generated from coulomb/repo-seed
265 lines
9.0 KiB
Markdown
265 lines
9.0 KiB
Markdown
---
|
|
id: KONT-WP-0008
|
|
type: workplan
|
|
title: "Traceable Transformations And Workflow Jobs"
|
|
domain: markitect
|
|
repo: kontextual-engine
|
|
status: completed
|
|
owner: codex
|
|
topic_slug: markitect
|
|
planning_priority: high
|
|
planning_order: 8
|
|
created: "2026-05-05"
|
|
updated: "2026-05-06"
|
|
state_hub_workstream_id: "1b7a6b04-7879-4862-bb3e-817f7f20fc59"
|
|
---
|
|
|
|
# KONT-WP-0008: Traceable Transformations And Workflow Jobs
|
|
|
|
## Purpose
|
|
|
|
Implement the operations layer that turns knowledge assets into traceable
|
|
outputs: transformation operations, derived artifacts, workflow templates,
|
|
workflow runs, job execution state, retries, cancellation, review gates,
|
|
exception queues, and operation audit.
|
|
|
|
## Requirement Coverage
|
|
|
|
Primary: FR-080 to FR-090 and FR-100 to FR-110.
|
|
|
|
Supporting: FR-083 to FR-085, FR-106, FR-144 to FR-145, FR-165,
|
|
FR-200 to FR-202.
|
|
|
|
## Architecture Constraint
|
|
|
|
Implement transformations and workflows through operation registries, workflow
|
|
services, repository ports, event ports, policy checks, and audit events
|
|
described in `docs/architecture-blueprint.md`. Execution may start embedded,
|
|
but contracts must allow later queue or workflow-engine adapters.
|
|
|
|
## markitect-tool Boundary Remark
|
|
|
|
Markdown-specific transformations should delegate to Markitect operations,
|
|
contracts, runtime checks, templates, document functions, processors, and
|
|
workflow helpers. The engine owns the operation registry, run state, actors,
|
|
policy checks, derived artifact identity, lineage, retries, review gates, and
|
|
audit events.
|
|
|
|
## Implementation Status
|
|
|
|
The foundation slices are implemented for transformation operations,
|
|
transformation run persistence, derived artifact lineage, workflow templates,
|
|
the MVP durable job runner, review gates, exception queues, and audit
|
|
reconstruction. See `docs/transformation-implementation.md` and
|
|
`docs/workflow-jobs-implementation.md`.
|
|
|
|
## O8.1 - Implement transformation operation registry
|
|
|
|
```task
|
|
id: KONT-WP-0008-T001
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "ee2471b1-fab3-48f5-8b2d-d8f624abfc35"
|
|
```
|
|
|
|
Create a registry for transformation operations such as summarize, extract,
|
|
classify, compose, validate, generate report, and produce structured view.
|
|
|
|
Acceptance:
|
|
|
|
- Operations declare inputs, outputs, parameters, required permissions, and
|
|
supported asset types.
|
|
- Provider-specific LLM behavior remains behind adapters.
|
|
- Unsupported operations return structured capability errors.
|
|
- Markdown compose, include, transform, and validate operations are registered
|
|
as adapter-backed operations rather than reimplemented.
|
|
|
|
Implemented:
|
|
|
|
- `TransformationOperationRegistry` and default operation descriptors are in
|
|
`src/kontextual_engine/services/transformation_service.py`.
|
|
- `structured_view` is executable inside the engine as a generic derived
|
|
structured representation.
|
|
- Markdown operations are registered with `adapter_ref="markitect-tool"` and
|
|
no local handler, preserving the boundary against reimplementing Markitect.
|
|
|
|
## O8.2 - Implement transformation runs with parameters actors and policy context
|
|
|
|
```task
|
|
id: KONT-WP-0008-T002
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "1eac7b47-8cff-4736-9f7d-599123218bad"
|
|
```
|
|
|
|
Represent each transformation as a run with source assets, source versions,
|
|
operation type, parameters, actor, policy context, timestamps, and status.
|
|
|
|
Acceptance:
|
|
|
|
- Transformations can be queued, run, completed, failed, retried, or canceled.
|
|
- Transformation permissions are checked before reading sources or writing
|
|
outputs.
|
|
- Parameters needed to interpret or reproduce the run are preserved.
|
|
|
|
Implemented:
|
|
|
|
- `TransformationRun` records include operation, source assets, source
|
|
versions, parameters, actor, correlation ID, policy context, diagnostics,
|
|
timestamps, status, retry metadata, and output asset IDs.
|
|
- Source-read and run-execution policy decisions are captured before handler
|
|
execution.
|
|
- Run state is persisted in memory and SQLite repositories.
|
|
|
|
## O8.3 - Persist derived artifacts and source lineage
|
|
|
|
```task
|
|
id: KONT-WP-0008-T003
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "837ad793-2e9a-41f0-bce6-0a75815b5c15"
|
|
```
|
|
|
|
Persist summaries, extracts, reports, structured representations, generated
|
|
artifacts, and composed outputs as governed derived artifacts.
|
|
|
|
Acceptance:
|
|
|
|
- Derived artifacts have stable identity and lineage to source assets.
|
|
- Lineage includes transformation run, source versions, actor, parameters,
|
|
policy context, and output representation.
|
|
- Re-runs create new traceable records rather than silently overwriting outputs.
|
|
|
|
Implemented:
|
|
|
|
- Executed transformations can persist governed derived assets through
|
|
`AssetRegistryService`.
|
|
- `DerivedArtifactLineage` records are saved and queryable by output asset,
|
|
source asset, or transformation run.
|
|
- Derived asset versions use `VersionChangeType.DERIVED_OUTPUT` and point back
|
|
to the transformation run and source parent version.
|
|
|
|
## O8.4 - Implement workflow templates steps dependencies and preconditions
|
|
|
|
```task
|
|
id: KONT-WP-0008-T004
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "2c55c5dd-f07b-466b-85a5-f229e41fd124"
|
|
```
|
|
|
|
Define reusable workflow templates containing steps, dependencies, inputs,
|
|
outputs, preconditions, policy checks, and failure behavior.
|
|
|
|
Acceptance:
|
|
|
|
- Templates can be created and invoked programmatically.
|
|
- Step dependencies prevent unsafe or premature execution.
|
|
- Workflow inputs can be assets, collections, queries, source events, or
|
|
submitted payloads.
|
|
|
|
Implemented:
|
|
|
|
- `WorkflowTemplate`, `WorkflowInputDefinition`, and
|
|
`WorkflowStepDefinition` capture reusable templates, dependencies, inputs,
|
|
outputs, preconditions, policy checks, and failure behavior.
|
|
- Template registration validates missing dependencies, duplicate step IDs,
|
|
unsupported failure behavior, and dependency cycles.
|
|
- Workflow invocations preserve asset, collection, query, source event, and
|
|
payload input bindings; the MVP runner resolves asset bindings for
|
|
transformation-backed steps.
|
|
|
|
## O8.5 - Implement job runner status retry resume and cancel behavior
|
|
|
|
```task
|
|
id: KONT-WP-0008-T005
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "5f4d6c88-904d-4369-90d5-eaa4d27e3010"
|
|
```
|
|
|
|
Implement a simple MVP job runner for workflows and transformations.
|
|
|
|
Acceptance:
|
|
|
|
- Runs expose queued, running, waiting, completed, failed, retried, canceled,
|
|
and partially completed states.
|
|
- Safe retry, resume, and cancellation behavior is defined per operation.
|
|
- Recovery actions do not require direct storage edits.
|
|
|
|
Implemented:
|
|
|
|
- `WorkflowService` can queue, invoke, resume, retry, and cancel workflow
|
|
runs programmatically.
|
|
- The MVP runner executes transformation-backed steps in dependency order and
|
|
persists workflow run/step state in memory and SQLite repositories.
|
|
- Retry creates fresh governed outputs when fixed template output IDs would
|
|
collide with existing assets.
|
|
|
|
## O8.6 - Implement review gates human tasks and exception queues
|
|
|
|
```task
|
|
id: KONT-WP-0008-T006
|
|
status: done
|
|
priority: medium
|
|
state_hub_task_id: "5fae9005-4d64-4fca-8c51-a19405512377"
|
|
```
|
|
|
|
Add workflow primitives for review, approval, correction, rejection,
|
|
low-confidence handling, policy conflicts, and blocked exceptions.
|
|
|
|
Acceptance:
|
|
|
|
- Sensitive or high-impact outputs can pause for human review.
|
|
- Exception queues expose failed, blocked, low-confidence, policy-conflicted,
|
|
or review-required items.
|
|
- Review decisions continue, reject, correct, retry, or escalate runs.
|
|
|
|
Implemented:
|
|
|
|
- Workflow step definitions support `review_gate` settings.
|
|
- Review-required outputs pause step and run execution with embedded
|
|
`WorkflowReviewTask` and `WorkflowExceptionRecord` state.
|
|
- Review decisions can continue, reject, correct, retry, or escalate a run.
|
|
- Exception queue listing exposes review-required, failed, blocked,
|
|
low-confidence, and policy-conflicted workflow exceptions.
|
|
|
|
## O8.7 - Audit workflow and transformation operations
|
|
|
|
```task
|
|
id: KONT-WP-0008-T007
|
|
status: done
|
|
priority: medium
|
|
state_hub_task_id: "9e06aa46-3988-4389-99ec-0a934c68af1b"
|
|
```
|
|
|
|
Audit template changes, run starts, step executions, retries, cancellations,
|
|
approvals, failures, outputs, and derived artifact changes.
|
|
|
|
Acceptance:
|
|
|
|
- A workflow run can be reconstructed from run records and audit events.
|
|
- Audit records include actor, operation, target, outcome, correlation ID, and
|
|
policy context.
|
|
- Derived artifact audit events connect to source lineage.
|
|
|
|
Implemented:
|
|
|
|
- Workflow template registration, run queue/start/final states, step
|
|
executions, retries, cancellations, review requests, review decisions,
|
|
failures, and exceptions emit audit events.
|
|
- `WorkflowService.reconstruct_run` returns run state, template, audit events,
|
|
transformation runs, derived lineage, review tasks, and exceptions.
|
|
- Derived artifact lineage emits `derived_artifact.lineage.linked` audit events
|
|
connected to output asset, transformation run, source assets, source
|
|
versions, and output representation.
|
|
|
|
## Definition Of Done
|
|
|
|
- Transformations and workflows produce inspectable run records and audit
|
|
events.
|
|
- Derived artifacts are persistent, governed, and lineage-linked.
|
|
- Transformation and workflow run models follow
|
|
`docs/architecture-blueprint.md`.
|
|
- `python3 -m pytest` passes.
|