kontextual-engine/workplans/KONT-WP-0008-transformations-workflow-jobs.md

id, type, title, domain, repo, status, owner, topic_slug, planning_priority, planning_order, created, updated, state_hub_workstream_id

id	type	title	domain	repo	status	owner	topic_slug	planning_priority	planning_order	created	updated	state_hub_workstream_id
KONT-WP-0008	workplan	Traceable Transformations And Workflow Jobs	markitect	kontextual-engine	completed	codex	markitect	high	8	2026-05-05	2026-05-06	1b7a6b04-7879-4862-bb3e-817f7f20fc59

KONT-WP-0008: Traceable Transformations And Workflow Jobs

Purpose

Implement the operations layer that turns knowledge assets into traceable outputs: transformation operations, derived artifacts, workflow templates, workflow runs, job execution state, retries, cancellation, review gates, exception queues, and operation audit.

Requirement Coverage

Primary: FR-080 to FR-090 and FR-100 to FR-110.

Supporting: FR-083 to FR-085, FR-106, FR-144 to FR-145, FR-165, FR-200 to FR-202.

Architecture Constraint

Implement transformations and workflows through operation registries, workflow services, repository ports, event ports, policy checks, and audit events described in docs/architecture-blueprint.md. Execution may start embedded, but contracts must allow later queue or workflow-engine adapters.

markitect-tool Boundary Remark

Markdown-specific transformations should delegate to Markitect operations, contracts, runtime checks, templates, document functions, processors, and workflow helpers. The engine owns the operation registry, run state, actors, policy checks, derived artifact identity, lineage, retries, review gates, and audit events.

Implementation Status

The foundation slices are implemented for transformation operations, transformation run persistence, derived artifact lineage, workflow templates, the MVP durable job runner, review gates, exception queues, and audit reconstruction. See docs/transformation-implementation.md and docs/workflow-jobs-implementation.md.

O8.1 - Implement transformation operation registry

id: KONT-WP-0008-T001
status: done
priority: high
state_hub_task_id: "ee2471b1-fab3-48f5-8b2d-d8f624abfc35"

Create a registry for transformation operations such as summarize, extract, classify, compose, validate, generate report, and produce structured view.

Acceptance:

• Operations declare inputs, outputs, parameters, required permissions, and supported asset types.
• Provider-specific LLM behavior remains behind adapters.
• Unsupported operations return structured capability errors.
• Markdown compose, include, transform, and validate operations are registered as adapter-backed operations rather than reimplemented.

Implemented:

• TransformationOperationRegistry and default operation descriptors are in src/kontextual_engine/services/transformation_service.py.
• structured_view is executable inside the engine as a generic derived structured representation.
• Markdown operations are registered with adapter_ref="markitect-tool" and no local handler, preserving the boundary against reimplementing Markitect.

O8.2 - Implement transformation runs with parameters actors and policy context

id: KONT-WP-0008-T002
status: done
priority: high
state_hub_task_id: "1eac7b47-8cff-4736-9f7d-599123218bad"

Represent each transformation as a run with source assets, source versions, operation type, parameters, actor, policy context, timestamps, and status.

Acceptance:

• Transformations can be queued, run, completed, failed, retried, or canceled.
• Transformation permissions are checked before reading sources or writing outputs.
• Parameters needed to interpret or reproduce the run are preserved.

Implemented:

• TransformationRun records include operation, source assets, source versions, parameters, actor, correlation ID, policy context, diagnostics, timestamps, status, retry metadata, and output asset IDs.
• Source-read and run-execution policy decisions are captured before handler execution.
• Run state is persisted in memory and SQLite repositories.

O8.3 - Persist derived artifacts and source lineage

id: KONT-WP-0008-T003
status: done
priority: high
state_hub_task_id: "837ad793-2e9a-41f0-bce6-0a75815b5c15"

Persist summaries, extracts, reports, structured representations, generated artifacts, and composed outputs as governed derived artifacts.

Acceptance:

• Derived artifacts have stable identity and lineage to source assets.
• Lineage includes transformation run, source versions, actor, parameters, policy context, and output representation.
• Re-runs create new traceable records rather than silently overwriting outputs.

Implemented:

• Executed transformations can persist governed derived assets through AssetRegistryService.
• DerivedArtifactLineage records are saved and queryable by output asset, source asset, or transformation run.
• Derived asset versions use VersionChangeType.DERIVED_OUTPUT and point back to the transformation run and source parent version.

O8.4 - Implement workflow templates steps dependencies and preconditions

id: KONT-WP-0008-T004
status: done
priority: high
state_hub_task_id: "2c55c5dd-f07b-466b-85a5-f229e41fd124"

Define reusable workflow templates containing steps, dependencies, inputs, outputs, preconditions, policy checks, and failure behavior.

Acceptance:

• Templates can be created and invoked programmatically.
• Step dependencies prevent unsafe or premature execution.
• Workflow inputs can be assets, collections, queries, source events, or submitted payloads.

Implemented:

• WorkflowTemplate, WorkflowInputDefinition, and WorkflowStepDefinition capture reusable templates, dependencies, inputs, outputs, preconditions, policy checks, and failure behavior.
• Template registration validates missing dependencies, duplicate step IDs, unsupported failure behavior, and dependency cycles.
• Workflow invocations preserve asset, collection, query, source event, and payload input bindings; the MVP runner resolves asset bindings for transformation-backed steps.

O8.5 - Implement job runner status retry resume and cancel behavior

id: KONT-WP-0008-T005
status: done
priority: high
state_hub_task_id: "5f4d6c88-904d-4369-90d5-eaa4d27e3010"

Implement a simple MVP job runner for workflows and transformations.

Acceptance:

• Runs expose queued, running, waiting, completed, failed, retried, canceled, and partially completed states.
• Safe retry, resume, and cancellation behavior is defined per operation.
• Recovery actions do not require direct storage edits.

Implemented:

• WorkflowService can queue, invoke, resume, retry, and cancel workflow runs programmatically.
• The MVP runner executes transformation-backed steps in dependency order and persists workflow run/step state in memory and SQLite repositories.
• Retry creates fresh governed outputs when fixed template output IDs would collide with existing assets.

O8.6 - Implement review gates human tasks and exception queues

id: KONT-WP-0008-T006
status: done
priority: medium
state_hub_task_id: "5fae9005-4d64-4fca-8c51-a19405512377"

Add workflow primitives for review, approval, correction, rejection, low-confidence handling, policy conflicts, and blocked exceptions.

Acceptance:

• Sensitive or high-impact outputs can pause for human review.
• Exception queues expose failed, blocked, low-confidence, policy-conflicted, or review-required items.
• Review decisions continue, reject, correct, retry, or escalate runs.

Implemented:

• Workflow step definitions support review_gate settings.
• Review-required outputs pause step and run execution with embedded WorkflowReviewTask and WorkflowExceptionRecord state.
• Review decisions can continue, reject, correct, retry, or escalate a run.
• Exception queue listing exposes review-required, failed, blocked, low-confidence, and policy-conflicted workflow exceptions.

O8.7 - Audit workflow and transformation operations

id: KONT-WP-0008-T007
status: done
priority: medium
state_hub_task_id: "9e06aa46-3988-4389-99ec-0a934c68af1b"

Audit template changes, run starts, step executions, retries, cancellations, approvals, failures, outputs, and derived artifact changes.

Acceptance:

• A workflow run can be reconstructed from run records and audit events.
• Audit records include actor, operation, target, outcome, correlation ID, and policy context.
• Derived artifact audit events connect to source lineage.

Implemented:

• Workflow template registration, run queue/start/final states, step executions, retries, cancellations, review requests, review decisions, failures, and exceptions emit audit events.
• WorkflowService.reconstruct_run returns run state, template, audit events, transformation runs, derived lineage, review tasks, and exceptions.
• Derived artifact lineage emits derived_artifact.lineage.linked audit events connected to output asset, transformation run, source assets, source versions, and output representation.

Definition Of Done

• Transformations and workflows produce inspectable run records and audit events.
• Derived artifacts are persistent, governed, and lineage-linked.
• Transformation and workflow run models follow docs/architecture-blueprint.md.
• python3 -m pytest passes.