kontextual-engine/docs/workflow-jobs-implementation.md

Workflow Jobs Implementation Note

Date: 2026-05-06

Status: completed foundation implementation note for KONT-WP-0008.

Purpose

This note records the first durable workflow and job-runner foundation. It extends traceable transformations into reusable workflow templates with dependency-ordered execution and recoverable run state.

Implemented Package Shape

src/kontextual_engine/
  core/
    workflow_jobs.py
  services/
    workflow_service.py
  ports/
    repositories.py
  adapters/
    memory/asset_registry.py
    sqlite/asset_registry.py

The workflow service uses the existing repository and policy ports. Executable steps delegate to TransformationService, so workflow runs do not bypass asset governance, transformation run records, derived lineage, or audit.

Implemented Capabilities

• WorkflowTemplate for reusable template identity, version, metadata, inputs, steps, policy checks, and failure behavior.
• WorkflowInputDefinition supports asset, collection, query, source event, and payload inputs.
• WorkflowStepDefinition captures step kind, operation ID, dependencies, input/output bindings, preconditions, review gates, failure behavior, and metadata.
• Template registration validates duplicate step IDs, missing dependencies, dependency cycles, and unsupported failure behavior.
• WorkflowRun and WorkflowStepRun persist queued, running, waiting, completed, partially completed, failed, retried, and canceled states.
• WorkflowService can register templates, queue runs, invoke runs, resume runs, retry runs, and cancel runs programmatically.
• The MVP runner executes transformation-backed steps in dependency order.
• Step inputs can resolve source asset IDs from invocation inputs or completed upstream step outputs.
• Retry and repeated invocation avoid silent overwrite by choosing a fresh output asset ID when a fixed template output ID already exists.
• Workflow template and run persistence are implemented for memory and SQLite.
• Review-required outputs pause the step and workflow run with embedded WorkflowReviewTask and WorkflowExceptionRecord state.
• Review decisions can continue, reject, correct, retry, or escalate workflow runs.
• Exception queue listing exposes review-required, failed, blocked, low-confidence, and policy-conflicted items.
• Audit events are emitted for template registration, run queue/start/final states, step start/final states, retry, cancellation, review requests, review decisions, and exception opening.
• WorkflowService.reconstruct_run returns run state, template, audit events, transformation runs, derived lineage, review tasks, and exceptions.

Current Boundaries

The runner executes only kind="transformation" steps. Non-transformation steps are represented and fail with structured diagnostics until a later adapter or job worker handles them.

Workflow inputs preserve collection, query, source event, and payload bindings, but the MVP runner only interprets asset bindings for transformation execution. Query expansion, source-event ingestion, and external queue-worker adapters stay in later implementation work.

Markdown-specific transformations remain adapter-backed through markitect-tool. Workflow orchestration may invoke those operations once the adapter boundary is wired, but the engine does not reimplement Markdown semantics.

Current SQLite Tables

WP-0008 adds shared registry persistence for:

• workflow_templates
• workflow_runs

The tables store compact JSON payloads with indexed lookup columns for template ID, template version, template name, run ID, run status, actor ID, correlation ID, queued timestamp, and updated timestamp.

Not Yet Implemented

• Queue-worker adapters beyond embedded execution.
• Rich retry policies by operation type.
• Query/input expansion into dynamic asset sets.
• Product/API views for review queues and reconstruction records.

Test Coverage

tests/test_workflow_service.py covers:

• template registration, input-kind persistence, missing dependency rejection, and dependency-cycle rejection,
• dependency-ordered execution of two transformation steps,
• queue, cancel, resume denial, and retry without direct storage edits,
• partial completion when a continue-on-failure step fails and another step succeeds,
• review gate pause, continue, reject, correct, retry, and escalation decisions,
• failed step exception queue items,
• reconstruction across workflow audit, transformation runs, and derived lineage,
• SQLite reload of workflow templates, workflow runs, step state, review state, exception state, and derived output representation state.