kontextual-engine/docs/workflow-jobs-implementation.md

Workflow Jobs Implementation Note

Date: 2026-05-06

Status: active implementation note for KONT-WP-0008.

Purpose

This note records the first durable workflow and job-runner foundation. It extends traceable transformations into reusable workflow templates with dependency-ordered execution and recoverable run state.

Implemented Package Shape

src/kontextual_engine/
  core/
    workflow_jobs.py
  services/
    workflow_service.py
  ports/
    repositories.py
  adapters/
    memory/asset_registry.py
    sqlite/asset_registry.py

The workflow service uses the existing repository and policy ports. Executable steps delegate to TransformationService, so workflow runs do not bypass asset governance, transformation run records, derived lineage, or audit.

Implemented Capabilities

• WorkflowTemplate for reusable template identity, version, metadata, inputs, steps, policy checks, and failure behavior.
• WorkflowInputDefinition supports asset, collection, query, source event, and payload inputs.
• WorkflowStepDefinition captures step kind, operation ID, dependencies, input/output bindings, preconditions, failure behavior, and metadata.
• Template registration validates duplicate step IDs, missing dependencies, dependency cycles, and unsupported failure behavior.
• WorkflowRun and WorkflowStepRun persist queued, running, waiting, completed, partially completed, failed, retried, and canceled states.
• WorkflowService can register templates, queue runs, invoke runs, resume runs, retry runs, and cancel runs programmatically.
• The MVP runner executes transformation-backed steps in dependency order.
• Step inputs can resolve source asset IDs from invocation inputs or completed upstream step outputs.
• Retry and repeated invocation avoid silent overwrite by choosing a fresh output asset ID when a fixed template output ID already exists.
• Workflow template and run persistence are implemented for memory and SQLite.
• Audit events are emitted for template registration, run queue/start/final states, step start/final states, retry, and cancellation.

Current Boundaries

The runner executes only kind="transformation" steps. Non-transformation steps are represented and fail with structured diagnostics until a later adapter or job worker handles them.

Workflow inputs preserve collection, query, source event, and payload bindings, but the MVP runner only interprets asset bindings for transformation execution. Query expansion, source-event ingestion, human tasks, and exception queues stay in later WP-0008 tasks.

Markdown-specific transformations remain adapter-backed through markitect-tool. Workflow orchestration may invoke those operations once the adapter boundary is wired, but the engine does not reimplement Markdown semantics.

Current SQLite Tables

WP-0008 adds shared registry persistence for:

• workflow_templates
• workflow_runs

The tables store compact JSON payloads with indexed lookup columns for template ID, template version, template name, run ID, run status, actor ID, correlation ID, queued timestamp, and updated timestamp.

Not Yet Implemented

• Human review gates and approval tasks.
• Exception queues for blocked, failed, low-confidence, or policy-conflicted items.
• Queue-worker adapters beyond embedded execution.
• Rich retry policies by operation type.
• Query/input expansion into dynamic asset sets.
• Full workflow reconstruction views across all audit and lineage records.

These remain in open tasks KONT-WP-0008-T006 and KONT-WP-0008-T007.

Test Coverage

tests/test_workflow_service.py covers:

• template registration, input-kind persistence, missing dependency rejection, and dependency-cycle rejection,
• dependency-ordered execution of two transformation steps,
• queue, cancel, resume denial, and retry without direct storage edits,
• partial completion when a continue-on-failure step fails and another step succeeds,
• SQLite reload of workflow templates, workflow runs, step state, and derived output representation state.