generated from coulomb/repo-seed
114 lines
4.6 KiB
Markdown
114 lines
4.6 KiB
Markdown
# Workflow Jobs Implementation Note
|
|
|
|
Date: 2026-05-06
|
|
|
|
Status: completed foundation implementation note for `KONT-WP-0008`.
|
|
|
|
## Purpose
|
|
|
|
This note records the first durable workflow and job-runner foundation. It
|
|
extends traceable transformations into reusable workflow templates with
|
|
dependency-ordered execution and recoverable run state.
|
|
|
|
## Implemented Package Shape
|
|
|
|
```text
|
|
src/kontextual_engine/
|
|
core/
|
|
workflow_jobs.py
|
|
services/
|
|
workflow_service.py
|
|
ports/
|
|
repositories.py
|
|
adapters/
|
|
memory/asset_registry.py
|
|
sqlite/asset_registry.py
|
|
```
|
|
|
|
The workflow service uses the existing repository and policy ports. Executable
|
|
steps delegate to `TransformationService`, so workflow runs do not bypass asset
|
|
governance, transformation run records, derived lineage, or audit.
|
|
|
|
## Implemented Capabilities
|
|
|
|
- `WorkflowTemplate` for reusable template identity, version, metadata,
|
|
inputs, steps, policy checks, and failure behavior.
|
|
- `WorkflowInputDefinition` supports asset, collection, query, source event,
|
|
and payload inputs.
|
|
- `WorkflowStepDefinition` captures step kind, operation ID, dependencies,
|
|
input/output bindings, preconditions, review gates, failure behavior, and
|
|
metadata.
|
|
- Template registration validates duplicate step IDs, missing dependencies,
|
|
dependency cycles, and unsupported failure behavior.
|
|
- `WorkflowRun` and `WorkflowStepRun` persist queued, running, waiting,
|
|
completed, partially completed, failed, retried, and canceled states.
|
|
- `WorkflowService` can register templates, queue runs, invoke runs, resume
|
|
runs, retry runs, and cancel runs programmatically.
|
|
- The MVP runner executes transformation-backed steps in dependency order.
|
|
- Step inputs can resolve source asset IDs from invocation inputs or completed
|
|
upstream step outputs.
|
|
- Retry and repeated invocation avoid silent overwrite by choosing a fresh
|
|
output asset ID when a fixed template output ID already exists.
|
|
- Workflow template and run persistence are implemented for memory and SQLite.
|
|
- Review-required outputs pause the step and workflow run with embedded
|
|
`WorkflowReviewTask` and `WorkflowExceptionRecord` state.
|
|
- Review decisions can continue, reject, correct, retry, or escalate workflow
|
|
runs.
|
|
- Exception queue listing exposes review-required, failed, blocked,
|
|
low-confidence, and policy-conflicted items.
|
|
- Audit events are emitted for template registration, run queue/start/final
|
|
states, step start/final states, retry, cancellation, review requests,
|
|
review decisions, and exception opening.
|
|
- `WorkflowService.reconstruct_run` returns run state, template, audit events,
|
|
transformation runs, derived lineage, review tasks, and exceptions.
|
|
|
|
## Current Boundaries
|
|
|
|
The runner executes only `kind="transformation"` steps. Non-transformation
|
|
steps are represented and fail with structured diagnostics until a later
|
|
adapter or job worker handles them.
|
|
|
|
Workflow inputs preserve collection, query, source event, and payload bindings,
|
|
but the MVP runner only interprets asset bindings for transformation execution.
|
|
Query expansion, source-event ingestion, and external queue-worker adapters stay
|
|
in later implementation work.
|
|
|
|
Markdown-specific transformations remain adapter-backed through markitect-tool.
|
|
Workflow orchestration may invoke those operations once the adapter boundary is
|
|
wired, but the engine does not reimplement Markdown semantics.
|
|
|
|
## Current SQLite Tables
|
|
|
|
WP-0008 adds shared registry persistence for:
|
|
|
|
- `workflow_templates`
|
|
- `workflow_runs`
|
|
|
|
The tables store compact JSON payloads with indexed lookup columns for template
|
|
ID, template version, template name, run ID, run status, actor ID, correlation
|
|
ID, queued timestamp, and updated timestamp.
|
|
|
|
## Not Yet Implemented
|
|
|
|
- Queue-worker adapters beyond embedded execution.
|
|
- Rich retry policies by operation type.
|
|
- Query/input expansion into dynamic asset sets.
|
|
- Product/API views for review queues and reconstruction records.
|
|
|
|
## Test Coverage
|
|
|
|
`tests/test_workflow_service.py` covers:
|
|
|
|
- template registration, input-kind persistence, missing dependency rejection,
|
|
and dependency-cycle rejection,
|
|
- dependency-ordered execution of two transformation steps,
|
|
- queue, cancel, resume denial, and retry without direct storage edits,
|
|
- partial completion when a continue-on-failure step fails and another step
|
|
succeeds,
|
|
- review gate pause, continue, reject, correct, retry, and escalation decisions,
|
|
- failed step exception queue items,
|
|
- reconstruction across workflow audit, transformation runs, and derived
|
|
lineage,
|
|
- SQLite reload of workflow templates, workflow runs, step state, review state,
|
|
exception state, and derived output representation state.
|