generated from coulomb/repo-seed
105 lines
4.0 KiB
Markdown
105 lines
4.0 KiB
Markdown
# Workflow Jobs Implementation Note
|
|
|
|
Date: 2026-05-06
|
|
|
|
Status: active implementation note for `KONT-WP-0008`.
|
|
|
|
## Purpose
|
|
|
|
This note records the first durable workflow and job-runner foundation. It
|
|
extends traceable transformations into reusable workflow templates with
|
|
dependency-ordered execution and recoverable run state.
|
|
|
|
## Implemented Package Shape
|
|
|
|
```text
|
|
src/kontextual_engine/
|
|
core/
|
|
workflow_jobs.py
|
|
services/
|
|
workflow_service.py
|
|
ports/
|
|
repositories.py
|
|
adapters/
|
|
memory/asset_registry.py
|
|
sqlite/asset_registry.py
|
|
```
|
|
|
|
The workflow service uses the existing repository and policy ports. Executable
|
|
steps delegate to `TransformationService`, so workflow runs do not bypass asset
|
|
governance, transformation run records, derived lineage, or audit.
|
|
|
|
## Implemented Capabilities
|
|
|
|
- `WorkflowTemplate` for reusable template identity, version, metadata,
|
|
inputs, steps, policy checks, and failure behavior.
|
|
- `WorkflowInputDefinition` supports asset, collection, query, source event,
|
|
and payload inputs.
|
|
- `WorkflowStepDefinition` captures step kind, operation ID, dependencies,
|
|
input/output bindings, preconditions, failure behavior, and metadata.
|
|
- Template registration validates duplicate step IDs, missing dependencies,
|
|
dependency cycles, and unsupported failure behavior.
|
|
- `WorkflowRun` and `WorkflowStepRun` persist queued, running, waiting,
|
|
completed, partially completed, failed, retried, and canceled states.
|
|
- `WorkflowService` can register templates, queue runs, invoke runs, resume
|
|
runs, retry runs, and cancel runs programmatically.
|
|
- The MVP runner executes transformation-backed steps in dependency order.
|
|
- Step inputs can resolve source asset IDs from invocation inputs or completed
|
|
upstream step outputs.
|
|
- Retry and repeated invocation avoid silent overwrite by choosing a fresh
|
|
output asset ID when a fixed template output ID already exists.
|
|
- Workflow template and run persistence are implemented for memory and SQLite.
|
|
- Audit events are emitted for template registration, run queue/start/final
|
|
states, step start/final states, retry, and cancellation.
|
|
|
|
## Current Boundaries
|
|
|
|
The runner executes only `kind="transformation"` steps. Non-transformation
|
|
steps are represented and fail with structured diagnostics until a later
|
|
adapter or job worker handles them.
|
|
|
|
Workflow inputs preserve collection, query, source event, and payload bindings,
|
|
but the MVP runner only interprets asset bindings for transformation execution.
|
|
Query expansion, source-event ingestion, human tasks, and exception queues stay
|
|
in later WP-0008 tasks.
|
|
|
|
Markdown-specific transformations remain adapter-backed through markitect-tool.
|
|
Workflow orchestration may invoke those operations once the adapter boundary is
|
|
wired, but the engine does not reimplement Markdown semantics.
|
|
|
|
## Current SQLite Tables
|
|
|
|
WP-0008 adds shared registry persistence for:
|
|
|
|
- `workflow_templates`
|
|
- `workflow_runs`
|
|
|
|
The tables store compact JSON payloads with indexed lookup columns for template
|
|
ID, template version, template name, run ID, run status, actor ID, correlation
|
|
ID, queued timestamp, and updated timestamp.
|
|
|
|
## Not Yet Implemented
|
|
|
|
- Human review gates and approval tasks.
|
|
- Exception queues for blocked, failed, low-confidence, or policy-conflicted
|
|
items.
|
|
- Queue-worker adapters beyond embedded execution.
|
|
- Rich retry policies by operation type.
|
|
- Query/input expansion into dynamic asset sets.
|
|
- Full workflow reconstruction views across all audit and lineage records.
|
|
|
|
These remain in open tasks `KONT-WP-0008-T006` and `KONT-WP-0008-T007`.
|
|
|
|
## Test Coverage
|
|
|
|
`tests/test_workflow_service.py` covers:
|
|
|
|
- template registration, input-kind persistence, missing dependency rejection,
|
|
and dependency-cycle rejection,
|
|
- dependency-ordered execution of two transformation steps,
|
|
- queue, cancel, resume denial, and retry without direct storage edits,
|
|
- partial completion when a continue-on-failure step fails and another step
|
|
succeeds,
|
|
- SQLite reload of workflow templates, workflow runs, step state, and derived
|
|
output representation state.
|