kontextual-engine/docs/transformation-implementation.md

Transformation Implementation Note

Date: 2026-05-06

Status: active implementation note for KONT-WP-0008.

Purpose

This note records the first traceable transformation foundation. The goal is to turn governed source assets into governed derived assets without losing actor, policy, run, version, and lineage context.

Implemented Package Shape

src/kontextual_engine/
  core/
    transformations.py
    provenance.py
  services/
    transformation_service.py
  ports/
    repositories.py
  adapters/
    memory/asset_registry.py
    sqlite/asset_registry.py

The transformation service depends on the asset registry repository and policy gateway ports. Derived outputs are persisted through AssetRegistryService rather than by bypassing existing asset governance.

Implemented Capabilities

• TransformationOperation descriptors with input specs, output specs, parameter schema, required permissions, supported asset types, adapter references, and metadata.
• TransformationOperationRegistry with a default operation catalog.
• Executable engine-owned structured_view operation that produces a JSON derived representation from source asset metadata and representation refs.
• Adapter-backed operation descriptors for summarize, classify, generate_report, and Markdown operations.
• Markdown compose/include/transform/validate operations are registered with adapter_ref="markitect-tool" and no engine-local handler.
• Structured capability diagnostics for unsupported operations, unsupported asset types, missing parameters, missing executable adapters, policy denial, and execution failure.
• TransformationRun records with queued, running, waiting, completed, partially completed, failed, retried, and canceled states.
• Run persistence in memory and SQLite repositories.
• Source-read and run-execute policy decisions captured in run policy context before transformation handler execution.
• Derived artifact creation through the governed asset registry service.
• Derived output versions use VersionChangeType.DERIVED_OUTPUT, store the transformation run ID, and link to the first source parent version when available.
• DerivedArtifactLineage persistence and lookup by output asset, source asset, or transformation run.
• derived_artifact.lineage.linked audit events connect output assets to transformation runs, source assets, source versions, and output representations.
• Audit events for queued, started, completed, failed, denied, and canceled transformation runs.

markitect-tool Boundary

Kontextual Engine owns run state, governance, derived artifact identity, lineage, policy context, and audit. Markitect owns Markdown-specific semantics.

The engine therefore registers Markdown operations as adapter-backed capabilities but does not implement Markdown syntax processing itself. Once the adapter integration is wired, these operations should call Markitect functions through a stable boundary and keep the resulting run/output/lineage state in Kontextual Engine.

Current SQLite Tables

WP-0008 adds shared registry persistence for:

• transformation_runs
• derived_lineage

The tables store compact JSON payloads with indexed lookup columns for run ID, operation ID, status, actor ID, correlation ID, queued/updated timestamps, output asset ID, and transformation run lineage.

Not Yet Implemented

• Workflow template persistence and invocation.
• Step dependencies, preconditions, and workflow input binding.
• Durable job runner execution model with resume semantics.
• Rich retry policy by operation type.
• Review gates, human tasks, and exception queues.
• Full workflow reconstruction across template, step, run, derived output, and audit records.
• Concrete adapter execution for Markitect and provider-backed operations.

These remain in open tasks KONT-WP-0008-T004 through KONT-WP-0008-T007.

Test Coverage

tests/test_transformation_service.py covers:

• default operation registration and the Markitect adapter boundary,
• structured view execution with derived asset, version, lineage, and audit persistence,
• SQLite reload of transformation runs and derived lineage,
• adapter-backed operation failure without local Markdown reimplementation,
• unsupported operation diagnostics without creating a run,
• source-read policy denial before handler execution or output creation.