transformation registry, transformation runs, and derived artifact lineage

docs/asset-registry-implementation.md

@@ -63,6 +63,8 @@ and SQLite repositories are adapters behind those ports.
 - Relationship changes create source-asset version records and audit events.
 - Idempotency records for safe asset creation retries.
 - Idempotency-key reuse with a different payload raises a validation error.
+- Transformation run and derived lineage persistence for traceable derived
+  artifact creation.
 - In-memory repository for deterministic tests.
 - SQLite repository for local-first durable asset registry state.
 - SQLite foreign-key enforcement for representation and metadata asset
@@ -84,11 +86,15 @@ and SQLite repositories are adapters behind those ports.
 - `audit_events`
 - `retrieval_feedback`
 - `idempotency_records`
+- `ingestion_jobs`
+- `transformation_runs`
+- `derived_lineage`
 
 Payloads are stored as compact JSON envelopes while indexed columns carry
 stable lookup fields such as asset ID, lifecycle, representation kind, digest,
-sequence, relationship source/target, actor ID, target, correlation ID, and
-idempotency key.
+sequence, relationship source/target, actor ID, target, correlation ID,
+idempotency key, transformation status, operation ID, and derived output asset
+ID.
 
 ## Not Yet Implemented
 

docs/transformation-implementation.md

@@ -0,0 +1,108 @@
+# Transformation Implementation Note
+
+Date: 2026-05-06
+
+Status: active implementation note for `KONT-WP-0008`.
+
+## Purpose
+
+This note records the first traceable transformation foundation. The goal is to
+turn governed source assets into governed derived assets without losing actor,
+policy, run, version, and lineage context.
+
+## Implemented Package Shape
+
+```text
+src/kontextual_engine/
+  core/
+    transformations.py
+    provenance.py
+  services/
+    transformation_service.py
+  ports/
+    repositories.py
+  adapters/
+    memory/asset_registry.py
+    sqlite/asset_registry.py
+```
+
+The transformation service depends on the asset registry repository and policy
+gateway ports. Derived outputs are persisted through `AssetRegistryService`
+rather than by bypassing existing asset governance.
+
+## Implemented Capabilities
+
+- `TransformationOperation` descriptors with input specs, output specs,
+  parameter schema, required permissions, supported asset types, adapter
+  references, and metadata.
+- `TransformationOperationRegistry` with a default operation catalog.
+- Executable engine-owned `structured_view` operation that produces a JSON
+  derived representation from source asset metadata and representation refs.
+- Adapter-backed operation descriptors for `summarize`, `classify`,
+  `generate_report`, and Markdown operations.
+- Markdown compose/include/transform/validate operations are registered with
+  `adapter_ref="markitect-tool"` and no engine-local handler.
+- Structured capability diagnostics for unsupported operations, unsupported
+  asset types, missing parameters, missing executable adapters, policy denial,
+  and execution failure.
+- `TransformationRun` records with queued, running, waiting, completed,
+  partially completed, failed, retried, and canceled states.
+- Run persistence in memory and SQLite repositories.
+- Source-read and run-execute policy decisions captured in run policy context
+  before transformation handler execution.
+- Derived artifact creation through the governed asset registry service.
+- Derived output versions use `VersionChangeType.DERIVED_OUTPUT`, store the
+  transformation run ID, and link to the first source parent version when
+  available.
+- `DerivedArtifactLineage` persistence and lookup by output asset, source
+  asset, or transformation run.
+- Audit events for queued, started, completed, failed, denied, and canceled
+  transformation runs.
+
+## markitect-tool Boundary
+
+Kontextual Engine owns run state, governance, derived artifact identity,
+lineage, policy context, and audit. Markitect owns Markdown-specific semantics.
+
+The engine therefore registers Markdown operations as adapter-backed
+capabilities but does not implement Markdown syntax processing itself. Once the
+adapter integration is wired, these operations should call Markitect functions
+through a stable boundary and keep the resulting run/output/lineage state in
+Kontextual Engine.
+
+## Current SQLite Tables
+
+WP-0008 adds shared registry persistence for:
+
+- `transformation_runs`
+- `derived_lineage`
+
+The tables store compact JSON payloads with indexed lookup columns for run ID,
+operation ID, status, actor ID, correlation ID, queued/updated timestamps,
+output asset ID, and transformation run lineage.
+
+## Not Yet Implemented
+
+- Workflow template persistence and invocation.
+- Step dependencies, preconditions, and workflow input binding.
+- Durable job runner execution model with resume semantics.
+- Rich retry policy by operation type.
+- Review gates, human tasks, and exception queues.
+- Full workflow reconstruction across template, step, run, derived output, and
+  audit records.
+- Concrete adapter execution for Markitect and provider-backed operations.
+
+These remain in open tasks `KONT-WP-0008-T004` through
+`KONT-WP-0008-T007`.
+
+## Test Coverage
+
+`tests/test_transformation_service.py` covers:
+
+- default operation registration and the Markitect adapter boundary,
+- structured view execution with derived asset, version, lineage, and audit
+  persistence,
+- SQLite reload of transformation runs and derived lineage,
+- adapter-backed operation failure without local Markdown reimplementation,
+- unsupported operation diagnostics without creating a run,
+- source-read policy denial before handler execution or output creation.