From aed08061f7948bae5b58ac260ff91ae1c70ed115 Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 5 May 2026 10:37:01 +0200 Subject: [PATCH] System boundry and persistance workplan --- README.md | 1 + docs/phase-memory-boundary.md | 107 +++++++++ workplans/KONT-WP-0004-durable-persistence.md | 226 ++++++++++++++++++ 3 files changed, 334 insertions(+) create mode 100644 docs/phase-memory-boundary.md create mode 100644 workplans/KONT-WP-0004-durable-persistence.md diff --git a/README.md b/README.md index 4ed4a33..82a05bb 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ Start here: - `docs/stack-decision.md` - `docs/markitect-main-scope-assessment.md` - `docs/markitect-tool-reuse-boundary.md` +- `docs/phase-memory-boundary.md` - `docs/system-layer-extraction-inventory.md` - `docs/system-layer-migration-backlog.md` - `docs/service-api-boundary.md` diff --git a/docs/phase-memory-boundary.md b/docs/phase-memory-boundary.md new file mode 100644 index 0000000..95712f5 --- /dev/null +++ b/docs/phase-memory-boundary.md @@ -0,0 +1,107 @@ +# Phase Memory Boundary + +Date: 2026-05-05 + +This note defines how `kontextual-engine` should relate to the sibling +`phase-memory` project while closing the current persistence gap. + +## Source Scope + +`phase-memory` exists to provide profile-driven memory infrastructure for +agentic systems. Its intent covers memory phases, memory graphs, event paths, +stores, compaction, retention, service profiles, activation planning, and +adapters for advanced memory backends. + +`kontextual-engine` exists to provide an AI-first, headless knowledge and +content engine. Its intent covers persistent knowledge artifacts, collections, +relationships, ingestion, workflow state, query, service access, and agent- +operable knowledge processes. + +The projects overlap around durable state, graph-shaped records, context +packages, provenance, and agent operation. They must stay separated by purpose. + +## Ownership Boundary + +`kontextual-engine` owns: + +- Persistent knowledge collections, artifacts, relationships, and revisions. +- Content identity, artifact metadata, source provenance, and lifecycle state. +- Durable workflow run records and operation manifests. +- Query and retrieval over knowledge artifacts and relationships. +- Service and programmatic APIs for operating knowledge systems. +- Adapter surfaces that can call `markitect-tool`, `phase-memory`, storage + backends, and future workflow engines. + +`phase-memory` owns: + +- Ephemeral, fluid, stabilized, and rigid memory phase semantics. +- Reasoning graphs, conversational paths, activation memory, identity memory, + preference memory, and reviewed memory records. +- Memory profiles with latency, token, byte, retention, decay, compaction, + freshness, merge, conflict, and policy parameters. +- Memory-specific lifecycle enforcement, including retention, deletion, + stabilization, compaction, and summarization. +- Memory retrieval planning and activation-package compilation. +- Adapters to graph databases, vector stores, audit sinks, policy systems, and + memory-specific backend infrastructure. + +## Interop Boundary + +`kontextual-engine` may persist memory-related artifacts when they are treated +as knowledge assets, such as: + +- reviewed decisions, +- source-backed facts, +- durable project context packages, +- workflow outputs, +- policy-visible provenance records, +- exported memory graph slices. + +It should not implement memory-specific behavior such as: + +- conversational path branching and merge semantics, +- memory decay and retention policy execution, +- compaction or summarization strategies, +- activation planning heuristics, +- preference or identity memory governance, +- graph/vector memory backend selection, +- memory service profile validation. + +Those behaviors belong to `phase-memory`. `kontextual-engine` should integrate +with them through explicit adapters and stable exchange formats. + +## Persistence Implication + +The persistence gap in `kontextual-engine` should be closed as durable knowledge +runtime storage, not as a general memory runtime. + +The first durable backend should persist: + +- collections, +- artifacts, +- artifact revisions, +- relationships, +- workflow runs, +- run manifests, +- change records, +- queryable metadata, +- context-package references. + +It should not persist phase-memory internals beyond opaque references or +exported artifacts unless an integration adapter explicitly maps them. + +## Integration Shape + +The preferred integration shape is: + +1. Core `kontextual-engine` repository contracts remain independent. +2. A future `phase-memory` adapter can read/write durable knowledge artifacts + through those contracts. +3. Context packages can move between the systems as inspectable payloads. +4. Memory-specific policies stay in `phase-memory`; knowledge persistence + policy stays in `kontextual-engine`. +5. Cross-repo references should use stable identifiers, content digests, and + provenance metadata rather than implicit filesystem coupling. + +This keeps `kontextual-engine` focused on persistent, operable knowledge while +leaving sophisticated agentic memory behavior to the project designed for it. diff --git a/workplans/KONT-WP-0004-durable-persistence.md b/workplans/KONT-WP-0004-durable-persistence.md new file mode 100644 index 0000000..c5a2e4a --- /dev/null +++ b/workplans/KONT-WP-0004-durable-persistence.md @@ -0,0 +1,226 @@ +--- +id: KONT-WP-0004 +type: workplan +title: "Durable Persistence Foundation" +domain: markitect +repo: kontextual-engine +status: planned +owner: codex +topic_slug: markitect +created: "2026-05-05" +updated: "2026-05-05" +state_hub_workstream_id: pending +--- + +# KONT-WP-0004: Durable Persistence Foundation + +## Purpose + +Close the persistence gap identified after `KONT-WP-0003` by turning the current +in-memory repository contract into a durable, local-first storage foundation for +knowledge artifacts, collections, relationships, workflow state, and context +references. + +This workplan deliberately does not implement `phase-memory` behavior. It uses +`docs/phase-memory-boundary.md` as the boundary: `kontextual-engine` persists +durable knowledge runtime state; `phase-memory` owns agentic memory phases, +profiles, compaction, retention, and activation planning. + +## Persistence Scope + +In scope: + +- Durable storage for collections, artifacts, artifact revisions, and + relationships. +- Durable storage for workflow runs and run manifests. +- Explicit update and delete behavior for artifacts and relationships. +- Change records that make artifact evolution inspectable. +- Query support for identifiers, names, digests, metadata, content text, and + relationships. +- Local-first SQLite backend with a repository interface that can later be + backed by service, PostgreSQL, graph, or object-storage adapters. +- Tests proving data survives repository re-instantiation. + +Out of scope: + +- Memory-phase lifecycle behavior from `phase-memory`. +- Vector search, embedding storage, and memory activation planning. +- Markdown parsing or markdown transformations; use `markitect-tool` adapters. +- LLM provider execution; use future `llm-connect` adapters. +- Remote multi-tenant deployment concerns beyond schema choices that do not + block later migration. + +## P4.1 - Finalize persistence boundary and ADR + +```task +id: KONT-WP-0004-T001 +status: todo +priority: high +``` + +Promote the storage decision from deferred to explicit: local-first SQLite for +the first durable backend, wrapped by repository contracts. Decide whether the +implementation uses direct `sqlite3` or SQLAlchemy for this slice. + +Output: update `docs/stack-decision.md` or add an ADR under `docs/`. + +Acceptance: + +- The backend choice is explicit and justified. +- The decision references `docs/phase-memory-boundary.md`. +- Future service-backed storage remains possible. + +## P4.2 - Complete repository contract semantics + +```task +id: KONT-WP-0004-T002 +status: todo +priority: high +``` + +Extend `KnowledgeRepository` from create/list/get into a durable lifecycle +contract. Define update, delete, revision, and transaction semantics without +binding callers to a specific backend. + +Output: `src/kontextual_engine/storage.py` and focused tests. + +Acceptance: + +- Artifact update produces explicit revision/change semantics. +- Artifact delete behavior is defined for relationships and query results. +- Duplicate-name and referential-integrity behavior remains deterministic. +- Existing in-memory tests continue to pass. + +## P4.3 - Design durable schema and migrations + +```task +id: KONT-WP-0004-T003 +status: todo +priority: high +``` + +Create the first durable schema for collections, artifacts, revisions, +relationships, workflow runs, run manifests, and change records. + +Output: schema/migration files under `src/kontextual_engine/storage/` or an +equivalent package-owned location. + +Acceptance: + +- Schema stores content digest, artifact type, size, metadata, timestamps, and + provenance. +- Relationships enforce valid source and target artifacts. +- JSON metadata is preserved roundtrip. +- Migrations can initialize an empty local database deterministically. + +## P4.4 - Implement SQLite repository backend + +```task +id: KONT-WP-0004-T004 +status: todo +priority: high +``` + +Implement `SQLiteKnowledgeRepository` behind the same repository contract used +by the in-memory backend. + +Output: durable repository implementation and tests. + +Acceptance: + +- Collections, artifacts, relationships, and metadata survive closing and + reopening the repository. +- Query behavior matches the in-memory repository for supported filters. +- Tests cover duplicate artifact names, missing relationship endpoints, and + deterministic ordering. +- No markdown or memory-runtime logic is introduced. + +## P4.5 - Persist artifact evolution + +```task +id: KONT-WP-0004-T005 +status: todo +priority: high +``` + +Add artifact revision and change-record support so persistent knowledge can be +versioned and audited over time. + +Output: model additions, repository methods, and tests. + +Acceptance: + +- Updating an artifact records old and new digests. +- Revision history can be retrieved by artifact id. +- Deletion is traceable through a change record. +- Change records are backend-neutral at the programmatic API boundary. + +## P4.6 - Persist workflow run state + +```task +id: KONT-WP-0004-T006 +status: todo +priority: medium +``` + +Persist `OperationRun`, `WorkflowStep`, `InputBundle`, and `RunManifest` +records so orchestration can resume and inspect prior execution. + +Output: repository methods and persistence tests for workflow records. + +Acceptance: + +- Run status transitions survive repository re-instantiation. +- Run manifests roundtrip with inputs, outputs, diagnostics, and timestamps. +- Artifact outputs can be linked to producing runs. + +## P4.7 - Add context and phase-memory reference hooks + +```task +id: KONT-WP-0004-T007 +status: todo +priority: medium +``` + +Add lightweight persistence for context-package references and external memory +references without implementing memory lifecycle behavior. + +Output: context reference model and tests. + +Acceptance: + +- Context packages can refer to artifacts, relationships, runs, and external + memory records. +- External memory references are opaque and provenance-tagged. +- No retention, decay, compaction, activation planning, or preference-memory + behavior is added to this repo. + +## P4.8 - Add import/export and smoke verification + +```task +id: KONT-WP-0004-T008 +status: todo +priority: medium +``` + +Provide a deterministic import/export path for repository snapshots so early +users and agents can inspect or migrate local state. + +Output: programmatic snapshot helpers and tests. + +Acceptance: + +- A repository snapshot can be exported and imported into a fresh backend. +- Imported data preserves ids, digests, metadata, relationships, revisions, and + run links. +- Snapshot format does not become a replacement for the service API. + +## Definition Of Done + +- `python3 -m pytest` passes. +- Existing in-memory behavior remains compatible unless explicitly revised. +- SQLite-backed tests prove durable behavior across repository + re-instantiation. +- Persistence docs explain what is durable now and what remains deferred. +- `docs/phase-memory-boundary.md` remains the boundary for memory-specific + behavior.