Layered context memory revision 0

workplans/MKTT-WP-0008-agent-working-memory-context-cache.md

@@ -3,17 +3,17 @@ id: MKTT-WP-0008
 type: workplan
 title: "Agent Working Memory Context Cache"
 domain: markitect
-status: todo
+status: done
 owner: markitect-tool
 topic_slug: markitect
-planning_priority: P3
+planning_priority: complete
 planning_order: 90
 depends_on_workplans:
   - MKTT-WP-0006
   - MKTT-WP-0007
   - MKTT-WP-0009
 created: "2026-05-03"
-updated: "2026-05-03"
+updated: "2026-05-04"
 state_hub_workstream_id: "6269f338-4f5c-40ee-90e5-0371f5c3874c"
 ---
 
@@ -25,6 +25,54 @@ Create activatable context packages that let agents drop, reactivate, and
 reuse project knowledge efficiently while preserving provenance and policy
 metadata.
 
+## Thought Experiment And Design Refinement
+
+The implementation was preceded by a wishful architecture pass documented in
+`docs/agent-working-memory-thought-experiment.md`.
+
+The useful model is layered memory:
+
+- reflex context for the current response
+- short working set for the active task
+- episodic memory for a thread or work session
+- project semantic memory for architecture and source knowledge
+- identity-like continuity memory for durable principles and preferences
+- policy/safety memory that must be rechecked before activation
+
+The key refinement is that Markitect should implement the local, explicit,
+inspectable package layer rather than hidden ambient agent memory. Long-term
+identity and learning memory can be represented as reviewed namespace-scoped
+packages later, but writes should remain visible and reversible.
+
+This shaped the implementation:
+
+- package creation and activation are explicit CLI/library operations
+- every item carries source spans, summaries, token estimates, provenance, and
+  policy metadata
+- deterministic summaries come first
+- package creation and activation can both run local policy checks
+- local files and the local SQLite index are enough for the first version
+- LLM summaries, embeddings, remote registries, retention/decay, and durable
+  identity packages remain future optional extensions
+
+## Implementation Summary
+
+Implemented the first local agent working-memory context cache:
+
+- `markitect_tool.memory` extension package with context package schema,
+  retrieval recipes, budgets, namespaces, summary layers, activation envelopes,
+  and filesystem-backed local registry.
+- Package creation from direct Markdown source queries, local index selector or
+  JSONPath queries, FTS search results, and YAML manifests.
+- Deterministic summaries and approximate token estimates.
+- Activation, deactivation, refresh, explain, list, and save/load lifecycle.
+- Optional local policy filtering at package creation and activation.
+- CLI commands under `mkt context`.
+- Built-in extension descriptor `memory.context-package`.
+- Documentation and example manifest in `docs/agent-working-memory.md`,
+  `docs/agent-working-memory-thought-experiment.md`, and
+  `examples/memory/workplan-context.manifest.yaml`.
+
 ## Architectural Boundary
 
 This workplan depends only on Markitect-local backend and policy contracts:
@@ -48,7 +96,7 @@ freshness metadata. These references are metadata, not hard dependencies.
 
 ```task
 id: MKTT-WP-0008-T001
-status: todo
+status: done
 priority: high
 state_hub_task_id: "21ee9c37-4add-4886-bd03-a7bb4b20e957"
 ```
@@ -68,11 +116,16 @@ The schema should include optional policy metadata:
 These fields must support local policy gateways first and external policy
 services only through optional adapters.
 
+Implemented: `ContextPackage`, `ContextPackageItem`, `SourceSpan`,
+`RetrievalRecipe`, `SummaryLayer`, `ContextBudget`, and `MemoryNamespace`
+define the local schema. Package items preserve source spans, summaries, token
+estimates, provenance, freshness, and policy metadata.
+
 ## P8.2 - Implement package creation from queries
 
 ```task
 id: MKTT-WP-0008-T002
-status: todo
+status: done
 priority: high
 state_hub_task_id: "4df06b93-13ce-41fb-a8c3-f04d4ad9d752"
 ```
@@ -84,11 +137,16 @@ Package creation should use current query/search APIs and policy-aware result
 filtering. It should not call flex-auth directly; future flex-auth-backed
 filtering can be injected through the existing policy gateway boundary.
 
+Implemented: packages can be created from direct Markdown source queries, local
+indexed selector/JSONPath queries, FTS search results, and YAML manifests.
+Optional local policy gateways filter package items using the existing
+`LocalLabelPolicyGateway` boundary.
+
 ## P8.3 - Implement activation lifecycle
 
 ```task
 id: MKTT-WP-0008-T003
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "9f3d9792-d655-482d-bae0-262df5fc0136"
 ```
@@ -99,11 +157,17 @@ Activation should re-check local policy metadata when a policy gateway is
 available. In the absence of an external service, activation remains fully
 local and explainable.
 
+Implemented: `activate_context_package`, `deactivate_context_package`,
+`refresh_context_package`, `explain_context_package`, and
+`LocalContextPackageRegistry` support the lifecycle locally. Activation
+rebuilds summaries after policy filtering so denied content does not leak
+through package-level summaries.
+
 ## P8.4 - Add memory namespaces
 
 ```task
 id: MKTT-WP-0008-T004
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "2d090494-0e10-44cd-8e2d-c418d7530b27"
 ```
@@ -115,11 +179,14 @@ Namespace design should leave room for enterprise subject ids and external
 resource ids, but must not require any particular SSO, IAM, or authorization
 provider.
 
+Implemented: `MemoryNamespace` supports project, user, agent, thread, task,
+and custom namespace fields without depending on any external agent platform.
+
 ## P8.5 - Add summary layers
 
 ```task
 id: MKTT-WP-0008-T005
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "4d1cf970-3d6d-4bd5-8da9-ec2399aa7efe"
 ```
@@ -130,11 +197,15 @@ through an injected adapter.
 Assisted summaries must be optional and policy/capability-gated before any
 prompt assembly happens.
 
+Implemented: deterministic extractive/count summaries are included. Assisted
+summaries remain a documented future adapter path and are not invoked by core
+package creation or activation.
+
 ## P8.6 - Add CLI commands
 
 ```task
 id: MKTT-WP-0008-T006
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "2f18386c-9d2c-4af1-b8e2-75cb487c1692"
 ```
@@ -152,6 +223,21 @@ CLI commands should work against local packages without flex-auth. Optional
 policy flags may accept local policy files or later external adapter
 configuration.
 
+Implemented:
+
+```text
+mkt context pack
+mkt context activate
+mkt context deactivate
+mkt context explain
+mkt context refresh
+mkt context list
+```
+
+Commands work with local package files and the `.markitect/context` registry.
+Policy flags use local label policy only; external policy systems remain
+optional adapters.
+
 ## Exit Criteria
 
 - Agents can reactivate project context by stable id.