Extension workplan for phased-memory

2026-05-04 21:07:35 +02:00
parent ac4bf50643
commit 56f920a4eb
4 changed files with 565 additions and 0 deletions
--- a/docs/agent-working-memory.md
+++ b/docs/agent-working-memory.md
@@ -164,6 +164,15 @@ Natural extensions are intentionally deferred:
 - vector retrieval and embedding caches
 - decay/retention policies
 - reviewed long term identity packages
+- reasoning memory as decision graphs
+- short-term memory as conversational event paths
+- long-term memory as knowledge graphs
+- memory service blueprints with size, latency, retention, refresh, compaction,
+  and policy parameters
 - signed policy decisions and durable decision logs
 - remote or enterprise memory registries
 - package merge/diff and conflict diagnostics
+
+The graph/profile direction is captured in
+`docs/agentic-memory-graph-blueprint-assessment.md` and
+`MKTT-WP-0016: Agentic Memory Graphs And Service Blueprints`.
--- a/docs/agentic-memory-graph-blueprint-assessment.md
+++ b/docs/agentic-memory-graph-blueprint-assessment.md
@@ -0,0 +1,274 @@
+# Agentic Memory Graph Blueprint Assessment
+
+Date: 2026-05-04
+
+## Question
+
+Can the current Markitect memory framework naturally extend into a system with:
+
+- reasoning memory based on decision graphs
+- short-term memory based on conversational paths
+- long-term memory based on knowledge graphs
+
+And can it do so efficiently by generating profiles or blueprints that describe
+the services, sizes, response times, retention rules, and policy constraints
+needed for a given agent memory deployment?
+
+## Short Answer
+
+Yes, but not directly as-is.
+
+The current `markitect_tool.memory` layer is a strong first layer for explicit
+activation packages. It gives us source spans, namespaces, summaries, budgets,
+provenance, freshness, local policy metadata, activation lifecycle, and
+refreshable retrieval recipes.
+
+That is enough to be the **payload and activation layer** for agent memory.
+
+It is not yet enough to be the **memory operating model**. A three-part memory
+system needs graph schemas, event logs, service blueprints, operational
+profiles, lifecycle policies, and graph-aware retrieval/compaction.
+
+## Three Memory Systems
+
+### A. Reasoning Memory: Decision Graphs
+
+Reasoning memory should store how the agent got somewhere:
+
+- claims
+- assumptions
+- decisions
+- rejected alternatives
+- tradeoffs
+- evidence links
+- unresolved questions
+- confidence
+- reversibility
+- affected artifacts
+- follow-up obligations
+
+The natural shape is a directed graph:
+
+```text
+question -> hypothesis -> evidence -> decision -> action -> outcome
+                       \-> rejected alternative -> reason
+```
+
+This differs from a context package. A package captures selected content for
+activation. A decision graph captures the evolution of reasoning and can later
+answer:
+
+- why did we choose this?
+- what alternatives were considered?
+- which assumptions are stale?
+- what decision depends on this source?
+- which outcomes invalidated prior reasoning?
+
+Current fit:
+
+- `ContextPackageItem.provenance` can point to reasoning nodes later.
+- `RetrievalRecipe` can reactivate relevant decision subgraphs.
+- `MemoryNamespace` can scope decisions to project/thread/task.
+
+Missing:
+
+- node and edge schema for reasoning graphs
+- evidence/claim/decision/outcome types
+- graph integrity rules
+- graph query and explanation API
+- compaction from detailed reasoning paths into durable decision summaries
+
+### B. Short-Term Memory: Conversational Paths
+
+Short-term memory should store the navigable shape of an interaction:
+
+- turns
+- tool calls
+- observations
+- plan changes
+- branch points
+- interruptions
+- user corrections
+- local state transitions
+- activation/deactivation events
+
+The natural shape is an event path with branches:
+
+```text
+turn -> plan -> tool_call -> observation -> edit -> validation
+                     \-> interruption -> revised plan
+```
+
+This is not merely chat history. A useful conversational memory stores which
+path led to the current state, which branches were abandoned, and which facts
+are still active.
+
+Current fit:
+
+- `ContextActivation` can be logged as a path event.
+- `ContextPackage` can represent a compact episode summary.
+- `MemoryNamespace.thread` and `MemoryNamespace.task` already provide useful
+  coordinates.
+
+Missing:
+
+- event envelope for turns, tools, observations, decisions, and activations
+- path compaction into episode packages
+- active/inactive fact tracking
+- branch/merge semantics
+- retention windows and discard rules
+- low-latency current-thread profile
+
+### C. Long-Term Memory: Knowledge Graphs
+
+Long-term memory should store stable project or identity knowledge:
+
+- entities
+- concepts
+- artifacts
+- people/roles
+- contracts
+- APIs
+- decisions
+- policies
+- relationships
+- durable preferences
+- source-backed facts
+
+The natural shape is a knowledge graph with provenance and policy on every
+node and edge:
+
+```text
+artifact -> implements -> capability
+decision -> affects -> module
+user_preference -> applies_to -> interaction_style
+policy_label -> governs -> source_span
+```
+
+Current fit:
+
+- local index and source spans give us source-backed evidence.
+- content classes and references already hint at entity/relationship modeling.
+- packages can activate graph neighborhoods.
+
+Missing:
+
+- canonical knowledge node/edge vocabulary
+- extraction pipeline from Markdown/reference/content-class/query outputs
+- stable entity identity and merge rules
+- confidence and freshness tracking
+- graph-backed retrieval recipes
+- profile-driven storage adapters
+
+## Efficient Instantiation Through Profiles
+
+A useful memory framework should let a project generate a blueprint such as:
+
+```yaml
+id: markitect-agent-memory
+intent: Support project implementation agents with local-first memory.
+profiles:
+  reasoning:
+    store: local-sqlite-graph
+    max_nodes: 50000
+    p95_read_ms: 50
+    retention: project
+    compaction:
+      mode: decision-summary
+      max_tokens_per_decision: 300
+  conversation:
+    store: local-event-log
+    hot_window_turns: 40
+    p95_read_ms: 20
+    retention: 14d
+    branch_policy: preserve_decision_branches
+  knowledge:
+    store: local-property-graph
+    max_nodes: 250000
+    p95_neighborhood_ms: 150
+    embedding_adapter: optional
+    refresh: on_snapshot_change
+policy:
+  activation_requires_recheck: true
+  default_action: read
+  external_pdp: optional
+outputs:
+  context_packages:
+    max_tokens: 6000
+    max_items: 40
+```
+
+Such a profile should not necessarily start services by itself. The first
+Markitect version should validate, normalize, and explain these profiles, then
+emit implementation manifests for local or external runtimes.
+
+## Operational Parameters To Model
+
+Every profile should be able to state:
+
+- target memory kind: reasoning, conversation, knowledge, package, identity
+- backing store kind and adapter id
+- maximum nodes, edges, events, packages, and bytes
+- target p50/p95 read and write latency
+- token budget for activation
+- freshness and refresh triggers
+- retention and deletion rules
+- compaction policy
+- merge/conflict policy
+- policy reauthorization requirements
+- observability events
+- failure modes and fallback behavior
+- export/import requirements
+
+This gives memory architecture a concrete shape instead of a vague promise of
+"more context."
+
+## Architecture Extension
+
+The current framework can extend naturally if we add four layers:
+
+1. **Memory Graph Model**
+   Shared node, edge, graph snapshot, and graph query envelopes.
+
+2. **Memory Event Log**
+   Append-only event model for conversation paths, activations, reasoning
+   transitions, and graph updates.
+
+3. **Memory Service Blueprint**
+   Declarative profile that captures service topology, operational limits,
+   latency targets, storage budgets, retention, compaction, and policy.
+
+4. **Graph-To-Package Compiler**
+   Deterministic compiler that turns graph neighborhoods, decision paths, and
+   conversation episodes into `ContextPackage` objects.
+
+## Recommendation
+
+Create a follow-on workplan for **Agentic Memory Graphs And Service
+Blueprints**.
+
+Do not replace the current context package layer. It remains the right
+activation artifact. The next layer should make packages graph-aware and
+profile-driven.
+
+The first implementation should stay local and deterministic:
+
+- YAML blueprint validation
+- graph/event dataclasses
+- local SQLite graph/event prototype or interface-first fake store
+- package compilation from graph neighborhoods
+- CLI inspection and planning commands
+- no mandatory LLM, embedding, vector database, or external policy service
+
+## What To Defer
+
+- autonomous long-term personality mutation
+- unsupervised durable preference writes
+- live vector search
+- graph embeddings
+- remote graph service orchestration
+- distributed memory synchronization
+- enterprise authorization administration
+
+Those are valuable later, but they should plug into a clean profile/adapter
+model rather than becoming assumptions in core.
--- a/docs/workplan-planning-map.md
+++ b/docs/workplan-planning-map.md
@@ -42,6 +42,7 @@ and descriptions mirror the operational view.
 | `MKTT-WP-0012` | complete | done | `MKTT-WP-0004`, `MKTT-WP-0010`, `MKTT-WP-0011` | Document function layer is complete: deterministic Markdown-native function descriptors, registry, inline/fenced syntax, pipelines, context bindings, CLI, docs, examples, diagnostics, provenance, and extension descriptor. |
 | `MKTT-WP-0008` | complete | done | `MKTT-WP-0006`, `MKTT-WP-0007`, `MKTT-WP-0009` | Agent working-memory context cache is complete: context package schema, local registry, package creation from queries/search/manifests, deterministic summaries, namespaces, activation/deactivation/refresh/explain lifecycle, policy re-checks, CLI, docs, and examples. |
 | `MKTT-WP-0015` | P2 | todo | `MKTT-WP-0010`, `MKTT-WP-0011`, `MKTT-WP-0012` | Future render and document-function extensions: typed values, richer syntax, document-local reusable functions, Quarkdown/export adapters, render-aware references, assets, and permission sandboxing. Defer unless publishing/export pressure becomes current. |
+| `MKTT-WP-0016` | P2 | todo | `MKTT-WP-0008`, `MKTT-WP-0007`, `MKTT-WP-0009`, `MKTT-WP-0013` | Follow-on agentic memory architecture: reasoning decision graphs, conversational paths, long-term knowledge graphs, memory service blueprints/profiles, graph-to-context-package compilation, and adapter boundaries. |

 ## Dependency Notes

@@ -109,6 +110,13 @@ hidden ambient agent memory. LLM-assisted summaries, embeddings, vector stores,
 remote registries, retention/decay, and reviewed long-term identity packages
 remain future optional extensions.

+`MKTT-WP-0016` captures the next memory architecture layer. It should extend
+the context package model into reasoning graphs, conversational event paths,
+knowledge graphs, and deployment blueprints with operational parameters such
+as memory size, latency, retention, refresh, compaction, activation budgets,
+and policy reauthorization. It should compile graph neighborhoods into
+`ContextPackage` objects rather than replacing the WP-0008 package layer.
+
 ## State Hub Mirror

 Native State Hub dependency edges should mirror the whole-workstream
@@ -142,3 +150,7 @@ dependencies:
 - `MKTT-WP-0015 -> MKTT-WP-0010`
 - `MKTT-WP-0015 -> MKTT-WP-0011`
 - `MKTT-WP-0015 -> MKTT-WP-0012`
+- `MKTT-WP-0016 -> MKTT-WP-0008`
+- `MKTT-WP-0016 -> MKTT-WP-0007`
+- `MKTT-WP-0016 -> MKTT-WP-0009`
+- `MKTT-WP-0016 -> MKTT-WP-0013`