diff --git a/docs/agent-working-memory.md b/docs/agent-working-memory.md index e43c3b7..15e51bf 100644 --- a/docs/agent-working-memory.md +++ b/docs/agent-working-memory.md @@ -164,6 +164,15 @@ Natural extensions are intentionally deferred: - vector retrieval and embedding caches - decay/retention policies - reviewed long term identity packages +- reasoning memory as decision graphs +- short-term memory as conversational event paths +- long-term memory as knowledge graphs +- memory service blueprints with size, latency, retention, refresh, compaction, + and policy parameters - signed policy decisions and durable decision logs - remote or enterprise memory registries - package merge/diff and conflict diagnostics + +The graph/profile direction is captured in +`docs/agentic-memory-graph-blueprint-assessment.md` and +`MKTT-WP-0016: Agentic Memory Graphs And Service Blueprints`. diff --git a/docs/agentic-memory-graph-blueprint-assessment.md b/docs/agentic-memory-graph-blueprint-assessment.md new file mode 100644 index 0000000..7809c19 --- /dev/null +++ b/docs/agentic-memory-graph-blueprint-assessment.md @@ -0,0 +1,274 @@ +# Agentic Memory Graph Blueprint Assessment + +Date: 2026-05-04 + +## Question + +Can the current Markitect memory framework naturally extend into a system with: + +- reasoning memory based on decision graphs +- short-term memory based on conversational paths +- long-term memory based on knowledge graphs + +And can it do so efficiently by generating profiles or blueprints that describe +the services, sizes, response times, retention rules, and policy constraints +needed for a given agent memory deployment? + +## Short Answer + +Yes, but not directly as-is. + +The current `markitect_tool.memory` layer is a strong first layer for explicit +activation packages. It gives us source spans, namespaces, summaries, budgets, +provenance, freshness, local policy metadata, activation lifecycle, and +refreshable retrieval recipes. + +That is enough to be the **payload and activation layer** for agent memory. + +It is not yet enough to be the **memory operating model**. A three-part memory +system needs graph schemas, event logs, service blueprints, operational +profiles, lifecycle policies, and graph-aware retrieval/compaction. + +## Three Memory Systems + +### A. Reasoning Memory: Decision Graphs + +Reasoning memory should store how the agent got somewhere: + +- claims +- assumptions +- decisions +- rejected alternatives +- tradeoffs +- evidence links +- unresolved questions +- confidence +- reversibility +- affected artifacts +- follow-up obligations + +The natural shape is a directed graph: + +```text +question -> hypothesis -> evidence -> decision -> action -> outcome + \-> rejected alternative -> reason +``` + +This differs from a context package. A package captures selected content for +activation. A decision graph captures the evolution of reasoning and can later +answer: + +- why did we choose this? +- what alternatives were considered? +- which assumptions are stale? +- what decision depends on this source? +- which outcomes invalidated prior reasoning? + +Current fit: + +- `ContextPackageItem.provenance` can point to reasoning nodes later. +- `RetrievalRecipe` can reactivate relevant decision subgraphs. +- `MemoryNamespace` can scope decisions to project/thread/task. + +Missing: + +- node and edge schema for reasoning graphs +- evidence/claim/decision/outcome types +- graph integrity rules +- graph query and explanation API +- compaction from detailed reasoning paths into durable decision summaries + +### B. Short-Term Memory: Conversational Paths + +Short-term memory should store the navigable shape of an interaction: + +- turns +- tool calls +- observations +- plan changes +- branch points +- interruptions +- user corrections +- local state transitions +- activation/deactivation events + +The natural shape is an event path with branches: + +```text +turn -> plan -> tool_call -> observation -> edit -> validation + \-> interruption -> revised plan +``` + +This is not merely chat history. A useful conversational memory stores which +path led to the current state, which branches were abandoned, and which facts +are still active. + +Current fit: + +- `ContextActivation` can be logged as a path event. +- `ContextPackage` can represent a compact episode summary. +- `MemoryNamespace.thread` and `MemoryNamespace.task` already provide useful + coordinates. + +Missing: + +- event envelope for turns, tools, observations, decisions, and activations +- path compaction into episode packages +- active/inactive fact tracking +- branch/merge semantics +- retention windows and discard rules +- low-latency current-thread profile + +### C. Long-Term Memory: Knowledge Graphs + +Long-term memory should store stable project or identity knowledge: + +- entities +- concepts +- artifacts +- people/roles +- contracts +- APIs +- decisions +- policies +- relationships +- durable preferences +- source-backed facts + +The natural shape is a knowledge graph with provenance and policy on every +node and edge: + +```text +artifact -> implements -> capability +decision -> affects -> module +user_preference -> applies_to -> interaction_style +policy_label -> governs -> source_span +``` + +Current fit: + +- local index and source spans give us source-backed evidence. +- content classes and references already hint at entity/relationship modeling. +- packages can activate graph neighborhoods. + +Missing: + +- canonical knowledge node/edge vocabulary +- extraction pipeline from Markdown/reference/content-class/query outputs +- stable entity identity and merge rules +- confidence and freshness tracking +- graph-backed retrieval recipes +- profile-driven storage adapters + +## Efficient Instantiation Through Profiles + +A useful memory framework should let a project generate a blueprint such as: + +```yaml +id: markitect-agent-memory +intent: Support project implementation agents with local-first memory. +profiles: + reasoning: + store: local-sqlite-graph + max_nodes: 50000 + p95_read_ms: 50 + retention: project + compaction: + mode: decision-summary + max_tokens_per_decision: 300 + conversation: + store: local-event-log + hot_window_turns: 40 + p95_read_ms: 20 + retention: 14d + branch_policy: preserve_decision_branches + knowledge: + store: local-property-graph + max_nodes: 250000 + p95_neighborhood_ms: 150 + embedding_adapter: optional + refresh: on_snapshot_change +policy: + activation_requires_recheck: true + default_action: read + external_pdp: optional +outputs: + context_packages: + max_tokens: 6000 + max_items: 40 +``` + +Such a profile should not necessarily start services by itself. The first +Markitect version should validate, normalize, and explain these profiles, then +emit implementation manifests for local or external runtimes. + +## Operational Parameters To Model + +Every profile should be able to state: + +- target memory kind: reasoning, conversation, knowledge, package, identity +- backing store kind and adapter id +- maximum nodes, edges, events, packages, and bytes +- target p50/p95 read and write latency +- token budget for activation +- freshness and refresh triggers +- retention and deletion rules +- compaction policy +- merge/conflict policy +- policy reauthorization requirements +- observability events +- failure modes and fallback behavior +- export/import requirements + +This gives memory architecture a concrete shape instead of a vague promise of +"more context." + +## Architecture Extension + +The current framework can extend naturally if we add four layers: + +1. **Memory Graph Model** + Shared node, edge, graph snapshot, and graph query envelopes. + +2. **Memory Event Log** + Append-only event model for conversation paths, activations, reasoning + transitions, and graph updates. + +3. **Memory Service Blueprint** + Declarative profile that captures service topology, operational limits, + latency targets, storage budgets, retention, compaction, and policy. + +4. **Graph-To-Package Compiler** + Deterministic compiler that turns graph neighborhoods, decision paths, and + conversation episodes into `ContextPackage` objects. + +## Recommendation + +Create a follow-on workplan for **Agentic Memory Graphs And Service +Blueprints**. + +Do not replace the current context package layer. It remains the right +activation artifact. The next layer should make packages graph-aware and +profile-driven. + +The first implementation should stay local and deterministic: + +- YAML blueprint validation +- graph/event dataclasses +- local SQLite graph/event prototype or interface-first fake store +- package compilation from graph neighborhoods +- CLI inspection and planning commands +- no mandatory LLM, embedding, vector database, or external policy service + +## What To Defer + +- autonomous long-term personality mutation +- unsupervised durable preference writes +- live vector search +- graph embeddings +- remote graph service orchestration +- distributed memory synchronization +- enterprise authorization administration + +Those are valuable later, but they should plug into a clean profile/adapter +model rather than becoming assumptions in core. diff --git a/docs/workplan-planning-map.md b/docs/workplan-planning-map.md index f3b4b2c..509299c 100644 --- a/docs/workplan-planning-map.md +++ b/docs/workplan-planning-map.md @@ -42,6 +42,7 @@ and descriptions mirror the operational view. | `MKTT-WP-0012` | complete | done | `MKTT-WP-0004`, `MKTT-WP-0010`, `MKTT-WP-0011` | Document function layer is complete: deterministic Markdown-native function descriptors, registry, inline/fenced syntax, pipelines, context bindings, CLI, docs, examples, diagnostics, provenance, and extension descriptor. | | `MKTT-WP-0008` | complete | done | `MKTT-WP-0006`, `MKTT-WP-0007`, `MKTT-WP-0009` | Agent working-memory context cache is complete: context package schema, local registry, package creation from queries/search/manifests, deterministic summaries, namespaces, activation/deactivation/refresh/explain lifecycle, policy re-checks, CLI, docs, and examples. | | `MKTT-WP-0015` | P2 | todo | `MKTT-WP-0010`, `MKTT-WP-0011`, `MKTT-WP-0012` | Future render and document-function extensions: typed values, richer syntax, document-local reusable functions, Quarkdown/export adapters, render-aware references, assets, and permission sandboxing. Defer unless publishing/export pressure becomes current. | +| `MKTT-WP-0016` | P2 | todo | `MKTT-WP-0008`, `MKTT-WP-0007`, `MKTT-WP-0009`, `MKTT-WP-0013` | Follow-on agentic memory architecture: reasoning decision graphs, conversational paths, long-term knowledge graphs, memory service blueprints/profiles, graph-to-context-package compilation, and adapter boundaries. | ## Dependency Notes @@ -109,6 +110,13 @@ hidden ambient agent memory. LLM-assisted summaries, embeddings, vector stores, remote registries, retention/decay, and reviewed long-term identity packages remain future optional extensions. +`MKTT-WP-0016` captures the next memory architecture layer. It should extend +the context package model into reasoning graphs, conversational event paths, +knowledge graphs, and deployment blueprints with operational parameters such +as memory size, latency, retention, refresh, compaction, activation budgets, +and policy reauthorization. It should compile graph neighborhoods into +`ContextPackage` objects rather than replacing the WP-0008 package layer. + ## State Hub Mirror Native State Hub dependency edges should mirror the whole-workstream @@ -142,3 +150,7 @@ dependencies: - `MKTT-WP-0015 -> MKTT-WP-0010` - `MKTT-WP-0015 -> MKTT-WP-0011` - `MKTT-WP-0015 -> MKTT-WP-0012` +- `MKTT-WP-0016 -> MKTT-WP-0008` +- `MKTT-WP-0016 -> MKTT-WP-0007` +- `MKTT-WP-0016 -> MKTT-WP-0009` +- `MKTT-WP-0016 -> MKTT-WP-0013` diff --git a/workplans/MKTT-WP-0016-agentic-memory-graphs-and-service-blueprints.md b/workplans/MKTT-WP-0016-agentic-memory-graphs-and-service-blueprints.md new file mode 100644 index 0000000..11065b2 --- /dev/null +++ b/workplans/MKTT-WP-0016-agentic-memory-graphs-and-service-blueprints.md @@ -0,0 +1,270 @@ +--- +id: MKTT-WP-0016 +type: workplan +title: "Agentic Memory Graphs And Service Blueprints" +domain: markitect +status: todo +owner: markitect-tool +topic_slug: markitect +planning_priority: P2 +planning_order: 140 +depends_on_workplans: + - MKTT-WP-0008 + - MKTT-WP-0007 + - MKTT-WP-0009 + - MKTT-WP-0013 +related_workplans: + - MKTT-WP-0011 + - MKTT-WP-0014 +created: "2026-05-04" +updated: "2026-05-04" +state_hub_workstream_id: "850cbc13-71cb-422c-86ec-1cc7679095cf" +--- + +# MKTT-WP-0016: Agentic Memory Graphs And Service Blueprints + +## Preamble: Repository Boundary + +This workplan is intentionally scoped as a boundary-setting bridge between +`markitect-tool` and a future sister project named `phased-memory`. + +The emerging memory architecture is bigger than Markitect itself. Its +sophisticated form should live in `phased-memory`, a dedicated repo for memory +concepts that range from ephemeral and fluid working context to durable, +stabilized long-term memory. That repo can own memory service runtimes, +specialized stores, compaction strategies, graph evolution, profile +instantiation, retention mechanics, and operational deployment concerns. + +`markitect-tool` should keep the smaller and more natural responsibility: +define markdown-friendly interface contracts, memory blueprint/profile shapes, +adapter descriptors, provenance expectations, and deterministic compilers from +memory selections into `ContextPackage` activations. Markitect may provide local +fixtures and validation tools, but should avoid becoming the full memory +runtime. + +The practical goal is separation of concerns. Markitect should be able to +describe, validate, and consume memory systems. `phased-memory` should be able +to implement and operate them. + +## Purpose + +Extend Markitect's explicit context-package memory layer into a graph-aware, +profile-driven memory architecture for agentic systems. + +The target model separates: + +- reasoning memory based on decision graphs +- short-term memory based on conversational paths +- long-term memory based on knowledge graphs + +The current `MKTT-WP-0008` implementation remains the activation and package +layer. This workplan adds the graph, event, and service-blueprint layers needed +to instantiate memory systems with explicit operational parameters. + +## Background + +The assessment in `docs/agentic-memory-graph-blueprint-assessment.md` concludes +that the current memory package framework is a natural substrate but not yet a +complete graph memory system. + +Current strengths: + +- source spans +- namespaces +- deterministic summaries +- activation/deactivation lifecycle +- local package registry +- retrieval recipes +- token budgets +- provenance and policy metadata + +Missing layers: + +- decision graph schema +- conversational event/path schema +- knowledge graph schema +- memory service profile and blueprint format +- operational parameters such as size, latency, retention, freshness, and + compaction +- graph-to-context-package compiler +- graph-aware CLI inspection/planning + +## Decision + +Implement this as an optional extension over the existing memory framework. + +Do not make LLMs, embeddings, vector stores, remote graph databases, flex-auth, +or external policy services required. The first version should be deterministic +and local-first, with adapter boundaries for more sophisticated deployments. + +## P16.1 - Define graph memory vocabulary + +```task +id: MKTT-WP-0016-T001 +status: todo +priority: high +state_hub_task_id: "e1523ffb-47e6-4017-88e5-907586297416" +``` + +Define common node and edge envelopes for: + +- reasoning graph nodes: question, claim, assumption, evidence, decision, + alternative, outcome, risk, follow-up +- conversational path nodes: turn, plan, tool call, observation, edit, + validation, interruption, activation +- knowledge graph nodes: entity, artifact, concept, capability, contract, + policy, preference, source fact +- edge types: supports, contradicts, depends_on, led_to, supersedes, + derived_from, affects, mentions, activates, governs + +Every node and edge should support provenance, freshness, confidence, policy +metadata, namespace, and source spans. + +Output: model docs, dataclasses or schema objects, and serialization tests. + +## P16.2 - Add memory event log + +```task +id: MKTT-WP-0016-T002 +status: todo +priority: high +state_hub_task_id: "a5b871ea-1f5b-4065-bae2-04765da3bc96" +``` + +Create an append-only event envelope for short-term conversational paths and +graph updates. + +Events should capture: + +- event id and timestamp +- actor/agent/thread/task namespace +- event kind +- references to context packages and activations +- graph node/edge updates +- branch and merge metadata +- policy metadata + +Output: local event log interface, deterministic tests, and examples. + +## P16.3 - Define memory service blueprint/profile standard + +```task +id: MKTT-WP-0016-T003 +status: todo +priority: high +state_hub_task_id: "07c94925-c593-4acb-8f2b-b6c61744deff" +``` + +Define a YAML/JSON profile that can describe a memory deployment: + +- enabled memory kinds +- stores/adapters per kind +- node/edge/event/package limits +- target p50/p95 latency +- retention and deletion rules +- refresh triggers +- compaction strategy +- activation token budgets +- policy reauthorization requirements +- observability events +- failure and fallback behavior + +Output: profile schema, validation, example profiles, and docs. + +## P16.4 - Implement local graph/event prototype + +```task +id: MKTT-WP-0016-T004 +status: todo +priority: medium +state_hub_task_id: "afcd7ce9-e657-4779-b3b2-0ae3f8e2d66e" +``` + +Implement a small local prototype, likely SQLite-backed or interface-first with +durable YAML fixtures, that can store and query: + +- decision graph subgraphs +- conversation paths +- knowledge graph neighborhoods + +This should be adequate for tests and local use, not a full graph database. + +Output: local store or adapter interface, graph query helpers, and tests. + +## P16.5 - Compile graph neighborhoods into context packages + +```task +id: MKTT-WP-0016-T005 +status: todo +priority: high +state_hub_task_id: "5dfbe77e-f926-4ce8-a8c3-3c10aec0f2f6" +``` + +Add deterministic compilers from graph/path selections into +`ContextPackage` objects: + +- decision path package +- conversation episode package +- knowledge neighborhood package +- mixed package with explicit budget and priority rules + +Output: compiler API, provenance-preserving package output, and activation +tests. + +## P16.6 - Add blueprint planning CLI + +```task +id: MKTT-WP-0016-T006 +status: todo +priority: medium +state_hub_task_id: "07fb3f0d-aaf7-4bc0-a7a3-0fd4983776bb" +``` + +Add CLI commands to inspect and plan memory systems: + +```text +mkt memory blueprint validate +mkt memory blueprint plan +mkt memory graph explain +mkt memory graph pack +``` + +These commands should validate profiles and produce implementation plans with +storage, latency, budget, and policy implications. They should not launch live +external services in the first version. + +Output: CLI commands, docs, and tests. + +## P16.7 - Define adapter boundaries for advanced services + +```task +id: MKTT-WP-0016-T007 +status: todo +priority: medium +state_hub_task_id: "ff78d449-0738-4f1b-a018-2efed3d8e878" +``` + +Define optional adapter boundaries for: + +- external graph databases +- vector stores and embedding providers +- LLM-assisted graph extraction and summaries +- enterprise policy decision points +- remote memory registries +- durable audit/event sinks + +These are adapter protocols only. Core graph/profile validation and local +package compilation must remain useful without them. + +Output: adapter interface notes and extension descriptors. + +## Exit Criteria + +- Memory graph vocabulary is explicit and serialized cleanly. +- Reasoning, conversation, and knowledge memory can be represented separately. +- A memory blueprint/profile can express operational size, latency, retention, + refresh, compaction, activation, and policy parameters. +- Local deterministic graph/event examples can compile into context packages. +- CLI can validate and explain memory blueprints. +- External LLM, embedding, vector, graph, policy, and registry services remain + optional.