diff --git a/docs/agent-working-memory-thought-experiment.md b/docs/agent-working-memory-thought-experiment.md new file mode 100644 index 0000000..190c2be --- /dev/null +++ b/docs/agent-working-memory-thought-experiment.md @@ -0,0 +1,104 @@ +# Agent Working Memory Thought Experiment + +Date: 2026-05-04 + +## Prompt + +Imagine an agent memory architecture optimized for short and mid term +efficiency across different reaction speeds, timescales, and cost constraints, +while preserving a personality-like long term identity and learning memory that +evolves over time. + +This is a wishful design note. It is not a promise that every layer belongs in +`markitect-tool` core. + +## Memory Layers + +| Layer | Reaction Speed | Lifetime | Cost Shape | Useful Contents | +| --- | --- | --- | --- | --- | +| Reflex context | milliseconds to seconds | one response | highest compute pressure | current user turn, active file, tool output, immediate errors | +| Working set | seconds to minutes | one task step | medium compute pressure | current plan, nearby code/docs, selected snippets, constraints | +| Episode memory | minutes to hours | one chat/thread/work session | moderate storage | decisions, explored dead ends, task state, user preferences for this effort | +| Project semantic memory | hours to weeks | project lifecycle | cheap local storage, occasional refresh | architecture docs, stable APIs, workplans, policies, source maps, examples | +| Identity and learning memory | weeks to years | agent/persona/project relationship | expensive to write, cheap to retrieve | values, style, durable preferences, repeated collaboration patterns | +| Policy and safety memory | all speeds | explicit freshness windows | must be rechecked before use | labels, trust zones, subject identity, decision ids, authorization context | + +The main design pressure is not just retrieval quality. It is deciding when a +memory should be cheap, fast, provisional, discardable, re-checkable, or +durable. + +## Wish List + +If a few things could be manifested instantly: + +1. A hot context router would keep the current working set small and sharp. It + would load exact spans and summaries only when they are needed, then release + them when the task moves on. +2. Every context item would carry provenance, policy metadata, freshness, + token cost, and a reason it was included. +3. Memory would be layered, not monolithic. Fast scratch memory, package memory, + project memory, and identity memory would have different write thresholds. +4. Durable memory writes would be explicit and inspectable. The agent could + propose "this seems worth remembering" rather than silently mutating itself. +5. Long term identity would be represented as small, versioned principles and + preferences, not as a bag of transcripts. +6. The system would support both activation and forgetting. Dropping context + should be as first-class as loading it. +7. Retrieval would be policy-aware at both package creation and activation. + A package created yesterday should not be blindly activated today. +8. Summaries would be multi-resolution: one-line, section-level, package-level, + project-level, and identity-level, each with its own freshness and source + links. +9. Local deterministic memory would work offline. LLM-assisted summaries, + embeddings, external stores, and enterprise authorization would be optional + adapters. +10. The agent could maintain a small "self continuity" layer: tone, + collaboration norms, current mission, and durable lessons, all with user + review and rollback. + +## Dynamic Balances + +The useful architecture has several moving balances: + +- Fast recall vs. exact provenance: the reflex layer can be approximate, but + activation packages must be explainable. +- Rich context vs. token budget: every item needs a token estimate and a + summary so a caller can choose the right resolution. +- Stability vs. freshness: project architecture memory is useful only if it + can be refreshed against current snapshots. +- Personalization vs. consent: long term identity memory should require higher + write friction than task-local memory. +- Local autonomy vs. enterprise control: local package creation must work + offline, while activation can re-check policy when a gateway is available. +- Retrieval breadth vs. actionability: packages should be small enough to + activate directly and large enough to explain why they matter. + +## Implications For Markitect + +Markitect should own the local, inspectable package layer: + +- schema for context packages +- package creation from selectors, search, and manifests +- deterministic summary layers +- activation/deactivation/refresh/explain lifecycle +- source spans, token estimates, provenance, and policy metadata +- optional adapter boundaries for assisted summaries, embeddings, vector + stores, and external policy decision points + +Markitect should not own hidden agent identity memory as an ambient service. +It can provide explicit package formats and namespaces for identity-like +memory, but durable writes should remain visible, reviewable, and policy-bound. + +## Best First Step + +The first implementation should be boring in the right way: + +- local files and local SQLite index only +- deterministic summaries only +- explicit package files in `.markitect/context` +- policy metadata preserved and optionally rechecked +- no live flex-auth, vector database, embedding provider, LLM provider, or + hidden agent state + +That gives future systems a trustworthy memory substrate without making the +core repo pretend to be a full cognitive architecture. diff --git a/docs/agent-working-memory.md b/docs/agent-working-memory.md new file mode 100644 index 0000000..e43c3b7 --- /dev/null +++ b/docs/agent-working-memory.md @@ -0,0 +1,169 @@ +# Agent Working Memory Context Cache + +Date: 2026-05-04 + +## Purpose + +Agent working memory packages are explicit, portable context bundles. They let +an agent or user gather relevant Markdown knowledge, drop it when it is no +longer needed, and reactivate it later by stable id. + +The implementation is local-first: + +- no live flex-auth requirement +- no network access +- no embedding or LLM provider requirement +- no hidden memory writes +- deterministic summaries first +- optional local policy checks at package creation and activation + +## Core Objects + +| Object | Role | +| --- | --- | +| `MemoryNamespace` | Project, user, agent, thread, task, and custom namespace coordinates. | +| `SourceSpan` | Path, snapshot id, unit kind/index, line span, selector, and engine. | +| `ContextPackageItem` | One included memory item with text, summary, token estimate, policy, provenance, and metadata. | +| `RetrievalRecipe` | Refreshable query/search recipe. | +| `SummaryLayer` | Deterministic package summary layer. | +| `ContextBudget` | Approximate max items/tokens and reserve. | +| `ContextPackage` | Portable package envelope. | +| `ContextActivation` | Activation/deactivation envelope with Markdown content and policy results. | +| `LocalContextPackageRegistry` | Filesystem registry rooted at `.markitect/context`. | + +## Use Cases + +### Task Reactivation + +Create a package from the current workplan or architecture sections, then +activate it in a later thread without rereading the whole repository. + +### Focused Code Review Context + +Pack only the sections, files, and examples relevant to a specific subsystem. +Use a small token budget so activation produces a compact review bundle. + +### Project Onboarding + +Create a package with intent, scope, architecture, and current workplan +summaries. A new agent can activate this instead of scanning every document. + +### Policy-Aware Knowledge Work + +Create and activate packages with local label policy. Package creation can +filter sensitive sources, and activation can re-check whether the current +subject may still read the items. + +### Identity-Like Continuity + +Represent durable collaboration norms or project principles as explicit +namespace-scoped context packages. These are not hidden personality mutations; +they are versioned and inspectable memory artifacts. + +### Mid-Term Episodic Memory + +Store decisions, explored alternatives, and task state for a thread or work +session. Keep it separate from long term project memory so it can expire or be +reviewed. + +## CLI + +Create from direct source files: + +```text +mkt context pack "sections[heading=Decision]" --source docs/example.md +``` + +Create from the local index: + +```text +mkt cache index docs --root . +mkt context pack "sections[heading~=Policy]" --path docs/access-control-policy-gateway.md +``` + +Create from FTS search: + +```text +mkt context pack "policy gateway" --search --limit 5 +``` + +Activate: + +```text +mkt context activate memory:package:... --target thread:wp-0008 +``` + +Explain: + +```text +mkt context explain memory:package:... +``` + +Refresh: + +```text +mkt context refresh memory:package:... +``` + +Deactivate: + +```text +mkt context deactivate memory:activation:... +``` + +## Manifest Shape + +```yaml +title: Workplan context +intent: Reactivate current workplan decisions. +namespace: + project: markitect-tool + task: MKTT-WP-0008 +budget: + max_items: 4 + max_tokens: 1200 +retrieval_recipes: + - kind: selector + query: sections[heading=Purpose] + engine: selector + sources: + - workplans/MKTT-WP-0008-agent-working-memory-context-cache.md +``` + +## Policy + +Policy metadata is copied from document frontmatter or the local index. When a +local policy gateway is supplied, Markitect filters package items by the +requested subject/action. + +Package creation typically uses action `package`; activation defaults to +action `read`. + +External policy services may be attached later through the existing policy +gateway boundary, but are not required. + +## Architecture Fit + +The memory layer is an internal extension over existing primitives: + +- local snapshots and FTS from `LocalSnapshotStore` +- selectors and optional JSONPath from the query engines +- local labels and trust zones from `LocalLabelPolicyGateway` +- provenance, diagnostics, and capability metadata from the extension + framework + +It is deliberately separate from the dataflow workflow engine. Workflows +orchestrate business processes; context packages capture reusable working +memory for agents. + +## Future Extensions + +Natural extensions are intentionally deferred: + +- LLM-assisted summaries through an injected adapter +- vector retrieval and embedding caches +- decay/retention policies +- reviewed long term identity packages +- signed policy decisions and durable decision logs +- remote or enterprise memory registries +- package merge/diff and conflict diagnostics diff --git a/docs/workplan-planning-map.md b/docs/workplan-planning-map.md index efed49a..f3b4b2c 100644 --- a/docs/workplan-planning-map.md +++ b/docs/workplan-planning-map.md @@ -40,7 +40,7 @@ and descriptions mirror the operational view. | `MKTT-WP-0009` | complete | done | `MKTT-WP-0006` | Access-controlled knowledge gateway is complete: local labels, trust zones, path rules, policy-aware cache query/search, decisions, diagnostics, and external adapter boundaries. | | `MKTT-WP-0014` | complete | done | `MKTT-WP-0009` | Markitect-side enterprise IAM access-control integration is complete: NetKingdom/key-cape-compatible identity claims, flex-auth resource/policy contract, directory group resolution fixtures, decision-log sink, workflow declarations, CLI commands, and external PDP request examples. | | `MKTT-WP-0012` | complete | done | `MKTT-WP-0004`, `MKTT-WP-0010`, `MKTT-WP-0011` | Document function layer is complete: deterministic Markdown-native function descriptors, registry, inline/fenced syntax, pipelines, context bindings, CLI, docs, examples, diagnostics, provenance, and extension descriptor. | -| `MKTT-WP-0008` | P3 | todo | `MKTT-WP-0006`, `MKTT-WP-0007`, `MKTT-WP-0009` | Agent working-memory cache after backend and policy floor are available. | +| `MKTT-WP-0008` | complete | done | `MKTT-WP-0006`, `MKTT-WP-0007`, `MKTT-WP-0009` | Agent working-memory context cache is complete: context package schema, local registry, package creation from queries/search/manifests, deterministic summaries, namespaces, activation/deactivation/refresh/explain lifecycle, policy re-checks, CLI, docs, and examples. | | `MKTT-WP-0015` | P2 | todo | `MKTT-WP-0010`, `MKTT-WP-0011`, `MKTT-WP-0012` | Future render and document-function extensions: typed values, richer syntax, document-local reusable functions, Quarkdown/export adapters, render-aware references, assets, and permission sandboxing. Defer unless publishing/export pressure becomes current. | ## Dependency Notes @@ -91,8 +91,8 @@ protocols. A live flex-auth service can improve enterprise deployment, central policy administration, and durable audit, but it is not a prerequisite for the document function layer or local agent context packages. -Remaining Markitect workplans, including `MKTT-WP-0008` and the future -`MKTT-WP-0015` extension track, should keep this policy posture: +Remaining Markitect workplans, including the future `MKTT-WP-0015` extension +track, should keep this policy posture: - use `AccessPolicyGateway`, `PolicySubject`, `PolicyObject`, and `PolicyDecision` as local contracts @@ -103,6 +103,12 @@ Remaining Markitect workplans, including `MKTT-WP-0008` and the future parsing, deterministic functions, workflows, cache queries, or context package lifecycle +`MKTT-WP-0008` completed the local agent working-memory package layer. It +deliberately implements explicit, inspectable context packages rather than +hidden ambient agent memory. LLM-assisted summaries, embeddings, vector stores, +remote registries, retention/decay, and reviewed long-term identity packages +remain future optional extensions. + ## State Hub Mirror Native State Hub dependency edges should mirror the whole-workstream diff --git a/examples/memory/workplan-context.manifest.yaml b/examples/memory/workplan-context.manifest.yaml new file mode 100644 index 0000000..81a1aeb --- /dev/null +++ b/examples/memory/workplan-context.manifest.yaml @@ -0,0 +1,19 @@ +title: Agent working memory workplan context +intent: Reactivate the purpose and implementation scope for WP-0008. +namespace: + project: markitect-tool + task: MKTT-WP-0008 +budget: + max_items: 4 + max_tokens: 1200 +retrieval_recipes: + - kind: selector + query: sections[heading=Purpose] + engine: selector + sources: + - workplans/MKTT-WP-0008-agent-working-memory-context-cache.md + - kind: selector + query: sections[heading=Architectural Boundary] + engine: selector + sources: + - workplans/MKTT-WP-0008-agent-working-memory-context-cache.md diff --git a/src/markitect_tool/__init__.py b/src/markitect_tool/__init__.py index a43b831..5f031ac 100644 --- a/src/markitect_tool/__init__.py +++ b/src/markitect_tool/__init__.py @@ -114,6 +114,26 @@ from markitect_tool.literate import ( weave_markdown, write_tangle_files, ) +from markitect_tool.memory import ( + ContextActivation, + ContextBudget, + ContextPackage, + ContextPackageError, + ContextPackageItem, + LocalContextPackageRegistry, + MemoryNamespace, + RetrievalRecipe, + SourceSpan as MemorySourceSpan, + SummaryLayer, + activate_context_package, + create_context_package_from_index, + create_context_package_from_manifest, + create_context_package_from_sources, + deactivate_context_package, + explain_context_package, + load_context_package_file, + refresh_context_package, +) from markitect_tool.ops import ( ComposeResult, IncludeError, @@ -314,6 +334,24 @@ __all__ = [ "tangle_markdown", "weave_markdown", "write_tangle_files", + "ContextActivation", + "ContextBudget", + "ContextPackage", + "ContextPackageError", + "ContextPackageItem", + "LocalContextPackageRegistry", + "MemoryNamespace", + "RetrievalRecipe", + "MemorySourceSpan", + "SummaryLayer", + "activate_context_package", + "create_context_package_from_index", + "create_context_package_from_manifest", + "create_context_package_from_sources", + "deactivate_context_package", + "explain_context_package", + "load_context_package_file", + "refresh_context_package", "ComposeResult", "IncludeError", "IncludeResult", diff --git a/src/markitect_tool/cli/main.py b/src/markitect_tool/cli/main.py index 90dd7ef..ab03050 100644 --- a/src/markitect_tool/cli/main.py +++ b/src/markitect_tool/cli/main.py @@ -57,6 +57,18 @@ from markitect_tool.generation import ( run_generation_plan, ) from markitect_tool.literate import tangle_markdown, weave_markdown, write_tangle_files +from markitect_tool.memory import ( + ContextBudget, + ContextPackageError, + LocalContextPackageRegistry, + MemoryNamespace, + activate_context_package, + create_context_package_from_index, + create_context_package_from_manifest, + create_context_package_from_sources, + explain_context_package, + refresh_context_package, +) from markitect_tool.ops import IncludeError, compose_files, resolve_includes, transform_markdown from markitect_tool.processor import ProcessorContext, run_fenced_processors from markitect_tool.policy import ( @@ -1392,6 +1404,373 @@ def search( _emit_search_results(data, output_format) +@main.group("context") +def context_group() -> None: + """Pack and activate agent working-memory context.""" + + +@context_group.command("pack") +@click.argument("query_or_manifest") +@click.option( + "--source", + "sources", + multiple=True, + type=click.Path(exists=True, dir_okay=False, path_type=Path), + help="Markdown source file to query directly. May be repeated.", +) +@click.option( + "--root", + type=click.Path(exists=True, file_okay=False, path_type=Path), + default=Path("."), + show_default=True, + help="Root used for relative paths and the local context registry.", +) +@click.option( + "--index-path", + type=click.Path(dir_okay=False, path_type=Path), + help="SQLite index path. Defaults to .markitect/cache/index.sqlite3 under root.", +) +@click.option("--search", is_flag=True, help="Treat the argument as an FTS search query.") +@click.option("--path", "paths", multiple=True, help="Restrict indexed query/search to relative paths.") +@click.option( + "--engine", + type=click.Choice(["selector", "jsonpath"], case_sensitive=False), + default="selector", + show_default=True, + help="Query engine for selector packages.", +) +@click.option("--limit", type=int, default=20, show_default=True, help="Maximum FTS search matches.") +@click.option("--title", help="Package title.") +@click.option("--intent", help="Package intent statement.") +@click.option("--project", help="Project namespace.") +@click.option("--user", "user_ns", help="User namespace.") +@click.option("--agent", help="Agent namespace.") +@click.option("--thread", help="Thread namespace.") +@click.option("--task", help="Task namespace.") +@click.option("--namespace", "namespace_pairs", multiple=True, metavar="KEY=VALUE", help="Custom namespace key.") +@click.option("--max-items", type=int, help="Maximum context items to include.") +@click.option("--max-tokens", type=int, help="Approximate maximum package tokens.") +@click.option("--reserve-tokens", type=int, default=0, show_default=True, help="Token reserve inside max tokens.") +@click.option( + "--policy", + "policy_file", + type=click.Path(exists=True, dir_okay=False, path_type=Path), + help="Local label policy file used to filter package items.", +) +@click.option("--subject", default="anonymous", show_default=True, help="Policy subject id.") +@click.option( + "--policy-mode", + type=click.Choice(["off", "audit", "enforce"], case_sensitive=False), + help="Override policy mode while packing.", +) +@click.option("--action", default="package", show_default=True, help="Policy action used while packing.") +@click.option("--output", type=click.Path(dir_okay=False, path_type=Path), help="Write package to this file.") +@click.option("--no-save", is_flag=True, help="Do not save to the local context registry.") +@click.option( + "--format", + "output_format", + type=click.Choice(["json", "yaml", "text"], case_sensitive=False), + default="text", + show_default=True, +) +def context_pack( + query_or_manifest: str, + sources: tuple[Path, ...], + root: Path, + index_path: Path | None, + search: bool, + paths: tuple[str, ...], + engine: str, + limit: int, + title: str | None, + intent: str | None, + project: str | None, + user_ns: str | None, + agent: str | None, + thread: str | None, + task: str | None, + namespace_pairs: tuple[str, ...], + max_items: int | None, + max_tokens: int | None, + reserve_tokens: int, + policy_file: Path | None, + subject: str, + policy_mode: str | None, + action: str, + output: Path | None, + no_save: bool, + output_format: str, +) -> None: + """Create an inspectable context package from a query, search, or manifest.""" + + try: + namespace = _memory_namespace(project, user_ns, agent, thread, task, namespace_pairs) + budget = ContextBudget(max_tokens=max_tokens, max_items=max_items, reserve_tokens=reserve_tokens) + gateway = _load_policy_gateway(policy_file, policy_mode) + manifest = Path(query_or_manifest) + if manifest.exists() and manifest.is_file() and not sources: + package = create_context_package_from_manifest( + manifest, + root=root, + budget=budget, + policy_gateway=gateway, + subject=subject, + action=action, + ) + elif sources: + package = create_context_package_from_sources( + query_or_manifest, + list(sources), + root=root, + engine=engine, + title=title, + intent=intent, + namespace=namespace, + budget=budget, + policy_gateway=gateway, + subject=subject, + action=action, + ) + else: + package = create_context_package_from_index( + query_or_manifest, + root=root, + index_path=index_path, + engine=engine, + paths=list(paths) or None, + search=search, + limit=limit, + title=title, + intent=intent, + namespace=namespace, + budget=budget, + policy_gateway=gateway, + subject=subject, + action=action, + ) + except (ContextPackageError, InvalidQueryError, ValueError) as exc: + raise click.ClickException(str(exc)) from exc + registry_path = None + if not no_save: + registry_path = LocalContextPackageRegistry(root).save(package) + if output: + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(yaml.safe_dump(package.to_dict(), sort_keys=False), encoding="utf-8") + _emit_context_package( + package.to_dict() + | { + "registry_path": str(registry_path) if registry_path else None, + "output_path": str(output) if output else None, + }, + output_format, + ) + + +@context_group.command("activate") +@click.argument("package") +@click.option( + "--root", + type=click.Path(exists=True, file_okay=False, path_type=Path), + default=Path("."), + show_default=True, + help="Root used for the local context registry.", +) +@click.option("--target", default="default", show_default=True, help="Thread/workspace activation target.") +@click.option("--max-items", type=int, help="Maximum context items to activate.") +@click.option("--max-tokens", type=int, help="Approximate maximum activation tokens.") +@click.option("--reserve-tokens", type=int, default=0, show_default=True, help="Token reserve inside max tokens.") +@click.option( + "--policy", + "policy_file", + type=click.Path(exists=True, dir_okay=False, path_type=Path), + help="Local label policy file used to re-check items before activation.", +) +@click.option("--subject", default="anonymous", show_default=True, help="Policy subject id.") +@click.option( + "--policy-mode", + type=click.Choice(["off", "audit", "enforce"], case_sensitive=False), + help="Override policy mode while activating.", +) +@click.option("--action", default="read", show_default=True, help="Policy action used while activating.") +@click.option( + "--format", + "output_format", + type=click.Choice(["markdown", "json", "yaml", "text"], case_sensitive=False), + default="markdown", + show_default=True, +) +def context_activate( + package: str, + root: Path, + target: str, + max_items: int | None, + max_tokens: int | None, + reserve_tokens: int, + policy_file: Path | None, + subject: str, + policy_mode: str | None, + action: str, + output_format: str, +) -> None: + """Activate a saved context package as Markdown working memory.""" + + try: + registry = LocalContextPackageRegistry(root) + loaded = registry.load(package) + activation = activate_context_package( + loaded, + target=target, + policy_gateway=_load_policy_gateway(policy_file, policy_mode), + subject=subject, + action=action, + budget=ContextBudget(max_tokens=max_tokens, max_items=max_items, reserve_tokens=reserve_tokens), + ) + activation_path = registry.save_activation(activation) + registry.save(loaded.with_activation_state("active")) + except (ContextPackageError, ValueError) as exc: + raise click.ClickException(str(exc)) from exc + _emit_context_activation( + activation.to_dict() | {"activation_path": str(activation_path)}, + output_format, + ) + + +@context_group.command("deactivate") +@click.argument("activation_id") +@click.option( + "--root", + type=click.Path(exists=True, file_okay=False, path_type=Path), + default=Path("."), + show_default=True, + help="Root used for the local context registry.", +) +@click.option( + "--format", + "output_format", + type=click.Choice(["json", "yaml", "text"], case_sensitive=False), + default="text", + show_default=True, +) +def context_deactivate(activation_id: str, root: Path, output_format: str) -> None: + """Deactivate a recorded context activation.""" + + try: + registry = LocalContextPackageRegistry(root) + registry.deactivate(activation_id) + activation = registry.load_activation(activation_id) + except ContextPackageError as exc: + raise click.ClickException(str(exc)) from exc + _emit_context_activation(activation.to_dict(), output_format) + + +@context_group.command("explain") +@click.argument("package") +@click.option( + "--root", + type=click.Path(exists=True, file_okay=False, path_type=Path), + default=Path("."), + show_default=True, + help="Root used for the local context registry.", +) +@click.option( + "--format", + "output_format", + type=click.Choice(["json", "yaml", "text"], case_sensitive=False), + default="text", + show_default=True, +) +def context_explain(package: str, root: Path, output_format: str) -> None: + """Explain context package contents, retrieval, budget, and policy metadata.""" + + try: + loaded = LocalContextPackageRegistry(root).load(package) + except ContextPackageError as exc: + raise click.ClickException(str(exc)) from exc + _emit_context_explain(explain_context_package(loaded), output_format) + + +@context_group.command("refresh") +@click.argument("package") +@click.option( + "--root", + type=click.Path(exists=True, file_okay=False, path_type=Path), + default=Path("."), + show_default=True, + help="Root used for the local context registry.", +) +@click.option( + "--policy", + "policy_file", + type=click.Path(exists=True, dir_okay=False, path_type=Path), + help="Local label policy file used to filter refreshed package items.", +) +@click.option("--subject", default="anonymous", show_default=True, help="Policy subject id.") +@click.option( + "--policy-mode", + type=click.Choice(["off", "audit", "enforce"], case_sensitive=False), + help="Override policy mode while refreshing.", +) +@click.option("--action", default="package", show_default=True, help="Policy action used while refreshing.") +@click.option("--no-save", is_flag=True, help="Do not save the refreshed package.") +@click.option( + "--format", + "output_format", + type=click.Choice(["json", "yaml", "text"], case_sensitive=False), + default="text", + show_default=True, +) +def context_refresh( + package: str, + root: Path, + policy_file: Path | None, + subject: str, + policy_mode: str | None, + action: str, + no_save: bool, + output_format: str, +) -> None: + """Refresh a package by re-running its retrieval recipes.""" + + try: + registry = LocalContextPackageRegistry(root) + loaded = registry.load(package) + refreshed = refresh_context_package( + loaded, + policy_gateway=_load_policy_gateway(policy_file, policy_mode), + subject=subject, + action=action, + ) + registry_path = None if no_save else registry.save(refreshed) + except (ContextPackageError, ValueError) as exc: + raise click.ClickException(str(exc)) from exc + _emit_context_package( + refreshed.to_dict() | {"registry_path": str(registry_path) if registry_path else None}, + output_format, + ) + + +@context_group.command("list") +@click.option( + "--root", + type=click.Path(exists=True, file_okay=False, path_type=Path), + default=Path("."), + show_default=True, + help="Root used for the local context registry.", +) +@click.option( + "--format", + "output_format", + type=click.Choice(["json", "yaml", "text"], case_sensitive=False), + default="text", + show_default=True, +) +def context_list(root: Path, output_format: str) -> None: + """List locally saved context packages.""" + + packages = [package.to_dict() for package in LocalContextPackageRegistry(root).list()] + _emit_context_package_list({"count": len(packages), "packages": packages}, output_format) + + @main.group() def workflow() -> None: """Inspect, plan, and run declarative Markdown workflows.""" @@ -1809,6 +2188,25 @@ def _load_policy_gateway( raise click.ClickException(str(exc)) from exc +def _memory_namespace( + project: str | None, + user: str | None, + agent: str | None, + thread: str | None, + task: str | None, + namespace_pairs: tuple[str, ...], +) -> MemoryNamespace: + custom = _parse_key_value_options(namespace_pairs) + return MemoryNamespace( + project=project, + user=user, + agent=agent, + thread=thread, + task=task, + custom=custom, + ) + + def _emit_result(data: dict, output_format: str) -> None: if output_format == "json": click.echo(json.dumps(data, indent=2, ensure_ascii=False)) @@ -2084,6 +2482,83 @@ def _emit_search_results(data: dict, output_format: str) -> None: click.echo(f"! [{diagnostic['severity']}] {diagnostic['code']}: {diagnostic['message']}") +def _emit_context_package(data: dict, output_format: str) -> None: + if output_format == "json": + click.echo(json.dumps(data, indent=2, ensure_ascii=False)) + elif output_format == "yaml": + click.echo(yaml.safe_dump(data, sort_keys=False)) + else: + click.echo(f"package: {data['id']}") + click.echo(f"title: {data.get('title', '')}") + click.echo(f"items: {len(data.get('items', []))}") + click.echo(f"token_estimate: {data.get('token_estimate', 0)}") + if data.get("registry_path"): + click.echo(f"registry_path: {data['registry_path']}") + if data.get("output_path"): + click.echo(f"output_path: {data['output_path']}") + if data.get("policy", {}).get("summary"): + _emit_policy_summary(data["policy"]["summary"]) + for summary in data.get("summaries", []): + click.echo(f"- {summary['name']}: {summary['text']}") + + +def _emit_context_activation(data: dict, output_format: str) -> None: + if output_format == "json": + click.echo(json.dumps(data, indent=2, ensure_ascii=False)) + elif output_format == "yaml": + click.echo(yaml.safe_dump(data, sort_keys=False)) + elif output_format == "markdown": + click.echo(data.get("content", ""), nl=False) + else: + click.echo(f"activation: {data['id']}") + click.echo(f"status: {data.get('status')}") + click.echo(f"package: {data.get('package_id')}") + click.echo(f"items: {len(data.get('items', []))}") + click.echo(f"token_estimate: {data.get('token_estimate', 0)}") + if data.get("activation_path"): + click.echo(f"activation_path: {data['activation_path']}") + if data.get("policy", {}).get("summary"): + _emit_policy_summary(data["policy"]["summary"]) + for diagnostic in data.get("diagnostics", []): + click.echo(f"! [{diagnostic['severity']}] {diagnostic['code']}: {diagnostic['message']}") + + +def _emit_context_explain(data: dict, output_format: str) -> None: + if output_format == "json": + click.echo(json.dumps(data, indent=2, ensure_ascii=False)) + elif output_format == "yaml": + click.echo(yaml.safe_dump(data, sort_keys=False)) + else: + click.echo(f"package: {data['id']}") + click.echo(f"title: {data.get('title', '')}") + click.echo(f"intent: {data.get('intent', '')}") + click.echo(f"activation_state: {data.get('activation_state', 'inactive')}") + click.echo(f"items: {data.get('items', 0)}") + click.echo(f"token_estimate: {data.get('token_estimate', 0)}") + if data.get("namespace"): + click.echo(f"namespace: {data['namespace']}") + if data.get("retrieval_recipes"): + click.echo("retrieval_recipes:") + for recipe in data["retrieval_recipes"]: + click.echo(f"- {recipe.get('kind')} {recipe.get('query')}") + if data.get("sources"): + click.echo("sources:") + for source in data["sources"]: + span = f":{source.get('line_start')}" if source.get("line_start") else "" + click.echo(f"- {source.get('path')}{span} {source.get('unit_kind', '')}") + + +def _emit_context_package_list(data: dict, output_format: str) -> None: + if output_format == "json": + click.echo(json.dumps(data, indent=2, ensure_ascii=False)) + elif output_format == "yaml": + click.echo(yaml.safe_dump(data, sort_keys=False)) + else: + click.echo(f"packages: {data.get('count', 0)}") + for package in data.get("packages", []): + click.echo(f"- {package['id']} {package.get('title', '')}") + + def _emit_policy_summary(policy_data: dict) -> None: click.echo( "policy: " diff --git a/src/markitect_tool/extension/builtins.py b/src/markitect_tool/extension/builtins.py index 845b71f..fdc01e2 100644 --- a/src/markitect_tool/extension/builtins.py +++ b/src/markitect_tool/extension/builtins.py @@ -19,6 +19,7 @@ def builtin_extension_registry() -> ExtensionRegistry: _runtime_assessment_descriptor(), _local_label_policy_descriptor(), _document_function_descriptor(), + _agent_memory_descriptor(), ]: registry.register(descriptor) return registry @@ -265,3 +266,51 @@ def _document_function_descriptor() -> ExtensionDescriptor: "external_policy_services_required": False, }, ) + + +def _agent_memory_descriptor() -> ExtensionDescriptor: + return ExtensionDescriptor( + id="memory.context-package", + kind="memory-registry", + summary="Local agent working-memory context package registry and activation lifecycle.", + capabilities=[ + ProcessingCapability(id="context_packages", kind="create"), + ProcessingCapability(id="context_activation", kind="execute"), + ProcessingCapability(id="snapshots", kind="read"), + ProcessingCapability(id="fts", kind="read"), + ProcessingCapability(id="policy_filter", kind="filter"), + ProcessingCapability(id="provenance", kind="emit"), + ProcessingCapability(id="diagnostics", kind="emit"), + ], + safety={ + "reads_files": True, + "writes_local_context_registry": True, + "network": False, + "assisted_generation": False, + "external_policy_engine": False, + }, + input_contract="Selector/search/manifest + local snapshots/documents", + output_contract="ContextPackage | ContextActivation", + diagnostics_namespace="memory", + provenance_prefix="memory.context_package", + cli={ + "commands": [ + "mkt context pack", + "mkt context activate", + "mkt context deactivate", + "mkt context explain", + "mkt context refresh", + "mkt context list", + ] + }, + docs=[ + "docs/agent-working-memory.md", + "docs/agent-working-memory-thought-experiment.md", + ], + examples=["examples/memory/workplan-context.manifest.yaml"], + metadata={ + "external_policy_services_required": False, + "assisted_summaries": "future adapter-only", + "default_registry": ".markitect/context", + }, + ) diff --git a/src/markitect_tool/memory/__init__.py b/src/markitect_tool/memory/__init__.py new file mode 100644 index 0000000..ef04bb3 --- /dev/null +++ b/src/markitect_tool/memory/__init__.py @@ -0,0 +1,43 @@ +"""Agent working-memory context packages.""" + +from markitect_tool.memory.engine import ( + ContextActivation, + ContextBudget, + ContextPackage, + ContextPackageError, + ContextPackageItem, + LocalContextPackageRegistry, + MemoryNamespace, + RetrievalRecipe, + SourceSpan, + SummaryLayer, + activate_context_package, + create_context_package_from_index, + create_context_package_from_manifest, + create_context_package_from_sources, + deactivate_context_package, + explain_context_package, + load_context_package_file, + refresh_context_package, +) + +__all__ = [ + "ContextActivation", + "ContextBudget", + "ContextPackage", + "ContextPackageError", + "ContextPackageItem", + "LocalContextPackageRegistry", + "MemoryNamespace", + "RetrievalRecipe", + "SourceSpan", + "SummaryLayer", + "activate_context_package", + "create_context_package_from_index", + "create_context_package_from_manifest", + "create_context_package_from_sources", + "deactivate_context_package", + "explain_context_package", + "load_context_package_file", + "refresh_context_package", +] diff --git a/src/markitect_tool/memory/engine.py b/src/markitect_tool/memory/engine.py new file mode 100644 index 0000000..3328b89 --- /dev/null +++ b/src/markitect_tool/memory/engine.py @@ -0,0 +1,1255 @@ +"""Local agent working-memory context package engine.""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import asdict, dataclass, field +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import yaml + +from markitect_tool.backend import LocalSnapshotStore, local_index_path_for +from markitect_tool.core import Document, parse_markdown_file +from markitect_tool.policy import LocalLabelPolicyGateway, policy_metadata_from_document +from markitect_tool.query import ( + InvalidQueryError, + QueryMatch, + query_document, + query_document_jsonpath, +) + + +DEFAULT_CONTEXT_REGISTRY_PATH = ".markitect/context" + + +class ContextPackageError(ValueError): + """Raised when a context package cannot be created, loaded, or activated.""" + + +def _now() -> str: + return datetime.now(timezone.utc).isoformat() + + +@dataclass(frozen=True) +class MemoryNamespace: + """Stable namespace coordinates for agent memory packages.""" + + project: str | None = None + user: str | None = None + agent: str | None = None + thread: str | None = None + task: str | None = None + custom: dict[str, Any] = field(default_factory=dict) + + @classmethod + def from_mapping(cls, data: dict[str, Any] | None) -> "MemoryNamespace": + raw = dict(data or {}) + known = { + "project": raw.pop("project", None), + "user": raw.pop("user", None), + "agent": raw.pop("agent", None), + "thread": raw.pop("thread", None), + "task": raw.pop("task", None), + } + custom = dict(raw.pop("custom", {}) or {}) | raw + return cls( + project=_optional_str(known["project"]), + user=_optional_str(known["user"]), + agent=_optional_str(known["agent"]), + thread=_optional_str(known["thread"]), + task=_optional_str(known["task"]), + custom=custom, + ) + + def to_dict(self) -> dict[str, Any]: + return _drop_empty(asdict(self)) + + +@dataclass(frozen=True) +class ContextBudget: + """Token and item budget applied while creating or activating context.""" + + max_tokens: int | None = None + max_items: int | None = None + reserve_tokens: int = 0 + strategy: str = "first-fit" + + @classmethod + def from_mapping(cls, data: dict[str, Any] | None) -> "ContextBudget": + raw = dict(data or {}) + return cls( + max_tokens=_optional_int(raw.get("max_tokens")), + max_items=_optional_int(raw.get("max_items")), + reserve_tokens=int(raw.get("reserve_tokens") or 0), + strategy=str(raw.get("strategy") or "first-fit"), + ) + + def to_dict(self) -> dict[str, Any]: + return _drop_empty(asdict(self)) + + +@dataclass(frozen=True) +class SourceSpan: + """Source range represented in a memory package.""" + + path: str + snapshot_id: str | None = None + unit_kind: str | None = None + unit_index: int | None = None + line_start: int | None = None + line_end: int | None = None + selector: str | None = None + engine: str | None = None + + @classmethod + def from_mapping(cls, data: dict[str, Any]) -> "SourceSpan": + return cls( + path=str(data["path"]), + snapshot_id=_optional_str(data.get("snapshot_id")), + unit_kind=_optional_str(data.get("unit_kind")), + unit_index=_optional_int(data.get("unit_index")), + line_start=_optional_int(data.get("line_start")), + line_end=_optional_int(data.get("line_end")), + selector=_optional_str(data.get("selector")), + engine=_optional_str(data.get("engine")), + ) + + def to_dict(self) -> dict[str, Any]: + return _drop_empty(asdict(self)) + + +@dataclass(frozen=True) +class RetrievalRecipe: + """Recipe that can refresh memory package contents.""" + + kind: str + query: str + engine: str = "selector" + sources: list[str] = field(default_factory=list) + paths: list[str] = field(default_factory=list) + root: str = "." + index_path: str | None = None + limit: int | None = None + metadata: dict[str, Any] = field(default_factory=dict) + + @classmethod + def from_mapping(cls, data: dict[str, Any]) -> "RetrievalRecipe": + return cls( + kind=str(data.get("kind") or data.get("type") or "selector"), + query=str(data.get("query") or data.get("selector") or data.get("text") or ""), + engine=str(data.get("engine") or "selector"), + sources=_string_list(data.get("sources") or data.get("source")), + paths=_string_list(data.get("paths") or data.get("path")), + root=str(data.get("root") or "."), + index_path=_optional_str(data.get("index_path")), + limit=_optional_int(data.get("limit")), + metadata=dict(data.get("metadata") or {}), + ) + + def to_dict(self) -> dict[str, Any]: + return _drop_empty(asdict(self)) + + +@dataclass(frozen=True) +class SummaryLayer: + """One deterministic or assisted summary layer.""" + + name: str + kind: str + text: str + token_estimate: int + generated_by: str = "markitect.deterministic" + metadata: dict[str, Any] = field(default_factory=dict) + + @classmethod + def from_mapping(cls, data: dict[str, Any]) -> "SummaryLayer": + return cls( + name=str(data["name"]), + kind=str(data.get("kind") or "extractive"), + text=str(data.get("text") or ""), + token_estimate=int(data.get("token_estimate") or estimate_tokens(str(data.get("text") or ""))), + generated_by=str(data.get("generated_by") or "markitect.deterministic"), + metadata=dict(data.get("metadata") or {}), + ) + + def to_dict(self) -> dict[str, Any]: + return _drop_empty(asdict(self)) + + +@dataclass(frozen=True) +class ContextPackageItem: + """One item included in an agent-ready context package.""" + + id: str + source: SourceSpan + text: str + summary: str + token_estimate: int + score: float | None = None + policy: dict[str, Any] = field(default_factory=dict) + provenance: dict[str, Any] = field(default_factory=dict) + metadata: dict[str, Any] = field(default_factory=dict) + + @classmethod + def create( + cls, + *, + source: SourceSpan, + text: str, + summary: str | None = None, + score: float | None = None, + policy: dict[str, Any] | None = None, + provenance: dict[str, Any] | None = None, + metadata: dict[str, Any] | None = None, + ) -> "ContextPackageItem": + item_summary = summary or summarize_text(text) + payload = { + "source": source.to_dict(), + "text_hash": _hash_text(text), + } + return cls( + id="memory:item:" + _stable_hash(payload), + source=source, + text=text, + summary=item_summary, + token_estimate=estimate_tokens(text), + score=score, + policy=policy or {}, + provenance=provenance or {}, + metadata=metadata or {}, + ) + + @classmethod + def from_mapping(cls, data: dict[str, Any]) -> "ContextPackageItem": + return cls( + id=str(data["id"]), + source=SourceSpan.from_mapping(data["source"]), + text=str(data.get("text") or ""), + summary=str(data.get("summary") or ""), + token_estimate=int(data.get("token_estimate") or estimate_tokens(str(data.get("text") or ""))), + score=float(data["score"]) if data.get("score") is not None else None, + policy=dict(data.get("policy") or {}), + provenance=dict(data.get("provenance") or {}), + metadata=dict(data.get("metadata") or {}), + ) + + def to_dict(self) -> dict[str, Any]: + return _drop_empty( + { + "id": self.id, + "source": self.source.to_dict(), + "text": self.text, + "summary": self.summary, + "token_estimate": self.token_estimate, + "score": self.score, + "policy": self.policy, + "provenance": self.provenance, + "metadata": self.metadata, + } + ) + + +@dataclass(frozen=True) +class ContextPackage: + """Portable, inspectable context package for agent working memory.""" + + id: str + title: str + intent: str + namespace: MemoryNamespace + items: list[ContextPackageItem] + retrieval_recipes: list[RetrievalRecipe] = field(default_factory=list) + summaries: list[SummaryLayer] = field(default_factory=list) + budget: ContextBudget = field(default_factory=ContextBudget) + activation_state: str = "inactive" + created_at: str = field(default_factory=_now) + updated_at: str = field(default_factory=_now) + freshness: dict[str, Any] = field(default_factory=dict) + policy: dict[str, Any] = field(default_factory=dict) + provenance: list[dict[str, Any]] = field(default_factory=list) + metadata: dict[str, Any] = field(default_factory=dict) + + @classmethod + def create( + cls, + *, + title: str, + intent: str, + namespace: MemoryNamespace | None = None, + items: list[ContextPackageItem] | None = None, + retrieval_recipes: list[RetrievalRecipe] | None = None, + summaries: list[SummaryLayer] | None = None, + budget: ContextBudget | None = None, + package_id: str | None = None, + freshness: dict[str, Any] | None = None, + policy: dict[str, Any] | None = None, + provenance: list[dict[str, Any]] | None = None, + metadata: dict[str, Any] | None = None, + ) -> "ContextPackage": + selected_items = _apply_budget(items or [], budget or ContextBudget()) + selected_summaries = summaries or build_summary_layers(selected_items) + package_payload = { + "title": title, + "intent": intent, + "namespace": (namespace or MemoryNamespace()).to_dict(), + "recipes": [recipe.to_dict() for recipe in retrieval_recipes or []], + "items": [item.id for item in selected_items], + "budget": (budget or ContextBudget()).to_dict(), + } + return cls( + id=package_id or "memory:package:" + _stable_hash(package_payload), + title=title, + intent=intent, + namespace=namespace or MemoryNamespace(), + items=selected_items, + retrieval_recipes=retrieval_recipes or [], + summaries=selected_summaries, + budget=budget or ContextBudget(), + freshness=freshness or _freshness(selected_items), + policy=policy or {}, + provenance=provenance or [], + metadata=metadata or {}, + ) + + @classmethod + def from_mapping(cls, data: dict[str, Any]) -> "ContextPackage": + return cls( + id=str(data["id"]), + title=str(data.get("title") or data["id"]), + intent=str(data.get("intent") or ""), + namespace=MemoryNamespace.from_mapping(data.get("namespace")), + items=[ContextPackageItem.from_mapping(item) for item in data.get("items", [])], + retrieval_recipes=[RetrievalRecipe.from_mapping(recipe) for recipe in data.get("retrieval_recipes", [])], + summaries=[SummaryLayer.from_mapping(summary) for summary in data.get("summaries", [])], + budget=ContextBudget.from_mapping(data.get("budget")), + activation_state=str(data.get("activation_state") or "inactive"), + created_at=str(data.get("created_at") or _now()), + updated_at=str(data.get("updated_at") or _now()), + freshness=dict(data.get("freshness") or {}), + policy=dict(data.get("policy") or {}), + provenance=list(data.get("provenance") or []), + metadata=dict(data.get("metadata") or {}), + ) + + @property + def token_estimate(self) -> int: + return sum(item.token_estimate for item in self.items) + sum( + summary.token_estimate for summary in self.summaries + ) + + def to_dict(self) -> dict[str, Any]: + return _drop_empty( + { + "id": self.id, + "title": self.title, + "intent": self.intent, + "namespace": self.namespace.to_dict(), + "activation_state": self.activation_state, + "created_at": self.created_at, + "updated_at": self.updated_at, + "freshness": self.freshness, + "budget": self.budget.to_dict(), + "token_estimate": self.token_estimate, + "retrieval_recipes": [recipe.to_dict() for recipe in self.retrieval_recipes], + "summaries": [summary.to_dict() for summary in self.summaries], + "items": [item.to_dict() for item in self.items], + "policy": self.policy, + "provenance": self.provenance, + "metadata": self.metadata, + } + ) + + def with_activation_state(self, state: str) -> "ContextPackage": + return ContextPackage( + id=self.id, + title=self.title, + intent=self.intent, + namespace=self.namespace, + items=self.items, + retrieval_recipes=self.retrieval_recipes, + summaries=self.summaries, + budget=self.budget, + activation_state=state, + created_at=self.created_at, + updated_at=_now(), + freshness=self.freshness, + policy=self.policy, + provenance=self.provenance, + metadata=self.metadata, + ) + + +@dataclass(frozen=True) +class ContextActivation: + """Result of activating or deactivating a context package.""" + + id: str + package_id: str + status: str + target: str + content: str + items: list[ContextPackageItem] + summaries: list[SummaryLayer] = field(default_factory=list) + token_estimate: int = 0 + created_at: str = field(default_factory=_now) + policy: dict[str, Any] = field(default_factory=dict) + policy_decisions: list[dict[str, Any]] = field(default_factory=list) + diagnostics: list[dict[str, Any]] = field(default_factory=list) + metadata: dict[str, Any] = field(default_factory=dict) + + @classmethod + def from_mapping(cls, data: dict[str, Any]) -> "ContextActivation": + return cls( + id=str(data["id"]), + package_id=str(data["package_id"]), + status=str(data.get("status") or "active"), + target=str(data.get("target") or "default"), + content=str(data.get("content") or ""), + items=[ContextPackageItem.from_mapping(item) for item in data.get("items", [])], + summaries=[SummaryLayer.from_mapping(summary) for summary in data.get("summaries", [])], + token_estimate=int(data.get("token_estimate") or 0), + created_at=str(data.get("created_at") or _now()), + policy=dict(data.get("policy") or {}), + policy_decisions=list(data.get("policy_decisions") or []), + diagnostics=list(data.get("diagnostics") or []), + metadata=dict(data.get("metadata") or {}), + ) + + def to_dict(self) -> dict[str, Any]: + return _drop_empty( + { + "id": self.id, + "package_id": self.package_id, + "status": self.status, + "target": self.target, + "content": self.content, + "items": [item.to_dict() for item in self.items], + "summaries": [summary.to_dict() for summary in self.summaries], + "token_estimate": self.token_estimate, + "created_at": self.created_at, + "policy": self.policy, + "policy_decisions": self.policy_decisions, + "diagnostics": self.diagnostics, + "metadata": self.metadata, + } + ) + + +class LocalContextPackageRegistry: + """Filesystem-backed local context package registry.""" + + registry_id = "memory.local-context" + + def __init__(self, root: str | Path = ".") -> None: + self.root = Path(root) + self.path = self.root / DEFAULT_CONTEXT_REGISTRY_PATH + self.activation_path = self.path / "activations" + + def initialize(self) -> None: + self.path.mkdir(parents=True, exist_ok=True) + self.activation_path.mkdir(parents=True, exist_ok=True) + + def save(self, package: ContextPackage) -> Path: + self.initialize() + path = self.path_for(package.id) + path.write_text(yaml.safe_dump(package.to_dict(), sort_keys=False), encoding="utf-8") + return path + + def load(self, package_id_or_path: str | Path) -> ContextPackage: + return load_context_package_file(self.resolve(package_id_or_path)) + + def list(self) -> list[ContextPackage]: + if not self.path.exists(): + return [] + return [ + load_context_package_file(path) + for path in sorted(self.path.glob("*.context.yaml")) + ] + + def path_for(self, package_id: str) -> Path: + safe_id = package_id.replace(":", "-").replace("/", "-") + return self.path / f"{safe_id}.context.yaml" + + def resolve(self, package_id_or_path: str | Path) -> Path: + candidate = Path(package_id_or_path) + if candidate.exists(): + return candidate + path = self.path_for(str(package_id_or_path)) + if path.exists(): + return path + raise ContextPackageError(f"Unknown context package `{package_id_or_path}`.") + + def create_package( + self, + query_or_manifest: dict[str, Any], + budget: dict[str, Any] | None = None, + policy: dict[str, Any] | None = None, + ) -> str: + """Create a package from a serialized package or manifest mapping.""" + + if "items" in query_or_manifest: + package = ContextPackage.from_mapping(query_or_manifest) + else: + package = create_context_package_from_manifest_mapping( + query_or_manifest, + root=self.root, + budget=ContextBudget.from_mapping(budget), + package_policy=policy, + ) + self.save(package) + return package.id + + def activate(self, package_id: str, thread_or_workspace: str) -> str: + package = self.load(package_id) + activation = activate_context_package(package, target=thread_or_workspace) + self.save_activation(activation) + self.save(package.with_activation_state("active")) + return activation.id + + def save_activation(self, activation: ContextActivation) -> Path: + self.initialize() + path = self.activation_path / f"{activation.id.replace(':', '-')}.yaml" + path.write_text(yaml.safe_dump(activation.to_dict(), sort_keys=False), encoding="utf-8") + return path + + def load_activation(self, activation_id: str) -> ContextActivation: + path = self.activation_path / f"{activation_id.replace(':', '-')}.yaml" + if not path.exists(): + raise ContextPackageError(f"Unknown context activation `{activation_id}`.") + return ContextActivation.from_mapping(yaml.safe_load(path.read_text(encoding="utf-8")) or {}) + + def deactivate(self, activation_id: str) -> None: + path = self.activation_path / f"{activation_id.replace(':', '-')}.yaml" + activation = self.load_activation(activation_id) + inactive = deactivate_context_package(activation) + path.write_text(yaml.safe_dump(inactive.to_dict(), sort_keys=False), encoding="utf-8") + + def refresh(self, package_id: str) -> str: + package = self.load(package_id) + refreshed = refresh_context_package(package) + self.save(refreshed) + return refreshed.id + + def explain(self, package_id: str) -> dict[str, Any]: + return explain_context_package(self.load(package_id)) + + +def create_context_package_from_sources( + query: str, + sources: list[str | Path], + *, + root: str | Path = ".", + engine: str = "selector", + title: str | None = None, + intent: str | None = None, + namespace: MemoryNamespace | None = None, + budget: ContextBudget | None = None, + policy_gateway: LocalLabelPolicyGateway | None = None, + subject: str = "anonymous", + action: str = "package", + package_id: str | None = None, +) -> ContextPackage: + """Create a context package by querying Markdown source files directly.""" + + root_path = Path(root).resolve() + items: list[ContextPackageItem] = [] + policy_result: dict[str, Any] | None = None + for source in sources: + source_path = Path(source) + document = parse_markdown_file(source_path) + relative_path = _relative(source_path, root_path) + matches = _query(document, query, engine) + policy_metadata = policy_metadata_from_document(document.to_dict(), path=relative_path) + for match in matches: + items.append( + _item_from_query_match( + match, + source_path=relative_path, + selector=query, + engine=engine, + policy=policy_metadata, + snapshot_id=None, + ) + ) + if policy_gateway: + filtered = policy_gateway.filter_results( + subject, + action, + [item.to_dict() for item in items], + ) + items = [ContextPackageItem.from_mapping(item) for item in filtered["results"]] + policy_result = filtered + recipe = RetrievalRecipe( + kind="selector", + query=query, + engine=engine, + sources=[_relative(Path(source), root_path) for source in sources], + root=str(root_path), + ) + return ContextPackage.create( + package_id=package_id, + title=title or f"Context package for {query}", + intent=intent or "Reactivate selected Markdown context.", + namespace=namespace, + items=items, + retrieval_recipes=[recipe], + budget=budget, + policy=_policy_envelope(policy_result, policy_gateway, subject, action), + provenance=[_provenance("memory.pack.sources", {"query": query, "engine": engine})], + ) + + +def create_context_package_from_index( + query: str, + *, + root: str | Path = ".", + index_path: str | Path | None = None, + engine: str = "selector", + paths: list[str] | None = None, + search: bool = False, + limit: int = 20, + title: str | None = None, + intent: str | None = None, + namespace: MemoryNamespace | None = None, + budget: ContextBudget | None = None, + policy_gateway: LocalLabelPolicyGateway | None = None, + subject: str = "anonymous", + action: str = "package", + package_id: str | None = None, +) -> ContextPackage: + """Create a context package from the local SQLite index.""" + + root_path = Path(root).resolve() + resolved_index = local_index_path_for(root_path, index_path) + store = LocalSnapshotStore(resolved_index) + items = ( + _items_from_search(store, query, limit=limit) + if search + else _items_from_index_query(store, query, engine=engine, paths=paths) + ) + policy_result = None + if policy_gateway: + filtered = policy_gateway.filter_results( + subject, + action, + [item.to_dict() for item in items], + ) + items = [ContextPackageItem.from_mapping(item) for item in filtered["results"]] + policy_result = filtered + recipe = RetrievalRecipe( + kind="search" if search else "selector", + query=query, + engine="fts" if search else engine, + paths=paths or [], + root=str(root_path), + index_path=str(resolved_index), + limit=limit if search else None, + ) + return ContextPackage.create( + package_id=package_id, + title=title or f"Context package for {query}", + intent=intent or "Reactivate indexed Markdown context.", + namespace=namespace, + items=items, + retrieval_recipes=[recipe], + budget=budget, + policy=_policy_envelope(policy_result, policy_gateway, subject, action), + provenance=[_provenance("memory.pack.index", {"query": query, "engine": recipe.engine})], + ) + + +def create_context_package_from_manifest( + manifest_path: str | Path, + *, + root: str | Path = ".", + budget: ContextBudget | None = None, + policy_gateway: LocalLabelPolicyGateway | None = None, + subject: str = "anonymous", + action: str = "package", +) -> ContextPackage: + """Create a context package from a YAML/JSON manifest file.""" + + path = Path(manifest_path) + data = _load_mapping_file(path) + manifest_root = Path(root) + if data.get("root"): + manifest_root = _resolve_manifest_path(path.parent, str(data["root"])) + return create_context_package_from_manifest_mapping( + data, + root=manifest_root, + budget=budget, + policy_gateway=policy_gateway, + subject=subject, + action=action, + ) + + +def create_context_package_from_manifest_mapping( + data: dict[str, Any], + *, + root: str | Path = ".", + budget: ContextBudget | None = None, + policy_gateway: LocalLabelPolicyGateway | None = None, + subject: str = "anonymous", + action: str = "package", + package_policy: dict[str, Any] | None = None, +) -> ContextPackage: + """Create a context package from a manifest mapping.""" + + root_path = Path(root).resolve() + recipes = [ + RetrievalRecipe.from_mapping(recipe) + for recipe in data.get("retrieval_recipes", data.get("recipes", [])) + ] + if not recipes: + query = data.get("query") or data.get("selector") or data.get("search") + if not query: + raise ContextPackageError("Context manifest needs `recipes`, `query`, `selector`, or `search`.") + recipes = [ + RetrievalRecipe( + kind="search" if data.get("search") else "selector", + query=str(query), + engine=str(data.get("engine") or ("fts" if data.get("search") else "selector")), + sources=_string_list(data.get("sources") or data.get("source")), + paths=_string_list(data.get("paths") or data.get("path")), + root=str(root_path), + index_path=_optional_str(data.get("index_path")), + limit=_optional_int(data.get("limit")), + ) + ] + items: list[ContextPackageItem] = [] + for recipe in recipes: + effective_root = Path(recipe.root).resolve() if recipe.root != "." else root_path + if recipe.sources: + package = create_context_package_from_sources( + recipe.query, + [_resolve_manifest_path(effective_root, source) for source in recipe.sources], + root=effective_root, + engine=recipe.engine, + budget=ContextBudget(), + policy_gateway=None, + ) + else: + package = create_context_package_from_index( + recipe.query, + root=effective_root, + index_path=recipe.index_path, + engine="selector" if recipe.engine == "fts" else recipe.engine, + paths=recipe.paths, + search=recipe.kind == "search" or recipe.engine == "fts", + limit=recipe.limit or 20, + budget=ContextBudget(), + policy_gateway=None, + ) + items.extend(package.items) + policy_result = None + if policy_gateway: + filtered = policy_gateway.filter_results(subject, action, [item.to_dict() for item in items]) + items = [ContextPackageItem.from_mapping(item) for item in filtered["results"]] + policy_result = filtered + package_budget = budget or ContextBudget.from_mapping(data.get("budget")) + return ContextPackage.create( + package_id=_optional_str(data.get("id")), + title=str(data.get("title") or "Context package"), + intent=str(data.get("intent") or "Reactivate manifest-selected context."), + namespace=MemoryNamespace.from_mapping(data.get("namespace")), + items=items, + retrieval_recipes=recipes, + budget=package_budget, + policy=package_policy or _policy_envelope(policy_result, policy_gateway, subject, action), + provenance=[_provenance("memory.pack.manifest", {"recipes": len(recipes)})], + metadata=dict(data.get("metadata") or {}), + ) + + +def activate_context_package( + package: ContextPackage, + *, + target: str = "default", + policy_gateway: LocalLabelPolicyGateway | None = None, + subject: str = "anonymous", + action: str = "read", + budget: ContextBudget | None = None, +) -> ContextActivation: + """Activate a package into a Markdown context bundle.""" + + items = _apply_budget(package.items, budget or package.budget) + policy_result = None + if policy_gateway: + filtered = policy_gateway.filter_results(subject, action, [item.to_dict() for item in items]) + items = [ContextPackageItem.from_mapping(item) for item in filtered["results"]] + policy_result = filtered + summaries = build_summary_layers(items) + content = render_activation_markdown(package, items, summaries=summaries) + payload = { + "package_id": package.id, + "target": target, + "items": [item.id for item in items], + "content_hash": _hash_text(content), + } + return ContextActivation( + id="memory:activation:" + _stable_hash(payload), + package_id=package.id, + status="active", + target=target, + content=content, + items=items, + summaries=summaries, + token_estimate=estimate_tokens(content), + policy=_policy_envelope(policy_result, policy_gateway, subject, action), + policy_decisions=(policy_result or {}).get("decisions", []), + diagnostics=(policy_result or {}).get("diagnostics", []), + metadata={"package_title": package.title}, + ) + + +def deactivate_context_package(activation: ContextActivation) -> ContextActivation: + """Return a deactivated activation envelope.""" + + return ContextActivation( + id=activation.id, + package_id=activation.package_id, + status="inactive", + target=activation.target, + content="", + items=[], + summaries=activation.summaries, + token_estimate=0, + created_at=activation.created_at, + policy=activation.policy, + policy_decisions=activation.policy_decisions, + diagnostics=activation.diagnostics, + metadata=activation.metadata | {"deactivated_at": _now()}, + ) + + +def refresh_context_package( + package: ContextPackage, + *, + policy_gateway: LocalLabelPolicyGateway | None = None, + subject: str = "anonymous", + action: str = "package", +) -> ContextPackage: + """Refresh a package by re-running its retrieval recipes.""" + + if not package.retrieval_recipes: + raise ContextPackageError(f"Context package `{package.id}` has no retrieval recipes.") + data = { + "id": package.id, + "title": package.title, + "intent": package.intent, + "namespace": package.namespace.to_dict(), + "budget": package.budget.to_dict(), + "retrieval_recipes": [recipe.to_dict() for recipe in package.retrieval_recipes], + "metadata": package.metadata | {"refreshed_from": package.updated_at}, + } + refreshed = create_context_package_from_manifest_mapping( + data, + root=".", + policy_gateway=policy_gateway, + subject=subject, + action=action, + ) + return ContextPackage( + id=package.id, + title=refreshed.title, + intent=refreshed.intent, + namespace=refreshed.namespace, + items=refreshed.items, + retrieval_recipes=refreshed.retrieval_recipes, + summaries=refreshed.summaries, + budget=refreshed.budget, + activation_state="inactive", + created_at=package.created_at, + updated_at=_now(), + freshness=refreshed.freshness, + policy=refreshed.policy, + provenance=package.provenance + [_provenance("memory.refresh", {"previous_updated_at": package.updated_at})], + metadata=refreshed.metadata, + ) + + +def explain_context_package(package: ContextPackage) -> dict[str, Any]: + """Return provenance, budget, policy, namespace, and retrieval details.""" + + return { + "id": package.id, + "title": package.title, + "intent": package.intent, + "namespace": package.namespace.to_dict(), + "activation_state": package.activation_state, + "items": len(package.items), + "token_estimate": package.token_estimate, + "budget": package.budget.to_dict(), + "freshness": package.freshness, + "retrieval_recipes": [recipe.to_dict() for recipe in package.retrieval_recipes], + "sources": [item.source.to_dict() for item in package.items], + "summaries": [summary.to_dict() for summary in package.summaries], + "policy": package.policy, + "provenance": package.provenance, + "metadata": package.metadata, + } + + +def render_activation_markdown( + package: ContextPackage, + items: list[ContextPackageItem], + *, + summaries: list[SummaryLayer] | None = None, +) -> str: + """Render an activation bundle as compact Markdown.""" + + lines = [ + f"# {package.title}", + "", + f"Package: `{package.id}`", + ] + if package.intent: + lines.extend(["", package.intent]) + active_summaries = summaries if summaries is not None else package.summaries + if active_summaries: + lines.extend(["", "## Summary"]) + for summary in active_summaries: + lines.append(f"- {summary.text}") + lines.extend(["", "## Context Items"]) + for index, item in enumerate(items, start=1): + source = item.source + line = f"{source.path}" + if source.line_start: + line += f":{source.line_start}" + if source.line_end and source.line_end != source.line_start: + line += f"-{source.line_end}" + lines.extend( + [ + "", + f"### {index}. {line}", + "", + item.text.strip(), + ] + ) + return "\n".join(lines).rstrip() + "\n" + + +def load_context_package_file(path: str | Path) -> ContextPackage: + """Load a context package from YAML or JSON.""" + + data = _load_mapping_file(Path(path)) + return ContextPackage.from_mapping(data) + + +def build_summary_layers(items: list[ContextPackageItem]) -> list[SummaryLayer]: + """Build deterministic package-level summaries.""" + + if not items: + return [] + source_count = len({item.source.path for item in items}) + overview = f"{len(items)} item(s) from {source_count} source(s), estimated {sum(item.token_estimate for item in items)} tokens." + highlights = " ".join(item.summary for item in items[:5] if item.summary) + summaries = [ + SummaryLayer( + name="overview", + kind="deterministic-counts", + text=overview, + token_estimate=estimate_tokens(overview), + ) + ] + if highlights: + summaries.append( + SummaryLayer( + name="highlights", + kind="extractive", + text=highlights, + token_estimate=estimate_tokens(highlights), + ) + ) + return summaries + + +def summarize_text(text: str, *, max_chars: int = 220) -> str: + """Return a compact deterministic summary from the first meaningful text.""" + + compact = " ".join(text.split()) + if len(compact) <= max_chars: + return compact + return compact[: max_chars - 1].rstrip() + "..." + + +def estimate_tokens(text: str) -> int: + """Cheap token estimate tuned for budget planning rather than billing.""" + + compact = " ".join(str(text).split()) + if not compact: + return 0 + return max(1, int(len(compact) / 4) + 1) + + +def _items_from_index_query( + store: LocalSnapshotStore, + query: str, + *, + engine: str, + paths: list[str] | None, +) -> list[ContextPackageItem]: + states = store.load_state() + snapshot_ids = {state.path: state.snapshot_id for state in states} + indexed_paths = sorted(paths or snapshot_ids) + items: list[ContextPackageItem] = [] + for indexed_path in indexed_paths: + try: + document = Document.from_dict(store.get_document(indexed_path)) + policy = store.policy_metadata(indexed_path) + except KeyError as exc: + raise ContextPackageError(str(exc)) from exc + try: + matches = _query(document, query, engine) + except InvalidQueryError as exc: + raise ContextPackageError(str(exc)) from exc + snapshot_id = snapshot_ids.get(indexed_path) + for match in matches: + items.append( + _item_from_query_match( + match, + source_path=indexed_path, + selector=query, + engine=engine, + policy=policy, + snapshot_id=snapshot_id, + ) + ) + return items + + +def _items_from_search( + store: LocalSnapshotStore, + query: str, + *, + limit: int, +) -> list[ContextPackageItem]: + try: + results = store.search(query, limit=limit) + except ValueError as exc: + raise ContextPackageError(str(exc)) from exc + items: list[ContextPackageItem] = [] + for result in results: + items.append( + ContextPackageItem.create( + source=SourceSpan( + path=result.path, + snapshot_id=result.snapshot_id, + unit_kind=result.unit_kind, + unit_index=result.unit_index, + line_start=result.line_start, + line_end=result.line_end, + selector=query, + engine="fts", + ), + text=result.text, + score=result.rank, + policy=store.policy_metadata(result.path), + provenance={ + "operation": "memory.pack.search", + "backend": "local-sqlite", + "snapshot_id": result.snapshot_id, + }, + metadata={"heading": result.heading}, + ) + ) + return items + + +def _item_from_query_match( + match: QueryMatch, + *, + source_path: str, + selector: str, + engine: str, + policy: dict[str, Any], + snapshot_id: str | None, +) -> ContextPackageItem: + text = match.text or _text_from_value(match.value) + line_end = _line_end_from_value(match.value) or match.line + return ContextPackageItem.create( + source=SourceSpan( + path=source_path, + snapshot_id=snapshot_id, + unit_kind=match.kind, + unit_index=_index_from_path(match.path), + line_start=match.line, + line_end=line_end, + selector=selector, + engine=engine, + ), + text=text, + policy=policy, + provenance={ + "operation": "memory.pack.query", + "engine": engine, + "selector": selector, + "snapshot_id": snapshot_id, + }, + metadata={"query_path": match.path, "kind": match.kind}, + ) + + +def _query(document: Document, query: str, engine: str) -> list[QueryMatch]: + if engine == "jsonpath": + return query_document_jsonpath(document, query) + if engine != "selector": + raise ContextPackageError(f"Unsupported context query engine `{engine}`.") + return query_document(document, query) + + +def _apply_budget(items: list[ContextPackageItem], budget: ContextBudget) -> list[ContextPackageItem]: + max_items = budget.max_items + max_tokens = None + if budget.max_tokens is not None: + max_tokens = max(0, budget.max_tokens - max(0, budget.reserve_tokens)) + selected: list[ContextPackageItem] = [] + total = 0 + for item in items: + if max_items is not None and len(selected) >= max_items: + break + if max_tokens is not None and selected and total + item.token_estimate > max_tokens: + break + if max_tokens is not None and not selected and item.token_estimate > max_tokens: + continue + selected.append(item) + total += item.token_estimate + return selected + + +def _policy_envelope( + policy_result: dict[str, Any] | None, + gateway: LocalLabelPolicyGateway | None, + subject: str, + action: str, +) -> dict[str, Any]: + if policy_result: + return { + "gateway": getattr(gateway, "gateway_id", None), + "subject": subject, + "action": action, + "summary": policy_result.get("policy"), + "decisions": policy_result.get("decisions"), + "diagnostics": policy_result.get("diagnostics"), + } + if gateway: + return {"gateway": getattr(gateway, "gateway_id", None), "subject": subject, "action": action} + return {} + + +def _freshness(items: list[ContextPackageItem]) -> dict[str, Any]: + return { + "created_at": _now(), + "item_count": len(items), + "source_count": len({item.source.path for item in items}), + "content_hash": _stable_hash([item.id for item in items]), + } + + +def _provenance(operation: str, metadata: dict[str, Any] | None = None) -> dict[str, Any]: + return { + "operation": operation, + "created_at": _now(), + "metadata": metadata or {}, + } + + +def _text_from_value(value: Any) -> str: + if isinstance(value, str): + return value + if isinstance(value, dict): + if "blocks" in value and isinstance(value["blocks"], list): + heading = value.get("heading", {}) + heading_text = "" + if isinstance(heading, dict) and heading.get("text"): + heading_text = f"{'#' * int(heading.get('level') or 2)} {heading['text']}\n\n" + return heading_text + "\n\n".join(str(block.get("text", "")) for block in value["blocks"]) + return json.dumps(value, sort_keys=True, ensure_ascii=False) + return "" if value is None else str(value) + + +def _line_end_from_value(value: Any) -> int | None: + if not isinstance(value, dict): + return None + blocks = value.get("blocks") + if isinstance(blocks, list): + ends = [block.get("line_end") for block in blocks if isinstance(block, dict) and block.get("line_end")] + return max(ends) if ends else None + return value.get("line_end") + + +def _index_from_path(path: str) -> int | None: + if "[" not in path or "]" not in path: + return None + try: + return int(path.rsplit("[", 1)[1].split("]", 1)[0]) + except ValueError: + return None + + +def _load_mapping_file(path: Path) -> dict[str, Any]: + try: + data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + except OSError as exc: + raise ContextPackageError(str(exc)) from exc + if not isinstance(data, dict): + raise ContextPackageError(f"Context package file `{path}` must contain a mapping.") + return data + + +def _resolve_manifest_path(base: Path, path: str) -> Path: + candidate = Path(path) + if candidate.is_absolute(): + return candidate + return base / candidate + + +def _relative(path: Path, root: Path) -> str: + resolved = path.resolve() + try: + return resolved.relative_to(root).as_posix() + except ValueError: + return resolved.as_posix() + + +def _stable_hash(value: Any) -> str: + return hashlib.sha256( + json.dumps(value, sort_keys=True, ensure_ascii=False, default=str).encode("utf-8") + ).hexdigest() + + +def _hash_text(text: str) -> str: + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def _optional_str(value: Any) -> str | None: + if value is None: + return None + text = str(value) + return text if text else None + + +def _optional_int(value: Any) -> int | None: + if value is None or value == "": + return None + return int(value) + + +def _string_list(value: Any) -> list[str]: + if value is None: + return [] + if isinstance(value, list): + return [str(item) for item in value if item is not None] + return [str(value)] + + +def _drop_empty(data: dict[str, Any]) -> dict[str, Any]: + return { + key: value + for key, value in data.items() + if value not in (None, [], {}, "") + } diff --git a/tests/test_agent_working_memory.py b/tests/test_agent_working_memory.py new file mode 100644 index 0000000..8743c93 --- /dev/null +++ b/tests/test_agent_working_memory.py @@ -0,0 +1,172 @@ +from pathlib import Path + +import yaml +from click.testing import CliRunner + +from markitect_tool.cli import main +from markitect_tool.memory import ( + ContextBudget, + LocalContextPackageRegistry, + MemoryNamespace, + activate_context_package, + create_context_package_from_index, + create_context_package_from_manifest, + create_context_package_from_sources, + refresh_context_package, +) +from markitect_tool.policy import LocalLabelPolicyGateway + + +def test_create_context_package_from_sources_preserves_spans_and_summary(tmp_path: Path): + source = tmp_path / "doc.md" + source.write_text("# Doc\n\n## Decision\n\nUse local memory packages.\n", encoding="utf-8") + + package = create_context_package_from_sources( + "sections[heading=Decision]", + [source], + root=tmp_path, + namespace=MemoryNamespace(project="markitect-tool", task="MKTT-WP-0008"), + budget=ContextBudget(max_items=1), + ) + + assert package.id.startswith("memory:package:") + assert package.namespace.project == "markitect-tool" + assert package.items[0].source.path == "doc.md" + assert package.items[0].source.unit_kind == "section" + assert "local memory packages" in package.items[0].text + assert package.summaries[0].name == "overview" + + +def test_local_context_registry_saves_loads_and_activates(tmp_path: Path): + source = tmp_path / "doc.md" + source.write_text("# Doc\n\n## Context\n\nReusable project facts.\n", encoding="utf-8") + package = create_context_package_from_sources("sections[heading=Context]", [source], root=tmp_path) + registry = LocalContextPackageRegistry(tmp_path) + + saved_path = registry.save(package) + loaded = registry.load(package.id) + activation_id = registry.activate(package.id, "thread:test") + activation = registry.load_activation(activation_id) + + assert saved_path.exists() + assert loaded.id == package.id + assert activation.status == "active" + assert "Reusable project facts" in activation.content + + +def test_activation_can_recheck_policy_and_drop_denied_items(tmp_path: Path): + public = tmp_path / "public.md" + private = tmp_path / "private.md" + public.write_text("---\npolicy:\n labels: [public]\n---\n# Public\n\nVisible.\n", encoding="utf-8") + private.write_text("---\npolicy:\n labels: [internal]\n---\n# Private\n\nHidden.\n", encoding="utf-8") + package = create_context_package_from_sources("document", [public, private], root=tmp_path) + gateway = LocalLabelPolicyGateway( + { + "id": "memory-test-policy", + "subjects": { + "reader": { + "allowed_labels": ["public"], + "allowed_actions": ["read"], + } + }, + "default_subject": "reader", + } + ) + + activation = activate_context_package(package, policy_gateway=gateway, subject="reader") + + assert len(activation.items) == 1 + assert "Visible" in activation.content + assert "Hidden" not in activation.content + assert activation.policy["summary"]["denied"] == 1 + + +def test_create_context_package_from_index_search_and_refresh(tmp_path: Path): + source = tmp_path / "doc.md" + source.write_text("# Doc\n\nSearchable memory fact.\n", encoding="utf-8") + runner = CliRunner() + indexed = runner.invoke(main, ["cache", "index", str(tmp_path), "--root", str(tmp_path)]) + assert indexed.exit_code == 0 + + package = create_context_package_from_index("Searchable", root=tmp_path, search=True) + source.write_text("# Doc\n\nSearchable memory fact changed.\n", encoding="utf-8") + reindexed = runner.invoke(main, ["cache", "index", str(tmp_path), "--root", str(tmp_path)]) + refreshed = refresh_context_package(package) + + assert reindexed.exit_code == 0 + assert package.items + assert "changed" in refreshed.items[0].text + assert refreshed.id == package.id + + +def test_create_context_package_from_manifest(tmp_path: Path): + source = tmp_path / "doc.md" + source.write_text("# Doc\n\n## Purpose\n\nManifest selected context.\n", encoding="utf-8") + manifest = tmp_path / "context.yaml" + manifest.write_text( + yaml.safe_dump( + { + "title": "Manifest package", + "namespace": {"project": "markitect-tool"}, + "retrieval_recipes": [ + { + "kind": "selector", + "query": "sections[heading=Purpose]", + "sources": ["doc.md"], + } + ], + }, + sort_keys=False, + ), + encoding="utf-8", + ) + + package = create_context_package_from_manifest(manifest, root=tmp_path) + + assert package.title == "Manifest package" + assert package.namespace.project == "markitect-tool" + assert "Manifest selected context" in package.items[0].text + + +def test_mkt_context_pack_activate_explain_and_deactivate(tmp_path: Path): + source = tmp_path / "doc.md" + source.write_text("# Doc\n\n## Decision\n\nUse explicit context packages.\n", encoding="utf-8") + runner = CliRunner() + + packed = runner.invoke( + main, + [ + "context", + "pack", + "sections[heading=Decision]", + "--source", + str(source), + "--root", + str(tmp_path), + "--format", + "json", + ], + ) + assert packed.exit_code == 0 + package_id = yaml.safe_load(packed.output)["id"] + + explained = runner.invoke(main, ["context", "explain", package_id, "--root", str(tmp_path)]) + activated = runner.invoke( + main, + [ + "context", + "activate", + package_id, + "--root", + str(tmp_path), + "--format", + "json", + ], + ) + activation_id = yaml.safe_load(activated.output)["id"] + deactivated = runner.invoke(main, ["context", "deactivate", activation_id, "--root", str(tmp_path)]) + + assert explained.exit_code == 0 + assert "Use explicit context packages" in activated.output + assert deactivated.exit_code == 0 + assert "inactive" in deactivated.output diff --git a/tests/test_builtin_extension_catalog.py b/tests/test_builtin_extension_catalog.py index 13d63aa..e88f440 100644 --- a/tests/test_builtin_extension_catalog.py +++ b/tests/test_builtin_extension_catalog.py @@ -19,6 +19,7 @@ def test_builtin_extension_registry_lists_query_processors_and_backend(): assert "runtime.assessment" in ids assert "policy.local-label" in ids assert "document.function" in ids + assert "memory.context-package" in ids def test_builtin_processor_descriptors_capture_safety_and_provenance(): @@ -123,3 +124,21 @@ def test_builtin_document_function_descriptor_exposes_deterministic_boundary(): "mkt function check", "mkt function render", ] + + +def test_builtin_memory_descriptor_exposes_local_optional_boundary(): + registry = builtin_extension_registry() + + descriptor = registry.get("memory.context-package") + + assert descriptor.kind == "memory-registry" + assert descriptor.safety["network"] is False + assert descriptor.safety["writes_local_context_registry"] is True + assert descriptor.metadata["external_policy_services_required"] is False + assert {capability.id for capability in descriptor.capabilities} >= { + "context_packages", + "context_activation", + "policy_filter", + } + assert "mkt context pack" in descriptor.cli["commands"] + assert "mkt context activate" in descriptor.cli["commands"] diff --git a/tests/test_cli_extension_specs.py b/tests/test_cli_extension_specs.py index 9db76dc..92e0da5 100644 --- a/tests/test_cli_extension_specs.py +++ b/tests/test_cli_extension_specs.py @@ -19,6 +19,7 @@ def test_collect_cli_command_specs_from_builtin_registry(): assert ("backend.local-sqlite", "mkt cache index") in commands assert ("backend.local-sqlite", "mkt search") in commands assert ("document.function", "mkt function render") in commands + assert ("memory.context-package", "mkt context pack") in commands def test_cli_command_spec_serializes_without_empty_fields(): diff --git a/workplans/MKTT-WP-0008-agent-working-memory-context-cache.md b/workplans/MKTT-WP-0008-agent-working-memory-context-cache.md index 3998f17..7a6227f 100644 --- a/workplans/MKTT-WP-0008-agent-working-memory-context-cache.md +++ b/workplans/MKTT-WP-0008-agent-working-memory-context-cache.md @@ -3,17 +3,17 @@ id: MKTT-WP-0008 type: workplan title: "Agent Working Memory Context Cache" domain: markitect -status: todo +status: done owner: markitect-tool topic_slug: markitect -planning_priority: P3 +planning_priority: complete planning_order: 90 depends_on_workplans: - MKTT-WP-0006 - MKTT-WP-0007 - MKTT-WP-0009 created: "2026-05-03" -updated: "2026-05-03" +updated: "2026-05-04" state_hub_workstream_id: "6269f338-4f5c-40ee-90e5-0371f5c3874c" --- @@ -25,6 +25,54 @@ Create activatable context packages that let agents drop, reactivate, and reuse project knowledge efficiently while preserving provenance and policy metadata. +## Thought Experiment And Design Refinement + +The implementation was preceded by a wishful architecture pass documented in +`docs/agent-working-memory-thought-experiment.md`. + +The useful model is layered memory: + +- reflex context for the current response +- short working set for the active task +- episodic memory for a thread or work session +- project semantic memory for architecture and source knowledge +- identity-like continuity memory for durable principles and preferences +- policy/safety memory that must be rechecked before activation + +The key refinement is that Markitect should implement the local, explicit, +inspectable package layer rather than hidden ambient agent memory. Long-term +identity and learning memory can be represented as reviewed namespace-scoped +packages later, but writes should remain visible and reversible. + +This shaped the implementation: + +- package creation and activation are explicit CLI/library operations +- every item carries source spans, summaries, token estimates, provenance, and + policy metadata +- deterministic summaries come first +- package creation and activation can both run local policy checks +- local files and the local SQLite index are enough for the first version +- LLM summaries, embeddings, remote registries, retention/decay, and durable + identity packages remain future optional extensions + +## Implementation Summary + +Implemented the first local agent working-memory context cache: + +- `markitect_tool.memory` extension package with context package schema, + retrieval recipes, budgets, namespaces, summary layers, activation envelopes, + and filesystem-backed local registry. +- Package creation from direct Markdown source queries, local index selector or + JSONPath queries, FTS search results, and YAML manifests. +- Deterministic summaries and approximate token estimates. +- Activation, deactivation, refresh, explain, list, and save/load lifecycle. +- Optional local policy filtering at package creation and activation. +- CLI commands under `mkt context`. +- Built-in extension descriptor `memory.context-package`. +- Documentation and example manifest in `docs/agent-working-memory.md`, + `docs/agent-working-memory-thought-experiment.md`, and + `examples/memory/workplan-context.manifest.yaml`. + ## Architectural Boundary This workplan depends only on Markitect-local backend and policy contracts: @@ -48,7 +96,7 @@ freshness metadata. These references are metadata, not hard dependencies. ```task id: MKTT-WP-0008-T001 -status: todo +status: done priority: high state_hub_task_id: "21ee9c37-4add-4886-bd03-a7bb4b20e957" ``` @@ -68,11 +116,16 @@ The schema should include optional policy metadata: These fields must support local policy gateways first and external policy services only through optional adapters. +Implemented: `ContextPackage`, `ContextPackageItem`, `SourceSpan`, +`RetrievalRecipe`, `SummaryLayer`, `ContextBudget`, and `MemoryNamespace` +define the local schema. Package items preserve source spans, summaries, token +estimates, provenance, freshness, and policy metadata. + ## P8.2 - Implement package creation from queries ```task id: MKTT-WP-0008-T002 -status: todo +status: done priority: high state_hub_task_id: "4df06b93-13ce-41fb-a8c3-f04d4ad9d752" ``` @@ -84,11 +137,16 @@ Package creation should use current query/search APIs and policy-aware result filtering. It should not call flex-auth directly; future flex-auth-backed filtering can be injected through the existing policy gateway boundary. +Implemented: packages can be created from direct Markdown source queries, local +indexed selector/JSONPath queries, FTS search results, and YAML manifests. +Optional local policy gateways filter package items using the existing +`LocalLabelPolicyGateway` boundary. + ## P8.3 - Implement activation lifecycle ```task id: MKTT-WP-0008-T003 -status: todo +status: done priority: medium state_hub_task_id: "9f3d9792-d655-482d-bae0-262df5fc0136" ``` @@ -99,11 +157,17 @@ Activation should re-check local policy metadata when a policy gateway is available. In the absence of an external service, activation remains fully local and explainable. +Implemented: `activate_context_package`, `deactivate_context_package`, +`refresh_context_package`, `explain_context_package`, and +`LocalContextPackageRegistry` support the lifecycle locally. Activation +rebuilds summaries after policy filtering so denied content does not leak +through package-level summaries. + ## P8.4 - Add memory namespaces ```task id: MKTT-WP-0008-T004 -status: todo +status: done priority: medium state_hub_task_id: "2d090494-0e10-44cd-8e2d-c418d7530b27" ``` @@ -115,11 +179,14 @@ Namespace design should leave room for enterprise subject ids and external resource ids, but must not require any particular SSO, IAM, or authorization provider. +Implemented: `MemoryNamespace` supports project, user, agent, thread, task, +and custom namespace fields without depending on any external agent platform. + ## P8.5 - Add summary layers ```task id: MKTT-WP-0008-T005 -status: todo +status: done priority: medium state_hub_task_id: "4d1cf970-3d6d-4bd5-8da9-ec2399aa7efe" ``` @@ -130,11 +197,15 @@ through an injected adapter. Assisted summaries must be optional and policy/capability-gated before any prompt assembly happens. +Implemented: deterministic extractive/count summaries are included. Assisted +summaries remain a documented future adapter path and are not invoked by core +package creation or activation. + ## P8.6 - Add CLI commands ```task id: MKTT-WP-0008-T006 -status: todo +status: done priority: medium state_hub_task_id: "2f18386c-9d2c-4af1-b8e2-75cb487c1692" ``` @@ -152,6 +223,21 @@ CLI commands should work against local packages without flex-auth. Optional policy flags may accept local policy files or later external adapter configuration. +Implemented: + +```text +mkt context pack +mkt context activate +mkt context deactivate +mkt context explain +mkt context refresh +mkt context list +``` + +Commands work with local package files and the `.markitect/context` registry. +Policy flags use local label policy only; external policy systems remain +optional adapters. + ## Exit Criteria - Agents can reactivate project context by stable id.