coulomb/markitect-tool
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Layered context memory revision 0

Browse Source
This commit is contained in:
Bernd Worsch
2026-05-04 20:33:20 +02:00
parent f940aa4b21
commit 46c1d804fc
13 changed files with 2448 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
docs/agent-working-memory-thought-experiment.md Normal file
View File

@@ -0,0 +1,104 @@
# Agent Working Memory Thought Experiment
Date: 2026-05-04
## Prompt
Imagine an agent memory architecture optimized for short and mid term
efficiency across different reaction speeds, timescales, and cost constraints,
while preserving a personality-like long term identity and learning memory that
evolves over time.
This is a wishful design note. It is not a promise that every layer belongs in
`markitect-tool` core.
## Memory Layers
| Layer | Reaction Speed | Lifetime | Cost Shape | Useful Contents |
| --- | --- | --- | --- | --- |
| Reflex context | milliseconds to seconds | one response | highest compute pressure | current user turn, active file, tool output, immediate errors |
| Working set | seconds to minutes | one task step | medium compute pressure | current plan, nearby code/docs, selected snippets, constraints |
| Episode memory | minutes to hours | one chat/thread/work session | moderate storage | decisions, explored dead ends, task state, user preferences for this effort |
| Project semantic memory | hours to weeks | project lifecycle | cheap local storage, occasional refresh | architecture docs, stable APIs, workplans, policies, source maps, examples |
| Identity and learning memory | weeks to years | agent/persona/project relationship | expensive to write, cheap to retrieve | values, style, durable preferences, repeated collaboration patterns |
| Policy and safety memory | all speeds | explicit freshness windows | must be rechecked before use | labels, trust zones, subject identity, decision ids, authorization context |
The main design pressure is not just retrieval quality. It is deciding when a
memory should be cheap, fast, provisional, discardable, re-checkable, or
durable.
## Wish List
If a few things could be manifested instantly:
1. A hot context router would keep the current working set small and sharp. It
would load exact spans and summaries only when they are needed, then release
them when the task moves on.
2. Every context item would carry provenance, policy metadata, freshness,
token cost, and a reason it was included.
3. Memory would be layered, not monolithic. Fast scratch memory, package memory,
project memory, and identity memory would have different write thresholds.
4. Durable memory writes would be explicit and inspectable. The agent could
propose "this seems worth remembering" rather than silently mutating itself.
5. Long term identity would be represented as small, versioned principles and
preferences, not as a bag of transcripts.
6. The system would support both activation and forgetting. Dropping context
should be as first-class as loading it.
7. Retrieval would be policy-aware at both package creation and activation.
A package created yesterday should not be blindly activated today.
8. Summaries would be multi-resolution: one-line, section-level, package-level,
project-level, and identity-level, each with its own freshness and source
links.
9. Local deterministic memory would work offline. LLM-assisted summaries,
embeddings, external stores, and enterprise authorization would be optional
adapters.
10. The agent could maintain a small "self continuity" layer: tone,
collaboration norms, current mission, and durable lessons, all with user
review and rollback.
## Dynamic Balances
The useful architecture has several moving balances:
- Fast recall vs. exact provenance: the reflex layer can be approximate, but
activation packages must be explainable.
- Rich context vs. token budget: every item needs a token estimate and a
summary so a caller can choose the right resolution.
- Stability vs. freshness: project architecture memory is useful only if it
can be refreshed against current snapshots.
- Personalization vs. consent: long term identity memory should require higher
write friction than task-local memory.
- Local autonomy vs. enterprise control: local package creation must work
offline, while activation can re-check policy when a gateway is available.
- Retrieval breadth vs. actionability: packages should be small enough to
activate directly and large enough to explain why they matter.
## Implications For Markitect
Markitect should own the local, inspectable package layer:
- schema for context packages
- package creation from selectors, search, and manifests
- deterministic summary layers
- activation/deactivation/refresh/explain lifecycle
- source spans, token estimates, provenance, and policy metadata
- optional adapter boundaries for assisted summaries, embeddings, vector
stores, and external policy decision points
Markitect should not own hidden agent identity memory as an ambient service.
It can provide explicit package formats and namespaces for identity-like
memory, but durable writes should remain visible, reviewable, and policy-bound.
## Best First Step
The first implementation should be boring in the right way:
- local files and local SQLite index only
- deterministic summaries only
- explicit package files in `.markitect/context`
- policy metadata preserved and optionally rechecked
- no live flex-auth, vector database, embedding provider, LLM provider, or
hidden agent state
That gives future systems a trustworthy memory substrate without making the
core repo pretend to be a full cognitive architecture.

169
docs/agent-working-memory.md Normal file
View File

@@ -0,0 +1,169 @@
# Agent Working Memory Context Cache
Date: 2026-05-04
## Purpose
Agent working memory packages are explicit, portable context bundles. They let
an agent or user gather relevant Markdown knowledge, drop it when it is no
longer needed, and reactivate it later by stable id.
The implementation is local-first:
- no live flex-auth requirement
- no network access
- no embedding or LLM provider requirement
- no hidden memory writes
- deterministic summaries first
- optional local policy checks at package creation and activation
## Core Objects
| Object | Role |
| --- | --- |
| `MemoryNamespace` | Project, user, agent, thread, task, and custom namespace coordinates. |
| `SourceSpan` | Path, snapshot id, unit kind/index, line span, selector, and engine. |
| `ContextPackageItem` | One included memory item with text, summary, token estimate, policy, provenance, and metadata. |
| `RetrievalRecipe` | Refreshable query/search recipe. |
| `SummaryLayer` | Deterministic package summary layer. |
| `ContextBudget` | Approximate max items/tokens and reserve. |
| `ContextPackage` | Portable package envelope. |
| `ContextActivation` | Activation/deactivation envelope with Markdown content and policy results. |
| `LocalContextPackageRegistry` | Filesystem registry rooted at `.markitect/context`. |
## Use Cases
### Task Reactivation
Create a package from the current workplan or architecture sections, then
activate it in a later thread without rereading the whole repository.
### Focused Code Review Context
Pack only the sections, files, and examples relevant to a specific subsystem.
Use a small token budget so activation produces a compact review bundle.
### Project Onboarding
Create a package with intent, scope, architecture, and current workplan
summaries. A new agent can activate this instead of scanning every document.
### Policy-Aware Knowledge Work
Create and activate packages with local label policy. Package creation can
filter sensitive sources, and activation can re-check whether the current
subject may still read the items.
### Identity-Like Continuity
Represent durable collaboration norms or project principles as explicit
namespace-scoped context packages. These are not hidden personality mutations;
they are versioned and inspectable memory artifacts.
### Mid-Term Episodic Memory
Store decisions, explored alternatives, and task state for a thread or work
session. Keep it separate from long term project memory so it can expire or be
reviewed.
## CLI
Create from direct source files:
```text
mkt context pack "sections[heading=Decision]" --source docs/example.md
```
Create from the local index:
```text
mkt cache index docs --root .
mkt context pack "sections[heading~=Policy]" --path docs/access-control-policy-gateway.md
```
Create from FTS search:
```text
mkt context pack "policy gateway" --search --limit 5
```
Activate:
```text
mkt context activate memory:package:... --target thread:wp-0008
```
Explain:
```text
mkt context explain memory:package:...
```
Refresh:
```text
mkt context refresh memory:package:...
```
Deactivate:
```text
mkt context deactivate memory:activation:...
```
## Manifest Shape
```yaml
title: Workplan context
intent: Reactivate current workplan decisions.
namespace:
project: markitect-tool
task: MKTT-WP-0008
budget:
max_items: 4
max_tokens: 1200
retrieval_recipes:
- kind: selector
query: sections[heading=Purpose]
engine: selector
sources:
- workplans/MKTT-WP-0008-agent-working-memory-context-cache.md
```
## Policy
Policy metadata is copied from document frontmatter or the local index. When a
local policy gateway is supplied, Markitect filters package items by the
requested subject/action.
Package creation typically uses action `package`; activation defaults to
action `read`.
External policy services may be attached later through the existing policy
gateway boundary, but are not required.
## Architecture Fit
The memory layer is an internal extension over existing primitives:
- local snapshots and FTS from `LocalSnapshotStore`
- selectors and optional JSONPath from the query engines
- local labels and trust zones from `LocalLabelPolicyGateway`
- provenance, diagnostics, and capability metadata from the extension
framework
It is deliberately separate from the dataflow workflow engine. Workflows
orchestrate business processes; context packages capture reusable working
memory for agents.
## Future Extensions
Natural extensions are intentionally deferred:
- LLM-assisted summaries through an injected adapter
- vector retrieval and embedding caches
- decay/retention policies
- reviewed long term identity packages
- signed policy decisions and durable decision logs
- remote or enterprise memory registries
- package merge/diff and conflict diagnostics

12
docs/workplan-planning-map.md
View File

@@ -40,7 +40,7 @@ and descriptions mirror the operational view.
| `MKTT-WP-0009` | complete | done | `MKTT-WP-0006` | Access-controlled knowledge gateway is complete: local labels, trust zones, path rules, policy-aware cache query/search, decisions, diagnostics, and external adapter boundaries. |
| `MKTT-WP-0014` | complete | done | `MKTT-WP-0009` | Markitect-side enterprise IAM access-control integration is complete: NetKingdom/key-cape-compatible identity claims, flex-auth resource/policy contract, directory group resolution fixtures, decision-log sink, workflow declarations, CLI commands, and external PDP request examples. |
| `MKTT-WP-0012` | complete | done | `MKTT-WP-0004`, `MKTT-WP-0010`, `MKTT-WP-0011` | Document function layer is complete: deterministic Markdown-native function descriptors, registry, inline/fenced syntax, pipelines, context bindings, CLI, docs, examples, diagnostics, provenance, and extension descriptor. |
| `MKTT-WP-0008` | P3 | todo | `MKTT-WP-0006`, `MKTT-WP-0007`, `MKTT-WP-0009` | Agent working-memory cache after backend and policy floor are available. |
| `MKTT-WP-0008` | complete | done | `MKTT-WP-0006`, `MKTT-WP-0007`, `MKTT-WP-0009` | Agent working-memory context cache is complete: context package schema, local registry, package creation from queries/search/manifests, deterministic summaries, namespaces, activation/deactivation/refresh/explain lifecycle, policy re-checks, CLI, docs, and examples. |
| `MKTT-WP-0015` | P2 | todo | `MKTT-WP-0010`, `MKTT-WP-0011`, `MKTT-WP-0012` | Future render and document-function extensions: typed values, richer syntax, document-local reusable functions, Quarkdown/export adapters, render-aware references, assets, and permission sandboxing. Defer unless publishing/export pressure becomes current. |
## Dependency Notes
@@ -91,8 +91,8 @@ protocols. A live flex-auth service can improve enterprise deployment, central
policy administration, and durable audit, but it is not a prerequisite for the
document function layer or local agent context packages.
Remaining Markitect workplans, including `MKTT-WP-0008` and the future
`MKTT-WP-0015` extension track, should keep this policy posture:
Remaining Markitect workplans, including the future `MKTT-WP-0015` extension
track, should keep this policy posture:
- use `AccessPolicyGateway`, `PolicySubject`, `PolicyObject`, and
`PolicyDecision` as local contracts
@@ -103,6 +103,12 @@ Remaining Markitect workplans, including `MKTT-WP-0008` and the future
parsing, deterministic functions, workflows, cache queries, or context
package lifecycle
`MKTT-WP-0008` completed the local agent working-memory package layer. It
deliberately implements explicit, inspectable context packages rather than
hidden ambient agent memory. LLM-assisted summaries, embeddings, vector stores,
remote registries, retention/decay, and reviewed long-term identity packages
remain future optional extensions.
## State Hub Mirror
Native State Hub dependency edges should mirror the whole-workstream

19
examples/memory/workplan-context.manifest.yaml Normal file
View File

@@ -0,0 +1,19 @@
title: Agent working memory workplan context
intent: Reactivate the purpose and implementation scope for WP-0008.
namespace:
project: markitect-tool
task: MKTT-WP-0008
budget:
max_items: 4
max_tokens: 1200
retrieval_recipes:
- kind: selector
query: sections[heading=Purpose]
engine: selector
sources:
- workplans/MKTT-WP-0008-agent-working-memory-context-cache.md
- kind: selector
query: sections[heading=Architectural Boundary]
engine: selector
sources:
- workplans/MKTT-WP-0008-agent-working-memory-context-cache.md

38
src/markitect_tool/__init__.py
View File

@@ -114,6 +114,26 @@ from markitect_tool.literate import (
weave_markdown,
write_tangle_files,
)
from markitect_tool.memory import (
ContextActivation,
ContextBudget,
ContextPackage,
ContextPackageError,
ContextPackageItem,
LocalContextPackageRegistry,
MemoryNamespace,
RetrievalRecipe,
SourceSpan as MemorySourceSpan,
SummaryLayer,
activate_context_package,
create_context_package_from_index,
create_context_package_from_manifest,
create_context_package_from_sources,
deactivate_context_package,
explain_context_package,
load_context_package_file,
refresh_context_package,
)
from markitect_tool.ops import (
ComposeResult,
IncludeError,
@@ -314,6 +334,24 @@ __all__ = [
"tangle_markdown",
"weave_markdown",
"write_tangle_files",
"ContextActivation",
"ContextBudget",
"ContextPackage",
"ContextPackageError",
"ContextPackageItem",
"LocalContextPackageRegistry",
"MemoryNamespace",
"RetrievalRecipe",
"MemorySourceSpan",
"SummaryLayer",
"activate_context_package",
"create_context_package_from_index",
"create_context_package_from_manifest",
"create_context_package_from_sources",
"deactivate_context_package",
"explain_context_package",
"load_context_package_file",
"refresh_context_package",
"ComposeResult",
"IncludeError",
"IncludeResult",

475
src/markitect_tool/cli/main.py
View File

@@ -57,6 +57,18 @@ from markitect_tool.generation import (
run_generation_plan,
)
from markitect_tool.literate import tangle_markdown, weave_markdown, write_tangle_files
from markitect_tool.memory import (
ContextBudget,
ContextPackageError,
LocalContextPackageRegistry,
MemoryNamespace,
activate_context_package,
create_context_package_from_index,
create_context_package_from_manifest,
create_context_package_from_sources,
explain_context_package,
refresh_context_package,
)
from markitect_tool.ops import IncludeError, compose_files, resolve_includes, transform_markdown
from markitect_tool.processor import ProcessorContext, run_fenced_processors
from markitect_tool.policy import (
@@ -1392,6 +1404,373 @@ def search(
_emit_search_results(data, output_format)
@main.group("context")
def context_group() -> None:
"""Pack and activate agent working-memory context."""
@context_group.command("pack")
@click.argument("query_or_manifest")
@click.option(
"--source",
"sources",
multiple=True,
type=click.Path(exists=True, dir_okay=False, path_type=Path),
help="Markdown source file to query directly. May be repeated.",
)
@click.option(
"--root",
type=click.Path(exists=True, file_okay=False, path_type=Path),
default=Path("."),
show_default=True,
help="Root used for relative paths and the local context registry.",
)
@click.option(
"--index-path",
type=click.Path(dir_okay=False, path_type=Path),
help="SQLite index path. Defaults to .markitect/cache/index.sqlite3 under root.",
)
@click.option("--search", is_flag=True, help="Treat the argument as an FTS search query.")
@click.option("--path", "paths", multiple=True, help="Restrict indexed query/search to relative paths.")
@click.option(
"--engine",
type=click.Choice(["selector", "jsonpath"], case_sensitive=False),
default="selector",
show_default=True,
help="Query engine for selector packages.",
)
@click.option("--limit", type=int, default=20, show_default=True, help="Maximum FTS search matches.")
@click.option("--title", help="Package title.")
@click.option("--intent", help="Package intent statement.")
@click.option("--project", help="Project namespace.")
@click.option("--user", "user_ns", help="User namespace.")
@click.option("--agent", help="Agent namespace.")
@click.option("--thread", help="Thread namespace.")
@click.option("--task", help="Task namespace.")
@click.option("--namespace", "namespace_pairs", multiple=True, metavar="KEY=VALUE", help="Custom namespace key.")
@click.option("--max-items", type=int, help="Maximum context items to include.")
@click.option("--max-tokens", type=int, help="Approximate maximum package tokens.")
@click.option("--reserve-tokens", type=int, default=0, show_default=True, help="Token reserve inside max tokens.")
@click.option(
"--policy",
"policy_file",
type=click.Path(exists=True, dir_okay=False, path_type=Path),
help="Local label policy file used to filter package items.",
)
@click.option("--subject", default="anonymous", show_default=True, help="Policy subject id.")
@click.option(
"--policy-mode",
type=click.Choice(["off", "audit", "enforce"], case_sensitive=False),
help="Override policy mode while packing.",
)
@click.option("--action", default="package", show_default=True, help="Policy action used while packing.")
@click.option("--output", type=click.Path(dir_okay=False, path_type=Path), help="Write package to this file.")
@click.option("--no-save", is_flag=True, help="Do not save to the local context registry.")
@click.option(
"--format",
"output_format",
type=click.Choice(["json", "yaml", "text"], case_sensitive=False),
default="text",
show_default=True,
)
def context_pack(
query_or_manifest: str,
sources: tuple[Path, ...],
root: Path,
index_path: Path | None,
search: bool,
paths: tuple[str, ...],
engine: str,
limit: int,
title: str | None,
intent: str | None,
project: str | None,
user_ns: str | None,
agent: str | None,
thread: str | None,
task: str | None,
namespace_pairs: tuple[str, ...],
max_items: int | None,
max_tokens: int | None,
reserve_tokens: int,
policy_file: Path | None,
subject: str,
policy_mode: str | None,
action: str,
output: Path | None,
no_save: bool,
output_format: str,
) -> None:
"""Create an inspectable context package from a query, search, or manifest."""
try:
namespace = _memory_namespace(project, user_ns, agent, thread, task, namespace_pairs)
budget = ContextBudget(max_tokens=max_tokens, max_items=max_items, reserve_tokens=reserve_tokens)
gateway = _load_policy_gateway(policy_file, policy_mode)
manifest = Path(query_or_manifest)
if manifest.exists() and manifest.is_file() and not sources:
package = create_context_package_from_manifest(
manifest,
root=root,
budget=budget,
policy_gateway=gateway,
subject=subject,
action=action,
)
elif sources:
package = create_context_package_from_sources(
query_or_manifest,
list(sources),
root=root,
engine=engine,
title=title,
intent=intent,
namespace=namespace,
budget=budget,
policy_gateway=gateway,
subject=subject,
action=action,
)
else:
package = create_context_package_from_index(
query_or_manifest,
root=root,
index_path=index_path,
engine=engine,
paths=list(paths) or None,
search=search,
limit=limit,
title=title,
intent=intent,
namespace=namespace,
budget=budget,
policy_gateway=gateway,
subject=subject,
action=action,
)
except (ContextPackageError, InvalidQueryError, ValueError) as exc:
raise click.ClickException(str(exc)) from exc
registry_path = None
if not no_save:
registry_path = LocalContextPackageRegistry(root).save(package)
if output:
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(yaml.safe_dump(package.to_dict(), sort_keys=False), encoding="utf-8")
_emit_context_package(
package.to_dict()
| {
"registry_path": str(registry_path) if registry_path else None,
"output_path": str(output) if output else None,
},
output_format,
)
@context_group.command("activate")
@click.argument("package")
@click.option(
"--root",
type=click.Path(exists=True, file_okay=False, path_type=Path),
default=Path("."),
show_default=True,
help="Root used for the local context registry.",
)
@click.option("--target", default="default", show_default=True, help="Thread/workspace activation target.")
@click.option("--max-items", type=int, help="Maximum context items to activate.")
@click.option("--max-tokens", type=int, help="Approximate maximum activation tokens.")
@click.option("--reserve-tokens", type=int, default=0, show_default=True, help="Token reserve inside max tokens.")
@click.option(
"--policy",
"policy_file",
type=click.Path(exists=True, dir_okay=False, path_type=Path),
help="Local label policy file used to re-check items before activation.",
)
@click.option("--subject", default="anonymous", show_default=True, help="Policy subject id.")
@click.option(
"--policy-mode",
type=click.Choice(["off", "audit", "enforce"], case_sensitive=False),
help="Override policy mode while activating.",
)
@click.option("--action", default="read", show_default=True, help="Policy action used while activating.")
@click.option(
"--format",
"output_format",
type=click.Choice(["markdown", "json", "yaml", "text"], case_sensitive=False),
default="markdown",
show_default=True,
)
def context_activate(
package: str,
root: Path,
target: str,
max_items: int | None,
max_tokens: int | None,
reserve_tokens: int,
policy_file: Path | None,
subject: str,
policy_mode: str | None,
action: str,
output_format: str,
) -> None:
"""Activate a saved context package as Markdown working memory."""
try:
registry = LocalContextPackageRegistry(root)
loaded = registry.load(package)
activation = activate_context_package(
loaded,
target=target,
policy_gateway=_load_policy_gateway(policy_file, policy_mode),
subject=subject,
action=action,
budget=ContextBudget(max_tokens=max_tokens, max_items=max_items, reserve_tokens=reserve_tokens),
)
activation_path = registry.save_activation(activation)
registry.save(loaded.with_activation_state("active"))
except (ContextPackageError, ValueError) as exc:
raise click.ClickException(str(exc)) from exc
_emit_context_activation(
activation.to_dict() | {"activation_path": str(activation_path)},
output_format,
)
@context_group.command("deactivate")
@click.argument("activation_id")
@click.option(
"--root",
type=click.Path(exists=True, file_okay=False, path_type=Path),
default=Path("."),
show_default=True,
help="Root used for the local context registry.",
)
@click.option(
"--format",
"output_format",
type=click.Choice(["json", "yaml", "text"], case_sensitive=False),
default="text",
show_default=True,
)
def context_deactivate(activation_id: str, root: Path, output_format: str) -> None:
"""Deactivate a recorded context activation."""
try:
registry = LocalContextPackageRegistry(root)
registry.deactivate(activation_id)
activation = registry.load_activation(activation_id)
except ContextPackageError as exc:
raise click.ClickException(str(exc)) from exc
_emit_context_activation(activation.to_dict(), output_format)
@context_group.command("explain")
@click.argument("package")
@click.option(
"--root",
type=click.Path(exists=True, file_okay=False, path_type=Path),
default=Path("."),
show_default=True,
help="Root used for the local context registry.",
)
@click.option(
"--format",
"output_format",
type=click.Choice(["json", "yaml", "text"], case_sensitive=False),
default="text",
show_default=True,
)
def context_explain(package: str, root: Path, output_format: str) -> None:
"""Explain context package contents, retrieval, budget, and policy metadata."""
try:
loaded = LocalContextPackageRegistry(root).load(package)
except ContextPackageError as exc:
raise click.ClickException(str(exc)) from exc
_emit_context_explain(explain_context_package(loaded), output_format)
@context_group.command("refresh")
@click.argument("package")
@click.option(
"--root",
type=click.Path(exists=True, file_okay=False, path_type=Path),
default=Path("."),
show_default=True,
help="Root used for the local context registry.",
)
@click.option(
"--policy",
"policy_file",
type=click.Path(exists=True, dir_okay=False, path_type=Path),
help="Local label policy file used to filter refreshed package items.",
)
@click.option("--subject", default="anonymous", show_default=True, help="Policy subject id.")
@click.option(
"--policy-mode",
type=click.Choice(["off", "audit", "enforce"], case_sensitive=False),
help="Override policy mode while refreshing.",
)
@click.option("--action", default="package", show_default=True, help="Policy action used while refreshing.")
@click.option("--no-save", is_flag=True, help="Do not save the refreshed package.")
@click.option(
"--format",
"output_format",
type=click.Choice(["json", "yaml", "text"], case_sensitive=False),
default="text",
show_default=True,
)
def context_refresh(
package: str,
root: Path,
policy_file: Path | None,
subject: str,
policy_mode: str | None,
action: str,
no_save: bool,
output_format: str,
) -> None:
"""Refresh a package by re-running its retrieval recipes."""
try:
registry = LocalContextPackageRegistry(root)
loaded = registry.load(package)
refreshed = refresh_context_package(
loaded,
policy_gateway=_load_policy_gateway(policy_file, policy_mode),
subject=subject,
action=action,
)
registry_path = None if no_save else registry.save(refreshed)
except (ContextPackageError, ValueError) as exc:
raise click.ClickException(str(exc)) from exc
_emit_context_package(
refreshed.to_dict() | {"registry_path": str(registry_path) if registry_path else None},
output_format,
)
@context_group.command("list")
@click.option(
"--root",
type=click.Path(exists=True, file_okay=False, path_type=Path),
default=Path("."),
show_default=True,
help="Root used for the local context registry.",
)
@click.option(
"--format",
"output_format",
type=click.Choice(["json", "yaml", "text"], case_sensitive=False),
default="text",
show_default=True,
)
def context_list(root: Path, output_format: str) -> None:
"""List locally saved context packages."""
packages = [package.to_dict() for package in LocalContextPackageRegistry(root).list()]
_emit_context_package_list({"count": len(packages), "packages": packages}, output_format)
@main.group()
def workflow() -> None:
"""Inspect, plan, and run declarative Markdown workflows."""
@@ -1809,6 +2188,25 @@ def _load_policy_gateway(
raise click.ClickException(str(exc)) from exc
def _memory_namespace(
project: str | None,
user: str | None,
agent: str | None,
thread: str | None,
task: str | None,
namespace_pairs: tuple[str, ...],
) -> MemoryNamespace:
custom = _parse_key_value_options(namespace_pairs)
return MemoryNamespace(
project=project,
user=user,
agent=agent,
thread=thread,
task=task,
custom=custom,
)
def _emit_result(data: dict, output_format: str) -> None:
if output_format == "json":
click.echo(json.dumps(data, indent=2, ensure_ascii=False))
@@ -2084,6 +2482,83 @@ def _emit_search_results(data: dict, output_format: str) -> None:
click.echo(f"! [{diagnostic['severity']}] {diagnostic['code']}: {diagnostic['message']}")
def _emit_context_package(data: dict, output_format: str) -> None:
if output_format == "json":
click.echo(json.dumps(data, indent=2, ensure_ascii=False))
elif output_format == "yaml":
click.echo(yaml.safe_dump(data, sort_keys=False))
else:
click.echo(f"package: {data['id']}")
click.echo(f"title: {data.get('title', '')}")
click.echo(f"items: {len(data.get('items', []))}")
click.echo(f"token_estimate: {data.get('token_estimate', 0)}")
if data.get("registry_path"):
click.echo(f"registry_path: {data['registry_path']}")
if data.get("output_path"):
click.echo(f"output_path: {data['output_path']}")
if data.get("policy", {}).get("summary"):
_emit_policy_summary(data["policy"]["summary"])
for summary in data.get("summaries", []):
click.echo(f"- {summary['name']}: {summary['text']}")
def _emit_context_activation(data: dict, output_format: str) -> None:
if output_format == "json":
click.echo(json.dumps(data, indent=2, ensure_ascii=False))
elif output_format == "yaml":
click.echo(yaml.safe_dump(data, sort_keys=False))
elif output_format == "markdown":
click.echo(data.get("content", ""), nl=False)
else:
click.echo(f"activation: {data['id']}")
click.echo(f"status: {data.get('status')}")
click.echo(f"package: {data.get('package_id')}")
click.echo(f"items: {len(data.get('items', []))}")
click.echo(f"token_estimate: {data.get('token_estimate', 0)}")
if data.get("activation_path"):
click.echo(f"activation_path: {data['activation_path']}")
if data.get("policy", {}).get("summary"):
_emit_policy_summary(data["policy"]["summary"])
for diagnostic in data.get("diagnostics", []):
click.echo(f"! [{diagnostic['severity']}] {diagnostic['code']}: {diagnostic['message']}")
def _emit_context_explain(data: dict, output_format: str) -> None:
if output_format == "json":
click.echo(json.dumps(data, indent=2, ensure_ascii=False))
elif output_format == "yaml":
click.echo(yaml.safe_dump(data, sort_keys=False))
else:
click.echo(f"package: {data['id']}")
click.echo(f"title: {data.get('title', '')}")
click.echo(f"intent: {data.get('intent', '')}")
click.echo(f"activation_state: {data.get('activation_state', 'inactive')}")
click.echo(f"items: {data.get('items', 0)}")
click.echo(f"token_estimate: {data.get('token_estimate', 0)}")
if data.get("namespace"):
click.echo(f"namespace: {data['namespace']}")
if data.get("retrieval_recipes"):
click.echo("retrieval_recipes:")
for recipe in data["retrieval_recipes"]:
click.echo(f"- {recipe.get('kind')} {recipe.get('query')}")
if data.get("sources"):
click.echo("sources:")
for source in data["sources"]:
span = f":{source.get('line_start')}" if source.get("line_start") else ""
click.echo(f"- {source.get('path')}{span} {source.get('unit_kind', '')}")
def _emit_context_package_list(data: dict, output_format: str) -> None:
if output_format == "json":
click.echo(json.dumps(data, indent=2, ensure_ascii=False))
elif output_format == "yaml":
click.echo(yaml.safe_dump(data, sort_keys=False))
else:
click.echo(f"packages: {data.get('count', 0)}")
for package in data.get("packages", []):
click.echo(f"- {package['id']} {package.get('title', '')}")
def _emit_policy_summary(policy_data: dict) -> None:
click.echo(
"policy: "

49
src/markitect_tool/extension/builtins.py
View File

@@ -19,6 +19,7 @@ def builtin_extension_registry() -> ExtensionRegistry:
_runtime_assessment_descriptor(),
_local_label_policy_descriptor(),
_document_function_descriptor(),
_agent_memory_descriptor(),
]:
registry.register(descriptor)
return registry
@@ -265,3 +266,51 @@ def _document_function_descriptor() -> ExtensionDescriptor:
"external_policy_services_required": False,
},
)
def _agent_memory_descriptor() -> ExtensionDescriptor:
return ExtensionDescriptor(
id="memory.context-package",
kind="memory-registry",
summary="Local agent working-memory context package registry and activation lifecycle.",
capabilities=[
ProcessingCapability(id="context_packages", kind="create"),
ProcessingCapability(id="context_activation", kind="execute"),
ProcessingCapability(id="snapshots", kind="read"),
ProcessingCapability(id="fts", kind="read"),
ProcessingCapability(id="policy_filter", kind="filter"),
ProcessingCapability(id="provenance", kind="emit"),
ProcessingCapability(id="diagnostics", kind="emit"),
],
safety={
"reads_files": True,
"writes_local_context_registry": True,
"network": False,
"assisted_generation": False,
"external_policy_engine": False,
},
input_contract="Selector/search/manifest + local snapshots/documents",
output_contract="ContextPackage | ContextActivation",
diagnostics_namespace="memory",
provenance_prefix="memory.context_package",
cli={
"commands": [
"mkt context pack",
"mkt context activate",
"mkt context deactivate",
"mkt context explain",
"mkt context refresh",
"mkt context list",
]
},
docs=[
"docs/agent-working-memory.md",
"docs/agent-working-memory-thought-experiment.md",
],
examples=["examples/memory/workplan-context.manifest.yaml"],
metadata={
"external_policy_services_required": False,
"assisted_summaries": "future adapter-only",
"default_registry": ".markitect/context",
},
)

43
src/markitect_tool/memory/__init__.py Normal file
View File

@@ -0,0 +1,43 @@
"""Agent working-memory context packages."""
from markitect_tool.memory.engine import (
ContextActivation,
ContextBudget,
ContextPackage,
ContextPackageError,
ContextPackageItem,
LocalContextPackageRegistry,
MemoryNamespace,
RetrievalRecipe,
SourceSpan,
SummaryLayer,
activate_context_package,
create_context_package_from_index,
create_context_package_from_manifest,
create_context_package_from_sources,
deactivate_context_package,
explain_context_package,
load_context_package_file,
refresh_context_package,
)
__all__ = [
"ContextActivation",
"ContextBudget",
"ContextPackage",
"ContextPackageError",
"ContextPackageItem",
"LocalContextPackageRegistry",
"MemoryNamespace",
"RetrievalRecipe",
"SourceSpan",
"SummaryLayer",
"activate_context_package",
"create_context_package_from_index",
"create_context_package_from_manifest",
"create_context_package_from_sources",
"deactivate_context_package",
"explain_context_package",
"load_context_package_file",
"refresh_context_package",
]

1255
src/markitect_tool/memory/engine.py Normal file
View File

File diff suppressed because it is too large Load Diff

172
tests/test_agent_working_memory.py Normal file
View File

@@ -0,0 +1,172 @@
from pathlib import Path
import yaml
from click.testing import CliRunner
from markitect_tool.cli import main
from markitect_tool.memory import (
ContextBudget,
LocalContextPackageRegistry,
MemoryNamespace,
activate_context_package,
create_context_package_from_index,
create_context_package_from_manifest,
create_context_package_from_sources,
refresh_context_package,
)
from markitect_tool.policy import LocalLabelPolicyGateway
def test_create_context_package_from_sources_preserves_spans_and_summary(tmp_path: Path):
source = tmp_path / "doc.md"
source.write_text("# Doc\n\n## Decision\n\nUse local memory packages.\n", encoding="utf-8")
package = create_context_package_from_sources(
"sections[heading=Decision]",
[source],
root=tmp_path,
namespace=MemoryNamespace(project="markitect-tool", task="MKTT-WP-0008"),
budget=ContextBudget(max_items=1),
)
assert package.id.startswith("memory:package:")
assert package.namespace.project == "markitect-tool"
assert package.items[0].source.path == "doc.md"
assert package.items[0].source.unit_kind == "section"
assert "local memory packages" in package.items[0].text
assert package.summaries[0].name == "overview"
def test_local_context_registry_saves_loads_and_activates(tmp_path: Path):
source = tmp_path / "doc.md"
source.write_text("# Doc\n\n## Context\n\nReusable project facts.\n", encoding="utf-8")
package = create_context_package_from_sources("sections[heading=Context]", [source], root=tmp_path)
registry = LocalContextPackageRegistry(tmp_path)
saved_path = registry.save(package)
loaded = registry.load(package.id)
activation_id = registry.activate(package.id, "thread:test")
activation = registry.load_activation(activation_id)
assert saved_path.exists()
assert loaded.id == package.id
assert activation.status == "active"
assert "Reusable project facts" in activation.content
def test_activation_can_recheck_policy_and_drop_denied_items(tmp_path: Path):
public = tmp_path / "public.md"
private = tmp_path / "private.md"
public.write_text("---\npolicy:\n labels: [public]\n---\n# Public\n\nVisible.\n", encoding="utf-8")
private.write_text("---\npolicy:\n labels: [internal]\n---\n# Private\n\nHidden.\n", encoding="utf-8")
package = create_context_package_from_sources("document", [public, private], root=tmp_path)
gateway = LocalLabelPolicyGateway(
{
"id": "memory-test-policy",
"subjects": {
"reader": {
"allowed_labels": ["public"],
"allowed_actions": ["read"],
}
},
"default_subject": "reader",
}
)
activation = activate_context_package(package, policy_gateway=gateway, subject="reader")
assert len(activation.items) == 1
assert "Visible" in activation.content
assert "Hidden" not in activation.content
assert activation.policy["summary"]["denied"] == 1
def test_create_context_package_from_index_search_and_refresh(tmp_path: Path):
source = tmp_path / "doc.md"
source.write_text("# Doc\n\nSearchable memory fact.\n", encoding="utf-8")
runner = CliRunner()
indexed = runner.invoke(main, ["cache", "index", str(tmp_path), "--root", str(tmp_path)])
assert indexed.exit_code == 0
package = create_context_package_from_index("Searchable", root=tmp_path, search=True)
source.write_text("# Doc\n\nSearchable memory fact changed.\n", encoding="utf-8")
reindexed = runner.invoke(main, ["cache", "index", str(tmp_path), "--root", str(tmp_path)])
refreshed = refresh_context_package(package)
assert reindexed.exit_code == 0
assert package.items
assert "changed" in refreshed.items[0].text
assert refreshed.id == package.id
def test_create_context_package_from_manifest(tmp_path: Path):
source = tmp_path / "doc.md"
source.write_text("# Doc\n\n## Purpose\n\nManifest selected context.\n", encoding="utf-8")
manifest = tmp_path / "context.yaml"
manifest.write_text(
yaml.safe_dump(
{
"title": "Manifest package",
"namespace": {"project": "markitect-tool"},
"retrieval_recipes": [
{
"kind": "selector",
"query": "sections[heading=Purpose]",
"sources": ["doc.md"],
}
],
},
sort_keys=False,
),
encoding="utf-8",
)
package = create_context_package_from_manifest(manifest, root=tmp_path)
assert package.title == "Manifest package"
assert package.namespace.project == "markitect-tool"
assert "Manifest selected context" in package.items[0].text
def test_mkt_context_pack_activate_explain_and_deactivate(tmp_path: Path):
source = tmp_path / "doc.md"
source.write_text("# Doc\n\n## Decision\n\nUse explicit context packages.\n", encoding="utf-8")
runner = CliRunner()
packed = runner.invoke(
main,
[
"context",
"pack",
"sections[heading=Decision]",
"--source",
str(source),
"--root",
str(tmp_path),
"--format",
"json",
],
)
assert packed.exit_code == 0
package_id = yaml.safe_load(packed.output)["id"]
explained = runner.invoke(main, ["context", "explain", package_id, "--root", str(tmp_path)])
activated = runner.invoke(
main,
[
"context",
"activate",
package_id,
"--root",
str(tmp_path),
"--format",
"json",
],
)
activation_id = yaml.safe_load(activated.output)["id"]
deactivated = runner.invoke(main, ["context", "deactivate", activation_id, "--root", str(tmp_path)])
assert explained.exit_code == 0
assert "Use explicit context packages" in activated.output
assert deactivated.exit_code == 0
assert "inactive" in deactivated.output

19
tests/test_builtin_extension_catalog.py
View File

@@ -19,6 +19,7 @@ def test_builtin_extension_registry_lists_query_processors_and_backend():
assert "runtime.assessment" in ids
assert "policy.local-label" in ids
assert "document.function" in ids
assert "memory.context-package" in ids
def test_builtin_processor_descriptors_capture_safety_and_provenance():
@@ -123,3 +124,21 @@ def test_builtin_document_function_descriptor_exposes_deterministic_boundary():
"mkt function check",
"mkt function render",
]
def test_builtin_memory_descriptor_exposes_local_optional_boundary():
registry = builtin_extension_registry()
descriptor = registry.get("memory.context-package")
assert descriptor.kind == "memory-registry"
assert descriptor.safety["network"] is False
assert descriptor.safety["writes_local_context_registry"] is True
assert descriptor.metadata["external_policy_services_required"] is False
assert {capability.id for capability in descriptor.capabilities} >= {
"context_packages",
"context_activation",
"policy_filter",
}
assert "mkt context pack" in descriptor.cli["commands"]
assert "mkt context activate" in descriptor.cli["commands"]

1
tests/test_cli_extension_specs.py
View File

@@ -19,6 +19,7 @@ def test_collect_cli_command_specs_from_builtin_registry():
assert ("backend.local-sqlite", "mkt cache index") in commands
assert ("backend.local-sqlite", "mkt search") in commands
assert ("document.function", "mkt function render") in commands
assert ("memory.context-package", "mkt context pack") in commands
def test_cli_command_spec_serializes_without_empty_fields():

104
workplans/MKTT-WP-0008-agent-working-memory-context-cache.md
View File

@@ -3,17 +3,17 @@ id: MKTT-WP-0008
type: workplan
title: "Agent Working Memory Context Cache"
domain: markitect
status: todo
status: done
owner: markitect-tool
topic_slug: markitect
planning_priority: P3
planning_priority: complete
planning_order: 90
depends_on_workplans:
- MKTT-WP-0006
- MKTT-WP-0007
- MKTT-WP-0009
created: "2026-05-03"
updated: "2026-05-03"
updated: "2026-05-04"
state_hub_workstream_id: "6269f338-4f5c-40ee-90e5-0371f5c3874c"
---
@@ -25,6 +25,54 @@ Create activatable context packages that let agents drop, reactivate, and
reuse project knowledge efficiently while preserving provenance and policy
metadata.
## Thought Experiment And Design Refinement
The implementation was preceded by a wishful architecture pass documented in
`docs/agent-working-memory-thought-experiment.md`.
The useful model is layered memory:
- reflex context for the current response
- short working set for the active task
- episodic memory for a thread or work session
- project semantic memory for architecture and source knowledge
- identity-like continuity memory for durable principles and preferences
- policy/safety memory that must be rechecked before activation
The key refinement is that Markitect should implement the local, explicit,
inspectable package layer rather than hidden ambient agent memory. Long-term
identity and learning memory can be represented as reviewed namespace-scoped
packages later, but writes should remain visible and reversible.
This shaped the implementation:
- package creation and activation are explicit CLI/library operations
- every item carries source spans, summaries, token estimates, provenance, and
policy metadata
- deterministic summaries come first
- package creation and activation can both run local policy checks
- local files and the local SQLite index are enough for the first version
- LLM summaries, embeddings, remote registries, retention/decay, and durable
identity packages remain future optional extensions
## Implementation Summary
Implemented the first local agent working-memory context cache:
- `markitect_tool.memory` extension package with context package schema,
retrieval recipes, budgets, namespaces, summary layers, activation envelopes,
and filesystem-backed local registry.
- Package creation from direct Markdown source queries, local index selector or
JSONPath queries, FTS search results, and YAML manifests.
- Deterministic summaries and approximate token estimates.
- Activation, deactivation, refresh, explain, list, and save/load lifecycle.
- Optional local policy filtering at package creation and activation.
- CLI commands under `mkt context`.
- Built-in extension descriptor `memory.context-package`.
- Documentation and example manifest in `docs/agent-working-memory.md`,
`docs/agent-working-memory-thought-experiment.md`, and
`examples/memory/workplan-context.manifest.yaml`.
## Architectural Boundary
This workplan depends only on Markitect-local backend and policy contracts:
@@ -48,7 +96,7 @@ freshness metadata. These references are metadata, not hard dependencies.
```task
id: MKTT-WP-0008-T001
status: todo
status: done
priority: high
state_hub_task_id: "21ee9c37-4add-4886-bd03-a7bb4b20e957"
```
@@ -68,11 +116,16 @@ The schema should include optional policy metadata:
These fields must support local policy gateways first and external policy
services only through optional adapters.
Implemented: `ContextPackage`, `ContextPackageItem`, `SourceSpan`,
`RetrievalRecipe`, `SummaryLayer`, `ContextBudget`, and `MemoryNamespace`
define the local schema. Package items preserve source spans, summaries, token
estimates, provenance, freshness, and policy metadata.
## P8.2 - Implement package creation from queries
```task
id: MKTT-WP-0008-T002
status: todo
status: done
priority: high
state_hub_task_id: "4df06b93-13ce-41fb-a8c3-f04d4ad9d752"
```
@@ -84,11 +137,16 @@ Package creation should use current query/search APIs and policy-aware result
filtering. It should not call flex-auth directly; future flex-auth-backed
filtering can be injected through the existing policy gateway boundary.
Implemented: packages can be created from direct Markdown source queries, local
indexed selector/JSONPath queries, FTS search results, and YAML manifests.
Optional local policy gateways filter package items using the existing
`LocalLabelPolicyGateway` boundary.
## P8.3 - Implement activation lifecycle
```task
id: MKTT-WP-0008-T003
status: todo
status: done
priority: medium
state_hub_task_id: "9f3d9792-d655-482d-bae0-262df5fc0136"
```
@@ -99,11 +157,17 @@ Activation should re-check local policy metadata when a policy gateway is
available. In the absence of an external service, activation remains fully
local and explainable.
Implemented: `activate_context_package`, `deactivate_context_package`,
`refresh_context_package`, `explain_context_package`, and
`LocalContextPackageRegistry` support the lifecycle locally. Activation
rebuilds summaries after policy filtering so denied content does not leak
through package-level summaries.
## P8.4 - Add memory namespaces
```task
id: MKTT-WP-0008-T004
status: todo
status: done
priority: medium
state_hub_task_id: "2d090494-0e10-44cd-8e2d-c418d7530b27"
```
@@ -115,11 +179,14 @@ Namespace design should leave room for enterprise subject ids and external
resource ids, but must not require any particular SSO, IAM, or authorization
provider.
Implemented: `MemoryNamespace` supports project, user, agent, thread, task,
and custom namespace fields without depending on any external agent platform.
## P8.5 - Add summary layers
```task
id: MKTT-WP-0008-T005
status: todo
status: done
priority: medium
state_hub_task_id: "4d1cf970-3d6d-4bd5-8da9-ec2399aa7efe"
```
@@ -130,11 +197,15 @@ through an injected adapter.
Assisted summaries must be optional and policy/capability-gated before any
prompt assembly happens.
Implemented: deterministic extractive/count summaries are included. Assisted
summaries remain a documented future adapter path and are not invoked by core
package creation or activation.
## P8.6 - Add CLI commands
```task
id: MKTT-WP-0008-T006
status: todo
status: done
priority: medium
state_hub_task_id: "2f18386c-9d2c-4af1-b8e2-75cb487c1692"
```
@@ -152,6 +223,21 @@ CLI commands should work against local packages without flex-auth. Optional
policy flags may accept local policy files or later external adapter
configuration.
Implemented:
```text
mkt context pack
mkt context activate
mkt context deactivate
mkt context explain
mkt context refresh
mkt context list
```
Commands work with local package files and the `.markitect/context` registry.
Policy flags use local label policy only; external policy systems remain
optional adapters.
## Exit Criteria
- Agents can reactivate project context by stable id.