phase-memory/workplans/PMEM-WP-0006-retrieval-activation-quality-and-evaluation.md

id, type, title, domain, repo, status, owner, topic_slug, planning_priority, planning_order, related_workplans, created, updated, state_hub_workstream_id

id	type	title	domain	repo	status	owner	topic_slug	planning_priority	planning_order	related_workplans	created	updated	state_hub_workstream_id
PMEM-WP-0006	workplan	Retrieval, Activation Quality, And Evaluation	markitect	phase-memory	finished	phase-memory	activation-quality	P2	60	PMEM-WP-0002 PMEM-WP-0005	2026-05-18	2026-05-18	68417e46-d0a4-4168-bdd6-9ba6ac7847c1

PMEM-WP-0006: Retrieval, Activation Quality, And Evaluation

Goal

Move activation planning beyond deterministic id ordering into explainable, policy-aware memory retrieval and evaluation.

INTENT.md expects activation memory to compile graph neighborhoods, decision paths, conversation episodes, and knowledge slices into bounded context packages. The current activation planner selects nodes under item and token budgets, but it does not yet model graph neighborhoods, event paths, freshness, confidence, or retrieval quality.

Current Evidence

Current activation behavior:

• prioritizes explicitly requested node ids
• sorts remaining nodes by id
• estimates tokens with word count
• includes events only when they reference packages or activations
• emits Markitect-compatible selection dictionaries
• explains budget omissions

This is enough for a first handoff, but not enough to satisfy the full activation-memory intent.

Non-Goals

• Do not require embeddings or vector stores in the default path.
• Do not use live LLM ranking in deterministic tests.
• Do not own benchmark dashboards that belong in infospace-bench.
• Do not optimize for a single application domain.

Implementation Update - 2026-05-18

The retrieval, activation quality, and evaluation slice is complete.

Implemented outputs:

• phase_memory.retrieval adds deterministic graph-neighborhood retrieval, candidate scoring, event-path selection, pluggable token estimator protocol, neighborhood activation planning, and activation quality reports.
• Retrieval supports max hops, edge-kind filters, direction filters, phase filters, and memory kind filters.
• Event-path activation selects bounded event windows from structured MemoryPath records and treats inactive paths as opt-in.
• Ranking signals include explicit priority, graph distance, phase, lifecycle state, confidence, source-backed status, freshness, and policy allowance.
• WordCountTokenEstimator provides deterministic local budget accounting.
• activation_quality_report emits selected expected nodes, omitted required nodes, policy-denied required nodes, token budget utilization, stale item count, provenance coverage, source span coverage, and explanation coverage.
• docs/activation-quality.md documents retrieval, event paths, scoring, estimator boundaries, and evaluation metrics.

Validation:

• python3 -m pytest -> 46 passed.

T01 - Add deterministic graph-neighborhood retrieval

id: PMEM-WP-0006-T01
status: done
priority: high
state_hub_task_id: "8ed0909f-9e8e-4d49-9312-dca267df29f5"

Add selection strategies that can expand from seed nodes across graph edges:

• one-hop neighbors
• bounded multi-hop neighbors
• edge-kind filters
• source and target direction filters
• phase filters
• memory kind filters

Output: retrieval planner and tests for stable graph-neighborhood selection.

T02 - Add event-path activation

id: PMEM-WP-0006-T02
status: done
priority: high
state_hub_task_id: "5d48ba91-fef0-4d4f-a560-836abed1c527"

Select conversational path episodes and event windows as first-class activation inputs.

Output: event-path selection support, path budget behavior, and tests around active, abandoned, and merged paths.

T03 - Add ranking signals and explanations

id: PMEM-WP-0006-T03
status: done
priority: high
state_hub_task_id: "0f6340ef-f7bd-408b-b98e-6d90188c5969"

Rank activation candidates using deterministic local signals:

• explicit priority
• graph distance from seed
• lifecycle state
• phase
• freshness
• confidence
• policy allowance
• source-backed status
• prior activation references

Output: scoring model, per-item selection reason, and omitted-item reason.

T04 - Improve token and budget accounting

id: PMEM-WP-0006-T04
status: done
priority: medium
state_hub_task_id: "12d83382-a767-45a8-b7cc-8c3f6f3f4c37"

Replace rough word-count behavior with a pluggable budget estimator that can stay deterministic locally and later delegate to provider-specific tokenizers.

Output: estimator protocol, default estimator, and tests for node, event, and package budget pressure.

T05 - Add evaluation fixture scenarios

id: PMEM-WP-0006-T05
status: done
priority: medium
state_hub_task_id: "509e9417-3aa7-4899-aed5-20749372fe00"

Add small evaluation fixtures inspired by adjacent infospace-bench pilots:

• restart package
• decision recall
• stale source refresh
• policy-denied activation
• compacted trace window
• conflicting preference update

Output: fixture set and expected activation plans.

T06 - Add maturity metrics for activation quality

id: PMEM-WP-0006-T06
status: done
priority: medium
state_hub_task_id: "477a896a-8013-42a5-b965-b1ccd2577fec"

Define local metrics that can be exported to infospace-bench later:

• selected expected nodes
• omitted required nodes
• policy-denied required nodes
• token budget utilization
• stale item activation count
• provenance coverage
• source span coverage
• explanation coverage

Output: metrics helper and JSON report fixture.

T07 - Document retrieval and evaluation behavior

id: PMEM-WP-0006-T07
status: done
priority: medium
state_hub_task_id: "551432e4-2551-49fa-b17b-f762853a6a50"

Document activation strategy, scoring inputs, limitations, and evaluation fixtures.

Output: activation planning guide and scorecard update.

Acceptance Criteria

• python3 -m pytest passes.
• Activation can select graph neighborhoods and event paths under budget.
• Every selected and omitted item has a machine-readable reason.
• Evaluation fixtures produce deterministic activation quality reports.
• Optional semantic indexes remain behind the SemanticIndex port.

Closure Review - 2026-05-18

Outcome: All tasks completed.

Completed

• PMEM-WP-0006-T01 - Add deterministic graph-neighborhood retrieval
• PMEM-WP-0006-T02 - Add event-path activation
• PMEM-WP-0006-T03 - Add ranking signals and explanations
• PMEM-WP-0006-T04 - Improve token and budget accounting
• PMEM-WP-0006-T05 - Add evaluation fixture scenarios
• PMEM-WP-0006-T06 - Add maturity metrics for activation quality
• PMEM-WP-0006-T07 - Document retrieval and evaluation behavior

Cancelled

None.

Carried Forward

Service contracts, runtime configuration, health diagnostics, and external adapter conformance remain in PMEM-WP-0007.