From 7c6f4358ee04e77b048710dfcb631f3fd757a5a3 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 6 Jun 2026 21:44:46 +0200 Subject: [PATCH] session-memory Phase 0: end-to-end verification + docs (T07) - verified full sweep over 85 real local Claude transcripts: 63 sessions ingested+analyzed, eviction under tiny cap freed 26MB with zero data loss, digest-preservation invariant holds, idempotent re-run - session_memory/README.md: usage, scheduling, retention knobs - design doc: OQ4 resolved (median ~49KB/session), OQ6 (multi-file sessions) - workplan AGENTIC-WP-0002 finished Co-Authored-By: Claude Opus 4.8 --- docs/DESIGN-session-memory.md | 12 ++- session_memory/README.md | 75 +++++++++++++++++++ .../AGENTIC-WP-0002-session-memory-phase0.md | 12 ++- 3 files changed, 95 insertions(+), 4 deletions(-) create mode 100644 session_memory/README.md diff --git a/docs/DESIGN-session-memory.md b/docs/DESIGN-session-memory.md index 7aa00e1..93e70f8 100644 --- a/docs/DESIGN-session-memory.md +++ b/docs/DESIGN-session-memory.md @@ -356,8 +356,16 @@ hub indexes). human-intervention markers), refine in Detect phase. - **OQ3** `task_ref` resolution — can we always map a session to a workplan task (via cwd + branch + state-hub), or only sometimes? -- **OQ4** Right default for `raw_soft_cap_bytes` — measure real per-session sizes - first (Grok dirs are heavier than Claude single-files). +- ~~**OQ4** Right default for `raw_soft_cap_bytes`.~~ **Measured** (Phase 0, 85 + real local Claude files / 63 distinct sessions): source bytes per session + min 396 · **median ~49 KB** · max 48 MB (one outlier) · ~103 MB total. Claude + defaults (4 GiB soft / 6 GiB hard) leave ample headroom; revisit once Grok dirs + (heavier, multi-file) are ingested in Phase 1. +- **OQ6 (new, found in Phase 0)** Multi-file sessions: ~84 transcript files mapped + to ~63 `session_uid`s — some sessions span multiple files (resume/sidechain + sharing a `sessionId`). Current behavior upserts (last file wins per + `(session_uid, seq)`); a future refinement is to *merge* events across files of + one session rather than overwrite. Acceptable for Phase 0. - **OQ5** Should push-hooks be opt-in per machine to avoid surprising the agents? --- diff --git a/session_memory/README.md b/session_memory/README.md new file mode 100644 index 0000000..2f48a5a --- /dev/null +++ b/session_memory/README.md @@ -0,0 +1,75 @@ +# session_memory + +Capture + retention layer for Helix Forge — the **Capture** stage of the loop in +[../docs/PRD-helix-forge.md](../docs/PRD-helix-forge.md), built to the +[../docs/DESIGN-session-memory.md](../docs/DESIGN-session-memory.md) spec. + +It scans coding-agent session logs, normalizes them into one schema, distills a +compact per-session digest, and ages out raw bulk under a **storage budget** +(dropping sessions once analyzed and once space is needed) rather than a fixed +time window. + +## Layout + +``` +session_memory/ + adapters/claude.py # Tier0 -> Tier1 normalizer (Codex/Grok land in Phase 1) + core/schema.py # Session / SessionEvent / Cost + core/store.py # SQLite rows + blob-dir bodies (Tier1) + digests (Tier2) + core/cursor.py # incremental ingest cursors + core/digest.py # Tier1 -> Tier2 promotion + outcome heuristic + core/retention.py # budget-based eviction sweep + ingest.py # one sweep: discover -> normalize -> store -> digest -> evict + config.toml # store paths, retention caps, sources, repo->domain map +``` + +The local store lives under `session_memory/.store/` (gitignored). + +## Run a sweep + +```bash +# from the repo root +python -m session_memory.ingest # ingest + analyze + evict +python -m session_memory.ingest --dry-run # discover + parse only, writes nothing +python -m session_memory.ingest --config path/to/config.toml +``` + +Output reports `discovered / ingested / skipped_unchanged / analyzed` and a +retention line (`freed`, `final_usage`, and per-pass eviction counts). Sweeps are +idempotent — re-running skips unchanged files via the cursor. + +## Scheduling (cadence) + +Retention is budget-based; the `cadence` in `config.toml` only decides how often +the sweep *runs*. Trigger it with the repo scheduler, e.g. daily: + +```bash +# Claude Code: schedule a daily routine that runs the sweep +/schedule "daily session-memory sweep" -- python -m session_memory.ingest +``` + +or a cron entry / `/loop` on a timer. Push-capture (agent Stop/SessionEnd hooks) +can also enqueue a sweep; see design §7. + +## Retention knobs (`[retention]` in config.toml) + +| Key | Meaning | +|-----|---------| +| `raw_soft_cap_bytes` | begin evicting **analyzed** sessions above this (oldest first) | +| `raw_hard_cap_bytes` | absolute Tier 1 ceiling; overflow path may, as a last resort, evict un-analyzed sessions and report `data_loss` | +| `raw_max_age_days` | backstop: analyzed raw older than this is evictable regardless of space | +| `distilled_cap_bytes` | Tier 2 ceiling — **alert only**, never auto-dropped | + +**Invariant:** a session's raw bytes are never dropped before its Tier 2 digest +exists, except the explicitly-reported hard-cap overflow path. + +## Tests + +```bash +python -m pytest # 26 tests: schema, adapter, store, digest, retention, ingest +``` + +## Status + +Phase 0 (AGENTIC-WP-0002): Claude adapter only, end to end. Codex and Grok +adapters are designed (schemas confirmed in the design doc) and land in Phase 1. diff --git a/workplans/AGENTIC-WP-0002-session-memory-phase0.md b/workplans/AGENTIC-WP-0002-session-memory-phase0.md index 267e668..d34f083 100644 --- a/workplans/AGENTIC-WP-0002-session-memory-phase0.md +++ b/workplans/AGENTIC-WP-0002-session-memory-phase0.md @@ -4,7 +4,7 @@ type: workplan title: "Coding Session Memory — Phase 0 (Capture + budget-based retention)" domain: helix_forge repo: agentic-resources -status: active +status: finished owner: codex topic_slug: helix-forge created: "2026-06-06" @@ -118,7 +118,7 @@ intended `cadence` trigger (`/schedule` daily/weekly) in the repo docs. ```task id: AGENTIC-WP-0002-T07 -status: progress +status: done priority: medium state_hub_task_id: "98d5cc7c-c285-4556-91a3-a85e0a2bb6df" ``` @@ -133,3 +133,11 @@ sizes). After workplan file updates, notify the custodian operator to run from ```bash make fix-consistency REPO=agentic-resources ``` + +**Verification results (2026-06-06):** full suite 26/26 green. Live sweep over 85 +real local Claude transcripts → 84 parsed (1 empty, tolerated) → 63 distinct +sessions ingested + analyzed. Eviction under a 2 MB soft cap freed 26 MB, ended at +1.34 MB (under cap), zero data loss, and every evicted session kept its Tier 2 +digest (invariant holds). Idempotent re-run skipped 84 unchanged, re-ingested 1 +(the live session). Outcomes 52 success / 11 abandoned. OQ4 sizes recorded in the +design doc; OQ6 (multi-file sessions) noted as a Phase-1 refinement.