Implement AGENTIC-WP-0011 kaizen correlation follow-up

Add bidirectional doc links (PRD §9.1, README, DESIGN §11), session-close HELIX_* env convention, stable digest JSON contract, and digest_lookup CLI for read-only correlate lookups. All tasks done; 163 tests green.
2026-06-19 20:27:00 +02:00
parent 292b656952
commit 01d2affc3b
6 changed files with 279 additions and 6 deletions
--- a/docs/DESIGN-session-memory.md
+++ b/docs/DESIGN-session-memory.md
@@ -394,10 +394,65 @@ into Helix Forge.
 **Related kaizen-agentic docs:** [ADR-004 project metrics convention](https://gitea.coulomb.social/coulomb/kaizen-agentic/src/branch/main/docs/adr/ADR-004-project-metrics-convention.md),
 [wiki/EcosystemIntegration.md](https://gitea.coulomb.social/coulomb/kaizen-agentic/src/branch/main/wiki/EcosystemIntegration.md).

+### 11.1 Session-close env export (dual-layer agents)
+
+Agents that run **both** Helix Forge capture and kaizen `metrics record` should
+export the following **after** the ingest sweep has written the session digest
+(`python -m session_memory.ingest` or an equivalent Stop/SessionEnd hook). Names
+match kaizen-agentic ADR-004 — do not invent parallel aliases.
+
+| Variable | Source in Helix Forge | Purpose |
+|----------|----------------------|---------|
+| `HELIX_SESSION_UID` | `Session.session_uid` | Primary correlation key → `helix_session_uid` |
+| `HELIX_REPO` | `digest.repo` | Project/repo scoping |
+| `HELIX_FLAVOR` | `digest.flavor` | Agent runtime (`claude` / `codex` / `grok`) |
+| `HELIX_TOKENS` | `digest.cost.input_tokens + digest.cost.output_tokens` | Token rollup → `tokens` |
+| `HELIX_INFRA_OVERHEAD_SHARE` | infra bucket share over `tool_histogram` (see `measure.metrics.session_metrics`) | MCP/plumbing overhead → `infra_overhead_share` |
+
+Example (after digest exists):
+
+```bash
+export HELIX_SESSION_UID="claude:abc-123"
+export HELIX_REPO="agentic-resources"
+export HELIX_FLAVOR="claude"
+export HELIX_TOKENS=125000
+export HELIX_INFRA_OVERHEAD_SHARE=0.117
+# optional — lets kaizen correlate without guessing the store location:
+export HELIX_STORE_DB="$(pwd)/session_memory/.store/mem.db"
+kaizen-agentic metrics record   # merges HELIX_* when present
+```
+
+### 11.2 Digest store location and read API
+
+- **`HELIX_STORE_DB`** — absolute path to the SQLite file holding Tier 2 digests.
+  Defaults to `config.toml` `[store].db_path` (`session_memory/.store/mem.db` relative
+  to the repo root). Export as an absolute path when setting the variable on session
+  close so `metrics correlate` works across hosts and working directories.
+- **Thin CLI** — `python -m session_memory.digest_lookup <session_uid> [--json]`
+  prints one digest without running ingest. Exit `0` on hit, `1` when missing.
+- **Programmatic** — `Store.get_digest(session_uid)` returns the JSON blob written
+  by `build_digest` / `analyze`.
+
+**Stable digest JSON shape** (fields consumers may rely on):
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `session_uid` | string | Normalized uid (`<flavor>:<native-id>`) |
+| `flavor`, `repo`, `domain` | string | Session attribution |
+| `model` | string | Model id when known |
+| `started_at`, `ended_at` | string | ISO timestamps |
+| `outcome` | string | `success` / `fail` / `abandoned` / `unknown` |
+| `cost` | object | `input_tokens`, `output_tokens`, `cache_tokens`, `wall_clock_s`, `turns`, `retries` |
+| `tool_histogram` | object | Tool name → call count |
+| `event_count`, `kind_counts`, `markers` | object/int | Compact activity summary |
+| `first_prompt`, `last_assistant` | string | Short text snippets |
+| `error_snippets` | array | `{fingerprint, sample, count, tool}` entries |
+| `schema_version` | int | Digest schema version |
+
 ---

-*Next step: [AGENTIC-WP-0002] implements Phase 0 — the schema, the Claude
-collector, the Tier1/Tier2 store, and the budget-based eviction sweep.*
+*Implemented:* Phases 0–4, weekly retro ([AGENTIC-WP-0002]–[AGENTIC-WP-0010]);
+kaizen correlation follow-up ([AGENTIC-WP-0011]).

 ## Sources

--- a/docs/PRD-helix-forge.md
+++ b/docs/PRD-helix-forge.md
@@ -5,7 +5,7 @@
 **Status:** Draft v0.1
 **Author:** Claude (drafted with Bernd Worsch)
 **Created:** 2026-06-06
-**Updated:** 2026-06-06
+**Updated:** 2026-06-19

 ---

@@ -223,6 +223,32 @@ record:
 - The hub remains a **read model**; Helix Forge writes its durable artifacts as files
  and lets the hub index them.

+### 9.1 Downstream: kaizen-agentic project metrics correlation
+
+Helix Forge is a **fleet-level** producer of normalized session digests. The
+**kaizen-agentic** framework is a **project-scoped** consumer of optional
+correlation fields on its execution metrics (ADR-004). The two layers link
+**by reference** — kaizen-agentic does not re-implement JSONL ingestion or write
+into the Helix Forge store.
+
+| Layer | Owner | What it stores |
+|-------|-------|----------------|
+| Fleet | agentic-resources (`session_memory`) | Per-session digests in the local SQLite store |
+| Project | kaizen-agentic | `.kaizen/metrics/<agent>/executions.jsonl` |
+
+**Canonical spec in this repo:** [DESIGN-session-memory.md §11](DESIGN-session-memory.md#11-project-metrics-correlation-kaizen-agentic)
+(session-close env export, digest read path, stable JSON shape).
+
+**Authoritative cross-repo contract (kaizen-agentic):**
+[Helix Forge Correlation Contract](https://gitea.coulomb.social/coulomb/kaizen-agentic/src/branch/main/docs/integrations/helix-forge-correlation.md).
+Field mapping: `Session.session_uid` → `helix_session_uid`; digest token totals →
+`tokens`; MCP/tool overhead share → `infra_overhead_share`.
+
+**Read path for consumers:** `HELIX_STORE_DB` points at the digest SQLite file
+(default `session_memory/.store/mem.db`); `python -m session_memory.digest_lookup
+<uid> --json` or `kaizen-agentic metrics correlate <uid>` performs a read-only
+lookup. No ingestion code belongs in kaizen-agentic.
+
 ## 10. Success Metrics

 | Metric | Meaning | Target (directional, v1) |
--- a/session_memory/README.md
+++ b/session_memory/README.md
@@ -45,6 +45,7 @@ session_memory/
  retro/build.py       # windowed top-3-per-repo suggestions
  retro/publish.py     # hub coding_retro read model + local report
  retro/__main__.py    # python -m session_memory.retro
+  digest_lookup.py     # python -m session_memory.digest_lookup (read one digest, no ingest)
  config.toml          # store paths, retention caps, sources, repo->domain map, curate gate
 ```

@@ -184,6 +185,39 @@ Writes `retro/last_retro.{json,md}` and (with `--publish`) posts an
 which emits one improvement task per relevant repo. Hub publish degrades
 gracefully when the hub is unreachable.

+## Correlation with kaizen-agentic
+
+Helix Forge owns **fleet-level** session digests; **kaizen-agentic** owns
+**project-scoped** execution metrics (ADR-004). The two layers correlate by
+optional `helix_session_uid` on project records — **link-by-reference only**;
+kaizen-agentic does not ingest JSONL into this store.
+
+| Layer | Storage |
+|-------|---------|
+| Fleet (here) | `session_memory/.store/mem.db` → `digests` table |
+| Project (kaizen) | `.kaizen/metrics/<agent>/executions.jsonl` |
+
+- **Spec:** [DESIGN-session-memory.md §11](../docs/DESIGN-session-memory.md#11-project-metrics-correlation-kaizen-agentic)
+- **Contract (kaizen-agentic):** [Helix Forge Correlation Contract](https://gitea.coulomb.social/coulomb/kaizen-agentic/src/branch/main/docs/integrations/helix-forge-correlation.md)
+
+### Session-close env export
+
+After ingest has written the digest, agents using both layers export `HELIX_*`
+vars for `kaizen-agentic metrics record` to merge (names match ADR-004):
+
+`HELIX_SESSION_UID`, `HELIX_REPO`, `HELIX_FLAVOR`, `HELIX_TOKENS`,
+`HELIX_INFRA_OVERHEAD_SHARE`, and optionally `HELIX_STORE_DB` (absolute path to
+`mem.db`). See DESIGN §11.1 for field sources.
+
+### Read one digest (for `metrics correlate`)
+
+```bash
+python -m session_memory.digest_lookup claude:abc-123 --json
+HELIX_STORE_DB=/abs/path/to/mem.db python -m session_memory.digest_lookup <uid>
+```
+
+Defaults to `[store].db_path` in `config.toml`. Read-only — does not run ingest.
+
 ## Retention knobs (`[retention]` in config.toml)

 | Key | Meaning |
@@ -220,3 +254,7 @@ python -m pytest          # schema, adapters, store, digest, retention, ingest,
 - **Phase 4** (AGENTIC-WP-0009): Measure — fleet baseline/trend + before/after
  per-pattern effectiveness. The Capture → Detect → Curate → Distribute → Measure
  loop is closed.
+- **Weekly retro** (AGENTIC-WP-0010): windowed top-3-per-repo + hub `coding_retro`
+  publish.
+- **Kaizen correlation** (AGENTIC-WP-0011): bidirectional doc links, session-close
+  `HELIX_*` env convention, `digest_lookup` read path.
--- a/session_memory/digest_lookup.py
+++ b/session_memory/digest_lookup.py
@@ -0,0 +1,76 @@
+"""Read a single session digest from the local store (AGENTIC-WP-0011 T03).
+
+Thin read path for ``kaizen-agentic metrics correlate`` and other consumers.
+Does not run ingest.
+
+Usage:
+    python -m session_memory.digest_lookup <session_uid> [--json]
+    HELIX_STORE_DB=/abs/path/to/mem.db python -m session_memory.digest_lookup <uid>
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+
+from .core.store import Store
+from .ingest import _expand, load_config
+
+
+def resolve_store_paths(*, config_path: str | None = None) -> tuple[str, str]:
+    """Resolve db + blob paths from HELIX_STORE_DB or config.toml [store]."""
+    env_db = os.environ.get("HELIX_STORE_DB")
+    if env_db:
+        db_path = _expand(env_db)
+        blob_dir = os.path.join(os.path.dirname(db_path), "blobs")
+        return db_path, blob_dir
+
+    here = os.path.dirname(os.path.abspath(__file__))
+    cfg_path = config_path or os.path.join(here, "config.toml")
+    store_cfg = load_config(cfg_path).get("store", {})
+    return _expand(store_cfg.get("db_path", "session_memory/.store/mem.db")), _expand(
+        store_cfg.get("blob_dir", "session_memory/.store/blobs")
+    )
+
+
+def lookup_digest(session_uid: str, *, config_path: str | None = None) -> dict | None:
+    db_path, blob_dir = resolve_store_paths(config_path=config_path)
+    store = Store(db_path, blob_dir)
+    try:
+        return store.get_digest(session_uid)
+    finally:
+        store.close()
+
+
+def main(argv: list[str] | None = None) -> int:
+    here = os.path.dirname(os.path.abspath(__file__))
+    ap = argparse.ArgumentParser(
+        description="Read one session digest from the Helix Forge store (no ingest)."
+    )
+    ap.add_argument("session_uid", help="Normalized session uid, e.g. claude:abc-123")
+    ap.add_argument("--config", default=os.path.join(here, "config.toml"),
+                    help="config.toml when HELIX_STORE_DB is unset")
+    ap.add_argument("--json", action="store_true", help="print digest JSON to stdout")
+    args = ap.parse_args(argv)
+
+    digest = lookup_digest(args.session_uid, config_path=args.config)
+    if digest is None:
+        print(f"digest not found: {args.session_uid}", file=sys.stderr)
+        return 1
+
+    if args.json:
+        print(json.dumps(digest, indent=2, sort_keys=True))
+    else:
+        cost = digest.get("cost") or {}
+        tokens = cost.get("input_tokens", 0) + cost.get("output_tokens", 0)
+        print(f"session_uid: {digest.get('session_uid')}")
+        print(f"repo: {digest.get('repo')}  flavor: {digest.get('flavor')}")
+        print(f"outcome: {digest.get('outcome')}  tokens: {tokens}")
+        print(f"started_at: {digest.get('started_at')}  ended_at: {digest.get('ended_at')}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
--- a/tests/test_digest_lookup.py
+++ b/tests/test_digest_lookup.py
@@ -0,0 +1,78 @@
+"""digest_lookup entrypoint tests (AGENTIC-WP-0011 T03)."""
+
+import json
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from session_memory.core.store import Store  # noqa: E402
+from session_memory.digest_lookup import lookup_digest, main, resolve_store_paths  # noqa: E402
+
+
+def _write_config(tmp_path) -> str:
+    store = tmp_path / ".store"
+    toml = tmp_path / "config.toml"
+    toml.write_text(
+        f'[store]\ndb_path = "{store / "m.db"}"\nblob_dir = "{store / "blobs"}"\n'
+        f'cursor = "{store / "c.json"}"\n')
+    return str(toml), str(store)
+
+
+def _seed(store_dir, uid="claude:test-uid"):
+    st = Store(os.path.join(store_dir, "m.db"), os.path.join(store_dir, "blobs"))
+    st.write_digest(uid, {
+        "session_uid": uid,
+        "flavor": "claude",
+        "repo": "agentic-resources",
+        "outcome": "success",
+        "started_at": "2026-06-19T10:00:00Z",
+        "ended_at": "2026-06-19T11:00:00Z",
+        "cost": {"input_tokens": 100, "output_tokens": 25},
+        "tool_histogram": {"Bash": 10, "Edit": 5},
+    })
+    st.close()
+    return uid
+
+
+def test_resolve_store_paths_from_config(tmp_path):
+    cfg_path, store_dir = _write_config(tmp_path)
+    db, blob = resolve_store_paths(config_path=cfg_path)
+    assert db.endswith("m.db")
+    assert blob.endswith("blobs")
+    assert store_dir in db
+
+
+def test_resolve_store_paths_from_env(tmp_path, monkeypatch):
+    db = tmp_path / "custom" / "mem.db"
+    db.parent.mkdir(parents=True)
+    monkeypatch.setenv("HELIX_STORE_DB", str(db))
+    resolved_db, blob = resolve_store_paths()
+    assert resolved_db == str(db)
+    assert blob == str(tmp_path / "custom" / "blobs")
+
+
+def test_lookup_digest_found_and_missing(tmp_path):
+    cfg_path, store_dir = _write_config(tmp_path)
+    uid = _seed(store_dir)
+    found = lookup_digest(uid, config_path=cfg_path)
+    assert found is not None and found["outcome"] == "success"
+    assert lookup_digest("claude:missing", config_path=cfg_path) is None
+
+
+def test_main_json_success(tmp_path, capsys):
+    cfg_path, store_dir = _write_config(tmp_path)
+    uid = _seed(store_dir)
+    rc = main(["--config", cfg_path, uid, "--json"])
+    assert rc == 0
+    data = json.loads(capsys.readouterr().out)
+    assert data["session_uid"] == uid
+    assert data["repo"] == "agentic-resources"
+
+
+def test_main_not_found(tmp_path, capsys):
+    cfg_path, store_dir = _write_config(tmp_path)
+    _seed(store_dir)
+    rc = main(["--config", cfg_path, "claude:missing"])
+    assert rc == 1
+    assert "not found" in capsys.readouterr().err.lower()
--- a/workplans/AGENTIC-WP-0011-kaizen-correlation-followup.md
+++ b/workplans/AGENTIC-WP-0011-kaizen-correlation-followup.md
@@ -28,7 +28,7 @@ into Helix Forge. Authoritative cross-repo contract stays in kaizen-agentic:

 ```task
 id: AGENTIC-WP-0011-T01
-status: todo
+status: done
 priority: high
 ```

@@ -46,7 +46,7 @@ Complete the bidirectional documentation surface requested by kaizen-agentic:

 ```task
 id: AGENTIC-WP-0011-T02
-status: todo
+status: done
 priority: high
 ```

@@ -69,7 +69,7 @@ table — do not invent parallel names.

 ```task
 id: AGENTIC-WP-0011-T03
-status: todo
+status: done
 priority: medium
 ```