tegwick
1641a3165d
Publish activity-core EventEnvelope payloads to NATS subject activity.kaizen.metrics.recorded after a successful append. Optional nats-py via kaizen-agentic[events]; project slug from KAIZEN_PROJECT_SLUG or directory basename. Skips emit on idempotency duplicates. Closes KAIZEN-WP-0008 T03.
1.9 KiB
Telemetry and Agent Effectiveness Tracking
WP-0001 T04 design — aligned with ADR-004 and WP-0004 ecosystem integration.
Two layers (do not merge)
| Layer | Question | Mechanism |
|---|---|---|
| Project | How is agent X performing in this repo? | kaizen-agentic metrics record → .kaizen/metrics/ |
| Fleet | How are coding sessions performing across repos? | agentic-resources Helix Forge |
kaizen-agentic does not ship a parallel session transcript ingestion pipeline.
Project telemetry (implemented)
Memory-enabled agents record per-session outcomes at close:
kaizen-agentic metrics record <agent> --success --time <s> --quality <0-1>
kaizen-agentic metrics record <agent> --success --time <s> --quality <0-1> --emit-event
kaizen-agentic metrics optimize [agent]
kaizen-agentic memory brief <agent> # includes Performance Summary
Optional fleet correlation via HELIX_SESSION_UID (see
integrations/helix-forge-correlation.md).
Fleet telemetry (agentic-resources)
Helix Forge owns session capture, digest storage, baselines, and weekly retro. kaizen-agentic consumes correlation fields only.
CLI install / usage analytics (future)
Potential v1.1 additions (not yet implemented):
- Opt-in anonymous counters on
install/memory init(no PII, no project paths) - Aggregate effectiveness reports via
metrics listacross a monorepo checkout
tele-mcp evaluation (deferred)
tele-mcp is a candidate MCP adapter for IDE-level telemetry (WP-0001 note). Assess before depending on it. Project and fleet layers above satisfy INTENT's "measurable agents" requirement without tele-mcp.
Feedback loop
User experience feedback uses FEEDBACK.md and Gitea issue templates — separate from execution metrics.