tegwick
80c60ebd7a
Add kaizen-agentic feedback CLI, Gitea issue templates, CI workflow, pre-commit hooks, FEEDBACK/TELEMETRY docs, and cross-platform path tests. Improve CLI registry error messages; remove agents_backup scaffolding. Apply black formatting across src/tests for CI consistency. State Hub message sent to agentic-resources for Helix correlation doc link.
1.8 KiB
Telemetry and Agent Effectiveness Tracking
WP-0001 T04 design — aligned with ADR-004 and WP-0004 ecosystem integration.
Two layers (do not merge)
| Layer | Question | Mechanism |
|---|---|---|
| Project | How is agent X performing in this repo? | kaizen-agentic metrics record → .kaizen/metrics/ |
| Fleet | How are coding sessions performing across repos? | agentic-resources Helix Forge |
kaizen-agentic does not ship a parallel session transcript ingestion pipeline.
Project telemetry (implemented)
Memory-enabled agents record per-session outcomes at close:
kaizen-agentic metrics record <agent> --success --time <s> --quality <0-1>
kaizen-agentic metrics optimize [agent]
kaizen-agentic memory brief <agent> # includes Performance Summary
Optional fleet correlation via HELIX_SESSION_UID (see
integrations/helix-forge-correlation.md).
Fleet telemetry (agentic-resources)
Helix Forge owns session capture, digest storage, baselines, and weekly retro. kaizen-agentic consumes correlation fields only.
CLI install / usage analytics (future)
Potential v1.1 additions (not yet implemented):
- Opt-in anonymous counters on
install/memory init(no PII, no project paths) - Aggregate effectiveness reports via
metrics listacross a monorepo checkout
tele-mcp evaluation (deferred)
tele-mcp is a candidate MCP adapter for IDE-level telemetry (WP-0001 note). Assess before depending on it. Project and fleet layers above satisfy INTENT's "measurable agents" requirement without tele-mcp.
Feedback loop
User experience feedback uses FEEDBACK.md and Gitea issue templates — separate from execution metrics.