From b5c5ed88157829482c2e6c66c06dfe60b2755bd0 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 29 Mar 2026 16:26:24 +0200 Subject: [PATCH] chore(consistency): sync task status from DB [auto] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated by fix-consistency on 2026-03-29: - CUST-WP-0029-T01: todo → done --- ...CUST-WP-0029-token-consumption-tracking.md | 256 ++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 workplans/CUST-WP-0029-token-consumption-tracking.md diff --git a/workplans/CUST-WP-0029-token-consumption-tracking.md b/workplans/CUST-WP-0029-token-consumption-tracking.md new file mode 100644 index 0000000..a5e7841 --- /dev/null +++ b/workplans/CUST-WP-0029-token-consumption-tracking.md @@ -0,0 +1,256 @@ +--- +id: CUST-WP-0029 +type: workplan +title: "Token Consumption Tracking" +domain: custodian +repo: the-custodian +status: active +owner: custodian +topic_slug: custodian +created: "2026-03-29" +updated: "2026-03-29" +state_hub_workstream_id: "6f459d9f-b4d4-46d7-a5d7-d5f10721b29e" +--- + +# Token Consumption Tracking + +## Goal + +Record AI token consumption at task granularity and aggregate it up to +workstream, repo, commit, and release level. Makes agent work visible as a +cost/effort metric — reviewable alongside tasks, workplans, and releases. + +## Background + +Sessions produce no durable token signal today. Without it, there is no way to +ask "how expensive was WP-0003?" or "which repo consumes the most tokens per +release?". Token counts are the closest proxy for AI effort and cost that can +be captured without external instrumentation. + +**Reporting model:** agents self-report tokens when completing a task (alongside +`update_task_status`). The `ralph-workplan` skill is updated to pass token data +per iteration. Commit/release tagging is optional and manual. + +## Schema + +``` +token_events + id UUID PK + task_id UUID FK tasks (nullable) + workstream_id UUID FK workstreams (nullable) + repo_id UUID FK managed_repos (nullable) + session_id TEXT -- agent session identifier + model TEXT -- e.g. "claude-sonnet-4-6" + tokens_in INT NOT NULL + tokens_out INT NOT NULL + agent TEXT -- "custodian", "ralph", etc. + ref_type TEXT -- 'task'|'workstream'|'commit'|'release'|'session' + ref_id TEXT -- commit SHA, release tag, etc. + note TEXT + created_at TIMESTAMPTZ server_default=now() +``` + +Derived field `tokens_total = tokens_in + tokens_out` computed at query time. +Aggregation endpoint rolls up by any FK axis. + +## Exit Criteria + +- Token events can be recorded via MCP tool +- Aggregation queries work for task / workstream / repo / commit / release +- Dashboard page shows token spend by repo, workplan, model +- `ralph-workplan` logs a token event per completed task iteration +- All tests passing; consistency check clean + +--- + +## Tasks + +### T01 — Migration: token_events table + +```task +id: CUST-WP-0029-T01 +status: done +priority: high +state_hub_task_id: "5a758a61-4021-44e8-8f99-60a63cba6e50" +``` + +Write Alembic migration for `token_events`. Table as per schema above. +Indexes: `ix_token_events_task_id`, `ix_token_events_workstream_id`, +`ix_token_events_repo_id`, `ix_token_events_created_at`. + +`down_revision` = current Alembic head (check with `alembic heads`). + +**Exit criteria:** `make migrate` succeeds; table visible in psql. + +--- + +### T02 — Model and schema + +```task +id: CUST-WP-0029-T02 +status: todo +priority: high +state_hub_task_id: "57d71132-001a-4c85-bc39-2d20155c4971" +``` + +Add `api/models/token_event.py` (SQLAlchemy ORM, relationships to Task, +Workstream, ManagedRepo). Add `api/schemas/token_event.py`: + +- `TokenEventCreate` — input (task_id, workstream_id, repo_id all nullable; + tokens_in, tokens_out required; model, agent, ref_type, ref_id, note optional) +- `TokenEventRead` — full row + `tokens_total: int` computed field +- `TokenSummary` — aggregated view: + `{ scope, scope_id, tokens_in, tokens_out, tokens_total, event_count, + by_model: dict[str, int], by_agent: dict[str, int] }` + +Register model in `api/models/__init__.py`. + +**Exit criteria:** models import cleanly; `tokens_total` computed correctly. + +--- + +### T03 — Router: CRUD + aggregation + +```task +id: CUST-WP-0029-T03 +status: todo +priority: high +state_hub_task_id: "14604f26-5aa6-455d-b65d-0e7a4ba42509" +``` + +Add `api/routers/token_events.py`: + +- `POST /token-events/` — create event; auto-populate `workstream_id` from + task if not provided (look up `task.workstream_id`) +- `GET /token-events/` — list with filters: `task_id`, `workstream_id`, + `repo_id`, `ref_type`, `ref_id`, `model`, `agent`; default limit 100 +- `GET /token-events/summary/` — aggregation; required query param `scope` + (`task`|`workstream`|`repo`|`commit`|`release`) + `id` (the FK value or + ref_id). Returns `TokenSummary`. + +Register router in `api/main.py`. + +**Exit criteria:** all three endpoints return correct data; summary rolls up +`tokens_in`/`tokens_out` and breaks down by model and agent. + +--- + +### T04 — MCP tools: record_token_event and get_token_summary + +```task +id: CUST-WP-0029-T04 +status: todo +priority: high +state_hub_task_id: "e032aa47-2bf9-4cd7-982c-4d21c9d5e286" +``` + +Add two tools to `mcp_server/server.py`: + +**`record_token_event(tokens_in, tokens_out, task_id?, workstream_id?, +repo_id?, model?, agent?, ref_type?, ref_id?, note?, session_id?)`** +- POSTs to `/token-events/` +- Returns the created event id and running total for the task/workstream + +**`get_token_summary(scope, id)`** +- GETs `/token-events/summary/?scope=X&id=Y` +- Returns `TokenSummary` formatted as a readable table string + +Update `TOOLS.md` with both tools. + +**Exit criteria:** both tools callable from Claude Code MCP; record → read +round-trip works. + +--- + +### T05 — Enrich update_task_status with optional token passthrough + +```task +id: CUST-WP-0029-T05 +status: todo +priority: medium +state_hub_task_id: "0c95c442-0b03-4acc-aba7-a986fd006416" +``` + +Extend the existing `update_task_status` MCP tool signature to accept optional +`tokens_in: int` and `tokens_out: int` parameters. When provided, automatically +call `record_token_event` internally (with the task's workstream_id and repo_id +auto-populated). This lets agents report tokens in one call instead of two. + +Keep the parameters optional — existing callers are unaffected. + +**Exit criteria:** `update_task_status(task_id, status='done', tokens_in=1200, +tokens_out=800)` creates both a status update and a token event. + +--- + +### T06 — Dashboard: token cost page + +```task +id: CUST-WP-0029-T06 +status: todo +priority: medium +state_hub_task_id: "02cc5d8e-a9da-4fb3-9c39-fdc05812d8d0" +``` + +Add `dashboard/src/token-cost.md` Observable page: + +- **By repo bar chart** — total tokens per repo (stacked in/out) +- **By workplan table** — workstream slug, title, tokens_total, event_count, + dominant model +- **By model breakdown** — pie or bar; shows model mix across all events +- **Top 10 tasks by tokens** — useful for identifying expensive tasks + +Data loader: `dashboard/src/data/token-summary.json.py` — calls +`GET /token-events/summary/` for each repo and workstream. + +Add page to `observablehq.config.js` nav under "Analytics". + +**Exit criteria:** page renders with real data; updates on refresh. + +--- + +### T07 — ralph-workplan: log token event per completed task + +```task +id: CUST-WP-0029-T07 +status: todo +priority: medium +state_hub_task_id: "676f2a94-b5a5-467d-8dd4-889817acb159" +``` + +Update `~/.claude/plugins/ralph-workplan/ralph-workplan.md` and +`~/.claude/commands/ralph-workplan.md`: + +Add a note in **Step 5** (loop active) instructing the agent that each time a +task is marked done, it should report tokens via `update_task_status` (with +`tokens_in`/`tokens_out`) or a standalone `record_token_event` call. + +Guidance: estimate tokens from the Claude Code status bar (input/output shown +at session end) or use a rough per-task heuristic (1000 in / 500 out) when +exact counts are unavailable. Log `model` from the known session model +(`claude-sonnet-4-6` by default). + +**Exit criteria:** both skill files updated; guidance is clear and actionable +for an agent running a loop. + +--- + +### T08 — Tests and consistency gate + +```task +id: CUST-WP-0029-T08 +status: todo +priority: high +state_hub_task_id: "a3627144-9d98-4a3b-aa64-3079fd087448" +``` + +Add tests to `state-hub/tests/`: + +- `test_token_events.py`: create event, list with filter, summary aggregation + (single task, cross-workstream rollup, by-model breakdown) +- `test_token_passthrough.py`: `update_task_status` with tokens creates event + +Run `make test`. Run `make fix-consistency REPO=the-custodian`. + +**Exit criteria:** all new tests pass; consistency check clean; Alembic head +matches DB.