Fixed and improved token tracking

docs/token-evidence-model.md

@@ -0,0 +1,57 @@
+# Token Evidence Model
+
+State Hub token events distinguish source-backed measurements from inferred
+operational signals. Dashboards and reports should use structured fields for
+quality and provenance; `note` remains human context only.
+
+## Measurement Kinds
+
+| Kind | Meaning | Default confidence |
+| --- | --- | --- |
+| `measured` | Parsed from a source that reports usage metadata, such as Codex session logs or Claude transcript usage blocks. | `1.0` |
+| `allocated` | A share of a larger known total, assigned to a task/workstream by a documented allocation method. | `0.70` |
+| `estimated` | A fallback or operator-entered estimate without direct source evidence. | `0.35` |
+| `superseded` | Historical rows retained for audit but excluded from active totals. | `0.0` |
+
+## Source Providers
+
+| Provider | Source |
+| --- | --- |
+| `codex_session` | Codex Desktop `.codex/sessions/**` and `.codex/archived_sessions/**` JSONL token_count events. |
+| `claude_transcript` | Claude Code `.claude/projects/**/*.jsonl` usage metadata. Transcript text is never stored. |
+| `llm_connect` | Future llm-connect usage metadata. |
+| `manual` | Explicit operator/API input. |
+| `task_fallback` | Fixed task-completion fallback rows created when no source data is available. |
+
+## Provenance Fields
+
+Each source-backed row should include:
+
+- `source_provider`, `source_id`, `source_path`, `source_created_at`
+- `parser_version`, `ingested_at`, `confidence`
+- `cached_input_tokens`, `reasoning_output_tokens`, `raw_total_tokens`
+- `raw_metadata` with parser and attribution metadata, never transcript content
+
+`tokens_in + tokens_out` remains the default active total. Cached input and
+reasoning output are preserved separately so dashboards can show both default
+and provider-style totals without rewriting history.
+
+## Idempotency
+
+Measured sources must be written with a stable `source_id`. State Hub enforces
+one row for each `(measurement_kind, source_provider, source_id)` tuple and
+`POST /token-events/upsert` updates a growing live session rather than creating
+duplicates.
+
+## Migration Playbook
+
+1. Run the token-event provenance migration.
+2. Run `python3 scripts/token_reconcile.py --since 2026-05-19` and inspect the
+   dry-run report.
+3. Run `python3 scripts/token_reconcile.py --since 2026-05-19 --apply` to
+   upsert measured Codex/Claude source rows.
+4. Run the same command with `--zero-superseded-fallbacks` only after measured
+   source rows cover the affected window.
+5. Check `/token-events/quality/` or the Token Cost dashboard for fallback,
+   missing-provenance, duplicate-source, and unattributed measured signals.
+6. Keep historical fallback rows as `superseded`; do not delete them.