coulomb/infospace-bench
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

IB-WP-0019-T03: rate-table cost computation

Browse Source 
Ship a starter model rate table at src/infospace_bench/model_rates.yaml
(prompt_per_1k / completion_per_1k for the OpenRouter models we have
actually touched: gpt-4o, gpt-4o-mini, gpt-4-turbo, claude 3.5 sonnet
and haiku, claude 3 opus, gemini 1.5 flash/pro, llama 3.1 70b) and a
load_rate_table() / estimate_cost_usd() pair that overlays an optional
<workspace>/model-rates.yaml on top of the bundled defaults.

generate run now passes a workspace-aware cost_resolver into
record_run_usage, so cost_usd_estimated lands on every usage bucket
whose model matches the table. Adapter-returned cost still wins
(cost_status="known"); rate-table cost is reported under
cost_status="estimated"; unmatched models are recorded as
cost_status="unknown" rather than silently zeroed. Rate-table file is
listed in pyproject.toml package-data so pip-installed users keep the
defaults.

106 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-05-17 19:54:30 +02:00
parent 678508226a
commit a4dde53fc3
7 changed files with 252 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/generic-source-generator.md
View File

@@ -48,6 +48,29 @@ infospace-bench generate status ./infospaces/book-space
shows chunk counts, generated artifact counts, evaluations, metrics, history,
and stale source/profile inputs.
### Budget and usage registry
Every `generate plan` invocation appends a compact snapshot to
`output/budget/plans.yaml` (deterministic 12-char `snapshot_id`, 50-entry
sliding retention). Every `generate run` invocation appends a usage
rollup to `output/budget/usage.yaml`, bucketed by `(workflow_id,
stage_id, provider, model)` with prompt and completion token counts,
known cost (when the adapter returned it), and estimated cost (when a
rate table entry matches the model).
The default rate table is bundled at
`src/infospace_bench/model_rates.yaml` and covers a handful of common
OpenRouter models at list price (see the file for the captured-at
timestamp). A workspace can override or extend entries by placing
`model-rates.yaml` next to its `infospaces/` directory; the workspace
file is overlaid on top of the package default so partial overrides
are fine.
Cost resolution order on each run: adapter-returned `cost` first, then
the rate table, then `cost_status="unknown"` (recorded explicitly,
never silently zeroed). The plan-vs-actual variance summary lands in
follow-on task T04.
### Profiles
Two profiles ship today: