coulomb/infospace-bench
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

IB-WP-0020-T04: example routing config + live routing smoke

Browse Source 
examples/routing/trading-literature.yaml is the checked-in starting
config for a Lefevre-style run. It applies the IB-WP-0018 task-type
taxonomy: cheap candidates for summary + evaluation, smart candidates
for entity + relation extraction, and a separate baseline rule wiring
claude_code for a follow-on T05 ShadowingAdapter step. Workspace-
relative ledger_path keeps adaptive observations with the workspace.

tests/test_routing_config.py gains a regression test that asserts the
shipped example parses cleanly, every stage in stage_to_task_type maps
to a declared task type, and the baseline candidate uses the
claude_code provider — so the example will not bit-rot silently.

tests/test_openrouter_live.py gains test_provider_routing_one_chapter_live_smoke
gated on the same INFOSPACE_BENCH_ENABLE_LIVE_OPENROUTER + OPENROUTER_API_KEY
opt-in as the existing static smoke. It builds a one-candidate routing
config, runs a single chapter through --provider routing, and asserts
the per-stage adapter-choices report section names the routed model
and the routed artifacts carry adapter_id provenance.

docs/generic-source-generator.md gains a "Live runs with --provider
routing" subsection that walks through the one-command routed run,
explains the --quality-floor override, and points at the parallel
live smoke test.

174 tests pass, 2 skipped (both live smokes, correctly gated).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-05-18 22:19:54 +02:00
parent d3562454d7
commit debd2b8e69
5 changed files with 221 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
docs/generic-source-generator.md
View File

@@ -94,6 +94,42 @@ skipped unless both `OPENROUTER_API_KEY` and
chapter through the same path and asserts the provider metadata
plumb-through.
### Live runs with `--provider routing`
When the routing CLI is what you want to exercise live, swap
`--provider openrouter --model ...` for the routing pair:
```bash
infospace-bench generate from-source ./LEFEVRE.epub \
--workspace ./infospaces \
--slug reminiscences-routed \
--name "Reminiscences (Routed)" \
--profile trading-literature \
--provider routing \
--routing-config ./examples/routing/trading-literature.yaml \
--chapter I \
--apply
```
`examples/routing/trading-literature.yaml` is a checked-in starting
config: cheap candidates for summary/evaluation, smart candidates for
entity/relation, a `claude_code` baseline rule for future shadow
sampling, and a workspace-relative `output/routing/quality.jsonl`
ledger so adaptive observations stay with the workspace.
`--quality-floor <float>` on the same command overrides the config's
`default_quality_floor` for a single invocation — useful for
tightening the bar for a specific run without editing the file. The
ledger fills up as the `AdaptiveRoutingPolicy` records each
observation; later runs against the same workspace get the benefit
without re-grading from scratch.
The parallel live-smoke test
(`test_provider_routing_one_chapter_live_smoke`) is also gated on
`INFOSPACE_BENCH_ENABLE_LIVE_OPENROUTER=1` + `OPENROUTER_API_KEY` and
asserts the per-stage adapter-choices report section names the routed
model.
### Budget and usage registry
Every `generate plan` invocation appends a compact snapshot to