IB-WP-0018-T03+T04: shadow sampling + report/CLI surfacing; close IB-WP-0018

T03 — wrap_with_shadow_sampling() helper in routing.py: builds a
llm-connect ShadowingAdapter around any candidate LLMAdapter with a
caller-supplied baseline, grader, and QualityLedger. async_shadow=True
by default so production load is not doubled; on_shadow_error escape
hatch keeps caller logs informed when a baseline outage swallows the
shadow path. The returned adapter is still an LLMAdapter so it slots
into a RoutingPolicy rule without further code change.

T04 — generation report enrichment plus a small CLI helper:

- _collect_adapter_choices walks artifact provenance, groups by
  (stage_id, adapter_id), and surfaces calls + prompt/completion tokens
  per (stage, adapter) pair in a new ## Per-stage adapter choices
  section. Runs that did not go through the bridge have no
  provider_metadata.adapter_id and emit an empty list, so fixture-only
  reports stay terse.
- summarise_quality_ledger() rolls a llm-connect QualityLedger up by
  (task_type, adapter_id) with mean quality, mean cost, observations,
  and cumulative tokens.
- infospace-bench routing ledger <path> CLI prints the rollup as JSON.

Five new tests cover shadow happy-path, shadow failure isolation,
ledger rollup, the routing CLI, and the report's adapter-choice
aggregation. Closes IB-WP-0018: T01-T05 are all done and the workplan
status flips from blocked to done now that LLM-WP-0004's primitives
have shipped.

144 tests pass, 1 skipped (the OpenRouter live smoke, gated as before).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-18 11:52:05 +02:00
parent 0a83e908ce
commit f818acfc62
5 changed files with 365 additions and 3 deletions

View File

@@ -4,11 +4,11 @@ type: workplan
title: "Adaptive LLM Routing — infospace-bench Consumer Wiring"
domain: markitect
repo: infospace-bench
status: blocked
status: done
owner: markitect
topic_slug: markitect
created: "2026-05-17"
updated: "2026-05-17"
updated: "2026-05-18"
depends_on_workplans:
- LLM-WP-0004
related_workplans:
@@ -33,7 +33,22 @@ list will be refined once that API is stable.
## Status
Blocked on `LLM-WP-0004` T01..T03.
Done. LLM-WP-0004 landed `QualityLedger`, `QualityObservation`,
`BaselineGrader`/`PairedGrader`/`ExactMatchJudge`/`EmbeddingSimilarityJudge`/
`LLMJudge`, `AdaptiveRoutingPolicy`, and `ShadowingAdapter` in
llm-connect; the five tasks below are all complete.
- T01 — task-type taxonomy (`docs/routing-task-types.md`)
- T02 — `RoutingAssistedGenerationAdapter` bridge in
`src/infospace_bench/routing.py`
- T03 — `wrap_with_shadow_sampling()` helper that opt-in installs
llm-connect's `ShadowingAdapter` around any candidate
- T04 — `## Per-stage adapter choices` section in
`reports/generation-summary.md` (driven from artifact
`provenance.provider_metadata`) and `infospace-bench routing ledger`
CLI subcommand
- T05 — `tests/test_routing_adapter.py` (13 tests, including a CLI
smoke and the adapter-choices unit test)
## Why this is a separate workplan