generated from coulomb/repo-seed
IB-WP-0018-T03+T04: shadow sampling + report/CLI surfacing; close IB-WP-0018
T03 — wrap_with_shadow_sampling() helper in routing.py: builds a llm-connect ShadowingAdapter around any candidate LLMAdapter with a caller-supplied baseline, grader, and QualityLedger. async_shadow=True by default so production load is not doubled; on_shadow_error escape hatch keeps caller logs informed when a baseline outage swallows the shadow path. The returned adapter is still an LLMAdapter so it slots into a RoutingPolicy rule without further code change. T04 — generation report enrichment plus a small CLI helper: - _collect_adapter_choices walks artifact provenance, groups by (stage_id, adapter_id), and surfaces calls + prompt/completion tokens per (stage, adapter) pair in a new ## Per-stage adapter choices section. Runs that did not go through the bridge have no provider_metadata.adapter_id and emit an empty list, so fixture-only reports stay terse. - summarise_quality_ledger() rolls a llm-connect QualityLedger up by (task_type, adapter_id) with mean quality, mean cost, observations, and cumulative tokens. - infospace-bench routing ledger <path> CLI prints the rollup as JSON. Five new tests cover shadow happy-path, shadow failure isolation, ledger rollup, the routing CLI, and the report's adapter-choice aggregation. Closes IB-WP-0018: T01-T05 are all done and the workplan status flips from blocked to done now that LLM-WP-0004's primitives have shipped. 144 tests pass, 1 skipped (the OpenRouter live smoke, gated as before). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -4,11 +4,11 @@ type: workplan
|
||||
title: "Adaptive LLM Routing — infospace-bench Consumer Wiring"
|
||||
domain: markitect
|
||||
repo: infospace-bench
|
||||
status: blocked
|
||||
status: done
|
||||
owner: markitect
|
||||
topic_slug: markitect
|
||||
created: "2026-05-17"
|
||||
updated: "2026-05-17"
|
||||
updated: "2026-05-18"
|
||||
depends_on_workplans:
|
||||
- LLM-WP-0004
|
||||
related_workplans:
|
||||
@@ -33,7 +33,22 @@ list will be refined once that API is stable.
|
||||
|
||||
## Status
|
||||
|
||||
Blocked on `LLM-WP-0004` T01..T03.
|
||||
Done. LLM-WP-0004 landed `QualityLedger`, `QualityObservation`,
|
||||
`BaselineGrader`/`PairedGrader`/`ExactMatchJudge`/`EmbeddingSimilarityJudge`/
|
||||
`LLMJudge`, `AdaptiveRoutingPolicy`, and `ShadowingAdapter` in
|
||||
llm-connect; the five tasks below are all complete.
|
||||
|
||||
- T01 — task-type taxonomy (`docs/routing-task-types.md`)
|
||||
- T02 — `RoutingAssistedGenerationAdapter` bridge in
|
||||
`src/infospace_bench/routing.py`
|
||||
- T03 — `wrap_with_shadow_sampling()` helper that opt-in installs
|
||||
llm-connect's `ShadowingAdapter` around any candidate
|
||||
- T04 — `## Per-stage adapter choices` section in
|
||||
`reports/generation-summary.md` (driven from artifact
|
||||
`provenance.provider_metadata`) and `infospace-bench routing ledger`
|
||||
CLI subcommand
|
||||
- T05 — `tests/test_routing_adapter.py` (13 tests, including a CLI
|
||||
smoke and the adapter-choices unit test)
|
||||
|
||||
## Why this is a separate workplan
|
||||
|
||||
|
||||
Reference in New Issue
Block a user