coulomb/infospace-bench
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

IB-WP-0020-T05: shadow-mode CLI flags; close IB-WP-0020

Browse Source 
Add --shadow-baseline <id> and --shadow-rate <float> opt-in flags to
generate run, generate resume, and generate from-source. When
--shadow-baseline names a candidate id from the routing config,
build_routing_policy_from_config wraps every other candidate in an
llm-connect ShadowingAdapter using that baseline plus a
PairedGrader(ExactMatchJudge()) and the workspace-resolved
QualityLedger. The baseline candidate itself is never wrapped — that
would shadow it against itself. --shadow-rate defaults to 0.1 when
--shadow-baseline is set; passing --shadow-rate without
--shadow-baseline fails fast with shadow_rate_without_baseline.
Setting --shadow-baseline without a ledger_path in the config fails
with missing_routing_ledger_for_shadow so observations have a place to
land before any call goes out.

run_generation grew shadow_baseline + shadow_rate kwargs and
_adapter_for("routing", ...) plumbs them into
build_routing_policy_from_config. The wrapped ShadowingAdapter slots
into the policy's prefer/fallback per task type via a
(candidate_id, task_type) reverse lookup, and adapters_by_id on the
adaptive policy gets the string-keyed entries.

Five new tests cover: shadow_rate without baseline fails fast, shadow
mode without a ledger fails fast, unknown shadow baseline id fails
fast, structural assertion that ShadowingAdapter wraps non-baseline
candidates and leaves the baseline raw, and a behavioural check that
shadow_rate=1.0 calls the baseline on every call while shadow_rate=0.0
skips entirely. Test forces async_shadow=False so the call counter is
deterministic.

Closes IB-WP-0020: T01-T05 all done. Workplan status flips from active
to finished. 179 tests pass, 2 skipped (both live OpenRouter smokes).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-05-18 23:30:36 +02:00
parent debd2b8e69
commit b0d67ae79e
5 changed files with 308 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
workplans/IB-WP-0020-provider-routing-cli.md
View File

@@ -4,7 +4,7 @@ type: workplan
title: "Provider Routing CLI Integration"
domain: markitect
repo: infospace-bench
status: active
status: finished
owner: markitect
topic_slug: markitect
created: "2026-05-18"
@@ -158,7 +158,7 @@ state_hub_task_id: "69288131-f265-4db5-a4b0-b0c8a6f55dd8"
```task
id: IB-WP-0020-T05
status: todo
status: done
priority: medium
state_hub_task_id: "02658420-056c-4d73-8055-e6a7ab51876b"
```