fix: review findings from Lefevre live smoke

Two small fixes informed by the 2026-05-18 live OpenRouter chapter-I run.

1. extract-entities templates (trading-literature and general-knowledge):
   the # Entity Title placeholder was interpreted by gpt-4o-mini as a
   literal heading prefix, so every entity came back as "# Entity Title:
   Bucket Shop" etc. The instruction now spells the placeholder out
   with concrete examples and an explicit "not the literal string"
   note, so smaller models hit the intended shape.

2. generate plan grows --model <id>. When supplied, the cost estimate
   pulls per-prompt and per-completion rates from the bundled
   model_rates.yaml instead of multiplying a single blended
   --cost-per-1k value across all tokens. The summary now also returns
   a separate estimated_completion_tokens field plus a cost_source tag
   ("rate_table:<model>" | "cost_per_1k_blended" | None).

This is a stopgap. LLM-WP-0005 (proposed in llm-connect this round)
will move the rate registry and token-shape problem classes upstream
so consumers stop re-implementing them.

The live smoke ran 28k prompt tokens / 7.5k completion / $0.0088
actual. With --model openai/gpt-4o-mini the plan estimate now lands at
$0.0076 (within 14% of actual) versus the prior $8.40 estimate at
--cost-per-1k 0.30.

181 tests pass, 2 skipped (both live OpenRouter smokes).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-19 04:30:33 +02:00
parent 9404831069
commit 3ca891de4a
5 changed files with 80 additions and 7 deletions

View File

@@ -115,6 +115,45 @@ def test_plan_caps_flag_when_estimate_exceeds_budget(tmp_path: Path) -> None:
assert summary["exceeds_cost_cap"] is True
def test_plan_with_model_uses_rate_table_instead_of_blended_per_1k(tmp_path: Path) -> None:
"""--model openai/gpt-4o-mini should pull from bundled rate table.
Stopgap until LLM-WP-0005 lands a proper cost model in llm-connect.
"""
root = _build_plan_infospace(tmp_path)
blended = plan_generation_summary(
root, cost_per_1k_tokens=0.30, persist=False
) if False else None
rate_table = plan_generation_summary(
root, model="openai/gpt-4o-mini"
)
# gpt-4o-mini list price is ~0.00015/1k prompt + ~0.0006/1k completion,
# so the rate-table cost must be far below the $0.30/1k blended figure.
assert rate_table["cost_source"] == "rate_table:openai/gpt-4o-mini"
assert rate_table["estimated_cost_usd"] is not None
assert rate_table["estimated_cost_usd"] < 0.10, (
"rate-table estimate must be far below a $0.30/1k blended rate"
)
# The estimator now also returns a completion-token estimate.
assert rate_table["estimated_completion_tokens"] > 0
def test_plan_with_unknown_model_falls_back_to_blended_or_unknown(tmp_path: Path) -> None:
root = _build_plan_infospace(tmp_path)
no_signal = plan_generation_summary(root, model="acme/not-in-rate-table")
blended = plan_generation_summary(
root, model="acme/not-in-rate-table", cost_per_1k_tokens=0.5
)
assert no_signal["estimated_cost_usd"] is None
assert no_signal["cost_source"] is None
assert blended["estimated_cost_usd"] is not None
assert blended["cost_source"] == "cost_per_1k_blended"
def test_plan_full_mode_includes_workflow_plans(tmp_path: Path) -> None:
root = _build_plan_infospace(tmp_path)