infospace-bench/docs/generic-source-generator.md

Generic Source Generator

Date: 2026-05-14

Purpose

infospace-bench generate turns a local article, ebook-like file, or folder of knowledge sources into a manifest-backed infospace. It generalizes the Wealth/VSM pilot into an explicit workflow path with deterministic fixture support and an optional OpenRouter provider.

Deterministic Run

Use fixture responses for repeatable tests and demos:

infospace-bench generate from-source ./examples/article.md \
  --workspace . \
  --slug article-space \
  --name "Article Space" \
  --profile general-knowledge \
  --fixture-responses ./examples/responses.yaml \
  --apply

The command creates normalized source chunks, installs the selected profile, runs the declared workflows, writes entities, relations, evaluations, metrics, history, and a generation report, then registers artifacts in artifacts/index.yaml.

Stepwise Workflow

infospace-bench generate init ./book.epub \
  --workspace . \
  --slug book-space \
  --name "Book Space" \
  --profile general-knowledge \
  --max-chunks 3

infospace-bench generate plan ./infospaces/book-space --stage all
infospace-bench generate run ./infospaces/book-space \
  --fixture-responses ./responses.yaml
infospace-bench generate status ./infospaces/book-space

--max-chunks caps early experiments and provider cost. generate status shows chunk counts, generated artifact counts, evaluations, metrics, history, and stale source/profile inputs.

Live OpenRouter runs (handle with care)

A single-chapter live run is the only OpenRouter shape the test suite covers today. Use --chapter (or --from-chapter / --to-chapter) on generate init or generate from-source to scope what gets registered before any provider calls happen:

export OPENROUTER_API_KEY=...

# Preview the cost first
infospace-bench generate plan ./infospaces/foo --chapter I --cost-per-1k 0.30

# Run only Chapter I against a cheap model
infospace-bench generate from-source ./LEFEVRE.epub \
  --workspace ./infospaces \
  --slug reminiscences-ch1 \
  --name "Reminiscences (Ch I)" \
  --profile trading-literature \
  --provider openrouter \
  --model openai/gpt-4o-mini \
  --chapter I \
  --apply

output/budget/plans.yaml, usage.yaml, and summary.yaml record what was estimated, what was actually spent, and the plan-vs-actual delta. output/workflows/runs/*.yaml carry the OpenRouter request_id, model, token usage, retry count, and per-call duration; the same metadata reaches the entity/relation/evaluation artifacts via provenance.provider_metadata.

Before scaling to the full book:

• Inspect each chapter's outputs and generation-summary.md
• Multiply the per-chapter total_provider_calls_estimate and estimated_cost_usd by the chapter count and compare to your budget
• Decide on a final model and confirm the rate-table entry exists in src/infospace_bench/model_rates.yaml or your workspace override

The optional live-smoke test in tests/test_openrouter_live.py is skipped unless both OPENROUTER_API_KEY and INFOSPACE_BENCH_ENABLE_LIVE_OPENROUTER=1 are set. It runs a single chapter through the same path and asserts the provider metadata plumb-through.

Live runs with --provider routing

When the routing CLI is what you want to exercise live, swap --provider openrouter --model ... for the routing pair:

infospace-bench generate from-source ./LEFEVRE.epub \
  --workspace ./infospaces \
  --slug reminiscences-routed \
  --name "Reminiscences (Routed)" \
  --profile trading-literature \
  --provider routing \
  --routing-config ./examples/routing/trading-literature.yaml \
  --chapter I \
  --apply

examples/routing/trading-literature.yaml is a checked-in starting config: cheap candidates for summary/evaluation, smart candidates for entity/relation, a claude_code baseline rule for future shadow sampling, and a workspace-relative output/routing/quality.jsonl ledger so adaptive observations stay with the workspace.

--quality-floor <float> on the same command overrides the config's default_quality_floor for a single invocation — useful for tightening the bar for a specific run without editing the file. The ledger fills up as the AdaptiveRoutingPolicy records each observation; later runs against the same workspace get the benefit without re-grading from scratch.

The parallel live-smoke test (test_provider_routing_one_chapter_live_smoke) is also gated on INFOSPACE_BENCH_ENABLE_LIVE_OPENROUTER=1 + OPENROUTER_API_KEY and asserts the per-stage adapter-choices report section names the routed model.

Budget and usage registry

Every generate plan invocation appends a compact snapshot to output/budget/plans.yaml (deterministic 12-char snapshot_id, 50-entry sliding retention). Every generate run invocation appends a usage rollup to output/budget/usage.yaml, bucketed by (workflow_id, stage_id, provider, model) with prompt and completion token counts, known cost (when the adapter returned it), and estimated cost (when a rate table entry matches the model).

The default rate table is bundled at src/infospace_bench/model_rates.yaml and covers a handful of common OpenRouter models at list price (see the file for the captured-at timestamp). A workspace can override or extend entries by placing model-rates.yaml next to its infospaces/ directory; the workspace file is overlaid on top of the package default so partial overrides are fine.

Cost resolution order on each run: adapter-returned cost first, then the rate table, then cost_status="unknown" (recorded explicitly, never silently zeroed). The plan-vs-actual variance summary lands in follow-on task T04.

Profiles

Two profiles ship today:

• general-knowledge — durable concepts, claims, methods, people, places, works, and objects across any source
• trading-literature — trading memoirs and market-structure texts; tunes entity categories (trader, market, strategy, error, psychological_pattern, institution, instrument, evidence_bearing_claim), relation types (cause_effect, lesson_evidence, risk_mitigation, actor_venue, strategy_outcome), and evaluation criteria (groundedness, lesson_clarity, historical_context, overgeneralization_risk)

Select via --profile trading-literature on generate init or generate from-source. The generic profile remains the default.

Scale-aware plan

generate plan returns a compact estimate by default — counts of selected chunks, calls per workflow, prompt-word and token estimates, and a rough USD cost when --cost-per-1k is supplied. Long corpora no longer dump hundreds of full prompts unless --full is set.

infospace-bench generate plan ./infospaces/book-space \
  --from-chapter 1 --to-chapter 3 \
  --cost-per-1k 0.30 \
  --max-calls 50 \
  --cost-cap 2.00

Selection filters:

• --chapter LABEL (repeatable) — match a chapter by roman/arabic label or numeric value (e.g. --chapter I or --chapter 2)
• --from-chapter N / --to-chapter N — numeric chapter range
• --chunk ID (repeatable) — exact source chunk id (e.g. chapter-01-part-002)

Budget flags --max-calls and --cost-cap are reported as exceeds_max_calls / exceeds_cost_cap booleans in the summary, so a caller can fail fast before invoking run. Use --full to opt back into the full per-workflow plan with prompts for deep inspection.

OpenRouter

Live model calls are explicit:

export OPENROUTER_API_KEY=...

infospace-bench generate run ./infospaces/book-space \
  --provider openrouter \
  --model openai/gpt-4o-mini \
  --stage all

Choose the --model value from OpenRouter model IDs. The API key is read from OPENROUTER_API_KEY; it is not written to infospace.yaml. Default tests never make live provider calls.

Resume

Use resume for interrupted or reviewed runs:

infospace-bench generate resume ./infospaces/book-space \
  --provider openrouter \
  --model openai/gpt-4o-mini

Unchanged completed runs are skipped. Use --force when you intentionally want to rerun completed work. Stale status is reported when source artifact digests or installed profile/template files change.

Review Path

After generation:

• inspect artifacts/sources/ for normalized input chunks
• inspect artifacts/entities/ and artifacts/relations/ for generated claims
• inspect output/evaluations/ for rubric output
• run infospace-bench validate <root> and infospace-bench graph <root>
• review reports/generation-summary.md

Move from the generic profile to a specialized profile when the source domain needs stricter terminology, narrower extraction granularity, or a discipline lens such as VSM.