infospace-bench/docs/routing-task-types.md

Task-Type Taxonomy for Routing

Workplan: IB-WP-0018 (T01) Depends on: llm-connect LLM-WP-0004 (RoutingPolicy, AdaptiveRoutingPolicy)

This file names the task types that infospace-bench emits when it routes each generation stage through llm-connect. The names are the consumer side of LLM-WP-0004's scope guardrail: llm-connect ships the routing primitives, infospace-bench owns the taxonomy.

Default identity mapping

RoutingAssistedGenerationAdapter (see src/infospace_bench/routing.py) maps stage ids to task types using the identity mapping below by default. Callers override individual entries via RoutingAssistedGenerationAdapter(..., stage_to_task_type={...}).

Stage id	Task type	Notes
summarize-source	summarize-source	One call per source chunk. Cheap, high-volume; small models usually clear the bar.
extract-entities	extract-entities	One call per source chunk. Quality matters most here — bad extractions cascade.
extract-relations	extract-relations	One call per source chunk. Quality close to extraction; relations rely on entity titles being stable.
evaluate-entity	evaluate-entity	One call per generated entity. Cheap, often a different model than extraction to avoid self-grading.
synthesize-report	synthesize-report	One call at the end. Volume-of-one; quality matters; cost negligible.

Quality expectations

AdaptiveRoutingPolicy.resolve(task_type, quality_floor=...) picks the cheapest adapter whose ledger-observed mean quality clears the floor. The recommended starting floors:

Task type	Quality floor	Rationale
summarize-source	0.70	Summaries are intermediate. Slight quality loss is recoverable downstream.
extract-entities	0.85	Entities are the durable output. Be strict.
extract-relations	0.80	Relations depend on entities; slightly looser is OK as long as evidence is intact.
evaluate-entity	0.80	Judge-level reliability. Self-grading bias is more of a concern than absolute score.
synthesize-report	0.70	The report is a review surface; tolerate looser language for cheaper models.

These are starting points. Bind them at the calling site (RoutingAssistedGenerationAdapter(..., quality_floor=0.85) for extraction stages) — they are not enforced by this taxonomy.

Common overrides

Callers may want to collapse task types to share observations across related stages, or split a task type to pin a specific model to a narrow workload. Two illustrative overrides:

# Collapse extraction stages so a single ledger drives both
stage_to_task_type = {
    "extract-entities": "extraction",
    "extract-relations": "extraction",
}

# Split entity evaluation by category — useful when a profile has very
# different quality bars for different entity categories (e.g.
# trading-literature's `evidence_bearing_claim` is harder to judge than
# `instrument`).
stage_to_task_type = {
    "evaluate-entity": "judge",
}

Anything not in the override map falls through to the identity mapping.

What this taxonomy does NOT decide

• Which adapter ships per task type. That belongs to the caller's RoutingPolicy rule list.
• Where the quality ledger lives. Caller-supplied path on the AdaptiveRoutingPolicy.
• When to refresh observations. Caller decides via the ledger's TTL helpers in llm-connect.
• What a quality score means. Each judge defines its own.