Three coordinated changes that let the pipeline produce a clean
chapter-by-chapter git history on long texts without archaeology after
the fact.
1. Richer commit messages. `SourcePipeline._git_commit` now diffs the
staged changes, buckets added files by output subdirectory (entities,
evaluations, classifications, mappings, analyses, metrics, logs), and
includes counts in the commit body. So `git log` reads "entities:
+23, evaluations: +23" per chapter instead of the same generic blurb
on every commit. Zero behaviour change when no output changed; falls
back to the original message if the diff query fails.
2. --eval-after-source / --classify-after-source on `infospace process`.
After a source's stages succeed, the pipeline identifies which entity
files are *new* (set diff of entity slugs before vs after), loads
their EntityMeta, and runs per-entity evaluation and/or
classification scoped to just those slugs before the per-source git
commit lands. Result: each chapter's commit is self-contained —
extraction + evaluation + classification in one atomic unit. Gated
behind explicit flags because the cost is real (LLM latency per
chapter rather than amortised across one bulk batch).
3. `markitect infospace chapters` subcommand. Lists source files in
canonical order with entity count, evaluated count, classified
count, and mean per-entity score per source. Text or JSON output.
Natural triage surface for long-text infospaces — spot chapters that
under-extracted or evaluated poorly.
Also: `docs/advanced-usage.md` gets a new "Systematic processing of
long texts" section with the recommended flag combo and the tradeoff
note on cost.
11 new unit tests cover the chapters command (text/json/no-sources),
the process flag wiring (help + provider requirement), and the
commit-body bucket logic. Full infospace+llm unit suite (315 tests)
green; 3 pre-existing infospace failures unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- `markitect infospace entity <name>`: single-entity lookup tolerating
hyphens/underscores/case, with substring matching, ambiguity listing,
and near-match hints. Prints slug, source path, domain, chapter, word
count, VSM system, overall score, evaluator, and evaluation file path.
- `markitect infospace evaluate --model-fallback <model>`: if any
entities fail with a rate-limit error, retry just those with a fresh
adapter on the fallback model (different free-tier models have
separate quota buckets).
- `markitect llm-check`: advisory when `OPENROUTER_API_KEY` is set but
not used by the resolved provider; targeted hint when OpenRouter
returns 401 (almost always a stale env key).
- `build_state`: raises `TypeError` with actionable message if passed a
path instead of an `InfospaceConfig` — prior failure mode was a
confusing `AttributeError` deep in the stack.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Five improvements that eliminate most of the agent-in-the-loop friction
observed while closing out the 988-entity WoN evaluation (C.1):
1. Gemini adapter now retries on 429 + 5xx with exponential backoff
(same pattern already used by OpenRouter/OpenAI). Removes the need
for shell-level retry wrappers when hitting free-tier rate limits.
2. evaluate CLI prints the underlying error ("ERROR — HTTP 503 …")
instead of a bare "ERROR", so agents don't have to drop into Python
to diagnose transient failures.
3. --entity/--chapter now respect existing evaluation files by default
(previously only the full-collection pass did). New --force flag
opts into re-evaluation. Stops silently burning free-tier quota on
re-runs of the same slug.
4. --entity accepts hyphenated slugs (matching entity filenames) and
normalizes them to the underscore form used on disk. On a miss the
CLI suggests near matches instead of a bare "not found".
5. eval-summary --update-metrics is no longer destructive:
read_metrics_file/write_metrics_file preserve structured values
(type_distribution) and don't flatten ints to floats. Fixes a
silent data loss observed on every run.
Bonus: the evaluator field in written evaluation frontmatter now
falls back from run_config.model_name to the adapter's resolved model
(or the model echoed back in the API response), so rows no longer
show `evaluator: null` when --model is omitted.
Tests: new tests/unit/llm/test_gemini.py covers retry behavior;
tests/unit/infospace/test_history.py gains a round-trip test that
pins the type_distribution / int-preservation invariants.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Fix evaluate dimensions to match template file:
definition_precision, source_grounding, domain_placement,
vsm_relevance, explanatory_value (was domain_relevance,
discipline_alignment, conceptual_clarity)
- Add VSM background context to evaluation prompt so LLM can
score vsm_relevance without macro injection
- Fix model_name bug: was sending literal "default" to API (HTTP 400)
- Refactor run_entity_evaluation to write files incrementally via
callback rather than all at once after the batch — long runs are
now resumable if interrupted
- Add incremental skip in CLI: entities with existing eval files
are skipped automatically on re-run (acts as resume)
- Add eval-summary command: reads all eval files, shows per-dimension
means, optionally writes per_entity_mean to metrics.yaml
- Fix record_check_results to merge rather than overwrite metrics.yaml
so per_entity_mean survives subsequent check runs
- Add per_entity_mean viability threshold (min: 3.5) to infospace.yaml
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- PipelineStage now supports max_tokens to override the 4096 default
- SourcePipeline records provider/model on each entity file as HTML comment
- output/processing-log.yaml tracks tokens, cost, duration, retries, errors
- _call_llm returns (content, metadata) for downstream traceability
- _http.py wraps JSON parse errors with body preview for debugging
- infospace.yaml stages: extract/map=6000 tokens, synthesize=3000 tokens
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extend PipelineStage with name, output_dir, output_macro,
split_entities, and macros fields for declarative pipeline config
- Add SourcePipeline class (pipeline.py) using simple @{macro}
substitution — no SQLite dependency, skip-if-exists per stage,
LLM retry on rate limits, git commit per source
- Add `markitect infospace process [GLOB_PATTERN]` CLI command with
--all, --provider, --model, --check-after-each, --no-commit flags
- Update infospace.yaml with output_dir, output_macro, split_entities,
and macros for each pipeline stage in the WoN example
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
History module with snapshot creation from check results, metrics file
I/O, auto-append to history after checks, date-based snapshot lookup,
and metric trend extraction. CLI commands: history, history-diff.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Evaluation pipeline builds prompts from entity metadata, delegates
to BatchEvaluator, parses structured LLM responses into ScoreEntry
objects, and writes evaluation files. CLI: 'markitect infospace evaluate'
with --provider, --entity, --chapter filters.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 'markitect infospace' command group with init (create config),
status (entity count/domains/disciplines), entities (list with sort),
and viability (threshold dashboard with pass/fail).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
