docs(tutorial): update §8-9 for eval-summary command and 6/6 viability

- Add eval-summary command documentation with dimension descriptions
- Document resumable evaluate (incremental skip on re-run)
- Fix --entity slug example to use underscores (not hyphens)
- Update viability output to show per_entity_mean as 6th threshold
- Add workflow note: check → eval-summary --update-metrics → viability

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-23 05:33:11 +01:00
parent 7f1eecbdb2
commit 34ed7a6fab

View File

@@ -391,13 +391,54 @@ markitect infospace evaluate --provider openrouter
# Evaluate entities from a specific chapter:
markitect infospace evaluate --chapter book-1-chapter-05 --provider openrouter
# Re-evaluate a single entity:
markitect infospace evaluate --entity division-of-labour --provider openrouter
# Re-evaluate a single entity (slugs use underscores):
markitect infospace evaluate --entity division_of_labour --provider openrouter
```
This runs the `evaluate-entity` prompt template against each entity,
scoring dimensions like definition precision, source grounding, and
VSM relevance. Results are written to `output/evaluations/`.
The command is resumable: entities with existing evaluation files are
skipped automatically. Re-run after an interruption and it picks up
where it left off. Results are written incrementally to
`output/evaluations/<slug>.md`.
Each entity is scored on five dimensions (15 scale):
- `definition_precision` — Is the definition precise and non-circular?
- `source_grounding` — Is the entity grounded in the actual source text?
- `domain_placement` — Is the economic domain assignment correct?
- `vsm_relevance` — Does the entity map naturally to a VSM system (S1S5)?
- `explanatory_value` — Does the entity add genuine explanatory power?
### Evaluation summary
After the evaluation run completes, compute aggregate statistics:
```bash
# Show per-dimension means:
markitect infospace eval-summary
# Also write per_entity_mean to metrics.yaml for viability checks:
markitect infospace eval-summary --update-metrics
```
Sample output (full corpus, 988 entities):
```
Evaluation summary — 988 entities evaluated
Dimension Mean
--------------------------------------
overall 4.XX
definition_precision 4.XX
domain_placement X.XX
explanatory_value 4.XX
source_grounding 4.XX
vsm_relevance 3.XX
Range: X.XX X.XX
```
`vsm_relevance` typically scores lower than the other dimensions —
many WoN concepts are foundational economic ideas that don't map
cleanly to a single VSM system. This is expected and informative.
### Collection-level checks (C1C5)
@@ -459,15 +500,20 @@ Compares the latest metrics against the thresholds declared in
```
Metric Value Threshold Status
---------------------------------------------------------------
redundancy_ratio 0.0059 max=0.1 PASS
redundancy_ratio 0.0061 max=0.1 PASS
coverage_ratio 0.6190 min=0.4 PASS
coherence_components 0.0000 max=3 PASS
consistency_cycles 0.0000 max=0 PASS
granularity_entropy 2.9533 min=1.0 PASS
granularity_entropy 2.6748 min=1.0 PASS
per_entity_mean 4.XXXX min=3.5 PASS
Viable: YES (5/5 thresholds met)
Viable: YES (6/6 thresholds met)
```
`per_entity_mean` only appears after running `eval-summary --update-metrics`.
Run `check` first (deterministic), then `eval-summary --update-metrics`,
then `viability` to see the full six-threshold dashboard.
During early processing (first few books), coverage will fall and
then stabilise as the domain × chapter matrix fills in. The threshold
of 0.40 reflects realistic expectations for a multi-book corpus where