feat(infospace): add eval-summary command and improve evaluate pipeline (S3.3)
- Fix evaluate dimensions to match template file: definition_precision, source_grounding, domain_placement, vsm_relevance, explanatory_value (was domain_relevance, discipline_alignment, conceptual_clarity) - Add VSM background context to evaluation prompt so LLM can score vsm_relevance without macro injection - Fix model_name bug: was sending literal "default" to API (HTTP 400) - Refactor run_entity_evaluation to write files incrementally via callback rather than all at once after the batch — long runs are now resumable if interrupted - Add incremental skip in CLI: entities with existing eval files are skipped automatically on re-run (acts as resume) - Add eval-summary command: reads all eval files, shows per-dimension means, optionally writes per_entity_mean to metrics.yaml - Fix record_check_results to merge rather than overwrite metrics.yaml so per_entity_mean survives subsequent check runs - Add per_entity_mean viability threshold (min: 3.5) to infospace.yaml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -131,8 +131,11 @@ def record_check_results(
|
||||
metrics_dir = root / config.metrics_dir
|
||||
metrics = check_report.metrics()
|
||||
|
||||
# Save latest metrics
|
||||
write_metrics_file(metrics, metrics_dir / "metrics.yaml")
|
||||
# Save latest metrics — merge with existing so other metric sources
|
||||
# (e.g. per-entity evaluation summary) are preserved across check runs.
|
||||
existing = read_metrics_file(metrics_dir / "metrics.yaml")
|
||||
merged = {**existing, **metrics} # check results overwrite on key conflict
|
||||
write_metrics_file(merged, metrics_dir / "metrics.yaml")
|
||||
|
||||
# Create and append snapshot
|
||||
snapshot = snapshot_from_checks(
|
||||
|
||||
Reference in New Issue
Block a user