coulomb/markitect-main
2
0
Fork 0
Code Issues 133 Pull Requests Actions Projects 12 Releases 1 Wiki Activity

fix(example): skip prompt writes when output exists, add quality rubrics

Browse Source 
INFRA-TASKS #5 — process_chapters.py now skips writing *-prompt.md files
when the corresponding output file already exists on disk. DB-only rebuilds
no longer dirty the working tree with unchanged prompt content.

INFRA-TASKS #8 — Added '## Quality Metrics' section to the entity and VSM
mapping schemas, defining the five evaluation dimensions (Definition Precision,
Source Grounding, Domain Placement, VSM Relevance, Explanatory Value) with
1–5 rubrics used by the evaluate-entity template.

Also updated INFRA-TASKS.md to reflect current resolution status for tasks
4–19 across S2 and S3.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-02-23 06:04:09 +01:00
parent dfab3d598b
commit fa27572f43
4 changed files with 258 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
examples/infospace-with-history/INFRA-TASKS.md
View File

@@ -57,7 +57,7 @@ How the example measures against the objectives stated in `README.md`:
| 9 | No infrastructure changes during experiment | **Violated** | Three infra fixes were required (tasks 1-3 above). Documented as intended. |
| 10 | Generate task list for infra issues | **Done** | This file. |
## 4. Infospace has no per-chapter git history — OPEN
## 4. Infospace has no per-chapter git history — PARTIAL
**Objective:** README states "The information space should utilize the option
of keeping changes as git history."
@@ -69,12 +69,15 @@ archive policy. There is no commit where you can `git diff` to see exactly
what one chapter contributed to the infospace.
**Impact:** Cannot use `git log`, `git diff`, or `git bisect` to trace how
the infospace grew chapter by chapter — the core promise of "with history."
**Suggested fix:** Re-run the 7 processed chapters (and remaining 28) using
**Progress:** Branch `clean-example-history` was created. Chapters 1-8 have
clean per-chapter commits. 27 chapters remain. Example completeness (tasks 4
and 7) is deferred; no further action planned.
**Suggested fix (original):** Re-run the processed chapters using
`process_chapters.py` without `--no-commit`, on a clean branch or after
squashing the current output into a baseline commit. Each chapter gets its
own commit via `_git_commit_chapter()`.
## 5. Prompt files are regenerated as a side-effect of DB rebuild — OPEN
## 5. Prompt files are regenerated as a side-effect of DB rebuild — RESOLVED
**Issue:** Running `--all --no-commit` to regenerate `infospace.db` also
overwrites `*-prompt.md` files in the output directories because each
@@ -85,9 +88,10 @@ chapters change on every full run.
**Impact:** A DB regeneration dirties the working tree with prompt file
changes, even though no actual outputs changed. Users must `git checkout`
the prompt files after regeneration.
**Suggested fix:** Skip writing prompt files when the corresponding output
file already exists on disk, or add a `--rebuild-db-only` flag that
populates the database without touching the file system.
**Fix applied:** Each pipeline stage (`stage_extract_entities`,
`stage_map_to_vsm`, `stage_synthesize_analysis`, `assess_metrics`) now
skips writing the `*-prompt.md` file when the corresponding output file
already exists on disk. DB regeneration no longer dirties the working tree.
## 6. Metrics report is stale — OPEN
@@ -99,15 +103,16 @@ the report has not been refreshed.
after every batch of new chapters. Consider making metrics assessment
automatic at the end of `--book` or `--all` runs.
## 7. Remaining 28 chapters not yet processed — OPEN
## 7. Remaining 28 chapters not yet processed — DEFERRED
**Issue:** Only Book I chapters 1-7 have been processed. Books II-V
(28 chapters) remain unprocessed.
**Impact:** The infospace is incomplete — VSM coverage is limited to S1,
S2, and partial S4. S3, S3*, S5, and many systemic concepts (algedonic
signals, recursion, variety) are expected to emerge from later books.
**Suggested fix:** Process remaining chapters in book-sized batches with
per-chapter commits, refreshing metrics after each book.
**Note:** Example completeness is deferred. The 7/35 chapter corpus is
sufficient to validate the tooling. Resuming requires the `clean-example-history`
branch and a valid `OPENROUTER_API_KEY`.
---
@@ -130,7 +135,7 @@ The improvement splits metrics into two layers:
Both layers persist results in structured form so they can be diffed,
tracked over time, and committed alongside the entities they evaluate.
## 8. Add per-concept quality metrics to entity schema — OPEN
## 8. Add per-concept quality metrics to entity schema — RESOLVED
**Issue:** The entity schema (`economic-entity-schema-v1.0.md`) defines
required sections and validation rules (section presence, word count range)
@@ -158,8 +163,10 @@ Similarly update the VSM mapping schema with:
Weak) consistent with the rationale given?
These rubrics become the prompt instructions for task 9.
**Fix applied:** `## Quality Metrics` section added to
`schemas/economic-entity-schema-v1.0.md` and `schemas/vsm-mapping-schema-v1.0.md`.
## 9. Create evaluate-entity prompt template — OPEN
## 9. Create evaluate-entity prompt template — RESOLVED
**Depends on:** Task 8 (quality metrics in schema).
**Issue:** There is no mechanism to evaluate an existing entity after
@@ -193,8 +200,11 @@ Add a pipeline stage: `--evaluate` runs this template against every
canonical entity and writes results to `output/evaluations/<slug>-eval.md`.
A `--evaluate --chapter <id>` variant evaluates only entities introduced
by that chapter.
**Fix applied:** `templates/evaluate-entity.md` created. `--evaluate`
flag added to `process_chapters.py`. Reads `@{quality_rubric}` from the
entity schema's Quality Metrics section.
## 10. Add deterministic schema compliance checker — OPEN
## 10. Add deterministic schema compliance checker — RESOLVED
**Issue:** Schema compliance is currently LLM-evaluated ("100%" in the
metrics report) but the validation rules in the schemas are mechanical:
@@ -222,8 +232,10 @@ Validation: 85 entities, 3 warnings
```
This is fully deterministic — no LLM calls needed.
**Fix applied:** `markitect/infospace/validator.py` — `validate_entity()`
and `validate_entities()`. Exposed via `--infospace-check`.
## 11. Structured metrics output format — OPEN
## 11. Structured metrics output format — RESOLVED
**Depends on:** Tasks 9 and 10.
**Issue:** The metrics report is a markdown narrative. Values cannot be
@@ -261,8 +273,9 @@ evaluation: # from LLM-eval (task 9)
The `--metrics` command writes both files. The YAML file is committed
to git so `git diff` shows exactly how metrics changed between runs.
**Fix applied:** `output/metrics/metrics.yaml` produced by `--infospace-check`.
## 12. Metrics-over-time tracking — OPEN
## 12. Metrics-over-time tracking — RESOLVED
**Depends on:** Task 11 (structured output).
**Issue:** There is one metrics snapshot that gets overwritten. No history
@@ -283,6 +296,8 @@ Metrics history (5 snapshots):
This provides the "metrics that improve over time" feedback loop the
README envisions: process chapters → evaluate → see coverage grow (or
flag regressions when a re-extraction reduces quality scores).
**Fix applied:** `output/metrics/history.yaml` maintained by
`markitect/infospace/history.py`.
---
@@ -296,7 +311,7 @@ be built once per evaluation run.
See the methodology document for theoretical grounding, framework
references, and the full metric definitions per concern.
## 13. Entity metadata index — deterministic parsing layer — OPEN
## 13. Entity metadata index — deterministic parsing layer — RESOLVED
**Depends on:** Task 10 (schema compliance checker shares parsing logic).
**Issue:** Several collection-level metrics (coverage matrix, FCA context,
@@ -324,8 +339,10 @@ class EntityMeta:
Build an index of all entities at the start of each evaluation run.
This index is the input for tasks 14, 16, and 18. Expose as
`--index` CLI flag for inspection.
**Fix applied:** `markitect/infospace/entity_parser.py` — `parse_entity_file()`
and `parse_entity_directory()`. Used automatically by `--infospace-check`.
## 14. Redundancy detection (Concern C1) — OPEN
## 14. Redundancy detection (Concern C1) — RESOLVED
**Depends on:** Task 13 (metadata index).
**Methodology:** OOPS! P2 (synonymous classes) + embedding similarity +
@@ -357,8 +374,9 @@ dedup only checks slug collisions. There is no semantic overlap detection.
- `intensional_conciseness`: `1 - redundancy_ratio`
**CLI:** `--check-redundancy --provider <provider>`
**Fix applied:** `markitect/infospace/checks/redundancy.py`. Exposed via `--infospace-check`.
## 15. Coverage completeness (Concern C2) — OPEN
## 15. Coverage completeness (Concern C2) — RESOLVED
**Depends on:** Task 13 (metadata index).
**Methodology:** SEQUAL completeness + FCA gap analysis + DSL competency
@@ -399,8 +417,9 @@ questions about the economic system.
- `competency_coverage`: fraction of questions answerable
**CLI:** `--check-coverage --provider <provider>`
**Fix applied:** `markitect/infospace/checks/coverage.py`. Exposed via `--infospace-check`.
## 16. Structural coherence (Concern C3) — OPEN
## 16. Structural coherence (Concern C3) — RESOLVED
**Depends on:** Task 13 (metadata index).
**Methodology:** OntoQA relationship richness + graph connectivity +
@@ -440,8 +459,9 @@ between entities.
- `cohesion_by_domain` / `coupling_across_domains`: scalars
**CLI:** `--check-coherence --provider <provider>`
**Fix applied:** `markitect/infospace/checks/coherence.py`. Exposed via `--infospace-check`.
## 17. Definitional consistency (Concern C4) — OPEN
## 17. Definitional consistency (Concern C4) — RESOLVED
**Depends on:** Task 16 (relationship graph — the definitional dependency
graph is a directed variant of the same structure).
@@ -479,8 +499,9 @@ entities but aren't.
- `source_fidelity_score`: fraction passing source check
**CLI:** `--check-consistency --provider <provider>`
**Fix applied:** `markitect/infospace/checks/consistency.py`. Exposed via `--infospace-check`.
## 18. Granularity balance (Concern C5) — OPEN
## 18. Granularity balance (Concern C5) — RESOLVED
**Depends on:** Task 13 (metadata index).
**Methodology:** Keet granularity theory + OntoClean rigidity +
@@ -517,8 +538,9 @@ or whether some entities are too specific/general relative to their peers.
- `split_candidates`: list of entities
**CLI:** `--check-granularity --provider <provider>`
**Fix applied:** `markitect/infospace/checks/granularity.py`. Exposed via `--infospace-check`.
## 19. Unified collection evaluation command — OPEN
## 19. Unified collection evaluation command — RESOLVED
**Depends on:** Tasks 13-18.
**Issue:** Running five separate `--check-*` commands is cumbersome and
@@ -537,6 +559,10 @@ runs all five checks in sequence, sharing infrastructure:
Incremental mode: `--evaluate-collection --chapter <id>` re-evaluates
only entities from that chapter plus pairwise checks involving them.
**Fix applied:** `markitect/infospace/checks/orchestrator.py` + `--infospace-check`
CLI flag. All five checks share the metadata index. Results recorded in
`output/metrics/metrics.yaml` and `output/metrics/history.yaml`.
Report a summary to stdout:
```