markitect-main/examples/infospace-with-history/INFRA-TASKS.md

Markitect Infrastructure Tasks

Issues discovered while building the infospace-with-history example. All three have been fixed in commit 706981c and the pipeline script refactored to use the fixed infrastructure directly.

1. Artifact Repository does not store content — RESOLVED

File: markitect/prompts/resolver/resolver.py, line 147-148 Issue: content = f"[Content of {artifact.name} from {space_id}]" — the resolver returns placeholder text instead of actual artifact content because the SQLiteArtifactRepository stores metadata (digest, name, type) but not the content itself. Impact: Consumers must maintain their own content cache alongside the repository, defeating the purpose of centralised artifact storage. Fix applied: Added content field to Artifact model, content TEXT column to SQLite schema (with migration for existing DBs), and replaced the resolver placeholder with artifact.content.

2. ContentMacro raw_text defaults to empty string — RESOLVED

File: markitect/prompts/templates/models.py, line 46 Issue: raw_text: str = "" — when macros are constructed programmatically (not parsed from template text), raw_text defaults to "". The ContextCompiler then calls str.replace("", resolved.content) which inserts content between every character, producing multi-gigabyte output. Impact: Silent data corruption; compiled prompts become unusable. Fix applied: Added __post_init__ to ContentMacro that auto-derives raw_text = f"@{{{self.target}}}" when not provided.

3. No TemplateAnalyzer support for @{target} syntax — RESOLVED

File: markitect/prompts/templates/parser.py Issue: The MacroParser parses {{kind:target}} syntax but the templates in this example use the simplified @{target} syntax. There's no automatic parsing for this format, requiring manual macro construction. Fix applied: Added SHORTHAND_PATTERN to MacroParser that recognises @{target} and maps it to MacroKind.REQUIRED. Updated has_macros(), count_macros(), and find_macro_positions() accordingly.

---

Assignment Assessment (18 Feb 2026)

How the example measures against the objectives stated in README.md:

#	Objective	Status	Notes
1	Capture knowledge from Wealth of Nations	Partial	7 of 35 chapters processed (Book I, ch. 1-7). 85 canonical entities extracted.
2	Transform to VSM concepts/entities	Done (for processed chapters)	Entities mapped to S1-S5 with strength ratings.
3	Consistent and complete	Not yet	Only 20% of chapters done. Metrics report exists but covers limited scope.
4	Schemas as scaffolding	Done	Four schemas defined and used across all stages.
5	Prompt dependency resolution	Done	@{macro} templates resolved via MultiSpaceResolutionStrategy.
6	Incremental chapter injection	Done	Pipeline processes one chapter at a time; @{existing_entities} prevents duplication.
7	Keep changes as git history	Not done	See task 4 below.
8	Metrics for completeness/consistency	Partial	Template and report exist but only cover 4 chapters (report predates ch. 5-7).
9	No infrastructure changes during experiment	Violated	Three infra fixes were required (tasks 1-3 above). Documented as intended.
10	Generate task list for infra issues	Done	This file.

4. Infospace has no per-chapter git history — OPEN

Objective: README states "The information space should utilize the option of keeping changes as git history." Issue: The 7 processed chapters were committed in mixed batches alongside infrastructure changes (LLM adapters, entity refactoring, archive policy). Chapters 1-2 are bundled into fecc2fd with the entire LLM module. Chapters 5-7 share a single commit (41773f1) with the OpenAI adapter and archive policy. There is no commit where you can git diff to see exactly what one chapter contributed to the infospace. Impact: Cannot use git log, git diff, or git bisect to trace how the infospace grew chapter by chapter — the core promise of "with history." Suggested fix: Re-run the 7 processed chapters (and remaining 28) using process_chapters.py without --no-commit, on a clean branch or after squashing the current output into a baseline commit. Each chapter gets its own commit via _git_commit_chapter().

5. Prompt files are regenerated as a side-effect of DB rebuild — OPEN

Issue: Running --all --no-commit to regenerate infospace.db also overwrites *-prompt.md files in the output directories because each pipeline stage unconditionally writes the compiled prompt before checking whether output already exists. The @{existing_entities} macro content shifts as earlier chapters are loaded, so prompt files for already-processed chapters change on every full run. Impact: A DB regeneration dirties the working tree with prompt file changes, even though no actual outputs changed. Users must git checkout the prompt files after regeneration. Suggested fix: Skip writing prompt files when the corresponding output file already exists on disk, or add a --rebuild-db-only flag that populates the database without touching the file system.

6. Metrics report is stale — OPEN

Issue: The metrics report (output/metrics/metrics-report.md) was generated after chapters 1-4. Chapters 5-7 have since been processed but the report has not been refreshed. Impact: The metrics do not reflect the current state of the infospace. Suggested fix: Re-run --metrics --provider <provider> --no-commit after every batch of new chapters. Consider making metrics assessment automatic at the end of --book or --all runs.

7. Remaining 28 chapters not yet processed — OPEN

Issue: Only Book I chapters 1-7 have been processed. Books II-V (28 chapters) remain unprocessed. Impact: The infospace is incomplete — VSM coverage is limited to S1, S2, and partial S4. S3, S3*, S5, and many systemic concepts (algedonic signals, recursion, variety) are expected to emerge from later books. Suggested fix: Process remaining chapters in book-sized batches with per-chapter commits, refreshing metrics after each book.

---

Per-Concept Metrics (tasks 8-12)

The current metrics system is a single LLM-evaluated narrative report that assesses the infospace as a whole. It produces no machine-readable output, cannot be tracked over time, and conflates per-concept quality with collection-level coherence.

The improvement splits metrics into two layers:

• LLM-Eval: A prompt template evaluates each concept individually against quality criteria defined in the schema. The LLM returns structured scores, not prose.
• Deterministic aggregation: process_chapters.py computes what it can from files on disk (schema compliance, word counts, section presence, coverage tallies) and aggregates LLM-eval scores into dashboard metrics.

Both layers persist results in structured form so they can be diffed, tracked over time, and committed alongside the entities they evaluate.

8. Add per-concept quality metrics to entity schema — OPEN

Issue: The entity schema (economic-entity-schema-v1.0.md) defines required sections and validation rules (section presence, word count range) but no quality criteria. There is no definition of what makes a good entity versus a merely compliant one. Suggested fix: Add a ## Quality Metrics section to the entity schema defining evaluation dimensions with scoring rubrics:

• Definition Precision (1-5): Is the definition specific, non-circular, and distinguishable from neighbouring concepts?
• Source Grounding (1-5): Is the entity grounded in a specific passage? Does the citation exist and support the definition?
• Domain Placement (1-5): Is the economic domain assignment correct and specific (not just "General Theory")?
• VSM Relevance (1-5): Does the entity connect meaningfully to at least one VSM system, or is it too granular/abstract to map?
• Explanatory Value (1-5): Does this entity contribute to explaining the economic system, or is it a restatement of another concept?

Similarly update the VSM mapping schema with:

• Rationale Rigour (1-5): Is the mapping justified with reference to Beer's definitions, not just surface-level analogy?
• Strength Calibration (1-5): Is the declared strength (Strong/Moderate/ Weak) consistent with the rationale given?

These rubrics become the prompt instructions for task 9.

9. Create evaluate-entity prompt template — OPEN

Depends on: Task 8 (quality metrics in schema). Issue: There is no mechanism to evaluate an existing entity after extraction. Quality is only judged implicitly during the global metrics assessment, which is too coarse to identify individual weak entities. Suggested fix: Create templates/evaluate-entity.md — a prompt template that:

1. Takes @{entity_content}, @{source_chapter}, @{vsm_framework}, and @{quality_rubric} (from the schema's quality metrics section).
2. Asks the LLM to score each dimension (1-5) with a one-sentence justification per score.
3. Outputs structured YAML front-matter (scores) followed by markdown (justifications), e.g.:

---
entity: division-of-labour
scores:
  definition_precision: 5
  source_grounding: 5
  domain_placement: 4
  vsm_relevance: 5
  explanatory_value: 5
overall: 4.8
flags: []
---

Add a pipeline stage: --evaluate runs this template against every canonical entity and writes results to output/evaluations/<slug>-eval.md. A --evaluate --chapter <id> variant evaluates only entities introduced by that chapter.

10. Add deterministic schema compliance checker — OPEN

Issue: Schema compliance is currently LLM-evaluated ("100%" in the metrics report) but the validation rules in the schemas are mechanical: section presence, word count ranges, heading format. These should be checked programmatically, not by an LLM. Suggested fix: Add a validate_entity(path) -> ValidationResult function to process_chapters.py (or a new validate.py module) that:

• Parses the markdown to extract H2 section headings
• Checks required sections are present (Definition, Source Chapter, Context, Economic Domain)
• Counts words in the Definition section (must be 20-150)
• Checks H1 heading exists and is not a slug (e.g. effectual-demand in chapter 7 has # effectual-demand instead of # Effectual Demand)
• Validates Source Chapter cites a specific book/chapter
• For mapping files: checks Mapping Strength is one of the enum values

Expose as --validate CLI flag. Output a structured report:

Validation: 85 entities, 3 warnings
  effectual-demand.md: H1 is slug format, not title case
  porter.md: Definition is 18 words (minimum 20)
  ...

This is fully deterministic — no LLM calls needed.

11. Structured metrics output format — OPEN

Depends on: Tasks 9 and 10. Issue: The metrics report is a markdown narrative. Values cannot be parsed programmatically, diffed meaningfully, or plotted over time. Suggested fix: Alongside the human-readable metrics-report.md, emit a machine-readable metrics.yaml (or .json) containing:

timestamp: "2026-02-18T12:00:00Z"
chapters_processed: 7
chapters_total: 35
entities_total: 85
entities_archived: 0
vsm_coverage:
  S1: 28
  S2: 12
  S3: 8
  S3_star: 0
  S4: 5
  S5: 0
  recursion: 1
  variety: 0
mapping_strength:
  strong: 64
  moderate: 18
  weak: 3
validation:
  schema_compliant: 82
  warnings: 3
evaluation:    # from LLM-eval (task 9)
  mean_overall: 4.2
  min_overall: 2.8
  flagged_entities: ["porter", "country-workman"]

The --metrics command writes both files. The YAML file is committed to git so git diff shows exactly how metrics changed between runs.

12. Metrics-over-time tracking — OPEN

Depends on: Task 11 (structured output). Issue: There is one metrics snapshot that gets overwritten. No history of how metrics evolved as chapters were added. Suggested fix: Append each metrics snapshot to a cumulative log file output/metrics/metrics-history.yaml (list of timestamped entries). This is committed to git alongside the current snapshot. The pipeline can optionally render a simple text-based progress summary:

Metrics history (5 snapshots):
  2026-02-10  ch 1/35   13 entities  41.7% VSM coverage
  2026-02-11  ch 4/35   38 entities  50.0% VSM coverage
  2026-02-11  ch 7/35   85 entities  58.3% VSM coverage
  ...

This provides the "metrics that improve over time" feedback loop the README envisions: process chapters → evaluate → see coverage grow (or flag regressions when a re-extraction reduces quality scores).

---

Collection-Level Metrics (tasks 13-19)

These tasks implement the five collection-level concerns described in METRICS-METHODOLOGY.md. They share underlying infrastructure (entity metadata index, definition embeddings, relationship graph) that should be built once per evaluation run.

See the methodology document for theoretical grounding, framework references, and the full metric definitions per concern.

13. Entity metadata index — deterministic parsing layer — OPEN

Depends on: Task 10 (schema compliance checker shares parsing logic). Issue: Several collection-level metrics (coverage matrix, FCA context, granularity distribution) require structured metadata extracted from entity files: H1 title, economic domain, VSM system(s), source chapter, section presence, word counts. Currently this information exists only as prose inside markdown files. Suggested fix: Add a parse_entity_metadata(path) -> EntityMeta function that extracts from each entity file:

@dataclass
class EntityMeta:
    slug: str
    title: str                  # from H1
    domain: str                 # from Economic Domain section
    source_chapter: str         # from Source Chapter section
    definition_words: int       # word count of Definition section
    has_original_wording: bool  # optional section present?
    has_modern_interpretation: bool
    vsm_systems: list[str]     # from mapping file if exists
    mapping_strengths: list[str]

Build an index of all entities at the start of each evaluation run. This index is the input for tasks 14, 16, and 18. Expose as --index CLI flag for inspection.

14. Redundancy detection (Concern C1) — OPEN

Depends on: Task 13 (metadata index). Methodology: OOPS! P2 (synonymous classes) + embedding similarity + LLM pairwise judgment. See METRICS-METHODOLOGY.md §4 C1. Issue: Entities with different slugs but overlapping meanings (e.g. natural-rate / ordinary-or-average-rate) survive extraction because dedup only checks slug collisions. There is no semantic overlap detection. Suggested fix: Implement in three stages:

1. Embed — Compute vector embeddings of all entity definitions using an embedding API (OpenRouter, OpenAI, or a local sentence-transformer). Cache embeddings in output/metrics/embeddings.json keyed by {slug: content_digest} so unchanged entities skip re-embedding.

2. Similarity matrix — Compute NxN cosine similarity. Write the full matrix to output/metrics/similarity-matrix.json. Flag all pairs with cosine > 0.80 as candidates.

3. LLM pairwise judgment — For each candidate pair, run a prompt: "Given these two entity definitions, are they (a) the same concept and should be merged, (b) genuinely distinct, or (c) partially overlapping and should be clarified?" Write results to output/metrics/redundancy-report.md + YAML.

Metrics produced:

• high_similarity_pairs: count and list
• confirmed_synonyms: count (LLM-confirmed same concept)
• redundancy_ratio: confirmed_synonyms / total_entities
• intensional_conciseness: 1 - redundancy_ratio

CLI: --check-redundancy --provider <provider>

15. Coverage completeness (Concern C2) — OPEN

Depends on: Task 13 (metadata index). Methodology: SEQUAL completeness + FCA gap analysis + DSL competency questions. See METRICS-METHODOLOGY.md §4 C2. Issue: Coverage is currently assessed by the LLM in a single narrative pass. There is no structured view of which domain × VSM cells are populated, and no way to test whether the entity set can answer specific questions about the economic system. Suggested fix: Implement in three stages:

1. Domain × VSM matrix — From the metadata index, count entities per {economic_domain, vsm_system} cell. Render as a table. Identify empty cells as specific, actionable gaps. Compute:

   • coverage_ratio = populated_cells / total_cells
   • vsm_balance_entropy = -Σ(pᵢ log pᵢ) across VSM systems

2. FCA lattice — Construct a formal context with objects = entities, attributes = {domain, vsm_system, source_book, abstraction_level}. Compute the concept lattice (Python concepts library). Extract attribute combinations with no corresponding entity — these are structural coverage gaps not visible in the simple matrix.

3. Competency questions — Define a set of 15-20 canonical questions the infospace should answer (stored in schemas/competency-questions.md). Example questions:

   • "How does the division of labour relate to market extent?"
   • "What mechanisms regulate wages toward their natural rate?"
   • "How do monopolies distort the viable system?" LLM-Eval tests whether current entities suffice to answer each. Unanswerable questions identify specific completeness gaps.

Metrics produced:

• domain_vsm_matrix: cell counts
• coverage_ratio: scalar
• vsm_balance_entropy: scalar
• empty_cells: list of {domain, vsm_system} gaps
• fca_gap_concepts: attribute combos with no entity
• competency_coverage: fraction of questions answerable

CLI: --check-coverage --provider <provider>

16. Structural coherence (Concern C3) — OPEN

Depends on: Task 13 (metadata index). Methodology: OntoQA relationship richness + graph connectivity + community detection. See METRICS-METHODOLOGY.md §4 C3. Issue: It is unknown whether the 85 entities form a connected explanatory web or a fragmented collection. No relationship graph exists between entities. Suggested fix: Implement in three stages:

1. Explicit cross-references — Scan each entity's definition for mentions of other entity slugs or titles (normalised string matching). This is deterministic and catches direct references.

2. LLM-inferred edges — For entity pairs not caught by string matching but in the same domain or VSM system, LLM-Eval: "Does A's definition conceptually depend on or explain B, or vice versa?" Run in batches. Write the combined graph to output/metrics/relationship-graph.json (adjacency list).

3. Graph analysis — Using networkx or equivalent:

   • Connected components (target: 1)
   • Graph density, average degree
   • Betweenness centrality → identify bridge concepts
   • Louvain community detection → compare to declared domains
   • OntoQA Relationship Richness
   • Cohesion per domain, coupling across domains
   • Orphan entities (degree 0 or 1)

Metrics produced:

• connected_components: count (target: 1)
• graph_density: scalar
• avg_degree: scalar
• relationship_richness: OntoQA RR
• modularity: Louvain score
• bridge_concepts: list (high betweenness centrality)
• orphan_entities: list (degree ≤ 1)
• cohesion_by_domain / coupling_across_domains: scalars

CLI: --check-coherence --provider <provider>

17. Definitional consistency (Concern C4) — OPEN

Depends on: Task 16 (relationship graph — the definitional dependency graph is a directed variant of the same structure). Methodology: OntoClean metaproperties + OOPS! P24 (circular definitions) + SEQUAL validity. See METRICS-METHODOLOGY.md §4 C4. Issue: No mechanism to detect circular definitions, contradictions between related entities, or terms used in definitions that should be entities but aren't. Suggested fix: Implement in four stages:

1. Definitional dependency graph — Directed version of the relationship graph: edge A→B means A's definition uses B's concept. Reuse cross-reference extraction from task 16.

2. Cycle detection — Find all cycles of length ≤ 3 in the directed graph. Short cycles are problematic (A defines B, B defines A). Compute grounding_ratio: fraction of entities traceable to terms outside the entity set without encountering a cycle.

3. Undefined dependencies — Extract terms from definitions that match entity-name patterns (capitalised noun phrases, kebab-case slugs) but have no corresponding entity file. These are concepts the infospace implicitly relies on but hasn't defined.

4. LLM consistency checks — For directly-connected entity pairs, LLM-Eval: "Do these definitions contradict each other?" For entities with Smith's Original Wording, LLM-Eval: "Does the definition accurately represent the cited passage?"

Metrics produced:

• circular_definitions: count and list of cycles (length ≤ 3)
• grounding_ratio: fraction of entities reaching primitives
• undefined_dependencies: list of missing terms
• contradiction_candidates: LLM-flagged pairs
• source_fidelity_score: fraction passing source check

CLI: --check-consistency --provider <provider>

18. Granularity balance (Concern C5) — OPEN

Depends on: Task 13 (metadata index). Methodology: Keet granularity theory + OntoClean rigidity + DSL laconicity. See METRICS-METHODOLOGY.md §4 C5. Issue: Entities range from broad sectors (agriculture) to specific market roles (effectual-demanders) to abstract principles (division-of-labour). It is unclear whether this range is appropriate or whether some entities are too specific/general relative to their peers. Suggested fix: Implement in three stages:

1. LLM classification — For each entity, LLM-Eval assigns:

   • Abstraction level: theory / mechanism / observation
   • Scope score: 1-5 (very specific → very general)
   • Indispensability: 1-5 ("if removed, how much explanatory power lost?") Write to output/evaluations/<slug>-classification.yaml.

2. Distribution analysis — Deterministic:

   • Count per abstraction level; compute entropy
   • Per-domain scope variance (flag domains with high variance)
   • Level × domain matrix (from FCA context in task 15)
   • Outlier detection: entities > 1.5σ from their domain's mean scope

3. Merge/split recommendations — For outlier entities, LLM-Eval: "Should this entity be merged into a broader concept, split into sub-concepts, or is its current granularity justified?" For entities with indispensability ≤ 2: "Could another entity serve this purpose?"

Metrics produced:

• abstraction_distribution: {theory: n, mechanism: n, observation: n}
• abstraction_entropy: scalar (higher = more balanced)
• scope_variance_by_domain: per-domain scalar
• dispensable_entities: list (indispensability ≤ 2)
• merge_candidates: list of pairs
• split_candidates: list of entities

CLI: --check-granularity --provider <provider>

19. Unified collection evaluation command — OPEN

Depends on: Tasks 13-18. Issue: Running five separate --check-* commands is cumbersome and repeats shared computation (metadata parsing, embedding, graph building). Suggested fix: Add --evaluate-collection --provider <provider> that runs all five checks in sequence, sharing infrastructure:

1. Parse entity metadata index (task 13) — used by all
2. Compute embeddings (task 14) — used by C1, C3
3. Build relationship graph (task 16) — used by C3, C4
4. Run all five concern checks
5. Write per-concern reports to output/metrics/
6. Write unified metrics.yaml with all collection metrics
7. Append to metrics-history.yaml (task 12)

Incremental mode: --evaluate-collection --chapter <id> re-evaluates only entities from that chapter plus pairwise checks involving them.

Report a summary to stdout:

Collection evaluation (85 entities, 7 chapters):
  Redundancy:   3 synonym candidates, conciseness 0.96
  Coverage:     58% VSM, 20% chapters, 4 domain gaps
  Coherence:    1 component, density 0.12, 2 orphans
  Consistency:  0 cycles, 5 undefined deps, 0 contradictions
  Granularity:  entropy 1.42, 1 dispensable, 2 merge candidates