markitect-main/examples/infospace-with-history/TUTORIAL.md

Building an Infospace with History — Tutorial

This tutorial walks through how we built a structured information space (infospace) from Adam Smith's The Wealth of Nations, mapping classical economic concepts to Stafford Beer's Viable System Model (VSM), using MarkiTect's prompt dependency resolution and LLM integration.

By the end you will understand how to:

1. Design schemas that scaffold structured LLM output
2. Write prompt templates with dependency injection (@{macro} syntax)
3. Populate source artifacts and reference material
4. Run an incremental, chapter-by-chapter pipeline
5. Track every change through git history
6. Measure completeness and consistency with metrics
7. Continue the work to process remaining chapters

---

1. The Idea

We want to transform a large body of text — the full public-domain text of The Wealth of Nations (5 books, 35 chapters) — into a curated collection of economic concepts and entities, each mapped to the VSM.

The challenge: this is too much for a single prompt. The text is hundreds of thousands of words. We need to work incrementally, one chapter at a time, building up the infospace and tracking progress.

MarkiTect's prompt dependency resolution lets us define templates with @{placeholder} macros that are filled from an artifact repository at execution time. The pipeline compiles each template into a complete prompt, sends it to an LLM, and stores the output — all tracked by git.

---

2. Project Layout

examples/infospace-with-history/
│
├── README.md                   # Project brief
├── TUTORIAL.md                 # This file
├── INFRA-TASKS.md              # Infrastructure issues found during the experiment
├── process_chapters.py         # Pipeline script
├── infospace.db                # SQLite artifact database (generated, not in git)
│
├── schemas/                    # Output structure definitions
│   ├── economic-entity-schema-v1.0.md
│   ├── vsm-concept-schema-v1.0.md
│   ├── vsm-mapping-schema-v1.0.md
│   └── chapter-analysis-schema-v1.0.md
│
├── templates/                  # Prompt templates (with @{macro} placeholders)
│   ├── extract-entities.md
│   ├── map-to-vsm.md
│   ├── synthesize-analysis.md
│   └── assess-metrics.md
│
├── artifacts/                  # Input artifacts
│   ├── sources/                # Chapter text (35 files)
│   ├── guidelines/             # Extraction and mapping rules
│   └── vsm-reference/         # VSM framework definition
│
└── output/                     # Generated artifacts (LLM outputs)
    ├── entities/               # Flat canonical entity set + chapter views
    │   ├── division-of-labour.md        # Canonical entity file (PRIMARY)
    │   ├── exchange.md
    │   ├── commercial-society.md
    │   ├── ...
    │   ├── book-1-chapter-01-entities.md  # Chapter view (transclusion)
    │   ├── book-1-chapter-01-prompt.md   # Compiled prompt
    │   ├── book-1-chapter-04-entities.md  # Also references division-of-labour.md
    │   └── ...
    ├── mappings/               # Per-chapter VSM mappings
    ├── analyses/               # Per-chapter synthesised analyses
    └── metrics/                # Cross-chapter metrics reports

Entity organisation: The infospace maintains a flat canonical set of entities — one markdown file per entity, stored directly in output/entities/. When a chapter mentions an entity that already exists (detected by slug collision), the duplicate is skipped and the original definition is kept. This builds a minimal necessary and sufficient set of entities across the entire book.

Per-chapter *-entities.md files are secondary views that use MarkiTect's transclusion engine ({{ include "entity.md" }}) to compose entity content by reference. The same entity (e.g., division-of-labour.md) can appear in multiple chapter views. Editing a canonical entity file automatically updates every chapter view that references it.

Deduplication: The pipeline tells the LLM which entities already exist (via the @{existing_entities} macro in the extraction template) so it focuses on genuinely new entities. At the file level, slug collisions are detected and skipped as a safety net.

Entity lifecycle: Once an entity enters the canonical set, it is never silently deleted. Entities may only be retired when they have been subsumed by another entity, found to partially map onto other entities, or otherwise determined to be redundant. Retired entities are archived — moved to output/entities/archive/ with a dated header documenting the reason. This preserves the intellectual history of the infospace: every decision to drop an entity is a deliberate, documented learning.

# Archive an entity that has been subsumed by another
python process_chapters.py --archive-entity enlarged-monopoly \
  --reason "Subsumed by monopoly-price — both describe the same market distortion"

# The archived file retains its full content with an explanatory header
cat output/entities/archive/enlarged-monopoly.md

The --list command shows both the active canonical set and the archive count.

---

3. Designing Schemas

Before writing any prompts we defined four schemas that tell the LLM exactly what sections each output document must contain. This ensures every generated document is machine-parseable and comparable across chapters.

Economic Entity Schema (schemas/economic-entity-schema-v1.0.md)

Every extracted entity must have:

• H1 heading with the entity name
• Definition (20-150 words)
• Source Chapter citing Book and Chapter
• Context — where in Smith's argument the entity appears
• Economic Domain (Production, Distribution, Exchange, etc.)

Optional: Smith's Original Wording, Modern Interpretation.

VSM Mapping Schema (schemas/vsm-mapping-schema-v1.0.md)

Every entity-to-VSM mapping must have:

• H1 heading in the format Entity Name -> VSM Concept Name
• Economic Entity Reference and VSM Concept Reference
• Mapping Rationale (minimum 30 words, grounded in Beer's definitions)
• Mapping Strength: Strong, Moderate, or Weak

Chapter Analysis Schema (schemas/chapter-analysis-schema-v1.0.md)

The per-chapter synthesis includes:

• Chapter Summary (50-300 words)
• Entities Extracted — bulleted list
• VSM Mappings — entity, concept, strength
• VSM Coverage — explicit assessment of S1 through S5 and S3*
• Gaps & Observations

Metrics Schema (implicit in assess-metrics template)

The metrics report computes:

• VSM Concept Coverage (% of S1-S5, recursion, variety, etc.)
• Chapter Coverage (% of 35 chapters processed)
• Entity and Mapping counts
• Terminology Consistency and Cross-reference Integrity scores

Key insight: Schemas are not code — they are markdown documents that the LLM reads as instructions. This means you can iterate on them without changing any infrastructure.

---

4. Writing Prompt Templates

Each template is a markdown file containing instructions for the LLM plus @{macro_name} placeholders that MarkiTect's resolver fills with artifact content at compile time.

Template 1: Extract Entities (templates/extract-entities.md)

# Extract Economic Entities

You are an analytical economist specialising in classical economic theory.
Your task is to extract distinct economic entities from a chapter of
Adam Smith's *The Wealth of Nations*.

## Source Chapter

@{chapter_text}

## Extraction Guidelines

@{extraction_rules}

## VSM Framework Context

@{vsm_framework}

## Existing Entities

@{existing_entities}

## Instructions
[... detailed step-by-step instructions ...]

## Output Format

Output each entity as a separate markdown document, delimited by
`--- ENTITY: <entity-name> ---` markers.

The four macros (chapter_text, extraction_rules, vsm_framework, existing_entities) are resolved by looking up artifacts by name in the relevant information spaces. The existing_entities list is dynamically generated at runtime from the canonical entity files already on disk, enabling incremental extraction without duplication.

Template 2: Map to VSM (templates/map-to-vsm.md)

Takes @{entities} (output from stage 1), @{vsm_framework}, and @{mapping_rules} as inputs.

Template 3: Synthesise Analysis (templates/synthesize-analysis.md)

Takes @{chapter_text}, @{entities} (stage 1 output), @{mappings} (stage 2 output), and @{vsm_framework}.

Template 4: Assess Metrics (templates/assess-metrics.md)

Takes @{all_analyses} (concatenation of all chapter analyses) and @{vsm_framework}. Runs across the entire infospace, not per-chapter.

Dependency chain per chapter:

chapter_text ─────┐
extraction_rules ──┤
vsm_framework ────┤
                   ▼
           extract-entities
                   │
                   ▼ entities
           map-to-vsm
                   │
                   ▼ mappings
           synthesize-analysis
                   │
                   ▼ analysis

After all chapters are processed, assess-metrics evaluates the complete infospace.

---

5. Populating Artifacts

Source chapters (artifacts/sources/)

35 markdown files containing the full public-domain text of each chapter. Named by convention: book-1-chapter-01.md through book-5-chapter-03.md, plus introduction.md.

These are loaded into the infospace-sources information space.

Guidelines (artifacts/guidelines/)

Two hand-written reference documents:

• extraction-rules.md — What constitutes an entity, granularity rules, naming conventions, quality checks.
• mapping-rules.md — How to map entities to VSM systems, what constitutes strong/moderate/weak mapping strength.

These are loaded into infospace-guidelines.

VSM reference (artifacts/vsm-reference/)

• vsm-framework.md — Complete description of Beer's VSM (S1-S5, S3*, recursion, variety, viability, attenuation/amplification, algedonic signals, autonomy). Includes economic interpretations for each system.

Loaded into infospace-vsm-reference.

---

6. The Pipeline Script

process_chapters.py orchestrates everything. It:

1. Initialises the artifact repository (SQLite) and information spaces
2. Loads all static artifacts (templates, guidelines, VSM reference)
3. For each chapter, runs the three-stage pipeline
4. Optionally calls an LLM to auto-generate outputs
5. Records dependency edges in the graph
6. Commits results to git

Running a single chapter

# Manual mode (writes prompts, waits for you to provide output files):
python process_chapters.py --chapter book-1-chapter-05 --no-commit

# Automatic mode via OpenRouter (recommended — fast, real token counts):
python process_chapters.py --chapter book-1-chapter-05 --provider openrouter --no-commit

# Automatic mode via Gemini free tier:
python process_chapters.py --chapter book-1-chapter-05 --provider gemini --no-commit

# Automatic mode via Claude Code CLI:
python process_chapters.py --chapter book-1-chapter-05 --provider claude-code --no-commit

# With a specific model:
python process_chapters.py --chapter book-1-chapter-05 --provider openrouter --model anthropic/claude-haiku-4-5-20251001 --no-commit
python process_chapters.py --chapter book-1-chapter-05 --provider gemini --model gemini-2.5-flash-lite --no-commit

Running a whole book

python process_chapters.py --book 1 --provider openrouter --no-commit

Running all chapters

python process_chapters.py --all --provider openrouter --no-commit

Checking progress

python process_chapters.py --list

Prints a table showing which chapters have completed each stage (entity counts reflect the chapter view's transclusion references, including shared entities from earlier chapters):

Available chapters (35):

  Chapter                        Entities     Mappings     Analysis
  ------------------------------ ------------ ------------ ------------
  book-1-chapter-01              done (13)    done         done
  book-1-chapter-02              done (7)     done         done
  book-1-chapter-03              done (18)    done         done
  book-1-chapter-04              done (5)     done         done
  book-1-chapter-05              done (1)     done         done
  ...

  Canonical entity set: 42 unique entities

Assessing metrics

After processing a batch of chapters, run the metrics assessment:

python process_chapters.py --metrics --provider openrouter --no-commit

This concatenates all completed analyses and asks the LLM to evaluate coverage, consistency, and completeness.

Dependency statistics

python process_chapters.py --stats

---

7. The Artifact Database (infospace.db)

The pipeline stores all artifacts (source text, templates, guidelines, generated outputs) and their dependency edges in a local SQLite database — infospace.db. This file is not checked into git because it is a derived cache that can be regenerated deterministically from the files already in the repository.

Why it is excluded

• Binary format — SQLite databases don't produce meaningful diffs and would bloat the git history with every pipeline run.
• Fully derived — every piece of data in the database originates from markdown files that are tracked in git (sources, templates, schemas, guidelines, and generated output).
• Reproducible — re-running the pipeline rebuilds the database from scratch without any LLM calls, because each stage checks for existing output files on disk before invoking the LLM.

How to regenerate it

If infospace.db is missing (e.g. after a fresh clone), rebuild it by re-running the pipeline over the chapters that already have output on disk:

# Regenerate the database from existing output files (no LLM calls needed):
python process_chapters.py --all --no-commit

This will:

1. Create a fresh infospace.db
2. Load all static artifacts (templates, guidelines, VSM reference)
3. For each chapter whose output files already exist, import them into the database and record dependency edges
4. Skip LLM calls entirely — existing files are detected and reused

After regeneration, --list and --stats work as normal:

python process_chapters.py --list
python process_chapters.py --stats

---

8. How the LLM Integration Works

The pipeline uses MarkiTect's markitect.llm module, which provides three adapter backends that implement the LLMAdapter interface:

Backend	How it works	Pros	Cons
openrouter	HTTP POST to OpenRouter API	Fast, real token counts, any model	Needs API key
claude-code	Shells out to claude --print	No API key needed if CLI installed	Slower, estimated token counts
gemini	HTTP POST to Google Generative Language API	Free tier available, real token counts	Rate limits on free tier

API key setup (OpenRouter)

Place your key in one of these locations (checked in order):

1. Pass --api-key on the command line (not yet implemented in the CLI)
2. Set OPENROUTER_API_KEY environment variable
3. Create apikey-openrouter.txt in the project root (git-ignored)

API key setup (Gemini)

Place your Google AI Studio key in one of these locations (checked in order):

1. Set GEMINI_API_KEY environment variable
2. Create apikey-geminifree.txt in the project root (git-ignored)

Get a free API key at https://aistudio.google.com/apikey. The free tier supports gemini-2.5-flash with generous rate limits.

What happens per stage

1. The pipeline resolves macro placeholders by looking up artifacts in the repository
2. It compiles the template into a complete prompt (macros replaced with real content)
3. It writes the compiled prompt to output/<stage>/<chapter>-prompt.md for inspection
4. If no output exists yet and an LLM adapter is configured, it executes the prompt
5. For entity extraction (stage 1): the pipeline first binds the list of already-existing entity slugs to @{existing_entities} so the LLM knows what to skip. The LLM returns combined content with --- ENTITY: <name> --- delimiters. The pipeline splits this into the flat canonical directory (output/entities/<slug>.md), skipping any entity whose slug already exists. It then generates the chapter view file with transclusion directives. The combined content is never persisted as a single file — canonical entity files are the source of truth.
6. For other stages: the result is written directly to its output file
7. The output is stored as a generated artifact in the repository
8. Dependency edges are recorded in the graph

---

9. Tracking History with Git

Every processed chapter produces a git commit containing:

• Compiled prompts (*-prompt.md) — so you can audit exactly what was sent
• Canonical entity files (output/entities/<slug>.md) — one file per entity, shared across chapters, first occurrence wins
• Chapter entity views (<chapter>-entities.md) — transclusion into the canonical entities relevant to each chapter
• Generated outputs (*-mappings.md, *-analysis.md)

This means:

• git log shows the chronological order of processing
• git diff between commits shows what each chapter contributed
• You can git bisect to find where quality degraded
• You can revert a chapter and re-process it with different settings

To let the script auto-commit (default):

python process_chapters.py --chapter book-1-chapter-05 --provider openrouter

To commit manually after reviewing:

python process_chapters.py --chapter book-1-chapter-05 --provider openrouter --no-commit
# review new entity files in output/entities/ (look for recently modified .md files)
# review chapter view in output/entities/book-1-chapter-05-entities.md
git add examples/infospace-with-history/output/
git commit -m "infospace: process book-1-chapter-05"

---

10. Cost and Performance

From our measurements processing chapters 3-5:

	Claude Code CLI	OpenRouter	Gemini (free)
Time per chapter	~5 minutes	~2 minutes	~45 seconds
Token counts	Estimated (4 chars/tok)	Real (from API)	Real (from API)
Cost per chapter	~$0.35 est.	~$0.07 est.	Free
Default model	claude-sonnet-4	anthropic/claude-sonnet-4	gemini-2.5-flash

Projected cost for all 35 chapters via OpenRouter: ~$2.50 (varies by chapter length; Book V chapters are longer).

Gemini free tier is a zero-cost option for experimentation and development. The gemini-2.5-flash model produces good results. Rate limits apply — the pipeline retries automatically on 429 responses.

To reduce costs further, use a cheaper model:

--provider openrouter --model anthropic/claude-haiku-4-5-20251001
--provider gemini --model gemini-2.5-flash-lite

---

11. Completing the Remaining Chapters

As of now, 5 of 35 chapters are processed (Book I, Chapters 1-5). Here is how to complete the rest.

Step-by-step

1. Process remaining Book I chapters (6-11):

python process_chapters.py --book 1 --provider openrouter --no-commit

Already-processed chapters are skipped (their chapter view files exist). Entities from earlier chapters are automatically shared — the LLM is told which entities already exist and avoids re-extracting them.

2. Process Books II-V:

python process_chapters.py --book 2 --provider openrouter --no-commit
python process_chapters.py --book 3 --provider openrouter --no-commit
python process_chapters.py --book 4 --provider openrouter --no-commit
python process_chapters.py --book 5 --provider openrouter --no-commit

Or all at once:

python process_chapters.py --all --provider openrouter --no-commit

3. Run metrics after each book (or at the end):

python process_chapters.py --metrics --provider openrouter --no-commit

4. Commit the results:

git add examples/infospace-with-history/output/
git commit -m "infospace: process all remaining chapters"

5. Review the metrics report:

Open output/metrics/metrics-report.md. It will show:

• Which VSM concepts (S1-S5, recursion, variety, etc.) now have mappings
• Total entity and mapping counts
• Consistency scores
• Recommendations for gaps

Expected progression

After	Chapters	Expected coverage
Book I (11 ch.)	11/35	S1, S2, S4 strong; S3 emerging
Books I-II (16 ch.)	16/35	S3 (capital control) covered
Books I-III (20 ch.)	20/35	Historical patterns add depth
Books I-IV (30 ch.)	30/35	S5 (policy, mercantilism) emerging
All (35 ch.)	35/35	Full coverage, S3* and algedonic signals likely from Book V

Book V (public revenue, taxation, sovereign duties) is expected to fill the remaining gaps in S3*, S5, and regulatory concepts.

---

12. Quality Improvement Loop

The infospace is designed to be iteratively refined:

1. Process chapters — run the pipeline
2. Assess metrics — identify gaps in VSM coverage and consistency
3. Refine guidelines — update extraction-rules.md or mapping-rules.md to address identified weaknesses
4. Re-process — delete output files for specific chapters and re-run with improved guidelines
5. Compare — use git diff to see how the refined guidelines changed the output

Example: if metrics show that S3* (Audit) is consistently missed, you could add a paragraph to extraction-rules.md explicitly asking the LLM to look for audit, inspection, and oversight mechanisms.

To re-process a specific chapter, remove its chapter view and downstream outputs. Note: canonical entity files in output/entities/ are shared across chapters — only delete individual entity files if you want them re-extracted from scratch.

rm -f examples/infospace-with-history/output/entities/book-1-chapter-03-entities.md
rm -f examples/infospace-with-history/output/mappings/book-1-chapter-03-mappings.md
rm -f examples/infospace-with-history/output/analyses/book-1-chapter-03-analysis.md
python process_chapters.py --chapter book-1-chapter-03 --provider openrouter --no-commit

Important: never silently delete canonical entity files. If an entity is no longer needed, archive it with a documented reason:

# Entity found to be redundant — archive it
python process_chapters.py --archive-entity extent-of-the-market \
  --reason "Subsumed by market-price and effectual-demand — the concept is fully covered by these two entities"

# Then re-process the chapter
python process_chapters.py --chapter book-1-chapter-03 --provider openrouter --no-commit

If you genuinely need to re-extract an entity with different content (e.g., improving its definition), archive the old version first, then delete the archive copy only after confirming the new version is better. The archive in output/entities/archive/ preserves the full intellectual history of the infospace — every refinement decision is traceable.

---

13. Infrastructure Issues Found and Fixed

During development we documented three issues with the MarkiTect infrastructure in INFRA-TASKS.md:

1. Artifact repo doesn't store content — the resolver returned placeholder text instead of actual artifact content.
2. ContentMacro raw_text defaults to "" — caused silent data corruption when macros were constructed programmatically.
3. No @{target} syntax in MacroParser — macros had to be constructed manually rather than auto-detected from template text.

All three have been fixed in the markitect infrastructure. The pipeline script (process_chapters.py) has been refactored to use the fixed infrastructure directly — the local content cache, manual macro construction, and manual substitution workarounds have been removed. See INFRA-TASKS.md for details on each fix.

---

14. Adapting This Pattern to Your Own Project

To build your own infospace using this pattern:

1. Choose your source corpus — any collection of documents you want to transform into structured knowledge.
2. Define your target ontology — what concepts, relationships, or categories you want to extract (our VSM is just one example).
3. Write schemas — markdown documents defining the required sections and validation rules for each output type.
4. Write extraction guidelines — rules that tell the LLM what to look for and how to handle edge cases.
5. Create prompt templates — use @{macro} syntax to inject source text and guidelines at compile time.
6. Build your pipeline — follow process_chapters.py as a reference for loading artifacts, resolving templates, and calling the LLM.
7. Process incrementally — work through your corpus one document at a time, tracking everything in git.
8. Measure and refine — define metrics, assess them periodically, and update your guidelines when gaps appear.

The key architectural insight is that schemas and guidelines are artifacts — they live in the same repository as your source text and can be versioned, diffed, and refined just like code.