Chapters with many pre-existing entities were still truncating at 6000 tokens
because the LLM needs space to output the full list of candidates even when
most are skipped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
map-to-vsm was consistently truncating at 6000 tokens; synthesize-analysis
sometimes truncated at 3000 for chapters with many entities.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- PipelineStage now supports max_tokens to override the 4096 default
- SourcePipeline records provider/model on each entity file as HTML comment
- output/processing-log.yaml tracks tokens, cost, duration, retries, errors
- _call_llm returns (content, metadata) for downstream traceability
- _http.py wraps JSON parse errors with body preview for debugging
- infospace.yaml stages: extract/map=6000 tokens, synthesize=3000 tokens
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- SourcePipeline: retry split_entities stage once when 0 entity delimiters
are found (free-tier models intermittently return short non-formatted
responses); save raw LLM response to <stage>-raw.md alongside prompts
- Return None (pause pipeline) rather than writing empty view file when
no entities found after max retries
- _http.py: wrap json.JSONDecodeError in LLMAPIError with body preview
- extract-entities.md: add explicit H2-heading format example to Output
Format section to prevent models from using inline "Section:" format
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extend PipelineStage with name, output_dir, output_macro,
split_entities, and macros fields for declarative pipeline config
- Add SourcePipeline class (pipeline.py) using simple @{macro}
substitution — no SQLite dependency, skip-if-exists per stage,
LLM retry on rate limits, git commit per source
- Add `markitect infospace process [GLOB_PATTERN]` CLI command with
--all, --provider, --model, --check-after-each, --no-commit flags
- Update infospace.yaml with output_dir, output_macro, split_entities,
and macros for each pipeline stage in the WoN example
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two root causes of metric fragmentation observed in collection checks:
1. Schema's Economic Domain used free-form examples ("labour economics,
trade theory") which overrode the enum in extraction-rules.md, causing
the LLM to produce multi-domain strings and non-canonical values.
Fix: schema now specifies the exact 7-value enum with descriptions.
2. Source Chapter had no format constraint, producing 9 different formats
for 7 chapters (full titles, mixed Roman/Arabic numerals, asterisks).
Fix: extraction-rules now mandate "Book [Roman], Chapter [n]" exactly.
These fixes are prerequisites for clean reprocessing (S3.2 continuation).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The SQLite artifact database is a derived cache regenerable from
committed files — no LLM calls needed. Added tutorial section
explaining why it is excluded and how to rebuild it after a fresh clone.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add OpenAIAdapter for the OpenAI chat completions API (apikey-chatgpt.txt
or OPENAI_API_KEY). Set default model to arcee-ai/trinity-large-preview:free
for the infospace pipeline and increase max_tokens from 4096 to 8192.
Reprocess chapter 05 with Trinity Large (was Gemini: 1 truncated entity,
now 19 complete entities). Process chapters 06 (Aurora Alpha, 10 entities)
and 07 (Trinity Large, 15 entities including regenerated violent-policy.md).
Canonical set now at 85 unique entities.
Add entity archive policy: entities are never silently deleted. Retired
entities move to output/entities/archive/ with a dated reason header.
New CLI option: --archive-entity <slug> --reason "...". The --list
output shows the archive count alongside the canonical set.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add GeminiAdapter calling Google's Generative Language REST API
(default model: gemini-2.5-flash). Register "gemini" as third
provider in the factory and CLI. Add rate-limit retry with
exponential backoff to the pipeline's _call_llm helper. Increase
default max_tokens from 2000 to 4096.
Process book-1-chapter-05 via Gemini free tier — 1 new entity
extracted (necessaries-conveniencies-and-amusements-of-life),
41 existing entities correctly skipped by dedup. Canonical set
now at 42 unique entities.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restructure entity storage from per-chapter subdirectories to a flat
canonical set in output/entities/. Each entity exists as a single file;
duplicates across chapters are detected by slug collision and skipped
(first occurrence wins). Chapter views use {{ include }} transclusion
to reference shared entity files.
Add @{existing_entities} macro to extract-entities template so the LLM
knows which entities already exist and focuses on genuinely new ones.
Refactor _call_llm() from _execute_llm() for callers that handle their
own file I/O. 41 unique entities from 4 chapters (2 duplicates removed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comprehensive walkthrough covering schema design, prompt templates,
artifact population, pipeline usage, LLM integration, git history
tracking, metrics, and how to complete the remaining 31 chapters.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 3 stages (entities, mappings, analysis) auto-generated.
1m53s wall time, 9,478 tokens (real), ~$0.07 est. cost.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Auto-generated mappings and analysis via Claude Code CLI adapter.
Entities were already present from a previous session.
Stats: 5m04s wall time, ~51K estimated tokens, ~$0.35 estimated cost.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements markitect/llm/ package with concrete LLMAdapter implementations:
- OpenRouterAdapter: HTTP via urllib with retry/backoff on 429/5xx
- ClaudeCodeAdapter: subprocess-based Claude CLI with stdin piping
- Factory pattern: create_adapter("openrouter") or create_adapter("claude-code")
- API key resolution chain: constructor > env var > project-root key file
- 42 unit tests, 2 integration tests (gated on API key / CLI availability)
Also adds the infospace-with-history example with Wealth of Nations VSM
analysis pipeline, templates, schemas, source chapters, and processed
output for chapters 1-2. process_chapters.py now supports --provider
and --model flags for automatic LLM-driven processing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This example demonstrates the full workflow of generating InfoTech primers
using MarkiTect's Prompt Dependency Resolution infrastructure.
Features demonstrated:
- Artifact creation and storage with content-based addressing
- PromptTemplate with @{macro} resolution across multiple spaces
- Automatic dependency tracking and graph construction
- Provenance tracing from outputs back to inputs
- Visualization export (Mermaid format)
- Incremental execution with change detection
Files added:
- generate_primers.py: Complete working example
- README.md: Quick start guide and architecture overview
- TUTORIAL.md: Comprehensive 500+ line tutorial
- templates/generate-primer.md: Template with macros
- artifacts/topics/: ETL and Microservices topic definitions
- artifacts/guidelines/: Authoring rules and research protocol
- prepdr/: Original manual system (preserved for reference)
Example output:
- Generates 2 primers (ETL, Microservices)
- Creates 8 artifacts across 4 information spaces
- Records 8 dependency edges in SQLite database
- Exports dependency graph visualization
Run with: cd examples/content-generator && python generate_primers.py
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit closes the schema-evolution topic (260105) by adding the final
deliverable (ADR schema) and fixing markdown schema support across commands.
**ADR Schema Created**:
- Comprehensive Architecture Decision Record validation schema
- 12 section classifications (7 required, 2 recommended, 2 optional, 3 improper/discouraged)
- Content pattern validation for ADR formatting rules (status dates, decision statements, rationale structure)
- Quality metrics for completeness (word counts, sentence counts)
- Follows title case naming convention (Status, Context, Decision, etc.)
**Markdown Schema Support Fixed**:
- Fixed `markitect validate` command to support .md schemas
- Added load_schema_from_path() for both .json and .md files
- Updated structural and semantic validation to use schema dict
- Fixed `markitect generate-stub` command to support .md schemas
- Uses load_schema_from_path() instead of direct JSON loading
- Created DocumentWrapper class in semantic_validator.py
- Extracts headings from AST tokens (heading_open, inline)
- Provides get_headings_by_level() interface expected by validators
- Enables section validation to work with real documents
**Topic Closure**:
- Updated SCHEMA_EVOLUTION_WORKPLAN.md with completion summary
- Phases 1-3: 100% complete (via Schema-of-Schemas and Semantic Validation)
- Phase 4: Deferred as future enhancement (15-20 sessions)
- Phase 5: 70% complete (docs done, CI/CD templates deferred)
- Created DONE.md with comprehensive task checklist
- Generated ADR template stub (examples/templates/adr-template.md)
- Moved topic from roadmap/ to history/260105-schema-evolution/
**Files Changed**:
- markitect/cli.py: Added markdown schema support to validate and generate-stub
- markitect/semantic_validator.py: Added DocumentWrapper class for AST parsing
- markitect/schemas/adr-schema-v1.0.md: New ADR validation schema (560 lines)
- examples/templates/adr-template.md: Generated ADR template stub
- history/260105-schema-evolution/: Moved completed topic to history
**Status**: Schema evolution topic successfully closed with ADR schema as final deliverable.
All schema commands now support markdown schemas. Section validation working correctly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated example documentation to use the new schema-of-schemas standard
with markdown schema format and multi-schema validation commands.
**Manpage Example Updates:**
- Changed schema reference from markdown-manpage-schema.json to manpage-schema-v1.0.md
- Updated all commands to use new multi-schema validation syntax
- Added examples of number-based validation (markitect schema-validate 2)
- Added examples of batch validation (--all, ranges, lists)
- Updated integration examples (CI/CD, pre-commit hooks, Makefile)
- Documented schema registry workflow
**Terminology Example Updates:**
- Changed schema reference from terminology-schema.json to terminology-schema-v1.0.md
- Updated all validation commands to use new CLI syntax
- Added examples of schema-list and numbered selection
- Added batch validation examples
- Updated GitHub Actions and pre-commit hook examples
- Documented schema registry access methods
**Key Changes:**
- All schema filenames now follow {domain}-schema-v{major}.{minor}.md convention
- Commands use schema registry with numbered or filename selection
- Batch validation examples added throughout
- Integration examples updated to new standard
- Documentation reflects markdown-first schema format
All schemas validated successfully against metaschema.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit completes Phase 2 of schema evolution work and establishes
a new example demonstrating schema usage for terminology documents.
## New Features
### Terminology Validation Example (examples/terminology/)
- Complete example terminology document with proper structure
- JSON schema with MarkiTect extensions for validation
- Demonstrates schema usage beyond manpages (glossaries, lexicons)
- Validates term structure: Definition, Synonyms, Related Terms, Examples
- Includes content control and quality validation rules
- Full documentation with usage examples and best practices
### Schema Registration System
- Registered terminology schema in markitect database
- Created schema catalog (markitect/schemas/schema-catalog.yaml)
- Copied schema to official location (markitect/schemas/)
- Provides metadata, features, and usage info for all schemas
### Improved schema-list Command
- Now displays creation timestamps in default output
- Table format includes Created/Updated columns
- Cleaner timestamp formatting (removed microseconds)
- Better visibility into when schemas were added
## Files Changed
Added:
- examples/terminology/README.md - Complete documentation
- examples/terminology/terminology-example.md - Example glossary
- examples/terminology/terminology-schema.json - Validation schema
- markitect/schemas/terminology-schema.json - Registered schema
- markitect/schemas/schema-catalog.yaml - Schema registry
Modified:
- markitect/cli.py - Enhanced schema-list with timestamps
- TODO.md - Documented Phase 2 completion and new example
Moved:
- SCHEMA_EVOLUTION_WORKPLAN.md → todo/ directory
## Schema Features Demonstrated
- Heading hierarchy validation (H1 → H2 → H3)
- Term structure validation with required/optional fields
- Content quality metrics (word counts, readability targets)
- MarkiTect extensions (x-markitect-sections, x-markitect-content-control)
- Classification system (required/recommended/optional/discouraged/improper)
## Usage
```bash
# List schemas with timestamps
markitect schema-list
# Validate terminology document
markitect validate glossary.md --schema terminology-schema.json
# View in table format
markitect schema-list --format table
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Comprehensively document the new classification system and content control
features added in Phase 1.
## Documentation Updates
### New Content Added
**1. Updated MarkiTect Extensions Section**
- Replaced deprecated x-markitect-required/recommended-sections
- Documented x-markitect-sections with five classification levels
- Documented x-markitect-content-control for content validation
**2. Added Section Classification System (150+ lines)**
- Detailed explanation of all five classification levels:
- required: Missing = ERROR
- recommended: Missing = WARNING
- optional: No validation impact
- discouraged: Present = WARNING
- improper: Present = ERROR
- Validation behavior for each classification
- JSON examples for each level
**3. Added Content Control Documentation**
- Pattern validation (required/discouraged/forbidden)
- Content quality metrics (word count, readability targets)
- Content instructions for authors
- Complete examples with explanations
**4. Updated Schema Design Best Practices**
- Replaced old extension examples with new classification system
- Added guidance on choosing appropriate classifications
- Examples showing required, recommended, optional, discouraged, improper
**5. Added Classification System Example**
- Complete working schema demonstrating all features
- Validation scenarios showing different outcomes
- Integration of sections and content-control extensions
## Changes Summary
**Lines Added**: ~200 lines of new documentation
**Sections Updated**: 4 major sections
**Examples Added**: 8 new code examples
**Key Topics Covered**:
- Five-level classification system (required → improper)
- Content pattern validation
- Quality metrics and readability targets
- Content instructions for document authors
- Validation behavior for each classification
- Complete working examples
## Validation
✅ Manual validates against improved markdown-manpage-schema.json
✅ All new features documented with examples
✅ Backward compatibility maintained
✅ Self-documenting: manual uses the features it documents
The manual now comprehensively documents the Phase 1 enhanced schema
system while itself validating against a schema using those features.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
