coulomb/markitect-main
2
0
Fork 0
Code Issues 61 Pull Requests Actions Projects 12 Releases 1 Wiki Activity

chore: establish schema-of-schemas workplan and reorganize roadmap

Browse Source 
This commit sets up the comprehensive workplan for implementing a
markdown-first schema management system with naming conventions,
versioning, and self-validation capabilities.

## Directory Reorganization

- Renamed `todo/` → `roadmap/` for better organization
- Created `roadmap/schema-of-schemas/` subdirectory
- Moved schema management planning artifacts to dedicated directory

## Planning Artifacts Created

### Workplan & Documentation
- **WORKPLAN.md** (19KB) - Comprehensive 6-phase implementation plan
- **SCHEMA_MANAGEMENT_PROPOSAL.md** - Full analysis with 4 options
- **SCHEMA_MANAGEMENT_SUMMARY.md** - Executive summary
- **README.md** - Quick reference guide

### Example Schema
- **examples/schemas/manpage-schema-v1.md** - Demonstrates markdown format

## Schema Management System Design

### Naming Convention
**Format:** `{domain}-schema-v{major}.{minor}.md`
**Examples:**
- `manpage-schema-v1.0.md`
- `terminology-schema-v1.0.md`
- `api-documentation-schema-v1.0.md`

### Markdown-First Format
Schemas will be markdown files with:
- YAML frontmatter for metadata
- Rich documentation sections
- Embedded JSON schema in code block
- Version history and examples

### Implementation Phases (8-10 days)

**Phase 0:** Planning & Setup ✅ (0.5 days) - COMPLETE
**Phase 1:** Filename Convention (1 day) - NEXT
**Phase 2:** Markdown Loader (2-3 days)
**Phase 3:** Schema-for-Schemas (2 days)
**Phase 4:** Schema Migration (1-2 days)
**Phase 5:** CLI & Documentation (1 day)
**Phase 6:** Testing & Validation (1 day)

### Goals

1. ✅ Establish naming convention
2. ⏳ Implement filename validation
3. ⏳ Create markdown schema loader
4. ⏳ Build schema-for-schemas metaschema
5. ⏳ Migrate 5 existing schemas (remove 2 duplicates)
6. ⏳ Update CLI and documentation

## Updated Tracking

### TODO.md
- Added Schema-of-Schemas as active work item
- Documented Phase 1 tasks and timeline
- Paused capability extraction work

### CHANGELOG.md
- Added schema management system to [Unreleased]
- Documented directory reorganization
- Added "In Progress" section for current work

## Next Steps

Begin Phase 1:
1. Implement schema_naming.py with validation
2. Add unit tests
3. Update CLI schema-ingest command
4. Create naming specification document

## Files Changed

- CHANGELOG.md - Added unreleased schema management features
- TODO.md - Updated active work tracking
- roadmap/ - Reorganized from todo/
- roadmap/schema-of-schemas/ - New planning directory
- examples/schemas/ - Example markdown schema

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-01-04 23:47:02 +01:00
parent 6df9b5df05
commit b6f95066a3
10 changed files with 2138 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

316
roadmap/ISSUE_DATAMODEL_OPTIMIZATION_GAMEPLAN.md Normal file
View File

@@ -0,0 +1,316 @@
# Issue Datamodel Optimization Gameplan
## Executive Summary
Comprehensive plan to optimize the issue object datamodel based on the Datamodel Optimization Specialist Agent analysis. This gameplan implements proven optimization patterns to achieve model consistency, performance improvements, and code reduction.
## Current State Analysis
### Identified Datamodel Structure
**Core Models:**
- `/home/worsch/markitect_project/domain/issues/models.py` - Domain Issue, IssueState, Label, LabelCategories
- `/home/worsch/markitect_project/gitea/models.py` - API-focused Issue, Label, User, Milestone
- `/home/worsch/markitect_project/markitect/issues/activity_tracker.py` - IssueActivity, ActivityType
**Architecture Strengths:**
- Clean Architecture with domain separation
- Rich activity tracking with cost integration
- Plugin architecture for different backends (Gitea, Local)
- Well-defined repository interfaces
**Identified Issues:**
- Multiple Issue models causing inconsistencies
- Label-based state management complexity
- Missing convenience methods and properties
- Potential serialization inefficiencies
- No unified datamodel optimization approach
## Optimization Strategy
### PRIORITY 1: Unified Issue Model Architecture
**Goal**: Create single source of truth Issue model
**Implementation Plan:**
1. **Unified Issue Model** in `domain/issues/models.py`:
- Combine best features from domain and Gitea models
- Add external_id field for API mappings
- Implement cached label categorization for performance
- Add convenience properties for common operations
2. **Backward Compatibility Layer**:
- Create adapter classes for existing API interactions
- Maintain existing interfaces during transition
- Gradual migration path for all consumers
**Expected Benefits:**
- Single source of truth eliminates mapping complexity
- Consistent interface across all usage patterns
- Reduced maintenance overhead
### PRIORITY 2: Enhanced State Management
**Goal**: Unified state enum with mapping capabilities
**Implementation Plan:**
1. **Enhanced IssueState Enum**:
- Add mapping methods for string conversion
- Include display names for UI presentation
- Support for state transitions and validation
2. **State Management Service**:
- Centralized state transition logic
- Business rules enforcement
- Audit trail integration
**Expected Benefits:**
- Consistent state representations across layers
- Improved state validation and transitions
- Better UI/CLI presentation
### PRIORITY 3: Optimized Label System
**Goal**: 60-80% performance improvement in label processing
**Implementation Plan:**
1. **Unified Label Model**:
- Frozen dataclass for immutability
- Cached category detection using @cached_property
- Efficient value extraction methods
2. **Single-Pass Label Categorization**:
- Replace multiple iterations with single categorization
- Cache results for repeated access
- Optimized label filtering methods
**Expected Benefits:**
- Significant performance improvement in label operations
- Reduced memory usage through caching
- Cleaner label manipulation APIs
### PRIORITY 4: Enhanced Database Integration
**Goal**: Proper referential integrity and data consistency
**Implementation Plan:**
1. **Database Schema Enhancement**:
- Add issues table with proper constraints
- Implement foreign key relationships
- Create indices for performance
2. **Migration Strategy**:
- Create 002_issue_integration.sql migration
- Data integrity validation
- Rollback procedures
**Expected Benefits:**
- Data integrity through foreign key constraints
- Better query performance with proper indices
- Cleaner database structure
### PRIORITY 5: Repository Pattern Enhancement
**Goal**: High-performance repository with caching
**Implementation Plan:**
1. **Cached Repository Implementation**:
- TTL-based caching layer
- Efficient query patterns
- Bulk operation optimizations
2. **Query Optimization**:
- Label-based filtering improvements
- Pagination and sorting enhancements
- Connection pooling considerations
**Expected Benefits:**
- Improved application performance
- Reduced database load
- Better scalability for larger datasets
### PRIORITY 6: Activity Tracking Integration
**Goal**: Integrated activity tracking with issue lifecycle
**Implementation Plan:**
1. **Enhanced Activity Model**:
- Tight coupling with issue state changes
- Automated activity generation for lifecycle events
- Rich activity querying capabilities
2. **Lifecycle Integration**:
- Automatic activity logging on state changes
- Business logic triggers for activities
- Activity-driven notifications
**Expected Benefits:**
- Complete audit trail for all issue changes
- Automated activity tracking
- Better business intelligence capabilities
## Implementation Phases
### Phase 1: Model Unification (Week 1)
**Scope**: Create unified Issue model with backward compatibility
**Tasks:**
1. Analyze current Issue model usage patterns
2. Design unified Issue model with all required fields
3. Implement backward compatibility adapters
4. Create migration utilities
5. Update unit tests for new model
**Deliverables:**
- Enhanced `domain/issues/models.py` with unified Issue model
- Compatibility adapter classes
- Comprehensive test coverage
- Migration documentation
**Success Criteria:**
- All existing tests pass with adapter layer
- No breaking changes to external interfaces
- Performance benchmarks established
### Phase 2: Performance Optimization (Week 2)
**Scope**: Implement caching and optimize label processing
**Tasks:**
1. Implement cached label categorization
2. Add performance-optimized properties
3. Create repository caching layer
4. Optimize database queries
5. Add performance monitoring
**Deliverables:**
- Optimized label processing with caching
- High-performance repository implementation
- Database query optimizations
- Performance benchmark results
**Success Criteria:**
- 60-80% improvement in label processing performance
- Repository query response time improvements
- Memory usage optimization validated
### Phase 3: Integration Enhancement (Week 3)
**Scope**: Integrate activity tracking and enhance relationships
**Tasks:**
1. Enhance database schema with proper constraints
2. Integrate activity tracking with issue lifecycle
3. Implement cost allocation relationships
4. Add comprehensive validation
5. Create data migration scripts
**Deliverables:**
- Enhanced database schema with foreign keys
- Integrated activity tracking system
- Data validation framework
- Migration scripts and procedures
**Success Criteria:**
- Database integrity constraints properly enforced
- Activity tracking automatically captures all issue changes
- Data validation prevents inconsistent states
### Phase 4: Testing & Migration (Week 4)
**Scope**: Comprehensive testing and production deployment
**Tasks:**
1. Comprehensive integration testing
2. Performance benchmarking
3. Production data migration
4. Documentation updates
5. Training and knowledge transfer
**Deliverables:**
- Complete test suite with integration tests
- Performance benchmark reports
- Production migration procedures
- Updated documentation
- Training materials
**Success Criteria:**
- 100% test coverage for optimized models
- Performance improvements validated in production-like environment
- Successful data migration without data loss
- Team trained on new datamodel capabilities
## Risk Mitigation
### Technical Risks
1. **Backward Compatibility**: Maintain adapter layer during transition
2. **Data Integrity**: Comprehensive validation and migration scripts
3. **Performance Regression**: Extensive benchmarking and monitoring
4. **Complex Dependencies**: Careful dependency analysis and phased rollout
### Mitigation Strategies
1. **Gradual Migration**: Phase-by-phase implementation with rollback points
2. **Comprehensive Testing**: Unit, integration, and performance tests
3. **Monitoring**: Real-time performance and error monitoring
4. **Documentation**: Clear migration guides and troubleshooting procedures
## Success Metrics
### Quantitative Goals
- **Performance**: 60-80% improvement in label processing
- **Code Reduction**: 15-25 lines of code reduction per optimized model
- **Test Reliability**: 90%+ reduction in test failures due to model inconsistencies
- **Query Performance**: 50%+ improvement in database query response times
### Qualitative Goals
- **Maintainability**: Easier to modify and extend issue models
- **Developer Experience**: Cleaner APIs and more intuitive interfaces
- **Data Consistency**: Reliable data integrity across all operations
- **System Reliability**: Reduced bugs due to model inconsistencies
## Resource Requirements
### Development Team
- 1 Senior Developer (Lead optimization implementation)
- 1 Database Specialist (Schema and migration work)
- 1 QA Engineer (Testing and validation)
### Timeline
- **Total Duration**: 4 weeks
- **Effort Estimate**: ~6-8 person-weeks
- **Critical Path**: Phase 1 (Model Unification) → Phase 2 (Performance) → Phase 3 (Integration) → Phase 4 (Testing)
### Infrastructure
- Development and staging environments for testing
- Performance testing tools and monitoring
- Database migration and rollback capabilities
## Next Steps
### Immediate Actions (Next 1-2 Days)
1. **Team Alignment**: Review gameplan with development team
2. **Environment Setup**: Prepare development and testing environments
3. **Baseline Establishment**: Create performance and functionality baselines
4. **Detailed Planning**: Break down Phase 1 tasks into specific work items
### Week 1 Kickoff
1. Begin Phase 1 implementation
2. Set up continuous integration for new model testing
3. Establish performance monitoring baselines
4. Create detailed migration documentation
---
## Appendix
### Related Documents
- [Datamodel Optimization Specialist Agent Documentation](../docs/sub_agents/datamodel_optimizer.md)
- [Datamodel Optimizer Tool](../tools/datamodel_optimizer.py)
- [Issue #126 - IssueActivity Optimization Success Case](../docs/issues/issue_126_analysis.md)
### References
- Clean Architecture Principles
- Domain-Driven Design Patterns
- Database Design Best Practices
- Performance Optimization Techniques
---
*This gameplan provides a comprehensive roadmap for optimizing the issue datamodel while maintaining system stability, performance, and developer productivity. The phased approach ensures risk mitigation while delivering measurable improvements at each stage.*

75
roadmap/RelevantClaudeIssues.md Normal file
View File

@@ -0,0 +1,75 @@
# Relevant Claude Code Issues
## Introduction
This document tracks Claude Code issues that directly impact our development workflows for the MarkiTect project. Each issue section provides a clear problem description, affected workflows, related GitHub issues for monitoring, available workarounds, and resolution tracking information.
**Purpose:**
- Document blocking Claude Code issues affecting project development
- Provide centralized tracking of GitHub issues to monitor for fixes
- Maintain awareness of workarounds and their trade-offs
- Enable quick assessment of when normal workflows can resume
**Maintenance:**
- Update when new blocking issues are discovered
- Check GitHub issue status weekly and update resolution monitoring
- Remove resolved issues after confirming fixes work in our environment
- Maintained by the agent-claude-documentation subagent as part of issue tracking responsibilities
**🎯 CRITICAL WORKFLOW REMINDER:**
When discussing project issues (not Claude Code issues), ALWAYS fetch from Gitea first. Gitea is the source of truth for all issue assessment, feasibility evaluation, and implementation planning. Local files are insufficient for decision-making about issues. See ISSUE_WORKFLOW_REMINDER.md for complete workflow.
---
## Resolved Issues
### ✅ RESOLVED: Custom Subagents Not Available
### Problem Description (Historical)
Custom subagents defined in `.claude/agents/` directory were not being detected or made available as `subagent_type` options in the Task tool. This prevented the intended workflow of using specialized subagents for domain-specific tasks like project management, documentation, or Claude Code expertise.
The custom subagent system appeared completely broken, with agents not being recognized despite proper YAML frontmatter configuration.
### Resolution Details
**Resolved Date:** 2025-09-23
**Resolution Method:** Issue appears to have been fixed in current Claude Code version
**Verification:** Successfully tested the following subagents:
- ✅ `general-purpose` - Full tool access for complex multi-step tasks
- ✅ `agent-claude-documentation` - Specialized for Claude Code documentation and features
- ✅ `agent-project-management` - Specialized for MarkiTect project status and development planning
**Resolution Confirmation Steps:**
1. ✅ Custom agents now appear as valid `subagent_type` options in Task tool
2. ✅ Successfully invoked custom subagents for specialized tasks
3. ✅ All intended multi-agent workflows are now functional
4. ✅ No workarounds needed - normal operation restored
### Impact of Resolution
**Restored Workflows:**
- **Specialized Task Delegation**: ✅ Custom subagents working for domain-specific tasks
- **Project Management**: ✅ agent-project-management subagent functional for status tracking and planning
- **Documentation Assistance**: ✅ agent-claude-documentation subagent operational for Claude Code expertise
- **Task Decomposition**: ✅ Full subagent ecosystem available
- **Workflow Automation**: ✅ Multi-agent collaborative workflows enabled
**Workarounds No Longer Needed:**
- No need to downgrade Claude Code versions
- No need to use only built-in general-purpose agent
- No need for manual role assignment
### Related GitHub Issues (Historical Reference)
- [#4623](https://github.com/anthropics/claude-code/issues/4623) - Custom agents not being detected
- [#4728](https://github.com/anthropics/claude-code/issues/4728) - Agent discovery mechanism broken
- [#4626](https://github.com/anthropics/claude-code/issues/4626) - Custom agents missing from UI
- [#5185](https://github.com/anthropics/claude-code/issues/5185) - Agent configuration not working
- [#4182](https://github.com/anthropics/claude-code/issues/4182) - Task tool limitations for nested agents
---
## Monitoring Schedule
This document should be reviewed weekly to check for issue resolution and update status. The agent-claude-documentation subagent is responsible for maintaining this tracking and updating the project team when workflows can resume normal operation.

787
roadmap/SCHEMA_EVOLUTION_WORKPLAN.md Normal file
View File

@@ -0,0 +1,787 @@
# MarkiTect Schema Evolution Workplan
## Executive Summary
**Current State**: MarkiTect validates document structure via JSON Schema, but is too rigid (exact counts) and structure-only (no content guidance).
**Target State**: A flexible schema system with content control, section classification, multi-schema conformance, and blueprint-based document generation.
**Timeline**: 5 phases, 15-20 development sessions, approximately 8-10 weeks.
---
## Problem Analysis
### Current Limitations
#### 1. Structural Rigidity
**Problem**: Auto-generated schemas use exact counts
```json
"paragraphs": { "minItems": 86, "maxItems": 86 }
```
**Impact**: Schemas are document-specific, not reusable patterns.
#### 2. Binary Structure Validation
**Problem**: Elements are either valid or invalid, no classification.
**Need**: Required, Recommended, Optional, Discouraged, Improper classifications.
#### 3. No Content Guidance
**Problem**: Schemas validate structure exists, not what content belongs there.
**Need**: Content instructions, semantic patterns, quality expectations.
#### 4. Single Schema Limitation
**Problem**: Documents can only conform to one schema.
**Need**: Multi-schema conformance (e.g., "manpage" + "API reference" + "tutorial").
#### 5. Template Generation Gap
**Problem**: `generate-stub` creates outline, but no content guidance or data binding.
**Need**: Blueprint system with content instructions and data templates.
---
## Proposed Architecture
### Three-Layer System
```
┌─────────────────────────────────────────────┐
│ BLUEPRINT LAYER │
│ (Multi-schema + Content + Data Templates) │
└─────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────┐
│ SCHEMA LAYER (Enhanced) │
│ (Structure + Classification + Instructions) │
└─────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────┐
│ VALIDATION LAYER │
│ (AST Validation + Content Analysis) │
└─────────────────────────────────────────────┘
```
### Key Concepts
**1. Schema Classification System**
- **Required**: Must be present, validation fails if missing
- **Recommended**: Should be present, warning if missing
- **Optional**: May be present, no validation impact
- **Discouraged**: Should not be present, warning if present
- **Improper**: Must not be present, validation fails if present
**2. Content Control**
- **Content Instructions**: Human-readable guidance for section content
- **Content Patterns**: Regex/template patterns for content validation
- **Content Quality Metrics**: Word count, readability, completeness scoring
**3. Multi-Schema Conformance**
- Documents can conform to multiple schemas simultaneously
- Schema composition and inheritance
- Conflict resolution strategies
**4. Blueprint System**
- Schemas + Instructions + Data Templates = Blueprints
- Blueprints generate documents with content guidance
- Data binding for dynamic document generation
---
## Phase 1: Enhanced Schema Format
**Goal**: Extend JSON Schema with MarkiTect-specific content control extensions.
### 1.1 Schema Classification Extensions
**New Properties**:
```json
{
"x-markitect-sections": {
"SYNOPSIS": {
"classification": "required",
"heading_level": 2,
"position": "after_title",
"content_instruction": "Brief command syntax showing all options",
"min_code_blocks": 1,
"max_code_blocks": 3
},
"EXAMPLES": {
"classification": "recommended",
"heading_level": 2,
"content_instruction": "Practical usage examples with explanations",
"min_code_blocks": 3,
"warning_if_missing": "Examples greatly improve documentation usability"
},
"DEPRECATED": {
"classification": "discouraged",
"heading_level": 2,
"warning_message": "DEPRECATED sections should be moved to historical docs"
},
"INTERNAL_NOTES": {
"classification": "improper",
"heading_level": 2,
"error_message": "Internal notes must not appear in published documentation"
}
}
}
```
### 1.2 Content Control Extensions
**New Properties**:
```json
{
"x-markitect-content-control": {
"synopsis_section": {
"min_paragraphs": 1,
"max_paragraphs": 3,
"required_patterns": [
"\\*\\*[a-z-]+\\*\\*.*\\[.*\\]" // Bold command with args
],
"content_quality": {
"min_words": 10,
"max_words": 100,
"readability_target": "technical"
},
"content_instructions": [
"Show command name in bold",
"Include all major options in synopsis",
"Use italic for arguments and placeholders"
]
}
}
}
```
### 1.3 Flexible Structure Constraints
**Replace rigid counts with ranges and classifications**:
```json
{
"properties": {
"headings": {
"properties": {
"level_2": {
"items": {
"properties": {
"content": {
"oneOf": [
{"const": "SYNOPSIS", "x-markitect-classification": "required"},
{"const": "DESCRIPTION", "x-markitect-classification": "required"},
{"const": "EXAMPLES", "x-markitect-classification": "recommended"},
{"const": "SEE ALSO", "x-markitect-classification": "optional"}
]
}
}
},
"minItems": 2, // At least required sections
"maxItems": 30 // Reasonable upper bound
}
}
}
}
}
```
### Tasks
- [ ] **Task 1.1**: Define `x-markitect-sections` schema extension format
- [ ] **Task 1.2**: Define `x-markitect-content-control` schema extension format
- [ ] **Task 1.3**: Update metaschema to validate new extensions
- [ ] **Task 1.4**: Create schema examples demonstrating all classifications
- [ ] **Task 1.5**: Document schema extension format
**Duration**: 3-4 sessions
**Dependencies**: None
**Deliverables**: Enhanced schema format specification, updated metaschema
---
## Phase 2: Schema Refinement Tools
**Goal**: Tools to transform rigid auto-generated schemas into flexible, classified schemas.
### 2.1 Schema Analysis Tool
**Command**: `markitect schema-analyze`
Analyzes existing schema and suggests improvements:
```bash
markitect schema-analyze rigid-schema.json
# Output:
⚠️ Exact counts detected (86 paragraphs)
Suggestion: Use range 50-150 for flexibility
⚠️ All sections unclassified
Suggestion: Classify sections as required/recommended/optional
⚠️ No content instructions
Suggestion: Add content guidance for key sections
✨ Run: markitect schema-refine rigid-schema.json
```
### 2.2 Schema Refinement Tool
**Command**: `markitect schema-refine`
Interactive or automated schema refinement:
```bash
# Automated: Apply common refinements
markitect schema-refine rigid-schema.json \
--loosen-counts \
--add-classifications \
--output flexible-schema.json
# Interactive: Guided refinement
markitect schema-refine rigid-schema.json --interactive
```
**Refinement Operations**:
- Convert exact counts to ranges (configurable tolerance)
- Classify sections based on conventions
- Add content instructions from templates
- Merge multiple schemas for common patterns
### 2.3 Schema Composition Tool
**Command**: `markitect schema-compose`
Combine multiple schemas:
```bash
# Create composite schema
markitect schema-compose \
--base manpage-schema.json \
--extend api-reference-schema.json \
--extend tutorial-schema.json \
--output composite-schema.json
```
### Tasks
- [ ] **Task 2.1**: Implement `schema-analyze` command
- [ ] **Task 2.2**: Implement `schema-refine` command with loosening logic
- [ ] **Task 2.3**: Implement `schema-refine --interactive` mode
- [ ] **Task 2.4**: Implement `schema-compose` command
- [ ] **Task 2.5**: Create schema refinement rule library
**Duration**: 3-4 sessions
**Dependencies**: Phase 1 complete
**Deliverables**: Schema analysis, refinement, and composition tools
---
## Phase 3: Enhanced Validation Engine
**Goal**: Validate classification levels, content patterns, and multi-schema conformance.
### 3.1 Classification-Aware Validation
**Validation Levels**:
```python
class ValidationResult:
status: Literal["valid", "valid_with_warnings", "invalid"]
errors: List[ValidationError] # Required/Improper violations
warnings: List[ValidationWarning] # Recommended/Discouraged violations
suggestions: List[str] # Optional improvements
```
**Example Output**:
```bash
markitect validate document.md schema.json --detailed-errors
❌ ERRORS (validation failed)
- Missing required section: SYNOPSIS
- Improper section present: INTERNAL_NOTES
⚠️ WARNINGS
- Missing recommended section: EXAMPLES
- Discouraged section present: DEPRECATED
💡 SUGGESTIONS
- Consider adding optional section: PERFORMANCE
- Content quality: DESCRIPTION section below recommended word count (45/100)
Status: INVALID (2 errors, 2 warnings)
```
### 3.2 Content Pattern Validation
**Validate content patterns**:
```python
# Schema specifies required patterns
"synopsis_section": {
"required_patterns": [
r"\*\*command\*\*", # Bold command name
r"\[.*\]" # Options in brackets
],
"discouraged_patterns": [
r"TODO", # No TODOs in published docs
r"FIXME"
]
}
```
### 3.3 Multi-Schema Validation
**Command**: `markitect validate --schemas`
```bash
# Validate against multiple schemas
markitect validate api-doc.md \
--schemas manpage.json,api-reference.json,tutorial.json \
--require-all
# Output shows conformance to each schema
✅ manpage.json: VALID
✅ api-reference.json: VALID (2 warnings)
❌ tutorial.json: INVALID (missing required section: GETTING STARTED)
Overall: INVALID (must conform to all schemas)
```
### 3.4 Content Quality Metrics
**Validate content quality**:
```bash
markitect validate document.md schema.json --quality-check
📊 Content Quality Report
- Word count: 487 (target: 300-1000) ✅
- Code examples: 3 (minimum: 3) ✅
- Readability: Technical (appropriate) ✅
- Link validity: 12/12 valid ✅
- Heading hierarchy: Valid ✅
Quality Score: 95/100
```
### Tasks
- [ ] **Task 3.1**: Implement classification-aware validator
- [ ] **Task 3.2**: Implement content pattern validation
- [ ] **Task 3.3**: Implement multi-schema validation
- [ ] **Task 3.4**: Implement content quality metrics
- [ ] **Task 3.5**: Enhanced error reporting with suggestions
**Duration**: 4-5 sessions
**Dependencies**: Phase 1 complete
**Deliverables**: Enhanced validation engine, quality metrics
---
## Phase 4: Blueprint System
**Goal**: Document generation system with schemas + content instructions + data templates.
### 4.1 Blueprint Format
**Blueprint Structure**:
```json
{
"$blueprint": "1.0",
"name": "api-documentation-blueprint",
"description": "Blueprint for API endpoint documentation",
"schemas": [
"manpage-schema.json",
"api-reference-schema.json"
],
"content_model": {
"synopsis": {
"template": "**{{command}}** [*OPTIONS*] *{{primary_argument}}*",
"data_source": "command_metadata.json",
"instruction": "Brief command syntax"
},
"description": {
"template": "{{description}}\n\nThis endpoint {{purpose}}.",
"min_paragraphs": 2,
"instruction": "Explain what the endpoint does and why to use it"
},
"parameters": {
"template": "{{#each parameters}}\n**{{name}}** *{{type}}*\n: {{description}}\n{{/each}}",
"data_source": "parameters",
"instruction": "Document all parameters with types and descriptions"
}
},
"data_schema": {
"type": "object",
"properties": {
"command": {"type": "string"},
"primary_argument": {"type": "string"},
"description": {"type": "string"},
"purpose": {"type": "string"},
"parameters": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"type": {"type": "string"},
"description": {"type": "string"}
}
}
}
}
},
"generation_rules": {
"heading_style": "atx",
"code_fence_style": "backticks",
"line_length": 80,
"include_metadata": true
}
}
```
### 4.2 Blueprint Commands
**Create Blueprint**:
```bash
# From existing schema
markitect blueprint-create --from-schema api-schema.json \
--output api-blueprint.json
# Interactive creation
markitect blueprint-create --interactive
```
**Generate from Blueprint**:
```bash
# Generate with data file
markitect blueprint-generate api-blueprint.json \
--data endpoint-data.json \
--output api-doc.md
# Generate with inline data
markitect blueprint-generate api-blueprint.json \
--data '{"command": "api-call", "description": "Make API call"}' \
--output api-doc.md
# Batch generation
markitect blueprint-generate-batch api-blueprint.json \
--data-dir ./endpoints/ \
--output-dir ./docs/api/
```
**Validate Blueprint**:
```bash
# Validate blueprint format
markitect blueprint-validate api-blueprint.json
# Test blueprint generation
markitect blueprint-test api-blueprint.json \
--sample-data test-data.json
```
### 4.3 Template Engine Integration
**Handlebars-style templates with MarkiTect extensions**:
```markdown
# {{command}}(1) - {{title}}
## SYNOPSIS
**{{command}}** {{#each options}}[*{{this}}*] {{/each}}*{{argument}}*
## DESCRIPTION
{{description}}
{{#markitect-section "technical-details"}}
Technical implementation details for {{command}}.
{{/markitect-section}}
## PARAMETERS
{{#each parameters}}
**--{{name}}** *{{type}}*
: {{description}}
: {{#if default}}Default: `{{default}}`{{/if}}
{{/each}}
{{#markitect-code-block "bash"}}
# Example usage
{{command}} {{#each examples.[0].args}}{{this}} {{/each}}
{{/markitect-code-block}}
```
### Tasks
- [ ] **Task 4.1**: Define blueprint format specification
- [ ] **Task 4.2**: Implement `blueprint-create` command
- [ ] **Task 4.3**: Implement `blueprint-generate` command
- [ ] **Task 4.4**: Implement template engine with Handlebars
- [ ] **Task 4.5**: Implement `blueprint-validate` command
- [ ] **Task 4.6**: Implement batch generation
- [ ] **Task 4.7**: Create blueprint library (common patterns)
**Duration**: 5-6 sessions
**Dependencies**: Phases 1 and 3 complete
**Deliverables**: Blueprint system, template engine, generation commands
---
## Phase 5: Documentation and Integration
**Goal**: Comprehensive documentation, examples, and ecosystem integration.
### 5.1 Documentation Suite
**Documents to Create**:
- [ ] Schema Evolution Guide (why and how)
- [ ] Schema Classification Reference
- [ ] Content Control Specification
- [ ] Blueprint System Guide
- [ ] Schema Design Best Practices
- [ ] Migration Guide (old schemas → new format)
- [ ] API Reference for programmatic usage
### 5.2 Example Gallery
**Create comprehensive examples**:
- [ ] Manpage blueprint (already started)
- [ ] API documentation blueprint
- [ ] Tutorial document blueprint
- [ ] Architecture Decision Record (ADR) blueprint
- [ ] RFC/specification blueprint
- [ ] Meeting notes blueprint
- [ ] Project README blueprint
### 5.3 CLI Integration
**Update existing commands**:
```bash
# schema-generate with classification
markitect schema-generate example.md \
--classify-sections \
--add-instructions \
--flexible \
--output smart-schema.json
# validate with multiple schemas
markitect validate doc.md \
--schemas schema1.json,schema2.json \
--classification-aware \
--quality-check
# generate-stub enhanced
markitect generate-stub schema.json \
--include-instructions \
--sample-content \
--output template.md
```
### 5.4 CI/CD Integration Templates
**Provide ready-to-use integrations**:
GitHub Actions:
```yaml
- name: Validate Documentation
uses: markitect/validate-action@v1
with:
schemas: docs/schemas/*.json
files: docs/**/*.md
classification-aware: true
fail-on: errors
warn-on: missing-recommended
```
Pre-commit hook:
```bash
#!/bin/bash
markitect validate-changed --schemas docs/schemas/ \
--classification-aware \
--fail-on errors
```
### Tasks
- [ ] **Task 5.1**: Write comprehensive documentation suite
- [ ] **Task 5.2**: Create example gallery with 7+ blueprints
- [ ] **Task 5.3**: Update all CLI commands for new features
- [ ] **Task 5.4**: Create CI/CD integration templates
- [ ] **Task 5.5**: Write migration guide for existing schemas
- [ ] **Task 5.6**: Create video tutorials/screencasts
**Duration**: 3-4 sessions
**Dependencies**: All previous phases complete
**Deliverables**: Complete documentation, examples, integrations
---
## Implementation Strategy
### Development Approach
**1. Test-Driven Development**
- Write tests for each classification level
- Test schema refinement transformations
- Test blueprint generation with various data
- Test multi-schema validation
**2. Backward Compatibility**
- Existing schemas continue to work
- New features are opt-in via extensions
- Clear migration path documented
**3. Incremental Rollout**
- Phase 1: Can be used immediately after completion
- Each phase delivers user value independently
- Later phases build on earlier phases
**4. Community Feedback**
- Alpha release after Phase 1
- Beta release after Phase 3
- Stable release after Phase 5
### Technical Considerations
**Schema Format**:
- JSON Schema draft-07 as foundation
- MarkiTect extensions namespaced with `x-markitect-`
- Validation via metaschema
- Clear upgrade path to future JSON Schema versions
**Performance**:
- Cache compiled schemas
- Lazy validation for large documents
- Parallel validation for multiple schemas
- Optimize content pattern matching
**API Design**:
- Programmatic access to all features
- Python API for schema manipulation
- Plugin system for custom validators
- Extensible template engine
---
## Success Metrics
### Phase 1 Success
- ✅ Schema with all 5 classifications validates correctly
- ✅ Content instructions appear in generated stubs
- ✅ Metaschema validates all extension formats
### Phase 2 Success
- ✅ Rigid schema refined to flexible schema automatically
- ✅ Multiple schemas composed without conflicts
- ✅ Interactive refinement completes end-to-end
### Phase 3 Success
- ✅ Validation distinguishes errors from warnings
- ✅ Content patterns detected and reported
- ✅ Multi-schema validation works with 3+ schemas
- ✅ Quality metrics provide actionable feedback
### Phase 4 Success
- ✅ Blueprint generates valid document from data
- ✅ Generated document validates against source schemas
- ✅ Batch generation processes 100+ documents
- ✅ Template engine supports complex logic
### Phase 5 Success
- ✅ Documentation covers all features
- ✅ 7+ working blueprint examples
- ✅ CI/CD integrations work in real projects
- ✅ Migration guide successfully upgrades old schemas
---
## Risk Assessment
### Technical Risks
**Risk**: Schema format complexity
**Mitigation**: Clear examples, validation tools, gradual adoption
**Risk**: Performance degradation with complex schemas
**Mitigation**: Caching, optimization, benchmarking
**Risk**: Template engine security (code injection)
**Mitigation**: Sandboxed execution, no eval, strict parsing
### Adoption Risks
**Risk**: Breaking changes to existing workflows
**Mitigation**: Full backward compatibility, opt-in features
**Risk**: Learning curve for new features
**Mitigation**: Excellent documentation, examples, tutorials
**Risk**: Feature bloat
**Mitigation**: Keep core simple, advanced features optional
---
## Future Enhancements (Post-MVP)
### Potential Future Features
**1. Semantic Validation**
- AI-powered content quality checking
- Grammar and style validation
- Factual consistency checking
- Link and reference validation
**2. Visual Schema Editor**
- Web-based GUI for schema creation
- Visual blueprint designer
- Live preview of generated documents
- Drag-and-drop section arrangement
**3. Schema Marketplace**
- Community schema repository
- Reusable blueprint library
- Rating and reviews system
- Version management
**4. Advanced Blueprint Features**
- Conditional sections based on data
- Dynamic schema selection
- Multi-language support
- Custom helper functions
**5. Integration Ecosystem**
- IDE plugins (VS Code, JetBrains)
- Documentation platforms (Read the Docs, Docusaurus)
- CMS integrations (Contentful, Strapi)
- Static site generators (Hugo, Jekyll)
---
## Conclusion
This workplan transforms MarkiTect from a structural validator to a comprehensive document control system:
**Current**: Rigid structure validation
**Target**: Flexible content control with blueprints
**Key Improvements**:
1. ✨ Classification system (required → improper)
2. ✨ Content guidance and instructions
3. ✨ Multi-schema conformance
4. ✨ Blueprint-based generation
5. ✨ Quality metrics and analysis
**Timeline**: ~8-10 weeks for full implementation
**Value**: Complete CMS-like document control for markdown
The system remains true to MarkiTect's philosophy of treating markdown as structured data while adding the flexibility and guidance needed for real-world content management.
---
## Next Steps
1. **Review and refine** this workplan
2. **Prioritize phases** based on user needs
3. **Create detailed specifications** for Phase 1
4. **Set up development environment** for new features
5. **Begin implementation** with TDD approach
**First Implementation Task**: Define `x-markitect-sections` format specification

94
roadmap/schema-of-schemas/README.md Normal file
View File

@@ -0,0 +1,94 @@
# Schema-of-Schemas Implementation
**Project:** Markdown-First Schema System
**Status:** Planning → Implementation
**Timeline:** 8-10 days
## Quick Links
- **[WORKPLAN.md](./WORKPLAN.md)** - Detailed implementation plan with phases
- **[SCHEMA_MANAGEMENT_PROPOSAL.md](./SCHEMA_MANAGEMENT_PROPOSAL.md)** - Full analysis and options
- **[SCHEMA_MANAGEMENT_SUMMARY.md](./SCHEMA_MANAGEMENT_SUMMARY.md)** - Executive summary
## What We're Building
### Goals
1. ✅ Filename convention: `{domain}-schema-v{version}.md`
2. ✅ Markdown-first schema format (documentation + embedded JSON)
3. ✅ Schema-for-schemas to validate all schemas
4. ✅ Migrate existing schemas to new format
5. ✅ Clean up duplicate/legacy schemas
### Why
- **Consistency:** Enforced naming and versioning
- **Alignment:** Markdown-first matches MarkiTect philosophy
- **Documentation:** Rich docs alongside schemas
- **Validation:** Schema-for-schemas ensures quality
- **Maintainability:** Clear versions and structure
## Implementation Phases
1. **Phase 0:** Planning & Setup (0.5 days) ← **Current**
2. **Phase 1:** Filename Convention (1 day)
3. **Phase 2:** Markdown Loader (2-3 days)
4. **Phase 3:** Schema-for-Schemas (2 days)
5. **Phase 4:** Schema Migration (1-2 days)
6. **Phase 5:** CLI & Docs (1 day)
7. **Phase 6:** Testing (1 day)
## Current Status
### Completed
- [x] Directory structure created
- [x] Planning documents moved to roadmap
- [x] Comprehensive workplan written
- [x] Example markdown schema created
### Next Steps
1. Complete Phase 0 planning artifacts
2. Begin Phase 1 implementation
3. Checkpoint review after Phase 1
## Key Decisions
### Naming Convention
**Format:** `{domain}-schema-v{major}.{minor}.md`
**Example:** `manpage-schema-v1.0.md`
### Schema Format
**Markdown with embedded JSON:**
```markdown
---
schema-id: "https://markitect.dev/schemas/manpage/v1"
version: "1.0.0"
---
# Manpage Schema v1.0
[Documentation...]
## Schema Definition
```json
{ ... JSON schema ... }
```
```
### Schema Migration Plan
```
Old → New
──────────────────────────────────────────────────
terminology-schema.json → terminology-schema-v1.0.md
api-documentation → api-documentation-schema-v1.0.md
enhanced-manpage → manpage-schema-v2.0.md
markdown-manpage → REMOVE (duplicate)
markdown-manpage-schema.json → REMOVE (duplicate)
```
## Progress Tracking
Track progress in: `roadmap/schema-of-schemas/IMPLEMENTATION_LOG.md` (to be created)
## Questions?
See the full workplan for detailed implementation steps, risks, and mitigation strategies.

569
roadmap/schema-of-schemas/SCHEMA_MANAGEMENT_PROPOSAL.md Normal file
View File

@@ -0,0 +1,569 @@
# Schema Management Proposal
**Status:** Draft
**Created:** 2026-01-04
**Author:** Analysis of current state and proposed improvements
## Problem Statement
### 1. Inconsistent Schema Naming
**Current State:**
```
terminology-schema.json ← Has ".json" suffix
api-documentation ← No suffix
enhanced-manpage ← No suffix
markdown-manpage ← No suffix, duplicate title
markdown-manpage-schema.json ← Has ".json" suffix, duplicate title
```
**Issues:**
- No naming convention enforced
- Duplicate schemas (3 manpage schemas!)
- Mix of suffixed (.json) and non-suffixed names
- No way to distinguish versions
### 2. Missing Versioning
**Current State:**
- No version in filenames
- No version in schema metadata (beyond optional `$id`)
- No way to track schema evolution
- Breaking changes not apparent
**Issues:**
- Can't have multiple versions simultaneously
- No migration path when schemas change
- Unclear which schema version a document uses
### 3. Format Mismatch: JSON vs Markdown
**The Philosophical Problem:**
> MarkiTect is a markdown-centric tool, yet schemas are JSON files.
> This creates a conceptual and practical mismatch.
**Current State:**
- Documents: Markdown (.md)
- Schemas: JSON (.json)
- No unified format for documentation + schema
- Schemas lack rich documentation capabilities
## Proposed Solutions
### Part 1: Naming Convention & Versioning
#### Option A: Filename-Based Versioning (Recommended)
**Format:** `{domain}-{type}-schema-v{major}.{minor}.json`
**Examples:**
```
manpage-schema-v1.0.json # Manpage schema v1.0
manpage-schema-v2.0.json # Breaking change → v2.0
terminology-schema-v1.0.json # Terminology schema
api-documentation-schema-v1.0.json
arc42-schema-v1.0.json
```
**Benefits:**
- Clear versioning in filename
- Easy to see multiple versions
- SemVer compatible (major.minor)
- Searchable/sortable
**Migration Strategy:**
```bash
# Rename existing schemas
markdown-manpage → manpage-schema-v1.0.json
enhanced-manpage → manpage-schema-v2.0.json # (breaking changes)
terminology-schema.json → terminology-schema-v1.0.json
```
#### Option B: $id-Based Versioning
**Keep simple filenames, use `$id` for versioning:**
```json
{
"$id": "https://markitect.dev/schemas/manpage/v1",
"$schema": "http://json-schema.org/draft-07/schema#",
"version": "1.0.0",
...
}
```
**Filenames:** `manpage-schema.json`, `terminology-schema.json`
**Benefits:**
- Clean filenames
- Versioning in metadata
- Follows JSON Schema best practices
**Drawbacks:**
- Can't have multiple versions in same database
- Harder to see versions at a glance
#### Recommendation: **Hybrid Approach**
Combine both for maximum clarity:
```json
// File: manpage-schema-v1.json
{
"$id": "https://markitect.dev/schemas/manpage/v1",
"$schema": "http://json-schema.org/draft-07/schema#",
"version": "1.0.0",
"title": "Unix Manual Page Schema",
...
}
```
### Part 2: Schema Metadata Standard
Add required metadata to all schemas:
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://markitect.dev/schemas/{domain}/v{major}",
// Required metadata
"version": "1.0.0", // SemVer
"title": "Human Readable Title",
"description": "Detailed description",
// Optional metadata
"x-markitect-schema-type": "document-schema",
"x-markitect-version": {
"major": 1,
"minor": 0,
"patch": 0
},
"x-markitect-author": "MarkiTect Project",
"x-markitect-created": "2026-01-04",
"x-markitect-updated": "2026-01-04",
"x-markitect-deprecated": false,
"x-markitect-superseded-by": null,
"x-markitect-document-types": ["manpage", "manual"],
"x-markitect-example": "examples/manpages/example.md",
// Schema content
"type": "object",
"properties": { ... }
}
```
### Part 3: Format Mismatch Solutions
#### Option 1: Markdown-First with Embedded JSON (Recommended)
**File Format:** Markdown with frontmatter and JSON code block
```markdown
---
schema-version: "1.0.0"
schema-id: "https://markitect.dev/schemas/manpage/v1"
document-type: manpage
status: stable
---
# Manpage Schema v1.0
## Overview
This schema validates Unix/Linux manual page documentation following
standard conventions (SYNOPSIS, DESCRIPTION, OPTIONS, etc.).
## Document Types
- Manual pages (man pages)
- CLI command documentation
- API reference pages
## Usage
\`\`\`bash
markitect validate mycommand.1.md --schema manpage-schema-v1
\`\`\`
## Examples
See [examples/manpages/](../../examples/manpages/) for complete examples.
## Schema Definition
\`\`\`json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://markitect.dev/schemas/manpage/v1",
"version": "1.0.0",
"title": "Unix Manual Page Schema",
"type": "object",
"properties": {
"headings": {
"type": "object",
"properties": {
"level_1": { ... }
}
}
},
"x-markitect-sections": { ... }
}
\`\`\`
## Validation Rules
### Required Sections
- **NAME** - Command name and brief description
- **SYNOPSIS** - Command syntax
- **DESCRIPTION** - Detailed description
### Optional Sections
- **OPTIONS** - Command-line options
- **EXAMPLES** - Usage examples
- **SEE ALSO** - Related commands
## Version History
### v1.0.0 (2026-01-04)
- Initial release
- Basic manpage structure validation
```
**Implementation:**
```python
class MarkdownSchemaLoader:
"""Load schemas from markdown files with embedded JSON."""
def load_schema_from_markdown(self, md_path: Path) -> dict:
"""Extract JSON schema from markdown file."""
content = md_path.read_text()
# Parse frontmatter
frontmatter = self._extract_frontmatter(content)
# Extract JSON from code block
schema_json = self._extract_json_from_code_block(content)
# Merge metadata
schema = json.loads(schema_json)
schema['x-markitect-metadata'] = frontmatter
return schema
def save_schema_to_markdown(self, schema: dict, md_path: Path):
"""Save schema as markdown with embedded JSON."""
# Generate markdown documentation
doc = self._generate_schema_documentation(schema)
# Embed JSON schema
json_block = f"```json\n{json.dumps(schema, indent=2)}\n```"
# Combine
full_content = f"{doc}\n\n## Schema Definition\n\n{json_block}"
md_path.write_text(full_content)
```
**Benefits:**
- ✅ Markdown-first (aligns with MarkiTect philosophy)
- ✅ Rich documentation alongside schema
- ✅ Human-readable and editable
- ✅ Version history in same file
- ✅ Examples and usage inline
- ✅ Can extract JSON when needed
**Drawbacks:**
- ⚠️ Requires parsing logic
- ⚠️ Two sources of truth (markdown + embedded JSON)
- ⚠️ More complex than pure JSON
#### Option 2: Markdown Documentation Generator
**Keep JSON schemas, auto-generate markdown docs:**
```
schemas/
manpage-schema-v1.json # Source of truth
manpage-schema-v1.md # Auto-generated docs
```
**Command:**
```bash
markitect schema-document manpage-schema-v1.json
# Generates: manpage-schema-v1.md
```
**Benefits:**
- ✅ Simple implementation
- ✅ JSON remains source of truth
- ✅ Auto-generated docs always in sync
**Drawbacks:**
- ⚠️ Two files to manage
- ⚠️ Can't hand-edit documentation (gets overwritten)
#### Option 3: Markdown Schema Language (DSL)
**Define schemas in markdown-native syntax:**
```markdown
# Manpage Schema v1.0
## Document Structure
### Required Sections (Level 1 Heading)
**NAME**
- Classification: required
- Content: Command name in bold, followed by description
- Pattern: `**command** - description`
**SYNOPSIS**
- Classification: required
- Content: Command syntax with options
- Min paragraphs: 1
- Max paragraphs: 3
### Optional Sections
**OPTIONS**
- Classification: recommended
- Content: Definition list of command-line options
```
**Parser generates JSON schema from markdown:**
```bash
markitect schema-compile manpage-schema-v1.md --output manpage-schema-v1.json
```
**Benefits:**
- ✅ Pure markdown
- ✅ Human-friendly syntax
- ✅ No JSON editing needed
**Drawbacks:**
- ⚠️ Complex parser implementation
- ⚠️ Limited to MarkiTect-specific features
- ⚠️ Can't use standard JSON Schema tools
#### Option 4: Literate Schema Programming
**Inspired by literate programming, mix documentation and schema:**
```markdown
# Manpage Schema v1.0
Manual pages follow a standard structure. The NAME section is required:
<<define-name-section>>=
{
"NAME": {
"classification": "required",
"heading_level": 2,
"content_instruction": "Command name and brief description"
}
}
The SYNOPSIS section shows command syntax:
<<define-synopsis-section>>=
{
"SYNOPSIS": {
"classification": "required",
"heading_level": 2,
"min_code_blocks": 1
}
}
Complete schema:
<<manpage-schema.json>>=
{
"$schema": "http://json-schema.org/draft-07/schema#",
"x-markitect-sections": {
<<define-name-section>>,
<<define-synopsis-section>>
}
}
```
**Benefits:**
- ✅ Documentation and schema interleaved
- ✅ Literate programming benefits
- ✅ Reusable schema fragments
**Drawbacks:**
- ⚠️ Complex tangling/weaving
- ⚠️ Unfamiliar paradigm
- ⚠️ Overkill for simple schemas
## Recommendations
### Short-Term (Immediate)
1. **Naming Convention:**
- Format: `{domain}-schema-v{major}.{minor}.json`
- Example: `manpage-schema-v1.0.json`
2. **Schema Metadata:**
- Add required `version`, `title`, `description` fields
- Add `x-markitect-*` metadata extensions
- Document in schema-catalog.yaml
3. **Duplicate Cleanup:**
- Consolidate 3 manpage schemas into versioned series
- Keep enhanced-manpage as v2.0 (breaking changes)
- Archive old schemas
### Medium-Term (Next Phase)
4. **Markdown Schema Format (Option 1):**
- Implement markdown-first schema format
- Markdown file with embedded JSON in code block
- Parser extracts JSON for validation
- Rich documentation alongside schema
5. **Schema Documentation Generator:**
- Auto-generate markdown docs from JSON schemas
- Include examples, usage, version history
- Link to example documents
### Long-Term (Future)
6. **Schema DSL (Option 3):**
- Evaluate markdown schema language
- Prototype parser for common patterns
- Consider if DSL adds value over JSON
7. **Schema Registry API:**
- REST API for schema discovery
- Version negotiation
- Schema evolution tracking
## Implementation Plan
### Phase 1: Naming & Versioning (1-2 days)
**Tasks:**
1. Define naming convention spec
2. Create schema metadata template
3. Rename existing schemas
4. Update schema-catalog.yaml
5. Update documentation
**Deliverables:**
- Schema naming convention spec
- Migrated schemas with versions
- Updated catalog
### Phase 2: Markdown Schema Format (3-5 days)
**Tasks:**
1. Design markdown schema format
2. Implement parser (extract JSON from markdown)
3. Implement generator (create markdown from JSON)
4. Convert existing schemas to markdown format
5. Update CLI to support .md schemas
6. Write documentation and examples
**Deliverables:**
- Markdown schema parser/generator
- All schemas in markdown format
- Updated CLI commands
- Migration guide
### Phase 3: Schema Validation (2-3 days)
**Tasks:**
1. Create metaschema for validating schemas
2. Add schema validation command
3. Validate all existing schemas
4. Add CI check for schema validity
**Deliverables:**
- Schema-for-schemas (metaschema)
- Validation command
- CI integration
## Cost-Benefit Analysis
### Option 1: Markdown-First (Recommended)
**Cost:**
- Parser implementation: ~200 lines
- CLI updates: ~100 lines
- Migration effort: 2-3 days
- Testing: 1 day
**Benefit:**
- Aligned with markdown philosophy ⭐⭐⭐⭐⭐
- Rich documentation ⭐⭐⭐⭐⭐
- Version history inline ⭐⭐⭐⭐
- Human-friendly ⭐⭐⭐⭐⭐
- Lower barrier to entry ⭐⭐⭐⭐
**Total:** High value for reasonable cost
### Option 2: Documentation Generator
**Cost:**
- Generator implementation: ~150 lines
- Template design: 1 day
- Testing: 0.5 days
**Benefit:**
- Simple implementation ⭐⭐⭐⭐
- Auto-sync docs ⭐⭐⭐⭐
- JSON remains source ⭐⭐⭐
**Total:** Good value, lower cost
### Option 3: Schema DSL
**Cost:**
- DSL design: 2-3 days
- Parser implementation: ~500 lines
- Compiler: ~300 lines
- Testing: 2 days
- Documentation: 1 day
**Benefit:**
- Pure markdown ⭐⭐⭐⭐⭐
- No JSON editing ⭐⭐⭐⭐
- Limited ecosystem ⭐⭐
**Total:** High cost, uncertain value
## Decision Matrix
| Criterion | Option 1: Markdown-First | Option 2: Doc Generator | Option 3: DSL |
|-----------|-------------------------|------------------------|---------------|
| Markdown alignment | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Implementation cost | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐ |
| Documentation quality | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| Tool ecosystem | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐ |
| Maintainability | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ |
| User-friendliness | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐ |
## Recommendation Summary
1. **Immediate:** Implement naming convention and versioning
2. **Short-term:** Choose **Option 1 (Markdown-First)** for schema format
3. **Fallback:** If Option 1 proves too complex, use **Option 2 (Doc Generator)**
4. **Future:** Evaluate DSL if community demand emerges
## Next Steps
1. Review and approve this proposal
2. Create naming convention specification
3. Prototype markdown schema parser
4. Migrate one schema as proof-of-concept
5. Gather feedback and iterate
6. Full migration of all schemas
---
## Appendix: Example Markdown Schema
See `examples/schemas/manpage-schema-v1.md` for a complete example of the proposed format.

154
roadmap/schema-of-schemas/SCHEMA_MANAGEMENT_SUMMARY.md Normal file
View File

@@ -0,0 +1,154 @@
# Schema Management: Executive Summary
**TL;DR:** Implement naming conventions, versioning, and markdown-first schema format to solve current schema management issues.
## Problems Identified
1. **Inconsistent Naming** - Mix of `schema.json` suffix and no suffix
2. **No Versioning** - Can't track schema evolution or maintain multiple versions
3. **Duplicate Schemas** - 3 manpage schemas with similar content
4. **Format Mismatch** - JSON schemas in markdown-centric tool
## Recommended Solution
### 1. Naming Convention (Immediate)
**Format:** `{domain}-schema-v{major}.{minor}.json` or `.md`
**Examples:**
```
manpage-schema-v1.0.json
terminology-schema-v1.0.json
api-documentation-schema-v1.0.json
```
**Migration:**
```
markdown-manpage → manpage-schema-v1.0.json
enhanced-manpage → manpage-schema-v2.0.json (breaking changes)
terminology-schema.json → terminology-schema-v1.0.json
```
### 2. Markdown-First Format (Short-term)
**Proposal:** Store schemas as markdown files with embedded JSON
**Benefits:**
- Aligns with markdown philosophy ✅
- Rich documentation alongside schema ✅
- Version history in same file ✅
- Examples and usage inline ✅
- Lower barrier to entry ✅
**Example:** See `examples/schemas/manpage-schema-v1.md`
**Format:**
```markdown
# Schema Title v1.0
## Documentation sections...
## Schema Definition
\`\`\`json
{ schema here }
\`\`\`
```
### 3. Schema Metadata Standard (Immediate)
**Required fields:**
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://markitect.dev/schemas/{domain}/v{major}",
"version": "1.0.0",
"title": "Human Readable Title",
"description": "Detailed description",
"x-markitect-metadata": {
"domain": "manpage",
"document-types": ["manual-page"],
"created": "2026-01-04",
"example": "examples/manpages/example.md"
}
}
```
## Implementation Phases
### Phase 1: Foundation (1-2 days)
- [x] Analyze current state
- [ ] Define naming convention spec
- [ ] Create schema metadata template
- [ ] Rename existing schemas
- [ ] Update schema-catalog.yaml
### Phase 2: Markdown Format (3-5 days)
- [ ] Design markdown schema format
- [ ] Implement parser (extract JSON from markdown)
- [ ] Convert 1 schema as proof-of-concept
- [ ] Test and iterate
- [ ] Migrate all schemas
### Phase 3: Tooling (2-3 days)
- [ ] Update CLI to support .md schemas
- [ ] Add schema validation command
- [ ] Create migration guide
- [ ] Update documentation
## Cost-Benefit Analysis
**Cost:** 6-10 days total effort
**Benefits:**
- Professional schema management ⭐⭐⭐⭐⭐
- Better discoverability ⭐⭐⭐⭐
- Easier maintenance ⭐⭐⭐⭐⭐
- Markdown alignment ⭐⭐⭐⭐⭐
- Version tracking ⭐⭐⭐⭐⭐
**ROI:** High - Foundational improvement that benefits all future schema work
## Alternative Considered
**Alternative:** Keep JSON, generate markdown docs automatically
**Pros:**
- Simpler implementation (2-3 days)
- JSON remains source of truth
- Standard tooling works
**Cons:**
- Doesn't solve format mismatch
- Documentation generated, not authored
- Two files to manage
**Verdict:** Markdown-first better aligns with project philosophy
## Quick Wins (Today)
1. **Rename schemas** with versioned names (30 minutes)
2. **Add metadata** to existing schemas (1 hour)
3. **Update catalog** with proper versioning (30 minutes)
## Questions to Resolve
1. **File extension:** `.md` or `.schema.md` for markdown schemas?
2. **JSON extraction:** Real-time or pre-compiled cache?
3. **Backward compatibility:** Support both formats during transition?
4. **CLI changes:** `--schema file.md` or auto-detect format?
## Next Steps
1. **Review** this proposal and example (examples/schemas/manpage-schema-v1.md)
2. **Decide** on markdown-first vs generated docs approach
3. **Prototype** parser for markdown schemas
4. **Migrate** one schema as proof-of-concept
5. **Iterate** based on feedback
6. **Full rollout** to all schemas
## References
- Full proposal: [SCHEMA_MANAGEMENT_PROPOSAL.md](./SCHEMA_MANAGEMENT_PROPOSAL.md)
- Example markdown schema: [examples/schemas/manpage-schema-v1.md](../../examples/schemas/manpage-schema-v1.md)
- Current schema catalog: [markitect/schemas/schema-catalog.yaml](../../markitect/schemas/schema-catalog.yaml)

962
roadmap/schema-of-schemas/WORKPLAN.md Normal file
View File

@@ -0,0 +1,962 @@
# Schema-of-Schemas Implementation Workplan
**Project:** Implement Markdown-First Schema System with Self-Description
**Created:** 2026-01-04
**Status:** Planning
**Duration:** 6-10 days
**Priority:** High - Foundation for all schema work
## Executive Summary
This workplan implements a comprehensive schema management system:
1. Filename conventions and versioning
2. Markdown-first schema format (`.md` with embedded JSON)
3. Schema-for-schemas (metaschema) for validation
4. Migration of existing schemas
5. Cleanup of legacy schemas from registry
## Project Goals
### Primary Goals
- [x] Establish filename convention: `{domain}-schema-v{version}.md`
- [ ] Implement markdown schema parser (extract JSON from markdown)
- [ ] Create schema-for-schemas to validate all schemas
- [ ] Migrate existing schemas to new format
- [ ] Remove legacy/duplicate schemas from registry
### Success Criteria
- ✅ All schemas follow naming convention
- ✅ Schemas stored as markdown files with embedded JSON
- ✅ Schema-for-schemas validates all schemas successfully
- ✅ No duplicate schemas in registry
- ✅ CLI commands work with `.md` schema files
- ✅ Documentation updated
## Architecture Overview
### Current State
```
Schemas: JSON files (.json)
Naming: Inconsistent (api-documentation, markdown-manpage-schema.json)
Versioning: None
Documentation: Separate or missing
Registry: Database with 5 schemas (3 duplicates)
```
### Target State
```
Schemas: Markdown files (.md) with embedded JSON
Naming: {domain}-schema-v{major}.{minor}.md
Versioning: SemVer in filename and metadata
Documentation: Inline with schema
Registry: Clean, versioned, no duplicates
Validation: Schema-for-schemas validates all schemas
```
### Components to Build
```
markitect/
├── schema_loader.py # NEW: Load schemas from markdown
├── schema_validator.py # UPDATED: Support .md schemas
├── cli.py # UPDATED: Accept .md schema files
└── schemas/
├── schema-schema-v1.md # NEW: Schema-for-schemas
└── ...versioned schemas...
examples/schemas/ # Markdown schema examples
└── manpage-schema-v1.md # Already created
roadmap/schema-of-schemas/ # Planning artifacts
├── WORKPLAN.md # This file
├── SCHEMA_NAMING_SPEC.md # Naming convention spec
└── IMPLEMENTATION_LOG.md # Progress tracking
```
## Phase Breakdown
### Phase 0: Planning & Setup ✅ (0.5 days)
**Goal:** Establish project structure and specifications
**Tasks:**
- [x] Create roadmap/schema-of-schemas directory
- [x] Move planning documents to roadmap
- [ ] Write naming convention specification
- [ ] Document schema metadata standard
- [ ] Create implementation checklist
**Deliverables:**
- [x] Directory structure
- [ ] SCHEMA_NAMING_SPEC.md
- [ ] SCHEMA_METADATA_SPEC.md
- [ ] This workplan
**Duration:** 0.5 days
**Status:** In Progress
---
### Phase 1: Filename Convention & Validation (1 day)
**Goal:** Establish and enforce filename conventions
**1.1 Define Naming Convention**
**Specification:**
```
Format: {domain}-schema-v{major}.{minor}.md
Components:
- domain: lowercase, hyphen-separated (e.g., "manpage", "api-documentation")
- schema: literal string "schema"
- version: SemVer major.minor (e.g., "v1.0", "v2.1")
- extension: ".md" (markdown)
Examples:
✓ manpage-schema-v1.0.md
✓ terminology-schema-v1.0.md
✓ api-documentation-schema-v1.0.md
✗ manpage.json (missing version)
✗ manpage-v1.md (missing "schema")
✗ ManPage-Schema-v1.0.md (wrong case)
```
**1.2 Implement Validation Function**
**File:** `markitect/schema_naming.py` (NEW)
```python
import re
from pathlib import Path
from typing import Tuple, Optional
SCHEMA_FILENAME_PATTERN = re.compile(
r'^(?P<domain>[a-z][a-z0-9-]*)-schema-v(?P<major>\d+)\.(?P<minor>\d+)\.md$'
)
def validate_schema_filename(filename: str) -> Tuple[bool, Optional[dict]]:
"""
Validate schema filename against convention.
Returns:
(is_valid, metadata_dict)
"""
match = SCHEMA_FILENAME_PATTERN.match(filename)
if not match:
return False, None
return True, {
'domain': match.group('domain'),
'version': f"{match.group('major')}.{match.group('minor')}",
'major': int(match.group('major')),
'minor': int(match.group('minor'))
}
def suggest_schema_filename(domain: str, version: str) -> str:
"""Generate correct schema filename from domain and version."""
# Normalize domain: lowercase, replace spaces with hyphens
domain_clean = domain.lower().replace(' ', '-').replace('_', '-')
return f"{domain_clean}-schema-v{version}.md"
```
**1.3 Add CLI Validation**
**Update:** `markitect/cli.py` - schema-ingest command
```python
@cli.command('schema-ingest')
@click.argument('schema_file', type=click.Path(exists=True, path_type=Path))
@click.option('--force', is_flag=True, help='Skip filename validation')
def schema_ingest(config, schema_file, force):
"""Ingest schema file with filename validation."""
from .schema_naming import validate_schema_filename, suggest_schema_filename
filename = schema_file.name
is_valid, metadata = validate_schema_filename(filename)
if not is_valid and not force:
click.echo(f"❌ Invalid schema filename: {filename}", err=True)
click.echo("\nExpected format: {domain}-schema-v{major}.{minor}.md")
click.echo("Example: manpage-schema-v1.0.md")
# Try to suggest correct name
# ... extract domain/version from file content ...
suggestion = suggest_schema_filename(domain, version)
click.echo(f"\nSuggested filename: {suggestion}")
click.echo("\nUse --force to skip validation")
sys.exit(1)
# Continue with ingestion...
```
**Tasks:**
- [ ] Write `markitect/schema_naming.py`
- [ ] Add unit tests for filename validation
- [ ] Update `schema-ingest` command with validation
- [ ] Test with valid and invalid filenames
**Deliverables:**
- [ ] schema_naming.py with validation logic
- [ ] Unit tests (tests/test_schema_naming.py)
- [ ] Updated CLI with validation
- [ ] SCHEMA_NAMING_SPEC.md documentation
**Duration:** 1 day
---
### Phase 2: Markdown Schema Loader (2-3 days)
**Goal:** Parse markdown files to extract JSON schemas
**2.1 Design Markdown Schema Format**
**Format Specification:**
```markdown
---
schema-id: "https://markitect.dev/schemas/{domain}/v{major}"
version: "{major}.{minor}.{patch}"
status: "stable|draft|deprecated"
---
# {Title} v{version}
## Overview
[Human-readable description]
## Usage
[Examples of how to use this schema]
## Schema Definition
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://markitect.dev/schemas/{domain}/v{major}",
"version": "{major}.{minor}.{patch}",
...
}
\```
## Validation Rules
[Explanation of schema rules]
## Version History
[Changelog]
```
**2.2 Implement Markdown Schema Loader**
**File:** `markitect/schema_loader.py` (NEW)
```python
"""
Schema Loader - Extract JSON schemas from markdown files.
Supports:
- YAML frontmatter for metadata
- JSON code block for schema definition
- Validation of schema structure
"""
import re
import json
import yaml
from pathlib import Path
from typing import Dict, Any, Optional, Tuple
class MarkdownSchemaLoader:
"""Load and parse markdown schema files."""
def __init__(self):
self.frontmatter_pattern = re.compile(
r'^---\s*\n(.*?)\n---\s*\n',
re.DOTALL | re.MULTILINE
)
self.json_code_block_pattern = re.compile(
r'```json\s*\n(.*?)\n```',
re.DOTALL | re.MULTILINE
)
def load_schema(self, md_path: Path) -> Dict[str, Any]:
"""
Load schema from markdown file.
Returns:
{
'schema': {...}, # Extracted JSON schema
'metadata': {...}, # Frontmatter metadata
'documentation': '...' # Full markdown content
}
"""
if not md_path.exists():
raise FileNotFoundError(f"Schema file not found: {md_path}")
content = md_path.read_text(encoding='utf-8')
# Extract frontmatter
metadata = self._extract_frontmatter(content)
# Extract JSON schema
schema = self._extract_json_schema(content)
if not schema:
raise ValueError(f"No JSON schema found in {md_path}")
# Merge metadata into schema
schema = self._merge_metadata(schema, metadata, md_path)
return {
'schema': schema,
'metadata': metadata,
'documentation': content,
'source_file': str(md_path)
}
def _extract_frontmatter(self, content: str) -> Dict[str, Any]:
"""Extract YAML frontmatter from markdown."""
match = self.frontmatter_pattern.search(content)
if not match:
return {}
try:
return yaml.safe_load(match.group(1)) or {}
except yaml.YAMLError as e:
raise ValueError(f"Invalid YAML frontmatter: {e}")
def _extract_json_schema(self, content: str) -> Optional[Dict[str, Any]]:
"""Extract JSON schema from code block."""
matches = self.json_code_block_pattern.findall(content)
if not matches:
return None
# Use the first JSON code block as schema
# (or could look for specific heading like "## Schema Definition")
try:
return json.loads(matches[0])
except json.JSONDecodeError as e:
raise ValueError(f"Invalid JSON schema: {e}")
def _merge_metadata(
self,
schema: Dict[str, Any],
metadata: Dict[str, Any],
source_file: Path
) -> Dict[str, Any]:
"""Merge frontmatter metadata into schema."""
# Add MarkiTect-specific metadata
schema['x-markitect-source'] = {
'file': str(source_file),
'format': 'markdown',
'frontmatter': metadata
}
# Override schema fields with frontmatter if present
if 'version' in metadata:
schema['version'] = metadata['version']
if 'schema-id' in metadata:
schema['$id'] = metadata['schema-id']
return schema
def save_schema(
self,
schema: Dict[str, Any],
md_path: Path,
template: Optional[str] = None
):
"""
Save schema as markdown file.
Args:
schema: JSON schema dict
md_path: Output path
template: Optional markdown template
"""
if template:
# Use provided template
content = self._render_template(template, schema)
else:
# Generate basic markdown
content = self._generate_markdown(schema)
md_path.write_text(content, encoding='utf-8')
def _generate_markdown(self, schema: Dict[str, Any]) -> str:
"""Generate markdown from schema."""
title = schema.get('title', 'Untitled Schema')
version = schema.get('version', '1.0.0')
description = schema.get('description', '')
# Generate frontmatter
frontmatter = yaml.dump({
'schema-id': schema.get('$id', ''),
'version': version,
'status': 'draft'
}, default_flow_style=False)
# Generate markdown
md = f"""---
{frontmatter}---
# {title} v{version}
## Overview
{description}
## Schema Definition
```json
{json.dumps(schema, indent=2)}
```
## Version History
### v{version}
- Initial version
"""
return md
class SchemaLoaderError(Exception):
"""Base exception for schema loading errors."""
pass
```
**2.3 Update Schema Validator**
**Update:** `markitect/schema_validator.py`
```python
from .schema_loader import MarkdownSchemaLoader
class SchemaValidator:
def __init__(self):
self.schema_generator = SchemaGenerator()
self.jsonschema_available = JSONSCHEMA_AVAILABLE
self.md_loader = MarkdownSchemaLoader() # NEW
def validate_file_against_schema_file(
self,
file_path: Path,
schema_file_path: Path
) -> bool:
"""Validate file against schema (supports .json and .md)."""
# Detect schema file format
if schema_file_path.suffix == '.md':
# Load from markdown
schema_data = self.md_loader.load_schema(schema_file_path)
schema = schema_data['schema']
else:
# Load from JSON (legacy)
schema_content = schema_file_path.read_text(encoding='utf-8')
schema = json.loads(schema_content)
return self.validate_file_against_schema(file_path, schema)
```
**Tasks:**
- [ ] Implement MarkdownSchemaLoader class
- [ ] Add frontmatter extraction (YAML)
- [ ] Add JSON code block extraction
- [ ] Add metadata merging logic
- [ ] Write comprehensive unit tests
- [ ] Update SchemaValidator to use loader
- [ ] Test with example markdown schemas
**Deliverables:**
- [ ] schema_loader.py implementation
- [ ] Unit tests (tests/test_schema_loader.py)
- [ ] Updated schema_validator.py
- [ ] Integration tests
**Duration:** 2-3 days
---
### Phase 3: Schema-for-Schemas (2 days)
**Goal:** Create metaschema to validate all schema files
**3.1 Design Schema-for-Schemas**
**File:** `markitect/schemas/schema-schema-v1.md`
**Purpose:** Validates that schema files follow MarkiTect conventions
**Validates:**
- Required fields ($schema, $id, version, title, description)
- Version format (SemVer)
- $id URL format
- x-markitect-* extensions
- Section classifications
- Content control structures
**3.2 Implement Schema-for-Schemas**
See separate file: `roadmap/schema-of-schemas/schema-schema-v1.md` (to be created)
**3.3 Add Schema Validation Command**
**New CLI command:** `markitect schema-validate`
```python
@cli.command('schema-validate')
@click.argument('schema_file', type=click.Path(exists=True, path_type=Path))
@click.option('--detailed-errors', is_flag=True)
def schema_validate(config, schema_file, detailed_errors):
"""
Validate a schema file against the schema-for-schemas.
Ensures schema files follow MarkiTect conventions and standards.
"""
from .schema_loader import MarkdownSchemaLoader
from .schema_validator import SchemaValidator
loader = MarkdownSchemaLoader()
validator = SchemaValidator()
# Load the schema
try:
schema_data = loader.load_schema(schema_file)
schema = schema_data['schema']
except Exception as e:
click.echo(f"❌ Failed to load schema: {e}", err=True)
sys.exit(1)
# Load schema-for-schemas
metaschema_path = Path(__file__).parent / 'schemas' / 'schema-schema-v1.md'
metaschema_data = loader.load_schema(metaschema_path)
metaschema = metaschema_data['schema']
# Validate
is_valid = validator.validate_schema_against_metaschema(schema, metaschema)
if is_valid:
click.echo(f"✅ Schema is valid: {schema_file.name}")
click.echo(f" Title: {schema.get('title')}")
click.echo(f" Version: {schema.get('version')}")
else:
click.echo(f"❌ Schema validation failed: {schema_file.name}", err=True)
if detailed_errors:
# Show detailed errors
pass
sys.exit(1)
```
**Tasks:**
- [ ] Design schema-for-schemas structure
- [ ] Implement schema-schema-v1.md
- [ ] Add schema validation logic
- [ ] Create `schema-validate` CLI command
- [ ] Test all existing schemas against metaschema
- [ ] Document validation rules
**Deliverables:**
- [ ] schema-schema-v1.md (metaschema)
- [ ] schema-validate command
- [ ] Validation documentation
- [ ] Test suite
**Duration:** 2 days
---
### Phase 4: Schema Migration (1-2 days)
**Goal:** Convert existing schemas to new format
**4.1 Inventory Current Schemas**
Current schemas in database:
```
1. terminology-schema.json → terminology-schema-v1.0.md
2. api-documentation → api-documentation-schema-v1.0.md
3. enhanced-manpage → manpage-schema-v2.0.md
4. markdown-manpage → manpage-schema-v1.0.md (DUPLICATE)
5. markdown-manpage-schema.json → manpage-schema-v1.0.md (DUPLICATE)
```
**Decision matrix:**
- Keep enhanced-manpage as v2.0 (has classifications)
- Merge markdown-manpage variants into v1.0
- Update terminology to v1.0
- Update api-documentation to v1.0
**4.2 Create Migration Script**
**File:** `scripts/migrate_schemas.py`
```python
#!/usr/bin/env python3
"""Migrate schemas to markdown format with versioning."""
from pathlib import Path
from markitect.schema_loader import MarkdownSchemaLoader
from markitect.database import DatabaseManager
def migrate_schema(
db_manager: DatabaseManager,
old_name: str,
new_name: str,
version: str,
domain: str
):
"""Migrate single schema to new format."""
# Get old schema from database
old_schema = db_manager.get_schema_file(old_name)
if not old_schema:
print(f"❌ Schema not found: {old_name}")
return
schema_json = json.loads(old_schema['schema_content'])
# Update metadata
schema_json['version'] = version
schema_json['$id'] = f"https://markitect.dev/schemas/{domain}/v{version.split('.')[0]}"
# Save as markdown
loader = MarkdownSchemaLoader()
md_path = Path(f"markitect/schemas/{new_name}")
loader.save_schema(schema_json, md_path)
print(f"✓ Migrated: {old_name} → {new_name}")
# Ingest new schema
# ... ingest markdown schema to database ...
return md_path
def main():
migrations = [
('terminology-schema.json', 'terminology-schema-v1.0.md', '1.0.0', 'terminology'),
('api-documentation', 'api-documentation-schema-v1.0.md', '1.0.0', 'api-documentation'),
('enhanced-manpage', 'manpage-schema-v2.0.md', '2.0.0', 'manpage'),
('markdown-manpage', 'manpage-schema-v1.0.md', '1.0.0', 'manpage'),
]
db = DatabaseManager('markitect.db')
for old, new, version, domain in migrations:
migrate_schema(db, old, new, version, domain)
```
**4.3 Execute Migration**
```bash
# Run migration script
python scripts/migrate_schemas.py
# Validate all new schemas
for schema in markitect/schemas/*-schema-v*.md; do
markitect schema-validate "$schema"
done
# Ingest new schemas
for schema in markitect/schemas/*-schema-v*.md; do
markitect schema-ingest "$schema"
done
```
**4.4 Clean Up Registry**
```bash
# Remove old schemas from database
markitect schema-delete markdown-manpage
markitect schema-delete markdown-manpage-schema.json
markitect schema-delete api-documentation
markitect schema-delete enhanced-manpage
markitect schema-delete terminology-schema.json
# Verify cleanup
markitect schema-list
# Should show only versioned .md schemas
```
**Tasks:**
- [ ] Create schema inventory
- [ ] Write migration script
- [ ] Test migration on one schema
- [ ] Execute full migration
- [ ] Validate all migrated schemas
- [ ] Remove old schemas from database
- [ ] Update examples to use new schema names
**Deliverables:**
- [ ] scripts/migrate_schemas.py
- [ ] All schemas migrated to markdown
- [ ] Clean registry (no duplicates)
- [ ] Migration report
**Duration:** 1-2 days
---
### Phase 5: CLI & Documentation Updates (1 day)
**Goal:** Update CLI and documentation for new system
**5.1 Update CLI Commands**
Commands to update:
- `schema-ingest` - Accept .md files, validate filename
- `schema-list` - Show version in output
- `schema-get` - Export as .md or .json
- `validate` - Accept .md schema files
- `generate-stub` - Work with .md schemas
- `schema-generate` - Output .md format option
- NEW: `schema-validate` - Validate against metaschema
**5.2 Update Documentation**
Files to update:
- README.md - Mention markdown schemas
- examples/terminology/README.md - Use new schema name
- docs/specifications/schema-extensions-spec.md - Document markdown format
- Create: docs/guides/schema-authoring-guide.md
**5.3 Add Schema Templates**
**File:** `templates/schema-template-v1.md`
```markdown
---
schema-id: "https://markitect.dev/schemas/DOMAIN/v1"
version: "1.0.0"
status: "draft"
---
# TITLE Schema v1.0
## Overview
[Description of what this schema validates]
## Document Types
- [Document type 1]
- [Document type 2]
## Usage
\`\`\`bash
markitect validate document.md --schema DOMAIN-schema-v1.0.md
\`\`\`
## Examples
See [examples/DOMAIN/example.md](../../examples/DOMAIN/example.md)
## Schema Definition
\`\`\`json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://markitect.dev/schemas/DOMAIN/v1",
"version": "1.0.0",
"title": "TITLE Schema",
"description": "Schema for validating DESCRIPTION",
"type": "object",
"properties": {
"headings": {
"type": "object"
}
},
"x-markitect-sections": {},
"x-markitect-content-control": {}
}
\`\`\`
## Validation Rules
### Required Sections
- **SECTION** - Description
### Optional Sections
- **SECTION** - Description
## Version History
### v1.0.0 (YYYY-MM-DD)
- Initial release
```
**Tasks:**
- [ ] Update all CLI commands for .md support
- [ ] Update documentation
- [ ] Create schema authoring guide
- [ ] Add schema template
- [ ] Update examples
- [ ] Test all workflows end-to-end
**Deliverables:**
- [ ] Updated CLI commands
- [ ] Schema authoring guide
- [ ] Schema template
- [ ] Updated examples
- [ ] End-to-end tests
**Duration:** 1 day
---
### Phase 6: Testing & Validation (1 day)
**Goal:** Comprehensive testing of new system
**6.1 Unit Tests**
Test coverage for:
- `schema_naming.py` - Filename validation
- `schema_loader.py` - Markdown parsing
- `schema_validator.py` - Validation with .md schemas
**6.2 Integration Tests**
End-to-end workflows:
1. Create new schema in markdown format
2. Validate schema against schema-for-schemas
3. Ingest schema to database
4. Use schema to validate documents
5. Generate stub from schema
6. Export schema
**6.3 Regression Tests**
Ensure existing functionality still works:
- JSON schemas still load (backward compatibility)
- All existing documents validate
- Schema generation still works
- Stub generation still works
**Tasks:**
- [ ] Write unit tests for new modules
- [ ] Create integration test suite
- [ ] Run regression tests
- [ ] Fix any issues found
- [ ] Achieve >80% code coverage
- [ ] Document test procedures
**Deliverables:**
- [ ] Unit tests (>80% coverage)
- [ ] Integration tests
- [ ] Regression test suite
- [ ] Test documentation
**Duration:** 1 day
---
## Timeline
```
Week 1:
Day 1: Phase 0 (Planning) + Phase 1 (Naming Convention)
Day 2-3: Phase 2 (Markdown Loader)
Day 4-5: Phase 3 (Schema-for-Schemas)
Week 2:
Day 6-7: Phase 4 (Migration)
Day 8: Phase 5 (CLI & Docs)
Day 9: Phase 6 (Testing)
Day 10: Buffer for issues/refinement
```
**Total:** 8-10 days
## Risks & Mitigation
### Risk 1: Parsing Complexity
**Risk:** Markdown parsing more complex than expected
**Probability:** Medium
**Impact:** High
**Mitigation:**
- Start with simple regex-based parser
- Test extensively with edge cases
- Have fallback to simpler format
### Risk 2: Backward Compatibility
**Risk:** Breaking existing workflows
**Probability:** Low
**Impact:** High
**Mitigation:**
- Support both .json and .md during transition
- Provide migration script
- Test thoroughly with existing documents
### Risk 3: Schema-for-Schemas Complexity
**Risk:** Self-referential validation complex
**Probability:** Medium
**Impact:** Medium
**Mitigation:**
- Start with simple metaschema
- Iterate based on actual schemas
- Don't over-engineer initially
## Success Metrics
- [ ] All schemas follow naming convention (5/5)
- [ ] All schemas in markdown format (5/5)
- [ ] All schemas validate against metaschema (5/5)
- [ ] Zero duplicate schemas in registry
- [ ] CLI commands work with .md schemas
- [ ] Documentation comprehensive
- [ ] Test coverage >80%
- [ ] No regression in existing functionality
## Deliverables Checklist
### Code
- [ ] markitect/schema_naming.py
- [ ] markitect/schema_loader.py
- [ ] markitect/schemas/schema-schema-v1.md
- [ ] scripts/migrate_schemas.py
- [ ] Updated CLI commands
- [ ] Unit tests
- [ ] Integration tests
### Documentation
- [ ] SCHEMA_NAMING_SPEC.md
- [ ] SCHEMA_METADATA_SPEC.md
- [ ] Schema authoring guide
- [ ] Migration guide
- [ ] Updated examples
- [ ] IMPLEMENTATION_LOG.md
### Schemas
- [ ] terminology-schema-v1.0.md
- [ ] api-documentation-schema-v1.0.md
- [ ] manpage-schema-v1.0.md
- [ ] manpage-schema-v2.0.md
- [ ] schema-schema-v1.0.md
### Registry
- [ ] Clean schema database
- [ ] Updated schema-catalog.yaml
- [ ] No duplicates
## Next Steps
1. **Review this workplan** - Get approval
2. **Phase 0** - Complete planning artifacts
3. **Phase 1** - Implement naming validation
4. **Checkpoint** - Review progress after Phase 1
5. **Continue** - Execute remaining phases
## Approval
- [ ] Workplan reviewed
- [ ] Approach approved
- [ ] Ready to begin implementation
---
**Status:** Awaiting approval
**Next Action:** Complete Phase 0 planning artifacts