coulomb/markitect-main
2
0
Fork 0
Code Issues 61 Pull Requests Actions Projects 12 Releases 1 Wiki Activity

feat: implement Phase 2 - Markdown Schema Loader

Browse Source 
Completed Phase 2 of the schema-of-schemas implementation with full
markdown schema support. This enables schemas to be authored as
markdown files with rich documentation and embedded JSON schemas.

Core Implementation (markitect/schema_loader.py):
- MarkdownSchemaLoader class with comprehensive parsing capabilities
- YAML frontmatter extraction with error handling
- JSON code block extraction with section preference (## Schema Definition)
- Metadata merging with x-markitect-source tracking
- Schema saving with template support and round-trip capability
- Helper methods: list_json_blocks(), validate_schema_structure()

Test Coverage (tests/test_schema_loader.py):
- 35 comprehensive unit tests (100% passing)
- Tests for loading, parsing, saving, round-trip conversion
- Edge case handling (empty files, binary files, malformed blocks)
- Fixed binary file test to use invalid UTF-8 sequences

Example Schema (markitect/schemas/manpage-schema-v1.0.md):
- First markdown schema following naming convention
- Complete manpage schema with frontmatter + documentation + JSON
- Demonstrates section classification and content control
- Shows proper structure for future schema authors

Documentation (roadmap/schema-of-schemas/SCHEMA_LOADER_GUIDE.md):
- Comprehensive user guide (600+ lines)
- API reference with examples
- Best practices and troubleshooting
- Integration patterns for CLI and validator

Progress Tracking:
- Updated TODO.md with Phase 2 completion
- Updated CHANGELOG.md with implementation details
- Next: Phase 3 - Schema-for-Schemas Metaschema

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-01-05 00:02:15 +01:00
parent 14108533fb
commit b81ce5631d
6 changed files with 2151 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

503
markitect/schema_loader.py Normal file
View File

@@ -0,0 +1,503 @@
"""
Schema Loader - Extract JSON schemas from markdown files.
This module provides functionality to load schemas from markdown files that
contain embedded JSON schemas in code blocks, along with YAML frontmatter
metadata and rich documentation.
Markdown Schema Format:
---
schema-id: "https://markitect.dev/schemas/domain/v1"
version: "1.0.0"
status: "stable|draft|deprecated"
---
# Schema Title v1.0
## Documentation sections...
## Schema Definition
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
...
}
```
This enables:
- Rich documentation alongside schemas
- Version history in same file
- Human-readable schema files
- Markdown-first approach aligned with MarkiTect philosophy
"""
import re
import json
import yaml
from pathlib import Path
from typing import Dict, Any, Optional, List, Tuple
class SchemaLoaderError(Exception):
"""Base exception for schema loading errors."""
pass
class InvalidSchemaFormatError(SchemaLoaderError):
"""Schema file format is invalid."""
pass
class SchemaNotFoundError(SchemaLoaderError):
"""No JSON schema found in markdown file."""
pass
class MarkdownSchemaLoader:
"""
Load and parse markdown schema files.
Supports:
- YAML frontmatter for metadata
- JSON code blocks for schema definition
- Validation of schema structure
- Metadata merging
Example:
>>> loader = MarkdownSchemaLoader()
>>> schema_data = loader.load_schema(Path("manpage-schema-v1.0.md"))
>>> schema = schema_data['schema']
>>> metadata = schema_data['metadata']
"""
def __init__(self):
"""Initialize the schema loader with regex patterns."""
# Pattern to match YAML frontmatter
# Matches: --- ... --- at start of file
self.frontmatter_pattern = re.compile(
r'^---\s*\n(.*?)\n---\s*\n',
re.DOTALL | re.MULTILINE
)
# Pattern to match JSON code blocks
# Matches: ```json ... ```
self.json_code_block_pattern = re.compile(
r'```json\s*\n(.*?)\n```',
re.DOTALL | re.MULTILINE
)
# Pattern to find Schema Definition section
# This helps us find the right JSON block if there are multiple
self.schema_section_pattern = re.compile(
r'##\s+Schema Definition\s*\n',
re.MULTILINE
)
def load_schema(self, md_path: Path) -> Dict[str, Any]:
"""
Load schema from markdown file.
Args:
md_path: Path to markdown schema file
Returns:
Dictionary containing:
- schema: Extracted JSON schema (dict)
- metadata: Frontmatter metadata (dict)
- documentation: Full markdown content (str)
- source_file: Source file path (str)
Raises:
FileNotFoundError: If schema file doesn't exist
InvalidSchemaFormatError: If file format is invalid
SchemaNotFoundError: If no JSON schema found
Example:
>>> loader = MarkdownSchemaLoader()
>>> data = loader.load_schema(Path("manpage-schema-v1.0.md"))
>>> print(data['schema']['title'])
'Unix Manual Page Schema'
"""
if not md_path.exists():
raise FileNotFoundError(f"Schema file not found: {md_path}")
# Read file content
try:
content = md_path.read_text(encoding='utf-8')
except Exception as e:
raise InvalidSchemaFormatError(f"Failed to read schema file: {e}")
# Extract frontmatter
metadata = self._extract_frontmatter(content)
# Extract JSON schema
schema = self._extract_json_schema(content)
if not schema:
raise SchemaNotFoundError(
f"No JSON schema found in {md_path}. "
f"Expected a ```json code block with schema definition."
)
# Merge metadata into schema
schema = self._merge_metadata(schema, metadata, md_path)
return {
'schema': schema,
'metadata': metadata,
'documentation': content,
'source_file': str(md_path)
}
def _extract_frontmatter(self, content: str) -> Dict[str, Any]:
"""
Extract YAML frontmatter from markdown content.
Args:
content: Markdown file content
Returns:
Dictionary of frontmatter metadata (empty if none found)
Raises:
InvalidSchemaFormatError: If YAML is malformed
"""
match = self.frontmatter_pattern.search(content)
if not match:
return {}
yaml_content = match.group(1)
try:
metadata = yaml.safe_load(yaml_content) or {}
if not isinstance(metadata, dict):
raise InvalidSchemaFormatError(
f"Frontmatter must be a YAML dictionary, got {type(metadata)}"
)
return metadata
except yaml.YAMLError as e:
raise InvalidSchemaFormatError(f"Invalid YAML frontmatter: {e}")
def _extract_json_schema(self, content: str) -> Optional[Dict[str, Any]]:
"""
Extract JSON schema from markdown code blocks.
Prefers JSON blocks under "## Schema Definition" section,
but will use first JSON block if no Schema Definition section found.
Args:
content: Markdown file content
Returns:
JSON schema dictionary or None if not found
Raises:
InvalidSchemaFormatError: If JSON is malformed
"""
# Find all JSON code blocks
json_blocks = self.json_code_block_pattern.findall(content)
if not json_blocks:
return None
# Try to find the Schema Definition section
schema_section_match = self.schema_section_pattern.search(content)
if schema_section_match:
# Find JSON block that comes after Schema Definition section
section_pos = schema_section_match.end()
# Re-search for JSON blocks starting from section position
remaining_content = content[section_pos:]
section_json_blocks = self.json_code_block_pattern.findall(remaining_content)
if section_json_blocks:
json_text = section_json_blocks[0]
else:
# Fallback to first JSON block in entire document
json_text = json_blocks[0]
else:
# No Schema Definition section, use first JSON block
json_text = json_blocks[0]
# Parse JSON
try:
schema = json.loads(json_text)
if not isinstance(schema, dict):
raise InvalidSchemaFormatError(
f"Schema must be a JSON object, got {type(schema)}"
)
return schema
except json.JSONDecodeError as e:
raise InvalidSchemaFormatError(f"Invalid JSON schema: {e}")
def _merge_metadata(
self,
schema: Dict[str, Any],
metadata: Dict[str, Any],
source_file: Path
) -> Dict[str, Any]:
"""
Merge frontmatter metadata into schema.
Adds x-markitect-source extension with file info and metadata.
Optionally overrides schema fields with frontmatter values.
Args:
schema: JSON schema dictionary
metadata: Frontmatter metadata dictionary
source_file: Path to source file
Returns:
Schema with merged metadata
"""
# Create a copy to avoid modifying original
merged_schema = schema.copy()
# Add MarkiTect-specific source metadata
merged_schema['x-markitect-source'] = {
'file': str(source_file),
'filename': source_file.name,
'format': 'markdown',
'frontmatter': metadata
}
# Override schema fields with frontmatter if present
# This allows frontmatter to be the source of truth for metadata
if 'version' in metadata:
merged_schema['version'] = metadata['version']
if 'schema-id' in metadata:
merged_schema['$id'] = metadata['schema-id']
if 'status' in metadata:
if 'x-markitect-metadata' not in merged_schema:
merged_schema['x-markitect-metadata'] = {}
merged_schema['x-markitect-metadata']['status'] = metadata['status']
return merged_schema
def save_schema(
self,
schema: Dict[str, Any],
md_path: Path,
template: Optional[str] = None,
frontmatter: Optional[Dict[str, Any]] = None
):
"""
Save schema as markdown file.
Args:
schema: JSON schema dictionary to save
md_path: Output path for markdown file
template: Optional markdown template string
frontmatter: Optional frontmatter metadata (extracted from schema if not provided)
Raises:
InvalidSchemaFormatError: If schema is invalid
Example:
>>> loader = MarkdownSchemaLoader()
>>> loader.save_schema(
... schema={'title': 'My Schema', ...},
... md_path=Path('my-schema-v1.0.md')
... )
"""
if template:
# Use provided template
content = self._render_template(template, schema, frontmatter)
else:
# Generate basic markdown
content = self._generate_markdown(schema, frontmatter)
# Create parent directory if needed
md_path.parent.mkdir(parents=True, exist_ok=True)
# Write file
try:
md_path.write_text(content, encoding='utf-8')
except Exception as e:
raise InvalidSchemaFormatError(f"Failed to write schema file: {e}")
def _generate_markdown(
self,
schema: Dict[str, Any],
frontmatter: Optional[Dict[str, Any]] = None
) -> str:
"""
Generate markdown from schema.
Args:
schema: JSON schema dictionary
frontmatter: Optional frontmatter metadata
Returns:
Markdown content as string
"""
# Extract metadata from schema
title = schema.get('title', 'Untitled Schema')
version = schema.get('version', '1.0.0')
description = schema.get('description', '')
schema_id = schema.get('$id', '')
# Build frontmatter
if frontmatter is None:
frontmatter = {}
# Set defaults
if 'schema-id' not in frontmatter and schema_id:
frontmatter['schema-id'] = schema_id
if 'version' not in frontmatter:
frontmatter['version'] = version
if 'status' not in frontmatter:
frontmatter['status'] = 'draft'
# Generate frontmatter YAML
frontmatter_yaml = yaml.dump(
frontmatter,
default_flow_style=False,
allow_unicode=True
).strip()
# Generate JSON (pretty-printed)
schema_json = json.dumps(schema, indent=2, ensure_ascii=False)
# Build markdown content
md_content = f"""---
{frontmatter_yaml}
---
# {title} v{version}
## Overview
{description}
## Usage
```bash
markitect validate document.md --schema {Path(frontmatter.get('schema-id', 'schema')).name}
```
## Schema Definition
```json
{schema_json}
```
## Version History
### v{version}
- Initial version
"""
return md_content
def _render_template(
self,
template: str,
schema: Dict[str, Any],
frontmatter: Optional[Dict[str, Any]] = None
) -> str:
"""
Render markdown from template.
Simple template rendering using string formatting.
For complex templates, consider using Jinja2 or similar.
Args:
template: Template string
schema: JSON schema dictionary
frontmatter: Optional frontmatter metadata
Returns:
Rendered markdown content
"""
# Build context for template
context = {
'title': schema.get('title', 'Untitled'),
'version': schema.get('version', '1.0.0'),
'description': schema.get('description', ''),
'schema_id': schema.get('$id', ''),
'schema_json': json.dumps(schema, indent=2, ensure_ascii=False),
'frontmatter': frontmatter or {},
}
# Simple template rendering
try:
return template.format(**context)
except KeyError as e:
raise InvalidSchemaFormatError(f"Template missing key: {e}")
def list_json_blocks(self, content: str) -> List[Tuple[int, str]]:
"""
List all JSON code blocks in markdown content.
Useful for debugging or when multiple JSON blocks exist.
Args:
content: Markdown file content
Returns:
List of (position, json_content) tuples
Example:
>>> loader = MarkdownSchemaLoader()
>>> content = Path('schema.md').read_text()
>>> blocks = loader.list_json_blocks(content)
>>> print(f"Found {len(blocks)} JSON blocks")
"""
blocks = []
for match in self.json_code_block_pattern.finditer(content):
blocks.append((match.start(), match.group(1)))
return blocks
def validate_schema_structure(self, schema: Dict[str, Any]) -> List[str]:
"""
Validate basic schema structure.
Checks for required JSON Schema fields and MarkiTect conventions.
Args:
schema: JSON schema dictionary
Returns:
List of warning/error messages (empty if valid)
Example:
>>> loader = MarkdownSchemaLoader()
>>> issues = loader.validate_schema_structure(schema)
>>> if issues:
... print("Schema issues:", issues)
"""
issues = []
# Check required JSON Schema fields
if '$schema' not in schema:
issues.append("Missing required field: $schema")
if 'type' not in schema:
issues.append("Missing recommended field: type")
if 'title' not in schema:
issues.append("Missing recommended field: title")
if 'description' not in schema:
issues.append("Missing recommended field: description")
# Check MarkiTect conventions
if 'version' not in schema:
issues.append("Missing MarkiTect convention: version field")
if '$id' not in schema:
issues.append("Missing recommended field: $id")
# Check $id format if present
if '$id' in schema:
schema_id = schema['$id']
if not isinstance(schema_id, str):
issues.append("$id must be a string")
elif not schema_id.startswith('https://'):
issues.append("$id should be a full HTTPS URL")
return issues

333
markitect/schemas/manpage-schema-v1.0.md Normal file
View File

@@ -0,0 +1,333 @@
---
schema-id: "https://markitect.dev/schemas/manpage/v1.0"
version: "1.0.0"
status: "stable"
domain: "manpage"
description: "JSON schema for Unix-style manual pages with section classification and content control"
---
# Unix Manual Page Schema v1.0
## Overview
This schema defines the structure and validation rules for Unix-style manual pages (manpages) in MarkiTect's markdown format. It includes comprehensive section classification, content control patterns, and quality guidelines to ensure consistent, high-quality documentation.
## Features
- **Section Classification System**: Categorizes manpage sections as required, recommended, optional, discouraged, or improper
- **Content Control**: Validates content patterns, quality metrics, and structural requirements
- **Flexible Section Names**: Supports alternative section names (e.g., "FLAGS" as alternative to "OPTIONS")
- **Quality Enforcement**: Minimum/maximum content requirements for paragraphs, code blocks, and words
## Section Classifications
### Required Sections
- **SYNOPSIS**: Brief command syntax with all options and arguments
- **DESCRIPTION**: Detailed explanation of command purpose and functionality
### Recommended Sections
- **EXAMPLES**: Practical usage examples demonstrating common use cases
- **OPTIONS**: Detailed option descriptions with all flags and behaviors
- **SEE ALSO**: Related commands and documentation references
### Optional Sections
- **BUGS**: Known issues and bug reporting information
- **AUTHORS**: Contributors and maintainers
- **COPYRIGHT**: License information
- **HISTORY**: Historical development information
### Discouraged Sections
- **DEPRECATED**: Legacy content (should move to HISTORY)
- **OLD_SYNTAX**: Outdated syntax (should move to HISTORY or be removed)
### Improper Sections
- **INTERNAL_NOTES**: Development notes (must not appear in published docs)
- **TODO**: Development tasks (remove before publication)
- **DRAFT**: Draft markers (remove before publication)
## Usage
### Validating a Manpage
```bash
markitect validate my-command.1.md --schema manpage-schema-v1.0
```
### Common Validation Errors
1. **Missing Required Sections**: Ensure SYNOPSIS and DESCRIPTION are present
2. **Content Too Brief**: DESCRIPTION should have at least 50 words
3. **No Examples**: While optional, EXAMPLES are highly recommended
4. **Improper Sections**: Remove TODO, DRAFT, and INTERNAL_NOTES before publication
## Content Quality Guidelines
### SYNOPSIS Section
- Show command name in bold: `**command**`
- Use brackets `[]` for optional arguments
- Use italic `*ARG*` for required arguments
- Keep concise (1-5 lines maximum)
- Include 5-150 words
### DESCRIPTION Section
- Start with what the command does
- Explain why users would use it
- Describe main functionality and features
- Minimum 50 words, maximum 1000 words
- At least 3 sentences
### EXAMPLES Section
- Use bash code blocks for commands
- Include comments explaining each example
- Start simple, progress to complex
- Show actual output when helpful
- Cover common use cases first
## Schema Definition
```json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Enhanced Markdown Manpage Schema with Classifications",
"description": "JSON schema for Unix-style manual pages with section classification and content control",
"x-markitect-sections": {
"SYNOPSIS": {
"classification": "required",
"heading_level": 2,
"position": "after_title",
"content_instruction": "Brief command syntax showing all options and arguments in standard format",
"min_paragraphs": 1,
"max_paragraphs": 5,
"min_code_blocks": 0,
"max_code_blocks": 3,
"error_message": "SYNOPSIS section is mandatory for all manpages per Unix conventions"
},
"DESCRIPTION": {
"classification": "required",
"heading_level": 2,
"content_instruction": "Detailed explanation of what the command does, its purpose, and main functionality",
"min_paragraphs": 2,
"max_paragraphs": 50,
"error_message": "DESCRIPTION section is mandatory for all manpages"
},
"EXAMPLES": {
"classification": "recommended",
"heading_level": 2,
"content_instruction": "Practical usage examples with explanations demonstrating common use cases",
"min_code_blocks": 3,
"max_code_blocks": 20,
"warning_if_missing": "Examples greatly improve manpage usability - highly recommended"
},
"SEE ALSO": {
"classification": "recommended",
"heading_level": 2,
"content_instruction": "Related commands, configuration files, and documentation references",
"min_paragraphs": 1,
"warning_if_missing": "Cross-references help users discover related functionality"
},
"OPTIONS": {
"classification": "recommended",
"heading_level": 2,
"content_instruction": "Detailed option descriptions with all flags and their behaviors",
"alternatives": ["GLOBAL OPTIONS", "COMMAND OPTIONS", "FLAGS"],
"warning_if_missing": "Documenting command options helps users understand available functionality"
},
"BUGS": {
"classification": "optional",
"heading_level": 2,
"content_instruction": "Known issues, limitations, and bug reporting information"
},
"AUTHORS": {
"classification": "optional",
"heading_level": 2,
"content_instruction": "List of contributors and maintainers"
},
"COPYRIGHT": {
"classification": "optional",
"heading_level": 2,
"content_instruction": "Copyright statement and license information"
},
"HISTORY": {
"classification": "optional",
"heading_level": 2,
"content_instruction": "Historical information about command development"
},
"DEPRECATED": {
"classification": "discouraged",
"heading_level": 2,
"warning_if_missing": "Consider moving deprecated content to historical documentation or HISTORY section"
},
"OLD_SYNTAX": {
"classification": "discouraged",
"heading_level": 2,
"warning_if_missing": "Old syntax should be documented in HISTORY or removed entirely"
},
"INTERNAL_NOTES": {
"classification": "improper",
"heading_level": 2,
"error_message": "Internal notes must not appear in published manpages - move to developer documentation"
},
"TODO": {
"classification": "improper",
"heading_level": 2,
"error_message": "TODO sections are for development only - remove before publication"
},
"DRAFT": {
"classification": "improper",
"heading_level": 2,
"error_message": "DRAFT markers must be removed before publication"
}
},
"x-markitect-content-control": {
"synopsis": {
"required_patterns": [
"\\*\\*[a-z][a-z0-9-]*\\*\\*",
"\\[.*\\]"
],
"discouraged_patterns": [
"TODO",
"FIXME",
"TBD"
],
"content_quality": {
"min_words": 5,
"max_words": 150,
"readability_target": "technical"
},
"content_instructions": [
"Show command name in bold (e.g., **command**)",
"Use brackets [] for optional arguments",
"Use italic *ARG* for required arguments",
"Keep synopsis concise (1-5 lines maximum)",
"Use ellipsis ... to indicate repeatable arguments"
]
},
"description": {
"discouraged_patterns": [
"TODO",
"FIXME",
"\\bWIP\\b",
"\\bXXX\\b"
],
"forbidden_patterns": [
"password\\s*=\\s*[\"'].*[\"']",
"api[_-]?key\\s*=\\s*[\"'].*[\"']",
"secret\\s*=\\s*[\"'].*[\"']"
],
"content_quality": {
"min_words": 50,
"max_words": 1000,
"readability_target": "technical",
"min_sentences": 3
},
"content_instructions": [
"Start with what the command does",
"Explain why users would use it",
"Describe main functionality and features",
"Mention any prerequisites or requirements",
"Keep technical but accessible"
],
"link_validation": {
"check_internal": true,
"check_external": false,
"allow_fragments": true
}
},
"examples": {
"required_patterns": [
"```",
"#"
],
"content_quality": {
"min_words": 100,
"max_words": 2000,
"readability_target": "general"
},
"content_instructions": [
"Use bash code blocks for command examples",
"Include comments explaining what each example does",
"Start with simple examples, progress to complex",
"Show actual output when helpful",
"Cover common use cases first"
]
}
},
"type": "object",
"properties": {
"headings": {
"type": "object",
"description": "Document heading structure",
"properties": {
"level_1": {
"type": "array",
"description": "Title heading in format: command(section) - description",
"items": {
"type": "object",
"properties": {
"content": {
"type": "string",
"pattern": "^[a-z0-9-]+\\([0-9]\\) - .+"
}
}
},
"minItems": 1,
"maxItems": 1
},
"level_2": {
"type": "array",
"description": "Main section headings",
"minItems": 3,
"maxItems": 30
},
"level_3": {
"type": "array",
"description": "Subsection headings",
"minItems": 0,
"maxItems": 50
}
},
"required": ["level_1", "level_2"]
},
"paragraphs": {
"type": "array",
"description": "Text paragraphs",
"minItems": 10,
"maxItems": 500
},
"code_blocks": {
"type": "array",
"description": "Code examples",
"minItems": 1,
"maxItems": 50
},
"lists": {
"type": "array",
"description": "Lists for options and structured information",
"minItems": 0,
"maxItems": 100
},
"emphasis": {
"type": "array",
"description": "Bold and italic text for commands and arguments",
"minItems": 20,
"maxItems": 500
}
},
"required": ["headings", "paragraphs", "code_blocks", "emphasis"]
}
```
## Version History
### v1.0.0 (2026-01-04)
- Initial markdown schema version
- Migrated from enhanced-manpage JSON schema
- Added comprehensive documentation
- Implemented section classification system
- Added content control and quality guidelines
## Related Documentation
- [Schema Naming Specification](../../roadmap/schema-of-schemas/SCHEMA_NAMING_SPEC.md)
- [Schema Management Workplan](../../roadmap/schema-of-schemas/WORKPLAN.md)
- [MarkiTect Documentation](../../README.md)