markitect-main/roadmap/schema-of-schemas/SCHEMA_MANAGEMENT_PROPOSAL.md

Schema Management Proposal

Status: Draft Created: 2026-01-04 Author: Analysis of current state and proposed improvements

Problem Statement

1. Inconsistent Schema Naming

Current State:

terminology-schema.json       ← Has ".json" suffix
api-documentation             ← No suffix
enhanced-manpage              ← No suffix
markdown-manpage              ← No suffix, duplicate title
markdown-manpage-schema.json  ← Has ".json" suffix, duplicate title

Issues:

• No naming convention enforced
• Duplicate schemas (3 manpage schemas!)
• Mix of suffixed (.json) and non-suffixed names
• No way to distinguish versions

2. Missing Versioning

Current State:

• No version in filenames
• No version in schema metadata (beyond optional $id)
• No way to track schema evolution
• Breaking changes not apparent

Issues:

• Can't have multiple versions simultaneously
• No migration path when schemas change
• Unclear which schema version a document uses

3. Format Mismatch: JSON vs Markdown

The Philosophical Problem:

MarkiTect is a markdown-centric tool, yet schemas are JSON files. This creates a conceptual and practical mismatch.

Current State:

• Documents: Markdown (.md)
• Schemas: JSON (.json)
• No unified format for documentation + schema
• Schemas lack rich documentation capabilities

Proposed Solutions

Part 1: Naming Convention & Versioning

Option A: Filename-Based Versioning (Recommended)

Format: {domain}-{type}-schema-v{major}.{minor}.json

Examples:

manpage-schema-v1.0.json          # Manpage schema v1.0
manpage-schema-v2.0.json          # Breaking change → v2.0
terminology-schema-v1.0.json      # Terminology schema
api-documentation-schema-v1.0.json
arc42-schema-v1.0.json

Benefits:

• Clear versioning in filename
• Easy to see multiple versions
• SemVer compatible (major.minor)
• Searchable/sortable

Migration Strategy:

# Rename existing schemas
markdown-manpage → manpage-schema-v1.0.json
enhanced-manpage → manpage-schema-v2.0.json  # (breaking changes)
terminology-schema.json → terminology-schema-v1.0.json

Option B: $id-Based Versioning

Keep simple filenames, use $id for versioning:

{
  "$id": "https://markitect.dev/schemas/manpage/v1",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "version": "1.0.0",
  ...
}

Filenames: manpage-schema.json, terminology-schema.json

Benefits:

• Clean filenames
• Versioning in metadata
• Follows JSON Schema best practices

Drawbacks:

• Can't have multiple versions in same database
• Harder to see versions at a glance

Recommendation: Hybrid Approach

Combine both for maximum clarity:

// File: manpage-schema-v1.json
{
  "$id": "https://markitect.dev/schemas/manpage/v1",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "version": "1.0.0",
  "title": "Unix Manual Page Schema",
  ...
}

Part 2: Schema Metadata Standard

Add required metadata to all schemas:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://markitect.dev/schemas/{domain}/v{major}",

  // Required metadata
  "version": "1.0.0",                    // SemVer
  "title": "Human Readable Title",
  "description": "Detailed description",

  // Optional metadata
  "x-markitect-schema-type": "document-schema",
  "x-markitect-version": {
    "major": 1,
    "minor": 0,
    "patch": 0
  },
  "x-markitect-author": "MarkiTect Project",
  "x-markitect-created": "2026-01-04",
  "x-markitect-updated": "2026-01-04",
  "x-markitect-deprecated": false,
  "x-markitect-superseded-by": null,
  "x-markitect-document-types": ["manpage", "manual"],
  "x-markitect-example": "examples/manpages/example.md",

  // Schema content
  "type": "object",
  "properties": { ... }
}

Part 3: Format Mismatch Solutions

Option 1: Markdown-First with Embedded JSON (Recommended)

File Format: Markdown with frontmatter and JSON code block

---
schema-version: "1.0.0"
schema-id: "https://markitect.dev/schemas/manpage/v1"
document-type: manpage
status: stable
---

# Manpage Schema v1.0

## Overview

This schema validates Unix/Linux manual page documentation following
standard conventions (SYNOPSIS, DESCRIPTION, OPTIONS, etc.).

## Document Types

- Manual pages (man pages)
- CLI command documentation
- API reference pages

## Usage

\`\`\`bash
markitect validate mycommand.1.md --schema manpage-schema-v1
\`\`\`

## Examples

See [examples/manpages/](../../examples/manpages/) for complete examples.

## Schema Definition

\`\`\`json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://markitect.dev/schemas/manpage/v1",
  "version": "1.0.0",
  "title": "Unix Manual Page Schema",
  "type": "object",
  "properties": {
    "headings": {
      "type": "object",
      "properties": {
        "level_1": { ... }
      }
    }
  },
  "x-markitect-sections": { ... }
}
\`\`\`

## Validation Rules

### Required Sections
- **NAME** - Command name and brief description
- **SYNOPSIS** - Command syntax
- **DESCRIPTION** - Detailed description

### Optional Sections
- **OPTIONS** - Command-line options
- **EXAMPLES** - Usage examples
- **SEE ALSO** - Related commands

## Version History

### v1.0.0 (2026-01-04)
- Initial release
- Basic manpage structure validation

Implementation:

class MarkdownSchemaLoader:
    """Load schemas from markdown files with embedded JSON."""

    def load_schema_from_markdown(self, md_path: Path) -> dict:
        """Extract JSON schema from markdown file."""
        content = md_path.read_text()

        # Parse frontmatter
        frontmatter = self._extract_frontmatter(content)

        # Extract JSON from code block
        schema_json = self._extract_json_from_code_block(content)

        # Merge metadata
        schema = json.loads(schema_json)
        schema['x-markitect-metadata'] = frontmatter

        return schema

    def save_schema_to_markdown(self, schema: dict, md_path: Path):
        """Save schema as markdown with embedded JSON."""
        # Generate markdown documentation
        doc = self._generate_schema_documentation(schema)

        # Embed JSON schema
        json_block = f"```json\n{json.dumps(schema, indent=2)}\n```"

        # Combine
        full_content = f"{doc}\n\n## Schema Definition\n\n{json_block}"
        md_path.write_text(full_content)

Benefits:

• ✅ Markdown-first (aligns with MarkiTect philosophy)
• ✅ Rich documentation alongside schema
• ✅ Human-readable and editable
• ✅ Version history in same file
• ✅ Examples and usage inline
• ✅ Can extract JSON when needed

Drawbacks:

• ⚠️ Requires parsing logic
• ⚠️ Two sources of truth (markdown + embedded JSON)
• ⚠️ More complex than pure JSON

Option 2: Markdown Documentation Generator

Keep JSON schemas, auto-generate markdown docs:

schemas/
  manpage-schema-v1.json          # Source of truth
  manpage-schema-v1.md            # Auto-generated docs

Command:

markitect schema-document manpage-schema-v1.json
# Generates: manpage-schema-v1.md

Benefits:

• ✅ Simple implementation
• ✅ JSON remains source of truth
• ✅ Auto-generated docs always in sync

Drawbacks:

• ⚠️ Two files to manage
• ⚠️ Can't hand-edit documentation (gets overwritten)

Option 3: Markdown Schema Language (DSL)

Define schemas in markdown-native syntax:

# Manpage Schema v1.0

## Document Structure

### Required Sections (Level 1 Heading)

**NAME**
- Classification: required
- Content: Command name in bold, followed by description
- Pattern: `**command** - description`

**SYNOPSIS**
- Classification: required
- Content: Command syntax with options
- Min paragraphs: 1
- Max paragraphs: 3

### Optional Sections

**OPTIONS**
- Classification: recommended
- Content: Definition list of command-line options

Parser generates JSON schema from markdown:

markitect schema-compile manpage-schema-v1.md --output manpage-schema-v1.json

Benefits:

• ✅ Pure markdown
• ✅ Human-friendly syntax
• ✅ No JSON editing needed

Drawbacks:

• ⚠️ Complex parser implementation
• ⚠️ Limited to MarkiTect-specific features
• ⚠️ Can't use standard JSON Schema tools

Option 4: Literate Schema Programming

Inspired by literate programming, mix documentation and schema:

# Manpage Schema v1.0

Manual pages follow a standard structure. The NAME section is required:

<<define-name-section>>=
{
  "NAME": {
    "classification": "required",
    "heading_level": 2,
    "content_instruction": "Command name and brief description"
  }
}

The SYNOPSIS section shows command syntax:

<<define-synopsis-section>>=
{
  "SYNOPSIS": {
    "classification": "required",
    "heading_level": 2,
    "min_code_blocks": 1
  }
}

Complete schema:

<<manpage-schema.json>>=
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "x-markitect-sections": {
    <<define-name-section>>,
    <<define-synopsis-section>>
  }
}

Benefits:

• ✅ Documentation and schema interleaved
• ✅ Literate programming benefits
• ✅ Reusable schema fragments

Drawbacks:

• ⚠️ Complex tangling/weaving
• ⚠️ Unfamiliar paradigm
• ⚠️ Overkill for simple schemas

Recommendations

Short-Term (Immediate)

1. Naming Convention:

   • Format: {domain}-schema-v{major}.{minor}.json
   • Example: manpage-schema-v1.0.json

2. Schema Metadata:

   • Add required version, title, description fields
   • Add x-markitect-* metadata extensions
   • Document in schema-catalog.yaml

3. Duplicate Cleanup:

   • Consolidate 3 manpage schemas into versioned series
   • Keep enhanced-manpage as v2.0 (breaking changes)
   • Archive old schemas

Medium-Term (Next Phase)

4. Markdown Schema Format (Option 1):

   • Implement markdown-first schema format
   • Markdown file with embedded JSON in code block
   • Parser extracts JSON for validation
   • Rich documentation alongside schema

5. Schema Documentation Generator:

   • Auto-generate markdown docs from JSON schemas
   • Include examples, usage, version history
   • Link to example documents

Long-Term (Future)

6. Schema DSL (Option 3):

   • Evaluate markdown schema language
   • Prototype parser for common patterns
   • Consider if DSL adds value over JSON

7. Schema Registry API:

   • REST API for schema discovery
   • Version negotiation
   • Schema evolution tracking

Implementation Plan

Phase 1: Naming & Versioning (1-2 days)

Tasks:

1. Define naming convention spec
2. Create schema metadata template
3. Rename existing schemas
4. Update schema-catalog.yaml
5. Update documentation

Deliverables:

• Schema naming convention spec
• Migrated schemas with versions
• Updated catalog

Phase 2: Markdown Schema Format (3-5 days)

Tasks:

1. Design markdown schema format
2. Implement parser (extract JSON from markdown)
3. Implement generator (create markdown from JSON)
4. Convert existing schemas to markdown format
5. Update CLI to support .md schemas
6. Write documentation and examples

Deliverables:

• Markdown schema parser/generator
• All schemas in markdown format
• Updated CLI commands
• Migration guide

Phase 3: Schema Validation (2-3 days)

Tasks:

1. Create metaschema for validating schemas
2. Add schema validation command
3. Validate all existing schemas
4. Add CI check for schema validity

Deliverables:

• Schema-for-schemas (metaschema)
• Validation command
• CI integration

Cost-Benefit Analysis

Option 1: Markdown-First (Recommended)

Cost:

• Parser implementation: ~200 lines
• CLI updates: ~100 lines
• Migration effort: 2-3 days
• Testing: 1 day

Benefit:

• Aligned with markdown philosophy ⭐⭐⭐⭐⭐
• Rich documentation ⭐⭐⭐⭐⭐
• Version history inline ⭐⭐⭐⭐
• Human-friendly ⭐⭐⭐⭐⭐
• Lower barrier to entry ⭐⭐⭐⭐

Total: High value for reasonable cost

Option 2: Documentation Generator

Cost:

• Generator implementation: ~150 lines
• Template design: 1 day
• Testing: 0.5 days

Benefit:

• Simple implementation ⭐⭐⭐⭐
• Auto-sync docs ⭐⭐⭐⭐
• JSON remains source ⭐⭐⭐

Total: Good value, lower cost

Option 3: Schema DSL

Cost:

• DSL design: 2-3 days
• Parser implementation: ~500 lines
• Compiler: ~300 lines
• Testing: 2 days
• Documentation: 1 day

Benefit:

• Pure markdown ⭐⭐⭐⭐⭐
• No JSON editing ⭐⭐⭐⭐
• Limited ecosystem ⭐⭐

Total: High cost, uncertain value

Decision Matrix

Criterion	Option 1: Markdown-First	Option 2: Doc Generator	Option 3: DSL
Markdown alignment	⭐⭐⭐⭐⭐	⭐⭐⭐	⭐⭐⭐⭐⭐
Implementation cost	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐
Documentation quality	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐⭐
Tool ecosystem	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐
Maintainability	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐
User-friendliness	⭐⭐⭐⭐⭐	⭐⭐⭐	⭐⭐⭐⭐

Recommendation Summary

1. Immediate: Implement naming convention and versioning
2. Short-term: Choose Option 1 (Markdown-First) for schema format
3. Fallback: If Option 1 proves too complex, use Option 2 (Doc Generator)
4. Future: Evaluate DSL if community demand emerges

Next Steps

1. Review and approve this proposal
2. Create naming convention specification
3. Prototype markdown schema parser
4. Migrate one schema as proof-of-concept
5. Gather feedback and iterate
6. Full migration of all schemas

---

Appendix: Example Markdown Schema

See examples/schemas/manpage-schema-v1.md for a complete example of the proposed format.