markitect-main/docs/user-guides/migration-guide.md

Migration Guide: Upgrading to MarkiTect's Enhanced Explode-Implode System

Step-by-step guide for migrating existing documentation workflows to MarkiTect's advanced explode-implode and packaging systems

Table of Contents

1. Overview
2. Pre-Migration Assessment
3. Migration Scenarios
4. Step-by-Step Migration
5. Validation and Testing
6. Common Issues and Solutions
7. Rollback Procedures
8. Best Practices

---

Overview

This guide helps you migrate from:

• Basic MarkiTect installations to the enhanced explode-implode system
• Manual directory organization to structured variant-based workflows
• Legacy documentation systems to MarkiTect's integrated approach
• Simple file-based workflows to advanced packaging and templating

What's New

Enhanced Features Available After Migration:

• 🔄 Three organizational variants (flat, hierarchical, semantic)
• 📦 MDZ packaging with asset management
• 🔗 Template-based transclusion (MDT format)
• 🤖 Auto-detection of exploded structures
• 📊 Manifest-based reversibility
• ⚡ CLI command integration

Compatibility

Supported Migration Paths:

• MarkiTect 0.x → 1.0+ (full migration required)
• Manual documentation workflows → MarkiTect structured workflows
• GitBook, MkDocs, Sphinx → MarkiTect (with content adaptation)
• Simple markdown files → Advanced explode-implode workflows

Pre-Migration Assessment

Step 1: Inventory Current Setup

Run this assessment to understand your current state:

# Check your current MarkiTect version
markitect version

# List your current documentation structure
find . -name "*.md" -type f | head -20

# Check for existing MarkiTect files
find . -name "*.mdd" -o -name "manifest.md" -o -name "*.mdz"

# Assess document sizes (identifies candidates for explosion)
find . -name "*.md" -exec wc -l {} \; | sort -nr | head -10

Step 2: Backup Current Setup

Critical: Always backup before migration

# Create timestamped backup
backup_dir="backup-$(date +%Y%m%d-%H%M%S)"
mkdir "$backup_dir"

# Backup all markdown and config files
cp -r docs/ "$backup_dir/" 2>/dev/null || true
cp -r *.md "$backup_dir/" 2>/dev/null || true
cp -r .markitect* "$backup_dir/" 2>/dev/null || true

echo "Backup created in: $backup_dir"

Step 3: Assess Migration Complexity

Use this checklist to determine your migration path:

Simple Migration (1-2 hours):

• Small documentation set (< 10 files)
• Basic markdown structure
• No custom build processes
• Willing to use recommended variants

Moderate Migration (4-8 hours):

• Medium documentation set (10-50 files)
• Some organizational structure exists
• Basic build/CI processes
• Custom asset management needs

Complex Migration (1-3 days):

• Large documentation set (50+ files)
• Complex existing organization
• Extensive build/CI integration
• Custom tooling and workflows

Migration Scenarios

Scenario A: From Basic MarkiTect

Situation: You have MarkiTect installed but haven't used explode-implode features.

Migration Steps:

1. Update MarkiTect:

   pip install --upgrade markitect
   markitect version  # Verify 1.0+
   

2. Test new commands:

   # Verify new commands are available
   markitect md-explode --help
   markitect md-package --help
   markitect md-transclude --help
   

3. Choose your first document:

   # Start with a medium-sized document
   markitect md-explode your-document.md --variant hierarchical --dry-run
   

4. Perform first explosion:

   markitect md-explode your-document.md --variant hierarchical --create-manifest
   

Scenario B: From Manual Directory Organization

Situation: You manually organize markdown files in directories.

Current Structure Example:

docs/
├── intro/
│   ├── overview.md
│   └── getting-started.md
├── tutorials/
│   ├── basic-usage.md
│   └── advanced-features.md
└── reference/
    └── api.md

Migration Steps:

1. Assess current organization:

   # Analyze your structure
   tree docs/ > current-structure.txt
   
   # Count sections across files
   grep -r "^#" docs/ | wc -l
   

2. Choose migration strategy:

   Option A: Convert to Semantic Variant

   # Combines files into semantic structure
   markitect md-implode docs/ --force-variant semantic --output combined-docs.md
   
   # Then re-explode with proper manifest
   markitect md-explode combined-docs.md --variant semantic --create-manifest
   

   Option B: Preserve as Flat Structure

   # Convert to flat variant maintaining file boundaries
   markitect md-implode docs/ --force-variant flat --output combined-docs.md
   markitect md-explode combined-docs.md --variant flat --create-manifest
   

3. Validate result:

   # Check the new structure
   tree combined-docs.mdd/
   
   # Test round-trip
   markitect md-implode combined-docs.mdd/ --output test-rebuild.md
   diff combined-docs.md test-rebuild.md
   

Scenario C: From GitBook

Situation: Migrating from GitBook to MarkiTect.

GitBook Structure:

docs/
├── SUMMARY.md
├── README.md
├── chapter1/
│   ├── README.md
│   └── section1.md
└── chapter2/
    └── README.md

Migration Steps:

1. Convert GitBook structure:

   # Create conversion script
   cat > convert-gitbook.py << 'EOF'
   #!/usr/bin/env python3
   import os
   from pathlib import Path
   
   def convert_gitbook_to_single_md():
       # Parse SUMMARY.md to understand structure
       # Combine files in order
       # Generate single markdown file
       pass
   
   if __name__ == "__main__":
       convert_gitbook_to_single_md()
   EOF
   
   python convert-gitbook.py
   

2. Apply MarkiTect structure:

   markitect md-explode converted-document.md --variant hierarchical --create-manifest
   

Scenario D: From Legacy Documentation Systems

Situation: Migrating from Sphinx, MkDocs, or similar systems.

Migration Approach:

1. Export to markdown:

   # For Sphinx (using pandoc)
   find . -name "*.rst" -exec pandoc {} -f rst -t markdown -o {}.md \;
   
   # For MkDocs, files may already be markdown
   # Check mkdocs.yml for structure information
   

2. Combine and structure:

   # Create combined document preserving hierarchy
   # (You may need custom script based on your system)
   
   # Then apply MarkiTect organization
   markitect md-explode legacy-combined.md --variant hierarchical
   

Step-by-Step Migration

Phase 1: Preparation (30 minutes)

1. Environment Setup:

   # Ensure latest version
   pip install --upgrade markitect
   
   # Verify installation
   markitect version
   markitect md-explode --help
   markitect md-package --help
   

2. Create working directory:

   mkdir markitect-migration
   cd markitect-migration
   
   # Copy source documents
   cp -r /path/to/current/docs ./source-docs/
   

3. Assess document complexity:

   # Find largest documents (good candidates for explosion)
   find source-docs/ -name "*.md" -exec wc -l {} \; | sort -nr > document-sizes.txt
   
   # Identify documents with many sections
   find source-docs/ -name "*.md" -exec bash -c 'echo -n "{}: "; grep -c "^#" "{}"' \; > section-counts.txt
   

Phase 2: Small-Scale Testing (1 hour)

1. Choose pilot document:

   # Select medium-sized document from analysis
   pilot_doc="source-docs/user-guide.md"  # Replace with your choice
   

2. Test explosion variants:

   # Test all variants with dry-run
   markitect md-explode "$pilot_doc" --variant flat --dry-run --verbose
   markitect md-explode "$pilot_doc" --variant hierarchical --dry-run --verbose
   markitect md-explode "$pilot_doc" --variant semantic --dry-run --verbose
   

3. Perform pilot explosion:

   # Choose best variant and execute
   markitect md-explode "$pilot_doc" --variant hierarchical --create-manifest --output pilot-exploded/
   
   # Validate round-trip
   markitect md-implode pilot-exploded/ --output pilot-rebuilt.md
   diff "$pilot_doc" pilot-rebuilt.md
   

Phase 3: Full Migration (2-4 hours)

1. Process all documents:

   # Create batch processing script
   cat > migrate-all.sh << 'EOF'
   #!/bin/bash
   
   # Configure migration settings
   VARIANT="hierarchical"  # Change as needed
   SOURCE_DIR="source-docs"
   OUTPUT_DIR="migrated-docs"
   
   mkdir -p "$OUTPUT_DIR"
   
   # Process each markdown file
   find "$SOURCE_DIR" -name "*.md" | while read -r file; do
       echo "Processing: $file"
   
       # Generate output path
       relative_path=$(realpath --relative-to="$SOURCE_DIR" "$file")
       basename_no_ext=$(basename "$file" .md)
       output_subdir="$OUTPUT_DIR/${basename_no_ext}.mdd"
   
       # Explode document
       markitect md-explode "$file" --variant "$VARIANT" --create-manifest --output "$output_subdir"
   
       # Validate
       temp_rebuilt="/tmp/${basename_no_ext}-rebuilt.md"
       markitect md-implode "$output_subdir" --output "$temp_rebuilt"
   
       if diff -q "$file" "$temp_rebuilt" > /dev/null; then
           echo "✓ Migration successful: $file"
       else
           echo "⚠ Migration issues detected: $file"
           echo "   Check: $temp_rebuilt vs $file"
       fi
   done
   EOF
   
   chmod +x migrate-all.sh
   ./migrate-all.sh
   

2. Validate migration results:

   # Check all migrations
   find migrated-docs/ -name "manifest.md" | wc -l  # Should equal input document count
   
   # Test random sampling
   find migrated-docs/ -name "*.mdd" | shuf | head -3 | while read -r exploded_dir; do
       echo "Testing: $exploded_dir"
       temp_rebuilt="/tmp/$(basename "$exploded_dir" .mdd)-test.md"
       markitect md-implode "$exploded_dir" --output "$temp_rebuilt"
       echo "Rebuilt to: $temp_rebuilt"
   done
   

Phase 4: Integration and Packaging (1-2 hours)

1. Set up packaging workflow:

   # Create packaging script
   cat > package-docs.sh << 'EOF'
   #!/bin/bash
   
   MIGRATED_DIR="migrated-docs"
   PACKAGES_DIR="packages"
   
   mkdir -p "$PACKAGES_DIR"
   
   # Package each exploded document
   find "$MIGRATED_DIR" -name "*.mdd" | while read -r exploded_dir; do
       basename_no_ext=$(basename "$exploded_dir" .mdd)
   
       # First implode to single document
       temp_doc="/tmp/${basename_no_ext}.md"
       markitect md-implode "$exploded_dir" --output "$temp_doc"
   
       # Then package
       output_package="$PACKAGES_DIR/${basename_no_ext}.mdz"
       markitect md-package create "$temp_doc" --format mdz --output "$output_package"
   
       echo "Packaged: $output_package"
   
       # Verify package
       markitect md-package info "$output_package"
   done
   EOF
   
   chmod +x package-docs.sh
   ./package-docs.sh
   

2. Set up templating (optional):

   # If you have template needs, create MDT templates
   mkdir templates/
   
   # Example template creation
   cat > templates/user-guide-template.mdt << 'EOF'
   # {{title}}
   
   **Version:** {{version}}
   **Author:** {{author}}
   **Updated:** {{date}}
   
   {{include "introduction.md"}}
   
   ## Features
   
   {{include "features.md"}}
   
   {{if include_advanced}}
   ## Advanced Topics
   
   {{include "advanced.md"}}
   {{endif}}
   EOF
   
   # Test template
   cat > templates/variables.json << 'EOF'
   {
     "title": "Migrated User Guide",
     "version": "2.0.0",
     "author": "Documentation Team",
     "date": "2025-10-14",
     "include_advanced": true
   }
   EOF
   
   markitect md-transclude validate templates/user-guide-template.mdt
   

Validation and Testing

Automated Validation

# Create validation script
cat > validate-migration.sh << 'EOF'
#!/bin/bash

echo "=== Migration Validation Report ==="

# Count original files
orig_count=$(find source-docs/ -name "*.md" | wc -l)
echo "Original documents: $orig_count"

# Count migrated structures
migrated_count=$(find migrated-docs/ -name "*.mdd" | wc -l)
echo "Migrated structures: $migrated_count"

# Count packages
package_count=$(find packages/ -name "*.mdz" | wc -l 2>/dev/null || echo "0")
echo "Created packages: $package_count"

# Test round-trip integrity
echo
echo "=== Round-trip Tests ==="
failed_tests=0

find migrated-docs/ -name "*.mdd" | head -5 | while read -r exploded_dir; do
    basename_no_ext=$(basename "$exploded_dir" .mdd)
    original_file="source-docs/${basename_no_ext}.md"

    if [ -f "$original_file" ]; then
        temp_rebuilt="/tmp/${basename_no_ext}-validation.md"
        markitect md-implode "$exploded_dir" --output "$temp_rebuilt"

        if diff -q "$original_file" "$temp_rebuilt" > /dev/null; then
            echo "✓ PASS: $basename_no_ext"
        else
            echo "✗ FAIL: $basename_no_ext"
            failed_tests=$((failed_tests + 1))
        fi
    fi
done

if [ $failed_tests -eq 0 ]; then
    echo
    echo "🎉 All validation tests passed!"
else
    echo
    echo "⚠ $failed_tests validation tests failed. Review differences manually."
fi
EOF

chmod +x validate-migration.sh
./validate-migration.sh

Manual Validation Checklist

Document Structure:

• All original sections are preserved
• Heading hierarchy is maintained
• Code blocks are intact
• Links and references work correctly
• Images and assets are accessible

Functionality:

• Explode operations complete without errors
• Implode operations produce identical content
• Variant detection works correctly
• Packaging creates valid MDZ files
• Templates process correctly (if used)

Integration:

• CLI commands integrate with existing workflows
• Build processes adapt successfully
• Version control handles new file structures
• Team members can use new workflows

Common Issues and Solutions

Issue 1: "Failed to detect variant" Error

Symptoms:

Error: Failed to detect variant for directory 'docs/'

Solutions:

# Option 1: Force specific variant
markitect md-implode docs/ --force-variant flat

# Option 2: Create proper structure first
markitect md-explode combined-doc.md --variant hierarchical --create-manifest

Issue 2: Large Document Memory Issues

Symptoms:

• Slow processing of large documents
• Out of memory errors

Solutions:

# Split large documents manually first
split -l 1000 large-document.md split-doc-

# Process parts separately
for part in split-doc-*; do
    markitect md-explode "$part" --variant flat
done

# Or use streaming approach (if available)
# markitect md-explode large-document.md --streaming --variant hierarchical

Issue 3: Asset Path Issues

Symptoms:

• Images don't display after migration
• Broken asset references

Solutions:

# Check asset discovery
markitect md-package create document.md --dry-run --verbose

# Fix relative paths in markdown
sed -i 's|\.\./assets/|./assets/|g' document.md

# Verify asset packaging
markitect md-package info document.mdz --verbose

Issue 4: Manifest Corruption

Symptoms:

Error: Invalid manifest format in 'manifest.md'

Solutions:

# Backup corrupted manifest
cp manifest.md manifest.md.backup

# Regenerate manifest
markitect md-explode original-document.md --variant hierarchical --create-manifest --force

# Or manually fix YAML frontmatter

Issue 5: Template Processing Errors

Symptoms:

Error: Circular reference detected in template

Solutions:

# Validate template structure
markitect md-transclude validate template.mdt --verbose

# Check for circular includes
grep -r "{{include" templates/ | sort

# Fix circular references by restructuring includes

Rollback Procedures

Complete Rollback

# Restore from backup
backup_dir="backup-20251014-100000"  # Your backup timestamp

# Stop and restore
rm -rf migrated-docs/ packages/ templates/
cp -r "$backup_dir/"* ./

echo "Rollback completed. You're back to the original state."

Selective Rollback

# Rollback specific documents only
problem_docs=("user-guide" "api-reference")

for doc in "${problem_docs[@]}"; do
    # Remove migrated version
    rm -rf "migrated-docs/${doc}.mdd"

    # Restore original
    cp "$backup_dir/${doc}.md" ./

    echo "Rolled back: $doc"
done

Partial Migration Approach

If full migration is problematic, use a gradual approach:

# Phase 1: Migrate only small documents
find source-docs/ -name "*.md" -exec wc -l {} \; | awk '$1 < 200' | cut -d: -f2 | while read -r file; do
    markitect md-explode "$file" --variant flat --create-manifest
done

# Phase 2: Migrate medium documents (when comfortable)
# Phase 3: Migrate large documents (with assistance)

Best Practices

Migration Planning

1. Start Small: Begin with 1-2 documents to understand the process
2. Test Variants: Try different variants to find the best fit
3. Validate Early: Check round-trip integrity frequently
4. Backup Everything: Multiple backups at different stages
5. Document Changes: Keep migration notes for team members

Workflow Integration

# Add to Makefile
migrate-docs:
	./migrate-all.sh

validate-migration:
	./validate-migration.sh

package-docs:
	./package-docs.sh

clean-migration:
	rm -rf migrated-docs/ packages/
	git checkout -- source-docs/

Team Collaboration

1. Training: Ensure team understands new commands
2. Documentation: Update team docs with new workflows
3. Gradual Adoption: Allow parallel workflows during transition
4. Support: Designate migration champions to help others

Maintenance

# Regular validation script
cat > .github/workflows/validate-docs.yml << 'EOF'
name: Validate Documentation Structure

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Install MarkiTect
      run: pip install markitect
    - name: Validate structures
      run: |
        find . -name "*.mdd" | while read -r dir; do
          echo "Validating: $dir"
          markitect md-implode "$dir" --dry-run --verbose
        done
EOF

---

Migration Checklist

Pre-Migration

• Backup all documentation
• Install latest MarkiTect version
• Assess current documentation structure
• Choose migration strategy
• Test with pilot documents

During Migration

• Process documents systematically
• Validate each step
• Document any issues encountered
• Test package creation
• Verify asset handling

Post-Migration

• Update team documentation
• Train team members on new workflows
• Update CI/CD processes
• Establish maintenance procedures
• Plan rollback procedures (just in case)

Migration Support: For complex migrations or issues not covered in this guide, create detailed issue reports with your specific setup and requirements.

---

Version: 1.0.0 Last Updated: 2025-10-14 Compatibility: MarkiTect 1.0+