markitect-main/docs/user-guides/migration-guide.md

# Migration Guide: Upgrading to MarkiTect's Enhanced Explode-Implode System

**Step-by-step guide for migrating existing documentation workflows to MarkiTect's advanced explode-implode and packaging systems**

## Table of Contents

1. [Overview](#overview)
2. [Pre-Migration Assessment](#pre-migration-assessment)
3. [Migration Scenarios](#migration-scenarios)
4. [Step-by-Step Migration](#step-by-step-migration)
5. [Validation and Testing](#validation-and-testing)
6. [Common Issues and Solutions](#common-issues-and-solutions)
7. [Rollback Procedures](#rollback-procedures)
8. [Best Practices](#best-practices)

---

## Overview

This guide helps you migrate from:
- **Basic MarkiTect installations** to the enhanced explode-implode system
- **Manual directory organization** to structured variant-based workflows
- **Legacy documentation systems** to MarkiTect's integrated approach
- **Simple file-based workflows** to advanced packaging and templating

### What's New

**Enhanced Features Available After Migration:**
- 🔄 **Three organizational variants** (flat, hierarchical, semantic)
- 📦 **MDZ packaging** with asset management
- 🔗 **Template-based transclusion** (MDT format)
- 🤖 **Auto-detection** of exploded structures
- 📊 **Manifest-based reversibility**
- ⚡ **CLI command integration**

### Compatibility

**Supported Migration Paths:**
- MarkiTect 0.x → 1.0+ (full migration required)
- Manual documentation workflows → MarkiTect structured workflows
- GitBook, MkDocs, Sphinx → MarkiTect (with content adaptation)
- Simple markdown files → Advanced explode-implode workflows

## Pre-Migration Assessment

### Step 1: Inventory Current Setup

Run this assessment to understand your current state:

```bash
# Check your current MarkiTect version
markitect version

# List your current documentation structure
find . -name "*.md" -type f | head -20

# Check for existing MarkiTect files
find . -name "*.mdd" -o -name "manifest.md" -o -name "*.mdz"

# Assess document sizes (identifies candidates for explosion)
find . -name "*.md" -exec wc -l {} \; | sort -nr | head -10
```

### Step 2: Backup Current Setup

**Critical: Always backup before migration**

```bash
# Create timestamped backup
backup_dir="backup-$(date +%Y%m%d-%H%M%S)"
mkdir "$backup_dir"

# Backup all markdown and config files
cp -r docs/ "$backup_dir/" 2>/dev/null || true
cp -r *.md "$backup_dir/" 2>/dev/null || true
cp -r .markitect* "$backup_dir/" 2>/dev/null || true

echo "Backup created in: $backup_dir"
```

### Step 3: Assess Migration Complexity

Use this checklist to determine your migration path:

**Simple Migration (1-2 hours):**
- [ ] Small documentation set (< 10 files)
- [ ] Basic markdown structure
- [ ] No custom build processes
- [ ] Willing to use recommended variants

**Moderate Migration (4-8 hours):**
- [ ] Medium documentation set (10-50 files)
- [ ] Some organizational structure exists
- [ ] Basic build/CI processes
- [ ] Custom asset management needs

**Complex Migration (1-3 days):**
- [ ] Large documentation set (50+ files)
- [ ] Complex existing organization
- [ ] Extensive build/CI integration
- [ ] Custom tooling and workflows

## Migration Scenarios

### Scenario A: From Basic MarkiTect

**Situation:** You have MarkiTect installed but haven't used explode-implode features.

**Migration Steps:**

1. **Update MarkiTect:**
   ```bash
   pip install --upgrade markitect
   markitect version  # Verify 1.0+
   ```

2. **Test new commands:**
   ```bash
   # Verify new commands are available
   markitect md-explode --help
   markitect md-package --help
   markitect md-transclude --help
   ```

3. **Choose your first document:**
   ```bash
   # Start with a medium-sized document
   markitect md-explode your-document.md --variant hierarchical --dry-run
   ```

4. **Perform first explosion:**
   ```bash
   markitect md-explode your-document.md --variant hierarchical --create-manifest
   ```

### Scenario B: From Manual Directory Organization

**Situation:** You manually organize markdown files in directories.

**Current Structure Example:**
```
docs/
├── intro/
│   ├── overview.md
│   └── getting-started.md
├── tutorials/
│   ├── basic-usage.md
│   └── advanced-features.md
└── reference/
    └── api.md
```

**Migration Steps:**

1. **Assess current organization:**
   ```bash
   # Analyze your structure
   tree docs/ > current-structure.txt

   # Count sections across files
   grep -r "^#" docs/ | wc -l
   ```

2. **Choose migration strategy:**

   **Option A: Convert to Semantic Variant**
   ```bash
   # Combines files into semantic structure
   markitect md-implode docs/ --force-variant semantic --output combined-docs.md

   # Then re-explode with proper manifest
   markitect md-explode combined-docs.md --variant semantic --create-manifest
   ```

   **Option B: Preserve as Flat Structure**
   ```bash
   # Convert to flat variant maintaining file boundaries
   markitect md-implode docs/ --force-variant flat --output combined-docs.md
   markitect md-explode combined-docs.md --variant flat --create-manifest
   ```

3. **Validate result:**
   ```bash
   # Check the new structure
   tree combined-docs.mdd/

   # Test round-trip
   markitect md-implode combined-docs.mdd/ --output test-rebuild.md
   diff combined-docs.md test-rebuild.md
   ```

### Scenario C: From GitBook

**Situation:** Migrating from GitBook to MarkiTect.

**GitBook Structure:**
```
docs/
├── SUMMARY.md
├── README.md
├── chapter1/
│   ├── README.md
│   └── section1.md
└── chapter2/
    └── README.md
```

**Migration Steps:**

1. **Convert GitBook structure:**
   ```bash
   # Create conversion script
   cat > convert-gitbook.py << 'EOF'
   #!/usr/bin/env python3
   import os
   from pathlib import Path

   def convert_gitbook_to_single_md():
       # Parse SUMMARY.md to understand structure
       # Combine files in order
       # Generate single markdown file
       pass

   if __name__ == "__main__":
       convert_gitbook_to_single_md()
   EOF

   python convert-gitbook.py
   ```

2. **Apply MarkiTect structure:**
   ```bash
   markitect md-explode converted-document.md --variant hierarchical --create-manifest
   ```

### Scenario D: From Legacy Documentation Systems

**Situation:** Migrating from Sphinx, MkDocs, or similar systems.

**Migration Approach:**

1. **Export to markdown:**
   ```bash
   # For Sphinx (using pandoc)
   find . -name "*.rst" -exec pandoc {} -f rst -t markdown -o {}.md \;

   # For MkDocs, files may already be markdown
   # Check mkdocs.yml for structure information
   ```

2. **Combine and structure:**
   ```bash
   # Create combined document preserving hierarchy
   # (You may need custom script based on your system)

   # Then apply MarkiTect organization
   markitect md-explode legacy-combined.md --variant hierarchical
   ```

## Step-by-Step Migration

### Phase 1: Preparation (30 minutes)

1. **Environment Setup:**
   ```bash
   # Ensure latest version
   pip install --upgrade markitect

   # Verify installation
   markitect version
   markitect md-explode --help
   markitect md-package --help
   ```

2. **Create working directory:**
   ```bash
   mkdir markitect-migration
   cd markitect-migration

   # Copy source documents
   cp -r /path/to/current/docs ./source-docs/
   ```

3. **Assess document complexity:**
   ```bash
   # Find largest documents (good candidates for explosion)
   find source-docs/ -name "*.md" -exec wc -l {} \; | sort -nr > document-sizes.txt

   # Identify documents with many sections
   find source-docs/ -name "*.md" -exec bash -c 'echo -n "{}: "; grep -c "^#" "{}"' \; > section-counts.txt
   ```

### Phase 2: Small-Scale Testing (1 hour)

1. **Choose pilot document:**
   ```bash
   # Select medium-sized document from analysis
   pilot_doc="source-docs/user-guide.md"  # Replace with your choice
   ```

2. **Test explosion variants:**
   ```bash
   # Test all variants with dry-run
   markitect md-explode "$pilot_doc" --variant flat --dry-run --verbose
   markitect md-explode "$pilot_doc" --variant hierarchical --dry-run --verbose
   markitect md-explode "$pilot_doc" --variant semantic --dry-run --verbose
   ```

3. **Perform pilot explosion:**
   ```bash
   # Choose best variant and execute
   markitect md-explode "$pilot_doc" --variant hierarchical --create-manifest --output pilot-exploded/

   # Validate round-trip
   markitect md-implode pilot-exploded/ --output pilot-rebuilt.md
   diff "$pilot_doc" pilot-rebuilt.md
   ```

### Phase 3: Full Migration (2-4 hours)

1. **Process all documents:**
   ```bash
   # Create batch processing script
   cat > migrate-all.sh << 'EOF'
   #!/bin/bash

   # Configure migration settings
   VARIANT="hierarchical"  # Change as needed
   SOURCE_DIR="source-docs"
   OUTPUT_DIR="migrated-docs"

   mkdir -p "$OUTPUT_DIR"

   # Process each markdown file
   find "$SOURCE_DIR" -name "*.md" | while read -r file; do
       echo "Processing: $file"

       # Generate output path
       relative_path=$(realpath --relative-to="$SOURCE_DIR" "$file")
       basename_no_ext=$(basename "$file" .md)
       output_subdir="$OUTPUT_DIR/${basename_no_ext}.mdd"

       # Explode document
       markitect md-explode "$file" --variant "$VARIANT" --create-manifest --output "$output_subdir"

       # Validate
       temp_rebuilt="/tmp/${basename_no_ext}-rebuilt.md"
       markitect md-implode "$output_subdir" --output "$temp_rebuilt"

       if diff -q "$file" "$temp_rebuilt" > /dev/null; then
           echo "✓ Migration successful: $file"
       else
           echo "⚠ Migration issues detected: $file"
           echo "   Check: $temp_rebuilt vs $file"
       fi
   done
   EOF

   chmod +x migrate-all.sh
   ./migrate-all.sh
   ```

2. **Validate migration results:**
   ```bash
   # Check all migrations
   find migrated-docs/ -name "manifest.md" | wc -l  # Should equal input document count

   # Test random sampling
   find migrated-docs/ -name "*.mdd" | shuf | head -3 | while read -r exploded_dir; do
       echo "Testing: $exploded_dir"
       temp_rebuilt="/tmp/$(basename "$exploded_dir" .mdd)-test.md"
       markitect md-implode "$exploded_dir" --output "$temp_rebuilt"
       echo "Rebuilt to: $temp_rebuilt"
   done
   ```

### Phase 4: Integration and Packaging (1-2 hours)

1. **Set up packaging workflow:**
   ```bash
   # Create packaging script
   cat > package-docs.sh << 'EOF'
   #!/bin/bash

   MIGRATED_DIR="migrated-docs"
   PACKAGES_DIR="packages"

   mkdir -p "$PACKAGES_DIR"

   # Package each exploded document
   find "$MIGRATED_DIR" -name "*.mdd" | while read -r exploded_dir; do
       basename_no_ext=$(basename "$exploded_dir" .mdd)

       # First implode to single document
       temp_doc="/tmp/${basename_no_ext}.md"
       markitect md-implode "$exploded_dir" --output "$temp_doc"

       # Then package
       output_package="$PACKAGES_DIR/${basename_no_ext}.mdz"
       markitect md-package create "$temp_doc" --format mdz --output "$output_package"

       echo "Packaged: $output_package"

       # Verify package
       markitect md-package info "$output_package"
   done
   EOF

   chmod +x package-docs.sh
   ./package-docs.sh
   ```

2. **Set up templating (optional):**
   ```bash
   # If you have template needs, create MDT templates
   mkdir templates/

   # Example template creation
   cat > templates/user-guide-template.mdt << 'EOF'
   # {{title}}

   **Version:** {{version}}
   **Author:** {{author}}
   **Updated:** {{date}}

   {{include "introduction.md"}}

   ## Features

   {{include "features.md"}}

   {{if include_advanced}}
   ## Advanced Topics

   {{include "advanced.md"}}
   {{endif}}
   EOF

   # Test template
   cat > templates/variables.json << 'EOF'
   {
     "title": "Migrated User Guide",
     "version": "2.0.0",
     "author": "Documentation Team",
     "date": "2025-10-14",
     "include_advanced": true
   }
   EOF

   markitect md-transclude validate templates/user-guide-template.mdt
   ```

## Validation and Testing

### Automated Validation

```bash
# Create validation script
cat > validate-migration.sh << 'EOF'
#!/bin/bash

echo "=== Migration Validation Report ==="

# Count original files
orig_count=$(find source-docs/ -name "*.md" | wc -l)
echo "Original documents: $orig_count"

# Count migrated structures
migrated_count=$(find migrated-docs/ -name "*.mdd" | wc -l)
echo "Migrated structures: $migrated_count"

# Count packages
package_count=$(find packages/ -name "*.mdz" | wc -l 2>/dev/null || echo "0")
echo "Created packages: $package_count"

# Test round-trip integrity
echo
echo "=== Round-trip Tests ==="
failed_tests=0

find migrated-docs/ -name "*.mdd" | head -5 | while read -r exploded_dir; do
    basename_no_ext=$(basename "$exploded_dir" .mdd)
    original_file="source-docs/${basename_no_ext}.md"

    if [ -f "$original_file" ]; then
        temp_rebuilt="/tmp/${basename_no_ext}-validation.md"
        markitect md-implode "$exploded_dir" --output "$temp_rebuilt"

        if diff -q "$original_file" "$temp_rebuilt" > /dev/null; then
            echo "✓ PASS: $basename_no_ext"
        else
            echo "✗ FAIL: $basename_no_ext"
            failed_tests=$((failed_tests + 1))
        fi
    fi
done

if [ $failed_tests -eq 0 ]; then
    echo
    echo "🎉 All validation tests passed!"
else
    echo
    echo "⚠ $failed_tests validation tests failed. Review differences manually."
fi
EOF

chmod +x validate-migration.sh
./validate-migration.sh
```

### Manual Validation Checklist

**Document Structure:**
- [ ] All original sections are preserved
- [ ] Heading hierarchy is maintained
- [ ] Code blocks are intact
- [ ] Links and references work correctly
- [ ] Images and assets are accessible

**Functionality:**
- [ ] Explode operations complete without errors
- [ ] Implode operations produce identical content
- [ ] Variant detection works correctly
- [ ] Packaging creates valid MDZ files
- [ ] Templates process correctly (if used)

**Integration:**
- [ ] CLI commands integrate with existing workflows
- [ ] Build processes adapt successfully
- [ ] Version control handles new file structures
- [ ] Team members can use new workflows

## Common Issues and Solutions

### Issue 1: "Failed to detect variant" Error

**Symptoms:**
```
Error: Failed to detect variant for directory 'docs/'
```

**Solutions:**
```bash
# Option 1: Force specific variant
markitect md-implode docs/ --force-variant flat

# Option 2: Create proper structure first
markitect md-explode combined-doc.md --variant hierarchical --create-manifest
```

### Issue 2: Large Document Memory Issues

**Symptoms:**
- Slow processing of large documents
- Out of memory errors

**Solutions:**
```bash
# Split large documents manually first
split -l 1000 large-document.md split-doc-

# Process parts separately
for part in split-doc-*; do
    markitect md-explode "$part" --variant flat
done

# Or use streaming approach (if available)
# markitect md-explode large-document.md --streaming --variant hierarchical
```

### Issue 3: Asset Path Issues

**Symptoms:**
- Images don't display after migration
- Broken asset references

**Solutions:**
```bash
# Check asset discovery
markitect md-package create document.md --dry-run --verbose

# Fix relative paths in markdown
sed -i 's|\.\./assets/|./assets/|g' document.md

# Verify asset packaging
markitect md-package info document.mdz --verbose
```

### Issue 4: Manifest Corruption

**Symptoms:**
```
Error: Invalid manifest format in 'manifest.md'
```

**Solutions:**
```bash
# Backup corrupted manifest
cp manifest.md manifest.md.backup

# Regenerate manifest
markitect md-explode original-document.md --variant hierarchical --create-manifest --force

# Or manually fix YAML frontmatter
```

### Issue 5: Template Processing Errors

**Symptoms:**
```
Error: Circular reference detected in template
```

**Solutions:**
```bash
# Validate template structure
markitect md-transclude validate template.mdt --verbose

# Check for circular includes
grep -r "{{include" templates/ | sort

# Fix circular references by restructuring includes
```

## Rollback Procedures

### Complete Rollback

```bash
# Restore from backup
backup_dir="backup-20251014-100000"  # Your backup timestamp

# Stop and restore
rm -rf migrated-docs/ packages/ templates/
cp -r "$backup_dir/"* ./

echo "Rollback completed. You're back to the original state."
```

### Selective Rollback

```bash
# Rollback specific documents only
problem_docs=("user-guide" "api-reference")

for doc in "${problem_docs[@]}"; do
    # Remove migrated version
    rm -rf "migrated-docs/${doc}.mdd"

    # Restore original
    cp "$backup_dir/${doc}.md" ./

    echo "Rolled back: $doc"
done
```

### Partial Migration Approach

If full migration is problematic, use a gradual approach:

```bash
# Phase 1: Migrate only small documents
find source-docs/ -name "*.md" -exec wc -l {} \; | awk '$1 < 200' | cut -d: -f2 | while read -r file; do
    markitect md-explode "$file" --variant flat --create-manifest
done

# Phase 2: Migrate medium documents (when comfortable)
# Phase 3: Migrate large documents (with assistance)
```

## Best Practices

### Migration Planning

1. **Start Small:** Begin with 1-2 documents to understand the process
2. **Test Variants:** Try different variants to find the best fit
3. **Validate Early:** Check round-trip integrity frequently
4. **Backup Everything:** Multiple backups at different stages
5. **Document Changes:** Keep migration notes for team members

### Workflow Integration

```bash
# Add to Makefile
migrate-docs:
	./migrate-all.sh

validate-migration:
	./validate-migration.sh

package-docs:
	./package-docs.sh

clean-migration:
	rm -rf migrated-docs/ packages/
	git checkout -- source-docs/
```

### Team Collaboration

1. **Training:** Ensure team understands new commands
2. **Documentation:** Update team docs with new workflows
3. **Gradual Adoption:** Allow parallel workflows during transition
4. **Support:** Designate migration champions to help others

### Maintenance

```bash
# Regular validation script
cat > .github/workflows/validate-docs.yml << 'EOF'
name: Validate Documentation Structure

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Install MarkiTect
      run: pip install markitect
    - name: Validate structures
      run: |
        find . -name "*.mdd" | while read -r dir; do
          echo "Validating: $dir"
          markitect md-implode "$dir" --dry-run --verbose
        done
EOF
```

---

## Migration Checklist

### Pre-Migration
- [ ] Backup all documentation
- [ ] Install latest MarkiTect version
- [ ] Assess current documentation structure
- [ ] Choose migration strategy
- [ ] Test with pilot documents

### During Migration
- [ ] Process documents systematically
- [ ] Validate each step
- [ ] Document any issues encountered
- [ ] Test package creation
- [ ] Verify asset handling

### Post-Migration
- [ ] Update team documentation
- [ ] Train team members on new workflows
- [ ] Update CI/CD processes
- [ ] Establish maintenance procedures
- [ ] Plan rollback procedures (just in case)

**Migration Support:** For complex migrations or issues not covered in this guide, create detailed issue reports with your specific setup and requirements.

---

**Version:** 1.0.0
**Last Updated:** 2025-10-14
**Compatibility:** MarkiTect 1.0+