feat: complete Issue #151 - Phase 4: Integration and Documentation

Implements comprehensive CLI integration and documentation for the explode-implode system, completing both Issues #147 and #151. Key Features Added: - md-package CLI command (create/extract/info actions) - md-transclude CLI command (process/validate actions) - Complete user guide (556 lines) with tutorials and examples - Technical API documentation (500 lines) for developers - Migration guide (761 lines) with step-by-step procedures - Cost analysis documenting ~85 hours of development value Technical Implementation: - Full MDZ packaging support with asset embedding - Template-based transclusion with variable substitution - Comprehensive error handling and verbose output modes - Integration with existing MarkiTect CLI architecture Documentation Suite: - docs/user-guides/explode-implode-complete-guide.md - docs/api/explode-variants.md - docs/user-guides/migration-guide.md - docs/cost-analysis/issues-147-151-implementation.md This implementation transforms MarkiTect from a simple markdown processor into a comprehensive document management platform with sophisticated organizational capabilities. Closes #147: Directory organization preservation fully implemented Closes #151: CLI integration and documentation completed 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-14 11:11:51 +02:00
parent e8e0fbaec3
commit 6ddd4ea6e3
7 changed files with 2389 additions and 1 deletions
--- a/docs/user-guides/migration-guide.md
+++ b/docs/user-guides/migration-guide.md
@@ -0,0 +1,762 @@
+# Migration Guide: Upgrading to MarkiTect's Enhanced Explode-Implode System
+
+**Step-by-step guide for migrating existing documentation workflows to MarkiTect's advanced explode-implode and packaging systems**
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Pre-Migration Assessment](#pre-migration-assessment)
+3. [Migration Scenarios](#migration-scenarios)
+4. [Step-by-Step Migration](#step-by-step-migration)
+5. [Validation and Testing](#validation-and-testing)
+6. [Common Issues and Solutions](#common-issues-and-solutions)
+7. [Rollback Procedures](#rollback-procedures)
+8. [Best Practices](#best-practices)
+
+---
+
+## Overview
+
+This guide helps you migrate from:
+- **Basic MarkiTect installations** to the enhanced explode-implode system
+- **Manual directory organization** to structured variant-based workflows
+- **Legacy documentation systems** to MarkiTect's integrated approach
+- **Simple file-based workflows** to advanced packaging and templating
+
+### What's New
+
+**Enhanced Features Available After Migration:**
+- 🔄 **Three organizational variants** (flat, hierarchical, semantic)
+- 📦 **MDZ packaging** with asset management
+- 🔗 **Template-based transclusion** (MDT format)
+- 🤖 **Auto-detection** of exploded structures
+- 📊 **Manifest-based reversibility**
+- ⚡ **CLI command integration**
+
+### Compatibility
+
+**Supported Migration Paths:**
+- MarkiTect 0.x → 1.0+ (full migration required)
+- Manual documentation workflows → MarkiTect structured workflows
+- GitBook, MkDocs, Sphinx → MarkiTect (with content adaptation)
+- Simple markdown files → Advanced explode-implode workflows
+
+## Pre-Migration Assessment
+
+### Step 1: Inventory Current Setup
+
+Run this assessment to understand your current state:
+
+```bash
+# Check your current MarkiTect version
+markitect version
+
+# List your current documentation structure
+find . -name "*.md" -type f | head -20
+
+# Check for existing MarkiTect files
+find . -name "*.mdd" -o -name "manifest.md" -o -name "*.mdz"
+
+# Assess document sizes (identifies candidates for explosion)
+find . -name "*.md" -exec wc -l {} \; | sort -nr | head -10
+```
+
+### Step 2: Backup Current Setup
+
+**Critical: Always backup before migration**
+
+```bash
+# Create timestamped backup
+backup_dir="backup-$(date +%Y%m%d-%H%M%S)"
+mkdir "$backup_dir"
+
+# Backup all markdown and config files
+cp -r docs/ "$backup_dir/" 2>/dev/null || true
+cp -r *.md "$backup_dir/" 2>/dev/null || true
+cp -r .markitect* "$backup_dir/" 2>/dev/null || true
+
+echo "Backup created in: $backup_dir"
+```
+
+### Step 3: Assess Migration Complexity
+
+Use this checklist to determine your migration path:
+
+**Simple Migration (1-2 hours):**
+- [ ] Small documentation set (< 10 files)
+- [ ] Basic markdown structure
+- [ ] No custom build processes
+- [ ] Willing to use recommended variants
+
+**Moderate Migration (4-8 hours):**
+- [ ] Medium documentation set (10-50 files)
+- [ ] Some organizational structure exists
+- [ ] Basic build/CI processes
+- [ ] Custom asset management needs
+
+**Complex Migration (1-3 days):**
+- [ ] Large documentation set (50+ files)
+- [ ] Complex existing organization
+- [ ] Extensive build/CI integration
+- [ ] Custom tooling and workflows
+
+## Migration Scenarios
+
+### Scenario A: From Basic MarkiTect
+
+**Situation:** You have MarkiTect installed but haven't used explode-implode features.
+
+**Migration Steps:**
+
+1. **Update MarkiTect:**
+   ```bash
+   pip install --upgrade markitect
+   markitect version  # Verify 1.0+
+   ```
+
+2. **Test new commands:**
+   ```bash
+   # Verify new commands are available
+   markitect md-explode --help
+   markitect md-package --help
+   markitect md-transclude --help
+   ```
+
+3. **Choose your first document:**
+   ```bash
+   # Start with a medium-sized document
+   markitect md-explode your-document.md --variant hierarchical --dry-run
+   ```
+
+4. **Perform first explosion:**
+   ```bash
+   markitect md-explode your-document.md --variant hierarchical --create-manifest
+   ```
+
+### Scenario B: From Manual Directory Organization
+
+**Situation:** You manually organize markdown files in directories.
+
+**Current Structure Example:**
+```
+docs/
+├── intro/
+│   ├── overview.md
+│   └── getting-started.md
+├── tutorials/
+│   ├── basic-usage.md
+│   └── advanced-features.md
+└── reference/
+    └── api.md
+```
+
+**Migration Steps:**
+
+1. **Assess current organization:**
+   ```bash
+   # Analyze your structure
+   tree docs/ > current-structure.txt
+
+   # Count sections across files
+   grep -r "^#" docs/ | wc -l
+   ```
+
+2. **Choose migration strategy:**
+
+   **Option A: Convert to Semantic Variant**
+   ```bash
+   # Combines files into semantic structure
+   markitect md-implode docs/ --force-variant semantic --output combined-docs.md
+
+   # Then re-explode with proper manifest
+   markitect md-explode combined-docs.md --variant semantic --create-manifest
+   ```
+
+   **Option B: Preserve as Flat Structure**
+   ```bash
+   # Convert to flat variant maintaining file boundaries
+   markitect md-implode docs/ --force-variant flat --output combined-docs.md
+   markitect md-explode combined-docs.md --variant flat --create-manifest
+   ```
+
+3. **Validate result:**
+   ```bash
+   # Check the new structure
+   tree combined-docs.mdd/
+
+   # Test round-trip
+   markitect md-implode combined-docs.mdd/ --output test-rebuild.md
+   diff combined-docs.md test-rebuild.md
+   ```
+
+### Scenario C: From GitBook
+
+**Situation:** Migrating from GitBook to MarkiTect.
+
+**GitBook Structure:**
+```
+docs/
+├── SUMMARY.md
+├── README.md
+├── chapter1/
+│   ├── README.md
+│   └── section1.md
+└── chapter2/
+    └── README.md
+```
+
+**Migration Steps:**
+
+1. **Convert GitBook structure:**
+   ```bash
+   # Create conversion script
+   cat > convert-gitbook.py << 'EOF'
+   #!/usr/bin/env python3
+   import os
+   from pathlib import Path
+
+   def convert_gitbook_to_single_md():
+       # Parse SUMMARY.md to understand structure
+       # Combine files in order
+       # Generate single markdown file
+       pass
+
+   if __name__ == "__main__":
+       convert_gitbook_to_single_md()
+   EOF
+
+   python convert-gitbook.py
+   ```
+
+2. **Apply MarkiTect structure:**
+   ```bash
+   markitect md-explode converted-document.md --variant hierarchical --create-manifest
+   ```
+
+### Scenario D: From Legacy Documentation Systems
+
+**Situation:** Migrating from Sphinx, MkDocs, or similar systems.
+
+**Migration Approach:**
+
+1. **Export to markdown:**
+   ```bash
+   # For Sphinx (using pandoc)
+   find . -name "*.rst" -exec pandoc {} -f rst -t markdown -o {}.md \;
+
+   # For MkDocs, files may already be markdown
+   # Check mkdocs.yml for structure information
+   ```
+
+2. **Combine and structure:**
+   ```bash
+   # Create combined document preserving hierarchy
+   # (You may need custom script based on your system)
+
+   # Then apply MarkiTect organization
+   markitect md-explode legacy-combined.md --variant hierarchical
+   ```
+
+## Step-by-Step Migration
+
+### Phase 1: Preparation (30 minutes)
+
+1. **Environment Setup:**
+   ```bash
+   # Ensure latest version
+   pip install --upgrade markitect
+
+   # Verify installation
+   markitect version
+   markitect md-explode --help
+   markitect md-package --help
+   ```
+
+2. **Create working directory:**
+   ```bash
+   mkdir markitect-migration
+   cd markitect-migration
+
+   # Copy source documents
+   cp -r /path/to/current/docs ./source-docs/
+   ```
+
+3. **Assess document complexity:**
+   ```bash
+   # Find largest documents (good candidates for explosion)
+   find source-docs/ -name "*.md" -exec wc -l {} \; | sort -nr > document-sizes.txt
+
+   # Identify documents with many sections
+   find source-docs/ -name "*.md" -exec bash -c 'echo -n "{}: "; grep -c "^#" "{}"' \; > section-counts.txt
+   ```
+
+### Phase 2: Small-Scale Testing (1 hour)
+
+1. **Choose pilot document:**
+   ```bash
+   # Select medium-sized document from analysis
+   pilot_doc="source-docs/user-guide.md"  # Replace with your choice
+   ```
+
+2. **Test explosion variants:**
+   ```bash
+   # Test all variants with dry-run
+   markitect md-explode "$pilot_doc" --variant flat --dry-run --verbose
+   markitect md-explode "$pilot_doc" --variant hierarchical --dry-run --verbose
+   markitect md-explode "$pilot_doc" --variant semantic --dry-run --verbose
+   ```
+
+3. **Perform pilot explosion:**
+   ```bash
+   # Choose best variant and execute
+   markitect md-explode "$pilot_doc" --variant hierarchical --create-manifest --output pilot-exploded/
+
+   # Validate round-trip
+   markitect md-implode pilot-exploded/ --output pilot-rebuilt.md
+   diff "$pilot_doc" pilot-rebuilt.md
+   ```
+
+### Phase 3: Full Migration (2-4 hours)
+
+1. **Process all documents:**
+   ```bash
+   # Create batch processing script
+   cat > migrate-all.sh << 'EOF'
+   #!/bin/bash
+
+   # Configure migration settings
+   VARIANT="hierarchical"  # Change as needed
+   SOURCE_DIR="source-docs"
+   OUTPUT_DIR="migrated-docs"
+
+   mkdir -p "$OUTPUT_DIR"
+
+   # Process each markdown file
+   find "$SOURCE_DIR" -name "*.md" | while read -r file; do
+       echo "Processing: $file"
+
+       # Generate output path
+       relative_path=$(realpath --relative-to="$SOURCE_DIR" "$file")
+       basename_no_ext=$(basename "$file" .md)
+       output_subdir="$OUTPUT_DIR/${basename_no_ext}.mdd"
+
+       # Explode document
+       markitect md-explode "$file" --variant "$VARIANT" --create-manifest --output "$output_subdir"
+
+       # Validate
+       temp_rebuilt="/tmp/${basename_no_ext}-rebuilt.md"
+       markitect md-implode "$output_subdir" --output "$temp_rebuilt"
+
+       if diff -q "$file" "$temp_rebuilt" > /dev/null; then
+           echo "✓ Migration successful: $file"
+       else
+           echo "⚠ Migration issues detected: $file"
+           echo "   Check: $temp_rebuilt vs $file"
+       fi
+   done
+   EOF
+
+   chmod +x migrate-all.sh
+   ./migrate-all.sh
+   ```
+
+2. **Validate migration results:**
+   ```bash
+   # Check all migrations
+   find migrated-docs/ -name "manifest.md" | wc -l  # Should equal input document count
+
+   # Test random sampling
+   find migrated-docs/ -name "*.mdd" | shuf | head -3 | while read -r exploded_dir; do
+       echo "Testing: $exploded_dir"
+       temp_rebuilt="/tmp/$(basename "$exploded_dir" .mdd)-test.md"
+       markitect md-implode "$exploded_dir" --output "$temp_rebuilt"
+       echo "Rebuilt to: $temp_rebuilt"
+   done
+   ```
+
+### Phase 4: Integration and Packaging (1-2 hours)
+
+1. **Set up packaging workflow:**
+   ```bash
+   # Create packaging script
+   cat > package-docs.sh << 'EOF'
+   #!/bin/bash
+
+   MIGRATED_DIR="migrated-docs"
+   PACKAGES_DIR="packages"
+
+   mkdir -p "$PACKAGES_DIR"
+
+   # Package each exploded document
+   find "$MIGRATED_DIR" -name "*.mdd" | while read -r exploded_dir; do
+       basename_no_ext=$(basename "$exploded_dir" .mdd)
+
+       # First implode to single document
+       temp_doc="/tmp/${basename_no_ext}.md"
+       markitect md-implode "$exploded_dir" --output "$temp_doc"
+
+       # Then package
+       output_package="$PACKAGES_DIR/${basename_no_ext}.mdz"
+       markitect md-package create "$temp_doc" --format mdz --output "$output_package"
+
+       echo "Packaged: $output_package"
+
+       # Verify package
+       markitect md-package info "$output_package"
+   done
+   EOF
+
+   chmod +x package-docs.sh
+   ./package-docs.sh
+   ```
+
+2. **Set up templating (optional):**
+   ```bash
+   # If you have template needs, create MDT templates
+   mkdir templates/
+
+   # Example template creation
+   cat > templates/user-guide-template.mdt << 'EOF'
+   # {{title}}
+
+   **Version:** {{version}}
+   **Author:** {{author}}
+   **Updated:** {{date}}
+
+   {{include "introduction.md"}}
+
+   ## Features
+
+   {{include "features.md"}}
+
+   {{if include_advanced}}
+   ## Advanced Topics
+
+   {{include "advanced.md"}}
+   {{endif}}
+   EOF
+
+   # Test template
+   cat > templates/variables.json << 'EOF'
+   {
+     "title": "Migrated User Guide",
+     "version": "2.0.0",
+     "author": "Documentation Team",
+     "date": "2025-10-14",
+     "include_advanced": true
+   }
+   EOF
+
+   markitect md-transclude validate templates/user-guide-template.mdt
+   ```
+
+## Validation and Testing
+
+### Automated Validation
+
+```bash
+# Create validation script
+cat > validate-migration.sh << 'EOF'
+#!/bin/bash
+
+echo "=== Migration Validation Report ==="
+
+# Count original files
+orig_count=$(find source-docs/ -name "*.md" | wc -l)
+echo "Original documents: $orig_count"
+
+# Count migrated structures
+migrated_count=$(find migrated-docs/ -name "*.mdd" | wc -l)
+echo "Migrated structures: $migrated_count"
+
+# Count packages
+package_count=$(find packages/ -name "*.mdz" | wc -l 2>/dev/null || echo "0")
+echo "Created packages: $package_count"
+
+# Test round-trip integrity
+echo
+echo "=== Round-trip Tests ==="
+failed_tests=0
+
+find migrated-docs/ -name "*.mdd" | head -5 | while read -r exploded_dir; do
+    basename_no_ext=$(basename "$exploded_dir" .mdd)
+    original_file="source-docs/${basename_no_ext}.md"
+
+    if [ -f "$original_file" ]; then
+        temp_rebuilt="/tmp/${basename_no_ext}-validation.md"
+        markitect md-implode "$exploded_dir" --output "$temp_rebuilt"
+
+        if diff -q "$original_file" "$temp_rebuilt" > /dev/null; then
+            echo "✓ PASS: $basename_no_ext"
+        else
+            echo "✗ FAIL: $basename_no_ext"
+            failed_tests=$((failed_tests + 1))
+        fi
+    fi
+done
+
+if [ $failed_tests -eq 0 ]; then
+    echo
+    echo "🎉 All validation tests passed!"
+else
+    echo
+    echo "⚠ $failed_tests validation tests failed. Review differences manually."
+fi
+EOF
+
+chmod +x validate-migration.sh
+./validate-migration.sh
+```
+
+### Manual Validation Checklist
+
+**Document Structure:**
+- [ ] All original sections are preserved
+- [ ] Heading hierarchy is maintained
+- [ ] Code blocks are intact
+- [ ] Links and references work correctly
+- [ ] Images and assets are accessible
+
+**Functionality:**
+- [ ] Explode operations complete without errors
+- [ ] Implode operations produce identical content
+- [ ] Variant detection works correctly
+- [ ] Packaging creates valid MDZ files
+- [ ] Templates process correctly (if used)
+
+**Integration:**
+- [ ] CLI commands integrate with existing workflows
+- [ ] Build processes adapt successfully
+- [ ] Version control handles new file structures
+- [ ] Team members can use new workflows
+
+## Common Issues and Solutions
+
+### Issue 1: "Failed to detect variant" Error
+
+**Symptoms:**
+```
+Error: Failed to detect variant for directory 'docs/'
+```
+
+**Solutions:**
+```bash
+# Option 1: Force specific variant
+markitect md-implode docs/ --force-variant flat
+
+# Option 2: Create proper structure first
+markitect md-explode combined-doc.md --variant hierarchical --create-manifest
+```
+
+### Issue 2: Large Document Memory Issues
+
+**Symptoms:**
+- Slow processing of large documents
+- Out of memory errors
+
+**Solutions:**
+```bash
+# Split large documents manually first
+split -l 1000 large-document.md split-doc-
+
+# Process parts separately
+for part in split-doc-*; do
+    markitect md-explode "$part" --variant flat
+done
+
+# Or use streaming approach (if available)
+# markitect md-explode large-document.md --streaming --variant hierarchical
+```
+
+### Issue 3: Asset Path Issues
+
+**Symptoms:**
+- Images don't display after migration
+- Broken asset references
+
+**Solutions:**
+```bash
+# Check asset discovery
+markitect md-package create document.md --dry-run --verbose
+
+# Fix relative paths in markdown
+sed -i 's|\.\./assets/|./assets/|g' document.md
+
+# Verify asset packaging
+markitect md-package info document.mdz --verbose
+```
+
+### Issue 4: Manifest Corruption
+
+**Symptoms:**
+```
+Error: Invalid manifest format in 'manifest.md'
+```
+
+**Solutions:**
+```bash
+# Backup corrupted manifest
+cp manifest.md manifest.md.backup
+
+# Regenerate manifest
+markitect md-explode original-document.md --variant hierarchical --create-manifest --force
+
+# Or manually fix YAML frontmatter
+```
+
+### Issue 5: Template Processing Errors
+
+**Symptoms:**
+```
+Error: Circular reference detected in template
+```
+
+**Solutions:**
+```bash
+# Validate template structure
+markitect md-transclude validate template.mdt --verbose
+
+# Check for circular includes
+grep -r "{{include" templates/ | sort
+
+# Fix circular references by restructuring includes
+```
+
+## Rollback Procedures
+
+### Complete Rollback
+
+```bash
+# Restore from backup
+backup_dir="backup-20251014-100000"  # Your backup timestamp
+
+# Stop and restore
+rm -rf migrated-docs/ packages/ templates/
+cp -r "$backup_dir/"* ./
+
+echo "Rollback completed. You're back to the original state."
+```
+
+### Selective Rollback
+
+```bash
+# Rollback specific documents only
+problem_docs=("user-guide" "api-reference")
+
+for doc in "${problem_docs[@]}"; do
+    # Remove migrated version
+    rm -rf "migrated-docs/${doc}.mdd"
+
+    # Restore original
+    cp "$backup_dir/${doc}.md" ./
+
+    echo "Rolled back: $doc"
+done
+```
+
+### Partial Migration Approach
+
+If full migration is problematic, use a gradual approach:
+
+```bash
+# Phase 1: Migrate only small documents
+find source-docs/ -name "*.md" -exec wc -l {} \; | awk '$1 < 200' | cut -d: -f2 | while read -r file; do
+    markitect md-explode "$file" --variant flat --create-manifest
+done
+
+# Phase 2: Migrate medium documents (when comfortable)
+# Phase 3: Migrate large documents (with assistance)
+```
+
+## Best Practices
+
+### Migration Planning
+
+1. **Start Small:** Begin with 1-2 documents to understand the process
+2. **Test Variants:** Try different variants to find the best fit
+3. **Validate Early:** Check round-trip integrity frequently
+4. **Backup Everything:** Multiple backups at different stages
+5. **Document Changes:** Keep migration notes for team members
+
+### Workflow Integration
+
+```bash
+# Add to Makefile
+migrate-docs:
+	./migrate-all.sh
+
+validate-migration:
+	./validate-migration.sh
+
+package-docs:
+	./package-docs.sh
+
+clean-migration:
+	rm -rf migrated-docs/ packages/
+	git checkout -- source-docs/
+```
+
+### Team Collaboration
+
+1. **Training:** Ensure team understands new commands
+2. **Documentation:** Update team docs with new workflows
+3. **Gradual Adoption:** Allow parallel workflows during transition
+4. **Support:** Designate migration champions to help others
+
+### Maintenance
+
+```bash
+# Regular validation script
+cat > .github/workflows/validate-docs.yml << 'EOF'
+name: Validate Documentation Structure
+
+on: [push, pull_request]
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Install MarkiTect
+      run: pip install markitect
+    - name: Validate structures
+      run: |
+        find . -name "*.mdd" | while read -r dir; do
+          echo "Validating: $dir"
+          markitect md-implode "$dir" --dry-run --verbose
+        done
+EOF
+```
+
+---
+
+## Migration Checklist
+
+### Pre-Migration
+- [ ] Backup all documentation
+- [ ] Install latest MarkiTect version
+- [ ] Assess current documentation structure
+- [ ] Choose migration strategy
+- [ ] Test with pilot documents
+
+### During Migration
+- [ ] Process documents systematically
+- [ ] Validate each step
+- [ ] Document any issues encountered
+- [ ] Test package creation
+- [ ] Verify asset handling
+
+### Post-Migration
+- [ ] Update team documentation
+- [ ] Train team members on new workflows
+- [ ] Update CI/CD processes
+- [ ] Establish maintenance procedures
+- [ ] Plan rollback procedures (just in case)
+
+**Migration Support:** For complex migrations or issues not covered in this guide, create detailed issue reports with your specific setup and requirements.
+
+---
+
+**Version:** 1.0.0
+**Last Updated:** 2025-10-14
+**Compatibility:** MarkiTect 1.0+