markitect-main/docs/CLI_TUTORIAL.md

# MarkiTect CLI Tutorial: Clever Command-Line Usage

## Table of Contents
1. [Getting Started](#getting-started)
2. [Core Workflow Patterns](#core-workflow-patterns)
3. [Document Processing](#document-processing)
4. [Template & Schema Workflows](#template--schema-workflows)
5. [Data Analysis & Querying](#data-analysis--querying)
6. [Advanced Techniques](#advanced-techniques)
7. [Business Document Automation](#business-document-automation)
8. [Troubleshooting & Optimization](#troubleshooting--optimization)

---

## Getting Started

### Installation & First Steps
```bash
# Check MarkiTect is properly installed
markitect --help

# View system statistics
markitect stats

# Check database status
markitect db-stats
```

### Essential Setup Commands
```bash
# Initialize workspace - process your first document
markitect ingest README.md

# List all processed files
markitect list

# Check specific file status
markitect stats README.md
```

---

## Core Workflow Patterns

### 1. Document Analysis Workflow
**Scenario**: Analyze and understand a markdown document structure

```bash
# Step 1: Ingest the document
markitect ingest document.md

# Step 2: View document metadata
markitect metadata document.md

# Step 3: Check frontmatter
markitect frontmatter-keys document.md
markitect frontmatter-get document.md title

# Step 4: Analyze AST structure
markitect ast-show document.md --format tree

# Step 5: Generate schema from structure
markitect schema-generate document.md --output document-schema.json
```

### 2. Content Extraction Workflow
**Scenario**: Extract specific content types from documents

```bash
# Extract pure content (no frontmatter/tailmatter)
markitect content-get document.md

# Get specific frontmatter values
markitect frontmatter-get document.md author
markitect frontmatter-get document.md config.theme  # nested values

# Extract contentmatter (MultiMarkdown key-value pairs)
markitect contentmatter-keys document.md
markitect contentmatter-get document.md project_id

# Check tailmatter (QA checklists, metadata)
markitect tailmatter-keys document.md
markitect tailmatter-get document.md qa.reviewed
```

### 3. Schema-Driven Development
**Scenario**: Use schemas to validate and generate documents

```bash
# Generate schema from example document
markitect schema-generate example.md --output project-schema.json

# Store schema in database
markitect schema-ingest project-schema.json

# Validate documents against schema
markitect validate document.md project-schema.json

# Generate stub from schema
markitect generate-stub project-schema.json --output new-document.md

# Generate multiple drafts
markitect generate-drafts project-schema.json data-source.json --output-dir ./drafts/
```

---

## Document Processing

### Batch Processing Techniques
```bash
# Process multiple files efficiently
for file in *.md; do
    markitect ingest "$file"
    echo "Processed: $file"
done

# Bulk validation
for file in docs/*.md; do
    markitect validate "$file" schema.json || echo "Validation failed: $file"
done

# Extract frontmatter from all files
markitect list --format json | jq -r '.[].filename' | while read file; do
    echo "=== $file ==="
    markitect frontmatter-keys "$file"
done
```

### Content Modification Workflows
```bash
# Add sections to existing documents
markitect modify document.md --add-section "New Section" --section-content "Content here"

# Update frontmatter programmatically
markitect frontmatter-set document.md last_updated="$(date)"
markitect frontmatter-set document.md version=2.1

# Set contentmatter values
markitect contentmatter-set document.md status=reviewed
markitect contentmatter-set document.md project.phase=complete
```

---

## Template & Schema Workflows

### Template-Driven Document Generation
**Scenario**: Generate business documents from templates

```bash
# Create invoice from template
markitect template-render invoice-template.md customer-data.json \
    --output "invoice-$(date +%Y%m%d).md" \
    --validate --check-data

# Generate report with YAML data
markitect template-render report-template.md quarterly-data.yaml \
    --format yaml --lenient --output quarterly-report.md

# Batch generate documents
for customer in customers/*.json; do
    customer_name=$(basename "$customer" .json)
    markitect template-render invoice-template.md "$customer" \
        --output "invoices/invoice-$customer_name.md"
done
```

### Schema Management
```bash
# List all stored schemas
markitect schema-list --format table

# Export schema for sharing
markitect schema-get project-schema --output exported-schema.json

# Update schema in database
markitect schema-delete old-schema
markitect schema-ingest updated-schema.json

# Validate schema compliance
markitect validate document.md schema-name --detailed-errors
```

---

## Data Analysis & Querying

### Database Queries
```bash
# View database schema
markitect db-schema

# Query processed files
markitect db-query "SELECT filename, processed_at FROM files WHERE processed_at > '2025-01-01'"

# Advanced frontmatter queries
markitect db-query "SELECT filename, frontmatter FROM files WHERE JSON_EXTRACT(frontmatter, '$.author') = 'John Doe'"

# Content statistics
markitect db-query "SELECT AVG(JSON_EXTRACT(metadata, '$.word_count')) as avg_words FROM files"
```

### AST Analysis
```bash
# Query AST structure with JSONPath
markitect ast-query document.md "$.children[?(@.type=='heading')].children[0].value"

# Find all links in document
markitect ast-query document.md "$..children[?(@.type=='link')].url"

# Extract code blocks
markitect ast-query document.md "$..children[?(@.type=='code')].value"

# Analyze heading structure
markitect ast-query document.md "$.children[?(@.type=='heading')].depth" --format json
```

### Statistical Analysis
```bash
# Document statistics
markitect content-stats document.md

# Frontmatter analysis across all files
markitect frontmatter-stats

# Contentmatter usage patterns
markitect contentmatter-stats

# System performance metrics
markitect cache-stats
markitect ast-stats
```

---

## Advanced Techniques

### Command Chaining & Pipelines
```bash
# Extract and process frontmatter
markitect frontmatter-get document.md title | tr '[:lower:]' '[:upper:]'

# Combine with standard tools
markitect list --format json | jq '.[] | select(.word_count > 1000) | .filename'

# Template generation pipeline
markitect schema-generate source.md | \
    markitect generate-stub --stdin | \
    markitect template-render --stdin data.json
```

### Conditional Processing
```bash
# Process only if file changed
if [ document.md -nt last-processed.timestamp ]; then
    markitect ingest document.md
    touch last-processed.timestamp
fi

# Validate before publishing
if markitect validate document.md schema.json --quiet; then
    echo "✅ Document valid - ready for publish"
    markitect template-render publish-template.md document-data.json
else
    echo "❌ Validation failed - fix errors first"
    markitect validate document.md schema.json --detailed-errors
fi
```

### Output Format Optimization
```bash
# Machine-readable output
markitect list --format json > files.json
markitect stats --format yaml > stats.yaml

# Human-readable reports
markitect list --format table --names-only
markitect db-stats --format simple

# Export for external tools
markitect db-query "SELECT * FROM files" --format json | jq '.[] | .filename'
```

---

## Business Document Automation

### Invoice Generation Workflow
```bash
# Setup: Create invoice template and customer database
# invoice-template.md contains {{customer.name}}, {{items}}, {{total}} etc.
# customers.json contains customer data array

# Generate monthly invoices
markitect template-render templates/invoice.md data/customer-001.json \
    --output "invoices/$(date +%Y-%m)/customer-001-invoice.md" \
    --validate --check-data

# Batch invoice generation
for customer in data/customers/*.json; do
    customer_id=$(basename "$customer" .json)
    markitect template-render templates/invoice.md "$customer" \
        --output "invoices/$(date +%Y-%m)/$customer_id-invoice.md" \
        --strict
done
```

### Report Generation Pipeline
```bash
# Generate quarterly business report
markitect template-render templates/quarterly-report.md data/q1-2025.yaml \
    --format yaml \
    --output "reports/Q1-2025-Business-Report.md" \
    --validate

# Validate report against company standards
markitect validate "reports/Q1-2025-Business-Report.md" schemas/report-schema.json

# Extract key metrics for dashboard
markitect frontmatter-get "reports/Q1-2025-Business-Report.md" metrics.revenue
markitect contentmatter-get "reports/Q1-2025-Business-Report.md" kpi.growth_rate
```

### Content Management Workflows
```bash
# Blog post publishing pipeline
markitect ingest drafts/new-post.md
markitect validate drafts/new-post.md schemas/blog-post.json
markitect frontmatter-set drafts/new-post.md published_date="$(date)"
markitect frontmatter-set drafts/new-post.md status=published

# Documentation maintenance
markitect schema-generate docs/api-reference.md --output schemas/api-doc.json
markitect generate-stub schemas/api-doc.json --output templates/api-template.md

# Quality assurance checks
markitect tailmatter-check document.md  # Run QA checklist
markitect validate document.md company-standards.json --detailed-errors
```

---

## Troubleshooting & Optimization

### Performance Optimization
```bash
# Check cache effectiveness
markitect cache-stats

# Clean cache if needed
markitect cache-clean

# Invalidate specific file cache
markitect cache-invalidate problematic-file.md

# Monitor database performance
markitect db-stats --format json | jq '.performance'
```

### Debugging Workflows
```bash
# Verbose output for debugging
markitect --verbose ingest document.md

# Check file processing status
markitect metadata document.md --format json | jq '.processing_errors'

# Validate template syntax
markitect template-render template.md data.json --validate

# Debug AST issues
markitect ast-show document.md --format json | jq '.errors'
```

### Database Maintenance
```bash
# Backup database
cp markitect.db markitect-backup-$(date +%Y%m%d).db

# Clean up orphaned records
markitect db-query "DELETE FROM files WHERE filename NOT IN (SELECT DISTINCT filename FROM current_files)"

# Optimize database
markitect db-query "VACUUM"

# Check database integrity
markitect db-query "PRAGMA integrity_check"
```

### Configuration Management
```bash
# Check configuration
markitect config-stats

# Use custom config file
markitect --config custom-config.yaml list

# Use different database
markitect --database project-specific.db ingest document.md
```

---

## Pro Tips & Best Practices

### 1. Workflow Automation
```bash
# Create alias for common operations
alias md-process='markitect ingest'
alias md-validate='markitect validate'
alias md-extract='markitect frontmatter-get'

# Setup environment variables
export MARKITECT_DB="/path/to/project.db"
export MARKITECT_CONFIG="/path/to/config.yaml"
```

### 2. Error Handling in Scripts
```bash
#!/bin/bash
# Robust document processing script

process_document() {
    local file="$1"

    # Check file exists
    if [[ ! -f "$file" ]]; then
        echo "Error: File $file not found" >&2
        return 1
    fi

    # Process with error handling
    if markitect ingest "$file"; then
        echo "✅ Processed: $file"

        # Validate if schema exists
        if [[ -f "schema.json" ]]; then
            if markitect validate "$file" schema.json --quiet; then
                echo "✅ Validated: $file"
            else
                echo "⚠️  Validation failed: $file" >&2
                markitect validate "$file" schema.json --detailed-errors >&2
            fi
        fi
    else
        echo "❌ Processing failed: $file" >&2
        return 1
    fi
}

# Process all markdown files
for file in *.md; do
    process_document "$file" || echo "Skipping $file due to errors"
done
```

### 3. Integration with Other Tools
```bash
# Combine with git hooks
# .git/hooks/pre-commit
markitect validate changed-docs/*.md schemas/doc-standard.json --quiet || {
    echo "Documentation validation failed"
    exit 1
}

# Integration with CI/CD
markitect list --format json | jq -r '.[] | select(.validation_status != "valid") | .filename' | while read file; do
    echo "::error file=$file::Document validation failed"
done

# Export for external analytics
markitect db-query "SELECT filename, JSON_EXTRACT(metadata, '$.word_count') as words FROM files" \
    --format json | jq -r '.[] | "\(.filename),\(.words)"' > document-metrics.csv
```

---

## Quick Reference

### Most Common Commands
```bash
# Basic document processing
markitect ingest document.md
markitect list
markitect stats document.md

# Content extraction
markitect frontmatter-get document.md key
markitect content-get document.md

# Template processing
markitect template-render template.md data.json

# Schema operations
markitect schema-generate document.md
markitect validate document.md schema.json

# Database queries
markitect db-query "SQL_QUERY"
markitect list --format json
```

### Output Formats
- `--format table` - Human-readable tables
- `--format json` - Machine-readable JSON
- `--format yaml` - YAML format
- `--format simple` - Plain text
- `--format compact` - Condensed output

### Global Options
- `--verbose` - Detailed output
- `--config CONFIG_FILE` - Custom configuration
- `--database DB_FILE` - Custom database
- `--help` - Command help

---

**🎯 Pro Tip**: Start with basic `ingest` and `list` commands, then gradually explore advanced features. Use `--help` on any command to see all available options!

**📚 Remember**: MarkiTect is designed for powerful document automation - combine commands creatively to build sophisticated workflows that match your specific needs.