MARKITECT(1)

NAME
markitect - high-performance markdown processing engine with AST caching

SYNOPSIS
markitect [OPTION]... [COMMAND] [FILE]...

DESCRIPTION
MarkiTect is a high-performance markdown processing engine that implements a "parse-once, manipulate-many" architecture with intelligent AST caching and database-first metadata management. The core innovation is that markdown files are parsed once and stored in multiple fast-access representations: JSON-serialized AST cache files, structured database metadata, and original content preservation. This enables complex document workflows without performance penalties.

COMMANDS

  Document Processing

ingest FILE
    Ingest a markdown file into the MarkiTect system. Creates AST cache and stores metadata in database. Performance: Initial parse creates overhead, but subsequent cache loads are < 50% of parse time.

ingest-batch DIRECTORY
    Batch process all markdown files in a directory. Supports recursive processing with --recursive option.

status FILE
    Show processing status and cache information for a file. Displays parse time, cache time, and cache validity.

  Cache Management

cache-info FILE
    Display detailed cache information including performance metrics. Shows cache hit/miss ratio and loading time statistics.

cache-invalidate FILE
    Force invalidation of AST cache for a file. Useful when manual cache refresh is needed.

cache-clean
    Remove all stale cache files based on source file modification times. Performs automatic cache maintenance.

  Database Operations

query SQL
    Execute SQL query against document metadata database. Enables relational operations on front matter data.

list
    List all ingested documents with metadata summary. Shows filename, title, modification time, and cache status.

show FILE
    Display complete metadata for a specific document. Includes front matter, processing times, and cache information.

export FORMAT
    Export document metadata in specified format (json, csv, yaml). Supports filtered exports with --filter option.

  AST Operations

ast-dump FILE
    Output AST representation of a markdown file. Useful for debugging and analysis. Uses cached AST if available.

ast-query FILE QUERY
    Query AST structure using JSONPath expressions. Examples: $..[?@.type=='heading_open'], $..[?@.level==1]

ast-transform FILE SCRIPT
    Apply transformation script to AST structure. Supports custom Python scripts for content modification.

  Performance Analysis

benchmark FILE
    Run performance benchmark comparing parse vs cache load times. Validates the < 50% cache loading performance requirement.

profile DIRECTORY
    Generate performance profile for a collection of documents. Identifies performance bottlenecks and optimization opportunities.

OPTIONS

  Global Options

--cache-dir DIRECTORY
    Specify custom cache directory (default: .ast_cache)

--database FILE
    Specify database file path (default: markitect.db)

--verbose, -v
    Enable verbose output with performance timing details

--quiet, -q
    Suppress non-essential output

--config FILE
    Use custom configuration file

  Processing Options

--recursive, -r
    Process directories recursively

--force, -f
    Force reprocessing even if cache is valid

--validate
    Validate performance requirements during processing

--no-cache
    Disable AST caching (parse every time)

  Output Options

--format FORMAT
    Output format: json, yaml, csv, table (default: table)

--output FILE
    Write output to file instead of stdout

--filter EXPRESSION
    Filter results using JSONPath expression

PERFORMANCE GUARANTEES
MarkiTect provides documented performance contracts:

Cache Loading Time
    AST cache loading guaranteed to be < 50% of original markdown parsing time. This is validated by automated tests and can be verified with markitect benchmark.

Database Queries
    Metadata queries typically complete in sub-millisecond time for collections up to 10,000 documents.

Memory Usage
    Constant memory usage for cache operations regardless of document size. Memory scaling is linear with the number of documents processed simultaneously.

CONFIGURATION
MarkiTect can be configured through:

Configuration File
    ~/.markitect/config.yaml or specified with --config option

Environment Variables
    MARKITECT_CACHE_DIR - Default cache directory
    MARKITECT_DATABASE - Default database file
    MARKITECT_VALIDATE_PERFORMANCE - Enable automatic performance validation

ARCHITECTURE

Parse-Once, Manipulate-Many
    Source files are parsed once to create multiple fast-access representations:
    - AST Cache: JSON-serialized Abstract Syntax Tree
    - Database Metadata: Structured front matter and document metadata
    - Original Content: Preserved for integrity validation

Intelligent Cache Invalidation
    Cache files are automatically invalidated based on source file modification times. No manual cache management required.

Database-First Metadata
    Front matter becomes queryable relational data with full SQL capabilities. Supports joins, aggregations, and complex filtering operations.

EXAMPLES

Basic Document Processing
    # Ingest a single markdown file
    markitect ingest document.md
    # Process all markdown files in a directory
    markitect ingest-batch docs/ --recursive
    # Show processing status
    markitect status document.md

Cache Operations
    # Display cache information
    markitect cache-info document.md
    # Clean stale cache files
    markitect cache-clean
    # Force cache regeneration
    markitect cache-invalidate document.md --force

Database Queries
    # List all documents
    markitect list
    # Query by metadata
    markitect query "SELECT * FROM markdown_files WHERE json_extract(front_matter, '$.author') = 'John Doe'"
    # Export metadata
    markitect export json --output metadata.json

AST Analysis
    # Dump AST structure
    markitect ast-dump document.md --format json
    # Query for all headings
    markitect ast-query document.md "$..[?@.type=='heading_open']"
    # Find level 1 headings
    markitect ast-query document.md "$..[?@.level==1]"

Performance Analysis
    # Benchmark a single file
    markitect benchmark document.md
    # Profile a document collection
    markitect profile docs/ --recursive
    # Validate performance requirements
    markitect ingest document.md --validate

EXIT STATUS

0   Success

1   General error (file not found, permission denied, etc.)

2   Performance requirement violation (cache loading >= 50% of parse time)

3   Database error (corruption, schema mismatch, etc.)

4   Cache error (corruption, permission denied, etc.)

FILES

~/.markitect/config.yaml
    User configuration file

.ast_cache/
    Default AST cache directory

markitect.db
    Default SQLite database file

.markitect_workspace/
    Workspace directory for development workflows

DIAGNOSTICS
Common diagnostic commands:

Performance Issues
    markitect benchmark problematic_file.md
    markitect profile slow_directory/ --verbose

Cache Problems
    markitect cache-info file.md
    markitect cache-clean --verbose

Database Issues
    markitect query "PRAGMA integrity_check"
    markitect list --validate

BUGS
Report bugs to: https://github.com/project/markitect/issues

SEE ALSO
markdown(1), sqlite3(1), jq(1)

AUTHORS
MarkiTect development team

COPYRIGHT
Copyright (C) 2025 MarkiTect Project. This is free software; see the source for copying conditions.