markitect-main/docs/user-guides/cache-management.md

Cache Management Guide

MarkiTect's caching system provides significant performance improvements by storing parsed AST representations of your markdown files. This guide explains how to monitor, maintain, and optimize your cache usage.

Overview

The cache system automatically manages performance optimization, but provides CLI tools for monitoring and maintenance when needed.

Cache Commands

markitect cache-info

Display detailed information about your current cache status.

markitect cache-info

Example Output:

Cache Directory: /home/user/project/.ast_cache
Total Files: 42
Cache Size: 2.1 MB

What it shows:

• Cache Directory: Where cache files are stored
• Total Files: Number of documents currently cached
• Cache Size: Total disk space used by cache

markitect cache-clean

Remove all cached files to free disk space or force fresh parsing.

markitect cache-clean

Example Output:

Cache cleaned successfully - removed 42 file(s).

When to use:

• Free up disk space
• Force fresh parsing of all documents
• Clear potentially corrupted cache
• Development debugging

markitect cache-invalidate <file>

Remove cache for a specific file, forcing it to be re-parsed next time.

markitect cache-invalidate docs/architecture.md

Example Output:

Cache invalidated for architecture.md.

When to use:

• File was modified outside of MarkiTect
• Testing parsing behavior
• Troubleshooting specific document issues

Understanding Cache Behavior

Automatic Cache Management

The cache system handles most operations automatically:

1. First Access: File is parsed and cached
2. Subsequent Access: Cache is loaded (60-85% faster)
3. File Modification: Cache is automatically invalidated
4. Next Access: File is re-parsed and re-cached

Cache Directory Structure

your-project/
├── docs/
│   ├── guide.md              # Your source files
│   └── api.md
├── .ast_cache/               # Auto-created cache directory
│   ├── guide.md.ast.json     # Cached AST for guide.md
│   └── api.md.ast.json       # Cached AST for api.md
└── .gitignore                # Should include .ast_cache/

Performance Optimization

Monitoring Cache Effectiveness

Use cache-info regularly to monitor cache usage:

# Check current cache status
markitect cache-info

# Process some files
markitect ingest docs/*.md
markitect query "SELECT COUNT(*) FROM markdown_files"

# Check cache growth
markitect cache-info

Cache Performance Characteristics

File Size	First Parse	Cached Load	Improvement
Small (< 1KB)	~10ms	~3ms	70%
Medium (1-10KB)	~50ms	~15ms	70%
Large (> 10KB)	~200ms	~25ms	85%

Best Practices

For Daily Usage

1. Let it work automatically - No manual intervention needed
2. Monitor disk usage - Use cache-info periodically
3. Clean when needed - Use cache-clean if disk space is limited

For Development

1. Add to .gitignore - Cache files shouldn't be version controlled
2. Clean during debugging - Use cache-invalidate for specific issues
3. Performance testing - Monitor cache effectiveness with cache-info

For Production

1. Plan disk space - Cache grows with content
2. Backup strategy - Source files matter, cache is regenerable
3. Monitoring - Include cache metrics in system monitoring

Troubleshooting

Common Issues

"Cache directory does not exist - nothing to clean"

• Normal when no files have been processed yet
• Cache directory is created automatically on first use

"No cache found for filename.md - nothing to invalidate"

• File hasn't been processed by MarkiTect yet
• Use markitect ingest filename.md first

Poor cache performance

• Check if files are being modified frequently
• Verify cache directory is on fast storage (SSD recommended)
• Monitor cache hit rates with repeated cache-info calls

Advanced Diagnostics

# Check if cache is being used effectively
markitect cache-info
markitect status docs/large-file.md  # Should be fast if cached
markitect cache-info  # File count should be same (cache hit)

# Force fresh parsing for comparison
markitect cache-invalidate docs/large-file.md
time markitect status docs/large-file.md  # Measure parse time
time markitect status docs/large-file.md  # Measure cache load time

Integration with Other Features

Database Queries

Cache improves performance of database operations that access document content:

markitect query "SELECT filename, title FROM markdown_files WHERE content LIKE '%architecture%'"

Batch Operations

Cache provides significant benefits for batch processing:

markitect ingest docs/*.md  # First run: parse + cache
markitect query "SELECT COUNT(*) FROM markdown_files"  # Subsequent: cache only

Technical Details

For detailed technical information about cache implementation, see:

• Architecture: Caching System
• Development: Performance Testing (coming soon)

---

The cache system is designed to be invisible during normal usage while providing powerful tools for monitoring and optimization when needed.