markitect-main/CAPABILITY_INCLUSION_GUIDE.md

# Capability Inclusion Guide

> **Complete guide to understanding and managing capability inclusion in the MarkiTect project**

## Overview

MarkiTect uses a **Capability Inclusion Pattern** where functionality is organized into distinct capabilities that can be:
- **Internal**: Provided by this repository (core MarkiTect functionality)
- **External**: Used by this repository (submodules, dependencies, extracted capabilities)

This approach enables clear separation of concerns, easy extension/bugfixing, and prevents code duplication.

---

## 📋 **Documentation Structure**

### Core Documentation Files

1. **`CAPABILITIES.md`** - **Internal Capability Inventory**
   - **Purpose**: Comprehensive analysis of all capabilities provided BY this repository
   - **Content**: 73+ internal capabilities, test coverage, extraction recommendations
   - **Scope**: What MarkiTect provides to the world

2. **`CAPABILITY_REGISTRY.md`** - **External Capability Registry**
   - **Purpose**: Registry of all capabilities USED BY this repository
   - **Content**: Submodules, local extracted capabilities, external dependencies
   - **Scope**: What MarkiTect consumes from other sources

3. **`CLAUDE_CAPABILITY_REFERENCE.md`** - **Quick Usage Reference**
   - **Purpose**: Prevent code duplication by guiding Claude to existing capabilities
   - **Content**: Quick lookup patterns and anti-duplication rules
   - **Scope**: Operational guidance for development

4. **`CAPABILITY_INCLUSION_GUIDE.md`** - **This Document**
   - **Purpose**: Explains the overall capability inclusion architecture
   - **Content**: Workflow, patterns, internal vs external organization
   - **Scope**: Architectural understanding and management

---

## 🏗️ **Capability Organization Architecture**

### Internal Capabilities (Provided BY MarkiTect)

**Location**: Throughout the main codebase
**Purpose**: Core functionality that MarkiTect provides to the world
**Management**: Documented in `CAPABILITIES.md`

#### Categories:
- **Core Processing**: AST-based markdown processing, database operations
- **CLI Commands**: Command-line interface functionality
- **Template Engine**: Document template processing
- **Caching System**: Multi-layer performance caching
- **Schema Validation**: Document structure validation
- **Export/Import**: Data transformation capabilities

#### Extraction Candidates:
- Capabilities that could be useful to other projects
- Self-contained functionality with clear boundaries
- Well-tested components (>80% coverage preferred)

**Example Internal Capability:**
```
markitect/database/       # Database operations capability
markitect/template/       # Template processing capability
markitect/cli/           # CLI framework capability
```

### External Capabilities (Used BY MarkiTect)

**Location**: Various inclusion patterns
**Purpose**: Functionality MarkiTect depends on from external sources
**Management**: Documented in `CAPABILITY_REGISTRY.md`

#### 1. **Submodule Capabilities** (Independent Repositories)
- **Pattern**: Git submodules pointing to external repositories
- **Benefits**: Independent versioning, separate development, easy updates
- **Examples**: `capabilities/issue-facade/`, `wiki/`

#### 2. **Local Extracted Capabilities** (Previously Internal, Now Separated)
- **Pattern**: Moved to `capabilities/` directory but still in this repo
- **Benefits**: Clear separation, preparation for future extraction
- **Examples**: `capabilities/markitect-content/`, `capabilities/markitect-utils/`

#### 3. **Package Dependencies** (Third-Party Libraries)
- **Pattern**: Standard pip dependencies in `pyproject.toml`
- **Benefits**: Mature, maintained, standard integration
- **Examples**: `click`, `pytest`, `sqlalchemy`

---

## 🔄 **Capability Inclusion Workflow**

### Phase 1: Internal Development
```
Developer creates functionality → Internal capability (in main codebase)
```

### Phase 2: Extraction Evaluation
```
Capability matures → Evaluate extraction criteria → Decide extraction pattern
```

### Phase 3: Capability Inclusion
```
Extract capability → Choose inclusion pattern → Update registries
```

### Inclusion Pattern Decision Tree:

1. **Will other projects use this capability?**
   - **Yes** → Consider **Submodule Capability** (extract to separate repo)
   - **No** → Consider **Local Capability** (move to `capabilities/`)

2. **Does it need independent versioning/development?**
   - **Yes** → **Submodule Capability**
   - **No** → **Local Capability**

3. **Is it a mature third-party solution?**
   - **Yes** → **Package Dependency**
   - **No** → Custom solution needed

### Example Extraction Journey:
```
Internal → markitect/issues/ (internal issue management)
Evaluation → Self-contained, reusable, independent development needed
Extraction → coulomb/issue-facade (separate repository)
Inclusion → capabilities/issue-facade/ (submodule capability)
Registration → CAPABILITY_REGISTRY.md updated
```

---

## 📊 **Current Capability Landscape**

### Internal Capabilities (73+ documented in CAPABILITIES.md)
```
markitect/                    # Core repository
├── database/                 # Database operations
├── template/                 # Template processing
├── cli/                      # CLI framework
├── packaging/                # Document packaging
├── finance/                  # Cost tracking
└── [... 68+ more capabilities]
```

### External Capabilities (5 documented in CAPABILITY_REGISTRY.md)
```
capabilities/
├── issue-facade/            # Submodule: Universal issue tracking
├── kaizen-agentic/          # Submodule: AI agent framework
├── markitect-content/       # Local: Content processing
└── markitect-utils/         # Local: Utility functions
wiki/                        # Submodule: Documentation
[External dependencies: click, pytest, sqlalchemy, ...]
```

---

## 🛠️ **Management Operations**

### Discovery and Validation
```bash
# Discover all external capabilities
make capability-report

# Search for existing functionality
make capability-search TERM=issue_management

# Validate proper usage
make capability-validate FILE=my_code.py
```

### Adding New External Capabilities

#### Submodule Capability:
```bash
git submodule add <repo-url> <local-path>
# Update CAPABILITY_REGISTRY.md
```

#### Local Capability:
```bash
mkdir capabilities/new-capability
# Move code, create README.md
# Update CAPABILITY_REGISTRY.md
```

#### Package Dependency:
```bash
# Update pyproject.toml
# Update CAPABILITY_REGISTRY.md
```

### Updating Capabilities

#### Submodules:
```bash
git submodule update --remote <submodule-path>
```

#### Local Capabilities:
```bash
# Direct code updates in capabilities/
```

#### Package Dependencies:
```bash
pip install --upgrade <package>
# Update pyproject.toml version constraints
```

---

## 🎯 **Best Practices**

### For Internal Capabilities (CAPABILITIES.md):
- **Document thoroughly**: Clear description, interfaces, test coverage
- **Evaluate extraction**: Regular review against extraction criteria
- **Maintain quality**: Adequate test coverage, clear boundaries
- **Consider reusability**: Could other projects benefit from this?

### For External Capabilities (CAPABILITY_REGISTRY.md):
- **Registry first**: Always check before implementing new functionality
- **Respect interfaces**: Use documented APIs, don't bypass capabilities
- **Update documentation**: Keep registry current with capability changes
- **Clear boundaries**: Don't duplicate external capability functionality

### For Claude and Developers:
- **Check before code**: Always consult `CAPABILITY_REGISTRY.md` first
- **Use discovery tools**: `make capability-search` before implementing
- **Follow patterns**: Use established integration patterns
- **Update registries**: Document new capabilities immediately

---

## 🔮 **Future Evolution**

### Extraction Pipeline:
```
Internal Capability → Evaluation → Local Capability → Submodule Capability
```

### Maturity Progression:
1. **Internal**: New functionality developed in main codebase
2. **Local**: Stable functionality moved to `capabilities/` for separation
3. **Submodule**: Mature functionality extracted to independent repository
4. **Package**: Published capabilities available via pip/pypi

### Success Metrics:
- **Zero duplication**: No accidental reimplementation of existing capabilities
- **Clear boundaries**: Well-defined interfaces between internal and external
- **Easy extension**: Simple to enhance or fix external capabilities
- **Efficient discovery**: Fast identification of existing functionality

---

## 📚 **Quick Reference**

| Need | Check | Use |
|------|--------|-----|
| Internal MarkiTect functionality | `CAPABILITIES.md` | Import from main codebase |
| External functionality | `CAPABILITY_REGISTRY.md` | Use documented interface |
| Prevent duplication | `CLAUDE_CAPABILITY_REFERENCE.md` | Follow anti-duplication rules |
| Understand architecture | `CAPABILITY_INCLUSION_GUIDE.md` | This document |

**Remember**: Internal capabilities are what MarkiTect **provides**, external capabilities are what MarkiTect **uses**.