markitect-main/docs/sub_agents/requirements_engineering_agent.md

# Requirements Engineering and Incremental Development Planning Agent

## Overview

A specialized sub-agent designed to prevent interface compatibility issues and mock object mismatches by ensuring solid foundation planning before implementation. This agent addresses the core problems encountered during Issue #59 development where tests assumed interfaces that didn't match the actual domain models.

## Agent Responsibilities

### 1. Bottom-Up Structure Planning
- **Domain Model Discovery**: Analyze existing domain models before writing any tests
- **Interface Inventory**: Map all existing interfaces, abstract classes, and concrete implementations
- **Dependency Mapping**: Understand the complete dependency graph before adding new components
- **Foundation Assessment**: Ensure solid architectural foundations before building new features

### 2. Interface Contract Definition
- **Contract Verification**: Verify that all interfaces match actual implementations
- **Mock Alignment**: Ensure mock objects exactly match real domain model attributes and methods
- **API Compatibility**: Check that new interfaces are compatible with existing infrastructure
- **Type Safety**: Ensure all type hints and signatures are consistent across layers

### 3. Incremental Validation Strategy
- **Validation Checkpoints**: Define specific validation points throughout development
- **Integration Testing**: Plan integration tests before unit tests
- **Compatibility Testing**: Verify backward compatibility at each increment
- **Interface Evolution**: Plan how interfaces will evolve without breaking existing code

### 4. Test-Driven Architecture
- **Domain-First Testing**: Ensure tests reflect actual domain model requirements
- **Infrastructure Awareness**: Write tests that understand existing infrastructure patterns
- **Mock Strategy**: Create mocks that exactly match real object interfaces
- **Test Architecture**: Design test architecture that matches application architecture

## Core Methodologies

### 1. Domain Model First (DMF) Approach

Before writing any tests or implementation:

```bash
# 1. Analyze existing domain models
grep -r "class.*:" domain/*/models.py
grep -r "def " domain/*/models.py

# 2. Map existing interfaces
find . -name "*.py" -exec grep -l "class.*ABC\|@abstractmethod" {} \;

# 3. Understand data flow
grep -r "Repository\|Service" infrastructure/ domain/
```

**Workflow:**
1. **Domain Discovery**: Map all existing domain models and their attributes
2. **Interface Analysis**: Understand all abstract base classes and interfaces
3. **Dependency Review**: Trace dependencies between layers
4. **Contract Documentation**: Document all interface contracts before modification

### 2. Interface-Contract-First (ICF) Testing

```python
# WRONG - Assumption-based mocking
mock_issue = Mock()
mock_issue.number = 59
mock_issue.title = "Test"
mock_issue.state = "open"  # String instead of enum!

# RIGHT - Contract-verified mocking
from domain.issues.models import Issue, IssueState, Label
mock_issue = Mock(spec=Issue)
mock_issue.number = 59
mock_issue.title = "Test Issue"
mock_issue.state = IssueState.OPEN  # Proper enum
mock_issue.labels = []
mock_issue.created_at = datetime.now(timezone.utc)
mock_issue.updated_at = datetime.now(timezone.utc)
```

**Workflow:**
1. **Spec-Based Mocking**: Always use `spec=` parameter with actual classes
2. **Attribute Verification**: Verify all mock attributes match real object attributes
3. **Type Consistency**: Ensure mock data types match domain model types
4. **Enum Handling**: Use actual enums instead of string representations

### 3. Incremental Architecture Validation (IAV)

**Validation Checkpoints:**
- **Checkpoint 1**: Domain model compatibility
- **Checkpoint 2**: Interface contract verification
- **Checkpoint 3**: Mock object alignment
- **Checkpoint 4**: Integration test validation
- **Checkpoint 5**: End-to-end workflow testing

**Implementation:**
```bash
# Validation script template
validate_domain_compatibility() {
    python -c "
    from domain.issues.models import Issue
    from markitect.issues.base import IssueBackend
    # Verify interface compatibility
    "
}

validate_mock_alignment() {
    # Run tests that verify mocks match real objects
    python -m pytest tests/test_mock_compatibility.py
}
```

### 4. Foundation-First Development (FFD)

**Principle**: Build on solid foundations before adding new layers.

**Workflow:**
1. **Foundation Assessment**: Verify existing infrastructure is solid
2. **Interface Stability**: Ensure base interfaces won't change during development
3. **Dependency Injection**: Plan dependency injection patterns
4. **Layer Separation**: Maintain clear separation between architectural layers

## Tools and Frameworks

### 1. Domain Analysis Tools

```bash
# Domain Model Inspector
analyze_domain_models() {
    echo "=== Domain Model Analysis ==="
    find domain/ -name "models.py" -exec echo "File: {}" \; -exec grep -n "class\|def " {} \;
}

# Interface Contract Checker
check_interface_contracts() {
    echo "=== Interface Contract Analysis ==="
    grep -r "@abstractmethod\|ABC" . --include="*.py"
}

# Mock Compatibility Validator
validate_mocks() {
    echo "=== Mock Compatibility Check ==="
    python -c "
    import inspect
    from domain.issues.models import Issue
    print('Issue attributes:', [attr for attr in dir(Issue) if not attr.startswith('_')])
    "
}
```

### 2. Test Architecture Framework

```python
# Test Base Classes for Interface Compliance
class DomainModelTestBase:
    """Base class ensuring tests match domain models."""

    def setUp(self):
        self.validate_test_setup()

    def validate_test_setup(self):
        """Verify test setup matches actual domain models."""
        pass

    def create_mock_with_spec(self, domain_class):
        """Create spec-compliant mock."""
        return Mock(spec=domain_class)

class IntegrationTestBase:
    """Base class for integration tests."""

    def setUp(self):
        self.verify_infrastructure_availability()

    def verify_infrastructure_availability(self):
        """Ensure required infrastructure is available."""
        pass
```

### 3. Interface Evolution Manager

```python
class InterfaceEvolutionManager:
    """Manages interface changes without breaking compatibility."""

    def plan_interface_change(self, interface_name, changes):
        """Plan interface changes with backward compatibility."""
        pass

    def validate_compatibility(self, old_interface, new_interface):
        """Validate that new interface is backward compatible."""
        pass

    def generate_migration_plan(self, changes):
        """Generate step-by-step migration plan."""
        pass
```

### 4. Mock Validation Framework

```python
class MockValidator:
    """Validates that mocks match real objects."""

    @staticmethod
    def validate_mock_spec(mock_obj, real_class):
        """Validate mock object matches real class specification."""
        mock_attrs = set(dir(mock_obj))
        real_attrs = set(dir(real_class))

        missing_attrs = real_attrs - mock_attrs
        extra_attrs = mock_attrs - real_attrs

        if missing_attrs:
            raise MockSpecError(f"Mock missing attributes: {missing_attrs}")

        return True

    @staticmethod
    def validate_mock_types(mock_obj, real_instance):
        """Validate mock attribute types match real object types."""
        for attr_name in dir(real_instance):
            if not attr_name.startswith('_'):
                real_value = getattr(real_instance, attr_name)
                mock_value = getattr(mock_obj, attr_name, None)

                if mock_value is not None and type(mock_value) != type(real_value):
                    raise MockTypeError(f"Type mismatch for {attr_name}")
```

## Example Workflows

### 1. Adding New CLI Command Workflow

**Phase 1: Foundation Analysis**
```bash
# 1. Analyze existing CLI structure
find cli/ -name "*.py" -exec grep -l "click\|@cli" {} \;

# 2. Understand existing domain models
python -c "
from domain.issues.models import Issue
import inspect
print(inspect.signature(Issue.__init__))
"

# 3. Map existing repository interfaces
grep -r "class.*Repository" infrastructure/
```

**Phase 2: Interface Contract Definition**
```python
# Define interface contract first
class IssueBackend(ABC):
    @abstractmethod
    def list_issues(self, state: Optional[str] = None) -> List[Issue]:
        """List issues with optional state filter."""
        pass

    @abstractmethod
    def get_issue(self, issue_id: str) -> Issue:
        """Get specific issue by ID."""
        pass
```

**Phase 3: Test Architecture Design**
```python
# Design tests that match actual interfaces
class TestIssuesCLIGroup:
    def setup_method(self):
        # Use actual domain model for mock spec
        self.mock_issue = Mock(spec=Issue)
        self.mock_issue.number = 59
        self.mock_issue.title = "Test Issue"
        self.mock_issue.state = IssueState.OPEN  # Use actual enum
        self.mock_issue.labels = []
        self.mock_issue.created_at = datetime.now(timezone.utc)
        self.mock_issue.updated_at = datetime.now(timezone.utc)
```

**Phase 4: Incremental Implementation**
- Implement abstract base class
- Create plugin system
- Add CLI commands
- Integrate with existing infrastructure

### 2. Domain Model Extension Workflow

**Phase 1: Impact Analysis**
```bash
# Find all usages of the domain model
grep -r "Issue" . --include="*.py" | grep -v __pycache__

# Check existing tests
grep -r "Issue" tests/ --include="*.py"

# Analyze database schemas
grep -r "Issue" infrastructure/repositories/
```

**Phase 2: Backward Compatibility Planning**
```python
# Plan extension that maintains compatibility
@dataclass
class Issue:
    # Existing attributes (DO NOT CHANGE)
    number: int
    title: str
    state: IssueState
    labels: List[Label]
    created_at: datetime
    updated_at: datetime

    # New attributes (with defaults for compatibility)
    body: str = ""  # Add with default
    assignees: List[str] = field(default_factory=list)
    html_url: str = ""
```

**Phase 3: Migration Strategy**
```python
# Create migration tests
class TestIssueModelMigration:
    def test_old_constructor_still_works(self):
        """Ensure old constructor calls still work."""
        issue = Issue(
            number=1,
            title="Test",
            state=IssueState.OPEN,
            labels=[],
            created_at=datetime.now(timezone.utc),
            updated_at=datetime.now(timezone.utc)
        )
        assert issue.body == ""  # Default value
```

### 3. Plugin System Development Workflow

**Phase 1: Architecture Planning**
```python
# Define plugin interface based on existing patterns
class IssueBackend(ABC):
    """Abstract base class matching existing repository patterns."""

    def __init__(self, config: Dict[str, Any]):
        self.config = config

    @abstractmethod
    def list_issues(self, state: Optional[str] = None) -> List[Issue]:
        pass
```

**Phase 2: Integration Strategy**
```python
# Plan integration with existing infrastructure
class GiteaPlugin(IssueBackend):
    def __init__(self, config: Dict[str, Any]):
        super().__init__(config)
        # Reuse existing infrastructure
        self.repository = GiteaIssueRepository(
            connection_manager=self._create_connection_manager()
        )

    def list_issues(self, state: Optional[str] = None) -> List[Issue]:
        # Use existing async infrastructure
        return asyncio.run(self.repository.get_issues(state=state))
```

## Integration Points with Existing Development Workflow

### 1. TDD8 Enhancement

**Enhanced TDD8 Workflow:**
1. **ANALYZE** - Analyze existing domain models and interfaces
2. **ISSUE** - Understand requirements in context of existing architecture
3. **TEST** - Write tests that match actual interfaces
4. **RED** - Verify tests fail for right reasons
5. **GREEN** - Implement with interface compatibility
6. **REFACTOR** - Maintain interface contracts
7. **DOCUMENT** - Update interface documentation
8. **PUBLISH** - Commit with interface change documentation

### 2. TodoWrite Integration

```python
# Enhanced TodoWrite with validation checkpoints
todos = [
    {
        "content": "Analyze existing Issue domain model",
        "status": "pending",
        "activeForm": "Analyzing existing Issue domain model",
        "checkpoint": "domain_analysis"
    },
    {
        "content": "Define IssueBackend interface contract",
        "status": "pending",
        "activeForm": "Defining IssueBackend interface contract",
        "checkpoint": "interface_definition"
    },
    {
        "content": "Create spec-compliant mocks",
        "status": "pending",
        "activeForm": "Creating spec-compliant mocks",
        "checkpoint": "mock_validation"
    }
]
```

### 3. CLI Help Integration

```bash
# Enhanced CLI with validation commands
markitect validate-interfaces    # Check interface compatibility
markitect analyze-domain        # Analyze domain models
markitect check-mocks          # Validate mock objects
markitect plan-migration       # Plan interface migrations
```

## Success Metrics

### 1. Interface Compatibility
- **Zero Mock Mismatches**: All mocks must match actual object interfaces
- **Type Safety**: 100% type consistency between tests and implementation
- **Backward Compatibility**: No breaking changes to existing interfaces

### 2. Test Quality
- **Domain Model Alignment**: Tests reflect actual domain model structure
- **Integration Coverage**: All integration points tested with real interfaces
- **Mock Validation**: All mocks validated against real object specifications

### 3. Development Efficiency
- **Reduced Debugging**: Fewer interface-related bugs
- **Faster Development**: Less time spent fixing mock mismatches
- **Better Architecture**: Cleaner interface design and evolution

## Prevention of Issue #59 Problems

This agent would have prevented the Issue #59 problems by:

1. **Domain Model Analysis**: Would have discovered the actual Issue model has `IssueState` enum, not string
2. **Interface Inventory**: Would have mapped existing GiteaIssueRepository before designing plugin interface
3. **Mock Validation**: Would have caught mock attribute mismatches before running tests
4. **Integration Planning**: Would have planned how new CLI integrates with existing infrastructure
5. **Contract Verification**: Would have ensured all interfaces match actual implementations

The result would be a solid, well-planned implementation that builds on existing foundations rather than making incorrect assumptions about interfaces and domain models.