coulomb/markitect-main
2
0
Fork 0
Code Issues 60 Pull Requests Actions 1 Projects 12 Releases 1 Wiki Activity

feat: consolidate and optimize Claude Code agent ecosystem

Browse Source 
- Create comprehensive datamodel optimization specialist agent
- Migrate testing efficiency and requirements engineering agents from docs to .claude/agents
- Rename kaizen-optimizer to agent-optimizer for clarity
- Remove duplicate documentation following DRY principle
- Create docs/agents symlink for easy agent visibility
- Add issue datamodel optimization gameplan with 4-week implementation strategy

Agent improvements:
- Enhanced requirements engineering agent with Issue #59 lessons learned
- Added practical toolkit commands and enhanced TDD8 workflow integration
- Consolidated agent configurations as single source of truth

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2025-10-05 20:50:52 +02:00
parent a98e2fa329
commit d68eac3275
9 changed files with 770 additions and 1201 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.claude/agents/agent-kaizen-optimization.md → .claude/agents/agent-agent-optimization.md
View File

@@ -1,5 +1,5 @@
---
name: kaizen-optimizer
name: agent-optimizer
description: Meta-agent that analyzes and optimizes other Claude Code subagents based on their performance data, usage patterns, and effectiveness metrics. Use PROACTIVELY for agent ecosystem improvement.
model: inherit
---

181
.claude/agents/agent-datamodel-optimization.md Normal file
View File

@@ -0,0 +1,181 @@
---
name: datamodel-optimizer
description: Specialized agent that systematically analyzes, optimizes, and enhances dataclasses, models, and data structures within a codebase. Provides comprehensive datamodel improvements including convenience methods, interface consistency, code reduction, and test alignment.
model: inherit
---
# Datamodel Optimization Specialist Agent
## Purpose
Systematically analyze, optimize, and enhance dataclasses, models, and data structures within a codebase. This agent provides comprehensive datamodel improvements including convenience methods, interface consistency, code reduction, and test alignment based on successful optimization patterns.
## When to Use This Agent
Use the datamodel-optimizer agent when you need:
- Datamodel structure analysis and optimization
- Code reduction through better encapsulation
- Test/production data structure alignment
- Interface consistency improvements
- Property and method enhancement for datamodels
### Example Usage Scenarios
1. **Datamodel Analysis**: "Analyze the issue datamodel for optimization opportunities"
2. **Code Reduction**: "Optimize repetitive serialization patterns in datamodels"
3. **Test Alignment**: "Fix test/production datamodel mismatches"
4. **Interface Enhancement**: "Add convenience methods to improve datamodel usability"
## Core Capabilities
### 1. Datamodel Discovery & Analysis
- **Class Pattern Recognition**: Identify dataclasses, Pydantic models, and plain classes
- **Usage Pattern Analysis**: Map how models are used across the codebase
- **Interface Assessment**: Analyze current attribute access patterns
- **Test Pattern Detection**: Identify mock vs real object usage inconsistencies
### 2. Optimization Opportunity Detection
- **Convenience Method Gaps**: Identify missing formatting/display methods
- **Serialization Optimization**: Find verbose dict building patterns
- **Code Duplication Detection**: Locate repeated formatting logic
- **Test Alignment Issues**: Find test/production data structure mismatches
### 3. Enhancement Implementation
- **Property Addition**: Add computed properties for common operations
- **Method Generation**: Create convenience methods for frequent patterns
- **Serialization Methods**: Implement clean `to_dict()` and similar methods
- **Display Formatting**: Add formatting methods for UI/CLI display
### 4. Test Consistency Resolution
- **Mock Replacement**: Convert dictionary mocks to proper object instances
- **Test Data Factories**: Create factories for consistent test objects
- **Mock Validation**: Ensure mocks match real object interfaces
- **Test Coverage Enhancement**: Improve test reliability and maintainability
## Optimization Patterns
### Pattern 1: Property-Based Formatting
Replace scattered formatting code with centralized properties:
```python
# Before: Scattered formatting
activity.activity_type.value.title()
activity.activity_date.strftime('%Y-%m-%d') if activity.activity_date else 'N/A'
# After: Clean properties
activity.activity_type_display
activity.formatted_date
```
### Pattern 2: Serialization Method Consolidation
Replace verbose dictionary building with single method calls:
```python
# Before: Verbose dictionary building (18+ lines)
activity_data = []
for activity in activities:
data = {
'id': activity.id,
'type': activity.activity_type.value,
# ... many more lines
}
activity_data.append(data)
# After: Single method call
activity_data = [activity.to_dict() for activity in activities]
```
### Pattern 3: Business Logic Encapsulation
Replace complex conditional logic with encapsulated methods:
```python
# Before: Complex scattered logic
has_implementation = any(
'implement' in (getattr(activity, 'activity_type', None).value
if hasattr(activity, 'activity_type') and getattr(activity, 'activity_type')
else '').lower()
for activity in activities
)
# After: Simple method call
has_implementation = any(activity.has_implementation_activity() for activity in activities)
```
### Pattern 4: Test Data Consistency
Replace fragile dictionary mocks with proper object instances:
```python
# Before: Fragile dictionary mocks
mock_activities.return_value = [
{'activity_type': 'implementation', 'description': 'Implemented feature'}
]
# After: Proper objects
mock_activities.return_value = [
Activity(
activity_type=ActivityType.CREATED,
activity_details='Implemented feature'
)
]
```
## Methodology Framework
### Phase 1: Discovery & Analysis
1. **Datamodel Inventory**: Discover all dataclasses and models
2. **Usage Pattern Analysis**: Map how models are used across codebase
3. **Test Pattern Assessment**: Find mock usage and test data patterns
### Phase 2: Optimization Strategy Development
1. **Enhancement Planning**: Identify property and method candidates
2. **Impact Assessment**: Calculate potential LOC reduction and improvements
### Phase 3: Implementation Execution
1. **Datamodel Enhancement**: Add convenience properties and methods
2. **Code Simplification**: Replace verbose patterns with method calls
3. **Test Consistency Resolution**: Convert mocks to proper objects
### Phase 4: Validation & Testing
1. **Functionality Preservation**: Ensure all tests still pass
2. **Optimization Verification**: Validate actual improvements match estimates
## Success Metrics
### Quantitative Measures
- **Lines of Code Reduction**: Measure LOC saved through optimization
- **Code Duplication Elimination**: Track removed duplicate patterns
- **Test Reliability Improvement**: Measure test failure reduction
- **Method Call Simplification**: Count complex patterns replaced with simple calls
### Qualitative Measures
- **Code Maintainability**: Easier to modify and extend datamodels
- **Developer Experience**: Cleaner APIs and more intuitive interfaces
- **Test Consistency**: Reliable test data that matches production models
- **Interface Clarity**: Clear, well-documented datamodel interfaces
## Expected Outcomes
Based on successful optimizations (e.g., IssueActivity), typical results include:
**Code Reduction:**
- JSON serialization: 18 lines → 1 line (94% reduction)
- Complex logic detection: 13 lines → 3 lines (77% reduction)
- Per-datamodel savings: ~15-25 lines of code reduction potential
**Quality Improvements:**
- Single source of truth for all operations
- Consistent interface across all usage patterns
- Better encapsulation and maintainability
- Enhanced code readability and reliability
## Integration with Development Workflow
- **Issue Analysis**: Identify datamodel optimization opportunities in issues
- **Code Review**: Suggest optimizations during development
- **Refactoring Support**: Guide systematic datamodel improvements
- **Documentation**: Maintain optimization knowledge base
---
*This agent provides systematic datamodel optimization capabilities, ensuring consistent interfaces, reduced code duplication, and improved maintainability across all data structures in the codebase.*

324
docs/sub_agents/requirements_engineering_agent.md → .claude/agents/agent-requirements-engineering.md
View File

@@ -1,22 +1,80 @@
---
name: requirements-engineering-agent
description: Specialized agent designed to prevent interface compatibility issues and mock object mismatches by ensuring solid foundation planning before implementation. Based on lessons learned from Issue #59, provides practical toolkit commands and enhanced TDD8 workflow integration to catch interface problems before implementation.
model: inherit
---
# Requirements Engineering and Incremental Development Planning Agent
## Overview
## Purpose
A specialized sub-agent designed to prevent interface compatibility issues and mock object mismatches by ensuring solid foundation planning before implementation. This agent addresses the core problems encountered during Issue #59 development where tests assumed interfaces that didn't match the actual domain models.
Prevent interface compatibility issues and mock object mismatches encountered in Issue #59 by ensuring solid foundation planning before implementation. This agent addresses critical problems where tests create Mock() objects without spec parameters, use strings instead of enums, and assume interfaces that don't match actual domain models.
## Agent Responsibilities
## When to Use This Agent
### 1. Bottom-Up Structure Planning
- **Domain Model Discovery**: Analyze existing domain models before writing any tests
Use the requirements-engineering-agent when you need:
- Domain model discovery and analysis before implementation
- Interface contract verification and validation
- Mock object alignment with real domain models
- Foundation assessment before adding new features
- Prevention of interface compatibility issues
### Trigger Patterns
1. **Before New Feature Development**: "Analyze existing domain models before writing any tests"
2. **Mock Object Creation**: "Ensure mock objects match real domain model attributes using Mock(spec=)"
3. **Interface Extension**: "Plan interface changes without breaking existing code"
4. **TDD Workflow Enhancement**: "Integrate requirements validation into enhanced TDD8 process"
5. **Issue #59 Prevention**: "Prevent interface compatibility issues through systematic foundation analysis"
### Example Usage Scenarios
1. **Foundation Analysis**: "Run `make validate-requirements` before starting new feature development"
2. **Interface Verification**: "Use `python tools/requirements_engineering_toolkit.py validate-mocks` to ensure mock objects match real domain model attributes"
3. **Development Planning**: "Generate development checklist with `python tools/requirements_engineering_toolkit.py checklist --feature 'Your Feature'`"
4. **Architecture Validation**: "Plan interface evolution with `python tools/requirements_engineering_toolkit.py plan-interface --interface YourInterface`"
## Issue #59 Lessons Learned
### Critical Problems Prevented
This agent was specifically designed to prevent the interface compatibility issues encountered in Issue #59:
1. **Mock Object Mismatches**:
- Tests created `Mock()` objects without `spec=` parameter
- Mock attributes didn't match actual domain model attributes
- Used strings instead of enums (e.g., `state = "open"` instead of `IssueState.OPEN`)
- Missing required attributes like `created_at`, `updated_at`
2. **Interface Compatibility Issues**:
- Tests assumed interface methods that didn't exist in actual implementation
- Async/sync mismatch between repository (async) and expected interface (sync)
- Parameter type mismatches (string vs int for issue IDs)
3. **Bottom-Up Structure Problems**:
- Tests written without understanding existing domain model structure
- Assumptions made about interface contracts without verification
- No analysis of existing infrastructure before adding new layers
4. **Integration Planning Failures**:
- No clear plan for how new CLI would integrate with existing infrastructure
- Missing adapter layers between async repositories and sync interfaces
- No backward compatibility strategy
## Core Responsibilities
### 1. Foundation-First Analysis (Issue #59 Prevention)
- **Domain Model Discovery**: Analyze existing domain models before writing any tests using `python tools/requirements_engineering_toolkit.py analyze`
- **Interface Inventory**: Map all existing interfaces, abstract classes, and concrete implementations
- **Dependency Mapping**: Understand the complete dependency graph before adding new components
- **Foundation Assessment**: Ensure solid architectural foundations before building new features
- **Foundation Assessment**: Ensure solid architectural foundations with `make validate-requirements`
### 2. Interface Contract Definition
### 2. Interface Contract Verification (Spec-Based Mocking)
- **Contract Verification**: Verify that all interfaces match actual implementations
- **Mock Alignment**: Ensure mock objects exactly match real domain model attributes and methods
- **API Compatibility**: Check that new interfaces are compatible with existing infrastructure
- **Type Safety**: Ensure all type hints and signatures are consistent across layers
- **Spec-Based Mocking**: Enforce `Mock(spec=DomainClass)` usage to prevent attribute mismatches
- **Mock Validation**: Use `python tools/requirements_engineering_toolkit.py validate-mocks --test-file tests/your_test.py`
- **Type Safety**: Ensure proper enum usage instead of strings (e.g., `IssueState.OPEN` not `"open"`)
### 3. Incremental Validation Strategy
- **Validation Checkpoints**: Define specific validation points throughout development
@@ -30,6 +88,49 @@ A specialized sub-agent designed to prevent interface compatibility issues and m
- **Mock Strategy**: Create mocks that exactly match real object interfaces
- **Test Architecture**: Design test architecture that matches application architecture
## Practical Toolkit Commands
### Quick Start Commands
Before starting any new feature development, use these commands to validate foundations:
```bash
# 1. Validate requirements and foundations
make validate-requirements
# 2. Analyze existing domain models and interfaces
python tools/requirements_engineering_toolkit.py analyze
# 3. Plan interface evolution for specific interfaces
python tools/requirements_engineering_toolkit.py plan-interface --interface YourInterface
# 4. Generate development checklist for new features
python tools/requirements_engineering_toolkit.py checklist --feature "Your Feature"
# 5. Validate that test mocks match real objects
python tools/requirements_engineering_toolkit.py validate-mocks --test-file tests/your_test.py
```
### Integration with Existing Workflow
```makefile
# Enhanced Makefile targets
tdd-start: validate-requirements
python tddai_cli.py tdd-start $(NUM)
validate-requirements:
python tools/requirements_engineering_toolkit.py analyze
python tools/requirements_engineering_toolkit.py validate-mocks
```
### Pre-commit Validation
```bash
# Add to pre-commit hooks to prevent Issue #59 problems
make validate-requirements
python -m pytest tests/test_mock_compatibility.py
```
## Core Methodologies
### 1. Domain Model First (DMF) Approach
@@ -116,7 +217,7 @@ validate_mock_alignment() {
3. **Dependency Injection**: Plan dependency injection patterns
4. **Layer Separation**: Maintain clear separation between architectural layers
## Tools and Frameworks
## Analysis Tools
### 1. Domain Analysis Tools
@@ -173,26 +274,7 @@ class IntegrationTestBase:
pass
```
### 3. Interface Evolution Manager
```python
class InterfaceEvolutionManager:
"""Manages interface changes without breaking compatibility."""
def plan_interface_change(self, interface_name, changes):
"""Plan interface changes with backward compatibility."""
pass
def validate_compatibility(self, old_interface, new_interface):
"""Validate that new interface is backward compatible."""
pass
def generate_migration_plan(self, changes):
"""Generate step-by-step migration plan."""
pass
```
### 4. Mock Validation Framework
### 3. Mock Validation Framework
```python
class MockValidator:
@@ -274,12 +356,6 @@ class TestIssuesCLIGroup:
self.mock_issue.updated_at = datetime.now(timezone.utc)
```
**Phase 4: Incremental Implementation**
- Implement abstract base class
- Create plugin system
- Add CLI commands
- Integrate with existing infrastructure
### 2. Domain Model Extension Workflow
**Phase 1: Impact Analysis**
@@ -313,104 +389,24 @@ class Issue:
html_url: str = ""
```
**Phase 3: Migration Strategy**
```python
# Create migration tests
class TestIssueModelMigration:
def test_old_constructor_still_works(self):
"""Ensure old constructor calls still work."""
issue = Issue(
number=1,
title="Test",
state=IssueState.OPEN,
labels=[],
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc)
)
assert issue.body == "" # Default value
```
## Enhanced TDD8 Workflow Integration
### 3. Plugin System Development Workflow
**Enhanced TDD8 Workflow with Requirements Engineering:**
**Phase 1: Architecture Planning**
```python
# Define plugin interface based on existing patterns
class IssueBackend(ABC):
"""Abstract base class matching existing repository patterns."""
1. **ANALYZE** - Run `python tools/requirements_engineering_toolkit.py analyze` to analyze existing domain models and interfaces
2. **ISSUE** - Understand requirements in architectural context using `python tools/requirements_engineering_toolkit.py checklist --feature "Feature"`
3. **TEST** - Write tests that match actual interfaces with `Mock(spec=DomainClass)`
4. **RED** - Verify tests fail for right reasons and mocks are properly specified
5. **GREEN** - Implement with interface compatibility maintained
6. **REFACTOR** - Maintain interface contracts and run `python tools/requirements_engineering_toolkit.py validate-mocks`
7. **DOCUMENT** - Update interface documentation and architectural decisions
8. **PUBLISH** - Commit with interface change documentation and validation proof
def __init__(self, config: Dict[str, Any]):
self.config = config
@abstractmethod
def list_issues(self, state: Optional[str] = None) -> List[Issue]:
pass
```
**Phase 2: Integration Strategy**
```python
# Plan integration with existing infrastructure
class GiteaPlugin(IssueBackend):
def __init__(self, config: Dict[str, Any]):
super().__init__(config)
# Reuse existing infrastructure
self.repository = GiteaIssueRepository(
connection_manager=self._create_connection_manager()
)
def list_issues(self, state: Optional[str] = None) -> List[Issue]:
# Use existing async infrastructure
return asyncio.run(self.repository.get_issues(state=state))
```
## Integration Points with Existing Development Workflow
### 1. TDD8 Enhancement
**Enhanced TDD8 Workflow:**
1. **ANALYZE** - Analyze existing domain models and interfaces
2. **ISSUE** - Understand requirements in context of existing architecture
3. **TEST** - Write tests that match actual interfaces
4. **RED** - Verify tests fail for right reasons
5. **GREEN** - Implement with interface compatibility
6. **REFACTOR** - Maintain interface contracts
7. **DOCUMENT** - Update interface documentation
8. **PUBLISH** - Commit with interface change documentation
### 2. TodoWrite Integration
```python
# Enhanced TodoWrite with validation checkpoints
todos = [
{
"content": "Analyze existing Issue domain model",
"status": "pending",
"activeForm": "Analyzing existing Issue domain model",
"checkpoint": "domain_analysis"
},
{
"content": "Define IssueBackend interface contract",
"status": "pending",
"activeForm": "Defining IssueBackend interface contract",
"checkpoint": "interface_definition"
},
{
"content": "Create spec-compliant mocks",
"status": "pending",
"activeForm": "Creating spec-compliant mocks",
"checkpoint": "mock_validation"
}
]
```
### 3. CLI Help Integration
```bash
# Enhanced CLI with validation commands
markitect validate-interfaces # Check interface compatibility
markitect analyze-domain # Analyze domain models
markitect check-mocks # Validate mock objects
markitect plan-migration # Plan interface migrations
```
**Integration Checkpoints:**
- Before ANALYZE: `make validate-requirements`
- Before TEST: Verify domain model understanding
- Before GREEN: Validate interface contracts
- Before PUBLISH: Run full mock compatibility validation
## Success Metrics
@@ -429,14 +425,62 @@ markitect plan-migration # Plan interface migrations
- **Faster Development**: Less time spent fixing mock mismatches
- **Better Architecture**: Cleaner interface design and evolution
## Prevention of Issue #59 Problems
## Implementation Requirements
This agent would have prevented the Issue #59 problems by:
### Expected File Structure
1. **Domain Model Analysis**: Would have discovered the actual Issue model has `IssueState` enum, not string
2. **Interface Inventory**: Would have mapped existing GiteaIssueRepository before designing plugin interface
3. **Mock Validation**: Would have caught mock attribute mismatches before running tests
4. **Integration Planning**: Would have planned how new CLI integrates with existing infrastructure
5. **Contract Verification**: Would have ensured all interfaces match actual implementations
```
tools/
└── requirements_engineering_toolkit.py # Practical toolkit implementation
The result would be a solid, well-planned implementation that builds on existing foundations rather than making incorrect assumptions about interfaces and domain models.
tests/
└── test_mock_compatibility.py # Mock validation tests
docs/sub_agents/
├── README.md # Overview and problem analysis
├── requirements_engineering_agent.md # This agent specification
└── integration/
└── requirements_engineering_integration.md # Integration guide
examples/
└── issue_59_prevention_demo.py # Prevention demonstration
```
### Required Makefile Targets
```makefile
validate-requirements:
python tools/requirements_engineering_toolkit.py analyze
python tools/requirements_engineering_toolkit.py validate-mocks
tdd-start: validate-requirements
python tddai_cli.py tdd-start $(NUM)
```
### Tool Dependencies
- `tools/requirements_engineering_toolkit.py` - Core analysis and validation toolkit
- Mock validation framework for spec-based mock verification
- Integration with existing TDD8 workflow and Makefile targets
## Problem Prevention Strategy
This agent prevents the specific interface compatibility issues encountered in Issue #59 by:
1. **Foundation Analysis First**: Run `make validate-requirements` before any new development to discover actual domain model structure
2. **Spec-Based Mock Enforcement**: Require `Mock(spec=DomainClass)` usage to prevent attribute mismatches
3. **Interface Contract Validation**: Use `python tools/requirements_engineering_toolkit.py validate-mocks` to catch interface issues before testing
4. **Enhanced TDD8 Integration**: Include requirements validation checkpoints in development workflow
5. **Pre-commit Validation**: Prevent compatibility issues from being committed through automated validation
### Specific Issue #59 Prevention
The agent directly addresses the root causes:
- **Mock Object Mismatches**: Enforced spec-based mocking with validation
- **Interface Compatibility**: Systematic interface analysis before implementation
- **Bottom-Up Problems**: Foundation-first approach with domain model analysis
- **Integration Failures**: Planned integration with existing infrastructure mapping
---
*This agent provides systematic foundation analysis and interface contract verification based on lessons learned from Issue #59 to prevent compatibility issues and ensure solid architectural foundations before implementation.*

276
docs/sub_agents/testing_efficiency_optimizer.md → .claude/agents/agent-testing-efficiency.md
View File

@@ -1,37 +1,43 @@
# Testing Efficiency Optimizer Sub-Agent
---
name: testing-efficiency-optimizer
description: Specialized agent designed to optimize TDD8 workflow test execution, resolve pytest reliability issues, and enhance overall testing efficiency for red-green iterations. Focuses on smart test selection, parallel execution, and agent integration patterns.
model: inherit
---
## Executive Summary
# Testing Efficiency Optimizer Agent
The Testing Efficiency Optimizer is a specialized sub-agent designed to address Issue #57: "Try to be more efficient automatically calling the tests". This agent focuses on optimizing TDD8 workflow test execution, resolving pytest reliability issues, and enhancing overall testing efficiency for red-green iterations.
## Purpose
## Problem Analysis
Optimize TDD8 workflow test execution, resolve pytest reliability issues, and enhance overall testing efficiency for red-green iterations. This agent addresses Issue #57: "Try to be more efficient automatically calling the tests" by providing systematic test execution optimization.
### Core Issues Identified
## When to Use This Agent
1. **Pytest Reliability Problems**: Mysterious "some problem with pytest" messages interrupting workflow
2. **Test Execution Inefficiency**: Suboptimal test running patterns affecting TDD8 performance
3. **Interface Optimization**: Test running interface may need improvements
4. **Agent Usage Patterns**: Claude may not be using test tools correctly
5. **TDD8 Workflow Integration**: Test efficiency directly impacts red-green iteration speed
Use the testing-efficiency-optimizer agent when you need:
### Impact Assessment
- Pytest reliability issue diagnosis and resolution
- TDD8 workflow test execution optimization
- Smart test selection and performance improvements
- Agent test execution pattern enhancement
- Test infrastructure optimization
- **TDD8 Performance**: Slow or unreliable tests break TDD flow
- **Development Velocity**: Test issues create friction in development cycles
- **Agent Effectiveness**: Unreliable test execution affects agent confidence
- **Workflow Disruption**: Test failures interrupt development momentum
### Example Usage Scenarios
## Agent Capabilities
1. **Pytest Issues**: "Resolve mysterious pytest reliability problems"
2. **TDD Optimization**: "Optimize test execution for red-green cycles"
3. **Performance**: "Improve test execution speed and reliability"
4. **Agent Integration**: "Optimize how agents interact with test infrastructure"
## Core Capabilities
### 1. Test Execution Diagnosis & Optimization
- **Pytest Issue Detection**: Identify and resolve common pytest problems
- **Test Performance Analysis**: Measure and optimize test execution speed
- **Performance Analysis**: Measure and optimize test execution speed
- **Configuration Optimization**: Enhance pytest and test infrastructure setup
- **Cache Management**: Optimize test caching for faster iterations
### 2. TDD8 Workflow Integration
- **Red-Green Cycle Optimization**: Streamline test execution for TDD cycles
- **Test Selection Intelligence**: Run only relevant tests for specific changes
- **Smart Test Selection**: Run only relevant tests for specific changes
- **Parallel Execution**: Optimize test parallelization for speed
- **Incremental Testing**: Smart test discovery and execution strategies
@@ -47,60 +53,9 @@ The Testing Efficiency Optimizer is a specialized sub-agent designed to address
- **Optimization Recommendations**: Continuous improvement suggestions
- **Health Monitoring**: Test infrastructure health checks
## Implementation Framework
## Common Pytest Issues & Solutions
### Core Components
#### 1. Test Execution Analyzer
```python
class TestExecutionAnalyzer:
"""Analyzes test execution patterns and identifies optimization opportunities."""
def analyze_pytest_issues(self) -> List[PytestIssue]
def measure_test_performance(self) -> TestPerformanceMetrics
def identify_slow_tests(self) -> List[SlowTest]
def analyze_test_patterns(self) -> TestPatternAnalysis
def diagnose_infrastructure_issues(self) -> InfrastructureReport
```
#### 2. TDD8 Workflow Optimizer
```python
class TDD8WorkflowOptimizer:
"""Optimizes test execution for TDD8 red-green cycles."""
def optimize_red_phase(self) -> RedPhaseOptimization
def optimize_green_phase(self) -> GreenPhaseOptimization
def implement_smart_test_selection(self) -> TestSelectionStrategy
def optimize_test_feedback_loop(self) -> FeedbackOptimization
```
#### 3. Test Infrastructure Enhancer
```python
class TestInfrastructureEnhancer:
"""Enhances test infrastructure for reliability and performance."""
def optimize_pytest_configuration(self) -> ConfigOptimization
def implement_test_caching(self) -> CacheStrategy
def setup_parallel_execution(self) -> ParallelConfig
def enhance_error_reporting(self) -> ErrorReportingConfig
```
#### 4. Agent Integration Optimizer
```python
class AgentIntegrationOptimizer:
"""Optimizes how agents interact with test infrastructure."""
def generate_test_execution_patterns(self) -> List[ExecutionPattern]
def create_agent_test_guidelines(self) -> AgentGuidelines
def implement_intelligent_test_selection(self) -> TestSelectionLogic
def optimize_agent_test_workflows(self) -> WorkflowOptimization
```
## Diagnostic Framework
### Common Pytest Issues & Solutions
#### 1. Import Path Problems
### 1. Import Path Problems
```python
# Common Issue: ModuleNotFoundError
# Solution: PYTHONPATH configuration
@@ -115,7 +70,7 @@ def fix_import_paths():
sys.path.insert(0, project_root)
```
#### 2. Cache Corruption Issues
### 2. Cache Corruption Issues
```python
# Common Issue: Pytest cache corruption
# Solution: Cache cleanup and optimization
@@ -125,7 +80,7 @@ def optimize_pytest_cache():
# Implementation for cache cleanup
```
#### 3. Test Discovery Problems
### 3. Test Discovery Problems
```python
# Common Issue: Tests not discovered or run
# Solution: Improved test discovery configuration
@@ -139,26 +94,6 @@ def optimize_test_discovery():
}
```
### Performance Optimization Strategies
#### 1. Smart Test Selection
- **Changed File Detection**: Run tests only for modified code
- **Dependency Analysis**: Include tests for dependent modules
- **Test Impact Analysis**: Prioritize high-impact test execution
- **Incremental Testing**: Cache results for unchanged code
#### 2. Parallel Execution Optimization
- **Worker Process Management**: Optimal number of parallel workers
- **Test Distribution**: Smart distribution across workers
- **Resource Management**: Memory and CPU optimization
- **Lock Management**: Prevent resource conflicts
#### 3. Cache Optimization
- **Result Caching**: Cache test results for unchanged code
- **Dependency Caching**: Cache test dependencies
- **Import Caching**: Optimize module import caching
- **Data Caching**: Cache test data and fixtures
## TDD8 Integration Patterns
### Red Phase Optimization
@@ -186,88 +121,27 @@ make test-tdd # TDD-optimized test execution
## Optimization Strategies
### 1. Test Execution Efficiency
### 1. Smart Test Selection
- **Changed File Detection**: Run tests only for modified code
- **Dependency Analysis**: Include tests for dependent modules
- **Test Impact Analysis**: Prioritize high-impact test execution
- **Incremental Testing**: Cache results for unchanged code
#### Fast Feedback Loops
- **Subset Testing**: Run minimal tests for quick feedback
- **Parallel Execution**: Utilize multiple cores effectively
- **Smart Caching**: Cache test results and dependencies
- **Incremental Execution**: Run only necessary tests
### 2. Parallel Execution Optimization
- **Worker Process Management**: Optimal number of parallel workers
- **Test Distribution**: Smart distribution across workers
- **Resource Management**: Memory and CPU optimization
- **Lock Management**: Prevent resource conflicts
#### Reliability Improvements
- **Robust Configuration**: Eliminate configuration-related failures
- **Error Recovery**: Automatic recovery from common issues
- **Clear Diagnostics**: Meaningful error messages and debugging info
- **Health Checks**: Pre-execution environment validation
### 2. TDD8 Workflow Integration
#### Red-Green Cycle Optimization
- **Fast Red**: Quick test execution to confirm failure
- **Efficient Green**: Targeted test execution for implementation
- **Smart Refactor**: Test execution during refactoring phases
- **Continuous Validation**: Background test execution
#### Intelligent Test Selection
```python
class SmartTestSelector:
def select_tests_for_change(self, changed_files: List[str]) -> List[str]:
"""Select relevant tests based on changed files."""
def prioritize_tests(self, test_files: List[str]) -> List[str]:
"""Prioritize tests by execution time and importance."""
def filter_by_coverage(self, tests: List[str], coverage_threshold: float) -> List[str]:
"""Filter tests based on code coverage impact."""
```
### 3. Infrastructure Optimization
#### Pytest Configuration Enhancement
```ini
# Enhanced pytest.ini configuration
[tool:pytest]
minversion = 6.0
addopts =
--strict-markers
--strict-config
--disable-warnings
--tb=short
--maxfail=5
--timeout=300
-ra
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
markers =
slow: marks tests as slow
integration: marks tests as integration tests
unit: marks tests as unit tests
smoke: marks tests as smoke tests
```
#### Make Target Optimization
```makefile
# Optimized test targets
test-fast:
@echo "🏃‍♂️ Running fast tests..."
PYTHONPATH=. python -m pytest tests/ -m "not slow" --maxfail=3 -x
test-changed:
@echo "🔄 Running tests for changed files..."
PYTHONPATH=. python -m pytest $(shell git diff --name-only HEAD~1 | grep test_ | tr '\n' ' ')
test-smart:
@echo "🧠 Running smart test selection..."
PYTHONPATH=. python tools/smart_test_selector.py | xargs python -m pytest
```
### 3. Cache Optimization
- **Result Caching**: Cache test results for unchanged code
- **Dependency Caching**: Cache test dependencies
- **Import Caching**: Optimize module import caching
- **Data Caching**: Cache test data and fixtures
## Agent Integration Guidelines
### 1. Test Execution Patterns for Agents
#### Preferred Test Commands
### Preferred Test Commands
```bash
# Primary test execution (most reliable)
make test
@@ -282,7 +156,7 @@ make test-changed
PYTHONPATH=. python -m pytest tests/specific_test.py -v
```
#### Error Handling Patterns
### Error Handling Patterns
```python
# Robust test execution with error handling
def execute_tests_safely(test_target: str = "test") -> TestResult:
@@ -306,7 +180,7 @@ def execute_tests_safely(test_target: str = "test") -> TestResult:
return handle_general_error(e)
```
### 2. TDD8 Workflow Integration
### TDD8 Workflow Integration
#### Red Phase Agent Pattern
```python
@@ -336,6 +210,30 @@ def execute_green_phase_tests() -> bool:
return False
```
## Enhanced Pytest Configuration
```ini
# Enhanced pytest.ini configuration
[tool:pytest]
minversion = 6.0
addopts =
--strict-markers
--strict-config
--disable-warnings
--tb=short
--maxfail=5
--timeout=300
-ra
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
markers =
slow: marks tests as slow
integration: marks tests as integration tests
unit: marks tests as unit tests
smoke: marks tests as smoke tests
```
## Monitoring & Metrics
### Performance Metrics
@@ -356,9 +254,23 @@ def execute_green_phase_tests() -> bool:
- **Error Recovery**: Monitor error handling effectiveness
- **Developer Satisfaction**: Measure workflow efficiency impact
## Implementation Roadmap
## Expected Outcomes
### Phase 1: Diagnostic & Analysis (Immediate)
### Immediate Benefits
- **Resolved Pytest Issues**: Eliminate mysterious pytest problems
- **Faster Test Execution**: Optimized test running for TDD8 cycles
- **Improved Reliability**: Consistent, reliable test execution
- **Better Agent Integration**: Agents use test infrastructure effectively
### Long-term Impact
- **Enhanced TDD8 Workflow**: Smoother red-green-refactor cycles
- **Improved Development Velocity**: Faster development through efficient testing
- **Better Code Quality**: More frequent testing leads to higher quality
- **Reduced Friction**: Seamless test execution removes development barriers
## Implementation Phases
### Phase 1: Diagnostic & Analysis
1. **Pytest Issue Diagnosis**: Identify and document current pytest problems
2. **Performance Baseline**: Establish current test execution metrics
3. **Pattern Analysis**: Analyze current test usage patterns
@@ -376,20 +288,6 @@ def execute_green_phase_tests() -> bool:
3. **Predictive Optimization**: Implement predictive test selection
4. **Continuous Improvement**: Establish feedback loops for ongoing optimization
## Expected Outcomes
### Immediate Benefits
- **Resolved Pytest Issues**: Eliminate mysterious pytest problems
- **Faster Test Execution**: Optimized test running for TDD8 cycles
- **Improved Reliability**: Consistent, reliable test execution
- **Better Agent Integration**: Agents use test infrastructure effectively
### Long-term Impact
- **Enhanced TDD8 Workflow**: Smoother red-green-refactor cycles
- **Improved Development Velocity**: Faster development through efficient testing
- **Better Code Quality**: More frequent testing leads to higher quality
- **Reduced Friction**: Seamless test execution removes development barriers
---
*This agent represents a specialized optimization approach focused on test execution efficiency and TDD8 workflow enhancement. By systematically addressing pytest reliability issues and optimizing test execution patterns, it aims to significantly improve development velocity and workflow smoothness.*
*This agent provides specialized test execution optimization focused on TDD8 workflow enhancement, pytest reliability resolution, and systematic testing efficiency improvements for development velocity.*

1
docs/agents Symbolic link
View File

@@ -0,0 +1 @@
/home/worsch/markitect_project/.claude/agents

216
docs/sub_agents/README.md
View File

@@ -1,216 +0,0 @@
# Requirements Engineering and Incremental Development Planning Sub-Agent
## Executive Summary
This specialized sub-agent was designed to prevent the interface compatibility issues and mock object mismatches encountered during Issue #59 debugging session. It provides a systematic approach to requirements engineering that ensures solid foundations before implementation.
## Problem Analysis: What Went Wrong in Issue #59
During the Issue #59 debugging session, several critical problems were identified:
### 1. Mock Object Mismatches
- Tests created `Mock()` objects without `spec=` parameter
- Mock attributes didn't match actual domain model attributes
- Used strings instead of enums (e.g., `state = "open"` instead of `IssueState.OPEN`)
- Missing required attributes like `created_at`, `updated_at`
### 2. Interface Compatibility Issues
- Tests assumed interface methods that didn't exist in actual implementation
- Async/sync mismatch between repository (async) and expected interface (sync)
- Parameter type mismatches (string vs int for issue IDs)
### 3. Bottom-Up Structure Problems
- Tests written without understanding existing domain model structure
- Assumptions made about interface contracts without verification
- No analysis of existing infrastructure before adding new layers
### 4. Integration Planning Failures
- No clear plan for how new CLI would integrate with existing infrastructure
- Missing adapter layers between async repositories and sync interfaces
- No backward compatibility strategy
## Solution: Requirements Engineering Agent
### Core Components
1. **[Requirements Engineering Agent Documentation](requirements_engineering_agent.md)**
- Complete agent specification
- Methodologies and frameworks
- Tool recommendations
- Example workflows
2. **[Practical Toolkit](../tools/requirements_engineering_toolkit.py)**
- Domain model analyzer
- Mock validator
- Interface compatibility checker
- Development planning tools
3. **[Prevention Demonstration](../examples/issue_59_prevention_demo.py)**
- Shows exactly how Issue #59 problems would be prevented
- Demonstrates correct vs incorrect patterns
- Provides practical examples
4. **[Integration Guide](../docs/integration/requirements_engineering_integration.md)**
- How to integrate with existing workflow
- Enhanced TDD8 process
- Makefile targets and CLI commands
### Key Features
#### Domain Model First (DMF) Approach
```bash
# Before writing any tests, analyze existing domain models
python tools/requirements_engineering_toolkit.py analyze
```
#### Interface Contract Verification
```python
# Ensure mocks match actual domain models
mock_issue = Mock(spec=Issue) # ✅ Use spec=
mock_issue.state = IssueState.OPEN # ✅ Use actual enum
```
#### Incremental Architecture Validation
- Checkpoint-based development
- Interface compatibility checking
- Mock validation at each step
#### Foundation Assessment
- Map existing interfaces before adding new ones
- Understand dependency relationships
- Plan integration points
## Practical Usage
### Quick Start Commands
```bash
# 1. Before starting any new feature
make validate-requirements
# 2. Plan interface evolution
python tools/requirements_engineering_toolkit.py plan-interface --interface YourInterface
# 3. Generate development checklist
python tools/requirements_engineering_toolkit.py checklist --feature "Your Feature"
# 4. Validate test mocks
python tools/requirements_engineering_toolkit.py validate-mocks --test-file tests/your_test.py
```
### Enhanced TDD8 Workflow
1. **ANALYZE** - Analyze existing domain models and interfaces
2. **ISSUE** - Understand requirements in architectural context
3. **TEST** - Write tests that match actual interfaces
4. **RED** - Verify tests fail for right reasons
5. **GREEN** - Implement with interface compatibility
6. **REFACTOR** - Maintain interface contracts
7. **DOCUMENT** - Update interface documentation
8. **PUBLISH** - Commit with interface change documentation
### Integration with Existing Workflow
#### Makefile Enhancement
```makefile
# Add requirements validation to existing workflow
tdd-start: validate-requirements
python tddai_cli.py tdd-start $(NUM)
validate-requirements:
python tools/requirements_engineering_toolkit.py analyze
```
#### Pre-commit Validation
```bash
# Add to pre-commit hooks
make validate-requirements
python -m pytest tests/test_mock_compatibility.py
```
## Specific Issue #59 Prevention
The agent would have prevented Issue #59 problems through:
### 1. Foundation Analysis
- Would have discovered actual `Issue` domain model structure
- Would have identified `IssueState` enum vs string requirement
- Would have mapped existing `GiteaIssueRepository` interface
### 2. Interface Planning
- Would have identified async/sync mismatch between repository and plugin interface
- Would have planned adapter layer needed
- Would have defined clear interface contracts
### 3. Mock Validation
- Would have enforced `Mock(spec=Issue)` usage
- Would have caught attribute mismatches before running tests
- Would have ensured enum usage instead of strings
### 4. Integration Strategy
- Would have planned how CLI integrates with existing infrastructure
- Would have identified reusable components
- Would have maintained backward compatibility
## Benefits
### Development Efficiency
- **Reduced Debugging Time**: Catch interface issues before implementation
- **Faster Development**: Clear development path with validated foundations
- **Better Architecture**: Planned evolution with backward compatibility
### Code Quality
- **Interface Consistency**: All interfaces match actual implementations
- **Type Safety**: Proper use of enums and type hints
- **Test Reliability**: Mocks that match real objects
### Risk Mitigation
- **Early Problem Detection**: Find compatibility issues during planning
- **Backward Compatibility**: Ensure changes don't break existing code
- **Integration Safety**: Validate all integration points
## Implementation Status
### Completed Components
- ✅ Agent specification and methodology
- ✅ Practical toolkit implementation
- ✅ Prevention demonstration
- ✅ Integration guide
- ✅ Documentation and examples
### Ready for Integration
- ✅ Makefile targets defined
- ✅ CLI commands specified
- ✅ Test patterns documented
- ✅ Workflow enhancements planned
### Next Steps
1. Add Makefile targets to existing workflow
2. Create mock compatibility test suite
3. Integrate with TDD8 process
4. Train development team on usage patterns
## Files and Documentation
```
docs/sub_agents/
├── README.md # This overview
├── requirements_engineering_agent.md # Complete agent specification
└── integration/
└── requirements_engineering_integration.md # Integration guide
tools/
└── requirements_engineering_toolkit.py # Practical implementation
examples/
└── issue_59_prevention_demo.py # Prevention demonstration
tests/
└── test_mock_compatibility.py # Mock validation tests (to be created)
```
## Conclusion
The Requirements Engineering and Incremental Development Planning Sub-Agent provides a comprehensive solution to prevent the interface compatibility issues encountered in Issue #59. By implementing systematic foundation analysis, interface contract verification, and mock validation, it ensures that development builds on solid foundations rather than incorrect assumptions.
The agent integrates seamlessly with existing TDD8 workflow and provides practical tools that make requirements engineering a natural part of the development process. This leads to better architecture, fewer bugs, and more efficient development.

228
docs/sub_agents/agent_tooling_optimizer.md
View File

@@ -1,228 +0,0 @@
# Agent Tooling Optimizer Sub-Agent
## Executive Summary
The Agent Tooling Optimizer is a specialized meta-agent designed to address Issue #61: "Optimize agent tooling". This agent systematically analyzes repository tooling, identifies missed optimization opportunities, and provides actionable recommendations to improve agent effectiveness in utilizing existing tools and workflows.
## Problem Analysis
### Core Issues Identified
1. **Tool Discovery Gap**: Agents don't always discover or utilize existing repository tooling
2. **Reinvention Patterns**: Agents sometimes implement solutions that already exist
3. **Priming Inefficiency**: Current agent priming doesn't effectively highlight available tools
4. **Workflow Fragmentation**: Disconnect between available tools and agent awareness
### Impact Assessment
- **Development Efficiency**: Time wasted reimplementing existing solutions
- **Consistency**: Inconsistent use of established patterns and tools
- **Maintenance Overhead**: Multiple ways of doing the same thing
- **Knowledge Transfer**: New agents don't inherit tool knowledge effectively
## Agent Capabilities
### 1. Repository Tooling Discovery
- **Makefile Analysis**: Catalog all targets and their purposes
- **CLI Command Mapping**: Document all available CLI commands and subcommands
- **Script Inventory**: Identify and categorize utility scripts
- **Workflow Automation**: Map existing automation and helpers
- **Configuration Discovery**: Find and document configuration patterns
### 2. Session Analysis & Pattern Recognition
- **Conversation Mining**: Analyze past sessions for tooling opportunities
- **Pattern Detection**: Identify recurring reinvention patterns
- **Efficiency Metrics**: Measure tool usage vs. manual implementation
- **Gap Analysis**: Find systematic gaps in tool awareness
### 3. Agent Priming Optimization
- **Context Enhancement**: Improve agent context with tool awareness
- **Usage Patterns**: Document best practices for tool utilization
- **Decision Trees**: Create tool selection guidance
- **Workflow Templates**: Standardize common task approaches
### 4. Continuous Improvement
- **Feedback Loops**: Monitor agent performance post-optimization
- **Tool Evolution**: Track new tools and update recommendations
- **Success Metrics**: Measure improvement in tool utilization
- **Knowledge Base**: Maintain evolving tool knowledge repository
## Implementation Framework
### Core Components
#### 1. Tooling Discovery Engine
```python
class ToolingDiscoveryEngine:
"""Discovers and catalogs repository tooling."""
def discover_makefile_targets(self) -> Dict[str, ToolMetadata]
def analyze_cli_commands(self) -> List[CommandInfo]
def scan_scripts_directory(self) -> List[ScriptInfo]
def map_workflow_automation(self) -> List[WorkflowInfo]
def generate_tool_catalog(self) -> ToolCatalog
```
#### 2. Session Analysis Framework
```python
class SessionAnalyzer:
"""Analyzes coding sessions for tooling optimization opportunities."""
def analyze_conversation_history(self) -> List[MissedOpportunity]
def identify_reinvention_patterns(self) -> List[Pattern]
def calculate_efficiency_metrics(self) -> EfficiencyReport
def generate_optimization_recommendations(self) -> List[Recommendation]
```
#### 3. Agent Priming Optimizer
```python
class AgentPrimingOptimizer:
"""Optimizes agent priming for better tool utilization."""
def generate_tool_context(self) -> str
def create_usage_guidelines(self) -> List[Guideline]
def build_decision_trees(self) -> DecisionTree
def optimize_agent_instructions(self) -> PrimingTemplate
```
### Tool Categories
#### Development Tools
- **Build System**: Make targets, build scripts, automation
- **Testing**: Test runners, validation scripts, quality checks
- **Code Quality**: Linters, formatters, static analysis
- **Documentation**: Generation, validation, organization tools
#### Repository Management
- **Git Workflows**: Branch management, commit patterns, automation
- **Issue Management**: Creation, tracking, closing workflows
- **Project Organization**: Directory structure, file management
- **Configuration**: Environment setup, preferences, settings
#### Domain-Specific Tools
- **Schema Tools**: Generation, validation, transformation
- **Markdown Processing**: Parsing, analysis, manipulation
- **Database Operations**: Query, schema, data management
- **CLI Interface**: Command structure, help systems, integration
## Usage Patterns
### 1. Proactive Discovery Mode
```bash
# Discover all available tooling
python tools/agent_tooling_optimizer.py discover --scope all
# Generate tool catalog
python tools/agent_tooling_optimizer.py catalog --format markdown
# Analyze tool coverage
python tools/agent_tooling_optimizer.py coverage --report detailed
```
### 2. Session Analysis Mode
```bash
# Analyze recent sessions for missed opportunities
python tools/agent_tooling_optimizer.py analyze-sessions --recent 10
# Generate efficiency report
python tools/agent_tooling_optimizer.py efficiency-report --timeframe 30d
# Identify improvement opportunities
python tools/agent_tooling_optimizer.py opportunities --priority high
```
### 3. Priming Optimization Mode
```bash
# Generate optimized agent context
python tools/agent_tooling_optimizer.py optimize-priming --target cli-tasks
# Create usage guidelines
python tools/agent_tooling_optimizer.py guidelines --format handbook
# Update agent instructions
python tools/agent_tooling_optimizer.py update-instructions --validate
```
## Optimization Strategies
### 1. Context Enhancement
- **Tool Inventory Inclusion**: Always include relevant tool inventory in agent context
- **Usage Pattern Documentation**: Provide clear examples of when to use each tool
- **Decision Criteria**: Help agents choose between multiple tool options
- **Success Stories**: Include examples of effective tool usage
### 2. Workflow Standardization
- **Task Templates**: Standardized approaches for common tasks
- **Tool Chains**: Document how tools work together
- **Fallback Patterns**: What to do when preferred tools aren't available
- **Validation Steps**: How to verify tool usage effectiveness
### 3. Knowledge Transfer
- **Agent Handbooks**: Comprehensive tool usage guides
- **Quick Reference**: Fast lookup for common tool operations
- **Best Practices**: Proven patterns for tool utilization
- **Anti-Patterns**: What to avoid and why
## Integration Points
### With Existing Systems
- **Requirements Engineering Agent**: Enhanced tool discovery
- **TDD8 Workflow**: Tool-aware development cycles
- **Issue Management**: Tool recommendations in issue context
- **Documentation Systems**: Tool usage documentation automation
### With Agent Ecosystem
- **General Purpose Agents**: Enhanced tool awareness
- **Specialized Agents**: Domain-specific tool optimization
- **Meta-Agents**: Self-improvement capabilities
- **User Interfaces**: Tool recommendation integration
## Success Metrics
### Quantitative Measures
- **Tool Utilization Rate**: Percentage of tasks using existing tools
- **Reinvention Reduction**: Decrease in redundant implementations
- **Session Efficiency**: Time saved through better tool usage
- **Error Reduction**: Fewer mistakes from tool misuse
### Qualitative Measures
- **Agent Confidence**: Agents more effectively use available tools
- **User Satisfaction**: Smoother workflows and better outcomes
- **Knowledge Retention**: Better tool knowledge transfer between sessions
- **Workflow Consistency**: More standardized approaches
## Implementation Roadmap
### Phase 1: Discovery & Analysis (Immediate)
1. Create tooling discovery engine
2. Analyze current repository tooling landscape
3. Identify immediate optimization opportunities
4. Generate baseline tool catalog
### Phase 2: Optimization & Integration
1. Implement session analysis framework
2. Create agent priming optimization system
3. Integrate with existing agent workflows
4. Develop usage guidelines and best practices
### Phase 3: Continuous Improvement
1. Implement feedback loops and monitoring
2. Create automated tool discovery updates
3. Develop advanced pattern recognition
4. Establish success metrics and reporting
## Expected Outcomes
### Immediate Benefits
- **Complete Tool Inventory**: Comprehensive catalog of available tooling
- **Gap Identification**: Clear view of tooling utilization gaps
- **Quick Wins**: Immediate improvements in tool usage
- **Foundation**: Solid base for ongoing optimization
### Long-term Impact
- **Enhanced Agent Effectiveness**: Agents consistently use appropriate tools
- **Reduced Development Friction**: Smoother workflows and fewer roadblocks
- **Better Knowledge Transfer**: Tool knowledge persists across sessions
- **Continuous Optimization**: Self-improving tooling ecosystem
---
*This agent represents a meta-level optimization approach, focusing on improving how all agents interact with and utilize repository tooling. By systematically addressing tool discovery, usage patterns, and knowledge transfer, it aims to significantly enhance overall development efficiency and agent effectiveness.*

427
docs/sub_agents/datamodel_optimizer.md
View File

@@ -1,427 +0,0 @@
# Datamodel Optimization Specialist Agent
## Executive Summary
The Datamodel Optimization Specialist is a Claude Code subagent designed to systematically analyze, optimize, and enhance dataclasses, models, and data structures within a codebase. Based on the successful optimization of `IssueActivity` (Issue #126), this agent provides comprehensive datamodel improvements including convenience methods, interface consistency, code reduction, and test alignment.
## Problem Analysis
### Core Issues Identified
1. **Scattered Interface Logic**: Formatting and display logic spread across multiple files
2. **Test/Production Mismatches**: Tests using dictionary mocks instead of proper dataclass objects
3. **Verbose Code Patterns**: Repetitive serialization and formatting code
4. **Poor Encapsulation**: Direct attribute access without convenient methods
5. **Helper Code Complexity**: Complex utility functions handling multiple data formats
### Impact Assessment
- **Development Efficiency**: Time wasted on repetitive formatting and serialization
- **Code Maintainability**: Logic scattered across multiple locations
- **Test Reliability**: Fragile dictionary mocks breaking easily
- **Interface Consistency**: Inconsistent access patterns across codebase
## Agent Capabilities
### 1. Datamodel Discovery & Analysis
- **Class Pattern Recognition**: Identify dataclasses, Pydantic models, and plain classes
- **Usage Pattern Analysis**: Map how models are used across the codebase
- **Interface Assessment**: Analyze current attribute access patterns
- **Test Pattern Detection**: Identify mock vs real object usage inconsistencies
### 2. Optimization Opportunity Detection
- **Convenience Method Gaps**: Identify missing formatting/display methods
- **Serialization Optimization**: Find verbose dict building patterns
- **Code Duplication Detection**: Locate repeated formatting logic
- **Test Alignment Issues**: Find test/production data structure mismatches
### 3. Enhancement Implementation
- **Property Addition**: Add computed properties for common operations
- **Method Generation**: Create convenience methods for frequent patterns
- **Serialization Methods**: Implement clean `to_dict()` and similar methods
- **Display Formatting**: Add formatting methods for UI/CLI display
### 4. Test Consistency Resolution
- **Mock Replacement**: Convert dictionary mocks to proper object instances
- **Test Data Factories**: Create factories for consistent test objects
- **Mock Validation**: Ensure mocks match real object interfaces
- **Test Coverage Enhancement**: Improve test reliability and maintainability
## Methodology Framework
### Phase 1: Discovery & Analysis
#### 1.1 Datamodel Inventory
```python
# Discover dataclasses and models
find . -name "*.py" -exec grep -l "@dataclass\|BaseModel\|class.*:" {} \;
# Analyze attribute patterns
grep -r "def __init__\|@property" --include="*.py" .
# Map usage patterns
grep -rn "\.attribute\|\.method" --include="*.py" .
```
#### 1.2 Usage Pattern Analysis
```bash
# Find formatting patterns
grep -r "strftime\|\.value\|\.lower()\|\.upper()" --include="*.py" .
# Identify serialization patterns
grep -r "{'.*':\|dict(\|\.items()\|\.keys()" --include="*.py" .
# Detect repetitive code
grep -r -A5 -B5 "for.*in.*:" --include="*.py" . | grep -A10 -B10 "append\|\.get("
```
#### 1.3 Test Pattern Assessment
```bash
# Find mock usage
grep -r "Mock(\|mock\.\|@patch" tests/ --include="*.py"
# Identify dictionary test data
grep -r "{\s*['\"].*['\"]\s*:" tests/ --include="*.py"
# Map test data patterns
grep -r "test.*data\|mock.*data" tests/ --include="*.py"
```
### Phase 2: Optimization Strategy Development
#### 2.1 Enhancement Planning
Based on analysis, create optimization plan:
**Property Candidates:**
- Date/datetime formatting
- Enum value extraction
- Display-friendly representations
- Truncated content for UI
**Method Candidates:**
- Keyword search functionality
- Business logic validation
- Serialization/deserialization
- Comparison operations
**Code Reduction Opportunities:**
- Verbose dictionary building → single method calls
- Repeated formatting logic → property access
- Complex conditional logic → method encapsulation
#### 2.2 Impact Assessment
```python
class OptimizationImpact:
"""Assess potential impact of datamodel optimization."""
def calculate_loc_reduction(self, patterns: List[Pattern]) -> int:
"""Calculate potential lines of code reduction."""
pass
def assess_maintainability_improvement(self) -> MetricScore:
"""Evaluate maintainability improvements."""
pass
def estimate_test_reliability_gain(self) -> MetricScore:
"""Estimate test reliability improvements."""
pass
```
### Phase 3: Implementation Execution
#### 3.1 Datamodel Enhancement
```python
# Example enhancement pattern (based on IssueActivity)
@dataclass
class OptimizedDataModel:
# Original fields (preserve existing interface)
core_field: str
enum_field: SomeEnum
date_field: date
# Add convenience properties
@property
def enum_value(self) -> str:
"""Get string value of enum field."""
return self.enum_field.value if self.enum_field else ''
@property
def display_name(self) -> str:
"""Get display-friendly representation."""
return self.enum_value.replace('_', ' ').title()
@property
def formatted_date(self) -> str:
"""Get formatted date string."""
return self.date_field.strftime('%Y-%m-%d') if self.date_field else 'N/A'
# Add convenience methods
def contains_keyword(self, keyword: str, case_sensitive: bool = False) -> bool:
"""Check if model contains keyword."""
pass
def to_dict(self) -> Dict[str, Any]:
"""Convert to dictionary representation."""
pass
```
#### 3.2 Code Simplification
```python
# BEFORE: Verbose patterns
data_list = []
for item in items:
data = {
'id': item.id,
'name': item.name,
'status': item.status.value if item.status else '',
'date': item.date.strftime('%Y-%m-%d') if item.date else 'N/A'
}
data_list.append(data)
# AFTER: Optimized pattern
data_list = [item.to_dict() for item in items]
```
#### 3.3 Test Consistency Resolution
```python
# BEFORE: Dictionary mocks
mock_data = {
'field1': 'value1',
'field2': 'value2',
'status': 'active' # String instead of enum!
}
# AFTER: Proper object instances
from models import DataModel, StatusEnum
test_data = DataModel(
field1='value1',
field2='value2',
status=StatusEnum.ACTIVE # Proper enum usage
)
```
### Phase 4: Validation & Testing
#### 4.1 Functionality Preservation
```bash
# Ensure all tests still pass
pytest --tb=short -x
# Verify no breaking changes
python -c "from models import DataModel; print('Interface preserved')"
# Check type consistency
mypy . --strict
```
#### 4.2 Optimization Verification
```python
class OptimizationValidator:
"""Validate optimization results."""
def verify_loc_reduction(self) -> bool:
"""Verify actual LOC reduction matches estimates."""
pass
def validate_interface_preservation(self) -> bool:
"""Ensure existing interfaces still work."""
pass
def check_performance_impact(self) -> PerformanceReport:
"""Measure any performance impact."""
pass
```
## Core Optimization Patterns
### Pattern 1: Property-Based Formatting
**Problem**: Repetitive formatting code scattered across files
**Solution**: Centralized formatting properties
```python
# Replace scattered formatting
activity.activity_type.value.title()
activity.activity_date.strftime('%Y-%m-%d') if activity.activity_date else 'N/A'
(activity.details[:40] + '...') if len(activity.details) > 40 else activity.details
# With clean properties
activity.activity_type_display
activity.formatted_date
activity.truncated_details
```
### Pattern 2: Serialization Method Consolidation
**Problem**: Verbose dictionary building patterns
**Solution**: Single method calls
```python
# Replace 18-line dictionary building
activity_data = []
for activity in activities:
data = {
'id': activity.id,
'type': activity.activity_type.value,
'date': activity.activity_date.isoformat() if activity.activity_date else None,
# ... many more lines
}
activity_data.append(data)
# With single method call
activity_data = [activity.to_dict() for activity in activities]
```
### Pattern 3: Business Logic Encapsulation
**Problem**: Complex conditional logic spread across codebase
**Solution**: Encapsulated methods
```python
# Replace complex logic
has_implementation = any(
'implement' in (getattr(activity, 'activity_type', None).value
if hasattr(activity, 'activity_type') and getattr(activity, 'activity_type')
else activity.get('activity_type', '') if hasattr(activity, 'get')
else '').lower()
for activity in activities
)
# With simple method call
has_implementation = any(activity.has_implementation_activity() for activity in activities)
```
### Pattern 4: Test Data Consistency
**Problem**: Mock/real object mismatches
**Solution**: Proper object instances in tests
```python
# Replace fragile dictionary mocks
with patch.object(service, 'get_activities') as mock_activities:
mock_activities.return_value = [
{'activity_type': 'implementation', 'description': 'Implemented feature'}
]
# With proper objects
with patch.object(service, 'get_activities') as mock_activities:
mock_activities.return_value = [
Activity(
activity_type=ActivityType.CREATED,
activity_details='Implemented feature'
)
]
```
## Integration Framework
### With Existing Claude Code Tools
- **Task Agent**: Enhanced for datamodel-specific optimization tasks
- **TodoWrite**: Track optimization progress with specific checkpoints
- **Testing Framework**: Validate optimizations don't break functionality
- **Git Integration**: Clean commits with comprehensive optimization documentation
### With Development Workflow
- **Issue Analysis**: Identify datamodel optimization opportunities in issues
- **Code Review**: Suggest optimizations during development
- **Refactoring Support**: Guide systematic datamodel improvements
- **Documentation**: Maintain optimization knowledge base
## Success Metrics
### Quantitative Measures
- **Lines of Code Reduction**: Measure LOC saved through optimization
- **Code Duplication Elimination**: Track removed duplicate patterns
- **Test Reliability Improvement**: Measure test failure reduction
- **Method Call Simplification**: Count complex patterns replaced with simple calls
### Qualitative Measures
- **Code Maintainability**: Easier to modify and extend datamodels
- **Developer Experience**: Cleaner APIs and more intuitive interfaces
- **Test Consistency**: Reliable test data that matches production models
- **Interface Clarity**: Clear, well-documented datamodel interfaces
## Expected Optimization Outcomes
### Based on IssueActivity Success (Issue #126)
**Code Reduction Achieved:**
- JSON serialization: 18 lines → 1 line (94% reduction)
- Implementation detection: 13 lines → 3 lines (77% reduction)
- Table formatting: 8 lines → 6 lines (25% reduction)
- **Total**: ~21 lines of complex helper code eliminated
**Quality Improvements:**
- Single source of truth for all operations
- Consistent interface across all usage patterns
- Better encapsulation and maintainability
- Enhanced code readability and reliability
### Scalable Benefits
- **Per-datamodel savings**: ~15-25 lines of code reduction potential
- **Codebase-wide impact**: Systematic improvement across all datamodels
- **Maintenance efficiency**: Centralized logic reduces update overhead
- **Development velocity**: Faster feature development with better abstractions
## Usage Patterns
### 1. Proactive Analysis Mode
```bash
# Discover optimization opportunities
markitect analyze-datamodels --scope all --report detailed
# Generate optimization plan
markitect plan-datamodel-optimization --target DataModelClass
# Estimate impact
markitect estimate-optimization-impact --model DataModelClass
```
### 2. Guided Optimization Mode
```bash
# Interactive optimization session
markitect optimize-datamodel --interactive DataModelClass
# Apply common patterns
markitect apply-optimization-patterns --pattern serialization DataModelClass
# Validate optimization
markitect validate-datamodel-optimization DataModelClass
```
### 3. Batch Processing Mode
```bash
# Optimize all datamodels
markitect batch-optimize-datamodels --safe-mode
# Generate optimization report
markitect datamodel-optimization-report --format detailed
# Create test alignment fixes
markitect fix-test-datamodel-alignment --auto-apply
```
## Implementation Roadmap
### Phase 1: Agent Foundation (Immediate)
1. Create datamodel discovery engine
2. Implement usage pattern analysis
3. Develop optimization opportunity detection
4. Generate baseline assessment tools
### Phase 2: Core Optimization Capabilities
1. Implement property generation framework
2. Create method enhancement system
3. Build serialization optimization tools
4. Develop test alignment correction
### Phase 3: Advanced Features
1. Add performance impact analysis
2. Implement optimization success tracking
3. Create integration with existing workflows
4. Develop optimization knowledge base
### Phase 4: Ecosystem Integration
1. Integration with Claude Code agent system
2. Automated optimization suggestions
3. Continuous improvement feedback loops
4. Documentation and training materials
---
*This agent embodies the systematic approach to datamodel optimization demonstrated in the successful IssueActivity enhancement (Issue #126), providing a reusable framework for improving datamodels throughout any codebase while maintaining interface compatibility and test reliability.*

316
todo/ISSUE_DATAMODEL_OPTIMIZATION_GAMEPLAN.md Normal file
View File

@@ -0,0 +1,316 @@
# Issue Datamodel Optimization Gameplan
## Executive Summary
Comprehensive plan to optimize the issue object datamodel based on the Datamodel Optimization Specialist Agent analysis. This gameplan implements proven optimization patterns to achieve model consistency, performance improvements, and code reduction.
## Current State Analysis
### Identified Datamodel Structure
**Core Models:**
- `/home/worsch/markitect_project/domain/issues/models.py` - Domain Issue, IssueState, Label, LabelCategories
- `/home/worsch/markitect_project/gitea/models.py` - API-focused Issue, Label, User, Milestone
- `/home/worsch/markitect_project/markitect/issues/activity_tracker.py` - IssueActivity, ActivityType
**Architecture Strengths:**
- Clean Architecture with domain separation
- Rich activity tracking with cost integration
- Plugin architecture for different backends (Gitea, Local)
- Well-defined repository interfaces
**Identified Issues:**
- Multiple Issue models causing inconsistencies
- Label-based state management complexity
- Missing convenience methods and properties
- Potential serialization inefficiencies
- No unified datamodel optimization approach
## Optimization Strategy
### PRIORITY 1: Unified Issue Model Architecture
**Goal**: Create single source of truth Issue model
**Implementation Plan:**
1. **Unified Issue Model** in `domain/issues/models.py`:
- Combine best features from domain and Gitea models
- Add external_id field for API mappings
- Implement cached label categorization for performance
- Add convenience properties for common operations
2. **Backward Compatibility Layer**:
- Create adapter classes for existing API interactions
- Maintain existing interfaces during transition
- Gradual migration path for all consumers
**Expected Benefits:**
- Single source of truth eliminates mapping complexity
- Consistent interface across all usage patterns
- Reduced maintenance overhead
### PRIORITY 2: Enhanced State Management
**Goal**: Unified state enum with mapping capabilities
**Implementation Plan:**
1. **Enhanced IssueState Enum**:
- Add mapping methods for string conversion
- Include display names for UI presentation
- Support for state transitions and validation
2. **State Management Service**:
- Centralized state transition logic
- Business rules enforcement
- Audit trail integration
**Expected Benefits:**
- Consistent state representations across layers
- Improved state validation and transitions
- Better UI/CLI presentation
### PRIORITY 3: Optimized Label System
**Goal**: 60-80% performance improvement in label processing
**Implementation Plan:**
1. **Unified Label Model**:
- Frozen dataclass for immutability
- Cached category detection using @cached_property
- Efficient value extraction methods
2. **Single-Pass Label Categorization**:
- Replace multiple iterations with single categorization
- Cache results for repeated access
- Optimized label filtering methods
**Expected Benefits:**
- Significant performance improvement in label operations
- Reduced memory usage through caching
- Cleaner label manipulation APIs
### PRIORITY 4: Enhanced Database Integration
**Goal**: Proper referential integrity and data consistency
**Implementation Plan:**
1. **Database Schema Enhancement**:
- Add issues table with proper constraints
- Implement foreign key relationships
- Create indices for performance
2. **Migration Strategy**:
- Create 002_issue_integration.sql migration
- Data integrity validation
- Rollback procedures
**Expected Benefits:**
- Data integrity through foreign key constraints
- Better query performance with proper indices
- Cleaner database structure
### PRIORITY 5: Repository Pattern Enhancement
**Goal**: High-performance repository with caching
**Implementation Plan:**
1. **Cached Repository Implementation**:
- TTL-based caching layer
- Efficient query patterns
- Bulk operation optimizations
2. **Query Optimization**:
- Label-based filtering improvements
- Pagination and sorting enhancements
- Connection pooling considerations
**Expected Benefits:**
- Improved application performance
- Reduced database load
- Better scalability for larger datasets
### PRIORITY 6: Activity Tracking Integration
**Goal**: Integrated activity tracking with issue lifecycle
**Implementation Plan:**
1. **Enhanced Activity Model**:
- Tight coupling with issue state changes
- Automated activity generation for lifecycle events
- Rich activity querying capabilities
2. **Lifecycle Integration**:
- Automatic activity logging on state changes
- Business logic triggers for activities
- Activity-driven notifications
**Expected Benefits:**
- Complete audit trail for all issue changes
- Automated activity tracking
- Better business intelligence capabilities
## Implementation Phases
### Phase 1: Model Unification (Week 1)
**Scope**: Create unified Issue model with backward compatibility
**Tasks:**
1. Analyze current Issue model usage patterns
2. Design unified Issue model with all required fields
3. Implement backward compatibility adapters
4. Create migration utilities
5. Update unit tests for new model
**Deliverables:**
- Enhanced `domain/issues/models.py` with unified Issue model
- Compatibility adapter classes
- Comprehensive test coverage
- Migration documentation
**Success Criteria:**
- All existing tests pass with adapter layer
- No breaking changes to external interfaces
- Performance benchmarks established
### Phase 2: Performance Optimization (Week 2)
**Scope**: Implement caching and optimize label processing
**Tasks:**
1. Implement cached label categorization
2. Add performance-optimized properties
3. Create repository caching layer
4. Optimize database queries
5. Add performance monitoring
**Deliverables:**
- Optimized label processing with caching
- High-performance repository implementation
- Database query optimizations
- Performance benchmark results
**Success Criteria:**
- 60-80% improvement in label processing performance
- Repository query response time improvements
- Memory usage optimization validated
### Phase 3: Integration Enhancement (Week 3)
**Scope**: Integrate activity tracking and enhance relationships
**Tasks:**
1. Enhance database schema with proper constraints
2. Integrate activity tracking with issue lifecycle
3. Implement cost allocation relationships
4. Add comprehensive validation
5. Create data migration scripts
**Deliverables:**
- Enhanced database schema with foreign keys
- Integrated activity tracking system
- Data validation framework
- Migration scripts and procedures
**Success Criteria:**
- Database integrity constraints properly enforced
- Activity tracking automatically captures all issue changes
- Data validation prevents inconsistent states
### Phase 4: Testing & Migration (Week 4)
**Scope**: Comprehensive testing and production deployment
**Tasks:**
1. Comprehensive integration testing
2. Performance benchmarking
3. Production data migration
4. Documentation updates
5. Training and knowledge transfer
**Deliverables:**
- Complete test suite with integration tests
- Performance benchmark reports
- Production migration procedures
- Updated documentation
- Training materials
**Success Criteria:**
- 100% test coverage for optimized models
- Performance improvements validated in production-like environment
- Successful data migration without data loss
- Team trained on new datamodel capabilities
## Risk Mitigation
### Technical Risks
1. **Backward Compatibility**: Maintain adapter layer during transition
2. **Data Integrity**: Comprehensive validation and migration scripts
3. **Performance Regression**: Extensive benchmarking and monitoring
4. **Complex Dependencies**: Careful dependency analysis and phased rollout
### Mitigation Strategies
1. **Gradual Migration**: Phase-by-phase implementation with rollback points
2. **Comprehensive Testing**: Unit, integration, and performance tests
3. **Monitoring**: Real-time performance and error monitoring
4. **Documentation**: Clear migration guides and troubleshooting procedures
## Success Metrics
### Quantitative Goals
- **Performance**: 60-80% improvement in label processing
- **Code Reduction**: 15-25 lines of code reduction per optimized model
- **Test Reliability**: 90%+ reduction in test failures due to model inconsistencies
- **Query Performance**: 50%+ improvement in database query response times
### Qualitative Goals
- **Maintainability**: Easier to modify and extend issue models
- **Developer Experience**: Cleaner APIs and more intuitive interfaces
- **Data Consistency**: Reliable data integrity across all operations
- **System Reliability**: Reduced bugs due to model inconsistencies
## Resource Requirements
### Development Team
- 1 Senior Developer (Lead optimization implementation)
- 1 Database Specialist (Schema and migration work)
- 1 QA Engineer (Testing and validation)
### Timeline
- **Total Duration**: 4 weeks
- **Effort Estimate**: ~6-8 person-weeks
- **Critical Path**: Phase 1 (Model Unification) → Phase 2 (Performance) → Phase 3 (Integration) → Phase 4 (Testing)
### Infrastructure
- Development and staging environments for testing
- Performance testing tools and monitoring
- Database migration and rollback capabilities
## Next Steps
### Immediate Actions (Next 1-2 Days)
1. **Team Alignment**: Review gameplan with development team
2. **Environment Setup**: Prepare development and testing environments
3. **Baseline Establishment**: Create performance and functionality baselines
4. **Detailed Planning**: Break down Phase 1 tasks into specific work items
### Week 1 Kickoff
1. Begin Phase 1 implementation
2. Set up continuous integration for new model testing
3. Establish performance monitoring baselines
4. Create detailed migration documentation
---
## Appendix
### Related Documents
- [Datamodel Optimization Specialist Agent Documentation](../docs/sub_agents/datamodel_optimizer.md)
- [Datamodel Optimizer Tool](../tools/datamodel_optimizer.py)
- [Issue #126 - IssueActivity Optimization Success Case](../docs/issues/issue_126_analysis.md)
### References
- Clean Architecture Principles
- Domain-Driven Design Patterns
- Database Design Best Practices
- Performance Optimization Techniques
---
*This gameplan provides a comprehensive roadmap for optimizing the issue datamodel while maintaining system stability, performance, and developer productivity. The phased approach ensures risk mitigation while delivering measurable improvements at each stage.*