423 lines
16 KiB
Markdown
423 lines
16 KiB
Markdown
# MarkiTect Test Architecture & Execution Strategy
|
|
|
|
> **Architectural test organization and reverse dependency execution order**
|
|
|
|
This document defines the test architecture based on the 7-layer system architecture, establishing clear naming conventions and optimal execution order for faster feedback and reduced debugging time.
|
|
|
|
## 🏗️ Test Architecture Mapping
|
|
|
|
### Layer-Based Test Organization
|
|
|
|
Tests are organized by architectural layer with execution order optimized for dependency flow:
|
|
|
|
```
|
|
Test Execution Order (Foundation → Presentation):
|
|
Foundation → Infrastructure → Integration → Domain → Service → Application → Presentation
|
|
↑ ↑ ↑ ↑ ↑ ↑ ↑
|
|
Fastest Cache Tests Gitea API Business Service Workflows CLI Tests
|
|
Feedback DB Tests Tests Logic Tests Tests E2E Tests
|
|
```
|
|
|
|
---
|
|
|
|
## 📋 Current Test Files → New Architecture Mapping
|
|
|
|
### **Layer 7: Foundation Tests** (Execute First - Fastest Feedback)
|
|
*Core technology and utility validation*
|
|
|
|
| Current Test File | New Test File | Test Focus | Execution Priority |
|
|
|-------------------|---------------|------------|-------------------|
|
|
| `test_parser.py` | `test_l7_foundation_markdown_parsing.py` | Core markdown parsing engine | **1** |
|
|
| `test_issue_1_database_initialization.py` | `test_l7_foundation_database_core.py` | SQLite database initialization | **2** |
|
|
|
|
**Rationale**: Foundation layer failures break everything downstream. Test these first for immediate feedback.
|
|
|
|
---
|
|
|
|
### **Layer 5: Infrastructure Tests** (Execute Second - Core Systems)
|
|
*Technical capabilities and system resources*
|
|
|
|
| Current Test File | New Test File | Test Focus | Execution Priority |
|
|
|-------------------|---------------|------------|-------------------|
|
|
| `test_issue_2_file_ingestion.py` | `test_l5_infrastructure_ast_processing.py` | AST processing and caching | **3** |
|
|
| `test_issue_13_cache_commands.py` | `test_l5_infrastructure_cache_management.py` | Cache system operations | **4** |
|
|
| `test_issue_13_cache_info_command.py` | `test_l5_infrastructure_cache_monitoring.py` | Cache statistics and monitoring | **5** |
|
|
| `test_issue_14_query_commands.py` | `test_l5_infrastructure_database_queries.py` | Database query interface | **6** |
|
|
| `test_config_cli_commands.py` | `test_l5_infrastructure_configuration.py` | Configuration management | **7** |
|
|
| `unit/infrastructure/test_testing_infrastructure.py` | `test_l5_infrastructure_test_framework.py` | Testing infrastructure | **8** |
|
|
|
|
**Rationale**: Infrastructure provides technical foundation for all business logic. Failures here cascade widely.
|
|
|
|
---
|
|
|
|
### **Layer 6: Integration Tests** (Execute Third - External Dependencies)
|
|
*External system integration and APIs*
|
|
|
|
| Current Test File | New Test File | Test Focus | Execution Priority |
|
|
|-------------------|---------------|------------|-------------------|
|
|
| `test_gitea_facade.py` | `test_l6_integration_gitea_api.py` | Gitea API client functionality | **9** |
|
|
| `test_issue_creator.py` | `test_l6_integration_issue_creation.py` | Issue creation via external APIs | **10** |
|
|
|
|
**Rationale**: Integration failures are often environmental. Test after core systems are validated.
|
|
|
|
---
|
|
|
|
### **Layer 3: Domain Tests** (Execute Fourth - Business Logic)
|
|
*Pure business logic validation*
|
|
|
|
| Current Test File | New Test File | Test Focus | Execution Priority |
|
|
|-------------------|---------------|------------|-------------------|
|
|
| `unit/domain/issues/test_issue_models.py` | `test_l3_domain_issue_models.py` | Issue domain models and business rules | **11** |
|
|
| `unit/domain/issues/test_issue_services.py` | `test_l3_domain_issue_services.py` | Issue business logic services | **12** |
|
|
| `unit/domain/projects/test_project_models.py` | `test_l3_domain_project_models.py` | Project domain models and logic | **13** |
|
|
| `e2e/performance/test_domain_performance.py` | `test_l3_domain_performance_validation.py` | Domain logic performance characteristics | **14** |
|
|
|
|
**Rationale**: Domain logic is independent of technical concerns. Test after infrastructure is stable.
|
|
|
|
---
|
|
|
|
### **Layer 4: Service Tests** (Execute Fifth - Application Services)
|
|
*Cross-cutting concerns and service coordination*
|
|
|
|
| Current Test File | New Test File | Test Focus | Execution Priority |
|
|
|-------------------|---------------|------------|-------------------|
|
|
| `test_issue_4_retrieve_all_files.py` | `test_l4_service_document_management.py` | Document service operations | **15** |
|
|
| `test_issue_2_get_modify_commands.py` | `test_l4_service_document_modification.py` | Document modification services | **16** |
|
|
| `test_issue_15_ast_commands.py` | `test_l4_service_ast_analysis.py` | AST analysis services | **17** |
|
|
| `test_issue_14_output_formatting.py` | `test_l4_service_output_formatting.py` | Output formatting services | **18** |
|
|
|
|
**Rationale**: Services coordinate domain and infrastructure. Test after both layers are stable.
|
|
|
|
---
|
|
|
|
### **Layer 2: Application Tests** (Execute Sixth - Use Cases & Workflows)
|
|
*Workflow orchestration and use case validation*
|
|
|
|
| Current Test File | New Test File | Test Focus | Execution Priority |
|
|
|-------------------|---------------|------------|-------------------|
|
|
| `test_issue_11_workspace_creation.py` | `test_l2_application_workspace_workflows.py` | TDD workspace management workflows | **19** |
|
|
| `test_issue_11_workspace_creation_validation.py` | `test_l2_application_workspace_validation.py` | Workspace workflow validation | **20** |
|
|
| `test_issue_11_workflow_integration.py` | `test_l2_application_tdd_workflows.py` | Complete TDD workflow integration | **21** |
|
|
| `test_issue_11_feature.py` | `test_l2_application_feature_workflows.py` | Feature development workflows | **22** |
|
|
|
|
**Rationale**: Applications orchestrate multiple services. Test after service layer is validated.
|
|
|
|
---
|
|
|
|
### **Layer 1: Presentation Tests** (Execute Last - User Interface)
|
|
*CLI and user interaction validation*
|
|
|
|
| Current Test File | New Test File | Test Focus | Execution Priority |
|
|
|-------------------|---------------|------------|-------------------|
|
|
| `e2e/cli/test_issue_commands_e2e.py` | `test_l1_presentation_cli_interface.py` | End-to-end CLI command testing | **23** |
|
|
|
|
**Rationale**: Presentation layer depends on all other layers. Test last for comprehensive integration validation.
|
|
|
|
---
|
|
|
|
## 🚀 Test Execution Configuration
|
|
|
|
### Reverse Dependency Order Execution
|
|
|
|
```bash
|
|
# Execute in dependency order for optimal feedback
|
|
pytest tests/test_l7_foundation_*.py # Foundation (fastest)
|
|
pytest tests/test_l5_infrastructure_*.py # Infrastructure
|
|
pytest tests/test_l6_integration_*.py # Integration
|
|
pytest tests/test_l3_domain_*.py # Domain
|
|
pytest tests/test_l4_service_*.py # Service
|
|
pytest tests/test_l2_application_*.py # Application
|
|
pytest tests/test_l1_presentation_*.py # Presentation (slowest)
|
|
```
|
|
|
|
### pytest.ini Configuration
|
|
|
|
Create optimized test execution configuration:
|
|
|
|
```ini
|
|
[tool:pytest]
|
|
testpaths = tests
|
|
python_files = test_*.py
|
|
python_classes = Test*
|
|
python_functions = test_*
|
|
|
|
# Test execution order optimization
|
|
collect_ignore = ["setup.py"]
|
|
addopts =
|
|
--strict-markers
|
|
--strict-config
|
|
--verbose
|
|
--tb=short
|
|
--durations=10
|
|
|
|
# Custom test execution order
|
|
markers =
|
|
foundation: Foundation layer tests (execute first)
|
|
infrastructure: Infrastructure layer tests
|
|
integration: Integration layer tests
|
|
domain: Domain layer tests
|
|
service: Service layer tests
|
|
application: Application layer tests
|
|
presentation: Presentation layer tests (execute last)
|
|
slow: Slow tests requiring longer timeout
|
|
performance: Performance validation tests
|
|
```
|
|
|
|
### Custom Test Runner Script
|
|
|
|
```python
|
|
#!/usr/bin/env python3
|
|
"""
|
|
Architectural test runner with dependency-optimized execution order.
|
|
"""
|
|
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
def run_layer_tests(layer_name: str, pattern: str) -> bool:
|
|
"""Run tests for a specific architectural layer."""
|
|
print(f"\n🧪 Testing {layer_name} Layer...")
|
|
print("=" * 50)
|
|
|
|
cmd = [
|
|
"python", "-m", "pytest",
|
|
f"tests/{pattern}",
|
|
"-v", "--tb=short",
|
|
f"--durations=5"
|
|
]
|
|
|
|
result = subprocess.run(cmd)
|
|
success = result.returncode == 0
|
|
|
|
if success:
|
|
print(f"✅ {layer_name} layer tests PASSED")
|
|
else:
|
|
print(f"❌ {layer_name} layer tests FAILED")
|
|
|
|
return success
|
|
|
|
def main():
|
|
"""Execute tests in reverse dependency order."""
|
|
layers = [
|
|
("Foundation", "test_l7_foundation_*.py"),
|
|
("Infrastructure", "test_l5_infrastructure_*.py"),
|
|
("Integration", "test_l6_integration_*.py"),
|
|
("Domain", "test_l3_domain_*.py"),
|
|
("Service", "test_l4_service_*.py"),
|
|
("Application", "test_l2_application_*.py"),
|
|
("Presentation", "test_l1_presentation_*.py")
|
|
]
|
|
|
|
print("🏗️ MarkiTect Architectural Test Runner")
|
|
print("Executing tests in reverse dependency order...")
|
|
|
|
failed_layers = []
|
|
|
|
for layer_name, pattern in layers:
|
|
if not run_layer_tests(layer_name, pattern):
|
|
failed_layers.append(layer_name)
|
|
print(f"\n⚠️ Stopping at {layer_name} layer failure")
|
|
break
|
|
|
|
if failed_layers:
|
|
print(f"\n❌ Test execution stopped at layer: {failed_layers[0]}")
|
|
print("Fix foundation issues before testing dependent layers.")
|
|
sys.exit(1)
|
|
else:
|
|
print("\n✅ All architectural layers passed!")
|
|
print("🎉 System is architecturally sound!")
|
|
|
|
if __name__ == "__main__":
|
|
main()
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Test Execution Benefits
|
|
|
|
### Performance Optimization
|
|
|
|
| **Strategy** | **Time Savings** | **Debugging Efficiency** |
|
|
|-------------|-------------------|---------------------------|
|
|
| **Foundation First** | 60-80% faster feedback | Fix root causes immediately |
|
|
| **Layer Isolation** | 40-60% reduced test time | Clear failure boundaries |
|
|
| **Dependency Order** | 70-90% fewer cascade failures | Targeted fixes |
|
|
| **Early Termination** | 80%+ time saved on foundation failures | Stop at first architectural break |
|
|
|
|
### Development Workflow Benefits
|
|
|
|
1. **Immediate Feedback**: Foundation failures caught in < 5 seconds
|
|
2. **Targeted Debugging**: Clear layer-specific failure isolation
|
|
3. **Reduced Context Switching**: Fix architectural layers systematically
|
|
4. **Confidence Building**: Green foundation = stable architecture
|
|
5. **Parallel Development**: Teams can work on different layers safely
|
|
|
|
---
|
|
|
|
## 🔄 Migration Plan
|
|
|
|
### Phase 1: Test File Renaming (1-2 days)
|
|
|
|
```bash
|
|
# Foundation Layer
|
|
mv test_parser.py test_l7_foundation_markdown_parsing.py
|
|
mv test_issue_1_database_initialization.py test_l7_foundation_database_core.py
|
|
|
|
# Infrastructure Layer
|
|
mv test_issue_2_file_ingestion.py test_l5_infrastructure_ast_processing.py
|
|
mv test_issue_13_cache_commands.py test_l5_infrastructure_cache_management.py
|
|
mv test_config_cli_commands.py test_l5_infrastructure_configuration.py
|
|
|
|
# Integration Layer
|
|
mv test_gitea_facade.py test_l6_integration_gitea_api.py
|
|
mv test_issue_creator.py test_l6_integration_issue_creation.py
|
|
|
|
# Domain Layer
|
|
mv unit/domain/issues/test_issue_models.py test_l3_domain_issue_models.py
|
|
mv unit/domain/issues/test_issue_services.py test_l3_domain_issue_services.py
|
|
mv unit/domain/projects/test_project_models.py test_l3_domain_project_models.py
|
|
|
|
# Service Layer
|
|
mv test_issue_4_retrieve_all_files.py test_l4_service_document_management.py
|
|
mv test_issue_15_ast_commands.py test_l4_service_ast_analysis.py
|
|
|
|
# Application Layer
|
|
mv test_issue_11_workspace_creation.py test_l2_application_workspace_workflows.py
|
|
mv test_issue_11_workflow_integration.py test_l2_application_tdd_workflows.py
|
|
|
|
# Presentation Layer
|
|
mv e2e/cli/test_issue_commands_e2e.py test_l1_presentation_cli_interface.py
|
|
```
|
|
|
|
### Phase 2: Test Configuration Setup (1 day)
|
|
|
|
1. Update `pytest.ini` with execution order markers
|
|
2. Create architectural test runner script
|
|
3. Update CI/CD pipeline configuration
|
|
4. Create test execution documentation
|
|
|
|
### Phase 3: Team Training (1 day)
|
|
|
|
1. Document new test architecture
|
|
2. Train team on execution strategy
|
|
3. Update development workflow docs
|
|
4. Create architectural test guidelines
|
|
|
|
---
|
|
|
|
## 🎯 Quality Gates
|
|
|
|
### Layer-Specific Quality Requirements
|
|
|
|
| **Layer** | **Coverage Target** | **Performance Target** | **Failure Rate Target** |
|
|
|-----------|-------------------|------------------------|-------------------------|
|
|
| **Foundation** | 98%+ | < 5s execution | < 0.1% failure rate |
|
|
| **Infrastructure** | 95%+ | < 15s execution | < 0.5% failure rate |
|
|
| **Integration** | 85%+ | < 30s execution | < 2% failure rate |
|
|
| **Domain** | 98%+ | < 10s execution | < 0.2% failure rate |
|
|
| **Service** | 90%+ | < 20s execution | < 1% failure rate |
|
|
| **Application** | 85%+ | < 45s execution | < 2% failure rate |
|
|
| **Presentation** | 80%+ | < 60s execution | < 5% failure rate |
|
|
|
|
### Architectural Health Metrics
|
|
|
|
1. **Foundation Stability**: > 99.5% pass rate
|
|
2. **Layer Isolation**: Zero upward dependency failures
|
|
3. **Execution Time**: < 3 minutes for full architectural test suite
|
|
4. **Debugging Efficiency**: Average fix time < 15 minutes per layer
|
|
|
|
---
|
|
|
|
## 📝 Architectural Test Patterns
|
|
|
|
### Foundation Layer Pattern
|
|
```python
|
|
# test_l7_foundation_markdown_parsing.py
|
|
class TestFoundationMarkdownParsing:
|
|
"""Foundation layer: Core markdown parsing engine validation."""
|
|
|
|
def test_markdown_parser_converts_heading_and_paragraph_to_ast_tokens(self):
|
|
"""FOUNDATION: Validate core parsing capability."""
|
|
# Fast, isolated, no dependencies
|
|
```
|
|
|
|
### Infrastructure Layer Pattern
|
|
```python
|
|
# test_l5_infrastructure_cache_management.py
|
|
class TestInfrastructureCacheManagement:
|
|
"""Infrastructure layer: Cache system technical capabilities."""
|
|
|
|
def test_cache_info_command_works_with_empty_and_populated_cache(self):
|
|
"""INFRASTRUCTURE: Validate cache system operations."""
|
|
# Technical capability, depends on foundation
|
|
```
|
|
|
|
### Domain Layer Pattern
|
|
```python
|
|
# test_l3_domain_issue_models.py
|
|
class TestDomainIssueModels:
|
|
"""Domain layer: Pure business logic validation."""
|
|
|
|
def test_issue_creation_with_valid_data(self):
|
|
"""DOMAIN: Validate business rules and logic."""
|
|
# No infrastructure dependencies, pure logic
|
|
```
|
|
|
|
---
|
|
|
|
## 🚦 Continuous Integration Integration
|
|
|
|
### GitHub Actions Workflow
|
|
|
|
```yaml
|
|
name: Architectural Test Suite
|
|
|
|
on: [push, pull_request]
|
|
|
|
jobs:
|
|
architectural-tests:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v3
|
|
- name: Set up Python
|
|
uses: actions/setup-python@v4
|
|
with:
|
|
python-version: '3.12'
|
|
|
|
- name: Install dependencies
|
|
run: |
|
|
pip install -r requirements.txt
|
|
pip install -r requirements-test.txt
|
|
|
|
- name: Run Foundation Tests (Layer 7)
|
|
run: pytest tests/test_l7_foundation_*.py -v
|
|
|
|
- name: Run Infrastructure Tests (Layer 5)
|
|
run: pytest tests/test_l5_infrastructure_*.py -v
|
|
if: success()
|
|
|
|
- name: Run Integration Tests (Layer 6)
|
|
run: pytest tests/test_l6_integration_*.py -v
|
|
if: success()
|
|
|
|
- name: Run Domain Tests (Layer 3)
|
|
run: pytest tests/test_l3_domain_*.py -v
|
|
if: success()
|
|
|
|
- name: Run Service Tests (Layer 4)
|
|
run: pytest tests/test_l4_service_*.py -v
|
|
if: success()
|
|
|
|
- name: Run Application Tests (Layer 2)
|
|
run: pytest tests/test_l2_application_*.py -v
|
|
if: success()
|
|
|
|
- name: Run Presentation Tests (Layer 1)
|
|
run: pytest tests/test_l1_presentation_*.py -v
|
|
if: success()
|
|
```
|
|
|
|
---
|
|
|
|
*This architectural test strategy transforms testing from a time-consuming bottleneck into a fast, targeted debugging tool that respects system architecture and optimizes developer productivity.* |