chore: Gameplan for repository cleanup

2025-09-27 21:15:14 +02:00
parent 1fa0f1e84a
commit 73ea849ee9
1 changed files with 508 additions and 0 deletions
--- a/DIRECTORY_STRUCTURE_OPTIMIZATION_GAMEPLAN.md
+++ b/DIRECTORY_STRUCTURE_OPTIMIZATION_GAMEPLAN.md
@@ -0,0 +1,508 @@
+# Directory Structure Optimization Gameplan
+
+**Status**: Draft
+**Created**: 2025-09-27
+**Priority**: Medium
+**Complexity**: High
+**Estimated Duration**: 18-25 hours over 3-5 days
+
+## Overview
+This gameplan outlines the systematic restructuring of the MarkiTect repository to follow modern Python packaging standards and improve maintainability. The migration will be done in phases to minimize disruption and ensure everything continues working.
+
+## Current Structure Analysis
+
+### Identified Issues
+1. **Multiple overlapping CLI implementations** (`markitect/cli.py`, `cli/`, `tddai_cli.py`)
+2. **Scattered application logic** across `markitect/`, `services/`, `application/`
+3. **Inconsistent module boundaries** - domain logic mixed with infrastructure concerns
+4. **Duplicate functionality** between `markitect/` and other modules
+5. **Documentation scattered** at root level cluttering main directory
+6. **Configuration spread** across multiple locations
+
+### Current Directory Tree
+```
+markitect_project/
+├── application/               # Application services (partial)
+├── cli/                      # Structured CLI commands
+├── config/                   # Configuration management
+├── domain/                   # Business logic
+├── infrastructure/           # Infrastructure concerns
+├── markitect/               # Core library + CLI (mixed concerns)
+├── services/                # More application services
+├── tddai/                   # TDD workflow tools
+├── gitea/                   # External API client
+├── tests/                   # Test suite (well organized)
+├── docs/                    # Some documentation
+├── *.md                     # 10+ documentation files at root
+└── scripts (*.sh)           # Setup scripts at root
+```
+
+## Target Structure
+
+```
+markitect_project/
+├── src/                          # Source code root (modern Python standard)
+│   └── markitect/               # Main package
+│       ├── __init__.py
+│       ├── domain/              # Business logic (move from ./domain/)
+│       │   ├── issues/
+│       │   ├── projects/
+│       │   ├── documents/
+│       │   └── workspaces/
+│       ├── application/         # Application services (consolidate from ./application/ and ./services/)
+│       │   ├── services/
+│       │   ├── use_cases/
+│       │   └── dto/
+│       ├── infrastructure/      # Infrastructure concerns (from ./infrastructure/)
+│       │   ├── persistence/     # Repositories
+│       │   ├── external/        # External APIs (Gitea)
+│       │   ├── logging/
+│       │   └── config/
+│       ├── interfaces/          # CLI and other interfaces (consolidate ./cli/ and ./markitect/cli.py)
+│       │   ├── cli/
+│       │   └── api/             # Future web API
+│       └── shared/              # Shared utilities
+│           ├── exceptions.py
+│           ├── types.py
+│           └── utils.py
+├── tools/                       # Development tools (move ./tddai/ here)
+│   ├── tddai/                   # TDD workflow tools
+│   └── scripts/                 # Build/deployment scripts
+├── docs/                        # All documentation (consolidate root .md files)
+│   ├── architecture/
+│   ├── development/
+│   ├── user-guides/
+│   └── planning/                # Move planning docs here
+│       ├── roadmap.md
+│       ├── features.md
+│       └── gameplans/
+├── tests/                       # Keep current structure (good)
+├── config/                      # Configuration files and templates
+├── examples/                    # Usage examples
+└── scripts/                     # Project scripts (install-*.sh)
+```
+
+## Phase 1: Documentation Cleanup (Low Risk, High Impact)
+**Estimated Duration**: 1-2 hours
+**Goal**: Clean up root directory and establish proper documentation structure
+
+### 1.1 Create Documentation Structure
+```bash
+mkdir -p docs/{architecture,development,user-guides,planning/gameplans}
+```
+
+### 1.2 Move and Organize Documentation Files
+- `README.md` → stays at root (essential for GitHub)
+- `CONFIG.md` → `docs/development/configuration.md`
+- `FEATURES.md` → `docs/planning/features.md`
+- `ROADMAP.md` → `docs/planning/roadmap.md`
+- `ERROR_HANDLING_GUIDE.md` → `docs/development/error-handling.md`
+- `TESTING_ARCHITECTURE_ENHANCEMENT_GAMEPLAN.md` → `docs/planning/gameplans/testing-architecture.md`
+- `DATA_ACCESS_IMPROVEMENTS_GAMEPLAN.md` → `docs/planning/gameplans/data-access.md`
+- `DOMAIN_LOGIC_SEPARATION_GAMEPLAN.md` → `docs/planning/gameplans/domain-logic.md`
+- `DOMAIN_LOGIC_SEPARATION_DEMO.md` → `docs/architecture/domain-separation-demo.md`
+- `ProjectDiary.md` → `docs/development/project-diary.md`
+- `ProjectStatusDigest.md` → `docs/development/status-digest.md`
+- `NEXT.md` → `docs/planning/next-steps.md`
+- `RelevantClaudeIssues.md` → `docs/development/claude-issues.md`
+
+### 1.3 Update Internal References
+- Update any references to moved files in remaining documentation
+- Create a `docs/README.md` with navigation guide
+
+### 1.4 Verification
+- Ensure all moved files are accessible
+- Verify no broken internal links
+- Run tests to ensure nothing depends on moved files
+
+## Phase 2: Prepare New Structure (Medium Risk)
+**Estimated Duration**: 2-3 hours
+**Goal**: Create new directory structure without moving code yet
+
+### 2.1 Create Source Structure
+```bash
+mkdir -p src/markitect/{domain,application,infrastructure,interfaces,shared}
+mkdir -p src/markitect/domain/{issues,projects,documents,workspaces}
+mkdir -p src/markitect/application/{services,use_cases,dto}
+mkdir -p src/markitect/infrastructure/{persistence,external,logging,config}
+mkdir -p src/markitect/interfaces/{cli,api}
+mkdir -p src/markitect/shared
+```
+
+### 2.2 Create Tools Structure
+```bash
+mkdir -p tools/{tddai,scripts}
+mkdir -p examples
+mkdir -p config/templates
+```
+
+### 2.3 Create Migration Scripts
+Create helper scripts for the migration:
+- `scripts/migrate-imports.py` - Script to update import paths
+- `scripts/verify-migration.py` - Script to verify migration completeness
+- `scripts/test-all-phases.sh` - Script to run tests after each phase
+
+### 2.4 Backup Current State
+```bash
+git checkout -b migration-backup
+git checkout main
+git checkout -b feature/directory-restructure
+```
+
+## Phase 3: Domain Logic Migration (Medium Risk)
+**Estimated Duration**: 3-4 hours
+**Goal**: Move domain logic as it has the fewest dependencies
+
+### 3.1 Move Domain Modules
+```bash
+# Move domain logic
+cp -r domain/* src/markitect/domain/
+# Add __init__.py files where needed
+find src/markitect/domain -type d -exec touch {}/__init__.py \;
+```
+
+### 3.2 Update Domain Imports
+- Update all imports within domain modules to use new paths
+- Update imports in tests that reference domain modules
+- Use migration script to automate this process
+
+### 3.3 Update pyproject.toml (First Update)
+```toml
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["markitect*"]
+exclude = ["tests*", "tools*"]
+```
+
+### 3.4 Test Domain Migration
+```bash
+PYTHONPATH=src python -m pytest tests/unit/domain/ -v
+```
+
+### 3.5 Update Import References
+Update all files that import from domain to use new paths:
+- `application/` modules
+- `services/` modules
+- `infrastructure/` modules
+- Test files
+
+## Phase 4: Infrastructure Migration (Medium Risk)
+**Estimated Duration**: 2-3 hours
+**Goal**: Move infrastructure code to new location
+
+### 4.1 Move Infrastructure Components
+```bash
+# Move infrastructure
+cp -r infrastructure/* src/markitect/infrastructure/
+# Move Gitea client to external
+cp -r gitea/* src/markitect/infrastructure/external/gitea/
+# Move config module
+cp -r config/* src/markitect/infrastructure/config/
+```
+
+### 4.2 Organize Infrastructure Submodules
+- `infrastructure/repositories/` → `src/markitect/infrastructure/persistence/`
+- `infrastructure/logging/` → `src/markitect/infrastructure/logging/`
+- `gitea/` → `src/markitect/infrastructure/external/gitea/`
+- `config/` → `src/markitect/infrastructure/config/`
+
+### 4.3 Update Infrastructure Imports
+- Update all internal infrastructure imports
+- Update imports from other modules that use infrastructure
+- Update test files
+
+### 4.4 Test Infrastructure Migration
+```bash
+PYTHONPATH=src python -m pytest tests/unit/infrastructure/ -v
+PYTHONPATH=src python -m pytest tests/integration/ -v
+```
+
+## Phase 5: Application Services Consolidation (High Risk)
+**Estimated Duration**: 4-5 hours
+**Goal**: Merge application/, services/, and markitect/ into unified application layer
+
+### 5.1 Analyze Current Application Logic
+- Map all services in `services/`
+- Map all application logic in `application/`
+- Map all business logic in `markitect/`
+- Identify overlaps and consolidation opportunities
+
+### 5.2 Create Unified Application Services
+```bash
+# Move and organize application services
+cp -r services/* src/markitect/application/services/
+cp -r application/* src/markitect/application/
+# Move relevant markitect modules to application
+cp markitect/document_manager.py src/markitect/application/services/
+cp markitect/cache_service.py src/markitect/application/services/
+```
+
+### 5.3 Consolidate Core Library Functions
+- `markitect/parser.py` → `src/markitect/shared/parser.py`
+- `markitect/serializer.py` → `src/markitect/shared/serializer.py`
+- `markitect/exceptions.py` → `src/markitect/shared/exceptions.py`
+- `markitect/database.py` → `src/markitect/infrastructure/persistence/database.py`
+- `markitect/frontmatter.py` → `src/markitect/shared/frontmatter.py`
+
+### 5.4 Update Application Imports
+- Update all imports to use new application service paths
+- Remove duplicate functionality
+- Update tests
+
+### 5.5 Test Application Migration
+```bash
+PYTHONPATH=src python -m pytest tests/ -v --tb=short
+```
+
+## Phase 6: CLI Consolidation (High Risk)
+**Estimated Duration**: 3-4 hours
+**Goal**: Merge all CLI implementations into single interface
+
+### 6.1 Analyze CLI Implementations
+- `markitect/cli.py` - Main CLI entry point
+- `cli/` - Structured CLI commands
+- `tddai_cli.py` - TDD workflow CLI
+- Identify overlaps and consolidation strategy
+
+### 6.2 Create Unified CLI Structure
+```bash
+# Create CLI structure
+mkdir -p src/markitect/interfaces/cli/{commands,presenters}
+# Move CLI components
+cp cli/commands/* src/markitect/interfaces/cli/commands/
+cp cli/presenters/* src/markitect/interfaces/cli/presenters/
+cp cli/core.py src/markitect/interfaces/cli/
+# Integrate markitect CLI
+# Move tddai CLI as subcommand
+```
+
+### 6.3 Update CLI Entry Points
+Update `pyproject.toml`:
+```toml
+[project.scripts]
+markitect = "markitect.interfaces.cli:main"
+tddai = "markitect.interfaces.cli.tddai:main"
+```
+
+### 6.4 Test CLI Integration
+```bash
+PYTHONPATH=src python -m markitect.interfaces.cli --help
+PYTHONPATH=src python -c "from markitect.interfaces.cli import main; main()"
+```
+
+## Phase 7: Tools Migration (Low Risk)
+**Estimated Duration**: 1-2 hours
+**Goal**: Move development tools to proper location
+
+### 7.1 Move TDD Tools
+```bash
+cp -r tddai/* tools/tddai/
+cp tddai_cli.py tools/tddai/cli.py
+```
+
+### 7.2 Move Scripts
+```bash
+mv install-*.sh scripts/
+mv tddai-setup.sh scripts/
+```
+
+### 7.3 Update Tool References
+- Update any references to tools in documentation
+- Update CI/CD scripts if they reference tools
+- Create tool entry points if needed
+
+## Phase 8: Final Cleanup and Testing (Medium Risk)
+**Estimated Duration**: 2-3 hours
+**Goal**: Complete migration and verify everything works
+
+### 8.1 Update Package Configuration
+Final `pyproject.toml` updates:
+```toml
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["markitect*"]
+exclude = ["tests*", "tools*", "docs*"]
+
+[project.scripts]
+markitect = "markitect.interfaces.cli:main"
+
+[tool.mypy]
+mypy_path = "src"
+```
+
+### 8.2 Remove Old Directories
+```bash
+# After verifying everything works
+rm -rf domain/ infrastructure/ application/ services/ markitect/ cli/ gitea/ config/ tddai/
+rm tddai_cli.py
+```
+
+### 8.3 Update Development Environment
+- Update IDE configurations
+- Update import settings
+- Update any development scripts
+
+### 8.4 Comprehensive Testing
+```bash
+# Install in development mode
+pip install -e .
+# Run all tests
+PYTHONPATH=src python -m pytest tests/ -v
+# Test CLI functionality
+markitect --help
+markitect list
+markitect schema
+```
+
+### 8.5 Update Documentation
+- Update installation instructions
+- Update development setup guide
+- Update contributor documentation
+
+## Phase 9: Verification and Rollback Plan (Critical)
+**Estimated Duration**: 1 hour
+**Goal**: Verify migration success and prepare rollback if needed
+
+### 9.1 Final Verification Checklist
+- [ ] All tests pass
+- [ ] CLI commands work
+- [ ] Package can be installed
+- [ ] Documentation is accessible
+- [ ] No import errors
+- [ ] Performance not degraded
+
+### 9.2 Rollback Plan
+If issues are discovered:
+```bash
+git checkout migration-backup
+# Or selectively revert problematic changes
+git checkout main -- problematic/path/
+```
+
+### 9.3 Success Criteria
+- Zero test failures
+- All CLI commands functional
+- Clean `pip install -e .`
+- Documentation updated
+- No broken imports
+
+## Risk Mitigation Strategies
+
+### Before Starting
+1. **Full test suite must be green** - Ensure all tests pass before beginning
+2. **Create comprehensive backup** - Branch with current state
+3. **Automate import updates** - Create scripts to handle bulk import changes
+4. **Test incrementally** - Run tests after each phase
+
+### During Migration
+1. **Phase-by-phase approach** - Complete each phase fully before next
+2. **Continuous testing** - Run relevant tests after each major change
+3. **Import tracking** - Keep list of all import changes for rollback
+4. **Documentation updates** - Update docs as you go, not at the end
+
+### Emergency Procedures
+1. **Quick rollback** - `git checkout migration-backup`
+2. **Partial rollback** - `git checkout main -- specific/path/`
+3. **Import fixes** - Use prepared scripts to bulk-fix imports
+4. **Test isolation** - Ability to test individual components
+
+## Benefits of This Structure
+
+### Code Quality
+1. **Standards Compliance**: Follows modern Python packaging standards
+2. **Clear Separation**: Domain, application, infrastructure clearly separated
+3. **Maintainability**: Easier to navigate and understand
+4. **Testability**: Better test organization mirrors source structure
+
+### Developer Experience
+1. **Professional Appearance**: Clean root directory, organized documentation
+2. **Scalability**: Structure supports future growth (web API, plugins, etc.)
+3. **Tool Integration**: Better IDE support and static analysis
+4. **Easier Onboarding**: Clear structure for new developers
+
+### Maintainability
+1. **Reduced Duplication**: Eliminates overlapping functionality
+2. **Better Dependencies**: Clearer dependency boundaries
+3. **Logical Organization**: Related code grouped together
+4. **Future-Proof**: Supports project growth and evolution
+
+## Potential Risks
+
+### Technical Risks
+1. **Import Path Changes**: All imports will need updating
+2. **IDE Configuration**: Development tools may need reconfiguration
+3. **CI/CD Updates**: Build scripts and workflows need adjustment
+4. **Documentation References**: Internal links will need updating
+
+### Process Risks
+1. **Migration Complexity**: Large-scale changes increase error probability
+2. **Testing Overhead**: Need to test thoroughly after each phase
+3. **Time Investment**: Significant time commitment required
+4. **Rollback Complexity**: May be difficult to rollback partial migrations
+
+## Success Metrics
+
+### Code Quality
+- All tests pass (305 passed, 2 skipped, 0 warnings)
+- No import errors
+- Clean package installation
+- Type checking passes
+
+### Developer Experience
+- Cleaner directory structure
+- Logical code organization
+- Easier navigation
+- Better IDE support
+
+### Maintainability
+- Clear separation of concerns
+- Reduced code duplication
+- Better dependency management
+- Easier onboarding for new developers
+
+## Timeline
+**Total Estimated Duration**: 18-25 hours over 3-5 days
+
+- **Day 1**: Phases 1-2 (Documentation + Structure Creation) - 3-5 hours
+- **Day 2**: Phases 3-4 (Domain + Infrastructure Migration) - 5-7 hours
+- **Day 3**: Phase 5 (Application Consolidation) - 4-5 hours
+- **Day 4**: Phases 6-7 (CLI + Tools Migration) - 4-5 hours
+- **Day 5**: Phases 8-9 (Cleanup + Verification) - 3 hours
+
+## Prerequisites
+
+### Technical Prerequisites
+1. **Green Test Suite**: All current tests must pass
+2. **Clean Git State**: No uncommitted changes
+3. **Backup Strategy**: Migration backup branch created
+4. **Development Environment**: Working local development setup
+
+### Planning Prerequisites
+1. **Team Alignment**: All team members aware of migration plan
+2. **Migration Scripts**: Helper scripts prepared and tested
+3. **Documentation Plan**: Clear plan for updating documentation
+4. **Rollback Strategy**: Clear rollback procedures defined
+
+## Post-Migration Tasks
+
+### Immediate (Day 1 after completion)
+1. **Verification Testing**: Comprehensive test suite execution
+2. **Performance Validation**: Ensure no performance degradation
+3. **Documentation Updates**: Update all development documentation
+4. **Tool Configuration**: Update IDE and development tool configurations
+
+### Short Term (Week 1)
+1. **Developer Onboarding**: Update onboarding documentation
+2. **CI/CD Updates**: Update build and deployment scripts
+3. **Monitoring**: Monitor for any issues with new structure
+4. **Feedback Collection**: Gather feedback from team members
+
+### Long Term (Month 1)
+1. **Structure Validation**: Assess if new structure meets goals
+2. **Further Optimizations**: Identify additional improvements
+3. **Documentation Completion**: Ensure all documentation is complete
+4. **Best Practices**: Document lessons learned for future migrations
+
+---
+
+This gameplan provides a systematic, risk-mitigated approach to restructuring the MarkiTect repository while maintaining functionality and minimizing disruption. The phased approach allows for incremental progress with verification at each step, ensuring the migration can be completed successfully or rolled back if necessary.