markitect-main/DIRECTORY_STRUCTURE_OPTIMIZATION_GAMEPLAN.md

Directory Structure Optimization Gameplan

Status: Draft Created: 2025-09-27 Priority: Medium Complexity: High Estimated Duration: 18-25 hours over 3-5 days

Overview

This gameplan outlines the systematic restructuring of the MarkiTect repository to follow modern Python packaging standards and improve maintainability. The migration will be done in phases to minimize disruption and ensure everything continues working.

Current Structure Analysis

Identified Issues

1. Multiple overlapping CLI implementations (markitect/cli.py, cli/, tddai_cli.py)
2. Scattered application logic across markitect/, services/, application/
3. Inconsistent module boundaries - domain logic mixed with infrastructure concerns
4. Duplicate functionality between markitect/ and other modules
5. Documentation scattered at root level cluttering main directory
6. Configuration spread across multiple locations

Current Directory Tree

markitect_project/
├── application/               # Application services (partial)
├── cli/                      # Structured CLI commands
├── config/                   # Configuration management
├── domain/                   # Business logic
├── infrastructure/           # Infrastructure concerns
├── markitect/               # Core library + CLI (mixed concerns)
├── services/                # More application services
├── tddai/                   # TDD workflow tools
├── gitea/                   # External API client
├── tests/                   # Test suite (well organized)
├── docs/                    # Some documentation
├── *.md                     # 10+ documentation files at root
└── scripts (*.sh)           # Setup scripts at root

Target Structure

markitect_project/
├── src/                          # Source code root (modern Python standard)
│   └── markitect/               # Main package
│       ├── __init__.py
│       ├── domain/              # Business logic (move from ./domain/)
│       │   ├── issues/
│       │   ├── projects/
│       │   ├── documents/
│       │   └── workspaces/
│       ├── application/         # Application services (consolidate from ./application/ and ./services/)
│       │   ├── services/
│       │   ├── use_cases/
│       │   └── dto/
│       ├── infrastructure/      # Infrastructure concerns (from ./infrastructure/)
│       │   ├── persistence/     # Repositories
│       │   ├── external/        # External APIs (Gitea)
│       │   ├── logging/
│       │   └── config/
│       ├── interfaces/          # CLI and other interfaces (consolidate ./cli/ and ./markitect/cli.py)
│       │   ├── cli/
│       │   └── api/             # Future web API
│       └── shared/              # Shared utilities
│           ├── exceptions.py
│           ├── types.py
│           └── utils.py
├── tools/                       # Development tools (move ./tddai/ here)
│   ├── tddai/                   # TDD workflow tools
│   └── scripts/                 # Build/deployment scripts
├── docs/                        # All documentation (consolidate root .md files)
│   ├── architecture/
│   ├── development/
│   ├── user-guides/
│   └── planning/                # Move planning docs here
│       ├── roadmap.md
│       ├── features.md
│       └── gameplans/
├── tests/                       # Keep current structure (good)
├── config/                      # Configuration files and templates
├── examples/                    # Usage examples
└── scripts/                     # Project scripts (install-*.sh)

Phase 1: Documentation Cleanup (Low Risk, High Impact)

Estimated Duration: 1-2 hours Goal: Clean up root directory and establish proper documentation structure

1.1 Create Documentation Structure

mkdir -p docs/{architecture,development,user-guides,planning/gameplans}

1.2 Move and Organize Documentation Files

• README.md → stays at root (essential for GitHub)
• CONFIG.md → docs/development/configuration.md
• FEATURES.md → docs/planning/features.md
• ROADMAP.md → docs/planning/roadmap.md
• ERROR_HANDLING_GUIDE.md → docs/development/error-handling.md
• TESTING_ARCHITECTURE_ENHANCEMENT_GAMEPLAN.md → docs/planning/gameplans/testing-architecture.md
• DATA_ACCESS_IMPROVEMENTS_GAMEPLAN.md → docs/planning/gameplans/data-access.md
• DOMAIN_LOGIC_SEPARATION_GAMEPLAN.md → docs/planning/gameplans/domain-logic.md
• DOMAIN_LOGIC_SEPARATION_DEMO.md → docs/architecture/domain-separation-demo.md
• ProjectDiary.md → docs/development/project-diary.md
• ProjectStatusDigest.md → docs/development/status-digest.md
• NEXT.md → docs/planning/next-steps.md
• RelevantClaudeIssues.md → docs/development/claude-issues.md

1.3 Update Internal References

• Update any references to moved files in remaining documentation
• Create a docs/README.md with navigation guide

1.4 Verification

• Ensure all moved files are accessible
• Verify no broken internal links
• Run tests to ensure nothing depends on moved files

Phase 2: Prepare New Structure (Medium Risk)

Estimated Duration: 2-3 hours Goal: Create new directory structure without moving code yet

2.1 Create Source Structure

mkdir -p src/markitect/{domain,application,infrastructure,interfaces,shared}
mkdir -p src/markitect/domain/{issues,projects,documents,workspaces}
mkdir -p src/markitect/application/{services,use_cases,dto}
mkdir -p src/markitect/infrastructure/{persistence,external,logging,config}
mkdir -p src/markitect/interfaces/{cli,api}
mkdir -p src/markitect/shared

2.2 Create Tools Structure

mkdir -p tools/{tddai,scripts}
mkdir -p examples
mkdir -p config/templates

2.3 Create Migration Scripts

Create helper scripts for the migration:

• scripts/migrate-imports.py - Script to update import paths
• scripts/verify-migration.py - Script to verify migration completeness
• scripts/test-all-phases.sh - Script to run tests after each phase

2.4 Backup Current State

git checkout -b migration-backup
git checkout main
git checkout -b feature/directory-restructure

Phase 3: Domain Logic Migration (Medium Risk)

Estimated Duration: 3-4 hours Goal: Move domain logic as it has the fewest dependencies

3.1 Move Domain Modules

# Move domain logic
cp -r domain/* src/markitect/domain/
# Add __init__.py files where needed
find src/markitect/domain -type d -exec touch {}/__init__.py \;

3.2 Update Domain Imports

• Update all imports within domain modules to use new paths
• Update imports in tests that reference domain modules
• Use migration script to automate this process

3.3 Update pyproject.toml (First Update)

[tool.setuptools.packages.find]
where = ["src"]
include = ["markitect*"]
exclude = ["tests*", "tools*"]

3.4 Test Domain Migration

PYTHONPATH=src python -m pytest tests/unit/domain/ -v

3.5 Update Import References

Update all files that import from domain to use new paths:

• application/ modules
• services/ modules
• infrastructure/ modules
• Test files

Phase 4: Infrastructure Migration (Medium Risk)

Estimated Duration: 2-3 hours Goal: Move infrastructure code to new location

4.1 Move Infrastructure Components

# Move infrastructure
cp -r infrastructure/* src/markitect/infrastructure/
# Move Gitea client to external
cp -r gitea/* src/markitect/infrastructure/external/gitea/
# Move config module
cp -r config/* src/markitect/infrastructure/config/

4.2 Organize Infrastructure Submodules

• infrastructure/repositories/ → src/markitect/infrastructure/persistence/
• infrastructure/logging/ → src/markitect/infrastructure/logging/
• gitea/ → src/markitect/infrastructure/external/gitea/
• config/ → src/markitect/infrastructure/config/

4.3 Update Infrastructure Imports

• Update all internal infrastructure imports
• Update imports from other modules that use infrastructure
• Update test files

4.4 Test Infrastructure Migration

PYTHONPATH=src python -m pytest tests/unit/infrastructure/ -v
PYTHONPATH=src python -m pytest tests/integration/ -v

Phase 5: Application Services Consolidation (High Risk)

Estimated Duration: 4-5 hours Goal: Merge application/, services/, and markitect/ into unified application layer

5.1 Analyze Current Application Logic

• Map all services in services/
• Map all application logic in application/
• Map all business logic in markitect/
• Identify overlaps and consolidation opportunities

5.2 Create Unified Application Services

# Move and organize application services
cp -r services/* src/markitect/application/services/
cp -r application/* src/markitect/application/
# Move relevant markitect modules to application
cp markitect/document_manager.py src/markitect/application/services/
cp markitect/cache_service.py src/markitect/application/services/

5.3 Consolidate Core Library Functions

• markitect/parser.py → src/markitect/shared/parser.py
• markitect/serializer.py → src/markitect/shared/serializer.py
• markitect/exceptions.py → src/markitect/shared/exceptions.py
• markitect/database.py → src/markitect/infrastructure/persistence/database.py
• markitect/frontmatter.py → src/markitect/shared/frontmatter.py

5.4 Update Application Imports

• Update all imports to use new application service paths
• Remove duplicate functionality
• Update tests

5.5 Test Application Migration

PYTHONPATH=src python -m pytest tests/ -v --tb=short

Phase 6: CLI Consolidation (High Risk)

Estimated Duration: 3-4 hours Goal: Merge all CLI implementations into single interface

6.1 Analyze CLI Implementations

• markitect/cli.py - Main CLI entry point
• cli/ - Structured CLI commands
• tddai_cli.py - TDD workflow CLI
• Identify overlaps and consolidation strategy

6.2 Create Unified CLI Structure

# Create CLI structure
mkdir -p src/markitect/interfaces/cli/{commands,presenters}
# Move CLI components
cp cli/commands/* src/markitect/interfaces/cli/commands/
cp cli/presenters/* src/markitect/interfaces/cli/presenters/
cp cli/core.py src/markitect/interfaces/cli/
# Integrate markitect CLI
# Move tddai CLI as subcommand

6.3 Update CLI Entry Points

Update pyproject.toml:

[project.scripts]
markitect = "markitect.interfaces.cli:main"
tddai = "markitect.interfaces.cli.tddai:main"

6.4 Test CLI Integration

PYTHONPATH=src python -m markitect.interfaces.cli --help
PYTHONPATH=src python -c "from markitect.interfaces.cli import main; main()"

Phase 7: Tools Migration (Low Risk)

Estimated Duration: 1-2 hours Goal: Move development tools to proper location

7.1 Move TDD Tools

cp -r tddai/* tools/tddai/
cp tddai_cli.py tools/tddai/cli.py

7.2 Move Scripts

mv install-*.sh scripts/
mv tddai-setup.sh scripts/

7.3 Update Tool References

• Update any references to tools in documentation
• Update CI/CD scripts if they reference tools
• Create tool entry points if needed

Phase 8: Final Cleanup and Testing (Medium Risk)

Estimated Duration: 2-3 hours Goal: Complete migration and verify everything works

8.1 Update Package Configuration

Final pyproject.toml updates:

[tool.setuptools.packages.find]
where = ["src"]
include = ["markitect*"]
exclude = ["tests*", "tools*", "docs*"]

[project.scripts]
markitect = "markitect.interfaces.cli:main"

[tool.mypy]
mypy_path = "src"

8.2 Remove Old Directories

# After verifying everything works
rm -rf domain/ infrastructure/ application/ services/ markitect/ cli/ gitea/ config/ tddai/
rm tddai_cli.py

8.3 Update Development Environment

• Update IDE configurations
• Update import settings
• Update any development scripts

8.4 Comprehensive Testing

# Install in development mode
pip install -e .
# Run all tests
PYTHONPATH=src python -m pytest tests/ -v
# Test CLI functionality
markitect --help
markitect list
markitect schema

8.5 Update Documentation

• Update installation instructions
• Update development setup guide
• Update contributor documentation

Phase 9: Verification and Rollback Plan (Critical)

Estimated Duration: 1 hour Goal: Verify migration success and prepare rollback if needed

9.1 Final Verification Checklist

• All tests pass
• CLI commands work
• Package can be installed
• Documentation is accessible
• No import errors
• Performance not degraded

9.2 Rollback Plan

If issues are discovered:

git checkout migration-backup
# Or selectively revert problematic changes
git checkout main -- problematic/path/

9.3 Success Criteria

• Zero test failures
• All CLI commands functional
• Clean pip install -e .
• Documentation updated
• No broken imports

Risk Mitigation Strategies

Before Starting

1. Full test suite must be green - Ensure all tests pass before beginning
2. Create comprehensive backup - Branch with current state
3. Automate import updates - Create scripts to handle bulk import changes
4. Test incrementally - Run tests after each phase

During Migration

1. Phase-by-phase approach - Complete each phase fully before next
2. Continuous testing - Run relevant tests after each major change
3. Import tracking - Keep list of all import changes for rollback
4. Documentation updates - Update docs as you go, not at the end

Emergency Procedures

1. Quick rollback - git checkout migration-backup
2. Partial rollback - git checkout main -- specific/path/
3. Import fixes - Use prepared scripts to bulk-fix imports
4. Test isolation - Ability to test individual components

Benefits of This Structure

Code Quality

1. Standards Compliance: Follows modern Python packaging standards
2. Clear Separation: Domain, application, infrastructure clearly separated
3. Maintainability: Easier to navigate and understand
4. Testability: Better test organization mirrors source structure

Developer Experience

1. Professional Appearance: Clean root directory, organized documentation
2. Scalability: Structure supports future growth (web API, plugins, etc.)
3. Tool Integration: Better IDE support and static analysis
4. Easier Onboarding: Clear structure for new developers

Maintainability

1. Reduced Duplication: Eliminates overlapping functionality
2. Better Dependencies: Clearer dependency boundaries
3. Logical Organization: Related code grouped together
4. Future-Proof: Supports project growth and evolution

Potential Risks

Technical Risks

1. Import Path Changes: All imports will need updating
2. IDE Configuration: Development tools may need reconfiguration
3. CI/CD Updates: Build scripts and workflows need adjustment
4. Documentation References: Internal links will need updating

Process Risks

1. Migration Complexity: Large-scale changes increase error probability
2. Testing Overhead: Need to test thoroughly after each phase
3. Time Investment: Significant time commitment required
4. Rollback Complexity: May be difficult to rollback partial migrations

Success Metrics

Code Quality

• All tests pass (305 passed, 2 skipped, 0 warnings)
• No import errors
• Clean package installation
• Type checking passes

Developer Experience

• Cleaner directory structure
• Logical code organization
• Easier navigation
• Better IDE support

Maintainability

• Clear separation of concerns
• Reduced code duplication
• Better dependency management
• Easier onboarding for new developers

Timeline

Total Estimated Duration: 18-25 hours over 3-5 days

• Day 1: Phases 1-2 (Documentation + Structure Creation) - 3-5 hours
• Day 2: Phases 3-4 (Domain + Infrastructure Migration) - 5-7 hours
• Day 3: Phase 5 (Application Consolidation) - 4-5 hours
• Day 4: Phases 6-7 (CLI + Tools Migration) - 4-5 hours
• Day 5: Phases 8-9 (Cleanup + Verification) - 3 hours

Prerequisites

Technical Prerequisites

1. Green Test Suite: All current tests must pass
2. Clean Git State: No uncommitted changes
3. Backup Strategy: Migration backup branch created
4. Development Environment: Working local development setup

Planning Prerequisites

1. Team Alignment: All team members aware of migration plan
2. Migration Scripts: Helper scripts prepared and tested
3. Documentation Plan: Clear plan for updating documentation
4. Rollback Strategy: Clear rollback procedures defined

Post-Migration Tasks

Immediate (Day 1 after completion)

1. Verification Testing: Comprehensive test suite execution
2. Performance Validation: Ensure no performance degradation
3. Documentation Updates: Update all development documentation
4. Tool Configuration: Update IDE and development tool configurations

Short Term (Week 1)

1. Developer Onboarding: Update onboarding documentation
2. CI/CD Updates: Update build and deployment scripts
3. Monitoring: Monitor for any issues with new structure
4. Feedback Collection: Gather feedback from team members

Long Term (Month 1)

1. Structure Validation: Assess if new structure meets goals
2. Further Optimizations: Identify additional improvements
3. Documentation Completion: Ensure all documentation is complete
4. Best Practices: Document lessons learned for future migrations

---

This gameplan provides a systematic, risk-mitigated approach to restructuring the MarkiTect repository while maintaining functionality and minimizing disruption. The phased approach allows for incremental progress with verification at each step, ensuring the migration can be completed successfully or rolled back if necessary.