direkt-vermittlung-de/docs/WORKPLAN_MainCodebase_Integration.md

Workplan: Main Codebase Integration & TDD Setup

Status: Draft v1.0 Date: 2025-12-01 Goal: Consolidate prototype learnings into a production-ready main codebase with TDD practices and proper ADR documentation structure.

---

1. Current State Assessment

Existing Assets

• 3 Prototype Implementations: chatgpt5 (most complete), geminiNbt3pro, grok4.1
• Documentation:
  • Single decisions.md file with all ADRs (needs restructuring)
  • implementation_guide.md (comprehensive)
  • api_docs.md (scenario-based)
  • openapi.yaml (API spec)
• Architecture: Clean Architecture pattern established in chatgpt5 prototype

Gaps

• No unified main codebase
• ADRs not in individual files (not easily referenceable)
• No TDD test suite defining core interfaces
• No CI/CD pipeline configuration
• Prototypes have overlapping implementations without consolidation

---

2. Target State Definition

Main Codebase Structure

/
├── docs/
│   ├── architecture/
│   │   ├── adr/                    # Individual ADR files
│   │   │   ├── 0001-split-payload-model.md
│   │   │   ├── 0002-stateless-auth.md
│   │   │   └── ...
│   │   ├── architecture-overview.md
│   │   └── design-patterns.md
│   ├── api/
│   │   ├── openapi.yaml
│   │   └── api-scenarios.md
│   └── development/
│       ├── testing-strategy.md
│       └── agentic-coding-guide.md
├── src/
│   ├── domain/          # Pure business logic (TDD tested)
│   ├── adapters/        # External integrations (mocked in tests)
│   ├── service/         # Application services (TDD tested)
│   ├── api/             # FastAPI routes (integration tested)
│   └── workers/         # Background jobs
├── tests/
│   ├── unit/            # TDD unit tests
│   ├── integration/     # Integration tests
│   └── fixtures/        # Test data and mocks
├── scripts/
│   └── init_db.py
├── pyproject.toml       # Dependencies & tool config
├── pytest.ini           # Test configuration
├── .github/
│   └── workflows/       # CI/CD pipelines
└── CLAUDE.md            # AI coding assistant guidance

TDD Approach

• Interface-first: Define Pydantic models and service interfaces via tests
• Red-Green-Refactor: Write failing test → implement → refactor
• Test pyramid: Many unit tests, fewer integration tests, minimal e2e tests
• Async testing: Use pytest-asyncio for async operations

ADR Documentation Standards

• Format: Follow Markdown Any Decision Records (MADR) template
• Naming: NNNN-title-with-dashes.md (e.g., 0001-split-payload-model.md)
• Location: /docs/architecture/adr/
• Template:

  # ADR-NNNN: [Title]
  
  **Status:** [Accepted|Proposed|Deprecated|Superseded]
  **Date:** YYYY-MM-DD
  **Deciders:** [Team/Role]
  
  ## Context and Problem Statement
  [What is the issue we're addressing?]
  
  ## Decision Drivers
  * [Driver 1]
  * [Driver 2]
  
  ## Considered Options
  * Option 1
  * Option 2
  
  ## Decision Outcome
  Chosen option: [option], because [rationale].
  
  ### Positive Consequences
  * [Consequence 1]
  
  ### Negative Consequences
  * [Consequence 1]
  
  ## Implementation Notes
  [Specific technical guidance]
  

---

3. Phase 1: Foundation Setup (Week 1)

3.1 ADR Restructuring

Goal: Extract individual ADRs from decisions.md into separate files.

Tasks:

• Create /docs/architecture/adr/ directory
• Create ADR template file: docs/architecture/adr/0000-template.md
• Extract ADR-001 (Split-Payload Model) → 0001-split-payload-model.md
• Extract ADR-002 (Stateless Auth) → 0002-stateless-authentication.md
• Extract ADR-003 (Pagination) → 0003-cursor-based-pagination.md
• Extract ADR-004 (Async Exports) → 0004-async-export-workflow.md
• Extract ADR-005 (Resource Naming) → 0005-rest-resource-structure.md
• Extract ADR-006 (Data Retention) → 0006-gdpr-retention-model.md
• Extract ADR-007 (Python & ProcessPoolExecutor) → 0007-hybrid-concurrency-pattern.md
• Create index file: docs/architecture/adr/README.md with ADR list and links
• Update decisions.md with deprecation notice and redirect to ADR directory

Acceptance Criteria:

• Each ADR is a standalone markdown file
• ADRs follow consistent template structure
• Index file allows easy navigation
• CLAUDE.md updated to reference new ADR location

3.2 Main Codebase Scaffolding

Goal: Create production-ready directory structure with tooling.

Tasks:

• Create /src/ directory with Clean Architecture structure
• Initialize pyproject.toml with dependencies from prototype-chatgpt5
• Add dev dependencies: pytest, pytest-asyncio, pytest-cov, httpx, respx, mypy, ruff
• Create pytest.ini with async and coverage configuration
• Create .gitignore (Python, IDE, env files)
• Create tests/ structure: unit/, integration/, fixtures/
• Create scripts/ directory
• Set up pre-commit hooks configuration (ruff, mypy)

Acceptance Criteria:

• Directory structure matches target state
• pip install -e ".[dev]" works
• pytest runs (even with 0 tests)
• Type checking with mypy src/ works

3.3 Documentation Organization

Goal: Restructure documentation for clarity and AI assistant consumption.

Tasks:

• Create /docs/architecture/ directory
• Create /docs/api/ directory
• Create /docs/development/ directory
• Move openapi.yaml → docs/api/openapi.yaml
• Move api_docs.md → docs/api/api-scenarios.md
• Refactor implementation_guide.md → split into:
  • docs/architecture/design-patterns.md (architectural patterns)
  • docs/development/testing-strategy.md (TDD approach)
  • docs/development/agentic-coding-guide.md (LLM-specific guidance)
• Create docs/architecture/architecture-overview.md (high-level system design)
• Update CLAUDE.md with new documentation structure

Acceptance Criteria:

• Documentation is logically organized by concern
• Each document has a single, clear purpose
• CLAUDE.md points to correct locations

---

4. Phase 2: TDD Interface Definition (Week 2)

4.1 Domain Models (TDD)

Goal: Define core domain models with comprehensive test coverage.

Approach: Write tests first, then implement models.

Tasks:

• Test: tests/unit/domain/test_document_metadata.py

  • Test: Valid metadata creation
  • Test: Invalid authorityId (empty, too long)
  • Test: Invalid referenceNumber format
  • Test: Future issuedAt date validation
  • Implement: src/domain/models.py → DocumentMetadata

• Test: tests/unit/domain/test_thread_models.py

  • Test: ThreadType enum values
  • Test: SenderRole enum values
  • Test: ThreadCreateRequest validation
  • Test: Message model with encrypted content
  • Implement: Thread-related models

• Test: tests/unit/domain/test_document_envelope.py

  • Test: Split payload structure
  • Test: encryptedPayload validation (base64, size limits)
  • Test: Status transitions (RECEIVED → ROUTED → ASSIGNED → CLOSED)
  • Implement: DocumentEnvelope and status management

Acceptance Criteria:

• All domain models have >90% test coverage
• Pydantic validation catches invalid inputs
• Tests run in <1 second
• Zero mypy type errors

4.2 Service Layer Interfaces (TDD)

Goal: Define service contracts via protocol/ABC classes with tests.

Tasks:

• Test: tests/unit/service/test_documents_service.py

  • Mock: Database adapter
  • Mock: Storage adapter (S3)
  • Mock: Routing engine
  • Test: create_document() - happy path
  • Test: create_document() - routing failure
  • Test: create_document() - storage failure
  • Test: get_document() - found
  • Test: get_document() - not found
  • Test: Retention date calculation (default 90 days)
  • Implement: src/service/documents_service.py

• Test: tests/unit/service/test_threads_service.py

  • Test: create_thread() - links to document
  • Test: create_thread() - document not found (404)
  • Test: list_messages() - cursor pagination
  • Test: add_message() - role validation
  • Implement: src/service/threads_service.py

• Test: tests/unit/service/test_exports_service.py

  • Test: create_export_job() - returns jobId
  • Test: get_export_status() - job states (QUEUED, RUNNING, COMPLETED, FAILED)
  • Test: Async job enqueuing (mock Redis/ARQ)
  • Implement: src/service/exports_service.py

Acceptance Criteria:

• Service layer has clear, testable interfaces
• All external dependencies are mocked
• Tests verify business logic, not infrastructure
• Each service method has both success and failure test cases

4.3 Adapter Contracts (Protocols)

Goal: Define adapter interfaces using Python Protocols for mockability.

Tasks:

• Create src/adapters/protocols.py:

  • StorageAdapter protocol (save_encrypted_payload, get_encrypted_payload)
  • DatabaseAdapter protocol (CRUD operations)
  • RoutingEngine protocol (route_document)
  • JobQueue protocol (enqueue, get_status)

• Create stub implementations for testing:

  • tests/fixtures/storage_stub.py (in-memory storage)
  • tests/fixtures/db_stub.py (in-memory DB)

Acceptance Criteria:

• Protocols are narrow and focused
• Test fixtures implement all protocols
• Production adapters can be swapped without changing service layer

---

5. Phase 3: Prototype Integration (Week 3)

5.1 Comparative Analysis

Goal: Identify best implementations across prototypes.

Tasks:

• Analyze prototype-chatgpt5/src/app/adapters/:

  • auth.py - OAuth2 implementation quality
  • db.py - SQLAlchemy async patterns
  • storage.py - S3 streaming approach
  • routing.py - Routing logic structure

• Analyze prototype-geminiNbt3pro/:

  • Identify unique features or better implementations

• Analyze prototype-grok4.1/:

  • Compare test coverage and patterns

• Document findings in: docs/development/prototype-analysis.md

  • Table: Feature vs Prototype vs Recommendation
  • Rationale for selections

Acceptance Criteria:

• Clear decision on which implementation to use for each component
• Documented rationale for selections
• Identified any missing features across all prototypes

5.2 Core Adapter Implementation

Goal: Implement production adapters based on best prototype code.

Tasks:

• Database Adapter (src/adapters/db.py):

  • Port SQLAlchemy models from chosen prototype
  • Implement async session management
  • Add connection pooling configuration
  • Write integration tests: tests/integration/test_db_adapter.py

• Storage Adapter (src/adapters/storage.py):

  • Implement S3 client using aiobotocore
  • Add streaming upload/download (no in-memory buffering)
  • Mock S3 in tests using moto or similar
  • Write tests: tests/integration/test_storage_adapter.py

• Routing Engine (src/adapters/routing.py):

  • Port routing logic from prototype
  • Make routing rules configurable (not hardcoded)
  • Add caching layer (Redis) for routing rules
  • Write tests: tests/unit/adapters/test_routing.py

• Authentication (src/adapters/auth.py):

  • Implement JWT validation
  • Add JWKS caching
  • Create FastAPI dependency for auth
  • Write tests: tests/unit/adapters/test_auth.py

Acceptance Criteria:

• All adapters follow Protocol contracts
• Integration tests use real dependencies (testcontainers)
• Unit tests use mocks
• Streaming works for large files (>50MB)

5.3 API Layer Implementation

Goal: Build FastAPI routes with OpenAPI compliance.

Tasks:

• Documents API (src/api/documents.py):

  • POST /documents - implement with streaming upload
  • GET /documents/{id} - implement with ETag support
  • Add request validation (Pydantic)
  • Write integration tests: tests/integration/test_documents_api.py

• Threads API (src/api/threads.py):

  • POST /documents/{id}/threads
  • GET /threads/{id}/messages (cursor pagination)
  • POST /threads/{id}/messages
  • Write integration tests: tests/integration/test_threads_api.py

• Exports API (src/api/exports.py):

  • POST /exports (async job creation)
  • GET /exports/{jobId} (status polling)
  • Write integration tests: tests/integration/test_exports_api.py

• Main App (src/main.py):

  • Configure FastAPI with CORS, middleware
  • Include all routers
  • Add exception handlers
  • Add health check endpoint: GET /health

Acceptance Criteria:

• OpenAPI schema matches docs/api/openapi.yaml
• All endpoints have auth middleware
• Integration tests achieve >80% coverage
• API responses match documented format

5.4 Background Workers

Goal: Implement async export worker.

Tasks:

• Choose task queue: ARQ (Redis-based, async) or Celery

• Implement src/workers/exports_worker.py:

  • Fetch document from storage
  • Fetch message history from DB
  • Generate export package (PDF + metadata)
  • Update job status

• Write worker tests: tests/unit/workers/test_exports_worker.py

• Document worker deployment in: docs/development/worker-deployment.md

Acceptance Criteria:

• Worker processes export jobs independently
• Failures are logged and job marked as FAILED
• Worker can be scaled horizontally

---

6. Phase 4: CI/CD & Quality Gates (Week 4)

6.1 GitHub Actions Workflows

Goal: Automate testing and quality checks.

Tasks:

• Create .github/workflows/test.yml:

  • Run on: push, pull_request
  • Matrix: Python 3.11, 3.12
  • Steps: Install deps, run pytest, upload coverage

• Create .github/workflows/lint.yml:

  • Run ruff linting
  • Run mypy type checking
  • Check code formatting

• Create .github/workflows/integration.yml:

  • Spin up PostgreSQL, Redis via services
  • Run integration tests with real dependencies

• Add status badges to README.md

Acceptance Criteria:

• All workflows pass on main branch
• Pull requests blocked if tests fail
• Coverage report available in PR comments

6.2 Pre-commit Hooks

Goal: Catch issues before commit.

Tasks:

• Create .pre-commit-config.yaml:

  • ruff linting and formatting
  • mypy type checking
  • trailing whitespace removal
  • YAML validation

• Document setup in: docs/development/setup-guide.md

Acceptance Criteria:

• Hooks auto-format code
• Hooks prevent commits with type errors
• Setup documented for new developers

6.3 Test Coverage Requirements

Goal: Enforce quality thresholds.

Tasks:

• Configure pytest-cov in pytest.ini:

  • Minimum coverage: 80%
  • Exclude: tests/, scripts/

• Add coverage badge to README.md

• Document coverage exemptions (e.g., # pragma: no cover)

Acceptance Criteria:

• pytest --cov fails if <80% coverage
• Coverage report generated in HTML format
• Uncovered lines are intentional and documented

---

7. Phase 5: Agentic Coding Enablement (Week 5)

7.1 Agentic Coding Guide

Goal: Create comprehensive guide for LLM-driven development.

Tasks:

• Create docs/development/agentic-coding-guide.md:

  • TDD workflow for Claude/GPT
  • Example prompts for generating tests
  • How to use Protocol adapters for mocking
  • Async testing patterns
  • Common pitfalls (GIL, blocking operations)

• Add example prompt templates:

  • "Write async pytest for POST /documents with ProcessPoolExecutor mock"
  • "Implement cursor pagination for messages following ADR-003"

• Update CLAUDE.md with agentic coding patterns

Acceptance Criteria:

• Guide includes concrete examples
• Prompts reference specific ADRs
• Guide covers both unit and integration test generation

7.2 Testing Utilities & Fixtures

Goal: Provide reusable test infrastructure.

Tasks:

• Create tests/fixtures/factories.py:

  • DocumentMetadataFactory (faker-based)
  • ThreadFactory
  • MessageFactory

• Create tests/fixtures/db_fixtures.py:

  • @pytest.fixture for async DB session
  • @pytest.fixture for testcontainers Postgres

• Create tests/fixtures/auth_fixtures.py:

  • Mock JWT tokens with different scopes
  • Mock JWKS endpoints

• Document in: docs/development/testing-utilities.md

Acceptance Criteria:

• Fixtures reduce boilerplate in tests
• Factories generate realistic test data
• Documentation shows usage examples

7.3 ADR for Agentic Development

Goal: Document the TDD + AI approach as architectural decision.

Tasks:

• Create docs/architecture/adr/0008-agentic-tdd-workflow.md:
  • Context: LLM-driven development velocity vs. quality
  • Decision: Interface-first TDD with AI assistance
  • Rationale: Tests serve as executable specification
  • Implementation: Workflow, tooling, prompts

Acceptance Criteria:

• ADR approved by team
• Links to agentic-coding-guide.md
• Referenced in CLAUDE.md

---

8. Phase 6: Migration & Validation (Week 6)

8.1 Prototype Deprecation

Goal: Mark prototypes as archived.

Tasks:

• Add README.md to each prototype directory:

  • Status: ARCHIVED
  • Reason: Consolidated into main codebase
  • Date: 2025-12-XX

• Document migration decisions in: docs/development/prototype-migration.md

• Keep prototypes in repo for reference (don't delete)

Acceptance Criteria:

• Clear indication that prototypes are not maintained
• Migration rationale documented

8.2 End-to-End Validation

Goal: Verify complete system integration.

Tasks:

• Write E2E test: tests/e2e/test_full_workflow.py:

  • Citizen uploads document
  • Document is routed
  • Thread is created
  • Messages are exchanged
  • Export is generated

• Run against local environment (Docker Compose)

• Measure performance against NFRs:

  • Document upload + routing: <500ms
  • Message retrieval: <300ms

• Document E2E setup in: docs/development/e2e-testing.md

Acceptance Criteria:

• E2E test passes consistently
• Performance targets met
• E2E environment reproducible via Docker Compose

8.3 Documentation Review

Goal: Ensure all documentation is accurate and complete.

Tasks:

• Review all ADRs for consistency
• Update CLAUDE.md with final structure
• Review API documentation against implementation
• Spell check and grammar check all docs
• Generate API documentation from OpenAPI spec (ReDoc or Swagger UI)

Acceptance Criteria:

• No broken links in documentation
• Code examples in docs are tested
• CLAUDE.md accurately reflects current state

---

9. Success Criteria (Overall)

Functional Requirements

• ✅ Main codebase has all features from best prototype
• ✅ All core APIs implemented and tested
• ✅ Background worker functional

Quality Requirements

• ✅ Test coverage >80%
• ✅ Zero mypy type errors
• ✅ All linting rules pass
• ✅ CI/CD pipeline green

Documentation Requirements

• ✅ ADRs in individual files with consistent structure
• ✅ All major decisions documented
• ✅ Agentic coding guide comprehensive
• ✅ CLAUDE.md accurate and complete

Performance Requirements

• ✅ Document routing <500ms (measured in E2E tests)
• ✅ Message retrieval <300ms (measured in E2E tests)
• ✅ Large file upload streaming works (>50MB test)

Process Requirements

• ✅ TDD workflow established and documented
• ✅ Pre-commit hooks prevent quality issues
• ✅ GitHub Actions enforce quality gates
• ✅ Agentic development patterns proven with at least 3 features

---

10. Risk Mitigation

Risk 1: Prototype Integration Conflicts

Mitigation: Complete comparative analysis (Phase 3.1) before implementation. Document decision rationale.

Risk 2: TDD Slowing Initial Progress

Mitigation: Front-load interface definition (Phase 2). Once interfaces stable, implementation accelerates.

Risk 3: Incomplete ADR Extraction

Mitigation: Use checklist approach. Review original decisions.md multiple times. Cross-reference with implementation guide.

Risk 4: Agentic Coding Learning Curve

Mitigation: Create example-driven guide. Include actual prompts that worked. Pair with human for first few features.

Risk 5: Performance Targets Not Met

Mitigation: Include performance testing from Phase 2 onwards. Identify bottlenecks early. Profile with py-spy or similar.

---

11. Next Steps

1. Review this workplan with the team
2. Adjust timeline based on team capacity
3. Start Phase 1 (Foundation Setup)
4. Daily standup to track progress and blockers
5. Weekly retrospective to improve agentic coding workflow

---

Appendix A: Recommended Tools

• Testing: pytest, pytest-asyncio, pytest-cov, hypothesis (property testing)
• Mocking: respx (HTTP), moto (AWS), testcontainers-python (real services)
• Linting: ruff (fast, replaces flake8 + isort + pyupgrade)
• Type Checking: mypy with strict mode
• Factories: factory_boy or custom Pydantic factories
• Performance: py-spy (profiling), locust (load testing)
• Pre-commit: pre-commit framework
• CI/CD: GitHub Actions (free for public repos)

---

Appendix B: ADR Numbering Convention

• 0001-0099: Core Architecture (payload model, auth, concurrency)
• 0100-0199: Data & Persistence (pagination, retention, schema)
• 0200-0299: API & Integration (REST structure, export workflow)
• 0300-0399: Development Process (TDD, agentic coding)
• 0400+: Future decisions

---

Document Owner: Backend Engineering Team Last Updated: 2025-12-01 Status: Ready for Review