markitect-main/docs/roadmap/information-space-service/WORKPLAN.md

# Headless Information Space Service - Implementation Workplan

## Overview

This workplan details the implementation phases for evolving markitect into a headless markdown transclusion-based information space service.

---

## Phase 0: Project Organization (Prerequisite)

### Current State Issues
- **Hybrid layout** - Mix of root-level packages and monolithic `/markitect/`
- **Flat root in markitect** - 34 .py files at `/markitect/` root level
- **No structure documentation** - Missing PROJECT_STRUCTURE.md (now created)

### Reorganization Tasks

| ID | Task | Description | Status |
|----|------|-------------|--------|
| ORG-001 | Create PROJECT_STRUCTURE.md | Document current layout and rationale | Done |
| ORG-002 | Create `/markitect/core/` | Move parser, serializer, document_manager | Pending |
| ORG-003 | Create `/markitect/schema/` | Consolidate 6 schema_*.py files | Pending |
| ORG-004 | Create `/markitect/storage/` | Group database.py, cache modules | Pending |
| ORG-005 | Update imports | Fix all import statements after moves | Pending |
| ORG-006 | Verify tests | Ensure all tests pass after moves | Pending |

### Target Structure After Phase 0

```
markitect/
├── core/                    # Core infrastructure
│   ├── __init__.py
│   ├── parser.py           # (from markitect/)
│   ├── serializer.py       # (from markitect/)
│   ├── document_manager.py # (from markitect/)
│   └── workspace.py        # (from markitect/)
├── schema/                  # Schema management
│   ├── __init__.py
│   ├── validator.py        # (from schema_validator.py)
│   ├── generator.py        # (from schema_generator.py)
│   ├── loader.py           # (from schema_loader.py)
│   ├── analyzer.py         # (from schema_analyzer.py)
│   ├── refiner.py          # (from schema_refiner.py)
│   └── naming.py           # (from schema_naming.py)
├── storage/                 # Storage concerns
│   ├── __init__.py
│   └── database.py         # (from markitect/)
└── spaces/                  # Information spaces (Phase 1+)
```

---

## Phase 1: Foundation

### Capability Requirements

| ID | Capability | Description | Priority |
|----|-----------|-------------|----------|
| CAP-001 | InformationSpace Entity | First-class space abstraction with identity, metadata, lifecycle | Critical |
| CAP-002 | SpaceRepository | CRUD operations for spaces with SQLite backing | Critical |
| CAP-003 | Document-Space Association | Link documents to spaces with membership tracking | Critical |
| CAP-004 | Space Metadata Schema | Extensible metadata schema for space configuration | High |
| CAP-005 | Database Migrations | Schema evolution for space-related tables | High |

### Implementation Tasks

**Week 1: Core Models**
- Create `markitect/spaces/models.py`
  - `InformationSpace` dataclass with id, name, description, metadata, config
  - `SpaceDocument` dataclass for document membership
  - `SpaceConfig` dataclass for space settings
- Create `markitect/spaces/repositories/interfaces.py`
- Unit tests for models

**Week 2: Repository Implementation**
- Create `markitect/spaces/repositories/sqlite.py`
- Implement `ISpaceRepository` for SQLite
- Implement `IDocumentAssociationRepository`
- Database migration scripts
- Repository unit tests

**Week 3: Basic SpaceService**
- Create `markitect/spaces/services/space_service.py`
- CRUD operations for spaces
- Document add/remove operations
- Integration tests

### Database Schema

```sql
CREATE TABLE spaces (
    id TEXT PRIMARY KEY,
    name TEXT UNIQUE NOT NULL,
    description TEXT,
    metadata JSON,
    config JSON,
    parent_space_id TEXT REFERENCES spaces(id),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE space_documents (
    id TEXT PRIMARY KEY,
    space_id TEXT NOT NULL REFERENCES spaces(id),
    document_id TEXT NOT NULL,
    space_path TEXT NOT NULL,
    order_index INTEGER DEFAULT 0,
    metadata JSON,
    added_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(space_id, space_path)
);
```

### Verification
```bash
pytest tests/unit/spaces/test_*_model.py
pytest tests/unit/spaces/test_*_repository.py
pytest tests/integration/spaces/test_space_service_integration.py
```

---

## Phase 2: Event System

### Capability Requirements

| ID | Capability | Description | Priority |
|----|-----------|-------------|----------|
| CAP-010 | SpaceEvent Base | Event dataclass with type, payload, timestamp | Critical |
| CAP-011 | Event Bus | In-process publish/subscribe for space events | Critical |
| CAP-012 | Event Handlers Registry | Register/unregister event handlers by type | High |
| CAP-013 | Change Detection | Detect document changes via content hash comparison | High |
| CAP-014 | Event Persistence | Store events for replay/audit | Medium |

### Implementation Tasks

**Week 4: Event Infrastructure**
- Create `markitect/spaces/events/models.py`
  - `SpaceEvent` dataclass with event_id, type, space_id, payload, timestamp
  - `SpaceEventType` enum (DOCUMENT_ADDED, DOCUMENT_UPDATED, DOCUMENT_REMOVED, etc.)
- Create `markitect/spaces/events/bus.py`
  - `EventBus` with sync/async handler support
  - Handler registration by event type
- Unit tests for event bus

**Week 5: Integration**
- Wire events into SpaceService (emit on document operations)
- Implement change detection (content hash comparison)
- Optional: event persistence table
- Integration tests for event flow

### Event Types

```python
class SpaceEventType(Enum):
    SPACE_CREATED = "space.created"
    SPACE_UPDATED = "space.updated"
    SPACE_DELETED = "space.deleted"
    DOCUMENT_ADDED = "document.added"
    DOCUMENT_UPDATED = "document.updated"
    DOCUMENT_REMOVED = "document.removed"
    DOCUMENT_MOVED = "document.moved"
    VARIABLE_SET = "variable.set"
    RENDER_COMPLETED = "render.completed"
    SYNC_COMPLETED = "sync.completed"
```

### Verification
```bash
pytest tests/unit/spaces/test_event_bus.py
pytest tests/integration/spaces/test_event_propagation.py
```

---

## Phase 3: Persistent Transclusion Context

### Capability Requirements

| ID | Capability | Description | Priority |
|----|-----------|-------------|----------|
| CAP-020 | Persistent TransclusionContext | Store context state in database | Critical |
| CAP-021 | Cross-Space References | Resolve transclusions across space boundaries | High |
| CAP-022 | Reference Graph | Track document dependencies for invalidation | High |
| CAP-023 | Variable Scope Layers | Space-level, document-level, request-level variables | Medium |
| CAP-024 | Transclusion Cache Invalidation | Invalidate rendered content on dependency change | High |

### Implementation Tasks

**Week 6: Persistent Context**
- Create `markitect/spaces/transclusion/persistent_context.py`
- Extend existing `TransclusionContext` with DB persistence
- Space-scoped variable storage

**Week 7: Reference Graph**
- Implement reference tracking during transclusion resolution
- Cross-space reference resolution with space:// protocol
- Variable scope layers (space → document → request)

**Week 8: Cache Invalidation**
- Wire change events to cache invalidation
- Dependency-aware cache clearing
- Integration tests

### Database Schema Additions

```sql
CREATE TABLE space_variables (
    space_id TEXT NOT NULL REFERENCES spaces(id),
    name TEXT NOT NULL,
    value JSON,
    scope TEXT DEFAULT 'space',
    PRIMARY KEY(space_id, name)
);

CREATE TABLE transclusion_references (
    source_doc_id TEXT NOT NULL,
    target_doc_id TEXT NOT NULL,
    space_id TEXT NOT NULL REFERENCES spaces(id),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY(source_doc_id, target_doc_id, space_id)
);
```

### Verification
```bash
pytest tests/unit/spaces/test_persistent_context.py
pytest tests/unit/spaces/test_reference_graph.py
pytest tests/integration/spaces/test_transclusion_persistence.py
```

---

## Phase 4: HTML Rendering Mode

### Capability Requirements

| ID | Capability | Description | Priority |
|----|-----------|-------------|----------|
| CAP-030 | SpaceRenderer Base | Abstract renderer interface | Critical |
| CAP-031 | MarkdownToHTMLRenderer | Render resolved markdown to HTML | Critical |
| CAP-032 | Rendering Cache | Cache rendered output with invalidation | High |
| CAP-033 | Theme Support | Apply themes to rendered HTML | Medium |
| CAP-034 | Incremental Rendering | Re-render only changed documents | Medium |

### Implementation Tasks

**Week 9: Renderer Base**
- Create `markitect/spaces/rendering/base.py` - SpaceRenderer ABC
- Create `markitect/spaces/rendering/html_renderer.py` - MarkdownToHTMLRenderer
- Integrate with existing `CleanDocumentManager`

**Week 10: Caching and Themes**
- Implement render output caching (keyed by content hash)
- Theme integration using existing theme system
- Invalidation on dependency change via events

**Week 11: Incremental Rendering**
- Re-render only affected documents on change
- Rendering events emission
- E2E tests for render workflow

### Verification
```bash
pytest tests/e2e/spaces/test_html_rendering_workflow.py
```

---

## Phase 5: Directory Mode

### Capability Requirements

| ID | Capability | Description | Priority |
|----|-----------|-------------|----------|
| CAP-040 | SpaceToDirectory Exporter | Export space to canonical directory structure | Critical |
| CAP-041 | DirectoryToSpace Importer | Import directory structure as space | Critical |
| CAP-042 | Bidirectional Sync | Detect and sync changes both directions | High |
| CAP-043 | Filesystem Watcher | Watch directory for external changes | Medium |
| CAP-044 | Conflict Resolution | Handle conflicts in bidirectional sync | Medium |

### Implementation Tasks

**Week 12: Export**
- Create `markitect/spaces/sync/directory_exporter.py`
- Integrate with existing `VariantFactory`
- Support flat/hierarchical/semantic variants

**Week 13: Import and Sync**
- Create `markitect/spaces/sync/directory_importer.py`
- Create `markitect/spaces/sync/bidirectional.py`
- Conflict detection (modification time, content hash)

**Week 14: Filesystem Watcher**
- Implement watcher using `watchdog` library
- Sync events emission
- E2E tests for bidirectional sync

### Canonical Directory Structure

```
.markitect/spaces/{space-name}/
├── .space.yaml          # Space metadata and config
├── documents/           # Document files
│   ├── intro.md
│   ├── getting-started.md
│   └── advanced/
│       └── topics.md
└── assets/              # Associated assets
```

### Verification
```bash
pytest tests/e2e/spaces/test_directory_mode_workflow.py
```

---

## Phase 6: API Layer

### Capability Requirements

| ID | Capability | Description | Priority |
|----|-----------|-------------|----------|
| CAP-060 | SpaceService | Service layer orchestrating space operations | Critical |
| CAP-061 | GraphQL Space Schema | Extend existing GraphQL with space types | High |
| CAP-062 | REST Endpoints | Alternative REST API for spaces | Medium |
| CAP-063 | WebSocket Subscriptions | Real-time event subscriptions | Medium |
| CAP-064 | CLI Space Commands | CLI commands for space management | High |

### Implementation Tasks

**Week 15: GraphQL Extension**
- Extend `markitect/graphql/schema.py` with Space types
- Add mutations: createSpace, updateSpace, deleteSpace
- Add queries: space, spaces, spaceDocuments
- Add subscriptions: onSpaceEvent

**Week 16: CLI Commands**
- Add to `markitect/cli.py`:
  - `markitect space create/list/show/delete`
  - `markitect space add-doc/remove-doc/list-docs`
  - `markitect space render`
  - `markitect space sync`

**Week 17: WebSocket and Polish**
- WebSocket subscriptions for real-time events
- Documentation updates
- Final integration testing

### GraphQL Schema Extensions

```graphql
type InformationSpace {
  id: ID!
  name: String!
  description: String
  documents: [SpaceDocument!]!
  config: SpaceConfig!
  parentSpace: InformationSpace
  createdAt: DateTime!
  updatedAt: DateTime!
}

type SpaceDocument {
  id: ID!
  spacePath: String!
  content: String!
  metadata: JSON
}

type Mutation {
  createSpace(input: CreateSpaceInput!): InformationSpace!
  addDocument(spaceId: ID!, input: AddDocumentInput!): SpaceDocument!
  renderSpace(spaceId: ID!, options: RenderOptions): RenderResult!
}

type Subscription {
  onSpaceEvent(spaceId: ID!): SpaceEvent!
}
```

### Verification
```bash
pytest tests/integration/spaces/
pytest tests/e2e/spaces/
markitect space --help  # Verify CLI
```

---

## Phase 7: Composability

### Capability Requirements

| ID | Capability | Description | Priority |
|----|-----------|-------------|----------|
| CAP-050 | Space References | Spaces can reference other spaces | High |
| CAP-051 | Space Inheritance | Child spaces inherit parent context | Medium |
| CAP-053 | Space Access Control | Basic permission model for space access | Medium |

### Implementation Tasks

**Week 18-19: Space References**
- Space-to-space references via space:// protocol
- Variable inheritance from parent spaces
- Basic access control (read/write/admin)

**Week 20: Final Integration**
- Complete E2E test suite
- Performance testing
- User documentation

### Space Reference Protocol

```markdown
<!-- Reference document from another space -->
{{transclude space://other-space/path/to/doc.md}}

<!-- Reference with variable override -->
{{transclude space://shared-components/header.md | title="My Page"}}
```

---

## Phase 8: Git History Tracking (Optional)

### Capability Requirements

| ID | Capability | Description | Priority |
|----|-----------|-------------|----------|
| CAP-070 | History Configuration | Per-space history tracking configuration | High |
| CAP-071 | HistoryBackend Interface | Abstract interface for history backends | High |
| CAP-072 | GitHistoryBackend | Git implementation of history backend | High |
| CAP-073 | Canonical Directory Binding | Bind space to canonical directory for git | High |
| CAP-074 | Event-Driven Commits | Commit on document change events | Medium |
| CAP-075 | History Query API | Query commits, diffs, branches | High |
| CAP-076 | History CLI Commands | CLI for log, diff, restore, checkout | High |
| CAP-077 | Versioned Read/Render | Read/render documents at specific versions | Medium |

### Implementation Tasks

**Week 21: History Infrastructure**
- Create `markitect/spaces/history/interfaces.py` - IHistoryBackend ABC
- Create `markitect/spaces/history/models.py` - Commit, HistoryEntry dataclasses
- Add `SpaceConfig` fields: history_enabled, history_backend, history_options
- Add `SPACE_SYNC` to PluginType enum

**Week 22: Git Backend**
- Create `markitect/spaces/history/git_backend.py`
- Leverage existing `legacy/git_tracker.py` patterns
- Create event handlers for auto-commit on document changes
- Integration tests

**Week 23: API and CLI**
- History query service for log, diff, branches
- CLI commands: `markitect space history log/diff/restore/checkout`
- Extend read/render with `--version` option
- E2E tests

### Integration Diagram

```
Document Update Flow (with history enabled):

User updates document
        │
        ▼
┌───────────────────┐
│   SpaceService    │
│  update_document()│
└────────┬──────────┘
         │ emit event
         ▼
┌───────────────────┐     ┌─────────────────────┐
│    Event Bus      │────▶│ GitHistoryHandler   │
│ DOCUMENT_UPDATED  │     │ (subscribed)        │
└───────────────────┘     └──────────┬──────────┘
                                     │
                          ┌──────────▼──────────┐
                          │ DirectorySyncService │
                          └──────────┬──────────┘
                                     │ writes to
                          ┌──────────▼──────────┐
                          │ Canonical Directory  │
                          │ .markitect/spaces/X/ │
                          └──────────┬──────────┘
                                     │
                          ┌──────────▼──────────┐
                          │   GitHistoryBackend  │
                          │ git add && git commit│
                          └─────────────────────┘
```

### Verification
```bash
markitect space create my-space --history-enabled
markitect space add-doc my-space --content "# V1"
markitect space update-doc my-space/doc.md --content "# V2"
markitect space history log my-space
markitect space history diff my-space --rev HEAD~1
```

---

## Timeline Summary

| Phase | Focus | Duration |
|-------|-------|----------|
| 0 | Project Organization | 1 week |
| 1 | Foundation | 3 weeks |
| 2 | Event System | 2 weeks |
| 3 | Persistent Transclusion | 3 weeks |
| 4 | HTML Rendering Mode | 3 weeks |
| 5 | Directory Mode | 3 weeks |
| 6 | API Layer | 3 weeks |
| 7 | Composability | 3 weeks |
| 8 | Git History (Optional) | 3 weeks |

**Total: 21-24 weeks** (5-6 months)

### Parallel Work Opportunities
- Phase 4 (HTML) and Phase 5 (Directory) can run in parallel after Phase 3
- Phase 8 can start in parallel with Phase 7
- Documentation can be written incrementally
- CLI commands can start in parallel with Phase 4/5

---

## Files to Create

### Phase 0
```
docs/PROJECT_STRUCTURE.md           # Done
roadmap/information-space-service/  # Done
├── README.md                       # Done
└── WORKPLAN.md                     # This file
markitect/core/                     # To do
markitect/schema/                   # To do
markitect/storage/                  # To do
```

### Phase 1+
```
markitect/spaces/
├── __init__.py
├── models.py
├── events/
│   ├── __init__.py
│   ├── models.py
│   └── bus.py
├── repositories/
│   ├── __init__.py
│   ├── interfaces.py
│   └── sqlite.py
├── transclusion/
│   ├── __init__.py
│   └── persistent_context.py
├── rendering/
│   ├── __init__.py
│   ├── base.py
│   └── html_renderer.py
├── sync/
│   ├── __init__.py
│   ├── directory_exporter.py
│   ├── directory_importer.py
│   └── bidirectional.py
├── history/                        # Phase 8
│   ├── __init__.py
│   ├── interfaces.py
│   ├── models.py
│   ├── git_backend.py
│   ├── events.py
│   └── queries.py
└── services/
    ├── __init__.py
    └── space_service.py
```

### Test Files
```
tests/unit/spaces/
tests/integration/spaces/
tests/e2e/spaces/
tests/fixtures/spaces.py
```

---

## Success Criteria

1. Phase 0 complete: project reorganized with docs/PROJECT_STRUCTURE.md
2. All phases complete with passing tests
3. HTML rendering mode fully functional
4. Directory mode with bidirectional sync working
5. GraphQL API exposing all space operations
6. CLI commands operational
7. Events propagating correctly
8. Cross-space transclusion resolving