markitect-main/docs/roadmap/information-space-service/WORKPLAN.md

Headless Information Space Service - Implementation Workplan

Overview

This workplan details the implementation phases for evolving markitect into a headless markdown transclusion-based information space service.

---

Phase 0: Project Organization (Prerequisite)

Current State Issues

• Hybrid layout - Mix of root-level packages and monolithic /markitect/
• Flat root in markitect - 34 .py files at /markitect/ root level
• No structure documentation - Missing PROJECT_STRUCTURE.md (now created)

Reorganization Tasks

ID	Task	Description	Status
ORG-001	Create PROJECT_STRUCTURE.md	Document current layout and rationale	Done
ORG-002	Create /markitect/core/	Move parser, serializer, document_manager	Pending
ORG-003	Create /markitect/schema/	Consolidate 6 schema_*.py files	Pending
ORG-004	Create /markitect/storage/	Group database.py, cache modules	Pending
ORG-005	Update imports	Fix all import statements after moves	Pending
ORG-006	Verify tests	Ensure all tests pass after moves	Pending

Target Structure After Phase 0

markitect/
├── core/                    # Core infrastructure
│   ├── __init__.py
│   ├── parser.py           # (from markitect/)
│   ├── serializer.py       # (from markitect/)
│   ├── document_manager.py # (from markitect/)
│   └── workspace.py        # (from markitect/)
├── schema/                  # Schema management
│   ├── __init__.py
│   ├── validator.py        # (from schema_validator.py)
│   ├── generator.py        # (from schema_generator.py)
│   ├── loader.py           # (from schema_loader.py)
│   ├── analyzer.py         # (from schema_analyzer.py)
│   ├── refiner.py          # (from schema_refiner.py)
│   └── naming.py           # (from schema_naming.py)
├── storage/                 # Storage concerns
│   ├── __init__.py
│   └── database.py         # (from markitect/)
└── spaces/                  # Information spaces (Phase 1+)

---

Phase 1: Foundation

Capability Requirements

ID	Capability	Description	Priority
CAP-001	InformationSpace Entity	First-class space abstraction with identity, metadata, lifecycle	Critical
CAP-002	SpaceRepository	CRUD operations for spaces with SQLite backing	Critical
CAP-003	Document-Space Association	Link documents to spaces with membership tracking	Critical
CAP-004	Space Metadata Schema	Extensible metadata schema for space configuration	High
CAP-005	Database Migrations	Schema evolution for space-related tables	High

Implementation Tasks

Week 1: Core Models

• Create markitect/spaces/models.py
  • InformationSpace dataclass with id, name, description, metadata, config
  • SpaceDocument dataclass for document membership
  • SpaceConfig dataclass for space settings
• Create markitect/spaces/repositories/interfaces.py
• Unit tests for models

Week 2: Repository Implementation

• Create markitect/spaces/repositories/sqlite.py
• Implement ISpaceRepository for SQLite
• Implement IDocumentAssociationRepository
• Database migration scripts
• Repository unit tests

Week 3: Basic SpaceService

• Create markitect/spaces/services/space_service.py
• CRUD operations for spaces
• Document add/remove operations
• Integration tests

Database Schema

CREATE TABLE spaces (
    id TEXT PRIMARY KEY,
    name TEXT UNIQUE NOT NULL,
    description TEXT,
    metadata JSON,
    config JSON,
    parent_space_id TEXT REFERENCES spaces(id),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE space_documents (
    id TEXT PRIMARY KEY,
    space_id TEXT NOT NULL REFERENCES spaces(id),
    document_id TEXT NOT NULL,
    space_path TEXT NOT NULL,
    order_index INTEGER DEFAULT 0,
    metadata JSON,
    added_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(space_id, space_path)
);

Verification

pytest tests/unit/spaces/test_*_model.py
pytest tests/unit/spaces/test_*_repository.py
pytest tests/integration/spaces/test_space_service_integration.py

---

Phase 2: Event System

Capability Requirements

ID	Capability	Description	Priority
CAP-010	SpaceEvent Base	Event dataclass with type, payload, timestamp	Critical
CAP-011	Event Bus	In-process publish/subscribe for space events	Critical
CAP-012	Event Handlers Registry	Register/unregister event handlers by type	High
CAP-013	Change Detection	Detect document changes via content hash comparison	High
CAP-014	Event Persistence	Store events for replay/audit	Medium

Implementation Tasks

Week 4: Event Infrastructure

• Create markitect/spaces/events/models.py
  • SpaceEvent dataclass with event_id, type, space_id, payload, timestamp
  • SpaceEventType enum (DOCUMENT_ADDED, DOCUMENT_UPDATED, DOCUMENT_REMOVED, etc.)
• Create markitect/spaces/events/bus.py
  • EventBus with sync/async handler support
  • Handler registration by event type
• Unit tests for event bus

Week 5: Integration

• Wire events into SpaceService (emit on document operations)
• Implement change detection (content hash comparison)
• Optional: event persistence table
• Integration tests for event flow

Event Types

class SpaceEventType(Enum):
    SPACE_CREATED = "space.created"
    SPACE_UPDATED = "space.updated"
    SPACE_DELETED = "space.deleted"
    DOCUMENT_ADDED = "document.added"
    DOCUMENT_UPDATED = "document.updated"
    DOCUMENT_REMOVED = "document.removed"
    DOCUMENT_MOVED = "document.moved"
    VARIABLE_SET = "variable.set"
    RENDER_COMPLETED = "render.completed"
    SYNC_COMPLETED = "sync.completed"

Verification

pytest tests/unit/spaces/test_event_bus.py
pytest tests/integration/spaces/test_event_propagation.py

---

Phase 3: Persistent Transclusion Context

Capability Requirements

ID	Capability	Description	Priority
CAP-020	Persistent TransclusionContext	Store context state in database	Critical
CAP-021	Cross-Space References	Resolve transclusions across space boundaries	High
CAP-022	Reference Graph	Track document dependencies for invalidation	High
CAP-023	Variable Scope Layers	Space-level, document-level, request-level variables	Medium
CAP-024	Transclusion Cache Invalidation	Invalidate rendered content on dependency change	High

Implementation Tasks

Week 6: Persistent Context

• Create markitect/spaces/transclusion/persistent_context.py
• Extend existing TransclusionContext with DB persistence
• Space-scoped variable storage

Week 7: Reference Graph

• Implement reference tracking during transclusion resolution
• Cross-space reference resolution with space:// protocol
• Variable scope layers (space → document → request)

Week 8: Cache Invalidation

• Wire change events to cache invalidation
• Dependency-aware cache clearing
• Integration tests

Database Schema Additions

CREATE TABLE space_variables (
    space_id TEXT NOT NULL REFERENCES spaces(id),
    name TEXT NOT NULL,
    value JSON,
    scope TEXT DEFAULT 'space',
    PRIMARY KEY(space_id, name)
);

CREATE TABLE transclusion_references (
    source_doc_id TEXT NOT NULL,
    target_doc_id TEXT NOT NULL,
    space_id TEXT NOT NULL REFERENCES spaces(id),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY(source_doc_id, target_doc_id, space_id)
);

Verification

pytest tests/unit/spaces/test_persistent_context.py
pytest tests/unit/spaces/test_reference_graph.py
pytest tests/integration/spaces/test_transclusion_persistence.py

---

Phase 4: HTML Rendering Mode

Capability Requirements

ID	Capability	Description	Priority
CAP-030	SpaceRenderer Base	Abstract renderer interface	Critical
CAP-031	MarkdownToHTMLRenderer	Render resolved markdown to HTML	Critical
CAP-032	Rendering Cache	Cache rendered output with invalidation	High
CAP-033	Theme Support	Apply themes to rendered HTML	Medium
CAP-034	Incremental Rendering	Re-render only changed documents	Medium

Implementation Tasks

Week 9: Renderer Base

• Create markitect/spaces/rendering/base.py - SpaceRenderer ABC
• Create markitect/spaces/rendering/html_renderer.py - MarkdownToHTMLRenderer
• Integrate with existing CleanDocumentManager

Week 10: Caching and Themes

• Implement render output caching (keyed by content hash)
• Theme integration using existing theme system
• Invalidation on dependency change via events

Week 11: Incremental Rendering

• Re-render only affected documents on change
• Rendering events emission
• E2E tests for render workflow

Verification

pytest tests/e2e/spaces/test_html_rendering_workflow.py

---

Phase 5: Directory Mode

Capability Requirements

ID	Capability	Description	Priority
CAP-040	SpaceToDirectory Exporter	Export space to canonical directory structure	Critical
CAP-041	DirectoryToSpace Importer	Import directory structure as space	Critical
CAP-042	Bidirectional Sync	Detect and sync changes both directions	High
CAP-043	Filesystem Watcher	Watch directory for external changes	Medium
CAP-044	Conflict Resolution	Handle conflicts in bidirectional sync	Medium

Implementation Tasks

Week 12: Export

• Create markitect/spaces/sync/directory_exporter.py
• Integrate with existing VariantFactory
• Support flat/hierarchical/semantic variants

Week 13: Import and Sync

• Create markitect/spaces/sync/directory_importer.py
• Create markitect/spaces/sync/bidirectional.py
• Conflict detection (modification time, content hash)

Week 14: Filesystem Watcher

• Implement watcher using watchdog library
• Sync events emission
• E2E tests for bidirectional sync

Canonical Directory Structure

.markitect/spaces/{space-name}/
├── .space.yaml          # Space metadata and config
├── documents/           # Document files
│   ├── intro.md
│   ├── getting-started.md
│   └── advanced/
│       └── topics.md
└── assets/              # Associated assets

Verification

pytest tests/e2e/spaces/test_directory_mode_workflow.py

---

Phase 6: API Layer

Capability Requirements

ID	Capability	Description	Priority
CAP-060	SpaceService	Service layer orchestrating space operations	Critical
CAP-061	GraphQL Space Schema	Extend existing GraphQL with space types	High
CAP-062	REST Endpoints	Alternative REST API for spaces	Medium
CAP-063	WebSocket Subscriptions	Real-time event subscriptions	Medium
CAP-064	CLI Space Commands	CLI commands for space management	High

Implementation Tasks

Week 15: GraphQL Extension

• Extend markitect/graphql/schema.py with Space types
• Add mutations: createSpace, updateSpace, deleteSpace
• Add queries: space, spaces, spaceDocuments
• Add subscriptions: onSpaceEvent

Week 16: CLI Commands

• Add to markitect/cli.py:
  • markitect space create/list/show/delete
  • markitect space add-doc/remove-doc/list-docs
  • markitect space render
  • markitect space sync

Week 17: WebSocket and Polish

• WebSocket subscriptions for real-time events
• Documentation updates
• Final integration testing

GraphQL Schema Extensions

type InformationSpace {
  id: ID!
  name: String!
  description: String
  documents: [SpaceDocument!]!
  config: SpaceConfig!
  parentSpace: InformationSpace
  createdAt: DateTime!
  updatedAt: DateTime!
}

type SpaceDocument {
  id: ID!
  spacePath: String!
  content: String!
  metadata: JSON
}

type Mutation {
  createSpace(input: CreateSpaceInput!): InformationSpace!
  addDocument(spaceId: ID!, input: AddDocumentInput!): SpaceDocument!
  renderSpace(spaceId: ID!, options: RenderOptions): RenderResult!
}

type Subscription {
  onSpaceEvent(spaceId: ID!): SpaceEvent!
}

Verification

pytest tests/integration/spaces/
pytest tests/e2e/spaces/
markitect space --help  # Verify CLI

---

Phase 7: Composability

Capability Requirements

ID	Capability	Description	Priority
CAP-050	Space References	Spaces can reference other spaces	High
CAP-051	Space Inheritance	Child spaces inherit parent context	Medium
CAP-053	Space Access Control	Basic permission model for space access	Medium

Implementation Tasks

Week 18-19: Space References

• Space-to-space references via space:// protocol
• Variable inheritance from parent spaces
• Basic access control (read/write/admin)

Week 20: Final Integration

• Complete E2E test suite
• Performance testing
• User documentation

Space Reference Protocol

<!-- Reference document from another space -->
{{transclude space://other-space/path/to/doc.md}}

<!-- Reference with variable override -->
{{transclude space://shared-components/header.md | title="My Page"}}

---

Phase 8: Git History Tracking (Optional)

Capability Requirements

ID	Capability	Description	Priority
CAP-070	History Configuration	Per-space history tracking configuration	High
CAP-071	HistoryBackend Interface	Abstract interface for history backends	High
CAP-072	GitHistoryBackend	Git implementation of history backend	High
CAP-073	Canonical Directory Binding	Bind space to canonical directory for git	High
CAP-074	Event-Driven Commits	Commit on document change events	Medium
CAP-075	History Query API	Query commits, diffs, branches	High
CAP-076	History CLI Commands	CLI for log, diff, restore, checkout	High
CAP-077	Versioned Read/Render	Read/render documents at specific versions	Medium

Implementation Tasks

Week 21: History Infrastructure

• Create markitect/spaces/history/interfaces.py - IHistoryBackend ABC
• Create markitect/spaces/history/models.py - Commit, HistoryEntry dataclasses
• Add SpaceConfig fields: history_enabled, history_backend, history_options
• Add SPACE_SYNC to PluginType enum

Week 22: Git Backend

• Create markitect/spaces/history/git_backend.py
• Leverage existing legacy/git_tracker.py patterns
• Create event handlers for auto-commit on document changes
• Integration tests

Week 23: API and CLI

• History query service for log, diff, branches
• CLI commands: markitect space history log/diff/restore/checkout
• Extend read/render with --version option
• E2E tests

Integration Diagram

Document Update Flow (with history enabled):

User updates document
        │
        ▼
┌───────────────────┐
│   SpaceService    │
│  update_document()│
└────────┬──────────┘
         │ emit event
         ▼
┌───────────────────┐     ┌─────────────────────┐
│    Event Bus      │────▶│ GitHistoryHandler   │
│ DOCUMENT_UPDATED  │     │ (subscribed)        │
└───────────────────┘     └──────────┬──────────┘
                                     │
                          ┌──────────▼──────────┐
                          │ DirectorySyncService │
                          └──────────┬──────────┘
                                     │ writes to
                          ┌──────────▼──────────┐
                          │ Canonical Directory  │
                          │ .markitect/spaces/X/ │
                          └──────────┬──────────┘
                                     │
                          ┌──────────▼──────────┐
                          │   GitHistoryBackend  │
                          │ git add && git commit│
                          └─────────────────────┘

Verification

markitect space create my-space --history-enabled
markitect space add-doc my-space --content "# V1"
markitect space update-doc my-space/doc.md --content "# V2"
markitect space history log my-space
markitect space history diff my-space --rev HEAD~1

---

Timeline Summary

Phase	Focus	Duration
0	Project Organization	1 week
1	Foundation	3 weeks
2	Event System	2 weeks
3	Persistent Transclusion	3 weeks
4	HTML Rendering Mode	3 weeks
5	Directory Mode	3 weeks
6	API Layer	3 weeks
7	Composability	3 weeks
8	Git History (Optional)	3 weeks

Total: 21-24 weeks (5-6 months)

Parallel Work Opportunities

• Phase 4 (HTML) and Phase 5 (Directory) can run in parallel after Phase 3
• Phase 8 can start in parallel with Phase 7
• Documentation can be written incrementally
• CLI commands can start in parallel with Phase 4/5

---

Files to Create

Phase 0

docs/PROJECT_STRUCTURE.md           # Done
roadmap/information-space-service/  # Done
├── README.md                       # Done
└── WORKPLAN.md                     # This file
markitect/core/                     # To do
markitect/schema/                   # To do
markitect/storage/                  # To do

Phase 1+

markitect/spaces/
├── __init__.py
├── models.py
├── events/
│   ├── __init__.py
│   ├── models.py
│   └── bus.py
├── repositories/
│   ├── __init__.py
│   ├── interfaces.py
│   └── sqlite.py
├── transclusion/
│   ├── __init__.py
│   └── persistent_context.py
├── rendering/
│   ├── __init__.py
│   ├── base.py
│   └── html_renderer.py
├── sync/
│   ├── __init__.py
│   ├── directory_exporter.py
│   ├── directory_importer.py
│   └── bidirectional.py
├── history/                        # Phase 8
│   ├── __init__.py
│   ├── interfaces.py
│   ├── models.py
│   ├── git_backend.py
│   ├── events.py
│   └── queries.py
└── services/
    ├── __init__.py
    └── space_service.py

Test Files

tests/unit/spaces/
tests/integration/spaces/
tests/e2e/spaces/
tests/fixtures/spaces.py

---

Success Criteria

1. Phase 0 complete: project reorganized with docs/PROJECT_STRUCTURE.md
2. All phases complete with passing tests
3. HTML rendering mode fully functional
4. Directory mode with bidirectional sync working
5. GraphQL API exposing all space operations
6. CLI commands operational
7. Events propagating correctly
8. Cross-space transclusion resolving