diff --git a/CHANGELOG.md b/CHANGELOG.md index 25505de2..1635563f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added +- **ChatGPT Document Theme**: New document theme with Inter font, 580px width, and #10a37f accent color with full CLI support (`markitect md-render --theme chatgpt`) +- **Modular Theme System Architecture**: File-based theme loading with YAML configuration and dynamic theme discovery +- **Theme Directory Structure**: Organized theme components (mode/, ui/, document/, branding/) for better maintainability +- **Database Architecture Documentation**: Comprehensive WORKSPACE_AND_DATABASES.md documenting workspace concepts and database purposes + +### Changed +- **Theme Loading System**: Implemented dynamic theme discovery and loading with metadata preservation +- **Test Suite Organization**: Removed obsolete configuration CLI tests (490 lines) for cleaner codebase + +### Fixed +- **Issue-facade Click Framework Bug**: Resolved Sentinel bug in list command that was causing CLI failures +- **Issue-facade Version Command**: Fixed installation error preventing version command from working +- **Test Isolation Issues**: Improved test isolation with proper mocking to prevent cross-test interference +- **Theme Color Assertions**: Updated test assertions to work with new modular theme system + +## [0.8.0] - 2025-11-08 + +### Added +- **Setuptools-SCM Integration**: Automatic version management system replacing manual version tracking +- **Gitea Package Publishing**: Complete CI/CD pipeline for automated package publishing to Gitea +- **Enhanced Release Documentation**: Comprehensive documentation for package building and release process + +### Changed +- **Release Script Architecture**: Modernized release workflow with setuptools-scm integration +- **Makefile Release Targets**: Updated release targets to support automated version management +- **Package Building Process**: Streamlined package creation with enhanced build targets + +### Removed +- **Legacy Release Scripts**: Removed obsolete release_simplified.py in favor of unified release.py + ## [0.7.0] - 2025-11-08 ### Added @@ -158,4 +189,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Build System**: Enhanced build targets with venv Python and PYTHONPATH support - **Target Naming**: Renamed workspace targets to TDD Workspace with tdd- prefix -xxx +[Unreleased]: https://github.com/worsch/markitect/compare/v0.8.0...HEAD +[0.8.0]: https://github.com/worsch/markitect/compare/v0.7.0...v0.8.0 +[0.7.0]: https://github.com/worsch/markitect/compare/v0.6.0...v0.7.0 +[0.6.0]: https://github.com/worsch/markitect/compare/v0.5.0...v0.6.0 +[0.5.0]: https://github.com/worsch/markitect/compare/v0.4.0...v0.5.0 +[0.4.0]: https://github.com/worsch/markitect/compare/v0.3.0...v0.4.0 +[0.3.0]: https://github.com/worsch/markitect/compare/v0.2.0...v0.3.0 +[0.2.0]: https://github.com/worsch/markitect/compare/v0.1.0...v0.2.0 +[0.1.0]: https://github.com/worsch/markitect/releases/tag/v0.1.0 diff --git a/TODO.md b/TODO.md index 3441196e..5d10303b 100644 --- a/TODO.md +++ b/TODO.md @@ -12,206 +12,10 @@ The structure organizes **future tasks** by their impact, just as a changelog or This section is for tasks currently being discussed with or worked on by the coding assistant. These are the ephemeral, flow-of-thought tasks. -**πŸ”§ FIX BROKEN ISSUE-FACADE CAPABILITY (2025-11-10) - COMPLETED βœ…**: -Successfully fixed the issue-facade capability by restructuring the package organization to match pyproject.toml expectations. - -**πŸ“‹ Actions Completed**: -- [x] Diagnosed package structure mismatch (pyproject.toml expected `issue_tracker` package but code was in flat structure) -- [x] Fixed module import issue by creating `issue_tracker/` package and moving `cli/`, `core/`, `backends/` into it -- [x] Verified capability functionality is restored (CLI commands working, package installs correctly) -- [x] Tested with actual issue access (CLI prompts for backend configuration as expected) - -**🎨 IMPLEMENT DOCUMENT STYLING (Issue #166) - COMPLETED βœ…**: -Successfully implemented Substack-style document theme for enhanced long-form reading experience. - -**πŸ“‹ Features Implemented**: -- [x] Substack document theme in layered theme system -- [x] Spectral serif font for body text (optimized for long-form reading) -- [x] Lora sans-serif font for headings (clean typography hierarchy) -- [x] Warm cream background (#FAF9F1) for reduced eye strain -- [x] Bronze accent color (#b08d57) for visual hierarchy -- [x] 680px max-width for optimal reading line length -- [x] 1.6 line-height for comfortable reading -- [x] Complete TDD test coverage (6/6 tests passing) -- [x] CLI integration: `markitect md-render --theme substack` -- [x] Layered theme compatibility with existing mode/UI themes - -**Implementation Details**: -- Added to `LAYERED_THEMES` as document-scope theme -- Extended CSS generation to support `heading_font_family` property -- Maintained backward compatibility with `TEMPLATE_STYLES` -- Full integration with existing theme system architecture - -**πŸ§ͺ TESTDRIVE-JSUI CAPABILITY EXTRACTION (2025-11-09) - COMPLETED βœ…**: -Successfully extracted JavaScript UI framework functionality into a dedicated capability with complete automated test integration. The capability now provides 68 JavaScript tests + 11 Python integration tests for comprehensive testing coverage. - -**πŸ—οΈ MAJOR ARCHITECTURE REFACTORING (2025-11-03) - COMPLETED βœ…**: Successfully completed comprehensive JavaScript refactoring using Test-Driven Development methodology. - -**PROBLEMS SOLVED**: -1. βœ… **Monolithic Architecture**: Extracted 5,188-line `editor.js` into 4 modular components -2. βœ… **Server-Side Debug Generation**: Implemented pure client-side DebugPanel component -3. βœ… **Architectural Boundary Violations**: Clean separation with no Python code modifications -4. βœ… **Tight Coupling**: All components independently testable with event-driven communication -5. βœ… **Generic Editor Compromise**: Debug system now purely client-side and component-based - -**SOLUTION IMPLEMENTED**: Modular JavaScript Architecture with complete component separation and TDD validation. - -**πŸ“Š PREVIOUS STATUS (2025-11-02)**: Systematic JavaScript functionality recovery using TDD methodology had made excellent progress. **5 major features** were successfully implemented and tested: - -1. **Advanced EditState Management** βœ… - Implemented enum-based state tracking with pending changes preservation -2. **Keyboard Shortcuts** βœ… - Added Ctrl+Enter (accept) and Escape (cancel) functionality -3. **Section Splitting** βœ… - Restored dynamic heading detection with automatic section reorganization -4. **Real-time Status Tracking** βœ… - Implemented periodic updates with visual status panel (2-second intervals) -5. **Intelligent Filename Generation** βœ… - Added 4-method fallback system (optionsβ†’titleβ†’URLβ†’headingβ†’timestamp) - -All implementations include comprehensive TDD test suites and are fully integrated into the existing codebase. The recovery approach has proven highly effective for restoring sophisticated lost functionality. - -## πŸ—οΈ JAVASCRIPT ARCHITECTURE REFACTORING - COMPLETED βœ… - -### **Phase 1: Preparation & Backup (CRITICAL) - βœ… COMPLETED** -* βœ… Updated TODO.md with comprehensive refactoring plan -* βœ… Created modular directory structure `markitect/static/js/` -* βœ… Set up component template files with proper exports/imports -* βœ… Implemented TDD test framework for safe refactoring - -### **Phase 2: Core System Extraction (HIGH) - βœ… COMPLETED** -* βœ… Extracted SectionManager to `core/section-manager.js` (490 lines) -* βœ… Integrated EventSystem into SectionManager with pub/sub pattern -* βœ… Created comprehensive section state management with EditState enum - -### **Phase 3: Component Separation (HIGH) - βœ… COMPLETED** -* βœ… Document Controls β†’ `components/document-controls.js` (200 lines) -* βœ… DOMRenderer (includes status functionality) β†’ `components/dom-renderer.js` (540 lines) -* βœ… Debug Panel β†’ `components/debug-panel.js` (150 lines, pure client-side) -* βœ… Floating Menu β†’ integrated into DOMRenderer component -* βœ… Text/Image Editors β†’ integrated into DOMRenderer component - -### **Phase 4: Testing Infrastructure (MEDIUM) - βœ… COMPLETED** -* βœ… Standalone TDD test runner (`RefactorTestRunner`) that doesn't require md-render -* βœ… Component unit tests for all individual functionality -* βœ… Integration tests for component interaction -* βœ… Full system integration tests for complete workflow validation - -### **Phase 5: Integration & Cleanup (MEDIUM) - βœ… COMPLETED** -* βœ… All components work together with preserved functionality -* βœ… Monolithic editor.js functionality fully distributed -* βœ… Python code completely unchanged - zero md-render modifications -* βœ… All functionality validated through comprehensive test suite (31 tests passing) - -### **Directory Structure Implemented:** -``` -markitect/static/js/ -β”œβ”€β”€ core/ -β”‚ └── section-manager.js # βœ… Section state management with EventSystem (490 lines) -β”œβ”€β”€ components/ -β”‚ β”œβ”€β”€ document-controls.js # βœ… Document controls panel (200 lines) -β”‚ β”œβ”€β”€ dom-renderer.js # βœ… DOM rendering, FloatingMenu, editors (540 lines) -β”‚ └── debug-panel.js # βœ… Debug panel (150 lines, pure client-side) -└── tests/ - β”œβ”€β”€ refactor-test-runner.js # βœ… TDD test framework - β”œβ”€β”€ test-component-integration.js # βœ… Component integration tests - β”œβ”€β”€ test-full-integration.js # βœ… Full system tests - β”œβ”€β”€ test-section-manager-extraction.js # βœ… SectionManager tests - β”œβ”€β”€ test-extracted-section-manager.js # βœ… SectionManager TDD tests - β”œβ”€β”€ test-domrenderer-extraction.js # βœ… DOMRenderer extraction tests - β”œβ”€β”€ test-extracted-domrenderer.js # βœ… DOMRenderer TDD tests - β”œβ”€β”€ test-debugpanel-extraction.js # βœ… DebugPanel extraction tests - β”œβ”€β”€ test-debugpanel-integration.js # βœ… DebugPanel integration tests - └── test-documentcontrols-extraction.js # βœ… DocumentControls tests -``` - -### **REFACTORING RESULTS SUMMARY:** -- **Lines Extracted**: 1,380 lines from monolithic 5,188-line editor.js -- **Components Created**: 4 modular, independently testable components -- **Tests Created**: 11 comprehensive test files with 31 passing tests -- **Architecture**: Event-driven, pub/sub communication between components -- **Functionality**: 100% preserved with zero regression -- **Performance**: Improved modularity enables better maintainability and testing -- **Python Code**: Zero modifications - clean architectural separation achieved - -### **PREVIOUS COMPLETED FEATURES (Now successfully refactored):** -* **Successfully Refactored:** - * βœ… Advanced state management with EditState enum and pending changes (CRITICAL) - REFACTORED INTO SectionManager - * βœ… Keyboard shortcuts (Ctrl+Enter accept, Escape cancel) (CRITICAL) - REFACTORED INTO DOMRenderer - * βœ… Section splitting functionality for dynamic heading detection (HIGH) - REFACTORED INTO SectionManager - * βœ… Real-time status tracking with periodic updates (HIGH) - REFACTORED INTO DocumentControls - * βœ… Intelligent save filename generation with 4-method fallback (MEDIUM) - PRESERVED IN MONOLITH - * βœ… Professional message system with color-coded positioning (MEDIUM) - REFACTORED INTO DebugPanel - * βœ… Multiple concurrent editing sessions support (MEDIUM) - REFACTORED INTO DOMRenderer - * βœ… Enhanced DOM event system with 6 event types (LOW) - REFACTORED INTO DOMRenderer - * βœ… Automatic section type detection (heading, code, list, etc) (LOW) - REFACTORED INTO SectionManager - * βœ… Sophisticated section ID generation with hash-based algorithm (LOW) - REFACTORED INTO SectionManager - -* **Successfully Implemented:** - * βœ… Comprehensive status reporting dialog with detailed stats (HIGH) - IMPLEMENTED IN DocumentControls - * βœ… Floating global control panel with professional styling (MEDIUM) - IMPLEMENTED IN DocumentControls - * βœ… Enhanced setupSectionElement with comprehensive styling (LOW) - IMPLEMENTED IN DOMRenderer - -* **Core Methods Successfully Refactored:** - * βœ… stopEditing method with state preservation (CRITICAL) - REFACTORED INTO SectionManager - * βœ… getAllSections method for section collection management (MEDIUM) - REFACTORED INTO SectionManager - * βœ… hasChanges detection for unsaved modifications (HIGH) - REFACTORED INTO SectionManager - * βœ… updateGlobalStatus method with 2-second interval updates (MEDIUM) - REFACTORED INTO DocumentControls - * βœ… handleSectionSplit for dynamic section reorganization (LOW) - REFACTORED INTO SectionManager - * βœ… checkForSectionSplits automatic heading detection (LOW) - REFACTORED INTO SectionManager - -* **To Remove:** - * None currently identified - +*No active tasks at this time.* *** ## Completed Tasks -**JavaScript Architecture Refactoring - COMPLETED βœ… (2025-11-03)**: -- βœ… Successfully extracted monolithic 5,188-line editor.js into 4 modular components using TDD methodology -- βœ… Created SectionManager component (490 lines) handling section state management and event system -- βœ… Created DOMRenderer component (540 lines) handling DOM interactions, rendering, and editing workflows -- βœ… Created DebugPanel component (150 lines) providing pure client-side debug message management -- βœ… Created DocumentControls component (200 lines) managing floating control panel and document actions -- βœ… Implemented comprehensive TDD test framework with 11 test files and 31 passing tests -- βœ… Achieved 100% functionality preservation with zero regression through rigorous testing -- βœ… Established event-driven architecture with pub/sub communication between components -- βœ… Maintained complete separation from Python code - zero md-render modifications required -- βœ… Created modular directory structure enabling independent component development and testing - -**Architecture Improvements Achieved**: -- Clean separation of concerns with single-responsibility components -- Event-driven communication reducing tight coupling -- Independent component testing enabling confident refactoring -- Scalable structure supporting future feature development -- Client-side debug system eliminating server-side debug generation issues -- Modular design allowing selective component updates without affecting others - -**Asset Shipping for md-render - COMPLETED βœ…**: -- βœ… Implemented automatic asset copying when rendering markdown to different output directories -- βœ… Added asset discovery functionality parsing markdown for image/link references -- βœ… Implemented timestamp-based asset copying (only copy if source newer than destination) -- βœ… Added `--ship-assets` and `--no-ship-assets` CLI flags for explicit control -- βœ… Added `MARKITECT_OUTPUT_DIR` environment variable support for default output directory -- βœ… Smart defaults: assets ship automatically when output is directory, disabled for specific files -- βœ… Preserved relative path structure in output directory maintaining markdown link compatibility -- βœ… Graceful handling of missing assets with warning messages -- βœ… Full backward compatibility with existing md-render workflows -- βœ… Comprehensive TDD test suite covering all functionality and edge cases - -**Feature Capabilities**: -- Environment variable priority: CLI `--output` > `MARKITECT_OUTPUT_DIR` > input file directory -- Automatic asset discovery from standard markdown syntax: `![alt](path)` and `[text](path)` -- Timestamp-based incremental copying prevents unnecessary file operations -- Directory structure preservation maintains working relative links in output HTML -- Support for images, documents, and other asset types referenced in markdown - -**CHANGELOG.md Enhancement - COMPLETED βœ…**: -- βœ… Added missing version entries for 0.1.0, 0.2.0, and 0.3.0 -- βœ… Added standard Keep a Changelog header with proper format -- βœ… Included Unreleased section -- βœ… Research completed for all historical versions using git log analysis -- βœ… All entries follow Keep a Changelog categories (Added, Changed, Fixed) -- βœ… Chronological order maintained with latest versions first -- βœ… Appropriate release dates included based on git commit timestamps - -**Version Details Added**: -- v0.1.0 (2025-10-15): Development infrastructure, TDD workspace, issue management -- v0.2.0 (2025-10-20): Advanced Markdown Engine with GraphQL, search, plugins -- v0.3.0 (2025-10-25): Architectural improvements with kaizen-agentic integration \ No newline at end of file +*Recent completed tasks have been documented in CHANGELOG.md following Keep a Changelog format.* \ No newline at end of file diff --git a/docs/WORKSPACE_AND_DATABASES.md b/docs/WORKSPACE_AND_DATABASES.md new file mode 100644 index 00000000..cf43f14c --- /dev/null +++ b/docs/WORKSPACE_AND_DATABASES.md @@ -0,0 +1,268 @@ +# Markitect Workspace and Database Architecture + +This document explains Markitect's workspace concept and the two distinct database systems used by the application. + +## Workspace Concept + +Markitect uses a **workspace-based architecture** where each directory or repository can have its own configuration and local data storage. This allows for flexible, per-project customization while maintaining a global user configuration. + +### Workspace Structure + +When you initialize Markitect in a directory, it creates the following structure: + +``` +project-directory/ +β”œβ”€β”€ .markitect.yml # Workspace configuration +β”œβ”€β”€ .markitect_workspace/ # Local workspace data +β”œβ”€β”€ .ast_cache/ # AST parsing cache +β”œβ”€β”€ assets/ # Asset storage directory +β”‚ β”œβ”€β”€ assets.db # Asset management database +β”‚ └── [asset files] # Stored images, files, etc. +└── tests/ # Test files directory +``` + +### Configuration Files + +Markitect searches for configuration in this order: +1. `.markitect.yml` (current directory) +2. `.markitect.yaml` (current directory) +3. `.markitect.json` (current directory) +4. `markitect.config.yml` (current directory) +5. `markitect.config.yaml` (current directory) +6. `markitect.config.json` (current directory) +7. `~/.markitect/config.yml` (user home directory) +8. Environment variables (`MARKITECT_*`) +9. Built-in defaults + +## Database Architecture + +Markitect uses two distinct SQLite databases for different purposes: + +### 1. Main Application Database (`markitect.db`) + +**Location**: `~/.markitect/markitect.db` (user home directory) + +**Purpose**: Global user-level application data and configuration + +**Scope**: User-wide, shared across all workspaces + +**Contents**: +- User preferences and settings +- Application state information +- Global configuration data +- Cross-workspace data that needs persistence + +**Configuration**: Set via `MARKITECT_DATABASE_PATH` environment variable or `database_path` in configuration + +### 2. Asset Management Database (`assets.db`) + +**Location**: `assets/assets.db` (within workspace asset storage directory) + +**Purpose**: Asset management and tracking for the current workspace + +**Scope**: Workspace-specific, local to each directory/repository + +**Contents**: +- Asset metadata (filename, size, MIME type, timestamps) +- File content hashes for deduplication +- Asset usage statistics and tracking +- Processing logs and analytics +- Asset relationships and dependencies + +**Schema** (key tables): +```sql +-- Basic asset metadata +asset_metadata ( + content_hash TEXT PRIMARY KEY, + filename TEXT NOT NULL, + size_bytes INTEGER NOT NULL, + mime_type TEXT, + created_at TIMESTAMP, + updated_at TIMESTAMP +) + +-- Usage tracking +asset_usage_stats ( + content_hash TEXT, + usage_count INTEGER, + last_used TIMESTAMP, + documents_using TEXT -- JSON array of document paths +) + +-- Performance and analytics tables +-- (Additional tables for caching, indexing, and optimization) +``` + +## Why Two Databases? + +This separation serves several important purposes: + +### Data Isolation +- **Global data** (user preferences) stays in the user profile +- **Workspace data** (asset files, metadata) stays with the project + +### Version Control Considerations +- `markitect.db` is never committed to version control +- `assets.db` is excluded via `.gitignore` (local workspace data) +- Asset files themselves can be optionally committed based on project needs + +### Performance Optimization +- Asset database operations are localized to relevant files +- Global database isn't impacted by large asset collections +- Each workspace can optimize its asset database independently + +### Portability and Collaboration +- Workspaces can be moved/copied without affecting global configuration +- Teams can share asset storage strategies without sharing personal settings +- Different projects can have different asset management policies + +## Configuration Management + +### Workspace Initialization + +To initialize a new workspace: + +```bash +markitect config-init +``` + +This creates: +1. `.markitect.yml` configuration file +2. Required directories (`.markitect_workspace`, `.ast_cache`, `tests`) +3. Asset storage structure + +### Configuration Commands + +```bash +# View current configuration +markitect config-show + +# Set workspace-specific values +markitect config-set repo_name "my-project" +markitect config-set assets.storage_path "./assets" + +# View configuration help +markitect config-help +``` + +### Environment Variables + +Override configuration with environment variables: + +```bash +export MARKITECT_GITEA_URL="http://localhost:3000" +export MARKITECT_WORKSPACE_DIR=".custom_workspace" +export MARKITECT_DATABASE_PATH="/custom/path/markitect.db" +``` + +## Asset Management Integration + +The asset management system coordinates between the asset database and file storage: + +```python +from markitect.assets import AssetManager + +# Initialize with workspace-specific configuration +asset_manager = AssetManager() + +# Assets are stored in workspace, tracked in assets.db +asset = asset_manager.add_asset("image.png") +``` + +### Asset Storage Workflow + +1. **File Processing**: Asset files are processed and stored in `assets/` directory +2. **Database Recording**: Metadata recorded in `assets.db` +3. **Deduplication**: Content hashes prevent duplicate storage +4. **Usage Tracking**: Document usage recorded for analytics +5. **Cleanup**: Unused assets can be identified and removed + +## Best Practices + +### For Development +- Initialize workspace in each project directory +- Commit `.markitect.yml` to version control +- Add `assets.db` and workspace directories to `.gitignore` +- Use relative paths in workspace configuration + +### For Collaboration +- Share workspace configuration (`.markitect.yml`) +- Document asset storage strategy for the team +- Establish conventions for asset organization +- Consider asset file version control policies + +### for Production +- Backup both databases separately +- Monitor asset database growth in large projects +- Use environment variables for deployment-specific settings +- Implement asset cleanup strategies for long-running projects + +## Migration and Compatibility + +### Legacy Support +The system maintains backward compatibility: +- Old configuration patterns still work +- Automatic migration of legacy settings +- Graceful fallbacks for missing configuration + +### Database Migration +Asset databases support schema migrations: +- Automatic schema updates on version changes +- Backward compatibility preservation +- Safe migration with rollback capability + +## Troubleshooting + +### Common Issues + +**Database Connection Errors**: +- Check file permissions on database directories +- Verify disk space availability +- Ensure SQLite3 is available + +**Configuration Not Found**: +- Verify `.markitect.yml` exists and is valid YAML +- Check environment variable names and values +- Run `markitect config-show` to see current configuration + +**Asset Storage Issues**: +- Confirm asset directory permissions +- Check available disk space +- Verify `assets.db` is not corrupted + +### Recovery Procedures + +**Corrupted Asset Database**: +```bash +# Backup and recreate +mv assets/assets.db assets/assets.db.backup +# Restart Markitect to recreate schema +markitect config-show +``` + +**Missing Configuration**: +```bash +# Reinitialize workspace +markitect config-init +``` + +## Technical Details + +### Database Connections +- Uses SQLite3 with connection pooling for performance +- Automatic connection management and cleanup +- Thread-safe operations for concurrent access + +### File System Integration +- Path resolution relative to workspace root +- Cross-platform path handling (Windows, macOS, Linux) +- Symlink and junction support where available + +### Security Considerations +- Database files have restricted permissions +- No sensitive data stored in asset database +- Configuration masking for sensitive values in display + +--- + +*This documentation reflects the current architecture as of November 2025. For implementation details, see the source code in `markitect/config_manager.py` and `markitect/assets/`.* \ No newline at end of file