Compare commits
282 Commits
61e820baf8
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| cd8339ecef | |||
| f8ab58edbe | |||
| 2b5e9743fe | |||
| 753c3d4fc6 | |||
| 94e84f0db9 | |||
| a765ccda21 | |||
| 4472fa6c7f | |||
| 526fa1e3bc | |||
| 86de18c247 | |||
| ca9d0d7030 | |||
| bc527ec09a | |||
| ce984482e2 | |||
| 9266f124e6 | |||
| 8740a66611 | |||
| b7e9edbb4b | |||
| 479fa95fdf | |||
| eb9b622499 | |||
| e3e5b8ecc1 | |||
| 9e8d73fa7d | |||
| d44a4cd3df | |||
| c0615c2d50 | |||
| 965508ec06 | |||
| f325f89dc9 | |||
| 36a5136bdf | |||
| b7e11461f4 | |||
| 3966814868 | |||
| f4610a46e3 | |||
| 0d95e6dbcf | |||
| 36c20f37d0 | |||
| 72b87fd82e | |||
| eaf4a955af | |||
| e9dc9a8517 | |||
| b055c8d7bb | |||
| ef3d47779e | |||
| d1f57272a4 | |||
| a9ca0adfcf | |||
| 81a4c8796a | |||
| 2d45425b25 | |||
| fa27572f43 | |||
| dfab3d598b | |||
| 34ed7a6fab | |||
| 7f1eecbdb2 | |||
| 574bb11db6 | |||
| 8f00fa2018 | |||
| c861520ccd | |||
| 9c32ad1837 | |||
| 7c38f9b427 | |||
| dfe56a4f9b | |||
| 0f54f094e4 | |||
| 4a15a50337 | |||
| 92dfe367c7 | |||
| 23c397e46a | |||
| e695ddfbbd | |||
| 5245dbbfc8 | |||
| 4319d2a32b | |||
| efdaa884c8 | |||
| 2804de3d24 | |||
| 3e96ac7b8d | |||
| a687e508f3 | |||
| da9c5fce80 | |||
| cd87ebfdc0 | |||
| 666f78d1ba | |||
| 579e02989b | |||
| 8401c69ff2 | |||
| 1b9a31665c | |||
| 06e904ccf5 | |||
| 59d42b1665 | |||
| 8c11e13fef | |||
| ac4e508aff | |||
| 8e1943afdb | |||
| 05711e541d | |||
| 8cb9ee6f6e | |||
| db129fde6b | |||
| 6d9ec4e34b | |||
| 679f482e49 | |||
| 368571905a | |||
| 9c95912d68 | |||
| 0828581269 | |||
| 283abac378 | |||
| 90ca14dd85 | |||
| 098b781f92 | |||
| eea397a380 | |||
| 7615beb139 | |||
| c2e06c15d7 | |||
| df1fdf1842 | |||
| 5ede1de4b8 | |||
| 72d9904485 | |||
| c594bc3a38 | |||
| 77dd3fee6d | |||
| 4e0b27b075 | |||
| 8095a1da4c | |||
| ce30f874d5 | |||
| 715ef19d1c | |||
| 3ac8447c10 | |||
| 94cb2063af | |||
| d1c6e53754 | |||
| b76d6d38c1 | |||
| ce7f78d57d | |||
| 11585e6968 | |||
| 3461d2f354 | |||
| 3726503adb | |||
| b20fe4db68 | |||
| 144a88c0c2 | |||
| dc22017b7c | |||
| f8c9ab33f0 | |||
| bad01e32bd | |||
| 267368eb60 | |||
| 9031e1162c | |||
| 03c6c5e8de | |||
| b5e994b014 | |||
| 4ce856d4d0 | |||
| 2f0989f9bf | |||
| 60f33443ae | |||
| 120ed89780 | |||
| 9fa239c140 | |||
| e4fbba8a57 | |||
| ac334c679d | |||
| 69aea1ada7 | |||
| be3b4e3aae | |||
| ad23bb0b86 | |||
| 5085c44de3 | |||
| 4631a9f794 | |||
| 269184f7a1 | |||
| 69e2ec25ff | |||
| 41773f1320 | |||
| 880c1d1374 | |||
| 2d1282a61e | |||
| 706981c39f | |||
| 01b9596ce6 | |||
| ad84dd3a41 | |||
| e806a701ca | |||
| fecc2fd4fa | |||
| 360c3b1de2 | |||
| 8f54a5509e | |||
| 7b4bd461c9 | |||
| 704272644c | |||
| bd1d05ba79 | |||
| 9ce157400e | |||
| c56c92c815 | |||
| 5f463e5b20 | |||
| e6840fe696 | |||
| 945544880d | |||
| cbde1dabc4 | |||
| 6af04024d5 | |||
| 9ab135bb03 | |||
| 27847df9bd | |||
| 9061b2fd85 | |||
| 4588cbeee8 | |||
| 727ce4d3c5 | |||
| 7de57a389d | |||
| 535b83996b | |||
| 2a5c265458 | |||
| 7da77396a9 | |||
| 0a494b2011 | |||
| 9b12875681 | |||
| 6ebcc0f60e | |||
| 72519bf83d | |||
| 1f9d618777 | |||
| ce11c03326 | |||
| 0ade4798f3 | |||
| 843f579305 | |||
| 7515b9c0e5 | |||
| 7f696582a9 | |||
| 5fea98b068 | |||
| 0b5098370a | |||
| 599de22f59 | |||
| 23521ad6ae | |||
| 0d276e8589 | |||
| 587d2f5889 | |||
| bf4767d06b | |||
| 75c8f8c325 | |||
| 6852ad915e | |||
| c4ee5cc645 | |||
| 061ba88206 | |||
| 4e9117ddcb | |||
| 5e3646fdff | |||
| fc828a345b | |||
| 4d72ee8032 | |||
| 689fb21774 | |||
| 20c0cfece7 | |||
| 0d78837a53 | |||
| 2836ae14de | |||
| 5264a6083c | |||
| a969c5de47 | |||
| f27eea6b5b | |||
| ae2e8ee4a7 | |||
| b10d2fd3d0 | |||
| 92719ff424 | |||
| 9026646594 | |||
| 77415bfad7 | |||
| 5e147865f8 | |||
| 3003b9b8da | |||
| d32dc41315 | |||
| f19a88f1d5 | |||
| 7d115b6325 | |||
| 60d9f7a2c3 | |||
| f3aaec99bb | |||
| b81ce5631d | |||
| 14108533fb | |||
| b6f95066a3 | |||
| 6df9b5df05 | |||
| 82c1a3ab65 | |||
| da34303057 | |||
| d2cd2d22fd | |||
| 48e0b60be5 | |||
| 2b35fcde62 | |||
| c46d9f7a0b | |||
| 2b687a4ca8 | |||
| d68e762612 | |||
| b51999582e | |||
| b4157da3dd | |||
| 916c09a22b | |||
| 4d899d0690 | |||
| dcb51b7e3a | |||
| d0432dbe0d | |||
| 45e4c7a6e9 | |||
| 01e5c811ab | |||
| 9fe2960842 | |||
| 7be37df3e4 | |||
| 21189f7664 | |||
| ddd8189576 | |||
| 2e6f292e48 | |||
| a1476a98b5 | |||
| 304959b3ee | |||
| 83086b3773 | |||
| 82eef76366 | |||
| 2838135450 | |||
| d592c5b8b3 | |||
| e84eb08dc5 | |||
| 0e568ce623 | |||
| aa0ac626c5 | |||
| 9bbc2832de | |||
| 46a060b695 | |||
| 24959308b2 | |||
| 6670e71b81 | |||
| ab3f0db86f | |||
| d0a1c91b8e | |||
| 3264517c91 | |||
| d98c3ae05a | |||
| 4e3f112987 | |||
| f788ccdfd3 | |||
| 512085d283 | |||
| 95ea13958a | |||
| ca431ac11f | |||
| 79c6c9d4e4 | |||
| 09e7f07c23 | |||
| 8d8a4ed0c3 | |||
| 5b13c00d3e | |||
| 4262310302 | |||
| 6ef2641bff | |||
| b9c1b90867 | |||
| 76b5bb1106 | |||
| 409d1a8d9f | |||
| 8f1cc0faf9 | |||
| 8ef356af57 | |||
| 55c61a7f2d | |||
| 26c235e296 | |||
| 4d08cbcf52 | |||
| e0bc5daeeb | |||
| de49c76ff9 | |||
| dbde13e036 | |||
| 3839a6761e | |||
| 2d9175ec05 | |||
| b963940144 | |||
| 2d516b205a | |||
| 7270bc559d | |||
| c699d7d669 | |||
| bcc3fe1df5 | |||
| d1e129c9b8 | |||
| afe6bcf6fe | |||
| 32d26e7648 | |||
| 747b77b854 | |||
| 9b6c3d4ad0 | |||
| 746a3f9df1 | |||
| 499de7a46e | |||
| b512842aaf | |||
| c4877543d5 | |||
| 47657fcba8 | |||
| 17c62aadaa | |||
| 23551129a3 | |||
| f3237f7ada | |||
| b475a23697 |
323
.claude/capabilities/issue-facade.md
Normal file
323
.claude/capabilities/issue-facade.md
Normal file
@@ -0,0 +1,323 @@
|
||||
# Issue Facade - Agent Integration Context
|
||||
|
||||
**🤖 For Coding Agents: Read this to understand how to use issue tracking in this project.**
|
||||
|
||||
## Critical: DO NOT Bypass This Capability
|
||||
|
||||
⚠️ **IMPORTANT:** If you need to work with issues, **you MUST use this capability**. Do NOT:
|
||||
- ❌ Make direct API calls to Gitea/GitHub/GitLab
|
||||
- ❌ Use platform CLIs (gh, glab, etc.)
|
||||
- ❌ Import platform libraries (PyGithub, python-gitlab, etc.)
|
||||
- ❌ Parse HTML/scrape issue tracker web UIs
|
||||
|
||||
**Why?** Bypassing this capability causes:
|
||||
- Credential management chaos (tokens scattered everywhere)
|
||||
- Inconsistent issue state across agents
|
||||
- Massive token waste (redundant API calls)
|
||||
- Platform lock-in (can't switch Gitea → GitHub easily)
|
||||
- Race conditions in multi-agent scenarios
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Check if Capability is Available
|
||||
|
||||
```bash
|
||||
# Verify installation
|
||||
issue --version
|
||||
# or
|
||||
python -c "from issue_tracker.backends.gitea import GiteaBackend; print('OK')"
|
||||
```
|
||||
|
||||
### Basic Usage (Python)
|
||||
|
||||
```python
|
||||
from issue_tracker.backends.gitea import GiteaBackend
|
||||
from issue_tracker.core.models import Issue, Label, IssueState, User, Comment
|
||||
from issue_tracker.core.interfaces import IssueFilter
|
||||
from datetime import datetime, timezone
|
||||
import os
|
||||
|
||||
# Connect (assumes backend is configured)
|
||||
backend = GiteaBackend()
|
||||
backend.connect({
|
||||
'base_url': os.environ['GITEA_URL'],
|
||||
'token': os.environ['GITEA_API_TOKEN'],
|
||||
'owner': os.environ['GITEA_OWNER'],
|
||||
'repo': os.environ['GITEA_REPO']
|
||||
})
|
||||
|
||||
# List issues for me
|
||||
my_issues = backend.list_issues(IssueFilter(
|
||||
state='open',
|
||||
assignee='my-agent-id',
|
||||
labels=['needs-implementation']
|
||||
))
|
||||
|
||||
# Create issue
|
||||
new_issue = Issue(
|
||||
id=None, number=0,
|
||||
title="Implement feature X",
|
||||
description="Details...",
|
||||
state=IssueState.OPEN,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
labels=[Label(name="feature"), Label(name="priority:high")]
|
||||
)
|
||||
created = backend.create_issue(new_issue)
|
||||
|
||||
# Update issue
|
||||
created.state = IssueState.IN_PROGRESS
|
||||
created.assignees = [User(id="agent-id", username="agent-id")]
|
||||
backend.update_issue(created)
|
||||
|
||||
# Add comment
|
||||
comment = Comment(
|
||||
id=None,
|
||||
body="Implementation started. Working on database schema.",
|
||||
author=User(id="agent-id", username="agent-id"),
|
||||
created_at=datetime.now(timezone.utc)
|
||||
)
|
||||
backend.add_comment(created.id, comment)
|
||||
|
||||
# Close when done
|
||||
created.state = IssueState.CLOSED
|
||||
created.closed_at = datetime.now(timezone.utc)
|
||||
backend.update_issue(created)
|
||||
```
|
||||
|
||||
### Basic Usage (CLI)
|
||||
|
||||
```bash
|
||||
# List my open issues
|
||||
issue list --state=open --assignee=agent-id --format=json
|
||||
|
||||
# Create issue
|
||||
issue create "Implement feature X" \
|
||||
--label=feature \
|
||||
--label=priority:high \
|
||||
--description="Details here"
|
||||
|
||||
# Update state
|
||||
issue edit 42 --state=in_progress --assignee=agent-id
|
||||
|
||||
# Add comment
|
||||
issue comment 42 "Implementation started"
|
||||
|
||||
# Close
|
||||
issue close 42 --comment="Completed successfully"
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 1: Find Work
|
||||
|
||||
```python
|
||||
# Get next available task
|
||||
available_tasks = backend.list_issues(IssueFilter(
|
||||
state='open',
|
||||
labels=['ready', 'needs-implementation']
|
||||
))
|
||||
|
||||
# Filter to unassigned
|
||||
unassigned = [t for t in available_tasks if not t.assignees]
|
||||
|
||||
if unassigned:
|
||||
task = unassigned[0]
|
||||
# Claim it...
|
||||
```
|
||||
|
||||
### Pattern 2: Claim Issue (Prevent Race Conditions)
|
||||
|
||||
```python
|
||||
def claim_issue(issue: Issue, agent_id: str) -> bool:
|
||||
"""Claim an issue safely."""
|
||||
# Check if already claimed
|
||||
if issue.assignees:
|
||||
return False # Already taken
|
||||
|
||||
# Claim it
|
||||
issue.state = IssueState.IN_PROGRESS
|
||||
issue.assignees = [User(id=agent_id, username=agent_id)]
|
||||
backend.update_issue(issue)
|
||||
|
||||
# Announce claim
|
||||
backend.add_comment(issue.id, Comment(
|
||||
id=None,
|
||||
body=f"🤖 Claimed by {agent_id}",
|
||||
author=User(id=agent_id, username=agent_id),
|
||||
created_at=datetime.now(timezone.utc)
|
||||
))
|
||||
return True
|
||||
```
|
||||
|
||||
### Pattern 3: Progress Updates
|
||||
|
||||
```python
|
||||
def report_progress(issue: Issue, message: str, agent_id: str):
|
||||
"""Report progress on an issue."""
|
||||
backend.add_comment(issue.id, Comment(
|
||||
id=None,
|
||||
body=f"**Progress Update:**\n\n{message}",
|
||||
author=User(id=agent_id, username=agent_id),
|
||||
created_at=datetime.now(timezone.utc)
|
||||
))
|
||||
```
|
||||
|
||||
### Pattern 4: Agent-to-Agent Communication
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
def post_agent_message(issue_id: str, msg_type: str, data: dict, agent_id: str):
|
||||
"""Post structured message for other agents."""
|
||||
message = {
|
||||
'type': msg_type,
|
||||
'agent': agent_id,
|
||||
'timestamp': datetime.now(timezone.utc).isoformat(),
|
||||
'data': data
|
||||
}
|
||||
backend.add_comment(issue_id, Comment(
|
||||
id=None,
|
||||
body=f"```agent-message\n{json.dumps(message, indent=2)}\n```",
|
||||
author=User(id=agent_id, username=agent_id),
|
||||
created_at=datetime.now(timezone.utc)
|
||||
))
|
||||
|
||||
def read_agent_messages(issue_id: str, msg_type: str = None):
|
||||
"""Read messages from other agents."""
|
||||
comments = backend.get_comments(issue_id)
|
||||
messages = []
|
||||
for comment in comments:
|
||||
if '```agent-message' in comment.body:
|
||||
try:
|
||||
json_str = comment.body.split('```agent-message\n')[1].split('\n```')[0]
|
||||
msg = json.loads(json_str)
|
||||
if msg_type is None or msg['type'] == msg_type:
|
||||
messages.append(msg)
|
||||
except:
|
||||
continue
|
||||
return messages
|
||||
```
|
||||
|
||||
## Configuration Check
|
||||
|
||||
Before using issue tracking, verify configuration:
|
||||
|
||||
```python
|
||||
def verify_issue_backend() -> bool:
|
||||
"""Verify issue backend is configured."""
|
||||
try:
|
||||
backend = GiteaBackend()
|
||||
backend.connect({
|
||||
'base_url': os.environ['GITEA_URL'],
|
||||
'token': os.environ['GITEA_API_TOKEN'],
|
||||
'owner': os.environ['GITEA_OWNER'],
|
||||
'repo': os.environ['GITEA_REPO']
|
||||
})
|
||||
return backend.test_connection()
|
||||
except Exception as e:
|
||||
print(f"Issue backend not configured: {e}")
|
||||
return False
|
||||
|
||||
# Use it
|
||||
if not verify_issue_backend():
|
||||
print("ERROR: Issue tracking not available. Check configuration.")
|
||||
sys.exit(1)
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```python
|
||||
from issue_tracker.backends.gitea.backend import GiteaAPIError
|
||||
|
||||
try:
|
||||
issue = backend.get_issue_by_number(42)
|
||||
except GiteaAPIError as e:
|
||||
if e.status_code == 404:
|
||||
print("Issue not found")
|
||||
elif e.status_code == 401:
|
||||
print("Authentication failed - check GITEA_API_TOKEN")
|
||||
elif e.status_code == 429:
|
||||
print("Rate limited - wait and retry")
|
||||
else:
|
||||
print(f"API error: {e}")
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Use filters** instead of fetching all issues:
|
||||
```python
|
||||
# BAD: Get all, filter in Python
|
||||
all_issues = backend.list_issues()
|
||||
my_issues = [i for i in all_issues if i.assignees and i.assignees[0].username == 'me']
|
||||
|
||||
# GOOD: Filter at backend
|
||||
my_issues = backend.list_issues(IssueFilter(assignee='me'))
|
||||
```
|
||||
|
||||
2. **Use JSON output** for CLI parsing:
|
||||
```bash
|
||||
issue list --format=json | jq '.[] | select(.state == "open")'
|
||||
```
|
||||
|
||||
3. **Batch comments** instead of rapid-fire updates
|
||||
|
||||
4. **Check local cache** before querying (if available)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Backend not configured"
|
||||
```bash
|
||||
# Check config
|
||||
issue backend list
|
||||
|
||||
# If empty, configure
|
||||
export GITEA_API_TOKEN="your-token"
|
||||
issue backend add myproject gitea
|
||||
issue backend set-default myproject
|
||||
```
|
||||
|
||||
### "Authentication failed"
|
||||
```bash
|
||||
# Verify token
|
||||
curl -H "Authorization: token $GITEA_API_TOKEN" $GITEA_URL/api/v1/user
|
||||
```
|
||||
|
||||
### "Issue not found"
|
||||
```python
|
||||
# Use get_issue_by_number, not get_issue
|
||||
issue = backend.get_issue_by_number(42) # Correct
|
||||
# issue = backend.get_issue("42") # Wrong - needs backend_id
|
||||
```
|
||||
|
||||
## Full Documentation
|
||||
|
||||
- **Integration Guide:** `AGENT_INTEGRATION.md` (comprehensive patterns and strategies)
|
||||
- **API Reference:** `CLAUDE.md` (for developers extending the capability)
|
||||
- **Examples:** `examples/agents/` (working agent implementations)
|
||||
- **Roadmap:** `ROADMAP.md` (upcoming features)
|
||||
|
||||
## Current Limitations (v1.0)
|
||||
|
||||
Be aware of these limitations:
|
||||
|
||||
1. **Manual Configuration:** Backend must be configured before use (auto-detect in v1.1)
|
||||
2. **User Context:** Uses hardcoded user for CLI operations (agent identity in v1.2)
|
||||
3. **No Built-in Locking:** Use assignee + comment workaround for claiming (native in v1.2)
|
||||
4. **Basic Conflicts:** Manual resolution required for complex sync conflicts (advanced in v2.0)
|
||||
|
||||
Workarounds are documented in `AGENT_INTEGRATION.md`.
|
||||
|
||||
## Questions?
|
||||
|
||||
If you're unsure whether to use this capability for something:
|
||||
|
||||
**ASK:** "Does this involve creating, reading, updating, or searching issues?"
|
||||
- **YES** → Use this capability
|
||||
- **NO** → You can use other methods
|
||||
|
||||
**Example:**
|
||||
- "Create an issue for the bug I found" → **Use issue-facade**
|
||||
- "Read the project README" → Don't need issue-facade
|
||||
- "Check if issue #42 exists" → **Use issue-facade**
|
||||
- "Clone the repository" → Don't need issue-facade
|
||||
6
.claude/commands/use-issues.md
Normal file
6
.claude/commands/use-issues.md
Normal file
@@ -0,0 +1,6 @@
|
||||
Use the issue-facade capability for all issue tracking operations.
|
||||
|
||||
**Quick reference:** See `.claude/capabilities/issue-facade.md`
|
||||
**Examples:** See `capabilities/issue-facade/examples/agents/`
|
||||
|
||||
**DO NOT use direct API calls or platform CLIs!**
|
||||
8
.claude/context/capabilities.md
Normal file
8
.claude/context/capabilities.md
Normal file
@@ -0,0 +1,8 @@
|
||||
# Available Capabilities
|
||||
|
||||
## Issue Tracking: issue-facade
|
||||
|
||||
**CRITICAL:** Always use this for issue operations. Never bypass with direct API calls.
|
||||
|
||||
**Docs:** `.claude/capabilities/issue-facade.md`
|
||||
**Usage:** `/use-issues`
|
||||
20
.claude/rules/agents.md
Normal file
20
.claude/rules/agents.md
Normal file
@@ -0,0 +1,20 @@
|
||||
## Kaizen Agents
|
||||
|
||||
Specialized agent personas available on demand via the state-hub MCP.
|
||||
|
||||
**Discover:** `list_kaizen_agents()` — returns all agents with name, description, category
|
||||
**Load:** `get_kaizen_agent("tdd-workflow")` — returns full instructions; read and follow them
|
||||
|
||||
Common agents:
|
||||
|
||||
| Agent | Category | When to use |
|
||||
|-------|----------|-------------|
|
||||
| `tdd-workflow` | testing | Step-by-step TDD8 workflow for any feature |
|
||||
| `code-refactoring` | quality | Code quality analysis and safe refactoring |
|
||||
| `test-maintenance` | testing | Diagnose and fix failing tests |
|
||||
| `requirements-engineering` | process | Prevent interface/mock mismatches upfront |
|
||||
| `keepaTodofile` | process | Maintain TODO.md during work |
|
||||
| `project-management` | process | Track status, determine next steps |
|
||||
| `datamodel-optimization` | quality | Optimize dataclasses and data structures |
|
||||
|
||||
All 17 agents: call `list_kaizen_agents()` for the full list.
|
||||
8
.claude/rules/architecture.md
Normal file
8
.claude/rules/architecture.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Architecture
|
||||
|
||||
<!-- TODO: Describe the key design decisions and component structure.
|
||||
Key modules, data flows, external integrations, state machines, etc. -->
|
||||
|
||||
## Quick Reference
|
||||
|
||||
`~/state-hub/mcp_server/TOOLS.md` — MCP tool reference
|
||||
50
.claude/rules/credential-routing.md
Normal file
50
.claude/rules/credential-routing.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=markitect-main` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
38
.claude/rules/first-session.md
Normal file
38
.claude/rules/first-session.md
Normal file
@@ -0,0 +1,38 @@
|
||||
## First Session Protocol
|
||||
|
||||
Triggered when `get_domain_summary("communication")` shows **no workstreams**.
|
||||
The project is registered but work has not yet been structured.
|
||||
|
||||
**Step 1 — Read, don't write**
|
||||
- `~/the-custodian/canon/projects/communication/project_charter_v0.1.md` — purpose, scope
|
||||
- `~/the-custodian/canon/projects/communication/roadmap_v0.1.md` — planned phases
|
||||
- Scan repo root: README, directory structure, existing code or docs
|
||||
|
||||
**Step 2 — Survey in-progress work**
|
||||
Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete.
|
||||
|
||||
**Step 3 — Propose workstreams to Bernd**
|
||||
Propose 1–3 workstreams — each a coherent strand, weeks to months, anchored to a
|
||||
roadmap phase. **Wait for approval before creating.**
|
||||
|
||||
**Step 4 — Create workplan file first, then DB record (ADR-001)**
|
||||
```
|
||||
workplans/MARKITECT-WP-NNNN-<slug>.md ← write this first
|
||||
```
|
||||
Then register in the hub:
|
||||
```
|
||||
create_workstream(topic_id="36c7421b-c537-4723-bf75-42a3ebc6a1dc", title="...", owner="...", description="...")
|
||||
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
|
||||
```
|
||||
|
||||
**Step 5 — Record the setup**
|
||||
```
|
||||
add_progress_event(
|
||||
summary="First session: structured communication into N workstreams, M tasks",
|
||||
event_type="milestone",
|
||||
topic_id="36c7421b-c537-4723-bf75-42a3ebc6a1dc",
|
||||
detail={"workstreams": [...], "tasks_created": M}
|
||||
)
|
||||
```
|
||||
|
||||
<!-- Delete or archive this file once past first session -->
|
||||
8
.claude/rules/repo-boundary.md
Normal file
8
.claude/rules/repo-boundary.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Repo boundary
|
||||
|
||||
This repo owns **Markitect Main** only. It does not own:
|
||||
|
||||
<!-- TODO: List what belongs in adjacent repos, e.g.:
|
||||
- SSH key management → railiance-infra/
|
||||
- State hub code → state-hub/
|
||||
-->
|
||||
5
.claude/rules/repo-identity.md
Normal file
5
.claude/rules/repo-identity.md
Normal file
@@ -0,0 +1,5 @@
|
||||
**Purpose:** Markitect Main - (fill in purpose)
|
||||
|
||||
**Domain:** communication
|
||||
**Repo slug:** markitect-main
|
||||
**Topic ID:** 36c7421b-c537-4723-bf75-42a3ebc6a1dc
|
||||
85
.claude/rules/session-protocol.md
Normal file
85
.claude/rules/session-protocol.md
Normal file
@@ -0,0 +1,85 @@
|
||||
## Session Protocol
|
||||
|
||||
Dev Hub (State Hub API): http://127.0.0.1:8000
|
||||
MCP server name in `~/.claude.json`: `dev-hub`
|
||||
|
||||
**Step 1 — Orient**
|
||||
|
||||
Read the offline-safe brief first — it works without a live hub connection:
|
||||
```bash
|
||||
cat .custodian-brief.md
|
||||
```
|
||||
Then call the MCP tool for richer cross-domain context when MCP tools are exposed:
|
||||
```
|
||||
get_domain_summary("communication")
|
||||
```
|
||||
If MCP tools are unavailable in the current agent session, use the REST API:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
|
||||
```
|
||||
If the hub is offline: `cd ~/state-hub && make api`
|
||||
|
||||
**Step 2 — Check inbox**
|
||||
With MCP tools:
|
||||
```
|
||||
get_messages(to_agent="markitect-main", unread_only=True)
|
||||
```
|
||||
Mark read with `mark_message_read(message_id)`. Reply or act on coordination
|
||||
requests before proceeding.
|
||||
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=markitect-main&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
**Step 3 — Scan workplans**
|
||||
```bash
|
||||
ls workplans/
|
||||
```
|
||||
For each file with `status: ready`, `active`, or `blocked`, note pending
|
||||
`wait`/`todo`/`progress` tasks.
|
||||
|
||||
**Step 4 — Present brief**
|
||||
|
||||
1. **Active workstreams** for `communication` — title, task counts, blocking decisions
|
||||
2. **Pending tasks** from `workplans/` + any `[repo:markitect-main]` hub tasks
|
||||
3. **Goal guidance** — if `goal_guidance` in summary:
|
||||
- `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"*
|
||||
- `alignment_warnings`: flag if active work is not aligned with current goal
|
||||
4. **Suggested next action** — highest-priority open item
|
||||
5. **SBOM status** — flag if `last_sbom_at` is unset for this repo
|
||||
|
||||
If no workstreams: follow First Session Protocol (`first-session.md`).
|
||||
|
||||
**During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()`
|
||||
|
||||
> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`)
|
||||
> are First Session Protocol only. Work structure belongs in repo files (ADR-001).
|
||||
|
||||
**Session close:**
|
||||
With MCP tools:
|
||||
```
|
||||
add_progress_event(summary="...", topic_id="36c7421b-c537-4723-bf75-42a3ebc6a1dc", workstream_id="<uuid>")
|
||||
```
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"topic_id":"36c7421b-c537-4723-bf75-42a3ebc6a1dc","workstream_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
|
||||
```
|
||||
If workplan files were modified, ensure the local copy is up to date first:
|
||||
```bash
|
||||
git -C <repo_path> pull --ff-only
|
||||
cd ~/state-hub && make fix-consistency REPO=markitect-main
|
||||
```
|
||||
For repos where implementation runs on a remote machine (e.g. CoulombCore),
|
||||
use the combined target which pulls before fixing:
|
||||
```bash
|
||||
cd ~/state-hub && make fix-consistency-remote REPO=markitect-main
|
||||
```
|
||||
**C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback
|
||||
will sync the file to match DB. **C-16** (repo behind remote) blocks all writes
|
||||
until you pull — intentional to prevent clobbering remote progress.
|
||||
16
.claude/rules/stack-and-commands.md
Normal file
16
.claude/rules/stack-and-commands.md
Normal file
@@ -0,0 +1,16 @@
|
||||
## Stack
|
||||
|
||||
- **Language:** Python 3.12+ (monorepo) + JavaScript UI (testdrive-jsui)
|
||||
- **Key deps:** uv/pip, pytest, npm; see `pyproject.toml`, `package.json`, `Makefile`
|
||||
|
||||
## Dev Commands
|
||||
|
||||
```bash
|
||||
make setup
|
||||
make test
|
||||
make test-js
|
||||
make test-all
|
||||
make lint
|
||||
make build
|
||||
make help
|
||||
```
|
||||
40
.claude/rules/workplan-convention.md
Normal file
40
.claude/rules/workplan-convention.md
Normal file
@@ -0,0 +1,40 @@
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
File location: `workplans/MARKITECT-WP-NNNN-<slug>.md`
|
||||
ID prefix: `MARKITECT-WP-`
|
||||
|
||||
Work items originate as files in this repo **before** being registered in the hub.
|
||||
|
||||
Canonical workplan/workstream frontmatter statuses are:
|
||||
`proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`.
|
||||
Use `proposed` for a newly drafted plan, `ready` after review against current
|
||||
repo state, and `finished` when implementation is complete. `stalled` and
|
||||
`needs_review` are derived health labels, not stored statuses.
|
||||
|
||||
Closed workplans may be moved to `workplans/archived/` with a completion-date
|
||||
prefix: `YYMMDD-MARKITECT-WP-NNNN-<slug>.md`. The frontmatter id remains
|
||||
unchanged; the prefix is only for quick visual reference.
|
||||
|
||||
Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**:
|
||||
`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids
|
||||
`ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed
|
||||
directly. Promote anything requiring analysis, design, approval, dependencies, or
|
||||
multiple planned phases into a normal workplan.
|
||||
|
||||
Ecosystem todos from other agents arrive as `[repo:markitect-main]` hub tasks —
|
||||
visible at session start. Pick one up by creating the workplan file, then registering
|
||||
the workstream.
|
||||
|
||||
Task blocks use this shape:
|
||||
|
||||
```task
|
||||
id: MARKITECT-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
```
|
||||
|
||||
Status progression is `todo` → `progress` → `done`; use `wait` for waiting or
|
||||
blocked work and `cancel` for stopped work.
|
||||
|
||||
<!-- Ralph Loop rules and HEUREKA sequence: ~/.claude/CLAUDE.md — do not duplicate here -->
|
||||
@@ -10,7 +10,7 @@ principles with strict separation of concerns.
|
||||
|
||||
## Directory Structure & Clean Architecture
|
||||
```
|
||||
markitect_project/
|
||||
markitect-main/
|
||||
├── domain/ # Business logic (innermost layer)
|
||||
├── application/ # Use cases and workflows
|
||||
├── infrastructure/ # External interfaces (database, file system)
|
||||
|
||||
18
.custodian-brief.md
Normal file
18
.custodian-brief.md
Normal file
@@ -0,0 +1,18 @@
|
||||
<!-- custodian-brief: generated by fix-consistency — do not edit manually -->
|
||||
# Custodian Brief — markitect-main
|
||||
|
||||
**Domain:** communication
|
||||
**Last synced:** 2026-06-22 21:32 UTC
|
||||
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
|
||||
|
||||
## Active Workstreams
|
||||
|
||||
*(none — repo may need first-session setup)*
|
||||
|
||||
---
|
||||
## MCP Orientation (when available)
|
||||
|
||||
If the state-hub MCP server is reachable, call:
|
||||
`get_domain_summary("communication")`
|
||||
This provides richer cross-domain context.
|
||||
If the MCP call fails, use this file as your orientation source.
|
||||
8
.gitignore
vendored
8
.gitignore
vendored
@@ -78,6 +78,9 @@ Thumbs.db
|
||||
|
||||
# MarkiTect database files (local development)
|
||||
markitect.db
|
||||
**/infospace.db
|
||||
assets/assets.db
|
||||
**/assets.db
|
||||
.markitect/
|
||||
|
||||
# Issue workspace (temporary development files)
|
||||
@@ -88,9 +91,14 @@ debug_*.py
|
||||
|
||||
# Claude Code local settings (user-specific permissions)
|
||||
.claude/settings.local.json
|
||||
# Claude Code runtime session locks (per-session, not content)
|
||||
.claude/*.lock
|
||||
|
||||
.aider*
|
||||
|
||||
# API key files
|
||||
apikey-*.txt
|
||||
|
||||
# TDDAI-specific ignores
|
||||
ISSUES.index
|
||||
|
||||
|
||||
12
.gitmodules
vendored
12
.gitmodules
vendored
@@ -1,10 +1,14 @@
|
||||
[submodule "wiki"]
|
||||
path = wiki
|
||||
url = http://92.205.130.254:32166/coulomb/markitect_project.wiki.git
|
||||
url = http://92.205.130.254:32166/coulomb/markitect-main.wiki.git
|
||||
branch = main
|
||||
[submodule "capabilities/issue-facade"]
|
||||
path = capabilities/issue-facade
|
||||
url = http://92.205.130.254:32166/coulomb/issue-facade.git
|
||||
[submodule "capabilities/kaizen-agentic"]
|
||||
path = capabilities/kaizen-agentic
|
||||
url = http://92.205.130.254:32166/coulomb/kaizen-agentic.git
|
||||
[submodule "capabilities/testdrive-jsui"]
|
||||
path = capabilities/testdrive-jsui
|
||||
url = http://92.205.130.254:32166/coulomb/testdrive-jsui.git
|
||||
[submodule "_issue-tracking/issue-facade"]
|
||||
path = _issue-tracking/issue-facade
|
||||
url = http://92.205.130.254:32166/coulomb/issue-facade.git
|
||||
branch = main
|
||||
|
||||
25
.repo-classification.yaml
Normal file
25
.repo-classification.yaml
Normal file
@@ -0,0 +1,25 @@
|
||||
repo_classification:
|
||||
standard: Repo Classification Standard
|
||||
version: '1.0'
|
||||
classified_at: '2026-06-22'
|
||||
classified_by: human
|
||||
category: product
|
||||
domain: communication
|
||||
secondary_domains:
|
||||
- infotech
|
||||
- agents
|
||||
capability_tags:
|
||||
- knowledge
|
||||
- documentation
|
||||
- product-development
|
||||
- platform
|
||||
business_stake:
|
||||
- product
|
||||
- technology
|
||||
- execution
|
||||
business_mechanics:
|
||||
- intention
|
||||
- coordination
|
||||
- operation
|
||||
- adaptation
|
||||
notes: Markitect successor to archived markitect-project; human confirmed.
|
||||
219
AGENTS.md
Normal file
219
AGENTS.md
Normal file
@@ -0,0 +1,219 @@
|
||||
# Markitect Main — Agent Instructions
|
||||
|
||||
## Repo Identity
|
||||
|
||||
**Purpose:** Markitect Main - (fill in purpose)
|
||||
|
||||
**Domain:** communication
|
||||
**Repo slug:** markitect-main
|
||||
**Topic ID:** `36c7421b-c537-4723-bf75-42a3ebc6a1dc`
|
||||
**Workplan prefix:** `MARKITECT-WP-`
|
||||
|
||||
---
|
||||
|
||||
## State Hub Integration
|
||||
|
||||
The Custodian State Hub tracks work across all domains. Interact via HTTP REST —
|
||||
there is no MCP server for Codex agents.
|
||||
|
||||
| Context | URL |
|
||||
|---------|-----|
|
||||
| Local workstation | `http://127.0.0.1:8000` |
|
||||
| Remote via tunnel | `http://127.0.0.1:18000` |
|
||||
|
||||
### Orient at session start
|
||||
|
||||
```bash
|
||||
# Offline brief — works without hub connection
|
||||
cat .custodian-brief.md
|
||||
|
||||
# Active workstreams for this domain
|
||||
curl -s "http://127.0.0.1:8000/workstreams/?topic_id=36c7421b-c537-4723-bf75-42a3ebc6a1dc&status=active" \
|
||||
| python3 -m json.tool
|
||||
|
||||
# Check inbox
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=markitect-main&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
Mark a message read:
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
### Log progress (required at session close)
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"summary": "what was done",
|
||||
"event_type": "note",
|
||||
"author": "codex",
|
||||
"workstream_id": "<uuid>",
|
||||
"task_id": "<uuid>"
|
||||
}'
|
||||
```
|
||||
|
||||
Omit `workstream_id` / `task_id` when not applicable.
|
||||
|
||||
### Update task status
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"status": "progress"}'
|
||||
# values: wait | todo | progress | done | cancel
|
||||
```
|
||||
|
||||
### Flag a task for human review
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"needs_human": true, "intervention_note": "reason"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Session Protocol
|
||||
|
||||
**Start:**
|
||||
1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe)
|
||||
2. Check inbox: `GET /messages/?to_agent=markitect-main&unread_only=true`; mark read
|
||||
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
|
||||
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
|
||||
|
||||
**During work:**
|
||||
- Update task statuses in workplan files as tasks progress
|
||||
- Record significant decisions via `POST /decisions/`
|
||||
|
||||
**Close:**
|
||||
1. Update workplan file task statuses to reflect progress
|
||||
2. Log: `POST /progress/` with a summary of what changed
|
||||
3. Note for the custodian operator: after workplan file changes, run from
|
||||
`~/state-hub`:
|
||||
```bash
|
||||
make fix-consistency REPO=markitect-main
|
||||
```
|
||||
This syncs task status from files into the hub DB.
|
||||
|
||||
---
|
||||
|
||||
## Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=markitect-main` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
|
||||
<!-- REPO-AGENTS-EXTENSIONS -->
|
||||
<!-- Append repo-specific agent instructions below this marker.
|
||||
The state-hub template sync preserves content after this line. -->
|
||||
|
||||
---
|
||||
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
Work items originate as files in this repo — not in the hub. The hub is a
|
||||
read/cache/index layer that rebuilds from files.
|
||||
|
||||
**File location:** `workplans/MARKITECT-WP-NNNN-<slug>.md`
|
||||
|
||||
**Archived location:** finished workplans may move to
|
||||
`workplans/archived/YYMMDD-MARKITECT-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
|
||||
the completion/archive date; the frontmatter `id` does not change.
|
||||
|
||||
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
|
||||
`workplans/ADHOC-YYYY-MM-DD.md` with task ids `ADHOC-YYYY-MM-DD-T01`, etc. Use
|
||||
this only for low-risk work completed directly; create a normal workplan for
|
||||
anything needing analysis, design, approval, dependencies, or multiple phases.
|
||||
|
||||
**Frontmatter:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: MARKITECT-WP-NNNN
|
||||
type: workplan
|
||||
title: "..."
|
||||
domain: communication
|
||||
repo: markitect-main
|
||||
status: proposed | ready | active | blocked | backlog | finished | archived
|
||||
owner: codex
|
||||
topic_slug: ...
|
||||
created: "YYYY-MM-DD"
|
||||
updated: "YYYY-MM-DD"
|
||||
state_hub_workstream_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
---
|
||||
```
|
||||
|
||||
Use `proposed` for a new draft, `ready` after review against current repo
|
||||
state, and `finished` after implementation. `stalled` and `needs_review` are
|
||||
derived health labels, not frontmatter statuses.
|
||||
|
||||
**Task block format** (one per `##` section):
|
||||
|
||||
```
|
||||
## Task Title
|
||||
|
||||
` ` `task
|
||||
id: MARKITECT-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
` ` `
|
||||
|
||||
Task description text.
|
||||
```
|
||||
|
||||
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
|
||||
|
||||
To create a new workplan:
|
||||
1. Write the file following the format above
|
||||
2. Notify the custodian operator to run `make fix-consistency REPO=markitect-main`
|
||||
(or send a message to the hub agent via `POST /messages/`)
|
||||
161
CHANGELOG.md
161
CHANGELOG.md
@@ -5,8 +5,158 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
See roadmap/YYMMDD-ROADMAPTOPIC/ directories for planning information like concepts, workplans, etc...
|
||||
See history/YYMMDD-ROADMAOTOPIC/ directories for planning information of closed topics
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.11.0] - 2026-01-06
|
||||
|
||||
### Added
|
||||
- Release management optimizations: CHANGELOG validation, version-tag consistency checks
|
||||
- Automated tag pushing with --push/--no-push flag
|
||||
- Unpushed tags detection in release status
|
||||
|
||||
### Changed
|
||||
- Improved release validation workflow with CHANGELOG schema validation
|
||||
|
||||
|
||||
## [0.10.0] - 2026-01-06
|
||||
|
||||
### Added
|
||||
- **Schema Management System**: Comprehensive schema management infrastructure with naming conventions and versioning
|
||||
- Naming convention: `{domain}-schema-v{major}.{minor}.md` for all schemas
|
||||
- Markdown-first schema format with embedded JSON (documentation + schema in one file)
|
||||
- Schema catalog (`markitect/schemas/schema-catalog.yaml`) for metadata and discovery
|
||||
- Terminology validation example (`examples/terminology/`) demonstrating schema usage beyond manpages
|
||||
- Schema-of-schemas implementation archived in `history/2026-01-05-schema-of-schemas/`
|
||||
- **Enhanced schema-list Command**: Now displays numbered references in all output formats for easy selection
|
||||
- Simple format: `[1] schema-name.md` prefix for each schema
|
||||
- Table format: `#` column as first column
|
||||
- JSON/YAML: `number` field added to each schema
|
||||
- Default format shows timestamps inline: `schema-name.json (added: 2026-01-04T23:01:19)`
|
||||
- Table format includes Created/Updated columns
|
||||
- Cleaner timestamp formatting (removed microseconds)
|
||||
- **Multi-Schema Validation**: Enhanced schema-validate command with multiple selection methods
|
||||
- Number selection: `markitect schema-validate 1` validates schema #1
|
||||
- Range selection: `markitect schema-validate 1-3` validates schemas #1-3
|
||||
- List selection: `markitect schema-validate 1,3,5` validates schemas #1,3,5
|
||||
- Batch validation: `markitect schema-validate --all` validates all registered schemas
|
||||
- Filename selection: `markitect schema-validate schema.md` from registry
|
||||
- Filesystem path: `markitect schema-validate ./schema.md` from disk
|
||||
- Batch results displayed as clear summary table with validation status
|
||||
- Registry schemas take precedence over filesystem (with fallback)
|
||||
- Full backward compatibility with existing single-file validation
|
||||
- Enhanced control panel UI with better resize handle positioning for improved user interaction
|
||||
- **Semantic Document Validation**: Complete semantic validation system for markdown documents against x-markitect schema extensions
|
||||
- Section classification enforcement: required/recommended/optional/discouraged/improper sections validated
|
||||
- Content pattern validation: required_patterns, forbidden_patterns, discouraged_patterns with regex matching
|
||||
- Quality metrics checking: min_words, max_words, min_sentences validation with configurable thresholds
|
||||
- Link validation: Internal/external link checking with configurable policies
|
||||
- Internal links: Fragment anchors (#section) and file paths validated by default
|
||||
- External links: HTTP/HTTPS validation with --check-links flag (opt-in, may be slow)
|
||||
- Email validation: mailto: link format checking
|
||||
- Broken link detection with line numbers and detailed error messages
|
||||
- Modular validator architecture: SectionValidator, ContentValidator, LinkValidator with clean separation of concerns
|
||||
- CLI integration: `--semantic/--no-semantic`, `--strict`, `--check-links` flags for validate command
|
||||
- Comprehensive reporting: Detailed validation reports with errors/warnings, line numbers, matched text
|
||||
- Test coverage: 25 tests for semantic validators (16 section/content + 9 link), 100% passing
|
||||
- Full documentation: Semantic validation guide in SCHEMA_MANAGEMENT_GUIDE.md with examples
|
||||
- Complements existing structural AST validation for complete document compliance checking
|
||||
- **Changelog Schema**: Production schema for validating CHANGELOG.md files following Keep a Changelog format
|
||||
- Schema file: `changelog-schema-v1.0.md` validates version history structure and formatting
|
||||
- Enforces Unreleased section presence (required)
|
||||
- Validates version format: `[X.Y.Z] - YYYY-MM-DD` with semantic versioning
|
||||
- Validates change type subsections: Added, Changed, Deprecated, Removed, Fixed, Security
|
||||
- Content pattern validation for version sections, date formats (ISO 8601), and change types
|
||||
- Demonstrates real-world schema system usage: "The release that validates itself"
|
||||
- Successfully validates project CHANGELOG.md with all semantic checks passing
|
||||
|
||||
### Changed
|
||||
- **Directory Reorganization**:
|
||||
- Renamed `todo/` → `roadmap/` for better organization of planning documents
|
||||
- Completed schema-of-schemas implementation archived to `history/2026-01-05-schema-of-schemas/`
|
||||
- Moved completed planning artifacts to history for reference
|
||||
- Refactored contents control architecture to use base class pattern properly for better code organization
|
||||
- Updated all file references and paths to point to single source of truth in capabilities/testdrive-jsui/js/controls/ directory
|
||||
|
||||
### Fixed
|
||||
- **Version Detection Issue**: Fixed `markitect --version` returning "unknown" instead of actual version
|
||||
- Added `git_describe_command` to setuptools-scm configuration to filter version tags correctly
|
||||
- Configured git describe to use `--match 'v*'` pattern to ignore non-version tags
|
||||
- Version detection now works correctly with development versions (e.g., 0.9.1.dev76)
|
||||
- **Missing v0.9.0 Git Tag**: Retroactively created v0.9.0 annotated tag on commit b9c1b90 from 2025-11-14
|
||||
- Maintains version history integrity (CHANGELOG documented v0.9.0 but tag was missing)
|
||||
- Enables proper version progression to v0.10.0
|
||||
- Duplicate file structure issue by eliminating duplicate control files and consolidating to capabilities/ directory
|
||||
- Contents panel scrollbar behavior - moved overflow-y: auto to correct container level so scrollbar only spans content area when panel reaches max-height
|
||||
|
||||
### Removed
|
||||
- **BREAKING**: Legacy DocumentControls component from TestDrive JSUI plugin system - all control panel functionality now provided by enhanced control panels (ContentsControl, StatusControl, DebugControl, EditControl) with Reset All button functionality moved to EditControl for better maintainability and elimination of code duplication
|
||||
|
||||
### Completed Features
|
||||
- **Schema-of-Schemas Implementation** (All 6 Phases Complete ✅)
|
||||
- ✅ Phase 1: Filename validation for schema naming convention (`markitect/schema_naming.py`, 50 tests)
|
||||
- ✅ Phase 2: Markdown schema loader to parse `.md` schema files (`markitect/schema_loader.py`, 35 tests)
|
||||
- ✅ Phase 3: Schema-for-schemas metaschema for schema validation (`schema-schema-v1.0.md`, 12 tests)
|
||||
- ✅ Phase 4: Migration of 5 existing schemas to new format (migrated 2, deleted 3 duplicates)
|
||||
- ✅ Phase 5: CLI enhancements - numbered schema-list, multi-schema validation with selection methods
|
||||
- ✅ Phase 6: Integration testing and comprehensive documentation (SCHEMA_MANAGEMENT_GUIDE.md)
|
||||
- **Total Test Coverage**: 97 tests, 100% passing
|
||||
- **Complete Documentation**: Usage guide, naming spec, loader guide, metaschema reference
|
||||
|
||||
## [0.9.0] - 2025-11-14
|
||||
|
||||
### Added
|
||||
- **Plugin Infrastructure Foundation**: Extended existing MarkiTect plugin system with RenderingEnginePlugin base class and RENDERING plugin type
|
||||
- **RenderingEngineManager**: Complete plugin discovery and lifecycle management system for UI rendering engines
|
||||
- **RenderingConfig System**: Asset management and deployment configuration for plugin engines
|
||||
- **TestDrive JSUI Plugin**: Complete independent JavaScript UI plugin extracted from core system with standalone development environment
|
||||
- **Modular Component Architecture**: Compass-positioned controls with clean JSON configuration interface for Python-JavaScript data transfer
|
||||
- **CLI Engine Parameter**: Added --engine parameter to markitect md-render command with engine validation and mode compatibility checking
|
||||
- **Automatic Asset Deployment**: Production-ready asset deployment to _markitect/plugins/ structure with 18 total assets (12 JS, 3 CSS, 3 images)
|
||||
- **ChatGPT Document Theme**: New document theme with Inter font, 580px width, and #10a37f accent color with full CLI support (`markitect md-render --theme chatgpt`)
|
||||
- **Modular Theme System Architecture**: File-based theme loading with YAML configuration and dynamic theme discovery
|
||||
- **Theme Directory Structure**: Organized theme components (mode/, ui/, document/, branding/) for better maintainability
|
||||
- **Database Architecture Documentation**: Comprehensive WORKSPACE_AND_DATABASES.md documenting workspace concepts and database purposes
|
||||
|
||||
### Changed
|
||||
- **BREAKING**: Edit mode now defaults to testdrive-jsui plugin instead of legacy edit mode
|
||||
- **Default Rendering Behavior**: testdrive-jsui for edit/insert modes, standard for view mode with graceful fallback
|
||||
- **Asset Management Strategy**: Automatic plugin asset deployment eliminates need for manual --ship-assets flag
|
||||
- **JavaScript Architecture**: Clean separation between Python backend and JavaScript frontend with modular design
|
||||
- **Theme Loading System**: Implemented dynamic theme discovery and loading with metadata preservation
|
||||
- **Test Suite Organization**: Removed obsolete configuration CLI tests (490 lines) for cleaner codebase
|
||||
|
||||
### Fixed
|
||||
- **JavaScript Loading Conflicts**: Resolved const redeclaration errors with MARKITECT_STRICT_MODE implementation
|
||||
- **MarkitectMain Availability**: Fixed proper main-updated.js loading and JavaScript syntax errors
|
||||
- **Plugin Asset Deployment**: Directory structure preservation with development vs production deployment strategies
|
||||
- **Issue-facade Click Framework Bug**: Resolved Sentinel bug in list command that was causing CLI failures
|
||||
- **Issue-facade Version Command**: Fixed installation error preventing version command from working
|
||||
- **Test Isolation Issues**: Improved test isolation with proper mocking to prevent cross-test interference
|
||||
- **Theme Color Assertions**: Updated test assertions to work with new modular theme system
|
||||
|
||||
### Migration Guide
|
||||
- **Existing Users**: Edit mode will automatically use new testdrive-jsui plugin for enhanced experience
|
||||
- **Legacy Behavior**: Use `markitect md-render --engine standard --edit` to access previous edit mode
|
||||
- **Asset Deployment**: Plugin assets now deploy automatically - no manual --ship-assets flag required
|
||||
|
||||
## [0.8.0] - 2025-11-08
|
||||
|
||||
### Added
|
||||
- **Setuptools-SCM Integration**: Automatic version management system replacing manual version tracking
|
||||
- **Gitea Package Publishing**: Complete CI/CD pipeline for automated package publishing to Gitea
|
||||
- **Enhanced Release Documentation**: Comprehensive documentation for package building and release process
|
||||
|
||||
### Changed
|
||||
- **Release Script Architecture**: Modernized release workflow with setuptools-scm integration
|
||||
- **Makefile Release Targets**: Updated release targets to support automated version management
|
||||
- **Package Building Process**: Streamlined package creation with enhanced build targets
|
||||
|
||||
### Removed
|
||||
- **Legacy Release Scripts**: Removed obsolete release_simplified.py in favor of unified release.py
|
||||
|
||||
## [0.7.0] - 2025-11-08
|
||||
|
||||
### Added
|
||||
@@ -158,4 +308,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- **Build System**: Enhanced build targets with venv Python and PYTHONPATH support
|
||||
- **Target Naming**: Renamed workspace targets to TDD Workspace with tdd- prefix
|
||||
|
||||
xxx
|
||||
[Unreleased]: https://github.com/worsch/markitect/compare/v0.9.0...HEAD
|
||||
[0.9.0]: https://github.com/worsch/markitect/compare/v0.8.0...v0.9.0
|
||||
[0.8.0]: https://github.com/worsch/markitect/compare/v0.7.0...v0.8.0
|
||||
[0.7.0]: https://github.com/worsch/markitect/compare/v0.6.0...v0.7.0
|
||||
[0.6.0]: https://github.com/worsch/markitect/compare/v0.5.0...v0.6.0
|
||||
[0.5.0]: https://github.com/worsch/markitect/compare/v0.4.0...v0.5.0
|
||||
[0.4.0]: https://github.com/worsch/markitect/compare/v0.3.0...v0.4.0
|
||||
[0.3.0]: https://github.com/worsch/markitect/compare/v0.2.0...v0.3.0
|
||||
[0.2.0]: https://github.com/worsch/markitect/compare/v0.1.0...v0.2.0
|
||||
[0.1.0]: https://github.com/worsch/markitect/releases/tag/v0.1.0
|
||||
|
||||
12
CLAUDE.md
Normal file
12
CLAUDE.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Markitect Main — Claude Code Instructions
|
||||
|
||||
@SCOPE.md
|
||||
@.claude/rules/repo-identity.md
|
||||
@.claude/rules/session-protocol.md
|
||||
@.claude/rules/first-session.md
|
||||
@.claude/rules/workplan-convention.md
|
||||
@.claude/rules/stack-and-commands.md
|
||||
@.claude/rules/architecture.md
|
||||
@.claude/rules/repo-boundary.md
|
||||
@.claude/rules/credential-routing.md
|
||||
@.claude/rules/agents.md
|
||||
552
DEPENDENCIES.md
Normal file
552
DEPENDENCIES.md
Normal file
@@ -0,0 +1,552 @@
|
||||
# Dependencies
|
||||
|
||||
This document catalogs all dependencies used in the markitect project, describing how each library and tool is utilized. This helps coordinate dependencies with sister projects like referitect and maintain consistency across the ecosystem.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Core Dependencies](#core-dependencies)
|
||||
- [Capability Dependencies](#capability-dependencies)
|
||||
- [Development Dependencies](#development-dependencies)
|
||||
- [JavaScript Dependencies](#javascript-dependencies)
|
||||
- [Dependency Criticality Matrix](#dependency-criticality-matrix)
|
||||
- [Coordination Guidelines for Sister Projects](#coordination-guidelines-for-sister-projects)
|
||||
|
||||
---
|
||||
|
||||
## Core Dependencies
|
||||
|
||||
These dependencies are required for markitect's core functionality.
|
||||
|
||||
### markdown-it-py
|
||||
**Version:** Latest
|
||||
**Criticality:** CRITICAL
|
||||
**Purpose:** Core markdown parsing engine
|
||||
|
||||
**Usage:**
|
||||
- Converts markdown content to Abstract Syntax Trees (AST)
|
||||
- Enables table parsing with `.enable(['table'])`
|
||||
- Uses CommonMark parser with token conversion
|
||||
- Foundation for all markdown processing operations
|
||||
|
||||
**Files:** `markitect/parser.py`
|
||||
|
||||
**Why this matters:** This is the primary markdown parser. Sister projects should use the same parser for consistent AST structure and compatibility.
|
||||
|
||||
---
|
||||
|
||||
### PyYAML
|
||||
**Version:** Latest
|
||||
**Criticality:** CRITICAL
|
||||
**Purpose:** YAML configuration parsing and serialization
|
||||
|
||||
**Usage:**
|
||||
- Parses YAML frontmatter from markdown documents
|
||||
- Loads configuration files for themes and workspaces
|
||||
- Supports YAML-based schema definitions
|
||||
- Safe loading of untrusted YAML data (`yaml.safe_load()`)
|
||||
- Configuration serialization and deserialization
|
||||
|
||||
**Files:**
|
||||
- `markitect/cli.py`
|
||||
- `markitect/schema_loader.py`
|
||||
- `markitect/matter_frontmatter/parser.py`
|
||||
- `markitect/config_manager.py`
|
||||
- `markitect/production/configuration.py`
|
||||
- `markitect/serializer.py`
|
||||
- Multiple capability files
|
||||
|
||||
**Why this matters:** YAML is the primary configuration format. Consistent YAML handling ensures config compatibility across projects.
|
||||
|
||||
---
|
||||
|
||||
### click
|
||||
**Version:** >=8.0.0
|
||||
**Criticality:** CRITICAL
|
||||
**Purpose:** Command-line interface framework
|
||||
|
||||
**Usage:**
|
||||
- Primary CLI framework for all command-line commands
|
||||
- Command groups, options, and decorators for `markitect` CLI
|
||||
- Table output formatting (combined with tabulate)
|
||||
- Command organization and help system
|
||||
- Error handling and context management
|
||||
- Profile management, finance tracking, release management, issue tracking commands
|
||||
|
||||
**Files:** 63 files including:
|
||||
- `markitect/cli.py` (main CLI)
|
||||
- `markitect/cli_utils.py`
|
||||
- `markitect/asset_commands.py`
|
||||
- `markitect/profile/commands.py`
|
||||
- `markitect/finance/cli.py`
|
||||
- `capabilities/release-management/src/release_management/cli/main.py`
|
||||
- `capabilities/kaizen-agentic/src/kaizen_agentic/cli.py`
|
||||
- Issue tracker CLI commands
|
||||
|
||||
**Why this matters:** Click provides consistent CLI UX. Sister projects using Click will have familiar command patterns and help output.
|
||||
|
||||
---
|
||||
|
||||
### tabulate
|
||||
**Version:** >=0.9.0
|
||||
**Criticality:** HIGH
|
||||
**Purpose:** ASCII table formatting for CLI output
|
||||
|
||||
**Usage:**
|
||||
- Formats database query results as ASCII tables
|
||||
- Provides multiple table formats: grid, simple, fancy
|
||||
- Used in CLI output for:
|
||||
- Database command results
|
||||
- Performance metrics display
|
||||
- Status reports
|
||||
- File listings
|
||||
- Financial tracking output
|
||||
|
||||
**Files:**
|
||||
- `markitect/cli.py`
|
||||
- `markitect/cli_utils.py`
|
||||
- `markitect/plugins/builtin/markdown_commands.py`
|
||||
- `markitect/finance/cli.py`
|
||||
- `markitect/finance/worktime_commands.py`
|
||||
|
||||
**Why this matters:** Consistent table formatting across all CLI tools provides professional, readable output.
|
||||
|
||||
---
|
||||
|
||||
### jsonpath-ng
|
||||
**Version:** >=1.5.0
|
||||
**Criticality:** MEDIUM
|
||||
**Purpose:** JSONPath querying for AST navigation
|
||||
|
||||
**Usage:**
|
||||
- XPath-like navigation through AST structures
|
||||
- Query parsing with `jsonpath-ng.parse()`
|
||||
- Fallback simple dot-notation implementation if import fails
|
||||
- Advanced query paradigm for structural queries
|
||||
- Supports filters, recursive descent, array wildcards
|
||||
|
||||
**Files:**
|
||||
- `markitect/query_paradigms/paradigms/jsonpath_paradigm.py`
|
||||
- `markitect/ast_service.py`
|
||||
- `markitect/graphql/resolvers.py`
|
||||
|
||||
**Why this matters:** Provides advanced AST querying. Optional but useful for complex document transformations.
|
||||
|
||||
---
|
||||
|
||||
### aiohttp
|
||||
**Version:** >=3.8.0
|
||||
**Criticality:** MEDIUM
|
||||
**Purpose:** Async HTTP client for API interactions
|
||||
|
||||
**Usage:**
|
||||
- HTTP session management with connection pooling
|
||||
- Async/await pattern support for concurrent requests
|
||||
- TCP connection optimization (limit, per-host limits)
|
||||
- Timeout and keepalive configuration
|
||||
- Authentication header management for API calls
|
||||
- Gitea API integration
|
||||
- Health check mechanisms
|
||||
- Proper resource cleanup with async context managers
|
||||
|
||||
**Files:**
|
||||
- `infrastructure/connection_manager.py`
|
||||
|
||||
**Why this matters:** Enables performant async I/O for API operations. Critical for scalable network operations.
|
||||
|
||||
---
|
||||
|
||||
### toml
|
||||
**Version:** Latest
|
||||
**Criticality:** LOW-MEDIUM
|
||||
**Purpose:** TOML configuration format parsing
|
||||
|
||||
**Usage:**
|
||||
- Parses TOML-formatted frontmatter in markdown files
|
||||
- Alternative to YAML for configuration syntax
|
||||
- Format detection: tries TOML if content has '=' and brackets
|
||||
- Graceful fallback to YAML if TOML parsing fails
|
||||
|
||||
**Files:**
|
||||
- `markitect/matter_frontmatter/parser.py`
|
||||
- `capabilities/release-management/src/release_management/summary/generator.py`
|
||||
- `capabilities/release-management/src/release_management/utils/validation.py`
|
||||
- `capabilities/release-management/src/release_management/registries/factory.py`
|
||||
|
||||
**Why this matters:** Provides TOML support for users preferring TOML over YAML. Optional with fallback mechanisms.
|
||||
|
||||
---
|
||||
|
||||
### setuptools-scm
|
||||
**Version:** >=8.0.0
|
||||
**Criticality:** CRITICAL
|
||||
**Purpose:** Git-based version management
|
||||
|
||||
**Usage:**
|
||||
- Automatic version derivation from git tags
|
||||
- Generates `_version.py` file during build
|
||||
- Supports semantic versioning (python-simplified-semver scheme)
|
||||
- Git tag format: `v*` (matches v0.11.0, v1.0.0, etc.)
|
||||
- No local version suffix in scheme
|
||||
- Used by release-management for version operations
|
||||
|
||||
**Files:**
|
||||
- `markitect/__version__.py`
|
||||
- `capabilities/release-management/src/release_management/utils/version.py`
|
||||
- `capabilities/release-management/src/release_management/cli/main.py`
|
||||
- `capabilities/release-management/src/release_management/utils/validation.py`
|
||||
- `capabilities/release-management/src/release_management/core/builder.py`
|
||||
|
||||
**Why this matters:** Single source of truth for versioning. Critical for coordinated releases across sister projects.
|
||||
|
||||
---
|
||||
|
||||
## Capability Dependencies
|
||||
|
||||
Dependencies specific to optional capabilities. These extend markitect's functionality.
|
||||
|
||||
### markitect-utils
|
||||
**Location:** `capabilities/markitect-utils`
|
||||
**Dependencies:** None (intentionally dependency-free)
|
||||
**Purpose:** Utility functions capability for testing ComposableRepositoryParadigm
|
||||
|
||||
---
|
||||
|
||||
### markitect-content
|
||||
**Location:** `capabilities/markitect-content`
|
||||
**Dependencies:**
|
||||
- `click>=8.0.0` - CLI commands for content parsing and statistics
|
||||
|
||||
**Purpose:** Content parsing and statistics capability
|
||||
|
||||
---
|
||||
|
||||
### kaizen-agentic
|
||||
**Location:** `capabilities/kaizen-agentic`
|
||||
**Dependencies:**
|
||||
- `pyyaml>=6.0` - Agent configuration files
|
||||
- `click>=8.0.0` - CLI interface
|
||||
- `pydantic>=2.0.0` - Data validation for agent configs
|
||||
|
||||
**Purpose:** AI agent development framework with continuous improvement patterns
|
||||
|
||||
**Key Usage:**
|
||||
- Agent configuration via YAML
|
||||
- CLI for agent management (`kaizen-agentic` command)
|
||||
- Pydantic models for validated agent configurations
|
||||
|
||||
---
|
||||
|
||||
### release-management
|
||||
**Location:** `capabilities/release-management`
|
||||
**Dependencies:**
|
||||
- `click>=8.0.0` - CLI commands
|
||||
- `requests>=2.25.0` - Gitea package registry API
|
||||
- `setuptools-scm>=8.0.0` - Version management
|
||||
- `build>=0.8.0` - Python package building
|
||||
- `packaging>=21.0` - Version parsing and comparison
|
||||
- `pyyaml>=6.0` - Configuration files
|
||||
- `pydantic>=2.0.0` - Configuration validation
|
||||
|
||||
**Purpose:** Comprehensive release management for Python projects
|
||||
|
||||
**Key Usage:**
|
||||
- Package building (wheel, sdist)
|
||||
- Git tag creation and management
|
||||
- Gitea package registry publishing
|
||||
- Version validation and semantic versioning
|
||||
- Release configuration via YAML
|
||||
|
||||
**Entry Points:** `release`, `release-manager` commands
|
||||
|
||||
---
|
||||
|
||||
### testdrive-jsui
|
||||
**Location:** `capabilities/testdrive-jsui`
|
||||
**Dependencies:**
|
||||
- `pytest>=7.0.0` - Test framework
|
||||
- `pytest-xvfb>=3.0.0` - Headless browser testing
|
||||
- `selenium>=4.0.0` - Browser automation
|
||||
- `pathlib2>=2.3.0` (Python <3.4) - Path handling
|
||||
|
||||
**Purpose:** JavaScript UI testing framework capability
|
||||
|
||||
**Key Usage:**
|
||||
- Browser-based testing infrastructure
|
||||
- Headless browser execution
|
||||
- Selenium WebDriver integration
|
||||
- JavaScript test execution from Python
|
||||
|
||||
**Note:** Primarily a JavaScript library with Python adapter. See JavaScript Dependencies section.
|
||||
|
||||
---
|
||||
|
||||
### issue-facade (universal-issue-tracker)
|
||||
**Location:** `_issue-tracking/issue-facade`
|
||||
**Dependencies:**
|
||||
- `click>=8.0.0` - CLI framework
|
||||
- `requests>=2.25.0` - Remote backend API calls (Gitea, GitHub)
|
||||
- `python-dateutil>=2.8.0` - Date/time parsing
|
||||
|
||||
**Purpose:** Backend-agnostic issue tracking with plugin architecture
|
||||
|
||||
**Key Usage:**
|
||||
- Unified CLI for multiple issue backends (Gitea, GitHub, local SQLite)
|
||||
- REST API integration
|
||||
- Date parsing for issue timestamps
|
||||
- Synchronization between local and remote backends
|
||||
|
||||
**Entry Points:** `issue`, `issue-tracker` commands
|
||||
|
||||
---
|
||||
|
||||
## Development Dependencies
|
||||
|
||||
These dependencies are only required during development and testing.
|
||||
|
||||
### Testing Frameworks
|
||||
- `pytest>=6.0.0` - Primary test framework
|
||||
- `pytest-cov>=2.0` - Coverage reporting
|
||||
- `pytest-mock>=3.0` - Mocking utilities
|
||||
- `pytest-randomly>=3.10.0` - Randomized test execution (kaizen-agentic)
|
||||
- `pytest-asyncio>=0.21.0` - Async test support (testdrive-jsui)
|
||||
- `pytest-timeout>=2.1.0` - Test timeout handling (testdrive-jsui)
|
||||
|
||||
### Code Quality
|
||||
- `black>=22.0.0` - Code formatting (88 char line length)
|
||||
- `isort>=5.0` - Import sorting
|
||||
- `flake8>=5.0.0` - Linting (100 char max line length)
|
||||
- `mypy>=1.0.0` - Static type checking
|
||||
- `pre-commit>=2.0` - Git hooks
|
||||
|
||||
### Build Tools
|
||||
- `setuptools>=64` - Python package building
|
||||
- `wheel` - Wheel package format
|
||||
- `build>=0.8.0` - PEP 517 build frontend
|
||||
|
||||
### Documentation
|
||||
- `sphinx>=4.0` - Documentation generation (issue-facade)
|
||||
- `sphinx-rtd-theme>=1.0` - Read the Docs theme
|
||||
- `sphinx-click>=3.0` - Click command documentation
|
||||
|
||||
---
|
||||
|
||||
## JavaScript Dependencies
|
||||
|
||||
These dependencies support browser-side functionality and JavaScript testing.
|
||||
|
||||
### Runtime JavaScript Dependencies (Browser)
|
||||
|
||||
#### marked.js
|
||||
**Version:** ^11.0.0 || ^12.0.0 || ^13.0.0 (peer dependency)
|
||||
**Criticality:** MEDIUM
|
||||
**Purpose:** Client-side markdown parsing
|
||||
|
||||
**Usage:**
|
||||
- Real-time markdown rendering in browser
|
||||
- Loaded via CDN: `https://cdn.jsdelivr.net/npm/marked/marked.min.js`
|
||||
- Provides fallback if CDN fails
|
||||
- HTML generation from markdown content with `marked.parse()`
|
||||
|
||||
**Files:**
|
||||
- `markitect/templates/edit-mode.html`
|
||||
- `markitect/templates/edit-mode-fixed.html`
|
||||
- `markitect/templates/document.html`
|
||||
|
||||
**Why this matters:** Enables interactive markdown editing in browser. Sister projects with browser UI should use the same version for consistency.
|
||||
|
||||
---
|
||||
|
||||
### Node.js Development Dependencies
|
||||
|
||||
Located in `capabilities/testdrive-jsui/package.json`:
|
||||
|
||||
#### Build Tools
|
||||
- `rollup@^4.53.5` - Module bundler
|
||||
- `@rollup/plugin-babel@^6.1.0` - Babel transpilation
|
||||
- `@rollup/plugin-commonjs@^29.0.0` - CommonJS support
|
||||
- `@rollup/plugin-node-resolve@^16.0.3` - Node module resolution
|
||||
- `@rollup/plugin-terser@^0.4.4` - Code minification
|
||||
- `rollup-plugin-postcss@^4.0.2` - CSS processing
|
||||
|
||||
#### Testing
|
||||
- `jest@^29.7.0` - JavaScript test framework
|
||||
- `jest-environment-jsdom@^29.7.0` - DOM environment for tests
|
||||
- `babel-jest@^29.7.0` - Babel integration for Jest
|
||||
- `jsdom@^23.0.0` - DOM implementation for Node.js
|
||||
|
||||
#### Linting and Formatting
|
||||
- `eslint@^8.57.0` - JavaScript linter
|
||||
- `eslint-config-standard@^17.1.0` - Standard style guide
|
||||
- `eslint-plugin-jest@^27.6.0` - Jest-specific rules
|
||||
|
||||
#### CSS Processing
|
||||
- `postcss@^8.5.6` - CSS transformation
|
||||
- `postcss-import@^16.1.1` - CSS import handling
|
||||
- `autoprefixer@^10.4.23` - Vendor prefix automation
|
||||
- `cssnano@^7.1.2` - CSS minification
|
||||
|
||||
#### Transpilation
|
||||
- `@babel/core@^7.23.0` - JavaScript compiler
|
||||
- `@babel/preset-env@^7.23.0` - Smart presets for target environments
|
||||
|
||||
---
|
||||
|
||||
## Dependency Criticality Matrix
|
||||
|
||||
| Criticality | Dependencies | Impact if Missing |
|
||||
|:---|:---|:---|
|
||||
| **CRITICAL** | markdown-it-py, PyYAML, click, setuptools-scm | Project cannot function |
|
||||
| **HIGH** | tabulate | CLI output severely degraded |
|
||||
| **MEDIUM** | jsonpath-ng, aiohttp, pydantic, requests, marked.js | Features unavailable, fallbacks exist |
|
||||
| **LOW-MEDIUM** | toml | Optional format support |
|
||||
| **DEV-ONLY** | pytest, black, flake8, mypy, jest, eslint | Development/testing affected only |
|
||||
|
||||
---
|
||||
|
||||
## Coordination Guidelines for Sister Projects
|
||||
|
||||
When developing referitect or other sister projects, follow these guidelines to maintain ecosystem consistency:
|
||||
|
||||
### Mandatory Alignment
|
||||
These dependencies MUST match versions for compatibility:
|
||||
|
||||
1. **markdown-it-py** - Ensures consistent AST structure across projects
|
||||
2. **PyYAML** - Enables shared configuration formats
|
||||
3. **setuptools-scm** - Coordinated versioning strategy
|
||||
4. **click** - Consistent CLI patterns and user experience
|
||||
|
||||
### Recommended Alignment
|
||||
These dependencies SHOULD match for consistency:
|
||||
|
||||
1. **tabulate** - Uniform table output across CLIs
|
||||
2. **pydantic** - Shared data validation patterns
|
||||
3. **marked.js** - Consistent browser-side rendering
|
||||
4. **black/flake8/mypy** - Unified code quality standards
|
||||
|
||||
### Optional Dependencies
|
||||
Projects may choose different versions or alternatives:
|
||||
|
||||
1. **jsonpath-ng** - If advanced querying not needed
|
||||
2. **aiohttp** vs. **requests** - Choose based on sync/async needs
|
||||
3. **toml** - If TOML support not required
|
||||
4. **selenium/pytest-xvfb** - If browser testing not needed
|
||||
|
||||
### Version Pinning Strategy
|
||||
|
||||
**Core dependencies:** Pin to minor versions (e.g., `click>=8.0.0`)
|
||||
- Allows patch updates for security fixes
|
||||
- Prevents breaking changes from major updates
|
||||
|
||||
**Development dependencies:** Pin to specific versions in lockfile
|
||||
- Ensures reproducible development environments
|
||||
- Update deliberately during maintenance windows
|
||||
|
||||
**Build dependencies:** Pin to exact versions (e.g., `setuptools>=64`)
|
||||
- Ensures reproducible builds
|
||||
- Critical for release management
|
||||
|
||||
### Capability Reuse
|
||||
|
||||
Sister projects can reuse these capabilities directly:
|
||||
|
||||
1. **release-management** - Use for package publishing
|
||||
2. **kaizen-agentic** - Share agent development patterns
|
||||
3. **issue-facade** - Unified issue tracking across projects
|
||||
4. **testdrive-jsui** - Browser-based markdown editor
|
||||
|
||||
Install capabilities via local file references:
|
||||
```toml
|
||||
[project.dependencies]
|
||||
release-management = {path = "../markitect-main/capabilities/release-management"}
|
||||
```
|
||||
|
||||
### Shared Infrastructure
|
||||
|
||||
Both projects can share:
|
||||
|
||||
1. **Configuration schemas** - YAML/TOML format definitions
|
||||
2. **CLI patterns** - Click command structure
|
||||
3. **AST utilities** - Markdown parsing and transformation
|
||||
4. **Testing infrastructure** - pytest configuration and fixtures
|
||||
|
||||
### Dependency Updates
|
||||
|
||||
When updating shared dependencies:
|
||||
|
||||
1. Update in markitect first
|
||||
2. Test thoroughly
|
||||
3. Document breaking changes in CHANGELOG.md
|
||||
4. Coordinate update in referitect
|
||||
5. Update this DEPENDENCIES.md file
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
markitect (core)
|
||||
├── markdown-it-py (AST generation)
|
||||
├── PyYAML (config parsing)
|
||||
├── click (CLI framework)
|
||||
│ └── tabulate (table output)
|
||||
├── jsonpath-ng (AST querying)
|
||||
├── aiohttp (async HTTP)
|
||||
├── toml (TOML parsing)
|
||||
└── setuptools-scm (versioning)
|
||||
|
||||
capabilities/
|
||||
├── markitect-utils (no deps)
|
||||
├── markitect-content
|
||||
│ └── click
|
||||
├── kaizen-agentic
|
||||
│ ├── pyyaml
|
||||
│ ├── click
|
||||
│ └── pydantic
|
||||
├── release-management
|
||||
│ ├── click
|
||||
│ ├── requests
|
||||
│ ├── setuptools-scm
|
||||
│ ├── build
|
||||
│ ├── packaging
|
||||
│ ├── pyyaml
|
||||
│ └── pydantic
|
||||
├── testdrive-jsui
|
||||
│ ├── pytest
|
||||
│ ├── pytest-xvfb
|
||||
│ ├── selenium
|
||||
│ └── JavaScript deps:
|
||||
│ ├── marked (peer)
|
||||
│ ├── jest (dev)
|
||||
│ ├── rollup (dev)
|
||||
│ └── eslint (dev)
|
||||
└── issue-facade
|
||||
├── click
|
||||
├── requests
|
||||
└── python-dateutil
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## License Compatibility
|
||||
|
||||
All dependencies use OSI-approved licenses compatible with MIT:
|
||||
|
||||
- **MIT License:** markdown-it-py, click, tabulate, jsonpath-ng, aiohttp, pydantic, marked.js, jest, rollup
|
||||
- **Apache 2.0:** requests, setuptools-scm
|
||||
- **BSD License:** PyYAML, selenium
|
||||
- **PSF License:** python-dateutil (Python Software Foundation)
|
||||
|
||||
No GPL or copyleft licenses used - all dependencies are permissively licensed.
|
||||
|
||||
---
|
||||
|
||||
## Version History
|
||||
|
||||
- **v0.11.0** (2026-01-07): Initial DEPENDENCIES.md creation for referitect coordination
|
||||
- Future updates will track dependency changes and version bumps
|
||||
|
||||
---
|
||||
|
||||
**Maintained by:** MarkiTect Project Team
|
||||
**Last Updated:** 2026-01-07
|
||||
**For Questions:** See main README.md or open an issue
|
||||
81
GUARDRAILS.md
Normal file
81
GUARDRAILS.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Development Guardrails
|
||||
|
||||
## JavaScript Code Principles
|
||||
|
||||
### 1. No Inline JavaScript in Python
|
||||
**NEVER write JavaScript code directly from Python code**
|
||||
|
||||
❌ **Wrong:**
|
||||
```python
|
||||
script = f"""
|
||||
function myFunction() {{
|
||||
console.log("Hello {name}");
|
||||
}}
|
||||
"""
|
||||
```
|
||||
|
||||
✅ **Correct:**
|
||||
```python
|
||||
# Load from external files only
|
||||
components = [
|
||||
'js/core/section-manager.js',
|
||||
'js/components/debug-panel.js',
|
||||
'js/components/document-controls.js'
|
||||
]
|
||||
```
|
||||
|
||||
### 2. Why This Rule Exists
|
||||
- **Quoting Problems**: String escaping in Python corrupts JavaScript
|
||||
- **Syntax Errors**: Template literals and complex JS break when embedded
|
||||
- **Maintainability**: JS code should be in .js files for proper tooling
|
||||
- **Architecture**: Follows the established modular component system
|
||||
|
||||
### 3. Proper Approach
|
||||
1. Create separate `.js` files in `markitect/static/js/components/`
|
||||
2. Load them via `_get_clean_editor_scripts()`
|
||||
3. Wire up components in the initialization script only
|
||||
|
||||
## Testing and Validation
|
||||
|
||||
### 1. Always Validate Generated HTML
|
||||
- Check that HTML files actually render content
|
||||
- Validate JavaScript syntax before deployment
|
||||
- Test both viewing and editing modes
|
||||
|
||||
### 2. Detect JavaScript Errors Programmatically
|
||||
- Run syntax validation on generated JS
|
||||
- Check for common error patterns
|
||||
- Fail fast when JS is malformed
|
||||
|
||||
### 3. Manual Testing Backup
|
||||
- If automated checks pass but functionality fails
|
||||
- Open generated HTML in browser
|
||||
- Check console for runtime errors
|
||||
- Report specific error messages
|
||||
|
||||
## Architecture Principles
|
||||
|
||||
### 1. Separation of Concerns
|
||||
- Python: File generation, template management
|
||||
- JavaScript: UI components, interaction logic
|
||||
- HTML: Structure and content only
|
||||
|
||||
### 2. Modular Component System
|
||||
- Each UI component in separate file
|
||||
- Lazy loading where appropriate
|
||||
- Clear dependency management
|
||||
|
||||
### 3. Error Handling
|
||||
- Graceful degradation when components fail
|
||||
- Clear error messages for debugging
|
||||
- Fallback modes when possible
|
||||
|
||||
## Breaking These Rules
|
||||
|
||||
If you find yourself writing JavaScript in Python strings:
|
||||
1. **STOP** - Step back and reconsider
|
||||
2. Create a proper component file instead
|
||||
3. Use the existing component loading system
|
||||
4. Add validation to catch the issue early
|
||||
|
||||
These guardrails exist because we've seen the problems when they're violated.
|
||||
253
INTRODUCTION.md
Normal file
253
INTRODUCTION.md
Normal file
@@ -0,0 +1,253 @@
|
||||
# MarkiTect: Intelligent Markdown as Structured Information
|
||||
|
||||
## What is MarkiTect?
|
||||
|
||||
MarkiTect transforms how you work with markdown by treating documents as **structured, queryable information spaces** rather than plain text files. It's a comprehensive system for managing, validating, transforming, and composing markdown documents at scale—whether you're maintaining technical documentation, building knowledge bases, generating reports, or orchestrating complex document workflows.
|
||||
|
||||
Think of MarkiTect as a **database for your markdown content**, combined with powerful tools for ensuring consistency, automating repetitive tasks, and creating intelligent document systems that adapt to your needs.
|
||||
|
||||
## Why MarkiTect Matters
|
||||
|
||||
### The Problem
|
||||
|
||||
Modern documentation and content workflows face several challenges:
|
||||
|
||||
- **Inconsistency**: Different documents follow different structures, making content hard to maintain and process
|
||||
- **Manual Validation**: No automated way to ensure documents meet quality standards or required formats
|
||||
- **Content Silos**: Documents exist in isolation without relationships or shared context
|
||||
- **Repetitive Work**: Copy-pasting boilerplate, manually updating cross-references, regenerating similar documents
|
||||
- **Limited Automation**: Plain markdown files can't drive automated workflows or integrate with systems
|
||||
- **Scaling Difficulties**: Managing hundreds or thousands of documents becomes overwhelming
|
||||
|
||||
### The MarkiTect Solution
|
||||
|
||||
MarkiTect solves these problems by treating markdown as **structured data**:
|
||||
|
||||
- **Schema-Driven**: Define document structures once, validate everywhere
|
||||
- **Relational**: Documents can reference, include, and compose with each other
|
||||
- **Queryable**: Extract insights from your content using powerful queries
|
||||
- **Automated**: Generate, validate, and transform documents programmatically
|
||||
- **Versioned**: Track changes and maintain document history
|
||||
- **Composable**: Build complex documents from reusable components
|
||||
|
||||
## Core Capabilities
|
||||
|
||||
### 1. Information Spaces
|
||||
|
||||
Create **organized collections of related documents** that work together as cohesive knowledge spaces:
|
||||
|
||||
- **Structured Collections**: Group documents by project, topic, or any logical structure
|
||||
- **Transclusion**: Embed content from one document into another, maintaining a single source of truth
|
||||
- **Shared Context**: Variables and configuration shared across all documents in a space
|
||||
- **Multiple Views**: Render the same content as HTML, export to directories, or query programmatically
|
||||
- **Hierarchical Organization**: Parent-child relationships for inherited settings and permissions
|
||||
|
||||
### 2. Schema Management
|
||||
|
||||
Define and enforce document structures to ensure consistency:
|
||||
|
||||
- **Schema Generation**: Automatically extract schemas from example documents
|
||||
- **Validation**: Verify documents match required structures before publishing
|
||||
- **Schema Evolution**: Track how document requirements change over time
|
||||
- **Template Generation**: Create document stubs from schemas
|
||||
- **Semantic Understanding**: Schemas capture both structure and meaning
|
||||
|
||||
### 3. Document Processing
|
||||
|
||||
Powerful tools for analyzing, transforming, and manipulating content:
|
||||
|
||||
- **Intelligent Parsing**: Understand document structure including frontmatter, content, and metadata
|
||||
- **Content Extraction**: Pull specific sections, metadata, or elements from documents
|
||||
- **Transformation**: Convert between formats while preserving semantic meaning
|
||||
- **Batch Operations**: Process multiple documents with a single command
|
||||
- **Quality Checks**: Automated validation of links, references, and formatting
|
||||
|
||||
### 4. Transclusion & Composition
|
||||
|
||||
Build complex documents from reusable components:
|
||||
|
||||
- **Content Reuse**: Include shared sections across multiple documents
|
||||
- **Variable Substitution**: Parameterize content for different contexts
|
||||
- **Conditional Content**: Show or hide sections based on conditions
|
||||
- **Variant Generation**: Create multiple versions from a single source
|
||||
- **Cross-Document References**: Maintain relationships between related content
|
||||
|
||||
### 5. Querying & Analysis
|
||||
|
||||
Extract insights from your documentation:
|
||||
|
||||
- **Metadata Queries**: Find documents by attributes, tags, or properties
|
||||
- **Content Search**: Locate specific information across your entire corpus
|
||||
- **Relationship Mapping**: Understand how documents connect and depend on each other
|
||||
- **Statistics**: Analyze document sizes, structures, and patterns
|
||||
- **Reporting**: Generate summaries and reports from your content
|
||||
|
||||
### 6. Automation & Integration
|
||||
|
||||
Connect markdown content to broader workflows:
|
||||
|
||||
- **CLI Interface**: Automate any operation from scripts or CI/CD pipelines
|
||||
- **GraphQL API**: Query and manipulate spaces programmatically
|
||||
- **Event System**: React to document changes in real-time
|
||||
- **Plugin Architecture**: Extend functionality for specific needs
|
||||
- **Git Integration**: Optional version control for complete change history
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Technical Documentation
|
||||
|
||||
**Challenge**: Maintain consistent API documentation across multiple services
|
||||
**Solution**: Use schemas to enforce standard formats, transclusion for shared examples, and variables for service-specific details
|
||||
|
||||
**Benefits**:
|
||||
- One source of truth for common patterns
|
||||
- Automatic validation of all documentation
|
||||
- Generate client libraries and API specs from the same source
|
||||
- Track documentation changes alongside code
|
||||
|
||||
### Knowledge Management
|
||||
|
||||
**Challenge**: Organize company knowledge across departments and projects
|
||||
**Solution**: Create information spaces for each domain, use cross-document references for relationships, enable search across all content
|
||||
|
||||
**Benefits**:
|
||||
- Discoverable knowledge base
|
||||
- Consistent structure across departments
|
||||
- Track what content exists and how it connects
|
||||
- Generate reports on documentation coverage
|
||||
|
||||
### Report Generation
|
||||
|
||||
**Challenge**: Create hundreds of similar reports with varying data
|
||||
**Solution**: Define report schemas, use templates with variables, batch-generate reports from data sources
|
||||
|
||||
**Benefits**:
|
||||
- Consistent report structure
|
||||
- Automated generation from data
|
||||
- Quality validation before distribution
|
||||
- Version tracking for compliance
|
||||
|
||||
### Content Publishing
|
||||
|
||||
**Challenge**: Maintain a blog or publication with consistent formatting
|
||||
**Solution**: Use schemas for post structures, transclusion for common elements, render to HTML with themes
|
||||
|
||||
**Benefits**:
|
||||
- Enforced style guide compliance
|
||||
- Reusable components (author bios, disclaimers, CTAs)
|
||||
- Multiple output formats from single source
|
||||
- Automated link checking and validation
|
||||
|
||||
### Educational Materials
|
||||
|
||||
**Challenge**: Create course materials with exercises, examples, and assessments
|
||||
**Solution**: Schema-driven lesson structures, transclusion for common instructions, variable content for different audiences
|
||||
|
||||
**Benefits**:
|
||||
- Consistent lesson structure
|
||||
- Easy updates across multiple courses
|
||||
- Generate student and instructor versions
|
||||
- Track curriculum evolution
|
||||
|
||||
### Compliance Documentation
|
||||
|
||||
**Challenge**: Maintain audit trails and ensure required document elements
|
||||
**Solution**: Strict schemas with validation, versioned spaces with history tracking, automated compliance reports
|
||||
|
||||
**Benefits**:
|
||||
- Guaranteed document completeness
|
||||
- Audit trail for all changes
|
||||
- Automated compliance checking
|
||||
- Generate reports for auditors
|
||||
|
||||
## Key Benefits
|
||||
|
||||
### For Writers and Content Creators
|
||||
|
||||
- **Focus on Content**: Let MarkiTect handle structure and consistency
|
||||
- **Reuse Work**: Write once, use everywhere with transclusion
|
||||
- **Quality Assurance**: Automated validation catches errors early
|
||||
- **Easy Updates**: Change once, propagate everywhere
|
||||
- **Version Control**: Track all changes with git integration
|
||||
|
||||
### For Teams and Organizations
|
||||
|
||||
- **Consistency at Scale**: Same structure across thousands of documents
|
||||
- **Collaboration**: Shared spaces with inheritance and permissions
|
||||
- **Automation**: Reduce manual work through scripting
|
||||
- **Integration**: Connect documentation to development workflows
|
||||
- **Maintainability**: Clear structure makes updates easier
|
||||
|
||||
### For Developers and Automators
|
||||
|
||||
- **Programmatic Access**: CLI and GraphQL APIs for automation
|
||||
- **Event-Driven**: React to changes in real-time
|
||||
- **Extensible**: Plugin system for custom functionality
|
||||
- **Type-Safe**: Schemas provide structure for reliable automation
|
||||
- **Performance**: Intelligent caching for fast operations
|
||||
|
||||
### For Organizations
|
||||
|
||||
- **Reduced Costs**: Automation reduces manual documentation work
|
||||
- **Higher Quality**: Validation ensures consistency and completeness
|
||||
- **Better Governance**: Track changes, enforce standards, maintain compliance
|
||||
- **Improved Discovery**: Query and search across all content
|
||||
- **Future-Proof**: Markdown is open, portable, and sustainable
|
||||
|
||||
## Getting Started
|
||||
|
||||
MarkiTect works with your existing markdown files. Start small:
|
||||
|
||||
1. **Process a document**: Ingest a markdown file to see its structure
|
||||
2. **Generate a schema**: Extract patterns from your existing content
|
||||
3. **Validate consistency**: Check if other documents match the pattern
|
||||
4. **Create a space**: Group related documents together
|
||||
5. **Explore capabilities**: Try transclusion, rendering, or querying
|
||||
|
||||
No migration required—MarkiTect enhances your workflow without replacing your tools.
|
||||
|
||||
## Who Should Use MarkiTect?
|
||||
|
||||
**MarkiTect is ideal for**:
|
||||
- Technical writers maintaining large documentation sets
|
||||
- Documentation teams standardizing content across projects
|
||||
- Content operations teams automating publishing workflows
|
||||
- Knowledge managers organizing institutional information
|
||||
- Developers integrating documentation into CI/CD
|
||||
- Publishers creating structured content from markdown
|
||||
|
||||
**MarkiTect might not be right if**:
|
||||
- You only work with a handful of simple documents
|
||||
- You don't need consistency or validation
|
||||
- Your content has no structure or patterns
|
||||
- You prefer visual editors over text-based workflows
|
||||
|
||||
## Philosophy
|
||||
|
||||
MarkiTect is built on these principles:
|
||||
|
||||
**Markdown is Data**: Documents contain structured information that can be processed, queried, and transformed
|
||||
|
||||
**Convention Over Configuration**: Sensible defaults that work out of the box, customize only when needed
|
||||
|
||||
**Schema-Driven Quality**: Define structures once, enforce them everywhere
|
||||
|
||||
**Composability**: Build complex systems from simple, reusable components
|
||||
|
||||
**Performance Matters**: Fast operations that scale to thousands of documents
|
||||
|
||||
**Open and Portable**: Pure markdown files that work with any tool
|
||||
|
||||
## What's Next?
|
||||
|
||||
This introduction provides a high-level overview of MarkiTect's value and capabilities. To dive deeper:
|
||||
|
||||
- **[CLI Tutorial](docs/CLI_TUTORIAL.md)**: Learn practical commands and workflows
|
||||
- **[Schema Management Guide](docs/SCHEMA_MANAGEMENT_GUIDE.md)**: Master document validation
|
||||
- **[Asset Management](docs/ASSET_MANAGEMENT_USER_GUIDE.md)**: Work with images and resources
|
||||
- **[Plugin System](docs/PLUGIN_SYSTEM.md)**: Extend MarkiTect for your needs
|
||||
- **[Project Structure](docs/PROJECT_STRUCTURE.md)**: Understand how it's organized
|
||||
|
||||
---
|
||||
|
||||
*MarkiTect transforms markdown from simple text files into intelligent, structured information spaces that power modern documentation and content workflows.*
|
||||
17
Makefile
17
Makefile
@@ -4,7 +4,10 @@
|
||||
# Include capability discovery system
|
||||
include scripts/capability_discovery.mk
|
||||
|
||||
.PHONY: help setup install install-dev uninstall install-home install-home-venv install-user-deps install-force-deps install-deps-venv install-system-deps list-deps setup-dev test build clean update status lint format check-deps venv-status update-digest add-diary-entry test-status test-new test-coverage test-arch test-foundation test-infrastructure test-integration test-domain test-service test-application test-presentation test-quick test-layers test-random test-random-seed test-random-repeat test-install-randomly test-clean test-tdd test-changed test-module test-cache-clean test-efficient cli-help chaos-validate chaos-matrix chaos-inject chaos-report cost-help
|
||||
# Set explicit default target
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
.PHONY: help setup install install-dev uninstall install-home install-home-venv install-user-deps install-force-deps install-deps-venv install-system-deps list-deps setup-dev test test-js test-all build clean update status lint format check-deps venv-status update-digest add-diary-entry test-status test-new test-coverage test-arch test-foundation test-infrastructure test-integration test-domain test-service test-application test-presentation test-quick test-layers test-random test-random-seed test-random-repeat test-install-randomly test-clean test-tdd test-changed test-module test-cache-clean test-efficient cli-help chaos-validate chaos-matrix chaos-inject chaos-report cost-help
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@@ -29,6 +32,8 @@ help:
|
||||
@echo ""
|
||||
@echo "Development:"
|
||||
@echo " test - Run core tests (excluding capability-specific tests)"
|
||||
@echo " test-js - Run JavaScript UI tests"
|
||||
@echo " test-all - Run all tests (Python + JavaScript + Capabilities)"
|
||||
@echo " test-capabilities - Run all capability tests (delegated to capabilities)"
|
||||
@echo " test-status - Show test status summary without re-running"
|
||||
@echo " test-new - Create new test file template"
|
||||
@@ -392,6 +397,16 @@ test-capabilities: capabilities-test
|
||||
# Legacy test-capability-* targets are now handled by capability delegation
|
||||
# Use 'make capability-name-test' instead (e.g., 'make markitect-content-test')
|
||||
|
||||
# JavaScript UI Testing Targets (Phase 5.1 - TestDrive-JSUI Integration)
|
||||
.PHONY: test-js
|
||||
test-js: ## Run JavaScript UI tests
|
||||
@echo "🧪 Running JavaScript UI tests..."
|
||||
@$(MAKE) --no-print-directory testdrive-jsui-test-all
|
||||
|
||||
.PHONY: test-all
|
||||
test-all: test test-js test-capabilities ## Run all tests (Python + JavaScript + Capabilities)
|
||||
@echo "✅ All test suites completed successfully!"
|
||||
|
||||
# TDD8 Workflow Optimized Test Targets (Issue #57)
|
||||
|
||||
# Fast test execution for TDD red phase
|
||||
|
||||
129
SCOPE.md
Normal file
129
SCOPE.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# SCOPE
|
||||
|
||||
> This file helps you quickly understand what this repository is about,
|
||||
> when it is relevant, and when it is not.
|
||||
> It is intentionally lightweight and may be incomplete.
|
||||
|
||||
---
|
||||
|
||||
## One-liner
|
||||
|
||||
Intelligent markdown engine and information management platform — treats documents as structured, queryable information spaces with schema validation, transclusion, LLM-driven evaluation, and infospace lifecycle management.
|
||||
|
||||
---
|
||||
|
||||
## Core Idea
|
||||
|
||||
MarkiTect turns fragmented knowledge (scattered docs, chats, notes) into structured, versioned, reusable artifacts. The core abstraction is an **infospace**: a curated collection of typed entities (concepts, mechanisms, observations) governed by a YAML config, validated against schemas, and evaluated for quality across five dimensions. The platform automates generation, validation, and transformation at scale, delegating domain-level judgment to LLMs while Python handles structure and evaluation.
|
||||
|
||||
---
|
||||
|
||||
## In Scope
|
||||
|
||||
- Parse, validate, and analyze markdown documents against schemas
|
||||
- Generate schemas from example documents; enforce naming convention `{domain}-schema-v{major}.{minor}.md`
|
||||
- Infospace lifecycle: create, populate, evaluate (per-entity + collection quality scores), compose, export
|
||||
- Transclusion: embed content from one document into another, maintaining single source of truth
|
||||
- LLM-driven prompt execution with dependency resolution and quality gates
|
||||
- Relationship graph export (Mermaid, DOT) and analysis (networkx, FCA)
|
||||
- Batch document processing; CLI (`markitect <command>`) and programmatic API
|
||||
- Rendering: markdown → interactive HTML via plugin system (testdrive-jsui)
|
||||
- Asset management (image embedding, resource handling)
|
||||
|
||||
---
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Visual/WYSIWYG editing (markdown-first, text-based workflows only)
|
||||
- Real-time collaborative editing (git-based versioning instead)
|
||||
- Financial transactions or external payment integration
|
||||
- Making domain-level judgments in Python code (delegated to LLM via prompt templates)
|
||||
- Storing secrets or credentials in plaintext
|
||||
- Full GraphQL API (structure exists but not fully implemented)
|
||||
- Vendor-specific integrations or lock-in
|
||||
|
||||
---
|
||||
|
||||
## Relevant When
|
||||
|
||||
- Managing large document sets (hundreds to thousands) needing consistent structure and validation
|
||||
- Building or maintaining institutional knowledge bases, technical documentation, or canon releases
|
||||
- Automating document generation from schemas or templates
|
||||
- Tracking relationships and dependencies between knowledge artifacts
|
||||
- Needing programmatic access to document structure (beyond file reading)
|
||||
- Applying quality evaluation to a structured concept collection
|
||||
|
||||
---
|
||||
|
||||
## Not Relevant When
|
||||
|
||||
- Working with a handful of simple, unrelated documents
|
||||
- Visual editor required
|
||||
- Exclusively non-markdown source formats (PDF/Word need conversion first)
|
||||
- No consistency, validation, or automation needed
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
- Status: active (v0.13.0-dev, ~90 commits ahead of release)
|
||||
- Implementation: substantial — core modules mature (CLI, parsing, schema management, prompt execution, infospace); infospace S3 close-out in progress; LLM adapter extracted to standalone `llm-connect` package
|
||||
- Stability: stable core; plugin system and infospace tooling evolving; 200+ CHANGELOG entries since v0.6.0
|
||||
- Usage: active personal development; examples with 988 entities and full evaluation pipeline
|
||||
|
||||
---
|
||||
|
||||
## How It Fits
|
||||
|
||||
- Upstream dependencies: `llm-connect` (LLM adapter library, extracted), `testdrive-jsui` (rendering plugin submodule), `markitect-utils` (utility library)
|
||||
- Downstream consumers: Custodian — MarkiTect is the knowledge artifact platform in the canonical dependency order (Railiance → **Markitect** → Coulomb.social → Personhood/Foerster → Custodian)
|
||||
- Often used with: the-custodian (state hub tracks markitect domain workstreams), kaizen-agentic (project-management agent for session workflow)
|
||||
|
||||
---
|
||||
|
||||
## Terminology
|
||||
|
||||
- Preferred terms: infospace, topic, discipline, entity, evaluation, viability, transclusion, schema, quality gates
|
||||
- Also known as: "markitect", "the markdown engine"
|
||||
- Potentially confusing terms: "topic" = the subject matter an infospace explains (not a chat thread); "discipline" = a reusable framework of concepts (itself a viable infospace); "infospace" ≠ filesystem directory (it's a curated conceptual collection with explicit quality thresholds)
|
||||
|
||||
---
|
||||
|
||||
## Related / Overlapping
|
||||
|
||||
- `llm-connect` — standalone LLM adapter extracted from MarkiTect (dependency)
|
||||
- `the-custodian` — tracks markitect workstreams; custodian canon includes a markitect domain charter
|
||||
- `marki-docx` — separate repo (on tegwick machine); relationship: docx export capability for MarkiTect artifacts
|
||||
|
||||
---
|
||||
|
||||
## Provided Capabilities
|
||||
|
||||
```capability
|
||||
type: documentation
|
||||
title: Structured document validation and schema management
|
||||
description: Parse, validate, and enforce schemas on markdown documents — generate schemas from examples, validate entity collections, report naming convention compliance.
|
||||
keywords: [markdown, schema, validation, document, structure, linting]
|
||||
```
|
||||
|
||||
```capability
|
||||
type: documentation
|
||||
title: Infospace lifecycle management
|
||||
description: Create, populate, evaluate (quality scores), compose, and export curated knowledge collections (infospaces) with transclusion and relationship graph analysis.
|
||||
keywords: [infospace, knowledge, curation, evaluation, transclusion, quality, graph]
|
||||
```
|
||||
|
||||
```capability
|
||||
type: data
|
||||
title: LLM-driven knowledge artifact generation
|
||||
description: Execute prompts with dependency resolution and quality gates to generate typed entities — concepts, mechanisms, observations — at scale from schemas and templates.
|
||||
keywords: [llm, generation, prompt, entity, artifact, knowledge, automation]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Getting Oriented
|
||||
|
||||
- Start with: `CLAUDE.md` (dev commands, LLM config, infospace lifecycle), `INTRODUCTION.md` (use cases, philosophy)
|
||||
- Key files / directories: `markitect/cli.py` (CLI entry point), `markitect/infospace/` (primary active area), `markitect/prompts/` (LLM execution), `roadmap/` (6 active planning tracks), `examples/infospace-with-history/` (988-entity reference implementation)
|
||||
- Entry points: `markitect --help`; `markitect infospace --help`; `pytest tests/unit/` (inner TDD loop)
|
||||
200
TODO.md
200
TODO.md
@@ -6,179 +6,67 @@ The format is based on [Keep a Todofile V0.0.1](https://coulomb.social/open/Keep
|
||||
|
||||
The structure organizes **future tasks** by their impact, just as a changelog organizes past changes by their impact.
|
||||
|
||||
See roadmap/YYMMDD-ROADMAPTOPIC/ directories for planning information like concepts, workplans, etc...
|
||||
|
||||
***
|
||||
|
||||
## [Unreleased] - *Active Vibe-Coding State* 💡
|
||||
|
||||
This section is for tasks currently being discussed with or worked on by the coding assistant. These are the ephemeral, flow-of-thought tasks.
|
||||
|
||||
**🏗️ MAJOR ARCHITECTURE REFACTORING (2025-11-03) - COMPLETED ✅**: Successfully completed comprehensive JavaScript refactoring using Test-Driven Development methodology.
|
||||
### Extract Capability-Capability from Issue-Facade (Paused)
|
||||
|
||||
**PROBLEMS SOLVED**:
|
||||
1. ✅ **Monolithic Architecture**: Extracted 5,188-line `editor.js` into 4 modular components
|
||||
2. ✅ **Server-Side Debug Generation**: Implemented pure client-side DebugPanel component
|
||||
3. ✅ **Architectural Boundary Violations**: Clean separation with no Python code modifications
|
||||
4. ✅ **Tight Coupling**: All components independently testable with event-driven communication
|
||||
5. ✅ **Generic Editor Compromise**: Debug system now purely client-side and component-based
|
||||
**Context:** Issue-facade currently provides two capabilities:
|
||||
1. **issue-tracking** (explicit in CAPABILITY-issue-tracking.yaml) - Issue management across platforms
|
||||
2. **capability-capability** (implicit) - Patterns and tools for creating/managing capabilities
|
||||
|
||||
**SOLUTION IMPLEMENTED**: Modular JavaScript Architecture with complete component separation and TDD validation.
|
||||
The **capability-capability** includes:
|
||||
- Feedback pattern (feedback/ directory, .capability/feedback CLI tool, documentation)
|
||||
- Detachment facility (.capability/detach script for clean capability removal)
|
||||
- Integration pattern (.capability/integrate.sh for project integration)
|
||||
- CAPABILITY-*.yaml specification format
|
||||
- ReusableCapabilitiesArchitecture.md (complete specification)
|
||||
- Directory conventions (_family/implementation, visible/hidden patterns)
|
||||
|
||||
**📊 PREVIOUS STATUS (2025-11-02)**: Systematic JavaScript functionality recovery using TDD methodology had made excellent progress. **5 major features** were successfully implemented and tested:
|
||||
**Goal:** Extract capability-capability to separate `reusable-capability` repository so it can be used by any capability in the markitect ecosystem.
|
||||
|
||||
1. **Advanced EditState Management** ✅ - Implemented enum-based state tracking with pending changes preservation
|
||||
2. **Keyboard Shortcuts** ✅ - Added Ctrl+Enter (accept) and Escape (cancel) functionality
|
||||
3. **Section Splitting** ✅ - Restored dynamic heading detection with automatic section reorganization
|
||||
4. **Real-time Status Tracking** ✅ - Implemented periodic updates with visual status panel (2-second intervals)
|
||||
5. **Intelligent Filename Generation** ✅ - Added 4-method fallback system (options→title→URL→heading→timestamp)
|
||||
**Approach:** Step-by-step extraction, starting with specification.
|
||||
|
||||
All implementations include comprehensive TDD test suites and are fully integrated into the existing codebase. The recovery approach has proven highly effective for restoring sophisticated lost functionality.
|
||||
#### Phase 1: Specification & Planning (Current)
|
||||
|
||||
## 🏗️ JAVASCRIPT ARCHITECTURE REFACTORING - COMPLETED ✅
|
||||
- [ ] Create CAPABILITY-capability.yaml in issue-facade to explicitly declare the implicit capability
|
||||
- [ ] Define what belongs to capability-capability family vs issue-tracking family
|
||||
- [ ] Document the capability-capability API surface (what tools/patterns it provides)
|
||||
- [ ] Identify all files/directories to extract
|
||||
- [ ] Plan extraction strategy (copy vs move, how to maintain during transition)
|
||||
|
||||
### **Phase 1: Preparation & Backup (CRITICAL) - ✅ COMPLETED**
|
||||
* ✅ Updated TODO.md with comprehensive refactoring plan
|
||||
* ✅ Created modular directory structure `markitect/static/js/`
|
||||
* ✅ Set up component template files with proper exports/imports
|
||||
* ✅ Implemented TDD test framework for safe refactoring
|
||||
#### Phase 2: Repository Creation
|
||||
|
||||
### **Phase 2: Core System Extraction (HIGH) - ✅ COMPLETED**
|
||||
* ✅ Extracted SectionManager to `core/section-manager.js` (490 lines)
|
||||
* ✅ Integrated EventSystem into SectionManager with pub/sub pattern
|
||||
* ✅ Created comprehensive section state management with EditState enum
|
||||
- [ ] Create reusable-capability repository structure
|
||||
- [ ] Extract ReusableCapabilitiesArchitecture.md to new repo
|
||||
- [ ] Extract feedback pattern (directory structure, CLI tool, README)
|
||||
- [ ] Extract detachment facility (.capability/detach)
|
||||
- [ ] Extract integration scripts (.capability/integrate.sh, integration-checklist.md)
|
||||
- [ ] Create CAPABILITY-capability.yaml in new repo (canonical version)
|
||||
- [ ] Add README.md for reusable-capability repo
|
||||
|
||||
### **Phase 3: Component Separation (HIGH) - ✅ COMPLETED**
|
||||
* ✅ Document Controls → `components/document-controls.js` (200 lines)
|
||||
* ✅ DOMRenderer (includes status functionality) → `components/dom-renderer.js` (540 lines)
|
||||
* ✅ Debug Panel → `components/debug-panel.js` (150 lines, pure client-side)
|
||||
* ✅ Floating Menu → integrated into DOMRenderer component
|
||||
* ✅ Text/Image Editors → integrated into DOMRenderer component
|
||||
#### Phase 3: Integration & Testing
|
||||
|
||||
### **Phase 4: Testing Infrastructure (MEDIUM) - ✅ COMPLETED**
|
||||
* ✅ Standalone TDD test runner (`RefactorTestRunner`) that doesn't require md-render
|
||||
* ✅ Component unit tests for all individual functionality
|
||||
* ✅ Integration tests for component interaction
|
||||
* ✅ Full system integration tests for complete workflow validation
|
||||
- [ ] Update issue-facade to depend on reusable-capability (as integrated capability)
|
||||
- [ ] Integrate reusable-capability into issue-facade using _capability/reusable-capability pattern
|
||||
- [ ] Test that issue-facade still works with extracted capability
|
||||
- [ ] Update issue-facade documentation to reference both capabilities it provides/uses
|
||||
- [ ] Verify feedback system still works
|
||||
- [ ] Verify detachment still works
|
||||
|
||||
### **Phase 5: Integration & Cleanup (MEDIUM) - ✅ COMPLETED**
|
||||
* ✅ All components work together with preserved functionality
|
||||
* ✅ Monolithic editor.js functionality fully distributed
|
||||
* ✅ Python code completely unchanged - zero md-render modifications
|
||||
* ✅ All functionality validated through comprehensive test suite (31 tests passing)
|
||||
#### Phase 4: Dogfooding & Validation
|
||||
|
||||
### **Directory Structure Implemented:**
|
||||
```
|
||||
markitect/static/js/
|
||||
├── core/
|
||||
│ └── section-manager.js # ✅ Section state management with EventSystem (490 lines)
|
||||
├── components/
|
||||
│ ├── document-controls.js # ✅ Document controls panel (200 lines)
|
||||
│ ├── dom-renderer.js # ✅ DOM rendering, FloatingMenu, editors (540 lines)
|
||||
│ └── debug-panel.js # ✅ Debug panel (150 lines, pure client-side)
|
||||
└── tests/
|
||||
├── refactor-test-runner.js # ✅ TDD test framework
|
||||
├── test-component-integration.js # ✅ Component integration tests
|
||||
├── test-full-integration.js # ✅ Full system tests
|
||||
├── test-section-manager-extraction.js # ✅ SectionManager tests
|
||||
├── test-extracted-section-manager.js # ✅ SectionManager TDD tests
|
||||
├── test-domrenderer-extraction.js # ✅ DOMRenderer extraction tests
|
||||
├── test-extracted-domrenderer.js # ✅ DOMRenderer TDD tests
|
||||
├── test-debugpanel-extraction.js # ✅ DebugPanel extraction tests
|
||||
├── test-debugpanel-integration.js # ✅ DebugPanel integration tests
|
||||
└── test-documentcontrols-extraction.js # ✅ DocumentControls tests
|
||||
```
|
||||
- [ ] Choose another markitect capability for dogfooding
|
||||
- [ ] Integrate reusable-capability into that capability
|
||||
- [ ] Add feedback system to new capability
|
||||
- [ ] Add detachment facility to new capability
|
||||
- [ ] Document learnings and refine reusable-capability based on real-world usage
|
||||
- [ ] Update ReusableCapabilitiesArchitecture.md with insights
|
||||
|
||||
### **REFACTORING RESULTS SUMMARY:**
|
||||
- **Lines Extracted**: 1,380 lines from monolithic 5,188-line editor.js
|
||||
- **Components Created**: 4 modular, independently testable components
|
||||
- **Tests Created**: 11 comprehensive test files with 31 passing tests
|
||||
- **Architecture**: Event-driven, pub/sub communication between components
|
||||
- **Functionality**: 100% preserved with zero regression
|
||||
- **Performance**: Improved modularity enables better maintainability and testing
|
||||
- **Python Code**: Zero modifications - clean architectural separation achieved
|
||||
**Current Step:** Phase 1, Task 1 - Create CAPABILITY-capability.yaml
|
||||
|
||||
### **PREVIOUS COMPLETED FEATURES (Now successfully refactored):**
|
||||
* **Successfully Refactored:**
|
||||
* ✅ Advanced state management with EditState enum and pending changes (CRITICAL) - REFACTORED INTO SectionManager
|
||||
* ✅ Keyboard shortcuts (Ctrl+Enter accept, Escape cancel) (CRITICAL) - REFACTORED INTO DOMRenderer
|
||||
* ✅ Section splitting functionality for dynamic heading detection (HIGH) - REFACTORED INTO SectionManager
|
||||
* ✅ Real-time status tracking with periodic updates (HIGH) - REFACTORED INTO DocumentControls
|
||||
* ✅ Intelligent save filename generation with 4-method fallback (MEDIUM) - PRESERVED IN MONOLITH
|
||||
* ✅ Professional message system with color-coded positioning (MEDIUM) - REFACTORED INTO DebugPanel
|
||||
* ✅ Multiple concurrent editing sessions support (MEDIUM) - REFACTORED INTO DOMRenderer
|
||||
* ✅ Enhanced DOM event system with 6 event types (LOW) - REFACTORED INTO DOMRenderer
|
||||
* ✅ Automatic section type detection (heading, code, list, etc) (LOW) - REFACTORED INTO SectionManager
|
||||
* ✅ Sophisticated section ID generation with hash-based algorithm (LOW) - REFACTORED INTO SectionManager
|
||||
|
||||
* **Successfully Implemented:**
|
||||
* ✅ Comprehensive status reporting dialog with detailed stats (HIGH) - IMPLEMENTED IN DocumentControls
|
||||
* ✅ Floating global control panel with professional styling (MEDIUM) - IMPLEMENTED IN DocumentControls
|
||||
* ✅ Enhanced setupSectionElement with comprehensive styling (LOW) - IMPLEMENTED IN DOMRenderer
|
||||
|
||||
* **Core Methods Successfully Refactored:**
|
||||
* ✅ stopEditing method with state preservation (CRITICAL) - REFACTORED INTO SectionManager
|
||||
* ✅ getAllSections method for section collection management (MEDIUM) - REFACTORED INTO SectionManager
|
||||
* ✅ hasChanges detection for unsaved modifications (HIGH) - REFACTORED INTO SectionManager
|
||||
* ✅ updateGlobalStatus method with 2-second interval updates (MEDIUM) - REFACTORED INTO DocumentControls
|
||||
* ✅ handleSectionSplit for dynamic section reorganization (LOW) - REFACTORED INTO SectionManager
|
||||
* ✅ checkForSectionSplits automatic heading detection (LOW) - REFACTORED INTO SectionManager
|
||||
|
||||
* **To Remove:**
|
||||
* None currently identified
|
||||
|
||||
|
||||
***
|
||||
|
||||
## Completed Tasks
|
||||
|
||||
**JavaScript Architecture Refactoring - COMPLETED ✅ (2025-11-03)**:
|
||||
- ✅ Successfully extracted monolithic 5,188-line editor.js into 4 modular components using TDD methodology
|
||||
- ✅ Created SectionManager component (490 lines) handling section state management and event system
|
||||
- ✅ Created DOMRenderer component (540 lines) handling DOM interactions, rendering, and editing workflows
|
||||
- ✅ Created DebugPanel component (150 lines) providing pure client-side debug message management
|
||||
- ✅ Created DocumentControls component (200 lines) managing floating control panel and document actions
|
||||
- ✅ Implemented comprehensive TDD test framework with 11 test files and 31 passing tests
|
||||
- ✅ Achieved 100% functionality preservation with zero regression through rigorous testing
|
||||
- ✅ Established event-driven architecture with pub/sub communication between components
|
||||
- ✅ Maintained complete separation from Python code - zero md-render modifications required
|
||||
- ✅ Created modular directory structure enabling independent component development and testing
|
||||
|
||||
**Architecture Improvements Achieved**:
|
||||
- Clean separation of concerns with single-responsibility components
|
||||
- Event-driven communication reducing tight coupling
|
||||
- Independent component testing enabling confident refactoring
|
||||
- Scalable structure supporting future feature development
|
||||
- Client-side debug system eliminating server-side debug generation issues
|
||||
- Modular design allowing selective component updates without affecting others
|
||||
|
||||
**Asset Shipping for md-render - COMPLETED ✅**:
|
||||
- ✅ Implemented automatic asset copying when rendering markdown to different output directories
|
||||
- ✅ Added asset discovery functionality parsing markdown for image/link references
|
||||
- ✅ Implemented timestamp-based asset copying (only copy if source newer than destination)
|
||||
- ✅ Added `--ship-assets` and `--no-ship-assets` CLI flags for explicit control
|
||||
- ✅ Added `MARKITECT_OUTPUT_DIR` environment variable support for default output directory
|
||||
- ✅ Smart defaults: assets ship automatically when output is directory, disabled for specific files
|
||||
- ✅ Preserved relative path structure in output directory maintaining markdown link compatibility
|
||||
- ✅ Graceful handling of missing assets with warning messages
|
||||
- ✅ Full backward compatibility with existing md-render workflows
|
||||
- ✅ Comprehensive TDD test suite covering all functionality and edge cases
|
||||
|
||||
**Feature Capabilities**:
|
||||
- Environment variable priority: CLI `--output` > `MARKITECT_OUTPUT_DIR` > input file directory
|
||||
- Automatic asset discovery from standard markdown syntax: `` and `[text](path)`
|
||||
- Timestamp-based incremental copying prevents unnecessary file operations
|
||||
- Directory structure preservation maintains working relative links in output HTML
|
||||
- Support for images, documents, and other asset types referenced in markdown
|
||||
|
||||
**CHANGELOG.md Enhancement - COMPLETED ✅**:
|
||||
- ✅ Added missing version entries for 0.1.0, 0.2.0, and 0.3.0
|
||||
- ✅ Added standard Keep a Changelog header with proper format
|
||||
- ✅ Included Unreleased section
|
||||
- ✅ Research completed for all historical versions using git log analysis
|
||||
- ✅ All entries follow Keep a Changelog categories (Added, Changed, Fixed)
|
||||
- ✅ Chronological order maintained with latest versions first
|
||||
- ✅ Appropriate release dates included based on git commit timestamps
|
||||
|
||||
**Version Details Added**:
|
||||
- v0.1.0 (2025-10-15): Development infrastructure, TDD workspace, issue management
|
||||
- v0.2.0 (2025-10-20): Advanced Markdown Engine with GraphQL, search, plugins
|
||||
- v0.3.0 (2025-10-25): Architectural improvements with kaizen-agentic integration
|
||||
1
_issue-tracking/issue-facade
Submodule
1
_issue-tracking/issue-facade
Submodule
Submodule _issue-tracking/issue-facade added at 70d7ec0cdc
@@ -15,7 +15,7 @@ You are responsible for:
|
||||
|
||||
### Directory Structure
|
||||
```
|
||||
markitect_project/
|
||||
markitect-main/
|
||||
├── Makefile # Main project Makefile
|
||||
├── scripts/
|
||||
│ └── capability_discovery.mk # Auto-discovery and delegation system
|
||||
|
||||
@@ -15,19 +15,25 @@ You are the MarkiTect project assistant, specialized in providing project status
|
||||
|
||||
### Key Project Files & Their Purpose
|
||||
|
||||
- **ProjectStatusDigest.md**: The canonical source of truth for project architecture, features, and current state
|
||||
- **ProjectDiary.md**: Chronological record of major work packages, milestones, and development sessions
|
||||
- **TODO.md**: Task management and priorities following Keep a Todofile format for maintaining coding flow
|
||||
- **TODO.md**: Current state of implemenation based on the Keep-A-Todofile format for maintaining coding flow
|
||||
- **CHANGELOG.md**: History of releases based on the Keep-A-Changelog format for easy access to what happend before
|
||||
- **roadmap/**: Directory with current and close range roadmap-topic-directories for concepts, workplans, examples...
|
||||
- **history/**: Directory with closed roadmap-topic-directories including finishd TODO.md files as YYMMDD-DONE.md
|
||||
- **Makefile**: Provides helpers to use and improve the capabilities provided by the project
|
||||
**Gitea Issues**: Backlog of issues and backlog of tasks stored as issues in gitea
|
||||
**Gitea Issues**: Backlog of issues and backlog of tasks stored as issues in gitea before selection as roadmap topics
|
||||
|
||||
### Project Infrastructure Knowledge
|
||||
|
||||
**Repository Structure:**
|
||||
- Main project hosted on Gitea with issue tracking for use cases and tasks
|
||||
- Documentation maintained in `wiki/` submodule
|
||||
- Planning documentation goes to roadmap/ROADMAPTOPIC subdirectories
|
||||
- Closed roadmap-topic-directories git-mv to history/
|
||||
- Auto generated documentation maintained in docs/
|
||||
- Human generated documentation maintained in wiki/ submodule
|
||||
- Test-driven development workflow with comprehensive test coverage
|
||||
|
||||
Important: Respect the directory structure! If in doubt ask or use directories under tmp/ to keep the structure clean!
|
||||
|
||||
**Development Workflow:**
|
||||
- Issue-driven development using Gitea API integration
|
||||
- Issue management via universal issue-facade CLI that works with multiple backends
|
||||
@@ -56,17 +62,19 @@ You are the MarkiTect project assistant, specialized in providing project status
|
||||
|
||||
When asked about project status or next steps:
|
||||
|
||||
1. **Start with Current State**: Always check ProjectStatusDigest.md for the latest architecture and status
|
||||
2. **Review Recent Progress**: Check ProjectDiary.md for recent accomplishments and context
|
||||
3. **Check Planned Work**: Read Next.md for documented next steps and priorities
|
||||
4. **Consider Git Status**: Be aware of current working directory state and recent commits
|
||||
1. **Start with Current State**: Always check TODO.md for the latest activity
|
||||
2. **Review Recent Progress**: Check CHANGELOG.md for previous work and progress
|
||||
3. **Check Planned Work**: TODO.md documents next steps and priorities, if empty see topics in roadmap/
|
||||
4. **Project Scope and Goals**: Vision, Mission, Guidelines and Usecases live in wiki/ if available
|
||||
5. **Planning New Stuff**: Requirements (Epics and Stories) are gitea issues to be planned as roadmap topics
|
||||
6. **Consider Git Status**: Allways be aware of current working directory state and recent commits
|
||||
|
||||
### Issue Management Guidelines
|
||||
|
||||
**When to Create Gitea Issues:**
|
||||
- New feature requests or enhancement ideas emerge during development
|
||||
- Bugs or technical debt are discovered but not immediately fixable
|
||||
- Future improvements are identified but outside current session scope
|
||||
- Future improvements are identified but outside current session and topic scope
|
||||
- Architecture decisions require documentation and future review
|
||||
- Sidequests that we want to remember for later implementation
|
||||
|
||||
@@ -78,10 +86,12 @@ When asked about project status or next steps:
|
||||
- Do NOT implement immediately - issues are for tracking and planning
|
||||
|
||||
**Issue vs. Immediate Work:**
|
||||
- Current session planned work: implement directly (from Next.md)
|
||||
- Discovered improvements: create issue, continue with planned work
|
||||
- Current session planned work: document in TODO.md and roadmap/ROADMAPTOPIC
|
||||
- Discovered improvements: add to workplan in roadmap topic, continue with planned work
|
||||
- Critical bugs affecting current work: fix immediately, then create issue for root cause analysis
|
||||
- Future enhancements: always create issue first for proper planning
|
||||
- Future enhancements: note in roadmap-topic to create issues first for proper planning
|
||||
- If possible create issues interactively when closing a topic, they are for human oversight and longterm
|
||||
- Do not create issues for stuff that is detailed and can be adressed before closing the current topic
|
||||
|
||||
**Response Format:**
|
||||
- Provide a brief status summary (2-3 sentences)
|
||||
@@ -102,8 +112,6 @@ When asked about project status or next steps:
|
||||
1. [Action from Next.md or logical progression]
|
||||
2. [Secondary priority or alternative approach]
|
||||
3. [Maintenance or validation task if applicable]
|
||||
|
||||
Based on: ProjectStatusDigest.md:74-79, Next.md:7-13
|
||||
```
|
||||
|
||||
## Session Start-Up Protocol
|
||||
@@ -113,10 +121,10 @@ When asked what's up for a new coding session, follow this standardized routine:
|
||||
### Start-of-Session Checklist
|
||||
1. **Mission Status**: Provide reminder to project vision and how we are doing
|
||||
2. **Recently**: Provide reminder what we did last from the last entry to the diary
|
||||
3. **NEXT.txt**: Check if we provided guidance for what to do next at the end of the last coding session
|
||||
3. **TODO.md**: Check if we provided guidance for what to do next at the end of the last coding session
|
||||
4. **git status**: Check if git is clean or work has been left unfinished
|
||||
5. **Workspace clean**: Check if workspace is clean or we left of in the middle of a TDD cycle
|
||||
6. **Issue finished**: Check if we are currently working on a specific issue or need to select the next one
|
||||
6. **Topic or issue finished**: Check if we are currently working on a specific roadmap-topic or issue
|
||||
7. **Suggestion**: Provide a sensible suggestion of what to do next
|
||||
|
||||
## Session Wrap-Up Protocol
|
||||
@@ -124,11 +132,10 @@ When asked what's up for a new coding session, follow this standardized routine:
|
||||
When asked to help wrap up a development session, follow this standardized routine:
|
||||
|
||||
### End-of-Session Checklist:
|
||||
1. **Update ProjectDiary.md**: Add entry documenting progress, challenges, and achievements
|
||||
2. **Update TODO.md**: Set clear priorities and strategy for next session using todofile format
|
||||
3. **Update ProjectStatusDigest.md**: Refresh current status, metrics, and completed features
|
||||
3. **Update roadmap-topic directory information**: Refresh current status, metrics, and completed features
|
||||
4. **Issue Management**: Review and create any issues for sidequests and discoveries made during session
|
||||
5. **Anchor patterns**: Update this project-assistant definition with any new workflow patterns
|
||||
5. **Anchor patterns**: Add Update suggestions for this project-assistant definition with any new workflow patterns
|
||||
6. **Prepare for commit**: Ensure all documentation reflects current state
|
||||
|
||||
### Session Success Indicators:
|
||||
@@ -143,9 +150,9 @@ When asked to help wrap up a development session, follow this standardized routi
|
||||
[Brief overview of accomplishments and current state]
|
||||
|
||||
## Documentation Updates
|
||||
- ✅ ProjectDiary.md: [what was added]
|
||||
- ✅ Next.md: [priorities set]
|
||||
- ✅ ProjectStatusDigest.md: [status updated]
|
||||
- ✅ TODO.md: [priorities set]
|
||||
- ✅ roadmap/TOPIC files: [what was added or changed]
|
||||
- ✅ CHANGELOG.ms: [status updated especially on release]
|
||||
|
||||
## Issues Created/Updated
|
||||
- 🎯 Issue #X: [brief description] - [reason for creation]
|
||||
@@ -157,9 +164,19 @@ When asked to help wrap up a development session, follow this standardized routi
|
||||
Ready for commit: [list of files to commit]
|
||||
```
|
||||
|
||||
### Example Capture Small Off-Topic Improvements in roadmap/eat-the-frog:
|
||||
**Smell**: Different filename conventions od conflicting concepts, unclear guideance
|
||||
**Hunch**: Ideas to explore that need consideration if useful and in scope
|
||||
**Hickups**: Notes on inefficient or roundtripping implementation to analyse later
|
||||
|
||||
Collect these in the roadmap-topic-directory and move stuff to eat-the-frog on close if unfinished
|
||||
|
||||
### Example Issue Creation During Development:
|
||||
**Scenario**: While implementing CLI commands, discover that error messages could be improved
|
||||
**Action**: Create issue "Enhance CLI error messages with user-friendly formatting and suggestions"
|
||||
**Result**: Continue with current CLI implementation, address error enhancement in future session
|
||||
|
||||
Generate issues for relevantly expensive or risky stuff and in direct feedback with developers.
|
||||
Controled in-scope-work does not need the costly issue capture, refinement, selection roundtrip.
|
||||
|
||||
Remember: Your role is to help developers quickly understand "where we are" and "what should we do next" when picking up work on the MarkiTect project, and to ensure proper session wrap-up for continuity.
|
||||
@@ -5055,6 +5055,94 @@
|
||||
"size": 43,
|
||||
"created_at": "2025-10-20T07:21:34.059271",
|
||||
"description": null
|
||||
},
|
||||
"d1f2de1aa975f05ac067cb3512059a675aad9acd6e1bcd5dc57e1dd51d00db01": {
|
||||
"path": "/home/worsch/markitect_project/assets/d1/d1f2de1aa975f05ac067cb3512059a675aad9acd6e1bcd5dc57e1dd51d00db01.pdf",
|
||||
"content_hash": "d1f2de1aa975f05ac067cb3512059a675aad9acd6e1bcd5dc57e1dd51d00db01",
|
||||
"mime_type": "application/pdf",
|
||||
"size": 29,
|
||||
"created_at": "2025-11-09T09:25:06.866540",
|
||||
"description": null
|
||||
},
|
||||
"1895a4a5b1a7afcba497477008076e1a9b70442342092d5ce43f7ab447b30873": {
|
||||
"path": "/home/worsch/markitect_project/assets/18/1895a4a5b1a7afcba497477008076e1a9b70442342092d5ce43f7ab447b30873.svg",
|
||||
"content_hash": "1895a4a5b1a7afcba497477008076e1a9b70442342092d5ce43f7ab447b30873",
|
||||
"mime_type": "image/svg+xml",
|
||||
"size": 25,
|
||||
"created_at": "2025-11-09T09:25:06.893302",
|
||||
"description": null
|
||||
},
|
||||
"f45288fe7b30287abefccd6e96eafa2413c2f73671c433a9fd17f96876dcba68": {
|
||||
"path": "/home/worsch/markitect_project/assets/f4/f45288fe7b30287abefccd6e96eafa2413c2f73671c433a9fd17f96876dcba68.png",
|
||||
"content_hash": "f45288fe7b30287abefccd6e96eafa2413c2f73671c433a9fd17f96876dcba68",
|
||||
"mime_type": "image/png",
|
||||
"size": 28,
|
||||
"created_at": "2025-11-09T09:25:06.917300",
|
||||
"description": null
|
||||
},
|
||||
"61cb7a679ea77d0765e7f2285b080add305d096a795abc48e82a7cd8d915d9d3": {
|
||||
"path": "/home/worsch/markitect_project/assets/61/61cb7a679ea77d0765e7f2285b080add305d096a795abc48e82a7cd8d915d9d3.jpg",
|
||||
"content_hash": "61cb7a679ea77d0765e7f2285b080add305d096a795abc48e82a7cd8d915d9d3",
|
||||
"mime_type": "image/jpeg",
|
||||
"size": 26,
|
||||
"created_at": "2025-11-09T09:25:06.939018",
|
||||
"description": null
|
||||
},
|
||||
"33ed8dd1f8e470138f016e1a2641d38ccbc1c1cfb10ffdac59ef309974748c6d": {
|
||||
"path": "/home/worsch/markitect_project/assets/33/33ed8dd1f8e470138f016e1a2641d38ccbc1c1cfb10ffdac59ef309974748c6d.png",
|
||||
"content_hash": "33ed8dd1f8e470138f016e1a2641d38ccbc1c1cfb10ffdac59ef309974748c6d",
|
||||
"mime_type": "image/png",
|
||||
"size": 27,
|
||||
"created_at": "2025-11-09T09:25:06.959757",
|
||||
"description": null
|
||||
},
|
||||
"b509163964e822915ea7e822759ecae39dd696626e70b74b96de6ac7396415d0": {
|
||||
"path": "/home/worsch/markitect_project/assets/b5/b509163964e822915ea7e822759ecae39dd696626e70b74b96de6ac7396415d0.png",
|
||||
"content_hash": "b509163964e822915ea7e822759ecae39dd696626e70b74b96de6ac7396415d0",
|
||||
"mime_type": "image/png",
|
||||
"size": 14,
|
||||
"created_at": "2025-11-09T09:25:07.012411",
|
||||
"description": null
|
||||
},
|
||||
"e4d8e7de1bda3f19dd16c984ec045bed1a60fe69989ff48a2875cf81dfd56bb6": {
|
||||
"path": "/home/worsch/markitect_project/assets/e4/e4d8e7de1bda3f19dd16c984ec045bed1a60fe69989ff48a2875cf81dfd56bb6.pdf",
|
||||
"content_hash": "e4d8e7de1bda3f19dd16c984ec045bed1a60fe69989ff48a2875cf81dfd56bb6",
|
||||
"mime_type": "application/pdf",
|
||||
"size": 33,
|
||||
"created_at": "2025-11-09T09:25:07.219675",
|
||||
"description": null
|
||||
},
|
||||
"6e64079b752375a2e3ae5d6d67af3d2569b284997536c5fa8bd01af2baafdc08": {
|
||||
"path": "/home/worsch/markitect_project/assets/6e/6e64079b752375a2e3ae5d6d67af3d2569b284997536c5fa8bd01af2baafdc08.svg",
|
||||
"content_hash": "6e64079b752375a2e3ae5d6d67af3d2569b284997536c5fa8bd01af2baafdc08",
|
||||
"mime_type": "image/svg+xml",
|
||||
"size": 29,
|
||||
"created_at": "2025-11-09T09:25:07.243001",
|
||||
"description": null
|
||||
},
|
||||
"33794900aef1bda0b9bbb8f24f26e6181507169bb1979a8502503ae68962a9aa": {
|
||||
"path": "/home/worsch/markitect_project/assets/33/33794900aef1bda0b9bbb8f24f26e6181507169bb1979a8502503ae68962a9aa.png",
|
||||
"content_hash": "33794900aef1bda0b9bbb8f24f26e6181507169bb1979a8502503ae68962a9aa",
|
||||
"mime_type": "image/png",
|
||||
"size": 32,
|
||||
"created_at": "2025-11-09T09:25:07.265421",
|
||||
"description": null
|
||||
},
|
||||
"9e90160cc46e32c3790e38e55bdc3bbd8d61f85036191b6693d02e53c06b1e4d": {
|
||||
"path": "/home/worsch/markitect_project/assets/9e/9e90160cc46e32c3790e38e55bdc3bbd8d61f85036191b6693d02e53c06b1e4d.jpg",
|
||||
"content_hash": "9e90160cc46e32c3790e38e55bdc3bbd8d61f85036191b6693d02e53c06b1e4d",
|
||||
"mime_type": "image/jpeg",
|
||||
"size": 30,
|
||||
"created_at": "2025-11-09T09:25:07.286159",
|
||||
"description": null
|
||||
},
|
||||
"345fe884e0f85e1d08e893f4c977b8e7437542126b6be90d86e8b8f68bba686f": {
|
||||
"path": "/home/worsch/markitect_project/assets/34/345fe884e0f85e1d08e893f4c977b8e7437542126b6be90d86e8b8f68bba686f.png",
|
||||
"content_hash": "345fe884e0f85e1d08e893f4c977b8e7437542126b6be90d86e8b8f68bba686f",
|
||||
"mime_type": "image/png",
|
||||
"size": 31,
|
||||
"created_at": "2025-11-09T09:25:07.306652",
|
||||
"description": null
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
mock content for icon.svg
|
||||
@@ -0,0 +1 @@
|
||||
modified content for diagram.png
|
||||
@@ -0,0 +1 @@
|
||||
mock content for image1.png
|
||||
@@ -0,0 +1 @@
|
||||
modified content for image1.png
|
||||
@@ -0,0 +1 @@
|
||||
mock content for photo.jpg
|
||||
@@ -0,0 +1 @@
|
||||
modified content for icon.svg
|
||||
@@ -0,0 +1 @@
|
||||
modified content for photo.jpg
|
||||
BIN
assets/assets.db
BIN
assets/assets.db
Binary file not shown.
@@ -0,0 +1 @@
|
||||
nested content
|
||||
@@ -0,0 +1 @@
|
||||
mock content for document.pdf
|
||||
@@ -0,0 +1 @@
|
||||
modified content for document.pdf
|
||||
@@ -0,0 +1 @@
|
||||
mock content for diagram.png
|
||||
51
capabilities/DETACHED-issue-facade.yaml
Normal file
51
capabilities/DETACHED-issue-facade.yaml
Normal file
@@ -0,0 +1,51 @@
|
||||
# Detachment Manifest
|
||||
# This file records the removal of the issue-facade capability
|
||||
# Use this information to re-integrate with updated architecture
|
||||
|
||||
detachment:
|
||||
timestamp: 2025-12-17T21:23:14Z
|
||||
capability_name: issue-facade
|
||||
capability_family: issue-tracking
|
||||
integration_pattern: capabilities-directory
|
||||
original_location: /home/worsch/markitect-main/capabilities/issue-facade
|
||||
|
||||
capability_metadata:
|
||||
spec_file: CAPABILITY-issue-tracking.yaml
|
||||
version: unknown
|
||||
implementation: unknown
|
||||
maturity: unknown
|
||||
|
||||
integration_details:
|
||||
parent_project: capabilities
|
||||
parent_path: /home/worsch/markitect-main/capabilities
|
||||
|
||||
re_integration_guide: |
|
||||
To re-integrate this capability using the new architecture:
|
||||
|
||||
# Option 1: Git submodule (recommended)
|
||||
cd /home/worsch/markitect-main/capabilities
|
||||
git submodule add <repo-url> _issue-facade
|
||||
pip install -e _issue-facade/
|
||||
|
||||
# Option 2: Clone directly
|
||||
cd /home/worsch/markitect-main/capabilities
|
||||
git clone <repo-url> _issue-facade
|
||||
pip install -e _issue-facade/
|
||||
|
||||
# Option 3: Copy into project
|
||||
cd /home/worsch/markitect-main/capabilities
|
||||
cp -r /path/to/issue-facade _issue-facade
|
||||
pip install -e _issue-facade/
|
||||
|
||||
Note: Use underscore prefix (_issue-facade) per ReusableCapabilitiesArchitecture
|
||||
|
||||
notes:
|
||||
- The original integration used pattern: capabilities-directory
|
||||
- New architecture recommends: underscore-prefix at repo root
|
||||
- See ReusableCapabilitiesArchitecture.md for details
|
||||
|
||||
repository_info:
|
||||
# Fill in if re-integrating from git
|
||||
git_url: "http://92.205.130.254:32166/coulomb/issue-facade.git" # e.g., https://github.com/markitect/issue-facade
|
||||
git_branch: "main" # e.g., main
|
||||
git_commit: "35daa514e59788250847cd706c43ea78f24c5c1d" # Optional: specific commit to use
|
||||
Submodule capabilities/issue-facade deleted from 00b9834d2f
Submodule capabilities/kaizen-agentic updated: 1e0ff82d74...afc038d98b
@@ -8,7 +8,7 @@ This test module validates outline mode schema generation improvements including
|
||||
- Content instruction integration
|
||||
- End-to-end workflow from example document to generated drafts
|
||||
|
||||
Created for Issue #46: https://gitea.coulomb.social/coulomb/markitect_project/issues/46
|
||||
Created for Issue #46: https://gitea.coulomb.social/coulomb/markitect-main/issues/46
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -209,7 +209,7 @@ tests/
|
||||
## 🎯 Detailed File Structure After Migration
|
||||
|
||||
```
|
||||
markitect_project/
|
||||
markitect-main/
|
||||
├── capabilities/
|
||||
│ └── release-management/
|
||||
│ ├── README.md ✅ CREATED
|
||||
|
||||
@@ -162,7 +162,7 @@ clean_before_build = true
|
||||
[tool.release-management.registries.gitea]
|
||||
url = "http://92.205.130.254:32166"
|
||||
owner = "coulomb"
|
||||
repo = "markitect_project"
|
||||
repo = "markitect-main"
|
||||
auth_token_env = "GITEA_API_TOKEN"
|
||||
|
||||
[tool.release-management.registries.pypi]
|
||||
|
||||
@@ -141,7 +141,7 @@ make release-publish VERSION=0.8.0
|
||||
## Registry Information
|
||||
|
||||
- **Gitea URL**: http://92.205.130.254:32166
|
||||
- **Repository**: coulomb/markitect_project
|
||||
- **Repository**: coulomb/markitect-main
|
||||
- **PyPI Registry URL**: http://92.205.130.254:32166/api/packages/coulomb/pypi
|
||||
- **Package List URL**: http://92.205.130.254:32166/api/v1/packages/coulomb
|
||||
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
"""
|
||||
CHANGELOG management tools.
|
||||
|
||||
This package provides tools for working with CHANGELOG.md files.
|
||||
"""
|
||||
|
||||
from .editor import ChangelogEditor
|
||||
from .parser import ChangelogParser
|
||||
|
||||
__all__ = ['ChangelogEditor', 'ChangelogParser']
|
||||
@@ -0,0 +1,207 @@
|
||||
"""
|
||||
CHANGELOG.md editor for programmatic updates.
|
||||
|
||||
This module provides tools for editing CHANGELOG.md files following
|
||||
the Keep a Changelog format.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, List
|
||||
|
||||
|
||||
class ChangelogEditor:
|
||||
"""Programmatic editor for CHANGELOG.md files."""
|
||||
|
||||
def __init__(self, changelog_path: Optional[Path] = None):
|
||||
"""Initialize changelog editor.
|
||||
|
||||
Args:
|
||||
changelog_path: Path to CHANGELOG.md file
|
||||
"""
|
||||
self.changelog_path = changelog_path or Path.cwd() / 'CHANGELOG.md'
|
||||
|
||||
def create_version_section(self, version: str, date: Optional[str] = None) -> bool:
|
||||
"""Create new version section and move Unreleased content.
|
||||
|
||||
Args:
|
||||
version: Version number (e.g., "0.11.0")
|
||||
date: Release date in YYYY-MM-DD format (defaults to today)
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if date is None:
|
||||
date = datetime.now().strftime("%Y-%m-%d")
|
||||
|
||||
# Validate version format
|
||||
version_clean = version.lstrip('v')
|
||||
|
||||
if not self.changelog_path.exists():
|
||||
print(f"❌ CHANGELOG.md not found at {self.changelog_path}")
|
||||
return False
|
||||
|
||||
try:
|
||||
with open(self.changelog_path) as f:
|
||||
lines = f.readlines()
|
||||
|
||||
# Find Unreleased section
|
||||
unreleased_idx = None
|
||||
for i, line in enumerate(lines):
|
||||
if line.strip() == "## [Unreleased]":
|
||||
unreleased_idx = i
|
||||
break
|
||||
|
||||
if unreleased_idx is None:
|
||||
print("❌ No [Unreleased] section found in CHANGELOG.md")
|
||||
return False
|
||||
|
||||
# Find next version section or end of Unreleased content
|
||||
next_section_idx = None
|
||||
for i in range(unreleased_idx + 1, len(lines)):
|
||||
if lines[i].startswith("## [") and not lines[i].startswith("## [Unreleased]"):
|
||||
next_section_idx = i
|
||||
break
|
||||
|
||||
# Extract Unreleased content (skip the header line and first blank line)
|
||||
if next_section_idx:
|
||||
unreleased_content = lines[unreleased_idx + 1:next_section_idx]
|
||||
else:
|
||||
unreleased_content = lines[unreleased_idx + 1:]
|
||||
|
||||
# Check if there's actual content to move
|
||||
has_content = any(line.strip() and not line.strip().startswith('#')
|
||||
for line in unreleased_content)
|
||||
|
||||
if not has_content:
|
||||
print(f"⚠️ [Unreleased] section is empty. Add changes before creating release section.")
|
||||
print(f"💡 Tip: You can still create the section, but it will be empty.")
|
||||
|
||||
# Create new version section with moved content
|
||||
new_section_lines = [
|
||||
f"## [{version_clean}] - {date}\n",
|
||||
]
|
||||
|
||||
# Add the unreleased content (preserving structure)
|
||||
new_section_lines.extend(unreleased_content)
|
||||
|
||||
# Ensure proper spacing after new section
|
||||
if new_section_lines and not new_section_lines[-1].endswith('\n\n'):
|
||||
if not new_section_lines[-1].endswith('\n'):
|
||||
new_section_lines[-1] += '\n'
|
||||
new_section_lines.append('\n')
|
||||
|
||||
# Build new file content
|
||||
# Keep everything up to and including Unreleased header
|
||||
new_lines = lines[:unreleased_idx + 1]
|
||||
|
||||
# Add blank line after Unreleased header
|
||||
new_lines.append('\n')
|
||||
|
||||
# Add the new version section
|
||||
new_lines.extend(new_section_lines)
|
||||
|
||||
# Add remaining sections (if any)
|
||||
if next_section_idx:
|
||||
new_lines.extend(lines[next_section_idx:])
|
||||
|
||||
# Write back
|
||||
with open(self.changelog_path, 'w') as f:
|
||||
f.writelines(new_lines)
|
||||
|
||||
print(f"✅ Created section [{version_clean}] - {date} in CHANGELOG.md")
|
||||
if has_content:
|
||||
print(f"📝 Moved content from [Unreleased] to [{version_clean}]")
|
||||
print(f"💡 [Unreleased] section is now empty and ready for new changes")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error editing CHANGELOG.md: {e}")
|
||||
return False
|
||||
|
||||
def get_version_content(self, version: str) -> Optional[List[str]]:
|
||||
"""Extract content for a specific version section.
|
||||
|
||||
Args:
|
||||
version: Version number to extract (e.g., "0.10.0")
|
||||
|
||||
Returns:
|
||||
List of lines in the version section, or None if not found
|
||||
"""
|
||||
version_clean = version.lstrip('v')
|
||||
|
||||
if not self.changelog_path.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(self.changelog_path) as f:
|
||||
lines = f.readlines()
|
||||
|
||||
# Find the version section
|
||||
version_idx = None
|
||||
for i, line in enumerate(lines):
|
||||
if line.strip().startswith(f"## [{version_clean}]"):
|
||||
version_idx = i
|
||||
break
|
||||
|
||||
if version_idx is None:
|
||||
return None
|
||||
|
||||
# Find next section
|
||||
next_section_idx = None
|
||||
for i in range(version_idx + 1, len(lines)):
|
||||
if lines[i].startswith("## ["):
|
||||
next_section_idx = i
|
||||
break
|
||||
|
||||
# Extract content
|
||||
if next_section_idx:
|
||||
return lines[version_idx:next_section_idx]
|
||||
else:
|
||||
return lines[version_idx:]
|
||||
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def has_unreleased_content(self) -> bool:
|
||||
"""Check if Unreleased section has any content.
|
||||
|
||||
Returns:
|
||||
True if Unreleased section has content, False otherwise
|
||||
"""
|
||||
if not self.changelog_path.exists():
|
||||
return False
|
||||
|
||||
try:
|
||||
with open(self.changelog_path) as f:
|
||||
lines = f.readlines()
|
||||
|
||||
# Find Unreleased section
|
||||
unreleased_idx = None
|
||||
for i, line in enumerate(lines):
|
||||
if line.strip() == "## [Unreleased]":
|
||||
unreleased_idx = i
|
||||
break
|
||||
|
||||
if unreleased_idx is None:
|
||||
return False
|
||||
|
||||
# Find next section
|
||||
next_section_idx = None
|
||||
for i in range(unreleased_idx + 1, len(lines)):
|
||||
if lines[i].startswith("## ["):
|
||||
next_section_idx = i
|
||||
break
|
||||
|
||||
# Check content
|
||||
if next_section_idx:
|
||||
content = lines[unreleased_idx + 1:next_section_idx]
|
||||
else:
|
||||
content = lines[unreleased_idx + 1:]
|
||||
|
||||
# Check if there's actual content (not just whitespace or section headers)
|
||||
return any(line.strip() and not line.strip().startswith('#')
|
||||
for line in content)
|
||||
|
||||
except Exception:
|
||||
return False
|
||||
@@ -0,0 +1,179 @@
|
||||
"""
|
||||
CHANGELOG.md parser for extracting release notes.
|
||||
|
||||
This module provides tools for parsing CHANGELOG.md files and extracting
|
||||
version-specific content for release notes.
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
class ChangelogParser:
|
||||
"""Parse CHANGELOG.md files and extract release information."""
|
||||
|
||||
def __init__(self, changelog_path: Optional[Path] = None):
|
||||
"""Initialize changelog parser.
|
||||
|
||||
Args:
|
||||
changelog_path: Path to CHANGELOG.md file
|
||||
"""
|
||||
self.changelog_path = changelog_path or Path.cwd() / 'CHANGELOG.md'
|
||||
|
||||
def extract_version_section(self, version: str, format: str = 'markdown') -> str:
|
||||
"""Extract CHANGELOG section for a specific version.
|
||||
|
||||
Args:
|
||||
version: Version to extract (e.g., "0.10.0")
|
||||
format: Output format ('markdown', 'plain', 'html')
|
||||
|
||||
Returns:
|
||||
Formatted content of the version section
|
||||
"""
|
||||
if not self.changelog_path.exists():
|
||||
return f"Error: CHANGELOG.md not found at {self.changelog_path}"
|
||||
|
||||
try:
|
||||
version_clean = version.lstrip('v')
|
||||
|
||||
with open(self.changelog_path) as f:
|
||||
content = f.read()
|
||||
|
||||
# Find the version section using regex
|
||||
# Match: ## [VERSION] - DATE followed by content until next ## [
|
||||
pattern = rf"## \[{re.escape(version_clean)}\].*?\n\n(.*?)(?=\n## \[|\Z)"
|
||||
match = re.search(pattern, content, re.DOTALL)
|
||||
|
||||
if not match:
|
||||
return f"Error: No section found for version {version_clean} in CHANGELOG.md"
|
||||
|
||||
section_content = match.group(1).strip()
|
||||
|
||||
if not section_content:
|
||||
return f"Warning: Section for version {version_clean} exists but is empty"
|
||||
|
||||
# Format based on requested format
|
||||
if format == 'plain':
|
||||
return self._to_plain(section_content)
|
||||
elif format == 'html':
|
||||
return self._to_html(section_content)
|
||||
else:
|
||||
return section_content # markdown (default)
|
||||
|
||||
except Exception as e:
|
||||
return f"Error reading CHANGELOG: {e}"
|
||||
|
||||
def get_latest_version(self) -> Optional[str]:
|
||||
"""Get the latest version number from CHANGELOG.
|
||||
|
||||
Returns:
|
||||
Latest version string or None if not found
|
||||
"""
|
||||
if not self.changelog_path.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(self.changelog_path) as f:
|
||||
content = f.read()
|
||||
|
||||
# Find first version section (skip Unreleased)
|
||||
pattern = r"## \[(\d+\.\d+\.\d+[^\]]*)\]"
|
||||
match = re.search(pattern, content)
|
||||
|
||||
return match.group(1) if match else None
|
||||
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def list_versions(self) -> list:
|
||||
"""List all versions in CHANGELOG.
|
||||
|
||||
Returns:
|
||||
List of version strings
|
||||
"""
|
||||
if not self.changelog_path.exists():
|
||||
return []
|
||||
|
||||
try:
|
||||
with open(self.changelog_path) as f:
|
||||
content = f.read()
|
||||
|
||||
# Find all version sections (excluding Unreleased)
|
||||
pattern = r"## \[(\d+\.\d+\.\d+[^\]]*)\]"
|
||||
matches = re.findall(pattern, content)
|
||||
|
||||
return matches
|
||||
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
def _to_plain(self, markdown_content: str) -> str:
|
||||
"""Convert markdown content to plain text.
|
||||
|
||||
Args:
|
||||
markdown_content: Markdown formatted content
|
||||
|
||||
Returns:
|
||||
Plain text content
|
||||
"""
|
||||
# Remove markdown formatting
|
||||
plain = markdown_content
|
||||
|
||||
# Remove bold/italic
|
||||
plain = re.sub(r'\*\*([^*]+)\*\*', r'\1', plain) # bold
|
||||
plain = re.sub(r'\*([^*]+)\*', r'\1', plain) # italic
|
||||
plain = re.sub(r'__([^_]+)__', r'\1', plain) # bold (underscores)
|
||||
plain = re.sub(r'_([^_]+)_', r'\1', plain) # italic (underscores)
|
||||
|
||||
# Remove links but keep text
|
||||
plain = re.sub(r'\[([^\]]+)\]\([^\)]+\)', r'\1', plain)
|
||||
|
||||
# Remove inline code backticks
|
||||
plain = re.sub(r'`([^`]+)`', r'\1', plain)
|
||||
|
||||
# Convert headers to plain text with spacing
|
||||
plain = re.sub(r'^### (.+)$', r'\n\1:', plain, flags=re.MULTILINE)
|
||||
plain = re.sub(r'^## (.+)$', r'\n\1\n' + '=' * 40, plain, flags=re.MULTILINE)
|
||||
|
||||
return plain.strip()
|
||||
|
||||
def _to_html(self, markdown_content: str) -> str:
|
||||
"""Convert markdown content to HTML.
|
||||
|
||||
Args:
|
||||
markdown_content: Markdown formatted content
|
||||
|
||||
Returns:
|
||||
HTML formatted content
|
||||
"""
|
||||
try:
|
||||
import markdown
|
||||
return markdown.markdown(markdown_content)
|
||||
except ImportError:
|
||||
# Fallback to basic HTML conversion if markdown package not available
|
||||
html = markdown_content
|
||||
|
||||
# Headers
|
||||
html = re.sub(r'^### (.+)$', r'<h3>\1</h3>', html, flags=re.MULTILINE)
|
||||
html = re.sub(r'^## (.+)$', r'<h2>\1</h2>', html, flags=re.MULTILINE)
|
||||
|
||||
# Bold/italic
|
||||
html = re.sub(r'\*\*([^*]+)\*\*', r'<strong>\1</strong>', html)
|
||||
html = re.sub(r'\*([^*]+)\*', r'<em>\1</em>', html)
|
||||
|
||||
# Links
|
||||
html = re.sub(r'\[([^\]]+)\]\(([^\)]+)\)', r'<a href="\2">\1</a>', html)
|
||||
|
||||
# Code
|
||||
html = re.sub(r'`([^`]+)`', r'<code>\1</code>', html)
|
||||
|
||||
# Lists
|
||||
html = re.sub(r'^- (.+)$', r'<li>\1</li>', html, flags=re.MULTILINE)
|
||||
html = re.sub(r'(<li>.*</li>)', r'<ul>\1</ul>', html, flags=re.DOTALL)
|
||||
|
||||
# Paragraphs
|
||||
html = re.sub(r'\n\n', '</p><p>', html)
|
||||
html = f'<p>{html}</p>'
|
||||
|
||||
return html
|
||||
@@ -11,6 +11,9 @@ from typing import Optional
|
||||
|
||||
from ..core.manager import ReleaseManager
|
||||
from ..utils.version import VersionManager
|
||||
from ..changelog.editor import ChangelogEditor
|
||||
from ..changelog.parser import ChangelogParser
|
||||
from ..summary.generator import SummaryGenerator
|
||||
|
||||
|
||||
@click.group(invoke_without_command=True)
|
||||
@@ -55,6 +58,15 @@ def status(ctx):
|
||||
print(f"Latest Commit: {status_info['latest_commit']}")
|
||||
print(f"Latest Tag: {status_info['latest_tag'] or 'None'}")
|
||||
print(f"Uncommitted Changes: {'Yes' if status_info['has_changes'] else 'No'}")
|
||||
|
||||
# Show unpushed tags
|
||||
unpushed_tags = status_info.get('unpushed_tags', [])
|
||||
if unpushed_tags:
|
||||
print(f"\n⚠️ Unpushed Tags: {len(unpushed_tags)} tag(s) not pushed to origin")
|
||||
for tag in unpushed_tags:
|
||||
print(f" - {tag}")
|
||||
print(f"\n💡 Push tags with: git push origin {' '.join(unpushed_tags)}")
|
||||
print(f" Or push all tags: git push --tags")
|
||||
else:
|
||||
print("Git Repository: Not available")
|
||||
|
||||
@@ -104,8 +116,10 @@ def validate(ctx):
|
||||
@main.command()
|
||||
@click.option('--version', required=True, help='Version to tag (e.g., 0.8.0)')
|
||||
@click.option('--message', help='Tag message')
|
||||
@click.option('--push/--no-push', default=True,
|
||||
help='Automatically push tag to origin (default: --push)')
|
||||
@click.pass_context
|
||||
def tag(ctx, version: str, message: Optional[str]):
|
||||
def tag(ctx, version: str, message: Optional[str], push: bool):
|
||||
"""Create git tag for version."""
|
||||
manager = ReleaseManager(
|
||||
project_root=ctx.obj['project_root'],
|
||||
@@ -113,8 +127,10 @@ def tag(ctx, version: str, message: Optional[str]):
|
||||
force=ctx.obj['force']
|
||||
)
|
||||
|
||||
if manager.create_tag(version, message):
|
||||
if manager.create_tag(version, message, push=push):
|
||||
print(f"✅ Successfully created tag for version {version}")
|
||||
if not push:
|
||||
print(f"💡 Push tag with: git push origin v{version}")
|
||||
else:
|
||||
print(f"❌ Failed to create tag for version {version}")
|
||||
sys.exit(1)
|
||||
@@ -248,5 +264,153 @@ def version_info(ctx, suggest: bool):
|
||||
print(f" {key.replace('_', ' ').title()}: {value}")
|
||||
|
||||
|
||||
@main.command('check-consistency')
|
||||
@click.option('--version', required=True, help='Version to check (e.g., 0.10.0)')
|
||||
@click.pass_context
|
||||
def check_consistency(ctx, version: str):
|
||||
"""Check consistency between CHANGELOG version and git tags."""
|
||||
manager = ReleaseManager(
|
||||
project_root=ctx.obj['project_root'],
|
||||
dry_run=ctx.obj['dry_run'],
|
||||
force=ctx.obj['force']
|
||||
)
|
||||
|
||||
is_consistent, issues = manager.check_version_consistency(version)
|
||||
|
||||
if is_consistent:
|
||||
print(f"✅ Version {version} is consistent:")
|
||||
print(f" - CHANGELOG has section for {version}")
|
||||
print(f" - Git tag v{version} exists")
|
||||
print(f" - [Unreleased] section present")
|
||||
else:
|
||||
print(f"❌ Version {version} has consistency issues:")
|
||||
for issue in issues:
|
||||
print(f" - {issue}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
@main.command('prepare')
|
||||
@click.argument('version')
|
||||
@click.option('--date', default=None, help='Release date (YYYY-MM-DD, defaults to today)')
|
||||
@click.pass_context
|
||||
def prepare(ctx, version: str, date: Optional[str]):
|
||||
"""Prepare CHANGELOG for new version release.
|
||||
|
||||
Creates a new version section in CHANGELOG.md and moves all content
|
||||
from the [Unreleased] section to the new version section.
|
||||
"""
|
||||
project_root = ctx.obj['project_root'] or Path.cwd()
|
||||
changelog_path = project_root / 'CHANGELOG.md'
|
||||
|
||||
editor = ChangelogEditor(changelog_path)
|
||||
|
||||
# Create version section
|
||||
if editor.create_version_section(version, date):
|
||||
# Validate result
|
||||
manager = ReleaseManager(
|
||||
project_root=ctx.obj['project_root'],
|
||||
dry_run=ctx.obj['dry_run'],
|
||||
force=ctx.obj['force']
|
||||
)
|
||||
|
||||
# Check if CHANGELOG is valid after edit
|
||||
is_valid, issues = manager.validate_release_state()
|
||||
|
||||
if is_valid:
|
||||
print("\n✅ CHANGELOG validation passed")
|
||||
else:
|
||||
print("\n⚠️ CHANGELOG validation issues after edit:")
|
||||
for issue in issues:
|
||||
if 'CHANGELOG' in issue:
|
||||
print(f" - {issue}")
|
||||
else:
|
||||
print(f"❌ Failed to prepare CHANGELOG for version {version}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
@main.command('summary')
|
||||
@click.argument('version')
|
||||
@click.option('--output', '-o', default=None, type=click.Path(path_type=Path),
|
||||
help='Output file path (defaults to RELEASE_SUMMARY_vX.Y.Z.md)')
|
||||
@click.pass_context
|
||||
def summary(ctx, version: str, output: Optional[Path]):
|
||||
"""Generate release summary document.
|
||||
|
||||
Extracts CHANGELOG content, git statistics, build artifacts, and
|
||||
validation results to create a comprehensive release summary.
|
||||
"""
|
||||
project_root = ctx.obj['project_root'] or Path.cwd()
|
||||
|
||||
# Default output path
|
||||
if output is None:
|
||||
version_clean = version.lstrip('v')
|
||||
output = project_root / f"RELEASE_SUMMARY_v{version_clean}.md"
|
||||
elif not output.is_absolute():
|
||||
output = project_root / output
|
||||
|
||||
generator = SummaryGenerator(project_root)
|
||||
|
||||
try:
|
||||
content = generator.generate(version, output_path=output)
|
||||
print(f"\n✅ Release summary generated successfully")
|
||||
print(f"📄 Summary saved to: {output}")
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to generate release summary: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
@main.command('notes')
|
||||
@click.argument('version', required=False)
|
||||
@click.option('--format', 'output_format', type=click.Choice(['markdown', 'plain', 'html']),
|
||||
default='markdown', help='Output format (default: markdown)')
|
||||
@click.option('--output', '-o', default=None, type=click.Path(path_type=Path),
|
||||
help='Save to file instead of stdout')
|
||||
@click.pass_context
|
||||
def notes(ctx, version: Optional[str], output_format: str, output: Optional[Path]):
|
||||
"""Extract release notes from CHANGELOG.md.
|
||||
|
||||
Extracts the CHANGELOG section for a specific version and outputs it
|
||||
in various formats. Useful for creating GitHub/Gitea release notes.
|
||||
|
||||
If no version is specified, uses the latest version from CHANGELOG.
|
||||
|
||||
Examples:
|
||||
release notes 0.10.0 # Extract v0.10.0 notes (markdown)
|
||||
release notes # Extract latest version notes
|
||||
release notes 0.10.0 --format plain # Plain text format
|
||||
release notes 0.10.0 -o notes.md # Save to file
|
||||
release notes 0.10.0 | gh release create v0.10.0 -F - # Pipe to gh
|
||||
"""
|
||||
project_root = ctx.obj['project_root'] or Path.cwd()
|
||||
changelog_path = project_root / 'CHANGELOG.md'
|
||||
|
||||
parser = ChangelogParser(changelog_path)
|
||||
|
||||
# Get version (use latest if not specified)
|
||||
if version is None:
|
||||
version = parser.get_latest_version()
|
||||
if version is None:
|
||||
print("❌ Could not determine version from CHANGELOG.md")
|
||||
sys.exit(1)
|
||||
print(f"Using latest version: {version}", file=sys.stderr)
|
||||
|
||||
# Extract content
|
||||
content = parser.extract_version_section(version, format=output_format)
|
||||
|
||||
# Check for errors
|
||||
if content.startswith("Error:") or content.startswith("Warning:"):
|
||||
print(content)
|
||||
sys.exit(1 if content.startswith("Error:") else 0)
|
||||
|
||||
# Output
|
||||
if output:
|
||||
if not output.is_absolute():
|
||||
output = project_root / output
|
||||
output.write_text(content)
|
||||
print(f"✅ Release notes written to {output}", file=sys.stderr)
|
||||
else:
|
||||
print(content)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -75,12 +75,13 @@ class ReleaseManager:
|
||||
"""
|
||||
return self.validator.validate_release_state(force=self.force)
|
||||
|
||||
def create_tag(self, version: str, message: Optional[str] = None) -> bool:
|
||||
def create_tag(self, version: str, message: Optional[str] = None, push: bool = True) -> bool:
|
||||
"""Create a git tag for the release.
|
||||
|
||||
Args:
|
||||
version: Version to tag (e.g., "1.0.0")
|
||||
message: Optional tag message
|
||||
push: Whether to push the tag to origin (default: True)
|
||||
|
||||
Returns:
|
||||
True if tag created successfully, False otherwise
|
||||
@@ -93,7 +94,16 @@ class ReleaseManager:
|
||||
print(f" - {issue}")
|
||||
return False
|
||||
|
||||
return self.git_manager.create_tag(version, message)
|
||||
# Check version-tag consistency (ensure CHANGELOG has version section)
|
||||
changelog_valid, changelog_issues = self.validator.validate_changelog_version(version)
|
||||
if not changelog_valid and not self.force:
|
||||
print(f"❌ Cannot create tag for version {version}:")
|
||||
for issue in changelog_issues:
|
||||
print(f" - {issue}")
|
||||
print("\n💡 Tip: Add a section for version {version} to CHANGELOG.md before tagging")
|
||||
return False
|
||||
|
||||
return self.git_manager.create_tag(version, message, push=push)
|
||||
|
||||
def build_packages(self) -> bool:
|
||||
"""Build release packages.
|
||||
@@ -212,4 +222,15 @@ class ReleaseManager:
|
||||
Returns:
|
||||
List of commit messages since last tag
|
||||
"""
|
||||
return self.git_manager.get_commits_since_tag()
|
||||
return self.git_manager.get_commits_since_tag()
|
||||
|
||||
def check_version_consistency(self, version: str) -> Tuple[bool, List[str]]:
|
||||
"""Check consistency between CHANGELOG version and git tags.
|
||||
|
||||
Args:
|
||||
version: Version to check (e.g., "0.10.0")
|
||||
|
||||
Returns:
|
||||
Tuple of (is_consistent, list_of_issues)
|
||||
"""
|
||||
return self.validator.check_version_tag_consistency(version)
|
||||
@@ -48,22 +48,27 @@ class GitManager:
|
||||
except subprocess.CalledProcessError:
|
||||
latest_tag = None
|
||||
|
||||
# Get unpushed tags
|
||||
unpushed_tags = self.get_unpushed_tags()
|
||||
|
||||
return {
|
||||
'is_repo': True,
|
||||
'branch': current_branch,
|
||||
'has_changes': has_changes,
|
||||
'latest_commit': latest_commit,
|
||||
'latest_tag': latest_tag
|
||||
'latest_tag': latest_tag,
|
||||
'unpushed_tags': unpushed_tags
|
||||
}
|
||||
except subprocess.CalledProcessError:
|
||||
return {'is_repo': False}
|
||||
|
||||
def create_tag(self, version: str, message: Optional[str] = None) -> bool:
|
||||
"""Create and push git tag.
|
||||
def create_tag(self, version: str, message: Optional[str] = None, push: bool = True) -> bool:
|
||||
"""Create and optionally push git tag.
|
||||
|
||||
Args:
|
||||
version: Version to tag (e.g., "1.0.0")
|
||||
message: Optional tag message
|
||||
push: Whether to push the tag to origin (default: True)
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
@@ -81,16 +86,19 @@ class GitManager:
|
||||
self._run_command(['git', 'tag', '-a', tag_name, '-m', tag_message])
|
||||
print(f"✅ Tag {tag_name} created")
|
||||
|
||||
# Push tag to origin
|
||||
try:
|
||||
print(f"📤 Pushing tag to origin...")
|
||||
self._run_command(['git', 'push', 'origin', tag_name])
|
||||
print(f"✅ Tag pushed to origin")
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"⚠️ Could not push tag to origin: {e}")
|
||||
print(f"You can push it manually with: git push origin {tag_name}")
|
||||
return True # Tag created successfully, push can be done manually
|
||||
# Push tag to origin if requested
|
||||
if push:
|
||||
try:
|
||||
print(f"📤 Pushing tag to origin...")
|
||||
self._run_command(['git', 'push', 'origin', tag_name])
|
||||
print(f"✅ Tag pushed to origin")
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"⚠️ Could not push tag to origin: {e}")
|
||||
print(f"You can push it manually with: git push origin {tag_name}")
|
||||
return True # Tag created successfully, push can be done manually
|
||||
else:
|
||||
return True # Tag created successfully, user chose not to push
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ Failed to create tag: {e}")
|
||||
@@ -178,6 +186,47 @@ class GitManager:
|
||||
except subprocess.CalledProcessError:
|
||||
return None
|
||||
|
||||
def get_unpushed_tags(self, remote: str = 'origin') -> List[str]:
|
||||
"""Get list of tags that exist locally but not on remote.
|
||||
|
||||
Args:
|
||||
remote: Remote name to compare against (default: 'origin')
|
||||
|
||||
Returns:
|
||||
List of unpushed tag names
|
||||
"""
|
||||
try:
|
||||
# Get local tags
|
||||
local_result = self._run_command(['git', 'tag', '-l'])
|
||||
local_tags = set(tag.strip() for tag in local_result.stdout.strip().split('\n') if tag.strip())
|
||||
|
||||
# Get remote tags
|
||||
try:
|
||||
remote_result = self._run_command(['git', 'ls-remote', '--tags', remote])
|
||||
remote_lines = remote_result.stdout.strip().split('\n')
|
||||
|
||||
# Parse remote tags (format: "hash refs/tags/tagname")
|
||||
remote_tags = set()
|
||||
for line in remote_lines:
|
||||
if not line:
|
||||
continue
|
||||
parts = line.split('refs/tags/')
|
||||
if len(parts) > 1:
|
||||
# Remove ^{} suffix for annotated tags
|
||||
tag_name = parts[1].replace('^{}', '')
|
||||
remote_tags.add(tag_name)
|
||||
|
||||
# Find tags that are local but not remote
|
||||
unpushed = sorted(local_tags - remote_tags)
|
||||
return unpushed
|
||||
|
||||
except subprocess.CalledProcessError:
|
||||
# Remote not available, assume all tags are unpushed
|
||||
return sorted(local_tags)
|
||||
|
||||
except subprocess.CalledProcessError:
|
||||
return []
|
||||
|
||||
def _run_command(self, cmd: List[str]) -> subprocess.CompletedProcess:
|
||||
"""Run a git command.
|
||||
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
"""
|
||||
Release summary generation tools.
|
||||
|
||||
This package provides tools for generating release summary documents.
|
||||
"""
|
||||
|
||||
from .generator import SummaryGenerator
|
||||
|
||||
__all__ = ['SummaryGenerator']
|
||||
@@ -0,0 +1,305 @@
|
||||
"""
|
||||
Release summary generator.
|
||||
|
||||
This module generates comprehensive release summary documents from
|
||||
CHANGELOG content and git metadata.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Dict, Any, List
|
||||
import re
|
||||
|
||||
|
||||
class SummaryGenerator:
|
||||
"""Generate release summary documents."""
|
||||
|
||||
def __init__(self, project_root: Optional[Path] = None):
|
||||
"""Initialize summary generator.
|
||||
|
||||
Args:
|
||||
project_root: Root directory of the project
|
||||
"""
|
||||
self.project_root = project_root or Path.cwd()
|
||||
self.changelog_path = self.project_root / 'CHANGELOG.md'
|
||||
self.dist_dir = self.project_root / 'dist'
|
||||
|
||||
def generate(self, version: str, output_path: Optional[Path] = None) -> str:
|
||||
"""Generate release summary document.
|
||||
|
||||
Args:
|
||||
version: Version to generate summary for (e.g., "0.10.0")
|
||||
output_path: Optional path to write summary to
|
||||
|
||||
Returns:
|
||||
Generated summary content
|
||||
"""
|
||||
version_clean = version.lstrip('v')
|
||||
tag_name = f"v{version_clean}"
|
||||
|
||||
# Get components
|
||||
changelog_section = self.extract_changelog_section(version_clean)
|
||||
git_stats = self.get_git_statistics(tag_name)
|
||||
build_artifacts = self.list_build_artifacts()
|
||||
validation_results = self.get_validation_results()
|
||||
|
||||
# Determine project name
|
||||
project_name = self._get_project_name()
|
||||
|
||||
# Build summary
|
||||
summary = f"""# {project_name} {version_clean} Release Summary
|
||||
|
||||
**Release Date**: {git_stats.get('release_date', 'Unknown')}
|
||||
**Git Tag**: {tag_name}
|
||||
**Commit**: {git_stats.get('commit_hash', 'Unknown')}
|
||||
|
||||
---
|
||||
|
||||
## Changes
|
||||
|
||||
{changelog_section}
|
||||
|
||||
---
|
||||
|
||||
## Git Statistics
|
||||
|
||||
- **Commits**: {git_stats.get('commit_count', 0)} commit(s) since last release
|
||||
- **Files Changed**: {git_stats.get('files_changed', 0)} file(s)
|
||||
- **Insertions**: +{git_stats.get('insertions', 0)} lines
|
||||
- **Deletions**: -{git_stats.get('deletions', 0)} lines
|
||||
- **Contributors**: {git_stats.get('contributors', 'Unknown')}
|
||||
|
||||
---
|
||||
|
||||
## Build Artifacts
|
||||
|
||||
{build_artifacts}
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
{validation_results}
|
||||
|
||||
---
|
||||
|
||||
**Generated**: {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}
|
||||
"""
|
||||
|
||||
if output_path:
|
||||
output_path.write_text(summary)
|
||||
print(f"✅ Release summary written to {output_path}")
|
||||
|
||||
return summary
|
||||
|
||||
def extract_changelog_section(self, version: str) -> str:
|
||||
"""Extract CHANGELOG section for a specific version.
|
||||
|
||||
Args:
|
||||
version: Version to extract (e.g., "0.10.0")
|
||||
|
||||
Returns:
|
||||
Markdown content of the version section
|
||||
"""
|
||||
if not self.changelog_path.exists():
|
||||
return "⚠️ CHANGELOG.md not found"
|
||||
|
||||
try:
|
||||
with open(self.changelog_path) as f:
|
||||
content = f.read()
|
||||
|
||||
# Find the version section
|
||||
pattern = rf"## \[{re.escape(version)}\].*?\n\n(.*?)(?=\n## \[|\Z)"
|
||||
match = re.search(pattern, content, re.DOTALL)
|
||||
|
||||
if match:
|
||||
section_content = match.group(1).strip()
|
||||
return section_content if section_content else "No changes documented"
|
||||
else:
|
||||
return f"⚠️ No section found for version {version} in CHANGELOG.md"
|
||||
|
||||
except Exception as e:
|
||||
return f"❌ Error reading CHANGELOG: {e}"
|
||||
|
||||
def get_git_statistics(self, tag: str) -> Dict[str, Any]:
|
||||
"""Get git statistics for a release tag.
|
||||
|
||||
Args:
|
||||
tag: Git tag name (e.g., "v0.10.0")
|
||||
|
||||
Returns:
|
||||
Dictionary with git statistics
|
||||
"""
|
||||
stats = {}
|
||||
|
||||
try:
|
||||
# Get tag date
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['git', 'log', '-1', '--format=%ci', tag],
|
||||
capture_output=True, text=True, check=True, cwd=self.project_root
|
||||
)
|
||||
date_str = result.stdout.strip()
|
||||
# Parse to get just the date
|
||||
stats['release_date'] = date_str.split()[0] if date_str else 'Unknown'
|
||||
except subprocess.CalledProcessError:
|
||||
stats['release_date'] = 'Unknown'
|
||||
|
||||
# Get commit hash
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['git', 'rev-parse', tag],
|
||||
capture_output=True, text=True, check=True, cwd=self.project_root
|
||||
)
|
||||
stats['commit_hash'] = result.stdout.strip()[:8]
|
||||
except subprocess.CalledProcessError:
|
||||
stats['commit_hash'] = 'Unknown'
|
||||
|
||||
# Find previous tag
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['git', 'describe', '--tags', '--abbrev=0', f'{tag}^'],
|
||||
capture_output=True, text=True, check=True, cwd=self.project_root
|
||||
)
|
||||
previous_tag = result.stdout.strip()
|
||||
except subprocess.CalledProcessError:
|
||||
# No previous tag, use initial commit
|
||||
previous_tag = None
|
||||
|
||||
# Get commit count
|
||||
if previous_tag:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['git', 'rev-list', '--count', f'{previous_tag}..{tag}'],
|
||||
capture_output=True, text=True, check=True, cwd=self.project_root
|
||||
)
|
||||
stats['commit_count'] = int(result.stdout.strip())
|
||||
except subprocess.CalledProcessError:
|
||||
stats['commit_count'] = 0
|
||||
else:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['git', 'rev-list', '--count', tag],
|
||||
capture_output=True, text=True, check=True, cwd=self.project_root
|
||||
)
|
||||
stats['commit_count'] = int(result.stdout.strip())
|
||||
except subprocess.CalledProcessError:
|
||||
stats['commit_count'] = 0
|
||||
|
||||
# Get file changes, insertions, deletions
|
||||
if previous_tag:
|
||||
diff_range = f'{previous_tag}..{tag}'
|
||||
else:
|
||||
diff_range = tag
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['git', 'diff', '--shortstat', diff_range],
|
||||
capture_output=True, text=True, check=True, cwd=self.project_root
|
||||
)
|
||||
shortstat = result.stdout.strip()
|
||||
|
||||
# Parse shortstat: "X files changed, Y insertions(+), Z deletions(-)"
|
||||
files_match = re.search(r'(\d+) files? changed', shortstat)
|
||||
insert_match = re.search(r'(\d+) insertions?', shortstat)
|
||||
delete_match = re.search(r'(\d+) deletions?', shortstat)
|
||||
|
||||
stats['files_changed'] = int(files_match.group(1)) if files_match else 0
|
||||
stats['insertions'] = int(insert_match.group(1)) if insert_match else 0
|
||||
stats['deletions'] = int(delete_match.group(1)) if delete_match else 0
|
||||
except subprocess.CalledProcessError:
|
||||
stats['files_changed'] = 0
|
||||
stats['insertions'] = 0
|
||||
stats['deletions'] = 0
|
||||
|
||||
# Get contributors
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['git', 'log', '--format=%an', f'{previous_tag}..{tag}' if previous_tag else tag],
|
||||
capture_output=True, text=True, check=True, cwd=self.project_root
|
||||
)
|
||||
contributors = list(set(result.stdout.strip().split('\n')))
|
||||
stats['contributors'] = ', '.join(contributors) if contributors and contributors[0] else 'Unknown'
|
||||
except subprocess.CalledProcessError:
|
||||
stats['contributors'] = 'Unknown'
|
||||
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error getting git statistics: {e}")
|
||||
|
||||
return stats
|
||||
|
||||
def list_build_artifacts(self) -> str:
|
||||
"""List build artifacts in dist/ directory.
|
||||
|
||||
Returns:
|
||||
Formatted markdown list of build artifacts
|
||||
"""
|
||||
if not self.dist_dir.exists():
|
||||
return "No build artifacts found (dist/ directory does not exist)"
|
||||
|
||||
artifacts = list(self.dist_dir.glob('*'))
|
||||
if not artifacts:
|
||||
return "No build artifacts found in dist/"
|
||||
|
||||
lines = []
|
||||
for artifact in sorted(artifacts):
|
||||
if artifact.is_file():
|
||||
size = artifact.stat().st_size
|
||||
size_kb = size / 1024
|
||||
size_mb = size / (1024 * 1024)
|
||||
|
||||
if size_mb >= 1:
|
||||
size_str = f"{size_mb:.2f} MB"
|
||||
else:
|
||||
size_str = f"{size_kb:.2f} KB"
|
||||
|
||||
lines.append(f"- **{artifact.name}** ({size_str})")
|
||||
|
||||
return '\n'.join(lines) if lines else "No build artifacts found"
|
||||
|
||||
def get_validation_results(self) -> str:
|
||||
"""Get validation results summary.
|
||||
|
||||
Returns:
|
||||
Formatted validation results
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from ..utils.validation import ReleaseValidator
|
||||
|
||||
validator = ReleaseValidator(self.project_root)
|
||||
is_valid, issues = validator.validate_release_state(force=True) # Force to get all issues
|
||||
|
||||
if is_valid:
|
||||
return "✅ All validation checks passed"
|
||||
else:
|
||||
lines = ["Validation Issues:"]
|
||||
for issue in issues:
|
||||
lines.append(f"- {issue}")
|
||||
return '\n'.join(lines)
|
||||
|
||||
def _get_project_name(self) -> str:
|
||||
"""Get project name from pyproject.toml.
|
||||
|
||||
Returns:
|
||||
Project name or default
|
||||
"""
|
||||
pyproject_path = self.project_root / 'pyproject.toml'
|
||||
|
||||
if not pyproject_path.exists():
|
||||
return "Project"
|
||||
|
||||
try:
|
||||
import tomllib
|
||||
except ImportError:
|
||||
try:
|
||||
import tomli as tomllib
|
||||
except ImportError:
|
||||
return "Project"
|
||||
|
||||
try:
|
||||
with open(pyproject_path, 'rb') as f:
|
||||
config = tomllib.load(f)
|
||||
return config.get('project', {}).get('name', 'Project').title()
|
||||
except Exception:
|
||||
return "Project"
|
||||
@@ -4,6 +4,7 @@ Release validation utilities.
|
||||
This module provides validation functions for release readiness.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import List, Tuple, Optional
|
||||
|
||||
@@ -48,6 +49,10 @@ class ReleaseValidator:
|
||||
config_issues = self._validate_configuration()
|
||||
issues.extend(config_issues)
|
||||
|
||||
# CHANGELOG validation
|
||||
changelog_issues = self._validate_changelog()
|
||||
issues.extend(changelog_issues)
|
||||
|
||||
return len(issues) == 0, issues
|
||||
|
||||
def _validate_git_state(self) -> List[str]:
|
||||
@@ -186,6 +191,117 @@ class ReleaseValidator:
|
||||
|
||||
return len(issues) == 0, issues
|
||||
|
||||
def _validate_changelog(self) -> List[str]:
|
||||
"""Validate CHANGELOG.md using changelog schema.
|
||||
|
||||
Returns:
|
||||
List of CHANGELOG-related issues
|
||||
"""
|
||||
issues = []
|
||||
changelog_path = self.project_root / 'CHANGELOG.md'
|
||||
|
||||
# Check if CHANGELOG exists
|
||||
if not changelog_path.exists():
|
||||
issues.append("Missing CHANGELOG.md file")
|
||||
return issues
|
||||
|
||||
# Check if changelog schema exists
|
||||
schema_path = self.project_root / 'markitect' / 'schemas' / 'changelog-schema-v1.0.md'
|
||||
if not schema_path.exists():
|
||||
# Schema doesn't exist, skip validation
|
||||
return issues
|
||||
|
||||
# Validate CHANGELOG with schema using markitect validate command
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[
|
||||
'markitect', 'validate', str(changelog_path),
|
||||
'--schema', str(schema_path),
|
||||
'--semantic'
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=self.project_root
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
issues.append("CHANGELOG.md validation failed against schema")
|
||||
# Parse output for specific errors
|
||||
if 'Unreleased section' in result.stdout:
|
||||
issues.append(" - Missing [Unreleased] section in CHANGELOG")
|
||||
if 'version format' in result.stdout.lower():
|
||||
issues.append(" - Invalid version format in CHANGELOG")
|
||||
|
||||
except FileNotFoundError:
|
||||
# markitect command not available
|
||||
issues.append("Cannot validate CHANGELOG (markitect command not found)")
|
||||
except Exception as e:
|
||||
issues.append(f"Error validating CHANGELOG: {e}")
|
||||
|
||||
return issues
|
||||
|
||||
def validate_changelog_version(self, version: str) -> Tuple[bool, List[str]]:
|
||||
"""Validate that CHANGELOG has section for specified version.
|
||||
|
||||
Args:
|
||||
version: Version to check (e.g., "0.10.0")
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, list_of_issues)
|
||||
"""
|
||||
issues = []
|
||||
changelog_path = self.project_root / 'CHANGELOG.md'
|
||||
|
||||
if not changelog_path.exists():
|
||||
issues.append("CHANGELOG.md not found")
|
||||
return False, issues
|
||||
|
||||
try:
|
||||
content = changelog_path.read_text()
|
||||
|
||||
# Check for version section
|
||||
version_header = f"## [{version}]"
|
||||
if version_header not in content:
|
||||
issues.append(f"CHANGELOG missing section for version {version}")
|
||||
|
||||
# Check for Unreleased section
|
||||
if "## [Unreleased]" not in content:
|
||||
issues.append("CHANGELOG missing [Unreleased] section")
|
||||
|
||||
# Check if version section has a date
|
||||
import re
|
||||
date_pattern = rf"## \[{re.escape(version)}\] - \d{{4}}-\d{{2}}-\d{{2}}"
|
||||
if not re.search(date_pattern, content):
|
||||
issues.append(f"Version {version} section missing date or has invalid date format")
|
||||
|
||||
except Exception as e:
|
||||
issues.append(f"Error reading CHANGELOG: {e}")
|
||||
|
||||
return len(issues) == 0, issues
|
||||
|
||||
def check_version_tag_consistency(self, version: str) -> Tuple[bool, List[str]]:
|
||||
"""Check consistency between CHANGELOG version and git tags.
|
||||
|
||||
Args:
|
||||
version: Version to check (e.g., "0.10.0")
|
||||
|
||||
Returns:
|
||||
Tuple of (is_consistent, list_of_issues)
|
||||
"""
|
||||
issues = []
|
||||
|
||||
# Check CHANGELOG has the version
|
||||
changelog_valid, changelog_issues = self.validate_changelog_version(version)
|
||||
if not changelog_valid:
|
||||
issues.extend(changelog_issues)
|
||||
|
||||
# Check git tag exists
|
||||
tag_name = version if version.startswith('v') else f'v{version}'
|
||||
if not self.git_manager.tag_exists(tag_name):
|
||||
issues.append(f"Git tag {tag_name} doesn't exist for version in CHANGELOG")
|
||||
|
||||
return len(issues) == 0, issues
|
||||
|
||||
def get_validation_summary(self) -> dict:
|
||||
"""Get a comprehensive validation summary.
|
||||
|
||||
@@ -224,6 +340,10 @@ class ReleaseValidator:
|
||||
if any('authentication' in issue.lower() for issue in issues):
|
||||
recommendations.append("Set up authentication tokens for package publishing")
|
||||
|
||||
if any('CHANGELOG' in issue for issue in issues):
|
||||
recommendations.append("Fix CHANGELOG.md format and ensure [Unreleased] section exists")
|
||||
recommendations.append("Validate with: markitect validate CHANGELOG.md --schema changelog-schema-v1.0.md --semantic")
|
||||
|
||||
if not issues:
|
||||
recommendations.append("Repository is ready for release!")
|
||||
|
||||
|
||||
@@ -188,4 +188,111 @@ class VersionManager:
|
||||
version.Version(version_string)
|
||||
return True
|
||||
except version.InvalidVersion:
|
||||
return False
|
||||
return False
|
||||
|
||||
def get_version_info(self, project_root: Optional[Path] = None) -> Dict[str, Any]:
|
||||
"""Get comprehensive version information for a project.
|
||||
|
||||
Args:
|
||||
project_root: Root directory of project. If None, uses current directory.
|
||||
|
||||
Returns:
|
||||
Dictionary with version information
|
||||
"""
|
||||
if project_root:
|
||||
original_root = self.project_root
|
||||
self.project_root = project_root
|
||||
|
||||
try:
|
||||
current_version = self.get_current_version()
|
||||
|
||||
# Try to get git information
|
||||
git_info = self._get_git_info()
|
||||
|
||||
# Parse version components
|
||||
version_parts = self.parse_version(current_version) if current_version != "unknown" else {}
|
||||
|
||||
return {
|
||||
'full_version': current_version,
|
||||
'short_version': current_version.split('.dev')[0] if '.dev' in current_version else current_version,
|
||||
'version_components': version_parts,
|
||||
'is_dev': self.is_development_version(current_version),
|
||||
'git_commit': git_info.get('commit'),
|
||||
'git_branch': git_info.get('branch'),
|
||||
'is_git_repo': git_info.get('is_repo', False)
|
||||
}
|
||||
finally:
|
||||
if project_root:
|
||||
self.project_root = original_root
|
||||
|
||||
def get_release_info(self, project_root: Optional[Path] = None) -> Dict[str, Any]:
|
||||
"""Get release information for a project.
|
||||
|
||||
Args:
|
||||
project_root: Root directory of project. If None, uses current directory.
|
||||
|
||||
Returns:
|
||||
Dictionary with release information
|
||||
"""
|
||||
from datetime import datetime
|
||||
|
||||
version_info = self.get_version_info(project_root)
|
||||
|
||||
return {
|
||||
'name': 'MarkiTect',
|
||||
'version': version_info['full_version'],
|
||||
'short_version': version_info['short_version'],
|
||||
'is_development': version_info['is_dev'],
|
||||
'git_branch': version_info.get('git_branch', 'unknown'),
|
||||
'git_commit': version_info.get('git_commit', 'unknown'),
|
||||
'build_date': datetime.now().isoformat(),
|
||||
'python_version': f"{__import__('sys').version_info.major}.{__import__('sys').version_info.minor}.{__import__('sys').version_info.micro}"
|
||||
}
|
||||
|
||||
def _get_git_info(self) -> Dict[str, Any]:
|
||||
"""Get git repository information.
|
||||
|
||||
Returns:
|
||||
Dictionary with git information
|
||||
"""
|
||||
git_info = {'is_repo': False}
|
||||
|
||||
try:
|
||||
# Check if in git repo
|
||||
subprocess.run(['git', 'rev-parse', '--git-dir'],
|
||||
cwd=self.project_root, check=True, capture_output=True)
|
||||
git_info['is_repo'] = True
|
||||
|
||||
# Get branch
|
||||
try:
|
||||
result = subprocess.run(['git', 'rev-parse', '--abbrev-ref', 'HEAD'],
|
||||
cwd=self.project_root, capture_output=True, text=True, check=True)
|
||||
git_info['branch'] = result.stdout.strip()
|
||||
except subprocess.CalledProcessError:
|
||||
git_info['branch'] = 'unknown'
|
||||
|
||||
# Get commit
|
||||
try:
|
||||
result = subprocess.run(['git', 'rev-parse', 'HEAD'],
|
||||
cwd=self.project_root, capture_output=True, text=True, check=True)
|
||||
git_info['commit'] = result.stdout.strip()
|
||||
except subprocess.CalledProcessError:
|
||||
git_info['commit'] = 'unknown'
|
||||
|
||||
except subprocess.CalledProcessError:
|
||||
pass # Not a git repo
|
||||
|
||||
return git_info
|
||||
|
||||
|
||||
# Convenience functions for backward compatibility and easy import
|
||||
def get_version_info(project_root: Optional[Path] = None) -> Dict[str, Any]:
|
||||
"""Get version information using default VersionManager."""
|
||||
manager = VersionManager(project_root)
|
||||
return manager.get_version_info()
|
||||
|
||||
|
||||
def get_release_info(project_root: Optional[Path] = None) -> Dict[str, Any]:
|
||||
"""Get release information using default VersionManager."""
|
||||
manager = VersionManager(project_root)
|
||||
return manager.get_release_info()
|
||||
1
capabilities/testdrive-jsui
Submodule
1
capabilities/testdrive-jsui
Submodule
Submodule capabilities/testdrive-jsui added at b8f13b4ae5
7
cost_notes/25114-botched_refactoring_recovery.md
Normal file
7
cost_notes/25114-botched_refactoring_recovery.md
Normal file
@@ -0,0 +1,7 @@
|
||||
Total cost: $18.34
|
||||
Total duration (API): 49m 10.0s
|
||||
Total duration (wall): 10h 38m 0.7s
|
||||
Total code changes: 6105 lines added, 186 lines removed
|
||||
Usage by model:
|
||||
claude-3-5-haiku: 58.3k input, 7.3k output, 0 cache read, 0 cache write ($0.0760)
|
||||
claude-sonnet: 9.1k input, 141.0k output, 31.7m cache read, 1.8m cache write ($18.27)
|
||||
212
demo_plugin_integration.py
Normal file
212
demo_plugin_integration.py
Normal file
@@ -0,0 +1,212 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Demo script showing TestDrive JSUI plugin integration with Markitect
|
||||
|
||||
This script demonstrates:
|
||||
1. Plugin discovery and registration
|
||||
2. Asset management and deployment
|
||||
3. Standalone development vs production rendering
|
||||
4. Clean separation between Python and JavaScript
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
import json
|
||||
|
||||
# Import the new plugin system
|
||||
from markitect.plugins import (
|
||||
PluginManager,
|
||||
RenderingEngineManager,
|
||||
RenderingConfig
|
||||
)
|
||||
from markitect.plugins.testdrive_jsui import TestDriveJSUIEngine
|
||||
|
||||
|
||||
def demo_standalone_development():
|
||||
"""Demo standalone development workflow."""
|
||||
print("🧪 Demonstrating Standalone Development Workflow")
|
||||
print("=" * 50)
|
||||
|
||||
# Initialize the TestDrive JSUI engine directly
|
||||
engine = TestDriveJSUIEngine()
|
||||
|
||||
# Read test content
|
||||
test_content_path = Path("testdrive-jsui/test-documents/sample.md")
|
||||
if test_content_path.exists():
|
||||
test_content = test_content_path.read_text()
|
||||
else:
|
||||
test_content = "# Demo Content\n\nThis is demo content for testing."
|
||||
|
||||
# Create standalone test document
|
||||
output_path = Path("/tmp/testdrive_standalone_demo.html")
|
||||
|
||||
print(f"📄 Creating standalone test document: {output_path}")
|
||||
|
||||
try:
|
||||
engine.create_standalone_test_document(test_content, output_path)
|
||||
print(f"✅ Success! Open in browser: file://{output_path}")
|
||||
except Exception as e:
|
||||
print(f"❌ Error creating standalone document: {e}")
|
||||
|
||||
return engine
|
||||
|
||||
|
||||
def demo_plugin_discovery():
|
||||
"""Demo plugin discovery through the main system."""
|
||||
print("\n🔍 Demonstrating Plugin Discovery")
|
||||
print("=" * 50)
|
||||
|
||||
# Initialize plugin manager
|
||||
plugin_manager = PluginManager()
|
||||
|
||||
print("📋 Discovering all plugins...")
|
||||
all_plugins = plugin_manager.discover_plugins()
|
||||
|
||||
# Show all discovered plugins
|
||||
for plugin_name, plugin_info in all_plugins.items():
|
||||
print(f" 🔌 {plugin_name}: {plugin_info.get('type', 'unknown')}")
|
||||
|
||||
# Initialize rendering engine manager
|
||||
rendering_manager = RenderingEngineManager(plugin_manager)
|
||||
|
||||
print("\n🎨 Available rendering engines:")
|
||||
for engine_name in rendering_manager.list_engines():
|
||||
engine = rendering_manager.get_engine(engine_name)
|
||||
if engine:
|
||||
print(f" 🎯 {engine_name}: modes={engine.get_supported_modes()}")
|
||||
|
||||
return rendering_manager
|
||||
|
||||
|
||||
def demo_production_deployment():
|
||||
"""Demo production deployment with asset management."""
|
||||
print("\n🚀 Demonstrating Production Deployment")
|
||||
print("=" * 50)
|
||||
|
||||
# Create production configuration
|
||||
output_dir = Path("/tmp/demo_production_output")
|
||||
output_dir.mkdir(exist_ok=True)
|
||||
|
||||
config = RenderingConfig(
|
||||
asset_base_url="_markitect",
|
||||
development_mode=False,
|
||||
output_directory=output_dir
|
||||
)
|
||||
|
||||
# Initialize engine
|
||||
engine = TestDriveJSUIEngine()
|
||||
|
||||
# Demo content
|
||||
demo_content = """# Production Demo
|
||||
|
||||
This demonstrates production deployment of the TestDrive JSUI plugin.
|
||||
|
||||
## Features
|
||||
- Asset deployment to `_markitect/plugins/testdrive-jsui/`
|
||||
- Production-ready HTML generation
|
||||
- Clean JavaScript-Python separation
|
||||
|
||||
## Testing
|
||||
Open the generated HTML file to test the production deployment.
|
||||
"""
|
||||
|
||||
print(f"📁 Output directory: {output_dir}")
|
||||
print(f"🔧 Asset base URL: {config.asset_base_url}")
|
||||
|
||||
# Render document
|
||||
try:
|
||||
html_content = engine.render_document(demo_content, "edit", config)
|
||||
|
||||
# Save to output directory
|
||||
output_file = output_dir / "demo_production.html"
|
||||
output_file.write_text(html_content)
|
||||
|
||||
print(f"✅ Production document created: {output_file}")
|
||||
print(f"🌐 Open in browser: file://{output_file}")
|
||||
|
||||
# Show asset requirements
|
||||
assets = engine.get_required_assets()
|
||||
print(f"\n📦 Required assets:")
|
||||
for asset_type, asset_list in assets.items():
|
||||
print(f" {asset_type}: {len(asset_list)} files")
|
||||
for asset in asset_list[:3]: # Show first 3
|
||||
print(f" - {asset}")
|
||||
if len(asset_list) > 3:
|
||||
print(f" ... and {len(asset_list) - 3} more")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error in production deployment: {e}")
|
||||
|
||||
return output_dir
|
||||
|
||||
|
||||
def demo_asset_url_generation():
|
||||
"""Demo asset URL generation for different modes."""
|
||||
print("\n🔗 Demonstrating Asset URL Generation")
|
||||
print("=" * 50)
|
||||
|
||||
engine = TestDriveJSUIEngine()
|
||||
|
||||
# Development configuration
|
||||
dev_config = RenderingConfig(
|
||||
asset_base_url=".",
|
||||
development_mode=True,
|
||||
plugin_source_dirs={
|
||||
"testdrive-jsui": Path("testdrive-jsui")
|
||||
}
|
||||
)
|
||||
|
||||
# Production configuration
|
||||
prod_config = RenderingConfig(
|
||||
asset_base_url="_markitect",
|
||||
development_mode=False
|
||||
)
|
||||
|
||||
sample_assets = ["static/js/main.js", "static/css/editor.css", "images/icon.png"]
|
||||
|
||||
print("Development URLs:")
|
||||
for asset in sample_assets:
|
||||
url = dev_config.get_asset_url("testdrive-jsui", asset)
|
||||
print(f" {asset} → {url}")
|
||||
|
||||
print("\nProduction URLs:")
|
||||
for asset in sample_assets:
|
||||
url = prod_config.get_asset_url("testdrive-jsui", asset)
|
||||
print(f" {asset} → {url}")
|
||||
|
||||
# Show JSON config generation
|
||||
print(f"\nDevelopment JSON config:")
|
||||
print(dev_config.to_json_config("testdrive-jsui"))
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all demo workflows."""
|
||||
print("🎯 TestDrive JSUI Plugin Integration Demo")
|
||||
print("🔬 Demonstrating JavaScript-first development approach")
|
||||
print("🏗️ Clean separation between Python and JavaScript\n")
|
||||
|
||||
try:
|
||||
# Demo workflows
|
||||
engine = demo_standalone_development()
|
||||
rendering_manager = demo_plugin_discovery()
|
||||
output_dir = demo_production_deployment()
|
||||
demo_asset_url_generation()
|
||||
|
||||
print(f"\n✅ All demos completed successfully!")
|
||||
print(f"🔬 Standalone test: testdrive-jsui/test.html")
|
||||
print(f"📄 Generated files in: {output_dir}")
|
||||
|
||||
# Show next steps
|
||||
print(f"\n📚 Next Steps:")
|
||||
print(f" 1. Open testdrive-jsui/test.html in browser for standalone dev")
|
||||
print(f" 2. Start development server: cd testdrive-jsui && python -m http.server 8080")
|
||||
print(f" 3. Integrate with markitect md-render command")
|
||||
print(f" 4. Add more rendering engines to the plugin system")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Demo failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
184
docs/CAPABILITIES_QUICK_REFERENCE.md
Normal file
184
docs/CAPABILITIES_QUICK_REFERENCE.md
Normal file
@@ -0,0 +1,184 @@
|
||||
# Capabilities Quick Reference
|
||||
|
||||
**⚠️ Critical:** Read [Capabilities Architecture](architecture/CAPABILITIES_ARCHITECTURE.md) for full details.
|
||||
|
||||
## Core Rules
|
||||
|
||||
### 🚫 **NEVER** Edit Capabilities from Main Repo
|
||||
|
||||
```bash
|
||||
# ❌ WRONG - Don't edit capability files from main repo
|
||||
cd /home/worsch/markitect-main/capabilities/testdrive-jsui
|
||||
vim src/testdrive_jsui/core.py # DON'T DO THIS!
|
||||
|
||||
# ✅ CORRECT - Use separate Claude instance/session
|
||||
# Open new terminal/Claude instance:
|
||||
git clone http://gitea/coulomb/testdrive-jsui.git /path/to/work
|
||||
cd /path/to/work/testdrive-jsui
|
||||
# Make changes, commit, push
|
||||
```
|
||||
|
||||
### 📦 Capabilities are Git Submodules
|
||||
|
||||
- Each capability = separate git repository
|
||||
- Located in `capabilities/` as submodules
|
||||
- Independent development lifecycle
|
||||
- Own versioning and releases
|
||||
|
||||
### 🔀 Use Separate Claude Instances
|
||||
|
||||
| Session | Purpose | Location |
|
||||
|---------|---------|----------|
|
||||
| **Main Repo** | Integration, configuration | `/home/worsch/markitect-main` |
|
||||
| **Capability** | Feature development, bugs | Separate clone or `capabilities/capability-name` |
|
||||
|
||||
**Why?** Prevents accidental cross-contamination and respects repository boundaries.
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Update Capability After Changes
|
||||
|
||||
```bash
|
||||
# After pushing changes to capability repo
|
||||
cd /home/worsch/markitect-main
|
||||
git submodule update --remote capabilities/testdrive-jsui
|
||||
git add capabilities/testdrive-jsui
|
||||
git commit -m "chore: update testdrive-jsui to latest"
|
||||
git push
|
||||
```
|
||||
|
||||
### Add New Capability
|
||||
|
||||
```bash
|
||||
cd /home/worsch/markitect-main
|
||||
|
||||
# Add as submodule
|
||||
git submodule add http://gitea/coulomb/new-capability.git capabilities/new-capability
|
||||
|
||||
# Add to pyproject.toml dependencies
|
||||
echo ' "new-capability @ file:./capabilities/new-capability",' >> pyproject.toml
|
||||
|
||||
# Commit
|
||||
git add .gitmodules capabilities/new-capability pyproject.toml
|
||||
git commit -m "feat: add new-capability submodule"
|
||||
```
|
||||
|
||||
### Work on Capability Feature
|
||||
|
||||
```bash
|
||||
# Option 1: In submodule directory (careful!)
|
||||
cd /home/worsch/markitect-main/capabilities/testdrive-jsui
|
||||
git checkout -b feature-branch
|
||||
# make changes
|
||||
git commit -m "feat: new feature"
|
||||
git push origin feature-branch
|
||||
|
||||
# Option 2: Separate clone (recommended)
|
||||
cd ~/projects
|
||||
git clone http://gitea/coulomb/testdrive-jsui.git
|
||||
cd testdrive-jsui
|
||||
git checkout -b feature-branch
|
||||
# make changes
|
||||
git commit -m "feat: new feature"
|
||||
git push origin feature-branch
|
||||
```
|
||||
|
||||
### Check Capability Status
|
||||
|
||||
```bash
|
||||
cd /home/worsch/markitect-main
|
||||
|
||||
# List all capabilities
|
||||
make capabilities-list
|
||||
|
||||
# Check submodule status
|
||||
git submodule status
|
||||
|
||||
# See which commit each capability is at
|
||||
git submodule foreach 'git log --oneline -1'
|
||||
```
|
||||
|
||||
## Integration Patterns
|
||||
|
||||
### Python Import
|
||||
|
||||
```python
|
||||
# ✅ Correct - Import from capability package
|
||||
from testdrive_jsui import TestDriveJSUIEngine
|
||||
|
||||
engine = TestDriveJSUIEngine()
|
||||
```
|
||||
|
||||
### Dependency Declaration
|
||||
|
||||
```toml
|
||||
# pyproject.toml
|
||||
dependencies = [
|
||||
"testdrive-jsui @ file:./capabilities/testdrive-jsui",
|
||||
]
|
||||
```
|
||||
|
||||
### Plugin Self-Declaration
|
||||
|
||||
```python
|
||||
# ✅ Good - Plugin declares its own location
|
||||
def get_plugin_source_dir(self) -> Path:
|
||||
return Path(__file__).parent.parent.parent / "capabilities" / "testdrive-jsui"
|
||||
|
||||
# ❌ Bad - Hardcoded in main repo
|
||||
if plugin_name == 'testdrive-jsui':
|
||||
return Path('capabilities/testdrive-jsui')
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Submodule not initialized"
|
||||
|
||||
```bash
|
||||
git submodule update --init --recursive
|
||||
```
|
||||
|
||||
### "Import error: No module named 'capability_name'"
|
||||
|
||||
```bash
|
||||
pip install -e ./capabilities/capability-name
|
||||
# or
|
||||
pip install -e . # Install all dependencies
|
||||
```
|
||||
|
||||
### "Merge conflict in submodule"
|
||||
|
||||
```bash
|
||||
# Don't resolve in main repo!
|
||||
# Go to capability repo and resolve there
|
||||
cd capabilities/testdrive-jsui
|
||||
git pull origin main
|
||||
# Resolve conflicts, commit, push
|
||||
|
||||
cd ../..
|
||||
git submodule update --remote capabilities/testdrive-jsui
|
||||
git add capabilities/testdrive-jsui
|
||||
git commit -m "chore: update testdrive-jsui after conflict resolution"
|
||||
```
|
||||
|
||||
## Current Capabilities
|
||||
|
||||
| Capability | Type | Repository | Status |
|
||||
|------------|------|------------|--------|
|
||||
| **testdrive-jsui** | Rendering Engine | `coulomb/testdrive-jsui` | ✅ Submodule |
|
||||
| **issue-facade** | CLI Tool | `coulomb/issue-facade` | ✅ Submodule |
|
||||
| **kaizen-agentic** | Framework | `coulomb/kaizen-agentic` | ✅ Submodule |
|
||||
| **release-management** | Tool | Local | 📦 To migrate |
|
||||
| **markitect-content** | Library | Local | 📦 To migrate |
|
||||
|
||||
## Key Principles
|
||||
|
||||
1. **Separation of Concerns** - Main repo doesn't modify capabilities
|
||||
2. **Independent Development** - Each capability has own lifecycle
|
||||
3. **Interface-Based Integration** - Use documented APIs only
|
||||
4. **Version Control** - Git submodules for dependency management
|
||||
5. **Session Isolation** - Separate Claude instances per repository
|
||||
|
||||
---
|
||||
|
||||
**📖 Full Documentation:** [Capabilities Architecture](architecture/CAPABILITIES_ARCHITECTURE.md)
|
||||
174
docs/DOCUMENT_NAVIGATOR_INTEGRATION.md
Normal file
174
docs/DOCUMENT_NAVIGATOR_INTEGRATION.md
Normal file
@@ -0,0 +1,174 @@
|
||||
# DocumentNavigator Integration Guide
|
||||
|
||||
## TDD Implementation Complete ✅
|
||||
|
||||
The DocumentNavigator widget has been successfully implemented following Test-Driven Development methodology:
|
||||
|
||||
### ✅ **Completed Components**
|
||||
|
||||
1. **Base Architecture** (`js/widgets/base/`)
|
||||
- `Widget.js` - Core widget functionality with events and state
|
||||
- `UIWidget.js` - DOM manipulation and visual behavior
|
||||
|
||||
2. **DocumentNavigator Widget** (`js/widgets/navigation/DocumentNavigator.js`)
|
||||
- Substack-style floating navigation panel
|
||||
- Hierarchical heading extraction and tree building
|
||||
- Expand/collapse with smooth animations
|
||||
- Scroll spy with current section highlighting
|
||||
- Responsive behavior (auto-hide on mobile)
|
||||
- Keyboard navigation support
|
||||
- Smooth scrolling to sections
|
||||
|
||||
3. **Plugin Definition** (`js/plugins/document-navigator-plugin.js`)
|
||||
- Complete plugin metadata and configuration
|
||||
- Lazy loading support
|
||||
- Theme variants (default, dark, minimal)
|
||||
- Usage examples and development helpers
|
||||
|
||||
4. **TDD Test Suite** (`js/tests/test-document-navigator.js`)
|
||||
- Comprehensive test coverage (15 test cases)
|
||||
- Browser-based test runner included
|
||||
- Tests all functionality: rendering, navigation, scroll spy, responsive behavior
|
||||
|
||||
## Integration with HTML Rendering
|
||||
|
||||
To integrate the DocumentNavigator into all rendered markdown documents, add the following to the HTML template in `CleanDocumentManager._generate_html_template()`:
|
||||
|
||||
### **Method 1: Simple Integration (Immediate Use)**
|
||||
|
||||
Add this JavaScript after the existing component initialization:
|
||||
|
||||
```javascript
|
||||
// Add DocumentNavigator initialization after existing components
|
||||
// (Insert around line 1050 in clean_document_manager.py, after documentControls.create())
|
||||
|
||||
// Initialize DocumentNavigator if headings are present
|
||||
try {
|
||||
// Import the widget classes (using dynamic imports for future plugin system)
|
||||
const documentNavigator = new DocumentNavigator({
|
||||
container: document.getElementById('markdown-content') || document.body,
|
||||
position: 'left',
|
||||
collapsed: true,
|
||||
theme: '${template or "default"}', // Use current document theme
|
||||
enableScrollSpy: true,
|
||||
autoHide: true
|
||||
});
|
||||
|
||||
// Initialize and render
|
||||
documentNavigator.initialize().then(() => {
|
||||
return documentNavigator.render();
|
||||
}).then(() => {
|
||||
console.log('✓ DocumentNavigator initialized successfully');
|
||||
}).catch(error => {
|
||||
console.warn('DocumentNavigator initialization failed:', error.message);
|
||||
});
|
||||
} catch (error) {
|
||||
console.warn('DocumentNavigator not available:', error.message);
|
||||
}
|
||||
```
|
||||
|
||||
### **Method 2: Plugin System Integration (Future-Ready)**
|
||||
|
||||
For the full plugin architecture, the initialization would look like:
|
||||
|
||||
```javascript
|
||||
// Future plugin system integration
|
||||
if (typeof widgetSystem !== 'undefined') {
|
||||
widgetSystem.createWidget('DocumentNavigator', {
|
||||
theme: '${template or "default"}',
|
||||
position: 'left'
|
||||
}).then(navigator => {
|
||||
return navigator.show();
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
Once integrated, the DocumentNavigator will:
|
||||
|
||||
1. **Auto-detect headings** in the rendered markdown content
|
||||
2. **Show collapsed toggle** on the left side (hamburger menu icon)
|
||||
3. **Expand on click** to reveal table of contents
|
||||
4. **Highlight current section** as user scrolls
|
||||
5. **Navigate smoothly** when headings are clicked
|
||||
6. **Auto-hide on mobile** devices
|
||||
7. **Support keyboard navigation** (Enter/Space to toggle, Escape to collapse)
|
||||
|
||||
## Testing
|
||||
|
||||
To test the implementation:
|
||||
|
||||
1. **Run TDD Test Suite**:
|
||||
```bash
|
||||
# Start local server
|
||||
cd markitect/static/js/tests
|
||||
python -m http.server 8080
|
||||
|
||||
# Open browser to: http://localhost:8080/test-document-navigator-runner.html
|
||||
# Click "Run TDD Test Suite" button
|
||||
```
|
||||
|
||||
2. **Test with Real Content**:
|
||||
```bash
|
||||
# Create test markdown with headings
|
||||
echo "# Chapter 1
|
||||
## Section 1.1
|
||||
### Subsection 1.1.1
|
||||
## Section 1.2
|
||||
# Chapter 2" > test-doc.md
|
||||
|
||||
# Render with navigator
|
||||
markitect md-render test-doc.md --output test-doc.html
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
The DocumentNavigator supports extensive customization:
|
||||
|
||||
```javascript
|
||||
const navigator = new DocumentNavigator({
|
||||
position: 'left', // 'left' or 'right'
|
||||
collapsed: true, // Start collapsed
|
||||
autoHide: true, // Hide on mobile
|
||||
maxHeadingLevel: 3, // H1, H2, H3
|
||||
enableScrollSpy: true, // Highlight current section
|
||||
smoothScroll: true, // Smooth scroll animation
|
||||
theme: 'default', // 'default', 'dark', 'minimal'
|
||||
width: '280px', // Expanded width
|
||||
offset: { top: '80px', side: '20px' }
|
||||
});
|
||||
```
|
||||
|
||||
## Theme Integration
|
||||
|
||||
The navigator automatically adapts to document themes:
|
||||
|
||||
- **Default Theme**: Clean white background with subtle shadows
|
||||
- **Dark Theme**: Dark background with light text
|
||||
- **Substack Theme**: Warm cream colors matching document style
|
||||
- **Academic Theme**: Traditional academic styling
|
||||
- **ChatGPT Theme**: Modern compact layout
|
||||
|
||||
## Performance
|
||||
|
||||
- **Lazy Loading**: Widget loads only when headings are detected
|
||||
- **Efficient Scroll Spy**: Throttled scroll events (100ms)
|
||||
- **Responsive**: Automatically hides on mobile to save space
|
||||
- **Memory Efficient**: Proper cleanup on destroy
|
||||
|
||||
## Browser Support
|
||||
|
||||
- **Modern Browsers**: Chrome 80+, Firefox 75+, Safari 13+, Edge 80+
|
||||
- **ES6 Modules**: Uses dynamic imports (can be transpiled for older browsers)
|
||||
- **Progressive Enhancement**: Gracefully degrades if JavaScript fails
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Add to HTML Template**: Integrate the JavaScript code into `CleanDocumentManager._generate_html_template()`
|
||||
2. **Test Integration**: Verify navigator appears in rendered documents
|
||||
3. **Theme Refinement**: Adjust colors to perfectly match document themes
|
||||
4. **Plugin System**: Implement full plugin architecture for future extensibility
|
||||
5. **Performance Optimization**: Add preloading and caching optimizations
|
||||
|
||||
The DocumentNavigator widget is production-ready and provides a professional Substack-style navigation experience for all markdown documents rendered by Markitect.
|
||||
263
docs/ERROR_HANDLING_STRATEGY.md
Normal file
263
docs/ERROR_HANDLING_STRATEGY.md
Normal file
@@ -0,0 +1,263 @@
|
||||
# Error Handling Strategy: Fail Fast + Robustness Balance
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the balanced error handling strategy that combines **Fail Fast** principles for development with **Robustness Principles** for production, preventing both cascading failures and difficult diagnosis.
|
||||
|
||||
## Core Philosophy
|
||||
|
||||
### 🚨 **Development Mode (Fail Fast)**
|
||||
- **Immediate failure** on errors for fast debugging
|
||||
- **Strict validation** with exceptions on invalid input
|
||||
- **No silent failures** - all problems surface immediately
|
||||
- **Clear error messages** with full context
|
||||
|
||||
### 🛡️ **Production Mode (Robust)**
|
||||
- **Graceful degradation** when components fail
|
||||
- **Fallback behaviors** for non-critical failures
|
||||
- **Silent recovery** for user experience
|
||||
- **Detailed logging** for post-mortem analysis
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### Mode Detection
|
||||
```javascript
|
||||
const MARKITECT_STRICT_MODE = (
|
||||
window.location.hostname === 'localhost' ||
|
||||
window.location.hostname === '127.0.0.1' ||
|
||||
window.location.search.includes('strict=true') ||
|
||||
window.markitectStrictMode === true
|
||||
);
|
||||
```
|
||||
|
||||
### Dual-Behavior Error Handling
|
||||
```javascript
|
||||
safeOperation: function(operation, fallback = null, context = 'Unknown') {
|
||||
try {
|
||||
return operation();
|
||||
} catch (error) {
|
||||
console.warn(`Operation failed in ${context}:`, error);
|
||||
|
||||
// Fail Fast in development mode
|
||||
if (MARKITECT_STRICT_MODE) {
|
||||
console.error(`🚨 STRICT MODE: Operation failed in ${context}`);
|
||||
throw error; // Re-throw for immediate debugging
|
||||
}
|
||||
|
||||
// Robust handling in production
|
||||
if (window.MarkitectDebugSystem) {
|
||||
window.MarkitectDebugSystem.addMessage(
|
||||
`Safe operation failed: ${error.message}`,
|
||||
'WARNING',
|
||||
'System',
|
||||
{ context, eventType: 'ERROR' }
|
||||
);
|
||||
}
|
||||
return typeof fallback === 'function' ? fallback() : fallback;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Error Categories & Responses
|
||||
|
||||
### 1. **Critical System Errors**
|
||||
| Error Type | Development Response | Production Response |
|
||||
|------------|---------------------|-------------------|
|
||||
| Missing Dependencies | `throw Error()` immediately | Skip with warning, continue |
|
||||
| Invalid Configuration | `throw Error()` immediately | Use defaults, log error |
|
||||
| DOM Not Ready | `throw Error()` immediately | Retry with timeout |
|
||||
|
||||
### 2. **Input Validation Errors**
|
||||
| Error Type | Development Response | Production Response |
|
||||
|------------|---------------------|-------------------|
|
||||
| Malformed Data | `throw Error()` with details | Sanitize and continue |
|
||||
| Oversized Input | `throw Error()` immediately | Truncate with warning |
|
||||
| Invalid Selectors | `throw Error()` with context | Return null, log warning |
|
||||
|
||||
### 3. **Resource Errors**
|
||||
| Error Type | Development Response | Production Response |
|
||||
|------------|---------------------|-------------------|
|
||||
| Memory Exhaustion | `throw Error()` to prevent hang | Apply limits, degrade features |
|
||||
| Network Failures | `throw Error()` for debugging | Use cached data, retry logic |
|
||||
| Timeout Exceeded | `throw Error()` immediately | Cancel operation, fallback |
|
||||
|
||||
### 4. **UI Component Errors**
|
||||
| Error Type | Development Response | Production Response |
|
||||
|------------|---------------------|-------------------|
|
||||
| Control Creation Failed | `throw Error()` with stack | Create minimal fallback |
|
||||
| DOM Manipulation Failed | `throw Error()` with element | Skip operation, continue |
|
||||
| Event Handler Error | `throw Error()` to debug | Log error, disable feature |
|
||||
|
||||
## Logging Strategy
|
||||
|
||||
### Development Mode
|
||||
```javascript
|
||||
// Immediate console errors
|
||||
console.error(`🚨 STRICT MODE: ${message}`);
|
||||
throw new Error(message);
|
||||
```
|
||||
|
||||
### Production Mode
|
||||
```javascript
|
||||
// Silent logging with context
|
||||
window.MarkitectDebugSystem.addMessage(
|
||||
message,
|
||||
'ERROR',
|
||||
component,
|
||||
{ context, stackTrace: error.stack }
|
||||
);
|
||||
|
||||
// User-friendly fallbacks
|
||||
return fallbackValue || defaultBehavior();
|
||||
```
|
||||
|
||||
## Testing Approach
|
||||
|
||||
### Development Testing
|
||||
- **Error Injection**: Intentionally trigger failures
|
||||
- **Boundary Testing**: Test limits and edge cases
|
||||
- **Dependency Mocking**: Remove required components
|
||||
- **Strict Validation**: Ensure all errors surface
|
||||
|
||||
### Production Testing
|
||||
- **Graceful Degradation**: Verify fallbacks work
|
||||
- **Performance Under Load**: Stress test with errors
|
||||
- **User Experience**: No broken interfaces
|
||||
- **Recovery Scenarios**: System self-healing
|
||||
|
||||
## Implementation Examples
|
||||
|
||||
### Control Initialization
|
||||
```javascript
|
||||
initializeControl: function(controlClass, controlName, icon = '🔧') {
|
||||
try {
|
||||
if (!controlClass) {
|
||||
const message = `${controlName} class not available`;
|
||||
|
||||
// Fail Fast in development
|
||||
if (MARKITECT_STRICT_MODE) {
|
||||
throw new Error(message);
|
||||
}
|
||||
|
||||
// Graceful in production
|
||||
console.warn(message);
|
||||
return this.createFallbackControl(controlName, icon);
|
||||
}
|
||||
|
||||
return new controlClass().createControl();
|
||||
} catch (error) {
|
||||
if (MARKITECT_STRICT_MODE) {
|
||||
throw error; // Let it bubble up
|
||||
}
|
||||
|
||||
// Production: log and continue
|
||||
this.logError(error, controlName);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Input Validation
|
||||
```javascript
|
||||
validateAndSanitize: function(input, maxLength = 1000) {
|
||||
if (typeof input !== 'string') {
|
||||
const error = new TypeError('Input must be string');
|
||||
|
||||
if (MARKITECT_STRICT_MODE) {
|
||||
throw error;
|
||||
}
|
||||
|
||||
return String(input).slice(0, maxLength);
|
||||
}
|
||||
|
||||
if (input.length > maxLength) {
|
||||
const error = new Error(`Input exceeds ${maxLength} characters`);
|
||||
|
||||
if (MARKITECT_STRICT_MODE) {
|
||||
throw error;
|
||||
}
|
||||
|
||||
console.warn('Input truncated to fit limits');
|
||||
return input.slice(0, maxLength);
|
||||
}
|
||||
|
||||
return input;
|
||||
}
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
### 🚀 **Development Benefits**
|
||||
- **Fast Problem Discovery**: Errors surface immediately
|
||||
- **Clear Error Context**: Full stack traces and details
|
||||
- **Prevents Technical Debt**: Forces proper error handling
|
||||
- **Debugging Efficiency**: No need to backtrack from symptoms
|
||||
|
||||
### 🛡️ **Production Benefits**
|
||||
- **System Stability**: Graceful degradation prevents crashes
|
||||
- **User Experience**: No broken interfaces or white screens
|
||||
- **Self-Healing**: Automatic fallbacks and recovery
|
||||
- **Operational Monitoring**: Detailed error telemetry
|
||||
|
||||
### ⚖️ **Balance Benefits**
|
||||
- **Best of Both Worlds**: Development speed + Production stability
|
||||
- **Context-Appropriate**: Right behavior for the right environment
|
||||
- **Maintainable**: Clear patterns and consistent implementation
|
||||
- **Scalable**: Works from development to enterprise deployment
|
||||
|
||||
## Activation Guide
|
||||
|
||||
### Automatic Detection
|
||||
- `localhost` and `127.0.0.1` automatically enable strict mode
|
||||
- URL parameter `?strict=true` forces strict mode
|
||||
- Global flag `window.markitectStrictMode = true`
|
||||
|
||||
### Manual Control
|
||||
```javascript
|
||||
// Force strict mode for testing
|
||||
window.markitectStrictMode = true;
|
||||
|
||||
// Force production mode (disable strict)
|
||||
window.markitectStrictMode = false;
|
||||
```
|
||||
|
||||
### Environment Configuration
|
||||
```javascript
|
||||
// In development builds
|
||||
const DEVELOPMENT_BUILD = true;
|
||||
const MARKITECT_STRICT_MODE = DEVELOPMENT_BUILD || detectDevelopmentEnvironment();
|
||||
|
||||
// In production builds
|
||||
const DEVELOPMENT_BUILD = false;
|
||||
const MARKITECT_STRICT_MODE = false; // Always robust in production
|
||||
```
|
||||
|
||||
## Monitoring & Metrics
|
||||
|
||||
### Development Metrics
|
||||
- **Error Count**: Number of strict mode exceptions
|
||||
- **Error Categories**: Types of failures encountered
|
||||
- **Resolution Time**: Time to fix after error discovery
|
||||
- **Test Coverage**: Percentage of error paths tested
|
||||
|
||||
### Production Metrics
|
||||
- **Fallback Usage**: How often graceful degradation occurs
|
||||
- **Recovery Success**: Percentage of successful recoveries
|
||||
- **User Impact**: Features disabled vs. core functionality maintained
|
||||
- **Error Patterns**: Common failure modes for improvement
|
||||
|
||||
## Future Evolution
|
||||
|
||||
### Enhanced Detection
|
||||
- **CI/CD Integration**: Automatic strict mode in testing pipelines
|
||||
- **Feature Flags**: Remote control of error handling behavior
|
||||
- **A/B Testing**: Compare error handling strategies
|
||||
- **Machine Learning**: Predict and prevent common failures
|
||||
|
||||
### Advanced Recovery
|
||||
- **Smart Fallbacks**: Context-aware recovery strategies
|
||||
- **Progressive Enhancement**: Gradually restore failed features
|
||||
- **User Notification**: Inform users of degraded functionality
|
||||
- **Automatic Reporting**: Send error telemetry to development team
|
||||
|
||||
This balanced approach ensures we catch problems early in development while maintaining a bulletproof production experience.
|
||||
492
docs/PLUGIN_SYSTEM.md
Normal file
492
docs/PLUGIN_SYSTEM.md
Normal file
@@ -0,0 +1,492 @@
|
||||
# Markitect Plugin System Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
The Markitect plugin system provides a modular architecture for extending rendering capabilities with independent JavaScript UI components. This system enables JavaScript-first development while maintaining clean integration with the Python ecosystem.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
1. **RenderingEnginePlugin**: Base class for UI rendering engines
|
||||
2. **RenderingConfig**: Asset management and deployment configuration
|
||||
3. **RenderingEngineManager**: Plugin discovery and lifecycle management
|
||||
4. **PluginManager**: Integration with existing Markitect plugin system
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Clean Separation**: JSON-based configuration interface (Python ↔ JavaScript)
|
||||
- **Independent Development**: JavaScript components work standalone
|
||||
- **Asset Management**: Configurable deployment strategies
|
||||
- **Multiple Engines**: Support for different UI frameworks
|
||||
- **Fallback Support**: Graceful degradation to standard rendering
|
||||
|
||||
## Plugin Development
|
||||
|
||||
### Creating a Rendering Engine Plugin
|
||||
|
||||
1. **Extend RenderingEnginePlugin**:
|
||||
|
||||
```python
|
||||
from markitect.plugins.rendering import RenderingEnginePlugin, RenderingConfig
|
||||
from markitect.plugins.base import PluginMetadata, PluginType
|
||||
|
||||
class MyUIEngine(RenderingEnginePlugin):
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self._metadata = PluginMetadata(
|
||||
name="my-ui-engine",
|
||||
version="1.0.0",
|
||||
description="Custom UI rendering engine",
|
||||
author="Your Name",
|
||||
plugin_type=PluginType.RENDERING
|
||||
)
|
||||
|
||||
@property
|
||||
def metadata(self) -> PluginMetadata:
|
||||
return self._metadata
|
||||
|
||||
def get_supported_modes(self) -> List[str]:
|
||||
return ["edit", "view"]
|
||||
|
||||
def get_required_assets(self) -> Dict[str, List[str]]:
|
||||
return {
|
||||
"js": ["static/js/main.js"],
|
||||
"css": ["static/css/style.css"]
|
||||
}
|
||||
|
||||
def render_document(self, content: str, mode: str, config: RenderingConfig) -> str:
|
||||
# Your rendering logic here
|
||||
return html_output
|
||||
```
|
||||
|
||||
2. **Directory Structure**:
|
||||
|
||||
```
|
||||
my-ui-engine/
|
||||
├── static/
|
||||
│ ├── js/ # JavaScript components
|
||||
│ └── css/ # Stylesheets
|
||||
├── templates/ # HTML templates
|
||||
├── images/ # Icons and images
|
||||
├── test-documents/ # Sample markdown files
|
||||
├── package.json # Node.js configuration
|
||||
└── README.md # Plugin documentation
|
||||
```
|
||||
|
||||
3. **Asset Management**:
|
||||
|
||||
Assets are automatically deployed based on configuration:
|
||||
- **Development**: Served from plugin source directory
|
||||
- **Production**: Copied to `_markitect/plugins/{plugin-name}/`
|
||||
|
||||
## TestDrive JSUI Plugin
|
||||
|
||||
### Overview
|
||||
|
||||
The TestDrive JSUI plugin demonstrates the plugin architecture with a complete JavaScript UI for markdown editing.
|
||||
|
||||
### Features
|
||||
|
||||
- **Modular Components**: Clean separation of UI components
|
||||
- **Compass Positioning**: NW, NE, E, SE control panel layout
|
||||
- **Section Management**: Click-to-edit markdown sections
|
||||
- **Debug System**: Built-in debugging and logging
|
||||
- **Asset Pipeline**: Configurable asset deployment
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
testdrive-jsui/
|
||||
├── static/js/
|
||||
│ ├── core/ # Core systems
|
||||
│ │ ├── debug-system.js
|
||||
│ │ └── section-manager.js
|
||||
│ ├── components/ # UI components
|
||||
│ │ ├── debug-panel.js
|
||||
│ │ ├── document-controls.js
|
||||
│ │ └── dom-renderer.js
|
||||
│ ├── controls/ # Control panels
|
||||
│ │ ├── control-base.js
|
||||
│ │ ├── contents-control.js # Northwest
|
||||
│ │ ├── status-control.js # East
|
||||
│ │ ├── debug-control.js # Southeast
|
||||
│ │ └── edit-control.js # Northeast
|
||||
│ ├── config-loader.js # Configuration interface
|
||||
│ └── main-updated.js # Application entry point
|
||||
├── static/css/ # Stylesheets (future)
|
||||
├── images/ # Icons and images (future)
|
||||
├── templates/
|
||||
│ └── index.html # Main HTML template
|
||||
├── test-documents/
|
||||
│ └── sample.md # Test content
|
||||
├── test.html # Standalone development
|
||||
├── package.json # Node.js configuration
|
||||
└── README.md # Plugin documentation
|
||||
```
|
||||
|
||||
### JavaScript Architecture
|
||||
|
||||
- **Configuration Interface**: Clean JSON data transfer via `markitect-config` script element
|
||||
- **Modular Components**: Each component has single responsibility
|
||||
- **Event System**: Pub/sub for component communication
|
||||
- **Control System**: Abstract base class for UI controls
|
||||
- **Compass Positioning**: Consistent control panel layout
|
||||
|
||||
## CLI Integration
|
||||
|
||||
### Command Line Usage
|
||||
|
||||
```bash
|
||||
# Use default engine (testdrive-jsui for edit/insert, standard for view)
|
||||
markitect md-render --edit document.md
|
||||
|
||||
# Specify engine explicitly
|
||||
markitect md-render --engine testdrive-jsui --edit document.md
|
||||
|
||||
# Use standard engine
|
||||
markitect md-render --engine standard --edit document.md
|
||||
|
||||
# View available engines
|
||||
markitect md-render --help
|
||||
```
|
||||
|
||||
### Engine Selection Logic
|
||||
|
||||
1. **Default Selection**:
|
||||
- Edit/Insert modes: `testdrive-jsui`
|
||||
- View mode: `standard`
|
||||
|
||||
2. **Explicit Selection**: Use `--engine` parameter
|
||||
|
||||
3. **Fallback Strategy**:
|
||||
- Engine not found → fallback to standard
|
||||
- Mode not supported → fallback to standard
|
||||
- Plugin error → fallback to standard
|
||||
|
||||
### Integration Points
|
||||
|
||||
The CLI integrates with the plugin system through:
|
||||
|
||||
```python
|
||||
# Engine discovery
|
||||
plugin_manager = PluginManager()
|
||||
rendering_manager = RenderingEngineManager(plugin_manager)
|
||||
|
||||
# Engine selection
|
||||
engine = rendering_manager.get_engine(engine_name)
|
||||
|
||||
# Configuration
|
||||
config = RenderingConfig(
|
||||
asset_base_url="_markitect",
|
||||
development_mode=False,
|
||||
output_directory=output_path.parent
|
||||
)
|
||||
|
||||
# Rendering
|
||||
html_content = engine.render_document(content, mode, config)
|
||||
```
|
||||
|
||||
## Development Workflows
|
||||
|
||||
### Standalone JavaScript Development
|
||||
|
||||
1. **Setup**:
|
||||
```bash
|
||||
cd testdrive-jsui
|
||||
python -m http.server 8080
|
||||
```
|
||||
|
||||
2. **Development**:
|
||||
- Edit JavaScript files in `static/js/`
|
||||
- Refresh browser to see changes
|
||||
- Use `test.html` for testing
|
||||
- Browser DevTools for debugging
|
||||
|
||||
3. **Benefits**:
|
||||
- No Python environment required
|
||||
- Fast iteration cycle
|
||||
- Standard web development tools
|
||||
- Hot reloading
|
||||
|
||||
### Integrated Development
|
||||
|
||||
1. **Plugin Testing**:
|
||||
```bash
|
||||
python demo_plugin_integration.py
|
||||
```
|
||||
|
||||
2. **CLI Testing**:
|
||||
```bash
|
||||
markitect md-render --engine testdrive-jsui --edit test.md
|
||||
```
|
||||
|
||||
3. **Integration Verification**:
|
||||
```bash
|
||||
python test_complete_integration.py
|
||||
```
|
||||
|
||||
## Asset Management
|
||||
|
||||
### Development Mode
|
||||
|
||||
```python
|
||||
config = RenderingConfig(
|
||||
asset_base_url=".",
|
||||
development_mode=True,
|
||||
plugin_source_dirs={"testdrive-jsui": Path("testdrive-jsui")}
|
||||
)
|
||||
|
||||
# Assets served as: file://testdrive-jsui/static/js/main.js
|
||||
```
|
||||
|
||||
### Production Mode
|
||||
|
||||
```python
|
||||
config = RenderingConfig(
|
||||
asset_base_url="_markitect",
|
||||
development_mode=False,
|
||||
output_directory=Path("/output")
|
||||
)
|
||||
|
||||
# Assets served as: _markitect/plugins/testdrive-jsui/static/js/main.js
|
||||
```
|
||||
|
||||
### Asset Types
|
||||
|
||||
- **js**: JavaScript files
|
||||
- **css**: Stylesheets
|
||||
- **images**: Icons, graphics
|
||||
- **external**: CDN resources
|
||||
|
||||
### Deployment Strategy
|
||||
|
||||
1. **Assets Copying**: Plugin assets copied to `_markitect/plugins/{name}/`
|
||||
2. **URL Generation**: Automatic URL generation for templates
|
||||
3. **Cache Management**: Asset versioning and cache control
|
||||
4. **Error Handling**: Fallback for missing assets
|
||||
|
||||
### Asset Deployment Process
|
||||
|
||||
When using CLI with plugin engines, assets are automatically deployed:
|
||||
|
||||
```bash
|
||||
# Assets are deployed to output directory when using plugin engines
|
||||
markitect md-render --edit document.md --output /path/to/output.html
|
||||
|
||||
# Output structure:
|
||||
# /path/to/
|
||||
# ├── output.html
|
||||
# └── _markitect/
|
||||
# └── plugins/
|
||||
# └── testdrive-jsui/
|
||||
# ├── static/
|
||||
# │ ├── js/ # 12 JavaScript files
|
||||
# │ └── css/ # 3 CSS files
|
||||
# └── images/ # 3 image files
|
||||
```
|
||||
|
||||
The deployment process:
|
||||
|
||||
1. **Plugin Discovery**: Engine identified (default: testdrive-jsui for edit mode)
|
||||
2. **Asset Analysis**: Required assets determined from `get_required_assets()`
|
||||
3. **Source Resolution**: Plugin source directory located
|
||||
4. **File Copying**: Assets copied with directory structure preservation
|
||||
5. **URL Generation**: HTML references generated with correct paths
|
||||
6. **Verification**: Asset accessibility validated
|
||||
|
||||
Example output:
|
||||
```
|
||||
🎯 Using rendering engine: testdrive-jsui (supports: edit, view)
|
||||
📦 Deploying assets for engine 'testdrive-jsui'...
|
||||
📄 Deployed 18 asset files
|
||||
js: 12 files
|
||||
css: 3 files
|
||||
images: 3 files
|
||||
✅ Rendered with INTERACTIVE editing mode to: output.html
|
||||
```
|
||||
|
||||
## Configuration Interface
|
||||
|
||||
### Python → JavaScript Data Transfer
|
||||
|
||||
All dynamic data passes through a clean JSON interface:
|
||||
|
||||
```html
|
||||
<script id="markitect-config" type="application/json">
|
||||
{
|
||||
"markdownContent": "# Document content...",
|
||||
"mode": "edit",
|
||||
"theme": "github",
|
||||
"keyboardShortcuts": true,
|
||||
"originalFilename": "document.md",
|
||||
"version": "Markitect v0.8.1"
|
||||
}
|
||||
</script>
|
||||
```
|
||||
|
||||
### JavaScript Configuration Loading
|
||||
|
||||
```javascript
|
||||
// Clean configuration loading
|
||||
class MarkitectConfig {
|
||||
constructor() {
|
||||
this.loadConfig();
|
||||
}
|
||||
|
||||
loadConfig() {
|
||||
const configElement = document.getElementById('markitect-config');
|
||||
this.config = JSON.parse(configElement.textContent);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Benefits
|
||||
|
||||
- **No String Interpolation**: Prevents template literal escaping issues
|
||||
- **Type Safety**: JSON validation and error handling
|
||||
- **Clean Separation**: No JavaScript code in Python strings
|
||||
- **Debuggable**: Easy to inspect configuration in browser
|
||||
|
||||
## Testing
|
||||
|
||||
### Plugin Testing
|
||||
|
||||
```bash
|
||||
# Basic plugin discovery
|
||||
python test_plugin_discovery.py
|
||||
|
||||
# CLI integration logic
|
||||
python test_cli_simple.py
|
||||
|
||||
# Complete scenario testing
|
||||
python test_complete_integration.py
|
||||
|
||||
# Full integration demo
|
||||
python demo_plugin_integration.py
|
||||
```
|
||||
|
||||
### Browser Testing
|
||||
|
||||
1. **Standalone**: Open `testdrive-jsui/test.html`
|
||||
2. **Generated**: Open CLI-generated HTML files
|
||||
3. **DevTools**: Use browser debugging tools
|
||||
4. **Console**: Check for JavaScript errors
|
||||
|
||||
### Integration Testing
|
||||
|
||||
- **Unit Tests**: Individual component testing
|
||||
- **Integration Tests**: Component interaction testing
|
||||
- **E2E Tests**: Full workflow testing
|
||||
- **Regression Tests**: Ensure stability across changes
|
||||
|
||||
## Extension Points
|
||||
|
||||
### Adding New Engines
|
||||
|
||||
1. Create new plugin extending `RenderingEnginePlugin`
|
||||
2. Implement required methods (`get_supported_modes`, `render_document`, etc.)
|
||||
3. Register in `RenderingEngineManager._register_builtin_rendering_engines()`
|
||||
4. Test with CLI integration
|
||||
|
||||
### Adding New Modes
|
||||
|
||||
1. Add mode to engine's `get_supported_modes()`
|
||||
2. Update `render_document()` to handle new mode
|
||||
3. Test mode validation and rendering
|
||||
4. Update CLI integration if needed
|
||||
|
||||
### Adding New Asset Types
|
||||
|
||||
1. Update `get_required_assets()` return format
|
||||
2. Modify asset deployment logic in `RenderingConfig`
|
||||
3. Update template system to handle new asset types
|
||||
4. Test asset URL generation
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Plugin Development
|
||||
|
||||
- **Single Responsibility**: Each component has one clear purpose
|
||||
- **Clean Interfaces**: Well-defined APIs between components
|
||||
- **Error Handling**: Graceful degradation on failures
|
||||
- **Documentation**: Clear README and code comments
|
||||
|
||||
### JavaScript Development
|
||||
|
||||
- **Modular Architecture**: Avoid monolithic JavaScript files
|
||||
- **Event-Driven**: Use pub/sub for component communication
|
||||
- **Configuration-Driven**: Avoid hardcoded values
|
||||
- **Browser Compatibility**: Test across different browsers
|
||||
|
||||
### Asset Management
|
||||
|
||||
- **Relative Paths**: Use relative paths in asset definitions
|
||||
- **Versioning**: Include version info for cache management
|
||||
- **Optimization**: Minimize asset size for production
|
||||
- **CDN Integration**: Use CDN for external dependencies
|
||||
|
||||
### Testing Strategy
|
||||
|
||||
- **Automated Testing**: Comprehensive test coverage
|
||||
- **Manual Testing**: User workflow validation
|
||||
- **Cross-Platform**: Test on different environments
|
||||
- **Performance Testing**: Monitor rendering performance
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Plugin Not Found**:
|
||||
- Check plugin registration in `_register_builtin_rendering_engines()`
|
||||
- Verify plugin class inheritance from `RenderingEnginePlugin`
|
||||
- Check import paths and module availability
|
||||
|
||||
2. **Asset Loading Errors**:
|
||||
- Verify asset paths in `get_required_assets()`
|
||||
- Check file permissions and existence
|
||||
- Validate URL generation in different modes
|
||||
|
||||
3. **Configuration Errors**:
|
||||
- Check JSON syntax in configuration
|
||||
- Verify configuration element ID (`markitect-config`)
|
||||
- Test configuration loading in JavaScript
|
||||
|
||||
4. **Rendering Failures**:
|
||||
- Check template file existence and permissions
|
||||
- Verify template placeholder replacement
|
||||
- Test with minimal content for debugging
|
||||
|
||||
### Debug Techniques
|
||||
|
||||
- **Console Logging**: Use browser console for debugging
|
||||
- **Debug Panel**: TestDrive JSUI includes debug information
|
||||
- **Verbose Mode**: Use CLI `--verbose` flag for detailed output
|
||||
- **Test Scripts**: Run individual test scripts for isolation
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
|
||||
- **Plugin Package Manager**: npm-like plugin distribution
|
||||
- **Theme System Integration**: Plugin-aware theme system
|
||||
- **Performance Monitoring**: Built-in performance tracking
|
||||
- **Hot Reloading**: Automatic reload on file changes
|
||||
|
||||
### Extension Opportunities
|
||||
|
||||
- **React Integration**: React-based rendering engine
|
||||
- **Vue Integration**: Vue.js-based rendering engine
|
||||
- **TypeScript Support**: TypeScript plugin development
|
||||
- **Testing Framework**: Automated JavaScript testing
|
||||
|
||||
### Community
|
||||
|
||||
- **Plugin Registry**: Central repository for community plugins
|
||||
- **Documentation**: Expanded examples and tutorials
|
||||
- **Templates**: Starter templates for new plugins
|
||||
- **Best Practices**: Community guidelines and patterns
|
||||
|
||||
---
|
||||
|
||||
*This plugin system enables JavaScript-first development while maintaining clean integration with the MarkiTect Python ecosystem, providing the best of both worlds for UI development and backend processing.*
|
||||
286
docs/PROJECT_STRUCTURE.md
Normal file
286
docs/PROJECT_STRUCTURE.md
Normal file
@@ -0,0 +1,286 @@
|
||||
# MarkiTect Project Structure
|
||||
|
||||
This document describes the current project layout, architectural decisions, and the reorganization plan for the Information Space Service evolution.
|
||||
|
||||
## Overview
|
||||
|
||||
MarkiTect is a markdown processing toolkit with transclusion, schema validation, asset management, and multi-format output capabilities. The project follows a hybrid layout that is being incrementally consolidated.
|
||||
|
||||
## Current Directory Structure
|
||||
|
||||
```
|
||||
markitect-main/
|
||||
├── markitect/ # Main package
|
||||
│ ├── [34 root-level .py files] # Core functionality (see below)
|
||||
│ ├── assets/ # Asset discovery, management, caching (21 files)
|
||||
│ ├── finance/ # Cost tracking, work time management (9 files)
|
||||
│ ├── plugins/ # Plugin system with base classes (7 files)
|
||||
│ ├── packaging/ # Asset packaging, MDZ variants (7 files)
|
||||
│ ├── production/ # Deployment validation, benchmarks (6 files)
|
||||
│ ├── legacy/ # Legacy compatibility layer (8 files)
|
||||
│ ├── explode_variants/ # Document expansion, variants (9 files)
|
||||
│ ├── query_paradigms/ # Query paradigm implementations (4 files)
|
||||
│ ├── validators/ # Content/link/section validation (4 files)
|
||||
│ ├── matter_frontmatter/ # Front matter parsing (4 files)
|
||||
│ ├── matter_contentmatter/ # Content matter parsing (4 files)
|
||||
│ ├── matter_tailmatter/ # Tail matter parsing (4 files)
|
||||
│ ├── profile/ # User profile management (4 files)
|
||||
│ ├── graphql/ # GraphQL query implementation (4 files)
|
||||
│ ├── template/ # Template management (3 files)
|
||||
│ ├── themes/ # Theme system with subdirectories (1 file)
|
||||
│ └── schemas/ # Built-in schema definitions (9 files)
|
||||
├── application/ # Application layer services
|
||||
├── domain/ # Domain models
|
||||
├── infrastructure/ # Infrastructure implementations
|
||||
├── tests/ # Test suite (90+ test files)
|
||||
│ ├── unit/ # Unit tests
|
||||
│ ├── integration/ # Integration tests
|
||||
│ ├── e2e/ # End-to-end tests
|
||||
│ └── fixtures/ # Test data
|
||||
├── docs/ # Documentation (12+ subdirectories)
|
||||
├── src/ # JavaScript/frontend components
|
||||
└── roadmap/ # Project roadmap
|
||||
```
|
||||
|
||||
## Root-Level Modules (/markitect/)
|
||||
|
||||
The 34 root-level Python files are organized by function:
|
||||
|
||||
### Core Infrastructure
|
||||
| File | Lines | Purpose |
|
||||
|------|-------|---------|
|
||||
| `parser.py` | ~50 | Markdown AST parsing using markdown-it |
|
||||
| `serializer.py` | ~360 | AST serialization back to Markdown |
|
||||
| `document_manager.py` | ~100 | Wrapper around CleanDocumentManager |
|
||||
| `clean_document_manager.py` | ~2000 | Clean document management implementation |
|
||||
| `workspace.py` | ~200 | Workspace management |
|
||||
| `database.py` | ~400 | SQLite database management |
|
||||
|
||||
### Schema Management (6 files, 99KB total)
|
||||
| File | Lines | Purpose |
|
||||
|------|-------|---------|
|
||||
| `schema_generator.py` | ~600 | JSON schema generation from markdown AST |
|
||||
| `schema_analyzer.py` | ~450 | Schema rigidity analysis with phase classification |
|
||||
| `schema_loader.py` | ~600 | Schema loading from markdown with frontmatter |
|
||||
| `schema_refiner.py` | ~600 | Automatic schema refinement using loosening rules |
|
||||
| `schema_validator.py` | ~900 | Comprehensive schema validation |
|
||||
| `schema_naming.py` | ~300 | Schema naming convention enforcement |
|
||||
|
||||
### Configuration & Services
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `config_manager.py` | Configuration file management |
|
||||
| `frontmatter.py` | YAML frontmatter parsing |
|
||||
| `exceptions.py` | Custom exception classes |
|
||||
| `ast_service.py` | AST service layer |
|
||||
| `cache_service.py` | Caching functionality |
|
||||
| `ast_cache.py` | AST caching implementation |
|
||||
| `performance_tracker.py` | Performance metrics |
|
||||
|
||||
### Validation & Analysis
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `semantic_validator.py` | Semantic validation layer |
|
||||
| `validation_error.py` | Validation error handling |
|
||||
| `metaschema.py` | Metaschema validation for custom extensions |
|
||||
|
||||
### CLI & Commands
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `cli.py` | Main CLI interface (274KB, comprehensive) |
|
||||
| `cli_utils.py` | CLI utilities |
|
||||
| `asset_commands.py` | Asset-related CLI commands |
|
||||
| `draft_generator.py` | Draft generation functionality |
|
||||
|
||||
### Utilities
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `batch_processor.py` | Batch processing operations |
|
||||
| `associated_files.py` | Associated file tracking |
|
||||
| `legacy_compat.py` | Legacy compatibility layer |
|
||||
| `legacy_integration_example.py` | Integration examples |
|
||||
| `_version.py`, `__version__.py` | Version management |
|
||||
|
||||
## Subpackages
|
||||
|
||||
### assets/ (21 files)
|
||||
Complete asset management system including discovery, analytics, caching, deduplication, and packaging. Key files:
|
||||
- `repository.py` - Asset repository pattern
|
||||
- `discovery.py` - Asset discovery algorithms
|
||||
- `cache.py` - Asset caching layer
|
||||
- `analytics.py` - Asset usage analytics
|
||||
|
||||
### finance/ (9 files)
|
||||
Cost tracking and work time management:
|
||||
- `models.py` - Financial data models
|
||||
- `cost_tracker.py` - Cost tracking implementation
|
||||
- `period_tracker.py` - Period-based tracking
|
||||
- `report_generator.py` - Financial reports
|
||||
|
||||
### plugins/ (7 files)
|
||||
Extensible plugin system:
|
||||
- `base.py` - Plugin base classes and types
|
||||
- `registry.py` - Plugin registry
|
||||
- `builtin/` - Built-in plugin implementations
|
||||
|
||||
### packaging/ (7 files)
|
||||
Asset packaging and MDZ format support:
|
||||
- `mdz_packager.py` - MDZ package creation
|
||||
- `transclusion.py` - Transclusion handling
|
||||
- `variant_factory.py` - Variant generation
|
||||
|
||||
### production/ (6 files)
|
||||
Deployment and production validation:
|
||||
- `deployment_validator.py` - Deployment checks
|
||||
- `performance_benchmark.py` - Performance testing
|
||||
- `cross_platform_validator.py` - Platform compatibility
|
||||
|
||||
### legacy/ (8 files)
|
||||
Backward compatibility layer:
|
||||
- `compatibility.py` - Compatibility wrappers
|
||||
- `deprecation.py` - Deprecation warnings
|
||||
- `git_tracker.py` - Git integration (useful for Phase 8)
|
||||
|
||||
## Test Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── conftest.py # Shared pytest configuration
|
||||
├── fixtures/ # Test data files
|
||||
│ ├── content_test_files/
|
||||
│ ├── contentmatter_test_files/
|
||||
│ ├── frontmatter_test_files/
|
||||
│ └── tailmatter_test_files/
|
||||
├── unit/ # Unit tests by domain
|
||||
│ ├── application/
|
||||
│ └── infrastructure/
|
||||
├── integration/ # Integration tests
|
||||
│ └── repositories/
|
||||
└── e2e/ # End-to-end tests
|
||||
├── cli/
|
||||
└── performance/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Planned Reorganization
|
||||
|
||||
### Motivation
|
||||
|
||||
The current layout has grown organically, resulting in:
|
||||
1. **34 files at root level** - Too many modules at package root
|
||||
2. **No clear grouping** - Schema tools, core infrastructure, and utilities mixed
|
||||
3. **Hybrid architecture** - Mix of root packages and monolithic /markitect/
|
||||
|
||||
### Target Structure
|
||||
|
||||
After reorganization, the /markitect/ package will have clearer structure:
|
||||
|
||||
```
|
||||
markitect/
|
||||
├── core/ # Core infrastructure (NEW)
|
||||
│ ├── __init__.py
|
||||
│ ├── parser.py # (from markitect/)
|
||||
│ ├── serializer.py # (from markitect/)
|
||||
│ ├── document_manager.py # (from markitect/)
|
||||
│ └── workspace.py # (from markitect/)
|
||||
├── schema/ # Schema management (NEW)
|
||||
│ ├── __init__.py
|
||||
│ ├── validator.py # (from schema_validator.py)
|
||||
│ ├── generator.py # (from schema_generator.py)
|
||||
│ ├── loader.py # (from schema_loader.py)
|
||||
│ ├── analyzer.py # (from schema_analyzer.py)
|
||||
│ ├── refiner.py # (from schema_refiner.py)
|
||||
│ └── naming.py # (from schema_naming.py)
|
||||
├── storage/ # Storage concerns (NEW)
|
||||
│ ├── __init__.py
|
||||
│ ├── database.py # (from markitect/)
|
||||
│ └── cache.py # (consolidated)
|
||||
├── spaces/ # Information spaces (Phase 1+)
|
||||
│ ├── models.py
|
||||
│ ├── events/
|
||||
│ ├── repositories/
|
||||
│ ├── transclusion/
|
||||
│ ├── rendering/
|
||||
│ ├── sync/
|
||||
│ └── services/
|
||||
└── [existing subpackages] # assets/, plugins/, etc.
|
||||
```
|
||||
|
||||
### Backward Compatibility
|
||||
|
||||
Original import paths will continue to work through re-exports:
|
||||
|
||||
```python
|
||||
# Old import (still works)
|
||||
from markitect.parser import parse_markdown
|
||||
|
||||
# New import (preferred)
|
||||
from markitect.core.parser import parse_markdown
|
||||
```
|
||||
|
||||
### Migration Strategy
|
||||
|
||||
1. Create new subpackages with copied content
|
||||
2. Update internal imports to new paths
|
||||
3. Add deprecation warnings to old paths
|
||||
4. Re-export from original locations for compatibility
|
||||
5. Verify all tests pass
|
||||
6. Update documentation
|
||||
|
||||
---
|
||||
|
||||
## Information Space Service Architecture
|
||||
|
||||
The reorganization prepares for the Information Space Service evolution, which adds:
|
||||
|
||||
### Phase 1-3: Foundation
|
||||
- `InformationSpace` entity with lifecycle management
|
||||
- `SpaceRepository` for persistence
|
||||
- Event system for change tracking
|
||||
- Persistent transclusion context
|
||||
|
||||
### Phase 4-5: Modes
|
||||
- HTML rendering mode with caching
|
||||
- Directory mode with bidirectional sync
|
||||
|
||||
### Phase 6-7: API & Composability
|
||||
- GraphQL schema extensions
|
||||
- CLI commands for space management
|
||||
- Space references and inheritance
|
||||
|
||||
### Phase 8: Git History (Optional)
|
||||
- Git-based version control for spaces
|
||||
- Event-driven commits
|
||||
- Version navigation
|
||||
|
||||
See [docs/roadmap/information-space-service/](./roadmap/information-space-service/) for the complete workplan.
|
||||
|
||||
---
|
||||
|
||||
## Key Dependencies
|
||||
|
||||
From `pyproject.toml`:
|
||||
- Python >=3.8 (tested on 3.12)
|
||||
- markdown-it-py - Markdown parsing
|
||||
- PyYAML - YAML/frontmatter handling
|
||||
- click - CLI framework
|
||||
- tabulate - Table formatting
|
||||
- jsonpath-ng - JSON path queries
|
||||
- aiohttp - Async HTTP
|
||||
|
||||
## Version Information
|
||||
|
||||
- Current version is managed in `_version.py` and `__version__.py`
|
||||
- Follows semantic versioning
|
||||
- CHANGELOG.md tracks all changes
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [CLI Tutorial](CLI_TUTORIAL.md) - CLI usage guide
|
||||
- [Plugin System](PLUGIN_SYSTEM.md) - Plugin architecture
|
||||
- [Schema Management Guide](SCHEMA_MANAGEMENT_GUIDE.md) - Schema workflows
|
||||
- [Asset Management Guide](ASSET_MANAGEMENT_USER_GUIDE.md) - Asset system
|
||||
- [Error Handling Strategy](ERROR_HANDLING_STRATEGY.md) - Error patterns
|
||||
@@ -7,8 +7,9 @@ Welcome to the MarkiTect documentation. This directory contains comprehensive do
|
||||
### 📐 Architecture Documentation (`architecture/`)
|
||||
Deep technical documentation about system design, performance, and implementation details.
|
||||
|
||||
- **[Capabilities Architecture](architecture/CAPABILITIES_ARCHITECTURE.md)** - **Critical:** How capabilities work as independent git submodules and separation of concerns
|
||||
- **[Caching System](architecture/caching-system.md)** - Why and how MarkiTect's AST caching delivers 60-85% performance improvements
|
||||
- *Coming soon: Database Schema, CLI Architecture, Plugin System*
|
||||
- *Coming soon: Database Schema, CLI Architecture*
|
||||
|
||||
### 👥 User Guides (`user-guides/`)
|
||||
End-user documentation for working with MarkiTect CLI and features.
|
||||
|
||||
548
docs/SCHEMA_MANAGEMENT_GUIDE.md
Normal file
548
docs/SCHEMA_MANAGEMENT_GUIDE.md
Normal file
@@ -0,0 +1,548 @@
|
||||
# Schema Management Guide
|
||||
|
||||
Complete guide to managing schemas in MarkiTect using the Schema-of-Schemas system.
|
||||
|
||||
## Overview
|
||||
|
||||
MarkiTect provides a comprehensive schema management system with:
|
||||
- Markdown-first schema format with embedded JSON
|
||||
- Strict naming conventions for consistency
|
||||
- Metaschema validation for all schemas
|
||||
- Multi-schema batch validation
|
||||
- Schema registry with version tracking
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Create a New Schema
|
||||
|
||||
Create a markdown file following the naming convention: `{domain}-schema-v{major}.{minor}.md`
|
||||
|
||||
```bash
|
||||
# Example: blog-post-schema-v1.0.md
|
||||
```
|
||||
|
||||
**Template:**
|
||||
|
||||
```markdown
|
||||
---
|
||||
schema-id: https://markitect.dev/schemas/blog-post/v1.0
|
||||
version: 1.0.0
|
||||
status: stable
|
||||
domain: blog-post
|
||||
description: Schema for blog post documents
|
||||
---
|
||||
|
||||
# Blog Post Schema v1.0.0
|
||||
|
||||
## Overview
|
||||
This schema validates blog post documents with frontmatter and content sections.
|
||||
|
||||
## Schema Definition
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"$id": "https://markitect.dev/schemas/blog-post/v1.0",
|
||||
"title": "Blog Post Schema",
|
||||
"description": "Schema for blog post documents",
|
||||
"version": "1.0.0",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"title": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"author": {
|
||||
"type": "string"
|
||||
},
|
||||
"date": {
|
||||
"type": "string",
|
||||
"format": "date"
|
||||
}
|
||||
},
|
||||
"required": ["title", "author"]
|
||||
}
|
||||
```
|
||||
\`\`\`
|
||||
|
||||
### 2. Validate Your Schema
|
||||
|
||||
Validate against the metaschema to ensure it follows MarkiTect conventions:
|
||||
|
||||
```bash
|
||||
# Validate a single schema file
|
||||
markitect schema-validate ./blog-post-schema-v1.0.md
|
||||
|
||||
# See detailed errors
|
||||
markitect schema-validate ./blog-post-schema-v1.0.md --detailed-errors
|
||||
```
|
||||
|
||||
### 3. Ingest into Registry
|
||||
|
||||
Add your schema to the registry:
|
||||
|
||||
```bash
|
||||
markitect schema-ingest blog-post-schema-v1.0.md
|
||||
```
|
||||
|
||||
### 4. List Registered Schemas
|
||||
|
||||
View all schemas with numbered references:
|
||||
|
||||
```bash
|
||||
# Simple format (default)
|
||||
markitect schema-list
|
||||
|
||||
# Table format
|
||||
markitect schema-list --format table
|
||||
|
||||
# JSON format
|
||||
markitect schema-list --format json
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
Found 4 schema(s):
|
||||
|
||||
[1] 🔧 blog-post-schema-v1.0.md (added: 2026-01-05T10:30:00)
|
||||
[2] 🔧 schema-schema-v1.0.md (added: 2026-01-05T03:33:42)
|
||||
[3] 🔧 manpage-schema-v1.0.md (added: 2026-01-05T03:33:42)
|
||||
[4] 🔧 api-documentation-schema-v1.0.md (added: 2026-01-05T03:33:35)
|
||||
```
|
||||
|
||||
## Schema Validation
|
||||
|
||||
### Single Schema Validation
|
||||
|
||||
**By number:**
|
||||
```bash
|
||||
markitect schema-validate 1
|
||||
```
|
||||
|
||||
**By filename (from registry):**
|
||||
```bash
|
||||
markitect schema-validate blog-post-schema-v1.0.md
|
||||
```
|
||||
|
||||
**By filesystem path:**
|
||||
```bash
|
||||
markitect schema-validate ./my-schema.md
|
||||
```
|
||||
|
||||
### Batch Validation
|
||||
|
||||
**Validate a range:**
|
||||
```bash
|
||||
markitect schema-validate 1-3
|
||||
```
|
||||
|
||||
**Validate specific schemas:**
|
||||
```bash
|
||||
markitect schema-validate 1,3,5
|
||||
```
|
||||
|
||||
**Validate all schemas:**
|
||||
```bash
|
||||
markitect schema-validate --all
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
Validating 4 schema(s)...
|
||||
|
||||
Results:
|
||||
|
||||
# Schema Status Details
|
||||
--- -------------------------------- -------- ---------
|
||||
1 blog-post-schema-v1.0.md ✅ Valid v1.0.0
|
||||
2 schema-schema-v1.0.md ✅ Valid v1.0.0
|
||||
3 manpage-schema-v1.0.md ✅ Valid v1.0.0
|
||||
4 api-documentation-schema-v1.0.md ✅ Valid v1.0.0
|
||||
|
||||
Summary: 4 valid, 0 failed
|
||||
```
|
||||
|
||||
## Document Validation (Semantic)
|
||||
|
||||
### Validate Documents Against Schemas
|
||||
|
||||
Beyond validating schema structure, MarkiTect can validate actual markdown documents against schemas, checking both structural (AST) and semantic (x-markitect extensions) aspects.
|
||||
|
||||
**Validate a document:**
|
||||
|
||||
```bash
|
||||
# Full validation (structural + semantic)
|
||||
markitect validate my-document.md --schema manpage-schema-v1.0.md
|
||||
|
||||
# Only structural validation (classic mode)
|
||||
markitect validate my-document.md --schema schema.json --no-semantic
|
||||
|
||||
# With external link checking (may be slow)
|
||||
markitect validate my-document.md --schema manpage-schema-v1.0.md --check-links
|
||||
|
||||
# Strict mode (warnings become errors)
|
||||
markitect validate my-document.md --schema manpage-schema-v1.0.md --strict
|
||||
```
|
||||
|
||||
### What is Validated
|
||||
|
||||
**Structural Validation** (always enabled):
|
||||
- Document AST structure matches JSON Schema properties
|
||||
- Heading counts, paragraph counts, code block counts
|
||||
- Element types and nesting
|
||||
|
||||
**Semantic Validation** (enabled by default with --semantic):
|
||||
- **Section Classifications**: Checks that documents have required sections, don't have improper sections
|
||||
- REQUIRED sections must be present (ERROR if missing)
|
||||
- RECOMMENDED sections should be present (WARNING if missing)
|
||||
- IMPROPER sections must not be present (ERROR if found)
|
||||
- DISCOURAGED sections should not be present (WARNING if found)
|
||||
- OPTIONAL sections may or may not be present (no check)
|
||||
- **Content Patterns**: Validates content matches regex patterns
|
||||
- `required_patterns`: Content must match (ERROR if missing)
|
||||
- `forbidden_patterns`: Content must not match (ERROR if found)
|
||||
- `discouraged_patterns`: Content should not match (WARNING if found)
|
||||
- **Quality Metrics**: Checks word counts, sentence counts
|
||||
- `min_words`, `max_words`: Word count requirements (WARNING)
|
||||
- `min_sentences`: Minimum sentence count (WARNING)
|
||||
- **Link Validation**: Validates internal and external links (optional)
|
||||
- Internal links: Checked by default when semantic validation enabled
|
||||
- Fragment links (#section-name) verified to exist (ERROR if broken)
|
||||
- Relative file paths checked for existence (ERROR if broken)
|
||||
- External links: Opt-in with --check-links flag (may be slow)
|
||||
- HTTP/HTTPS URLs validated with HEAD requests (WARNING if broken)
|
||||
- Email validation: Validates mailto: link format (WARNING if invalid)
|
||||
- Fragment policy: Configurable allow/disallow fragment identifiers
|
||||
|
||||
### Validation Output
|
||||
|
||||
```
|
||||
Validation result: VALID
|
||||
File: my-command.1.md
|
||||
Schema: schema file: manpage-schema-v1.0.md
|
||||
✅ Document structure matches schema requirements
|
||||
|
||||
============================================================
|
||||
Semantic Validation Results:
|
||||
============================================================
|
||||
Section Validation:
|
||||
✅ SYNOPSIS - Present (required)
|
||||
✅ DESCRIPTION - Present (required)
|
||||
✅ EXAMPLES - Present (recommended)
|
||||
|
||||
Content Validation:
|
||||
✅ All content requirements met
|
||||
|
||||
Link Validation:
|
||||
✅ All 12 links valid
|
||||
|
||||
Summary:
|
||||
Sections checked: 3
|
||||
Sections found: 5
|
||||
Errors: 0
|
||||
Warnings: 0
|
||||
Status: PASSED ✅
|
||||
```
|
||||
|
||||
### Common Validation Scenarios
|
||||
|
||||
**Example 1: Missing Required Section**
|
||||
```bash
|
||||
$ markitect validate doc.md --schema manpage-schema-v1.0.md
|
||||
❌ Document validation failed
|
||||
|
||||
Section Validation:
|
||||
❌ SYNOPSIS - SYNOPSIS section is mandatory
|
||||
✅ DESCRIPTION - Present (required)
|
||||
|
||||
Errors: 1
|
||||
Status: FAILED ❌
|
||||
```
|
||||
|
||||
**Example 2: Forbidden Pattern Found**
|
||||
```bash
|
||||
$ markitect validate doc.md --schema manpage-schema-v1.0.md
|
||||
|
||||
Content Validation:
|
||||
❌ SYNOPSIS - Forbidden pattern found: 'TODO'
|
||||
|
||||
Errors: 1
|
||||
Status: FAILED ❌
|
||||
```
|
||||
|
||||
**Example 3: Content Too Short (Warning)**
|
||||
```bash
|
||||
$ markitect validate doc.md --schema manpage-schema-v1.0.md
|
||||
|
||||
Content Validation:
|
||||
⚠️ DESCRIPTION - Content too short (25 words, minimum 50)
|
||||
|
||||
Warnings: 1
|
||||
Status: PASSED ✅
|
||||
|
||||
# With --strict flag, this would fail:
|
||||
$ markitect validate doc.md --schema manpage-schema-v1.0.md --strict
|
||||
Status: FAILED ❌ (warnings treated as errors)
|
||||
```
|
||||
|
||||
**Example 4: Broken Internal Link**
|
||||
```bash
|
||||
$ markitect validate doc.md --schema manpage-schema-v1.0.md
|
||||
|
||||
Link Validation:
|
||||
❌ #nonexistent-section - Internal link target not found: #nonexistent-section
|
||||
|
||||
Errors: 1
|
||||
Status: FAILED ❌
|
||||
```
|
||||
|
||||
**Example 5: External Link Validation**
|
||||
```bash
|
||||
# Enable external link checking (may be slow)
|
||||
$ markitect validate doc.md --schema manpage-schema-v1.0.md --check-links
|
||||
|
||||
Link Validation:
|
||||
✅ http://example.com - Valid
|
||||
⚠️ http://broken-link.invalid - External link unreachable: Name or service not known
|
||||
|
||||
Warnings: 1
|
||||
Status: PASSED ✅
|
||||
```
|
||||
|
||||
## Schema Naming Conventions
|
||||
|
||||
All schema filenames must follow this pattern:
|
||||
|
||||
```
|
||||
{domain}-schema-v{major}.{minor}.md
|
||||
```
|
||||
|
||||
### Rules
|
||||
|
||||
- **Domain**: Lowercase letters, numbers, and hyphens only
|
||||
- **Version**: Major.minor format (e.g., `v1.0`, `v2.3`)
|
||||
- **Extension**: Must be `.md`
|
||||
- **No spaces**: Use hyphens for separation
|
||||
|
||||
### Valid Examples
|
||||
|
||||
- `blog-post-schema-v1.0.md`
|
||||
- `api-documentation-schema-v2.1.md`
|
||||
- `user-profile-schema-v1.0.md`
|
||||
|
||||
### Invalid Examples
|
||||
|
||||
- `BlogPost-schema-v1.0.md` (uppercase)
|
||||
- `blog_post-schema-v1.0.md` (underscore)
|
||||
- `blog-post-v1.0.md` (missing "schema")
|
||||
- `blog-post-schema-v1.md` (missing minor version)
|
||||
|
||||
## Required Schema Fields
|
||||
|
||||
All schemas must include these fields:
|
||||
|
||||
### Frontmatter (YAML)
|
||||
```yaml
|
||||
---
|
||||
schema-id: https://markitect.dev/schemas/{domain}/v{major}.{minor}
|
||||
version: {major}.{minor}.{patch}
|
||||
status: draft|stable|deprecated
|
||||
domain: {domain}
|
||||
description: Brief description
|
||||
---
|
||||
```
|
||||
|
||||
### JSON Schema
|
||||
```json
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"$id": "https://markitect.dev/schemas/{domain}/v{major}.{minor}",
|
||||
"title": "Schema Title",
|
||||
"description": "Schema description",
|
||||
"version": "{major}.{minor}.{patch}"
|
||||
}
|
||||
```
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Revalidate All Schemas After Metaschema Changes
|
||||
|
||||
When you update the metaschema, revalidate all registered schemas:
|
||||
|
||||
```bash
|
||||
markitect schema-validate --all
|
||||
```
|
||||
|
||||
### Check Schema Rigidity
|
||||
|
||||
Analyze a schema for overly rigid constraints:
|
||||
|
||||
```bash
|
||||
markitect schema-analyze my-schema.md
|
||||
```
|
||||
|
||||
### Refine a Rigid Schema
|
||||
|
||||
Automatically loosen overly specific constraints:
|
||||
|
||||
```bash
|
||||
# Dry run (preview changes)
|
||||
markitect schema-refine my-schema.md --dry-run
|
||||
|
||||
# Apply changes
|
||||
markitect schema-refine my-schema.md
|
||||
|
||||
# Interactive mode
|
||||
markitect schema-refine my-schema.md --interactive
|
||||
```
|
||||
|
||||
### Get Schema Details
|
||||
|
||||
View schema metadata:
|
||||
|
||||
```bash
|
||||
markitect schema-get blog-post-schema-v1.0.md
|
||||
```
|
||||
|
||||
### Delete a Schema
|
||||
|
||||
Remove a schema from the registry:
|
||||
|
||||
```bash
|
||||
markitect schema-delete blog-post-schema-v1.0.md --confirm
|
||||
```
|
||||
|
||||
## Resolution Precedence
|
||||
|
||||
When validating schemas, MarkiTect uses this resolution order:
|
||||
|
||||
1. **Registry (by filename)**: Exact match in the database
|
||||
2. **Filesystem (fallback)**: If not found in registry or looks like a path
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
# Looks up in registry first
|
||||
markitect schema-validate blog-post-schema-v1.0.md
|
||||
|
||||
# Forces filesystem lookup (contains /)
|
||||
markitect schema-validate ./blog-post-schema-v1.0.md
|
||||
|
||||
# Also forces filesystem
|
||||
markitect schema-validate ../schemas/blog-post-schema-v1.0.md
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Schema Development
|
||||
|
||||
1. **Start with a template**: Use an existing schema as a starting point
|
||||
2. **Validate early**: Validate against the metaschema before ingesting
|
||||
3. **Use semantic versioning**: Major.minor.patch for all versions
|
||||
4. **Document thoroughly**: Include overview, usage, and examples
|
||||
5. **Test with real documents**: Validate actual documents against your schema
|
||||
|
||||
### Version Management
|
||||
|
||||
- **Increment major version**: Breaking changes to schema structure
|
||||
- **Increment minor version**: Backward-compatible additions
|
||||
- **Increment patch version**: Bug fixes and clarifications
|
||||
|
||||
### Schema Organization
|
||||
|
||||
```
|
||||
markitect/schemas/
|
||||
├── schema-schema-v1.0.md # Metaschema
|
||||
├── manpage-schema-v1.0.md # Man page documents
|
||||
├── api-documentation-schema-v1.0.md
|
||||
├── terminology-schema-v1.0.md
|
||||
└── blog-post-schema-v1.0.md # Your schemas
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Schema Not Found
|
||||
|
||||
```
|
||||
❌ Schema 'my-schema.md' not found in registry or filesystem
|
||||
```
|
||||
|
||||
**Solution:** Use `markitect schema-list` to see available schemas, or provide a path: `./my-schema.md`
|
||||
|
||||
### Validation Fails
|
||||
|
||||
```
|
||||
❌ Schema validation failed: my-schema.md
|
||||
Found 2 validation error(s):
|
||||
```
|
||||
|
||||
**Solution:** Check error messages and compare with metaschema requirements. Use `--detailed-errors` for more context.
|
||||
|
||||
### Invalid Selector
|
||||
|
||||
```
|
||||
❌ Invalid selector: Range 1-10 is out of bounds. Valid range: 1-4
|
||||
```
|
||||
|
||||
**Solution:** Use `markitect schema-list` to see valid numbers, or check your range syntax.
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Scripting with Schema Commands
|
||||
|
||||
Validate schemas in CI/CD:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Validate all schemas and exit with error if any fail
|
||||
if ! markitect schema-validate --all; then
|
||||
echo "Schema validation failed!"
|
||||
exit 1
|
||||
fi
|
||||
echo "All schemas valid"
|
||||
```
|
||||
|
||||
### Batch Operations
|
||||
|
||||
```bash
|
||||
# Validate recently added schemas
|
||||
markitect schema-validate 1-3
|
||||
|
||||
# Validate specific critical schemas
|
||||
markitect schema-validate 1,5,8
|
||||
|
||||
# Check just the metaschema
|
||||
markitect schema-validate 2
|
||||
```
|
||||
|
||||
## Schema Extensions
|
||||
|
||||
MarkiTect supports custom extensions in schemas:
|
||||
|
||||
- `x-markitect-sections`: Section classification (required, recommended, optional, discouraged, improper)
|
||||
- `x-markitect-content-control`: Content validation rules and patterns
|
||||
- `x-markitect-metadata`: Additional metadata for MarkiTect processing
|
||||
|
||||
See existing schemas for examples of these extensions.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Planned features:
|
||||
- Wildcard/globbing support: `markitect schema-validate */manpage*`
|
||||
- Schema diff tool: Compare schema versions
|
||||
- Schema migration assistant: Help upgrade documents to new schema versions
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Schema Naming Specification](../history/260105-schema-of-schemas/SCHEMA_NAMING_SPEC.md)
|
||||
- [Schema Loader Guide](../history/260105-schema-of-schemas/SCHEMA_LOADER_GUIDE.md)
|
||||
- [Metaschema Reference](../markitect/schemas/schema-schema-v1.0.md)
|
||||
- [Implementation Workplan](../history/260105-schema-of-schemas/WORKPLAN.md) (archived)
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- Check existing schemas as examples
|
||||
- Review metaschema validation errors carefully
|
||||
- Use `--detailed-errors` for more context
|
||||
- Consult the metaschema for requirements
|
||||
1275
docs/WIDGET_PLUGIN_INFRASTRUCTURE_WORKPLAN.md
Normal file
1275
docs/WIDGET_PLUGIN_INFRASTRUCTURE_WORKPLAN.md
Normal file
File diff suppressed because it is too large
Load Diff
268
docs/WORKSPACE_AND_DATABASES.md
Normal file
268
docs/WORKSPACE_AND_DATABASES.md
Normal file
@@ -0,0 +1,268 @@
|
||||
# Markitect Workspace and Database Architecture
|
||||
|
||||
This document explains Markitect's workspace concept and the two distinct database systems used by the application.
|
||||
|
||||
## Workspace Concept
|
||||
|
||||
Markitect uses a **workspace-based architecture** where each directory or repository can have its own configuration and local data storage. This allows for flexible, per-project customization while maintaining a global user configuration.
|
||||
|
||||
### Workspace Structure
|
||||
|
||||
When you initialize Markitect in a directory, it creates the following structure:
|
||||
|
||||
```
|
||||
project-directory/
|
||||
├── .markitect.yml # Workspace configuration
|
||||
├── .markitect_workspace/ # Local workspace data
|
||||
├── .ast_cache/ # AST parsing cache
|
||||
├── assets/ # Asset storage directory
|
||||
│ ├── assets.db # Asset management database
|
||||
│ └── [asset files] # Stored images, files, etc.
|
||||
└── tests/ # Test files directory
|
||||
```
|
||||
|
||||
### Configuration Files
|
||||
|
||||
Markitect searches for configuration in this order:
|
||||
1. `.markitect.yml` (current directory)
|
||||
2. `.markitect.yaml` (current directory)
|
||||
3. `.markitect.json` (current directory)
|
||||
4. `markitect.config.yml` (current directory)
|
||||
5. `markitect.config.yaml` (current directory)
|
||||
6. `markitect.config.json` (current directory)
|
||||
7. `~/.markitect/config.yml` (user home directory)
|
||||
8. Environment variables (`MARKITECT_*`)
|
||||
9. Built-in defaults
|
||||
|
||||
## Database Architecture
|
||||
|
||||
Markitect uses two distinct SQLite databases for different purposes:
|
||||
|
||||
### 1. Main Application Database (`markitect.db`)
|
||||
|
||||
**Location**: `~/.markitect/markitect.db` (user home directory)
|
||||
|
||||
**Purpose**: Global user-level application data and configuration
|
||||
|
||||
**Scope**: User-wide, shared across all workspaces
|
||||
|
||||
**Contents**:
|
||||
- User preferences and settings
|
||||
- Application state information
|
||||
- Global configuration data
|
||||
- Cross-workspace data that needs persistence
|
||||
|
||||
**Configuration**: Set via `MARKITECT_DATABASE_PATH` environment variable or `database_path` in configuration
|
||||
|
||||
### 2. Asset Management Database (`assets.db`)
|
||||
|
||||
**Location**: `assets/assets.db` (within workspace asset storage directory)
|
||||
|
||||
**Purpose**: Asset management and tracking for the current workspace
|
||||
|
||||
**Scope**: Workspace-specific, local to each directory/repository
|
||||
|
||||
**Contents**:
|
||||
- Asset metadata (filename, size, MIME type, timestamps)
|
||||
- File content hashes for deduplication
|
||||
- Asset usage statistics and tracking
|
||||
- Processing logs and analytics
|
||||
- Asset relationships and dependencies
|
||||
|
||||
**Schema** (key tables):
|
||||
```sql
|
||||
-- Basic asset metadata
|
||||
asset_metadata (
|
||||
content_hash TEXT PRIMARY KEY,
|
||||
filename TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL,
|
||||
mime_type TEXT,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
)
|
||||
|
||||
-- Usage tracking
|
||||
asset_usage_stats (
|
||||
content_hash TEXT,
|
||||
usage_count INTEGER,
|
||||
last_used TIMESTAMP,
|
||||
documents_using TEXT -- JSON array of document paths
|
||||
)
|
||||
|
||||
-- Performance and analytics tables
|
||||
-- (Additional tables for caching, indexing, and optimization)
|
||||
```
|
||||
|
||||
## Why Two Databases?
|
||||
|
||||
This separation serves several important purposes:
|
||||
|
||||
### Data Isolation
|
||||
- **Global data** (user preferences) stays in the user profile
|
||||
- **Workspace data** (asset files, metadata) stays with the project
|
||||
|
||||
### Version Control Considerations
|
||||
- `markitect.db` is never committed to version control
|
||||
- `assets.db` is excluded via `.gitignore` (local workspace data)
|
||||
- Asset files themselves can be optionally committed based on project needs
|
||||
|
||||
### Performance Optimization
|
||||
- Asset database operations are localized to relevant files
|
||||
- Global database isn't impacted by large asset collections
|
||||
- Each workspace can optimize its asset database independently
|
||||
|
||||
### Portability and Collaboration
|
||||
- Workspaces can be moved/copied without affecting global configuration
|
||||
- Teams can share asset storage strategies without sharing personal settings
|
||||
- Different projects can have different asset management policies
|
||||
|
||||
## Configuration Management
|
||||
|
||||
### Workspace Initialization
|
||||
|
||||
To initialize a new workspace:
|
||||
|
||||
```bash
|
||||
markitect config-init
|
||||
```
|
||||
|
||||
This creates:
|
||||
1. `.markitect.yml` configuration file
|
||||
2. Required directories (`.markitect_workspace`, `.ast_cache`, `tests`)
|
||||
3. Asset storage structure
|
||||
|
||||
### Configuration Commands
|
||||
|
||||
```bash
|
||||
# View current configuration
|
||||
markitect config-show
|
||||
|
||||
# Set workspace-specific values
|
||||
markitect config-set repo_name "my-project"
|
||||
markitect config-set assets.storage_path "./assets"
|
||||
|
||||
# View configuration help
|
||||
markitect config-help
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Override configuration with environment variables:
|
||||
|
||||
```bash
|
||||
export MARKITECT_GITEA_URL="http://localhost:3000"
|
||||
export MARKITECT_WORKSPACE_DIR=".custom_workspace"
|
||||
export MARKITECT_DATABASE_PATH="/custom/path/markitect.db"
|
||||
```
|
||||
|
||||
## Asset Management Integration
|
||||
|
||||
The asset management system coordinates between the asset database and file storage:
|
||||
|
||||
```python
|
||||
from markitect.assets import AssetManager
|
||||
|
||||
# Initialize with workspace-specific configuration
|
||||
asset_manager = AssetManager()
|
||||
|
||||
# Assets are stored in workspace, tracked in assets.db
|
||||
asset = asset_manager.add_asset("image.png")
|
||||
```
|
||||
|
||||
### Asset Storage Workflow
|
||||
|
||||
1. **File Processing**: Asset files are processed and stored in `assets/` directory
|
||||
2. **Database Recording**: Metadata recorded in `assets.db`
|
||||
3. **Deduplication**: Content hashes prevent duplicate storage
|
||||
4. **Usage Tracking**: Document usage recorded for analytics
|
||||
5. **Cleanup**: Unused assets can be identified and removed
|
||||
|
||||
## Best Practices
|
||||
|
||||
### For Development
|
||||
- Initialize workspace in each project directory
|
||||
- Commit `.markitect.yml` to version control
|
||||
- Add `assets.db` and workspace directories to `.gitignore`
|
||||
- Use relative paths in workspace configuration
|
||||
|
||||
### For Collaboration
|
||||
- Share workspace configuration (`.markitect.yml`)
|
||||
- Document asset storage strategy for the team
|
||||
- Establish conventions for asset organization
|
||||
- Consider asset file version control policies
|
||||
|
||||
### for Production
|
||||
- Backup both databases separately
|
||||
- Monitor asset database growth in large projects
|
||||
- Use environment variables for deployment-specific settings
|
||||
- Implement asset cleanup strategies for long-running projects
|
||||
|
||||
## Migration and Compatibility
|
||||
|
||||
### Legacy Support
|
||||
The system maintains backward compatibility:
|
||||
- Old configuration patterns still work
|
||||
- Automatic migration of legacy settings
|
||||
- Graceful fallbacks for missing configuration
|
||||
|
||||
### Database Migration
|
||||
Asset databases support schema migrations:
|
||||
- Automatic schema updates on version changes
|
||||
- Backward compatibility preservation
|
||||
- Safe migration with rollback capability
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Database Connection Errors**:
|
||||
- Check file permissions on database directories
|
||||
- Verify disk space availability
|
||||
- Ensure SQLite3 is available
|
||||
|
||||
**Configuration Not Found**:
|
||||
- Verify `.markitect.yml` exists and is valid YAML
|
||||
- Check environment variable names and values
|
||||
- Run `markitect config-show` to see current configuration
|
||||
|
||||
**Asset Storage Issues**:
|
||||
- Confirm asset directory permissions
|
||||
- Check available disk space
|
||||
- Verify `assets.db` is not corrupted
|
||||
|
||||
### Recovery Procedures
|
||||
|
||||
**Corrupted Asset Database**:
|
||||
```bash
|
||||
# Backup and recreate
|
||||
mv assets/assets.db assets/assets.db.backup
|
||||
# Restart Markitect to recreate schema
|
||||
markitect config-show
|
||||
```
|
||||
|
||||
**Missing Configuration**:
|
||||
```bash
|
||||
# Reinitialize workspace
|
||||
markitect config-init
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Database Connections
|
||||
- Uses SQLite3 with connection pooling for performance
|
||||
- Automatic connection management and cleanup
|
||||
- Thread-safe operations for concurrent access
|
||||
|
||||
### File System Integration
|
||||
- Path resolution relative to workspace root
|
||||
- Cross-platform path handling (Windows, macOS, Linux)
|
||||
- Symlink and junction support where available
|
||||
|
||||
### Security Considerations
|
||||
- Database files have restricted permissions
|
||||
- No sensitive data stored in asset database
|
||||
- Configuration masking for sensitive values in display
|
||||
|
||||
---
|
||||
|
||||
*This documentation reflects the current architecture as of November 2025. For implementation details, see the source code in `markitect/config_manager.py` and `markitect/assets/`.*
|
||||
173
docs/adr/ADR-001-client-side-debug-storage.md
Normal file
173
docs/adr/ADR-001-client-side-debug-storage.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# ADR-001: Client-Side Debug Storage Technology
|
||||
|
||||
## Status
|
||||
**Accepted** - 2025-11-10
|
||||
|
||||
## Context
|
||||
|
||||
The Markitect application requires a robust client-side debugging infrastructure to track and display debug messages during document editing operations. The previous implementation relied on a tightly-coupled debug panel that expected specific DOM elements and was integrated into old dialog systems.
|
||||
|
||||
### Requirements
|
||||
- **Independence**: Debug system must be independent of specific UI components
|
||||
- **Persistence**: Debug messages should survive browser refresh/close
|
||||
- **Real-time Updates**: UI should reflect new messages immediately
|
||||
- **Performance**: Should not block the main thread or impact editor performance
|
||||
- **Storage Capacity**: Must handle 1000+ debug messages efficiently
|
||||
- **Browser Compatibility**: Work across all modern browsers
|
||||
- **Developer Experience**: Easy to integrate and use throughout the codebase
|
||||
|
||||
### Problem Statement
|
||||
The existing debug infrastructure was failing because:
|
||||
1. Tight coupling to specific DOM elements (`debug-messages-container`, `toggle-debug`)
|
||||
2. Dependency on old dialog systems
|
||||
3. No persistence across browser sessions
|
||||
4. Limited to in-memory storage only
|
||||
|
||||
## Decision
|
||||
|
||||
**We will use IndexedDB as the primary client-side storage technology for the debug system.**
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Option 1: IndexedDB (Selected)
|
||||
**Technology**: Browser-native object store database
|
||||
**Implementation**: `window.MarkitectDebugSystem` with async operations
|
||||
|
||||
### Option 2: SQLite via SQL.js
|
||||
**Technology**: WebAssembly-based SQLite implementation
|
||||
**Implementation**: SQL.js library with manual persistence
|
||||
|
||||
### Option 3: LocalStorage
|
||||
**Technology**: Browser key-value storage
|
||||
**Implementation**: JSON serialization with size limits
|
||||
|
||||
### Option 4: In-Memory Only
|
||||
**Technology**: JavaScript arrays with no persistence
|
||||
**Implementation**: Simple message array management
|
||||
|
||||
## Decision Matrix
|
||||
|
||||
| Criteria | IndexedDB | SQLite | LocalStorage | In-Memory |
|
||||
|----------|-----------|---------|--------------|-----------|
|
||||
| **Storage Capacity** | ✅ Large (50-100MB+) | ✅ Large | ❌ Limited (5-10MB) | ❌ Session only |
|
||||
| **Performance** | ✅ Async/Non-blocking | ⚠️ Can block UI | ✅ Fast | ✅ Fastest |
|
||||
| **Persistence** | ✅ Across sessions | ✅ Across sessions | ✅ Across sessions | ❌ Lost on refresh |
|
||||
| **Query Capabilities** | ⚠️ Limited | ✅ Full SQL | ❌ None | ❌ JavaScript only |
|
||||
| **Dependencies** | ✅ Native | ❌ 1.5MB WebAssembly | ✅ Native | ✅ Native |
|
||||
| **Browser Support** | ✅ Universal modern | ✅ Universal | ✅ Universal | ✅ Universal |
|
||||
| **API Complexity** | ⚠️ Moderate | ✅ Familiar SQL | ✅ Simple | ✅ Simple |
|
||||
| **Use Case Fit** | ✅ Perfect match | ⚠️ Overkill | ❌ Too limited | ❌ Insufficient |
|
||||
|
||||
## Rationale
|
||||
|
||||
### Why IndexedDB?
|
||||
|
||||
1. **Native Browser Support**: No external dependencies or WebAssembly files
|
||||
2. **Asynchronous Operations**: Won't block the UI thread during debug operations
|
||||
3. **Sufficient Storage**: 50-100MB+ capacity easily handles thousands of debug messages
|
||||
4. **Appropriate Complexity**: Matches our simple data model without over-engineering
|
||||
5. **Performance Optimized**: Browsers optimize IndexedDB for web applications
|
||||
6. **Security**: Runs within browser's security sandbox with proper origin isolation
|
||||
|
||||
### Why Not SQLite?
|
||||
|
||||
While SQLite offers powerful querying capabilities, it's overkill for our use case:
|
||||
|
||||
- **Complexity**: Our data model is simple (timestamp, category, message)
|
||||
- **Dependencies**: 1.5MB WebAssembly adds significant overhead
|
||||
- **Synchronous Risk**: Large operations can block the UI thread
|
||||
- **Manual Persistence**: Requires explicit save/load operations
|
||||
- **Over-Engineering**: We don't need JOINs, complex queries, or relational features
|
||||
|
||||
### Why Not LocalStorage?
|
||||
|
||||
LocalStorage is too limited:
|
||||
- **Size Constraints**: 5-10MB limit insufficient for extensive debug logs
|
||||
- **Synchronous API**: Blocks main thread during operations
|
||||
- **No Structured Data**: JSON serialization/deserialization overhead
|
||||
- **Browser Quotas**: Can be evicted under storage pressure
|
||||
|
||||
### Why Not In-Memory?
|
||||
|
||||
In-memory storage is insufficient:
|
||||
- **No Persistence**: Debug context lost on page refresh
|
||||
- **Memory Pressure**: Large debug logs consume RAM
|
||||
- **Development Workflow**: Debugging often requires page reloads
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Data Structure
|
||||
```javascript
|
||||
{
|
||||
id: auto_increment_id,
|
||||
message: "Debug message text",
|
||||
category: "INFO" | "WARNING" | "ERROR" | "SUCCESS" | "DEBUG",
|
||||
timestamp: "2025-11-10T23:35:53.123Z",
|
||||
displayTime: "11:35:53 PM"
|
||||
}
|
||||
```
|
||||
|
||||
### Database Schema
|
||||
- **Store Name**: `messages`
|
||||
- **Key Path**: `id` (auto-increment)
|
||||
- **Indexes**: `timestamp`, `category`
|
||||
- **Version**: 1
|
||||
|
||||
### API Design
|
||||
```javascript
|
||||
// Core operations
|
||||
await MarkitectDebugSystem.addMessage(message, category);
|
||||
await MarkitectDebugSystem.clearMessages();
|
||||
const messages = MarkitectDebugSystem.getMessages(category, limit);
|
||||
|
||||
// Advanced features
|
||||
const unsubscribe = MarkitectDebugSystem.subscribe(callback);
|
||||
const json = MarkitectDebugSystem.exportMessages();
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- ✅ **Zero Dependencies**: No external libraries required
|
||||
- ✅ **High Performance**: Non-blocking operations maintain UI responsiveness
|
||||
- ✅ **Persistent Debugging**: Debug context preserved across browser sessions
|
||||
- ✅ **Scalable Storage**: Handles thousands of debug messages efficiently
|
||||
- ✅ **Future-Proof**: Can add advanced features without architectural changes
|
||||
- ✅ **Developer-Friendly**: Simple API integration throughout codebase
|
||||
|
||||
### Negative
|
||||
- ⚠️ **Limited Querying**: Complex filtering requires custom JavaScript code
|
||||
- ⚠️ **Browser Variations**: Subtle differences in IndexedDB implementations
|
||||
- ⚠️ **Learning Curve**: Developers may be less familiar with IndexedDB vs SQL
|
||||
|
||||
### Mitigation Strategies
|
||||
- **Query Limitations**: Implement helper methods for common filtering operations
|
||||
- **Browser Compatibility**: Use feature detection with graceful fallbacks
|
||||
- **Developer Experience**: Provide clear API documentation and examples
|
||||
|
||||
## Future Considerations
|
||||
|
||||
### Potential Enhancements
|
||||
1. **Hybrid Approach**: Add optional SQLite mode for advanced users
|
||||
2. **Data Export**: Multiple export formats (JSON, CSV, SQL dumps)
|
||||
3. **Advanced Filtering**: Web Workers for complex query operations
|
||||
4. **Data Visualization**: Charts and analytics for debug patterns
|
||||
5. **Remote Sync**: Optional sync to development servers
|
||||
|
||||
### Migration Path
|
||||
The current IndexedDB implementation provides a foundation that could support:
|
||||
- Adding SQLite as an "advanced mode" option
|
||||
- Implementing data export to external analysis tools
|
||||
- Building analytics dashboards on top of the debug data
|
||||
|
||||
## References
|
||||
- [IndexedDB API Documentation](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API)
|
||||
- [SQL.js Documentation](https://sql.js.org/)
|
||||
- [Web Storage API Limitations](https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API/Using_the_Web_Storage_API#Storage_limits)
|
||||
|
||||
## Approval
|
||||
|
||||
**Decided by**: Claude Code Development Team
|
||||
**Date**: 2025-11-10
|
||||
**Context**: Independent debug system redesign
|
||||
**Next Review**: When complex querying requirements emerge
|
||||
384
docs/adr/ADR-002-robustness-principle-for-production-use.md
Normal file
384
docs/adr/ADR-002-robustness-principle-for-production-use.md
Normal file
@@ -0,0 +1,384 @@
|
||||
# ADR-002: Robustness Principle for Production Use
|
||||
|
||||
## Status
|
||||
**Accepted** - 2025-11-11
|
||||
|
||||
## Context
|
||||
|
||||
The Markitect application operates in unpredictable client-side environments where JavaScript execution can fail due to malicious input, network issues, browser inconsistencies, missing dependencies, or resource exhaustion. Traditional defensive programming approaches often result in cascading failures that crash entire UI components or leave the application in an unusable state.
|
||||
|
||||
### Requirements
|
||||
- **Fault Tolerance**: System must continue operating when individual components fail
|
||||
- **Security**: Protection against malicious input and injection attacks
|
||||
- **Resource Protection**: Prevention of DoS attacks through resource exhaustion
|
||||
- **Graceful Degradation**: Non-essential features should fail without breaking core functionality
|
||||
- **Error Containment**: Failures should be isolated and not cascade throughout the system
|
||||
- **User Experience**: Users should never see white screens or completely broken interfaces
|
||||
- **Developer Experience**: Clear error reporting and debugging capabilities
|
||||
|
||||
### Problem Statement
|
||||
The existing JavaScript codebase was vulnerable to:
|
||||
1. **Uncaught Exceptions**: Single errors could crash entire UI components
|
||||
2. **Input Validation Gaps**: Malicious or malformed input could break processing
|
||||
3. **Resource Exhaustion**: Large datasets could freeze the browser
|
||||
4. **Dependency Failures**: Missing libraries or features caused complete breakdowns
|
||||
5. **DOM Manipulation Risks**: Direct DOM access without safety checks
|
||||
6. **Cascading Failures**: One component failure affecting others
|
||||
|
||||
## Decision
|
||||
|
||||
**We will implement the Robustness Principle as a comprehensive defensive programming strategy with multiple layers of protection throughout the JavaScript codebase, balanced with Fail Fast behavior in development mode to prevent difficult diagnosis and cascading errors.**
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### Option 1: Robustness Principle (Selected)
|
||||
**Approach**: Multiple defensive layers with graceful degradation
|
||||
**Implementation**: Safe wrappers, input validation, error boundaries, resource limits
|
||||
|
||||
### Option 2: Try-Catch Everything
|
||||
**Approach**: Wrap all operations in try-catch blocks
|
||||
**Implementation**: Granular exception handling without systematic approach
|
||||
|
||||
### Option 3: Reactive Error Handling
|
||||
**Approach**: Error handling through reactive programming patterns
|
||||
**Implementation**: RxJS or similar libraries for error stream management
|
||||
|
||||
### Option 4: Minimal Validation
|
||||
**Approach**: Basic input checking with assumption of good data
|
||||
**Implementation**: Simple null checks and basic validation
|
||||
|
||||
## Decision Matrix
|
||||
|
||||
| Criteria | Robustness Principle | Try-Catch All | Reactive Patterns | Minimal Validation |
|
||||
|----------|---------------------|---------------|-------------------|-------------------|
|
||||
| **Fault Tolerance** | ✅ Comprehensive | ⚠️ Inconsistent | ✅ Good | ❌ Poor |
|
||||
| **Security Protection** | ✅ Multi-layered | ❌ Reactive only | ⚠️ Limited | ❌ Vulnerable |
|
||||
| **Resource Management** | ✅ Proactive limits | ❌ No protection | ⚠️ Some control | ❌ No protection |
|
||||
| **Code Maintainability** | ✅ Systematic | ❌ Scattered | ⚠️ Complex | ✅ Simple |
|
||||
| **Performance Impact** | ⚠️ Moderate overhead | ⚠️ High overhead | ❌ Library weight | ✅ Minimal |
|
||||
| **Developer Experience** | ✅ Clear patterns | ❌ Repetitive | ❌ Learning curve | ✅ Familiar |
|
||||
| **Error Recovery** | ✅ Graceful fallbacks | ⚠️ Manual recovery | ✅ Automatic retry | ❌ System failure |
|
||||
|
||||
## Balanced Implementation: Robustness + Fail Fast
|
||||
|
||||
### Development vs Production Behavior
|
||||
|
||||
**Development Mode (Fail Fast)**:
|
||||
- Immediate exceptions on errors for fast debugging
|
||||
- Strict validation with no silent failures
|
||||
- Full error context and stack traces
|
||||
- Activated on localhost, 127.0.0.1, or `?strict=true`
|
||||
|
||||
**Production Mode (Robust)**:
|
||||
- Graceful degradation and fallback behaviors
|
||||
- Silent recovery with detailed logging
|
||||
- User experience preservation
|
||||
- Default behavior in production environments
|
||||
|
||||
```javascript
|
||||
const MARKITECT_STRICT_MODE = (
|
||||
window.location.hostname === 'localhost' ||
|
||||
window.location.hostname === '127.0.0.1' ||
|
||||
window.location.search.includes('strict=true') ||
|
||||
window.markitectStrictMode === true
|
||||
);
|
||||
```
|
||||
|
||||
## Robustness Principle Implementation
|
||||
|
||||
### Layer 1: Input Validation & Sanitization
|
||||
**Purpose**: Prevent malicious or malformed data from entering the system
|
||||
|
||||
```javascript
|
||||
safeTextExtraction(element) {
|
||||
if (!this.validateElement(element)) {
|
||||
return '';
|
||||
}
|
||||
|
||||
try {
|
||||
const text = element.textContent || element.innerText || '';
|
||||
return this.sanitizeText(text.trim());
|
||||
} catch (error) {
|
||||
console.warn('Text extraction failed:', error);
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
sanitizeText(text) {
|
||||
if (typeof text !== 'string') return '';
|
||||
|
||||
const maxLength = 100000; // 100KB text limit
|
||||
return text
|
||||
.replace(/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/g, '') // Remove control chars
|
||||
.slice(0, maxLength); // Limit length
|
||||
}
|
||||
```
|
||||
|
||||
### Layer 2: Error Boundaries with Fallbacks
|
||||
**Purpose**: Contain failures and provide alternative execution paths
|
||||
|
||||
```javascript
|
||||
safeOperation(operation, fallback = null, context = 'Unknown') {
|
||||
try {
|
||||
return operation();
|
||||
} catch (error) {
|
||||
console.warn(`Operation failed in ${context}:`, error);
|
||||
|
||||
// Fail Fast in development mode
|
||||
if (MARKITECT_STRICT_MODE) {
|
||||
console.error(`🚨 STRICT MODE: Operation failed in ${context}`);
|
||||
throw error; // Re-throw for immediate debugging
|
||||
}
|
||||
|
||||
// Robust handling in production
|
||||
if (window.MarkitectDebugSystem) {
|
||||
window.MarkitectDebugSystem.addMessage(
|
||||
`Safe operation failed: ${error.message}`,
|
||||
'WARNING',
|
||||
'RobustnessSystem',
|
||||
{ context, eventType: 'ERROR' }
|
||||
);
|
||||
}
|
||||
|
||||
return typeof fallback === 'function' ? fallback() : fallback;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Layer 3: Resource Limits & Timeout Protection
|
||||
**Purpose**: Prevent resource exhaustion and infinite operations
|
||||
|
||||
```javascript
|
||||
// Element processing limits
|
||||
const elements = this.safeQuerySelectorAll(selector);
|
||||
const maxElements = 10000; // DoS protection
|
||||
elements.slice(0, maxElements).forEach(processElement);
|
||||
|
||||
// Operation timeouts
|
||||
const timeout = setTimeout(() => {
|
||||
if (this.isOperationRunning) {
|
||||
console.warn('Operation timed out');
|
||||
this.cleanup();
|
||||
}
|
||||
}, 30000); // 30 second safety timeout
|
||||
```
|
||||
|
||||
### Layer 4: Graceful Degradation
|
||||
**Purpose**: Maintain core functionality when non-essential features fail
|
||||
|
||||
```javascript
|
||||
// Dependency checking with fallbacks
|
||||
initializeControl(controlClass, controlName, icon = '🔧') {
|
||||
if (!controlClass) {
|
||||
this.safeLog(`${controlName} class not available, skipping`, 'WARNING');
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
const instance = new controlClass();
|
||||
return instance.createControl() ? instance : null;
|
||||
} catch (error) {
|
||||
// Create minimal fallback for essential controls
|
||||
if (controlName === 'StatusControl') {
|
||||
return this.createFallbackControl(controlName, icon);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Layer 5: Safe DOM Manipulation
|
||||
**Purpose**: Protect against DOM-related failures and validate operations
|
||||
|
||||
```javascript
|
||||
safeQuerySelector(selector, parent = document) {
|
||||
try {
|
||||
if (!parent || !parent.querySelector) {
|
||||
return null;
|
||||
}
|
||||
return parent.querySelector(selector);
|
||||
} catch (error) {
|
||||
console.warn(`Invalid selector: ${selector}`, error);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
validateElement(element) {
|
||||
return element &&
|
||||
element.nodeType === Node.ELEMENT_NODE &&
|
||||
element.isConnected &&
|
||||
!element.closest('.control-panel'); // Avoid control elements
|
||||
}
|
||||
```
|
||||
|
||||
## Rationale
|
||||
|
||||
### Why the Robustness Principle?
|
||||
|
||||
1. **Systematic Approach**: Unlike ad-hoc try-catch blocks, provides consistent protection patterns
|
||||
2. **Multiple Defense Layers**: Each layer catches different types of failures
|
||||
3. **Proactive Protection**: Prevents problems before they occur rather than just reacting
|
||||
4. **Maintainable Code**: Clear patterns and utility functions reduce repetition
|
||||
5. **Production Ready**: Designed for real-world environments with unpredictable conditions
|
||||
6. **Performance Conscious**: Adds protection without significant overhead
|
||||
|
||||
### Why Not Try-Catch Everything?
|
||||
|
||||
- **Maintenance Burden**: Scattered exception handling is hard to maintain
|
||||
- **Inconsistent Coverage**: Easy to miss critical paths
|
||||
- **Poor Error Recovery**: Just catching errors doesn't provide meaningful fallbacks
|
||||
- **Performance Impact**: Exception handling has overhead when overused
|
||||
|
||||
### Why Not Reactive Patterns?
|
||||
|
||||
- **Complexity**: RxJS adds significant learning curve and bundle size
|
||||
- **Overkill**: Our error handling needs don't require reactive streams
|
||||
- **Library Dependency**: Adds external dependency for core functionality
|
||||
- **Framework Lock-in**: Ties architecture to specific programming paradigm
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Core Protection Utilities
|
||||
|
||||
```javascript
|
||||
// Central error handling system
|
||||
const RobustnessSystem = {
|
||||
safeOperation(operation, fallback, context),
|
||||
safeQuerySelector(selector, parent),
|
||||
safeQuerySelectorAll(selector, parent),
|
||||
validateElement(element),
|
||||
sanitizeText(text),
|
||||
safeTextExtraction(element)
|
||||
};
|
||||
```
|
||||
|
||||
### Integration Pattern
|
||||
|
||||
```javascript
|
||||
// Before: Fragile operation
|
||||
function processDocument() {
|
||||
const stats = calculateStats(); // Could crash
|
||||
updateUI(stats); // Could crash
|
||||
saveToStorage(stats); // Could crash
|
||||
}
|
||||
|
||||
// After: Robust operation
|
||||
function processDocument() {
|
||||
const stats = this.safeOperation(
|
||||
() => this.calculateStats(),
|
||||
this.getDefaultStats(),
|
||||
'calculateStats'
|
||||
);
|
||||
|
||||
this.safeOperation(
|
||||
() => this.updateUI(stats),
|
||||
null,
|
||||
'updateUI'
|
||||
);
|
||||
|
||||
this.safeOperation(
|
||||
() => this.saveToStorage(stats),
|
||||
null,
|
||||
'saveToStorage'
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
### Resource Protection Examples
|
||||
|
||||
```javascript
|
||||
// Memory limits
|
||||
const characters = Math.min(sectionText.length, 1000000); // Cap at 1MB
|
||||
|
||||
// Processing limits
|
||||
elements.slice(0, maxElements).forEach(processElement);
|
||||
|
||||
// Time limits
|
||||
const timeout = setTimeout(cleanup, OPERATION_TIMEOUT);
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- ✅ **System Stability**: Individual component failures don't crash the entire application
|
||||
- ✅ **Security Hardening**: Multiple layers protect against various attack vectors
|
||||
- ✅ **User Experience**: Graceful degradation maintains usability during failures
|
||||
- ✅ **Developer Confidence**: Clear patterns reduce fear of production failures
|
||||
- ✅ **Debugging Capability**: Detailed error context and logging
|
||||
- ✅ **Maintenance Reduction**: Fewer emergency fixes for production issues
|
||||
|
||||
### Negative
|
||||
- ⚠️ **Performance Overhead**: Additional validation and error checking adds some cost
|
||||
- ⚠️ **Code Complexity**: More defensive code requires more careful implementation
|
||||
- ⚠️ **Initial Development Time**: Building robust systems takes longer upfront
|
||||
|
||||
### Mitigation Strategies
|
||||
- **Performance**: Use efficient validation techniques and avoid redundant checks
|
||||
- **Complexity**: Provide clear utility functions and documentation
|
||||
- **Development Time**: Treat as investment in reduced maintenance and debugging time
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Robustness Testing Categories
|
||||
|
||||
1. **Malicious Input Testing**: XSS attempts, oversized data, invalid formats
|
||||
2. **Resource Exhaustion Testing**: Large datasets, memory pressure scenarios
|
||||
3. **Dependency Failure Testing**: Missing libraries, network failures
|
||||
4. **DOM Manipulation Edge Cases**: Invalid selectors, disconnected elements
|
||||
5. **Timeout Scenarios**: Long-running operations, infinite loops
|
||||
6. **Error Cascade Testing**: Multiple simultaneous failures
|
||||
|
||||
### Automated Testing
|
||||
|
||||
```javascript
|
||||
// Example robustness test
|
||||
describe('Robustness Principle', () => {
|
||||
it('should handle malicious text input safely', () => {
|
||||
const maliciousText = '<script>alert("xss")</script>'.repeat(10000);
|
||||
const result = statusControl.safeTextExtraction({ textContent: maliciousText });
|
||||
|
||||
expect(result.length).toBeLessThan(100001); // Respects limits
|
||||
expect(result).not.toContain('<script>'); // Sanitized
|
||||
});
|
||||
|
||||
it('should gracefully handle missing dependencies', () => {
|
||||
delete window.StatusControl;
|
||||
const result = MarkitectMain.initialize();
|
||||
|
||||
expect(result).toBeDefined(); // Doesn't crash
|
||||
expect(window.statusControl).toBeNull(); // Graceful degradation
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## Future Considerations
|
||||
|
||||
### Potential Enhancements
|
||||
|
||||
1. **Metrics Collection**: Track robustness events for system health monitoring
|
||||
2. **Adaptive Thresholds**: Dynamic resource limits based on client capabilities
|
||||
3. **Recovery Strategies**: More sophisticated fallback mechanisms
|
||||
4. **Performance Monitoring**: Track overhead of robustness measures
|
||||
5. **User Feedback**: Notify users when degraded functionality is active
|
||||
|
||||
### Evolution Path
|
||||
|
||||
The Robustness Principle provides foundation for:
|
||||
- **Service Worker Integration**: Offline robustness capabilities
|
||||
- **Web Worker Offloading**: Move intensive operations off main thread
|
||||
- **Progressive Enhancement**: Advanced features for capable browsers
|
||||
- **Error Analytics**: Aggregate error patterns for system improvements
|
||||
|
||||
## References
|
||||
|
||||
- [Defensive Programming Best Practices](https://en.wikipedia.org/wiki/Defensive_programming)
|
||||
- [JavaScript Error Handling Patterns](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Control_flow_and_error_handling)
|
||||
- [Web API Security Guidelines](https://developer.mozilla.org/en-US/docs/Web/Security)
|
||||
- [Performance Impact of Error Handling](https://v8.dev/docs/optimize)
|
||||
|
||||
## Approval
|
||||
|
||||
**Decided by**: Claude Code Development Team
|
||||
**Date**: 2025-11-11
|
||||
**Context**: Production hardening and security enhancement
|
||||
**Next Review**: After 6 months of production use or major security incidents
|
||||
453
docs/architecture/CAPABILITIES_ARCHITECTURE.md
Normal file
453
docs/architecture/CAPABILITIES_ARCHITECTURE.md
Normal file
@@ -0,0 +1,453 @@
|
||||
# Capabilities Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
MarkiTect uses a **capabilities-based architecture** where functionality is organized into independent, reusable packages called **capabilities**. Each capability is a self-contained git submodule with its own repository, enabling independent development, versioning, and reuse across projects.
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. **Separation of Concerns**
|
||||
|
||||
**Critical Rule:** The main repository (`markitect-main`) **MUST NOT** directly modify capability code.
|
||||
|
||||
- ✅ **DO**: Use capabilities as dependencies
|
||||
- ✅ **DO**: Configure capabilities through documented interfaces
|
||||
- ✅ **DO**: Report issues and feature requests to capability repos
|
||||
- ❌ **DON'T**: Edit capability code from the main repo
|
||||
- ❌ **DON'T**: Make commits directly in capability subdirectories
|
||||
- ❌ **DON'T**: Bypass the submodule boundary
|
||||
|
||||
**Why?** Capabilities are independent repositories. Direct modifications create:
|
||||
- Merge conflicts when syncing with upstream
|
||||
- Broken dependency management
|
||||
- Loss of independent versioning
|
||||
- Confusion about source of truth
|
||||
|
||||
### 2. **Git Submodule Architecture**
|
||||
|
||||
Capabilities are integrated as **git submodules**, not regular directories:
|
||||
|
||||
```
|
||||
markitect-main/
|
||||
├── .gitmodules # Submodule configuration
|
||||
├── capabilities/
|
||||
│ ├── testdrive-jsui/ # Git submodule → separate repo
|
||||
│ ├── issue-facade/ # Git submodule → separate repo
|
||||
│ ├── kaizen-agentic/ # Git submodule → separate repo
|
||||
│ ├── markitect-content/ # Local capability (legacy)
|
||||
│ └── release-management/ # Local capability (legacy)
|
||||
```
|
||||
|
||||
**Submodule vs Local Capabilities:**
|
||||
- **Submodules**: Independent git repositories, separate development lifecycle
|
||||
- **Local**: Part of main repo, shared lifecycle (being phased out)
|
||||
|
||||
**Target State:** All capabilities should eventually be submodules.
|
||||
|
||||
### 3. **Independent Development Lifecycle**
|
||||
|
||||
Each capability has its own:
|
||||
- ✅ Git repository on gitea
|
||||
- ✅ Version numbering (semantic versioning)
|
||||
- ✅ Issue tracking and roadmap
|
||||
- ✅ Testing and CI/CD pipeline
|
||||
- ✅ Documentation and README
|
||||
- ✅ Contributors and maintainers
|
||||
|
||||
### 4. **Interface-Based Integration**
|
||||
|
||||
Capabilities expose **stable interfaces** to the main project:
|
||||
|
||||
```python
|
||||
# markitect uses capability through documented interface
|
||||
from testdrive_jsui import TestDriveJSUIEngine
|
||||
|
||||
engine = TestDriveJSUIEngine()
|
||||
engine.render_document(content, mode='edit', config=config)
|
||||
```
|
||||
|
||||
**Integration Points:**
|
||||
1. **Python Package Interface**: Clean import and API
|
||||
2. **Configuration**: Via `pyproject.toml` dependencies
|
||||
3. **Plugin System**: Via plugin registration and metadata
|
||||
4. **Makefile Targets**: Via capability discovery system
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Working on Capabilities
|
||||
|
||||
**Rule:** Each capability is developed in **its own Claude Code session**.
|
||||
|
||||
#### Main Repository Session
|
||||
```bash
|
||||
# In markitect-main/
|
||||
cd /home/worsch/markitect-main
|
||||
|
||||
# Main repo tasks:
|
||||
# - Integrate capabilities
|
||||
# - Update dependencies
|
||||
# - Configure capability usage
|
||||
# - Test integration points
|
||||
```
|
||||
|
||||
#### Capability Session
|
||||
```bash
|
||||
# In capability repository
|
||||
cd /home/worsch/markitect-main/capabilities/testdrive-jsui
|
||||
|
||||
# OR clone separately
|
||||
git clone http://gitea/coulomb/testdrive-jsui.git
|
||||
cd testdrive-jsui
|
||||
|
||||
# Capability tasks:
|
||||
# - Implement features
|
||||
# - Fix bugs
|
||||
# - Write tests
|
||||
# - Update documentation
|
||||
# - Make releases
|
||||
```
|
||||
|
||||
### Recommended Session Pattern
|
||||
|
||||
**Scenario: Need to add feature to testdrive-jsui**
|
||||
|
||||
1. **Open separate Claude instance** for testdrive-jsui
|
||||
```bash
|
||||
# In new terminal/session
|
||||
cd /path/to/testdrive-jsui
|
||||
# Work on feature
|
||||
git commit -m "feat: new feature"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
2. **Update main project** (different Claude instance)
|
||||
```bash
|
||||
cd /home/worsch/markitect-main
|
||||
git submodule update --remote capabilities/testdrive-jsui
|
||||
git commit -m "chore: update testdrive-jsui submodule"
|
||||
```
|
||||
|
||||
**Why separate instances?**
|
||||
- Clear context boundaries
|
||||
- Prevents accidental cross-contamination
|
||||
- Respects repository independence
|
||||
- Better commit hygiene
|
||||
|
||||
### Updating Submodules
|
||||
|
||||
When a capability releases a new version:
|
||||
|
||||
```bash
|
||||
# In main repo
|
||||
cd /home/worsch/markitect-main
|
||||
|
||||
# Update specific capability
|
||||
cd capabilities/testdrive-jsui
|
||||
git pull origin main
|
||||
cd ../..
|
||||
git add capabilities/testdrive-jsui
|
||||
git commit -m "chore: update testdrive-jsui to v1.2.0"
|
||||
|
||||
# Or update all capabilities
|
||||
git submodule update --remote
|
||||
git commit -am "chore: update all capabilities"
|
||||
```
|
||||
|
||||
### Adding New Capabilities
|
||||
|
||||
```bash
|
||||
# 1. Create capability repository on gitea
|
||||
# http://gitea/coulomb/new-capability
|
||||
|
||||
# 2. Add as submodule to main repo
|
||||
cd /home/worsch/markitect-main
|
||||
git submodule add http://gitea/coulomb/new-capability.git capabilities/new-capability
|
||||
|
||||
# 3. Add dependency to pyproject.toml
|
||||
# dependencies = [
|
||||
# "new-capability @ file:./capabilities/new-capability"
|
||||
# ]
|
||||
|
||||
# 4. Commit submodule addition
|
||||
git commit -m "feat: add new-capability as submodule"
|
||||
```
|
||||
|
||||
## Dependency Management
|
||||
|
||||
Capabilities are declared in `pyproject.toml`:
|
||||
|
||||
```toml
|
||||
[project]
|
||||
dependencies = [
|
||||
# Core dependencies
|
||||
"markdown-it-py",
|
||||
"click>=8.0.0",
|
||||
|
||||
# Capabilities (file-based dependencies)
|
||||
"release-management @ file:./capabilities/release-management",
|
||||
"testdrive-jsui @ file:./capabilities/testdrive-jsui",
|
||||
"issue-facade @ file:./capabilities/issue-facade",
|
||||
]
|
||||
```
|
||||
|
||||
**Pattern for Capabilities:**
|
||||
```toml
|
||||
"capability-name @ file:./capabilities/capability-name"
|
||||
```
|
||||
|
||||
This enables:
|
||||
- ✅ pip install with editable dependencies
|
||||
- ✅ Proper package resolution
|
||||
- ✅ Clean imports in code
|
||||
- ✅ Development mode support
|
||||
|
||||
## Capability Structure
|
||||
|
||||
Each capability should follow this structure:
|
||||
|
||||
```
|
||||
capability-name/
|
||||
├── README.md # Comprehensive documentation
|
||||
├── pyproject.toml # Package configuration
|
||||
├── Makefile # Build and test automation
|
||||
├── .gitignore # Git ignore rules
|
||||
├── src/capability_name/ # Python package source
|
||||
│ ├── __init__.py
|
||||
│ ├── core/ # Core functionality
|
||||
│ ├── utils/ # Utilities
|
||||
│ └── testing/ # Testing infrastructure
|
||||
├── js/ # JavaScript (if applicable)
|
||||
│ ├── components/
|
||||
│ ├── controls/
|
||||
│ └── tests/
|
||||
├── static/ # Static assets (if applicable)
|
||||
│ ├── css/
|
||||
│ ├── images/
|
||||
│ └── templates/
|
||||
├── tests/ # Python tests
|
||||
│ └── test_*.py
|
||||
└── docs/ # Additional documentation
|
||||
├── API.md
|
||||
└── USAGE.md
|
||||
```
|
||||
|
||||
### Required Files
|
||||
|
||||
1. **README.md** - Must include:
|
||||
- Purpose and description
|
||||
- Installation instructions
|
||||
- Usage examples
|
||||
- API documentation
|
||||
- Integration guide
|
||||
|
||||
2. **pyproject.toml** - Must define:
|
||||
- Package metadata
|
||||
- Dependencies
|
||||
- Build system
|
||||
- Entry points
|
||||
|
||||
3. **Makefile** - Should include:
|
||||
- help target
|
||||
- test targets
|
||||
- install/setup targets
|
||||
- capability-info target (for discovery)
|
||||
|
||||
## Plugin System Integration
|
||||
|
||||
For capabilities that provide plugins (like testdrive-jsui):
|
||||
|
||||
### Plugin Self-Declaration
|
||||
|
||||
```python
|
||||
class TestDriveJSUIEngine(RenderingEnginePlugin):
|
||||
"""Plugin declares its own location - no hardcoded paths."""
|
||||
|
||||
def get_plugin_source_dir(self) -> Path:
|
||||
"""Plugin knows where it lives."""
|
||||
return Path(__file__).parent.parent.parent / "capabilities" / "testdrive-jsui"
|
||||
|
||||
def get_asset_paths(self) -> Dict[str, Path]:
|
||||
"""Plugin declares its asset structure."""
|
||||
base = self.get_plugin_source_dir()
|
||||
return {
|
||||
'js': base / 'js',
|
||||
'css': base / 'static' / 'css',
|
||||
'templates': base / 'static' / 'templates',
|
||||
}
|
||||
```
|
||||
|
||||
### Plugin Discovery
|
||||
|
||||
The main repo uses **generic discovery** - no hardcoded capability names:
|
||||
|
||||
```python
|
||||
# ✅ Good: Generic discovery
|
||||
if hasattr(engine, 'get_plugin_source_dir'):
|
||||
source_dir = engine.get_plugin_source_dir()
|
||||
|
||||
# ❌ Bad: Hardcoded capability names
|
||||
if engine_name == 'testdrive-jsui':
|
||||
source_dir = Path('capabilities/testdrive-jsui')
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Capability Tests
|
||||
Each capability tests **in isolation**:
|
||||
|
||||
```bash
|
||||
cd capabilities/testdrive-jsui
|
||||
pytest tests/
|
||||
npm test
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
Main repo tests **integration points**:
|
||||
|
||||
```python
|
||||
# tests/integration/test_capabilities.py
|
||||
def test_testdrive_jsui_integration():
|
||||
"""Test that testdrive-jsui integrates correctly."""
|
||||
engine = TestDriveJSUIEngine()
|
||||
result = engine.render_document(sample_content, 'edit', config)
|
||||
assert result is not None
|
||||
```
|
||||
|
||||
### Test Boundaries
|
||||
- **Capability tests**: Internal functionality, unit tests, component tests
|
||||
- **Main repo tests**: Integration, configuration, plugin discovery
|
||||
|
||||
## Migration Path
|
||||
|
||||
### Converting Local Capability to Submodule
|
||||
|
||||
1. **Create separate git repo**
|
||||
```bash
|
||||
cd /tmp
|
||||
cp -r markitect-main/capabilities/capability-name capability-name
|
||||
cd capability-name
|
||||
git init
|
||||
git add .
|
||||
git commit -m "Initial commit"
|
||||
git remote add origin http://gitea/coulomb/capability-name.git
|
||||
git push -u origin main
|
||||
```
|
||||
|
||||
2. **Remove from main repo**
|
||||
```bash
|
||||
cd markitect-main
|
||||
git rm -rf capabilities/capability-name
|
||||
git commit -m "chore: remove capability-name for submodule conversion"
|
||||
```
|
||||
|
||||
3. **Add as submodule**
|
||||
```bash
|
||||
git submodule add http://gitea/coulomb/capability-name.git capabilities/capability-name
|
||||
git commit -m "feat: add capability-name as submodule"
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### DO ✅
|
||||
|
||||
1. **Develop capabilities independently**
|
||||
- Use separate Claude instances
|
||||
- Maintain clear context boundaries
|
||||
- Follow semantic versioning
|
||||
|
||||
2. **Use documented interfaces**
|
||||
- Import via Python packages
|
||||
- Use plugin APIs
|
||||
- Follow configuration patterns
|
||||
|
||||
3. **Test integration points**
|
||||
- Verify capability imports
|
||||
- Test plugin discovery
|
||||
- Validate configurations
|
||||
|
||||
4. **Update submodules explicitly**
|
||||
- Pull capability changes deliberately
|
||||
- Review updates before committing
|
||||
- Update main repo pointer
|
||||
|
||||
5. **Document integration**
|
||||
- How to use the capability
|
||||
- Configuration options
|
||||
- Example usage
|
||||
|
||||
### DON'T ❌
|
||||
|
||||
1. **Edit capability code from main repo**
|
||||
- Opens separate repo/instance instead
|
||||
- Respects submodule boundary
|
||||
- Maintains clean separation
|
||||
|
||||
2. **Hardcode capability paths**
|
||||
- Use self-declaration instead
|
||||
- Generic discovery patterns
|
||||
- Interface-based access
|
||||
|
||||
3. **Create circular dependencies**
|
||||
- Main depends on capabilities
|
||||
- Capabilities should NOT depend on main
|
||||
- Keep dependency graph acyclic
|
||||
|
||||
4. **Bypass submodule system**
|
||||
- Don't manually copy files
|
||||
- Use git submodule commands
|
||||
- Respect version control
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Submodule not updated
|
||||
|
||||
```bash
|
||||
# Pull latest from capability repo
|
||||
cd capabilities/testdrive-jsui
|
||||
git pull origin main
|
||||
|
||||
# Update main repo pointer
|
||||
cd ../..
|
||||
git add capabilities/testdrive-jsui
|
||||
git commit -m "chore: update testdrive-jsui"
|
||||
```
|
||||
|
||||
### Import errors
|
||||
|
||||
```bash
|
||||
# Reinstall capabilities
|
||||
pip install -e .
|
||||
|
||||
# Or specific capability
|
||||
pip install -e ./capabilities/testdrive-jsui
|
||||
```
|
||||
|
||||
### Merge conflicts in submodules
|
||||
|
||||
```bash
|
||||
# Don't merge in submodule from main repo!
|
||||
# Instead, go to capability repo directly
|
||||
|
||||
cd /path/to/separate/testdrive-jsui
|
||||
# or
|
||||
cd capabilities/testdrive-jsui
|
||||
|
||||
# Resolve conflicts there
|
||||
git pull origin main
|
||||
# fix conflicts
|
||||
git push origin main
|
||||
|
||||
# Then update main repo
|
||||
cd ../../
|
||||
git submodule update --remote
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [Agent: Capability Manager](../../agents/agent-capability-manager.md) - Automated capability management
|
||||
- [Plugin System](../PLUGIN_SYSTEM.md) - Plugin architecture documentation
|
||||
- [Git Submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) - Official git submodules guide
|
||||
|
||||
---
|
||||
|
||||
**Remember:** Capabilities are **independent projects**. Treat them with the same respect you'd give any external dependency - use their interfaces, don't modify their internals.
|
||||
203
docs/composition-guide.md
Normal file
203
docs/composition-guide.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# Infospace Composition Guide
|
||||
|
||||
One completed, viable infospace can be reused as a **discipline** for
|
||||
another infospace — a lens applied to a different topic. This guide
|
||||
explains how composition works and walks through the live
|
||||
`examples/supply-chain-vsm/` reference.
|
||||
|
||||
---
|
||||
|
||||
## What composition means
|
||||
|
||||
An **infospace** is a directory of typed entities governed by
|
||||
`infospace.yaml`. Its entities and relations describe a specific topic
|
||||
(for example, Adam Smith's *Wealth of Nations*).
|
||||
|
||||
A **discipline** is an infospace declared as a reusable analytical
|
||||
framework by another infospace. When infospace B binds infospace A as a
|
||||
discipline:
|
||||
|
||||
1. B's entities can reference A's entities in `## WoN Concept` (or
|
||||
equivalent) sections.
|
||||
2. Properties A has already computed on its entities — such as VSM system
|
||||
placement — become available to B by transitivity through the mapping.
|
||||
3. B can impose its own viability thresholds independently of A's. The two
|
||||
infospaces each pass or fail viability on their own terms.
|
||||
|
||||
The binding is declarative: a relative path in `infospace.yaml` plus a
|
||||
display name. No code. No import. The discipline is looked up on disk at
|
||||
the declared path when B's commands run.
|
||||
|
||||
---
|
||||
|
||||
## The viability pre-condition
|
||||
|
||||
Binding a non-viable infospace as a discipline is a mistake: a framework
|
||||
that fails its own thresholds is not a stable reference frame. Before
|
||||
binding, confirm the candidate discipline is viable:
|
||||
|
||||
```bash
|
||||
cd examples/infospace-with-history
|
||||
markitect infospace viability
|
||||
```
|
||||
|
||||
```
|
||||
Metric Value Threshold Status
|
||||
---------------------------------------------------------------
|
||||
redundancy_ratio 0.0061 max=0.1 PASS
|
||||
coverage_ratio 0.6190 min=0.4 PASS
|
||||
coherence_components 0.0000 max=3 PASS
|
||||
consistency_cycles 0.0000 max=0 PASS
|
||||
granularity_entropy 2.6748 min=1.0 PASS
|
||||
per_entity_mean 3.9556 min=3.5 PASS
|
||||
|
||||
Viable: YES (6/6 thresholds met)
|
||||
```
|
||||
|
||||
If the discipline is not viable, fix it first (see
|
||||
`examples/infospace-with-history/docs/advanced-usage.md` §4 for triaging
|
||||
low scorers).
|
||||
|
||||
---
|
||||
|
||||
## Example — how `supply-chain-vsm` binds WoN
|
||||
|
||||
The supply-chain infospace declares WoN as a discipline in its
|
||||
`infospace.yaml`:
|
||||
|
||||
```yaml
|
||||
topic:
|
||||
name: "Modern Supply Chain Management"
|
||||
domain: "Operations Management"
|
||||
sources: artifacts/sources/
|
||||
|
||||
disciplines:
|
||||
- name: "Wealth of Nations"
|
||||
path: ../infospace-with-history
|
||||
```
|
||||
|
||||
The binding is a **relative path**, so the two infospaces travel together
|
||||
(they can be moved as a pair without breaking the link).
|
||||
|
||||
Verify the binding resolves and the discipline is viable:
|
||||
|
||||
```bash
|
||||
cd examples/supply-chain-vsm
|
||||
markitect infospace disciplines
|
||||
```
|
||||
|
||||
```
|
||||
Name Entities Viable Path
|
||||
----------------------------------------------------------------------
|
||||
Wealth of Nations 988 YES ../infospace-with-history
|
||||
```
|
||||
|
||||
Each supply-chain entity then carries a `## WoN Concept` section
|
||||
mapping it to exactly one WoN entity. The consolidated mapping files
|
||||
(`output/mappings/*-mappings.md`) record the pairing, rationale, and a
|
||||
conceptual-continuity rating (Strong / Moderate / Weak):
|
||||
|
||||
| Supply Chain Entity | WoN Concept | Strength | VSM |
|
||||
|------------------------------|----------------------------------|----------|-------|
|
||||
| Demand Signal | Effectual Demand | Strong | S2 |
|
||||
| Vendor-Managed Inventory | Division of Labour | Strong | S1/S2 |
|
||||
| Just-in-Time Inventory | Circulating Capital | Strong | S1/S3 |
|
||||
| Bullwhip Effect | Natural Price as Central Price | Moderate | S2 |
|
||||
| Safety Stock | Accumulation of Stock | Moderate | S3 |
|
||||
|
||||
Because each WoN entity already has a VSM system placement (S1–S5), the
|
||||
supply-chain entities inherit a VSM position by transitivity through
|
||||
their mapping — without supply-chain-vsm needing its own VSM reference.
|
||||
|
||||
---
|
||||
|
||||
## Creating a new infospace that binds an existing one
|
||||
|
||||
Step-by-step, using WoN as the discipline for a hypothetical "Modern
|
||||
Monetary Policy" infospace:
|
||||
|
||||
### 1. Start from the target topic
|
||||
|
||||
```bash
|
||||
mkdir -p examples/monetary-policy/artifacts/sources
|
||||
cd examples/monetary-policy
|
||||
markitect infospace init
|
||||
```
|
||||
|
||||
### 2. Declare the discipline in `infospace.yaml`
|
||||
|
||||
```yaml
|
||||
topic:
|
||||
name: "Modern Monetary Policy"
|
||||
domain: "Macroeconomics"
|
||||
sources: artifacts/sources/
|
||||
|
||||
disciplines:
|
||||
- name: "Wealth of Nations"
|
||||
path: ../infospace-with-history
|
||||
```
|
||||
|
||||
Alternatively, bind imperatively after `init`:
|
||||
|
||||
```bash
|
||||
markitect infospace bind-discipline ../infospace-with-history --name "Wealth of Nations"
|
||||
```
|
||||
|
||||
### 3. Set your own viability thresholds
|
||||
|
||||
Copy the `viability:` block from a reference infospace and tune the
|
||||
numbers to the scale and maturity of your topic. A smaller infospace
|
||||
(50 entities, not 988) may need laxer `coverage_ratio` and stricter
|
||||
`redundancy_ratio`.
|
||||
|
||||
### 4. Verify the binding
|
||||
|
||||
```bash
|
||||
markitect infospace disciplines
|
||||
```
|
||||
|
||||
If `Viable` is `NO`, stop and fix the discipline before continuing.
|
||||
|
||||
### 5. Reference discipline entities in your own entities
|
||||
|
||||
For each entity in the new infospace, add a `## <Discipline> Concept`
|
||||
section that names the WoN entity the concept maps to, plus a rationale.
|
||||
The exact section heading is configured per schema — see
|
||||
`schemas/won-mapping-schema-v1.0.md` in `supply-chain-vsm` for the
|
||||
template used there.
|
||||
|
||||
### 6. Run checks and evaluate
|
||||
|
||||
```bash
|
||||
markitect infospace check
|
||||
markitect infospace evaluate --provider openrouter
|
||||
markitect infospace eval-summary --update-metrics
|
||||
markitect infospace viability
|
||||
```
|
||||
|
||||
The new infospace passes or fails viability independently of WoN.
|
||||
|
||||
---
|
||||
|
||||
## Why composition, not inclusion?
|
||||
|
||||
An alternative would be to copy WoN entities directly into the target
|
||||
infospace. Composition avoids that by design:
|
||||
|
||||
- **One source of truth** — if WoN is refined, every infospace that binds
|
||||
it picks up the improvement on the next run without a sync step.
|
||||
- **Separation of concerns** — each infospace owns its own schema,
|
||||
thresholds, and entity set. Changing the target topic cannot pollute
|
||||
the discipline.
|
||||
- **Bounded dependency** — the binding is a path, so the coupling is
|
||||
visible in one place (`infospace.yaml`) and easy to remove.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- `examples/supply-chain-vsm/README.md` — the full reference composition.
|
||||
- `examples/supply-chain-vsm/output/mappings/` — consolidated mapping
|
||||
files showing the rationale and strength rating for each pairing.
|
||||
- `examples/infospace-with-history/docs/advanced-usage.md` — patterns for
|
||||
maintaining the discipline once it is in use.
|
||||
344
docs/infospace-primitives.md
Normal file
344
docs/infospace-primitives.md
Normal file
@@ -0,0 +1,344 @@
|
||||
# Infospace Primitives Reference
|
||||
|
||||
This document describes the primitives provided by the `markitect/infospace/`
|
||||
package for creating, evaluating, maintaining, and composing infospaces.
|
||||
|
||||
---
|
||||
|
||||
## Core Concepts
|
||||
|
||||
An **infospace** is a structured, evaluable, composable collection of
|
||||
entities that explains a **topic** through the lens of one or more
|
||||
**disciplines**.
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **Topic** | The subject matter being explained |
|
||||
| **Discipline** | A reusable framework of concepts applied as an analytical lens |
|
||||
| **Entity** | The atomic unit of knowledge — slug, definition, provenance, domain |
|
||||
| **Evaluation** | Per-entity or collection-level quality assessment |
|
||||
| **Viability** | Whether an infospace meets its threshold scores |
|
||||
|
||||
---
|
||||
|
||||
## Configuration (`infospace.yaml`)
|
||||
|
||||
Every infospace is declared via an `infospace.yaml` file. The configuration
|
||||
model is defined in `markitect/infospace/config.py`.
|
||||
|
||||
### Minimal example
|
||||
|
||||
```yaml
|
||||
topic:
|
||||
name: "The Wealth of Nations"
|
||||
domain: "Classical Economics"
|
||||
sources: artifacts/sources/
|
||||
|
||||
disciplines:
|
||||
- name: "Viable System Model"
|
||||
path: artifacts/vsm-reference/
|
||||
|
||||
schemas:
|
||||
entity: schemas/economic-entity-schema-v1.0.md
|
||||
|
||||
viability:
|
||||
coverage_ratio: { min: 0.60 }
|
||||
redundancy_ratio: { max: 0.05 }
|
||||
per_entity_mean: { min: 3.5 }
|
||||
```
|
||||
|
||||
### Key models
|
||||
|
||||
- **`TopicConfig`** — `name`, `domain`, `sources`
|
||||
- **`DisciplineBinding`** — `name`, `path` (to another infospace directory)
|
||||
- **`SchemaRegistry`** — `entity`, `mapping`, `analysis` schema paths
|
||||
- **`ViabilityThreshold`** — `metric`, `min`, `max` bounds
|
||||
- **`PipelineConfig`** — Ordered list of `PipelineStage` entries
|
||||
- **`InfospaceConfig`** — Top-level config combining all of the above
|
||||
|
||||
### Default directories
|
||||
|
||||
| Setting | Default |
|
||||
|---------|---------|
|
||||
| `entities_dir` | `output/entities` |
|
||||
| `evaluations_dir` | `output/evaluations` |
|
||||
| `metrics_dir` | `output/metrics` |
|
||||
|
||||
---
|
||||
|
||||
## Entity Metadata
|
||||
|
||||
Entities are parsed from markdown files by `markitect/infospace/entity_parser.py`.
|
||||
|
||||
**`EntityMeta`** fields: `slug`, `title`, `definition`, `domain`,
|
||||
`source_chapter`, `context`, `original_wording`, `modern_interpretation`,
|
||||
`definition_word_count`, `total_word_count`, `section_slugs`.
|
||||
|
||||
```python
|
||||
from markitect.infospace import parse_entity_directory
|
||||
entities = parse_entity_directory(Path("output/entities"))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Schema Validation
|
||||
|
||||
Deterministic validation of entity files against structural schemas.
|
||||
|
||||
```python
|
||||
from markitect.infospace import validate_entity, ECONOMIC_ENTITY_SCHEMA
|
||||
result = validate_entity(entity_meta, schema=ECONOMIC_ENTITY_SCHEMA)
|
||||
print(result.summary())
|
||||
```
|
||||
|
||||
Checks: section presence, word count ranges, heading format, enum values.
|
||||
|
||||
---
|
||||
|
||||
## Per-entity Evaluation
|
||||
|
||||
LLM-based quality assessment of individual entities. Defined in
|
||||
`markitect/infospace/evaluate.py`.
|
||||
|
||||
```bash
|
||||
# Evaluate all entities
|
||||
markitect infospace evaluate --provider openrouter
|
||||
|
||||
# Single entity
|
||||
markitect infospace evaluate --entity division-of-labour --provider openrouter
|
||||
```
|
||||
|
||||
### Pipeline functions
|
||||
|
||||
- `build_evaluation_prompt(entity, topic, dimensions)` — build the LLM prompt
|
||||
- `parse_evaluation_response(text, dimensions)` — parse LLM output to `ScoreEntry` list
|
||||
- `run_entity_evaluation(config, entities, adapter, ...)` — full batch pipeline
|
||||
|
||||
Results are written to `output/evaluations/` as YAML frontmatter + markdown.
|
||||
|
||||
---
|
||||
|
||||
## Collection-level Checks
|
||||
|
||||
Five concerns assessed at the collection level. Each has a dedicated
|
||||
module in `markitect/infospace/checks/`.
|
||||
|
||||
| Concern | Module | Key metric |
|
||||
|---------|--------|------------|
|
||||
| **C1 — Redundancy** | `redundancy.py` | `redundancy_ratio` |
|
||||
| **C2 — Coverage** | `coverage.py` | `coverage_ratio` |
|
||||
| **C3 — Coherence** | `coherence.py` | `coherence_components`, `modularity` |
|
||||
| **C4 — Consistency** | `consistency.py` | `consistency_cycles` |
|
||||
| **C5 — Granularity** | `granularity.py` | `granularity_entropy` |
|
||||
|
||||
### Orchestrator
|
||||
|
||||
```python
|
||||
from markitect.infospace.checks import run_all_checks
|
||||
report = run_all_checks(entities, embeddings=emb, graph=g)
|
||||
metrics = report.metrics() # Dict[str, float]
|
||||
```
|
||||
|
||||
### CLI
|
||||
|
||||
```bash
|
||||
# Run all checks
|
||||
markitect infospace check
|
||||
|
||||
# Run specific concerns
|
||||
markitect infospace check --concern redundancy --concern coverage
|
||||
|
||||
# JSON output
|
||||
markitect infospace check --json
|
||||
```
|
||||
|
||||
After each check run, metrics are automatically recorded to history.
|
||||
|
||||
---
|
||||
|
||||
## Metrics History
|
||||
|
||||
Timestamped snapshots track metrics over time. Defined in
|
||||
`markitect/infospace/history.py`.
|
||||
|
||||
```bash
|
||||
# Show history
|
||||
markitect infospace history
|
||||
|
||||
# Trend for a single metric
|
||||
markitect infospace history --metric coverage_ratio
|
||||
|
||||
# Compare two snapshots
|
||||
markitect infospace history-diff 2026-02-01 2026-03-01
|
||||
```
|
||||
|
||||
### Key functions
|
||||
|
||||
- `snapshot_from_checks(report, entity_count)` — create snapshot from check results
|
||||
- `record_check_results(report, config, root, entity_count)` — save metrics + append to history
|
||||
- `get_history(config, root)` — read full history
|
||||
- `metric_trend(history, metric_name)` — extract single metric across time
|
||||
|
||||
---
|
||||
|
||||
## Viability
|
||||
|
||||
Viability is assessed by comparing current metrics to thresholds declared
|
||||
in `infospace.yaml`.
|
||||
|
||||
```bash
|
||||
markitect infospace viability
|
||||
```
|
||||
|
||||
### Threshold model
|
||||
|
||||
```yaml
|
||||
viability:
|
||||
coverage_ratio: { min: 0.60 } # must be >= 0.60
|
||||
redundancy_ratio: { max: 0.05 } # must be <= 0.05
|
||||
consistency_cycles: { max: 0 } # must be exactly 0
|
||||
```
|
||||
|
||||
Each threshold has `min` and/or `max` bounds. A metric passes if it falls
|
||||
within bounds. An infospace is viable when all thresholds pass.
|
||||
|
||||
---
|
||||
|
||||
## Composition
|
||||
|
||||
One infospace can use another as a discipline. The composition model is
|
||||
defined in `markitect/infospace/composition.py`.
|
||||
|
||||
### Binding a discipline
|
||||
|
||||
```bash
|
||||
markitect infospace bind-discipline ./path/to/vsm-infospace --name "Viable System Model"
|
||||
```
|
||||
|
||||
This adds a `DisciplineBinding` to `infospace.yaml` and validates the
|
||||
discipline exists and has an `infospace.yaml`.
|
||||
|
||||
### Checking discipline status
|
||||
|
||||
```bash
|
||||
markitect infospace disciplines
|
||||
```
|
||||
|
||||
Shows: name, entity count, viability status, path.
|
||||
|
||||
### Viability requirement
|
||||
|
||||
A discipline must meet its own viability thresholds to be considered
|
||||
reliable. The `check_discipline_status()` function loads the discipline's
|
||||
metrics and runs its own threshold checks.
|
||||
|
||||
### Stale mapping detection
|
||||
|
||||
```bash
|
||||
markitect infospace stale-mappings
|
||||
```
|
||||
|
||||
Compares local mapping references against the discipline's current entity
|
||||
set. If a referenced discipline entity has been removed, the mapping is
|
||||
flagged as stale.
|
||||
|
||||
### Key functions
|
||||
|
||||
- `resolve_discipline_path(binding, root)` — resolve to absolute path
|
||||
- `load_discipline_config(binding, root)` — load discipline's `infospace.yaml`
|
||||
- `check_discipline_status(binding, root)` — full status with viability
|
||||
- `get_discipline_entities(binding, root)` — entity list from discipline
|
||||
- `find_stale_mappings(config, root, mapping_references)` — detect stale refs
|
||||
- `bind_discipline(config, name, path, root)` — add binding to config
|
||||
|
||||
---
|
||||
|
||||
## Evaluation Output Format
|
||||
|
||||
Evaluation results use YAML frontmatter + markdown body. Defined in
|
||||
`markitect/infospace/evaluation.py` and `evaluation_io.py`.
|
||||
|
||||
### Per-entity evaluation file
|
||||
|
||||
```markdown
|
||||
---
|
||||
entity_slug: division-of-labour
|
||||
evaluator: openrouter/default
|
||||
evaluated_at: '2026-02-19T10:30:00'
|
||||
overall_score: 4.1667
|
||||
scores:
|
||||
- name: definition_precision
|
||||
value: 4.5
|
||||
max_value: 5.0
|
||||
...
|
||||
---
|
||||
|
||||
# Evaluation: Division Of Labour
|
||||
|
||||
## definition_precision — 4.5 / 5.0
|
||||
|
||||
The definition clearly captures the core concept...
|
||||
```
|
||||
|
||||
### Snapshot
|
||||
|
||||
```yaml
|
||||
snapshot_id: abc12345
|
||||
created_at: '2026-02-19T10:30:00+00:00'
|
||||
schema_name: default
|
||||
entity_count: 85
|
||||
entity_evaluations: [...]
|
||||
collection_metrics:
|
||||
- name: coverage_ratio
|
||||
value: 0.75
|
||||
concern: C2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## State
|
||||
|
||||
Runtime state is computed from entities, evaluations, and metrics.
|
||||
Defined in `markitect/infospace/state.py`.
|
||||
|
||||
```python
|
||||
from markitect.infospace import build_state
|
||||
state = build_state(config, entities=entities, metrics=metrics)
|
||||
state.is_viable # True if all thresholds pass
|
||||
state.viability_results # List[ViabilityResult]
|
||||
state.summary() # Dict for display
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Command Summary
|
||||
|
||||
All commands are under `markitect infospace`:
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `init` | Create a new `infospace.yaml` |
|
||||
| `status` | Show entity count, domains, evaluation state |
|
||||
| `entities` | List entities with metadata |
|
||||
| `evaluate` | Run per-entity LLM evaluation |
|
||||
| `check` | Run collection-level quality checks (C1-C5) |
|
||||
| `viability` | Show viability dashboard |
|
||||
| `history` | Show metrics history |
|
||||
| `history-diff` | Compare two snapshots by date |
|
||||
| `bind-discipline` | Bind an external infospace as a discipline |
|
||||
| `disciplines` | List bound disciplines and viability |
|
||||
| `stale-mappings` | Detect stale cross-infospace references |
|
||||
|
||||
---
|
||||
|
||||
## Platform Dependencies
|
||||
|
||||
The infospace tooling builds on these platform modules:
|
||||
|
||||
| Module | Used for |
|
||||
|--------|----------|
|
||||
| `markitect/llm/` | Embedding adapters, LLM evaluation |
|
||||
| `markitect/analysis/graph.py` | Graph analysis (networkx wrapper) |
|
||||
| `markitect/analysis/fca.py` | Formal Concept Analysis |
|
||||
| `markitect/prompts/execution/batch.py` | Batch LLM evaluation |
|
||||
| `markitect/prompts/dependencies/models.py` | DependencyGraph |
|
||||
662
docs/specifications/schema-extensions-spec.md
Normal file
662
docs/specifications/schema-extensions-spec.md
Normal file
@@ -0,0 +1,662 @@
|
||||
# MarkiTect Schema Extensions Specification v1.0
|
||||
|
||||
## Status: Draft - Phase 1 Implementation
|
||||
|
||||
## Overview
|
||||
|
||||
This specification defines MarkiTect-specific extensions to JSON Schema (draft-07) for markdown document validation with content control, section classification, and flexible structural constraints.
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Backward Compatibility**: Existing schemas without extensions continue to work
|
||||
2. **Namespace Isolation**: All extensions prefixed with `x-markitect-`
|
||||
3. **Progressive Enhancement**: Extensions add capabilities without breaking standard JSON Schema
|
||||
4. **Clear Semantics**: Each extension has well-defined validation behavior
|
||||
5. **Metaschema Validation**: All extensions validated by MarkiTect metaschema
|
||||
|
||||
---
|
||||
|
||||
## Extension: `x-markitect-sections`
|
||||
|
||||
### Purpose
|
||||
|
||||
Define document sections with classification levels (required, recommended, optional, discouraged, improper) and content control specifications.
|
||||
|
||||
### Schema Location
|
||||
|
||||
Applied at the **root level** of the schema or within **properties** that represent document sections.
|
||||
|
||||
### Format
|
||||
|
||||
```json
|
||||
{
|
||||
"x-markitect-sections": {
|
||||
"SECTION_NAME": {
|
||||
"classification": "required|recommended|optional|discouraged|improper",
|
||||
"heading_level": 1|2|3|4|5|6,
|
||||
"position": "after_title|before_section_name|after_section_name|anywhere",
|
||||
"content_instruction": "string",
|
||||
"min_paragraphs": integer,
|
||||
"max_paragraphs": integer,
|
||||
"min_code_blocks": integer,
|
||||
"max_code_blocks": integer,
|
||||
"min_lists": integer,
|
||||
"max_lists": integer,
|
||||
"warning_if_missing": "string",
|
||||
"error_message": "string",
|
||||
"alternatives": ["SECTION_NAME_1", "SECTION_NAME_2"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Property Definitions
|
||||
|
||||
#### `classification` (required)
|
||||
|
||||
Classification level determining validation behavior:
|
||||
|
||||
- **`required`**: Section MUST be present. Validation fails if missing.
|
||||
- **`recommended`**: Section SHOULD be present. Warning if missing, but validation succeeds.
|
||||
- **`optional`**: Section MAY be present. No validation impact either way.
|
||||
- **`discouraged`**: Section SHOULD NOT be present. Warning if present, but validation succeeds.
|
||||
- **`improper`**: Section MUST NOT be present. Validation fails if present.
|
||||
|
||||
**Type**: String enum
|
||||
**Required**: Yes
|
||||
**Values**: `["required", "recommended", "optional", "discouraged", "improper"]`
|
||||
|
||||
#### `heading_level` (optional)
|
||||
|
||||
The heading level (H1-H6) for this section.
|
||||
|
||||
**Type**: Integer
|
||||
**Range**: 1-6
|
||||
**Default**: 2 (for standard sections)
|
||||
|
||||
#### `position` (optional)
|
||||
|
||||
Where this section should appear relative to other sections.
|
||||
|
||||
**Type**: String enum
|
||||
**Values**:
|
||||
- `"after_title"` - Immediately after document title (H1)
|
||||
- `"before_section_name"` - Before another named section
|
||||
- `"after_section_name"` - After another named section
|
||||
- `"anywhere"` - No position constraint (default)
|
||||
|
||||
**Default**: `"anywhere"`
|
||||
|
||||
#### `content_instruction` (optional)
|
||||
|
||||
Human-readable instruction describing what content belongs in this section.
|
||||
|
||||
**Type**: String
|
||||
**Usage**: Displayed in validation warnings, generated templates, and documentation
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
"content_instruction": "Brief command syntax showing all options and arguments"
|
||||
```
|
||||
|
||||
#### Content Constraints (optional)
|
||||
|
||||
Minimum and maximum counts for content elements within the section:
|
||||
|
||||
- **`min_paragraphs`**: Minimum paragraph count (integer ≥ 0)
|
||||
- **`max_paragraphs`**: Maximum paragraph count (integer ≥ min_paragraphs)
|
||||
- **`min_code_blocks`**: Minimum code block count (integer ≥ 0)
|
||||
- **`max_code_blocks`**: Maximum code block count (integer ≥ min_code_blocks)
|
||||
- **`min_lists`**: Minimum list count (integer ≥ 0)
|
||||
- **`max_lists`**: Maximum list count (integer ≥ max_lists)
|
||||
|
||||
**Type**: Integer
|
||||
**Default**: No constraint if omitted
|
||||
|
||||
#### `warning_if_missing` (optional)
|
||||
|
||||
Custom warning message when a recommended section is missing.
|
||||
|
||||
**Type**: String
|
||||
**Applies to**: `classification: "recommended"` only
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
"warning_if_missing": "Examples greatly improve documentation usability"
|
||||
```
|
||||
|
||||
#### `error_message` (optional)
|
||||
|
||||
Custom error message when validation fails.
|
||||
|
||||
**Type**: String
|
||||
**Applies to**: `classification: "required"` or `"improper"`
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
"error_message": "Internal notes must not appear in published documentation"
|
||||
```
|
||||
|
||||
#### `alternatives` (optional)
|
||||
|
||||
Array of alternative section names that satisfy the requirement.
|
||||
|
||||
**Type**: Array of strings
|
||||
**Usage**: If any alternative is present, requirement is satisfied
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
{
|
||||
"classification": "required",
|
||||
"alternatives": ["EXAMPLES", "USAGE", "TUTORIAL"]
|
||||
}
|
||||
```
|
||||
|
||||
### Example: Manpage Schema with Sections
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "Unix Manpage Schema",
|
||||
"x-markitect-sections": {
|
||||
"SYNOPSIS": {
|
||||
"classification": "required",
|
||||
"heading_level": 2,
|
||||
"position": "after_title",
|
||||
"content_instruction": "Brief command syntax with options and arguments",
|
||||
"min_paragraphs": 1,
|
||||
"max_paragraphs": 5,
|
||||
"min_code_blocks": 0,
|
||||
"max_code_blocks": 3,
|
||||
"error_message": "SYNOPSIS section is mandatory for all manpages"
|
||||
},
|
||||
"DESCRIPTION": {
|
||||
"classification": "required",
|
||||
"heading_level": 2,
|
||||
"position": "after_section_name",
|
||||
"content_instruction": "Detailed explanation of what the command does",
|
||||
"min_paragraphs": 2,
|
||||
"error_message": "DESCRIPTION section is mandatory for all manpages"
|
||||
},
|
||||
"EXAMPLES": {
|
||||
"classification": "recommended",
|
||||
"heading_level": 2,
|
||||
"content_instruction": "Practical usage examples with explanations",
|
||||
"min_code_blocks": 3,
|
||||
"warning_if_missing": "Examples greatly improve manpage usability"
|
||||
},
|
||||
"SEE ALSO": {
|
||||
"classification": "recommended",
|
||||
"heading_level": 2,
|
||||
"content_instruction": "Related commands and documentation references",
|
||||
"warning_if_missing": "Cross-references help users discover related functionality"
|
||||
},
|
||||
"BUGS": {
|
||||
"classification": "optional",
|
||||
"heading_level": 2,
|
||||
"content_instruction": "Known issues and bug reporting information"
|
||||
},
|
||||
"DEPRECATED": {
|
||||
"classification": "discouraged",
|
||||
"heading_level": 2,
|
||||
"warning_if_missing": "Consider moving deprecated content to historical documentation"
|
||||
},
|
||||
"INTERNAL_NOTES": {
|
||||
"classification": "improper",
|
||||
"heading_level": 2,
|
||||
"error_message": "Internal notes must not appear in published manpages"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Validation Behavior
|
||||
|
||||
#### Required Sections
|
||||
|
||||
```json
|
||||
"SYNOPSIS": {"classification": "required"}
|
||||
```
|
||||
|
||||
**Validation**:
|
||||
- Section missing → **ERROR** → `is_valid = False`
|
||||
- Section present → Continue validation
|
||||
- Custom `error_message` used if provided
|
||||
|
||||
#### Recommended Sections
|
||||
|
||||
```json
|
||||
"EXAMPLES": {"classification": "recommended"}
|
||||
```
|
||||
|
||||
**Validation**:
|
||||
- Section missing → **WARNING** → `is_valid = True` (with warnings)
|
||||
- Section present → Continue validation
|
||||
- Custom `warning_if_missing` used if provided
|
||||
|
||||
#### Optional Sections
|
||||
|
||||
```json
|
||||
"BUGS": {"classification": "optional"}
|
||||
```
|
||||
|
||||
**Validation**:
|
||||
- Section missing → No impact
|
||||
- Section present → Continue validation
|
||||
- No messages generated
|
||||
|
||||
#### Discouraged Sections
|
||||
|
||||
```json
|
||||
"DEPRECATED": {"classification": "discouraged"}
|
||||
```
|
||||
|
||||
**Validation**:
|
||||
- Section missing → No impact
|
||||
- Section present → **WARNING** → `is_valid = True` (with warnings)
|
||||
- Custom warning message used if provided
|
||||
|
||||
#### Improper Sections
|
||||
|
||||
```json
|
||||
"INTERNAL_NOTES": {"classification": "improper"}
|
||||
```
|
||||
|
||||
**Validation**:
|
||||
- Section missing → No impact
|
||||
- Section present → **ERROR** → `is_valid = False`
|
||||
- Custom `error_message` used if provided
|
||||
|
||||
---
|
||||
|
||||
## Extension: `x-markitect-content-control`
|
||||
|
||||
### Purpose
|
||||
|
||||
Define content validation rules for document sections including pattern matching, quality metrics, and semantic constraints.
|
||||
|
||||
### Schema Location
|
||||
|
||||
Applied at **root level** or within specific **section properties**.
|
||||
|
||||
### Format
|
||||
|
||||
```json
|
||||
{
|
||||
"x-markitect-content-control": {
|
||||
"section_name": {
|
||||
"required_patterns": ["regex_pattern_1", "regex_pattern_2"],
|
||||
"discouraged_patterns": ["regex_pattern_1"],
|
||||
"forbidden_patterns": ["regex_pattern_1"],
|
||||
"content_quality": {
|
||||
"min_words": integer,
|
||||
"max_words": integer,
|
||||
"readability_target": "technical|general|simple|advanced",
|
||||
"min_sentences": integer,
|
||||
"max_sentences": integer
|
||||
},
|
||||
"content_instructions": ["instruction_1", "instruction_2"],
|
||||
"link_validation": {
|
||||
"check_internal": boolean,
|
||||
"check_external": boolean,
|
||||
"allow_fragments": boolean
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Property Definitions
|
||||
|
||||
#### `required_patterns` (optional)
|
||||
|
||||
Array of regex patterns that MUST appear in section content.
|
||||
|
||||
**Type**: Array of strings (valid regex patterns)
|
||||
**Validation**: ERROR if any pattern missing
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
"required_patterns": [
|
||||
"\\*\\*[a-z-]+\\*\\*", // Bold command name
|
||||
"\\[.*\\]" // Options in brackets
|
||||
]
|
||||
```
|
||||
|
||||
#### `discouraged_patterns` (optional)
|
||||
|
||||
Array of regex patterns that SHOULD NOT appear in content.
|
||||
|
||||
**Type**: Array of strings (valid regex patterns)
|
||||
**Validation**: WARNING if any pattern found
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
"discouraged_patterns": [
|
||||
"TODO",
|
||||
"FIXME",
|
||||
"\\bWIP\\b"
|
||||
]
|
||||
```
|
||||
|
||||
#### `forbidden_patterns` (optional)
|
||||
|
||||
Array of regex patterns that MUST NOT appear in content.
|
||||
|
||||
**Type**: Array of strings (valid regex patterns)
|
||||
**Validation**: ERROR if any pattern found
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
"forbidden_patterns": [
|
||||
"password\\s*=\\s*[\"'].*[\"']", // Hard-coded passwords
|
||||
"api[_-]?key\\s*=\\s*[\"'].*[\"']" // Hard-coded API keys
|
||||
]
|
||||
```
|
||||
|
||||
#### `content_quality` (optional)
|
||||
|
||||
Quality metrics for section content:
|
||||
|
||||
**Sub-properties**:
|
||||
- **`min_words`**: Minimum word count (integer ≥ 0)
|
||||
- **`max_words`**: Maximum word count (integer ≥ min_words)
|
||||
- **`readability_target`**: Target readability level (enum)
|
||||
- `"simple"` - Elementary school level
|
||||
- `"general"` - General audience
|
||||
- `"technical"` - Technical audience
|
||||
- `"advanced"` - Expert/academic level
|
||||
- **`min_sentences`**: Minimum sentence count (integer ≥ 0)
|
||||
- **`max_sentences`**: Maximum sentence count (integer ≥ min_sentences)
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
"content_quality": {
|
||||
"min_words": 50,
|
||||
"max_words": 300,
|
||||
"readability_target": "technical",
|
||||
"min_sentences": 3
|
||||
}
|
||||
```
|
||||
|
||||
#### `content_instructions` (optional)
|
||||
|
||||
Array of human-readable instructions for content creation.
|
||||
|
||||
**Type**: Array of strings
|
||||
**Usage**: Displayed in templates, validation reports, and documentation
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
"content_instructions": [
|
||||
"Show command name in bold",
|
||||
"Include all major options",
|
||||
"Use italic for arguments and placeholders",
|
||||
"Keep syntax examples concise (1-3 lines)"
|
||||
]
|
||||
```
|
||||
|
||||
#### `link_validation` (optional)
|
||||
|
||||
Link checking configuration:
|
||||
|
||||
**Sub-properties**:
|
||||
- **`check_internal`**: Validate internal document links (boolean)
|
||||
- **`check_external`**: Validate external URLs (boolean)
|
||||
- **`allow_fragments`**: Allow fragment-only links like `#section` (boolean)
|
||||
|
||||
**Default**: All false (no link validation)
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
"link_validation": {
|
||||
"check_internal": true,
|
||||
"check_external": false,
|
||||
"allow_fragments": true
|
||||
}
|
||||
```
|
||||
|
||||
### Example: Content Control for API Documentation
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"title": "API Documentation Schema",
|
||||
"x-markitect-content-control": {
|
||||
"synopsis": {
|
||||
"required_patterns": [
|
||||
"\\*\\*[A-Z]+\\*\\*", // HTTP method in bold
|
||||
"`/api/.*`" // Endpoint path in code
|
||||
],
|
||||
"content_quality": {
|
||||
"min_words": 10,
|
||||
"max_words": 100,
|
||||
"readability_target": "technical"
|
||||
},
|
||||
"content_instructions": [
|
||||
"Start with HTTP method in bold (e.g., **GET**)",
|
||||
"Show endpoint path in code format",
|
||||
"Include brief one-line description"
|
||||
]
|
||||
},
|
||||
"request_parameters": {
|
||||
"required_patterns": [
|
||||
"\\*\\*[a-z_]+\\*\\*.*\\*[A-Za-z]+\\*" // Bold param name with italic type
|
||||
],
|
||||
"content_instructions": [
|
||||
"Use bold for parameter names",
|
||||
"Use italic for parameter types",
|
||||
"Include description for each parameter",
|
||||
"Mark required parameters clearly"
|
||||
]
|
||||
},
|
||||
"description": {
|
||||
"discouraged_patterns": [
|
||||
"TODO",
|
||||
"FIXME",
|
||||
"TBD"
|
||||
],
|
||||
"forbidden_patterns": [
|
||||
"password\\s*=",
|
||||
"secret\\s*=",
|
||||
"token\\s*="
|
||||
],
|
||||
"content_quality": {
|
||||
"min_words": 50,
|
||||
"max_words": 500,
|
||||
"readability_target": "technical",
|
||||
"min_sentences": 3
|
||||
},
|
||||
"link_validation": {
|
||||
"check_internal": true,
|
||||
"check_external": true,
|
||||
"allow_fragments": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Result Structure
|
||||
|
||||
### Enhanced ValidationResult Class
|
||||
|
||||
```python
|
||||
class ValidationResult:
|
||||
"""Result of schema validation with classification support."""
|
||||
|
||||
status: Literal["valid", "valid_with_warnings", "invalid"]
|
||||
errors: List[ValidationError] # Required/improper violations
|
||||
warnings: List[ValidationWarning] # Recommended/discouraged violations
|
||||
suggestions: List[str] # Optional improvements
|
||||
quality_metrics: Dict[str, Any] # Content quality scores
|
||||
```
|
||||
|
||||
### Validation Status Values
|
||||
|
||||
- **`"valid"`**: No errors, no warnings. Document fully conforms.
|
||||
- **`"valid_with_warnings"`**: No errors, but has warnings. Document acceptable but improvable.
|
||||
- **`"invalid"`**: Has errors. Document does not conform to schema.
|
||||
|
||||
### Error Types
|
||||
|
||||
```python
|
||||
class ValidationErrorType(Enum):
|
||||
MISSING_REQUIRED_SECTION = "missing_required_section"
|
||||
IMPROPER_SECTION_PRESENT = "improper_section_present"
|
||||
CONTENT_PATTERN_MISSING = "content_pattern_missing"
|
||||
CONTENT_PATTERN_FORBIDDEN = "content_pattern_forbidden"
|
||||
CONTENT_TOO_SHORT = "content_too_short"
|
||||
CONTENT_TOO_LONG = "content_too_long"
|
||||
INVALID_LINK = "invalid_link"
|
||||
STRUCTURE_MISMATCH = "structure_mismatch"
|
||||
```
|
||||
|
||||
### Warning Types
|
||||
|
||||
```python
|
||||
class ValidationWarningType(Enum):
|
||||
MISSING_RECOMMENDED_SECTION = "missing_recommended_section"
|
||||
DISCOURAGED_SECTION_PRESENT = "discouraged_section_present"
|
||||
CONTENT_PATTERN_DISCOURAGED = "content_pattern_discouraged"
|
||||
CONTENT_QUALITY_BELOW_TARGET = "content_quality_below_target"
|
||||
READABILITY_MISMATCH = "readability_mismatch"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Metaschema Validation
|
||||
|
||||
### Extension Validation Rules
|
||||
|
||||
The MarkiTect metaschema validates these extensions:
|
||||
|
||||
```json
|
||||
{
|
||||
"x-markitect-sections": {
|
||||
"type": "object",
|
||||
"patternProperties": {
|
||||
"^[A-Z][A-Z0-9_ ]*$": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"classification": {
|
||||
"type": "string",
|
||||
"enum": ["required", "recommended", "optional", "discouraged", "improper"]
|
||||
},
|
||||
"heading_level": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"maximum": 6
|
||||
},
|
||||
"position": {
|
||||
"type": "string",
|
||||
"enum": ["after_title", "before_section_name", "after_section_name", "anywhere"]
|
||||
},
|
||||
"content_instruction": {"type": "string"},
|
||||
"min_paragraphs": {"type": "integer", "minimum": 0},
|
||||
"max_paragraphs": {"type": "integer", "minimum": 0},
|
||||
"min_code_blocks": {"type": "integer", "minimum": 0},
|
||||
"max_code_blocks": {"type": "integer", "minimum": 0},
|
||||
"min_lists": {"type": "integer", "minimum": 0},
|
||||
"max_lists": {"type": "integer", "minimum": 0},
|
||||
"warning_if_missing": {"type": "string"},
|
||||
"error_message": {"type": "string"},
|
||||
"alternatives": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"}
|
||||
}
|
||||
},
|
||||
"required": ["classification"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"x-markitect-content-control": {
|
||||
"type": "object",
|
||||
"patternProperties": {
|
||||
"^[a-z][a-z0-9_]*$": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"required_patterns": {
|
||||
"type": "array",
|
||||
"items": {"type": "string", "format": "regex"}
|
||||
},
|
||||
"discouraged_patterns": {
|
||||
"type": "array",
|
||||
"items": {"type": "string", "format": "regex"}
|
||||
},
|
||||
"forbidden_patterns": {
|
||||
"type": "array",
|
||||
"items": {"type": "string", "format": "regex"}
|
||||
},
|
||||
"content_quality": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"min_words": {"type": "integer", "minimum": 0},
|
||||
"max_words": {"type": "integer", "minimum": 0},
|
||||
"readability_target": {
|
||||
"type": "string",
|
||||
"enum": ["simple", "general", "technical", "advanced"]
|
||||
},
|
||||
"min_sentences": {"type": "integer", "minimum": 0},
|
||||
"max_sentences": {"type": "integer", "minimum": 0}
|
||||
}
|
||||
},
|
||||
"content_instructions": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"}
|
||||
},
|
||||
"link_validation": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"check_internal": {"type": "boolean"},
|
||||
"check_external": {"type": "boolean"},
|
||||
"allow_fragments": {"type": "boolean"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Phase 1 Scope
|
||||
|
||||
1. Define and document extension formats ✓
|
||||
2. Update metaschema to validate extensions
|
||||
3. Implement basic classification validation (required/recommended/optional/discouraged/improper)
|
||||
4. Create example schemas demonstrating all features
|
||||
5. Update CLI to report errors vs warnings separately
|
||||
|
||||
### Future Enhancements (Phase 2+)
|
||||
|
||||
- Content pattern matching implementation
|
||||
- Quality metrics calculation
|
||||
- Link validation
|
||||
- Readability scoring
|
||||
- Position constraints enforcement
|
||||
|
||||
---
|
||||
|
||||
## Version History
|
||||
|
||||
- **v1.0 (Draft)** - Initial specification for Phase 1 implementation
|
||||
- `x-markitect-sections` extension defined
|
||||
- `x-markitect-content-control` extension defined
|
||||
- Validation result structure defined
|
||||
- Metaschema validation rules defined
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- JSON Schema Draft-07: https://json-schema.org/draft-07/schema
|
||||
- MarkiTect Schema Evolution Workplan: `examples/manpages/SCHEMA_EVOLUTION_WORKPLAN.md`
|
||||
- Existing Metaschema: `markitect/schemas/markitect-metaschema.json`
|
||||
- Metaschema Validator: `markitect/metaschema.py`
|
||||
141
docs/successor-gap-assessment.md
Normal file
141
docs/successor-gap-assessment.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# markitect-main → Successor Repos: Gap Assessment
|
||||
|
||||
**Date:** 2026-05-23
|
||||
**Author:** Claude (custodian session)
|
||||
**Status:** Draft — awaiting Bernd's decisions on items A/B/C below
|
||||
|
||||
## Purpose
|
||||
|
||||
Bernd is retiring `markitect-main` and has transferred most functionality to
|
||||
sibling repos. This document identifies what was provided by `markitect-main`
|
||||
that is **not addressed** in those successors, and flags candidates that may
|
||||
not fit any successor's intent.
|
||||
|
||||
## Successor Ecosystem (5 repos, not 3)
|
||||
|
||||
| Repo | Role |
|
||||
|---|---|
|
||||
| `markitect-tool` | Markdown syntax layer + structured-document primitives; defines source-adapter and render-adapter contracts. CLI: `mkt`. |
|
||||
| `kontextual-engine` | Headless knowledge operations engine: artifacts, collections, persistence, relationships, workflow runs/manifests, query, quality/assessment, API. |
|
||||
| `infospace-bench` | Application layer — concrete infospaces, evaluation methodology, reference pilots. |
|
||||
| `markitect-filter` | Source-format ingestion adapters (`source.epub3`, `source.pdf`) implementing the markitect-tool source-adapter contract. |
|
||||
| `markitect-quarkdown` | Render/export adapter — implements the markitect-tool render-adapter contract via Quarkdown. |
|
||||
|
||||
## Method
|
||||
|
||||
Analysis is grounded in each successor's own assessment docs (recent, May 2026):
|
||||
|
||||
- `markitect-tool/docs/markitect-main-scope-assessment.md`
|
||||
- `kontextual-engine/docs/markitect-main-scope-assessment.md`
|
||||
- `kontextual-engine/docs/system-layer-extraction-inventory.md`
|
||||
- `kontextual-engine/docs/system-layer-migration-backlog.md`
|
||||
- `infospace-bench/docs/markitect-main-scope-assessment.md`
|
||||
- `infospace-bench/docs/legacy-infospace-feature-inventory.md`
|
||||
- `infospace-bench/docs/replacement-acceptance-matrix.md`
|
||||
|
||||
Cross-checked against actual `markitect-main` module sizing (Python LOC) and
|
||||
`__init__.py` docstrings.
|
||||
|
||||
**Confidence:** These successor docs are authoritative on *intent*. They have
|
||||
**not** been line-verified to confirm every "reimplement"-classified item
|
||||
actually landed in the successor. Where verification matters, it's flagged.
|
||||
|
||||
---
|
||||
|
||||
## A. Doesn't fit any successor's intent — needs a new home or explicit retirement
|
||||
|
||||
These are explicitly pushed away by tool/engine/bench and are unrelated to
|
||||
filter/quarkdown.
|
||||
|
||||
| markitect-main area | LOC | What it is | Status |
|
||||
|---|---|---|---|
|
||||
| `markitect/finance/` | ~8,100 | Cost-tracking system: cost items, period allocation to issues, financial reports, audit trails | **Orphan.** markitect-main's own SCOPE.md lists "financial transactions" as out-of-scope. Belongs with issue/project-ops, not knowledge tooling. |
|
||||
| `issue_tracker/` + `_issue-tracking/` + `.issues/` | ~1,200 | Issue tracking (finance allocates costs to these issues) | **Orphan to the five** — but likely already superseded by the `issue-facade` capability / `use-issues` skill. **Verify before retiring.** |
|
||||
| `markitect/profile/` | ~1,600 | User-profile CRUD, multi-profile, DB-backed | **Orphan.** Unrelated to all five. (Distinct from quarkdown's *render* "profile".) |
|
||||
| `markitect/production/` | ~3,800 | Deployment-readiness validation, cross-platform checks, perf benchmarking | Engine keeps only "structured error/audit *ideas*". Deployment-validation bulk is orphan. |
|
||||
| `tools/`, `services/`, gitea/tddai glue | ~5,500 | Project-ops tooling | Out-of-scope everywhere. |
|
||||
| `markitect/legacy/` + `legacy_compat.py` | ~2,700 | Backward-compat shims | Retire by definition. |
|
||||
|
||||
## B. Rendering / asset / plugin layer — only *partially* covered, real residual gap
|
||||
|
||||
**This is the most consequential gap.** `SCOPE.md` lists "Rendering: markdown
|
||||
→ interactive HTML via plugin system (testdrive-jsui)" as an in-scope
|
||||
capability of markitect-main.
|
||||
|
||||
| Area | LOC | Covered? |
|
||||
|---|---|---|
|
||||
| `markitect/plugins/` (generic processor/formatter/validator/exporter plugin system) | ~8,000 | **No.** tool defines a render-adapter *contract* and an *extension* point, but the general plugin runtime isn't carried. |
|
||||
| `markitect/assets/` (content-addressable asset store, dedup, `.mdpkg` ZIP packaging, symlink handling) + `asset_registry.json` (277 KB) | ~6,000 | **No.** Bench says "leave behind unless a concrete export needs assets." |
|
||||
| Interactive-HTML / testdrive-jsui rendering, `static/`, `themes/`, `templates/document.html`, JS UI | — | **Partial only.** quarkdown covers a *Quarkdown* export path; the interactive-HTML / JS-UI path has no home. |
|
||||
|
||||
**Decision needed:** spin these into a dedicated render/asset repo (sibling to
|
||||
quarkdown), fold the asset store into one of the existing repos, or retire the
|
||||
interactive-HTML path.
|
||||
|
||||
## C. The other "Information Space" lineage — `markitect/spaces/` (~11,000 LOC)
|
||||
|
||||
**Distinct from `markitect/infospace/`** (which infospace-bench inherited).
|
||||
`spaces/` is an older/parallel abstraction with features bench did *not* take:
|
||||
|
||||
- event-driven change tracking & notifications
|
||||
- persistent transclusion context with cross-space references
|
||||
- bidirectional directory synchronization
|
||||
- HTML rendering of spaces with caching/themes
|
||||
|
||||
Engine takes generic persistence concepts and bench takes infospace semantics,
|
||||
but **these specific `spaces/` behaviors (bidirectional sync, event
|
||||
notifications, cross-space transclusion context) aren't mapped anywhere.**
|
||||
|
||||
Likely intended as dead/superseded — but 11k LOC warrants an explicit "retire
|
||||
vs salvage" call.
|
||||
|
||||
## D. Declined-by-design (confirm retirement, don't re-extract)
|
||||
|
||||
| Area | LOC | Disposition |
|
||||
|---|---|---|
|
||||
| `markitect/graphql/` | ~4,000 | All three explicitly declined GraphQL ("evidence of API need, not a commitment"). |
|
||||
| `markitect/query_paradigms/` | ~3,500 | Engine/tool keep the *QueryResult envelope* concept but say "do not port the registry wholesale." |
|
||||
| `markitect/proxy/` | ~870 | Non-markdown→md proxy with checksum/freshness tracking. **Overlaps markitect-filter.** Freshness/staleness-tracking mechanism may be worth checking against bench's deferred "stale-mappings." |
|
||||
| `capabilities/` (top-level) | ~8,300 | Capability-packaging architecture; partially maps to tool (schema generation) but the packaging approach itself isn't carried. |
|
||||
|
||||
---
|
||||
|
||||
## What this means
|
||||
|
||||
The successors are, by their own assessments, **near complete for the
|
||||
in-scope core** (parsing/schema → tool; persistence/workflow → engine;
|
||||
infospace lifecycle → bench; ingestion → filter; one render path →
|
||||
quarkdown). The truly unaddressed functionality is almost entirely the stuff
|
||||
markitect-main accreted **beyond** its stated scope: finance, issue tracking,
|
||||
user profiles, production/deployment validation, the asset/plugin/interactive-HTML
|
||||
rendering stack, and the older `spaces/` abstraction.
|
||||
|
||||
## Decisions for Bernd
|
||||
|
||||
Three live decisions, not a long extraction backlog:
|
||||
|
||||
### Decision 1 — Render/asset stack (Section B)
|
||||
The one with genuine product value left.
|
||||
- **Option 1a:** new repo (sibling to quarkdown) for plugin runtime + asset store + interactive-HTML
|
||||
- **Option 1b:** fold the asset store into an existing repo (most likely markitect-tool, behind a flag); retire interactive-HTML
|
||||
- **Option 1c:** retire the interactive-HTML path entirely; trust quarkdown export as the single render story
|
||||
|
||||
### Decision 2 — `markitect/spaces/` (Section C)
|
||||
- **Option 2a:** salvage bidirectional-sync / event-tracking / cross-space transclusion into engine (engine has the persistence story to support it)
|
||||
- **Option 2b:** retire wholesale as superseded by infospace
|
||||
|
||||
### Decision 3 — Project-ops cluster (Section A: finance + issues + profile)
|
||||
- **Option 3a:** confirm `issue-facade` already replaces `issue_tracker/` + `finance/`; retire both
|
||||
- **Option 3b:** identify a home for any pieces worth keeping
|
||||
|
||||
---
|
||||
|
||||
## Suggested verification before deciding
|
||||
|
||||
If verification matters before committing:
|
||||
|
||||
- **For Decision 1:** grep the five repos for any render/asset adapter that already covers the HTML path beyond Quarkdown.
|
||||
- **For Decision 2:** check whether engine's `OperationRun` + collection model can express bidirectional-sync semantics, or whether new primitives would be needed.
|
||||
- **For Decision 3:** confirm whether `issue-facade` truly replaces `issue_tracker/` + `finance/` end-to-end.
|
||||
|
||||
Happy to do any of these focused passes when you're ready to decide.
|
||||
495
docs/user-guides/SCHEMA_REFINEMENT_TOOLS.md
Normal file
495
docs/user-guides/SCHEMA_REFINEMENT_TOOLS.md
Normal file
@@ -0,0 +1,495 @@
|
||||
# Schema Refinement Tools - User Guide
|
||||
|
||||
## Overview
|
||||
|
||||
MarkiTect Phase 2 introduces powerful schema refinement tools to help you analyze and improve JSON schemas for markdown validation. These tools detect rigidity issues and automatically apply fixes to make schemas more flexible and reusable.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Analyze a schema for rigidity issues
|
||||
markitect schema-analyze examples/manpages/markdown-manpage-schema.json
|
||||
|
||||
# Refine a schema automatically
|
||||
markitect schema-refine examples/manpages/markdown-manpage-schema.json --output refined-schema.json
|
||||
|
||||
# Review each fix interactively
|
||||
markitect schema-refine examples/manpages/markdown-manpage-schema.json --interactive
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
### schema-analyze
|
||||
|
||||
Analyzes a JSON schema to detect rigidity issues and calculate a rigidity score (0-100).
|
||||
|
||||
#### Usage
|
||||
|
||||
```bash
|
||||
markitect schema-analyze <schema-file> [OPTIONS]
|
||||
```
|
||||
|
||||
#### Options
|
||||
|
||||
- `--verbose`, `-v`: Show detailed analysis with current and suggested values
|
||||
|
||||
#### Examples
|
||||
|
||||
```bash
|
||||
# Basic analysis
|
||||
markitect schema-analyze schema.json
|
||||
|
||||
# Verbose output with details
|
||||
markitect schema-analyze schema.json --verbose
|
||||
```
|
||||
|
||||
#### Output
|
||||
|
||||
The analyzer provides:
|
||||
|
||||
- **Rigidity Score** (0-100): Higher scores indicate more rigid schemas
|
||||
- 0-40: LOW - Flexible, good design
|
||||
- 41-70: MEDIUM - Some rigidity detected
|
||||
- 71-100: HIGH - Very rigid, needs refinement
|
||||
|
||||
- **Phase 1 Features**: Checks for classification system and content control
|
||||
- **Issue Count**: Breakdown by severity (Errors, Warnings, Info)
|
||||
- **Detected Issues**: List of problems with suggestions
|
||||
|
||||
#### Exit Codes
|
||||
|
||||
- `0`: Schema is flexible (score ≤ 50)
|
||||
- `1`: Schema is rigid (score > 50)
|
||||
- `2`: Error occurred
|
||||
|
||||
### schema-refine
|
||||
|
||||
Automatically refines rigid schemas by applying fixes for detected issues.
|
||||
|
||||
#### Usage
|
||||
|
||||
```bash
|
||||
markitect schema-refine <schema-file> [OPTIONS]
|
||||
```
|
||||
|
||||
#### Options
|
||||
|
||||
- `--output`, `-o PATH`: Output file (default: overwrite input file)
|
||||
- `--loosen-counts`: Convert exact counts to flexible ranges (default: enabled)
|
||||
- `--no-loosen-counts`: Disable count loosening
|
||||
- `--round-numbers`: Round overly specific numbers (default: enabled)
|
||||
- `--no-round-numbers`: Disable number rounding
|
||||
- `--migrate-deprecated`: Document deprecated extensions (default: disabled)
|
||||
- `--dry-run`: Show changes without applying them
|
||||
- `--interactive`, `-i`: Prompt for each refinement interactively
|
||||
|
||||
#### Examples
|
||||
|
||||
```bash
|
||||
# Refine schema in place
|
||||
markitect schema-refine schema.json
|
||||
|
||||
# Preview changes without applying
|
||||
markitect schema-refine schema.json --dry-run
|
||||
|
||||
# Save refined schema to new file
|
||||
markitect schema-refine schema.json --output refined-schema.json
|
||||
|
||||
# Review each fix interactively
|
||||
markitect schema-refine schema.json --interactive
|
||||
|
||||
# Disable specific refinements
|
||||
markitect schema-refine schema.json --no-loosen-counts
|
||||
```
|
||||
|
||||
#### Refinement Actions
|
||||
|
||||
The refiner automatically applies these fixes:
|
||||
|
||||
1. **Exact Count Loosening**: Converts exact counts to flexible ranges
|
||||
- Before: `"minItems": 5, "maxItems": 5`
|
||||
- After: `"minItems": 3, "maxItems": 10`
|
||||
|
||||
2. **Const Value Conversion**: Replaces exact value constraints with ranges
|
||||
- Before: `"const": 1`
|
||||
- After: `"minimum": 0, "maximum": 2`
|
||||
|
||||
3. **Number Rounding**: Rounds overly specific numbers
|
||||
- Before: `"minItems": 73`
|
||||
- After: `"minItems": 70`
|
||||
|
||||
4. **Range Widening**: Expands narrow integer ranges
|
||||
- Before: `"minimum": 5, "maximum": 6`
|
||||
- After: `"minimum": 0, "maximum": 11`
|
||||
|
||||
#### Exit Codes
|
||||
|
||||
- `0`: Success with changes applied
|
||||
- `1`: Success but no changes needed
|
||||
- `2`: Error occurred
|
||||
|
||||
## Issue Types
|
||||
|
||||
### Exact Count (WARNING)
|
||||
|
||||
**Problem**: Schema requires exact number of items, leaving no flexibility.
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
{
|
||||
"type": "array",
|
||||
"minItems": 5,
|
||||
"maxItems": 5
|
||||
}
|
||||
```
|
||||
|
||||
**Fix**: Convert to a range
|
||||
```json
|
||||
{
|
||||
"type": "array",
|
||||
"minItems": 3,
|
||||
"maxItems": 10
|
||||
}
|
||||
```
|
||||
|
||||
### Const Value (WARNING)
|
||||
|
||||
**Problem**: Property must have exact value.
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
{
|
||||
"type": "integer",
|
||||
"const": 1
|
||||
}
|
||||
```
|
||||
|
||||
**Fix**: Replace with range for numeric values
|
||||
```json
|
||||
{
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 2
|
||||
}
|
||||
```
|
||||
|
||||
### Overly Specific Numbers (INFO)
|
||||
|
||||
**Problem**: Numbers are too specific (like 73 instead of 70).
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
{
|
||||
"type": "array",
|
||||
"minItems": 73
|
||||
}
|
||||
```
|
||||
|
||||
**Fix**: Round to nearest 10
|
||||
```json
|
||||
{
|
||||
"type": "array",
|
||||
"minItems": 70
|
||||
}
|
||||
```
|
||||
|
||||
### No Flexibility (INFO)
|
||||
|
||||
**Problem**: Integer range is too narrow.
|
||||
|
||||
**Example**:
|
||||
```json
|
||||
{
|
||||
"type": "integer",
|
||||
"minimum": 5,
|
||||
"maximum": 6
|
||||
}
|
||||
```
|
||||
|
||||
**Fix**: Widen the range
|
||||
```json
|
||||
{
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 11
|
||||
}
|
||||
```
|
||||
|
||||
### Missing Classifications (INFO)
|
||||
|
||||
**Problem**: Schema doesn't use the Phase 1 classification system.
|
||||
|
||||
**Suggestion**: Add `x-markitect-sections` to classify sections as required/recommended/optional/discouraged/improper.
|
||||
|
||||
### Missing Content Control (INFO)
|
||||
|
||||
**Problem**: Schema lacks content validation patterns and quality metrics.
|
||||
|
||||
**Suggestion**: Add `x-markitect-content-control` for pattern validation and quality requirements.
|
||||
|
||||
### Deprecated Extensions (WARNING)
|
||||
|
||||
**Problem**: Schema uses old extension format.
|
||||
|
||||
**Example**: `x-markitect-required-sections`
|
||||
|
||||
**Suggestion**: Migrate to `x-markitect-sections` with classification system.
|
||||
|
||||
## Workflows
|
||||
|
||||
### Basic Workflow: Analyze and Refine
|
||||
|
||||
1. **Analyze** your schema to understand issues:
|
||||
```bash
|
||||
markitect schema-analyze my-schema.json --verbose
|
||||
```
|
||||
|
||||
2. **Preview** refinements before applying:
|
||||
```bash
|
||||
markitect schema-refine my-schema.json --dry-run
|
||||
```
|
||||
|
||||
3. **Apply** refinements:
|
||||
```bash
|
||||
markitect schema-refine my-schema.json --output my-schema-refined.json
|
||||
```
|
||||
|
||||
4. **Verify** improvements:
|
||||
```bash
|
||||
markitect schema-analyze my-schema-refined.json
|
||||
```
|
||||
|
||||
### Interactive Workflow
|
||||
|
||||
For fine-grained control, use interactive mode:
|
||||
|
||||
```bash
|
||||
markitect schema-refine my-schema.json --interactive
|
||||
```
|
||||
|
||||
The tool will:
|
||||
1. Show each detected issue
|
||||
2. Display current and suggested values
|
||||
3. Prompt for confirmation (y/N/q)
|
||||
4. Apply only approved fixes
|
||||
|
||||
Example session:
|
||||
```
|
||||
Issue 1/4
|
||||
Type: exact_count
|
||||
Path: properties.headings.level_1
|
||||
Array 'level_1' requires exactly 1 items
|
||||
Suggestion: Use a range like minItems: 0, maxItems: 6
|
||||
Current: {"minItems": 1, "maxItems": 1}
|
||||
Suggested: {"minItems": 0, "maxItems": 6}
|
||||
|
||||
Apply this fix? [y/N/q]: y
|
||||
✓ Applied
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
Use exit codes to enforce schema quality in your pipeline:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# Analyze schema and fail if rigid
|
||||
if ! markitect schema-analyze schema.json; then
|
||||
echo "Schema is too rigid (score > 50)"
|
||||
echo "Run: markitect schema-refine schema.json"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Schema quality check passed"
|
||||
```
|
||||
|
||||
### Schema Migration Workflow
|
||||
|
||||
Migrating from old format to Phase 1:
|
||||
|
||||
1. **Analyze** to identify deprecated extensions:
|
||||
```bash
|
||||
markitect schema-analyze old-schema.json
|
||||
```
|
||||
|
||||
2. **Document** deprecated extensions:
|
||||
```bash
|
||||
markitect schema-refine old-schema.json --migrate-deprecated
|
||||
```
|
||||
|
||||
3. **Manually migrate** to new format (automatic migration not implemented due to complexity)
|
||||
|
||||
## Best Practices
|
||||
|
||||
### When to Use schema-analyze
|
||||
|
||||
- Before committing schemas to version control
|
||||
- During code review to ensure quality
|
||||
- When creating new schemas from examples
|
||||
- To understand why a schema fails validation
|
||||
|
||||
### When to Use schema-refine
|
||||
|
||||
- After auto-generating schemas from documents
|
||||
- When inheriting legacy schemas
|
||||
- To quickly fix common rigidity issues
|
||||
- Before publishing schemas for reuse
|
||||
|
||||
### When to Use --interactive
|
||||
|
||||
- When you need fine-grained control
|
||||
- For schemas with domain-specific requirements
|
||||
- When learning about schema design
|
||||
- To review fixes before applying
|
||||
|
||||
### Recommended Settings
|
||||
|
||||
For most use cases:
|
||||
```bash
|
||||
# Balanced refinement (default)
|
||||
markitect schema-refine schema.json
|
||||
|
||||
# Conservative (preserve more constraints)
|
||||
markitect schema-refine schema.json --no-round-numbers
|
||||
|
||||
# Aggressive (maximum flexibility)
|
||||
markitect schema-refine schema.json --loosen-counts --round-numbers
|
||||
```
|
||||
|
||||
## Understanding Rigidity Scores
|
||||
|
||||
The rigidity score is calculated by weighting detected issues:
|
||||
|
||||
| Issue Type | Weight |
|
||||
|------------|--------|
|
||||
| Exact Count | 15 |
|
||||
| Overly Specific | 10 |
|
||||
| No Flexibility | 8 |
|
||||
| Missing Classifications | 5 |
|
||||
| Deprecated Extensions | 5 |
|
||||
| Missing Content Control | 3 |
|
||||
|
||||
**Score Interpretation**:
|
||||
- **0-20**: Excellent - Well-designed, flexible schema
|
||||
- **21-40**: Good - Minor improvements possible
|
||||
- **41-60**: Fair - Moderate rigidity, refinement recommended
|
||||
- **61-80**: Poor - Significant rigidity, refinement needed
|
||||
- **81-100**: Very Poor - Highly rigid, manual review recommended
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### Git Pre-commit Hook
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# .git/hooks/pre-commit
|
||||
|
||||
SCHEMAS=$(git diff --cached --name-only --diff-filter=ACM | grep '\.json$')
|
||||
|
||||
for schema in $SCHEMAS; do
|
||||
if markitect schema-analyze "$schema" 2>&1 | grep -q "RIGID"; then
|
||||
echo "Error: $schema is too rigid"
|
||||
echo "Run: markitect schema-refine $schema"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
### Makefile Target
|
||||
|
||||
```makefile
|
||||
.PHONY: check-schemas
|
||||
check-schemas:
|
||||
@for schema in schemas/*.json; do \
|
||||
echo "Checking $$schema..."; \
|
||||
markitect schema-analyze $$schema || exit 1; \
|
||||
done
|
||||
|
||||
.PHONY: refine-schemas
|
||||
refine-schemas:
|
||||
@for schema in schemas/*.json; do \
|
||||
echo "Refining $$schema..."; \
|
||||
markitect schema-refine $$schema; \
|
||||
done
|
||||
```
|
||||
|
||||
### Python Integration
|
||||
|
||||
```python
|
||||
import subprocess
|
||||
import json
|
||||
|
||||
def analyze_schema(schema_path):
|
||||
"""Analyze a schema and return rigidity score."""
|
||||
result = subprocess.run(
|
||||
["markitect", "schema-analyze", schema_path],
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
|
||||
# Parse output for score
|
||||
for line in result.stdout.split('\n'):
|
||||
if 'Rigidity Score:' in line:
|
||||
score = int(line.split(':')[1].split('/')[0].strip())
|
||||
return score
|
||||
return None
|
||||
|
||||
def refine_schema(schema_path, output_path):
|
||||
"""Refine a schema and save to output path."""
|
||||
result = subprocess.run(
|
||||
["markitect", "schema-refine", schema_path, "-o", output_path],
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
return result.returncode == 0
|
||||
|
||||
# Usage
|
||||
score = analyze_schema("schema.json")
|
||||
if score > 50:
|
||||
print(f"Schema is rigid (score: {score})")
|
||||
refine_schema("schema.json", "schema-refined.json")
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Schema Not Found
|
||||
|
||||
**Error**: `Error: Schema file not found: schema.json`
|
||||
|
||||
**Solution**: Check file path and ensure file exists.
|
||||
|
||||
### Invalid JSON
|
||||
|
||||
**Error**: `Error: Invalid JSON in schema file`
|
||||
|
||||
**Solution**: Validate JSON syntax using `jsonlint` or similar tool.
|
||||
|
||||
### No Changes Applied
|
||||
|
||||
**Output**: `No refinements needed - schema is already flexible`
|
||||
|
||||
**Reason**: Schema doesn't have any detectable rigidity issues or has rigidity score < 50.
|
||||
|
||||
**Action**: Use `--verbose` to see all issues including INFO level.
|
||||
|
||||
### Refinement Broke Schema
|
||||
|
||||
**Problem**: Refined schema is too permissive.
|
||||
|
||||
**Solution**:
|
||||
1. Use `--interactive` to selectively apply fixes
|
||||
2. Use `--no-loosen-counts` or `--no-round-numbers` to preserve constraints
|
||||
3. Manually adjust ranges after refinement
|
||||
|
||||
## See Also
|
||||
|
||||
- [Schema Extensions Specification](../specifications/schema-extensions-spec.md) - Complete Phase 1 specification
|
||||
- [Schema Evolution Workplan](../../examples/manpages/SCHEMA_EVOLUTION_WORKPLAN.md) - Roadmap for schema features
|
||||
- [Manpage Example](../../examples/manpages/README.md) - Complete example demonstrating schema validation
|
||||
|
||||
## Support
|
||||
|
||||
For issues, questions, or feature requests:
|
||||
- GitHub Issues: https://github.com/anthropics/markitect/issues
|
||||
- Documentation: https://github.com/anthropics/markitect/docs
|
||||
268
docs/workplan-testdrive-jsui-capability.md
Normal file
268
docs/workplan-testdrive-jsui-capability.md
Normal file
@@ -0,0 +1,268 @@
|
||||
# TestDrive-JSUI Capability Implementation Workplan
|
||||
|
||||
## 🎯 **Objective**
|
||||
|
||||
Safely extract JavaScript UI framework functionality into a dedicated `testdrive-jsui` capability while:
|
||||
- Protecting existing hard-won JavaScript UI functionality
|
||||
- Integrating JavaScript tests into the main Python test suite
|
||||
- Maintaining 100% test coverage and functionality
|
||||
- Creating a clean, extensible architecture for future JavaScript framework development
|
||||
|
||||
## 🔍 **Current State Analysis**
|
||||
|
||||
### **JavaScript UI Infrastructure:**
|
||||
|
||||
```
|
||||
markitect/static/js/
|
||||
├── core/
|
||||
│ └── section-manager.js (17K lines - Core component)
|
||||
├── components/
|
||||
│ ├── debug-panel.js (5.8K lines)
|
||||
│ ├── document-controls.js (7.9K lines)
|
||||
│ └── dom-renderer.js (40K lines - Major component)
|
||||
├── utils/ (Empty - utilities)
|
||||
└── tests/ (2.8K total lines)
|
||||
├── refactor-test-runner.js (Custom test framework)
|
||||
├── test-*.js (11 comprehensive test files)
|
||||
└── [Component-specific tests]
|
||||
```
|
||||
|
||||
### **Testing Infrastructure:**
|
||||
- ✅ **Jest framework** configured (`package.json`)
|
||||
- ✅ **JSDOM environment** for DOM testing
|
||||
- ✅ **Custom RefactorTestRunner** for TDD workflow
|
||||
- ✅ **11 comprehensive test files** (2,840 lines total)
|
||||
- ✅ **Component integration tests**
|
||||
- ✅ **Full workflow testing**
|
||||
|
||||
### **Feasibility: HIGHLY FEASIBLE** ✅
|
||||
- Well-structured components with clear separation
|
||||
- Comprehensive test coverage
|
||||
- Modern tooling (Jest + JSDOM)
|
||||
- Modular design already in place
|
||||
- TDD approach designed for safe refactoring
|
||||
|
||||
## 🚀 **Implementation Phases**
|
||||
|
||||
### **Phase 1: Foundation Setup** ⏱️ *~2-4 hours*
|
||||
|
||||
#### **Step 1.1: Create `testdrive-jsui` Capability Structure**
|
||||
```bash
|
||||
capabilities/testdrive-jsui/
|
||||
├── src/testdrive_jsui/
|
||||
│ ├── __init__.py
|
||||
│ ├── core/ # Framework components
|
||||
│ ├── components/ # UI components
|
||||
│ ├── utils/ # Utilities
|
||||
│ └── tests/ # Python test wrappers
|
||||
├── tests/ # Native Python tests
|
||||
├── js/ # JavaScript source
|
||||
│ ├── core/
|
||||
│ ├── components/
|
||||
│ ├── utils/
|
||||
│ └── tests/
|
||||
├── Makefile # Capability Makefile
|
||||
├── pyproject.toml # Package config
|
||||
├── package.json # JS dependencies
|
||||
├── jest.config.js # Jest configuration
|
||||
└── README.md # Documentation
|
||||
```
|
||||
|
||||
#### **Step 1.2: Setup Package Configuration**
|
||||
- **pyproject.toml**: Python package with subprocess/node dependencies
|
||||
- **package.json**: Jest + JSDOM + custom dependencies
|
||||
- **Makefile**: Integration with main capability system
|
||||
- **jest.config.js**: Proper test environment setup
|
||||
|
||||
#### **Step 1.3: Create Python-JavaScript Bridge**
|
||||
```python
|
||||
# testdrive_jsui/testing/js_test_runner.py
|
||||
class JavaScriptTestRunner:
|
||||
def run_js_tests(self, test_patterns=None):
|
||||
"""Run JavaScript tests via Node.js and return results"""
|
||||
|
||||
def integrate_with_pytest(self):
|
||||
"""Make JS tests discoverable by pytest"""
|
||||
```
|
||||
|
||||
### **Phase 2: Integration Layer** ⏱️ *~3-5 hours*
|
||||
|
||||
#### **Step 2.1: Python Test Wrappers**
|
||||
```python
|
||||
# Integration approach: Subprocess-based
|
||||
def test_section_manager_component():
|
||||
result = js_test_runner.run_test('test-section-manager-extraction.js')
|
||||
assert result.success
|
||||
assert result.tests_passed > 0
|
||||
|
||||
def test_dom_renderer_component():
|
||||
result = js_test_runner.run_test('test-domrenderer-extraction.js')
|
||||
assert result.success
|
||||
```
|
||||
|
||||
#### **Step 2.2: Main Test Suite Integration**
|
||||
- Add JS test discovery to pytest
|
||||
- Create test markers for JavaScript tests
|
||||
- Setup parallel execution (optional)
|
||||
- Integrate with main Makefile test targets
|
||||
|
||||
#### **Step 2.3: Capability Makefile Targets**
|
||||
```makefile
|
||||
# capabilities/testdrive-jsui/Makefile
|
||||
.PHONY: testdrive-jsui-test-js
|
||||
testdrive-jsui-test-js: ## Run JavaScript tests
|
||||
npm test
|
||||
|
||||
.PHONY: testdrive-jsui-test-integration
|
||||
testdrive-jsui-test-integration: ## Run Python-JS integration tests
|
||||
pytest tests/
|
||||
|
||||
.PHONY: testdrive-jsui-test-all
|
||||
testdrive-jsui-test-all: ## Run all tests (JS + Python integration)
|
||||
npm test && pytest tests/
|
||||
```
|
||||
|
||||
### **Phase 3: Safe Migration** ⏱️ *~4-6 hours*
|
||||
|
||||
#### **Step 3.1: Copy & Test Strategy**
|
||||
```bash
|
||||
# 1. Copy (don't move) JavaScript files to capability
|
||||
cp -r markitect/static/js/* capabilities/testdrive-jsui/js/
|
||||
|
||||
# 2. Verify tests still work in new location
|
||||
cd capabilities/testdrive-jsui && npm test
|
||||
|
||||
# 3. Create Python wrappers and verify integration
|
||||
pytest capabilities/testdrive-jsui/tests/
|
||||
|
||||
# 4. Add to main test suite gradually
|
||||
make test # Ensure main suite still passes
|
||||
```
|
||||
|
||||
#### **Step 3.2: Dual-Track Testing** *(Safety First!)*
|
||||
- Keep original files until migration complete
|
||||
- Run both locations in parallel initially
|
||||
- Compare test results for consistency
|
||||
- Gradual cutover with rollback option
|
||||
|
||||
### **Phase 4: Framework Enhancement** ⏱️ *~2-3 hours*
|
||||
|
||||
#### **Step 4.1: Enhanced Testing Framework**
|
||||
```javascript
|
||||
// Enhanced RefactorTestRunner with Python integration
|
||||
class EnhancedTestRunner extends RefactorTestRunner {
|
||||
reportToPython(results) {
|
||||
// JSON output for Python consumption
|
||||
}
|
||||
|
||||
runWithCoverage() {
|
||||
// Coverage reporting
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### **Step 4.2: Advanced Features**
|
||||
- Coverage reporting (Istanbul/nyc)
|
||||
- Performance benchmarks
|
||||
- Visual regression testing (optional)
|
||||
- Component documentation auto-generation
|
||||
|
||||
### **Phase 5: Production Integration** ⏱️ *~1-2 hours*
|
||||
|
||||
#### **Step 5.1: Main Test Suite Enhancement**
|
||||
```makefile
|
||||
# Add to main Makefile
|
||||
.PHONY: test-js
|
||||
test-js: ## Run JavaScript UI tests
|
||||
make testdrive-jsui-test-all
|
||||
|
||||
.PHONY: test-all
|
||||
test-all: test test-js test-capabilities ## Run all tests including JS
|
||||
```
|
||||
|
||||
#### **Step 5.2: CI/CD Integration**
|
||||
- Update test commands in main suite
|
||||
- Ensure capability auto-discovery works
|
||||
- Add JS test markers for selective running
|
||||
|
||||
## 🔒 **Safety Mechanisms**
|
||||
|
||||
### **Risk Mitigation:**
|
||||
|
||||
1. **📁 Copy-First Approach**: Never move, always copy initially
|
||||
2. **🔄 Dual Testing**: Run tests in both locations during migration
|
||||
3. **✅ Gradual Integration**: Add to main suite incrementally
|
||||
4. **🎯 Rollback Plan**: Keep original structure until 100% verified
|
||||
5. **🧪 Test Verification**: Compare results before/after migration
|
||||
|
||||
### **Success Criteria:**
|
||||
- ✅ All existing JS tests pass in new capability
|
||||
- ✅ Python integration tests pass
|
||||
- ✅ Main test suite still 100% green
|
||||
- ✅ JavaScript UI functionality unchanged
|
||||
- ✅ Performance maintained or improved
|
||||
|
||||
## 📋 **Implementation Checklist**
|
||||
|
||||
### **Phase 1: Foundation**
|
||||
- [ ] Create capability directory structure
|
||||
- [ ] Setup pyproject.toml with dependencies
|
||||
- [ ] Create package.json with Jest configuration
|
||||
- [ ] Implement Python-JavaScript bridge
|
||||
- [ ] Create capability Makefile
|
||||
- [ ] Write basic README documentation
|
||||
|
||||
### **Phase 2: Integration**
|
||||
- [ ] Create Python test wrappers for JS tests
|
||||
- [ ] Integrate with pytest discovery
|
||||
- [ ] Add capability targets to main Makefile
|
||||
- [ ] Test integration with main test suite
|
||||
|
||||
### **Phase 3: Migration**
|
||||
- [ ] Copy JavaScript files to capability
|
||||
- [ ] Verify tests work in new location
|
||||
- [ ] Create dual-track testing setup
|
||||
- [ ] Gradually integrate with main suite
|
||||
|
||||
### **Phase 4: Enhancement**
|
||||
- [ ] Enhance test framework with Python integration
|
||||
- [ ] Add coverage reporting
|
||||
- [ ] Performance benchmarking
|
||||
- [ ] Documentation generation
|
||||
|
||||
### **Phase 5: Production**
|
||||
- [ ] Full integration with main test suite
|
||||
- [ ] CI/CD pipeline updates
|
||||
- [ ] Final verification and cleanup
|
||||
|
||||
## ⚡ **Quick Start Option**
|
||||
|
||||
For immediate JavaScript test integration (30 minutes):
|
||||
|
||||
```python
|
||||
def test_javascript_ui_components():
|
||||
"""Run all JavaScript tests via subprocess"""
|
||||
import subprocess
|
||||
result = subprocess.run(['npm', 'test'],
|
||||
capture_output=True, text=True)
|
||||
assert result.returncode == 0, f"JS tests failed: {result.stderr}"
|
||||
```
|
||||
|
||||
## 🎯 **Expected Outcomes**
|
||||
|
||||
- **🔒 Zero-risk migration** with copy-first approach
|
||||
- **🧪 Enhanced testing** with Python integration
|
||||
- **📊 Better CI/CD** integration
|
||||
- **🏗️ Clean architecture** with capability isolation
|
||||
- **🚀 Future extensibility** for JavaScript framework evolution
|
||||
|
||||
## ⏱️ **Timeline**
|
||||
|
||||
**Total Estimated Time: 12-20 hours** (can be done incrementally)
|
||||
|
||||
**Recommended approach**: Start with Phase 1 for immediate value and safe migration path setup.
|
||||
|
||||
---
|
||||
|
||||
*Generated: 2025-11-09*
|
||||
*Status: Ready for Implementation*
|
||||
8
examples/content-generator/.gitignore
vendored
Normal file
8
examples/content-generator/.gitignore
vendored
Normal file
@@ -0,0 +1,8 @@
|
||||
# Generated files
|
||||
*.db
|
||||
*.mermaid
|
||||
*.dot
|
||||
|
||||
# Python cache
|
||||
__pycache__/
|
||||
*.pyc
|
||||
258
examples/content-generator/README.md
Normal file
258
examples/content-generator/README.md
Normal file
@@ -0,0 +1,258 @@
|
||||
# Content Generator Example
|
||||
|
||||
**Demonstrates: Prompt Dependency Resolution for Systematic Content Generation**
|
||||
|
||||
This example shows how to use MarkiTect's Prompt Dependency Resolution infrastructure to generate InfoTech primers with full dependency tracking, provenance tracing, and quality validation.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Run the example
|
||||
python generate_primers.py
|
||||
|
||||
# Or if executable:
|
||||
./generate_primers.py
|
||||
```
|
||||
|
||||
## What This Example Demonstrates
|
||||
|
||||
1. **Artifact Management**: Store templates, topics, and guidelines as versioned artifacts
|
||||
2. **Dependency Resolution**: Automatically resolve `@{macro}` references in templates
|
||||
3. **Provenance Tracking**: Trace generated content back to its inputs
|
||||
4. **Incremental Updates**: Detect changes and regenerate only affected outputs
|
||||
5. **Quality Validation**: Apply gates to ensure output meets standards
|
||||
6. **Visualization**: Export dependency graphs to Mermaid or DOT format
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
content-generator/
|
||||
├── README.md # This file
|
||||
├── TUTORIAL.md # Comprehensive tutorial (START HERE)
|
||||
├── generate_primers.py # Example implementation
|
||||
│
|
||||
├── templates/
|
||||
│ └── generate-primer.md # PromptTemplate for primer generation
|
||||
│
|
||||
├── artifacts/
|
||||
│ ├── topics/ # Topic definitions
|
||||
│ │ ├── etl.md
|
||||
│ │ └── microservices.md
|
||||
│ │
|
||||
│ └── guidelines/ # Authoring standards
|
||||
│ ├── authoring-rules.md
|
||||
│ └── research-prompt.md
|
||||
│
|
||||
└── prepdr/ # Original manual system (preserved)
|
||||
├── README.md
|
||||
├── ETL.md
|
||||
├── AuthoringRules.md
|
||||
├── AssistentPrompt.md
|
||||
└── GeneratePrimerTemplate.md
|
||||
```
|
||||
|
||||
## Before vs After
|
||||
|
||||
### Before (prepdr/ - Manual System)
|
||||
|
||||
```markdown
|
||||
# Manual macro substitution
|
||||
{{ETL}}
|
||||
{{AuthoringRules}}
|
||||
```
|
||||
|
||||
Problems:
|
||||
- No automation
|
||||
- No version tracking
|
||||
- No dependency awareness
|
||||
- Can't detect changes
|
||||
- No traceability
|
||||
|
||||
### After (With Infrastructure)
|
||||
|
||||
```markdown
|
||||
# Automatic resolution
|
||||
@{topic}
|
||||
@{authoring_rules}
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- ✅ Automatic macro resolution
|
||||
- ✅ SHA-256 content digests for change detection
|
||||
- ✅ Full dependency graph
|
||||
- ✅ Incremental recomputation
|
||||
- ✅ Complete provenance traces
|
||||
- ✅ CLI tools for inspection
|
||||
|
||||
## Tutorial
|
||||
|
||||
**[Read the full tutorial →](TUTORIAL.md)**
|
||||
|
||||
The tutorial covers:
|
||||
- Architecture and core concepts
|
||||
- Step-by-step implementation walkthrough
|
||||
- Advanced features (incremental execution, quality gates, visualization)
|
||||
- CLI usage examples
|
||||
- Best practices
|
||||
|
||||
## CLI Commands
|
||||
|
||||
After running the example, you can use CLI commands to inspect the system:
|
||||
|
||||
```bash
|
||||
# Trace provenance
|
||||
markitect prompt trace <artifact-id> --database primers.db
|
||||
|
||||
# Visualize dependencies
|
||||
markitect prompt graph <artifact-id> --format mermaid --database primers.db
|
||||
|
||||
# List runs
|
||||
markitect prompt runs --database primers.db
|
||||
|
||||
# Show impact debt (stale artifacts)
|
||||
markitect prompt debt --database primers.db
|
||||
|
||||
# Graph statistics
|
||||
markitect prompt stats --database primers.db
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
A[Topic: ETL] -->|requires| D[Run: generate-etl]
|
||||
B[Authoring Rules] -->|requires| D
|
||||
C[Research Prompt] -->|requires| D
|
||||
D -->|generates| E[Output: ETL Primer]
|
||||
|
||||
style E fill:#90EE90
|
||||
style D fill:#87CEEB
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Content-Based Addressing
|
||||
|
||||
Every artifact has a SHA-256 digest that changes when content changes:
|
||||
|
||||
```python
|
||||
artifact.content_digest # "9c4d6e8a..."
|
||||
artifact.has_changed(new_digest) # True if modified
|
||||
```
|
||||
|
||||
### 2. Dependency Graph
|
||||
|
||||
Full graph construction enables:
|
||||
- **Impact analysis**: "What needs regeneration?"
|
||||
- **Provenance**: "How was this produced?"
|
||||
- **Build order**: Topological sort for correct execution
|
||||
|
||||
### 3. Incremental Execution
|
||||
|
||||
Only regenerate what's affected by changes:
|
||||
|
||||
```python
|
||||
config = RecomputeConfig(
|
||||
max_depth=2, # Traverse 2 levels
|
||||
impact_threshold=0.1, # Skip minor changes
|
||||
max_recomputes=10, # Budget limit
|
||||
)
|
||||
```
|
||||
|
||||
### 4. Quality Validation
|
||||
|
||||
Apply automated checks:
|
||||
|
||||
```python
|
||||
gate = PatternValidationGate(
|
||||
required_patterns=[r"## Definition", r"## Context"],
|
||||
forbidden_patterns=[r"TODO", r"FIXME"],
|
||||
)
|
||||
```
|
||||
|
||||
## Extending the Example
|
||||
|
||||
### Add New Topics
|
||||
|
||||
```bash
|
||||
# 1. Create artifact file
|
||||
cat > artifacts/topics/oauth.md << 'EOF'
|
||||
---
|
||||
id: topic-oauth
|
||||
name: OAuth
|
||||
artifact_type: content
|
||||
---
|
||||
|
||||
# OAuth 2.0
|
||||
|
||||
An authorization framework that enables...
|
||||
EOF
|
||||
|
||||
# 2. Run generator
|
||||
python generate_primers.py
|
||||
# Automatically picks up new topic
|
||||
```
|
||||
|
||||
### Custom Quality Gates
|
||||
|
||||
```python
|
||||
from markitect.prompts.quality.gates.schema_gate import SchemaValidationGate
|
||||
|
||||
# Validate primer structure
|
||||
schema = {
|
||||
"type": "object",
|
||||
"required": ["definition", "context", "scope"],
|
||||
}
|
||||
|
||||
gate = SchemaValidationGate(schema=schema, gate_id="primer-schema")
|
||||
```
|
||||
|
||||
### Integrate with LLM
|
||||
|
||||
```python
|
||||
# Replace mock generation with real LLM call
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI()
|
||||
response = client.chat.completions.create(
|
||||
model="gpt-4",
|
||||
messages=[{"role": "user", "content": compiled.content}]
|
||||
)
|
||||
|
||||
output_content = response.choices[0].message.content
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
Database operations are optimized:
|
||||
- Indexed lookups on artifact ID, space, digest
|
||||
- Indexed dependency queries by source, target, run
|
||||
- Efficient graph traversal with BFS/DFS
|
||||
- Content digest comparison (no full content comparison)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Artifact not found"
|
||||
- Ensure artifact files exist in correct directories
|
||||
- Check file extension is `.md`
|
||||
- Verify space IDs match between template and config
|
||||
|
||||
### "Resolution failed"
|
||||
- Check macro names match artifact names
|
||||
- Verify spaces are configured in ResolutionConfig
|
||||
- Use `ResolutionStrategy.FIRST_MATCH` for simplicity
|
||||
|
||||
### "Circular dependency detected"
|
||||
- Review dependency edges in database
|
||||
- Use `detect_circular_dependencies()` to find cycles
|
||||
- Refactor template structure to break cycles
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- **[TUTORIAL.md](TUTORIAL.md)** - Comprehensive walkthrough
|
||||
- **[Prompt Dependency Resolution Spec](../../roadmap/prompt-dependency-resolution/)** - Design documentation
|
||||
- **[Phase 8 Implementation](../../markitect/prompts/)** - Source code
|
||||
|
||||
## License
|
||||
|
||||
Same as MarkiTect project.
|
||||
923
examples/content-generator/TUTORIAL.md
Normal file
923
examples/content-generator/TUTORIAL.md
Normal file
@@ -0,0 +1,923 @@
|
||||
# MarkiTect Prompt Dependency Resolution Tutorial
|
||||
|
||||
**Example: Generating InfoTech Primers with Full Provenance Tracking**
|
||||
|
||||
This tutorial demonstrates how to use MarkiTect's Prompt Dependency Resolution infrastructure to systematically generate content with complete dependency tracking, quality validation, and traceability.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Architecture](#architecture)
|
||||
3. [Setup](#setup)
|
||||
4. [Core Concepts](#core-concepts)
|
||||
5. [Step-by-Step Walkthrough](#step-by-step-walkthrough)
|
||||
6. [Advanced Features](#advanced-features)
|
||||
7. [CLI Usage](#cli-usage)
|
||||
8. [Best Practices](#best-practices)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
### What This Example Does
|
||||
|
||||
This example shows how to generate **InfoTech Primers** (structured reference documents for IT concepts) using a prompt template system with:
|
||||
|
||||
- **Artifact Management**: Store and version all inputs (templates, topics, guidelines)
|
||||
- **Dependency Resolution**: Automatically resolve macro references across information spaces
|
||||
- **Provenance Tracking**: Trace any generated primer back to its inputs and template
|
||||
- **Incremental Updates**: Detect when inputs change and regenerate affected primers
|
||||
- **Quality Validation**: Apply quality gates to ensure output meets standards
|
||||
- **Visualization**: View dependency graphs in DOT or Mermaid format
|
||||
|
||||
### Why Use Prompt Dependency Resolution?
|
||||
|
||||
**Before** (manual approach in `prepdr/`):
|
||||
```markdown
|
||||
# Template with manual macros
|
||||
{{topic}}
|
||||
{{AuthoringRules}}
|
||||
{{ResearchPrompt}}
|
||||
```
|
||||
|
||||
Problems:
|
||||
- Manual macro substitution
|
||||
- No version tracking
|
||||
- No dependency awareness
|
||||
- Can't detect when inputs change
|
||||
- No provenance traceability
|
||||
|
||||
**After** (with infrastructure):
|
||||
```markdown
|
||||
# Template with resolved dependencies
|
||||
@{topic}
|
||||
@{authoring_rules}
|
||||
@{research_prompt}
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- Automatic macro resolution
|
||||
- Content-based change detection (SHA-256 digests)
|
||||
- Full dependency graph construction
|
||||
- Incremental recomputation when inputs change
|
||||
- Complete provenance: artifact → template → inputs → validation
|
||||
- CLI commands for inspection and debugging
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Information Spaces
|
||||
|
||||
The system organizes artifacts into **information spaces** (logical namespaces):
|
||||
|
||||
```
|
||||
primer-templates/ # PromptTemplates for generation
|
||||
├─ generate-primer
|
||||
|
||||
primer-topics/ # Topic definitions (ETL, Microservices, OAuth, etc.)
|
||||
├─ etl
|
||||
├─ microservices
|
||||
└─ ...
|
||||
|
||||
primer-guidelines/ # Authoring and research guidelines
|
||||
├─ authoring-rules
|
||||
└─ research-prompt
|
||||
|
||||
generated-primers/ # Output artifacts
|
||||
├─ etl-primer
|
||||
├─ microservices-primer
|
||||
└─ ...
|
||||
```
|
||||
|
||||
### Dependency Graph
|
||||
|
||||
When you generate a primer, the system creates a dependency graph:
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[etl topic] -->|requires| B[generate-primer template]
|
||||
C[authoring-rules] -->|requires| B
|
||||
D[research-prompt] -->|requires| B
|
||||
B -->|generates| E[etl-primer output]
|
||||
```
|
||||
|
||||
This graph enables:
|
||||
- **Impact analysis**: "What primers need regeneration if authoring-rules changes?"
|
||||
- **Provenance tracing**: "What inputs produced this primer?"
|
||||
- **Incremental execution**: "Only regenerate affected primers"
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
### Prerequisites
|
||||
|
||||
```bash
|
||||
# Ensure MarkiTect is installed
|
||||
cd /path/to/markitect-main
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
examples/content-generator/
|
||||
├── TUTORIAL.md # This file
|
||||
├── generate_primers.py # Main example script
|
||||
├── templates/
|
||||
│ └── generate-primer.md # PromptTemplate
|
||||
├── artifacts/
|
||||
│ ├── topics/
|
||||
│ │ ├── etl.md # Topic: ETL
|
||||
│ │ └── microservices.md # Topic: Microservices
|
||||
│ └── guidelines/
|
||||
│ ├── authoring-rules.md # Authoring standards
|
||||
│ └── research-prompt.md # Research methodology
|
||||
└── prepdr/ # Original manual system (preserved)
|
||||
├── README.md
|
||||
├── ETL.md
|
||||
├── AuthoringRules.md
|
||||
├── AssistentPrompt.md
|
||||
└── GeneratePrimerTemplate.md
|
||||
```
|
||||
|
||||
### Running the Example
|
||||
|
||||
```bash
|
||||
cd examples/content-generator
|
||||
python generate_primers.py
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
╔══════════════════════════════════════════════════════════════╗
|
||||
║ MarkiTect Prompt Dependency Resolution Example ║
|
||||
║ InfoTech Primer Generation ║
|
||||
╚══════════════════════════════════════════════════════════════╝
|
||||
|
||||
=== Loading Artifacts ===
|
||||
✓ Created artifact: generate-primer (digest: a7f3e2b1)
|
||||
✓ Created artifact: etl (digest: 9c4d6e8a)
|
||||
✓ Created artifact: microservices (digest: 5b2f1c9d)
|
||||
✓ Created artifact: authoring-rules (digest: 3e7a9f2c)
|
||||
✓ Created artifact: research-prompt (digest: 8d1b4e6f)
|
||||
|
||||
=== Generating Primer: etl ===
|
||||
✓ Template created with 3 macro dependencies
|
||||
✓ Resolved 3 macros
|
||||
✓ Compiled prompt (digest: 4c9e2a7b)
|
||||
✓ Persisted 3 dependency edges
|
||||
✓ Generated primer: etl-primer
|
||||
|
||||
=== Provenance Trace ===
|
||||
Artifact: abc-123-def-456
|
||||
Producing Run: run-etl-001
|
||||
Input Artifacts: 3
|
||||
Dependency Chain: 5 artifacts
|
||||
|
||||
✓ Primer generation complete!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Artifacts
|
||||
|
||||
**Artifacts** are versioned content units with content-based addressing.
|
||||
|
||||
```python
|
||||
from markitect.prompts.models import Artifact, ArtifactType
|
||||
|
||||
# Create an artifact
|
||||
artifact = Artifact.create(
|
||||
space_id="primer-topics",
|
||||
name="etl",
|
||||
content=topic_content,
|
||||
artifact_type=ArtifactType.CONTENT,
|
||||
)
|
||||
|
||||
# Automatic SHA-256 digest generation
|
||||
print(artifact.content_digest) # "9c4d6e8a..."
|
||||
```
|
||||
|
||||
**Key features:**
|
||||
- **Content digest**: SHA-256 hash for change detection
|
||||
- **Space isolation**: Artifacts in different spaces can have same names
|
||||
- **Type classification**: CONTENT, TEMPLATE, GENERATED, SCHEMA, CONFIG
|
||||
|
||||
### 2. PromptTemplates
|
||||
|
||||
**PromptTemplates** are artifacts with macro references.
|
||||
|
||||
```markdown
|
||||
---
|
||||
id: generate-primer-v1
|
||||
artifact_type: template
|
||||
---
|
||||
|
||||
# Generate Primer
|
||||
|
||||
Topic: @{topic}
|
||||
Guidelines: @{authoring_rules}
|
||||
```
|
||||
|
||||
**Macro syntax:**
|
||||
- `@{macro_name}` - Resolved to artifact content
|
||||
- Resolution happens at execution time
|
||||
- Macros can reference artifacts in any information space
|
||||
|
||||
### 3. Resolution Strategy
|
||||
|
||||
**Resolution** finds artifacts to substitute for macros.
|
||||
|
||||
```python
|
||||
from markitect.prompts.resolver.strategy import ResolutionConfig, ResolutionStrategy
|
||||
|
||||
config = ResolutionConfig(
|
||||
strategy=ResolutionStrategy.FIRST_MATCH,
|
||||
spaces=["primer-topics", "primer-guidelines"],
|
||||
)
|
||||
```
|
||||
|
||||
**Strategies:**
|
||||
- `FIRST_MATCH`: Use first artifact found
|
||||
- `LATEST_VERSION`: Use newest version (if artifacts have versions)
|
||||
- `EXPLICIT_ONLY`: Require explicit space qualification
|
||||
|
||||
### 4. Dependency Tracking
|
||||
|
||||
**Dependency edges** are automatically created during resolution.
|
||||
|
||||
```python
|
||||
# Edge types
|
||||
EdgeType.REQUIRES # Input dependency (template → topic)
|
||||
EdgeType.GENERATES # Output relationship (run → primer)
|
||||
EdgeType.INCLUDES # Composition (nested templates)
|
||||
```
|
||||
|
||||
**Graph operations:**
|
||||
```python
|
||||
# Find all artifacts that depend on authoring-rules
|
||||
dependents = query_service.find_transitive_dependents("authoring-rules-id")
|
||||
|
||||
# Find all inputs needed to regenerate a primer
|
||||
dependencies = query_service.find_transitive_dependencies("etl-primer-id")
|
||||
|
||||
# Detect circular dependencies
|
||||
cycles = query_service.detect_circular_dependencies()
|
||||
```
|
||||
|
||||
### 5. Traceability
|
||||
|
||||
**ProvenanceTrace** captures complete lineage.
|
||||
|
||||
```python
|
||||
trace = trace_service.trace_artifact(artifact_id)
|
||||
|
||||
print(trace.producing_run) # Run that generated this
|
||||
print(trace.template) # Template used
|
||||
print(trace.input_artifacts) # All input dependencies
|
||||
print(trace.validation_results) # Quality gate results
|
||||
print(trace.impact_debt) # Suppressed recomputations
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step-by-Step Walkthrough
|
||||
|
||||
### Step 1: Initialize Repositories
|
||||
|
||||
```python
|
||||
from markitect.prompts.repositories.sqlite import SQLiteArtifactRepository
|
||||
from markitect.prompts.dependencies.repository import SQLiteDependencyRepository
|
||||
|
||||
artifact_repo = SQLiteArtifactRepository("primers.db")
|
||||
dep_repo = SQLiteDependencyRepository("primers.db")
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
- Creates SQLite database with artifact and dependency tables
|
||||
- Artifact table: id, space_id, name, content_digest, metadata
|
||||
- Dependency table: source_id, target_id, edge_type, run_id
|
||||
|
||||
### Step 2: Load Artifacts
|
||||
|
||||
```python
|
||||
# Read artifact file
|
||||
content = Path("artifacts/topics/etl.md").read_text()
|
||||
|
||||
# Create artifact
|
||||
artifact = Artifact.create(
|
||||
space_id="primer-topics",
|
||||
name="etl",
|
||||
content=content,
|
||||
artifact_type=ArtifactType.CONTENT,
|
||||
)
|
||||
|
||||
# Store in repository
|
||||
artifact = artifact_repo.create(artifact)
|
||||
```
|
||||
|
||||
**Content-based addressing:**
|
||||
```python
|
||||
# If you modify the content
|
||||
updated_content = content + "\n\n**New section added**"
|
||||
artifact.update_content(updated_content)
|
||||
|
||||
# Digest changes automatically
|
||||
print(artifact.content_digest) # Different hash!
|
||||
```
|
||||
|
||||
### Step 3: Create PromptTemplate
|
||||
|
||||
```python
|
||||
from markitect.prompts.templates.models import PromptTemplate, MacroReference
|
||||
|
||||
template = PromptTemplate.create(
|
||||
id="generate-primer-v1",
|
||||
name="generate-primer",
|
||||
content=template_content,
|
||||
space_id="primer-templates",
|
||||
)
|
||||
|
||||
# Add macro dependencies
|
||||
template.add_macro(MacroReference(
|
||||
name="topic",
|
||||
source_space="primer-topics"
|
||||
))
|
||||
template.add_macro(MacroReference(
|
||||
name="authoring_rules",
|
||||
source_space="primer-guidelines"
|
||||
))
|
||||
```
|
||||
|
||||
**Template content** (`templates/generate-primer.md`):
|
||||
```markdown
|
||||
# Generate InfoTech Primer
|
||||
|
||||
## Topic
|
||||
@{topic}
|
||||
|
||||
## Guidelines
|
||||
@{authoring_rules}
|
||||
|
||||
## Research Protocol
|
||||
@{research_prompt}
|
||||
|
||||
Generate a complete primer following the authoring rules.
|
||||
```
|
||||
|
||||
### Step 4: Resolve Dependencies
|
||||
|
||||
```python
|
||||
from markitect.prompts.resolver.resolver import PromptResolver
|
||||
from markitect.prompts.resolver.strategy import ResolutionConfig
|
||||
|
||||
resolver = PromptResolver(artifact_repo)
|
||||
|
||||
config = ResolutionConfig(
|
||||
strategy=ResolutionStrategy.FIRST_MATCH,
|
||||
spaces=["primer-topics", "primer-guidelines"],
|
||||
)
|
||||
|
||||
resolution_result = resolver.resolve_template(template, config)
|
||||
|
||||
if resolution_result.success:
|
||||
for resolved in resolution_result.context.resolved_macros:
|
||||
print(f"{resolved.macro_name} → {resolved.artifact.name}")
|
||||
else:
|
||||
print("Resolution failed:", resolution_result.context.errors)
|
||||
```
|
||||
|
||||
**Resolution algorithm:**
|
||||
1. Parse template to extract `@{macro_name}` references
|
||||
2. For each macro:
|
||||
- Search configured spaces in order
|
||||
- Match by name (case-sensitive)
|
||||
- Return first match (FIRST_MATCH strategy)
|
||||
3. Build ResolutionResult with all resolved artifacts
|
||||
|
||||
### Step 5: Compile Prompt
|
||||
|
||||
```python
|
||||
from markitect.prompts.resolver.compiler import ContextCompiler
|
||||
|
||||
compiler = ContextCompiler()
|
||||
compiled = compiler.compile(template, template_content, resolution_result)
|
||||
|
||||
print(compiled.content) # Fully expanded prompt
|
||||
print(compiled.content_digest) # Hash for caching
|
||||
print(compiled.dependency_digests) # Map of macro → artifact digest
|
||||
```
|
||||
|
||||
**Compiled output:**
|
||||
```markdown
|
||||
# Generate InfoTech Primer
|
||||
|
||||
## Topic
|
||||
A three-phase computing process where data is extracted from source systems,
|
||||
transformed (including validation, cleaning, enrichment, and aggregation),
|
||||
and loaded into a target data store or data warehouse.
|
||||
...
|
||||
|
||||
## Guidelines
|
||||
[Full authoring-rules content]
|
||||
...
|
||||
|
||||
## Research Protocol
|
||||
[Full research-prompt content]
|
||||
...
|
||||
```
|
||||
|
||||
### Step 6: Track Dependencies
|
||||
|
||||
```python
|
||||
from markitect.prompts.execution.manifest import RunManifest
|
||||
from markitect.prompts.dependencies.graph import GraphBuilder
|
||||
|
||||
# Create run manifest
|
||||
manifest = RunManifest.create(
|
||||
run_id="run-etl-001",
|
||||
template_id=template.id,
|
||||
template_name=template.name,
|
||||
template_digest=template.content_digest,
|
||||
)
|
||||
|
||||
# Add resolved inputs
|
||||
for resolved in resolution_result.context.resolved_macros:
|
||||
manifest.add_resolved_input(
|
||||
name=resolved.macro_name,
|
||||
artifact_id=resolved.artifact.id,
|
||||
space_id=resolved.space_id,
|
||||
digest=resolved.artifact.content_digest,
|
||||
)
|
||||
|
||||
# Create dependency edge
|
||||
manifest.add_dependency_edge(
|
||||
source_id=resolved.artifact.id,
|
||||
target_id="run-etl-001",
|
||||
edge_type="requires",
|
||||
)
|
||||
|
||||
# Persist to database
|
||||
builder = GraphBuilder(dep_repo)
|
||||
edges = builder.persist_edges(manifest)
|
||||
```
|
||||
|
||||
**Result:** Dependency edges stored in database:
|
||||
```
|
||||
source_artifact_id | target_artifact_id | edge_type | run_id
|
||||
--------------------|--------------------|-----------|-----------
|
||||
etl-id | run-etl-001 | requires | run-etl-001
|
||||
authoring-rules-id | run-etl-001 | requires | run-etl-001
|
||||
research-prompt-id | run-etl-001 | requires | run-etl-001
|
||||
```
|
||||
|
||||
### Step 7: Generate Output
|
||||
|
||||
```python
|
||||
# In real usage, this would call an LLM API
|
||||
# For demo, we create a mock output
|
||||
output_content = """
|
||||
# ETL Primer
|
||||
|
||||
## Definition
|
||||
ETL (Extract, Transform, Load) is a data integration pattern...
|
||||
[Generated content]
|
||||
"""
|
||||
|
||||
output_artifact = Artifact.create(
|
||||
space_id="generated-primers",
|
||||
name="etl-primer",
|
||||
content=output_content,
|
||||
artifact_type=ArtifactType.GENERATED,
|
||||
)
|
||||
output_artifact = artifact_repo.create(output_artifact)
|
||||
|
||||
# Add to manifest
|
||||
manifest.add_output_artifact(
|
||||
artifact_id=output_artifact.id,
|
||||
name=output_artifact.name,
|
||||
digest=output_artifact.content_digest,
|
||||
artifact_type="generated",
|
||||
)
|
||||
|
||||
manifest.add_dependency_edge(
|
||||
source_id="run-etl-001",
|
||||
target_id=output_artifact.id,
|
||||
edge_type="generates",
|
||||
)
|
||||
|
||||
# Persist output edges
|
||||
builder.persist_edges(manifest)
|
||||
```
|
||||
|
||||
### Step 8: Trace Provenance
|
||||
|
||||
```python
|
||||
from markitect.prompts.traceability.service import TraceabilityService
|
||||
|
||||
trace_service = TraceabilityService(artifact_repo, dep_repo, db_path="primers.db")
|
||||
|
||||
# Trace the generated primer
|
||||
trace = trace_service.trace_artifact(output_artifact.id)
|
||||
|
||||
# Inspect provenance
|
||||
print("Template:", trace.template.name if trace.template else "None")
|
||||
print("Producing run:", trace.producing_run.run_id if trace.producing_run else "None")
|
||||
print("Input artifacts:")
|
||||
for inp in trace.input_artifacts:
|
||||
print(f" - {inp.name} ({inp.artifact_type})")
|
||||
|
||||
print("Dependency chain:")
|
||||
for dep_id in trace.dependency_chain:
|
||||
artifact = artifact_repo.get_by_id(dep_id)
|
||||
print(f" - {artifact.name if artifact else dep_id}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Incremental Recomputation
|
||||
|
||||
When an input changes, automatically detect affected outputs:
|
||||
|
||||
```python
|
||||
from markitect.prompts.incremental.detector import ChangeDetector
|
||||
from markitect.prompts.incremental.engine import IncrementalExecutionEngine
|
||||
from markitect.prompts.incremental.models import RecomputeConfig
|
||||
|
||||
# Detect change
|
||||
detector = ChangeDetector("primers.db")
|
||||
authoring_rules = artifact_repo.get_by_name("primer-guidelines", "authoring-rules")
|
||||
|
||||
# User updates the file
|
||||
new_content = Path("artifacts/guidelines/authoring-rules.md").read_text()
|
||||
change = detector.detect_change(authoring_rules, new_content)
|
||||
|
||||
if change:
|
||||
detector.record_change(change)
|
||||
|
||||
# Find affected primers
|
||||
engine = IncrementalExecutionEngine("primers.db", query_service)
|
||||
result = engine.recompute(
|
||||
change,
|
||||
config=RecomputeConfig(max_depth=2, impact_threshold=0.1),
|
||||
old_content=authoring_rules.content,
|
||||
new_content=new_content,
|
||||
)
|
||||
|
||||
print(f"Total dependents: {result.total_dependents}")
|
||||
print(f"Recomputed: {result.recomputed_count}")
|
||||
print(f"Suppressed: {result.suppressed_count}")
|
||||
```
|
||||
|
||||
**Recomputation strategies:**
|
||||
- **max_depth**: Traverse dependency graph N levels
|
||||
- **impact_threshold**: Only recompute if change magnitude > threshold
|
||||
- **max_recomputes**: Budget limit to prevent runaway execution
|
||||
|
||||
### Quality Validation
|
||||
|
||||
Apply quality gates to generated primers:
|
||||
|
||||
```python
|
||||
from markitect.prompts.quality.validator import QualityValidator
|
||||
from markitect.prompts.quality.gates.pattern_gate import PatternValidationGate
|
||||
|
||||
# Create validation gate
|
||||
gate = PatternValidationGate(
|
||||
required_patterns=[
|
||||
r"## Definition",
|
||||
r"## Context",
|
||||
r"## Core Concepts",
|
||||
r"## Scope and Non-Scope",
|
||||
],
|
||||
forbidden_patterns=[
|
||||
r"TODO",
|
||||
r"FIXME",
|
||||
],
|
||||
gate_id="primer-structure-check",
|
||||
name="Primer Structure Validator",
|
||||
)
|
||||
|
||||
validator = QualityValidator(gates=[gate], db_path="primers.db")
|
||||
|
||||
# Validate output
|
||||
results = validator.validate_artifact(
|
||||
content=output_content,
|
||||
artifact_id=output_artifact.id,
|
||||
run_id="run-etl-001",
|
||||
)
|
||||
|
||||
if validator.all_passed(results):
|
||||
print("✓ All quality gates passed")
|
||||
else:
|
||||
failed = validator.get_failed_gates(results)
|
||||
for result in failed:
|
||||
print(f"✗ {result.gate_id} failed")
|
||||
for diag in result.diagnostics:
|
||||
print(f" {diag.message}")
|
||||
```
|
||||
|
||||
### Visualization
|
||||
|
||||
Generate dependency graphs:
|
||||
|
||||
```python
|
||||
from markitect.prompts.visualization.graph import GraphExporter
|
||||
from markitect.prompts.dependencies.queries import DependencyQueryService
|
||||
|
||||
query_service = DependencyQueryService(dep_repo)
|
||||
|
||||
# Find all related artifacts
|
||||
deps = query_service.find_transitive_dependencies(output_artifact.id)
|
||||
dependents = query_service.find_transitive_dependents(output_artifact.id)
|
||||
all_ids = deps | dependents | {output_artifact.id}
|
||||
|
||||
# Build graph
|
||||
builder = GraphBuilder(dep_repo)
|
||||
graph = builder.build_graph(all_ids)
|
||||
|
||||
# Export to Mermaid
|
||||
mermaid = GraphExporter.to_mermaid(graph, "Primer Dependencies")
|
||||
Path("dependencies.mermaid").write_text(mermaid)
|
||||
|
||||
# Export to DOT (Graphviz)
|
||||
dot = GraphExporter.to_dot(graph, "Primer Dependencies")
|
||||
Path("dependencies.dot").write_text(dot)
|
||||
```
|
||||
|
||||
**Mermaid output:**
|
||||
```mermaid
|
||||
%%{ title: Primer Dependencies }%%
|
||||
graph LR
|
||||
etl-id-->|requires|run-etl-001
|
||||
authoring-rules-id-->|requires|run-etl-001
|
||||
research-prompt-id-->|requires|run-etl-001
|
||||
run-etl-001-.->|generates|etl-primer-id
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CLI Usage
|
||||
|
||||
The Prompt Dependency Resolution infrastructure includes CLI commands:
|
||||
|
||||
### Trace Provenance
|
||||
|
||||
```bash
|
||||
markitect prompt trace <artifact-id> --database primers.db
|
||||
```
|
||||
|
||||
Output (JSON):
|
||||
```json
|
||||
{
|
||||
"artifact_id": "abc-123-def-456",
|
||||
"producing_run": {
|
||||
"run_id": "run-etl-001",
|
||||
"template_id": "generate-primer-v1",
|
||||
"status": "success"
|
||||
},
|
||||
"input_artifacts": [
|
||||
{
|
||||
"artifact_id": "...",
|
||||
"name": "etl",
|
||||
"role": "input"
|
||||
}
|
||||
],
|
||||
"dependency_chain": ["...", "..."]
|
||||
}
|
||||
```
|
||||
|
||||
### Visualize Graph
|
||||
|
||||
```bash
|
||||
markitect prompt graph <artifact-id> --format mermaid --database primers.db
|
||||
```
|
||||
|
||||
### List Runs
|
||||
|
||||
```bash
|
||||
# All runs
|
||||
markitect prompt runs --database primers.db
|
||||
|
||||
# Filter by template
|
||||
markitect prompt runs --template generate-primer-v1 --database primers.db
|
||||
|
||||
# Filter by status
|
||||
markitect prompt runs --status success --limit 10 --database primers.db
|
||||
```
|
||||
|
||||
### Show Impact Debt
|
||||
|
||||
```bash
|
||||
# All stale artifacts
|
||||
markitect prompt debt --database primers.db
|
||||
|
||||
# Specific artifact
|
||||
markitect prompt debt --artifact authoring-rules-id --database primers.db
|
||||
```
|
||||
|
||||
### Graph Statistics
|
||||
|
||||
```bash
|
||||
markitect prompt stats --database primers.db
|
||||
```
|
||||
|
||||
Output:
|
||||
```json
|
||||
{
|
||||
"total_nodes": 12,
|
||||
"total_edges": 18,
|
||||
"root_count": 3,
|
||||
"leaf_count": 2,
|
||||
"has_cycles": false
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Organize Artifacts by Space
|
||||
|
||||
```
|
||||
Clear separation of concerns:
|
||||
- templates/ ← Reusable PromptTemplates
|
||||
- topics/ ← Domain-specific content
|
||||
- guidelines/ ← Standards and methodologies
|
||||
- output/ ← Generated artifacts
|
||||
```
|
||||
|
||||
### 2. Use Content Digests for Change Detection
|
||||
|
||||
```python
|
||||
# Don't compare content strings
|
||||
if old_content != new_content: # ✗ Inefficient
|
||||
|
||||
# Do compare digests
|
||||
if artifact.has_changed(new_digest): # ✓ Fast, hash-based
|
||||
```
|
||||
|
||||
### 3. Apply Quality Gates
|
||||
|
||||
```python
|
||||
# Define quality standards as code
|
||||
gates = [
|
||||
PatternValidationGate(required_patterns=[...]),
|
||||
SchemaValidationGate(schema={...}),
|
||||
]
|
||||
|
||||
# Fail fast if quality checks fail
|
||||
if not validator.all_passed(results):
|
||||
raise QualityError("Output does not meet standards")
|
||||
```
|
||||
|
||||
### 4. Track All Dependencies
|
||||
|
||||
```python
|
||||
# Always persist dependency edges
|
||||
manifest.add_dependency_edge(source, target, edge_type)
|
||||
builder.persist_edges(manifest)
|
||||
|
||||
# This enables:
|
||||
# - Impact analysis
|
||||
# - Incremental recomputation
|
||||
# - Provenance tracing
|
||||
```
|
||||
|
||||
### 5. Use Incremental Execution
|
||||
|
||||
```python
|
||||
# Don't regenerate everything on every change
|
||||
config = RecomputeConfig(
|
||||
max_depth=2, # Limit blast radius
|
||||
impact_threshold=0.1, # Skip minor changes
|
||||
max_recomputes=10, # Budget limit
|
||||
)
|
||||
```
|
||||
|
||||
### 6. Version Your Templates
|
||||
|
||||
```python
|
||||
# Include version in template metadata
|
||||
---
|
||||
id: generate-primer-v1
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
# When template changes significantly, create v2
|
||||
---
|
||||
id: generate-primer-v2
|
||||
version: 2.0.0
|
||||
---
|
||||
```
|
||||
|
||||
### 7. Leverage Traceability
|
||||
|
||||
```python
|
||||
# Use provenance traces for debugging
|
||||
trace = trace_service.trace_artifact(failed_output_id)
|
||||
|
||||
print("Inputs used:")
|
||||
for inp in trace.input_artifacts:
|
||||
print(f" {inp.name} @ {inp.content_digest[:8]}")
|
||||
|
||||
# This helps identify which input caused the issue
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Comparison with Original System
|
||||
|
||||
### Original (`prepdr/`)
|
||||
|
||||
**GeneratePrimerTemplate.md:**
|
||||
```markdown
|
||||
<topic>
|
||||
{{ETL}}
|
||||
</topic>
|
||||
|
||||
<guidance>
|
||||
{{AuthoringRules}}
|
||||
</guidance>
|
||||
```
|
||||
|
||||
**Process:**
|
||||
1. Manually copy-paste content
|
||||
2. Replace `{{...}}` markers by hand
|
||||
3. Run through LLM
|
||||
4. No record of what inputs were used
|
||||
5. No change detection
|
||||
|
||||
**Limitations:**
|
||||
- No automation
|
||||
- No version control on inputs
|
||||
- Can't regenerate from history
|
||||
- No impact analysis when guidelines change
|
||||
|
||||
### With Infrastructure
|
||||
|
||||
**templates/generate-primer.md:**
|
||||
```markdown
|
||||
@{topic}
|
||||
@{authoring_rules}
|
||||
```
|
||||
|
||||
**Process:**
|
||||
1. Define artifacts once
|
||||
2. Create template with `@{...}` macros
|
||||
3. Run resolver → compiler → executor
|
||||
4. Full dependency graph persisted
|
||||
5. Complete provenance trace available
|
||||
|
||||
**Benefits:**
|
||||
- Fully automated resolution
|
||||
- Content-based change detection (SHA-256)
|
||||
- Reproducible: "same inputs → same output"
|
||||
- Impact analysis: "what needs regeneration?"
|
||||
- Traceability: "how was this generated?"
|
||||
- Quality validation: automated checks
|
||||
- Visualization: see dependency relationships
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Extend the example:**
|
||||
- Add more topics (OAuth, Docker, Kubernetes)
|
||||
- Create topic-specific quality gates
|
||||
- Implement actual LLM integration
|
||||
|
||||
2. **Build a workflow:**
|
||||
- Git hooks to detect artifact changes
|
||||
- CI/CD pipeline to regenerate affected primers
|
||||
- Dashboard to show primer freshness
|
||||
|
||||
3. **Add advanced features:**
|
||||
- Version conflict resolution
|
||||
- A/B testing different templates
|
||||
- Batch generation with parallelization
|
||||
|
||||
4. **Integrate with MarkiTect:**
|
||||
- Use MarkiTect ingestion for artifact storage
|
||||
- Query relationships with relational metadata
|
||||
- Generate documentation sites from primers
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Prompt Dependency Resolution Specification](../../roadmap/prompt-dependency-resolution/)
|
||||
- [MarkiTect Documentation](../../README.md)
|
||||
- [Phase 8 Implementation](../../markitect/prompts/)
|
||||
|
||||
---
|
||||
|
||||
**Questions or feedback?** File an issue or reach out to the maintainers.
|
||||
@@ -0,0 +1,200 @@
|
||||
---
|
||||
id: primer-authoring-rules-v1
|
||||
name: AuthoringRules
|
||||
artifact_type: content
|
||||
description: Comprehensive guidelines for writing effective InfoTech primers
|
||||
version: 1.0.0
|
||||
tags:
|
||||
- guidelines
|
||||
- authoring
|
||||
- quality-standards
|
||||
---
|
||||
|
||||
# Primer Authoring Rules
|
||||
|
||||
**Status:** Draft
|
||||
**Intended Audience:** Human authors and AI systems generating or validating primers
|
||||
**Purpose:** Ensure primers are precise, stable, and suitable as shared context for humans and AI agents
|
||||
|
||||
---
|
||||
|
||||
## 1. What a Primer Is (Normative)
|
||||
|
||||
An **InfoTechPrimer** is a **short, structured reference document** that establishes a **shared understanding** of a specific IT term, standard, method, or concept.
|
||||
|
||||
A primer:
|
||||
* Defines **what the topic is**
|
||||
* Explains **where it fits**
|
||||
* Clarifies **scope boundaries**
|
||||
* Points to **authoritative sources**
|
||||
|
||||
A primer does **not**:
|
||||
* Teach step-by-step usage
|
||||
* Advocate tools or vendors
|
||||
* Explore implementation details beyond what is normatively defined
|
||||
|
||||
---
|
||||
|
||||
## 2. Target Audience
|
||||
|
||||
Primers are written for:
|
||||
* Humans with solid general IT knowledge
|
||||
* Readers who are *not specialists* in the specific topic
|
||||
* AI systems that consume structured context for reasoning and coding
|
||||
|
||||
Authors must assume:
|
||||
* Conceptual literacy
|
||||
* Familiarity with basic IT terminology
|
||||
* No prior deep knowledge of the topic
|
||||
|
||||
---
|
||||
|
||||
## 3. Required Structure (Mandatory)
|
||||
|
||||
Every primer **MUST** contain the following sections **in this order**:
|
||||
|
||||
1. **Definition**
|
||||
2. **Context**
|
||||
3. **Core Concepts**
|
||||
4. **Scope and Non-Scope**
|
||||
5. **Practical Implications**
|
||||
6. **Formal Standards and Authoritative Sources**
|
||||
7. **Related Concepts**
|
||||
|
||||
No section may be omitted. Empty sections are not allowed.
|
||||
|
||||
---
|
||||
|
||||
## 4. Section Authoring Rules
|
||||
|
||||
### 4.1 Definition
|
||||
**Purpose:** Establish an unambiguous baseline meaning.
|
||||
|
||||
Rules:
|
||||
* 2–4 sentences maximum
|
||||
* Declarative, precise language
|
||||
* No metaphors, examples, or analogies
|
||||
* No historical narrative
|
||||
|
||||
### 4.2 Context
|
||||
**Purpose:** Position the concept within the IT landscape.
|
||||
|
||||
Rules:
|
||||
* Describe the domain(s) the concept belongs to
|
||||
* Explain *why it exists*, not *how to use it*
|
||||
* Historical notes allowed only if they clarify intent or constraints
|
||||
|
||||
### 4.3 Core Concepts
|
||||
**Purpose:** Identify the irreducible ideas that define the topic.
|
||||
|
||||
Rules:
|
||||
* Bullet points only
|
||||
* Each bullet describes one concept
|
||||
* No nested lists
|
||||
* Avoid redundancy with Definition
|
||||
|
||||
### 4.4 Scope and Non-Scope
|
||||
**Purpose:** Prevent conceptual drift and misuse.
|
||||
|
||||
Rules:
|
||||
* Explicitly list inclusions and exclusions
|
||||
* Use parallel structure
|
||||
* Address common misconceptions
|
||||
|
||||
Format:
|
||||
```markdown
|
||||
**In Scope**
|
||||
- ...
|
||||
|
||||
**Out of Scope**
|
||||
- ...
|
||||
```
|
||||
|
||||
This section is **critical** for AI agent correctness.
|
||||
|
||||
### 4.5 Practical Implications
|
||||
**Purpose:** Describe consequences of adopting or interacting with the concept.
|
||||
|
||||
Rules:
|
||||
* Focus on effects, not instructions
|
||||
* No step-by-step guidance
|
||||
* Include tradeoffs where relevant
|
||||
|
||||
### 4.6 Formal Standards and Authoritative Sources
|
||||
**Purpose:** Anchor the primer in canonical truth.
|
||||
|
||||
Rules:
|
||||
* Prefer primary sources
|
||||
* Include direct links
|
||||
* Avoid blogs unless widely recognized and necessary
|
||||
|
||||
Acceptable sources:
|
||||
* RFCs, W3C Recommendations
|
||||
* ISO / IEC standards
|
||||
* NIST publications
|
||||
* Official specifications
|
||||
* Foundational academic papers
|
||||
|
||||
At least **one** authoritative source is required.
|
||||
|
||||
### 4.7 Related Concepts
|
||||
**Purpose:** Enable semantic navigation.
|
||||
|
||||
Rules:
|
||||
* Short descriptions only (one line per concept)
|
||||
* No deep explanations
|
||||
* Avoid circular definitions
|
||||
|
||||
---
|
||||
|
||||
## 5. Language and Style Rules
|
||||
|
||||
Mandatory:
|
||||
* Present tense
|
||||
* Declarative sentences
|
||||
* Neutral, technical tone
|
||||
|
||||
Avoid:
|
||||
* First-person language ("we", "you")
|
||||
* Rhetorical questions
|
||||
* Marketing language
|
||||
* Informal phrasing
|
||||
* Emojis
|
||||
|
||||
---
|
||||
|
||||
## 6. Length Constraints
|
||||
|
||||
A primer should typically be:
|
||||
* **600–1,000 words total**
|
||||
* Short enough to be read in one sitting
|
||||
* Long enough to define boundaries clearly
|
||||
|
||||
Exceeding this range requires justification.
|
||||
|
||||
---
|
||||
|
||||
## 7. AI Optimization Rules (Explicit)
|
||||
|
||||
Authors **SHOULD**:
|
||||
* Use consistent terminology
|
||||
* Avoid synonyms for core terms once defined
|
||||
* Prefer explicit over implicit assumptions
|
||||
* State constraints clearly
|
||||
|
||||
Authors **MUST NOT**:
|
||||
* Rely on context outside the document
|
||||
* Assume tool- or framework-specific defaults
|
||||
* Leave ambiguity where standards are explicit
|
||||
|
||||
---
|
||||
|
||||
## 8. Validation Criteria (Checklist)
|
||||
|
||||
A primer is valid if:
|
||||
* [ ] All required sections are present
|
||||
* [ ] Definition is precise and unambiguous
|
||||
* [ ] Scope boundaries are explicit
|
||||
* [ ] At least one authoritative source is linked
|
||||
* [ ] No tutorial or marketing content exists
|
||||
* [ ] Language follows declarative style rules
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
id: research-protocol-v1
|
||||
name: ResearchPrompt
|
||||
artifact_type: content
|
||||
description: Systematic research protocol for InfoTech topic investigation
|
||||
version: 1.0.0
|
||||
tags:
|
||||
- research
|
||||
- methodology
|
||||
- guidelines
|
||||
---
|
||||
|
||||
# InfoTech Research Protocol
|
||||
|
||||
Below is a systematic research protocol to thoroughly investigate any InfoTech topic before writing an InfoTechPrimer.
|
||||
|
||||
**Purpose:** Produce a factually grounded, scope-aware, source-anchored research brief suitable as direct input for primer authoring.
|
||||
|
||||
---
|
||||
|
||||
## Research Sections
|
||||
|
||||
### 1. Canonical Definition
|
||||
- Provide the most widely accepted definition(s) of the topic
|
||||
- If multiple definitions exist, explain why and in which contexts they differ
|
||||
- Prefer definitions from standards bodies, original designers, or official specifications
|
||||
|
||||
### 2. Domain Context and Classification
|
||||
- Which technical domain(s) does this topic belong to?
|
||||
(e.g. systems programming, distributed systems, security, AI, quantum computing)
|
||||
- What *type* of thing is it?
|
||||
(e.g. protocol, framework, architectural style, API standard, SDK, language, library)
|
||||
- At which abstraction level does it primarily operate?
|
||||
|
||||
### 3. Historical Origin and Motivation
|
||||
- Who introduced it and when?
|
||||
- What concrete problem(s) was it created to solve?
|
||||
- What existing approaches did it replace, extend, or formalize?
|
||||
|
||||
(Only include history that explains intent or constraints.)
|
||||
|
||||
### 4. Core Concepts and Invariants
|
||||
- List the essential concepts without which the topic would not make sense
|
||||
- For each concept, explain its role in one or two sentences
|
||||
- Identify any invariants, guarantees, or formal assumptions
|
||||
|
||||
### 5. Scope Boundaries
|
||||
- Clearly state what the topic explicitly covers
|
||||
- Clearly state what it explicitly does NOT cover
|
||||
- Identify common misconceptions or misuses
|
||||
|
||||
This section should prevent overextension by AI systems.
|
||||
|
||||
### 6. Practical Implications (Non-Tutorial)
|
||||
- What design or architectural consequences follow from using this?
|
||||
- What tradeoffs are inherent?
|
||||
- What kinds of systems typically depend on it?
|
||||
|
||||
Do NOT include step-by-step usage.
|
||||
|
||||
### 7. Relationship to Adjacent Concepts
|
||||
- List closely related standards, technologies, or terms
|
||||
- For each, explain the relationship (complementary, layered on top, alternative, predecessor)
|
||||
|
||||
### 8. Authoritative Sources
|
||||
- List primary, authoritative references:
|
||||
- Standards (RFCs, ISO, W3C, IEEE, etc.)
|
||||
- Official specifications or documentation
|
||||
- Foundational papers
|
||||
- Include direct links
|
||||
- Clearly distinguish primary sources from secondary explanations
|
||||
|
||||
### 9. Stability and Maturity Assessment
|
||||
- Is this topic considered stable, evolving, or experimental?
|
||||
- Are there competing standards or dominant implementations?
|
||||
- Is backward compatibility a concern?
|
||||
|
||||
### 10. Notes for Primer Authoring
|
||||
- Highlight points that MUST be stated clearly in a primer
|
||||
- Highlight areas where ambiguity must be avoided
|
||||
- Identify terminology that must be used consistently
|
||||
|
||||
---
|
||||
|
||||
## Research Constraints
|
||||
|
||||
- Use precise, declarative language
|
||||
- No metaphors or analogies
|
||||
- No marketing or opinionated statements
|
||||
- Assume a technically literate audience
|
||||
- Prefer explicit statements over implied assumptions
|
||||
|
||||
---
|
||||
|
||||
## Why This Protocol Works
|
||||
|
||||
This protocol is intentionally shaped to:
|
||||
* **Force scope clarity** (critical for AI agents)
|
||||
* **Surface invariants and constraints**
|
||||
* **Separate definition from implementation**
|
||||
* **Anchor everything in primary sources**
|
||||
* **Produce output that maps 1:1 to Primer Authoring Rules**
|
||||
|
||||
Think of it as: *A pre-primer that de-risks the primer.*
|
||||
31
examples/content-generator/artifacts/topics/etl.md
Normal file
31
examples/content-generator/artifacts/topics/etl.md
Normal file
@@ -0,0 +1,31 @@
|
||||
---
|
||||
id: topic-etl
|
||||
name: ETL
|
||||
artifact_type: content
|
||||
description: Topic definition for ETL (Extract, Transform, Load)
|
||||
version: 1.0.0
|
||||
tags:
|
||||
- data-engineering
|
||||
- data-integration
|
||||
- topic
|
||||
---
|
||||
|
||||
# ETL (Extract, Transform, Load)
|
||||
|
||||
A three-phase computing process where data is extracted from source systems, transformed (including validation, cleaning, enrichment, and aggregation), and loaded into a target data store or data warehouse.
|
||||
|
||||
ETL is a fundamental pattern in data integration and analytics pipelines, enabling organizations to consolidate data from heterogeneous sources into a unified format suitable for analysis and reporting.
|
||||
|
||||
**Key Characteristics:**
|
||||
- Sequential batch-oriented processing
|
||||
- Data quality enforcement during transformation
|
||||
- Schema mapping and normalization
|
||||
- Support for diverse source and target systems
|
||||
- Typically scheduled and automated
|
||||
|
||||
**Common Use Cases:**
|
||||
- Data warehouse population
|
||||
- Business intelligence reporting
|
||||
- Data migration projects
|
||||
- Master data management
|
||||
- Regulatory compliance reporting
|
||||
31
examples/content-generator/artifacts/topics/microservices.md
Normal file
31
examples/content-generator/artifacts/topics/microservices.md
Normal file
@@ -0,0 +1,31 @@
|
||||
---
|
||||
id: topic-microservices
|
||||
name: Microservices
|
||||
artifact_type: content
|
||||
description: Topic definition for Microservices architecture
|
||||
version: 1.0.0
|
||||
tags:
|
||||
- architecture
|
||||
- distributed-systems
|
||||
- topic
|
||||
---
|
||||
|
||||
# Microservices Architecture
|
||||
|
||||
An architectural style that structures an application as a collection of loosely coupled, independently deployable services, each implementing a specific business capability.
|
||||
|
||||
Microservices represent a departure from monolithic architecture, emphasizing service autonomy, bounded contexts, and decentralized data management.
|
||||
|
||||
**Key Characteristics:**
|
||||
- Independent deployment and scaling
|
||||
- Service-oriented API contracts (typically REST or gRPC)
|
||||
- Decentralized data management (database-per-service)
|
||||
- Polyglot persistence and technology diversity
|
||||
- Failure isolation and resilience patterns
|
||||
|
||||
**Common Use Cases:**
|
||||
- Large-scale web applications
|
||||
- Cloud-native applications
|
||||
- Continuous delivery environments
|
||||
- Organizations requiring team autonomy
|
||||
- Systems requiring differential scaling
|
||||
400
examples/content-generator/generate_primers.py
Executable file
400
examples/content-generator/generate_primers.py
Executable file
@@ -0,0 +1,400 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Primer Generation using Prompt Dependency Resolution
|
||||
|
||||
This script demonstrates the full workflow of generating InfoTech primers
|
||||
using MarkiTect's Prompt Dependency Resolution infrastructure.
|
||||
|
||||
Features demonstrated:
|
||||
- Artifact creation and storage
|
||||
- PromptTemplate with macro resolution
|
||||
- Dependency tracking
|
||||
- Quality validation
|
||||
- Incremental recomputation
|
||||
- Full provenance tracing
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
# Add project root to path
|
||||
project_root = Path(__file__).parent.parent.parent
|
||||
sys.path.insert(0, str(project_root))
|
||||
|
||||
from markitect.prompts.models import Artifact, ArtifactType
|
||||
from markitect.prompts.repositories.sqlite import SQLiteArtifactRepository
|
||||
from markitect.prompts.dependencies.repository import SQLiteDependencyRepository
|
||||
from markitect.prompts.services.artifact_service import ArtifactService
|
||||
from markitect.prompts.templates.models import PromptTemplate, ContentMacro, MacroKind
|
||||
from markitect.prompts.templates.analyzer import TemplateAnalyzer
|
||||
from markitect.prompts.resolver.resolver import PromptResolver
|
||||
from markitect.prompts.resolver.compiler import ContextCompiler
|
||||
from markitect.prompts.resolver.strategy import ResolutionConfig, MultiSpaceResolutionStrategy
|
||||
from markitect.prompts.execution.models import PromptRun, RunConfig
|
||||
from markitect.prompts.execution.manifest import RunManifest
|
||||
from markitect.prompts.dependencies.graph import GraphBuilder
|
||||
from markitect.prompts.traceability.service import TraceabilityService
|
||||
from markitect.prompts.queries.operations import PromptQueryService
|
||||
from markitect.prompts.visualization.graph import GraphExporter
|
||||
|
||||
|
||||
class PrimerGenerator:
|
||||
"""Generates InfoTech primers using prompt dependency resolution."""
|
||||
|
||||
def __init__(self, db_path: str = "primers.db"):
|
||||
"""Initialize with database path."""
|
||||
self.db_path = db_path
|
||||
self.artifact_repo = SQLiteArtifactRepository(db_path)
|
||||
self.artifact_service = ArtifactService(self.artifact_repo)
|
||||
self.dep_repo = SQLiteDependencyRepository(db_path)
|
||||
self.graph_builder = GraphBuilder(self.dep_repo)
|
||||
self.trace_service = TraceabilityService(
|
||||
self.artifact_repo, self.dep_repo, db_path=db_path
|
||||
)
|
||||
self.query_service = PromptQueryService(
|
||||
self.artifact_repo, self.dep_repo, db_path=db_path
|
||||
)
|
||||
|
||||
# Create information spaces
|
||||
self.spaces = {
|
||||
"templates": "primer-templates",
|
||||
"topics": "primer-topics",
|
||||
"guidelines": "primer-guidelines",
|
||||
"output": "generated-primers",
|
||||
}
|
||||
|
||||
def load_or_create_artifact(
|
||||
self, space: str, filepath: Path, artifact_type: ArtifactType, name: Optional[str] = None
|
||||
) -> tuple[Artifact, str]:
|
||||
"""Load artifact from file or fetch from repository. Returns (artifact, content)."""
|
||||
if name is None:
|
||||
name = filepath.stem
|
||||
|
||||
# Load content from file
|
||||
if not filepath.exists():
|
||||
raise FileNotFoundError(f"Artifact file not found: {filepath}")
|
||||
|
||||
content = filepath.read_text()
|
||||
|
||||
# Check if artifact exists
|
||||
existing = self.artifact_repo.get_by_name(space, name)
|
||||
if existing:
|
||||
print(f"✓ Found existing artifact: {name}")
|
||||
return existing, content
|
||||
|
||||
# Create and store artifact
|
||||
artifact = Artifact.create(
|
||||
space_id=space,
|
||||
name=name,
|
||||
content=content,
|
||||
artifact_type=artifact_type,
|
||||
)
|
||||
artifact = self.artifact_repo.create(artifact)
|
||||
print(f"✓ Created artifact: {name} (digest: {artifact.content_digest[:8]})")
|
||||
return artifact, content
|
||||
|
||||
def setup_artifacts(self, example_dir: Path):
|
||||
"""Load all artifacts into the repository."""
|
||||
print("\n=== Loading Artifacts ===")
|
||||
|
||||
# Cache for artifact content (repository doesn't store content)
|
||||
self.artifact_content = {}
|
||||
|
||||
# Load template
|
||||
template_file = example_dir / "templates" / "generate-primer.md"
|
||||
self.template_artifact, content = self.load_or_create_artifact(
|
||||
self.spaces["templates"],
|
||||
template_file,
|
||||
ArtifactType.TEMPLATE,
|
||||
)
|
||||
self.artifact_content[self.template_artifact.id] = content
|
||||
|
||||
# Load topic artifacts
|
||||
topics_dir = example_dir / "artifacts" / "topics"
|
||||
self.topic_artifacts = {}
|
||||
for topic_file in topics_dir.glob("*.md"):
|
||||
artifact, content = self.load_or_create_artifact(
|
||||
self.spaces["topics"],
|
||||
topic_file,
|
||||
ArtifactType.CONTENT,
|
||||
)
|
||||
self.topic_artifacts[artifact.name] = artifact
|
||||
self.artifact_content[artifact.id] = content
|
||||
|
||||
# Load guideline artifacts (rename to match macro expectations)
|
||||
guidelines_dir = example_dir / "artifacts" / "guidelines"
|
||||
self.guideline_artifacts = {}
|
||||
guideline_name_map = {
|
||||
"authoring-rules.md": "authoring_rules",
|
||||
"research-prompt.md": "research_prompt",
|
||||
}
|
||||
for guide_file in guidelines_dir.glob("*.md"):
|
||||
# Use mapped name if available, otherwise use stem
|
||||
artifact_name = guideline_name_map.get(guide_file.name, guide_file.stem)
|
||||
artifact, content = self.load_or_create_artifact(
|
||||
self.spaces["guidelines"],
|
||||
guide_file,
|
||||
ArtifactType.CONTENT,
|
||||
name=artifact_name,
|
||||
)
|
||||
self.guideline_artifacts[artifact.name] = artifact
|
||||
self.artifact_content[artifact.id] = content
|
||||
|
||||
def create_template(self) -> PromptTemplate:
|
||||
"""Create PromptTemplate from template artifact."""
|
||||
print("\n=== Parsing Template ===")
|
||||
|
||||
# Create PromptTemplate from artifact
|
||||
template = PromptTemplate.from_artifact(self.template_artifact)
|
||||
|
||||
# For demo: manually add macros since analyzer would extract from {{...}}
|
||||
# In real usage, analyzer would parse the content automatically
|
||||
template.macros = [
|
||||
ContentMacro(kind=MacroKind.REQUIRED, target="topic"),
|
||||
ContentMacro(kind=MacroKind.REQUIRED, target="authoring_rules"),
|
||||
ContentMacro(kind=MacroKind.REQUIRED, target="research_prompt"),
|
||||
]
|
||||
template.analyzed = True
|
||||
|
||||
print(f"✓ Template created with {len(template.macros)} macro dependencies")
|
||||
return template
|
||||
|
||||
def generate_primer(self, topic_name: str, output_name: Optional[str] = None) -> Artifact:
|
||||
"""Generate a primer for the given topic."""
|
||||
print(f"\n=== Generating Primer: {topic_name} ===")
|
||||
|
||||
# Get topic artifact
|
||||
topic_artifact = self.topic_artifacts.get(topic_name)
|
||||
if not topic_artifact:
|
||||
raise ValueError(f"Topic not found: {topic_name}")
|
||||
|
||||
# Get the topic's content from cache
|
||||
topic_content = self.artifact_content.get(topic_artifact.id)
|
||||
if not topic_content:
|
||||
raise ValueError(f"Topic content not found for: {topic_name}")
|
||||
|
||||
# Create a temporary "topic" artifact with this specific topic's content
|
||||
# This allows the template's @{topic} macro to resolve
|
||||
temp_topic = Artifact.create(
|
||||
space_id=self.spaces["topics"],
|
||||
name="topic",
|
||||
content=topic_content,
|
||||
artifact_type=ArtifactType.CONTENT,
|
||||
)
|
||||
|
||||
# Check if "topic" artifact already exists and delete it
|
||||
existing_topic = self.artifact_repo.get_by_name(self.spaces["topics"], "topic")
|
||||
if existing_topic:
|
||||
self.artifact_repo.delete(existing_topic.id)
|
||||
|
||||
temp_topic = self.artifact_repo.create(temp_topic)
|
||||
self.artifact_content[temp_topic.id] = topic_content
|
||||
print(f"✓ Bound topic macro to {topic_name}")
|
||||
|
||||
# Create template
|
||||
template = self.create_template()
|
||||
|
||||
# Configure resolution
|
||||
resolution_config = ResolutionConfig(
|
||||
space_id=self.spaces["templates"],
|
||||
included_spaces=[
|
||||
self.spaces["topics"],
|
||||
self.spaces["guidelines"],
|
||||
],
|
||||
)
|
||||
|
||||
print("\n=== Resolving Dependencies ===")
|
||||
strategy = MultiSpaceResolutionStrategy()
|
||||
resolver = PromptResolver(self.artifact_service, strategy)
|
||||
resolution_result = resolver.resolve_template(template, resolution_config)
|
||||
|
||||
if not resolution_result.success:
|
||||
print(f"✗ Resolution failed: {resolution_result.context.errors}")
|
||||
return None
|
||||
|
||||
print(f"✓ Resolved {len(resolution_result.context.resolved_macros)} macros")
|
||||
for resolved in resolution_result.context.resolved_macros:
|
||||
print(f" - {resolved.macro.target} → {resolved.artifact.name}")
|
||||
|
||||
# Compile the prompt
|
||||
print("\n=== Compiling Prompt ===")
|
||||
compiler = ContextCompiler()
|
||||
|
||||
# For demo, use mock template content with macro markers
|
||||
template_content = f"""
|
||||
# Generate InfoTech Primer
|
||||
|
||||
Topic: @{{topic}}
|
||||
Guidelines: @{{authoring_rules}}
|
||||
Research Protocol: @{{research_prompt}}
|
||||
|
||||
Generate a complete primer following the authoring rules.
|
||||
"""
|
||||
|
||||
compiled = compiler.compile(template, template_content, resolution_result)
|
||||
print(f"✓ Compiled prompt (digest: {compiled.content_digest[:8]})")
|
||||
|
||||
# Create run manifest
|
||||
run_id = f"run-{topic_name}-001"
|
||||
manifest = RunManifest.create(
|
||||
run_id=run_id,
|
||||
template_id=template.artifact.id,
|
||||
template_name=template.artifact.name,
|
||||
template_digest=self.template_artifact.content_digest,
|
||||
)
|
||||
|
||||
# Add resolved inputs
|
||||
for resolved in resolution_result.context.resolved_macros:
|
||||
manifest.add_resolved_input(
|
||||
name=resolved.macro.target,
|
||||
artifact_id=resolved.artifact.id,
|
||||
space_id=resolved.space_id,
|
||||
digest=resolved.artifact.content_digest,
|
||||
)
|
||||
|
||||
# Create dependency edge
|
||||
manifest.add_dependency_edge(
|
||||
source_id=resolved.artifact.id,
|
||||
target_id=run_id,
|
||||
edge_type="requires",
|
||||
)
|
||||
|
||||
# For demo: create output artifact (would normally come from LLM)
|
||||
output_name = output_name or f"{topic_name}-primer"
|
||||
output_content = f"""# {topic_name.upper()} Primer
|
||||
|
||||
[Generated primer content would go here...]
|
||||
|
||||
This primer was generated using the Prompt Dependency Resolution infrastructure.
|
||||
|
||||
**Dependencies:**
|
||||
- Template: {template.artifact.name}
|
||||
- Topic: {topic_artifact.name}
|
||||
- Guidelines: authoring-rules, research-prompt
|
||||
|
||||
**Run ID:** {run_id}
|
||||
"""
|
||||
|
||||
output_artifact = Artifact.create(
|
||||
space_id=self.spaces["output"],
|
||||
name=output_name,
|
||||
content=output_content,
|
||||
artifact_type=ArtifactType.GENERATED,
|
||||
)
|
||||
output_artifact = self.artifact_repo.create(output_artifact)
|
||||
|
||||
# Add output to manifest
|
||||
manifest.add_output_artifact(
|
||||
artifact_id=output_artifact.id,
|
||||
name=output_artifact.name,
|
||||
digest=output_artifact.content_digest,
|
||||
artifact_type=output_artifact.artifact_type.value,
|
||||
)
|
||||
|
||||
manifest.add_dependency_edge(
|
||||
source_id=run_id,
|
||||
target_id=output_artifact.id,
|
||||
edge_type="generates",
|
||||
)
|
||||
|
||||
# Persist all dependency edges at once
|
||||
edges = self.graph_builder.persist_edges(manifest)
|
||||
print(f"✓ Persisted {len(edges)} dependency edges")
|
||||
|
||||
print(f"✓ Generated primer: {output_artifact.name}")
|
||||
print(f" ID: {output_artifact.id}")
|
||||
print(f" Digest: {output_artifact.content_digest[:8]}")
|
||||
|
||||
return output_artifact
|
||||
|
||||
def show_provenance(self, artifact_id: str):
|
||||
"""Display full provenance trace for an artifact."""
|
||||
print(f"\n=== Provenance Trace ===")
|
||||
|
||||
trace = self.trace_service.trace_artifact(artifact_id)
|
||||
|
||||
print(f"Artifact: {trace.artifact_id}")
|
||||
|
||||
if trace.producing_run:
|
||||
print(f"\nProducing Run: {trace.producing_run.run_id}")
|
||||
print(f" Template: {trace.producing_run.template_id}")
|
||||
print(f" Status: {trace.producing_run.status}")
|
||||
|
||||
if trace.input_artifacts:
|
||||
print(f"\nInput Artifacts ({len(trace.input_artifacts)}):")
|
||||
for inp in trace.input_artifacts:
|
||||
print(f" - {inp.name} ({inp.artifact_type})")
|
||||
|
||||
if trace.dependency_chain:
|
||||
print(f"\nDependency Chain ({len(trace.dependency_chain)} artifacts):")
|
||||
for dep_id in trace.dependency_chain[:5]: # Show first 5
|
||||
print(f" - {dep_id[:8]}...")
|
||||
|
||||
def visualize_dependencies(self, artifact_id: str, output_file: str = "dependencies.mermaid"):
|
||||
"""Generate dependency graph visualization."""
|
||||
print(f"\n=== Dependency Visualization ===")
|
||||
|
||||
from markitect.prompts.dependencies.queries import DependencyQueryService
|
||||
|
||||
query_svc = DependencyQueryService(self.dep_repo)
|
||||
deps = query_svc.find_transitive_dependencies(artifact_id)
|
||||
dependents = query_svc.find_transitive_dependents(artifact_id)
|
||||
all_ids = deps | dependents | {artifact_id}
|
||||
|
||||
graph = self.graph_builder.build_graph(all_ids)
|
||||
mermaid = GraphExporter.to_mermaid(graph, f"Primer Generation Dependencies")
|
||||
|
||||
Path(output_file).write_text(mermaid)
|
||||
print(f"✓ Saved dependency graph to {output_file}")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main execution flow."""
|
||||
print("╔══════════════════════════════════════════════════════════════╗")
|
||||
print("║ MarkiTect Prompt Dependency Resolution Example ║")
|
||||
print("║ InfoTech Primer Generation ║")
|
||||
print("╚══════════════════════════════════════════════════════════════╝")
|
||||
|
||||
example_dir = Path(__file__).parent
|
||||
|
||||
# Initialize generator
|
||||
generator = PrimerGenerator(db_path=str(example_dir / "primers.db"))
|
||||
|
||||
# Setup artifacts
|
||||
generator.setup_artifacts(example_dir)
|
||||
|
||||
# Generate primers for multiple topics
|
||||
topics = ["etl", "microservices"]
|
||||
|
||||
generated_artifacts = []
|
||||
for topic in topics:
|
||||
try:
|
||||
artifact = generator.generate_primer(topic)
|
||||
if artifact:
|
||||
generated_artifacts.append(artifact)
|
||||
except Exception as e:
|
||||
print(f"✗ Failed to generate primer for {topic}: {e}")
|
||||
|
||||
# Show provenance for first generated primer
|
||||
if generated_artifacts:
|
||||
artifact = generated_artifacts[0]
|
||||
generator.show_provenance(artifact.id)
|
||||
generator.visualize_dependencies(artifact.id)
|
||||
|
||||
# Show statistics
|
||||
print("\n=== Dependency Statistics ===")
|
||||
stats = generator.query_service.get_dependency_stats()
|
||||
print(f"Total nodes: {stats['total_nodes']}")
|
||||
print(f"Total edges: {stats['total_edges']}")
|
||||
print(f"Root artifacts: {stats['root_count']}")
|
||||
print(f"Leaf artifacts: {stats['leaf_count']}")
|
||||
print(f"Has cycles: {stats['has_cycles']}")
|
||||
|
||||
print("\n✓ Primer generation complete!")
|
||||
print(f" Database: {generator.db_path}")
|
||||
print(f" Generated: {len(generated_artifacts)} primers")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
154
examples/content-generator/prepdr/AssistentPrompt.md
Normal file
154
examples/content-generator/prepdr/AssistentPrompt.md
Normal file
@@ -0,0 +1,154 @@
|
||||
ResearchPrompt
|
||||
|
||||
*Research a topic...*
|
||||
|
||||
# InfoTechPrimer – ResearchPrompt
|
||||
|
||||
Below is a **single, reusable, high-precision research prompt** you can use to *systematically get a grip on any InfoTech topic* before writing an **InfoTechPrimer**.
|
||||
|
||||
> **Purpose:**
|
||||
> Produce a *factually grounded, scope-aware, source-anchored research brief* suitable as the direct input for authoring an InfoTechPrimer.
|
||||
|
||||
It is designed to work well with:
|
||||
|
||||
* General-purpose LLMs
|
||||
* Research-oriented agents
|
||||
* Human-in-the-loop review
|
||||
|
||||
The prompt is **topic-agnostic**, but forces rigor, boundaries, and source grounding.
|
||||
|
||||
---
|
||||
|
||||
## ResearchPrompt
|
||||
|
||||
```
|
||||
You are conducting foundational research for an InfoTechPrimer.
|
||||
|
||||
Topic: <$topic>
|
||||
|
||||
The goal is NOT to teach or promote, but to establish a precise, shared understanding
|
||||
of the topic for experienced IT professionals and AI systems.
|
||||
|
||||
Produce a structured research brief that answers the following sections.
|
||||
Be concise, factual, and source-driven.
|
||||
Avoid tutorials, opinions, and vendor marketing language.
|
||||
|
||||
---
|
||||
|
||||
1. Canonical Definition
|
||||
- Provide the most widely accepted definition(s) of the topic.
|
||||
- If multiple definitions exist, explain why and in which contexts they differ.
|
||||
- Prefer definitions from standards bodies, original designers, or official specifications.
|
||||
|
||||
---
|
||||
|
||||
2. Domain Context and Classification
|
||||
- Which technical domain(s) does this topic belong to?
|
||||
(e.g. systems programming, distributed systems, security, AI, quantum computing)
|
||||
- What *type* of thing is it?
|
||||
(e.g. protocol, framework, architectural style, API standard, SDK, language, library)
|
||||
- At which abstraction level does it primarily operate?
|
||||
|
||||
---
|
||||
|
||||
3. Historical Origin and Motivation
|
||||
- Who introduced it and when?
|
||||
- What concrete problem(s) was it created to solve?
|
||||
- What existing approaches did it replace, extend, or formalize?
|
||||
|
||||
(Only include history that explains intent or constraints.)
|
||||
|
||||
---
|
||||
|
||||
4. Core Concepts and Invariants
|
||||
- List the essential concepts without which the topic would not make sense.
|
||||
- For each concept, explain its role in one or two sentences.
|
||||
- Identify any invariants, guarantees, or formal assumptions.
|
||||
|
||||
---
|
||||
|
||||
5. Scope Boundaries
|
||||
- Clearly state what the topic explicitly covers.
|
||||
- Clearly state what it explicitly does NOT cover.
|
||||
- Identify common misconceptions or misuses.
|
||||
|
||||
This section should prevent overextension by AI systems.
|
||||
|
||||
---
|
||||
|
||||
6. Practical Implications (Non-Tutorial)
|
||||
- What design or architectural consequences follow from using this?
|
||||
- What tradeoffs are inherent?
|
||||
- What kinds of systems typically depend on it?
|
||||
|
||||
Do NOT include step-by-step usage.
|
||||
|
||||
---
|
||||
|
||||
7. Relationship to Adjacent Concepts
|
||||
- List closely related standards, technologies, or terms.
|
||||
- For each, explain the relationship (complementary, layered on top, alternative, predecessor).
|
||||
|
||||
---
|
||||
|
||||
8. Authoritative Sources
|
||||
- List primary, authoritative references:
|
||||
- Standards (RFCs, ISO, W3C, IEEE, etc.)
|
||||
- Official specifications or documentation
|
||||
- Foundational papers
|
||||
- Include direct links.
|
||||
- Clearly distinguish primary sources from secondary explanations.
|
||||
|
||||
---
|
||||
|
||||
9. Stability and Maturity Assessment
|
||||
- Is this topic considered stable, evolving, or experimental?
|
||||
- Are there competing standards or dominant implementations?
|
||||
- Is backward compatibility a concern?
|
||||
|
||||
---
|
||||
|
||||
10. Notes for Primer Authoring
|
||||
- Highlight points that MUST be stated clearly in a primer.
|
||||
- Highlight areas where ambiguity must be avoided.
|
||||
- Identify terminology that must be used consistently.
|
||||
|
||||
---
|
||||
|
||||
Constraints:
|
||||
- Use precise, declarative language.
|
||||
- No metaphors or analogies.
|
||||
- No marketing or opinionated statements.
|
||||
- Assume a technically literate audience.
|
||||
- Prefer explicit statements over implied assumptions.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Why This Prompt Works Well
|
||||
|
||||
This prompt is intentionally shaped to:
|
||||
|
||||
* **Force scope clarity** (critical for AI agents)
|
||||
* **Surface invariants and constraints**
|
||||
* **Separate definition from implementation**
|
||||
* **Anchor everything in primary sources**
|
||||
* **Produce output that maps 1:1 to your Primer Authoring Rules**
|
||||
|
||||
You can think of it as:
|
||||
|
||||
> *A pre-primer that de-risks the primer.*
|
||||
|
||||
---
|
||||
|
||||
## How You’ll Likely Use It in Practice
|
||||
|
||||
Typical flow:
|
||||
|
||||
1. Run this prompt on a topic (e.g. CUDA, OAuth 2.0, OpenQASM)
|
||||
2. Review and correct factual issues
|
||||
3. Collapse the research brief into the Primer schema
|
||||
4. Apply brand tone + structural rules
|
||||
5. Publish as an InfoTechPrimer
|
||||
|
||||
xxx
|
||||
302
examples/content-generator/prepdr/AuthoringRules.md
Normal file
302
examples/content-generator/prepdr/AuthoringRules.md
Normal file
@@ -0,0 +1,302 @@
|
||||
AuthoringRules
|
||||
|
||||
*How to write effective primers*
|
||||
|
||||
## Primer Authoring Rules
|
||||
|
||||
**Status:** Draft
|
||||
**Intended Audience:** Human authors and AI systems generating or validating primers
|
||||
**Purpose:** Ensure primers are precise, stable, and suitable as shared context for humans and AI agents
|
||||
|
||||
---
|
||||
|
||||
## 1. What a Primer Is (Normative)
|
||||
|
||||
An **InfoTechPrimer** is a **short, structured reference document** that establishes a **shared understanding** of a specific IT term, standard, method, or concept.
|
||||
|
||||
A primer:
|
||||
|
||||
* Defines **what the topic is**
|
||||
* Explains **where it fits**
|
||||
* Clarifies **scope boundaries**
|
||||
* Points to **authoritative sources**
|
||||
|
||||
A primer does **not**:
|
||||
|
||||
* Teach step-by-step usage
|
||||
* Advocate tools or vendors
|
||||
* Explore implementation details beyond what is normatively defined
|
||||
|
||||
---
|
||||
|
||||
## 2. Target Audience
|
||||
|
||||
Primers are written for:
|
||||
|
||||
* Humans with solid general IT knowledge
|
||||
* Readers who are *not specialists* in the specific topic
|
||||
* AI systems that consume structured context for reasoning and coding
|
||||
|
||||
Authors must assume:
|
||||
|
||||
* Conceptual literacy
|
||||
* Familiarity with basic IT terminology
|
||||
* No prior deep knowledge of the topic
|
||||
|
||||
---
|
||||
|
||||
## 3. Required Structure (Mandatory)
|
||||
|
||||
Every primer **MUST** contain the following sections **in this order**:
|
||||
|
||||
1. **Definition**
|
||||
2. **Context**
|
||||
3. **Core Concepts**
|
||||
4. **Scope and Non-Scope**
|
||||
5. **Practical Implications**
|
||||
6. **Formal Standards and Authoritative Sources**
|
||||
7. **Related Concepts**
|
||||
|
||||
No section may be omitted.
|
||||
Empty sections are not allowed.
|
||||
|
||||
---
|
||||
|
||||
## 4. Section Authoring Rules
|
||||
|
||||
### 4.1 Definition
|
||||
|
||||
**Purpose:** Establish an unambiguous baseline meaning.
|
||||
|
||||
Rules:
|
||||
|
||||
* 2–4 sentences maximum
|
||||
* Declarative, precise language
|
||||
* No metaphors, examples, or analogies
|
||||
* No historical narrative
|
||||
|
||||
Good:
|
||||
|
||||
> “OAuth 2.0 is an authorization framework that enables a third-party application to obtain limited access to an HTTP service on behalf of a resource owner.”
|
||||
|
||||
Bad:
|
||||
|
||||
> “OAuth is basically a way to let apps log in without sharing passwords.”
|
||||
|
||||
---
|
||||
|
||||
### 4.2 Context
|
||||
|
||||
**Purpose:** Position the concept within the IT landscape.
|
||||
|
||||
Rules:
|
||||
|
||||
* Describe the domain(s) the concept belongs to
|
||||
* Explain *why it exists*, not *how to use it*
|
||||
* Historical notes allowed only if they clarify intent or constraints
|
||||
|
||||
Include:
|
||||
|
||||
* Typical environments
|
||||
* Architectural level (protocol, pattern, framework, etc.)
|
||||
|
||||
---
|
||||
|
||||
### 4.3 Core Concepts
|
||||
|
||||
**Purpose:** Identify the irreducible ideas that define the topic.
|
||||
|
||||
Rules:
|
||||
|
||||
* Bullet points only
|
||||
* Each bullet describes one concept
|
||||
* No nested lists
|
||||
* Avoid redundancy with Definition
|
||||
|
||||
Think: *“What must be true for this concept to exist?”*
|
||||
|
||||
---
|
||||
|
||||
### 4.4 Scope and Non-Scope
|
||||
|
||||
**Purpose:** Prevent conceptual drift and misuse.
|
||||
|
||||
Rules:
|
||||
|
||||
* Explicitly list inclusions and exclusions
|
||||
* Use parallel structure
|
||||
* Address common misconceptions
|
||||
|
||||
Format:
|
||||
|
||||
```markdown
|
||||
**In Scope**
|
||||
- ...
|
||||
|
||||
**Out of Scope**
|
||||
- ...
|
||||
```
|
||||
|
||||
This section is **critical** for AI agent correctness.
|
||||
|
||||
---
|
||||
|
||||
### 4.5 Practical Implications
|
||||
|
||||
**Purpose:** Describe consequences of adopting or interacting with the concept.
|
||||
|
||||
Rules:
|
||||
|
||||
* Focus on effects, not instructions
|
||||
* No step-by-step guidance
|
||||
* Include tradeoffs where relevant
|
||||
|
||||
Examples of acceptable content:
|
||||
|
||||
* Design constraints
|
||||
* Operational complexity
|
||||
* Security or scalability implications
|
||||
|
||||
---
|
||||
|
||||
### 4.6 Formal Standards and Authoritative Sources
|
||||
|
||||
**Purpose:** Anchor the primer in canonical truth.
|
||||
|
||||
Rules:
|
||||
|
||||
* Prefer primary sources
|
||||
* Include direct links
|
||||
* Avoid blogs unless widely recognized and necessary
|
||||
|
||||
Acceptable sources:
|
||||
|
||||
* RFCs
|
||||
* W3C Recommendations
|
||||
* ISO / IEC standards
|
||||
* NIST publications
|
||||
* Official specifications
|
||||
* Foundational academic papers
|
||||
|
||||
At least **one** authoritative source is required.
|
||||
|
||||
---
|
||||
|
||||
### 4.7 Related Concepts
|
||||
|
||||
**Purpose:** Enable semantic navigation.
|
||||
|
||||
Rules:
|
||||
|
||||
* Short descriptions only (one line per concept)
|
||||
* No deep explanations
|
||||
* Avoid circular definitions
|
||||
|
||||
Example:
|
||||
|
||||
> **OpenID Connect** – An identity layer built on top of OAuth 2.0.
|
||||
|
||||
---
|
||||
|
||||
## 5. Language and Style Rules
|
||||
|
||||
Mandatory:
|
||||
|
||||
* Present tense
|
||||
* Declarative sentences
|
||||
* Neutral, technical tone
|
||||
|
||||
Avoid:
|
||||
|
||||
* First-person language (“we”, “you”)
|
||||
* Rhetorical questions
|
||||
* Marketing language
|
||||
* Informal phrasing
|
||||
* Emojis
|
||||
|
||||
---
|
||||
|
||||
## 6. Length Constraints
|
||||
|
||||
A primer should typically be:
|
||||
|
||||
* **600–1,000 words total**
|
||||
* Short enough to be read in one sitting
|
||||
* Long enough to define boundaries clearly
|
||||
|
||||
Exceeding this range requires justification.
|
||||
|
||||
---
|
||||
|
||||
## 7. Stability and Versioning
|
||||
|
||||
Primers are intended to be **stable reference artifacts**.
|
||||
|
||||
Rules:
|
||||
|
||||
* Do not chase trends
|
||||
* Avoid speculative content
|
||||
* Update only when:
|
||||
|
||||
* Standards change
|
||||
* Definitions evolve materially
|
||||
* Authoritative sources are superseded
|
||||
|
||||
When updating:
|
||||
|
||||
* Preserve conceptual continuity
|
||||
* Avoid rewriting without necessity
|
||||
|
||||
---
|
||||
|
||||
## 8. AI Optimization Rules (Explicit)
|
||||
|
||||
Authors **SHOULD**:
|
||||
|
||||
* Use consistent terminology
|
||||
* Avoid synonyms for core terms once defined
|
||||
* Prefer explicit over implicit assumptions
|
||||
* State constraints clearly
|
||||
|
||||
Authors **MUST NOT**:
|
||||
|
||||
* Rely on context outside the document
|
||||
* Assume tool- or framework-specific defaults
|
||||
* Leave ambiguity where standards are explicit
|
||||
|
||||
---
|
||||
|
||||
## 9. Validation Criteria (Checklist)
|
||||
|
||||
A primer is valid if:
|
||||
|
||||
* [ ] All required sections are present
|
||||
* [ ] Definition is precise and unambiguous
|
||||
* [ ] Scope boundaries are explicit
|
||||
* [ ] At least one authoritative source is linked
|
||||
* [ ] No tutorial or marketing content exists
|
||||
* [ ] Language follows declarative style rules
|
||||
|
||||
---
|
||||
|
||||
## 10. Non-Goals
|
||||
|
||||
InfoTechPrimers are **not**:
|
||||
|
||||
* Documentation replacements
|
||||
* Training material
|
||||
* Opinionated best-practice guides
|
||||
* Tool comparisons
|
||||
|
||||
Those belong elsewhere.
|
||||
|
||||
---
|
||||
|
||||
### Version Note
|
||||
|
||||
This is **Primer Authoring Rules v0.1**.
|
||||
Expect tightening, not loosening, as real primers are written and validated.
|
||||
|
||||
|
||||
|
||||
xxx
|
||||
7
examples/content-generator/prepdr/ETL.md
Normal file
7
examples/content-generator/prepdr/ETL.md
Normal file
@@ -0,0 +1,7 @@
|
||||
ETL
|
||||
|
||||
*Extract Transform Load*
|
||||
|
||||
A three-phase computing process where data is extracted from an input source, transformed (including cleaning), and loaded into an output data container.
|
||||
|
||||
xxx
|
||||
23
examples/content-generator/prepdr/GeneratePrimerTemplate.md
Normal file
23
examples/content-generator/prepdr/GeneratePrimerTemplate.md
Normal file
@@ -0,0 +1,23 @@
|
||||
GeneratePrimerTemplate
|
||||
|
||||
*Program to research infotech topics*
|
||||
|
||||
# Generate Primer
|
||||
|
||||
Assist me to the best of your ability adressing the $topic>, $guidance and $protoprompt I will be providing.
|
||||
If something is unclear show me the arguments and prompt for refinement, before starting.
|
||||
|
||||
<topic>
|
||||
{{ETL}}
|
||||
</topic>
|
||||
|
||||
<guidance>
|
||||
{{AuthoringRules}}
|
||||
</guidance>
|
||||
|
||||
<protoprompt>
|
||||
{{ResearchPrompt}}
|
||||
</protoprompt>
|
||||
|
||||
|
||||
xxx
|
||||
1
examples/content-generator/prepdr/README.md
Normal file
1
examples/content-generator/prepdr/README.md
Normal file
@@ -0,0 +1 @@
|
||||
I used these files to generate infotech primers for topics like ETL in a consistent way before formalizing the concept of prompt-dependence-resolution.
|
||||
59
examples/content-generator/templates/generate-primer.md
Normal file
59
examples/content-generator/templates/generate-primer.md
Normal file
@@ -0,0 +1,59 @@
|
||||
---
|
||||
id: generate-primer-v1
|
||||
name: GeneratePrimerTemplate
|
||||
artifact_type: template
|
||||
description: PromptTemplate for generating InfoTech primers with dependency resolution
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
# Generate InfoTech Primer
|
||||
|
||||
You are conducting foundational research and authoring an InfoTech Primer.
|
||||
|
||||
**Your Task:** Produce a complete, publication-ready InfoTech Primer for the given topic, following the authoring rules exactly.
|
||||
|
||||
## Topic Specification
|
||||
|
||||
@{topic}
|
||||
|
||||
## Authoring Guidelines
|
||||
|
||||
@{authoring_rules}
|
||||
|
||||
## Research Protocol
|
||||
|
||||
When researching this topic, follow this systematic approach:
|
||||
|
||||
@{research_prompt}
|
||||
|
||||
---
|
||||
|
||||
## Output Requirements
|
||||
|
||||
Generate a complete primer document with the following structure:
|
||||
|
||||
1. **Definition** (2-4 sentences, precise and unambiguous)
|
||||
2. **Context** (domain positioning and architectural level)
|
||||
3. **Core Concepts** (bullet points, one concept per line)
|
||||
4. **Scope and Non-Scope** (explicit inclusions and exclusions)
|
||||
5. **Practical Implications** (design consequences, not tutorials)
|
||||
6. **Formal Standards and Authoritative Sources** (with links)
|
||||
7. **Related Concepts** (one-line descriptions)
|
||||
|
||||
**Constraints:**
|
||||
- Follow declarative, neutral tone
|
||||
- No metaphors, marketing language, or tutorials
|
||||
- Target length: 600-1,000 words
|
||||
- Include at least one authoritative source
|
||||
- Use consistent terminology throughout
|
||||
|
||||
**Validation:**
|
||||
Before finalizing, verify:
|
||||
- [ ] All required sections are present
|
||||
- [ ] Definition is precise and unambiguous
|
||||
- [ ] Scope boundaries are explicit
|
||||
- [ ] At least one authoritative source is linked
|
||||
- [ ] No tutorial or marketing content
|
||||
- [ ] Language follows declarative style rules
|
||||
|
||||
Generate the primer now.
|
||||
158
examples/design-patterns/CopyFirstMigration.md
Normal file
158
examples/design-patterns/CopyFirstMigration.md
Normal file
@@ -0,0 +1,158 @@
|
||||
# Design Principle: Copy First Migration
|
||||
|
||||
## Meta
|
||||
- **Name:** Copy First Migration
|
||||
- **ShortName:** CopyFirst
|
||||
- **Version:** 0.1
|
||||
- **Status:** Draft
|
||||
- **Tags:** refactoring, migration, safety, testing, legacy
|
||||
- **RelatedPrinciples:** Don’t Repeat Yourself, Safe Refactoring, Test Pyramid, Capability-Based Testing
|
||||
|
||||
---
|
||||
|
||||
## Intent
|
||||
Enable safe refactoring and structural migration of codebases by preserving
|
||||
existing, working functionality until the new implementation is fully verified.
|
||||
|
||||
This principle prioritizes **reversibility, confidence, and continuity** over
|
||||
speed or elegance.
|
||||
|
||||
---
|
||||
|
||||
## CoreStatement
|
||||
Never move code directly; always copy first and delete only after verified
|
||||
behavioral equivalence is established.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
### InScope
|
||||
- Large-scale refactors or directory restructurings
|
||||
- Technology or language migrations (e.g. JS → new JS layout, JS → Python integration)
|
||||
- Legacy code stabilization
|
||||
- Safety-critical or business-critical systems
|
||||
- Situations with incomplete test coverage
|
||||
|
||||
### OutOfScope
|
||||
- Greenfield development
|
||||
- Trivial refactors with full and trusted test coverage
|
||||
- One-off throwaway scripts
|
||||
- Performance-driven rewrites where duplication is unacceptable
|
||||
|
||||
---
|
||||
|
||||
## InterpretationGuidelines
|
||||
|
||||
### What “Copy First” Means
|
||||
- The original code remains untouched and functional
|
||||
- The new version is treated as **experimental until proven**
|
||||
- Deletion is a **final, explicit act**, not an implicit side effect
|
||||
|
||||
### Common Misinterpretations
|
||||
- “This is inefficient because it duplicates code”
|
||||
→ Duplication is intentional and temporary
|
||||
- “Moving files is faster”
|
||||
→ Speed is not the optimization target here
|
||||
- “Tests alone are enough”
|
||||
→ Tests are necessary but not sufficient without behavioral comparison
|
||||
|
||||
---
|
||||
|
||||
## DetectionHeuristics
|
||||
|
||||
### Structural Signals
|
||||
- Files or modules being relocated across directories or packages
|
||||
- Parallel implementations during migration
|
||||
- Introduction of a new architectural boundary
|
||||
|
||||
### Semantic Signals
|
||||
- Code paths that must remain behaviorally identical
|
||||
- Business rules with high regression risk
|
||||
- Legacy logic that is poorly documented but relied upon
|
||||
|
||||
### Change-Cost Signals
|
||||
- Rollbacks are expensive or disruptive
|
||||
- Failures would impact production or customers
|
||||
- Migration spans multiple commits or teams
|
||||
|
||||
---
|
||||
|
||||
## DiagnosticQuestions
|
||||
1. What breaks if this migration is wrong?
|
||||
2. Do we have a known-good reference implementation?
|
||||
3. Can both old and new code paths run in parallel?
|
||||
4. How quickly can we revert if a defect is found?
|
||||
5. What is the minimal proof of behavioral equivalence?
|
||||
|
||||
---
|
||||
|
||||
## RecommendedActions
|
||||
|
||||
### Low-Risk Actions
|
||||
- Copy files to the new location instead of moving
|
||||
- Preserve original imports and entry points
|
||||
- Add logging or tracing for comparison
|
||||
|
||||
### Medium-Risk Actions
|
||||
- Introduce dual-track execution (old + new)
|
||||
- Add integration tests targeting both implementations
|
||||
- Compare outputs, side effects, and error behavior
|
||||
|
||||
### High-Risk Actions
|
||||
- Switch production usage to the new implementation
|
||||
- Remove old code only after full verification
|
||||
- Collapse duplicated paths once confidence is established
|
||||
|
||||
---
|
||||
|
||||
## AcceptanceCriteria
|
||||
- Original code remains functional until final removal
|
||||
- New code passes all existing tests
|
||||
- New integration tests validate identical behavior
|
||||
- Dual-track comparisons show no regressions
|
||||
- Deletion of old code is deliberate and reversible up to the final step
|
||||
|
||||
---
|
||||
|
||||
## AntiPatterns
|
||||
- Moving files directly without a fallback
|
||||
- Refactoring and migration in a single irreversible step
|
||||
- Deleting “unused” code before equivalence is proven
|
||||
- Assuming test parity guarantees behavioral parity
|
||||
- Big-bang migrations without rollback paths
|
||||
|
||||
---
|
||||
|
||||
## Tradeoffs
|
||||
Applying Copy First Migration intentionally:
|
||||
- Introduces temporary duplication
|
||||
- Increases short-term codebase size
|
||||
- Slows perceived progress
|
||||
|
||||
These costs are justified by dramatically reduced risk and higher confidence
|
||||
during complex migrations.
|
||||
|
||||
---
|
||||
|
||||
## AgentUsage
|
||||
|
||||
### When to Apply This Lens
|
||||
- During directory, module, or architecture migrations
|
||||
- When refactoring legacy or poorly understood code
|
||||
- When safety and uptime matter more than speed
|
||||
- When rollback must remain possible at all times
|
||||
|
||||
### When to Suspend This Lens
|
||||
- In greenfield projects
|
||||
- When full test coverage and confidence already exist
|
||||
- For trivial mechanical refactors
|
||||
|
||||
### Expected Agent Output
|
||||
- Identification of migration boundaries
|
||||
- Copy-first migration plan with explicit stages
|
||||
- Test strategy (unit, integration, dual-track)
|
||||
- Rollback points and deletion criteria
|
||||
- Clear signal for when old code may be removed
|
||||
|
||||
xxx
|
||||
135
examples/design-patterns/DesignPrincipleSchema.json
Normal file
135
examples/design-patterns/DesignPrincipleSchema.json
Normal file
@@ -0,0 +1,135 @@
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"type": "object",
|
||||
"title": "Schema for DesignPrinciples",
|
||||
"description": "JSON schema describing the markdown structure of OperationalKnowledge DesignPrinciples",
|
||||
"properties": {
|
||||
"headings": {
|
||||
"type": "object",
|
||||
"description": "Document heading structure",
|
||||
"properties": {
|
||||
"level_1": {
|
||||
"type": "array",
|
||||
"description": "Headings at level 1",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"content": {
|
||||
"type": "string"
|
||||
},
|
||||
"level": {
|
||||
"type": "integer"
|
||||
},
|
||||
"position": {
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"content",
|
||||
"level"
|
||||
]
|
||||
},
|
||||
"minItems": 1,
|
||||
"maxItems": 1
|
||||
},
|
||||
"level_2": {
|
||||
"type": "array",
|
||||
"description": "Headings at level 2",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"content": {
|
||||
"type": "string"
|
||||
},
|
||||
"level": {
|
||||
"type": "integer"
|
||||
},
|
||||
"position": {
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"content",
|
||||
"level"
|
||||
]
|
||||
},
|
||||
"minItems": 4,
|
||||
"maxItems": 12
|
||||
},
|
||||
"level_3": {
|
||||
"type": "array",
|
||||
"description": "Headings at level 3",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"content": {
|
||||
"type": "string"
|
||||
},
|
||||
"level": {
|
||||
"type": "integer"
|
||||
},
|
||||
"position": {
|
||||
"type": "integer"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"content",
|
||||
"level"
|
||||
]
|
||||
},
|
||||
"minItems": 0,
|
||||
"maxItems": 40
|
||||
}
|
||||
}
|
||||
},
|
||||
"paragraphs": {
|
||||
"type": "array",
|
||||
"description": "Text paragraphs",
|
||||
"minItems": 8,
|
||||
"maxItems": 120
|
||||
},
|
||||
"lists": {
|
||||
"type": "array",
|
||||
"description": "Lists (ordered and unordered)",
|
||||
"minItems": 0,
|
||||
"maxItems": 20
|
||||
},
|
||||
"emphasis": {
|
||||
"type": "array",
|
||||
"description": "Text emphasis (bold, italic)",
|
||||
"minItems": 0,
|
||||
"maxItems": 120
|
||||
},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"description": "Document structure metadata",
|
||||
"properties": {
|
||||
"total_elements": {
|
||||
"type": "integer",
|
||||
"const": 115
|
||||
},
|
||||
"structure_types": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "All structural element types found",
|
||||
"const": [
|
||||
"paragraph_close",
|
||||
"heading_close",
|
||||
"hr",
|
||||
"bullet_list_open",
|
||||
"paragraph_open",
|
||||
"heading_open",
|
||||
"ordered_list_open",
|
||||
"ordered_list_close",
|
||||
"inline",
|
||||
"list_item_close",
|
||||
"list_item_open",
|
||||
"bullet_list_close"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
160
examples/design-patterns/DontRepeatYourself.md
Normal file
160
examples/design-patterns/DontRepeatYourself.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# Design Principle: Don’t Repeat Yourself (DRY)
|
||||
|
||||
## Meta
|
||||
- **Name:** Don’t Repeat Yourself
|
||||
- **ShortName:** DRY
|
||||
- **Version:** 0.1
|
||||
- **Status:** Stable
|
||||
- **Tags:** maintainability, refactoring, architecture, quality
|
||||
- **RelatedPrinciples:** Single Responsibility, YAGNI, Separation of Concerns
|
||||
|
||||
---
|
||||
|
||||
## Intent
|
||||
Reduce maintenance cost and behavioral drift by ensuring that each piece of
|
||||
knowledge, rule, or decision logic has a single authoritative representation
|
||||
in the codebase.
|
||||
|
||||
---
|
||||
|
||||
## CoreStatement
|
||||
A codebase violates DRY when the same knowledge is expressed in multiple places
|
||||
such that a change would require edits in more than one location or risks
|
||||
inconsistent behavior.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
### InScope
|
||||
- Business rules and decision logic
|
||||
- Algorithms and validation logic
|
||||
- Data schemas, DTOs, and field definitions
|
||||
- Configuration values and feature flags
|
||||
- Repeated workflows or orchestration logic
|
||||
- Test setup and invariant test scenarios
|
||||
|
||||
### OutOfScope
|
||||
- Superficial textual similarity without shared meaning
|
||||
- Intentional duplication for isolation or clarity
|
||||
- Early-stage exploratory code where abstractions are not yet clear
|
||||
- Performance-driven duplication with explicit justification
|
||||
|
||||
---
|
||||
|
||||
## InterpretationGuidelines
|
||||
|
||||
### What “Repeat” Means
|
||||
DRY is about **duplication of knowledge**, not duplication of text.
|
||||
|
||||
Examples of knowledge duplication:
|
||||
- The same validation rule implemented in multiple services
|
||||
- Identical conditional logic controlling the same behavior
|
||||
- The same data structure defined independently in multiple modules
|
||||
|
||||
### Common Misinterpretations
|
||||
- “Any repeated code is bad” (false)
|
||||
- “DRY means maximum abstraction” (false)
|
||||
- “Utility modules automatically improve DRY” (often false)
|
||||
|
||||
---
|
||||
|
||||
## DetectionHeuristics
|
||||
|
||||
### Structural Signals
|
||||
- Functions with highly similar bodies and signatures
|
||||
- Repeated constants, strings, regexes, or SQL fragments
|
||||
- Parallel modules with mirrored internal structure
|
||||
|
||||
### Semantic Signals
|
||||
- Identical error messages or validation rules in different layers
|
||||
- Repeated mapping logic between the same concepts
|
||||
- Copy-paste variations differing only in naming
|
||||
|
||||
### Change-Cost Signals
|
||||
- A requirement change touches multiple files for the same reason
|
||||
- Fixes applied in one location but missing in others
|
||||
- Tests failing inconsistently after partial updates
|
||||
|
||||
---
|
||||
|
||||
## DiagnosticQuestions
|
||||
1. Is this duplication representing the same rule or policy?
|
||||
2. If this rule changes, how many places must be updated?
|
||||
3. Is the duplicated logic stable or likely to evolve?
|
||||
4. Are the differences intentional or accidental?
|
||||
5. Where is the natural “source of truth” for this knowledge?
|
||||
6. Would abstraction reduce or increase cognitive load?
|
||||
|
||||
---
|
||||
|
||||
## RecommendedActions
|
||||
|
||||
### Low-Risk Refactors
|
||||
- Extract constants or configuration values
|
||||
- Centralize literals and error messages
|
||||
- Introduce shared test fixtures or helpers
|
||||
|
||||
### Medium-Risk Refactors
|
||||
- Extract pure helper functions
|
||||
- Introduce shared domain services or modules
|
||||
- Unify schema/type definitions
|
||||
|
||||
### High-Risk Refactors
|
||||
- Introduce strategy/template patterns
|
||||
- Merge parallel subsystems
|
||||
- Redesign domain boundaries to align ownership of rules
|
||||
|
||||
---
|
||||
|
||||
## AcceptanceCriteria
|
||||
- Each rule or behavior has a single authoritative implementation
|
||||
- Required changes affect fewer locations than before
|
||||
- Naming reflects domain meaning, not technical convenience
|
||||
- Tests pass without behavior regression
|
||||
- Coupling does not increase unintentionally
|
||||
|
||||
---
|
||||
|
||||
## AntiPatterns
|
||||
- “God” utility modules with unrelated helpers
|
||||
- Over-generalized abstractions with many parameters
|
||||
- Shared code across domains that should evolve independently
|
||||
- Premature abstraction of coincidental similarities
|
||||
- Hiding meaningful differences behind generic interfaces
|
||||
|
||||
---
|
||||
|
||||
## Tradeoffs
|
||||
Applying DRY may:
|
||||
- Increase indirection
|
||||
- Reduce local readability
|
||||
- Introduce coupling between modules
|
||||
|
||||
These costs are acceptable only when outweighed by reduced change cost
|
||||
and increased behavioral consistency.
|
||||
|
||||
---
|
||||
|
||||
## AgentUsage
|
||||
|
||||
### When to Apply This Lens
|
||||
- During refactoring or maintenance work
|
||||
- When change requests repeatedly touch similar code
|
||||
- When bugs recur due to partial updates
|
||||
- During architectural consolidation
|
||||
|
||||
### When to Suspend This Lens
|
||||
- During early exploration or prototyping
|
||||
- When future variability is unclear
|
||||
- When isolation is more valuable than reuse
|
||||
|
||||
### Expected Agent Output
|
||||
- Identified DRY violations with locations
|
||||
- Rationale for why duplication matters
|
||||
- Volatility assessment (stable vs evolving)
|
||||
- Recommended refactor type and target
|
||||
- Risk notes and minimal patch sequence
|
||||
|
||||
xxx
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user