coulomb/markitect-tool
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

scoped migration filter workplan and docs

Browse Source
This commit is contained in:
Bernd Worsch
2026-05-03 21:12:22 +02:00
parent f50d9567a1
commit 68753a4500
5 changed files with 349 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
docs/compatibility-policy.md Normal file
View File

@@ -0,0 +1,47 @@
# Compatibility Policy
`markitect-tool` is a successor, not a compatibility release of
`markitect-main`.
## Default
Do not preserve legacy module paths, database-backed workflows, or command names
unless the new PRD/FRS requires the behavior.
The durable compatibility target is the new product contract:
- Markdown input/output
- Structured document representation
- Schema validation and reporting
- Transform/query/template operations
- `mkt` CLI and library API
## What May Be Compatible
Compatibility is acceptable for:
- Markdown schema file format, if it remains clean and provider-neutral.
- Validation error concepts and output formats, if they fit the new structured
error model.
- Template variable syntax, if deterministic and documented.
- Transclusion/include directive syntax, if made explicit and testable.
- Selected fixture files, if they do not encode platform assumptions.
## What Should Not Be Compatible
Do not carry forward:
- `markitect.*` import paths as public API.
- `markitect` CLI command names as MVP aliases.
- SQLite/database ingest/list/query behavior.
- GraphQL API behavior.
- Infospace, spaces, prompt execution, LLM provider configuration, rendering,
asset, finance, wishlist, or legacy-management behavior.
- Backward compatibility shims whose only purpose is old internal tooling.
## Migration Stance
Consumers should migrate to `mkt` and the new library API intentionally. Any
compatibility shim must have a named downstream consumer, a removal condition or
stability commitment, and tests proving it does not pull platform concerns into
the core.

54
docs/dependency-classification.md Normal file
View File

@@ -0,0 +1,54 @@
# Dependency Classification
Source: `/home/worsch/markitect-main/pyproject.toml` and `package.json`
## Core Candidates
| Dependency | Recommendation | Reason |
| --- | --- | --- |
| `markdown-it-py` | Keep | Mature Markdown parser; good fit for deterministic parse contract. |
| `PyYAML` | Keep | Needed for frontmatter, config, and markdown schema metadata. |
| `click>=8` | Keep or replace deliberately | Legacy CLI uses Click. Keeping it lowers migration cost. |
| `jsonpath-ng>=1.5` | Conditional | Useful only if `mkt query` adopts JSONPath over structured docs. |
| `toml` / stdlib `tomllib` | Conditional | Use `tomllib` for read-only TOML on Python 3.11+; add a writer only if needed. |
| `tabulate` | Optional | Nice for human tables; structured output should not depend on it. |
## Optional Integration Candidates
| Dependency | Recommendation | Reason |
| --- | --- | --- |
| `llm-connect` | Optional extra only | FR-042 allows LLM-assisted generation, but core must remain provider-neutral. |
| `networkx` | Defer | Maybe useful later for transclusion reference graphs; mostly higher-layer. |
| `markdownify`, `markitdown-no-magika`, `pymupdf4llm` | Do not include in core | Multi-format proxy conversion is out of markdown-native scope. |
## Drop from Successor Core
| Dependency/source | Reason |
| --- | --- |
| Local capability packages | Old project ecosystem/tooling, not syntax-layer contract. |
| `aiohttp` | Only needed by old service/provider integrations. |
| `jsdom`, `jest` | Browser/editor/rendering tests are out of scope for core toolkit. |
## Recommended Initial Stack
Assuming Python 3.12+:
```toml
[project]
dependencies = [
"markdown-it-py",
"PyYAML",
"click>=8.0",
]
[project.optional-dependencies]
query = ["jsonpath-ng>=1.5"]
tables = ["tabulate>=0.9"]
llm = ["llm-connect @ file:///home/worsch/llm-connect"]
dev = ["pytest"]
```
## Packaging Decision Needed
`MKTT-WP-0001-T004` should decide Python minimum version, package import name,
CLI entry point, build backend, versioning, test runner, and lint/type tools.

84
docs/legacy-cli-command-map.md Normal file
View File

@@ -0,0 +1,84 @@
# Legacy CLI Command Map
Source command: `markitect`
Successor command: `mkt`
This map classifies `markitect-main` commands as migrate, reimplement, defer,
or drop for `markitect-tool`.
## Command Policy
`mkt` should expose composable syntax-layer operations. It should not expose
database lifecycle, platform services, rendering applications, project finance,
wishlist, or legacy management commands.
## Core Commands to Reimplement
| Legacy command | Future `mkt` shape | Decision |
| --- | --- | --- |
| `ast-show` | `mkt parse <file> --format tree/json` | Reimplement with new representation. |
| `ast-query` | `mkt query <file> <selector>` | Reimplement; selector syntax TBD. |
| `ast-stats` | `mkt inspect <file> --stats` | Reimplement if lightweight. |
| `metadata` | `mkt matter get <file>` or parse output field | Reimplement around matter contract. |
| `frontmatter-*` | `mkt matter front ...` | Reimplement if matter editing remains in scope. |
| `contentmatter-*` | `mkt matter content ...` | Reimplement only if contentmatter is formalized. |
| `tailmatter-*` | `mkt matter tail ...` | Reimplement only if tailmatter is formalized. |
| `schema-generate` | `mkt schema derive <file>` | Reimplement. |
| `validate` | `mkt validate <file> --schema <schema>` | Reimplement. |
| `schema-validate` | `mkt schema validate <schema>` | Reimplement. |
| `schema-analyze` | `mkt schema inspect <schema>` | Defer unless needed for MVP. |
| `schema-refine` | `mkt schema refine <schema>` | Defer; only deterministic refinements. |
| `generate-stub` | `mkt generate stub --schema <schema>` | Reimplement. |
| `generate-drafts` | `mkt generate drafts --schema <schema> --data <data>` | Reimplement later. |
| `template-render` | `mkt template render <template> --data <data>` | Reimplement. |
| `ingest-dir` | `mkt batch parse <dir>` | Rename; avoid database implication. |
| `batch-process` | `mkt batch run <glob> --op <operation>` | Reimplement around operation pipeline. |
| `recursive` | `mkt batch run <dir> --recursive --depth <n>` | Fold into batch options. |
| `cache-stats` | `mkt cache stats` | Reimplement after cache contract. |
| `cache-clean` | `mkt cache clean` | Reimplement after cache contract. |
| `cache-invalidate` | `mkt cache invalidate <file>` | Reimplement after cache contract. |
| `config-show` | `mkt config show` | Reimplement. |
| `config-set` | `mkt config set <key> <value>` | Reimplement. |
| `config-init` | `mkt config init` | Reimplement. |
| `config-validate` | `mkt config validate` | Reimplement. |
| `config-help` | `mkt config help [key]` | Reimplement if docs stay embedded. |
## Commands to Drop or Leave to Higher Layers
| Legacy command/group | Reason |
| --- | --- |
| `db-*` | Persistent database/service layer is out of scope. |
| `schema-ingest/list/get/delete` | Old behavior stores schemas in DB; successor should use files. |
| `graphql-*` | GraphQL service is not syntax toolkit core. |
| `plugin-*` | Old plugin system is mostly rendering/application infrastructure. |
| `search *` | FTS index service is out of initial scope; structured query comes first. |
| `legacy *` | Old-project maintenance. |
| `cost`, `worktime`, `profile` | Project/finance/user tooling. |
| `wish *` | Product-management workflow belongs to State Hub/Custodian. |
| `llm-*`, `helper` | Provider-specific LLM operations belong in `llm-connect` or higher layers. |
| `prompt *` | Prompt dependency execution and quality gates are out of scope. |
| `infospace *`, `spaces *` | Knowledge lifecycle/platform layer. |
| `proxy *` | Multi-format conversion is external preprocessing. |
| `asset *`, `package *`, `workspace *` | Asset/rendering/package application concerns. |
| `perf-*` | Keep focused benchmarks as tests instead. |
## Proposed `mkt` MVP Surface
```text
mkt parse <file> [--format json|yaml|tree]
mkt query <file-or-dir> <selector> [--format json|yaml|text]
mkt validate <file-or-dir> --schema <schema>
mkt schema derive <file-or-dir> [--output <schema.md>]
mkt schema validate <schema>
mkt transform <file> --operation <op> [--output <file>]
mkt compose <manifest-or-files> [--output <file>]
mkt include <file> [--base-dir <dir>]
mkt template render <template> --data <data>
mkt generate stub --schema <schema>
mkt batch run <path-or-glob> --op <parse|validate|query|transform>
mkt config show|set|init|validate|help
mkt cache stats|clean|invalidate
```
Start MVP with parse, validate, schema derive/validate, template render, batch
discovery, and config.

152
docs/markitect-main-test-migration-inventory.md Normal file
View File

@@ -0,0 +1,152 @@
# markitect-main Test Migration Inventory
Source repo: `/home/worsch/markitect-main`
This inventory classifies old tests and fixtures by the `markitect-tool` FRS.
It is a migration filter, not a promise to copy tests verbatim.
## Migration Rule
Migrate tests when they verify provider-neutral markdown behavior. Rewrite tests
when the old assertion depends on database state, platform lifecycle, UI
rendering, domain-specific infospace semantics, or provider-specific LLM calls.
## FR-001/002: Markdown Parsing and Structuring
Candidate tests and fixtures:
| Source | Use |
| --- | --- |
| `tests/test_l7_foundation_markdown_parsing.py` | Minimal parser smoke test for heading and paragraph tokens. |
| `tests/fixtures/frontmatter_test_files/*.md` | YAML/JSON/TOML/no-frontmatter fixtures. |
| `tests/fixtures/content_test_files/*.md` | Frontmatter/content/tailmatter segmentation fixtures. |
| `tests/fixtures/markdown_samples.py` | Reusable samples if not coupled to legacy AST shape. |
| `testdata/test_frontmatter.md`, `testdata/test_roundtrip.md` | Roundtrip and frontmatter fixtures. |
| `tests/test_roundtrip_consolidated.py` | Review for parse/serialize expectations. |
| `tests/test_issue_51_outline_mode.py`, `tests/test_issue_52_heading_text_capture.py` | Review for heading tree behavior. |
Define the new structured representation first, then port only behavior that
still applies.
## FR-010/012: Schema Definition and Validation
Strong candidates:
| Source | Use |
| --- | --- |
| `tests/test_schema_loader.py` | Markdown schema load/save/roundtrip, frontmatter extraction, JSON extraction, edge cases. |
| `tests/test_issue_5_schema_generation.py` | Schema derivation from simple/complex markdown, depth limits, empty/file-not-found behavior. |
| `tests/test_issue_7_schema_validation.py` | Matching/non-matching schemas, invalid inputs, depth-limited schema validation. |
| `tests/test_issue_8_validation_errors.py` | Structured validation errors, formatting, summaries, file/string inputs. |
| `tests/test_schema_metaschema.py` | Metaschema validation fixtures. |
| `tests/test_schema_naming.py` | Migrate only if schema naming remains part of the contract. |
| `markitect/schemas/*.md`, `markitect/schemas/markitect-metaschema.json` | Candidate fixtures, not automatically built-in product schemas. |
Do not migrate database-backed schema ingest/list/get/delete behavior.
## FR-020/022: Transformation and Composition
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_issue_150_transclusion_engine.py` | Includes, variables, conditionals, circular references, depth limits, partial errors. |
| `tests/test_issue_149_explode_implode_variants.py` | Review explode/implode variants as transform primitives. |
| `tests/unit/infospace/test_composition.py` | Extract only generic markdown composition expectations. |
| `tests/unit/spaces/test_transclusion.py` | Extract only generic transclusion rules. |
Rewrite around a small operation API. Avoid importing `infospace`, `spaces`, or
packaging semantics into the toolkit core.
## FR-030/031: Query and Extraction
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_l4_service_ast_analysis.py` | AST display/query/stats behavior and JSONPath error handling. |
| `markitect/query_paradigms/tests/test_query_paradigms.py` | Review for adapter boundary ideas only. |
| `markitect/graphql/tests/*` | Do not migrate; GraphQL service is out of scope. |
Initial query tests should cover selector matches, empty matches, invalid
selectors, and structured output over parsed documents.
## FR-040/042: Templating and Generation
Strong candidates:
| Source | Use |
| --- | --- |
| `tests/test_issue_65_template_parser.py` | Variable detection, nested variables, duplicates, malformed braces, empty templates. |
| `tests/test_issue_65_template_substitution.py` | Strict/lenient substitution, nested data, markdown formatting, special characters, large templates. |
| `tests/test_issue_6_stub_generation.py` | Review for schema-to-markdown stub generation. |
Do not migrate prompt execution or LLM quality gates into core. FR-042 should be
an optional adapter hook after deterministic generation works.
## FR-050/052: CLI, Batch, and Repeatability
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_issue_17_batch_processing.py` | File discovery, recursion, depth limits, glob behavior, stop/continue errors. |
| `tests/test_cli_simple.py`, `tests/test_cli_integration.py`, `tests/test_issue_6_cli_integration.py` | Review command ergonomics only; command names change to `mkt`. |
| `tests/test_l4_service_output_formatting.py` | Structured output formatting expectations. |
Rewrite around new `mkt` commands. Do not keep database ingest/status/list as
first-class behavior.
## FR-060/061: Configuration Handling
Strong candidates:
| Source | Use |
| --- | --- |
| `tests/test_issue_18_config_management.py` | Config show/set/init/validate/help, YAML/JSON parsing, env vars, sensitive masking. |
| `tests/test_issue_37_configuration_integration.py` | Review for config precedence if not provider-specific. |
Drop LLM provider preference handling from core config.
## FR-070/071: Caching and Incremental Processing
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_l5_infrastructure_cache_management.py` | Cache stats/clean/invalidate behavior and feedback. |
| `markitect/ast_cache.py`, `cache_service.py` | Reference after parser representation stabilizes. |
Do not migrate prompt incremental recompute tests.
## FR-080/081: Structured Errors
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_issue_8_validation_errors.py` | Highest-value source for typed validation failures and formatting. |
| `tests/test_issue_152_153_edge_cases.py` | Review for generic edge cases. |
| `markitect/exceptions.py`, `validation_error.py` | Reference for error taxonomy. |
Structured errors should be introduced early because they shape CLI and API
behavior across all FRS areas.
## Explicitly Out of Scope
Do not migrate:
- `tests/unit/infospace/*`, except tiny generic markdown composition ideas.
- `tests/unit/spaces/*`, except generic transclusion ideas.
- `tests/unit/prompts/*`, `tests/integration/prompts/*`.
- `tests/unit/llm/*`, `tests/integration/llm/*`.
- Asset/package/rendering/plugin/browser/editor/theme tests.
- `markitect/finance/tests/*`, profile tests, legacy management tests,
wishlist/project tooling tests.
## First Test Migration Slice
Start with parser/frontmatter fixtures, `tests/test_schema_loader.py`,
`tests/test_issue_7_schema_validation.py`, `tests/test_issue_8_validation_errors.py`,
template parser/substitution tests, and batch discovery tests without database
processors.

16
workplans/MKTT-WP-0002-markitect-main-migration.md
View File

@@ -34,7 +34,7 @@ out-of-scope areas, and migration principles.
```task ```task
id: MKTT-WP-0002-T002 id: MKTT-WP-0002-T002
status: todo status: done
priority: high priority: high
state_hub_task_id: "994a8c17-bd13-476b-9b8e-86c50e1c4897" state_hub_task_id: "994a8c17-bd13-476b-9b8e-86c50e1c4897"
``` ```
@@ -44,11 +44,13 @@ tests. Create a candidate test migration list organized by FRS section.
Technical debt: `TD-MKTT-002`. Technical debt: `TD-MKTT-002`.
Output: `docs/markitect-main-test-migration-inventory.md`.
## P2.3 - Map legacy commands to `mkt` ## P2.3 - Map legacy commands to `mkt`
```task ```task
id: MKTT-WP-0002-T003 id: MKTT-WP-0002-T003
status: todo status: done
priority: high priority: high
state_hub_task_id: "f82b1f2c-887b-4378-b70b-84c1cb051742" state_hub_task_id: "f82b1f2c-887b-4378-b70b-84c1cb051742"
``` ```
@@ -56,11 +58,13 @@ state_hub_task_id: "f82b1f2c-887b-4378-b70b-84c1cb051742"
Inventory old `markitect` CLI commands and classify each as migrate, Inventory old `markitect` CLI commands and classify each as migrate,
reimplement, rename, defer, or out of scope for `mkt`. reimplement, rename, defer, or out of scope for `mkt`.
Output: `docs/legacy-cli-command-map.md`.
## P2.4 - Decide compatibility policy ## P2.4 - Decide compatibility policy
```task ```task
id: MKTT-WP-0002-T004 id: MKTT-WP-0002-T004
status: todo status: done
priority: medium priority: medium
state_hub_task_id: "56a12905-c3ec-42b0-893a-47ef023ea10a" state_hub_task_id: "56a12905-c3ec-42b0-893a-47ef023ea10a"
``` ```
@@ -68,6 +72,8 @@ state_hub_task_id: "56a12905-c3ec-42b0-893a-47ef023ea10a"
Define whether any old module paths, command names, file formats, or schema Define whether any old module paths, command names, file formats, or schema
names deserve compatibility shims in the successor. names deserve compatibility shims in the successor.
Output: `docs/compatibility-policy.md`.
## Extension Points ## Extension Points
`EP-MKTT-002` tracks the query backend adapter boundary: keep the initial query `EP-MKTT-002` tracks the query backend adapter boundary: keep the initial query
@@ -78,10 +84,12 @@ the provider-neutral contract.
```task ```task
id: MKTT-WP-0002-T005 id: MKTT-WP-0002-T005
status: todo status: done
priority: medium priority: medium
state_hub_task_id: "4d51e763-bf5f-483b-8948-6306ff5a58bb" state_hub_task_id: "4d51e763-bf5f-483b-8948-6306ff5a58bb"
``` ```
Separate core dependencies from optional integrations. Initial candidates: Separate core dependencies from optional integrations. Initial candidates:
`markdown-it-py`, `PyYAML`, `jsonpath-ng`, and optional `llm-connect`. `markdown-it-py`, `PyYAML`, `jsonpath-ng`, and optional `llm-connect`.
Output: `docs/dependency-classification.md`.