coulomb/markitect-tool
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

scoped migration filter workplan and docs

Browse Source
This commit is contained in:
Bernd Worsch
2026-05-03 21:12:22 +02:00
parent f50d9567a1
commit 68753a4500
5 changed files with 349 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
docs/compatibility-policy.md Normal file
View File

@@ -0,0 +1,47 @@
# Compatibility Policy
`markitect-tool` is a successor, not a compatibility release of
`markitect-main`.
## Default
Do not preserve legacy module paths, database-backed workflows, or command names
unless the new PRD/FRS requires the behavior.
The durable compatibility target is the new product contract:
- Markdown input/output
- Structured document representation
- Schema validation and reporting
- Transform/query/template operations
- `mkt` CLI and library API
## What May Be Compatible
Compatibility is acceptable for:
- Markdown schema file format, if it remains clean and provider-neutral.
- Validation error concepts and output formats, if they fit the new structured
error model.
- Template variable syntax, if deterministic and documented.
- Transclusion/include directive syntax, if made explicit and testable.
- Selected fixture files, if they do not encode platform assumptions.
## What Should Not Be Compatible
Do not carry forward:
- `markitect.*` import paths as public API.
- `markitect` CLI command names as MVP aliases.
- SQLite/database ingest/list/query behavior.
- GraphQL API behavior.
- Infospace, spaces, prompt execution, LLM provider configuration, rendering,
asset, finance, wishlist, or legacy-management behavior.
- Backward compatibility shims whose only purpose is old internal tooling.
## Migration Stance
Consumers should migrate to `mkt` and the new library API intentionally. Any
compatibility shim must have a named downstream consumer, a removal condition or
stability commitment, and tests proving it does not pull platform concerns into
the core.

54
docs/dependency-classification.md Normal file
View File

@@ -0,0 +1,54 @@
# Dependency Classification
Source: `/home/worsch/markitect-main/pyproject.toml` and `package.json`
## Core Candidates
| Dependency | Recommendation | Reason |
| --- | --- | --- |
| `markdown-it-py` | Keep | Mature Markdown parser; good fit for deterministic parse contract. |
| `PyYAML` | Keep | Needed for frontmatter, config, and markdown schema metadata. |
| `click>=8` | Keep or replace deliberately | Legacy CLI uses Click. Keeping it lowers migration cost. |
| `jsonpath-ng>=1.5` | Conditional | Useful only if `mkt query` adopts JSONPath over structured docs. |
| `toml` / stdlib `tomllib` | Conditional | Use `tomllib` for read-only TOML on Python 3.11+; add a writer only if needed. |
| `tabulate` | Optional | Nice for human tables; structured output should not depend on it. |
## Optional Integration Candidates
| Dependency | Recommendation | Reason |
| --- | --- | --- |
| `llm-connect` | Optional extra only | FR-042 allows LLM-assisted generation, but core must remain provider-neutral. |
| `networkx` | Defer | Maybe useful later for transclusion reference graphs; mostly higher-layer. |
| `markdownify`, `markitdown-no-magika`, `pymupdf4llm` | Do not include in core | Multi-format proxy conversion is out of markdown-native scope. |
## Drop from Successor Core
| Dependency/source | Reason |
| --- | --- |
| Local capability packages | Old project ecosystem/tooling, not syntax-layer contract. |
| `aiohttp` | Only needed by old service/provider integrations. |
| `jsdom`, `jest` | Browser/editor/rendering tests are out of scope for core toolkit. |
## Recommended Initial Stack
Assuming Python 3.12+:
```toml
[project]
dependencies = [
"markdown-it-py",
"PyYAML",
"click>=8.0",
]
[project.optional-dependencies]
query = ["jsonpath-ng>=1.5"]
tables = ["tabulate>=0.9"]
llm = ["llm-connect @ file:///home/worsch/llm-connect"]
dev = ["pytest"]
```
## Packaging Decision Needed
`MKTT-WP-0001-T004` should decide Python minimum version, package import name,
CLI entry point, build backend, versioning, test runner, and lint/type tools.

84
docs/legacy-cli-command-map.md Normal file
View File

@@ -0,0 +1,84 @@
# Legacy CLI Command Map
Source command: `markitect`
Successor command: `mkt`
This map classifies `markitect-main` commands as migrate, reimplement, defer,
or drop for `markitect-tool`.
## Command Policy
`mkt` should expose composable syntax-layer operations. It should not expose
database lifecycle, platform services, rendering applications, project finance,
wishlist, or legacy management commands.
## Core Commands to Reimplement
| Legacy command | Future `mkt` shape | Decision |
| --- | --- | --- |
| `ast-show` | `mkt parse <file> --format tree/json` | Reimplement with new representation. |
| `ast-query` | `mkt query <file> <selector>` | Reimplement; selector syntax TBD. |
| `ast-stats` | `mkt inspect <file> --stats` | Reimplement if lightweight. |
| `metadata` | `mkt matter get <file>` or parse output field | Reimplement around matter contract. |
| `frontmatter-*` | `mkt matter front ...` | Reimplement if matter editing remains in scope. |
| `contentmatter-*` | `mkt matter content ...` | Reimplement only if contentmatter is formalized. |
| `tailmatter-*` | `mkt matter tail ...` | Reimplement only if tailmatter is formalized. |
| `schema-generate` | `mkt schema derive <file>` | Reimplement. |
| `validate` | `mkt validate <file> --schema <schema>` | Reimplement. |
| `schema-validate` | `mkt schema validate <schema>` | Reimplement. |
| `schema-analyze` | `mkt schema inspect <schema>` | Defer unless needed for MVP. |
| `schema-refine` | `mkt schema refine <schema>` | Defer; only deterministic refinements. |
| `generate-stub` | `mkt generate stub --schema <schema>` | Reimplement. |
| `generate-drafts` | `mkt generate drafts --schema <schema> --data <data>` | Reimplement later. |
| `template-render` | `mkt template render <template> --data <data>` | Reimplement. |
| `ingest-dir` | `mkt batch parse <dir>` | Rename; avoid database implication. |
| `batch-process` | `mkt batch run <glob> --op <operation>` | Reimplement around operation pipeline. |
| `recursive` | `mkt batch run <dir> --recursive --depth <n>` | Fold into batch options. |
| `cache-stats` | `mkt cache stats` | Reimplement after cache contract. |
| `cache-clean` | `mkt cache clean` | Reimplement after cache contract. |
| `cache-invalidate` | `mkt cache invalidate <file>` | Reimplement after cache contract. |
| `config-show` | `mkt config show` | Reimplement. |
| `config-set` | `mkt config set <key> <value>` | Reimplement. |
| `config-init` | `mkt config init` | Reimplement. |
| `config-validate` | `mkt config validate` | Reimplement. |
| `config-help` | `mkt config help [key]` | Reimplement if docs stay embedded. |
## Commands to Drop or Leave to Higher Layers
| Legacy command/group | Reason |
| --- | --- |
| `db-*` | Persistent database/service layer is out of scope. |
| `schema-ingest/list/get/delete` | Old behavior stores schemas in DB; successor should use files. |
| `graphql-*` | GraphQL service is not syntax toolkit core. |
| `plugin-*` | Old plugin system is mostly rendering/application infrastructure. |
| `search *` | FTS index service is out of initial scope; structured query comes first. |
| `legacy *` | Old-project maintenance. |
| `cost`, `worktime`, `profile` | Project/finance/user tooling. |
| `wish *` | Product-management workflow belongs to State Hub/Custodian. |
| `llm-*`, `helper` | Provider-specific LLM operations belong in `llm-connect` or higher layers. |
| `prompt *` | Prompt dependency execution and quality gates are out of scope. |
| `infospace *`, `spaces *` | Knowledge lifecycle/platform layer. |
| `proxy *` | Multi-format conversion is external preprocessing. |
| `asset *`, `package *`, `workspace *` | Asset/rendering/package application concerns. |
| `perf-*` | Keep focused benchmarks as tests instead. |
## Proposed `mkt` MVP Surface
```text
mkt parse <file> [--format json|yaml|tree]
mkt query <file-or-dir> <selector> [--format json|yaml|text]
mkt validate <file-or-dir> --schema <schema>
mkt schema derive <file-or-dir> [--output <schema.md>]
mkt schema validate <schema>
mkt transform <file> --operation <op> [--output <file>]
mkt compose <manifest-or-files> [--output <file>]
mkt include <file> [--base-dir <dir>]
mkt template render <template> --data <data>
mkt generate stub --schema <schema>
mkt batch run <path-or-glob> --op <parse|validate|query|transform>
mkt config show|set|init|validate|help
mkt cache stats|clean|invalidate
```
Start MVP with parse, validate, schema derive/validate, template render, batch
discovery, and config.

152
docs/markitect-main-test-migration-inventory.md Normal file
View File

@@ -0,0 +1,152 @@
# markitect-main Test Migration Inventory
Source repo: `/home/worsch/markitect-main`
This inventory classifies old tests and fixtures by the `markitect-tool` FRS.
It is a migration filter, not a promise to copy tests verbatim.
## Migration Rule
Migrate tests when they verify provider-neutral markdown behavior. Rewrite tests
when the old assertion depends on database state, platform lifecycle, UI
rendering, domain-specific infospace semantics, or provider-specific LLM calls.
## FR-001/002: Markdown Parsing and Structuring
Candidate tests and fixtures:
| Source | Use |
| --- | --- |
| `tests/test_l7_foundation_markdown_parsing.py` | Minimal parser smoke test for heading and paragraph tokens. |
| `tests/fixtures/frontmatter_test_files/*.md` | YAML/JSON/TOML/no-frontmatter fixtures. |
| `tests/fixtures/content_test_files/*.md` | Frontmatter/content/tailmatter segmentation fixtures. |
| `tests/fixtures/markdown_samples.py` | Reusable samples if not coupled to legacy AST shape. |
| `testdata/test_frontmatter.md`, `testdata/test_roundtrip.md` | Roundtrip and frontmatter fixtures. |
| `tests/test_roundtrip_consolidated.py` | Review for parse/serialize expectations. |
| `tests/test_issue_51_outline_mode.py`, `tests/test_issue_52_heading_text_capture.py` | Review for heading tree behavior. |
Define the new structured representation first, then port only behavior that
still applies.
## FR-010/012: Schema Definition and Validation
Strong candidates:
| Source | Use |
| --- | --- |
| `tests/test_schema_loader.py` | Markdown schema load/save/roundtrip, frontmatter extraction, JSON extraction, edge cases. |
| `tests/test_issue_5_schema_generation.py` | Schema derivation from simple/complex markdown, depth limits, empty/file-not-found behavior. |
| `tests/test_issue_7_schema_validation.py` | Matching/non-matching schemas, invalid inputs, depth-limited schema validation. |
| `tests/test_issue_8_validation_errors.py` | Structured validation errors, formatting, summaries, file/string inputs. |
| `tests/test_schema_metaschema.py` | Metaschema validation fixtures. |
| `tests/test_schema_naming.py` | Migrate only if schema naming remains part of the contract. |
| `markitect/schemas/*.md`, `markitect/schemas/markitect-metaschema.json` | Candidate fixtures, not automatically built-in product schemas. |
Do not migrate database-backed schema ingest/list/get/delete behavior.
## FR-020/022: Transformation and Composition
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_issue_150_transclusion_engine.py` | Includes, variables, conditionals, circular references, depth limits, partial errors. |
| `tests/test_issue_149_explode_implode_variants.py` | Review explode/implode variants as transform primitives. |
| `tests/unit/infospace/test_composition.py` | Extract only generic markdown composition expectations. |
| `tests/unit/spaces/test_transclusion.py` | Extract only generic transclusion rules. |
Rewrite around a small operation API. Avoid importing `infospace`, `spaces`, or
packaging semantics into the toolkit core.
## FR-030/031: Query and Extraction
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_l4_service_ast_analysis.py` | AST display/query/stats behavior and JSONPath error handling. |
| `markitect/query_paradigms/tests/test_query_paradigms.py` | Review for adapter boundary ideas only. |
| `markitect/graphql/tests/*` | Do not migrate; GraphQL service is out of scope. |
Initial query tests should cover selector matches, empty matches, invalid
selectors, and structured output over parsed documents.
## FR-040/042: Templating and Generation
Strong candidates:
| Source | Use |
| --- | --- |
| `tests/test_issue_65_template_parser.py` | Variable detection, nested variables, duplicates, malformed braces, empty templates. |
| `tests/test_issue_65_template_substitution.py` | Strict/lenient substitution, nested data, markdown formatting, special characters, large templates. |
| `tests/test_issue_6_stub_generation.py` | Review for schema-to-markdown stub generation. |
Do not migrate prompt execution or LLM quality gates into core. FR-042 should be
an optional adapter hook after deterministic generation works.
## FR-050/052: CLI, Batch, and Repeatability
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_issue_17_batch_processing.py` | File discovery, recursion, depth limits, glob behavior, stop/continue errors. |
| `tests/test_cli_simple.py`, `tests/test_cli_integration.py`, `tests/test_issue_6_cli_integration.py` | Review command ergonomics only; command names change to `mkt`. |
| `tests/test_l4_service_output_formatting.py` | Structured output formatting expectations. |
Rewrite around new `mkt` commands. Do not keep database ingest/status/list as
first-class behavior.
## FR-060/061: Configuration Handling
Strong candidates:
| Source | Use |
| --- | --- |
| `tests/test_issue_18_config_management.py` | Config show/set/init/validate/help, YAML/JSON parsing, env vars, sensitive masking. |
| `tests/test_issue_37_configuration_integration.py` | Review for config precedence if not provider-specific. |
Drop LLM provider preference handling from core config.
## FR-070/071: Caching and Incremental Processing
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_l5_infrastructure_cache_management.py` | Cache stats/clean/invalidate behavior and feedback. |
| `markitect/ast_cache.py`, `cache_service.py` | Reference after parser representation stabilizes. |
Do not migrate prompt incremental recompute tests.
## FR-080/081: Structured Errors
Candidate tests:
| Source | Use |
| --- | --- |
| `tests/test_issue_8_validation_errors.py` | Highest-value source for typed validation failures and formatting. |
| `tests/test_issue_152_153_edge_cases.py` | Review for generic edge cases. |
| `markitect/exceptions.py`, `validation_error.py` | Reference for error taxonomy. |
Structured errors should be introduced early because they shape CLI and API
behavior across all FRS areas.
## Explicitly Out of Scope
Do not migrate:
- `tests/unit/infospace/*`, except tiny generic markdown composition ideas.
- `tests/unit/spaces/*`, except generic transclusion ideas.
- `tests/unit/prompts/*`, `tests/integration/prompts/*`.
- `tests/unit/llm/*`, `tests/integration/llm/*`.
- Asset/package/rendering/plugin/browser/editor/theme tests.
- `markitect/finance/tests/*`, profile tests, legacy management tests,
wishlist/project tooling tests.
## First Test Migration Slice
Start with parser/frontmatter fixtures, `tests/test_schema_loader.py`,
`tests/test_issue_7_schema_validation.py`, `tests/test_issue_8_validation_errors.py`,
template parser/substitution tests, and batch discovery tests without database
processors.

16
workplans/MKTT-WP-0002-markitect-main-migration.md
View File

@@ -34,7 +34,7 @@ out-of-scope areas, and migration principles.
```task
id: MKTT-WP-0002-T002
status: todo
status: done
priority: high
state_hub_task_id: "994a8c17-bd13-476b-9b8e-86c50e1c4897"
```
@@ -44,11 +44,13 @@ tests. Create a candidate test migration list organized by FRS section.
Technical debt: `TD-MKTT-002`.
Output: `docs/markitect-main-test-migration-inventory.md`.
## P2.3 - Map legacy commands to `mkt`
```task
id: MKTT-WP-0002-T003
status: todo
status: done
priority: high
state_hub_task_id: "f82b1f2c-887b-4378-b70b-84c1cb051742"
```
@@ -56,11 +58,13 @@ state_hub_task_id: "f82b1f2c-887b-4378-b70b-84c1cb051742"
Inventory old `markitect` CLI commands and classify each as migrate,
reimplement, rename, defer, or out of scope for `mkt`.
Output: `docs/legacy-cli-command-map.md`.
## P2.4 - Decide compatibility policy
```task
id: MKTT-WP-0002-T004
status: todo
status: done
priority: medium
state_hub_task_id: "56a12905-c3ec-42b0-893a-47ef023ea10a"
```
@@ -68,6 +72,8 @@ state_hub_task_id: "56a12905-c3ec-42b0-893a-47ef023ea10a"
Define whether any old module paths, command names, file formats, or schema
names deserve compatibility shims in the successor.
Output: `docs/compatibility-policy.md`.
## Extension Points
`EP-MKTT-002` tracks the query backend adapter boundary: keep the initial query
@@ -78,10 +84,12 @@ the provider-neutral contract.
```task
id: MKTT-WP-0002-T005
status: todo
status: done
priority: medium
state_hub_task_id: "4d51e763-bf5f-483b-8948-6306ff5a58bb"
```
Separate core dependencies from optional integrations. Initial candidates:
`markdown-it-py`, `PyYAML`, `jsonpath-ng`, and optional `llm-connect`.
Output: `docs/dependency-classification.md`.