diff --git a/INTENT.md b/INTENT.md index 80be1c6..7bc36ef 100644 --- a/INTENT.md +++ b/INTENT.md @@ -23,6 +23,8 @@ It turns markdown from plain text into a **programmable knowledge substrate**. The current practical usecase matrix is maintained in `docs/practical-usecase-relevance.md`. That document translates this intent into adoption-oriented scenarios, relevance expectations, and E2E coverage targets. +First-use documentation lives in `docs/getting-started.md`, with a command +cheatsheet and examples index beside it. --- diff --git a/README.md b/README.md index 5c18605..dac63a6 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,10 @@ requirements documents in `wiki/`. - `wiki/FunctionalRequirementsSpecification.md` - observable functional behavior - `docs/markitect-main-scope-assessment.md` - migration assessment from `markitect-main` - `docs/practical-usecase-relevance.md` - practical adoption usecases and relevance expectations +- `docs/getting-started.md` - first-use guide from checkout to practical commands +- `docs/command-cheatsheet.md` - command-oriented workflow cheat sheet +- `docs/examples-index.md` - map from examples to usecases and commands +- `docs/performance-notes.md` - local performance posture and smoke coverage - `docs/cli-reference.md` - generated `mkt` command reference - `docs/api-reference.md` - generated public API reference - `workplans/` - source-of-truth implementation and migration plans @@ -32,6 +36,13 @@ Run the tests: python3 -m pytest ``` +Install locally from a checkout: + +```bash +python3 -m pip install -e . +mkt --help +``` + Try the parser CLI from a checkout: ```bash diff --git a/docs/command-cheatsheet.md b/docs/command-cheatsheet.md new file mode 100644 index 0000000..0aab229 --- /dev/null +++ b/docs/command-cheatsheet.md @@ -0,0 +1,108 @@ +# Command Cheatsheet + +Use `mkt --help` and `mkt --help` for the live command surface. This +cheatsheet is organized by intent. + +## Inspect Markdown + +```bash +mkt parse file.md --format json +mkt parse file.md --format tree +mkt ast show file.md --format yaml +mkt ast stats file.md +mkt metrics file.md +``` + +## Validate Structure + +```bash +mkt validate file.md --schema schema.md +mkt schema validate schema.md +mkt contract validate contract.md +mkt contract check file.md --contract contract.md +mkt contract form-state file.md --contract contract.md --context context.yaml +``` + +## Query And Extract + +```bash +mkt query file.md 'sections[heading=Decision]' +mkt extract file.md 'sections[heading=Decision]' --format text +mkt query file.md '$.sections[*].heading.text' --engine jsonpath +``` + +## Transform And Reuse + +```bash +mkt transform file.md --set status=ready +mkt compose one.md two.md --title Combined +mkt include file.md --base-dir . +mkt ref resolve context.md 'std:clauses.md#payment-terms' --root examples/references +mkt process file.md --root . +``` + +## Split And Literate Workflows + +```bash +mkt explode file.md --output-dir parts +mkt implode parts --output rebuilt.md +mkt tangle examples/literate/app.md --output-dir build +mkt weave examples/literate/app.md --output woven.md +``` + +## Generate Markdown + +```bash +mkt template inspect template.md +mkt template render template.md --data data.yaml +mkt generate stub --contract contract.md --include-optional +mkt generate rules generation.md --output-dir out +mkt function list +mkt function check file.md +mkt function render file.md +``` + +## Index, Search, And Refresh + +```bash +mkt cache init --root . +mkt cache build docs --root . +mkt cache status docs --root . +mkt cache index docs --root . +mkt cache query 'sections[heading=Decision]' --root . +mkt search latency --root . --limit 10 +mkt backend refresh-plan docs --root . +``` + +## Policy-Aware Access + +```bash +mkt policy check public-agent read object-id --policy policy.yaml +mkt policy subject claims.yaml --policy-map policy-map.yaml +mkt policy resource-manifest manifest.yaml +mkt search internal --root . --policy policy.yaml --subject internal-agent +``` + +## Workflows And Context Packages + +```bash +mkt workflow inspect workflow.md +mkt workflow plan workflow.md --output-dir out +mkt workflow run workflow.md --output-dir out +mkt context pack query --source file.md --max-items 5 +mkt context activate package-id +mkt context explain package-id +mkt context refresh package-id +mkt context list +``` + +## Discovery And Docs + +```bash +mkt completion bash --instructions +mkt extension list +mkt extension inspect backend.local-sqlite +mkt extension commands +mkt docs cli --output docs/cli-reference.md +mkt docs api --output docs/api-reference.md +``` diff --git a/docs/examples-index.md b/docs/examples-index.md new file mode 100644 index 0000000..a435623 --- /dev/null +++ b/docs/examples-index.md @@ -0,0 +1,50 @@ +# Examples Index + +This index maps example files to practical usecases and useful commands. + +## Contracts And Diagnostics + +| Files | Usecase | Try | +| --- | --- | --- | +| `examples/contracts/adr.contract.md`, `examples/documents/adr-valid.md` | ADR contract validation | `mkt contract check examples/documents/adr-valid.md --contract examples/contracts/adr.contract.md` | +| `examples/contracts/business-letter.contract.md`, `examples/documents/business-letter-valid.md` | Form-like business document validation | `mkt contract form-state examples/documents/business-letter-valid.md --contract examples/contracts/business-letter.contract.md --context examples/runtime/business-letter.context.yaml` | +| `examples/contracts/prd-frs.contract.md`, `examples/documents/prd-frs-valid.md` | Larger product/specification contract | `mkt contract check examples/documents/prd-frs-valid.md --contract examples/contracts/prd-frs.contract.md` | +| `examples/diagnostics/*.expected-diagnostics.md` | Expected failure explanations | Compare with `mkt contract check ... --format text` | + +## Query, References, And Processors + +| Files | Usecase | Try | +| --- | --- | --- | +| `examples/references/context.md` | Namespace, region, fence, and section references | `mkt ref resolve examples/references/context.md 'std:clauses.md#payment-terms' --root examples/references` | +| `examples/references/standard/clauses.md` | Referenced reusable content | Use with `mkt ref resolve` or `mkt process` | +| `examples/migration/legacy-path-include.md` | Migration-style include handling | `mkt include examples/migration/legacy-path-include.md --base-dir examples/migration` | +| `examples/functions/basic-functions.md` | Deterministic document functions | `mkt function render examples/functions/basic-functions.md` | + +## Templates, Generation, And Workflows + +| Files | Usecase | Try | +| --- | --- | --- | +| `examples/templates/adr-summary.template.md`, `examples/templates/adr-summary.data.yaml` | Data-bound Markdown template | `mkt template render examples/templates/adr-summary.template.md --data examples/templates/adr-summary.data.yaml` | +| `examples/templates/adr-summary.generation.md` | Rule-based generation plan | `mkt generate rules examples/templates/adr-summary.generation.md --output-dir /tmp/markitect-generated` | +| `examples/workflows/source-snippets.workflow.md` | Extract snippets from Markdown sources | `mkt workflow plan examples/workflows/source-snippets.workflow.md --output-dir /tmp/markitect-workflow` | +| `examples/workflows/adr-release-notes.workflow.md` | ADR release notes workflow | `mkt workflow plan examples/workflows/adr-release-notes.workflow.md --output-dir /tmp/markitect-workflow` | +| `examples/workflows/assisted-review.workflow.md` | Assisted-generation boundary shape | Inspect with `mkt workflow inspect` | + +## Cache, Backend, Policy, And Context + +| Files | Usecase | Try | +| --- | --- | --- | +| `examples/backends/local-sqlite-backend.md` | Backend manifest inspection | `mkt backend inspect local-sqlite --path examples/backends/local-sqlite-backend.md` | +| `examples/backend-state/snapshot-state.yaml` | Refresh planning fixture | `mkt backend refresh-plan examples/documents --state examples/backend-state/snapshot-state.yaml --root .` | +| `examples/policy/local-label-policy.yaml` | Local policy gateway | `mkt policy check public-agent read note --policy examples/policy/local-label-policy.yaml --label public` | +| `examples/policy/enterprise-policy-map.yaml` | Enterprise IAM mapping fixture | `mkt policy subject examples/policy/netkingdom-claims.yaml --policy-map examples/policy/enterprise-policy-map.yaml` | +| `examples/memory/workplan-context.manifest.yaml` | Context package manifest | `mkt context pack examples/memory/workplan-context.manifest.yaml --root . --no-save` | + +## Literate And Migration + +| Files | Usecase | Try | +| --- | --- | --- | +| `examples/literate/app.md` | Weave/tangle style code/document workflow | `mkt tangle examples/literate/app.md --output-dir /tmp/markitect-tangle` | +| `examples/migration/legacy-explode-source.md` | Reversible section splitting | `mkt explode examples/migration/legacy-explode-source.md --output-dir /tmp/markitect-parts` | +| `examples/migration/legacy-literate.md` | Legacy literate migration sample | `mkt weave examples/migration/legacy-literate.md` | +| `examples/classes/prd-classes.yaml` | Content class composition | `mkt class resolve examples/classes/prd-classes.yaml product-prd` | diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..7bbfeac --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,130 @@ +# Getting Started + +This guide gets from a checkout to practical Markitect use without requiring a +service, database server, LLM provider, or project configuration. + +## Install For Local Use + +From the repository root: + +```bash +python3 -m pip install -e . +mkt --help +``` + +When working from a checkout without installing: + +```bash +PYTHONPATH=src python3 -m markitect_tool --help +``` + +## Add Shell Completion + +Generate completion for your shell: + +```bash +mkt completion bash --instructions +mkt completion zsh --instructions +mkt completion fish --instructions +``` + +For Bash, the direct form is: + +```bash +mkt completion bash > ~/.mkt-complete.bash +echo '. ~/.mkt-complete.bash' >> ~/.bashrc +``` + +## First Useful Commands + +Parse a Markdown file: + +```bash +mkt parse README.md --format tree +``` + +Check document size and complexity: + +```bash +mkt metrics examples/documents/adr-valid.md +``` + +Validate a document contract: + +```bash +mkt contract check examples/documents/adr-valid.md \ + --contract examples/contracts/adr.contract.md +``` + +Query a section: + +```bash +mkt query examples/documents/adr-valid.md 'sections[heading=Decision]' --format text +``` + +Build and search a local index: + +```bash +mkt cache index examples/documents --root . +mkt search contracts --root . --limit 5 +``` + +Generate from a template: + +```bash +mkt template render examples/templates/adr-summary.template.md \ + --data examples/templates/adr-summary.data.yaml +``` + +Run a declarative workflow in plan mode: + +```bash +mkt workflow plan examples/workflows/adr-release-notes.workflow.md \ + --output-dir /tmp/markitect-out +``` + +Create an inspectable context package: + +```bash +mkt context pack contracts --search --root . --max-items 5 --format yaml +``` + +## Discover The Surface + +List built-in extension descriptors: + +```bash +mkt extension list +mkt extension commands +mkt extension inspect memory.context-package +``` + +Regenerate reference docs from the live implementation: + +```bash +mkt docs cli --output docs/cli-reference.md +mkt docs api --output docs/api-reference.md +``` + +## Suggested First Workflow + +1. Start with `mkt parse` and `mkt metrics` on one document. +2. Use `mkt contract check` with one of the example contracts. +3. Use `mkt query` or `mkt extract` to grab a section. +4. Move to `mkt cache index` and `mkt search` once you have a directory. +5. Use `mkt workflow plan` before writing workflow outputs. +6. Use `mkt extension commands` when looking for the command attached to a + capability. + +## Diagnostics + +Most validation-oriented commands return structured JSON or YAML when requested: + +```bash +mkt contract check examples/documents/adr-invalid.md \ + --contract examples/contracts/adr.contract.md \ + --format json +``` + +Use machine-readable output in automation, and text output for quick local +inspection. diff --git a/docs/performance-notes.md b/docs/performance-notes.md new file mode 100644 index 0000000..1a65426 --- /dev/null +++ b/docs/performance-notes.md @@ -0,0 +1,60 @@ +# Performance Notes + +Markitect is designed to remain useful without persistent services. The current +performance posture is therefore local-first: + +- parse individual Markdown files directly for one-off use +- build a local SQLite index for repeated corpus operations +- use refresh planning to avoid unnecessary parse/index work +- keep policy filtering and context package creation deterministic + +## Current Smoke Coverage + +`tests/test_practical_usecases_e2e.py` includes a large-corpus smoke test that +creates 120 synthetic Markdown files, indexes them locally, and runs an FTS +search. + +The thresholds are intentionally generous: + +- local cache/index build: under 30 seconds +- local indexed search: under 5 seconds + +These are not benchmark claims. They are regression guards to catch accidental +algorithmic or IO mistakes while keeping the test portable. + +## Practical Guidance + +For one file: + +```bash +mkt parse file.md +mkt query file.md 'sections[heading=Decision]' +``` + +For a directory you will query repeatedly: + +```bash +mkt cache index docs --root . +mkt search keyword --root . +mkt cache query 'sections[heading=Decision]' --root . +``` + +Before refreshing derived work: + +```bash +mkt backend refresh-plan docs --root . --verify-hashes +``` + +## Future Measurement + +If performance becomes a release gate, add a separate benchmark suite instead +of tightening normal E2E tests. Good benchmark dimensions would be: + +- number of documents +- total bytes +- heading/section density +- frontmatter size +- number of reference/include relationships +- policy labels per document +- index rebuild versus incremental refresh +- context package item and token budgets diff --git a/docs/workplan-planning-map.md b/docs/workplan-planning-map.md index d7c31db..01da16c 100644 --- a/docs/workplan-planning-map.md +++ b/docs/workplan-planning-map.md @@ -41,7 +41,7 @@ and descriptions mirror the operational view. | `MKTT-WP-0014` | complete | done | `MKTT-WP-0009` | Markitect-side enterprise IAM access-control integration is complete: NetKingdom/key-cape-compatible identity claims, flex-auth resource/policy contract, directory group resolution fixtures, decision-log sink, workflow declarations, CLI commands, and external PDP request examples. | | `MKTT-WP-0012` | complete | done | `MKTT-WP-0004`, `MKTT-WP-0010`, `MKTT-WP-0011` | Document function layer is complete: deterministic Markdown-native function descriptors, registry, inline/fenced syntax, pipelines, context bindings, CLI, docs, examples, diagnostics, provenance, and extension descriptor. | | `MKTT-WP-0008` | complete | done | `MKTT-WP-0006`, `MKTT-WP-0007`, `MKTT-WP-0009` | Agent working-memory context cache is complete: context package schema, local registry, package creation from queries/search/manifests, deterministic summaries, namespaces, activation/deactivation/refresh/explain lifecycle, policy re-checks, CLI, docs, and examples. | -| `MKTT-WP-0017` | P1 | active | `MKTT-WP-0003`, `MKTT-WP-0013` | Current polish/adoption track: CLI/API completeness, shell completion, generated command/API docs, usecase relevance matrix, E2E fixture planning, accessibility, and test/performance iteration before more advanced feature work. | +| `MKTT-WP-0017` | complete | done | `MKTT-WP-0003`, `MKTT-WP-0013` | CLI/API polish and practical adoption track is complete: shell completion, extension discovery, generated CLI/API docs, usecase relevance matrix, E2E fixture matrix, large-corpus smoke coverage, first-use docs, examples index, and command cheat sheet. | | `MKTT-WP-0015` | P2 | todo | `MKTT-WP-0010`, `MKTT-WP-0011`, `MKTT-WP-0012` | Future render and document-function extensions: typed values, richer syntax, document-local reusable functions, Quarkdown/export adapters, render-aware references, assets, and permission sandboxing. Defer unless publishing/export pressure becomes current. | | `MKTT-WP-0016` | P2 | todo | `MKTT-WP-0008`, `MKTT-WP-0007`, `MKTT-WP-0009`, `MKTT-WP-0013` | Follow-on agentic memory architecture: reasoning decision graphs, conversational paths, long-term knowledge graphs, memory service blueprints/profiles, graph-to-context-package compilation, and adapter boundaries. | diff --git a/tests/test_practical_usecases_e2e.py b/tests/test_practical_usecases_e2e.py index 0245211..3476fcf 100644 --- a/tests/test_practical_usecases_e2e.py +++ b/tests/test_practical_usecases_e2e.py @@ -1,4 +1,5 @@ import json +import time from pathlib import Path import pytest @@ -203,3 +204,202 @@ def test_structure_reuse_policy_and_literate_usecases(tmp_path: Path): implode = runner.invoke(main, ["implode", str(exploded), "--format", "markdown"]) assert implode.exit_code == 0, implode.output assert "# Use Markdown Contracts" in implode.output + + +@pytest.mark.parametrize( + ("label", "document", "contract", "context"), + [ + ("small", "examples/documents/concept-note-valid.md", "examples/contracts/concept-note.contract.md", None), + ( + "typical", + "examples/documents/business-letter-valid.md", + "examples/contracts/business-letter.contract.md", + "examples/runtime/business-letter.context.yaml", + ), + ("large", "examples/documents/prd-frs-valid.md", "examples/contracts/prd-frs.contract.md", None), + ], +) +def test_contract_and_runtime_usecases_small_typical_large( + label: str, + document: str, + contract: str, + context: str | None, +): + runner = CliRunner() + command = ["contract", "check", document, "--contract", contract, "--format", "json"] + if context: + command.extend(["--context", context]) + + check = runner.invoke(main, command) + + assert check.exit_code == 0, check.output + check_data = json.loads(check.output) + assert check_data["valid"] is True + assert check_data["document_path"].endswith(Path(document).name) + + if context: + form_state = runner.invoke( + main, + [ + "contract", + "form-state", + document, + "--contract", + contract, + "--context", + context, + "--format", + "json", + ], + ) + assert form_state.exit_code == 0, form_state.output + form_data = json.loads(form_state.output) + assert form_data["valid"] is True + assert form_data["fields"], label + + +@pytest.mark.parametrize( + ("label", "count"), + [ + ("small", 1), + ("typical", 5), + ("large", 30), + ], +) +def test_transform_compose_and_include_usecases_small_typical_large( + tmp_path: Path, + label: str, + count: int, +): + docs = tmp_path / "sources" + docs.mkdir() + for index in range(count): + (docs / f"source-{index:02d}.md").write_text( + f"""--- +review: + state: draft +--- + +# Source {index} + +## Decision + +The {label} transform pipeline keeps source {index} reusable. +""", + encoding="utf-8", + ) + + runner = CliRunner() + transform = runner.invoke( + main, + [ + "transform", + str(docs / "source-00.md"), + "--set", + "review.state=ready", + "--heading-delta", + "1", + ], + ) + assert transform.exit_code == 0, transform.output + assert "state: ready" in transform.output + assert "## Source 0" in transform.output + + compose = runner.invoke( + main, + [ + "compose", + *[str(path) for path in sorted(docs.glob("*.md"))], + "--title", + f"{label.title()} Corpus", + "--heading-delta", + "1", + "--format", + "markdown", + ], + ) + assert compose.exit_code == 0, compose.output + assert f"# {label.title()} Corpus" in compose.output + assert compose.output.count("### Decision") == count + + include_file = tmp_path / "include.md" + include_file.write_text( + '', + encoding="utf-8", + ) + include = runner.invoke(main, ["include", str(include_file), "--base-dir", str(tmp_path)]) + assert include.exit_code == 0, include.output + assert "### Decision" in include.output + assert f"The {label} transform pipeline" in include.output + + +@pytest.mark.parametrize( + "workflow_file", + [ + "examples/workflows/source-snippets.workflow.md", + "examples/workflows/adr-release-notes.workflow.md", + ], +) +def test_workflow_and_refresh_planning_usecases(tmp_path: Path, workflow_file: str): + runner = CliRunner() + + inspect = runner.invoke(main, ["workflow", "inspect", workflow_file, "--format", "json"]) + assert inspect.exit_code == 0, inspect.output + assert json.loads(inspect.output)["valid"] is True + + plan = runner.invoke( + main, + ["workflow", "plan", workflow_file, "--output-dir", str(tmp_path / "out"), "--format", "json"], + ) + assert plan.exit_code == 0, plan.output + plan_data = json.loads(plan.output) + assert plan_data["valid"] is True + assert plan_data["dry_run"] is True + + refresh = runner.invoke( + main, + ["backend", "refresh-plan", "examples/documents", "--root", ".", "--format", "json"], + ) + assert refresh.exit_code == 1, refresh.output + refresh_data = json.loads(refresh.output) + assert refresh_data["dirty"] is True + assert refresh_data["counts"]["needs_parse"] >= 1 + + +def test_large_corpus_performance_smoke_stays_bounded(tmp_path: Path): + docs = tmp_path / "docs" + docs.mkdir() + for index in range(120): + (docs / f"note-{index:03d}.md").write_text( + f"""# Note {index} + +## Summary + +This synthetic performance smoke document mentions adoption latency and search. + +## Details + +The index should stay responsive for local corpora without requiring services. +""", + encoding="utf-8", + ) + + runner = CliRunner() + started = time.perf_counter() + index = runner.invoke(main, ["cache", "index", str(docs), "--root", str(tmp_path), "--format", "json"]) + index_seconds = time.perf_counter() - started + + assert index.exit_code == 0, index.output + assert len(json.loads(index.output)["indexed"]) == 120 + assert index_seconds < 30 + + started = time.perf_counter() + search = runner.invoke( + main, + ["search", "adoption", "--root", str(tmp_path), "--limit", "10", "--format", "json"], + ) + search_seconds = time.perf_counter() - started + + assert search.exit_code == 0, search.output + assert json.loads(search.output)["count"] >= 1 + assert search_seconds < 5 diff --git a/workplans/MKTT-WP-0017-cli-api-polish-and-practical-adoption.md b/workplans/MKTT-WP-0017-cli-api-polish-and-practical-adoption.md index 5fcbf81..7f09b03 100644 --- a/workplans/MKTT-WP-0017-cli-api-polish-and-practical-adoption.md +++ b/workplans/MKTT-WP-0017-cli-api-polish-and-practical-adoption.md @@ -3,7 +3,7 @@ id: MKTT-WP-0017 type: workplan title: "CLI/API Polish And Practical Adoption" domain: markitect -status: active +status: done owner: markitect-tool topic_slug: markitect planning_priority: P1 @@ -123,7 +123,7 @@ Output: `docs/practical-usecase-relevance.md` and links from intent/README. ```task id: MKTT-WP-0017-T005 -status: in_progress +status: done priority: high state_hub_task_id: "1159ec82-00e6-45cd-8a2c-e61ae03e758a" ``` @@ -149,22 +149,22 @@ Required coverage: Output: deterministic tests and fixtures with no network dependency. -Initial coverage has been added for: +Coverage has been added for: - small, typical, and large synthetic corpus parse/query/cache/search/context package flow - example-backed contract, template, function, workflow, reference, processor, policy, literate tangle, and explode/implode flows - -Remaining work: expand the matrix into explicit small/typical/large variants -for every high-relevance workflow and add performance assertions where they are -stable. +- small, typical, and large contract/form-state validation flows +- small, typical, and large transform/compose/include flows +- workflow inspect/plan and backend refresh-planning flows +- large local cache/index/search performance smoke ## P17.6 - Run coverage and performance iteration ```task id: MKTT-WP-0017-T006 -status: todo +status: done priority: high state_hub_task_id: "8dd5afe5-cefa-4388-bf4c-a975ba5c77c3" ``` @@ -175,11 +175,15 @@ or performance issues exposed by the E2E matrix. Output: passing test suite, documented coverage gaps, and targeted performance notes for large synthetic corpora. +Implementation adds `docs/performance-notes.md` and a portable smoke test with +generous bounds for local index build and indexed search. Full-suite validation +is run before closing the workplan. + ## P17.7 - Improve first-use accessibility ```task id: MKTT-WP-0017-T007 -status: todo +status: done priority: medium state_hub_task_id: "590af805-0cc0-4585-b85f-9e95679afa77" ``` @@ -198,6 +202,14 @@ Possible improvements: Output: README/docs/examples polish and tests where executable. +Implementation adds: + +- `docs/getting-started.md` +- `docs/command-cheatsheet.md` +- `docs/examples-index.md` +- `docs/performance-notes.md` +- README and intent links to the practical docs + ## Exit Criteria - Every implemented subsystem has CLI access, public API access, docs, examples,