Extensible canonical internal processing refactoring

docs/internal-extension-framework.md

@@ -0,0 +1,149 @@
+# Internal Extension Framework
+
+## Purpose
+
+Markitect has reached the point where optional features are useful but are
+starting to concentrate wiring in central modules. Query engines, processors,
+backend stores, references, contract checks, templates, generation adapters, and
+CLI commands all need some combination of registration, capability metadata,
+diagnostics, provenance, and optional dependency handling.
+
+The internal extension framework should make those seams explicit without
+turning the project into a heavy external plugin platform.
+
+## Boundary
+
+This framework is about internal extensibility:
+
+```text
+feature descriptor -> registry -> processing request/context/result
+                   -> diagnostics/provenance/capabilities
+                   -> CLI/API/backend integration
+```
+
+It is not the same as `MKTT-WP-0011` dataflow workflows. Workflows organize
+business-facing processing steps for a document pipeline. The extension
+framework organizes how Markitect itself exposes and composes capabilities.
+
+## Extension Taxonomy
+
+| Kind | Examples | Primary Contract |
+| --- | --- | --- |
+| `query-engine` | selector, JSONPath | document/data in, matches out |
+| `processor` | identity, uppercase, include | fenced block in, processed result out |
+| `backend` | local SQLite index | snapshots/index/search storage |
+| `reference-provider` | section, region, fence, line | address in, content units out |
+| `validator` | schema, contract, section assertion | document/context in, diagnostics out |
+| `template-engine` | deterministic templates | template/data in, Markdown out |
+| `generation-adapter` | provider-neutral assisted generation | request in, generated candidate out |
+| `cli-group` | cache, backend, ref, class | command descriptors or registration hook |
+| `render-export` | future Quarkdown/export adapters | Markdown source in, rendered/exported artifact out |
+| `document-function` | future function layer | function call in, typed document value out |
+
+## Canonical Lifecycle
+
+An extension should be describable before it runs:
+
+1. Register descriptor.
+2. Check optional dependencies.
+3. Check capabilities and policy labels.
+4. Build processing context.
+5. Execute operation.
+6. Normalize result, diagnostics, provenance, and trace data.
+7. Expose output through library API, CLI, backend, or workflow layer.
+
+The framework should allow deterministic extensions to stay simple. Assisted,
+external, networked, or filesystem-mutating extensions should declare that
+explicitly before execution.
+
+## Descriptor Shape
+
+The first descriptor model should cover:
+
+- stable id
+- kind
+- version
+- summary
+- implementation reference
+- capability declarations
+- optional dependency declarations
+- safety flags
+- input and output contract names
+- diagnostics namespace
+- provenance operation prefix
+- documentation and example links
+- CLI affordances where applicable
+
+Descriptors are not meant to replace implementation modules. They are the small
+declarative surface that lets Markitect inspect, list, validate, and compose
+capabilities consistently.
+
+## Processing Model
+
+The canonical processing model should define a small set of shared envelopes:
+
+- `ProcessingRequest`: operation id, input payload, options, scope
+- `ProcessingContext`: root, source path, namespaces, variables, policy, backend
+  handles, and caller metadata
+- `ProcessingResult`: output payload, diagnostics, provenance, dependencies,
+  trace events, and validity
+- `ProcessingDiagnostic`: severity, code, message, source, help
+- `ProcessingCapability`: declared feature or permission requirement
+- `ProcessingProvenance`: operation, source identity, snapshot/content hashes,
+  dependencies, backend/provider metadata
+
+Subsystem-specific types may remain richer. The canonical model is the bridge,
+not a forced replacement for every local dataclass.
+
+## Registration Strategy
+
+Start with in-package registration:
+
+```text
+markitect_tool/extensions/
+  query_selector.py
+  query_jsonpath.py
+  backend_local_sqlite.py
+  processors_builtin.py
+```
+
+Each module exposes one or more descriptors plus a registration function. The
+root registry can be assembled explicitly at import time or by a small internal
+discovery list. Package entry points can be added later if external extension
+packages become a real requirement.
+
+See `docs/extension-authoring.md` for the extension authoring checklist and
+descriptor template.
+
+## Compatibility Rules
+
+The refactor must preserve:
+
+- current library APIs such as `query_document`
+- current CLI commands and output envelopes
+- current diagnostic codes where users may rely on them
+- current provenance operation strings unless intentionally deprecated
+- optional dependency behavior for JSONPath and future adapters
+- cache/index file compatibility unless a migration is documented
+
+The first implementation adds canonical processing envelopes, extension
+descriptors, registries, lifecycle callbacks, query-engine registry shims,
+built-in extension descriptors, and CLI command specs while preserving existing
+public commands.
+
+## Characterization Coverage
+
+Before refactoring, lock down:
+
+- selector query and extraction
+- optional JSONPath diagnostics
+- processor registry behavior and provenance
+- backend manifest registry and capability checks
+- local SQLite snapshot/index/search behavior
+- content reference resolution
+- representative CLI command envelopes
+- provenance and diagnostic shapes
+
+These tests are deliberately a little redundant with unit tests. Their job is
+to protect the current public behavior while internals move behind extension
+descriptors and registries.