coulomb/markitect-tool
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

context loading, path resolution, form state, dynamic rules, and provider-neutral assessment runner/cache boundary

Browse Source
This commit is contained in:
Bernd Worsch
2026-05-04 13:52:29 +02:00
parent eccf1874fb
commit 8361f9ea45
29 changed files with 2809 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
docs/contract-framework.md
View File

@@ -9,9 +9,9 @@ Markdown as the authoring surface and uses fenced YAML as a structured extension
for rules that need machine interpretation.
The first implementation is deterministic. It checks document type, fields,
sections, ordering, metric bands, and text assertions. Forms, context, and LLM
rubrics are represented in the contract vocabulary as extension points before
runtime adapters are added.
sections, ordering, metric bands, and text assertions. Runtime context, forms,
dynamic rules, and provider-neutral assessment requests are implemented as
extensions around the same contract vocabulary.
## Contract File Shape
@@ -112,10 +112,10 @@ fields, and metric bands. This is the bridge to later LLM rubrics: semantic
checks can become additional assessments without changing how failures are
reported.
## Forms And Context
## Forms, Context, And Runtime Rules
Field specs are the first step toward form-backed Markdown generation. Runtime
form handling should build on the same field vocabulary:
Field specs are the foundation for form-backed Markdown generation and
context-aware checks. Runtime form handling uses the same field vocabulary:
- `id`
- `type`
@@ -128,15 +128,25 @@ form handling should build on the same field vocabulary:
- `min` / `max`
- `min_length` / `max_length`
Dynamic requiredness, visibility, calculations, and prefill should be declared
as context-aware rules in later work. The contract should remain the source of
truth, while UI and generation layers act as adapters.
Runtime context can be supplied as local YAML or JSON:
```text
mkt contract check <document.md> --contract <contract.md> --context <context.yaml>
mkt contract form-state <document.md> --contract <contract.md> --context <context.yaml>
```
The runtime resolves fields in this order: document value, context source,
default, missing. Document values win over context and conflicts are diagnostics.
Dynamic rules support small deterministic `if` / `then` / `else` expressions
for requiredness, visibility, allowed values, calculated values, context
assertions, and dynamic section presence. See
`docs/runtime-context-forms-assessments.md`.
## LLM Assessment Extension
LLM-assisted checks should be declared as rubrics, scoped to document or section
roles. Core Markitect should not call a provider directly. A future adapter
should accept a provider-neutral request:
LLM-assisted checks are declared as rubrics, scoped to document or section roles.
Core Markitect does not call a provider directly. It creates provider-neutral
assessment requests for injected adapters:
- contract id and rule id
- document or section text
@@ -152,26 +162,11 @@ It should return:
- model/provider metadata
- diagnostics using the shared diagnostic model
## Deferred Runtime Work
The deterministic contract framework is ready now. The runtime engines are
deferred to `MKTT-WP-0005-runtime-context-and-assessment-engines.md`.
Pick that work up when one of these becomes true:
- contract checks need external user, project, or entity context
- generation needs reliable field prefill before rendering
- a UI or agent workflow needs form state, defaults, and dynamic requiredness
- deterministic section assertions are not enough and rubric-based semantic
assessment becomes necessary
The intended order is context and form runtime first, deterministic dynamic
rules second, LLM assessment execution third.
## CLI
```text
mkt contract validate <contract.md>
mkt contract check <document.md> --contract <contract.md>
mkt contract check <document.md> --contract <contract.md> [--context <context.yaml>]
mkt contract form-state <document.md> --contract <contract.md> [--context <context.yaml>]
mkt metrics <document.md>
```

2
docs/internal-extension-framework.md
View File

@@ -34,6 +34,8 @@ framework organizes how Markitect itself exposes and composes capabilities.
| `backend` | local SQLite index | snapshots/index/search storage |
| `reference-provider` | section, region, fence, line | address in, content units out |
| `validator` | schema, contract, section assertion | document/context in, diagnostics out |
| `runtime` | context loader, form state, dynamic rules | document/contract/context in, diagnostics and state out |
| `assessment-runner` | provider-neutral rubric execution | assessment request in, normalized result out |
| `template-engine` | deterministic templates | template/data in, Markdown out |
| `generation-adapter` | provider-neutral assisted generation | request in, generated candidate out |
| `cli-group` | cache, backend, ref, class | command descriptors or registration hook |

3
docs/markdown-workflows.md
View File

@@ -72,7 +72,8 @@ This makes workflows useful without provider dependencies.
| `transform` | transformed `markdown`, operations, provenance. |
| `include` | include-resolved `markdown`, included paths, provenance. |
| `contract_stub` | generated contract stub Markdown. |
| `contract_check` | contract diagnostics and metrics. |
| `contract_check` | contract diagnostics, metrics, and optional runtime context results. |
| `form_state` | field values, origins, dynamic rule results, and diagnostics. |
| `assisted` | generated Markdown if a hook is supplied, otherwise skipped/diagnostic. |
## Data Bindings

220
docs/runtime-context-forms-assessments.md Normal file
View File

@@ -0,0 +1,220 @@
# Runtime Context, Forms, Rules, And Assessments
Date: 2026-05-04
## Purpose
The runtime layer turns contract extension points into executable behavior while
keeping the deterministic contract framework intact. Static checks still handle
document type, sections, assertions, and metric bands. Runtime checks add
external context, field prefill, UI-neutral form state, dynamic rules, and a
provider-neutral assessment protocol.
The layer is deliberately local-first. Core Markitect reads YAML or JSON context
files and runs deterministic rules. Network calls, application lookups, and LLM
providers belong behind adapters.
## Context Files
Runtime context can be a raw YAML/JSON mapping:
```yaml
recipient:
name: Ada Lovelace
sender:
name: Markitect Team
```
or an envelope with metadata and schema:
```yaml
metadata:
case_id: case-42
schema:
type: object
required: [recipient, sender]
context:
recipient:
name: Ada Lovelace
sender:
name: Markitect Team
```
The value under `context` is bound as `context` in field sources and dynamic
rules. `schema` validates the full context object. `schemas` can validate named
objects individually.
Malformed context and schema failures produce normal diagnostics:
- `runtime.context.malformed`
- `runtime.context.schema_invalid`
- `runtime.context.schema_target_missing`
- `runtime.context.schema`
## Field Runtime
Field specs continue to live in the contract:
```yaml
fields:
recipient_name:
type: string
required: true
source: context.recipient.name
delivery_channel:
type: string
default: email
enum: [email, print]
```
Runtime resolution order is:
1. Manual document value from `path`, usually frontmatter.
2. Context value from `source` or `sources`.
3. Contract `default`.
4. Missing.
Manual document values win over context. If both exist and differ, Markitect
emits `runtime.field.conflict` as a warning by default. A field can set
`conflict: error` to make that stricter. Multiple context sources with distinct
values produce `runtime.field.ambiguous`.
`mkt contract check` uses runtime evaluation only when `--context` is supplied:
```text
mkt contract check document.md --contract contract.md --context context.yaml
```
`mkt contract form-state` always emits the UI-neutral runtime view:
```text
mkt contract form-state document.md --contract contract.md --context context.yaml
```
## Form State
Form state is not a UI framework. It is a stable contract that future UIs,
agents, generators, and workflow steps can render:
- field id
- value
- origin: `manual`, `prefilled`, `defaulted`, `calculated`, or `missing`
- required/optional
- visible/hidden
- enabled/disabled
- allowed values
- diagnostics
- metadata
Hidden fields are not required unless a future adapter explicitly asks for
hidden validation. This matches practical form behavior and avoids punishing
authors for data that the current context made irrelevant.
## Dynamic Rules
Rules are deterministic YAML. They use a deliberately small condition language:
```yaml
rules:
- id: postal-address-for-print
if:
path: fields.delivery_channel.value
equals: print
then:
required: [postal_address]
visible:
postal_address: true
else:
hidden: [postal_address]
```
Supported condition operators:
- `exists`
- `equals` / `eq`
- `not_equals`
- `in`
- `contains`
- `matches`
- `gt`, `gte`, `lt`, `lte`
- `all`, `any`, `not`
Supported actions:
- `required` / `optional`
- `visible` / `hidden`
- `enabled` / `disabled`
- `allowed_values`
- `set`
- `assert`
- `sections`
Calculated values can reference runtime paths:
```yaml
then:
set:
contact_label: "${fields.sender_name.value} <${context.sender.email}>"
```
Context assertions use the same condition vocabulary:
```yaml
assert:
path: context.sender.email
matches: "@example\\.com$"
message: Sender email must come from example.com.
severity: warning
```
Dynamic section rules are intentionally narrow. They can require, recommend,
discourage, or forbid section specs already declared in the contract.
## Assessment Protocol
Rubrics remain provider-neutral contract declarations:
```yaml
rubrics:
- id: tone-fit
scope: section.body
criteria: The body should match the recipient relationship.
threshold: 0.75
```
Core Markitect turns rubrics into `AssessmentRequest` objects and normalizes
adapter results into `AssessmentResult` and diagnostics. It does not call an LLM
provider directly. The cache key includes contract id, rule id, scope, text,
criteria, context, structured inputs, threshold, provider, model, and metadata.
Adapters can be injected from workflows, applications, or tests. A transparent
in-memory cache exists for tests and short runs; persistent storage remains a
backend concern.
## Workflow Integration
Workflow `contract_check` steps accept `context`:
```yaml
steps:
- id: check-letter
kind: contract_check
document: letter.md
contract: letter.contract.md
context: letter.context.yaml
```
Workflow `form_state` steps expose the runtime state as a step result:
```yaml
steps:
- id: form
kind: form_state
document: letter.md
contract: letter.contract.md
context: letter.context.yaml
```
This keeps workflow orchestration separate from the runtime engine. The runtime
engine answers "what does this contract imply in this context"; the workflow
engine decides when to run it and where to send the output.

3
docs/workflow-definition-standard.md
View File

@@ -188,7 +188,8 @@ First implementation step kinds:
| `transform` | Apply deterministic Markdown transforms. |
| `include` | Resolve include markers in Markdown. |
| `contract_stub` | Generate a Markdown stub from a contract. |
| `contract_check` | Check a Markdown document against a contract. |
| `contract_check` | Check a Markdown document against a contract, optionally with runtime context. |
| `form_state` | Evaluate UI-neutral field prefill, validation, and dynamic rules. |
| `assisted` | Provider-neutral assisted step boundary, optional by default. |
## Data Bindings

2
docs/workplan-planning-map.md
View File

@@ -35,7 +35,7 @@ and descriptions mirror the operational view.
| `MKTT-WP-0010` | complete | done | `MKTT-WP-0004`; task-level trigger: `MKTT-WP-0003-T006` | Content references, processors, explode/implode, weave/tangle, content classes, and migration examples are complete as the first WP-0010 extension layer. |
| `MKTT-WP-0007` | complete | done | `MKTT-WP-0006` | Advanced query and local index backend is complete: AST inspection, optional JSONPath, SQLite snapshots/metadata, FTS5 search, incremental refresh, and local index CLI. |
| `MKTT-WP-0013` | complete | done | `MKTT-WP-0003`, `MKTT-WP-0004`, `MKTT-WP-0006`, `MKTT-WP-0007`, `MKTT-WP-0010` | Internal extension framework is complete: characterization tests, canonical processing model, descriptors, registries, lifecycle callbacks, query-engine registry, built-in extension catalog, CLI command specs, and authoring guide. |
| `MKTT-WP-0005` | P2 | todo | `MKTT-WP-0003`, `MKTT-WP-0004` | Pick up when generation/form/context or semantic assessment pressure appears. |
| `MKTT-WP-0005` | complete | done | `MKTT-WP-0003`, `MKTT-WP-0004` | Runtime context, form state, dynamic rules, workflow integration, and provider-neutral assessment boundary are complete. |
| `MKTT-WP-0011` | complete | done | `MKTT-WP-0003`; task-level triggers: `MKTT-WP-0010-T001`, `MKTT-WP-0010-T005` | Markdown dataflow workflow layer is complete: workflow standard, source collectors, binding model, deterministic steps, assisted boundary, safe outputs, CLI, docs, and examples. |
| `MKTT-WP-0009` | P2 | todo | `MKTT-WP-0006` | Establish access-control gateway before security-sensitive cache/context use. |
| `MKTT-WP-0012` | P3 | todo | `MKTT-WP-0004`, `MKTT-WP-0010`, `MKTT-WP-0011` | Future Quarkdown-inspired document function layer: reusable Markdown-native function calls over processors, references, contracts, workflows, and later assisted steps. |

23
examples/runtime/business-letter-prefill.md Normal file
View File

@@ -0,0 +1,23 @@
---
document_type: business-letter
---
# Follow-Up Letter
## Greeting
Dear Ada Lovelace,
## Body
Thank you for the thoughtful discussion about structured Markdown generation.
We reviewed the requirements and will send a concise proposal that separates
document contracts, field prefill, validation diagnostics, and optional semantic
assessment. This keeps the implementation practical while leaving room for
future automation.
## Closing
Kind regards,
Markitect Team

19
examples/runtime/business-letter.context.yaml Normal file
View File

@@ -0,0 +1,19 @@
metadata:
case_id: letter-ada-001
schema:
type: object
required: [recipient, sender]
properties:
recipient:
type: object
required: [name]
sender:
type: object
required: [name, email]
context:
recipient:
name: Ada Lovelace
relationship: research collaborator
sender:
name: Markitect Team
email: hello@example.com

27
examples/runtime/concept-note-assessment.contract.md Normal file
View File

@@ -0,0 +1,27 @@
# Concept Note Assessment Contract
```yaml contract
id: concept-note-assessment-v1
document:
type: concept-note
fields:
status:
type: string
required: true
enum: [draft, review, accepted]
sections:
- id: concept
title: Concept
presence: required
level: 2
- id: utility
title: Utility
presence: required
level: 2
rubrics:
- id: utility-is-practical
scope: section.utility
criteria: The utility section should explain who benefits, what changes in practice, and how success can be recognized.
threshold: 0.75
severity: warning
```

13
examples/runtime/prd-frs.context.yaml Normal file
View File

@@ -0,0 +1,13 @@
metadata:
release: runtime-context-alpha
context:
product:
name: Markitect Tool
owner: Platform Architecture
regulatory_tier: internal
project:
target_date: "2026-06-30"
stakeholders:
- Documentation
- Architecture
- Product Operations

25
examples/runtime/runtime-expected-diagnostics.md Normal file
View File

@@ -0,0 +1,25 @@
# Runtime Expected Diagnostics
These examples are meant for manual inspection and future snapshot fixtures.
```text
mkt contract check examples/runtime/workplan-done-missing-decision.md \
--contract examples/runtime/workplan-dynamic.contract.md \
--format text
```
Expected diagnostic:
- `runtime.section.missing`: status `done` requires a `Decision Point` section.
```text
mkt contract check examples/runtime/business-letter-prefill.md \
--contract examples/contracts/business-letter.contract.md \
--context examples/runtime/business-letter.context.yaml \
--format text
```
Expected runtime utility:
- `recipient_name` and `sender_name` are prefilled from context.
- No field-missing diagnostic is emitted for those fields.

18
examples/runtime/workplan-done-missing-decision.md Normal file
View File

@@ -0,0 +1,18 @@
---
document_type: workplan
id: MKTT-WP-EXAMPLE
status: done
---
# Example Workplan
## Purpose
Show how a dynamic section rule reports missing completion context.
## Tasks
```task
id: MKTT-WP-EXAMPLE-T001
status: done
```

46
examples/runtime/workplan-dynamic.contract.md Normal file
View File

@@ -0,0 +1,46 @@
# Dynamic Workplan Contract
```yaml contract
id: dynamic-workplan-contract-v1
document:
type: workplan
fields:
id:
type: string
required: true
status:
type: string
required: true
enum: [proposed, active, done, deferred]
owner:
type: string
source: context.workplan.owner
sections:
- id: purpose
title: Purpose
presence: required
level: 2
- id: tasks
title: Tasks
presence: required
level: 2
- id: decision-point
title: Decision Point
presence: optional
level: 2
rules:
- id: require-decision-point-when-done
if:
path: fields.status.value
equals: done
then:
sections:
decision-point:
presence: required
- id: owner-needed-for-active-work
if:
path: fields.status.value
equals: active
then:
required: [owner]
```

32
src/markitect_tool/__init__.py
View File

@@ -140,6 +140,23 @@ from markitect_tool.reference import (
parse_reference,
resolve_reference,
)
from markitect_tool.runtime import (
AssessmentRequest,
AssessmentResult,
AssessmentRunResult,
AssessmentRunner,
FieldState,
FormState,
MemoryAssessmentCache,
RuntimeContext,
RuntimeContextLoadResult,
RuntimeContextSource,
assessment_requests_for_contract,
evaluate_form_state,
load_runtime_context_file,
load_runtime_context_file_result,
run_contract_assessments,
)
from markitect_tool.schema import (
MarkdownSchema,
SchemaValidationResult,
@@ -290,6 +307,21 @@ __all__ = [
"load_namespaces",
"parse_reference",
"resolve_reference",
"AssessmentRequest",
"AssessmentResult",
"AssessmentRunResult",
"AssessmentRunner",
"FieldState",
"FormState",
"MemoryAssessmentCache",
"RuntimeContext",
"RuntimeContextLoadResult",
"RuntimeContextSource",
"assessment_requests_for_contract",
"evaluate_form_state",
"load_runtime_context_file",
"load_runtime_context_file_result",
"run_contract_assessments",
"MissingTemplateVariable",
"TemplateAnalysis",
"TemplateError",

81
src/markitect_tool/cli/main.py
View File

@@ -65,6 +65,7 @@ from markitect_tool.reference import (
load_namespaces,
resolve_reference,
)
from markitect_tool.runtime import evaluate_form_state, load_runtime_context_file
from markitect_tool.schema import load_schema_file, validate_markdown_file, validate_schema
from markitect_tool.template import (
MissingTemplateVariable,
@@ -1466,17 +1467,68 @@ def contract_validate(contract_file: Path, output_format: str) -> None:
default="text",
show_default=True,
)
def contract_check(file: Path, contract_file: Path, output_format: str) -> None:
@click.option(
"--context",
"context_file",
type=click.Path(exists=True, dir_okay=False, path_type=Path),
help="YAML or JSON runtime context used for field prefill and dynamic rules.",
)
def contract_check(
file: Path,
contract_file: Path,
output_format: str,
context_file: Path | None,
) -> None:
"""Check a Markdown file against a Markdown document contract."""
try:
result = check_markdown_file(file, contract_file)
result = check_markdown_file(file, contract_file, context_path=context_file)
except ContractLoaderError as exc:
raise click.ClickException(str(exc)) from exc
_emit_diagnostic_result(result.to_dict(), output_format)
raise click.exceptions.Exit(0 if result.valid else 1)
@contract.command("form-state")
@click.argument("file", type=click.Path(exists=True, dir_okay=False, path_type=Path))
@click.option(
"--contract",
"contract_file",
required=True,
type=click.Path(exists=True, dir_okay=False, path_type=Path),
)
@click.option(
"--context",
"context_file",
type=click.Path(exists=True, dir_okay=False, path_type=Path),
help="YAML or JSON runtime context used for field prefill and dynamic rules.",
)
@click.option(
"--format",
"output_format",
type=click.Choice(["json", "yaml", "text"], case_sensitive=False),
default="text",
show_default=True,
)
def contract_form_state(
file: Path,
contract_file: Path,
context_file: Path | None,
output_format: str,
) -> None:
"""Evaluate UI-neutral form state for a document contract."""
try:
document = parse_markdown_file(file)
contract_definition = load_contract_file(contract_file)
context = load_runtime_context_file(context_file) if context_file else None
form_state = evaluate_form_state(document, contract_definition, context)
except ContractLoaderError as exc:
raise click.ClickException(str(exc)) from exc
_emit_form_state(form_state.to_dict(), output_format)
raise click.exceptions.Exit(0 if form_state.valid else 1)
def _emit_result(data: dict, output_format: str) -> None:
if output_format == "json":
click.echo(json.dumps(data, indent=2, ensure_ascii=False))
@@ -1511,6 +1563,31 @@ def _emit_diagnostic_result(data: dict, output_format: str) -> None:
click.echo(f" guidance: {diagnostic['guidance']}")
def _emit_form_state(data: dict, output_format: str) -> None:
if output_format == "json":
click.echo(json.dumps(data, indent=2, ensure_ascii=False))
elif output_format == "yaml":
click.echo(yaml.safe_dump(data, sort_keys=False))
else:
click.echo("valid" if data.get("valid") else "invalid")
for field in data.get("fields", []):
value = field.get("value", "<missing>") if field.get("exists") else "<missing>"
flags = []
if field.get("required"):
flags.append("required")
if field.get("visible") is False:
flags.append("hidden")
if field.get("enabled") is False:
flags.append("disabled")
suffix = f" ({', '.join(flags)})" if flags else ""
click.echo(f"- {field['id']}: {value} [{field.get('origin', 'unknown')}]{suffix}")
for diagnostic in data.get("diagnostics", []):
click.echo(
f" [{diagnostic['severity']}] {diagnostic['code']}: "
f"{diagnostic['message']}"
)
def _emit_metrics(data: dict, output_format: str) -> None:
if output_format == "json":
click.echo(json.dumps(data, indent=2, ensure_ascii=False))

34
src/markitect_tool/contract/checker.py
View File

@@ -27,6 +27,7 @@ from markitect_tool.diagnostics import (
has_error,
valid_severity,
)
from markitect_tool.runtime import RuntimeContext, evaluate_form_state, load_runtime_context_file
@dataclass(frozen=True)
@@ -55,6 +56,7 @@ class ContractCheckResult:
document_path: str | None = None
contract_path: str | None = None
metrics: dict[str, Any] = field(default_factory=dict)
runtime: dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> dict[str, Any]:
data = {
@@ -63,6 +65,7 @@ class ContractCheckResult:
"document_path": self.document_path,
"contract_path": self.contract_path,
"metrics": self.metrics or None,
"runtime": self.runtime or None,
}
return {key: value for key, value in data.items() if value is not None}
@@ -131,26 +134,48 @@ def validate_contract(contract: DocumentContract) -> ContractValidationResult:
def check_markdown_file(
markdown_path: str | Path, contract_path: str | Path
markdown_path: str | Path,
contract_path: str | Path,
*,
context_path: str | Path | None = None,
runtime_context: RuntimeContext | None = None,
) -> ContractCheckResult:
"""Parse and check a Markdown file against a contract file."""
document = parse_markdown_file(markdown_path)
contract = load_contract_file(contract_path)
return check_document_contract(document, contract)
context = runtime_context
if context_path is not None:
context = load_runtime_context_file(context_path)
return check_document_contract(document, contract, runtime_context=context)
def check_document_contract(
document: Document, contract: DocumentContract
document: Document,
contract: DocumentContract,
*,
runtime_context: RuntimeContext | None = None,
) -> ContractCheckResult:
"""Check a parsed Markdown document against a document contract."""
contract_validation = validate_contract(contract)
document_metrics = collect_metrics(document)
diagnostics = list(contract_validation.diagnostics)
runtime: dict[str, Any] = {}
if contract_validation.valid:
diagnostics.extend(_check_document_type(document, contract))
diagnostics.extend(_check_fields(document, contract))
if runtime_context is None and not contract.rules:
diagnostics.extend(_check_fields(document, contract))
else:
form_state = evaluate_form_state(
document,
contract,
runtime_context or RuntimeContext.empty(),
)
diagnostics.extend(form_state.diagnostics)
context = runtime_context or RuntimeContext.empty()
runtime["context"] = context.to_dict()
runtime["form_state"] = form_state.to_dict()
diagnostics.extend(_check_document_metrics(document, contract, document_metrics))
diagnostics.extend(_check_assertions(document.body, contract.assertions, document, contract))
diagnostics.extend(_check_sections(document, contract, document_metrics))
@@ -161,6 +186,7 @@ def check_document_contract(
document_path=document.source_path,
contract_path=contract.source_path,
metrics=document_metrics.to_dict(),
runtime=runtime,
)

1
src/markitect_tool/contract/loader.py
View File

@@ -120,6 +120,7 @@ def _looks_like_contract(data: dict[str, Any]) -> bool:
"metric_bands",
"assertions",
"forms",
"rules",
"rubrics",
}
)

3
src/markitect_tool/contract/model.py
View File

@@ -223,6 +223,7 @@ class DocumentContract:
assertions: list[AssertionSpec] = field(default_factory=list)
forms: list[dict[str, Any]] = field(default_factory=list)
context: dict[str, Any] = field(default_factory=dict)
rules: list[dict[str, Any]] = field(default_factory=list)
rubrics: list[dict[str, Any]] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
raw: dict[str, Any] = field(default_factory=dict)
@@ -264,6 +265,7 @@ class DocumentContract:
assertions=assertions_from_value(raw.get("assertions")),
forms=raw.get("forms") if isinstance(raw.get("forms"), list) else [],
context=raw.get("context") if isinstance(raw.get("context"), dict) else {},
rules=raw.get("rules") if isinstance(raw.get("rules"), list) else [],
rubrics=raw.get("rubrics") if isinstance(raw.get("rubrics"), list) else [],
metadata=metadata,
raw=raw,
@@ -284,6 +286,7 @@ class DocumentContract:
"assertions": [assertion.raw for assertion in self.assertions],
"forms": self.forms,
"context": self.context,
"rules": self.rules,
"rubrics": self.rubrics,
"source_path": self.source_path,
}

68
src/markitect_tool/extension/builtins.py
View File

@@ -14,6 +14,9 @@ def builtin_extension_registry() -> ExtensionRegistry:
for descriptor in _processor_descriptors() + [
_local_sqlite_backend_descriptor(),
_workflow_engine_descriptor(),
_runtime_context_descriptor(),
_runtime_form_state_descriptor(),
_runtime_assessment_descriptor(),
]:
registry.register(descriptor)
return registry
@@ -120,3 +123,68 @@ def _workflow_engine_descriptor() -> ExtensionDescriptor:
docs=["docs/workflow-definition-standard.md"],
examples=["examples/workflows/adr-release-notes.workflow.md"],
)
def _runtime_context_descriptor() -> ExtensionDescriptor:
return ExtensionDescriptor(
id="runtime.context",
kind="runtime",
summary="YAML/JSON runtime context loader for document contracts.",
capabilities=[
ProcessingCapability(id="context", kind="read"),
ProcessingCapability(id="json-schema", kind="validate"),
ProcessingCapability(id="diagnostics", kind="emit"),
],
safety={"reads_files": True, "network": False},
input_contract="YAML/JSON runtime context file",
output_contract="RuntimeContext",
diagnostics_namespace="runtime.context",
provenance_prefix="runtime.context",
cli={"commands": ["mkt contract check --context", "mkt contract form-state"]},
docs=["docs/runtime-context-forms-assessments.md"],
examples=["examples/runtime/business-letter.context.yaml"],
)
def _runtime_form_state_descriptor() -> ExtensionDescriptor:
return ExtensionDescriptor(
id="runtime.form-state",
kind="runtime",
summary="UI-neutral field prefill, validation, and dynamic rule engine.",
capabilities=[
ProcessingCapability(id="forms", kind="evaluate"),
ProcessingCapability(id="rules", kind="evaluate"),
ProcessingCapability(id="contracts", kind="validate"),
ProcessingCapability(id="diagnostics", kind="emit"),
],
safety={"reads_files": False, "network": False},
input_contract="Document + DocumentContract + RuntimeContext",
output_contract="FormState",
diagnostics_namespace="runtime",
provenance_prefix="runtime.form_state",
cli={"commands": ["mkt contract form-state"]},
docs=["docs/runtime-context-forms-assessments.md"],
examples=["examples/runtime/workplan-dynamic.contract.md"],
)
def _runtime_assessment_descriptor() -> ExtensionDescriptor:
return ExtensionDescriptor(
id="runtime.assessment",
kind="assessment-runner",
summary="Provider-neutral rubric assessment request, result, and cache boundary.",
capabilities=[
ProcessingCapability(id="assessment", kind="execute"),
ProcessingCapability(id="rubrics", kind="read"),
ProcessingCapability(id="cache-key", kind="compute"),
ProcessingCapability(id="diagnostics", kind="emit"),
],
safety={"network": "adapter-only", "provider_calls": "adapter-only"},
input_contract="AssessmentRequest",
output_contract="AssessmentResult",
diagnostics_namespace="runtime.assessment",
provenance_prefix="runtime.assessment",
docs=["docs/runtime-context-forms-assessments.md"],
examples=["examples/runtime/concept-note-assessment.contract.md"],
metadata={"provider_implementation": "external adapter required"},
)

53
src/markitect_tool/runtime/__init__.py Normal file
View File

@@ -0,0 +1,53 @@
"""Runtime context, form state, rules, and assessment extension APIs."""
from markitect_tool.runtime.assessment import (
AssessmentAdapter,
AssessmentCache,
AssessmentRequest,
AssessmentResult,
AssessmentRunResult,
AssessmentRunner,
MemoryAssessmentCache,
assessment_requests_for_contract,
run_contract_assessments,
)
from markitect_tool.runtime.context import (
RuntimeContext,
RuntimeContextLoadResult,
RuntimeContextSource,
load_runtime_context_file,
load_runtime_context_file_result,
)
from markitect_tool.runtime.forms import (
FieldState,
FormState,
build_runtime_bindings,
evaluate_form_state,
)
from markitect_tool.runtime.paths import comparable_value, resolve_path
from markitect_tool.runtime.rules import ConditionResult, evaluate_condition
__all__ = [
"AssessmentAdapter",
"AssessmentCache",
"AssessmentRequest",
"AssessmentResult",
"AssessmentRunResult",
"AssessmentRunner",
"ConditionResult",
"FieldState",
"FormState",
"MemoryAssessmentCache",
"RuntimeContext",
"RuntimeContextLoadResult",
"RuntimeContextSource",
"assessment_requests_for_contract",
"build_runtime_bindings",
"comparable_value",
"evaluate_condition",
"evaluate_form_state",
"load_runtime_context_file",
"load_runtime_context_file_result",
"resolve_path",
"run_contract_assessments",
]

309
src/markitect_tool/runtime/assessment.py Normal file
View File

@@ -0,0 +1,309 @@
"""Provider-neutral assessment protocol for rubric-backed contract checks."""
from __future__ import annotations
import hashlib
import json
from dataclasses import asdict, dataclass, field, replace
from typing import Any, Protocol
from markitect_tool.contract.model import DocumentContract
from markitect_tool.core import Document, Section
from markitect_tool.diagnostics import Diagnostic, SourceLocation, has_error
from markitect_tool.runtime.context import RuntimeContext
@dataclass(frozen=True)
class AssessmentRequest:
"""A provider-neutral request for rubric assessment."""
contract_id: str | None
rule_id: str
scope: str
text: str
criteria: Any
context: dict[str, Any] = field(default_factory=dict)
structured_inputs: dict[str, Any] = field(default_factory=dict)
severity: str = "error"
threshold: float | None = None
provider: str | None = None
model: str | None = None
metadata: dict[str, Any] = field(default_factory=dict)
@property
def cache_key(self) -> str:
payload = {
"contract_id": self.contract_id,
"rule_id": self.rule_id,
"scope": self.scope,
"text": self.text,
"criteria": self.criteria,
"context": self.context,
"structured_inputs": self.structured_inputs,
"threshold": self.threshold,
"provider": self.provider,
"model": self.model,
"metadata": self.metadata,
}
return "assessment:" + hashlib.sha256(
json.dumps(payload, sort_keys=True, ensure_ascii=False, default=str).encode("utf-8")
).hexdigest()
def to_dict(self) -> dict[str, Any]:
data = asdict(self)
data["cache_key"] = self.cache_key
return _drop_empty(data)
@dataclass(frozen=True)
class AssessmentResult:
"""Normalized result returned by an assessment adapter."""
rule_id: str
passed: bool
score: float | None = None
reason: str | None = None
diagnostics: list[Diagnostic] = field(default_factory=list)
provider: str | None = None
model: str | None = None
cache_key: str | None = None
cached: bool = False
metadata: dict[str, Any] = field(default_factory=dict)
@property
def valid(self) -> bool:
return self.passed and not has_error(self.diagnostics)
def to_dict(self) -> dict[str, Any]:
data = {
"rule_id": self.rule_id,
"passed": self.passed,
"score": self.score,
"reason": self.reason,
"diagnostics": [diagnostic.to_dict() for diagnostic in self.diagnostics],
"provider": self.provider,
"model": self.model,
"cache_key": self.cache_key,
"cached": self.cached,
"metadata": self.metadata,
}
return _drop_empty(data)
class AssessmentAdapter(Protocol):
"""Adapter boundary for an LLM or other semantic grader."""
def assess(self, request: AssessmentRequest) -> AssessmentResult | dict[str, Any]:
"""Assess a request and return a normalized result or mapping."""
class AssessmentCache(Protocol):
"""Minimal pluggable assessment cache."""
def get(self, cache_key: str) -> AssessmentResult | None:
"""Return a cached assessment result if available."""
def set(self, cache_key: str, result: AssessmentResult) -> None:
"""Store an assessment result."""
class MemoryAssessmentCache:
"""Transparent in-memory cache useful for tests and short workflow runs."""
def __init__(self) -> None:
self._results: dict[str, AssessmentResult] = {}
def get(self, cache_key: str) -> AssessmentResult | None:
return self._results.get(cache_key)
def set(self, cache_key: str, result: AssessmentResult) -> None:
self._results[cache_key] = result
@dataclass(frozen=True)
class AssessmentRunResult:
"""Result of executing one or more rubric assessments."""
assessments: list[AssessmentResult] = field(default_factory=list)
diagnostics: list[Diagnostic] = field(default_factory=list)
@property
def valid(self) -> bool:
return not has_error(self.diagnostics)
def to_dict(self) -> dict[str, Any]:
data = {
"valid": self.valid,
"assessments": [assessment.to_dict() for assessment in self.assessments],
"diagnostics": [diagnostic.to_dict() for diagnostic in self.diagnostics],
}
return _drop_empty(data)
class AssessmentRunner:
"""Invoke an injected assessment adapter and normalize diagnostics."""
def __init__(
self,
adapter: AssessmentAdapter,
*,
cache: AssessmentCache | None = None,
) -> None:
self.adapter = adapter
self.cache = cache
def assess(self, request: AssessmentRequest) -> AssessmentResult:
if self.cache:
cached = self.cache.get(request.cache_key)
if cached:
return replace(cached, cached=True)
raw_result = self.adapter.assess(request)
result = _normalize_assessment_result(raw_result, request)
if self.cache:
self.cache.set(request.cache_key, result)
return result
def assess_all(self, requests: list[AssessmentRequest]) -> AssessmentRunResult:
assessments = [self.assess(request) for request in requests]
diagnostics: list[Diagnostic] = []
for request, assessment in zip(requests, assessments, strict=True):
diagnostics.extend(_diagnostics_for_result(request, assessment))
return AssessmentRunResult(assessments=assessments, diagnostics=diagnostics)
def assessment_requests_for_contract(
document: Document,
contract: DocumentContract,
runtime_context: RuntimeContext | None = None,
) -> list[AssessmentRequest]:
"""Create assessment requests for contract rubric declarations."""
context = runtime_context or RuntimeContext.empty()
requests: list[AssessmentRequest] = []
for index, rubric in enumerate(contract.rubrics):
if not isinstance(rubric, dict):
continue
rule_id = str(rubric.get("id") or f"rubric-{index + 1}")
scope = str(rubric.get("scope") or "document")
text = _text_for_scope(document, contract, scope)
requests.append(
AssessmentRequest(
contract_id=contract.id,
rule_id=rule_id,
scope=scope,
text=text,
criteria=rubric.get("criteria") or rubric.get("prompt") or rubric,
context=context.binding(),
structured_inputs={
"frontmatter": document.frontmatter,
"contract": contract.to_dict(),
},
severity=str(rubric.get("severity", "error")),
threshold=rubric.get("threshold"),
provider=rubric.get("provider"),
model=rubric.get("model"),
metadata={"rubric": rubric},
)
)
return requests
def run_contract_assessments(
document: Document,
contract: DocumentContract,
adapter: AssessmentAdapter,
*,
runtime_context: RuntimeContext | None = None,
cache: AssessmentCache | None = None,
) -> AssessmentRunResult:
"""Run all rubrics in a contract with an injected assessment adapter."""
runner = AssessmentRunner(adapter, cache=cache)
return runner.assess_all(
assessment_requests_for_contract(document, contract, runtime_context)
)
def _normalize_assessment_result(
raw_result: AssessmentResult | dict[str, Any],
request: AssessmentRequest,
) -> AssessmentResult:
if isinstance(raw_result, AssessmentResult):
result = raw_result
else:
result = AssessmentResult(
rule_id=str(raw_result.get("rule_id") or request.rule_id),
passed=bool(raw_result.get("passed")),
score=raw_result.get("score"),
reason=raw_result.get("reason"),
diagnostics=list(raw_result.get("diagnostics", [])),
provider=raw_result.get("provider"),
model=raw_result.get("model"),
metadata=raw_result.get("metadata", {}),
)
return replace(
result,
rule_id=result.rule_id or request.rule_id,
cache_key=request.cache_key,
provider=result.provider or request.provider,
model=result.model or request.model,
)
def _diagnostics_for_result(
request: AssessmentRequest, assessment: AssessmentResult
) -> list[Diagnostic]:
diagnostics = list(assessment.diagnostics)
if not assessment.passed:
diagnostics.append(
Diagnostic(
severity=request.severity,
code="runtime.assessment.failed",
message=assessment.reason or f"Assessment `{request.rule_id}` failed.",
rule_id=request.rule_id,
details={
"scope": request.scope,
"score": assessment.score,
"threshold": request.threshold,
"provider": assessment.provider,
"model": assessment.model,
"cache_key": assessment.cache_key,
},
)
)
return diagnostics
def _text_for_scope(document: Document, contract: DocumentContract, scope: str) -> str:
if scope == "document":
return document.body
if scope.startswith("section."):
section_id = scope.split(".", 1)[1]
for section_spec in contract.sections:
if section_spec.id != section_id:
continue
section = _matching_section(document, section_spec.headings)
if section:
return "\n".join(block.text for block in section.blocks if block.text)
if scope.startswith("field."):
field_id = scope.split(".", 1)[1]
value = document.frontmatter.get(field_id)
return "" if value is None else str(value)
return document.body
def _matching_section(document: Document, headings: list[str]) -> Section | None:
expected = {heading.strip().lower() for heading in headings}
for section in document.sections:
if section.heading.text.strip().lower() in expected:
return section
return None
def _drop_empty(data: dict[str, Any]) -> dict[str, Any]:
return {
key: value
for key, value in data.items()
if value not in (None, [], {}, "")
}

297
src/markitect_tool/runtime/context.py Normal file
View File

@@ -0,0 +1,297 @@
"""Runtime context loading and validation for contract execution."""
from __future__ import annotations
import json
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any
import yaml
from jsonschema import Draft202012Validator, SchemaError, ValidationError
from markitect_tool.diagnostics import Diagnostic, SourceLocation, has_error
CONTEXT_RESERVED_KEYS = {"context", "metadata", "schema", "schemas", "sources"}
@dataclass(frozen=True)
class RuntimeContextSource:
"""Origin metadata for one loaded context object."""
name: str | None = None
path: str | None = None
kind: str = "file"
metadata: dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> dict[str, Any]:
return _drop_empty(asdict(self))
@dataclass(frozen=True)
class RuntimeContext:
"""Named external data available to contract checks and generation."""
data: dict[str, Any] = field(default_factory=dict)
metadata: dict[str, Any] = field(default_factory=dict)
source_path: str | None = None
sources: list[RuntimeContextSource] = field(default_factory=list)
schemas: dict[str, Any] = field(default_factory=dict)
diagnostics: list[Diagnostic] = field(default_factory=list)
@property
def valid(self) -> bool:
return not has_error(self.diagnostics)
def binding(self) -> dict[str, Any]:
"""Return the object bound at `context` in runtime expressions."""
return self.data
def to_dict(self) -> dict[str, Any]:
data = {
"valid": self.valid,
"source_path": self.source_path,
"data": self.data,
"metadata": self.metadata,
"sources": [source.to_dict() for source in self.sources],
"schemas": self.schemas,
"diagnostics": [diagnostic.to_dict() for diagnostic in self.diagnostics],
}
return _drop_empty(data)
@classmethod
def empty(cls) -> "RuntimeContext":
return cls()
@classmethod
def from_mapping(
cls,
raw: dict[str, Any],
*,
source_path: str | None = None,
) -> "RuntimeContext":
if "context" in raw:
context_data = raw.get("context") or {}
if not isinstance(context_data, dict):
return cls(
source_path=source_path,
diagnostics=[
_diagnostic(
"runtime.context.invalid",
"`context` must be a mapping.",
source_path=source_path,
)
],
)
else:
context_data = {
key: value for key, value in raw.items() if key not in CONTEXT_RESERVED_KEYS
}
metadata = raw.get("metadata") if isinstance(raw.get("metadata"), dict) else {}
schemas: dict[str, Any] = {}
if isinstance(raw.get("schema"), dict):
schemas["$context"] = raw["schema"]
if isinstance(raw.get("schemas"), dict):
schemas.update(raw["schemas"])
sources = _sources_from_raw(raw.get("sources"), source_path)
diagnostics = _validate_context_schemas(context_data, schemas, source_path)
return cls(
data=context_data,
metadata=metadata,
source_path=source_path,
sources=sources,
schemas=schemas,
diagnostics=diagnostics,
)
@dataclass(frozen=True)
class RuntimeContextLoadResult:
"""Context load result that can carry diagnostics instead of raising."""
context: RuntimeContext
@property
def valid(self) -> bool:
return self.context.valid
@property
def diagnostics(self) -> list[Diagnostic]:
return self.context.diagnostics
def to_dict(self) -> dict[str, Any]:
return self.context.to_dict()
def load_runtime_context_file(path: str | Path) -> RuntimeContext:
"""Load a runtime context from JSON or YAML."""
return load_runtime_context_file_result(path).context
def load_runtime_context_file_result(path: str | Path) -> RuntimeContextLoadResult:
"""Load a runtime context and represent malformed input as diagnostics."""
context_path = Path(path)
try:
text = context_path.read_text(encoding="utf-8")
except OSError as exc:
return RuntimeContextLoadResult(
RuntimeContext(
source_path=str(context_path),
diagnostics=[
_diagnostic(
"runtime.context.read_failed",
f"Cannot read runtime context: {exc}",
source_path=str(context_path),
)
],
)
)
try:
raw = _load_mapping(text, context_path)
except ValueError as exc:
return RuntimeContextLoadResult(
RuntimeContext(
source_path=str(context_path),
diagnostics=[
_diagnostic(
"runtime.context.malformed",
str(exc),
source_path=str(context_path),
)
],
)
)
return RuntimeContextLoadResult(
RuntimeContext.from_mapping(raw, source_path=str(context_path))
)
def _load_mapping(text: str, path: Path) -> dict[str, Any]:
suffix = path.suffix.lower()
try:
if suffix == ".json":
data = json.loads(text) if text.strip() else {}
else:
data = yaml.safe_load(text) if text.strip() else {}
except (json.JSONDecodeError, yaml.YAMLError) as exc:
raise ValueError(f"Invalid context file `{path}`: {exc}") from exc
if data is None:
data = {}
if not isinstance(data, dict):
raise ValueError("Runtime context file must contain a mapping.")
return data
def _sources_from_raw(
raw_sources: Any, source_path: str | None
) -> list[RuntimeContextSource]:
if not raw_sources:
return [RuntimeContextSource(path=source_path)] if source_path else []
if isinstance(raw_sources, dict):
return [
RuntimeContextSource(
name=str(name),
path=str(raw.get("path")) if isinstance(raw, dict) and raw.get("path") else None,
kind=str(raw.get("kind", "file")) if isinstance(raw, dict) else "file",
metadata=raw.get("metadata", {}) if isinstance(raw, dict) else {},
)
for name, raw in raw_sources.items()
]
if isinstance(raw_sources, list):
sources: list[RuntimeContextSource] = []
for item in raw_sources:
if isinstance(item, dict):
sources.append(
RuntimeContextSource(
name=item.get("name"),
path=item.get("path"),
kind=str(item.get("kind", "file")),
metadata=item.get("metadata", {}),
)
)
return sources
return [RuntimeContextSource(path=source_path)] if source_path else []
def _validate_context_schemas(
data: dict[str, Any], schemas: dict[str, Any], source_path: str | None
) -> list[Diagnostic]:
diagnostics: list[Diagnostic] = []
for name, schema in schemas.items():
if not isinstance(schema, dict):
diagnostics.append(
_diagnostic(
"runtime.context.schema_invalid",
f"Context schema `{name}` must be a mapping.",
source_path=source_path,
rule_id=str(name),
)
)
continue
target = data if name == "$context" else data.get(name)
if name != "$context" and name not in data:
diagnostics.append(
_diagnostic(
"runtime.context.schema_target_missing",
f"Context schema `{name}` has no matching context object.",
source_path=source_path,
rule_id=str(name),
)
)
continue
try:
Draft202012Validator.check_schema(schema)
Draft202012Validator(schema).validate(target)
except SchemaError as exc:
diagnostics.append(
_diagnostic(
"runtime.context.schema_invalid",
f"Invalid context schema `{name}`: {exc.message}",
source_path=source_path,
rule_id=str(name),
)
)
except ValidationError as exc:
diagnostics.append(
_diagnostic(
"runtime.context.schema",
f"Context object `{name}` does not match its schema: {exc.message}",
source_path=source_path,
rule_id=str(name),
details={"path": list(exc.absolute_path)},
)
)
return diagnostics
def _diagnostic(
code: str,
message: str,
*,
source_path: str | None = None,
rule_id: str | None = None,
details: dict[str, Any] | None = None,
) -> Diagnostic:
return Diagnostic(
severity="error",
code=code,
message=message,
source=SourceLocation(path=source_path) if source_path else None,
rule_id=rule_id,
details=details or {},
)
def _drop_empty(data: dict[str, Any]) -> dict[str, Any]:
return {
key: value
for key, value in data.items()
if value not in (None, [], {}, "")
}

839
src/markitect_tool/runtime/forms.py Normal file
View File

@@ -0,0 +1,839 @@
"""Form state and deterministic runtime rule evaluation."""
from __future__ import annotations
import re
from dataclasses import asdict, dataclass, field, replace
from typing import Any
from markitect_tool.contract.model import DocumentContract, FieldSpec, SectionSpec
from markitect_tool.core import Document
from markitect_tool.diagnostics import (
Diagnostic,
SourceLocation,
has_error,
valid_severity,
)
from markitect_tool.runtime.context import RuntimeContext
from markitect_tool.runtime.paths import comparable_value, resolve_path
from markitect_tool.runtime.rules import evaluate_condition
@dataclass(frozen=True)
class FieldState:
"""UI-neutral runtime state for one contract field."""
id: str
path: str | None = None
source: str | None = None
type: str | None = None
label: str | None = None
description: str | None = None
value: Any = None
exists: bool = False
origin: str = "missing"
required: bool = False
visible: bool = True
enabled: bool = True
allowed_values: list[Any] | None = None
diagnostics: list[Diagnostic] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
@property
def valid(self) -> bool:
return not has_error(self.diagnostics)
def to_dict(self) -> dict[str, Any]:
data = asdict(self)
data["diagnostics"] = [diagnostic.to_dict() for diagnostic in self.diagnostics]
return _drop_empty(data)
@dataclass(frozen=True)
class FormState:
"""A complete runtime form state derived from a document contract."""
contract_id: str | None
document_path: str | None = None
context_path: str | None = None
fields: list[FieldState] = field(default_factory=list)
diagnostics: list[Diagnostic] = field(default_factory=list)
rules_applied: list[str] = field(default_factory=list)
metadata: dict[str, Any] = field(default_factory=dict)
@property
def valid(self) -> bool:
return not has_error(self.diagnostics)
@property
def field_values(self) -> dict[str, Any]:
return {field.id: field.value for field in self.fields if field.exists}
def to_dict(self) -> dict[str, Any]:
data = {
"valid": self.valid,
"contract_id": self.contract_id,
"document_path": self.document_path,
"context_path": self.context_path,
"field_values": self.field_values,
"fields": [field.to_dict() for field in self.fields],
"diagnostics": [diagnostic.to_dict() for diagnostic in self.diagnostics],
"rules_applied": self.rules_applied,
"metadata": self.metadata,
}
return _drop_empty(data)
def evaluate_form_state(
document: Document,
contract: DocumentContract,
runtime_context: RuntimeContext | None = None,
) -> FormState:
"""Evaluate contract fields, prefill sources, and dynamic runtime rules."""
context = runtime_context or RuntimeContext.empty()
bindings = build_runtime_bindings(document, contract, context)
fields = [
_resolve_field_state(field_spec, document, contract, bindings)
for field_spec in contract.fields
]
diagnostics: list[Diagnostic] = list(context.diagnostics)
rules_applied: list[str] = []
fields_by_id = {field.id: field for field in fields}
bindings = build_runtime_bindings(document, contract, context, fields_by_id)
fields_by_id, rule_diagnostics, rules_applied = _apply_dynamic_rules(
fields_by_id,
document,
contract,
context,
bindings,
)
diagnostics.extend(rule_diagnostics)
validated_fields: list[FieldState] = []
for field_spec in contract.fields:
field_state = fields_by_id[_field_key(field_spec)]
field_diagnostics = [
*field_state.diagnostics,
*_validate_field_state(field_spec, field_state, document, contract),
]
validated_fields.append(replace(field_state, diagnostics=field_diagnostics))
diagnostics.extend(field_diagnostics)
return FormState(
contract_id=contract.id,
document_path=document.source_path,
context_path=context.source_path,
fields=validated_fields,
diagnostics=diagnostics,
rules_applied=rules_applied,
)
def build_runtime_bindings(
document: Document,
contract: DocumentContract,
context: RuntimeContext,
fields_by_id: dict[str, FieldState] | None = None,
) -> dict[str, Any]:
"""Build deterministic bindings used by field sources and rule conditions."""
field_states = fields_by_id or {}
return {
"document": document.to_dict(),
"frontmatter": document.frontmatter,
"context": context.binding(),
"contract": contract.to_dict(),
"fields": {
field_id: field.to_dict() | {"value": field.value}
for field_id, field in field_states.items()
},
"field_values": {
field_id: field.value
for field_id, field in field_states.items()
if field.exists
},
}
def _resolve_field_state(
field_spec: FieldSpec,
document: Document,
contract: DocumentContract,
bindings: dict[str, Any],
) -> FieldState:
field_id = field_spec.id or "<missing>"
path = field_spec.path or (f"frontmatter.{field_id}" if field_spec.id else None)
source_candidates = _source_candidates(field_spec)
diagnostics: list[Diagnostic] = []
document_value, document_exists = resolve_path(bindings, path)
source_values = [
(source, value)
for source in source_candidates
for value, exists in [resolve_path(bindings, source)]
if exists
]
if document_exists:
origin = "manual"
value = document_value
exists = True
for source, source_value in source_values:
if comparable_value(source_value) != comparable_value(document_value):
diagnostics.append(
_diagnostic(
"runtime.field.conflict",
(
f"Field `{field_id}` is provided by the document and "
f"context source `{source}` with different values."
),
document=document,
contract=contract,
rule_id=field_id,
severity=_conflict_severity(field_spec),
details={
"path": path,
"source": source,
"document_value": comparable_value(document_value),
"context_value": comparable_value(source_value),
},
)
)
elif source_values:
origin = "prefilled"
value = source_values[0][1]
exists = True
distinct = {
repr(comparable_value(item_value))
for _source, item_value in source_values
}
if len(distinct) > 1:
diagnostics.append(
_diagnostic(
"runtime.field.ambiguous",
f"Field `{field_id}` has multiple source values.",
document=document,
contract=contract,
rule_id=field_id,
details={"sources": [source for source, _value in source_values]},
)
)
elif field_spec.default is not None:
origin = "defaulted"
value = field_spec.default
exists = True
else:
origin = "missing"
value = None
exists = False
return FieldState(
id=field_id,
path=path,
source=source_candidates[0] if source_candidates else field_spec.source,
type=field_spec.type,
label=field_spec.label,
description=field_spec.description,
value=value,
exists=exists,
origin=origin,
required=field_spec.required,
allowed_values=field_spec.enum,
diagnostics=diagnostics,
metadata={
"sources": source_candidates,
"coerce": bool(field_spec.raw.get("coerce", False))
if isinstance(field_spec.raw, dict)
else False,
},
)
def _source_candidates(field_spec: FieldSpec) -> list[str]:
raw = field_spec.raw if isinstance(field_spec.raw, dict) else {}
candidates: list[str] = []
source = raw.get("source", field_spec.source)
if isinstance(source, list):
candidates.extend(str(item) for item in source if item)
elif source:
candidates.append(str(source))
sources = raw.get("sources")
if isinstance(sources, list):
candidates.extend(str(item) for item in sources if item)
return list(dict.fromkeys(candidates))
def _field_key(field_spec: FieldSpec) -> str:
return field_spec.id or "<missing>"
def _apply_dynamic_rules(
fields_by_id: dict[str, FieldState],
document: Document,
contract: DocumentContract,
context: RuntimeContext,
bindings: dict[str, Any],
) -> tuple[dict[str, FieldState], list[Diagnostic], list[str]]:
diagnostics: list[Diagnostic] = []
applied: list[str] = []
rules = contract.rules
for index, rule in enumerate(rules):
if not isinstance(rule, dict):
diagnostics.append(
_diagnostic(
"runtime.rule.invalid",
"Dynamic rule must be a mapping.",
document=document,
contract=contract,
)
)
continue
rule_id = str(rule.get("id") or f"rule-{index + 1}")
result = evaluate_condition(rule.get("if"), bindings, rule_id=rule_id)
diagnostics.extend(
_with_contract_location(item, document=document, contract=contract)
for item in result.diagnostics
)
action = rule.get("then") if result.matched else rule.get("else")
if result.matched:
applied.append(rule_id)
if action is not None:
fields_by_id, action_diagnostics = _apply_rule_action(
fields_by_id,
action,
document,
contract,
context,
rule_id,
)
diagnostics.extend(action_diagnostics)
bindings = build_runtime_bindings(document, contract, context, fields_by_id)
if "assert" in rule and result.matched:
diagnostics.extend(
_evaluate_runtime_assertions(
rule["assert"],
bindings,
document,
contract,
rule_id,
rule,
)
)
return fields_by_id, diagnostics, applied
def _apply_rule_action(
fields_by_id: dict[str, FieldState],
action: Any,
document: Document,
contract: DocumentContract,
context: RuntimeContext,
rule_id: str,
) -> tuple[dict[str, FieldState], list[Diagnostic]]:
if not isinstance(action, dict):
return fields_by_id, [
_diagnostic(
"runtime.rule.action_invalid",
f"Dynamic rule `{rule_id}` action must be a mapping.",
document=document,
contract=contract,
rule_id=rule_id,
)
]
diagnostics: list[Diagnostic] = []
updated = dict(fields_by_id)
bindings = build_runtime_bindings(document, contract, context, updated)
for field_id in _field_id_list(action.get("required")):
updated, diagnostics = _set_field_attr(
updated, diagnostics, field_id, {"required": True}, document, contract, rule_id
)
for field_id in _field_id_list(action.get("optional")):
updated, diagnostics = _set_field_attr(
updated, diagnostics, field_id, {"required": False}, document, contract, rule_id
)
for field_id, visible in _field_bool_mapping(action.get("visible")).items():
updated, diagnostics = _set_field_attr(
updated, diagnostics, field_id, {"visible": visible}, document, contract, rule_id
)
for field_id in _field_id_list(action.get("hidden")):
updated, diagnostics = _set_field_attr(
updated, diagnostics, field_id, {"visible": False}, document, contract, rule_id
)
for field_id, enabled in _field_bool_mapping(action.get("enabled")).items():
updated, diagnostics = _set_field_attr(
updated, diagnostics, field_id, {"enabled": enabled}, document, contract, rule_id
)
for field_id in _field_id_list(action.get("disabled")):
updated, diagnostics = _set_field_attr(
updated, diagnostics, field_id, {"enabled": False}, document, contract, rule_id
)
if isinstance(action.get("allowed_values"), dict):
for field_id, allowed in action["allowed_values"].items():
updated, diagnostics = _set_field_attr(
updated,
diagnostics,
str(field_id),
{"allowed_values": allowed if isinstance(allowed, list) else [allowed]},
document,
contract,
rule_id,
)
if isinstance(action.get("set"), dict):
for field_id, raw_value in action["set"].items():
value = _resolve_template_value(raw_value, bindings)
updated, diagnostics = _set_field_attr(
updated,
diagnostics,
str(field_id),
{"value": value, "exists": True, "origin": "calculated"},
document,
contract,
rule_id,
)
if "assert" in action:
diagnostics.extend(
_evaluate_runtime_assertions(
action["assert"],
build_runtime_bindings(document, contract, context, updated),
document,
contract,
rule_id,
action,
)
)
if "sections" in action:
diagnostics.extend(_check_dynamic_sections(action["sections"], document, contract, rule_id))
return updated, diagnostics
def _set_field_attr(
fields_by_id: dict[str, FieldState],
diagnostics: list[Diagnostic],
field_id: str,
updates: dict[str, Any],
document: Document,
contract: DocumentContract,
rule_id: str,
) -> tuple[dict[str, FieldState], list[Diagnostic]]:
if field_id not in fields_by_id:
diagnostics.append(
_diagnostic(
"runtime.rule.unknown_field",
f"Dynamic rule `{rule_id}` references unknown field `{field_id}`.",
document=document,
contract=contract,
rule_id=rule_id,
)
)
return fields_by_id, diagnostics
fields_by_id[field_id] = replace(fields_by_id[field_id], **updates)
return fields_by_id, diagnostics
def _evaluate_runtime_assertions(
raw_assertions: Any,
bindings: dict[str, Any],
document: Document,
contract: DocumentContract,
rule_id: str,
rule: dict[str, Any],
) -> list[Diagnostic]:
assertions = raw_assertions if isinstance(raw_assertions, list) else [raw_assertions]
diagnostics: list[Diagnostic] = []
for assertion in assertions:
result = evaluate_condition(assertion, bindings, rule_id=rule_id)
diagnostics.extend(
_with_contract_location(item, document=document, contract=contract)
for item in result.diagnostics
)
if not result.matched:
severity = _severity_from_mapping(assertion, rule)
message = (
assertion.get("message")
if isinstance(assertion, dict) and assertion.get("message")
else rule.get("message")
or f"Runtime assertion `{rule_id}` was not satisfied."
)
diagnostics.append(
_diagnostic(
"runtime.rule.assertion_failed",
str(message),
document=document,
contract=contract,
rule_id=rule_id,
severity=severity,
details={"assertion": assertion if isinstance(assertion, dict) else None},
)
)
return diagnostics
def _check_dynamic_sections(
raw_sections: Any,
document: Document,
contract: DocumentContract,
rule_id: str,
) -> list[Diagnostic]:
if not isinstance(raw_sections, dict):
return [
_diagnostic(
"runtime.rule.sections_invalid",
"`sections` action must be a mapping.",
document=document,
contract=contract,
rule_id=rule_id,
)
]
section_specs = {section.id: section for section in contract.sections if section.id}
diagnostics: list[Diagnostic] = []
for section_id, raw in raw_sections.items():
section_spec = section_specs.get(str(section_id))
if section_spec is None:
diagnostics.append(
_diagnostic(
"runtime.rule.unknown_section",
f"Dynamic rule `{rule_id}` references unknown section `{section_id}`.",
document=document,
contract=contract,
rule_id=rule_id,
)
)
continue
presence = raw.get("presence") if isinstance(raw, dict) else raw
diagnostics.extend(_check_dynamic_section_presence(document, contract, section_spec, str(presence), rule_id))
return diagnostics
def _check_dynamic_section_presence(
document: Document,
contract: DocumentContract,
section_spec: SectionSpec,
presence: str,
rule_id: str,
) -> list[Diagnostic]:
matches = _matching_section_lines(document, section_spec)
if presence == "required" and not matches:
return [
_diagnostic(
"runtime.section.missing",
f"Dynamic rule requires section `{section_spec.id}`.",
document=document,
contract=contract,
rule_id=rule_id,
guidance=f"Add a {'#' * (section_spec.level or 2)} {section_spec.title or section_spec.id} section.",
)
]
if presence == "recommended" and not matches:
return [
_diagnostic(
"runtime.section.recommended_missing",
f"Dynamic rule recommends section `{section_spec.id}`.",
document=document,
contract=contract,
rule_id=rule_id,
severity="warning",
)
]
if presence == "forbidden" and matches:
return [
_diagnostic(
"runtime.section.forbidden",
f"Dynamic rule forbids section `{section_spec.id}`.",
document=document,
contract=contract,
rule_id=rule_id,
source_line=matches[0],
)
]
if presence == "discouraged" and matches:
return [
_diagnostic(
"runtime.section.discouraged",
f"Dynamic rule discourages section `{section_spec.id}`.",
document=document,
contract=contract,
rule_id=rule_id,
source_line=matches[0],
severity="warning",
)
]
return []
def _validate_field_state(
field_spec: FieldSpec,
field_state: FieldState,
document: Document,
contract: DocumentContract,
) -> list[Diagnostic]:
diagnostics: list[Diagnostic] = []
if field_state.required and field_state.visible and not field_state.exists:
diagnostics.append(
_diagnostic(
"contract.field.missing",
f"Required field `{field_state.id}` is missing.",
document=document,
contract=contract,
rule_id=field_state.id,
guidance=f"Provide `{field_state.path}` in the document or context.",
)
)
return diagnostics
if not field_state.exists:
return diagnostics
value = field_state.value
if field_state.metadata.get("coerce"):
value, coerced, coercion_error = _coerce_value(value, field_state.type)
if coercion_error:
diagnostics.append(
_diagnostic(
"runtime.field.coercion_failed",
f"Field `{field_state.id}` could not be coerced to `{field_state.type}`.",
document=document,
contract=contract,
rule_id=field_state.id,
)
)
elif coerced:
object.__setattr__(field_state, "value", value)
if field_state.type and not _value_matches_type(value, field_state.type):
diagnostics.append(
_diagnostic(
"contract.field.type_mismatch",
(
f"Field `{field_state.id}` must be `{field_state.type}`, "
f"got `{type(value).__name__}`."
),
document=document,
contract=contract,
rule_id=field_state.id,
)
)
allowed_values = field_state.allowed_values
if allowed_values is not None and value not in allowed_values:
diagnostics.append(
_diagnostic(
"contract.field.enum",
f"Field `{field_state.id}` must be one of {allowed_values}.",
document=document,
contract=contract,
rule_id=field_state.id,
)
)
if field_spec.pattern and isinstance(value, str) and not re.search(field_spec.pattern, value):
diagnostics.append(
_diagnostic(
"contract.field.pattern",
f"Field `{field_state.id}` does not match its required pattern.",
document=document,
contract=contract,
rule_id=field_state.id,
)
)
if field_spec.min_length is not None and hasattr(value, "__len__") and len(value) < field_spec.min_length:
diagnostics.append(
_diagnostic(
"contract.field.min_length",
f"Field `{field_state.id}` is shorter than {field_spec.min_length}.",
document=document,
contract=contract,
rule_id=field_state.id,
)
)
if field_spec.max_length is not None and hasattr(value, "__len__") and len(value) > field_spec.max_length:
diagnostics.append(
_diagnostic(
"contract.field.max_length",
f"Field `{field_state.id}` is longer than {field_spec.max_length}.",
document=document,
contract=contract,
rule_id=field_state.id,
)
)
if field_spec.min is not None and isinstance(value, int | float) and value < field_spec.min:
diagnostics.append(
_diagnostic(
"contract.field.min",
f"Field `{field_state.id}` is below {field_spec.min}.",
document=document,
contract=contract,
rule_id=field_state.id,
)
)
if field_spec.max is not None and isinstance(value, int | float) and value > field_spec.max:
diagnostics.append(
_diagnostic(
"contract.field.max",
f"Field `{field_state.id}` is above {field_spec.max}.",
document=document,
contract=contract,
rule_id=field_state.id,
)
)
return diagnostics
def _coerce_value(value: Any, expected_type: str | None) -> tuple[Any, bool, bool]:
if expected_type == "string" and not isinstance(value, str):
return str(value), True, False
if expected_type == "integer" and isinstance(value, str):
try:
return int(value), True, False
except ValueError:
return value, False, True
if expected_type == "number" and isinstance(value, str):
try:
return float(value), True, False
except ValueError:
return value, False, True
if expected_type == "boolean" and isinstance(value, str):
normalized = value.strip().lower()
if normalized in {"true", "yes", "1"}:
return True, True, False
if normalized in {"false", "no", "0"}:
return False, True, False
return value, False, True
return value, False, False
def _value_matches_type(value: Any, expected_type: str) -> bool:
if expected_type == "string":
return isinstance(value, str)
if expected_type == "number":
return isinstance(value, int | float) and not isinstance(value, bool)
if expected_type == "integer":
return isinstance(value, int) and not isinstance(value, bool)
if expected_type == "boolean":
return isinstance(value, bool)
if expected_type == "array":
return isinstance(value, list)
if expected_type == "object":
return isinstance(value, dict)
if expected_type == "date":
return isinstance(value, str)
return True
def _field_id_list(value: Any) -> list[str]:
if value is None:
return []
if isinstance(value, str):
return [value]
if isinstance(value, list):
return [str(item) for item in value if item]
if isinstance(value, dict):
return [str(key) for key, enabled in value.items() if enabled]
return []
def _field_bool_mapping(value: Any) -> dict[str, bool]:
if value is None:
return {}
if isinstance(value, dict):
return {str(key): bool(item) for key, item in value.items()}
if isinstance(value, str):
return {value: True}
if isinstance(value, list):
return {str(item): True for item in value if item}
return {}
def _resolve_template_value(value: Any, bindings: dict[str, Any]) -> Any:
if isinstance(value, str):
full = re.fullmatch(r"\$\{([^}]+)\}", value.strip())
if full:
resolved, exists = resolve_path(bindings, full.group(1))
return resolved if exists else value
def replace_match(match: re.Match[str]) -> str:
resolved, exists = resolve_path(bindings, match.group(1))
return str(resolved) if exists else match.group(0)
return re.sub(r"\$\{([^}]+)\}", replace_match, value)
if isinstance(value, list):
return [_resolve_template_value(item, bindings) for item in value]
if isinstance(value, dict):
return {key: _resolve_template_value(item, bindings) for key, item in value.items()}
return value
def _conflict_severity(field_spec: FieldSpec) -> str:
raw = field_spec.raw if isinstance(field_spec.raw, dict) else {}
severity = raw.get("conflict") or raw.get("conflict_severity") or "warning"
return str(severity) if valid_severity(str(severity)) else "warning"
def _severity_from_mapping(*items: Any) -> str:
for item in items:
if isinstance(item, dict) and valid_severity(str(item.get("severity"))):
return str(item["severity"])
return "error"
def _matching_section_lines(document: Document, section_spec: SectionSpec) -> list[int]:
expected = {_normalize_heading(value) for value in section_spec.headings}
return [
section.heading.line
for section in document.sections
if _normalize_heading(section.heading.text) in expected
]
def _normalize_heading(text: str) -> str:
return re.sub(r"\s+", " ", text.strip().lower())
def _with_contract_location(
diagnostic: Diagnostic,
*,
document: Document,
contract: DocumentContract,
) -> Diagnostic:
return Diagnostic(
severity=diagnostic.severity,
code=diagnostic.code,
message=diagnostic.message,
source=diagnostic.source or SourceLocation(path=document.source_path),
contract=diagnostic.contract or SourceLocation(path=contract.source_path, line=contract.source_line),
rule_id=diagnostic.rule_id,
guidance=diagnostic.guidance,
details=diagnostic.details,
)
def _diagnostic(
code: str,
message: str,
*,
document: Document,
contract: DocumentContract,
rule_id: str | None = None,
severity: str = "error",
guidance: str | None = None,
details: dict[str, Any] | None = None,
source_line: int | None = 1,
) -> Diagnostic:
return Diagnostic(
severity=severity,
code=code,
message=message,
source=SourceLocation(path=document.source_path, line=source_line),
contract=SourceLocation(path=contract.source_path, line=contract.source_line),
rule_id=rule_id,
guidance=guidance,
details=details or {},
)
def _drop_empty(data: dict[str, Any]) -> dict[str, Any]:
return {
key: value
for key, value in data.items()
if value not in (None, [], {}, "")
}

45
src/markitect_tool/runtime/paths.py Normal file
View File

@@ -0,0 +1,45 @@
"""Small path helpers for runtime context and rule evaluation."""
from __future__ import annotations
from typing import Any
def resolve_path(data: Any, path: str | None) -> tuple[Any, bool]:
"""Resolve a dotted path against dictionaries and lists.
The runtime path vocabulary is intentionally small: `context.user.name`,
`frontmatter.status`, `fields.owner.value`, and numeric list indexes are
enough for contract checks, form state, and deterministic rules without
embedding a general expression language.
"""
if not path:
return None, False
current = data
for part in _path_parts(path):
if isinstance(current, dict) and part in current:
current = current[part]
continue
if isinstance(current, list) and part.isdigit():
index = int(part)
if index < len(current):
current = current[index]
continue
return None, False
return current, True
def comparable_value(value: Any) -> Any:
"""Return a stable scalar-ish value for diagnostics and equality checks."""
if isinstance(value, dict):
return {key: comparable_value(value[key]) for key in sorted(value)}
if isinstance(value, list):
return [comparable_value(item) for item in value]
return value
def _path_parts(path: str) -> list[str]:
normalized = str(path).strip().removeprefix("$.")
return [part for part in normalized.split(".") if part]

162
src/markitect_tool/runtime/rules.py Normal file
View File

@@ -0,0 +1,162 @@
"""Deterministic rule conditions for runtime forms and checks."""
from __future__ import annotations
import re
from dataclasses import dataclass, field
from typing import Any
from markitect_tool.diagnostics import Diagnostic
from markitect_tool.runtime.paths import resolve_path
@dataclass(frozen=True)
class ConditionResult:
"""Result of evaluating one deterministic condition."""
matched: bool
diagnostics: list[Diagnostic] = field(default_factory=list)
def evaluate_condition(
condition: Any,
bindings: dict[str, Any],
*,
rule_id: str | None = None,
) -> ConditionResult:
"""Evaluate a small Markitect-native condition mapping."""
if condition is None:
return ConditionResult(True)
if isinstance(condition, bool):
return ConditionResult(condition)
if isinstance(condition, list):
diagnostics: list[Diagnostic] = []
results = [evaluate_condition(item, bindings, rule_id=rule_id) for item in condition]
for result in results:
diagnostics.extend(result.diagnostics)
return ConditionResult(all(result.matched for result in results), diagnostics)
if not isinstance(condition, dict):
return ConditionResult(
False,
[
_diagnostic(
"runtime.rule.condition_invalid",
"Rule condition must be a mapping, list, or boolean.",
rule_id=rule_id,
)
],
)
if "all" in condition:
return _combine_conditions(condition["all"], bindings, rule_id, all)
if "any" in condition:
return _combine_conditions(condition["any"], bindings, rule_id, any)
if "not" in condition:
result = evaluate_condition(condition["not"], bindings, rule_id=rule_id)
return ConditionResult(not result.matched, result.diagnostics)
path = condition.get("path")
if not path:
return ConditionResult(
False,
[
_diagnostic(
"runtime.rule.condition_missing_path",
"Rule condition must declare `path`.",
rule_id=rule_id,
)
],
)
value, exists = resolve_path(bindings, str(path))
diagnostics: list[Diagnostic] = []
matched = True
if "exists" in condition:
matched = matched and (exists is bool(condition["exists"]))
elif not exists:
matched = False
if exists:
matched = matched and _operator_matches(value, condition, diagnostics, rule_id)
return ConditionResult(matched, diagnostics)
def _combine_conditions(
raw_conditions: Any,
bindings: dict[str, Any],
rule_id: str | None,
combiner: Any,
) -> ConditionResult:
conditions = raw_conditions if isinstance(raw_conditions, list) else [raw_conditions]
results = [evaluate_condition(item, bindings, rule_id=rule_id) for item in conditions]
diagnostics: list[Diagnostic] = []
for result in results:
diagnostics.extend(result.diagnostics)
return ConditionResult(combiner(result.matched for result in results), diagnostics)
def _operator_matches(
value: Any,
condition: dict[str, Any],
diagnostics: list[Diagnostic],
rule_id: str | None,
) -> bool:
matched = True
if "equals" in condition:
matched = matched and value == condition["equals"]
if "eq" in condition:
matched = matched and value == condition["eq"]
if "not_equals" in condition:
matched = matched and value != condition["not_equals"]
if "in" in condition:
expected = condition["in"]
matched = matched and isinstance(expected, list) and value in expected
if "contains" in condition:
expected = condition["contains"]
matched = matched and _contains(value, expected)
if "matches" in condition:
pattern = str(condition["matches"])
try:
matched = matched and re.search(pattern, str(value)) is not None
except re.error as exc:
diagnostics.append(
_diagnostic(
"runtime.rule.regex_invalid",
f"Invalid rule regular expression `{pattern}`: {exc}",
rule_id=rule_id,
)
)
matched = False
for key, predicate in {
"gt": lambda actual, expected: actual > expected,
"gte": lambda actual, expected: actual >= expected,
"lt": lambda actual, expected: actual < expected,
"lte": lambda actual, expected: actual <= expected,
}.items():
if key not in condition:
continue
try:
matched = matched and predicate(value, condition[key])
except TypeError:
matched = False
return matched
def _contains(value: Any, expected: Any) -> bool:
if isinstance(value, str):
return str(expected) in value
if isinstance(value, list | tuple | set):
return expected in value
if isinstance(value, dict):
return expected in value
return False
def _diagnostic(code: str, message: str, *, rule_id: str | None = None) -> Diagnostic:
return Diagnostic(
severity="error",
code=code,
message=message,
rule_id=rule_id,
)

24
src/markitect_tool/workflow/engine.py
View File

@@ -25,6 +25,7 @@ from markitect_tool.query import (
extract_document_with_engine,
query_document_with_engine,
)
from markitect_tool.runtime import evaluate_form_state, load_runtime_context_file
from markitect_tool.template import MissingTemplateVariable, TemplateError, render_template
@@ -322,6 +323,8 @@ class WorkflowRunner:
return self._step_contract_stub(step)
if kind == "contract_check":
return self._step_contract_check(step)
if kind == "form_state":
return self._step_form_state(step)
if kind == "assisted":
return self._step_assisted(step)
raise WorkflowError(f"Unsupported workflow step kind `{kind}`")
@@ -404,9 +407,28 @@ class WorkflowRunner:
def _step_contract_check(self, step: dict[str, Any]) -> dict[str, Any]:
document_path = _safe_input_path(self.base_dir, step.get("document"))
contract_path = _safe_input_path(self.base_dir, step.get("contract"))
result = check_markdown_file(document_path, contract_path)
context_path = (
_safe_input_path(self.base_dir, step.get("context"))
if step.get("context")
else None
)
result = check_markdown_file(document_path, contract_path, context_path=context_path)
return result.to_dict() | {"kind": "contract_check"}
def _step_form_state(self, step: dict[str, Any]) -> dict[str, Any]:
document_path = _safe_input_path(self.base_dir, step.get("document"))
contract_path = _safe_input_path(self.base_dir, step.get("contract"))
context_path = (
_safe_input_path(self.base_dir, step.get("context"))
if step.get("context")
else None
)
document = parse_markdown_file(document_path)
contract = load_contract_file(contract_path)
context = load_runtime_context_file(context_path) if context_path else None
result = evaluate_form_state(document, contract, context)
return result.to_dict() | {"kind": "form_state"}
def _step_assisted(self, step: dict[str, Any]) -> dict[str, Any]:
optional = bool(step.get("optional", True))
if self.assisted_hook is None:

17
tests/test_builtin_extension_catalog.py
View File

@@ -14,6 +14,9 @@ def test_builtin_extension_registry_lists_query_processors_and_backend():
assert "processor.include" in ids
assert "backend.local-sqlite" in ids
assert "workflow.markdown-dataflow" in ids
assert "runtime.context" in ids
assert "runtime.form-state" in ids
assert "runtime.assessment" in ids
def test_builtin_processor_descriptors_capture_safety_and_provenance():
@@ -64,3 +67,17 @@ def test_builtin_workflow_descriptor_exposes_cli_and_safety():
"mkt workflow plan",
"mkt workflow run",
]
def test_builtin_runtime_descriptors_expose_boundaries():
registry = builtin_extension_registry()
context = registry.get("runtime.context")
form_state = registry.get("runtime.form-state")
assessment = registry.get("runtime.assessment")
assert context.safety["reads_files"] is True
assert "mkt contract check --context" in context.cli["commands"]
assert {capability.id for capability in form_state.capabilities} >= {"forms", "rules"}
assert assessment.kind == "assessment-runner"
assert assessment.safety["provider_calls"] == "adapter-only"

388
tests/test_runtime_context_forms_assessment.py Normal file
View File

@@ -0,0 +1,388 @@
from pathlib import Path
from click.testing import CliRunner
from markitect_tool.cli import main
from markitect_tool.contract import check_markdown_file, load_contract_file
from markitect_tool.core import parse_markdown, parse_markdown_file
from markitect_tool.runtime import (
AssessmentResult,
AssessmentRunner,
MemoryAssessmentCache,
assessment_requests_for_contract,
evaluate_form_state,
load_runtime_context_file,
)
from markitect_tool.workflow import WorkflowRunner, load_workflow_file
LETTER_CONTRACT = """# Letter Contract
```yaml contract
id: letter-runtime-v1
document:
type: business-letter
fields:
recipient_name:
type: string
required: true
source: context.recipient.name
sender_name:
type: string
required: true
source: context.sender.name
sender_email:
type: string
required: true
source: context.sender.email
pattern: "@example\\\\.com$"
delivery_channel:
type: string
required: true
default: email
enum: [email, print]
postal_address:
type: string
contact_label:
type: string
rules:
- id: postal-address-for-print
if:
path: fields.delivery_channel.value
equals: print
then:
required: [postal_address]
visible:
postal_address: true
else:
hidden: [postal_address]
- id: calculate-contact-label
then:
set:
contact_label: "${fields.sender_name.value} <${fields.sender_email.value}>"
- id: sender-email-domain
assert:
path: context.sender.email
matches: "@example\\\\.com$"
message: Sender email must come from example.com.
severity: warning
sections:
- id: greeting
title: Greeting
presence: required
level: 2
- id: body
title: Body
presence: required
level: 2
- id: closing
title: Closing
presence: required
level: 2
rubrics:
- id: tone-fit
scope: section.body
criteria: The body should match the recipient relationship.
```
"""
LETTER_DOC = """---
document_type: business-letter
---
# Follow Up
## Greeting
Dear Ada,
## Body
Thank you for the productive discussion. We will follow up with a concise
proposal and next steps for the Markdown workflow.
## Closing
Kind regards
"""
LETTER_CONTEXT = """metadata:
case_id: case-42
schema:
type: object
required: [recipient, sender]
properties:
recipient:
type: object
required: [name]
sender:
type: object
required: [name, email]
context:
recipient:
name: Ada Lovelace
sender:
name: Markitect Team
email: hello@example.com
"""
def test_runtime_context_loads_yaml_and_validates_schema(tmp_path: Path):
context_file = tmp_path / "context.yaml"
context_file.write_text(LETTER_CONTEXT, encoding="utf-8")
context = load_runtime_context_file(context_file)
assert context.valid is True
assert context.data["recipient"]["name"] == "Ada Lovelace"
assert context.metadata["case_id"] == "case-42"
def test_runtime_context_reports_schema_failure(tmp_path: Path):
context_file = tmp_path / "context.yaml"
context_file.write_text(
"schema:\n type: object\n required: [recipient]\ncontext: {}\n",
encoding="utf-8",
)
context = load_runtime_context_file(context_file)
assert context.valid is False
assert context.diagnostics[0].code == "runtime.context.schema"
def test_form_state_prefills_defaults_hides_fields_and_calculates_values(tmp_path: Path):
contract_file = tmp_path / "letter.contract.md"
context_file = tmp_path / "context.yaml"
contract_file.write_text(LETTER_CONTRACT, encoding="utf-8")
context_file.write_text(LETTER_CONTEXT, encoding="utf-8")
document = parse_markdown(LETTER_DOC, source_path="letter.md")
form_state = evaluate_form_state(
document,
load_contract_file(contract_file),
load_runtime_context_file(context_file),
)
fields = {field.id: field for field in form_state.fields}
assert form_state.valid is True
assert fields["recipient_name"].origin == "prefilled"
assert fields["delivery_channel"].origin == "defaulted"
assert fields["postal_address"].visible is False
assert fields["contact_label"].origin == "calculated"
assert fields["contact_label"].value == "Markitect Team <hello@example.com>"
def test_context_conflict_keeps_manual_document_value_as_warning(tmp_path: Path):
contract_file = tmp_path / "letter.contract.md"
context_file = tmp_path / "context.yaml"
contract_file.write_text(LETTER_CONTRACT, encoding="utf-8")
context_file.write_text(LETTER_CONTEXT, encoding="utf-8")
document = parse_markdown(
LETTER_DOC.replace(
"document_type: business-letter",
"document_type: business-letter\nrecipient_name: Grace Hopper",
),
source_path="letter.md",
)
form_state = evaluate_form_state(
document,
load_contract_file(contract_file),
load_runtime_context_file(context_file),
)
recipient = next(field for field in form_state.fields if field.id == "recipient_name")
assert form_state.valid is True
assert recipient.value == "Grace Hopper"
assert "runtime.field.conflict" in {diagnostic.code for diagnostic in form_state.diagnostics}
def test_dynamic_rule_requires_print_address(tmp_path: Path):
contract_file = tmp_path / "letter.contract.md"
context_file = tmp_path / "context.yaml"
contract_file.write_text(LETTER_CONTRACT, encoding="utf-8")
context_file.write_text(LETTER_CONTEXT, encoding="utf-8")
document = parse_markdown(
LETTER_DOC.replace(
"document_type: business-letter",
"document_type: business-letter\ndelivery_channel: print",
),
source_path="letter.md",
)
form_state = evaluate_form_state(
document,
load_contract_file(contract_file),
load_runtime_context_file(context_file),
)
assert form_state.valid is False
assert "contract.field.missing" in {diagnostic.code for diagnostic in form_state.diagnostics}
def test_dynamic_section_rule_can_require_section(tmp_path: Path):
contract_file = tmp_path / "workplan.contract.md"
contract_file.write_text(
"""# Workplan Contract
```yaml contract
id: dynamic-workplan-v1
document:
type: workplan
fields:
status:
type: string
required: true
sections:
- id: purpose
title: Purpose
presence: required
- id: decision-point
title: Decision Point
presence: optional
rules:
- id: require-decision-when-done
if:
path: fields.status.value
equals: done
then:
sections:
decision-point:
presence: required
```
""",
encoding="utf-8",
)
document = parse_markdown(
"---\ndocument_type: workplan\nstatus: done\n---\n# WP\n\n## Purpose\n\nDone.\n",
source_path="workplan.md",
)
form_state = evaluate_form_state(document, load_contract_file(contract_file))
assert form_state.valid is False
assert "runtime.section.missing" in {diagnostic.code for diagnostic in form_state.diagnostics}
def test_contract_check_uses_runtime_context(tmp_path: Path):
contract_file = tmp_path / "letter.contract.md"
document_file = tmp_path / "letter.md"
context_file = tmp_path / "context.yaml"
contract_file.write_text(LETTER_CONTRACT, encoding="utf-8")
document_file.write_text(LETTER_DOC, encoding="utf-8")
context_file.write_text(LETTER_CONTEXT, encoding="utf-8")
result = check_markdown_file(document_file, contract_file, context_path=context_file)
assert result.valid is True
assert result.runtime["form_state"]["field_values"]["recipient_name"] == "Ada Lovelace"
def test_contract_cli_accepts_context_and_reports_form_state(tmp_path: Path):
contract_file = tmp_path / "letter.contract.md"
document_file = tmp_path / "letter.md"
context_file = tmp_path / "context.yaml"
contract_file.write_text(LETTER_CONTRACT, encoding="utf-8")
document_file.write_text(LETTER_DOC, encoding="utf-8")
context_file.write_text(LETTER_CONTEXT, encoding="utf-8")
check = CliRunner().invoke(
main,
[
"contract",
"check",
str(document_file),
"--contract",
str(contract_file),
"--context",
str(context_file),
"--format",
"json",
],
)
form_state = CliRunner().invoke(
main,
[
"contract",
"form-state",
str(document_file),
"--contract",
str(contract_file),
"--context",
str(context_file),
"--format",
"text",
],
)
assert check.exit_code == 0
assert '"runtime"' in check.output
assert form_state.exit_code == 0
assert "recipient_name: Ada Lovelace [prefilled]" in form_state.output
def test_assessment_runner_normalizes_cache_and_failure_diagnostics(tmp_path: Path):
contract_file = tmp_path / "letter.contract.md"
contract_file.write_text(LETTER_CONTRACT, encoding="utf-8")
document = parse_markdown_file(_write_file(tmp_path / "letter.md", LETTER_DOC))
requests = assessment_requests_for_contract(document, load_contract_file(contract_file))
class Adapter:
calls = 0
def assess(self, request):
self.calls += 1
return AssessmentResult(
rule_id=request.rule_id,
passed=False,
score=0.2,
reason="Tone is too vague.",
provider="mock",
model="mock-grader",
)
adapter = Adapter()
runner = AssessmentRunner(adapter, cache=MemoryAssessmentCache())
first = runner.assess(requests[0])
second = runner.assess(requests[0])
run = runner.assess_all(requests)
assert first.cached is False
assert second.cached is True
assert adapter.calls == 1
assert "runtime.assessment.failed" in {diagnostic.code for diagnostic in run.diagnostics}
def test_workflow_form_state_step_uses_context(tmp_path: Path):
contract_file = tmp_path / "letter.contract.md"
document_file = tmp_path / "letter.md"
context_file = tmp_path / "context.yaml"
workflow_file = tmp_path / "workflow.yaml"
contract_file.write_text(LETTER_CONTRACT, encoding="utf-8")
document_file.write_text(LETTER_DOC, encoding="utf-8")
context_file.write_text(LETTER_CONTEXT, encoding="utf-8")
workflow_file.write_text(
"""metadata:
id: runtime-workflow
steps:
- id: form
kind: form_state
document: letter.md
contract: letter.contract.md
context: context.yaml
""",
encoding="utf-8",
)
result = WorkflowRunner(load_workflow_file(workflow_file)).run()
assert result.valid is True
assert result.steps["form"]["field_values"]["recipient_name"] == "Ada Lovelace"
def _write_file(path: Path, text: str) -> Path:
path.write_text(text, encoding="utf-8")
return path

67
workplans/MKTT-WP-0005-runtime-context-and-assessment-engines.md
View File

@@ -3,10 +3,10 @@ id: MKTT-WP-0005
type: workplan
title: "Runtime Context, Form, and Assessment Engines"
domain: markitect
status: todo
status: done
owner: markitect-tool
topic_slug: markitect
planning_priority: P2
planning_priority: complete
planning_order: 70
depends_on_workplans:
- MKTT-WP-0003
@@ -31,7 +31,11 @@ This workplan picks up the deferred runtime scope from
## Decision
Do not start this immediately unless one of these is true:
This workplan is implemented. It was picked up after the workflow and internal
extension framework work made runtime form/context evaluation useful and cleanly
integrable.
The original trigger conditions were:
- We are implementing template/generation flows that need reliable field
prefill and pre-render validation.
@@ -41,16 +45,12 @@ Do not start this immediately unless one of these is true:
- A user-facing or agent-facing workflow needs structured form state, defaults,
conditional requiredness, or guided repair.
Recommended sequencing:
Implemented sequencing:
1. Implement context and form runtime first.
2. Add deterministic context-aware rules.
3. Add LLM assessment execution only after the diagnostic/caching boundary is
stable.
This is probably not the next immediate implementation if we want to first
finish core query/extraction and deterministic transform primitives. It should
come before serious generation pipelines or any LLM review loop.
3. Add provider-neutral assessment execution after the diagnostic/caching
boundary was stable.
## Background
@@ -76,7 +76,7 @@ It does not yet execute:
```task
id: MKTT-WP-0005-T001
status: todo
status: done
priority: high
state_hub_task_id: "e24e6238-efef-41c4-9f1e-ca677c1be89b"
```
@@ -96,7 +96,7 @@ Expected output: design notes and tests for context loading.
```task
id: MKTT-WP-0005-T002
status: todo
status: done
priority: high
state_hub_task_id: "d180bb6d-dae8-4305-88de-64c80b708b8a"
```
@@ -114,7 +114,7 @@ application-specific data access belongs in adapters outside the core package.
```task
id: MKTT-WP-0005-T003
status: todo
status: done
priority: high
state_hub_task_id: "b954984a-6f67-4e5b-8744-35e3c4fcc992"
```
@@ -135,7 +135,7 @@ document exists.
```task
id: MKTT-WP-0005-T004
status: todo
status: done
priority: medium
state_hub_task_id: "cccdf868-2308-42a1-b564-8b54fccd3c8b"
```
@@ -157,7 +157,7 @@ This should support future UI adapters while remaining useful from the CLI.
```task
id: MKTT-WP-0005-T005
status: todo
status: done
priority: medium
state_hub_task_id: "6e420e1e-2465-40d3-8e64-d8681a294e63"
```
@@ -178,7 +178,7 @@ small set of operators over embedding a general programming language.
```task
id: MKTT-WP-0005-T006
status: todo
status: done
priority: medium
state_hub_task_id: "24b22b3a-e89e-4946-81f4-94f971a11979"
```
@@ -205,7 +205,7 @@ implementation.
```task
id: MKTT-WP-0005-T007
status: todo
status: done
priority: medium
state_hub_task_id: "b09b77e2-59c0-4d31-b246-685b742d111f"
```
@@ -220,7 +220,7 @@ cache may be local file-based only if it remains transparent and easy to reset.
```task
id: MKTT-WP-0005-T008
status: todo
status: done
priority: high
state_hub_task_id: "2efb8233-3154-4824-a898-6fcde37330c5"
```
@@ -237,14 +237,29 @@ prefill, invalid dynamic rules, and assessment failures.
## Open Questions
- Should context values override frontmatter, or should conflicts always be
diagnostics until explicitly resolved?
- Should the first dynamic rule syntax reuse JSON Schema conditionals or define
a smaller Markitect-native rule vocabulary?
- Should LLM assessment execution live behind an optional extra, or only in
external adapters?
- What cache invalidation metadata is sufficient for assessment reproducibility
without pretending model judgments are deterministic?
- Resolved: document/frontmatter values win over context; conflicts are
diagnostics and can be escalated per field.
- Resolved: dynamic rules use a smaller Markitect-native vocabulary over
JSON/YAML paths instead of embedding JSON Schema conditionals.
- Resolved: provider execution remains external; core defines request/result,
runner, diagnostics, and cache boundary only.
- Resolved for core: assessment cache keys include contract/rule/scope/text,
criteria, context, structured inputs, provider/model metadata, threshold, and
metadata. Persistent cache invalidation remains a backend concern.
## Implementation Notes
- Added `markitect_tool.runtime` with context loading, path resolution,
deterministic condition evaluation, form state, dynamic rules, assessment
request/result models, runner, and in-memory cache boundary.
- `mkt contract check` now accepts `--context`.
- Added `mkt contract form-state`.
- Workflow `contract_check` accepts context and workflow `form_state` exposes
runtime field state as a step result.
- Built-in extension descriptors now include `runtime.context`,
`runtime.form-state`, and `runtime.assessment`.
- Documentation lives in `docs/runtime-context-forms-assessments.md`.
- Examples live in `examples/runtime/`.
## Exit Criteria