generated from coulomb/repo-seed
Followup workplan for more in depth document information processing
This commit is contained in:
@@ -152,6 +152,22 @@ It should return:
|
|||||||
- model/provider metadata
|
- model/provider metadata
|
||||||
- diagnostics using the shared diagnostic model
|
- diagnostics using the shared diagnostic model
|
||||||
|
|
||||||
|
## Deferred Runtime Work
|
||||||
|
|
||||||
|
The deterministic contract framework is ready now. The runtime engines are
|
||||||
|
deferred to `MKTT-WP-0005-runtime-context-and-assessment-engines.md`.
|
||||||
|
|
||||||
|
Pick that work up when one of these becomes true:
|
||||||
|
|
||||||
|
- contract checks need external user, project, or entity context
|
||||||
|
- generation needs reliable field prefill before rendering
|
||||||
|
- a UI or agent workflow needs form state, defaults, and dynamic requiredness
|
||||||
|
- deterministic section assertions are not enough and rubric-based semantic
|
||||||
|
assessment becomes necessary
|
||||||
|
|
||||||
|
The intended order is context and form runtime first, deterministic dynamic
|
||||||
|
rules second, LLM assessment execution third.
|
||||||
|
|
||||||
## CLI
|
## CLI
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
|||||||
249
workplans/MKTT-WP-0005-runtime-context-and-assessment-engines.md
Normal file
249
workplans/MKTT-WP-0005-runtime-context-and-assessment-engines.md
Normal file
@@ -0,0 +1,249 @@
|
|||||||
|
---
|
||||||
|
id: MKTT-WP-0005
|
||||||
|
type: workplan
|
||||||
|
title: "Runtime Context, Form, and Assessment Engines"
|
||||||
|
domain: markitect
|
||||||
|
status: todo
|
||||||
|
owner: markitect-tool
|
||||||
|
topic_slug: markitect
|
||||||
|
created: "2026-05-03"
|
||||||
|
updated: "2026-05-03"
|
||||||
|
state_hub_workstream_id: "7918687e-2364-46b1-ab7e-65aa77cb8449"
|
||||||
|
---
|
||||||
|
|
||||||
|
# MKTT-WP-0005: Runtime Context, Form, and Assessment Engines
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Turn the contract framework extension points into executable runtime engines:
|
||||||
|
context loading, field prefill, form state evaluation, dynamic rules, and
|
||||||
|
provider-neutral LLM assessment execution.
|
||||||
|
|
||||||
|
This workplan picks up the deferred runtime scope from
|
||||||
|
`MKTT-WP-0004-practical-contract-framework.md`.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Do not start this immediately unless one of these is true:
|
||||||
|
|
||||||
|
- We are implementing template/generation flows that need reliable field
|
||||||
|
prefill and pre-render validation.
|
||||||
|
- We need document checks that depend on case-specific external context.
|
||||||
|
- Deterministic assertions are no longer enough to assess whether sections do
|
||||||
|
their semantic job.
|
||||||
|
- A user-facing or agent-facing workflow needs structured form state, defaults,
|
||||||
|
conditional requiredness, or guided repair.
|
||||||
|
|
||||||
|
Recommended sequencing:
|
||||||
|
|
||||||
|
1. Implement context and form runtime first.
|
||||||
|
2. Add deterministic context-aware rules.
|
||||||
|
3. Add LLM assessment execution only after the diagnostic/caching boundary is
|
||||||
|
stable.
|
||||||
|
|
||||||
|
This is probably not the next immediate implementation if we want to first
|
||||||
|
finish core query/extraction and deterministic transform primitives. It should
|
||||||
|
come before serious generation pipelines or any LLM review loop.
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
The deterministic contract framework already supports:
|
||||||
|
|
||||||
|
- field declarations
|
||||||
|
- deterministic section assertions
|
||||||
|
- metric bands
|
||||||
|
- provider-neutral rubric declarations as contract vocabulary
|
||||||
|
- one shared diagnostic model
|
||||||
|
|
||||||
|
It does not yet execute:
|
||||||
|
|
||||||
|
- context resolvers
|
||||||
|
- form state evaluation
|
||||||
|
- dynamic requiredness or visibility
|
||||||
|
- calculated values
|
||||||
|
- prefill
|
||||||
|
- provider-neutral LLM assessment requests
|
||||||
|
- assessment result caching
|
||||||
|
|
||||||
|
## P5.1 - Define runtime context model
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: MKTT-WP-0005-T001
|
||||||
|
status: todo
|
||||||
|
priority: high
|
||||||
|
state_hub_task_id: "e24e6238-efef-41c4-9f1e-ca677c1be89b"
|
||||||
|
```
|
||||||
|
|
||||||
|
Define how external context is supplied to contract checks and generation:
|
||||||
|
|
||||||
|
- inline YAML/JSON files
|
||||||
|
- named context objects
|
||||||
|
- typed context schemas
|
||||||
|
- explicit source paths
|
||||||
|
- conflict behavior when frontmatter and context both provide a value
|
||||||
|
- diagnostic behavior for missing or malformed context
|
||||||
|
|
||||||
|
Expected output: design notes and tests for context loading.
|
||||||
|
|
||||||
|
## P5.2 - Implement context resolver API and CLI input
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: MKTT-WP-0005-T002
|
||||||
|
status: todo
|
||||||
|
priority: high
|
||||||
|
state_hub_task_id: "d180bb6d-dae8-4305-88de-64c80b708b8a"
|
||||||
|
```
|
||||||
|
|
||||||
|
Add a small runtime API and CLI option such as:
|
||||||
|
|
||||||
|
```text
|
||||||
|
mkt contract check <document.md> --contract <contract.md> --context <context.yaml>
|
||||||
|
```
|
||||||
|
|
||||||
|
Resolvers must be deterministic, local-first, and provider-neutral. Network or
|
||||||
|
application-specific data access belongs in adapters outside the core package.
|
||||||
|
|
||||||
|
## P5.3 - Implement field prefill and validation runtime
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: MKTT-WP-0005-T003
|
||||||
|
status: todo
|
||||||
|
priority: high
|
||||||
|
state_hub_task_id: "b954984a-6f67-4e5b-8744-35e3c4fcc992"
|
||||||
|
```
|
||||||
|
|
||||||
|
Evaluate field specs against document frontmatter and context:
|
||||||
|
|
||||||
|
- required fields
|
||||||
|
- defaults
|
||||||
|
- source paths
|
||||||
|
- enum/pattern/range validation
|
||||||
|
- type coercion policy
|
||||||
|
- diagnostics for missing, ambiguous, or conflicting values
|
||||||
|
|
||||||
|
Expected utility: contracts become useful before generation, not only after a
|
||||||
|
document exists.
|
||||||
|
|
||||||
|
## P5.4 - Implement form state model
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: MKTT-WP-0005-T004
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
state_hub_task_id: "cccdf868-2308-42a1-b564-8b54fccd3c8b"
|
||||||
|
```
|
||||||
|
|
||||||
|
Represent form state without binding to a UI framework:
|
||||||
|
|
||||||
|
- field id
|
||||||
|
- value
|
||||||
|
- defaulted/prefilled/manual/calculated origin
|
||||||
|
- visible/hidden
|
||||||
|
- enabled/disabled
|
||||||
|
- required/optional
|
||||||
|
- validation diagnostics
|
||||||
|
- display hints as metadata, not behavior
|
||||||
|
|
||||||
|
This should support future UI adapters while remaining useful from the CLI.
|
||||||
|
|
||||||
|
## P5.5 - Implement dynamic rules
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: MKTT-WP-0005-T005
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
state_hub_task_id: "6e420e1e-2465-40d3-8e64-d8681a294e63"
|
||||||
|
```
|
||||||
|
|
||||||
|
Add deterministic dynamic rules for field and section behavior:
|
||||||
|
|
||||||
|
- `if` / `then` / `else`
|
||||||
|
- requiredness
|
||||||
|
- visibility
|
||||||
|
- allowed values
|
||||||
|
- calculated values
|
||||||
|
- context-dependent assertions
|
||||||
|
|
||||||
|
Keep the expression language intentionally small. Prefer JSON/YAML paths and a
|
||||||
|
small set of operators over embedding a general programming language.
|
||||||
|
|
||||||
|
## P5.6 - Define LLM assessment execution interface
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: MKTT-WP-0005-T006
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
state_hub_task_id: "24b22b3a-e89e-4946-81f4-94f971a11979"
|
||||||
|
```
|
||||||
|
|
||||||
|
Define provider-neutral request/response models for rubric execution:
|
||||||
|
|
||||||
|
- contract id
|
||||||
|
- rule id
|
||||||
|
- scope: document, section, or field
|
||||||
|
- text and structured inputs
|
||||||
|
- context snapshot
|
||||||
|
- rubric criteria
|
||||||
|
- cache key material
|
||||||
|
- pass/fail
|
||||||
|
- score
|
||||||
|
- reason
|
||||||
|
- model/provider metadata
|
||||||
|
- diagnostics
|
||||||
|
|
||||||
|
Core package should define the protocol and result model, not a provider
|
||||||
|
implementation.
|
||||||
|
|
||||||
|
## P5.7 - Add assessment runner and cache boundary
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: MKTT-WP-0005-T007
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
state_hub_task_id: "b09b77e2-59c0-4d31-b246-685b742d111f"
|
||||||
|
```
|
||||||
|
|
||||||
|
Implement a runner that can invoke an injected assessment adapter and normalize
|
||||||
|
results into diagnostics.
|
||||||
|
|
||||||
|
Add deterministic cache key calculation but keep storage pluggable. The default
|
||||||
|
cache may be local file-based only if it remains transparent and easy to reset.
|
||||||
|
|
||||||
|
## P5.8 - Add examples and failure diagnostics
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: MKTT-WP-0005-T008
|
||||||
|
status: todo
|
||||||
|
priority: high
|
||||||
|
state_hub_task_id: "2efb8233-3154-4824-a898-6fcde37330c5"
|
||||||
|
```
|
||||||
|
|
||||||
|
Create examples that show the practical value:
|
||||||
|
|
||||||
|
- business letter with context-prefilled recipient/sender data
|
||||||
|
- PRD/FRS with context-dependent product metadata
|
||||||
|
- workplan where task requirements depend on status
|
||||||
|
- concept note with LLM rubric declaration and mocked assessment output
|
||||||
|
|
||||||
|
Each example should include expected diagnostics for missing context, ambiguous
|
||||||
|
prefill, invalid dynamic rules, and assessment failures.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
- Should context values override frontmatter, or should conflicts always be
|
||||||
|
diagnostics until explicitly resolved?
|
||||||
|
- Should the first dynamic rule syntax reuse JSON Schema conditionals or define
|
||||||
|
a smaller Markitect-native rule vocabulary?
|
||||||
|
- Should LLM assessment execution live behind an optional extra, or only in
|
||||||
|
external adapters?
|
||||||
|
- What cache invalidation metadata is sufficient for assessment reproducibility
|
||||||
|
without pretending model judgments are deterministic?
|
||||||
|
|
||||||
|
## Exit Criteria
|
||||||
|
|
||||||
|
- Runtime context can be supplied to contract checks.
|
||||||
|
- Field prefill and validation produce unified diagnostics.
|
||||||
|
- Form state can be rendered by a future adapter without changing core models.
|
||||||
|
- Dynamic rules cover common requiredness, visibility, and context assertions.
|
||||||
|
- LLM rubric execution has a provider-neutral protocol and mocked test adapter.
|
||||||
|
- Examples demonstrate utility beyond static document validation.
|
||||||
Reference in New Issue
Block a user