markitect-tool/docs/practical-schema-framework-research.md

Practical Schema Framework Research

Date: 2026-05-03

Purpose

This document reassesses markitect-tool schema utility before further implementation. The concern is that pure structural validation, such as heading counts and min/max depth constraints, is rarely enough to make markdown document pipelines useful.

The practical opportunity is to define a stronger framework for markdown-native document contracts: section specifications, content assertions, form fields, context-aware rules, LLM-assisted assessments, and high-quality diagnostics.

Research Signals

Structured Authoring

DITA is the strongest analogue for typed, reusable textual units. It emphasizes information typing, semantic markup, modularity, reuse, interchange, and multiple deliverables from one source. A DITA topic is the unit of authoring and reuse; topics may be generic or specialized into roles such as concept, task, or reference.

Relevance for markitect-tool:

• A markdown document or section should have an explicit information type.
• Information type should imply expected structure and reader purpose.
• Reuse and composition need stable addressing of sections, not only files.
• Specialization is a better mental model than ad hoc schema forks.

Sources:

• https://dita-lang.org/dita/archspec/base/basic-concepts
• https://dita-lang.org/dita/archspec/base/introduction-to-dita

Document Schemas With Assertions

DocBook remains relevant because it combines formal document schemas with Schematron-style assertions. That is the missing layer in many simplistic JSON Schema approaches: grammar says what may exist; assertions say what must be true in context.

Relevance for markitect-tool:

• JSON Schema over Document.to_dict() is useful but insufficient.
• We need a second assertion layer for document-specific semantics.
• Diagnostics must point to the document location and rule intention.

Source:

• https://docbook.org/schemas/docbook/

Dynamic Form Rules

JSON Schema supports conditional validation through dependentRequired, dependentSchemas, and if/then/else. JSON Forms separates data schema from UI schema and uses rules to show, hide, enable, or disable UI elements based on JSON Schema conditions. Form.io’s architecture treats the form schema as a single source of truth for validation and conditional logic across client and server.

Relevance for markitect-tool:

• Forms should be first-class, not bolted onto document generation.
• Field definitions need static validation and dynamic rules.
• Prefill, visibility, requiredness, and calculated values should come from the same contract used for generation and validation.
• Context data must be explicit and typed.

Sources:

• https://json-schema.org/understanding-json-schema/reference/conditionals
• https://jsonforms.io/docs/uischema/rules/
• https://form.io/features/form-conditional-logic-form-validation/

LLM-Assisted Assessment

Modern evaluation frameworks treat LLM assessment as explicit graders or rubrics. OpenAI graders return scores in a 0–1 range and can combine grader types. Promptfoo’s llm-rubric uses explicit criteria and expects structured judge output with reason, score, and pass/fail.

Relevance for markitect-tool:

• LLM checks should be declared as assessment rules, not hidden in prompts.
• Deterministic validation and LLM assessment should produce one diagnostic model.
• Section-level rubrics are more useful than whole-document vague grading.
• The LLM provider must remain external; markitect-tool defines contracts and reports.

Sources:

• https://developers.openai.com/api/docs/guides/graders
• https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/

Markdown Structure

CommonMark gives markdown a well-defined block/inline model. mdast gives a language-neutral tree vocabulary for Markdown nodes. Both point toward keeping the parse layer separate from domain/schema layers.

Relevance for markitect-tool:

• The core document model should stay close to CommonMark/mdast concepts.
• Practical document contracts should sit above the parse model.
• Section addressing, source spans, and block identity are foundational for good diagnostics.

Sources:

• https://spec.commonmark.org/0.31.2/
• https://github.com/syntax-tree/mdast

Terminology Proposal

Term	Meaning
Document	A markdown artifact parsed into frontmatter, blocks, headings, sections, and source spans.
Section	A heading-led document region with content, children, source location, and stable identity.
Document Type	A named contract for a whole document, e.g. ADR, PRD, invoice letter, support reply, concept note.
Section Type	A reusable role for a section, e.g. Context, Decision, Risks, Procedure, Evidence, Conclusion.
Field	A typed value expected in frontmatter, inline matter, a section, or an external data record.
Form	A field collection with UI hints, validation rules, defaults, dynamic visibility, and calculations.
Context	External data available during validation/generation, such as user data, project data, dates, or related entities.
Rule	A deterministic condition evaluated against document, fields, context, or pipeline state.
Assertion	A claim that must hold for content, usually richer than shape validation.
Metric Band	A soft or hard target for size/complexity, such as word count, sentence count, section count, or reading level.
Assessment	A deterministic or LLM-assisted evaluation that returns pass/fail, score, reason, and diagnostics.
Rubric	A human-readable criterion for LLM-assisted assessment, scoped to a document or section type.
Diagnostic	A structured finding with severity, code, message, source location, rule id, and suggested repair.
Contract	The full specification for a document type: structure, sections, fields, rules, forms, assertions, rubrics, and outputs.
Pipeline	A repeatable sequence of parse, prefill, generate, validate, assess, transform, and compose operations.

Most Relevant Use Cases

UC-001: Typed Document Contract

Define a document type such as ADR, PRD, FRS, workplan, customer letter, or meeting brief. Specify required sections by semantic role, allowed alternatives, field requirements, and diagnostics.

Practical value:

• Prevents missing critical content.
• Makes generated documents predictable.
• Creates an explicit contract for humans and agents.

Needed tooling:

• mkt contract check <doc> --contract <contract.md>
• Section matching by heading text, aliases, ids, or section type markers.
• Diagnostics that say which section/field/assertion failed and why.

UC-002: Section-Level Content Expectations

Specify what a section is expected to contain: assertions, required evidence, forbidden omissions, content patterns, examples, and reviewer prompts.

Practical value:

• Moves beyond “has a heading” toward “does the section do its job?”
• Enables review of generated or human-authored text.

Needed tooling:

• Deterministic assertions for regex, presence, references, counts, and field values.
• Optional LLM rubrics for semantic content checks.
• Per-section diagnostic reports.

UC-003: Size and Complexity Bands

Define soft/hard bands for document and section size: words, characters, sentences, paragraphs, sections, list items, code blocks, and nesting depth.

Practical value:

• Controls generation output size.
• Keeps templates from becoming bloated or underdeveloped.
• Helps compare intended vs actual document complexity.

Needed tooling:

• Metrics extractor.
• Rule severities: info, warning, error.
• “Too small/too large” diagnostics with actual and target values.

UC-004: Form-Backed Markdown Generation

Define forms that collect or prefill structured fields, then render markdown documents. Fields may be static, calculated, conditional, or context-derived.

Practical value:

• Bridges structured data capture and prose generation.
• Supports repeatable business documents.
• Makes prefill from user/project/entity data explicit.

Needed tooling:

• Field schema.
• UI schema or form hints.
• Dynamic rules for requiredness, visibility, defaults, and calculations.
• Template rendering with validation before and after render.

UC-005: Context-Aware Validation

Validate a document against external context: user data, project metadata, related entities, dates, policy constraints, or canonical terminology.

Practical value:

• Checks whether a document is correct for this case, not only generally well-formed.
• Enables pipelines like personalized letters, compliance reports, and project-specific workplans.

Needed tooling:

• Context object schema.
• Resolvers for local files, JSON/YAML data, and later higher-layer systems.
• Rule expressions that can reference document and context paths.

UC-006: LLM-Assisted Section Assessment

Attach rubrics to section types. Use an external LLM adapter to assess whether a section satisfies the rubric, returning score, reason, and pass/fail.

Practical value:

• Handles semantic checks that deterministic rules cannot.
• Supports review loops for generated text.
• Makes subjective requirements explicit and auditable.

Needed tooling:

• Rubric declaration format.
• Provider-neutral assessment request/response models.
• Caching and reproducibility metadata.
• Clear distinction between deterministic errors and model-judged findings.

UC-007: Pipeline Diagnostics and Repair Guidance

Run a document pipeline and get one coherent diagnostic report from parsing, schema checks, field validation, assertions, generation, composition, and LLM-assisted assessments.

Practical value:

• Makes failures debuggable.
• Helps humans and agents repair documents.
• Avoids scattered errors from unrelated subsystems.

Needed tooling:

• Common diagnostic model.
• Error codes and severities.
• Source spans and rule ids.
• Suggested repair text or structured patches when safe.

Comparison With markitect-main

markitect-main had several useful seeds:

• x-markitect-sections for required/recommended/optional/discouraged/improper sections.
• x-markitect-content-control for required, discouraged, and forbidden patterns plus word-count metrics.
• Section and content validators with warnings/errors.
• Schema generation and validation experiments.
• Draft generation with x-markitect-field-mapping.
• Prompt quality gates with schema and pattern validators.
• Infospace entity parsing and LLM classification/evaluation.

The problem was not lack of ideas. The problem was that the ideas lived in separate subsystems with different models:

• Schema validation compared generated schemas rather than validating a stable document contract.
• Semantic validation used x-markitect-* extensions but was not integrated into a unified contract framework.
• Field mapping existed in draft generation, not in a general form/context model.
• LLM quality gates existed inside prompt execution, not as provider-neutral document assessments.
• Infospace checks were domain/application layer behavior, not syntax-layer primitives.

Strategic Direction

The successor should introduce a framework layer above parsing:

Markdown parse model
  -> document contract
      -> section specifications
      -> field/form specifications
      -> deterministic rules/assertions
      -> metric bands
      -> optional LLM rubrics
      -> unified diagnostics

This should not replace JSON Schema. JSON Schema remains useful for typed data and machine validation. The new layer should make document-specific semantics natural.

Recommendation

Do not continue straight into generic query/transform work until this framework direction is captured. The next implementation slice should be a small, deterministic version of document contracts:

1. Define the contract schema and terminology.
2. Implement section specifications.
3. Implement metric bands.
4. Implement the unified diagnostic model.
5. Leave LLM rubrics and form dynamics as designed extension points for the next slice.

This is the utility inflection point. It will make markitect-tool practically useful instead of merely structurally correct.