generated from coulomb/repo-seed
Compare commits
2 Commits
61b31e4ebf
...
ab7914890e
| Author | SHA1 | Date | |
|---|---|---|---|
| ab7914890e | |||
| 955643554f |
@@ -2,16 +2,18 @@
|
|||||||
# Custodian Brief — guide-board
|
# Custodian Brief — guide-board
|
||||||
|
|
||||||
**Domain:** markitect
|
**Domain:** markitect
|
||||||
**Last synced:** 2026-05-15 12:43 UTC
|
**Last synced:** 2026-05-15 13:10 UTC
|
||||||
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
|
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
|
||||||
|
|
||||||
## Active Workstreams
|
## Active Workstreams
|
||||||
|
|
||||||
### Assessment Operations Baseline
|
### Extension SDK Maturity
|
||||||
Progress: 5/6 done | workstream_id: `fc5b1573-91b2-4a19-b6a9-dd4d17057d9b`
|
Progress: 1/4 done | workstream_id: `26aa9511-cd5c-4dd5-989c-d2838ba3b50d`
|
||||||
|
|
||||||
**Open tasks:**
|
**Open tasks:**
|
||||||
- · D2.6 - External Extension Acceptance Path `65fbf1df`
|
- · D3.2 - Normalizer Plug-in Contract `b87e68c1`
|
||||||
|
- · D3.3 - SDK Fixture Extension And Acceptance Tests `f3738751`
|
||||||
|
- · D3.4 - Extension Authoring Documentation Refresh `3d390bd4`
|
||||||
|
|
||||||
---
|
---
|
||||||
## MCP Orientation (when available)
|
## MCP Orientation (when available)
|
||||||
|
|||||||
@@ -54,6 +54,7 @@ See:
|
|||||||
- [docs/CANDIDATE-HANDOFF.md](docs/CANDIDATE-HANDOFF.md)
|
- [docs/CANDIDATE-HANDOFF.md](docs/CANDIDATE-HANDOFF.md)
|
||||||
- [docs/COMPLIANCE-EVIDENCE-PACKS.md](docs/COMPLIANCE-EVIDENCE-PACKS.md)
|
- [docs/COMPLIANCE-EVIDENCE-PACKS.md](docs/COMPLIANCE-EVIDENCE-PACKS.md)
|
||||||
- [docs/CONTAINER.md](docs/CONTAINER.md)
|
- [docs/CONTAINER.md](docs/CONTAINER.md)
|
||||||
|
- [docs/EXTERNAL-EXTENSION-ACCEPTANCE.md](docs/EXTERNAL-EXTENSION-ACCEPTANCE.md)
|
||||||
- [docs/EXTENSION-SDK.md](docs/EXTENSION-SDK.md)
|
- [docs/EXTENSION-SDK.md](docs/EXTENSION-SDK.md)
|
||||||
- [docs/LOCAL-SERVICE-API.md](docs/LOCAL-SERVICE-API.md)
|
- [docs/LOCAL-SERVICE-API.md](docs/LOCAL-SERVICE-API.md)
|
||||||
- [docs/SERVICE-JOB-DURABILITY.md](docs/SERVICE-JOB-DURABILITY.md)
|
- [docs/SERVICE-JOB-DURABILITY.md](docs/SERVICE-JOB-DURABILITY.md)
|
||||||
|
|||||||
@@ -62,6 +62,9 @@ PYTHONPATH=src python3 -m guide_board \
|
|||||||
|
|
||||||
The same extension roots can be provided through `GUIDE_BOARD_EXTENSION_PATHS`
|
The same extension roots can be provided through `GUIDE_BOARD_EXTENSION_PATHS`
|
||||||
when a wrapper script or container entrypoint should keep commands shorter.
|
when a wrapper script or container entrypoint should keep commands shorter.
|
||||||
|
For the repeatable external extension acceptance path, including validation,
|
||||||
|
planning, live execution, and retained result review, see
|
||||||
|
`docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
|
||||||
|
|
||||||
## CLI Results
|
## CLI Results
|
||||||
|
|
||||||
|
|||||||
@@ -79,6 +79,45 @@ The key runtime fields are:
|
|||||||
- `certification_boundary`: explicit statement of what the extension does not
|
- `certification_boundary`: explicit statement of what the extension does not
|
||||||
certify.
|
certify.
|
||||||
|
|
||||||
|
`profile_schemas` may use the original string shorthand for core schemas:
|
||||||
|
|
||||||
|
```json
|
||||||
|
["target-profile", "assessment-profile"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Extensions that need stricter domain-specific validation can add schema
|
||||||
|
descriptors:
|
||||||
|
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
"target-profile",
|
||||||
|
"assessment-profile",
|
||||||
|
{
|
||||||
|
"id": "cmis-browser-target",
|
||||||
|
"profile_kind": "target",
|
||||||
|
"path": "schemas/cmis-browser-target.schema.json",
|
||||||
|
"subject_type": "cmis-browser-binding-endpoint",
|
||||||
|
"description": "Requires the target shape expected by the CMIS Browser Binding harness."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
Descriptor fields:
|
||||||
|
|
||||||
|
- `id`: stable schema descriptor ID used in validation errors.
|
||||||
|
- `profile_kind`: `target` or `assessment`.
|
||||||
|
- `path`: JSON schema path relative to the extension root.
|
||||||
|
- `subject_type`: optional target-profile selector. When present, the schema is
|
||||||
|
applied only to targets with that `subject_type`.
|
||||||
|
- `description`: optional authoring note.
|
||||||
|
|
||||||
|
The core validates the generic guide-board schema first, then applies matching
|
||||||
|
extension-owned schemas during `profile validate-*`, `plan`, and `run`.
|
||||||
|
Extension schema paths must stay inside the extension root. The baseline
|
||||||
|
validator intentionally supports the small JSON Schema subset used by
|
||||||
|
guide-board contracts: `type`, `enum`, `required`, `properties`,
|
||||||
|
`additionalProperties`, `items`, and `minItems`.
|
||||||
|
|
||||||
## Runner Entry Points
|
## Runner Entry Points
|
||||||
|
|
||||||
Runner entry points currently support these kinds:
|
Runner entry points currently support these kinds:
|
||||||
|
|||||||
144
docs/EXTERNAL-EXTENSION-ACCEPTANCE.md
Normal file
144
docs/EXTERNAL-EXTENSION-ACCEPTANCE.md
Normal file
@@ -0,0 +1,144 @@
|
|||||||
|
# External Extension Acceptance
|
||||||
|
|
||||||
|
Status: draft
|
||||||
|
Created: 2026-05-15
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document defines the repeatable acceptance path for a guide-board
|
||||||
|
extension that lives outside the core repository. It proves that guide-board can
|
||||||
|
discover the extension, validate candidate profiles, build a traceable run plan,
|
||||||
|
execute the selected checks, and review retained results without moving
|
||||||
|
domain-specific harness logic into the core.
|
||||||
|
|
||||||
|
`open-cmis-tck` is the first concrete example. CMIS-specific profiles, runners,
|
||||||
|
runtime dependencies, and harness behavior remain owned by that extension
|
||||||
|
repository.
|
||||||
|
|
||||||
|
## Acceptance Stages
|
||||||
|
|
||||||
|
Run these stages from the guide-board repository.
|
||||||
|
|
||||||
|
1. Confirm the extension repository shape:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
test -f ../open-cmis-tck/extension.json
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Validate extension discovery and manifest contracts:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
PYTHONPATH=src python3 -m guide_board \
|
||||||
|
--extension-dir ../open-cmis-tck \
|
||||||
|
extensions validate
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Validate the target and assessment profiles:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
PYTHONPATH=src python3 -m guide_board \
|
||||||
|
--extension-dir ../open-cmis-tck \
|
||||||
|
profile validate-target \
|
||||||
|
../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json
|
||||||
|
|
||||||
|
PYTHONPATH=src python3 -m guide_board \
|
||||||
|
--extension-dir ../open-cmis-tck \
|
||||||
|
profile validate-assessment \
|
||||||
|
../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Generate and retain a run plan:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
mkdir -p /tmp/guide-board-external-extension-acceptance
|
||||||
|
PYTHONPATH=src python3 -m guide_board \
|
||||||
|
--extension-dir ../open-cmis-tck \
|
||||||
|
plan \
|
||||||
|
--target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
|
||||||
|
--assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
|
||||||
|
--output /tmp/guide-board-external-extension-acceptance/plan.json
|
||||||
|
```
|
||||||
|
|
||||||
|
5. Execute a live run when the candidate endpoint and extension runtime
|
||||||
|
dependencies are available:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
PYTHONPATH=src python3 -m guide_board \
|
||||||
|
--extension-dir ../open-cmis-tck \
|
||||||
|
run \
|
||||||
|
--target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
|
||||||
|
--assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
|
||||||
|
--output-dir /tmp/guide-board-external-extension-acceptance/open-cmis-tck-baseline
|
||||||
|
```
|
||||||
|
|
||||||
|
6. Review retained results:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
PYTHONPATH=src python3 -m guide_board runs report \
|
||||||
|
--runs-dir /tmp/guide-board-external-extension-acceptance \
|
||||||
|
--target kontextual-cmis-compat \
|
||||||
|
--assessment cmis-browser-baseline
|
||||||
|
```
|
||||||
|
|
||||||
|
The run directory should contain:
|
||||||
|
|
||||||
|
```text
|
||||||
|
run.json
|
||||||
|
plan.json
|
||||||
|
retention-summary.json
|
||||||
|
normalized/evidence.json
|
||||||
|
normalized/findings.json
|
||||||
|
normalized/mappings.json
|
||||||
|
reports/assessment-package.json
|
||||||
|
reports/report.md
|
||||||
|
artifacts/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Scripted Acceptance
|
||||||
|
|
||||||
|
The scripted path keeps the same commands in one place:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
scripts/external_extension_acceptance.sh plan
|
||||||
|
```
|
||||||
|
|
||||||
|
The default `plan` mode validates the extension and profiles, then writes a run
|
||||||
|
plan under `/tmp/guide-board-external-extension-acceptance`. It does not contact
|
||||||
|
the candidate system.
|
||||||
|
|
||||||
|
Use `run` mode for the live acceptance after the candidate endpoint and harness
|
||||||
|
dependencies are ready:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
scripts/external_extension_acceptance.sh run
|
||||||
|
```
|
||||||
|
|
||||||
|
Override the extension, profiles, or output location when accepting another
|
||||||
|
extension:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GUIDE_BOARD_ACCEPT_EXTENSION_DIR=../example-extension \
|
||||||
|
GUIDE_BOARD_ACCEPT_TARGET=../example-extension/profiles/targets/example.json \
|
||||||
|
GUIDE_BOARD_ACCEPT_ASSESSMENT=../example-extension/profiles/assessments/example.json \
|
||||||
|
GUIDE_BOARD_ACCEPT_OUTPUT_DIR=/tmp/guide-board-example-extension \
|
||||||
|
scripts/external_extension_acceptance.sh plan
|
||||||
|
```
|
||||||
|
|
||||||
|
## Ownership Boundary
|
||||||
|
|
||||||
|
Guide-board owns:
|
||||||
|
|
||||||
|
- extension discovery,
|
||||||
|
- extension manifest validation,
|
||||||
|
- target and assessment profile validation,
|
||||||
|
- run plan construction,
|
||||||
|
- run artifact layout,
|
||||||
|
- retained run summary and report lookup.
|
||||||
|
|
||||||
|
The external extension repository owns:
|
||||||
|
|
||||||
|
- domain-specific target and assessment profiles,
|
||||||
|
- harness adapters and runtime dependencies,
|
||||||
|
- credentials or restricted asset instructions,
|
||||||
|
- check group behavior,
|
||||||
|
- interpretation of domain-specific evidence.
|
||||||
@@ -43,7 +43,21 @@
|
|||||||
},
|
},
|
||||||
"supported_frameworks": { "type": "array", "items": { "type": "string" } },
|
"supported_frameworks": { "type": "array", "items": { "type": "string" } },
|
||||||
"authorities": { "type": "array", "items": { "type": "string" } },
|
"authorities": { "type": "array", "items": { "type": "string" } },
|
||||||
"profile_schemas": { "type": "array", "items": { "type": "string" } },
|
"profile_schemas": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": ["string", "object"],
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["id", "profile_kind", "path"],
|
||||||
|
"properties": {
|
||||||
|
"id": { "type": "string" },
|
||||||
|
"profile_kind": { "type": "string", "enum": ["target", "assessment"] },
|
||||||
|
"path": { "type": "string" },
|
||||||
|
"subject_type": { "type": ["string", "null"] },
|
||||||
|
"description": { "type": ["string", "null"] }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
"check_groups": {
|
"check_groups": {
|
||||||
"type": "array",
|
"type": "array",
|
||||||
"items": {
|
"items": {
|
||||||
|
|||||||
106
scripts/external_extension_acceptance.sh
Executable file
106
scripts/external_extension_acceptance.sh
Executable file
@@ -0,0 +1,106 @@
|
|||||||
|
#!/usr/bin/env sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
ROOT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)"
|
||||||
|
MODE="${1:-plan}"
|
||||||
|
EXTENSION_DIR="${GUIDE_BOARD_ACCEPT_EXTENSION_DIR:-../open-cmis-tck}"
|
||||||
|
TARGET="${GUIDE_BOARD_ACCEPT_TARGET:-../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json}"
|
||||||
|
ASSESSMENT="${GUIDE_BOARD_ACCEPT_ASSESSMENT:-../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json}"
|
||||||
|
OUTPUT_DIR="${GUIDE_BOARD_ACCEPT_OUTPUT_DIR:-${TMPDIR:-/tmp}/guide-board-external-extension-acceptance}"
|
||||||
|
PLAN_OUTPUT="${GUIDE_BOARD_ACCEPT_PLAN_OUTPUT:-$OUTPUT_DIR/plan.json}"
|
||||||
|
RUN_DIR="${GUIDE_BOARD_ACCEPT_RUN_DIR:-$OUTPUT_DIR/open-cmis-tck-baseline}"
|
||||||
|
PYTHON="${PYTHON:-python3}"
|
||||||
|
|
||||||
|
case "$MODE" in
|
||||||
|
plan|run)
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 [plan|run]" >&2
|
||||||
|
exit 64
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
cd "$ROOT_DIR"
|
||||||
|
export PYTHONPATH="$ROOT_DIR/src${PYTHONPATH:+:$PYTHONPATH}"
|
||||||
|
|
||||||
|
if [ ! -f "$EXTENSION_DIR/extension.json" ]; then
|
||||||
|
echo "ERROR: extension manifest not found at $EXTENSION_DIR/extension.json" >&2
|
||||||
|
exit 66
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$OUTPUT_DIR"
|
||||||
|
|
||||||
|
echo "==> Listing discovered extensions"
|
||||||
|
"$PYTHON" -m guide_board \
|
||||||
|
--root "$ROOT_DIR" \
|
||||||
|
--extension-dir "$EXTENSION_DIR" \
|
||||||
|
extensions list
|
||||||
|
|
||||||
|
echo "==> Validating extension manifests"
|
||||||
|
"$PYTHON" -m guide_board \
|
||||||
|
--root "$ROOT_DIR" \
|
||||||
|
--extension-dir "$EXTENSION_DIR" \
|
||||||
|
extensions validate
|
||||||
|
|
||||||
|
echo "==> Validating target profile"
|
||||||
|
"$PYTHON" -m guide_board \
|
||||||
|
--root "$ROOT_DIR" \
|
||||||
|
--extension-dir "$EXTENSION_DIR" \
|
||||||
|
profile validate-target "$TARGET"
|
||||||
|
|
||||||
|
echo "==> Validating assessment profile"
|
||||||
|
"$PYTHON" -m guide_board \
|
||||||
|
--root "$ROOT_DIR" \
|
||||||
|
--extension-dir "$EXTENSION_DIR" \
|
||||||
|
profile validate-assessment "$ASSESSMENT"
|
||||||
|
|
||||||
|
echo "==> Generating run plan"
|
||||||
|
"$PYTHON" -m guide_board \
|
||||||
|
--root "$ROOT_DIR" \
|
||||||
|
--extension-dir "$EXTENSION_DIR" \
|
||||||
|
plan \
|
||||||
|
--target "$TARGET" \
|
||||||
|
--assessment "$ASSESSMENT" \
|
||||||
|
--output "$PLAN_OUTPUT"
|
||||||
|
|
||||||
|
if [ "$MODE" = "plan" ]; then
|
||||||
|
echo "External extension acceptance plan passed."
|
||||||
|
echo "Plan output: $PLAN_OUTPUT"
|
||||||
|
echo "Run live acceptance with: $0 run"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Running live assessment"
|
||||||
|
"$PYTHON" -m guide_board \
|
||||||
|
--root "$ROOT_DIR" \
|
||||||
|
--extension-dir "$EXTENSION_DIR" \
|
||||||
|
run \
|
||||||
|
--target "$TARGET" \
|
||||||
|
--assessment "$ASSESSMENT" \
|
||||||
|
--output-dir "$RUN_DIR"
|
||||||
|
|
||||||
|
echo "==> Verifying retained run artifacts"
|
||||||
|
for path in \
|
||||||
|
"$RUN_DIR/run.json" \
|
||||||
|
"$RUN_DIR/plan.json" \
|
||||||
|
"$RUN_DIR/retention-summary.json" \
|
||||||
|
"$RUN_DIR/normalized/evidence.json" \
|
||||||
|
"$RUN_DIR/normalized/findings.json" \
|
||||||
|
"$RUN_DIR/normalized/mappings.json" \
|
||||||
|
"$RUN_DIR/reports/assessment-package.json" \
|
||||||
|
"$RUN_DIR/reports/report.md"
|
||||||
|
do
|
||||||
|
if [ ! -f "$path" ]; then
|
||||||
|
echo "ERROR: expected artifact missing: $path" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "==> Reviewing retained report paths"
|
||||||
|
"$PYTHON" -m guide_board \
|
||||||
|
--root "$ROOT_DIR" \
|
||||||
|
runs report \
|
||||||
|
--runs-dir "$OUTPUT_DIR"
|
||||||
|
|
||||||
|
echo "External extension live acceptance passed."
|
||||||
|
echo "Run artifacts: $RUN_DIR"
|
||||||
@@ -155,12 +155,18 @@ def cmd_extensions_validate(args: argparse.Namespace) -> dict[str, Any]:
|
|||||||
|
|
||||||
|
|
||||||
def cmd_validate_target(args: argparse.Namespace) -> dict[str, Any]:
|
def cmd_validate_target(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
profile = validate_target_profile(args.path)
|
profile = validate_target_profile(
|
||||||
|
args.path,
|
||||||
|
discover_extensions(args.root, args.extension_dir),
|
||||||
|
)
|
||||||
return {"status": "valid", "target_profile": profile["id"]}
|
return {"status": "valid", "target_profile": profile["id"]}
|
||||||
|
|
||||||
|
|
||||||
def cmd_validate_assessment(args: argparse.Namespace) -> dict[str, Any]:
|
def cmd_validate_assessment(args: argparse.Namespace) -> dict[str, Any]:
|
||||||
profile = validate_assessment_profile(args.path)
|
profile = validate_assessment_profile(
|
||||||
|
args.path,
|
||||||
|
discover_extensions(args.root, args.extension_dir),
|
||||||
|
)
|
||||||
return {"status": "valid", "assessment_profile": profile["id"]}
|
return {"status": "valid", "assessment_profile": profile["id"]}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -6,21 +6,32 @@ from datetime import datetime, timezone
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from guide_board.discovery import discover_extensions
|
from guide_board.discovery import Extension, discover_extensions
|
||||||
from guide_board.errors import ValidationError
|
from guide_board.errors import ValidationError
|
||||||
from guide_board.io import load_json
|
from guide_board.io import load_json
|
||||||
from guide_board.schema import assert_valid
|
from guide_board.schema import assert_valid, validate_document
|
||||||
|
|
||||||
|
|
||||||
def validate_target_profile(path: Path) -> dict[str, Any]:
|
def validate_target_profile(
|
||||||
|
path: Path,
|
||||||
|
extensions: list[Extension] | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
document = load_json(path)
|
document = load_json(path)
|
||||||
assert_valid(document, "target-profile")
|
assert_valid(document, "target-profile")
|
||||||
|
if extensions:
|
||||||
|
_validate_extension_profile_schemas(document, "target", extensions)
|
||||||
return document
|
return document
|
||||||
|
|
||||||
|
|
||||||
def validate_assessment_profile(path: Path) -> dict[str, Any]:
|
def validate_assessment_profile(
|
||||||
|
path: Path,
|
||||||
|
extensions: list[Extension] | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
document = load_json(path)
|
document = load_json(path)
|
||||||
assert_valid(document, "assessment-profile")
|
assert_valid(document, "assessment-profile")
|
||||||
|
if extensions:
|
||||||
|
selected_extensions = _selected_assessment_extensions(document, extensions)
|
||||||
|
_validate_extension_profile_schemas(document, "assessment", selected_extensions)
|
||||||
return document
|
return document
|
||||||
|
|
||||||
|
|
||||||
@@ -41,6 +52,10 @@ def build_run_plan(
|
|||||||
missing = [extension_id for extension_id in selected_extensions if extension_id not in extensions]
|
missing = [extension_id for extension_id in selected_extensions if extension_id not in extensions]
|
||||||
if missing:
|
if missing:
|
||||||
raise ValidationError(f"assessment references unknown extension(s): {', '.join(missing)}")
|
raise ValidationError(f"assessment references unknown extension(s): {', '.join(missing)}")
|
||||||
|
selected_extension_records = [extensions[extension_id] for extension_id in selected_extensions]
|
||||||
|
|
||||||
|
_validate_extension_profile_schemas(target, "target", selected_extension_records)
|
||||||
|
_validate_extension_profile_schemas(assessment, "assessment", selected_extension_records)
|
||||||
|
|
||||||
if assessment["target_profile_ref"] != target["id"]:
|
if assessment["target_profile_ref"] != target["id"]:
|
||||||
raise ValidationError(
|
raise ValidationError(
|
||||||
@@ -119,6 +134,80 @@ def _credential_refs(target: dict[str, Any]) -> list[str]:
|
|||||||
return []
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _selected_assessment_extensions(
|
||||||
|
assessment: dict[str, Any],
|
||||||
|
extensions: list[Extension],
|
||||||
|
) -> list[Extension]:
|
||||||
|
by_id = {extension.id: extension for extension in extensions}
|
||||||
|
selected_ids = assessment.get("extension_refs", [])
|
||||||
|
selected_extensions = []
|
||||||
|
missing = []
|
||||||
|
for extension_id in selected_ids:
|
||||||
|
if extension_id in by_id:
|
||||||
|
selected_extensions.append(by_id[extension_id])
|
||||||
|
else:
|
||||||
|
missing.append(extension_id)
|
||||||
|
if missing:
|
||||||
|
raise ValidationError(f"assessment references unknown extension(s): {', '.join(missing)}")
|
||||||
|
return selected_extensions
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_extension_profile_schemas(
|
||||||
|
profile: dict[str, Any],
|
||||||
|
profile_kind: str,
|
||||||
|
extensions: list[Extension],
|
||||||
|
) -> None:
|
||||||
|
for extension in extensions:
|
||||||
|
for descriptor in _profile_schema_descriptors(extension, profile_kind, profile):
|
||||||
|
schema = _load_extension_profile_schema(extension, descriptor)
|
||||||
|
errors = validate_document(profile, schema)
|
||||||
|
if errors:
|
||||||
|
formatted = "\n".join(f"- {error}" for error in errors)
|
||||||
|
raise ValidationError(
|
||||||
|
f"{extension.id}:{descriptor['id']} profile schema validation failed:\n"
|
||||||
|
f"{formatted}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _profile_schema_descriptors(
|
||||||
|
extension: Extension,
|
||||||
|
profile_kind: str,
|
||||||
|
profile: dict[str, Any],
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
descriptors = []
|
||||||
|
for raw_descriptor in extension.manifest.get("profile_schemas", []):
|
||||||
|
if not isinstance(raw_descriptor, dict):
|
||||||
|
continue
|
||||||
|
if raw_descriptor.get("profile_kind") != profile_kind:
|
||||||
|
continue
|
||||||
|
subject_type = raw_descriptor.get("subject_type")
|
||||||
|
if profile_kind == "target" and subject_type and subject_type != profile.get("subject_type"):
|
||||||
|
continue
|
||||||
|
descriptors.append(raw_descriptor)
|
||||||
|
return descriptors
|
||||||
|
|
||||||
|
|
||||||
|
def _load_extension_profile_schema(
|
||||||
|
extension: Extension,
|
||||||
|
descriptor: dict[str, Any],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
raw_path = descriptor["path"]
|
||||||
|
schema_path = (extension.path / raw_path).resolve()
|
||||||
|
extension_root = extension.path.resolve()
|
||||||
|
try:
|
||||||
|
schema_path.relative_to(extension_root)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ValidationError(
|
||||||
|
f"{extension.id}:{descriptor['id']} profile schema path escapes extension root: "
|
||||||
|
f"{raw_path!r}"
|
||||||
|
) from exc
|
||||||
|
if not schema_path.is_file():
|
||||||
|
raise ValidationError(
|
||||||
|
f"{extension.id}:{descriptor['id']} profile schema not found: {raw_path!r}"
|
||||||
|
)
|
||||||
|
return load_json(schema_path)
|
||||||
|
|
||||||
|
|
||||||
def _extension_path_ref(root: Path, path: Path) -> str:
|
def _extension_path_ref(root: Path, path: Path) -> str:
|
||||||
try:
|
try:
|
||||||
return str(path.resolve().relative_to(root.resolve()))
|
return str(path.resolve().relative_to(root.resolve()))
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ from tempfile import TemporaryDirectory
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from guide_board.discovery import discover_extensions
|
from guide_board.discovery import discover_extensions
|
||||||
|
from guide_board.errors import ValidationError
|
||||||
from guide_board.execution import run_assessment
|
from guide_board.execution import run_assessment
|
||||||
from guide_board.gates import evaluate_trend_gates
|
from guide_board.gates import evaluate_trend_gates
|
||||||
from guide_board.io import load_json
|
from guide_board.io import load_json
|
||||||
@@ -143,6 +144,55 @@ class CoreArchitectureTests(unittest.TestCase):
|
|||||||
self.assertEqual(plan["extension_snapshots"][0]["path"], str(extension_dir))
|
self.assertEqual(plan["extension_snapshots"][0]["path"], str(extension_dir))
|
||||||
self.assertEqual([item["result"] for item in evidence], ["skipped", "manual"])
|
self.assertEqual([item["result"] for item in evidence], ["skipped", "manual"])
|
||||||
|
|
||||||
|
def test_applies_external_extension_profile_schemas(self) -> None:
|
||||||
|
with TemporaryDirectory() as temporary_directory:
|
||||||
|
temp_root = Path(temporary_directory)
|
||||||
|
extension_dir = temp_root / "schema-noop"
|
||||||
|
_write_schema_extension(extension_dir)
|
||||||
|
extensions = discover_extensions(ROOT, [extension_dir])
|
||||||
|
target_path = temp_root / "target.json"
|
||||||
|
assessment_path = temp_root / "assessment.json"
|
||||||
|
_write_schema_target(target_path, endpoints=[{
|
||||||
|
"id": "api",
|
||||||
|
"url": "http://127.0.0.1:8080",
|
||||||
|
"binding": "example",
|
||||||
|
}])
|
||||||
|
_write_schema_assessment(assessment_path, runtime_policy={"offline": True})
|
||||||
|
|
||||||
|
target = validate_target_profile(target_path, extensions)
|
||||||
|
assessment = validate_assessment_profile(assessment_path, extensions)
|
||||||
|
plan = build_run_plan(ROOT, target_path, assessment_path, [extension_dir])
|
||||||
|
|
||||||
|
self.assertEqual(target["subject_type"], "schema-subject")
|
||||||
|
self.assertEqual(assessment["runtime_policy"], {"offline": True})
|
||||||
|
self.assertEqual(plan["extension_snapshots"][0]["id"], "schema-noop")
|
||||||
|
|
||||||
|
_write_schema_target(target_path, endpoints=[])
|
||||||
|
with self.assertRaisesRegex(
|
||||||
|
ValidationError,
|
||||||
|
"schema-noop:schema-target profile schema validation failed",
|
||||||
|
):
|
||||||
|
validate_target_profile(target_path, extensions)
|
||||||
|
|
||||||
|
def test_rejects_extension_profile_schema_paths_outside_extension_root(self) -> None:
|
||||||
|
with TemporaryDirectory() as temporary_directory:
|
||||||
|
temp_root = Path(temporary_directory)
|
||||||
|
extension_dir = temp_root / "schema-noop"
|
||||||
|
_write_schema_extension(extension_dir, target_schema_path="../outside.schema.json")
|
||||||
|
target_path = temp_root / "target.json"
|
||||||
|
_write_schema_target(target_path, endpoints=[{
|
||||||
|
"id": "api",
|
||||||
|
"url": "http://127.0.0.1:8080",
|
||||||
|
"binding": "example",
|
||||||
|
}])
|
||||||
|
|
||||||
|
extensions = discover_extensions(ROOT, [extension_dir])
|
||||||
|
with self.assertRaisesRegex(
|
||||||
|
ValidationError,
|
||||||
|
"profile schema path escapes extension root",
|
||||||
|
):
|
||||||
|
validate_target_profile(target_path, extensions)
|
||||||
|
|
||||||
def test_runs_sample_noop_assessment(self) -> None:
|
def test_runs_sample_noop_assessment(self) -> None:
|
||||||
with TemporaryDirectory() as temporary_directory:
|
with TemporaryDirectory() as temporary_directory:
|
||||||
result = run_assessment(
|
result = run_assessment(
|
||||||
@@ -460,5 +510,135 @@ def _write_external_extension(extension_dir: Path) -> None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_schema_extension(
|
||||||
|
extension_dir: Path,
|
||||||
|
target_schema_path: str = "schemas/schema-target.schema.json",
|
||||||
|
) -> None:
|
||||||
|
extension_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
schema_dir = extension_dir / "schemas"
|
||||||
|
schema_dir.mkdir()
|
||||||
|
(schema_dir / "schema-target.schema.json").write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"type": "object",
|
||||||
|
"required": ["subject_type", "endpoints"],
|
||||||
|
"properties": {
|
||||||
|
"subject_type": {"enum": ["schema-subject"]},
|
||||||
|
"endpoints": {"type": "array", "minItems": 1},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(schema_dir / "schema-assessment.schema.json").write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"type": "object",
|
||||||
|
"required": ["runtime_policy"],
|
||||||
|
"properties": {
|
||||||
|
"runtime_policy": {
|
||||||
|
"type": "object",
|
||||||
|
"required": ["offline"],
|
||||||
|
"properties": {"offline": {"type": "boolean"}},
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(extension_dir / "extension.json").write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"id": "schema-noop",
|
||||||
|
"name": "Schema No-op",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"extension_type": "repository_quality",
|
||||||
|
"lifecycle_status": "incubating",
|
||||||
|
"supported_frameworks": ["schema.readiness.v1"],
|
||||||
|
"authorities": [],
|
||||||
|
"profile_schemas": [
|
||||||
|
"target-profile",
|
||||||
|
"assessment-profile",
|
||||||
|
{
|
||||||
|
"id": "schema-target",
|
||||||
|
"profile_kind": "target",
|
||||||
|
"path": target_schema_path,
|
||||||
|
"subject_type": "schema-subject",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "schema-assessment",
|
||||||
|
"profile_kind": "assessment",
|
||||||
|
"path": "schemas/schema-assessment.schema.json",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"check_groups": [
|
||||||
|
{
|
||||||
|
"id": "shape",
|
||||||
|
"name": "Shape",
|
||||||
|
"check_type": "repository_quality",
|
||||||
|
"requirement_refs": ["schema.shape"],
|
||||||
|
"runner_ref": None,
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"preflight_runner": None,
|
||||||
|
"runner_entrypoints": [],
|
||||||
|
"normalizers": [],
|
||||||
|
"mappings": [],
|
||||||
|
"report_fragments": [],
|
||||||
|
"dependencies": [],
|
||||||
|
"restricted_assets": [],
|
||||||
|
"certification_boundary": "Test fixture only.",
|
||||||
|
}
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_schema_target(path: Path, endpoints: list[dict[str, str]]) -> None:
|
||||||
|
path.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"id": "schema-target",
|
||||||
|
"subject_type": "schema-subject",
|
||||||
|
"subject_name": "Schema Target",
|
||||||
|
"environment": "test",
|
||||||
|
"scope": ["schema"],
|
||||||
|
"endpoints": endpoints,
|
||||||
|
"artifacts": [],
|
||||||
|
"credentials_ref": None,
|
||||||
|
"declared_capabilities": [],
|
||||||
|
"known_gaps": [],
|
||||||
|
}
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _write_schema_assessment(path: Path, runtime_policy: dict[str, object]) -> None:
|
||||||
|
path.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"id": "schema-assessment",
|
||||||
|
"framework_refs": ["schema.readiness.v1"],
|
||||||
|
"extension_refs": ["schema-noop"],
|
||||||
|
"target_profile_ref": "schema-target",
|
||||||
|
"selected_check_groups": {"schema-noop": ["shape"]},
|
||||||
|
"expectations_ref": None,
|
||||||
|
"waivers_ref": None,
|
||||||
|
"output_policy": {
|
||||||
|
"report_formats": ["json", "markdown"],
|
||||||
|
"artifact_retention": "summary-only",
|
||||||
|
},
|
||||||
|
"retention_policy": {
|
||||||
|
"summary_days": 365,
|
||||||
|
"raw_artifact_days": 0,
|
||||||
|
},
|
||||||
|
"runtime_policy": runtime_policy,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ type: workplan
|
|||||||
title: "Assessment Operations Baseline"
|
title: "Assessment Operations Baseline"
|
||||||
repo: guide-board
|
repo: guide-board
|
||||||
domain: markitect
|
domain: markitect
|
||||||
status: active
|
status: completed
|
||||||
owner: codex
|
owner: codex
|
||||||
planning_priority: high
|
planning_priority: high
|
||||||
planning_order: 2
|
planning_order: 2
|
||||||
@@ -174,7 +174,7 @@ Progress:
|
|||||||
|
|
||||||
```task
|
```task
|
||||||
id: GUIDE-BOARD-WP-0002-T006
|
id: GUIDE-BOARD-WP-0002-T006
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "65fbf1df-caef-40f6-abee-8308daf27fbc"
|
state_hub_task_id: "65fbf1df-caef-40f6-abee-8308daf27fbc"
|
||||||
```
|
```
|
||||||
@@ -187,6 +187,13 @@ Acceptance:
|
|||||||
- Cover extension validation, target/assessment profile validation, plan
|
- Cover extension validation, target/assessment profile validation, plan
|
||||||
generation, run execution, and result review.
|
generation, run execution, and result review.
|
||||||
|
|
||||||
|
Progress:
|
||||||
|
|
||||||
|
- Added `docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
|
||||||
|
- Added `scripts/external_extension_acceptance.sh`.
|
||||||
|
- Verified `open-cmis-tck` extension validation, profile validation, and plan
|
||||||
|
generation in scripted plan mode.
|
||||||
|
|
||||||
## Definition Of Done
|
## Definition Of Done
|
||||||
|
|
||||||
- The repo has a clear assessment operations guide.
|
- The repo has a clear assessment operations guide.
|
||||||
|
|||||||
123
workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md
Normal file
123
workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md
Normal file
@@ -0,0 +1,123 @@
|
|||||||
|
---
|
||||||
|
id: GUIDE-BOARD-WP-0003
|
||||||
|
type: workplan
|
||||||
|
title: "Extension SDK Maturity"
|
||||||
|
repo: guide-board
|
||||||
|
domain: markitect
|
||||||
|
status: active
|
||||||
|
owner: codex
|
||||||
|
planning_priority: high
|
||||||
|
planning_order: 3
|
||||||
|
created: "2026-05-15"
|
||||||
|
updated: "2026-05-15"
|
||||||
|
state_hub_workstream_id: "26aa9511-cd5c-4dd5-989c-d2838ba3b50d"
|
||||||
|
---
|
||||||
|
|
||||||
|
# GUIDE-BOARD-WP-0003: Extension SDK Maturity
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Harden the external extension SDK now that guide-board has a repeatable
|
||||||
|
assessment operations baseline. External extension repositories should be able
|
||||||
|
to declare their own validation surfaces, normalization boundaries, and
|
||||||
|
acceptance fixtures without pushing domain logic back into the core.
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
`GUIDE-BOARD-WP-0002` made assessment operation repeatable for CLI, service,
|
||||||
|
container, candidate handoff, retained results, and external extension
|
||||||
|
acceptance. The next repo-level gap is SDK maturity: the core can discover
|
||||||
|
external extensions and run their entrypoints, but extension-owned validation
|
||||||
|
and normalizer contracts are still mostly prose.
|
||||||
|
|
||||||
|
## Boundary
|
||||||
|
|
||||||
|
This workplan owns extension-neutral SDK contracts and core enforcement points.
|
||||||
|
Domain-specific schemas, CMIS runner behavior, harness dependencies, and
|
||||||
|
certification interpretations remain owned by external extension repositories.
|
||||||
|
|
||||||
|
## D3.1 - Extension-Owned Profile Schema Validation
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0003-T001
|
||||||
|
status: done
|
||||||
|
priority: high
|
||||||
|
state_hub_task_id: "1bc729ec-683c-410e-8b47-1b13eb61da00"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Allow extension manifests to declare profile schema descriptors without
|
||||||
|
breaking the existing string shorthand.
|
||||||
|
- Validate extension-owned target and assessment profile schemas during CLI
|
||||||
|
profile validation and run planning.
|
||||||
|
- Keep extension schemas loaded from the extension root and reject schema paths
|
||||||
|
that escape that root.
|
||||||
|
- Add focused tests and SDK documentation.
|
||||||
|
|
||||||
|
Progress:
|
||||||
|
|
||||||
|
- Extended `profile_schemas` to support descriptor objects while preserving the
|
||||||
|
existing string shorthand.
|
||||||
|
- Applied extension-owned target and assessment schema validation in CLI profile
|
||||||
|
validation and run planning.
|
||||||
|
- Added tests for successful extension-owned validation, validation failure, and
|
||||||
|
schema-path root containment.
|
||||||
|
- Documented the descriptor contract in `docs/EXTENSION-SDK.md`.
|
||||||
|
|
||||||
|
## D3.2 - Normalizer Plug-in Contract
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0003-T002
|
||||||
|
status: todo
|
||||||
|
priority: high
|
||||||
|
state_hub_task_id: "b87e68c1-6eca-4274-8e3f-6e2854c5a1e1"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Define how extension normalizers are declared, loaded, and invoked.
|
||||||
|
- Preserve the current runner-result contract while allowing an extension to
|
||||||
|
normalize native result artifacts explicitly.
|
||||||
|
- Add tests that prove a normalizer can map native output into evidence.
|
||||||
|
|
||||||
|
## D3.3 - SDK Fixture Extension And Acceptance Tests
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0003-T003
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
state_hub_task_id: "f3738751-5a0d-4eaf-85b1-75e599a78060"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Add a compact SDK fixture extension that exercises the mature contracts.
|
||||||
|
- Keep the fixture dependency-light and suitable for unit tests.
|
||||||
|
- Cover external repo discovery, schema validation, normalizer invocation, plan
|
||||||
|
generation, and result package shape.
|
||||||
|
|
||||||
|
## D3.4 - Extension Authoring Documentation Refresh
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0003-T004
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
state_hub_task_id: "3d390bd4-755b-462a-9e16-9c859990d99e"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Refresh `docs/EXTENSION-SDK.md` with the finalized profile-schema and
|
||||||
|
normalizer contracts.
|
||||||
|
- Update templates or examples so extension authors can copy working shapes.
|
||||||
|
- Link the SDK maturity guidance from the assessment operations and external
|
||||||
|
extension acceptance docs where useful.
|
||||||
|
|
||||||
|
## Definition Of Done
|
||||||
|
|
||||||
|
- External extension repositories can declare and test domain-specific profile
|
||||||
|
validation without core code changes.
|
||||||
|
- Normalizer plug-ins have a documented and tested core contract.
|
||||||
|
- The SDK includes a small fixture path that future extension work can reuse.
|
||||||
|
- Operator docs and authoring docs agree on the supported extension lifecycle.
|
||||||
Reference in New Issue
Block a user