generated from coulomb/repo-seed
Complete extension SDK maturity
This commit is contained in:
@@ -1,8 +1,8 @@
|
||||
<!-- custodian-brief: generated by fix-consistency — do not edit manually -->
|
||||
# Custodian Brief — guide-board
|
||||
|
||||
**Domain:** markitect
|
||||
**Last synced:** 2026-05-15 13:30 UTC
|
||||
**Domain:** markitect
|
||||
**Last synced:** 2026-05-15 13:30 UTC
|
||||
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
|
||||
|
||||
## Active Workstreams
|
||||
|
||||
@@ -44,7 +44,9 @@ those contracts for service and container operation; see
|
||||
[docs/LOCAL-SERVICE-API.md](docs/LOCAL-SERVICE-API.md).
|
||||
|
||||
The `sample-noop` extension exercises the guide-board contracts without invoking
|
||||
an external harness. `open-cmis-tck` is the first real seed extension.
|
||||
an external harness. `sdk-fixture` demonstrates the extension SDK contracts for
|
||||
schemas, normalizers, mappings, and fixture profiles. `open-cmis-tck` is the
|
||||
first real seed extension.
|
||||
|
||||
See:
|
||||
|
||||
@@ -60,3 +62,5 @@ See:
|
||||
- [docs/SERVICE-JOB-DURABILITY.md](docs/SERVICE-JOB-DURABILITY.md)
|
||||
- [extensions/CANDIDATES.md](extensions/CANDIDATES.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0001-bootstrapping.md](workplans/GUIDE-BOARD-WP-0001-bootstrapping.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md](workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md](workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md)
|
||||
|
||||
@@ -65,6 +65,8 @@ when a wrapper script or container entrypoint should keep commands shorter.
|
||||
For the repeatable external extension acceptance path, including validation,
|
||||
planning, live execution, and retained result review, see
|
||||
`docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
|
||||
For extension-author contracts such as profile schema descriptors and
|
||||
normalizer plug-ins, see `docs/EXTENSION-SDK.md`.
|
||||
|
||||
## CLI Results
|
||||
|
||||
|
||||
@@ -75,6 +75,8 @@ The key runtime fields are:
|
||||
- `check_groups`: named groups that assessment profiles can select.
|
||||
- `preflight_runner`: optional runner ID used before selected check groups.
|
||||
- `runner_entrypoints`: concrete runner declarations.
|
||||
- `normalizers`: optional plug-ins that convert native runner output into the
|
||||
stable runner-result shape before evidence is written.
|
||||
- `mappings`: mapping set IDs under `mappings/<mapping-id>.json`.
|
||||
- `certification_boundary`: explicit statement of what the extension does not
|
||||
certify.
|
||||
@@ -283,6 +285,71 @@ or `infrastructure_error`, downstream check groups for that extension are not
|
||||
executed; they receive `blocked` evidence with `blocked_reason:
|
||||
preflight_failed`.
|
||||
|
||||
## Normalizer Plug-ins
|
||||
|
||||
Runners can keep returning guide-board-ready result objects directly. When a
|
||||
runner wraps a native harness or scanner that writes its own result format, the
|
||||
extension can add a normalizer descriptor:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "native-probe-normalizer",
|
||||
"kind": "python_module",
|
||||
"module_path": "normalizers/native_probe.py",
|
||||
"callable": "normalize",
|
||||
"runner_ref": "native-probe",
|
||||
"description": "Converts native runner output into guide-board evidence."
|
||||
}
|
||||
```
|
||||
|
||||
Normalizers are declared in `extension.json` under `normalizers`. The original
|
||||
string shorthand remains valid for descriptive-only entries, but only descriptor
|
||||
objects are loaded and invoked by the core.
|
||||
|
||||
The first supported normalizer kind is `python_module`. Its module path is
|
||||
resolved relative to the extension root and must stay inside that root. The
|
||||
callable receives one context object:
|
||||
|
||||
- `root`: guide-board core root path as a string.
|
||||
- `extension_path`: extension root path as a string.
|
||||
- `run_dir`: output run directory path as a string.
|
||||
- `run_id`: current run ID.
|
||||
- `plan`: full run plan snapshot.
|
||||
- `step`: the step being normalized.
|
||||
- `target_profile`: target profile snapshot.
|
||||
- `assessment_profile`: assessment profile snapshot.
|
||||
- `normalizer`: manifest normalizer descriptor.
|
||||
- `runner_result`: the current runner-result object.
|
||||
|
||||
A normalizer returns any subset of the runner-result fields:
|
||||
|
||||
```python
|
||||
def normalize(context: dict) -> dict:
|
||||
return {
|
||||
"result": "pass",
|
||||
"observations": ["Native result was normalized."],
|
||||
"facts": {"native_status": "ok"},
|
||||
"artifact_refs": ["artifacts/native-result.json"],
|
||||
"requirement_refs": ["framework.requirement"],
|
||||
}
|
||||
```
|
||||
|
||||
The core merges the normalizer output over the runner result:
|
||||
|
||||
- `result` replaces the previous result.
|
||||
- `observations` are appended.
|
||||
- `facts` are merged.
|
||||
- `artifact_refs` and `requirement_refs` are deduplicated.
|
||||
- `normalizer_refs` is recorded in evidence facts when any normalizer runs.
|
||||
|
||||
If a normalizer raises an exception, the step becomes
|
||||
`infrastructure_error` evidence and the run still produces its normal artifact
|
||||
set.
|
||||
|
||||
The bundled `extensions/sdk-fixture` extension is the copyable reference path
|
||||
for profile schemas, a native-output runner, a normalizer, mappings, and fixture
|
||||
profiles.
|
||||
|
||||
## Result Statuses
|
||||
|
||||
Initial statuses:
|
||||
@@ -303,11 +370,14 @@ Initial statuses:
|
||||
## Current Extension Examples
|
||||
|
||||
- `sample-noop`: no runner, used to validate the core contracts.
|
||||
- `sdk-fixture`: compact SDK fixture covering profile schemas, runner output,
|
||||
normalizer invocation, mapping, and fixture profiles.
|
||||
- `open-cmis-tck`: provides a Python CMIS Browser Binding preflight runner and
|
||||
declares the future external OpenCMIS TCK runner.
|
||||
|
||||
## Next SDK Steps
|
||||
|
||||
- Add normalizer plug-in contracts.
|
||||
- Add extension-owned schema validation for domain-specific target profile
|
||||
fields.
|
||||
- Broaden normalizer examples as real external extensions adopt native harness
|
||||
result formats.
|
||||
- Add more extension-owned schema validation examples for assessment-specific
|
||||
domain constraints.
|
||||
|
||||
@@ -15,6 +15,9 @@ domain-specific harness logic into the core.
|
||||
runtime dependencies, and harness behavior remain owned by that extension
|
||||
repository.
|
||||
|
||||
For a dependency-light SDK reference extension that can be copied into a
|
||||
temporary external repository, see `extensions/sdk-fixture`.
|
||||
|
||||
## Acceptance Stages
|
||||
|
||||
Run these stages from the guide-board repository.
|
||||
|
||||
@@ -93,7 +93,22 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"normalizers": { "type": "array", "items": { "type": "string" } },
|
||||
"normalizers": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": ["string", "object"],
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "kind", "module_path", "callable"],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"kind": { "type": "string", "enum": ["python_module"] },
|
||||
"module_path": { "type": "string" },
|
||||
"callable": { "type": "string" },
|
||||
"runner_ref": { "type": ["string", "null"] },
|
||||
"description": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
},
|
||||
"mappings": { "type": "array", "items": { "type": "string" } },
|
||||
"report_fragments": { "type": "array", "items": { "type": "string" } },
|
||||
"dependencies": { "type": "array", "items": { "type": "string" } },
|
||||
|
||||
13
extensions/sdk-fixture/INTENT.md
Normal file
13
extensions/sdk-fixture/INTENT.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# SDK Fixture Extension
|
||||
|
||||
`sdk-fixture` is a dependency-light guide-board extension used to exercise the
|
||||
extension SDK contracts. It is intentionally small and is not a real assessment
|
||||
program.
|
||||
|
||||
The fixture demonstrates:
|
||||
|
||||
- extension-owned target and assessment profile schemas,
|
||||
- a Python runner that writes native output,
|
||||
- a Python normalizer that converts native output into guide-board evidence,
|
||||
- a mapping set for normalized requirement refs,
|
||||
- copyable profiles for SDK acceptance tests.
|
||||
67
extensions/sdk-fixture/extension.json
Normal file
67
extensions/sdk-fixture/extension.json
Normal file
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"id": "sdk-fixture",
|
||||
"name": "SDK Fixture Extension",
|
||||
"version": "0.1.0",
|
||||
"extension_type": "repository_quality",
|
||||
"lifecycle_status": "incubating",
|
||||
"supported_frameworks": [
|
||||
"guide-board.sdk-fixture.v1"
|
||||
],
|
||||
"authorities": [],
|
||||
"profile_schemas": [
|
||||
"target-profile",
|
||||
"assessment-profile",
|
||||
{
|
||||
"id": "sdk-fixture-target",
|
||||
"profile_kind": "target",
|
||||
"path": "schemas/sdk-fixture-target.schema.json",
|
||||
"subject_type": "sdk-fixture-target",
|
||||
"description": "Requires the target shape used by the SDK fixture runner."
|
||||
},
|
||||
{
|
||||
"id": "sdk-fixture-assessment",
|
||||
"profile_kind": "assessment",
|
||||
"path": "schemas/sdk-fixture-assessment.schema.json",
|
||||
"description": "Requires the runtime policy used by the SDK fixture normalizer test."
|
||||
}
|
||||
],
|
||||
"check_groups": [
|
||||
{
|
||||
"id": "native-output",
|
||||
"name": "Native Output Normalization",
|
||||
"check_type": "repository_quality",
|
||||
"requirement_refs": [
|
||||
"guide-board.sdk-fixture.v1.native-output"
|
||||
],
|
||||
"runner_ref": "native-probe"
|
||||
}
|
||||
],
|
||||
"preflight_runner": null,
|
||||
"runner_entrypoints": [
|
||||
{
|
||||
"id": "native-probe",
|
||||
"kind": "python_module",
|
||||
"module_path": "runners/native_probe.py",
|
||||
"callable": "run",
|
||||
"command": null,
|
||||
"description": "Writes a tiny native result artifact for the SDK fixture normalizer."
|
||||
}
|
||||
],
|
||||
"normalizers": [
|
||||
{
|
||||
"id": "native-probe-normalizer",
|
||||
"kind": "python_module",
|
||||
"module_path": "normalizers/native_probe.py",
|
||||
"callable": "normalize",
|
||||
"runner_ref": "native-probe",
|
||||
"description": "Converts the SDK fixture native result artifact into guide-board evidence."
|
||||
}
|
||||
],
|
||||
"mappings": [
|
||||
"sdk-fixture-map"
|
||||
],
|
||||
"report_fragments": [],
|
||||
"dependencies": [],
|
||||
"restricted_assets": [],
|
||||
"certification_boundary": "SDK fixture only. It does not certify any product, process, or repository."
|
||||
}
|
||||
16
extensions/sdk-fixture/mappings/sdk-fixture-map.json
Normal file
16
extensions/sdk-fixture/mappings/sdk-fixture-map.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"id": "sdk-fixture-map",
|
||||
"extension_id": "sdk-fixture",
|
||||
"framework_refs": [
|
||||
"guide-board.sdk-fixture.v1"
|
||||
],
|
||||
"mappings": [
|
||||
{
|
||||
"requirement_ref": "guide-board.sdk-fixture.v1.native-output",
|
||||
"target_type": "sdk_contract",
|
||||
"target_id": "normalizer-plugin",
|
||||
"label": "Normalizer Plug-in Contract",
|
||||
"description": "The extension runner can emit native output that a normalizer converts into guide-board evidence."
|
||||
}
|
||||
]
|
||||
}
|
||||
28
extensions/sdk-fixture/normalizers/native_probe.py
Normal file
28
extensions/sdk-fixture/normalizers/native_probe.py
Normal file
@@ -0,0 +1,28 @@
|
||||
"""SDK fixture normalizer for native runner output."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def normalize(context: dict) -> dict:
|
||||
run_dir = Path(context["run_dir"])
|
||||
runner_result = context["runner_result"]
|
||||
artifact_ref = runner_result["facts"]["native_result_ref"]
|
||||
native_result = json.loads((run_dir / artifact_ref).read_text(encoding="utf-8"))
|
||||
native_status = native_result.get("native_status")
|
||||
result = "pass" if native_status == "ok" else "fail"
|
||||
return {
|
||||
"result": result,
|
||||
"observations": native_result.get("observations", []),
|
||||
"facts": {
|
||||
"native_status": native_status,
|
||||
"native_score": native_result.get("native_score"),
|
||||
"normalized_by": "native-probe-normalizer"
|
||||
},
|
||||
"artifact_refs": [
|
||||
artifact_ref
|
||||
],
|
||||
"requirement_refs": native_result.get("requirement_refs", []),
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "sdk-fixture-assessment",
|
||||
"framework_refs": [
|
||||
"guide-board.sdk-fixture.v1"
|
||||
],
|
||||
"extension_refs": [
|
||||
"sdk-fixture"
|
||||
],
|
||||
"target_profile_ref": "sdk-fixture-target",
|
||||
"selected_check_groups": {
|
||||
"sdk-fixture": [
|
||||
"native-output"
|
||||
]
|
||||
},
|
||||
"expectations_ref": null,
|
||||
"waivers_ref": null,
|
||||
"output_policy": {
|
||||
"report_formats": [
|
||||
"json",
|
||||
"markdown"
|
||||
],
|
||||
"artifact_retention": "raw-logs-plus-summary"
|
||||
},
|
||||
"retention_policy": {
|
||||
"summary_days": 365,
|
||||
"raw_artifact_days": 30
|
||||
},
|
||||
"runtime_policy": {
|
||||
"offline": true,
|
||||
"timeout_seconds": 30,
|
||||
"fixture_mode": "native-result"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"id": "sdk-fixture-target",
|
||||
"subject_type": "sdk-fixture-target",
|
||||
"subject_name": "SDK Fixture Target",
|
||||
"environment": "test",
|
||||
"scope": [
|
||||
"Extension SDK fixture validation"
|
||||
],
|
||||
"endpoints": [],
|
||||
"artifacts": [
|
||||
"extension.json"
|
||||
],
|
||||
"credentials_ref": null,
|
||||
"declared_capabilities": [
|
||||
"guide-board.sdk-fixture.v1.native-output"
|
||||
],
|
||||
"known_gaps": []
|
||||
}
|
||||
36
extensions/sdk-fixture/runners/native_probe.py
Normal file
36
extensions/sdk-fixture/runners/native_probe.py
Normal file
@@ -0,0 +1,36 @@
|
||||
"""SDK fixture runner that writes a native result artifact."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def run(context: dict) -> dict:
|
||||
run_dir = Path(context["run_dir"])
|
||||
artifact_path = run_dir / "artifacts" / "sdk-fixture" / "native-result.json"
|
||||
artifact_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
native_result = {
|
||||
"native_status": "ok",
|
||||
"native_score": 98,
|
||||
"observations": [
|
||||
"SDK fixture native probe completed."
|
||||
],
|
||||
"requirement_refs": [
|
||||
"guide-board.sdk-fixture.v1.native-output"
|
||||
],
|
||||
}
|
||||
artifact_path.write_text(json.dumps(native_result, indent=2, sort_keys=True), encoding="utf-8")
|
||||
artifact_ref = str(artifact_path.relative_to(run_dir))
|
||||
return {
|
||||
"result": "unknown",
|
||||
"observations": [
|
||||
"SDK fixture runner wrote native output for normalization."
|
||||
],
|
||||
"facts": {
|
||||
"native_result_ref": artifact_ref
|
||||
},
|
||||
"artifact_refs": [
|
||||
artifact_ref
|
||||
],
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "SDK Fixture Assessment Profile",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"runtime_policy"
|
||||
],
|
||||
"properties": {
|
||||
"runtime_policy": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"fixture_mode"
|
||||
],
|
||||
"properties": {
|
||||
"fixture_mode": { "enum": ["native-result"] }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "SDK Fixture Target Profile",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"subject_type",
|
||||
"artifacts",
|
||||
"declared_capabilities"
|
||||
],
|
||||
"properties": {
|
||||
"subject_type": { "enum": ["sdk-fixture-target"] },
|
||||
"artifacts": { "type": "array", "minItems": 1 },
|
||||
"declared_capabilities": { "type": "array", "minItems": 1 }
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@ from typing import Any
|
||||
from guide_board.artifacts import build_artifact_manifest
|
||||
from guide_board.io import write_json
|
||||
from guide_board.mapping import build_mapping_records, summarize_mappings
|
||||
from guide_board.normalizers import normalize_step_result
|
||||
from guide_board.planning import build_run_plan
|
||||
from guide_board.policy import apply_policy
|
||||
from guide_board.retention import build_retention_summary
|
||||
@@ -153,6 +154,7 @@ def _evidence_for_step(
|
||||
now = _now()
|
||||
runner_ref = step.get("runner_ref")
|
||||
runner_result = run_step(root, run_dir, run_id, plan, step)
|
||||
runner_result = normalize_step_result(root, run_dir, run_id, plan, step, runner_result)
|
||||
|
||||
return {
|
||||
"id": f"evidence:{step['id']}",
|
||||
@@ -167,17 +169,44 @@ def _evidence_for_step(
|
||||
"runner_ref": runner_ref,
|
||||
**runner_result["facts"],
|
||||
},
|
||||
"requirement_refs": _requirement_refs(plan, step),
|
||||
"requirement_refs": _requirement_refs(plan, step, runner_result),
|
||||
"artifact_refs": runner_result["artifact_refs"],
|
||||
"started_at": now,
|
||||
"completed_at": now,
|
||||
}
|
||||
|
||||
|
||||
def _requirement_refs(plan: dict[str, Any], step: dict[str, Any]) -> list[str]:
|
||||
def _requirement_refs(
|
||||
plan: dict[str, Any],
|
||||
step: dict[str, Any],
|
||||
runner_result: dict[str, Any] | None = None,
|
||||
) -> list[str]:
|
||||
refs = []
|
||||
if step["kind"] != "check_group":
|
||||
return _runner_requirement_refs(runner_result)
|
||||
refs.extend(step.get("requirement_refs", []))
|
||||
refs.extend(_runner_requirement_refs(runner_result))
|
||||
return _dedupe(refs)
|
||||
|
||||
|
||||
def _runner_requirement_refs(runner_result: dict[str, Any] | None) -> list[str]:
|
||||
if not runner_result:
|
||||
return []
|
||||
return list(step.get("requirement_refs", []))
|
||||
refs = runner_result.get("requirement_refs", [])
|
||||
if not isinstance(refs, list):
|
||||
return []
|
||||
return [ref for ref in refs if isinstance(ref, str)]
|
||||
|
||||
|
||||
def _dedupe(values: list[str]) -> list[str]:
|
||||
seen = set()
|
||||
deduped = []
|
||||
for value in values:
|
||||
if value in seen:
|
||||
continue
|
||||
seen.add(value)
|
||||
deduped.append(value)
|
||||
return deduped
|
||||
|
||||
|
||||
def _findings_for_evidence(run_id: str, evidence: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
|
||||
224
src/guide_board/normalizers.py
Normal file
224
src/guide_board/normalizers.py
Normal file
@@ -0,0 +1,224 @@
|
||||
"""Normalizer plug-in bridge for extension-provided runner output."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import Any
|
||||
|
||||
from guide_board.errors import ValidationError
|
||||
from guide_board.io import load_json
|
||||
|
||||
|
||||
def normalize_step_result(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
step: dict[str, Any],
|
||||
runner_result: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
"""Apply matching extension normalizers to a runner result."""
|
||||
extension = _extension_snapshot(plan, step["extension_id"])
|
||||
extension_path = _snapshot_path(root, extension)
|
||||
manifest = load_json(extension_path / "extension.json")
|
||||
result = _coerce_result(runner_result)
|
||||
applied: list[str] = []
|
||||
|
||||
for normalizer in _matching_normalizers(manifest, step):
|
||||
normalized = _run_normalizer(
|
||||
root,
|
||||
run_dir,
|
||||
run_id,
|
||||
plan,
|
||||
step,
|
||||
extension_path,
|
||||
normalizer,
|
||||
result,
|
||||
)
|
||||
if _is_normalizer_error(normalized):
|
||||
return normalized
|
||||
result = _merge_result(result, normalized)
|
||||
applied.append(normalizer["id"])
|
||||
|
||||
if applied:
|
||||
facts = dict(result.get("facts", {}))
|
||||
facts["normalizer_refs"] = applied
|
||||
result["facts"] = facts
|
||||
return result
|
||||
|
||||
|
||||
def _matching_normalizers(
|
||||
manifest: dict[str, Any],
|
||||
step: dict[str, Any],
|
||||
) -> list[dict[str, Any]]:
|
||||
matching = []
|
||||
runner_ref = step.get("runner_ref")
|
||||
for normalizer in manifest.get("normalizers", []):
|
||||
if not isinstance(normalizer, dict):
|
||||
continue
|
||||
normalizer_runner_ref = normalizer.get("runner_ref")
|
||||
if normalizer_runner_ref and normalizer_runner_ref != runner_ref:
|
||||
continue
|
||||
matching.append(normalizer)
|
||||
return matching
|
||||
|
||||
|
||||
def _run_normalizer(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
step: dict[str, Any],
|
||||
extension_path: Path,
|
||||
normalizer: dict[str, Any],
|
||||
runner_result: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
if normalizer["kind"] != "python_module":
|
||||
raise ValidationError(
|
||||
f"{normalizer['id']}: unsupported normalizer kind {normalizer['kind']!r}"
|
||||
)
|
||||
|
||||
module_path = normalizer.get("module_path")
|
||||
callable_name = normalizer.get("callable")
|
||||
if not module_path or not callable_name:
|
||||
raise ValidationError(
|
||||
f"{normalizer['id']}: python_module normalizers need module_path and callable"
|
||||
)
|
||||
|
||||
module_file = (extension_path / module_path).resolve()
|
||||
try:
|
||||
module_file.relative_to(extension_path.resolve())
|
||||
except ValueError as exc:
|
||||
raise ValidationError(
|
||||
f"{normalizer['id']}: module_path must stay inside the extension directory"
|
||||
) from exc
|
||||
|
||||
module = _load_module(module_file, normalizer["id"])
|
||||
normalizer_callable = getattr(module, callable_name, None)
|
||||
if not callable(normalizer_callable):
|
||||
raise ValidationError(f"{normalizer['id']}: callable {callable_name!r} was not found")
|
||||
|
||||
context = {
|
||||
"root": str(root),
|
||||
"run_dir": str(run_dir),
|
||||
"run_id": run_id,
|
||||
"plan": plan,
|
||||
"step": step,
|
||||
"target_profile": plan["target_profile_snapshot"],
|
||||
"assessment_profile": plan["assessment_profile_snapshot"],
|
||||
"extension_path": str(extension_path),
|
||||
"normalizer": normalizer,
|
||||
"runner_result": runner_result,
|
||||
}
|
||||
try:
|
||||
result = normalizer_callable(context)
|
||||
except Exception as exc: # noqa: BLE001 - extension failures become evidence.
|
||||
return {
|
||||
"result": "infrastructure_error",
|
||||
"observations": [
|
||||
f"Normalizer {normalizer['id']!r} failed before producing evidence: {exc}"
|
||||
],
|
||||
"facts": {
|
||||
"normalizer_ref": normalizer["id"],
|
||||
"normalizer_kind": normalizer["kind"],
|
||||
"error_type": type(exc).__name__,
|
||||
},
|
||||
"artifact_refs": runner_result.get("artifact_refs", []),
|
||||
"requirement_refs": runner_result.get("requirement_refs", []),
|
||||
}
|
||||
|
||||
if not isinstance(result, dict):
|
||||
raise ValidationError(f"{normalizer['id']}: normalizer must return an object")
|
||||
return result
|
||||
|
||||
|
||||
def _merge_result(
|
||||
base: dict[str, Any],
|
||||
update: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
merged = dict(base)
|
||||
if "result" in update:
|
||||
merged["result"] = update["result"]
|
||||
if "observations" in update:
|
||||
merged["observations"] = _string_list(base.get("observations", []))
|
||||
merged["observations"].extend(_string_list(update.get("observations", [])))
|
||||
if "facts" in update:
|
||||
facts = dict(base.get("facts", {}))
|
||||
update_facts = update.get("facts", {})
|
||||
if isinstance(update_facts, dict):
|
||||
facts.update(update_facts)
|
||||
merged["facts"] = facts
|
||||
if "artifact_refs" in update:
|
||||
merged["artifact_refs"] = _dedupe(
|
||||
_string_list(base.get("artifact_refs", []))
|
||||
+ _string_list(update.get("artifact_refs", []))
|
||||
)
|
||||
if "requirement_refs" in update:
|
||||
merged["requirement_refs"] = _dedupe(
|
||||
_string_list(base.get("requirement_refs", []))
|
||||
+ _string_list(update.get("requirement_refs", []))
|
||||
)
|
||||
return _coerce_result(merged)
|
||||
|
||||
|
||||
def _coerce_result(value: dict[str, Any]) -> dict[str, Any]:
|
||||
facts = value.get("facts", {})
|
||||
if not isinstance(facts, dict):
|
||||
facts = {}
|
||||
return {
|
||||
"result": value.get("result", "unknown"),
|
||||
"observations": _string_list(value.get("observations", [])),
|
||||
"facts": facts,
|
||||
"artifact_refs": _string_list(value.get("artifact_refs", [])),
|
||||
"requirement_refs": _string_list(value.get("requirement_refs", [])),
|
||||
}
|
||||
|
||||
|
||||
def _is_normalizer_error(result: dict[str, Any]) -> bool:
|
||||
return (
|
||||
result.get("result") == "infrastructure_error"
|
||||
and "normalizer_ref" in result.get("facts", {})
|
||||
)
|
||||
|
||||
|
||||
def _string_list(value: Any) -> list[str]:
|
||||
if not isinstance(value, list):
|
||||
return []
|
||||
return [item for item in value if isinstance(item, str)]
|
||||
|
||||
|
||||
def _dedupe(values: list[str]) -> list[str]:
|
||||
seen = set()
|
||||
deduped = []
|
||||
for value in values:
|
||||
if value in seen:
|
||||
continue
|
||||
seen.add(value)
|
||||
deduped.append(value)
|
||||
return deduped
|
||||
|
||||
|
||||
def _load_module(path: Path, normalizer_id: str) -> ModuleType:
|
||||
if not path.exists():
|
||||
raise ValidationError(f"{normalizer_id}: module not found: {path}")
|
||||
module_name = f"_guide_board_normalizer_{normalizer_id.replace('-', '_')}"
|
||||
spec = importlib.util.spec_from_file_location(module_name, path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise ValidationError(f"{normalizer_id}: unable to load module from {path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _extension_snapshot(plan: dict[str, Any], extension_id: str) -> dict[str, Any]:
|
||||
for extension in plan["extension_snapshots"]:
|
||||
if extension["id"] == extension_id:
|
||||
return extension
|
||||
raise ValidationError(f"step references unknown extension {extension_id!r}")
|
||||
|
||||
|
||||
def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path:
|
||||
path = Path(extension["path"])
|
||||
return path if path.is_absolute() else root / path
|
||||
@@ -2,6 +2,7 @@ from __future__ import annotations
|
||||
|
||||
import http.client
|
||||
import json
|
||||
import shutil
|
||||
import time
|
||||
import unittest
|
||||
from tempfile import TemporaryDirectory
|
||||
@@ -193,6 +194,52 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
):
|
||||
validate_target_profile(target_path, extensions)
|
||||
|
||||
def test_runs_sdk_fixture_from_external_extension_repo(self) -> None:
|
||||
with TemporaryDirectory() as temporary_directory:
|
||||
temp_root = Path(temporary_directory)
|
||||
extension_dir = temp_root / "sdk-fixture"
|
||||
shutil.copytree(ROOT / "extensions" / "sdk-fixture", extension_dir)
|
||||
|
||||
result = run_assessment(
|
||||
temp_root,
|
||||
extension_dir / "profiles" / "targets" / "sdk-fixture-target.json",
|
||||
extension_dir / "profiles" / "assessments" / "sdk-fixture-assessment.json",
|
||||
temp_root / "runs" / "sdk-fixture",
|
||||
[extension_dir],
|
||||
)
|
||||
run_dir = Path(result["run_dir"])
|
||||
plan = json.loads((run_dir / "plan.json").read_text(encoding="utf-8"))
|
||||
evidence = json.loads(
|
||||
(run_dir / "normalized" / "evidence.json").read_text(encoding="utf-8")
|
||||
)["evidence"]
|
||||
mappings = json.loads(
|
||||
(run_dir / "normalized" / "mappings.json").read_text(encoding="utf-8")
|
||||
)["mappings"]
|
||||
assessment_package = json.loads(
|
||||
(run_dir / "reports" / "assessment-package.json").read_text(encoding="utf-8")
|
||||
)
|
||||
|
||||
self.assertEqual(result["status"], "completed")
|
||||
self.assertEqual(plan["extension_snapshots"][0]["source"], "external")
|
||||
self.assertEqual(plan["target_profile_snapshot"]["subject_type"], "sdk-fixture-target")
|
||||
self.assertEqual([item["result"] for item in evidence], ["skipped", "pass"])
|
||||
check_evidence = evidence[1]
|
||||
self.assertEqual(
|
||||
check_evidence["facts"]["normalizer_refs"],
|
||||
["native-probe-normalizer"],
|
||||
)
|
||||
self.assertEqual(check_evidence["facts"]["native_score"], 98)
|
||||
self.assertEqual(
|
||||
check_evidence["requirement_refs"],
|
||||
["guide-board.sdk-fixture.v1.native-output"],
|
||||
)
|
||||
self.assertEqual(
|
||||
check_evidence["artifact_refs"],
|
||||
["artifacts/sdk-fixture/native-result.json"],
|
||||
)
|
||||
self.assertEqual(mappings[0]["target_id"], "normalizer-plugin")
|
||||
self.assertEqual(assessment_package["summary"], {"pass": 1, "skipped": 1})
|
||||
|
||||
def test_runs_sample_noop_assessment(self) -> None:
|
||||
with TemporaryDirectory() as temporary_directory:
|
||||
result = run_assessment(
|
||||
|
||||
@@ -4,7 +4,7 @@ type: workplan
|
||||
title: "Extension SDK Maturity"
|
||||
repo: guide-board
|
||||
domain: markitect
|
||||
status: active
|
||||
status: completed
|
||||
owner: codex
|
||||
planning_priority: high
|
||||
planning_order: 3
|
||||
@@ -69,7 +69,7 @@ Progress:
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0003-T002
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "b87e68c1-6eca-4274-8e3f-6e2854c5a1e1"
|
||||
```
|
||||
@@ -81,11 +81,22 @@ Acceptance:
|
||||
normalize native result artifacts explicitly.
|
||||
- Add tests that prove a normalizer can map native output into evidence.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `guide_board.normalizers.normalize_step_result`.
|
||||
- Extended `normalizers` manifest entries to support Python module descriptor
|
||||
objects while preserving the existing string shorthand.
|
||||
- Invoked matching normalizers after runner execution and before evidence
|
||||
writing.
|
||||
- Merged normalizer result fields over runner results and recorded
|
||||
`normalizer_refs` in evidence facts.
|
||||
- Added test coverage through the SDK fixture run.
|
||||
|
||||
## D3.3 - SDK Fixture Extension And Acceptance Tests
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0003-T003
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "f3738751-5a0d-4eaf-85b1-75e599a78060"
|
||||
```
|
||||
@@ -97,11 +108,19 @@ Acceptance:
|
||||
- Cover external repo discovery, schema validation, normalizer invocation, plan
|
||||
generation, and result package shape.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `extensions/sdk-fixture`.
|
||||
- Included extension-owned target and assessment schemas, fixture profiles, a
|
||||
native-output runner, a normalizer, and a mapping set.
|
||||
- Added a unit test that copies the fixture as an external extension repository
|
||||
and verifies plan, evidence, mapping, and assessment package output.
|
||||
|
||||
## D3.4 - Extension Authoring Documentation Refresh
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0003-T004
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "3d390bd4-755b-462a-9e16-9c859990d99e"
|
||||
```
|
||||
@@ -114,6 +133,14 @@ Acceptance:
|
||||
- Link the SDK maturity guidance from the assessment operations and external
|
||||
extension acceptance docs where useful.
|
||||
|
||||
Progress:
|
||||
|
||||
- Refreshed `docs/EXTENSION-SDK.md` with profile schema descriptors,
|
||||
normalizer descriptors, context fields, merge semantics, and fixture guidance.
|
||||
- Linked SDK authoring contracts from `docs/ASSESSMENT-OPERATIONS.md`.
|
||||
- Linked `extensions/sdk-fixture` from `docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
|
||||
- Added README references for the SDK fixture and WP3.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
- External extension repositories can declare and test domain-specific profile
|
||||
|
||||
Reference in New Issue
Block a user