diff --git a/.custodian-brief.md b/.custodian-brief.md index 7d51ac5..3b9844f 100644 --- a/.custodian-brief.md +++ b/.custodian-brief.md @@ -1,8 +1,8 @@ # Custodian Brief — guide-board -**Domain:** markitect -**Last synced:** 2026-05-15 13:30 UTC +**Domain:** markitect +**Last synced:** 2026-05-15 13:30 UTC **State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)* ## Active Workstreams diff --git a/README.md b/README.md index c0f5aaf..f9a5da3 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,9 @@ those contracts for service and container operation; see [docs/LOCAL-SERVICE-API.md](docs/LOCAL-SERVICE-API.md). The `sample-noop` extension exercises the guide-board contracts without invoking -an external harness. `open-cmis-tck` is the first real seed extension. +an external harness. `sdk-fixture` demonstrates the extension SDK contracts for +schemas, normalizers, mappings, and fixture profiles. `open-cmis-tck` is the +first real seed extension. See: @@ -60,3 +62,5 @@ See: - [docs/SERVICE-JOB-DURABILITY.md](docs/SERVICE-JOB-DURABILITY.md) - [extensions/CANDIDATES.md](extensions/CANDIDATES.md) - [workplans/GUIDE-BOARD-WP-0001-bootstrapping.md](workplans/GUIDE-BOARD-WP-0001-bootstrapping.md) +- [workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md](workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md) +- [workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md](workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md) diff --git a/docs/ASSESSMENT-OPERATIONS.md b/docs/ASSESSMENT-OPERATIONS.md index 052c07a..d9d34d5 100644 --- a/docs/ASSESSMENT-OPERATIONS.md +++ b/docs/ASSESSMENT-OPERATIONS.md @@ -65,6 +65,8 @@ when a wrapper script or container entrypoint should keep commands shorter. For the repeatable external extension acceptance path, including validation, planning, live execution, and retained result review, see `docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`. +For extension-author contracts such as profile schema descriptors and +normalizer plug-ins, see `docs/EXTENSION-SDK.md`. ## CLI Results diff --git a/docs/EXTENSION-SDK.md b/docs/EXTENSION-SDK.md index 2f5bf01..04ba25d 100644 --- a/docs/EXTENSION-SDK.md +++ b/docs/EXTENSION-SDK.md @@ -75,6 +75,8 @@ The key runtime fields are: - `check_groups`: named groups that assessment profiles can select. - `preflight_runner`: optional runner ID used before selected check groups. - `runner_entrypoints`: concrete runner declarations. +- `normalizers`: optional plug-ins that convert native runner output into the + stable runner-result shape before evidence is written. - `mappings`: mapping set IDs under `mappings/.json`. - `certification_boundary`: explicit statement of what the extension does not certify. @@ -283,6 +285,71 @@ or `infrastructure_error`, downstream check groups for that extension are not executed; they receive `blocked` evidence with `blocked_reason: preflight_failed`. +## Normalizer Plug-ins + +Runners can keep returning guide-board-ready result objects directly. When a +runner wraps a native harness or scanner that writes its own result format, the +extension can add a normalizer descriptor: + +```json +{ + "id": "native-probe-normalizer", + "kind": "python_module", + "module_path": "normalizers/native_probe.py", + "callable": "normalize", + "runner_ref": "native-probe", + "description": "Converts native runner output into guide-board evidence." +} +``` + +Normalizers are declared in `extension.json` under `normalizers`. The original +string shorthand remains valid for descriptive-only entries, but only descriptor +objects are loaded and invoked by the core. + +The first supported normalizer kind is `python_module`. Its module path is +resolved relative to the extension root and must stay inside that root. The +callable receives one context object: + +- `root`: guide-board core root path as a string. +- `extension_path`: extension root path as a string. +- `run_dir`: output run directory path as a string. +- `run_id`: current run ID. +- `plan`: full run plan snapshot. +- `step`: the step being normalized. +- `target_profile`: target profile snapshot. +- `assessment_profile`: assessment profile snapshot. +- `normalizer`: manifest normalizer descriptor. +- `runner_result`: the current runner-result object. + +A normalizer returns any subset of the runner-result fields: + +```python +def normalize(context: dict) -> dict: + return { + "result": "pass", + "observations": ["Native result was normalized."], + "facts": {"native_status": "ok"}, + "artifact_refs": ["artifacts/native-result.json"], + "requirement_refs": ["framework.requirement"], + } +``` + +The core merges the normalizer output over the runner result: + +- `result` replaces the previous result. +- `observations` are appended. +- `facts` are merged. +- `artifact_refs` and `requirement_refs` are deduplicated. +- `normalizer_refs` is recorded in evidence facts when any normalizer runs. + +If a normalizer raises an exception, the step becomes +`infrastructure_error` evidence and the run still produces its normal artifact +set. + +The bundled `extensions/sdk-fixture` extension is the copyable reference path +for profile schemas, a native-output runner, a normalizer, mappings, and fixture +profiles. + ## Result Statuses Initial statuses: @@ -303,11 +370,14 @@ Initial statuses: ## Current Extension Examples - `sample-noop`: no runner, used to validate the core contracts. +- `sdk-fixture`: compact SDK fixture covering profile schemas, runner output, + normalizer invocation, mapping, and fixture profiles. - `open-cmis-tck`: provides a Python CMIS Browser Binding preflight runner and declares the future external OpenCMIS TCK runner. ## Next SDK Steps -- Add normalizer plug-in contracts. -- Add extension-owned schema validation for domain-specific target profile - fields. +- Broaden normalizer examples as real external extensions adopt native harness + result formats. +- Add more extension-owned schema validation examples for assessment-specific + domain constraints. diff --git a/docs/EXTERNAL-EXTENSION-ACCEPTANCE.md b/docs/EXTERNAL-EXTENSION-ACCEPTANCE.md index 2a810e4..c362259 100644 --- a/docs/EXTERNAL-EXTENSION-ACCEPTANCE.md +++ b/docs/EXTERNAL-EXTENSION-ACCEPTANCE.md @@ -15,6 +15,9 @@ domain-specific harness logic into the core. runtime dependencies, and harness behavior remain owned by that extension repository. +For a dependency-light SDK reference extension that can be copied into a +temporary external repository, see `extensions/sdk-fixture`. + ## Acceptance Stages Run these stages from the guide-board repository. diff --git a/docs/schemas/extension-manifest.schema.json b/docs/schemas/extension-manifest.schema.json index 1a12ebe..9946708 100644 --- a/docs/schemas/extension-manifest.schema.json +++ b/docs/schemas/extension-manifest.schema.json @@ -93,7 +93,22 @@ } } }, - "normalizers": { "type": "array", "items": { "type": "string" } }, + "normalizers": { + "type": "array", + "items": { + "type": ["string", "object"], + "additionalProperties": false, + "required": ["id", "kind", "module_path", "callable"], + "properties": { + "id": { "type": "string" }, + "kind": { "type": "string", "enum": ["python_module"] }, + "module_path": { "type": "string" }, + "callable": { "type": "string" }, + "runner_ref": { "type": ["string", "null"] }, + "description": { "type": ["string", "null"] } + } + } + }, "mappings": { "type": "array", "items": { "type": "string" } }, "report_fragments": { "type": "array", "items": { "type": "string" } }, "dependencies": { "type": "array", "items": { "type": "string" } }, diff --git a/extensions/sdk-fixture/INTENT.md b/extensions/sdk-fixture/INTENT.md new file mode 100644 index 0000000..e9643b3 --- /dev/null +++ b/extensions/sdk-fixture/INTENT.md @@ -0,0 +1,13 @@ +# SDK Fixture Extension + +`sdk-fixture` is a dependency-light guide-board extension used to exercise the +extension SDK contracts. It is intentionally small and is not a real assessment +program. + +The fixture demonstrates: + +- extension-owned target and assessment profile schemas, +- a Python runner that writes native output, +- a Python normalizer that converts native output into guide-board evidence, +- a mapping set for normalized requirement refs, +- copyable profiles for SDK acceptance tests. diff --git a/extensions/sdk-fixture/extension.json b/extensions/sdk-fixture/extension.json new file mode 100644 index 0000000..87f0233 --- /dev/null +++ b/extensions/sdk-fixture/extension.json @@ -0,0 +1,67 @@ +{ + "id": "sdk-fixture", + "name": "SDK Fixture Extension", + "version": "0.1.0", + "extension_type": "repository_quality", + "lifecycle_status": "incubating", + "supported_frameworks": [ + "guide-board.sdk-fixture.v1" + ], + "authorities": [], + "profile_schemas": [ + "target-profile", + "assessment-profile", + { + "id": "sdk-fixture-target", + "profile_kind": "target", + "path": "schemas/sdk-fixture-target.schema.json", + "subject_type": "sdk-fixture-target", + "description": "Requires the target shape used by the SDK fixture runner." + }, + { + "id": "sdk-fixture-assessment", + "profile_kind": "assessment", + "path": "schemas/sdk-fixture-assessment.schema.json", + "description": "Requires the runtime policy used by the SDK fixture normalizer test." + } + ], + "check_groups": [ + { + "id": "native-output", + "name": "Native Output Normalization", + "check_type": "repository_quality", + "requirement_refs": [ + "guide-board.sdk-fixture.v1.native-output" + ], + "runner_ref": "native-probe" + } + ], + "preflight_runner": null, + "runner_entrypoints": [ + { + "id": "native-probe", + "kind": "python_module", + "module_path": "runners/native_probe.py", + "callable": "run", + "command": null, + "description": "Writes a tiny native result artifact for the SDK fixture normalizer." + } + ], + "normalizers": [ + { + "id": "native-probe-normalizer", + "kind": "python_module", + "module_path": "normalizers/native_probe.py", + "callable": "normalize", + "runner_ref": "native-probe", + "description": "Converts the SDK fixture native result artifact into guide-board evidence." + } + ], + "mappings": [ + "sdk-fixture-map" + ], + "report_fragments": [], + "dependencies": [], + "restricted_assets": [], + "certification_boundary": "SDK fixture only. It does not certify any product, process, or repository." +} diff --git a/extensions/sdk-fixture/mappings/sdk-fixture-map.json b/extensions/sdk-fixture/mappings/sdk-fixture-map.json new file mode 100644 index 0000000..acc3aac --- /dev/null +++ b/extensions/sdk-fixture/mappings/sdk-fixture-map.json @@ -0,0 +1,16 @@ +{ + "id": "sdk-fixture-map", + "extension_id": "sdk-fixture", + "framework_refs": [ + "guide-board.sdk-fixture.v1" + ], + "mappings": [ + { + "requirement_ref": "guide-board.sdk-fixture.v1.native-output", + "target_type": "sdk_contract", + "target_id": "normalizer-plugin", + "label": "Normalizer Plug-in Contract", + "description": "The extension runner can emit native output that a normalizer converts into guide-board evidence." + } + ] +} diff --git a/extensions/sdk-fixture/normalizers/native_probe.py b/extensions/sdk-fixture/normalizers/native_probe.py new file mode 100644 index 0000000..1a0bc9b --- /dev/null +++ b/extensions/sdk-fixture/normalizers/native_probe.py @@ -0,0 +1,28 @@ +"""SDK fixture normalizer for native runner output.""" + +from __future__ import annotations + +import json +from pathlib import Path + + +def normalize(context: dict) -> dict: + run_dir = Path(context["run_dir"]) + runner_result = context["runner_result"] + artifact_ref = runner_result["facts"]["native_result_ref"] + native_result = json.loads((run_dir / artifact_ref).read_text(encoding="utf-8")) + native_status = native_result.get("native_status") + result = "pass" if native_status == "ok" else "fail" + return { + "result": result, + "observations": native_result.get("observations", []), + "facts": { + "native_status": native_status, + "native_score": native_result.get("native_score"), + "normalized_by": "native-probe-normalizer" + }, + "artifact_refs": [ + artifact_ref + ], + "requirement_refs": native_result.get("requirement_refs", []), + } diff --git a/extensions/sdk-fixture/profiles/assessments/sdk-fixture-assessment.json b/extensions/sdk-fixture/profiles/assessments/sdk-fixture-assessment.json new file mode 100644 index 0000000..b60be77 --- /dev/null +++ b/extensions/sdk-fixture/profiles/assessments/sdk-fixture-assessment.json @@ -0,0 +1,33 @@ +{ + "id": "sdk-fixture-assessment", + "framework_refs": [ + "guide-board.sdk-fixture.v1" + ], + "extension_refs": [ + "sdk-fixture" + ], + "target_profile_ref": "sdk-fixture-target", + "selected_check_groups": { + "sdk-fixture": [ + "native-output" + ] + }, + "expectations_ref": null, + "waivers_ref": null, + "output_policy": { + "report_formats": [ + "json", + "markdown" + ], + "artifact_retention": "raw-logs-plus-summary" + }, + "retention_policy": { + "summary_days": 365, + "raw_artifact_days": 30 + }, + "runtime_policy": { + "offline": true, + "timeout_seconds": 30, + "fixture_mode": "native-result" + } +} diff --git a/extensions/sdk-fixture/profiles/targets/sdk-fixture-target.json b/extensions/sdk-fixture/profiles/targets/sdk-fixture-target.json new file mode 100644 index 0000000..855c345 --- /dev/null +++ b/extensions/sdk-fixture/profiles/targets/sdk-fixture-target.json @@ -0,0 +1,18 @@ +{ + "id": "sdk-fixture-target", + "subject_type": "sdk-fixture-target", + "subject_name": "SDK Fixture Target", + "environment": "test", + "scope": [ + "Extension SDK fixture validation" + ], + "endpoints": [], + "artifacts": [ + "extension.json" + ], + "credentials_ref": null, + "declared_capabilities": [ + "guide-board.sdk-fixture.v1.native-output" + ], + "known_gaps": [] +} diff --git a/extensions/sdk-fixture/runners/native_probe.py b/extensions/sdk-fixture/runners/native_probe.py new file mode 100644 index 0000000..f725f80 --- /dev/null +++ b/extensions/sdk-fixture/runners/native_probe.py @@ -0,0 +1,36 @@ +"""SDK fixture runner that writes a native result artifact.""" + +from __future__ import annotations + +import json +from pathlib import Path + + +def run(context: dict) -> dict: + run_dir = Path(context["run_dir"]) + artifact_path = run_dir / "artifacts" / "sdk-fixture" / "native-result.json" + artifact_path.parent.mkdir(parents=True, exist_ok=True) + native_result = { + "native_status": "ok", + "native_score": 98, + "observations": [ + "SDK fixture native probe completed." + ], + "requirement_refs": [ + "guide-board.sdk-fixture.v1.native-output" + ], + } + artifact_path.write_text(json.dumps(native_result, indent=2, sort_keys=True), encoding="utf-8") + artifact_ref = str(artifact_path.relative_to(run_dir)) + return { + "result": "unknown", + "observations": [ + "SDK fixture runner wrote native output for normalization." + ], + "facts": { + "native_result_ref": artifact_ref + }, + "artifact_refs": [ + artifact_ref + ], + } diff --git a/extensions/sdk-fixture/schemas/sdk-fixture-assessment.schema.json b/extensions/sdk-fixture/schemas/sdk-fixture-assessment.schema.json new file mode 100644 index 0000000..2af7a43 --- /dev/null +++ b/extensions/sdk-fixture/schemas/sdk-fixture-assessment.schema.json @@ -0,0 +1,19 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "SDK Fixture Assessment Profile", + "type": "object", + "required": [ + "runtime_policy" + ], + "properties": { + "runtime_policy": { + "type": "object", + "required": [ + "fixture_mode" + ], + "properties": { + "fixture_mode": { "enum": ["native-result"] } + } + } + } +} diff --git a/extensions/sdk-fixture/schemas/sdk-fixture-target.schema.json b/extensions/sdk-fixture/schemas/sdk-fixture-target.schema.json new file mode 100644 index 0000000..ada23b4 --- /dev/null +++ b/extensions/sdk-fixture/schemas/sdk-fixture-target.schema.json @@ -0,0 +1,15 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "SDK Fixture Target Profile", + "type": "object", + "required": [ + "subject_type", + "artifacts", + "declared_capabilities" + ], + "properties": { + "subject_type": { "enum": ["sdk-fixture-target"] }, + "artifacts": { "type": "array", "minItems": 1 }, + "declared_capabilities": { "type": "array", "minItems": 1 } + } +} diff --git a/src/guide_board/execution.py b/src/guide_board/execution.py index e8abd41..543722a 100644 --- a/src/guide_board/execution.py +++ b/src/guide_board/execution.py @@ -10,6 +10,7 @@ from typing import Any from guide_board.artifacts import build_artifact_manifest from guide_board.io import write_json from guide_board.mapping import build_mapping_records, summarize_mappings +from guide_board.normalizers import normalize_step_result from guide_board.planning import build_run_plan from guide_board.policy import apply_policy from guide_board.retention import build_retention_summary @@ -153,6 +154,7 @@ def _evidence_for_step( now = _now() runner_ref = step.get("runner_ref") runner_result = run_step(root, run_dir, run_id, plan, step) + runner_result = normalize_step_result(root, run_dir, run_id, plan, step, runner_result) return { "id": f"evidence:{step['id']}", @@ -167,17 +169,44 @@ def _evidence_for_step( "runner_ref": runner_ref, **runner_result["facts"], }, - "requirement_refs": _requirement_refs(plan, step), + "requirement_refs": _requirement_refs(plan, step, runner_result), "artifact_refs": runner_result["artifact_refs"], "started_at": now, "completed_at": now, } -def _requirement_refs(plan: dict[str, Any], step: dict[str, Any]) -> list[str]: +def _requirement_refs( + plan: dict[str, Any], + step: dict[str, Any], + runner_result: dict[str, Any] | None = None, +) -> list[str]: + refs = [] if step["kind"] != "check_group": + return _runner_requirement_refs(runner_result) + refs.extend(step.get("requirement_refs", [])) + refs.extend(_runner_requirement_refs(runner_result)) + return _dedupe(refs) + + +def _runner_requirement_refs(runner_result: dict[str, Any] | None) -> list[str]: + if not runner_result: return [] - return list(step.get("requirement_refs", [])) + refs = runner_result.get("requirement_refs", []) + if not isinstance(refs, list): + return [] + return [ref for ref in refs if isinstance(ref, str)] + + +def _dedupe(values: list[str]) -> list[str]: + seen = set() + deduped = [] + for value in values: + if value in seen: + continue + seen.add(value) + deduped.append(value) + return deduped def _findings_for_evidence(run_id: str, evidence: list[dict[str, Any]]) -> list[dict[str, Any]]: diff --git a/src/guide_board/normalizers.py b/src/guide_board/normalizers.py new file mode 100644 index 0000000..4b23cd2 --- /dev/null +++ b/src/guide_board/normalizers.py @@ -0,0 +1,224 @@ +"""Normalizer plug-in bridge for extension-provided runner output.""" + +from __future__ import annotations + +import importlib.util +from pathlib import Path +from types import ModuleType +from typing import Any + +from guide_board.errors import ValidationError +from guide_board.io import load_json + + +def normalize_step_result( + root: Path, + run_dir: Path, + run_id: str, + plan: dict[str, Any], + step: dict[str, Any], + runner_result: dict[str, Any], +) -> dict[str, Any]: + """Apply matching extension normalizers to a runner result.""" + extension = _extension_snapshot(plan, step["extension_id"]) + extension_path = _snapshot_path(root, extension) + manifest = load_json(extension_path / "extension.json") + result = _coerce_result(runner_result) + applied: list[str] = [] + + for normalizer in _matching_normalizers(manifest, step): + normalized = _run_normalizer( + root, + run_dir, + run_id, + plan, + step, + extension_path, + normalizer, + result, + ) + if _is_normalizer_error(normalized): + return normalized + result = _merge_result(result, normalized) + applied.append(normalizer["id"]) + + if applied: + facts = dict(result.get("facts", {})) + facts["normalizer_refs"] = applied + result["facts"] = facts + return result + + +def _matching_normalizers( + manifest: dict[str, Any], + step: dict[str, Any], +) -> list[dict[str, Any]]: + matching = [] + runner_ref = step.get("runner_ref") + for normalizer in manifest.get("normalizers", []): + if not isinstance(normalizer, dict): + continue + normalizer_runner_ref = normalizer.get("runner_ref") + if normalizer_runner_ref and normalizer_runner_ref != runner_ref: + continue + matching.append(normalizer) + return matching + + +def _run_normalizer( + root: Path, + run_dir: Path, + run_id: str, + plan: dict[str, Any], + step: dict[str, Any], + extension_path: Path, + normalizer: dict[str, Any], + runner_result: dict[str, Any], +) -> dict[str, Any]: + if normalizer["kind"] != "python_module": + raise ValidationError( + f"{normalizer['id']}: unsupported normalizer kind {normalizer['kind']!r}" + ) + + module_path = normalizer.get("module_path") + callable_name = normalizer.get("callable") + if not module_path or not callable_name: + raise ValidationError( + f"{normalizer['id']}: python_module normalizers need module_path and callable" + ) + + module_file = (extension_path / module_path).resolve() + try: + module_file.relative_to(extension_path.resolve()) + except ValueError as exc: + raise ValidationError( + f"{normalizer['id']}: module_path must stay inside the extension directory" + ) from exc + + module = _load_module(module_file, normalizer["id"]) + normalizer_callable = getattr(module, callable_name, None) + if not callable(normalizer_callable): + raise ValidationError(f"{normalizer['id']}: callable {callable_name!r} was not found") + + context = { + "root": str(root), + "run_dir": str(run_dir), + "run_id": run_id, + "plan": plan, + "step": step, + "target_profile": plan["target_profile_snapshot"], + "assessment_profile": plan["assessment_profile_snapshot"], + "extension_path": str(extension_path), + "normalizer": normalizer, + "runner_result": runner_result, + } + try: + result = normalizer_callable(context) + except Exception as exc: # noqa: BLE001 - extension failures become evidence. + return { + "result": "infrastructure_error", + "observations": [ + f"Normalizer {normalizer['id']!r} failed before producing evidence: {exc}" + ], + "facts": { + "normalizer_ref": normalizer["id"], + "normalizer_kind": normalizer["kind"], + "error_type": type(exc).__name__, + }, + "artifact_refs": runner_result.get("artifact_refs", []), + "requirement_refs": runner_result.get("requirement_refs", []), + } + + if not isinstance(result, dict): + raise ValidationError(f"{normalizer['id']}: normalizer must return an object") + return result + + +def _merge_result( + base: dict[str, Any], + update: dict[str, Any], +) -> dict[str, Any]: + merged = dict(base) + if "result" in update: + merged["result"] = update["result"] + if "observations" in update: + merged["observations"] = _string_list(base.get("observations", [])) + merged["observations"].extend(_string_list(update.get("observations", []))) + if "facts" in update: + facts = dict(base.get("facts", {})) + update_facts = update.get("facts", {}) + if isinstance(update_facts, dict): + facts.update(update_facts) + merged["facts"] = facts + if "artifact_refs" in update: + merged["artifact_refs"] = _dedupe( + _string_list(base.get("artifact_refs", [])) + + _string_list(update.get("artifact_refs", [])) + ) + if "requirement_refs" in update: + merged["requirement_refs"] = _dedupe( + _string_list(base.get("requirement_refs", [])) + + _string_list(update.get("requirement_refs", [])) + ) + return _coerce_result(merged) + + +def _coerce_result(value: dict[str, Any]) -> dict[str, Any]: + facts = value.get("facts", {}) + if not isinstance(facts, dict): + facts = {} + return { + "result": value.get("result", "unknown"), + "observations": _string_list(value.get("observations", [])), + "facts": facts, + "artifact_refs": _string_list(value.get("artifact_refs", [])), + "requirement_refs": _string_list(value.get("requirement_refs", [])), + } + + +def _is_normalizer_error(result: dict[str, Any]) -> bool: + return ( + result.get("result") == "infrastructure_error" + and "normalizer_ref" in result.get("facts", {}) + ) + + +def _string_list(value: Any) -> list[str]: + if not isinstance(value, list): + return [] + return [item for item in value if isinstance(item, str)] + + +def _dedupe(values: list[str]) -> list[str]: + seen = set() + deduped = [] + for value in values: + if value in seen: + continue + seen.add(value) + deduped.append(value) + return deduped + + +def _load_module(path: Path, normalizer_id: str) -> ModuleType: + if not path.exists(): + raise ValidationError(f"{normalizer_id}: module not found: {path}") + module_name = f"_guide_board_normalizer_{normalizer_id.replace('-', '_')}" + spec = importlib.util.spec_from_file_location(module_name, path) + if spec is None or spec.loader is None: + raise ValidationError(f"{normalizer_id}: unable to load module from {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _extension_snapshot(plan: dict[str, Any], extension_id: str) -> dict[str, Any]: + for extension in plan["extension_snapshots"]: + if extension["id"] == extension_id: + return extension + raise ValidationError(f"step references unknown extension {extension_id!r}") + + +def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path: + path = Path(extension["path"]) + return path if path.is_absolute() else root / path diff --git a/tests/test_core.py b/tests/test_core.py index 6d47984..932d295 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -2,6 +2,7 @@ from __future__ import annotations import http.client import json +import shutil import time import unittest from tempfile import TemporaryDirectory @@ -193,6 +194,52 @@ class CoreArchitectureTests(unittest.TestCase): ): validate_target_profile(target_path, extensions) + def test_runs_sdk_fixture_from_external_extension_repo(self) -> None: + with TemporaryDirectory() as temporary_directory: + temp_root = Path(temporary_directory) + extension_dir = temp_root / "sdk-fixture" + shutil.copytree(ROOT / "extensions" / "sdk-fixture", extension_dir) + + result = run_assessment( + temp_root, + extension_dir / "profiles" / "targets" / "sdk-fixture-target.json", + extension_dir / "profiles" / "assessments" / "sdk-fixture-assessment.json", + temp_root / "runs" / "sdk-fixture", + [extension_dir], + ) + run_dir = Path(result["run_dir"]) + plan = json.loads((run_dir / "plan.json").read_text(encoding="utf-8")) + evidence = json.loads( + (run_dir / "normalized" / "evidence.json").read_text(encoding="utf-8") + )["evidence"] + mappings = json.loads( + (run_dir / "normalized" / "mappings.json").read_text(encoding="utf-8") + )["mappings"] + assessment_package = json.loads( + (run_dir / "reports" / "assessment-package.json").read_text(encoding="utf-8") + ) + + self.assertEqual(result["status"], "completed") + self.assertEqual(plan["extension_snapshots"][0]["source"], "external") + self.assertEqual(plan["target_profile_snapshot"]["subject_type"], "sdk-fixture-target") + self.assertEqual([item["result"] for item in evidence], ["skipped", "pass"]) + check_evidence = evidence[1] + self.assertEqual( + check_evidence["facts"]["normalizer_refs"], + ["native-probe-normalizer"], + ) + self.assertEqual(check_evidence["facts"]["native_score"], 98) + self.assertEqual( + check_evidence["requirement_refs"], + ["guide-board.sdk-fixture.v1.native-output"], + ) + self.assertEqual( + check_evidence["artifact_refs"], + ["artifacts/sdk-fixture/native-result.json"], + ) + self.assertEqual(mappings[0]["target_id"], "normalizer-plugin") + self.assertEqual(assessment_package["summary"], {"pass": 1, "skipped": 1}) + def test_runs_sample_noop_assessment(self) -> None: with TemporaryDirectory() as temporary_directory: result = run_assessment( diff --git a/workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md b/workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md index 5954d15..8f291a8 100644 --- a/workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md +++ b/workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md @@ -4,7 +4,7 @@ type: workplan title: "Extension SDK Maturity" repo: guide-board domain: markitect -status: active +status: completed owner: codex planning_priority: high planning_order: 3 @@ -69,7 +69,7 @@ Progress: ```task id: GUIDE-BOARD-WP-0003-T002 -status: todo +status: done priority: high state_hub_task_id: "b87e68c1-6eca-4274-8e3f-6e2854c5a1e1" ``` @@ -81,11 +81,22 @@ Acceptance: normalize native result artifacts explicitly. - Add tests that prove a normalizer can map native output into evidence. +Progress: + +- Added `guide_board.normalizers.normalize_step_result`. +- Extended `normalizers` manifest entries to support Python module descriptor + objects while preserving the existing string shorthand. +- Invoked matching normalizers after runner execution and before evidence + writing. +- Merged normalizer result fields over runner results and recorded + `normalizer_refs` in evidence facts. +- Added test coverage through the SDK fixture run. + ## D3.3 - SDK Fixture Extension And Acceptance Tests ```task id: GUIDE-BOARD-WP-0003-T003 -status: todo +status: done priority: medium state_hub_task_id: "f3738751-5a0d-4eaf-85b1-75e599a78060" ``` @@ -97,11 +108,19 @@ Acceptance: - Cover external repo discovery, schema validation, normalizer invocation, plan generation, and result package shape. +Progress: + +- Added `extensions/sdk-fixture`. +- Included extension-owned target and assessment schemas, fixture profiles, a + native-output runner, a normalizer, and a mapping set. +- Added a unit test that copies the fixture as an external extension repository + and verifies plan, evidence, mapping, and assessment package output. + ## D3.4 - Extension Authoring Documentation Refresh ```task id: GUIDE-BOARD-WP-0003-T004 -status: todo +status: done priority: medium state_hub_task_id: "3d390bd4-755b-462a-9e16-9c859990d99e" ``` @@ -114,6 +133,14 @@ Acceptance: - Link the SDK maturity guidance from the assessment operations and external extension acceptance docs where useful. +Progress: + +- Refreshed `docs/EXTENSION-SDK.md` with profile schema descriptors, + normalizer descriptors, context fields, merge semantics, and fixture guidance. +- Linked SDK authoring contracts from `docs/ASSESSMENT-OPERATIONS.md`. +- Linked `extensions/sdk-fixture` from `docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`. +- Added README references for the SDK fixture and WP3. + ## Definition Of Done - External extension repositories can declare and test domain-specific profile