diff --git a/README.md b/README.md index 00e7f6b..55082de 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,9 @@ The same CLI contracts are packaged by the container baseline. See [docs/CONTAINER.md](docs/CONTAINER.md). The dependency-light local API wraps those contracts for service and container operation; see [docs/LOCAL-SERVICE-API.md](docs/LOCAL-SERVICE-API.md). +New runs write a richer `sources.lock.json` and +`reports/submission-package.json` alongside the assessment package so reviewers +can inspect source, metadata, artifact, and boundary references. The `sample-noop` extension exercises the guide-board contracts without invoking an external harness. `sdk-fixture` demonstrates the extension SDK contracts for diff --git a/docs/ARCHITECTURE-BLUEPRINT.md b/docs/ARCHITECTURE-BLUEPRINT.md index 9125799..5776d38 100644 --- a/docs/ARCHITECTURE-BLUEPRINT.md +++ b/docs/ARCHITECTURE-BLUEPRINT.md @@ -355,7 +355,9 @@ Stores run artifacts by reference and checksum: The first implementation builds the assessment package artifact manifest from runner-emitted artifact refs and computes checksums for files inside the run -directory. +directory. New runs also write a source lock and a submission package manifest +that fingerprint reviewable run files and summarize runner or normalizer +metadata reported by extensions. ### Normalizer @@ -559,6 +561,18 @@ building complex runtime code. - `artifact_policy` - `runtime_policy` +### `SourceLock` + +- `framework_refs` +- `extension_refs` +- `frameworks` +- `extensions` +- `mapping_sets` +- `profiles` +- `policy_refs` +- `authorities` +- `metadata_hooks` + ### `RawArtifact` - `id` @@ -626,6 +640,19 @@ building complex runtime code. - `certification_boundary` - `created_at` +### `SubmissionPackage` + +- `run_id` +- `package_identity` +- `source_lock_ref` +- `source_lock` +- `reports` +- `normalized_outputs` +- `profile_snapshots` +- `artifact_manifest` +- `reported_metadata` +- `certification_boundary` + ## Result Vocabulary The evidence model should allow these statuses: @@ -714,6 +741,7 @@ runs// reports/ report.md assessment-package.json + submission-package.json exports/ ``` @@ -787,7 +815,12 @@ Each run should lock: - test suite IDs, - mapping version, - target profile snapshot, -- waiver snapshot. +- expectation and waiver refs. + +The current source lock remains backward-compatible with the original +`framework_refs` and `extension_refs` fields while adding checksummed profiles, +mapping-set refs, optional policy refs, authority descriptors, and metadata +hooks for runners and normalizers. ## Implementation Sequence diff --git a/docs/ASSESSMENT-OPERATIONS.md b/docs/ASSESSMENT-OPERATIONS.md index d9d34d5..c5c1a52 100644 --- a/docs/ASSESSMENT-OPERATIONS.md +++ b/docs/ASSESSMENT-OPERATIONS.md @@ -77,6 +77,7 @@ A completed CLI command prints a JSON result with: - `run_dir`: output directory, - `assessment_package`: JSON assessment package path, - `report`: Markdown report path, +- `submission_package`: portable submission package manifest path, - `retention_summary`: compact durable summary path. The output directory uses this contract: @@ -84,15 +85,27 @@ The output directory uses this contract: ```text run.json plan.json +sources.lock.json +target-profile.snapshot.json +assessment-profile.snapshot.json retention-summary.json normalized/evidence.json normalized/findings.json normalized/mappings.json reports/assessment-package.json reports/report.md +reports/submission-package.json artifacts/ ``` +`sources.lock.json` records the framework refs, extension versions, mapping +sets, profile snapshots, policy refs, authority refs, and extension metadata +hooks used for the run. `reports/submission-package.json` points at the +reviewable package files, includes checksums where files exist, carries the raw +artifact manifest, and repeats the certification boundary. It is a portable +handoff manifest for preparation evidence, not an authority-specific final +submission. + Use the retained run helpers for history: ```sh diff --git a/docs/EXTENSION-SDK.md b/docs/EXTENSION-SDK.md index 04ba25d..9bcfb48 100644 --- a/docs/EXTENSION-SDK.md +++ b/docs/EXTENSION-SDK.md @@ -71,7 +71,12 @@ The key runtime fields are: - `extension_type`: one of the supported archetypes from the architecture blueprint. - `supported_frameworks`: framework IDs this extension can contribute evidence - for. + for. Descriptor objects with `id`, `version`, `source_url`, and + `authority_ref` may be used when source metadata is available. +- `authorities`: authority IDs or descriptor objects with optional source URL, + version, license, and access notes. +- `metadata`: optional extension-level metadata such as adapter version or + source URL. The core preserves it in source locks and evidence metadata. - `check_groups`: named groups that assessment profiles can select. - `preflight_runner`: optional runner ID used before selected check groups. - `runner_entrypoints`: concrete runner declarations. @@ -141,6 +146,11 @@ Example: "module_path": "src/open_cmis_tck/preflight.py", "callable": "run", "command": null, + "metadata": { + "harness_id": "opencmis-tck", + "harness_version": "extension-detected-or-declared", + "source_url": "https://chemistry.apache.org/java/opencmis.html" + }, "description": "Checks whether the CMIS Browser Binding endpoint is reachable." } ``` @@ -272,11 +282,20 @@ Result fields: - `observations`: human-readable observations. - `facts`: structured facts extracted by the runner. - `artifact_refs`: references to raw artifacts written by the runner. +- `requirement_refs`: optional requirement refs discovered by the runner. +- `metadata`: optional generic metadata such as `harness_version`, + `test_suite_id`, `adapter_version`, `source_url`, or native result IDs. Artifact refs must be paths relative to the run directory. After runner execution, the core fingerprints existing artifact refs into the assessment package `artifact_manifest`. +Runner metadata is merged with manifest entrypoint metadata and preserved under +evidence `facts.source_metadata`. The same metadata is also summarized in the +submission package manifest, which lets reviewers distinguish the extension +version from the harness or native test-suite version without adding +domain-specific fields to the core. + If a Python runner raises an exception, the core converts that failure into `infrastructure_error` evidence so the assessment package remains complete. @@ -298,6 +317,9 @@ extension can add a normalizer descriptor: "module_path": "normalizers/native_probe.py", "callable": "normalize", "runner_ref": "native-probe", + "metadata": { + "adapter_version": "0.1.0" + }, "description": "Converts native runner output into guide-board evidence." } ``` @@ -340,6 +362,7 @@ The core merges the normalizer output over the runner result: - `observations` are appended. - `facts` are merged. - `artifact_refs` and `requirement_refs` are deduplicated. +- `metadata` is merged. - `normalizer_refs` is recorded in evidence facts when any normalizer runs. If a normalizer raises an exception, the step becomes @@ -350,6 +373,25 @@ The bundled `extensions/sdk-fixture` extension is the copyable reference path for profile schemas, a native-output runner, a normalizer, mappings, and fixture profiles. +## Source Lock And Submission Package + +Every new run writes `sources.lock.json` and +`reports/submission-package.json`. Extension authors should treat source +metadata as part of the evidence contract: + +- declare extension, authority, framework, runner, and normalizer metadata in + `extension.json` when it is static; +- return runner or normalizer `metadata` when versions, native result IDs, or + test-suite IDs are detected at runtime; +- keep mapping sets under `mappings/` so the core can checksum them in the + source lock; +- keep restricted or licensed assets referenced by metadata or artifacts rather + than vendored into the core. + +The submission package manifest is generic guide-board output. Authority-specific +final submissions, trademark assertions, or certification conclusions remain +extension-owned or reviewer-owned. + ## Result Statuses Initial statuses: diff --git a/docs/schemas/extension-manifest.schema.json b/docs/schemas/extension-manifest.schema.json index 9946708..d08c569 100644 --- a/docs/schemas/extension-manifest.schema.json +++ b/docs/schemas/extension-manifest.schema.json @@ -41,8 +41,38 @@ "type": "string", "enum": ["candidate", "incubating", "active", "external", "deprecated"] }, - "supported_frameworks": { "type": "array", "items": { "type": "string" } }, - "authorities": { "type": "array", "items": { "type": "string" } }, + "supported_frameworks": { + "type": "array", + "items": { + "type": ["string", "object"], + "additionalProperties": false, + "required": ["id"], + "properties": { + "id": { "type": "string" }, + "version": { "type": ["string", "null"] }, + "source_url": { "type": ["string", "null"] }, + "authority_ref": { "type": ["string", "null"] }, + "description": { "type": ["string", "null"] } + } + } + }, + "authorities": { + "type": "array", + "items": { + "type": ["string", "object"], + "additionalProperties": false, + "required": ["id"], + "properties": { + "id": { "type": "string" }, + "name": { "type": ["string", "null"] }, + "version": { "type": ["string", "null"] }, + "source_url": { "type": ["string", "null"] }, + "license": { "type": ["string", "null"] }, + "access": { "type": ["string", "null"] } + } + } + }, + "metadata": { "type": "object" }, "profile_schemas": { "type": "array", "items": { @@ -89,6 +119,7 @@ "module_path": { "type": ["string", "null"] }, "callable": { "type": ["string", "null"] }, "command": { "type": ["array", "null"], "items": { "type": "string" } }, + "metadata": { "type": "object" }, "description": { "type": ["string", "null"] } } } @@ -105,6 +136,7 @@ "module_path": { "type": "string" }, "callable": { "type": "string" }, "runner_ref": { "type": ["string", "null"] }, + "metadata": { "type": "object" }, "description": { "type": ["string", "null"] } } } diff --git a/docs/schemas/source-lock.schema.json b/docs/schemas/source-lock.schema.json new file mode 100644 index 0000000..6a4c151 --- /dev/null +++ b/docs/schemas/source-lock.schema.json @@ -0,0 +1,34 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Guide Board Source Lock", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "schema_version", + "created_at", + "framework_refs", + "extension_refs", + "frameworks", + "extensions", + "mapping_sets", + "profiles", + "policy_refs", + "authorities", + "metadata_hooks" + ], + "properties": { + "id": { "type": "string" }, + "schema_version": { "type": "string" }, + "created_at": { "type": "string" }, + "framework_refs": { "type": "array", "items": { "type": "string" } }, + "extension_refs": { "type": "array", "items": { "type": "string" } }, + "frameworks": { "type": "array", "items": { "type": "object" } }, + "extensions": { "type": "array", "items": { "type": "object" } }, + "mapping_sets": { "type": "array", "items": { "type": "object" } }, + "profiles": { "type": "object" }, + "policy_refs": { "type": "object" }, + "authorities": { "type": "array", "items": { "type": "object" } }, + "metadata_hooks": { "type": "object" } + } +} diff --git a/docs/schemas/submission-package.schema.json b/docs/schemas/submission-package.schema.json new file mode 100644 index 0000000..1557288 --- /dev/null +++ b/docs/schemas/submission-package.schema.json @@ -0,0 +1,36 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Guide Board Submission Package Manifest", + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "schema_version", + "run_id", + "created_at", + "package_identity", + "source_lock_ref", + "source_lock", + "reports", + "normalized_outputs", + "profile_snapshots", + "artifact_manifest", + "reported_metadata", + "certification_boundary" + ], + "properties": { + "id": { "type": "string" }, + "schema_version": { "type": "string" }, + "run_id": { "type": "string" }, + "created_at": { "type": "string" }, + "package_identity": { "type": "object" }, + "source_lock_ref": { "type": "string" }, + "source_lock": { "type": "object" }, + "reports": { "type": "array", "items": { "type": "object" } }, + "normalized_outputs": { "type": "array", "items": { "type": "object" } }, + "profile_snapshots": { "type": "array", "items": { "type": "object" } }, + "artifact_manifest": { "type": "array", "items": { "type": "object" } }, + "reported_metadata": { "type": "array", "items": { "type": "object" } }, + "certification_boundary": { "type": "string" } + } +} diff --git a/extensions/sdk-fixture/extension.json b/extensions/sdk-fixture/extension.json index 87f0233..f08d4a6 100644 --- a/extensions/sdk-fixture/extension.json +++ b/extensions/sdk-fixture/extension.json @@ -8,6 +8,10 @@ "guide-board.sdk-fixture.v1" ], "authorities": [], + "metadata": { + "adapter_version": "0.1.0", + "source_url": "https://example.invalid/guide-board/sdk-fixture" + }, "profile_schemas": [ "target-profile", "assessment-profile", @@ -44,6 +48,12 @@ "module_path": "runners/native_probe.py", "callable": "run", "command": null, + "metadata": { + "harness_id": "sdk-fixture-native-probe", + "harness_version": "1.0.0", + "test_suite_id": "sdk-fixture-suite-v1", + "source_url": "https://example.invalid/guide-board/sdk-fixture/native-probe" + }, "description": "Writes a tiny native result artifact for the SDK fixture normalizer." } ], @@ -54,6 +64,10 @@ "module_path": "normalizers/native_probe.py", "callable": "normalize", "runner_ref": "native-probe", + "metadata": { + "adapter_version": "0.1.0", + "source_url": "https://example.invalid/guide-board/sdk-fixture/native-normalizer" + }, "description": "Converts the SDK fixture native result artifact into guide-board evidence." } ], diff --git a/extensions/sdk-fixture/normalizers/native_probe.py b/extensions/sdk-fixture/normalizers/native_probe.py index 1a0bc9b..0985ec5 100644 --- a/extensions/sdk-fixture/normalizers/native_probe.py +++ b/extensions/sdk-fixture/normalizers/native_probe.py @@ -25,4 +25,8 @@ def normalize(context: dict) -> dict: artifact_ref ], "requirement_refs": native_result.get("requirement_refs", []), + "metadata": { + "normalizer_id": "native-probe-normalizer", + "native_result_id": "sdk-fixture-native-result" + }, } diff --git a/extensions/sdk-fixture/runners/native_probe.py b/extensions/sdk-fixture/runners/native_probe.py index f725f80..6abd914 100644 --- a/extensions/sdk-fixture/runners/native_probe.py +++ b/extensions/sdk-fixture/runners/native_probe.py @@ -33,4 +33,8 @@ def run(context: dict) -> dict: "artifact_refs": [ artifact_ref ], + "metadata": { + "native_result_id": "sdk-fixture-native-result", + "test_suite_id": "sdk-fixture-suite-v1" + }, } diff --git a/src/guide_board/artifacts.py b/src/guide_board/artifacts.py index b91193d..6bd5552 100644 --- a/src/guide_board/artifacts.py +++ b/src/guide_board/artifacts.py @@ -46,6 +46,107 @@ def build_artifact_manifest( return artifacts +def build_submission_manifest( + run_dir: Path, + run_metadata: dict[str, Any], + plan: dict[str, Any], + evidence: list[dict[str, Any]], + assessment_package: dict[str, Any], +) -> dict[str, Any]: + """Build a portable manifest for the files that make up a review package.""" + source_lock = plan["source_lock"] + manifest = { + "id": f"submission-package:{run_metadata['id']}", + "schema_version": "guide-board.submission-package.v1", + "run_id": run_metadata["id"], + "created_at": datetime.now(timezone.utc).isoformat(), + "package_identity": { + "target_profile_ref": run_metadata["target_profile_ref"], + "assessment_profile_ref": run_metadata["assessment_profile_ref"], + "framework_refs": source_lock.get("framework_refs", []), + "extension_refs": source_lock.get("extension_refs", []), + }, + "source_lock_ref": "sources.lock.json", + "source_lock": { + "id": source_lock.get("id"), + "schema_version": source_lock.get("schema_version"), + "checksum": _file_entry(run_dir, "sources.lock.json").get("checksum"), + "framework_refs": source_lock.get("framework_refs", []), + "extension_refs": source_lock.get("extension_refs", []), + }, + "reports": _existing_file_entries( + run_dir, + [ + ("assessment-package", "reports/assessment-package.json"), + ("markdown-report", "reports/report.md"), + ], + ), + "normalized_outputs": _existing_file_entries( + run_dir, + [ + ("evidence", "normalized/evidence.json"), + ("findings", "normalized/findings.json"), + ("mappings", "normalized/mappings.json"), + ], + ), + "profile_snapshots": _existing_file_entries( + run_dir, + [ + ("target-profile", "target-profile.snapshot.json"), + ("assessment-profile", "assessment-profile.snapshot.json"), + ], + ), + "artifact_manifest": assessment_package.get("artifact_manifest", []), + "reported_metadata": _reported_metadata(evidence), + "certification_boundary": assessment_package["certification_boundary"], + } + assert_valid(manifest, "submission-package") + return manifest + + +def _existing_file_entries(run_dir: Path, refs: list[tuple[str, str]]) -> list[dict[str, Any]]: + entries = [] + for entry_id, ref in refs: + entry = _file_entry(run_dir, ref) + if entry: + entry["id"] = entry_id + entries.append(entry) + return entries + + +def _file_entry(run_dir: Path, ref: str) -> dict[str, Any]: + path = (run_dir / ref).resolve() + try: + path.relative_to(run_dir.resolve()) + except ValueError: + return {} + if not path.is_file(): + return {} + return { + "path": ref, + "media_type": _media_type(path), + "checksum": f"sha256:{_sha256(path)}", + "size_bytes": path.stat().st_size, + } + + +def _reported_metadata(evidence: list[dict[str, Any]]) -> list[dict[str, Any]]: + records = [] + for item in evidence: + metadata = item.get("facts", {}).get("source_metadata") + if not isinstance(metadata, dict) or not metadata: + continue + records.append( + { + "evidence_ref": item["id"], + "check_id": item["check_id"], + "extension_id": item["extension_id"], + "metadata": metadata, + } + ) + return records + + def _sha256(path: Path) -> str: digest = hashlib.sha256() with path.open("rb") as handle: diff --git a/src/guide_board/execution.py b/src/guide_board/execution.py index 543722a..ec33f0f 100644 --- a/src/guide_board/execution.py +++ b/src/guide_board/execution.py @@ -7,8 +7,8 @@ from datetime import datetime, timezone from pathlib import Path from typing import Any -from guide_board.artifacts import build_artifact_manifest -from guide_board.io import write_json +from guide_board.artifacts import build_artifact_manifest, build_submission_manifest +from guide_board.io import load_json, write_json from guide_board.mapping import build_mapping_records, summarize_mappings from guide_board.normalizers import normalize_step_result from guide_board.planning import build_run_plan @@ -83,6 +83,7 @@ def run_assessment( "run_dir": str(run_dir), "assessment_package": str(run_dir / "reports" / "assessment-package.json"), "report": str(run_dir / "reports" / "report.md"), + "submission_package": str(run_dir / "reports" / "submission-package.json"), "retention_summary": str(run_dir / "retention-summary.json"), } @@ -155,6 +156,14 @@ def _evidence_for_step( runner_ref = step.get("runner_ref") runner_result = run_step(root, run_dir, run_id, plan, step) runner_result = normalize_step_result(root, run_dir, run_id, plan, step, runner_result) + facts = { + "step_kind": step["kind"], + "runner_ref": runner_ref, + **runner_result["facts"], + } + source_metadata = _source_metadata_for_step(root, plan, step, runner_result) + if source_metadata: + facts["source_metadata"] = source_metadata return { "id": f"evidence:{step['id']}", @@ -164,11 +173,7 @@ def _evidence_for_step( "subject_ref": plan["target_profile_snapshot"]["id"], "result": runner_result["result"], "observations": runner_result["observations"], - "facts": { - "step_kind": step["kind"], - "runner_ref": runner_ref, - **runner_result["facts"], - }, + "facts": facts, "requirement_refs": _requirement_refs(plan, step, runner_result), "artifact_refs": runner_result["artifact_refs"], "started_at": now, @@ -198,6 +203,95 @@ def _runner_requirement_refs(runner_result: dict[str, Any] | None) -> list[str]: return [ref for ref in refs if isinstance(ref, str)] +def _source_metadata_for_step( + root: Path, + plan: dict[str, Any], + step: dict[str, Any], + runner_result: dict[str, Any], +) -> dict[str, Any]: + runner_ref = step.get("runner_ref") + extension = _extension_snapshot(plan, step["extension_id"]) + extension_path = _snapshot_path(root, extension) + manifest = load_json(extension_path / "extension.json") + metadata: dict[str, Any] = { + "extension": { + "id": step["extension_id"], + "version": extension.get("version"), + "metadata": _object_or_empty(manifest.get("metadata")), + } + } + + if runner_ref: + entrypoint = _runner_entrypoint(manifest, runner_ref) + metadata["runner"] = { + "id": runner_ref, + "kind": entrypoint.get("kind"), + "metadata": _object_or_empty(entrypoint.get("metadata")), + } + + applied_normalizers = runner_result.get("facts", {}).get("normalizer_refs", []) + normalizers = [] + if isinstance(applied_normalizers, list): + normalizer_ids = {item for item in applied_normalizers if isinstance(item, str)} + for normalizer in manifest.get("normalizers", []): + if isinstance(normalizer, dict) and normalizer.get("id") in normalizer_ids: + normalizers.append( + { + "id": normalizer["id"], + "kind": normalizer.get("kind"), + "runner_ref": normalizer.get("runner_ref"), + "metadata": _object_or_empty(normalizer.get("metadata")), + } + ) + if normalizers: + metadata["normalizers"] = normalizers + + reported = _object_or_empty(runner_result.get("metadata")) + if reported: + metadata["reported"] = reported + + return _drop_empty_metadata(metadata) + + +def _extension_snapshot(plan: dict[str, Any], extension_id: str) -> dict[str, Any]: + for extension in plan["extension_snapshots"]: + if extension["id"] == extension_id: + return extension + return {} + + +def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path: + path = Path(extension["path"]) + return path if path.is_absolute() else root / path + + +def _runner_entrypoint(manifest: dict[str, Any], runner_ref: str) -> dict[str, Any]: + for entrypoint in manifest.get("runner_entrypoints", []): + if entrypoint.get("id") == runner_ref: + return entrypoint + return {} + + +def _object_or_empty(value: Any) -> dict[str, Any]: + return value if isinstance(value, dict) else {} + + +def _drop_empty_metadata(value: dict[str, Any]) -> dict[str, Any]: + compact = {} + for key, child in value.items(): + if isinstance(child, dict): + child = _drop_empty_metadata(child) + if isinstance(child, list): + child = [ + _drop_empty_metadata(item) if isinstance(item, dict) else item + for item in child + ] + child = [item for item in child if item] + if child: + compact[key] = child + return compact + + def _dedupe(values: list[str]) -> list[str]: seen = set() deduped = [] @@ -340,6 +434,14 @@ def _write_run_directory( _markdown_report(run_metadata, assessment_package), encoding="utf-8", ) + submission_manifest = build_submission_manifest( + run_dir, + run_metadata, + plan, + evidence, + assessment_package, + ) + write_json(run_dir / "reports" / "submission-package.json", submission_manifest) def _markdown_report(run_metadata: dict[str, Any], package: dict[str, Any]) -> str: diff --git a/src/guide_board/normalizers.py b/src/guide_board/normalizers.py index 4b23cd2..6e726be 100644 --- a/src/guide_board/normalizers.py +++ b/src/guide_board/normalizers.py @@ -127,6 +127,7 @@ def _run_normalizer( }, "artifact_refs": runner_result.get("artifact_refs", []), "requirement_refs": runner_result.get("requirement_refs", []), + "metadata": runner_result.get("metadata", {}), } if not isinstance(result, dict): @@ -160,6 +161,12 @@ def _merge_result( _string_list(base.get("requirement_refs", [])) + _string_list(update.get("requirement_refs", [])) ) + if "metadata" in update: + metadata = dict(base.get("metadata", {})) + update_metadata = update.get("metadata", {}) + if isinstance(update_metadata, dict): + metadata.update(update_metadata) + merged["metadata"] = metadata return _coerce_result(merged) @@ -173,6 +180,7 @@ def _coerce_result(value: dict[str, Any]) -> dict[str, Any]: "facts": facts, "artifact_refs": _string_list(value.get("artifact_refs", [])), "requirement_refs": _string_list(value.get("requirement_refs", [])), + "metadata": _object_or_empty(value.get("metadata")), } @@ -189,6 +197,10 @@ def _string_list(value: Any) -> list[str]: return [item for item in value if isinstance(item, str)] +def _object_or_empty(value: Any) -> dict[str, Any]: + return value if isinstance(value, dict) else {} + + def _dedupe(values: list[str]) -> list[str]: seen = set() deduped = [] diff --git a/src/guide_board/planning.py b/src/guide_board/planning.py index a4c9b91..9776f52 100644 --- a/src/guide_board/planning.py +++ b/src/guide_board/planning.py @@ -2,6 +2,7 @@ from __future__ import annotations +import hashlib from datetime import datetime, timezone from pathlib import Path from typing import Any @@ -96,6 +97,16 @@ def build_run_plan( } ) + source_lock = _build_source_lock( + root, + target_path, + assessment_path, + target, + assessment, + [extensions[extension_id] for extension_id in selected_extensions], + ) + assert_valid(source_lock, "source-lock") + plan = { "id": f"plan-{_timestamp()}", "assessment_profile_snapshot": assessment, @@ -109,10 +120,7 @@ def build_run_plan( } for extension_id in selected_extensions ], - "source_lock": { - "framework_refs": assessment["framework_refs"], - "extension_refs": selected_extensions, - }, + "source_lock": source_lock, "profile_paths": { "target_profile_path": str(target_path.resolve()), "assessment_profile_path": str(assessment_path.resolve()), @@ -208,6 +216,270 @@ def _load_extension_profile_schema( return load_json(schema_path) +def _build_source_lock( + root: Path, + target_path: Path, + assessment_path: Path, + target: dict[str, Any], + assessment: dict[str, Any], + extensions: list[Extension], +) -> dict[str, Any]: + framework_refs = assessment["framework_refs"] + extension_refs = [extension.id for extension in extensions] + return { + "id": f"source-lock:{assessment['id']}:{target['id']}", + "schema_version": "guide-board.source-lock.v1", + "created_at": _now(), + "framework_refs": framework_refs, + "extension_refs": extension_refs, + "frameworks": _framework_records(framework_refs, extensions), + "extensions": [_extension_source_record(root, extension) for extension in extensions], + "mapping_sets": _mapping_source_records(root, extensions), + "profiles": { + "target": _file_source_record( + "target-profile", + target["id"], + target_path, + "target-profile.snapshot.json", + ), + "assessment": _file_source_record( + "assessment-profile", + assessment["id"], + assessment_path, + "assessment-profile.snapshot.json", + ), + }, + "policy_refs": { + "expectations": _optional_policy_source_record( + root, + assessment_path, + assessment.get("expectations_ref"), + "expectation-set", + ), + "waivers": _optional_policy_source_record( + root, + assessment_path, + assessment.get("waivers_ref"), + "waiver-set", + ), + }, + "authorities": _authority_source_records(extensions), + "metadata_hooks": { + "runner_entrypoints": _entrypoint_metadata_records(extensions), + "normalizers": _normalizer_metadata_records(extensions), + }, + } + + +def _framework_records( + framework_refs: list[str], + extensions: list[Extension], +) -> list[dict[str, Any]]: + records = [] + for framework_ref in framework_refs: + declaring_extensions = [ + extension.id + for extension in extensions + if framework_ref in _manifest_framework_ids(extension.manifest) + ] + records.append( + { + "id": framework_ref, + "version": _version_hint(framework_ref), + "declared_by_extensions": declaring_extensions, + } + ) + return records + + +def _extension_source_record(root: Path, extension: Extension) -> dict[str, Any]: + manifest_path = extension.path / "extension.json" + return { + "id": extension.id, + "version": extension.manifest["version"], + "path": _extension_path_ref(root, extension.path), + "source": extension.source, + "manifest_path": _display_path(root, manifest_path), + "manifest_checksum": _checksum_if_file(manifest_path), + "supported_frameworks": _manifest_framework_ids(extension.manifest), + "authorities": _authority_ids(extension.manifest.get("authorities", [])), + "certification_boundary": extension.manifest["certification_boundary"], + "metadata": _object_or_empty(extension.manifest.get("metadata")), + } + + +def _mapping_source_records(root: Path, extensions: list[Extension]) -> list[dict[str, Any]]: + records = [] + for extension in extensions: + for mapping_id in extension.manifest.get("mappings", []): + if not isinstance(mapping_id, str): + continue + mapping_path = extension.path / "mappings" / f"{mapping_id}.json" + record = { + "id": mapping_id, + "extension_id": extension.id, + "path": _display_path(root, mapping_path), + "exists": mapping_path.is_file(), + "checksum": _checksum_if_file(mapping_path), + "framework_refs": [], + } + if mapping_path.is_file(): + mapping_set = load_json(mapping_path) + record["framework_refs"] = _string_list(mapping_set.get("framework_refs", [])) + records.append(record) + return records + + +def _file_source_record( + kind: str, + profile_id: str, + path: Path, + snapshot_ref: str, +) -> dict[str, Any]: + return { + "kind": kind, + "id": profile_id, + "path": str(path.resolve()), + "checksum": _checksum_if_file(path), + "snapshot_ref": snapshot_ref, + } + + +def _optional_policy_source_record( + root: Path, + assessment_path: Path, + ref: Any, + kind: str, +) -> dict[str, Any] | None: + if not isinstance(ref, str) or not ref: + return None + path = _resolve_assessment_ref(root, assessment_path, ref) + return { + "kind": kind, + "ref": ref, + "path": str(path.resolve()), + "exists": path.is_file(), + "checksum": _checksum_if_file(path), + } + + +def _authority_source_records(extensions: list[Extension]) -> list[dict[str, Any]]: + records = [] + for extension in extensions: + for authority in extension.manifest.get("authorities", []): + if isinstance(authority, str): + records.append({"id": authority, "extension_id": extension.id}) + elif isinstance(authority, dict): + record = { + "id": authority.get("id"), + "extension_id": extension.id, + } + for key in ("name", "version", "source_url", "license", "access"): + if key in authority: + record[key] = authority[key] + records.append(record) + return [record for record in records if isinstance(record.get("id"), str)] + + +def _entrypoint_metadata_records(extensions: list[Extension]) -> list[dict[str, Any]]: + records = [] + for extension in extensions: + for entrypoint in extension.manifest.get("runner_entrypoints", []): + if not isinstance(entrypoint, dict): + continue + records.append( + { + "extension_id": extension.id, + "id": entrypoint.get("id"), + "kind": entrypoint.get("kind"), + "metadata": _object_or_empty(entrypoint.get("metadata")), + } + ) + return records + + +def _normalizer_metadata_records(extensions: list[Extension]) -> list[dict[str, Any]]: + records = [] + for extension in extensions: + for normalizer in extension.manifest.get("normalizers", []): + if not isinstance(normalizer, dict): + continue + records.append( + { + "extension_id": extension.id, + "id": normalizer.get("id"), + "kind": normalizer.get("kind"), + "runner_ref": normalizer.get("runner_ref"), + "metadata": _object_or_empty(normalizer.get("metadata")), + } + ) + return records + + +def _manifest_framework_ids(manifest: dict[str, Any]) -> list[str]: + values = [] + for framework in manifest.get("supported_frameworks", []): + if isinstance(framework, str): + values.append(framework) + elif isinstance(framework, dict) and isinstance(framework.get("id"), str): + values.append(framework["id"]) + return values + + +def _authority_ids(authorities: list[Any]) -> list[str]: + values = [] + for authority in authorities: + if isinstance(authority, str): + values.append(authority) + elif isinstance(authority, dict) and isinstance(authority.get("id"), str): + values.append(authority["id"]) + return values + + +def _resolve_assessment_ref(root: Path, assessment_path: Path, ref: str) -> Path: + ref_path = Path(ref) + if ref_path.is_absolute(): + return ref_path + root_relative = root / ref_path + if root_relative.exists(): + return root_relative + return assessment_path.resolve().parent / ref_path + + +def _checksum_if_file(path: Path) -> str | None: + if not path.is_file(): + return None + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return f"sha256:{digest.hexdigest()}" + + +def _version_hint(ref: str) -> str | None: + for part in reversed(ref.replace("-", ".").split(".")): + if len(part) > 1 and part[0].lower() == "v" and any(char.isdigit() for char in part[1:]): + return part + return None + + +def _object_or_empty(value: Any) -> dict[str, Any]: + return value if isinstance(value, dict) else {} + + +def _string_list(value: Any) -> list[str]: + if not isinstance(value, list): + return [] + return [item for item in value if isinstance(item, str)] + + +def _display_path(root: Path, path: Path) -> str: + try: + return str(path.resolve().relative_to(root.resolve())) + except ValueError: + return str(path.resolve()) + + def _extension_path_ref(root: Path, path: Path) -> str: try: return str(path.resolve().relative_to(root.resolve())) @@ -215,5 +487,9 @@ def _extension_path_ref(root: Path, path: Path) -> str: return str(path.resolve()) +def _now() -> str: + return datetime.now(timezone.utc).isoformat() + + def _timestamp() -> str: return datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") diff --git a/src/guide_board/retention.py b/src/guide_board/retention.py index ef01806..eab367c 100644 --- a/src/guide_board/retention.py +++ b/src/guide_board/retention.py @@ -45,6 +45,7 @@ def build_retention_summary( "report_refs": [ "reports/assessment-package.json", "reports/report.md", + "reports/submission-package.json", ], "artifact_retention": { "policy": plan["assessment_profile_snapshot"].get("retention_policy", {}), diff --git a/src/guide_board/runners.py b/src/guide_board/runners.py index 975321f..20e2336 100644 --- a/src/guide_board/runners.py +++ b/src/guide_board/runners.py @@ -45,6 +45,8 @@ def run_step( "runner_kind": "external", }, "artifact_refs": [], + "requirement_refs": [], + "metadata": _object_or_empty(entrypoint.get("metadata")), } if entrypoint["kind"] == "command": return _run_command(root, run_dir, run_id, plan, step, extension_path, entrypoint) @@ -63,6 +65,8 @@ def _no_runner_result(step: dict[str, Any]) -> dict[str, Any]: "runner_kind": None, }, "artifact_refs": [], + "requirement_refs": [], + "metadata": {}, } @@ -118,6 +122,8 @@ def _run_python_module( "error_type": type(exc).__name__, }, "artifact_refs": [], + "requirement_refs": [], + "metadata": _object_or_empty(entrypoint.get("metadata")), } if not isinstance(result, dict): raise ValidationError(f"{entrypoint['id']}: runner must return an object") @@ -126,6 +132,8 @@ def _run_python_module( "observations": result.get("observations", []), "facts": result.get("facts", {}), "artifact_refs": result.get("artifact_refs", []), + "requirement_refs": result.get("requirement_refs", []), + "metadata": _merge_metadata(entrypoint.get("metadata"), result.get("metadata")), } @@ -192,6 +200,8 @@ def _run_command( "command": command, }, "artifact_refs": [str(context_path.relative_to(run_dir))], + "requirement_refs": [], + "metadata": _object_or_empty(entrypoint.get("metadata")), } except subprocess.TimeoutExpired: return { @@ -206,6 +216,8 @@ def _run_command( "command": command, }, "artifact_refs": [str(context_path.relative_to(run_dir))], + "requirement_refs": [], + "metadata": _object_or_empty(entrypoint.get("metadata")), } parsed = _parse_runner_stdout(completed.stdout) @@ -225,6 +237,8 @@ def _run_command( "command": command, }, "artifact_refs": [str(context_path.relative_to(run_dir))], + "requirement_refs": [], + "metadata": _object_or_empty(entrypoint.get("metadata")), } facts = parsed.get("facts", {}) @@ -245,6 +259,9 @@ def _run_command( if not isinstance(artifact_refs, list): artifact_refs = [] artifact_refs.append(str(context_path.relative_to(run_dir))) + requirement_refs = parsed.get("requirement_refs", []) + if not isinstance(requirement_refs, list): + requirement_refs = [] result = parsed.get("result", "unknown") if completed.returncode != 0 and result in {"pass", "warning", "manual", "skipped"}: @@ -258,6 +275,8 @@ def _run_command( "observations": observations, "facts": facts, "artifact_refs": artifact_refs, + "requirement_refs": requirement_refs, + "metadata": _merge_metadata(entrypoint.get("metadata"), parsed.get("metadata")), } @@ -328,5 +347,17 @@ def _parse_runner_stdout(stdout: str) -> dict[str, Any] | None: return parsed +def _merge_metadata(*values: Any) -> dict[str, Any]: + merged: dict[str, Any] = {} + for value in values: + if isinstance(value, dict): + merged.update(value) + return merged + + +def _object_or_empty(value: Any) -> dict[str, Any]: + return value if isinstance(value, dict) else {} + + def _safe_id(value: str) -> str: return "".join(char if char.isalnum() or char in {"-", "_"} else "_" for char in value) diff --git a/tests/test_core.py b/tests/test_core.py index 932d295..60abc68 100644 --- a/tests/test_core.py +++ b/tests/test_core.py @@ -75,6 +75,17 @@ class CoreArchitectureTests(unittest.TestCase): plan["ordered_steps"][1]["requirement_refs"], ["guide-board.sample-readiness.v0.profile-shape"], ) + assert_valid(plan["source_lock"], "source-lock") + self.assertEqual(plan["source_lock"]["schema_version"], "guide-board.source-lock.v1") + self.assertEqual(plan["source_lock"]["framework_refs"], ["guide-board.sample-readiness.v0"]) + self.assertEqual(plan["source_lock"]["extension_refs"], ["sample-noop"]) + self.assertEqual( + plan["source_lock"]["profiles"]["target"]["snapshot_ref"], + "target-profile.snapshot.json", + ) + self.assertTrue(plan["source_lock"]["profiles"]["target"]["checksum"].startswith("sha256:")) + self.assertEqual(plan["source_lock"]["mapping_sets"][0]["id"], "sample-readiness-map") + self.assertTrue(plan["source_lock"]["mapping_sets"][0]["checksum"].startswith("sha256:")) def test_runs_external_extension_from_separate_repo(self) -> None: with TemporaryDirectory() as temporary_directory: @@ -237,8 +248,37 @@ class CoreArchitectureTests(unittest.TestCase): check_evidence["artifact_refs"], ["artifacts/sdk-fixture/native-result.json"], ) + self.assertEqual( + check_evidence["facts"]["source_metadata"]["runner"]["metadata"]["harness_version"], + "1.0.0", + ) + self.assertEqual( + check_evidence["facts"]["source_metadata"]["reported"]["native_result_id"], + "sdk-fixture-native-result", + ) self.assertEqual(mappings[0]["target_id"], "normalizer-plugin") self.assertEqual(assessment_package["summary"], {"pass": 1, "skipped": 1}) + self.assertEqual( + assessment_package["source_lock"]["metadata_hooks"]["runner_entrypoints"][0][ + "metadata" + ]["harness_id"], + "sdk-fixture-native-probe", + ) + submission_package = json.loads( + (run_dir / "reports" / "submission-package.json").read_text(encoding="utf-8") + ) + assert_valid(submission_package, "submission-package") + self.assertEqual(submission_package["source_lock"]["id"], "source-lock:sdk-fixture-assessment:sdk-fixture-target") + self.assertEqual( + submission_package["reported_metadata"][1]["metadata"]["reported"][ + "native_result_id" + ], + "sdk-fixture-native-result", + ) + self.assertEqual( + submission_package["artifact_manifest"][0]["checksum"], + assessment_package["artifact_manifest"][0]["checksum"], + ) def test_runs_sample_noop_assessment(self) -> None: with TemporaryDirectory() as temporary_directory: @@ -256,6 +296,7 @@ class CoreArchitectureTests(unittest.TestCase): self.assertTrue((run_dir / "normalized" / "evidence.json").exists()) self.assertTrue((run_dir / "reports" / "assessment-package.json").exists()) self.assertTrue((run_dir / "reports" / "report.md").exists()) + self.assertTrue((run_dir / "reports" / "submission-package.json").exists()) retention = json.loads( (run_dir / "retention-summary.json").read_text(encoding="utf-8") ) @@ -263,12 +304,26 @@ class CoreArchitectureTests(unittest.TestCase): result["retention_summary"], str(run_dir / "retention-summary.json"), ) + self.assertEqual( + result["submission_package"], + str(run_dir / "reports" / "submission-package.json"), + ) self.assertEqual(retention["summary"]["status"], "completed") self.assertEqual(retention["summary"]["artifact_count"], 0) + self.assertIn("reports/submission-package.json", retention["report_refs"]) self.assertEqual( retention["artifact_retention"]["policy"], {"raw_artifact_days": 0, "summary_days": 365}, ) + submission = json.loads( + (run_dir / "reports" / "submission-package.json").read_text(encoding="utf-8") + ) + assert_valid(submission, "submission-package") + self.assertEqual(submission["package_identity"]["target_profile_ref"], "sample-repository") + self.assertEqual( + [entry["path"] for entry in submission["reports"]], + ["reports/assessment-package.json", "reports/report.md"], + ) self.assertEqual( [run["run_id"] for run in list_retained_runs(Path(temporary_directory))], [result["run_id"]], diff --git a/workplans/GUIDE-BOARD-WP-0004-source-lock-and-submission-package-baseline.md b/workplans/GUIDE-BOARD-WP-0004-source-lock-and-submission-package-baseline.md index d65e513..c1d4fc6 100644 --- a/workplans/GUIDE-BOARD-WP-0004-source-lock-and-submission-package-baseline.md +++ b/workplans/GUIDE-BOARD-WP-0004-source-lock-and-submission-package-baseline.md @@ -4,12 +4,12 @@ type: workplan title: "Source Lock And Submission Package Baseline" repo: guide-board domain: markitect -status: active +status: completed owner: codex planning_priority: high planning_order: 4 created: "2026-05-15" -updated: "2026-05-15" +updated: "2026-05-16" state_hub_workstream_id: "6dd2832b-d1d9-43bc-ad5c-d16f399930dc" --- @@ -41,7 +41,7 @@ submission rules, and licensed or restricted assets remain extension-owned. ```task id: GUIDE-BOARD-WP-0004-T001 -status: todo +status: done priority: high state_hub_task_id: "d5a7a18f-941b-47b8-9992-2cb54bc5ad06" ``` @@ -55,11 +55,20 @@ Acceptance: - Keep the schema backward-compatible with existing retained runs. - Add tests for source lock shape and retained run compatibility. +Progress: + +- Added `docs/schemas/source-lock.schema.json`. +- Expanded run-plan source locks with framework, extension, mapping-set, + profile snapshot, policy-ref, authority, and metadata-hook records. +- Preserved the original `framework_refs` and `extension_refs` fields for + retained-run compatibility. +- Added tests for source-lock shape and older retained summary compatibility. + ## D4.2 - Harness And Extension Metadata Hooks ```task id: GUIDE-BOARD-WP-0004-T002 -status: todo +status: done priority: high state_hub_task_id: "7abd5a66-5784-41b9-a361-6572290923cc" ``` @@ -75,11 +84,20 @@ Acceptance: provide this metadata yet. - Cover the SDK fixture and at least one no-metadata extension path in tests. +Progress: + +- Added optional manifest metadata for extensions, authorities, frameworks, + runner entrypoints, and normalizers. +- Preserved runner and normalizer returned `metadata` and requirement refs. +- Recorded merged metadata under evidence `facts.source_metadata`. +- Updated `sdk-fixture` to exercise harness, test-suite, adapter, source URL, + and native result metadata while keeping `sample-noop` as a no-metadata path. + ## D4.3 - Submission Package Manifest ```task id: GUIDE-BOARD-WP-0004-T003 -status: todo +status: done priority: medium state_hub_task_id: "c54273d6-1fc2-4444-92cf-74f2a5e614ec" ``` @@ -95,11 +113,21 @@ Acceptance: packs. - Document how this differs from an authority-specific final submission. +Progress: + +- Added `docs/schemas/submission-package.schema.json`. +- Wrote `reports/submission-package.json` for each run. +- Included package identity, source lock checksum, report paths, normalized + output paths, profile snapshots, artifact manifest entries, reported + metadata, and the certification boundary. +- Exposed the submission manifest path in CLI/service run results and retained + report refs. + ## D4.4 - Documentation And Acceptance Tests ```task id: GUIDE-BOARD-WP-0004-T004 -status: todo +status: done priority: medium state_hub_task_id: "ad37baeb-973c-4399-96d0-c9cb7fc6b761" ``` @@ -113,6 +141,15 @@ Acceptance: - Include compatibility notes for older retained runs. - Keep the output paths aligned with existing CLI and service result retrieval. +Progress: + +- Updated assessment operations, extension SDK, architecture blueprint, and + README references. +- Added focused unit assertions for the sample and SDK fixture assessment + outputs. +- Kept retained run listing compatible with older `retention-summary.json` + files that do not reference a submission manifest. + ## Definition Of Done - Every new run writes a richer source lock and submission package manifest.