generated from coulomb/repo-seed
Add report fragments and export manifest
This commit is contained in:
@@ -42,9 +42,10 @@ The same CLI contracts are packaged by the container baseline. See
|
||||
[docs/CONTAINER.md](docs/CONTAINER.md). The dependency-light local API wraps
|
||||
those contracts for service and container operation; see
|
||||
[docs/LOCAL-SERVICE-API.md](docs/LOCAL-SERVICE-API.md).
|
||||
New runs write a richer `sources.lock.json` and
|
||||
`reports/submission-package.json` alongside the assessment package so reviewers
|
||||
can inspect source, metadata, artifact, and boundary references.
|
||||
New runs write a richer `sources.lock.json`,
|
||||
`reports/submission-package.json`, extension report fragments, and
|
||||
`exports/export-manifest.json` alongside the assessment package so reviewers can
|
||||
inspect source, metadata, artifact, export, and boundary references.
|
||||
|
||||
The `sample-noop` extension exercises the guide-board contracts without invoking
|
||||
an external harness. `sdk-fixture` demonstrates the extension SDK contracts for
|
||||
|
||||
@@ -653,6 +653,18 @@ building complex runtime code.
|
||||
- `reported_metadata`
|
||||
- `certification_boundary`
|
||||
|
||||
### `ExportManifest`
|
||||
|
||||
- `export_type`
|
||||
- `source_package_ref`
|
||||
- `source_lock_ref`
|
||||
- `summary`
|
||||
- `policy_summary`
|
||||
- `mapping_summary`
|
||||
- `report_fragments`
|
||||
- `counts`
|
||||
- `certification_boundary`
|
||||
|
||||
## Result Vocabulary
|
||||
|
||||
The evidence model should allow these statuses:
|
||||
@@ -740,9 +752,11 @@ runs/<run-id>/
|
||||
mappings.json
|
||||
reports/
|
||||
report.md
|
||||
fragments.json
|
||||
assessment-package.json
|
||||
submission-package.json
|
||||
exports/
|
||||
export-manifest.json
|
||||
```
|
||||
|
||||
## Container And Service Model
|
||||
@@ -836,6 +850,8 @@ hooks for runners and normalizers.
|
||||
6. Add container design after the CLI baseline is stable.
|
||||
7. Add optional service API around the CLI job model.
|
||||
8. Add OSCAL export and procedural evidence-pack support after the internal
|
||||
evidence model proves itself with executable extensions.
|
||||
evidence model proves itself with executable extensions. The first generic
|
||||
export is `exports/export-manifest.json`; authority-specific interchange
|
||||
remains extension-owned until the internal model is stable.
|
||||
|
||||
The first extension SDK contract is documented in `docs/EXTENSION-SDK.md`.
|
||||
|
||||
@@ -79,6 +79,7 @@ A completed CLI command prints a JSON result with:
|
||||
- `assessment_package`: JSON assessment package path,
|
||||
- `report`: Markdown report path,
|
||||
- `submission_package`: portable submission package manifest path,
|
||||
- `export_manifest`: portable generic JSON export manifest path,
|
||||
- `retention_summary`: compact durable summary path.
|
||||
|
||||
The output directory uses this contract:
|
||||
@@ -95,7 +96,9 @@ normalized/findings.json
|
||||
normalized/mappings.json
|
||||
reports/assessment-package.json
|
||||
reports/report.md
|
||||
reports/fragments.json
|
||||
reports/submission-package.json
|
||||
exports/export-manifest.json
|
||||
artifacts/
|
||||
```
|
||||
|
||||
@@ -107,6 +110,12 @@ raw artifact manifest, and repeats the certification boundary. It is a portable
|
||||
handoff manifest for preparation evidence, not an authority-specific final
|
||||
submission.
|
||||
|
||||
Extension report fragments are recorded in `reports/fragments.json`, embedded in
|
||||
`reports/assessment-package.json`, and rendered into the Markdown report.
|
||||
`exports/export-manifest.json` is the first generic portable export surface. It
|
||||
is derived from the assessment package and carries summary, policy, mapping,
|
||||
fragment, count, source-lock, and boundary references.
|
||||
|
||||
Use the retained run helpers for history:
|
||||
|
||||
```sh
|
||||
@@ -121,6 +130,10 @@ PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs
|
||||
PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs
|
||||
```
|
||||
|
||||
Trend summaries include status changes, unexpected finding deltas, unresolved
|
||||
review deltas, mapping target deltas, evidence result deltas, and a compact
|
||||
human-readable summary string for each target/assessment pair.
|
||||
|
||||
## Local Service Flow
|
||||
|
||||
Start the service from the guide-board repository:
|
||||
|
||||
@@ -62,7 +62,8 @@ The script:
|
||||
- builds `guide-board-core:smoke`,
|
||||
- mounts a host output directory at `/runs`,
|
||||
- runs the bundled sample assessment,
|
||||
- verifies that the expected run artifacts are present on the host.
|
||||
- verifies that the expected run artifacts, report fragments, submission
|
||||
manifest, and generic export manifest are present on the host.
|
||||
|
||||
Override the runtime, image name, or output directory when needed:
|
||||
|
||||
|
||||
@@ -83,6 +83,8 @@ The key runtime fields are:
|
||||
- `normalizers`: optional plug-ins that convert native runner output into the
|
||||
stable runner-result shape before evidence is written.
|
||||
- `mappings`: mapping set IDs under `mappings/<mapping-id>.json`.
|
||||
- `report_fragments`: optional Markdown file or Python module descriptors for
|
||||
extension-owned report content.
|
||||
- `certification_boundary`: explicit statement of what the extension does not
|
||||
certify.
|
||||
|
||||
@@ -209,6 +211,53 @@ to extension-owned mappings and writes normalized mapping records to:
|
||||
runs/<run-id>/normalized/mappings.json
|
||||
```
|
||||
|
||||
## Report Fragments
|
||||
|
||||
Extensions can contribute report fragments through `report_fragments`.
|
||||
|
||||
Static Markdown file:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "overview",
|
||||
"kind": "markdown_file",
|
||||
"path": "reports/overview.md",
|
||||
"title": "Overview"
|
||||
}
|
||||
```
|
||||
|
||||
Dynamic Python fragment:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "sdk-fixture-summary",
|
||||
"kind": "python_module",
|
||||
"module_path": "reports/sdk_fixture_summary.py",
|
||||
"callable": "build_fragment",
|
||||
"path": null,
|
||||
"title": "SDK Fixture Summary"
|
||||
}
|
||||
```
|
||||
|
||||
Fragment paths are resolved relative to the extension root and must stay inside
|
||||
that root. A Python fragment receives `root`, `run_dir`, `run_id`, `plan`,
|
||||
`evidence`, `findings`, `mappings`, `assessment_package`, `policy_summary`,
|
||||
`source_lock`, `extension_path`, and `report_fragment`.
|
||||
|
||||
It returns:
|
||||
|
||||
```python
|
||||
def build_fragment(context: dict) -> dict:
|
||||
return {
|
||||
"markdown": "### Extension Summary\n\n- evidence items: 2",
|
||||
"structured": {"evidence_count": 2},
|
||||
}
|
||||
```
|
||||
|
||||
Fragments are written to `reports/fragments.json`, embedded in the assessment
|
||||
package, rendered in `reports/report.md`, and summarized in
|
||||
`exports/export-manifest.json`.
|
||||
|
||||
## Evidence Request Sets
|
||||
|
||||
Procedural and hybrid compliance extensions may include evidence request sets
|
||||
@@ -402,9 +451,9 @@ profiles.
|
||||
|
||||
## Source Lock And Submission Package
|
||||
|
||||
Every new run writes `sources.lock.json` and
|
||||
`reports/submission-package.json`. Extension authors should treat source
|
||||
metadata as part of the evidence contract:
|
||||
Every new run writes `sources.lock.json`, `reports/submission-package.json`,
|
||||
and the generic portable export manifest at `exports/export-manifest.json`.
|
||||
Extension authors should treat source metadata as part of the evidence contract:
|
||||
|
||||
- declare extension, authority, framework, runner, and normalizer metadata in
|
||||
`extension.json` when it is static;
|
||||
|
||||
@@ -98,8 +98,8 @@ errors.
|
||||
### `GET /runs/{job_id}/reports`
|
||||
|
||||
Returns the Markdown report content, assessment package JSON, retention summary,
|
||||
submission package JSON when present, and their filesystem paths after a job has
|
||||
succeeded.
|
||||
submission package JSON, export manifest JSON when present, and their filesystem
|
||||
paths after a job has succeeded.
|
||||
|
||||
### `GET /retained-runs`
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@
|
||||
"waivers",
|
||||
"challenges",
|
||||
"exclusions",
|
||||
"report_fragments",
|
||||
"certification_boundary",
|
||||
"created_at"
|
||||
],
|
||||
@@ -38,6 +39,7 @@
|
||||
"waivers": { "type": "array", "items": { "type": "object" } },
|
||||
"challenges": { "type": "array", "items": { "type": "object" } },
|
||||
"exclusions": { "type": "array", "items": { "type": "object" } },
|
||||
"report_fragments": { "type": "array", "items": { "type": "object" } },
|
||||
"certification_boundary": { "type": "string" },
|
||||
"created_at": { "type": "string" }
|
||||
}
|
||||
|
||||
36
docs/schemas/export-manifest.schema.json
Normal file
36
docs/schemas/export-manifest.schema.json
Normal file
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Guide Board Export Manifest",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"schema_version",
|
||||
"export_type",
|
||||
"run_id",
|
||||
"created_at",
|
||||
"source_package_ref",
|
||||
"source_lock_ref",
|
||||
"summary",
|
||||
"policy_summary",
|
||||
"mapping_summary",
|
||||
"report_fragments",
|
||||
"counts",
|
||||
"certification_boundary"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"schema_version": { "type": "string" },
|
||||
"export_type": { "type": "string" },
|
||||
"run_id": { "type": "string" },
|
||||
"created_at": { "type": "string" },
|
||||
"source_package_ref": { "type": "string" },
|
||||
"source_lock_ref": { "type": "string" },
|
||||
"summary": { "type": "object" },
|
||||
"policy_summary": { "type": "object" },
|
||||
"mapping_summary": { "type": "object" },
|
||||
"report_fragments": { "type": "array", "items": { "type": "object" } },
|
||||
"counts": { "type": "object" },
|
||||
"certification_boundary": { "type": "string" }
|
||||
}
|
||||
}
|
||||
@@ -142,7 +142,23 @@
|
||||
}
|
||||
},
|
||||
"mappings": { "type": "array", "items": { "type": "string" } },
|
||||
"report_fragments": { "type": "array", "items": { "type": "string" } },
|
||||
"report_fragments": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": ["string", "object"],
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "kind"],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"kind": { "type": "string", "enum": ["markdown_file", "python_module"] },
|
||||
"path": { "type": ["string", "null"] },
|
||||
"module_path": { "type": ["string", "null"] },
|
||||
"callable": { "type": ["string", "null"] },
|
||||
"title": { "type": ["string", "null"] },
|
||||
"description": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
},
|
||||
"dependencies": { "type": "array", "items": { "type": "string" } },
|
||||
"restricted_assets": { "type": "array", "items": { "type": "string" } },
|
||||
"certification_boundary": { "type": "string" }
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
"created_at": { "type": "string" },
|
||||
"summary": { "type": "object" },
|
||||
"report_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"export_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"artifact_retention": { "type": "object" }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -74,7 +74,17 @@
|
||||
"mappings": [
|
||||
"sdk-fixture-map"
|
||||
],
|
||||
"report_fragments": [],
|
||||
"report_fragments": [
|
||||
{
|
||||
"id": "sdk-fixture-summary",
|
||||
"kind": "python_module",
|
||||
"module_path": "reports/sdk_fixture_summary.py",
|
||||
"callable": "build_fragment",
|
||||
"path": null,
|
||||
"title": "SDK Fixture Summary",
|
||||
"description": "Summarizes SDK fixture evidence for report fragment tests."
|
||||
}
|
||||
],
|
||||
"dependencies": [],
|
||||
"restricted_assets": [],
|
||||
"certification_boundary": "SDK fixture only. It does not certify any product, process, or repository."
|
||||
|
||||
26
extensions/sdk-fixture/reports/sdk_fixture_summary.py
Normal file
26
extensions/sdk-fixture/reports/sdk_fixture_summary.py
Normal file
@@ -0,0 +1,26 @@
|
||||
"""Report fragment for the SDK fixture extension."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def build_fragment(context: dict) -> dict:
|
||||
evidence_count = len(context["evidence"])
|
||||
finding_count = len(context["findings"])
|
||||
source_lock = context["source_lock"]
|
||||
markdown = "\n".join(
|
||||
[
|
||||
"### SDK Fixture Summary",
|
||||
"",
|
||||
f"- evidence items: {evidence_count}",
|
||||
f"- findings: {finding_count}",
|
||||
f"- source lock: {source_lock.get('id')}",
|
||||
]
|
||||
)
|
||||
return {
|
||||
"markdown": markdown,
|
||||
"structured": {
|
||||
"evidence_count": evidence_count,
|
||||
"finding_count": finding_count,
|
||||
"source_lock_ref": source_lock.get("id"),
|
||||
},
|
||||
}
|
||||
@@ -42,7 +42,9 @@ for path in \
|
||||
"$RUNS_DIR/sample-noop/normalized/mappings.json" \
|
||||
"$RUNS_DIR/sample-noop/reports/assessment-package.json" \
|
||||
"$RUNS_DIR/sample-noop/reports/report.md" \
|
||||
"$RUNS_DIR/sample-noop/reports/submission-package.json"
|
||||
"$RUNS_DIR/sample-noop/reports/fragments.json" \
|
||||
"$RUNS_DIR/sample-noop/reports/submission-package.json" \
|
||||
"$RUNS_DIR/sample-noop/exports/export-manifest.json"
|
||||
do
|
||||
if [ ! -f "$path" ]; then
|
||||
echo "ERROR: expected artifact missing: $path" >&2
|
||||
|
||||
@@ -8,11 +8,13 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from guide_board.artifacts import build_artifact_manifest, build_submission_manifest
|
||||
from guide_board.exports import build_export_manifest
|
||||
from guide_board.io import load_json, write_json
|
||||
from guide_board.mapping import build_mapping_records, summarize_mappings
|
||||
from guide_board.normalizers import normalize_step_result
|
||||
from guide_board.planning import build_run_plan
|
||||
from guide_board.policy import apply_policy
|
||||
from guide_board.reports import build_report_fragments, markdown_for_fragments
|
||||
from guide_board.retention import build_retention_summary
|
||||
from guide_board.runners import run_step
|
||||
from guide_board.schema import assert_valid
|
||||
@@ -64,7 +66,19 @@ def run_assessment(
|
||||
applied_exclusions,
|
||||
created_at,
|
||||
)
|
||||
report_fragments = build_report_fragments(
|
||||
root,
|
||||
run_dir,
|
||||
run_id,
|
||||
plan,
|
||||
evidence,
|
||||
findings,
|
||||
mapping_records,
|
||||
assessment_package,
|
||||
)
|
||||
assessment_package["report_fragments"] = report_fragments
|
||||
assert_valid(assessment_package, "assessment-package")
|
||||
export_manifest = build_export_manifest(assessment_package)
|
||||
|
||||
run_metadata = {
|
||||
"id": run_id,
|
||||
@@ -85,6 +99,7 @@ def run_assessment(
|
||||
findings,
|
||||
mapping_records,
|
||||
assessment_package,
|
||||
export_manifest,
|
||||
retention_summary,
|
||||
)
|
||||
return {
|
||||
@@ -94,6 +109,7 @@ def run_assessment(
|
||||
"assessment_package": str(run_dir / "reports" / "assessment-package.json"),
|
||||
"report": str(run_dir / "reports" / "report.md"),
|
||||
"submission_package": str(run_dir / "reports" / "submission-package.json"),
|
||||
"export_manifest": str(run_dir / "exports" / "export-manifest.json"),
|
||||
"retention_summary": str(run_dir / "retention-summary.json"),
|
||||
}
|
||||
|
||||
@@ -419,6 +435,7 @@ def _assessment_package(
|
||||
"waivers": applied_waivers,
|
||||
"challenges": applied_challenges,
|
||||
"exclusions": applied_exclusions,
|
||||
"report_fragments": [],
|
||||
"certification_boundary": "Guide Board produces preparation evidence only and does not issue certifications or audit assurance.",
|
||||
"created_at": created_at,
|
||||
}
|
||||
@@ -432,6 +449,7 @@ def _write_run_directory(
|
||||
findings: list[dict[str, Any]],
|
||||
mapping_records: list[dict[str, Any]],
|
||||
assessment_package: dict[str, Any],
|
||||
export_manifest: dict[str, Any],
|
||||
retention_summary: dict[str, Any],
|
||||
) -> None:
|
||||
write_json(run_dir / "run.json", run_metadata)
|
||||
@@ -447,6 +465,8 @@ def _write_run_directory(
|
||||
write_json(run_dir / "normalized" / "findings.json", {"findings": findings})
|
||||
write_json(run_dir / "normalized" / "mappings.json", {"mappings": mapping_records})
|
||||
write_json(run_dir / "reports" / "assessment-package.json", assessment_package)
|
||||
write_json(run_dir / "reports" / "fragments.json", {"fragments": assessment_package["report_fragments"]})
|
||||
write_json(run_dir / "exports" / "export-manifest.json", export_manifest)
|
||||
(run_dir / "reports").mkdir(parents=True, exist_ok=True)
|
||||
(run_dir / "reports" / "report.md").write_text(
|
||||
_markdown_report(run_metadata, assessment_package),
|
||||
@@ -471,6 +491,7 @@ def _markdown_report(run_metadata: dict[str, Any], package: dict[str, Any]) -> s
|
||||
mapping_lines = _mapping_summary_lines(package)
|
||||
policy_lines = _policy_summary_lines(package)
|
||||
review_lines = _review_summary_lines(package)
|
||||
fragment_lines = markdown_for_fragments(package.get("report_fragments", []))
|
||||
|
||||
return "\n".join(
|
||||
[
|
||||
@@ -496,6 +517,10 @@ def _markdown_report(run_metadata: dict[str, Any], package: dict[str, Any]) -> s
|
||||
"",
|
||||
review_lines,
|
||||
"",
|
||||
"## Extension Fragments",
|
||||
"",
|
||||
fragment_lines,
|
||||
"",
|
||||
"## Boundary",
|
||||
"",
|
||||
package["certification_boundary"],
|
||||
|
||||
50
src/guide_board/exports.py
Normal file
50
src/guide_board/exports.py
Normal file
@@ -0,0 +1,50 @@
|
||||
"""Portable export builders derived from assessment packages."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from guide_board.schema import assert_valid
|
||||
|
||||
|
||||
def build_export_manifest(assessment_package: dict[str, Any]) -> dict[str, Any]:
|
||||
manifest = {
|
||||
"id": f"export-manifest:{assessment_package['run_id']}",
|
||||
"schema_version": "guide-board.export-manifest.v1",
|
||||
"export_type": "guide-board.generic-json.v1",
|
||||
"run_id": assessment_package["run_id"],
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
"source_package_ref": "reports/assessment-package.json",
|
||||
"source_lock_ref": "sources.lock.json",
|
||||
"summary": assessment_package.get("summary", {}),
|
||||
"policy_summary": assessment_package.get("policy_summary", {}),
|
||||
"mapping_summary": assessment_package.get("mapping_summary", {}),
|
||||
"report_fragments": [
|
||||
_export_fragment(fragment)
|
||||
for fragment in assessment_package.get("report_fragments", [])
|
||||
if isinstance(fragment, dict)
|
||||
],
|
||||
"counts": {
|
||||
"evidence_refs": len(assessment_package.get("evidence_refs", [])),
|
||||
"findings": len(assessment_package.get("findings", [])),
|
||||
"artifacts": len(assessment_package.get("artifact_manifest", [])),
|
||||
"waivers": len(assessment_package.get("waivers", [])),
|
||||
"challenges": len(assessment_package.get("challenges", [])),
|
||||
"exclusions": len(assessment_package.get("exclusions", [])),
|
||||
"report_fragments": len(assessment_package.get("report_fragments", [])),
|
||||
},
|
||||
"certification_boundary": assessment_package["certification_boundary"],
|
||||
}
|
||||
assert_valid(manifest, "export-manifest")
|
||||
return manifest
|
||||
|
||||
|
||||
def _export_fragment(fragment: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"id": fragment.get("id"),
|
||||
"extension_id": fragment.get("extension_id"),
|
||||
"title": fragment.get("title"),
|
||||
"kind": fragment.get("kind"),
|
||||
"structured": fragment.get("structured", {}),
|
||||
}
|
||||
204
src/guide_board/reports.py
Normal file
204
src/guide_board/reports.py
Normal file
@@ -0,0 +1,204 @@
|
||||
"""Report fragment loading for extension-contributed report content."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import Any
|
||||
|
||||
from guide_board.errors import ValidationError
|
||||
from guide_board.io import load_json
|
||||
|
||||
|
||||
def build_report_fragments(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
evidence: list[dict[str, Any]],
|
||||
findings: list[dict[str, Any]],
|
||||
mapping_records: list[dict[str, Any]],
|
||||
assessment_package: dict[str, Any],
|
||||
) -> list[dict[str, Any]]:
|
||||
fragments: list[dict[str, Any]] = []
|
||||
for extension in plan["extension_snapshots"]:
|
||||
extension_path = _snapshot_path(root, extension)
|
||||
manifest = load_json(extension_path / "extension.json")
|
||||
for descriptor in manifest.get("report_fragments", []):
|
||||
fragment = _load_fragment(
|
||||
root,
|
||||
run_dir,
|
||||
run_id,
|
||||
plan,
|
||||
evidence,
|
||||
findings,
|
||||
mapping_records,
|
||||
assessment_package,
|
||||
extension,
|
||||
extension_path,
|
||||
descriptor,
|
||||
)
|
||||
if fragment is not None:
|
||||
fragments.append(fragment)
|
||||
return fragments
|
||||
|
||||
|
||||
def markdown_for_fragments(fragments: list[dict[str, Any]]) -> str:
|
||||
markdown_blocks = [
|
||||
fragment.get("markdown", "")
|
||||
for fragment in fragments
|
||||
if isinstance(fragment.get("markdown"), str) and fragment.get("markdown")
|
||||
]
|
||||
if not markdown_blocks:
|
||||
return "- no extension report fragments"
|
||||
return "\n\n".join(markdown_blocks)
|
||||
|
||||
|
||||
def _load_fragment(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
evidence: list[dict[str, Any]],
|
||||
findings: list[dict[str, Any]],
|
||||
mapping_records: list[dict[str, Any]],
|
||||
assessment_package: dict[str, Any],
|
||||
extension: dict[str, Any],
|
||||
extension_path: Path,
|
||||
descriptor: Any,
|
||||
) -> dict[str, Any] | None:
|
||||
if isinstance(descriptor, str):
|
||||
descriptor = {
|
||||
"id": descriptor,
|
||||
"kind": "markdown_file",
|
||||
"path": f"reports/{descriptor}.md",
|
||||
"title": descriptor,
|
||||
}
|
||||
if not isinstance(descriptor, dict):
|
||||
return None
|
||||
|
||||
if descriptor["kind"] == "markdown_file":
|
||||
markdown = _load_markdown_fragment(extension_path, descriptor)
|
||||
return _fragment_record(extension["id"], descriptor, markdown, {})
|
||||
if descriptor["kind"] == "python_module":
|
||||
result = _run_python_fragment(
|
||||
root,
|
||||
run_dir,
|
||||
run_id,
|
||||
plan,
|
||||
evidence,
|
||||
findings,
|
||||
mapping_records,
|
||||
assessment_package,
|
||||
extension_path,
|
||||
descriptor,
|
||||
)
|
||||
return _fragment_record(
|
||||
extension["id"],
|
||||
descriptor,
|
||||
result.get("markdown", ""),
|
||||
_object_or_empty(result.get("structured")),
|
||||
)
|
||||
raise ValidationError(f"{descriptor['id']}: unsupported report fragment kind")
|
||||
|
||||
|
||||
def _load_markdown_fragment(extension_path: Path, descriptor: dict[str, Any]) -> str:
|
||||
raw_path = descriptor.get("path") or f"reports/{descriptor['id']}.md"
|
||||
fragment_path = _safe_extension_path(extension_path, raw_path, descriptor["id"])
|
||||
if not fragment_path.is_file():
|
||||
raise ValidationError(f"{descriptor['id']}: report fragment not found: {raw_path}")
|
||||
return fragment_path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _run_python_fragment(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
evidence: list[dict[str, Any]],
|
||||
findings: list[dict[str, Any]],
|
||||
mapping_records: list[dict[str, Any]],
|
||||
assessment_package: dict[str, Any],
|
||||
extension_path: Path,
|
||||
descriptor: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
module_path = descriptor.get("module_path")
|
||||
callable_name = descriptor.get("callable")
|
||||
if not module_path or not callable_name:
|
||||
raise ValidationError(
|
||||
f"{descriptor['id']}: python_module report fragments need module_path and callable"
|
||||
)
|
||||
|
||||
module_file = _safe_extension_path(extension_path, module_path, descriptor["id"])
|
||||
module = _load_module(module_file, descriptor["id"])
|
||||
fragment_callable = getattr(module, callable_name, None)
|
||||
if not callable(fragment_callable):
|
||||
raise ValidationError(f"{descriptor['id']}: callable {callable_name!r} was not found")
|
||||
|
||||
context = {
|
||||
"root": str(root),
|
||||
"run_dir": str(run_dir),
|
||||
"run_id": run_id,
|
||||
"plan": plan,
|
||||
"evidence": evidence,
|
||||
"findings": findings,
|
||||
"mappings": mapping_records,
|
||||
"assessment_package": assessment_package,
|
||||
"policy_summary": assessment_package.get("policy_summary", {}),
|
||||
"source_lock": assessment_package.get("source_lock", {}),
|
||||
"extension_path": str(extension_path),
|
||||
"report_fragment": descriptor,
|
||||
}
|
||||
result = fragment_callable(context)
|
||||
if not isinstance(result, dict):
|
||||
raise ValidationError(f"{descriptor['id']}: report fragment must return an object")
|
||||
return result
|
||||
|
||||
|
||||
def _fragment_record(
|
||||
extension_id: str,
|
||||
descriptor: dict[str, Any],
|
||||
markdown: str,
|
||||
structured: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"id": descriptor["id"],
|
||||
"extension_id": extension_id,
|
||||
"title": descriptor.get("title") or descriptor["id"],
|
||||
"kind": descriptor["kind"],
|
||||
"markdown": markdown if isinstance(markdown, str) else "",
|
||||
"structured": structured,
|
||||
}
|
||||
|
||||
|
||||
def _safe_extension_path(extension_path: Path, raw_path: str, fragment_id: str) -> Path:
|
||||
path = (extension_path / raw_path).resolve()
|
||||
try:
|
||||
path.relative_to(extension_path.resolve())
|
||||
except ValueError as exc:
|
||||
raise ValidationError(
|
||||
f"{fragment_id}: report fragment path must stay inside the extension directory"
|
||||
) from exc
|
||||
return path
|
||||
|
||||
|
||||
def _load_module(path: Path, fragment_id: str) -> ModuleType:
|
||||
if not path.exists():
|
||||
raise ValidationError(f"{fragment_id}: module not found: {path}")
|
||||
module_name = f"_guide_board_report_fragment_{fragment_id.replace('-', '_')}"
|
||||
spec = importlib.util.spec_from_file_location(module_name, path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise ValidationError(f"{fragment_id}: unable to load module from {path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path:
|
||||
path = Path(extension["path"])
|
||||
return path if path.is_absolute() else root / path
|
||||
|
||||
|
||||
def _object_or_empty(value: Any) -> dict[str, Any]:
|
||||
return value if isinstance(value, dict) else {}
|
||||
@@ -49,8 +49,12 @@ def build_retention_summary(
|
||||
"report_refs": [
|
||||
"reports/assessment-package.json",
|
||||
"reports/report.md",
|
||||
"reports/fragments.json",
|
||||
"reports/submission-package.json",
|
||||
],
|
||||
"export_refs": [
|
||||
"exports/export-manifest.json",
|
||||
],
|
||||
"artifact_retention": {
|
||||
"policy": plan["assessment_profile_snapshot"].get("retention_policy", {}),
|
||||
"output_artifact_retention": plan["assessment_profile_snapshot"]
|
||||
@@ -200,6 +204,7 @@ def _run_projection(run: dict[str, Any]) -> dict[str, Any]:
|
||||
"status": summary.get("status", "unknown"),
|
||||
"unexpected_findings": _summary_int(summary, "unexpected_findings"),
|
||||
"finding_count": _summary_int(summary, "finding_count"),
|
||||
"mapping_target_count": _summary_int(summary, "mapping_target_count"),
|
||||
"artifact_count": _summary_int(summary, "artifact_count"),
|
||||
"challenged_findings": _summary_int(summary, "challenged_findings"),
|
||||
"authority_exclusions": _summary_int(summary, "authority_exclusions"),
|
||||
@@ -217,12 +222,18 @@ def _trend_between(
|
||||
return {
|
||||
"direction": "insufficient-history",
|
||||
"status_changed": False,
|
||||
"status_change": {
|
||||
"from": None,
|
||||
"to": _status_for(latest),
|
||||
},
|
||||
"unexpected_findings_delta": 0,
|
||||
"finding_count_delta": 0,
|
||||
"artifact_count_delta": 0,
|
||||
"unresolved_review_items_delta": 0,
|
||||
"evidence_result_deltas": {},
|
||||
}
|
||||
"artifact_count_delta": 0,
|
||||
"unresolved_review_items_delta": 0,
|
||||
"mapping_target_count_delta": 0,
|
||||
"evidence_result_deltas": {},
|
||||
"summary_text": "No previous retained run is available for comparison.",
|
||||
}
|
||||
|
||||
previous_summary = previous.get("summary", {})
|
||||
latest_summary = latest.get("summary", {})
|
||||
@@ -242,17 +253,34 @@ def _trend_between(
|
||||
review_delta = _summary_int(latest_summary, "unresolved_review_items") - _summary_int(
|
||||
previous_summary, "unresolved_review_items"
|
||||
)
|
||||
mapping_target_delta = _summary_int(latest_summary, "mapping_target_count") - _summary_int(
|
||||
previous_summary, "mapping_target_count"
|
||||
)
|
||||
previous_status = _status_for(previous)
|
||||
latest_status = _status_for(latest)
|
||||
direction = _trend_direction(previous_status, latest_status, unexpected_delta)
|
||||
|
||||
return {
|
||||
"direction": _trend_direction(previous_status, latest_status, unexpected_delta),
|
||||
"direction": direction,
|
||||
"status_changed": previous_status != latest_status,
|
||||
"status_change": {
|
||||
"from": previous_status,
|
||||
"to": latest_status,
|
||||
},
|
||||
"unexpected_findings_delta": unexpected_delta,
|
||||
"finding_count_delta": finding_delta,
|
||||
"artifact_count_delta": artifact_delta,
|
||||
"unresolved_review_items_delta": review_delta,
|
||||
"mapping_target_count_delta": mapping_target_delta,
|
||||
"evidence_result_deltas": evidence_deltas,
|
||||
"summary_text": _trend_summary_text(
|
||||
direction,
|
||||
previous_status,
|
||||
latest_status,
|
||||
unexpected_delta,
|
||||
review_delta,
|
||||
mapping_target_delta,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -294,6 +322,24 @@ def _summary_int(summary: dict[str, Any], key: str) -> int:
|
||||
return value if isinstance(value, int) and not isinstance(value, bool) else 0
|
||||
|
||||
|
||||
def _trend_summary_text(
|
||||
direction: str,
|
||||
previous_status: str,
|
||||
latest_status: str,
|
||||
unexpected_delta: int,
|
||||
review_delta: int,
|
||||
mapping_target_delta: int,
|
||||
) -> str:
|
||||
parts = [
|
||||
f"Trend {direction}",
|
||||
f"status {previous_status} -> {latest_status}",
|
||||
f"unexpected findings delta {unexpected_delta}",
|
||||
f"unresolved review delta {review_delta}",
|
||||
f"mapping target delta {mapping_target_delta}",
|
||||
]
|
||||
return "; ".join(parts) + "."
|
||||
|
||||
|
||||
def _dict_deltas(previous: Any, latest: Any) -> dict[str, int]:
|
||||
previous_dict = previous if isinstance(previous, dict) else {}
|
||||
latest_dict = latest if isinstance(latest, dict) else {}
|
||||
|
||||
@@ -285,6 +285,12 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
if isinstance(submission_value, str) and submission_value
|
||||
else None
|
||||
)
|
||||
export_value = result.get("export_manifest")
|
||||
export_path = (
|
||||
Path(export_value)
|
||||
if isinstance(export_value, str) and export_value
|
||||
else None
|
||||
)
|
||||
try:
|
||||
report_markdown = report_path.read_text(encoding="utf-8")
|
||||
assessment_package = load_json(package_path)
|
||||
@@ -294,6 +300,11 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
if submission_path is not None and submission_path.is_file()
|
||||
else None
|
||||
)
|
||||
export_manifest = (
|
||||
load_json(export_path)
|
||||
if export_path is not None and export_path.is_file()
|
||||
else None
|
||||
)
|
||||
except OSError as exc:
|
||||
raise HttpProblem(404, f"run report artifact is missing: {exc}") from exc
|
||||
|
||||
@@ -307,6 +318,7 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
"assessment_package": str(package_path),
|
||||
"retention_summary": str(retention_path),
|
||||
"submission_package": str(submission_path) if submission_path is not None else None,
|
||||
"export_manifest": str(export_path) if export_path is not None else None,
|
||||
},
|
||||
"report": {
|
||||
"path": str(report_path),
|
||||
@@ -324,6 +336,10 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
"path": str(submission_path) if submission_package else None,
|
||||
"json": submission_package,
|
||||
},
|
||||
"export_manifest": {
|
||||
"path": str(export_path) if export_manifest else None,
|
||||
"json": export_manifest,
|
||||
},
|
||||
}
|
||||
|
||||
def _retained_runs(self, query: dict[str, str]) -> dict[str, Any]:
|
||||
|
||||
17
tests/golden/export-manifest-shape.json
Normal file
17
tests/golden/export-manifest-shape.json
Normal file
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"top_level_keys": [
|
||||
"certification_boundary",
|
||||
"counts",
|
||||
"created_at",
|
||||
"export_type",
|
||||
"id",
|
||||
"mapping_summary",
|
||||
"policy_summary",
|
||||
"report_fragments",
|
||||
"run_id",
|
||||
"schema_version",
|
||||
"source_lock_ref",
|
||||
"source_package_ref",
|
||||
"summary"
|
||||
]
|
||||
}
|
||||
5
tests/golden/sdk-fixture-report-fragment.md
Normal file
5
tests/golden/sdk-fixture-report-fragment.md
Normal file
@@ -0,0 +1,5 @@
|
||||
### SDK Fixture Summary
|
||||
|
||||
- evidence items: 2
|
||||
- findings: 0
|
||||
- source lock: source-lock:sdk-fixture-assessment:sdk-fixture-target
|
||||
@@ -230,6 +230,10 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
assessment_package = json.loads(
|
||||
(run_dir / "reports" / "assessment-package.json").read_text(encoding="utf-8")
|
||||
)
|
||||
report = (run_dir / "reports" / "report.md").read_text(encoding="utf-8")
|
||||
export_manifest = json.loads(
|
||||
(run_dir / "exports" / "export-manifest.json").read_text(encoding="utf-8")
|
||||
)
|
||||
|
||||
self.assertEqual(result["status"], "completed")
|
||||
self.assertEqual(plan["extension_snapshots"][0]["source"], "external")
|
||||
@@ -259,6 +263,21 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
)
|
||||
self.assertEqual(mappings[0]["target_id"], "normalizer-plugin")
|
||||
self.assertEqual(assessment_package["summary"], {"pass": 1, "skipped": 1})
|
||||
self.assertEqual(
|
||||
assessment_package["report_fragments"][0]["markdown"],
|
||||
(ROOT / "tests" / "golden" / "sdk-fixture-report-fragment.md")
|
||||
.read_text(encoding="utf-8")
|
||||
.rstrip(),
|
||||
)
|
||||
self.assertIn("### SDK Fixture Summary", report)
|
||||
assert_valid(export_manifest, "export-manifest")
|
||||
export_shape = load_json(ROOT / "tests" / "golden" / "export-manifest-shape.json")
|
||||
self.assertEqual(sorted(export_manifest), export_shape["top_level_keys"])
|
||||
self.assertEqual(export_manifest["counts"]["report_fragments"], 1)
|
||||
self.assertEqual(
|
||||
export_manifest["report_fragments"][0]["structured"]["evidence_count"],
|
||||
2,
|
||||
)
|
||||
self.assertEqual(
|
||||
assessment_package["source_lock"]["metadata_hooks"]["runner_entrypoints"][0][
|
||||
"metadata"
|
||||
@@ -297,7 +316,9 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
self.assertTrue((run_dir / "normalized" / "evidence.json").exists())
|
||||
self.assertTrue((run_dir / "reports" / "assessment-package.json").exists())
|
||||
self.assertTrue((run_dir / "reports" / "report.md").exists())
|
||||
self.assertTrue((run_dir / "reports" / "fragments.json").exists())
|
||||
self.assertTrue((run_dir / "reports" / "submission-package.json").exists())
|
||||
self.assertTrue((run_dir / "exports" / "export-manifest.json").exists())
|
||||
retention = json.loads(
|
||||
(run_dir / "retention-summary.json").read_text(encoding="utf-8")
|
||||
)
|
||||
@@ -309,9 +330,14 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
result["submission_package"],
|
||||
str(run_dir / "reports" / "submission-package.json"),
|
||||
)
|
||||
self.assertEqual(
|
||||
result["export_manifest"],
|
||||
str(run_dir / "exports" / "export-manifest.json"),
|
||||
)
|
||||
self.assertEqual(retention["summary"]["status"], "completed")
|
||||
self.assertEqual(retention["summary"]["artifact_count"], 0)
|
||||
self.assertIn("reports/submission-package.json", retention["report_refs"])
|
||||
self.assertIn("exports/export-manifest.json", retention["export_refs"])
|
||||
self.assertEqual(
|
||||
retention["artifact_retention"]["policy"],
|
||||
{"raw_artifact_days": 0, "summary_days": 365},
|
||||
@@ -463,6 +489,10 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
reports["submission_package"]["json"]["run_id"],
|
||||
status["result"]["run_id"],
|
||||
)
|
||||
self.assertEqual(
|
||||
reports["export_manifest"]["json"]["run_id"],
|
||||
status["result"]["run_id"],
|
||||
)
|
||||
finally:
|
||||
service.stop()
|
||||
|
||||
@@ -552,7 +582,13 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
self.assertEqual(group["previous_run"]["run_id"], "run-old")
|
||||
self.assertEqual(group["trend"]["direction"], "improved")
|
||||
self.assertTrue(group["trend"]["status_changed"])
|
||||
self.assertEqual(
|
||||
group["trend"]["status_change"],
|
||||
{"from": "blocked", "to": "completed"},
|
||||
)
|
||||
self.assertEqual(group["trend"]["unexpected_findings_delta"], -1)
|
||||
self.assertEqual(group["trend"]["mapping_target_count_delta"], 0)
|
||||
self.assertIn("Trend improved", group["trend"]["summary_text"])
|
||||
self.assertEqual(
|
||||
group["trend"]["evidence_result_deltas"],
|
||||
{"blocked": -1, "manual": 1, "skipped": 1},
|
||||
|
||||
@@ -4,12 +4,12 @@ type: workplan
|
||||
title: "Report And Export Maturity"
|
||||
repo: guide-board
|
||||
domain: markitect
|
||||
status: active
|
||||
status: completed
|
||||
owner: codex
|
||||
planning_priority: medium
|
||||
planning_order: 7
|
||||
created: "2026-05-15"
|
||||
updated: "2026-05-15"
|
||||
updated: "2026-05-16"
|
||||
state_hub_workstream_id: "ef9351d2-e99c-470e-aeec-f17aa51eae14"
|
||||
---
|
||||
|
||||
@@ -41,7 +41,7 @@ extension-provided fragments or exporters.
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0007-T001
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "bf3fe163-b06d-4c2e-9b45-31721864e1f2"
|
||||
```
|
||||
@@ -55,11 +55,19 @@ Acceptance:
|
||||
summary, and source lock data.
|
||||
- Add a fixture fragment and tests.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added Markdown file and Python module report fragment descriptors.
|
||||
- Loaded fragment paths safely from extension roots.
|
||||
- Added fragment context for assessment package, evidence, findings, mappings,
|
||||
policy summary, and source lock data.
|
||||
- Added an SDK fixture Python report fragment and focused tests.
|
||||
|
||||
## D7.2 - Portable Export Formats
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0007-T002
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "fda51e62-98aa-408e-a057-4db40fe7c644"
|
||||
```
|
||||
@@ -72,11 +80,20 @@ Acceptance:
|
||||
- Preserve certification boundary and source lock references in each export.
|
||||
- Document which exports are generic and which must remain extension-owned.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `docs/schemas/export-manifest.schema.json`.
|
||||
- Wrote `exports/export-manifest.json` for each run.
|
||||
- Included source package refs, source lock refs, summaries, policy summaries,
|
||||
mapping summaries, report fragments, counts, and certification boundary.
|
||||
- Documented the generic export boundary; authority-specific formats remain
|
||||
extension-owned.
|
||||
|
||||
## D7.3 - Trend And Gate Report Improvements
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0007-T003
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "33c3089a-9d5e-4605-89c4-a1e070bc12ad"
|
||||
```
|
||||
@@ -89,11 +106,19 @@ Acceptance:
|
||||
- Keep machine-readable gate summaries stable for automation.
|
||||
- Add CLI report helpers or Markdown summaries where useful.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added trend status-change details, unresolved review deltas, mapping target
|
||||
deltas, and compact summary text.
|
||||
- Kept existing gate summary shape stable while improving trend inputs for
|
||||
human review.
|
||||
- Added assertions for the richer trend output.
|
||||
|
||||
## D7.4 - Golden Fixtures And Documentation
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0007-T004
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "66669f68-6728-4484-9ec9-267ffe025027"
|
||||
```
|
||||
@@ -106,6 +131,15 @@ Acceptance:
|
||||
- Ensure report text remains clear about preparation evidence versus formal
|
||||
certification.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added golden fixture expectations for report fragment text and export manifest
|
||||
top-level shape.
|
||||
- Updated extension SDK, assessment operations, architecture, service, container,
|
||||
and README docs.
|
||||
- Kept report and export boundary language tied to preparation evidence, not
|
||||
formal certification.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
- Extensions can contribute report fragments through a documented contract.
|
||||
|
||||
Reference in New Issue
Block a user