Compare commits

...

19 Commits

Author SHA1 Message Date
ea092120ea chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-22:
  - update .custodian-brief.md for guide-board
2026-06-22 23:22:01 +02:00
2b2f0c959a Normalize agent instructions and workplan frontmatter (STATE-WP-0067)
- Align agent files with on-disk workplan prefixes (infer from workplan ids)
- Set workplan domain to registered domain_slug; add topic_slug where applicable
- Repair frontmatter delimiter formatting; migrate legacy task status literals
- Regenerate AGENTS.md, CLAUDE.md, and .claude/rules from State Hub templates
2026-06-22 23:16:25 +02:00
b1dc0cbe2e Add .repo-classification.yaml (CUST-WP-0050 T11 agent first-pass) 2026-06-22 17:47:36 +02:00
c2810e00e4 Add credential routing instructions for all agent runtimes
Propagate shared credential-routing section (Codex, Claude, Grok, llm-connect)
from state-hub template via scripts/propagate_credential_routing.py.
2026-06-18 22:48:38 +02:00
ac0ee92cc1 Add capability registry scaffold (REUSE-WP-0014-T04 B02)
Empty helix_forge registry layout for federation publishing.
2026-06-16 01:52:38 +02:00
d2cbe61962 Refresh agent instruction files 2026-05-18 16:55:42 +02:00
65c3cba8c7 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-05-16:
  - update .custodian-brief.md for guide-board
2026-05-16 03:15:37 +02:00
6c467dd1f4 Add report fragments and export manifest 2026-05-16 03:11:56 +02:00
2a1a53c140 Expose retained runs through service API 2026-05-16 03:04:17 +02:00
2412f30975 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-05-16:
  - update .custodian-brief.md for guide-board
2026-05-16 02:58:26 +02:00
b1dff0440d Add challenge and exclusion review handling 2026-05-16 02:58:18 +02:00
c8ac42154c Implement source lock and submission package baseline 2026-05-16 02:51:00 +02:00
d73a73b455 Refinement with workplans 2026-05-15 23:04:11 +02:00
4b3f65572f chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-05-15:
  - update .custodian-brief.md for guide-board
2026-05-15 16:03:23 +02:00
6758b3992c Complete extension SDK maturity 2026-05-15 15:34:55 +02:00
67f2fc5346 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-05-15:
  - update .custodian-brief.md for guide-board
2026-05-15 15:30:59 +02:00
fdeea87d4a chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-05-15:
  - update .custodian-brief.md for guide-board
2026-05-15 15:14:20 +02:00
ab7914890e Add extension profile schema validation 2026-05-15 15:13:03 +02:00
955643554f Add external extension acceptance path 2026-05-15 14:59:16 +02:00
70 changed files with 5015 additions and 70 deletions

20
.claude/rules/agents.md Normal file
View File

@@ -0,0 +1,20 @@
## Kaizen Agents
Specialized agent personas available on demand via the state-hub MCP.
**Discover:** `list_kaizen_agents()` — returns all agents with name, description, category
**Load:** `get_kaizen_agent("tdd-workflow")` — returns full instructions; read and follow them
Common agents:
| Agent | Category | When to use |
|-------|----------|-------------|
| `tdd-workflow` | testing | Step-by-step TDD8 workflow for any feature |
| `code-refactoring` | quality | Code quality analysis and safe refactoring |
| `test-maintenance` | testing | Diagnose and fix failing tests |
| `requirements-engineering` | process | Prevent interface/mock mismatches upfront |
| `keepaTodofile` | process | Maintain TODO.md during work |
| `project-management` | process | Track status, determine next steps |
| `datamodel-optimization` | quality | Optimize dataclasses and data structures |
All 17 agents: call `list_kaizen_agents()` for the full list.

View File

@@ -0,0 +1,8 @@
## Architecture
<!-- TODO: Describe the key design decisions and component structure.
Key modules, data flows, external integrations, state machines, etc. -->
## Quick Reference
`~/state-hub/mcp_server/TOOLS.md` — MCP tool reference

View File

@@ -0,0 +1,50 @@
# Credential and access routing
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
for inference. Run this check **before** requesting secrets, API keys, SSH access,
login tokens, or database passwords — in any repo, not only `ops-warden`.
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
other credential need belongs to another subsystem. **Do not** message
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
### Lookup (do this first)
```bash
warden route find "<describe your need>" --json
warden route show <catalog-id> --json
```
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
| Agent runtime | How to orient |
| --- | --- |
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=guide-board` is for coordination, not secret vending |
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
### Quick routing table
| I need… | Owner | ops-warden executes? |
| --- | --- | --- |
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes**`warden sign` |
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
| Authorization decision | flex-auth | No — route only |
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
### Anti-patterns (do not do these)
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
- Pasting secrets into Git, State Hub, workplans, logs, or chat
### Other capabilities (reuse-surface)
Non-credential capabilities are usually discovered through **reuse-surface** federation
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
every repo's agent instructions because it is high-frequency, high-risk, and easy to
get wrong.
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`

View File

@@ -0,0 +1,38 @@
## First Session Protocol
Triggered when `get_domain_summary("communication")` shows **no workstreams**.
The project is registered but work has not yet been structured.
**Step 1 — Read, don't write**
- `~/the-custodian/canon/projects/communication/project_charter_v0.1.md` — purpose, scope
- `~/the-custodian/canon/projects/communication/roadmap_v0.1.md` — planned phases
- Scan repo root: README, directory structure, existing code or docs
**Step 2 — Survey in-progress work**
Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete.
**Step 3 — Propose workstreams to Bernd**
Propose 13 workstreams — each a coherent strand, weeks to months, anchored to a
roadmap phase. **Wait for approval before creating.**
**Step 4 — Create workplan file first, then DB record (ADR-001)**
```
workplans/GUIDE-BOARD-WP-NNNN-<slug>.md ← write this first
```
Then register in the hub:
```
create_workstream(topic_id="5571d954-0d30-4950-980d-7bcaaad8e3e2", title="...", owner="...", description="...")
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
```
**Step 5 — Record the setup**
```
add_progress_event(
summary="First session: structured communication into N workstreams, M tasks",
event_type="milestone",
topic_id="5571d954-0d30-4950-980d-7bcaaad8e3e2",
detail={"workstreams": [...], "tasks_created": M}
)
```
<!-- Delete or archive this file once past first session -->

View File

@@ -0,0 +1,8 @@
## Repo boundary
This repo owns **guide-board** only. It does not own:
<!-- TODO: List what belongs in adjacent repos, e.g.:
- SSH key management → railiance-infra/
- State hub code → state-hub/
-->

View File

@@ -0,0 +1,5 @@
**Purpose:** Certification and compliance preparation framework core with external extension support.
**Domain:** communication
**Repo slug:** guide-board
**Topic ID:** 5571d954-0d30-4950-980d-7bcaaad8e3e2

View File

@@ -0,0 +1,85 @@
## Session Protocol
Dev Hub (State Hub API): http://127.0.0.1:8000
MCP server name in `~/.claude.json`: `dev-hub`
**Step 1 — Orient**
Read the offline-safe brief first — it works without a live hub connection:
```bash
cat .custodian-brief.md
```
Then call the MCP tool for richer cross-domain context when MCP tools are exposed:
```
get_domain_summary("communication")
```
If MCP tools are unavailable in the current agent session, use the REST API:
```bash
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
```
If the hub is offline: `cd ~/state-hub && make api`
**Step 2 — Check inbox**
With MCP tools:
```
get_messages(to_agent="guide-board", unread_only=True)
```
Mark read with `mark_message_read(message_id)`. Reply or act on coordination
requests before proceeding.
Without MCP tools:
```bash
curl -s "http://127.0.0.1:8000/messages/?to_agent=guide-board&unread_only=true" \
| python3 -m json.tool
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
-H "Content-Type: application/json" -d '{}'
```
**Step 3 — Scan workplans**
```bash
ls workplans/
```
For each file with `status: ready`, `active`, or `blocked`, note pending
`wait`/`todo`/`progress` tasks.
**Step 4 — Present brief**
1. **Active workstreams** for `communication` — title, task counts, blocking decisions
2. **Pending tasks** from `workplans/` + any `[repo:guide-board]` hub tasks
3. **Goal guidance** — if `goal_guidance` in summary:
- `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"*
- `alignment_warnings`: flag if active work is not aligned with current goal
4. **Suggested next action** — highest-priority open item
5. **SBOM status** — flag if `last_sbom_at` is unset for this repo
If no workstreams: follow First Session Protocol (`first-session.md`).
**During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()`
> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`)
> are First Session Protocol only. Work structure belongs in repo files (ADR-001).
**Session close:**
With MCP tools:
```
add_progress_event(summary="...", topic_id="5571d954-0d30-4950-980d-7bcaaad8e3e2", workstream_id="<uuid>")
```
Without MCP tools:
```bash
curl -s -X POST http://127.0.0.1:8000/progress/ \
-H "Content-Type: application/json" \
-d '{"topic_id":"5571d954-0d30-4950-980d-7bcaaad8e3e2","workstream_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
```
If workplan files were modified, ensure the local copy is up to date first:
```bash
git -C <repo_path> pull --ff-only
cd ~/state-hub && make fix-consistency REPO=guide-board
```
For repos where implementation runs on a remote machine (e.g. CoulombCore),
use the combined target which pulls before fixing:
```bash
cd ~/state-hub && make fix-consistency-remote REPO=guide-board
```
**C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback
will sync the file to match DB. **C-16** (repo behind remote) blocks all writes
until you pull — intentional to prevent clobbering remote progress.

View File

@@ -0,0 +1,19 @@
## Stack
<!-- TODO: Fill in language, frameworks, and key dependencies -->
- **Language:**
- **Key deps:**
## Dev Commands
```bash
# TODO: Fill in the standard commands for this repo
# Install dependencies
# Run tests
# Lint / type check
# Build / package (if applicable)
```

View File

@@ -0,0 +1,40 @@
## Workplan Convention (ADR-001)
File location: `workplans/GUIDE-BOARD-WP-NNNN-<slug>.md`
ID prefix: `GUIDE-BOARD-WP-`
Work items originate as files in this repo **before** being registered in the hub.
Canonical workplan/workstream frontmatter statuses are:
`proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`.
Use `proposed` for a newly drafted plan, `ready` after review against current
repo state, and `finished` when implementation is complete. `stalled` and
`needs_review` are derived health labels, not stored statuses.
Closed workplans may be moved to `workplans/archived/` with a completion-date
prefix: `YYMMDD-GUIDE-BOARD-WP-NNNN-<slug>.md`. The frontmatter id remains
unchanged; the prefix is only for quick visual reference.
Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**:
`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids
`ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed
directly. Promote anything requiring analysis, design, approval, dependencies, or
multiple planned phases into a normal workplan.
Ecosystem todos from other agents arrive as `[repo:guide-board]` hub tasks —
visible at session start. Pick one up by creating the workplan file, then registering
the workstream.
Task blocks use this shape:
```task
id: GUIDE-BOARD-WP-NNNN-T01
status: wait | todo | progress | done | cancel
priority: high | medium | low
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
```
Status progression is `todo``progress``done`; use `wait` for waiting or
blocked work and `cancel` for stopped work.
<!-- Ralph Loop rules and HEUREKA sequence: ~/.claude/CLAUDE.md — do not duplicate here -->

View File

@@ -1,22 +1,18 @@
<!-- custodian-brief: generated by fix-consistency — do not edit manually -->
# Custodian Brief — guide-board
**Domain:** markitect
**Last synced:** 2026-05-15 12:43 UTC
**Domain:** communication
**Last synced:** 2026-06-22 21:22 UTC
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
## Active Workstreams
### Assessment Operations Baseline
Progress: 5/6 done | workstream_id: `fc5b1573-91b2-4a19-b6a9-dd4d17057d9b`
**Open tasks:**
- · D2.6 - External Extension Acceptance Path `65fbf1df`
*(none — repo may need first-session setup)*
---
## MCP Orientation (when available)
If the state-hub MCP server is reachable, call:
`get_domain_summary("markitect")`
`get_domain_summary("communication")`
This provides richer cross-domain context.
If the MCP call fails, use this file as your orientation source.

17
.repo-classification.yaml Normal file
View File

@@ -0,0 +1,17 @@
repo_classification:
standard: Repo Classification Standard
version: '1.0'
classified_at: '2026-06-22'
classified_by: agent
category: project
domain: communication
secondary_domains: []
capability_tags:
- knowledge
- documentation
business_stake:
- technology
- product
business_mechanics:
- coordination
- operation

219
AGENTS.md Normal file
View File

@@ -0,0 +1,219 @@
# guide-board — Agent Instructions
## Repo Identity
**Purpose:** Certification and compliance preparation framework core with external extension support.
**Domain:** communication
**Repo slug:** guide-board
**Topic ID:** `5571d954-0d30-4950-980d-7bcaaad8e3e2`
**Workplan prefix:** `GUIDE-BOARD-WP-`
---
## State Hub Integration
The Custodian State Hub tracks work across all domains. Interact via HTTP REST —
there is no MCP server for Codex agents.
| Context | URL |
|---------|-----|
| Local workstation | `http://127.0.0.1:8000` |
| Remote via tunnel | `http://127.0.0.1:18000` |
### Orient at session start
```bash
# Offline brief — works without hub connection
cat .custodian-brief.md
# Active workstreams for this domain
curl -s "http://127.0.0.1:8000/workstreams/?topic_id=5571d954-0d30-4950-980d-7bcaaad8e3e2&status=active" \
| python3 -m json.tool
# Check inbox
curl -s "http://127.0.0.1:8000/messages/?to_agent=guide-board&unread_only=true" \
| python3 -m json.tool
```
Mark a message read:
```bash
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
-H "Content-Type: application/json" -d '{}'
```
### Log progress (required at session close)
```bash
curl -s -X POST http://127.0.0.1:8000/progress/ \
-H "Content-Type: application/json" \
-d '{
"summary": "what was done",
"event_type": "note",
"author": "codex",
"workstream_id": "<uuid>",
"task_id": "<uuid>"
}'
```
Omit `workstream_id` / `task_id` when not applicable.
### Update task status
```bash
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
-H "Content-Type: application/json" \
-d '{"status": "progress"}'
# values: wait | todo | progress | done | cancel
```
### Flag a task for human review
```bash
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
-H "Content-Type: application/json" \
-d '{"needs_human": true, "intervention_note": "reason"}'
```
---
## Session Protocol
**Start:**
1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe)
2. Check inbox: `GET /messages/?to_agent=guide-board&unread_only=true`; mark read
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
**During work:**
- Update task statuses in workplan files as tasks progress
- Record significant decisions via `POST /decisions/`
**Close:**
1. Update workplan file task statuses to reflect progress
2. Log: `POST /progress/` with a summary of what changed
3. Note for the custodian operator: after workplan file changes, run from
`~/state-hub`:
```bash
make fix-consistency REPO=guide-board
```
This syncs task status from files into the hub DB.
---
## Credential and access routing
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
for inference. Run this check **before** requesting secrets, API keys, SSH access,
login tokens, or database passwords — in any repo, not only `ops-warden`.
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
other credential need belongs to another subsystem. **Do not** message
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
### Lookup (do this first)
```bash
warden route find "<describe your need>" --json
warden route show <catalog-id> --json
```
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
| Agent runtime | How to orient |
| --- | --- |
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=guide-board` is for coordination, not secret vending |
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
### Quick routing table
| I need… | Owner | ops-warden executes? |
| --- | --- | --- |
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
| Authorization decision | flex-auth | No — route only |
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
### Anti-patterns (do not do these)
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
- Pasting secrets into Git, State Hub, workplans, logs, or chat
### Other capabilities (reuse-surface)
Non-credential capabilities are usually discovered through **reuse-surface** federation
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
every repo's agent instructions because it is high-frequency, high-risk, and easy to
get wrong.
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
<!-- REPO-AGENTS-EXTENSIONS -->
<!-- Append repo-specific agent instructions below this marker.
The state-hub template sync preserves content after this line. -->
---
## Workplan Convention (ADR-001)
Work items originate as files in this repo — not in the hub. The hub is a
read/cache/index layer that rebuilds from files.
**File location:** `workplans/GUIDE-WP-NNNN-<slug>.md`
**Archived location:** finished workplans may move to
`workplans/archived/YYMMDD-GUIDE-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
the completion/archive date; the frontmatter `id` does not change.
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
`workplans/ADHOC-YYYY-MM-DD.md` with task ids `ADHOC-YYYY-MM-DD-T01`, etc. Use
this only for low-risk work completed directly; create a normal workplan for
anything needing analysis, design, approval, dependencies, or multiple phases.
**Frontmatter:**
```yaml
---
id: GUIDE-WP-NNNN
type: workplan
title: "..."
domain: communication
repo: guide-board
status: proposed | ready | active | blocked | backlog | finished | archived
owner: codex
topic_slug: ...
created: "YYYY-MM-DD"
updated: "YYYY-MM-DD"
state_hub_workstream_id: "<uuid>" # written by fix-consistency — do not edit
---
```
Use `proposed` for a new draft, `ready` after review against current repo
state, and `finished` after implementation. `stalled` and `needs_review` are
derived health labels, not frontmatter statuses.
**Task block format** (one per `##` section):
```
## Task Title
` ` `task
id: GUIDE-WP-NNNN-T01
status: wait | todo | progress | done | cancel
priority: high | medium | low
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
` ` `
Task description text.
```
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
To create a new workplan:
1. Write the file following the format above
2. Notify the custodian operator to run `make fix-consistency REPO=guide-board`
(or send a message to the hub agent via `POST /messages/`)

12
CLAUDE.md Normal file
View File

@@ -0,0 +1,12 @@
# guide-board — Claude Code Instructions
@SCOPE.md
@.claude/rules/repo-identity.md
@.claude/rules/session-protocol.md
@.claude/rules/first-session.md
@.claude/rules/workplan-convention.md
@.claude/rules/stack-and-commands.md
@.claude/rules/architecture.md
@.claude/rules/repo-boundary.md
@.claude/rules/credential-routing.md
@.claude/rules/agents.md

View File

@@ -42,9 +42,15 @@ The same CLI contracts are packaged by the container baseline. See
[docs/CONTAINER.md](docs/CONTAINER.md). The dependency-light local API wraps
those contracts for service and container operation; see
[docs/LOCAL-SERVICE-API.md](docs/LOCAL-SERVICE-API.md).
New runs write a richer `sources.lock.json`,
`reports/submission-package.json`, extension report fragments, and
`exports/export-manifest.json` alongside the assessment package so reviewers can
inspect source, metadata, artifact, export, and boundary references.
The `sample-noop` extension exercises the guide-board contracts without invoking
an external harness. `open-cmis-tck` is the first real seed extension.
an external harness. `sdk-fixture` demonstrates the extension SDK contracts for
schemas, normalizers, mappings, and fixture profiles. `open-cmis-tck` is the
first real seed extension.
See:
@@ -54,8 +60,15 @@ See:
- [docs/CANDIDATE-HANDOFF.md](docs/CANDIDATE-HANDOFF.md)
- [docs/COMPLIANCE-EVIDENCE-PACKS.md](docs/COMPLIANCE-EVIDENCE-PACKS.md)
- [docs/CONTAINER.md](docs/CONTAINER.md)
- [docs/EXTERNAL-EXTENSION-ACCEPTANCE.md](docs/EXTERNAL-EXTENSION-ACCEPTANCE.md)
- [docs/EXTENSION-SDK.md](docs/EXTENSION-SDK.md)
- [docs/LOCAL-SERVICE-API.md](docs/LOCAL-SERVICE-API.md)
- [docs/SERVICE-JOB-DURABILITY.md](docs/SERVICE-JOB-DURABILITY.md)
- [extensions/CANDIDATES.md](extensions/CANDIDATES.md)
- [workplans/GUIDE-BOARD-WP-0001-bootstrapping.md](workplans/GUIDE-BOARD-WP-0001-bootstrapping.md)
- [workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md](workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md)
- [workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md](workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md)
- [workplans/GUIDE-BOARD-WP-0004-source-lock-and-submission-package-baseline.md](workplans/GUIDE-BOARD-WP-0004-source-lock-and-submission-package-baseline.md)
- [workplans/GUIDE-BOARD-WP-0005-challenge-and-exclusion-handling.md](workplans/GUIDE-BOARD-WP-0005-challenge-and-exclusion-handling.md)
- [workplans/GUIDE-BOARD-WP-0006-service-artifact-access-and-durable-run-index.md](workplans/GUIDE-BOARD-WP-0006-service-artifact-access-and-durable-run-index.md)
- [workplans/GUIDE-BOARD-WP-0007-report-and-export-maturity.md](workplans/GUIDE-BOARD-WP-0007-report-and-export-maturity.md)

137
SCOPE.md Normal file
View File

@@ -0,0 +1,137 @@
# SCOPE
> This file helps you quickly understand what this repository is about,
> when it is relevant, and when it is not.
> It is intentionally lightweight and may be incomplete.
---
## One-liner
<!-- Describe the purpose of this repository in one precise sentence. -->
<!-- Example: "Provides a lightweight event router for Kubernetes-native systems." -->
---
## Core Idea
<!-- What is the main capability or idea behind this repository? -->
<!-- What problem does it try to solve? -->
---
## In Scope
<!-- What this repository is responsible for. -->
<!-- Be explicit and concrete. -->
-
-
-
---
## Out of Scope
<!-- What this repository deliberately does NOT do. -->
<!-- This is often more important than "In Scope". -->
-
-
-
---
## Relevant When
<!-- When should someone consider using or exploring this repository? -->
-
-
-
---
## Not Relevant When
<!-- When should someone ignore this repository? -->
-
-
-
---
## Current State
<!-- Rough indication of maturity. No strict format required. -->
- Status: <!-- e.g. concept / experimental / active / stable / deprecated -->
- Implementation: <!-- e.g. idea / partial / substantial / complete -->
- Stability: <!-- e.g. unstable / evolving / stable -->
- Usage: <!-- e.g. none / personal / internal / production -->
<!-- Add any notes that help set expectations. -->
---
## How It Fits
<!-- Where does this repository sit in the bigger picture? -->
- Upstream dependencies:
- Downstream consumers:
- Often used with:
---
## Terminology
<!-- Terms that are important to understand this repo. -->
<!-- Especially useful if naming differs from other repos. -->
- Preferred terms:
- Also known as:
- Potentially confusing terms:
---
## Related / Overlapping Repositories
<!-- List repositories that have similar or adjacent responsibilities. -->
<!-- Helps detect duplication and navigate the ecosystem. -->
- <repo-name> — <!-- how it relates -->
---
## Getting Oriented
<!-- If someone decides to look deeper, where should they start? -->
- Start with:
- Key files / directories:
- Entry points:
---
## Provided Capabilities
<!-- What can this repo's domain provide to other domains on request? -->
<!-- Each capability block is parsed by the state-hub capability catalog ingest. -->
<!-- Remove the examples and add your own, or leave empty if none. -->
<!--
```capability
type: infrastructure
title: Example capability title
description: What this capability provides, in one or two sentences.
keywords: [keyword1, keyword2, keyword3]
```
-->
---
## Notes
<!-- Anything else worth knowing. Keep it short. -->

View File

@@ -355,7 +355,9 @@ Stores run artifacts by reference and checksum:
The first implementation builds the assessment package artifact manifest from
runner-emitted artifact refs and computes checksums for files inside the run
directory.
directory. New runs also write a source lock and a submission package manifest
that fingerprint reviewable run files and summarize runner or normalizer
metadata reported by extensions.
### Normalizer
@@ -559,6 +561,18 @@ building complex runtime code.
- `artifact_policy`
- `runtime_policy`
### `SourceLock`
- `framework_refs`
- `extension_refs`
- `frameworks`
- `extensions`
- `mapping_sets`
- `profiles`
- `policy_refs`
- `authorities`
- `metadata_hooks`
### `RawArtifact`
- `id`
@@ -626,6 +640,31 @@ building complex runtime code.
- `certification_boundary`
- `created_at`
### `SubmissionPackage`
- `run_id`
- `package_identity`
- `source_lock_ref`
- `source_lock`
- `reports`
- `normalized_outputs`
- `profile_snapshots`
- `artifact_manifest`
- `reported_metadata`
- `certification_boundary`
### `ExportManifest`
- `export_type`
- `source_package_ref`
- `source_lock_ref`
- `summary`
- `policy_summary`
- `mapping_summary`
- `report_fragments`
- `counts`
- `certification_boundary`
## Result Vocabulary
The evidence model should allow these statuses:
@@ -713,8 +752,11 @@ runs/<run-id>/
mappings.json
reports/
report.md
fragments.json
assessment-package.json
submission-package.json
exports/
export-manifest.json
```
## Container And Service Model
@@ -775,6 +817,9 @@ Use separate concepts:
- defect: unexpected product or process failure.
The report must make these visible separately.
The current policy layer loads challenge and exclusion refs from assessment
profiles, annotates findings and evidence, and keeps `unexpected_findings`
visible for gate semantics unless a finding is separately expected or waived.
### Source Locking
@@ -787,7 +832,12 @@ Each run should lock:
- test suite IDs,
- mapping version,
- target profile snapshot,
- waiver snapshot.
- expectation and waiver refs.
The current source lock remains backward-compatible with the original
`framework_refs` and `extension_refs` fields while adding checksummed profiles,
mapping-set refs, optional policy refs, authority descriptors, and metadata
hooks for runners and normalizers.
## Implementation Sequence
@@ -800,6 +850,8 @@ Each run should lock:
6. Add container design after the CLI baseline is stable.
7. Add optional service API around the CLI job model.
8. Add OSCAL export and procedural evidence-pack support after the internal
evidence model proves itself with executable extensions.
evidence model proves itself with executable extensions. The first generic
export is `exports/export-manifest.json`; authority-specific interchange
remains extension-owned until the internal model is stable.
The first extension SDK contract is documented in `docs/EXTENSION-SDK.md`.

View File

@@ -27,7 +27,8 @@ Every run needs:
The target profile describes the candidate system or artifact being assessed.
The assessment profile selects frameworks, extensions, check groups, runtime
policy, waivers, expectations, and output policy.
policy, expectations, waivers, challenges, authority exclusions, and output
policy.
## CLI Flow
@@ -62,6 +63,11 @@ PYTHONPATH=src python3 -m guide_board \
The same extension roots can be provided through `GUIDE_BOARD_EXTENSION_PATHS`
when a wrapper script or container entrypoint should keep commands shorter.
For the repeatable external extension acceptance path, including validation,
planning, live execution, and retained result review, see
`docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
For extension-author contracts such as profile schema descriptors and
normalizer plug-ins, see `docs/EXTENSION-SDK.md`.
## CLI Results
@@ -72,6 +78,8 @@ A completed CLI command prints a JSON result with:
- `run_dir`: output directory,
- `assessment_package`: JSON assessment package path,
- `report`: Markdown report path,
- `submission_package`: portable submission package manifest path,
- `export_manifest`: portable generic JSON export manifest path,
- `retention_summary`: compact durable summary path.
The output directory uses this contract:
@@ -79,15 +87,35 @@ The output directory uses this contract:
```text
run.json
plan.json
sources.lock.json
target-profile.snapshot.json
assessment-profile.snapshot.json
retention-summary.json
normalized/evidence.json
normalized/findings.json
normalized/mappings.json
reports/assessment-package.json
reports/report.md
reports/fragments.json
reports/submission-package.json
exports/export-manifest.json
artifacts/
```
`sources.lock.json` records the framework refs, extension versions, mapping
sets, profile snapshots, policy and review refs, authority refs, and extension
metadata hooks used for the run. `reports/submission-package.json` points at
the reviewable package files, includes checksums where files exist, carries the
raw artifact manifest, and repeats the certification boundary. It is a portable
handoff manifest for preparation evidence, not an authority-specific final
submission.
Extension report fragments are recorded in `reports/fragments.json`, embedded in
`reports/assessment-package.json`, and rendered into the Markdown report.
`exports/export-manifest.json` is the first generic portable export surface. It
is derived from the assessment package and carries summary, policy, mapping,
fragment, count, source-lock, and boundary references.
Use the retained run helpers for history:
```sh
@@ -102,6 +130,10 @@ PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs
PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs
```
Trend summaries include status changes, unexpected finding deltas, unresolved
review deltas, mapping target deltas, evidence result deltas, and a compact
human-readable summary string for each target/assessment pair.
## Local Service Flow
Start the service from the guide-board repository:
@@ -145,6 +177,15 @@ Fetch reports after the job status is `succeeded`:
curl -sf http://127.0.0.1:8080/runs/JOB_ID/reports | python3 -m json.tool
```
Inspect retained run history, including runs produced before the current
service process started:
```sh
curl -sf "http://127.0.0.1:8080/retained-runs?runs_dir=runs" | python3 -m json.tool
curl -sf "http://127.0.0.1:8080/retained-runs/latest?runs_dir=runs" | python3 -m json.tool
curl -sf "http://127.0.0.1:8080/retained-runs/RUN_ID/artifact-manifest?runs_dir=runs" | python3 -m json.tool
```
Service job state is currently in memory for the running service process. Run
artifacts are durable in the output directory and can still be inspected after a
service restart. See `docs/SERVICE-JOB-DURABILITY.md` for the restart and
@@ -182,6 +223,23 @@ Individual evidence items use:
- `expected_gap`
- `infrastructure_error`
## Review State
Assessment profiles may reference:
- `expectations_ref`: known target posture, optional scope, or accepted gaps,
- `waivers_ref`: approved, time-bounded exceptions,
- `challenges_ref`: review claims that a finding, check, mapping, or native
result should be challenged,
- `exclusions_ref`: authority or program exclusions that apply to selected
findings.
Challenges and exclusions annotate findings and evidence. They do not silently
turn failures into passing evidence and they do not reduce the
`unexpected_findings` count used by default gates. Retained summaries expose
separate counts for expected findings, waived findings, challenged findings,
authority exclusions, unresolved defects, and unresolved review items.
## Candidate System Checklist
Before starting a run against candidate software, confirm:

View File

@@ -8,8 +8,8 @@ Created: 2026-05-07
Compliance evidence packs cover frameworks where guide-board cannot rely on an
official executable harness. They help prepare and perform assessments by
organizing evidence requests, expected artifacts, reviewer workflow, waivers,
and run reports. They do not replace auditors, accredited certification bodies,
legal counsel, or official standard text.
challenges, authority exclusions, and run reports. They do not replace auditors,
accredited certification bodies, legal counsel, or official standard text.
Examples include GDPR, SOC 2, HIPAA, NF Z 42-013, NF 461, ISO 14641, ISO 15489,
and similar procedural or control-oriented frameworks.
@@ -83,7 +83,7 @@ Each request should include:
Requests should be phrased as collection guidance, not as legal conclusions.
## Waivers And Expected Gaps
## Review Policy Records
Evidence packs use the same expectation and waiver model as executable
extensions.
@@ -103,6 +103,16 @@ Use waivers for:
Every waiver should include owner, reason, approval status, and expiry.
Use challenges for disputed checks, disputed mappings, imported native result
questions, or evidence that needs a reviewer decision before it can be treated
as a defect. Use authority exclusions only when a program, standard, or
authorized reviewer excludes a requirement or check from the assessment scope.
Both records should cite stable requirement refs, check refs, evidence refs, or
authority source refs rather than reproducing restricted standard text.
Challenges and exclusions make review state visible; they do not by themselves
claim compliance or remove default gate-visible unexpected findings.
## Framework Notes
GDPR packs should emphasize processing inventory, lawful basis records, data
@@ -129,6 +139,7 @@ extensions:
- normalized evidence,
- findings,
- review annotations for expectations, waivers, challenges, and exclusions,
- mapping records,
- assessment packages,
- retention summaries,

View File

@@ -62,7 +62,8 @@ The script:
- builds `guide-board-core:smoke`,
- mounts a host output directory at `/runs`,
- runs the bundled sample assessment,
- verifies that the expected run artifacts are present on the host.
- verifies that the expected run artifacts, report fragments, submission
manifest, and generic export manifest are present on the host.
Override the runtime, image name, or output directory when needed:
@@ -144,4 +145,9 @@ podman run --rm -p 8080:8080 \
The service layer adds in-memory job tracking and HTTP transport. Execution
semantics remain the CLI/core semantics documented in
`docs/LOCAL-SERVICE-API.md`.
`docs/LOCAL-SERVICE-API.md`. Mounted run directories remain discoverable through
the retained-run endpoints, for example:
```sh
curl -sf "http://127.0.0.1:8080/retained-runs?runs_dir=/runs" | python3 -m json.tool
```

View File

@@ -71,14 +71,62 @@ The key runtime fields are:
- `extension_type`: one of the supported archetypes from the architecture
blueprint.
- `supported_frameworks`: framework IDs this extension can contribute evidence
for.
for. Descriptor objects with `id`, `version`, `source_url`, and
`authority_ref` may be used when source metadata is available.
- `authorities`: authority IDs or descriptor objects with optional source URL,
version, license, and access notes.
- `metadata`: optional extension-level metadata such as adapter version or
source URL. The core preserves it in source locks and evidence metadata.
- `check_groups`: named groups that assessment profiles can select.
- `preflight_runner`: optional runner ID used before selected check groups.
- `runner_entrypoints`: concrete runner declarations.
- `normalizers`: optional plug-ins that convert native runner output into the
stable runner-result shape before evidence is written.
- `mappings`: mapping set IDs under `mappings/<mapping-id>.json`.
- `report_fragments`: optional Markdown file or Python module descriptors for
extension-owned report content.
- `certification_boundary`: explicit statement of what the extension does not
certify.
`profile_schemas` may use the original string shorthand for core schemas:
```json
["target-profile", "assessment-profile"]
```
Extensions that need stricter domain-specific validation can add schema
descriptors:
```json
[
"target-profile",
"assessment-profile",
{
"id": "cmis-browser-target",
"profile_kind": "target",
"path": "schemas/cmis-browser-target.schema.json",
"subject_type": "cmis-browser-binding-endpoint",
"description": "Requires the target shape expected by the CMIS Browser Binding harness."
}
]
```
Descriptor fields:
- `id`: stable schema descriptor ID used in validation errors.
- `profile_kind`: `target` or `assessment`.
- `path`: JSON schema path relative to the extension root.
- `subject_type`: optional target-profile selector. When present, the schema is
applied only to targets with that `subject_type`.
- `description`: optional authoring note.
The core validates the generic guide-board schema first, then applies matching
extension-owned schemas during `profile validate-*`, `plan`, and `run`.
Extension schema paths must stay inside the extension root. The baseline
validator intentionally supports the small JSON Schema subset used by
guide-board contracts: `type`, `enum`, `required`, `properties`,
`additionalProperties`, `items`, and `minItems`.
## Runner Entry Points
Runner entry points currently support these kinds:
@@ -100,6 +148,11 @@ Example:
"module_path": "src/open_cmis_tck/preflight.py",
"callable": "run",
"command": null,
"metadata": {
"harness_id": "opencmis-tck",
"harness_version": "extension-detected-or-declared",
"source_url": "https://chemistry.apache.org/java/opencmis.html"
},
"description": "Checks whether the CMIS Browser Binding endpoint is reachable."
}
```
@@ -158,6 +211,53 @@ to extension-owned mappings and writes normalized mapping records to:
runs/<run-id>/normalized/mappings.json
```
## Report Fragments
Extensions can contribute report fragments through `report_fragments`.
Static Markdown file:
```json
{
"id": "overview",
"kind": "markdown_file",
"path": "reports/overview.md",
"title": "Overview"
}
```
Dynamic Python fragment:
```json
{
"id": "sdk-fixture-summary",
"kind": "python_module",
"module_path": "reports/sdk_fixture_summary.py",
"callable": "build_fragment",
"path": null,
"title": "SDK Fixture Summary"
}
```
Fragment paths are resolved relative to the extension root and must stay inside
that root. A Python fragment receives `root`, `run_dir`, `run_id`, `plan`,
`evidence`, `findings`, `mappings`, `assessment_package`, `policy_summary`,
`source_lock`, `extension_path`, and `report_fragment`.
It returns:
```python
def build_fragment(context: dict) -> dict:
return {
"markdown": "### Extension Summary\n\n- evidence items: 2",
"structured": {"evidence_count": 2},
}
```
Fragments are written to `reports/fragments.json`, embedded in the assessment
package, rendered in `reports/report.md`, and summarized in
`exports/export-manifest.json`.
## Evidence Request Sets
Procedural and hybrid compliance extensions may include evidence request sets
@@ -199,6 +299,33 @@ Expectation sets mark known posture as expected. Waiver sets mark approved,
time-bounded exceptions. Both are applied after findings are generated, and the
assessment package records policy summary counts.
## Challenges And Authority Exclusions
Assessment profiles may also reference challenge and exclusion sets:
```json
{
"challenges_ref": "profiles/challenges/example.json",
"exclusions_ref": "profiles/exclusions/example.json"
}
```
Challenge sets validate against `docs/schemas/challenge-set.schema.json`.
Exclusion sets validate against `docs/schemas/exclusion-set.schema.json`.
Records can match findings by requirement refs, check refs, evidence refs,
result refs, or classification refs. They also carry owner, review status,
rationale, authority source refs, review dates, optional expiry, native IDs,
and free-form metadata.
Use challenges when an extension author or assessment team believes a finding
needs review because a check is invalid, a native harness result is disputed, or
a mapping is wrong. Use exclusions when an authority or program explicitly
removes a requirement, check, or result from the assessment scope. The core
preserves these distinctions in findings, evidence review annotations,
assessment packages, reports, and retained summaries, but default gate semantics
still count the underlying finding as unexpected unless it is separately
expected or waived.
## Python Runner Contract
A Python runner receives one context object and returns one result object.
@@ -231,11 +358,20 @@ Result fields:
- `observations`: human-readable observations.
- `facts`: structured facts extracted by the runner.
- `artifact_refs`: references to raw artifacts written by the runner.
- `requirement_refs`: optional requirement refs discovered by the runner.
- `metadata`: optional generic metadata such as `harness_version`,
`test_suite_id`, `adapter_version`, `source_url`, or native result IDs.
Artifact refs must be paths relative to the run directory. After runner
execution, the core fingerprints existing artifact refs into the assessment
package `artifact_manifest`.
Runner metadata is merged with manifest entrypoint metadata and preserved under
evidence `facts.source_metadata`. The same metadata is also summarized in the
submission package manifest, which lets reviewers distinguish the extension
version from the harness or native test-suite version without adding
domain-specific fields to the core.
If a Python runner raises an exception, the core converts that failure into
`infrastructure_error` evidence so the assessment package remains complete.
@@ -244,6 +380,94 @@ or `infrastructure_error`, downstream check groups for that extension are not
executed; they receive `blocked` evidence with `blocked_reason:
preflight_failed`.
## Normalizer Plug-ins
Runners can keep returning guide-board-ready result objects directly. When a
runner wraps a native harness or scanner that writes its own result format, the
extension can add a normalizer descriptor:
```json
{
"id": "native-probe-normalizer",
"kind": "python_module",
"module_path": "normalizers/native_probe.py",
"callable": "normalize",
"runner_ref": "native-probe",
"metadata": {
"adapter_version": "0.1.0"
},
"description": "Converts native runner output into guide-board evidence."
}
```
Normalizers are declared in `extension.json` under `normalizers`. The original
string shorthand remains valid for descriptive-only entries, but only descriptor
objects are loaded and invoked by the core.
The first supported normalizer kind is `python_module`. Its module path is
resolved relative to the extension root and must stay inside that root. The
callable receives one context object:
- `root`: guide-board core root path as a string.
- `extension_path`: extension root path as a string.
- `run_dir`: output run directory path as a string.
- `run_id`: current run ID.
- `plan`: full run plan snapshot.
- `step`: the step being normalized.
- `target_profile`: target profile snapshot.
- `assessment_profile`: assessment profile snapshot.
- `normalizer`: manifest normalizer descriptor.
- `runner_result`: the current runner-result object.
A normalizer returns any subset of the runner-result fields:
```python
def normalize(context: dict) -> dict:
return {
"result": "pass",
"observations": ["Native result was normalized."],
"facts": {"native_status": "ok"},
"artifact_refs": ["artifacts/native-result.json"],
"requirement_refs": ["framework.requirement"],
}
```
The core merges the normalizer output over the runner result:
- `result` replaces the previous result.
- `observations` are appended.
- `facts` are merged.
- `artifact_refs` and `requirement_refs` are deduplicated.
- `metadata` is merged.
- `normalizer_refs` is recorded in evidence facts when any normalizer runs.
If a normalizer raises an exception, the step becomes
`infrastructure_error` evidence and the run still produces its normal artifact
set.
The bundled `extensions/sdk-fixture` extension is the copyable reference path
for profile schemas, a native-output runner, a normalizer, mappings, and fixture
profiles.
## Source Lock And Submission Package
Every new run writes `sources.lock.json`, `reports/submission-package.json`,
and the generic portable export manifest at `exports/export-manifest.json`.
Extension authors should treat source metadata as part of the evidence contract:
- declare extension, authority, framework, runner, and normalizer metadata in
`extension.json` when it is static;
- return runner or normalizer `metadata` when versions, native result IDs, or
test-suite IDs are detected at runtime;
- keep mapping sets under `mappings/` so the core can checksum them in the
source lock;
- keep restricted or licensed assets referenced by metadata or artifacts rather
than vendored into the core.
The submission package manifest is generic guide-board output. Authority-specific
final submissions, trademark assertions, or certification conclusions remain
extension-owned or reviewer-owned.
## Result Statuses
Initial statuses:
@@ -264,11 +488,14 @@ Initial statuses:
## Current Extension Examples
- `sample-noop`: no runner, used to validate the core contracts.
- `sdk-fixture`: compact SDK fixture covering profile schemas, runner output,
normalizer invocation, mapping, and fixture profiles.
- `open-cmis-tck`: provides a Python CMIS Browser Binding preflight runner and
declares the future external OpenCMIS TCK runner.
## Next SDK Steps
- Add normalizer plug-in contracts.
- Add extension-owned schema validation for domain-specific target profile
fields.
- Broaden normalizer examples as real external extensions adopt native harness
result formats.
- Add more extension-owned schema validation examples for assessment-specific
domain constraints.

View File

@@ -0,0 +1,147 @@
# External Extension Acceptance
Status: draft
Created: 2026-05-15
## Purpose
This document defines the repeatable acceptance path for a guide-board
extension that lives outside the core repository. It proves that guide-board can
discover the extension, validate candidate profiles, build a traceable run plan,
execute the selected checks, and review retained results without moving
domain-specific harness logic into the core.
`open-cmis-tck` is the first concrete example. CMIS-specific profiles, runners,
runtime dependencies, and harness behavior remain owned by that extension
repository.
For a dependency-light SDK reference extension that can be copied into a
temporary external repository, see `extensions/sdk-fixture`.
## Acceptance Stages
Run these stages from the guide-board repository.
1. Confirm the extension repository shape:
```sh
test -f ../open-cmis-tck/extension.json
```
2. Validate extension discovery and manifest contracts:
```sh
PYTHONPATH=src python3 -m guide_board \
--extension-dir ../open-cmis-tck \
extensions validate
```
3. Validate the target and assessment profiles:
```sh
PYTHONPATH=src python3 -m guide_board \
--extension-dir ../open-cmis-tck \
profile validate-target \
../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json
PYTHONPATH=src python3 -m guide_board \
--extension-dir ../open-cmis-tck \
profile validate-assessment \
../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json
```
4. Generate and retain a run plan:
```sh
mkdir -p /tmp/guide-board-external-extension-acceptance
PYTHONPATH=src python3 -m guide_board \
--extension-dir ../open-cmis-tck \
plan \
--target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
--assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
--output /tmp/guide-board-external-extension-acceptance/plan.json
```
5. Execute a live run when the candidate endpoint and extension runtime
dependencies are available:
```sh
PYTHONPATH=src python3 -m guide_board \
--extension-dir ../open-cmis-tck \
run \
--target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
--assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
--output-dir /tmp/guide-board-external-extension-acceptance/open-cmis-tck-baseline
```
6. Review retained results:
```sh
PYTHONPATH=src python3 -m guide_board runs report \
--runs-dir /tmp/guide-board-external-extension-acceptance \
--target kontextual-cmis-compat \
--assessment cmis-browser-baseline
```
The run directory should contain:
```text
run.json
plan.json
retention-summary.json
normalized/evidence.json
normalized/findings.json
normalized/mappings.json
reports/assessment-package.json
reports/report.md
artifacts/
```
## Scripted Acceptance
The scripted path keeps the same commands in one place:
```sh
scripts/external_extension_acceptance.sh plan
```
The default `plan` mode validates the extension and profiles, then writes a run
plan under `/tmp/guide-board-external-extension-acceptance`. It does not contact
the candidate system.
Use `run` mode for the live acceptance after the candidate endpoint and harness
dependencies are ready:
```sh
scripts/external_extension_acceptance.sh run
```
Override the extension, profiles, or output location when accepting another
extension:
```sh
GUIDE_BOARD_ACCEPT_EXTENSION_DIR=../example-extension \
GUIDE_BOARD_ACCEPT_TARGET=../example-extension/profiles/targets/example.json \
GUIDE_BOARD_ACCEPT_ASSESSMENT=../example-extension/profiles/assessments/example.json \
GUIDE_BOARD_ACCEPT_OUTPUT_DIR=/tmp/guide-board-example-extension \
scripts/external_extension_acceptance.sh plan
```
## Ownership Boundary
Guide-board owns:
- extension discovery,
- extension manifest validation,
- target and assessment profile validation,
- run plan construction,
- run artifact layout,
- retained run summary and report lookup.
The external extension repository owns:
- domain-specific target and assessment profiles,
- harness adapters and runtime dependencies,
- credentials or restricted asset instructions,
- check group behavior,
- interpretation of domain-specific evidence.

View File

@@ -98,7 +98,41 @@ errors.
### `GET /runs/{job_id}/reports`
Returns the Markdown report content, assessment package JSON, retention summary,
and their filesystem paths after a job has succeeded.
submission package JSON, export manifest JSON when present, and their filesystem
paths after a job has succeeded.
### `GET /retained-runs`
Lists durable retained run summaries by scanning a runs directory. Without a
query parameter, the service scans `<root>/runs`.
```text
GET /retained-runs?runs_dir=/runs
```
### `GET /retained-runs/latest`
Selects the latest retained run, optionally filtered by target and assessment
profile refs.
```text
GET /retained-runs/latest?runs_dir=/runs&target=sample-repository&assessment=sample-noop-assessment
```
### `GET /retained-runs/{run_id}/reports`
Returns the retained summary plus safe report paths for a durable run. This
works after a service restart because it reads `retention-summary.json` from
disk instead of in-memory job records.
### `GET /retained-runs/{run_id}/artifact-manifest`
Returns the assessment package `artifact_manifest` for a retained run. If the
run predates assessment packages, the response is compatible and returns an
empty manifest with `compatibility: "assessment-package-missing"`.
Retained-run endpoints validate report and artifact paths before returning
them. A path that escapes the selected run directory is rejected.
## Container Mode
@@ -112,5 +146,6 @@ podman run --rm -p 8080:8080 \
```
The service keeps job state in memory. Durable run evidence remains in the
mounted output directory. See `docs/SERVICE-JOB-DURABILITY.md` for the explicit
restart and recovery contract.
mounted output directory and can be discovered through `GET /retained-runs`
after restart. See `docs/SERVICE-JOB-DURABILITY.md` for the explicit recovery
contract.

View File

@@ -13,16 +13,19 @@ Durable state lives in run directories:
- `run.json`
- `plan.json`
- `sources.lock.json`
- `retention-summary.json`
- `normalized/evidence.json`
- `normalized/findings.json`
- `normalized/mappings.json`
- `reports/assessment-package.json`
- `reports/report.md`
- `reports/submission-package.json`
- `artifacts/`
The durable recovery index is the set of `retention-summary.json` files under a
runs directory.
runs directory. No separate durable service index is required for the baseline;
the service reconstructs retained-run views by scanning those summaries.
## Why In-Memory Jobs Stay The Baseline
@@ -47,9 +50,14 @@ After a service restart:
- old `job_id` values are invalid,
- `GET /runs/{job_id}` cannot recover pre-restart job metadata,
- `GET /runs/{job_id}/reports` only works for jobs known to the current process,
- run artifacts from earlier service processes remain available on disk.
- run artifacts from earlier service processes remain available on disk,
- `GET /retained-runs`, `GET /retained-runs/latest`,
`GET /retained-runs/{run_id}/reports`, and
`GET /retained-runs/{run_id}/artifact-manifest` can expose completed retained
runs after restart.
Operators should recover previous results with the CLI run-history commands:
Operators can recover previous results with either the CLI run-history commands
or the retained-run service endpoints:
```sh
PYTHONPATH=src python3 -m guide_board runs list --runs-dir runs
@@ -57,6 +65,12 @@ PYTHONPATH=src python3 -m guide_board runs latest --runs-dir runs
PYTHONPATH=src python3 -m guide_board runs report --runs-dir runs --run-id RUN_ID
```
```sh
curl -sf "http://127.0.0.1:8080/retained-runs?runs_dir=runs" | python3 -m json.tool
curl -sf "http://127.0.0.1:8080/retained-runs/RUN_ID/reports?runs_dir=runs" | python3 -m json.tool
curl -sf "http://127.0.0.1:8080/retained-runs/RUN_ID/artifact-manifest?runs_dir=runs" | python3 -m json.tool
```
## Recovery Flow
Use this flow when the service process restarted or a browser/UI lost its job
@@ -64,8 +78,9 @@ state:
1. Identify the output directory passed to `POST /runs`.
2. Confirm whether `retention-summary.json` exists.
3. If it exists, use `guide-board runs report --runs-dir <parent>` to retrieve
report paths.
3. If it exists, use `guide-board runs report --runs-dir <parent>` or
`GET /retained-runs/{run_id}/reports?runs_dir=<parent>` to retrieve report
paths.
4. If only partial files exist, inspect `run.json`, `plan.json`, and artifacts
before rerunning.
5. Rerun into a fresh output directory when the prior status is unclear.
@@ -73,8 +88,8 @@ state:
## Future Durable Index Option
A future durable service index may be added if UI or automation workflows need
cross-restart job lookup. If added, it should remain reconstructable from run
directories and should not become the authority for assessment results.
cross-restart transport job lookup. If added, it should remain reconstructable
from run directories and should not become the authority for assessment results.
The minimum acceptable durable index would contain:

View File

@@ -17,6 +17,9 @@
"evidence_refs",
"artifact_manifest",
"waivers",
"challenges",
"exclusions",
"report_fragments",
"certification_boundary",
"created_at"
],
@@ -34,6 +37,9 @@
"evidence_refs": { "type": "array", "items": { "type": "string" } },
"artifact_manifest": { "type": "array", "items": { "type": "object" } },
"waivers": { "type": "array", "items": { "type": "object" } },
"challenges": { "type": "array", "items": { "type": "object" } },
"exclusions": { "type": "array", "items": { "type": "object" } },
"report_fragments": { "type": "array", "items": { "type": "object" } },
"certification_boundary": { "type": "string" },
"created_at": { "type": "string" }
}

View File

@@ -28,6 +28,8 @@
},
"expectations_ref": { "type": ["string", "null"] },
"waivers_ref": { "type": ["string", "null"] },
"challenges_ref": { "type": ["string", "null"] },
"exclusions_ref": { "type": ["string", "null"] },
"output_policy": { "type": "object" },
"retention_policy": { "type": "object" },
"runtime_policy": { "type": "object" }

View File

@@ -0,0 +1,56 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Guide Board Challenge Set",
"type": "object",
"additionalProperties": false,
"required": [
"id",
"target_profile_ref",
"challenges"
],
"properties": {
"id": { "type": "string" },
"target_profile_ref": { "type": "string" },
"challenges": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"id",
"requirement_refs",
"check_refs",
"evidence_refs",
"result_refs",
"classification_refs",
"authority_source_refs",
"owner",
"review_status",
"rationale",
"created_at",
"review_due_at",
"expires_at",
"native_challenge_id",
"metadata"
],
"properties": {
"id": { "type": "string" },
"requirement_refs": { "type": "array", "items": { "type": "string" } },
"check_refs": { "type": "array", "items": { "type": "string" } },
"evidence_refs": { "type": "array", "items": { "type": "string" } },
"result_refs": { "type": "array", "items": { "type": "string" } },
"classification_refs": { "type": "array", "items": { "type": "string" } },
"authority_source_refs": { "type": "array", "items": { "type": "string" } },
"owner": { "type": "string" },
"review_status": { "type": "string" },
"rationale": { "type": "string" },
"created_at": { "type": "string" },
"review_due_at": { "type": ["string", "null"] },
"expires_at": { "type": ["string", "null"] },
"native_challenge_id": { "type": ["string", "null"] },
"metadata": { "type": "object" }
}
}
}
}
}

View File

@@ -42,6 +42,7 @@
},
"observations": { "type": "array", "items": { "type": "string" } },
"facts": { "type": "object" },
"review": { "type": "object" },
"requirement_refs": { "type": "array", "items": { "type": "string" } },
"artifact_refs": { "type": "array", "items": { "type": "string" } },
"started_at": { "type": "string" },

View File

@@ -0,0 +1,60 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Guide Board Authority Exclusion Set",
"type": "object",
"additionalProperties": false,
"required": [
"id",
"target_profile_ref",
"exclusions"
],
"properties": {
"id": { "type": "string" },
"target_profile_ref": { "type": "string" },
"exclusions": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"id",
"authority_ref",
"requirement_refs",
"check_refs",
"evidence_refs",
"result_refs",
"classification_refs",
"authority_source_refs",
"owner",
"approved_by",
"review_status",
"rationale",
"created_at",
"review_due_at",
"expires_at",
"native_exclusion_id",
"metadata"
],
"properties": {
"id": { "type": "string" },
"authority_ref": { "type": "string" },
"requirement_refs": { "type": "array", "items": { "type": "string" } },
"check_refs": { "type": "array", "items": { "type": "string" } },
"evidence_refs": { "type": "array", "items": { "type": "string" } },
"result_refs": { "type": "array", "items": { "type": "string" } },
"classification_refs": { "type": "array", "items": { "type": "string" } },
"authority_source_refs": { "type": "array", "items": { "type": "string" } },
"owner": { "type": "string" },
"approved_by": { "type": ["string", "null"] },
"review_status": { "type": "string" },
"rationale": { "type": "string" },
"created_at": { "type": "string" },
"review_due_at": { "type": ["string", "null"] },
"expires_at": { "type": ["string", "null"] },
"native_exclusion_id": { "type": ["string", "null"] },
"metadata": { "type": "object" }
}
}
}
}
}

View File

@@ -0,0 +1,36 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Guide Board Export Manifest",
"type": "object",
"additionalProperties": false,
"required": [
"id",
"schema_version",
"export_type",
"run_id",
"created_at",
"source_package_ref",
"source_lock_ref",
"summary",
"policy_summary",
"mapping_summary",
"report_fragments",
"counts",
"certification_boundary"
],
"properties": {
"id": { "type": "string" },
"schema_version": { "type": "string" },
"export_type": { "type": "string" },
"run_id": { "type": "string" },
"created_at": { "type": "string" },
"source_package_ref": { "type": "string" },
"source_lock_ref": { "type": "string" },
"summary": { "type": "object" },
"policy_summary": { "type": "object" },
"mapping_summary": { "type": "object" },
"report_fragments": { "type": "array", "items": { "type": "object" } },
"counts": { "type": "object" },
"certification_boundary": { "type": "string" }
}
}

View File

@@ -41,9 +41,53 @@
"type": "string",
"enum": ["candidate", "incubating", "active", "external", "deprecated"]
},
"supported_frameworks": { "type": "array", "items": { "type": "string" } },
"authorities": { "type": "array", "items": { "type": "string" } },
"profile_schemas": { "type": "array", "items": { "type": "string" } },
"supported_frameworks": {
"type": "array",
"items": {
"type": ["string", "object"],
"additionalProperties": false,
"required": ["id"],
"properties": {
"id": { "type": "string" },
"version": { "type": ["string", "null"] },
"source_url": { "type": ["string", "null"] },
"authority_ref": { "type": ["string", "null"] },
"description": { "type": ["string", "null"] }
}
}
},
"authorities": {
"type": "array",
"items": {
"type": ["string", "object"],
"additionalProperties": false,
"required": ["id"],
"properties": {
"id": { "type": "string" },
"name": { "type": ["string", "null"] },
"version": { "type": ["string", "null"] },
"source_url": { "type": ["string", "null"] },
"license": { "type": ["string", "null"] },
"access": { "type": ["string", "null"] }
}
}
},
"metadata": { "type": "object" },
"profile_schemas": {
"type": "array",
"items": {
"type": ["string", "object"],
"additionalProperties": false,
"required": ["id", "profile_kind", "path"],
"properties": {
"id": { "type": "string" },
"profile_kind": { "type": "string", "enum": ["target", "assessment"] },
"path": { "type": "string" },
"subject_type": { "type": ["string", "null"] },
"description": { "type": ["string", "null"] }
}
}
},
"check_groups": {
"type": "array",
"items": {
@@ -75,13 +119,46 @@
"module_path": { "type": ["string", "null"] },
"callable": { "type": ["string", "null"] },
"command": { "type": ["array", "null"], "items": { "type": "string" } },
"metadata": { "type": "object" },
"description": { "type": ["string", "null"] }
}
}
},
"normalizers": {
"type": "array",
"items": {
"type": ["string", "object"],
"additionalProperties": false,
"required": ["id", "kind", "module_path", "callable"],
"properties": {
"id": { "type": "string" },
"kind": { "type": "string", "enum": ["python_module"] },
"module_path": { "type": "string" },
"callable": { "type": "string" },
"runner_ref": { "type": ["string", "null"] },
"metadata": { "type": "object" },
"description": { "type": ["string", "null"] }
}
}
},
"normalizers": { "type": "array", "items": { "type": "string" } },
"mappings": { "type": "array", "items": { "type": "string" } },
"report_fragments": { "type": "array", "items": { "type": "string" } },
"report_fragments": {
"type": "array",
"items": {
"type": ["string", "object"],
"additionalProperties": false,
"required": ["id", "kind"],
"properties": {
"id": { "type": "string" },
"kind": { "type": "string", "enum": ["markdown_file", "python_module"] },
"path": { "type": ["string", "null"] },
"module_path": { "type": ["string", "null"] },
"callable": { "type": ["string", "null"] },
"title": { "type": ["string", "null"] },
"description": { "type": ["string", "null"] }
}
}
},
"dependencies": { "type": "array", "items": { "type": "string" } },
"restricted_assets": { "type": "array", "items": { "type": "string" } },
"certification_boundary": { "type": "string" }

View File

@@ -14,7 +14,10 @@
"evidence_refs",
"expected",
"waiver_ref",
"challenge_ref",
"exclusion_ref",
"policy_ref",
"review_status",
"remediation"
],
"properties": {
@@ -28,7 +31,10 @@
"evidence_refs": { "type": "array", "items": { "type": "string" } },
"expected": { "type": "boolean" },
"waiver_ref": { "type": ["string", "null"] },
"challenge_ref": { "type": ["string", "null"] },
"exclusion_ref": { "type": ["string", "null"] },
"policy_ref": { "type": ["string", "null"] },
"review_status": { "type": "string" },
"remediation": { "type": ["string", "null"] }
}
}

View File

@@ -21,6 +21,7 @@
"created_at": { "type": "string" },
"summary": { "type": "object" },
"report_refs": { "type": "array", "items": { "type": "string" } },
"export_refs": { "type": "array", "items": { "type": "string" } },
"artifact_retention": { "type": "object" }
}
}

View File

@@ -0,0 +1,34 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Guide Board Source Lock",
"type": "object",
"additionalProperties": false,
"required": [
"id",
"schema_version",
"created_at",
"framework_refs",
"extension_refs",
"frameworks",
"extensions",
"mapping_sets",
"profiles",
"policy_refs",
"authorities",
"metadata_hooks"
],
"properties": {
"id": { "type": "string" },
"schema_version": { "type": "string" },
"created_at": { "type": "string" },
"framework_refs": { "type": "array", "items": { "type": "string" } },
"extension_refs": { "type": "array", "items": { "type": "string" } },
"frameworks": { "type": "array", "items": { "type": "object" } },
"extensions": { "type": "array", "items": { "type": "object" } },
"mapping_sets": { "type": "array", "items": { "type": "object" } },
"profiles": { "type": "object" },
"policy_refs": { "type": "object" },
"authorities": { "type": "array", "items": { "type": "object" } },
"metadata_hooks": { "type": "object" }
}
}

View File

@@ -0,0 +1,36 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Guide Board Submission Package Manifest",
"type": "object",
"additionalProperties": false,
"required": [
"id",
"schema_version",
"run_id",
"created_at",
"package_identity",
"source_lock_ref",
"source_lock",
"reports",
"normalized_outputs",
"profile_snapshots",
"artifact_manifest",
"reported_metadata",
"certification_boundary"
],
"properties": {
"id": { "type": "string" },
"schema_version": { "type": "string" },
"run_id": { "type": "string" },
"created_at": { "type": "string" },
"package_identity": { "type": "object" },
"source_lock_ref": { "type": "string" },
"source_lock": { "type": "object" },
"reports": { "type": "array", "items": { "type": "object" } },
"normalized_outputs": { "type": "array", "items": { "type": "object" } },
"profile_snapshots": { "type": "array", "items": { "type": "object" } },
"artifact_manifest": { "type": "array", "items": { "type": "object" } },
"reported_metadata": { "type": "array", "items": { "type": "object" } },
"certification_boundary": { "type": "string" }
}
}

View File

@@ -0,0 +1,13 @@
# SDK Fixture Extension
`sdk-fixture` is a dependency-light guide-board extension used to exercise the
extension SDK contracts. It is intentionally small and is not a real assessment
program.
The fixture demonstrates:
- extension-owned target and assessment profile schemas,
- a Python runner that writes native output,
- a Python normalizer that converts native output into guide-board evidence,
- a mapping set for normalized requirement refs,
- copyable profiles for SDK acceptance tests.

View File

@@ -0,0 +1,91 @@
{
"id": "sdk-fixture",
"name": "SDK Fixture Extension",
"version": "0.1.0",
"extension_type": "repository_quality",
"lifecycle_status": "incubating",
"supported_frameworks": [
"guide-board.sdk-fixture.v1"
],
"authorities": [],
"metadata": {
"adapter_version": "0.1.0",
"source_url": "https://example.invalid/guide-board/sdk-fixture"
},
"profile_schemas": [
"target-profile",
"assessment-profile",
{
"id": "sdk-fixture-target",
"profile_kind": "target",
"path": "schemas/sdk-fixture-target.schema.json",
"subject_type": "sdk-fixture-target",
"description": "Requires the target shape used by the SDK fixture runner."
},
{
"id": "sdk-fixture-assessment",
"profile_kind": "assessment",
"path": "schemas/sdk-fixture-assessment.schema.json",
"description": "Requires the runtime policy used by the SDK fixture normalizer test."
}
],
"check_groups": [
{
"id": "native-output",
"name": "Native Output Normalization",
"check_type": "repository_quality",
"requirement_refs": [
"guide-board.sdk-fixture.v1.native-output"
],
"runner_ref": "native-probe"
}
],
"preflight_runner": null,
"runner_entrypoints": [
{
"id": "native-probe",
"kind": "python_module",
"module_path": "runners/native_probe.py",
"callable": "run",
"command": null,
"metadata": {
"harness_id": "sdk-fixture-native-probe",
"harness_version": "1.0.0",
"test_suite_id": "sdk-fixture-suite-v1",
"source_url": "https://example.invalid/guide-board/sdk-fixture/native-probe"
},
"description": "Writes a tiny native result artifact for the SDK fixture normalizer."
}
],
"normalizers": [
{
"id": "native-probe-normalizer",
"kind": "python_module",
"module_path": "normalizers/native_probe.py",
"callable": "normalize",
"runner_ref": "native-probe",
"metadata": {
"adapter_version": "0.1.0",
"source_url": "https://example.invalid/guide-board/sdk-fixture/native-normalizer"
},
"description": "Converts the SDK fixture native result artifact into guide-board evidence."
}
],
"mappings": [
"sdk-fixture-map"
],
"report_fragments": [
{
"id": "sdk-fixture-summary",
"kind": "python_module",
"module_path": "reports/sdk_fixture_summary.py",
"callable": "build_fragment",
"path": null,
"title": "SDK Fixture Summary",
"description": "Summarizes SDK fixture evidence for report fragment tests."
}
],
"dependencies": [],
"restricted_assets": [],
"certification_boundary": "SDK fixture only. It does not certify any product, process, or repository."
}

View File

@@ -0,0 +1,16 @@
{
"id": "sdk-fixture-map",
"extension_id": "sdk-fixture",
"framework_refs": [
"guide-board.sdk-fixture.v1"
],
"mappings": [
{
"requirement_ref": "guide-board.sdk-fixture.v1.native-output",
"target_type": "sdk_contract",
"target_id": "normalizer-plugin",
"label": "Normalizer Plug-in Contract",
"description": "The extension runner can emit native output that a normalizer converts into guide-board evidence."
}
]
}

View File

@@ -0,0 +1,32 @@
"""SDK fixture normalizer for native runner output."""
from __future__ import annotations
import json
from pathlib import Path
def normalize(context: dict) -> dict:
run_dir = Path(context["run_dir"])
runner_result = context["runner_result"]
artifact_ref = runner_result["facts"]["native_result_ref"]
native_result = json.loads((run_dir / artifact_ref).read_text(encoding="utf-8"))
native_status = native_result.get("native_status")
result = "pass" if native_status == "ok" else "fail"
return {
"result": result,
"observations": native_result.get("observations", []),
"facts": {
"native_status": native_status,
"native_score": native_result.get("native_score"),
"normalized_by": "native-probe-normalizer"
},
"artifact_refs": [
artifact_ref
],
"requirement_refs": native_result.get("requirement_refs", []),
"metadata": {
"normalizer_id": "native-probe-normalizer",
"native_result_id": "sdk-fixture-native-result"
},
}

View File

@@ -0,0 +1,33 @@
{
"id": "sdk-fixture-assessment",
"framework_refs": [
"guide-board.sdk-fixture.v1"
],
"extension_refs": [
"sdk-fixture"
],
"target_profile_ref": "sdk-fixture-target",
"selected_check_groups": {
"sdk-fixture": [
"native-output"
]
},
"expectations_ref": null,
"waivers_ref": null,
"output_policy": {
"report_formats": [
"json",
"markdown"
],
"artifact_retention": "raw-logs-plus-summary"
},
"retention_policy": {
"summary_days": 365,
"raw_artifact_days": 30
},
"runtime_policy": {
"offline": true,
"timeout_seconds": 30,
"fixture_mode": "native-result"
}
}

View File

@@ -0,0 +1,18 @@
{
"id": "sdk-fixture-target",
"subject_type": "sdk-fixture-target",
"subject_name": "SDK Fixture Target",
"environment": "test",
"scope": [
"Extension SDK fixture validation"
],
"endpoints": [],
"artifacts": [
"extension.json"
],
"credentials_ref": null,
"declared_capabilities": [
"guide-board.sdk-fixture.v1.native-output"
],
"known_gaps": []
}

View File

@@ -0,0 +1,26 @@
"""Report fragment for the SDK fixture extension."""
from __future__ import annotations
def build_fragment(context: dict) -> dict:
evidence_count = len(context["evidence"])
finding_count = len(context["findings"])
source_lock = context["source_lock"]
markdown = "\n".join(
[
"### SDK Fixture Summary",
"",
f"- evidence items: {evidence_count}",
f"- findings: {finding_count}",
f"- source lock: {source_lock.get('id')}",
]
)
return {
"markdown": markdown,
"structured": {
"evidence_count": evidence_count,
"finding_count": finding_count,
"source_lock_ref": source_lock.get("id"),
},
}

View File

@@ -0,0 +1,40 @@
"""SDK fixture runner that writes a native result artifact."""
from __future__ import annotations
import json
from pathlib import Path
def run(context: dict) -> dict:
run_dir = Path(context["run_dir"])
artifact_path = run_dir / "artifacts" / "sdk-fixture" / "native-result.json"
artifact_path.parent.mkdir(parents=True, exist_ok=True)
native_result = {
"native_status": "ok",
"native_score": 98,
"observations": [
"SDK fixture native probe completed."
],
"requirement_refs": [
"guide-board.sdk-fixture.v1.native-output"
],
}
artifact_path.write_text(json.dumps(native_result, indent=2, sort_keys=True), encoding="utf-8")
artifact_ref = str(artifact_path.relative_to(run_dir))
return {
"result": "unknown",
"observations": [
"SDK fixture runner wrote native output for normalization."
],
"facts": {
"native_result_ref": artifact_ref
},
"artifact_refs": [
artifact_ref
],
"metadata": {
"native_result_id": "sdk-fixture-native-result",
"test_suite_id": "sdk-fixture-suite-v1"
},
}

View File

@@ -0,0 +1,19 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "SDK Fixture Assessment Profile",
"type": "object",
"required": [
"runtime_policy"
],
"properties": {
"runtime_policy": {
"type": "object",
"required": [
"fixture_mode"
],
"properties": {
"fixture_mode": { "enum": ["native-result"] }
}
}
}
}

View File

@@ -0,0 +1,15 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "SDK Fixture Target Profile",
"type": "object",
"required": [
"subject_type",
"artifacts",
"declared_capabilities"
],
"properties": {
"subject_type": { "enum": ["sdk-fixture-target"] },
"artifacts": { "type": "array", "minItems": 1 },
"declared_capabilities": { "type": "array", "minItems": 1 }
}
}

12
registry/README.md Normal file
View File

@@ -0,0 +1,12 @@
# Capability Registry
Markdown-first capability index for federation and reuse planning.
## Authoring
1. Copy a capability entry template (see reuse-surface `templates/capability-entry.template.md`).
2. Add the row to `indexes/capabilities.yaml`.
3. Run `reuse-surface validate` from a checkout with the CLI installed.
4. Merge to `main` and verify publish with `reuse-surface establish --publish-check`.
Federation contract: reuse-surface `docs/RegistryFederation.md`.

View File

View File

@@ -0,0 +1,4 @@
version: 1
updated: '2026-06-16'
domain: helix_forge
capabilities: []

View File

@@ -35,12 +35,16 @@ echo "==> Verifying mounted run artifacts"
for path in \
"$RUNS_DIR/sample-noop/run.json" \
"$RUNS_DIR/sample-noop/plan.json" \
"$RUNS_DIR/sample-noop/sources.lock.json" \
"$RUNS_DIR/sample-noop/retention-summary.json" \
"$RUNS_DIR/sample-noop/normalized/evidence.json" \
"$RUNS_DIR/sample-noop/normalized/findings.json" \
"$RUNS_DIR/sample-noop/normalized/mappings.json" \
"$RUNS_DIR/sample-noop/reports/assessment-package.json" \
"$RUNS_DIR/sample-noop/reports/report.md"
"$RUNS_DIR/sample-noop/reports/report.md" \
"$RUNS_DIR/sample-noop/reports/fragments.json" \
"$RUNS_DIR/sample-noop/reports/submission-package.json" \
"$RUNS_DIR/sample-noop/exports/export-manifest.json"
do
if [ ! -f "$path" ]; then
echo "ERROR: expected artifact missing: $path" >&2

View File

@@ -0,0 +1,106 @@
#!/usr/bin/env sh
set -eu
ROOT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)"
MODE="${1:-plan}"
EXTENSION_DIR="${GUIDE_BOARD_ACCEPT_EXTENSION_DIR:-../open-cmis-tck}"
TARGET="${GUIDE_BOARD_ACCEPT_TARGET:-../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json}"
ASSESSMENT="${GUIDE_BOARD_ACCEPT_ASSESSMENT:-../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json}"
OUTPUT_DIR="${GUIDE_BOARD_ACCEPT_OUTPUT_DIR:-${TMPDIR:-/tmp}/guide-board-external-extension-acceptance}"
PLAN_OUTPUT="${GUIDE_BOARD_ACCEPT_PLAN_OUTPUT:-$OUTPUT_DIR/plan.json}"
RUN_DIR="${GUIDE_BOARD_ACCEPT_RUN_DIR:-$OUTPUT_DIR/open-cmis-tck-baseline}"
PYTHON="${PYTHON:-python3}"
case "$MODE" in
plan|run)
;;
*)
echo "Usage: $0 [plan|run]" >&2
exit 64
;;
esac
cd "$ROOT_DIR"
export PYTHONPATH="$ROOT_DIR/src${PYTHONPATH:+:$PYTHONPATH}"
if [ ! -f "$EXTENSION_DIR/extension.json" ]; then
echo "ERROR: extension manifest not found at $EXTENSION_DIR/extension.json" >&2
exit 66
fi
mkdir -p "$OUTPUT_DIR"
echo "==> Listing discovered extensions"
"$PYTHON" -m guide_board \
--root "$ROOT_DIR" \
--extension-dir "$EXTENSION_DIR" \
extensions list
echo "==> Validating extension manifests"
"$PYTHON" -m guide_board \
--root "$ROOT_DIR" \
--extension-dir "$EXTENSION_DIR" \
extensions validate
echo "==> Validating target profile"
"$PYTHON" -m guide_board \
--root "$ROOT_DIR" \
--extension-dir "$EXTENSION_DIR" \
profile validate-target "$TARGET"
echo "==> Validating assessment profile"
"$PYTHON" -m guide_board \
--root "$ROOT_DIR" \
--extension-dir "$EXTENSION_DIR" \
profile validate-assessment "$ASSESSMENT"
echo "==> Generating run plan"
"$PYTHON" -m guide_board \
--root "$ROOT_DIR" \
--extension-dir "$EXTENSION_DIR" \
plan \
--target "$TARGET" \
--assessment "$ASSESSMENT" \
--output "$PLAN_OUTPUT"
if [ "$MODE" = "plan" ]; then
echo "External extension acceptance plan passed."
echo "Plan output: $PLAN_OUTPUT"
echo "Run live acceptance with: $0 run"
exit 0
fi
echo "==> Running live assessment"
"$PYTHON" -m guide_board \
--root "$ROOT_DIR" \
--extension-dir "$EXTENSION_DIR" \
run \
--target "$TARGET" \
--assessment "$ASSESSMENT" \
--output-dir "$RUN_DIR"
echo "==> Verifying retained run artifacts"
for path in \
"$RUN_DIR/run.json" \
"$RUN_DIR/plan.json" \
"$RUN_DIR/retention-summary.json" \
"$RUN_DIR/normalized/evidence.json" \
"$RUN_DIR/normalized/findings.json" \
"$RUN_DIR/normalized/mappings.json" \
"$RUN_DIR/reports/assessment-package.json" \
"$RUN_DIR/reports/report.md"
do
if [ ! -f "$path" ]; then
echo "ERROR: expected artifact missing: $path" >&2
exit 1
fi
done
echo "==> Reviewing retained report paths"
"$PYTHON" -m guide_board \
--root "$ROOT_DIR" \
runs report \
--runs-dir "$OUTPUT_DIR"
echo "External extension live acceptance passed."
echo "Run artifacts: $RUN_DIR"

View File

@@ -46,6 +46,107 @@ def build_artifact_manifest(
return artifacts
def build_submission_manifest(
run_dir: Path,
run_metadata: dict[str, Any],
plan: dict[str, Any],
evidence: list[dict[str, Any]],
assessment_package: dict[str, Any],
) -> dict[str, Any]:
"""Build a portable manifest for the files that make up a review package."""
source_lock = plan["source_lock"]
manifest = {
"id": f"submission-package:{run_metadata['id']}",
"schema_version": "guide-board.submission-package.v1",
"run_id": run_metadata["id"],
"created_at": datetime.now(timezone.utc).isoformat(),
"package_identity": {
"target_profile_ref": run_metadata["target_profile_ref"],
"assessment_profile_ref": run_metadata["assessment_profile_ref"],
"framework_refs": source_lock.get("framework_refs", []),
"extension_refs": source_lock.get("extension_refs", []),
},
"source_lock_ref": "sources.lock.json",
"source_lock": {
"id": source_lock.get("id"),
"schema_version": source_lock.get("schema_version"),
"checksum": _file_entry(run_dir, "sources.lock.json").get("checksum"),
"framework_refs": source_lock.get("framework_refs", []),
"extension_refs": source_lock.get("extension_refs", []),
},
"reports": _existing_file_entries(
run_dir,
[
("assessment-package", "reports/assessment-package.json"),
("markdown-report", "reports/report.md"),
],
),
"normalized_outputs": _existing_file_entries(
run_dir,
[
("evidence", "normalized/evidence.json"),
("findings", "normalized/findings.json"),
("mappings", "normalized/mappings.json"),
],
),
"profile_snapshots": _existing_file_entries(
run_dir,
[
("target-profile", "target-profile.snapshot.json"),
("assessment-profile", "assessment-profile.snapshot.json"),
],
),
"artifact_manifest": assessment_package.get("artifact_manifest", []),
"reported_metadata": _reported_metadata(evidence),
"certification_boundary": assessment_package["certification_boundary"],
}
assert_valid(manifest, "submission-package")
return manifest
def _existing_file_entries(run_dir: Path, refs: list[tuple[str, str]]) -> list[dict[str, Any]]:
entries = []
for entry_id, ref in refs:
entry = _file_entry(run_dir, ref)
if entry:
entry["id"] = entry_id
entries.append(entry)
return entries
def _file_entry(run_dir: Path, ref: str) -> dict[str, Any]:
path = (run_dir / ref).resolve()
try:
path.relative_to(run_dir.resolve())
except ValueError:
return {}
if not path.is_file():
return {}
return {
"path": ref,
"media_type": _media_type(path),
"checksum": f"sha256:{_sha256(path)}",
"size_bytes": path.stat().st_size,
}
def _reported_metadata(evidence: list[dict[str, Any]]) -> list[dict[str, Any]]:
records = []
for item in evidence:
metadata = item.get("facts", {}).get("source_metadata")
if not isinstance(metadata, dict) or not metadata:
continue
records.append(
{
"evidence_ref": item["id"],
"check_id": item["check_id"],
"extension_id": item["extension_id"],
"metadata": metadata,
}
)
return records
def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:

View File

@@ -155,12 +155,18 @@ def cmd_extensions_validate(args: argparse.Namespace) -> dict[str, Any]:
def cmd_validate_target(args: argparse.Namespace) -> dict[str, Any]:
profile = validate_target_profile(args.path)
profile = validate_target_profile(
args.path,
discover_extensions(args.root, args.extension_dir),
)
return {"status": "valid", "target_profile": profile["id"]}
def cmd_validate_assessment(args: argparse.Namespace) -> dict[str, Any]:
profile = validate_assessment_profile(args.path)
profile = validate_assessment_profile(
args.path,
discover_extensions(args.root, args.extension_dir),
)
return {"status": "valid", "assessment_profile": profile["id"]}

View File

@@ -7,11 +7,14 @@ from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from guide_board.artifacts import build_artifact_manifest
from guide_board.io import write_json
from guide_board.artifacts import build_artifact_manifest, build_submission_manifest
from guide_board.exports import build_export_manifest
from guide_board.io import load_json, write_json
from guide_board.mapping import build_mapping_records, summarize_mappings
from guide_board.normalizers import normalize_step_result
from guide_board.planning import build_run_plan
from guide_board.policy import apply_policy
from guide_board.reports import build_report_fragments, markdown_for_fragments
from guide_board.retention import build_retention_summary
from guide_board.runners import run_step
from guide_board.schema import assert_valid
@@ -34,7 +37,15 @@ def run_assessment(
assert_valid(item, "evidence-item")
findings = _findings_for_evidence(run_id, evidence)
findings, policy_summary, applied_waivers = apply_policy(root, plan, findings)
(
findings,
policy_summary,
applied_waivers,
applied_challenges,
applied_exclusions,
) = apply_policy(root, plan, evidence, findings)
for item in evidence:
assert_valid(item, "evidence-item")
for finding in findings:
assert_valid(finding, "finding")
@@ -51,9 +62,23 @@ def run_assessment(
mapping_summary,
policy_summary,
applied_waivers,
applied_challenges,
applied_exclusions,
created_at,
)
report_fragments = build_report_fragments(
root,
run_dir,
run_id,
plan,
evidence,
findings,
mapping_records,
assessment_package,
)
assessment_package["report_fragments"] = report_fragments
assert_valid(assessment_package, "assessment-package")
export_manifest = build_export_manifest(assessment_package)
run_metadata = {
"id": run_id,
@@ -74,6 +99,7 @@ def run_assessment(
findings,
mapping_records,
assessment_package,
export_manifest,
retention_summary,
)
return {
@@ -82,6 +108,8 @@ def run_assessment(
"run_dir": str(run_dir),
"assessment_package": str(run_dir / "reports" / "assessment-package.json"),
"report": str(run_dir / "reports" / "report.md"),
"submission_package": str(run_dir / "reports" / "submission-package.json"),
"export_manifest": str(run_dir / "exports" / "export-manifest.json"),
"retention_summary": str(run_dir / "retention-summary.json"),
}
@@ -153,6 +181,15 @@ def _evidence_for_step(
now = _now()
runner_ref = step.get("runner_ref")
runner_result = run_step(root, run_dir, run_id, plan, step)
runner_result = normalize_step_result(root, run_dir, run_id, plan, step, runner_result)
facts = {
"step_kind": step["kind"],
"runner_ref": runner_ref,
**runner_result["facts"],
}
source_metadata = _source_metadata_for_step(root, plan, step, runner_result)
if source_metadata:
facts["source_metadata"] = source_metadata
return {
"id": f"evidence:{step['id']}",
@@ -162,22 +199,134 @@ def _evidence_for_step(
"subject_ref": plan["target_profile_snapshot"]["id"],
"result": runner_result["result"],
"observations": runner_result["observations"],
"facts": {
"step_kind": step["kind"],
"runner_ref": runner_ref,
**runner_result["facts"],
},
"requirement_refs": _requirement_refs(plan, step),
"facts": facts,
"requirement_refs": _requirement_refs(plan, step, runner_result),
"artifact_refs": runner_result["artifact_refs"],
"started_at": now,
"completed_at": now,
}
def _requirement_refs(plan: dict[str, Any], step: dict[str, Any]) -> list[str]:
def _requirement_refs(
plan: dict[str, Any],
step: dict[str, Any],
runner_result: dict[str, Any] | None = None,
) -> list[str]:
refs = []
if step["kind"] != "check_group":
return _runner_requirement_refs(runner_result)
refs.extend(step.get("requirement_refs", []))
refs.extend(_runner_requirement_refs(runner_result))
return _dedupe(refs)
def _runner_requirement_refs(runner_result: dict[str, Any] | None) -> list[str]:
if not runner_result:
return []
return list(step.get("requirement_refs", []))
refs = runner_result.get("requirement_refs", [])
if not isinstance(refs, list):
return []
return [ref for ref in refs if isinstance(ref, str)]
def _source_metadata_for_step(
root: Path,
plan: dict[str, Any],
step: dict[str, Any],
runner_result: dict[str, Any],
) -> dict[str, Any]:
runner_ref = step.get("runner_ref")
extension = _extension_snapshot(plan, step["extension_id"])
extension_path = _snapshot_path(root, extension)
manifest = load_json(extension_path / "extension.json")
metadata: dict[str, Any] = {
"extension": {
"id": step["extension_id"],
"version": extension.get("version"),
"metadata": _object_or_empty(manifest.get("metadata")),
}
}
if runner_ref:
entrypoint = _runner_entrypoint(manifest, runner_ref)
metadata["runner"] = {
"id": runner_ref,
"kind": entrypoint.get("kind"),
"metadata": _object_or_empty(entrypoint.get("metadata")),
}
applied_normalizers = runner_result.get("facts", {}).get("normalizer_refs", [])
normalizers = []
if isinstance(applied_normalizers, list):
normalizer_ids = {item for item in applied_normalizers if isinstance(item, str)}
for normalizer in manifest.get("normalizers", []):
if isinstance(normalizer, dict) and normalizer.get("id") in normalizer_ids:
normalizers.append(
{
"id": normalizer["id"],
"kind": normalizer.get("kind"),
"runner_ref": normalizer.get("runner_ref"),
"metadata": _object_or_empty(normalizer.get("metadata")),
}
)
if normalizers:
metadata["normalizers"] = normalizers
reported = _object_or_empty(runner_result.get("metadata"))
if reported:
metadata["reported"] = reported
return _drop_empty_metadata(metadata)
def _extension_snapshot(plan: dict[str, Any], extension_id: str) -> dict[str, Any]:
for extension in plan["extension_snapshots"]:
if extension["id"] == extension_id:
return extension
return {}
def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path:
path = Path(extension["path"])
return path if path.is_absolute() else root / path
def _runner_entrypoint(manifest: dict[str, Any], runner_ref: str) -> dict[str, Any]:
for entrypoint in manifest.get("runner_entrypoints", []):
if entrypoint.get("id") == runner_ref:
return entrypoint
return {}
def _object_or_empty(value: Any) -> dict[str, Any]:
return value if isinstance(value, dict) else {}
def _drop_empty_metadata(value: dict[str, Any]) -> dict[str, Any]:
compact = {}
for key, child in value.items():
if isinstance(child, dict):
child = _drop_empty_metadata(child)
if isinstance(child, list):
child = [
_drop_empty_metadata(item) if isinstance(item, dict) else item
for item in child
]
child = [item for item in child if item]
if child:
compact[key] = child
return compact
def _dedupe(values: list[str]) -> list[str]:
seen = set()
deduped = []
for value in values:
if value in seen:
continue
seen.add(value)
deduped.append(value)
return deduped
def _findings_for_evidence(run_id: str, evidence: list[dict[str, Any]]) -> list[dict[str, Any]]:
@@ -185,6 +334,7 @@ def _findings_for_evidence(run_id: str, evidence: list[dict[str, Any]]) -> list[
for item in evidence:
if item["result"] not in {"blocked", "fail", "infrastructure_error"}:
continue
expected = _expected_for_item(item)
findings.append(
{
"id": f"finding:{item['check_id']}",
@@ -195,9 +345,12 @@ def _findings_for_evidence(run_id: str, evidence: list[dict[str, Any]]) -> list[
"classification": _classification_for_item(item),
"requirement_refs": item["requirement_refs"],
"evidence_refs": [item["id"]],
"expected": _expected_for_item(item),
"expected": expected,
"waiver_ref": None,
"challenge_ref": None,
"exclusion_ref": None,
"policy_ref": None,
"review_status": "expected" if expected else "unresolved_defect",
"remediation": _remediation_for_item(item),
}
)
@@ -259,6 +412,8 @@ def _assessment_package(
mapping_summary: dict[str, Any],
policy_summary: dict[str, Any],
applied_waivers: list[dict[str, Any]],
applied_challenges: list[dict[str, Any]],
applied_exclusions: list[dict[str, Any]],
created_at: str,
) -> dict[str, Any]:
summary = dict(Counter(item["result"] for item in evidence))
@@ -278,6 +433,9 @@ def _assessment_package(
"evidence_refs": [item["id"] for item in evidence],
"artifact_manifest": artifact_manifest,
"waivers": applied_waivers,
"challenges": applied_challenges,
"exclusions": applied_exclusions,
"report_fragments": [],
"certification_boundary": "Guide Board produces preparation evidence only and does not issue certifications or audit assurance.",
"created_at": created_at,
}
@@ -291,6 +449,7 @@ def _write_run_directory(
findings: list[dict[str, Any]],
mapping_records: list[dict[str, Any]],
assessment_package: dict[str, Any],
export_manifest: dict[str, Any],
retention_summary: dict[str, Any],
) -> None:
write_json(run_dir / "run.json", run_metadata)
@@ -306,11 +465,21 @@ def _write_run_directory(
write_json(run_dir / "normalized" / "findings.json", {"findings": findings})
write_json(run_dir / "normalized" / "mappings.json", {"mappings": mapping_records})
write_json(run_dir / "reports" / "assessment-package.json", assessment_package)
write_json(run_dir / "reports" / "fragments.json", {"fragments": assessment_package["report_fragments"]})
write_json(run_dir / "exports" / "export-manifest.json", export_manifest)
(run_dir / "reports").mkdir(parents=True, exist_ok=True)
(run_dir / "reports" / "report.md").write_text(
_markdown_report(run_metadata, assessment_package),
encoding="utf-8",
)
submission_manifest = build_submission_manifest(
run_dir,
run_metadata,
plan,
evidence,
assessment_package,
)
write_json(run_dir / "reports" / "submission-package.json", submission_manifest)
def _markdown_report(run_metadata: dict[str, Any], package: dict[str, Any]) -> str:
@@ -321,6 +490,8 @@ def _markdown_report(run_metadata: dict[str, Any], package: dict[str, Any]) -> s
summary_lines = "- no evidence produced"
mapping_lines = _mapping_summary_lines(package)
policy_lines = _policy_summary_lines(package)
review_lines = _review_summary_lines(package)
fragment_lines = markdown_for_fragments(package.get("report_fragments", []))
return "\n".join(
[
@@ -342,6 +513,14 @@ def _markdown_report(run_metadata: dict[str, Any], package: dict[str, Any]) -> s
"",
policy_lines,
"",
"## Review",
"",
review_lines,
"",
"## Extension Fragments",
"",
fragment_lines,
"",
"## Boundary",
"",
package["certification_boundary"],
@@ -371,10 +550,27 @@ def _policy_summary_lines(package: dict[str, Any]) -> str:
f"- applied expectations: {summary.get('applied_expectations', 0)}",
f"- applied waivers: {summary.get('applied_waivers', 0)}",
f"- unexpected findings: {summary.get('unexpected_findings', 0)}",
f"- challenged findings: {summary.get('challenged_findings', 0)}",
f"- authority exclusions: {summary.get('authority_exclusions', 0)}",
f"- unresolved defects: {summary.get('unresolved_defects', 0)}",
]
)
def _review_summary_lines(package: dict[str, Any]) -> str:
findings = package.get("findings", [])
if not findings:
return "- no findings requiring review"
counts = Counter(
finding.get("review_status", "unreviewed")
for finding in findings
if isinstance(finding, dict)
)
return "\n".join(
f"- {status}: {count}" for status, count in sorted(counts.items())
)
def _run_status(evidence: list[dict[str, Any]]) -> str:
if any(item["result"] == "fail" for item in evidence):
return "failed"

View File

@@ -0,0 +1,50 @@
"""Portable export builders derived from assessment packages."""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any
from guide_board.schema import assert_valid
def build_export_manifest(assessment_package: dict[str, Any]) -> dict[str, Any]:
manifest = {
"id": f"export-manifest:{assessment_package['run_id']}",
"schema_version": "guide-board.export-manifest.v1",
"export_type": "guide-board.generic-json.v1",
"run_id": assessment_package["run_id"],
"created_at": datetime.now(timezone.utc).isoformat(),
"source_package_ref": "reports/assessment-package.json",
"source_lock_ref": "sources.lock.json",
"summary": assessment_package.get("summary", {}),
"policy_summary": assessment_package.get("policy_summary", {}),
"mapping_summary": assessment_package.get("mapping_summary", {}),
"report_fragments": [
_export_fragment(fragment)
for fragment in assessment_package.get("report_fragments", [])
if isinstance(fragment, dict)
],
"counts": {
"evidence_refs": len(assessment_package.get("evidence_refs", [])),
"findings": len(assessment_package.get("findings", [])),
"artifacts": len(assessment_package.get("artifact_manifest", [])),
"waivers": len(assessment_package.get("waivers", [])),
"challenges": len(assessment_package.get("challenges", [])),
"exclusions": len(assessment_package.get("exclusions", [])),
"report_fragments": len(assessment_package.get("report_fragments", [])),
},
"certification_boundary": assessment_package["certification_boundary"],
}
assert_valid(manifest, "export-manifest")
return manifest
def _export_fragment(fragment: dict[str, Any]) -> dict[str, Any]:
return {
"id": fragment.get("id"),
"extension_id": fragment.get("extension_id"),
"title": fragment.get("title"),
"kind": fragment.get("kind"),
"structured": fragment.get("structured", {}),
}

View File

@@ -0,0 +1,236 @@
"""Normalizer plug-in bridge for extension-provided runner output."""
from __future__ import annotations
import importlib.util
from pathlib import Path
from types import ModuleType
from typing import Any
from guide_board.errors import ValidationError
from guide_board.io import load_json
def normalize_step_result(
root: Path,
run_dir: Path,
run_id: str,
plan: dict[str, Any],
step: dict[str, Any],
runner_result: dict[str, Any],
) -> dict[str, Any]:
"""Apply matching extension normalizers to a runner result."""
extension = _extension_snapshot(plan, step["extension_id"])
extension_path = _snapshot_path(root, extension)
manifest = load_json(extension_path / "extension.json")
result = _coerce_result(runner_result)
applied: list[str] = []
for normalizer in _matching_normalizers(manifest, step):
normalized = _run_normalizer(
root,
run_dir,
run_id,
plan,
step,
extension_path,
normalizer,
result,
)
if _is_normalizer_error(normalized):
return normalized
result = _merge_result(result, normalized)
applied.append(normalizer["id"])
if applied:
facts = dict(result.get("facts", {}))
facts["normalizer_refs"] = applied
result["facts"] = facts
return result
def _matching_normalizers(
manifest: dict[str, Any],
step: dict[str, Any],
) -> list[dict[str, Any]]:
matching = []
runner_ref = step.get("runner_ref")
for normalizer in manifest.get("normalizers", []):
if not isinstance(normalizer, dict):
continue
normalizer_runner_ref = normalizer.get("runner_ref")
if normalizer_runner_ref and normalizer_runner_ref != runner_ref:
continue
matching.append(normalizer)
return matching
def _run_normalizer(
root: Path,
run_dir: Path,
run_id: str,
plan: dict[str, Any],
step: dict[str, Any],
extension_path: Path,
normalizer: dict[str, Any],
runner_result: dict[str, Any],
) -> dict[str, Any]:
if normalizer["kind"] != "python_module":
raise ValidationError(
f"{normalizer['id']}: unsupported normalizer kind {normalizer['kind']!r}"
)
module_path = normalizer.get("module_path")
callable_name = normalizer.get("callable")
if not module_path or not callable_name:
raise ValidationError(
f"{normalizer['id']}: python_module normalizers need module_path and callable"
)
module_file = (extension_path / module_path).resolve()
try:
module_file.relative_to(extension_path.resolve())
except ValueError as exc:
raise ValidationError(
f"{normalizer['id']}: module_path must stay inside the extension directory"
) from exc
module = _load_module(module_file, normalizer["id"])
normalizer_callable = getattr(module, callable_name, None)
if not callable(normalizer_callable):
raise ValidationError(f"{normalizer['id']}: callable {callable_name!r} was not found")
context = {
"root": str(root),
"run_dir": str(run_dir),
"run_id": run_id,
"plan": plan,
"step": step,
"target_profile": plan["target_profile_snapshot"],
"assessment_profile": plan["assessment_profile_snapshot"],
"extension_path": str(extension_path),
"normalizer": normalizer,
"runner_result": runner_result,
}
try:
result = normalizer_callable(context)
except Exception as exc: # noqa: BLE001 - extension failures become evidence.
return {
"result": "infrastructure_error",
"observations": [
f"Normalizer {normalizer['id']!r} failed before producing evidence: {exc}"
],
"facts": {
"normalizer_ref": normalizer["id"],
"normalizer_kind": normalizer["kind"],
"error_type": type(exc).__name__,
},
"artifact_refs": runner_result.get("artifact_refs", []),
"requirement_refs": runner_result.get("requirement_refs", []),
"metadata": runner_result.get("metadata", {}),
}
if not isinstance(result, dict):
raise ValidationError(f"{normalizer['id']}: normalizer must return an object")
return result
def _merge_result(
base: dict[str, Any],
update: dict[str, Any],
) -> dict[str, Any]:
merged = dict(base)
if "result" in update:
merged["result"] = update["result"]
if "observations" in update:
merged["observations"] = _string_list(base.get("observations", []))
merged["observations"].extend(_string_list(update.get("observations", [])))
if "facts" in update:
facts = dict(base.get("facts", {}))
update_facts = update.get("facts", {})
if isinstance(update_facts, dict):
facts.update(update_facts)
merged["facts"] = facts
if "artifact_refs" in update:
merged["artifact_refs"] = _dedupe(
_string_list(base.get("artifact_refs", []))
+ _string_list(update.get("artifact_refs", []))
)
if "requirement_refs" in update:
merged["requirement_refs"] = _dedupe(
_string_list(base.get("requirement_refs", []))
+ _string_list(update.get("requirement_refs", []))
)
if "metadata" in update:
metadata = dict(base.get("metadata", {}))
update_metadata = update.get("metadata", {})
if isinstance(update_metadata, dict):
metadata.update(update_metadata)
merged["metadata"] = metadata
return _coerce_result(merged)
def _coerce_result(value: dict[str, Any]) -> dict[str, Any]:
facts = value.get("facts", {})
if not isinstance(facts, dict):
facts = {}
return {
"result": value.get("result", "unknown"),
"observations": _string_list(value.get("observations", [])),
"facts": facts,
"artifact_refs": _string_list(value.get("artifact_refs", [])),
"requirement_refs": _string_list(value.get("requirement_refs", [])),
"metadata": _object_or_empty(value.get("metadata")),
}
def _is_normalizer_error(result: dict[str, Any]) -> bool:
return (
result.get("result") == "infrastructure_error"
and "normalizer_ref" in result.get("facts", {})
)
def _string_list(value: Any) -> list[str]:
if not isinstance(value, list):
return []
return [item for item in value if isinstance(item, str)]
def _object_or_empty(value: Any) -> dict[str, Any]:
return value if isinstance(value, dict) else {}
def _dedupe(values: list[str]) -> list[str]:
seen = set()
deduped = []
for value in values:
if value in seen:
continue
seen.add(value)
deduped.append(value)
return deduped
def _load_module(path: Path, normalizer_id: str) -> ModuleType:
if not path.exists():
raise ValidationError(f"{normalizer_id}: module not found: {path}")
module_name = f"_guide_board_normalizer_{normalizer_id.replace('-', '_')}"
spec = importlib.util.spec_from_file_location(module_name, path)
if spec is None or spec.loader is None:
raise ValidationError(f"{normalizer_id}: unable to load module from {path}")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def _extension_snapshot(plan: dict[str, Any], extension_id: str) -> dict[str, Any]:
for extension in plan["extension_snapshots"]:
if extension["id"] == extension_id:
return extension
raise ValidationError(f"step references unknown extension {extension_id!r}")
def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path:
path = Path(extension["path"])
return path if path.is_absolute() else root / path

View File

@@ -2,25 +2,37 @@
from __future__ import annotations
import hashlib
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from guide_board.discovery import discover_extensions
from guide_board.discovery import Extension, discover_extensions
from guide_board.errors import ValidationError
from guide_board.io import load_json
from guide_board.schema import assert_valid
from guide_board.schema import assert_valid, validate_document
def validate_target_profile(path: Path) -> dict[str, Any]:
def validate_target_profile(
path: Path,
extensions: list[Extension] | None = None,
) -> dict[str, Any]:
document = load_json(path)
assert_valid(document, "target-profile")
if extensions:
_validate_extension_profile_schemas(document, "target", extensions)
return document
def validate_assessment_profile(path: Path) -> dict[str, Any]:
def validate_assessment_profile(
path: Path,
extensions: list[Extension] | None = None,
) -> dict[str, Any]:
document = load_json(path)
assert_valid(document, "assessment-profile")
if extensions:
selected_extensions = _selected_assessment_extensions(document, extensions)
_validate_extension_profile_schemas(document, "assessment", selected_extensions)
return document
@@ -41,6 +53,10 @@ def build_run_plan(
missing = [extension_id for extension_id in selected_extensions if extension_id not in extensions]
if missing:
raise ValidationError(f"assessment references unknown extension(s): {', '.join(missing)}")
selected_extension_records = [extensions[extension_id] for extension_id in selected_extensions]
_validate_extension_profile_schemas(target, "target", selected_extension_records)
_validate_extension_profile_schemas(assessment, "assessment", selected_extension_records)
if assessment["target_profile_ref"] != target["id"]:
raise ValidationError(
@@ -81,6 +97,16 @@ def build_run_plan(
}
)
source_lock = _build_source_lock(
root,
target_path,
assessment_path,
target,
assessment,
[extensions[extension_id] for extension_id in selected_extensions],
)
assert_valid(source_lock, "source-lock")
plan = {
"id": f"plan-{_timestamp()}",
"assessment_profile_snapshot": assessment,
@@ -94,10 +120,7 @@ def build_run_plan(
}
for extension_id in selected_extensions
],
"source_lock": {
"framework_refs": assessment["framework_refs"],
"extension_refs": selected_extensions,
},
"source_lock": source_lock,
"profile_paths": {
"target_profile_path": str(target_path.resolve()),
"assessment_profile_path": str(assessment_path.resolve()),
@@ -119,6 +142,356 @@ def _credential_refs(target: dict[str, Any]) -> list[str]:
return []
def _selected_assessment_extensions(
assessment: dict[str, Any],
extensions: list[Extension],
) -> list[Extension]:
by_id = {extension.id: extension for extension in extensions}
selected_ids = assessment.get("extension_refs", [])
selected_extensions = []
missing = []
for extension_id in selected_ids:
if extension_id in by_id:
selected_extensions.append(by_id[extension_id])
else:
missing.append(extension_id)
if missing:
raise ValidationError(f"assessment references unknown extension(s): {', '.join(missing)}")
return selected_extensions
def _validate_extension_profile_schemas(
profile: dict[str, Any],
profile_kind: str,
extensions: list[Extension],
) -> None:
for extension in extensions:
for descriptor in _profile_schema_descriptors(extension, profile_kind, profile):
schema = _load_extension_profile_schema(extension, descriptor)
errors = validate_document(profile, schema)
if errors:
formatted = "\n".join(f"- {error}" for error in errors)
raise ValidationError(
f"{extension.id}:{descriptor['id']} profile schema validation failed:\n"
f"{formatted}"
)
def _profile_schema_descriptors(
extension: Extension,
profile_kind: str,
profile: dict[str, Any],
) -> list[dict[str, Any]]:
descriptors = []
for raw_descriptor in extension.manifest.get("profile_schemas", []):
if not isinstance(raw_descriptor, dict):
continue
if raw_descriptor.get("profile_kind") != profile_kind:
continue
subject_type = raw_descriptor.get("subject_type")
if profile_kind == "target" and subject_type and subject_type != profile.get("subject_type"):
continue
descriptors.append(raw_descriptor)
return descriptors
def _load_extension_profile_schema(
extension: Extension,
descriptor: dict[str, Any],
) -> dict[str, Any]:
raw_path = descriptor["path"]
schema_path = (extension.path / raw_path).resolve()
extension_root = extension.path.resolve()
try:
schema_path.relative_to(extension_root)
except ValueError as exc:
raise ValidationError(
f"{extension.id}:{descriptor['id']} profile schema path escapes extension root: "
f"{raw_path!r}"
) from exc
if not schema_path.is_file():
raise ValidationError(
f"{extension.id}:{descriptor['id']} profile schema not found: {raw_path!r}"
)
return load_json(schema_path)
def _build_source_lock(
root: Path,
target_path: Path,
assessment_path: Path,
target: dict[str, Any],
assessment: dict[str, Any],
extensions: list[Extension],
) -> dict[str, Any]:
framework_refs = assessment["framework_refs"]
extension_refs = [extension.id for extension in extensions]
return {
"id": f"source-lock:{assessment['id']}:{target['id']}",
"schema_version": "guide-board.source-lock.v1",
"created_at": _now(),
"framework_refs": framework_refs,
"extension_refs": extension_refs,
"frameworks": _framework_records(framework_refs, extensions),
"extensions": [_extension_source_record(root, extension) for extension in extensions],
"mapping_sets": _mapping_source_records(root, extensions),
"profiles": {
"target": _file_source_record(
"target-profile",
target["id"],
target_path,
"target-profile.snapshot.json",
),
"assessment": _file_source_record(
"assessment-profile",
assessment["id"],
assessment_path,
"assessment-profile.snapshot.json",
),
},
"policy_refs": {
"expectations": _optional_policy_source_record(
root,
assessment_path,
assessment.get("expectations_ref"),
"expectation-set",
),
"waivers": _optional_policy_source_record(
root,
assessment_path,
assessment.get("waivers_ref"),
"waiver-set",
),
"challenges": _optional_policy_source_record(
root,
assessment_path,
assessment.get("challenges_ref"),
"challenge-set",
),
"exclusions": _optional_policy_source_record(
root,
assessment_path,
assessment.get("exclusions_ref"),
"exclusion-set",
),
},
"authorities": _authority_source_records(extensions),
"metadata_hooks": {
"runner_entrypoints": _entrypoint_metadata_records(extensions),
"normalizers": _normalizer_metadata_records(extensions),
},
}
def _framework_records(
framework_refs: list[str],
extensions: list[Extension],
) -> list[dict[str, Any]]:
records = []
for framework_ref in framework_refs:
declaring_extensions = [
extension.id
for extension in extensions
if framework_ref in _manifest_framework_ids(extension.manifest)
]
records.append(
{
"id": framework_ref,
"version": _version_hint(framework_ref),
"declared_by_extensions": declaring_extensions,
}
)
return records
def _extension_source_record(root: Path, extension: Extension) -> dict[str, Any]:
manifest_path = extension.path / "extension.json"
return {
"id": extension.id,
"version": extension.manifest["version"],
"path": _extension_path_ref(root, extension.path),
"source": extension.source,
"manifest_path": _display_path(root, manifest_path),
"manifest_checksum": _checksum_if_file(manifest_path),
"supported_frameworks": _manifest_framework_ids(extension.manifest),
"authorities": _authority_ids(extension.manifest.get("authorities", [])),
"certification_boundary": extension.manifest["certification_boundary"],
"metadata": _object_or_empty(extension.manifest.get("metadata")),
}
def _mapping_source_records(root: Path, extensions: list[Extension]) -> list[dict[str, Any]]:
records = []
for extension in extensions:
for mapping_id in extension.manifest.get("mappings", []):
if not isinstance(mapping_id, str):
continue
mapping_path = extension.path / "mappings" / f"{mapping_id}.json"
record = {
"id": mapping_id,
"extension_id": extension.id,
"path": _display_path(root, mapping_path),
"exists": mapping_path.is_file(),
"checksum": _checksum_if_file(mapping_path),
"framework_refs": [],
}
if mapping_path.is_file():
mapping_set = load_json(mapping_path)
record["framework_refs"] = _string_list(mapping_set.get("framework_refs", []))
records.append(record)
return records
def _file_source_record(
kind: str,
profile_id: str,
path: Path,
snapshot_ref: str,
) -> dict[str, Any]:
return {
"kind": kind,
"id": profile_id,
"path": str(path.resolve()),
"checksum": _checksum_if_file(path),
"snapshot_ref": snapshot_ref,
}
def _optional_policy_source_record(
root: Path,
assessment_path: Path,
ref: Any,
kind: str,
) -> dict[str, Any] | None:
if not isinstance(ref, str) or not ref:
return None
path = _resolve_assessment_ref(root, assessment_path, ref)
return {
"kind": kind,
"ref": ref,
"path": str(path.resolve()),
"exists": path.is_file(),
"checksum": _checksum_if_file(path),
}
def _authority_source_records(extensions: list[Extension]) -> list[dict[str, Any]]:
records = []
for extension in extensions:
for authority in extension.manifest.get("authorities", []):
if isinstance(authority, str):
records.append({"id": authority, "extension_id": extension.id})
elif isinstance(authority, dict):
record = {
"id": authority.get("id"),
"extension_id": extension.id,
}
for key in ("name", "version", "source_url", "license", "access"):
if key in authority:
record[key] = authority[key]
records.append(record)
return [record for record in records if isinstance(record.get("id"), str)]
def _entrypoint_metadata_records(extensions: list[Extension]) -> list[dict[str, Any]]:
records = []
for extension in extensions:
for entrypoint in extension.manifest.get("runner_entrypoints", []):
if not isinstance(entrypoint, dict):
continue
records.append(
{
"extension_id": extension.id,
"id": entrypoint.get("id"),
"kind": entrypoint.get("kind"),
"metadata": _object_or_empty(entrypoint.get("metadata")),
}
)
return records
def _normalizer_metadata_records(extensions: list[Extension]) -> list[dict[str, Any]]:
records = []
for extension in extensions:
for normalizer in extension.manifest.get("normalizers", []):
if not isinstance(normalizer, dict):
continue
records.append(
{
"extension_id": extension.id,
"id": normalizer.get("id"),
"kind": normalizer.get("kind"),
"runner_ref": normalizer.get("runner_ref"),
"metadata": _object_or_empty(normalizer.get("metadata")),
}
)
return records
def _manifest_framework_ids(manifest: dict[str, Any]) -> list[str]:
values = []
for framework in manifest.get("supported_frameworks", []):
if isinstance(framework, str):
values.append(framework)
elif isinstance(framework, dict) and isinstance(framework.get("id"), str):
values.append(framework["id"])
return values
def _authority_ids(authorities: list[Any]) -> list[str]:
values = []
for authority in authorities:
if isinstance(authority, str):
values.append(authority)
elif isinstance(authority, dict) and isinstance(authority.get("id"), str):
values.append(authority["id"])
return values
def _resolve_assessment_ref(root: Path, assessment_path: Path, ref: str) -> Path:
ref_path = Path(ref)
if ref_path.is_absolute():
return ref_path
root_relative = root / ref_path
if root_relative.exists():
return root_relative
return assessment_path.resolve().parent / ref_path
def _checksum_if_file(path: Path) -> str | None:
if not path.is_file():
return None
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return f"sha256:{digest.hexdigest()}"
def _version_hint(ref: str) -> str | None:
for part in reversed(ref.replace("-", ".").split(".")):
if len(part) > 1 and part[0].lower() == "v" and any(char.isdigit() for char in part[1:]):
return part
return None
def _object_or_empty(value: Any) -> dict[str, Any]:
return value if isinstance(value, dict) else {}
def _string_list(value: Any) -> list[str]:
if not isinstance(value, list):
return []
return [item for item in value if isinstance(item, str)]
def _display_path(root: Path, path: Path) -> str:
try:
return str(path.resolve().relative_to(root.resolve()))
except ValueError:
return str(path.resolve())
def _extension_path_ref(root: Path, path: Path) -> str:
try:
return str(path.resolve().relative_to(root.resolve()))
@@ -126,5 +499,9 @@ def _extension_path_ref(root: Path, path: Path) -> str:
return str(path.resolve())
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _timestamp() -> str:
return datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")

View File

@@ -13,20 +13,36 @@ from guide_board.schema import assert_valid
def apply_policy(
root: Path,
plan: dict[str, Any],
evidence: list[dict[str, Any]],
findings: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], dict[str, Any], list[dict[str, Any]]]:
) -> tuple[
list[dict[str, Any]],
dict[str, Any],
list[dict[str, Any]],
list[dict[str, Any]],
list[dict[str, Any]],
]:
expectations = _load_optional_set(root, plan, "expectations_ref", "expectation-set")
waiver_set = _load_optional_set(root, plan, "waivers_ref", "waiver-set")
challenge_set = _load_optional_set(root, plan, "challenges_ref", "challenge-set")
exclusion_set = _load_optional_set(root, plan, "exclusions_ref", "exclusion-set")
waivers = waiver_set.get("waivers", []) if waiver_set else []
challenges = challenge_set.get("challenges", []) if challenge_set else []
exclusions = exclusion_set.get("exclusions", []) if exclusion_set else []
applied_expectations = 0
applied_waivers: list[dict[str, Any]] = []
applied_challenges: list[dict[str, Any]] = []
applied_exclusions: list[dict[str, Any]] = []
evidence_by_id = {item["id"]: item for item in evidence}
for finding in findings:
for expectation in expectations.get("expectations", []) if expectations else []:
if _matches_rule(finding, expectation):
finding["expected"] = expectation["expected"]
finding["policy_ref"] = expectation["id"]
finding["review_status"] = "expected" if expectation["expected"] else "unresolved_defect"
_annotate_evidence(evidence_by_id, finding, "expectation_refs", expectation["id"])
applied_expectations += 1
break
@@ -37,20 +53,60 @@ def apply_policy(
finding["waiver_ref"] = waiver["id"]
finding["expected"] = True
finding["policy_ref"] = waiver["id"]
finding["review_status"] = "waived"
finding["remediation"] = f"Waived: {waiver['reason']}"
applied_waivers.append(waiver)
_annotate_evidence(evidence_by_id, finding, "waiver_refs", waiver["id"])
break
for exclusion in exclusions:
if not _review_record_active(exclusion):
continue
if _matches_rule(finding, exclusion):
finding["exclusion_ref"] = exclusion["id"]
if finding.get("review_status") == "unresolved_defect":
finding["review_status"] = "authority_excluded"
applied_exclusions.append(exclusion)
_annotate_evidence(evidence_by_id, finding, "exclusion_refs", exclusion["id"])
break
for challenge in challenges:
if not _review_record_active(challenge):
continue
if _matches_rule(finding, challenge):
finding["challenge_ref"] = challenge["id"]
if finding.get("review_status") == "unresolved_defect":
finding["review_status"] = "challenged"
applied_challenges.append(challenge)
_annotate_evidence(evidence_by_id, finding, "challenge_refs", challenge["id"])
break
policy_summary = {
"expectations_ref": plan["assessment_profile_snapshot"].get("expectations_ref"),
"waivers_ref": plan["assessment_profile_snapshot"].get("waivers_ref"),
"challenges_ref": plan["assessment_profile_snapshot"].get("challenges_ref"),
"exclusions_ref": plan["assessment_profile_snapshot"].get("exclusions_ref"),
"applied_expectations": applied_expectations,
"applied_waivers": len(applied_waivers),
"challenged_findings": _unique_applied_count(findings, "challenge_ref"),
"authority_exclusions": _unique_applied_count(findings, "exclusion_ref"),
"unexpected_findings": sum(
1 for finding in findings if not finding.get("expected") and not finding.get("waiver_ref")
),
"unresolved_defects": sum(
1 for finding in findings if finding.get("review_status") == "unresolved_defect"
),
"unresolved_review_items": sum(
1 for finding in findings if finding.get("review_status") in {"challenged", "authority_excluded"}
),
}
return findings, policy_summary, applied_waivers
return (
findings,
policy_summary,
_dedupe_records(applied_waivers),
_dedupe_records(applied_challenges),
_dedupe_records(applied_exclusions),
)
def _load_optional_set(
@@ -94,6 +150,7 @@ def _matches_rule(finding: dict[str, Any], rule: dict[str, Any]) -> bool:
return (
_matches_any(finding.get("requirement_refs", []), rule.get("requirement_refs", []))
and _matches_any([finding.get("check_id", "")], rule.get("check_refs", []))
and _matches_any(finding.get("evidence_refs", []), rule.get("evidence_refs", []))
and _matches_scalar(finding.get("status"), rule.get("result_refs", []))
and _matches_scalar(finding.get("classification"), rule.get("classification_refs", []))
)
@@ -122,3 +179,57 @@ def _waiver_active(waiver: dict[str, Any]) -> bool:
except ValueError:
return False
return expiry >= date.today()
def _review_record_active(record: dict[str, Any]) -> bool:
status = record.get("review_status")
if status in {"rejected", "withdrawn", "closed", "expired"}:
return False
expires_at = record.get("expires_at")
if not expires_at:
return True
try:
expiry = date.fromisoformat(expires_at)
except ValueError:
return False
return expiry >= date.today()
def _annotate_evidence(
evidence_by_id: dict[str, dict[str, Any]],
finding: dict[str, Any],
ref_key: str,
ref_value: str,
) -> None:
for evidence_ref in finding.get("evidence_refs", []):
item = evidence_by_id.get(evidence_ref)
if item is None:
continue
review = item.setdefault(
"review",
{
"expectation_refs": [],
"waiver_refs": [],
"challenge_refs": [],
"exclusion_refs": [],
},
)
refs = review.setdefault(ref_key, [])
if ref_value not in refs:
refs.append(ref_value)
def _unique_applied_count(findings: list[dict[str, Any]], ref_name: str) -> int:
return sum(1 for finding in findings if finding.get(ref_name))
def _dedupe_records(records: list[dict[str, Any]]) -> list[dict[str, Any]]:
seen = set()
deduped = []
for record in records:
record_id = record.get("id")
if not isinstance(record_id, str) or record_id in seen:
continue
seen.add(record_id)
deduped.append(record)
return deduped

204
src/guide_board/reports.py Normal file
View File

@@ -0,0 +1,204 @@
"""Report fragment loading for extension-contributed report content."""
from __future__ import annotations
import importlib.util
from pathlib import Path
from types import ModuleType
from typing import Any
from guide_board.errors import ValidationError
from guide_board.io import load_json
def build_report_fragments(
root: Path,
run_dir: Path,
run_id: str,
plan: dict[str, Any],
evidence: list[dict[str, Any]],
findings: list[dict[str, Any]],
mapping_records: list[dict[str, Any]],
assessment_package: dict[str, Any],
) -> list[dict[str, Any]]:
fragments: list[dict[str, Any]] = []
for extension in plan["extension_snapshots"]:
extension_path = _snapshot_path(root, extension)
manifest = load_json(extension_path / "extension.json")
for descriptor in manifest.get("report_fragments", []):
fragment = _load_fragment(
root,
run_dir,
run_id,
plan,
evidence,
findings,
mapping_records,
assessment_package,
extension,
extension_path,
descriptor,
)
if fragment is not None:
fragments.append(fragment)
return fragments
def markdown_for_fragments(fragments: list[dict[str, Any]]) -> str:
markdown_blocks = [
fragment.get("markdown", "")
for fragment in fragments
if isinstance(fragment.get("markdown"), str) and fragment.get("markdown")
]
if not markdown_blocks:
return "- no extension report fragments"
return "\n\n".join(markdown_blocks)
def _load_fragment(
root: Path,
run_dir: Path,
run_id: str,
plan: dict[str, Any],
evidence: list[dict[str, Any]],
findings: list[dict[str, Any]],
mapping_records: list[dict[str, Any]],
assessment_package: dict[str, Any],
extension: dict[str, Any],
extension_path: Path,
descriptor: Any,
) -> dict[str, Any] | None:
if isinstance(descriptor, str):
descriptor = {
"id": descriptor,
"kind": "markdown_file",
"path": f"reports/{descriptor}.md",
"title": descriptor,
}
if not isinstance(descriptor, dict):
return None
if descriptor["kind"] == "markdown_file":
markdown = _load_markdown_fragment(extension_path, descriptor)
return _fragment_record(extension["id"], descriptor, markdown, {})
if descriptor["kind"] == "python_module":
result = _run_python_fragment(
root,
run_dir,
run_id,
plan,
evidence,
findings,
mapping_records,
assessment_package,
extension_path,
descriptor,
)
return _fragment_record(
extension["id"],
descriptor,
result.get("markdown", ""),
_object_or_empty(result.get("structured")),
)
raise ValidationError(f"{descriptor['id']}: unsupported report fragment kind")
def _load_markdown_fragment(extension_path: Path, descriptor: dict[str, Any]) -> str:
raw_path = descriptor.get("path") or f"reports/{descriptor['id']}.md"
fragment_path = _safe_extension_path(extension_path, raw_path, descriptor["id"])
if not fragment_path.is_file():
raise ValidationError(f"{descriptor['id']}: report fragment not found: {raw_path}")
return fragment_path.read_text(encoding="utf-8")
def _run_python_fragment(
root: Path,
run_dir: Path,
run_id: str,
plan: dict[str, Any],
evidence: list[dict[str, Any]],
findings: list[dict[str, Any]],
mapping_records: list[dict[str, Any]],
assessment_package: dict[str, Any],
extension_path: Path,
descriptor: dict[str, Any],
) -> dict[str, Any]:
module_path = descriptor.get("module_path")
callable_name = descriptor.get("callable")
if not module_path or not callable_name:
raise ValidationError(
f"{descriptor['id']}: python_module report fragments need module_path and callable"
)
module_file = _safe_extension_path(extension_path, module_path, descriptor["id"])
module = _load_module(module_file, descriptor["id"])
fragment_callable = getattr(module, callable_name, None)
if not callable(fragment_callable):
raise ValidationError(f"{descriptor['id']}: callable {callable_name!r} was not found")
context = {
"root": str(root),
"run_dir": str(run_dir),
"run_id": run_id,
"plan": plan,
"evidence": evidence,
"findings": findings,
"mappings": mapping_records,
"assessment_package": assessment_package,
"policy_summary": assessment_package.get("policy_summary", {}),
"source_lock": assessment_package.get("source_lock", {}),
"extension_path": str(extension_path),
"report_fragment": descriptor,
}
result = fragment_callable(context)
if not isinstance(result, dict):
raise ValidationError(f"{descriptor['id']}: report fragment must return an object")
return result
def _fragment_record(
extension_id: str,
descriptor: dict[str, Any],
markdown: str,
structured: dict[str, Any],
) -> dict[str, Any]:
return {
"id": descriptor["id"],
"extension_id": extension_id,
"title": descriptor.get("title") or descriptor["id"],
"kind": descriptor["kind"],
"markdown": markdown if isinstance(markdown, str) else "",
"structured": structured,
}
def _safe_extension_path(extension_path: Path, raw_path: str, fragment_id: str) -> Path:
path = (extension_path / raw_path).resolve()
try:
path.relative_to(extension_path.resolve())
except ValueError as exc:
raise ValidationError(
f"{fragment_id}: report fragment path must stay inside the extension directory"
) from exc
return path
def _load_module(path: Path, fragment_id: str) -> ModuleType:
if not path.exists():
raise ValidationError(f"{fragment_id}: module not found: {path}")
module_name = f"_guide_board_report_fragment_{fragment_id.replace('-', '_')}"
spec = importlib.util.spec_from_file_location(module_name, path)
if spec is None or spec.loader is None:
raise ValidationError(f"{fragment_id}: unable to load module from {path}")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path:
path = Path(extension["path"])
return path if path.is_absolute() else root / path
def _object_or_empty(value: Any) -> dict[str, Any]:
return value if isinstance(value, dict) else {}

View File

@@ -37,6 +37,10 @@ def build_retention_summary(
"unexpected_findings": policy_summary.get("unexpected_findings", 0),
"expected_findings": sum(1 for finding in findings if finding.get("expected")),
"waived_findings": sum(1 for finding in findings if finding.get("waiver_ref")),
"challenged_findings": policy_summary.get("challenged_findings", 0),
"authority_exclusions": policy_summary.get("authority_exclusions", 0),
"unresolved_defects": policy_summary.get("unresolved_defects", 0),
"unresolved_review_items": policy_summary.get("unresolved_review_items", 0),
"mapping_target_count": len(
assessment_package.get("mapping_summary", {}).get("targets", [])
),
@@ -45,6 +49,11 @@ def build_retention_summary(
"report_refs": [
"reports/assessment-package.json",
"reports/report.md",
"reports/fragments.json",
"reports/submission-package.json",
],
"export_refs": [
"exports/export-manifest.json",
],
"artifact_retention": {
"policy": plan["assessment_profile_snapshot"].get("retention_policy", {}),
@@ -195,7 +204,12 @@ def _run_projection(run: dict[str, Any]) -> dict[str, Any]:
"status": summary.get("status", "unknown"),
"unexpected_findings": _summary_int(summary, "unexpected_findings"),
"finding_count": _summary_int(summary, "finding_count"),
"mapping_target_count": _summary_int(summary, "mapping_target_count"),
"artifact_count": _summary_int(summary, "artifact_count"),
"challenged_findings": _summary_int(summary, "challenged_findings"),
"authority_exclusions": _summary_int(summary, "authority_exclusions"),
"unresolved_defects": _summary_int(summary, "unresolved_defects"),
"unresolved_review_items": _summary_int(summary, "unresolved_review_items"),
"run_dir": run.get("run_dir"),
}
@@ -208,10 +222,17 @@ def _trend_between(
return {
"direction": "insufficient-history",
"status_changed": False,
"status_change": {
"from": None,
"to": _status_for(latest),
},
"unexpected_findings_delta": 0,
"finding_count_delta": 0,
"artifact_count_delta": 0,
"unresolved_review_items_delta": 0,
"mapping_target_count_delta": 0,
"evidence_result_deltas": {},
"summary_text": "No previous retained run is available for comparison.",
}
previous_summary = previous.get("summary", {})
@@ -229,16 +250,37 @@ def _trend_between(
artifact_delta = _summary_int(latest_summary, "artifact_count") - _summary_int(
previous_summary, "artifact_count"
)
review_delta = _summary_int(latest_summary, "unresolved_review_items") - _summary_int(
previous_summary, "unresolved_review_items"
)
mapping_target_delta = _summary_int(latest_summary, "mapping_target_count") - _summary_int(
previous_summary, "mapping_target_count"
)
previous_status = _status_for(previous)
latest_status = _status_for(latest)
direction = _trend_direction(previous_status, latest_status, unexpected_delta)
return {
"direction": _trend_direction(previous_status, latest_status, unexpected_delta),
"direction": direction,
"status_changed": previous_status != latest_status,
"status_change": {
"from": previous_status,
"to": latest_status,
},
"unexpected_findings_delta": unexpected_delta,
"finding_count_delta": finding_delta,
"artifact_count_delta": artifact_delta,
"unresolved_review_items_delta": review_delta,
"mapping_target_count_delta": mapping_target_delta,
"evidence_result_deltas": evidence_deltas,
"summary_text": _trend_summary_text(
direction,
previous_status,
latest_status,
unexpected_delta,
review_delta,
mapping_target_delta,
),
}
@@ -280,6 +322,24 @@ def _summary_int(summary: dict[str, Any], key: str) -> int:
return value if isinstance(value, int) and not isinstance(value, bool) else 0
def _trend_summary_text(
direction: str,
previous_status: str,
latest_status: str,
unexpected_delta: int,
review_delta: int,
mapping_target_delta: int,
) -> str:
parts = [
f"Trend {direction}",
f"status {previous_status} -> {latest_status}",
f"unexpected findings delta {unexpected_delta}",
f"unresolved review delta {review_delta}",
f"mapping target delta {mapping_target_delta}",
]
return "; ".join(parts) + "."
def _dict_deltas(previous: Any, latest: Any) -> dict[str, int]:
previous_dict = previous if isinstance(previous, dict) else {}
latest_dict = latest if isinstance(latest, dict) else {}

View File

@@ -45,6 +45,8 @@ def run_step(
"runner_kind": "external",
},
"artifact_refs": [],
"requirement_refs": [],
"metadata": _object_or_empty(entrypoint.get("metadata")),
}
if entrypoint["kind"] == "command":
return _run_command(root, run_dir, run_id, plan, step, extension_path, entrypoint)
@@ -63,6 +65,8 @@ def _no_runner_result(step: dict[str, Any]) -> dict[str, Any]:
"runner_kind": None,
},
"artifact_refs": [],
"requirement_refs": [],
"metadata": {},
}
@@ -118,6 +122,8 @@ def _run_python_module(
"error_type": type(exc).__name__,
},
"artifact_refs": [],
"requirement_refs": [],
"metadata": _object_or_empty(entrypoint.get("metadata")),
}
if not isinstance(result, dict):
raise ValidationError(f"{entrypoint['id']}: runner must return an object")
@@ -126,6 +132,8 @@ def _run_python_module(
"observations": result.get("observations", []),
"facts": result.get("facts", {}),
"artifact_refs": result.get("artifact_refs", []),
"requirement_refs": result.get("requirement_refs", []),
"metadata": _merge_metadata(entrypoint.get("metadata"), result.get("metadata")),
}
@@ -192,6 +200,8 @@ def _run_command(
"command": command,
},
"artifact_refs": [str(context_path.relative_to(run_dir))],
"requirement_refs": [],
"metadata": _object_or_empty(entrypoint.get("metadata")),
}
except subprocess.TimeoutExpired:
return {
@@ -206,6 +216,8 @@ def _run_command(
"command": command,
},
"artifact_refs": [str(context_path.relative_to(run_dir))],
"requirement_refs": [],
"metadata": _object_or_empty(entrypoint.get("metadata")),
}
parsed = _parse_runner_stdout(completed.stdout)
@@ -225,6 +237,8 @@ def _run_command(
"command": command,
},
"artifact_refs": [str(context_path.relative_to(run_dir))],
"requirement_refs": [],
"metadata": _object_or_empty(entrypoint.get("metadata")),
}
facts = parsed.get("facts", {})
@@ -245,6 +259,9 @@ def _run_command(
if not isinstance(artifact_refs, list):
artifact_refs = []
artifact_refs.append(str(context_path.relative_to(run_dir)))
requirement_refs = parsed.get("requirement_refs", [])
if not isinstance(requirement_refs, list):
requirement_refs = []
result = parsed.get("result", "unknown")
if completed.returncode != 0 and result in {"pass", "warning", "manual", "skipped"}:
@@ -258,6 +275,8 @@ def _run_command(
"observations": observations,
"facts": facts,
"artifact_refs": artifact_refs,
"requirement_refs": requirement_refs,
"metadata": _merge_metadata(entrypoint.get("metadata"), parsed.get("metadata")),
}
@@ -328,5 +347,17 @@ def _parse_runner_stdout(stdout: str) -> dict[str, Any] | None:
return parsed
def _merge_metadata(*values: Any) -> dict[str, Any]:
merged: dict[str, Any] = {}
for value in values:
if isinstance(value, dict):
merged.update(value)
return merged
def _object_or_empty(value: Any) -> dict[str, Any]:
return value if isinstance(value, dict) else {}
def _safe_id(value: str) -> str:
return "".join(char if char.isalnum() or char in {"-", "_"} else "_" for char in value)

View File

@@ -10,7 +10,7 @@ from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
from urllib.parse import parse_qs, unquote, urlparse
from guide_board.discovery import discover_extensions
from guide_board.errors import GuideBoardError
@@ -21,6 +21,11 @@ from guide_board.planning import (
validate_assessment_profile,
validate_target_profile,
)
from guide_board.retention import (
list_retained_runs,
retained_run_report_paths,
select_retained_run,
)
@dataclass(frozen=True)
@@ -131,7 +136,7 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
def _handle(self, method: str) -> None:
parsed = urlparse(self.path)
try:
response, status_code = self._route(method, parsed.path)
response, status_code = self._route(method, parsed.path, parsed.query)
except HttpProblem as exc:
response = _error_response(exc.message, exc.__class__.__name__, exc.status_code)
status_code = exc.status_code
@@ -147,7 +152,8 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
self._send_json(status_code, response)
def _route(self, method: str, path: str) -> tuple[dict[str, Any], int]:
def _route(self, method: str, path: str, query: str = "") -> tuple[dict[str, Any], int]:
query_params = _query_params(query)
if method == "GET" and path == "/health":
return self._health(), 200
if method == "GET" and path == "/extensions":
@@ -160,6 +166,10 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
return {"runs": self.server.context.jobs.list()}, 200
if method == "POST" and path == "/runs":
return self._start_run(), 202
if method == "GET" and path == "/retained-runs":
return self._retained_runs(query_params), 200
if method == "GET" and path == "/retained-runs/latest":
return self._retained_latest(query_params), 200
run_match = _match_run_path(path)
if method == "GET" and run_match is not None:
@@ -169,6 +179,14 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
if suffix == "reports":
return self._run_reports(job_id), 200
retained_match = _match_retained_run_path(path)
if method == "GET" and retained_match is not None:
run_id, suffix = retained_match
if suffix == "reports":
return self._retained_run_reports(run_id, query_params), 200
if suffix == "artifact-manifest":
return self._retained_artifact_manifest(run_id, query_params), 200
raise HttpProblem(404, f"endpoint not found: {method} {path}")
def _health(self) -> dict[str, Any]:
@@ -261,10 +279,32 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
report_path = Path(result["report"])
package_path = Path(result["assessment_package"])
retention_path = Path(result["retention_summary"])
submission_value = result.get("submission_package")
submission_path = (
Path(submission_value)
if isinstance(submission_value, str) and submission_value
else None
)
export_value = result.get("export_manifest")
export_path = (
Path(export_value)
if isinstance(export_value, str) and export_value
else None
)
try:
report_markdown = report_path.read_text(encoding="utf-8")
assessment_package = load_json(package_path)
retention_summary = load_json(retention_path)
submission_package = (
load_json(submission_path)
if submission_path is not None and submission_path.is_file()
else None
)
export_manifest = (
load_json(export_path)
if export_path is not None and export_path.is_file()
else None
)
except OSError as exc:
raise HttpProblem(404, f"run report artifact is missing: {exc}") from exc
@@ -277,6 +317,8 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
"report": str(report_path),
"assessment_package": str(package_path),
"retention_summary": str(retention_path),
"submission_package": str(submission_path) if submission_path is not None else None,
"export_manifest": str(export_path) if export_path is not None else None,
},
"report": {
"path": str(report_path),
@@ -290,6 +332,73 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
"path": str(retention_path),
"json": retention_summary,
},
"submission_package": {
"path": str(submission_path) if submission_package else None,
"json": submission_package,
},
"export_manifest": {
"path": str(export_path) if export_manifest else None,
"json": export_manifest,
},
}
def _retained_runs(self, query: dict[str, str]) -> dict[str, Any]:
runs_dir = _runs_dir_from_query(self.server.context.root, query)
return {
"runs_dir": str(runs_dir),
"runs": list_retained_runs(runs_dir),
}
def _retained_latest(self, query: dict[str, str]) -> dict[str, Any]:
runs_dir = _runs_dir_from_query(self.server.context.root, query)
run = select_retained_run(
runs_dir,
target_profile_ref=query.get("target"),
assessment_profile_ref=query.get("assessment"),
)
return {
"runs_dir": str(runs_dir),
"selection": {
"target_profile_ref": query.get("target"),
"assessment_profile_ref": query.get("assessment"),
},
"run": _retained_run_with_paths(run) if run else None,
}
def _retained_run_reports(self, run_id: str, query: dict[str, str]) -> dict[str, Any]:
runs_dir = _runs_dir_from_query(self.server.context.root, query)
run = _select_retained_run_or_404(runs_dir, run_id)
return {
"runs_dir": str(runs_dir),
"run": _retained_run_with_paths(run),
}
def _retained_artifact_manifest(self, run_id: str, query: dict[str, str]) -> dict[str, Any]:
runs_dir = _runs_dir_from_query(self.server.context.root, query)
run = _select_retained_run_or_404(runs_dir, run_id)
run_dir = _safe_run_dir(runs_dir, run)
package_path = run_dir / "reports" / "assessment-package.json"
if not package_path.exists():
return {
"runs_dir": str(runs_dir),
"run_id": run_id,
"run_dir": str(run_dir),
"artifact_manifest": [],
"compatibility": "assessment-package-missing",
}
package = load_json(package_path)
artifacts = package.get("artifact_manifest", [])
if not isinstance(artifacts, list):
raise HttpProblem(400, f"{package_path}: artifact_manifest must be a list")
for artifact in artifacts:
if isinstance(artifact, dict):
_safe_run_ref(run_dir, artifact.get("path"))
return {
"runs_dir": str(runs_dir),
"run_id": run_id,
"run_dir": str(run_dir),
"artifact_manifest": artifacts,
"compatibility": "current",
}
def _read_payload(self) -> dict[str, Any]:
@@ -430,6 +539,81 @@ def _match_run_path(path: str) -> tuple[str, str | None] | None:
return None
def _match_retained_run_path(path: str) -> tuple[str, str] | None:
parts = [unquote(part) for part in path.split("/") if part]
if len(parts) == 3 and parts[0] == "retained-runs":
return parts[1], parts[2]
return None
def _query_params(query: str) -> dict[str, str]:
parsed = parse_qs(query, keep_blank_values=False)
params = {}
for key, values in parsed.items():
if values:
params[key] = values[-1]
return params
def _runs_dir_from_query(root: Path, query: dict[str, str]) -> Path:
runs_dir = query.get("runs_dir")
if not runs_dir:
return (root / "runs").resolve()
return _resolve_path(root, runs_dir)
def _select_retained_run_or_404(runs_dir: Path, run_id: str) -> dict[str, Any]:
run = select_retained_run(runs_dir, run_id=run_id)
if run is None:
raise HttpProblem(404, f"retained run not found: {run_id}")
return run
def _retained_run_with_paths(run: dict[str, Any] | None) -> dict[str, Any] | None:
if run is None:
return None
paths = retained_run_report_paths(run)
run_dir = Path(run["run_dir"]).resolve()
safe_paths = {}
for key, value in paths.items():
path = Path(value).resolve()
try:
path.relative_to(run_dir)
except ValueError as exc:
raise HttpProblem(
400,
f"retained run report path escapes run directory: {value}",
) from exc
safe_paths[key] = str(path)
return {
**run,
"paths": dict(sorted(safe_paths.items())),
}
def _safe_run_dir(runs_dir: Path, run: dict[str, Any]) -> Path:
run_dir_value = run.get("run_dir")
if not isinstance(run_dir_value, str) or not run_dir_value:
raise HttpProblem(400, "retained run is missing run_dir")
run_dir = Path(run_dir_value).resolve()
try:
run_dir.relative_to(runs_dir.resolve())
except ValueError as exc:
raise HttpProblem(400, f"retained run escapes runs_dir: {run_dir}") from exc
return run_dir
def _safe_run_ref(run_dir: Path, ref: Any) -> Path:
if not isinstance(ref, str) or not ref:
raise HttpProblem(400, "artifact manifest entry path must be a non-empty string")
path = (run_dir / ref).resolve()
try:
path.relative_to(run_dir.resolve())
except ValueError as exc:
raise HttpProblem(400, f"artifact path escapes run directory: {ref}") from exc
return path
def _display_path(root: Path, path: Path) -> str:
try:
return str(path.resolve().relative_to(root.resolve()))

View File

@@ -0,0 +1,17 @@
{
"top_level_keys": [
"certification_boundary",
"counts",
"created_at",
"export_type",
"id",
"mapping_summary",
"policy_summary",
"report_fragments",
"run_id",
"schema_version",
"source_lock_ref",
"source_package_ref",
"summary"
]
}

View File

@@ -0,0 +1,5 @@
### SDK Fixture Summary
- evidence items: 2
- findings: 0
- source lock: source-lock:sdk-fixture-assessment:sdk-fixture-target

View File

@@ -2,12 +2,15 @@ from __future__ import annotations
import http.client
import json
import shutil
import time
import unittest
from tempfile import TemporaryDirectory
from pathlib import Path
from urllib.parse import quote
from guide_board.discovery import discover_extensions
from guide_board.errors import ValidationError
from guide_board.execution import run_assessment
from guide_board.gates import evaluate_trend_gates
from guide_board.io import load_json
@@ -73,6 +76,17 @@ class CoreArchitectureTests(unittest.TestCase):
plan["ordered_steps"][1]["requirement_refs"],
["guide-board.sample-readiness.v0.profile-shape"],
)
assert_valid(plan["source_lock"], "source-lock")
self.assertEqual(plan["source_lock"]["schema_version"], "guide-board.source-lock.v1")
self.assertEqual(plan["source_lock"]["framework_refs"], ["guide-board.sample-readiness.v0"])
self.assertEqual(plan["source_lock"]["extension_refs"], ["sample-noop"])
self.assertEqual(
plan["source_lock"]["profiles"]["target"]["snapshot_ref"],
"target-profile.snapshot.json",
)
self.assertTrue(plan["source_lock"]["profiles"]["target"]["checksum"].startswith("sha256:"))
self.assertEqual(plan["source_lock"]["mapping_sets"][0]["id"], "sample-readiness-map")
self.assertTrue(plan["source_lock"]["mapping_sets"][0]["checksum"].startswith("sha256:"))
def test_runs_external_extension_from_separate_repo(self) -> None:
with TemporaryDirectory() as temporary_directory:
@@ -143,6 +157,149 @@ class CoreArchitectureTests(unittest.TestCase):
self.assertEqual(plan["extension_snapshots"][0]["path"], str(extension_dir))
self.assertEqual([item["result"] for item in evidence], ["skipped", "manual"])
def test_applies_external_extension_profile_schemas(self) -> None:
with TemporaryDirectory() as temporary_directory:
temp_root = Path(temporary_directory)
extension_dir = temp_root / "schema-noop"
_write_schema_extension(extension_dir)
extensions = discover_extensions(ROOT, [extension_dir])
target_path = temp_root / "target.json"
assessment_path = temp_root / "assessment.json"
_write_schema_target(target_path, endpoints=[{
"id": "api",
"url": "http://127.0.0.1:8080",
"binding": "example",
}])
_write_schema_assessment(assessment_path, runtime_policy={"offline": True})
target = validate_target_profile(target_path, extensions)
assessment = validate_assessment_profile(assessment_path, extensions)
plan = build_run_plan(ROOT, target_path, assessment_path, [extension_dir])
self.assertEqual(target["subject_type"], "schema-subject")
self.assertEqual(assessment["runtime_policy"], {"offline": True})
self.assertEqual(plan["extension_snapshots"][0]["id"], "schema-noop")
_write_schema_target(target_path, endpoints=[])
with self.assertRaisesRegex(
ValidationError,
"schema-noop:schema-target profile schema validation failed",
):
validate_target_profile(target_path, extensions)
def test_rejects_extension_profile_schema_paths_outside_extension_root(self) -> None:
with TemporaryDirectory() as temporary_directory:
temp_root = Path(temporary_directory)
extension_dir = temp_root / "schema-noop"
_write_schema_extension(extension_dir, target_schema_path="../outside.schema.json")
target_path = temp_root / "target.json"
_write_schema_target(target_path, endpoints=[{
"id": "api",
"url": "http://127.0.0.1:8080",
"binding": "example",
}])
extensions = discover_extensions(ROOT, [extension_dir])
with self.assertRaisesRegex(
ValidationError,
"profile schema path escapes extension root",
):
validate_target_profile(target_path, extensions)
def test_runs_sdk_fixture_from_external_extension_repo(self) -> None:
with TemporaryDirectory() as temporary_directory:
temp_root = Path(temporary_directory)
extension_dir = temp_root / "sdk-fixture"
shutil.copytree(ROOT / "extensions" / "sdk-fixture", extension_dir)
result = run_assessment(
temp_root,
extension_dir / "profiles" / "targets" / "sdk-fixture-target.json",
extension_dir / "profiles" / "assessments" / "sdk-fixture-assessment.json",
temp_root / "runs" / "sdk-fixture",
[extension_dir],
)
run_dir = Path(result["run_dir"])
plan = json.loads((run_dir / "plan.json").read_text(encoding="utf-8"))
evidence = json.loads(
(run_dir / "normalized" / "evidence.json").read_text(encoding="utf-8")
)["evidence"]
mappings = json.loads(
(run_dir / "normalized" / "mappings.json").read_text(encoding="utf-8")
)["mappings"]
assessment_package = json.loads(
(run_dir / "reports" / "assessment-package.json").read_text(encoding="utf-8")
)
report = (run_dir / "reports" / "report.md").read_text(encoding="utf-8")
export_manifest = json.loads(
(run_dir / "exports" / "export-manifest.json").read_text(encoding="utf-8")
)
self.assertEqual(result["status"], "completed")
self.assertEqual(plan["extension_snapshots"][0]["source"], "external")
self.assertEqual(plan["target_profile_snapshot"]["subject_type"], "sdk-fixture-target")
self.assertEqual([item["result"] for item in evidence], ["skipped", "pass"])
check_evidence = evidence[1]
self.assertEqual(
check_evidence["facts"]["normalizer_refs"],
["native-probe-normalizer"],
)
self.assertEqual(check_evidence["facts"]["native_score"], 98)
self.assertEqual(
check_evidence["requirement_refs"],
["guide-board.sdk-fixture.v1.native-output"],
)
self.assertEqual(
check_evidence["artifact_refs"],
["artifacts/sdk-fixture/native-result.json"],
)
self.assertEqual(
check_evidence["facts"]["source_metadata"]["runner"]["metadata"]["harness_version"],
"1.0.0",
)
self.assertEqual(
check_evidence["facts"]["source_metadata"]["reported"]["native_result_id"],
"sdk-fixture-native-result",
)
self.assertEqual(mappings[0]["target_id"], "normalizer-plugin")
self.assertEqual(assessment_package["summary"], {"pass": 1, "skipped": 1})
self.assertEqual(
assessment_package["report_fragments"][0]["markdown"],
(ROOT / "tests" / "golden" / "sdk-fixture-report-fragment.md")
.read_text(encoding="utf-8")
.rstrip(),
)
self.assertIn("### SDK Fixture Summary", report)
assert_valid(export_manifest, "export-manifest")
export_shape = load_json(ROOT / "tests" / "golden" / "export-manifest-shape.json")
self.assertEqual(sorted(export_manifest), export_shape["top_level_keys"])
self.assertEqual(export_manifest["counts"]["report_fragments"], 1)
self.assertEqual(
export_manifest["report_fragments"][0]["structured"]["evidence_count"],
2,
)
self.assertEqual(
assessment_package["source_lock"]["metadata_hooks"]["runner_entrypoints"][0][
"metadata"
]["harness_id"],
"sdk-fixture-native-probe",
)
submission_package = json.loads(
(run_dir / "reports" / "submission-package.json").read_text(encoding="utf-8")
)
assert_valid(submission_package, "submission-package")
self.assertEqual(submission_package["source_lock"]["id"], "source-lock:sdk-fixture-assessment:sdk-fixture-target")
self.assertEqual(
submission_package["reported_metadata"][1]["metadata"]["reported"][
"native_result_id"
],
"sdk-fixture-native-result",
)
self.assertEqual(
submission_package["artifact_manifest"][0]["checksum"],
assessment_package["artifact_manifest"][0]["checksum"],
)
def test_runs_sample_noop_assessment(self) -> None:
with TemporaryDirectory() as temporary_directory:
result = run_assessment(
@@ -159,6 +316,9 @@ class CoreArchitectureTests(unittest.TestCase):
self.assertTrue((run_dir / "normalized" / "evidence.json").exists())
self.assertTrue((run_dir / "reports" / "assessment-package.json").exists())
self.assertTrue((run_dir / "reports" / "report.md").exists())
self.assertTrue((run_dir / "reports" / "fragments.json").exists())
self.assertTrue((run_dir / "reports" / "submission-package.json").exists())
self.assertTrue((run_dir / "exports" / "export-manifest.json").exists())
retention = json.loads(
(run_dir / "retention-summary.json").read_text(encoding="utf-8")
)
@@ -166,12 +326,31 @@ class CoreArchitectureTests(unittest.TestCase):
result["retention_summary"],
str(run_dir / "retention-summary.json"),
)
self.assertEqual(
result["submission_package"],
str(run_dir / "reports" / "submission-package.json"),
)
self.assertEqual(
result["export_manifest"],
str(run_dir / "exports" / "export-manifest.json"),
)
self.assertEqual(retention["summary"]["status"], "completed")
self.assertEqual(retention["summary"]["artifact_count"], 0)
self.assertIn("reports/submission-package.json", retention["report_refs"])
self.assertIn("exports/export-manifest.json", retention["export_refs"])
self.assertEqual(
retention["artifact_retention"]["policy"],
{"raw_artifact_days": 0, "summary_days": 365},
)
submission = json.loads(
(run_dir / "reports" / "submission-package.json").read_text(encoding="utf-8")
)
assert_valid(submission, "submission-package")
self.assertEqual(submission["package_identity"]["target_profile_ref"], "sample-repository")
self.assertEqual(
[entry["path"] for entry in submission["reports"]],
["reports/assessment-package.json", "reports/report.md"],
)
self.assertEqual(
[run["run_id"] for run in list_retained_runs(Path(temporary_directory))],
[result["run_id"]],
@@ -182,6 +361,69 @@ class CoreArchitectureTests(unittest.TestCase):
self.assertEqual(len(mappings), 1)
self.assertEqual(mappings[0]["target_id"], "profile-readiness")
def test_applies_challenges_and_exclusions_without_hiding_gate_failures(self) -> None:
with TemporaryDirectory() as temporary_directory:
temp_root = Path(temporary_directory)
extension_dir = temp_root / "review-noop"
_write_review_extension(extension_dir)
target_path = temp_root / "review-target.json"
assessment_path = temp_root / "review-assessment.json"
challenge_path = temp_root / "review-challenges.json"
exclusion_path = temp_root / "review-exclusions.json"
_write_review_target(target_path)
_write_review_assessment(assessment_path)
_write_review_challenges(challenge_path)
_write_review_exclusions(exclusion_path)
result = run_assessment(
ROOT,
target_path,
assessment_path,
temp_root / "runs" / "review",
[extension_dir],
)
run_dir = Path(result["run_dir"])
evidence = json.loads(
(run_dir / "normalized" / "evidence.json").read_text(encoding="utf-8")
)["evidence"]
assessment_package = json.loads(
(run_dir / "reports" / "assessment-package.json").read_text(encoding="utf-8")
)
retention = json.loads(
(run_dir / "retention-summary.json").read_text(encoding="utf-8")
)
report = (run_dir / "reports" / "report.md").read_text(encoding="utf-8")
self.assertEqual(result["status"], "blocked")
finding = assessment_package["findings"][0]
self.assertEqual(finding["challenge_ref"], "challenge-review-blocked")
self.assertEqual(finding["exclusion_ref"], "exclusion-review-blocked")
self.assertEqual(finding["review_status"], "authority_excluded")
self.assertFalse(finding["expected"])
self.assertEqual(assessment_package["policy_summary"]["unexpected_findings"], 1)
self.assertEqual(assessment_package["policy_summary"]["challenged_findings"], 1)
self.assertEqual(assessment_package["policy_summary"]["authority_exclusions"], 1)
self.assertEqual(assessment_package["policy_summary"]["unresolved_defects"], 0)
self.assertEqual(
evidence[1]["review"]["challenge_refs"],
["challenge-review-blocked"],
)
self.assertEqual(
evidence[1]["review"]["exclusion_refs"],
["exclusion-review-blocked"],
)
self.assertEqual(assessment_package["challenges"][0]["owner"], "qa")
self.assertEqual(assessment_package["exclusions"][0]["authority_ref"], "review-authority")
self.assertEqual(retention["summary"]["challenged_findings"], 1)
self.assertEqual(retention["summary"]["authority_exclusions"], 1)
self.assertEqual(retention["summary"]["unresolved_review_items"], 1)
self.assertIn("- authority_excluded: 1", report)
gate = evaluate_trend_gates(build_trend_summary(temp_root / "runs"))
self.assertEqual(gate["status"], "failed")
checks = {check["id"]: check for check in gate["groups"][0]["checks"]}
self.assertEqual(checks["unexpected-findings"]["observed"], 1)
def test_serves_local_api_run_lifecycle(self) -> None:
with TemporaryDirectory() as temporary_directory:
service = start_service(ROOT, host="127.0.0.1", port=0)
@@ -243,6 +485,68 @@ class CoreArchitectureTests(unittest.TestCase):
reports["assessment_package"]["json"]["run_id"],
status["result"]["run_id"],
)
self.assertEqual(
reports["submission_package"]["json"]["run_id"],
status["result"]["run_id"],
)
self.assertEqual(
reports["export_manifest"]["json"]["run_id"],
status["result"]["run_id"],
)
finally:
service.stop()
def test_service_exposes_retained_runs_after_restart(self) -> None:
with TemporaryDirectory() as temporary_directory:
runs_dir = Path(temporary_directory) / "runs"
result = run_assessment(
ROOT,
ROOT / "profiles" / "targets" / "sample-repository.json",
ROOT / "profiles" / "assessments" / "sample-noop.json",
runs_dir / "sample",
)
_write_unsafe_artifact_run(runs_dir / "unsafe-run")
service = start_service(ROOT, host="127.0.0.1", port=0)
try:
query = f"runs_dir={quote(str(runs_dir), safe='')}"
listing = _request_json(service, "GET", f"/retained-runs?{query}")
self.assertEqual(listing["runs_dir"], str(runs_dir))
self.assertIn(result["run_id"], [run["run_id"] for run in listing["runs"]])
latest = _request_json(
service,
"GET",
f"/retained-runs/latest?{query}&target=sample-repository&assessment=sample-noop-assessment",
)
self.assertEqual(latest["run"]["run_id"], result["run_id"])
self.assertIn("submission_package", latest["run"]["paths"])
reports = _request_json(
service,
"GET",
f"/retained-runs/{result['run_id']}/reports?{query}",
)
self.assertEqual(
reports["run"]["paths"]["assessment_package"],
str(runs_dir / "sample" / "reports" / "assessment-package.json"),
)
artifacts = _request_json(
service,
"GET",
f"/retained-runs/{result['run_id']}/artifact-manifest?{query}",
)
self.assertEqual(artifacts["artifact_manifest"], [])
self.assertEqual(artifacts["compatibility"], "current")
unsafe = _request_json(
service,
"GET",
f"/retained-runs/unsafe-run/artifact-manifest?{query}",
expected_status=400,
)
self.assertIn("escapes run directory", unsafe["error"]["message"])
finally:
service.stop()
@@ -278,7 +582,13 @@ class CoreArchitectureTests(unittest.TestCase):
self.assertEqual(group["previous_run"]["run_id"], "run-old")
self.assertEqual(group["trend"]["direction"], "improved")
self.assertTrue(group["trend"]["status_changed"])
self.assertEqual(
group["trend"]["status_change"],
{"from": "blocked", "to": "completed"},
)
self.assertEqual(group["trend"]["unexpected_findings_delta"], -1)
self.assertEqual(group["trend"]["mapping_target_count_delta"], 0)
self.assertIn("Trend improved", group["trend"]["summary_text"])
self.assertEqual(
group["trend"]["evidence_result_deltas"],
{"blocked": -1, "manual": 1, "skipped": 1},
@@ -388,6 +698,34 @@ def _write_retention_summary(
)
def _write_unsafe_artifact_run(run_dir: Path) -> None:
_write_retention_summary(
run_dir,
"unsafe-run",
"2026-05-07T12:00:00+00:00",
"completed",
{"pass": 1},
0,
1,
)
reports_dir = run_dir / "reports"
reports_dir.mkdir(parents=True, exist_ok=True)
(reports_dir / "assessment-package.json").write_text(
json.dumps(
{
"artifact_manifest": [
{
"id": "artifact:unsafe",
"path": "../outside.txt",
"checksum": "sha256:unsafe",
}
]
}
),
encoding="utf-8",
)
def _request_json(
service: ServiceHandle,
method: str,
@@ -460,5 +798,296 @@ def _write_external_extension(extension_dir: Path) -> None:
)
def _write_schema_extension(
extension_dir: Path,
target_schema_path: str = "schemas/schema-target.schema.json",
) -> None:
extension_dir.mkdir(parents=True, exist_ok=True)
schema_dir = extension_dir / "schemas"
schema_dir.mkdir()
(schema_dir / "schema-target.schema.json").write_text(
json.dumps(
{
"type": "object",
"required": ["subject_type", "endpoints"],
"properties": {
"subject_type": {"enum": ["schema-subject"]},
"endpoints": {"type": "array", "minItems": 1},
},
}
),
encoding="utf-8",
)
(schema_dir / "schema-assessment.schema.json").write_text(
json.dumps(
{
"type": "object",
"required": ["runtime_policy"],
"properties": {
"runtime_policy": {
"type": "object",
"required": ["offline"],
"properties": {"offline": {"type": "boolean"}},
}
},
}
),
encoding="utf-8",
)
(extension_dir / "extension.json").write_text(
json.dumps(
{
"id": "schema-noop",
"name": "Schema No-op",
"version": "0.1.0",
"extension_type": "repository_quality",
"lifecycle_status": "incubating",
"supported_frameworks": ["schema.readiness.v1"],
"authorities": [],
"profile_schemas": [
"target-profile",
"assessment-profile",
{
"id": "schema-target",
"profile_kind": "target",
"path": target_schema_path,
"subject_type": "schema-subject",
},
{
"id": "schema-assessment",
"profile_kind": "assessment",
"path": "schemas/schema-assessment.schema.json",
},
],
"check_groups": [
{
"id": "shape",
"name": "Shape",
"check_type": "repository_quality",
"requirement_refs": ["schema.shape"],
"runner_ref": None,
}
],
"preflight_runner": None,
"runner_entrypoints": [],
"normalizers": [],
"mappings": [],
"report_fragments": [],
"dependencies": [],
"restricted_assets": [],
"certification_boundary": "Test fixture only.",
}
),
encoding="utf-8",
)
def _write_schema_target(path: Path, endpoints: list[dict[str, str]]) -> None:
path.write_text(
json.dumps(
{
"id": "schema-target",
"subject_type": "schema-subject",
"subject_name": "Schema Target",
"environment": "test",
"scope": ["schema"],
"endpoints": endpoints,
"artifacts": [],
"credentials_ref": None,
"declared_capabilities": [],
"known_gaps": [],
}
),
encoding="utf-8",
)
def _write_schema_assessment(path: Path, runtime_policy: dict[str, object]) -> None:
path.write_text(
json.dumps(
{
"id": "schema-assessment",
"framework_refs": ["schema.readiness.v1"],
"extension_refs": ["schema-noop"],
"target_profile_ref": "schema-target",
"selected_check_groups": {"schema-noop": ["shape"]},
"expectations_ref": None,
"waivers_ref": None,
"output_policy": {
"report_formats": ["json", "markdown"],
"artifact_retention": "summary-only",
},
"retention_policy": {
"summary_days": 365,
"raw_artifact_days": 0,
},
"runtime_policy": runtime_policy,
}
),
encoding="utf-8",
)
def _write_review_extension(extension_dir: Path) -> None:
extension_dir.mkdir(parents=True, exist_ok=True)
(extension_dir / "extension.json").write_text(
json.dumps(
{
"id": "review-noop",
"name": "Review No-op",
"version": "0.1.0",
"extension_type": "repository_quality",
"lifecycle_status": "incubating",
"supported_frameworks": ["review.framework.v1"],
"authorities": ["review-authority"],
"profile_schemas": ["target-profile", "assessment-profile"],
"check_groups": [
{
"id": "review",
"name": "Review",
"check_type": "repository_quality",
"requirement_refs": ["review.requirement"],
"runner_ref": "external-review",
}
],
"preflight_runner": None,
"runner_entrypoints": [
{
"id": "external-review",
"kind": "external",
"module_path": None,
"callable": None,
"command": None,
"metadata": {"test_suite_id": "review-suite"},
"description": "External runner used to produce reviewable blocked evidence.",
}
],
"normalizers": [],
"mappings": [],
"report_fragments": [],
"dependencies": [],
"restricted_assets": [],
"certification_boundary": "Review fixture only.",
}
),
encoding="utf-8",
)
def _write_review_target(path: Path) -> None:
path.write_text(
json.dumps(
{
"id": "review-target",
"subject_type": "repository",
"subject_name": "Review Target",
"environment": "test",
"scope": ["review"],
"endpoints": [],
"artifacts": [],
"credentials_ref": None,
"declared_capabilities": [],
"known_gaps": [],
}
),
encoding="utf-8",
)
def _write_review_assessment(path: Path) -> None:
path.write_text(
json.dumps(
{
"id": "review-assessment",
"framework_refs": ["review.framework.v1"],
"extension_refs": ["review-noop"],
"target_profile_ref": "review-target",
"selected_check_groups": {"review-noop": ["review"]},
"expectations_ref": None,
"waivers_ref": None,
"challenges_ref": "review-challenges.json",
"exclusions_ref": "review-exclusions.json",
"output_policy": {
"report_formats": ["json", "markdown"],
"artifact_retention": "summary-only",
},
"retention_policy": {
"summary_days": 365,
"raw_artifact_days": 0,
},
"runtime_policy": {
"offline": True,
"timeout_seconds": 2,
},
}
),
encoding="utf-8",
)
def _write_review_challenges(path: Path) -> None:
path.write_text(
json.dumps(
{
"id": "review-challenges",
"target_profile_ref": "review-target",
"challenges": [
{
"id": "challenge-review-blocked",
"requirement_refs": ["review.requirement"],
"check_refs": ["check-group:review-noop:review"],
"evidence_refs": [],
"result_refs": ["blocked"],
"classification_refs": ["runner_not_implemented"],
"authority_source_refs": ["review-authority:rule-1"],
"owner": "qa",
"review_status": "open",
"rationale": "The external suite is not wired in this fixture.",
"created_at": "2026-05-16",
"review_due_at": "2026-06-16",
"expires_at": None,
"native_challenge_id": "native-challenge-1",
"metadata": {"kind": "fixture"},
}
],
}
),
encoding="utf-8",
)
def _write_review_exclusions(path: Path) -> None:
path.write_text(
json.dumps(
{
"id": "review-exclusions",
"target_profile_ref": "review-target",
"exclusions": [
{
"id": "exclusion-review-blocked",
"authority_ref": "review-authority",
"requirement_refs": ["review.requirement"],
"check_refs": ["check-group:review-noop:review"],
"evidence_refs": [],
"result_refs": ["blocked"],
"classification_refs": ["runner_not_implemented"],
"authority_source_refs": ["review-authority:rule-1"],
"owner": "qa",
"approved_by": "authority-reviewer",
"review_status": "approved",
"rationale": "Fixture demonstrates authority exclusion annotation.",
"created_at": "2026-05-16",
"review_due_at": "2026-06-16",
"expires_at": None,
"native_exclusion_id": "native-exclusion-1",
"metadata": {"kind": "fixture"},
}
],
}
),
encoding="utf-8",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -3,7 +3,8 @@ id: GUIDE-BOARD-WP-0001
type: workplan
title: "Guide Board Bootstrapping"
repo: guide-board
domain: markitect
domain: communication
topic_slug: markitect
status: completed
owner: codex
planning_priority: high

View File

@@ -3,8 +3,9 @@ id: GUIDE-BOARD-WP-0002
type: workplan
title: "Assessment Operations Baseline"
repo: guide-board
domain: markitect
status: active
domain: communication
topic_slug: markitect
status: completed
owner: codex
planning_priority: high
planning_order: 2
@@ -174,7 +175,7 @@ Progress:
```task
id: GUIDE-BOARD-WP-0002-T006
status: todo
status: done
priority: high
state_hub_task_id: "65fbf1df-caef-40f6-abee-8308daf27fbc"
```
@@ -187,6 +188,13 @@ Acceptance:
- Cover extension validation, target/assessment profile validation, plan
generation, run execution, and result review.
Progress:
- Added `docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
- Added `scripts/external_extension_acceptance.sh`.
- Verified `open-cmis-tck` extension validation, profile validation, and plan
generation in scripted plan mode.
## Definition Of Done
- The repo has a clear assessment operations guide.

View File

@@ -0,0 +1,151 @@
---
id: GUIDE-BOARD-WP-0003
type: workplan
title: "Extension SDK Maturity"
repo: guide-board
domain: communication
topic_slug: markitect
status: completed
owner: codex
planning_priority: high
planning_order: 3
created: "2026-05-15"
updated: "2026-05-15"
state_hub_workstream_id: "26aa9511-cd5c-4dd5-989c-d2838ba3b50d"
---
# GUIDE-BOARD-WP-0003: Extension SDK Maturity
## Purpose
Harden the external extension SDK now that guide-board has a repeatable
assessment operations baseline. External extension repositories should be able
to declare their own validation surfaces, normalization boundaries, and
acceptance fixtures without pushing domain logic back into the core.
## Background
`GUIDE-BOARD-WP-0002` made assessment operation repeatable for CLI, service,
container, candidate handoff, retained results, and external extension
acceptance. The next repo-level gap is SDK maturity: the core can discover
external extensions and run their entrypoints, but extension-owned validation
and normalizer contracts are still mostly prose.
## Boundary
This workplan owns extension-neutral SDK contracts and core enforcement points.
Domain-specific schemas, CMIS runner behavior, harness dependencies, and
certification interpretations remain owned by external extension repositories.
## D3.1 - Extension-Owned Profile Schema Validation
```task
id: GUIDE-BOARD-WP-0003-T001
status: done
priority: high
state_hub_task_id: "1bc729ec-683c-410e-8b47-1b13eb61da00"
```
Acceptance:
- Allow extension manifests to declare profile schema descriptors without
breaking the existing string shorthand.
- Validate extension-owned target and assessment profile schemas during CLI
profile validation and run planning.
- Keep extension schemas loaded from the extension root and reject schema paths
that escape that root.
- Add focused tests and SDK documentation.
Progress:
- Extended `profile_schemas` to support descriptor objects while preserving the
existing string shorthand.
- Applied extension-owned target and assessment schema validation in CLI profile
validation and run planning.
- Added tests for successful extension-owned validation, validation failure, and
schema-path root containment.
- Documented the descriptor contract in `docs/EXTENSION-SDK.md`.
## D3.2 - Normalizer Plug-in Contract
```task
id: GUIDE-BOARD-WP-0003-T002
status: done
priority: high
state_hub_task_id: "b87e68c1-6eca-4274-8e3f-6e2854c5a1e1"
```
Acceptance:
- Define how extension normalizers are declared, loaded, and invoked.
- Preserve the current runner-result contract while allowing an extension to
normalize native result artifacts explicitly.
- Add tests that prove a normalizer can map native output into evidence.
Progress:
- Added `guide_board.normalizers.normalize_step_result`.
- Extended `normalizers` manifest entries to support Python module descriptor
objects while preserving the existing string shorthand.
- Invoked matching normalizers after runner execution and before evidence
writing.
- Merged normalizer result fields over runner results and recorded
`normalizer_refs` in evidence facts.
- Added test coverage through the SDK fixture run.
## D3.3 - SDK Fixture Extension And Acceptance Tests
```task
id: GUIDE-BOARD-WP-0003-T003
status: done
priority: medium
state_hub_task_id: "f3738751-5a0d-4eaf-85b1-75e599a78060"
```
Acceptance:
- Add a compact SDK fixture extension that exercises the mature contracts.
- Keep the fixture dependency-light and suitable for unit tests.
- Cover external repo discovery, schema validation, normalizer invocation, plan
generation, and result package shape.
Progress:
- Added `extensions/sdk-fixture`.
- Included extension-owned target and assessment schemas, fixture profiles, a
native-output runner, a normalizer, and a mapping set.
- Added a unit test that copies the fixture as an external extension repository
and verifies plan, evidence, mapping, and assessment package output.
## D3.4 - Extension Authoring Documentation Refresh
```task
id: GUIDE-BOARD-WP-0003-T004
status: done
priority: medium
state_hub_task_id: "3d390bd4-755b-462a-9e16-9c859990d99e"
```
Acceptance:
- Refresh `docs/EXTENSION-SDK.md` with the finalized profile-schema and
normalizer contracts.
- Update templates or examples so extension authors can copy working shapes.
- Link the SDK maturity guidance from the assessment operations and external
extension acceptance docs where useful.
Progress:
- Refreshed `docs/EXTENSION-SDK.md` with profile schema descriptors,
normalizer descriptors, context fields, merge semantics, and fixture guidance.
- Linked SDK authoring contracts from `docs/ASSESSMENT-OPERATIONS.md`.
- Linked `extensions/sdk-fixture` from `docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
- Added README references for the SDK fixture and WP3.
## Definition Of Done
- External extension repositories can declare and test domain-specific profile
validation without core code changes.
- Normalizer plug-ins have a documented and tested core contract.
- The SDK includes a small fixture path that future extension work can reuse.
- Operator docs and authoring docs agree on the supported extension lifecycle.

View File

@@ -0,0 +1,159 @@
---
id: GUIDE-BOARD-WP-0004
type: workplan
title: "Source Lock And Submission Package Baseline"
repo: guide-board
domain: communication
topic_slug: markitect
status: completed
owner: codex
planning_priority: high
planning_order: 4
created: "2026-05-15"
updated: "2026-05-16"
state_hub_workstream_id: "6dd2832b-d1d9-43bc-ad5c-d16f399930dc"
---
# GUIDE-BOARD-WP-0004: Source Lock And Submission Package Baseline
## Purpose
Make guide-board assessment packages source-complete enough for serious review.
Runs already snapshot target and assessment profiles and preserve normalized
evidence. The next maturity layer is a stronger source lock and a submission
package manifest that records which framework, extension, harness, mapping,
profile, waiver, and artifact sources produced the assessment.
## Background
The architecture blueprint calls out source locking as part of credible
assessment evidence. A reviewer should be able to distinguish a normal local run
from a package that is ready to hand to another team, auditor, authority, or
certification-preparation process. This does not turn guide-board into a
certification body; it makes the evidence boundary clearer and more portable.
## Boundary
This workplan owns extension-neutral source lock fields and package manifest
generation. Extension-specific harness version detection, authority-specific
submission rules, and licensed or restricted assets remain extension-owned.
## D4.1 - Source Lock Schema And Capture
```task
id: GUIDE-BOARD-WP-0004-T001
status: done
priority: high
state_hub_task_id: "d5a7a18f-941b-47b8-9992-2cb54bc5ad06"
```
Acceptance:
- Extend the source lock contract beyond framework and extension IDs.
- Capture stable references for framework versions, extension versions, mapping
sets, target profile snapshots, assessment profile snapshots, expectation
sets, waiver sets, and authority source URLs when available.
- Keep the schema backward-compatible with existing retained runs.
- Add tests for source lock shape and retained run compatibility.
Progress:
- Added `docs/schemas/source-lock.schema.json`.
- Expanded run-plan source locks with framework, extension, mapping-set,
profile snapshot, policy-ref, authority, and metadata-hook records.
- Preserved the original `framework_refs` and `extension_refs` fields for
retained-run compatibility.
- Added tests for source-lock shape and older retained summary compatibility.
## D4.2 - Harness And Extension Metadata Hooks
```task
id: GUIDE-BOARD-WP-0004-T002
status: done
priority: high
state_hub_task_id: "7abd5a66-5784-41b9-a361-6572290923cc"
```
Acceptance:
- Define how extensions and runner or normalizer results can report harness
versions, test suite IDs, adapter versions, source URLs, and native result
identifiers.
- Persist this metadata in run plans, evidence facts, source locks, or package
manifests without inventing domain-specific fields in the core.
- Preserve current runner and normalizer contracts for extensions that do not
provide this metadata yet.
- Cover the SDK fixture and at least one no-metadata extension path in tests.
Progress:
- Added optional manifest metadata for extensions, authorities, frameworks,
runner entrypoints, and normalizers.
- Preserved runner and normalizer returned `metadata` and requirement refs.
- Recorded merged metadata under evidence `facts.source_metadata`.
- Updated `sdk-fixture` to exercise harness, test-suite, adapter, source URL,
and native result metadata while keeping `sample-noop` as a no-metadata path.
## D4.3 - Submission Package Manifest
```task
id: GUIDE-BOARD-WP-0004-T003
status: done
priority: medium
state_hub_task_id: "c54273d6-1fc2-4444-92cf-74f2a5e614ec"
```
Acceptance:
- Add a machine-readable submission package manifest under each run report
directory.
- Include package identity, source lock references, report paths, normalized
evidence paths, artifact manifest entries, checksums where available, and the
certification boundary.
- Keep the manifest useful for both executable harnesses and procedural evidence
packs.
- Document how this differs from an authority-specific final submission.
Progress:
- Added `docs/schemas/submission-package.schema.json`.
- Wrote `reports/submission-package.json` for each run.
- Included package identity, source lock checksum, report paths, normalized
output paths, profile snapshots, artifact manifest entries, reported
metadata, and the certification boundary.
- Exposed the submission manifest path in CLI/service run results and retained
report refs.
## D4.4 - Documentation And Acceptance Tests
```task
id: GUIDE-BOARD-WP-0004-T004
status: done
priority: medium
state_hub_task_id: "ad37baeb-973c-4399-96d0-c9cb7fc6b761"
```
Acceptance:
- Update assessment operations, extension SDK, and architecture docs with the
source lock and submission package contracts.
- Add tests that run a sample or SDK fixture assessment and assert the source
lock and manifest outputs.
- Include compatibility notes for older retained runs.
- Keep the output paths aligned with existing CLI and service result retrieval.
Progress:
- Updated assessment operations, extension SDK, architecture blueprint, and
README references.
- Added focused unit assertions for the sample and SDK fixture assessment
outputs.
- Kept retained run listing compatible with older `retention-summary.json`
files that do not reference a submission manifest.
## Definition Of Done
- Every new run writes a richer source lock and submission package manifest.
- Extension-provided harness metadata has a stable path into the package.
- Older retained runs remain readable.
- Operators and extension authors know what the package can and cannot claim.

View File

@@ -0,0 +1,156 @@
---
id: GUIDE-BOARD-WP-0005
type: workplan
title: "Challenge And Exclusion Handling"
repo: guide-board
domain: communication
topic_slug: markitect
status: completed
owner: codex
planning_priority: high
planning_order: 5
created: "2026-05-15"
updated: "2026-05-16"
state_hub_workstream_id: "fb11e1c7-6c0c-4ec7-a163-da98b2fe9f8f"
---
# GUIDE-BOARD-WP-0005: Challenge And Exclusion Handling
## Purpose
Represent authority exclusions, extension challenges, target expectations,
waivers, and defects as distinct review concepts. Guide-board already supports
expectations and waivers, but real assessments also need a way to record that a
test was excluded by an authority, challenged as invalid or mis-mapped, or
identified as a target defect.
## Background
Conformance work often includes dispute and exclusion paths. A failed check may
be a product issue, a harness issue, an unsupported optional feature, a known
waived risk, an authority-approved exclusion, or a local challenge awaiting
review. Reports need to make those distinctions visible instead of flattening
everything into pass, fail, blocked, or waived.
## Boundary
This workplan owns guide-board's generic challenge and exclusion model. It does
not decide authority-specific challenge rules, certification program policy, or
whether a challenge is valid. Extensions may provide authority-specific fields,
but the core should preserve them without embedding domain policy.
## D5.1 - Challenge And Exclusion Data Contracts
```task
id: GUIDE-BOARD-WP-0005-T001
status: done
priority: high
state_hub_task_id: "6ff4e6f7-bce6-4e7f-a5af-e0c67cfa7e55"
```
Acceptance:
- Define schemas for authority exclusions and extension challenges.
- Distinguish challenge/exclusion records from expectations and waivers.
- Support links to requirement refs, check IDs, evidence IDs, authority source
refs, owners, review status, rationale, expiry or review dates, and native
challenge IDs where available.
- Keep the data contract usable by executable harnesses, hosted suites, and
procedural packs.
Progress:
- Added `docs/schemas/challenge-set.schema.json` and
`docs/schemas/exclusion-set.schema.json`.
- Added optional `challenges_ref` and `exclusions_ref` assessment profile
fields.
- Supported requirement, check, evidence, result, classification, authority
source, owner, review status, rationale, review date, expiry, native ID, and
metadata fields.
## D5.2 - Policy Application And Finding Annotation
```task
id: GUIDE-BOARD-WP-0005-T002
status: done
priority: high
state_hub_task_id: "fd384bd3-40c4-4344-8b7d-cb123dbf2cac"
```
Acceptance:
- Load challenge and exclusion references from assessment profiles.
- Annotate findings and evidence with challenge or exclusion state without
silently hiding unexpected failures.
- Preserve separate counts for expected findings, waived findings, challenged
findings, authority exclusions, and unresolved defects.
- Add tests that prove challenge and exclusion records affect reporting without
corrupting gate semantics.
Progress:
- Loaded challenge and exclusion refs through the policy layer.
- Annotated findings with challenge refs, exclusion refs, and review status.
- Annotated matching evidence with review refs.
- Kept default `unexpected_findings` gate semantics visible unless a finding is
separately expected or waived.
- Added tests proving challenged and excluded findings remain gate-visible.
## D5.3 - Report Visibility And Review Workflow
```task
id: GUIDE-BOARD-WP-0005-T003
status: done
priority: medium
state_hub_task_id: "791071c0-8a9a-462b-83b3-75548bb8524f"
```
Acceptance:
- Show challenge, exclusion, waiver, expectation, and defect distinctions in
Markdown and JSON assessment packages.
- Make unresolved review items easy to find in retained run summaries.
- Provide CLI history or report helpers that expose review state for the latest
run.
- Document how an operator should treat challenged or excluded findings.
Progress:
- Added Markdown report review summaries.
- Added challenge, exclusion, unresolved defect, and unresolved review counts to
retention summaries and trend projections.
- Included applied challenge and exclusion records in JSON assessment packages.
- Exposed review counts through existing retained run helpers.
## D5.4 - Tests And Documentation
```task
id: GUIDE-BOARD-WP-0005-T004
status: done
priority: medium
state_hub_task_id: "43b966da-af8d-479b-93bd-6b6741fdab37"
```
Acceptance:
- Add schema tests and policy tests for challenges and exclusions.
- Add a sample or SDK fixture scenario that produces at least one challenged or
excluded finding.
- Update assessment operations, extension SDK, and compliance evidence pack docs.
- Keep certification boundary language explicit.
Progress:
- Added focused schema and policy tests through a fixture extension scenario.
- Updated assessment operations, extension SDK, compliance evidence pack, and
architecture docs.
- Kept boundary language explicit: challenges and exclusions are review state,
not certification conclusions.
## Definition Of Done
- The core has separate, tested concepts for expectations, waivers, challenges,
authority exclusions, and defects.
- Reports surface those concepts without overstating certification conclusions.
- Retained summaries expose unresolved review work.
- Extension authors know how to supply authority-specific challenge metadata.

View File

@@ -0,0 +1,151 @@
---
id: GUIDE-BOARD-WP-0006
type: workplan
title: "Service Artifact Access And Durable Run Index"
repo: guide-board
domain: communication
topic_slug: markitect
status: completed
owner: codex
planning_priority: medium
planning_order: 6
created: "2026-05-15"
updated: "2026-05-16"
state_hub_workstream_id: "ba008283-1631-467b-868e-1052c3870ab9"
---
# GUIDE-BOARD-WP-0006: Service Artifact Access And Durable Run Index
## Purpose
Move the local service beyond in-memory job visibility while preserving the CLI
as the execution source of truth. Operators and future UI clients should be able
to discover retained runs, retrieve reports, inspect artifacts, and recover
after service restarts without needing private knowledge of the run directory.
## Background
WP2 intentionally kept service job state in memory and documented run
directories as the durable source. That is a good baseline, but the service
already wraps run start, status, and report retrieval. The next step is to expose
the durable run history and artifact paths directly through service contracts.
## Boundary
This workplan owns local service API and durable run index behavior. It should
not change CLI run semantics, introduce a database dependency, or create a
distributed execution service. Any durable index must be reconstructable from
existing run artifacts.
## D6.1 - Durable Run Index Design
```task
id: GUIDE-BOARD-WP-0006-T001
status: done
priority: high
state_hub_task_id: "4d392fc5-6a1c-46f7-9cbf-6c02bbd744c6"
```
Acceptance:
- Decide whether the service needs a separate durable index file or can rely on
retained run summaries with helper scans.
- Define reconstruction behavior after service restart.
- Preserve compatibility with existing `retention-summary.json` and run
directory layout.
- Document the operational tradeoff and failure modes.
Decision:
- Keep the durable index as retained run summaries and helper scans.
- Do not add a separate service index file for the baseline.
Progress:
- Documented reconstruction from `retention-summary.json` files.
- Kept compatibility with older runs that lack newer assessment package or
submission manifest files.
## D6.2 - Service Run History And Artifact Endpoints
```task
id: GUIDE-BOARD-WP-0006-T002
status: done
priority: high
state_hub_task_id: "8f209920-6b14-4d6f-bfa1-8f1d03bcdbf1"
```
Acceptance:
- Add service endpoints for retained run listing, latest run selection, report
path lookup, and artifact manifest access.
- Keep endpoint responses aligned with existing CLI `runs` commands.
- Avoid serving arbitrary filesystem paths outside configured run directories.
- Add tests for successful retrieval and path-safety failures.
Progress:
- Added `GET /retained-runs`.
- Added `GET /retained-runs/latest`.
- Added `GET /retained-runs/{run_id}/reports`.
- Added `GET /retained-runs/{run_id}/artifact-manifest`.
- Added path containment checks for report refs and artifact manifest paths.
- Added service tests for retained history retrieval after a fresh service
process and unsafe artifact path rejection.
## D6.3 - Restart Recovery And Compatibility
```task
id: GUIDE-BOARD-WP-0006-T003
status: done
priority: medium
state_hub_task_id: "0857e7d8-3d23-4426-b7fa-73362d7041a0"
```
Acceptance:
- Prove that a service restart can still expose retained run reports and
artifacts.
- Keep in-memory job status semantics clear for currently running jobs.
- Add compatibility handling for older run directories that lack newer manifest
files.
- Update service durability documentation with examples.
Progress:
- Preserved `/runs` as in-memory job history.
- Exposed durable run results through retained-run endpoints after restart.
- Returned a compatibility marker when an older retained run lacks an
assessment package artifact manifest.
- Updated service durability and local API docs.
## D6.4 - Container And Service Acceptance Tests
```task
id: GUIDE-BOARD-WP-0006-T004
status: done
priority: medium
state_hub_task_id: "900a70fa-65ff-4815-9c0c-31f0da4019f0"
```
Acceptance:
- Add focused service tests for durable run lookup and artifact/report retrieval.
- Extend container or scripted acceptance to prove mounted run directories remain
readable through service contracts.
- Document service endpoint usage in local and container modes.
- Keep tests dependency-light.
Progress:
- Added dependency-light service tests for durable run lookup, report paths, and
artifact manifest retrieval.
- Updated container smoke artifact expectations for current run outputs.
- Documented retained-run endpoint usage in local and container modes.
## Definition Of Done
- The local service can expose retained runs and artifacts after restart.
- Endpoint behavior matches CLI run history semantics.
- Filesystem access is constrained to intended run outputs.
- Operators have documented recovery and artifact retrieval paths.

View File

@@ -0,0 +1,150 @@
---
id: GUIDE-BOARD-WP-0007
type: workplan
title: "Report And Export Maturity"
repo: guide-board
domain: communication
topic_slug: markitect
status: completed
owner: codex
planning_priority: medium
planning_order: 7
created: "2026-05-15"
updated: "2026-05-16"
state_hub_workstream_id: "ef9351d2-e99c-470e-aeec-f17aa51eae14"
---
# GUIDE-BOARD-WP-0007: Report And Export Maturity
## Purpose
Improve guide-board's human and machine-readable reporting so assessment output
is easier to review, compare, and hand off. The current JSON assessment package
and Markdown report are useful baselines. The next layer should support
extension report fragments, portable export formats, and stronger trend and gate
summaries.
## Background
Real assessment consumers need different report surfaces. Engineers need concise
run feedback, product teams need readiness summaries, compliance reviewers need
traceable evidence and boundaries, and external systems may need portable JSON
or later OSCAL-style interchange. This workplan keeps those surfaces derived
from the same evidence package.
## Boundary
This workplan owns generic report and export mechanics. It does not implement
authority-specific final submission formats unless they can be represented as
extension-provided fragments or exporters.
## D7.1 - Report Fragment Plug-in Contract
```task
id: GUIDE-BOARD-WP-0007-T001
status: done
priority: high
state_hub_task_id: "bf3fe163-b06d-4c2e-9b45-31721864e1f2"
```
Acceptance:
- Define how extensions declare Markdown or structured report fragments.
- Load fragments safely from extension roots and include them in run reports
without allowing arbitrary file access.
- Give fragments access to the assessment package, evidence, mappings, policy
summary, and source lock data.
- Add a fixture fragment and tests.
Progress:
- Added Markdown file and Python module report fragment descriptors.
- Loaded fragment paths safely from extension roots.
- Added fragment context for assessment package, evidence, findings, mappings,
policy summary, and source lock data.
- Added an SDK fixture Python report fragment and focused tests.
## D7.2 - Portable Export Formats
```task
id: GUIDE-BOARD-WP-0007-T002
status: done
priority: high
state_hub_task_id: "fda51e62-98aa-408e-a057-4db40fe7c644"
```
Acceptance:
- Add one or more portable export outputs derived from the assessment package.
- Start with a stable JSON export manifest before considering OSCAL or other
external interchange formats.
- Preserve certification boundary and source lock references in each export.
- Document which exports are generic and which must remain extension-owned.
Progress:
- Added `docs/schemas/export-manifest.schema.json`.
- Wrote `exports/export-manifest.json` for each run.
- Included source package refs, source lock refs, summaries, policy summaries,
mapping summaries, report fragments, counts, and certification boundary.
- Documented the generic export boundary; authority-specific formats remain
extension-owned.
## D7.3 - Trend And Gate Report Improvements
```task
id: GUIDE-BOARD-WP-0007-T003
status: done
priority: medium
state_hub_task_id: "33c3089a-9d5e-4605-89c4-a1e070bc12ad"
```
Acceptance:
- Improve retained trend and gate outputs for human review.
- Surface status changes, unexpected finding deltas, mapping target changes, and
unresolved review items.
- Keep machine-readable gate summaries stable for automation.
- Add CLI report helpers or Markdown summaries where useful.
Progress:
- Added trend status-change details, unresolved review deltas, mapping target
deltas, and compact summary text.
- Kept existing gate summary shape stable while improving trend inputs for
human review.
- Added assertions for the richer trend output.
## D7.4 - Golden Fixtures And Documentation
```task
id: GUIDE-BOARD-WP-0007-T004
status: done
priority: medium
state_hub_task_id: "66669f68-6728-4484-9ec9-267ffe025027"
```
Acceptance:
- Add golden fixture outputs for reports and exports.
- Document report fragment and export authoring in the extension SDK.
- Update assessment operations with report/export retrieval examples.
- Ensure report text remains clear about preparation evidence versus formal
certification.
Progress:
- Added golden fixture expectations for report fragment text and export manifest
top-level shape.
- Updated extension SDK, assessment operations, architecture, service, container,
and README docs.
- Kept report and export boundary language tied to preparation evidence, not
formal certification.
## Definition Of Done
- Extensions can contribute report fragments through a documented contract.
- The core emits portable export artifacts derived from the assessment package.
- Trend and gate summaries are more useful to humans without breaking
automation.
- Golden fixtures guard report and export shape.