generated from coulomb/repo-seed
Compare commits
17 Commits
ab7914890e
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| ea092120ea | |||
| 2b2f0c959a | |||
| b1dc0cbe2e | |||
| c2810e00e4 | |||
| ac0ee92cc1 | |||
| d2cbe61962 | |||
| 65c3cba8c7 | |||
| 6c467dd1f4 | |||
| 2a1a53c140 | |||
| 2412f30975 | |||
| b1dff0440d | |||
| c8ac42154c | |||
| d73a73b455 | |||
| 4b3f65572f | |||
| 6758b3992c | |||
| 67f2fc5346 | |||
| fdeea87d4a |
20
.claude/rules/agents.md
Normal file
20
.claude/rules/agents.md
Normal file
@@ -0,0 +1,20 @@
|
||||
## Kaizen Agents
|
||||
|
||||
Specialized agent personas available on demand via the state-hub MCP.
|
||||
|
||||
**Discover:** `list_kaizen_agents()` — returns all agents with name, description, category
|
||||
**Load:** `get_kaizen_agent("tdd-workflow")` — returns full instructions; read and follow them
|
||||
|
||||
Common agents:
|
||||
|
||||
| Agent | Category | When to use |
|
||||
|-------|----------|-------------|
|
||||
| `tdd-workflow` | testing | Step-by-step TDD8 workflow for any feature |
|
||||
| `code-refactoring` | quality | Code quality analysis and safe refactoring |
|
||||
| `test-maintenance` | testing | Diagnose and fix failing tests |
|
||||
| `requirements-engineering` | process | Prevent interface/mock mismatches upfront |
|
||||
| `keepaTodofile` | process | Maintain TODO.md during work |
|
||||
| `project-management` | process | Track status, determine next steps |
|
||||
| `datamodel-optimization` | quality | Optimize dataclasses and data structures |
|
||||
|
||||
All 17 agents: call `list_kaizen_agents()` for the full list.
|
||||
8
.claude/rules/architecture.md
Normal file
8
.claude/rules/architecture.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Architecture
|
||||
|
||||
<!-- TODO: Describe the key design decisions and component structure.
|
||||
Key modules, data flows, external integrations, state machines, etc. -->
|
||||
|
||||
## Quick Reference
|
||||
|
||||
`~/state-hub/mcp_server/TOOLS.md` — MCP tool reference
|
||||
50
.claude/rules/credential-routing.md
Normal file
50
.claude/rules/credential-routing.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=guide-board` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
38
.claude/rules/first-session.md
Normal file
38
.claude/rules/first-session.md
Normal file
@@ -0,0 +1,38 @@
|
||||
## First Session Protocol
|
||||
|
||||
Triggered when `get_domain_summary("communication")` shows **no workstreams**.
|
||||
The project is registered but work has not yet been structured.
|
||||
|
||||
**Step 1 — Read, don't write**
|
||||
- `~/the-custodian/canon/projects/communication/project_charter_v0.1.md` — purpose, scope
|
||||
- `~/the-custodian/canon/projects/communication/roadmap_v0.1.md` — planned phases
|
||||
- Scan repo root: README, directory structure, existing code or docs
|
||||
|
||||
**Step 2 — Survey in-progress work**
|
||||
Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete.
|
||||
|
||||
**Step 3 — Propose workstreams to Bernd**
|
||||
Propose 1–3 workstreams — each a coherent strand, weeks to months, anchored to a
|
||||
roadmap phase. **Wait for approval before creating.**
|
||||
|
||||
**Step 4 — Create workplan file first, then DB record (ADR-001)**
|
||||
```
|
||||
workplans/GUIDE-BOARD-WP-NNNN-<slug>.md ← write this first
|
||||
```
|
||||
Then register in the hub:
|
||||
```
|
||||
create_workstream(topic_id="5571d954-0d30-4950-980d-7bcaaad8e3e2", title="...", owner="...", description="...")
|
||||
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
|
||||
```
|
||||
|
||||
**Step 5 — Record the setup**
|
||||
```
|
||||
add_progress_event(
|
||||
summary="First session: structured communication into N workstreams, M tasks",
|
||||
event_type="milestone",
|
||||
topic_id="5571d954-0d30-4950-980d-7bcaaad8e3e2",
|
||||
detail={"workstreams": [...], "tasks_created": M}
|
||||
)
|
||||
```
|
||||
|
||||
<!-- Delete or archive this file once past first session -->
|
||||
8
.claude/rules/repo-boundary.md
Normal file
8
.claude/rules/repo-boundary.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Repo boundary
|
||||
|
||||
This repo owns **guide-board** only. It does not own:
|
||||
|
||||
<!-- TODO: List what belongs in adjacent repos, e.g.:
|
||||
- SSH key management → railiance-infra/
|
||||
- State hub code → state-hub/
|
||||
-->
|
||||
5
.claude/rules/repo-identity.md
Normal file
5
.claude/rules/repo-identity.md
Normal file
@@ -0,0 +1,5 @@
|
||||
**Purpose:** Certification and compliance preparation framework core with external extension support.
|
||||
|
||||
**Domain:** communication
|
||||
**Repo slug:** guide-board
|
||||
**Topic ID:** 5571d954-0d30-4950-980d-7bcaaad8e3e2
|
||||
85
.claude/rules/session-protocol.md
Normal file
85
.claude/rules/session-protocol.md
Normal file
@@ -0,0 +1,85 @@
|
||||
## Session Protocol
|
||||
|
||||
Dev Hub (State Hub API): http://127.0.0.1:8000
|
||||
MCP server name in `~/.claude.json`: `dev-hub`
|
||||
|
||||
**Step 1 — Orient**
|
||||
|
||||
Read the offline-safe brief first — it works without a live hub connection:
|
||||
```bash
|
||||
cat .custodian-brief.md
|
||||
```
|
||||
Then call the MCP tool for richer cross-domain context when MCP tools are exposed:
|
||||
```
|
||||
get_domain_summary("communication")
|
||||
```
|
||||
If MCP tools are unavailable in the current agent session, use the REST API:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
|
||||
```
|
||||
If the hub is offline: `cd ~/state-hub && make api`
|
||||
|
||||
**Step 2 — Check inbox**
|
||||
With MCP tools:
|
||||
```
|
||||
get_messages(to_agent="guide-board", unread_only=True)
|
||||
```
|
||||
Mark read with `mark_message_read(message_id)`. Reply or act on coordination
|
||||
requests before proceeding.
|
||||
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=guide-board&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
**Step 3 — Scan workplans**
|
||||
```bash
|
||||
ls workplans/
|
||||
```
|
||||
For each file with `status: ready`, `active`, or `blocked`, note pending
|
||||
`wait`/`todo`/`progress` tasks.
|
||||
|
||||
**Step 4 — Present brief**
|
||||
|
||||
1. **Active workstreams** for `communication` — title, task counts, blocking decisions
|
||||
2. **Pending tasks** from `workplans/` + any `[repo:guide-board]` hub tasks
|
||||
3. **Goal guidance** — if `goal_guidance` in summary:
|
||||
- `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"*
|
||||
- `alignment_warnings`: flag if active work is not aligned with current goal
|
||||
4. **Suggested next action** — highest-priority open item
|
||||
5. **SBOM status** — flag if `last_sbom_at` is unset for this repo
|
||||
|
||||
If no workstreams: follow First Session Protocol (`first-session.md`).
|
||||
|
||||
**During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()`
|
||||
|
||||
> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`)
|
||||
> are First Session Protocol only. Work structure belongs in repo files (ADR-001).
|
||||
|
||||
**Session close:**
|
||||
With MCP tools:
|
||||
```
|
||||
add_progress_event(summary="...", topic_id="5571d954-0d30-4950-980d-7bcaaad8e3e2", workstream_id="<uuid>")
|
||||
```
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"topic_id":"5571d954-0d30-4950-980d-7bcaaad8e3e2","workstream_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
|
||||
```
|
||||
If workplan files were modified, ensure the local copy is up to date first:
|
||||
```bash
|
||||
git -C <repo_path> pull --ff-only
|
||||
cd ~/state-hub && make fix-consistency REPO=guide-board
|
||||
```
|
||||
For repos where implementation runs on a remote machine (e.g. CoulombCore),
|
||||
use the combined target which pulls before fixing:
|
||||
```bash
|
||||
cd ~/state-hub && make fix-consistency-remote REPO=guide-board
|
||||
```
|
||||
**C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback
|
||||
will sync the file to match DB. **C-16** (repo behind remote) blocks all writes
|
||||
until you pull — intentional to prevent clobbering remote progress.
|
||||
19
.claude/rules/stack-and-commands.md
Normal file
19
.claude/rules/stack-and-commands.md
Normal file
@@ -0,0 +1,19 @@
|
||||
## Stack
|
||||
|
||||
<!-- TODO: Fill in language, frameworks, and key dependencies -->
|
||||
- **Language:**
|
||||
- **Key deps:**
|
||||
|
||||
## Dev Commands
|
||||
|
||||
```bash
|
||||
# TODO: Fill in the standard commands for this repo
|
||||
|
||||
# Install dependencies
|
||||
|
||||
# Run tests
|
||||
|
||||
# Lint / type check
|
||||
|
||||
# Build / package (if applicable)
|
||||
```
|
||||
40
.claude/rules/workplan-convention.md
Normal file
40
.claude/rules/workplan-convention.md
Normal file
@@ -0,0 +1,40 @@
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
File location: `workplans/GUIDE-BOARD-WP-NNNN-<slug>.md`
|
||||
ID prefix: `GUIDE-BOARD-WP-`
|
||||
|
||||
Work items originate as files in this repo **before** being registered in the hub.
|
||||
|
||||
Canonical workplan/workstream frontmatter statuses are:
|
||||
`proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`.
|
||||
Use `proposed` for a newly drafted plan, `ready` after review against current
|
||||
repo state, and `finished` when implementation is complete. `stalled` and
|
||||
`needs_review` are derived health labels, not stored statuses.
|
||||
|
||||
Closed workplans may be moved to `workplans/archived/` with a completion-date
|
||||
prefix: `YYMMDD-GUIDE-BOARD-WP-NNNN-<slug>.md`. The frontmatter id remains
|
||||
unchanged; the prefix is only for quick visual reference.
|
||||
|
||||
Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**:
|
||||
`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids
|
||||
`ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed
|
||||
directly. Promote anything requiring analysis, design, approval, dependencies, or
|
||||
multiple planned phases into a normal workplan.
|
||||
|
||||
Ecosystem todos from other agents arrive as `[repo:guide-board]` hub tasks —
|
||||
visible at session start. Pick one up by creating the workplan file, then registering
|
||||
the workstream.
|
||||
|
||||
Task blocks use this shape:
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
```
|
||||
|
||||
Status progression is `todo` → `progress` → `done`; use `wait` for waiting or
|
||||
blocked work and `cancel` for stopped work.
|
||||
|
||||
<!-- Ralph Loop rules and HEUREKA sequence: ~/.claude/CLAUDE.md — do not duplicate here -->
|
||||
@@ -1,24 +1,18 @@
|
||||
<!-- custodian-brief: generated by fix-consistency — do not edit manually -->
|
||||
# Custodian Brief — guide-board
|
||||
|
||||
**Domain:** markitect
|
||||
**Last synced:** 2026-05-15 13:10 UTC
|
||||
**Domain:** communication
|
||||
**Last synced:** 2026-06-22 21:22 UTC
|
||||
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
|
||||
|
||||
## Active Workstreams
|
||||
|
||||
### Extension SDK Maturity
|
||||
Progress: 1/4 done | workstream_id: `26aa9511-cd5c-4dd5-989c-d2838ba3b50d`
|
||||
|
||||
**Open tasks:**
|
||||
- · D3.2 - Normalizer Plug-in Contract `b87e68c1`
|
||||
- · D3.3 - SDK Fixture Extension And Acceptance Tests `f3738751`
|
||||
- · D3.4 - Extension Authoring Documentation Refresh `3d390bd4`
|
||||
*(none — repo may need first-session setup)*
|
||||
|
||||
---
|
||||
## MCP Orientation (when available)
|
||||
|
||||
If the state-hub MCP server is reachable, call:
|
||||
`get_domain_summary("markitect")`
|
||||
`get_domain_summary("communication")`
|
||||
This provides richer cross-domain context.
|
||||
If the MCP call fails, use this file as your orientation source.
|
||||
|
||||
17
.repo-classification.yaml
Normal file
17
.repo-classification.yaml
Normal file
@@ -0,0 +1,17 @@
|
||||
repo_classification:
|
||||
standard: Repo Classification Standard
|
||||
version: '1.0'
|
||||
classified_at: '2026-06-22'
|
||||
classified_by: agent
|
||||
category: project
|
||||
domain: communication
|
||||
secondary_domains: []
|
||||
capability_tags:
|
||||
- knowledge
|
||||
- documentation
|
||||
business_stake:
|
||||
- technology
|
||||
- product
|
||||
business_mechanics:
|
||||
- coordination
|
||||
- operation
|
||||
219
AGENTS.md
Normal file
219
AGENTS.md
Normal file
@@ -0,0 +1,219 @@
|
||||
# guide-board — Agent Instructions
|
||||
|
||||
## Repo Identity
|
||||
|
||||
**Purpose:** Certification and compliance preparation framework core with external extension support.
|
||||
|
||||
**Domain:** communication
|
||||
**Repo slug:** guide-board
|
||||
**Topic ID:** `5571d954-0d30-4950-980d-7bcaaad8e3e2`
|
||||
**Workplan prefix:** `GUIDE-BOARD-WP-`
|
||||
|
||||
---
|
||||
|
||||
## State Hub Integration
|
||||
|
||||
The Custodian State Hub tracks work across all domains. Interact via HTTP REST —
|
||||
there is no MCP server for Codex agents.
|
||||
|
||||
| Context | URL |
|
||||
|---------|-----|
|
||||
| Local workstation | `http://127.0.0.1:8000` |
|
||||
| Remote via tunnel | `http://127.0.0.1:18000` |
|
||||
|
||||
### Orient at session start
|
||||
|
||||
```bash
|
||||
# Offline brief — works without hub connection
|
||||
cat .custodian-brief.md
|
||||
|
||||
# Active workstreams for this domain
|
||||
curl -s "http://127.0.0.1:8000/workstreams/?topic_id=5571d954-0d30-4950-980d-7bcaaad8e3e2&status=active" \
|
||||
| python3 -m json.tool
|
||||
|
||||
# Check inbox
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=guide-board&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
Mark a message read:
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
### Log progress (required at session close)
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"summary": "what was done",
|
||||
"event_type": "note",
|
||||
"author": "codex",
|
||||
"workstream_id": "<uuid>",
|
||||
"task_id": "<uuid>"
|
||||
}'
|
||||
```
|
||||
|
||||
Omit `workstream_id` / `task_id` when not applicable.
|
||||
|
||||
### Update task status
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"status": "progress"}'
|
||||
# values: wait | todo | progress | done | cancel
|
||||
```
|
||||
|
||||
### Flag a task for human review
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"needs_human": true, "intervention_note": "reason"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Session Protocol
|
||||
|
||||
**Start:**
|
||||
1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe)
|
||||
2. Check inbox: `GET /messages/?to_agent=guide-board&unread_only=true`; mark read
|
||||
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
|
||||
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
|
||||
|
||||
**During work:**
|
||||
- Update task statuses in workplan files as tasks progress
|
||||
- Record significant decisions via `POST /decisions/`
|
||||
|
||||
**Close:**
|
||||
1. Update workplan file task statuses to reflect progress
|
||||
2. Log: `POST /progress/` with a summary of what changed
|
||||
3. Note for the custodian operator: after workplan file changes, run from
|
||||
`~/state-hub`:
|
||||
```bash
|
||||
make fix-consistency REPO=guide-board
|
||||
```
|
||||
This syncs task status from files into the hub DB.
|
||||
|
||||
---
|
||||
|
||||
## Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=guide-board` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
|
||||
<!-- REPO-AGENTS-EXTENSIONS -->
|
||||
<!-- Append repo-specific agent instructions below this marker.
|
||||
The state-hub template sync preserves content after this line. -->
|
||||
|
||||
---
|
||||
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
Work items originate as files in this repo — not in the hub. The hub is a
|
||||
read/cache/index layer that rebuilds from files.
|
||||
|
||||
**File location:** `workplans/GUIDE-WP-NNNN-<slug>.md`
|
||||
|
||||
**Archived location:** finished workplans may move to
|
||||
`workplans/archived/YYMMDD-GUIDE-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
|
||||
the completion/archive date; the frontmatter `id` does not change.
|
||||
|
||||
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
|
||||
`workplans/ADHOC-YYYY-MM-DD.md` with task ids `ADHOC-YYYY-MM-DD-T01`, etc. Use
|
||||
this only for low-risk work completed directly; create a normal workplan for
|
||||
anything needing analysis, design, approval, dependencies, or multiple phases.
|
||||
|
||||
**Frontmatter:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: GUIDE-WP-NNNN
|
||||
type: workplan
|
||||
title: "..."
|
||||
domain: communication
|
||||
repo: guide-board
|
||||
status: proposed | ready | active | blocked | backlog | finished | archived
|
||||
owner: codex
|
||||
topic_slug: ...
|
||||
created: "YYYY-MM-DD"
|
||||
updated: "YYYY-MM-DD"
|
||||
state_hub_workstream_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
---
|
||||
```
|
||||
|
||||
Use `proposed` for a new draft, `ready` after review against current repo
|
||||
state, and `finished` after implementation. `stalled` and `needs_review` are
|
||||
derived health labels, not frontmatter statuses.
|
||||
|
||||
**Task block format** (one per `##` section):
|
||||
|
||||
```
|
||||
## Task Title
|
||||
|
||||
` ` `task
|
||||
id: GUIDE-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
` ` `
|
||||
|
||||
Task description text.
|
||||
```
|
||||
|
||||
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
|
||||
|
||||
To create a new workplan:
|
||||
1. Write the file following the format above
|
||||
2. Notify the custodian operator to run `make fix-consistency REPO=guide-board`
|
||||
(or send a message to the hub agent via `POST /messages/`)
|
||||
12
CLAUDE.md
Normal file
12
CLAUDE.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# guide-board — Claude Code Instructions
|
||||
|
||||
@SCOPE.md
|
||||
@.claude/rules/repo-identity.md
|
||||
@.claude/rules/session-protocol.md
|
||||
@.claude/rules/first-session.md
|
||||
@.claude/rules/workplan-convention.md
|
||||
@.claude/rules/stack-and-commands.md
|
||||
@.claude/rules/architecture.md
|
||||
@.claude/rules/repo-boundary.md
|
||||
@.claude/rules/credential-routing.md
|
||||
@.claude/rules/agents.md
|
||||
14
README.md
14
README.md
@@ -42,9 +42,15 @@ The same CLI contracts are packaged by the container baseline. See
|
||||
[docs/CONTAINER.md](docs/CONTAINER.md). The dependency-light local API wraps
|
||||
those contracts for service and container operation; see
|
||||
[docs/LOCAL-SERVICE-API.md](docs/LOCAL-SERVICE-API.md).
|
||||
New runs write a richer `sources.lock.json`,
|
||||
`reports/submission-package.json`, extension report fragments, and
|
||||
`exports/export-manifest.json` alongside the assessment package so reviewers can
|
||||
inspect source, metadata, artifact, export, and boundary references.
|
||||
|
||||
The `sample-noop` extension exercises the guide-board contracts without invoking
|
||||
an external harness. `open-cmis-tck` is the first real seed extension.
|
||||
an external harness. `sdk-fixture` demonstrates the extension SDK contracts for
|
||||
schemas, normalizers, mappings, and fixture profiles. `open-cmis-tck` is the
|
||||
first real seed extension.
|
||||
|
||||
See:
|
||||
|
||||
@@ -60,3 +66,9 @@ See:
|
||||
- [docs/SERVICE-JOB-DURABILITY.md](docs/SERVICE-JOB-DURABILITY.md)
|
||||
- [extensions/CANDIDATES.md](extensions/CANDIDATES.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0001-bootstrapping.md](workplans/GUIDE-BOARD-WP-0001-bootstrapping.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md](workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md](workplans/GUIDE-BOARD-WP-0003-extension-sdk-maturity.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0004-source-lock-and-submission-package-baseline.md](workplans/GUIDE-BOARD-WP-0004-source-lock-and-submission-package-baseline.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0005-challenge-and-exclusion-handling.md](workplans/GUIDE-BOARD-WP-0005-challenge-and-exclusion-handling.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0006-service-artifact-access-and-durable-run-index.md](workplans/GUIDE-BOARD-WP-0006-service-artifact-access-and-durable-run-index.md)
|
||||
- [workplans/GUIDE-BOARD-WP-0007-report-and-export-maturity.md](workplans/GUIDE-BOARD-WP-0007-report-and-export-maturity.md)
|
||||
|
||||
137
SCOPE.md
Normal file
137
SCOPE.md
Normal file
@@ -0,0 +1,137 @@
|
||||
# SCOPE
|
||||
|
||||
> This file helps you quickly understand what this repository is about,
|
||||
> when it is relevant, and when it is not.
|
||||
> It is intentionally lightweight and may be incomplete.
|
||||
|
||||
---
|
||||
|
||||
## One-liner
|
||||
|
||||
<!-- Describe the purpose of this repository in one precise sentence. -->
|
||||
<!-- Example: "Provides a lightweight event router for Kubernetes-native systems." -->
|
||||
|
||||
---
|
||||
|
||||
## Core Idea
|
||||
|
||||
<!-- What is the main capability or idea behind this repository? -->
|
||||
<!-- What problem does it try to solve? -->
|
||||
|
||||
---
|
||||
|
||||
## In Scope
|
||||
|
||||
<!-- What this repository is responsible for. -->
|
||||
<!-- Be explicit and concrete. -->
|
||||
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
---
|
||||
|
||||
## Out of Scope
|
||||
|
||||
<!-- What this repository deliberately does NOT do. -->
|
||||
<!-- This is often more important than "In Scope". -->
|
||||
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
---
|
||||
|
||||
## Relevant When
|
||||
|
||||
<!-- When should someone consider using or exploring this repository? -->
|
||||
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
---
|
||||
|
||||
## Not Relevant When
|
||||
|
||||
<!-- When should someone ignore this repository? -->
|
||||
|
||||
-
|
||||
-
|
||||
-
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
<!-- Rough indication of maturity. No strict format required. -->
|
||||
|
||||
- Status: <!-- e.g. concept / experimental / active / stable / deprecated -->
|
||||
- Implementation: <!-- e.g. idea / partial / substantial / complete -->
|
||||
- Stability: <!-- e.g. unstable / evolving / stable -->
|
||||
- Usage: <!-- e.g. none / personal / internal / production -->
|
||||
|
||||
<!-- Add any notes that help set expectations. -->
|
||||
|
||||
---
|
||||
|
||||
## How It Fits
|
||||
|
||||
<!-- Where does this repository sit in the bigger picture? -->
|
||||
|
||||
- Upstream dependencies:
|
||||
- Downstream consumers:
|
||||
- Often used with:
|
||||
|
||||
---
|
||||
|
||||
## Terminology
|
||||
|
||||
<!-- Terms that are important to understand this repo. -->
|
||||
<!-- Especially useful if naming differs from other repos. -->
|
||||
|
||||
- Preferred terms:
|
||||
- Also known as:
|
||||
- Potentially confusing terms:
|
||||
|
||||
---
|
||||
|
||||
## Related / Overlapping Repositories
|
||||
|
||||
<!-- List repositories that have similar or adjacent responsibilities. -->
|
||||
<!-- Helps detect duplication and navigate the ecosystem. -->
|
||||
|
||||
- <repo-name> — <!-- how it relates -->
|
||||
|
||||
---
|
||||
|
||||
## Getting Oriented
|
||||
|
||||
<!-- If someone decides to look deeper, where should they start? -->
|
||||
|
||||
- Start with:
|
||||
- Key files / directories:
|
||||
- Entry points:
|
||||
|
||||
---
|
||||
|
||||
## Provided Capabilities
|
||||
|
||||
<!-- What can this repo's domain provide to other domains on request? -->
|
||||
<!-- Each capability block is parsed by the state-hub capability catalog ingest. -->
|
||||
<!-- Remove the examples and add your own, or leave empty if none. -->
|
||||
|
||||
<!--
|
||||
```capability
|
||||
type: infrastructure
|
||||
title: Example capability title
|
||||
description: What this capability provides, in one or two sentences.
|
||||
keywords: [keyword1, keyword2, keyword3]
|
||||
```
|
||||
-->
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
<!-- Anything else worth knowing. Keep it short. -->
|
||||
@@ -355,7 +355,9 @@ Stores run artifacts by reference and checksum:
|
||||
|
||||
The first implementation builds the assessment package artifact manifest from
|
||||
runner-emitted artifact refs and computes checksums for files inside the run
|
||||
directory.
|
||||
directory. New runs also write a source lock and a submission package manifest
|
||||
that fingerprint reviewable run files and summarize runner or normalizer
|
||||
metadata reported by extensions.
|
||||
|
||||
### Normalizer
|
||||
|
||||
@@ -559,6 +561,18 @@ building complex runtime code.
|
||||
- `artifact_policy`
|
||||
- `runtime_policy`
|
||||
|
||||
### `SourceLock`
|
||||
|
||||
- `framework_refs`
|
||||
- `extension_refs`
|
||||
- `frameworks`
|
||||
- `extensions`
|
||||
- `mapping_sets`
|
||||
- `profiles`
|
||||
- `policy_refs`
|
||||
- `authorities`
|
||||
- `metadata_hooks`
|
||||
|
||||
### `RawArtifact`
|
||||
|
||||
- `id`
|
||||
@@ -626,6 +640,31 @@ building complex runtime code.
|
||||
- `certification_boundary`
|
||||
- `created_at`
|
||||
|
||||
### `SubmissionPackage`
|
||||
|
||||
- `run_id`
|
||||
- `package_identity`
|
||||
- `source_lock_ref`
|
||||
- `source_lock`
|
||||
- `reports`
|
||||
- `normalized_outputs`
|
||||
- `profile_snapshots`
|
||||
- `artifact_manifest`
|
||||
- `reported_metadata`
|
||||
- `certification_boundary`
|
||||
|
||||
### `ExportManifest`
|
||||
|
||||
- `export_type`
|
||||
- `source_package_ref`
|
||||
- `source_lock_ref`
|
||||
- `summary`
|
||||
- `policy_summary`
|
||||
- `mapping_summary`
|
||||
- `report_fragments`
|
||||
- `counts`
|
||||
- `certification_boundary`
|
||||
|
||||
## Result Vocabulary
|
||||
|
||||
The evidence model should allow these statuses:
|
||||
@@ -713,8 +752,11 @@ runs/<run-id>/
|
||||
mappings.json
|
||||
reports/
|
||||
report.md
|
||||
fragments.json
|
||||
assessment-package.json
|
||||
submission-package.json
|
||||
exports/
|
||||
export-manifest.json
|
||||
```
|
||||
|
||||
## Container And Service Model
|
||||
@@ -775,6 +817,9 @@ Use separate concepts:
|
||||
- defect: unexpected product or process failure.
|
||||
|
||||
The report must make these visible separately.
|
||||
The current policy layer loads challenge and exclusion refs from assessment
|
||||
profiles, annotates findings and evidence, and keeps `unexpected_findings`
|
||||
visible for gate semantics unless a finding is separately expected or waived.
|
||||
|
||||
### Source Locking
|
||||
|
||||
@@ -787,7 +832,12 @@ Each run should lock:
|
||||
- test suite IDs,
|
||||
- mapping version,
|
||||
- target profile snapshot,
|
||||
- waiver snapshot.
|
||||
- expectation and waiver refs.
|
||||
|
||||
The current source lock remains backward-compatible with the original
|
||||
`framework_refs` and `extension_refs` fields while adding checksummed profiles,
|
||||
mapping-set refs, optional policy refs, authority descriptors, and metadata
|
||||
hooks for runners and normalizers.
|
||||
|
||||
## Implementation Sequence
|
||||
|
||||
@@ -800,6 +850,8 @@ Each run should lock:
|
||||
6. Add container design after the CLI baseline is stable.
|
||||
7. Add optional service API around the CLI job model.
|
||||
8. Add OSCAL export and procedural evidence-pack support after the internal
|
||||
evidence model proves itself with executable extensions.
|
||||
evidence model proves itself with executable extensions. The first generic
|
||||
export is `exports/export-manifest.json`; authority-specific interchange
|
||||
remains extension-owned until the internal model is stable.
|
||||
|
||||
The first extension SDK contract is documented in `docs/EXTENSION-SDK.md`.
|
||||
|
||||
@@ -27,7 +27,8 @@ Every run needs:
|
||||
|
||||
The target profile describes the candidate system or artifact being assessed.
|
||||
The assessment profile selects frameworks, extensions, check groups, runtime
|
||||
policy, waivers, expectations, and output policy.
|
||||
policy, expectations, waivers, challenges, authority exclusions, and output
|
||||
policy.
|
||||
|
||||
## CLI Flow
|
||||
|
||||
@@ -65,6 +66,8 @@ when a wrapper script or container entrypoint should keep commands shorter.
|
||||
For the repeatable external extension acceptance path, including validation,
|
||||
planning, live execution, and retained result review, see
|
||||
`docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
|
||||
For extension-author contracts such as profile schema descriptors and
|
||||
normalizer plug-ins, see `docs/EXTENSION-SDK.md`.
|
||||
|
||||
## CLI Results
|
||||
|
||||
@@ -75,6 +78,8 @@ A completed CLI command prints a JSON result with:
|
||||
- `run_dir`: output directory,
|
||||
- `assessment_package`: JSON assessment package path,
|
||||
- `report`: Markdown report path,
|
||||
- `submission_package`: portable submission package manifest path,
|
||||
- `export_manifest`: portable generic JSON export manifest path,
|
||||
- `retention_summary`: compact durable summary path.
|
||||
|
||||
The output directory uses this contract:
|
||||
@@ -82,15 +87,35 @@ The output directory uses this contract:
|
||||
```text
|
||||
run.json
|
||||
plan.json
|
||||
sources.lock.json
|
||||
target-profile.snapshot.json
|
||||
assessment-profile.snapshot.json
|
||||
retention-summary.json
|
||||
normalized/evidence.json
|
||||
normalized/findings.json
|
||||
normalized/mappings.json
|
||||
reports/assessment-package.json
|
||||
reports/report.md
|
||||
reports/fragments.json
|
||||
reports/submission-package.json
|
||||
exports/export-manifest.json
|
||||
artifacts/
|
||||
```
|
||||
|
||||
`sources.lock.json` records the framework refs, extension versions, mapping
|
||||
sets, profile snapshots, policy and review refs, authority refs, and extension
|
||||
metadata hooks used for the run. `reports/submission-package.json` points at
|
||||
the reviewable package files, includes checksums where files exist, carries the
|
||||
raw artifact manifest, and repeats the certification boundary. It is a portable
|
||||
handoff manifest for preparation evidence, not an authority-specific final
|
||||
submission.
|
||||
|
||||
Extension report fragments are recorded in `reports/fragments.json`, embedded in
|
||||
`reports/assessment-package.json`, and rendered into the Markdown report.
|
||||
`exports/export-manifest.json` is the first generic portable export surface. It
|
||||
is derived from the assessment package and carries summary, policy, mapping,
|
||||
fragment, count, source-lock, and boundary references.
|
||||
|
||||
Use the retained run helpers for history:
|
||||
|
||||
```sh
|
||||
@@ -105,6 +130,10 @@ PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs
|
||||
PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs
|
||||
```
|
||||
|
||||
Trend summaries include status changes, unexpected finding deltas, unresolved
|
||||
review deltas, mapping target deltas, evidence result deltas, and a compact
|
||||
human-readable summary string for each target/assessment pair.
|
||||
|
||||
## Local Service Flow
|
||||
|
||||
Start the service from the guide-board repository:
|
||||
@@ -148,6 +177,15 @@ Fetch reports after the job status is `succeeded`:
|
||||
curl -sf http://127.0.0.1:8080/runs/JOB_ID/reports | python3 -m json.tool
|
||||
```
|
||||
|
||||
Inspect retained run history, including runs produced before the current
|
||||
service process started:
|
||||
|
||||
```sh
|
||||
curl -sf "http://127.0.0.1:8080/retained-runs?runs_dir=runs" | python3 -m json.tool
|
||||
curl -sf "http://127.0.0.1:8080/retained-runs/latest?runs_dir=runs" | python3 -m json.tool
|
||||
curl -sf "http://127.0.0.1:8080/retained-runs/RUN_ID/artifact-manifest?runs_dir=runs" | python3 -m json.tool
|
||||
```
|
||||
|
||||
Service job state is currently in memory for the running service process. Run
|
||||
artifacts are durable in the output directory and can still be inspected after a
|
||||
service restart. See `docs/SERVICE-JOB-DURABILITY.md` for the restart and
|
||||
@@ -185,6 +223,23 @@ Individual evidence items use:
|
||||
- `expected_gap`
|
||||
- `infrastructure_error`
|
||||
|
||||
## Review State
|
||||
|
||||
Assessment profiles may reference:
|
||||
|
||||
- `expectations_ref`: known target posture, optional scope, or accepted gaps,
|
||||
- `waivers_ref`: approved, time-bounded exceptions,
|
||||
- `challenges_ref`: review claims that a finding, check, mapping, or native
|
||||
result should be challenged,
|
||||
- `exclusions_ref`: authority or program exclusions that apply to selected
|
||||
findings.
|
||||
|
||||
Challenges and exclusions annotate findings and evidence. They do not silently
|
||||
turn failures into passing evidence and they do not reduce the
|
||||
`unexpected_findings` count used by default gates. Retained summaries expose
|
||||
separate counts for expected findings, waived findings, challenged findings,
|
||||
authority exclusions, unresolved defects, and unresolved review items.
|
||||
|
||||
## Candidate System Checklist
|
||||
|
||||
Before starting a run against candidate software, confirm:
|
||||
|
||||
@@ -8,8 +8,8 @@ Created: 2026-05-07
|
||||
Compliance evidence packs cover frameworks where guide-board cannot rely on an
|
||||
official executable harness. They help prepare and perform assessments by
|
||||
organizing evidence requests, expected artifacts, reviewer workflow, waivers,
|
||||
and run reports. They do not replace auditors, accredited certification bodies,
|
||||
legal counsel, or official standard text.
|
||||
challenges, authority exclusions, and run reports. They do not replace auditors,
|
||||
accredited certification bodies, legal counsel, or official standard text.
|
||||
|
||||
Examples include GDPR, SOC 2, HIPAA, NF Z 42-013, NF 461, ISO 14641, ISO 15489,
|
||||
and similar procedural or control-oriented frameworks.
|
||||
@@ -83,7 +83,7 @@ Each request should include:
|
||||
|
||||
Requests should be phrased as collection guidance, not as legal conclusions.
|
||||
|
||||
## Waivers And Expected Gaps
|
||||
## Review Policy Records
|
||||
|
||||
Evidence packs use the same expectation and waiver model as executable
|
||||
extensions.
|
||||
@@ -103,6 +103,16 @@ Use waivers for:
|
||||
|
||||
Every waiver should include owner, reason, approval status, and expiry.
|
||||
|
||||
Use challenges for disputed checks, disputed mappings, imported native result
|
||||
questions, or evidence that needs a reviewer decision before it can be treated
|
||||
as a defect. Use authority exclusions only when a program, standard, or
|
||||
authorized reviewer excludes a requirement or check from the assessment scope.
|
||||
Both records should cite stable requirement refs, check refs, evidence refs, or
|
||||
authority source refs rather than reproducing restricted standard text.
|
||||
|
||||
Challenges and exclusions make review state visible; they do not by themselves
|
||||
claim compliance or remove default gate-visible unexpected findings.
|
||||
|
||||
## Framework Notes
|
||||
|
||||
GDPR packs should emphasize processing inventory, lawful basis records, data
|
||||
@@ -129,6 +139,7 @@ extensions:
|
||||
|
||||
- normalized evidence,
|
||||
- findings,
|
||||
- review annotations for expectations, waivers, challenges, and exclusions,
|
||||
- mapping records,
|
||||
- assessment packages,
|
||||
- retention summaries,
|
||||
|
||||
@@ -62,7 +62,8 @@ The script:
|
||||
- builds `guide-board-core:smoke`,
|
||||
- mounts a host output directory at `/runs`,
|
||||
- runs the bundled sample assessment,
|
||||
- verifies that the expected run artifacts are present on the host.
|
||||
- verifies that the expected run artifacts, report fragments, submission
|
||||
manifest, and generic export manifest are present on the host.
|
||||
|
||||
Override the runtime, image name, or output directory when needed:
|
||||
|
||||
@@ -144,4 +145,9 @@ podman run --rm -p 8080:8080 \
|
||||
|
||||
The service layer adds in-memory job tracking and HTTP transport. Execution
|
||||
semantics remain the CLI/core semantics documented in
|
||||
`docs/LOCAL-SERVICE-API.md`.
|
||||
`docs/LOCAL-SERVICE-API.md`. Mounted run directories remain discoverable through
|
||||
the retained-run endpoints, for example:
|
||||
|
||||
```sh
|
||||
curl -sf "http://127.0.0.1:8080/retained-runs?runs_dir=/runs" | python3 -m json.tool
|
||||
```
|
||||
|
||||
@@ -71,11 +71,20 @@ The key runtime fields are:
|
||||
- `extension_type`: one of the supported archetypes from the architecture
|
||||
blueprint.
|
||||
- `supported_frameworks`: framework IDs this extension can contribute evidence
|
||||
for.
|
||||
for. Descriptor objects with `id`, `version`, `source_url`, and
|
||||
`authority_ref` may be used when source metadata is available.
|
||||
- `authorities`: authority IDs or descriptor objects with optional source URL,
|
||||
version, license, and access notes.
|
||||
- `metadata`: optional extension-level metadata such as adapter version or
|
||||
source URL. The core preserves it in source locks and evidence metadata.
|
||||
- `check_groups`: named groups that assessment profiles can select.
|
||||
- `preflight_runner`: optional runner ID used before selected check groups.
|
||||
- `runner_entrypoints`: concrete runner declarations.
|
||||
- `normalizers`: optional plug-ins that convert native runner output into the
|
||||
stable runner-result shape before evidence is written.
|
||||
- `mappings`: mapping set IDs under `mappings/<mapping-id>.json`.
|
||||
- `report_fragments`: optional Markdown file or Python module descriptors for
|
||||
extension-owned report content.
|
||||
- `certification_boundary`: explicit statement of what the extension does not
|
||||
certify.
|
||||
|
||||
@@ -139,6 +148,11 @@ Example:
|
||||
"module_path": "src/open_cmis_tck/preflight.py",
|
||||
"callable": "run",
|
||||
"command": null,
|
||||
"metadata": {
|
||||
"harness_id": "opencmis-tck",
|
||||
"harness_version": "extension-detected-or-declared",
|
||||
"source_url": "https://chemistry.apache.org/java/opencmis.html"
|
||||
},
|
||||
"description": "Checks whether the CMIS Browser Binding endpoint is reachable."
|
||||
}
|
||||
```
|
||||
@@ -197,6 +211,53 @@ to extension-owned mappings and writes normalized mapping records to:
|
||||
runs/<run-id>/normalized/mappings.json
|
||||
```
|
||||
|
||||
## Report Fragments
|
||||
|
||||
Extensions can contribute report fragments through `report_fragments`.
|
||||
|
||||
Static Markdown file:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "overview",
|
||||
"kind": "markdown_file",
|
||||
"path": "reports/overview.md",
|
||||
"title": "Overview"
|
||||
}
|
||||
```
|
||||
|
||||
Dynamic Python fragment:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "sdk-fixture-summary",
|
||||
"kind": "python_module",
|
||||
"module_path": "reports/sdk_fixture_summary.py",
|
||||
"callable": "build_fragment",
|
||||
"path": null,
|
||||
"title": "SDK Fixture Summary"
|
||||
}
|
||||
```
|
||||
|
||||
Fragment paths are resolved relative to the extension root and must stay inside
|
||||
that root. A Python fragment receives `root`, `run_dir`, `run_id`, `plan`,
|
||||
`evidence`, `findings`, `mappings`, `assessment_package`, `policy_summary`,
|
||||
`source_lock`, `extension_path`, and `report_fragment`.
|
||||
|
||||
It returns:
|
||||
|
||||
```python
|
||||
def build_fragment(context: dict) -> dict:
|
||||
return {
|
||||
"markdown": "### Extension Summary\n\n- evidence items: 2",
|
||||
"structured": {"evidence_count": 2},
|
||||
}
|
||||
```
|
||||
|
||||
Fragments are written to `reports/fragments.json`, embedded in the assessment
|
||||
package, rendered in `reports/report.md`, and summarized in
|
||||
`exports/export-manifest.json`.
|
||||
|
||||
## Evidence Request Sets
|
||||
|
||||
Procedural and hybrid compliance extensions may include evidence request sets
|
||||
@@ -238,6 +299,33 @@ Expectation sets mark known posture as expected. Waiver sets mark approved,
|
||||
time-bounded exceptions. Both are applied after findings are generated, and the
|
||||
assessment package records policy summary counts.
|
||||
|
||||
## Challenges And Authority Exclusions
|
||||
|
||||
Assessment profiles may also reference challenge and exclusion sets:
|
||||
|
||||
```json
|
||||
{
|
||||
"challenges_ref": "profiles/challenges/example.json",
|
||||
"exclusions_ref": "profiles/exclusions/example.json"
|
||||
}
|
||||
```
|
||||
|
||||
Challenge sets validate against `docs/schemas/challenge-set.schema.json`.
|
||||
Exclusion sets validate against `docs/schemas/exclusion-set.schema.json`.
|
||||
Records can match findings by requirement refs, check refs, evidence refs,
|
||||
result refs, or classification refs. They also carry owner, review status,
|
||||
rationale, authority source refs, review dates, optional expiry, native IDs,
|
||||
and free-form metadata.
|
||||
|
||||
Use challenges when an extension author or assessment team believes a finding
|
||||
needs review because a check is invalid, a native harness result is disputed, or
|
||||
a mapping is wrong. Use exclusions when an authority or program explicitly
|
||||
removes a requirement, check, or result from the assessment scope. The core
|
||||
preserves these distinctions in findings, evidence review annotations,
|
||||
assessment packages, reports, and retained summaries, but default gate semantics
|
||||
still count the underlying finding as unexpected unless it is separately
|
||||
expected or waived.
|
||||
|
||||
## Python Runner Contract
|
||||
|
||||
A Python runner receives one context object and returns one result object.
|
||||
@@ -270,11 +358,20 @@ Result fields:
|
||||
- `observations`: human-readable observations.
|
||||
- `facts`: structured facts extracted by the runner.
|
||||
- `artifact_refs`: references to raw artifacts written by the runner.
|
||||
- `requirement_refs`: optional requirement refs discovered by the runner.
|
||||
- `metadata`: optional generic metadata such as `harness_version`,
|
||||
`test_suite_id`, `adapter_version`, `source_url`, or native result IDs.
|
||||
|
||||
Artifact refs must be paths relative to the run directory. After runner
|
||||
execution, the core fingerprints existing artifact refs into the assessment
|
||||
package `artifact_manifest`.
|
||||
|
||||
Runner metadata is merged with manifest entrypoint metadata and preserved under
|
||||
evidence `facts.source_metadata`. The same metadata is also summarized in the
|
||||
submission package manifest, which lets reviewers distinguish the extension
|
||||
version from the harness or native test-suite version without adding
|
||||
domain-specific fields to the core.
|
||||
|
||||
If a Python runner raises an exception, the core converts that failure into
|
||||
`infrastructure_error` evidence so the assessment package remains complete.
|
||||
|
||||
@@ -283,6 +380,94 @@ or `infrastructure_error`, downstream check groups for that extension are not
|
||||
executed; they receive `blocked` evidence with `blocked_reason:
|
||||
preflight_failed`.
|
||||
|
||||
## Normalizer Plug-ins
|
||||
|
||||
Runners can keep returning guide-board-ready result objects directly. When a
|
||||
runner wraps a native harness or scanner that writes its own result format, the
|
||||
extension can add a normalizer descriptor:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "native-probe-normalizer",
|
||||
"kind": "python_module",
|
||||
"module_path": "normalizers/native_probe.py",
|
||||
"callable": "normalize",
|
||||
"runner_ref": "native-probe",
|
||||
"metadata": {
|
||||
"adapter_version": "0.1.0"
|
||||
},
|
||||
"description": "Converts native runner output into guide-board evidence."
|
||||
}
|
||||
```
|
||||
|
||||
Normalizers are declared in `extension.json` under `normalizers`. The original
|
||||
string shorthand remains valid for descriptive-only entries, but only descriptor
|
||||
objects are loaded and invoked by the core.
|
||||
|
||||
The first supported normalizer kind is `python_module`. Its module path is
|
||||
resolved relative to the extension root and must stay inside that root. The
|
||||
callable receives one context object:
|
||||
|
||||
- `root`: guide-board core root path as a string.
|
||||
- `extension_path`: extension root path as a string.
|
||||
- `run_dir`: output run directory path as a string.
|
||||
- `run_id`: current run ID.
|
||||
- `plan`: full run plan snapshot.
|
||||
- `step`: the step being normalized.
|
||||
- `target_profile`: target profile snapshot.
|
||||
- `assessment_profile`: assessment profile snapshot.
|
||||
- `normalizer`: manifest normalizer descriptor.
|
||||
- `runner_result`: the current runner-result object.
|
||||
|
||||
A normalizer returns any subset of the runner-result fields:
|
||||
|
||||
```python
|
||||
def normalize(context: dict) -> dict:
|
||||
return {
|
||||
"result": "pass",
|
||||
"observations": ["Native result was normalized."],
|
||||
"facts": {"native_status": "ok"},
|
||||
"artifact_refs": ["artifacts/native-result.json"],
|
||||
"requirement_refs": ["framework.requirement"],
|
||||
}
|
||||
```
|
||||
|
||||
The core merges the normalizer output over the runner result:
|
||||
|
||||
- `result` replaces the previous result.
|
||||
- `observations` are appended.
|
||||
- `facts` are merged.
|
||||
- `artifact_refs` and `requirement_refs` are deduplicated.
|
||||
- `metadata` is merged.
|
||||
- `normalizer_refs` is recorded in evidence facts when any normalizer runs.
|
||||
|
||||
If a normalizer raises an exception, the step becomes
|
||||
`infrastructure_error` evidence and the run still produces its normal artifact
|
||||
set.
|
||||
|
||||
The bundled `extensions/sdk-fixture` extension is the copyable reference path
|
||||
for profile schemas, a native-output runner, a normalizer, mappings, and fixture
|
||||
profiles.
|
||||
|
||||
## Source Lock And Submission Package
|
||||
|
||||
Every new run writes `sources.lock.json`, `reports/submission-package.json`,
|
||||
and the generic portable export manifest at `exports/export-manifest.json`.
|
||||
Extension authors should treat source metadata as part of the evidence contract:
|
||||
|
||||
- declare extension, authority, framework, runner, and normalizer metadata in
|
||||
`extension.json` when it is static;
|
||||
- return runner or normalizer `metadata` when versions, native result IDs, or
|
||||
test-suite IDs are detected at runtime;
|
||||
- keep mapping sets under `mappings/` so the core can checksum them in the
|
||||
source lock;
|
||||
- keep restricted or licensed assets referenced by metadata or artifacts rather
|
||||
than vendored into the core.
|
||||
|
||||
The submission package manifest is generic guide-board output. Authority-specific
|
||||
final submissions, trademark assertions, or certification conclusions remain
|
||||
extension-owned or reviewer-owned.
|
||||
|
||||
## Result Statuses
|
||||
|
||||
Initial statuses:
|
||||
@@ -303,11 +488,14 @@ Initial statuses:
|
||||
## Current Extension Examples
|
||||
|
||||
- `sample-noop`: no runner, used to validate the core contracts.
|
||||
- `sdk-fixture`: compact SDK fixture covering profile schemas, runner output,
|
||||
normalizer invocation, mapping, and fixture profiles.
|
||||
- `open-cmis-tck`: provides a Python CMIS Browser Binding preflight runner and
|
||||
declares the future external OpenCMIS TCK runner.
|
||||
|
||||
## Next SDK Steps
|
||||
|
||||
- Add normalizer plug-in contracts.
|
||||
- Add extension-owned schema validation for domain-specific target profile
|
||||
fields.
|
||||
- Broaden normalizer examples as real external extensions adopt native harness
|
||||
result formats.
|
||||
- Add more extension-owned schema validation examples for assessment-specific
|
||||
domain constraints.
|
||||
|
||||
@@ -15,6 +15,9 @@ domain-specific harness logic into the core.
|
||||
runtime dependencies, and harness behavior remain owned by that extension
|
||||
repository.
|
||||
|
||||
For a dependency-light SDK reference extension that can be copied into a
|
||||
temporary external repository, see `extensions/sdk-fixture`.
|
||||
|
||||
## Acceptance Stages
|
||||
|
||||
Run these stages from the guide-board repository.
|
||||
|
||||
@@ -98,7 +98,41 @@ errors.
|
||||
### `GET /runs/{job_id}/reports`
|
||||
|
||||
Returns the Markdown report content, assessment package JSON, retention summary,
|
||||
and their filesystem paths after a job has succeeded.
|
||||
submission package JSON, export manifest JSON when present, and their filesystem
|
||||
paths after a job has succeeded.
|
||||
|
||||
### `GET /retained-runs`
|
||||
|
||||
Lists durable retained run summaries by scanning a runs directory. Without a
|
||||
query parameter, the service scans `<root>/runs`.
|
||||
|
||||
```text
|
||||
GET /retained-runs?runs_dir=/runs
|
||||
```
|
||||
|
||||
### `GET /retained-runs/latest`
|
||||
|
||||
Selects the latest retained run, optionally filtered by target and assessment
|
||||
profile refs.
|
||||
|
||||
```text
|
||||
GET /retained-runs/latest?runs_dir=/runs&target=sample-repository&assessment=sample-noop-assessment
|
||||
```
|
||||
|
||||
### `GET /retained-runs/{run_id}/reports`
|
||||
|
||||
Returns the retained summary plus safe report paths for a durable run. This
|
||||
works after a service restart because it reads `retention-summary.json` from
|
||||
disk instead of in-memory job records.
|
||||
|
||||
### `GET /retained-runs/{run_id}/artifact-manifest`
|
||||
|
||||
Returns the assessment package `artifact_manifest` for a retained run. If the
|
||||
run predates assessment packages, the response is compatible and returns an
|
||||
empty manifest with `compatibility: "assessment-package-missing"`.
|
||||
|
||||
Retained-run endpoints validate report and artifact paths before returning
|
||||
them. A path that escapes the selected run directory is rejected.
|
||||
|
||||
## Container Mode
|
||||
|
||||
@@ -112,5 +146,6 @@ podman run --rm -p 8080:8080 \
|
||||
```
|
||||
|
||||
The service keeps job state in memory. Durable run evidence remains in the
|
||||
mounted output directory. See `docs/SERVICE-JOB-DURABILITY.md` for the explicit
|
||||
restart and recovery contract.
|
||||
mounted output directory and can be discovered through `GET /retained-runs`
|
||||
after restart. See `docs/SERVICE-JOB-DURABILITY.md` for the explicit recovery
|
||||
contract.
|
||||
|
||||
@@ -13,16 +13,19 @@ Durable state lives in run directories:
|
||||
|
||||
- `run.json`
|
||||
- `plan.json`
|
||||
- `sources.lock.json`
|
||||
- `retention-summary.json`
|
||||
- `normalized/evidence.json`
|
||||
- `normalized/findings.json`
|
||||
- `normalized/mappings.json`
|
||||
- `reports/assessment-package.json`
|
||||
- `reports/report.md`
|
||||
- `reports/submission-package.json`
|
||||
- `artifacts/`
|
||||
|
||||
The durable recovery index is the set of `retention-summary.json` files under a
|
||||
runs directory.
|
||||
runs directory. No separate durable service index is required for the baseline;
|
||||
the service reconstructs retained-run views by scanning those summaries.
|
||||
|
||||
## Why In-Memory Jobs Stay The Baseline
|
||||
|
||||
@@ -47,9 +50,14 @@ After a service restart:
|
||||
- old `job_id` values are invalid,
|
||||
- `GET /runs/{job_id}` cannot recover pre-restart job metadata,
|
||||
- `GET /runs/{job_id}/reports` only works for jobs known to the current process,
|
||||
- run artifacts from earlier service processes remain available on disk.
|
||||
- run artifacts from earlier service processes remain available on disk,
|
||||
- `GET /retained-runs`, `GET /retained-runs/latest`,
|
||||
`GET /retained-runs/{run_id}/reports`, and
|
||||
`GET /retained-runs/{run_id}/artifact-manifest` can expose completed retained
|
||||
runs after restart.
|
||||
|
||||
Operators should recover previous results with the CLI run-history commands:
|
||||
Operators can recover previous results with either the CLI run-history commands
|
||||
or the retained-run service endpoints:
|
||||
|
||||
```sh
|
||||
PYTHONPATH=src python3 -m guide_board runs list --runs-dir runs
|
||||
@@ -57,6 +65,12 @@ PYTHONPATH=src python3 -m guide_board runs latest --runs-dir runs
|
||||
PYTHONPATH=src python3 -m guide_board runs report --runs-dir runs --run-id RUN_ID
|
||||
```
|
||||
|
||||
```sh
|
||||
curl -sf "http://127.0.0.1:8080/retained-runs?runs_dir=runs" | python3 -m json.tool
|
||||
curl -sf "http://127.0.0.1:8080/retained-runs/RUN_ID/reports?runs_dir=runs" | python3 -m json.tool
|
||||
curl -sf "http://127.0.0.1:8080/retained-runs/RUN_ID/artifact-manifest?runs_dir=runs" | python3 -m json.tool
|
||||
```
|
||||
|
||||
## Recovery Flow
|
||||
|
||||
Use this flow when the service process restarted or a browser/UI lost its job
|
||||
@@ -64,8 +78,9 @@ state:
|
||||
|
||||
1. Identify the output directory passed to `POST /runs`.
|
||||
2. Confirm whether `retention-summary.json` exists.
|
||||
3. If it exists, use `guide-board runs report --runs-dir <parent>` to retrieve
|
||||
report paths.
|
||||
3. If it exists, use `guide-board runs report --runs-dir <parent>` or
|
||||
`GET /retained-runs/{run_id}/reports?runs_dir=<parent>` to retrieve report
|
||||
paths.
|
||||
4. If only partial files exist, inspect `run.json`, `plan.json`, and artifacts
|
||||
before rerunning.
|
||||
5. Rerun into a fresh output directory when the prior status is unclear.
|
||||
@@ -73,8 +88,8 @@ state:
|
||||
## Future Durable Index Option
|
||||
|
||||
A future durable service index may be added if UI or automation workflows need
|
||||
cross-restart job lookup. If added, it should remain reconstructable from run
|
||||
directories and should not become the authority for assessment results.
|
||||
cross-restart transport job lookup. If added, it should remain reconstructable
|
||||
from run directories and should not become the authority for assessment results.
|
||||
|
||||
The minimum acceptable durable index would contain:
|
||||
|
||||
|
||||
@@ -17,6 +17,9 @@
|
||||
"evidence_refs",
|
||||
"artifact_manifest",
|
||||
"waivers",
|
||||
"challenges",
|
||||
"exclusions",
|
||||
"report_fragments",
|
||||
"certification_boundary",
|
||||
"created_at"
|
||||
],
|
||||
@@ -34,6 +37,9 @@
|
||||
"evidence_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"artifact_manifest": { "type": "array", "items": { "type": "object" } },
|
||||
"waivers": { "type": "array", "items": { "type": "object" } },
|
||||
"challenges": { "type": "array", "items": { "type": "object" } },
|
||||
"exclusions": { "type": "array", "items": { "type": "object" } },
|
||||
"report_fragments": { "type": "array", "items": { "type": "object" } },
|
||||
"certification_boundary": { "type": "string" },
|
||||
"created_at": { "type": "string" }
|
||||
}
|
||||
|
||||
@@ -28,6 +28,8 @@
|
||||
},
|
||||
"expectations_ref": { "type": ["string", "null"] },
|
||||
"waivers_ref": { "type": ["string", "null"] },
|
||||
"challenges_ref": { "type": ["string", "null"] },
|
||||
"exclusions_ref": { "type": ["string", "null"] },
|
||||
"output_policy": { "type": "object" },
|
||||
"retention_policy": { "type": "object" },
|
||||
"runtime_policy": { "type": "object" }
|
||||
|
||||
56
docs/schemas/challenge-set.schema.json
Normal file
56
docs/schemas/challenge-set.schema.json
Normal file
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Guide Board Challenge Set",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"target_profile_ref",
|
||||
"challenges"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"target_profile_ref": { "type": "string" },
|
||||
"challenges": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"requirement_refs",
|
||||
"check_refs",
|
||||
"evidence_refs",
|
||||
"result_refs",
|
||||
"classification_refs",
|
||||
"authority_source_refs",
|
||||
"owner",
|
||||
"review_status",
|
||||
"rationale",
|
||||
"created_at",
|
||||
"review_due_at",
|
||||
"expires_at",
|
||||
"native_challenge_id",
|
||||
"metadata"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"requirement_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"check_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"evidence_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"result_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"classification_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"authority_source_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"owner": { "type": "string" },
|
||||
"review_status": { "type": "string" },
|
||||
"rationale": { "type": "string" },
|
||||
"created_at": { "type": "string" },
|
||||
"review_due_at": { "type": ["string", "null"] },
|
||||
"expires_at": { "type": ["string", "null"] },
|
||||
"native_challenge_id": { "type": ["string", "null"] },
|
||||
"metadata": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -42,6 +42,7 @@
|
||||
},
|
||||
"observations": { "type": "array", "items": { "type": "string" } },
|
||||
"facts": { "type": "object" },
|
||||
"review": { "type": "object" },
|
||||
"requirement_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"artifact_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"started_at": { "type": "string" },
|
||||
|
||||
60
docs/schemas/exclusion-set.schema.json
Normal file
60
docs/schemas/exclusion-set.schema.json
Normal file
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Guide Board Authority Exclusion Set",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"target_profile_ref",
|
||||
"exclusions"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"target_profile_ref": { "type": "string" },
|
||||
"exclusions": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"authority_ref",
|
||||
"requirement_refs",
|
||||
"check_refs",
|
||||
"evidence_refs",
|
||||
"result_refs",
|
||||
"classification_refs",
|
||||
"authority_source_refs",
|
||||
"owner",
|
||||
"approved_by",
|
||||
"review_status",
|
||||
"rationale",
|
||||
"created_at",
|
||||
"review_due_at",
|
||||
"expires_at",
|
||||
"native_exclusion_id",
|
||||
"metadata"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"authority_ref": { "type": "string" },
|
||||
"requirement_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"check_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"evidence_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"result_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"classification_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"authority_source_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"owner": { "type": "string" },
|
||||
"approved_by": { "type": ["string", "null"] },
|
||||
"review_status": { "type": "string" },
|
||||
"rationale": { "type": "string" },
|
||||
"created_at": { "type": "string" },
|
||||
"review_due_at": { "type": ["string", "null"] },
|
||||
"expires_at": { "type": ["string", "null"] },
|
||||
"native_exclusion_id": { "type": ["string", "null"] },
|
||||
"metadata": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
36
docs/schemas/export-manifest.schema.json
Normal file
36
docs/schemas/export-manifest.schema.json
Normal file
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Guide Board Export Manifest",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"schema_version",
|
||||
"export_type",
|
||||
"run_id",
|
||||
"created_at",
|
||||
"source_package_ref",
|
||||
"source_lock_ref",
|
||||
"summary",
|
||||
"policy_summary",
|
||||
"mapping_summary",
|
||||
"report_fragments",
|
||||
"counts",
|
||||
"certification_boundary"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"schema_version": { "type": "string" },
|
||||
"export_type": { "type": "string" },
|
||||
"run_id": { "type": "string" },
|
||||
"created_at": { "type": "string" },
|
||||
"source_package_ref": { "type": "string" },
|
||||
"source_lock_ref": { "type": "string" },
|
||||
"summary": { "type": "object" },
|
||||
"policy_summary": { "type": "object" },
|
||||
"mapping_summary": { "type": "object" },
|
||||
"report_fragments": { "type": "array", "items": { "type": "object" } },
|
||||
"counts": { "type": "object" },
|
||||
"certification_boundary": { "type": "string" }
|
||||
}
|
||||
}
|
||||
@@ -41,8 +41,38 @@
|
||||
"type": "string",
|
||||
"enum": ["candidate", "incubating", "active", "external", "deprecated"]
|
||||
},
|
||||
"supported_frameworks": { "type": "array", "items": { "type": "string" } },
|
||||
"authorities": { "type": "array", "items": { "type": "string" } },
|
||||
"supported_frameworks": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": ["string", "object"],
|
||||
"additionalProperties": false,
|
||||
"required": ["id"],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"version": { "type": ["string", "null"] },
|
||||
"source_url": { "type": ["string", "null"] },
|
||||
"authority_ref": { "type": ["string", "null"] },
|
||||
"description": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
},
|
||||
"authorities": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": ["string", "object"],
|
||||
"additionalProperties": false,
|
||||
"required": ["id"],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": ["string", "null"] },
|
||||
"version": { "type": ["string", "null"] },
|
||||
"source_url": { "type": ["string", "null"] },
|
||||
"license": { "type": ["string", "null"] },
|
||||
"access": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
},
|
||||
"metadata": { "type": "object" },
|
||||
"profile_schemas": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
@@ -89,13 +119,46 @@
|
||||
"module_path": { "type": ["string", "null"] },
|
||||
"callable": { "type": ["string", "null"] },
|
||||
"command": { "type": ["array", "null"], "items": { "type": "string" } },
|
||||
"metadata": { "type": "object" },
|
||||
"description": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
},
|
||||
"normalizers": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": ["string", "object"],
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "kind", "module_path", "callable"],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"kind": { "type": "string", "enum": ["python_module"] },
|
||||
"module_path": { "type": "string" },
|
||||
"callable": { "type": "string" },
|
||||
"runner_ref": { "type": ["string", "null"] },
|
||||
"metadata": { "type": "object" },
|
||||
"description": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
},
|
||||
"normalizers": { "type": "array", "items": { "type": "string" } },
|
||||
"mappings": { "type": "array", "items": { "type": "string" } },
|
||||
"report_fragments": { "type": "array", "items": { "type": "string" } },
|
||||
"report_fragments": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": ["string", "object"],
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "kind"],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"kind": { "type": "string", "enum": ["markdown_file", "python_module"] },
|
||||
"path": { "type": ["string", "null"] },
|
||||
"module_path": { "type": ["string", "null"] },
|
||||
"callable": { "type": ["string", "null"] },
|
||||
"title": { "type": ["string", "null"] },
|
||||
"description": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
},
|
||||
"dependencies": { "type": "array", "items": { "type": "string" } },
|
||||
"restricted_assets": { "type": "array", "items": { "type": "string" } },
|
||||
"certification_boundary": { "type": "string" }
|
||||
|
||||
@@ -14,7 +14,10 @@
|
||||
"evidence_refs",
|
||||
"expected",
|
||||
"waiver_ref",
|
||||
"challenge_ref",
|
||||
"exclusion_ref",
|
||||
"policy_ref",
|
||||
"review_status",
|
||||
"remediation"
|
||||
],
|
||||
"properties": {
|
||||
@@ -28,7 +31,10 @@
|
||||
"evidence_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"expected": { "type": "boolean" },
|
||||
"waiver_ref": { "type": ["string", "null"] },
|
||||
"challenge_ref": { "type": ["string", "null"] },
|
||||
"exclusion_ref": { "type": ["string", "null"] },
|
||||
"policy_ref": { "type": ["string", "null"] },
|
||||
"review_status": { "type": "string" },
|
||||
"remediation": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
"created_at": { "type": "string" },
|
||||
"summary": { "type": "object" },
|
||||
"report_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"export_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"artifact_retention": { "type": "object" }
|
||||
}
|
||||
}
|
||||
|
||||
34
docs/schemas/source-lock.schema.json
Normal file
34
docs/schemas/source-lock.schema.json
Normal file
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Guide Board Source Lock",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"schema_version",
|
||||
"created_at",
|
||||
"framework_refs",
|
||||
"extension_refs",
|
||||
"frameworks",
|
||||
"extensions",
|
||||
"mapping_sets",
|
||||
"profiles",
|
||||
"policy_refs",
|
||||
"authorities",
|
||||
"metadata_hooks"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"schema_version": { "type": "string" },
|
||||
"created_at": { "type": "string" },
|
||||
"framework_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"extension_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"frameworks": { "type": "array", "items": { "type": "object" } },
|
||||
"extensions": { "type": "array", "items": { "type": "object" } },
|
||||
"mapping_sets": { "type": "array", "items": { "type": "object" } },
|
||||
"profiles": { "type": "object" },
|
||||
"policy_refs": { "type": "object" },
|
||||
"authorities": { "type": "array", "items": { "type": "object" } },
|
||||
"metadata_hooks": { "type": "object" }
|
||||
}
|
||||
}
|
||||
36
docs/schemas/submission-package.schema.json
Normal file
36
docs/schemas/submission-package.schema.json
Normal file
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Guide Board Submission Package Manifest",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"schema_version",
|
||||
"run_id",
|
||||
"created_at",
|
||||
"package_identity",
|
||||
"source_lock_ref",
|
||||
"source_lock",
|
||||
"reports",
|
||||
"normalized_outputs",
|
||||
"profile_snapshots",
|
||||
"artifact_manifest",
|
||||
"reported_metadata",
|
||||
"certification_boundary"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"schema_version": { "type": "string" },
|
||||
"run_id": { "type": "string" },
|
||||
"created_at": { "type": "string" },
|
||||
"package_identity": { "type": "object" },
|
||||
"source_lock_ref": { "type": "string" },
|
||||
"source_lock": { "type": "object" },
|
||||
"reports": { "type": "array", "items": { "type": "object" } },
|
||||
"normalized_outputs": { "type": "array", "items": { "type": "object" } },
|
||||
"profile_snapshots": { "type": "array", "items": { "type": "object" } },
|
||||
"artifact_manifest": { "type": "array", "items": { "type": "object" } },
|
||||
"reported_metadata": { "type": "array", "items": { "type": "object" } },
|
||||
"certification_boundary": { "type": "string" }
|
||||
}
|
||||
}
|
||||
13
extensions/sdk-fixture/INTENT.md
Normal file
13
extensions/sdk-fixture/INTENT.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# SDK Fixture Extension
|
||||
|
||||
`sdk-fixture` is a dependency-light guide-board extension used to exercise the
|
||||
extension SDK contracts. It is intentionally small and is not a real assessment
|
||||
program.
|
||||
|
||||
The fixture demonstrates:
|
||||
|
||||
- extension-owned target and assessment profile schemas,
|
||||
- a Python runner that writes native output,
|
||||
- a Python normalizer that converts native output into guide-board evidence,
|
||||
- a mapping set for normalized requirement refs,
|
||||
- copyable profiles for SDK acceptance tests.
|
||||
91
extensions/sdk-fixture/extension.json
Normal file
91
extensions/sdk-fixture/extension.json
Normal file
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"id": "sdk-fixture",
|
||||
"name": "SDK Fixture Extension",
|
||||
"version": "0.1.0",
|
||||
"extension_type": "repository_quality",
|
||||
"lifecycle_status": "incubating",
|
||||
"supported_frameworks": [
|
||||
"guide-board.sdk-fixture.v1"
|
||||
],
|
||||
"authorities": [],
|
||||
"metadata": {
|
||||
"adapter_version": "0.1.0",
|
||||
"source_url": "https://example.invalid/guide-board/sdk-fixture"
|
||||
},
|
||||
"profile_schemas": [
|
||||
"target-profile",
|
||||
"assessment-profile",
|
||||
{
|
||||
"id": "sdk-fixture-target",
|
||||
"profile_kind": "target",
|
||||
"path": "schemas/sdk-fixture-target.schema.json",
|
||||
"subject_type": "sdk-fixture-target",
|
||||
"description": "Requires the target shape used by the SDK fixture runner."
|
||||
},
|
||||
{
|
||||
"id": "sdk-fixture-assessment",
|
||||
"profile_kind": "assessment",
|
||||
"path": "schemas/sdk-fixture-assessment.schema.json",
|
||||
"description": "Requires the runtime policy used by the SDK fixture normalizer test."
|
||||
}
|
||||
],
|
||||
"check_groups": [
|
||||
{
|
||||
"id": "native-output",
|
||||
"name": "Native Output Normalization",
|
||||
"check_type": "repository_quality",
|
||||
"requirement_refs": [
|
||||
"guide-board.sdk-fixture.v1.native-output"
|
||||
],
|
||||
"runner_ref": "native-probe"
|
||||
}
|
||||
],
|
||||
"preflight_runner": null,
|
||||
"runner_entrypoints": [
|
||||
{
|
||||
"id": "native-probe",
|
||||
"kind": "python_module",
|
||||
"module_path": "runners/native_probe.py",
|
||||
"callable": "run",
|
||||
"command": null,
|
||||
"metadata": {
|
||||
"harness_id": "sdk-fixture-native-probe",
|
||||
"harness_version": "1.0.0",
|
||||
"test_suite_id": "sdk-fixture-suite-v1",
|
||||
"source_url": "https://example.invalid/guide-board/sdk-fixture/native-probe"
|
||||
},
|
||||
"description": "Writes a tiny native result artifact for the SDK fixture normalizer."
|
||||
}
|
||||
],
|
||||
"normalizers": [
|
||||
{
|
||||
"id": "native-probe-normalizer",
|
||||
"kind": "python_module",
|
||||
"module_path": "normalizers/native_probe.py",
|
||||
"callable": "normalize",
|
||||
"runner_ref": "native-probe",
|
||||
"metadata": {
|
||||
"adapter_version": "0.1.0",
|
||||
"source_url": "https://example.invalid/guide-board/sdk-fixture/native-normalizer"
|
||||
},
|
||||
"description": "Converts the SDK fixture native result artifact into guide-board evidence."
|
||||
}
|
||||
],
|
||||
"mappings": [
|
||||
"sdk-fixture-map"
|
||||
],
|
||||
"report_fragments": [
|
||||
{
|
||||
"id": "sdk-fixture-summary",
|
||||
"kind": "python_module",
|
||||
"module_path": "reports/sdk_fixture_summary.py",
|
||||
"callable": "build_fragment",
|
||||
"path": null,
|
||||
"title": "SDK Fixture Summary",
|
||||
"description": "Summarizes SDK fixture evidence for report fragment tests."
|
||||
}
|
||||
],
|
||||
"dependencies": [],
|
||||
"restricted_assets": [],
|
||||
"certification_boundary": "SDK fixture only. It does not certify any product, process, or repository."
|
||||
}
|
||||
16
extensions/sdk-fixture/mappings/sdk-fixture-map.json
Normal file
16
extensions/sdk-fixture/mappings/sdk-fixture-map.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"id": "sdk-fixture-map",
|
||||
"extension_id": "sdk-fixture",
|
||||
"framework_refs": [
|
||||
"guide-board.sdk-fixture.v1"
|
||||
],
|
||||
"mappings": [
|
||||
{
|
||||
"requirement_ref": "guide-board.sdk-fixture.v1.native-output",
|
||||
"target_type": "sdk_contract",
|
||||
"target_id": "normalizer-plugin",
|
||||
"label": "Normalizer Plug-in Contract",
|
||||
"description": "The extension runner can emit native output that a normalizer converts into guide-board evidence."
|
||||
}
|
||||
]
|
||||
}
|
||||
32
extensions/sdk-fixture/normalizers/native_probe.py
Normal file
32
extensions/sdk-fixture/normalizers/native_probe.py
Normal file
@@ -0,0 +1,32 @@
|
||||
"""SDK fixture normalizer for native runner output."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def normalize(context: dict) -> dict:
|
||||
run_dir = Path(context["run_dir"])
|
||||
runner_result = context["runner_result"]
|
||||
artifact_ref = runner_result["facts"]["native_result_ref"]
|
||||
native_result = json.loads((run_dir / artifact_ref).read_text(encoding="utf-8"))
|
||||
native_status = native_result.get("native_status")
|
||||
result = "pass" if native_status == "ok" else "fail"
|
||||
return {
|
||||
"result": result,
|
||||
"observations": native_result.get("observations", []),
|
||||
"facts": {
|
||||
"native_status": native_status,
|
||||
"native_score": native_result.get("native_score"),
|
||||
"normalized_by": "native-probe-normalizer"
|
||||
},
|
||||
"artifact_refs": [
|
||||
artifact_ref
|
||||
],
|
||||
"requirement_refs": native_result.get("requirement_refs", []),
|
||||
"metadata": {
|
||||
"normalizer_id": "native-probe-normalizer",
|
||||
"native_result_id": "sdk-fixture-native-result"
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"id": "sdk-fixture-assessment",
|
||||
"framework_refs": [
|
||||
"guide-board.sdk-fixture.v1"
|
||||
],
|
||||
"extension_refs": [
|
||||
"sdk-fixture"
|
||||
],
|
||||
"target_profile_ref": "sdk-fixture-target",
|
||||
"selected_check_groups": {
|
||||
"sdk-fixture": [
|
||||
"native-output"
|
||||
]
|
||||
},
|
||||
"expectations_ref": null,
|
||||
"waivers_ref": null,
|
||||
"output_policy": {
|
||||
"report_formats": [
|
||||
"json",
|
||||
"markdown"
|
||||
],
|
||||
"artifact_retention": "raw-logs-plus-summary"
|
||||
},
|
||||
"retention_policy": {
|
||||
"summary_days": 365,
|
||||
"raw_artifact_days": 30
|
||||
},
|
||||
"runtime_policy": {
|
||||
"offline": true,
|
||||
"timeout_seconds": 30,
|
||||
"fixture_mode": "native-result"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"id": "sdk-fixture-target",
|
||||
"subject_type": "sdk-fixture-target",
|
||||
"subject_name": "SDK Fixture Target",
|
||||
"environment": "test",
|
||||
"scope": [
|
||||
"Extension SDK fixture validation"
|
||||
],
|
||||
"endpoints": [],
|
||||
"artifacts": [
|
||||
"extension.json"
|
||||
],
|
||||
"credentials_ref": null,
|
||||
"declared_capabilities": [
|
||||
"guide-board.sdk-fixture.v1.native-output"
|
||||
],
|
||||
"known_gaps": []
|
||||
}
|
||||
26
extensions/sdk-fixture/reports/sdk_fixture_summary.py
Normal file
26
extensions/sdk-fixture/reports/sdk_fixture_summary.py
Normal file
@@ -0,0 +1,26 @@
|
||||
"""Report fragment for the SDK fixture extension."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def build_fragment(context: dict) -> dict:
|
||||
evidence_count = len(context["evidence"])
|
||||
finding_count = len(context["findings"])
|
||||
source_lock = context["source_lock"]
|
||||
markdown = "\n".join(
|
||||
[
|
||||
"### SDK Fixture Summary",
|
||||
"",
|
||||
f"- evidence items: {evidence_count}",
|
||||
f"- findings: {finding_count}",
|
||||
f"- source lock: {source_lock.get('id')}",
|
||||
]
|
||||
)
|
||||
return {
|
||||
"markdown": markdown,
|
||||
"structured": {
|
||||
"evidence_count": evidence_count,
|
||||
"finding_count": finding_count,
|
||||
"source_lock_ref": source_lock.get("id"),
|
||||
},
|
||||
}
|
||||
40
extensions/sdk-fixture/runners/native_probe.py
Normal file
40
extensions/sdk-fixture/runners/native_probe.py
Normal file
@@ -0,0 +1,40 @@
|
||||
"""SDK fixture runner that writes a native result artifact."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def run(context: dict) -> dict:
|
||||
run_dir = Path(context["run_dir"])
|
||||
artifact_path = run_dir / "artifacts" / "sdk-fixture" / "native-result.json"
|
||||
artifact_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
native_result = {
|
||||
"native_status": "ok",
|
||||
"native_score": 98,
|
||||
"observations": [
|
||||
"SDK fixture native probe completed."
|
||||
],
|
||||
"requirement_refs": [
|
||||
"guide-board.sdk-fixture.v1.native-output"
|
||||
],
|
||||
}
|
||||
artifact_path.write_text(json.dumps(native_result, indent=2, sort_keys=True), encoding="utf-8")
|
||||
artifact_ref = str(artifact_path.relative_to(run_dir))
|
||||
return {
|
||||
"result": "unknown",
|
||||
"observations": [
|
||||
"SDK fixture runner wrote native output for normalization."
|
||||
],
|
||||
"facts": {
|
||||
"native_result_ref": artifact_ref
|
||||
},
|
||||
"artifact_refs": [
|
||||
artifact_ref
|
||||
],
|
||||
"metadata": {
|
||||
"native_result_id": "sdk-fixture-native-result",
|
||||
"test_suite_id": "sdk-fixture-suite-v1"
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "SDK Fixture Assessment Profile",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"runtime_policy"
|
||||
],
|
||||
"properties": {
|
||||
"runtime_policy": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"fixture_mode"
|
||||
],
|
||||
"properties": {
|
||||
"fixture_mode": { "enum": ["native-result"] }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "SDK Fixture Target Profile",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"subject_type",
|
||||
"artifacts",
|
||||
"declared_capabilities"
|
||||
],
|
||||
"properties": {
|
||||
"subject_type": { "enum": ["sdk-fixture-target"] },
|
||||
"artifacts": { "type": "array", "minItems": 1 },
|
||||
"declared_capabilities": { "type": "array", "minItems": 1 }
|
||||
}
|
||||
}
|
||||
12
registry/README.md
Normal file
12
registry/README.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Capability Registry
|
||||
|
||||
Markdown-first capability index for federation and reuse planning.
|
||||
|
||||
## Authoring
|
||||
|
||||
1. Copy a capability entry template (see reuse-surface `templates/capability-entry.template.md`).
|
||||
2. Add the row to `indexes/capabilities.yaml`.
|
||||
3. Run `reuse-surface validate` from a checkout with the CLI installed.
|
||||
4. Merge to `main` and verify publish with `reuse-surface establish --publish-check`.
|
||||
|
||||
Federation contract: reuse-surface `docs/RegistryFederation.md`.
|
||||
0
registry/capabilities/.gitkeep
Normal file
0
registry/capabilities/.gitkeep
Normal file
4
registry/indexes/capabilities.yaml
Normal file
4
registry/indexes/capabilities.yaml
Normal file
@@ -0,0 +1,4 @@
|
||||
version: 1
|
||||
updated: '2026-06-16'
|
||||
domain: helix_forge
|
||||
capabilities: []
|
||||
@@ -35,12 +35,16 @@ echo "==> Verifying mounted run artifacts"
|
||||
for path in \
|
||||
"$RUNS_DIR/sample-noop/run.json" \
|
||||
"$RUNS_DIR/sample-noop/plan.json" \
|
||||
"$RUNS_DIR/sample-noop/sources.lock.json" \
|
||||
"$RUNS_DIR/sample-noop/retention-summary.json" \
|
||||
"$RUNS_DIR/sample-noop/normalized/evidence.json" \
|
||||
"$RUNS_DIR/sample-noop/normalized/findings.json" \
|
||||
"$RUNS_DIR/sample-noop/normalized/mappings.json" \
|
||||
"$RUNS_DIR/sample-noop/reports/assessment-package.json" \
|
||||
"$RUNS_DIR/sample-noop/reports/report.md"
|
||||
"$RUNS_DIR/sample-noop/reports/report.md" \
|
||||
"$RUNS_DIR/sample-noop/reports/fragments.json" \
|
||||
"$RUNS_DIR/sample-noop/reports/submission-package.json" \
|
||||
"$RUNS_DIR/sample-noop/exports/export-manifest.json"
|
||||
do
|
||||
if [ ! -f "$path" ]; then
|
||||
echo "ERROR: expected artifact missing: $path" >&2
|
||||
|
||||
@@ -46,6 +46,107 @@ def build_artifact_manifest(
|
||||
return artifacts
|
||||
|
||||
|
||||
def build_submission_manifest(
|
||||
run_dir: Path,
|
||||
run_metadata: dict[str, Any],
|
||||
plan: dict[str, Any],
|
||||
evidence: list[dict[str, Any]],
|
||||
assessment_package: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
"""Build a portable manifest for the files that make up a review package."""
|
||||
source_lock = plan["source_lock"]
|
||||
manifest = {
|
||||
"id": f"submission-package:{run_metadata['id']}",
|
||||
"schema_version": "guide-board.submission-package.v1",
|
||||
"run_id": run_metadata["id"],
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
"package_identity": {
|
||||
"target_profile_ref": run_metadata["target_profile_ref"],
|
||||
"assessment_profile_ref": run_metadata["assessment_profile_ref"],
|
||||
"framework_refs": source_lock.get("framework_refs", []),
|
||||
"extension_refs": source_lock.get("extension_refs", []),
|
||||
},
|
||||
"source_lock_ref": "sources.lock.json",
|
||||
"source_lock": {
|
||||
"id": source_lock.get("id"),
|
||||
"schema_version": source_lock.get("schema_version"),
|
||||
"checksum": _file_entry(run_dir, "sources.lock.json").get("checksum"),
|
||||
"framework_refs": source_lock.get("framework_refs", []),
|
||||
"extension_refs": source_lock.get("extension_refs", []),
|
||||
},
|
||||
"reports": _existing_file_entries(
|
||||
run_dir,
|
||||
[
|
||||
("assessment-package", "reports/assessment-package.json"),
|
||||
("markdown-report", "reports/report.md"),
|
||||
],
|
||||
),
|
||||
"normalized_outputs": _existing_file_entries(
|
||||
run_dir,
|
||||
[
|
||||
("evidence", "normalized/evidence.json"),
|
||||
("findings", "normalized/findings.json"),
|
||||
("mappings", "normalized/mappings.json"),
|
||||
],
|
||||
),
|
||||
"profile_snapshots": _existing_file_entries(
|
||||
run_dir,
|
||||
[
|
||||
("target-profile", "target-profile.snapshot.json"),
|
||||
("assessment-profile", "assessment-profile.snapshot.json"),
|
||||
],
|
||||
),
|
||||
"artifact_manifest": assessment_package.get("artifact_manifest", []),
|
||||
"reported_metadata": _reported_metadata(evidence),
|
||||
"certification_boundary": assessment_package["certification_boundary"],
|
||||
}
|
||||
assert_valid(manifest, "submission-package")
|
||||
return manifest
|
||||
|
||||
|
||||
def _existing_file_entries(run_dir: Path, refs: list[tuple[str, str]]) -> list[dict[str, Any]]:
|
||||
entries = []
|
||||
for entry_id, ref in refs:
|
||||
entry = _file_entry(run_dir, ref)
|
||||
if entry:
|
||||
entry["id"] = entry_id
|
||||
entries.append(entry)
|
||||
return entries
|
||||
|
||||
|
||||
def _file_entry(run_dir: Path, ref: str) -> dict[str, Any]:
|
||||
path = (run_dir / ref).resolve()
|
||||
try:
|
||||
path.relative_to(run_dir.resolve())
|
||||
except ValueError:
|
||||
return {}
|
||||
if not path.is_file():
|
||||
return {}
|
||||
return {
|
||||
"path": ref,
|
||||
"media_type": _media_type(path),
|
||||
"checksum": f"sha256:{_sha256(path)}",
|
||||
"size_bytes": path.stat().st_size,
|
||||
}
|
||||
|
||||
|
||||
def _reported_metadata(evidence: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
records = []
|
||||
for item in evidence:
|
||||
metadata = item.get("facts", {}).get("source_metadata")
|
||||
if not isinstance(metadata, dict) or not metadata:
|
||||
continue
|
||||
records.append(
|
||||
{
|
||||
"evidence_ref": item["id"],
|
||||
"check_id": item["check_id"],
|
||||
"extension_id": item["extension_id"],
|
||||
"metadata": metadata,
|
||||
}
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
|
||||
@@ -7,11 +7,14 @@ from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from guide_board.artifacts import build_artifact_manifest
|
||||
from guide_board.io import write_json
|
||||
from guide_board.artifacts import build_artifact_manifest, build_submission_manifest
|
||||
from guide_board.exports import build_export_manifest
|
||||
from guide_board.io import load_json, write_json
|
||||
from guide_board.mapping import build_mapping_records, summarize_mappings
|
||||
from guide_board.normalizers import normalize_step_result
|
||||
from guide_board.planning import build_run_plan
|
||||
from guide_board.policy import apply_policy
|
||||
from guide_board.reports import build_report_fragments, markdown_for_fragments
|
||||
from guide_board.retention import build_retention_summary
|
||||
from guide_board.runners import run_step
|
||||
from guide_board.schema import assert_valid
|
||||
@@ -34,7 +37,15 @@ def run_assessment(
|
||||
assert_valid(item, "evidence-item")
|
||||
|
||||
findings = _findings_for_evidence(run_id, evidence)
|
||||
findings, policy_summary, applied_waivers = apply_policy(root, plan, findings)
|
||||
(
|
||||
findings,
|
||||
policy_summary,
|
||||
applied_waivers,
|
||||
applied_challenges,
|
||||
applied_exclusions,
|
||||
) = apply_policy(root, plan, evidence, findings)
|
||||
for item in evidence:
|
||||
assert_valid(item, "evidence-item")
|
||||
for finding in findings:
|
||||
assert_valid(finding, "finding")
|
||||
|
||||
@@ -51,9 +62,23 @@ def run_assessment(
|
||||
mapping_summary,
|
||||
policy_summary,
|
||||
applied_waivers,
|
||||
applied_challenges,
|
||||
applied_exclusions,
|
||||
created_at,
|
||||
)
|
||||
report_fragments = build_report_fragments(
|
||||
root,
|
||||
run_dir,
|
||||
run_id,
|
||||
plan,
|
||||
evidence,
|
||||
findings,
|
||||
mapping_records,
|
||||
assessment_package,
|
||||
)
|
||||
assessment_package["report_fragments"] = report_fragments
|
||||
assert_valid(assessment_package, "assessment-package")
|
||||
export_manifest = build_export_manifest(assessment_package)
|
||||
|
||||
run_metadata = {
|
||||
"id": run_id,
|
||||
@@ -74,6 +99,7 @@ def run_assessment(
|
||||
findings,
|
||||
mapping_records,
|
||||
assessment_package,
|
||||
export_manifest,
|
||||
retention_summary,
|
||||
)
|
||||
return {
|
||||
@@ -82,6 +108,8 @@ def run_assessment(
|
||||
"run_dir": str(run_dir),
|
||||
"assessment_package": str(run_dir / "reports" / "assessment-package.json"),
|
||||
"report": str(run_dir / "reports" / "report.md"),
|
||||
"submission_package": str(run_dir / "reports" / "submission-package.json"),
|
||||
"export_manifest": str(run_dir / "exports" / "export-manifest.json"),
|
||||
"retention_summary": str(run_dir / "retention-summary.json"),
|
||||
}
|
||||
|
||||
@@ -153,6 +181,15 @@ def _evidence_for_step(
|
||||
now = _now()
|
||||
runner_ref = step.get("runner_ref")
|
||||
runner_result = run_step(root, run_dir, run_id, plan, step)
|
||||
runner_result = normalize_step_result(root, run_dir, run_id, plan, step, runner_result)
|
||||
facts = {
|
||||
"step_kind": step["kind"],
|
||||
"runner_ref": runner_ref,
|
||||
**runner_result["facts"],
|
||||
}
|
||||
source_metadata = _source_metadata_for_step(root, plan, step, runner_result)
|
||||
if source_metadata:
|
||||
facts["source_metadata"] = source_metadata
|
||||
|
||||
return {
|
||||
"id": f"evidence:{step['id']}",
|
||||
@@ -162,22 +199,134 @@ def _evidence_for_step(
|
||||
"subject_ref": plan["target_profile_snapshot"]["id"],
|
||||
"result": runner_result["result"],
|
||||
"observations": runner_result["observations"],
|
||||
"facts": {
|
||||
"step_kind": step["kind"],
|
||||
"runner_ref": runner_ref,
|
||||
**runner_result["facts"],
|
||||
},
|
||||
"requirement_refs": _requirement_refs(plan, step),
|
||||
"facts": facts,
|
||||
"requirement_refs": _requirement_refs(plan, step, runner_result),
|
||||
"artifact_refs": runner_result["artifact_refs"],
|
||||
"started_at": now,
|
||||
"completed_at": now,
|
||||
}
|
||||
|
||||
|
||||
def _requirement_refs(plan: dict[str, Any], step: dict[str, Any]) -> list[str]:
|
||||
def _requirement_refs(
|
||||
plan: dict[str, Any],
|
||||
step: dict[str, Any],
|
||||
runner_result: dict[str, Any] | None = None,
|
||||
) -> list[str]:
|
||||
refs = []
|
||||
if step["kind"] != "check_group":
|
||||
return _runner_requirement_refs(runner_result)
|
||||
refs.extend(step.get("requirement_refs", []))
|
||||
refs.extend(_runner_requirement_refs(runner_result))
|
||||
return _dedupe(refs)
|
||||
|
||||
|
||||
def _runner_requirement_refs(runner_result: dict[str, Any] | None) -> list[str]:
|
||||
if not runner_result:
|
||||
return []
|
||||
return list(step.get("requirement_refs", []))
|
||||
refs = runner_result.get("requirement_refs", [])
|
||||
if not isinstance(refs, list):
|
||||
return []
|
||||
return [ref for ref in refs if isinstance(ref, str)]
|
||||
|
||||
|
||||
def _source_metadata_for_step(
|
||||
root: Path,
|
||||
plan: dict[str, Any],
|
||||
step: dict[str, Any],
|
||||
runner_result: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
runner_ref = step.get("runner_ref")
|
||||
extension = _extension_snapshot(plan, step["extension_id"])
|
||||
extension_path = _snapshot_path(root, extension)
|
||||
manifest = load_json(extension_path / "extension.json")
|
||||
metadata: dict[str, Any] = {
|
||||
"extension": {
|
||||
"id": step["extension_id"],
|
||||
"version": extension.get("version"),
|
||||
"metadata": _object_or_empty(manifest.get("metadata")),
|
||||
}
|
||||
}
|
||||
|
||||
if runner_ref:
|
||||
entrypoint = _runner_entrypoint(manifest, runner_ref)
|
||||
metadata["runner"] = {
|
||||
"id": runner_ref,
|
||||
"kind": entrypoint.get("kind"),
|
||||
"metadata": _object_or_empty(entrypoint.get("metadata")),
|
||||
}
|
||||
|
||||
applied_normalizers = runner_result.get("facts", {}).get("normalizer_refs", [])
|
||||
normalizers = []
|
||||
if isinstance(applied_normalizers, list):
|
||||
normalizer_ids = {item for item in applied_normalizers if isinstance(item, str)}
|
||||
for normalizer in manifest.get("normalizers", []):
|
||||
if isinstance(normalizer, dict) and normalizer.get("id") in normalizer_ids:
|
||||
normalizers.append(
|
||||
{
|
||||
"id": normalizer["id"],
|
||||
"kind": normalizer.get("kind"),
|
||||
"runner_ref": normalizer.get("runner_ref"),
|
||||
"metadata": _object_or_empty(normalizer.get("metadata")),
|
||||
}
|
||||
)
|
||||
if normalizers:
|
||||
metadata["normalizers"] = normalizers
|
||||
|
||||
reported = _object_or_empty(runner_result.get("metadata"))
|
||||
if reported:
|
||||
metadata["reported"] = reported
|
||||
|
||||
return _drop_empty_metadata(metadata)
|
||||
|
||||
|
||||
def _extension_snapshot(plan: dict[str, Any], extension_id: str) -> dict[str, Any]:
|
||||
for extension in plan["extension_snapshots"]:
|
||||
if extension["id"] == extension_id:
|
||||
return extension
|
||||
return {}
|
||||
|
||||
|
||||
def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path:
|
||||
path = Path(extension["path"])
|
||||
return path if path.is_absolute() else root / path
|
||||
|
||||
|
||||
def _runner_entrypoint(manifest: dict[str, Any], runner_ref: str) -> dict[str, Any]:
|
||||
for entrypoint in manifest.get("runner_entrypoints", []):
|
||||
if entrypoint.get("id") == runner_ref:
|
||||
return entrypoint
|
||||
return {}
|
||||
|
||||
|
||||
def _object_or_empty(value: Any) -> dict[str, Any]:
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def _drop_empty_metadata(value: dict[str, Any]) -> dict[str, Any]:
|
||||
compact = {}
|
||||
for key, child in value.items():
|
||||
if isinstance(child, dict):
|
||||
child = _drop_empty_metadata(child)
|
||||
if isinstance(child, list):
|
||||
child = [
|
||||
_drop_empty_metadata(item) if isinstance(item, dict) else item
|
||||
for item in child
|
||||
]
|
||||
child = [item for item in child if item]
|
||||
if child:
|
||||
compact[key] = child
|
||||
return compact
|
||||
|
||||
|
||||
def _dedupe(values: list[str]) -> list[str]:
|
||||
seen = set()
|
||||
deduped = []
|
||||
for value in values:
|
||||
if value in seen:
|
||||
continue
|
||||
seen.add(value)
|
||||
deduped.append(value)
|
||||
return deduped
|
||||
|
||||
|
||||
def _findings_for_evidence(run_id: str, evidence: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
@@ -185,6 +334,7 @@ def _findings_for_evidence(run_id: str, evidence: list[dict[str, Any]]) -> list[
|
||||
for item in evidence:
|
||||
if item["result"] not in {"blocked", "fail", "infrastructure_error"}:
|
||||
continue
|
||||
expected = _expected_for_item(item)
|
||||
findings.append(
|
||||
{
|
||||
"id": f"finding:{item['check_id']}",
|
||||
@@ -195,9 +345,12 @@ def _findings_for_evidence(run_id: str, evidence: list[dict[str, Any]]) -> list[
|
||||
"classification": _classification_for_item(item),
|
||||
"requirement_refs": item["requirement_refs"],
|
||||
"evidence_refs": [item["id"]],
|
||||
"expected": _expected_for_item(item),
|
||||
"expected": expected,
|
||||
"waiver_ref": None,
|
||||
"challenge_ref": None,
|
||||
"exclusion_ref": None,
|
||||
"policy_ref": None,
|
||||
"review_status": "expected" if expected else "unresolved_defect",
|
||||
"remediation": _remediation_for_item(item),
|
||||
}
|
||||
)
|
||||
@@ -259,6 +412,8 @@ def _assessment_package(
|
||||
mapping_summary: dict[str, Any],
|
||||
policy_summary: dict[str, Any],
|
||||
applied_waivers: list[dict[str, Any]],
|
||||
applied_challenges: list[dict[str, Any]],
|
||||
applied_exclusions: list[dict[str, Any]],
|
||||
created_at: str,
|
||||
) -> dict[str, Any]:
|
||||
summary = dict(Counter(item["result"] for item in evidence))
|
||||
@@ -278,6 +433,9 @@ def _assessment_package(
|
||||
"evidence_refs": [item["id"] for item in evidence],
|
||||
"artifact_manifest": artifact_manifest,
|
||||
"waivers": applied_waivers,
|
||||
"challenges": applied_challenges,
|
||||
"exclusions": applied_exclusions,
|
||||
"report_fragments": [],
|
||||
"certification_boundary": "Guide Board produces preparation evidence only and does not issue certifications or audit assurance.",
|
||||
"created_at": created_at,
|
||||
}
|
||||
@@ -291,6 +449,7 @@ def _write_run_directory(
|
||||
findings: list[dict[str, Any]],
|
||||
mapping_records: list[dict[str, Any]],
|
||||
assessment_package: dict[str, Any],
|
||||
export_manifest: dict[str, Any],
|
||||
retention_summary: dict[str, Any],
|
||||
) -> None:
|
||||
write_json(run_dir / "run.json", run_metadata)
|
||||
@@ -306,11 +465,21 @@ def _write_run_directory(
|
||||
write_json(run_dir / "normalized" / "findings.json", {"findings": findings})
|
||||
write_json(run_dir / "normalized" / "mappings.json", {"mappings": mapping_records})
|
||||
write_json(run_dir / "reports" / "assessment-package.json", assessment_package)
|
||||
write_json(run_dir / "reports" / "fragments.json", {"fragments": assessment_package["report_fragments"]})
|
||||
write_json(run_dir / "exports" / "export-manifest.json", export_manifest)
|
||||
(run_dir / "reports").mkdir(parents=True, exist_ok=True)
|
||||
(run_dir / "reports" / "report.md").write_text(
|
||||
_markdown_report(run_metadata, assessment_package),
|
||||
encoding="utf-8",
|
||||
)
|
||||
submission_manifest = build_submission_manifest(
|
||||
run_dir,
|
||||
run_metadata,
|
||||
plan,
|
||||
evidence,
|
||||
assessment_package,
|
||||
)
|
||||
write_json(run_dir / "reports" / "submission-package.json", submission_manifest)
|
||||
|
||||
|
||||
def _markdown_report(run_metadata: dict[str, Any], package: dict[str, Any]) -> str:
|
||||
@@ -321,6 +490,8 @@ def _markdown_report(run_metadata: dict[str, Any], package: dict[str, Any]) -> s
|
||||
summary_lines = "- no evidence produced"
|
||||
mapping_lines = _mapping_summary_lines(package)
|
||||
policy_lines = _policy_summary_lines(package)
|
||||
review_lines = _review_summary_lines(package)
|
||||
fragment_lines = markdown_for_fragments(package.get("report_fragments", []))
|
||||
|
||||
return "\n".join(
|
||||
[
|
||||
@@ -342,6 +513,14 @@ def _markdown_report(run_metadata: dict[str, Any], package: dict[str, Any]) -> s
|
||||
"",
|
||||
policy_lines,
|
||||
"",
|
||||
"## Review",
|
||||
"",
|
||||
review_lines,
|
||||
"",
|
||||
"## Extension Fragments",
|
||||
"",
|
||||
fragment_lines,
|
||||
"",
|
||||
"## Boundary",
|
||||
"",
|
||||
package["certification_boundary"],
|
||||
@@ -371,10 +550,27 @@ def _policy_summary_lines(package: dict[str, Any]) -> str:
|
||||
f"- applied expectations: {summary.get('applied_expectations', 0)}",
|
||||
f"- applied waivers: {summary.get('applied_waivers', 0)}",
|
||||
f"- unexpected findings: {summary.get('unexpected_findings', 0)}",
|
||||
f"- challenged findings: {summary.get('challenged_findings', 0)}",
|
||||
f"- authority exclusions: {summary.get('authority_exclusions', 0)}",
|
||||
f"- unresolved defects: {summary.get('unresolved_defects', 0)}",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _review_summary_lines(package: dict[str, Any]) -> str:
|
||||
findings = package.get("findings", [])
|
||||
if not findings:
|
||||
return "- no findings requiring review"
|
||||
counts = Counter(
|
||||
finding.get("review_status", "unreviewed")
|
||||
for finding in findings
|
||||
if isinstance(finding, dict)
|
||||
)
|
||||
return "\n".join(
|
||||
f"- {status}: {count}" for status, count in sorted(counts.items())
|
||||
)
|
||||
|
||||
|
||||
def _run_status(evidence: list[dict[str, Any]]) -> str:
|
||||
if any(item["result"] == "fail" for item in evidence):
|
||||
return "failed"
|
||||
|
||||
50
src/guide_board/exports.py
Normal file
50
src/guide_board/exports.py
Normal file
@@ -0,0 +1,50 @@
|
||||
"""Portable export builders derived from assessment packages."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from guide_board.schema import assert_valid
|
||||
|
||||
|
||||
def build_export_manifest(assessment_package: dict[str, Any]) -> dict[str, Any]:
|
||||
manifest = {
|
||||
"id": f"export-manifest:{assessment_package['run_id']}",
|
||||
"schema_version": "guide-board.export-manifest.v1",
|
||||
"export_type": "guide-board.generic-json.v1",
|
||||
"run_id": assessment_package["run_id"],
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
"source_package_ref": "reports/assessment-package.json",
|
||||
"source_lock_ref": "sources.lock.json",
|
||||
"summary": assessment_package.get("summary", {}),
|
||||
"policy_summary": assessment_package.get("policy_summary", {}),
|
||||
"mapping_summary": assessment_package.get("mapping_summary", {}),
|
||||
"report_fragments": [
|
||||
_export_fragment(fragment)
|
||||
for fragment in assessment_package.get("report_fragments", [])
|
||||
if isinstance(fragment, dict)
|
||||
],
|
||||
"counts": {
|
||||
"evidence_refs": len(assessment_package.get("evidence_refs", [])),
|
||||
"findings": len(assessment_package.get("findings", [])),
|
||||
"artifacts": len(assessment_package.get("artifact_manifest", [])),
|
||||
"waivers": len(assessment_package.get("waivers", [])),
|
||||
"challenges": len(assessment_package.get("challenges", [])),
|
||||
"exclusions": len(assessment_package.get("exclusions", [])),
|
||||
"report_fragments": len(assessment_package.get("report_fragments", [])),
|
||||
},
|
||||
"certification_boundary": assessment_package["certification_boundary"],
|
||||
}
|
||||
assert_valid(manifest, "export-manifest")
|
||||
return manifest
|
||||
|
||||
|
||||
def _export_fragment(fragment: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"id": fragment.get("id"),
|
||||
"extension_id": fragment.get("extension_id"),
|
||||
"title": fragment.get("title"),
|
||||
"kind": fragment.get("kind"),
|
||||
"structured": fragment.get("structured", {}),
|
||||
}
|
||||
236
src/guide_board/normalizers.py
Normal file
236
src/guide_board/normalizers.py
Normal file
@@ -0,0 +1,236 @@
|
||||
"""Normalizer plug-in bridge for extension-provided runner output."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import Any
|
||||
|
||||
from guide_board.errors import ValidationError
|
||||
from guide_board.io import load_json
|
||||
|
||||
|
||||
def normalize_step_result(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
step: dict[str, Any],
|
||||
runner_result: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
"""Apply matching extension normalizers to a runner result."""
|
||||
extension = _extension_snapshot(plan, step["extension_id"])
|
||||
extension_path = _snapshot_path(root, extension)
|
||||
manifest = load_json(extension_path / "extension.json")
|
||||
result = _coerce_result(runner_result)
|
||||
applied: list[str] = []
|
||||
|
||||
for normalizer in _matching_normalizers(manifest, step):
|
||||
normalized = _run_normalizer(
|
||||
root,
|
||||
run_dir,
|
||||
run_id,
|
||||
plan,
|
||||
step,
|
||||
extension_path,
|
||||
normalizer,
|
||||
result,
|
||||
)
|
||||
if _is_normalizer_error(normalized):
|
||||
return normalized
|
||||
result = _merge_result(result, normalized)
|
||||
applied.append(normalizer["id"])
|
||||
|
||||
if applied:
|
||||
facts = dict(result.get("facts", {}))
|
||||
facts["normalizer_refs"] = applied
|
||||
result["facts"] = facts
|
||||
return result
|
||||
|
||||
|
||||
def _matching_normalizers(
|
||||
manifest: dict[str, Any],
|
||||
step: dict[str, Any],
|
||||
) -> list[dict[str, Any]]:
|
||||
matching = []
|
||||
runner_ref = step.get("runner_ref")
|
||||
for normalizer in manifest.get("normalizers", []):
|
||||
if not isinstance(normalizer, dict):
|
||||
continue
|
||||
normalizer_runner_ref = normalizer.get("runner_ref")
|
||||
if normalizer_runner_ref and normalizer_runner_ref != runner_ref:
|
||||
continue
|
||||
matching.append(normalizer)
|
||||
return matching
|
||||
|
||||
|
||||
def _run_normalizer(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
step: dict[str, Any],
|
||||
extension_path: Path,
|
||||
normalizer: dict[str, Any],
|
||||
runner_result: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
if normalizer["kind"] != "python_module":
|
||||
raise ValidationError(
|
||||
f"{normalizer['id']}: unsupported normalizer kind {normalizer['kind']!r}"
|
||||
)
|
||||
|
||||
module_path = normalizer.get("module_path")
|
||||
callable_name = normalizer.get("callable")
|
||||
if not module_path or not callable_name:
|
||||
raise ValidationError(
|
||||
f"{normalizer['id']}: python_module normalizers need module_path and callable"
|
||||
)
|
||||
|
||||
module_file = (extension_path / module_path).resolve()
|
||||
try:
|
||||
module_file.relative_to(extension_path.resolve())
|
||||
except ValueError as exc:
|
||||
raise ValidationError(
|
||||
f"{normalizer['id']}: module_path must stay inside the extension directory"
|
||||
) from exc
|
||||
|
||||
module = _load_module(module_file, normalizer["id"])
|
||||
normalizer_callable = getattr(module, callable_name, None)
|
||||
if not callable(normalizer_callable):
|
||||
raise ValidationError(f"{normalizer['id']}: callable {callable_name!r} was not found")
|
||||
|
||||
context = {
|
||||
"root": str(root),
|
||||
"run_dir": str(run_dir),
|
||||
"run_id": run_id,
|
||||
"plan": plan,
|
||||
"step": step,
|
||||
"target_profile": plan["target_profile_snapshot"],
|
||||
"assessment_profile": plan["assessment_profile_snapshot"],
|
||||
"extension_path": str(extension_path),
|
||||
"normalizer": normalizer,
|
||||
"runner_result": runner_result,
|
||||
}
|
||||
try:
|
||||
result = normalizer_callable(context)
|
||||
except Exception as exc: # noqa: BLE001 - extension failures become evidence.
|
||||
return {
|
||||
"result": "infrastructure_error",
|
||||
"observations": [
|
||||
f"Normalizer {normalizer['id']!r} failed before producing evidence: {exc}"
|
||||
],
|
||||
"facts": {
|
||||
"normalizer_ref": normalizer["id"],
|
||||
"normalizer_kind": normalizer["kind"],
|
||||
"error_type": type(exc).__name__,
|
||||
},
|
||||
"artifact_refs": runner_result.get("artifact_refs", []),
|
||||
"requirement_refs": runner_result.get("requirement_refs", []),
|
||||
"metadata": runner_result.get("metadata", {}),
|
||||
}
|
||||
|
||||
if not isinstance(result, dict):
|
||||
raise ValidationError(f"{normalizer['id']}: normalizer must return an object")
|
||||
return result
|
||||
|
||||
|
||||
def _merge_result(
|
||||
base: dict[str, Any],
|
||||
update: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
merged = dict(base)
|
||||
if "result" in update:
|
||||
merged["result"] = update["result"]
|
||||
if "observations" in update:
|
||||
merged["observations"] = _string_list(base.get("observations", []))
|
||||
merged["observations"].extend(_string_list(update.get("observations", [])))
|
||||
if "facts" in update:
|
||||
facts = dict(base.get("facts", {}))
|
||||
update_facts = update.get("facts", {})
|
||||
if isinstance(update_facts, dict):
|
||||
facts.update(update_facts)
|
||||
merged["facts"] = facts
|
||||
if "artifact_refs" in update:
|
||||
merged["artifact_refs"] = _dedupe(
|
||||
_string_list(base.get("artifact_refs", []))
|
||||
+ _string_list(update.get("artifact_refs", []))
|
||||
)
|
||||
if "requirement_refs" in update:
|
||||
merged["requirement_refs"] = _dedupe(
|
||||
_string_list(base.get("requirement_refs", []))
|
||||
+ _string_list(update.get("requirement_refs", []))
|
||||
)
|
||||
if "metadata" in update:
|
||||
metadata = dict(base.get("metadata", {}))
|
||||
update_metadata = update.get("metadata", {})
|
||||
if isinstance(update_metadata, dict):
|
||||
metadata.update(update_metadata)
|
||||
merged["metadata"] = metadata
|
||||
return _coerce_result(merged)
|
||||
|
||||
|
||||
def _coerce_result(value: dict[str, Any]) -> dict[str, Any]:
|
||||
facts = value.get("facts", {})
|
||||
if not isinstance(facts, dict):
|
||||
facts = {}
|
||||
return {
|
||||
"result": value.get("result", "unknown"),
|
||||
"observations": _string_list(value.get("observations", [])),
|
||||
"facts": facts,
|
||||
"artifact_refs": _string_list(value.get("artifact_refs", [])),
|
||||
"requirement_refs": _string_list(value.get("requirement_refs", [])),
|
||||
"metadata": _object_or_empty(value.get("metadata")),
|
||||
}
|
||||
|
||||
|
||||
def _is_normalizer_error(result: dict[str, Any]) -> bool:
|
||||
return (
|
||||
result.get("result") == "infrastructure_error"
|
||||
and "normalizer_ref" in result.get("facts", {})
|
||||
)
|
||||
|
||||
|
||||
def _string_list(value: Any) -> list[str]:
|
||||
if not isinstance(value, list):
|
||||
return []
|
||||
return [item for item in value if isinstance(item, str)]
|
||||
|
||||
|
||||
def _object_or_empty(value: Any) -> dict[str, Any]:
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def _dedupe(values: list[str]) -> list[str]:
|
||||
seen = set()
|
||||
deduped = []
|
||||
for value in values:
|
||||
if value in seen:
|
||||
continue
|
||||
seen.add(value)
|
||||
deduped.append(value)
|
||||
return deduped
|
||||
|
||||
|
||||
def _load_module(path: Path, normalizer_id: str) -> ModuleType:
|
||||
if not path.exists():
|
||||
raise ValidationError(f"{normalizer_id}: module not found: {path}")
|
||||
module_name = f"_guide_board_normalizer_{normalizer_id.replace('-', '_')}"
|
||||
spec = importlib.util.spec_from_file_location(module_name, path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise ValidationError(f"{normalizer_id}: unable to load module from {path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _extension_snapshot(plan: dict[str, Any], extension_id: str) -> dict[str, Any]:
|
||||
for extension in plan["extension_snapshots"]:
|
||||
if extension["id"] == extension_id:
|
||||
return extension
|
||||
raise ValidationError(f"step references unknown extension {extension_id!r}")
|
||||
|
||||
|
||||
def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path:
|
||||
path = Path(extension["path"])
|
||||
return path if path.is_absolute() else root / path
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -96,6 +97,16 @@ def build_run_plan(
|
||||
}
|
||||
)
|
||||
|
||||
source_lock = _build_source_lock(
|
||||
root,
|
||||
target_path,
|
||||
assessment_path,
|
||||
target,
|
||||
assessment,
|
||||
[extensions[extension_id] for extension_id in selected_extensions],
|
||||
)
|
||||
assert_valid(source_lock, "source-lock")
|
||||
|
||||
plan = {
|
||||
"id": f"plan-{_timestamp()}",
|
||||
"assessment_profile_snapshot": assessment,
|
||||
@@ -109,10 +120,7 @@ def build_run_plan(
|
||||
}
|
||||
for extension_id in selected_extensions
|
||||
],
|
||||
"source_lock": {
|
||||
"framework_refs": assessment["framework_refs"],
|
||||
"extension_refs": selected_extensions,
|
||||
},
|
||||
"source_lock": source_lock,
|
||||
"profile_paths": {
|
||||
"target_profile_path": str(target_path.resolve()),
|
||||
"assessment_profile_path": str(assessment_path.resolve()),
|
||||
@@ -208,6 +216,282 @@ def _load_extension_profile_schema(
|
||||
return load_json(schema_path)
|
||||
|
||||
|
||||
def _build_source_lock(
|
||||
root: Path,
|
||||
target_path: Path,
|
||||
assessment_path: Path,
|
||||
target: dict[str, Any],
|
||||
assessment: dict[str, Any],
|
||||
extensions: list[Extension],
|
||||
) -> dict[str, Any]:
|
||||
framework_refs = assessment["framework_refs"]
|
||||
extension_refs = [extension.id for extension in extensions]
|
||||
return {
|
||||
"id": f"source-lock:{assessment['id']}:{target['id']}",
|
||||
"schema_version": "guide-board.source-lock.v1",
|
||||
"created_at": _now(),
|
||||
"framework_refs": framework_refs,
|
||||
"extension_refs": extension_refs,
|
||||
"frameworks": _framework_records(framework_refs, extensions),
|
||||
"extensions": [_extension_source_record(root, extension) for extension in extensions],
|
||||
"mapping_sets": _mapping_source_records(root, extensions),
|
||||
"profiles": {
|
||||
"target": _file_source_record(
|
||||
"target-profile",
|
||||
target["id"],
|
||||
target_path,
|
||||
"target-profile.snapshot.json",
|
||||
),
|
||||
"assessment": _file_source_record(
|
||||
"assessment-profile",
|
||||
assessment["id"],
|
||||
assessment_path,
|
||||
"assessment-profile.snapshot.json",
|
||||
),
|
||||
},
|
||||
"policy_refs": {
|
||||
"expectations": _optional_policy_source_record(
|
||||
root,
|
||||
assessment_path,
|
||||
assessment.get("expectations_ref"),
|
||||
"expectation-set",
|
||||
),
|
||||
"waivers": _optional_policy_source_record(
|
||||
root,
|
||||
assessment_path,
|
||||
assessment.get("waivers_ref"),
|
||||
"waiver-set",
|
||||
),
|
||||
"challenges": _optional_policy_source_record(
|
||||
root,
|
||||
assessment_path,
|
||||
assessment.get("challenges_ref"),
|
||||
"challenge-set",
|
||||
),
|
||||
"exclusions": _optional_policy_source_record(
|
||||
root,
|
||||
assessment_path,
|
||||
assessment.get("exclusions_ref"),
|
||||
"exclusion-set",
|
||||
),
|
||||
},
|
||||
"authorities": _authority_source_records(extensions),
|
||||
"metadata_hooks": {
|
||||
"runner_entrypoints": _entrypoint_metadata_records(extensions),
|
||||
"normalizers": _normalizer_metadata_records(extensions),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _framework_records(
|
||||
framework_refs: list[str],
|
||||
extensions: list[Extension],
|
||||
) -> list[dict[str, Any]]:
|
||||
records = []
|
||||
for framework_ref in framework_refs:
|
||||
declaring_extensions = [
|
||||
extension.id
|
||||
for extension in extensions
|
||||
if framework_ref in _manifest_framework_ids(extension.manifest)
|
||||
]
|
||||
records.append(
|
||||
{
|
||||
"id": framework_ref,
|
||||
"version": _version_hint(framework_ref),
|
||||
"declared_by_extensions": declaring_extensions,
|
||||
}
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def _extension_source_record(root: Path, extension: Extension) -> dict[str, Any]:
|
||||
manifest_path = extension.path / "extension.json"
|
||||
return {
|
||||
"id": extension.id,
|
||||
"version": extension.manifest["version"],
|
||||
"path": _extension_path_ref(root, extension.path),
|
||||
"source": extension.source,
|
||||
"manifest_path": _display_path(root, manifest_path),
|
||||
"manifest_checksum": _checksum_if_file(manifest_path),
|
||||
"supported_frameworks": _manifest_framework_ids(extension.manifest),
|
||||
"authorities": _authority_ids(extension.manifest.get("authorities", [])),
|
||||
"certification_boundary": extension.manifest["certification_boundary"],
|
||||
"metadata": _object_or_empty(extension.manifest.get("metadata")),
|
||||
}
|
||||
|
||||
|
||||
def _mapping_source_records(root: Path, extensions: list[Extension]) -> list[dict[str, Any]]:
|
||||
records = []
|
||||
for extension in extensions:
|
||||
for mapping_id in extension.manifest.get("mappings", []):
|
||||
if not isinstance(mapping_id, str):
|
||||
continue
|
||||
mapping_path = extension.path / "mappings" / f"{mapping_id}.json"
|
||||
record = {
|
||||
"id": mapping_id,
|
||||
"extension_id": extension.id,
|
||||
"path": _display_path(root, mapping_path),
|
||||
"exists": mapping_path.is_file(),
|
||||
"checksum": _checksum_if_file(mapping_path),
|
||||
"framework_refs": [],
|
||||
}
|
||||
if mapping_path.is_file():
|
||||
mapping_set = load_json(mapping_path)
|
||||
record["framework_refs"] = _string_list(mapping_set.get("framework_refs", []))
|
||||
records.append(record)
|
||||
return records
|
||||
|
||||
|
||||
def _file_source_record(
|
||||
kind: str,
|
||||
profile_id: str,
|
||||
path: Path,
|
||||
snapshot_ref: str,
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"kind": kind,
|
||||
"id": profile_id,
|
||||
"path": str(path.resolve()),
|
||||
"checksum": _checksum_if_file(path),
|
||||
"snapshot_ref": snapshot_ref,
|
||||
}
|
||||
|
||||
|
||||
def _optional_policy_source_record(
|
||||
root: Path,
|
||||
assessment_path: Path,
|
||||
ref: Any,
|
||||
kind: str,
|
||||
) -> dict[str, Any] | None:
|
||||
if not isinstance(ref, str) or not ref:
|
||||
return None
|
||||
path = _resolve_assessment_ref(root, assessment_path, ref)
|
||||
return {
|
||||
"kind": kind,
|
||||
"ref": ref,
|
||||
"path": str(path.resolve()),
|
||||
"exists": path.is_file(),
|
||||
"checksum": _checksum_if_file(path),
|
||||
}
|
||||
|
||||
|
||||
def _authority_source_records(extensions: list[Extension]) -> list[dict[str, Any]]:
|
||||
records = []
|
||||
for extension in extensions:
|
||||
for authority in extension.manifest.get("authorities", []):
|
||||
if isinstance(authority, str):
|
||||
records.append({"id": authority, "extension_id": extension.id})
|
||||
elif isinstance(authority, dict):
|
||||
record = {
|
||||
"id": authority.get("id"),
|
||||
"extension_id": extension.id,
|
||||
}
|
||||
for key in ("name", "version", "source_url", "license", "access"):
|
||||
if key in authority:
|
||||
record[key] = authority[key]
|
||||
records.append(record)
|
||||
return [record for record in records if isinstance(record.get("id"), str)]
|
||||
|
||||
|
||||
def _entrypoint_metadata_records(extensions: list[Extension]) -> list[dict[str, Any]]:
|
||||
records = []
|
||||
for extension in extensions:
|
||||
for entrypoint in extension.manifest.get("runner_entrypoints", []):
|
||||
if not isinstance(entrypoint, dict):
|
||||
continue
|
||||
records.append(
|
||||
{
|
||||
"extension_id": extension.id,
|
||||
"id": entrypoint.get("id"),
|
||||
"kind": entrypoint.get("kind"),
|
||||
"metadata": _object_or_empty(entrypoint.get("metadata")),
|
||||
}
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def _normalizer_metadata_records(extensions: list[Extension]) -> list[dict[str, Any]]:
|
||||
records = []
|
||||
for extension in extensions:
|
||||
for normalizer in extension.manifest.get("normalizers", []):
|
||||
if not isinstance(normalizer, dict):
|
||||
continue
|
||||
records.append(
|
||||
{
|
||||
"extension_id": extension.id,
|
||||
"id": normalizer.get("id"),
|
||||
"kind": normalizer.get("kind"),
|
||||
"runner_ref": normalizer.get("runner_ref"),
|
||||
"metadata": _object_or_empty(normalizer.get("metadata")),
|
||||
}
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def _manifest_framework_ids(manifest: dict[str, Any]) -> list[str]:
|
||||
values = []
|
||||
for framework in manifest.get("supported_frameworks", []):
|
||||
if isinstance(framework, str):
|
||||
values.append(framework)
|
||||
elif isinstance(framework, dict) and isinstance(framework.get("id"), str):
|
||||
values.append(framework["id"])
|
||||
return values
|
||||
|
||||
|
||||
def _authority_ids(authorities: list[Any]) -> list[str]:
|
||||
values = []
|
||||
for authority in authorities:
|
||||
if isinstance(authority, str):
|
||||
values.append(authority)
|
||||
elif isinstance(authority, dict) and isinstance(authority.get("id"), str):
|
||||
values.append(authority["id"])
|
||||
return values
|
||||
|
||||
|
||||
def _resolve_assessment_ref(root: Path, assessment_path: Path, ref: str) -> Path:
|
||||
ref_path = Path(ref)
|
||||
if ref_path.is_absolute():
|
||||
return ref_path
|
||||
root_relative = root / ref_path
|
||||
if root_relative.exists():
|
||||
return root_relative
|
||||
return assessment_path.resolve().parent / ref_path
|
||||
|
||||
|
||||
def _checksum_if_file(path: Path) -> str | None:
|
||||
if not path.is_file():
|
||||
return None
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return f"sha256:{digest.hexdigest()}"
|
||||
|
||||
|
||||
def _version_hint(ref: str) -> str | None:
|
||||
for part in reversed(ref.replace("-", ".").split(".")):
|
||||
if len(part) > 1 and part[0].lower() == "v" and any(char.isdigit() for char in part[1:]):
|
||||
return part
|
||||
return None
|
||||
|
||||
|
||||
def _object_or_empty(value: Any) -> dict[str, Any]:
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def _string_list(value: Any) -> list[str]:
|
||||
if not isinstance(value, list):
|
||||
return []
|
||||
return [item for item in value if isinstance(item, str)]
|
||||
|
||||
|
||||
def _display_path(root: Path, path: Path) -> str:
|
||||
try:
|
||||
return str(path.resolve().relative_to(root.resolve()))
|
||||
except ValueError:
|
||||
return str(path.resolve())
|
||||
|
||||
|
||||
def _extension_path_ref(root: Path, path: Path) -> str:
|
||||
try:
|
||||
return str(path.resolve().relative_to(root.resolve()))
|
||||
@@ -215,5 +499,9 @@ def _extension_path_ref(root: Path, path: Path) -> str:
|
||||
return str(path.resolve())
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _timestamp() -> str:
|
||||
return datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
|
||||
@@ -13,20 +13,36 @@ from guide_board.schema import assert_valid
|
||||
def apply_policy(
|
||||
root: Path,
|
||||
plan: dict[str, Any],
|
||||
evidence: list[dict[str, Any]],
|
||||
findings: list[dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], dict[str, Any], list[dict[str, Any]]]:
|
||||
) -> tuple[
|
||||
list[dict[str, Any]],
|
||||
dict[str, Any],
|
||||
list[dict[str, Any]],
|
||||
list[dict[str, Any]],
|
||||
list[dict[str, Any]],
|
||||
]:
|
||||
expectations = _load_optional_set(root, plan, "expectations_ref", "expectation-set")
|
||||
waiver_set = _load_optional_set(root, plan, "waivers_ref", "waiver-set")
|
||||
challenge_set = _load_optional_set(root, plan, "challenges_ref", "challenge-set")
|
||||
exclusion_set = _load_optional_set(root, plan, "exclusions_ref", "exclusion-set")
|
||||
waivers = waiver_set.get("waivers", []) if waiver_set else []
|
||||
challenges = challenge_set.get("challenges", []) if challenge_set else []
|
||||
exclusions = exclusion_set.get("exclusions", []) if exclusion_set else []
|
||||
|
||||
applied_expectations = 0
|
||||
applied_waivers: list[dict[str, Any]] = []
|
||||
applied_challenges: list[dict[str, Any]] = []
|
||||
applied_exclusions: list[dict[str, Any]] = []
|
||||
evidence_by_id = {item["id"]: item for item in evidence}
|
||||
|
||||
for finding in findings:
|
||||
for expectation in expectations.get("expectations", []) if expectations else []:
|
||||
if _matches_rule(finding, expectation):
|
||||
finding["expected"] = expectation["expected"]
|
||||
finding["policy_ref"] = expectation["id"]
|
||||
finding["review_status"] = "expected" if expectation["expected"] else "unresolved_defect"
|
||||
_annotate_evidence(evidence_by_id, finding, "expectation_refs", expectation["id"])
|
||||
applied_expectations += 1
|
||||
break
|
||||
|
||||
@@ -37,20 +53,60 @@ def apply_policy(
|
||||
finding["waiver_ref"] = waiver["id"]
|
||||
finding["expected"] = True
|
||||
finding["policy_ref"] = waiver["id"]
|
||||
finding["review_status"] = "waived"
|
||||
finding["remediation"] = f"Waived: {waiver['reason']}"
|
||||
applied_waivers.append(waiver)
|
||||
_annotate_evidence(evidence_by_id, finding, "waiver_refs", waiver["id"])
|
||||
break
|
||||
|
||||
for exclusion in exclusions:
|
||||
if not _review_record_active(exclusion):
|
||||
continue
|
||||
if _matches_rule(finding, exclusion):
|
||||
finding["exclusion_ref"] = exclusion["id"]
|
||||
if finding.get("review_status") == "unresolved_defect":
|
||||
finding["review_status"] = "authority_excluded"
|
||||
applied_exclusions.append(exclusion)
|
||||
_annotate_evidence(evidence_by_id, finding, "exclusion_refs", exclusion["id"])
|
||||
break
|
||||
|
||||
for challenge in challenges:
|
||||
if not _review_record_active(challenge):
|
||||
continue
|
||||
if _matches_rule(finding, challenge):
|
||||
finding["challenge_ref"] = challenge["id"]
|
||||
if finding.get("review_status") == "unresolved_defect":
|
||||
finding["review_status"] = "challenged"
|
||||
applied_challenges.append(challenge)
|
||||
_annotate_evidence(evidence_by_id, finding, "challenge_refs", challenge["id"])
|
||||
break
|
||||
|
||||
policy_summary = {
|
||||
"expectations_ref": plan["assessment_profile_snapshot"].get("expectations_ref"),
|
||||
"waivers_ref": plan["assessment_profile_snapshot"].get("waivers_ref"),
|
||||
"challenges_ref": plan["assessment_profile_snapshot"].get("challenges_ref"),
|
||||
"exclusions_ref": plan["assessment_profile_snapshot"].get("exclusions_ref"),
|
||||
"applied_expectations": applied_expectations,
|
||||
"applied_waivers": len(applied_waivers),
|
||||
"challenged_findings": _unique_applied_count(findings, "challenge_ref"),
|
||||
"authority_exclusions": _unique_applied_count(findings, "exclusion_ref"),
|
||||
"unexpected_findings": sum(
|
||||
1 for finding in findings if not finding.get("expected") and not finding.get("waiver_ref")
|
||||
),
|
||||
"unresolved_defects": sum(
|
||||
1 for finding in findings if finding.get("review_status") == "unresolved_defect"
|
||||
),
|
||||
"unresolved_review_items": sum(
|
||||
1 for finding in findings if finding.get("review_status") in {"challenged", "authority_excluded"}
|
||||
),
|
||||
}
|
||||
return findings, policy_summary, applied_waivers
|
||||
return (
|
||||
findings,
|
||||
policy_summary,
|
||||
_dedupe_records(applied_waivers),
|
||||
_dedupe_records(applied_challenges),
|
||||
_dedupe_records(applied_exclusions),
|
||||
)
|
||||
|
||||
|
||||
def _load_optional_set(
|
||||
@@ -94,6 +150,7 @@ def _matches_rule(finding: dict[str, Any], rule: dict[str, Any]) -> bool:
|
||||
return (
|
||||
_matches_any(finding.get("requirement_refs", []), rule.get("requirement_refs", []))
|
||||
and _matches_any([finding.get("check_id", "")], rule.get("check_refs", []))
|
||||
and _matches_any(finding.get("evidence_refs", []), rule.get("evidence_refs", []))
|
||||
and _matches_scalar(finding.get("status"), rule.get("result_refs", []))
|
||||
and _matches_scalar(finding.get("classification"), rule.get("classification_refs", []))
|
||||
)
|
||||
@@ -122,3 +179,57 @@ def _waiver_active(waiver: dict[str, Any]) -> bool:
|
||||
except ValueError:
|
||||
return False
|
||||
return expiry >= date.today()
|
||||
|
||||
|
||||
def _review_record_active(record: dict[str, Any]) -> bool:
|
||||
status = record.get("review_status")
|
||||
if status in {"rejected", "withdrawn", "closed", "expired"}:
|
||||
return False
|
||||
expires_at = record.get("expires_at")
|
||||
if not expires_at:
|
||||
return True
|
||||
try:
|
||||
expiry = date.fromisoformat(expires_at)
|
||||
except ValueError:
|
||||
return False
|
||||
return expiry >= date.today()
|
||||
|
||||
|
||||
def _annotate_evidence(
|
||||
evidence_by_id: dict[str, dict[str, Any]],
|
||||
finding: dict[str, Any],
|
||||
ref_key: str,
|
||||
ref_value: str,
|
||||
) -> None:
|
||||
for evidence_ref in finding.get("evidence_refs", []):
|
||||
item = evidence_by_id.get(evidence_ref)
|
||||
if item is None:
|
||||
continue
|
||||
review = item.setdefault(
|
||||
"review",
|
||||
{
|
||||
"expectation_refs": [],
|
||||
"waiver_refs": [],
|
||||
"challenge_refs": [],
|
||||
"exclusion_refs": [],
|
||||
},
|
||||
)
|
||||
refs = review.setdefault(ref_key, [])
|
||||
if ref_value not in refs:
|
||||
refs.append(ref_value)
|
||||
|
||||
|
||||
def _unique_applied_count(findings: list[dict[str, Any]], ref_name: str) -> int:
|
||||
return sum(1 for finding in findings if finding.get(ref_name))
|
||||
|
||||
|
||||
def _dedupe_records(records: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
seen = set()
|
||||
deduped = []
|
||||
for record in records:
|
||||
record_id = record.get("id")
|
||||
if not isinstance(record_id, str) or record_id in seen:
|
||||
continue
|
||||
seen.add(record_id)
|
||||
deduped.append(record)
|
||||
return deduped
|
||||
|
||||
204
src/guide_board/reports.py
Normal file
204
src/guide_board/reports.py
Normal file
@@ -0,0 +1,204 @@
|
||||
"""Report fragment loading for extension-contributed report content."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import Any
|
||||
|
||||
from guide_board.errors import ValidationError
|
||||
from guide_board.io import load_json
|
||||
|
||||
|
||||
def build_report_fragments(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
evidence: list[dict[str, Any]],
|
||||
findings: list[dict[str, Any]],
|
||||
mapping_records: list[dict[str, Any]],
|
||||
assessment_package: dict[str, Any],
|
||||
) -> list[dict[str, Any]]:
|
||||
fragments: list[dict[str, Any]] = []
|
||||
for extension in plan["extension_snapshots"]:
|
||||
extension_path = _snapshot_path(root, extension)
|
||||
manifest = load_json(extension_path / "extension.json")
|
||||
for descriptor in manifest.get("report_fragments", []):
|
||||
fragment = _load_fragment(
|
||||
root,
|
||||
run_dir,
|
||||
run_id,
|
||||
plan,
|
||||
evidence,
|
||||
findings,
|
||||
mapping_records,
|
||||
assessment_package,
|
||||
extension,
|
||||
extension_path,
|
||||
descriptor,
|
||||
)
|
||||
if fragment is not None:
|
||||
fragments.append(fragment)
|
||||
return fragments
|
||||
|
||||
|
||||
def markdown_for_fragments(fragments: list[dict[str, Any]]) -> str:
|
||||
markdown_blocks = [
|
||||
fragment.get("markdown", "")
|
||||
for fragment in fragments
|
||||
if isinstance(fragment.get("markdown"), str) and fragment.get("markdown")
|
||||
]
|
||||
if not markdown_blocks:
|
||||
return "- no extension report fragments"
|
||||
return "\n\n".join(markdown_blocks)
|
||||
|
||||
|
||||
def _load_fragment(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
evidence: list[dict[str, Any]],
|
||||
findings: list[dict[str, Any]],
|
||||
mapping_records: list[dict[str, Any]],
|
||||
assessment_package: dict[str, Any],
|
||||
extension: dict[str, Any],
|
||||
extension_path: Path,
|
||||
descriptor: Any,
|
||||
) -> dict[str, Any] | None:
|
||||
if isinstance(descriptor, str):
|
||||
descriptor = {
|
||||
"id": descriptor,
|
||||
"kind": "markdown_file",
|
||||
"path": f"reports/{descriptor}.md",
|
||||
"title": descriptor,
|
||||
}
|
||||
if not isinstance(descriptor, dict):
|
||||
return None
|
||||
|
||||
if descriptor["kind"] == "markdown_file":
|
||||
markdown = _load_markdown_fragment(extension_path, descriptor)
|
||||
return _fragment_record(extension["id"], descriptor, markdown, {})
|
||||
if descriptor["kind"] == "python_module":
|
||||
result = _run_python_fragment(
|
||||
root,
|
||||
run_dir,
|
||||
run_id,
|
||||
plan,
|
||||
evidence,
|
||||
findings,
|
||||
mapping_records,
|
||||
assessment_package,
|
||||
extension_path,
|
||||
descriptor,
|
||||
)
|
||||
return _fragment_record(
|
||||
extension["id"],
|
||||
descriptor,
|
||||
result.get("markdown", ""),
|
||||
_object_or_empty(result.get("structured")),
|
||||
)
|
||||
raise ValidationError(f"{descriptor['id']}: unsupported report fragment kind")
|
||||
|
||||
|
||||
def _load_markdown_fragment(extension_path: Path, descriptor: dict[str, Any]) -> str:
|
||||
raw_path = descriptor.get("path") or f"reports/{descriptor['id']}.md"
|
||||
fragment_path = _safe_extension_path(extension_path, raw_path, descriptor["id"])
|
||||
if not fragment_path.is_file():
|
||||
raise ValidationError(f"{descriptor['id']}: report fragment not found: {raw_path}")
|
||||
return fragment_path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _run_python_fragment(
|
||||
root: Path,
|
||||
run_dir: Path,
|
||||
run_id: str,
|
||||
plan: dict[str, Any],
|
||||
evidence: list[dict[str, Any]],
|
||||
findings: list[dict[str, Any]],
|
||||
mapping_records: list[dict[str, Any]],
|
||||
assessment_package: dict[str, Any],
|
||||
extension_path: Path,
|
||||
descriptor: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
module_path = descriptor.get("module_path")
|
||||
callable_name = descriptor.get("callable")
|
||||
if not module_path or not callable_name:
|
||||
raise ValidationError(
|
||||
f"{descriptor['id']}: python_module report fragments need module_path and callable"
|
||||
)
|
||||
|
||||
module_file = _safe_extension_path(extension_path, module_path, descriptor["id"])
|
||||
module = _load_module(module_file, descriptor["id"])
|
||||
fragment_callable = getattr(module, callable_name, None)
|
||||
if not callable(fragment_callable):
|
||||
raise ValidationError(f"{descriptor['id']}: callable {callable_name!r} was not found")
|
||||
|
||||
context = {
|
||||
"root": str(root),
|
||||
"run_dir": str(run_dir),
|
||||
"run_id": run_id,
|
||||
"plan": plan,
|
||||
"evidence": evidence,
|
||||
"findings": findings,
|
||||
"mappings": mapping_records,
|
||||
"assessment_package": assessment_package,
|
||||
"policy_summary": assessment_package.get("policy_summary", {}),
|
||||
"source_lock": assessment_package.get("source_lock", {}),
|
||||
"extension_path": str(extension_path),
|
||||
"report_fragment": descriptor,
|
||||
}
|
||||
result = fragment_callable(context)
|
||||
if not isinstance(result, dict):
|
||||
raise ValidationError(f"{descriptor['id']}: report fragment must return an object")
|
||||
return result
|
||||
|
||||
|
||||
def _fragment_record(
|
||||
extension_id: str,
|
||||
descriptor: dict[str, Any],
|
||||
markdown: str,
|
||||
structured: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"id": descriptor["id"],
|
||||
"extension_id": extension_id,
|
||||
"title": descriptor.get("title") or descriptor["id"],
|
||||
"kind": descriptor["kind"],
|
||||
"markdown": markdown if isinstance(markdown, str) else "",
|
||||
"structured": structured,
|
||||
}
|
||||
|
||||
|
||||
def _safe_extension_path(extension_path: Path, raw_path: str, fragment_id: str) -> Path:
|
||||
path = (extension_path / raw_path).resolve()
|
||||
try:
|
||||
path.relative_to(extension_path.resolve())
|
||||
except ValueError as exc:
|
||||
raise ValidationError(
|
||||
f"{fragment_id}: report fragment path must stay inside the extension directory"
|
||||
) from exc
|
||||
return path
|
||||
|
||||
|
||||
def _load_module(path: Path, fragment_id: str) -> ModuleType:
|
||||
if not path.exists():
|
||||
raise ValidationError(f"{fragment_id}: module not found: {path}")
|
||||
module_name = f"_guide_board_report_fragment_{fragment_id.replace('-', '_')}"
|
||||
spec = importlib.util.spec_from_file_location(module_name, path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise ValidationError(f"{fragment_id}: unable to load module from {path}")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _snapshot_path(root: Path, extension: dict[str, Any]) -> Path:
|
||||
path = Path(extension["path"])
|
||||
return path if path.is_absolute() else root / path
|
||||
|
||||
|
||||
def _object_or_empty(value: Any) -> dict[str, Any]:
|
||||
return value if isinstance(value, dict) else {}
|
||||
@@ -37,6 +37,10 @@ def build_retention_summary(
|
||||
"unexpected_findings": policy_summary.get("unexpected_findings", 0),
|
||||
"expected_findings": sum(1 for finding in findings if finding.get("expected")),
|
||||
"waived_findings": sum(1 for finding in findings if finding.get("waiver_ref")),
|
||||
"challenged_findings": policy_summary.get("challenged_findings", 0),
|
||||
"authority_exclusions": policy_summary.get("authority_exclusions", 0),
|
||||
"unresolved_defects": policy_summary.get("unresolved_defects", 0),
|
||||
"unresolved_review_items": policy_summary.get("unresolved_review_items", 0),
|
||||
"mapping_target_count": len(
|
||||
assessment_package.get("mapping_summary", {}).get("targets", [])
|
||||
),
|
||||
@@ -45,6 +49,11 @@ def build_retention_summary(
|
||||
"report_refs": [
|
||||
"reports/assessment-package.json",
|
||||
"reports/report.md",
|
||||
"reports/fragments.json",
|
||||
"reports/submission-package.json",
|
||||
],
|
||||
"export_refs": [
|
||||
"exports/export-manifest.json",
|
||||
],
|
||||
"artifact_retention": {
|
||||
"policy": plan["assessment_profile_snapshot"].get("retention_policy", {}),
|
||||
@@ -195,7 +204,12 @@ def _run_projection(run: dict[str, Any]) -> dict[str, Any]:
|
||||
"status": summary.get("status", "unknown"),
|
||||
"unexpected_findings": _summary_int(summary, "unexpected_findings"),
|
||||
"finding_count": _summary_int(summary, "finding_count"),
|
||||
"mapping_target_count": _summary_int(summary, "mapping_target_count"),
|
||||
"artifact_count": _summary_int(summary, "artifact_count"),
|
||||
"challenged_findings": _summary_int(summary, "challenged_findings"),
|
||||
"authority_exclusions": _summary_int(summary, "authority_exclusions"),
|
||||
"unresolved_defects": _summary_int(summary, "unresolved_defects"),
|
||||
"unresolved_review_items": _summary_int(summary, "unresolved_review_items"),
|
||||
"run_dir": run.get("run_dir"),
|
||||
}
|
||||
|
||||
@@ -208,10 +222,17 @@ def _trend_between(
|
||||
return {
|
||||
"direction": "insufficient-history",
|
||||
"status_changed": False,
|
||||
"status_change": {
|
||||
"from": None,
|
||||
"to": _status_for(latest),
|
||||
},
|
||||
"unexpected_findings_delta": 0,
|
||||
"finding_count_delta": 0,
|
||||
"artifact_count_delta": 0,
|
||||
"unresolved_review_items_delta": 0,
|
||||
"mapping_target_count_delta": 0,
|
||||
"evidence_result_deltas": {},
|
||||
"summary_text": "No previous retained run is available for comparison.",
|
||||
}
|
||||
|
||||
previous_summary = previous.get("summary", {})
|
||||
@@ -229,16 +250,37 @@ def _trend_between(
|
||||
artifact_delta = _summary_int(latest_summary, "artifact_count") - _summary_int(
|
||||
previous_summary, "artifact_count"
|
||||
)
|
||||
review_delta = _summary_int(latest_summary, "unresolved_review_items") - _summary_int(
|
||||
previous_summary, "unresolved_review_items"
|
||||
)
|
||||
mapping_target_delta = _summary_int(latest_summary, "mapping_target_count") - _summary_int(
|
||||
previous_summary, "mapping_target_count"
|
||||
)
|
||||
previous_status = _status_for(previous)
|
||||
latest_status = _status_for(latest)
|
||||
direction = _trend_direction(previous_status, latest_status, unexpected_delta)
|
||||
|
||||
return {
|
||||
"direction": _trend_direction(previous_status, latest_status, unexpected_delta),
|
||||
"direction": direction,
|
||||
"status_changed": previous_status != latest_status,
|
||||
"status_change": {
|
||||
"from": previous_status,
|
||||
"to": latest_status,
|
||||
},
|
||||
"unexpected_findings_delta": unexpected_delta,
|
||||
"finding_count_delta": finding_delta,
|
||||
"artifact_count_delta": artifact_delta,
|
||||
"unresolved_review_items_delta": review_delta,
|
||||
"mapping_target_count_delta": mapping_target_delta,
|
||||
"evidence_result_deltas": evidence_deltas,
|
||||
"summary_text": _trend_summary_text(
|
||||
direction,
|
||||
previous_status,
|
||||
latest_status,
|
||||
unexpected_delta,
|
||||
review_delta,
|
||||
mapping_target_delta,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -280,6 +322,24 @@ def _summary_int(summary: dict[str, Any], key: str) -> int:
|
||||
return value if isinstance(value, int) and not isinstance(value, bool) else 0
|
||||
|
||||
|
||||
def _trend_summary_text(
|
||||
direction: str,
|
||||
previous_status: str,
|
||||
latest_status: str,
|
||||
unexpected_delta: int,
|
||||
review_delta: int,
|
||||
mapping_target_delta: int,
|
||||
) -> str:
|
||||
parts = [
|
||||
f"Trend {direction}",
|
||||
f"status {previous_status} -> {latest_status}",
|
||||
f"unexpected findings delta {unexpected_delta}",
|
||||
f"unresolved review delta {review_delta}",
|
||||
f"mapping target delta {mapping_target_delta}",
|
||||
]
|
||||
return "; ".join(parts) + "."
|
||||
|
||||
|
||||
def _dict_deltas(previous: Any, latest: Any) -> dict[str, int]:
|
||||
previous_dict = previous if isinstance(previous, dict) else {}
|
||||
latest_dict = latest if isinstance(latest, dict) else {}
|
||||
|
||||
@@ -45,6 +45,8 @@ def run_step(
|
||||
"runner_kind": "external",
|
||||
},
|
||||
"artifact_refs": [],
|
||||
"requirement_refs": [],
|
||||
"metadata": _object_or_empty(entrypoint.get("metadata")),
|
||||
}
|
||||
if entrypoint["kind"] == "command":
|
||||
return _run_command(root, run_dir, run_id, plan, step, extension_path, entrypoint)
|
||||
@@ -63,6 +65,8 @@ def _no_runner_result(step: dict[str, Any]) -> dict[str, Any]:
|
||||
"runner_kind": None,
|
||||
},
|
||||
"artifact_refs": [],
|
||||
"requirement_refs": [],
|
||||
"metadata": {},
|
||||
}
|
||||
|
||||
|
||||
@@ -118,6 +122,8 @@ def _run_python_module(
|
||||
"error_type": type(exc).__name__,
|
||||
},
|
||||
"artifact_refs": [],
|
||||
"requirement_refs": [],
|
||||
"metadata": _object_or_empty(entrypoint.get("metadata")),
|
||||
}
|
||||
if not isinstance(result, dict):
|
||||
raise ValidationError(f"{entrypoint['id']}: runner must return an object")
|
||||
@@ -126,6 +132,8 @@ def _run_python_module(
|
||||
"observations": result.get("observations", []),
|
||||
"facts": result.get("facts", {}),
|
||||
"artifact_refs": result.get("artifact_refs", []),
|
||||
"requirement_refs": result.get("requirement_refs", []),
|
||||
"metadata": _merge_metadata(entrypoint.get("metadata"), result.get("metadata")),
|
||||
}
|
||||
|
||||
|
||||
@@ -192,6 +200,8 @@ def _run_command(
|
||||
"command": command,
|
||||
},
|
||||
"artifact_refs": [str(context_path.relative_to(run_dir))],
|
||||
"requirement_refs": [],
|
||||
"metadata": _object_or_empty(entrypoint.get("metadata")),
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {
|
||||
@@ -206,6 +216,8 @@ def _run_command(
|
||||
"command": command,
|
||||
},
|
||||
"artifact_refs": [str(context_path.relative_to(run_dir))],
|
||||
"requirement_refs": [],
|
||||
"metadata": _object_or_empty(entrypoint.get("metadata")),
|
||||
}
|
||||
|
||||
parsed = _parse_runner_stdout(completed.stdout)
|
||||
@@ -225,6 +237,8 @@ def _run_command(
|
||||
"command": command,
|
||||
},
|
||||
"artifact_refs": [str(context_path.relative_to(run_dir))],
|
||||
"requirement_refs": [],
|
||||
"metadata": _object_or_empty(entrypoint.get("metadata")),
|
||||
}
|
||||
|
||||
facts = parsed.get("facts", {})
|
||||
@@ -245,6 +259,9 @@ def _run_command(
|
||||
if not isinstance(artifact_refs, list):
|
||||
artifact_refs = []
|
||||
artifact_refs.append(str(context_path.relative_to(run_dir)))
|
||||
requirement_refs = parsed.get("requirement_refs", [])
|
||||
if not isinstance(requirement_refs, list):
|
||||
requirement_refs = []
|
||||
|
||||
result = parsed.get("result", "unknown")
|
||||
if completed.returncode != 0 and result in {"pass", "warning", "manual", "skipped"}:
|
||||
@@ -258,6 +275,8 @@ def _run_command(
|
||||
"observations": observations,
|
||||
"facts": facts,
|
||||
"artifact_refs": artifact_refs,
|
||||
"requirement_refs": requirement_refs,
|
||||
"metadata": _merge_metadata(entrypoint.get("metadata"), parsed.get("metadata")),
|
||||
}
|
||||
|
||||
|
||||
@@ -328,5 +347,17 @@ def _parse_runner_stdout(stdout: str) -> dict[str, Any] | None:
|
||||
return parsed
|
||||
|
||||
|
||||
def _merge_metadata(*values: Any) -> dict[str, Any]:
|
||||
merged: dict[str, Any] = {}
|
||||
for value in values:
|
||||
if isinstance(value, dict):
|
||||
merged.update(value)
|
||||
return merged
|
||||
|
||||
|
||||
def _object_or_empty(value: Any) -> dict[str, Any]:
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def _safe_id(value: str) -> str:
|
||||
return "".join(char if char.isalnum() or char in {"-", "_"} else "_" for char in value)
|
||||
|
||||
@@ -10,7 +10,7 @@ from datetime import datetime, timezone
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urlparse
|
||||
from urllib.parse import parse_qs, unquote, urlparse
|
||||
|
||||
from guide_board.discovery import discover_extensions
|
||||
from guide_board.errors import GuideBoardError
|
||||
@@ -21,6 +21,11 @@ from guide_board.planning import (
|
||||
validate_assessment_profile,
|
||||
validate_target_profile,
|
||||
)
|
||||
from guide_board.retention import (
|
||||
list_retained_runs,
|
||||
retained_run_report_paths,
|
||||
select_retained_run,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -131,7 +136,7 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
def _handle(self, method: str) -> None:
|
||||
parsed = urlparse(self.path)
|
||||
try:
|
||||
response, status_code = self._route(method, parsed.path)
|
||||
response, status_code = self._route(method, parsed.path, parsed.query)
|
||||
except HttpProblem as exc:
|
||||
response = _error_response(exc.message, exc.__class__.__name__, exc.status_code)
|
||||
status_code = exc.status_code
|
||||
@@ -147,7 +152,8 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
|
||||
self._send_json(status_code, response)
|
||||
|
||||
def _route(self, method: str, path: str) -> tuple[dict[str, Any], int]:
|
||||
def _route(self, method: str, path: str, query: str = "") -> tuple[dict[str, Any], int]:
|
||||
query_params = _query_params(query)
|
||||
if method == "GET" and path == "/health":
|
||||
return self._health(), 200
|
||||
if method == "GET" and path == "/extensions":
|
||||
@@ -160,6 +166,10 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
return {"runs": self.server.context.jobs.list()}, 200
|
||||
if method == "POST" and path == "/runs":
|
||||
return self._start_run(), 202
|
||||
if method == "GET" and path == "/retained-runs":
|
||||
return self._retained_runs(query_params), 200
|
||||
if method == "GET" and path == "/retained-runs/latest":
|
||||
return self._retained_latest(query_params), 200
|
||||
|
||||
run_match = _match_run_path(path)
|
||||
if method == "GET" and run_match is not None:
|
||||
@@ -169,6 +179,14 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
if suffix == "reports":
|
||||
return self._run_reports(job_id), 200
|
||||
|
||||
retained_match = _match_retained_run_path(path)
|
||||
if method == "GET" and retained_match is not None:
|
||||
run_id, suffix = retained_match
|
||||
if suffix == "reports":
|
||||
return self._retained_run_reports(run_id, query_params), 200
|
||||
if suffix == "artifact-manifest":
|
||||
return self._retained_artifact_manifest(run_id, query_params), 200
|
||||
|
||||
raise HttpProblem(404, f"endpoint not found: {method} {path}")
|
||||
|
||||
def _health(self) -> dict[str, Any]:
|
||||
@@ -261,10 +279,32 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
report_path = Path(result["report"])
|
||||
package_path = Path(result["assessment_package"])
|
||||
retention_path = Path(result["retention_summary"])
|
||||
submission_value = result.get("submission_package")
|
||||
submission_path = (
|
||||
Path(submission_value)
|
||||
if isinstance(submission_value, str) and submission_value
|
||||
else None
|
||||
)
|
||||
export_value = result.get("export_manifest")
|
||||
export_path = (
|
||||
Path(export_value)
|
||||
if isinstance(export_value, str) and export_value
|
||||
else None
|
||||
)
|
||||
try:
|
||||
report_markdown = report_path.read_text(encoding="utf-8")
|
||||
assessment_package = load_json(package_path)
|
||||
retention_summary = load_json(retention_path)
|
||||
submission_package = (
|
||||
load_json(submission_path)
|
||||
if submission_path is not None and submission_path.is_file()
|
||||
else None
|
||||
)
|
||||
export_manifest = (
|
||||
load_json(export_path)
|
||||
if export_path is not None and export_path.is_file()
|
||||
else None
|
||||
)
|
||||
except OSError as exc:
|
||||
raise HttpProblem(404, f"run report artifact is missing: {exc}") from exc
|
||||
|
||||
@@ -277,6 +317,8 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
"report": str(report_path),
|
||||
"assessment_package": str(package_path),
|
||||
"retention_summary": str(retention_path),
|
||||
"submission_package": str(submission_path) if submission_path is not None else None,
|
||||
"export_manifest": str(export_path) if export_path is not None else None,
|
||||
},
|
||||
"report": {
|
||||
"path": str(report_path),
|
||||
@@ -290,6 +332,73 @@ class GuideBoardRequestHandler(BaseHTTPRequestHandler):
|
||||
"path": str(retention_path),
|
||||
"json": retention_summary,
|
||||
},
|
||||
"submission_package": {
|
||||
"path": str(submission_path) if submission_package else None,
|
||||
"json": submission_package,
|
||||
},
|
||||
"export_manifest": {
|
||||
"path": str(export_path) if export_manifest else None,
|
||||
"json": export_manifest,
|
||||
},
|
||||
}
|
||||
|
||||
def _retained_runs(self, query: dict[str, str]) -> dict[str, Any]:
|
||||
runs_dir = _runs_dir_from_query(self.server.context.root, query)
|
||||
return {
|
||||
"runs_dir": str(runs_dir),
|
||||
"runs": list_retained_runs(runs_dir),
|
||||
}
|
||||
|
||||
def _retained_latest(self, query: dict[str, str]) -> dict[str, Any]:
|
||||
runs_dir = _runs_dir_from_query(self.server.context.root, query)
|
||||
run = select_retained_run(
|
||||
runs_dir,
|
||||
target_profile_ref=query.get("target"),
|
||||
assessment_profile_ref=query.get("assessment"),
|
||||
)
|
||||
return {
|
||||
"runs_dir": str(runs_dir),
|
||||
"selection": {
|
||||
"target_profile_ref": query.get("target"),
|
||||
"assessment_profile_ref": query.get("assessment"),
|
||||
},
|
||||
"run": _retained_run_with_paths(run) if run else None,
|
||||
}
|
||||
|
||||
def _retained_run_reports(self, run_id: str, query: dict[str, str]) -> dict[str, Any]:
|
||||
runs_dir = _runs_dir_from_query(self.server.context.root, query)
|
||||
run = _select_retained_run_or_404(runs_dir, run_id)
|
||||
return {
|
||||
"runs_dir": str(runs_dir),
|
||||
"run": _retained_run_with_paths(run),
|
||||
}
|
||||
|
||||
def _retained_artifact_manifest(self, run_id: str, query: dict[str, str]) -> dict[str, Any]:
|
||||
runs_dir = _runs_dir_from_query(self.server.context.root, query)
|
||||
run = _select_retained_run_or_404(runs_dir, run_id)
|
||||
run_dir = _safe_run_dir(runs_dir, run)
|
||||
package_path = run_dir / "reports" / "assessment-package.json"
|
||||
if not package_path.exists():
|
||||
return {
|
||||
"runs_dir": str(runs_dir),
|
||||
"run_id": run_id,
|
||||
"run_dir": str(run_dir),
|
||||
"artifact_manifest": [],
|
||||
"compatibility": "assessment-package-missing",
|
||||
}
|
||||
package = load_json(package_path)
|
||||
artifacts = package.get("artifact_manifest", [])
|
||||
if not isinstance(artifacts, list):
|
||||
raise HttpProblem(400, f"{package_path}: artifact_manifest must be a list")
|
||||
for artifact in artifacts:
|
||||
if isinstance(artifact, dict):
|
||||
_safe_run_ref(run_dir, artifact.get("path"))
|
||||
return {
|
||||
"runs_dir": str(runs_dir),
|
||||
"run_id": run_id,
|
||||
"run_dir": str(run_dir),
|
||||
"artifact_manifest": artifacts,
|
||||
"compatibility": "current",
|
||||
}
|
||||
|
||||
def _read_payload(self) -> dict[str, Any]:
|
||||
@@ -430,6 +539,81 @@ def _match_run_path(path: str) -> tuple[str, str | None] | None:
|
||||
return None
|
||||
|
||||
|
||||
def _match_retained_run_path(path: str) -> tuple[str, str] | None:
|
||||
parts = [unquote(part) for part in path.split("/") if part]
|
||||
if len(parts) == 3 and parts[0] == "retained-runs":
|
||||
return parts[1], parts[2]
|
||||
return None
|
||||
|
||||
|
||||
def _query_params(query: str) -> dict[str, str]:
|
||||
parsed = parse_qs(query, keep_blank_values=False)
|
||||
params = {}
|
||||
for key, values in parsed.items():
|
||||
if values:
|
||||
params[key] = values[-1]
|
||||
return params
|
||||
|
||||
|
||||
def _runs_dir_from_query(root: Path, query: dict[str, str]) -> Path:
|
||||
runs_dir = query.get("runs_dir")
|
||||
if not runs_dir:
|
||||
return (root / "runs").resolve()
|
||||
return _resolve_path(root, runs_dir)
|
||||
|
||||
|
||||
def _select_retained_run_or_404(runs_dir: Path, run_id: str) -> dict[str, Any]:
|
||||
run = select_retained_run(runs_dir, run_id=run_id)
|
||||
if run is None:
|
||||
raise HttpProblem(404, f"retained run not found: {run_id}")
|
||||
return run
|
||||
|
||||
|
||||
def _retained_run_with_paths(run: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||
if run is None:
|
||||
return None
|
||||
paths = retained_run_report_paths(run)
|
||||
run_dir = Path(run["run_dir"]).resolve()
|
||||
safe_paths = {}
|
||||
for key, value in paths.items():
|
||||
path = Path(value).resolve()
|
||||
try:
|
||||
path.relative_to(run_dir)
|
||||
except ValueError as exc:
|
||||
raise HttpProblem(
|
||||
400,
|
||||
f"retained run report path escapes run directory: {value}",
|
||||
) from exc
|
||||
safe_paths[key] = str(path)
|
||||
return {
|
||||
**run,
|
||||
"paths": dict(sorted(safe_paths.items())),
|
||||
}
|
||||
|
||||
|
||||
def _safe_run_dir(runs_dir: Path, run: dict[str, Any]) -> Path:
|
||||
run_dir_value = run.get("run_dir")
|
||||
if not isinstance(run_dir_value, str) or not run_dir_value:
|
||||
raise HttpProblem(400, "retained run is missing run_dir")
|
||||
run_dir = Path(run_dir_value).resolve()
|
||||
try:
|
||||
run_dir.relative_to(runs_dir.resolve())
|
||||
except ValueError as exc:
|
||||
raise HttpProblem(400, f"retained run escapes runs_dir: {run_dir}") from exc
|
||||
return run_dir
|
||||
|
||||
|
||||
def _safe_run_ref(run_dir: Path, ref: Any) -> Path:
|
||||
if not isinstance(ref, str) or not ref:
|
||||
raise HttpProblem(400, "artifact manifest entry path must be a non-empty string")
|
||||
path = (run_dir / ref).resolve()
|
||||
try:
|
||||
path.relative_to(run_dir.resolve())
|
||||
except ValueError as exc:
|
||||
raise HttpProblem(400, f"artifact path escapes run directory: {ref}") from exc
|
||||
return path
|
||||
|
||||
|
||||
def _display_path(root: Path, path: Path) -> str:
|
||||
try:
|
||||
return str(path.resolve().relative_to(root.resolve()))
|
||||
|
||||
17
tests/golden/export-manifest-shape.json
Normal file
17
tests/golden/export-manifest-shape.json
Normal file
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"top_level_keys": [
|
||||
"certification_boundary",
|
||||
"counts",
|
||||
"created_at",
|
||||
"export_type",
|
||||
"id",
|
||||
"mapping_summary",
|
||||
"policy_summary",
|
||||
"report_fragments",
|
||||
"run_id",
|
||||
"schema_version",
|
||||
"source_lock_ref",
|
||||
"source_package_ref",
|
||||
"summary"
|
||||
]
|
||||
}
|
||||
5
tests/golden/sdk-fixture-report-fragment.md
Normal file
5
tests/golden/sdk-fixture-report-fragment.md
Normal file
@@ -0,0 +1,5 @@
|
||||
### SDK Fixture Summary
|
||||
|
||||
- evidence items: 2
|
||||
- findings: 0
|
||||
- source lock: source-lock:sdk-fixture-assessment:sdk-fixture-target
|
||||
@@ -2,10 +2,12 @@ from __future__ import annotations
|
||||
|
||||
import http.client
|
||||
import json
|
||||
import shutil
|
||||
import time
|
||||
import unittest
|
||||
from tempfile import TemporaryDirectory
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote
|
||||
|
||||
from guide_board.discovery import discover_extensions
|
||||
from guide_board.errors import ValidationError
|
||||
@@ -74,6 +76,17 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
plan["ordered_steps"][1]["requirement_refs"],
|
||||
["guide-board.sample-readiness.v0.profile-shape"],
|
||||
)
|
||||
assert_valid(plan["source_lock"], "source-lock")
|
||||
self.assertEqual(plan["source_lock"]["schema_version"], "guide-board.source-lock.v1")
|
||||
self.assertEqual(plan["source_lock"]["framework_refs"], ["guide-board.sample-readiness.v0"])
|
||||
self.assertEqual(plan["source_lock"]["extension_refs"], ["sample-noop"])
|
||||
self.assertEqual(
|
||||
plan["source_lock"]["profiles"]["target"]["snapshot_ref"],
|
||||
"target-profile.snapshot.json",
|
||||
)
|
||||
self.assertTrue(plan["source_lock"]["profiles"]["target"]["checksum"].startswith("sha256:"))
|
||||
self.assertEqual(plan["source_lock"]["mapping_sets"][0]["id"], "sample-readiness-map")
|
||||
self.assertTrue(plan["source_lock"]["mapping_sets"][0]["checksum"].startswith("sha256:"))
|
||||
|
||||
def test_runs_external_extension_from_separate_repo(self) -> None:
|
||||
with TemporaryDirectory() as temporary_directory:
|
||||
@@ -193,6 +206,100 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
):
|
||||
validate_target_profile(target_path, extensions)
|
||||
|
||||
def test_runs_sdk_fixture_from_external_extension_repo(self) -> None:
|
||||
with TemporaryDirectory() as temporary_directory:
|
||||
temp_root = Path(temporary_directory)
|
||||
extension_dir = temp_root / "sdk-fixture"
|
||||
shutil.copytree(ROOT / "extensions" / "sdk-fixture", extension_dir)
|
||||
|
||||
result = run_assessment(
|
||||
temp_root,
|
||||
extension_dir / "profiles" / "targets" / "sdk-fixture-target.json",
|
||||
extension_dir / "profiles" / "assessments" / "sdk-fixture-assessment.json",
|
||||
temp_root / "runs" / "sdk-fixture",
|
||||
[extension_dir],
|
||||
)
|
||||
run_dir = Path(result["run_dir"])
|
||||
plan = json.loads((run_dir / "plan.json").read_text(encoding="utf-8"))
|
||||
evidence = json.loads(
|
||||
(run_dir / "normalized" / "evidence.json").read_text(encoding="utf-8")
|
||||
)["evidence"]
|
||||
mappings = json.loads(
|
||||
(run_dir / "normalized" / "mappings.json").read_text(encoding="utf-8")
|
||||
)["mappings"]
|
||||
assessment_package = json.loads(
|
||||
(run_dir / "reports" / "assessment-package.json").read_text(encoding="utf-8")
|
||||
)
|
||||
report = (run_dir / "reports" / "report.md").read_text(encoding="utf-8")
|
||||
export_manifest = json.loads(
|
||||
(run_dir / "exports" / "export-manifest.json").read_text(encoding="utf-8")
|
||||
)
|
||||
|
||||
self.assertEqual(result["status"], "completed")
|
||||
self.assertEqual(plan["extension_snapshots"][0]["source"], "external")
|
||||
self.assertEqual(plan["target_profile_snapshot"]["subject_type"], "sdk-fixture-target")
|
||||
self.assertEqual([item["result"] for item in evidence], ["skipped", "pass"])
|
||||
check_evidence = evidence[1]
|
||||
self.assertEqual(
|
||||
check_evidence["facts"]["normalizer_refs"],
|
||||
["native-probe-normalizer"],
|
||||
)
|
||||
self.assertEqual(check_evidence["facts"]["native_score"], 98)
|
||||
self.assertEqual(
|
||||
check_evidence["requirement_refs"],
|
||||
["guide-board.sdk-fixture.v1.native-output"],
|
||||
)
|
||||
self.assertEqual(
|
||||
check_evidence["artifact_refs"],
|
||||
["artifacts/sdk-fixture/native-result.json"],
|
||||
)
|
||||
self.assertEqual(
|
||||
check_evidence["facts"]["source_metadata"]["runner"]["metadata"]["harness_version"],
|
||||
"1.0.0",
|
||||
)
|
||||
self.assertEqual(
|
||||
check_evidence["facts"]["source_metadata"]["reported"]["native_result_id"],
|
||||
"sdk-fixture-native-result",
|
||||
)
|
||||
self.assertEqual(mappings[0]["target_id"], "normalizer-plugin")
|
||||
self.assertEqual(assessment_package["summary"], {"pass": 1, "skipped": 1})
|
||||
self.assertEqual(
|
||||
assessment_package["report_fragments"][0]["markdown"],
|
||||
(ROOT / "tests" / "golden" / "sdk-fixture-report-fragment.md")
|
||||
.read_text(encoding="utf-8")
|
||||
.rstrip(),
|
||||
)
|
||||
self.assertIn("### SDK Fixture Summary", report)
|
||||
assert_valid(export_manifest, "export-manifest")
|
||||
export_shape = load_json(ROOT / "tests" / "golden" / "export-manifest-shape.json")
|
||||
self.assertEqual(sorted(export_manifest), export_shape["top_level_keys"])
|
||||
self.assertEqual(export_manifest["counts"]["report_fragments"], 1)
|
||||
self.assertEqual(
|
||||
export_manifest["report_fragments"][0]["structured"]["evidence_count"],
|
||||
2,
|
||||
)
|
||||
self.assertEqual(
|
||||
assessment_package["source_lock"]["metadata_hooks"]["runner_entrypoints"][0][
|
||||
"metadata"
|
||||
]["harness_id"],
|
||||
"sdk-fixture-native-probe",
|
||||
)
|
||||
submission_package = json.loads(
|
||||
(run_dir / "reports" / "submission-package.json").read_text(encoding="utf-8")
|
||||
)
|
||||
assert_valid(submission_package, "submission-package")
|
||||
self.assertEqual(submission_package["source_lock"]["id"], "source-lock:sdk-fixture-assessment:sdk-fixture-target")
|
||||
self.assertEqual(
|
||||
submission_package["reported_metadata"][1]["metadata"]["reported"][
|
||||
"native_result_id"
|
||||
],
|
||||
"sdk-fixture-native-result",
|
||||
)
|
||||
self.assertEqual(
|
||||
submission_package["artifact_manifest"][0]["checksum"],
|
||||
assessment_package["artifact_manifest"][0]["checksum"],
|
||||
)
|
||||
|
||||
def test_runs_sample_noop_assessment(self) -> None:
|
||||
with TemporaryDirectory() as temporary_directory:
|
||||
result = run_assessment(
|
||||
@@ -209,6 +316,9 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
self.assertTrue((run_dir / "normalized" / "evidence.json").exists())
|
||||
self.assertTrue((run_dir / "reports" / "assessment-package.json").exists())
|
||||
self.assertTrue((run_dir / "reports" / "report.md").exists())
|
||||
self.assertTrue((run_dir / "reports" / "fragments.json").exists())
|
||||
self.assertTrue((run_dir / "reports" / "submission-package.json").exists())
|
||||
self.assertTrue((run_dir / "exports" / "export-manifest.json").exists())
|
||||
retention = json.loads(
|
||||
(run_dir / "retention-summary.json").read_text(encoding="utf-8")
|
||||
)
|
||||
@@ -216,12 +326,31 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
result["retention_summary"],
|
||||
str(run_dir / "retention-summary.json"),
|
||||
)
|
||||
self.assertEqual(
|
||||
result["submission_package"],
|
||||
str(run_dir / "reports" / "submission-package.json"),
|
||||
)
|
||||
self.assertEqual(
|
||||
result["export_manifest"],
|
||||
str(run_dir / "exports" / "export-manifest.json"),
|
||||
)
|
||||
self.assertEqual(retention["summary"]["status"], "completed")
|
||||
self.assertEqual(retention["summary"]["artifact_count"], 0)
|
||||
self.assertIn("reports/submission-package.json", retention["report_refs"])
|
||||
self.assertIn("exports/export-manifest.json", retention["export_refs"])
|
||||
self.assertEqual(
|
||||
retention["artifact_retention"]["policy"],
|
||||
{"raw_artifact_days": 0, "summary_days": 365},
|
||||
)
|
||||
submission = json.loads(
|
||||
(run_dir / "reports" / "submission-package.json").read_text(encoding="utf-8")
|
||||
)
|
||||
assert_valid(submission, "submission-package")
|
||||
self.assertEqual(submission["package_identity"]["target_profile_ref"], "sample-repository")
|
||||
self.assertEqual(
|
||||
[entry["path"] for entry in submission["reports"]],
|
||||
["reports/assessment-package.json", "reports/report.md"],
|
||||
)
|
||||
self.assertEqual(
|
||||
[run["run_id"] for run in list_retained_runs(Path(temporary_directory))],
|
||||
[result["run_id"]],
|
||||
@@ -232,6 +361,69 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
self.assertEqual(len(mappings), 1)
|
||||
self.assertEqual(mappings[0]["target_id"], "profile-readiness")
|
||||
|
||||
def test_applies_challenges_and_exclusions_without_hiding_gate_failures(self) -> None:
|
||||
with TemporaryDirectory() as temporary_directory:
|
||||
temp_root = Path(temporary_directory)
|
||||
extension_dir = temp_root / "review-noop"
|
||||
_write_review_extension(extension_dir)
|
||||
target_path = temp_root / "review-target.json"
|
||||
assessment_path = temp_root / "review-assessment.json"
|
||||
challenge_path = temp_root / "review-challenges.json"
|
||||
exclusion_path = temp_root / "review-exclusions.json"
|
||||
_write_review_target(target_path)
|
||||
_write_review_assessment(assessment_path)
|
||||
_write_review_challenges(challenge_path)
|
||||
_write_review_exclusions(exclusion_path)
|
||||
|
||||
result = run_assessment(
|
||||
ROOT,
|
||||
target_path,
|
||||
assessment_path,
|
||||
temp_root / "runs" / "review",
|
||||
[extension_dir],
|
||||
)
|
||||
run_dir = Path(result["run_dir"])
|
||||
evidence = json.loads(
|
||||
(run_dir / "normalized" / "evidence.json").read_text(encoding="utf-8")
|
||||
)["evidence"]
|
||||
assessment_package = json.loads(
|
||||
(run_dir / "reports" / "assessment-package.json").read_text(encoding="utf-8")
|
||||
)
|
||||
retention = json.loads(
|
||||
(run_dir / "retention-summary.json").read_text(encoding="utf-8")
|
||||
)
|
||||
report = (run_dir / "reports" / "report.md").read_text(encoding="utf-8")
|
||||
|
||||
self.assertEqual(result["status"], "blocked")
|
||||
finding = assessment_package["findings"][0]
|
||||
self.assertEqual(finding["challenge_ref"], "challenge-review-blocked")
|
||||
self.assertEqual(finding["exclusion_ref"], "exclusion-review-blocked")
|
||||
self.assertEqual(finding["review_status"], "authority_excluded")
|
||||
self.assertFalse(finding["expected"])
|
||||
self.assertEqual(assessment_package["policy_summary"]["unexpected_findings"], 1)
|
||||
self.assertEqual(assessment_package["policy_summary"]["challenged_findings"], 1)
|
||||
self.assertEqual(assessment_package["policy_summary"]["authority_exclusions"], 1)
|
||||
self.assertEqual(assessment_package["policy_summary"]["unresolved_defects"], 0)
|
||||
self.assertEqual(
|
||||
evidence[1]["review"]["challenge_refs"],
|
||||
["challenge-review-blocked"],
|
||||
)
|
||||
self.assertEqual(
|
||||
evidence[1]["review"]["exclusion_refs"],
|
||||
["exclusion-review-blocked"],
|
||||
)
|
||||
self.assertEqual(assessment_package["challenges"][0]["owner"], "qa")
|
||||
self.assertEqual(assessment_package["exclusions"][0]["authority_ref"], "review-authority")
|
||||
self.assertEqual(retention["summary"]["challenged_findings"], 1)
|
||||
self.assertEqual(retention["summary"]["authority_exclusions"], 1)
|
||||
self.assertEqual(retention["summary"]["unresolved_review_items"], 1)
|
||||
self.assertIn("- authority_excluded: 1", report)
|
||||
|
||||
gate = evaluate_trend_gates(build_trend_summary(temp_root / "runs"))
|
||||
self.assertEqual(gate["status"], "failed")
|
||||
checks = {check["id"]: check for check in gate["groups"][0]["checks"]}
|
||||
self.assertEqual(checks["unexpected-findings"]["observed"], 1)
|
||||
|
||||
def test_serves_local_api_run_lifecycle(self) -> None:
|
||||
with TemporaryDirectory() as temporary_directory:
|
||||
service = start_service(ROOT, host="127.0.0.1", port=0)
|
||||
@@ -293,6 +485,68 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
reports["assessment_package"]["json"]["run_id"],
|
||||
status["result"]["run_id"],
|
||||
)
|
||||
self.assertEqual(
|
||||
reports["submission_package"]["json"]["run_id"],
|
||||
status["result"]["run_id"],
|
||||
)
|
||||
self.assertEqual(
|
||||
reports["export_manifest"]["json"]["run_id"],
|
||||
status["result"]["run_id"],
|
||||
)
|
||||
finally:
|
||||
service.stop()
|
||||
|
||||
def test_service_exposes_retained_runs_after_restart(self) -> None:
|
||||
with TemporaryDirectory() as temporary_directory:
|
||||
runs_dir = Path(temporary_directory) / "runs"
|
||||
result = run_assessment(
|
||||
ROOT,
|
||||
ROOT / "profiles" / "targets" / "sample-repository.json",
|
||||
ROOT / "profiles" / "assessments" / "sample-noop.json",
|
||||
runs_dir / "sample",
|
||||
)
|
||||
_write_unsafe_artifact_run(runs_dir / "unsafe-run")
|
||||
|
||||
service = start_service(ROOT, host="127.0.0.1", port=0)
|
||||
try:
|
||||
query = f"runs_dir={quote(str(runs_dir), safe='')}"
|
||||
listing = _request_json(service, "GET", f"/retained-runs?{query}")
|
||||
self.assertEqual(listing["runs_dir"], str(runs_dir))
|
||||
self.assertIn(result["run_id"], [run["run_id"] for run in listing["runs"]])
|
||||
|
||||
latest = _request_json(
|
||||
service,
|
||||
"GET",
|
||||
f"/retained-runs/latest?{query}&target=sample-repository&assessment=sample-noop-assessment",
|
||||
)
|
||||
self.assertEqual(latest["run"]["run_id"], result["run_id"])
|
||||
self.assertIn("submission_package", latest["run"]["paths"])
|
||||
|
||||
reports = _request_json(
|
||||
service,
|
||||
"GET",
|
||||
f"/retained-runs/{result['run_id']}/reports?{query}",
|
||||
)
|
||||
self.assertEqual(
|
||||
reports["run"]["paths"]["assessment_package"],
|
||||
str(runs_dir / "sample" / "reports" / "assessment-package.json"),
|
||||
)
|
||||
|
||||
artifacts = _request_json(
|
||||
service,
|
||||
"GET",
|
||||
f"/retained-runs/{result['run_id']}/artifact-manifest?{query}",
|
||||
)
|
||||
self.assertEqual(artifacts["artifact_manifest"], [])
|
||||
self.assertEqual(artifacts["compatibility"], "current")
|
||||
|
||||
unsafe = _request_json(
|
||||
service,
|
||||
"GET",
|
||||
f"/retained-runs/unsafe-run/artifact-manifest?{query}",
|
||||
expected_status=400,
|
||||
)
|
||||
self.assertIn("escapes run directory", unsafe["error"]["message"])
|
||||
finally:
|
||||
service.stop()
|
||||
|
||||
@@ -328,7 +582,13 @@ class CoreArchitectureTests(unittest.TestCase):
|
||||
self.assertEqual(group["previous_run"]["run_id"], "run-old")
|
||||
self.assertEqual(group["trend"]["direction"], "improved")
|
||||
self.assertTrue(group["trend"]["status_changed"])
|
||||
self.assertEqual(
|
||||
group["trend"]["status_change"],
|
||||
{"from": "blocked", "to": "completed"},
|
||||
)
|
||||
self.assertEqual(group["trend"]["unexpected_findings_delta"], -1)
|
||||
self.assertEqual(group["trend"]["mapping_target_count_delta"], 0)
|
||||
self.assertIn("Trend improved", group["trend"]["summary_text"])
|
||||
self.assertEqual(
|
||||
group["trend"]["evidence_result_deltas"],
|
||||
{"blocked": -1, "manual": 1, "skipped": 1},
|
||||
@@ -438,6 +698,34 @@ def _write_retention_summary(
|
||||
)
|
||||
|
||||
|
||||
def _write_unsafe_artifact_run(run_dir: Path) -> None:
|
||||
_write_retention_summary(
|
||||
run_dir,
|
||||
"unsafe-run",
|
||||
"2026-05-07T12:00:00+00:00",
|
||||
"completed",
|
||||
{"pass": 1},
|
||||
0,
|
||||
1,
|
||||
)
|
||||
reports_dir = run_dir / "reports"
|
||||
reports_dir.mkdir(parents=True, exist_ok=True)
|
||||
(reports_dir / "assessment-package.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"artifact_manifest": [
|
||||
{
|
||||
"id": "artifact:unsafe",
|
||||
"path": "../outside.txt",
|
||||
"checksum": "sha256:unsafe",
|
||||
}
|
||||
]
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _request_json(
|
||||
service: ServiceHandle,
|
||||
method: str,
|
||||
@@ -640,5 +928,166 @@ def _write_schema_assessment(path: Path, runtime_policy: dict[str, object]) -> N
|
||||
)
|
||||
|
||||
|
||||
def _write_review_extension(extension_dir: Path) -> None:
|
||||
extension_dir.mkdir(parents=True, exist_ok=True)
|
||||
(extension_dir / "extension.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "review-noop",
|
||||
"name": "Review No-op",
|
||||
"version": "0.1.0",
|
||||
"extension_type": "repository_quality",
|
||||
"lifecycle_status": "incubating",
|
||||
"supported_frameworks": ["review.framework.v1"],
|
||||
"authorities": ["review-authority"],
|
||||
"profile_schemas": ["target-profile", "assessment-profile"],
|
||||
"check_groups": [
|
||||
{
|
||||
"id": "review",
|
||||
"name": "Review",
|
||||
"check_type": "repository_quality",
|
||||
"requirement_refs": ["review.requirement"],
|
||||
"runner_ref": "external-review",
|
||||
}
|
||||
],
|
||||
"preflight_runner": None,
|
||||
"runner_entrypoints": [
|
||||
{
|
||||
"id": "external-review",
|
||||
"kind": "external",
|
||||
"module_path": None,
|
||||
"callable": None,
|
||||
"command": None,
|
||||
"metadata": {"test_suite_id": "review-suite"},
|
||||
"description": "External runner used to produce reviewable blocked evidence.",
|
||||
}
|
||||
],
|
||||
"normalizers": [],
|
||||
"mappings": [],
|
||||
"report_fragments": [],
|
||||
"dependencies": [],
|
||||
"restricted_assets": [],
|
||||
"certification_boundary": "Review fixture only.",
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _write_review_target(path: Path) -> None:
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "review-target",
|
||||
"subject_type": "repository",
|
||||
"subject_name": "Review Target",
|
||||
"environment": "test",
|
||||
"scope": ["review"],
|
||||
"endpoints": [],
|
||||
"artifacts": [],
|
||||
"credentials_ref": None,
|
||||
"declared_capabilities": [],
|
||||
"known_gaps": [],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _write_review_assessment(path: Path) -> None:
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "review-assessment",
|
||||
"framework_refs": ["review.framework.v1"],
|
||||
"extension_refs": ["review-noop"],
|
||||
"target_profile_ref": "review-target",
|
||||
"selected_check_groups": {"review-noop": ["review"]},
|
||||
"expectations_ref": None,
|
||||
"waivers_ref": None,
|
||||
"challenges_ref": "review-challenges.json",
|
||||
"exclusions_ref": "review-exclusions.json",
|
||||
"output_policy": {
|
||||
"report_formats": ["json", "markdown"],
|
||||
"artifact_retention": "summary-only",
|
||||
},
|
||||
"retention_policy": {
|
||||
"summary_days": 365,
|
||||
"raw_artifact_days": 0,
|
||||
},
|
||||
"runtime_policy": {
|
||||
"offline": True,
|
||||
"timeout_seconds": 2,
|
||||
},
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _write_review_challenges(path: Path) -> None:
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "review-challenges",
|
||||
"target_profile_ref": "review-target",
|
||||
"challenges": [
|
||||
{
|
||||
"id": "challenge-review-blocked",
|
||||
"requirement_refs": ["review.requirement"],
|
||||
"check_refs": ["check-group:review-noop:review"],
|
||||
"evidence_refs": [],
|
||||
"result_refs": ["blocked"],
|
||||
"classification_refs": ["runner_not_implemented"],
|
||||
"authority_source_refs": ["review-authority:rule-1"],
|
||||
"owner": "qa",
|
||||
"review_status": "open",
|
||||
"rationale": "The external suite is not wired in this fixture.",
|
||||
"created_at": "2026-05-16",
|
||||
"review_due_at": "2026-06-16",
|
||||
"expires_at": None,
|
||||
"native_challenge_id": "native-challenge-1",
|
||||
"metadata": {"kind": "fixture"},
|
||||
}
|
||||
],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _write_review_exclusions(path: Path) -> None:
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"id": "review-exclusions",
|
||||
"target_profile_ref": "review-target",
|
||||
"exclusions": [
|
||||
{
|
||||
"id": "exclusion-review-blocked",
|
||||
"authority_ref": "review-authority",
|
||||
"requirement_refs": ["review.requirement"],
|
||||
"check_refs": ["check-group:review-noop:review"],
|
||||
"evidence_refs": [],
|
||||
"result_refs": ["blocked"],
|
||||
"classification_refs": ["runner_not_implemented"],
|
||||
"authority_source_refs": ["review-authority:rule-1"],
|
||||
"owner": "qa",
|
||||
"approved_by": "authority-reviewer",
|
||||
"review_status": "approved",
|
||||
"rationale": "Fixture demonstrates authority exclusion annotation.",
|
||||
"created_at": "2026-05-16",
|
||||
"review_due_at": "2026-06-16",
|
||||
"expires_at": None,
|
||||
"native_exclusion_id": "native-exclusion-1",
|
||||
"metadata": {"kind": "fixture"},
|
||||
}
|
||||
],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -3,7 +3,8 @@ id: GUIDE-BOARD-WP-0001
|
||||
type: workplan
|
||||
title: "Guide Board Bootstrapping"
|
||||
repo: guide-board
|
||||
domain: markitect
|
||||
domain: communication
|
||||
topic_slug: markitect
|
||||
status: completed
|
||||
owner: codex
|
||||
planning_priority: high
|
||||
|
||||
@@ -3,7 +3,8 @@ id: GUIDE-BOARD-WP-0002
|
||||
type: workplan
|
||||
title: "Assessment Operations Baseline"
|
||||
repo: guide-board
|
||||
domain: markitect
|
||||
domain: communication
|
||||
topic_slug: markitect
|
||||
status: completed
|
||||
owner: codex
|
||||
planning_priority: high
|
||||
|
||||
@@ -3,8 +3,9 @@ id: GUIDE-BOARD-WP-0003
|
||||
type: workplan
|
||||
title: "Extension SDK Maturity"
|
||||
repo: guide-board
|
||||
domain: markitect
|
||||
status: active
|
||||
domain: communication
|
||||
topic_slug: markitect
|
||||
status: completed
|
||||
owner: codex
|
||||
planning_priority: high
|
||||
planning_order: 3
|
||||
@@ -69,7 +70,7 @@ Progress:
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0003-T002
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "b87e68c1-6eca-4274-8e3f-6e2854c5a1e1"
|
||||
```
|
||||
@@ -81,11 +82,22 @@ Acceptance:
|
||||
normalize native result artifacts explicitly.
|
||||
- Add tests that prove a normalizer can map native output into evidence.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `guide_board.normalizers.normalize_step_result`.
|
||||
- Extended `normalizers` manifest entries to support Python module descriptor
|
||||
objects while preserving the existing string shorthand.
|
||||
- Invoked matching normalizers after runner execution and before evidence
|
||||
writing.
|
||||
- Merged normalizer result fields over runner results and recorded
|
||||
`normalizer_refs` in evidence facts.
|
||||
- Added test coverage through the SDK fixture run.
|
||||
|
||||
## D3.3 - SDK Fixture Extension And Acceptance Tests
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0003-T003
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "f3738751-5a0d-4eaf-85b1-75e599a78060"
|
||||
```
|
||||
@@ -97,11 +109,19 @@ Acceptance:
|
||||
- Cover external repo discovery, schema validation, normalizer invocation, plan
|
||||
generation, and result package shape.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `extensions/sdk-fixture`.
|
||||
- Included extension-owned target and assessment schemas, fixture profiles, a
|
||||
native-output runner, a normalizer, and a mapping set.
|
||||
- Added a unit test that copies the fixture as an external extension repository
|
||||
and verifies plan, evidence, mapping, and assessment package output.
|
||||
|
||||
## D3.4 - Extension Authoring Documentation Refresh
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0003-T004
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "3d390bd4-755b-462a-9e16-9c859990d99e"
|
||||
```
|
||||
@@ -114,6 +134,14 @@ Acceptance:
|
||||
- Link the SDK maturity guidance from the assessment operations and external
|
||||
extension acceptance docs where useful.
|
||||
|
||||
Progress:
|
||||
|
||||
- Refreshed `docs/EXTENSION-SDK.md` with profile schema descriptors,
|
||||
normalizer descriptors, context fields, merge semantics, and fixture guidance.
|
||||
- Linked SDK authoring contracts from `docs/ASSESSMENT-OPERATIONS.md`.
|
||||
- Linked `extensions/sdk-fixture` from `docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
|
||||
- Added README references for the SDK fixture and WP3.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
- External extension repositories can declare and test domain-specific profile
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
id: GUIDE-BOARD-WP-0004
|
||||
type: workplan
|
||||
title: "Source Lock And Submission Package Baseline"
|
||||
repo: guide-board
|
||||
domain: communication
|
||||
topic_slug: markitect
|
||||
status: completed
|
||||
owner: codex
|
||||
planning_priority: high
|
||||
planning_order: 4
|
||||
created: "2026-05-15"
|
||||
updated: "2026-05-16"
|
||||
state_hub_workstream_id: "6dd2832b-d1d9-43bc-ad5c-d16f399930dc"
|
||||
---
|
||||
|
||||
# GUIDE-BOARD-WP-0004: Source Lock And Submission Package Baseline
|
||||
|
||||
## Purpose
|
||||
|
||||
Make guide-board assessment packages source-complete enough for serious review.
|
||||
Runs already snapshot target and assessment profiles and preserve normalized
|
||||
evidence. The next maturity layer is a stronger source lock and a submission
|
||||
package manifest that records which framework, extension, harness, mapping,
|
||||
profile, waiver, and artifact sources produced the assessment.
|
||||
|
||||
## Background
|
||||
|
||||
The architecture blueprint calls out source locking as part of credible
|
||||
assessment evidence. A reviewer should be able to distinguish a normal local run
|
||||
from a package that is ready to hand to another team, auditor, authority, or
|
||||
certification-preparation process. This does not turn guide-board into a
|
||||
certification body; it makes the evidence boundary clearer and more portable.
|
||||
|
||||
## Boundary
|
||||
|
||||
This workplan owns extension-neutral source lock fields and package manifest
|
||||
generation. Extension-specific harness version detection, authority-specific
|
||||
submission rules, and licensed or restricted assets remain extension-owned.
|
||||
|
||||
## D4.1 - Source Lock Schema And Capture
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0004-T001
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "d5a7a18f-941b-47b8-9992-2cb54bc5ad06"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Extend the source lock contract beyond framework and extension IDs.
|
||||
- Capture stable references for framework versions, extension versions, mapping
|
||||
sets, target profile snapshots, assessment profile snapshots, expectation
|
||||
sets, waiver sets, and authority source URLs when available.
|
||||
- Keep the schema backward-compatible with existing retained runs.
|
||||
- Add tests for source lock shape and retained run compatibility.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `docs/schemas/source-lock.schema.json`.
|
||||
- Expanded run-plan source locks with framework, extension, mapping-set,
|
||||
profile snapshot, policy-ref, authority, and metadata-hook records.
|
||||
- Preserved the original `framework_refs` and `extension_refs` fields for
|
||||
retained-run compatibility.
|
||||
- Added tests for source-lock shape and older retained summary compatibility.
|
||||
|
||||
## D4.2 - Harness And Extension Metadata Hooks
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0004-T002
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "7abd5a66-5784-41b9-a361-6572290923cc"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Define how extensions and runner or normalizer results can report harness
|
||||
versions, test suite IDs, adapter versions, source URLs, and native result
|
||||
identifiers.
|
||||
- Persist this metadata in run plans, evidence facts, source locks, or package
|
||||
manifests without inventing domain-specific fields in the core.
|
||||
- Preserve current runner and normalizer contracts for extensions that do not
|
||||
provide this metadata yet.
|
||||
- Cover the SDK fixture and at least one no-metadata extension path in tests.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added optional manifest metadata for extensions, authorities, frameworks,
|
||||
runner entrypoints, and normalizers.
|
||||
- Preserved runner and normalizer returned `metadata` and requirement refs.
|
||||
- Recorded merged metadata under evidence `facts.source_metadata`.
|
||||
- Updated `sdk-fixture` to exercise harness, test-suite, adapter, source URL,
|
||||
and native result metadata while keeping `sample-noop` as a no-metadata path.
|
||||
|
||||
## D4.3 - Submission Package Manifest
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0004-T003
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "c54273d6-1fc2-4444-92cf-74f2a5e614ec"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Add a machine-readable submission package manifest under each run report
|
||||
directory.
|
||||
- Include package identity, source lock references, report paths, normalized
|
||||
evidence paths, artifact manifest entries, checksums where available, and the
|
||||
certification boundary.
|
||||
- Keep the manifest useful for both executable harnesses and procedural evidence
|
||||
packs.
|
||||
- Document how this differs from an authority-specific final submission.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `docs/schemas/submission-package.schema.json`.
|
||||
- Wrote `reports/submission-package.json` for each run.
|
||||
- Included package identity, source lock checksum, report paths, normalized
|
||||
output paths, profile snapshots, artifact manifest entries, reported
|
||||
metadata, and the certification boundary.
|
||||
- Exposed the submission manifest path in CLI/service run results and retained
|
||||
report refs.
|
||||
|
||||
## D4.4 - Documentation And Acceptance Tests
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0004-T004
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "ad37baeb-973c-4399-96d0-c9cb7fc6b761"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Update assessment operations, extension SDK, and architecture docs with the
|
||||
source lock and submission package contracts.
|
||||
- Add tests that run a sample or SDK fixture assessment and assert the source
|
||||
lock and manifest outputs.
|
||||
- Include compatibility notes for older retained runs.
|
||||
- Keep the output paths aligned with existing CLI and service result retrieval.
|
||||
|
||||
Progress:
|
||||
|
||||
- Updated assessment operations, extension SDK, architecture blueprint, and
|
||||
README references.
|
||||
- Added focused unit assertions for the sample and SDK fixture assessment
|
||||
outputs.
|
||||
- Kept retained run listing compatible with older `retention-summary.json`
|
||||
files that do not reference a submission manifest.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
- Every new run writes a richer source lock and submission package manifest.
|
||||
- Extension-provided harness metadata has a stable path into the package.
|
||||
- Older retained runs remain readable.
|
||||
- Operators and extension authors know what the package can and cannot claim.
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
id: GUIDE-BOARD-WP-0005
|
||||
type: workplan
|
||||
title: "Challenge And Exclusion Handling"
|
||||
repo: guide-board
|
||||
domain: communication
|
||||
topic_slug: markitect
|
||||
status: completed
|
||||
owner: codex
|
||||
planning_priority: high
|
||||
planning_order: 5
|
||||
created: "2026-05-15"
|
||||
updated: "2026-05-16"
|
||||
state_hub_workstream_id: "fb11e1c7-6c0c-4ec7-a163-da98b2fe9f8f"
|
||||
---
|
||||
|
||||
# GUIDE-BOARD-WP-0005: Challenge And Exclusion Handling
|
||||
|
||||
## Purpose
|
||||
|
||||
Represent authority exclusions, extension challenges, target expectations,
|
||||
waivers, and defects as distinct review concepts. Guide-board already supports
|
||||
expectations and waivers, but real assessments also need a way to record that a
|
||||
test was excluded by an authority, challenged as invalid or mis-mapped, or
|
||||
identified as a target defect.
|
||||
|
||||
## Background
|
||||
|
||||
Conformance work often includes dispute and exclusion paths. A failed check may
|
||||
be a product issue, a harness issue, an unsupported optional feature, a known
|
||||
waived risk, an authority-approved exclusion, or a local challenge awaiting
|
||||
review. Reports need to make those distinctions visible instead of flattening
|
||||
everything into pass, fail, blocked, or waived.
|
||||
|
||||
## Boundary
|
||||
|
||||
This workplan owns guide-board's generic challenge and exclusion model. It does
|
||||
not decide authority-specific challenge rules, certification program policy, or
|
||||
whether a challenge is valid. Extensions may provide authority-specific fields,
|
||||
but the core should preserve them without embedding domain policy.
|
||||
|
||||
## D5.1 - Challenge And Exclusion Data Contracts
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0005-T001
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "6ff4e6f7-bce6-4e7f-a5af-e0c67cfa7e55"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Define schemas for authority exclusions and extension challenges.
|
||||
- Distinguish challenge/exclusion records from expectations and waivers.
|
||||
- Support links to requirement refs, check IDs, evidence IDs, authority source
|
||||
refs, owners, review status, rationale, expiry or review dates, and native
|
||||
challenge IDs where available.
|
||||
- Keep the data contract usable by executable harnesses, hosted suites, and
|
||||
procedural packs.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `docs/schemas/challenge-set.schema.json` and
|
||||
`docs/schemas/exclusion-set.schema.json`.
|
||||
- Added optional `challenges_ref` and `exclusions_ref` assessment profile
|
||||
fields.
|
||||
- Supported requirement, check, evidence, result, classification, authority
|
||||
source, owner, review status, rationale, review date, expiry, native ID, and
|
||||
metadata fields.
|
||||
|
||||
## D5.2 - Policy Application And Finding Annotation
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0005-T002
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "fd384bd3-40c4-4344-8b7d-cb123dbf2cac"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Load challenge and exclusion references from assessment profiles.
|
||||
- Annotate findings and evidence with challenge or exclusion state without
|
||||
silently hiding unexpected failures.
|
||||
- Preserve separate counts for expected findings, waived findings, challenged
|
||||
findings, authority exclusions, and unresolved defects.
|
||||
- Add tests that prove challenge and exclusion records affect reporting without
|
||||
corrupting gate semantics.
|
||||
|
||||
Progress:
|
||||
|
||||
- Loaded challenge and exclusion refs through the policy layer.
|
||||
- Annotated findings with challenge refs, exclusion refs, and review status.
|
||||
- Annotated matching evidence with review refs.
|
||||
- Kept default `unexpected_findings` gate semantics visible unless a finding is
|
||||
separately expected or waived.
|
||||
- Added tests proving challenged and excluded findings remain gate-visible.
|
||||
|
||||
## D5.3 - Report Visibility And Review Workflow
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0005-T003
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "791071c0-8a9a-462b-83b3-75548bb8524f"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Show challenge, exclusion, waiver, expectation, and defect distinctions in
|
||||
Markdown and JSON assessment packages.
|
||||
- Make unresolved review items easy to find in retained run summaries.
|
||||
- Provide CLI history or report helpers that expose review state for the latest
|
||||
run.
|
||||
- Document how an operator should treat challenged or excluded findings.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added Markdown report review summaries.
|
||||
- Added challenge, exclusion, unresolved defect, and unresolved review counts to
|
||||
retention summaries and trend projections.
|
||||
- Included applied challenge and exclusion records in JSON assessment packages.
|
||||
- Exposed review counts through existing retained run helpers.
|
||||
|
||||
## D5.4 - Tests And Documentation
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0005-T004
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "43b966da-af8d-479b-93bd-6b6741fdab37"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Add schema tests and policy tests for challenges and exclusions.
|
||||
- Add a sample or SDK fixture scenario that produces at least one challenged or
|
||||
excluded finding.
|
||||
- Update assessment operations, extension SDK, and compliance evidence pack docs.
|
||||
- Keep certification boundary language explicit.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added focused schema and policy tests through a fixture extension scenario.
|
||||
- Updated assessment operations, extension SDK, compliance evidence pack, and
|
||||
architecture docs.
|
||||
- Kept boundary language explicit: challenges and exclusions are review state,
|
||||
not certification conclusions.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
- The core has separate, tested concepts for expectations, waivers, challenges,
|
||||
authority exclusions, and defects.
|
||||
- Reports surface those concepts without overstating certification conclusions.
|
||||
- Retained summaries expose unresolved review work.
|
||||
- Extension authors know how to supply authority-specific challenge metadata.
|
||||
@@ -0,0 +1,151 @@
|
||||
---
|
||||
id: GUIDE-BOARD-WP-0006
|
||||
type: workplan
|
||||
title: "Service Artifact Access And Durable Run Index"
|
||||
repo: guide-board
|
||||
domain: communication
|
||||
topic_slug: markitect
|
||||
status: completed
|
||||
owner: codex
|
||||
planning_priority: medium
|
||||
planning_order: 6
|
||||
created: "2026-05-15"
|
||||
updated: "2026-05-16"
|
||||
state_hub_workstream_id: "ba008283-1631-467b-868e-1052c3870ab9"
|
||||
---
|
||||
|
||||
# GUIDE-BOARD-WP-0006: Service Artifact Access And Durable Run Index
|
||||
|
||||
## Purpose
|
||||
|
||||
Move the local service beyond in-memory job visibility while preserving the CLI
|
||||
as the execution source of truth. Operators and future UI clients should be able
|
||||
to discover retained runs, retrieve reports, inspect artifacts, and recover
|
||||
after service restarts without needing private knowledge of the run directory.
|
||||
|
||||
## Background
|
||||
|
||||
WP2 intentionally kept service job state in memory and documented run
|
||||
directories as the durable source. That is a good baseline, but the service
|
||||
already wraps run start, status, and report retrieval. The next step is to expose
|
||||
the durable run history and artifact paths directly through service contracts.
|
||||
|
||||
## Boundary
|
||||
|
||||
This workplan owns local service API and durable run index behavior. It should
|
||||
not change CLI run semantics, introduce a database dependency, or create a
|
||||
distributed execution service. Any durable index must be reconstructable from
|
||||
existing run artifacts.
|
||||
|
||||
## D6.1 - Durable Run Index Design
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0006-T001
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "4d392fc5-6a1c-46f7-9cbf-6c02bbd744c6"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Decide whether the service needs a separate durable index file or can rely on
|
||||
retained run summaries with helper scans.
|
||||
- Define reconstruction behavior after service restart.
|
||||
- Preserve compatibility with existing `retention-summary.json` and run
|
||||
directory layout.
|
||||
- Document the operational tradeoff and failure modes.
|
||||
|
||||
Decision:
|
||||
|
||||
- Keep the durable index as retained run summaries and helper scans.
|
||||
- Do not add a separate service index file for the baseline.
|
||||
|
||||
Progress:
|
||||
|
||||
- Documented reconstruction from `retention-summary.json` files.
|
||||
- Kept compatibility with older runs that lack newer assessment package or
|
||||
submission manifest files.
|
||||
|
||||
## D6.2 - Service Run History And Artifact Endpoints
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0006-T002
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "8f209920-6b14-4d6f-bfa1-8f1d03bcdbf1"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Add service endpoints for retained run listing, latest run selection, report
|
||||
path lookup, and artifact manifest access.
|
||||
- Keep endpoint responses aligned with existing CLI `runs` commands.
|
||||
- Avoid serving arbitrary filesystem paths outside configured run directories.
|
||||
- Add tests for successful retrieval and path-safety failures.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `GET /retained-runs`.
|
||||
- Added `GET /retained-runs/latest`.
|
||||
- Added `GET /retained-runs/{run_id}/reports`.
|
||||
- Added `GET /retained-runs/{run_id}/artifact-manifest`.
|
||||
- Added path containment checks for report refs and artifact manifest paths.
|
||||
- Added service tests for retained history retrieval after a fresh service
|
||||
process and unsafe artifact path rejection.
|
||||
|
||||
## D6.3 - Restart Recovery And Compatibility
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0006-T003
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "0857e7d8-3d23-4426-b7fa-73362d7041a0"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Prove that a service restart can still expose retained run reports and
|
||||
artifacts.
|
||||
- Keep in-memory job status semantics clear for currently running jobs.
|
||||
- Add compatibility handling for older run directories that lack newer manifest
|
||||
files.
|
||||
- Update service durability documentation with examples.
|
||||
|
||||
Progress:
|
||||
|
||||
- Preserved `/runs` as in-memory job history.
|
||||
- Exposed durable run results through retained-run endpoints after restart.
|
||||
- Returned a compatibility marker when an older retained run lacks an
|
||||
assessment package artifact manifest.
|
||||
- Updated service durability and local API docs.
|
||||
|
||||
## D6.4 - Container And Service Acceptance Tests
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0006-T004
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "900a70fa-65ff-4815-9c0c-31f0da4019f0"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Add focused service tests for durable run lookup and artifact/report retrieval.
|
||||
- Extend container or scripted acceptance to prove mounted run directories remain
|
||||
readable through service contracts.
|
||||
- Document service endpoint usage in local and container modes.
|
||||
- Keep tests dependency-light.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added dependency-light service tests for durable run lookup, report paths, and
|
||||
artifact manifest retrieval.
|
||||
- Updated container smoke artifact expectations for current run outputs.
|
||||
- Documented retained-run endpoint usage in local and container modes.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
- The local service can expose retained runs and artifacts after restart.
|
||||
- Endpoint behavior matches CLI run history semantics.
|
||||
- Filesystem access is constrained to intended run outputs.
|
||||
- Operators have documented recovery and artifact retrieval paths.
|
||||
150
workplans/GUIDE-BOARD-WP-0007-report-and-export-maturity.md
Normal file
150
workplans/GUIDE-BOARD-WP-0007-report-and-export-maturity.md
Normal file
@@ -0,0 +1,150 @@
|
||||
---
|
||||
id: GUIDE-BOARD-WP-0007
|
||||
type: workplan
|
||||
title: "Report And Export Maturity"
|
||||
repo: guide-board
|
||||
domain: communication
|
||||
topic_slug: markitect
|
||||
status: completed
|
||||
owner: codex
|
||||
planning_priority: medium
|
||||
planning_order: 7
|
||||
created: "2026-05-15"
|
||||
updated: "2026-05-16"
|
||||
state_hub_workstream_id: "ef9351d2-e99c-470e-aeec-f17aa51eae14"
|
||||
---
|
||||
|
||||
# GUIDE-BOARD-WP-0007: Report And Export Maturity
|
||||
|
||||
## Purpose
|
||||
|
||||
Improve guide-board's human and machine-readable reporting so assessment output
|
||||
is easier to review, compare, and hand off. The current JSON assessment package
|
||||
and Markdown report are useful baselines. The next layer should support
|
||||
extension report fragments, portable export formats, and stronger trend and gate
|
||||
summaries.
|
||||
|
||||
## Background
|
||||
|
||||
Real assessment consumers need different report surfaces. Engineers need concise
|
||||
run feedback, product teams need readiness summaries, compliance reviewers need
|
||||
traceable evidence and boundaries, and external systems may need portable JSON
|
||||
or later OSCAL-style interchange. This workplan keeps those surfaces derived
|
||||
from the same evidence package.
|
||||
|
||||
## Boundary
|
||||
|
||||
This workplan owns generic report and export mechanics. It does not implement
|
||||
authority-specific final submission formats unless they can be represented as
|
||||
extension-provided fragments or exporters.
|
||||
|
||||
## D7.1 - Report Fragment Plug-in Contract
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0007-T001
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "bf3fe163-b06d-4c2e-9b45-31721864e1f2"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Define how extensions declare Markdown or structured report fragments.
|
||||
- Load fragments safely from extension roots and include them in run reports
|
||||
without allowing arbitrary file access.
|
||||
- Give fragments access to the assessment package, evidence, mappings, policy
|
||||
summary, and source lock data.
|
||||
- Add a fixture fragment and tests.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added Markdown file and Python module report fragment descriptors.
|
||||
- Loaded fragment paths safely from extension roots.
|
||||
- Added fragment context for assessment package, evidence, findings, mappings,
|
||||
policy summary, and source lock data.
|
||||
- Added an SDK fixture Python report fragment and focused tests.
|
||||
|
||||
## D7.2 - Portable Export Formats
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0007-T002
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "fda51e62-98aa-408e-a057-4db40fe7c644"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Add one or more portable export outputs derived from the assessment package.
|
||||
- Start with a stable JSON export manifest before considering OSCAL or other
|
||||
external interchange formats.
|
||||
- Preserve certification boundary and source lock references in each export.
|
||||
- Document which exports are generic and which must remain extension-owned.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added `docs/schemas/export-manifest.schema.json`.
|
||||
- Wrote `exports/export-manifest.json` for each run.
|
||||
- Included source package refs, source lock refs, summaries, policy summaries,
|
||||
mapping summaries, report fragments, counts, and certification boundary.
|
||||
- Documented the generic export boundary; authority-specific formats remain
|
||||
extension-owned.
|
||||
|
||||
## D7.3 - Trend And Gate Report Improvements
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0007-T003
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "33c3089a-9d5e-4605-89c4-a1e070bc12ad"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Improve retained trend and gate outputs for human review.
|
||||
- Surface status changes, unexpected finding deltas, mapping target changes, and
|
||||
unresolved review items.
|
||||
- Keep machine-readable gate summaries stable for automation.
|
||||
- Add CLI report helpers or Markdown summaries where useful.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added trend status-change details, unresolved review deltas, mapping target
|
||||
deltas, and compact summary text.
|
||||
- Kept existing gate summary shape stable while improving trend inputs for
|
||||
human review.
|
||||
- Added assertions for the richer trend output.
|
||||
|
||||
## D7.4 - Golden Fixtures And Documentation
|
||||
|
||||
```task
|
||||
id: GUIDE-BOARD-WP-0007-T004
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "66669f68-6728-4484-9ec9-267ffe025027"
|
||||
```
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Add golden fixture outputs for reports and exports.
|
||||
- Document report fragment and export authoring in the extension SDK.
|
||||
- Update assessment operations with report/export retrieval examples.
|
||||
- Ensure report text remains clear about preparation evidence versus formal
|
||||
certification.
|
||||
|
||||
Progress:
|
||||
|
||||
- Added golden fixture expectations for report fragment text and export manifest
|
||||
top-level shape.
|
||||
- Updated extension SDK, assessment operations, architecture, service, container,
|
||||
and README docs.
|
||||
- Kept report and export boundary language tied to preparation evidence, not
|
||||
formal certification.
|
||||
|
||||
## Definition Of Done
|
||||
|
||||
- Extensions can contribute report fragments through a documented contract.
|
||||
- The core emits portable export artifacts derived from the assessment package.
|
||||
- Trend and gate summaries are more useful to humans without breaking
|
||||
automation.
|
||||
- Golden fixtures guard report and export shape.
|
||||
Reference in New Issue
Block a user