Add challenge and exclusion review handling

This commit is contained in:
2026-05-16 02:58:18 +02:00
parent c8ac42154c
commit b1dff0440d
16 changed files with 644 additions and 21 deletions

View File

@@ -803,6 +803,9 @@ Use separate concepts:
- defect: unexpected product or process failure.
The report must make these visible separately.
The current policy layer loads challenge and exclusion refs from assessment
profiles, annotates findings and evidence, and keeps `unexpected_findings`
visible for gate semantics unless a finding is separately expected or waived.
### Source Locking

View File

@@ -27,7 +27,8 @@ Every run needs:
The target profile describes the candidate system or artifact being assessed.
The assessment profile selects frameworks, extensions, check groups, runtime
policy, waivers, expectations, and output policy.
policy, expectations, waivers, challenges, authority exclusions, and output
policy.
## CLI Flow
@@ -99,10 +100,10 @@ artifacts/
```
`sources.lock.json` records the framework refs, extension versions, mapping
sets, profile snapshots, policy refs, authority refs, and extension metadata
hooks used for the run. `reports/submission-package.json` points at the
reviewable package files, includes checksums where files exist, carries the raw
artifact manifest, and repeats the certification boundary. It is a portable
sets, profile snapshots, policy and review refs, authority refs, and extension
metadata hooks used for the run. `reports/submission-package.json` points at
the reviewable package files, includes checksums where files exist, carries the
raw artifact manifest, and repeats the certification boundary. It is a portable
handoff manifest for preparation evidence, not an authority-specific final
submission.
@@ -200,6 +201,23 @@ Individual evidence items use:
- `expected_gap`
- `infrastructure_error`
## Review State
Assessment profiles may reference:
- `expectations_ref`: known target posture, optional scope, or accepted gaps,
- `waivers_ref`: approved, time-bounded exceptions,
- `challenges_ref`: review claims that a finding, check, mapping, or native
result should be challenged,
- `exclusions_ref`: authority or program exclusions that apply to selected
findings.
Challenges and exclusions annotate findings and evidence. They do not silently
turn failures into passing evidence and they do not reduce the
`unexpected_findings` count used by default gates. Retained summaries expose
separate counts for expected findings, waived findings, challenged findings,
authority exclusions, unresolved defects, and unresolved review items.
## Candidate System Checklist
Before starting a run against candidate software, confirm:

View File

@@ -8,8 +8,8 @@ Created: 2026-05-07
Compliance evidence packs cover frameworks where guide-board cannot rely on an
official executable harness. They help prepare and perform assessments by
organizing evidence requests, expected artifacts, reviewer workflow, waivers,
and run reports. They do not replace auditors, accredited certification bodies,
legal counsel, or official standard text.
challenges, authority exclusions, and run reports. They do not replace auditors,
accredited certification bodies, legal counsel, or official standard text.
Examples include GDPR, SOC 2, HIPAA, NF Z 42-013, NF 461, ISO 14641, ISO 15489,
and similar procedural or control-oriented frameworks.
@@ -83,7 +83,7 @@ Each request should include:
Requests should be phrased as collection guidance, not as legal conclusions.
## Waivers And Expected Gaps
## Review Policy Records
Evidence packs use the same expectation and waiver model as executable
extensions.
@@ -103,6 +103,16 @@ Use waivers for:
Every waiver should include owner, reason, approval status, and expiry.
Use challenges for disputed checks, disputed mappings, imported native result
questions, or evidence that needs a reviewer decision before it can be treated
as a defect. Use authority exclusions only when a program, standard, or
authorized reviewer excludes a requirement or check from the assessment scope.
Both records should cite stable requirement refs, check refs, evidence refs, or
authority source refs rather than reproducing restricted standard text.
Challenges and exclusions make review state visible; they do not by themselves
claim compliance or remove default gate-visible unexpected findings.
## Framework Notes
GDPR packs should emphasize processing inventory, lawful basis records, data
@@ -129,6 +139,7 @@ extensions:
- normalized evidence,
- findings,
- review annotations for expectations, waivers, challenges, and exclusions,
- mapping records,
- assessment packages,
- retention summaries,

View File

@@ -250,6 +250,33 @@ Expectation sets mark known posture as expected. Waiver sets mark approved,
time-bounded exceptions. Both are applied after findings are generated, and the
assessment package records policy summary counts.
## Challenges And Authority Exclusions
Assessment profiles may also reference challenge and exclusion sets:
```json
{
"challenges_ref": "profiles/challenges/example.json",
"exclusions_ref": "profiles/exclusions/example.json"
}
```
Challenge sets validate against `docs/schemas/challenge-set.schema.json`.
Exclusion sets validate against `docs/schemas/exclusion-set.schema.json`.
Records can match findings by requirement refs, check refs, evidence refs,
result refs, or classification refs. They also carry owner, review status,
rationale, authority source refs, review dates, optional expiry, native IDs,
and free-form metadata.
Use challenges when an extension author or assessment team believes a finding
needs review because a check is invalid, a native harness result is disputed, or
a mapping is wrong. Use exclusions when an authority or program explicitly
removes a requirement, check, or result from the assessment scope. The core
preserves these distinctions in findings, evidence review annotations,
assessment packages, reports, and retained summaries, but default gate semantics
still count the underlying finding as unexpected unless it is separately
expected or waived.
## Python Runner Contract
A Python runner receives one context object and returns one result object.

View File

@@ -17,6 +17,8 @@
"evidence_refs",
"artifact_manifest",
"waivers",
"challenges",
"exclusions",
"certification_boundary",
"created_at"
],
@@ -34,6 +36,8 @@
"evidence_refs": { "type": "array", "items": { "type": "string" } },
"artifact_manifest": { "type": "array", "items": { "type": "object" } },
"waivers": { "type": "array", "items": { "type": "object" } },
"challenges": { "type": "array", "items": { "type": "object" } },
"exclusions": { "type": "array", "items": { "type": "object" } },
"certification_boundary": { "type": "string" },
"created_at": { "type": "string" }
}

View File

@@ -28,6 +28,8 @@
},
"expectations_ref": { "type": ["string", "null"] },
"waivers_ref": { "type": ["string", "null"] },
"challenges_ref": { "type": ["string", "null"] },
"exclusions_ref": { "type": ["string", "null"] },
"output_policy": { "type": "object" },
"retention_policy": { "type": "object" },
"runtime_policy": { "type": "object" }

View File

@@ -0,0 +1,56 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Guide Board Challenge Set",
"type": "object",
"additionalProperties": false,
"required": [
"id",
"target_profile_ref",
"challenges"
],
"properties": {
"id": { "type": "string" },
"target_profile_ref": { "type": "string" },
"challenges": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"id",
"requirement_refs",
"check_refs",
"evidence_refs",
"result_refs",
"classification_refs",
"authority_source_refs",
"owner",
"review_status",
"rationale",
"created_at",
"review_due_at",
"expires_at",
"native_challenge_id",
"metadata"
],
"properties": {
"id": { "type": "string" },
"requirement_refs": { "type": "array", "items": { "type": "string" } },
"check_refs": { "type": "array", "items": { "type": "string" } },
"evidence_refs": { "type": "array", "items": { "type": "string" } },
"result_refs": { "type": "array", "items": { "type": "string" } },
"classification_refs": { "type": "array", "items": { "type": "string" } },
"authority_source_refs": { "type": "array", "items": { "type": "string" } },
"owner": { "type": "string" },
"review_status": { "type": "string" },
"rationale": { "type": "string" },
"created_at": { "type": "string" },
"review_due_at": { "type": ["string", "null"] },
"expires_at": { "type": ["string", "null"] },
"native_challenge_id": { "type": ["string", "null"] },
"metadata": { "type": "object" }
}
}
}
}
}

View File

@@ -42,6 +42,7 @@
},
"observations": { "type": "array", "items": { "type": "string" } },
"facts": { "type": "object" },
"review": { "type": "object" },
"requirement_refs": { "type": "array", "items": { "type": "string" } },
"artifact_refs": { "type": "array", "items": { "type": "string" } },
"started_at": { "type": "string" },

View File

@@ -0,0 +1,60 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Guide Board Authority Exclusion Set",
"type": "object",
"additionalProperties": false,
"required": [
"id",
"target_profile_ref",
"exclusions"
],
"properties": {
"id": { "type": "string" },
"target_profile_ref": { "type": "string" },
"exclusions": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"id",
"authority_ref",
"requirement_refs",
"check_refs",
"evidence_refs",
"result_refs",
"classification_refs",
"authority_source_refs",
"owner",
"approved_by",
"review_status",
"rationale",
"created_at",
"review_due_at",
"expires_at",
"native_exclusion_id",
"metadata"
],
"properties": {
"id": { "type": "string" },
"authority_ref": { "type": "string" },
"requirement_refs": { "type": "array", "items": { "type": "string" } },
"check_refs": { "type": "array", "items": { "type": "string" } },
"evidence_refs": { "type": "array", "items": { "type": "string" } },
"result_refs": { "type": "array", "items": { "type": "string" } },
"classification_refs": { "type": "array", "items": { "type": "string" } },
"authority_source_refs": { "type": "array", "items": { "type": "string" } },
"owner": { "type": "string" },
"approved_by": { "type": ["string", "null"] },
"review_status": { "type": "string" },
"rationale": { "type": "string" },
"created_at": { "type": "string" },
"review_due_at": { "type": ["string", "null"] },
"expires_at": { "type": ["string", "null"] },
"native_exclusion_id": { "type": ["string", "null"] },
"metadata": { "type": "object" }
}
}
}
}
}

View File

@@ -14,7 +14,10 @@
"evidence_refs",
"expected",
"waiver_ref",
"challenge_ref",
"exclusion_ref",
"policy_ref",
"review_status",
"remediation"
],
"properties": {
@@ -28,7 +31,10 @@
"evidence_refs": { "type": "array", "items": { "type": "string" } },
"expected": { "type": "boolean" },
"waiver_ref": { "type": ["string", "null"] },
"challenge_ref": { "type": ["string", "null"] },
"exclusion_ref": { "type": ["string", "null"] },
"policy_ref": { "type": ["string", "null"] },
"review_status": { "type": "string" },
"remediation": { "type": ["string", "null"] }
}
}