generated from coulomb/repo-seed
Add challenge and exclusion review handling
This commit is contained in:
@@ -803,6 +803,9 @@ Use separate concepts:
|
||||
- defect: unexpected product or process failure.
|
||||
|
||||
The report must make these visible separately.
|
||||
The current policy layer loads challenge and exclusion refs from assessment
|
||||
profiles, annotates findings and evidence, and keeps `unexpected_findings`
|
||||
visible for gate semantics unless a finding is separately expected or waived.
|
||||
|
||||
### Source Locking
|
||||
|
||||
|
||||
@@ -27,7 +27,8 @@ Every run needs:
|
||||
|
||||
The target profile describes the candidate system or artifact being assessed.
|
||||
The assessment profile selects frameworks, extensions, check groups, runtime
|
||||
policy, waivers, expectations, and output policy.
|
||||
policy, expectations, waivers, challenges, authority exclusions, and output
|
||||
policy.
|
||||
|
||||
## CLI Flow
|
||||
|
||||
@@ -99,10 +100,10 @@ artifacts/
|
||||
```
|
||||
|
||||
`sources.lock.json` records the framework refs, extension versions, mapping
|
||||
sets, profile snapshots, policy refs, authority refs, and extension metadata
|
||||
hooks used for the run. `reports/submission-package.json` points at the
|
||||
reviewable package files, includes checksums where files exist, carries the raw
|
||||
artifact manifest, and repeats the certification boundary. It is a portable
|
||||
sets, profile snapshots, policy and review refs, authority refs, and extension
|
||||
metadata hooks used for the run. `reports/submission-package.json` points at
|
||||
the reviewable package files, includes checksums where files exist, carries the
|
||||
raw artifact manifest, and repeats the certification boundary. It is a portable
|
||||
handoff manifest for preparation evidence, not an authority-specific final
|
||||
submission.
|
||||
|
||||
@@ -200,6 +201,23 @@ Individual evidence items use:
|
||||
- `expected_gap`
|
||||
- `infrastructure_error`
|
||||
|
||||
## Review State
|
||||
|
||||
Assessment profiles may reference:
|
||||
|
||||
- `expectations_ref`: known target posture, optional scope, or accepted gaps,
|
||||
- `waivers_ref`: approved, time-bounded exceptions,
|
||||
- `challenges_ref`: review claims that a finding, check, mapping, or native
|
||||
result should be challenged,
|
||||
- `exclusions_ref`: authority or program exclusions that apply to selected
|
||||
findings.
|
||||
|
||||
Challenges and exclusions annotate findings and evidence. They do not silently
|
||||
turn failures into passing evidence and they do not reduce the
|
||||
`unexpected_findings` count used by default gates. Retained summaries expose
|
||||
separate counts for expected findings, waived findings, challenged findings,
|
||||
authority exclusions, unresolved defects, and unresolved review items.
|
||||
|
||||
## Candidate System Checklist
|
||||
|
||||
Before starting a run against candidate software, confirm:
|
||||
|
||||
@@ -8,8 +8,8 @@ Created: 2026-05-07
|
||||
Compliance evidence packs cover frameworks where guide-board cannot rely on an
|
||||
official executable harness. They help prepare and perform assessments by
|
||||
organizing evidence requests, expected artifacts, reviewer workflow, waivers,
|
||||
and run reports. They do not replace auditors, accredited certification bodies,
|
||||
legal counsel, or official standard text.
|
||||
challenges, authority exclusions, and run reports. They do not replace auditors,
|
||||
accredited certification bodies, legal counsel, or official standard text.
|
||||
|
||||
Examples include GDPR, SOC 2, HIPAA, NF Z 42-013, NF 461, ISO 14641, ISO 15489,
|
||||
and similar procedural or control-oriented frameworks.
|
||||
@@ -83,7 +83,7 @@ Each request should include:
|
||||
|
||||
Requests should be phrased as collection guidance, not as legal conclusions.
|
||||
|
||||
## Waivers And Expected Gaps
|
||||
## Review Policy Records
|
||||
|
||||
Evidence packs use the same expectation and waiver model as executable
|
||||
extensions.
|
||||
@@ -103,6 +103,16 @@ Use waivers for:
|
||||
|
||||
Every waiver should include owner, reason, approval status, and expiry.
|
||||
|
||||
Use challenges for disputed checks, disputed mappings, imported native result
|
||||
questions, or evidence that needs a reviewer decision before it can be treated
|
||||
as a defect. Use authority exclusions only when a program, standard, or
|
||||
authorized reviewer excludes a requirement or check from the assessment scope.
|
||||
Both records should cite stable requirement refs, check refs, evidence refs, or
|
||||
authority source refs rather than reproducing restricted standard text.
|
||||
|
||||
Challenges and exclusions make review state visible; they do not by themselves
|
||||
claim compliance or remove default gate-visible unexpected findings.
|
||||
|
||||
## Framework Notes
|
||||
|
||||
GDPR packs should emphasize processing inventory, lawful basis records, data
|
||||
@@ -129,6 +139,7 @@ extensions:
|
||||
|
||||
- normalized evidence,
|
||||
- findings,
|
||||
- review annotations for expectations, waivers, challenges, and exclusions,
|
||||
- mapping records,
|
||||
- assessment packages,
|
||||
- retention summaries,
|
||||
|
||||
@@ -250,6 +250,33 @@ Expectation sets mark known posture as expected. Waiver sets mark approved,
|
||||
time-bounded exceptions. Both are applied after findings are generated, and the
|
||||
assessment package records policy summary counts.
|
||||
|
||||
## Challenges And Authority Exclusions
|
||||
|
||||
Assessment profiles may also reference challenge and exclusion sets:
|
||||
|
||||
```json
|
||||
{
|
||||
"challenges_ref": "profiles/challenges/example.json",
|
||||
"exclusions_ref": "profiles/exclusions/example.json"
|
||||
}
|
||||
```
|
||||
|
||||
Challenge sets validate against `docs/schemas/challenge-set.schema.json`.
|
||||
Exclusion sets validate against `docs/schemas/exclusion-set.schema.json`.
|
||||
Records can match findings by requirement refs, check refs, evidence refs,
|
||||
result refs, or classification refs. They also carry owner, review status,
|
||||
rationale, authority source refs, review dates, optional expiry, native IDs,
|
||||
and free-form metadata.
|
||||
|
||||
Use challenges when an extension author or assessment team believes a finding
|
||||
needs review because a check is invalid, a native harness result is disputed, or
|
||||
a mapping is wrong. Use exclusions when an authority or program explicitly
|
||||
removes a requirement, check, or result from the assessment scope. The core
|
||||
preserves these distinctions in findings, evidence review annotations,
|
||||
assessment packages, reports, and retained summaries, but default gate semantics
|
||||
still count the underlying finding as unexpected unless it is separately
|
||||
expected or waived.
|
||||
|
||||
## Python Runner Contract
|
||||
|
||||
A Python runner receives one context object and returns one result object.
|
||||
|
||||
@@ -17,6 +17,8 @@
|
||||
"evidence_refs",
|
||||
"artifact_manifest",
|
||||
"waivers",
|
||||
"challenges",
|
||||
"exclusions",
|
||||
"certification_boundary",
|
||||
"created_at"
|
||||
],
|
||||
@@ -34,6 +36,8 @@
|
||||
"evidence_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"artifact_manifest": { "type": "array", "items": { "type": "object" } },
|
||||
"waivers": { "type": "array", "items": { "type": "object" } },
|
||||
"challenges": { "type": "array", "items": { "type": "object" } },
|
||||
"exclusions": { "type": "array", "items": { "type": "object" } },
|
||||
"certification_boundary": { "type": "string" },
|
||||
"created_at": { "type": "string" }
|
||||
}
|
||||
|
||||
@@ -28,6 +28,8 @@
|
||||
},
|
||||
"expectations_ref": { "type": ["string", "null"] },
|
||||
"waivers_ref": { "type": ["string", "null"] },
|
||||
"challenges_ref": { "type": ["string", "null"] },
|
||||
"exclusions_ref": { "type": ["string", "null"] },
|
||||
"output_policy": { "type": "object" },
|
||||
"retention_policy": { "type": "object" },
|
||||
"runtime_policy": { "type": "object" }
|
||||
|
||||
56
docs/schemas/challenge-set.schema.json
Normal file
56
docs/schemas/challenge-set.schema.json
Normal file
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Guide Board Challenge Set",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"target_profile_ref",
|
||||
"challenges"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"target_profile_ref": { "type": "string" },
|
||||
"challenges": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"requirement_refs",
|
||||
"check_refs",
|
||||
"evidence_refs",
|
||||
"result_refs",
|
||||
"classification_refs",
|
||||
"authority_source_refs",
|
||||
"owner",
|
||||
"review_status",
|
||||
"rationale",
|
||||
"created_at",
|
||||
"review_due_at",
|
||||
"expires_at",
|
||||
"native_challenge_id",
|
||||
"metadata"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"requirement_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"check_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"evidence_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"result_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"classification_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"authority_source_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"owner": { "type": "string" },
|
||||
"review_status": { "type": "string" },
|
||||
"rationale": { "type": "string" },
|
||||
"created_at": { "type": "string" },
|
||||
"review_due_at": { "type": ["string", "null"] },
|
||||
"expires_at": { "type": ["string", "null"] },
|
||||
"native_challenge_id": { "type": ["string", "null"] },
|
||||
"metadata": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -42,6 +42,7 @@
|
||||
},
|
||||
"observations": { "type": "array", "items": { "type": "string" } },
|
||||
"facts": { "type": "object" },
|
||||
"review": { "type": "object" },
|
||||
"requirement_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"artifact_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"started_at": { "type": "string" },
|
||||
|
||||
60
docs/schemas/exclusion-set.schema.json
Normal file
60
docs/schemas/exclusion-set.schema.json
Normal file
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Guide Board Authority Exclusion Set",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"target_profile_ref",
|
||||
"exclusions"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"target_profile_ref": { "type": "string" },
|
||||
"exclusions": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"authority_ref",
|
||||
"requirement_refs",
|
||||
"check_refs",
|
||||
"evidence_refs",
|
||||
"result_refs",
|
||||
"classification_refs",
|
||||
"authority_source_refs",
|
||||
"owner",
|
||||
"approved_by",
|
||||
"review_status",
|
||||
"rationale",
|
||||
"created_at",
|
||||
"review_due_at",
|
||||
"expires_at",
|
||||
"native_exclusion_id",
|
||||
"metadata"
|
||||
],
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"authority_ref": { "type": "string" },
|
||||
"requirement_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"check_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"evidence_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"result_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"classification_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"authority_source_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"owner": { "type": "string" },
|
||||
"approved_by": { "type": ["string", "null"] },
|
||||
"review_status": { "type": "string" },
|
||||
"rationale": { "type": "string" },
|
||||
"created_at": { "type": "string" },
|
||||
"review_due_at": { "type": ["string", "null"] },
|
||||
"expires_at": { "type": ["string", "null"] },
|
||||
"native_exclusion_id": { "type": ["string", "null"] },
|
||||
"metadata": { "type": "object" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,7 +14,10 @@
|
||||
"evidence_refs",
|
||||
"expected",
|
||||
"waiver_ref",
|
||||
"challenge_ref",
|
||||
"exclusion_ref",
|
||||
"policy_ref",
|
||||
"review_status",
|
||||
"remediation"
|
||||
],
|
||||
"properties": {
|
||||
@@ -28,7 +31,10 @@
|
||||
"evidence_refs": { "type": "array", "items": { "type": "string" } },
|
||||
"expected": { "type": "boolean" },
|
||||
"waiver_ref": { "type": ["string", "null"] },
|
||||
"challenge_ref": { "type": ["string", "null"] },
|
||||
"exclusion_ref": { "type": ["string", "null"] },
|
||||
"policy_ref": { "type": ["string", "null"] },
|
||||
"review_status": { "type": "string" },
|
||||
"remediation": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user