generated from coulomb/repo-seed
261 lines
8.6 KiB
Markdown
261 lines
8.6 KiB
Markdown
# Assessment Operations
|
|
|
|
Status: draft
|
|
Created: 2026-05-15
|
|
|
|
## Purpose
|
|
|
|
This document is the generic operator path for running guide-board assessments.
|
|
It covers the local CLI and the dependency-light local service API. Extension
|
|
repositories can provide domain-specific profiles and runtime setup, but the
|
|
run, status, and result contracts are owned here.
|
|
|
|
Guide-board prepares structured evidence and assessment packages. It does not
|
|
issue certifications, provide audit assurance, or replace an accredited
|
|
assessment process.
|
|
|
|
## Inputs
|
|
|
|
Every run needs:
|
|
|
|
- a target profile,
|
|
- an assessment profile,
|
|
- any extension repository roots required by the assessment profile,
|
|
- optional credentials or mounted harness assets referenced by the target or
|
|
assessment profile,
|
|
- an output directory for run artifacts.
|
|
|
|
The target profile describes the candidate system or artifact being assessed.
|
|
The assessment profile selects frameworks, extensions, check groups, runtime
|
|
policy, expectations, waivers, challenges, authority exclusions, and output
|
|
policy.
|
|
|
|
## CLI Flow
|
|
|
|
Run from the guide-board repository:
|
|
|
|
```sh
|
|
cd /home/worsch/guide-board
|
|
PYTHONPATH=src python3 -m guide_board extensions validate
|
|
PYTHONPATH=src python3 -m guide_board profile validate-target profiles/targets/sample-repository.json
|
|
PYTHONPATH=src python3 -m guide_board profile validate-assessment profiles/assessments/sample-noop.json
|
|
PYTHONPATH=src python3 -m guide_board plan \
|
|
--target profiles/targets/sample-repository.json \
|
|
--assessment profiles/assessments/sample-noop.json
|
|
PYTHONPATH=src python3 -m guide_board run \
|
|
--target profiles/targets/sample-repository.json \
|
|
--assessment profiles/assessments/sample-noop.json \
|
|
--output-dir runs/sample-noop
|
|
```
|
|
|
|
For an external extension repository, pass `--extension-dir` before the
|
|
subcommand:
|
|
|
|
```sh
|
|
cd /home/worsch/guide-board
|
|
PYTHONPATH=src python3 -m guide_board \
|
|
--extension-dir ../open-cmis-tck \
|
|
run \
|
|
--target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
|
|
--assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
|
|
--output-dir runs/open-cmis-tck-baseline
|
|
```
|
|
|
|
The same extension roots can be provided through `GUIDE_BOARD_EXTENSION_PATHS`
|
|
when a wrapper script or container entrypoint should keep commands shorter.
|
|
For the repeatable external extension acceptance path, including validation,
|
|
planning, live execution, and retained result review, see
|
|
`docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
|
|
For extension-author contracts such as profile schema descriptors and
|
|
normalizer plug-ins, see `docs/EXTENSION-SDK.md`.
|
|
|
|
## CLI Results
|
|
|
|
A completed CLI command prints a JSON result with:
|
|
|
|
- `status`: assessment outcome,
|
|
- `run_id`: generated run identifier,
|
|
- `run_dir`: output directory,
|
|
- `assessment_package`: JSON assessment package path,
|
|
- `report`: Markdown report path,
|
|
- `submission_package`: portable submission package manifest path,
|
|
- `export_manifest`: portable generic JSON export manifest path,
|
|
- `retention_summary`: compact durable summary path.
|
|
|
|
The output directory uses this contract:
|
|
|
|
```text
|
|
run.json
|
|
plan.json
|
|
sources.lock.json
|
|
target-profile.snapshot.json
|
|
assessment-profile.snapshot.json
|
|
retention-summary.json
|
|
normalized/evidence.json
|
|
normalized/findings.json
|
|
normalized/mappings.json
|
|
reports/assessment-package.json
|
|
reports/report.md
|
|
reports/fragments.json
|
|
reports/submission-package.json
|
|
exports/export-manifest.json
|
|
artifacts/
|
|
```
|
|
|
|
`sources.lock.json` records the framework refs, extension versions, mapping
|
|
sets, profile snapshots, policy and review refs, authority refs, and extension
|
|
metadata hooks used for the run. `reports/submission-package.json` points at
|
|
the reviewable package files, includes checksums where files exist, carries the
|
|
raw artifact manifest, and repeats the certification boundary. It is a portable
|
|
handoff manifest for preparation evidence, not an authority-specific final
|
|
submission.
|
|
|
|
Extension report fragments are recorded in `reports/fragments.json`, embedded in
|
|
`reports/assessment-package.json`, and rendered into the Markdown report.
|
|
`exports/export-manifest.json` is the first generic portable export surface. It
|
|
is derived from the assessment package and carries summary, policy, mapping,
|
|
fragment, count, source-lock, and boundary references.
|
|
|
|
Use the retained run helpers for history:
|
|
|
|
```sh
|
|
PYTHONPATH=src python3 -m guide_board runs list --runs-dir runs
|
|
PYTHONPATH=src python3 -m guide_board runs latest --runs-dir runs \
|
|
--target sample-repository \
|
|
--assessment sample-noop-assessment
|
|
PYTHONPATH=src python3 -m guide_board runs report --runs-dir runs \
|
|
--target sample-repository \
|
|
--assessment sample-noop-assessment
|
|
PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs
|
|
PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs
|
|
```
|
|
|
|
Trend summaries include status changes, unexpected finding deltas, unresolved
|
|
review deltas, mapping target deltas, evidence result deltas, and a compact
|
|
human-readable summary string for each target/assessment pair.
|
|
|
|
## Local Service Flow
|
|
|
|
Start the service from the guide-board repository:
|
|
|
|
```sh
|
|
cd /home/worsch/guide-board
|
|
PYTHONPATH=src python3 -m guide_board \
|
|
--extension-dir ../open-cmis-tck \
|
|
serve --host 127.0.0.1 --port 8080
|
|
```
|
|
|
|
Check health:
|
|
|
|
```sh
|
|
curl -sf http://127.0.0.1:8080/health | python3 -m json.tool
|
|
```
|
|
|
|
Start a run:
|
|
|
|
```sh
|
|
curl -sf -X POST http://127.0.0.1:8080/runs \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"target": "../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json",
|
|
"assessment": "../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json",
|
|
"output_dir": "runs/open-cmis-tck-baseline"
|
|
}' | python3 -m json.tool
|
|
```
|
|
|
|
The response includes `job_id`.
|
|
|
|
Inspect job status:
|
|
|
|
```sh
|
|
curl -sf http://127.0.0.1:8080/runs/JOB_ID | python3 -m json.tool
|
|
```
|
|
|
|
Fetch reports after the job status is `succeeded`:
|
|
|
|
```sh
|
|
curl -sf http://127.0.0.1:8080/runs/JOB_ID/reports | python3 -m json.tool
|
|
```
|
|
|
|
Inspect retained run history, including runs produced before the current
|
|
service process started:
|
|
|
|
```sh
|
|
curl -sf "http://127.0.0.1:8080/retained-runs?runs_dir=runs" | python3 -m json.tool
|
|
curl -sf "http://127.0.0.1:8080/retained-runs/latest?runs_dir=runs" | python3 -m json.tool
|
|
curl -sf "http://127.0.0.1:8080/retained-runs/RUN_ID/artifact-manifest?runs_dir=runs" | python3 -m json.tool
|
|
```
|
|
|
|
Service job state is currently in memory for the running service process. Run
|
|
artifacts are durable in the output directory and can still be inspected after a
|
|
service restart. See `docs/SERVICE-JOB-DURABILITY.md` for the restart and
|
|
recovery contract.
|
|
|
|
## Status Vocabulary
|
|
|
|
Service jobs use transport status:
|
|
|
|
- `queued`
|
|
- `running`
|
|
- `succeeded`
|
|
- `failed`
|
|
|
|
Assessment runs use evidence status:
|
|
|
|
- `completed`
|
|
- `failed`
|
|
- `blocked`
|
|
- `infrastructure_error`
|
|
|
|
A service job can be `succeeded` while the assessment status is `blocked` or
|
|
`infrastructure_error`. That means guide-board completed the run machinery and
|
|
produced evidence explaining why the assessment could not proceed cleanly.
|
|
|
|
Individual evidence items use:
|
|
|
|
- `pass`
|
|
- `warning`
|
|
- `fail`
|
|
- `manual`
|
|
- `skipped`
|
|
- `blocked`
|
|
- `not_applicable`
|
|
- `expected_gap`
|
|
- `infrastructure_error`
|
|
|
|
## Review State
|
|
|
|
Assessment profiles may reference:
|
|
|
|
- `expectations_ref`: known target posture, optional scope, or accepted gaps,
|
|
- `waivers_ref`: approved, time-bounded exceptions,
|
|
- `challenges_ref`: review claims that a finding, check, mapping, or native
|
|
result should be challenged,
|
|
- `exclusions_ref`: authority or program exclusions that apply to selected
|
|
findings.
|
|
|
|
Challenges and exclusions annotate findings and evidence. They do not silently
|
|
turn failures into passing evidence and they do not reduce the
|
|
`unexpected_findings` count used by default gates. Retained summaries expose
|
|
separate counts for expected findings, waived findings, challenged findings,
|
|
authority exclusions, unresolved defects, and unresolved review items.
|
|
|
|
## Candidate System Checklist
|
|
|
|
Before starting a run against candidate software, confirm:
|
|
|
|
- the candidate service, repository, or artifact is in the state expected by
|
|
the target profile,
|
|
- endpoint URLs in the target profile are reachable from the guide-board
|
|
process or container,
|
|
- credentials are referenced through environment variables or mounted files,
|
|
not committed into profiles,
|
|
- the assessment profile selects only check groups that are safe for the
|
|
candidate environment,
|
|
- the output directory is outside source-controlled paths or is ignored.
|
|
|
|
Candidate repositories should provide their own short handoff document with the
|
|
startup command, profile path, required credentials, and expected endpoint URL.
|
|
See `docs/CANDIDATE-HANDOFF.md` and
|
|
`docs/templates/CANDIDATE-ASSESSMENT-HANDOFF.md` for the reusable contract.
|