generated from coulomb/repo-seed
208 lines
6.1 KiB
Markdown
208 lines
6.1 KiB
Markdown
# Assessment Operations
|
|
|
|
Status: draft
|
|
Created: 2026-05-15
|
|
|
|
## Purpose
|
|
|
|
This document is the generic operator path for running guide-board assessments.
|
|
It covers the local CLI and the dependency-light local service API. Extension
|
|
repositories can provide domain-specific profiles and runtime setup, but the
|
|
run, status, and result contracts are owned here.
|
|
|
|
Guide-board prepares structured evidence and assessment packages. It does not
|
|
issue certifications, provide audit assurance, or replace an accredited
|
|
assessment process.
|
|
|
|
## Inputs
|
|
|
|
Every run needs:
|
|
|
|
- a target profile,
|
|
- an assessment profile,
|
|
- any extension repository roots required by the assessment profile,
|
|
- optional credentials or mounted harness assets referenced by the target or
|
|
assessment profile,
|
|
- an output directory for run artifacts.
|
|
|
|
The target profile describes the candidate system or artifact being assessed.
|
|
The assessment profile selects frameworks, extensions, check groups, runtime
|
|
policy, waivers, expectations, and output policy.
|
|
|
|
## CLI Flow
|
|
|
|
Run from the guide-board repository:
|
|
|
|
```sh
|
|
cd /home/worsch/guide-board
|
|
PYTHONPATH=src python3 -m guide_board extensions validate
|
|
PYTHONPATH=src python3 -m guide_board profile validate-target profiles/targets/sample-repository.json
|
|
PYTHONPATH=src python3 -m guide_board profile validate-assessment profiles/assessments/sample-noop.json
|
|
PYTHONPATH=src python3 -m guide_board plan \
|
|
--target profiles/targets/sample-repository.json \
|
|
--assessment profiles/assessments/sample-noop.json
|
|
PYTHONPATH=src python3 -m guide_board run \
|
|
--target profiles/targets/sample-repository.json \
|
|
--assessment profiles/assessments/sample-noop.json \
|
|
--output-dir runs/sample-noop
|
|
```
|
|
|
|
For an external extension repository, pass `--extension-dir` before the
|
|
subcommand:
|
|
|
|
```sh
|
|
cd /home/worsch/guide-board
|
|
PYTHONPATH=src python3 -m guide_board \
|
|
--extension-dir ../open-cmis-tck \
|
|
run \
|
|
--target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
|
|
--assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
|
|
--output-dir runs/open-cmis-tck-baseline
|
|
```
|
|
|
|
The same extension roots can be provided through `GUIDE_BOARD_EXTENSION_PATHS`
|
|
when a wrapper script or container entrypoint should keep commands shorter.
|
|
For the repeatable external extension acceptance path, including validation,
|
|
planning, live execution, and retained result review, see
|
|
`docs/EXTERNAL-EXTENSION-ACCEPTANCE.md`.
|
|
For extension-author contracts such as profile schema descriptors and
|
|
normalizer plug-ins, see `docs/EXTENSION-SDK.md`.
|
|
|
|
## CLI Results
|
|
|
|
A completed CLI command prints a JSON result with:
|
|
|
|
- `status`: assessment outcome,
|
|
- `run_id`: generated run identifier,
|
|
- `run_dir`: output directory,
|
|
- `assessment_package`: JSON assessment package path,
|
|
- `report`: Markdown report path,
|
|
- `retention_summary`: compact durable summary path.
|
|
|
|
The output directory uses this contract:
|
|
|
|
```text
|
|
run.json
|
|
plan.json
|
|
retention-summary.json
|
|
normalized/evidence.json
|
|
normalized/findings.json
|
|
normalized/mappings.json
|
|
reports/assessment-package.json
|
|
reports/report.md
|
|
artifacts/
|
|
```
|
|
|
|
Use the retained run helpers for history:
|
|
|
|
```sh
|
|
PYTHONPATH=src python3 -m guide_board runs list --runs-dir runs
|
|
PYTHONPATH=src python3 -m guide_board runs latest --runs-dir runs \
|
|
--target sample-repository \
|
|
--assessment sample-noop-assessment
|
|
PYTHONPATH=src python3 -m guide_board runs report --runs-dir runs \
|
|
--target sample-repository \
|
|
--assessment sample-noop-assessment
|
|
PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs
|
|
PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs
|
|
```
|
|
|
|
## Local Service Flow
|
|
|
|
Start the service from the guide-board repository:
|
|
|
|
```sh
|
|
cd /home/worsch/guide-board
|
|
PYTHONPATH=src python3 -m guide_board \
|
|
--extension-dir ../open-cmis-tck \
|
|
serve --host 127.0.0.1 --port 8080
|
|
```
|
|
|
|
Check health:
|
|
|
|
```sh
|
|
curl -sf http://127.0.0.1:8080/health | python3 -m json.tool
|
|
```
|
|
|
|
Start a run:
|
|
|
|
```sh
|
|
curl -sf -X POST http://127.0.0.1:8080/runs \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"target": "../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json",
|
|
"assessment": "../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json",
|
|
"output_dir": "runs/open-cmis-tck-baseline"
|
|
}' | python3 -m json.tool
|
|
```
|
|
|
|
The response includes `job_id`.
|
|
|
|
Inspect job status:
|
|
|
|
```sh
|
|
curl -sf http://127.0.0.1:8080/runs/JOB_ID | python3 -m json.tool
|
|
```
|
|
|
|
Fetch reports after the job status is `succeeded`:
|
|
|
|
```sh
|
|
curl -sf http://127.0.0.1:8080/runs/JOB_ID/reports | python3 -m json.tool
|
|
```
|
|
|
|
Service job state is currently in memory for the running service process. Run
|
|
artifacts are durable in the output directory and can still be inspected after a
|
|
service restart. See `docs/SERVICE-JOB-DURABILITY.md` for the restart and
|
|
recovery contract.
|
|
|
|
## Status Vocabulary
|
|
|
|
Service jobs use transport status:
|
|
|
|
- `queued`
|
|
- `running`
|
|
- `succeeded`
|
|
- `failed`
|
|
|
|
Assessment runs use evidence status:
|
|
|
|
- `completed`
|
|
- `failed`
|
|
- `blocked`
|
|
- `infrastructure_error`
|
|
|
|
A service job can be `succeeded` while the assessment status is `blocked` or
|
|
`infrastructure_error`. That means guide-board completed the run machinery and
|
|
produced evidence explaining why the assessment could not proceed cleanly.
|
|
|
|
Individual evidence items use:
|
|
|
|
- `pass`
|
|
- `warning`
|
|
- `fail`
|
|
- `manual`
|
|
- `skipped`
|
|
- `blocked`
|
|
- `not_applicable`
|
|
- `expected_gap`
|
|
- `infrastructure_error`
|
|
|
|
## Candidate System Checklist
|
|
|
|
Before starting a run against candidate software, confirm:
|
|
|
|
- the candidate service, repository, or artifact is in the state expected by
|
|
the target profile,
|
|
- endpoint URLs in the target profile are reachable from the guide-board
|
|
process or container,
|
|
- credentials are referenced through environment variables or mounted files,
|
|
not committed into profiles,
|
|
- the assessment profile selects only check groups that are safe for the
|
|
candidate environment,
|
|
- the output directory is outside source-controlled paths or is ignored.
|
|
|
|
Candidate repositories should provide their own short handoff document with the
|
|
startup command, profile path, required credentials, and expected endpoint URL.
|
|
See `docs/CANDIDATE-HANDOFF.md` and
|
|
`docs/templates/CANDIDATE-ASSESSMENT-HANDOFF.md` for the reusable contract.
|