Close bootstrap and start assessment operations

2026-05-15 12:33:21 +02:00
parent 4357452af1
commit 2a7da8c04b
4 changed files with 360 additions and 2 deletions
--- a/docs/ASSESSMENT-OPERATIONS.md
+++ b/docs/ASSESSMENT-OPERATIONS.md
@@ -0,0 +1,193 @@
+# Assessment Operations
+
+Status: draft
+Created: 2026-05-15
+
+## Purpose
+
+This document is the generic operator path for running guide-board assessments.
+It covers the local CLI and the dependency-light local service API. Extension
+repositories can provide domain-specific profiles and runtime setup, but the
+run, status, and result contracts are owned here.
+
+Guide-board prepares structured evidence and assessment packages. It does not
+issue certifications, provide audit assurance, or replace an accredited
+assessment process.
+
+## Inputs
+
+Every run needs:
+
+- a target profile,
+- an assessment profile,
+- any extension repository roots required by the assessment profile,
+- optional credentials or mounted harness assets referenced by the target or
+  assessment profile,
+- an output directory for run artifacts.
+
+The target profile describes the candidate system or artifact being assessed.
+The assessment profile selects frameworks, extensions, check groups, runtime
+policy, waivers, expectations, and output policy.
+
+## CLI Flow
+
+Run from the guide-board repository:
+
+```sh
+cd /home/worsch/guide-board
+PYTHONPATH=src python3 -m guide_board extensions validate
+PYTHONPATH=src python3 -m guide_board profile validate-target profiles/targets/sample-repository.json
+PYTHONPATH=src python3 -m guide_board profile validate-assessment profiles/assessments/sample-noop.json
+PYTHONPATH=src python3 -m guide_board plan \
+  --target profiles/targets/sample-repository.json \
+  --assessment profiles/assessments/sample-noop.json
+PYTHONPATH=src python3 -m guide_board run \
+  --target profiles/targets/sample-repository.json \
+  --assessment profiles/assessments/sample-noop.json \
+  --output-dir runs/sample-noop
+```
+
+For an external extension repository, pass `--extension-dir` before the
+subcommand:
+
+```sh
+cd /home/worsch/guide-board
+PYTHONPATH=src python3 -m guide_board \
+  --extension-dir ../open-cmis-tck \
+  run \
+  --target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
+  --assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
+  --output-dir runs/open-cmis-tck-baseline
+```
+
+The same extension roots can be provided through `GUIDE_BOARD_EXTENSION_PATHS`
+when a wrapper script or container entrypoint should keep commands shorter.
+
+## CLI Results
+
+A completed CLI command prints a JSON result with:
+
+- `status`: assessment outcome,
+- `run_id`: generated run identifier,
+- `run_dir`: output directory,
+- `assessment_package`: JSON assessment package path,
+- `report`: Markdown report path,
+- `retention_summary`: compact durable summary path.
+
+The output directory uses this contract:
+
+```text
+run.json
+plan.json
+retention-summary.json
+normalized/evidence.json
+normalized/findings.json
+normalized/mappings.json
+reports/assessment-package.json
+reports/report.md
+artifacts/
+```
+
+Use the retained run helpers for history:
+
+```sh
+PYTHONPATH=src python3 -m guide_board runs list --runs-dir runs
+PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs
+PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs
+```
+
+## Local Service Flow
+
+Start the service from the guide-board repository:
+
+```sh
+cd /home/worsch/guide-board
+PYTHONPATH=src python3 -m guide_board \
+  --extension-dir ../open-cmis-tck \
+  serve --host 127.0.0.1 --port 8080
+```
+
+Check health:
+
+```sh
+curl -sf http://127.0.0.1:8080/health | python3 -m json.tool
+```
+
+Start a run:
+
+```sh
+curl -sf -X POST http://127.0.0.1:8080/runs \
+  -H "Content-Type: application/json" \
+  -d '{
+    "target": "../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json",
+    "assessment": "../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json",
+    "output_dir": "runs/open-cmis-tck-baseline"
+  }' | python3 -m json.tool
+```
+
+The response includes `job_id`.
+
+Inspect job status:
+
+```sh
+curl -sf http://127.0.0.1:8080/runs/JOB_ID | python3 -m json.tool
+```
+
+Fetch reports after the job status is `succeeded`:
+
+```sh
+curl -sf http://127.0.0.1:8080/runs/JOB_ID/reports | python3 -m json.tool
+```
+
+Service job state is currently in memory for the running service process. Run
+artifacts are durable in the output directory and can still be inspected after a
+service restart.
+
+## Status Vocabulary
+
+Service jobs use transport status:
+
+- `queued`
+- `running`
+- `succeeded`
+- `failed`
+
+Assessment runs use evidence status:
+
+- `completed`
+- `failed`
+- `blocked`
+- `infrastructure_error`
+
+A service job can be `succeeded` while the assessment status is `blocked` or
+`infrastructure_error`. That means guide-board completed the run machinery and
+produced evidence explaining why the assessment could not proceed cleanly.
+
+Individual evidence items use:
+
+- `pass`
+- `warning`
+- `fail`
+- `manual`
+- `skipped`
+- `blocked`
+- `not_applicable`
+- `expected_gap`
+- `infrastructure_error`
+
+## Candidate System Checklist
+
+Before starting a run against candidate software, confirm:
+
+- the candidate service, repository, or artifact is in the state expected by
+  the target profile,
+- endpoint URLs in the target profile are reachable from the guide-board
+  process or container,
+- credentials are referenced through environment variables or mounted files,
+  not committed into profiles,
+- the assessment profile selects only check groups that are safe for the
+  candidate environment,
+- the output directory is outside source-controlled paths or is ignored.
+
+Candidate repositories should provide their own short handoff document with the
+startup command, profile path, required credentials, and expected endpoint URL.