guide-board/docs/ASSESSMENT-OPERATIONS.md

Assessment Operations

Status: draft Created: 2026-05-15

Purpose

This document is the generic operator path for running guide-board assessments. It covers the local CLI and the dependency-light local service API. Extension repositories can provide domain-specific profiles and runtime setup, but the run, status, and result contracts are owned here.

Guide-board prepares structured evidence and assessment packages. It does not issue certifications, provide audit assurance, or replace an accredited assessment process.

Inputs

Every run needs:

• a target profile,
• an assessment profile,
• any extension repository roots required by the assessment profile,
• optional credentials or mounted harness assets referenced by the target or assessment profile,
• an output directory for run artifacts.

The target profile describes the candidate system or artifact being assessed. The assessment profile selects frameworks, extensions, check groups, runtime policy, waivers, expectations, and output policy.

CLI Flow

Run from the guide-board repository:

cd /home/worsch/guide-board
PYTHONPATH=src python3 -m guide_board extensions validate
PYTHONPATH=src python3 -m guide_board profile validate-target profiles/targets/sample-repository.json
PYTHONPATH=src python3 -m guide_board profile validate-assessment profiles/assessments/sample-noop.json
PYTHONPATH=src python3 -m guide_board plan \
  --target profiles/targets/sample-repository.json \
  --assessment profiles/assessments/sample-noop.json
PYTHONPATH=src python3 -m guide_board run \
  --target profiles/targets/sample-repository.json \
  --assessment profiles/assessments/sample-noop.json \
  --output-dir runs/sample-noop

For an external extension repository, pass --extension-dir before the subcommand:

cd /home/worsch/guide-board
PYTHONPATH=src python3 -m guide_board \
  --extension-dir ../open-cmis-tck \
  run \
  --target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
  --assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
  --output-dir runs/open-cmis-tck-baseline

The same extension roots can be provided through GUIDE_BOARD_EXTENSION_PATHS when a wrapper script or container entrypoint should keep commands shorter.

CLI Results

A completed CLI command prints a JSON result with:

• status: assessment outcome,
• run_id: generated run identifier,
• run_dir: output directory,
• assessment_package: JSON assessment package path,
• report: Markdown report path,
• retention_summary: compact durable summary path.

The output directory uses this contract:

run.json
plan.json
retention-summary.json
normalized/evidence.json
normalized/findings.json
normalized/mappings.json
reports/assessment-package.json
reports/report.md
artifacts/

Use the retained run helpers for history:

PYTHONPATH=src python3 -m guide_board runs list --runs-dir runs
PYTHONPATH=src python3 -m guide_board runs latest --runs-dir runs \
  --target sample-repository \
  --assessment sample-noop-assessment
PYTHONPATH=src python3 -m guide_board runs report --runs-dir runs \
  --target sample-repository \
  --assessment sample-noop-assessment
PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs
PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs

Local Service Flow

Start the service from the guide-board repository:

cd /home/worsch/guide-board
PYTHONPATH=src python3 -m guide_board \
  --extension-dir ../open-cmis-tck \
  serve --host 127.0.0.1 --port 8080

Check health:

curl -sf http://127.0.0.1:8080/health | python3 -m json.tool

Start a run:

curl -sf -X POST http://127.0.0.1:8080/runs \
  -H "Content-Type: application/json" \
  -d '{
    "target": "../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json",
    "assessment": "../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json",
    "output_dir": "runs/open-cmis-tck-baseline"
  }' | python3 -m json.tool

The response includes job_id.

Inspect job status:

curl -sf http://127.0.0.1:8080/runs/JOB_ID | python3 -m json.tool

Fetch reports after the job status is succeeded:

curl -sf http://127.0.0.1:8080/runs/JOB_ID/reports | python3 -m json.tool

Service job state is currently in memory for the running service process. Run artifacts are durable in the output directory and can still be inspected after a service restart.

Status Vocabulary

Service jobs use transport status:

• queued
• running
• succeeded
• failed

Assessment runs use evidence status:

• completed
• failed
• blocked
• infrastructure_error

A service job can be succeeded while the assessment status is blocked or infrastructure_error. That means guide-board completed the run machinery and produced evidence explaining why the assessment could not proceed cleanly.

Individual evidence items use:

• pass
• warning
• fail
• manual
• skipped
• blocked
• not_applicable
• expected_gap
• infrastructure_error

Candidate System Checklist

Before starting a run against candidate software, confirm:

• the candidate service, repository, or artifact is in the state expected by the target profile,
• endpoint URLs in the target profile are reachable from the guide-board process or container,
• credentials are referenced through environment variables or mounted files, not committed into profiles,
• the assessment profile selects only check groups that are safe for the candidate environment,
• the output directory is outside source-controlled paths or is ignored.

Candidate repositories should provide their own short handoff document with the startup command, profile path, required credentials, and expected endpoint URL. See docs/CANDIDATE-HANDOFF.md and docs/templates/CANDIDATE-ASSESSMENT-HANDOFF.md for the reusable contract.