guide-board/docs/ASSESSMENT-OPERATIONS.md

Assessment Operations

Status: draft Created: 2026-05-15

Purpose

This document is the generic operator path for running guide-board assessments. It covers the local CLI and the dependency-light local service API. Extension repositories can provide domain-specific profiles and runtime setup, but the run, status, and result contracts are owned here.

Guide-board prepares structured evidence and assessment packages. It does not issue certifications, provide audit assurance, or replace an accredited assessment process.

Inputs

Every run needs:

• a target profile,
• an assessment profile,
• any extension repository roots required by the assessment profile,
• optional credentials or mounted harness assets referenced by the target or assessment profile,
• an output directory for run artifacts.

The target profile describes the candidate system or artifact being assessed. The assessment profile selects frameworks, extensions, check groups, runtime policy, expectations, waivers, challenges, authority exclusions, and output policy.

CLI Flow

Run from the guide-board repository:

cd /home/worsch/guide-board
PYTHONPATH=src python3 -m guide_board extensions validate
PYTHONPATH=src python3 -m guide_board profile validate-target profiles/targets/sample-repository.json
PYTHONPATH=src python3 -m guide_board profile validate-assessment profiles/assessments/sample-noop.json
PYTHONPATH=src python3 -m guide_board plan \
  --target profiles/targets/sample-repository.json \
  --assessment profiles/assessments/sample-noop.json
PYTHONPATH=src python3 -m guide_board run \
  --target profiles/targets/sample-repository.json \
  --assessment profiles/assessments/sample-noop.json \
  --output-dir runs/sample-noop

For an external extension repository, pass --extension-dir before the subcommand:

cd /home/worsch/guide-board
PYTHONPATH=src python3 -m guide_board \
  --extension-dir ../open-cmis-tck \
  run \
  --target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
  --assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
  --output-dir runs/open-cmis-tck-baseline

The same extension roots can be provided through GUIDE_BOARD_EXTENSION_PATHS when a wrapper script or container entrypoint should keep commands shorter. For the repeatable external extension acceptance path, including validation, planning, live execution, and retained result review, see docs/EXTERNAL-EXTENSION-ACCEPTANCE.md. For extension-author contracts such as profile schema descriptors and normalizer plug-ins, see docs/EXTENSION-SDK.md.

CLI Results

A completed CLI command prints a JSON result with:

• status: assessment outcome,
• run_id: generated run identifier,
• run_dir: output directory,
• assessment_package: JSON assessment package path,
• report: Markdown report path,
• submission_package: portable submission package manifest path,
• export_manifest: portable generic JSON export manifest path,
• retention_summary: compact durable summary path.

The output directory uses this contract:

run.json
plan.json
sources.lock.json
target-profile.snapshot.json
assessment-profile.snapshot.json
retention-summary.json
normalized/evidence.json
normalized/findings.json
normalized/mappings.json
reports/assessment-package.json
reports/report.md
reports/fragments.json
reports/submission-package.json
exports/export-manifest.json
artifacts/

sources.lock.json records the framework refs, extension versions, mapping sets, profile snapshots, policy and review refs, authority refs, and extension metadata hooks used for the run. reports/submission-package.json points at the reviewable package files, includes checksums where files exist, carries the raw artifact manifest, and repeats the certification boundary. It is a portable handoff manifest for preparation evidence, not an authority-specific final submission.

Extension report fragments are recorded in reports/fragments.json, embedded in reports/assessment-package.json, and rendered into the Markdown report. exports/export-manifest.json is the first generic portable export surface. It is derived from the assessment package and carries summary, policy, mapping, fragment, count, source-lock, and boundary references.

Use the retained run helpers for history:

PYTHONPATH=src python3 -m guide_board runs list --runs-dir runs
PYTHONPATH=src python3 -m guide_board runs latest --runs-dir runs \
  --target sample-repository \
  --assessment sample-noop-assessment
PYTHONPATH=src python3 -m guide_board runs report --runs-dir runs \
  --target sample-repository \
  --assessment sample-noop-assessment
PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs
PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs

Trend summaries include status changes, unexpected finding deltas, unresolved review deltas, mapping target deltas, evidence result deltas, and a compact human-readable summary string for each target/assessment pair.

Local Service Flow

Start the service from the guide-board repository:

cd /home/worsch/guide-board
PYTHONPATH=src python3 -m guide_board \
  --extension-dir ../open-cmis-tck \
  serve --host 127.0.0.1 --port 8080

Check health:

curl -sf http://127.0.0.1:8080/health | python3 -m json.tool

Start a run:

curl -sf -X POST http://127.0.0.1:8080/runs \
  -H "Content-Type: application/json" \
  -d '{
    "target": "../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json",
    "assessment": "../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json",
    "output_dir": "runs/open-cmis-tck-baseline"
  }' | python3 -m json.tool

The response includes job_id.

Inspect job status:

curl -sf http://127.0.0.1:8080/runs/JOB_ID | python3 -m json.tool

Fetch reports after the job status is succeeded:

curl -sf http://127.0.0.1:8080/runs/JOB_ID/reports | python3 -m json.tool

Inspect retained run history, including runs produced before the current service process started:

curl -sf "http://127.0.0.1:8080/retained-runs?runs_dir=runs" | python3 -m json.tool
curl -sf "http://127.0.0.1:8080/retained-runs/latest?runs_dir=runs" | python3 -m json.tool
curl -sf "http://127.0.0.1:8080/retained-runs/RUN_ID/artifact-manifest?runs_dir=runs" | python3 -m json.tool

Service job state is currently in memory for the running service process. Run artifacts are durable in the output directory and can still be inspected after a service restart. See docs/SERVICE-JOB-DURABILITY.md for the restart and recovery contract.

Status Vocabulary

Service jobs use transport status:

• queued
• running
• succeeded
• failed

Assessment runs use evidence status:

• completed
• failed
• blocked
• infrastructure_error

A service job can be succeeded while the assessment status is blocked or infrastructure_error. That means guide-board completed the run machinery and produced evidence explaining why the assessment could not proceed cleanly.

Individual evidence items use:

• pass
• warning
• fail
• manual
• skipped
• blocked
• not_applicable
• expected_gap
• infrastructure_error

Review State

Assessment profiles may reference:

• expectations_ref: known target posture, optional scope, or accepted gaps,
• waivers_ref: approved, time-bounded exceptions,
• challenges_ref: review claims that a finding, check, mapping, or native result should be challenged,
• exclusions_ref: authority or program exclusions that apply to selected findings.

Challenges and exclusions annotate findings and evidence. They do not silently turn failures into passing evidence and they do not reduce the unexpected_findings count used by default gates. Retained summaries expose separate counts for expected findings, waived findings, challenged findings, authority exclusions, unresolved defects, and unresolved review items.

Candidate System Checklist

Before starting a run against candidate software, confirm:

• the candidate service, repository, or artifact is in the state expected by the target profile,
• endpoint URLs in the target profile are reachable from the guide-board process or container,
• credentials are referenced through environment variables or mounted files, not committed into profiles,
• the assessment profile selects only check groups that are safe for the candidate environment,
• the output directory is outside source-controlled paths or is ignored.

Candidate repositories should provide their own short handoff document with the startup command, profile path, required credentials, and expected endpoint URL. See docs/CANDIDATE-HANDOFF.md and docs/templates/CANDIDATE-ASSESSMENT-HANDOFF.md for the reusable contract.