From 2a7da8c04bc7b6e3aa3eddc7383cc1f3fd974d16 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 15 May 2026 12:33:21 +0200 Subject: [PATCH] Close bootstrap and start assessment operations --- README.md | 1 + docs/ASSESSMENT-OPERATIONS.md | 193 ++++++++++++++++++ .../GUIDE-BOARD-WP-0001-bootstrapping.md | 4 +- ...-WP-0002-assessment-operations-baseline.md | 164 +++++++++++++++ 4 files changed, 360 insertions(+), 2 deletions(-) create mode 100644 docs/ASSESSMENT-OPERATIONS.md create mode 100644 workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md diff --git a/README.md b/README.md index 62a5d7b..805838f 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,7 @@ See: - [INTENT.md](INTENT.md) - [docs/ARCHITECTURE-BLUEPRINT.md](docs/ARCHITECTURE-BLUEPRINT.md) +- [docs/ASSESSMENT-OPERATIONS.md](docs/ASSESSMENT-OPERATIONS.md) - [docs/COMPLIANCE-EVIDENCE-PACKS.md](docs/COMPLIANCE-EVIDENCE-PACKS.md) - [docs/CONTAINER.md](docs/CONTAINER.md) - [docs/EXTENSION-SDK.md](docs/EXTENSION-SDK.md) diff --git a/docs/ASSESSMENT-OPERATIONS.md b/docs/ASSESSMENT-OPERATIONS.md new file mode 100644 index 0000000..a1454f7 --- /dev/null +++ b/docs/ASSESSMENT-OPERATIONS.md @@ -0,0 +1,193 @@ +# Assessment Operations + +Status: draft +Created: 2026-05-15 + +## Purpose + +This document is the generic operator path for running guide-board assessments. +It covers the local CLI and the dependency-light local service API. Extension +repositories can provide domain-specific profiles and runtime setup, but the +run, status, and result contracts are owned here. + +Guide-board prepares structured evidence and assessment packages. It does not +issue certifications, provide audit assurance, or replace an accredited +assessment process. + +## Inputs + +Every run needs: + +- a target profile, +- an assessment profile, +- any extension repository roots required by the assessment profile, +- optional credentials or mounted harness assets referenced by the target or + assessment profile, +- an output directory for run artifacts. + +The target profile describes the candidate system or artifact being assessed. +The assessment profile selects frameworks, extensions, check groups, runtime +policy, waivers, expectations, and output policy. + +## CLI Flow + +Run from the guide-board repository: + +```sh +cd /home/worsch/guide-board +PYTHONPATH=src python3 -m guide_board extensions validate +PYTHONPATH=src python3 -m guide_board profile validate-target profiles/targets/sample-repository.json +PYTHONPATH=src python3 -m guide_board profile validate-assessment profiles/assessments/sample-noop.json +PYTHONPATH=src python3 -m guide_board plan \ + --target profiles/targets/sample-repository.json \ + --assessment profiles/assessments/sample-noop.json +PYTHONPATH=src python3 -m guide_board run \ + --target profiles/targets/sample-repository.json \ + --assessment profiles/assessments/sample-noop.json \ + --output-dir runs/sample-noop +``` + +For an external extension repository, pass `--extension-dir` before the +subcommand: + +```sh +cd /home/worsch/guide-board +PYTHONPATH=src python3 -m guide_board \ + --extension-dir ../open-cmis-tck \ + run \ + --target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \ + --assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \ + --output-dir runs/open-cmis-tck-baseline +``` + +The same extension roots can be provided through `GUIDE_BOARD_EXTENSION_PATHS` +when a wrapper script or container entrypoint should keep commands shorter. + +## CLI Results + +A completed CLI command prints a JSON result with: + +- `status`: assessment outcome, +- `run_id`: generated run identifier, +- `run_dir`: output directory, +- `assessment_package`: JSON assessment package path, +- `report`: Markdown report path, +- `retention_summary`: compact durable summary path. + +The output directory uses this contract: + +```text +run.json +plan.json +retention-summary.json +normalized/evidence.json +normalized/findings.json +normalized/mappings.json +reports/assessment-package.json +reports/report.md +artifacts/ +``` + +Use the retained run helpers for history: + +```sh +PYTHONPATH=src python3 -m guide_board runs list --runs-dir runs +PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs +PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs +``` + +## Local Service Flow + +Start the service from the guide-board repository: + +```sh +cd /home/worsch/guide-board +PYTHONPATH=src python3 -m guide_board \ + --extension-dir ../open-cmis-tck \ + serve --host 127.0.0.1 --port 8080 +``` + +Check health: + +```sh +curl -sf http://127.0.0.1:8080/health | python3 -m json.tool +``` + +Start a run: + +```sh +curl -sf -X POST http://127.0.0.1:8080/runs \ + -H "Content-Type: application/json" \ + -d '{ + "target": "../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json", + "assessment": "../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json", + "output_dir": "runs/open-cmis-tck-baseline" + }' | python3 -m json.tool +``` + +The response includes `job_id`. + +Inspect job status: + +```sh +curl -sf http://127.0.0.1:8080/runs/JOB_ID | python3 -m json.tool +``` + +Fetch reports after the job status is `succeeded`: + +```sh +curl -sf http://127.0.0.1:8080/runs/JOB_ID/reports | python3 -m json.tool +``` + +Service job state is currently in memory for the running service process. Run +artifacts are durable in the output directory and can still be inspected after a +service restart. + +## Status Vocabulary + +Service jobs use transport status: + +- `queued` +- `running` +- `succeeded` +- `failed` + +Assessment runs use evidence status: + +- `completed` +- `failed` +- `blocked` +- `infrastructure_error` + +A service job can be `succeeded` while the assessment status is `blocked` or +`infrastructure_error`. That means guide-board completed the run machinery and +produced evidence explaining why the assessment could not proceed cleanly. + +Individual evidence items use: + +- `pass` +- `warning` +- `fail` +- `manual` +- `skipped` +- `blocked` +- `not_applicable` +- `expected_gap` +- `infrastructure_error` + +## Candidate System Checklist + +Before starting a run against candidate software, confirm: + +- the candidate service, repository, or artifact is in the state expected by + the target profile, +- endpoint URLs in the target profile are reachable from the guide-board + process or container, +- credentials are referenced through environment variables or mounted files, + not committed into profiles, +- the assessment profile selects only check groups that are safe for the + candidate environment, +- the output directory is outside source-controlled paths or is ignored. + +Candidate repositories should provide their own short handoff document with the +startup command, profile path, required credentials, and expected endpoint URL. diff --git a/workplans/GUIDE-BOARD-WP-0001-bootstrapping.md b/workplans/GUIDE-BOARD-WP-0001-bootstrapping.md index ef1a44f..f4f788e 100644 --- a/workplans/GUIDE-BOARD-WP-0001-bootstrapping.md +++ b/workplans/GUIDE-BOARD-WP-0001-bootstrapping.md @@ -4,12 +4,12 @@ type: workplan title: "Guide Board Bootstrapping" repo: guide-board domain: markitect -status: active +status: completed owner: codex planning_priority: high planning_order: 1 created: "2026-05-07" -updated: "2026-05-07" +updated: "2026-05-15" supersedes: - "OPEN-CMIS-TCK-WP-0001" state_hub_workstream_id: "1d812b7d-74f0-4d71-86a7-0f390b22daf7" diff --git a/workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md b/workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md new file mode 100644 index 0000000..c0720fe --- /dev/null +++ b/workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md @@ -0,0 +1,164 @@ +--- +id: GUIDE-BOARD-WP-0002 +type: workplan +title: "Assessment Operations Baseline" +repo: guide-board +domain: markitect +status: active +owner: codex +planning_priority: high +planning_order: 2 +created: "2026-05-15" +updated: "2026-05-15" +state_hub_workstream_id: "fc5b1573-91b2-4a19-b6a9-dd4d17057d9b" +--- + +# GUIDE-BOARD-WP-0002: Assessment Operations Baseline + +## Purpose + +Turn the bootstrapped framework into an operator-ready assessment tool. The +next layer should make it obvious how a candidate software repository or +operator starts an assessment, follows status, retrieves results, and packages +evidence without needing to understand guide-board internals. + +## Background + +`GUIDE-BOARD-WP-0001` established the core contracts: schemas, extension +discovery, profile validation, planning, execution, service transport, +container baseline, retention summaries, and the first external extension path +with `open-cmis-tck`. + +The next gap is operational clarity. A user should be able to start from a +candidate system and know: + +- what target and assessment profiles are required, +- which command or HTTP endpoint starts the run, +- how status differs between service jobs and assessment outcomes, +- where reports, normalized evidence, raw artifacts, and retention summaries + are written, +- what a candidate repository should document when it wants to be assessed. + +## Boundary + +This workplan owns generic guide-board assessment operation. It may reference +`open-cmis-tck` as a concrete example, but CMIS-specific harness behavior, +target profiles, and runtime dependencies remain owned by the external +`open-cmis-tck` extension repository. + +## D2.1 - Assessment Operations Runbook + +```task +id: GUIDE-BOARD-WP-0002-T001 +status: done +priority: high +state_hub_task_id: "4592bb4f-096a-49e4-8785-06f9c3f619ac" +``` + +Acceptance: + +- Add a generic operator document for CLI and local service assessment flows. +- Document start, status, report retrieval, output directories, and result + states. +- Include the external-extension shape without making CMIS the root product. +- Link the runbook from README. + +Progress: + +- Added `docs/ASSESSMENT-OPERATIONS.md`. +- Linked the runbook from README. + +## D2.2 - Candidate Repository Handoff Contract + +```task +id: GUIDE-BOARD-WP-0002-T002 +status: todo +priority: high +state_hub_task_id: "7d1df7d0-49cd-487f-a507-9e558a90af31" +``` + +Acceptance: + +- Define the minimal instruction file a candidate repository should provide + when it wants guide-board assessment support. +- Cover required endpoint/profile facts, credential references, startup + command, expected runtime state, and where candidate-specific results should + be referenced. +- Provide a template that candidate repos can copy without importing + guide-board internals. + +## D2.3 - Run History And Result UX + +```task +id: GUIDE-BOARD-WP-0002-T003 +status: todo +priority: medium +state_hub_task_id: "ce18a2dc-7cc9-4fff-9272-5333b6118030" +``` + +Acceptance: + +- Make it easy to find the latest run for a target and assessment pair. +- Document or add commands for listing run reports and opening the latest + report paths. +- Preserve the existing run directory contract and retention summary model. + +## D2.4 - Service Job Durability Contract + +```task +id: GUIDE-BOARD-WP-0002-T004 +status: todo +priority: medium +state_hub_task_id: "10e4003c-dc11-4a8e-aecc-7815559ac439" +``` + +Acceptance: + +- Decide whether local service job state remains intentionally in-memory or + gains a small durable index. +- If in-memory remains the baseline, document restart semantics and the durable + fallback through run directories. +- If durable indexing is added, keep it dependency-light and reconstructable + from retained run artifacts. + +## D2.5 - Container Smoke Acceptance + +```task +id: GUIDE-BOARD-WP-0002-T005 +status: todo +priority: medium +state_hub_task_id: "9e2e7fa7-8a97-4f07-9925-e637e0366003" +``` + +Acceptance: + +- Add or document a reproducible container smoke check for the bundled sample + assessment. +- Verify mounted output directories receive run artifacts. +- Keep restricted or extension-specific harness assets outside the core image. + +## D2.6 - External Extension Acceptance Path + +```task +id: GUIDE-BOARD-WP-0002-T006 +status: todo +priority: high +state_hub_task_id: "65fbf1df-caef-40f6-abee-8308daf27fbc" +``` + +Acceptance: + +- Define a repeatable acceptance path for an external extension repository. +- Use `open-cmis-tck` as the first example without moving CMIS-specific logic + back into the core. +- Cover extension validation, target/assessment profile validation, plan + generation, run execution, and result review. + +## Definition Of Done + +- The repo has a clear assessment operations guide. +- Candidate repositories know what handoff information to provide. +- Operators can start runs, inspect status, and retrieve results through CLI or + local service. +- The service and container baselines have documented operational limits. +- The next external extension acceptance workflow is repeatable.