generated from coulomb/repo-seed
Close bootstrap and start assessment operations
This commit is contained in:
@@ -50,6 +50,7 @@ See:
|
|||||||
|
|
||||||
- [INTENT.md](INTENT.md)
|
- [INTENT.md](INTENT.md)
|
||||||
- [docs/ARCHITECTURE-BLUEPRINT.md](docs/ARCHITECTURE-BLUEPRINT.md)
|
- [docs/ARCHITECTURE-BLUEPRINT.md](docs/ARCHITECTURE-BLUEPRINT.md)
|
||||||
|
- [docs/ASSESSMENT-OPERATIONS.md](docs/ASSESSMENT-OPERATIONS.md)
|
||||||
- [docs/COMPLIANCE-EVIDENCE-PACKS.md](docs/COMPLIANCE-EVIDENCE-PACKS.md)
|
- [docs/COMPLIANCE-EVIDENCE-PACKS.md](docs/COMPLIANCE-EVIDENCE-PACKS.md)
|
||||||
- [docs/CONTAINER.md](docs/CONTAINER.md)
|
- [docs/CONTAINER.md](docs/CONTAINER.md)
|
||||||
- [docs/EXTENSION-SDK.md](docs/EXTENSION-SDK.md)
|
- [docs/EXTENSION-SDK.md](docs/EXTENSION-SDK.md)
|
||||||
|
|||||||
193
docs/ASSESSMENT-OPERATIONS.md
Normal file
193
docs/ASSESSMENT-OPERATIONS.md
Normal file
@@ -0,0 +1,193 @@
|
|||||||
|
# Assessment Operations
|
||||||
|
|
||||||
|
Status: draft
|
||||||
|
Created: 2026-05-15
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document is the generic operator path for running guide-board assessments.
|
||||||
|
It covers the local CLI and the dependency-light local service API. Extension
|
||||||
|
repositories can provide domain-specific profiles and runtime setup, but the
|
||||||
|
run, status, and result contracts are owned here.
|
||||||
|
|
||||||
|
Guide-board prepares structured evidence and assessment packages. It does not
|
||||||
|
issue certifications, provide audit assurance, or replace an accredited
|
||||||
|
assessment process.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
Every run needs:
|
||||||
|
|
||||||
|
- a target profile,
|
||||||
|
- an assessment profile,
|
||||||
|
- any extension repository roots required by the assessment profile,
|
||||||
|
- optional credentials or mounted harness assets referenced by the target or
|
||||||
|
assessment profile,
|
||||||
|
- an output directory for run artifacts.
|
||||||
|
|
||||||
|
The target profile describes the candidate system or artifact being assessed.
|
||||||
|
The assessment profile selects frameworks, extensions, check groups, runtime
|
||||||
|
policy, waivers, expectations, and output policy.
|
||||||
|
|
||||||
|
## CLI Flow
|
||||||
|
|
||||||
|
Run from the guide-board repository:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd /home/worsch/guide-board
|
||||||
|
PYTHONPATH=src python3 -m guide_board extensions validate
|
||||||
|
PYTHONPATH=src python3 -m guide_board profile validate-target profiles/targets/sample-repository.json
|
||||||
|
PYTHONPATH=src python3 -m guide_board profile validate-assessment profiles/assessments/sample-noop.json
|
||||||
|
PYTHONPATH=src python3 -m guide_board plan \
|
||||||
|
--target profiles/targets/sample-repository.json \
|
||||||
|
--assessment profiles/assessments/sample-noop.json
|
||||||
|
PYTHONPATH=src python3 -m guide_board run \
|
||||||
|
--target profiles/targets/sample-repository.json \
|
||||||
|
--assessment profiles/assessments/sample-noop.json \
|
||||||
|
--output-dir runs/sample-noop
|
||||||
|
```
|
||||||
|
|
||||||
|
For an external extension repository, pass `--extension-dir` before the
|
||||||
|
subcommand:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd /home/worsch/guide-board
|
||||||
|
PYTHONPATH=src python3 -m guide_board \
|
||||||
|
--extension-dir ../open-cmis-tck \
|
||||||
|
run \
|
||||||
|
--target ../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json \
|
||||||
|
--assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
|
||||||
|
--output-dir runs/open-cmis-tck-baseline
|
||||||
|
```
|
||||||
|
|
||||||
|
The same extension roots can be provided through `GUIDE_BOARD_EXTENSION_PATHS`
|
||||||
|
when a wrapper script or container entrypoint should keep commands shorter.
|
||||||
|
|
||||||
|
## CLI Results
|
||||||
|
|
||||||
|
A completed CLI command prints a JSON result with:
|
||||||
|
|
||||||
|
- `status`: assessment outcome,
|
||||||
|
- `run_id`: generated run identifier,
|
||||||
|
- `run_dir`: output directory,
|
||||||
|
- `assessment_package`: JSON assessment package path,
|
||||||
|
- `report`: Markdown report path,
|
||||||
|
- `retention_summary`: compact durable summary path.
|
||||||
|
|
||||||
|
The output directory uses this contract:
|
||||||
|
|
||||||
|
```text
|
||||||
|
run.json
|
||||||
|
plan.json
|
||||||
|
retention-summary.json
|
||||||
|
normalized/evidence.json
|
||||||
|
normalized/findings.json
|
||||||
|
normalized/mappings.json
|
||||||
|
reports/assessment-package.json
|
||||||
|
reports/report.md
|
||||||
|
artifacts/
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the retained run helpers for history:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
PYTHONPATH=src python3 -m guide_board runs list --runs-dir runs
|
||||||
|
PYTHONPATH=src python3 -m guide_board runs trend --runs-dir runs
|
||||||
|
PYTHONPATH=src python3 -m guide_board runs gate --runs-dir runs
|
||||||
|
```
|
||||||
|
|
||||||
|
## Local Service Flow
|
||||||
|
|
||||||
|
Start the service from the guide-board repository:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd /home/worsch/guide-board
|
||||||
|
PYTHONPATH=src python3 -m guide_board \
|
||||||
|
--extension-dir ../open-cmis-tck \
|
||||||
|
serve --host 127.0.0.1 --port 8080
|
||||||
|
```
|
||||||
|
|
||||||
|
Check health:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -sf http://127.0.0.1:8080/health | python3 -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
Start a run:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -sf -X POST http://127.0.0.1:8080/runs \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"target": "../open-cmis-tck/profiles/targets/kontextual-cmis-compat.json",
|
||||||
|
"assessment": "../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json",
|
||||||
|
"output_dir": "runs/open-cmis-tck-baseline"
|
||||||
|
}' | python3 -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
The response includes `job_id`.
|
||||||
|
|
||||||
|
Inspect job status:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -sf http://127.0.0.1:8080/runs/JOB_ID | python3 -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
Fetch reports after the job status is `succeeded`:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
curl -sf http://127.0.0.1:8080/runs/JOB_ID/reports | python3 -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
Service job state is currently in memory for the running service process. Run
|
||||||
|
artifacts are durable in the output directory and can still be inspected after a
|
||||||
|
service restart.
|
||||||
|
|
||||||
|
## Status Vocabulary
|
||||||
|
|
||||||
|
Service jobs use transport status:
|
||||||
|
|
||||||
|
- `queued`
|
||||||
|
- `running`
|
||||||
|
- `succeeded`
|
||||||
|
- `failed`
|
||||||
|
|
||||||
|
Assessment runs use evidence status:
|
||||||
|
|
||||||
|
- `completed`
|
||||||
|
- `failed`
|
||||||
|
- `blocked`
|
||||||
|
- `infrastructure_error`
|
||||||
|
|
||||||
|
A service job can be `succeeded` while the assessment status is `blocked` or
|
||||||
|
`infrastructure_error`. That means guide-board completed the run machinery and
|
||||||
|
produced evidence explaining why the assessment could not proceed cleanly.
|
||||||
|
|
||||||
|
Individual evidence items use:
|
||||||
|
|
||||||
|
- `pass`
|
||||||
|
- `warning`
|
||||||
|
- `fail`
|
||||||
|
- `manual`
|
||||||
|
- `skipped`
|
||||||
|
- `blocked`
|
||||||
|
- `not_applicable`
|
||||||
|
- `expected_gap`
|
||||||
|
- `infrastructure_error`
|
||||||
|
|
||||||
|
## Candidate System Checklist
|
||||||
|
|
||||||
|
Before starting a run against candidate software, confirm:
|
||||||
|
|
||||||
|
- the candidate service, repository, or artifact is in the state expected by
|
||||||
|
the target profile,
|
||||||
|
- endpoint URLs in the target profile are reachable from the guide-board
|
||||||
|
process or container,
|
||||||
|
- credentials are referenced through environment variables or mounted files,
|
||||||
|
not committed into profiles,
|
||||||
|
- the assessment profile selects only check groups that are safe for the
|
||||||
|
candidate environment,
|
||||||
|
- the output directory is outside source-controlled paths or is ignored.
|
||||||
|
|
||||||
|
Candidate repositories should provide their own short handoff document with the
|
||||||
|
startup command, profile path, required credentials, and expected endpoint URL.
|
||||||
@@ -4,12 +4,12 @@ type: workplan
|
|||||||
title: "Guide Board Bootstrapping"
|
title: "Guide Board Bootstrapping"
|
||||||
repo: guide-board
|
repo: guide-board
|
||||||
domain: markitect
|
domain: markitect
|
||||||
status: active
|
status: completed
|
||||||
owner: codex
|
owner: codex
|
||||||
planning_priority: high
|
planning_priority: high
|
||||||
planning_order: 1
|
planning_order: 1
|
||||||
created: "2026-05-07"
|
created: "2026-05-07"
|
||||||
updated: "2026-05-07"
|
updated: "2026-05-15"
|
||||||
supersedes:
|
supersedes:
|
||||||
- "OPEN-CMIS-TCK-WP-0001"
|
- "OPEN-CMIS-TCK-WP-0001"
|
||||||
state_hub_workstream_id: "1d812b7d-74f0-4d71-86a7-0f390b22daf7"
|
state_hub_workstream_id: "1d812b7d-74f0-4d71-86a7-0f390b22daf7"
|
||||||
|
|||||||
164
workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md
Normal file
164
workplans/GUIDE-BOARD-WP-0002-assessment-operations-baseline.md
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
---
|
||||||
|
id: GUIDE-BOARD-WP-0002
|
||||||
|
type: workplan
|
||||||
|
title: "Assessment Operations Baseline"
|
||||||
|
repo: guide-board
|
||||||
|
domain: markitect
|
||||||
|
status: active
|
||||||
|
owner: codex
|
||||||
|
planning_priority: high
|
||||||
|
planning_order: 2
|
||||||
|
created: "2026-05-15"
|
||||||
|
updated: "2026-05-15"
|
||||||
|
state_hub_workstream_id: "fc5b1573-91b2-4a19-b6a9-dd4d17057d9b"
|
||||||
|
---
|
||||||
|
|
||||||
|
# GUIDE-BOARD-WP-0002: Assessment Operations Baseline
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Turn the bootstrapped framework into an operator-ready assessment tool. The
|
||||||
|
next layer should make it obvious how a candidate software repository or
|
||||||
|
operator starts an assessment, follows status, retrieves results, and packages
|
||||||
|
evidence without needing to understand guide-board internals.
|
||||||
|
|
||||||
|
## Background
|
||||||
|
|
||||||
|
`GUIDE-BOARD-WP-0001` established the core contracts: schemas, extension
|
||||||
|
discovery, profile validation, planning, execution, service transport,
|
||||||
|
container baseline, retention summaries, and the first external extension path
|
||||||
|
with `open-cmis-tck`.
|
||||||
|
|
||||||
|
The next gap is operational clarity. A user should be able to start from a
|
||||||
|
candidate system and know:
|
||||||
|
|
||||||
|
- what target and assessment profiles are required,
|
||||||
|
- which command or HTTP endpoint starts the run,
|
||||||
|
- how status differs between service jobs and assessment outcomes,
|
||||||
|
- where reports, normalized evidence, raw artifacts, and retention summaries
|
||||||
|
are written,
|
||||||
|
- what a candidate repository should document when it wants to be assessed.
|
||||||
|
|
||||||
|
## Boundary
|
||||||
|
|
||||||
|
This workplan owns generic guide-board assessment operation. It may reference
|
||||||
|
`open-cmis-tck` as a concrete example, but CMIS-specific harness behavior,
|
||||||
|
target profiles, and runtime dependencies remain owned by the external
|
||||||
|
`open-cmis-tck` extension repository.
|
||||||
|
|
||||||
|
## D2.1 - Assessment Operations Runbook
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0002-T001
|
||||||
|
status: done
|
||||||
|
priority: high
|
||||||
|
state_hub_task_id: "4592bb4f-096a-49e4-8785-06f9c3f619ac"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Add a generic operator document for CLI and local service assessment flows.
|
||||||
|
- Document start, status, report retrieval, output directories, and result
|
||||||
|
states.
|
||||||
|
- Include the external-extension shape without making CMIS the root product.
|
||||||
|
- Link the runbook from README.
|
||||||
|
|
||||||
|
Progress:
|
||||||
|
|
||||||
|
- Added `docs/ASSESSMENT-OPERATIONS.md`.
|
||||||
|
- Linked the runbook from README.
|
||||||
|
|
||||||
|
## D2.2 - Candidate Repository Handoff Contract
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0002-T002
|
||||||
|
status: todo
|
||||||
|
priority: high
|
||||||
|
state_hub_task_id: "7d1df7d0-49cd-487f-a507-9e558a90af31"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Define the minimal instruction file a candidate repository should provide
|
||||||
|
when it wants guide-board assessment support.
|
||||||
|
- Cover required endpoint/profile facts, credential references, startup
|
||||||
|
command, expected runtime state, and where candidate-specific results should
|
||||||
|
be referenced.
|
||||||
|
- Provide a template that candidate repos can copy without importing
|
||||||
|
guide-board internals.
|
||||||
|
|
||||||
|
## D2.3 - Run History And Result UX
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0002-T003
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
state_hub_task_id: "ce18a2dc-7cc9-4fff-9272-5333b6118030"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Make it easy to find the latest run for a target and assessment pair.
|
||||||
|
- Document or add commands for listing run reports and opening the latest
|
||||||
|
report paths.
|
||||||
|
- Preserve the existing run directory contract and retention summary model.
|
||||||
|
|
||||||
|
## D2.4 - Service Job Durability Contract
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0002-T004
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
state_hub_task_id: "10e4003c-dc11-4a8e-aecc-7815559ac439"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Decide whether local service job state remains intentionally in-memory or
|
||||||
|
gains a small durable index.
|
||||||
|
- If in-memory remains the baseline, document restart semantics and the durable
|
||||||
|
fallback through run directories.
|
||||||
|
- If durable indexing is added, keep it dependency-light and reconstructable
|
||||||
|
from retained run artifacts.
|
||||||
|
|
||||||
|
## D2.5 - Container Smoke Acceptance
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0002-T005
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
state_hub_task_id: "9e2e7fa7-8a97-4f07-9925-e637e0366003"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Add or document a reproducible container smoke check for the bundled sample
|
||||||
|
assessment.
|
||||||
|
- Verify mounted output directories receive run artifacts.
|
||||||
|
- Keep restricted or extension-specific harness assets outside the core image.
|
||||||
|
|
||||||
|
## D2.6 - External Extension Acceptance Path
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: GUIDE-BOARD-WP-0002-T006
|
||||||
|
status: todo
|
||||||
|
priority: high
|
||||||
|
state_hub_task_id: "65fbf1df-caef-40f6-abee-8308daf27fbc"
|
||||||
|
```
|
||||||
|
|
||||||
|
Acceptance:
|
||||||
|
|
||||||
|
- Define a repeatable acceptance path for an external extension repository.
|
||||||
|
- Use `open-cmis-tck` as the first example without moving CMIS-specific logic
|
||||||
|
back into the core.
|
||||||
|
- Cover extension validation, target/assessment profile validation, plan
|
||||||
|
generation, run execution, and result review.
|
||||||
|
|
||||||
|
## Definition Of Done
|
||||||
|
|
||||||
|
- The repo has a clear assessment operations guide.
|
||||||
|
- Candidate repositories know what handoff information to provide.
|
||||||
|
- Operators can start runs, inspect status, and retrieve results through CLI or
|
||||||
|
local service.
|
||||||
|
- The service and container baselines have documented operational limits.
|
||||||
|
- The next external extension acceptance workflow is repeatable.
|
||||||
Reference in New Issue
Block a user