generated from coulomb/repo-seed
maturity scorecard generation
This commit is contained in:
77
docs/CMIS-MATURITY-SCORECARD.md
Normal file
77
docs/CMIS-MATURITY-SCORECARD.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# CMIS Capability Maturity Scorecard
|
||||
|
||||
Status: draft
|
||||
Created: 2026-05-08
|
||||
|
||||
## Purpose
|
||||
|
||||
The CMIS scorecard is an interpretation layer over guide-board evidence. It does
|
||||
not replace the raw evidence, findings, mappings, or assessment package. It
|
||||
summarizes how mature the target appears across Browser Binding capability
|
||||
groups.
|
||||
|
||||
It does not certify CMIS conformance.
|
||||
|
||||
## Generate
|
||||
|
||||
After a guide-board run:
|
||||
|
||||
```sh
|
||||
cd /home/worsch/open-cmis-tck
|
||||
PYTHONPATH=src python3 scripts/cmis_scorecard.py \
|
||||
--run-dir /tmp/open-cmis-tck-live
|
||||
```
|
||||
|
||||
Outputs:
|
||||
|
||||
```text
|
||||
/tmp/open-cmis-tck-live/reports/cmis-maturity-scorecard.json
|
||||
/tmp/open-cmis-tck-live/reports/cmis-maturity-scorecard.md
|
||||
```
|
||||
|
||||
To preview without writing:
|
||||
|
||||
```sh
|
||||
PYTHONPATH=src python3 scripts/cmis_scorecard.py \
|
||||
--run-dir /tmp/open-cmis-tck-live \
|
||||
--print
|
||||
```
|
||||
|
||||
## Capability Groups
|
||||
|
||||
The first model scores:
|
||||
|
||||
- repository/type metadata
|
||||
- object/content services
|
||||
- navigation services
|
||||
- query
|
||||
- relationships
|
||||
- ACL/policy
|
||||
- versioning
|
||||
- change log
|
||||
- extensions and known gaps
|
||||
|
||||
## Levels
|
||||
|
||||
Each group receives a 0-4 score:
|
||||
|
||||
- `0`: not assessed or unknown
|
||||
- `1`: blocked, infrastructure-blocked, or unexpectedly failing
|
||||
- `2`: scoped gap, unsupported by design, skipped, manual, or incomplete
|
||||
- `3`: partial, warnings need review
|
||||
- `4`: demonstrated by mapped passing evidence
|
||||
|
||||
The overall score is a weighted percentage across all groups. Core repository
|
||||
and object/content groups carry more weight than optional or gap-review groups.
|
||||
|
||||
## Evidence Boundary
|
||||
|
||||
The scorecard uses:
|
||||
|
||||
- `normalized/evidence.json`
|
||||
- `normalized/findings.json`
|
||||
- `normalized/mappings.json`
|
||||
- `reports/assessment-package.json`
|
||||
|
||||
It keeps conformance evidence separate from maturity interpretation. A formal
|
||||
review should always inspect the underlying evidence and raw artifacts.
|
||||
@@ -61,9 +61,18 @@ Expected dry-run artifacts:
|
||||
|
||||
If preflight fails, fix the target profile or endpoint before continuing.
|
||||
|
||||
Generate a maturity scorecard from the dry-run output:
|
||||
|
||||
```sh
|
||||
cd /home/worsch/open-cmis-tck
|
||||
PYTHONPATH=src python3 scripts/cmis_scorecard.py \
|
||||
--run-dir /tmp/open-cmis-tck-dry-run
|
||||
```
|
||||
|
||||
## Install Java And Maven
|
||||
|
||||
The current WSL environment needs Java and Maven before the real TCK can run:
|
||||
The current WSL environment does not expose system `java` and `mvn` on `PATH`.
|
||||
Either install them as system packages:
|
||||
|
||||
```sh
|
||||
sudo apt-get update
|
||||
@@ -73,10 +82,33 @@ sudo apt-get install -y openjdk-17-jdk maven
|
||||
Use a managed local Java/Maven installation instead if preferred. The bootstrap
|
||||
only requires `java` and `mvn` on `PATH`.
|
||||
|
||||
Or use the repo-local toolchain under `.local/`:
|
||||
|
||||
```sh
|
||||
cd /home/worsch/open-cmis-tck
|
||||
python3 scripts/install_local_toolchain.py
|
||||
source .local/toolchains/env.sh
|
||||
PYTHONPATH=src python3 scripts/bootstrap_opencmis_tck.py --resolve
|
||||
```
|
||||
|
||||
The local installer downloads a Linux x64 Temurin JDK 17 archive and Apache
|
||||
Maven 3.9.11, extracts them under `.local/toolchains`, verifies Maven's SHA-512
|
||||
checksum, writes `.local/toolchains/env.sh`, and leaves the downloaded binaries
|
||||
outside version control.
|
||||
|
||||
This workspace has already been bootstrapped with the repo-local path. In a new
|
||||
shell, source the environment file before running live TCK commands:
|
||||
|
||||
```sh
|
||||
cd /home/worsch/open-cmis-tck
|
||||
source .local/toolchains/env.sh
|
||||
```
|
||||
|
||||
## Resolve The TCK Runtime
|
||||
|
||||
```sh
|
||||
cd /home/worsch/open-cmis-tck
|
||||
source .local/toolchains/env.sh
|
||||
PYTHONPATH=src python3 scripts/bootstrap_opencmis_tck.py --resolve
|
||||
```
|
||||
|
||||
@@ -100,6 +132,7 @@ After bootstrap reports `ready`, run the baseline assessment:
|
||||
|
||||
```sh
|
||||
cd /home/worsch/guide-board
|
||||
source /home/worsch/open-cmis-tck/.local/toolchains/env.sh
|
||||
PYTHONPATH=src python3 -m guide_board \
|
||||
--extension-dir ../open-cmis-tck \
|
||||
run \
|
||||
@@ -116,6 +149,26 @@ The baseline currently selects:
|
||||
Expand selected check groups only after the repository/type run produces useful
|
||||
output.
|
||||
|
||||
Then generate the maturity scorecard:
|
||||
|
||||
```sh
|
||||
cd /home/worsch/open-cmis-tck
|
||||
PYTHONPATH=src python3 scripts/cmis_scorecard.py \
|
||||
--run-dir /tmp/open-cmis-tck-live
|
||||
```
|
||||
|
||||
Review the real TCK evidence before expanding the scope:
|
||||
|
||||
```text
|
||||
/tmp/open-cmis-tck-live/normalized/evidence.json
|
||||
/tmp/open-cmis-tck-live/artifacts/open-cmis-tck/tck/<check-group>/console-runner-stdout.txt
|
||||
/tmp/open-cmis-tck-live/artifacts/open-cmis-tck/tck/<check-group>/normalized-runner-result.json
|
||||
```
|
||||
|
||||
The normalizer preserves native OpenCMIS statuses (`OK`, `WARNING`, `FAILURE`,
|
||||
`SKIPPED`, `UNEXPECTED_EXCEPTION`, and `INFO`) as case-level facts while mapping
|
||||
the overall check group to guide-board's result vocabulary.
|
||||
|
||||
## Authenticated Targets
|
||||
|
||||
For environment credentials:
|
||||
|
||||
@@ -74,6 +74,17 @@ sudo apt-get install -y openjdk-17-jdk maven
|
||||
Use an already managed Java/Maven installation instead if this workstation has
|
||||
one outside WSL.
|
||||
|
||||
When sudo is not available, use the local toolchain installer:
|
||||
|
||||
```sh
|
||||
python3 scripts/install_local_toolchain.py
|
||||
source .local/toolchains/env.sh
|
||||
```
|
||||
|
||||
This keeps the JDK and Maven under `.local/toolchains`, which is ignored by git.
|
||||
It is intended as a workstation bootstrap convenience, not as a committed
|
||||
runtime artifact.
|
||||
|
||||
## Guide-Board Invocation
|
||||
|
||||
For the full local sequence, see `docs/LOCAL-RUNBOOK.md`.
|
||||
@@ -101,6 +112,11 @@ runtime/opencmis-tck/pom.xml
|
||||
That Maven descriptor pulls the OpenCMIS TCK artifact and runs
|
||||
`ConsoleRunner`.
|
||||
|
||||
The adapter normalizes the native `ConsoleRunner` text report into guide-board
|
||||
case evidence. The retained raw stdout/stderr files remain the audit trail; the
|
||||
normalized result records OpenCMIS result statuses, test names, messages,
|
||||
source locations where present, and per-status counts.
|
||||
|
||||
## Session Parameters
|
||||
|
||||
For Browser Binding runs, the adapter writes OpenCMIS session parameters such
|
||||
|
||||
@@ -97,11 +97,18 @@ The wrapper normalizes, in order:
|
||||
|
||||
1. JSON written to stdout with a `tests`, `cases`, or `results` array.
|
||||
2. JUnit-style XML files written directly into `{artifact_dir}`.
|
||||
3. Exit code only, when no structured output is found.
|
||||
3. Native OpenCMIS `TextReport` output written by `ConsoleRunner`.
|
||||
4. Exit code only, when no structured output is found.
|
||||
|
||||
Case statuses normalize to guide-board result vocabulary: `pass`, `fail`,
|
||||
`skipped`, `expected_gap`, `unsupported_by_design`, `infrastructure_error`, and
|
||||
related core statuses.
|
||||
|
||||
This is enough to run a real local TCK adapter while preserving raw logs for
|
||||
future Apache Chemistry-specific parsing refinements.
|
||||
The native text normalizer follows Apache Chemistry's `TextReport` structure:
|
||||
group headings, test headings, and result lines whose statuses are `INFO`,
|
||||
`SKIPPED`, `OK`, `WARNING`, `FAILURE`, or `UNEXPECTED_EXCEPTION`. It preserves
|
||||
the OpenCMIS status, test name, group name, source file/line where present,
|
||||
message, duration, selected check group, and raw artifact references.
|
||||
|
||||
This is enough to run the real local ConsoleRunner adapter while retaining the
|
||||
raw logs needed for later pilot-review refinements.
|
||||
|
||||
Reference in New Issue
Block a user