coulomb/open-cmis-tck
Archived
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restructured as guide-board compliance framework

Browse Source
This commit is contained in:
Bernd Worsch
2026-05-07 09:45:11 +02:00
parent 6108f3ab51
commit 1201695ebe
7 changed files with 744 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files

185
extensions/CANDIDATES.md Normal file
View File

@@ -0,0 +1,185 @@
# Extension Candidates
This registry captures official or authority-backed conformance harnesses that
can shape `guide-board` extension design. A candidate does not imply immediate
implementation; it marks a useful pattern for later inclusion.
## Selection Criteria
- The harness, validator, or test program is maintained by a standards body,
regulator, certification authority, foundation, or recognized upstream project.
- It produces executable or structured evidence, not only narrative guidance.
- It teaches a reusable architecture pattern for profiles, checks, artifacts,
normalization, mappings, waivers, or certification boundaries.
- Its license and access model can be represented honestly, even when the tool
itself is restricted.
## Registered Candidates
### `open-cmis-tck`
- Status: seed extension.
- Domain: CMIS repository interoperability.
- Authority and sources: OASIS CMIS standard family, Apache Chemistry OpenCMIS
TCK artifacts and APIs.
- Harness pattern: Java/Maven TCK wrapper, target endpoint profile, selected test
groups, raw logs, normalized capability evidence.
- Notes: Apache Chemistry/OpenCMIS appears retired, so this extension must track
maintenance status and avoid presenting TCK results as formal certification.
- Sources:
- [Apache Chemistry OpenCMIS TCK package](https://chemistry.apache.org/java/javadoc/org/apache/chemistry/opencmis/tck/package-summary.html)
- [Maven Central artifact](https://central.sonatype.com/artifact/org.apache.chemistry.opencmis/chemistry-opencmis-test-tck)
### `ogc-team-engine`
- Status: high-priority candidate.
- Domain: geospatial services, APIs, schemas, clients, and data.
- Authority and sources: Open Geospatial Consortium Compliance Testing Program.
- Harness pattern: multi-suite conformance engine, OGC CTL/TestNG tests, web and
command-line execution, session storage, conformance classes, certification
submission boundary.
- Why it matters: it is one of the clearest examples of a general conformance
engine plus installable executable test suites.
- Sources:
- [TEAM Engine documentation](https://opengeospatial.github.io/teamengine/)
- [OGC validator](https://cite.opengeospatial.org/teamengine/)
### `openid-conformance-suite`
- Status: high-priority candidate.
- Domain: identity, authentication, authorization, OpenID Connect, FAPI.
- Authority and sources: OpenID Foundation.
- Harness pattern: open source conformance suite, hosted service, local Docker
install, test plans, public/staging environments, CI runner, certification fee
boundary.
- Why it matters: it cleanly separates free self-testing from formal OpenID
certification and shows how runnable profiles become certification packages.
- Sources:
- [OpenID Conformance Suite](https://openid.net/certification/about-conformance-suite/)
- [OpenID Certification](https://openid.net/certification/)
### `kubernetes-conformance`
- Status: candidate.
- Domain: cloud-native platform API conformance.
- Authority and sources: Cloud Native Computing Foundation.
- Harness pattern: run the same open source conformance application used for
certification, collect result artifacts, submit results for review.
- Why it matters: it shows a strong public result-submission workflow and a
versioned conformance program tied to ecosystem trust.
- Source:
- [CNCF Certified Kubernetes Software Conformance](https://www.cncf.io/certification/software-conformance/)
### `web-platform-tests`
- Status: candidate.
- Domain: browser and web platform interoperability.
- Authority and sources: web-platform-tests project across W3C, WHATWG, and web
standards communities.
- Harness pattern: massive shared test repository, manifest generation, local
runner, public live deployment, public result aggregation.
- Why it matters: it shows how specification-linked tests, shared infrastructure,
and cross-implementation result comparison can scale.
- Sources:
- [web-platform-tests documentation](https://web-platform-tests.org/)
- [web-platform-tests repository](https://github.com/web-platform-tests/wpt)
### `khronos-cts`
- Status: candidate.
- Domain: graphics, compute, and XR API conformance.
- Authority and sources: Khronos Group.
- Harness pattern: conformance test suites, submission packages, explicit CTS
versioning, automated and interactive tests, trademark/adopter boundary.
- Why it matters: it teaches artifact packaging, conformance version metadata,
and distinction between development testing and formal adopter conformance.
- Sources:
- [Vulkan CTS guide](https://docs.vulkan.org/guide/latest/vulkan_cts.html)
- [OpenXR CTS usage guide](https://registry.khronos.org/OpenXR/conformance/cts_usage.html)
### `nist-acvp`
- Status: candidate.
- Domain: cryptographic algorithm validation.
- Authority and sources: National Institute of Standards and Technology.
- Harness pattern: validation protocol, client-server vector exchange, demo and
production environments, credentialed access, standardized result reporting.
- Why it matters: it is a reference model for authority-operated validation where
the external service participates directly in evidence generation.
- Source:
- [NIST ACVP documentation](https://pages.nist.gov/ACVP/)
### `hl7-fhir-inferno`
- Status: high-priority candidate.
- Domain: healthcare interoperability and FHIR implementation guides.
- Authority and sources: HL7 FHIR ecosystem and ONC Health IT Certification
Program test tooling.
- Harness pattern: executable test kits, implementation-guide profiles, hosted
demonstration service, local execution, approved test-method boundary for
specific ONC criteria.
- Why it matters: it bridges technical API conformance and regulated health IT
certification preparation.
- Sources:
- [FHIR testing framework](https://hl7.org/fhir/testing.html)
- [Inferno on HealthIT.gov](https://fhir.healthit.gov/about/)
- [ONC Certification API Test Kit](https://fhir.healthit.gov/suites/g10_certification)
### `jakarta-ee-tck`
- Status: candidate.
- Domain: enterprise Java platform and specification compatibility.
- Authority and sources: Eclipse Foundation Jakarta EE Specification Process.
- Harness pattern: formal TCK process, compatibility rules, challenge process,
exclusions, self-certification, license boundary.
- Why it matters: it provides a mature process model for TCK governance, appeals,
and controlled claims of compatibility.
- Sources:
- [Jakarta EE TCK Process](https://jakarta.ee/committees/specification/tckprocess/)
- [Jakarta EE compatible implementations](https://jakarta.ee/committees/specification/compatibility/)
### `opc-ua-ctt`
- Status: candidate with access restrictions.
- Domain: industrial automation interoperability.
- Authority and sources: OPC Foundation.
- Harness pattern: compliance test tool for clients and servers, profiles,
facets, conformance units, CLI automation, member-only redistribution boundary.
- Why it matters: it shows how guide-board should represent restricted tools
without copying or redistributing them.
- Source:
- [OPC UA Compliance Test Tool](https://opcfoundation.org/developer-tools/certification-test-tools/opc-ua-compliance-test-tool-uactt/)
## Non-Harness Evidence Packs
Some important frameworks may not have an official executable test harness in the
same sense as a TCK or conformance suite. They should still be candidates for
guide-board evidence packs, but the extension type is different: procedural
control evidence, policy review, artifact checks, and auditor-facing readiness
reports.
Initial non-harness families to evaluate later:
- GDPR readiness and data protection evidence.
- SOC 2 Trust Services Criteria evidence.
- HIPAA privacy and security readiness evidence.
- NF Z 42-013 and NF 461 electronic archiving evidence.
- ISO 14641 electronic archiving evidence.
- ISO 15489 records-management evidence.
These packs should cite official sources and licensed standards metadata, but
must not redistribute proprietary standard text or imply automated certification.
## Architecture Lessons
The candidates point to the same core abstractions:
- authority catalog with source links, version, license, and access constraints,
- extension manifest with harness type and execution model,
- target profile schema per extension,
- run plan that separates preflight, setup, execution, teardown, and reporting,
- raw artifact store plus normalized evidence model,
- conformance class, capability, control, or requirement mapping,
- expectation and waiver model for optional or unsupported behavior,
- result package suitable for human review and possible certification submission,
- explicit boundary between preparation evidence and certification decision.

108
extensions/open-cmis-tck/INTENT.md Normal file
View File

@@ -0,0 +1,108 @@
# INTENT
## Extension Name
`open-cmis-tck`
## Parent Project
`guide-board`
## Purpose
`open-cmis-tck` is the first guide-board extension. It provides a reusable CMIS
compatibility test facility around selected Apache Chemistry OpenCMIS TCK checks.
The extension turns CMIS capability claims into guide-board evidence:
- execute selected OpenCMIS TCK groups against a target access point,
- capture raw logs and run metadata,
- normalize results into guide-board-compatible evidence,
- classify outcomes by CMIS capability area,
- distinguish unsupported-by-design behavior from implementation defects,
- produce scorecard and assessment inputs for downstream projects.
## Boundary
This extension is a test facility, not a CMIS implementation and not a
certification authority. It does not implement server behavior, replace the CMIS
specification, or claim formal certification.
Target systems own:
- CMIS endpoints,
- repository data fixtures,
- authentication and authorization behavior,
- supported and unsupported capability decisions,
- product scorecards.
This extension owns:
- target profile shape for CMIS Browser Binding checks,
- CMIS preflight probes,
- OpenCMIS TCK orchestration,
- CMIS result normalization,
- CMIS capability mapping,
- guide-board report fragments.
## Primary Use Case
Given a running CMIS Browser Binding access point, such as:
```text
http://127.0.0.1:8000/cmis/compat-tck/browser
```
the extension should:
1. load a CMIS target profile,
2. verify target reachability and repository posture,
3. run selected OpenCMIS TCK checks,
4. normalize raw TCK output,
5. map pass, fail, skip, expected gap, and infrastructure outcomes to CMIS
capability groups,
6. write guide-board-compatible JSON and Markdown reports.
## First Target Integration
The first target profile is `kontextual-engine` `compat-tck`.
Expected target URL:
```text
http://127.0.0.1:8000/cmis/compat-tck/browser
```
Initial expected posture:
- selected repository/type checks should pass,
- selected object/content checks should pass,
- navigation, query, ACL, and versioning checks may partially pass,
- AtomPub and Web Services are not target bindings,
- unsupported optional capabilities should be treated as expected skips or
explained gaps, not failures by default.
## Source Posture
The extension should track source authority and maintenance status explicitly.
Apache Chemistry OpenCMIS provides TCK package APIs and Maven artifacts, but the
Apache Chemistry project appears retired. The extension must therefore preserve
harness version metadata and avoid overstating the meaning of a TCK pass.
Useful sources:
- [Apache Chemistry OpenCMIS TCK package](https://chemistry.apache.org/java/javadoc/org/apache/chemistry/opencmis/tck/package-summary.html)
- [Maven Central artifact](https://central.sonatype.com/artifact/org.apache.chemistry.opencmis/chemistry-opencmis-test-tck)
## Success Criteria
The extension is useful when it can:
- run from a clean checkout with documented Java/Maven requirements,
- validate a CMIS Browser Binding target before TCK execution,
- execute or cleanly skip selected OpenCMIS TCK groups,
- preserve raw TCK output as optional artifacts,
- emit normalized guide-board evidence,
- map results to CMIS capability groups,
- identify expected gaps separately from unexpected failures,
- provide enough evidence to update downstream CMIS capability scorecards.

196
extensions/open-cmis-tck/workplans/OPEN-CMIS-TCK-WP-0001-harness-foundation.md Normal file
View File

@@ -0,0 +1,196 @@
---
id: OPEN-CMIS-TCK-WP-0001
type: extension-workplan
title: "OpenCMIS TCK Harness Foundation"
repo: guide-board
extension: open-cmis-tck
domain: markitect
status: active
owner: codex
planning_priority: high
planning_order: 2
created: "2026-05-07"
updated: "2026-05-07"
superseded_root_workplan: "workplans/OPEN-CMIS-TCK-WP-0001-harness-foundation.md"
state_hub_workstream_id: "276483ee-1497-495b-8710-fd8bb7dace22"
---
# OPEN-CMIS-TCK-WP-0001: OpenCMIS TCK Harness Foundation
## Purpose
Establish `open-cmis-tck` as the first guide-board extension. The first usable
outcome is a local harness that can run selected Apache Chemistry OpenCMIS TCK
checks against a configured CMIS Browser Binding target and produce normalized
guide-board evidence.
## Background
`kontextual-engine` now exposes a conservative CMIS 1.1 Browser Binding subset
with explicit unsupported flags. Its current scorecard is estimation-based.
Running selected OpenCMIS TCK checks is the next step to turn that estimate into
evidence-backed capability scoring.
This extension keeps CMIS-specific Java/Maven orchestration, raw TCK output, and
capability mapping outside the guide-board core.
## Target Architecture
```text
CMIS target profile
-> CMIS preflight probe
-> OpenCMIS TCK runner
-> raw run artifacts
-> guide-board normalized evidence
-> CMIS capability mapper
-> JSON/Markdown report fragments
```
## Boundary
This workplan builds a guide-board extension. It does not implement CMIS server
features, modify `kontextual-engine`, issue certifications, provide audit
assurance, or replace legal, regulatory, or accredited assessment work.
## D1.1 - Extension Foundation
```task
id: OPEN-CMIS-TCK-WP-0001-T001
status: todo
priority: high
state_hub_task_id: "add6a26d-38a8-4500-8a3e-6fdac43fee42"
```
Acceptance:
- `extensions/open-cmis-tck/INTENT.md` captures the extension purpose, scope,
boundaries, and first target.
- Extension README identifies how the extension plugs into guide-board.
- Basic directory layout exists for configs, scripts, reports, docs, and tests.
- Local generated artifacts are ignored where appropriate.
## D1.2 - CMIS Target Profile Schema
```task
id: OPEN-CMIS-TCK-WP-0001-T002
status: todo
priority: high
state_hub_task_id: "2ccc74a7-bed9-4769-8608-d579fdf3a0cd"
```
Acceptance:
- A CMIS target profile config format defines endpoint URL, binding, repository
ID, credentials or auth mode, expected capability groups, known gaps, and
timeout settings.
- A `kontextual-engine` `compat-tck` example profile is included.
- Profile validation produces actionable diagnostics for missing or invalid
fields.
## D1.3 - CMIS Preflight Probe
```task
id: OPEN-CMIS-TCK-WP-0001-T003
status: todo
priority: high
state_hub_task_id: "6d45885b-78a4-4e8b-8fcc-b8d6488e703b"
```
Acceptance:
- Preflight checks verify target reachability, repository info, CMIS version,
binding posture, and capability flags before invoking the Java TCK.
- Unsupported optional capabilities can be accepted as expected gaps.
- Preflight output is captured as structured JSON.
## D1.4 - OpenCMIS TCK Runner Wrapper
```task
id: OPEN-CMIS-TCK-WP-0001-T004
status: todo
priority: high
state_hub_task_id: "502d7586-6f9e-475e-9683-43260666d5d9"
```
Acceptance:
- The extension documents Java and Maven requirements.
- A runner command can invoke selected OpenCMIS TCK groups against one target.
- Raw logs and machine-readable run metadata are written under a run directory.
- TCK execution can be skipped cleanly when Java/Maven are unavailable.
## D1.5 - CMIS Result Normalization
```task
id: OPEN-CMIS-TCK-WP-0001-T005
status: todo
priority: high
state_hub_task_id: "716486b6-6f14-41f8-8417-5015ba746005"
```
Acceptance:
- Raw TCK output is normalized into the guide-board evidence schema.
- Results distinguish pass, fail, expected skip, unsupported by design, and
infrastructure error.
- Failures include enough context to map back to TCK group, capability group,
target profile, and raw artifact paths.
## D1.6 - Capability Mapping And Reports
```task
id: OPEN-CMIS-TCK-WP-0001-T006
status: todo
priority: high
state_hub_task_id: "9f7dacc5-4d19-4755-aa9a-8572d4285514"
```
Acceptance:
- Capability-group mapping aligns TCK groups with repository/type, navigation,
object/content, versioning, query, relationships, ACL/policy, change log, and
extension gaps.
- JSON reports are compact and suitable for scorecard ingestion.
- Markdown reports summarize pass/fail/skip counts and unexpected gaps.
- Known gaps do not hide unexpected failures in the same capability area.
## D1.7 - Optional Local Service API Adapter
```task
id: OPEN-CMIS-TCK-WP-0001-T007
status: todo
priority: medium
state_hub_task_id: "a05e47bd-88db-4878-aef4-bf328790c3f0"
```
Acceptance:
- The guide-board service can list CMIS profiles, start a CMIS run, inspect run
status, and fetch normalized CMIS reports.
- CLI operation remains the primary path.
- Long-running TCK jobs are tracked without blocking the API process.
## D1.8 - Historical Result Retention
```task
id: OPEN-CMIS-TCK-WP-0001-T008
status: todo
priority: medium
state_hub_task_id: "c27ea43f-41ec-49d0-a890-3681455f7c6c"
```
Acceptance:
- The extension contributes compact run summaries over time.
- Retention avoids unbounded raw log growth.
- Summaries are suitable for trend charts and downstream capability-score
updates.
## Definition Of Done
- A developer can configure a CMIS Browser Binding endpoint.
- Preflight can verify the target before TCK execution.
- Selected OpenCMIS TCK checks can be invoked or cleanly skipped when
dependencies are missing.
- Normalized guide-board JSON and Markdown report fragments are generated.
- The `kontextual-engine` `compat-tck` target profile is represented.