First guide-board CMIS assessment run

2026-05-08 10:26:18 +02:00
parent ec60a67df6
commit 6382a5a7ab
3 changed files with 533 additions and 0 deletions
--- a/docs/cmis-1-1-capability-scorecard.md
+++ b/docs/cmis-1-1-capability-scorecard.md
@@ -2,6 +2,13 @@

 Date: 2026-05-07

+Evidence update: the 2026-05-08 OpenCMIS TCK run found that broad commodity
+CMIS client compatibility is currently pre-compliance because OpenCMIS cannot
+create a Browser Binding session. See
+`docs/cmis-opencmis-tck-assessment-2026-05-08T063312Z.md`. The score below is
+therefore retained as the pre-TCK product-depth estimate, not as external CMIS
+compatibility evidence.
+
 Status: baseline scorecard for the current Browser Binding subset.

 ## Purpose
--- a/docs/cmis-opencmis-tck-assessment-2026-05-08T063312Z.md
+++ b/docs/cmis-opencmis-tck-assessment-2026-05-08T063312Z.md
@@ -0,0 +1,277 @@
+# CMIS OpenCMIS TCK Assessment
+
+Run timestamp: 2026-05-08T06:33:12Z
+Local timestamp: 2026-05-08T08:33:12+02:00
+Status: evidence-backed compatibility blocker found
+
+## Purpose
+
+Persist the first real `guide-board` + `open-cmis-tck` assessment against a
+running `kontextual-engine` CMIS access point. This document supersedes prior
+pre-TCK estimates for broad commodity CMIS client compatibility.
+
+The assessment distinguishes two separate facts:
+
+- The native Kontextual CMIS-inspired route layer exposes useful document,
+  content-stream, query, ACL, relationship, parent, and profile behavior.
+- The current endpoint is not yet consumable as a standard CMIS 1.1 Browser
+  Binding endpoint by Apache Chemistry OpenCMIS.
+
+## Scope
+
+Target repository: `kontextual-engine`
+
+External test facility:
+
+- `guide-board`
+- `open-cmis-tck`
+- Apache Chemistry OpenCMIS TCK 1.1.0
+- Browser Binding profile: `cmis-browser-baseline`
+
+Runtime target used for this run:
+
+- URL: `http://127.0.0.1:8010/cmis/compat-tck/browser`
+- Access point: `compat-tck`
+- Reported repository ID: `kontextual-compat-tck`
+- TCK profile repository ID: `compat-tck`
+
+The production profile file currently points to port `8000`; this run used a
+temporary copy redirected to port `8010` because port `8000` was already owned
+by another local service that returned 404 for the CMIS probes.
+
+## Commands Run
+
+Validate the `open-cmis-tck` guide-board extension:
+
+```bash
+cd /home/worsch/guide-board
+PYTHONPATH=src python3 -m guide_board --extension-dir ../open-cmis-tck extensions validate
+```
+
+Run the live OpenCMIS baseline against the temporary target profile:
+
+```bash
+cd /home/worsch/guide-board
+source /home/worsch/open-cmis-tck/.local/toolchains/env.sh
+PYTHONPATH=src python3 -m guide_board \
+  --extension-dir ../open-cmis-tck \
+  run \
+  --target /tmp/kontextual-cmis-compat-8010.json \
+  --assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
+  --output-dir /tmp/open-cmis-tck-kontextual-live-20260508
+```
+
+Generate the scorecard:
+
+```bash
+cd /home/worsch/open-cmis-tck
+PYTHONPATH=src python3 scripts/cmis_scorecard.py \
+  --run-dir /tmp/open-cmis-tck-kontextual-live-20260508
+```
+
+Run focused internal CMIS tests with service dependencies available:
+
+```bash
+cd /home/worsch/kontextual-engine
+.venv/bin/python -m pytest tests/cmis --perf-history-disable
+```
+
+## External TCK Result
+
+Guide Board live run:
+
+- Run ID: `run-20260508T063312Z`
+- Assessment status: `infrastructure_error`
+- Summary: `pass: 1`, `infrastructure_error: 2`
+- Preflight: pass; endpoint was reachable and returned parseable JSON.
+- Repository/type TCK group: blocked before test cases could be parsed.
+- Object/content TCK group: blocked before test cases could be parsed.
+
+Generated scorecard:
+
+- Maturity score: `9.52`
+- Coverage: `2/9` groups, `22.22%`
+- Repository and type metadata: blocked, score `1/4`
+- Object and content services: blocked, score `1/4`
+- Navigation, query, relationships, ACL/policy, versioning, change log, and
+  extensions: not assessed.
+
+Primary blocker from OpenCMIS ConsoleRunner:
+
+```text
+Found invalid Repository Info!
+```
+
+The same failure occurs in:
+
+- `BasicsTestGroup` and `TypesTestGroup`
+- `CRUDTestGroup`
+
+No OpenCMIS semantic cases were parsed because the OpenCMIS client could not
+establish a valid CMIS Browser Binding session.
+
+## Evidence Artifacts
+
+Run artifacts were generated under `/tmp` and may be ephemeral. The critical
+facts are copied into this document.
+
+- `/tmp/open-cmis-tck-kontextual-live-20260508/reports/report.md`
+- `/tmp/open-cmis-tck-kontextual-live-20260508/reports/cmis-maturity-scorecard.md`
+- `/tmp/open-cmis-tck-kontextual-live-20260508/normalized/evidence.json`
+- `/tmp/open-cmis-tck-kontextual-live-20260508/normalized/findings.json`
+- `/tmp/open-cmis-tck-kontextual-live-20260508/artifacts/open-cmis-tck/preflight/response-body.bin`
+- `/tmp/open-cmis-tck-kontextual-live-20260508/artifacts/open-cmis-tck/tck/repository-type/console-runner-stdout.txt`
+- `/tmp/open-cmis-tck-kontextual-live-20260508/artifacts/open-cmis-tck/tck/object-content/console-runner-stdout.txt`
+
+## Repository Info Shape Observed
+
+Preflight observed a JSON object with these top-level keys:
+
+```text
+binding
+capabilities
+cmis_version_supported
+compliance
+principal_anonymous
+principal_anyone
+product_name
+product_version
+profile
+repository_description
+repository_features
+repository_id
+repository_name
+root_folder_id
+unsupported_features
+vendor_name
+```
+
+The endpoint reports:
+
+- `repository_id`: `kontextual-compat-tck`
+- `repository_name`: `Kontextual Engine compat-tck`
+- `cmis_version_supported`: `1.1`
+- `root_folder_id`: `cmis-root`
+- `binding`: `browser`
+
+The capability map also uses snake_case keys, for example:
+
+- `capability_query`
+- `capability_acl`
+- `capability_content_stream_updatability`
+- `capability_multifiling`
+
+OpenCMIS expects CMIS Browser Binding repository information/service document
+shape, including CMIS field names such as:
+
+- `repositoryId`
+- `repositoryName`
+- `cmisVersionSupported`
+- `rootFolderId`
+- Browser Binding capability names in the shape expected by the OpenCMIS client
+
+The assessment therefore classifies the immediate blocker as protocol contract
+shape, not absence of all underlying engine capabilities.
+
+## Internal CMIS Test Result
+
+Focused internal test run:
+
+- Command: `.venv/bin/python -m pytest tests/cmis --perf-history-disable`
+- Result: `42 passed`, `3 failed`
+
+Passing areas include:
+
+- access profile modeling
+- compliance flags
+- contract examples
+- domain mapper
+- fixture integration
+- runtime Browser Binding behavior
+
+Failures:
+
+1. OpenAPI generation fails around the streaming route return annotation.
+   Pydantic reports that the `StreamingResponse` forward reference is not fully
+   defined when generating `/openapi.json`.
+2. `test_cmis_readonly_children_object_content_query_relationships_and_changes`
+   expects root `/children` to include documents directly. Current behavior
+   returns first-level projected folders at root.
+3. `test_cmis_profile_gates_visibility_by_access_point` has the same root
+   `/children` expectation for admin/export visibility.
+
+The navigation failures are contract alignment issues: either root children
+should expose documents directly for the CMIS profile, or the tests should ask
+for the folder that contains those projected documents. The OpenAPI failure is
+a concrete implementation bug.
+
+## Native Route Probe Result
+
+Direct live probes confirmed useful native behavior:
+
+- `/cmis` lists multiple profiled access points.
+- `/cmis/compat-tck/browser` returns repository metadata and conservative
+  unsupported-feature diagnostics.
+- `/cmis/compat-tck/browser/types` returns six type projections.
+- `/cmis/compat-tck/browser/children` returns synthetic root folders.
+- `/cmis/compat-tck/browser/query?q=SELECT * FROM cmis:document` returns
+  document projections.
+- `/cmis/compat-tck/browser/object/{object_id}` returns object properties,
+  content stream metadata, allowable actions, version projection, and ACL
+  projection.
+- `/cmis/compat-tck/browser/content-bytes/{object_id}` returns real
+  `text/markdown` bytes.
+- `/cmis/compat-tck/browser/parents/{object_id}` returns projection-only parent
+  folders.
+- Unsupported query/property-update cases return structured diagnostics.
+
+This confirms that content-stream and object-projection foundations exist. The
+OpenCMIS blocker is the external CMIS Browser Binding adapter contract.
+
+## Capability Interpretation
+
+Evidence-backed commodity CMIS compatibility is currently lower than the
+pre-TCK estimate:
+
+- Broad OpenCMIS client compatibility: `pre-compliance`, approximately
+  `0-10%` until session creation succeeds.
+- Harness maturity score from the live run: `9.52`.
+- Controlled-client usefulness of the native CMIS-inspired routes remains
+  materially higher, approximately `55-60%`, because known clients can use the
+  documented JSON routes and profile semantics.
+
+The previous `docs/cmis-1-1-capability-scorecard.md` estimate remains useful as
+a product-depth hypothesis, but it must not be presented as external CMIS
+compatibility evidence until OpenCMIS can connect and execute test cases.
+
+## Findings
+
+| ID | Severity | Finding | Evidence | Required response |
+| --- | --- | --- | --- | --- |
+| CMIS-TCK-20260508-001 | Critical | OpenCMIS cannot create a Browser Binding session because repository info is invalid for CMIS expectations. | ConsoleRunner: `Found invalid Repository Info!` | Implement CMIS-shaped service document/repository info and rerun repository/type group. |
+| CMIS-TCK-20260508-002 | High | The endpoint and TCK target disagree on repository ID. | Endpoint reports `kontextual-compat-tck`; TCK profile uses `compat-tck`. | Align repository ID or provide a stable alias before rerunning TCK. |
+| CMIS-TCK-20260508-003 | High | Browser Binding response fields use snake_case/native DTO shape instead of CMIS Browser Binding shape. | Preflight top-level keys and capability keys are snake_case. | Add CMIS protocol serializers that translate from native DTOs to CMIS Browser Binding JSON. |
+| CMIS-TCK-20260508-004 | Medium | `/openapi.json` generation fails for the content byte streaming route. | Focused internal CMIS suite failure in `test_cmis_browser_binding_routes_are_advertised_in_openapi`. | Fix FastAPI route metadata for `StreamingResponse`. |
+| CMIS-TCK-20260508-005 | Medium | Root `/children` contract is ambiguous between synthetic-folder discovery and direct document listing. | Focused internal CMIS suite failures for readonly and admin/export children expectations. | Decide CMIS root behavior and align implementation/tests with Browser Binding expectations. |
+| CMIS-TCK-20260508-006 | Medium | The TCK facility reports target protocol failures as generic `infrastructure_error`. | Normalized findings classify both TCK groups as infrastructure errors. | Improve classifier/preflight in `open-cmis-tck` so invalid repository info is reported as target protocol shape failure. |
+
+## Recommended Fix Sequence
+
+1. Define exact CMIS Browser Binding service-root/repository-info examples based
+   on OpenCMIS expectations.
+2. Implement a separate CMIS protocol serialization layer rather than exposing
+   native snake_case DTOs directly.
+3. Align `compat-tck` repository ID and target profile.
+4. Make preflight reject native-shaped repository info before invoking Maven.
+5. Fix OpenAPI generation for streaming routes.
+6. Resolve root navigation semantics and add explicit tests for root,
+   synthetic folders, and object-by-path behavior.
+7. Rerun only the repository/type group until OpenCMIS session creation passes.
+8. Then enable object/content TCK and work through semantic failures.
+9. Update the public scorecard only from TCK-backed evidence.
+
+## Boundary
+
+This assessment does not claim CMIS certification or full conformance. It is a
+local engineering assessment using Apache Chemistry OpenCMIS TCK as the first
+external compatibility oracle.