coulomb/kontextual-engine
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5c450fcaa5d7de8a184ccd760f0353a0715cb76f
kontextual-engine/docs/cmis-opencmis-tck-assessment-2026-05-08T063312Z.md

11 KiB
Raw Blame History

CMIS OpenCMIS TCK Assessment

Run timestamp: 2026-05-08T06:33:12Z Local timestamp: 2026-05-08T08:33:12+02:00 Status: evidence-backed compatibility blocker found

Superseded implementation evidence: the blocker described here was resolved by KONT-WP-0013. See docs/cmis-opencmis-tck-implementation-evidence-2026-05-08T092113Z.md for the latest completed OpenCMIS run.

Purpose

Persist the first real guide-board + open-cmis-tck assessment against a running kontextual-engine CMIS access point. This document supersedes prior pre-TCK estimates for broad commodity CMIS client compatibility.

The assessment distinguishes two separate facts:

  • The native Kontextual CMIS-inspired route layer exposes useful document, content-stream, query, ACL, relationship, parent, and profile behavior.
  • The current endpoint is not yet consumable as a standard CMIS 1.1 Browser Binding endpoint by Apache Chemistry OpenCMIS.

Scope

Target repository: kontextual-engine

External test facility:

  • guide-board
  • open-cmis-tck
  • Apache Chemistry OpenCMIS TCK 1.1.0
  • Browser Binding profile: cmis-browser-baseline

Runtime target used for this run:

  • URL: http://127.0.0.1:8010/cmis/compat-tck/browser
  • Access point: compat-tck
  • Reported repository ID: kontextual-compat-tck
  • TCK profile repository ID: compat-tck

The production profile file currently points to port 8000; this run used a temporary copy redirected to port 8010 because port 8000 was already owned by another local service that returned 404 for the CMIS probes.

Commands Run

Validate the open-cmis-tck guide-board extension:

cd /home/worsch/guide-board
PYTHONPATH=src python3 -m guide_board --extension-dir ../open-cmis-tck extensions validate

Run the live OpenCMIS baseline against the temporary target profile:

cd /home/worsch/guide-board
source /home/worsch/open-cmis-tck/.local/toolchains/env.sh
PYTHONPATH=src python3 -m guide_board \
  --extension-dir ../open-cmis-tck \
  run \
  --target /tmp/kontextual-cmis-compat-8010.json \
  --assessment ../open-cmis-tck/profiles/assessments/cmis-browser-baseline.json \
  --output-dir /tmp/open-cmis-tck-kontextual-live-20260508

Generate the scorecard:

cd /home/worsch/open-cmis-tck
PYTHONPATH=src python3 scripts/cmis_scorecard.py \
  --run-dir /tmp/open-cmis-tck-kontextual-live-20260508

Run focused internal CMIS tests with service dependencies available:

cd /home/worsch/kontextual-engine
.venv/bin/python -m pytest tests/cmis --perf-history-disable

External TCK Result

Guide Board live run:

  • Run ID: run-20260508T063312Z
  • Assessment status: infrastructure_error
  • Summary: pass: 1, infrastructure_error: 2
  • Preflight: pass; endpoint was reachable and returned parseable JSON.
  • Repository/type TCK group: blocked before test cases could be parsed.
  • Object/content TCK group: blocked before test cases could be parsed.

Generated scorecard:

  • Maturity score: 9.52
  • Coverage: 2/9 groups, 22.22%
  • Repository and type metadata: blocked, score 1/4
  • Object and content services: blocked, score 1/4
  • Navigation, query, relationships, ACL/policy, versioning, change log, and extensions: not assessed.

Primary blocker from OpenCMIS ConsoleRunner:

Found invalid Repository Info!

The same failure occurs in:

  • BasicsTestGroup and TypesTestGroup
  • CRUDTestGroup

No OpenCMIS semantic cases were parsed because the OpenCMIS client could not establish a valid CMIS Browser Binding session.

Evidence Artifacts

Run artifacts were generated under /tmp and may be ephemeral. The critical facts are copied into this document.

  • /tmp/open-cmis-tck-kontextual-live-20260508/reports/report.md
  • /tmp/open-cmis-tck-kontextual-live-20260508/reports/cmis-maturity-scorecard.md
  • /tmp/open-cmis-tck-kontextual-live-20260508/normalized/evidence.json
  • /tmp/open-cmis-tck-kontextual-live-20260508/normalized/findings.json
  • /tmp/open-cmis-tck-kontextual-live-20260508/artifacts/open-cmis-tck/preflight/response-body.bin
  • /tmp/open-cmis-tck-kontextual-live-20260508/artifacts/open-cmis-tck/tck/repository-type/console-runner-stdout.txt
  • /tmp/open-cmis-tck-kontextual-live-20260508/artifacts/open-cmis-tck/tck/object-content/console-runner-stdout.txt

Repository Info Shape Observed

Preflight observed a JSON object with these top-level keys:

binding
capabilities
cmis_version_supported
compliance
principal_anonymous
principal_anyone
product_name
product_version
profile
repository_description
repository_features
repository_id
repository_name
root_folder_id
unsupported_features
vendor_name

The endpoint reports:

  • repository_id: kontextual-compat-tck
  • repository_name: Kontextual Engine compat-tck
  • cmis_version_supported: 1.1
  • root_folder_id: cmis-root
  • binding: browser

The capability map also uses snake_case keys, for example:

  • capability_query
  • capability_acl
  • capability_content_stream_updatability
  • capability_multifiling

OpenCMIS expects CMIS Browser Binding repository information/service document shape, including CMIS field names such as:

  • repositoryId
  • repositoryName
  • cmisVersionSupported
  • rootFolderId
  • Browser Binding capability names in the shape expected by the OpenCMIS client

The assessment therefore classifies the immediate blocker as protocol contract shape, not absence of all underlying engine capabilities.

Internal CMIS Test Result

Focused internal test run:

  • Command: .venv/bin/python -m pytest tests/cmis --perf-history-disable
  • Result: 42 passed, 3 failed

Passing areas include:

  • access profile modeling
  • compliance flags
  • contract examples
  • domain mapper
  • fixture integration
  • runtime Browser Binding behavior

Failures:

  1. OpenAPI generation fails around the streaming route return annotation. Pydantic reports that the StreamingResponse forward reference is not fully defined when generating /openapi.json.
  2. test_cmis_readonly_children_object_content_query_relationships_and_changes expects root /children to include documents directly. Current behavior returns first-level projected folders at root.
  3. test_cmis_profile_gates_visibility_by_access_point has the same root /children expectation for admin/export visibility.

The navigation failures are contract alignment issues: either root children should expose documents directly for the CMIS profile, or the tests should ask for the folder that contains those projected documents. The OpenAPI failure is a concrete implementation bug.

Native Route Probe Result

Direct live probes confirmed useful native behavior:

  • /cmis lists multiple profiled access points.
  • /cmis/compat-tck/browser returns repository metadata and conservative unsupported-feature diagnostics.
  • /cmis/compat-tck/browser/types returns six type projections.
  • /cmis/compat-tck/browser/children returns synthetic root folders.
  • /cmis/compat-tck/browser/query?q=SELECT * FROM cmis:document returns document projections.
  • /cmis/compat-tck/browser/object/{object_id} returns object properties, content stream metadata, allowable actions, version projection, and ACL projection.
  • /cmis/compat-tck/browser/content-bytes/{object_id} returns real text/markdown bytes.
  • /cmis/compat-tck/browser/parents/{object_id} returns projection-only parent folders.
  • Unsupported query/property-update cases return structured diagnostics.

This confirms that content-stream and object-projection foundations exist. The OpenCMIS blocker is the external CMIS Browser Binding adapter contract.

Capability Interpretation

Evidence-backed commodity CMIS compatibility is currently lower than the pre-TCK estimate:

  • Broad OpenCMIS client compatibility: pre-compliance, approximately 0-10% until session creation succeeds.
  • Harness maturity score from the live run: 9.52.
  • Controlled-client usefulness of the native CMIS-inspired routes remains materially higher, approximately 55-60%, because known clients can use the documented JSON routes and profile semantics.

The previous docs/cmis-1-1-capability-scorecard.md estimate remains useful as a product-depth hypothesis, but it must not be presented as external CMIS compatibility evidence until OpenCMIS can connect and execute test cases.

Findings

ID Severity Finding Evidence Required response
CMIS-TCK-20260508-001 Critical OpenCMIS cannot create a Browser Binding session because repository info is invalid for CMIS expectations. ConsoleRunner: Found invalid Repository Info! Implement CMIS-shaped service document/repository info and rerun repository/type group.
CMIS-TCK-20260508-002 High The endpoint and TCK target disagree on repository ID. Endpoint reports kontextual-compat-tck; TCK profile uses compat-tck. Align repository ID or provide a stable alias before rerunning TCK.
CMIS-TCK-20260508-003 High Browser Binding response fields use snake_case/native DTO shape instead of CMIS Browser Binding shape. Preflight top-level keys and capability keys are snake_case. Add CMIS protocol serializers that translate from native DTOs to CMIS Browser Binding JSON.
CMIS-TCK-20260508-004 Medium /openapi.json generation fails for the content byte streaming route. Focused internal CMIS suite failure in test_cmis_browser_binding_routes_are_advertised_in_openapi. Fix FastAPI route metadata for StreamingResponse.
CMIS-TCK-20260508-005 Medium Root /children contract is ambiguous between synthetic-folder discovery and direct document listing. Focused internal CMIS suite failures for readonly and admin/export children expectations. Decide CMIS root behavior and align implementation/tests with Browser Binding expectations.
CMIS-TCK-20260508-006 Medium The TCK facility reports target protocol failures as generic infrastructure_error. Normalized findings classify both TCK groups as infrastructure errors. Improve classifier/preflight in open-cmis-tck so invalid repository info is reported as target protocol shape failure.

Recommended Fix Sequence

  1. Define exact CMIS Browser Binding service-root/repository-info examples based on OpenCMIS expectations.
  2. Implement a separate CMIS protocol serialization layer rather than exposing native snake_case DTOs directly.
  3. Align compat-tck repository ID and target profile.
  4. Make preflight reject native-shaped repository info before invoking Maven.
  5. Fix OpenAPI generation for streaming routes.
  6. Resolve root navigation semantics and add explicit tests for root, synthetic folders, and object-by-path behavior.
  7. Rerun only the repository/type group until OpenCMIS session creation passes.
  8. Then enable object/content TCK and work through semantic failures.
  9. Update the public scorecard only from TCK-backed evidence.

Boundary

This assessment does not claim CMIS certification or full conformance. It is a local engineering assessment using Apache Chemistry OpenCMIS TCK as the first external compatibility oracle.

Reference in New Issue View Git Blame Copy Permalink