--- id: OPEN-CMIS-TCK-WP-0003 type: extension-workplan title: "Assessment Log Review And Hardening" repo: open-cmis-tck extension: open-cmis-tck domain: markitect status: completed owner: codex planning_priority: high planning_order: 4 created: "2026-05-14" updated: "2026-05-14" depends_on: - "OPEN-CMIS-TCK-WP-0002" state_hub_workstream_id: "5711ee2f-eaa9-428a-a4b2-e7383bfbf18a" --- # OPEN-CMIS-TCK-WP-0003: Assessment Log Review And Hardening ## Purpose Use the first real `kontextual-engine` OpenCMIS TCK assessment runs to harden the `open-cmis-tck` guide-board extension around warning policy, durable evidence retention, and repeatable log review. The latest `kontextual-engine` release-readiness run is healthy for the selected Browser Binding baseline: no hard TCK failures, no infrastructure errors, no unexpected findings, and empty stderr artifacts. The remaining work is mostly facility maturity: make the one current warning intentional, preserve raw evidence outside ephemeral `/tmp` paths, and give future assessments a compact "what should we fix next" report instead of relying on manual `rg` passes. ## Evidence Reviewed - Latest release-readiness evidence: `/home/worsch/kontextual-engine/docs/cmis-opencmis-tck-release-readiness-evidence-2026-05-13T223537Z.md` - Latest raw release-readiness run: `/tmp/kontextual-cmis-release-20260514-toolchain` - Prior raw run before `appendContentStream()` support: `/tmp/open-cmis-tck-kontextual-20260513T230205Z` - Earlier implementation evidence: `/home/worsch/kontextual-engine/docs/cmis-opencmis-tck-implementation-evidence-2026-05-08T092113Z.md` - Local extension self-test run: `/home/worsch/open-cmis-tck/.local/runs/opencmis-inmemory-pilot` - Local OpenCMIS in-memory server logs: `/home/worsch/open-cmis-tck/.local/opencmis-inmemory/logs` ## Current Findings 1. The latest `kontextual-engine` run completed with Guide Board summary `pass: 2`, `warning: 1`, `unexpected_findings: 0`. 2. `console-runner-stderr.txt` and `stderr.log` are empty for both selected TCK groups in the latest run. 3. The remaining current warning is from OpenCMIS `SecurityTest.java:67`: `HTTPS is not used. Credentials might be transferred as plain text!` 4. The previous `appendContentStream()` warning in `SetAndDeleteContentTest.java:200` is closed in the latest run. 5. The latest object/content run still has skipped cases for non-creatable relationship, policy, and item types, plus folder-name change-token subcases. These align with declared capability boundaries and are not errors, but they should remain visible as maturity scope. 6. The local OpenCMIS in-memory pilot has two repository/type warnings: loopback HTTP and `Thin client URI is not set!`. Tomcat and in-memory server logs did not show warning/error/exception lines in the scan. 7. Evidence retention is fragile: several useful raw runs live under `/tmp`, and at least one earlier `/tmp` run was already unavailable when later evidence was written. ## Boundary This workplan hardens the `open-cmis-tck` extension and its assessment operations. Product changes for `kontextual-engine` belong in that repository. This workplan may document product-facing follow-up candidates, but it should not modify the product repo directly. ## D3.1 - Capture Current Log Triage Baseline ```task id: OPEN-CMIS-TCK-WP-0003-T001 status: done priority: high state_hub_task_id: "1a262cad-a945-4a93-a957-02f2fdb497f1" ``` Acceptance: - Inspect the latest persisted evidence and raw run artifacts for `kontextual-engine`. - Separate current findings from older findings that have already been closed. - Inspect the local in-memory pilot logs so extension self-test warnings are not confused with product warnings. - Record the baseline in this workplan. Progress: - Confirmed the latest raw release-readiness run has no `fail`, `infrastructure_error`, unexpected finding, stderr output, or exception trace. - Confirmed the only current `kontextual-engine` TCK warning is local HTTP transport. - Confirmed `appendContentStream()` was a warning in the prior raw run and is gone in the latest raw run. - Confirmed the local in-memory pilot still reports loopback HTTP and missing thin-client URI warnings, while server logs are clean. ## D3.2 - Durable Assessment Archive Path ```task id: OPEN-CMIS-TCK-WP-0003-T002 status: done priority: high state_hub_task_id: "1e31b306-f21e-4bac-8e69-56d586d6712e" ``` Acceptance: - Provide a recommended non-ephemeral output layout for local product assessments, for example `.local/runs//` or a configured workspace archive path. - Add an operator command or documented copy/import step that preserves raw TCK stdout/stderr, normalized evidence, findings, mappings, run metadata, report, scorecard, and artifact manifest before `/tmp` cleanup can remove them. - Preserve artifact hashes or package metadata so copied evidence remains auditable. - Update the local runbook and service/retention notes with the durable path. Progress: - Added `src/open_cmis_tck/archive.py` and `scripts/archive_assessment_run.py`. - The archive command copies a run into `.local/runs/archive//` by default and writes `archive-manifest.json` with SHA-256 hashes, file sizes, source path, archive path, run ID, target profile reference, and assessment profile reference. - Updated README, local runbook, and service/retention docs with the archive command. - Archived the latest kontextual release-readiness run to `.local/runs/archive/kontextual-cmis-compat/run-20260513T223537Z`. ## D3.3 - Warning Policy And HTTPS Deployment Gate ```task id: OPEN-CMIS-TCK-WP-0003-T003 status: done priority: high state_hub_task_id: "58d2cf64-0db6-4f9a-a8d5-df9ef870557b" ``` Acceptance: - Define how warning policy distinguishes local loopback test topology from a deployment or release gate. - Treat the OpenCMIS HTTP warning as acceptable only for explicit local loopback profiles or documented local waivers. - Make non-loopback or release-target HTTP warnings visible as deployment blockers, even when the TCK group return code is `0`. - Record warning policy in a profile, expectation, or waiver file rather than burying it in narrative evidence. Progress: - Added `profiles/expectations/opencmis-warning-policy.json`. - Classified the OpenCMIS HTTP warning as accepted only for local/test loopback HTTP endpoints. - Non-loopback or production-like HTTP warnings are now classified as `deployment_transport_blocker` by the log-review command. ## D3.4 - OpenCMIS In-Memory Pilot Warning Cleanup ```task id: OPEN-CMIS-TCK-WP-0003-T004 status: done priority: medium state_hub_task_id: "12331abc-c28f-4140-9a04-a70eb761bccf" ``` Acceptance: - Investigate whether the local OpenCMIS in-memory server can expose a `thinClientURI` through configuration. - If the upstream in-memory server cannot be configured cleanly, mark the warning as an expected self-test limitation with a precise source location and explanation. - Keep the in-memory pilot useful as an extension smoke test without making its target-specific warnings look like guide-board defects. - Document the expected warning posture in the local runbook. Progress: - Added an explicit policy entry for the `opencmis-inmemory-local` `Thin client URI is not set!` warning. - The in-memory pilot review now classifies loopback HTTP and missing thin-client URI as accepted local self-test warnings. - Optional external server-log findings are reported as context without changing the run status by themselves, because those log directories may include historical startup attempts outside the assessed run. ## D3.5 - Automated Log Review Report ```task id: OPEN-CMIS-TCK-WP-0003-T005 status: done priority: high state_hub_task_id: "e445f909-678b-4236-a025-d5913e5473ed" ``` Acceptance: - Add a command that scans a guide-board run directory for OpenCMIS stdout, stderr, normalized results, findings, and known server logs. - Generate `reports/opencmis-log-review.json` and `reports/opencmis-log-review.md`. - Highlight hard errors, non-empty stderr, new warnings, known accepted warnings, skipped cases, unexpected findings, and closed-warning comparisons when a previous run is supplied. - Add regression tests with sanitized fixtures for the current HTTP warning, the now-closed `appendContentStream()` warning, empty stderr, and skipped capability-boundary cases. Progress: - Added `src/open_cmis_tck/log_review.py` and `scripts/opencmis_log_review.py`. - The command writes `reports/opencmis-log-review.json` and `reports/opencmis-log-review.md`. - Verified it against `/tmp/kontextual-cmis-release-20260514-toolchain` with `/tmp/open-cmis-tck-kontextual-20260513T230205Z` as the previous run. - Added regression coverage for accepted HTTP warnings, non-loopback deployment blockers, closed append warnings, stderr handling, and skipped capability boundaries. ## D3.6 - Skip And Capability Boundary Interpretation ```task id: OPEN-CMIS-TCK-WP-0003-T006 status: done priority: medium state_hub_task_id: "6df3e8c5-c2d2-4eb3-973e-76644ef7ee8f" ``` Acceptance: - Group skipped OpenCMIS cases by declared repository capability or type creatability boundary. - Distinguish "expected because the capability is not advertised" from "skipped because the target could not exercise an advertised capability." - Keep skipped cases visible in reports and maturity scorecards without treating them as failures when they match the advertised capability profile. - Add coverage for relationship, policy, item, type-subtype, and folder-name change-token skips seen in the latest raw run. Progress: - Added skip-boundary classification in the log-review report. - Current expected skip rules cover relationship, policy, item, document subtype, and folder-name mutation cases. - If the target advertises the required capability and OpenCMIS still skips the case, the review becomes `review_required`. ## D3.7 - Next Coverage Frontier ```task id: OPEN-CMIS-TCK-WP-0003-T007 status: done priority: medium state_hub_task_id: "f793e1b8-a2b1-4f9b-9972-b6b18d1ba56a" ``` Acceptance: - Identify which additional OpenCMIS TCK groups are realistic after the current repository/type and object/content baseline. - For each candidate group, record target preconditions, likely product capability requirements, and expected unsupported-by-design boundaries. - Do not expand the default baseline until the warning policy and durable evidence path are in place. - Produce a short recommendation for the next maturity slice, likely navigation, query, ACL/policy, versioning/PWC, or change-log depth. Progress: - Added the next-coverage recommendation to `docs/LOG-REVIEW.md`. - Recommended order is navigation/read-path depth first, metadata query second, ACL/policy discovery third, and versioning/PWC/change-log only after the product deliberately advertises those capabilities. - Left the default baseline unchanged at `repository-type` plus `object-content`. ## D3.8 - State Hub And Operator Docs ```task id: OPEN-CMIS-TCK-WP-0003-T008 status: done priority: medium state_hub_task_id: "32724628-d427-47c2-ac40-3f3f90e3a2b9" ``` Acceptance: - Sync this workplan with the state hub. - Update README/runbook references so operators know how to review warnings after a run. - Make it clear that guide-board produces preparation evidence and operational readiness signals, not certification or audit assurance. - Ensure doc updates cite the latest raw and persisted evidence baselines. Progress: - Added `docs/LOG-REVIEW.md`. - Updated README, local runbook, and service/retention docs. - Synced the completed workplan and task statuses into the state hub. ## Definition Of Done - Future local CMIS assessments keep raw evidence in a durable run location. - HTTP transport warnings are policy-classified rather than manually explained after every run. - The local in-memory pilot has either zero unexpected warnings or a documented expected-warning profile. - A log-review report can be generated from any guide-board run directory. - Skipped OpenCMIS cases are interpreted against advertised CMIS capability boundaries. - The next coverage frontier is explicit and does not blur preparation evidence with formal certification.