Log analysis functionality for self-assessment

2026-05-14 02:47:00 +02:00
parent 97a4a1fa37
commit cd43c7cfec
12 changed files with 1573 additions and 2 deletions
--- a/workplans/OPEN-CMIS-TCK-WP-0002-live-test-infrastructure.md
+++ b/workplans/OPEN-CMIS-TCK-WP-0002-live-test-infrastructure.md
@@ -5,12 +5,12 @@ title: "Live OpenCMIS TCK Execution And Capability Maturity"
 repo: open-cmis-tck
 extension: open-cmis-tck
 domain: markitect
-status: active
+status: completed
 owner: codex
 planning_priority: high
 planning_order: 3
 created: "2026-05-07"
-updated: "2026-05-08"
+updated: "2026-05-14"
 depends_on:
  - "OPEN-CMIS-TCK-WP-0001"
 state_hub_workstream_id: "da3f0d16-ba8e-4147-b0fc-ab3462e0b7b0"
--- a/workplans/OPEN-CMIS-TCK-WP-0003-assessment-log-review-and-hardening.md
+++ b/workplans/OPEN-CMIS-TCK-WP-0003-assessment-log-review-and-hardening.md
@@ -0,0 +1,326 @@
+---
+id: OPEN-CMIS-TCK-WP-0003
+type: extension-workplan
+title: "Assessment Log Review And Hardening"
+repo: open-cmis-tck
+extension: open-cmis-tck
+domain: markitect
+status: completed
+owner: codex
+planning_priority: high
+planning_order: 4
+created: "2026-05-14"
+updated: "2026-05-14"
+depends_on:
+  - "OPEN-CMIS-TCK-WP-0002"
+state_hub_workstream_id: "5711ee2f-eaa9-428a-a4b2-e7383bfbf18a"
+---
+
+# OPEN-CMIS-TCK-WP-0003: Assessment Log Review And Hardening
+
+## Purpose
+
+Use the first real `kontextual-engine` OpenCMIS TCK assessment runs to harden
+the `open-cmis-tck` guide-board extension around warning policy, durable
+evidence retention, and repeatable log review.
+
+The latest `kontextual-engine` release-readiness run is healthy for the selected
+Browser Binding baseline: no hard TCK failures, no infrastructure errors, no
+unexpected findings, and empty stderr artifacts. The remaining work is mostly
+facility maturity: make the one current warning intentional, preserve raw
+evidence outside ephemeral `/tmp` paths, and give future assessments a compact
+"what should we fix next" report instead of relying on manual `rg` passes.
+
+## Evidence Reviewed
+
+- Latest release-readiness evidence:
+  `/home/worsch/kontextual-engine/docs/cmis-opencmis-tck-release-readiness-evidence-2026-05-13T223537Z.md`
+- Latest raw release-readiness run:
+  `/tmp/kontextual-cmis-release-20260514-toolchain`
+- Prior raw run before `appendContentStream()` support:
+  `/tmp/open-cmis-tck-kontextual-20260513T230205Z`
+- Earlier implementation evidence:
+  `/home/worsch/kontextual-engine/docs/cmis-opencmis-tck-implementation-evidence-2026-05-08T092113Z.md`
+- Local extension self-test run:
+  `/home/worsch/open-cmis-tck/.local/runs/opencmis-inmemory-pilot`
+- Local OpenCMIS in-memory server logs:
+  `/home/worsch/open-cmis-tck/.local/opencmis-inmemory/logs`
+
+## Current Findings
+
+1. The latest `kontextual-engine` run completed with Guide Board summary
+   `pass: 2`, `warning: 1`, `unexpected_findings: 0`.
+2. `console-runner-stderr.txt` and `stderr.log` are empty for both selected TCK
+   groups in the latest run.
+3. The remaining current warning is from OpenCMIS
+   `SecurityTest.java:67`: `HTTPS is not used. Credentials might be transferred
+   as plain text!`
+4. The previous `appendContentStream()` warning in
+   `SetAndDeleteContentTest.java:200` is closed in the latest run.
+5. The latest object/content run still has skipped cases for non-creatable
+   relationship, policy, and item types, plus folder-name change-token subcases.
+   These align with declared capability boundaries and are not errors, but they
+   should remain visible as maturity scope.
+6. The local OpenCMIS in-memory pilot has two repository/type warnings:
+   loopback HTTP and `Thin client URI is not set!`. Tomcat and in-memory server
+   logs did not show warning/error/exception lines in the scan.
+7. Evidence retention is fragile: several useful raw runs live under `/tmp`, and
+   at least one earlier `/tmp` run was already unavailable when later evidence
+   was written.
+
+## Boundary
+
+This workplan hardens the `open-cmis-tck` extension and its assessment
+operations. Product changes for `kontextual-engine` belong in that repository.
+This workplan may document product-facing follow-up candidates, but it should
+not modify the product repo directly.
+
+## D3.1 - Capture Current Log Triage Baseline
+
+```task
+id: OPEN-CMIS-TCK-WP-0003-T001
+status: done
+priority: high
+state_hub_task_id: "1a262cad-a945-4a93-a957-02f2fdb497f1"
+```
+
+Acceptance:
+
+- Inspect the latest persisted evidence and raw run artifacts for
+  `kontextual-engine`.
+- Separate current findings from older findings that have already been closed.
+- Inspect the local in-memory pilot logs so extension self-test warnings are not
+  confused with product warnings.
+- Record the baseline in this workplan.
+
+Progress:
+
+- Confirmed the latest raw release-readiness run has no `fail`,
+  `infrastructure_error`, unexpected finding, stderr output, or exception trace.
+- Confirmed the only current `kontextual-engine` TCK warning is local HTTP
+  transport.
+- Confirmed `appendContentStream()` was a warning in the prior raw run and is
+  gone in the latest raw run.
+- Confirmed the local in-memory pilot still reports loopback HTTP and missing
+  thin-client URI warnings, while server logs are clean.
+
+## D3.2 - Durable Assessment Archive Path
+
+```task
+id: OPEN-CMIS-TCK-WP-0003-T002
+status: done
+priority: high
+state_hub_task_id: "1e31b306-f21e-4bac-8e69-56d586d6712e"
+```
+
+Acceptance:
+
+- Provide a recommended non-ephemeral output layout for local product
+  assessments, for example `.local/runs/<target>/<run-id>` or a configured
+  workspace archive path.
+- Add an operator command or documented copy/import step that preserves raw
+  TCK stdout/stderr, normalized evidence, findings, mappings, run metadata,
+  report, scorecard, and artifact manifest before `/tmp` cleanup can remove
+  them.
+- Preserve artifact hashes or package metadata so copied evidence remains
+  auditable.
+- Update the local runbook and service/retention notes with the durable path.
+
+Progress:
+
+- Added `src/open_cmis_tck/archive.py` and
+  `scripts/archive_assessment_run.py`.
+- The archive command copies a run into `.local/runs/archive/<target>/<run-id>`
+  by default and writes `archive-manifest.json` with SHA-256 hashes, file
+  sizes, source path, archive path, run ID, target profile reference, and
+  assessment profile reference.
+- Updated README, local runbook, and service/retention docs with the archive
+  command.
+- Archived the latest kontextual release-readiness run to
+  `.local/runs/archive/kontextual-cmis-compat/run-20260513T223537Z`.
+
+## D3.3 - Warning Policy And HTTPS Deployment Gate
+
+```task
+id: OPEN-CMIS-TCK-WP-0003-T003
+status: done
+priority: high
+state_hub_task_id: "58d2cf64-0db6-4f9a-a8d5-df9ef870557b"
+```
+
+Acceptance:
+
+- Define how warning policy distinguishes local loopback test topology from a
+  deployment or release gate.
+- Treat the OpenCMIS HTTP warning as acceptable only for explicit local
+  loopback profiles or documented local waivers.
+- Make non-loopback or release-target HTTP warnings visible as deployment
+  blockers, even when the TCK group return code is `0`.
+- Record warning policy in a profile, expectation, or waiver file rather than
+  burying it in narrative evidence.
+
+Progress:
+
+- Added `profiles/expectations/opencmis-warning-policy.json`.
+- Classified the OpenCMIS HTTP warning as accepted only for local/test loopback
+  HTTP endpoints.
+- Non-loopback or production-like HTTP warnings are now classified as
+  `deployment_transport_blocker` by the log-review command.
+
+## D3.4 - OpenCMIS In-Memory Pilot Warning Cleanup
+
+```task
+id: OPEN-CMIS-TCK-WP-0003-T004
+status: done
+priority: medium
+state_hub_task_id: "12331abc-c28f-4140-9a04-a70eb761bccf"
+```
+
+Acceptance:
+
+- Investigate whether the local OpenCMIS in-memory server can expose a
+  `thinClientURI` through configuration.
+- If the upstream in-memory server cannot be configured cleanly, mark the
+  warning as an expected self-test limitation with a precise source location and
+  explanation.
+- Keep the in-memory pilot useful as an extension smoke test without making its
+  target-specific warnings look like guide-board defects.
+- Document the expected warning posture in the local runbook.
+
+Progress:
+
+- Added an explicit policy entry for the `opencmis-inmemory-local`
+  `Thin client URI is not set!` warning.
+- The in-memory pilot review now classifies loopback HTTP and missing
+  thin-client URI as accepted local self-test warnings.
+- Optional external server-log findings are reported as context without
+  changing the run status by themselves, because those log directories may
+  include historical startup attempts outside the assessed run.
+
+## D3.5 - Automated Log Review Report
+
+```task
+id: OPEN-CMIS-TCK-WP-0003-T005
+status: done
+priority: high
+state_hub_task_id: "e445f909-678b-4236-a025-d5913e5473ed"
+```
+
+Acceptance:
+
+- Add a command that scans a guide-board run directory for OpenCMIS stdout,
+  stderr, normalized results, findings, and known server logs.
+- Generate `reports/opencmis-log-review.json` and
+  `reports/opencmis-log-review.md`.
+- Highlight hard errors, non-empty stderr, new warnings, known accepted
+  warnings, skipped cases, unexpected findings, and closed-warning comparisons
+  when a previous run is supplied.
+- Add regression tests with sanitized fixtures for the current HTTP warning,
+  the now-closed `appendContentStream()` warning, empty stderr, and skipped
+  capability-boundary cases.
+
+Progress:
+
+- Added `src/open_cmis_tck/log_review.py` and
+  `scripts/opencmis_log_review.py`.
+- The command writes `reports/opencmis-log-review.json` and
+  `reports/opencmis-log-review.md`.
+- Verified it against `/tmp/kontextual-cmis-release-20260514-toolchain` with
+  `/tmp/open-cmis-tck-kontextual-20260513T230205Z` as the previous run.
+- Added regression coverage for accepted HTTP warnings, non-loopback deployment
+  blockers, closed append warnings, stderr handling, and skipped capability
+  boundaries.
+
+## D3.6 - Skip And Capability Boundary Interpretation
+
+```task
+id: OPEN-CMIS-TCK-WP-0003-T006
+status: done
+priority: medium
+state_hub_task_id: "6df3e8c5-c2d2-4eb3-973e-76644ef7ee8f"
+```
+
+Acceptance:
+
+- Group skipped OpenCMIS cases by declared repository capability or type
+  creatability boundary.
+- Distinguish "expected because the capability is not advertised" from "skipped
+  because the target could not exercise an advertised capability."
+- Keep skipped cases visible in reports and maturity scorecards without treating
+  them as failures when they match the advertised capability profile.
+- Add coverage for relationship, policy, item, type-subtype, and folder-name
+  change-token skips seen in the latest raw run.
+
+Progress:
+
+- Added skip-boundary classification in the log-review report.
+- Current expected skip rules cover relationship, policy, item, document
+  subtype, and folder-name mutation cases.
+- If the target advertises the required capability and OpenCMIS still skips the
+  case, the review becomes `review_required`.
+
+## D3.7 - Next Coverage Frontier
+
+```task
+id: OPEN-CMIS-TCK-WP-0003-T007
+status: done
+priority: medium
+state_hub_task_id: "f793e1b8-a2b1-4f9b-9972-b6b18d1ba56a"
+```
+
+Acceptance:
+
+- Identify which additional OpenCMIS TCK groups are realistic after the current
+  repository/type and object/content baseline.
+- For each candidate group, record target preconditions, likely product
+  capability requirements, and expected unsupported-by-design boundaries.
+- Do not expand the default baseline until the warning policy and durable
+  evidence path are in place.
+- Produce a short recommendation for the next maturity slice, likely navigation,
+  query, ACL/policy, versioning/PWC, or change-log depth.
+
+Progress:
+
+- Added the next-coverage recommendation to `docs/LOG-REVIEW.md`.
+- Recommended order is navigation/read-path depth first, metadata query second,
+  ACL/policy discovery third, and versioning/PWC/change-log only after the
+  product deliberately advertises those capabilities.
+- Left the default baseline unchanged at `repository-type` plus
+  `object-content`.
+
+## D3.8 - State Hub And Operator Docs
+
+```task
+id: OPEN-CMIS-TCK-WP-0003-T008
+status: done
+priority: medium
+state_hub_task_id: "32724628-d427-47c2-ac40-3f3f90e3a2b9"
+```
+
+Acceptance:
+
+- Sync this workplan with the state hub.
+- Update README/runbook references so operators know how to review warnings
+  after a run.
+- Make it clear that guide-board produces preparation evidence and operational
+  readiness signals, not certification or audit assurance.
+- Ensure doc updates cite the latest raw and persisted evidence baselines.
+
+Progress:
+
+- Added `docs/LOG-REVIEW.md`.
+- Updated README, local runbook, and service/retention docs.
+- Synced the completed workplan and task statuses into the state hub.
+
+## Definition Of Done
+
+- Future local CMIS assessments keep raw evidence in a durable run location.
+- HTTP transport warnings are policy-classified rather than manually explained
+  after every run.
+- The local in-memory pilot has either zero unexpected warnings or a documented
+  expected-warning profile.
+- A log-review report can be generated from any guide-board run directory.
+- Skipped OpenCMIS cases are interpreted against advertised CMIS capability
+  boundaries.
+- The next coverage frontier is explicit and does not blur preparation evidence
+  with formal certification.