diff --git a/README.md b/README.md index ffbf462..b0aaf0e 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,18 @@ If the workstation does not have Java and Maven in WSL, see [docs/LOCAL-RUNBOOK.md](docs/LOCAL-RUNBOOK.md) for the repo-local toolchain installer path. +After a run, generate the OpenCMIS warning/stderr/skip review and archive any +important product evidence outside `/tmp`: + +```sh +cd ../open-cmis-tck +PYTHONPATH=src python3 scripts/opencmis_log_review.py \ + --run-dir /tmp/open-cmis-tck-baseline +PYTHONPATH=src python3 scripts/archive_assessment_run.py \ + --run-dir /tmp/open-cmis-tck-baseline \ + --archive-root .local/runs/archive +``` + ## Tests Run extension tests with the guide-board core on `PYTHONPATH`: @@ -81,6 +93,7 @@ PYTHONPATH=../guide-board/src python3 -m unittest discover -s tests - [docs/CONTAINER-HANDOFF.md](docs/CONTAINER-HANDOFF.md) - [docs/LOCAL-RUNBOOK.md](docs/LOCAL-RUNBOOK.md) - [docs/LOCAL-TCK-RUNTIME.md](docs/LOCAL-TCK-RUNTIME.md) +- [docs/LOG-REVIEW.md](docs/LOG-REVIEW.md) - [docs/OPENCMIS-TCK-RUNNER.md](docs/OPENCMIS-TCK-RUNNER.md) - [docs/SERVICE-AND-RETENTION.md](docs/SERVICE-AND-RETENTION.md) diff --git a/docs/LOCAL-RUNBOOK.md b/docs/LOCAL-RUNBOOK.md index cf85248..b370a97 100644 --- a/docs/LOCAL-RUNBOOK.md +++ b/docs/LOCAL-RUNBOOK.md @@ -193,12 +193,35 @@ PYTHONPATH=src python3 scripts/cmis_scorecard.py \ --run-dir /tmp/open-cmis-tck-live ``` +Generate the log review after the scorecard: + +```sh +PYTHONPATH=src python3 scripts/opencmis_log_review.py \ + --run-dir /tmp/open-cmis-tck-live +``` + +The review classifies warnings, hard errors, stderr, skipped cases, and closed +warnings when a previous run is supplied. The default warning policy accepts the +OpenCMIS HTTP transport warning only for local/test loopback profiles. + +For important product assessments, archive the run before `/tmp` cleanup: + +```sh +PYTHONPATH=src python3 scripts/archive_assessment_run.py \ + --run-dir /tmp/open-cmis-tck-live \ + --archive-root .local/runs/archive +``` + +The archive writes `.local/runs/archive///archive-manifest.json` +with SHA-256 hashes for every copied file. + Review the real TCK evidence before expanding the scope: ```text /tmp/open-cmis-tck-live/normalized/evidence.json /tmp/open-cmis-tck-live/artifacts/open-cmis-tck/tck//console-runner-stdout.txt /tmp/open-cmis-tck-live/artifacts/open-cmis-tck/tck//normalized-runner-result.json +/tmp/open-cmis-tck-live/reports/opencmis-log-review.md ``` The normalizer preserves native OpenCMIS statuses (`OK`, `WARNING`, `FAILURE`, diff --git a/docs/LOG-REVIEW.md b/docs/LOG-REVIEW.md new file mode 100644 index 0000000..81634dc --- /dev/null +++ b/docs/LOG-REVIEW.md @@ -0,0 +1,132 @@ +# OpenCMIS Log Review + +Status: draft +Created: 2026-05-14 + +## Purpose + +The log review command turns a guide-board OpenCMIS run directory into a compact +fix-or-acceptance report. It reads normalized OpenCMIS cases, findings, +stderr artifacts, optional server logs, and the warning policy profile. + +It answers four operator questions: + +- Did any hard OpenCMIS case fail? +- Did the runner or server write unexpected stderr/log errors? +- Are warnings accepted local-test conditions or release blockers? +- Do skipped cases match the target's advertised capability boundary? + +## Command + +Generate a review after a run: + +```sh +cd /home/worsch/open-cmis-tck +PYTHONPATH=src python3 scripts/opencmis_log_review.py \ + --run-dir /tmp/open-cmis-tck-live +``` + +The command writes: + +```text +/tmp/open-cmis-tck-live/reports/opencmis-log-review.json +/tmp/open-cmis-tck-live/reports/opencmis-log-review.md +``` + +Compare with a previous run to surface closed and new warnings: + +```sh +PYTHONPATH=src python3 scripts/opencmis_log_review.py \ + --run-dir /tmp/kontextual-cmis-release-20260514-toolchain \ + --previous-run-dir /tmp/open-cmis-tck-kontextual-20260513T230205Z +``` + +Include known server logs when available: + +```sh +PYTHONPATH=src python3 scripts/opencmis_log_review.py \ + --run-dir .local/runs/opencmis-inmemory-pilot \ + --server-log .local/opencmis-inmemory/logs \ + --server-log .local/opencmis-inmemory/containers/apache-tomcat-9.0.117/logs +``` + +Server-log matches are reported as review context. They do not change the +overall review status by themselves because external log directories often +contain historical startup attempts from before the assessed run. + +## Warning Policy + +The default policy is: + +```text +profiles/expectations/opencmis-warning-policy.json +``` + +It currently classifies: + +- `HTTPS is not used...` as acceptable only for explicit local/test loopback + HTTP endpoints. The same warning becomes a deployment transport blocker for + non-loopback or production-like targets. +- `Thin client URI is not set!` as an accepted limitation only for the + `opencmis-inmemory-local` self-test target. + +Policies intentionally live outside narrative evidence. A release run should +show whether each warning is accepted by policy, not rely on a human remembering +the rule. + +## Skip Interpretation + +Skipped OpenCMIS cases are grouped by capability boundary. A skip is treated as +expected when the corresponding capability is not advertised by the target +profile. For example: + +- non-creatable `cmis:relationship` -> requires `cmis.relationships` +- non-creatable `cmis:policy` -> requires `cmis.policy-mutation` +- non-creatable `cmis:item` -> requires `cmis.item-services` +- document sub-type mutation -> requires `cmis.type-mutability` +- folder-name mutation -> requires `cmis.folder-name-mutation` + +If a target advertises the relevant capability and the TCK still skips the case, +the review status becomes `review_required`. + +## Durable Archive + +Do not leave important product assessment evidence only under `/tmp`. Archive a +run after scorecard/log-review generation: + +```sh +cd /home/worsch/open-cmis-tck +PYTHONPATH=src python3 scripts/archive_assessment_run.py \ + --run-dir /tmp/kontextual-cmis-release-20260514-toolchain \ + --archive-root .local/runs/archive +``` + +The archive command copies the run directory and writes: + +```text +.local/runs/archive///archive-manifest.json +``` + +The manifest records the source path, archive path, run metadata, file count, +total bytes, and SHA-256 for every copied file. + +## Next Coverage Frontier + +Keep the default baseline at `repository-type` plus `object-content` until +warning policy and durable evidence are routine. The next recommended maturity +slice is navigation/read-path depth, because it sits closest to current +object/content behavior and reveals path, parent, children, and filing semantics +without immediately requiring versioning or full query depth. + +Candidate order: + +1. Navigation/read-path checks with unsupported filing mutations clearly scoped. +2. Metadata query checks for the advertised `metadataonly` query posture. +3. ACL/policy discovery depth before ACL/policy mutation. +4. Versioning/PWC and change-log depth only after the product deliberately + advertises those capabilities. + +## Boundary + +This log review supports preparation evidence and operational readiness. It +does not issue CMIS certification or audit assurance. diff --git a/docs/SERVICE-AND-RETENTION.md b/docs/SERVICE-AND-RETENTION.md index e6d2746..83840b2 100644 --- a/docs/SERVICE-AND-RETENTION.md +++ b/docs/SERVICE-AND-RETENTION.md @@ -39,6 +39,13 @@ CMIS runs use the guide-board run directory contract. Each run writes: - `reports/assessment-package.json` - `reports/report.md` +Extension-side post-processing can also add: + +- `reports/cmis-maturity-scorecard.json` +- `reports/cmis-maturity-scorecard.md` +- `reports/opencmis-log-review.json` +- `reports/opencmis-log-review.md` + The sample assessment profile keeps summaries for 365 days and raw artifacts for 30 days: @@ -54,3 +61,23 @@ The sample assessment profile keeps summaries for 365 days and raw artifacts for Compact `retention-summary.json` files are suitable for guide-board trend summaries and downstream CMIS capability scorecards without retaining unbounded raw TCK logs. + +## Durable Local Archive + +For product-facing assessment evidence, do not rely on `/tmp` as the only copy. +Archive completed runs after scorecard and log-review generation: + +```sh +cd /home/worsch/open-cmis-tck +PYTHONPATH=src python3 scripts/archive_assessment_run.py \ + --run-dir /tmp/open-cmis-tck-live \ + --archive-root .local/runs/archive +``` + +The archive command copies the full run directory and writes +`archive-manifest.json` with SHA-256 hashes, file sizes, source path, archive +path, run ID, target profile reference, and assessment profile reference. + +The default local archive root remains under `.local/`, so it is not committed. +Move selected archive packages into a controlled evidence store when the run is +used for release or external audit preparation. diff --git a/profiles/expectations/opencmis-warning-policy.json b/profiles/expectations/opencmis-warning-policy.json new file mode 100644 index 0000000..63725b8 --- /dev/null +++ b/profiles/expectations/opencmis-warning-policy.json @@ -0,0 +1,54 @@ +{ + "id": "opencmis-warning-policy", + "description": "Warning classification policy for local OpenCMIS TCK preparation runs.", + "warning_policies": [ + { + "id": "local-loopback-http-transport", + "match": { + "message_contains": "HTTPS is not used", + "source_location": { + "file": "SecurityTest.java" + } + }, + "accepted_when": { + "scheme": "http", + "host_scope": "loopback", + "environments": [ + "local", + "test", + "development" + ] + }, + "classification": "accepted_local_loopback_transport", + "severity": "info", + "reason": "OpenCMIS warns about HTTP credentials. This is acceptable for explicit loopback-only local runs and must not be used as a deployment-release claim.", + "unaccepted_classification": "deployment_transport_blocker", + "unaccepted_severity": "blocker", + "unaccepted_reason": "OpenCMIS reported plain HTTP outside the accepted local loopback boundary. Use HTTPS termination or an explicit approved waiver for deployment-like targets." + }, + { + "id": "opencmis-inmemory-thin-client-uri", + "match": { + "message_contains": "Thin client URI is not set", + "source_location": { + "file": "RepositoryInfoTest.java" + } + }, + "accepted_when": { + "target_profile_refs": [ + "opencmis-inmemory-local" + ], + "environments": [ + "local", + "test" + ] + }, + "classification": "accepted_inmemory_self_test_limitation", + "severity": "info", + "reason": "The Apache Chemistry in-memory server is a local extension smoke target. Missing thinClientURI is a target-specific self-test limitation, not a guide-board extension defect.", + "unaccepted_classification": "repository_info_warning", + "unaccepted_severity": "warning", + "unaccepted_reason": "A non-in-memory target should expose or intentionally document its thin client URI behavior." + } + ] +} diff --git a/scripts/archive_assessment_run.py b/scripts/archive_assessment_run.py new file mode 100644 index 0000000..75efd93 --- /dev/null +++ b/scripts/archive_assessment_run.py @@ -0,0 +1,35 @@ +#!/usr/bin/env python3 +"""Archive a guide-board OpenCMIS assessment run with a hash manifest.""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from open_cmis_tck.archive import archive_run # noqa: E402 + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--run-dir", type=Path, required=True) + parser.add_argument("--archive-root", type=Path, default=Path(".local/runs/archive")) + parser.add_argument("--target-id") + parser.add_argument("--archive-name") + args = parser.parse_args() + + manifest = archive_run( + args.run_dir, + args.archive_root, + target_id=args.target_id, + archive_name=args.archive_name, + ) + print(json.dumps(manifest, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/opencmis_log_review.py b/scripts/opencmis_log_review.py new file mode 100644 index 0000000..0b56a75 --- /dev/null +++ b/scripts/opencmis_log_review.py @@ -0,0 +1,37 @@ +#!/usr/bin/env python3 +"""Generate OpenCMIS TCK warning, stderr, and skip review reports.""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) + +from open_cmis_tck.log_review import write_log_review # noqa: E402 + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--run-dir", type=Path, required=True) + parser.add_argument("--output-dir", type=Path) + parser.add_argument("--policy", type=Path) + parser.add_argument("--previous-run-dir", type=Path) + parser.add_argument("--server-log", type=Path, action="append", default=[]) + args = parser.parse_args() + + result = write_log_review( + args.run_dir, + output_dir=args.output_dir, + policy_path=args.policy, + previous_run_dir=args.previous_run_dir, + server_log_paths=args.server_log, + ) + print(json.dumps(result, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/open_cmis_tck/archive.py b/src/open_cmis_tck/archive.py new file mode 100644 index 0000000..8a8b8b6 --- /dev/null +++ b/src/open_cmis_tck/archive.py @@ -0,0 +1,99 @@ +"""Durable archive helpers for guide-board OpenCMIS assessment runs.""" + +from __future__ import annotations + +import hashlib +import json +import shutil +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + + +MANIFEST_NAME = "archive-manifest.json" + + +def archive_run( + run_dir: Path, + archive_root: Path, + *, + target_id: str | None = None, + archive_name: str | None = None, +) -> dict[str, Any]: + """Copy a guide-board run directory into a durable archive path.""" + + source = run_dir.resolve() + if not source.exists() or not source.is_dir(): + raise FileNotFoundError(f"run directory does not exist: {source}") + + run_metadata = _load_json(source / "run.json") + target_profile = _load_json(source / "target-profile.snapshot.json") + resolved_target = target_id or target_profile.get("id") or run_metadata.get("target_profile_ref") or "unknown-target" + resolved_run_id = archive_name or run_metadata.get("id") or source.name + archive_dir = archive_root.resolve() / _safe_segment(str(resolved_target)) / _safe_segment(str(resolved_run_id)) + if archive_dir.exists(): + raise FileExistsError(f"archive directory already exists: {archive_dir}") + + archive_dir.mkdir(parents=True) + copied_files = _copy_tree(source, archive_dir) + files = [_file_manifest_entry(archive_dir, relative_path) for relative_path in copied_files] + manifest = { + "id": f"opencmis-run-archive:{resolved_run_id}", + "created_at": _now(), + "source_run_dir": str(source), + "archive_dir": str(archive_dir), + "run_id": run_metadata.get("id") or source.name, + "target_profile_ref": run_metadata.get("target_profile_ref") or target_profile.get("id"), + "assessment_profile_ref": run_metadata.get("assessment_profile_ref"), + "file_count": len(files), + "total_bytes": sum(item["size_bytes"] for item in files), + "files": files, + } + (archive_dir / MANIFEST_NAME).write_text( + json.dumps(manifest, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + return manifest + + +def _copy_tree(source: Path, destination: Path) -> list[Path]: + copied: list[Path] = [] + for path in sorted(source.rglob("*")): + if not path.is_file(): + continue + relative_path = path.relative_to(source) + target = destination / relative_path + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(path, target) + copied.append(relative_path) + return copied + + +def _file_manifest_entry(root: Path, relative_path: Path) -> dict[str, Any]: + path = root / relative_path + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return { + "path": relative_path.as_posix(), + "size_bytes": path.stat().st_size, + "sha256": digest.hexdigest(), + } + + +def _load_json(path: Path) -> dict[str, Any]: + if not path.exists(): + return {} + payload = json.loads(path.read_text(encoding="utf-8")) + return payload if isinstance(payload, dict) else {} + + +def _safe_segment(value: str) -> str: + safe = "".join(char if char.isalnum() or char in {"-", "_", "."} else "-" for char in value.strip()) + safe = "-".join(part for part in safe.split("-") if part) + return safe or "unknown" + + +def _now() -> str: + return datetime.now(timezone.utc).isoformat() diff --git a/src/open_cmis_tck/log_review.py b/src/open_cmis_tck/log_review.py new file mode 100644 index 0000000..420c531 --- /dev/null +++ b/src/open_cmis_tck/log_review.py @@ -0,0 +1,562 @@ +"""OpenCMIS TCK run log review and warning policy classification.""" + +from __future__ import annotations + +import json +import re +from collections import Counter +from datetime import datetime, timezone +from ipaddress import ip_address +from pathlib import Path +from typing import Any +from urllib.parse import urlparse + + +DEFAULT_POLICY_PATH = Path(__file__).resolve().parents[2] / "profiles" / "expectations" / "opencmis-warning-policy.json" +ERROR_TERMS = ("warn", "warning", "error", "severe", "exception", "caused by", "failed") + +SKIP_BOUNDARY_RULES = [ + { + "id": "relationship-type-not-creatable", + "message_contains": "Relationship type 'cmis:relationship' is not creatable", + "required_capability": "cmis.relationships", + "classification": "declared_type_creatability_boundary", + }, + { + "id": "policy-type-not-creatable", + "message_contains": "Policy type 'cmis:policy' is not creatable", + "required_capability": "cmis.policy-mutation", + "classification": "declared_type_creatability_boundary", + }, + { + "id": "item-type-not-creatable", + "message_contains": "Item type 'cmis:item' is not creatable", + "required_capability": "cmis.item-services", + "classification": "declared_type_creatability_boundary", + }, + { + "id": "document-subtype-not-creatable", + "message_contains": "Test document type doesn't allow creating a sub-type", + "required_capability": "cmis.type-mutability", + "classification": "declared_type_mutability_boundary", + }, + { + "id": "folder-name-change-not-supported", + "message_contains": "Folder name can't be changed", + "required_capability": "cmis.folder-name-mutation", + "classification": "declared_folder_mutation_boundary", + }, +] + + +def build_log_review( + run_dir: Path, + *, + policy_path: Path | None = None, + previous_run_dir: Path | None = None, + server_log_paths: list[Path] | None = None, +) -> dict[str, Any]: + """Build a compact review of warnings, stderr, skips, and hard errors.""" + + run = _load_run(run_dir) + policy = _load_policy(policy_path) + previous = _load_run(previous_run_dir) if previous_run_dir is not None else None + warning_reviews = [ + _review_warning(case, run, policy) + for case in run["cases"] + if case.get("status") == "warning" + ] + hard_errors = [ + _case_summary(case, run) + for case in run["cases"] + if case.get("status") in {"fail", "infrastructure_error", "blocked"} + ] + skip_reviews = [ + _review_skip(case, run) + for case in run["cases"] + if case.get("status") == "skipped" + ] + stderr_files = _collect_stderr_files(run["run_dir"]) + server_log_findings = _scan_server_logs(server_log_paths or [], run["run_dir"]) + previous_warning_signatures = { + _case_signature(case) + for case in (previous or {}).get("cases", []) + if case.get("status") == "warning" + } + current_warning_signatures = { + _case_signature(case) + for case in run["cases"] + if case.get("status") == "warning" + } + closed_warnings = [ + _case_summary(case, previous or {}) + for case in (previous or {}).get("cases", []) + if case.get("status") == "warning" + and _case_signature(case) not in current_warning_signatures + ] + new_warnings = [ + item + for item in warning_reviews + if item["signature"] not in previous_warning_signatures + ] + unexpected_findings = [ + finding + for finding in run["findings"] + if not finding.get("expected") + ] + unaccepted_warnings = [item for item in warning_reviews if not item["accepted"]] + unexpected_skips = [item for item in skip_reviews if not item["expected"]] + nonempty_stderr = [item for item in stderr_files if item["size_bytes"] > 0] + + status = _review_status( + hard_errors=hard_errors, + unexpected_findings=unexpected_findings, + unaccepted_warnings=unaccepted_warnings, + unexpected_skips=unexpected_skips, + nonempty_stderr=nonempty_stderr, + warning_reviews=warning_reviews, + skip_reviews=skip_reviews, + ) + summary = { + "status": status, + "case_count": len(run["cases"]), + "case_status_counts": dict(sorted(Counter(case.get("status", "unknown") for case in run["cases"]).items())), + "warning_count": len(warning_reviews), + "accepted_warnings": sum(1 for item in warning_reviews if item["accepted"]), + "unaccepted_warnings": len(unaccepted_warnings), + "new_warnings": len(new_warnings), + "closed_warnings": len(closed_warnings), + "hard_error_count": len(hard_errors), + "stderr_files": len(stderr_files), + "nonempty_stderr_files": len(nonempty_stderr), + "skipped_cases": len(skip_reviews), + "expected_skips": sum(1 for item in skip_reviews if item["expected"]), + "unexpected_skips": len(unexpected_skips), + "unexpected_findings": len(unexpected_findings), + "server_log_findings": len(server_log_findings), + } + return { + "id": f"opencmis-log-review:{run['run_id']}", + "created_at": _now(), + "run": { + "run_id": run["run_id"], + "run_dir": str(run["run_dir"]), + "target_profile_ref": run["target_profile_ref"], + "assessment_profile_ref": run["assessment_profile_ref"], + "target_environment": run["target_environment"], + "browser_binding_url": run["browser_binding_url"], + "declared_capabilities": run["declared_capabilities"], + }, + "policy": { + "id": policy.get("id"), + "path": str(policy.get("_path")) if policy.get("_path") else None, + }, + "summary": summary, + "warnings": warning_reviews, + "new_warnings": new_warnings, + "closed_warnings": closed_warnings, + "hard_errors": hard_errors, + "stderr": stderr_files, + "skips": skip_reviews, + "unexpected_findings": unexpected_findings, + "server_log_findings": server_log_findings, + "certification_boundary": "This log review supports preparation and operational readiness only; it does not certify CMIS conformance.", + } + + +def write_log_review( + run_dir: Path, + *, + output_dir: Path | None = None, + policy_path: Path | None = None, + previous_run_dir: Path | None = None, + server_log_paths: list[Path] | None = None, +) -> dict[str, str]: + output = output_dir or run_dir / "reports" + output.mkdir(parents=True, exist_ok=True) + review = build_log_review( + run_dir, + policy_path=policy_path, + previous_run_dir=previous_run_dir, + server_log_paths=server_log_paths, + ) + json_path = output / "opencmis-log-review.json" + markdown_path = output / "opencmis-log-review.md" + json_path.write_text(json.dumps(review, indent=2, sort_keys=True) + "\n", encoding="utf-8") + markdown_path.write_text(markdown_log_review(review), encoding="utf-8") + return { + "status": "written", + "json": str(json_path), + "markdown": str(markdown_path), + } + + +def markdown_log_review(review: dict[str, Any]) -> str: + summary = review["summary"] + lines = [ + f"# OpenCMIS Log Review: {review['run']['run_id']}", + "", + f"Target: {review['run']['target_profile_ref']}", + f"Assessment: {review['run']['assessment_profile_ref']}", + f"Status: {summary['status']}", + "", + "## Summary", + "", + f"- cases: {summary['case_count']}", + f"- warnings: {summary['warning_count']} ({summary['accepted_warnings']} accepted, {summary['unaccepted_warnings']} unaccepted)", + f"- hard errors: {summary['hard_error_count']}", + f"- stderr files: {summary['nonempty_stderr_files']} non-empty / {summary['stderr_files']} scanned", + f"- skipped cases: {summary['skipped_cases']} ({summary['expected_skips']} expected, {summary['unexpected_skips']} needs review)", + f"- unexpected findings: {summary['unexpected_findings']}", + f"- server log findings: {summary['server_log_findings']}", + "", + "## Warnings", + "", + ] + if review["warnings"]: + for item in review["warnings"]: + lines.extend( + [ + f"- {item['severity']}: {item['classification']} ({item.get('policy_id') or 'no-policy'})", + f" {item['selected_check_group']} / {item['test_name']}: {item['message']}", + ] + ) + else: + lines.append("- none") + lines.extend(["", "## Skips", ""]) + if review["skips"]: + for item in review["skips"]: + expected = "expected" if item["expected"] else "needs review" + lines.append( + f"- {expected}: {item['classification']} / {item['selected_check_group']} / {item['test_name']}: {item['message']}" + ) + else: + lines.append("- none") + lines.extend(["", "## Hard Errors And Stderr", ""]) + if review["hard_errors"]: + for item in review["hard_errors"]: + lines.append(f"- {item['status']}: {item['selected_check_group']} / {item['test_name']}: {item['message']}") + else: + lines.append("- no hard OpenCMIS case errors") + for item in review["stderr"]: + if item["size_bytes"] > 0: + lines.append(f"- non-empty stderr: {item['path']} ({item['size_bytes']} bytes)") + lines.extend(["", "## Closed Warnings", ""]) + if review["closed_warnings"]: + for item in review["closed_warnings"]: + lines.append(f"- {item['selected_check_group']} / {item['test_name']}: {item['message']}") + else: + lines.append("- none") + lines.extend(["", "## Boundary", "", review["certification_boundary"], ""]) + return "\n".join(lines) + + +def _load_run(run_dir: Path | None) -> dict[str, Any]: + if run_dir is None: + return {} + resolved = run_dir.resolve() + run_metadata = _load_json(resolved / "run.json") + target_profile = _load_json(resolved / "target-profile.snapshot.json") + assessment_profile = _load_json(resolved / "assessment-profile.snapshot.json") + evidence = _load_json(resolved / "normalized" / "evidence.json").get("evidence", []) + findings = _load_json(resolved / "normalized" / "findings.json").get("findings", []) + endpoint = _browser_binding_url(target_profile, evidence) + cases = _cases_from_evidence(evidence) + return { + "run_dir": resolved, + "run_id": run_metadata.get("id") or resolved.name, + "target_profile_ref": run_metadata.get("target_profile_ref") or target_profile.get("id"), + "assessment_profile_ref": run_metadata.get("assessment_profile_ref") or assessment_profile.get("id"), + "target_environment": target_profile.get("environment"), + "target_profile": target_profile, + "assessment_profile": assessment_profile, + "browser_binding_url": endpoint, + "declared_capabilities": sorted(target_profile.get("declared_capabilities") or []), + "cases": cases, + "findings": findings if isinstance(findings, list) else [], + } + + +def _cases_from_evidence(evidence: list[dict[str, Any]]) -> list[dict[str, Any]]: + cases: list[dict[str, Any]] = [] + for item in evidence: + facts = item.get("facts") or {} + for case in facts.get("cases") or []: + if not isinstance(case, dict): + continue + enriched = dict(case) + enriched.setdefault("selected_check_group", facts.get("selected_check_group") or facts.get("check_group")) + enriched.setdefault("check_id", item.get("check_id")) + enriched.setdefault("evidence_id", item.get("id")) + cases.append(enriched) + return cases + + +def _review_warning(case: dict[str, Any], run: dict[str, Any], policy: dict[str, Any]) -> dict[str, Any]: + policy_item = _matching_warning_policy(case, policy) + accepted = _warning_is_accepted(policy_item, run) if policy_item else False + if not policy_item: + classification = "unclassified_warning" + severity = "warning" + reason = "No warning policy matched this OpenCMIS warning." + policy_id = None + elif accepted: + classification = policy_item.get("classification", "accepted_warning") + severity = policy_item.get("severity", "info") + reason = policy_item.get("reason", "") + policy_id = policy_item.get("id") + else: + classification = policy_item.get("unaccepted_classification", "unaccepted_warning") + severity = policy_item.get("unaccepted_severity", "warning") + reason = policy_item.get("unaccepted_reason", policy_item.get("reason", "Warning policy matched but acceptance conditions were not met.")) + policy_id = policy_item.get("id") + summary = _case_summary(case, run) + summary.update( + { + "accepted": accepted, + "classification": classification, + "severity": severity, + "reason": reason, + "policy_id": policy_id, + "signature": _case_signature(case), + } + ) + return summary + + +def _matching_warning_policy(case: dict[str, Any], policy: dict[str, Any]) -> dict[str, Any] | None: + for item in policy.get("warning_policies") or []: + if _policy_matches_case(item.get("match") or {}, case): + return item + return None + + +def _policy_matches_case(match: dict[str, Any], case: dict[str, Any]) -> bool: + message = str(case.get("message") or "") + if match.get("message_contains") and str(match["message_contains"]) not in message: + return False + if match.get("test_name_contains") and str(match["test_name_contains"]) not in str(case.get("test_name") or ""): + return False + if match.get("selected_check_group") and match["selected_check_group"] != case.get("selected_check_group"): + return False + source_match = match.get("source_location") or {} + source_location = case.get("source_location") or {} + if source_match.get("file") and source_match["file"] != source_location.get("file"): + return False + if source_match.get("line") and source_match["line"] != source_location.get("line"): + return False + return True + + +def _warning_is_accepted(policy_item: dict[str, Any] | None, run: dict[str, Any]) -> bool: + if not policy_item: + return False + accepted_when = policy_item.get("accepted_when") or {} + target_refs = accepted_when.get("target_profile_refs") + if target_refs and run.get("target_profile_ref") not in target_refs: + return False + environments = accepted_when.get("environments") + if environments and run.get("target_environment") not in environments: + return False + scheme = accepted_when.get("scheme") + parsed = urlparse(str(run.get("browser_binding_url") or "")) + if scheme and parsed.scheme != scheme: + return False + if accepted_when.get("host_scope") == "loopback" and not _is_loopback_host(parsed.hostname): + return False + return True + + +def _review_skip(case: dict[str, Any], run: dict[str, Any]) -> dict[str, Any]: + message = str(case.get("message") or "") + declared = set(run.get("declared_capabilities") or []) + for rule in SKIP_BOUNDARY_RULES: + if rule["message_contains"] not in message: + continue + required = rule["required_capability"] + expected = required not in declared + classification = ( + rule["classification"] + if expected + else "advertised_capability_not_exercised" + ) + summary = _case_summary(case, run) + summary.update( + { + "expected": expected, + "classification": classification, + "required_capability": required, + "rule_id": rule["id"], + } + ) + return summary + summary = _case_summary(case, run) + summary.update( + { + "expected": False, + "classification": "unclassified_skip", + "required_capability": None, + "rule_id": None, + } + ) + return summary + + +def _case_summary(case: dict[str, Any], run: dict[str, Any]) -> dict[str, Any]: + source_location = case.get("source_location") or {} + return { + "id": case.get("id"), + "status": case.get("status"), + "status_native": case.get("status_native"), + "selected_check_group": case.get("selected_check_group"), + "group_name": case.get("group_name"), + "test_name": case.get("test_name"), + "message": case.get("message"), + "source_location": source_location, + "evidence_id": case.get("evidence_id"), + "run_id": run.get("run_id"), + } + + +def _case_signature(case: dict[str, Any]) -> str: + source = case.get("source_location") or {} + parts = [ + str(case.get("selected_check_group") or ""), + str(case.get("test_name") or ""), + str(case.get("message") or ""), + str(source.get("file") or ""), + str(source.get("line") or ""), + ] + return "|".join(parts) + + +def _collect_stderr_files(run_dir: Path) -> list[dict[str, Any]]: + patterns = [ + "artifacts/open-cmis-tck/tck/**/console-runner-stderr.txt", + "artifacts/open-cmis-tck/tck/**/stderr.log", + ] + files: list[Path] = [] + seen: set[Path] = set() + for pattern in patterns: + for path in sorted(run_dir.glob(pattern)): + resolved = path.resolve() + if path.is_file() and resolved not in seen: + files.append(path) + seen.add(resolved) + return [ + { + "path": _relative(path, run_dir), + "size_bytes": path.stat().st_size, + "excerpt": _excerpt(path) if path.stat().st_size else "", + } + for path in files + ] + + +def _scan_server_logs(paths: list[Path], run_dir: Path) -> list[dict[str, Any]]: + findings: list[dict[str, Any]] = [] + for root in paths: + for path in _expand_log_paths(root): + for line_number, line in _matching_log_lines(path): + findings.append( + { + "path": _relative(path, run_dir), + "line": line_number, + "message": line.strip(), + } + ) + return findings + + +def _expand_log_paths(path: Path) -> list[Path]: + if path.is_file(): + return [path] + if path.is_dir(): + return sorted(item for item in path.rglob("*") if item.is_file()) + return [] + + +def _matching_log_lines(path: Path) -> list[tuple[int, str]]: + matches: list[tuple[int, str]] = [] + try: + with path.open("r", encoding="utf-8", errors="replace") as handle: + for index, line in enumerate(handle, start=1): + lowered = line.lower() + if any(term in lowered for term in ERROR_TERMS): + matches.append((index, line)) + if len(matches) >= 50: + break + except OSError: + return [] + return matches + + +def _review_status( + *, + hard_errors: list[dict[str, Any]], + unexpected_findings: list[dict[str, Any]], + unaccepted_warnings: list[dict[str, Any]], + unexpected_skips: list[dict[str, Any]], + nonempty_stderr: list[dict[str, Any]], + warning_reviews: list[dict[str, Any]], + skip_reviews: list[dict[str, Any]], +) -> str: + if hard_errors or unexpected_findings or unaccepted_warnings or unexpected_skips or nonempty_stderr: + return "review_required" + if warning_reviews or skip_reviews: + return "pass_with_review_notes" + return "pass" + + +def _browser_binding_url(target_profile: dict[str, Any], evidence: list[dict[str, Any]]) -> str | None: + for endpoint in target_profile.get("endpoints") or []: + if endpoint.get("binding") == "cmis-browser": + return endpoint.get("url") + for item in evidence: + facts = item.get("facts") or {} + value = facts.get("browser_binding_url") or facts.get("url") + if value: + return str(value) + return None + + +def _load_policy(policy_path: Path | None) -> dict[str, Any]: + path = policy_path or DEFAULT_POLICY_PATH + if not path.exists(): + return {"id": "none", "warning_policies": [], "_path": None} + policy = _load_json(path) + policy["_path"] = path + return policy + + +def _load_json(path: Path) -> dict[str, Any]: + if not path.exists(): + return {} + payload = json.loads(path.read_text(encoding="utf-8")) + return payload if isinstance(payload, dict) else {} + + +def _is_loopback_host(host: str | None) -> bool: + if not host: + return False + if host in {"localhost"}: + return True + try: + return ip_address(host).is_loopback + except ValueError: + return False + + +def _excerpt(path: Path, limit: int = 500) -> str: + return path.read_text(encoding="utf-8", errors="replace")[:limit] + + +def _relative(path: Path, root: Path) -> str: + try: + return str(path.resolve().relative_to(root.resolve())) + except ValueError: + return str(path.resolve()) + + +def _now() -> str: + return datetime.now(timezone.utc).isoformat() diff --git a/tests/test_open_cmis_tck.py b/tests/test_open_cmis_tck.py index 137720a..596f839 100644 --- a/tests/test_open_cmis_tck.py +++ b/tests/test_open_cmis_tck.py @@ -22,7 +22,9 @@ from guide_board.planning import ( ) from guide_board.retention import build_trend_summary from guide_board.service import ServiceHandle, start_service +from open_cmis_tck.archive import archive_run from open_cmis_tck.bootstrap import TCK_COORDINATE, check_runtime +from open_cmis_tck.log_review import build_log_review, write_log_review from open_cmis_tck.normalization import ( aggregate_case_result, parse_text_report, @@ -198,6 +200,169 @@ class OpenCmisTckExtensionTests(unittest.TestCase): self.assertEqual(warning["source_location"], {"file": "SecurityTest.java", "line": 52}) self.assertEqual(failure["message"], "Test folder could not be created.") + def test_log_review_classifies_loopback_warning_and_closed_warning(self) -> None: + with TemporaryDirectory() as temporary_directory: + temp_root = Path(temporary_directory) + previous_run = temp_root / "previous" + current_run = temp_root / "current" + _write_review_run( + previous_run, + "run-previous", + "http://127.0.0.1:8010/cmis/browser", + ["cmis.repository-info", "cmis.object-services"], + [ + _opencmis_case( + "repository-type", + "warning", + "WARNING", + "Security Test (BROWSER)", + "HTTPS is not used. Credentials might be transferred as plain text!", + "SecurityTest.java", + 67, + ), + _opencmis_case( + "object-content", + "warning", + "WARNING", + "Set, Append, and Delete Content Test (BROWSER)", + "appendContentStream() is not supported!", + "SetAndDeleteContentTest.java", + 200, + ), + ], + ) + _write_review_run( + current_run, + "run-current", + "http://127.0.0.1:8010/cmis/browser", + ["cmis.repository-info", "cmis.object-services"], + [ + _opencmis_case( + "repository-type", + "warning", + "WARNING", + "Security Test (BROWSER)", + "HTTPS is not used. Credentials might be transferred as plain text!", + "SecurityTest.java", + 67, + ), + _opencmis_case( + "object-content", + "skipped", + "SKIPPED", + "Create and Delete Relationship Test (BROWSER)", + "Relationship type 'cmis:relationship' is not creatable!", + "AbstractSessionTest.java", + 634, + ), + ], + ) + + review = build_log_review(current_run, previous_run_dir=previous_run) + written = write_log_review(current_run, previous_run_dir=previous_run) + + self.assertEqual(review["summary"]["status"], "pass_with_review_notes") + self.assertEqual(review["summary"]["accepted_warnings"], 1) + self.assertEqual(review["summary"]["unaccepted_warnings"], 0) + self.assertEqual(review["summary"]["closed_warnings"], 1) + self.assertEqual(review["summary"]["expected_skips"], 1) + self.assertEqual( + review["warnings"][0]["classification"], + "accepted_local_loopback_transport", + ) + self.assertEqual( + review["closed_warnings"][0]["message"], + "appendContentStream() is not supported!", + ) + self.assertTrue(Path(written["json"]).exists()) + self.assertIn( + "OpenCMIS Log Review", + Path(written["markdown"]).read_text(encoding="utf-8"), + ) + + def test_log_review_flags_non_loopback_http_warning_as_deployment_blocker(self) -> None: + with TemporaryDirectory() as temporary_directory: + run_dir = Path(temporary_directory) / "run" + _write_review_run( + run_dir, + "run-production-http", + "http://cmis.example.test/browser", + ["cmis.repository-info"], + [ + _opencmis_case( + "repository-type", + "warning", + "WARNING", + "Security Test (BROWSER)", + "HTTPS is not used. Credentials might be transferred as plain text!", + "SecurityTest.java", + 67, + ) + ], + environment="production", + ) + + review = build_log_review(run_dir) + + self.assertEqual(review["summary"]["status"], "review_required") + self.assertEqual(review["summary"]["unaccepted_warnings"], 1) + self.assertEqual(review["warnings"][0]["classification"], "deployment_transport_blocker") + self.assertEqual(review["warnings"][0]["severity"], "blocker") + + def test_log_review_marks_advertised_capability_skip_for_review(self) -> None: + with TemporaryDirectory() as temporary_directory: + run_dir = Path(temporary_directory) / "run" + _write_review_run( + run_dir, + "run-relationship-skip", + "http://127.0.0.1:8010/cmis/browser", + ["cmis.repository-info", "cmis.relationships"], + [ + _opencmis_case( + "object-content", + "skipped", + "SKIPPED", + "Create and Delete Relationship Test (BROWSER)", + "Relationship type 'cmis:relationship' is not creatable!", + "AbstractSessionTest.java", + 634, + ) + ], + ) + + review = build_log_review(run_dir) + + self.assertEqual(review["summary"]["status"], "review_required") + self.assertEqual(review["summary"]["unexpected_skips"], 1) + self.assertFalse(review["skips"][0]["expected"]) + self.assertEqual(review["skips"][0]["classification"], "advertised_capability_not_exercised") + + def test_archive_run_copies_evidence_and_writes_hash_manifest(self) -> None: + with TemporaryDirectory() as temporary_directory: + temp_root = Path(temporary_directory) + run_dir = temp_root / "run" + _write_review_run( + run_dir, + "run-archive", + "http://127.0.0.1:8010/cmis/browser", + ["cmis.repository-info"], + [], + ) + (run_dir / "reports" / "report.md").write_text("# Report\n", encoding="utf-8") + + manifest = archive_run(run_dir, temp_root / "archive") + archive_dir = Path(manifest["archive_dir"]) + manifest_path = archive_dir / "archive-manifest.json" + + self.assertTrue((archive_dir / "reports" / "report.md").exists()) + self.assertTrue(manifest_path.exists()) + self.assertEqual(manifest["run_id"], "run-archive") + self.assertIn( + "normalized/evidence.json", + {item["path"] for item in manifest["files"]}, + ) + self.assertTrue(all(len(item["sha256"]) == 64 for item in manifest["files"])) + def test_console_adapter_dry_run_writes_session_and_group_files(self) -> None: with TemporaryDirectory() as temporary_directory: temp_root = Path(temporary_directory) @@ -877,6 +1042,104 @@ class OpenCmisTckExtensionTests(unittest.TestCase): self.assertTrue(findings[1]["expected"]) +def _write_review_run( + path: Path, + run_id: str, + browser_url: str, + declared_capabilities: list[str], + cases: list[dict[str, object]], + *, + target_id: str = "kontextual-cmis-compat", + environment: str = "local", +) -> None: + groups = sorted({str(case["selected_check_group"]) for case in cases}) or ["repository-type"] + (path / "normalized").mkdir(parents=True) + (path / "reports").mkdir() + for group in groups: + artifact_dir = path / "artifacts" / "open-cmis-tck" / "tck" / group + artifact_dir.mkdir(parents=True, exist_ok=True) + (artifact_dir / "console-runner-stderr.txt").write_text("", encoding="utf-8") + (artifact_dir / "stderr.log").write_text("", encoding="utf-8") + path.joinpath("run.json").write_text( + json.dumps( + { + "id": run_id, + "target_profile_ref": target_id, + "assessment_profile_ref": "cmis-browser-baseline", + } + ), + encoding="utf-8", + ) + path.joinpath("target-profile.snapshot.json").write_text( + json.dumps( + { + "id": target_id, + "environment": environment, + "endpoints": [ + { + "id": "browser-binding", + "url": browser_url, + "binding": "cmis-browser", + } + ], + "declared_capabilities": declared_capabilities, + } + ), + encoding="utf-8", + ) + path.joinpath("assessment-profile.snapshot.json").write_text( + json.dumps({"id": "cmis-browser-baseline"}), + encoding="utf-8", + ) + evidence = [] + for group in groups: + group_cases = [case for case in cases if case["selected_check_group"] == group] + evidence.append( + { + "id": f"evidence:check-group:open-cmis-tck:{group}", + "check_id": f"check-group:open-cmis-tck:{group}", + "result": "warning" if any(case["status"] == "warning" for case in group_cases) else "pass", + "facts": { + "selected_check_group": group, + "browser_binding_url": browser_url, + "cases": group_cases, + }, + } + ) + path.joinpath("normalized", "evidence.json").write_text( + json.dumps({"evidence": evidence}), + encoding="utf-8", + ) + path.joinpath("normalized", "findings.json").write_text( + json.dumps({"findings": []}), + encoding="utf-8", + ) + + +def _opencmis_case( + selected_check_group: str, + status: str, + status_native: str, + test_name: str, + message: str, + source_file: str, + source_line: int, +) -> dict[str, object]: + return { + "id": f"opencmis-tck:{selected_check_group}:{test_name.lower().replace(' ', '-')}", + "status": status, + "status_native": status_native, + "selected_check_group": selected_check_group, + "group_name": "OpenCMIS Test Group", + "test_name": test_name, + "message": message, + "source_location": { + "file": source_file, + "line": source_line, + }, + } + + def _write_target(path: Path, port: int, target_id: str) -> None: path.write_text( json.dumps( diff --git a/workplans/OPEN-CMIS-TCK-WP-0002-live-test-infrastructure.md b/workplans/OPEN-CMIS-TCK-WP-0002-live-test-infrastructure.md index 8cd2d2c..a349ebf 100644 --- a/workplans/OPEN-CMIS-TCK-WP-0002-live-test-infrastructure.md +++ b/workplans/OPEN-CMIS-TCK-WP-0002-live-test-infrastructure.md @@ -5,12 +5,12 @@ title: "Live OpenCMIS TCK Execution And Capability Maturity" repo: open-cmis-tck extension: open-cmis-tck domain: markitect -status: active +status: completed owner: codex planning_priority: high planning_order: 3 created: "2026-05-07" -updated: "2026-05-08" +updated: "2026-05-14" depends_on: - "OPEN-CMIS-TCK-WP-0001" state_hub_workstream_id: "da3f0d16-ba8e-4147-b0fc-ab3462e0b7b0" diff --git a/workplans/OPEN-CMIS-TCK-WP-0003-assessment-log-review-and-hardening.md b/workplans/OPEN-CMIS-TCK-WP-0003-assessment-log-review-and-hardening.md new file mode 100644 index 0000000..0c8ee49 --- /dev/null +++ b/workplans/OPEN-CMIS-TCK-WP-0003-assessment-log-review-and-hardening.md @@ -0,0 +1,326 @@ +--- +id: OPEN-CMIS-TCK-WP-0003 +type: extension-workplan +title: "Assessment Log Review And Hardening" +repo: open-cmis-tck +extension: open-cmis-tck +domain: markitect +status: completed +owner: codex +planning_priority: high +planning_order: 4 +created: "2026-05-14" +updated: "2026-05-14" +depends_on: + - "OPEN-CMIS-TCK-WP-0002" +state_hub_workstream_id: "5711ee2f-eaa9-428a-a4b2-e7383bfbf18a" +--- + +# OPEN-CMIS-TCK-WP-0003: Assessment Log Review And Hardening + +## Purpose + +Use the first real `kontextual-engine` OpenCMIS TCK assessment runs to harden +the `open-cmis-tck` guide-board extension around warning policy, durable +evidence retention, and repeatable log review. + +The latest `kontextual-engine` release-readiness run is healthy for the selected +Browser Binding baseline: no hard TCK failures, no infrastructure errors, no +unexpected findings, and empty stderr artifacts. The remaining work is mostly +facility maturity: make the one current warning intentional, preserve raw +evidence outside ephemeral `/tmp` paths, and give future assessments a compact +"what should we fix next" report instead of relying on manual `rg` passes. + +## Evidence Reviewed + +- Latest release-readiness evidence: + `/home/worsch/kontextual-engine/docs/cmis-opencmis-tck-release-readiness-evidence-2026-05-13T223537Z.md` +- Latest raw release-readiness run: + `/tmp/kontextual-cmis-release-20260514-toolchain` +- Prior raw run before `appendContentStream()` support: + `/tmp/open-cmis-tck-kontextual-20260513T230205Z` +- Earlier implementation evidence: + `/home/worsch/kontextual-engine/docs/cmis-opencmis-tck-implementation-evidence-2026-05-08T092113Z.md` +- Local extension self-test run: + `/home/worsch/open-cmis-tck/.local/runs/opencmis-inmemory-pilot` +- Local OpenCMIS in-memory server logs: + `/home/worsch/open-cmis-tck/.local/opencmis-inmemory/logs` + +## Current Findings + +1. The latest `kontextual-engine` run completed with Guide Board summary + `pass: 2`, `warning: 1`, `unexpected_findings: 0`. +2. `console-runner-stderr.txt` and `stderr.log` are empty for both selected TCK + groups in the latest run. +3. The remaining current warning is from OpenCMIS + `SecurityTest.java:67`: `HTTPS is not used. Credentials might be transferred + as plain text!` +4. The previous `appendContentStream()` warning in + `SetAndDeleteContentTest.java:200` is closed in the latest run. +5. The latest object/content run still has skipped cases for non-creatable + relationship, policy, and item types, plus folder-name change-token subcases. + These align with declared capability boundaries and are not errors, but they + should remain visible as maturity scope. +6. The local OpenCMIS in-memory pilot has two repository/type warnings: + loopback HTTP and `Thin client URI is not set!`. Tomcat and in-memory server + logs did not show warning/error/exception lines in the scan. +7. Evidence retention is fragile: several useful raw runs live under `/tmp`, and + at least one earlier `/tmp` run was already unavailable when later evidence + was written. + +## Boundary + +This workplan hardens the `open-cmis-tck` extension and its assessment +operations. Product changes for `kontextual-engine` belong in that repository. +This workplan may document product-facing follow-up candidates, but it should +not modify the product repo directly. + +## D3.1 - Capture Current Log Triage Baseline + +```task +id: OPEN-CMIS-TCK-WP-0003-T001 +status: done +priority: high +state_hub_task_id: "1a262cad-a945-4a93-a957-02f2fdb497f1" +``` + +Acceptance: + +- Inspect the latest persisted evidence and raw run artifacts for + `kontextual-engine`. +- Separate current findings from older findings that have already been closed. +- Inspect the local in-memory pilot logs so extension self-test warnings are not + confused with product warnings. +- Record the baseline in this workplan. + +Progress: + +- Confirmed the latest raw release-readiness run has no `fail`, + `infrastructure_error`, unexpected finding, stderr output, or exception trace. +- Confirmed the only current `kontextual-engine` TCK warning is local HTTP + transport. +- Confirmed `appendContentStream()` was a warning in the prior raw run and is + gone in the latest raw run. +- Confirmed the local in-memory pilot still reports loopback HTTP and missing + thin-client URI warnings, while server logs are clean. + +## D3.2 - Durable Assessment Archive Path + +```task +id: OPEN-CMIS-TCK-WP-0003-T002 +status: done +priority: high +state_hub_task_id: "1e31b306-f21e-4bac-8e69-56d586d6712e" +``` + +Acceptance: + +- Provide a recommended non-ephemeral output layout for local product + assessments, for example `.local/runs//` or a configured + workspace archive path. +- Add an operator command or documented copy/import step that preserves raw + TCK stdout/stderr, normalized evidence, findings, mappings, run metadata, + report, scorecard, and artifact manifest before `/tmp` cleanup can remove + them. +- Preserve artifact hashes or package metadata so copied evidence remains + auditable. +- Update the local runbook and service/retention notes with the durable path. + +Progress: + +- Added `src/open_cmis_tck/archive.py` and + `scripts/archive_assessment_run.py`. +- The archive command copies a run into `.local/runs/archive//` + by default and writes `archive-manifest.json` with SHA-256 hashes, file + sizes, source path, archive path, run ID, target profile reference, and + assessment profile reference. +- Updated README, local runbook, and service/retention docs with the archive + command. +- Archived the latest kontextual release-readiness run to + `.local/runs/archive/kontextual-cmis-compat/run-20260513T223537Z`. + +## D3.3 - Warning Policy And HTTPS Deployment Gate + +```task +id: OPEN-CMIS-TCK-WP-0003-T003 +status: done +priority: high +state_hub_task_id: "58d2cf64-0db6-4f9a-a8d5-df9ef870557b" +``` + +Acceptance: + +- Define how warning policy distinguishes local loopback test topology from a + deployment or release gate. +- Treat the OpenCMIS HTTP warning as acceptable only for explicit local + loopback profiles or documented local waivers. +- Make non-loopback or release-target HTTP warnings visible as deployment + blockers, even when the TCK group return code is `0`. +- Record warning policy in a profile, expectation, or waiver file rather than + burying it in narrative evidence. + +Progress: + +- Added `profiles/expectations/opencmis-warning-policy.json`. +- Classified the OpenCMIS HTTP warning as accepted only for local/test loopback + HTTP endpoints. +- Non-loopback or production-like HTTP warnings are now classified as + `deployment_transport_blocker` by the log-review command. + +## D3.4 - OpenCMIS In-Memory Pilot Warning Cleanup + +```task +id: OPEN-CMIS-TCK-WP-0003-T004 +status: done +priority: medium +state_hub_task_id: "12331abc-c28f-4140-9a04-a70eb761bccf" +``` + +Acceptance: + +- Investigate whether the local OpenCMIS in-memory server can expose a + `thinClientURI` through configuration. +- If the upstream in-memory server cannot be configured cleanly, mark the + warning as an expected self-test limitation with a precise source location and + explanation. +- Keep the in-memory pilot useful as an extension smoke test without making its + target-specific warnings look like guide-board defects. +- Document the expected warning posture in the local runbook. + +Progress: + +- Added an explicit policy entry for the `opencmis-inmemory-local` + `Thin client URI is not set!` warning. +- The in-memory pilot review now classifies loopback HTTP and missing + thin-client URI as accepted local self-test warnings. +- Optional external server-log findings are reported as context without + changing the run status by themselves, because those log directories may + include historical startup attempts outside the assessed run. + +## D3.5 - Automated Log Review Report + +```task +id: OPEN-CMIS-TCK-WP-0003-T005 +status: done +priority: high +state_hub_task_id: "e445f909-678b-4236-a025-d5913e5473ed" +``` + +Acceptance: + +- Add a command that scans a guide-board run directory for OpenCMIS stdout, + stderr, normalized results, findings, and known server logs. +- Generate `reports/opencmis-log-review.json` and + `reports/opencmis-log-review.md`. +- Highlight hard errors, non-empty stderr, new warnings, known accepted + warnings, skipped cases, unexpected findings, and closed-warning comparisons + when a previous run is supplied. +- Add regression tests with sanitized fixtures for the current HTTP warning, + the now-closed `appendContentStream()` warning, empty stderr, and skipped + capability-boundary cases. + +Progress: + +- Added `src/open_cmis_tck/log_review.py` and + `scripts/opencmis_log_review.py`. +- The command writes `reports/opencmis-log-review.json` and + `reports/opencmis-log-review.md`. +- Verified it against `/tmp/kontextual-cmis-release-20260514-toolchain` with + `/tmp/open-cmis-tck-kontextual-20260513T230205Z` as the previous run. +- Added regression coverage for accepted HTTP warnings, non-loopback deployment + blockers, closed append warnings, stderr handling, and skipped capability + boundaries. + +## D3.6 - Skip And Capability Boundary Interpretation + +```task +id: OPEN-CMIS-TCK-WP-0003-T006 +status: done +priority: medium +state_hub_task_id: "6df3e8c5-c2d2-4eb3-973e-76644ef7ee8f" +``` + +Acceptance: + +- Group skipped OpenCMIS cases by declared repository capability or type + creatability boundary. +- Distinguish "expected because the capability is not advertised" from "skipped + because the target could not exercise an advertised capability." +- Keep skipped cases visible in reports and maturity scorecards without treating + them as failures when they match the advertised capability profile. +- Add coverage for relationship, policy, item, type-subtype, and folder-name + change-token skips seen in the latest raw run. + +Progress: + +- Added skip-boundary classification in the log-review report. +- Current expected skip rules cover relationship, policy, item, document + subtype, and folder-name mutation cases. +- If the target advertises the required capability and OpenCMIS still skips the + case, the review becomes `review_required`. + +## D3.7 - Next Coverage Frontier + +```task +id: OPEN-CMIS-TCK-WP-0003-T007 +status: done +priority: medium +state_hub_task_id: "f793e1b8-a2b1-4f9b-9972-b6b18d1ba56a" +``` + +Acceptance: + +- Identify which additional OpenCMIS TCK groups are realistic after the current + repository/type and object/content baseline. +- For each candidate group, record target preconditions, likely product + capability requirements, and expected unsupported-by-design boundaries. +- Do not expand the default baseline until the warning policy and durable + evidence path are in place. +- Produce a short recommendation for the next maturity slice, likely navigation, + query, ACL/policy, versioning/PWC, or change-log depth. + +Progress: + +- Added the next-coverage recommendation to `docs/LOG-REVIEW.md`. +- Recommended order is navigation/read-path depth first, metadata query second, + ACL/policy discovery third, and versioning/PWC/change-log only after the + product deliberately advertises those capabilities. +- Left the default baseline unchanged at `repository-type` plus + `object-content`. + +## D3.8 - State Hub And Operator Docs + +```task +id: OPEN-CMIS-TCK-WP-0003-T008 +status: done +priority: medium +state_hub_task_id: "32724628-d427-47c2-ac40-3f3f90e3a2b9" +``` + +Acceptance: + +- Sync this workplan with the state hub. +- Update README/runbook references so operators know how to review warnings + after a run. +- Make it clear that guide-board produces preparation evidence and operational + readiness signals, not certification or audit assurance. +- Ensure doc updates cite the latest raw and persisted evidence baselines. + +Progress: + +- Added `docs/LOG-REVIEW.md`. +- Updated README, local runbook, and service/retention docs. +- Synced the completed workplan and task statuses into the state hub. + +## Definition Of Done + +- Future local CMIS assessments keep raw evidence in a durable run location. +- HTTP transport warnings are policy-classified rather than manually explained + after every run. +- The local in-memory pilot has either zero unexpected warnings or a documented + expected-warning profile. +- A log-review report can be generated from any guide-board run directory. +- Skipped OpenCMIS cases are interpreted against advertised CMIS capability + boundaries. +- The next coverage frontier is explicit and does not blur preparation evidence + with formal certification.