coulomb/open-cmis-tck
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Log analysis functionality for self-assessment

Browse Source
This commit is contained in:
Bernd Worsch
2026-05-14 02:47:00 +02:00
parent 97a4a1fa37
commit cd43c7cfec
12 changed files with 1573 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
README.md
View File

@@ -66,6 +66,18 @@ If the workstation does not have Java and Maven in WSL, see
[docs/LOCAL-RUNBOOK.md](docs/LOCAL-RUNBOOK.md) for the repo-local toolchain
installer path.
After a run, generate the OpenCMIS warning/stderr/skip review and archive any
important product evidence outside `/tmp`:
```sh
cd ../open-cmis-tck
PYTHONPATH=src python3 scripts/opencmis_log_review.py \
--run-dir /tmp/open-cmis-tck-baseline
PYTHONPATH=src python3 scripts/archive_assessment_run.py \
--run-dir /tmp/open-cmis-tck-baseline \
--archive-root .local/runs/archive
```
## Tests
Run extension tests with the guide-board core on `PYTHONPATH`:
@@ -81,6 +93,7 @@ PYTHONPATH=../guide-board/src python3 -m unittest discover -s tests
- [docs/CONTAINER-HANDOFF.md](docs/CONTAINER-HANDOFF.md)
- [docs/LOCAL-RUNBOOK.md](docs/LOCAL-RUNBOOK.md)
- [docs/LOCAL-TCK-RUNTIME.md](docs/LOCAL-TCK-RUNTIME.md)
- [docs/LOG-REVIEW.md](docs/LOG-REVIEW.md)
- [docs/OPENCMIS-TCK-RUNNER.md](docs/OPENCMIS-TCK-RUNNER.md)
- [docs/SERVICE-AND-RETENTION.md](docs/SERVICE-AND-RETENTION.md)

23
docs/LOCAL-RUNBOOK.md
View File

@@ -193,12 +193,35 @@ PYTHONPATH=src python3 scripts/cmis_scorecard.py \
--run-dir /tmp/open-cmis-tck-live
```
Generate the log review after the scorecard:
```sh
PYTHONPATH=src python3 scripts/opencmis_log_review.py \
--run-dir /tmp/open-cmis-tck-live
```
The review classifies warnings, hard errors, stderr, skipped cases, and closed
warnings when a previous run is supplied. The default warning policy accepts the
OpenCMIS HTTP transport warning only for local/test loopback profiles.
For important product assessments, archive the run before `/tmp` cleanup:
```sh
PYTHONPATH=src python3 scripts/archive_assessment_run.py \
--run-dir /tmp/open-cmis-tck-live \
--archive-root .local/runs/archive
```
The archive writes `.local/runs/archive/<target>/<run-id>/archive-manifest.json`
with SHA-256 hashes for every copied file.
Review the real TCK evidence before expanding the scope:
```text
/tmp/open-cmis-tck-live/normalized/evidence.json
/tmp/open-cmis-tck-live/artifacts/open-cmis-tck/tck/<check-group>/console-runner-stdout.txt
/tmp/open-cmis-tck-live/artifacts/open-cmis-tck/tck/<check-group>/normalized-runner-result.json
/tmp/open-cmis-tck-live/reports/opencmis-log-review.md
```
The normalizer preserves native OpenCMIS statuses (`OK`, `WARNING`, `FAILURE`,

132
docs/LOG-REVIEW.md Normal file
View File

@@ -0,0 +1,132 @@
# OpenCMIS Log Review
Status: draft
Created: 2026-05-14
## Purpose
The log review command turns a guide-board OpenCMIS run directory into a compact
fix-or-acceptance report. It reads normalized OpenCMIS cases, findings,
stderr artifacts, optional server logs, and the warning policy profile.
It answers four operator questions:
- Did any hard OpenCMIS case fail?
- Did the runner or server write unexpected stderr/log errors?
- Are warnings accepted local-test conditions or release blockers?
- Do skipped cases match the target's advertised capability boundary?
## Command
Generate a review after a run:
```sh
cd /home/worsch/open-cmis-tck
PYTHONPATH=src python3 scripts/opencmis_log_review.py \
--run-dir /tmp/open-cmis-tck-live
```
The command writes:
```text
/tmp/open-cmis-tck-live/reports/opencmis-log-review.json
/tmp/open-cmis-tck-live/reports/opencmis-log-review.md
```
Compare with a previous run to surface closed and new warnings:
```sh
PYTHONPATH=src python3 scripts/opencmis_log_review.py \
--run-dir /tmp/kontextual-cmis-release-20260514-toolchain \
--previous-run-dir /tmp/open-cmis-tck-kontextual-20260513T230205Z
```
Include known server logs when available:
```sh
PYTHONPATH=src python3 scripts/opencmis_log_review.py \
--run-dir .local/runs/opencmis-inmemory-pilot \
--server-log .local/opencmis-inmemory/logs \
--server-log .local/opencmis-inmemory/containers/apache-tomcat-9.0.117/logs
```
Server-log matches are reported as review context. They do not change the
overall review status by themselves because external log directories often
contain historical startup attempts from before the assessed run.
## Warning Policy
The default policy is:
```text
profiles/expectations/opencmis-warning-policy.json
```
It currently classifies:
- `HTTPS is not used...` as acceptable only for explicit local/test loopback
HTTP endpoints. The same warning becomes a deployment transport blocker for
non-loopback or production-like targets.
- `Thin client URI is not set!` as an accepted limitation only for the
`opencmis-inmemory-local` self-test target.
Policies intentionally live outside narrative evidence. A release run should
show whether each warning is accepted by policy, not rely on a human remembering
the rule.
## Skip Interpretation
Skipped OpenCMIS cases are grouped by capability boundary. A skip is treated as
expected when the corresponding capability is not advertised by the target
profile. For example:
- non-creatable `cmis:relationship` -> requires `cmis.relationships`
- non-creatable `cmis:policy` -> requires `cmis.policy-mutation`
- non-creatable `cmis:item` -> requires `cmis.item-services`
- document sub-type mutation -> requires `cmis.type-mutability`
- folder-name mutation -> requires `cmis.folder-name-mutation`
If a target advertises the relevant capability and the TCK still skips the case,
the review status becomes `review_required`.
## Durable Archive
Do not leave important product assessment evidence only under `/tmp`. Archive a
run after scorecard/log-review generation:
```sh
cd /home/worsch/open-cmis-tck
PYTHONPATH=src python3 scripts/archive_assessment_run.py \
--run-dir /tmp/kontextual-cmis-release-20260514-toolchain \
--archive-root .local/runs/archive
```
The archive command copies the run directory and writes:
```text
.local/runs/archive/<target>/<run-id>/archive-manifest.json
```
The manifest records the source path, archive path, run metadata, file count,
total bytes, and SHA-256 for every copied file.
## Next Coverage Frontier
Keep the default baseline at `repository-type` plus `object-content` until
warning policy and durable evidence are routine. The next recommended maturity
slice is navigation/read-path depth, because it sits closest to current
object/content behavior and reveals path, parent, children, and filing semantics
without immediately requiring versioning or full query depth.
Candidate order:
1. Navigation/read-path checks with unsupported filing mutations clearly scoped.
2. Metadata query checks for the advertised `metadataonly` query posture.
3. ACL/policy discovery depth before ACL/policy mutation.
4. Versioning/PWC and change-log depth only after the product deliberately
advertises those capabilities.
## Boundary
This log review supports preparation evidence and operational readiness. It
does not issue CMIS certification or audit assurance.

27
docs/SERVICE-AND-RETENTION.md
View File

@@ -39,6 +39,13 @@ CMIS runs use the guide-board run directory contract. Each run writes:
- `reports/assessment-package.json`
- `reports/report.md`
Extension-side post-processing can also add:
- `reports/cmis-maturity-scorecard.json`
- `reports/cmis-maturity-scorecard.md`
- `reports/opencmis-log-review.json`
- `reports/opencmis-log-review.md`
The sample assessment profile keeps summaries for 365 days and raw artifacts for
30 days:
@@ -54,3 +61,23 @@ The sample assessment profile keeps summaries for 365 days and raw artifacts for
Compact `retention-summary.json` files are suitable for guide-board trend
summaries and downstream CMIS capability scorecards without retaining unbounded
raw TCK logs.
## Durable Local Archive
For product-facing assessment evidence, do not rely on `/tmp` as the only copy.
Archive completed runs after scorecard and log-review generation:
```sh
cd /home/worsch/open-cmis-tck
PYTHONPATH=src python3 scripts/archive_assessment_run.py \
--run-dir /tmp/open-cmis-tck-live \
--archive-root .local/runs/archive
```
The archive command copies the full run directory and writes
`archive-manifest.json` with SHA-256 hashes, file sizes, source path, archive
path, run ID, target profile reference, and assessment profile reference.
The default local archive root remains under `.local/`, so it is not committed.
Move selected archive packages into a controlled evidence store when the run is
used for release or external audit preparation.

54
profiles/expectations/opencmis-warning-policy.json Normal file
View File

@@ -0,0 +1,54 @@
{
"id": "opencmis-warning-policy",
"description": "Warning classification policy for local OpenCMIS TCK preparation runs.",
"warning_policies": [
{
"id": "local-loopback-http-transport",
"match": {
"message_contains": "HTTPS is not used",
"source_location": {
"file": "SecurityTest.java"
}
},
"accepted_when": {
"scheme": "http",
"host_scope": "loopback",
"environments": [
"local",
"test",
"development"
]
},
"classification": "accepted_local_loopback_transport",
"severity": "info",
"reason": "OpenCMIS warns about HTTP credentials. This is acceptable for explicit loopback-only local runs and must not be used as a deployment-release claim.",
"unaccepted_classification": "deployment_transport_blocker",
"unaccepted_severity": "blocker",
"unaccepted_reason": "OpenCMIS reported plain HTTP outside the accepted local loopback boundary. Use HTTPS termination or an explicit approved waiver for deployment-like targets."
},
{
"id": "opencmis-inmemory-thin-client-uri",
"match": {
"message_contains": "Thin client URI is not set",
"source_location": {
"file": "RepositoryInfoTest.java"
}
},
"accepted_when": {
"target_profile_refs": [
"opencmis-inmemory-local"
],
"environments": [
"local",
"test"
]
},
"classification": "accepted_inmemory_self_test_limitation",
"severity": "info",
"reason": "The Apache Chemistry in-memory server is a local extension smoke target. Missing thinClientURI is a target-specific self-test limitation, not a guide-board extension defect.",
"unaccepted_classification": "repository_info_warning",
"unaccepted_severity": "warning",
"unaccepted_reason": "A non-in-memory target should expose or intentionally document its thin client URI behavior."
}
]
}

35
scripts/archive_assessment_run.py Normal file
View File

@@ -0,0 +1,35 @@
#!/usr/bin/env python3
"""Archive a guide-board OpenCMIS assessment run with a hash manifest."""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from open_cmis_tck.archive import archive_run # noqa: E402
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--run-dir", type=Path, required=True)
parser.add_argument("--archive-root", type=Path, default=Path(".local/runs/archive"))
parser.add_argument("--target-id")
parser.add_argument("--archive-name")
args = parser.parse_args()
manifest = archive_run(
args.run_dir,
args.archive_root,
target_id=args.target_id,
archive_name=args.archive_name,
)
print(json.dumps(manifest, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())

37
scripts/opencmis_log_review.py Normal file
View File

@@ -0,0 +1,37 @@
#!/usr/bin/env python3
"""Generate OpenCMIS TCK warning, stderr, and skip review reports."""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from open_cmis_tck.log_review import write_log_review # noqa: E402
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--run-dir", type=Path, required=True)
parser.add_argument("--output-dir", type=Path)
parser.add_argument("--policy", type=Path)
parser.add_argument("--previous-run-dir", type=Path)
parser.add_argument("--server-log", type=Path, action="append", default=[])
args = parser.parse_args()
result = write_log_review(
args.run_dir,
output_dir=args.output_dir,
policy_path=args.policy,
previous_run_dir=args.previous_run_dir,
server_log_paths=args.server_log,
)
print(json.dumps(result, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())

99
src/open_cmis_tck/archive.py Normal file
View File

@@ -0,0 +1,99 @@
"""Durable archive helpers for guide-board OpenCMIS assessment runs."""
from __future__ import annotations
import hashlib
import json
import shutil
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
MANIFEST_NAME = "archive-manifest.json"
def archive_run(
run_dir: Path,
archive_root: Path,
*,
target_id: str | None = None,
archive_name: str | None = None,
) -> dict[str, Any]:
"""Copy a guide-board run directory into a durable archive path."""
source = run_dir.resolve()
if not source.exists() or not source.is_dir():
raise FileNotFoundError(f"run directory does not exist: {source}")
run_metadata = _load_json(source / "run.json")
target_profile = _load_json(source / "target-profile.snapshot.json")
resolved_target = target_id or target_profile.get("id") or run_metadata.get("target_profile_ref") or "unknown-target"
resolved_run_id = archive_name or run_metadata.get("id") or source.name
archive_dir = archive_root.resolve() / _safe_segment(str(resolved_target)) / _safe_segment(str(resolved_run_id))
if archive_dir.exists():
raise FileExistsError(f"archive directory already exists: {archive_dir}")
archive_dir.mkdir(parents=True)
copied_files = _copy_tree(source, archive_dir)
files = [_file_manifest_entry(archive_dir, relative_path) for relative_path in copied_files]
manifest = {
"id": f"opencmis-run-archive:{resolved_run_id}",
"created_at": _now(),
"source_run_dir": str(source),
"archive_dir": str(archive_dir),
"run_id": run_metadata.get("id") or source.name,
"target_profile_ref": run_metadata.get("target_profile_ref") or target_profile.get("id"),
"assessment_profile_ref": run_metadata.get("assessment_profile_ref"),
"file_count": len(files),
"total_bytes": sum(item["size_bytes"] for item in files),
"files": files,
}
(archive_dir / MANIFEST_NAME).write_text(
json.dumps(manifest, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
return manifest
def _copy_tree(source: Path, destination: Path) -> list[Path]:
copied: list[Path] = []
for path in sorted(source.rglob("*")):
if not path.is_file():
continue
relative_path = path.relative_to(source)
target = destination / relative_path
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(path, target)
copied.append(relative_path)
return copied
def _file_manifest_entry(root: Path, relative_path: Path) -> dict[str, Any]:
path = root / relative_path
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return {
"path": relative_path.as_posix(),
"size_bytes": path.stat().st_size,
"sha256": digest.hexdigest(),
}
def _load_json(path: Path) -> dict[str, Any]:
if not path.exists():
return {}
payload = json.loads(path.read_text(encoding="utf-8"))
return payload if isinstance(payload, dict) else {}
def _safe_segment(value: str) -> str:
safe = "".join(char if char.isalnum() or char in {"-", "_", "."} else "-" for char in value.strip())
safe = "-".join(part for part in safe.split("-") if part)
return safe or "unknown"
def _now() -> str:
return datetime.now(timezone.utc).isoformat()

562
src/open_cmis_tck/log_review.py Normal file
View File

@@ -0,0 +1,562 @@
"""OpenCMIS TCK run log review and warning policy classification."""
from __future__ import annotations
import json
import re
from collections import Counter
from datetime import datetime, timezone
from ipaddress import ip_address
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
DEFAULT_POLICY_PATH = Path(__file__).resolve().parents[2] / "profiles" / "expectations" / "opencmis-warning-policy.json"
ERROR_TERMS = ("warn", "warning", "error", "severe", "exception", "caused by", "failed")
SKIP_BOUNDARY_RULES = [
{
"id": "relationship-type-not-creatable",
"message_contains": "Relationship type 'cmis:relationship' is not creatable",
"required_capability": "cmis.relationships",
"classification": "declared_type_creatability_boundary",
},
{
"id": "policy-type-not-creatable",
"message_contains": "Policy type 'cmis:policy' is not creatable",
"required_capability": "cmis.policy-mutation",
"classification": "declared_type_creatability_boundary",
},
{
"id": "item-type-not-creatable",
"message_contains": "Item type 'cmis:item' is not creatable",
"required_capability": "cmis.item-services",
"classification": "declared_type_creatability_boundary",
},
{
"id": "document-subtype-not-creatable",
"message_contains": "Test document type doesn't allow creating a sub-type",
"required_capability": "cmis.type-mutability",
"classification": "declared_type_mutability_boundary",
},
{
"id": "folder-name-change-not-supported",
"message_contains": "Folder name can't be changed",
"required_capability": "cmis.folder-name-mutation",
"classification": "declared_folder_mutation_boundary",
},
]
def build_log_review(
run_dir: Path,
*,
policy_path: Path | None = None,
previous_run_dir: Path | None = None,
server_log_paths: list[Path] | None = None,
) -> dict[str, Any]:
"""Build a compact review of warnings, stderr, skips, and hard errors."""
run = _load_run(run_dir)
policy = _load_policy(policy_path)
previous = _load_run(previous_run_dir) if previous_run_dir is not None else None
warning_reviews = [
_review_warning(case, run, policy)
for case in run["cases"]
if case.get("status") == "warning"
]
hard_errors = [
_case_summary(case, run)
for case in run["cases"]
if case.get("status") in {"fail", "infrastructure_error", "blocked"}
]
skip_reviews = [
_review_skip(case, run)
for case in run["cases"]
if case.get("status") == "skipped"
]
stderr_files = _collect_stderr_files(run["run_dir"])
server_log_findings = _scan_server_logs(server_log_paths or [], run["run_dir"])
previous_warning_signatures = {
_case_signature(case)
for case in (previous or {}).get("cases", [])
if case.get("status") == "warning"
}
current_warning_signatures = {
_case_signature(case)
for case in run["cases"]
if case.get("status") == "warning"
}
closed_warnings = [
_case_summary(case, previous or {})
for case in (previous or {}).get("cases", [])
if case.get("status") == "warning"
and _case_signature(case) not in current_warning_signatures
]
new_warnings = [
item
for item in warning_reviews
if item["signature"] not in previous_warning_signatures
]
unexpected_findings = [
finding
for finding in run["findings"]
if not finding.get("expected")
]
unaccepted_warnings = [item for item in warning_reviews if not item["accepted"]]
unexpected_skips = [item for item in skip_reviews if not item["expected"]]
nonempty_stderr = [item for item in stderr_files if item["size_bytes"] > 0]
status = _review_status(
hard_errors=hard_errors,
unexpected_findings=unexpected_findings,
unaccepted_warnings=unaccepted_warnings,
unexpected_skips=unexpected_skips,
nonempty_stderr=nonempty_stderr,
warning_reviews=warning_reviews,
skip_reviews=skip_reviews,
)
summary = {
"status": status,
"case_count": len(run["cases"]),
"case_status_counts": dict(sorted(Counter(case.get("status", "unknown") for case in run["cases"]).items())),
"warning_count": len(warning_reviews),
"accepted_warnings": sum(1 for item in warning_reviews if item["accepted"]),
"unaccepted_warnings": len(unaccepted_warnings),
"new_warnings": len(new_warnings),
"closed_warnings": len(closed_warnings),
"hard_error_count": len(hard_errors),
"stderr_files": len(stderr_files),
"nonempty_stderr_files": len(nonempty_stderr),
"skipped_cases": len(skip_reviews),
"expected_skips": sum(1 for item in skip_reviews if item["expected"]),
"unexpected_skips": len(unexpected_skips),
"unexpected_findings": len(unexpected_findings),
"server_log_findings": len(server_log_findings),
}
return {
"id": f"opencmis-log-review:{run['run_id']}",
"created_at": _now(),
"run": {
"run_id": run["run_id"],
"run_dir": str(run["run_dir"]),
"target_profile_ref": run["target_profile_ref"],
"assessment_profile_ref": run["assessment_profile_ref"],
"target_environment": run["target_environment"],
"browser_binding_url": run["browser_binding_url"],
"declared_capabilities": run["declared_capabilities"],
},
"policy": {
"id": policy.get("id"),
"path": str(policy.get("_path")) if policy.get("_path") else None,
},
"summary": summary,
"warnings": warning_reviews,
"new_warnings": new_warnings,
"closed_warnings": closed_warnings,
"hard_errors": hard_errors,
"stderr": stderr_files,
"skips": skip_reviews,
"unexpected_findings": unexpected_findings,
"server_log_findings": server_log_findings,
"certification_boundary": "This log review supports preparation and operational readiness only; it does not certify CMIS conformance.",
}
def write_log_review(
run_dir: Path,
*,
output_dir: Path | None = None,
policy_path: Path | None = None,
previous_run_dir: Path | None = None,
server_log_paths: list[Path] | None = None,
) -> dict[str, str]:
output = output_dir or run_dir / "reports"
output.mkdir(parents=True, exist_ok=True)
review = build_log_review(
run_dir,
policy_path=policy_path,
previous_run_dir=previous_run_dir,
server_log_paths=server_log_paths,
)
json_path = output / "opencmis-log-review.json"
markdown_path = output / "opencmis-log-review.md"
json_path.write_text(json.dumps(review, indent=2, sort_keys=True) + "\n", encoding="utf-8")
markdown_path.write_text(markdown_log_review(review), encoding="utf-8")
return {
"status": "written",
"json": str(json_path),
"markdown": str(markdown_path),
}
def markdown_log_review(review: dict[str, Any]) -> str:
summary = review["summary"]
lines = [
f"# OpenCMIS Log Review: {review['run']['run_id']}",
"",
f"Target: {review['run']['target_profile_ref']}",
f"Assessment: {review['run']['assessment_profile_ref']}",
f"Status: {summary['status']}",
"",
"## Summary",
"",
f"- cases: {summary['case_count']}",
f"- warnings: {summary['warning_count']} ({summary['accepted_warnings']} accepted, {summary['unaccepted_warnings']} unaccepted)",
f"- hard errors: {summary['hard_error_count']}",
f"- stderr files: {summary['nonempty_stderr_files']} non-empty / {summary['stderr_files']} scanned",
f"- skipped cases: {summary['skipped_cases']} ({summary['expected_skips']} expected, {summary['unexpected_skips']} needs review)",
f"- unexpected findings: {summary['unexpected_findings']}",
f"- server log findings: {summary['server_log_findings']}",
"",
"## Warnings",
"",
]
if review["warnings"]:
for item in review["warnings"]:
lines.extend(
[
f"- {item['severity']}: {item['classification']} ({item.get('policy_id') or 'no-policy'})",
f" {item['selected_check_group']} / {item['test_name']}: {item['message']}",
]
)
else:
lines.append("- none")
lines.extend(["", "## Skips", ""])
if review["skips"]:
for item in review["skips"]:
expected = "expected" if item["expected"] else "needs review"
lines.append(
f"- {expected}: {item['classification']} / {item['selected_check_group']} / {item['test_name']}: {item['message']}"
)
else:
lines.append("- none")
lines.extend(["", "## Hard Errors And Stderr", ""])
if review["hard_errors"]:
for item in review["hard_errors"]:
lines.append(f"- {item['status']}: {item['selected_check_group']} / {item['test_name']}: {item['message']}")
else:
lines.append("- no hard OpenCMIS case errors")
for item in review["stderr"]:
if item["size_bytes"] > 0:
lines.append(f"- non-empty stderr: {item['path']} ({item['size_bytes']} bytes)")
lines.extend(["", "## Closed Warnings", ""])
if review["closed_warnings"]:
for item in review["closed_warnings"]:
lines.append(f"- {item['selected_check_group']} / {item['test_name']}: {item['message']}")
else:
lines.append("- none")
lines.extend(["", "## Boundary", "", review["certification_boundary"], ""])
return "\n".join(lines)
def _load_run(run_dir: Path | None) -> dict[str, Any]:
if run_dir is None:
return {}
resolved = run_dir.resolve()
run_metadata = _load_json(resolved / "run.json")
target_profile = _load_json(resolved / "target-profile.snapshot.json")
assessment_profile = _load_json(resolved / "assessment-profile.snapshot.json")
evidence = _load_json(resolved / "normalized" / "evidence.json").get("evidence", [])
findings = _load_json(resolved / "normalized" / "findings.json").get("findings", [])
endpoint = _browser_binding_url(target_profile, evidence)
cases = _cases_from_evidence(evidence)
return {
"run_dir": resolved,
"run_id": run_metadata.get("id") or resolved.name,
"target_profile_ref": run_metadata.get("target_profile_ref") or target_profile.get("id"),
"assessment_profile_ref": run_metadata.get("assessment_profile_ref") or assessment_profile.get("id"),
"target_environment": target_profile.get("environment"),
"target_profile": target_profile,
"assessment_profile": assessment_profile,
"browser_binding_url": endpoint,
"declared_capabilities": sorted(target_profile.get("declared_capabilities") or []),
"cases": cases,
"findings": findings if isinstance(findings, list) else [],
}
def _cases_from_evidence(evidence: list[dict[str, Any]]) -> list[dict[str, Any]]:
cases: list[dict[str, Any]] = []
for item in evidence:
facts = item.get("facts") or {}
for case in facts.get("cases") or []:
if not isinstance(case, dict):
continue
enriched = dict(case)
enriched.setdefault("selected_check_group", facts.get("selected_check_group") or facts.get("check_group"))
enriched.setdefault("check_id", item.get("check_id"))
enriched.setdefault("evidence_id", item.get("id"))
cases.append(enriched)
return cases
def _review_warning(case: dict[str, Any], run: dict[str, Any], policy: dict[str, Any]) -> dict[str, Any]:
policy_item = _matching_warning_policy(case, policy)
accepted = _warning_is_accepted(policy_item, run) if policy_item else False
if not policy_item:
classification = "unclassified_warning"
severity = "warning"
reason = "No warning policy matched this OpenCMIS warning."
policy_id = None
elif accepted:
classification = policy_item.get("classification", "accepted_warning")
severity = policy_item.get("severity", "info")
reason = policy_item.get("reason", "")
policy_id = policy_item.get("id")
else:
classification = policy_item.get("unaccepted_classification", "unaccepted_warning")
severity = policy_item.get("unaccepted_severity", "warning")
reason = policy_item.get("unaccepted_reason", policy_item.get("reason", "Warning policy matched but acceptance conditions were not met."))
policy_id = policy_item.get("id")
summary = _case_summary(case, run)
summary.update(
{
"accepted": accepted,
"classification": classification,
"severity": severity,
"reason": reason,
"policy_id": policy_id,
"signature": _case_signature(case),
}
)
return summary
def _matching_warning_policy(case: dict[str, Any], policy: dict[str, Any]) -> dict[str, Any] | None:
for item in policy.get("warning_policies") or []:
if _policy_matches_case(item.get("match") or {}, case):
return item
return None
def _policy_matches_case(match: dict[str, Any], case: dict[str, Any]) -> bool:
message = str(case.get("message") or "")
if match.get("message_contains") and str(match["message_contains"]) not in message:
return False
if match.get("test_name_contains") and str(match["test_name_contains"]) not in str(case.get("test_name") or ""):
return False
if match.get("selected_check_group") and match["selected_check_group"] != case.get("selected_check_group"):
return False
source_match = match.get("source_location") or {}
source_location = case.get("source_location") or {}
if source_match.get("file") and source_match["file"] != source_location.get("file"):
return False
if source_match.get("line") and source_match["line"] != source_location.get("line"):
return False
return True
def _warning_is_accepted(policy_item: dict[str, Any] | None, run: dict[str, Any]) -> bool:
if not policy_item:
return False
accepted_when = policy_item.get("accepted_when") or {}
target_refs = accepted_when.get("target_profile_refs")
if target_refs and run.get("target_profile_ref") not in target_refs:
return False
environments = accepted_when.get("environments")
if environments and run.get("target_environment") not in environments:
return False
scheme = accepted_when.get("scheme")
parsed = urlparse(str(run.get("browser_binding_url") or ""))
if scheme and parsed.scheme != scheme:
return False
if accepted_when.get("host_scope") == "loopback" and not _is_loopback_host(parsed.hostname):
return False
return True
def _review_skip(case: dict[str, Any], run: dict[str, Any]) -> dict[str, Any]:
message = str(case.get("message") or "")
declared = set(run.get("declared_capabilities") or [])
for rule in SKIP_BOUNDARY_RULES:
if rule["message_contains"] not in message:
continue
required = rule["required_capability"]
expected = required not in declared
classification = (
rule["classification"]
if expected
else "advertised_capability_not_exercised"
)
summary = _case_summary(case, run)
summary.update(
{
"expected": expected,
"classification": classification,
"required_capability": required,
"rule_id": rule["id"],
}
)
return summary
summary = _case_summary(case, run)
summary.update(
{
"expected": False,
"classification": "unclassified_skip",
"required_capability": None,
"rule_id": None,
}
)
return summary
def _case_summary(case: dict[str, Any], run: dict[str, Any]) -> dict[str, Any]:
source_location = case.get("source_location") or {}
return {
"id": case.get("id"),
"status": case.get("status"),
"status_native": case.get("status_native"),
"selected_check_group": case.get("selected_check_group"),
"group_name": case.get("group_name"),
"test_name": case.get("test_name"),
"message": case.get("message"),
"source_location": source_location,
"evidence_id": case.get("evidence_id"),
"run_id": run.get("run_id"),
}
def _case_signature(case: dict[str, Any]) -> str:
source = case.get("source_location") or {}
parts = [
str(case.get("selected_check_group") or ""),
str(case.get("test_name") or ""),
str(case.get("message") or ""),
str(source.get("file") or ""),
str(source.get("line") or ""),
]
return "|".join(parts)
def _collect_stderr_files(run_dir: Path) -> list[dict[str, Any]]:
patterns = [
"artifacts/open-cmis-tck/tck/**/console-runner-stderr.txt",
"artifacts/open-cmis-tck/tck/**/stderr.log",
]
files: list[Path] = []
seen: set[Path] = set()
for pattern in patterns:
for path in sorted(run_dir.glob(pattern)):
resolved = path.resolve()
if path.is_file() and resolved not in seen:
files.append(path)
seen.add(resolved)
return [
{
"path": _relative(path, run_dir),
"size_bytes": path.stat().st_size,
"excerpt": _excerpt(path) if path.stat().st_size else "",
}
for path in files
]
def _scan_server_logs(paths: list[Path], run_dir: Path) -> list[dict[str, Any]]:
findings: list[dict[str, Any]] = []
for root in paths:
for path in _expand_log_paths(root):
for line_number, line in _matching_log_lines(path):
findings.append(
{
"path": _relative(path, run_dir),
"line": line_number,
"message": line.strip(),
}
)
return findings
def _expand_log_paths(path: Path) -> list[Path]:
if path.is_file():
return [path]
if path.is_dir():
return sorted(item for item in path.rglob("*") if item.is_file())
return []
def _matching_log_lines(path: Path) -> list[tuple[int, str]]:
matches: list[tuple[int, str]] = []
try:
with path.open("r", encoding="utf-8", errors="replace") as handle:
for index, line in enumerate(handle, start=1):
lowered = line.lower()
if any(term in lowered for term in ERROR_TERMS):
matches.append((index, line))
if len(matches) >= 50:
break
except OSError:
return []
return matches
def _review_status(
*,
hard_errors: list[dict[str, Any]],
unexpected_findings: list[dict[str, Any]],
unaccepted_warnings: list[dict[str, Any]],
unexpected_skips: list[dict[str, Any]],
nonempty_stderr: list[dict[str, Any]],
warning_reviews: list[dict[str, Any]],
skip_reviews: list[dict[str, Any]],
) -> str:
if hard_errors or unexpected_findings or unaccepted_warnings or unexpected_skips or nonempty_stderr:
return "review_required"
if warning_reviews or skip_reviews:
return "pass_with_review_notes"
return "pass"
def _browser_binding_url(target_profile: dict[str, Any], evidence: list[dict[str, Any]]) -> str | None:
for endpoint in target_profile.get("endpoints") or []:
if endpoint.get("binding") == "cmis-browser":
return endpoint.get("url")
for item in evidence:
facts = item.get("facts") or {}
value = facts.get("browser_binding_url") or facts.get("url")
if value:
return str(value)
return None
def _load_policy(policy_path: Path | None) -> dict[str, Any]:
path = policy_path or DEFAULT_POLICY_PATH
if not path.exists():
return {"id": "none", "warning_policies": [], "_path": None}
policy = _load_json(path)
policy["_path"] = path
return policy
def _load_json(path: Path) -> dict[str, Any]:
if not path.exists():
return {}
payload = json.loads(path.read_text(encoding="utf-8"))
return payload if isinstance(payload, dict) else {}
def _is_loopback_host(host: str | None) -> bool:
if not host:
return False
if host in {"localhost"}:
return True
try:
return ip_address(host).is_loopback
except ValueError:
return False
def _excerpt(path: Path, limit: int = 500) -> str:
return path.read_text(encoding="utf-8", errors="replace")[:limit]
def _relative(path: Path, root: Path) -> str:
try:
return str(path.resolve().relative_to(root.resolve()))
except ValueError:
return str(path.resolve())
def _now() -> str:
return datetime.now(timezone.utc).isoformat()

263
tests/test_open_cmis_tck.py
View File

@@ -22,7 +22,9 @@ from guide_board.planning import (
)
from guide_board.retention import build_trend_summary
from guide_board.service import ServiceHandle, start_service
from open_cmis_tck.archive import archive_run
from open_cmis_tck.bootstrap import TCK_COORDINATE, check_runtime
from open_cmis_tck.log_review import build_log_review, write_log_review
from open_cmis_tck.normalization import (
aggregate_case_result,
parse_text_report,
@@ -198,6 +200,169 @@ class OpenCmisTckExtensionTests(unittest.TestCase):
self.assertEqual(warning["source_location"], {"file": "SecurityTest.java", "line": 52})
self.assertEqual(failure["message"], "Test folder could not be created.")
def test_log_review_classifies_loopback_warning_and_closed_warning(self) -> None:
with TemporaryDirectory() as temporary_directory:
temp_root = Path(temporary_directory)
previous_run = temp_root / "previous"
current_run = temp_root / "current"
_write_review_run(
previous_run,
"run-previous",
"http://127.0.0.1:8010/cmis/browser",
["cmis.repository-info", "cmis.object-services"],
[
_opencmis_case(
"repository-type",
"warning",
"WARNING",
"Security Test (BROWSER)",
"HTTPS is not used. Credentials might be transferred as plain text!",
"SecurityTest.java",
67,
),
_opencmis_case(
"object-content",
"warning",
"WARNING",
"Set, Append, and Delete Content Test (BROWSER)",
"appendContentStream() is not supported!",
"SetAndDeleteContentTest.java",
200,
),
],
)
_write_review_run(
current_run,
"run-current",
"http://127.0.0.1:8010/cmis/browser",
["cmis.repository-info", "cmis.object-services"],
[
_opencmis_case(
"repository-type",
"warning",
"WARNING",
"Security Test (BROWSER)",
"HTTPS is not used. Credentials might be transferred as plain text!",
"SecurityTest.java",
67,
),
_opencmis_case(
"object-content",
"skipped",
"SKIPPED",
"Create and Delete Relationship Test (BROWSER)",
"Relationship type 'cmis:relationship' is not creatable!",
"AbstractSessionTest.java",
634,
),
],
)
review = build_log_review(current_run, previous_run_dir=previous_run)
written = write_log_review(current_run, previous_run_dir=previous_run)
self.assertEqual(review["summary"]["status"], "pass_with_review_notes")
self.assertEqual(review["summary"]["accepted_warnings"], 1)
self.assertEqual(review["summary"]["unaccepted_warnings"], 0)
self.assertEqual(review["summary"]["closed_warnings"], 1)
self.assertEqual(review["summary"]["expected_skips"], 1)
self.assertEqual(
review["warnings"][0]["classification"],
"accepted_local_loopback_transport",
)
self.assertEqual(
review["closed_warnings"][0]["message"],
"appendContentStream() is not supported!",
)
self.assertTrue(Path(written["json"]).exists())
self.assertIn(
"OpenCMIS Log Review",
Path(written["markdown"]).read_text(encoding="utf-8"),
)
def test_log_review_flags_non_loopback_http_warning_as_deployment_blocker(self) -> None:
with TemporaryDirectory() as temporary_directory:
run_dir = Path(temporary_directory) / "run"
_write_review_run(
run_dir,
"run-production-http",
"http://cmis.example.test/browser",
["cmis.repository-info"],
[
_opencmis_case(
"repository-type",
"warning",
"WARNING",
"Security Test (BROWSER)",
"HTTPS is not used. Credentials might be transferred as plain text!",
"SecurityTest.java",
67,
)
],
environment="production",
)
review = build_log_review(run_dir)
self.assertEqual(review["summary"]["status"], "review_required")
self.assertEqual(review["summary"]["unaccepted_warnings"], 1)
self.assertEqual(review["warnings"][0]["classification"], "deployment_transport_blocker")
self.assertEqual(review["warnings"][0]["severity"], "blocker")
def test_log_review_marks_advertised_capability_skip_for_review(self) -> None:
with TemporaryDirectory() as temporary_directory:
run_dir = Path(temporary_directory) / "run"
_write_review_run(
run_dir,
"run-relationship-skip",
"http://127.0.0.1:8010/cmis/browser",
["cmis.repository-info", "cmis.relationships"],
[
_opencmis_case(
"object-content",
"skipped",
"SKIPPED",
"Create and Delete Relationship Test (BROWSER)",
"Relationship type 'cmis:relationship' is not creatable!",
"AbstractSessionTest.java",
634,
)
],
)
review = build_log_review(run_dir)
self.assertEqual(review["summary"]["status"], "review_required")
self.assertEqual(review["summary"]["unexpected_skips"], 1)
self.assertFalse(review["skips"][0]["expected"])
self.assertEqual(review["skips"][0]["classification"], "advertised_capability_not_exercised")
def test_archive_run_copies_evidence_and_writes_hash_manifest(self) -> None:
with TemporaryDirectory() as temporary_directory:
temp_root = Path(temporary_directory)
run_dir = temp_root / "run"
_write_review_run(
run_dir,
"run-archive",
"http://127.0.0.1:8010/cmis/browser",
["cmis.repository-info"],
[],
)
(run_dir / "reports" / "report.md").write_text("# Report\n", encoding="utf-8")
manifest = archive_run(run_dir, temp_root / "archive")
archive_dir = Path(manifest["archive_dir"])
manifest_path = archive_dir / "archive-manifest.json"
self.assertTrue((archive_dir / "reports" / "report.md").exists())
self.assertTrue(manifest_path.exists())
self.assertEqual(manifest["run_id"], "run-archive")
self.assertIn(
"normalized/evidence.json",
{item["path"] for item in manifest["files"]},
)
self.assertTrue(all(len(item["sha256"]) == 64 for item in manifest["files"]))
def test_console_adapter_dry_run_writes_session_and_group_files(self) -> None:
with TemporaryDirectory() as temporary_directory:
temp_root = Path(temporary_directory)
@@ -877,6 +1042,104 @@ class OpenCmisTckExtensionTests(unittest.TestCase):
self.assertTrue(findings[1]["expected"])
def _write_review_run(
path: Path,
run_id: str,
browser_url: str,
declared_capabilities: list[str],
cases: list[dict[str, object]],
*,
target_id: str = "kontextual-cmis-compat",
environment: str = "local",
) -> None:
groups = sorted({str(case["selected_check_group"]) for case in cases}) or ["repository-type"]
(path / "normalized").mkdir(parents=True)
(path / "reports").mkdir()
for group in groups:
artifact_dir = path / "artifacts" / "open-cmis-tck" / "tck" / group
artifact_dir.mkdir(parents=True, exist_ok=True)
(artifact_dir / "console-runner-stderr.txt").write_text("", encoding="utf-8")
(artifact_dir / "stderr.log").write_text("", encoding="utf-8")
path.joinpath("run.json").write_text(
json.dumps(
{
"id": run_id,
"target_profile_ref": target_id,
"assessment_profile_ref": "cmis-browser-baseline",
}
),
encoding="utf-8",
)
path.joinpath("target-profile.snapshot.json").write_text(
json.dumps(
{
"id": target_id,
"environment": environment,
"endpoints": [
{
"id": "browser-binding",
"url": browser_url,
"binding": "cmis-browser",
}
],
"declared_capabilities": declared_capabilities,
}
),
encoding="utf-8",
)
path.joinpath("assessment-profile.snapshot.json").write_text(
json.dumps({"id": "cmis-browser-baseline"}),
encoding="utf-8",
)
evidence = []
for group in groups:
group_cases = [case for case in cases if case["selected_check_group"] == group]
evidence.append(
{
"id": f"evidence:check-group:open-cmis-tck:{group}",
"check_id": f"check-group:open-cmis-tck:{group}",
"result": "warning" if any(case["status"] == "warning" for case in group_cases) else "pass",
"facts": {
"selected_check_group": group,
"browser_binding_url": browser_url,
"cases": group_cases,
},
}
)
path.joinpath("normalized", "evidence.json").write_text(
json.dumps({"evidence": evidence}),
encoding="utf-8",
)
path.joinpath("normalized", "findings.json").write_text(
json.dumps({"findings": []}),
encoding="utf-8",
)
def _opencmis_case(
selected_check_group: str,
status: str,
status_native: str,
test_name: str,
message: str,
source_file: str,
source_line: int,
) -> dict[str, object]:
return {
"id": f"opencmis-tck:{selected_check_group}:{test_name.lower().replace(' ', '-')}",
"status": status,
"status_native": status_native,
"selected_check_group": selected_check_group,
"group_name": "OpenCMIS Test Group",
"test_name": test_name,
"message": message,
"source_location": {
"file": source_file,
"line": source_line,
},
}
def _write_target(path: Path, port: int, target_id: str) -> None:
path.write_text(
json.dumps(

4
workplans/OPEN-CMIS-TCK-WP-0002-live-test-infrastructure.md
View File

@@ -5,12 +5,12 @@ title: "Live OpenCMIS TCK Execution And Capability Maturity"
repo: open-cmis-tck
extension: open-cmis-tck
domain: markitect
status: active
status: completed
owner: codex
planning_priority: high
planning_order: 3
created: "2026-05-07"
updated: "2026-05-08"
updated: "2026-05-14"
depends_on:
- "OPEN-CMIS-TCK-WP-0001"
state_hub_workstream_id: "da3f0d16-ba8e-4147-b0fc-ab3462e0b7b0"

326
workplans/OPEN-CMIS-TCK-WP-0003-assessment-log-review-and-hardening.md Normal file
View File

@@ -0,0 +1,326 @@
---
id: OPEN-CMIS-TCK-WP-0003
type: extension-workplan
title: "Assessment Log Review And Hardening"
repo: open-cmis-tck
extension: open-cmis-tck
domain: markitect
status: completed
owner: codex
planning_priority: high
planning_order: 4
created: "2026-05-14"
updated: "2026-05-14"
depends_on:
- "OPEN-CMIS-TCK-WP-0002"
state_hub_workstream_id: "5711ee2f-eaa9-428a-a4b2-e7383bfbf18a"
---
# OPEN-CMIS-TCK-WP-0003: Assessment Log Review And Hardening
## Purpose
Use the first real `kontextual-engine` OpenCMIS TCK assessment runs to harden
the `open-cmis-tck` guide-board extension around warning policy, durable
evidence retention, and repeatable log review.
The latest `kontextual-engine` release-readiness run is healthy for the selected
Browser Binding baseline: no hard TCK failures, no infrastructure errors, no
unexpected findings, and empty stderr artifacts. The remaining work is mostly
facility maturity: make the one current warning intentional, preserve raw
evidence outside ephemeral `/tmp` paths, and give future assessments a compact
"what should we fix next" report instead of relying on manual `rg` passes.
## Evidence Reviewed
- Latest release-readiness evidence:
`/home/worsch/kontextual-engine/docs/cmis-opencmis-tck-release-readiness-evidence-2026-05-13T223537Z.md`
- Latest raw release-readiness run:
`/tmp/kontextual-cmis-release-20260514-toolchain`
- Prior raw run before `appendContentStream()` support:
`/tmp/open-cmis-tck-kontextual-20260513T230205Z`
- Earlier implementation evidence:
`/home/worsch/kontextual-engine/docs/cmis-opencmis-tck-implementation-evidence-2026-05-08T092113Z.md`
- Local extension self-test run:
`/home/worsch/open-cmis-tck/.local/runs/opencmis-inmemory-pilot`
- Local OpenCMIS in-memory server logs:
`/home/worsch/open-cmis-tck/.local/opencmis-inmemory/logs`
## Current Findings
1. The latest `kontextual-engine` run completed with Guide Board summary
`pass: 2`, `warning: 1`, `unexpected_findings: 0`.
2. `console-runner-stderr.txt` and `stderr.log` are empty for both selected TCK
groups in the latest run.
3. The remaining current warning is from OpenCMIS
`SecurityTest.java:67`: `HTTPS is not used. Credentials might be transferred
as plain text!`
4. The previous `appendContentStream()` warning in
`SetAndDeleteContentTest.java:200` is closed in the latest run.
5. The latest object/content run still has skipped cases for non-creatable
relationship, policy, and item types, plus folder-name change-token subcases.
These align with declared capability boundaries and are not errors, but they
should remain visible as maturity scope.
6. The local OpenCMIS in-memory pilot has two repository/type warnings:
loopback HTTP and `Thin client URI is not set!`. Tomcat and in-memory server
logs did not show warning/error/exception lines in the scan.
7. Evidence retention is fragile: several useful raw runs live under `/tmp`, and
at least one earlier `/tmp` run was already unavailable when later evidence
was written.
## Boundary
This workplan hardens the `open-cmis-tck` extension and its assessment
operations. Product changes for `kontextual-engine` belong in that repository.
This workplan may document product-facing follow-up candidates, but it should
not modify the product repo directly.
## D3.1 - Capture Current Log Triage Baseline
```task
id: OPEN-CMIS-TCK-WP-0003-T001
status: done
priority: high
state_hub_task_id: "1a262cad-a945-4a93-a957-02f2fdb497f1"
```
Acceptance:
- Inspect the latest persisted evidence and raw run artifacts for
`kontextual-engine`.
- Separate current findings from older findings that have already been closed.
- Inspect the local in-memory pilot logs so extension self-test warnings are not
confused with product warnings.
- Record the baseline in this workplan.
Progress:
- Confirmed the latest raw release-readiness run has no `fail`,
`infrastructure_error`, unexpected finding, stderr output, or exception trace.
- Confirmed the only current `kontextual-engine` TCK warning is local HTTP
transport.
- Confirmed `appendContentStream()` was a warning in the prior raw run and is
gone in the latest raw run.
- Confirmed the local in-memory pilot still reports loopback HTTP and missing
thin-client URI warnings, while server logs are clean.
## D3.2 - Durable Assessment Archive Path
```task
id: OPEN-CMIS-TCK-WP-0003-T002
status: done
priority: high
state_hub_task_id: "1e31b306-f21e-4bac-8e69-56d586d6712e"
```
Acceptance:
- Provide a recommended non-ephemeral output layout for local product
assessments, for example `.local/runs/<target>/<run-id>` or a configured
workspace archive path.
- Add an operator command or documented copy/import step that preserves raw
TCK stdout/stderr, normalized evidence, findings, mappings, run metadata,
report, scorecard, and artifact manifest before `/tmp` cleanup can remove
them.
- Preserve artifact hashes or package metadata so copied evidence remains
auditable.
- Update the local runbook and service/retention notes with the durable path.
Progress:
- Added `src/open_cmis_tck/archive.py` and
`scripts/archive_assessment_run.py`.
- The archive command copies a run into `.local/runs/archive/<target>/<run-id>`
by default and writes `archive-manifest.json` with SHA-256 hashes, file
sizes, source path, archive path, run ID, target profile reference, and
assessment profile reference.
- Updated README, local runbook, and service/retention docs with the archive
command.
- Archived the latest kontextual release-readiness run to
`.local/runs/archive/kontextual-cmis-compat/run-20260513T223537Z`.
## D3.3 - Warning Policy And HTTPS Deployment Gate
```task
id: OPEN-CMIS-TCK-WP-0003-T003
status: done
priority: high
state_hub_task_id: "58d2cf64-0db6-4f9a-a8d5-df9ef870557b"
```
Acceptance:
- Define how warning policy distinguishes local loopback test topology from a
deployment or release gate.
- Treat the OpenCMIS HTTP warning as acceptable only for explicit local
loopback profiles or documented local waivers.
- Make non-loopback or release-target HTTP warnings visible as deployment
blockers, even when the TCK group return code is `0`.
- Record warning policy in a profile, expectation, or waiver file rather than
burying it in narrative evidence.
Progress:
- Added `profiles/expectations/opencmis-warning-policy.json`.
- Classified the OpenCMIS HTTP warning as accepted only for local/test loopback
HTTP endpoints.
- Non-loopback or production-like HTTP warnings are now classified as
`deployment_transport_blocker` by the log-review command.
## D3.4 - OpenCMIS In-Memory Pilot Warning Cleanup
```task
id: OPEN-CMIS-TCK-WP-0003-T004
status: done
priority: medium
state_hub_task_id: "12331abc-c28f-4140-9a04-a70eb761bccf"
```
Acceptance:
- Investigate whether the local OpenCMIS in-memory server can expose a
`thinClientURI` through configuration.
- If the upstream in-memory server cannot be configured cleanly, mark the
warning as an expected self-test limitation with a precise source location and
explanation.
- Keep the in-memory pilot useful as an extension smoke test without making its
target-specific warnings look like guide-board defects.
- Document the expected warning posture in the local runbook.
Progress:
- Added an explicit policy entry for the `opencmis-inmemory-local`
`Thin client URI is not set!` warning.
- The in-memory pilot review now classifies loopback HTTP and missing
thin-client URI as accepted local self-test warnings.
- Optional external server-log findings are reported as context without
changing the run status by themselves, because those log directories may
include historical startup attempts outside the assessed run.
## D3.5 - Automated Log Review Report
```task
id: OPEN-CMIS-TCK-WP-0003-T005
status: done
priority: high
state_hub_task_id: "e445f909-678b-4236-a025-d5913e5473ed"
```
Acceptance:
- Add a command that scans a guide-board run directory for OpenCMIS stdout,
stderr, normalized results, findings, and known server logs.
- Generate `reports/opencmis-log-review.json` and
`reports/opencmis-log-review.md`.
- Highlight hard errors, non-empty stderr, new warnings, known accepted
warnings, skipped cases, unexpected findings, and closed-warning comparisons
when a previous run is supplied.
- Add regression tests with sanitized fixtures for the current HTTP warning,
the now-closed `appendContentStream()` warning, empty stderr, and skipped
capability-boundary cases.
Progress:
- Added `src/open_cmis_tck/log_review.py` and
`scripts/opencmis_log_review.py`.
- The command writes `reports/opencmis-log-review.json` and
`reports/opencmis-log-review.md`.
- Verified it against `/tmp/kontextual-cmis-release-20260514-toolchain` with
`/tmp/open-cmis-tck-kontextual-20260513T230205Z` as the previous run.
- Added regression coverage for accepted HTTP warnings, non-loopback deployment
blockers, closed append warnings, stderr handling, and skipped capability
boundaries.
## D3.6 - Skip And Capability Boundary Interpretation
```task
id: OPEN-CMIS-TCK-WP-0003-T006
status: done
priority: medium
state_hub_task_id: "6df3e8c5-c2d2-4eb3-973e-76644ef7ee8f"
```
Acceptance:
- Group skipped OpenCMIS cases by declared repository capability or type
creatability boundary.
- Distinguish "expected because the capability is not advertised" from "skipped
because the target could not exercise an advertised capability."
- Keep skipped cases visible in reports and maturity scorecards without treating
them as failures when they match the advertised capability profile.
- Add coverage for relationship, policy, item, type-subtype, and folder-name
change-token skips seen in the latest raw run.
Progress:
- Added skip-boundary classification in the log-review report.
- Current expected skip rules cover relationship, policy, item, document
subtype, and folder-name mutation cases.
- If the target advertises the required capability and OpenCMIS still skips the
case, the review becomes `review_required`.
## D3.7 - Next Coverage Frontier
```task
id: OPEN-CMIS-TCK-WP-0003-T007
status: done
priority: medium
state_hub_task_id: "f793e1b8-a2b1-4f9b-9972-b6b18d1ba56a"
```
Acceptance:
- Identify which additional OpenCMIS TCK groups are realistic after the current
repository/type and object/content baseline.
- For each candidate group, record target preconditions, likely product
capability requirements, and expected unsupported-by-design boundaries.
- Do not expand the default baseline until the warning policy and durable
evidence path are in place.
- Produce a short recommendation for the next maturity slice, likely navigation,
query, ACL/policy, versioning/PWC, or change-log depth.
Progress:
- Added the next-coverage recommendation to `docs/LOG-REVIEW.md`.
- Recommended order is navigation/read-path depth first, metadata query second,
ACL/policy discovery third, and versioning/PWC/change-log only after the
product deliberately advertises those capabilities.
- Left the default baseline unchanged at `repository-type` plus
`object-content`.
## D3.8 - State Hub And Operator Docs
```task
id: OPEN-CMIS-TCK-WP-0003-T008
status: done
priority: medium
state_hub_task_id: "32724628-d427-47c2-ac40-3f3f90e3a2b9"
```
Acceptance:
- Sync this workplan with the state hub.
- Update README/runbook references so operators know how to review warnings
after a run.
- Make it clear that guide-board produces preparation evidence and operational
readiness signals, not certification or audit assurance.
- Ensure doc updates cite the latest raw and persisted evidence baselines.
Progress:
- Added `docs/LOG-REVIEW.md`.
- Updated README, local runbook, and service/retention docs.
- Synced the completed workplan and task statuses into the state hub.
## Definition Of Done
- Future local CMIS assessments keep raw evidence in a durable run location.
- HTTP transport warnings are policy-classified rather than manually explained
after every run.
- The local in-memory pilot has either zero unexpected warnings or a documented
expected-warning profile.
- A log-review report can be generated from any guide-board run directory.
- Skipped OpenCMIS cases are interpreted against advertised CMIS capability
boundaries.
- The next coverage frontier is explicit and does not blur preparation evidence
with formal certification.