docs(CUST-WP-0054-T05): open Phase 5 stabilization window

Record kickoff baseline checks, 72h monitoring gates, and a reusable phase5-stabilization-check.sh script for daily health snapshots.
2026-07-06 19:41:42 +02:00
parent c9764ba73c
commit 104c66763e
3 changed files with 145 additions and 1 deletions
--- a/docs/state-hub-railiance01-cutover-phase5-2026-07-06.md
+++ b/docs/state-hub-railiance01-cutover-phase5-2026-07-06.md
@@ -0,0 +1,80 @@
+# T05 Phase 5 Evidence — Stabilization Window (2026-07-06)
+
+Workplan: `CUST-WP-0054-T05`  
+Parent runbook: `docs/state-hub-railiance01-cutover-plan.md`  
+`no_secret_material_recorded: true`
+
+## Window
+
+| Field | Value |
+| --- | --- |
+| Started | 2026-07-06 ~17:35 UTC |
+| Target close | 2026-07-09 17:35 UTC (72h) |
+| Owner | custodian |
+| Rollback | coulombcore `state-hub` Deployment + dumps retained ≥7 days |
+
+**Phase 5 status: in progress.**
+
+## Kickoff baseline (2026-07-06T17:41Z)
+
+| Check | Result | Evidence |
+| --- | --- | --- |
+| Hub `/state/health` | **pass** | `{"status":"ok","db":"connected"}` via `state-hub-primary` |
+| Row-count gate | **pass** | workstreams **640**, tasks **4002**, topics **14** (matches Phase 2 gate) |
+| railiance01 pod | **pass** | `state-hub-67f8b5d5d-9mmd5` Running, **0** restarts |
+| CNPG `state-hub-db` | **pass** | `Cluster in healthy state`, 1/1 |
+| coulombcore rollback | **pass** | `state-hub` replicas=**0** (not deleted) |
+| activity-core hub path | **pass** | in-cluster `http://state-hub.state-hub.svc.cluster.local:8000`; bridge **0** |
+| Consistency sweeps | **pass** | 6/6 recent events `exit_code=0`, `skipped_missing=0` (post-17:15 UTC) |
+| Sweep writeback host | **pass** | repos resolve `/home/tegwick/*` (Phase 4) |
+| Workstation tunnel | **pass** | `state-hub-primary` connected + LIVE → railiance01 |
+| Daily triage (post-cutover) | **pending** | last `daily_triage` 2026-07-06T05:20Z (pre-Phase-3); next cron 07:20 Europe/Berlin |
+| issue-core emission | **unchanged** | `issue-core-coulombcore` tunnel connected; null-sink audit mode (Wave 4) |
+
+Kickoff sweep (`source=phase5-kickoff`): **exit_code=0**, 10 repos processed, 0 missing.
+
+## Monitoring
+
+Run daily (or after incidents):
+
+```bash
+~/the-custodian/tools/phase5-stabilization-check.sh
+```
+
+Sweep history (24h):
+
+```bash
+cd ~/state-hub
+uv run python scripts/compare_consistency_sweep_parallel.py --since-hours 24
+```
+
+## Exit gates (T05 done)
+
+From `docs/state-hub-railiance01-cutover-plan.md` verification checklist:
+
+- [x] `/state/summary` totals match pre-cutover (±0)
+- [ ] activity-core **daily triage** succeeds once post-cutover (watch 2026-07-07 07:20 CEST)
+- [x] activity-core **consistency sweeps** succeed without `fleet-state-hub-coulombcore`
+- [x] Workstation `:8000` serves railiance01 hub
+- [x] Writeback lands on railiance01 clone path
+- [x] coulombcore hub scaled to 0
+- [x] Cutover progress notes recorded (Phases 0–5)
+
+**Close Phase 5 / mark T05 done when:** 72h window passes with no data-loss signals,
+sweeps stay `exit_code=0` (lock_skipped acceptable), one post-cutover daily triage
+succeeds, and operator approves coulombcore hub teardown window.
+
+## Known watch items
+
+1. **Sweep budget** — 300s processes ~8–10 dirty repos per tick; expect gradual
+   reconciliation across 15m cron cycles (not a blocker).
+2. **Image tag** — production uses hand-imported `main-phase4-sweep`; promote via
+   Forgejo CI before next unrelated deploy.
+3. **Stale tunnel** — `state-hub-coulombcore` shows `[STALE]`; safe to retire after
+   stabilization if unused.
+4. **issue-core** — still coulombcore-chained; tracked under Wave 4 / T06, not T05.
+
+## References
+
+- Phase 4: `docs/state-hub-railiance01-cutover-phase4-2026-07-06.md`
+- Sweep runbook: `state-hub/docs/consistency-sweep-runbook.md`
--- a/tools/phase5-stabilization-check.sh
+++ b/tools/phase5-stabilization-check.sh
@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Phase 5 stabilization health snapshot for CUST-WP-0054-T05.
+# Usage: phase5-stabilization-check.sh
+set -euo pipefail
+
+API_BASE="${API_BASE:-http://127.0.0.1:8000}"
+KUBECONFIG_RAILIANCE="${KUBECONFIG_RAILIANCE:-$HOME/.kube/config-hosteurope}"
+KUBECONFIG_COULOMB="${KUBECONFIG_COULOMB:-$HOME/.kube/config}"
+
+echo "=== State Hub API (${API_BASE}) ==="
+curl -fsS "${API_BASE}/state/health" | python3 -m json.tool
+python3 -c "
+import json, urllib.request
+s = json.load(urllib.request.urlopen('${API_BASE}/state/summary'))
+t = s['totals']
+print('totals: workstreams', t['workstreams']['total'], 'tasks', t['tasks']['total'], 'topics', t['topics']['total'])
+"
+
+echo
+echo "=== railiance01 state-hub + CNPG ==="
+KUBECONFIG="${KUBECONFIG_RAILIANCE}" kubectl -n state-hub get pods -l app.kubernetes.io/name=state-hub 2>/dev/null \
+  | awk 'NR==1 || /state-hub-/ {print}'
+KUBECONFIG="${KUBECONFIG_RAILIANCE}" kubectl -n databases get cluster state-hub-db \
+  -o jsonpath='phase={.status.phase} ready={.status.readyInstances}/{.spec.instances}{"\n"}' 2>/dev/null
+
+echo
+echo "=== coulombcore rollback copy ==="
+KUBECONFIG="${KUBECONFIG_COULOMB}" kubectl -n state-hub get deploy state-hub \
+  -o jsonpath='replicas={.spec.replicas}{"\n"}' 2>/dev/null || echo "coulombcore unreachable"
+
+echo
+echo "=== activity-core STATE_HUB_URL ==="
+KUBECONFIG="${KUBECONFIG_RAILIANCE}" kubectl -n activity-core get configmap actcore-runtime-config \
+  -o jsonpath='{.data.STATE_HUB_URL}{"\n"}' 2>/dev/null
+KUBECONFIG="${KUBECONFIG_RAILIANCE}" kubectl -n activity-core get deploy actcore-state-hub-bridge \
+  -o jsonpath='bridge_replicas={.spec.replicas}{"\n"}' 2>/dev/null
+
+echo
+echo "=== last 6 consistency sweeps ==="
+curl -fsS "${API_BASE}/progress/?event_type=consistency_sweep_remote_all&limit=6" | python3 -c "
+import json,sys
+for e in json.load(sys.stdin):
+    d=e.get('detail') or {}
+    print(e['created_at'][:19], 'src='+str(d.get('source','?')),
+          'exit='+str(d.get('exit_code')), 'err='+str(d.get('automation_error')),
+          'proc='+str(len(d.get('repos_processed') or [])),
+          'missing='+str(len(d.get('skipped_missing') or [])))
+"
+
+echo
+echo "=== last daily_triage ==="
+curl -fsS "${API_BASE}/progress/?event_type=daily_triage&limit=1" | python3 -c "
+import json,sys
+ev=json.load(sys.stdin)
+print(ev[0]['created_at'][:19] if ev else 'none', (ev[0].get('summary') or '')[:60] if ev else '')
+" 2>/dev/null || echo "no daily_triage events"
+
+echo
+echo "=== ops-bridge primary tunnel ==="
+bridge status 2>/dev/null | grep -E 'state-hub-primary|issue-core-coulombcore' || echo "bridge status unavailable"
--- a/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md
+++ b/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md
@@ -230,7 +230,11 @@ writer still frozen (rollback).
 **Phase 4 complete** (2026-07-06): 74 repos cloned under `/home/tegwick/`,
 `host_paths[239.62.205.92.host.secureserver.net]` registered, state-hub pod mounts
 clone tree + sweep hostname override, remote-all sweep exit_code=0 —
-`docs/state-hub-railiance01-cutover-phase4-2026-07-06.md`. Phase 5 stabilization next.
+`docs/state-hub-railiance01-cutover-phase4-2026-07-06.md`.
+**Phase 5 started** (2026-07-06): 72h stabilization window open; kickoff baseline
+green (640/4002/14, CNPG healthy, sweeps exit_code=0, coulombcore replicas=0) —
+`docs/state-hub-railiance01-cutover-phase5-2026-07-06.md`. Watch: post-cutover
+daily triage (2026-07-07 07:20 CEST), Forgejo CI image promotion.

 ## Task: Working-memory and sink path decoupling