From 104c66763e8368c264b9cc8323242e2fed1575b7 Mon Sep 17 00:00:00 2001 From: codex Date: Mon, 6 Jul 2026 19:41:42 +0200 Subject: [PATCH] docs(CUST-WP-0054-T05): open Phase 5 stabilization window Record kickoff baseline checks, 72h monitoring gates, and a reusable phase5-stabilization-check.sh script for daily health snapshots. --- ...b-railiance01-cutover-phase5-2026-07-06.md | 80 +++++++++++++++++++ tools/phase5-stabilization-check.sh | 60 ++++++++++++++ ...tion-independence-and-fleet-realignment.md | 6 +- 3 files changed, 145 insertions(+), 1 deletion(-) create mode 100644 docs/state-hub-railiance01-cutover-phase5-2026-07-06.md create mode 100755 tools/phase5-stabilization-check.sh diff --git a/docs/state-hub-railiance01-cutover-phase5-2026-07-06.md b/docs/state-hub-railiance01-cutover-phase5-2026-07-06.md new file mode 100644 index 0000000..f5c2d42 --- /dev/null +++ b/docs/state-hub-railiance01-cutover-phase5-2026-07-06.md @@ -0,0 +1,80 @@ +# T05 Phase 5 Evidence — Stabilization Window (2026-07-06) + +Workplan: `CUST-WP-0054-T05` +Parent runbook: `docs/state-hub-railiance01-cutover-plan.md` +`no_secret_material_recorded: true` + +## Window + +| Field | Value | +| --- | --- | +| Started | 2026-07-06 ~17:35 UTC | +| Target close | 2026-07-09 17:35 UTC (72h) | +| Owner | custodian | +| Rollback | coulombcore `state-hub` Deployment + dumps retained ≥7 days | + +**Phase 5 status: in progress.** + +## Kickoff baseline (2026-07-06T17:41Z) + +| Check | Result | Evidence | +| --- | --- | --- | +| Hub `/state/health` | **pass** | `{"status":"ok","db":"connected"}` via `state-hub-primary` | +| Row-count gate | **pass** | workstreams **640**, tasks **4002**, topics **14** (matches Phase 2 gate) | +| railiance01 pod | **pass** | `state-hub-67f8b5d5d-9mmd5` Running, **0** restarts | +| CNPG `state-hub-db` | **pass** | `Cluster in healthy state`, 1/1 | +| coulombcore rollback | **pass** | `state-hub` replicas=**0** (not deleted) | +| activity-core hub path | **pass** | in-cluster `http://state-hub.state-hub.svc.cluster.local:8000`; bridge **0** | +| Consistency sweeps | **pass** | 6/6 recent events `exit_code=0`, `skipped_missing=0` (post-17:15 UTC) | +| Sweep writeback host | **pass** | repos resolve `/home/tegwick/*` (Phase 4) | +| Workstation tunnel | **pass** | `state-hub-primary` connected + LIVE → railiance01 | +| Daily triage (post-cutover) | **pending** | last `daily_triage` 2026-07-06T05:20Z (pre-Phase-3); next cron 07:20 Europe/Berlin | +| issue-core emission | **unchanged** | `issue-core-coulombcore` tunnel connected; null-sink audit mode (Wave 4) | + +Kickoff sweep (`source=phase5-kickoff`): **exit_code=0**, 10 repos processed, 0 missing. + +## Monitoring + +Run daily (or after incidents): + +```bash +~/the-custodian/tools/phase5-stabilization-check.sh +``` + +Sweep history (24h): + +```bash +cd ~/state-hub +uv run python scripts/compare_consistency_sweep_parallel.py --since-hours 24 +``` + +## Exit gates (T05 done) + +From `docs/state-hub-railiance01-cutover-plan.md` verification checklist: + +- [x] `/state/summary` totals match pre-cutover (±0) +- [ ] activity-core **daily triage** succeeds once post-cutover (watch 2026-07-07 07:20 CEST) +- [x] activity-core **consistency sweeps** succeed without `fleet-state-hub-coulombcore` +- [x] Workstation `:8000` serves railiance01 hub +- [x] Writeback lands on railiance01 clone path +- [x] coulombcore hub scaled to 0 +- [x] Cutover progress notes recorded (Phases 0–5) + +**Close Phase 5 / mark T05 done when:** 72h window passes with no data-loss signals, +sweeps stay `exit_code=0` (lock_skipped acceptable), one post-cutover daily triage +succeeds, and operator approves coulombcore hub teardown window. + +## Known watch items + +1. **Sweep budget** — 300s processes ~8–10 dirty repos per tick; expect gradual + reconciliation across 15m cron cycles (not a blocker). +2. **Image tag** — production uses hand-imported `main-phase4-sweep`; promote via + Forgejo CI before next unrelated deploy. +3. **Stale tunnel** — `state-hub-coulombcore` shows `[STALE]`; safe to retire after + stabilization if unused. +4. **issue-core** — still coulombcore-chained; tracked under Wave 4 / T06, not T05. + +## References + +- Phase 4: `docs/state-hub-railiance01-cutover-phase4-2026-07-06.md` +- Sweep runbook: `state-hub/docs/consistency-sweep-runbook.md` \ No newline at end of file diff --git a/tools/phase5-stabilization-check.sh b/tools/phase5-stabilization-check.sh new file mode 100755 index 0000000..71db2bb --- /dev/null +++ b/tools/phase5-stabilization-check.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Phase 5 stabilization health snapshot for CUST-WP-0054-T05. +# Usage: phase5-stabilization-check.sh +set -euo pipefail + +API_BASE="${API_BASE:-http://127.0.0.1:8000}" +KUBECONFIG_RAILIANCE="${KUBECONFIG_RAILIANCE:-$HOME/.kube/config-hosteurope}" +KUBECONFIG_COULOMB="${KUBECONFIG_COULOMB:-$HOME/.kube/config}" + +echo "=== State Hub API (${API_BASE}) ===" +curl -fsS "${API_BASE}/state/health" | python3 -m json.tool +python3 -c " +import json, urllib.request +s = json.load(urllib.request.urlopen('${API_BASE}/state/summary')) +t = s['totals'] +print('totals: workstreams', t['workstreams']['total'], 'tasks', t['tasks']['total'], 'topics', t['topics']['total']) +" + +echo +echo "=== railiance01 state-hub + CNPG ===" +KUBECONFIG="${KUBECONFIG_RAILIANCE}" kubectl -n state-hub get pods -l app.kubernetes.io/name=state-hub 2>/dev/null \ + | awk 'NR==1 || /state-hub-/ {print}' +KUBECONFIG="${KUBECONFIG_RAILIANCE}" kubectl -n databases get cluster state-hub-db \ + -o jsonpath='phase={.status.phase} ready={.status.readyInstances}/{.spec.instances}{"\n"}' 2>/dev/null + +echo +echo "=== coulombcore rollback copy ===" +KUBECONFIG="${KUBECONFIG_COULOMB}" kubectl -n state-hub get deploy state-hub \ + -o jsonpath='replicas={.spec.replicas}{"\n"}' 2>/dev/null || echo "coulombcore unreachable" + +echo +echo "=== activity-core STATE_HUB_URL ===" +KUBECONFIG="${KUBECONFIG_RAILIANCE}" kubectl -n activity-core get configmap actcore-runtime-config \ + -o jsonpath='{.data.STATE_HUB_URL}{"\n"}' 2>/dev/null +KUBECONFIG="${KUBECONFIG_RAILIANCE}" kubectl -n activity-core get deploy actcore-state-hub-bridge \ + -o jsonpath='bridge_replicas={.spec.replicas}{"\n"}' 2>/dev/null + +echo +echo "=== last 6 consistency sweeps ===" +curl -fsS "${API_BASE}/progress/?event_type=consistency_sweep_remote_all&limit=6" | python3 -c " +import json,sys +for e in json.load(sys.stdin): + d=e.get('detail') or {} + print(e['created_at'][:19], 'src='+str(d.get('source','?')), + 'exit='+str(d.get('exit_code')), 'err='+str(d.get('automation_error')), + 'proc='+str(len(d.get('repos_processed') or [])), + 'missing='+str(len(d.get('skipped_missing') or []))) +" + +echo +echo "=== last daily_triage ===" +curl -fsS "${API_BASE}/progress/?event_type=daily_triage&limit=1" | python3 -c " +import json,sys +ev=json.load(sys.stdin) +print(ev[0]['created_at'][:19] if ev else 'none', (ev[0].get('summary') or '')[:60] if ev else '') +" 2>/dev/null || echo "no daily_triage events" + +echo +echo "=== ops-bridge primary tunnel ===" +bridge status 2>/dev/null | grep -E 'state-hub-primary|issue-core-coulombcore' || echo "bridge status unavailable" \ No newline at end of file diff --git a/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md b/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md index 9fa12eb..606ace9 100644 --- a/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md +++ b/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md @@ -230,7 +230,11 @@ writer still frozen (rollback). **Phase 4 complete** (2026-07-06): 74 repos cloned under `/home/tegwick/`, `host_paths[239.62.205.92.host.secureserver.net]` registered, state-hub pod mounts clone tree + sweep hostname override, remote-all sweep exit_code=0 — -`docs/state-hub-railiance01-cutover-phase4-2026-07-06.md`. Phase 5 stabilization next. +`docs/state-hub-railiance01-cutover-phase4-2026-07-06.md`. +**Phase 5 started** (2026-07-06): 72h stabilization window open; kickoff baseline +green (640/4002/14, CNPG healthy, sweeps exit_code=0, coulombcore replicas=0) — +`docs/state-hub-railiance01-cutover-phase5-2026-07-06.md`. Watch: post-cutover +daily triage (2026-07-07 07:20 CEST), Forgejo CI image promotion. ## Task: Working-memory and sink path decoupling