Add phase5 daily/closeout ActivityDefinitions, document schedules in Phase 5 evidence, and migrate daily triage sinks to custodian:// runtime URIs.
4.6 KiB
T05 Phase 5 Evidence — Stabilization Window (2026-07-06)
Workplan: CUST-WP-0054-T05
Parent runbook: docs/state-hub-railiance01-cutover-plan.md
no_secret_material_recorded: true
Window
| Field | Value |
|---|---|
| Started | 2026-07-06 ~17:35 UTC |
| Target close | 2026-07-09 17:35 UTC (72h) |
| Owner | custodian |
| Rollback | coulombcore state-hub Deployment + dumps retained ≥7 days |
Phase 5 status: in progress.
Kickoff baseline (2026-07-06T17:41Z)
| Check | Result | Evidence |
|---|---|---|
Hub /state/health |
pass | {"status":"ok","db":"connected"} via state-hub-primary |
| Row-count gate | pass | workstreams 640, tasks 4002, topics 14 (matches Phase 2 gate) |
| railiance01 pod | pass | state-hub-76fdffdc9d-dtvpb Running, 0 restarts (rev 7 @ main-375961c) |
CNPG state-hub-db |
pass | Cluster in healthy state, 1/1 |
| coulombcore rollback | pass | state-hub replicas=0 (not deleted) |
| activity-core hub path | pass | in-cluster http://state-hub.state-hub.svc.cluster.local:8000; bridge 0 |
| Consistency sweeps | pass | 6/6 recent events exit_code=0, skipped_missing=0 (post-17:15 UTC) |
| Sweep writeback host | pass | repos resolve /home/tegwick/* (Phase 4) |
| Workstation tunnel | pass | state-hub-primary connected + LIVE → railiance01 |
| Daily triage (post-cutover) | pass | manual trigger 2026-07-06T17:50Z; event a99e644e, report daily-triage-2026-07-06-67685f1c.md |
| issue-core emission | unchanged | issue-core-coulombcore tunnel connected; null-sink audit mode (Wave 4) |
Kickoff sweep (source=phase5-kickoff): exit_code=0, 10 repos processed, 0 missing.
Day-1 check (2026-07-06T22:35Z)
tools/phase5-stabilization-check.sh — all green:
| Check | Result |
|---|---|
/state/health + totals |
pass — 640 workstreams / 4002 tasks / 14 topics |
| railiance01 pod | pass — state-hub-76fdffdc9d-dtvpb, 0 restarts, age ~4h |
| CNPG | pass — healthy 1/1 |
| coulombcore rollback | pass — replicas=0 |
| activity-core | pass — in-cluster URL; bridge replicas=0 |
| Sweeps (last 6) | pass — exit_code=0, proc=11, missing=0 |
| Daily triage | pass — last run 2026-07-06T17:50Z |
| Tunnels | pass — state-hub-primary connected + LIVE |
Window closes 2026-07-09 ~17:35 UTC (~65h remaining at check time).
Scheduled checks (activity-core)
| Definition | Schedule | Event type |
|---|---|---|
phase5-stabilization-daily (f3a8c2e1…) |
0 9 * * * Europe/Berlin |
phase5_stabilization_check |
phase5-stabilization-closeout (e7d2b5a8…) |
once 2026-07-09T17:35Z |
phase5_stabilization_closeout |
Hub-visible gates: health, totals (640/4002/14), last 6 sweeps, daily triage
freshness ≤36h. Closeout fails visibly when any gate is red; passes record
operator_signoff_needed for coulombcore teardown approval.
Monitoring
Run daily (or after incidents):
~/the-custodian/tools/phase5-stabilization-check.sh
Sweep history (24h):
cd ~/state-hub
uv run python scripts/compare_consistency_sweep_parallel.py --since-hours 24
Exit gates (T05 done)
From docs/state-hub-railiance01-cutover-plan.md verification checklist:
/state/summarytotals match pre-cutover (±0)- activity-core daily triage succeeds once post-cutover (manual 2026-07-06T17:50Z)
- activity-core consistency sweeps succeed without
fleet-state-hub-coulombcore - Workstation
:8000serves railiance01 hub - Writeback lands on railiance01 clone path
- coulombcore hub scaled to 0
- Cutover progress notes recorded (Phases 0–5)
Close Phase 5 / mark T05 done when: 72h window passes with no data-loss signals,
sweeps stay exit_code=0 (lock_skipped acceptable), one post-cutover daily triage
succeeds, and operator approves coulombcore hub teardown window.
Known watch items
- Sweep budget — 300s processes ~8–10 dirty repos per tick; expect gradual reconciliation across 15m cron cycles (not a blocker).
Image tag— resolved 2026-07-06T18:15Z: Forgejo CIimage.yaml #26publishedmain-375961c; Helm rev 7 deployed; post-upgrade sweepexit_code=0.- Stale tunnel —
state-hub-coulombcoreshows[STALE]; safe to retire after stabilization if unused. - issue-core — still coulombcore-chained; tracked under Wave 4 / T06, not T05.
References
- Phase 4:
docs/state-hub-railiance01-cutover-phase4-2026-07-06.md - Sweep runbook:
state-hub/docs/consistency-sweep-runbook.md