104c66763e
Record kickoff baseline checks, 72h monitoring gates, and a reusable phase5-stabilization-check.sh script for daily health snapshots.
80 lines
3.3 KiB
Markdown
80 lines
3.3 KiB
Markdown
# T05 Phase 5 Evidence — Stabilization Window (2026-07-06)
|
||
|
||
Workplan: `CUST-WP-0054-T05`
|
||
Parent runbook: `docs/state-hub-railiance01-cutover-plan.md`
|
||
`no_secret_material_recorded: true`
|
||
|
||
## Window
|
||
|
||
| Field | Value |
|
||
| --- | --- |
|
||
| Started | 2026-07-06 ~17:35 UTC |
|
||
| Target close | 2026-07-09 17:35 UTC (72h) |
|
||
| Owner | custodian |
|
||
| Rollback | coulombcore `state-hub` Deployment + dumps retained ≥7 days |
|
||
|
||
**Phase 5 status: in progress.**
|
||
|
||
## Kickoff baseline (2026-07-06T17:41Z)
|
||
|
||
| Check | Result | Evidence |
|
||
| --- | --- | --- |
|
||
| Hub `/state/health` | **pass** | `{"status":"ok","db":"connected"}` via `state-hub-primary` |
|
||
| Row-count gate | **pass** | workstreams **640**, tasks **4002**, topics **14** (matches Phase 2 gate) |
|
||
| railiance01 pod | **pass** | `state-hub-67f8b5d5d-9mmd5` Running, **0** restarts |
|
||
| CNPG `state-hub-db` | **pass** | `Cluster in healthy state`, 1/1 |
|
||
| coulombcore rollback | **pass** | `state-hub` replicas=**0** (not deleted) |
|
||
| activity-core hub path | **pass** | in-cluster `http://state-hub.state-hub.svc.cluster.local:8000`; bridge **0** |
|
||
| Consistency sweeps | **pass** | 6/6 recent events `exit_code=0`, `skipped_missing=0` (post-17:15 UTC) |
|
||
| Sweep writeback host | **pass** | repos resolve `/home/tegwick/*` (Phase 4) |
|
||
| Workstation tunnel | **pass** | `state-hub-primary` connected + LIVE → railiance01 |
|
||
| Daily triage (post-cutover) | **pending** | last `daily_triage` 2026-07-06T05:20Z (pre-Phase-3); next cron 07:20 Europe/Berlin |
|
||
| issue-core emission | **unchanged** | `issue-core-coulombcore` tunnel connected; null-sink audit mode (Wave 4) |
|
||
|
||
Kickoff sweep (`source=phase5-kickoff`): **exit_code=0**, 10 repos processed, 0 missing.
|
||
|
||
## Monitoring
|
||
|
||
Run daily (or after incidents):
|
||
|
||
```bash
|
||
~/the-custodian/tools/phase5-stabilization-check.sh
|
||
```
|
||
|
||
Sweep history (24h):
|
||
|
||
```bash
|
||
cd ~/state-hub
|
||
uv run python scripts/compare_consistency_sweep_parallel.py --since-hours 24
|
||
```
|
||
|
||
## Exit gates (T05 done)
|
||
|
||
From `docs/state-hub-railiance01-cutover-plan.md` verification checklist:
|
||
|
||
- [x] `/state/summary` totals match pre-cutover (±0)
|
||
- [ ] activity-core **daily triage** succeeds once post-cutover (watch 2026-07-07 07:20 CEST)
|
||
- [x] activity-core **consistency sweeps** succeed without `fleet-state-hub-coulombcore`
|
||
- [x] Workstation `:8000` serves railiance01 hub
|
||
- [x] Writeback lands on railiance01 clone path
|
||
- [x] coulombcore hub scaled to 0
|
||
- [x] Cutover progress notes recorded (Phases 0–5)
|
||
|
||
**Close Phase 5 / mark T05 done when:** 72h window passes with no data-loss signals,
|
||
sweeps stay `exit_code=0` (lock_skipped acceptable), one post-cutover daily triage
|
||
succeeds, and operator approves coulombcore hub teardown window.
|
||
|
||
## Known watch items
|
||
|
||
1. **Sweep budget** — 300s processes ~8–10 dirty repos per tick; expect gradual
|
||
reconciliation across 15m cron cycles (not a blocker).
|
||
2. **Image tag** — production uses hand-imported `main-phase4-sweep`; promote via
|
||
Forgejo CI before next unrelated deploy.
|
||
3. **Stale tunnel** — `state-hub-coulombcore` shows `[STALE]`; safe to retire after
|
||
stabilization if unused.
|
||
4. **issue-core** — still coulombcore-chained; tracked under Wave 4 / T06, not T05.
|
||
|
||
## References
|
||
|
||
- Phase 4: `docs/state-hub-railiance01-cutover-phase4-2026-07-06.md`
|
||
- Sweep runbook: `state-hub/docs/consistency-sweep-runbook.md` |