8baa2d10ff
Add phase5 daily/closeout ActivityDefinitions, document schedules in Phase 5 evidence, and migrate daily triage sinks to custodian:// runtime URIs.
108 lines
4.6 KiB
Markdown
108 lines
4.6 KiB
Markdown
# T05 Phase 5 Evidence — Stabilization Window (2026-07-06)
|
||
|
||
Workplan: `CUST-WP-0054-T05`
|
||
Parent runbook: `docs/state-hub-railiance01-cutover-plan.md`
|
||
`no_secret_material_recorded: true`
|
||
|
||
## Window
|
||
|
||
| Field | Value |
|
||
| --- | --- |
|
||
| Started | 2026-07-06 ~17:35 UTC |
|
||
| Target close | 2026-07-09 17:35 UTC (72h) |
|
||
| Owner | custodian |
|
||
| Rollback | coulombcore `state-hub` Deployment + dumps retained ≥7 days |
|
||
|
||
**Phase 5 status: in progress.**
|
||
|
||
## Kickoff baseline (2026-07-06T17:41Z)
|
||
|
||
| Check | Result | Evidence |
|
||
| --- | --- | --- |
|
||
| Hub `/state/health` | **pass** | `{"status":"ok","db":"connected"}` via `state-hub-primary` |
|
||
| Row-count gate | **pass** | workstreams **640**, tasks **4002**, topics **14** (matches Phase 2 gate) |
|
||
| railiance01 pod | **pass** | `state-hub-76fdffdc9d-dtvpb` Running, **0** restarts (rev 7 @ `main-375961c`) |
|
||
| CNPG `state-hub-db` | **pass** | `Cluster in healthy state`, 1/1 |
|
||
| coulombcore rollback | **pass** | `state-hub` replicas=**0** (not deleted) |
|
||
| activity-core hub path | **pass** | in-cluster `http://state-hub.state-hub.svc.cluster.local:8000`; bridge **0** |
|
||
| Consistency sweeps | **pass** | 6/6 recent events `exit_code=0`, `skipped_missing=0` (post-17:15 UTC) |
|
||
| Sweep writeback host | **pass** | repos resolve `/home/tegwick/*` (Phase 4) |
|
||
| Workstation tunnel | **pass** | `state-hub-primary` connected + LIVE → railiance01 |
|
||
| Daily triage (post-cutover) | **pass** | manual trigger 2026-07-06T17:50Z; event `a99e644e`, report `daily-triage-2026-07-06-67685f1c.md` |
|
||
| issue-core emission | **unchanged** | `issue-core-coulombcore` tunnel connected; null-sink audit mode (Wave 4) |
|
||
|
||
Kickoff sweep (`source=phase5-kickoff`): **exit_code=0**, 10 repos processed, 0 missing.
|
||
|
||
## Day-1 check (2026-07-06T22:35Z)
|
||
|
||
`tools/phase5-stabilization-check.sh` — all green:
|
||
|
||
| Check | Result |
|
||
| --- | --- |
|
||
| `/state/health` + totals | **pass** — 640 workstreams / 4002 tasks / 14 topics |
|
||
| railiance01 pod | **pass** — `state-hub-76fdffdc9d-dtvpb`, 0 restarts, age ~4h |
|
||
| CNPG | **pass** — healthy 1/1 |
|
||
| coulombcore rollback | **pass** — replicas=0 |
|
||
| activity-core | **pass** — in-cluster URL; bridge replicas=0 |
|
||
| Sweeps (last 6) | **pass** — exit_code=0, proc=11, missing=0 |
|
||
| Daily triage | **pass** — last run 2026-07-06T17:50Z |
|
||
| Tunnels | **pass** — `state-hub-primary` connected + LIVE |
|
||
|
||
Window closes **2026-07-09 ~17:35 UTC** (~65h remaining at check time).
|
||
|
||
## Scheduled checks (activity-core)
|
||
|
||
| Definition | Schedule | Event type |
|
||
| --- | --- | --- |
|
||
| `phase5-stabilization-daily` (`f3a8c2e1…`) | `0 9 * * *` Europe/Berlin | `phase5_stabilization_check` |
|
||
| `phase5-stabilization-closeout` (`e7d2b5a8…`) | once `2026-07-09T17:35Z` | `phase5_stabilization_closeout` |
|
||
|
||
Hub-visible gates: health, totals (640/4002/14), last 6 sweeps, daily triage
|
||
freshness ≤36h. Closeout fails visibly when any gate is red; passes record
|
||
`operator_signoff_needed` for coulombcore teardown approval.
|
||
|
||
## Monitoring
|
||
|
||
Run daily (or after incidents):
|
||
|
||
```bash
|
||
~/the-custodian/tools/phase5-stabilization-check.sh
|
||
```
|
||
|
||
Sweep history (24h):
|
||
|
||
```bash
|
||
cd ~/state-hub
|
||
uv run python scripts/compare_consistency_sweep_parallel.py --since-hours 24
|
||
```
|
||
|
||
## Exit gates (T05 done)
|
||
|
||
From `docs/state-hub-railiance01-cutover-plan.md` verification checklist:
|
||
|
||
- [x] `/state/summary` totals match pre-cutover (±0)
|
||
- [x] activity-core **daily triage** succeeds once post-cutover (manual 2026-07-06T17:50Z)
|
||
- [x] activity-core **consistency sweeps** succeed without `fleet-state-hub-coulombcore`
|
||
- [x] Workstation `:8000` serves railiance01 hub
|
||
- [x] Writeback lands on railiance01 clone path
|
||
- [x] coulombcore hub scaled to 0
|
||
- [x] Cutover progress notes recorded (Phases 0–5)
|
||
|
||
**Close Phase 5 / mark T05 done when:** 72h window passes with no data-loss signals,
|
||
sweeps stay `exit_code=0` (lock_skipped acceptable), one post-cutover daily triage
|
||
succeeds, and operator approves coulombcore hub teardown window.
|
||
|
||
## Known watch items
|
||
|
||
1. **Sweep budget** — 300s processes ~8–10 dirty repos per tick; expect gradual
|
||
reconciliation across 15m cron cycles (not a blocker).
|
||
2. ~~**Image tag**~~ — **resolved 2026-07-06T18:15Z**: Forgejo CI `image.yaml #26`
|
||
published `main-375961c`; Helm rev 7 deployed; post-upgrade sweep `exit_code=0`.
|
||
3. **Stale tunnel** — `state-hub-coulombcore` shows `[STALE]`; safe to retire after
|
||
stabilization if unused.
|
||
4. **issue-core** — still coulombcore-chained; tracked under Wave 4 / T06, not T05.
|
||
|
||
## References
|
||
|
||
- Phase 4: `docs/state-hub-railiance01-cutover-phase4-2026-07-06.md`
|
||
- Sweep runbook: `state-hub/docs/consistency-sweep-runbook.md` |