CUST-WP-0054-T05: State Hub railiance01 cutover plan and task progress

2026-07-06 17:25:09 +02:00
parent 0d890a7155
commit fa715c29d5
2 changed files with 241 additions and 1 deletions
--- a/docs/state-hub-railiance01-cutover-plan.md
+++ b/docs/state-hub-railiance01-cutover-plan.md
@@ -0,0 +1,231 @@
+# State Hub railiance01 Cutover Plan (CUST-WP-0054-T05)
+
+Date: 2026-07-06  
+Workplans: `CUST-WP-0054-T05`, `CUST-WP-0011-T07` (playbook), Wave 2 in `docs/coulombcore-drain-placement-plan.md`  
+`no_secret_material_recorded: true`
+
+## Purpose
+
+Move the **production State Hub primary** from **coulombcore-k3s** to **railiance01-k3s**
+so the automation loop (activity-core, Temporal, daily triage, consistency sweeps)
+is machine-local and survives coulombcore or workstation outages.
+
+This is a **second cutover** in the pragmatic migration arc:
+
+| Cutover | Date | From | To |
+| --- | --- | --- | --- |
+| CUST-WP-0011-T07 | 2026-07-03 | WSL2 | coulombcore cluster |
+| **CUST-WP-0054-T05** | planned | coulombcore cluster | railiance01 cluster |
+
+## Current state (2026-07-06 inventory)
+
+### Production writer
+
+| Item | Value |
+| --- | --- |
+| API workload | coulombcore-k3s, namespace `state-hub` |
+| Database | CNPG `state-hub-db`, namespace `databases` on coulombcore |
+| Image (deployed) | `gitea.coulomb.social/coulomb/state-hub:f2e042a` (Helm handoff pin) |
+| Image (Forgejo CI) | `forgejo.coulomb.social/coulomb/state-hub:f9f0091` — ready for cutover |
+| Hub totals | 637 workstreams, 3983 tasks, 14 topics (`GET /state/summary`) |
+
+### Access paths
+
+| Consumer | Path today | Serves |
+| --- | --- | --- |
+| Workstation agents | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` → coulombcore cluster Svc |
+| activity-core (railiance01) | `actcore-state-hub-bridge` → `127.0.0.1:18000` | systemd `fleet-state-hub-coulombcore` → coulombcore ClusterIP `10.43.170.94:8000` |
+| WSL2 fallback | `make api` (local compose) | Retained; not normal writer |
+
+### railiance01 cluster (target)
+
+| Item | Status |
+| --- | --- |
+| CNPG operator | Healthy (`forgejo-db`, `net-kingdom-pg` in `databases`) |
+| Namespace `state-hub` | **Does not exist** |
+| CNPG `state-hub-db` on railiance01 | **Not deployed** |
+| Hostname for `host_paths` | `239.62.205.92.host.secureserver.net` (not `railiance01`) |
+| Clone tree | Partial (`activity-core`, `llm-connect`, … under `/home/tegwick/`) |
+| Forgejo | Canonical (`forgejo.coulomb.social`) — Wave 1 complete |
+
+### Prerequisites satisfied
+
+- [x] T04 tier-3 production repos on Forgejo (images build in CI)
+- [x] Fleet mesh de-hub (T02) — production does not chain through workstation
+- [x] CUST-WP-0011 deploy handoff (`state-hub/deploy/railiance/`)
+- [ ] **Operator approval** for freeze/final restore (constitution gate)
+- [ ] CNPG + storage sign-off (Wave 2.1)
+
+## Target end state
+
+```
+railiance01-k3s
+  databases/state-hub-db (CNPG, 1 instance)
+  state-hub/state-hub Deployment + ClusterIP :8000
+       ↑
+       ├── activity-core: in-cluster http://state-hub.state-hub.svc:8000
+       │   (retire actcore-state-hub-bridge + fleet-state-hub-coulombcore)
+       ├── consistency sweep: host_paths[239.62.205.92.host.secureserver.net]
+       │   → /home/tegwick/<repo> (Forgejo clones)
+       └── workstation (dev): ops-bridge tunnel → railiance01 cluster Svc
+
+coulombcore: state-hub scaled to 0 / read-only rollback copy (not writer)
+WSL2: fallback only (CUST-WP-0011-T08/T09 retirement folds into T10)
+```
+
+## Safety contract (from CUST-WP-0011)
+
+- coulombcore remains rollback-capable until stabilization passes.
+- No cutover without **exact-count** row comparison after final restore.
+- Failed railiance01 deploy must not destroy coulombcore data.
+- Explicit human approval before freezing the production writer.
+
+## Cutover phases
+
+### Phase 0 — Preflight and baseline (no production impact)
+
+**Owner:** platform + custodian  
+**Done when:** checklist green, baseline row counts recorded.
+
+| # | Action | Command / artifact |
+| --- | --- | --- |
+| 0.1 | Record baseline totals | `curl -s http://127.0.0.1:8000/state/summary` |
+| 0.2 | Record per-table counts | `pg_dump --schema-only` + `SELECT count(*)` on coulombcore CNPG (via port-forward) |
+| 0.3 | Confirm Forgejo image pull | `crictl pull forgejo.coulomb.social/coulomb/state-hub:f9f0091` on railiance01 |
+| 0.4 | Dry-run manifests | `cd ~/state-hub && KUBECONFIG=~/.kube/config-hosteurope make railiance-state-hub-server-dry-run` |
+| 0.5 | Storage review | Confirm `local-path` (or chosen class) has ≥10Gi for `state-hub-db` PVC |
+| 0.6 | Operator approval | `POST /decisions/` or workplan note with approval id |
+
+Baseline totals to match at cutover (2026-07-06):
+
+```text
+workstreams.total: 637
+tasks.total: 3983
+topics.total: 14
+```
+
+Extend with explicit `pg_catalog` table counts for `workplans`, `tasks`, `progress_events`, `managed_repos`, `token_events` (same set as CUST-WP-0011-T07).
+
+### Phase 1 — Deploy empty stack on railiance01
+
+**Owner:** `railiance-platform` + `railiance-apps` promotion from `state-hub/deploy/railiance/`  
+**Done when:** empty hub healthy on railiance01, Alembic at head, **no production data yet**.
+
+| # | Action | Notes |
+| --- | --- | --- |
+| 1.1 | Create `state-hub` namespace | `apps/manifests/state-hub-namespace.yaml` |
+| 1.2 | Deploy `state-hub-db` CNPG | `platform/state-hub-db-cluster.yaml` + credentials Secret |
+| 1.3 | Apply DB NetworkPolicies | `platform/state-hub-db-networkpolicies.yaml` |
+| 1.4 | Create `state-hub-env` Secret | `DATABASE_URL` → `state-hub-db-rw.databases.svc.cluster.local` |
+| 1.5 | Update image to Forgejo | `forgejo.coulomb.social/coulomb/state-hub:f9f0091` in values |
+| 1.6 | Run Alembic Job | Same pattern as CUST-WP-0011-T05 |
+| 1.7 | Helm install `state-hub` | `namespace.create=false`; verify `/state/health` via port-forward |
+
+Promotion path: copy handoff assets into `railiance-platform` / `railiance-apps` repos
+(GitOps) or apply from `state-hub/deploy/railiance/` with operator approval.
+
+### Phase 2 — Data migration (CUST-WP-0011-T07 playbook)
+
+**Owner:** custodian operator  
+**Done when:** railiance01 DB is exact copy; coulombcore API still available for rollback.
+
+| # | Action | Notes |
+| --- | --- | --- |
+| 2.1 | Fresh `pg_dump` from coulombcore `state_hub` | Custom format (`-Fc`); timestamp recorded |
+| 2.2 | **Freeze** coulombcore writer | Scale `state-hub` Deployment to 0 on coulombcore |
+| 2.3 | Final `pg_dump` | Immediately after freeze |
+| 2.4 | Restore into railiance01 `state-hub-db` | `pg_restore` with `SET ROLE state_hub`; scale railiance01 API to 0 during restore |
+| 2.5 | Exact-count comparison | All public tables; mismatch → abort, restore coulombcore |
+| 2.6 | Scale railiance01 API to 1 | Verify `/state/health` + `/state/summary` via port-forward |
+
+### Phase 3 — Rewire access paths
+
+**Owner:** ops-bridge + activity-core  
+**Done when:** all production consumers reach railiance01 hub without coulombcore hop.
+
+| # | Consumer | Before | After |
+| --- | --- | --- | --- |
+| 3.1 | activity-core | bridge → `:18000` → fleet tunnel → coulombcore | In-cluster `http://state-hub.state-hub.svc.cluster.local:8000` (retire bridge Deployment) |
+| 3.2 | railiance01 systemd | `fleet-state-hub-coulombcore` active | **Disable/stop** unit |
+| 3.3 | Workstation | `state-hub-primary` → coulombcore | Repoint tunnel to railiance01 cluster Svc ClusterIP |
+| 3.4 | MCP / agents | `API_BASE=http://127.0.0.1:8000` | Unchanged URL; tunnel backend moves |
+| 3.5 | Smoke write | `POST /progress/` from activity-core | Record event id as cutover proof |
+
+ClusterIP for railiance01 `state-hub` Service must be captured at deploy time for
+tunnel `remote_host` if using SSH local-forward pattern from T02.
+
+### Phase 4 — Sweep checkout migration (Wave 2.3)
+
+**Owner:** custodian + STATE-WP-0064  
+**Done when:** production writebacks use railiance01 paths, not workstation.
+
+| # | Action | Notes |
+| --- | --- | --- |
+| 4.1 | Define clone root | `/home/tegwick/<slug>` (existing partial tree) |
+| 4.2 | Bulk clone from Forgejo | Script over registered repos (`GET /repos/`) |
+| 4.3 | Register `host_paths` | `POST /repos/{slug}/paths` with host=`239.62.205.92.host.secureserver.net` |
+| 4.4 | Run sweep on railiance01 | `POST /consistency/sweep/remote-all` or activity-core cron |
+| 4.5 | Verify git writeback | fix-consistency mutates railiance01 clone, not `/home/worsch/*` |
+
+Workstation `host_paths` entries may remain for dev; production sweep host becomes
+the railiance01 FQDN.
+
+### Phase 5 — Stabilization (48–72h)
+
+**Owner:** custodian  
+**Done when:** no data loss, triage + sweeps + emission green without coulombcore.
+
+Monitor:
+
+- State Hub pod restarts (railiance01)
+- CNPG `state-hub-db` health
+- Daily triage ActivityRun success
+- Consistency sweep `exit_code: 0` every 15m
+- issue-core emission (still via coulombcore tunnel until Wave 4)
+
+coulombcore state-hub remains **stopped, not deleted** until stabilization passes.
+
+## Rollback
+
+| Trigger | Action |
+| --- | --- |
+| Row-count mismatch | Do not rewire; scale coulombcore API back to 1 |
+| railiance01 API unhealthy post-rewire | Re-enable `fleet-state-hub-coulombcore`; repoint workstation tunnel to coulombcore |
+| Data corruption suspected | Restore coulombcore from pre-cutover dump; freeze railiance01 |
+
+Rollback window: keep coulombcore dump and Deployment manifests for ≥7 days.
+
+## Verification checklist (T05 done)
+
+- [ ] `GET /state/summary` on railiance01 matches pre-cutover totals (±0 rows)
+- [ ] activity-core triage + sweep ActivityRuns succeed without `fleet-state-hub-coulombcore`
+- [ ] Workstation `127.0.0.1:8000/state/health` serves railiance01 hub
+- [ ] `fix-consistency` writeback lands on railiance01 clone path
+- [ ] coulombcore hub Deployment scaled to 0 (not deleted)
+- [ ] Progress event recorded with cutover summary + operator approval ref
+
+## Sequencing with other CUST-WP-0054 tasks
+
+| Task | Relationship |
+| --- | --- |
+| T04 | **Prerequisite met** — Forgejo is clone/image source |
+| T06 | Can proceed in parallel; sink paths still reference workstation until T06 |
+| T07 | Dev beachhead independent |
+| T10 | WSL2 fallback retirement + 24h workstation-off proof after T05 stabilizes |
+
+## Open decisions (need operator input before Phase 2)
+
+1. **Maintenance window** — suggested low-traffic window; freeze blocks hub writes ~15–30 min.
+2. **GitOps vs manual apply** — promote `deploy/railiance/` into `railiance-platform`/`railiance-apps` first, or operator apply for speed?
+3. **Sweep clone scope** — all 74 registered repos at cutover, or phased (state-hub + the-custodian + activity-core first)?
+4. **coulombcore rollback retention** — how long to keep stopped Deployment + dump?
+
+## References
+
+- `state-hub/workplans/archived/260706-CUST-WP-0011-state-hub-threephoenix-migration.md` (T07 sequence)
+- `state-hub/docs/cluster-operating-model.md` (current coulombcore reality — update post-cutover)
+- `state-hub/deploy/railiance/README.md`
+- `docs/fleet-mesh-dehub-runbook.md`
+- `docs/forgejo-tier3-remote-url-sweep-playbook.md` (§ Railiance01 sweep paths)
+- `docs/coulombcore-drain-placement-plan.md` (Wave 2)
+- `infra/fleet-mesh/railiance01-tunnels.yaml`
--- a/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md
+++ b/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md
@@ -199,7 +199,7 @@ of remaining registered repos.

 ```task
 id: CUST-WP-0054-T05
-status: todo
+status: progress
 priority: high
 state_hub_task_id: "e91db8d0-973d-4a31-b3c2-ca37fd002ec7"
 ```
@@ -212,6 +212,15 @@ Prereq: railiance01 CNPG + storage reviewed (T03). Also relocate the
 consistency-sweep repo checkouts to railiance01 (clones from the T04 forge)
 so file writebacks no longer touch workstation paths.

+**Planning (2026-07-06):** Cutover runbook at
+`docs/state-hub-railiance01-cutover-plan.md`. Inventory: primary on coulombcore
+(637 ws / 3983 tasks); railiance01 has CNPG operator + Forgejo but **no**
+`state-hub` namespace or `state-hub-db` yet; activity-core reaches hub via
+`fleet-state-hub-coulombcore` → `:18000`. Target hostname for sweep paths:
+`239.62.205.92.host.secureserver.net`. Image for deploy:
+`forgejo.coulomb.social/coulomb/state-hub:f9f0091`. **Blocked on operator
+approval** before Phase 2 freeze/restore.
+
 ## Task: Working-memory and sink path decoupling

 ```task