# State Hub railiance01 Cutover Plan (CUST-WP-0054-T05)

Date: 2026-07-06  
Workplans: `CUST-WP-0054-T05`, `CUST-WP-0011-T07` (playbook), Wave 2 in `docs/coulombcore-drain-placement-plan.md`  
`no_secret_material_recorded: true`

## Purpose

Move the **production State Hub primary** from **coulombcore-k3s** to **railiance01-k3s**
so the automation loop (activity-core, Temporal, daily triage, consistency sweeps)
is machine-local and survives coulombcore or workstation outages.

This is a **second cutover** in the pragmatic migration arc:

| Cutover | Date | From | To |
| --- | --- | --- | --- |
| CUST-WP-0011-T07 | 2026-07-03 | WSL2 | coulombcore cluster |
| **CUST-WP-0054-T05** | planned | coulombcore cluster | railiance01 cluster |

## Current state (2026-07-06 inventory)

### Production writer

| Item | Value |
| --- | --- |
| API workload | coulombcore-k3s, namespace `state-hub` |
| Database | CNPG `state-hub-db`, namespace `databases` on coulombcore |
| Image (deployed) | `gitea.coulomb.social/coulomb/state-hub:f2e042a` (Helm handoff pin) |
| Image (Forgejo CI) | `forgejo.coulomb.social/coulomb/state-hub:f9f0091` — ready for cutover |
| Hub totals | 637 workstreams, 3983 tasks, 14 topics (`GET /state/summary`) |

### Access paths

| Consumer | Path today | Serves |
| --- | --- | --- |
| Workstation agents | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` → coulombcore cluster Svc |
| activity-core (railiance01) | `actcore-state-hub-bridge` → `127.0.0.1:18000` | systemd `fleet-state-hub-coulombcore` → coulombcore ClusterIP `10.43.170.94:8000` |
| WSL2 fallback | `make api` (local compose) | Retained; not normal writer |

### railiance01 cluster (target)

| Item | Status |
| --- | --- |
| CNPG operator | Healthy (`forgejo-db`, `net-kingdom-pg` in `databases`) |
| Namespace `state-hub` | **Does not exist** |
| CNPG `state-hub-db` on railiance01 | **Not deployed** |
| Hostname for `host_paths` | `239.62.205.92.host.secureserver.net` (not `railiance01`) |
| Clone tree | Partial (`activity-core`, `llm-connect`, … under `/home/tegwick/`) |
| Forgejo | Canonical (`forgejo.coulomb.social`) — Wave 1 complete |

### Prerequisites satisfied

- [x] T04 tier-3 production repos on Forgejo (images build in CI)
- [x] Fleet mesh de-hub (T02) — production does not chain through workstation
- [x] CUST-WP-0011 deploy handoff (`state-hub/deploy/railiance/`)
- [x] **Phase 0 preflight** (2026-07-06) — `docs/state-hub-railiance01-cutover-preflight-2026-07-06.md`
- [x] CNPG + storage sign-off (Wave 2.1) — `local-path`, 75G free, operator healthy
- [ ] **Operator approval** for freeze/final restore (constitution gate)

## Target end state

```
railiance01-k3s
  databases/state-hub-db (CNPG, 1 instance)
  state-hub/state-hub Deployment + ClusterIP :8000
       ↑
       ├── activity-core: in-cluster http://state-hub.state-hub.svc:8000
       │   (retire actcore-state-hub-bridge + fleet-state-hub-coulombcore)
       ├── consistency sweep: host_paths[239.62.205.92.host.secureserver.net]
       │   → /home/tegwick/<repo> (Forgejo clones)
       └── workstation (dev): ops-bridge tunnel → railiance01 cluster Svc

coulombcore: state-hub scaled to 0 / read-only rollback copy (not writer)
WSL2: fallback only (CUST-WP-0011-T08/T09 retirement folds into T10)
```

## Safety contract (from CUST-WP-0011)

- coulombcore remains rollback-capable until stabilization passes.
- No cutover without **exact-count** row comparison after final restore.
- Failed railiance01 deploy must not destroy coulombcore data.
- Explicit human approval before freezing the production writer.

## Cutover phases

### Phase 0 — Preflight and baseline (no production impact)

**Owner:** platform + custodian  
**Done when:** checklist green, baseline row counts recorded.

| # | Action | Command / artifact |
| --- | --- | --- |
| 0.1 | Record baseline totals | `curl -s http://127.0.0.1:8000/state/summary` |
| 0.2 | Record per-table counts | `pg_dump --schema-only` + `SELECT count(*)` on coulombcore CNPG (via port-forward) |
| 0.3 | Confirm Forgejo image pull | `crictl pull forgejo.coulomb.social/coulomb/state-hub:f9f0091` on railiance01 |
| 0.4 | Dry-run manifests | `cd ~/state-hub && KUBECONFIG=~/.kube/config-hosteurope make railiance-state-hub-server-dry-run` |
| 0.5 | Storage review | Confirm `local-path` (or chosen class) has ≥10Gi for `state-hub-db` PVC |
| 0.6 | Operator approval | `POST /decisions/` or workplan note with approval id |

**Recorded 2026-07-06 (Phase 0 complete):** see
`docs/state-hub-railiance01-cutover-preflight-2026-07-06.md`.

```text
workplans: 637        tasks: 3983         topics: 14
progress_events: 8717 token_events: 1936   managed_repos: 74
decisions: 79
```

Post-restore counts on railiance01 must match exactly.

### Phase 1 — Deploy empty stack on railiance01

**Owner:** `railiance-platform` + `railiance-apps` promotion from `state-hub/deploy/railiance/`  
**Done when:** empty hub healthy on railiance01, Alembic at head, **no production data yet**.

| # | Action | Notes |
| --- | --- | --- |
| 1.1 | Create `state-hub` namespace | `apps/manifests/state-hub-namespace.yaml` |
| 1.2 | Deploy `state-hub-db` CNPG | `platform/state-hub-db-cluster.yaml` + credentials Secret |
| 1.3 | Apply DB NetworkPolicies | `platform/state-hub-db-networkpolicies.yaml` |
| 1.4 | Create `state-hub-env` Secret | `DATABASE_URL` → `state-hub-db-rw.databases.svc.cluster.local` |
| 1.5 | Update image to Forgejo | `forgejo.coulomb.social/coulomb/state-hub:f9f0091` in values |
| 1.6 | Run Alembic Job | Same pattern as CUST-WP-0011-T05 |
| 1.7 | Helm install `state-hub` | `namespace.create=false`; verify `/state/health` via port-forward |

Promotion path: copy handoff assets into `railiance-platform` / `railiance-apps` repos
(GitOps) or apply from `state-hub/deploy/railiance/` with operator approval.

### Phase 2 — Data migration (CUST-WP-0011-T07 playbook)

**Owner:** custodian operator  
**Done when:** railiance01 DB is exact copy; coulombcore API still available for rollback.

| # | Action | Notes |
| --- | --- | --- |
| 2.1 | Fresh `pg_dump` from coulombcore `state_hub` | Custom format (`-Fc`); timestamp recorded |
| 2.2 | **Freeze** coulombcore writer | Scale `state-hub` Deployment to 0 on coulombcore |
| 2.3 | Final `pg_dump` | Immediately after freeze |
| 2.4 | Restore into railiance01 `state-hub-db` | `pg_restore` with `SET ROLE state_hub`; scale railiance01 API to 0 during restore |
| 2.5 | Exact-count comparison | All public tables; mismatch → abort, restore coulombcore |
| 2.6 | Scale railiance01 API to 1 | Verify `/state/health` + `/state/summary` via port-forward |

### Phase 3 — Rewire access paths

**Owner:** ops-bridge + activity-core  
**Done when:** all production consumers reach railiance01 hub without coulombcore hop.

| # | Consumer | Before | After |
| --- | --- | --- | --- |
| 3.1 | activity-core | bridge → `:18000` → fleet tunnel → coulombcore | In-cluster `http://state-hub.state-hub.svc.cluster.local:8000` (retire bridge Deployment) |
| 3.2 | railiance01 systemd | `fleet-state-hub-coulombcore` active | **Disable/stop** unit |
| 3.3 | Workstation | `state-hub-primary` → coulombcore | Repoint tunnel to railiance01 cluster Svc ClusterIP |
| 3.4 | MCP / agents | `API_BASE=http://127.0.0.1:8000` | Unchanged URL; tunnel backend moves |
| 3.5 | Smoke write | `POST /progress/` from activity-core | Record event id as cutover proof |

ClusterIP for railiance01 `state-hub` Service must be captured at deploy time for
tunnel `remote_host` if using SSH local-forward pattern from T02.

### Phase 4 — Sweep checkout migration (Wave 2.3)

**Owner:** custodian + STATE-WP-0064  
**Done when:** production writebacks use railiance01 paths, not workstation.

| # | Action | Notes |
| --- | --- | --- |
| 4.1 | Define clone root | `/home/tegwick/<slug>` (existing partial tree) |
| 4.2 | Bulk clone from Forgejo | Script over registered repos (`GET /repos/`) |
| 4.3 | Register `host_paths` | `POST /repos/{slug}/paths` with host=`239.62.205.92.host.secureserver.net` |
| 4.4 | Run sweep on railiance01 | `POST /consistency/sweep/remote-all` or activity-core cron |
| 4.5 | Verify git writeback | fix-consistency mutates railiance01 clone, not `/home/worsch/*` |

Workstation `host_paths` entries may remain for dev; production sweep host becomes
the railiance01 FQDN.

### Phase 5 — Stabilization (48–72h)

**Owner:** custodian  
**Done when:** no data loss, triage + sweeps + emission green without coulombcore.

Monitor:

- State Hub pod restarts (railiance01)
- CNPG `state-hub-db` health
- Daily triage ActivityRun success
- Consistency sweep `exit_code: 0` every 15m
- issue-core emission (still via coulombcore tunnel until Wave 4)

coulombcore state-hub remains **stopped, not deleted** until stabilization passes.

## Rollback

| Trigger | Action |
| --- | --- |
| Row-count mismatch | Do not rewire; scale coulombcore API back to 1 |
| railiance01 API unhealthy post-rewire | Re-enable `fleet-state-hub-coulombcore`; repoint workstation tunnel to coulombcore |
| Data corruption suspected | Restore coulombcore from pre-cutover dump; freeze railiance01 |

Rollback window: keep coulombcore dump and Deployment manifests for ≥7 days.

## Verification checklist (T05 done)

- [ ] `GET /state/summary` on railiance01 matches pre-cutover totals (±0 rows)
- [ ] activity-core triage + sweep ActivityRuns succeed without `fleet-state-hub-coulombcore`
- [ ] Workstation `127.0.0.1:8000/state/health` serves railiance01 hub
- [ ] `fix-consistency` writeback lands on railiance01 clone path
- [ ] coulombcore hub Deployment scaled to 0 (not deleted)
- [ ] Progress event recorded with cutover summary + operator approval ref

## Sequencing with other CUST-WP-0054 tasks

| Task | Relationship |
| --- | --- |
| T04 | **Prerequisite met** — Forgejo is clone/image source |
| T06 | Can proceed in parallel; sink paths still reference workstation until T06 |
| T07 | Dev beachhead independent |
| T10 | WSL2 fallback retirement + 24h workstation-off proof after T05 stabilizes |

## Open decisions (need operator input before Phase 2)

1. **Maintenance window** — suggested low-traffic window; freeze blocks hub writes ~15–30 min.
2. **GitOps vs manual apply** — promote `deploy/railiance/` into `railiance-platform`/`railiance-apps` first, or operator apply for speed?
3. **Sweep clone scope** — all 74 registered repos at cutover, or phased (state-hub + the-custodian + activity-core first)?
4. **coulombcore rollback retention** — how long to keep stopped Deployment + dump?

## References

- `state-hub/workplans/archived/260706-CUST-WP-0011-state-hub-threephoenix-migration.md` (T07 sequence)
- `state-hub/docs/cluster-operating-model.md` (current coulombcore reality — update post-cutover)
- `state-hub/deploy/railiance/README.md`
- `docs/fleet-mesh-dehub-runbook.md`
- `docs/forgejo-tier3-remote-url-sweep-playbook.md` (§ Railiance01 sweep paths)
- `docs/coulombcore-drain-placement-plan.md` (Wave 2)
- `infra/fleet-mesh/railiance01-tunnels.yaml`