11 KiB
11 KiB
State Hub railiance01 Cutover Plan (CUST-WP-0054-T05)
Date: 2026-07-06
Workplans: CUST-WP-0054-T05, CUST-WP-0011-T07 (playbook), Wave 2 in docs/coulombcore-drain-placement-plan.md
no_secret_material_recorded: true
Purpose
Move the production State Hub primary from coulombcore-k3s to railiance01-k3s so the automation loop (activity-core, Temporal, daily triage, consistency sweeps) is machine-local and survives coulombcore or workstation outages.
This is a second cutover in the pragmatic migration arc:
| Cutover | Date | From | To |
|---|---|---|---|
| CUST-WP-0011-T07 | 2026-07-03 | WSL2 | coulombcore cluster |
| CUST-WP-0054-T05 | planned | coulombcore cluster | railiance01 cluster |
Current state (2026-07-06 inventory)
Production writer
| Item | Value |
|---|---|
| API workload | coulombcore-k3s, namespace state-hub |
| Database | CNPG state-hub-db, namespace databases on coulombcore |
| Image (deployed) | gitea.coulomb.social/coulomb/state-hub:f2e042a (Helm handoff pin) |
| Image (Forgejo CI) | forgejo.coulomb.social/coulomb/state-hub:f9f0091 — ready for cutover |
| Hub totals | 637 workstreams, 3983 tasks, 14 topics (GET /state/summary) |
Access paths
| Consumer | Path today | Serves |
|---|---|---|
| Workstation agents | http://127.0.0.1:8000 |
ops-bridge state-hub-primary → coulombcore cluster Svc |
| activity-core (railiance01) | actcore-state-hub-bridge → 127.0.0.1:18000 |
systemd fleet-state-hub-coulombcore → coulombcore ClusterIP 10.43.170.94:8000 |
| WSL2 fallback | make api (local compose) |
Retained; not normal writer |
railiance01 cluster (target)
| Item | Status |
|---|---|
| CNPG operator | Healthy (forgejo-db, net-kingdom-pg in databases) |
Namespace state-hub |
Does not exist |
CNPG state-hub-db on railiance01 |
Not deployed |
Hostname for host_paths |
239.62.205.92.host.secureserver.net (not railiance01) |
| Clone tree | Partial (activity-core, llm-connect, … under /home/tegwick/) |
| Forgejo | Canonical (forgejo.coulomb.social) — Wave 1 complete |
Prerequisites satisfied
- T04 tier-3 production repos on Forgejo (images build in CI)
- Fleet mesh de-hub (T02) — production does not chain through workstation
- CUST-WP-0011 deploy handoff (
state-hub/deploy/railiance/) - Phase 0 preflight (2026-07-06) —
docs/state-hub-railiance01-cutover-preflight-2026-07-06.md - CNPG + storage sign-off (Wave 2.1) —
local-path, 75G free, operator healthy - Operator approval for freeze/final restore (constitution gate)
Target end state
railiance01-k3s
databases/state-hub-db (CNPG, 1 instance)
state-hub/state-hub Deployment + ClusterIP :8000
↑
├── activity-core: in-cluster http://state-hub.state-hub.svc:8000
│ (retire actcore-state-hub-bridge + fleet-state-hub-coulombcore)
├── consistency sweep: host_paths[239.62.205.92.host.secureserver.net]
│ → /home/tegwick/<repo> (Forgejo clones)
└── workstation (dev): ops-bridge tunnel → railiance01 cluster Svc
coulombcore: state-hub scaled to 0 / read-only rollback copy (not writer)
WSL2: fallback only (CUST-WP-0011-T08/T09 retirement folds into T10)
Safety contract (from CUST-WP-0011)
- coulombcore remains rollback-capable until stabilization passes.
- No cutover without exact-count row comparison after final restore.
- Failed railiance01 deploy must not destroy coulombcore data.
- Explicit human approval before freezing the production writer.
Cutover phases
Phase 0 — Preflight and baseline (no production impact)
Owner: platform + custodian
Done when: checklist green, baseline row counts recorded.
| # | Action | Command / artifact |
|---|---|---|
| 0.1 | Record baseline totals | curl -s http://127.0.0.1:8000/state/summary |
| 0.2 | Record per-table counts | pg_dump --schema-only + SELECT count(*) on coulombcore CNPG (via port-forward) |
| 0.3 | Confirm Forgejo image pull | crictl pull forgejo.coulomb.social/coulomb/state-hub:f9f0091 on railiance01 |
| 0.4 | Dry-run manifests | cd ~/state-hub && KUBECONFIG=~/.kube/config-hosteurope make railiance-state-hub-server-dry-run |
| 0.5 | Storage review | Confirm local-path (or chosen class) has ≥10Gi for state-hub-db PVC |
| 0.6 | Operator approval | POST /decisions/ or workplan note with approval id |
Recorded 2026-07-06 (Phase 0 complete): see
docs/state-hub-railiance01-cutover-preflight-2026-07-06.md.
workplans: 637 tasks: 3983 topics: 14
progress_events: 8717 token_events: 1936 managed_repos: 74
decisions: 79
Post-restore counts on railiance01 must match exactly.
Phase 1 — Deploy empty stack on railiance01
Owner: railiance-platform + railiance-apps promotion from state-hub/deploy/railiance/
Done when: empty hub healthy on railiance01, Alembic at head, no production data yet.
| # | Action | Notes |
|---|---|---|
| 1.1 | Create state-hub namespace |
apps/manifests/state-hub-namespace.yaml |
| 1.2 | Deploy state-hub-db CNPG |
platform/state-hub-db-cluster.yaml + credentials Secret |
| 1.3 | Apply DB NetworkPolicies | platform/state-hub-db-networkpolicies.yaml |
| 1.4 | Create state-hub-env Secret |
DATABASE_URL → state-hub-db-rw.databases.svc.cluster.local |
| 1.5 | Update image to Forgejo | forgejo.coulomb.social/coulomb/state-hub:f9f0091 in values |
| 1.6 | Run Alembic Job | Same pattern as CUST-WP-0011-T05 |
| 1.7 | Helm install state-hub |
namespace.create=false; verify /state/health via port-forward |
Promotion path: copy handoff assets into railiance-platform / railiance-apps repos
(GitOps) or apply from state-hub/deploy/railiance/ with operator approval.
Phase 2 — Data migration (CUST-WP-0011-T07 playbook)
Owner: custodian operator
Done when: railiance01 DB is exact copy; coulombcore API still available for rollback.
| # | Action | Notes |
|---|---|---|
| 2.1 | Fresh pg_dump from coulombcore state_hub |
Custom format (-Fc); timestamp recorded |
| 2.2 | Freeze coulombcore writer | Scale state-hub Deployment to 0 on coulombcore |
| 2.3 | Final pg_dump |
Immediately after freeze |
| 2.4 | Restore into railiance01 state-hub-db |
pg_restore with SET ROLE state_hub; scale railiance01 API to 0 during restore |
| 2.5 | Exact-count comparison | All public tables; mismatch → abort, restore coulombcore |
| 2.6 | Scale railiance01 API to 1 | Verify /state/health + /state/summary via port-forward |
Phase 3 — Rewire access paths
Owner: ops-bridge + activity-core
Done when: all production consumers reach railiance01 hub without coulombcore hop.
| # | Consumer | Before | After |
|---|---|---|---|
| 3.1 | activity-core | bridge → :18000 → fleet tunnel → coulombcore |
In-cluster http://state-hub.state-hub.svc.cluster.local:8000 (retire bridge Deployment) |
| 3.2 | railiance01 systemd | fleet-state-hub-coulombcore active |
Disable/stop unit |
| 3.3 | Workstation | state-hub-primary → coulombcore |
Repoint tunnel to railiance01 cluster Svc ClusterIP |
| 3.4 | MCP / agents | API_BASE=http://127.0.0.1:8000 |
Unchanged URL; tunnel backend moves |
| 3.5 | Smoke write | POST /progress/ from activity-core |
Record event id as cutover proof |
ClusterIP for railiance01 state-hub Service must be captured at deploy time for
tunnel remote_host if using SSH local-forward pattern from T02.
Phase 4 — Sweep checkout migration (Wave 2.3)
Owner: custodian + STATE-WP-0064
Done when: production writebacks use railiance01 paths, not workstation.
| # | Action | Notes |
|---|---|---|
| 4.1 | Define clone root | /home/tegwick/<slug> (existing partial tree) |
| 4.2 | Bulk clone from Forgejo | Script over registered repos (GET /repos/) |
| 4.3 | Register host_paths |
POST /repos/{slug}/paths with host=239.62.205.92.host.secureserver.net |
| 4.4 | Run sweep on railiance01 | POST /consistency/sweep/remote-all or activity-core cron |
| 4.5 | Verify git writeback | fix-consistency mutates railiance01 clone, not /home/worsch/* |
Workstation host_paths entries may remain for dev; production sweep host becomes
the railiance01 FQDN.
Phase 5 — Stabilization (48–72h)
Owner: custodian
Done when: no data loss, triage + sweeps + emission green without coulombcore.
Monitor:
- State Hub pod restarts (railiance01)
- CNPG
state-hub-dbhealth - Daily triage ActivityRun success
- Consistency sweep
exit_code: 0every 15m - issue-core emission (still via coulombcore tunnel until Wave 4)
coulombcore state-hub remains stopped, not deleted until stabilization passes.
Rollback
| Trigger | Action |
|---|---|
| Row-count mismatch | Do not rewire; scale coulombcore API back to 1 |
| railiance01 API unhealthy post-rewire | Re-enable fleet-state-hub-coulombcore; repoint workstation tunnel to coulombcore |
| Data corruption suspected | Restore coulombcore from pre-cutover dump; freeze railiance01 |
Rollback window: keep coulombcore dump and Deployment manifests for ≥7 days.
Verification checklist (T05 done)
GET /state/summaryon railiance01 matches pre-cutover totals (±0 rows)- activity-core triage + sweep ActivityRuns succeed without
fleet-state-hub-coulombcore - Workstation
127.0.0.1:8000/state/healthserves railiance01 hub fix-consistencywriteback lands on railiance01 clone path- coulombcore hub Deployment scaled to 0 (not deleted)
- Progress event recorded with cutover summary + operator approval ref
Sequencing with other CUST-WP-0054 tasks
| Task | Relationship |
|---|---|
| T04 | Prerequisite met — Forgejo is clone/image source |
| T06 | Can proceed in parallel; sink paths still reference workstation until T06 |
| T07 | Dev beachhead independent |
| T10 | WSL2 fallback retirement + 24h workstation-off proof after T05 stabilizes |
Open decisions (need operator input before Phase 2)
- Maintenance window — suggested low-traffic window; freeze blocks hub writes ~15–30 min.
- GitOps vs manual apply — promote
deploy/railiance/intorailiance-platform/railiance-appsfirst, or operator apply for speed? - Sweep clone scope — all 74 registered repos at cutover, or phased (state-hub + the-custodian + activity-core first)?
- coulombcore rollback retention — how long to keep stopped Deployment + dump?
References
state-hub/workplans/archived/260706-CUST-WP-0011-state-hub-threephoenix-migration.md(T07 sequence)state-hub/docs/cluster-operating-model.md(current coulombcore reality — update post-cutover)state-hub/deploy/railiance/README.mddocs/fleet-mesh-dehub-runbook.mddocs/forgejo-tier3-remote-url-sweep-playbook.md(§ Railiance01 sweep paths)docs/coulombcore-drain-placement-plan.md(Wave 2)infra/fleet-mesh/railiance01-tunnels.yaml