233 lines
11 KiB
Markdown
233 lines
11 KiB
Markdown
# State Hub railiance01 Cutover Plan (CUST-WP-0054-T05)
|
||
|
||
Date: 2026-07-06
|
||
Workplans: `CUST-WP-0054-T05`, `CUST-WP-0011-T07` (playbook), Wave 2 in `docs/coulombcore-drain-placement-plan.md`
|
||
`no_secret_material_recorded: true`
|
||
|
||
## Purpose
|
||
|
||
Move the **production State Hub primary** from **coulombcore-k3s** to **railiance01-k3s**
|
||
so the automation loop (activity-core, Temporal, daily triage, consistency sweeps)
|
||
is machine-local and survives coulombcore or workstation outages.
|
||
|
||
This is a **second cutover** in the pragmatic migration arc:
|
||
|
||
| Cutover | Date | From | To |
|
||
| --- | --- | --- | --- |
|
||
| CUST-WP-0011-T07 | 2026-07-03 | WSL2 | coulombcore cluster |
|
||
| **CUST-WP-0054-T05** | planned | coulombcore cluster | railiance01 cluster |
|
||
|
||
## Current state (2026-07-06 inventory)
|
||
|
||
### Production writer
|
||
|
||
| Item | Value |
|
||
| --- | --- |
|
||
| API workload | coulombcore-k3s, namespace `state-hub` |
|
||
| Database | CNPG `state-hub-db`, namespace `databases` on coulombcore |
|
||
| Image (deployed) | `gitea.coulomb.social/coulomb/state-hub:f2e042a` (Helm handoff pin) |
|
||
| Image (Forgejo CI) | `forgejo.coulomb.social/coulomb/state-hub:f9f0091` — ready for cutover |
|
||
| Hub totals | 637 workstreams, 3983 tasks, 14 topics (`GET /state/summary`) |
|
||
|
||
### Access paths
|
||
|
||
| Consumer | Path today | Serves |
|
||
| --- | --- | --- |
|
||
| Workstation agents | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` → coulombcore cluster Svc |
|
||
| activity-core (railiance01) | `actcore-state-hub-bridge` → `127.0.0.1:18000` | systemd `fleet-state-hub-coulombcore` → coulombcore ClusterIP `10.43.170.94:8000` |
|
||
| WSL2 fallback | `make api` (local compose) | Retained; not normal writer |
|
||
|
||
### railiance01 cluster (target)
|
||
|
||
| Item | Status |
|
||
| --- | --- |
|
||
| CNPG operator | Healthy (`forgejo-db`, `net-kingdom-pg` in `databases`) |
|
||
| Namespace `state-hub` | **Does not exist** |
|
||
| CNPG `state-hub-db` on railiance01 | **Not deployed** |
|
||
| Hostname for `host_paths` | `239.62.205.92.host.secureserver.net` (not `railiance01`) |
|
||
| Clone tree | Partial (`activity-core`, `llm-connect`, … under `/home/tegwick/`) |
|
||
| Forgejo | Canonical (`forgejo.coulomb.social`) — Wave 1 complete |
|
||
|
||
### Prerequisites satisfied
|
||
|
||
- [x] T04 tier-3 production repos on Forgejo (images build in CI)
|
||
- [x] Fleet mesh de-hub (T02) — production does not chain through workstation
|
||
- [x] CUST-WP-0011 deploy handoff (`state-hub/deploy/railiance/`)
|
||
- [x] **Phase 0 preflight** (2026-07-06) — `docs/state-hub-railiance01-cutover-preflight-2026-07-06.md`
|
||
- [x] CNPG + storage sign-off (Wave 2.1) — `local-path`, 75G free, operator healthy
|
||
- [ ] **Operator approval** for freeze/final restore (constitution gate)
|
||
|
||
## Target end state
|
||
|
||
```
|
||
railiance01-k3s
|
||
databases/state-hub-db (CNPG, 1 instance)
|
||
state-hub/state-hub Deployment + ClusterIP :8000
|
||
↑
|
||
├── activity-core: in-cluster http://state-hub.state-hub.svc:8000
|
||
│ (retire actcore-state-hub-bridge + fleet-state-hub-coulombcore)
|
||
├── consistency sweep: host_paths[239.62.205.92.host.secureserver.net]
|
||
│ → /home/tegwick/<repo> (Forgejo clones)
|
||
└── workstation (dev): ops-bridge tunnel → railiance01 cluster Svc
|
||
|
||
coulombcore: state-hub scaled to 0 / read-only rollback copy (not writer)
|
||
WSL2: fallback only (CUST-WP-0011-T08/T09 retirement folds into T10)
|
||
```
|
||
|
||
## Safety contract (from CUST-WP-0011)
|
||
|
||
- coulombcore remains rollback-capable until stabilization passes.
|
||
- No cutover without **exact-count** row comparison after final restore.
|
||
- Failed railiance01 deploy must not destroy coulombcore data.
|
||
- Explicit human approval before freezing the production writer.
|
||
|
||
## Cutover phases
|
||
|
||
### Phase 0 — Preflight and baseline (no production impact)
|
||
|
||
**Owner:** platform + custodian
|
||
**Done when:** checklist green, baseline row counts recorded.
|
||
|
||
| # | Action | Command / artifact |
|
||
| --- | --- | --- |
|
||
| 0.1 | Record baseline totals | `curl -s http://127.0.0.1:8000/state/summary` |
|
||
| 0.2 | Record per-table counts | `pg_dump --schema-only` + `SELECT count(*)` on coulombcore CNPG (via port-forward) |
|
||
| 0.3 | Confirm Forgejo image pull | `crictl pull forgejo.coulomb.social/coulomb/state-hub:f9f0091` on railiance01 |
|
||
| 0.4 | Dry-run manifests | `cd ~/state-hub && KUBECONFIG=~/.kube/config-hosteurope make railiance-state-hub-server-dry-run` |
|
||
| 0.5 | Storage review | Confirm `local-path` (or chosen class) has ≥10Gi for `state-hub-db` PVC |
|
||
| 0.6 | Operator approval | `POST /decisions/` or workplan note with approval id |
|
||
|
||
**Recorded 2026-07-06 (Phase 0 complete):** see
|
||
`docs/state-hub-railiance01-cutover-preflight-2026-07-06.md`.
|
||
|
||
```text
|
||
workplans: 637 tasks: 3983 topics: 14
|
||
progress_events: 8717 token_events: 1936 managed_repos: 74
|
||
decisions: 79
|
||
```
|
||
|
||
Post-restore counts on railiance01 must match exactly.
|
||
|
||
### Phase 1 — Deploy empty stack on railiance01
|
||
|
||
**Owner:** `railiance-platform` + `railiance-apps` promotion from `state-hub/deploy/railiance/`
|
||
**Done when:** empty hub healthy on railiance01, Alembic at head, **no production data yet**.
|
||
|
||
| # | Action | Notes |
|
||
| --- | --- | --- |
|
||
| 1.1 | Create `state-hub` namespace | `apps/manifests/state-hub-namespace.yaml` |
|
||
| 1.2 | Deploy `state-hub-db` CNPG | `platform/state-hub-db-cluster.yaml` + credentials Secret |
|
||
| 1.3 | Apply DB NetworkPolicies | `platform/state-hub-db-networkpolicies.yaml` |
|
||
| 1.4 | Create `state-hub-env` Secret | `DATABASE_URL` → `state-hub-db-rw.databases.svc.cluster.local` |
|
||
| 1.5 | Update image to Forgejo | `forgejo.coulomb.social/coulomb/state-hub:f9f0091` in values |
|
||
| 1.6 | Run Alembic Job | Same pattern as CUST-WP-0011-T05 |
|
||
| 1.7 | Helm install `state-hub` | `namespace.create=false`; verify `/state/health` via port-forward |
|
||
|
||
Promotion path: copy handoff assets into `railiance-platform` / `railiance-apps` repos
|
||
(GitOps) or apply from `state-hub/deploy/railiance/` with operator approval.
|
||
|
||
### Phase 2 — Data migration (CUST-WP-0011-T07 playbook)
|
||
|
||
**Owner:** custodian operator
|
||
**Done when:** railiance01 DB is exact copy; coulombcore API still available for rollback.
|
||
|
||
| # | Action | Notes |
|
||
| --- | --- | --- |
|
||
| 2.1 | Fresh `pg_dump` from coulombcore `state_hub` | Custom format (`-Fc`); timestamp recorded |
|
||
| 2.2 | **Freeze** coulombcore writer | Scale `state-hub` Deployment to 0 on coulombcore |
|
||
| 2.3 | Final `pg_dump` | Immediately after freeze |
|
||
| 2.4 | Restore into railiance01 `state-hub-db` | `pg_restore` with `SET ROLE state_hub`; scale railiance01 API to 0 during restore |
|
||
| 2.5 | Exact-count comparison | All public tables; mismatch → abort, restore coulombcore |
|
||
| 2.6 | Scale railiance01 API to 1 | Verify `/state/health` + `/state/summary` via port-forward |
|
||
|
||
### Phase 3 — Rewire access paths
|
||
|
||
**Owner:** ops-bridge + activity-core
|
||
**Done when:** all production consumers reach railiance01 hub without coulombcore hop.
|
||
|
||
| # | Consumer | Before | After |
|
||
| --- | --- | --- | --- |
|
||
| 3.1 | activity-core | bridge → `:18000` → fleet tunnel → coulombcore | In-cluster `http://state-hub.state-hub.svc.cluster.local:8000` (retire bridge Deployment) |
|
||
| 3.2 | railiance01 systemd | `fleet-state-hub-coulombcore` active | **Disable/stop** unit |
|
||
| 3.3 | Workstation | `state-hub-primary` → coulombcore | Repoint tunnel to railiance01 cluster Svc ClusterIP |
|
||
| 3.4 | MCP / agents | `API_BASE=http://127.0.0.1:8000` | Unchanged URL; tunnel backend moves |
|
||
| 3.5 | Smoke write | `POST /progress/` from activity-core | Record event id as cutover proof |
|
||
|
||
ClusterIP for railiance01 `state-hub` Service must be captured at deploy time for
|
||
tunnel `remote_host` if using SSH local-forward pattern from T02.
|
||
|
||
### Phase 4 — Sweep checkout migration (Wave 2.3)
|
||
|
||
**Owner:** custodian + STATE-WP-0064
|
||
**Done when:** production writebacks use railiance01 paths, not workstation.
|
||
|
||
| # | Action | Notes |
|
||
| --- | --- | --- |
|
||
| 4.1 | Define clone root | `/home/tegwick/<slug>` (existing partial tree) |
|
||
| 4.2 | Bulk clone from Forgejo | Script over registered repos (`GET /repos/`) |
|
||
| 4.3 | Register `host_paths` | `POST /repos/{slug}/paths` with host=`239.62.205.92.host.secureserver.net` |
|
||
| 4.4 | Run sweep on railiance01 | `POST /consistency/sweep/remote-all` or activity-core cron |
|
||
| 4.5 | Verify git writeback | fix-consistency mutates railiance01 clone, not `/home/worsch/*` |
|
||
|
||
Workstation `host_paths` entries may remain for dev; production sweep host becomes
|
||
the railiance01 FQDN.
|
||
|
||
### Phase 5 — Stabilization (48–72h)
|
||
|
||
**Owner:** custodian
|
||
**Done when:** no data loss, triage + sweeps + emission green without coulombcore.
|
||
|
||
Monitor:
|
||
|
||
- State Hub pod restarts (railiance01)
|
||
- CNPG `state-hub-db` health
|
||
- Daily triage ActivityRun success
|
||
- Consistency sweep `exit_code: 0` every 15m
|
||
- issue-core emission (still via coulombcore tunnel until Wave 4)
|
||
|
||
coulombcore state-hub remains **stopped, not deleted** until stabilization passes.
|
||
|
||
## Rollback
|
||
|
||
| Trigger | Action |
|
||
| --- | --- |
|
||
| Row-count mismatch | Do not rewire; scale coulombcore API back to 1 |
|
||
| railiance01 API unhealthy post-rewire | Re-enable `fleet-state-hub-coulombcore`; repoint workstation tunnel to coulombcore |
|
||
| Data corruption suspected | Restore coulombcore from pre-cutover dump; freeze railiance01 |
|
||
|
||
Rollback window: keep coulombcore dump and Deployment manifests for ≥7 days.
|
||
|
||
## Verification checklist (T05 done)
|
||
|
||
- [ ] `GET /state/summary` on railiance01 matches pre-cutover totals (±0 rows)
|
||
- [ ] activity-core triage + sweep ActivityRuns succeed without `fleet-state-hub-coulombcore`
|
||
- [ ] Workstation `127.0.0.1:8000/state/health` serves railiance01 hub
|
||
- [ ] `fix-consistency` writeback lands on railiance01 clone path
|
||
- [ ] coulombcore hub Deployment scaled to 0 (not deleted)
|
||
- [ ] Progress event recorded with cutover summary + operator approval ref
|
||
|
||
## Sequencing with other CUST-WP-0054 tasks
|
||
|
||
| Task | Relationship |
|
||
| --- | --- |
|
||
| T04 | **Prerequisite met** — Forgejo is clone/image source |
|
||
| T06 | Can proceed in parallel; sink paths still reference workstation until T06 |
|
||
| T07 | Dev beachhead independent |
|
||
| T10 | WSL2 fallback retirement + 24h workstation-off proof after T05 stabilizes |
|
||
|
||
## Open decisions (need operator input before Phase 2)
|
||
|
||
1. **Maintenance window** — suggested low-traffic window; freeze blocks hub writes ~15–30 min.
|
||
2. **GitOps vs manual apply** — promote `deploy/railiance/` into `railiance-platform`/`railiance-apps` first, or operator apply for speed?
|
||
3. **Sweep clone scope** — all 74 registered repos at cutover, or phased (state-hub + the-custodian + activity-core first)?
|
||
4. **coulombcore rollback retention** — how long to keep stopped Deployment + dump?
|
||
|
||
## References
|
||
|
||
- `state-hub/workplans/archived/260706-CUST-WP-0011-state-hub-threephoenix-migration.md` (T07 sequence)
|
||
- `state-hub/docs/cluster-operating-model.md` (current coulombcore reality — update post-cutover)
|
||
- `state-hub/deploy/railiance/README.md`
|
||
- `docs/fleet-mesh-dehub-runbook.md`
|
||
- `docs/forgejo-tier3-remote-url-sweep-playbook.md` (§ Railiance01 sweep paths)
|
||
- `docs/coulombcore-drain-placement-plan.md` (Wave 2)
|
||
- `infra/fleet-mesh/railiance01-tunnels.yaml` |