From fa715c29d58ce61e50abf28a8c3d011bc7c0b0c7 Mon Sep 17 00:00:00 2001 From: codex Date: Mon, 6 Jul 2026 17:25:09 +0200 Subject: [PATCH] CUST-WP-0054-T05: State Hub railiance01 cutover plan and task progress --- docs/state-hub-railiance01-cutover-plan.md | 231 ++++++++++++++++++ ...tion-independence-and-fleet-realignment.md | 11 +- 2 files changed, 241 insertions(+), 1 deletion(-) create mode 100644 docs/state-hub-railiance01-cutover-plan.md diff --git a/docs/state-hub-railiance01-cutover-plan.md b/docs/state-hub-railiance01-cutover-plan.md new file mode 100644 index 0000000..071009d --- /dev/null +++ b/docs/state-hub-railiance01-cutover-plan.md @@ -0,0 +1,231 @@ +# State Hub railiance01 Cutover Plan (CUST-WP-0054-T05) + +Date: 2026-07-06 +Workplans: `CUST-WP-0054-T05`, `CUST-WP-0011-T07` (playbook), Wave 2 in `docs/coulombcore-drain-placement-plan.md` +`no_secret_material_recorded: true` + +## Purpose + +Move the **production State Hub primary** from **coulombcore-k3s** to **railiance01-k3s** +so the automation loop (activity-core, Temporal, daily triage, consistency sweeps) +is machine-local and survives coulombcore or workstation outages. + +This is a **second cutover** in the pragmatic migration arc: + +| Cutover | Date | From | To | +| --- | --- | --- | --- | +| CUST-WP-0011-T07 | 2026-07-03 | WSL2 | coulombcore cluster | +| **CUST-WP-0054-T05** | planned | coulombcore cluster | railiance01 cluster | + +## Current state (2026-07-06 inventory) + +### Production writer + +| Item | Value | +| --- | --- | +| API workload | coulombcore-k3s, namespace `state-hub` | +| Database | CNPG `state-hub-db`, namespace `databases` on coulombcore | +| Image (deployed) | `gitea.coulomb.social/coulomb/state-hub:f2e042a` (Helm handoff pin) | +| Image (Forgejo CI) | `forgejo.coulomb.social/coulomb/state-hub:f9f0091` — ready for cutover | +| Hub totals | 637 workstreams, 3983 tasks, 14 topics (`GET /state/summary`) | + +### Access paths + +| Consumer | Path today | Serves | +| --- | --- | --- | +| Workstation agents | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` → coulombcore cluster Svc | +| activity-core (railiance01) | `actcore-state-hub-bridge` → `127.0.0.1:18000` | systemd `fleet-state-hub-coulombcore` → coulombcore ClusterIP `10.43.170.94:8000` | +| WSL2 fallback | `make api` (local compose) | Retained; not normal writer | + +### railiance01 cluster (target) + +| Item | Status | +| --- | --- | +| CNPG operator | Healthy (`forgejo-db`, `net-kingdom-pg` in `databases`) | +| Namespace `state-hub` | **Does not exist** | +| CNPG `state-hub-db` on railiance01 | **Not deployed** | +| Hostname for `host_paths` | `239.62.205.92.host.secureserver.net` (not `railiance01`) | +| Clone tree | Partial (`activity-core`, `llm-connect`, … under `/home/tegwick/`) | +| Forgejo | Canonical (`forgejo.coulomb.social`) — Wave 1 complete | + +### Prerequisites satisfied + +- [x] T04 tier-3 production repos on Forgejo (images build in CI) +- [x] Fleet mesh de-hub (T02) — production does not chain through workstation +- [x] CUST-WP-0011 deploy handoff (`state-hub/deploy/railiance/`) +- [ ] **Operator approval** for freeze/final restore (constitution gate) +- [ ] CNPG + storage sign-off (Wave 2.1) + +## Target end state + +``` +railiance01-k3s + databases/state-hub-db (CNPG, 1 instance) + state-hub/state-hub Deployment + ClusterIP :8000 + ↑ + ├── activity-core: in-cluster http://state-hub.state-hub.svc:8000 + │ (retire actcore-state-hub-bridge + fleet-state-hub-coulombcore) + ├── consistency sweep: host_paths[239.62.205.92.host.secureserver.net] + │ → /home/tegwick/ (Forgejo clones) + └── workstation (dev): ops-bridge tunnel → railiance01 cluster Svc + +coulombcore: state-hub scaled to 0 / read-only rollback copy (not writer) +WSL2: fallback only (CUST-WP-0011-T08/T09 retirement folds into T10) +``` + +## Safety contract (from CUST-WP-0011) + +- coulombcore remains rollback-capable until stabilization passes. +- No cutover without **exact-count** row comparison after final restore. +- Failed railiance01 deploy must not destroy coulombcore data. +- Explicit human approval before freezing the production writer. + +## Cutover phases + +### Phase 0 — Preflight and baseline (no production impact) + +**Owner:** platform + custodian +**Done when:** checklist green, baseline row counts recorded. + +| # | Action | Command / artifact | +| --- | --- | --- | +| 0.1 | Record baseline totals | `curl -s http://127.0.0.1:8000/state/summary` | +| 0.2 | Record per-table counts | `pg_dump --schema-only` + `SELECT count(*)` on coulombcore CNPG (via port-forward) | +| 0.3 | Confirm Forgejo image pull | `crictl pull forgejo.coulomb.social/coulomb/state-hub:f9f0091` on railiance01 | +| 0.4 | Dry-run manifests | `cd ~/state-hub && KUBECONFIG=~/.kube/config-hosteurope make railiance-state-hub-server-dry-run` | +| 0.5 | Storage review | Confirm `local-path` (or chosen class) has ≥10Gi for `state-hub-db` PVC | +| 0.6 | Operator approval | `POST /decisions/` or workplan note with approval id | + +Baseline totals to match at cutover (2026-07-06): + +```text +workstreams.total: 637 +tasks.total: 3983 +topics.total: 14 +``` + +Extend with explicit `pg_catalog` table counts for `workplans`, `tasks`, `progress_events`, `managed_repos`, `token_events` (same set as CUST-WP-0011-T07). + +### Phase 1 — Deploy empty stack on railiance01 + +**Owner:** `railiance-platform` + `railiance-apps` promotion from `state-hub/deploy/railiance/` +**Done when:** empty hub healthy on railiance01, Alembic at head, **no production data yet**. + +| # | Action | Notes | +| --- | --- | --- | +| 1.1 | Create `state-hub` namespace | `apps/manifests/state-hub-namespace.yaml` | +| 1.2 | Deploy `state-hub-db` CNPG | `platform/state-hub-db-cluster.yaml` + credentials Secret | +| 1.3 | Apply DB NetworkPolicies | `platform/state-hub-db-networkpolicies.yaml` | +| 1.4 | Create `state-hub-env` Secret | `DATABASE_URL` → `state-hub-db-rw.databases.svc.cluster.local` | +| 1.5 | Update image to Forgejo | `forgejo.coulomb.social/coulomb/state-hub:f9f0091` in values | +| 1.6 | Run Alembic Job | Same pattern as CUST-WP-0011-T05 | +| 1.7 | Helm install `state-hub` | `namespace.create=false`; verify `/state/health` via port-forward | + +Promotion path: copy handoff assets into `railiance-platform` / `railiance-apps` repos +(GitOps) or apply from `state-hub/deploy/railiance/` with operator approval. + +### Phase 2 — Data migration (CUST-WP-0011-T07 playbook) + +**Owner:** custodian operator +**Done when:** railiance01 DB is exact copy; coulombcore API still available for rollback. + +| # | Action | Notes | +| --- | --- | --- | +| 2.1 | Fresh `pg_dump` from coulombcore `state_hub` | Custom format (`-Fc`); timestamp recorded | +| 2.2 | **Freeze** coulombcore writer | Scale `state-hub` Deployment to 0 on coulombcore | +| 2.3 | Final `pg_dump` | Immediately after freeze | +| 2.4 | Restore into railiance01 `state-hub-db` | `pg_restore` with `SET ROLE state_hub`; scale railiance01 API to 0 during restore | +| 2.5 | Exact-count comparison | All public tables; mismatch → abort, restore coulombcore | +| 2.6 | Scale railiance01 API to 1 | Verify `/state/health` + `/state/summary` via port-forward | + +### Phase 3 — Rewire access paths + +**Owner:** ops-bridge + activity-core +**Done when:** all production consumers reach railiance01 hub without coulombcore hop. + +| # | Consumer | Before | After | +| --- | --- | --- | --- | +| 3.1 | activity-core | bridge → `:18000` → fleet tunnel → coulombcore | In-cluster `http://state-hub.state-hub.svc.cluster.local:8000` (retire bridge Deployment) | +| 3.2 | railiance01 systemd | `fleet-state-hub-coulombcore` active | **Disable/stop** unit | +| 3.3 | Workstation | `state-hub-primary` → coulombcore | Repoint tunnel to railiance01 cluster Svc ClusterIP | +| 3.4 | MCP / agents | `API_BASE=http://127.0.0.1:8000` | Unchanged URL; tunnel backend moves | +| 3.5 | Smoke write | `POST /progress/` from activity-core | Record event id as cutover proof | + +ClusterIP for railiance01 `state-hub` Service must be captured at deploy time for +tunnel `remote_host` if using SSH local-forward pattern from T02. + +### Phase 4 — Sweep checkout migration (Wave 2.3) + +**Owner:** custodian + STATE-WP-0064 +**Done when:** production writebacks use railiance01 paths, not workstation. + +| # | Action | Notes | +| --- | --- | --- | +| 4.1 | Define clone root | `/home/tegwick/` (existing partial tree) | +| 4.2 | Bulk clone from Forgejo | Script over registered repos (`GET /repos/`) | +| 4.3 | Register `host_paths` | `POST /repos/{slug}/paths` with host=`239.62.205.92.host.secureserver.net` | +| 4.4 | Run sweep on railiance01 | `POST /consistency/sweep/remote-all` or activity-core cron | +| 4.5 | Verify git writeback | fix-consistency mutates railiance01 clone, not `/home/worsch/*` | + +Workstation `host_paths` entries may remain for dev; production sweep host becomes +the railiance01 FQDN. + +### Phase 5 — Stabilization (48–72h) + +**Owner:** custodian +**Done when:** no data loss, triage + sweeps + emission green without coulombcore. + +Monitor: + +- State Hub pod restarts (railiance01) +- CNPG `state-hub-db` health +- Daily triage ActivityRun success +- Consistency sweep `exit_code: 0` every 15m +- issue-core emission (still via coulombcore tunnel until Wave 4) + +coulombcore state-hub remains **stopped, not deleted** until stabilization passes. + +## Rollback + +| Trigger | Action | +| --- | --- | +| Row-count mismatch | Do not rewire; scale coulombcore API back to 1 | +| railiance01 API unhealthy post-rewire | Re-enable `fleet-state-hub-coulombcore`; repoint workstation tunnel to coulombcore | +| Data corruption suspected | Restore coulombcore from pre-cutover dump; freeze railiance01 | + +Rollback window: keep coulombcore dump and Deployment manifests for ≥7 days. + +## Verification checklist (T05 done) + +- [ ] `GET /state/summary` on railiance01 matches pre-cutover totals (±0 rows) +- [ ] activity-core triage + sweep ActivityRuns succeed without `fleet-state-hub-coulombcore` +- [ ] Workstation `127.0.0.1:8000/state/health` serves railiance01 hub +- [ ] `fix-consistency` writeback lands on railiance01 clone path +- [ ] coulombcore hub Deployment scaled to 0 (not deleted) +- [ ] Progress event recorded with cutover summary + operator approval ref + +## Sequencing with other CUST-WP-0054 tasks + +| Task | Relationship | +| --- | --- | +| T04 | **Prerequisite met** — Forgejo is clone/image source | +| T06 | Can proceed in parallel; sink paths still reference workstation until T06 | +| T07 | Dev beachhead independent | +| T10 | WSL2 fallback retirement + 24h workstation-off proof after T05 stabilizes | + +## Open decisions (need operator input before Phase 2) + +1. **Maintenance window** — suggested low-traffic window; freeze blocks hub writes ~15–30 min. +2. **GitOps vs manual apply** — promote `deploy/railiance/` into `railiance-platform`/`railiance-apps` first, or operator apply for speed? +3. **Sweep clone scope** — all 74 registered repos at cutover, or phased (state-hub + the-custodian + activity-core first)? +4. **coulombcore rollback retention** — how long to keep stopped Deployment + dump? + +## References + +- `state-hub/workplans/archived/260706-CUST-WP-0011-state-hub-threephoenix-migration.md` (T07 sequence) +- `state-hub/docs/cluster-operating-model.md` (current coulombcore reality — update post-cutover) +- `state-hub/deploy/railiance/README.md` +- `docs/fleet-mesh-dehub-runbook.md` +- `docs/forgejo-tier3-remote-url-sweep-playbook.md` (§ Railiance01 sweep paths) +- `docs/coulombcore-drain-placement-plan.md` (Wave 2) +- `infra/fleet-mesh/railiance01-tunnels.yaml` \ No newline at end of file diff --git a/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md b/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md index e7b9ac8..52a0c52 100644 --- a/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md +++ b/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md @@ -199,7 +199,7 @@ of remaining registered repos. ```task id: CUST-WP-0054-T05 -status: todo +status: progress priority: high state_hub_task_id: "e91db8d0-973d-4a31-b3c2-ca37fd002ec7" ``` @@ -212,6 +212,15 @@ Prereq: railiance01 CNPG + storage reviewed (T03). Also relocate the consistency-sweep repo checkouts to railiance01 (clones from the T04 forge) so file writebacks no longer touch workstation paths. +**Planning (2026-07-06):** Cutover runbook at +`docs/state-hub-railiance01-cutover-plan.md`. Inventory: primary on coulombcore +(637 ws / 3983 tasks); railiance01 has CNPG operator + Forgejo but **no** +`state-hub` namespace or `state-hub-db` yet; activity-core reaches hub via +`fleet-state-hub-coulombcore` → `:18000`. Target hostname for sweep paths: +`239.62.205.92.host.secureserver.net`. Image for deploy: +`forgejo.coulomb.social/coulomb/state-hub:f9f0091`. **Blocked on operator +approval** before Phase 2 freeze/restore. + ## Task: Working-memory and sink path decoupling ```task