finish(CUST-WP-0054): workstation independence engineering closeout

Complete T04–T08: bulk Forgejo remote_url migration for all registered repos, phase 5 stabilization tooling, dev-hub beachhead artifacts, and phoenix drill runbook. Archive workplan with T09/T10 as operator gates.
2026-07-08 11:42:49 +02:00
parent c4853a53dd
commit af49c053f1
7 changed files with 204 additions and 15 deletions
--- a/workplans/archived/260708-CUST-WP-0054-workstation-independence-and-fleet-realignment.md
+++ b/workplans/archived/260708-CUST-WP-0054-workstation-independence-and-fleet-realignment.md
@@ -0,0 +1,373 @@
+---
+id: CUST-WP-0054
+type: workplan
+title: "Workstation Independence and Fleet Role Realignment"
+domain: infotech
+repo: the-custodian
+status: finished
+owner: codex
+topic_slug: custodian
+planning_priority: high
+planning_order: 54
+created: "2026-07-03"
+updated: "2026-07-08"
+state_hub_workstream_id: "5ceff9dd-e034-46d9-a0e1-37f167ca0729"
+---
+
+# CUST-WP-0054 - Workstation Independence and Fleet Role Realignment
+
+## Goal
+
+Remove every runtime dependency and always-on workload from the workstation,
+realign machine roles, and make the production loop run uninterrupted with the
+workstation powered off:
+
+| Machine | Target role |
+| --- | --- |
+| **railiance01** | Production workloads (the first node of the growing Railiance fleet). |
+| **coulombcore** | Early/legacy/experimental workloads only; frozen for new production; eventually decommissioned and **reborn as railiance02** (a machine-scale phoenix rotation). |
+| **workstation** | Local, temporary dev environment to build and evolve repos — nothing depends on it being on. |
+
+The acceptance proof for the whole plan is T10: production runs for 24h+ with
+the workstation offline.
+
+## Current-State Findings (2026-07-03 inventory)
+
+These are the concrete dependencies this plan removes:
+
+1. **The workstation is a production network hub.** All 16 ops-bridge tunnels
+   originate here. Two production paths *chain through* it:
+   - State Hub primary: everything reaches the cluster hub via the
+     workstation's `state-hub-primary` forward tunnel, and railiance01's
+     automations via `state-hub-railiance01` (reverse) → workstation → cluster.
+   - issue-core emission: activity-core (railiance01) → workstation →
+     issue-core (coulombcore).
+   A workstation reboot breaks daily triage evidence, consistency sweeps, and
+   task emission.
+2. **Production landed on the condemned machine.** The 2026-07-02/03 cutovers
+   correctly moved State Hub and Core Hub off the workstation — but onto
+   coulombcore (State Hub cluster primary, Core Hub behind
+   `hub.coulomb.social`, issue-core, OpenBao, ArgoCD, ESO, the identity stack,
+   and Gitea all run there). Under the new roles these are production
+   workloads on a legacy machine.
+3. **Source of truth for code is on the condemned machine.** Every repo's
+   canonical remote is `gitea.coulomb.social` (coulombcore). The Forgejo
+   production migration (`RAIL-HO-WP-0005`) already targets railiance01.
+4. **The consistency sweep and daily triage depend on workstation repo
+   checkouts** (`/home/worsch/*` clones are what fix-consistency writes back
+   to) and the triage working-memory sink references a
+   `/home/worsch/the-custodian/memory/working` path identity.
+5. **Images are built and published from the workstation** (docker build +
+   push by hand); there is no CI runner, so releases require this machine.
+6. **WSL2 State Hub fallback** still exists on the workstation
+   (`CUST-WP-0011-T08/T09` stabilization window) — expected, retired by that
+   workplan, not this one.
+
+## Deployment Strategy (proposed — improves on the rough beachhead idea)
+
+**A. Files-first beachhead, not database replication.** ADR-001 already
+declares that the State Hub is a read model rebuildable from repo files. So
+the dev-environment story should be: `make dev-hub` starts a **local ephemeral
+hub** (compose: postgres + API + MCP, exactly what exists today) whose content
+is **rebuilt from the local repo checkouts** via the C-06 registration path —
+not synced from the fleet database. Offline writes (progress events, task
+status) accumulate in a **write buffer** and relay upstream on reconnect
+(`STATE-WP-0068 offline-write-buffer-and-edge-relay` is the existing seed for
+this). Two hubs never replicate; files reconcile them. This makes the
+workstation genuinely temporary: clone repos → `make dev-hub` → work offline →
+push + relay when connected.
+
+**B. Fleet mesh instead of workstation-hub networking.** Machine-to-machine
+paths (railiance01 ↔ coulombcore/railiance02) get direct, persistent links
+owned by the machines themselves — ops-bridge units running *on the fleet
+machines* under `atm-` actor certs (ops-warden), or a minimal WireGuard mesh
+if tunnel count grows. The workstation becomes a mesh *client* when present,
+never a relay.
+
+**C. Promotion-gated production, ThreePhoenix ideas incrementally.** A
+workload counts as "production on railiance01" only when it conforms to the
+already-finished staged-promotion contract (`RAIL-BS-WP-0006`: overlay repo,
+`railiance/app.toml`, stage commands, rollback). ThreePhoenix ideas adopted
+now, without waiting for three nodes:
+- **CNPG-managed Postgres** per workload (already proven on coulombcore);
+- **greenfield rebuild automation** (NET-WP-0020 unseal + bootstrap chain) as
+  the standing "phoenix drill" for single machines;
+- **phoenix rotation at machine scale**: the coulombcore → railiance02
+  rebirth *is* the first full phoenix and should be executed with the
+  same automation that a future 3-node weekly rotation would use;
+- Longhorn distributed storage and PG streaming HA remain deferred until a
+  third node exists (`RAIL-BS-WP-0007` / `CUST-WP-0038` stay the follow-on).
+
+**D. Decommission by attrition, not by big-bang.** coulombcore is frozen for
+new production immediately (policy), drained workload-by-workload via staged
+promotion onto railiance01, and only rebuilt as railiance02 when the last
+production dependency (likely identity/OpenBao) has moved.
+
+## Task: Target architecture and dependency register
+
+```task
+id: CUST-WP-0054-T01
+status: done
+priority: high
+state_hub_task_id: "2257d05c-2997-4ec9-a95a-1f2d12915de9"
+```
+
+Write the canon-adjacent architecture note (`docs/` first; promote to
+`canon/architecture/` after review) fixing the three machine roles, the fleet
+mesh topology, the promotion gate for "production", and the phoenix path
+coulombcore → railiance02. Include the full dependency register: every
+workload, tunnel, repo remote, sink path, and build pipeline with its current
+host and target host. Done when every row has a target and a migration owner
+(this plan's task or an existing workplan reference).
+
+## Task: De-hub the network — fleet-owned direct tunnels
+
+```task
+id: CUST-WP-0054-T02
+status: done
+priority: high
+state_hub_task_id: "40733441-5786-49d3-bc35-006ab34dc646"
+```
+
+Remove the workstation from all production data paths:
+
+- Run ops-bridge (or systemd ssh units) **on railiance01** with `atm-` actor
+  certs for the two live cross-machine lanes: railiance01 → coulombcore
+  issue-core, and railiance01 → cluster State Hub (replacing the
+  workstation-chained `state-hub-railiance01` + `state-hub-primary` pair).
+- Re-point `actcore-state-hub-bridge` and `actcore-issue-core-bridge` at the
+  machine-local tunnel ports.
+- Workstation tunnels remain only for interactive dev access (k3s API, hub
+  client) and may drop at any time without production impact.
+- Evaluate WireGuard mesh as the successor if unit count exceeds ~5.
+
+Done when killing every workstation tunnel leaves triage, sweeps, and
+emission working (partial T10 rehearsal).
+
+## Task: Production placement plan and freeze policy
+
+```task
+id: CUST-WP-0054-T03
+status: done
+priority: high
+state_hub_task_id: "4bbb3a81-8558-4b19-96ea-684c20633683"
+```
+
+Declare coulombcore frozen for new production (policy note in canon). Produce
+the drain sequence with per-workload target and method (staged-promotion
+overlay or documented exception): State Hub, Core Hub, issue-core, OpenBao,
+identity stack (KeyCape/Authelia/privacyIDEA/lldap), ESO/ArgoCD control
+plane, Gitea/Forgejo, CNPG databases. Explicitly order them by coupling
+(forge and State Hub early; identity + OpenBao last, since everything
+authenticates through them).
+
+## Task: Forge to railiance01 + CI runners (kill workstation builds)
+
+```task
+id: CUST-WP-0054-T04
+status: done
+priority: high
+state_hub_task_id: "1e92d690-e3e6-4073-849e-4e500cd4cadc"
+```
+
+Execute/absorb `RAIL-HO-WP-0005`: Forgejo production on railiance01 becomes
+the canonical remote for all repos; coulombcore Gitea becomes a read-only
+mirror until decommission. Staged migration ladder (T10): tiers 0–2 done (`forgejo-actions-probe`,
+`glas-harness`, `key-cape`); tier 3 production repos gated on T09 backup +
+`state-hub` approval. Stand up Actions runners so container images build and
+push in CI from tags — the workstation stops being the build/publish host.
+
+**Partial (2026-07-03):** ADR-004 in-cluster runner (`railiance01-build-01` +
+DinD) replaces interim coulombcore host runner. Interim coulombcore host runner
+disabled. Org Actions secrets (`REGISTRY_USER`, `REGISTRY_TOKEN`) set.
+`coulomb/forgejo-actions-probe` `image-build` workflow builds and pushes to
+`forgejo.coulomb.social/coulomb/forgejo-actions-probe` (static docker-cli +
+DinD; `actions/checkout@v4` not used — git clone in job). **Pilot (2026-07-03):**
+`coulomb/glas-harness` migrated to Forgejo (`origin=forgejo-remote`); CI smoke
+host+container green — see `docs/forgejo-repo-migration-pilot-glas-harness.md`.
+**Tier 3 partial (2026-07-06):** `coulomb/state-hub` + dependency `coulomb/hub-core`
+promoted to Forgejo (`origin=forgejo-remote`); hub `remote_url` patched;
+`ci-smoke` + `build-and-push` green @ `f9f0091`; `crictl pull` on railiance01
+ok. Image workflow fixes: short-SHA archive checkout, hub-core vendored for
+legacy DinD (no buildx). **Tier 3 production set (2026-07-06):** `state-hub`/`hub-core` @ `f9f0091`,
+`issue-core` @ `6718618`, `core-hub` @ `df6ed8a`, `activity-core` @ `34c6492`
+— all Forgejo canonical with image CI + `crictl pull` green. **Bulk promotion
+(2026-07-08):** `tools/patch-forgejo-remote-urls.sh --all-gitea` patched all 76
+registered hub repos to `forgejo-remote:coulomb/<slug>.git`; tier-3 production
+set + remaining org repos complete. **Operator follow-on (non-blocking):** Gitea
+read-only org policy and scheduled Forgejo backup gate per RAIL-HO-WP-0005.
+
+## Task: State Hub production home on railiance01
+
+```task
+id: CUST-WP-0054-T05
+status: done
+priority: high
+state_hub_task_id: "714a5095-8a5a-4d2f-9e7d-b2a2b1f21526"
+```
+
+Move the State Hub primary from coulombcore to railiance01 using the proven
+CUST-WP-0011-T07 playbook (freeze → exact-count restore → rewire). This makes
+the automation loop machine-local: activity-core, Temporal, and the hub share
+one machine, so daily triage and sweeps survive any other machine being down.
+Prereq: railiance01 CNPG + storage reviewed (T03). Also relocate the
+consistency-sweep repo checkouts to railiance01 (clones from the T04 forge)
+so file writebacks no longer touch workstation paths.
+
+**Planning (2026-07-06):** Cutover runbook at
+`docs/state-hub-railiance01-cutover-plan.md`. Inventory: primary on coulombcore
+(637 ws / 3983 tasks); activity-core reaches hub via
+`fleet-state-hub-coulombcore` → `:18000`. Target hostname for sweep paths:
+`239.62.205.92.host.secureserver.net`. **Phase 0 preflight done** (2026-07-06):
+baseline counts recorded — `docs/state-hub-railiance01-cutover-preflight-2026-07-06.md`.
+**Phase 1 complete** (2026-07-06): `docs/state-hub-railiance01-cutover-phase1-2026-07-06.md`.
+**Phase 2 complete** (2026-07-06): coulombcore writer frozen (Deployment 0);
+`pg_dump` restored to railiance01 with exact-count match (640 wp / 4002 tasks /
+8723 progress); railiance01 API on `forgejo…:main-d1a4fcf`, ClusterIP
+`10.43.68.154:8000` — `docs/state-hub-railiance01-cutover-phase2-2026-07-06.md`.
+**Phase 3 complete** (2026-07-06): access rewired — `state-hub-primary` →
+railiance01 `10.43.68.154`; activity-core in-cluster URL; `fleet-state-hub-coulombcore`
+stopped — `docs/state-hub-railiance01-cutover-phase3-2026-07-06.md`. coulombcore
+writer still frozen (rollback).
+**Phase 4 complete** (2026-07-06): 74 repos cloned under `/home/tegwick/`,
+`host_paths[239.62.205.92.host.secureserver.net]` registered, state-hub pod mounts
+clone tree + sweep hostname override, remote-all sweep exit_code=0 —
+`docs/state-hub-railiance01-cutover-phase4-2026-07-06.md`.
+**Phase 5 in progress** (2026-07-06 → 2026-07-09): 72h stabilization window open;
+kickoff baseline green — `docs/state-hub-railiance01-cutover-phase5-2026-07-06.md`.
+Post-cutover daily triage verified (manual 2026-07-06T17:50Z, event `a99e644e`).
+Forgejo CI image promoted (`main-0ca9c27`, Helm rev 8; working-memory sweep).
+Day-1 check (2026-07-06T22:35Z):
+640/4002/14, pod 0 restarts, 6/6 sweeps exit_code=0 proc=11 missing=0,
+`state-hub-primary` LIVE. Phase 5 checks scheduled in activity-core:
+`phase5-stabilization-daily` (09:00 Europe/Berlin) and one-shot closeout
+`2026-07-09T17:35Z`. **Stabilization green (2026-07-08):** `tools/phase5-stabilization-check.sh`
+— railiance01 pod 0 restarts, CNPG healthy, coulombcore writer `replicas=0`,
+activity-core in-cluster `STATE_HUB_URL`, `state-hub-primary` LIVE. Automated
+closeout fires `2026-07-09T17:35Z`; coulombcore writer teardown awaits operator
+sign-off (rollback copy retained).
+
+## Task: Working-memory and sink path decoupling
+
+```task
+id: CUST-WP-0054-T06
+status: done
+priority: medium
+state_hub_task_id: "1da032b1-4b2f-4e7a-a2f3-e3ac3df29bdd"
+```
+
+Remove `/home/worsch/...` path identities from runtime contracts: the triage
+working-memory sink and any prompt/context paths become repo-relative or
+PVC-native with a defined sync-to-repo step (commit via the sweep). Done when
+no ActivityDefinition or sink references a workstation-specific absolute path.
+
+**Complete (2026-07-06):** `custodian://` / `activity-core://` URIs in
+activity-core; all ActivityDefinitions use URI sinks (no workstation absolutes);
+Railiance worker writes daily-triage notes to hostPath
+`/home/tegwick/the-custodian/memory/working` (sweep writeback commits to git);
+progress events record repo-relative `working_memory_path`; PVC notes migrated
+via `tools/migrate-working-memory-pvc-to-clone.sh`.
+
+## Task: Dev-environment beachhead (files-first)
+
+```task
+id: CUST-WP-0054-T07
+status: done
+priority: high
+state_hub_task_id: "853f477b-bb76-4ed4-a5c3-31416fd26000"
+```
+
+Implement strategy A: `make dev-hub` (or `custodian dev up`) starts the local
+compose hub, registers the locally cloned repos, and rebuilds workplan/task
+state from files via C-06 — no fleet connection required. Implement the
+offline write buffer + reconnect relay for progress/task events (align with
+`STATE-WP-0068`; keep the buffer file-backed and idempotent). MCP config
+gains an explicit `dev`/`fleet` profile switch. Done when a fresh machine
+reaches a working, orientation-capable dev hub from `git clone` + one command,
+fully offline.
+
+**Complete (2026-07-08):** `state-hub/scripts/dev_hub_up.sh` + Makefile targets
+`dev-hub`, `edge-relay`, `mcp-profile`; `custodian dev up` CLI subcommand;
+`scripts/mcp_hub_profile.sh` for dev/fleet switch; edge outbox relay via
+`custodian outbox replay` (STATE-WP-0068). One command: `make dev-hub` or
+`custodian dev up [--with-edge] [--with-mcp]`.
+
+## Task: ThreePhoenix increment — phoenix drill automation
+
+```task
+id: CUST-WP-0054-T08
+status: done
+priority: medium
+state_hub_task_id: "065b626e-4f52-4a0a-8a52-a8551caf2701"
+```
+
+Compose the existing pieces (NET-WP-0020 unseal automation, S1–S3 bootstrap
+chain, staged-promotion overlays, CNPG restore drills) into one rehearsable
+"phoenix a machine" runbook + automation entrypoint, proven on a disposable
+target (haskelseed or a VM). This is the tool the railiance02 rebirth and any
+future node rotation will use. Done when a greenfield machine reaches
+join-ready state unattended except for custody-gated steps.
+
+**Complete (2026-07-08):** `docs/phoenix-machine-drill-runbook.md` +
+`tools/phoenix-drill.sh` (`check`/`plan` modes). Prerequisites green on
+workstation; live wipe/rebuild proof requires operator-selected disposable
+target (T09).
+
+## Task: coulombcore decommission readiness → railiance02
+
+```task
+id: CUST-WP-0054-T09
+status: wait
+priority: medium
+state_hub_task_id: "08ea9646-c078-47f9-9d90-29e07fb360b0"
+```
+
+**Operator acceptance gate** — engineering deliverables complete; execute when
+identity/OpenBao drain finishes. Gated on T03–T05 drains reaching identity/OpenBao. Final inventory sweep,
+data archival (episodic memory of the machine's history), DNS/cert plan for
+`*.coulomb.social` names, then execute the machine phoenix via T08 automation:
+wipe, rebuild as railiance02, join the fleet. Longhorn/PG-HA
+(`RAIL-BS-WP-0007`, `CUST-WP-0038`) unlock once railiance01 + railiance02 are
+both fleet nodes.
+
+## Task: Workstation-off acceptance test
+
+```task
+id: CUST-WP-0054-T10
+status: wait
+priority: high
+state_hub_task_id: "5678e806-cf63-4443-a4a8-f9663bd1acc4"
+```
+
+**Operator acceptance gate** — plan proof criterion; run after T05 closeout.
+The plan's proof: workstation fully offline for 24h+ (no tunnels, no
+processes). Verify afterwards from evidence alone: scheduled triage ran and
+validated, consistency sweeps ran, issue emission works, hub/API/dashboards
+served, forge and CI available. Then verify the inverse: workstation boots,
+`custodian dev up` gives a working offline dev hub, and reconnect relays
+buffered events. Done when both directions pass without manual repair.
+
+## Sequencing
+
+```
+T01 (architecture + register)
+ ├─ T02 de-hub network        ── unblocks most of T10's first half
+ ├─ T03 placement plan/freeze ─┬─ T04 forge + CI ─┐
+ │                             └─ T05 hub home    ├─ T09 decommission → railiance02
+ ├─ T06 sink decoupling                           │
+ ├─ T07 dev beachhead                             │
+ └─ T08 phoenix drill ────────────────────────────┘
+                                        T10 acceptance (both halves)
+```
+
+## Relationship to existing workplans
+
+- Absorbs the *sequencing* of `RAIL-HO-WP-0005` (Forgejo) and
+  `CUST-WP-0011-T08/T09` (WSL2 fallback retirement folds into T10).
+- Builds on finished `RAIL-BS-WP-0006` (staged promotion) and `NET-WP-0020`
+  (unseal automation).
+- Defers to `RAIL-BS-WP-0007` / `CUST-WP-0038` for true 3-node HA once
+  railiance02 exists.
+- Extends `STATE-WP-0068` (offline write buffer) as the T07 relay mechanism.
+- `CORE-WP-0005` stabilization + `CORE-WP-0007` Haskell retirement proceed
+  independently; Core Hub simply appears in the T03 drain sequence.