CUST-WP-0054 (proposed): workstation independence and fleet role realignment
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
codex
@@ -0,0 +1,289 @@
|
||||
---
|
||||
id: CUST-WP-0054
|
||||
type: workplan
|
||||
title: "Workstation Independence and Fleet Role Realignment"
|
||||
domain: infotech
|
||||
repo: the-custodian
|
||||
status: proposed
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
planning_priority: high
|
||||
planning_order: 54
|
||||
created: "2026-07-03"
|
||||
updated: "2026-07-03"
|
||||
---
|
||||
|
||||
# CUST-WP-0054 - Workstation Independence and Fleet Role Realignment
|
||||
|
||||
## Goal
|
||||
|
||||
Remove every runtime dependency and always-on workload from the workstation,
|
||||
realign machine roles, and make the production loop run uninterrupted with the
|
||||
workstation powered off:
|
||||
|
||||
| Machine | Target role |
|
||||
| --- | --- |
|
||||
| **railiance01** | Production workloads (the first node of the growing Railiance fleet). |
|
||||
| **coulombcore** | Early/legacy/experimental workloads only; frozen for new production; eventually decommissioned and **reborn as railiance02** (a machine-scale phoenix rotation). |
|
||||
| **workstation** | Local, temporary dev environment to build and evolve repos — nothing depends on it being on. |
|
||||
|
||||
The acceptance proof for the whole plan is T10: production runs for 24h+ with
|
||||
the workstation offline.
|
||||
|
||||
## Current-State Findings (2026-07-03 inventory)
|
||||
|
||||
These are the concrete dependencies this plan removes:
|
||||
|
||||
1. **The workstation is a production network hub.** All 16 ops-bridge tunnels
|
||||
originate here. Two production paths *chain through* it:
|
||||
- State Hub primary: everything reaches the cluster hub via the
|
||||
workstation's `state-hub-primary` forward tunnel, and railiance01's
|
||||
automations via `state-hub-railiance01` (reverse) → workstation → cluster.
|
||||
- issue-core emission: activity-core (railiance01) → workstation →
|
||||
issue-core (coulombcore).
|
||||
A workstation reboot breaks daily triage evidence, consistency sweeps, and
|
||||
task emission.
|
||||
2. **Production landed on the condemned machine.** The 2026-07-02/03 cutovers
|
||||
correctly moved State Hub and Core Hub off the workstation — but onto
|
||||
coulombcore (State Hub cluster primary, Core Hub behind
|
||||
`hub.coulomb.social`, issue-core, OpenBao, ArgoCD, ESO, the identity stack,
|
||||
and Gitea all run there). Under the new roles these are production
|
||||
workloads on a legacy machine.
|
||||
3. **Source of truth for code is on the condemned machine.** Every repo's
|
||||
canonical remote is `gitea.coulomb.social` (coulombcore). The Forgejo
|
||||
production migration (`RAIL-HO-WP-0005`) already targets railiance01.
|
||||
4. **The consistency sweep and daily triage depend on workstation repo
|
||||
checkouts** (`/home/worsch/*` clones are what fix-consistency writes back
|
||||
to) and the triage working-memory sink references a
|
||||
`/home/worsch/the-custodian/memory/working` path identity.
|
||||
5. **Images are built and published from the workstation** (docker build +
|
||||
push by hand); there is no CI runner, so releases require this machine.
|
||||
6. **WSL2 State Hub fallback** still exists on the workstation
|
||||
(`CUST-WP-0011-T08/T09` stabilization window) — expected, retired by that
|
||||
workplan, not this one.
|
||||
|
||||
## Deployment Strategy (proposed — improves on the rough beachhead idea)
|
||||
|
||||
**A. Files-first beachhead, not database replication.** ADR-001 already
|
||||
declares that the State Hub is a read model rebuildable from repo files. So
|
||||
the dev-environment story should be: `make dev-hub` starts a **local ephemeral
|
||||
hub** (compose: postgres + API + MCP, exactly what exists today) whose content
|
||||
is **rebuilt from the local repo checkouts** via the C-06 registration path —
|
||||
not synced from the fleet database. Offline writes (progress events, task
|
||||
status) accumulate in a **write buffer** and relay upstream on reconnect
|
||||
(`STATE-WP-0068 offline-write-buffer-and-edge-relay` is the existing seed for
|
||||
this). Two hubs never replicate; files reconcile them. This makes the
|
||||
workstation genuinely temporary: clone repos → `make dev-hub` → work offline →
|
||||
push + relay when connected.
|
||||
|
||||
**B. Fleet mesh instead of workstation-hub networking.** Machine-to-machine
|
||||
paths (railiance01 ↔ coulombcore/railiance02) get direct, persistent links
|
||||
owned by the machines themselves — ops-bridge units running *on the fleet
|
||||
machines* under `atm-` actor certs (ops-warden), or a minimal WireGuard mesh
|
||||
if tunnel count grows. The workstation becomes a mesh *client* when present,
|
||||
never a relay.
|
||||
|
||||
**C. Promotion-gated production, ThreePhoenix ideas incrementally.** A
|
||||
workload counts as "production on railiance01" only when it conforms to the
|
||||
already-finished staged-promotion contract (`RAIL-BS-WP-0006`: overlay repo,
|
||||
`railiance/app.toml`, stage commands, rollback). ThreePhoenix ideas adopted
|
||||
now, without waiting for three nodes:
|
||||
- **CNPG-managed Postgres** per workload (already proven on coulombcore);
|
||||
- **greenfield rebuild automation** (NET-WP-0020 unseal + bootstrap chain) as
|
||||
the standing "phoenix drill" for single machines;
|
||||
- **phoenix rotation at machine scale**: the coulombcore → railiance02
|
||||
rebirth *is* the first full phoenix and should be executed with the
|
||||
same automation that a future 3-node weekly rotation would use;
|
||||
- Longhorn distributed storage and PG streaming HA remain deferred until a
|
||||
third node exists (`RAIL-BS-WP-0007` / `CUST-WP-0038` stay the follow-on).
|
||||
|
||||
**D. Decommission by attrition, not by big-bang.** coulombcore is frozen for
|
||||
new production immediately (policy), drained workload-by-workload via staged
|
||||
promotion onto railiance01, and only rebuilt as railiance02 when the last
|
||||
production dependency (likely identity/OpenBao) has moved.
|
||||
|
||||
## Task: Target architecture and dependency register
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Write the canon-adjacent architecture note (`docs/` first; promote to
|
||||
`canon/architecture/` after review) fixing the three machine roles, the fleet
|
||||
mesh topology, the promotion gate for "production", and the phoenix path
|
||||
coulombcore → railiance02. Include the full dependency register: every
|
||||
workload, tunnel, repo remote, sink path, and build pipeline with its current
|
||||
host and target host. Done when every row has a target and a migration owner
|
||||
(this plan's task or an existing workplan reference).
|
||||
|
||||
## Task: De-hub the network — fleet-owned direct tunnels
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Remove the workstation from all production data paths:
|
||||
|
||||
- Run ops-bridge (or systemd ssh units) **on railiance01** with `atm-` actor
|
||||
certs for the two live cross-machine lanes: railiance01 → coulombcore
|
||||
issue-core, and railiance01 → cluster State Hub (replacing the
|
||||
workstation-chained `state-hub-railiance01` + `state-hub-primary` pair).
|
||||
- Re-point `actcore-state-hub-bridge` and `actcore-issue-core-bridge` at the
|
||||
machine-local tunnel ports.
|
||||
- Workstation tunnels remain only for interactive dev access (k3s API, hub
|
||||
client) and may drop at any time without production impact.
|
||||
- Evaluate WireGuard mesh as the successor if unit count exceeds ~5.
|
||||
|
||||
Done when killing every workstation tunnel leaves triage, sweeps, and
|
||||
emission working (partial T10 rehearsal).
|
||||
|
||||
## Task: Production placement plan and freeze policy
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Declare coulombcore frozen for new production (policy note in canon). Produce
|
||||
the drain sequence with per-workload target and method (staged-promotion
|
||||
overlay or documented exception): State Hub, Core Hub, issue-core, OpenBao,
|
||||
identity stack (KeyCape/Authelia/privacyIDEA/lldap), ESO/ArgoCD control
|
||||
plane, Gitea/Forgejo, CNPG databases. Explicitly order them by coupling
|
||||
(forge and State Hub early; identity + OpenBao last, since everything
|
||||
authenticates through them).
|
||||
|
||||
## Task: Forge to railiance01 + CI runners (kill workstation builds)
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Execute/absorb `RAIL-HO-WP-0005`: Forgejo production on railiance01 becomes
|
||||
the canonical remote for all repos; coulombcore Gitea becomes a read-only
|
||||
mirror until decommission. Stand up Actions runners so container images
|
||||
(state-hub, core-hub, issue-core, activity-core) build and push in CI from
|
||||
tags — the workstation stops being the build/publish host. Done when a
|
||||
release ships with the workstation off.
|
||||
|
||||
## Task: State Hub production home on railiance01
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T05
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Move the State Hub primary from coulombcore to railiance01 using the proven
|
||||
CUST-WP-0011-T07 playbook (freeze → exact-count restore → rewire). This makes
|
||||
the automation loop machine-local: activity-core, Temporal, and the hub share
|
||||
one machine, so daily triage and sweeps survive any other machine being down.
|
||||
Prereq: railiance01 CNPG + storage reviewed (T03). Also relocate the
|
||||
consistency-sweep repo checkouts to railiance01 (clones from the T04 forge)
|
||||
so file writebacks no longer touch workstation paths.
|
||||
|
||||
## Task: Working-memory and sink path decoupling
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Remove `/home/worsch/...` path identities from runtime contracts: the triage
|
||||
working-memory sink and any prompt/context paths become repo-relative or
|
||||
PVC-native with a defined sync-to-repo step (commit via the sweep). Done when
|
||||
no ActivityDefinition or sink references a workstation-specific absolute path.
|
||||
|
||||
## Task: Dev-environment beachhead (files-first)
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T07
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Implement strategy A: `make dev-hub` (or `custodian dev up`) starts the local
|
||||
compose hub, registers the locally cloned repos, and rebuilds workplan/task
|
||||
state from files via C-06 — no fleet connection required. Implement the
|
||||
offline write buffer + reconnect relay for progress/task events (align with
|
||||
`STATE-WP-0068`; keep the buffer file-backed and idempotent). MCP config
|
||||
gains an explicit `dev`/`fleet` profile switch. Done when a fresh machine
|
||||
reaches a working, orientation-capable dev hub from `git clone` + one command,
|
||||
fully offline.
|
||||
|
||||
## Task: ThreePhoenix increment — phoenix drill automation
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T08
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Compose the existing pieces (NET-WP-0020 unseal automation, S1–S3 bootstrap
|
||||
chain, staged-promotion overlays, CNPG restore drills) into one rehearsable
|
||||
"phoenix a machine" runbook + automation entrypoint, proven on a disposable
|
||||
target (haskelseed or a VM). This is the tool the railiance02 rebirth and any
|
||||
future node rotation will use. Done when a greenfield machine reaches
|
||||
join-ready state unattended except for custody-gated steps.
|
||||
|
||||
## Task: coulombcore decommission readiness → railiance02
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T09
|
||||
status: wait
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Gated on T03–T05 drains reaching identity/OpenBao. Final inventory sweep,
|
||||
data archival (episodic memory of the machine's history), DNS/cert plan for
|
||||
`*.coulomb.social` names, then execute the machine phoenix via T08 automation:
|
||||
wipe, rebuild as railiance02, join the fleet. Longhorn/PG-HA
|
||||
(`RAIL-BS-WP-0007`, `CUST-WP-0038`) unlock once railiance01 + railiance02 are
|
||||
both fleet nodes.
|
||||
|
||||
## Task: Workstation-off acceptance test
|
||||
|
||||
```task
|
||||
id: CUST-WP-0054-T10
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
The plan's proof: workstation fully offline for 24h+ (no tunnels, no
|
||||
processes). Verify afterwards from evidence alone: scheduled triage ran and
|
||||
validated, consistency sweeps ran, issue emission works, hub/API/dashboards
|
||||
served, forge and CI available. Then verify the inverse: workstation boots,
|
||||
`custodian dev up` gives a working offline dev hub, and reconnect relays
|
||||
buffered events. Done when both directions pass without manual repair.
|
||||
|
||||
## Sequencing
|
||||
|
||||
```
|
||||
T01 (architecture + register)
|
||||
├─ T02 de-hub network ── unblocks most of T10's first half
|
||||
├─ T03 placement plan/freeze ─┬─ T04 forge + CI ─┐
|
||||
│ └─ T05 hub home ├─ T09 decommission → railiance02
|
||||
├─ T06 sink decoupling │
|
||||
├─ T07 dev beachhead │
|
||||
└─ T08 phoenix drill ────────────────────────────┘
|
||||
T10 acceptance (both halves)
|
||||
```
|
||||
|
||||
## Relationship to existing workplans
|
||||
|
||||
- Absorbs the *sequencing* of `RAIL-HO-WP-0005` (Forgejo) and
|
||||
`CUST-WP-0011-T08/T09` (WSL2 fallback retirement folds into T10).
|
||||
- Builds on finished `RAIL-BS-WP-0006` (staged promotion) and `NET-WP-0020`
|
||||
(unseal automation).
|
||||
- Defers to `RAIL-BS-WP-0007` / `CUST-WP-0038` for true 3-node HA once
|
||||
railiance02 exists.
|
||||
- Extends `STATE-WP-0068` (offline write buffer) as the T07 relay mechanism.
|
||||
- `CORE-WP-0005` stabilization + `CORE-WP-0007` Haskell retirement proceed
|
||||
independently; Core Hub simply appears in the T03 drain sequence.
|
||||
Reference in New Issue
Block a user