coulomb/the-custodian
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1745c37f2cee2adbda2fda6b7f6413c8164c9a51
the-custodian/docs/workstation-independence-fleet-architecture.md
codex cf4be716e1 CUST-WP-0054 T01-T03: fleet architecture, de-hub runbook, drain plan 
Documents the three-machine role model, fleet mesh topology, coulombcore
freeze policy, and ordered drain sequence. Adds railiance01 systemd tunnel
install assets and refreshes ops service inventory to reflect 2026-07-03
production placement (cluster State Hub, fleet mesh, draining coulombcore).
2026-07-04 00:29:55 +02:00

298 lines
17 KiB
Markdown
Raw Blame History

# Workstation Independence and Fleet Role Architecture
Date: 2026-07-03
Status: draft (canon-adjacent; promote to `canon/architecture/` after review)
Workplan: `CUST-WP-0054` T01
Related: `ADR-001`, `ADR-004`, `RAIL-BS-WP-0006`, `RAIL-HO-WP-0005`, `CUST-WP-0011`
## Purpose
Fix the three-machine role model, the fleet mesh topology, the promotion gate
for "production", and the phoenix path `coulombcore → railiance02`. Provide a
dependency register so every workload, tunnel, repo remote, sink path, and
build pipeline has a **current host**, **target host**, and **migration owner**.
The acceptance proof for the whole plan is `CUST-WP-0054-T10`: production runs
24h+ with the workstation fully offline.
## Machine Roles
| Machine | IP / identity | Current role (2026-07-03) | Target role |
| --- | --- | --- | --- |
| **railiance01** | `92.205.62.239` | First ThreePhoenix foundation node; hosts activity-core production, partial State Hub cluster footprint, automation schedules | **Production home** — first node of the growing Railiance fleet; hosts State Hub primary, forge, CI runners, and the automation loop |
| **coulombcore** | `92.205.130.254` | De-facto production host: State Hub cluster primary, Core Hub (`hub.coulomb.social`), issue-core, OpenBao, identity stack, ESO/ArgoCD, Gitea/registry | **Frozen legacy** — no new production; drain workload-by-workload; eventually wiped and **reborn as railiance02** |
| **workstation** | `bnt-lap001` / WSL2 | Production network hub (all 16 ops-bridge tunnels), State Hub client endpoint (`127.0.0.1:8000`), consistency-sweep writebacks, image build/publish, dev checkouts for 74 registered repos | **Temporary dev environment** — clone repos, run `make dev-hub`, push when connected; nothing in the production loop may depend on it being on |
### Role invariants
1. Production workloads authenticate, schedule, emit, and reconcile without the
workstation.
2. `coulombcore` is frozen for new production immediately (policy; see T03).
3. A workload counts as "production on railiance01" only after passing the
staged-promotion gate (see below).
4. Files remain authoritative per ADR-001; fleet databases are disposable caches.
## Fleet Mesh Topology
### Current topology (workstation as hub)
All ops-bridge tunnels originate on the workstation. Two production data paths
**chain through** it:
```
railiance01 workstation coulombcore
─────────── ─────────── ───────────
activity-core ──(state-hub-railiance01 reverse)──► :18000 ──(state-hub-primary forward)──► State Hub cluster
activity-core ──(issue-core-railiance01 reverse)──► :local ──(issue-core-coulombcore forward)──► issue-core
```
Live tunnel inventory (2026-07-03, `bridge status`):
| Tunnel | Direction | Actor | Production-critical? |
| --- | --- | --- | --- |
| `state-hub-primary` | workstation → coulombcore cluster | `agt-claude-coulombcore` | **yes** — MCP/agents reach cluster hub via `127.0.0.1:8000` |
| `state-hub-cluster-coulombcore` | workstation → coulombcore | `agt-claude-coulombcore` | dev/ops access |
| `state-hub-railiance01` | railiance01 → workstation (reverse) | `agt-claude-railiance01` | **yes** — activity-core reaches hub |
| `state-hub-mcp-railiance01` | railiance01 → workstation (reverse) | `agt-claude-railiance01` | dev MCP |
| `issue-core-railiance01` | railiance01 → workstation (reverse) | `agt-claude-railiance01` | **yes** — emission lane |
| `issue-core-coulombcore` | workstation → coulombcore | `agt-claude-coulombcore` | **yes** — completes emission chain |
| `state-hub-coulombcore` | workstation → coulombcore | `agt-claude-coulombcore` | legacy/dev |
| `state-hub-mcp-coulombcore` | workstation → coulombcore | `agt-claude-coulombcore` | dev MCP |
| `k3s-api-coulombcore` | workstation → coulombcore | `agt-claude-coulombcore` | operator dev |
| `k3s-api-haskelseed` | workstation → haskelseed | `agt-claude-haskelseed` | experimental |
| `flex-auth-coulombcore` | workstation → coulombcore | `agt-claude-coulombcore` | identity dev |
| `core-hub-staging-coulombcore` | workstation → coulombcore | `agt-claude-coulombcore` | staging |
| `inter-hub-coulombcore` | workstation → coulombcore | `agt-claude-coulombcore` | legacy Inter-Hub |
| `state-hub-haskelseed` | haskelseed → workstation | `agt-claude-haskelseed` | experimental |
| `state-hub-mcp-haskelseed` | haskelseed → workstation | `agt-claude-haskelseed` | experimental |
| `nix-daemon-haskelseed` | haskelseed → workstation | `agt-claude-haskelseed` | build dev |
A workstation reboot breaks daily triage evidence, consistency sweeps, and
issue emission until tunnels recover.
### Target topology (fleet-owned mesh)
```
railiance01 ◄────────────────────────────────────► coulombcore (draining)
│ direct atm- tunnels (ops-bridge on-host) │
│ State Hub API │ legacy until drain complete
│ issue-core REST │
└─ activity-core, Temporal, sweep checkouts └─ identity, OpenBao (last to move)
workstation (optional client)
│ interactive-only: k3s API, hub read, dev-hub
└─ may disconnect without production impact
```
Implementation owner: `CUST-WP-0054-T02`.
Key changes:
- ops-bridge (or systemd ssh units) runs **on railiance01** with `atm-` actor
certs for cross-machine lanes.
- `actcore-state-hub-bridge` and `actcore-issue-core-bridge` point at
machine-local tunnel ports, not workstation forwards.
- Workstation tunnels remain for interactive dev only.
- Evaluate WireGuard mesh when persistent unit count exceeds ~5.
This posture extends ADR-004 (connectivity-first) from "workstation connects
everything" to "fleet machines connect each other; workstation is a client."
## Production Promotion Gate
A workload is **production on railiance01** only when it conforms to the
finished staged-promotion contract (`RAIL-BS-WP-0006`):
| Gate | Requirement |
| --- | --- |
| Overlay repo | `railiance/<app>/` with `app.toml` and stage manifests |
| Stage commands | `stage deploy`, `stage observe`, `stage promote`, `stage rollback` proven |
| Evidence | Backup/restore drill, canary observation, operator approval recorded |
| Registry | Image in forge OCI registry with immutable tag |
**Exceptions** must be documented in the placement plan (T03) with explicit
rollback. No exception bypasses backup evidence for stateful workloads.
`coulombcore` workloads still running in production today are **grandfathered
legacy** until their drain task completes — not newly promoted production.
## Phoenix Path: coulombcore → railiance02
Machine-scale phoenix rotation reuses the same automation intended for future
3-node weekly rotations (`RAIL-BS-WP-0007`, `CUST-WP-0038` deferred until
railiance02 exists).
### Preconditions (drain complete)
All production dependencies moved off coulombcore per T03 ordering:
1. Forge + CI (T04) — repos and images no longer depend on `gitea.coulomb.social`
2. State Hub primary (T05) — cluster DB and sweep checkouts on railiance01
3. Core Hub, issue-core, Inter-Hub legacy — per T03 sequence
4. Identity + OpenBao — **last** (everything authenticates through them)
### Phoenix execution
Owner: `CUST-WP-0054-T09`, automation: `CUST-WP-0054-T08`.
| Phase | Action | Tooling |
| --- | --- | --- |
| S0 | Final inventory sweep, DNS/cert plan for `*.coulomb.social`, data archival | T09 |
| S1 | Wipe and greenfield rebuild | `NET-WP-0020` unseal + bootstrap chain |
| S2 | Join as `railiance02` | `railiance-cluster` overlay, `atm-` certs |
| S3 | Prove join-ready | Phoenix drill on disposable target first (T08) |
Longhorn distributed storage and PG streaming HA unlock once railiance01 +
railiance02 are both fleet nodes.
## Dev Environment (Files-First Beachhead)
Strategy A from the workplan; owner: `CUST-WP-0054-T07`.
```
git clone → make dev-hub → local ephemeral hub (compose)
│
├─ C-06 registration rebuilds workplan/task state from files
├─ offline write buffer (STATE-WP-0068) for progress/task events
└─ reconnect relay upstream; files reconcile, databases do not replicate
```
MCP config gains explicit `dev` / `fleet` profile switch. The workstation is
genuinely temporary: no fleet DB sync required for orientation.
## Dependency Register
### Workloads
| Workload | Current host | Target host | Migration owner | Method / notes |
| --- | --- | --- | --- | --- |
| State Hub API (primary) | coulombcore CNPG cluster via workstation tunnel `state-hub-primary` → `127.0.0.1:8000` | railiance01 | `CUST-WP-0054-T05` | `CUST-WP-0011-T07` playbook: freeze → exact-count restore → rewire |
| State Hub API (WSL2 fallback) | workstation WSL2 | retired | `CUST-WP-0011-T08/T09` → absorbed by `CUST-WP-0054-T10` | Stabilization window; not part of target architecture |
| activity-core | railiance01 k3s (`activity-core` ns) | railiance01 (retain) | — | Already on target machine; fix bridges in T02 |
| issue-core | coulombcore k3s | railiance01 | `CUST-WP-0054-T03` drain seq. | `ISSUE-WP-0003` live; emission chain fixed in T02 |
| Core Hub | coulombcore (`hub.coulomb.social`) | railiance01 | `CORE-WP-0005` + `CUST-WP-0054-T03` | Staging on coulombcore; production cutover human-gated |
| Inter-Hub (legacy Haskell) | coulombcore external | retired | `CORE-WP-0007` | Rollback-only after Core Hub cutover |
| Gitea + OCI registry | coulombcore k3s | railiance01 Forgejo | `RAIL-HO-WP-0005` / `CUST-WP-0054-T04` | Read-only mirror on coulombcore until decommission |
| OpenBao | coulombcore | railiance01 | `CUST-WP-0054-T03` (last) | NET-WP-0020 unseal automation |
| Identity stack (KeyCape, Authelia, privacyIDEA, lldap) | coulombcore | railiance01 | `CUST-WP-0054-T03` (last) | Coupled to OpenBao |
| ESO + ArgoCD control plane | coulombcore | railiance01 | `CUST-WP-0054-T03` | GitOps follows forge move |
| CNPG databases (per workload) | coulombcore / railiance01 | railiance01 per workload | `CUST-WP-0054-T03`, `CUST-WP-0054-T05` | CNPG pattern proven; migrate with workload |
| llm-connect | TBD cluster | railiance01 | near-term lanes board | `CCR-2026-0003` credential lane active |
| ops-hub (widget/evidence) | files + Inter-Hub widgets | railiance01 via Core Hub | `CUST-WP-0025`, `CUST-WP-0049` | Not blocking workstation independence |
| Temporal (activity-core) | railiance01 | railiance01 (retain) | — | Co-locate with activity-core |
| NATS (activity-core) | railiance01 | railiance01 (retain) | — | Co-locate with activity-core |
### Network tunnels (production-critical)
| Lane | Current path | Target path | Owner |
| --- | --- | --- | --- |
| activity-core → State Hub | railiance01 reverse → workstation → `state-hub-primary` → coulombcore | railiance01 `atm-` forward → railiance01 State Hub (local or short hop) | `CUST-WP-0054-T02` |
| Agents/MCP → State Hub | workstation `127.0.0.1:8000` → `state-hub-primary` → coulombcore | workstation `127.0.0.1:8000` → tunnel to railiance01 hub (dev client) or fleet endpoint | `CUST-WP-0054-T05` + T07 profiles |
| railiance01 automations → State Hub | `:18000` chain via workstation | railiance01-local bridge port | `CUST-WP-0054-T02` |
| activity-core → issue-core | railiance01 reverse → workstation → `issue-core-coulombcore` | railiance01 `atm-` forward → issue-core (on railiance01 post-drain) | `CUST-WP-0054-T02`, then T03 |
| Operator k3s access | workstation forwards (`k3s-api-*`) | workstation interactive (non-critical) | — |
### Repo remotes
All checked 2026-07-03; pattern is uniform:
| Repo (sample) | Current remote | Target remote | Owner |
| --- | --- | --- | --- |
| the-custodian | `gitea.coulomb.social/coulomb/the-custodian.git` | `forgejo.coulomb.social/coulomb/the-custodian.git` | `CUST-WP-0054-T04` |
| state-hub | `gitea.coulomb.social/coulomb/state-hub.git` | `forgejo.coulomb.social/coulomb/state-hub.git` | `CUST-WP-0054-T04` |
| activity-core | `gitea.coulomb.social/coulomb/activity-core.git` | `forgejo.coulomb.social/coulomb/activity-core.git` | `CUST-WP-0054-T04` |
| issue-core | `gitea.coulomb.social/coulomb/issue-core.git` | `forgejo.coulomb.social/coulomb/issue-core.git` | `CUST-WP-0054-T04` |
| ops-bridge | `gitea.coulomb.social/coulomb/ops-bridge.git` | `forgejo.coulomb.social/coulomb/ops-bridge.git` | `CUST-WP-0054-T04` |
| ops-warden | `gitea.coulomb.social/coulomb/ops-warden.git` | `forgejo.coulomb.social/coulomb/ops-warden.git` | `CUST-WP-0054-T04` |
| core-hub | `gitea.coulomb.social/coulomb/core-hub.git` | `forgejo.coulomb.social/coulomb/core-hub.git` | `CUST-WP-0054-T04` |
| *(all 74 registered repos)* | `gitea.coulomb.social/coulomb/<slug>.git` | `forgejo.coulomb.social/coulomb/<slug>.git` | `CUST-WP-0054-T04` |
### State Hub repo checkout paths
| Concern | Current | Target | Owner |
| --- | --- | --- | --- |
| `local_path` for 74 repos | `/home/worsch/<repo>` on workstation | railiance01 clone tree (e.g. `/home/tegwick/<repo>` or gitops-managed path) | `CUST-WP-0054-T05` |
| Consistency sweep writeback host | workstation (`consistency_check.py --remote` via API) | railiance01 checkouts from forge | `CUST-WP-0054-T05`, `STATE-WP-0064` |
| COULOMBCORE `host_paths` | `/home/tegwick/<repo>` (11 repos, `CUST-WP-0021`) | retired with coulombcore drain | `CUST-WP-0054-T09` |
| Multi-host path resolution | `host_paths` map per hostname | fleet-primary host only + dev-hub local | `CUST-WP-0054-T07` |
### Sink and prompt paths
| Sink / path | Current | Target | Owner |
| --- | --- | --- | --- |
| Daily triage working-memory | `/home/worsch/the-custodian/memory/working` (ActivityDefinition + PVC mount) | repo-relative or PVC-native path + sweep sync-to-repo | `CUST-WP-0054-T06` |
| Daily triage State Hub progress | cluster hub via workstation tunnel | railiance01 hub direct | `CUST-WP-0054-T02`, `T05` |
| Consistency sweep progress event | via workstation-hosted sweep | railiance01-hosted sweep | `CUST-WP-0054-T05`, `STATE-WP-0064` |
| Agent session traces (`runtime/agent.py`) | `memory/working/agent-session-*.md` on workstation | dev-hub local buffer; commit on reconnect | `CUST-WP-0054-T07` |
| `output_schema` in ActivityDefinitions | absolute paths under `/home/worsch/the-custodian/` | repo-relative resolution in activity-core | `CUST-WP-0054-T06` |
### Build and publish pipelines
| Image / artifact | Current build host | Current registry | Target build | Target registry | Owner |
| --- | --- | --- | --- | --- | --- |
| state-hub | workstation `docker build` | `gitea.coulomb.social/coulomb/state-hub` | Forgejo Actions runner on railiance01 | railiance01 forge OCI | `CUST-WP-0054-T04` |
| core-hub | workstation / railiance-forge docs | `gitea.coulomb.social/coulomb/core-hub` | CI runner | railiance01 forge OCI | `CUST-WP-0054-T04` |
| activity-core | workstation manual rebuild + scp | railiance01 k3s import / Gitea | CI on tag push | railiance01 forge OCI | `CUST-WP-0054-T04` |
| issue-core | workstation / manual | `gitea.coulomb.social/coulomb/issue-core` | CI runner | railiance01 forge OCI | `CUST-WP-0054-T04` |
| Haskell build agent | workstation VM (`haskell-build-vm`) | n/a | retired (`CORE-WP-0007`) | n/a | `CORE-WP-0007` |
Done criterion for T01: every row above has a target and migration owner. ✓
## Drain Sequence
Detailed plan: `docs/coulombcore-drain-placement-plan.md`
Freeze policy: `canon/standards/coulombcore-production-freeze_v0.1.md`
```
Wave 1 Forge + CI (T04)
Wave 2 State Hub primary (T05)
Wave 3 Core Hub (CORE-WP-0005)
Wave 4 issue-core
Wave 5 ESO / ArgoCD
Wave 6 Supporting apps
Wave 7 OpenBao + identity (LAST)
Wave 8 coulombcore phoenix → railiance02 (T09)
```
## Sequencing Map
```
T01 (this document) ✓
├─ T02 de-hub network ✓
├─ T03 placement plan / freeze ✓
│ ├─ T04 forge + CI
│ └─ T05 State Hub home on railiance01
├─ T06 sink decoupling
├─ T07 dev beachhead
└─ T08 phoenix drill
└─ T09 coulombcore → railiance02
└─ T10 workstation-off acceptance
```
## Evidence and Inventory Sources
- Live tunnel state: `bridge status` (2026-07-03)
- State Hub health: `http://127.0.0.1:8000/state/health` (cluster primary via tunnel)
- Registered repos: `GET /repos/` — 74 repos, all `local_path` under `/home/worsch/`
- `ops/service-inventory.yml` (2026-06-05; predates cluster cutover — refresh in T03)
- `docs/infrastructure-stabilization-pickup-checkpoint.md` (2026-07-03 metaplan closeout)
- Activity definitions: `activity-definitions/daily-statehub-wsjf-triage.md`,
`activity-definitions/state-hub-consistency-sweep.md`
## Open Gaps (not T01 blockers)
| Gap | Follow-on |
| --- | --- |
| Forgejo production hostname / SMTP / exposure decisions | `RAIL-HO-WP-0005-T02` (human) |
| `ops/service-inventory.yml` stale environment labels | Refresh during T03 |
| Core Hub widget-type registry prerequisite | `CORE-WP-0005-T04` |
| HA Postgres / Longhorn across 2+ nodes | `RAIL-BS-WP-0007`, `CUST-WP-0038` after railiance02 |
## Promotion to Canon
After operator review:
1. Move to `canon/architecture/adr-006-workstation-independence-fleet-roles.md`
(or equivalent ADR number).
2. Update `ops/service-inventory.yml` environment and service rows to match.
3. Link from `SCOPE.md` and `.custodian-brief.md` generation inputs.
Reference in New Issue View Git Blame Copy Permalink