coulomb/the-custodian
Archived
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated workplans for migrating the custodian to Railiance01

Browse Source
This commit is contained in:
Bernd Worsch
2026-05-02 23:38:56 +02:00
parent 94916cbfb0
commit dd887c8c81
2 changed files with 418 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files

363
workplans/CUST-WP-0011-state-hub-threephoenix-migration.md
View File

@@ -1,80 +1,107 @@
---
id: CUST-WP-0011
type: workplan
title: "Migrate Custodian State Hub to ThreePhoenix Cluster"
title: "Pragmatic State Hub Migration to railiance01"
domain: custodian
repo: the-custodian
status: active
owner: custodian
topic_slug: custodian
created: "2026-03-11"
updated: "2026-03-11"
updated: "2026-05-02"
state_hub_workstream_id: "967baafb-d92d-405a-ba0b-0d00d37c4940"
supersedes_intent_from: "Migrate Custodian State Hub to ThreePhoenix Cluster"
follow_up_workplan: CUST-WP-0038
---
# Migrate Custodian State Hub to ThreePhoenix Cluster
# Pragmatic State Hub Migration to railiance01
## Goal
Move the Custodian State Hub (FastAPI + PostgreSQL) from its current home on
the WSL2 operator workstation to the ThreePhoenix Kubernetes cluster
(Railiance01/02/03), making it available to Claude Code sessions running on
any machine with cluster access — without public internet exposure.
Move the Custodian State Hub (FastAPI + PostgreSQL) from the WSL2 operator
workstation to the current railiance01 Kubernetes environment, using the
Railiance production-readiness path that exists now:
The State Hub is **irreplaceable episodic memory**. This migration must be
executed with zero tolerance for data loss and a tested rollback path at
every stage.
- CloudNative PG (`cnpg`) for the State Hub database in the `databases`
namespace.
- State Hub as an S5 workload in `railiance-apps`.
- Platform/database ownership in `railiance-platform`.
- Access through the existing private tunnel/ops-bridge model, not public
exposure.
- WSL2 retained as a disaster-recovery fallback until the cluster deployment
has proven stable.
## Pre-conditions (gate — do not start until all satisfied)
This is a deliberate pragmatic step. It improves durability and multi-machine
access before the full ThreePhoenix target is ready. The ultimate multi-node,
replicated, long-term cluster goal is preserved in `CUST-WP-0038`.
- [ ] ThreePhoenix cluster has three healthy nodes (Railiance01 confirmed, Railiance02 + Railiance03 joined)
- [ ] Longhorn distributed storage installed and verified (replication factor ≥ 2)
- [ ] HA failover test passes (`tests/test_ha_failover.sh` exits 0 on the cluster)
- [ ] S2 integrated backup operational and tested on the cluster
- [ ] A full WSL2 State Hub backup has been taken and restore-drilled **within 24h of starting this workplan**
## Context Update
These gates are mandatory. A single-node cluster or unverified storage is not
an acceptable migration target for the Custodian.
The original 2026-03-11 version of this workplan targeted a future
ThreePhoenix cluster with Railiance01/02/03, Longhorn, and full HA gates before
starting. That was correct as an end-state, but it blocks useful progress now.
## Architecture after migration
The current Railiance architecture has moved on:
- `railiance-infra/workplans/RAIL-HO-WP-0004-production-readiness.md`
supersedes the older Bitnami PostgreSQL HA platform baseline.
- CloudNative PG is the deployed database operator.
- `RAIL-HO-WP-0004-T09` is the Railiance-side task for deploying State Hub to
the cluster, and it still requires human decisions before live data
migration.
This workplan is now the Custodian-side coordination and safety plan for that
T09 effort.
## Safety Contract
State Hub is irreplaceable episodic memory. This migration may prepare, deploy,
test, and compare as much as needed, but it must not make the cluster the only
source of truth until the explicit cutover gate is satisfied.
Rules:
- A fresh WSL2 backup and restore drill is mandatory before data migration.
- The WSL2 State Hub remains available as rollback until stabilisation passes.
- Any task that changes the live writer endpoint requires explicit human
approval.
- A failed cluster deploy must leave the WSL2 instance untouched and usable.
- Row counts and key API checks must match before cutover.
## Target Architecture After This Workplan
```
COULOMBCORE / operator workstation (WSL2)
└─ Claude Code
└─ MCP server subprocess (Python, local clone of the-custodian)
└─ HTTP → ssh -L 8000:state-hub-svc:8000 tegwick@92.205.62.239
└─ Railiance01 k3s
└─ state-hub ClusterIP service
├─ FastAPI pod (1–2 replicas)
└─ PostgreSQL PVC (Longhorn, 2-way replicated)
Operator workstation / COULOMBCORE / other agent hosts
-> local MCP server subprocess
-> http://127.0.0.1:8000 or configured API_BASE
-> private tunnel / ops-bridge
-> railiance01 k3s
-> state-hub Service
-> FastAPI Deployment
-> state-hub-db CloudNative PG Cluster
```
Key properties:
- **Not publicly exposed** — ClusterIP only; access via SSH port-forward
- **Replicated storage** — Longhorn replicates the PG data volume across nodes
- **WSL2 instance retained as DR fallback** during the stabilisation period
- **MCP config unchanged** — subprocess still calls `http://127.0.0.1:8000`;
the SSH port-forward provides the binding
## Backup and disaster recovery contract
- Single-node pragmatic deployment on railiance01.
- No public unauthenticated exposure.
- Database managed by cnpg, not an ad-hoc Postgres StatefulSet.
- WSL2 retained as DR fallback during stabilisation.
- Future multi-node HA and storage replication are deferred to `CUST-WP-0038`.
Before and during migration, the following must hold at all times:
## Open Human Decisions
| Asset | Backup mechanism | RPO | Tested? |
|---|---|---|---|
| State Hub PostgreSQL DB | `make backup` (pg_dump → age-encrypted, Nextcloud offsite) | Daily | Must be drilled before T03 |
| State Hub DB on cluster | Longhorn snapshot + age-encrypted copy to `/opt/backup/` | Daily | Must be drilled before T06 |
| WSL2 instance | Remains live during stabilisation period | — | Running |
Resolve these before T04/T05 can become live migration work:
**Rollback rule:** at any task boundary, if something is wrong, revert to
WSL2. No task should leave the system in a state where both WSL2 and cluster
are broken.
---
1. Final State Hub hostname or tunnel-only endpoint.
2. Container registry choice: Gitea registry vs external interim registry.
3. Exposure model: ClusterIP plus tunnel, private ingress, or both.
4. Approval window for freezing WSL2 writes and migrating the production DB.
5. Stabilisation duration before WSL2 can be considered non-primary fallback.
## Tasks
### T01 — Drill WSL2 backup restore end-to-end
### T01 — Drill WSL2 State Hub backup restore
```task
id: T01
@@ -83,29 +110,23 @@ priority: high
state_hub_task_id: "b0caf112-dc1d-43a8-9f27-d627dd4aa2bf"
```
Before touching anything, prove the current backup can actually be restored:
Take a fresh State Hub backup from the current WSL2 instance and restore it
into an isolated test PostgreSQL instance.
```bash
# In the-custodian/state-hub/
make backup # take fresh backup
# Spin up a test postgres container
docker run -d --name pg-restore-test -e POSTGRES_PASSWORD=test \
-p 5433:5432 postgres:16
# Decrypt and restore
age -d -i ~/.config/sops/age/keys.txt \
/opt/backup/custodian/state-hub-latest.sql.gz.age | \
gunzip | psql -h 127.0.0.1 -p 5433 -U postgres state_hub
# Spot-check: count topics
psql -h 127.0.0.1 -p 5433 -U postgres -c "SELECT COUNT(*) FROM topics;" state_hub
docker rm -f pg-restore-test
```
Minimum checks:
**Done when:** restore completes, topic count matches production, drill logged
in `memory/episodic/`.
- Restore completes without errors.
- Core table row counts match the live WSL2 database.
- `/state/summary` can be served from the restored copy if wired to a test API.
- Drill result is recorded in State Hub progress and, if useful, episodic
memory.
**Done when:** backup and restore are proven within 24 hours of live migration
work.
---
### T02 — Helm chart for State Hub (new: railiance-platform)
### T02 — Align with Railiance deployment plan
```task
id: T02
@@ -114,34 +135,22 @@ priority: high
state_hub_task_id: "24887dd9-7d50-4cc4-add7-bffa1454b80c"
```
Create `helm/state-hub/` in `railiance-platform` (S3 layer owns platform
services). The chart must deploy:
Update the cross-repo plan so this Custodian workplan and
`RAIL-HO-WP-0004-T09` point to the same architecture.
- **FastAPI deployment** — image built from `the-custodian/state-hub/`,
1 replica initially (scale to 2 after T06)
- **PostgreSQL StatefulSet** — single instance backed by a Longhorn PVC
(minimum 5 Gi); HA not required here — Longhorn replication IS the HA
- **ClusterIP service** `state-hub` on port 8000
- **ConfigMap** for non-secret config (DB URL template, log level)
- **Secret** for DB credentials (SOPS-encrypted values file)
- **Liveness/readiness probe** — `GET /state/health`
Expected outputs:
Values:
```yaml
image:
repository: gitea.local/custodian/state-hub
tag: latest
postgres:
storageClass: longhorn
size: 5Gi
replicaCount: 1
```
- `RAIL-HO-WP-0004-T09` remains the Railiance-side execution task.
- This workplan remains the Custodian-side safety/cutover task list.
- Any stale Longhorn/Postgres StatefulSet assumptions are removed from the
near-term migration plan.
- The future HA goal is referenced through `CUST-WP-0038`.
**Done when:** `helm lint` passes; chart committed in railiance-platform.
**Done when:** both workplans describe compatible responsibilities and gates.
---
### T03 — Build and push State Hub container image
### T03 — Build and publish State Hub container image
```task
id: T03
@@ -150,31 +159,22 @@ priority: high
state_hub_task_id: "79908ade-3e38-451b-a403-2361a16a3f3a"
```
Add `state-hub/Dockerfile` to the-custodian:
Package `state-hub/` as a production image.
```dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN pip install uv && uv sync --frozen --no-dev
COPY api/ ./api/
COPY mcp_server/ ./mcp_server/
CMD ["uv", "run", "uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
Requirements:
Build and push to the cluster-local Gitea registry:
- Dockerfile builds from the current Python/uv project.
- Alembic and runtime dependencies are available inside the image.
- Image exposes the FastAPI service on port 8000.
- Image tag is pushed to the chosen registry.
- Build provenance is documented in the commit/workplan.
```bash
docker build -t gitea.local/custodian/state-hub:latest .
docker push gitea.local/custodian/state-hub:latest
```
**Done when:** image available in Gitea registry; `helm install --dry-run`
resolves the image.
**Done when:** railiance01 can pull the image and a dry-run deployment resolves
it.
---
### T04 — Deploy to cluster and run Alembic migrations
### T04 — Define State Hub database and app manifests
```task
id: T04
@@ -183,26 +183,20 @@ priority: high
state_hub_task_id: "a7baf2eb-abd7-4aa3-b2cb-a5370ac09844"
```
```bash
# From operator workstation via SSH port-forward to k3s API
helm install state-hub ./helm/state-hub/ \
-n custodian --create-namespace \
-f helm/state-hub/values-production.yaml
Create the cluster-side deployment assets using current Railiance boundaries:
# Wait for pods
kubectl -n custodian rollout status deployment/state-hub
- `railiance-platform`: `state-hub-db` cnpg cluster and database credentials.
- `railiance-apps`: State Hub Deployment, Service, ConfigMap, Secret/External
Secret reference, and optional private Ingress.
- Health probes use `GET /state/health`.
- Environment includes `DATABASE_URL` and any required API settings.
# Run migrations inside the pod
kubectl -n custodian exec -it deploy/state-hub -- \
uv run alembic upgrade head
```
**Done when:** pod Running, `/state/health` returns 200, Alembic reports
"head" from inside the pod.
**Done when:** manifests lint/apply in a non-destructive dry run and ownership
boundaries are documented.
---
### T05 — Migrate data from WSL2 to cluster
### T05 — Deploy empty State Hub and run migrations on railiance01
```task
id: T05
@@ -211,33 +205,21 @@ priority: high
state_hub_task_id: "a307dd46-a8e2-49df-b016-c187759ebcf1"
```
This is the point of no return for the DB — execute with care:
Deploy State Hub against an empty `state-hub-db` cnpg database and run Alembic
migrations in the cluster environment.
```bash
# 1. Take final WSL2 backup
make -C ~/the-custodian/state-hub backup
Checks:
# 2. Copy dump into the cluster postgres pod
kubectl -n custodian cp /tmp/state-hub-migration.sql \
$(kubectl -n custodian get pod -l app=state-hub-postgres -o name):/tmp/
- Pod reaches Ready.
- `/state/health` returns healthy through the intended private access path.
- Alembic reports head.
- Logs show no repeated crash/restart loop.
# 3. Restore
kubectl -n custodian exec -it deploy/state-hub-postgres -- \
psql -U postgres -d state_hub -f /tmp/state-hub-migration.sql
# 4. Spot-check counts match WSL2
kubectl -n custodian exec -it deploy/state-hub -- \
psql -c "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;"
```
**Rollback:** if counts differ, delete cluster DB data, re-run from T04.
WSL2 is still live and unchanged.
**Done when:** all table row counts match the WSL2 instance.
**Done when:** an empty but structurally valid State Hub runs on railiance01.
---
### T06 — Drill cluster backup restore
### T06 — Restore WSL2 data copy into cluster and compare
```task
id: T06
@@ -246,53 +228,49 @@ priority: high
state_hub_task_id: "03753b88-824c-4448-97b2-f7315d145060"
```
Before cutting over, prove the cluster backup can be restored:
Restore a fresh WSL2 dump into the cluster database while WSL2 remains the live
source of truth.
```bash
# Trigger a backup via the cluster cron (or manually)
kubectl -n custodian create job --from=cronjob/state-hub-backup backup-drill-01
Required comparison:
# Verify output in /opt/backup/ on the node holding the PVC
# Decrypt and restore to a test namespace
kubectl create ns restore-test
# ... restore steps similar to T01 but against cluster postgres
```
- Table row counts match.
- Representative workstreams, tasks, decisions, progress events, repos, and
token events are queryable.
- Dashboard and MCP summary calls return expected data through the cluster API.
- Any mismatch is investigated before proceeding.
**Done when:** restore drill passes; drill logged.
**Done when:** cluster data is a verified copy of WSL2, but not yet the only
writer.
---
### T07 — Cutover: redirect MCP config to cluster
### T07 — Cut over private access to cluster State Hub
```task
id: T07
status: todo
priority: medium
state_hub_task_id: "ff1de25e-c301-4b86-9420-84dfe72e565e"
needs_human: true
intervention_note: "Requires explicit approval to freeze WSL2 writes and make the cluster State Hub the primary endpoint."
```
Update the MCP config on every operator workstation (WSL2, COULOMBCORE) to
reach the cluster state hub via SSH port-forward instead of the local process.
With human approval, freeze WSL2 writes, take a final dump, restore it to the
cluster, compare counts again, and redirect the active private access path to
the cluster API.
The MCP server subprocess still runs locally (Python, same `server.py`).
Only the API endpoint it calls changes — via a persistent port-forward:
Accepted approaches:
```bash
# On operator workstation — keep this running (add to tunnel-daemon or tunnel-loop)
ssh -L 8000:state-hub.custodian.svc.cluster.local:8000 tegwick@92.205.62.239
```
- Keep local MCP config pointed at `http://127.0.0.1:8000` and move that port
to an ops-bridge/SSH tunnel.
- Or set the MCP server `API_BASE` to the chosen private cluster endpoint.
No change to `.mcp.json` needed — subprocess still calls `http://127.0.0.1:8000`.
Alternatively: update the MCP server's `API_BASE` env var to point directly
to the port-forward. Either approach is valid; document the chosen one.
**Done when:** `claude /mcp` shows `state-hub` connected; `get_state_summary()`
returns live cluster data.
**Done when:** `get_state_summary()` and dashboard live data are served by the
cluster State Hub, and WSL2 is no longer receiving normal writes.
---
### T08 — Stabilisation period (2 weeks minimum)
### T08 — Stabilise with WSL2 retained as fallback
```task
id: T08
@@ -301,19 +279,23 @@ priority: medium
state_hub_task_id: "e06a59a0-5310-4c1c-9ba5-7cfaadda62e2"
```
Run the cluster state hub as the primary for two weeks before retiring WSL2:
Run the cluster State Hub as primary while keeping the WSL2 instance available
as a fallback.
- Keep WSL2 state hub running (but frozen — no writes) as DR fallback
- Monitor cluster pod restarts, storage health, backup cron
- Run `get_state_summary()` at the start of each session; confirm data is live
- Test failover: kill the FastAPI pod; verify it restarts and responds within 60s
Monitor:
**Done when:** two weeks elapsed with no data loss events; all backup drills
passed.
- State Hub pod restarts.
- cnpg cluster health.
- Backup job success.
- Dashboard and MCP behavior from each operator machine.
- Consistency sync behavior for file-backed workplans.
**Done when:** the agreed stabilisation window passes without data loss or
unresolved operational defects.
---
### T09 — Retire WSL2 instance
### T09 — Document operating model and defer final WSL2 retirement
```task
id: T09
@@ -322,25 +304,24 @@ priority: low
state_hub_task_id: "d75a2d49-f3b1-4bdd-b9e1-a1c6a9744681"
```
Once T08 stabilisation passes:
Document the new operating model:
1. Take a final WSL2 backup (archive, keep indefinitely)
2. Stop the WSL2 Docker container: `make -C ~/the-custodian/state-hub clean`
3. Update `CLAUDE.md` global and project to remove WSL2 state hub start instructions
4. Update MEMORY.md — state hub is now cluster-hosted
5. Record a decision in the state hub: "State Hub WSL2 instance retired"
- How agents reach State Hub.
- How backups and restores work.
- How to roll back to WSL2 if needed.
- Which parts remain pragmatic/single-node.
- Which long-term requirements moved to `CUST-WP-0038`.
**Done when:** WSL2 state hub no longer running; documentation updated.
Do not permanently retire WSL2 in this workplan unless a separate human
decision is recorded. Retirement belongs after proven stability or in the
future HA workplan.
---
**Done when:** runbooks and project instructions match the deployed reality.
## References
- Constitution constraint: irreversible actions require human approval — T05
(data migration) and T09 (WSL2 retirement) require explicit sign-off
- OAS layer: S3 Platform Services (railiance-platform)
- DR dependency: Longhorn storage (railiance-cluster WP to be linked)
- Extension point: EP-RAIL-005 (full-stack backup) — state hub must implement
`make backup` / `make restore` standard interface before T06
- Domain goal: `6f96c712-60e6-4ea9-ab06-168878eafbce` (Three-Phoenix Secure
Kubernetes Infrastructure)
- `railiance-infra/workplans/RAIL-HO-WP-0004-production-readiness.md`
- `RAIL-HO-WP-0004-T09` — Railiance-side State Hub deployment task
- `CUST-WP-0038` — future full ThreePhoenix HA State Hub migration
- Constitution constraint: production data migration and fallback retirement
require explicit human approval