coulomb/the-custodian
Archived
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated workplans for migrating the custodian to Railiance01

Browse Source
This commit is contained in:
Bernd Worsch
2026-05-02 23:38:56 +02:00
parent 94916cbfb0
commit dd887c8c81
2 changed files with 418 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files

363
workplans/CUST-WP-0011-state-hub-threephoenix-migration.md
View File

@@ -1,80 +1,107 @@
--- ---
id: CUST-WP-0011 id: CUST-WP-0011
type: workplan type: workplan
title: "Migrate Custodian State Hub to ThreePhoenix Cluster" title: "Pragmatic State Hub Migration to railiance01"
domain: custodian domain: custodian
repo: the-custodian repo: the-custodian
status: active status: active
owner: custodian owner: custodian
topic_slug: custodian topic_slug: custodian
created: "2026-03-11" created: "2026-03-11"
updated: "2026-03-11" updated: "2026-05-02"
state_hub_workstream_id: "967baafb-d92d-405a-ba0b-0d00d37c4940" state_hub_workstream_id: "967baafb-d92d-405a-ba0b-0d00d37c4940"
supersedes_intent_from: "Migrate Custodian State Hub to ThreePhoenix Cluster"
follow_up_workplan: CUST-WP-0038
--- ---
# Migrate Custodian State Hub to ThreePhoenix Cluster # Pragmatic State Hub Migration to railiance01
## Goal ## Goal
Move the Custodian State Hub (FastAPI + PostgreSQL) from its current home on Move the Custodian State Hub (FastAPI + PostgreSQL) from the WSL2 operator
the WSL2 operator workstation to the ThreePhoenix Kubernetes cluster workstation to the current railiance01 Kubernetes environment, using the
(Railiance01/02/03), making it available to Claude Code sessions running on Railiance production-readiness path that exists now:
any machine with cluster access — without public internet exposure.
The State Hub is **irreplaceable episodic memory**. This migration must be - CloudNative PG (`cnpg`) for the State Hub database in the `databases`
executed with zero tolerance for data loss and a tested rollback path at namespace.
every stage. - State Hub as an S5 workload in `railiance-apps`.
- Platform/database ownership in `railiance-platform`.
- Access through the existing private tunnel/ops-bridge model, not public
exposure.
- WSL2 retained as a disaster-recovery fallback until the cluster deployment
has proven stable.
## Pre-conditions (gate — do not start until all satisfied) This is a deliberate pragmatic step. It improves durability and multi-machine
access before the full ThreePhoenix target is ready. The ultimate multi-node,
replicated, long-term cluster goal is preserved in `CUST-WP-0038`.
- [ ] ThreePhoenix cluster has three healthy nodes (Railiance01 confirmed, Railiance02 + Railiance03 joined) ## Context Update
- [ ] Longhorn distributed storage installed and verified (replication factor ≥ 2)
- [ ] HA failover test passes (`tests/test_ha_failover.sh` exits 0 on the cluster)
- [ ] S2 integrated backup operational and tested on the cluster
- [ ] A full WSL2 State Hub backup has been taken and restore-drilled **within 24h of starting this workplan**
These gates are mandatory. A single-node cluster or unverified storage is not The original 2026-03-11 version of this workplan targeted a future
an acceptable migration target for the Custodian. ThreePhoenix cluster with Railiance01/02/03, Longhorn, and full HA gates before
starting. That was correct as an end-state, but it blocks useful progress now.
## Architecture after migration The current Railiance architecture has moved on:
- `railiance-infra/workplans/RAIL-HO-WP-0004-production-readiness.md`
supersedes the older Bitnami PostgreSQL HA platform baseline.
- CloudNative PG is the deployed database operator.
- `RAIL-HO-WP-0004-T09` is the Railiance-side task for deploying State Hub to
the cluster, and it still requires human decisions before live data
migration.
This workplan is now the Custodian-side coordination and safety plan for that
T09 effort.
## Safety Contract
State Hub is irreplaceable episodic memory. This migration may prepare, deploy,
test, and compare as much as needed, but it must not make the cluster the only
source of truth until the explicit cutover gate is satisfied.
Rules:
- A fresh WSL2 backup and restore drill is mandatory before data migration.
- The WSL2 State Hub remains available as rollback until stabilisation passes.
- Any task that changes the live writer endpoint requires explicit human
approval.
- A failed cluster deploy must leave the WSL2 instance untouched and usable.
- Row counts and key API checks must match before cutover.
## Target Architecture After This Workplan
``` ```
COULOMBCORE / operator workstation (WSL2) Operator workstation / COULOMBCORE / other agent hosts
└─ Claude Code -> local MCP server subprocess
└─ MCP server subprocess (Python, local clone of the-custodian) -> http://127.0.0.1:8000 or configured API_BASE
└─ HTTP → ssh -L 8000:state-hub-svc:8000 tegwick@92.205.62.239 -> private tunnel / ops-bridge
└─ Railiance01 k3s -> railiance01 k3s
└─ state-hub ClusterIP service -> state-hub Service
├─ FastAPI pod (1–2 replicas) -> FastAPI Deployment
└─ PostgreSQL PVC (Longhorn, 2-way replicated) -> state-hub-db CloudNative PG Cluster
``` ```
Key properties: Key properties:
- **Not publicly exposed** — ClusterIP only; access via SSH port-forward
- **Replicated storage** — Longhorn replicates the PG data volume across nodes
- **WSL2 instance retained as DR fallback** during the stabilisation period
- **MCP config unchanged** — subprocess still calls `http://127.0.0.1:8000`;
the SSH port-forward provides the binding
## Backup and disaster recovery contract - Single-node pragmatic deployment on railiance01.
- No public unauthenticated exposure.
- Database managed by cnpg, not an ad-hoc Postgres StatefulSet.
- WSL2 retained as DR fallback during stabilisation.
- Future multi-node HA and storage replication are deferred to `CUST-WP-0038`.
Before and during migration, the following must hold at all times: ## Open Human Decisions
| Asset | Backup mechanism | RPO | Tested? | Resolve these before T04/T05 can become live migration work:
|---|---|---|---|
| State Hub PostgreSQL DB | `make backup` (pg_dump → age-encrypted, Nextcloud offsite) | Daily | Must be drilled before T03 |
| State Hub DB on cluster | Longhorn snapshot + age-encrypted copy to `/opt/backup/` | Daily | Must be drilled before T06 |
| WSL2 instance | Remains live during stabilisation period | — | Running |
**Rollback rule:** at any task boundary, if something is wrong, revert to 1. Final State Hub hostname or tunnel-only endpoint.
WSL2. No task should leave the system in a state where both WSL2 and cluster 2. Container registry choice: Gitea registry vs external interim registry.
are broken. 3. Exposure model: ClusterIP plus tunnel, private ingress, or both.
4. Approval window for freezing WSL2 writes and migrating the production DB.
--- 5. Stabilisation duration before WSL2 can be considered non-primary fallback.
## Tasks ## Tasks
### T01 — Drill WSL2 backup restore end-to-end ### T01 — Drill WSL2 State Hub backup restore
```task ```task
id: T01 id: T01
@@ -83,29 +110,23 @@ priority: high
state_hub_task_id: "b0caf112-dc1d-43a8-9f27-d627dd4aa2bf" state_hub_task_id: "b0caf112-dc1d-43a8-9f27-d627dd4aa2bf"
``` ```
Before touching anything, prove the current backup can actually be restored: Take a fresh State Hub backup from the current WSL2 instance and restore it
into an isolated test PostgreSQL instance.
```bash Minimum checks:
# In the-custodian/state-hub/
make backup # take fresh backup
# Spin up a test postgres container
docker run -d --name pg-restore-test -e POSTGRES_PASSWORD=test \
-p 5433:5432 postgres:16
# Decrypt and restore
age -d -i ~/.config/sops/age/keys.txt \
/opt/backup/custodian/state-hub-latest.sql.gz.age | \
gunzip | psql -h 127.0.0.1 -p 5433 -U postgres state_hub
# Spot-check: count topics
psql -h 127.0.0.1 -p 5433 -U postgres -c "SELECT COUNT(*) FROM topics;" state_hub
docker rm -f pg-restore-test
```
**Done when:** restore completes, topic count matches production, drill logged - Restore completes without errors.
in `memory/episodic/`. - Core table row counts match the live WSL2 database.
- `/state/summary` can be served from the restored copy if wired to a test API.
- Drill result is recorded in State Hub progress and, if useful, episodic
memory.
**Done when:** backup and restore are proven within 24 hours of live migration
work.
--- ---
### T02 — Helm chart for State Hub (new: railiance-platform) ### T02 — Align with Railiance deployment plan
```task ```task
id: T02 id: T02
@@ -114,34 +135,22 @@ priority: high
state_hub_task_id: "24887dd9-7d50-4cc4-add7-bffa1454b80c" state_hub_task_id: "24887dd9-7d50-4cc4-add7-bffa1454b80c"
``` ```
Create `helm/state-hub/` in `railiance-platform` (S3 layer owns platform Update the cross-repo plan so this Custodian workplan and
services). The chart must deploy: `RAIL-HO-WP-0004-T09` point to the same architecture.
- **FastAPI deployment** — image built from `the-custodian/state-hub/`, Expected outputs:
1 replica initially (scale to 2 after T06)
- **PostgreSQL StatefulSet** — single instance backed by a Longhorn PVC
(minimum 5 Gi); HA not required here — Longhorn replication IS the HA
- **ClusterIP service** `state-hub` on port 8000
- **ConfigMap** for non-secret config (DB URL template, log level)
- **Secret** for DB credentials (SOPS-encrypted values file)
- **Liveness/readiness probe** — `GET /state/health`
Values: - `RAIL-HO-WP-0004-T09` remains the Railiance-side execution task.
```yaml - This workplan remains the Custodian-side safety/cutover task list.
image: - Any stale Longhorn/Postgres StatefulSet assumptions are removed from the
repository: gitea.local/custodian/state-hub near-term migration plan.
tag: latest - The future HA goal is referenced through `CUST-WP-0038`.
postgres:
storageClass: longhorn
size: 5Gi
replicaCount: 1
```
**Done when:** `helm lint` passes; chart committed in railiance-platform. **Done when:** both workplans describe compatible responsibilities and gates.
--- ---
### T03 — Build and push State Hub container image ### T03 — Build and publish State Hub container image
```task ```task
id: T03 id: T03
@@ -150,31 +159,22 @@ priority: high
state_hub_task_id: "79908ade-3e38-451b-a403-2361a16a3f3a" state_hub_task_id: "79908ade-3e38-451b-a403-2361a16a3f3a"
``` ```
Add `state-hub/Dockerfile` to the-custodian: Package `state-hub/` as a production image.
```dockerfile Requirements:
FROM python:3.12-slim
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN pip install uv && uv sync --frozen --no-dev
COPY api/ ./api/
COPY mcp_server/ ./mcp_server/
CMD ["uv", "run", "uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
Build and push to the cluster-local Gitea registry: - Dockerfile builds from the current Python/uv project.
- Alembic and runtime dependencies are available inside the image.
- Image exposes the FastAPI service on port 8000.
- Image tag is pushed to the chosen registry.
- Build provenance is documented in the commit/workplan.
```bash **Done when:** railiance01 can pull the image and a dry-run deployment resolves
docker build -t gitea.local/custodian/state-hub:latest . it.
docker push gitea.local/custodian/state-hub:latest
```
**Done when:** image available in Gitea registry; `helm install --dry-run`
resolves the image.
--- ---
### T04 — Deploy to cluster and run Alembic migrations ### T04 — Define State Hub database and app manifests
```task ```task
id: T04 id: T04
@@ -183,26 +183,20 @@ priority: high
state_hub_task_id: "a7baf2eb-abd7-4aa3-b2cb-a5370ac09844" state_hub_task_id: "a7baf2eb-abd7-4aa3-b2cb-a5370ac09844"
``` ```
```bash Create the cluster-side deployment assets using current Railiance boundaries:
# From operator workstation via SSH port-forward to k3s API
helm install state-hub ./helm/state-hub/ \
-n custodian --create-namespace \
-f helm/state-hub/values-production.yaml
# Wait for pods - `railiance-platform`: `state-hub-db` cnpg cluster and database credentials.
kubectl -n custodian rollout status deployment/state-hub - `railiance-apps`: State Hub Deployment, Service, ConfigMap, Secret/External
Secret reference, and optional private Ingress.
- Health probes use `GET /state/health`.
- Environment includes `DATABASE_URL` and any required API settings.
# Run migrations inside the pod **Done when:** manifests lint/apply in a non-destructive dry run and ownership
kubectl -n custodian exec -it deploy/state-hub -- \ boundaries are documented.
uv run alembic upgrade head
```
**Done when:** pod Running, `/state/health` returns 200, Alembic reports
"head" from inside the pod.
--- ---
### T05 — Migrate data from WSL2 to cluster ### T05 — Deploy empty State Hub and run migrations on railiance01
```task ```task
id: T05 id: T05
@@ -211,33 +205,21 @@ priority: high
state_hub_task_id: "a307dd46-a8e2-49df-b016-c187759ebcf1" state_hub_task_id: "a307dd46-a8e2-49df-b016-c187759ebcf1"
``` ```
This is the point of no return for the DB — execute with care: Deploy State Hub against an empty `state-hub-db` cnpg database and run Alembic
migrations in the cluster environment.
```bash Checks:
# 1. Take final WSL2 backup
make -C ~/the-custodian/state-hub backup
# 2. Copy dump into the cluster postgres pod - Pod reaches Ready.
kubectl -n custodian cp /tmp/state-hub-migration.sql \ - `/state/health` returns healthy through the intended private access path.
$(kubectl -n custodian get pod -l app=state-hub-postgres -o name):/tmp/ - Alembic reports head.
- Logs show no repeated crash/restart loop.
# 3. Restore **Done when:** an empty but structurally valid State Hub runs on railiance01.
kubectl -n custodian exec -it deploy/state-hub-postgres -- \
psql -U postgres -d state_hub -f /tmp/state-hub-migration.sql
# 4. Spot-check counts match WSL2
kubectl -n custodian exec -it deploy/state-hub -- \
psql -c "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;"
```
**Rollback:** if counts differ, delete cluster DB data, re-run from T04.
WSL2 is still live and unchanged.
**Done when:** all table row counts match the WSL2 instance.
--- ---
### T06 — Drill cluster backup restore ### T06 — Restore WSL2 data copy into cluster and compare
```task ```task
id: T06 id: T06
@@ -246,53 +228,49 @@ priority: high
state_hub_task_id: "03753b88-824c-4448-97b2-f7315d145060" state_hub_task_id: "03753b88-824c-4448-97b2-f7315d145060"
``` ```
Before cutting over, prove the cluster backup can be restored: Restore a fresh WSL2 dump into the cluster database while WSL2 remains the live
source of truth.
```bash Required comparison:
# Trigger a backup via the cluster cron (or manually)
kubectl -n custodian create job --from=cronjob/state-hub-backup backup-drill-01
# Verify output in /opt/backup/ on the node holding the PVC - Table row counts match.
# Decrypt and restore to a test namespace - Representative workstreams, tasks, decisions, progress events, repos, and
kubectl create ns restore-test token events are queryable.
# ... restore steps similar to T01 but against cluster postgres - Dashboard and MCP summary calls return expected data through the cluster API.
``` - Any mismatch is investigated before proceeding.
**Done when:** restore drill passes; drill logged. **Done when:** cluster data is a verified copy of WSL2, but not yet the only
writer.
--- ---
### T07 — Cutover: redirect MCP config to cluster ### T07 — Cut over private access to cluster State Hub
```task ```task
id: T07 id: T07
status: todo status: todo
priority: medium priority: medium
state_hub_task_id: "ff1de25e-c301-4b86-9420-84dfe72e565e" state_hub_task_id: "ff1de25e-c301-4b86-9420-84dfe72e565e"
needs_human: true
intervention_note: "Requires explicit approval to freeze WSL2 writes and make the cluster State Hub the primary endpoint."
``` ```
Update the MCP config on every operator workstation (WSL2, COULOMBCORE) to With human approval, freeze WSL2 writes, take a final dump, restore it to the
reach the cluster state hub via SSH port-forward instead of the local process. cluster, compare counts again, and redirect the active private access path to
the cluster API.
The MCP server subprocess still runs locally (Python, same `server.py`). Accepted approaches:
Only the API endpoint it calls changes — via a persistent port-forward:
```bash - Keep local MCP config pointed at `http://127.0.0.1:8000` and move that port
# On operator workstation — keep this running (add to tunnel-daemon or tunnel-loop) to an ops-bridge/SSH tunnel.
ssh -L 8000:state-hub.custodian.svc.cluster.local:8000 tegwick@92.205.62.239 - Or set the MCP server `API_BASE` to the chosen private cluster endpoint.
```
No change to `.mcp.json` needed — subprocess still calls `http://127.0.0.1:8000`. **Done when:** `get_state_summary()` and dashboard live data are served by the
cluster State Hub, and WSL2 is no longer receiving normal writes.
Alternatively: update the MCP server's `API_BASE` env var to point directly
to the port-forward. Either approach is valid; document the chosen one.
**Done when:** `claude /mcp` shows `state-hub` connected; `get_state_summary()`
returns live cluster data.
--- ---
### T08 — Stabilisation period (2 weeks minimum) ### T08 — Stabilise with WSL2 retained as fallback
```task ```task
id: T08 id: T08
@@ -301,19 +279,23 @@ priority: medium
state_hub_task_id: "e06a59a0-5310-4c1c-9ba5-7cfaadda62e2" state_hub_task_id: "e06a59a0-5310-4c1c-9ba5-7cfaadda62e2"
``` ```
Run the cluster state hub as the primary for two weeks before retiring WSL2: Run the cluster State Hub as primary while keeping the WSL2 instance available
as a fallback.
- Keep WSL2 state hub running (but frozen — no writes) as DR fallback Monitor:
- Monitor cluster pod restarts, storage health, backup cron
- Run `get_state_summary()` at the start of each session; confirm data is live
- Test failover: kill the FastAPI pod; verify it restarts and responds within 60s
**Done when:** two weeks elapsed with no data loss events; all backup drills - State Hub pod restarts.
passed. - cnpg cluster health.
- Backup job success.
- Dashboard and MCP behavior from each operator machine.
- Consistency sync behavior for file-backed workplans.
**Done when:** the agreed stabilisation window passes without data loss or
unresolved operational defects.
--- ---
### T09 — Retire WSL2 instance ### T09 — Document operating model and defer final WSL2 retirement
```task ```task
id: T09 id: T09
@@ -322,25 +304,24 @@ priority: low
state_hub_task_id: "d75a2d49-f3b1-4bdd-b9e1-a1c6a9744681" state_hub_task_id: "d75a2d49-f3b1-4bdd-b9e1-a1c6a9744681"
``` ```
Once T08 stabilisation passes: Document the new operating model:
1. Take a final WSL2 backup (archive, keep indefinitely) - How agents reach State Hub.
2. Stop the WSL2 Docker container: `make -C ~/the-custodian/state-hub clean` - How backups and restores work.
3. Update `CLAUDE.md` global and project to remove WSL2 state hub start instructions - How to roll back to WSL2 if needed.
4. Update MEMORY.md — state hub is now cluster-hosted - Which parts remain pragmatic/single-node.
5. Record a decision in the state hub: "State Hub WSL2 instance retired" - Which long-term requirements moved to `CUST-WP-0038`.
**Done when:** WSL2 state hub no longer running; documentation updated. Do not permanently retire WSL2 in this workplan unless a separate human
decision is recorded. Retirement belongs after proven stability or in the
future HA workplan.
--- **Done when:** runbooks and project instructions match the deployed reality.
## References ## References
- Constitution constraint: irreversible actions require human approval — T05 - `railiance-infra/workplans/RAIL-HO-WP-0004-production-readiness.md`
(data migration) and T09 (WSL2 retirement) require explicit sign-off - `RAIL-HO-WP-0004-T09` — Railiance-side State Hub deployment task
- OAS layer: S3 Platform Services (railiance-platform) - `CUST-WP-0038` — future full ThreePhoenix HA State Hub migration
- DR dependency: Longhorn storage (railiance-cluster WP to be linked) - Constitution constraint: production data migration and fallback retirement
- Extension point: EP-RAIL-005 (full-stack backup) — state hub must implement require explicit human approval
`make backup` / `make restore` standard interface before T06
- Domain goal: `6f96c712-60e6-4ea9-ab06-168878eafbce` (Three-Phoenix Secure
Kubernetes Infrastructure)

246
workplans/CUST-WP-0038-state-hub-threephoenix-ha.md Normal file
View File

@@ -0,0 +1,246 @@
---
id: CUST-WP-0038
type: workplan
title: "State Hub Full ThreePhoenix HA Migration"
domain: custodian
repo: the-custodian
status: proposed
owner: custodian
topic_slug: custodian
created: "2026-05-02"
updated: "2026-05-02"
depends_on: CUST-WP-0011
state_hub_workstream_id: "8d0c1b5d-44da-4b91-8357-e6526d3e0a85"
---
# State Hub Full ThreePhoenix HA Migration
## Goal
Preserve the original long-term State Hub infrastructure goal while
`CUST-WP-0011` takes the pragmatic railiance01 path.
This workplan completes the migration from a useful single-node cluster-hosted
State Hub to a full ThreePhoenix-grade service: multi-node Kubernetes,
replicated storage, tested failover, tested restore, and retirement of the WSL2
fallback only after operational confidence is earned.
## Why This Exists
The near-term State Hub migration should not wait for every HA precondition,
because the workstation-hosted State Hub is already a bottleneck for
multi-machine work.
But the original requirement remains valid:
- State Hub is irreplaceable episodic memory.
- A single node is not a final home.
- Backup and restore must be drilled, not assumed.
- Long-term operations must survive node loss and operator-machine loss.
`CUST-WP-0011` moves State Hub to railiance01 pragmatically. This workplan
keeps the ultimate target visible and reviewable.
## Entry Criteria
- `CUST-WP-0011` completed or explicitly superseded.
- Cluster-hosted State Hub has passed its stabilisation period.
- railiance01 is not the only planned durable node.
- Railiance architecture decision for storage replication is current:
Longhorn, cnpg replication, external backup, or a documented replacement.
- Backup and restore tooling has an owner and runbook.
## Target Properties
- Three healthy Kubernetes nodes: Railiance01, Railiance02, Railiance03.
- State Hub database survives loss of one node.
- State Hub API recovers from pod loss without manual repair.
- Backups are encrypted, off-node, and restorable into a test namespace.
- Agent access remains private.
- WSL2 is no longer needed as the primary disaster-recovery fallback.
## Tasks
### T01 — Confirm ThreePhoenix cluster readiness
```task
id: T01
status: todo
priority: high
state_hub_task_id: "aa1bf291-3b6c-4940-a4f5-7680b0349110"
```
Verify the target cluster state:
- Three nodes are joined and Ready.
- Control-plane and worker roles are documented.
- Cluster version and node resources are recorded.
- Smoke tests pass from the operator machine and from CoulombCore.
**Done when:** a current readiness report exists and no node is marked
NotReady or operationally unknown.
---
### T02 — Establish replicated storage/database strategy
```task
id: T02
status: todo
priority: high
state_hub_task_id: "5575f244-5cef-47aa-a168-24027cd08140"
```
Choose and document the durable data strategy for State Hub:
- cnpg multi-instance PostgreSQL cluster, and/or
- Longhorn-backed storage with suitable replication, and/or
- another explicitly approved architecture.
The decision must define RPO, RTO, failover behavior, and restore procedure.
**Done when:** the selected architecture is documented and approved before any
production data movement.
---
### T03 — Implement HA State Hub database
```task
id: T03
status: todo
priority: high
state_hub_task_id: "5330fcc3-684b-49f6-8d28-ea8c929733d6"
```
Apply the chosen database/storage architecture to State Hub.
Requirements:
- Database credentials remain SOPS/secret-managed.
- The database has automated backup configured.
- The database exposes a stable service endpoint for the API.
- Health and replication status are observable.
**Done when:** State Hub can run against the HA database in a test or staging
namespace.
---
### T04 — Add State Hub API high-availability behavior
```task
id: T04
status: todo
priority: medium
state_hub_task_id: "64175ed0-af36-47ea-9401-74c4b15ffe24"
```
Run State Hub API with the right availability posture for its workload:
- At least one replica, optionally more if DB/session behavior permits.
- Readiness and liveness probes.
- Rolling update behavior documented.
- Resource requests/limits set.
**Done when:** killing an API pod does not require manual recovery.
---
### T05 — Drill database failover
```task
id: T05
status: todo
priority: high
state_hub_task_id: "73c5008a-380e-42bf-ad57-1c9d0bda3a86"
```
Perform a controlled failover drill for the State Hub database.
Checks:
- Failure trigger is documented.
- API behavior during failover is observed.
- Recovery time is measured.
- No data loss is detected after recovery.
**Done when:** the failover drill passes and results are logged.
---
### T06 — Drill backup restore to isolated namespace
```task
id: T06
status: todo
priority: high
state_hub_task_id: "4e5b97ff-ef1c-414d-812b-39b87b242c74"
```
Restore the latest encrypted State Hub backup into an isolated namespace or
separate test database.
Checks:
- Backup can be decrypted with the documented key path.
- Restore completes from off-node backup material.
- Row counts and representative records match.
- Restored API can serve `/state/health` and `/state/summary` when pointed at
the restored database.
**Done when:** restore drill passes without depending on the live database.
---
### T07 — Update agent access and runbooks for HA endpoint
```task
id: T07
status: todo
priority: medium
state_hub_task_id: "959062d8-decb-4969-a60b-0d3b618a8d6c"
```
Update the private access model after the HA endpoint is available:
- ops-bridge or tunnel target.
- MCP `API_BASE` or local port-forward convention.
- Dashboard access.
- Operator recovery instructions.
**Done when:** each active operator machine can reach the HA State Hub endpoint
through the documented path.
---
### T08 — Retire WSL2 fallback after explicit approval
```task
id: T08
status: todo
priority: low
needs_human: true
intervention_note: "Requires explicit approval after HA failover and restore drills pass."
state_hub_task_id: "d4a7ca26-c338-48a1-b8b1-85a356550add"
```
Retire the WSL2 State Hub as a disaster-recovery fallback only after the HA
cluster path has passed drills.
Steps:
1. Take and archive a final WSL2 backup.
2. Stop local WSL2 State Hub services.
3. Update global and repo instructions.
4. Record the retirement decision in State Hub.
**Done when:** WSL2 is no longer part of the normal or fallback operating
model, and the cluster runbook is the source of truth.
## References
- `CUST-WP-0011` — pragmatic railiance01 migration
- Railiance ThreePhoenix infrastructure goal
- State Hub backup/restore runbooks
- Constitution constraint: irreversible retirement requires human approval