state-hub/docs/cluster-operating-model.md

State Hub Cluster Operating Model

This document describes how State Hub runs after the pragmatic cluster migration (CUST-WP-0011). It is the operator runbook for day-to-day use, rollback, and known pragmatic limitations.

Runtime Summary

Component	Location	Notes
API workload	coulombcore-k3s, namespace state-hub	Single-replica Deployment
Database	CNPG cluster state-hub-db, namespace databases	One instance, healthy
Image registry	gitea.coulomb.social/coulomb/state-hub	Tag pinned in Helm values
Primary access	http://127.0.0.1:8000	ops-bridge state-hub-primary forward tunnel
WSL2 fallback	make api + local Docker Postgres	Retained; not the normal writer

State Hub is not publicly exposed. Access stays on the private tunnel / ops-bridge path.

Deployment handoff assets live under deploy/railiance/ and were promoted to the coulombcore cluster during cutover (2026-07-03).

How Agents Reach State Hub

Primary operator workstation (WSL2)

The cluster API is the production writer. Port 8000 on the workstation is forwarded to the cluster service through ops-bridge:

bridge status    # state-hub-primary should be connected
curl -fsS http://127.0.0.1:8000/state/health

Local MCP registration (default):

make register-mcp
make mcp-http    # SSE on :8001

Remote machines (Railiance01, CoulombCore, Haskelseed, …)

Bring up the managed tunnel mesh, then register MCP against the remote API port:

make bridges
make register-mcp MCP_URL=http://127.0.0.1:18001/sse API_BASE=http://127.0.0.1:18000

Restart the agent runtime after MCP registration.

Onboarding details: docs/onboarding.md.

Claude Code / Codex session start

cat .custodian-brief.md
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool

When MCP tools are available, prefer get_domain_summary("infotech") or equivalent State Hub MCP helpers.

Backups and Restores

Cluster database (CNPG)

The state-hub-db cluster is managed by CloudNativePG on coulombcore-k3s. Scheduled CNPG backups are not yet configured — treat manual dumps as the current backup path until CUST-WP-0038 or a disaster-control workplan adds automated retention.

Manual cluster dump (operator):

# Port-forward the rw service, then pg_dump from an operator shell
KUBECONFIG=~/.kube/config kubectl port-forward -n databases svc/state-hub-db-rw 15432:5432
pg_dump -h 127.0.0.1 -p 15432 -U state_hub -Fc state_hub > state-hub-$(date +%Y%m%d).dump

Restore into an isolated test database before any production restore attempt. The T01 drill (2026-05-02) proved the WSL2 dump/restore path; repeat that discipline before any live restore.

WSL2 fallback database

The legacy Docker Postgres (infra-postgres-1) remains available for rollback. It is not receiving normal writes after cutover.

To take a WSL2 snapshot while fallback is stopped:

docker exec infra-postgres-1 pg_dump -U custodian -Fc custodian > wsl2-state-hub.dump

Roll Back to WSL2

Use this when the cluster deployment is unhealthy and operators need the last known-good local writer.

1. Stop forwarding the primary tunnel:

   bridge down state-hub-primary
   

2. Start the local stack:

   cd ~/state-hub
   make api
   

3. Verify local health:

   curl -fsS http://127.0.0.1:8000/state/health
   

4. Re-register MCP if needed (make register-mcp without tunnel overrides).

5. Record a progress event documenting the rollback and the triggering incident.

Returning to cluster-primary:

bridge up state-hub-primary
# stop local uvicorn if it would conflict on :8000
fuser -k 8000/tcp 2>/dev/null || true
curl -fsS http://127.0.0.1:8000/state/health

Cutover sequence reference: CUST-WP-0011-T07 (2026-07-03).

Consistency Sync

File-backed workplans remain authoritative (ADR-001). After commits:

make fix-consistency REPO=<slug>
# or from repo root:
make fix-consistency-here

The 15-minute all-repo sweep is owned by activity-core on Railiance01. It reaches the API through the actcore-state-hub-bridge proxy chain. Manual invocation from the workstation still works:

curl -s -X POST http://127.0.0.1:8000/consistency/sweep/remote-all \
  -H "Content-Type: application/json" \
  -d '{"max_seconds": 300}' | python3 -m json.tool

Runbook: docs/consistency-sweep-runbook.md.

Known gap: scheduled activity-core sweeps paused after the 2026-07-03 cutover while the bridge target chain was rewired. Manual sweeps succeed. Re-enablement is tracked outside this workplan (service-inventory gap).

Pragmatic Limitations (Single-Node)

This deployment is intentionally not highly available:

• One API replica on one k3s node.
• One CNPG instance (no synchronous replica).
• No public ingress; tunnel dependency for all remote access.
• Cluster and tunnel outages require the WSL2 fallback or the offline write buffer (docs/offline-write-buffer.md).

Long-term HA, replicated storage, tested failover, and WSL2 retirement belong to CUST-WP-0038.

WSL2 Retirement

Do not retire the WSL2 State Hub instance in normal operations. It remains the disaster-recovery fallback until CUST-WP-0038 (or a separate human decision) explicitly approves retirement.

Operator Checklist

Daily or after infra changes:

bridge check
curl -fsS http://127.0.0.1:8000/state/health
KUBECONFIG=~/.kube/config kubectl get pods -n state-hub
KUBECONFIG=~/.kube/config kubectl get cluster -n databases state-hub-db

After image or chart changes, see deploy/railiance/README.md and docs/container-image.md.

References

• workplans/CUST-WP-0011-state-hub-threephoenix-migration.md — migration plan
• workplans/CUST-WP-0038-state-hub-threephoenix-ha.md — future HA target
• deploy/railiance/README.md — Helm/CNPG handoff
• the-custodian/ops/service-inventory.yml — live endpoint inventory