state-hub/docs/cluster-operating-model.md

# State Hub Cluster Operating Model

This document describes how State Hub runs after the pragmatic cluster migration
(`CUST-WP-0011`). It is the operator runbook for day-to-day use, rollback, and
known pragmatic limitations.

## Runtime Summary

| Component | Location | Notes |
|-----------|----------|-------|
| API workload | `coulombcore-k3s`, namespace `state-hub` | Single-replica Deployment |
| Database | CNPG cluster `state-hub-db`, namespace `databases` | One instance, healthy |
| Image registry | `gitea.coulomb.social/coulomb/state-hub` | Tag pinned in Helm values |
| Primary access | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` forward tunnel |
| WSL2 fallback | `make api` + local Docker Postgres | Retained; not the normal writer |

State Hub is **not** publicly exposed. Access stays on the private tunnel /
ops-bridge path.

Deployment handoff assets live under `deploy/railiance/` and were promoted to
the coulombcore cluster during cutover (2026-07-03).

## How Agents Reach State Hub

### Primary operator workstation (WSL2)

The cluster API is the production writer. Port `8000` on the workstation is
forwarded to the cluster service through ops-bridge:

```bash
bridge status    # state-hub-primary should be connected
curl -fsS http://127.0.0.1:8000/state/health
```

Local MCP registration (default):

```bash
make register-mcp
make mcp-http    # SSE on :8001
```

### Remote machines (Railiance01, CoulombCore, Haskelseed, …)

Bring up the managed tunnel mesh, then register MCP against the remote API port:

```bash
make bridges
make register-mcp MCP_URL=http://127.0.0.1:18001/sse API_BASE=http://127.0.0.1:18000
```

Restart the agent runtime after MCP registration.

Onboarding details: [`docs/onboarding.md`](onboarding.md).

### Claude Code / Codex session start

```bash
cat .custodian-brief.md
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
```

When MCP tools are available, prefer `get_domain_summary("infotech")` or
equivalent State Hub MCP helpers.

## Backups and Restores

### Cluster database (CNPG)

The `state-hub-db` cluster is managed by CloudNativePG on coulombcore-k3s.
Scheduled CNPG backups are **not yet configured** — treat manual dumps as the
current backup path until `CUST-WP-0038` or a disaster-control workplan adds
automated retention.

Manual cluster dump (operator):

```bash
# Port-forward the rw service, then pg_dump from an operator shell
KUBECONFIG=~/.kube/config kubectl port-forward -n databases svc/state-hub-db-rw 15432:5432
pg_dump -h 127.0.0.1 -p 15432 -U state_hub -Fc state_hub > state-hub-$(date +%Y%m%d).dump
```

Restore into an isolated test database before any production restore attempt.
The T01 drill (2026-05-02) proved the WSL2 dump/restore path; repeat that
discipline before any live restore.

### WSL2 fallback database

The legacy Docker Postgres (`infra-postgres-1`) remains available for rollback.
It is **not** receiving normal writes after cutover.

To take a WSL2 snapshot while fallback is stopped:

```bash
docker exec infra-postgres-1 pg_dump -U custodian -Fc custodian > wsl2-state-hub.dump
```

## Roll Back to WSL2

Use this when the cluster deployment is unhealthy and operators need the last
known-good local writer.

1. Stop forwarding the primary tunnel:

   ```bash
   bridge down state-hub-primary
   ```

2. Start the local stack:

   ```bash
   cd ~/state-hub
   make api
   ```

3. Verify local health:

   ```bash
   curl -fsS http://127.0.0.1:8000/state/health
   ```

4. Re-register MCP if needed (`make register-mcp` without tunnel overrides).

5. Record a progress event documenting the rollback and the triggering incident.

Returning to cluster-primary:

```bash
bridge up state-hub-primary
# stop local uvicorn if it would conflict on :8000
fuser -k 8000/tcp 2>/dev/null || true
curl -fsS http://127.0.0.1:8000/state/health
```

Cutover sequence reference: `CUST-WP-0011-T07` (2026-07-03).

## Consistency Sync

File-backed workplans remain authoritative (ADR-001). After commits:

```bash
make fix-consistency REPO=<slug>
# or from repo root:
make fix-consistency-here
```

The 15-minute all-repo sweep is owned by activity-core on Railiance01. It
reaches the API through the `actcore-state-hub-bridge` proxy chain. Manual
invocation from the workstation still works:

```bash
curl -s -X POST http://127.0.0.1:8000/consistency/sweep/remote-all \
  -H "Content-Type: application/json" \
  -d '{"max_seconds": 300}' | python3 -m json.tool
```

Runbook: [`docs/consistency-sweep-runbook.md`](consistency-sweep-runbook.md).

**Known gap:** scheduled activity-core sweeps paused after the 2026-07-03
cutover while the bridge target chain was rewired. Manual sweeps succeed.
Re-enablement is tracked outside this workplan (service-inventory gap).

## Pragmatic Limitations (Single-Node)

This deployment is intentionally **not** highly available:

- One API replica on one k3s node.
- One CNPG instance (no synchronous replica).
- No public ingress; tunnel dependency for all remote access.
- Cluster and tunnel outages require the WSL2 fallback or the offline write
  buffer (`docs/offline-write-buffer.md`).

Long-term HA, replicated storage, tested failover, and WSL2 retirement belong
to **`CUST-WP-0038`**.

## WSL2 Retirement

Do **not** retire the WSL2 State Hub instance in normal operations. It remains
the disaster-recovery fallback until `CUST-WP-0038` (or a separate human
decision) explicitly approves retirement.

## Operator Checklist

Daily or after infra changes:

```bash
bridge check
curl -fsS http://127.0.0.1:8000/state/health
KUBECONFIG=~/.kube/config kubectl get pods -n state-hub
KUBECONFIG=~/.kube/config kubectl get cluster -n databases state-hub-db
```

After image or chart changes, see `deploy/railiance/README.md` and
`docs/container-image.md`.

## References

- `workplans/CUST-WP-0011-state-hub-threephoenix-migration.md` — migration plan
- `workplans/CUST-WP-0038-state-hub-threephoenix-ha.md` — future HA target
- `deploy/railiance/README.md` — Helm/CNPG handoff
- `the-custodian/ops/service-inventory.yml` — live endpoint inventory