generated from coulomb/repo-seed
Finish CUST-WP-0011 and implement STATE-WP-0061 suggestion backlog.
Add persisted Suggestion entities with WSJF ranking, relevance bumps, REST/MCP write surfaces, /suggestions dashboard page, and daily triage digest integration. Document the cluster operating model, archive the completed migration workplan, and seed WARDEN-WP-0012 routing scenarios.
This commit is contained in:
@@ -67,7 +67,7 @@ unset.
|
||||
- DB schema + Alembic migrations
|
||||
- API endpoints (CRUD + status transitions + read-model queries)
|
||||
- MCP tools (read + sanctioned writes: `resolve_decision`,
|
||||
`add_progress_event`, `get_next_steps`)
|
||||
`add_progress_event`, `get_next_steps`, suggestion backlog writes)
|
||||
- The consistency engine (`scripts/consistency_check.py`) — it owns
|
||||
ADR-001 reconciliation between workplan files and the DB.
|
||||
- The `cleanup_stale_tasks.py` *script* (not its schedule) — it owns
|
||||
|
||||
200
docs/cluster-operating-model.md
Normal file
200
docs/cluster-operating-model.md
Normal file
@@ -0,0 +1,200 @@
|
||||
# State Hub Cluster Operating Model
|
||||
|
||||
This document describes how State Hub runs after the pragmatic cluster migration
|
||||
(`CUST-WP-0011`). It is the operator runbook for day-to-day use, rollback, and
|
||||
known pragmatic limitations.
|
||||
|
||||
## Runtime Summary
|
||||
|
||||
| Component | Location | Notes |
|
||||
|-----------|----------|-------|
|
||||
| API workload | `coulombcore-k3s`, namespace `state-hub` | Single-replica Deployment |
|
||||
| Database | CNPG cluster `state-hub-db`, namespace `databases` | One instance, healthy |
|
||||
| Image registry | `gitea.coulomb.social/coulomb/state-hub` | Tag pinned in Helm values |
|
||||
| Primary access | `http://127.0.0.1:8000` | ops-bridge `state-hub-primary` forward tunnel |
|
||||
| WSL2 fallback | `make api` + local Docker Postgres | Retained; not the normal writer |
|
||||
|
||||
State Hub is **not** publicly exposed. Access stays on the private tunnel /
|
||||
ops-bridge path.
|
||||
|
||||
Deployment handoff assets live under `deploy/railiance/` and were promoted to
|
||||
the coulombcore cluster during cutover (2026-07-03).
|
||||
|
||||
## How Agents Reach State Hub
|
||||
|
||||
### Primary operator workstation (WSL2)
|
||||
|
||||
The cluster API is the production writer. Port `8000` on the workstation is
|
||||
forwarded to the cluster service through ops-bridge:
|
||||
|
||||
```bash
|
||||
bridge status # state-hub-primary should be connected
|
||||
curl -fsS http://127.0.0.1:8000/state/health
|
||||
```
|
||||
|
||||
Local MCP registration (default):
|
||||
|
||||
```bash
|
||||
make register-mcp
|
||||
make mcp-http # SSE on :8001
|
||||
```
|
||||
|
||||
### Remote machines (Railiance01, CoulombCore, Haskelseed, …)
|
||||
|
||||
Bring up the managed tunnel mesh, then register MCP against the remote API port:
|
||||
|
||||
```bash
|
||||
make bridges
|
||||
make register-mcp MCP_URL=http://127.0.0.1:18001/sse API_BASE=http://127.0.0.1:18000
|
||||
```
|
||||
|
||||
Restart the agent runtime after MCP registration.
|
||||
|
||||
Onboarding details: [`docs/onboarding.md`](onboarding.md).
|
||||
|
||||
### Claude Code / Codex session start
|
||||
|
||||
```bash
|
||||
cat .custodian-brief.md
|
||||
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
|
||||
```
|
||||
|
||||
When MCP tools are available, prefer `get_domain_summary("infotech")` or
|
||||
equivalent State Hub MCP helpers.
|
||||
|
||||
## Backups and Restores
|
||||
|
||||
### Cluster database (CNPG)
|
||||
|
||||
The `state-hub-db` cluster is managed by CloudNativePG on coulombcore-k3s.
|
||||
Scheduled CNPG backups are **not yet configured** — treat manual dumps as the
|
||||
current backup path until `CUST-WP-0038` or a disaster-control workplan adds
|
||||
automated retention.
|
||||
|
||||
Manual cluster dump (operator):
|
||||
|
||||
```bash
|
||||
# Port-forward the rw service, then pg_dump from an operator shell
|
||||
KUBECONFIG=~/.kube/config kubectl port-forward -n databases svc/state-hub-db-rw 15432:5432
|
||||
pg_dump -h 127.0.0.1 -p 15432 -U state_hub -Fc state_hub > state-hub-$(date +%Y%m%d).dump
|
||||
```
|
||||
|
||||
Restore into an isolated test database before any production restore attempt.
|
||||
The T01 drill (2026-05-02) proved the WSL2 dump/restore path; repeat that
|
||||
discipline before any live restore.
|
||||
|
||||
### WSL2 fallback database
|
||||
|
||||
The legacy Docker Postgres (`infra-postgres-1`) remains available for rollback.
|
||||
It is **not** receiving normal writes after cutover.
|
||||
|
||||
To take a WSL2 snapshot while fallback is stopped:
|
||||
|
||||
```bash
|
||||
docker exec infra-postgres-1 pg_dump -U custodian -Fc custodian > wsl2-state-hub.dump
|
||||
```
|
||||
|
||||
## Roll Back to WSL2
|
||||
|
||||
Use this when the cluster deployment is unhealthy and operators need the last
|
||||
known-good local writer.
|
||||
|
||||
1. Stop forwarding the primary tunnel:
|
||||
|
||||
```bash
|
||||
bridge down state-hub-primary
|
||||
```
|
||||
|
||||
2. Start the local stack:
|
||||
|
||||
```bash
|
||||
cd ~/state-hub
|
||||
make api
|
||||
```
|
||||
|
||||
3. Verify local health:
|
||||
|
||||
```bash
|
||||
curl -fsS http://127.0.0.1:8000/state/health
|
||||
```
|
||||
|
||||
4. Re-register MCP if needed (`make register-mcp` without tunnel overrides).
|
||||
|
||||
5. Record a progress event documenting the rollback and the triggering incident.
|
||||
|
||||
Returning to cluster-primary:
|
||||
|
||||
```bash
|
||||
bridge up state-hub-primary
|
||||
# stop local uvicorn if it would conflict on :8000
|
||||
fuser -k 8000/tcp 2>/dev/null || true
|
||||
curl -fsS http://127.0.0.1:8000/state/health
|
||||
```
|
||||
|
||||
Cutover sequence reference: `CUST-WP-0011-T07` (2026-07-03).
|
||||
|
||||
## Consistency Sync
|
||||
|
||||
File-backed workplans remain authoritative (ADR-001). After commits:
|
||||
|
||||
```bash
|
||||
make fix-consistency REPO=<slug>
|
||||
# or from repo root:
|
||||
make fix-consistency-here
|
||||
```
|
||||
|
||||
The 15-minute all-repo sweep is owned by activity-core on Railiance01. It
|
||||
reaches the API through the `actcore-state-hub-bridge` proxy chain. Manual
|
||||
invocation from the workstation still works:
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/consistency/sweep/remote-all \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"max_seconds": 300}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
Runbook: [`docs/consistency-sweep-runbook.md`](consistency-sweep-runbook.md).
|
||||
|
||||
**Known gap:** scheduled activity-core sweeps paused after the 2026-07-03
|
||||
cutover while the bridge target chain was rewired. Manual sweeps succeed.
|
||||
Re-enablement is tracked outside this workplan (service-inventory gap).
|
||||
|
||||
## Pragmatic Limitations (Single-Node)
|
||||
|
||||
This deployment is intentionally **not** highly available:
|
||||
|
||||
- One API replica on one k3s node.
|
||||
- One CNPG instance (no synchronous replica).
|
||||
- No public ingress; tunnel dependency for all remote access.
|
||||
- Cluster and tunnel outages require the WSL2 fallback or the offline write
|
||||
buffer (`docs/offline-write-buffer.md`).
|
||||
|
||||
Long-term HA, replicated storage, tested failover, and WSL2 retirement belong
|
||||
to **`CUST-WP-0038`**.
|
||||
|
||||
## WSL2 Retirement
|
||||
|
||||
Do **not** retire the WSL2 State Hub instance in normal operations. It remains
|
||||
the disaster-recovery fallback until `CUST-WP-0038` (or a separate human
|
||||
decision) explicitly approves retirement.
|
||||
|
||||
## Operator Checklist
|
||||
|
||||
Daily or after infra changes:
|
||||
|
||||
```bash
|
||||
bridge check
|
||||
curl -fsS http://127.0.0.1:8000/state/health
|
||||
KUBECONFIG=~/.kube/config kubectl get pods -n state-hub
|
||||
KUBECONFIG=~/.kube/config kubectl get cluster -n databases state-hub-db
|
||||
```
|
||||
|
||||
After image or chart changes, see `deploy/railiance/README.md` and
|
||||
`docs/container-image.md`.
|
||||
|
||||
## References
|
||||
|
||||
- `workplans/CUST-WP-0011-state-hub-threephoenix-migration.md` — migration plan
|
||||
- `workplans/CUST-WP-0038-state-hub-threephoenix-ha.md` — future HA target
|
||||
- `deploy/railiance/README.md` — Helm/CNPG handoff
|
||||
- `the-custodian/ops/service-inventory.yml` — live endpoint inventory
|
||||
@@ -79,19 +79,24 @@ Restart Claude Code after MCP registration.
|
||||
- `tegwick@92.205.62.239` for Railiance01
|
||||
- `tegwick@92.205.130.254` for CoulombCore
|
||||
|
||||
5. Start or connect to State Hub:
|
||||
5. Connect to State Hub:
|
||||
|
||||
Primary operators reach the cluster deployment on port `8000` through
|
||||
ops-bridge (`state-hub-primary`). Verify before starting a local API:
|
||||
|
||||
```bash
|
||||
make api
|
||||
curl -fsS http://127.0.0.1:8000/state/health || make api
|
||||
make mcp-http
|
||||
```
|
||||
|
||||
If the hub is remote, use ops-bridge:
|
||||
Remote machines use the tunnel mesh:
|
||||
|
||||
```bash
|
||||
make bridges
|
||||
```
|
||||
|
||||
Operating model: [`docs/cluster-operating-model.md`](cluster-operating-model.md)
|
||||
|
||||
6. Restart Claude Code and verify that `state-hub` appears in the MCP server
|
||||
list. In the first session, call `get_state_summary()` when MCP tools are
|
||||
available. If not, use:
|
||||
|
||||
Reference in New Issue
Block a user