inter-hub/deploy/railiance/RUNBOOK.md

# inter-hub on Railiance01 — Runbook

## Architecture

- **Cluster:** Railiance01 (K3s, 92.205.62.239)
- **Namespace:** `inter-hub`
- **Image registry:** `92.205.130.254:32166/coulomb/inter-hub:<sha>` (Gitea on CoulombCore)
- **Database:** CloudNativePG cluster `net-kingdom-pg` in `databases` namespace
  - RW endpoint: `net-kingdom-pg-rw.databases.svc.cluster.local:5432`
  - Database: `interhub`, User: `interhub`
- **Ingress:** Traefik → `hub.coulomb.social` (TLS via letsencrypt-prod)
- **Secrets:** `inter-hub-env` Secret in `inter-hub` namespace

## Deployment

```bash
# From workstation (image already built and pushed):
helm upgrade --install inter-hub deploy/helm/inter-hub \
  --namespace inter-hub --create-namespace \
  --set image.tag=<sha>
```

## Image Build (on haskelseed)

```bash
ssh root@192.168.178.135
cd /root/inter-hub
# Build:
nix build .#docker --log-format raw > /tmp/build.log 2>&1

# Push — Gitea registry token realm points to gitea.coulomb.social:80 but Gitea
# only listens on port 32166; skopeo must use a pre-fetched token:
SHA=$(git rev-parse --short HEAD)
SKOPEO=/nix/store/fwdagky9lfsyrgzxiq14zijcziazfdsn-skopeo-1.22.2/bin/skopeo
TOKEN=$(curl -s \
  "http://92.205.130.254:32166/v2/token?service=container_registry&scope=repository:coulomb/inter-hub:push,pull" \
  -u 'tegwick:<GITEA_API_KEY>' | awk -F'"' '/token/{print $4}')
$SKOPEO copy --insecure-policy --dest-tls-verify=false \
  --dest-registry-token "$TOKEN" \
  docker-archive:result \
  docker://92.205.130.254:32166/coulomb/inter-hub:$SHA
```

**Notes:**
- `skopeo` is in the Nix profile but not on PATH — use the full store path above.
- The IHP Nix Docker image has NO `/bin/RunProdServer` symlink. The binary lives at
  `/nix/store/<hash>-inter-hub/bin/RunProdServer` (hash changes per build).
  Use `kubectl exec deploy/inter-hub -- /nix/store/*-inter-hub/bin/RunProdServer <cmd>`
  if a shell is not available (the Nix image has no `/bin/sh`).

## Gitea Registry Credentials

The Gitea token for registry push is stored in `~/.config/tea/config.yml` on the
workstation. If the token has expired, generate a new one:
1. Go to http://92.205.130.254:32166 → Settings → Applications → Generate new token
2. Scope: `package:write`
3. Update `~/.config/tea/config.yml` on the workstation
4. Update the `GITEA_TOKEN` in any CI/CD secrets

## Database Migration

IHP migrations run automatically on startup via the init container in the Deployment.
To run migrations manually:

```bash
kubectl exec -n inter-hub deploy/inter-hub -- /bin/RunProdServer migrate
```

To check migration status:
```bash
kubectl exec -n databases net-kingdom-pg-1 -- psql -U postgres interhub -c "\dt"
```

## Logs

```bash
kubectl logs -n inter-hub -l app=inter-hub --tail=100 -f
# Previous pod logs:
kubectl logs -n inter-hub -l app=inter-hub --previous --tail=50
```

## Restart / Rollback

```bash
# Restart:
kubectl rollout restart deployment/inter-hub -n inter-hub
kubectl rollout status deployment/inter-hub -n inter-hub

# Rollback to previous image:
kubectl rollout undo deployment/inter-hub -n inter-hub

# Rollback to specific version:
helm rollback inter-hub 1 --namespace inter-hub
```

## Secret Rotation

To rotate the session secret:
```bash
kubectl create secret generic inter-hub-env \
  --namespace inter-hub \
  --from-literal=DATABASE_URL='...' \
  --from-literal=IHP_SESSION_SECRET='<new-64-char-hex>' \
  --from-literal=IHP_BASEURL='https://hub.coulomb.social' \
  --from-literal=PORT='8000' \
  --from-literal=IHP_ENV='Production' \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/inter-hub -n inter-hub
```

To rotate the database password:
1. Update the password in PostgreSQL (via kubectl exec to the CNPG pod)
2. Update the `inter-hub-env` secret
3. Restart the deployment

## Smoke Test

```bash
curl -s https://hub.coulomb.social/ | grep "Inter-Hub"           # Landing 200
curl -s https://hub.coulomb.social/capabilities | grep "Capabilities"
curl -s https://hub.coulomb.social/api/v2/hubs                   # 401 expected
curl -H "Authorization: Bearer <api-key>" https://hub.coulomb.social/api/v2/hubs  # 200
```

## Database Connection Check

The IHP Nix image has no `/bin/sh`. Connect via the CNPG pod instead:
```bash
kubectl exec -n databases net-kingdom-pg-1 -- psql -U postgres -d interhub -c "SELECT version();"
```

## Password Hashing

IHP uses `pwstore-fast` (`Crypto.PasswordStore`) — **not bcrypt**. Hash format:
```
sha256|17|<base64-salt>|<base64-hash>
```

To generate a correct hash (requires GHC with pwstore-fast available on haskelseed):
```bash
ssh root@192.168.178.135
cat > /tmp/genhash.hs << 'EOF'
import qualified Crypto.PasswordStore as PS
import qualified Data.ByteString.Char8 as B8
main :: IO ()
main = do
    h <- PS.makePassword (B8.pack "yourpassword") 17
    B8.putStrLn h
EOF
/nix/store/yp23474ys67f1fd2z2ff1nn3q5wrmjng-ghc-9.10.3-with-packages/bin/runghc /tmp/genhash.hs
```

## haskelseed Build VM

- **Host:** 192.168.178.135
- **Access:** `ssh root@192.168.178.135` (password in team secrets)
- **Repo:** `/root/inter-hub` (git initialized locally; pull requires Gitea token)
- **Build logs:** `/tmp/nix-build-docker.log`
- **Nix store:** `/dev/sdb1` (100 GB, mounted at `/nix`)