generated from coulomb/repo-seed
tegwick
9cbf4caadf
Generated.Types imports 119 modules, pushing the combined .hi read past a ~287 MB binary-deserialization limit in GHC 9.10.3. Fix by adding a nixpkgs overlay that patches the inter-hub-models derivation: replaces Generated/Types.hs with a thin TypesPart1/TypesPart2 re-export wrapper after build-generated-code runs, and adds the two split modules to the cabal exposed-modules list. Also fix the production binary name from /bin/App to /bin/RunProdServer in deployment.yaml and RUNBOOK.md (the IHP NixSupport build produces RunProdServer, not App). Switch packages.docker to IHP's built-in unoptimized-docker-image which already uses the correct binary path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
125 lines
3.8 KiB
Markdown
125 lines
3.8 KiB
Markdown
# inter-hub on Railiance01 — Runbook
|
|
|
|
## Architecture
|
|
|
|
- **Cluster:** Railiance01 (K3s, 92.205.62.239)
|
|
- **Namespace:** `inter-hub`
|
|
- **Image registry:** `92.205.130.254:32166/coulomb/inter-hub:<sha>` (Gitea on CoulombCore)
|
|
- **Database:** CloudNativePG cluster `net-kingdom-pg` in `databases` namespace
|
|
- RW endpoint: `net-kingdom-pg-rw.databases.svc.cluster.local:5432`
|
|
- Database: `interhub`, User: `interhub`
|
|
- **Ingress:** Traefik → `hub.coulomb.social` (TLS via letsencrypt-prod)
|
|
- **Secrets:** `inter-hub-env` Secret in `inter-hub` namespace
|
|
|
|
## Deployment
|
|
|
|
```bash
|
|
# From workstation (image already built and pushed):
|
|
helm upgrade --install inter-hub deploy/helm/inter-hub \
|
|
--namespace inter-hub --create-namespace \
|
|
--set image.tag=<sha>
|
|
```
|
|
|
|
## Image Build (on haskelseed)
|
|
|
|
```bash
|
|
ssh root@192.168.178.135
|
|
cd /root/inter-hub
|
|
git pull # (requires Gitea auth — see Gitea credentials section)
|
|
nix build .#docker --accept-flake-config --option lazy-trees false
|
|
# Push to Gitea registry:
|
|
skopeo copy docker-archive:result \
|
|
docker://92.205.130.254:32166/coulomb/inter-hub:<sha> \
|
|
--dest-creds "tegwick:<GITEA_TOKEN>" \
|
|
--dest-tls-verify=false
|
|
```
|
|
|
|
## Gitea Registry Credentials
|
|
|
|
The Gitea token for registry push is stored in `~/.config/tea/config.yml` on the
|
|
workstation. If the token has expired, generate a new one:
|
|
1. Go to http://92.205.130.254:32166 → Settings → Applications → Generate new token
|
|
2. Scope: `package:write`
|
|
3. Update `~/.config/tea/config.yml` on the workstation
|
|
4. Update the `GITEA_TOKEN` in any CI/CD secrets
|
|
|
|
## Database Migration
|
|
|
|
IHP migrations run automatically on startup via the init container in the Deployment.
|
|
To run migrations manually:
|
|
|
|
```bash
|
|
kubectl exec -n inter-hub deploy/inter-hub -- /bin/RunProdServer migrate
|
|
```
|
|
|
|
To check migration status:
|
|
```bash
|
|
kubectl exec -n databases net-kingdom-pg-1 -- psql -U postgres interhub -c "\dt"
|
|
```
|
|
|
|
## Logs
|
|
|
|
```bash
|
|
kubectl logs -n inter-hub -l app=inter-hub --tail=100 -f
|
|
# Previous pod logs:
|
|
kubectl logs -n inter-hub -l app=inter-hub --previous --tail=50
|
|
```
|
|
|
|
## Restart / Rollback
|
|
|
|
```bash
|
|
# Restart:
|
|
kubectl rollout restart deployment/inter-hub -n inter-hub
|
|
kubectl rollout status deployment/inter-hub -n inter-hub
|
|
|
|
# Rollback to previous image:
|
|
kubectl rollout undo deployment/inter-hub -n inter-hub
|
|
|
|
# Rollback to specific version:
|
|
helm rollback inter-hub 1 --namespace inter-hub
|
|
```
|
|
|
|
## Secret Rotation
|
|
|
|
To rotate the session secret:
|
|
```bash
|
|
kubectl create secret generic inter-hub-env \
|
|
--namespace inter-hub \
|
|
--from-literal=DATABASE_URL='...' \
|
|
--from-literal=IHP_SESSION_SECRET='<new-64-char-hex>' \
|
|
--from-literal=IHP_BASEURL='https://hub.coulomb.social' \
|
|
--from-literal=PORT='8000' \
|
|
--from-literal=IHP_ENV='Production' \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
kubectl rollout restart deployment/inter-hub -n inter-hub
|
|
```
|
|
|
|
To rotate the database password:
|
|
1. Update the password in PostgreSQL (via kubectl exec to the CNPG pod)
|
|
2. Update the `inter-hub-env` secret
|
|
3. Restart the deployment
|
|
|
|
## Smoke Test
|
|
|
|
```bash
|
|
curl -s https://hub.coulomb.social/ | grep "Inter-Hub" # Landing 200
|
|
curl -s https://hub.coulomb.social/capabilities | grep "Capabilities"
|
|
curl -s https://hub.coulomb.social/api/v2/hubs # 401 expected
|
|
curl -H "Authorization: Bearer <api-key>" https://hub.coulomb.social/api/v2/hubs # 200
|
|
```
|
|
|
|
## Database Connection Check
|
|
|
|
```bash
|
|
kubectl exec -n inter-hub deploy/inter-hub -- \
|
|
/bin/sh -c 'psql $DATABASE_URL -c "SELECT version();"'
|
|
```
|
|
|
|
## haskelseed Build VM
|
|
|
|
- **Host:** 192.168.178.135
|
|
- **Access:** `ssh root@192.168.178.135` (password in team secrets)
|
|
- **Repo:** `/root/inter-hub` (git initialized locally; pull requires Gitea token)
|
|
- **Build logs:** `/tmp/nix-build-docker.log`
|
|
- **Nix store:** `/dev/sdb1` (100 GB, mounted at `/nix`)
|