generated from coulomb/repo-seed
tegwick
a60cc24914
Some checks failed
Build and Deploy / build-push-deploy (push) Has been cancelled
The Helm init container used /bin/RunProdServer which doesn't exist in IHP's Nix Docker image (binary is at a Nix store path). Additionally, IHP v1.5's RunProdServer starts the server after migrating — it never exits — so init containers are the wrong pattern. IHP applies schema changes on startup. Changes: - Remove initContainers block from deployment.yaml entirely - Set runMigrations: false as default in values.yaml - Update RUNBOOK.md with correct skopeo push procedure (pre-fetch bearer token to work around Gitea's misconfigured token realm URL: port 80 vs actual 32166) - Add note that the Nix image has no /bin/sh or /bin/RunProdServer wrapper k3s registry auth: credentials added to /etc/rancher/k3s/registries.yaml and iptables DNAT rule added on Railiance01 (92.205.130.254:80 → 32166) so the ACME token realm redirect works. hub.coulomb.social DNS A record still needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
139 lines
4.5 KiB
Markdown
139 lines
4.5 KiB
Markdown
# inter-hub on Railiance01 — Runbook
|
|
|
|
## Architecture
|
|
|
|
- **Cluster:** Railiance01 (K3s, 92.205.62.239)
|
|
- **Namespace:** `inter-hub`
|
|
- **Image registry:** `92.205.130.254:32166/coulomb/inter-hub:<sha>` (Gitea on CoulombCore)
|
|
- **Database:** CloudNativePG cluster `net-kingdom-pg` in `databases` namespace
|
|
- RW endpoint: `net-kingdom-pg-rw.databases.svc.cluster.local:5432`
|
|
- Database: `interhub`, User: `interhub`
|
|
- **Ingress:** Traefik → `hub.coulomb.social` (TLS via letsencrypt-prod)
|
|
- **Secrets:** `inter-hub-env` Secret in `inter-hub` namespace
|
|
|
|
## Deployment
|
|
|
|
```bash
|
|
# From workstation (image already built and pushed):
|
|
helm upgrade --install inter-hub deploy/helm/inter-hub \
|
|
--namespace inter-hub --create-namespace \
|
|
--set image.tag=<sha>
|
|
```
|
|
|
|
## Image Build (on haskelseed)
|
|
|
|
```bash
|
|
ssh root@192.168.178.135
|
|
cd /root/inter-hub
|
|
# Build:
|
|
nix build .#docker --log-format raw > /tmp/build.log 2>&1
|
|
|
|
# Push — Gitea registry token realm points to gitea.coulomb.social:80 but Gitea
|
|
# only listens on port 32166; skopeo must use a pre-fetched token:
|
|
SHA=$(git rev-parse --short HEAD)
|
|
SKOPEO=/nix/store/fwdagky9lfsyrgzxiq14zijcziazfdsn-skopeo-1.22.2/bin/skopeo
|
|
TOKEN=$(curl -s \
|
|
"http://92.205.130.254:32166/v2/token?service=container_registry&scope=repository:coulomb/inter-hub:push,pull" \
|
|
-u 'tegwick:<GITEA_API_KEY>' | awk -F'"' '/token/{print $4}')
|
|
$SKOPEO copy --insecure-policy --dest-tls-verify=false \
|
|
--dest-registry-token "$TOKEN" \
|
|
docker-archive:result \
|
|
docker://92.205.130.254:32166/coulomb/inter-hub:$SHA
|
|
```
|
|
|
|
**Notes:**
|
|
- `skopeo` is in the Nix profile but not on PATH — use the full store path above.
|
|
- The IHP Nix Docker image has NO `/bin/RunProdServer` symlink. The binary lives at
|
|
`/nix/store/<hash>-inter-hub/bin/RunProdServer` (hash changes per build).
|
|
Use `kubectl exec deploy/inter-hub -- /nix/store/*-inter-hub/bin/RunProdServer <cmd>`
|
|
if a shell is not available (the Nix image has no `/bin/sh`).
|
|
|
|
## Gitea Registry Credentials
|
|
|
|
The Gitea token for registry push is stored in `~/.config/tea/config.yml` on the
|
|
workstation. If the token has expired, generate a new one:
|
|
1. Go to http://92.205.130.254:32166 → Settings → Applications → Generate new token
|
|
2. Scope: `package:write`
|
|
3. Update `~/.config/tea/config.yml` on the workstation
|
|
4. Update the `GITEA_TOKEN` in any CI/CD secrets
|
|
|
|
## Database Migration
|
|
|
|
IHP migrations run automatically on startup via the init container in the Deployment.
|
|
To run migrations manually:
|
|
|
|
```bash
|
|
kubectl exec -n inter-hub deploy/inter-hub -- /bin/RunProdServer migrate
|
|
```
|
|
|
|
To check migration status:
|
|
```bash
|
|
kubectl exec -n databases net-kingdom-pg-1 -- psql -U postgres interhub -c "\dt"
|
|
```
|
|
|
|
## Logs
|
|
|
|
```bash
|
|
kubectl logs -n inter-hub -l app=inter-hub --tail=100 -f
|
|
# Previous pod logs:
|
|
kubectl logs -n inter-hub -l app=inter-hub --previous --tail=50
|
|
```
|
|
|
|
## Restart / Rollback
|
|
|
|
```bash
|
|
# Restart:
|
|
kubectl rollout restart deployment/inter-hub -n inter-hub
|
|
kubectl rollout status deployment/inter-hub -n inter-hub
|
|
|
|
# Rollback to previous image:
|
|
kubectl rollout undo deployment/inter-hub -n inter-hub
|
|
|
|
# Rollback to specific version:
|
|
helm rollback inter-hub 1 --namespace inter-hub
|
|
```
|
|
|
|
## Secret Rotation
|
|
|
|
To rotate the session secret:
|
|
```bash
|
|
kubectl create secret generic inter-hub-env \
|
|
--namespace inter-hub \
|
|
--from-literal=DATABASE_URL='...' \
|
|
--from-literal=IHP_SESSION_SECRET='<new-64-char-hex>' \
|
|
--from-literal=IHP_BASEURL='https://hub.coulomb.social' \
|
|
--from-literal=PORT='8000' \
|
|
--from-literal=IHP_ENV='Production' \
|
|
--dry-run=client -o yaml | kubectl apply -f -
|
|
kubectl rollout restart deployment/inter-hub -n inter-hub
|
|
```
|
|
|
|
To rotate the database password:
|
|
1. Update the password in PostgreSQL (via kubectl exec to the CNPG pod)
|
|
2. Update the `inter-hub-env` secret
|
|
3. Restart the deployment
|
|
|
|
## Smoke Test
|
|
|
|
```bash
|
|
curl -s https://hub.coulomb.social/ | grep "Inter-Hub" # Landing 200
|
|
curl -s https://hub.coulomb.social/capabilities | grep "Capabilities"
|
|
curl -s https://hub.coulomb.social/api/v2/hubs # 401 expected
|
|
curl -H "Authorization: Bearer <api-key>" https://hub.coulomb.social/api/v2/hubs # 200
|
|
```
|
|
|
|
## Database Connection Check
|
|
|
|
```bash
|
|
kubectl exec -n inter-hub deploy/inter-hub -- \
|
|
/bin/sh -c 'psql $DATABASE_URL -c "SELECT version();"'
|
|
```
|
|
|
|
## haskelseed Build VM
|
|
|
|
- **Host:** 192.168.178.135
|
|
- **Access:** `ssh root@192.168.178.135` (password in team secrets)
|
|
- **Repo:** `/root/inter-hub` (git initialized locally; pull requires Gitea token)
|
|
- **Build logs:** `/tmp/nix-build-docker.log`
|
|
- **Nix store:** `/dev/sdb1` (100 GB, mounted at `/nix`)
|