Files

Build and Deploy / build-push-deploy (push) Has been cancelled

Details

fix(deploy): remove broken init container, document registry push workaround

The Helm init container used /bin/RunProdServer which doesn't exist in IHP's
Nix Docker image (binary is at a Nix store path). Additionally, IHP v1.5's
RunProdServer starts the server after migrating — it never exits — so init
containers are the wrong pattern. IHP applies schema changes on startup.

Changes:
- Remove initContainers block from deployment.yaml entirely
- Set runMigrations: false as default in values.yaml
- Update RUNBOOK.md with correct skopeo push procedure (pre-fetch bearer token
  to work around Gitea's misconfigured token realm URL: port 80 vs actual 32166)
- Add note that the Nix image has no /bin/sh or /bin/RunProdServer wrapper

k3s registry auth: credentials added to /etc/rancher/k3s/registries.yaml and
iptables DNAT rule added on Railiance01 (92.205.130.254:80 → 32166) so the
ACME token realm redirect works. hub.coulomb.social DNS A record still needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-05-02 17:45:59 +02:00

4.5 KiB

Raw Blame History

inter-hub on Railiance01 — Runbook

Architecture

Cluster: Railiance01 (K3s, 92.205.62.239)
Namespace: inter-hub
Image registry: 92.205.130.254:32166/coulomb/inter-hub:<sha> (Gitea on CoulombCore)
Database: CloudNativePG cluster net-kingdom-pg in databases namespace
- RW endpoint: net-kingdom-pg-rw.databases.svc.cluster.local:5432
- Database: interhub, User: interhub
Ingress: Traefik → hub.coulomb.social (TLS via letsencrypt-prod)
Secrets: inter-hub-env Secret in inter-hub namespace

Deployment

# From workstation (image already built and pushed):
helm upgrade --install inter-hub deploy/helm/inter-hub \
  --namespace inter-hub --create-namespace \
  --set image.tag=<sha>

Image Build (on haskelseed)

ssh root@192.168.178.135
cd /root/inter-hub
# Build:
nix build .#docker --log-format raw > /tmp/build.log 2>&1

# Push — Gitea registry token realm points to gitea.coulomb.social:80 but Gitea
# only listens on port 32166; skopeo must use a pre-fetched token:
SHA=$(git rev-parse --short HEAD)
SKOPEO=/nix/store/fwdagky9lfsyrgzxiq14zijcziazfdsn-skopeo-1.22.2/bin/skopeo
TOKEN=$(curl -s \
  "http://92.205.130.254:32166/v2/token?service=container_registry&scope=repository:coulomb/inter-hub:push,pull" \
  -u 'tegwick:<GITEA_API_KEY>' | awk -F'"' '/token/{print $4}')
$SKOPEO copy --insecure-policy --dest-tls-verify=false \
  --dest-registry-token "$TOKEN" \
  docker-archive:result \
  docker://92.205.130.254:32166/coulomb/inter-hub:$SHA

Notes:

skopeo is in the Nix profile but not on PATH — use the full store path above.
The IHP Nix Docker image has NO /bin/RunProdServer symlink. The binary lives at /nix/store/<hash>-inter-hub/bin/RunProdServer (hash changes per build). Use kubectl exec deploy/inter-hub -- /nix/store/*-inter-hub/bin/RunProdServer <cmd> if a shell is not available (the Nix image has no /bin/sh).

Gitea Registry Credentials

The Gitea token for registry push is stored in ~/.config/tea/config.yml on the workstation. If the token has expired, generate a new one:

Go to http://92.205.130.254:32166 → Settings → Applications → Generate new token
Scope: package:write
Update ~/.config/tea/config.yml on the workstation
Update the GITEA_TOKEN in any CI/CD secrets

Database Migration

IHP migrations run automatically on startup via the init container in the Deployment. To run migrations manually:

kubectl exec -n inter-hub deploy/inter-hub -- /bin/RunProdServer migrate

To check migration status:

kubectl exec -n databases net-kingdom-pg-1 -- psql -U postgres interhub -c "\dt"

Logs

kubectl logs -n inter-hub -l app=inter-hub --tail=100 -f
# Previous pod logs:
kubectl logs -n inter-hub -l app=inter-hub --previous --tail=50

Restart / Rollback

# Restart:
kubectl rollout restart deployment/inter-hub -n inter-hub
kubectl rollout status deployment/inter-hub -n inter-hub

# Rollback to previous image:
kubectl rollout undo deployment/inter-hub -n inter-hub

# Rollback to specific version:
helm rollback inter-hub 1 --namespace inter-hub

Secret Rotation

To rotate the session secret:

kubectl create secret generic inter-hub-env \
  --namespace inter-hub \
  --from-literal=DATABASE_URL='...' \
  --from-literal=IHP_SESSION_SECRET='<new-64-char-hex>' \
  --from-literal=IHP_BASEURL='https://hub.coulomb.social' \
  --from-literal=PORT='8000' \
  --from-literal=IHP_ENV='Production' \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/inter-hub -n inter-hub

To rotate the database password:

Update the password in PostgreSQL (via kubectl exec to the CNPG pod)
Update the inter-hub-env secret
Restart the deployment

Smoke Test

curl -s https://hub.coulomb.social/ | grep "Inter-Hub"           # Landing 200
curl -s https://hub.coulomb.social/capabilities | grep "Capabilities"
curl -s https://hub.coulomb.social/api/v2/hubs                   # 401 expected
curl -H "Authorization: Bearer <api-key>" https://hub.coulomb.social/api/v2/hubs  # 200

Database Connection Check

kubectl exec -n inter-hub deploy/inter-hub -- \
  /bin/sh -c 'psql $DATABASE_URL -c "SELECT version();"'

haskelseed Build VM

Host: 192.168.178.135
Access: ssh root@192.168.178.135 (password in team secrets)
Repo: /root/inter-hub (git initialized locally; pull requires Gitea token)
Build logs: /tmp/nix-build-docker.log
Nix store: /dev/sdb1 (100 GB, mounted at /nix)

4.5 KiB Raw Blame History