generated from coulomb/repo-seed
feat(WP-0018/R6): Helm chart and runbook for Railiance01 deployment
Some checks failed
Test / test (push) Has been cancelled
Some checks failed
Test / test (push) Has been cancelled
Helm chart at deploy/helm/inter-hub/ with Deployment, Service, Ingress (Traefik + letsencrypt-prod), and migration init container. Runbook at deploy/railiance/RUNBOOK.md with build, push, rotate, rollback procedures. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
124
deploy/railiance/RUNBOOK.md
Normal file
124
deploy/railiance/RUNBOOK.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# inter-hub on Railiance01 — Runbook
|
||||
|
||||
## Architecture
|
||||
|
||||
- **Cluster:** Railiance01 (K3s, 92.205.62.239)
|
||||
- **Namespace:** `inter-hub`
|
||||
- **Image registry:** `92.205.130.254:32166/coulomb/inter-hub:<sha>` (Gitea on CoulombCore)
|
||||
- **Database:** CloudNativePG cluster `net-kingdom-pg` in `databases` namespace
|
||||
- RW endpoint: `net-kingdom-pg-rw.databases.svc.cluster.local:5432`
|
||||
- Database: `interhub`, User: `interhub`
|
||||
- **Ingress:** Traefik → `hub.coulomb.social` (TLS via letsencrypt-prod)
|
||||
- **Secrets:** `inter-hub-env` Secret in `inter-hub` namespace
|
||||
|
||||
## Deployment
|
||||
|
||||
```bash
|
||||
# From workstation (image already built and pushed):
|
||||
helm upgrade --install inter-hub deploy/helm/inter-hub \
|
||||
--namespace inter-hub --create-namespace \
|
||||
--set image.tag=<sha>
|
||||
```
|
||||
|
||||
## Image Build (on haskelseed)
|
||||
|
||||
```bash
|
||||
ssh root@192.168.178.135
|
||||
cd /root/inter-hub
|
||||
git pull # (requires Gitea auth — see Gitea credentials section)
|
||||
nix build .#docker --accept-flake-config --option lazy-trees false
|
||||
# Push to Gitea registry:
|
||||
skopeo copy docker-archive:result \
|
||||
docker://92.205.130.254:32166/coulomb/inter-hub:<sha> \
|
||||
--dest-creds "tegwick:<GITEA_TOKEN>" \
|
||||
--dest-tls-verify=false
|
||||
```
|
||||
|
||||
## Gitea Registry Credentials
|
||||
|
||||
The Gitea token for registry push is stored in `~/.config/tea/config.yml` on the
|
||||
workstation. If the token has expired, generate a new one:
|
||||
1. Go to http://92.205.130.254:32166 → Settings → Applications → Generate new token
|
||||
2. Scope: `package:write`
|
||||
3. Update `~/.config/tea/config.yml` on the workstation
|
||||
4. Update the `GITEA_TOKEN` in any CI/CD secrets
|
||||
|
||||
## Database Migration
|
||||
|
||||
IHP migrations run automatically on startup via the init container in the Deployment.
|
||||
To run migrations manually:
|
||||
|
||||
```bash
|
||||
kubectl exec -n inter-hub deploy/inter-hub -- /bin/App migrate
|
||||
```
|
||||
|
||||
To check migration status:
|
||||
```bash
|
||||
kubectl exec -n databases net-kingdom-pg-1 -- psql -U postgres interhub -c "\dt"
|
||||
```
|
||||
|
||||
## Logs
|
||||
|
||||
```bash
|
||||
kubectl logs -n inter-hub -l app=inter-hub --tail=100 -f
|
||||
# Previous pod logs:
|
||||
kubectl logs -n inter-hub -l app=inter-hub --previous --tail=50
|
||||
```
|
||||
|
||||
## Restart / Rollback
|
||||
|
||||
```bash
|
||||
# Restart:
|
||||
kubectl rollout restart deployment/inter-hub -n inter-hub
|
||||
kubectl rollout status deployment/inter-hub -n inter-hub
|
||||
|
||||
# Rollback to previous image:
|
||||
kubectl rollout undo deployment/inter-hub -n inter-hub
|
||||
|
||||
# Rollback to specific version:
|
||||
helm rollback inter-hub 1 --namespace inter-hub
|
||||
```
|
||||
|
||||
## Secret Rotation
|
||||
|
||||
To rotate the session secret:
|
||||
```bash
|
||||
kubectl create secret generic inter-hub-env \
|
||||
--namespace inter-hub \
|
||||
--from-literal=DATABASE_URL='...' \
|
||||
--from-literal=IHP_SESSION_SECRET='<new-64-char-hex>' \
|
||||
--from-literal=IHP_BASEURL='https://hub.coulomb.social' \
|
||||
--from-literal=PORT='8000' \
|
||||
--from-literal=IHP_ENV='Production' \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
kubectl rollout restart deployment/inter-hub -n inter-hub
|
||||
```
|
||||
|
||||
To rotate the database password:
|
||||
1. Update the password in PostgreSQL (via kubectl exec to the CNPG pod)
|
||||
2. Update the `inter-hub-env` secret
|
||||
3. Restart the deployment
|
||||
|
||||
## Smoke Test
|
||||
|
||||
```bash
|
||||
curl -s https://hub.coulomb.social/ | grep "Inter-Hub" # Landing 200
|
||||
curl -s https://hub.coulomb.social/capabilities | grep "Capabilities"
|
||||
curl -s https://hub.coulomb.social/api/v2/hubs # 401 expected
|
||||
curl -H "Authorization: Bearer <api-key>" https://hub.coulomb.social/api/v2/hubs # 200
|
||||
```
|
||||
|
||||
## Database Connection Check
|
||||
|
||||
```bash
|
||||
kubectl exec -n inter-hub deploy/inter-hub -- \
|
||||
/bin/sh -c 'psql $DATABASE_URL -c "SELECT version();"'
|
||||
```
|
||||
|
||||
## haskelseed Build VM
|
||||
|
||||
- **Host:** 192.168.178.135
|
||||
- **Access:** `ssh root@192.168.178.135` (password in team secrets)
|
||||
- **Repo:** `/root/inter-hub` (git initialized locally; pull requires Gitea token)
|
||||
- **Build logs:** `/tmp/nix-build-docker.log`
|
||||
- **Nix store:** `/dev/sdb1` (100 GB, mounted at `/nix`)
|
||||
Reference in New Issue
Block a user