generated from coulomb/repo-seed
chore(deploy): add railiance handoff guardrails [skip ci]
This commit is contained in:
@@ -4,20 +4,43 @@
|
||||
|
||||
- **Cluster:** Railiance01 (K3s, 92.205.62.239)
|
||||
- **Namespace:** `inter-hub`
|
||||
- **Image registry:** `92.205.130.254:32166/coulomb/inter-hub:<sha>` (Gitea on CoulombCore)
|
||||
- **Image registry:** `gitea.coulomb.social/coulomb/inter-hub:<sha>`
|
||||
- **Database:** CloudNativePG cluster `net-kingdom-pg` in `databases` namespace
|
||||
- RW endpoint: `net-kingdom-pg-rw.databases.svc.cluster.local:5432`
|
||||
- Database: `interhub`, User: `interhub`
|
||||
- **Ingress:** Traefik → `hub.coulomb.social` (TLS via letsencrypt-prod)
|
||||
- **Secrets:** `inter-hub-env` Secret in `inter-hub` namespace
|
||||
- **App handoff:** `app.toml` points Railiance operators to
|
||||
`railiance-apps/charts/inter-hub` with values from
|
||||
`railiance-apps/helm/inter-hub-values.yaml`
|
||||
|
||||
## Deployment
|
||||
|
||||
Normal deployment is handled by Gitea Actions on push to `main`:
|
||||
|
||||
- runner labels: `self-hosted`, `haskelseed`
|
||||
- build: `nix build .#docker`
|
||||
- publish: `gitea.coulomb.social/coulomb/inter-hub:<short-sha>` and `latest`
|
||||
- deploy: `helm upgrade --install inter-hub deploy/helm/inter-hub ...`
|
||||
- smoke: public landing page and v2 auth gate
|
||||
|
||||
Manual deployment from this repo:
|
||||
|
||||
```bash
|
||||
# From workstation (image already built and pushed):
|
||||
helm upgrade --install inter-hub deploy/helm/inter-hub \
|
||||
--namespace inter-hub --create-namespace \
|
||||
--set image.tag=<sha>
|
||||
--set image.tag=<short-sha> \
|
||||
--wait --timeout 5m
|
||||
```
|
||||
|
||||
Manual deployment through the Railiance app handoff chart:
|
||||
|
||||
```bash
|
||||
helm upgrade --install inter-hub /home/worsch/railiance-apps/charts/inter-hub \
|
||||
--namespace inter-hub --create-namespace \
|
||||
-f /home/worsch/railiance-apps/helm/inter-hub-values.yaml \
|
||||
--set image.tag=<short-sha> \
|
||||
--wait --timeout 5m
|
||||
```
|
||||
|
||||
## Image Build (on haskelseed)
|
||||
@@ -28,42 +51,76 @@ cd /root/inter-hub
|
||||
# Build:
|
||||
nix build .#docker --log-format raw > /tmp/build.log 2>&1
|
||||
|
||||
# Push — Gitea registry token realm points to gitea.coulomb.social:80 but Gitea
|
||||
# only listens on port 32166; skopeo must use a pre-fetched token:
|
||||
# Push:
|
||||
SHA=$(git rev-parse --short HEAD)
|
||||
SKOPEO=/nix/store/fwdagky9lfsyrgzxiq14zijcziazfdsn-skopeo-1.22.2/bin/skopeo
|
||||
TOKEN=$(curl -s \
|
||||
"http://92.205.130.254:32166/v2/token?service=container_registry&scope=repository:coulomb/inter-hub:push,pull" \
|
||||
-u 'tegwick:<GITEA_API_KEY>' | awk -F'"' '/token/{print $4}')
|
||||
$SKOPEO copy --insecure-policy --dest-tls-verify=false \
|
||||
TOKEN=$(curl -fsS \
|
||||
"https://gitea.coulomb.social/v2/token?service=container_registry&scope=repository:coulomb/inter-hub:push,pull" \
|
||||
-u "tegwick:<REGISTRY_TOKEN>" | awk -F'"' '/token/{print $4}')
|
||||
skopeo copy --insecure-policy \
|
||||
--dest-registry-token "$TOKEN" \
|
||||
docker-archive:result \
|
||||
docker://92.205.130.254:32166/coulomb/inter-hub:$SHA
|
||||
docker://gitea.coulomb.social/coulomb/inter-hub:$SHA
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
- `skopeo` is in the Nix profile but not on PATH — use the full store path above.
|
||||
- The IHP Nix Docker image has NO `/bin/RunProdServer` symlink. The binary lives at
|
||||
`/nix/store/<hash>-inter-hub/bin/RunProdServer` (hash changes per build).
|
||||
Use `kubectl exec deploy/inter-hub -- /nix/store/*-inter-hub/bin/RunProdServer <cmd>`
|
||||
if a shell is not available (the Nix image has no `/bin/sh`).
|
||||
- Haskelseed is a build/deploy runner, not the production app host.
|
||||
- The IHP Nix Docker image may not have `/bin/sh`. Prefer Kubernetes-native
|
||||
checks from other pods or the database pod when possible.
|
||||
|
||||
## Gitea Registry Credentials
|
||||
|
||||
The Gitea token for registry push is stored in `~/.config/tea/config.yml` on the
|
||||
workstation. If the token has expired, generate a new one:
|
||||
1. Go to http://92.205.130.254:32166 → Settings → Applications → Generate new token
|
||||
2. Scope: `package:write`
|
||||
3. Update `~/.config/tea/config.yml` on the workstation
|
||||
4. Update the `GITEA_TOKEN` in any CI/CD secrets
|
||||
The deploy workflow uses the repository Actions secret `REGISTRY_TOKEN` to
|
||||
request a short-lived registry bearer token from
|
||||
`https://gitea.coulomb.social/v2/token`.
|
||||
|
||||
If publishing starts failing with an authentication error:
|
||||
1. Generate or rotate a Gitea token with package write access.
|
||||
2. Update the `REGISTRY_TOKEN` Actions secret for `coulomb/inter-hub`.
|
||||
3. Rerun the workflow or push a non-production test commit.
|
||||
|
||||
Do not print token values in logs, State Hub, or commits.
|
||||
|
||||
## Runtime Secret Source
|
||||
|
||||
The live deployment currently consumes the Kubernetes Secret
|
||||
`inter-hub/inter-hub-env`. The durable source file is:
|
||||
|
||||
```text
|
||||
deploy/railiance/secrets/inter-hub.env.sops.yaml
|
||||
```
|
||||
|
||||
Create or refresh it from the live Secret using:
|
||||
|
||||
```bash
|
||||
tmp="$(mktemp)"
|
||||
trap 'rm -f "$tmp"' EXIT
|
||||
|
||||
kubectl -n inter-hub get secret inter-hub-env -o json \
|
||||
| python3 deploy/railiance/secrets/k8s-secret-json-to-sops-input.py \
|
||||
> "$tmp"
|
||||
|
||||
sops --encrypt \
|
||||
--age age1aq8twfd78wvpra0had8cezcnj96tj4q0068edrz5jez8d6xwmflqdepsh4 \
|
||||
"$tmp" > deploy/railiance/secrets/inter-hub.env.sops.yaml
|
||||
```
|
||||
|
||||
Apply the encrypted source:
|
||||
|
||||
```bash
|
||||
sops -d deploy/railiance/secrets/inter-hub.env.sops.yaml \
|
||||
| kubectl apply -f -
|
||||
kubectl rollout restart deployment/inter-hub -n inter-hub
|
||||
kubectl rollout status deployment/inter-hub -n inter-hub
|
||||
```
|
||||
|
||||
## Database Migration
|
||||
|
||||
IHP migrations run automatically on startup via the init container in the Deployment.
|
||||
To run migrations manually:
|
||||
IHP migrations can be run from the production image when needed. Because the
|
||||
image is Nix-built and may not contain a shell, first inspect the binary path:
|
||||
|
||||
```bash
|
||||
kubectl exec -n inter-hub deploy/inter-hub -- /bin/RunProdServer migrate
|
||||
kubectl exec -n inter-hub deploy/inter-hub -- find /nix/store -path '*inter-hub*/bin/RunProdServer'
|
||||
kubectl exec -n inter-hub deploy/inter-hub -- /nix/store/<hash>-inter-hub/bin/RunProdServer migrate
|
||||
```
|
||||
|
||||
To check migration status:
|
||||
@@ -97,14 +154,8 @@ helm rollback inter-hub 1 --namespace inter-hub
|
||||
|
||||
To rotate the session secret:
|
||||
```bash
|
||||
kubectl create secret generic inter-hub-env \
|
||||
--namespace inter-hub \
|
||||
--from-literal=DATABASE_URL='...' \
|
||||
--from-literal=IHP_SESSION_SECRET='<new-64-char-hex>' \
|
||||
--from-literal=IHP_BASEURL='https://hub.coulomb.social' \
|
||||
--from-literal=PORT='8000' \
|
||||
--from-literal=IHP_ENV='Production' \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
sops deploy/railiance/secrets/inter-hub.env.sops.yaml
|
||||
sops -d deploy/railiance/secrets/inter-hub.env.sops.yaml | kubectl apply -f -
|
||||
kubectl rollout restart deployment/inter-hub -n inter-hub
|
||||
```
|
||||
|
||||
@@ -116,10 +167,9 @@ To rotate the database password:
|
||||
## Smoke Test
|
||||
|
||||
```bash
|
||||
curl -s https://hub.coulomb.social/ | grep "Inter-Hub" # Landing 200
|
||||
curl -s https://hub.coulomb.social/capabilities | grep "Capabilities"
|
||||
curl -s https://hub.coulomb.social/api/v2/hubs # 401 expected
|
||||
curl -H "Authorization: Bearer <api-key>" https://hub.coulomb.social/api/v2/hubs # 200
|
||||
curl -fsS https://hub.coulomb.social/ | grep "inter-hub"
|
||||
curl -fsS https://hub.coulomb.social/api/v2/openapi.json >/dev/null
|
||||
curl -s -o /dev/null -w "%{http_code}" https://hub.coulomb.social/api/v2/widgets | grep 401
|
||||
```
|
||||
|
||||
## Database Connection Check
|
||||
@@ -153,7 +203,8 @@ EOF
|
||||
## haskelseed Build VM
|
||||
|
||||
- **Host:** 192.168.178.135
|
||||
- **Access:** `ssh root@192.168.178.135` (password in team secrets)
|
||||
- **Repo:** `/root/inter-hub` (git initialized locally; pull requires Gitea token)
|
||||
- **Build logs:** `/tmp/nix-build-docker.log`
|
||||
- **Access:** ops-bridge SSH path with the approved operator key
|
||||
- **Role:** self-hosted Gitea Actions runner and Nix build machine only
|
||||
- **Runner:** OpenRC `act_runner` service registered to `https://gitea.coulomb.social`
|
||||
- **Build logs:** Gitea Actions logs and temporary runner work directories
|
||||
- **Nix store:** `/dev/sdb1` (100 GB, mounted at `/nix`)
|
||||
|
||||
Reference in New Issue
Block a user