chore(deploy): add railiance handoff guardrails [skip ci]

2026-06-14 16:47:24 +02:00
parent fde5525170
commit 333fbcc237
10 changed files with 279 additions and 74 deletions
--- a/deploy/railiance/RUNBOOK.md
+++ b/deploy/railiance/RUNBOOK.md
@@ -4,20 +4,43 @@

 - **Cluster:** Railiance01 (K3s, 92.205.62.239)
 - **Namespace:** `inter-hub`
- **Image registry:** `92.205.130.254:32166/coulomb/inter-hub:<sha>` (Gitea on CoulombCore)
+- **Image registry:** `gitea.coulomb.social/coulomb/inter-hub:<sha>`
 - **Database:** CloudNativePG cluster `net-kingdom-pg` in `databases` namespace
  - RW endpoint: `net-kingdom-pg-rw.databases.svc.cluster.local:5432`
  - Database: `interhub`, User: `interhub`
 - **Ingress:** Traefik → `hub.coulomb.social` (TLS via letsencrypt-prod)
 - **Secrets:** `inter-hub-env` Secret in `inter-hub` namespace
+- **App handoff:** `app.toml` points Railiance operators to
+  `railiance-apps/charts/inter-hub` with values from
+  `railiance-apps/helm/inter-hub-values.yaml`

 ## Deployment

+Normal deployment is handled by Gitea Actions on push to `main`:
+
+- runner labels: `self-hosted`, `haskelseed`
+- build: `nix build .#docker`
+- publish: `gitea.coulomb.social/coulomb/inter-hub:<short-sha>` and `latest`
+- deploy: `helm upgrade --install inter-hub deploy/helm/inter-hub ...`
+- smoke: public landing page and v2 auth gate
+
+Manual deployment from this repo:
+
 ```bash
-# From workstation (image already built and pushed):
 helm upgrade --install inter-hub deploy/helm/inter-hub \
  --namespace inter-hub --create-namespace \
-  --set image.tag=<sha>
+  --set image.tag=<short-sha> \
+  --wait --timeout 5m
+```
+
+Manual deployment through the Railiance app handoff chart:
+
+```bash
+helm upgrade --install inter-hub /home/worsch/railiance-apps/charts/inter-hub \
+  --namespace inter-hub --create-namespace \
+  -f /home/worsch/railiance-apps/helm/inter-hub-values.yaml \
+  --set image.tag=<short-sha> \
+  --wait --timeout 5m
 ```

 ## Image Build (on haskelseed)
@@ -28,42 +51,76 @@ cd /root/inter-hub
 # Build:
 nix build .#docker --log-format raw > /tmp/build.log 2>&1

-# Push — Gitea registry token realm points to gitea.coulomb.social:80 but Gitea
-# only listens on port 32166; skopeo must use a pre-fetched token:
+# Push:
 SHA=$(git rev-parse --short HEAD)
-SKOPEO=/nix/store/fwdagky9lfsyrgzxiq14zijcziazfdsn-skopeo-1.22.2/bin/skopeo
-TOKEN=$(curl -s \
-  "http://92.205.130.254:32166/v2/token?service=container_registry&scope=repository:coulomb/inter-hub:push,pull" \
-  -u 'tegwick:<GITEA_API_KEY>' | awk -F'"' '/token/{print $4}')
-$SKOPEO copy --insecure-policy --dest-tls-verify=false \
+TOKEN=$(curl -fsS \
+  "https://gitea.coulomb.social/v2/token?service=container_registry&scope=repository:coulomb/inter-hub:push,pull" \
+  -u "tegwick:<REGISTRY_TOKEN>" | awk -F'"' '/token/{print $4}')
+skopeo copy --insecure-policy \
  --dest-registry-token "$TOKEN" \
  docker-archive:result \
-  docker://92.205.130.254:32166/coulomb/inter-hub:$SHA
+  docker://gitea.coulomb.social/coulomb/inter-hub:$SHA
 ```

 **Notes:**
- `skopeo` is in the Nix profile but not on PATH — use the full store path above.
- The IHP Nix Docker image has NO `/bin/RunProdServer` symlink. The binary lives at
-  `/nix/store/<hash>-inter-hub/bin/RunProdServer` (hash changes per build).
-  Use `kubectl exec deploy/inter-hub -- /nix/store/*-inter-hub/bin/RunProdServer <cmd>`
-  if a shell is not available (the Nix image has no `/bin/sh`).
+- Haskelseed is a build/deploy runner, not the production app host.
+- The IHP Nix Docker image may not have `/bin/sh`. Prefer Kubernetes-native
+  checks from other pods or the database pod when possible.

 ## Gitea Registry Credentials

-The Gitea token for registry push is stored in `~/.config/tea/config.yml` on the
-workstation. If the token has expired, generate a new one:
-1. Go to http://92.205.130.254:32166 → Settings → Applications → Generate new token
-2. Scope: `package:write`
-3. Update `~/.config/tea/config.yml` on the workstation
-4. Update the `GITEA_TOKEN` in any CI/CD secrets
+The deploy workflow uses the repository Actions secret `REGISTRY_TOKEN` to
+request a short-lived registry bearer token from
+`https://gitea.coulomb.social/v2/token`.
+
+If publishing starts failing with an authentication error:
+1. Generate or rotate a Gitea token with package write access.
+2. Update the `REGISTRY_TOKEN` Actions secret for `coulomb/inter-hub`.
+3. Rerun the workflow or push a non-production test commit.
+
+Do not print token values in logs, State Hub, or commits.
+
+## Runtime Secret Source
+
+The live deployment currently consumes the Kubernetes Secret
+`inter-hub/inter-hub-env`. The durable source file is:
+
+```text
+deploy/railiance/secrets/inter-hub.env.sops.yaml
+```
+
+Create or refresh it from the live Secret using:
+
+```bash
+tmp="$(mktemp)"
+trap 'rm -f "$tmp"' EXIT
+
+kubectl -n inter-hub get secret inter-hub-env -o json \
+  | python3 deploy/railiance/secrets/k8s-secret-json-to-sops-input.py \
+  > "$tmp"
+
+sops --encrypt \
+  --age age1aq8twfd78wvpra0had8cezcnj96tj4q0068edrz5jez8d6xwmflqdepsh4 \
+  "$tmp" > deploy/railiance/secrets/inter-hub.env.sops.yaml
+```
+
+Apply the encrypted source:
+
+```bash
+sops -d deploy/railiance/secrets/inter-hub.env.sops.yaml \
+  | kubectl apply -f -
+kubectl rollout restart deployment/inter-hub -n inter-hub
+kubectl rollout status deployment/inter-hub -n inter-hub
+```

 ## Database Migration

-IHP migrations run automatically on startup via the init container in the Deployment.
-To run migrations manually:
+IHP migrations can be run from the production image when needed. Because the
+image is Nix-built and may not contain a shell, first inspect the binary path:

 ```bash
-kubectl exec -n inter-hub deploy/inter-hub -- /bin/RunProdServer migrate
+kubectl exec -n inter-hub deploy/inter-hub -- find /nix/store -path '*inter-hub*/bin/RunProdServer'
+kubectl exec -n inter-hub deploy/inter-hub -- /nix/store/<hash>-inter-hub/bin/RunProdServer migrate
 ```

 To check migration status:
@@ -97,14 +154,8 @@ helm rollback inter-hub 1 --namespace inter-hub

 To rotate the session secret:
 ```bash
-kubectl create secret generic inter-hub-env \
-  --namespace inter-hub \
-  --from-literal=DATABASE_URL='...' \
-  --from-literal=IHP_SESSION_SECRET='<new-64-char-hex>' \
-  --from-literal=IHP_BASEURL='https://hub.coulomb.social' \
-  --from-literal=PORT='8000' \
-  --from-literal=IHP_ENV='Production' \
-  --dry-run=client -o yaml | kubectl apply -f -
+sops deploy/railiance/secrets/inter-hub.env.sops.yaml
+sops -d deploy/railiance/secrets/inter-hub.env.sops.yaml | kubectl apply -f -
 kubectl rollout restart deployment/inter-hub -n inter-hub
 ```

@@ -116,10 +167,9 @@ To rotate the database password:
 ## Smoke Test

 ```bash
-curl -s https://hub.coulomb.social/ | grep "Inter-Hub"           # Landing 200
-curl -s https://hub.coulomb.social/capabilities | grep "Capabilities"
-curl -s https://hub.coulomb.social/api/v2/hubs                   # 401 expected
-curl -H "Authorization: Bearer <api-key>" https://hub.coulomb.social/api/v2/hubs  # 200
+curl -fsS https://hub.coulomb.social/ | grep "inter-hub"
+curl -fsS https://hub.coulomb.social/api/v2/openapi.json >/dev/null
+curl -s -o /dev/null -w "%{http_code}" https://hub.coulomb.social/api/v2/widgets | grep 401
 ```

 ## Database Connection Check
@@ -153,7 +203,8 @@ EOF
 ## haskelseed Build VM

 - **Host:** 192.168.178.135
- **Access:** `ssh root@192.168.178.135` (password in team secrets)
- **Repo:** `/root/inter-hub` (git initialized locally; pull requires Gitea token)
- **Build logs:** `/tmp/nix-build-docker.log`
+- **Access:** ops-bridge SSH path with the approved operator key
+- **Role:** self-hosted Gitea Actions runner and Nix build machine only
+- **Runner:** OpenRC `act_runner` service registered to `https://gitea.coulomb.social`
+- **Build logs:** Gitea Actions logs and temporary runner work directories
 - **Nix store:** `/dev/sdb1` (100 GB, mounted at `/nix`)
--- a/deploy/railiance/secrets/.gitignore
+++ b/deploy/railiance/secrets/.gitignore
@@ -0,0 +1,6 @@
+*
+!.gitignore
+!README.md
+!*.example.yaml
+!*.sops.yaml
+!*.py
--- a/deploy/railiance/secrets/README.md
+++ b/deploy/railiance/secrets/README.md
@@ -0,0 +1,51 @@
+# inter-hub Runtime Secret
+
+`inter-hub.env.sops.yaml` is the durable source for the production
+`inter-hub/inter-hub-env` Kubernetes Secret. The file is encrypted with the
+shared Railiance age recipient declared in the repo root `.sops.yaml`.
+
+Do not commit plaintext secret material. This directory ignores plaintext files
+by default; only `*.sops.yaml`, examples, docs, and helper scripts are tracked.
+
+## Create Or Refresh
+
+Use an attended operator shell with `kubectl`, `sops`, and access to the shared
+Railiance age identity:
+
+```bash
+tmp="$(mktemp)"
+trap 'rm -f "$tmp"' EXIT
+
+kubectl -n inter-hub get secret inter-hub-env -o json \
+  | python3 deploy/railiance/secrets/k8s-secret-json-to-sops-input.py \
+  > "$tmp"
+
+sops --encrypt \
+  --age age1aq8twfd78wvpra0had8cezcnj96tj4q0068edrz5jez8d6xwmflqdepsh4 \
+  "$tmp" > deploy/railiance/secrets/inter-hub.env.sops.yaml
+```
+
+Review only non-secret metadata before committing:
+
+```bash
+sops -d deploy/railiance/secrets/inter-hub.env.sops.yaml \
+  | sed -n '1,8p'
+```
+
+## Apply
+
+```bash
+sops -d deploy/railiance/secrets/inter-hub.env.sops.yaml \
+  | kubectl apply -f -
+
+kubectl rollout restart deployment/inter-hub -n inter-hub
+kubectl rollout status deployment/inter-hub -n inter-hub
+```
+
+## Expected Keys
+
+- `DATABASE_URL`
+- `IHP_SESSION_SECRET`
+- `IHP_BASEURL`
+- `PORT`
+- `IHP_ENV`
--- a/deploy/railiance/secrets/inter-hub.env.example.yaml
+++ b/deploy/railiance/secrets/inter-hub.env.example.yaml
@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: inter-hub-env
+  namespace: inter-hub
+type: Opaque
+stringData:
+  DATABASE_URL: "postgresql://interhub:<password>@net-kingdom-pg-rw.databases.svc.cluster.local:5432/interhub?sslmode=disable"
+  IHP_SESSION_SECRET: "<64-char-hex>"
+  IHP_BASEURL: "https://hub.coulomb.social"
+  PORT: "8000"
+  IHP_ENV: "Production"
--- a/deploy/railiance/secrets/k8s-secret-json-to-sops-input.py
+++ b/deploy/railiance/secrets/k8s-secret-json-to-sops-input.py
@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+"""Convert a Kubernetes Secret JSON document into a SOPS-ready Secret manifest.
+
+The output contains decoded secret values under stringData and must be redirected
+to a temporary file, encrypted with sops, and removed immediately.
+"""
+
+import base64
+import json
+import sys
+
+
+def yaml_string(value: str) -> str:
+    return json.dumps(value)
+
+
+source = json.load(sys.stdin)
+metadata = source.get("metadata", {})
+name = metadata.get("name", "inter-hub-env")
+namespace = metadata.get("namespace", "inter-hub")
+data = source.get("data", {})
+
+print("apiVersion: v1")
+print("kind: Secret")
+print("metadata:")
+print(f"  name: {yaml_string(name)}")
+print(f"  namespace: {yaml_string(namespace)}")
+print("type: Opaque")
+print("stringData:")
+
+for key in sorted(data):
+    decoded = base64.b64decode(data[key]).decode("utf-8")
+    print(f"  {key}: {yaml_string(decoded)}")