Implement Gitea Actions runner substrate

2026-06-08 00:31:06 +02:00
parent 3fb63c9a03
commit 19ee47fe82
15 changed files with 1032 additions and 3 deletions
--- a/docs/gitea-actions-runner-substrate.md
+++ b/docs/gitea-actions-runner-substrate.md
@@ -0,0 +1,245 @@
+# Gitea Actions Runner Substrate
+
+Last reviewed: 2026-06-07
+
+Status: implementation contract v1 and attended operating runbook. This
+document does not contain runner registration tokens, package credentials,
+kubeconfigs, SSH keys, or decrypted secret values.
+
+## Purpose
+
+Railiance needs a forge-owned Gitea Actions runner substrate so source and app
+workflows can build, publish, and verify artifacts without each repo inventing
+its own runner assumptions. The immediate blocker is inter-hub: its deployment
+workflow currently targets `self-hosted` and `haskelseed`, but no forge-owned
+runner deployment or health evidence existed after the first Gitea ownership
+move.
+
+This runbook implements the first supported runner model and keeps the
+separation clear:
+
+- `railiance-forge` owns runner placement, registration, labels, health, and
+  recovery.
+- Source and app repos own workflow files and app-specific build/deploy logic.
+- Cluster deploy authority is not bundled into the build runner unless a
+  separate human-reviewed approval says so.
+
+Primary upstream reference: https://docs.gitea.com/usage/actions/act-runner
+
+## First Supported Runner
+
+Use one attended Gitea `act_runner` on haskelseed as the compatibility runner
+for the inter-hub unblock, unless an operator explicitly chooses a separate
+runner host before registration.
+
+Initial contract:
+
+| Field | Value |
+| --- | --- |
+| Runner name | `railiance-haskelseed-build-01` |
+| Runner scope | `coulomb` organization runner, or repository runner if org scope is not approved |
+| Initial host | `haskelseed` |
+| Capacity | `1` |
+| Runtime | Docker-backed or host-backed `act_runner`, confirmed during attended install |
+| Cluster deploy authority | Not included in baseline runner approval |
+| Registry publish authority | Allowed only through repo/workflow-scoped Gitea secrets |
+
+The first runner exists to prove scheduling, build tooling, and registry publish
+readiness. It must not silently become a cluster deployment runner. If a
+workflow needs live cluster access, add a separate label and approval note such
+as `cluster-deploy` or shift deployment to an app-owned release path.
+
+## Label Contract
+
+Register the first runner with compatibility labels for the inter-hub workflow
+and semantic labels for future workflows:
+
+```text
+self-hosted:host,haskelseed:host,linux:host,linux_amd64:host,container-build:host,registry-publish:host
+```
+
+Rules:
+
+- `self-hosted` and `haskelseed` are compatibility labels for the first unblock.
+- New workflows should prefer semantic labels such as `linux`,
+  `container-build`, and `registry-publish`.
+- `registry-publish` means the runner may execute jobs that receive
+  repo-scoped registry credentials from Gitea secrets.
+- Do not add `cluster-read`, `cluster-dry-run`, `cluster-deploy`, or
+  `s5-release-check` until the cluster/platform credential path is reviewed.
+- Keep capacity at `1` until haskelseed resource contention and workload safety
+  are understood.
+
+Gitea runner labels are supplied at registration with `--labels` and, on modern
+Gitea/act_runner versions, can also be adjusted in the generated runner config.
+Use the generated config from the installed runner version as the source of
+truth for exact YAML keys.
+
+## Secret Boundaries
+
+Runner registration and publish credentials are live secrets. They must be
+created and handled outside Git.
+
+Allowed in this repo:
+
+- secret names;
+- operator prompts;
+- paths such as `/run/secrets/railiance/gitea-act-runner-registration-token`;
+- non-secret evidence such as runner name, labels, service status, workflow id,
+  commit SHA, image tag, and digest.
+
+Forbidden in this repo:
+
+- registration token values;
+- Gitea package or registry tokens;
+- tokenized URLs;
+- SSH private keys;
+- kubeconfigs or OpenBao tokens;
+- complete environment dumps from runner jobs.
+
+## Attended Install
+
+Run these steps on the selected runner host. Adjust only the installation path
+and package manager details for the host; do not change the label contract
+without updating this document and State Hub.
+
+1. Prepare the host:
+
+   ```bash
+   sudo install -d -o root -g root -m 0755 /etc/act_runner
+   sudo install -d -o act_runner -g act_runner -m 0750 /var/lib/act_runner
+   sudo install -d -o act_runner -g act_runner -m 0750 /var/cache/act_runner
+   ```
+
+2. Install `act_runner` using the current Gitea-supported binary or package
+   source for the host. Record the version, but do not commit downloaded
+   binaries into this repo.
+
+3. Generate the runner config on the host:
+
+   ```bash
+   sudo -u act_runner /usr/local/bin/act_runner generate-config \
+     | sudo tee /etc/act_runner/config.yaml >/dev/null
+   sudo chmod 0640 /etc/act_runner/config.yaml
+   sudo chown root:act_runner /etc/act_runner/config.yaml
+   ```
+
+4. Edit the generated config so capacity is `1` and labels match this document.
+   If the generated config for the installed version uses a different section
+   name, keep the generated structure and only change the corresponding values.
+
+5. Place the runner registration token in the approved secret path, for example:
+
+   ```text
+   /run/secrets/railiance/gitea-act-runner-registration-token
+   ```
+
+6. Register non-interactively without printing the token:
+
+   ```bash
+   sudo -u act_runner sh -lc '
+     set -eu
+     token="$(cat /run/secrets/railiance/gitea-act-runner-registration-token)"
+     exec /usr/local/bin/act_runner --config /etc/act_runner/config.yaml \
+       register --no-interactive \
+       --instance https://gitea.coulomb.social/ \
+       --token "$token" \
+       --name railiance-haskelseed-build-01 \
+       --labels self-hosted:host,haskelseed:host,linux:host,linux_amd64:host,container-build:host,registry-publish:host
+   '
+   ```
+
+7. Install the matching service template, adjusted to the host paths if needed.
+   For systemd hosts, use `runner/act-runner.service.example`:
+
+   ```bash
+   sudo cp runner/act-runner.service.example /etc/systemd/system/act_runner.service
+   sudo systemctl daemon-reload
+   sudo systemctl enable --now act_runner
+   ```
+
+   For Alpine/OpenRC hosts such as haskelseed, use
+   `runner/act-runner.openrc.example`:
+
+   ```bash
+   sudo install -m 0755 runner/act-runner.openrc.example /etc/init.d/act_runner
+   sudo rc-update add act_runner default
+   sudo rc-service act_runner start
+   ```
+
+   If reusing the current haskelseed root registration at `/root/.runner`, use
+   `runner/act-runner-haskelseed.openrc.example` instead. This is less
+   portable than the dedicated `act_runner` user layout, but it matches the
+   existing registration state without printing or replacing the token.
+
+8. Inspect without exposing secrets:
+
+   ```bash
+   systemctl status act_runner --no-pager
+   journalctl -u act_runner -n 100 --no-pager
+   rc-service act_runner status
+   tail -n 100 /var/log/act_runner.log
+   ```
+
+9. From this repo, run:
+
+   ```bash
+   make runner-status
+
+   # If using the current ops-bridge haskelseed path:
+   RUNNER_HOST=192.168.178.135 \
+   RUNNER_SSH_USER=root \
+   RUNNER_SSH_KEY=/home/worsch/.ssh/id_ops \
+   make runner-status
+   ```
+
+## Recovery
+
+Stop or drain:
+
+```bash
+sudo systemctl stop act_runner
+sudo systemctl disable act_runner
+```
+
+Replace a runner:
+
+1. Stop the old service.
+2. Revoke or rotate the old registration token in Gitea.
+3. Move the old `/var/lib/act_runner/.runner` aside for evidence, not reuse.
+4. Register the replacement with the same approved label contract.
+5. Run the smoke workflow before re-enabling consumer workflows.
+
+Disable a bad registration:
+
+- Stop the service immediately.
+- Remove or disable the runner in Gitea admin/org settings.
+- Rotate any repo secrets that were available to failed jobs if exposure is
+  plausible.
+- Record a State Hub note with runner name, labels, time window, and affected
+  workflows.
+
+## Smoke Workflow
+
+The repo includes `.gitea/workflows/forge-runner-smoke.yaml`. It is deliberately
+small: it checks scheduling, basic host context, Docker availability if present,
+and verifies that baseline cluster/registry secret environment variables are not
+accidentally present.
+
+Do not rerun inter-hub production deploys until this smoke workflow has passed
+on the approved runner and the result is recorded in
+`docs/gitea-actions-runner-evidence.md`.
+
+## Current Blockers
+
+- Direct non-interactive SSH to the bare `haskelseed` alias timed out from this
+  environment on 2026-06-07. The current ops-bridge path reaches haskelseed at
+  `root@192.168.178.135` with `/home/worsch/.ssh/id_ops`.
+- Haskelseed has `act_runner v0.6.1-1-g8e6b3be9` installed and `/root/.runner`
+  registered as `haskelseed`. On 2026-06-07 the OpenRC service was installed
+  from `runner/act-runner-haskelseed.openrc.example`, labels were updated to the
+  first compatibility contract, and the runner daemon declared successfully.
+- `skopeo` is not installed in this environment, so registry tag inspection must
+  run from a host with `skopeo` or use an approved equivalent.
+- Local `make check-tools` still lacks `kubectl`, `helm`, and `sops`; those are
+  separate operator-tool prerequisites for deploy-capable Gitea work.