railiance-forge/docs/gitea-actions-runner-substrate.md

# Gitea Actions Runner Substrate

Last reviewed: 2026-06-07

Status: implementation contract v1 and attended operating runbook. This
document does not contain runner registration tokens, package credentials,
kubeconfigs, SSH keys, or decrypted secret values.

## Purpose

Railiance needs a forge-owned Gitea Actions runner substrate so source and app
workflows can build, publish, and verify artifacts without each repo inventing
its own runner assumptions. The immediate blocker is inter-hub: its deployment
workflow currently targets `self-hosted` and `haskelseed`, but no forge-owned
runner deployment or health evidence existed after the first Gitea ownership
move.

This runbook implements the first supported runner model and keeps the
separation clear:

- `railiance-forge` owns runner placement, registration, labels, health, and
  recovery.
- Source and app repos own workflow files and app-specific build/deploy logic.
- Cluster deploy authority is not bundled into the build runner unless a
  separate human-reviewed approval says so.

Primary upstream reference: https://docs.gitea.com/usage/actions/act-runner

## First Supported Runner

Use one attended Gitea `act_runner` on haskelseed as the compatibility runner
for the inter-hub unblock, unless an operator explicitly chooses a separate
runner host before registration.

Initial contract:

| Field | Value |
| --- | --- |
| Runner name | `railiance-haskelseed-build-01` |
| Runner scope | `coulomb` organization runner, or repository runner if org scope is not approved |
| Initial host | `haskelseed` |
| Capacity | `1` |
| Runtime | Docker-backed or host-backed `act_runner`, confirmed during attended install |
| Cluster deploy authority | Not included in baseline runner approval |
| Registry publish authority | Allowed only through repo/workflow-scoped Gitea secrets |

The first runner exists to prove scheduling, build tooling, and registry publish
readiness. It must not silently become a cluster deployment runner. If a
workflow needs live cluster access, add a separate label and approval note such
as `cluster-deploy` or shift deployment to an app-owned release path.

## Label Contract

Register the first runner with compatibility labels for the inter-hub workflow
and semantic labels for future workflows:

```text
self-hosted:host,haskelseed:host,linux:host,linux_amd64:host,container-build:host,registry-publish:host
```

Rules:

- `self-hosted` and `haskelseed` are compatibility labels for the first unblock.
- New workflows should prefer semantic labels such as `linux`,
  `container-build`, and `registry-publish`.
- `registry-publish` means the runner may execute jobs that receive
  repo-scoped registry credentials from Gitea secrets.
- Do not add `cluster-read`, `cluster-dry-run`, `cluster-deploy`, or
  `s5-release-check` until the cluster/platform credential path is reviewed.
- Keep capacity at `1` until haskelseed resource contention and workload safety
  are understood.

Gitea runner labels are supplied at registration with `--labels` and, on modern
Gitea/act_runner versions, can also be adjusted in the generated runner config.
Use the generated config from the installed runner version as the source of
truth for exact YAML keys.

## Secret Boundaries

Runner registration and publish credentials are live secrets. They must be
created and handled outside Git.

Allowed in this repo:

- secret names;
- operator prompts;
- paths such as `/run/secrets/railiance/gitea-act-runner-registration-token`;
- non-secret evidence such as runner name, labels, service status, workflow id,
  commit SHA, image tag, and digest.

Forbidden in this repo:

- registration token values;
- Gitea package or registry tokens;
- tokenized URLs;
- SSH private keys;
- kubeconfigs or OpenBao tokens;
- complete environment dumps from runner jobs.

## Attended Install

Run these steps on the selected runner host. Adjust only the installation path
and package manager details for the host; do not change the label contract
without updating this document and State Hub.

1. Prepare the host:

   ```bash
   sudo install -d -o root -g root -m 0755 /etc/act_runner
   sudo install -d -o act_runner -g act_runner -m 0750 /var/lib/act_runner
   sudo install -d -o act_runner -g act_runner -m 0750 /var/cache/act_runner
   ```

2. Install `act_runner` using the current Gitea-supported binary or package
   source for the host. Record the version, but do not commit downloaded
   binaries into this repo.

3. Generate the runner config on the host:

   ```bash
   sudo -u act_runner /usr/local/bin/act_runner generate-config \
     | sudo tee /etc/act_runner/config.yaml >/dev/null
   sudo chmod 0640 /etc/act_runner/config.yaml
   sudo chown root:act_runner /etc/act_runner/config.yaml
   ```

4. Edit the generated config so capacity is `1` and labels match this document.
   If the generated config for the installed version uses a different section
   name, keep the generated structure and only change the corresponding values.

5. Place the runner registration token in the approved secret path, for example:

   ```text
   /run/secrets/railiance/gitea-act-runner-registration-token
   ```

6. Register non-interactively without printing the token:

   ```bash
   sudo -u act_runner sh -lc '
     set -eu
     token="$(cat /run/secrets/railiance/gitea-act-runner-registration-token)"
     exec /usr/local/bin/act_runner --config /etc/act_runner/config.yaml \
       register --no-interactive \
       --instance https://gitea.coulomb.social/ \
       --token "$token" \
       --name railiance-haskelseed-build-01 \
       --labels self-hosted:host,haskelseed:host,linux:host,linux_amd64:host,container-build:host,registry-publish:host
   '
   ```

7. Install the matching service template, adjusted to the host paths if needed.
   For systemd hosts, use `runner/act-runner.service.example`:

   ```bash
   sudo cp runner/act-runner.service.example /etc/systemd/system/act_runner.service
   sudo systemctl daemon-reload
   sudo systemctl enable --now act_runner
   ```

   For Alpine/OpenRC hosts such as haskelseed, use
   `runner/act-runner.openrc.example`:

   ```bash
   sudo install -m 0755 runner/act-runner.openrc.example /etc/init.d/act_runner
   sudo rc-update add act_runner default
   sudo rc-service act_runner start
   ```

   If reusing the current haskelseed root registration at `/root/.runner`, use
   `runner/act-runner-haskelseed.openrc.example` instead. This is less
   portable than the dedicated `act_runner` user layout, but it matches the
   existing registration state without printing or replacing the token.

8. Inspect without exposing secrets:

   ```bash
   systemctl status act_runner --no-pager
   journalctl -u act_runner -n 100 --no-pager
   rc-service act_runner status
   tail -n 100 /var/log/act_runner.log
   ```

9. From this repo, run:

   ```bash
   make runner-status

   # If using the current ops-bridge haskelseed path:
   RUNNER_HOST=192.168.178.135 \
   RUNNER_SSH_USER=root \
   RUNNER_SSH_KEY=/home/worsch/.ssh/id_ops \
   make runner-status
   ```

## Recovery

Stop or drain:

```bash
sudo systemctl stop act_runner
sudo systemctl disable act_runner
```

Replace a runner:

1. Stop the old service.
2. Revoke or rotate the old registration token in Gitea.
3. Move the old `/var/lib/act_runner/.runner` aside for evidence, not reuse.
4. Register the replacement with the same approved label contract.
5. Run the smoke workflow before re-enabling consumer workflows.

Disable a bad registration:

- Stop the service immediately.
- Remove or disable the runner in Gitea admin/org settings.
- Rotate any repo secrets that were available to failed jobs if exposure is
  plausible.
- Record a State Hub note with runner name, labels, time window, and affected
  workflows.

## Smoke Workflow

The repo includes `.gitea/workflows/forge-runner-smoke.yaml`. It is deliberately
small: it checks scheduling, basic host context, Docker availability if present,
and verifies that baseline cluster/registry secret environment variables are not
accidentally present.

Do not rerun inter-hub production deploys until this smoke workflow has passed
on the approved runner and the result is recorded in
`docs/gitea-actions-runner-evidence.md`.

## Current Blockers

- Direct non-interactive SSH to the bare `haskelseed` alias timed out from this
  environment on 2026-06-07. The current ops-bridge path reaches haskelseed at
  `root@192.168.178.135` with `/home/worsch/.ssh/id_ops`.
- Haskelseed has `act_runner v0.6.1-1-g8e6b3be9` installed and `/root/.runner`
  registered as `haskelseed`. On 2026-06-07 the OpenRC service was installed
  from `runner/act-runner-haskelseed.openrc.example`, labels were updated to the
  first compatibility contract, and the runner daemon declared successfully.
- Gitea created `forge-runner-smoke.yaml #1` for commit `19ee47fe82`, but the
  run is still `Waiting`; authenticated Gitea Actions inspection is needed to
  decide whether this is approval, scheduling, or runner scope.
- `skopeo` is not installed in this environment, so registry tag inspection must
  run from a host with `skopeo` or use an approved equivalent.
- Local `make check-tools` still lacks `kubectl`, `helm`, and `sops`; those are
  separate operator-tool prerequisites for deploy-capable Gitea work.