railiance-forge/docs/gitea-actions-runner-evidence.md

# Gitea Actions Runner Evidence

Last updated: 2026-06-08

Status: haskelseed is registered as a `coulomb` organization Gitea Actions
runner and the forge smoke workflow has passed. Inter-hub now fails after runner
scheduling, so the remaining blocker is workload-specific rather than runner
visibility.

## Workstream

- Workplan: `FORGE-WP-0003-actions-runner-substrate`
- State Hub workstream: `149a0316-64d1-4664-96d0-274577c32e63`
- Immediate consumer blocker: inter-hub `R7` waits on a runner matching
  `self-hosted` and `haskelseed`.

## Local Probe Results

Collected from `/home/worsch/railiance-forge` on 2026-06-07.

| Probe | Result | Note |
| --- | --- | --- |
| `curl` | available at `/usr/bin/curl` | Used for public endpoint checks. |
| `ssh` | available at `/usr/bin/ssh` | Direct `haskelseed` alias timed out; ops-bridge path reaches `root@192.168.178.135`. |
| `docker` | available at `/usr/bin/docker` | Local presence only; runner host Docker still pending. |
| `skopeo` | not available | Registry tag inspection pending. |
| local `act_runner` | not available | Haskelseed has `/usr/local/bin/act_runner`; local workstation does not. |
| `kubectl`, `helm`, `sops` | not available in `make check-tools` | Separate operator-tool gap for deploy-capable targets. |

Public endpoint checks from this environment:

Historical note: the root/API `404` results below were collected before
`FORGE-WP-0004` made `https://gitea.coulomb.social/` the standard public Gitea
web endpoint on 2026-06-13.

| Endpoint | Result | Interpretation |
| --- | --- | --- |
| `https://gitea.coulomb.social/` | HTTP `404` | Public root route is not a useful Gitea web health signal here. |
| `https://gitea.coulomb.social/api/v1/version` | HTTP `404` | Public API version route is not exposed through the current ingress path. |
| `https://gitea.coulomb.social/v2/` | HTTP `401` | OCI registry route responds with an auth challenge. |
| `https://gitea.coulomb.social/api/packages/coulomb/pypi/simple/` | HTTP `404` | Public package route reachable but package/simple root did not return an index. |
| `https://hub.coulomb.social/api/v2/hubs` | HTTP `404` | Confirms inter-hub production still lacks the expected API surface. |

Direct haskelseed alias probe:

```text
ssh -o BatchMode=yes -o ConnectTimeout=5 haskelseed hostname
```

Result:

```text
ssh: connect to host haskelseed port 22: Connection timed out
```

This does not prove the runner host is down; it proves this session does not
currently have the bare SSH alias path needed to inspect it.

Ops-bridge haskelseed path:

```bash
RUNNER_HOST=192.168.178.135 \
RUNNER_SSH_USER=root \
RUNNER_SSH_KEY=/home/worsch/.ssh/id_ops \
make runner-status
```

Observed on 2026-06-07:

| Field | Result |
| --- | --- |
| Hostname | `haskelseed.coulomb.social` |
| Kernel | `Linux 6.18.22-0-virt` on Alpine |
| `act_runner` | `/usr/local/bin/act_runner` |
| `act_runner --version` | `v0.6.1-1-g8e6b3be9` |
| `nix` | `/usr/local/bin/nix`, Determinate Nix `3.18.1`, Nix `2.33.4` |
| Init system | OpenRC (`/sbin/rc-service`) |
| `act_runner` OpenRC service | initially not present; installed and started on 2026-06-07 |
| `gitea-act-runner` OpenRC service | not present |
| live runner process | PID `5911` after activation |
| registration file | `/root/.runner`, mode `0644`, owner `root:root` |
| registration name | `haskelseed` |
| historical registration address | `http://92.205.130.254:32166` before the public NodePort was retired under `FORGE-WP-0005` |
| registration labels before activation | `haskelseed:host`, `linux:host`, `x86_64:host` |
| registration labels after activation | `self-hosted:host`, `haskelseed:host`, `linux:host`, `linux_amd64:host`, `x86_64:host`, `container-build:host`, `registry-publish:host` |
| ephemeral | `false` |
| runner backup | `/root/.runner.bak-20260607225905` |
| org registration | re-registered under the `coulomb` organization on 2026-06-08 using an attended token from the Gitea UI; token was not recorded |
| smoke workflow run | `forge-runner-smoke.yaml #4`, run `/coulomb/railiance-forge/actions/runs/4`, commit `de6178764c` |
| smoke workflow status | `Success` as of 2026-06-08 |

Activation evidence:

```text
rc-service act_runner restart
status: started
act_runner PID: 5911
runner declared successfully with labels:
self-hosted, haskelseed, linux, linux_amd64, x86_64, container-build, registry-publish
```

## Runner Inventory

Known from repo and State Hub:

- Before `FORGE-WP-0003`, this repo had runner ownership contracts but no
  runner deployment files, status script, smoke workflow, or runner evidence
  file.
- Inter-hub reported that commits intended to trigger deployment did not update
  production and that its workflow targets `self-hosted` and `haskelseed`.
- A local registration file exists on haskelseed and `act_runner` is running as
  an OpenRC service. Gitea runner admin access has not yet been used to confirm
  the runner in the UI, but the daemon log reports successful declaration.

Pending attended checks:

```bash
make runner-status

RUNNER_HOST=192.168.178.135 \
RUNNER_SSH_USER=root \
RUNNER_SSH_KEY=/home/worsch/.ssh/id_ops \
make runner-status

ssh haskelseed 'hostname; command -v act_runner || true'
ssh haskelseed 'systemctl status act_runner --no-pager || systemctl status gitea-act-runner --no-pager || true'
ssh haskelseed 'journalctl -u act_runner -n 200 --no-pager || journalctl -u gitea-act-runner -n 200 --no-pager || true'
```

If Gitea runner admin access is available, verify the `coulomb` organization or
instance runner page for:

- runner name `railiance-haskelseed-build-01`;
- labels `self-hosted`, `haskelseed`, `linux`, `linux_amd64`,
  `container-build`, and `registry-publish`;
- online status;
- last contact time;
- workflow ids for the smoke run and later inter-hub run.

## Registry Tag Evidence

Tag inspection remains pending because `skopeo` is unavailable in this
environment.

Run from an operator host with registry access:

```bash
for tag in 91037a4 ae9e497 fa96fb8 7cc3173 latest; do
  skopeo inspect \
    "docker://gitea.coulomb.social/coulomb/inter-hub:${tag}" \
    --format "${tag} {{.Name}} {{.Digest}}"
done
```

Record only image names, tags, digests, and status. Do not record registry
tokens.

Haskelseed inventory on 2026-06-08:

| Tag | Result |
| --- | --- |
| `91037a4` | `manifest unknown` |
| `ae9e497` | `manifest unknown` |
| `fa96fb8` | `manifest unknown` |
| `7cc3173` | `manifest unknown` |
| `latest` | `manifest unknown` |

This means the blocked inter-hub workflow did not publish the expected image
tags to the current Gitea registry path.

## Smoke Evidence

The smoke workflow passed after haskelseed was re-registered under the `coulomb`
organization runner scope. The earlier waiting runs were caused by Gitea not
having an eligible org/repo runner record even though a local haskelseed
registration file existed.

| Field | Value |
| --- | --- |
| Date | 2026-06-08 |
| Runner name | `haskelseed` |
| Labels used | `self-hosted`, `haskelseed` |
| Workflow id/url | `/coulomb/railiance-forge/actions/runs/4` |
| Repo commit | `de6178764c` |
| Docker availability | smoke step completes; public run status is `Success` |
| Cluster credential absent | smoke workflow asserts `KUBECONFIG` is unset |
| Registry credential absent in smoke | smoke workflow asserts `REGISTRY_PASSWORD` and `GITEA_RUNNER_REGISTRATION_TOKEN` are unset |

## Inter-Hub Unblock Slot

The runner substrate is no longer the primary blocker: the forge smoke workflow
passes and Gitea can schedule jobs on haskelseed. The first inter-hub job picked
up after org registration failed, according to the authenticated UI observation
reported by the operator on 2026-06-08.

Forge reproduced the non-secret build phase on haskelseed from a committed-only
archive of inter-hub `84ee797`. The build fails before publish/deploy:

```text
Web/Controller/Api/V2/Annotations.hs:20:23: error: GHC-87543
Ambiguous occurrence `createAnnotation`.
It could refer to Generated.Types.createAnnotation
or Web.Controller.Api.V2.Annotations.createAnnotation.

Web/Controller/Api/V2/Annotations.hs:29:42: error: GHC-87543
Ambiguous occurrence `createAnnotation`.
```

Registry inspection from haskelseed still reports `manifest unknown` for
`84ee797`, `7cc3173`, `fa96fb8`, and `latest`, matching a failure before or
during publish. Since the reproduced failure happens in `nix build .#docker`,
the next fix belongs in the inter-hub source repo.

Expected evidence after inter-hub rerun:

| Field | Value |
| --- | --- |
| Inter-hub commit | `TODO` |
| Workflow id/url | `TODO` |
| Image tag/digest | `TODO` |
| Deployment result | failed before image publish/deploy |
| `https://hub.coulomb.social/api/v2/hubs` | still old surface; no new image tag deployed |
| Remaining blocker | fix inter-hub GHC ambiguity in `Web/Controller/Api/V2/Annotations.hs` |