Compare commits
19 Commits
be81d49a7b
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 3e992d2551 | |||
| c664aebd1d | |||
| 67beab1525 | |||
| e7eae7f7f5 | |||
| 092315895f | |||
| 2d62317ada | |||
| 67b259f6dc | |||
| 6b0ededee2 | |||
| b32c56db4f | |||
| 3f2564748c | |||
| 73a30f0a44 | |||
| 723f23d05d | |||
| 48f40b8e24 | |||
| e065e9ced4 | |||
| 1720a56294 | |||
| 8f6c332685 | |||
| 6d3968699f | |||
| 1cae8ea14e | |||
| c31d70b53f |
50
.claude/rules/credential-routing.md
Normal file
50
.claude/rules/credential-routing.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=railiance-infra` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workplans; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
@@ -1,37 +1,41 @@
|
||||
## First Session Protocol
|
||||
|
||||
Triggered when `get_domain_summary("railiance")` shows **no workstreams**.
|
||||
Triggered when `get_domain_summary("financials")` shows **no workplans**.
|
||||
The project is registered but work has not yet been structured.
|
||||
|
||||
**Step 1 — Read, don't write**
|
||||
- `~/the-custodian/canon/projects/railiance/project_charter_v0.1.md` — purpose, scope
|
||||
- `~/the-custodian/canon/projects/railiance/roadmap_v0.1.md` — planned phases
|
||||
- `~/the-custodian/canon/projects/financials/project_charter_v0.1.md` — purpose, scope
|
||||
- `~/the-custodian/canon/projects/financials/roadmap_v0.1.md` — planned phases
|
||||
- Scan repo root: README, directory structure, existing code or docs
|
||||
|
||||
**Step 2 — Survey in-progress work**
|
||||
Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete.
|
||||
|
||||
**Step 3 — Propose workstreams to Bernd**
|
||||
Propose 1–3 workstreams — each a coherent strand, weeks to months, anchored to a
|
||||
**Step 3 — Propose workplans to Bernd**
|
||||
Propose 1–3 workplans — each a coherent strand, weeks to months, anchored to a
|
||||
roadmap phase. **Wait for approval before creating.**
|
||||
|
||||
**Step 4 — Create workplan file first, then DB record (ADR-001)**
|
||||
**Step 4 — Write the workplan file; fix-consistency registers it (ADR-001)**
|
||||
```
|
||||
workplans/railiance-hosts-WP-NNNN-<slug>.md ← write this first
|
||||
workplans/RAIL-HO-WP-NNNN-<slug>.md ← write this, commit it
|
||||
```
|
||||
Then register in the hub:
|
||||
```
|
||||
create_workstream(topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", title="...", owner="...", description="...")
|
||||
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
|
||||
Then register by running the consistency check — do **not** call
|
||||
`create_workplan`/`create_task` (or legacy `create_workstream`) yourself;
|
||||
manual registration duplicates what C-06 creates from the file:
|
||||
```bash
|
||||
statehub fix-consistency --repo railiance-infra
|
||||
```
|
||||
C-06 creates the hub workplan + tasks and writes `state_hub_workstream_id` /
|
||||
`state_hub_task_id` back into the file (legacy field names, kept for
|
||||
compatibility — they hold workplan/task IDs).
|
||||
|
||||
**Step 5 — Record the setup**
|
||||
```
|
||||
add_progress_event(
|
||||
summary="First session: structured railiance into N workstreams, M tasks",
|
||||
summary="First session: structured financials into N workplans, M tasks",
|
||||
event_type="milestone",
|
||||
topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38",
|
||||
detail={"workstreams": [...], "tasks_created": M}
|
||||
detail={"workplans": [...], "tasks_created": M}
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
## Repo boundary
|
||||
|
||||
This repo owns **railiance-hosts** only. It does not own:
|
||||
This repo owns **railiance-infra** only. It does not own:
|
||||
|
||||
<!-- TODO: List what belongs in adjacent repos, e.g.:
|
||||
- SSH key management → railiance-infra/
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
**Purpose:** Host inventory and node-level configuration for railiance infrastructure. Tracks server specs, network topology, and per-host service assignments.
|
||||
**Purpose:** OAS S1 Infrastructure Substrate — OS provisioning, Ansible roles, Goss spec, Terraform, inventory
|
||||
|
||||
**Domain:** railiance
|
||||
**Repo slug:** railiance-hosts
|
||||
**Domain:** financials
|
||||
**Repo slug:** railiance-infra
|
||||
**Topic ID:** ca369340-a64e-442e-98f1-a4fa7dc74a38
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
## Session Protocol
|
||||
|
||||
State Hub: http://127.0.0.1:8000
|
||||
Dev Hub (State Hub API): http://127.0.0.1:8000
|
||||
MCP server name in `~/.claude.json`: `dev-hub`
|
||||
|
||||
**Step 1 — Orient**
|
||||
|
||||
@@ -10,7 +11,7 @@ cat .custodian-brief.md
|
||||
```
|
||||
Then call the MCP tool for richer cross-domain context when MCP tools are exposed:
|
||||
```
|
||||
get_domain_summary("railiance")
|
||||
get_domain_summary("financials")
|
||||
```
|
||||
If MCP tools are unavailable in the current agent session, use the REST API:
|
||||
```bash
|
||||
@@ -21,14 +22,14 @@ If the hub is offline: `cd ~/state-hub && make api`
|
||||
**Step 2 — Check inbox**
|
||||
With MCP tools:
|
||||
```
|
||||
get_messages(to_agent="railiance-hosts", unread_only=True)
|
||||
get_messages(to_agent="railiance-infra", unread_only=True)
|
||||
```
|
||||
Mark read with `mark_message_read(message_id)`. Reply or act on coordination
|
||||
requests before proceeding.
|
||||
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=railiance-hosts&unread_only=true" \
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=railiance-infra&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
@@ -39,45 +40,54 @@ curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
ls workplans/
|
||||
```
|
||||
For each file with `status: ready`, `active`, or `blocked`, note pending
|
||||
`todo`/`in_progress` tasks.
|
||||
`wait`/`todo`/`progress` tasks.
|
||||
|
||||
**Step 4 — Present brief**
|
||||
|
||||
1. **Active workstreams** for `railiance` — title, task counts, blocking decisions
|
||||
2. **Pending tasks** from `workplans/` + any `[repo:railiance-hosts]` hub tasks
|
||||
1. **Active workplans** for `financials` — title, task counts, blocking decisions
|
||||
2. **Pending tasks** from `workplans/` + any `[repo:railiance-infra]` hub tasks
|
||||
3. **Goal guidance** — if `goal_guidance` in summary:
|
||||
- `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"*
|
||||
- `alignment_warnings`: flag if active work is not aligned with current goal
|
||||
4. **Suggested next action** — highest-priority open item
|
||||
5. **SBOM status** — flag if `last_sbom_at` is unset for this repo
|
||||
|
||||
If no workstreams: follow First Session Protocol (`first-session.md`).
|
||||
If no workplans: follow First Session Protocol (`first-session.md`).
|
||||
|
||||
**During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()`
|
||||
|
||||
> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`)
|
||||
> are First Session Protocol only. Work structure belongs in repo files (ADR-001).
|
||||
> State Hub is a *read model*. **Never register workplans or tasks by hand**
|
||||
> (`create_workplan`, `create_task`, or the legacy `create_workstream`) — write
|
||||
> the workplan file in `workplans/` and run `fix-consistency`; its C-06 check
|
||||
> registers the workplan and its tasks in the hub and writes the IDs back into
|
||||
> the file. Manual registration creates duplicates the moment fix-consistency
|
||||
> runs. Work structure belongs in repo files (ADR-001).
|
||||
>
|
||||
> Terminology: "workstream" is the legacy name for workplan. Some API/frontmatter
|
||||
> field names keep it for compatibility (`state_hub_workstream_id`,
|
||||
> `workstream_id` params) — treat them as workplan IDs.
|
||||
|
||||
**Session close:**
|
||||
With MCP tools:
|
||||
```
|
||||
add_progress_event(summary="...", topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", workstream_id="<uuid>")
|
||||
add_progress_event(summary="...", topic_id="ca369340-a64e-442e-98f1-a4fa7dc74a38", workplan_id="<uuid>")
|
||||
```
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"topic_id":"ca369340-a64e-442e-98f1-a4fa7dc74a38","workstream_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
|
||||
-d '{"topic_id":"ca369340-a64e-442e-98f1-a4fa7dc74a38","workplan_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
|
||||
```
|
||||
If workplan files were modified, ensure the local copy is up to date first:
|
||||
If workplan files were modified, ensure the local copy is up to date first,
|
||||
then sync from the repo checkout:
|
||||
```bash
|
||||
git -C <repo_path> pull --ff-only
|
||||
cd ~/state-hub && make fix-consistency REPO=railiance-hosts
|
||||
git pull --ff-only
|
||||
statehub fix-consistency
|
||||
```
|
||||
For repos where implementation runs on a remote machine (e.g. CoulombCore),
|
||||
use the combined target which pulls before fixing:
|
||||
use the pull-before-fix mode from any shell with the State Hub CLI:
|
||||
```bash
|
||||
cd ~/state-hub && make fix-consistency-remote REPO=railiance-hosts
|
||||
statehub fix-consistency --repo railiance-infra --remote
|
||||
```
|
||||
**C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback
|
||||
will sync the file to match DB. **C-16** (repo behind remote) blocks all writes
|
||||
|
||||
@@ -1,19 +1,22 @@
|
||||
## Stack
|
||||
|
||||
<!-- TODO: Fill in language, frameworks, and key dependencies -->
|
||||
- **Language:**
|
||||
- **Key deps:**
|
||||
- **Language:** Terraform (HCL), cloud-init, Ansible, Bash
|
||||
- **Key deps:** Hetzner/HostEurope provisioning, SOPS + age for secrets, repo-local git hooks (`.githooks`)
|
||||
|
||||
## Dev Commands
|
||||
|
||||
```bash
|
||||
# TODO: Fill in the standard commands for this repo
|
||||
|
||||
# Install dependencies
|
||||
|
||||
# Run tests
|
||||
|
||||
# Lint / type check
|
||||
|
||||
# Build / package (if applicable)
|
||||
make help # list all targets
|
||||
make hooks # configure repo-local git hooks (secrets guard)
|
||||
make check-secrets # fail if anything in secrets/ is not SOPS-encrypted
|
||||
make sops-edit # edit global secrets with SOPS
|
||||
make sops-encrypt FILE=secrets/foo.yaml
|
||||
make tf-fmt # terraform fmt
|
||||
make tf-init && make tf-plan # plan (needs decrypted HCLOUD_TOKEN)
|
||||
make tf-apply # provision — production-touching
|
||||
make tf-destroy # tear down — destructive, operator only
|
||||
```
|
||||
|
||||
`tf-apply`/`tf-destroy` change production infrastructure — operator approval
|
||||
required. No unit test suite; validation is `tf-plan` + the server test suite
|
||||
workplans.
|
||||
|
||||
@@ -1,28 +1,45 @@
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
File location: `workplans/railiance-hosts-WP-NNNN-<slug>.md`
|
||||
ID prefix: `RAILIANCE-WP`
|
||||
File location: `workplans/RAIL-HO-WP-NNNN-<slug>.md`
|
||||
ID prefix: `RAIL-HO-WP-`
|
||||
|
||||
Work items originate as files in this repo **before** being registered in the hub.
|
||||
|
||||
Canonical workplan/workstream frontmatter statuses are:
|
||||
Canonical workplan frontmatter statuses are:
|
||||
`proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`.
|
||||
Use `proposed` for a newly drafted plan, `ready` after review against current
|
||||
repo state, and `finished` when implementation is complete. `stalled` and
|
||||
`needs_review` are derived health labels, not stored statuses.
|
||||
|
||||
Closed workplans may be moved to `workplans/archived/` with a completion-date
|
||||
prefix: `YYMMDD-railiance-hosts-WP-NNNN-<slug>.md`. The frontmatter id remains
|
||||
prefix: `YYMMDD-RAIL-HO-WP-NNNN-<slug>.md`. The frontmatter id remains
|
||||
unchanged; the prefix is only for quick visual reference.
|
||||
|
||||
Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**:
|
||||
`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids
|
||||
`workplans/ADHOC-YYYY-MM-DD.md`, workplan slug `adhoc-YYYY-MM-DD`, and task ids
|
||||
`ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed
|
||||
directly. Promote anything requiring analysis, design, approval, dependencies, or
|
||||
multiple planned phases into a normal workplan.
|
||||
|
||||
Ecosystem todos from other agents arrive as `[repo:railiance-hosts]` hub tasks —
|
||||
visible at session start. Pick one up by creating the workplan file, then registering
|
||||
the workstream.
|
||||
Ecosystem todos from other agents arrive as `[repo:railiance-infra]` hub tasks —
|
||||
visible at session start. Pick one up by creating the workplan file, committing,
|
||||
and running `statehub fix-consistency` — C-06 registers the workplan in the hub.
|
||||
Never register by hand with `create_workplan`/`create_workstream`.
|
||||
|
||||
Task blocks use this shape:
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
```
|
||||
|
||||
Status progression is `todo` → `progress` → `done`; use `wait` for waiting or
|
||||
blocked work and `cancel` for stopped work.
|
||||
|
||||
Workplan frontmatter carries `state_hub_workstream_id` — a legacy field name
|
||||
kept for compatibility ("workstream" is the old term for workplan); it holds
|
||||
the hub workplan id and is written by fix-consistency. Do not edit or rename it.
|
||||
|
||||
<!-- Ralph Loop rules and HEUREKA sequence: ~/.claude/CLAUDE.md — do not duplicate here -->
|
||||
|
||||
@@ -1,40 +1,29 @@
|
||||
<!-- custodian-brief: generated by fix-consistency — do not edit manually -->
|
||||
# Custodian Brief — railiance-hosts
|
||||
# Custodian Brief — railiance-infra
|
||||
|
||||
**Domain:** railiance
|
||||
**Last synced:** 2026-05-17 12:50 UTC
|
||||
**Domain:** financials
|
||||
**Last synced:** 2026-07-04 09:27 UTC
|
||||
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
|
||||
|
||||
## Current Goal
|
||||
|
||||
Secure Single-Server Bootstrap at HostEurope
|
||||
|
||||
## Active Workstreams
|
||||
|
||||
### Forgejo Production Migration on railiance01
|
||||
Progress: 0/12 done | workstream_id: `84e17675-0d15-4268-a8bd-540124d37018`
|
||||
Progress: 1/12 done | workstream_id: `84e17675-0d15-4268-a8bd-540124d37018`
|
||||
|
||||
**Open tasks:**
|
||||
- · T01 — Inventory current Gitea functionality and migration requirements `cf59d171`
|
||||
- · T02 — Resolve Forgejo production design decisions `f88115bf`
|
||||
- · T03 — Build forgejo-railiance-probe `b516018a`
|
||||
- ► T01 — Inventory current Gitea functionality and migration requirements `cf59d171`
|
||||
- ► T02 — Resolve Forgejo production design decisions `f88115bf`
|
||||
- ► T05 — Define production Forgejo application deployment `11540ba4`
|
||||
- ► T08 — Enable Forgejo Actions `f45f98c9`
|
||||
- ► T09 — Implement Forgejo backup and restore automation `25892007`
|
||||
- ► T10 — Drill Gitea to Forgejo migration `6befde73`
|
||||
- · T04 — Define Forgejo platform services `28b351fe`
|
||||
- · T05 — Define production Forgejo application deployment `11540ba4`
|
||||
- · T06 — Implement usable email recovery cycle `417faa4d`
|
||||
- · T07 — Enable and harden package registry base `9578f672`
|
||||
- … and 5 more open tasks
|
||||
|
||||
### Railiance Production Readiness — Automated, Reproducible Stack
|
||||
Progress: 8/10 done | workstream_id: `cee078e9-b18c-4f84-8a8a-6f27c2f9f407`
|
||||
|
||||
**Open tasks:**
|
||||
- · T09 — Deploy state-hub to cluster (S5) `d2afe78a`
|
||||
- · T10 — Deploy activity-core to cluster (S5) `34d73215`
|
||||
- … and 4 more open tasks
|
||||
|
||||
---
|
||||
## MCP Orientation (when available)
|
||||
|
||||
If the state-hub MCP server is reachable, call:
|
||||
`get_domain_summary("railiance")`
|
||||
`get_domain_summary("financials")`
|
||||
This provides richer cross-domain context.
|
||||
If the MCP call fails, use this file as your orientation source.
|
||||
|
||||
29
.forgejo/workflows/ci-smoke.yaml
Normal file
29
.forgejo/workflows/ci-smoke.yaml
Normal file
@@ -0,0 +1,29 @@
|
||||
# Canonical CI smoke template (tier 1 routing drill).
|
||||
# Copy to: .forgejo/workflows/ci-smoke.yaml in consumer repos.
|
||||
name: CI Smoke
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
host-smoke:
|
||||
runs-on: self-hosted
|
||||
steps:
|
||||
- name: Routing probe (host runner)
|
||||
run: |
|
||||
set -eu
|
||||
echo "repository=${GITHUB_REPOSITORY:-unknown}"
|
||||
echo "sha=${GITHUB_SHA:-unknown}"
|
||||
echo "runner=${RUNNER_NAME:-unknown}"
|
||||
uname -a
|
||||
|
||||
container-smoke:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Routing probe (container label)
|
||||
run: |
|
||||
set -eu
|
||||
echo "container-smoke ok for ${GITHUB_REPOSITORY:-unknown}"
|
||||
17
.repo-classification.yaml
Normal file
17
.repo-classification.yaml
Normal file
@@ -0,0 +1,17 @@
|
||||
repo_classification:
|
||||
standard: Repo Classification Standard
|
||||
version: '1.0'
|
||||
classified_at: '2026-06-22'
|
||||
classified_by: agent
|
||||
category: project
|
||||
domain: financials
|
||||
secondary_domains: []
|
||||
capability_tags:
|
||||
- platform
|
||||
- operations
|
||||
business_stake:
|
||||
- technology
|
||||
- operations
|
||||
business_mechanics:
|
||||
- coordination
|
||||
- operation
|
||||
111
AGENTS.md
111
AGENTS.md
@@ -1,13 +1,13 @@
|
||||
# railiance-hosts — Agent Instructions
|
||||
# railiance-infra — Agent Instructions
|
||||
|
||||
## Repo Identity
|
||||
|
||||
**Purpose:** Host inventory and node-level configuration for railiance infrastructure. Tracks server specs, network topology, and per-host service assignments.
|
||||
**Purpose:** OAS S1 Infrastructure Substrate — OS provisioning, Ansible roles, Goss spec, Terraform, inventory
|
||||
|
||||
**Domain:** railiance
|
||||
**Repo slug:** railiance-hosts
|
||||
**Domain:** financials
|
||||
**Repo slug:** railiance-infra
|
||||
**Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38`
|
||||
**Workplan prefix:** `RAILIANCE-WP-`
|
||||
**Workplan prefix:** `RAIL-HO-WP-`
|
||||
|
||||
---
|
||||
|
||||
@@ -20,6 +20,12 @@ there is no MCP server for Codex agents.
|
||||
|---------|-----|
|
||||
| Local workstation | `http://127.0.0.1:8000` |
|
||||
| Remote via tunnel | `http://127.0.0.1:18000` |
|
||||
| Optional local edge relay | http://127.0.0.1:18080 |
|
||||
|
||||
When an operator has enabled the edge relay, set API_BASE to the relay URL.
|
||||
Queueable writes return an explicit queued receipt if the central hub is
|
||||
unreachable. Treat that as pending local evidence, then ask the operator to run
|
||||
statehub outbox status/replay after connectivity returns.
|
||||
|
||||
### Orient at session start
|
||||
|
||||
@@ -27,12 +33,12 @@ there is no MCP server for Codex agents.
|
||||
# Offline brief — works without hub connection
|
||||
cat .custodian-brief.md
|
||||
|
||||
# Active workstreams for this domain
|
||||
curl -s "http://127.0.0.1:8000/workstreams/?topic_id=ca369340-a64e-442e-98f1-a4fa7dc74a38&status=active" \
|
||||
# Active workplans for this domain
|
||||
curl -s "http://127.0.0.1:8000/workplans/?topic_id=ca369340-a64e-442e-98f1-a4fa7dc74a38&status=active" \
|
||||
| python3 -m json.tool
|
||||
|
||||
# Check inbox
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=railiance-hosts&unread_only=true" \
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=railiance-infra&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
@@ -51,20 +57,20 @@ curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
"summary": "what was done",
|
||||
"event_type": "note",
|
||||
"author": "codex",
|
||||
"workstream_id": "<uuid>",
|
||||
"workplan_id": "<uuid>",
|
||||
"task_id": "<uuid>"
|
||||
}'
|
||||
```
|
||||
|
||||
Omit `workstream_id` / `task_id` when not applicable.
|
||||
Omit `workplan_id` / `task_id` when not applicable.
|
||||
|
||||
### Update task status
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"status": "in_progress"}'
|
||||
# values: todo | in_progress | done | blocked
|
||||
-d '{"status": "progress"}'
|
||||
# values: wait | todo | progress | done | cancel
|
||||
```
|
||||
|
||||
### Flag a task for human review
|
||||
@@ -80,10 +86,10 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
## Session Protocol
|
||||
|
||||
**Start:**
|
||||
1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe)
|
||||
2. Check inbox: `GET /messages/?to_agent=railiance-hosts&unread_only=true`; mark read
|
||||
1. `cat .custodian-brief.md` — domain goal and open workplans (offline-safe)
|
||||
2. Check inbox: `GET /messages/?to_agent=railiance-infra&unread_only=true`; mark read
|
||||
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
|
||||
4. Check blocked tasks: `GET /tasks/?needs_human=true`
|
||||
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
|
||||
|
||||
**During work:**
|
||||
- Update task statuses in workplan files as tasks progress
|
||||
@@ -92,12 +98,69 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
**Close:**
|
||||
1. Update workplan file task statuses to reflect progress
|
||||
2. Log: `POST /progress/` with a summary of what changed
|
||||
3. Note for the custodian operator: after workplan file changes, run from
|
||||
`~/state-hub`:
|
||||
3. After workplan file changes, run:
|
||||
```bash
|
||||
make fix-consistency REPO=railiance-hosts
|
||||
statehub fix-consistency
|
||||
```
|
||||
This syncs task status from files into the hub DB.
|
||||
Coding agents should run this directly; ask the operator only if the CLI or
|
||||
State Hub API is unavailable. This syncs task status from files into the hub DB.
|
||||
|
||||
---
|
||||
|
||||
## Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=railiance-infra` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workplans; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
|
||||
<!-- REPO-AGENTS-EXTENSIONS -->
|
||||
<!-- Append repo-specific agent instructions below this marker.
|
||||
The state-hub template sync preserves content after this line. -->
|
||||
|
||||
---
|
||||
|
||||
@@ -124,8 +187,8 @@ anything needing analysis, design, approval, dependencies, or multiple phases.
|
||||
id: RAILIANCE-WP-NNNN
|
||||
type: workplan
|
||||
title: "..."
|
||||
domain: railiance
|
||||
repo: railiance-hosts
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: proposed | ready | active | blocked | backlog | finished | archived
|
||||
owner: codex
|
||||
topic_slug: ...
|
||||
@@ -146,7 +209,7 @@ derived health labels, not frontmatter statuses.
|
||||
|
||||
` ` `task
|
||||
id: RAILIANCE-WP-NNNN-T01
|
||||
status: todo | in_progress | done | blocked
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
` ` `
|
||||
@@ -154,9 +217,9 @@ state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
Task description text.
|
||||
```
|
||||
|
||||
Status progression: `todo` → `in_progress` → `done` (or `blocked`)
|
||||
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
|
||||
|
||||
To create a new workplan:
|
||||
1. Write the file following the format above
|
||||
2. Notify the custodian operator to run `make fix-consistency REPO=railiance-hosts`
|
||||
2. Notify the custodian operator to run `make fix-consistency REPO=railiance-infra`
|
||||
(or send a message to the hub agent via `POST /messages/`)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# railiance-hosts — Claude Code Instructions
|
||||
# railiance-infra — Claude Code Instructions
|
||||
|
||||
@SCOPE.md
|
||||
@.claude/rules/repo-identity.md
|
||||
@@ -8,4 +8,5 @@
|
||||
@.claude/rules/stack-and-commands.md
|
||||
@.claude/rules/architecture.md
|
||||
@.claude/rules/repo-boundary.md
|
||||
@.claude/rules/credential-routing.md
|
||||
@.claude/rules/agents.md
|
||||
|
||||
11
Makefile
11
Makefile
@@ -143,6 +143,17 @@ provision-custodian-agent-host: ## Deploy custodian agent key to one host: make
|
||||
cd ansible && ansible-playbook playbooks/custodian-agent.yaml -u $(SSH_USER) \
|
||||
--limit "$(HOST)"
|
||||
|
||||
bootstrap-ssh-ca: ## Deploy OpenBao SSH CA trust + auth_principals: make bootstrap-ssh-ca SSH_CA_PUBKEY=/path/to/ca_user.pub
|
||||
@test -n "$(SSH_CA_PUBKEY)" || (echo "Usage: make bootstrap-ssh-ca SSH_CA_PUBKEY=/path/to/ca_user.pub [HOST=Railiance01]"; exit 1)
|
||||
cd ansible && ansible-playbook playbooks/bootstrap-ssh-ca.yaml -u $(SSH_USER) \
|
||||
-e ssh_ca_pubkey_path="$(SSH_CA_PUBKEY)" \
|
||||
$(if $(HOST),--limit "$(HOST)",)
|
||||
|
||||
bootstrap-ssh-ca-host: ## Deploy SSH CA trust to one host: make bootstrap-ssh-ca-host HOST=Railiance01 SSH_CA_PUBKEY=...
|
||||
@test -n "$(HOST)" && test -n "$(SSH_CA_PUBKEY)" || \
|
||||
(echo "Usage: make bootstrap-ssh-ca-host HOST=Railiance01 SSH_CA_PUBKEY=/path/to/ca_user.pub"; exit 1)
|
||||
$(MAKE) bootstrap-ssh-ca SSH_CA_PUBKEY="$(SSH_CA_PUBKEY)" HOST="$(HOST)"
|
||||
|
||||
# ---- Orchestration ----
|
||||
apply: tf-fmt tf-apply ansible-bootstrap ## Provision via Terraform then converge via Ansible
|
||||
|
||||
|
||||
@@ -4,5 +4,6 @@ roles_path = ./roles
|
||||
host_key_checking = False
|
||||
retry_files_enabled = False
|
||||
interpreter_python = auto
|
||||
stdout_callback = yaml
|
||||
stdout_callback = default
|
||||
result_format = yaml
|
||||
forks = 20
|
||||
|
||||
19
ansible/inventory/ssh_principals.yaml
Normal file
19
ansible/inventory/ssh_principals.yaml
Normal file
@@ -0,0 +1,19 @@
|
||||
# Central SSH certificate principals inventory — synced with ops-warden actor
|
||||
# principals (wiki/ActorInventoryPatterns.md). Public data only.
|
||||
|
||||
ssh_principals:
|
||||
CoulombCore:
|
||||
users:
|
||||
tegwick:
|
||||
- agt-task-bridge
|
||||
- agt-interhub-bootstrap
|
||||
- adm-full
|
||||
- atm-backup-daily
|
||||
|
||||
Railiance01:
|
||||
users:
|
||||
tegwick:
|
||||
- agt-task-bridge
|
||||
- agt-interhub-bootstrap
|
||||
- adm-full
|
||||
- atm-backup-daily
|
||||
19
ansible/playbooks/bootstrap-ssh-ca.yaml
Normal file
19
ansible/playbooks/bootstrap-ssh-ca.yaml
Normal file
@@ -0,0 +1,19 @@
|
||||
---
|
||||
# Deploy OpenBao SSH user CA trust and per-user auth_principals.
|
||||
#
|
||||
# Prerequisite: railiance-platform openbao-configure-ssh (exports CA pubkey).
|
||||
#
|
||||
# cd ~/railiance-platform
|
||||
# OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token \
|
||||
# OPENBAO_SSH_CA_PUBKEY_OUT=/tmp/openbao-ssh-ca.pub \
|
||||
# make openbao-configure-ssh
|
||||
#
|
||||
# cd ~/railiance-infra
|
||||
# make bootstrap-ssh-ca SSH_CA_PUBKEY=/tmp/openbao-ssh-ca.pub
|
||||
|
||||
- hosts: all
|
||||
become: true
|
||||
vars_files:
|
||||
- ../inventory/ssh_principals.yaml
|
||||
roles:
|
||||
- role: ssh_ca_host
|
||||
5
ansible/roles/ssh_ca_host/handlers/main.yml
Normal file
5
ansible/roles/ssh_ca_host/handlers/main.yml
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
- name: Restart sshd
|
||||
ansible.builtin.service:
|
||||
name: ssh
|
||||
state: restarted
|
||||
85
ansible/roles/ssh_ca_host/tasks/main.yml
Normal file
85
ansible/roles/ssh_ca_host/tasks/main.yml
Normal file
@@ -0,0 +1,85 @@
|
||||
---
|
||||
- name: Require SSH CA public key path
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ssh_ca_pubkey_path is defined
|
||||
- ssh_ca_pubkey_path | length > 0
|
||||
fail_msg: >-
|
||||
Set ssh_ca_pubkey_path to the OpenBao SSH CA public key file
|
||||
(from railiance-platform openbao-configure-ssh).
|
||||
|
||||
- name: Stat SSH CA public key source
|
||||
ansible.builtin.stat:
|
||||
path: "{{ ssh_ca_pubkey_path }}"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
register: ssh_ca_pubkey_stat
|
||||
|
||||
- name: Fail when SSH CA public key is missing
|
||||
ansible.builtin.fail:
|
||||
msg: "SSH CA public key not found on controller: {{ ssh_ca_pubkey_path }}"
|
||||
when: not ssh_ca_pubkey_stat.stat.exists
|
||||
|
||||
- name: Ensure SSH CA directory exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/ssh/ca
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
- name: Install SSH user CA public key
|
||||
ansible.builtin.copy:
|
||||
src: "{{ ssh_ca_pubkey_path }}"
|
||||
dest: /etc/ssh/ca/ca_user.pub
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
notify: Restart sshd
|
||||
|
||||
- name: Configure SSH certificate trust
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/ssh/sshd_config.d/20-ssh-ca.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
TrustedUserCAKeys /etc/ssh/ca/ca_user.pub
|
||||
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
|
||||
notify: Restart sshd
|
||||
|
||||
- name: Ensure auth_principals directory exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/ssh/auth_principals
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
|
||||
- name: Resolve principals for this host
|
||||
ansible.builtin.set_fact:
|
||||
ssh_ca_host_principals: >-
|
||||
{{
|
||||
(ssh_principals[inventory_hostname].users
|
||||
if ssh_principals is defined
|
||||
and inventory_hostname in ssh_principals
|
||||
else {})
|
||||
}}
|
||||
|
||||
- name: Deploy auth_principals files per user
|
||||
ansible.builtin.copy:
|
||||
dest: "/etc/ssh/auth_principals/{{ item.key }}"
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: "{{ item.value | join('\n') }}\n"
|
||||
loop: "{{ ssh_ca_host_principals | dict2items }}"
|
||||
when: ssh_ca_host_principals | length > 0
|
||||
notify: Restart sshd
|
||||
|
||||
- name: Warn when no principals configured for host
|
||||
ansible.builtin.debug:
|
||||
msg: >-
|
||||
No principals in ssh_principals.yaml for {{ inventory_hostname }} —
|
||||
CA trust installed; add users under hosts.<name>.users to enable cert login.
|
||||
when: ssh_ca_host_principals | length == 0
|
||||
104
docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md
Normal file
104
docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# ADR-004 — Forgejo In-Cluster Actions Runner on railiance01
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-03
|
||||
**Deciders:** Bernd Worsch (operator), custodian agents
|
||||
**Workplans:** `RAIL-HO-WP-0005-T02`, `CUST-WP-0054-T04`
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Forgejo production runs on **railiance01 k3s** (`railiance-apps`, S5). An interim
|
||||
**host runner** on coulombcore proved Actions scheduling (`coulomb/forgejo-actions-probe`)
|
||||
but:
|
||||
|
||||
- coulombcore is a legacy machine slated for drain (CUST-WP-0054-T03).
|
||||
- Host runners require Docker or Podman on the OS — not installed, not desired on
|
||||
coulombcore long term.
|
||||
- Forgejo upstream recommends **not** co-locating runners on the same machine as the
|
||||
forge instance; in-cluster **separate pods** satisfy isolation while staying on the
|
||||
production fleet node.
|
||||
- `RAIL-HO-WP-0005-T02` left the runner model undecided among host, in-cluster, and
|
||||
ephemeral options.
|
||||
|
||||
Goal: a **coherent Kubernetes-from-the-start** CI substrate — Forgejo app, database,
|
||||
ingress, and Actions runner all lifecycle-managed on railiance01.
|
||||
|
||||
## Decision
|
||||
|
||||
### Runner placement
|
||||
|
||||
Deploy **one long-lived Forgejo Actions runner Deployment** in the `forgejo` namespace
|
||||
on railiance01:
|
||||
|
||||
| Component | Implementation |
|
||||
| --- | --- |
|
||||
| Runner | `data.forgejo.org/forgejo/runner:6.3.1` |
|
||||
| Container runtime for jobs | `docker:dind` sidecar (privileged) |
|
||||
| State | PVC `forgejo-runner-data` (`.runner`, `config.yaml`, action cache) |
|
||||
| Registration scope | `coulomb` organization |
|
||||
| Runner name | `railiance01-build-01` |
|
||||
| Deploy surface | `railiance-apps/manifests/forgejo-runner.yaml` |
|
||||
| Operator targets | `make forgejo-runner-deploy`, `forgejo-runner-status` |
|
||||
|
||||
### Label contract
|
||||
|
||||
Preserve Gitea migration compatibility and semantic capability labels:
|
||||
|
||||
```text
|
||||
self-hosted:host,linux:host,linux_amd64:host,container-build:host,registry-publish:host,railiance01:host,ubuntu-latest:docker://node:20-bookworm,docker:docker://node:20-bookworm
|
||||
```
|
||||
|
||||
### Security boundaries
|
||||
|
||||
- Runner pod receives **no** cluster-admin kubeconfig and **no** OpenBao tokens by default.
|
||||
- `registry-publish` jobs use **repo/org-scoped Forgejo secrets** only.
|
||||
- DinD sidecar runs **privileged** — accepted for single-node railiance01 with
|
||||
dedicated `forgejo` namespace; revisit when a third node or multi-tenant runners appear.
|
||||
- Registration tokens live in Kubernetes Secret `forgejo-runner-registration` (SOPS
|
||||
template committed; live value never in Git).
|
||||
|
||||
### Retire interim host runner
|
||||
|
||||
Stop and disable `forgejo-runner.service` on coulombcore after in-cluster runner is
|
||||
healthy. Do not register new host runners without an explicit ADR amendment.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
| Option | Outcome |
|
||||
| --- | --- |
|
||||
| Host runner + Docker on coulombcore | Rejected — legacy host, contradicts drain plan |
|
||||
| Host runner + Podman on haskelseed | Viable fallback; not chosen as primary |
|
||||
| Kaniko/Buildah without DinD | Deferred — higher workflow churn during Gitea migration |
|
||||
| Multiple ephemeral runner Jobs | Deferred — start with capacity=1 long-lived pod |
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- Single-machine production loop: forge + runner on railiance01, workstation not required.
|
||||
- Container image CI (`docker build` / `docker push`) works without OS-level Docker.
|
||||
- Runner upgrades roll with Git-managed manifests and `kubectl`/Makefile.
|
||||
|
||||
**Negative / follow-on**
|
||||
|
||||
- Privileged DinD increases blast radius within the node — monitor and restrict namespace RBAC.
|
||||
- SOPS-encrypted registration secret still requires operator age key.
|
||||
- `cluster-deploy` / `s5-release-check` labels remain **out of scope** until credential paths reviewed.
|
||||
|
||||
## Ownership (OAS)
|
||||
|
||||
| Concern | Repo | Layer |
|
||||
| --- | --- | --- |
|
||||
| ADR + umbrella sequencing | `railiance-infra` | S1 |
|
||||
| Runner manifests + Makefile | `railiance-apps` | S5 |
|
||||
| Label contract + runner evidence docs | `railiance-forge` | S5 forge substrate |
|
||||
| Reusable workflow templates | `railiance-enablement` | S4 |
|
||||
|
||||
## References
|
||||
|
||||
- `railiance-apps/docs/forgejo-on-railiance01.md`
|
||||
- `railiance-forge/docs/forgejo-actions-runner-substrate.md`
|
||||
- `the-custodian/docs/forgejo-production-decisions.md`
|
||||
- [Forgejo runner installation](https://forgejo.org/docs/v11.0/admin/actions/runner-installation/)
|
||||
@@ -204,3 +204,33 @@ lost or left with an untracked remote.
|
||||
This first pass satisfies the public and infrastructure metadata part of T01.
|
||||
T01 should remain open until the authenticated admin inventory and missing repo
|
||||
classification are complete.
|
||||
|
||||
## Addendum (2026-07-04) — migration ladder and new repos
|
||||
|
||||
`RAIL-HO-WP-0005` now uses a **staged per-repo ladder** instead of an isolated
|
||||
probe namespace (T03 cancelled). Repos to add or re-classify on next inventory
|
||||
refresh:
|
||||
|
||||
| Repo | On Gitea (2026-06) | On Forgejo (2026-07-04) | Tier | Notes |
|
||||
| --- | --- | --- | ---: | --- |
|
||||
| `forgejo-actions-probe` | — | yes | 0 | Disposable runner/OCI probe |
|
||||
| `glas-harness` | yes (not in table above) | yes (canonical) | 1 | Git+SSH+CI pilot |
|
||||
| `key-cape` | yes | yes (canonical) | 2 | Image CI + `crictl pull` on railiance01 |
|
||||
| `railiance-enablement` | yes | yes (canonical) | 2.5 | S4 workflow templates |
|
||||
| `railiance-infra` | yes | yes (canonical) | 2.5 | Stack promotion |
|
||||
| `railiance-apps` | yes | yes (canonical) | 2.5 | Stack promotion |
|
||||
| `railiance-platform` | yes | yes (canonical) | 2.5 | Stack promotion |
|
||||
| `railiance-cluster` | yes | yes (canonical) | 2.5 | Stack promotion |
|
||||
|
||||
Evidence: `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`
|
||||
|
||||
**Tier definitions** (for per-repo `migration tier` column in a future refresh):
|
||||
|
||||
| Tier | Criteria | Examples |
|
||||
| ---: | --- | --- |
|
||||
| 0 | Disposable integration probes | `forgejo-actions-probe` |
|
||||
| 1 | Non-production; git+CI only | `glas-harness` |
|
||||
| 2 | Non-production with container image + registry pull | `key-cape` (done) |
|
||||
| 3 | Production drain wave / sweep registration | `state-hub`, `issue-core`, … |
|
||||
|
||||
Production repos stay on Gitea until tier 0–2 gates and T09 backup drill pass.
|
||||
|
||||
93
docs/forgejo-restore-drill-evidence.md
Normal file
93
docs/forgejo-restore-drill-evidence.md
Normal file
@@ -0,0 +1,93 @@
|
||||
# Forgejo Backup/Restore Drill Evidence
|
||||
|
||||
Date: 2026-07-04
|
||||
Workplan: RAIL-HO-WP-0005
|
||||
Task: RAIL-HO-WP-0005-T09
|
||||
`no_secret_material_recorded: true`
|
||||
|
||||
## Purpose
|
||||
|
||||
Prove that a production `forgejo dump` can be restored into an isolated
|
||||
namespace and serve repository metadata without touching production Forgejo or
|
||||
Gitea.
|
||||
|
||||
## Backup source
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| Method | `forgejo dump` from production pod |
|
||||
| Production pod | `forgejo-gitea-64c5b57684-ph9vt` (namespace `forgejo`) |
|
||||
| Archive path (workstation) | `/tmp/forgejo-drill/forgejo-drill-backup.zip` |
|
||||
| Archive size | 12,284,847 bytes (~11.7 MiB) |
|
||||
| Archive timestamp | 2026-07-04 11:20 +0200 |
|
||||
| Archive contents (top-level) | `repos/`, `data/`, `forgejo-db.sql`, `app.ini` |
|
||||
|
||||
Repos present in dump: `forgejo-actions-probe`, `glas-harness`, `key-cape`
|
||||
(all under `repos/coulomb/`).
|
||||
|
||||
## Restore target
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| Namespace | `forgejo-restore-drill` |
|
||||
| Database | CNPG cluster `forgejo-db-restore` (isolated, 1 instance) |
|
||||
| App data PVC | `forgejo-restore-data` (`local-path`, 10Gi) |
|
||||
| Helm release | `forgejo-restore` (`gitea-charts/gitea` 12.5.0) |
|
||||
| Orchestration | `tools/forgejo-restore-drill.sh` |
|
||||
|
||||
Restore path (Forgejo 11.0.3 has no `forgejo restore` CLI):
|
||||
|
||||
1. Unzip dump into import pod staging area.
|
||||
2. Copy `repos/` → `/data/git/gitea-repositories/`.
|
||||
3. Copy `data/` → `/data/` (packages, attachments, avatars).
|
||||
4. Import `forgejo-db.sql` via `psql` into `forgejo-db-restore`.
|
||||
5. Deploy isolated Helm release bound to restored PVC + restore DB host.
|
||||
|
||||
## Post-restore checks (2026-07-04)
|
||||
|
||||
Port-forward: `svc/forgejo-restore-gitea-http` → `127.0.0.1:13000`
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| `GET /` health | HTTP 200 |
|
||||
| `GET /api/v1/repos/coulomb/glas-harness` | `full_name=coulomb/glas-harness`, `default_branch=main` |
|
||||
| `GET /api/v1/repos/coulomb/key-cape` | `full_name=coulomb/key-cape`, `default_branch=main` |
|
||||
| `GET /api/v1/orgs/coulomb/repos` | 3 repos: `forgejo-actions-probe`, `glas-harness`, `key-cape` |
|
||||
|
||||
Script exit marker: `restore-drill-complete`
|
||||
|
||||
## RPO / RTO (drill scope)
|
||||
|
||||
| Metric | Observed / assumed |
|
||||
| --- | --- |
|
||||
| RPO (manual dump) | Point-in-time of `forgejo dump` execution; no scheduled backup yet |
|
||||
| RTO (isolated restore) | ~3–5 minutes for CNPG ready + import + Helm deploy on railiance01 |
|
||||
| Production impact | None — read-only dump from running pod; separate namespace |
|
||||
|
||||
## Gaps (not closed by this drill)
|
||||
|
||||
- **Scheduled backups:** CNPG `Backup` CRs and off-cluster target not configured
|
||||
(`kubectl cnpg` plugin absent on workstation).
|
||||
- **Encryption at rest:** dump stored locally on workstation for drill only; no
|
||||
approved backup target wired.
|
||||
- **Automation:** `forgejo dump` is manual; T04/T09 still need cron/operator
|
||||
schedule and retention policy (T02 decision).
|
||||
- **Re-run hygiene:** concurrent or repeat runs require `DRILL_CLEAN=1` to wipe
|
||||
`forgejo-restore-drill` before import (SQL import is not idempotent).
|
||||
|
||||
## Cleanup
|
||||
|
||||
After evidence capture, delete the drill namespace:
|
||||
|
||||
```bash
|
||||
kubectl delete namespace forgejo-restore-drill --wait=true
|
||||
```
|
||||
|
||||
Production Forgejo (`forgejo` namespace) and Gitea remain unchanged.
|
||||
|
||||
## References
|
||||
|
||||
- `infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml`
|
||||
- `infra/forgejo-restore-drill/restore-job.yaml`
|
||||
- `tools/forgejo-restore-drill.sh`
|
||||
- `workplans/RAIL-HO-WP-0005-forgejo-production-migration.md` (T09)
|
||||
21
infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml
Normal file
21
infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml
Normal file
@@ -0,0 +1,21 @@
|
||||
---
|
||||
apiVersion: postgresql.cnpg.io/v1
|
||||
kind: Cluster
|
||||
metadata:
|
||||
name: forgejo-db-restore
|
||||
namespace: forgejo-restore-drill
|
||||
labels:
|
||||
app.kubernetes.io/name: forgejo-db-restore
|
||||
railiance.io/layer: s3-platform
|
||||
railiance.io/consumer: forgejo-restore-drill
|
||||
spec:
|
||||
instances: 1
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:16
|
||||
storage:
|
||||
size: 10Gi
|
||||
bootstrap:
|
||||
initdb:
|
||||
database: forgejo
|
||||
owner: forgejo
|
||||
secret:
|
||||
name: forgejo-db-credentials
|
||||
12
infra/forgejo-restore-drill/restore-job.yaml
Normal file
12
infra/forgejo-restore-drill/restore-job.yaml
Normal file
@@ -0,0 +1,12 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: forgejo-restore-data
|
||||
namespace: forgejo-restore-drill
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
storageClassName: local-path
|
||||
115
tools/forgejo-restore-drill.sh
Executable file
115
tools/forgejo-restore-drill.sh
Executable file
@@ -0,0 +1,115 @@
|
||||
#!/usr/bin/env bash
|
||||
# Non-production Forgejo backup/restore drill (RAIL-HO-WP-0005-T09).
|
||||
# Re-run: DRILL_CLEAN=1 ./tools/forgejo-restore-drill.sh (wipes namespace first)
|
||||
set -euo pipefail
|
||||
|
||||
KUBECONFIG="${KUBECONFIG:-$HOME/.kube/config-hosteurope}"
|
||||
export KUBECONFIG
|
||||
NS=forgejo-restore-drill
|
||||
DRILL_CLEAN="${DRILL_CLEAN:-0}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
BACKUP_LOCAL="${BACKUP_LOCAL:-/tmp/forgejo-drill/forgejo-drill-backup.zip}"
|
||||
PROD_POD="${PROD_POD:-$(kubectl get pods -n forgejo -l app.kubernetes.io/instance=forgejo -o jsonpath='{.items[0].metadata.name}')}"
|
||||
|
||||
step() { echo "==> $*"; }
|
||||
|
||||
if [[ "${DRILL_CLEAN}" == "1" ]]; then
|
||||
step "Clean prior drill namespace ${NS}"
|
||||
kubectl delete namespace "${NS}" --wait=true --timeout=5m || true
|
||||
fi
|
||||
|
||||
step "Create namespace ${NS}"
|
||||
kubectl create namespace "${NS}" --dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
step "Copy forgejo-db-credentials into ${NS}"
|
||||
kubectl get secret forgejo-db-credentials -n databases -o json \
|
||||
| python3 -c "import json,sys; s=json.load(sys.stdin); s['metadata']={k:v for k,v in s['metadata'].items() if k in ('name','labels','annotations')}; s['metadata']['namespace']='${NS}'; print(json.dumps(s))" \
|
||||
| kubectl apply -f -
|
||||
|
||||
step "Deploy restore CNPG cluster"
|
||||
kubectl apply -f "${ROOT_DIR}/infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml"
|
||||
kubectl wait --for=condition=Ready cluster/forgejo-db-restore -n "${NS}" --timeout=10m
|
||||
|
||||
step "Ensure local backup exists"
|
||||
if [[ ! -f "${BACKUP_LOCAL}" ]]; then
|
||||
kubectl exec -n forgejo "${PROD_POD}" -c gitea -- forgejo dump -f /tmp/forgejo-drill-backup.zip
|
||||
mkdir -p "$(dirname "${BACKUP_LOCAL}")"
|
||||
kubectl cp "forgejo/${PROD_POD}:/tmp/forgejo-drill-backup.zip" "${BACKUP_LOCAL}" -c gitea
|
||||
fi
|
||||
ls -lh "${BACKUP_LOCAL}"
|
||||
|
||||
step "Apply restore PVC"
|
||||
kubectl apply -f "${ROOT_DIR}/infra/forgejo-restore-drill/restore-job.yaml"
|
||||
|
||||
step "Run restore pod (stage backup, import files + SQL)"
|
||||
kubectl delete pod forgejo-restore-import -n "${NS}" --ignore-not-found --wait=true
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: forgejo-restore-import
|
||||
namespace: ${NS}
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: restore
|
||||
image: code.forgejo.org/forgejo/forgejo:11.0.3
|
||||
command: ["sleep", "3600"]
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
- name: backup
|
||||
mountPath: /backup
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: forgejo-restore-data
|
||||
- name: backup
|
||||
emptyDir: {}
|
||||
EOF
|
||||
kubectl wait --for=condition=Ready pod/forgejo-restore-import -n "${NS}" --timeout=3m
|
||||
kubectl cp "${BACKUP_LOCAL}" "${NS}/forgejo-restore-import:/backup/forgejo-drill-backup.zip" -c restore
|
||||
DB_PASS="$(kubectl get secret forgejo-db-credentials -n "${NS}" -o jsonpath='{.data.password}' | base64 -d)"
|
||||
kubectl exec -n "${NS}" forgejo-restore-import -c restore -- env POSTGRES_PASSWORD="${DB_PASS}" sh -c '
|
||||
set -eu
|
||||
apk add --no-cache unzip postgresql-client >/dev/null
|
||||
rm -rf /data/*
|
||||
mkdir -p /data/git/gitea-repositories
|
||||
unzip -q /backup/forgejo-drill-backup.zip -d /tmp/dump
|
||||
cp -a /tmp/dump/repos/. /data/git/gitea-repositories/
|
||||
cp -a /tmp/dump/data/. /data/
|
||||
chown -R git:git /data
|
||||
PGPASSWORD="${POSTGRES_PASSWORD}" psql -h forgejo-db-restore-rw.forgejo-restore-drill.svc.cluster.local -U forgejo -d forgejo -v ON_ERROR_STOP=1 -f /tmp/dump/forgejo-db.sql
|
||||
echo restore-import-ok
|
||||
'
|
||||
unset DB_PASS
|
||||
kubectl delete pod forgejo-restore-import -n "${NS}" --wait=true
|
||||
|
||||
step "Deploy isolated Forgejo release"
|
||||
cd "${HOME}/railiance-apps"
|
||||
DB_PASS="$(kubectl get secret forgejo-db-credentials -n "${NS}" -o jsonpath='{.data.password}' | base64 -d)"
|
||||
helm upgrade --install forgejo-restore gitea-charts/gitea --version 12.5.0 \
|
||||
--namespace "${NS}" --create-namespace \
|
||||
-f helm/forgejo-values.yaml \
|
||||
-f helm/forgejo-registry-values.yaml \
|
||||
--set strategy.type=Recreate \
|
||||
--set persistence.existingClaim=forgejo-restore-data \
|
||||
--set gitea.config.database.HOST=forgejo-db-restore-rw.${NS}.svc.cluster.local:5432 \
|
||||
--set gitea.config.database.PASSWD="${DB_PASS}" \
|
||||
--set gitea.config.server.DOMAIN=forgejo-restore.local \
|
||||
--set gitea.config.server.ROOT_URL=http://forgejo-restore.local:3000/ \
|
||||
--set gitea.admin.password=restore-drill-local-only \
|
||||
--set ingress.enabled=false \
|
||||
--wait --timeout=10m
|
||||
unset DB_PASS
|
||||
|
||||
step "Post-restore checks via port-forward"
|
||||
kubectl port-forward -n "${NS}" svc/forgejo-restore-gitea-http 13000:3000 >/tmp/forgejo-restore-pf.log 2>&1 &
|
||||
PF_PID=$!
|
||||
sleep 5
|
||||
curl -fsS -o /dev/null -w 'health:%{http_code}\n' http://127.0.0.1:13000/
|
||||
curl -fsS http://127.0.0.1:13000/api/v1/repos/coulomb/glas-harness | python3 -c "import json,sys; d=json.load(sys.stdin); print('repo', d.get('full_name'), d.get('default_branch'))"
|
||||
curl -fsS http://127.0.0.1:13000/api/v1/repos/coulomb/key-cape | python3 -c "import json,sys; d=json.load(sys.stdin); print('repo', d.get('full_name'), d.get('default_branch'))"
|
||||
kill "${PF_PID}" 2>/dev/null || true
|
||||
echo "restore-drill-complete"
|
||||
@@ -2,13 +2,13 @@
|
||||
id: RAIL-HO-WP-0005
|
||||
type: workplan
|
||||
title: "Forgejo Production Migration on railiance01"
|
||||
domain: railiance
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: active
|
||||
owner: railiance
|
||||
topic_slug: railiance
|
||||
created: "2026-05-03"
|
||||
updated: "2026-06-04"
|
||||
updated: "2026-07-04"
|
||||
state_hub_workstream_id: "84e17675-0d15-4268-a8bd-540124d37018"
|
||||
---
|
||||
|
||||
@@ -24,6 +24,13 @@ Forgejo will become the heart of Railiance infrastructure. The work must be
|
||||
fully automated, backup-backed, recovery-drilled, and suitable for long-lived
|
||||
operation on railiance01 before any production cutover happens.
|
||||
|
||||
**Sequencing update (2026-07-04):** Production Forgejo is live on railiance01
|
||||
with Gitea still canonical per the safety contract. Repo cutover proceeds
|
||||
**staged per-repo** using a migration ladder (disposable probes → non-production
|
||||
pilots → image-capable pilots → production repos). `state-hub` is last. See
|
||||
`CUST-WP-0054-T04` and
|
||||
`the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`.
|
||||
|
||||
## Placement in the Railiance Tooling Set
|
||||
|
||||
This workplan lives in `railiance-infra` because it is the cross-layer
|
||||
@@ -46,18 +53,23 @@ change is made there.
|
||||
|
||||
## Key Decisions to Confirm
|
||||
|
||||
1. Public/private hostname for Forgejo and whether Gitea remains reachable
|
||||
during the transition.
|
||||
1. ~~Public/private hostname for Forgejo~~ **DECIDED 2026-07-03:**
|
||||
`forgejo.coulomb.social` → railiance01 (`92.205.62.239`). DNS active;
|
||||
Traefik edge live; Forgejo workload deployed and serving HTTPS. Gitea remains
|
||||
canonical until migration drills pass. Record:
|
||||
`the-custodian/docs/forgejo-production-decisions.md`.
|
||||
2. Mail delivery path for password reset and account recovery
|
||||
(SMTP relay, sender domain, SPF/DKIM/DMARC expectations).
|
||||
3. Package registry scope: container images only at first, or also generic,
|
||||
npm, PyPI, Go, Maven, and Helm packages.
|
||||
4. Actions runner model: in-cluster ephemeral runners, long-lived runner pod,
|
||||
or isolated host runner.
|
||||
4. ~~Actions runner model~~ **DECIDED 2026-07-03:** in-cluster long-lived runner
|
||||
Deployment with DinD sidecar on railiance01 (`ADR-004`). Interim coulombcore
|
||||
host runner retired after cutover.
|
||||
5. Backup destination and retention target for database, repositories,
|
||||
attachments, LFS, Actions artifacts/logs, and package data.
|
||||
6. Cutover mode: freeze-and-migrate all repos in one window, or staged
|
||||
project-by-project transition.
|
||||
6. Cutover mode: ~~freeze-all vs staged~~ **LEANING staged per-repo (2026-07-04)**
|
||||
based on `glas-harness` pilot; operator confirmation still needed. Freeze-all
|
||||
remains fallback for final production wave if drift risk is unacceptable.
|
||||
|
||||
## Safety Contract
|
||||
|
||||
@@ -76,30 +88,38 @@ change is made there.
|
||||
repo. No plaintext SMTP passwords, admin tokens, runner tokens, or registry
|
||||
credentials in Git.
|
||||
|
||||
## Probe Strategy
|
||||
## Probe and pilot strategy (revised 2026-07-04)
|
||||
|
||||
A `forgejo-railiance-probe` is reasonable and should be treated as a disposable
|
||||
S5/S4 integration probe, not as the production install.
|
||||
Original T03 planned a **disposable isolated-namespace probe** before any
|
||||
production install. That path was **superseded**: production Forgejo deployed on
|
||||
railiance01 under the safety contract (Gitea remains canonical; no Gitea deletes).
|
||||
|
||||
The probe should prove:
|
||||
Integration evidence now comes from **in-production probes and repo pilots**:
|
||||
|
||||
- Helm values and cnpg database wiring converge cleanly.
|
||||
- Initial admin bootstrap is automated and repeatable.
|
||||
- SMTP/password reset works end-to-end.
|
||||
- Package registry endpoints work for the package types Railiance needs first.
|
||||
- Forgejo Actions can run a minimal workflow and publish a test package.
|
||||
- Backup and restore works in an isolated namespace.
|
||||
- Migration from a sample Gitea repo preserves git history, issues, releases,
|
||||
wiki, LFS or attachments where applicable.
|
||||
| Tier | Repo | Purpose | Status |
|
||||
| --- | --- | --- | --- |
|
||||
| 0 | `coulomb/forgejo-actions-probe` | Runner scheduling, DinD, OCI image-build | **done** |
|
||||
| 1 | `coulomb/glas-harness` | Non-production git+SSH+CI routing drill | **done** |
|
||||
| 2 | `coulomb/key-cape` | Image-build workflow + registry pull on railiance01 | **done** |
|
||||
| 2.5 | `railiance-enablement`, `railiance-infra`, `railiance-apps`, `railiance-platform`, `railiance-cluster` | Stack repos on Forgejo + S4 templates | **done** |
|
||||
| 3 | Production set (`state-hub`, `issue-core`, …) | Canonical remotes, sweep paths, deploy loops | **gated** |
|
||||
|
||||
The probe is destroyed or explicitly archived after production Forgejo is live.
|
||||
Each tier must pass before the next. T03 (isolated probe namespace) is cancelled;
|
||||
acceptance criteria below are tracked across T05, T07, T08, and T10 instead.
|
||||
|
||||
Still to prove before T11:
|
||||
|
||||
- SMTP/password reset end-to-end (T06).
|
||||
- Backup and restore in isolated namespace (T09) — **drill passed 2026-07-04**;
|
||||
scheduled automation pending.
|
||||
- Issues/releases/wiki/LFS per inventory classification (T10 matrix).
|
||||
- Operator SSH identity on Forgejo beyond interim `forgejo_admin` keys (T02/T10).
|
||||
|
||||
## Target Architecture
|
||||
|
||||
```
|
||||
operator / agents / developers
|
||||
-> private HTTPS endpoint
|
||||
-> railiance01 ingress
|
||||
-> https://forgejo.coulomb.social (railiance01 Traefik ingress)
|
||||
-> forgejo Service in forgejo namespace
|
||||
-> Forgejo Deployment/StatefulSet
|
||||
-> forgejo-db CloudNative PG Cluster in databases namespace
|
||||
@@ -115,7 +135,7 @@ operator / agents / developers
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0005-T01
|
||||
status: in_progress
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "cf59d171-5629-45c9-9d44-8d6499827ffc"
|
||||
```
|
||||
@@ -138,13 +158,17 @@ Minimum inventory:
|
||||
Forgejo before cutover and classifies each migration item as automatic,
|
||||
manual, unsupported, or explicitly out of scope.
|
||||
|
||||
**Gap (2026-07-04):** first-pass inventory predates repos created after
|
||||
2026-06-04 (e.g. `glas-harness`, `forgejo-actions-probe`). Refresh org repo
|
||||
list and add a **migration tier** column (0–3) per repo before T11.
|
||||
|
||||
---
|
||||
|
||||
### T02 — Resolve Forgejo production design decisions
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0005-T02
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
needs_human: true
|
||||
state_hub_task_id: "f88115bf-4f99-49ef-a415-0b23750141b3"
|
||||
@@ -152,10 +176,16 @@ state_hub_task_id: "f88115bf-4f99-49ef-a415-0b23750141b3"
|
||||
|
||||
Decide the production choices listed in "Key Decisions to Confirm".
|
||||
|
||||
**Partial (2026-07-04):** hostname, exposure, deployment pattern, live deploy,
|
||||
and in-cluster runner model decided (`ADR-004`). Cutover mode **leaning** staged
|
||||
per-repo (glas-harness pilot). Remaining operator decisions: SMTP, package scope
|
||||
beyond OCI, backup target, final cutover confirmation. See
|
||||
`the-custodian/docs/forgejo-production-decisions.md`.
|
||||
|
||||
Expected output:
|
||||
|
||||
- A short decision record in this workplan or a dedicated ADR.
|
||||
- Hostname and exposure model.
|
||||
- Hostname and exposure model. ✓ hostname; exposure follows railiance01 Traefik
|
||||
- SMTP provider and sender identity.
|
||||
- Package registry scope.
|
||||
- Actions runner isolation model.
|
||||
@@ -167,36 +197,21 @@ choices.
|
||||
|
||||
---
|
||||
|
||||
### T03 — Build forgejo-railiance-probe
|
||||
### T03 — Build forgejo-railiance-probe (isolated namespace)
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0005-T03
|
||||
status: todo
|
||||
status: cancel
|
||||
priority: high
|
||||
state_hub_task_id: "b516018a-415e-4a58-8c62-07c14ece9353"
|
||||
```
|
||||
|
||||
Create a disposable probe environment for Forgejo before touching production.
|
||||
|
||||
Expected repo ownership:
|
||||
|
||||
- `railiance-platform`: probe cnpg database and storage dependencies.
|
||||
- `railiance-apps`: probe Forgejo Helm values and namespace.
|
||||
- `railiance-enablement`: probe Actions runner template and workflows.
|
||||
|
||||
Probe acceptance:
|
||||
|
||||
- `make forgejo-probe-deploy` or equivalent converges from a clean cluster
|
||||
state.
|
||||
- Admin bootstrap is automated.
|
||||
- A test user can reset a password via email.
|
||||
- A test repository can be created, cloned, pushed, and protected.
|
||||
- A test package can be published and pulled.
|
||||
- A test Forgejo Actions workflow runs successfully.
|
||||
- A probe backup restores into an isolated namespace.
|
||||
|
||||
**Done when:** the probe demonstrates the whole lifecycle without manual
|
||||
cluster surgery.
|
||||
**Cancelled 2026-07-04:** superseded by production Forgejo on railiance01 (T05)
|
||||
plus in-production integration probes (`forgejo-actions-probe`, `glas-harness`).
|
||||
Isolated-namespace probe added latency without reducing risk given the safety
|
||||
contract (Gitea canonical, no deletes). Remaining T03 acceptance items map to:
|
||||
T05 (deploy), T06 (mail), T07 (packages), T08 (Actions), T09 (backup restore),
|
||||
T10 (repo migration drill).
|
||||
|
||||
---
|
||||
|
||||
@@ -220,6 +235,11 @@ Minimum scope:
|
||||
packages, Actions artifacts, and logs.
|
||||
- Restore runbook for database and blob/package data.
|
||||
|
||||
**Partial (2026-07-04):** `forgejo-db` CNPG cluster healthy on railiance01
|
||||
(`make forgejo-db-status` → Cluster in healthy state). SOPS secret path and
|
||||
network policies in `railiance-platform`. Remaining: backup/WAL archiving to
|
||||
approved target, blob/package storage restore drill (feeds T09).
|
||||
|
||||
**Done when:** platform dependencies can be deployed and restored without the
|
||||
Forgejo app running.
|
||||
|
||||
@@ -229,7 +249,7 @@ Forgejo app running.
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0005-T05
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "11540ba4-d31c-4f64-836b-c6de69107aa4"
|
||||
```
|
||||
@@ -245,6 +265,12 @@ Minimum scope:
|
||||
- Health/status targets in the Makefile.
|
||||
- Migration-safe configuration for coexistence with Gitea during the cutover.
|
||||
|
||||
**Partial (2026-07-04):** `railiance-apps` deploy live — HTTPS smoke pass,
|
||||
ingress + TLS, SSH NodePort `30022`, Actions enabled, `coulomb` org,
|
||||
`railiance01-build-01` runner (ADR-004). Git push/pull via HTTPS and
|
||||
`forgejo-remote` SSH proven. Remaining: SOPS hardening for all secrets,
|
||||
SMTP (T06), operator user accounts beyond `forgejo_admin`.
|
||||
|
||||
**Done when:** Forgejo runs on railiance01 against production platform
|
||||
services and can serve login, git clone/push, package registry, and admin
|
||||
operations.
|
||||
@@ -301,8 +327,14 @@ Acceptance:
|
||||
- Retention and cleanup expectations are documented.
|
||||
- Package data is included in backup and restore drills.
|
||||
|
||||
**Done when:** `state-hub` or a probe image can be published to Forgejo and
|
||||
pulled by railiance01.
|
||||
**Partial (2026-07-04):** OCI registry live (`/v2/` auth challenge). Tier-0/2
|
||||
images built and pulled on railiance01: `forgejo-actions-probe`, `key-cape`
|
||||
(`crictl pull forgejo.coulomb.social/coulomb/key-cape:latest` succeeded).
|
||||
Remaining: `state-hub` image after tier-3 approval; document retention; include
|
||||
packages in backup drill (T09).
|
||||
|
||||
**Done when:** tier-2 gate is fully satisfied (✓) and tier-3 production images
|
||||
follow the same pattern after explicit approval.
|
||||
|
||||
---
|
||||
|
||||
@@ -310,7 +342,7 @@ pulled by railiance01.
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0005-T08
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "f45f98c9-2f02-4224-bbfd-c2e1ec38581e"
|
||||
```
|
||||
@@ -326,8 +358,22 @@ Minimum scope:
|
||||
- Secret handling policy for Actions.
|
||||
- Resource limits to avoid repeating previous single-node overload patterns.
|
||||
|
||||
**Done when:** a representative repository can run Forgejo Actions and publish
|
||||
a test artifact without privileged cluster-wide credentials.
|
||||
**Partial (2026-07-04):** in-cluster runner live (`railiance-apps/manifests/
|
||||
forgejo-runner.yaml`, ADR-004). Proven workflows: `forgejo-actions-probe`
|
||||
(image-build), `glas-harness` (host+container CI smoke). Org secrets
|
||||
`REGISTRY_USER`/`REGISTRY_TOKEN` set. Documented constraints: host runner is
|
||||
non-root (static docker-cli, no `apk add`); `actions/checkout@v4` fails — use
|
||||
`git clone` in job. Remaining: reusable workflow templates in
|
||||
`railiance-enablement` (S4); resource limits review; no cluster-admin on runner.
|
||||
|
||||
**Partial (2026-07-04):** tier-2 satisfied by `key-cape` (`container-build`,
|
||||
archive checkout, static docker-cli). **Templates (2026-07-04):** S4
|
||||
`railiance-enablement/workflows/` (`ci-smoke`, `container-build-push`,
|
||||
`container-build-push-multirepo` for `hub-core` contexts) +
|
||||
`docs/forgejo-actions-workflow-templates.md`. Remaining: resource limits review.
|
||||
|
||||
**Done when:** tier-2 pilot repo runs Forgejo Actions end-to-end and publishes
|
||||
a pullable image without privileged cluster-wide credentials. **Tier 2: done.**
|
||||
|
||||
---
|
||||
|
||||
@@ -335,7 +381,7 @@ a test artifact without privileged cluster-wide credentials.
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0005-T09
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "25892007-36ca-4bd9-8adf-84d505465d7d"
|
||||
```
|
||||
@@ -360,34 +406,63 @@ Acceptance:
|
||||
- Restore into an isolated namespace is drilled and documented.
|
||||
- RPO/RTO expectations are recorded.
|
||||
|
||||
**Partial (2026-07-04):** isolated restore drill **passed**. Production
|
||||
`forgejo dump` (~11.7 MiB) restored into `forgejo-restore-drill` namespace;
|
||||
post-restore API checks: health 200, `coulomb/glas-harness` and
|
||||
`coulomb/key-cape` on `main`, 3 org repos visible. Evidence:
|
||||
`docs/forgejo-restore-drill-evidence.md`. Assets: `infra/forgejo-restore-drill/`,
|
||||
`tools/forgejo-restore-drill.sh`. Remaining: scheduled CNPG/off-cluster backups,
|
||||
encryption/approved target (T02/T04), automated dump schedule.
|
||||
|
||||
**Done when:** a fresh backup restores to a working isolated Forgejo instance
|
||||
with repository, package, and user recovery checks passing.
|
||||
with repository, package, and user recovery checks passing **and** scheduled
|
||||
backups run without manual intervention.
|
||||
|
||||
---
|
||||
|
||||
### T10 — Drill Gitea to Forgejo migration
|
||||
### T10 — Drill Gitea to Forgejo migration (staged ladder)
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0005-T10
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "6befde73-00bc-4643-be0b-a7ce7944e75f"
|
||||
```
|
||||
|
||||
Run a non-production migration drill from Gitea to Forgejo.
|
||||
Run staged migration drills from Gitea to Forgejo before production repos move.
|
||||
|
||||
Minimum checks:
|
||||
**Tier 1 complete (2026-07-04):** `glas-harness` — git history preserved,
|
||||
`origin` on Forgejo, `gitea` legacy remote retained, SSH+HTTPS push, CI smoke
|
||||
green. Result matrix:
|
||||
`the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`.
|
||||
|
||||
Minimum checks (per tier):
|
||||
|
||||
- Git history and default branches preserved.
|
||||
- Issues, labels, milestones, releases, wiki, and attachments handled per
|
||||
inventory classification.
|
||||
- SSH/HTTPS clone and push paths work.
|
||||
- Existing local remotes can be transformed predictably.
|
||||
- State Hub registered repo remotes can be updated safely.
|
||||
- Rollback plan is rehearsed.
|
||||
inventory classification (N/A for tier-1 git-only repos).
|
||||
- SSH/HTTPS clone and push paths work (`forgejo-remote` in `~/.ssh/config`).
|
||||
- Existing local remotes can be transformed predictably (`origin`/`gitea` split).
|
||||
- State Hub registered repo remotes can be updated safely (deferred for tier-1).
|
||||
- Rollback plan is rehearsed (Gitea copy unchanged).
|
||||
|
||||
**Done when:** a sample migration has a written result matrix and no unknown
|
||||
critical migration gaps remain.
|
||||
**Tier 2 complete (2026-07-04):** `key-cape` — multi-stage Dockerfile built and
|
||||
pushed via archive-checkout workflow; `crictl pull` on railiance01 succeeded.
|
||||
Evidence in `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`
|
||||
(tier 2 section).
|
||||
|
||||
**Tier 2.5 complete (2026-07-04):** railiance stack repos on Forgejo with
|
||||
`origin=forgejo-remote`, `gitea` legacy remote, `ci-smoke` workflows; operator
|
||||
`tegwick` SSH on Forgejo; enablement templates published. Evidence: pilot doc
|
||||
tier 2.5 section.
|
||||
|
||||
**Not ready:** `state-hub` (tier 3) until playbook is **executed** for
|
||||
`state-hub` (playbook drafted 2026-07-04:
|
||||
`the-custodian/docs/forgejo-tier3-remote-url-sweep-playbook.md`) and scheduled
|
||||
backups gate is closed (disaster-control).
|
||||
|
||||
**Done when:** tiers 0–2 pass with written result matrices and no unknown
|
||||
critical migration gaps remain for production repos.
|
||||
|
||||
---
|
||||
|
||||
@@ -401,19 +476,21 @@ needs_human: true
|
||||
state_hub_task_id: "b1b66687-ca33-4971-b312-743c8e059c5e"
|
||||
```
|
||||
|
||||
Execute the production migration only after the probe, backup restore, package
|
||||
registry, email recovery, and Actions gates pass.
|
||||
Execute production migration only after T06, T07, T08, T09, and T10 tier 0–2
|
||||
gates pass. `state-hub` and other Wave-1 production repos require explicit
|
||||
operator approval per `CUST-WP-0054` drain sequence.
|
||||
|
||||
Cutover sequence:
|
||||
**Preferred cutover (staged per-repo):**
|
||||
|
||||
1. Announce freeze window.
|
||||
2. Take final Gitea backup and verify it exists.
|
||||
3. Freeze Gitea writes.
|
||||
4. Migrate repositories and metadata to Forgejo.
|
||||
5. Validate critical repositories and package pulls.
|
||||
6. Update State Hub repo remotes and host paths as needed.
|
||||
7. Update local and railiance01 remotes.
|
||||
8. Keep Gitea read-only as rollback until the stabilization window passes.
|
||||
1. Per repo: Gitea backup snapshot (or org-wide before each wave).
|
||||
2. Mirror git to Forgejo; switch workstation `origin` to `forgejo-remote`.
|
||||
3. Port/verify Actions workflows on Forgejo runner.
|
||||
4. Update State Hub `remote_url` and railiance01 sweep checkouts when promoted.
|
||||
5. Mark Gitea repo read-only (org policy); do not delete.
|
||||
6. Repeat until production set complete.
|
||||
|
||||
**Freeze-all fallback:** single window if staged drift is unacceptable — same
|
||||
steps but all repos in one maintenance period.
|
||||
|
||||
**Done when:** all Railiance/Custodian repos use Forgejo as primary, Gitea is
|
||||
read-only fallback, and rollback instructions are documented.
|
||||
@@ -447,19 +524,31 @@ legacy Gitea either archived or intentionally retained as documented fallback.
|
||||
## Phasing and Dependencies
|
||||
|
||||
```
|
||||
T01 inventory ─┬─► T02 decisions ─┬─► T03 probe ─┬─► T04 platform
|
||||
│ │ ├─► T05 app
|
||||
│ │ ├─► T06 mail recovery
|
||||
│ │ ├─► T07 packages
|
||||
│ │ ├─► T08 actions
|
||||
│ │ └─► T09 backups
|
||||
└────────────────────────────────────► T10 migration drill
|
||||
T01 inventory ──► T02 decisions ──┬──► T04 platform (forgejo-db ✓ partial)
|
||||
├──► T05 app (live ✓ partial)
|
||||
├──► T06 mail recovery
|
||||
├──► T07 packages (OCI probe ✓ partial)
|
||||
├──► T08 actions (runner ✓ partial)
|
||||
└──► T09 backups
|
||||
|
||||
T03-T10 all pass ─► T11 production cutover ─► T12 legacy Gitea retirement
|
||||
T05+T08 ──► T10 migration ladder ──► T11 production cutover ──► T12 Gitea retire
|
||||
tier0 probe ✓
|
||||
tier1 glas-harness ✓
|
||||
tier2 key-cape ✓
|
||||
tier2.5 railiance-stack ✓
|
||||
tier3 production (gated)
|
||||
|
||||
T03 isolated probe: CANCELLED (superseded by T05 + in-production pilots)
|
||||
```
|
||||
|
||||
Recommended first slice: T01, T02, T03. Do not start T11 until T06, T07, T08,
|
||||
T09, and T10 are complete.
|
||||
**Current focus (2026-07-04):** T10 tiers 0–2 **complete**; T09 restore drill
|
||||
**passed** (scheduled backups + backup target still open); T02 decisions (SMTP,
|
||||
backup target) before tier-3 production repos.
|
||||
Do not start T11 `state-hub` until T09 complete and `CUST-WP-0054` Wave-1
|
||||
gates satisfied.
|
||||
|
||||
**Absorbed by `CUST-WP-0054-T04`:** forge + CI on railiance01; workstation
|
||||
build retirement; staged repo promotion before State Hub primary move (T05).
|
||||
|
||||
## railiance-bootstrap Note
|
||||
|
||||
@@ -479,7 +568,14 @@ purpose is identified.
|
||||
|
||||
- `RAIL-HO-WP-0004-production-readiness.md`
|
||||
- `RAIL-HO-WP-0003-5repo-stack-restructure.md`
|
||||
- `CUST-WP-0054-workstation-independence-and-fleet-realignment.md` (T04 forge+CI)
|
||||
- `CUST-WP-0014-repo-sync-automation.md`
|
||||
- `CUST-WP-0021-multi-host-repo-paths.md`
|
||||
- `docs/adr/ADR-004-forgejo-in-cluster-actions-runner.md`
|
||||
- `docs/forgejo-migration-inventory.md`
|
||||
- `the-custodian/docs/forgejo-production-decisions.md`
|
||||
- `the-custodian/docs/forgejo-repo-migration-pilot-glas-harness.md`
|
||||
- `railiance-apps/docs/forgejo-on-railiance01.md`
|
||||
- `railiance-forge/docs/forgejo-actions-runner-substrate.md`
|
||||
- `ops/incidents/2026-03-25-gitea-pgpool-crashloop.md`
|
||||
- `ops/incidents/2026-03-26-coulombcore-runaway-agent-overload.md`
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
id: RAIL-HO-WP-0001
|
||||
type: workplan
|
||||
title: "Secure Single-Server Bootstrap at HostEurope"
|
||||
domain: railiance
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: completed
|
||||
status: finished
|
||||
owner: railiance
|
||||
topic_slug: railiance
|
||||
repo_goal_id: 9e835b82-acca-493a-943f-2553ffe0bf54
|
||||
@@ -18,8 +18,7 @@ handoff_note: >
|
||||
done in railiance-bootstrap (harden.yml, bootstrap.yml) is superseded by
|
||||
the roles/base and roles/sops_agent structure in this repo. The HostEurope
|
||||
server IP is 92.205.62.239 (hosts.ini is gitignored — recreate from
|
||||
inventory/servers.yaml when working on the host).
|
||||
---
|
||||
inventory/servers.yaml when working on the host).---
|
||||
|
||||
# Secure Single-Server Bootstrap at HostEurope
|
||||
|
||||
@@ -56,7 +55,6 @@ ssh -R 8000:127.0.0.1:8000 <user>@92.205.62.239
|
||||
|
||||
Then Claude on the remote host can reach `http://127.0.0.1:8000` as normal.
|
||||
See also: the tunnel setup note in `CLAUDE.md`.
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
@@ -2,9 +2,9 @@
|
||||
id: RAIL-HO-WP-0002
|
||||
type: workplan
|
||||
title: "Server Specification and Automated Test Suite"
|
||||
domain: railiance
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: completed
|
||||
status: finished
|
||||
owner: railiance
|
||||
topic_slug: railiance
|
||||
state_hub_workstream_id: "8fed53c2-4c39-4471-8bb9-61f58771fe0c"
|
||||
@@ -2,9 +2,9 @@
|
||||
id: RAIL-HO-WP-0003
|
||||
type: workplan
|
||||
title: "Railiance 5-Repo Stack Restructure"
|
||||
domain: railiance
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: completed
|
||||
status: finished
|
||||
owner: railiance
|
||||
topic_slug: railiance
|
||||
state_hub_workstream_id: "3ae0afc5-13f2-4e6c-aea7-1c1fb9f1ab81"
|
||||
@@ -2,7 +2,7 @@
|
||||
id: RAIL-HO-WP-0004
|
||||
type: workplan
|
||||
title: "Railiance Production Readiness — Automated, Reproducible Stack"
|
||||
domain: railiance
|
||||
domain: financials
|
||||
repo: railiance-infra
|
||||
status: finished
|
||||
owner: worsch
|
||||
@@ -437,7 +437,7 @@ context.
|
||||
|
||||
```task
|
||||
id: RAIL-HO-WP-0004-T09
|
||||
status: cancelled
|
||||
status: cancel
|
||||
priority: medium
|
||||
state_hub_task_id: "d2afe78a-eb51-4ce9-b332-f181323d2370"
|
||||
needs_human: false
|
||||
Reference in New Issue
Block a user