coulomb/the-custodian
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f353c039afe8d7e04b2682d7a34f88ed931b6adf
the-custodian/docs/forgejo-production-decisions.md
codex 3ea0291f54 Record T02 cutover Option A and close design decision set 
Staged per-repo promotion as primary plan; freeze-all emergency fallback only.
All WP-0005 T02 production choices are now decided.
2026-07-07 17:00:05 +02:00

10 KiB
Raw Blame History

Forgejo Production Decisions (Wave 1)

Date: 2026-07-03
Workplans: RAIL-HO-WP-0005-T02, CUST-WP-0054-T04
Operator input: 2026-07-03

Decision log

# Topic Decision Status Evidence (2026-07-03)
1 Production hostname forgejo.coulomb.social decided DNS A → 92.205.62.239 (railiance01); HTTPS reaches Traefik on railiance01
2 Exposure model Private HTTPS via railiance01 Traefik ingress + cert-manager letsencrypt-prod decided Same pattern as Gitea (manifests/forgejo-ingress.yaml in railiance-apps)
2b Deployment pattern gitea-charts/gitea 12.5.0 Helm + Forgejo image; CNPG forgejo-db in railiance-platform; Makefile in railiance-apps decided Chart 12.6+ requires Gitea 1.26 config edit-ini (incompatible with Forgejo 11); see railiance-apps/docs/forgejo-on-railiance01.md
2c Live deploy Forgejo pod + ingress + TLS on railiance01 done (2026-07-03) make forgejo-smoke → HTTP 200 + OCI /v2/ 401 challenge; cert forgejo-tls Ready
3 Gitea during transition gitea.coulomb.social on coulombcore remains canonical until Forgejo restore/migration drills pass; then read-only mirror unchanged Per RAIL-HO-WP-0005 safety contract
4 SMTP / password reset IONOS SMTP; sender forgejo@coulomb.social; OpenBao/ESO mailer on railiance01 done (2026-07-08) smtp.ionos.de:587 + smtp+starttls; T06 verified
5 Package registry scope Multi-ecosystem (Option C): OCI + npm + generic; PyPI/Go/Maven/Helm as inventory proves need decided (2026-07-08) OCI live; npm is launch requirement — see § Package scope
6 Actions runner model In-cluster on railiance01: forgejo-runner Deployment + DinD (railiance01-build-01) done (2026-07-03) Runner 2/2 Ready; coulombcore host runner disabled; image-build probe pushed OCI image via org secrets
7 Backup target + retention Option A: age-encrypted forgejo dump + pg_dump → Nextcloud WebDAV decided (2026-07-09) See § Backup; T04/T09 implementation
8 Cutover mode Option A: staged per-repo promotion (no org-wide freeze) decided (2026-07-09) See § Cutover; tiers 0–2.5 + partial tier-3 proven

Hostname decision detail

Chosen hostname: https://forgejo.coulomb.social

Field Value
DNS forgejo.coulomb.social → 92.205.62.239 (railiance01)
Edge railiance01 k3s Traefik (kube-system/traefik LoadBalancer)
Target machine railiance01 (production home per CUST-WP-0054)
Canonical git remote (post-cutover) https://forgejo.coulomb.social/coulomb/<repo>.git
OCI registry (post-cutover) forgejo.coulomb.social/coulomb/<image>

Live probe (2026-07-03, post-deploy)

getent hosts forgejo.coulomb.social   # 92.205.62.239
curl -fsS -o /dev/null -w '%{http_code}\n' https://forgejo.coulomb.social/   # 200
curl -sSI -X GET https://forgejo.coulomb.social/v2/ | grep -i docker-distribution  # registry/2.0
KUBECONFIG=~/.kube/config-hosteurope kubectl get pods,ingress,certificate -n forgejo

Forgejo is serving HTTPS with a valid Let's Encrypt cert. Gitea on coulombcore remains canonical for git remotes until migration drills pass.

Implications for CUST-WP-0054

  • Wave 1 can proceed with a fixed hostname for overlays, ingress manifests, and CI IMAGE_REPOSITORY variables.
  • State Hub / sweep checkouts on railiance01 (T05) should clone from forgejo.coulomb.social once cutover completes.
  • Runners and image-build CI are proven; first repo migration pilot (glas-harness) documents git/SSH/CI routing in docs/forgejo-repo-migration-pilot-glas-harness.md.
  • Remaining blockers for production cutover: scheduled backup automation (T04/T09 per Option A), npm/package hardening (T07), and staged per-repo migration execution (RAIL-HO-WP-0005-T10 / T11).

SMTP decision (IONOS) — 2026-07-07

Provider: IONOS mail for coulomb.social and subdomains (operator owns DNS zone via IONOS; mailboxes provisioned in IONOS control panel).

Recommended Forgejo mailer settings (non-secret; password in railiance-apps/helm/forgejo-secrets.sops.yaml):

Field Value
MAILER_TYPE smtp
HOST smtp.ionos.com:587 (EU accounts may use smtp.ionos.de:587)
IS_TLS_ENABLED true (STARTTLS on 587)
FROM forgejo@coulomb.social (or dedicated noreply@… mailbox)
USER full mailbox address (same as FROM unless using a shared relay mailbox)
PASSWD IONOS mailbox password — SOPS only, never Git plaintext

Operator setup checklist:

  1. Create mailbox in IONOS (e.g. forgejo@coulomb.social).
  2. Confirm SPF includes IONOS (include:_spf-eu.ionos.com or current IONOS guidance for the zone).
  3. Enable DKIM in IONOS for coulomb.social if offered.
  4. Store mailbox password in helm/forgejo-secrets.sops.yaml under gitea.config.mailer.PASSWD; redeploy Forgejo.
  5. T06 gate: trigger password reset for a controlled non-admin test account; confirm delivery and link works.

Implementation: railiance-apps/docs/forgejo-on-railiance01.md (SMTP section). Password in OpenBao platform/workloads/forgejo/forgejo-mailer (field PASSWD).

Package scope decision (Option C) — 2026-07-08

Operator choice: Multi-ecosystem registry — not OCI-only at launch.

Phase Package types Status
Now (proven) OCI container images Live — tier 0–3 CI (container-build-push templates)
Launch requirement npm Decide → implement in T07 (auth, publish/pull docs, CI template)
In scope Generic (release tarballs/binaries) Enable with retention policy
As needed PyPI, Go, Maven, Helm Add per repo only after T01 inventory shows Gitea package data or an explicit consumer need

Rationale: Railiance needs npm publishing; other ecosystems follow evidence, not a big-bang enable-everything default.

Implications:

  • T07 expands beyond OCI smoke: npm publish/pull runbook, org/repo token policy, retention/cleanup for non-OCI blobs.
  • T09 backups must include all enabled package types under /data/packages.
  • T01 authenticated Gitea inventory should classify existing npm (and other) package data before cutover waves.

Forgejo endpoints (post-cutover):

Type URL pattern
OCI forgejo.coulomb.social/coulomb/<image>
npm https://forgejo.coulomb.social/api/packages/coulomb/npm/ (configure scope in T07)
Generic per-repo generic package API

Backup decision (Option A) — 2026-07-09

Operator choice: Extend the existing Railiance platform backup lane — age-encrypted artifacts uploaded to Nextcloud WebDAV (same pattern as railiance-platform make backup / coulombcore railiance-backup).

Component Method Schedule
Forgejo blob state forgejo dump zip (repos, packages incl. OCI/npm/generic, attachments, LFS, avatars) Daily
PostgreSQL pg_dump from CNPG forgejo-db (logical; no WAL/PITR in Phase 1) Daily
Encryption age (platform backup public key) before upload per artifact
Destination Nextcloud WebDAV file drop (off-node) upload after each run
Restore proof Re-run tools/forgejo-restore-drill.sh from automated backup quarterly manual gate

Retention (balanced profile):

Artifact Keep
Daily dumps 14 rotations
Weekly dumps 4 rotations (promote Sunday daily or explicit weekly job)
Local cache (if any) 7 copies per type (match existing railiance-backup prune)

RPO / RTO targets:

Metric Target
RPO 24 hours (daily schedule)
RTO 4 hours (operator response + restore drill path + validation)

Implementation owners:

  • railiance-platform — make forgejo-backup (or extend make backup): cron schedule, age encrypt, Nextcloud upload, retention prune.
  • railiance-infra — restore runbook + quarterly drill evidence (T09).
  • railiance-apps — no backup secrets in Git; dump runs against live pod.

Explicitly not in scope (Phase 1): CNPG WAL archiving to S3/MinIO; on-node-only backups without Nextcloud upload. Revisit if dump size or RPO requirements outgrow daily logical backup.

Promotion gate: No further tier-3 repo cutovers until automated daily backups have succeeded 7 consecutive days and one restore drill uses a Nextcloud artifact (not workstation /tmp).

Cutover decision (Option A) — 2026-07-09

Operator choice: Staged per-repo promotion — continue the proven migration ladder; no org-wide freeze-all window.

Per-repo sequence (T11):

  1. Gitea backup snapshot before promotion.
  2. Mirror/sync git to Forgejo if not already present.
  3. Verify Forgejo Actions and packages (OCI/npm per repo).
  4. Switch workstation origin → forgejo-remote; retain gitea legacy remote.
  5. Update State Hub remote_url and railiance01 sweep checkouts when promoted.
  6. Mark that Gitea repo read-only; do not delete (safety contract).
  7. Stabilization window before the next repo (operator judgment; state-hub used 7d).

Explicitly rejected for primary plan: org-wide freeze-all cutover (Option B). Freeze-all remains documented only as an emergency fallback if staged drift becomes unacceptable — not the default path.

Rollback (per repo): Re-point origin to gitea-remote; Forgejo copy becomes stale mirror. No Gitea repo deletes for ≥90 days after promotion. Gitea Helm stays until T12 explicit approval.

Gates before each tier-3 promotion:

  • T09 automated daily backups: 7 consecutive successes (Option A).
  • T01 inventory classification for that repo (issues/wiki/LFS/packages).
  • Operator explicit approval for Wave-1 production repos (CUST-WP-0054).

Evidence: tiers 0–2.5 complete; glas-harness pilot; state-hub cutover under CUST-WP-0054-T05.

T02 decision set — complete (2026-07-09)

All Wave-1 production design choices for Forgejo are decided:

# Topic Option
Hostname / exposure / deploy decided 2026-07-03
SMTP IONOS + OpenBao/ESO (T06 done)
Package scope Option C (OCI + npm + generic)
Actions runner In-cluster ADR-004
Backup Option A (Nextcloud + age)
Cutover Option A (staged per-repo)

Open decisions (need operator input)

All T02 items decided as of 2026-07-09. Implementation tracks: T04/T09 backup automation, T07 npm hardening, T10/T11 staged migration waves.

Reference in New Issue View Git Blame Copy Permalink