the-custodian/docs/forgejo-production-decisions.md

Forgejo Production Decisions (Wave 1)

Date: 2026-07-03
Workplans: RAIL-HO-WP-0005-T02, CUST-WP-0054-T04
Operator input: 2026-07-03

Decision log

#	Topic	Decision	Status	Evidence (2026-07-03)
1	Production hostname	forgejo.coulomb.social	decided	DNS A → 92.205.62.239 (railiance01); HTTPS reaches Traefik on railiance01
2	Exposure model	Private HTTPS via railiance01 Traefik ingress + cert-manager letsencrypt-prod	decided	Same pattern as Gitea (manifests/forgejo-ingress.yaml in railiance-apps)
2b	Deployment pattern	gitea-charts/gitea 12.5.0 Helm + Forgejo image; CNPG forgejo-db in railiance-platform; Makefile in railiance-apps	decided	Chart 12.6+ requires Gitea 1.26 config edit-ini (incompatible with Forgejo 11); see railiance-apps/docs/forgejo-on-railiance01.md
2c	Live deploy	Forgejo pod + ingress + TLS on railiance01	done (2026-07-03)	make forgejo-smoke → HTTP 200 + OCI /v2/ 401 challenge; cert forgejo-tls Ready
3	Gitea during transition	gitea.coulomb.social on coulombcore remains canonical until Forgejo restore/migration drills pass; then read-only mirror	unchanged	Per RAIL-HO-WP-0005 safety contract
4	SMTP / password reset	IONOS SMTP; sender forgejo@coulomb.social; OpenBao/ESO mailer on railiance01	done (2026-07-08)	smtp.ionos.de:587 + smtp+starttls; T06 verified
5	Package registry scope	Multi-ecosystem (Option C): OCI + npm + generic; PyPI/Go/Maven/Helm as inventory proves need	decided (2026-07-08)	OCI live; npm is launch requirement — see § Package scope
6	Actions runner model	In-cluster on railiance01: forgejo-runner Deployment + DinD (railiance01-build-01)	done (2026-07-03)	Runner 2/2 Ready; coulombcore host runner disabled; image-build probe pushed OCI image via org secrets
7	Backup target + retention	Option A: age-encrypted forgejo dump + pg_dump → Nextcloud WebDAV	decided (2026-07-09)	See § Backup; T04/T09 implementation
8	Cutover mode	Option A: staged per-repo promotion (no org-wide freeze)	decided (2026-07-09)	See § Cutover; tiers 0–2.5 + partial tier-3 proven

Hostname decision detail

Chosen hostname: https://forgejo.coulomb.social

Field	Value
DNS	forgejo.coulomb.social → 92.205.62.239 (railiance01)
Edge	railiance01 k3s Traefik (kube-system/traefik LoadBalancer)
Target machine	railiance01 (production home per CUST-WP-0054)
Canonical git remote (post-cutover)	https://forgejo.coulomb.social/coulomb/<repo>.git
OCI registry (post-cutover)	forgejo.coulomb.social/coulomb/<image>

Live probe (2026-07-03, post-deploy)

getent hosts forgejo.coulomb.social   # 92.205.62.239
curl -fsS -o /dev/null -w '%{http_code}\n' https://forgejo.coulomb.social/   # 200
curl -sSI -X GET https://forgejo.coulomb.social/v2/ | grep -i docker-distribution  # registry/2.0
KUBECONFIG=~/.kube/config-hosteurope kubectl get pods,ingress,certificate -n forgejo

Forgejo is serving HTTPS with a valid Let's Encrypt cert. Gitea on coulombcore remains canonical for git remotes until migration drills pass.

Implications for CUST-WP-0054

• Wave 1 can proceed with a fixed hostname for overlays, ingress manifests, and CI IMAGE_REPOSITORY variables.
• State Hub / sweep checkouts on railiance01 (T05) should clone from forgejo.coulomb.social once cutover completes.
• Runners and image-build CI are proven; first repo migration pilot (glas-harness) documents git/SSH/CI routing in docs/forgejo-repo-migration-pilot-glas-harness.md.
• Remaining blockers for production cutover: scheduled backup automation (T04/T09 per Option A), npm/package hardening (T07), and staged per-repo migration execution (RAIL-HO-WP-0005-T10 / T11).

SMTP decision (IONOS) — 2026-07-07

Provider: IONOS mail for coulomb.social and subdomains (operator owns DNS zone via IONOS; mailboxes provisioned in IONOS control panel).

Recommended Forgejo mailer settings (non-secret; password in railiance-apps/helm/forgejo-secrets.sops.yaml):

Field	Value
MAILER_TYPE	smtp
HOST	smtp.ionos.com:587 (EU accounts may use smtp.ionos.de:587)
IS_TLS_ENABLED	true (STARTTLS on 587)
FROM	forgejo@coulomb.social (or dedicated noreply@… mailbox)
USER	full mailbox address (same as FROM unless using a shared relay mailbox)
PASSWD	IONOS mailbox password — SOPS only, never Git plaintext

Operator setup checklist:

1. Create mailbox in IONOS (e.g. forgejo@coulomb.social).
2. Confirm SPF includes IONOS (include:_spf-eu.ionos.com or current IONOS guidance for the zone).
3. Enable DKIM in IONOS for coulomb.social if offered.
4. Store mailbox password in helm/forgejo-secrets.sops.yaml under gitea.config.mailer.PASSWD; redeploy Forgejo.
5. T06 gate: trigger password reset for a controlled non-admin test account; confirm delivery and link works.

Implementation: railiance-apps/docs/forgejo-on-railiance01.md (SMTP section). Password in OpenBao platform/workloads/forgejo/forgejo-mailer (field PASSWD).

Package scope decision (Option C) — 2026-07-08

Operator choice: Multi-ecosystem registry — not OCI-only at launch.

Phase	Package types	Status
Now (proven)	OCI container images	Live — tier 0–3 CI (container-build-push templates)
Launch requirement	npm	Decide → implement in T07 (auth, publish/pull docs, CI template)
In scope	Generic (release tarballs/binaries)	Enable with retention policy
As needed	PyPI, Go, Maven, Helm	Add per repo only after T01 inventory shows Gitea package data or an explicit consumer need

Rationale: Railiance needs npm publishing; other ecosystems follow evidence, not a big-bang enable-everything default.

Implications:

• T07 expands beyond OCI smoke: npm publish/pull runbook, org/repo token policy, retention/cleanup for non-OCI blobs.
• T09 backups must include all enabled package types under /data/packages.
• T01 authenticated Gitea inventory should classify existing npm (and other) package data before cutover waves.

Forgejo endpoints (post-cutover):

Type	URL pattern
OCI	forgejo.coulomb.social/coulomb/<image>
npm	https://forgejo.coulomb.social/api/packages/coulomb/npm/ (configure scope in T07)
Generic	per-repo generic package API

Backup decision (Option A) — 2026-07-09

Operator choice: Extend the existing Railiance platform backup lane — age-encrypted artifacts uploaded to Nextcloud WebDAV (same pattern as railiance-platform make backup / coulombcore railiance-backup).

Component	Method	Schedule
Forgejo blob state	forgejo dump zip (repos, packages incl. OCI/npm/generic, attachments, LFS, avatars)	Daily
PostgreSQL	pg_dump from CNPG forgejo-db (logical; no WAL/PITR in Phase 1)	Daily
Encryption	age (platform backup public key) before upload	per artifact
Destination	Nextcloud WebDAV file drop (off-node)	upload after each run
Restore proof	Re-run tools/forgejo-restore-drill.sh from automated backup quarterly	manual gate

Retention (balanced profile):

Artifact	Keep
Daily dumps	14 rotations
Weekly dumps	4 rotations (promote Sunday daily or explicit weekly job)
Local cache (if any)	7 copies per type (match existing railiance-backup prune)

RPO / RTO targets:

Metric	Target
RPO	24 hours (daily schedule)
RTO	4 hours (operator response + restore drill path + validation)

Implementation owners:

• railiance-platform — make forgejo-backup (or extend make backup): cron schedule, age encrypt, Nextcloud upload, retention prune.
• railiance-infra — restore runbook + quarterly drill evidence (T09).
• railiance-apps — no backup secrets in Git; dump runs against live pod.

Explicitly not in scope (Phase 1): CNPG WAL archiving to S3/MinIO; on-node-only backups without Nextcloud upload. Revisit if dump size or RPO requirements outgrow daily logical backup.

Promotion gate: No further tier-3 repo cutovers until automated daily backups have succeeded 7 consecutive days and one restore drill uses a Nextcloud artifact (not workstation /tmp).

Cutover decision (Option A) — 2026-07-09

Operator choice: Staged per-repo promotion — continue the proven migration ladder; no org-wide freeze-all window.

Per-repo sequence (T11):

1. Gitea backup snapshot before promotion.
2. Mirror/sync git to Forgejo if not already present.
3. Verify Forgejo Actions and packages (OCI/npm per repo).
4. Switch workstation origin → forgejo-remote; retain gitea legacy remote.
5. Update State Hub remote_url and railiance01 sweep checkouts when promoted.
6. Mark that Gitea repo read-only; do not delete (safety contract).
7. Stabilization window before the next repo (operator judgment; state-hub used 7d).

Explicitly rejected for primary plan: org-wide freeze-all cutover (Option B). Freeze-all remains documented only as an emergency fallback if staged drift becomes unacceptable — not the default path.

Rollback (per repo): Re-point origin to gitea-remote; Forgejo copy becomes stale mirror. No Gitea repo deletes for ≥90 days after promotion. Gitea Helm stays until T12 explicit approval.

Gates before each tier-3 promotion:

• T09 automated daily backups: 7 consecutive successes (Option A).
• T01 inventory classification for that repo (issues/wiki/LFS/packages).
• Operator explicit approval for Wave-1 production repos (CUST-WP-0054).

Evidence: tiers 0–2.5 complete; glas-harness pilot; state-hub cutover under CUST-WP-0054-T05.

T02 decision set — complete (2026-07-09)

All Wave-1 production design choices for Forgejo are decided:

#	Topic	Option
Hostname / exposure / deploy	decided 2026-07-03
SMTP	IONOS + OpenBao/ESO (T06 done)
Package scope	Option C (OCI + npm + generic)
Actions runner	In-cluster ADR-004
Backup	Option A (Nextcloud + age)
Cutover	Option A (staged per-repo)

Open decisions (need operator input)

All T02 items decided as of 2026-07-09. Implementation tracks: T04/T09 backup automation, T07 npm hardening, T10/T11 staged migration waves.