Close S5 app readiness workplan

2026-06-05 17:59:35 +02:00
parent 3e2eae6b14
commit 9c2713f9c4
8 changed files with 376 additions and 32 deletions
--- a/SCOPE.md
+++ b/SCOPE.md
@@ -71,6 +71,9 @@ lessons into reusable S5 app release patterns.
  - scripts under `tools/`.
 - S5 app runbooks and recipes in `docs/`, especially:
  - `docs/vergabe-teilnahme.md`;
+  - `docs/s5-app-onboarding-checklist.md`;
+  - `docs/app-data-backup-restore-handoff.md`;
+  - `docs/manifest-server-dry-run.md`;
  - `docs/django-on-railiance.md`;
  - `docs/operator-setup.md`;
  - `docs/operator-recipes.md`.
@@ -135,6 +138,9 @@ lessons into reusable S5 app release patterns.
  `vergabe-teilnahme` lock in its source repo.
 - `vergabe-teilnahme` is represented as a local Helm chart plus values,
  ingress, Makefile targets, and an operator runbook.
+- Reusable S5 app onboarding, app data restore handoff, and manifest
+  server-dry-run prerequisite docs now capture the repeatable parts of the
+  first app release.
 - Several deployment lessons are now repo-local guardrails: URL-encoded
  database URL secret creation, Django probe Host headers, operator tool checks,
  SOPS age-key checks, server-side manifest dry-run, and persistent-pod smoke
@@ -147,16 +153,16 @@ lessons into reusable S5 app release patterns.

 ## Known Gaps And Opportunities

- The first-app lessons from `vergabe-teilnahme` are documented, but there is
-  no reusable "new S5 app release checklist" yet.
- The manifest dry-run workflow assumes access to a representative cluster and
-  CRDs. Forge-owned runner labels, placement, and credential prerequisites are
+- The reusable S5 app onboarding checklist exists, but still needs to be
+  exercised by the next real app release.
+- The manifest dry-run prerequisite contract exists. Enforcing it in CI still
+  depends on forge-owned runner labels, placement, and credential evidence
  defined in
  `/home/worsch/railiance-forge/docs/ci-runner-actions-gitops-ownership.md`;
-  the app-side workflow behavior still needs explicit S5 readiness docs.
- App-level backup and restore responsibilities need clearer handoff contracts
-  with `railiance-platform`, especially for shared CNPG databases consumed by
-  S5 apps. Forge artifact restore and secret-custody evidence is defined in
+  plus representative cluster access from lower layers.
+- App-level backup and restore handoffs are documented, but production-trust use
+  of `vergabe_db` still depends on platform-owned `apps-pg` backup and restore
+  evidence. Forge artifact restore and secret-custody evidence is defined in
  `/home/worsch/railiance-forge/docs/backup-restore-secret-handoff.md`.
 - Forge release-readiness evidence that S5 app runbooks may cite is defined in
  `/home/worsch/railiance-forge/docs/observability-operating-evidence.md`.
@@ -252,9 +258,14 @@ keywords: [operator, runbook, sops, cnpg, dry-run, smoke-test, deployment]
 5. For Gitea registry work, use the forge-owned docs directly:
   `/home/worsch/railiance-forge/docs/gitea-container-registry.md` and
   `/home/worsch/railiance-forge/docs/gitea-package-registry.md`.
-6. For `vergabe-teilnahme`, start with `docs/vergabe-teilnahme.md`, then inspect
+6. For a new S5 app release, start with
+   `docs/s5-app-onboarding-checklist.md`.
+7. For `vergabe-teilnahme`, start with `docs/vergabe-teilnahme.md`, then inspect
   `charts/vergabe-teilnahme/`, `helm/vergabe-teilnahme-values.yaml`, and
   `manifests/vergabe-teilnahme-ingress.yaml`.
-7. Run `make check-tools` before deploy work. Run
+8. For data durability and server-side dry-run readiness, read
+   `docs/app-data-backup-restore-handoff.md` and
+   `docs/manifest-server-dry-run.md`.
+9. Run `make check-tools` before deploy work. Run
   `SOPS_SENTINEL=<encrypted-file> make check-sops` when app release work
   touches encrypted SOPS files.
--- a/docs/app-data-backup-restore-handoff.md
+++ b/docs/app-data-backup-restore-handoff.md
@@ -0,0 +1,94 @@
+# App Data Backup And Restore Handoff
+
+This document defines the S5 app release boundary for data durability. It does
+not create backup jobs, authorize a live restore drill, or move platform
+backup ownership into `railiance-apps`.
+
+## Current App Data
+
+`vergabe-teilnahme` stores relational app data in `vergabe_db` on the shared
+CloudNativePG cluster `apps-pg` in the `databases` namespace. The cluster is an
+S3 platform service owned by `railiance-platform`; see
+`/home/worsch/railiance-platform/docs/apps-pg.md`.
+
+The app currently has no durable media PVC enabled. `persistence.media.enabled`
+is `false`, so uploaded media is deferred rather than an S5 durability promise.
+
+## Ownership Matrix
+
+| Concern | S5 app repo owns | Upstream owner |
+| --- | --- | --- |
+| App database request | App name, namespace, database name, role name, intended use, and production-readiness need | `railiance-platform` reviews and provisions the role/database |
+| Runtime DB Secret use | Secret name in the app namespace and URL-encoded DSN rebuild helper | `railiance-platform` owns platform credential source and future secret delivery |
+| Database backup job | Readiness gate and consumer evidence requirement | `railiance-platform` owns CNPG backup and restore implementation |
+| App restore verification | App-specific post-restore checks, migrations, login/smoke path, and rollback note | `railiance-platform` restores the backing database |
+| Forge images/packages | Artifact identity and consumer evidence cited by app runbooks | `railiance-forge` owns registry/package restore evidence |
+| App media/blob data | PVC declaration and app-level restore checks if enabled | `railiance-platform` owns storage backup mechanism once media is production-critical |
+
+## Production Readiness Gate
+
+Before an app release treats data as production-critical, the app runbook should
+record:
+
+- data class: disposable, externally reproducible, or production-critical;
+- owning platform workplan or doc for the backup mechanism;
+- latest non-secret backup evidence reference;
+- latest restore-drill evidence reference, ideally from an isolated
+  environment;
+- app-specific post-restore checks, such as migrations, health endpoint,
+  login/admin path, and representative business workflow;
+- rollback or disable path if restore fails;
+- assertion that no secret material was copied into Git, logs, screenshots, or
+  State Hub notes.
+
+If this gate is missing, the app can still be used for smoke, development, or
+migration validation, but promotion beyond that should create or link a
+`railiance-platform` workplan.
+
+## `vergabe-teilnahme` Gate
+
+Current posture:
+
+- database: `vergabe_db` on `databases/apps-pg`;
+- app role Secret: `vergabe-app-credentials`;
+- env Secret: `vergabe-teilnahme-env`;
+- current backup status: platform docs state that `apps-pg` backup coverage is
+  follow-up work;
+- restore status: no app-level restore drill evidence is recorded in this repo.
+
+Minimum evidence before production-trust use:
+
+- platform confirms `apps-pg` backup coverage for `vergabe_db`;
+- an isolated restore drill proves the database can be restored;
+- `vergabe-teilnahme` runs migrations successfully after restore;
+- health endpoint and HTTPS smoke checks pass;
+- operator verifies the app can complete a representative tender-management
+  workflow after restore;
+- any app media path remains disabled or has its own storage restore evidence.
+
+## Forge Artifact Evidence
+
+S5 runbooks may cite forge-owned package and blob restore evidence, but must not
+own Gitea package backup procedures or registry credentials. Use
+`/home/worsch/railiance-forge/docs/backup-restore-secret-handoff.md` for the
+forge artifact boundary.
+
+For app releases, cite:
+
+- image repository, tag, and digest when available;
+- source commit and package version;
+- forge publish job or evidence reference;
+- package/blob restore drill evidence when the artifact is production-critical;
+- namespace-local pull Secret or approved workload secret path, without token
+  values.
+
+## Filing Upstream Gaps
+
+When the missing durability item is not local to S5:
+
+1. Keep the S5 task focused on the app release impact.
+2. Create or link the platform/forge workplan that owns the missing mechanism.
+3. Mark the S5 task `blocked` only when the app release cannot safely continue
+   without that upstream evidence.
+4. Record the State Hub workstream/task id in the app runbook or workplan.
+5. Revisit the S5 promotion gate after upstream evidence exists.
--- a/docs/manifest-server-dry-run.md
+++ b/docs/manifest-server-dry-run.md
@@ -0,0 +1,94 @@
+# Manifest Server Dry-Run Prerequisites
+
+`make k8s-server-dry-run` checks committed manifests and rendered app charts
+against a Kubernetes API server using server-side dry-run. It catches schema and
+admission drift that Helm rendering alone cannot see.
+
+## What The Command Does
+
+The helper in `tools/k8s-server-dry-run.sh`:
+
+1. verifies `kubectl` and `helm` are installed;
+2. verifies a Kubernetes API server is reachable;
+3. optionally creates the target namespace when
+   `DRY_RUN_CREATE_NAMESPACES=true`;
+4. renders `charts/vergabe-teilnahme/` with
+   `helm/vergabe-teilnahme-values.yaml`;
+5. runs `kubectl apply --dry-run=server -f manifests`;
+6. runs `kubectl apply --dry-run=server` against the rendered chart output.
+
+The namespace creation step is a real apply, not a dry-run. Use
+`DRY_RUN_CREATE_NAMESPACES=true` only against a disposable or approved
+representative cluster where creating the app namespace is acceptable.
+
+## Representative Cluster Requirement
+
+The check expects a live Kubernetes API server whose version, admission
+webhooks, and installed APIs are close enough to Railiance to be meaningful. A
+pure local render or unseeded kind cluster is not enough.
+
+The current `vergabe-teilnahme` release uses built-in Kubernetes APIs:
+
+- `apps/v1` Deployment;
+- `v1` Service;
+- `v1` PersistentVolumeClaim when media persistence is enabled;
+- `networking.k8s.io/v1` Ingress.
+
+For realistic S5 validation, the representative cluster should also have the
+same ingress class, cert-manager issuer posture, NetworkPolicy posture, and
+admission policies as Railiance. Future app manifests that introduce CNPG,
+cert-manager, Traefik, External Secrets, or other CRDs require those CRDs and
+webhooks to be installed before the dry-run result is meaningful.
+
+## Runner And Credential Requirements
+
+Local operator runs require:
+
+- `kubectl` context pointed at the representative cluster;
+- credentials with `get` access for API discovery;
+- server-side dry-run permission for the rendered resources;
+- namespace create/apply permission only when
+  `DRY_RUN_CREATE_NAMESPACES=true`.
+
+CI runs require forge-owned runner prerequisites:
+
+- a runner label approved for S5 release checks, such as `s5-release-check` or
+  `cluster-dry-run`;
+- approved kubeconfig or equivalent cluster access delivery;
+- runner placement that is allowed to reach the representative API server;
+- no kubeconfig, bearer token, package token, or secret value stored in Git.
+
+The runner label contract and secret boundary live in
+`/home/worsch/railiance-forge/docs/ci-runner-actions-gitops-ownership.md`.
+
+## Failure Classification
+
+Treat these as release-blocking when the representative cluster and runner are
+known-good:
+
+- Helm render fails;
+- server-side dry-run rejects a changed manifest;
+- Kubernetes schema or admission policy rejects an app resource;
+- the rendered image reference or required Secret name is structurally invalid.
+
+Treat these as prerequisite gaps rather than app release failures:
+
+- no runner with the required label is available;
+- the runner cannot reach the representative cluster;
+- kubeconfig or secret delivery is missing;
+- required CRDs/admission webhooks are absent from the representative cluster;
+- namespace creation is forbidden while `DRY_RUN_CREATE_NAMESPACES=true`.
+
+For prerequisite gaps, link or create the owning forge, cluster, platform, or
+enablement workplan instead of weakening the S5 app chart.
+
+## Enforcement Gate
+
+The workflow in `.gitea/workflows/manifest-server-dry-run.yaml` is ready to
+enforce PR checks only when:
+
+- forge has published the runner label and placement evidence;
+- cluster/platform have provided representative API access and secret delivery;
+- the namespace side effect is accepted or pre-provisioned;
+- at least one successful dry-run result is recorded for the current release
+  surface.
--- a/docs/operator-recipes.md
+++ b/docs/operator-recipes.md
@@ -38,6 +38,11 @@ make k8s-server-dry-run
 ```

 The command expects a representative Kubernetes API server with the same
-CRDs as the Railiance cluster. CI should run it against a disposable kind
-cluster seeded with CNPG, cert-manager, Traefik, and any other CRDs used
-by changed manifests.
+APIs, CRDs, admission webhooks, ingress posture, and cert-manager posture as
+the Railiance cluster. The CI workflow sets `DRY_RUN_CREATE_NAMESPACES=true`,
+which creates the app namespace before server-side dry-run so namespaced
+resources can validate. Use that mode only against a disposable or approved
+representative cluster.
+
+See `docs/manifest-server-dry-run.md` for runner, credential, and failure
+classification rules.
--- a/docs/s5-app-onboarding-checklist.md
+++ b/docs/s5-app-onboarding-checklist.md
@@ -0,0 +1,100 @@
+# S5 App Onboarding Checklist
+
+Use this checklist when adding a new user-facing Railiance workload to
+`railiance-apps`. It turns the `vergabe-teilnahme` lessons into a repeatable
+starting point so new app releases do not have to read the historical
+workplans first.
+
+## Scope And Planning
+
+- [ ] Create or update a repo-local workplan under `workplans/`.
+- [ ] Confirm the work is S5 app release wiring, not application source code,
+  forge runtime operation, platform service provisioning, or cluster addon work.
+- [ ] Name the source application repo and the owning package/image release
+  path.
+- [ ] Record upstream dependencies on `railiance-platform`, `railiance-forge`,
+  or `railiance-enablement` instead of hiding them in app values.
+- [ ] Run State Hub consistency sync after task-status edits.
+
+## Release Files
+
+- [ ] Add a Helm chart under `charts/<app>/`.
+- [ ] Add non-secret production values under `helm/<app>-values.yaml`.
+- [ ] Add app-specific manifests under `manifests/` only when they do not fit
+  cleanly in the chart.
+- [ ] Keep `image.tag` pinned by git SHA or immutable version.
+- [ ] Keep committed values non-secret. Runtime secrets must come from
+  Kubernetes Secrets, approved SOPS files, or a platform secret-delivery path.
+- [ ] Add Makefile targets for render, deploy, status, logs, migrations, and
+  any app-specific secret rebuild helpers.
+
+## Image And Artifact Consumption
+
+- [ ] Use a forge-owned image or package registry path.
+- [ ] Link to source-repo publish instructions instead of duplicating build
+  pipelines here.
+- [ ] Record the source commit, image tag, package version, and evidence needed
+  by the app runbook.
+- [ ] For private images or packages, name the consuming Secret or approved
+  secret-delivery path without storing tokenized URLs.
+- [ ] Verify a cluster can pull the image before promoting the release beyond
+  smoke-test use.
+
+## Database And Runtime Secrets
+
+- [ ] Request app database, role, and Secret handoff through
+  `railiance-platform` when using shared platform databases.
+- [ ] Use an app-scoped database and role, for example `<app>_db` and `<app>`.
+- [ ] Mirror only the app role credential into the app namespace.
+- [ ] If the app consumes `DATABASE_URL`, URL-encode generated passwords before
+  writing the env Secret.
+- [ ] Prefer separate PostgreSQL env vars when a framework does not require a
+  single DSN string.
+- [ ] Document secret rotation commands without printing or committing secret
+  values.
+
+## Ingress, TLS, And Probes
+
+- [ ] Name the public host, namespace, Helm release, ingress, and TLS Secret in
+  the app runbook.
+- [ ] Confirm ingress class and cert-manager issuer ownership with
+  `railiance-cluster`.
+- [ ] Keep certificate lifecycle in cert-manager, not in app scripts.
+- [ ] For framework apps with strict host validation, set HTTP probe `Host`
+  headers to a value included in the app's allowed hosts.
+- [ ] Keep readiness and liveness paths stable and unauthenticated.
+
+## Validation And Smoke Tests
+
+- [ ] Run `make check-tools`.
+- [ ] Run `make <app>-dry-run` or the equivalent Helm render before deploy.
+- [ ] Run `make k8s-server-dry-run` against a representative cluster before
+  enforcing PR checks.
+- [ ] Use the persistent-pod plus `kubectl exec` smoke pattern from
+  `docs/operator-recipes.md`.
+- [ ] Capture app-level deployment evidence: dry-run result, rollout status,
+  HTTPS or service smoke check, migration result when applicable, and rollback
+  note.
+
+## Runbook Baseline
+
+Each S5 app runbook should include:
+
+- identity table with URL, namespace, release, chart, values, ingress, image,
+  database, and TLS Secret;
+- secrets and rotation section;
+- day-to-day operator commands;
+- image promotion steps;
+- rollback behavior and migration warning;
+- troubleshooting for probes, database URLs, TLS, and app-specific failure
+  modes;
+- backup and restore readiness gate;
+- cross-references to source repo, platform handoff docs, forge artifact docs,
+  and common S5 recipes.
+
+## Done Gate
+
+A new app is ready for routine S5 operation when an operator can deploy, verify,
+roll back, inspect logs, rotate app-owned runtime secrets, understand upstream
+data durability gates, and sync the workplan without reading old app-specific
+history.
--- a/docs/vergabe-teilnahme.md
+++ b/docs/vergabe-teilnahme.md
@@ -125,18 +125,26 @@ fail, cert-manager keeps serving the old cert until it expires.
 Investigate with `kubectl describe certificate vergabe-teilnahme-tls
 -n vergabe-teilnahme`.

-## Backup posture (open)
+## Data durability and restore readiness

-The shared `apps-pg` cluster is not yet covered by an automated
-backup job — only the legacy PostgreSQL-HA setup is. Manual logical
-dump for now:
+`vergabe_db` lives on the shared `apps-pg` CNPG cluster owned by
+`railiance-platform`. S5 owns the app release runbook and post-restore app
+checks; platform owns the database backup and restore mechanism.
+
+Current status: `apps-pg` backup coverage is still platform follow-up work, so
+`vergabe-teilnahme` should not be treated as production-critical data until the
+gate in `docs/app-data-backup-restore-handoff.md` is satisfied.
+
+Manual logical dump is a break-glass or inspection option, not the durable
+backup contract:

 ```bash
 kubectl exec -n databases apps-pg-1 -- pg_dump -U postgres -Fc vergabe_db > vergabe_db-$(date +%F).dump
 ```

-Tracked as a follow-up in `RAILIANCE-WP-0003 Notes` (CNPG backup
-configuration belongs to `railiance-platform`).
+Before promotion beyond smoke or development use, record platform backup
+evidence, an isolated restore drill, migration result, health check, HTTPS
+smoke check, and representative app workflow verification.

 ## Deferred for v1

@@ -155,6 +163,9 @@ configuration belongs to `railiance-platform`).
 - Shared DB cluster: `railiance-platform/docs/apps-pg.md`
 - Container registry: `/home/worsch/railiance-forge/docs/gitea-container-registry.md`
 - Python package registry: `/home/worsch/railiance-forge/docs/gitea-package-registry.md`
+- S5 app onboarding checklist: `docs/s5-app-onboarding-checklist.md`
+- App data backup handoff: `docs/app-data-backup-restore-handoff.md`
+- Manifest dry-run prerequisites: `docs/manifest-server-dry-run.md`
 - Django deployment recipe: `docs/django-on-railiance.md`
 - Operator setup: `docs/operator-setup.md`
 - Operator recipes: `docs/operator-recipes.md`
--- a/tools/k8s-server-dry-run.sh
+++ b/tools/k8s-server-dry-run.sh
@@ -17,9 +17,16 @@ for cmd in kubectl helm; do
  fi
 done

-kubectl api-resources >/dev/null
+echo "server dry-run: checking Kubernetes API discovery"
+if ! kubectl api-resources >/dev/null; then
+  echo "ERROR: cannot reach a representative Kubernetes API server" >&2
+  echo "Check kubeconfig, runner placement, and cluster access prerequisites." >&2
+  echo "See docs/manifest-server-dry-run.md." >&2
+  exit 1
+fi

 if [[ "$DRY_RUN_CREATE_NAMESPACES" == "true" ]]; then
+  echo "server dry-run: ensuring namespace $VERGABE_NAMESPACE exists"
  kubectl create namespace "$VERGABE_NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
 fi

--- a/workplans/RAILIANCE-WP-0005-s5-app-release-readiness.md
+++ b/workplans/RAILIANCE-WP-0005-s5-app-release-readiness.md
@@ -4,7 +4,7 @@ type: workplan
 title: "S5 app release readiness and scope alignment"
 domain: railiance
 repo: railiance-apps
-status: active
+status: finished
 owner: codex
 topic_slug: railiance
 planning_priority: medium
@@ -27,16 +27,18 @@ The 2026-06-05 `railiance-forge` extraction moved canonical registry operating
 docs and registry-retention policy into the new forge layer. This workplan now
 keeps only app-release readiness items in S5.

-The same review found several gaps:
+The same review found several planning-time gaps that are closed by this
+workplan:

- `INTENT.md` is missing, so purpose and scope are collapsed into one document.
- The `vergabe-teilnahme` runbook still contains stale image-promotion guidance
-  tied to the old local `issue-core` build context.
- First-app lessons are documented, but there is no reusable checklist for the
-  next S5 app release.
- Forge package storage and app database backup responsibilities need clearer
+- `INTENT.md` was missing, so purpose and scope were collapsed into one
+  document.
+- The `vergabe-teilnahme` runbook contained stale image-promotion guidance tied
+  to the old local `issue-core` build context.
+- First-app lessons were documented, but not yet turned into a reusable
+  checklist for the next S5 app release.
+- Forge package storage and app database backup responsibilities needed clearer
  contracts with platform-layer work.
- The server-side dry-run workflow does not state its live-cluster/CRD
+- The server-side dry-run workflow did not state its live-cluster/CRD
  prerequisites clearly enough for a future runner.

 This workplan turns those scope gaps into the next improvement strand.
@@ -100,7 +102,7 @@ compatibility pointers were removed later by `RAILIANCE-WP-0006-T10`.

 ```task
 id: RAILIANCE-WP-0005-T03
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "4eab93a9-ad1b-46ca-97ef-18a059f64ab5"
 ```
@@ -122,13 +124,19 @@ The checklist should cover:
 Done when a new app can start from the checklist without reading all historical
 workplans first.

+Completed on 2026-06-05 by adding
+`docs/s5-app-onboarding-checklist.md` and cross-linking it from `SCOPE.md` and
+the `vergabe-teilnahme` runbook. The checklist covers app scope, chart/values
+layout, forge artifact consumption, database and secret handoff, ingress/TLS,
+probe Host headers, smoke tests, runbook baseline, and State Hub sync.
+
 ---

 ## T04 - Define app data backup and restore handoffs

 ```task
 id: RAILIANCE-WP-0005-T04
-status: todo
+status: done
 priority: high
 state_hub_task_id: "299d9623-3a54-4e85-9a70-016e8356c3d9"
 ```
@@ -151,13 +159,20 @@ Done when `SCOPE.md` and app runbooks clearly separate S5 release ownership from
 S3 backup implementation while still giving operators an actionable restore
 readiness gate.

+Completed on 2026-06-05 by adding
+`docs/app-data-backup-restore-handoff.md`, updating
+`docs/vergabe-teilnahme.md`, and refreshing `SCOPE.md`. The handoff states that
+S5 owns app release evidence and post-restore app checks, while
+`railiance-platform` owns `apps-pg` backup/restore mechanisms and
+`railiance-forge` owns artifact restore evidence.
+
 ---

 ## T05 - Make manifest dry-run workflow prerequisites explicit

 ```task
 id: RAILIANCE-WP-0005-T05
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "6cf0e662-d7e2-48b1-b1f2-c7636240dd81"
 ```
@@ -177,6 +192,13 @@ Questions to answer:
 Done when a future operator can tell whether the workflow is ready to enforce
 PR checks or still needs runner/cluster preparation.

+Completed on 2026-06-05 by adding
+`docs/manifest-server-dry-run.md`, updating `docs/operator-recipes.md`, and
+making `tools/k8s-server-dry-run.sh` print an explicit representative-cluster
+preflight error. The docs classify runner/cluster prerequisite gaps separately
+from release-blocking manifest failures and note that
+`DRY_RUN_CREATE_NAMESPACES=true` creates the namespace as a real side effect.
+
 ---

 ## T06 - Hand off Gitea package registry storage and retention posture