feat(sso-mfa): T03 PostgreSQL manifests (NK-WP-0001-T03)

CloudNativePG Cluster CR (net-kingdom-pg, PostgreSQL 16) with two application databases: keycloak_db (owner: keycloak) and privacyidea_db (owner: privacyidea). Passwords managed continuously via managed.roles. WAL archiving section stubbed and commented; activate when object storage is available. ScheduledBackup CR included (daily 02:00 UTC, 7d retention). Also: sync workplan status for T01 (Phase 0a done), T02 (manifests done), T03 (manifests done, restore drill pending); close NK-WP-0002. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 09:22:13 +01:00
parent 2ebb231f19
commit 8929bf65bc
7 changed files with 533 additions and 7 deletions
--- a/sso-mfa/k8s/postgresql/README.md
+++ b/sso-mfa/k8s/postgresql/README.md
@@ -0,0 +1,143 @@
+# T03 — PostgreSQL (CloudNativePG)
+
+Phase 2 of NK-WP-0001: CloudNativePG cluster with `keycloak_db` and `privacyidea_db`.
+
+## Prerequisites
+
+- T02 complete: `databases` namespace and NetworkPolicies applied
+- `kubectl` configured with cluster access
+- `gen-secrets.sh` run and output stored in KeePassXC
+
+## Apply order
+
+### 1. Install CloudNativePG operator
+
+```bash
+helm repo add cnpg https://cloudnative-pg.github.io/charts
+helm repo update
+helm install cnpg cnpg/cloudnative-pg \
+  --namespace cnpg-system \
+  --create-namespace \
+  --wait
+```
+
+Verify:
+```bash
+kubectl get pods -n cnpg-system
+kubectl get crd clusters.postgresql.cnpg.io
+```
+
+### 2. Create K8s Secrets
+
+```bash
+# From the postgresql/ directory:
+chmod +x create-secrets.sh
+./create-secrets.sh ../../bootstrap/secrets
+```
+
+Alternatively, if you've already shredded the generated files, reconstruct from KeePassXC:
+
+```bash
+kubectl create secret generic net-kingdom-pg-keycloak-app \
+  --namespace=databases \
+  --from-literal=username=keycloak \
+  --from-literal=password='<PG_KEYCLOAK_PASSWORD from KeePassXC>'
+
+kubectl create secret generic net-kingdom-pg-privacyidea-app \
+  --namespace=databases \
+  --from-literal=username=privacyidea \
+  --from-literal=password='<PI_DB_PASSWORD from KeePassXC>'
+```
+
+### 3. Deploy the cluster
+
+```bash
+kubectl apply -f cluster.yaml
+```
+
+Wait for cluster to become ready (this provisions PVCs and runs initdb — allow 2–3 minutes):
+```bash
+kubectl wait --for=condition=Ready cluster/net-kingdom-pg \
+  -n databases --timeout=300s
+```
+
+Check status:
+```bash
+kubectl get cluster -n databases
+kubectl describe cluster net-kingdom-pg -n databases
+kubectl get pods -n databases
+```
+
+### 4. Verify databases and users
+
+```bash
+# Connect as superuser to verify setup
+kubectl exec -it -n databases \
+  $(kubectl get pod -n databases -l cnpg.io/cluster=net-kingdom-pg,role=primary -o name) \
+  -- psql -U postgres
+
+# In psql:
+\l                          -- list databases
+\du                         -- list roles
+\q
+```
+
+Expected output: `keycloak_db`, `privacyidea_db`, roles `keycloak` and `privacyidea`.
+
+### 5. Configure backup (when object storage is available)
+
+Uncomment the `backup:` section in `cluster.yaml` and fill in the object store endpoint.
+Create the S3 credentials secret:
+
+```bash
+kubectl create secret generic net-kingdom-pg-backup-s3 \
+  --namespace=databases \
+  --from-literal=ACCESS_KEY_ID='<access key>' \
+  --from-literal=SECRET_ACCESS_KEY='<secret key>'
+```
+
+Apply the updated cluster.yaml, then:
+```bash
+kubectl apply -f scheduled-backup.yaml
+```
+
+### 6. Run the restore drill
+
+**Mandatory before marking T03 done.**
+
+```bash
+# Trigger a manual backup first
+kubectl cnpg backup net-kingdom-pg -n databases
+
+# Wait for backup to complete
+kubectl get backup -n databases --watch
+
+# Restore to a new cluster to verify
+# (See CloudNativePG docs: kubectl cnpg restore or Cluster bootstrap.recovery)
+```
+
+### 7. Run the full verification script
+
+```bash
+chmod +x ../verify-t03.sh
+../verify-t03.sh
+```
+
+## Secrets reference
+
+| Secret name | Keys | Purpose |
+|---|---|---|
+| `net-kingdom-pg-keycloak-app` | `username`, `password` | Keycloak DB user (also bootstrap owner) |
+| `net-kingdom-pg-privacyidea-app` | `username`, `password` | privacyIDEA DB user |
+| `net-kingdom-pg-backup-s3` | `ACCESS_KEY_ID`, `SECRET_ACCESS_KEY` | Object store backup (optional until backup enabled) |
+| `net-kingdom-pg-superuser` | auto-created by CNPG | PostgreSQL superuser (operator-managed) |
+| `net-kingdom-pg-app` | auto-created by CNPG | Initial app user (unused — we use named secrets) |
+
+## Notes
+
+- `cnpg.io/cluster: net-kingdom-pg` label on pods is what the NetworkPolicies in T02 target.
+  Do not rename the cluster without also updating netpol-databases.yaml.
+- `instances: 1` is intentional for dev/staging. Change to 3 before ThreePhoenix HA production
+  deployment (requires at least 3 schedulable nodes).
+- Password rotation: update the K8s Secret values and CNPG's managed.roles reconciler will
+  apply the change at the next reconciliation cycle (within seconds).
--- a/sso-mfa/k8s/postgresql/cluster.yaml
+++ b/sso-mfa/k8s/postgresql/cluster.yaml
@@ -0,0 +1,102 @@
+# CloudNativePG Cluster — net-kingdom-pg
+#
+# Creates a PostgreSQL 16 cluster with two application databases:
+#   keycloak_db   (owner: keycloak)
+#   privacyidea_db (owner: privacyidea)
+#
+# Prerequisites:
+#   - CloudNativePG operator installed (see README.md)
+#   - K8s Secrets created (see create-secrets.sh)
+#   - databases namespace exists (T02)
+#
+# Adjust `instances` before production: 1 for dev/staging, 3 for HA.
+# Adjust `storage.size` to match available PVC capacity.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: net-kingdom-pg
+  namespace: databases
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: databases
+spec:
+  # ── Instance count ───────────────────────────────────────────────────────────
+  # 1 = dev/single-node. Increase to 3 for ThreePhoenix HA production deployment.
+  instances: 1
+
+  imageName: ghcr.io/cloudnative-pg/postgresql:16
+
+  # ── Bootstrap ────────────────────────────────────────────────────────────────
+  # Creates keycloak_db with owner keycloak. privacyidea_db and the
+  # privacyidea role are created in postInitSQL (runs as superuser).
+  # managed.roles below reconciles passwords for both users continuously.
+  bootstrap:
+    initdb:
+      database: keycloak_db
+      owner: keycloak
+      secret:
+        name: net-kingdom-pg-keycloak-app
+      postInitSQL:
+        - "CREATE ROLE privacyidea WITH LOGIN;"
+        - "CREATE DATABASE privacyidea_db OWNER privacyidea;"
+        - "REVOKE CONNECT ON DATABASE privacyidea_db FROM PUBLIC;"
+        - "REVOKE CONNECT ON DATABASE keycloak_db FROM PUBLIC;"
+        - "GRANT CONNECT ON DATABASE keycloak_db TO keycloak;"
+        - "GRANT CONNECT ON DATABASE privacyidea_db TO privacyidea;"
+
+  # ── Managed roles ────────────────────────────────────────────────────────────
+  # Operator reconciles these passwords continuously from K8s Secrets.
+  # This ensures password rotation in KeePassXC/Vault propagates to PG.
+  managed:
+    roles:
+      - name: keycloak
+        ensure: present
+        login: true
+        passwordSecret:
+          name: net-kingdom-pg-keycloak-app
+      - name: privacyidea
+        ensure: present
+        login: true
+        passwordSecret:
+          name: net-kingdom-pg-privacyidea-app
+
+  # ── Storage ──────────────────────────────────────────────────────────────────
+  storage:
+    size: 10Gi
+    # storageClass: local-path   # uncomment to pin StorageClass explicitly
+
+  # ── WAL archiving (backup prerequisite) ─────────────────────────────────────
+  # Uncomment the backup section when object storage is available (MinIO/S3).
+  # WAL archiving must be enabled here before ScheduledBackup will function.
+  #
+  # backup:
+  #   barmanObjectStore:
+  #     destinationPath: "s3://net-kingdom-backups/postgres/"
+  #     endpointURL: "http://minio.minio-system.svc.cluster.local:9000"
+  #     s3Credentials:
+  #       accessKeyId:
+  #         name: net-kingdom-pg-backup-s3
+  #         key: ACCESS_KEY_ID
+  #       secretAccessKey:
+  #         name: net-kingdom-pg-backup-s3
+  #         key: SECRET_ACCESS_KEY
+  #     wal:
+  #       compression: gzip
+  #     data:
+  #       compression: gzip
+  #       immediateCheckpoint: true
+  #   retentionPolicy: "7d"
+
+  # ── Resource limits ──────────────────────────────────────────────────────────
+  resources:
+    requests:
+      memory: "256Mi"
+      cpu: "100m"
+    limits:
+      memory: "1Gi"
+      cpu: "1000m"
+
+  # ── Monitoring ───────────────────────────────────────────────────────────────
+  # Set enablePodMonitor: true when Prometheus / kube-prometheus-stack is deployed.
+  monitoring:
+    enablePodMonitor: false
--- a/sso-mfa/k8s/postgresql/create-secrets.sh
+++ b/sso-mfa/k8s/postgresql/create-secrets.sh
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# create-secrets.sh — create K8s Secrets for PostgreSQL from gen-secrets.sh output
+#
+# Usage:
+#   ./create-secrets.sh <secrets-dir>
+#
+# <secrets-dir> is the output directory produced by sso-mfa/bootstrap/gen-secrets.sh
+# (default: ../../bootstrap/secrets).
+#
+# Creates two K8s Secrets in the databases namespace:
+#   net-kingdom-pg-keycloak-app       — keycloak DB credentials
+#   net-kingdom-pg-privacyidea-app    — privacyIDEA DB credentials
+#
+# These secrets must exist before applying cluster.yaml.
+# Re-run this script whenever you rotate passwords in KeePassXC / gen-secrets.sh.
+
+set -euo pipefail
+
+SECRETS_DIR="${1:-../../bootstrap/secrets}"
+
+if [[ ! -d "$SECRETS_DIR" ]]; then
+    echo "ERROR: secrets directory not found: $SECRETS_DIR" >&2
+    echo "Run sso-mfa/bootstrap/gen-secrets.sh first, then re-run this script." >&2
+    exit 1
+fi
+
+PG_SECRETS="$SECRETS_DIR/postgres/secrets.env"
+PI_SECRETS="$SECRETS_DIR/privacyidea/secrets.env"
+
+if [[ ! -f "$PG_SECRETS" ]]; then
+    echo "ERROR: $PG_SECRETS not found" >&2
+    exit 1
+fi
+if [[ ! -f "$PI_SECRETS" ]]; then
+    echo "ERROR: $PI_SECRETS not found" >&2
+    exit 1
+fi
+
+# Source the generated env files (they contain KEY=VALUE pairs, no export)
+# Use a subshell to avoid polluting the current environment.
+PG_KC_PASS=$(bash -c "source $PG_SECRETS 2>/dev/null; echo \$PG_KEYCLOAK_PASSWORD")
+PI_DB_PASS=$(bash -c "source $PI_SECRETS 2>/dev/null; echo \$PI_DB_PASSWORD")
+
+if [[ -z "$PG_KC_PASS" || -z "$PI_DB_PASS" ]]; then
+    echo "ERROR: could not read passwords from secrets files." >&2
+    echo "Check that gen-secrets.sh ran successfully and the files are intact." >&2
+    exit 1
+fi
+
+echo "Creating K8s Secret: net-kingdom-pg-keycloak-app"
+kubectl create secret generic net-kingdom-pg-keycloak-app \
+    --namespace=databases \
+    --from-literal=username=keycloak \
+    --from-literal=password="$PG_KC_PASS" \
+    --dry-run=client -o yaml | kubectl apply -f -
+
+echo "Creating K8s Secret: net-kingdom-pg-privacyidea-app"
+kubectl create secret generic net-kingdom-pg-privacyidea-app \
+    --namespace=databases \
+    --from-literal=username=privacyidea \
+    --from-literal=password="$PI_DB_PASS" \
+    --dry-run=client -o yaml | kubectl apply -f -
+
+echo ""
+echo "Done. Secrets created in namespace: databases"
+echo ""
+echo "Verify:"
+echo "  kubectl get secrets -n databases"
+echo "  kubectl describe secret net-kingdom-pg-keycloak-app -n databases"
--- a/sso-mfa/k8s/postgresql/scheduled-backup.yaml
+++ b/sso-mfa/k8s/postgresql/scheduled-backup.yaml
@@ -0,0 +1,26 @@
+# CloudNativePG ScheduledBackup — net-kingdom-pg
+#
+# PREREQUISITE: WAL archiving must be enabled in cluster.yaml (backup.barmanObjectStore
+# section) before this ScheduledBackup will succeed. Uncomment cluster.yaml backup
+# block first, apply it, confirm WAL archiving is healthy, then apply this file.
+#
+# Schedule: daily at 02:00 UTC, keeping 7 daily backups.
+# Adjust schedule and retentionPolicy to match your RPO/RTO requirements.
+#
+# See T03 restore drill procedure in README.md before marking T03 done.
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: net-kingdom-pg-daily
+  namespace: databases
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: databases
+spec:
+  # Daily at 02:00 UTC
+  schedule: "0 0 2 * * *"   # CloudNativePG uses Go cron format: seconds minutes hours dom month dow
+  backupOwnerReference: self
+  cluster:
+    name: net-kingdom-pg
+  # Immediate: if the schedule is missed (e.g. pod restart), take a backup immediately
+  immediate: true