net-kingdom/sso-mfa/k8s/postgresql/README.md

T03 — PostgreSQL (CloudNativePG)

Phase 2 of NK-WP-0001: CloudNativePG cluster with keycloak_db and privacyidea_db.

Prerequisites

• T02 complete: databases namespace and NetworkPolicies applied
• kubectl configured with cluster access
• gen-secrets.sh run and output stored in KeePassXC

Apply order

1. Install CloudNativePG operator

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
helm install cnpg cnpg/cloudnative-pg \
  --namespace cnpg-system \
  --create-namespace \
  --wait

Verify:

kubectl get pods -n cnpg-system
kubectl get crd clusters.postgresql.cnpg.io

2. Create K8s Secrets

# From the postgresql/ directory:
chmod +x create-secrets.sh
./create-secrets.sh ../../bootstrap/secrets

Alternatively, if you've already shredded the generated files, reconstruct from KeePassXC:

kubectl create secret generic net-kingdom-pg-keycloak-app \
  --namespace=databases \
  --from-literal=username=keycloak \
  --from-literal=password='<PG_KEYCLOAK_PASSWORD from KeePassXC>'

kubectl create secret generic net-kingdom-pg-privacyidea-app \
  --namespace=databases \
  --from-literal=username=privacyidea \
  --from-literal=password='<PI_DB_PASSWORD from KeePassXC>'

3. Deploy the cluster

kubectl apply -f cluster.yaml

Wait for cluster to become ready (this provisions PVCs and runs initdb — allow 2–3 minutes):

kubectl wait --for=condition=Ready cluster/net-kingdom-pg \
  -n databases --timeout=300s

Check status:

kubectl get cluster -n databases
kubectl describe cluster net-kingdom-pg -n databases
kubectl get pods -n databases

4. Verify databases and users

# Connect as superuser to verify setup
kubectl exec -it -n databases \
  $(kubectl get pod -n databases -l cnpg.io/cluster=net-kingdom-pg,role=primary -o name) \
  -- psql -U postgres

# In psql:
\l                          -- list databases
\du                         -- list roles
\q

Expected output: keycloak_db, privacyidea_db, roles keycloak and privacyidea.

5. Configure backup (when object storage is available)

Uncomment the backup: section in cluster.yaml and fill in the object store endpoint. Create the S3 credentials secret:

kubectl create secret generic net-kingdom-pg-backup-s3 \
  --namespace=databases \
  --from-literal=ACCESS_KEY_ID='<access key>' \
  --from-literal=SECRET_ACCESS_KEY='<secret key>'

Apply the updated cluster.yaml, then:

kubectl apply -f scheduled-backup.yaml

6. Run the restore drill

Mandatory before marking T03 done.

# Trigger a manual backup first
kubectl cnpg backup net-kingdom-pg -n databases

# Wait for backup to complete
kubectl get backup -n databases --watch

# Restore to a new cluster to verify
# (See CloudNativePG docs: kubectl cnpg restore or Cluster bootstrap.recovery)

7. Run the full verification script

chmod +x ../verify-t03.sh
../verify-t03.sh

Secrets reference

Secret name	Keys	Purpose
net-kingdom-pg-keycloak-app	username, password	Keycloak DB user (also bootstrap owner)
net-kingdom-pg-privacyidea-app	username, password	privacyIDEA DB user
net-kingdom-pg-backup-s3	ACCESS_KEY_ID, SECRET_ACCESS_KEY	Object store backup (optional until backup enabled)
net-kingdom-pg-superuser	auto-created by CNPG	PostgreSQL superuser (operator-managed)
net-kingdom-pg-app	auto-created by CNPG	Initial app user (unused — we use named secrets)

Notes

• cnpg.io/cluster: net-kingdom-pg label on pods is what the NetworkPolicies in T02 target. Do not rename the cluster without also updating netpol-databases.yaml.
• instances: 1 is intentional for dev/staging. Change to 3 before ThreePhoenix HA production deployment (requires at least 3 schedulable nodes).
• Password rotation: update the K8s Secret values and CNPG's managed.roles reconciler will apply the change at the next reconciliation cycle (within seconds).