feat(sso-mfa): T04 privacyIDEA manifests (NK-WP-0001-T04)

Deploy privacyIDEA (MFA core) in the mfa namespace: - pvc.yaml: privacyidea-data (5Gi) and privacyidea-logs (2Gi) - configmap.yaml: pi.cfg reading secrets from env vars - deployment.yaml: Deployment + ClusterIP Service (port 8080) - middleware.yaml: Traefik RateLimit + admin IP AllowList - ingress.yaml: pink.coulomb.social (portal + admin), pink-account.coulomb.social (self-service) - create-secrets.sh: creates privacyidea-config Secret - enckey-bootstrap.sh: post-deploy key extraction + DR Secrets - bootstrap-admin.sh: pi-admin, trigger-admin, privacyidea-trigger-admin Secret - verify-t04.sh: 8-section done-criteria checker Config points CP-NK-002 (pink.coulomb.social) and CP-NK-003 (pink-account.coulomb.social) registered in CONFIG.md. pink = PrivacyIDEA Net Knights (project mnemonic). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 01:22:41 +00:00
parent 87d85261ca
commit 1d94652ba1
13 changed files with 1213 additions and 7 deletions
--- a/CONFIG.md
+++ b/CONFIG.md
@@ -21,6 +21,44 @@ If yes to any of the above, don't add it here.
 | ID | Name | Value | Location(s) |
 |----|------|-------|-------------|
 | CP-NK-001 | ACME contact email | `bernd.worsch+netkingdom@gmail.com` | `sso-mfa/k8s/cert-manager/issuers.yaml:38` |
+| CP-NK-002 | privacyIDEA portal hostname | `pink.coulomb.social` | `sso-mfa/k8s/privacyidea/ingress.yaml` |
+| CP-NK-003 | privacyIDEA self-service hostname | `pink-account.coulomb.social` | `sso-mfa/k8s/privacyidea/ingress.yaml` |
+
+---
+
+## CP-NK-002 — privacyIDEA portal hostname
+
+**Value:** `pink.coulomb.social`
+**Set:** 2026-03-19
+**Set by:** worsch
+
+**Location(s):**
+- `sso-mfa/k8s/privacyidea/ingress.yaml` — all three Ingress `host` fields
+
+**Why non-default:** Subdomain prefix must be chosen by the operator; no naming
+convention existed in the repo before T04. `pink` = **P**rivacy**I**DEA
+**N**et **K**nights (project-specific mnemonic).
+
+**Scope:** TLS certificate, Traefik routing, and all references to the
+privacyIDEA public URL (including Keycloak Provider config in T05/T06).
+
+---
+
+## CP-NK-003 — privacyIDEA self-service portal hostname
+
+**Value:** `pink-account.coulomb.social`
+**Set:** 2026-03-19
+**Set by:** worsch
+
+**Location(s):**
+- `sso-mfa/k8s/privacyidea/ingress.yaml` — `privacyidea-account` Ingress `host` field
+
+**Why non-default:** Separate hostname for the self-service portal allows
+different firewall/allowlist rules from the admin portal. Follows the
+`<service>-account` naming convention used in the workplan design.
+
+**Scope:** TLS certificate and Traefik routing for the user-facing
+self-service token enrolment portal.

 ---

--- a/sso-mfa/k8s/network-policies/netpol-mfa.yaml
+++ b/sso-mfa/k8s/network-policies/netpol-mfa.yaml
@@ -21,7 +21,7 @@ spec:
    - Egress
 ---
 # ── Allow ingress from Traefik ───────────────────────────────────────────────
-# pi.yourdomain.com and pi-account.yourdomain.com both terminate at Traefik.
+# pink.coulomb.social and pink-account.coulomb.social both terminate at Traefik.
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
--- a/sso-mfa/k8s/privacyidea/README.md
+++ b/sso-mfa/k8s/privacyidea/README.md
@@ -0,0 +1,188 @@
+# T04 — Phase 3: Deploy privacyIDEA
+
+Phase 3 of NK-WP-0001: deploys the MFA core (privacyIDEA) in the `mfa` namespace.
+
+**Hostnames (config points CP-NK-002 / CP-NK-003):**
+- `pink.coulomb.social` — main portal and API
+- `pink-account.coulomb.social` — self-service token portal
+
+**Prerequisites:**
+- T02 complete: `mfa` namespace and NetworkPolicies applied, cert-manager running.
+- T03 complete: PostgreSQL cluster `net-kingdom-pg` in `databases` namespace is Ready.
+- T01 Phase 0a complete: `gen-secrets.sh` run, all secrets in KeePassXC.
+
+---
+
+## Apply order
+
+### Step 1 — Create the config Secret
+
+```bash
+cd sso-mfa/k8s/privacyidea
+chmod +x create-secrets.sh enckey-bootstrap.sh bootstrap-admin.sh
+./create-secrets.sh
+```
+
+Creates `privacyidea-config` in the `mfa` namespace (PI_SECRET_KEY, PI_PEPPER,
+PI_SQLALCHEMY_DATABASE_URI).
+
+---
+
+### Step 2 — Apply manifests
+
+```bash
+# From sso-mfa/k8s/privacyidea/
+kubectl apply -f pvc.yaml
+kubectl apply -f configmap.yaml
+kubectl apply -f middleware.yaml
+kubectl apply -f deployment.yaml
+kubectl apply -f ingress.yaml
+```
+
+**Wait for the pod to reach Running state** (DB migrations run on first start —
+allow up to 3 minutes):
+
+```bash
+kubectl get pods -n mfa -w
+# Expected: privacyidea-<hash>   1/1   Running
+```
+
+If the pod is stuck in `Init` or `CrashLoopBackOff`, check logs:
+```bash
+kubectl logs -n mfa -l app.kubernetes.io/name=privacyidea --previous
+```
+
+Common causes:
+- `privacyidea-config` Secret missing → run `create-secrets.sh`
+- PostgreSQL not reachable → verify T03, check NetworkPolicies
+- Wrong DB password → re-run `create-secrets.sh` with corrected secrets
+
+---
+
+### Step 3 — Extract key material and create DR Secrets
+
+Run **once** after the pod reaches `Running`:
+
+```bash
+./enckey-bootstrap.sh
+```
+
+This generates RSA audit keys (if not already created), extracts the encryption
+key and audit keys from the pod, and creates two K8s Secrets as disaster-recovery
+copies:
+- `privacyidea-enckey`
+- `privacyidea-auditkeys`
+
+Follow the printed instructions to store the key files in KeePassXC, then shred
+the local copies.
+
+---
+
+### Step 4 — Bootstrap admin accounts
+
+```bash
+./bootstrap-admin.sh
+```
+
+Creates `pi-admin` (full admin) and `trigger-admin` (triggerchallenge only).
+Also creates the `privacyidea-trigger-admin` K8s Secret used by Keycloak in T05.
+
+**Immediately after the script completes:**
+1. Log in to `https://pink.coulomb.social` as `pi-admin`.
+2. Navigate to **Users → pi-admin → Enroll token** and enroll a TOTP or hardware token.
+3. Log out, log back in — the MFA challenge must appear.
+4. Verify the `trigger-admin-rights` policy at Config → Policies.
+
+---
+
+### Step 5 — Verify
+
+```bash
+cd sso-mfa/k8s
+chmod +x verify-t04.sh
+./verify-t04.sh
+```
+
+---
+
+## Container port note
+
+The deployment uses `containerPort: 8080`. The official `privacyidea/privacyidea`
+image uses nginx internally; the default nginx port may be 80 depending on the
+image version.
+
+**If the pod starts but requests return "connection refused":**
+```bash
+# Check what port the container actually listens on:
+kubectl exec -n mfa <pod> -- ss -tlnp | grep LISTEN
+```
+
+If the container uses port 80, update:
+1. `deployment.yaml`: `containerPort: 80`, Service `targetPort: 80`
+2. `sso-mfa/k8s/network-policies/netpol-mfa.yaml`: `port: 80` in privacyIDEA rules
+3. Reapply both files.
+
+---
+
+## NetworkPolicy design
+
+privacyIDEA sits entirely behind the NetworkPolicies applied in T02 (netpol-mfa.yaml):
+
+| Source | Destination | Port | Purpose |
+|--------|-------------|------|---------|
+| Traefik (kube-system) | privacyIDEA (mfa) | 8080 | User-facing portal |
+| Keycloak (sso) | privacyIDEA (mfa) | 8080 | Provider API (triggerchallenge) |
+| privacyIDEA (mfa) | PostgreSQL (databases) | 5432 | Database |
+
+Outbound to anything other than PostgreSQL and kube-dns is denied.
+
+---
+
+## Post-deploy steps (after verify-t04.sh passes)
+
+### Rate limiting adjustment
+
+The default rate limit (20 req/min, burst 5) is conservative. If Keycloak's
+triggerchallenge calls trigger false positives (HTTP 429), raise the average
+in `middleware.yaml` and reapply. Alternatively, the Keycloak-to-PI path is
+cluster-internal and not subject to the Ingress middleware.
+
+### Admin WebUI IP restriction
+
+Update `middleware.yaml` `privacyidea-admin-allowlist.spec.ipAllowList.sourceRange`
+to your actual VPN/office CIDRs and reapply:
+
+```bash
+kubectl apply -f middleware.yaml
+```
+
+### Self-service portal
+
+Enable the self-service portal policy in privacyIDEA:
+Config → Policies → New policy:
+- Scope: `user`
+- Action: `enrollTOTP`, `enrollHOTP`, or token types you want users to manage
+- URL for self-service: `https://pink-account.coulomb.social`
+
+---
+
+## Disaster recovery
+
+If the `privacyidea-data` PVC is lost:
+
+1. Create a new PVC with `pvc.yaml`.
+2. Restore enckey and audit keys from KeePassXC:
+   ```bash
+   # Copy pi.enc and private/public.pem from KeePassXC into a temporary pod
+   kubectl run restore-helper --image=busybox -n mfa --restart=Never -- sleep 3600
+   kubectl cp ./pi.enc        mfa/restore-helper:/etc/privacyidea/enckey
+   kubectl cp ./private.pem   mfa/restore-helper:/etc/privacyidea/private.pem
+   kubectl cp ./public.pem    mfa/restore-helper:/etc/privacyidea/public.pem
+   kubectl delete pod -n mfa restore-helper
+   ```
+   Or, if the K8s Secrets survived (created by `enckey-bootstrap.sh`):
+   ```bash
+   kubectl get secret privacyidea-enckey -n mfa -o jsonpath='{.data.enckey}' | base64 -d > pi.enc
+   ```
+3. Restart the privacyIDEA deployment — it will run DB migrations and use the
+   restored key material.
--- a/sso-mfa/k8s/privacyidea/bootstrap-admin.sh
+++ b/sso-mfa/k8s/privacyidea/bootstrap-admin.sh
@@ -0,0 +1,162 @@
+#!/usr/bin/env bash
+# bootstrap-admin.sh — create pi-admin and trigger-admin in privacyIDEA
+#
+# Run AFTER enckey-bootstrap.sh (key material must exist before admin setup).
+#
+# What it does:
+#   1. Creates pi-admin — full admin (single-credential bootstrap moment).
+#   2. Creates trigger-admin — limited admin for Keycloak's triggerchallenge calls.
+#   3. Creates the trigger-admin policy (triggerchallenge right only, via REST API).
+#   4. Prints instructions to immediately enroll MFA for pi-admin.
+#
+# Usage:
+#   ./bootstrap-admin.sh [secrets-dir] [pi-url]
+#
+# <secrets-dir> default: ../../bootstrap/secrets
+# <pi-url>      default: https://pink.coulomb.social
+#
+# The script uses the in-cluster kubectl exec path for creating admin users
+# (avoids the need for a network route to pink.coulomb.social during bootstrap).
+# The trigger-admin policy is created via REST API, so the URL must be reachable.
+
+set -euo pipefail
+
+NAMESPACE="mfa"
+SECRETS_DIR="${1:-../../bootstrap/secrets}"
+PI_URL="${2:-https://pink.coulomb.social}"
+PI_ENV="$SECRETS_DIR/privacyidea/secrets.env"
+
+if [[ ! -f "$PI_ENV" ]]; then
+    echo "ERROR: $PI_ENV not found — run gen-secrets.sh first." >&2
+    exit 1
+fi
+
+PI_ADMIN_PASS=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_ADMIN_PASSWORD")
+if [[ -z "$PI_ADMIN_PASS" ]]; then
+    echo "ERROR: PI_ADMIN_PASSWORD not found in $PI_ENV" >&2
+    exit 1
+fi
+
+# trigger-admin gets its own random password (generated here, stored in KeePassXC).
+TRIGGER_PASS=$(openssl rand -base64 32 | tr -d '\n/+=' | head -c 40)
+
+# ── 1. Find running pod ───────────────────────────────────────────────────────
+PI_POD=$(kubectl get pod -n "$NAMESPACE" \
+    -l app.kubernetes.io/name=privacyidea \
+    --field-selector=status.phase=Running \
+    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
+
+if [[ -z "$PI_POD" ]]; then
+    echo "ERROR: no Running privacyIDEA pod found in namespace '$NAMESPACE'." >&2
+    exit 1
+fi
+echo "Using pod: $PI_POD"
+
+# ── 2. Create pi-admin (full admin) ──────────────────────────────────────────
+echo ""
+echo "Creating admin: pi-admin"
+if kubectl exec -n "$NAMESPACE" "$PI_POD" -- \
+    pi-manage admin add pi-admin --password "$PI_ADMIN_PASS" 2>&1 | grep -q "already exist"; then
+    echo "  pi-admin already exists — skipping."
+else
+    echo "  pi-admin created."
+fi
+
+# ── 3. Create trigger-admin (limited admin) ───────────────────────────────────
+echo ""
+echo "Creating admin: trigger-admin"
+if kubectl exec -n "$NAMESPACE" "$PI_POD" -- \
+    pi-manage admin add trigger-admin --password "$TRIGGER_PASS" 2>&1 | grep -q "already exist"; then
+    echo "  trigger-admin already exists — skipping password update."
+    echo "  WARNING: if you need to rotate trigger-admin's password, do so manually."
+else
+    echo "  trigger-admin created."
+fi
+
+# ── 4. Create K8s Secret for trigger-admin credentials ───────────────────────
+# Keycloak's privacyIDEA Provider reads the trigger-admin credentials from
+# a K8s Secret that is referenced in the Keycloak realm configuration (T05/T06).
+echo ""
+echo "Creating K8s Secret: privacyidea-trigger-admin (namespace: mfa)"
+kubectl create secret generic privacyidea-trigger-admin \
+    --namespace=mfa \
+    --from-literal=username=trigger-admin \
+    --from-literal=password="$TRIGGER_PASS" \
+    --dry-run=client -o yaml | kubectl apply -f -
+
+echo "  Done."
+
+# ── 5. Create trigger-admin policy via REST API ───────────────────────────────
+# This restricts trigger-admin to the triggerchallenge action only.
+# Requires pink.coulomb.social to be reachable.
+echo ""
+echo "Creating trigger-admin policy via REST API ($PI_URL)..."
+echo "  (Skip this step if $PI_URL is not yet reachable — do it via the WebUI instead.)"
+echo ""
+
+# Authenticate as pi-admin to get a bearer token.
+AUTH_RESPONSE=$(curl -sf -X POST "$PI_URL/auth" \
+    -H "Content-Type: application/json" \
+    -d "{\"username\":\"pi-admin\",\"password\":\"$PI_ADMIN_PASS\"}" || echo "CURL_FAILED")
+
+if [[ "$AUTH_RESPONSE" == "CURL_FAILED" ]]; then
+    echo "  Could not reach $PI_URL — skipping REST policy creation."
+    echo "  Create the trigger-admin policy manually (see README.md — Post-deploy steps)."
+else
+    PI_TOKEN=$(echo "$AUTH_RESPONSE" | python3 -c \
+        "import sys,json; print(json.load(sys.stdin)['result']['value']['token'])" 2>/dev/null || echo "")
+
+    if [[ -z "$PI_TOKEN" ]]; then
+        echo "  ERROR: could not extract auth token from response." >&2
+        echo "  Create the trigger-admin policy manually (see README.md)." >&2
+    else
+        # Create policy: trigger-admin can only call triggerchallenge.
+        POLICY_RESP=$(curl -sf -X POST "$PI_URL/policy/trigger-admin-rights" \
+            -H "Authorization: $PI_TOKEN" \
+            -H "Content-Type: application/json" \
+            -d '{
+                "scope": "admin",
+                "action": "triggerchallenge",
+                "adminrealm": "*",
+                "adminuser": "trigger-admin",
+                "realm": "*",
+                "priority": 1
+            }' || echo "CURL_FAILED")
+
+        if [[ "$POLICY_RESP" == "CURL_FAILED" ]]; then
+            echo "  WARNING: policy creation request failed." >&2
+            echo "  Create the trigger-admin policy manually (see README.md)." >&2
+        else
+            echo "  Policy 'trigger-admin-rights' created."
+        fi
+    fi
+fi
+
+# ── 6. Summary ────────────────────────────────────────────────────────────────
+echo ""
+echo "════════════════════════════════════════════════════════════"
+echo "  Admin bootstrap complete."
+echo "════════════════════════════════════════════════════════════"
+echo ""
+echo "CRITICAL — Do these steps NOW:"
+echo ""
+echo "  1. Store trigger-admin password in KeePassXC:"
+echo "       KeePassXC group: net-kingdom/privacyIDEA"
+echo "       Entry: trigger-admin  →  username=trigger-admin  password=<below>"
+echo "       trigger-admin password: $TRIGGER_PASS"
+echo "     Then shred this terminal history."
+echo ""
+echo "  2. Log in to the privacyIDEA WebUI as pi-admin:"
+echo "       $PI_URL"
+echo "     Enroll MFA for pi-admin IMMEDIATELY (TOTP or hardware token)."
+echo "     Until MFA is enrolled, pi-admin has only password authentication."
+echo ""
+echo "  3. Verify the trigger-admin policy:"
+echo "       WebUI → Config → Policies → trigger-admin-rights"
+echo "       Scope: admin  Action: triggerchallenge  AdminUser: trigger-admin"
+echo "     If it was not created automatically, create it here."
+echo ""
+echo "  4. Test admin MFA:"
+echo "       Log out, log back in as pi-admin — MFA challenge must appear."
+echo ""
+echo "Next step: run ../verify-t04.sh"
--- a/sso-mfa/k8s/privacyidea/configmap.yaml
+++ b/sso-mfa/k8s/privacyidea/configmap.yaml
@@ -0,0 +1,45 @@
+# ConfigMap: privacyidea-cfg
+#
+# Provides /etc/privacyidea/pi.cfg (the privacyIDEA Python config file).
+# Mounted as a subPath into the privacyidea-data PVC, so it overlays just
+# that one file while the rest of /etc/privacyidea/ remains on the PVC.
+#
+# Sensitive values (SECRET_KEY, PI_PEPPER, SQLALCHEMY_DATABASE_URI) are
+# injected as environment variables from the privacyidea-config Secret.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: privacyidea-cfg
+  namespace: mfa
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+data:
+  pi.cfg: |
+    # /etc/privacyidea/pi.cfg
+    # Sensitive values injected via environment (privacyidea-config Secret).
+    import os
+
+    # Flask session secret — never type this; comes from vault/KeePassXC.
+    SECRET_KEY = os.environ["PI_SECRET_KEY"]
+
+    # Password hashing pepper — added to all hashed passwords.
+    PI_PEPPER = os.environ["PI_PEPPER"]
+
+    # Encryption key for token secrets (auto-generated on first start if missing).
+    PI_ENCFILE = "/etc/privacyidea/enckey"
+
+    # Audit log RSA signing keys (generated by pi-manage create_audit_keys).
+    PI_AUDIT_KEY_PRIVATE = "/etc/privacyidea/private.pem"
+    PI_AUDIT_KEY_PUBLIC  = "/etc/privacyidea/public.pem"
+
+    # Database connection (full URI, password from vault/KeePassXC).
+    SQLALCHEMY_DATABASE_URI = os.environ["PI_SQLALCHEMY_DATABASE_URI"]
+
+    # Application logging.
+    PI_LOGFILE  = "/var/log/privacyidea/privacyidea.log"
+    PI_LOGLEVEL = 20   # 10=DEBUG  20=INFO  30=WARNING  40=ERROR
+
+    # Scripts folder for event handlers (leave default unless customising).
+    PI_SCRIPT_FOLDER = "/etc/privacyidea/scripts"
--- a/sso-mfa/k8s/privacyidea/create-secrets.sh
+++ b/sso-mfa/k8s/privacyidea/create-secrets.sh
@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# create-secrets.sh — create the privacyidea-config K8s Secret
+#
+# Usage:
+#   ./create-secrets.sh [secrets-dir]
+#
+# <secrets-dir> is the output directory from sso-mfa/bootstrap/gen-secrets.sh
+# (default: ../../bootstrap/secrets).
+#
+# Creates ONE Secret in the mfa namespace:
+#   privacyidea-config   — PI_SECRET_KEY, PI_PEPPER, PI_SQLALCHEMY_DATABASE_URI
+#
+# This secret must exist before applying deployment.yaml.
+#
+# The enckey and auditkey Secrets (privacyidea-enckey, privacyidea-auditkeys)
+# are created separately by enckey-bootstrap.sh AFTER the first pod start,
+# because those keys are auto-generated by the container on first run.
+#
+# Re-run this script if you rotate PI_SECRET_KEY or PI_PEPPER in KeePassXC.
+
+set -euo pipefail
+
+SECRETS_DIR="${1:-../../bootstrap/secrets}"
+PI_ENV="$SECRETS_DIR/privacyidea/secrets.env"
+
+if [[ ! -d "$SECRETS_DIR" ]]; then
+    echo "ERROR: secrets directory not found: $SECRETS_DIR" >&2
+    echo "Run sso-mfa/bootstrap/gen-secrets.sh first." >&2
+    exit 1
+fi
+
+if [[ ! -f "$PI_ENV" ]]; then
+    echo "ERROR: $PI_ENV not found" >&2
+    exit 1
+fi
+
+# Read values from the generated env file in a subshell to avoid polluting env.
+PI_SECRET_KEY=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_SECRET_KEY")
+PI_PEPPER=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_PEPPER")
+PI_DB_PASSWORD=$(bash -c "source '$PI_ENV' 2>/dev/null; echo \$PI_DB_PASSWORD")
+
+if [[ -z "$PI_SECRET_KEY" || -z "$PI_PEPPER" || -z "$PI_DB_PASSWORD" ]]; then
+    echo "ERROR: could not read PI_SECRET_KEY, PI_PEPPER, or PI_DB_PASSWORD from $PI_ENV" >&2
+    echo "Check that gen-secrets.sh ran successfully." >&2
+    exit 1
+fi
+
+# Construct the SQLAlchemy database URI.
+# CloudNativePG read-write service: net-kingdom-pg-rw.databases.svc.cluster.local
+PI_DB_URI="postgresql://privacyidea:${PI_DB_PASSWORD}@net-kingdom-pg-rw.databases.svc.cluster.local:5432/privacyidea_db"
+
+echo "Creating K8s Secret: privacyidea-config (namespace: mfa)"
+kubectl create secret generic privacyidea-config \
+    --namespace=mfa \
+    --from-literal=PI_SECRET_KEY="$PI_SECRET_KEY" \
+    --from-literal=PI_PEPPER="$PI_PEPPER" \
+    --from-literal=PI_SQLALCHEMY_DATABASE_URI="$PI_DB_URI" \
+    --dry-run=client -o yaml | kubectl apply -f -
+
+echo ""
+echo "Done. Secret privacyidea-config created in namespace: mfa"
+echo ""
+echo "Next:"
+echo "  Apply manifests (see README.md apply order)."
+echo "  After the pod is Running, run: ./enckey-bootstrap.sh"
--- a/sso-mfa/k8s/privacyidea/deployment.yaml
+++ b/sso-mfa/k8s/privacyidea/deployment.yaml
@@ -0,0 +1,152 @@
+# Deployment + Service — privacyIDEA (namespace: mfa)
+#
+# Prerequisites (apply in order):
+#   1. pvc.yaml          — privacyidea-data and privacyidea-logs PVCs
+#   2. configmap.yaml    — privacyidea-cfg (pi.cfg template)
+#   3. create-secrets.sh — privacyidea-config Secret (PI_SECRET_KEY, PI_PEPPER, DB URI)
+#   4. This file
+#
+# After first pod starts successfully:
+#   5. enckey-bootstrap.sh — extract enckey + audit keys, create DR Secrets
+#   6. bootstrap-admin.sh  — create pi-admin (+ MFA enrolment) and trigger-admin
+#
+# Container port: 8080.
+# The official privacyidea/privacyidea image uses nginx internally.
+# If the image you pull listens on port 80 instead of 8080:
+#   - Change containerPort below to 80
+#   - Change the Service targetPort to 80
+#   - Update sso-mfa/k8s/network-policies/netpol-mfa.yaml ports to 80
+#   - Reapply both files
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: privacyidea
+  namespace: mfa
+  labels:
+    app.kubernetes.io/name: privacyidea
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: privacyidea
+  strategy:
+    type: Recreate   # single-node — avoid split-brain on PVC
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: privacyidea
+        app.kubernetes.io/part-of: net-kingdom-sso-mfa
+        net-kingdom/component: mfa
+    spec:
+      # ── Security context ───────────────────────────────────────────────────
+      securityContext:
+        runAsNonRoot: false   # privacyIDEA nginx needs root to bind port; revisit
+        fsGroup: 999          # privacyidea group inside container
+
+      # ── Init: ensure log dir exists and has correct permissions ───────────
+      initContainers:
+        - name: init-logdir
+          image: busybox:1.36
+          command: ["sh", "-c", "mkdir -p /var/log/privacyidea && chmod 777 /var/log/privacyidea"]
+          volumeMounts:
+            - name: logs
+              mountPath: /var/log/privacyidea
+
+      containers:
+        - name: privacyidea
+          # Pin to a specific release; update via image update policy.
+          # Check https://hub.docker.com/r/privacyidea/privacyidea for latest stable.
+          image: privacyidea/privacyidea:3.12
+          imagePullPolicy: IfNotPresent
+
+          ports:
+            - name: http
+              containerPort: 8080
+              protocol: TCP
+
+          # ── Environment — sensitive values from Secret ──────────────────
+          envFrom:
+            - secretRef:
+                name: privacyidea-config
+
+          # ── Volume mounts ───────────────────────────────────────────────
+          volumeMounts:
+            # pi.cfg overlaid into the data PVC as a single file (subPath).
+            - name: config
+              mountPath: /etc/privacyidea/pi.cfg
+              subPath: pi.cfg
+              readOnly: true
+            # Data PVC: enckey, audit keys, scripts, and other PI runtime files.
+            - name: data
+              mountPath: /etc/privacyidea
+            # Logs PVC: persistent application logs.
+            - name: logs
+              mountPath: /var/log/privacyidea
+
+          # ── Probes ──────────────────────────────────────────────────────
+          # Startup probe: give PI up to 3 min to run DB migrations on first boot.
+          startupProbe:
+            tcpSocket:
+              port: 8080
+            initialDelaySeconds: 15
+            periodSeconds: 10
+            failureThreshold: 18   # 18 × 10s = 3 min
+          livenessProbe:
+            tcpSocket:
+              port: 8080
+            initialDelaySeconds: 0
+            periodSeconds: 15
+            failureThreshold: 3
+          readinessProbe:
+            httpGet:
+              path: /token/
+              port: 8080
+            initialDelaySeconds: 0
+            periodSeconds: 10
+            failureThreshold: 3
+
+          # ── Resources ───────────────────────────────────────────────────
+          # Raise limits for production; privacyIDEA handles crypto and DB queries.
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "256Mi"
+            limits:
+              cpu: "500m"
+              memory: "512Mi"
+
+      # ── Volumes ─────────────────────────────────────────────────────────
+      volumes:
+        - name: config
+          configMap:
+            name: privacyidea-cfg
+        - name: data
+          persistentVolumeClaim:
+            claimName: privacyidea-data
+        - name: logs
+          persistentVolumeClaim:
+            claimName: privacyidea-logs
+
+---
+# Service — ClusterIP; Traefik and Keycloak reach privacyIDEA via this.
+apiVersion: v1
+kind: Service
+metadata:
+  name: privacyidea
+  namespace: mfa
+  labels:
+    app.kubernetes.io/name: privacyidea
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: privacyidea
+  ports:
+    - name: http
+      port: 8080
+      targetPort: 8080
+      protocol: TCP
--- a/sso-mfa/k8s/privacyidea/enckey-bootstrap.sh
+++ b/sso-mfa/k8s/privacyidea/enckey-bootstrap.sh
@@ -0,0 +1,121 @@
+#!/usr/bin/env bash
+# enckey-bootstrap.sh — extract privacyIDEA key material and create DR Secrets
+#
+# Run ONCE after the first successful pod start (deployment.yaml applied and Running).
+#
+# What it does:
+#   1. Finds the running privacyIDEA pod.
+#   2. Generates RSA audit keys inside the pod if they do not exist yet.
+#   3. Extracts: enckey, private.pem, public.pem  →  local files.
+#   4. Creates K8s Secrets privacyidea-enckey and privacyidea-auditkeys.
+#      These are disaster-recovery copies — the authoritative key material
+#      lives in the privacyidea-data PVC.
+#   5. Prints instructions to copy the extracted files into KeePassXC.
+#
+# Usage:
+#   ./enckey-bootstrap.sh [secrets-dir]
+#
+# <secrets-dir> is where local copies are written (default: ../../bootstrap/secrets).
+# Files in secrets-dir are sensitive — shred them after entering into KeePassXC.
+#
+# Re-running is safe: kubectl apply is idempotent; pi-manage create_audit_keys
+# does not overwrite existing keys.
+
+set -euo pipefail
+
+NAMESPACE="mfa"
+SECRETS_DIR="${1:-../../bootstrap/secrets}"
+PI_DIR="$SECRETS_DIR/privacyidea"
+
+mkdir -p "$PI_DIR"
+
+# ── 1. Find the running pod ───────────────────────────────────────────────────
+echo "Looking for running privacyIDEA pod in namespace: $NAMESPACE"
+PI_POD=$(kubectl get pod -n "$NAMESPACE" \
+    -l app.kubernetes.io/name=privacyidea \
+    --field-selector=status.phase=Running \
+    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
+
+if [[ -z "$PI_POD" ]]; then
+    echo "ERROR: no Running privacyIDEA pod found in namespace '$NAMESPACE'." >&2
+    echo "  Apply the manifests first and wait for the pod to reach Running state:" >&2
+    echo "    kubectl get pods -n $NAMESPACE -w" >&2
+    exit 1
+fi
+
+echo "Found pod: $PI_POD"
+
+# ── 2. Generate audit keys if missing ────────────────────────────────────────
+echo ""
+echo "Checking for audit keys..."
+if kubectl exec -n "$NAMESPACE" "$PI_POD" -- test -f /etc/privacyidea/private.pem 2>/dev/null; then
+    echo "  Audit keys already exist — skipping generation."
+else
+    echo "  Generating audit keys (pi-manage create_audit_keys)..."
+    kubectl exec -n "$NAMESPACE" "$PI_POD" -- pi-manage create_audit_keys
+    echo "  Done."
+fi
+
+# ── 3. Check enckey exists (auto-generated by PI on first DB init) ────────────
+echo ""
+echo "Checking for encryption key..."
+if ! kubectl exec -n "$NAMESPACE" "$PI_POD" -- test -f /etc/privacyidea/enckey 2>/dev/null; then
+    echo "  enckey not found — generating (pi-manage create_enckey)..."
+    kubectl exec -n "$NAMESPACE" "$PI_POD" -- pi-manage create_enckey
+    echo "  Done."
+else
+    echo "  enckey exists."
+fi
+
+# ── 4. Extract key material to local files ────────────────────────────────────
+echo ""
+echo "Extracting key material..."
+
+kubectl exec -n "$NAMESPACE" "$PI_POD" -- \
+    cat /etc/privacyidea/enckey > "$PI_DIR/pi.enc"
+echo "  Wrote: $PI_DIR/pi.enc"
+
+kubectl exec -n "$NAMESPACE" "$PI_POD" -- \
+    cat /etc/privacyidea/private.pem > "$PI_DIR/private.pem"
+echo "  Wrote: $PI_DIR/private.pem"
+
+kubectl exec -n "$NAMESPACE" "$PI_POD" -- \
+    cat /etc/privacyidea/public.pem > "$PI_DIR/public.pem"
+echo "  Wrote: $PI_DIR/public.pem"
+
+# ── 5. Create K8s Secrets (DR copies) ────────────────────────────────────────
+echo ""
+echo "Creating K8s Secret: privacyidea-enckey (namespace: $NAMESPACE)"
+kubectl create secret generic privacyidea-enckey \
+    --namespace="$NAMESPACE" \
+    --from-file=enckey="$PI_DIR/pi.enc" \
+    --dry-run=client -o yaml | kubectl apply -f -
+
+echo "Creating K8s Secret: privacyidea-auditkeys (namespace: $NAMESPACE)"
+kubectl create secret generic privacyidea-auditkeys \
+    --namespace="$NAMESPACE" \
+    --from-file=private.pem="$PI_DIR/private.pem" \
+    --from-file=public.pem="$PI_DIR/public.pem" \
+    --dry-run=client -o yaml | kubectl apply -f -
+
+# ── 6. Summary and KeePassXC instructions ─────────────────────────────────────
+echo ""
+echo "════════════════════════════════════════════════════════════"
+echo "  Key material extracted and K8s Secrets created."
+echo "════════════════════════════════════════════════════════════"
+echo ""
+echo "IMPORTANT — Store the extracted files in KeePassXC NOW:"
+echo ""
+echo "  KeePassXC group: net-kingdom/privacyIDEA"
+echo "  Entry: PI_ENCFILE"
+echo "    → Attach $PI_DIR/pi.enc as a binary attachment."
+echo "  Entry: Audit Keys"
+echo "    → Attach $PI_DIR/private.pem and $PI_DIR/public.pem"
+echo ""
+echo "  Then shred the local copies:"
+echo "    find '$PI_DIR' -name '*.enc' -o -name '*.pem' | xargs shred -u"
+echo ""
+echo "DR restore: if the privacyidea-data PVC is lost, copy these files"
+echo "  back into a fresh PVC before starting the pod."
+echo ""
+echo "Next step: run ./bootstrap-admin.sh"
--- a/sso-mfa/k8s/privacyidea/ingress.yaml
+++ b/sso-mfa/k8s/privacyidea/ingress.yaml
@@ -0,0 +1,121 @@
+# Ingress — privacyIDEA (namespace: mfa)
+#
+# pink.coulomb.social       — main portal (login, self-service, API)
+# pink-account.coulomb.social — self-service token portal
+#
+# Both hostnames resolve to the same privacyIDEA Service.
+# privacyIDEA serves the self-service portal at /account/ when the
+# "privacyideaserver" policy for self-service is enabled (configured
+# in bootstrap-admin.sh / T04 README).
+#
+# TLS: cert-manager issues certificates via the letsencrypt-prod ClusterIssuer
+# (T02). Public DNS for both hostnames must resolve to the cluster's external IP
+# before cert-manager can complete the ACME HTTP-01 challenge.
+#
+# Rate limiting: the privacyidea-rate-limit middleware (middleware.yaml) is
+# applied to pink.coulomb.social. Admin paths are further restricted by
+# privacyidea-admin-allowlist applied in the separate /admin Ingress below.
+#
+# Config points (see CONFIG.md):
+#   CP-NK-002  pink.coulomb.social
+#   CP-NK-003  pink-account.coulomb.social
+
+# ── Main portal — pink.coulomb.social ────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: privacyidea
+  namespace: mfa
+  labels:
+    app.kubernetes.io/name: privacyidea
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    # Rate-limit middleware (Traefik v3 format; see middleware.yaml for v2 note).
+    traefik.ingress.kubernetes.io/router.middlewares: "mfa-privacyidea-rate-limit@kubernetescrd"
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: pink.coulomb.social
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: privacyidea
+                port:
+                  number: 8080
+  tls:
+    - secretName: pink-tls
+      hosts:
+        - pink.coulomb.social
+---
+# ── Admin WebUI — pink.coulomb.social/admin — restricted to VPN/office IPs ──
+# Separate Ingress so the admin-allowlist middleware applies only to /admin/*.
+# The main Ingress above already handles / (which includes /admin/ by prefix);
+# this Ingress's more-specific /admin path takes precedence in Traefik routing.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: privacyidea-admin
+  namespace: mfa
+  labels:
+    app.kubernetes.io/name: privacyidea
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    # Both rate-limit AND IP allowlist for admin paths.
+    traefik.ingress.kubernetes.io/router.middlewares: >-
+      mfa-privacyidea-rate-limit@kubernetescrd,
+      mfa-privacyidea-admin-allowlist@kubernetescrd
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: pink.coulomb.social
+      http:
+        paths:
+          - path: /admin
+            pathType: Prefix
+            backend:
+              service:
+                name: privacyidea
+                port:
+                  number: 8080
+  tls:
+    - secretName: pink-tls
+      hosts:
+        - pink.coulomb.social
+---
+# ── Self-service portal — pink-account.coulomb.social ────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: privacyidea-account
+  namespace: mfa
+  labels:
+    app.kubernetes.io/name: privacyidea
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    traefik.ingress.kubernetes.io/router.middlewares: "mfa-privacyidea-rate-limit@kubernetescrd"
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: pink-account.coulomb.social
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: privacyidea
+                port:
+                  number: 8080
+  tls:
+    - secretName: pink-account-tls
+      hosts:
+        - pink-account.coulomb.social
--- a/sso-mfa/k8s/privacyidea/middleware.yaml
+++ b/sso-mfa/k8s/privacyidea/middleware.yaml
@@ -0,0 +1,56 @@
+# Traefik Middlewares for privacyIDEA (namespace: mfa)
+#
+# Middleware names follow the pattern referenced in ingress.yaml annotations:
+#   mfa-privacyidea-rate-limit@kubernetescrd
+#   mfa-privacyidea-admin-allowlist@kubernetescrd
+#
+# Traefik API version:
+#   Traefik v3 (K3s >= 1.30): traefik.io/v1alpha1
+#   Traefik v2 (K3s <  1.30): traefik.containo.us/v1alpha1
+# Check: kubectl get middleware -n mfa -o yaml | grep apiVersion
+# Update both documents below if you need the v2 apiVersion.
+
+# ── Rate limit — all PI endpoints ────────────────────────────────────────────
+# Applies globally to pink.coulomb.social.
+# Primary protection for /validate/check (OTP verification) and /auth.
+# 20 requests/minute per client IP; burst of 5 allowed.
+# Adjust average/burst upward if legitimate automation (e.g. Keycloak Provider)
+# triggers false positives — or exclude the Keycloak source IP at network level.
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: privacyidea-rate-limit
+  namespace: mfa
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+spec:
+  rateLimit:
+    average: 20
+    period: 1m
+    burst: 5
+---
+# ── Admin path allowlist — restrict WebUI to internal/VPN IPs ────────────────
+# Applied to the /admin/* Ingress (see ingress.yaml — separate Ingress for /admin/).
+#
+# ADJUST sourceRange to your actual VPN / office CIDR(s) before going live.
+# Leaving RFC-1918 ranges here is only a dev/staging default.
+#
+# Traefik v3 uses ipAllowList; Traefik v2 uses ipWhiteList.
+# Check your Traefik version and update accordingly.
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: privacyidea-admin-allowlist
+  namespace: mfa
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+spec:
+  ipAllowList:
+    # EDIT: replace with your VPN/office CIDRs (see CONFIG.md for the pattern).
+    # Example VPN: "10.8.0.0/24"
+    sourceRange:
+      - "10.0.0.0/8"
+      - "172.16.0.0/12"
+      - "192.168.0.0/16"
--- a/sso-mfa/k8s/privacyidea/pvc.yaml
+++ b/sso-mfa/k8s/privacyidea/pvc.yaml
@@ -0,0 +1,40 @@
+# PersistentVolumeClaims for privacyIDEA (namespace: mfa)
+#
+# privacyidea-data — /etc/privacyidea/
+#   Holds: enckey, audit signing keys, and any runtime PI config.
+#   PI auto-generates missing key material here on first start.
+#   Run enckey-bootstrap.sh after first deploy to extract keys into
+#   KeePassXC and K8s Secrets (disaster recovery copies).
+#
+# privacyidea-logs — /var/log/privacyidea/
+#   Application log files; separate PVC keeps data PVC clean.
+#
+# Adjust storage sizes before production deployment.
+
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: privacyidea-data
+  namespace: mfa
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+spec:
+  accessModes: [ReadWriteOnce]
+  resources:
+    requests:
+      storage: 5Gi
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: privacyidea-logs
+  namespace: mfa
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: mfa
+spec:
+  accessModes: [ReadWriteOnce]
+  resources:
+    requests:
+      storage: 2Gi
--- a/sso-mfa/k8s/verify-t04.sh
+++ b/sso-mfa/k8s/verify-t04.sh
@@ -0,0 +1,217 @@
+#!/usr/bin/env bash
+# verify-t04.sh — verify NK-WP-0001-T04 done-criteria
+#
+# Checks:
+#   1. privacyIDEA pod is Running in the mfa namespace
+#   2. privacyidea Service exists
+#   3. Traefik Middlewares exist
+#   4. Ingress resources exist with correct hostnames
+#   5. TLS certificates issued by cert-manager
+#   6. Required K8s Secrets are present
+#   7. PVCs are Bound
+#   8. Enckey and auditkey Secrets present (from enckey-bootstrap.sh)
+#
+# Usage:
+#   chmod +x verify-t04.sh
+#   ./verify-t04.sh
+
+set -euo pipefail
+
+NAMESPACE="mfa"
+PASS=0
+FAIL=0
+WARN=0
+
+pass() { echo "  [PASS] $1"; ((PASS++)); }
+fail() { echo "  [FAIL] $1"; ((FAIL++)); }
+warn() { echo "  [WARN] $1"; ((WARN++)); }
+
+section() { echo ""; echo "── $1 ──────────────────────────────────────"; }
+
+# ── 1. privacyIDEA pod ────────────────────────────────────────────────────────
+section "1. privacyIDEA pod (namespace: $NAMESPACE)"
+
+PI_POD=$(kubectl get pod -n "$NAMESPACE" \
+    -l app.kubernetes.io/name=privacyidea \
+    --field-selector=status.phase=Running \
+    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
+
+if [[ -n "$PI_POD" ]]; then
+    pass "Pod Running: $PI_POD"
+
+    READY=$(kubectl get pod -n "$NAMESPACE" "$PI_POD" \
+        -o jsonpath='{.status.containerStatuses[0].ready}' 2>/dev/null || echo "false")
+    if [[ "$READY" == "true" ]]; then
+        pass "Pod readiness probe passing"
+    else
+        fail "Pod is Running but not Ready (probe failing — check logs)"
+    fi
+else
+    PENDING=$(kubectl get pod -n "$NAMESPACE" \
+        -l app.kubernetes.io/name=privacyidea \
+        -o name 2>/dev/null | wc -l || echo 0)
+    if [[ "$PENDING" -gt 0 ]]; then
+        fail "privacyIDEA pod(s) exist but none are Running (check kubectl describe pod -n $NAMESPACE)"
+    else
+        fail "No privacyIDEA pods found in namespace $NAMESPACE — apply deployment.yaml"
+    fi
+fi
+
+# ── 2. Service ────────────────────────────────────────────────────────────────
+section "2. Service"
+
+if kubectl get service privacyidea -n "$NAMESPACE" &>/dev/null; then
+    pass "Service privacyidea exists"
+    PORT=$(kubectl get service privacyidea -n "$NAMESPACE" \
+        -o jsonpath='{.spec.ports[0].port}' 2>/dev/null || echo "")
+    if [[ "$PORT" == "8080" ]]; then
+        pass "Service port: 8080"
+    else
+        warn "Service port is $PORT (expected 8080 — check container port and netpol)"
+    fi
+else
+    fail "Service privacyidea not found in namespace $NAMESPACE"
+fi
+
+# ── 3. Traefik Middlewares ────────────────────────────────────────────────────
+section "3. Traefik Middlewares"
+
+for mw in privacyidea-rate-limit privacyidea-admin-allowlist; do
+    if kubectl get middleware "$mw" -n "$NAMESPACE" &>/dev/null; then
+        pass "Middleware $mw exists"
+    else
+        fail "Middleware $mw not found — apply middleware.yaml"
+    fi
+done
+
+# ── 4. Ingress resources ──────────────────────────────────────────────────────
+section "4. Ingress resources"
+
+for ing in privacyidea privacyidea-admin privacyidea-account; do
+    if kubectl get ingress "$ing" -n "$NAMESPACE" &>/dev/null; then
+        pass "Ingress $ing exists"
+    else
+        fail "Ingress $ing not found — apply ingress.yaml"
+    fi
+done
+
+# Verify hostnames in the main Ingress
+PI_HOST=$(kubectl get ingress privacyidea -n "$NAMESPACE" \
+    -o jsonpath='{.spec.rules[0].host}' 2>/dev/null || echo "")
+if [[ "$PI_HOST" == "pink.coulomb.social" ]]; then
+    pass "Ingress host: pink.coulomb.social"
+else
+    fail "Ingress host is '$PI_HOST' (expected pink.coulomb.social)"
+fi
+
+ACCOUNT_HOST=$(kubectl get ingress privacyidea-account -n "$NAMESPACE" \
+    -o jsonpath='{.spec.rules[0].host}' 2>/dev/null || echo "")
+if [[ "$ACCOUNT_HOST" == "pink-account.coulomb.social" ]]; then
+    pass "Self-service host: pink-account.coulomb.social"
+else
+    fail "Self-service host is '$ACCOUNT_HOST' (expected pink-account.coulomb.social)"
+fi
+
+# ── 5. TLS certificates ───────────────────────────────────────────────────────
+section "5. TLS certificates"
+
+for cert_secret in pink-tls pink-account-tls; do
+    if kubectl get secret "$cert_secret" -n "$NAMESPACE" &>/dev/null; then
+        # Check cert-manager Ready condition
+        CERT_NAME="${cert_secret%-tls}"
+        CERT_READY=$(kubectl get certificate "$CERT_NAME" -n "$NAMESPACE" \
+            -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "")
+        if [[ "$CERT_READY" == "True" ]]; then
+            pass "Certificate $cert_name is Ready (TLS secret $cert_secret exists)"
+        else
+            warn "TLS secret $cert_secret exists but certificate status is not Ready (DNS propagation pending?)"
+        fi
+    else
+        warn "TLS secret $cert_secret not yet issued (cert-manager pending — check DNS and ACME)"
+    fi
+done
+
+# ── 6. K8s Secrets ────────────────────────────────────────────────────────────
+section "6. K8s Secrets (namespace: $NAMESPACE)"
+
+for secret in privacyidea-config; do
+    if kubectl get secret "$secret" -n "$NAMESPACE" &>/dev/null; then
+        pass "Secret $secret exists"
+    else
+        fail "Secret $secret not found — run create-secrets.sh"
+    fi
+done
+
+# DR secrets — created by enckey-bootstrap.sh (may not exist yet on first run)
+for secret in privacyidea-enckey privacyidea-auditkeys privacyidea-trigger-admin; do
+    if kubectl get secret "$secret" -n "$NAMESPACE" &>/dev/null; then
+        pass "Secret $secret exists"
+    else
+        warn "Secret $secret not found — run enckey-bootstrap.sh / bootstrap-admin.sh"
+    fi
+done
+
+# ── 7. PVCs ───────────────────────────────────────────────────────────────────
+section "7. PersistentVolumeClaims"
+
+for pvc in privacyidea-data privacyidea-logs; do
+    STATUS=$(kubectl get pvc "$pvc" -n "$NAMESPACE" \
+        -o jsonpath='{.status.phase}' 2>/dev/null || echo "not found")
+    if [[ "$STATUS" == "Bound" ]]; then
+        pass "PVC $pvc: Bound"
+    elif [[ "$STATUS" == "not found" ]]; then
+        fail "PVC $pvc not found — apply pvc.yaml"
+    else
+        fail "PVC $pvc status: $STATUS (expected Bound)"
+    fi
+done
+
+# ── 8. Key material check ─────────────────────────────────────────────────────
+section "8. Key material (inside pod)"
+
+if [[ -n "$PI_POD" ]]; then
+    if kubectl exec -n "$NAMESPACE" "$PI_POD" -- test -f /etc/privacyidea/enckey 2>/dev/null; then
+        pass "enckey present in pod"
+    else
+        warn "enckey not found in pod — run enckey-bootstrap.sh (or wait for PI to generate on first DB init)"
+    fi
+
+    if kubectl exec -n "$NAMESPACE" "$PI_POD" -- test -f /etc/privacyidea/private.pem 2>/dev/null; then
+        pass "audit private.pem present in pod"
+    else
+        warn "audit private.pem not found — run enckey-bootstrap.sh"
+    fi
+
+    if kubectl exec -n "$NAMESPACE" "$PI_POD" -- \
+        pi-manage admin list 2>/dev/null | grep -q "pi-admin"; then
+        pass "Admin user pi-admin exists"
+    else
+        warn "pi-admin not found — run bootstrap-admin.sh"
+    fi
+
+    if kubectl exec -n "$NAMESPACE" "$PI_POD" -- \
+        pi-manage admin list 2>/dev/null | grep -q "trigger-admin"; then
+        pass "Admin user trigger-admin exists"
+    else
+        warn "trigger-admin not found — run bootstrap-admin.sh"
+    fi
+else
+    warn "Skipping key material checks — no running pod"
+fi
+
+# ── Summary ───────────────────────────────────────────────────────────────────
+echo ""
+echo "════════════════════════════════════════════════════════════"
+echo "  T04 verification: PASS=$PASS  WARN=$WARN  FAIL=$FAIL"
+echo "════════════════════════════════════════════════════════════"
+
+if [[ "$FAIL" -gt 0 ]]; then
+    echo "  Result: INCOMPLETE — resolve FAIL items before proceeding to T05"
+    exit 1
+elif [[ "$WARN" -gt 0 ]]; then
+    echo "  Result: PARTIAL — T04 core is up; WARN items should be resolved before T05"
+    exit 0
+else
+    echo "  Result: COMPLETE — T04 done-criteria met; proceed to T05 (Keycloak)"
+    exit 0
+fi
--- a/workplans/NK-WP-0001-sso-mfa-platform.md
+++ b/workplans/NK-WP-0001-sso-mfa-platform.md
@@ -196,8 +196,9 @@ restore drill passed.
 ```task
 id: NK-WP-0001-T04
 state_hub_task_id: 6ad1296a-a488-4031-b665-f77030e971ed
-status: todo
+status: in_progress
 priority: high
+note: Manifests committed (pvc, configmap, deployment, middleware, ingress). Scripts: create-secrets.sh, enckey-bootstrap.sh, bootstrap-admin.sh. verify-t04.sh. Domain pink.coulomb.social (CP-NK-002/003). Pending: apply to live cluster, run enckey-bootstrap.sh, bootstrap-admin.sh.
 ```

 Deploy privacyIDEA via `gpappsoft/privacyidea` Helm chart (Artifact Hub) or
@@ -217,7 +218,7 @@ privacyidea:
    key: PI_ENCFILE
  ingress:
    enabled: true
-    hostname: pi.yourdomain.com
+    hostname: pink.coulomb.social
    tls: true
 ```

@@ -233,7 +234,7 @@ WAF rules at Traefik level.
 4. Apply policies: WebUI restricted to VPN/office IPs; MFA required for
   all admin actions.

-**Done when:** privacyIDEA reachable at pi.yourdomain.com with valid TLS,
+**Done when:** privacyIDEA reachable at pink.coulomb.social with valid TLS,
 pi-admin enrolled with MFA, trigger-admin created, rate-limiting active.

 ---
@@ -287,7 +288,7 @@ In Keycloak:
   the enterprise tier (not in scope for this workplan phase).
 2. Create Authentication Flow "privacyIDEA Browser":
   - Add privacyIDEA execution step (REQUIRED)
-   - Config: privacyIDEA URL = `https://pi.yourdomain.com`, service account
+   - Config: privacyIDEA URL = `https://pink.coulomb.social`, service account
     = `trigger-admin` (secret from K8s Secret)
   - Optional: bypass group (break-glass) with strict restrictions + alerts
 3. Set this flow as the default browser flow.
@@ -326,7 +327,7 @@ Define policies in privacyIDEA:
 - Enrollment rules (who can self-enroll, which token types)
 - Admin rights separation: super-admin vs. helpdesk-admin

-Enable self-service portal at `pi-account.yourdomain.com` for user token
+Enable self-service portal at `pink-account.coulomb.social` for user token
 enrollment/replacement.

 Configure auditing and log shipping: privacyIDEA audit logs + Keycloak
@@ -395,7 +396,7 @@ documented and tested, HSTS and NetworkPolicies verified.
 - [ ] `sso`, `mfa`, `databases` namespaces + NetworkPolicies deployed
 - [ ] TLS everywhere via cert-manager (Traefik ingress)
 - [ ] PostgreSQL live; both DBs created; backup + restore tested
- [ ] privacyIDEA running at `pi.yourdomain.com`; pi-admin MFA enrolled;
+- [ ] privacyIDEA running at `pink.coulomb.social`; pi-admin MFA enrolled;
      trigger-admin created with least-privilege rights
 - [ ] Keycloak running from custom image including privacyIDEA Provider JAR
 - [ ] Keycloak "privacyIDEA Browser" flow enforced as default