feat(sso-mfa): T05 Keycloak manifests (NK-WP-0001-T05)

Deploys Keycloak (SSO core) in the sso namespace. Files: sso-mfa/k8s/keycloak/pvc.yaml — keycloak-data PVC (build cache) sso-mfa/k8s/keycloak/middleware.yaml — rate-limit, admin-allowlist, HSTS sso-mfa/k8s/keycloak/deployment.yaml — Deployment + Service; init container downloads privacyIDEA provider JAR sso-mfa/k8s/keycloak/ingress.yaml — Ingress for kc.coulomb.social (CP-NK-004) sso-mfa/k8s/keycloak/create-secrets.sh — keycloak-config Secret sso-mfa/k8s/keycloak/bootstrap-realm.sh— hardens master realm, creates net-kingdom realm sso-mfa/k8s/keycloak/README.md — apply order, custom image guide, DR sso-mfa/k8s/verify-t05.sh — T05 done-criteria verification script Config points added: CP-NK-004 (kc.coulomb.social), CP-NK-005 (provider JAR URL). CP-NK-005 must be set before applying deployment.yaml. Pending: apply to live cluster, set CP-NK-005, run bootstrap-realm.sh, verify-t05.sh. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 02:00:51 +00:00
parent 1d94652ba1
commit d0ed7d9cd6
9 changed files with 1116 additions and 0 deletions
--- a/CONFIG.md
+++ b/CONFIG.md
@@ -23,6 +23,8 @@ If yes to any of the above, don't add it here.
 | CP-NK-001 | ACME contact email | `bernd.worsch+netkingdom@gmail.com` | `sso-mfa/k8s/cert-manager/issuers.yaml:38` |
 | CP-NK-002 | privacyIDEA portal hostname | `pink.coulomb.social` | `sso-mfa/k8s/privacyidea/ingress.yaml` |
 | CP-NK-003 | privacyIDEA self-service hostname | `pink-account.coulomb.social` | `sso-mfa/k8s/privacyidea/ingress.yaml` |
+| CP-NK-004 | Keycloak SSO hostname | `kc.coulomb.social` | `sso-mfa/k8s/keycloak/deployment.yaml`, `sso-mfa/k8s/keycloak/ingress.yaml` |
+| CP-NK-005 | privacyIDEA Keycloak Provider JAR URL | *(not set — edit before apply)* | `sso-mfa/k8s/keycloak/deployment.yaml` |

 ---

@@ -83,3 +85,48 @@ gains a structured "operator contact" concept.

 **Scope:** All TLS certificates issued by the `letsencrypt-prod` ClusterIssuer across
 the entire cluster.
+
+---
+
+## CP-NK-004 — Keycloak SSO hostname
+
+**Value:** `kc.coulomb.social`
+**Set:** 2026-03-19
+**Set by:** worsch
+
+**Location(s):**
+- `sso-mfa/k8s/keycloak/deployment.yaml` — `KC_HOSTNAME` env var
+- `sso-mfa/k8s/keycloak/ingress.yaml` — both Ingress `host` fields
+
+**Why non-default:** Subdomain prefix must be chosen by the operator. `kc` =
+**K**ey**c**loak, consistent with the service-initial naming pattern.
+
+**Scope:** TLS certificate, Traefik routing, Keycloak's internal hostname strictness
+check, and all OIDC/SAML redirect URIs registered in this realm. Changing this
+hostname after clients are registered requires updating all registered redirect URIs.
+
+---
+
+## CP-NK-005 — privacyIDEA Keycloak Provider JAR URL
+
+**Value:** *(not set — operator must edit before applying T05)*
+**Set:** —
+**Set by:** —
+
+**Location(s):**
+- `sso-mfa/k8s/keycloak/deployment.yaml` — `PROVIDER_JAR_URL` env var in the
+  `install-privacyidea-provider` init container
+
+**Why non-default:** The JAR URL depends on the chosen release version, which must
+be verified for compatibility with the deployed Keycloak image version. There is no
+stable "latest" URL suitable for automation.
+
+**How to set:**
+1. Browse https://github.com/privacyIDEA/keycloak-provider/releases
+2. Choose a release compatible with the Keycloak image version in `deployment.yaml`.
+3. Edit `deployment.yaml`: replace `EDIT_BEFORE_APPLY` with the `.jar` download URL.
+4. Update this entry with the chosen URL and version.
+
+**Scope:** Keycloak init container only. If switching to a custom Keycloak image
+(see T05 README "Custom image" section), this config point becomes obsolete and
+can be removed.
--- a/sso-mfa/k8s/keycloak/README.md
+++ b/sso-mfa/k8s/keycloak/README.md
@@ -0,0 +1,203 @@
+# T05 — Phase 4: Deploy Keycloak
+
+Phase 4 of NK-WP-0001: deploys the SSO core (Keycloak) in the `sso` namespace.
+
+**Hostname (config point CP-NK-004):**
+- `kc.coulomb.social` — OIDC/SAML SSO portal, admin console
+
+**Prerequisites:**
+- T02 complete: `sso` namespace and NetworkPolicies applied, cert-manager running.
+- T03 complete: PostgreSQL cluster `net-kingdom-pg` in `databases` namespace is Ready.
+- T04 complete: privacyIDEA is Running; `bootstrap-admin.sh` has been run so the
+  `privacyidea-trigger-admin` Secret exists in the `mfa` namespace.
+- T01 Phase 0a complete: `gen-secrets.sh` run, all secrets in KeePassXC.
+
+---
+
+## Before you apply: two required edits
+
+### Edit 1 — Provider JAR URL (CP-NK-005, required)
+
+The init container in `deployment.yaml` downloads the privacyIDEA Keycloak Provider
+JAR. You must set the URL before applying:
+
+1. Go to https://github.com/privacyIDEA/keycloak-provider/releases
+2. Download the JAR for a release compatible with your Keycloak image version.
+3. Edit `deployment.yaml`: find `PROVIDER_JAR_URL` and replace `EDIT_BEFORE_APPLY`
+   with the real URL.
+4. Add the URL as CP-NK-005 in `CONFIG.md` (see bottom of this README).
+
+If your cluster has no egress internet access, see **Custom image** below.
+
+### Edit 2 — Admin console IP allowlist (optional but recommended)
+
+Edit `middleware.yaml`: update `keycloak-admin-allowlist.spec.ipAllowList.sourceRange`
+to your actual VPN/office CIDRs.
+
+---
+
+## Apply order
+
+### Step 1 — Create secrets
+
+```bash
+cd sso-mfa/k8s/keycloak
+chmod +x create-secrets.sh bootstrap-realm.sh
+./create-secrets.sh
+```
+
+Creates `keycloak-config` in the `sso` namespace (KC_DB_URL, KC_DB_PASSWORD,
+KC_BOOTSTRAP_ADMIN_PASSWORD).
+
+---
+
+### Step 2 — Set provider JAR URL and apply manifests
+
+After editing `PROVIDER_JAR_URL` in `deployment.yaml`:
+
+```bash
+# From sso-mfa/k8s/keycloak/
+kubectl apply -f pvc.yaml
+kubectl apply -f middleware.yaml
+kubectl apply -f deployment.yaml
+kubectl apply -f ingress.yaml
+```
+
+**Wait for the pod to reach Running+Ready** (DB migrations + provider build on first
+boot — allow up to 5 minutes):
+
+```bash
+kubectl get pods -n sso -w
+# Expected: keycloak-<hash>   1/1   Running
+```
+
+If the pod is stuck in `Init`, check the init container logs first:
+```bash
+kubectl logs -n sso -l app.kubernetes.io/name=keycloak -c install-privacyidea-provider
+```
+
+Common causes of `Init` failure:
+- `PROVIDER_JAR_URL` still set to `EDIT_BEFORE_APPLY` → edit and reapply
+- No egress internet access → use custom image (see below)
+
+If the pod is in `CrashLoopBackOff`, check main container logs:
+```bash
+kubectl logs -n sso -l app.kubernetes.io/name=keycloak --previous
+```
+
+Common causes of Keycloak crash:
+- `keycloak-config` Secret missing → run `create-secrets.sh`
+- PostgreSQL not reachable → verify T03, check NetworkPolicies
+- Wrong DB password → re-run `create-secrets.sh` with corrected secrets
+- `KC_DB_URL` format incorrect → must be `jdbc:postgresql://...` (not SQLAlchemy format)
+
+---
+
+### Step 3 — Bootstrap realm
+
+After the pod is Running and Ready:
+
+```bash
+./bootstrap-realm.sh
+```
+
+This:
+1. Authenticates to the Keycloak admin REST API inside the pod.
+2. Hardens the master realm (SSL required, brute-force protection, token lifetimes).
+3. Creates the `net-kingdom` application realm with equivalent hardening.
+
+**Immediately after bootstrap completes:**
+1. Log in to `https://kc.coulomb.social/admin` as `admin`.
+2. Create a permanent admin account (with MFA — configure MFA flow in T06 first).
+3. Disable or delete the bootstrap `admin` account once the permanent admin is enrolled.
+
+---
+
+### Step 4 — Verify
+
+```bash
+cd sso-mfa/k8s
+chmod +x verify-t05.sh
+./verify-t05.sh
+```
+
+---
+
+## Custom image (recommended for production)
+
+The init-container approach downloads the provider JAR from the internet on every pod
+restart. For production or air-gapped clusters, build a custom image:
+
+```dockerfile
+FROM quay.io/keycloak/keycloak:26.0
+# Add the privacyIDEA provider JAR (download from GitHub releases first)
+COPY keycloak-provider-VERSION.jar /opt/keycloak/providers/
+# Build an optimized image — this bakes in the provider at build time
+RUN /opt/keycloak/bin/kc.sh build
+```
+
+Push to your registry, then:
+1. Update `deployment.yaml`: change the `keycloak` container `image` to your custom image.
+2. Change `args: ["start"]` to `args: ["start", "--optimized"]` for faster startup.
+3. Remove the `install-privacyidea-provider` init container entirely.
+4. The `providers` emptyDir volume and its mount can also be removed.
+
+---
+
+## NetworkPolicy design
+
+Keycloak sits behind the NetworkPolicies applied in T02 (netpol-sso.yaml):
+
+| Source | Destination | Port | Purpose |
+|--------|-------------|------|---------|
+| Traefik (kube-system) | Keycloak (sso) | 8080 | OIDC/SAML login pages |
+| Keycloak (sso) | privacyIDEA (mfa) | 8080 | MFA challenge API (T06) |
+| Keycloak (sso) | PostgreSQL (databases) | 5432 | Database |
+
+Outbound to anything other than privacyIDEA, PostgreSQL, and kube-dns is denied.
+
+---
+
+## Post-deploy steps (after verify-t05.sh passes)
+
+### Rotate the bootstrap admin password
+
+The `KC_BOOTSTRAP_ADMIN_PASSWORD` in `keycloak-config` was the initial credential.
+After creating a permanent admin:
+
+1. In Keycloak admin console: Users → admin → Disable account (or delete it).
+2. Rotate `KC_BOOTSTRAP_ADMIN_PASSWORD` in KeePassXC and re-run `create-secrets.sh`.
+3. Restart the Keycloak deployment: `kubectl rollout restart deployment/keycloak -n sso`
+
+### Admin console IP restriction
+
+Update `middleware.yaml` `keycloak-admin-allowlist.spec.ipAllowList.sourceRange`
+to your actual VPN/office CIDRs:
+
+```bash
+kubectl apply -f middleware.yaml
+```
+
+---
+
+## Adding CP-NK-005 to CONFIG.md
+
+After setting the provider JAR URL, add it to `CONFIG.md` as CP-NK-005:
+
+```markdown
+| CP-NK-005 | privacyIDEA provider JAR URL | <the URL you used> | `sso-mfa/k8s/keycloak/deployment.yaml` |
+```
+
+---
+
+## Disaster recovery
+
+If the `keycloak-data` PVC is lost (build cache only — all state is in PostgreSQL):
+
+1. Create a new PVC with `pvc.yaml`.
+2. Restart the deployment — Keycloak will rebuild from PostgreSQL on first start.
+   Allow 3–5 minutes for the rebuild + DB reconnect.
+3. Realm config, clients, and users are preserved in the database.
+
+If the PostgreSQL `keycloak_db` database is lost, restore from the CNPG backup
+(T03 procedure) before restarting Keycloak.
--- a/sso-mfa/k8s/keycloak/bootstrap-realm.sh
+++ b/sso-mfa/k8s/keycloak/bootstrap-realm.sh
@@ -0,0 +1,150 @@
+#!/usr/bin/env bash
+# bootstrap-realm.sh — Phase 1 Keycloak bootstrap for T05
+#
+# Runs AFTER the Keycloak pod is Running and Ready (verify with verify-t05.sh).
+#
+# Actions (T05 scope):
+#   1. Authenticate to Keycloak admin REST API via kcadm.sh in the pod
+#   2. Harden master realm: brute-force protection, token lifetimes, SSL required
+#   3. Create the net-kingdom application realm
+#   4. Enable realm-level brute-force protection and SSL
+#
+# Out of scope for T05 (handled in T06):
+#   - privacyIDEA authentication flow configuration
+#   - User federation / LDAP resolver setup
+#   - Per-client OIDC settings
+#   - Break-glass admin account (T07)
+#
+# Usage:
+#   chmod +x bootstrap-realm.sh
+#   ./bootstrap-realm.sh [--realm-name NAME]
+#
+# Options:
+#   --realm-name NAME   Name for the application realm (default: net-kingdom)
+#
+# Prerequisites:
+#   - kubectl configured with cluster access
+#   - Keycloak pod Running+Ready in the sso namespace
+#   - KC_ADMIN_PASSWORD available (prompted interactively if not set as env var)
+
+set -euo pipefail
+
+NAMESPACE="sso"
+REALM_NAME="net-kingdom"
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --realm-name) REALM_NAME="$2"; shift 2 ;;
+        *) echo "Unknown option: $1" >&2; exit 1 ;;
+    esac
+done
+
+# ── Find the running Keycloak pod ─────────────────────────────────────────────
+KC_POD=$(kubectl get pod -n "$NAMESPACE" \
+    -l app.kubernetes.io/name=keycloak \
+    --field-selector=status.phase=Running \
+    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
+
+if [[ -z "$KC_POD" ]]; then
+    echo "ERROR: No running Keycloak pod found in namespace $NAMESPACE." >&2
+    echo "Run verify-t05.sh to diagnose." >&2
+    exit 1
+fi
+
+echo "Using pod: $KC_POD"
+
+# ── Admin password ────────────────────────────────────────────────────────────
+if [[ -z "${KC_ADMIN_PASSWORD:-}" ]]; then
+    echo -n "Keycloak admin password (KC_ADMIN_PASSWORD): "
+    read -rs KC_ADMIN_PASSWORD
+    echo ""
+fi
+
+if [[ -z "$KC_ADMIN_PASSWORD" ]]; then
+    echo "ERROR: KC_ADMIN_PASSWORD is required." >&2
+    exit 1
+fi
+
+# ── Helper: run kcadm.sh inside the pod ──────────────────────────────────────
+kcadm() {
+    kubectl exec -n "$NAMESPACE" "$KC_POD" -- \
+        /opt/keycloak/bin/kcadm.sh "$@"
+}
+
+# ── 1. Authenticate ───────────────────────────────────────────────────────────
+echo ""
+echo "── Authenticating ───────────────────────────────────────────────────────"
+kcadm config credentials \
+    --server http://localhost:8080 \
+    --realm master \
+    --user admin \
+    --password "$KC_ADMIN_PASSWORD"
+
+echo "  Authenticated as admin@master"
+
+# ── 2. Harden master realm ────────────────────────────────────────────────────
+echo ""
+echo "── Hardening master realm ───────────────────────────────────────────────"
+
+# Require SSL for all connections to master realm (external access requires HTTPS).
+kcadm update realms/master \
+    --set 'sslRequired=external'
+echo "  sslRequired=external (master)"
+
+# Brute-force protection: lock after 5 failures for 5 minutes.
+kcadm update realms/master \
+    --set 'bruteForceProtected=true' \
+    --set 'failureFactor=5' \
+    --set 'waitIncrementSeconds=60' \
+    --set 'maxFailureWaitSeconds=900' \
+    --set 'minimumQuickLoginWaitSeconds=60'
+echo "  Brute-force protection enabled (master)"
+
+# Shorten access token lifetime for security.
+kcadm update realms/master \
+    --set 'accessTokenLifespan=300'       # 5 min
+echo "  Access token lifetime: 300s (master)"
+
+# ── 3. Create the net-kingdom application realm ───────────────────────────────
+echo ""
+echo "── Creating realm: $REALM_NAME ─────────────────────────────────────────"
+
+if kcadm get realms/"$REALM_NAME" &>/dev/null; then
+    echo "  Realm $REALM_NAME already exists — skipping creation."
+else
+    kcadm create realms \
+        --set "realm=$REALM_NAME" \
+        --set 'enabled=true' \
+        --set 'displayName=net-kingdom' \
+        --set 'sslRequired=external' \
+        --set 'registrationAllowed=false' \
+        --set 'bruteForceProtected=true' \
+        --set 'failureFactor=5' \
+        --set 'waitIncrementSeconds=60' \
+        --set 'maxFailureWaitSeconds=900' \
+        --set 'minimumQuickLoginWaitSeconds=60' \
+        --set 'accessTokenLifespan=300' \
+        --set 'refreshTokenMaxReuse=0' \
+        --set 'revokeRefreshToken=true'
+    echo "  Realm $REALM_NAME created."
+fi
+
+# ── 4. Summary ────────────────────────────────────────────────────────────────
+echo ""
+echo "════════════════════════════════════════════════════════════"
+echo "  T05 realm bootstrap complete."
+echo "════════════════════════════════════════════════════════════"
+echo ""
+echo "What was done:"
+echo "  - master realm: SSL required, brute-force protection, short token lifetimes"
+echo "  - realm '$REALM_NAME' created with equivalent hardening"
+echo ""
+echo "Next steps (T06):"
+echo "  1. Log in to https://kc.coulomb.social/admin with the bootstrap admin."
+echo "     Immediately create a permanent admin account, then delete/disable"
+echo "     the bootstrap 'admin' account (or rotate KC_BOOTSTRAP_ADMIN_PASSWORD)."
+echo "  2. In realm '$REALM_NAME':"
+echo "     - Configure the privacyIDEA authentication flow (see T06 README)."
+echo "     - Set privacyIDEA base URL: https://pink.coulomb.social"
+echo "     - Set trigger-admin credentials from the privacyidea-trigger-admin Secret."
+echo "  3. Run verify-t05.sh to confirm all T05 done-criteria."
--- a/sso-mfa/k8s/keycloak/create-secrets.sh
+++ b/sso-mfa/k8s/keycloak/create-secrets.sh
@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# create-secrets.sh — create the keycloak-config K8s Secret
+#
+# Usage:
+#   ./create-secrets.sh [secrets-dir]
+#
+# <secrets-dir> is the output directory from sso-mfa/bootstrap/gen-secrets.sh
+# (default: ../../bootstrap/secrets).
+#
+# Creates ONE Secret in the sso namespace:
+#   keycloak-config   — KC_DB_URL, KC_DB_PASSWORD, KC_BOOTSTRAP_ADMIN_PASSWORD
+#
+# This secret must exist before applying deployment.yaml.
+#
+# Re-run with --rotate to update secrets after a rotation in KeePassXC.
+
+set -euo pipefail
+
+SECRETS_DIR="${1:-../../bootstrap/secrets}"
+KC_ENV="$SECRETS_DIR/keycloak/secrets.env"
+PG_ENV="$SECRETS_DIR/postgres/secrets.env"
+
+if [[ ! -d "$SECRETS_DIR" ]]; then
+    echo "ERROR: secrets directory not found: $SECRETS_DIR" >&2
+    echo "Run sso-mfa/bootstrap/gen-secrets.sh first." >&2
+    exit 1
+fi
+
+for f in "$KC_ENV" "$PG_ENV"; do
+    if [[ ! -f "$f" ]]; then
+        echo "ERROR: $f not found" >&2
+        exit 1
+    fi
+done
+
+# Read values from the generated env files in subshells to avoid polluting env.
+KC_ADMIN_PASSWORD=$(bash -c "source '$KC_ENV' 2>/dev/null; echo \$KC_ADMIN_PASSWORD")
+KC_DB_PASSWORD=$(bash -c "source '$KC_ENV' 2>/dev/null; echo \$KC_DB_PASSWORD")
+
+if [[ -z "$KC_ADMIN_PASSWORD" || -z "$KC_DB_PASSWORD" ]]; then
+    echo "ERROR: could not read KC_ADMIN_PASSWORD or KC_DB_PASSWORD from $KC_ENV" >&2
+    echo "Check that gen-secrets.sh ran successfully." >&2
+    exit 1
+fi
+
+# Construct the JDBC database URL.
+# CloudNativePG read-write service: net-kingdom-pg-rw.databases.svc.cluster.local
+# Keycloak uses JDBC format (jdbc:postgresql://...) — NOT the SQLAlchemy URI format.
+KC_DB_URL="jdbc:postgresql://net-kingdom-pg-rw.databases.svc.cluster.local:5432/keycloak_db"
+
+echo "Creating K8s Secret: keycloak-config (namespace: sso)"
+kubectl create secret generic keycloak-config \
+    --namespace=sso \
+    --from-literal=KC_DB_URL="$KC_DB_URL" \
+    --from-literal=KC_DB_PASSWORD="$KC_DB_PASSWORD" \
+    --from-literal=KC_BOOTSTRAP_ADMIN_PASSWORD="$KC_ADMIN_PASSWORD" \
+    --dry-run=client -o yaml | kubectl apply -f -
+
+echo ""
+echo "Done. Secret keycloak-config created in namespace: sso"
+echo ""
+echo "Next:"
+echo "  1. Edit deployment.yaml: set PROVIDER_JAR_URL to the privacyIDEA provider JAR URL (CP-NK-005)."
+echo "  2. Apply manifests (see README.md apply order)."
+echo "  3. After the pod is Running+Ready, run: ./bootstrap-realm.sh"
--- a/sso-mfa/k8s/keycloak/deployment.yaml
+++ b/sso-mfa/k8s/keycloak/deployment.yaml
@@ -0,0 +1,251 @@
+# Deployment + Service — Keycloak (namespace: sso)
+#
+# Prerequisites (apply in order):
+#   1. pvc.yaml          — keycloak-data PVC
+#   2. middleware.yaml   — Traefik middlewares
+#   3. create-secrets.sh — keycloak-config Secret (KC_DB_URL, KC_DB_PASSWORD,
+#                          KC_BOOTSTRAP_ADMIN_PASSWORD)
+#   4. Edit PROVIDER_JAR_URL in the init container below (CP-NK-005 — see CONFIG.md)
+#   5. This file
+#
+# After first pod starts successfully:
+#   6. bootstrap-realm.sh — configure master realm, create net-kingdom realm
+#
+# privacyIDEA Keycloak Provider (init container):
+#   The init container downloads the provider JAR to /opt/keycloak/providers/
+#   before Keycloak starts. Keycloak detects providers and rebuilds automatically
+#   (no --optimized flag). Build output is cached in the keycloak-data PVC so
+#   subsequent restarts skip the full rebuild.
+#
+#   For production, prefer building a custom image with the provider pre-baked:
+#     FROM quay.io/keycloak/keycloak:VERSION
+#     COPY keycloak-provider-VERSION.jar /opt/keycloak/providers/
+#     RUN /opt/keycloak/bin/kc.sh build
+#   A custom image avoids the internet dependency and ensures a reproducible build.
+#   See README.md "Custom image" section for details.
+#
+# Container ports:
+#   8080 — HTTP (Traefik ingress; TLS terminated at Traefik)
+#   9000 — Management (health/live, health/ready, health/started — kubelet only)
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: keycloak
+  namespace: sso
+  labels:
+    app.kubernetes.io/name: keycloak
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: sso
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak
+  strategy:
+    type: Recreate   # single replica — avoid two pods racing on the build-cache PVC
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: keycloak
+        app.kubernetes.io/part-of: net-kingdom-sso-mfa
+        net-kingdom/component: sso
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000      # keycloak user inside the official image
+        fsGroup: 1000
+
+      # ── Init: download privacyIDEA Keycloak Provider JAR ─────────────────
+      # The JAR is placed in /opt/keycloak/providers/ via an emptyDir volume.
+      # Keycloak detects the new provider on startup and rebuilds automatically.
+      #
+      # NOTE: This requires outbound HTTPS from the cluster to GitHub.
+      # If your cluster has no egress internet access, pre-stage the JAR in an
+      # internal registry or use a custom Keycloak image (see comment above).
+      initContainers:
+        - name: install-privacyidea-provider
+          # Pin curl image version alongside the Keycloak image.
+          image: curlimages/curl:8.10.1
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 65534   # nobody — curl image default
+          env:
+            - name: PROVIDER_JAR_URL
+              # CP-NK-005 — EDIT this value before applying.
+              # Find the correct release at:
+              #   https://github.com/privacyIDEA/keycloak-provider/releases
+              # Choose a version compatible with your Keycloak image version above.
+              # Example:
+              #   https://github.com/privacyIDEA/keycloak-provider/releases/download/v0.9/keycloak-provider-0.9.jar
+              value: "EDIT_BEFORE_APPLY"
+          command:
+            - sh
+            - -c
+            - |
+              if [ "$PROVIDER_JAR_URL" = "EDIT_BEFORE_APPLY" ]; then
+                echo "ERROR: PROVIDER_JAR_URL not set." >&2
+                echo "Edit deployment.yaml and replace PROVIDER_JAR_URL with the real JAR URL." >&2
+                echo "See CONFIG.md CP-NK-005 and README.md." >&2
+                exit 1
+              fi
+              echo "Downloading privacyIDEA Keycloak Provider from: $PROVIDER_JAR_URL"
+              curl -fsSL -o /providers/keycloak-provider.jar "$PROVIDER_JAR_URL"
+              BYTES=$(wc -c < /providers/keycloak-provider.jar)
+              echo "Downloaded: ${BYTES} bytes -> /providers/keycloak-provider.jar"
+          volumeMounts:
+            - name: providers
+              mountPath: /providers
+
+      containers:
+        - name: keycloak
+          # Pin to a specific release; update via image update policy.
+          # Check https://quay.io/repository/keycloak/keycloak for latest stable.
+          image: quay.io/keycloak/keycloak:26.0
+          imagePullPolicy: IfNotPresent
+
+          # kc.sh start — no --optimized flag so Keycloak rebuilds when providers change.
+          # After the first successful start, subsequent starts use the cached build
+          # in the keycloak-data PVC and restart within ~30 seconds.
+          args: ["start"]
+
+          ports:
+            - name: http
+              containerPort: 8080
+              protocol: TCP
+            - name: management
+              containerPort: 9000
+              protocol: TCP
+
+          # ── Environment — sensitive values from Secret ──────────────────
+          env:
+            # Database
+            - name: KC_DB
+              value: postgres
+            - name: KC_DB_URL
+              valueFrom:
+                secretKeyRef:
+                  name: keycloak-config
+                  key: KC_DB_URL
+            - name: KC_DB_USERNAME
+              value: keycloak
+            - name: KC_DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: keycloak-config
+                  key: KC_DB_PASSWORD
+
+            # Bootstrap admin (used only on first start to create the admin user)
+            - name: KC_BOOTSTRAP_ADMIN_USERNAME
+              value: admin
+            - name: KC_BOOTSTRAP_ADMIN_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: keycloak-config
+                  key: KC_BOOTSTRAP_ADMIN_PASSWORD
+
+            # Hostname & proxy
+            # CP-NK-004 — update CONFIG.md if you change this value.
+            - name: KC_HOSTNAME
+              value: kc.coulomb.social
+            # Traefik passes X-Forwarded-For and X-Forwarded-Proto headers.
+            - name: KC_PROXY_HEADERS
+              value: xforwarded
+            # TLS is terminated at Traefik; Keycloak serves HTTP inside the cluster.
+            - name: KC_HTTP_ENABLED
+              value: "true"
+
+            # Observability
+            - name: KC_HEALTH_ENABLED
+              value: "true"
+            - name: KC_METRICS_ENABLED
+              value: "true"
+            - name: KC_LOG_LEVEL
+              value: INFO
+
+            # Caching — local = in-JVM Infinispan; switch to ispn for multi-replica HA.
+            - name: KC_CACHE
+              value: local
+
+          # ── Volume mounts ───────────────────────────────────────────────
+          volumeMounts:
+            # providers emptyDir: populated by init container before Keycloak starts
+            - name: providers
+              mountPath: /opt/keycloak/providers
+              readOnly: true
+            # data PVC: Keycloak build cache (data/generated/) and runtime data
+            - name: data
+              mountPath: /opt/keycloak/data
+
+          # ── Probes ──────────────────────────────────────────────────────
+          # Keycloak 24+: health endpoints on management port 9000.
+          # /health/started — true once the application has completed startup.
+          # /health/live    — true unless the application is in an unrecoverable state.
+          # /health/ready   — true once Keycloak can serve requests.
+          #
+          # Startup: allow up to 5 min for DB migrations + provider build on first boot.
+          startupProbe:
+            httpGet:
+              path: /health/started
+              port: 9000
+            initialDelaySeconds: 20
+            periodSeconds: 10
+            failureThreshold: 30   # 30 × 10s = 5 min
+          livenessProbe:
+            httpGet:
+              path: /health/live
+              port: 9000
+            initialDelaySeconds: 0
+            periodSeconds: 15
+            failureThreshold: 3
+          readinessProbe:
+            httpGet:
+              path: /health/ready
+              port: 9000
+            initialDelaySeconds: 0
+            periodSeconds: 10
+            failureThreshold: 3
+
+          # ── Resources ───────────────────────────────────────────────────
+          # Keycloak is JVM-based; the initial provider build spikes CPU briefly.
+          # Raise limits for production.
+          resources:
+            requests:
+              cpu: "250m"
+              memory: "512Mi"
+            limits:
+              cpu: "1000m"
+              memory: "1Gi"
+
+      # ── Volumes ─────────────────────────────────────────────────────────
+      volumes:
+        # providers emptyDir: re-populated by init container on every pod start.
+        # Keycloak detects the JAR and checks whether a rebuild is needed.
+        # If the JAR hash matches the cached build, startup is fast (~30s).
+        - name: providers
+          emptyDir: {}
+        - name: data
+          persistentVolumeClaim:
+            claimName: keycloak-data
+
+---
+# Service — ClusterIP; Traefik reaches Keycloak via port 8080.
+# Port 9000 (management) is NOT exposed — kubelet probes reach it directly on the pod.
+apiVersion: v1
+kind: Service
+metadata:
+  name: keycloak
+  namespace: sso
+  labels:
+    app.kubernetes.io/name: keycloak
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: sso
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: keycloak
+  ports:
+    - name: http
+      port: 8080
+      targetPort: 8080
+      protocol: TCP
--- a/sso-mfa/k8s/keycloak/ingress.yaml
+++ b/sso-mfa/k8s/keycloak/ingress.yaml
@@ -0,0 +1,87 @@
+# Ingress — Keycloak (namespace: sso)
+#
+# kc.coulomb.social — SSO portal, OIDC/SAML endpoints, user login
+#
+# TLS: cert-manager issues the certificate via the letsencrypt-prod ClusterIssuer
+# (T02). Public DNS for kc.coulomb.social must resolve to the cluster's external
+# IP before cert-manager can complete the ACME HTTP-01 challenge.
+#
+# Middlewares:
+#   keycloak-rate-limit      — 100 req/min per IP (all paths)
+#   keycloak-admin-allowlist — restrict /admin/* to VPN/office IPs (see middleware.yaml)
+#   keycloak-hsts            — inject HSTS header on all HTTPS responses
+#
+# Config points (see CONFIG.md):
+#   CP-NK-004  kc.coulomb.social
+
+# ── Main portal — kc.coulomb.social ──────────────────────────────────────────
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: keycloak
+  namespace: sso
+  labels:
+    app.kubernetes.io/name: keycloak
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: sso
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    traefik.ingress.kubernetes.io/router.middlewares: >-
+      sso-keycloak-rate-limit@kubernetescrd,
+      sso-keycloak-hsts@kubernetescrd
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: kc.coulomb.social
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: keycloak
+                port:
+                  number: 8080
+  tls:
+    - secretName: kc-tls
+      hosts:
+        - kc.coulomb.social
+---
+# ── Admin console — kc.coulomb.social/admin — restricted to VPN/office IPs ──
+# Separate Ingress so the admin-allowlist middleware applies only to /admin/*.
+# Traefik prefers the more-specific /admin prefix over the / prefix above.
+#
+# The admin console path in Keycloak 20+ is /admin/.
+# Realm-specific admin REST API is at /admin/realms/{realm}/...
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: keycloak-admin
+  namespace: sso
+  labels:
+    app.kubernetes.io/name: keycloak
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: sso
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    traefik.ingress.kubernetes.io/router.middlewares: >-
+      sso-keycloak-rate-limit@kubernetescrd,
+      sso-keycloak-admin-allowlist@kubernetescrd,
+      sso-keycloak-hsts@kubernetescrd
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: kc.coulomb.social
+      http:
+        paths:
+          - path: /admin
+            pathType: Prefix
+            backend:
+              service:
+                name: keycloak
+                port:
+                  number: 8080
+  tls:
+    - secretName: kc-tls
+      hosts:
+        - kc.coulomb.social
--- a/sso-mfa/k8s/keycloak/middleware.yaml
+++ b/sso-mfa/k8s/keycloak/middleware.yaml
@@ -0,0 +1,71 @@
+# Traefik Middlewares for Keycloak (namespace: sso)
+#
+# Middleware names follow the pattern referenced in ingress.yaml annotations:
+#   sso-keycloak-rate-limit@kubernetescrd
+#   sso-keycloak-admin-allowlist@kubernetescrd
+#   sso-keycloak-hsts@kubernetescrd
+#
+# Traefik API version:
+#   Traefik v3 (K3s >= 1.30): traefik.io/v1alpha1
+#   Traefik v2 (K3s <  1.30): traefik.containo.us/v1alpha1
+# Check: kubectl get middleware -n sso -o yaml | grep apiVersion
+# Update all documents below if you need the v2 apiVersion.
+
+# ── Rate limit — all KC endpoints ────────────────────────────────────────────
+# 100 requests/minute per client IP; burst of 20 allowed.
+# Higher than privacyIDEA because OIDC discovery + JS app calls are bursty.
+# The /realms/{realm}/.well-known/openid-configuration call alone counts.
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: keycloak-rate-limit
+  namespace: sso
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: sso
+spec:
+  rateLimit:
+    average: 100
+    period: 1m
+    burst: 20
+---
+# ── Admin console allowlist — restrict /admin to VPN/office IPs ──────────────
+# Applied to the /admin Ingress (see ingress.yaml — separate Ingress for /admin/).
+#
+# ADJUST sourceRange to your actual VPN / office CIDR(s) before going live.
+# Leaving RFC-1918 ranges here is only a dev/staging default.
+#
+# Traefik v3 uses ipAllowList; Traefik v2 uses ipWhiteList.
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: keycloak-admin-allowlist
+  namespace: sso
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: sso
+spec:
+  ipAllowList:
+    # EDIT: replace with your VPN/office CIDRs (see CONFIG.md for the pattern).
+    sourceRange:
+      - "10.0.0.0/8"
+      - "172.16.0.0/12"
+      - "192.168.0.0/16"
+---
+# ── HSTS — HTTP Strict Transport Security ────────────────────────────────────
+# Keycloak docs recommend HSTS for all deployments.
+# Traefik terminates TLS; Keycloak runs HTTP internally.
+# This header is injected by Traefik on all HTTPS responses.
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: keycloak-hsts
+  namespace: sso
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: sso
+spec:
+  headers:
+    stsSeconds: 31536000          # 1 year
+    stsIncludeSubdomains: true
+    stsPreload: true
--- a/sso-mfa/k8s/keycloak/pvc.yaml
+++ b/sso-mfa/k8s/keycloak/pvc.yaml
@@ -0,0 +1,22 @@
+# PersistentVolumeClaim for Keycloak (namespace: sso)
+#
+# keycloak-data — /opt/keycloak/data
+#   Holds: Keycloak build cache (data/generated/) produced by kc.sh build.
+#   Persisting this avoids a full provider rebuild on every pod restart.
+#   Also holds H2 emergency data (only used if PostgreSQL is unreachable).
+#
+# Adjust storage size before production deployment.
+
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: keycloak-data
+  namespace: sso
+  labels:
+    app.kubernetes.io/part-of: net-kingdom-sso-mfa
+    net-kingdom/component: sso
+spec:
+  accessModes: [ReadWriteOnce]
+  resources:
+    requests:
+      storage: 2Gi
--- a/sso-mfa/k8s/verify-t05.sh
+++ b/sso-mfa/k8s/verify-t05.sh
@@ -0,0 +1,220 @@
+#!/usr/bin/env bash
+# verify-t05.sh — verify NK-WP-0001-T05 done-criteria
+#
+# Checks:
+#   1. Keycloak pod is Running+Ready in the sso namespace
+#   2. Keycloak Service exists on port 8080
+#   3. Traefik Middlewares exist
+#   4. Ingress resources exist with correct hostname
+#   5. TLS certificate issued by cert-manager
+#   6. Required K8s Secrets are present
+#   7. PVC is Bound
+#   8. Provider JAR is present inside the pod
+#   9. Keycloak health endpoints respond (started, ready)
+#  10. net-kingdom realm exists
+#
+# Usage:
+#   chmod +x verify-t05.sh
+#   ./verify-t05.sh
+
+set -euo pipefail
+
+NAMESPACE="sso"
+KC_HOSTNAME="kc.coulomb.social"
+REALM_NAME="net-kingdom"
+PASS=0
+FAIL=0
+WARN=0
+
+pass() { echo "  [PASS] $1"; ((PASS++)); }
+fail() { echo "  [FAIL] $1"; ((FAIL++)); }
+warn() { echo "  [WARN] $1"; ((WARN++)); }
+
+section() { echo ""; echo "── $1 ──────────────────────────────────────"; }
+
+# ── 1. Keycloak pod ───────────────────────────────────────────────────────────
+section "1. Keycloak pod (namespace: $NAMESPACE)"
+
+KC_POD=$(kubectl get pod -n "$NAMESPACE" \
+    -l app.kubernetes.io/name=keycloak \
+    --field-selector=status.phase=Running \
+    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
+
+if [[ -n "$KC_POD" ]]; then
+    pass "Pod Running: $KC_POD"
+
+    READY=$(kubectl get pod -n "$NAMESPACE" "$KC_POD" \
+        -o jsonpath='{.status.containerStatuses[0].ready}' 2>/dev/null || echo "false")
+    if [[ "$READY" == "true" ]]; then
+        pass "Pod readiness probe passing"
+    else
+        fail "Pod is Running but not Ready (probe failing — check logs)"
+    fi
+else
+    PENDING=$(kubectl get pod -n "$NAMESPACE" \
+        -l app.kubernetes.io/name=keycloak \
+        -o name 2>/dev/null | wc -l || echo 0)
+    if [[ "$PENDING" -gt 0 ]]; then
+        fail "Keycloak pod(s) exist but none are Running (check: kubectl describe pod -n $NAMESPACE)"
+    else
+        fail "No Keycloak pods found in namespace $NAMESPACE — apply deployment.yaml"
+    fi
+fi
+
+# ── 2. Service ────────────────────────────────────────────────────────────────
+section "2. Service"
+
+if kubectl get service keycloak -n "$NAMESPACE" &>/dev/null; then
+    pass "Service keycloak exists"
+    PORT=$(kubectl get service keycloak -n "$NAMESPACE" \
+        -o jsonpath='{.spec.ports[0].port}' 2>/dev/null || echo "")
+    if [[ "$PORT" == "8080" ]]; then
+        pass "Service port: 8080"
+    else
+        warn "Service port is $PORT (expected 8080)"
+    fi
+else
+    fail "Service keycloak not found in namespace $NAMESPACE"
+fi
+
+# ── 3. Traefik Middlewares ────────────────────────────────────────────────────
+section "3. Traefik Middlewares"
+
+for mw in keycloak-rate-limit keycloak-admin-allowlist keycloak-hsts; do
+    if kubectl get middleware "$mw" -n "$NAMESPACE" &>/dev/null; then
+        pass "Middleware $mw exists"
+    else
+        fail "Middleware $mw not found — apply middleware.yaml"
+    fi
+done
+
+# ── 4. Ingress resources ──────────────────────────────────────────────────────
+section "4. Ingress resources"
+
+for ing in keycloak keycloak-admin; do
+    if kubectl get ingress "$ing" -n "$NAMESPACE" &>/dev/null; then
+        pass "Ingress $ing exists"
+    else
+        fail "Ingress $ing not found — apply ingress.yaml"
+    fi
+done
+
+KC_HOST=$(kubectl get ingress keycloak -n "$NAMESPACE" \
+    -o jsonpath='{.spec.rules[0].host}' 2>/dev/null || echo "")
+if [[ "$KC_HOST" == "$KC_HOSTNAME" ]]; then
+    pass "Ingress host: $KC_HOSTNAME"
+else
+    fail "Ingress host is '$KC_HOST' (expected $KC_HOSTNAME)"
+fi
+
+# ── 5. TLS certificate ────────────────────────────────────────────────────────
+section "5. TLS certificate"
+
+if kubectl get secret kc-tls -n "$NAMESPACE" &>/dev/null; then
+    CERT_READY=$(kubectl get certificate kc -n "$NAMESPACE" \
+        -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "")
+    if [[ "$CERT_READY" == "True" ]]; then
+        pass "Certificate kc is Ready (TLS secret kc-tls exists)"
+    else
+        warn "TLS secret kc-tls exists but certificate status is not Ready (DNS propagation pending?)"
+    fi
+else
+    warn "TLS secret kc-tls not yet issued (cert-manager pending — check DNS and ACME)"
+fi
+
+# ── 6. K8s Secrets ────────────────────────────────────────────────────────────
+section "6. K8s Secrets (namespace: $NAMESPACE)"
+
+if kubectl get secret keycloak-config -n "$NAMESPACE" &>/dev/null; then
+    pass "Secret keycloak-config exists"
+else
+    fail "Secret keycloak-config not found — run create-secrets.sh"
+fi
+
+# ── 7. PVC ────────────────────────────────────────────────────────────────────
+section "7. PersistentVolumeClaim"
+
+STATUS=$(kubectl get pvc keycloak-data -n "$NAMESPACE" \
+    -o jsonpath='{.status.phase}' 2>/dev/null || echo "not found")
+if [[ "$STATUS" == "Bound" ]]; then
+    pass "PVC keycloak-data: Bound"
+elif [[ "$STATUS" == "not found" ]]; then
+    fail "PVC keycloak-data not found — apply pvc.yaml"
+else
+    fail "PVC keycloak-data status: $STATUS (expected Bound)"
+fi
+
+# ── 8. Provider JAR ───────────────────────────────────────────────────────────
+section "8. privacyIDEA provider JAR (inside pod)"
+
+if [[ -n "$KC_POD" ]]; then
+    if kubectl exec -n "$NAMESPACE" "$KC_POD" -- \
+        test -f /opt/keycloak/providers/keycloak-provider.jar 2>/dev/null; then
+        pass "Provider JAR present: /opt/keycloak/providers/keycloak-provider.jar"
+    else
+        fail "Provider JAR not found — check init container logs: kubectl logs -n $NAMESPACE $KC_POD -c install-privacyidea-provider"
+    fi
+
+    # Check that the provider was picked up (appears in the build output directory)
+    if kubectl exec -n "$NAMESPACE" "$KC_POD" -- \
+        ls /opt/keycloak/data/generated/ 2>/dev/null | grep -q .; then
+        pass "Keycloak build cache present (provider build completed)"
+    else
+        warn "Build cache empty — Keycloak may still be building (check pod logs)"
+    fi
+else
+    warn "Skipping provider JAR check — no running pod"
+fi
+
+# ── 9. Health endpoints ───────────────────────────────────────────────────────
+section "9. Keycloak health endpoints (via kubectl exec)"
+
+if [[ -n "$KC_POD" ]]; then
+    for endpoint in /health/started /health/ready /health/live; do
+        STATUS_CODE=$(kubectl exec -n "$NAMESPACE" "$KC_POD" -- \
+            curl -s -o /dev/null -w "%{http_code}" "http://localhost:9000${endpoint}" \
+            2>/dev/null || echo "")
+        if [[ "$STATUS_CODE" == "200" ]]; then
+            pass "GET localhost:9000${endpoint} → 200"
+        else
+            fail "GET localhost:9000${endpoint} → $STATUS_CODE (expected 200)"
+        fi
+    done
+else
+    warn "Skipping health endpoint checks — no running pod"
+fi
+
+# ── 10. Realm exists ──────────────────────────────────────────────────────────
+section "10. Keycloak realm: $REALM_NAME"
+
+if [[ -n "$KC_POD" ]]; then
+    REALM_STATUS=$(kubectl exec -n "$NAMESPACE" "$KC_POD" -- \
+        curl -s -o /dev/null -w "%{http_code}" \
+        "http://localhost:8080/realms/${REALM_NAME}" 2>/dev/null || echo "")
+    if [[ "$REALM_STATUS" == "200" ]]; then
+        pass "Realm $REALM_NAME exists and responds"
+    elif [[ "$REALM_STATUS" == "404" ]]; then
+        warn "Realm $REALM_NAME not found — run bootstrap-realm.sh"
+    else
+        warn "Realm $REALM_NAME check returned HTTP $REALM_STATUS"
+    fi
+else
+    warn "Skipping realm check — no running pod"
+fi
+
+# ── Summary ───────────────────────────────────────────────────────────────────
+echo ""
+echo "════════════════════════════════════════════════════════════"
+echo "  T05 verification: PASS=$PASS  WARN=$WARN  FAIL=$FAIL"
+echo "════════════════════════════════════════════════════════════"
+
+if [[ "$FAIL" -gt 0 ]]; then
+    echo "  Result: INCOMPLETE — resolve FAIL items before proceeding to T06"
+    exit 1
+elif [[ "$WARN" -gt 0 ]]; then
+    echo "  Result: PARTIAL — T05 core is up; WARN items should be resolved before T06"
+    exit 0
+else
+    echo "  Result: COMPLETE — T05 done-criteria met; proceed to T06 (realm config & MFA flow)"
+    exit 0
+fi