net-kingdom/sso-mfa/k8s/keycloak

T05 — Phase 4: Deploy Keycloak

Phase 4 of NK-WP-0001: deploys the SSO core (Keycloak) in the sso namespace.

Hostname (config point CP-NK-004):

• kc.coulomb.social — OIDC/SAML SSO portal, admin console

Prerequisites:

• T02 complete: sso namespace and NetworkPolicies applied, cert-manager running.
• T03 complete: PostgreSQL cluster net-kingdom-pg in databases namespace is Ready.
• T04 complete: privacyIDEA is Running; bootstrap-admin.sh has been run so the privacyidea-trigger-admin Secret exists in the mfa namespace.
• T01 Phase 0a complete: gen-secrets.sh run, all secrets in KeePassXC.

---

Before you apply: two required edits

Edit 1 — Provider JAR URL (CP-NK-005, required)

The init container in deployment.yaml downloads the privacyIDEA Keycloak Provider JAR. You must set the URL before applying:

1. Go to https://github.com/privacyIDEA/keycloak-provider/releases
2. Download the JAR for a release compatible with your Keycloak image version.
3. Edit deployment.yaml: find PROVIDER_JAR_URL and replace EDIT_BEFORE_APPLY with the real URL.
4. Add the URL as CP-NK-005 in CONFIG.md (see bottom of this README).

If your cluster has no egress internet access, see Custom image below.

Edit 2 — Admin console IP allowlist (optional but recommended)

Edit middleware.yaml: update keycloak-admin-allowlist.spec.ipAllowList.sourceRange to your actual VPN/office CIDRs.

---

Apply order

Step 1 — Create secrets

cd sso-mfa/k8s/keycloak
chmod +x create-secrets.sh bootstrap-realm.sh
./create-secrets.sh

Creates keycloak-config in the sso namespace (KC_DB_URL, KC_DB_PASSWORD, KC_BOOTSTRAP_ADMIN_PASSWORD).

---

Step 2 — Set provider JAR URL and apply manifests

After editing PROVIDER_JAR_URL in deployment.yaml:

# From sso-mfa/k8s/keycloak/
kubectl apply -f pvc.yaml
kubectl apply -f middleware.yaml
kubectl apply -f deployment.yaml
kubectl apply -f ingress.yaml

Wait for the pod to reach Running+Ready (DB migrations + provider build on first boot — allow up to 5 minutes):

kubectl get pods -n sso -w
# Expected: keycloak-<hash>   1/1   Running

If the pod is stuck in Init, check the init container logs first:

kubectl logs -n sso -l app.kubernetes.io/name=keycloak -c install-privacyidea-provider

Common causes of Init failure:

• PROVIDER_JAR_URL still set to EDIT_BEFORE_APPLY → edit and reapply
• No egress internet access → use custom image (see below)

If the pod is in CrashLoopBackOff, check main container logs:

kubectl logs -n sso -l app.kubernetes.io/name=keycloak --previous

Common causes of Keycloak crash:

• keycloak-config Secret missing → run create-secrets.sh
• PostgreSQL not reachable → verify T03, check NetworkPolicies
• Wrong DB password → re-run create-secrets.sh with corrected secrets
• KC_DB_URL format incorrect → must be jdbc:postgresql://... (not SQLAlchemy format)

---

Step 3 — Bootstrap realm

After the pod is Running and Ready:

./bootstrap-realm.sh

This:

1. Authenticates to the Keycloak admin REST API inside the pod.
2. Hardens the master realm (SSL required, brute-force protection, token lifetimes).
3. Creates the net-kingdom application realm with equivalent hardening.

Immediately after bootstrap completes:

1. Log in to https://kc.coulomb.social/admin as admin.
2. Create a permanent admin account (with MFA — configure MFA flow in T06 first).
3. Disable or delete the bootstrap admin account once the permanent admin is enrolled.

---

Step 4 — Verify

cd sso-mfa/k8s
chmod +x verify-t05.sh
./verify-t05.sh

---

Custom image (recommended for production)

The init-container approach downloads the provider JAR from the internet on every pod restart. For production or air-gapped clusters, build a custom image:

FROM quay.io/keycloak/keycloak:26.0
# Add the privacyIDEA provider JAR (download from GitHub releases first)
COPY keycloak-provider-VERSION.jar /opt/keycloak/providers/
# Build an optimized image — this bakes in the provider at build time
RUN /opt/keycloak/bin/kc.sh build

Push to your registry, then:

1. Update deployment.yaml: change the keycloak container image to your custom image.
2. Change args: ["start"] to args: ["start", "--optimized"] for faster startup.
3. Remove the install-privacyidea-provider init container entirely.
4. The providers emptyDir volume and its mount can also be removed.

---

NetworkPolicy design

Keycloak sits behind the NetworkPolicies applied in T02 (netpol-sso.yaml):

Source	Destination	Port	Purpose
Traefik (kube-system)	Keycloak (sso)	8080	OIDC/SAML login pages
Keycloak (sso)	privacyIDEA (mfa)	8080	MFA challenge API (T06)
Keycloak (sso)	PostgreSQL (databases)	5432	Database

Outbound to anything other than privacyIDEA, PostgreSQL, and kube-dns is denied.

---

Post-deploy steps (after verify-t05.sh passes)

Rotate the bootstrap admin password

The KC_BOOTSTRAP_ADMIN_PASSWORD in keycloak-config was the initial credential. After creating a permanent admin:

1. In Keycloak admin console: Users → admin → Disable account (or delete it).
2. Rotate KC_BOOTSTRAP_ADMIN_PASSWORD in KeePassXC and re-run create-secrets.sh.
3. Restart the Keycloak deployment: kubectl rollout restart deployment/keycloak -n sso

Admin console IP restriction

Update middleware.yaml keycloak-admin-allowlist.spec.ipAllowList.sourceRange to your actual VPN/office CIDRs:

kubectl apply -f middleware.yaml

---

Adding CP-NK-005 to CONFIG.md

After setting the provider JAR URL, add it to CONFIG.md as CP-NK-005:

| CP-NK-005 | privacyIDEA provider JAR URL | <the URL you used> | `sso-mfa/k8s/keycloak/deployment.yaml` |

---

Disaster recovery

If the keycloak-data PVC is lost (build cache only — all state is in PostgreSQL):

1. Create a new PVC with pvc.yaml.
2. Restart the deployment — Keycloak will rebuild from PostgreSQL on first start. Allow 3–5 minutes for the rebuild + DB reconnect.
3. Realm config, clients, and users are preserved in the database.

If the PostgreSQL keycloak_db database is lost, restore from the CNPG backup (T03 procedure) before restarting Keycloak.