Files
net-kingdom/sso-mfa/k8s
Bernd Worsch c054241a5c feat(t09): backup, break-glass, DR drill — NK-WP-0003-T09 done
- Apply SQLite backup CronJobs (LLDAP, Authelia, privacyIDEA) — all verified running
- Fix authelia-backup: remove scale-down/up dance; concurrent local-path PVC mount
  works on single-node k3s, sqlite3 .backup is safe for concurrent access
- Fix privacyidea-backup: add supplementalGroups: [999] so uid=1000 can read enckey
- Add allow-backup-to-kube-api NetworkPolicy (backup pod → 10.43.0.1:443)
- Create break-glass LLDAP account (net-kingdom-admins); fix ((PASS++)) set-e trap
- SQLite restore drill: LLDAP backup valid (2 users, all tables)
- verify-t08.sh: PASS=15, FAIL=0; fix counter bug + enckey PVC path (/etc/privacyidea)
- Update DR-RUNBOOK.md Authelia restore procedure
- T09 deferred: CNPG backup (needs MinIO/S3), Prometheus (needs kube-prometheus-stack)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-25 23:56:40 +00:00
..

T02 — K8s Foundations

Phase 1 of NK-WP-0001: namespaces, NetworkPolicies, cert-manager, StorageClass.

SSO stack overview

The sso namespace hosts three components:

  • KeyCape (kc.coulomb.social) — OIDC orchestration layer, stateless
  • Authelia (auth.coulomb.social) — password authentication frontend
  • LLDAP (lldap.coulomb.social) — lightweight LDAP directory (admin UI restricted)

The mfa namespace hosts:

  • privacyIDEA (pink.coulomb.social) — MFA engine, called by KeyCape

Prerequisites

  • K3s cluster running (ThreePhoenix HA or single-node dev)
  • T01 Phase 0a complete (KeePassXC vault populated, ops bundle exported)
  • kubectl configured with cluster access

Apply order

# 1. Install cert-manager (if not already on cluster)
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true

# Wait for cert-manager to be ready
kubectl rollout status deployment/cert-manager -n cert-manager

# 2. Create namespaces
kubectl apply -f namespaces/namespaces.yaml

# 3. Apply NetworkPolicies
kubectl apply -f network-policies/netpol-sso.yaml
kubectl apply -f network-policies/netpol-mfa.yaml
kubectl apply -f network-policies/netpol-databases.yaml

# 4. Create ClusterIssuers
#    Edit issuers.yaml first: replace ACME_EMAIL with your address
kubectl apply -f cert-manager/issuers.yaml

# 5. Verify cert-manager with test certificate
kubectl apply -f cert-manager/test-certificate.yaml
kubectl wait --for=condition=Ready certificate/selfsigned-test \
  -n cert-manager-test --timeout=60s
kubectl delete namespace cert-manager-test

# 6. Verify StorageClass
kubectl apply -f storage/verify-pvc.yaml
kubectl wait --for=condition=Ready pod/storage-test \
  -n storage-test --timeout=60s
kubectl logs -n storage-test storage-test
kubectl delete namespace storage-test

# 7. Run the full verification script
chmod +x verify-t02.sh
./verify-t02.sh

NetworkPolicy design

All three namespaces follow a default-deny-all posture. Only the minimal required paths are opened:

Source Destination Port Purpose
Traefik (kube-system) KeyCape (sso) 8080 OIDC endpoints — public
Traefik (kube-system) Authelia (sso) 9091 Login portal — public
Traefik (kube-system) LLDAP (sso) 17170 Admin web UI — IP-restricted
Traefik (kube-system) privacyIDEA (mfa) 8080 MFA portal — public
KeyCape (sso) Authelia (sso) 9091 OIDC token exchange
KeyCape (sso) LLDAP (sso) 3890 User attribute lookup
KeyCape (sso) privacyIDEA (mfa) 8080 MFA challenge + validation
Authelia (sso) LLDAP (sso) 3890 Credential validation
privacyIDEA (mfa) PostgreSQL (databases) 5432 DB
CNPG operator (cnpg-system) PostgreSQL (databases) 5432/9187 Operator + metrics
All pods kube-dns (kube-system) 53 DNS resolution
CNPG pods K8s API 6443 Status updates

Verifying denied paths (manual)

After applying NetworkPolicies, confirm that illegal paths are blocked:

# Test: KeyCape → privacyIDEA (should be ALLOWED)
kubectl run test-allowed -n sso --rm -it --image=busybox --restart=Never \
  -- nc -zv privacyidea.mfa.svc.cluster.local 8080

# Test: Authelia → privacyIDEA (should be DENIED — only KeyCape calls privacyIDEA)
kubectl run test-denied -n sso --rm -it --image=busybox --restart=Never \
  -l app.kubernetes.io/name=authelia \
  -- nc -zv -w3 privacyidea.mfa.svc.cluster.local 8080

# Test: databases → sso (should be DENIED — DB pods must not initiate connections)
kubectl run test-denied2 -n databases --rm -it --image=busybox --restart=Never \
  -- nc -zw3 keycape.sso.svc.cluster.local 8080

Notes

  • net-kingdom/component labels on namespaces are used by NetworkPolicy namespaceSelector rules. Do not remove them.
  • cnpg.io/cluster: net-kingdom-pg in netpol-databases.yaml must match the name of the CloudNativePG Cluster CR you create in T03.
  • The letsencrypt-prod ClusterIssuer requires public DNS and port 80 open to Let's Encrypt servers. Update ACME_EMAIL before applying.