feat(sso-mfa): T07/T08 user mgmt, backups, DR & break-glass (NK-WP-0001-T07/T08)

T07 — User management & self-service:
- k8s/lldap/bootstrap-users.sh: creates net-kingdom-users and net-kingdom-admins
  groups in LLDAP via GraphQL API; idempotent.
- k8s/lldap/break-glass.sh: creates break-glass bypass account in LLDAP,
  sets BREAKGLASS_PASSWORD, assigns to net-kingdom-admins.
- k8s/verify-t07.sh: 6 checks — groups, break-glass, self-service portal,
  KeyCape OIDC client registrations.

T08 — Backups, DR, break-glass:
- k8s/backup/cronjob-sqlite-backups.yaml: daily CronJobs for LLDAP SQLite,
  Authelia SQLite (with scale-down/up RBAC), and privacyIDEA enckey backup.
  7-day retention, 03:00/03:15/03:30 UTC staggered schedule.
- k8s/backup/DR-RUNBOOK.md: full restore runbook — scenarios, restore order,
  LLDAP/Authelia/PI SQLite restore procedure, full node rebuild sequence,
  offsite age-encrypted export.
- k8s/verify-t08.sh: 9 checks — CronJobs, RBAC, run history, backup files
  on PVCs, DR runbook presence, offsite backup (manual confirmation).
- WORKPLAN.md: T07/T08 sections with done-criteria added.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-03-19 09:17:03 +00:00
parent 69e900ddb1
commit 6c062e1295
7 changed files with 1288 additions and 3 deletions

View File

@@ -0,0 +1,187 @@
# Disaster Recovery Runbook — net-kingdom SSO/MFA Platform
**Stack:** LLDAP + Authelia + KeyCape (sso namespace) + privacyIDEA (mfa namespace)
**PostgreSQL:** Managed separately by CNPG (`postgresql/scheduled-backup.yaml`)
---
## Recovery scenarios
| Scenario | Impact | Recovery |
|----------|--------|----------|
| Pod crash / OOM | Stateless pods (KeyCape) recover automatically. Stateful pods (LLDAP, Authelia, PI) restart and reload from PVC. | K8s self-heals. Verify with `verify-t05.sh`. |
| PVC data corruption | Users/sessions/tokens lost. | Restore from SQLite backup (see below). |
| Node failure (single-node K3s) | All pods lost. PVCs intact on host. | Re-apply all manifests (idempotent). Pods re-attach to PVCs. |
| Node total loss (disk gone) | Everything lost. | Full restore from backup + KeePassXC. |
| Stack locked out (SSO broken, can't log in) | No user access to OIDC-protected apps. | Use break-glass account. |
| enckey lost (privacyIDEA) | All enrolled MFA tokens invalid. Users must re-enroll. | Restore from enckey backup or re-enroll all tokens. |
---
## Break-glass access
When the SSO stack is broken and no user can authenticate:
```bash
# 1. Access LLDAP admin UI directly (requires VPN / IP-allowlisted access)
# URL: https://lldap.coulomb.social
# Username: break-glass
# Password: from KeePassXC → net-kingdom/Break-glass/break-glass
#
# 2. Or access LLDAP via kubectl exec (no network required)
kubectl exec -n sso deployment/lldap -- /bin/sh
# Inside container: use ldapwhoami / ldapsearch to verify directory state
# 3. Access privacyIDEA admin UI
# URL: https://pink.coulomb.social
# Username: pi-admin
# Password: from KeePassXC → net-kingdom/privacyIDEA/pi-admin
# NOTE: pi-admin has MFA enrolled — if privacyIDEA MFA is down, use:
kubectl exec -n mfa deployment/privacyidea -- pi-manage admin list
```
---
## Restore order
**CRITICAL: Always restore in this order.** Components depend on each other
at startup: privacyIDEA needs PostgreSQL, KeyCape needs all three.
```
1. PostgreSQL (databases ns) — CNPG operator handles restore
2. privacyIDEA (mfa ns) — needs PG + enckey PVC
3. LLDAP (sso ns) — standalone
4. Authelia (sso ns) — needs LLDAP (LDAP bind at startup check)
5. KeyCape (sso ns) — needs Authelia + LLDAP + privacyIDEA
```
---
## Restore from SQLite backup (PVC data corruption)
### LLDAP
```bash
# 1. Scale down LLDAP
kubectl scale deployment/lldap -n sso --replicas=0
# 2. Start a restore pod on the lldap-data PVC
kubectl run -n sso lldap-restore --image=nouchka/sqlite3:latest \
--restart=Never \
--overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"lldap-data"}}],"containers":[{"name":"lldap-restore","image":"nouchka/sqlite3:latest","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}]}}'
# 3. Copy backup file into the pod (or it's already on the PVC under /data/backups/)
kubectl exec -n sso lldap-restore -- ls /data/backups/
# 4. Restore from the chosen backup
kubectl exec -n sso lldap-restore -- \
sqlite3 /data/backups/users.backup.YYYY-MM-DD ".dump" | \
sqlite3 /data/users.db
# 5. Clean up and restart
kubectl delete pod -n sso lldap-restore
kubectl scale deployment/lldap -n sso --replicas=1
kubectl rollout status deployment/lldap -n sso --timeout=120s
```
### Authelia
```bash
# Same pattern as LLDAP, using authelia-data PVC and authelia.backup.YYYY-MM-DD
kubectl scale deployment/authelia -n sso --replicas=0
# ... (run restore pod, restore db.sqlite3, scale back up)
kubectl scale deployment/authelia -n sso --replicas=1
```
### privacyIDEA enckey
```bash
# If the enckey is lost, restore it from KeePassXC binary attachment PI_ENCFILE.
# Extract it to a local file first, then:
kubectl create secret generic privacyidea-enckey \
--from-file=PI_ENCFILE=./pi.enc \
--namespace mfa \
--dry-run=client -o yaml | kubectl apply -f -
# Restart privacyIDEA to pick up the restored key
kubectl rollout restart deployment/privacyidea -n mfa
# If the enckey is truly lost and unrecoverable:
# All enrolled MFA tokens are invalid.
# Generate a new enckey with: kubectl exec -n mfa ... -- pi-manage create_enckey
# All users must re-enroll their TOTP/hardware tokens.
```
---
## Full node restore (new host)
```bash
# Prerequisites on new host:
# - K3s installed
# - Traefik ingress (bundled with K3s)
# - cert-manager installed (helm install cert-manager ...)
# - DNS records pointing to new node IP
# - KeePassXC vault accessible (offline copy or age-encrypted bundle)
# 1. Restore PostgreSQL from CNPG backup
# (See CNPG documentation for cluster restore from barmanObjectStore)
# 2. Re-apply all manifests in order
cd sso-mfa/k8s
kubectl apply -f namespaces/namespaces.yaml
kubectl apply -f network-policies/
kubectl apply -f cert-manager/issuers.yaml
# 3. Restore secrets from KeePassXC
# Run each create-secrets.sh in order:
cd postgresql && ./create-secrets.sh && cd ..
cd privacyidea && ./create-secrets.sh && cd ..
cd lldap && ./create-secrets.sh && cd ..
cd authelia && ./create-secrets.sh && cd ..
cd keycape && ./create-secrets.sh && cd ..
# 4. Apply workloads in restore order
kubectl apply -f postgresql/cluster.yaml
kubectl apply -f privacyidea/{pvc.yaml,configmap.yaml,deployment.yaml,middleware.yaml,ingress.yaml}
kubectl apply -f lldap/{pvc.yaml,deployment.yaml,middleware.yaml,ingress.yaml}
kubectl apply -f authelia/{pvc.yaml,configmap.yaml,deployment.yaml,ingress.yaml}
kubectl apply -f keycape/{deployment.yaml,middleware.yaml,ingress.yaml}
# 5. Wait for everything to be Ready
kubectl rollout status deployment/privacyidea -n mfa --timeout=300s
kubectl rollout status deployment/lldap -n sso --timeout=120s
kubectl rollout status deployment/authelia -n sso --timeout=120s
kubectl rollout status deployment/keycape -n sso --timeout=60s
# 6. Re-run bootstrap scripts if PVC data was lost
cd privacyidea && ./enckey-bootstrap.sh && ./bootstrap-admin.sh && ./bootstrap-realm.sh
cd ../lldap && ./bootstrap-users.sh && ./break-glass.sh
cd ../keycape && ./create-pi-token.sh && ./create-secrets.sh
kubectl rollout restart deployment/keycape -n sso
# 7. Verify
./verify-t04.sh && ./verify-t05.sh && ./verify-t06.sh && ./verify-t07.sh && ./verify-t08.sh
```
---
## Backup offsite export
The SQLite backup files land on the PVCs but are not offsite until exported.
Run this on the node host to pull them out and encrypt for offsite storage:
```bash
# Pull backup files from pods
kubectl exec -n sso deployment/lldap -- \
cat /data/backups/users.backup.$(date +%Y-%m-%d) > /tmp/lldap-backup.db
kubectl exec -n sso deployment/authelia -- \
cat /data/backups/authelia.backup.$(date +%Y-%m-%d) > /tmp/authelia-backup.db
# Encrypt with age and send offsite (same key as the ops bundle)
age -r "$(cat ~/net-kingdom-ops-bundle.key | grep 'public key' | awk '{print $NF}')" \
-o /tmp/lldap-backup.db.age /tmp/lldap-backup.db
# Shred plaintext copies
shred -u /tmp/lldap-backup.db /tmp/authelia-backup.db
```

View File

@@ -0,0 +1,304 @@
# SQLite backup CronJobs — sso and mfa namespaces
#
# Three CronJobs, one per stateful SQLite database:
# 1. lldap-backup — LLDAP user/group store (namespace: sso)
# 2. authelia-backup — Authelia session/storage DB (namespace: sso)
# 3. privacyidea-backup — privacyIDEA token store (namespace: mfa)
#
# Each CronJob runs daily at 03:00 UTC. It uses `sqlite3 .backup` for a
# hot backup that is consistent even while the parent pod is running.
# Backups land on the same PVC next to the live database — to protect
# against pod failure, not PVC failure. Export the backup files offsite
# using pack-bundle.sh or a separate volume snapshot mechanism.
#
# PostgreSQL (privacyIDEA DB) is handled by CNPG ScheduledBackup in
# postgresql/scheduled-backup.yaml. Do not duplicate it here.
#
# Backup file naming:
# <db>.backup.<YYYY-MM-DD> — created daily, pruned after 7 days
#
# Prerequisites:
# - SQLite3 available in the target pod (privacyIDEA and LLDAP images
# include it; Authelia's distroless image does NOT — so Authelia backup
# runs in a separate Job pod with sqlite:alpine image mounted on the PVC).
#
# Apply:
# kubectl apply -f cronjob-sqlite-backups.yaml
---
# ── 1. LLDAP backup (namespace: sso) ─────────────────────────────────────────
# LLDAP includes sqlite3 in its image — run the backup inside the live pod
# via a sidecar-style CronJob that mounts the same PVC.
apiVersion: batch/v1
kind: CronJob
metadata:
name: lldap-backup
namespace: sso
labels:
app.kubernetes.io/name: lldap-backup
app.kubernetes.io/part-of: net-kingdom-sso-mfa
net-kingdom/component: backup
spec:
schedule: "0 3 * * *" # daily at 03:00 UTC
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
metadata:
labels:
app.kubernetes.io/name: lldap-backup
net-kingdom/component: backup
spec:
restartPolicy: OnFailure
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
volumes:
- name: data
persistentVolumeClaim:
claimName: lldap-data
containers:
- name: backup
# Use a lightweight SQLite image — LLDAP's image may not have sqlite3 CLI
image: nouchka/sqlite3:latest
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- |
set -eu
DB=/data/users.db
BACKUP_DIR=/data/backups
DATE=$(date +%Y-%m-%d)
mkdir -p "$BACKUP_DIR"
if [ ! -f "$DB" ]; then
echo "WARN: $DB not found — LLDAP may not have been bootstrapped yet"
exit 0
fi
sqlite3 "$DB" ".backup '$BACKUP_DIR/users.backup.$DATE'"
echo "OK: backed up $DB to $BACKUP_DIR/users.backup.$DATE"
# Prune backups older than 7 days
find "$BACKUP_DIR" -name 'users.backup.*' -mtime +7 -delete
echo "OK: pruned backups older than 7 days"
volumeMounts:
- name: data
mountPath: /data
resources:
requests:
cpu: "10m"
memory: "32Mi"
limits:
cpu: "100m"
memory: "64Mi"
---
# ── 2. Authelia backup (namespace: sso) ──────────────────────────────────────
# Authelia uses a distroless image — run backup in a separate pod on the same PVC.
# NOTE: Authelia uses ReadWriteOnce PVC. The backup pod and Authelia pod cannot
# both mount it simultaneously on most K3s setups. This CronJob scales Authelia
# to 0 replicas, takes the backup, then restores the replica count.
# For production: prefer a storage-level snapshot (Longhorn/Velero) instead.
apiVersion: batch/v1
kind: CronJob
metadata:
name: authelia-backup
namespace: sso
labels:
app.kubernetes.io/name: authelia-backup
app.kubernetes.io/part-of: net-kingdom-sso-mfa
net-kingdom/component: backup
spec:
schedule: "15 3 * * *" # 03:15 UTC — offset from lldap-backup
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
metadata:
labels:
app.kubernetes.io/name: authelia-backup
net-kingdom/component: backup
spec:
restartPolicy: OnFailure
serviceAccountName: backup-sa # needs scale permission — see RBAC below
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
volumes:
- name: data
persistentVolumeClaim:
claimName: authelia-data
initContainers:
# Scale Authelia to 0 to release the PVC before mounting
- name: scale-down
image: bitnami/kubectl:latest
imagePullPolicy: IfNotPresent
command:
- kubectl
- scale
- deployment/authelia
- --replicas=0
- -n
- sso
resources:
requests:
cpu: "10m"
memory: "32Mi"
limits:
cpu: "100m"
memory: "64Mi"
containers:
- name: backup
image: nouchka/sqlite3:latest
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- |
set -eu
DB=/data/db.sqlite3
BACKUP_DIR=/data/backups
DATE=$(date +%Y-%m-%d)
mkdir -p "$BACKUP_DIR"
if [ ! -f "$DB" ]; then
echo "WARN: $DB not found — Authelia may not have been bootstrapped yet"
else
sqlite3 "$DB" ".backup '$BACKUP_DIR/authelia.backup.$DATE'"
echo "OK: backed up $DB to $BACKUP_DIR/authelia.backup.$DATE"
find "$BACKUP_DIR" -name 'authelia.backup.*' -mtime +7 -delete
echo "OK: pruned backups older than 7 days"
fi
# Always scale Authelia back up, even on backup failure
kubectl scale deployment/authelia --replicas=1 -n sso || true
volumeMounts:
- name: data
mountPath: /data
resources:
requests:
cpu: "10m"
memory: "32Mi"
limits:
cpu: "100m"
memory: "64Mi"
---
# ── 3. privacyIDEA backup (namespace: mfa) ───────────────────────────────────
# privacyIDEA's enckey and token store live in the PVC.
# The SQLite database (if configured) and enckey are both backed up here.
# NOTE: The main PI database is PostgreSQL (handled by CNPG). This backs up
# the PI_ENCFILE (encryption key) stored on the PVC and any local config files.
apiVersion: batch/v1
kind: CronJob
metadata:
name: privacyidea-backup
namespace: mfa
labels:
app.kubernetes.io/name: privacyidea-backup
app.kubernetes.io/part-of: net-kingdom-sso-mfa
net-kingdom/component: backup
spec:
schedule: "30 3 * * *" # 03:30 UTC — offset from previous jobs
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
metadata:
labels:
app.kubernetes.io/name: privacyidea-backup
net-kingdom/component: backup
spec:
restartPolicy: OnFailure
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
volumes:
- name: data
persistentVolumeClaim:
claimName: privacyidea-data
containers:
- name: backup
image: busybox:stable
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- |
set -eu
BACKUP_DIR=/data/backups
DATE=$(date +%Y-%m-%d)
mkdir -p "$BACKUP_DIR"
# Back up the enckey — this is the most critical file on this PVC.
# Loss of enckey = all enrolled MFA tokens become invalid.
if [ -f /data/enckey ]; then
cp /data/enckey "$BACKUP_DIR/enckey.backup.$DATE"
echo "OK: backed up enckey to $BACKUP_DIR/enckey.backup.$DATE"
else
echo "WARN: /data/enckey not found — enckey-bootstrap.sh may not have run yet"
fi
# Back up any local config files
if [ -f /data/privacyidea.cfg ]; then
cp /data/privacyidea.cfg "$BACKUP_DIR/privacyidea.cfg.backup.$DATE"
fi
# Prune files older than 7 days
find "$BACKUP_DIR" \( -name 'enckey.backup.*' -o -name '*.cfg.backup.*' \) \
-mtime +7 -delete
echo "OK: pruned backups older than 7 days"
volumeMounts:
- name: data
mountPath: /data
resources:
requests:
cpu: "10m"
memory: "16Mi"
limits:
cpu: "50m"
memory: "32Mi"
---
# ── RBAC for backup-sa (Authelia scale-down/up) ───────────────────────────────
apiVersion: v1
kind: ServiceAccount
metadata:
name: backup-sa
namespace: sso
labels:
app.kubernetes.io/part-of: net-kingdom-sso-mfa
net-kingdom/component: backup
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: backup-scaler
namespace: sso
labels:
app.kubernetes.io/part-of: net-kingdom-sso-mfa
net-kingdom/component: backup
rules:
- apiGroups: ["apps"]
resources: ["deployments/scale", "deployments"]
verbs: ["get", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: backup-sa-scaler
namespace: sso
labels:
app.kubernetes.io/part-of: net-kingdom-sso-mfa
net-kingdom/component: backup
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: backup-scaler
subjects:
- kind: ServiceAccount
name: backup-sa
namespace: sso