coulomb/net-kingdom
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(t09): backup, break-glass, DR drill — NK-WP-0003-T09 done

Browse Source 
- Apply SQLite backup CronJobs (LLDAP, Authelia, privacyIDEA) — all verified running
- Fix authelia-backup: remove scale-down/up dance; concurrent local-path PVC mount
  works on single-node k3s, sqlite3 .backup is safe for concurrent access
- Fix privacyidea-backup: add supplementalGroups: [999] so uid=1000 can read enckey
- Add allow-backup-to-kube-api NetworkPolicy (backup pod → 10.43.0.1:443)
- Create break-glass LLDAP account (net-kingdom-admins); fix ((PASS++)) set-e trap
- SQLite restore drill: LLDAP backup valid (2 users, all tables)
- verify-t08.sh: PASS=15, FAIL=0; fix counter bug + enckey PVC path (/etc/privacyidea)
- Update DR-RUNBOOK.md Authelia restore procedure
- T09 deferred: CNPG backup (needs MinIO/S3), Prometheus (needs kube-prometheus-stack)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-03-25 23:56:40 +00:00
parent 4c47c9035f
commit c054241a5c
6 changed files with 72 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
sso-mfa/k8s/backup/DR-RUNBOOK.md
View File

@@ -87,10 +87,20 @@ kubectl rollout status deployment/lldap -n sso --timeout=120s
### Authelia
```bash
# Same pattern as LLDAP, using authelia-data PVC and authelia.backup.YYYY-MM-DD
# On single-node k3s (local-path PVCs are hostPath-backed), a restore pod can mount
# authelia-data alongside the running Authelia pod. Scale down only if you need to
# replace the live db.sqlite3 in-place (Authelia must be stopped to avoid corruption).
kubectl scale deployment/authelia -n sso --replicas=0
# ... (run restore pod, restore db.sqlite3, scale back up)
kubectl run -n sso authelia-restore --image=nouchka/sqlite3:latest \
--restart=Never \
--overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"authelia-data"}}],"containers":[{"name":"authelia-restore","image":"nouchka/sqlite3:latest","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}]}}'
kubectl exec -n sso authelia-restore -- ls /data/backups/
kubectl exec -n sso authelia-restore -- \
sqlite3 /data/backups/authelia.backup.YYYY-MM-DD ".dump" | \
sqlite3 /data/db.sqlite3
kubectl delete pod -n sso authelia-restore
kubectl scale deployment/authelia -n sso --replicas=1
kubectl rollout status deployment/authelia -n sso --timeout=120s
```
### privacyIDEA enckey

44
sso-mfa/k8s/backup/cronjob-sqlite-backups.yaml
View File

@@ -96,11 +96,12 @@ spec:
---
# ── 2. Authelia backup (namespace: sso) ──────────────────────────────────────
# Authelia uses a distroless image — run backup in a separate pod on the same PVC.
# NOTE: Authelia uses ReadWriteOnce PVC. The backup pod and Authelia pod cannot
# both mount it simultaneously on most K3s setups. This CronJob scales Authelia
# to 0 replicas, takes the backup, then restores the replica count.
# For production: prefer a storage-level snapshot (Longhorn/Velero) instead.
# Authelia uses a distroless image — backup runs in a separate pod on the same PVC.
#
# On a single-node k3s cluster, local-path (hostPath-backed) PVCs can be mounted
# by multiple pods on the same node simultaneously, even with accessMode: RWO.
# SQLite's `.backup` command is safe for concurrent use (uses shared-mode locking).
# This means we do NOT need to scale Authelia down — just mount and backup directly.
apiVersion: batch/v1
kind: CronJob
metadata:
@@ -124,7 +125,6 @@ spec:
net-kingdom/component: backup
spec:
restartPolicy: OnFailure
serviceAccountName: backup-sa # needs scale permission — see RBAC below
securityContext:
runAsNonRoot: true
runAsUser: 1000
@@ -133,25 +133,6 @@ spec:
- name: data
persistentVolumeClaim:
claimName: authelia-data
initContainers:
# Scale Authelia to 0 to release the PVC before mounting
- name: scale-down
image: bitnami/kubectl:latest
imagePullPolicy: IfNotPresent
command:
- kubectl
- scale
- deployment/authelia
- --replicas=0
- -n
- sso
resources:
requests:
cpu: "10m"
memory: "32Mi"
limits:
cpu: "100m"
memory: "64Mi"
containers:
- name: backup
image: nouchka/sqlite3:latest
@@ -167,14 +148,12 @@ spec:
mkdir -p "$BACKUP_DIR"
if [ ! -f "$DB" ]; then
echo "WARN: $DB not found — Authelia may not have been bootstrapped yet"
else
sqlite3 "$DB" ".backup '$BACKUP_DIR/authelia.backup.$DATE'"
echo "OK: backed up $DB to $BACKUP_DIR/authelia.backup.$DATE"
find "$BACKUP_DIR" -name 'authelia.backup.*' -mtime +7 -delete
echo "OK: pruned backups older than 7 days"
exit 0
fi
# Always scale Authelia back up, even on backup failure
kubectl scale deployment/authelia --replicas=1 -n sso || true
sqlite3 "$DB" ".backup '$BACKUP_DIR/authelia.backup.$DATE'"
echo "OK: backed up $DB to $BACKUP_DIR/authelia.backup.$DATE"
find "$BACKUP_DIR" -name 'authelia.backup.*' -mtime +7 -delete
echo "OK: pruned backups older than 7 days"
volumeMounts:
- name: data
mountPath: /data
@@ -219,6 +198,7 @@ spec:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
supplementalGroups: [999] # PI PVC files are group 999 (privacyidea gid)
volumes:
- name: data
persistentVolumeClaim:

4
sso-mfa/k8s/lldap/break-glass.sh
View File

@@ -32,8 +32,8 @@ BG_DISPLAY="Break-glass Account"
PASS_COUNT=0
FAIL_COUNT=0
ok() { echo " [OK] $1"; ((PASS_COUNT++)); }
fail() { echo " [FAIL] $1"; ((FAIL_COUNT++)); }
ok() { echo " [OK] $1"; PASS_COUNT=$((PASS_COUNT + 1)); }
fail() { echo " [FAIL] $1"; FAIL_COUNT=$((FAIL_COUNT + 1)); }
info() { echo " [INFO] $1"; }
for f in "$LLDAP_ENV" "$BG_ENV"; do

24
sso-mfa/k8s/network-policies/netpol-sso.yaml
View File

@@ -297,6 +297,30 @@ spec:
- port: 8089
protocol: TCP
---
# ── Allow backup pod egress to Kubernetes API server ─────────────────────────
# The authelia-backup CronJob pod (net-kingdom/component=backup) needs to call
# kubectl scale to restore Authelia after taking the backup.
# kube-apiserver ClusterIP: 10.43.0.1 (k3s default service CIDR)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-backup-to-kube-api
namespace: sso
spec:
podSelector:
matchLabels:
net-kingdom/component: backup
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 10.43.0.1/32
ports:
- port: 443
protocol: TCP
---
# ── Allow egress DNS (all pods) ──────────────────────────────────────────────
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy

9
sso-mfa/k8s/verify-t08.sh
View File

@@ -28,9 +28,9 @@ PASS=0
FAIL=0
WARN=0
pass() { echo " [PASS] $1"; ((PASS++)); }
fail() { echo " [FAIL] $1"; ((FAIL++)); }
warn() { echo " [WARN] $1"; ((WARN++)); }
pass() { echo " [PASS] $1"; PASS=$((PASS + 1)); }
fail() { echo " [FAIL] $1"; FAIL=$((FAIL + 1)); }
warn() { echo " [WARN] $1"; WARN=$((WARN + 1)); }
section() { echo ""; echo "── $1 ──────────────────────────────────────"; }
@@ -107,8 +107,9 @@ PI_POD=$(kubectl get pod -n "$MFA_NAMESPACE" \
--field-selector=status.phase=Running \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
if [[ -n "$PI_POD" ]]; then
# PI PVC is mounted at /etc/privacyidea (not /data) in the privacyIDEA container
BACKUP_COUNT=$(kubectl exec -n "$MFA_NAMESPACE" "$PI_POD" -- \
sh -c 'ls /data/backups/enckey.backup.* 2>/dev/null | wc -l' 2>/dev/null || echo "0")
sh -c 'ls /etc/privacyidea/backups/enckey.backup.* 2>/dev/null | wc -l' 2>/dev/null || echo "0")
BACKUP_COUNT="${BACKUP_COUNT// /}"
if [[ "$BACKUP_COUNT" -gt 0 ]]; then
pass "privacyIDEA enckey backups found on PVC ($BACKUP_COUNT file(s))"