coulomb/net-kingdom
Archived
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(t09): backup, break-glass, DR drill — NK-WP-0003-T09 done

Browse Source 
- Apply SQLite backup CronJobs (LLDAP, Authelia, privacyIDEA) — all verified running
- Fix authelia-backup: remove scale-down/up dance; concurrent local-path PVC mount
  works on single-node k3s, sqlite3 .backup is safe for concurrent access
- Fix privacyidea-backup: add supplementalGroups: [999] so uid=1000 can read enckey
- Add allow-backup-to-kube-api NetworkPolicy (backup pod → 10.43.0.1:443)
- Create break-glass LLDAP account (net-kingdom-admins); fix ((PASS++)) set-e trap
- SQLite restore drill: LLDAP backup valid (2 users, all tables)
- verify-t08.sh: PASS=15, FAIL=0; fix counter bug + enckey PVC path (/etc/privacyidea)
- Update DR-RUNBOOK.md Authelia restore procedure
- T09 deferred: CNPG backup (needs MinIO/S3), Prometheus (needs kube-prometheus-stack)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-03-25 23:56:40 +00:00
parent 4c47c9035f
commit c054241a5c
6 changed files with 72 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
sso-mfa/k8s/backup/cronjob-sqlite-backups.yaml
View File

@@ -96,11 +96,12 @@ spec:
---
# ── 2. Authelia backup (namespace: sso) ──────────────────────────────────────
# Authelia uses a distroless image — run backup in a separate pod on the same PVC.
# NOTE: Authelia uses ReadWriteOnce PVC. The backup pod and Authelia pod cannot
# both mount it simultaneously on most K3s setups. This CronJob scales Authelia
# to 0 replicas, takes the backup, then restores the replica count.
# For production: prefer a storage-level snapshot (Longhorn/Velero) instead.
# Authelia uses a distroless image — backup runs in a separate pod on the same PVC.
#
# On a single-node k3s cluster, local-path (hostPath-backed) PVCs can be mounted
# by multiple pods on the same node simultaneously, even with accessMode: RWO.
# SQLite's `.backup` command is safe for concurrent use (uses shared-mode locking).
# This means we do NOT need to scale Authelia down — just mount and backup directly.
apiVersion: batch/v1
kind: CronJob
metadata:
@@ -124,7 +125,6 @@ spec:
net-kingdom/component: backup
spec:
restartPolicy: OnFailure
serviceAccountName: backup-sa # needs scale permission — see RBAC below
securityContext:
runAsNonRoot: true
runAsUser: 1000
@@ -133,25 +133,6 @@ spec:
- name: data
persistentVolumeClaim:
claimName: authelia-data
initContainers:
# Scale Authelia to 0 to release the PVC before mounting
- name: scale-down
image: bitnami/kubectl:latest
imagePullPolicy: IfNotPresent
command:
- kubectl
- scale
- deployment/authelia
- --replicas=0
- -n
- sso
resources:
requests:
cpu: "10m"
memory: "32Mi"
limits:
cpu: "100m"
memory: "64Mi"
containers:
- name: backup
image: nouchka/sqlite3:latest
@@ -167,14 +148,12 @@ spec:
mkdir -p "$BACKUP_DIR"
if [ ! -f "$DB" ]; then
echo "WARN: $DB not found — Authelia may not have been bootstrapped yet"
else
sqlite3 "$DB" ".backup '$BACKUP_DIR/authelia.backup.$DATE'"
echo "OK: backed up $DB to $BACKUP_DIR/authelia.backup.$DATE"
find "$BACKUP_DIR" -name 'authelia.backup.*' -mtime +7 -delete
echo "OK: pruned backups older than 7 days"
exit 0
fi
# Always scale Authelia back up, even on backup failure
kubectl scale deployment/authelia --replicas=1 -n sso || true
sqlite3 "$DB" ".backup '$BACKUP_DIR/authelia.backup.$DATE'"
echo "OK: backed up $DB to $BACKUP_DIR/authelia.backup.$DATE"
find "$BACKUP_DIR" -name 'authelia.backup.*' -mtime +7 -delete
echo "OK: pruned backups older than 7 days"
volumeMounts:
- name: data
mountPath: /data
@@ -219,6 +198,7 @@ spec:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
supplementalGroups: [999] # PI PVC files are group 999 (privacyidea gid)
volumes:
- name: data
persistentVolumeClaim: