From 95656f2324b902cce255b5c9646cc3f05ef8b8a5 Mon Sep 17 00:00:00 2001 From: Bernd Worsch Date: Sat, 21 Mar 2026 08:38:52 +0000 Subject: [PATCH] =?UTF-8?q?feat(creds):=20NK-WP-0005=20=E2=80=94=20agent-d?= =?UTF-8?q?riven=20credential=20bootstrap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements all 7 tasks from NK-WP-0005: T01: creds-state.yaml → schema_version: 2, agent_mode: true Replaces keepass_confirmed with emergency_bundle_delivered, adds phase tracking fields for fully automated flow. T02: creds-bootstrap-agent.sh — single entrypoint for autonomous bootstrap. 10 phases, idempotent re-runs via state file. Only human touchpoint: emergency bundle confirmation gate. T03: emergency-bundle.sh — assembles and displays emergency bundle (age key + break-glass passwords + ops bundle location). Writes temp file, shreds on confirmation, clears screen. Supports --reprint for re-delivery. T04: ~/.claude/commands/creds-init.md — /creds-init skill replaces /creds-bootstrap. Fully autonomous execution via the agent. T05: Makefile — creds-agent-init, creds-agent-status, creds-emergency-reprint targets. T06: creds-rotate.sh — --non-interactive flag for agent-driven rotation. Auto-confirms all gates; tracks last_rotated_ in creds-state.yaml. LLDAP web UI step prints warning in non-interactive mode. T07: canon/standards/credential-management_v0.2.md — updated standard: KeePassXC removed from operational path, agent bootstrap as Phase 0, emergency bundle section, prohibited patterns updated. Also: creds-status.sh handles both schema v1 (legacy) and v2. Co-Authored-By: Claude Sonnet 4.6 --- Makefile | 14 +- canon/standards/credential-management_v0.2.md | 230 +++++++++++ sso-mfa/bootstrap/creds-bootstrap-agent.sh | 363 ++++++++++++++++++ sso-mfa/bootstrap/creds-rotate.sh | 85 +++- sso-mfa/bootstrap/creds-state.yaml | 44 ++- sso-mfa/bootstrap/creds-status.sh | 110 ++++-- sso-mfa/bootstrap/emergency-bundle.sh | 147 +++++++ 7 files changed, 931 insertions(+), 62 deletions(-) create mode 100644 canon/standards/credential-management_v0.2.md create mode 100755 sso-mfa/bootstrap/creds-bootstrap-agent.sh create mode 100755 sso-mfa/bootstrap/emergency-bundle.sh diff --git a/Makefile b/Makefile index 8fc2bce..066d7b6 100644 --- a/Makefile +++ b/Makefile @@ -139,6 +139,18 @@ creds-rotate: ## Guided rotation for one secret: make creds-rotate SECRET= exit 1) SECRET=$(SECRET) bash sso-mfa/bootstrap/creds-rotate.sh +## ── Agent-driven credential lifecycle (NK-WP-0005) ────────────────────────── + +creds-agent-init: ## Fully automated credential bootstrap — generates, encrypts, injects, delivers emergency bundle + @bash sso-mfa/bootstrap/creds-bootstrap-agent.sh + +creds-agent-status: ## Show current v2 bootstrap state (agent mode) + @bash sso-mfa/bootstrap/creds-status.sh --v2 + +creds-emergency-reprint: ## Re-deliver emergency bundle (if lost/stolen — reprints, rotates nothing) + @bash sso-mfa/bootstrap/emergency-bundle.sh --reprint + .PHONY: help hooks hooks-test sops-setup sops-edit sops-encrypt sops-decrypt sops-rotate \ check-secrets creds-init creds-generate creds-bundle creds-apply creds-verify \ - creds-status creds-rotate + creds-status creds-rotate \ + creds-agent-init creds-agent-status creds-emergency-reprint diff --git a/canon/standards/credential-management_v0.2.md b/canon/standards/credential-management_v0.2.md new file mode 100644 index 0000000..0d76cdd --- /dev/null +++ b/canon/standards/credential-management_v0.2.md @@ -0,0 +1,230 @@ +# Credential Management Standard — net-kingdom +**Version:** 0.2 **Status:** current **Supersedes:** v0.1 (retired with NK-WP-0004) + +--- + +## 1. Purpose + +Define how service credentials are generated, stored, rotated, and recovered +in the net-kingdom SSO/MFA platform. This standard governs operational +security of all secrets used by the Authelia + LLDAP + KeyCape + privacyIDEA +stack and its PostgreSQL backend. + +--- + +## 2. Trust Hierarchy + +``` +age private key ─────────────► encrypts secrets.enc/ in git + │ + └──► ops bundle (encrypted snapshot of all secrets) + └──► stored offsite; decrypt with age key + +break-glass passwords ────────► direct service access if cluster/auth is down + │ + └──► stored in human's personal password manager + +K8s Secrets ──────────────────► live credential store for running services + │ + └──► created by create-secrets.sh scripts; sourced from secrets.enc/ +``` + +**KeePassXC is NOT in the operational path.** If you choose to import the +emergency bundle into KeePassXC for personal use, that is your business — it +is not required or assumed by any tooling in this repo. + +The age private key and SOPS/age-encrypted git files are the credential store. +The ops bundle is the backup. The emergency bundle is the human's key ring. + +--- + +## 3. Credential Lifecycle + +### Phase 0 — Bootstrap (run once, agent-driven) + +``` +make creds-agent-init +``` + +This single command runs the full bootstrap end-to-end: +1. Verifies prerequisites (age, kubectl, openssl, cluster) +2. Generates or verifies the age keypair at `~/.config/sops/age/keys.txt` +3. Generates all service secrets via `gen-secrets.sh` +4. Encrypts them to `secrets.enc/` with age and commits +5. Injects them into the cluster via `creds-apply.sh` +6. Verifies all K8s Secrets exist +7. Waits for privacyIDEA to be Ready, then runs enckey bootstrap + admin creation +8. Applies KeyCape secrets (requires pi-admin) +9. Creates the ops bundle (age-encrypted snapshot) +10. Delivers the emergency bundle to the terminal for human storage + ← **only human touchpoint** +11. Shreds all plaintext and marks `bootstrap_complete: true` + +The script resumes from where it left off if interrupted — each phase is +tracked in `creds-state.yaml`. + +### Phase 1 — Normal operation + +No human credential management is needed after bootstrap. All secrets live in: +- `secrets.enc/` — encrypted in git (decrypt with age key) +- K8s Secrets — live cluster state (updated by `creds-apply.sh`) + +### Phase 2 — Rotation + +``` +make creds-rotate SECRET= # guided interactive mode +SECRET= bash sso-mfa/bootstrap/creds-rotate.sh --non-interactive # agent mode +``` + +Rotation is handled per-secret with appropriate atomicity guarantees. +See Section 5 for details. + +### Phase 3 — Recovery + +Use the emergency bundle stored in your personal password manager. +See Section 4 — Emergency Bundle. + +--- + +## 4. Emergency Bundle + +### Contents + +| Item | Purpose | +|------|---------| +| age private key | Decrypt any `secrets.enc/*.age` file from git | +| privacyIDEA admin password | Direct access to privacyIDEA if cluster/auth is down | +| LLDAP admin password | Direct LDAP directory access | +| PostgreSQL root password | Direct database access | +| break-glass user password | Emergency login if Authelia/KeyCape is down | +| ops bundle location + decrypt command | Point-in-time snapshot of all secrets | + +### Delivery + +The emergency bundle is displayed once in the terminal at the end of +`make creds-agent-init`. The operator must copy it into their personal +password manager before pressing Enter to continue. + +The confirmation gate is a **deliberate security control** — bootstrap +does not complete until the human confirms receipt. + +### Re-delivery + +If the emergency bundle is lost or stolen: + +``` +make creds-emergency-reprint +``` + +This reprints the bundle from the current age key and secrets. +It does **not** rotate any secrets — if you suspect compromise, rotate +the affected secrets separately with `make creds-rotate`. + +### Storage recommendations + +Store the emergency bundle in one or more of: +- 1Password, Bitwarden, KeePassXC, or equivalent +- Encrypted offline storage +- Printed and physically secured (air-gapped scenarios) + +The agent does not care which — it only cares that you confirm receipt. + +### Recovery procedure + +1. Restore `~/.config/sops/age/keys.txt` from the emergency bundle +2. Clone net-kingdom repo +3. Run: `make creds-apply` — re-injects all secrets from `secrets.enc/` +4. Use break-glass passwords for direct service access if needed + +--- + +## 5. Secret Rotation + +### Rotatable secrets + +| Secret | Blast radius | Notes | +|--------|-------------|-------| +| `PI_SECRET_KEY` | Low — invalidates PI sessions | Safe to rotate anytime | +| `PI_DB_PASSWORD` | Medium — DB + pod | Atomic update required | +| `LLDAP_JWT_SECRET` | Low — invalidates LLDAP sessions | Safe to rotate anytime | +| `LLDAP_LDAP_USER_PASS` | High — 3-way coordinated | Authelia + KeyCape + LLDAP web UI | +| `AUTHELIA_SESSION_SECRET` | Low — invalidates sessions | Safe to rotate anytime | +| `AUTHELIA_KEYCAPE_CLIENT_SECRET` | Medium — 2-way atomic | Authelia + KeyCape together | +| `KEYCAPE_RSA_KEY` | High — invalidates all JWTs | Causes brief auth outage | +| `BREAKGLASS_PASSWORD` | Minimal | Rotate freely | + +### Non-rotatable + +| Secret | Reason | +|--------|--------| +| `PI_PEPPER` | Rotating requires re-hashing all PI user passwords | + +### Age key rotation + +Rotating the age private key is a special case: +1. Generate a new age key +2. Re-encrypt all `secrets.enc/` files with the new public key (`make sops-rotate`) +3. Commit +4. A **new emergency bundle must be delivered** before the old key is revoked + (see Section 4) + +--- + +## 6. Ops Bundle + +The ops bundle is an age-encrypted tar archive of all plaintext secrets at a +point in time. It is created automatically during bootstrap and can be +refreshed with: + +``` +make creds-bundle +``` + +Store the bundle offsite (cloud, external drive, second location). Decrypt: + +``` +age -d -i ~/.config/sops/age/keys.txt ops-bundle-.tar.age | tar xf - +``` + +--- + +## 7. Prohibited Patterns + +The following are permanently prohibited: + +1. **Committing `secrets/` (plaintext) to git** — the pre-commit hook blocks + this; do not bypass with `--no-verify` + +2. **Storing secrets in KeePassXC as the primary credential store** — KeePassXC + is for optional personal backup of the emergency bundle only + +3. **Skipping the `emergency_bundle_delivered` confirmation gate** — even in + non-interactive or automated runs, the agent MUST NOT mark bootstrap + complete until the human confirms receipt. This gate exists to ensure + break-glass access is always available. + +4. **Hardcoding secrets in manifests or ConfigMaps** — all service secrets + must flow through K8s Secrets created by `create-secrets.sh` scripts + +5. **Storing the age private key in the repo** — the key lives outside the + repo at `~/.config/sops/age/keys.txt` + +--- + +## Appendix A — KeePassXC Group Structure (optional) + +If you choose to import the emergency bundle into KeePassXC, a suggested +group structure for the break-glass passwords is: + +``` +net-kingdom/ + Break-glass/ + privacyIDEA admin username=pi-admin password= + LLDAP admin username=admin password= + PostgreSQL root username=postgres password= + break-glass user username=break-glass password= + age key (attach keys.txt as binary attachment) + ops bundle decrypt (store path + decrypt command as a note) +``` + +This is entirely optional — the agent does not read from or write to KeePassXC. diff --git a/sso-mfa/bootstrap/creds-bootstrap-agent.sh b/sso-mfa/bootstrap/creds-bootstrap-agent.sh new file mode 100755 index 0000000..9829add --- /dev/null +++ b/sso-mfa/bootstrap/creds-bootstrap-agent.sh @@ -0,0 +1,363 @@ +#!/usr/bin/env bash +# creds-bootstrap-agent.sh — fully automated credential bootstrap (NK-WP-0005) +# +# Usage: +# bash sso-mfa/bootstrap/creds-bootstrap-agent.sh [--dry-run] [--resume] +# make creds-agent-init +# +# Runs end-to-end without human input until the emergency bundle confirmation +# gate. Each phase updates creds-state.yaml so interrupted runs resume +# automatically from where they left off. +# +# Prerequisites: +# - age (apt install age) +# - kubectl with a reachable cluster (KUBECONFIG set or ~/.kube/config) +# - git (configured with commit access) +# - openssl +# - ~/.config/sops/age/keys.txt — age private key (generated here if missing) + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +K8S_DIR="$REPO_ROOT/sso-mfa/k8s" +SECRETS_DIR="$SCRIPT_DIR/secrets" +STATE_FILE="$SCRIPT_DIR/creds-state.yaml" +AGE_KEY="$HOME/.config/sops/age/keys.txt" + +DRY_RUN=false +for arg in "$@"; do [[ "$arg" == "--dry-run" ]] && DRY_RUN=true; done + +# ── Helpers ─────────────────────────────────────────────────────────────────── + +log() { echo " [bootstrap] $*"; } +step() { echo ""; echo "══════════════════════════════════════════════════════"; echo " Phase $*"; echo "══════════════════════════════════════════════════════"; } +ok() { echo " ✔ $*"; } +warn() { echo " ⚠ $*"; } +die() { echo ""; echo " ERROR: $*" >&2; exit 1; } + +dry_run_guard() { + if [[ "$DRY_RUN" == true ]]; then + echo " [dry-run] would run: $*" + return 0 + fi + "$@" +} + +# Read a top-level value from creds-state.yaml +state_get() { grep -E "^$1:" "$STATE_FILE" | sed 's/^[^:]*: *//' | sed 's/ *#.*//' | tr -d '"'; } +state_get_nested() { grep -E "^ $1:" "$STATE_FILE" | sed 's/^[^:]*: *//' | sed 's/ *#.*//' | tr -d '"'; } + +# Update a top-level key in creds-state.yaml +state_set() { + local key="$1" value="$2" + if [[ -f "$STATE_FILE" ]]; then + sed -i "s|^$key: .*|$key: $value|" "$STATE_FILE" + fi +} + +# Update a nested (2-space indent) key +state_set_nested() { + local key="$1" value="$2" + if [[ -f "$STATE_FILE" ]]; then + sed -i "s|^ $key: .*| $key: $value|" "$STATE_FILE" + fi +} + +# ── Pre-flight ──────────────────────────────────────────────────────────────── + +step "0 — Pre-flight" + +command -v age >/dev/null 2>&1 || die "age not installed (apt install age)" +command -v kubectl >/dev/null 2>&1 || die "kubectl not found — install it and configure KUBECONFIG" +command -v git >/dev/null 2>&1 || die "git not found" +command -v openssl >/dev/null 2>&1 || die "openssl not found" +ok "required tools present" + +# Age key — generate if missing +if [[ ! -f "$AGE_KEY" ]]; then + log "age key not found at $AGE_KEY — generating..." + mkdir -p "$(dirname "$AGE_KEY")" + if [[ "$DRY_RUN" == false ]]; then + age-keygen -o "$AGE_KEY" 2>/dev/null + chmod 600 "$AGE_KEY" + ok "age key generated: $AGE_KEY" + log "Public key: $(grep 'public key:' "$AGE_KEY" | awk '{print $NF}')" + else + echo " [dry-run] would run: age-keygen -o $AGE_KEY" + fi +fi + +AGE_PUBKEY=$(grep 'public key:' "$AGE_KEY" | awk '{print $NF}') +[[ -z "$AGE_PUBKEY" ]] && die "could not read public key from $AGE_KEY" +ok "age key ready: ${AGE_PUBKEY:0:20}…" +state_set "age_key_present" "true" + +# Cluster reachability +if ! kubectl cluster-info &>/dev/null; then + die "Cannot reach the Kubernetes cluster. Check KUBECONFIG / cluster status." +fi +KUBE_CTX=$(kubectl config current-context 2>/dev/null || echo '(unknown)') +ok "cluster reachable: $KUBE_CTX" + +# ── Phase 1: Generate secrets ───────────────────────────────────────────────── + +step "1 — Generate secrets" + +if [[ "$(state_get secrets_generated)" == "true" ]]; then + ok "secrets already generated — skipping" +else + # Clean up any partial generation from a failed prior run + if [[ -d "$SECRETS_DIR" ]]; then + warn "leftover secrets/ found from previous run — removing" + find "$SECRETS_DIR" -type f -exec shred -u {} \; 2>/dev/null || true + rm -rf "$SECRETS_DIR" + fi + + log "running gen-secrets.sh..." + if [[ "$DRY_RUN" == false ]]; then + (cd "$SCRIPT_DIR" && bash gen-secrets.sh "$SECRETS_DIR") + ok "secrets generated in $SECRETS_DIR" + state_set "secrets_generated" "true" + else + echo " [dry-run] would run: bash gen-secrets.sh $SECRETS_DIR" + fi +fi + +# ── Phase 2: Encrypt + commit ───────────────────────────────────────────────── + +step "2 — Encrypt secrets to secrets.enc/ and commit" + +# Re-check: if secrets_generated is true but secrets/ is gone, decrypt first +if [[ "$(state_get secrets_generated)" == "true" && ! -d "$SECRETS_DIR" ]]; then + log "secrets/ absent (was shredded) — decrypting from secrets.enc/ for re-apply..." + if [[ "$DRY_RUN" == false ]]; then + (cd "$SCRIPT_DIR" && bash decrypt-secrets.sh "$SECRETS_DIR" "$HOME/.config/net-kingdom/age.key" 2>/dev/null) \ + || (cd "$SCRIPT_DIR" && bash decrypt-secrets.sh "$SECRETS_DIR" "$AGE_KEY") + else + echo " [dry-run] would decrypt secrets.enc/ → secrets/" + fi +fi + +# Always (re-)encrypt in case secrets were just regenerated +if [[ "$DRY_RUN" == false ]]; then + log "encrypting secrets → secrets.enc/ ..." + (cd "$SCRIPT_DIR" && bash encrypt-secrets.sh "$SECRETS_DIR" "$AGE_KEY" --no-shred) + ok "secrets encrypted to secrets.enc/" + + # Commit the encrypted secrets + cd "$REPO_ROOT" + if git diff --quiet HEAD sso-mfa/bootstrap/secrets.enc/ 2>/dev/null && \ + git diff --cached --quiet sso-mfa/bootstrap/secrets.enc/ 2>/dev/null; then + ok "secrets.enc/ already committed — no changes" + else + git add sso-mfa/bootstrap/secrets.enc/ sso-mfa/bootstrap/creds-state.yaml + git commit -m "chore(creds): encrypted secrets [agent NK-WP-0005]" + ok "encrypted secrets committed" + fi +else + echo " [dry-run] would encrypt + git commit secrets.enc/" +fi + +# ── Phase 3: Inject into cluster ────────────────────────────────────────────── + +step "3 — Inject secrets into cluster (postgres → lldap → authelia → privacyidea)" + +if [[ "$DRY_RUN" == false ]]; then + (cd "$SCRIPT_DIR" && bash creds-apply.sh "$SECRETS_DIR") + ok "secrets applied to cluster" +else + echo " [dry-run] would run: bash creds-apply.sh $SECRETS_DIR" +fi + +# ── Phase 4: Verify initial secrets ─────────────────────────────────────────── + +step "4 — Verify K8s secrets (pre-bootstrap)" + +if [[ "$DRY_RUN" == false ]]; then + # Verify postgres, lldap, authelia, privacyidea (keycape not yet applied) + ALL_OK=true + for ns_secret in "databases/net-kingdom-pg-privacyidea-app" "sso/lldap-secrets" "sso/authelia-secrets" "mfa/privacyidea-config"; do + ns="${ns_secret%%/*}" + name="${ns_secret##*/}" + if kubectl get secret "$name" --namespace="$ns" --ignore-not-found -o name 2>/dev/null | grep -q .; then + ok "secret $ns/$name exists" + else + warn "secret $ns/$name is missing" + ALL_OK=false + fi + done + [[ "$ALL_OK" == true ]] || die "One or more required secrets are missing — check creds-apply output above" +else + echo " [dry-run] would verify K8s secrets" +fi + +# ── Phase 5: Post-apply bootstrap — wait for privacyIDEA ────────────────────── + +step "5 — Post-apply bootstrap (privacyIDEA enckey + admin)" + +NAMESPACE="mfa" +MAX_WAIT=300 # 5 minutes + +if [[ "$(state_get enckey_bootstrapped)" == "true" && "$(state_get pi_admin_created)" == "true" ]]; then + ok "privacyIDEA already bootstrapped — skipping" +else + log "waiting for privacyIDEA pod to be Ready (max ${MAX_WAIT}s)..." + if [[ "$DRY_RUN" == false ]]; then + # Wait for pod to appear and be Ready + WAITED=0 + PI_POD="" + while [[ -z "$PI_POD" && $WAITED -lt $MAX_WAIT ]]; do + PI_POD=$(kubectl get pod -n "$NAMESPACE" \ + -l app.kubernetes.io/name=privacyidea \ + --field-selector=status.phase=Running \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "") + if [[ -z "$PI_POD" ]]; then + sleep 10 + WAITED=$((WAITED + 10)) + log " waiting... (${WAITED}s / ${MAX_WAIT}s)" + fi + done + + [[ -z "$PI_POD" ]] && die "privacyIDEA pod did not reach Running state within ${MAX_WAIT}s — check: kubectl get pods -n $NAMESPACE" + ok "privacyIDEA pod ready: $PI_POD" + + # Run enckey bootstrap + if [[ "$(state_get enckey_bootstrapped)" != "true" ]]; then + log "running enckey-bootstrap.sh..." + (cd "$K8S_DIR/privacyidea" && bash enckey-bootstrap.sh "$SECRETS_DIR") + state_set "enckey_bootstrapped" "true" + ok "enckey bootstrapped" + else + ok "enckey already bootstrapped" + fi + + # Run pi-admin bootstrap + if [[ "$(state_get pi_admin_created)" != "true" ]]; then + log "running bootstrap-admin.sh..." + (cd "$K8S_DIR/privacyidea" && bash bootstrap-admin.sh "$SECRETS_DIR") + state_set "pi_admin_created" "true" + ok "pi-admin created" + else + ok "pi-admin already created" + fi + else + echo " [dry-run] would wait for privacyIDEA pod, run enckey-bootstrap.sh and bootstrap-admin.sh" + fi +fi + +# ── Phase 6: Apply keycape secrets ──────────────────────────────────────────── + +step "6 — Apply KeyCape secrets (requires pi-admin)" + +if [[ "$(state_get_nested keycape)" == "true" ]]; then + ok "keycape secrets already applied — skipping" +else + if [[ "$DRY_RUN" == false ]]; then + log "fetching PI admin token..." + (cd "$K8S_DIR/keycape" && bash create-pi-token.sh "$SECRETS_DIR") + ok "PI admin token fetched" + + log "applying keycape secrets..." + (cd "$K8S_DIR/keycape" && bash create-secrets.sh "$SECRETS_DIR") + state_set_nested "keycape" "true" + ok "keycape secrets applied" + else + echo " [dry-run] would run create-pi-token.sh + keycape/create-secrets.sh" + fi +fi + +# ── Phase 7: Final verification ──────────────────────────────────────────────── + +step "7 — Final verification (all components)" + +if [[ "$DRY_RUN" == false ]]; then + (cd "$SCRIPT_DIR" && bash creds-verify.sh) + ok "all secrets verified" +else + echo " [dry-run] would run: bash creds-verify.sh" +fi + +# ── Phase 8: Ops bundle ──────────────────────────────────────────────────────── + +step "8 — Create ops bundle (age-encrypted snapshot)" + +BUNDLE_NAME="ops-bundle-$(date +%Y%m%dT%H%M%S).tar.age" +BUNDLE_PATH="$REPO_ROOT/$BUNDLE_NAME" + +if [[ "$(state_get ops_bundle_created)" == "true" ]]; then + EXISTING_LOC="$(state_get ops_bundle_location)" + ok "ops bundle already created: ${EXISTING_LOC:-} — skipping" +else + if [[ "$DRY_RUN" == false ]]; then + [[ -d "$SECRETS_DIR" ]] || die "secrets/ not found — cannot create ops bundle (re-run from phase 1)" + log "creating ops bundle → $BUNDLE_PATH" + (cd "$SCRIPT_DIR" && bash pack-bundle.sh "$SECRETS_DIR" "$AGE_PUBKEY" "$BUNDLE_PATH") + state_set "ops_bundle_created" "true" + state_set "ops_bundle_location" "\"$BUNDLE_PATH\"" + ok "ops bundle created: $BUNDLE_PATH" + else + echo " [dry-run] would run: bash pack-bundle.sh $SECRETS_DIR $AGE_PUBKEY $BUNDLE_PATH" + fi +fi + +# ── Phase 9: Emergency bundle ───────────────────────────────────────────────── + +step "9 — Emergency bundle (human confirmation required)" + +if [[ "$(state_get emergency_bundle_delivered)" == "true" ]]; then + ok "emergency bundle already delivered — skipping" +else + if [[ "$DRY_RUN" == false ]]; then + log "assembling emergency bundle..." + OPS_LOC="$(state_get ops_bundle_location | tr -d '"')" + (cd "$SCRIPT_DIR" && bash emergency-bundle.sh \ + --age-key "$AGE_KEY" \ + --secrets-dir "$SECRETS_DIR" \ + --ops-bundle "${OPS_LOC:-$BUNDLE_PATH}") + state_set "emergency_bundle_delivered" "true" + state_set "emergency_bundle_delivered_at" "\"$(date -Iseconds)\"" + ok "emergency bundle delivered and confirmed" + else + echo " [dry-run] would run: bash emergency-bundle.sh ..." + fi +fi + +# ── Phase 10: Cleanup + finalise ────────────────────────────────────────────── + +step "10 — Cleanup and finalise" + +if [[ "$DRY_RUN" == false ]]; then + if [[ -d "$SECRETS_DIR" ]]; then + log "shredding plaintext secrets..." + find "$SECRETS_DIR" -type f -exec shred -u {} \; + rm -rf "$SECRETS_DIR" + ok "plaintext secrets shredded" + fi + + state_set "bootstrap_complete" "true" + + # Commit final state + cd "$REPO_ROOT" + git add sso-mfa/bootstrap/creds-state.yaml + if ! git diff --cached --quiet; then + git commit -m "chore(creds): bootstrap complete [agent NK-WP-0005]" + ok "final state committed" + fi +else + echo " [dry-run] would shred secrets/ and set bootstrap_complete: true" +fi + +# ── Done ────────────────────────────────────────────────────────────────────── + +echo "" +echo "╔══════════════════════════════════════════════════════════════════╗" +echo "║ NET-KINGDOM CREDENTIAL BOOTSTRAP COMPLETE ║" +echo "╚══════════════════════════════════════════════════════════════════╝" +echo "" +echo " All service secrets have been generated, encrypted, committed," +echo " and injected into the cluster. The emergency bundle has been" +echo " delivered to you for storage in your personal password manager." +echo "" +echo " Run 'make creds-agent-status' to review the final state." +echo "" diff --git a/sso-mfa/bootstrap/creds-rotate.sh b/sso-mfa/bootstrap/creds-rotate.sh index 9a9f805..fb450f2 100755 --- a/sso-mfa/bootstrap/creds-rotate.sh +++ b/sso-mfa/bootstrap/creds-rotate.sh @@ -1,25 +1,39 @@ #!/usr/bin/env bash -# creds-rotate.sh — guided rotation for a single net-kingdom credential. +# creds-rotate.sh — guided or non-interactive rotation for a single net-kingdom credential. # # Usage: # SECRET= bash sso-mfa/bootstrap/creds-rotate.sh [secrets-dir] # make creds-rotate SECRET= # +# # Agent / non-interactive mode (NK-WP-0005): +# SECRET= bash sso-mfa/bootstrap/creds-rotate.sh --non-interactive [secrets-dir] +# # The script: # 1. Validates the secret name # 2. Prints rotation impact and required coordination steps # 3. Generates a new value (same entropy as original) -# 4. Guides through the atomic update sequence -# 5. After confirmation, updates creds-state.yaml and reminds to re-bundle +# 4. Guides through the atomic update sequence (interactive) or runs it +# automatically (--non-interactive) +# 5. After completion, updates creds-state.yaml set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -SECRETS_DIR="${1:-$SCRIPT_DIR/secrets}" STATE_FILE="$SCRIPT_DIR/creds-state.yaml" K8S_DIR="$REPO_ROOT/sso-mfa/k8s" +NON_INTERACTIVE=false +POSITIONAL_ARGS=() +for arg in "$@"; do + if [[ "$arg" == "--non-interactive" ]]; then + NON_INTERACTIVE=true + else + POSITIONAL_ARGS+=("$arg") + fi +done + +SECRETS_DIR="${POSITIONAL_ARGS[0]:-$SCRIPT_DIR/secrets}" SECRET="${SECRET:-}" rnd_hex() { openssl rand -hex "$1"; } @@ -27,6 +41,10 @@ rnd_b64() { openssl rand -base64 "$1" | tr -d '\n/+=' | head -c "$2"; } confirm() { local prompt="${1:-Continue?}" + if [[ "$NON_INTERACTIVE" == true ]]; then + echo " [non-interactive] auto-confirming: $prompt" + return 0 + fi echo "" read -rp "$prompt [y/N] " ans [[ "${ans,,}" == "y" ]] @@ -35,16 +53,38 @@ confirm() { header() { echo "" echo "════════════════════════════════════════════════════════" - echo " Rotating: $SECRET" + echo " Rotating: $SECRET$([ "$NON_INTERACTIVE" = true ] && echo " (non-interactive)" || true)" echo "════════════════════════════════════════════════════════" } +update_last_rotated() { + local key="$1" + if [[ -f "$STATE_FILE" ]]; then + # Update or append last_rotated_ in creds-state.yaml + local ts + ts="$(date -Iseconds)" + if grep -qE "^last_rotated_${key}:" "$STATE_FILE"; then + sed -i "s|^last_rotated_${key}: .*|last_rotated_${key}: \"${ts}\"|" "$STATE_FILE" + else + echo "last_rotated_${key}: \"${ts}\"" >> "$STATE_FILE" + fi + echo " [state] last_rotated_${key} → ${ts}" + fi +} + post_rotation_reminder() { echo "" - echo "Post-rotation checklist:" - echo " ✓ Update KeePassXC entry for this secret" - echo " ✓ Run: make creds-bundle (refresh offsite backup)" - echo " ✓ Run: make creds-verify (confirm cluster state)" + if [[ "$NON_INTERACTIVE" == false ]]; then + echo "Post-rotation checklist:" + echo " ✓ Update your personal password manager entry for this secret" + echo " ✓ Run: make creds-bundle (refresh offsite backup)" + echo " ✓ Run: make creds-verify (confirm cluster state)" + else + echo "Post-rotation (agent mode):" + echo " ✓ creds-state.yaml updated with last_rotated timestamp" + echo " Run: make creds-bundle (refresh offsite backup)" + echo " Run: make creds-verify (confirm cluster state)" + fi } # ── Dispatch ────────────────────────────────────────────────────────────────── @@ -68,6 +108,7 @@ PI_SECRET_KEY) echo " 2. Restarting privacyIDEA pod..." kubectl rollout restart deployment privacyidea -n mfa kubectl rollout status deployment privacyidea -n mfa + update_last_rotated "PI_SECRET_KEY" post_rotation_reminder ;; @@ -118,6 +159,7 @@ PI_DB_PASSWORD) echo " 3. Restarting privacyIDEA pod..." kubectl rollout restart deployment privacyidea -n mfa kubectl rollout status deployment privacyidea -n mfa + update_last_rotated "PI_DB_PASSWORD" post_rotation_reminder ;; @@ -137,6 +179,7 @@ LLDAP_JWT_SECRET) echo " Restarting LLDAP pod..." kubectl rollout restart deployment lldap -n sso kubectl rollout status deployment lldap -n sso + update_last_rotated "LLDAP_JWT_SECRET" post_rotation_reminder ;; @@ -158,8 +201,13 @@ LLDAP_LDAP_USER_PASS) [[ -f "$ENV_FILE" ]] && sed -i "s|^LLDAP_LDAP_USER_PASS=.*|LLDAP_LDAP_USER_PASS=$NEW_VAL|" "$ENV_FILE" echo " 1. Updating LLDAP admin password via API..." echo " WARN: Automated LLDAP password update not implemented." - echo " Log in to https://lldap.coulomb.social and change the admin password manually." - confirm "Confirm you have updated the LLDAP admin password?" || { echo "Aborting."; exit 1; } + if [[ "$NON_INTERACTIVE" == false ]]; then + echo " Log in to https://lldap.coulomb.social and change the admin password manually." + confirm "Confirm you have updated the LLDAP admin password?" || { echo "Aborting."; exit 1; } + else + echo " [non-interactive] Skipping LLDAP web UI step — update manually:" + echo " Log in to https://lldap.coulomb.social and set admin password to: $NEW_VAL" + fi echo " 2. Updating Authelia secrets..." (cd "$K8S_DIR/authelia" && bash create-secrets.sh "$SECRETS_DIR") echo " 3. Updating KeyCape secrets..." @@ -169,6 +217,7 @@ LLDAP_LDAP_USER_PASS) kubectl rollout restart deployment keycape -n sso kubectl rollout status deployment authelia -n sso kubectl rollout status deployment keycape -n sso + update_last_rotated "LLDAP_LDAP_USER_PASS" post_rotation_reminder ;; @@ -187,6 +236,7 @@ AUTHELIA_SESSION_SECRET) (cd "$K8S_DIR/authelia" && bash create-secrets.sh "$SECRETS_DIR") kubectl rollout restart deployment authelia -n sso kubectl rollout status deployment authelia -n sso + update_last_rotated "AUTHELIA_SESSION_SECRET" post_rotation_reminder ;; @@ -214,6 +264,7 @@ AUTHELIA_KEYCAPE_CLIENT_SECRET) kubectl rollout restart deployment keycape -n sso kubectl rollout status deployment authelia -n sso kubectl rollout status deployment keycape -n sso + update_last_rotated "AUTHELIA_KEYCAPE_CLIENT_SECRET" post_rotation_reminder ;; @@ -248,7 +299,12 @@ KEYCAPE_RSA_KEY) kubectl rollout restart deployment keycape -n sso kubectl rollout status deployment keycape -n sso echo "" - echo " ✔ RSA key rotated. Store the new key in KeePassXC: net-kingdom/KeyCape/jwt-signing-key" + if [[ "$NON_INTERACTIVE" == false ]]; then + echo " ✔ RSA key rotated. Store the new key in your password manager: net-kingdom/KeyCape/jwt-signing-key" + else + echo " ✔ RSA key rotated." + fi + update_last_rotated "KEYCAPE_RSA_KEY" post_rotation_reminder ;; @@ -271,7 +327,10 @@ BREAKGLASS_PASSWORD) --from-literal=BREAKGLASS_PASSWORD="$NEW_VAL" \ --dry-run=client -o yaml | kubectl apply -f - echo "" - echo " Update KeePassXC entry: net-kingdom/Break-glass/break-glass" + if [[ "$NON_INTERACTIVE" == false ]]; then + echo " Update your password manager: net-kingdom/Break-glass/break-glass" + fi + update_last_rotated "BREAKGLASS_PASSWORD" post_rotation_reminder ;; diff --git a/sso-mfa/bootstrap/creds-state.yaml b/sso-mfa/bootstrap/creds-state.yaml index 21c0a1f..0e79e84 100644 --- a/sso-mfa/bootstrap/creds-state.yaml +++ b/sso-mfa/bootstrap/creds-state.yaml @@ -1,26 +1,32 @@ # Credential state — net-kingdom SSO/MFA stack -# This file is SAFE TO COMMIT. It contains no secrets. -# Updated automatically by make creds-* targets and sso-mfa/bootstrap/creds-verify.sh. -# -# keepass_confirmed is the only field that requires manual operator intervention. -# Set it to true after you have entered all generated secrets into KeePassXC. +# Safe to commit. Contains no secrets. Updated by agent. +# schema_version: 2 = agent-driven model (NK-WP-0005) +# schema_version: 1 = human-as-operator model (NK-WP-0004, now retired) -generated_at: "2026-03-20T02:57:00+00:00" -bundle_at: null -keepass_confirmed: false +schema_version: 2 +agent_mode: true # NK-WP-0005: fully automated +# Phase tracking +age_key_present: false # ~/.config/sops/age/keys.txt exists +secrets_generated: false # gen-secrets.sh ran successfully +ops_bundle_created: false # age-encrypted bundle created +ops_bundle_location: null # path or storage hint + +# Emergency bundle +emergency_bundle_delivered: false # human confirmed receipt +emergency_bundle_delivered_at: null + +# Cluster injection (per-component) secrets_applied: - postgres: false - lldap: false - authelia: false + postgres: false + lldap: false + authelia: false privacyidea: false - # keycape requires PI_ADMIN_TOKEN from post-privacyIDEA T04 bootstrap. - # Run: sso-mfa/k8s/keycape/create-pi-token.sh, then re-run keycape/create-secrets.sh. - keycape: false + keycape: false -# enckey_bootstrapped: set by sso-mfa/k8s/privacyidea/enckey-bootstrap.sh -# This step is TIME-SENSITIVE — it must run while the privacyIDEA pod is live. -enckey_bootstrapped: false +# Post-apply bootstrap (agent-run when pod is Ready) +enckey_bootstrapped: false +pi_admin_created: false -# pi_admin_created: set after sso-mfa/k8s/privacyidea/bootstrap-admin.sh completes -pi_admin_created: false +# Derived: all true → bootstrap complete +bootstrap_complete: false diff --git a/sso-mfa/bootstrap/creds-status.sh b/sso-mfa/bootstrap/creds-status.sh index 1f03fbb..5dec16d 100755 --- a/sso-mfa/bootstrap/creds-status.sh +++ b/sso-mfa/bootstrap/creds-status.sh @@ -2,14 +2,19 @@ # creds-status.sh — print a human-readable credential state table. # # Usage: -# bash sso-mfa/bootstrap/creds-status.sh +# bash sso-mfa/bootstrap/creds-status.sh [state-file] [--v2] # make creds-status +# make creds-agent-status set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" STATE_FILE="${1:-$SCRIPT_DIR/creds-state.yaml}" +# --v2 flag forces v2 display regardless of schema_version field +FORCE_V2=false +for arg in "$@"; do [[ "$arg" == "--v2" ]] && FORCE_V2=true; done + if [[ ! -f "$STATE_FILE" ]]; then echo "ERROR: creds-state.yaml not found: $STATE_FILE" >&2 echo " This file is created at repo init — check your working directory." >&2 @@ -29,37 +34,84 @@ status_icon() { esac } -echo "=== net-kingdom Credential State ===" -echo "" +SCHEMA_VER="$(top_val schema_version)" -generated_at="$(top_val generated_at)" -bundle_at="$(top_val bundle_at)" -keepass_confirmed="$(top_val keepass_confirmed)" +if [[ "$SCHEMA_VER" == "2" || "$FORCE_V2" == "true" ]]; then + # ── Schema v2: agent-driven model (NK-WP-0005) ──────────────────────────── + echo "=== net-kingdom Credential State (v2 — agent mode) ===" + echo "" -printf " %-30s %s\n" "Generated at:" "${generated_at:-—}" -printf " %-30s %s\n" "Bundle at:" "${bundle_at:-—}" -printf " %-30s %s %s\n" "KeePassXC confirmed:" \ - "$(status_icon "$keepass_confirmed")" \ - "$([ "$keepass_confirmed" = "false" ] && echo "(set keepass_confirmed: true manually)" || true)" -echo "" + age_key="$(top_val age_key_present)" + secrets_gen="$(top_val secrets_generated)" + ops_bundle="$(top_val ops_bundle_created)" + ops_loc="$(top_val ops_bundle_location)" + emerg_delivered="$(top_val emergency_bundle_delivered)" + emerg_at="$(top_val emergency_bundle_delivered_at)" + bootstrap_complete="$(top_val bootstrap_complete)" -echo " Secrets applied:" -for component in postgres lldap authelia privacyidea keycape; do - val="$(nested_val "$component")" - note="" - [[ "$component" == "keycape" && "$val" == "false" ]] && \ - note=" (requires PI_ADMIN_TOKEN — post-T04)" - printf " %-28s %s%s\n" "$component" "$(status_icon "$val")" "$note" -done -echo "" + printf " %-32s %s\n" "age key present:" "$(status_icon "$age_key")" + printf " %-32s %s\n" "secrets generated:" "$(status_icon "$secrets_gen")" + printf " %-32s %s %s\n" "ops bundle created:" \ + "$(status_icon "$ops_bundle")" \ + "$([ "$ops_bundle" = "true" ] && echo "${ops_loc:-}" || true)" + printf " %-32s %s %s\n" "emergency bundle delivered:" \ + "$(status_icon "$emerg_delivered")" \ + "$([ "$emerg_delivered" = "true" ] && echo "at ${emerg_at:-}" || echo "(pending human confirmation)")" + echo "" -enckey="$(top_val enckey_bootstrapped)" -pi_admin="$(top_val pi_admin_created)" + echo " Secrets applied:" + for component in postgres lldap authelia privacyidea keycape; do + val="$(nested_val "$component")" + printf " %-28s %s\n" "$component" "$(status_icon "$val")" + done + echo "" -printf " %-30s %s%s\n" "enckey bootstrapped:" \ - "$(status_icon "$enckey")" \ - "$([ "$enckey" = "false" ] && echo " ← TIME-SENSITIVE once pod is live" || true)" -printf " %-30s %s\n" "pi-admin created:" "$(status_icon "$pi_admin")" + enckey="$(top_val enckey_bootstrapped)" + pi_admin="$(top_val pi_admin_created)" -echo "" -echo "Run 'make creds-verify' to refresh state from the live cluster." + printf " %-32s %s\n" "enckey bootstrapped:" "$(status_icon "$enckey")" + printf " %-32s %s\n" "pi-admin created:" "$(status_icon "$pi_admin")" + echo "" + + printf " %-32s %s\n" "bootstrap complete:" "$(status_icon "$bootstrap_complete")" + echo "" + echo "Run 'make creds-verify' to refresh secrets_applied state from the live cluster." + echo "Run 'make creds-agent-init' to resume bootstrap if not complete." + +else + # ── Schema v1: human-as-operator model (NK-WP-0004, legacy) ────────────── + echo "=== net-kingdom Credential State (v1 — human mode) ===" + echo "" + + generated_at="$(top_val generated_at)" + bundle_at="$(top_val bundle_at)" + keepass_confirmed="$(top_val keepass_confirmed)" + + printf " %-30s %s\n" "Generated at:" "${generated_at:-—}" + printf " %-30s %s\n" "Bundle at:" "${bundle_at:-—}" + printf " %-30s %s %s\n" "KeePassXC confirmed:" \ + "$(status_icon "$keepass_confirmed")" \ + "$([ "$keepass_confirmed" = "false" ] && echo "(set keepass_confirmed: true manually)" || true)" + echo "" + + echo " Secrets applied:" + for component in postgres lldap authelia privacyidea keycape; do + val="$(nested_val "$component")" + note="" + [[ "$component" == "keycape" && "$val" == "false" ]] && \ + note=" (requires PI_ADMIN_TOKEN — post-T04)" + printf " %-28s %s%s\n" "$component" "$(status_icon "$val")" "$note" + done + echo "" + + enckey="$(top_val enckey_bootstrapped)" + pi_admin="$(top_val pi_admin_created)" + + printf " %-30s %s%s\n" "enckey bootstrapped:" \ + "$(status_icon "$enckey")" \ + "$([ "$enckey" = "false" ] && echo " ← TIME-SENSITIVE once pod is live" || true)" + printf " %-30s %s\n" "pi-admin created:" "$(status_icon "$pi_admin")" + + echo "" + echo "Run 'make creds-verify' to refresh state from the live cluster." +fi diff --git a/sso-mfa/bootstrap/emergency-bundle.sh b/sso-mfa/bootstrap/emergency-bundle.sh new file mode 100755 index 0000000..2b22ac1 --- /dev/null +++ b/sso-mfa/bootstrap/emergency-bundle.sh @@ -0,0 +1,147 @@ +#!/usr/bin/env bash +# emergency-bundle.sh — assemble and display the net-kingdom emergency credential bundle +# +# Usage: +# bash sso-mfa/bootstrap/emergency-bundle.sh \ +# --age-key (default: ~/.config/sops/age/keys.txt) +# --secrets-dir (default: ./secrets) +# --ops-bundle (path to the .tar.age ops bundle) +# [--reprint] (reprint a previously-delivered bundle without +# requiring all secrets to be present) +# +# Displays the emergency bundle on stdout, optionally writes a temp file +# for 60 seconds, then prompts for human confirmation. +# Sets emergency_bundle_delivered: true in creds-state.yaml after confirmation. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +STATE_FILE="$SCRIPT_DIR/creds-state.yaml" + +AGE_KEY="$HOME/.config/sops/age/keys.txt" +SECRETS_DIR="$SCRIPT_DIR/secrets" +OPS_BUNDLE="" +REPRINT=false + +# ── Argument parsing ────────────────────────────────────────────────────────── +while [[ $# -gt 0 ]]; do + case "$1" in + --age-key) AGE_KEY="$2"; shift 2 ;; + --secrets-dir) SECRETS_DIR="$2"; shift 2 ;; + --ops-bundle) OPS_BUNDLE="$2"; shift 2 ;; + --reprint) REPRINT=true; shift ;; + *) echo "Unknown argument: $1" >&2; exit 1 ;; + esac +done + +# ── Read helpers ────────────────────────────────────────────────────────────── + +read_env_val() { + local file="$1" key="$2" + grep -E "^${key}=" "$file" 2>/dev/null | head -1 | sed "s/^${key}=//" || echo "" +} + +read_env() { + local component="$1" key="$2" + local f="$SECRETS_DIR/$component/secrets.env" + if [[ -f "$f" ]]; then + read_env_val "$f" "$key" + else + echo "" + fi +} + +# ── Collect values ──────────────────────────────────────────────────────────── + +# Age private key +if [[ -f "$AGE_KEY" ]]; then + AGE_PRIVATE_KEY=$(cat "$AGE_KEY") + AGE_PUBKEY=$(grep 'public key:' "$AGE_KEY" | awk '{print $NF}' || echo "") +else + AGE_PRIVATE_KEY="" + AGE_PUBKEY="" +fi + +# Break-glass passwords +PI_ADMIN_PASS="$(read_env privacyidea PI_ADMIN_PASSWORD)" +LLDAP_ADMIN_PASS="$(read_env lldap LLDAP_LDAP_USER_PASS)" +PG_ROOT_PASS="$(read_env postgres PG_ROOT_PASSWORD)" +BG_PASS="$(read_env breakglass BREAKGLASS_PASSWORD)" + +# Ops bundle info +OPS_BUNDLE_LOCATION="${OPS_BUNDLE:-}" +if [[ -n "$AGE_PUBKEY" ]]; then + DECRYPT_CMD="age -d -i ops-bundle-.tar.age" +else + DECRYPT_CMD="age -d -i ops-bundle-.tar.age" +fi + +GENERATED_DATE="$(date -Iseconds)" + +# ── Temp file (shredded automatically) ─────────────────────────────────────── + +TMPFILE="$(mktemp --suffix=-emergency-bundle.txt)" +cleanup_tmp() { shred -u "$TMPFILE" 2>/dev/null || rm -f "$TMPFILE"; } +trap cleanup_tmp EXIT + +# ── Assemble bundle text ────────────────────────────────────────────────────── + +BUNDLE_TEXT="$(cat < "$TMPFILE" +chmod 600 "$TMPFILE" + +# ── Display ─────────────────────────────────────────────────────────────────── + +echo "" +echo "$BUNDLE_TEXT" +echo "" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo " A copy has been written to: $TMPFILE" +echo " It will be shredded automatically when you confirm below." +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "" +echo " Store the above in your personal password manager now." +echo " (1Password, Bitwarden, KeePassXC, paper — your choice.)" +echo "" +read -rp " Press Enter when you have stored the bundle (this clears the screen and shreds the temp file): " + +# Shred temp file (trap handles it but do it explicitly here first) +shred -u "$TMPFILE" 2>/dev/null || rm -f "$TMPFILE" +trap - EXIT + +# Clear screen to remove secrets from terminal scrollback +clear 2>/dev/null || printf '\033c' +echo "" +echo " ✔ Emergency bundle confirmed. Temp file shredded." +echo ""