--- title: "Credential Management Standard" version: "0.1" status: "Draft Standard" domain: custodian scope: all-domains created: "2026-03-20" --- # Credential Management Standard **Version:** 0.1 **Status:** Draft Standard **Scope:** All domains and repositories in the federated organization --- ## 1. Purpose This standard defines how credentials, secrets, and key material are managed across all systems — from a developer workstation with no infrastructure, to a fully operational Kubernetes cluster. The core principle is a **single root of trust**: one operator keypair anchors all credential storage and encryption. Every secret can be traced back to that root. No secret lives outside this hierarchy. --- ## 2. Trust Hierarchy ``` Operator passphrase (human memory only — never stored anywhere) │ └── age keypair (~/.config/sops/age/key.txt — one per operator) │ ├── SOPS encryption (GitOps secrets in all repos) │ └── secrets/**/*.sops.yaml — encrypted at rest in git │ ├── Ops bundle (age-encrypted tar — offsite backup) │ └── ops-bundle-.tar.age │ └── all service secrets at point-in-time │ └── KeePassXC (pre-cluster primary credential store) │ └── master password = operator passphrase (or derived) │ ├── Infrastructure credentials │ ├── SSH keys (server access) │ ├── API tokens (Gitea, HostEurope, Hetzner) │ └── Cloud credentials │ ├── Service secrets (per-domain groups) │ ├── net-kingdom/privacyidea/ │ ├── net-kingdom/lldap/ │ ├── net-kingdom/authelia/ │ ├── net-kingdom/keycape/ │ └── railiance/postgres/ │ └── Vault root token (in-cluster phase, stored here) └── HashiCorp Vault └── External Secrets Operator (ESO) └── K8s Secrets → pods ``` --- ## 3. Phases ### Phase 0 — Pre-cluster (bootstrap) **Used when:** No Kubernetes cluster is available. Local development, initial server provisioning, CI bootstrap. **Tools:** age keypair + KeePassXC + ops bundle **Flow:** 1. Generate service secrets with a `gen-secrets.sh` script 2. Copy each secret manually into KeePassXC (under the appropriate group) 3. Encrypt a point-in-time ops bundle: `pack-bundle.sh ` 4. Store the ops bundle offsite (separate physical location from KeePassXC) 5. Shred the plaintext secrets directory: `find secrets/ -type f -exec shred -u {} \;` 6. When deploying to k8s, read each secret from KeePassXC and inject via `create-secrets.sh` scripts that produce K8s Secrets **Invariant:** Plaintext secrets MUST NOT persist on disk after being stored in KeePassXC. The only durable forms are: KeePassXC + ops bundle. --- ### Phase 1 — GitOps secrets (SOPS) **Used when:** Secrets need to live alongside infrastructure code in git. All repos with infrastructure manifests use this pattern. **Tools:** SOPS + age **Configuration (`.sops.yaml` in repo root):** ```yaml creation_rules: - path_regex: secrets/.*$ age: >- - path_regex: .*\.sops\.yaml$ age: >- ``` **Multi-operator:** When a second operator joins, add their age public key as an additional recipient and re-encrypt all secrets with `sops updatekeys`. Both keys can decrypt independently — no single point of failure. **Invariant:** The age private key is NEVER committed to git. The public key is committed (in `.sops.yaml` and `keys/age.pub`). Encrypted values in git are safe to store and review. --- ### Phase 2 — In-cluster (HashiCorp Vault) **Used when:** Kubernetes cluster is operational and stable. **Tools:** HashiCorp Vault + External Secrets Operator (ESO) **Why ESO over Vault Agent Injector:** ESO produces standard K8s Secrets, which are compatible with plain Helm charts and do not require pod annotation changes. Decision D4 (net-kingdom DECISIONS.md). **Flow:** 1. Bootstrap Vault with the root token stored in KeePassXC 2. Enable Kubernetes auth method (`vault auth enable kubernetes`) 3. Create per-service policies with least-privilege access 4. Migrate each service secret from KeePassXC into Vault 5. Deploy ESO `SecretStore` pointing to Vault 6. Replace `create-secrets.sh` calls with `ExternalSecret` manifests 7. Vault reconciles secrets into K8s Secrets automatically **KeePassXC post-cluster:** Remains the source of truth for: - The Vault root/unseal keys (emergency only) - Dev/sandbox systems that do not connect to in-cluster Vault - New secrets before they are migrated into Vault --- ## 4. KeePassXC Group Structure All service secrets are organized under a standardized group hierarchy: ``` KeePassXC root ├── Infrastructure │ ├── SSH Keys │ │ └── (private key as attachment, public key as note) │ ├── API Tokens │ │ ├── gitea-admin │ │ ├── hosteurope-api │ │ └── hetzner-api │ └── Cloud Credentials │ └── │ ├── net-kingdom │ ├── privacyidea │ │ ├── PI_SECRET_KEY │ │ ├── PI_PEPPER │ │ ├── PI_DB_PASSWORD │ │ ├── pi-admin (password + totp-seed) │ │ ├── trigger-admin (password + API token) │ │ └── enckey (attachment: enckey file + audit keypair) │ ├── lldap │ │ ├── LLDAP_JWT_SECRET │ │ └── LLDAP_LDAP_USER_PASS │ ├── authelia │ │ ├── AUTHELIA_JWT_SECRET │ │ ├── AUTHELIA_SESSION_SECRET │ │ ├── AUTHELIA_STORAGE_ENCRYPTION_KEY │ │ ├── AUTHELIA_OIDC_HMAC_SECRET │ │ └── AUTHELIA_KEYCAPE_CLIENT_SECRET │ └── keycape │ ├── RSA signing key (attachment: private + public PEM) │ └── PI_ADMIN_TOKEN │ ├── railiance │ ├── postgres │ │ └── PG_ROOT_PASSWORD │ └── sops-age │ └── age private key (attachment: key.txt) │ └── vault ├── root-token └── unseal-keys (attachment: unseal-keys.txt, gpg-encrypted) ``` --- ## 5. Age Keypair Management **One keypair per operator.** The same key is used for: - SOPS encryption across all repos - Ops bundle encryption **Generate:** ```bash age-keygen -o ~/.config/sops/age/key.txt # output: Public key: age1... ``` **Add to repos:** Copy the public key into `.sops.yaml` of each repo and into `keys/age.pub`. Commit both. **Back up:** The private key file MUST be stored in KeePassXC as an attachment under `railiance/sops-age/age private key`. The KeePassXC database is the disaster recovery path for the age private key. **Rotation:** If the private key is compromised, generate a new keypair, add the new public key to all repos, re-encrypt all secrets with `sops updatekeys`, then revoke the old key from all `.sops.yaml` files. --- ## 6. Ops Bundle The ops bundle is a point-in-time snapshot of all service secrets, encrypted with age and stored offsite. **Create:** ```bash bash gen-secrets.sh ./secrets # generates all secrets as env files # ... enter each into KeePassXC ... bash pack-bundle.sh ./secrets # → ops-bundle-.tar.age find secrets/ -type f -exec shred -u {} \; # shred plaintext ``` **Restore:** ```bash age -d -i ~/.config/sops/age/key.txt -o secrets.tar ops-bundle-.tar.age tar xf secrets.tar # re-run create-secrets.sh scripts from restored env files ``` **Frequency:** Create a new ops bundle: - Before any major cluster operation (migration, upgrade, rekey) - After adding or rotating any service secret - At least once per quarter --- ## 7. Prohibited Patterns These are hard violations regardless of context: | Pattern | Why prohibited | |---------|----------------| | Plaintext secrets committed to git | Unrecoverable leak | | Secrets in environment variables in shell history | ~/.bash_history exposure | | Sharing secrets via chat, email, or issue trackers | Uncontrolled propagation | | Using the same password for multiple services | Single-point compromise | | Storing age private key only on a single machine | Catastrophic loss on disk failure | | Hardcoded secrets in application code or Helm values | Accidental publishing | --- ## 8. Multi-operator Extension When a second operator needs access: 1. They generate their own age keypair (`age-keygen`) 2. Share only the **public key** (never the private key) 3. Primary operator adds it to `.sops.yaml` in all repos 4. Primary operator runs `sops updatekeys ` on all encrypted files 5. Both operators can now encrypt and decrypt independently 6. Share KeePassXC database via an encrypted channel (never plaintext) — the other operator opens it with their own master password after import --- ## 9. Vault Migration Checklist When the cluster is stable enough to operate Vault: - [ ] Deploy Vault via Helm with HA mode (3 replicas minimum) - [ ] Store root token and unseal keys in KeePassXC (vault/ group) - [ ] Enable Kubernetes auth method - [ ] Create per-service Vault policies (least privilege) - [ ] Deploy ESO `ClusterSecretStore` pointing to Vault - [ ] For each service: create `ExternalSecret` manifest, verify K8s Secret reconciles correctly, then delete the manually-created K8s Secret - [ ] Verify ESO auto-rotation works (reduce TTL to 1h, confirm rotation) - [ ] Remove `create-secrets.sh` scripts from deployment runbooks - [ ] Update this standard to Phase 2 operational status --- ## 10. Summary | Situation | Tool | Source of truth | |-----------|------|----------------| | No cluster, local dev | KeePassXC + create-secrets.sh | KeePassXC | | GitOps secrets in repo | SOPS + age | Git (ciphertext) | | Cluster operational | Vault + ESO | Vault (KeePassXC holds root) | | Disaster recovery | Ops bundle (age) | Offsite encrypted archive | | Multi-operator | SOPS multi-recipient | Each operator's age keypair |