diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..396c3dc --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,87 @@ +# net-kingdom — Claude Code Instructions + +## Custodian State Hub Integration + +This project is tracked as the **netkingdom** domain in the Custodian State Hub. +Hub topic ID: `a6c6e745-bf54-4465-9340-1534a2be493e` + +The State Hub runs locally at http://127.0.0.1:8000. The MCP server (`state-hub`) +exposes tools for reading and writing state without touching the API directly. + +### Session Protocol + +**On receiving your first message — before writing any response text — call +`get_state_summary()` immediately.** Do not greet, do not ask what to do. +Call the tool first, then respond based on what you find. + +**At the start of every session:** +1. Call `get_state_summary()` — orients you to active workstreams, blocking decisions, + and recent progress. If it fails, the API is likely offline: + ``` + cd ~/the-custodian/state-hub && make api + ``` +2. Call `get_next_steps()` — surfaces contextual suggestions from recently resolved + decisions and cleared workstream dependencies. Act on these before starting new work. +3. Check whether this domain has any open workstreams in the summary. + - **If workstreams exist:** review blocking decisions before starting work. + - **If no workstreams exist:** follow the First Session Protocol below. + +**During work:** +- Use `record_decision()` for any decision that affects direction or dependencies. +- Use `add_progress_event()` for notable events (milestones, blockers, insights). +- Use `resolve_decision()` to close a decision once the choice is made — this is one + of the two sanctioned write operations in the hub. + +> **Design boundary:** The State Hub is a *read model*. Two write operations are +> permanently sanctioned: **Resolving Decisions** and **Suggesting Next Steps** (v0.2). +> The bootstrap tools (`create_workstream`, `create_task`, `update_task_status`) are +> only for First Session Protocol. Formal work structure — requirements, workplans, +> milestones, tasks — belongs in the domain repo, not managed through the hub. + +**At the end of every session:** +- Call `add_progress_event()` with a summary of what was accomplished or decided. + Include `topic_id: a6c6e745-bf54-4465-9340-1534a2be493e` and the relevant `workstream_id`. + +### First Session Protocol + +Triggered when `get_state_summary()` shows **no workstreams** for the `netkingdom` topic. +This means the project is registered but work has not yet been structured. + +**Step 1 — Understand the project (read, don't write)** +- `canon/projects/netkingdom/project_charter_v0.1.md` — purpose, scope, success criteria +- `canon/projects/netkingdom/roadmap_v0.1.md` — planned phases +- Scan the repo root: README, directory structure, any existing code or docs + +**Step 2 — Survey in-progress work** +- Look for TODOs, open branches, half-finished files, or notes +- Note what is already done vs. what is clearly started but incomplete + +**Step 3 — Propose workstreams to Bernd** +Based on what you found, propose 1–3 workstreams. Each workstream should be: +- A coherent strand of work lasting weeks to months (not a single task) +- Named clearly enough that its scope is obvious +- Anchored to a phase in the roadmap if possible + +Present the proposals and **wait for approval before creating anything**. + +**Step 4 — Create and populate (after approval)** +``` +create_workstream(topic_id="a6c6e745-bf54-4465-9340-1534a2be493e", title="...", owner="...", description="...") +create_task(workstream_id="", title="...", priority="high|medium|low") +# repeat for each task in the workstream +``` +Aim for 3–7 tasks per workstream at this stage. Tasks should be concrete and actionable. + +**Step 5 — Record the setup** +``` +add_progress_event( + summary="First session: structured netkingdom work into N workstreams, M tasks", + event_type="milestone", + topic_id="a6c6e745-bf54-4465-9340-1534a2be493e", + detail={"workstreams": [...], "tasks_created": M} +) +``` + +### Quick Reference + +See `~/the-custodian/state-hub/mcp_server/TOOLS.md` for a compact tool reference. diff --git a/README.md b/README.md index fcd7b8f..cc34b85 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,3 @@ -# repo-seed +# NetKingdom -A git repository template to bootstrap coulomb projects from. \ No newline at end of file +NetKingdom provides a dynamic self optimizing full circle security-platform for kubernetes deployed IT-infrastructures. diff --git a/wiki/NetKingdom.md b/wiki/NetKingdom.md new file mode 100644 index 0000000..b06b0a3 --- /dev/null +++ b/wiki/NetKingdom.md @@ -0,0 +1,8 @@ +NetKingdom + +*Open security core for dev sec ops on kubernetes* + +Agent powered automation drives the cambrian explosion of vulnerabilities and security measures in modern IT. +This is a blueprint to bootstrap foundational dynamic security building on open source components minimizing threat exposure for critical use cases. + +xxx diff --git a/wiki/WorkplanOneChatgpt.md b/wiki/WorkplanOneChatgpt.md new file mode 100644 index 0000000..3f75db0 --- /dev/null +++ b/wiki/WorkplanOneChatgpt.md @@ -0,0 +1,273 @@ +WorkplanOneChatgpt + +*How to start according to chatgpt* + +This is a protoplan not to be implemented directly but used as inspiration for an actual plan for implemenation! + +Below is an integrated, Kubernetes-focused workplan that keeps the **“single private credential” bootstrap** idea, then rapidly evolves to **MFA-first admin access**, **least-privilege service accounts**, and **Keycloak SSO (OIDC/SAML) with privacyIDEA as the MFA engine**—exactly the combination privacyIDEA recommends. ([PrivacyIDEA][1]) + +--- + +## Architecture (Kubernetes target state) + +**Ingress (TLS)** + +* `keycloak.yourdomain.com` → Keycloak (SSO / sessions / OIDC+SAML) +* `pi.yourdomain.com` → privacyIDEA (MFA policies + token mgmt) +* optional: `pi-account.yourdomain.com` → privacyIDEA self-service portal + +**Data** + +* Keycloak DB (PostgreSQL) +* privacyIDEA DB (PostgreSQL or MariaDB; Postgres is a common choice) +* privacyIDEA encryption + audit key material stored as **K8s Secrets** (backed up securely) + +**Integration** + +* Keycloak runs the login flow +* privacyIDEA provides MFA via the **privacyIDEA Keycloak Provider** (JAR) ([GitHub][2]) + +--- + +## Workplan (phased deployment on Kubernetes) + +### Phase 0 — Prereqs and “single private credential” rule + +1. **Pick your one private credential** + Use it to unlock a vault (recommended), not as a daily admin password. The vault stores: + + * initial privacyIDEA admin password (typed once) + * privacyIDEA crypto secrets (encryption/audit keys) + * Keycloak admin bootstrap secret + * DB credentials + +2. Cluster prerequisites + +* Ingress Controller (nginx / Traefik) +* `cert-manager` for TLS (Let’s Encrypt or internal CA) +* StorageClass for persistent volumes +* (Optional but recommended) External Secrets Operator + real vault + +--- + +### Phase 1 — Create namespaces, storage, network boundaries + +1. Namespaces + +* `sso` (Keycloak) +* `mfa` (privacyIDEA) +* `databases` (if you want shared DB ops) + +2. NetworkPolicies (recommended) + +* Only ingress controller can reach Keycloak/privacyIDEA services +* Only Keycloak pods can call privacyIDEA API endpoints needed for MFA +* Only app pods (or ingress) can reach Keycloak + +This prevents lateral movement if any pod is compromised. + +--- + +### Phase 2 — Deploy databases (prod-grade) + +**Recommended**: PostgreSQL via a battle-tested chart/operator (CloudNativePG / Zalando / Bitnami Postgres chart). + +1. Create DBs and users: + +* `keycloak_db`, user `keycloak` +* `privacyidea_db`, user `privacyidea` + +2. Store DB credentials as K8s Secrets (or ExternalSecrets). + +3. Backups: + +* Enable automated DB backups to object storage (S3/MinIO/etc.) +* Test restore early + +--- + +### Phase 3 — Deploy privacyIDEA on Kubernetes (MFA core) + +You have two common routes: + +**Route A: Helm chart** + +* There are community Helm approaches (example repo exists, but treat it as a starting point you should review/harden for production). ([GitHub][3]) + +**Route B: Roll your own manifests** + +* Deployment + Service + Ingress + PVC + Secrets +* Often preferred for security review & long-term ownership + +#### privacyIDEA critical secrets (do this before first real use) + +privacyIDEA relies on secrets like `SECRET_KEY`, `PI_PEPPER`, and an encryption key file (`PI_ENCFILE`) / related key material. Generate once, mount as Secrets, back up carefully. ([GitHub][4]) +(Those secrets are “if you lose them, you may lose access to encrypted token data”-class.) + +**Work items** + +1. Create K8s Secrets: + +* `privacyidea-config` (env or ini fragments) +* `privacyidea-enckey` (encryption key material) +* `privacyidea-auditkeys` (audit signing/encryption keys) + +2. Deploy privacyIDEA pods (Deployment) + Service +3. Add Ingress + TLS (`pi.yourdomain.com`) +4. Add rate limiting / WAF features at ingress (recommended) + +#### Bootstrap: “single private credential” + +1. Exec into the privacyIDEA pod and run the one-time admin creation (or equivalent `pi-manage` command). +2. That admin password is the **only password you ever manually type**. +3. Immediately log into WebUI and **enroll MFA for that admin** (TOTP/hardware token). +4. Immediately create: + +* a **limited “trigger-admin”** service account used by Keycloak provider with *only* the `triggerchallenge`-type permission (least privilege) +* policies requiring MFA for admin actions + +This aligns with the second opinion: bootstrap by one password, then transition to MFA + least privilege. + +--- + +### Phase 4 — Deploy Keycloak on Kubernetes (SSO core) + +Two common options: + +**Option A: KeycloakX Helm chart (codecentric)** + +* Widely used Helm chart for a StatefulSet-based Keycloak deployment. ([Artifact Hub][5]) + +**Option B: Keycloak Operator** + +* Good for CRD-based lifecycle management; if your org prefers operators. + +**Work items** + +1. Deploy Keycloak pointing at `keycloak_db` +2. Configure Ingress + TLS (`keycloak.yourdomain.com`) +3. Bootstrap Keycloak admin secret (stored in vault / K8s Secret) +4. Set production settings: + +* hostname strictness +* proxy mode +* metrics/logging +* realm import strategy (GitOps-friendly) + +--- + +### Phase 5 — Add the privacyIDEA Keycloak Provider (JAR) in Kubernetes + +The privacyIDEA provider is published on GitHub releases and needs to match your Keycloak version. ([GitHub][2]) + +**Kubernetes-friendly pattern** + +1. Build a **custom Keycloak image**: + +* Start from Keycloak base image +* Copy the provider JAR into `/opt/keycloak/providers/` +* Run `kc.sh build` + +2. Deploy that image via Helm values (or operator spec) + +This avoids “kubectl cp into running pods” and is the clean GitOps approach. + +--- + +### Phase 6 — Configure the MFA flow (Keycloak → privacyIDEA) + +**Work items** + +1. In Keycloak: + +* Create/choose realm +* Configure users source of truth (Keycloak internal or LDAP/AD) + +2. Create Authentication Flow “privacyIDEA Browser” + +* Add privacyIDEA execution step (REQUIRED) +* Configure: + + * privacyIDEA base URL: `https://pi.yourdomain.com` + * service account: `trigger-admin` + secret (from K8s Secret) + * optional bypass group (“break-glass”) with strong restrictions & alerts + +3. Set this flow as the default browser flow +4. Test: + +* normal user MFA +* admin console MFA +* failure modes (OTP wrong, token missing, privacyIDEA down) + +The provider’s releases/changelog track features like passkey handling and flow behavior—use the matching release notes when selecting versions. ([GitHub][2]) + +--- + +### Phase 7 — User management & token lifecycle (make it operable) + +privacyIDEA 3.12 introduced a **Keycloak user resolver** (and Entra ID resolver), which can simplify identity alignment depending on your setup. ([NetKnights][6]) + +**Work items** + +1. Decide identity source of truth: + +* simplest: Keycloak internal users → privacyIDEA resolver (Keycloak resolver if you want that pattern) +* enterprise: LDAP/AD shared by both + +2. Enable self-service portal where appropriate +3. Define policies: + +* allowed token types +* enrollment rules +* admin rights separation (super-admin vs helpdesk-admin) + +4. Add auditing / log shipping (privacyIDEA audit logs + Keycloak events) + +--- + +### Phase 8 — Backups, DR, upgrades, and break-glass + +**Backups** + +* DB backups (Keycloak + privacyIDEA) +* privacyIDEA encryption/audit keys Secrets (encrypted export) +* realm exports (Keycloak) as code / git + +**Disaster recovery drill** + +* restore DB + keys into a fresh namespace +* verify token validation still works (this catches key/secret mistakes) + +**Break-glass** + +* keep a *disabled-by-default* Keycloak admin path or group exemption +* store break-glass credentials offline + vault +* alert on every use + +--- + +## Deliverables checklist (what “done” looks like) + +* [ ] `mfa` + `sso` namespaces deployed via GitOps/Helm +* [ ] TLS everywhere (cert-manager) +* [ ] privacyIDEA secrets generated, mounted, backed up +* [ ] privacyIDEA admin created once (single typed password), MFA enrolled immediately +* [ ] `trigger-admin` exists with minimal rights +* [ ] Keycloak runs from custom image including privacyIDEA provider JAR +* [ ] Keycloak “privacyIDEA Browser” flow enforced +* [ ] DB backups + restore tested +* [ ] NetworkPolicies in place + +--- + +If you tell me **which Kubernetes flavor** you’re using (k3s / AKS / EKS / GKE / OpenShift) and whether you want **Postgres operator vs Helm**, I’ll turn this into a concrete, copy-pastable “repo plan” (folders, Helm values skeletons, Secrets/ExternalSecrets layout, and the exact build steps for the custom Keycloak image) without adding unnecessary complexity. + +[1]: https://www.privacyidea.org/?feed=&utm_source=chatgpt.com "privacyIDEA 3.12 is available" +[2]: https://github.com/privacyidea/keycloak-provider/releases?utm_source=chatgpt.com "Releases · privacyidea/keycloak-provider" +[3]: https://github.com/gpappsoft/privacyidea-helm?utm_source=chatgpt.com "A helm chart to install a privacyIDEA stack" +[4]: https://github.com/privacyidea/privacyidea?utm_source=chatgpt.com "privacyidea/privacyidea: :closed_lock_with_key" +[5]: https://artifacthub.io/packages/helm/codecentric/keycloakx?utm_source=chatgpt.com "keycloakx 7.1.8" +[6]: https://netknights.it/en/press/multi-faktor-authentifizierung-privacyidea-in-version-3-12-erschienen/?utm_source=chatgpt.com "Multi-Factor Authentication privacyIDEA version 3.12 released" + + +xxx diff --git a/wiki/WorkplanOneGrok.md b/wiki/WorkplanOneGrok.md new file mode 100644 index 0000000..65e5ede --- /dev/null +++ b/wiki/WorkplanOneGrok.md @@ -0,0 +1,171 @@ +WorkplanOneGrok + +*How to start according to grok* + +This is a protoplan not to be implemented directly but used as inspiration for an actual plan for implemenation! + +**Integrated architecture (merging both opinions)** + +Your single private credential remains the **root-of-trust** that only ever unlocks a vault. Everything else (initial admin passwords, database credentials, privacyIDEA crypto roots, TLS keys, Keycloak admin secret) lives inside the vault and is never typed manually again. + +- **privacyIDEA** = MFA/token/policy engine (user resolvers, self-service enrollment, token types, audit) +- **Keycloak** = full SSO IdP (OIDC/SAML clients, realms, groups, session management) +- **Integration** = Keycloak PrivacyIDEA provider (MFA step in the browser flow) +- **Vault** = single unlock point (KeePassXC/Bitwarden local, or HashiCorp Vault in-cluster) +- **Databases** = private, encrypted, backed up with keys from the vault +- **K8s layer** = Helm + Official Keycloak Operator + community privacyIDEA Helm chart (gpappsoft/privacyidea) + cert-manager + Ingress + +This satisfies both sources: you bootstrap from literally one secret, immediately move to MFA-protected admins, treat crypto roots (SECRET_KEY, PI_PEPPER, PI_ENCFILE) as crown jewels, and run everything declaratively on Kubernetes. + +### Kubernetes Workplan (Day 0–7) + +**Prerequisites** +- Running Kubernetes cluster (k3s, kind, EKS, AKS, GKE – any with StorageClass and cert-manager installed). +- kubectl, Helm 3. +- Ingress controller (nginx-ingress or Traefik). +- cert-manager for automatic TLS. +- Your single master credential ready (KeePassXC file or age private key). + +#### Day 0 – Vault & Secret Bootstrap (single-credential principle) +1. Create your vault (recommended: KeePassXC .kdbx file or self-hosted Bitwarden; for production add HashiCorp Vault later). +2. Inside the vault generate/store (all random, never reuse): + - privacyIDEA: `SECRET_KEY` (64+ chars), `PI_PEPPER` (32+ chars), encryption key file content (`pi-manage create_enckey`). + - MariaDB root + privacyIDEA DB user passwords. + - Keycloak admin initial secret + DB password. + - TLS ACME account key (if not using cert-manager fully). + - Break-glass admin credentials + offline recovery OTP seed. +3. Export an encrypted “ops bundle” (age-encrypted tar of all secret YAML manifests) – this bundle is the only thing you ever decrypt with your single credential. +4. Enable cluster encryption-at-rest (if not already). + +#### Day 1 – Foundation & Databases +```bash +# Deploy HashiCorp Vault (optional but ideal for rotation) +helm repo add hashicorp https://helm.releases.hashicorp.com +helm install vault hashicorp/vault --namespace vault --create-namespace -f vault-values.yaml + +# Or use simple sops/age for GitOps +``` +- Deploy MariaDB (Bitnami Helm or Crunchy Postgres Operator). +- Create Kubernetes Secrets from the vault (or use External Secrets Operator + Vault backend): + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: privacyidea-secrets + data: + SECRET_KEY: + PI_PEPPER: + PI_ENCFILE: + ``` +- Apply network policies: DBs only reachable from privacyIDEA/Keycloak pods. + +#### Day 2–3 – Deploy privacyIDEA (MFA core) +The community Helm chart (gpappsoft/privacyidea on Artifact Hub) deploys the full stack (privacyIDEA + MariaDB + optional Redis/RADIUS). + +```bash +helm repo add privacyidea https://gpappsoft.github.io/helm-charts +helm install privacyidea privacyidea/privacyidea \ + --namespace privacyidea --create-namespace \ + -f values-privacyidea.yaml +``` + +Key parts of `values-privacyidea.yaml` (populated from vault): +```yaml +database: + password: +privacyidea: + config: + SECRET_KEY: + PI_PEPPER: + encfile: # mounted as secret volume + enabled: true + existingSecret: privacyidea-secrets + key: PI_ENCFILE + admin: + bootstrap: true # chart runs pi-manage internally + ingress: + enabled: true + hostname: pi.yourdomain.com + tls: true +``` +Post-install (one-time job or manual): +- `kubectl exec` into privacyIDEA pod and run `pi-manage admin add pi-admin --email admin@yourdomain.com` (password from vault). +- Immediately enroll an MFA token for `pi-admin` via the UI. +- Create limited “trigger-admin” (only `triggerchallenge` right) for Keycloak. +- Apply day-1 policies: WebUI restricted to VPN/office IPs, MFA required for all admin actions, enrollment policies locked down. + +#### Day 4–5 – Deploy Keycloak + privacyIDEA MFA integration +1. Install the official Keycloak Operator: + ```bash + kubectl apply -f https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/main/kubernetes/keycloaks.k8s.keycloak.org-v1.yml + kubectl apply -f https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/main/kubernetes/kubernetes.yml -n keycloak + ``` +2. Prepare the PrivacyIDEA provider JAR: + ```bash + wget https://github.com/privacyidea/keycloak-provider/releases/latest/download/PrivacyIDEA-Provider.jar + kubectl create configmap privacyidea-provider --from-file=PrivacyIDEA-Provider.jar -n keycloak + ``` +3. Create Keycloak CR (with custom provider mount): + ```yaml + apiVersion: k8s.keycloak.org/v2alpha1 + kind: Keycloak + metadata: + name: keycloak + namespace: keycloak + spec: + instances: 2 + db: + vendor: postgres # or mariadb + username: keycloak + passwordSecret: keycloak-db-secret # from vault + unsupported: + podTemplate: + spec: + volumes: + - name: providers + configMap: + name: privacyidea-provider + containers: + - name: keycloak + volumeMounts: + - name: providers + mountPath: /opt/keycloak/providers + additionalOptions: + - name: spi-authenticator-privacyidea-enabled + value: "true" + - name: spi-authenticator-privacyidea-url + value: "https://pi.yourdomain.com" + - name: spi-authenticator-privacyidea-service-account + value: "trigger-admin" # limited rights only + ingress: + enabled: true + ``` +4. In Keycloak UI (or via CR import): create realm, import same user resolver as privacyIDEA (LDAP/Keycloak internal/Entra), create custom browser flow with “PrivacyIDEA Forms” execution (REQUIRED). Bind as default flow. + +#### Day 6–7 – Policies, Self-Service, Break-Glass & Testing +- privacyIDEA: enable self-service enrollment policies + registration tokens. +- Keycloak: require MFA step-up for admin console and sensitive clients. +- Break-glass: store offline admin + recovery OTP in sealed envelope + vault. Alert on usage. +- End-to-end test: app → Keycloak → privacyIDEA OTP/push → SSO session. +- Configure backups (Velero or operator) encrypted with vault key. +- Monitoring: Prometheus + Grafana (optional). + +### Security & Operations Checklist (do these before going live) +- All external traffic via Ingress + HSTS + strict TLS. +- privacyIDEA encryption keys backed up and never in image. +- Least-privilege service accounts everywhere. +- NetworkPolicies + PodSecurityPolicies. +- Rotate secrets via vault (External Secrets Operator handles re-injection). +- Test recovery: vault unlock → restore DB + keys → everything works. + +This plan is production-ready, fully declarative, and scales. You start with one memorized credential and end with a hardened, auditable SSO+MFA platform. + +If you tell me: +- Cluster type (k3s, EKS, bare-metal…) +- Existing user store (LDAP/AD/Entra/Keycloak internal) +- Target apps (Nextcloud, GitLab, custom…) +- Preference for GitOps (ArgoCD/Flux) or manual Helm + +I can give you the exact `values.yaml` files, CRDs, and one-click scripts. + +xxx diff --git a/workplans/NK-WP-0001-sso-mfa-platform.md b/workplans/NK-WP-0001-sso-mfa-platform.md new file mode 100644 index 0000000..87d2be4 --- /dev/null +++ b/workplans/NK-WP-0001-sso-mfa-platform.md @@ -0,0 +1,361 @@ +--- +id: NK-WP-0001 +type: workplan +title: "SSO & MFA Platform — Keycloak + privacyIDEA on Kubernetes" +domain: netkingdom +status: active +owner: worsch +topic_slug: netkingdom +state_hub_workstream_id: 39263c4b-ef70-4053-b782-350834b7e1be +created: "2026-02-28" +updated: "2026-02-28" +--- + +# SSO & MFA Platform — Keycloak + privacyIDEA on Kubernetes + +## Summary + +Deploy a hardened SSO and MFA platform on Kubernetes: Keycloak as the +OIDC/SAML identity provider, privacyIDEA as the MFA/token engine, +integrated via the privacyIDEA Keycloak Provider. This is the foundational +security layer for the net-kingdom DevSecOps platform. + +## Context + +Synthesised from two AI protoplans (wiki/WorkplanOneChatgpt.md and +wiki/WorkplanOneGrok.md). Both sources converge on the same architecture; +this plan picks the most concrete and production-aligned choices from each: + +- **Single-credential bootstrap** (Grok) — one master secret unlocks the + vault; all other credentials are vault-managed and never typed manually. +- **Phase structure** (ChatGPT) — eight sequential phases reducing blast + radius at each step. +- **Tooling choices** (both) — Keycloak Operator or codecentric Helm, + gpappsoft privacyIDEA Helm, CloudNativePG for PostgreSQL, cert-manager + for TLS, Traefik as ingress (K3s native, aligned with Railiance). +- **Custom Keycloak image** (both) — JAR baked into image via `kc.sh build` + rather than `kubectl cp`; clean GitOps pattern. + +## Architecture + +``` + Internet + │ TLS (cert-manager / Let's Encrypt) + ┌──────┴──────┐ + │ Traefik │ (K3s native ingress) + └──┬───────┬──┘ + │ │ + keycloak.… pi.… pi-account.… + │ │ │ + ┌──────┘ ┌────┘ │ + ▼ ▼ │ + [Keycloak] [privacyIDEA]◄──┘ (self-service portal) + │ │ + └────┬────┘ + ▼ + [PostgreSQL] (CloudNativePG, namespace: databases) + │ + [Vault / K8s Secrets] ← single credential unlocks +``` + +**Namespaces:** `sso` (Keycloak), `mfa` (privacyIDEA), `databases` + +**Integration:** Keycloak runs the browser login flow; privacyIDEA provides +MFA via the privacyIDEA Keycloak Provider JAR (baked into custom image). + +## Dependencies + +- Depends on: `railiance/three-phoenix-ha-cluster` — full production + deployment targets the ThreePhoenix K3s HA cluster. Development/staging + can proceed on a single-node k3s instance. +- Depends on: `railiance/phase-0-operational-baseline` — cert-manager, TLS, + backup strategy must be operational before going live. + +## Tasks + +### T01 — Phase 0: Vault & secret bootstrap (single-credential principle) + +```task +id: NK-WP-0001-T01 +state_hub_task_id: 7992528c-d533-44e5-bcce-f92aaa2b75b2 +status: todo +priority: critical +``` + +Create the vault (KeePassXC .kdbx or self-hosted Bitwarden; HashiCorp Vault +for later production hardening). Generate and store all secrets inside the +vault — never typed again: + +- privacyIDEA: `SECRET_KEY` (64+ chars), `PI_PEPPER` (32+ chars), + `PI_ENCFILE` content (`pi-manage create_enckey`). +- PostgreSQL: root + `keycloak` + `privacyidea` user passwords. +- Keycloak: admin bootstrap secret + DB password. +- TLS: ACME account key (if not delegated fully to cert-manager). +- Break-glass: admin credentials + offline recovery OTP seed. + +Export an age-encrypted ops bundle (encrypted tar of all secret YAML +manifests). Enable K8s encryption-at-rest. Confirm secret injection +strategy: External Secrets Operator + Vault backend, or sops/age for GitOps. + +**Done when:** vault created, all secrets generated, encrypted ops bundle +exported and stored offsite. Secret injection strategy decided. + +--- + +### T02 — Phase 1: K8s foundations (namespaces, NetworkPolicies, cert-manager) + +```task +id: NK-WP-0001-T02 +state_hub_task_id: 721ca6b2-0cf4-4008-a966-87b1563550fa +status: todo +priority: high +``` + +Create namespaces: `sso`, `mfa`, `databases`. Verify cert-manager is +installed and functional on the K3s cluster (Traefik ingress). Define and +apply NetworkPolicies to prevent lateral movement: + +- Only ingress controller reaches Keycloak/privacyIDEA service ports. +- Only Keycloak pods call the privacyIDEA API. +- Only app pods/ingress reach Keycloak. +- DB pods reachable only from `sso` and `mfa` namespaces. + +Verify StorageClass for PVCs. + +**Done when:** namespaces exist, NetworkPolicies applied and tested (verify +denied paths), cert-manager issues a test certificate. + +--- + +### T03 — Phase 2: PostgreSQL deployment (Keycloak + privacyIDEA DBs) + +```task +id: NK-WP-0001-T03 +state_hub_task_id: 7fa60004-deb2-4db5-a470-f95dda07f6ab +status: todo +priority: high +``` + +Deploy PostgreSQL via CloudNativePG operator (preferred: aligns with +ThreePhoenix HA posture) or Bitnami Helm chart as fallback. Create: + +- Database `keycloak_db`, user `keycloak` +- Database `privacyidea_db`, user `privacyidea` + +Store DB credentials as K8s Secrets (or ExternalSecrets from vault). +Configure automated DB backups to object storage (S3 or MinIO). +**Run a restore drill before proceeding** — a failed restore later is a +critical blocker. + +**Done when:** both DBs live, credentials in K8s Secrets, backup running, +restore drill passed. + +--- + +### T04 — Phase 3: Deploy privacyIDEA (MFA core) + +```task +id: NK-WP-0001-T04 +state_hub_task_id: 6ad1296a-a488-4031-b665-f77030e971ed +status: todo +priority: high +``` + +Deploy privacyIDEA via `gpappsoft/privacyidea` Helm chart (Artifact Hub) or +custom manifests (Deployment + Service + Ingress + PVC + Secrets). Key +Helm values: + +```yaml +database: + password: +privacyidea: + config: + SECRET_KEY: + PI_PEPPER: + encfile: + enabled: true + existingSecret: privacyidea-secrets + key: PI_ENCFILE + ingress: + enabled: true + hostname: pi.yourdomain.com + tls: true +``` + +Create K8s Secrets: `privacyidea-config`, `privacyidea-enckey`, +`privacyidea-auditkeys`. Configure Ingress + TLS. Add rate-limiting and +WAF rules at Traefik level. + +**Bootstrap (single-credential moment):** +1. `kubectl exec` into pod, run `pi-manage admin add pi-admin` — password + comes from vault (only time a password is typed). +2. Immediately enroll MFA for `pi-admin` (TOTP or hardware token). +3. Create `trigger-admin` with `triggerchallenge` right only. +4. Apply policies: WebUI restricted to VPN/office IPs; MFA required for + all admin actions. + +**Done when:** privacyIDEA reachable at pi.yourdomain.com with valid TLS, +pi-admin enrolled with MFA, trigger-admin created, rate-limiting active. + +--- + +### T05 — Phase 4: Deploy Keycloak (SSO core) + +```task +id: NK-WP-0001-T05 +state_hub_task_id: b9f73aa6-9035-4643-9905-64e73a29b298 +status: todo +priority: high +``` + +Build a **custom Keycloak image** that includes the privacyIDEA Provider JAR: + +```dockerfile +FROM quay.io/keycloak/keycloak: +COPY PrivacyIDEA-Provider.jar /opt/keycloak/providers/ +RUN /opt/keycloak/bin/kc.sh build +``` + +Deploy via official Keycloak Operator (CRD-based) or codecentric KeycloakX +Helm chart. Configure: + +- DB: `keycloak_db` (credentials from K8s Secret) +- Ingress + TLS: `keycloak.yourdomain.com` (Traefik + cert-manager) +- Hostname strictness + proxy mode (Traefik forward headers) +- Metrics/logging (Prometheus annotations) +- Admin bootstrap secret from vault +- Realm import strategy: GitOps-friendly (realm JSON in git or CR) + +**Done when:** Keycloak reachable with valid TLS, admin console accessible, +custom image with privacyIDEA JAR deployed and verified. + +--- + +### T06 — Phase 5: Realm config & MFA authentication flow + +```task +id: NK-WP-0001-T06 +state_hub_task_id: 3b6379a4-a27b-4d25-82be-bc600879f036 +status: todo +priority: medium +``` + +In Keycloak: + +1. Create/configure realm; set identity source of truth (Keycloak internal + users recommended for initial deployment; LDAP/AD or Entra as extension). +2. Create Authentication Flow "privacyIDEA Browser": + - Add privacyIDEA execution step (REQUIRED) + - Config: privacyIDEA URL = `https://pi.yourdomain.com`, service account + = `trigger-admin` (secret from K8s Secret) + - Optional: bypass group (break-glass) with strict restrictions + alerts +3. Set this flow as the default browser flow. +4. Require MFA step-up for admin console and sensitive OIDC clients. + +Test: +- Normal user: password → MFA OTP → session established +- Admin console: MFA required +- Failure modes: wrong OTP, token missing, privacyIDEA unreachable +- Break-glass: bypass works, alert fires + +**Done when:** end-to-end auth works for normal and admin paths, all failure +modes handled gracefully. + +--- + +### T07 — Phase 6: User management, policies & self-service portal + +```task +id: NK-WP-0001-T07 +state_hub_task_id: c7cf902a-b480-4545-a536-293070945206 +status: todo +priority: medium +``` + +Decide and implement identity source of truth (Keycloak internal → +privacyIDEA Keycloak resolver, or LDAP/AD shared). The privacyIDEA 3.12+ +Keycloak user resolver simplifies alignment. + +Define policies in privacyIDEA: +- Allowed token types: TOTP, hardware (YubiKey), passkey +- Enrollment rules (who can self-enroll, which token types) +- Admin rights separation: super-admin vs. helpdesk-admin + +Enable self-service portal at `pi-account.yourdomain.com` for user token +enrollment/replacement. + +Configure auditing and log shipping: privacyIDEA audit logs + Keycloak +events → centralized logging (ELK/Loki or equivalent). Token lifecycle +policies: enrollment, revocation, re-enrollment on device loss. + +**Done when:** policies documented and applied, self-service portal live, +audit logs flowing. + +--- + +### T08 — Phase 7: Backups, DR, break-glass & monitoring + +```task +id: NK-WP-0001-T08 +state_hub_task_id: 9cbd1d89-b5bf-491e-9d16-b1c7d57076fb +status: todo +priority: medium +``` + +**Backups:** +- DB backups: Keycloak + privacyIDEA (Velero or CloudNativePG scheduled + backup to S3/MinIO). Test restore. +- privacyIDEA encryption/audit key Secrets: encrypted export, versioned. +- Keycloak realm exports: stored as JSON in git (GitOps-friendly). + +**Disaster recovery drill** (mandatory before production): +1. Restore DB + keys into a fresh namespace. +2. Verify token validation still works — this catches key/secret mistakes. + +**Break-glass procedure:** +- Disabled-by-default Keycloak admin path or group exemption. +- Break-glass credentials stored offline + vault. Alert (PagerDuty/webhook) + on every use. + +**Monitoring:** +- Prometheus scraping Keycloak + privacyIDEA metrics. +- Grafana dashboards: auth success/failure rates, MFA challenge latency, + token count by type. +- Alert: privacyIDEA unreachable (blocks all logins). + +**Final validation:** +- All external traffic: Ingress + HSTS + strict TLS. +- NetworkPolicies verified (no unintended open paths). +- End-to-end: app → Keycloak → privacyIDEA OTP → SSO session established. + +**Done when:** DR drill passed, monitoring live, break-glass procedure +documented and tested, HSTS and NetworkPolicies verified. + +--- + +## Deliverables Checklist + +- [ ] Vault created; all secrets generated and encrypted ops bundle exported +- [ ] `sso`, `mfa`, `databases` namespaces + NetworkPolicies deployed +- [ ] TLS everywhere via cert-manager (Traefik ingress) +- [ ] PostgreSQL live; both DBs created; backup + restore tested +- [ ] privacyIDEA running at `pi.yourdomain.com`; pi-admin MFA enrolled; + trigger-admin created with least-privilege rights +- [ ] Keycloak running from custom image including privacyIDEA Provider JAR +- [ ] Keycloak "privacyIDEA Browser" flow enforced as default +- [ ] Realm exported to git; admin secret from vault +- [ ] Self-service portal live; token lifecycle policies defined +- [ ] DR drill passed; monitoring live; break-glass documented and tested + +## Open Questions / Extension Points + +- **Vault backend**: KeePassXC (simple) vs HashiCorp Vault in-cluster + (rotation, audit trail). Start with KeePassXC; upgrade to Vault when + ThreePhoenix cluster is stable. +- **Identity source of truth**: Keycloak-internal vs LDAP/AD/Entra. + Decision needed before T07. +- **GitOps tooling**: ArgoCD or Flux for declarative Helm management? + Aligns with Railiance staged-promotion-lifecycle workstream. +- **Cluster target**: Development on single-node k3s; production on + ThreePhoenix (3-node HA). Workplan covers both; HA-specific steps noted + where they diverge.