Add CLAUDE.md, wiki protoplans, and NK-WP-0001 workplan

Initialises the net-kingdom project structure:
- README.md: updated title and description
- CLAUDE.md: project instructions and State Hub integration config
- wiki/: three reference docs (NetKingdom overview, ChatGPT and Grok
  protoplans for the SSO/MFA platform)
- workplans/NK-WP-0001-sso-mfa-platform.md: combined workplan (8 phases,
  8 tasks) synthesised from the two protoplans; registered in the
  Custodian State Hub (workstream 39263c4b)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-28 17:21:51 +01:00
parent a852627f0c
commit 004a8d6e6b
6 changed files with 902 additions and 2 deletions

87
CLAUDE.md Normal file
View File

@@ -0,0 +1,87 @@
# net-kingdom — Claude Code Instructions
## Custodian State Hub Integration
This project is tracked as the **netkingdom** domain in the Custodian State Hub.
Hub topic ID: `a6c6e745-bf54-4465-9340-1534a2be493e`
The State Hub runs locally at http://127.0.0.1:8000. The MCP server (`state-hub`)
exposes tools for reading and writing state without touching the API directly.
### Session Protocol
**On receiving your first message — before writing any response text — call
`get_state_summary()` immediately.** Do not greet, do not ask what to do.
Call the tool first, then respond based on what you find.
**At the start of every session:**
1. Call `get_state_summary()` — orients you to active workstreams, blocking decisions,
and recent progress. If it fails, the API is likely offline:
```
cd ~/the-custodian/state-hub && make api
```
2. Call `get_next_steps()` — surfaces contextual suggestions from recently resolved
decisions and cleared workstream dependencies. Act on these before starting new work.
3. Check whether this domain has any open workstreams in the summary.
- **If workstreams exist:** review blocking decisions before starting work.
- **If no workstreams exist:** follow the First Session Protocol below.
**During work:**
- Use `record_decision()` for any decision that affects direction or dependencies.
- Use `add_progress_event()` for notable events (milestones, blockers, insights).
- Use `resolve_decision()` to close a decision once the choice is made — this is one
of the two sanctioned write operations in the hub.
> **Design boundary:** The State Hub is a *read model*. Two write operations are
> permanently sanctioned: **Resolving Decisions** and **Suggesting Next Steps** (v0.2).
> The bootstrap tools (`create_workstream`, `create_task`, `update_task_status`) are
> only for First Session Protocol. Formal work structure — requirements, workplans,
> milestones, tasks — belongs in the domain repo, not managed through the hub.
**At the end of every session:**
- Call `add_progress_event()` with a summary of what was accomplished or decided.
Include `topic_id: a6c6e745-bf54-4465-9340-1534a2be493e` and the relevant `workstream_id`.
### First Session Protocol
Triggered when `get_state_summary()` shows **no workstreams** for the `netkingdom` topic.
This means the project is registered but work has not yet been structured.
**Step 1 — Understand the project (read, don't write)**
- `canon/projects/netkingdom/project_charter_v0.1.md` — purpose, scope, success criteria
- `canon/projects/netkingdom/roadmap_v0.1.md` — planned phases
- Scan the repo root: README, directory structure, any existing code or docs
**Step 2 — Survey in-progress work**
- Look for TODOs, open branches, half-finished files, or notes
- Note what is already done vs. what is clearly started but incomplete
**Step 3 — Propose workstreams to Bernd**
Based on what you found, propose 13 workstreams. Each workstream should be:
- A coherent strand of work lasting weeks to months (not a single task)
- Named clearly enough that its scope is obvious
- Anchored to a phase in the roadmap if possible
Present the proposals and **wait for approval before creating anything**.
**Step 4 — Create and populate (after approval)**
```
create_workstream(topic_id="a6c6e745-bf54-4465-9340-1534a2be493e", title="...", owner="...", description="...")
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
# repeat for each task in the workstream
```
Aim for 37 tasks per workstream at this stage. Tasks should be concrete and actionable.
**Step 5 — Record the setup**
```
add_progress_event(
summary="First session: structured netkingdom work into N workstreams, M tasks",
event_type="milestone",
topic_id="a6c6e745-bf54-4465-9340-1534a2be493e",
detail={"workstreams": [...], "tasks_created": M}
)
```
### Quick Reference
See `~/the-custodian/state-hub/mcp_server/TOOLS.md` for a compact tool reference.

View File

@@ -1,3 +1,3 @@
# repo-seed
# NetKingdom
A git repository template to bootstrap coulomb projects from.
NetKingdom provides a dynamic self optimizing full circle security-platform for kubernetes deployed IT-infrastructures.

8
wiki/NetKingdom.md Normal file
View File

@@ -0,0 +1,8 @@
NetKingdom
*Open security core for dev sec ops on kubernetes*
Agent powered automation drives the cambrian explosion of vulnerabilities and security measures in modern IT.
This is a blueprint to bootstrap foundational dynamic security building on open source components minimizing threat exposure for critical use cases.
xxx

273
wiki/WorkplanOneChatgpt.md Normal file
View File

@@ -0,0 +1,273 @@
WorkplanOneChatgpt
*How to start according to chatgpt*
This is a protoplan not to be implemented directly but used as inspiration for an actual plan for implemenation!
Below is an integrated, Kubernetes-focused workplan that keeps the **“single private credential” bootstrap** idea, then rapidly evolves to **MFA-first admin access**, **least-privilege service accounts**, and **Keycloak SSO (OIDC/SAML) with privacyIDEA as the MFA engine**—exactly the combination privacyIDEA recommends. ([PrivacyIDEA][1])
---
## Architecture (Kubernetes target state)
**Ingress (TLS)**
* `keycloak.yourdomain.com` → Keycloak (SSO / sessions / OIDC+SAML)
* `pi.yourdomain.com` → privacyIDEA (MFA policies + token mgmt)
* optional: `pi-account.yourdomain.com` → privacyIDEA self-service portal
**Data**
* Keycloak DB (PostgreSQL)
* privacyIDEA DB (PostgreSQL or MariaDB; Postgres is a common choice)
* privacyIDEA encryption + audit key material stored as **K8s Secrets** (backed up securely)
**Integration**
* Keycloak runs the login flow
* privacyIDEA provides MFA via the **privacyIDEA Keycloak Provider** (JAR) ([GitHub][2])
---
## Workplan (phased deployment on Kubernetes)
### Phase 0 — Prereqs and “single private credential” rule
1. **Pick your one private credential**
Use it to unlock a vault (recommended), not as a daily admin password. The vault stores:
* initial privacyIDEA admin password (typed once)
* privacyIDEA crypto secrets (encryption/audit keys)
* Keycloak admin bootstrap secret
* DB credentials
2. Cluster prerequisites
* Ingress Controller (nginx / Traefik)
* `cert-manager` for TLS (Lets Encrypt or internal CA)
* StorageClass for persistent volumes
* (Optional but recommended) External Secrets Operator + real vault
---
### Phase 1 — Create namespaces, storage, network boundaries
1. Namespaces
* `sso` (Keycloak)
* `mfa` (privacyIDEA)
* `databases` (if you want shared DB ops)
2. NetworkPolicies (recommended)
* Only ingress controller can reach Keycloak/privacyIDEA services
* Only Keycloak pods can call privacyIDEA API endpoints needed for MFA
* Only app pods (or ingress) can reach Keycloak
This prevents lateral movement if any pod is compromised.
---
### Phase 2 — Deploy databases (prod-grade)
**Recommended**: PostgreSQL via a battle-tested chart/operator (CloudNativePG / Zalando / Bitnami Postgres chart).
1. Create DBs and users:
* `keycloak_db`, user `keycloak`
* `privacyidea_db`, user `privacyidea`
2. Store DB credentials as K8s Secrets (or ExternalSecrets).
3. Backups:
* Enable automated DB backups to object storage (S3/MinIO/etc.)
* Test restore early
---
### Phase 3 — Deploy privacyIDEA on Kubernetes (MFA core)
You have two common routes:
**Route A: Helm chart**
* There are community Helm approaches (example repo exists, but treat it as a starting point you should review/harden for production). ([GitHub][3])
**Route B: Roll your own manifests**
* Deployment + Service + Ingress + PVC + Secrets
* Often preferred for security review & long-term ownership
#### privacyIDEA critical secrets (do this before first real use)
privacyIDEA relies on secrets like `SECRET_KEY`, `PI_PEPPER`, and an encryption key file (`PI_ENCFILE`) / related key material. Generate once, mount as Secrets, back up carefully. ([GitHub][4])
(Those secrets are “if you lose them, you may lose access to encrypted token data”-class.)
**Work items**
1. Create K8s Secrets:
* `privacyidea-config` (env or ini fragments)
* `privacyidea-enckey` (encryption key material)
* `privacyidea-auditkeys` (audit signing/encryption keys)
2. Deploy privacyIDEA pods (Deployment) + Service
3. Add Ingress + TLS (`pi.yourdomain.com`)
4. Add rate limiting / WAF features at ingress (recommended)
#### Bootstrap: “single private credential”
1. Exec into the privacyIDEA pod and run the one-time admin creation (or equivalent `pi-manage` command).
2. That admin password is the **only password you ever manually type**.
3. Immediately log into WebUI and **enroll MFA for that admin** (TOTP/hardware token).
4. Immediately create:
* a **limited “trigger-admin”** service account used by Keycloak provider with *only* the `triggerchallenge`-type permission (least privilege)
* policies requiring MFA for admin actions
This aligns with the second opinion: bootstrap by one password, then transition to MFA + least privilege.
---
### Phase 4 — Deploy Keycloak on Kubernetes (SSO core)
Two common options:
**Option A: KeycloakX Helm chart (codecentric)**
* Widely used Helm chart for a StatefulSet-based Keycloak deployment. ([Artifact Hub][5])
**Option B: Keycloak Operator**
* Good for CRD-based lifecycle management; if your org prefers operators.
**Work items**
1. Deploy Keycloak pointing at `keycloak_db`
2. Configure Ingress + TLS (`keycloak.yourdomain.com`)
3. Bootstrap Keycloak admin secret (stored in vault / K8s Secret)
4. Set production settings:
* hostname strictness
* proxy mode
* metrics/logging
* realm import strategy (GitOps-friendly)
---
### Phase 5 — Add the privacyIDEA Keycloak Provider (JAR) in Kubernetes
The privacyIDEA provider is published on GitHub releases and needs to match your Keycloak version. ([GitHub][2])
**Kubernetes-friendly pattern**
1. Build a **custom Keycloak image**:
* Start from Keycloak base image
* Copy the provider JAR into `/opt/keycloak/providers/`
* Run `kc.sh build`
2. Deploy that image via Helm values (or operator spec)
This avoids “kubectl cp into running pods” and is the clean GitOps approach.
---
### Phase 6 — Configure the MFA flow (Keycloak → privacyIDEA)
**Work items**
1. In Keycloak:
* Create/choose realm
* Configure users source of truth (Keycloak internal or LDAP/AD)
2. Create Authentication Flow “privacyIDEA Browser”
* Add privacyIDEA execution step (REQUIRED)
* Configure:
* privacyIDEA base URL: `https://pi.yourdomain.com`
* service account: `trigger-admin` + secret (from K8s Secret)
* optional bypass group (“break-glass”) with strong restrictions & alerts
3. Set this flow as the default browser flow
4. Test:
* normal user MFA
* admin console MFA
* failure modes (OTP wrong, token missing, privacyIDEA down)
The providers releases/changelog track features like passkey handling and flow behavior—use the matching release notes when selecting versions. ([GitHub][2])
---
### Phase 7 — User management & token lifecycle (make it operable)
privacyIDEA 3.12 introduced a **Keycloak user resolver** (and Entra ID resolver), which can simplify identity alignment depending on your setup. ([NetKnights][6])
**Work items**
1. Decide identity source of truth:
* simplest: Keycloak internal users → privacyIDEA resolver (Keycloak resolver if you want that pattern)
* enterprise: LDAP/AD shared by both
2. Enable self-service portal where appropriate
3. Define policies:
* allowed token types
* enrollment rules
* admin rights separation (super-admin vs helpdesk-admin)
4. Add auditing / log shipping (privacyIDEA audit logs + Keycloak events)
---
### Phase 8 — Backups, DR, upgrades, and break-glass
**Backups**
* DB backups (Keycloak + privacyIDEA)
* privacyIDEA encryption/audit keys Secrets (encrypted export)
* realm exports (Keycloak) as code / git
**Disaster recovery drill**
* restore DB + keys into a fresh namespace
* verify token validation still works (this catches key/secret mistakes)
**Break-glass**
* keep a *disabled-by-default* Keycloak admin path or group exemption
* store break-glass credentials offline + vault
* alert on every use
---
## Deliverables checklist (what “done” looks like)
* [ ] `mfa` + `sso` namespaces deployed via GitOps/Helm
* [ ] TLS everywhere (cert-manager)
* [ ] privacyIDEA secrets generated, mounted, backed up
* [ ] privacyIDEA admin created once (single typed password), MFA enrolled immediately
* [ ] `trigger-admin` exists with minimal rights
* [ ] Keycloak runs from custom image including privacyIDEA provider JAR
* [ ] Keycloak “privacyIDEA Browser” flow enforced
* [ ] DB backups + restore tested
* [ ] NetworkPolicies in place
---
If you tell me **which Kubernetes flavor** youre using (k3s / AKS / EKS / GKE / OpenShift) and whether you want **Postgres operator vs Helm**, Ill turn this into a concrete, copy-pastable “repo plan” (folders, Helm values skeletons, Secrets/ExternalSecrets layout, and the exact build steps for the custom Keycloak image) without adding unnecessary complexity.
[1]: https://www.privacyidea.org/?feed=&utm_source=chatgpt.com "privacyIDEA 3.12 is available"
[2]: https://github.com/privacyidea/keycloak-provider/releases?utm_source=chatgpt.com "Releases · privacyidea/keycloak-provider"
[3]: https://github.com/gpappsoft/privacyidea-helm?utm_source=chatgpt.com "A helm chart to install a privacyIDEA stack"
[4]: https://github.com/privacyidea/privacyidea?utm_source=chatgpt.com "privacyidea/privacyidea: :closed_lock_with_key"
[5]: https://artifacthub.io/packages/helm/codecentric/keycloakx?utm_source=chatgpt.com "keycloakx 7.1.8"
[6]: https://netknights.it/en/press/multi-faktor-authentifizierung-privacyidea-in-version-3-12-erschienen/?utm_source=chatgpt.com "Multi-Factor Authentication privacyIDEA version 3.12 released"
xxx

171
wiki/WorkplanOneGrok.md Normal file
View File

@@ -0,0 +1,171 @@
WorkplanOneGrok
*How to start according to grok*
This is a protoplan not to be implemented directly but used as inspiration for an actual plan for implemenation!
**Integrated architecture (merging both opinions)**
Your single private credential remains the **root-of-trust** that only ever unlocks a vault. Everything else (initial admin passwords, database credentials, privacyIDEA crypto roots, TLS keys, Keycloak admin secret) lives inside the vault and is never typed manually again.
- **privacyIDEA** = MFA/token/policy engine (user resolvers, self-service enrollment, token types, audit)
- **Keycloak** = full SSO IdP (OIDC/SAML clients, realms, groups, session management)
- **Integration** = Keycloak PrivacyIDEA provider (MFA step in the browser flow)
- **Vault** = single unlock point (KeePassXC/Bitwarden local, or HashiCorp Vault in-cluster)
- **Databases** = private, encrypted, backed up with keys from the vault
- **K8s layer** = Helm + Official Keycloak Operator + community privacyIDEA Helm chart (gpappsoft/privacyidea) + cert-manager + Ingress
This satisfies both sources: you bootstrap from literally one secret, immediately move to MFA-protected admins, treat crypto roots (SECRET_KEY, PI_PEPPER, PI_ENCFILE) as crown jewels, and run everything declaratively on Kubernetes.
### Kubernetes Workplan (Day 07)
**Prerequisites**
- Running Kubernetes cluster (k3s, kind, EKS, AKS, GKE any with StorageClass and cert-manager installed).
- kubectl, Helm 3.
- Ingress controller (nginx-ingress or Traefik).
- cert-manager for automatic TLS.
- Your single master credential ready (KeePassXC file or age private key).
#### Day 0 Vault & Secret Bootstrap (single-credential principle)
1. Create your vault (recommended: KeePassXC .kdbx file or self-hosted Bitwarden; for production add HashiCorp Vault later).
2. Inside the vault generate/store (all random, never reuse):
- privacyIDEA: `SECRET_KEY` (64+ chars), `PI_PEPPER` (32+ chars), encryption key file content (`pi-manage create_enckey`).
- MariaDB root + privacyIDEA DB user passwords.
- Keycloak admin initial secret + DB password.
- TLS ACME account key (if not using cert-manager fully).
- Break-glass admin credentials + offline recovery OTP seed.
3. Export an encrypted “ops bundle” (age-encrypted tar of all secret YAML manifests) this bundle is the only thing you ever decrypt with your single credential.
4. Enable cluster encryption-at-rest (if not already).
#### Day 1 Foundation & Databases
```bash
# Deploy HashiCorp Vault (optional but ideal for rotation)
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault --namespace vault --create-namespace -f vault-values.yaml
# Or use simple sops/age for GitOps
```
- Deploy MariaDB (Bitnami Helm or Crunchy Postgres Operator).
- Create Kubernetes Secrets from the vault (or use External Secrets Operator + Vault backend):
```yaml
apiVersion: v1
kind: Secret
metadata:
name: privacyidea-secrets
data:
SECRET_KEY: <base64-from-vault>
PI_PEPPER: <base64-from-vault>
PI_ENCFILE: <base64-content>
```
- Apply network policies: DBs only reachable from privacyIDEA/Keycloak pods.
#### Day 23 Deploy privacyIDEA (MFA core)
The community Helm chart (gpappsoft/privacyidea on Artifact Hub) deploys the full stack (privacyIDEA + MariaDB + optional Redis/RADIUS).
```bash
helm repo add privacyidea https://gpappsoft.github.io/helm-charts
helm install privacyidea privacyidea/privacyidea \
--namespace privacyidea --create-namespace \
-f values-privacyidea.yaml
```
Key parts of `values-privacyidea.yaml` (populated from vault):
```yaml
database:
password: <from-vault>
privacyidea:
config:
SECRET_KEY: <from-vault>
PI_PEPPER: <from-vault>
encfile: # mounted as secret volume
enabled: true
existingSecret: privacyidea-secrets
key: PI_ENCFILE
admin:
bootstrap: true # chart runs pi-manage internally
ingress:
enabled: true
hostname: pi.yourdomain.com
tls: true
```
Post-install (one-time job or manual):
- `kubectl exec` into privacyIDEA pod and run `pi-manage admin add pi-admin --email admin@yourdomain.com` (password from vault).
- Immediately enroll an MFA token for `pi-admin` via the UI.
- Create limited “trigger-admin” (only `triggerchallenge` right) for Keycloak.
- Apply day-1 policies: WebUI restricted to VPN/office IPs, MFA required for all admin actions, enrollment policies locked down.
#### Day 45 Deploy Keycloak + privacyIDEA MFA integration
1. Install the official Keycloak Operator:
```bash
kubectl apply -f https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/main/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
kubectl apply -f https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/main/kubernetes/kubernetes.yml -n keycloak
```
2. Prepare the PrivacyIDEA provider JAR:
```bash
wget https://github.com/privacyidea/keycloak-provider/releases/latest/download/PrivacyIDEA-Provider.jar
kubectl create configmap privacyidea-provider --from-file=PrivacyIDEA-Provider.jar -n keycloak
```
3. Create Keycloak CR (with custom provider mount):
```yaml
apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
name: keycloak
namespace: keycloak
spec:
instances: 2
db:
vendor: postgres # or mariadb
username: keycloak
passwordSecret: keycloak-db-secret # from vault
unsupported:
podTemplate:
spec:
volumes:
- name: providers
configMap:
name: privacyidea-provider
containers:
- name: keycloak
volumeMounts:
- name: providers
mountPath: /opt/keycloak/providers
additionalOptions:
- name: spi-authenticator-privacyidea-enabled
value: "true"
- name: spi-authenticator-privacyidea-url
value: "https://pi.yourdomain.com"
- name: spi-authenticator-privacyidea-service-account
value: "trigger-admin" # limited rights only
ingress:
enabled: true
```
4. In Keycloak UI (or via CR import): create realm, import same user resolver as privacyIDEA (LDAP/Keycloak internal/Entra), create custom browser flow with “PrivacyIDEA Forms” execution (REQUIRED). Bind as default flow.
#### Day 67 Policies, Self-Service, Break-Glass & Testing
- privacyIDEA: enable self-service enrollment policies + registration tokens.
- Keycloak: require MFA step-up for admin console and sensitive clients.
- Break-glass: store offline admin + recovery OTP in sealed envelope + vault. Alert on usage.
- End-to-end test: app → Keycloak → privacyIDEA OTP/push → SSO session.
- Configure backups (Velero or operator) encrypted with vault key.
- Monitoring: Prometheus + Grafana (optional).
### Security & Operations Checklist (do these before going live)
- All external traffic via Ingress + HSTS + strict TLS.
- privacyIDEA encryption keys backed up and never in image.
- Least-privilege service accounts everywhere.
- NetworkPolicies + PodSecurityPolicies.
- Rotate secrets via vault (External Secrets Operator handles re-injection).
- Test recovery: vault unlock → restore DB + keys → everything works.
This plan is production-ready, fully declarative, and scales. You start with one memorized credential and end with a hardened, auditable SSO+MFA platform.
If you tell me:
- Cluster type (k3s, EKS, bare-metal…)
- Existing user store (LDAP/AD/Entra/Keycloak internal)
- Target apps (Nextcloud, GitLab, custom…)
- Preference for GitOps (ArgoCD/Flux) or manual Helm
I can give you the exact `values.yaml` files, CRDs, and one-click scripts.
xxx

View File

@@ -0,0 +1,361 @@
---
id: NK-WP-0001
type: workplan
title: "SSO & MFA Platform — Keycloak + privacyIDEA on Kubernetes"
domain: netkingdom
status: active
owner: worsch
topic_slug: netkingdom
state_hub_workstream_id: 39263c4b-ef70-4053-b782-350834b7e1be
created: "2026-02-28"
updated: "2026-02-28"
---
# SSO & MFA Platform — Keycloak + privacyIDEA on Kubernetes
## Summary
Deploy a hardened SSO and MFA platform on Kubernetes: Keycloak as the
OIDC/SAML identity provider, privacyIDEA as the MFA/token engine,
integrated via the privacyIDEA Keycloak Provider. This is the foundational
security layer for the net-kingdom DevSecOps platform.
## Context
Synthesised from two AI protoplans (wiki/WorkplanOneChatgpt.md and
wiki/WorkplanOneGrok.md). Both sources converge on the same architecture;
this plan picks the most concrete and production-aligned choices from each:
- **Single-credential bootstrap** (Grok) — one master secret unlocks the
vault; all other credentials are vault-managed and never typed manually.
- **Phase structure** (ChatGPT) — eight sequential phases reducing blast
radius at each step.
- **Tooling choices** (both) — Keycloak Operator or codecentric Helm,
gpappsoft privacyIDEA Helm, CloudNativePG for PostgreSQL, cert-manager
for TLS, Traefik as ingress (K3s native, aligned with Railiance).
- **Custom Keycloak image** (both) — JAR baked into image via `kc.sh build`
rather than `kubectl cp`; clean GitOps pattern.
## Architecture
```
Internet
│ TLS (cert-manager / Let's Encrypt)
┌──────┴──────┐
│ Traefik │ (K3s native ingress)
└──┬───────┬──┘
│ │
keycloak.… pi.… pi-account.…
│ │ │
┌──────┘ ┌────┘ │
▼ ▼ │
[Keycloak] [privacyIDEA]◄──┘ (self-service portal)
│ │
└────┬────┘
[PostgreSQL] (CloudNativePG, namespace: databases)
[Vault / K8s Secrets] ← single credential unlocks
```
**Namespaces:** `sso` (Keycloak), `mfa` (privacyIDEA), `databases`
**Integration:** Keycloak runs the browser login flow; privacyIDEA provides
MFA via the privacyIDEA Keycloak Provider JAR (baked into custom image).
## Dependencies
- Depends on: `railiance/three-phoenix-ha-cluster` — full production
deployment targets the ThreePhoenix K3s HA cluster. Development/staging
can proceed on a single-node k3s instance.
- Depends on: `railiance/phase-0-operational-baseline` — cert-manager, TLS,
backup strategy must be operational before going live.
## Tasks
### T01 — Phase 0: Vault & secret bootstrap (single-credential principle)
```task
id: NK-WP-0001-T01
state_hub_task_id: 7992528c-d533-44e5-bcce-f92aaa2b75b2
status: todo
priority: critical
```
Create the vault (KeePassXC .kdbx or self-hosted Bitwarden; HashiCorp Vault
for later production hardening). Generate and store all secrets inside the
vault — never typed again:
- privacyIDEA: `SECRET_KEY` (64+ chars), `PI_PEPPER` (32+ chars),
`PI_ENCFILE` content (`pi-manage create_enckey`).
- PostgreSQL: root + `keycloak` + `privacyidea` user passwords.
- Keycloak: admin bootstrap secret + DB password.
- TLS: ACME account key (if not delegated fully to cert-manager).
- Break-glass: admin credentials + offline recovery OTP seed.
Export an age-encrypted ops bundle (encrypted tar of all secret YAML
manifests). Enable K8s encryption-at-rest. Confirm secret injection
strategy: External Secrets Operator + Vault backend, or sops/age for GitOps.
**Done when:** vault created, all secrets generated, encrypted ops bundle
exported and stored offsite. Secret injection strategy decided.
---
### T02 — Phase 1: K8s foundations (namespaces, NetworkPolicies, cert-manager)
```task
id: NK-WP-0001-T02
state_hub_task_id: 721ca6b2-0cf4-4008-a966-87b1563550fa
status: todo
priority: high
```
Create namespaces: `sso`, `mfa`, `databases`. Verify cert-manager is
installed and functional on the K3s cluster (Traefik ingress). Define and
apply NetworkPolicies to prevent lateral movement:
- Only ingress controller reaches Keycloak/privacyIDEA service ports.
- Only Keycloak pods call the privacyIDEA API.
- Only app pods/ingress reach Keycloak.
- DB pods reachable only from `sso` and `mfa` namespaces.
Verify StorageClass for PVCs.
**Done when:** namespaces exist, NetworkPolicies applied and tested (verify
denied paths), cert-manager issues a test certificate.
---
### T03 — Phase 2: PostgreSQL deployment (Keycloak + privacyIDEA DBs)
```task
id: NK-WP-0001-T03
state_hub_task_id: 7fa60004-deb2-4db5-a470-f95dda07f6ab
status: todo
priority: high
```
Deploy PostgreSQL via CloudNativePG operator (preferred: aligns with
ThreePhoenix HA posture) or Bitnami Helm chart as fallback. Create:
- Database `keycloak_db`, user `keycloak`
- Database `privacyidea_db`, user `privacyidea`
Store DB credentials as K8s Secrets (or ExternalSecrets from vault).
Configure automated DB backups to object storage (S3 or MinIO).
**Run a restore drill before proceeding** — a failed restore later is a
critical blocker.
**Done when:** both DBs live, credentials in K8s Secrets, backup running,
restore drill passed.
---
### T04 — Phase 3: Deploy privacyIDEA (MFA core)
```task
id: NK-WP-0001-T04
state_hub_task_id: 6ad1296a-a488-4031-b665-f77030e971ed
status: todo
priority: high
```
Deploy privacyIDEA via `gpappsoft/privacyidea` Helm chart (Artifact Hub) or
custom manifests (Deployment + Service + Ingress + PVC + Secrets). Key
Helm values:
```yaml
database:
password: <from-vault>
privacyidea:
config:
SECRET_KEY: <from-vault>
PI_PEPPER: <from-vault>
encfile:
enabled: true
existingSecret: privacyidea-secrets
key: PI_ENCFILE
ingress:
enabled: true
hostname: pi.yourdomain.com
tls: true
```
Create K8s Secrets: `privacyidea-config`, `privacyidea-enckey`,
`privacyidea-auditkeys`. Configure Ingress + TLS. Add rate-limiting and
WAF rules at Traefik level.
**Bootstrap (single-credential moment):**
1. `kubectl exec` into pod, run `pi-manage admin add pi-admin` — password
comes from vault (only time a password is typed).
2. Immediately enroll MFA for `pi-admin` (TOTP or hardware token).
3. Create `trigger-admin` with `triggerchallenge` right only.
4. Apply policies: WebUI restricted to VPN/office IPs; MFA required for
all admin actions.
**Done when:** privacyIDEA reachable at pi.yourdomain.com with valid TLS,
pi-admin enrolled with MFA, trigger-admin created, rate-limiting active.
---
### T05 — Phase 4: Deploy Keycloak (SSO core)
```task
id: NK-WP-0001-T05
state_hub_task_id: b9f73aa6-9035-4643-9905-64e73a29b298
status: todo
priority: high
```
Build a **custom Keycloak image** that includes the privacyIDEA Provider JAR:
```dockerfile
FROM quay.io/keycloak/keycloak:<version>
COPY PrivacyIDEA-Provider.jar /opt/keycloak/providers/
RUN /opt/keycloak/bin/kc.sh build
```
Deploy via official Keycloak Operator (CRD-based) or codecentric KeycloakX
Helm chart. Configure:
- DB: `keycloak_db` (credentials from K8s Secret)
- Ingress + TLS: `keycloak.yourdomain.com` (Traefik + cert-manager)
- Hostname strictness + proxy mode (Traefik forward headers)
- Metrics/logging (Prometheus annotations)
- Admin bootstrap secret from vault
- Realm import strategy: GitOps-friendly (realm JSON in git or CR)
**Done when:** Keycloak reachable with valid TLS, admin console accessible,
custom image with privacyIDEA JAR deployed and verified.
---
### T06 — Phase 5: Realm config & MFA authentication flow
```task
id: NK-WP-0001-T06
state_hub_task_id: 3b6379a4-a27b-4d25-82be-bc600879f036
status: todo
priority: medium
```
In Keycloak:
1. Create/configure realm; set identity source of truth (Keycloak internal
users recommended for initial deployment; LDAP/AD or Entra as extension).
2. Create Authentication Flow "privacyIDEA Browser":
- Add privacyIDEA execution step (REQUIRED)
- Config: privacyIDEA URL = `https://pi.yourdomain.com`, service account
= `trigger-admin` (secret from K8s Secret)
- Optional: bypass group (break-glass) with strict restrictions + alerts
3. Set this flow as the default browser flow.
4. Require MFA step-up for admin console and sensitive OIDC clients.
Test:
- Normal user: password → MFA OTP → session established
- Admin console: MFA required
- Failure modes: wrong OTP, token missing, privacyIDEA unreachable
- Break-glass: bypass works, alert fires
**Done when:** end-to-end auth works for normal and admin paths, all failure
modes handled gracefully.
---
### T07 — Phase 6: User management, policies & self-service portal
```task
id: NK-WP-0001-T07
state_hub_task_id: c7cf902a-b480-4545-a536-293070945206
status: todo
priority: medium
```
Decide and implement identity source of truth (Keycloak internal →
privacyIDEA Keycloak resolver, or LDAP/AD shared). The privacyIDEA 3.12+
Keycloak user resolver simplifies alignment.
Define policies in privacyIDEA:
- Allowed token types: TOTP, hardware (YubiKey), passkey
- Enrollment rules (who can self-enroll, which token types)
- Admin rights separation: super-admin vs. helpdesk-admin
Enable self-service portal at `pi-account.yourdomain.com` for user token
enrollment/replacement.
Configure auditing and log shipping: privacyIDEA audit logs + Keycloak
events → centralized logging (ELK/Loki or equivalent). Token lifecycle
policies: enrollment, revocation, re-enrollment on device loss.
**Done when:** policies documented and applied, self-service portal live,
audit logs flowing.
---
### T08 — Phase 7: Backups, DR, break-glass & monitoring
```task
id: NK-WP-0001-T08
state_hub_task_id: 9cbd1d89-b5bf-491e-9d16-b1c7d57076fb
status: todo
priority: medium
```
**Backups:**
- DB backups: Keycloak + privacyIDEA (Velero or CloudNativePG scheduled
backup to S3/MinIO). Test restore.
- privacyIDEA encryption/audit key Secrets: encrypted export, versioned.
- Keycloak realm exports: stored as JSON in git (GitOps-friendly).
**Disaster recovery drill** (mandatory before production):
1. Restore DB + keys into a fresh namespace.
2. Verify token validation still works — this catches key/secret mistakes.
**Break-glass procedure:**
- Disabled-by-default Keycloak admin path or group exemption.
- Break-glass credentials stored offline + vault. Alert (PagerDuty/webhook)
on every use.
**Monitoring:**
- Prometheus scraping Keycloak + privacyIDEA metrics.
- Grafana dashboards: auth success/failure rates, MFA challenge latency,
token count by type.
- Alert: privacyIDEA unreachable (blocks all logins).
**Final validation:**
- All external traffic: Ingress + HSTS + strict TLS.
- NetworkPolicies verified (no unintended open paths).
- End-to-end: app → Keycloak → privacyIDEA OTP → SSO session established.
**Done when:** DR drill passed, monitoring live, break-glass procedure
documented and tested, HSTS and NetworkPolicies verified.
---
## Deliverables Checklist
- [ ] Vault created; all secrets generated and encrypted ops bundle exported
- [ ] `sso`, `mfa`, `databases` namespaces + NetworkPolicies deployed
- [ ] TLS everywhere via cert-manager (Traefik ingress)
- [ ] PostgreSQL live; both DBs created; backup + restore tested
- [ ] privacyIDEA running at `pi.yourdomain.com`; pi-admin MFA enrolled;
trigger-admin created with least-privilege rights
- [ ] Keycloak running from custom image including privacyIDEA Provider JAR
- [ ] Keycloak "privacyIDEA Browser" flow enforced as default
- [ ] Realm exported to git; admin secret from vault
- [ ] Self-service portal live; token lifecycle policies defined
- [ ] DR drill passed; monitoring live; break-glass documented and tested
## Open Questions / Extension Points
- **Vault backend**: KeePassXC (simple) vs HashiCorp Vault in-cluster
(rotation, audit trail). Start with KeePassXC; upgrade to Vault when
ThreePhoenix cluster is stable.
- **Identity source of truth**: Keycloak-internal vs LDAP/AD/Entra.
Decision needed before T07.
- **GitOps tooling**: ArgoCD or Flux for declarative Helm management?
Aligns with Railiance staged-promotion-lifecycle workstream.
- **Cluster target**: Development on single-node k3s; production on
ThreePhoenix (3-node HA). Workplan covers both; HA-specific steps noted
where they diverge.