net-kingdom/workplans/NK-WP-0003-keycape-privacyidea-cluster-deployment.md

id, type, title, domain, repo, status, owner, topic_slug, created, updated, state_hub_workstream_id

id	type	title	domain	repo	status	owner	topic_slug	created	updated	state_hub_workstream_id
NK-WP-0003	workplan	KeyCape + privacyIDEA Stack — Cluster Deployment	netkingdom	net-kingdom	active	custodian	netkingdom	2026-03-20	2026-03-20	f24cefd4-a09b-4fa1-9b25-94bf783b425e

KeyCape + privacyIDEA Stack — Cluster Deployment

Goal

Deploy the full NetKingdom identity stack on the live k3s cluster without Keycloak. KeyCape (v0.1, complete) is the OIDC orchestration layer; it binds LLDAP (directory), Authelia (auth sessions), and privacyIDEA (MFA).

NK-WP-0001 was scoped around Keycloak and is deferred. This workplan covers everything needed to reach a production-ready identity plane.

Pre-conditions

• k3s cluster healthy — RAIL-BS-WP-0002 ✓
• kubeconfig available at ~/.kube/config-hosteurope — RAIL-BS-WP-0005 ✓
• All manifests committed — net-kingdom sso-mfa/k8s/ ✓
• KeyCape v0.1 complete — KEY-WP-0001 ✓
• SOPS + age integrated into net-kingdom (T01 below)
• Credential ops-bundle generated and stored in KeePassXC (T01 below)

Architecture

Internet → Traefik (k3s) → cert-manager TLS
                ├── auth.coulomb.social   → Authelia
                ├── pink.coulomb.social   → privacyIDEA portal
                └── id.coulomb.social     → KeyCape (OIDC)

KeyCape ──► Authelia (session, password)
        ──► LLDAP   (directory, user lookup)
        ──► privacyIDEA (MFA challenges via trigger-admin token)

privacyIDEA ──► PostgreSQL (privacyidea_db via CloudNativePG)
LLDAP       ──► PostgreSQL (lldap_db via CloudNativePG)
Authelia    ──► PostgreSQL (authelia_db via CloudNativePG)

Tasks

T01 — Credential setup: SOPS + age + ops-bundle

id: NK-WP-0003-T01
status: todo
priority: high
state_hub_task_id: "6a22e17e-5854-4f8b-b419-9dc86d490357"

Net-kingdom currently uses a manual KeePassXC + age-bundle approach while railiance-infra uses SOPS with age keys. This task aligns them under the Credential Management Standard (canon/standards/credential-management_v0.1.md).

Steps:

1. Verify the operator age keypair exists at ~/.config/sops/age/key.txt (reuse the railiance key if already present — one keypair per operator)
2. Add .sops.yaml to net-kingdom root (mirror railiance-infra pattern):
   • Encrypt files matching secrets/.* and **/*.sops.yaml
   • Recipient: operator age public key
3. Run sso-mfa/bootstrap/gen-secrets.sh ./secrets to generate all service secrets
4. Store each secret in KeePassXC under the net-kingdom/ group hierarchy (see credential management standard for group layout)
5. Run sso-mfa/bootstrap/pack-bundle.sh ./secrets <age-pub-key> → encrypted ops bundle
6. Store ops bundle offsite (separate from KeePassXC)
7. Shred plaintext secrets: find secrets/ -type f -exec shred -u {} \;

T02 — Apply cluster foundations

id: NK-WP-0003-T02
status: todo
priority: high
state_hub_task_id: "a14e3a6b-18ee-4172-8a47-bd531f21e55a"

Apply the K8s infrastructure foundations. All manifests already committed.

export KUBECONFIG=~/.kube/config-hosteurope
kubectl apply -f sso-mfa/k8s/namespaces/
kubectl apply -f sso-mfa/k8s/network-policies/
kubectl apply -f sso-mfa/k8s/cert-manager/

Verify: bash sso-mfa/k8s/verify-t02.sh

Expected: namespaces sso, mfa, databases exist; NetworkPolicies applied; cert-manager pods Running.

T03 — Deploy PostgreSQL (CloudNativePG)

id: NK-WP-0003-T03
status: todo
priority: high
state_hub_task_id: "19e375d0-66bd-4cf0-9c2d-59d5c0d5989e"

Deploy the shared database cluster with three databases:

• privacyidea_db — privacyIDEA
• lldap_db — LLDAP
• authelia_db — Authelia

kubectl apply -f sso-mfa/k8s/postgres/

Wait for cluster to be Ready, then verify: bash sso-mfa/k8s/verify-t03.sh

Note: Do not proceed to T04 until the CloudNativePG cluster is fully healthy. Migration jobs will fail on a partially-started cluster.

T04 — Deploy privacyIDEA

id: NK-WP-0003-T04
status: todo
priority: high
state_hub_task_id: "9c9c1ec9-0cf5-4546-a83e-d74dbf3b27af"

Deploy privacyIDEA into the mfa namespace.

Image fix applied (2026-03-20): privacyidea/privacyidea:3.12 does not exist. Corrected to privacyidea/otpserver:3.12.2 on port 5001. Updated: deployment.yaml, ingress.yaml, netpol-mfa.yaml, netpol-sso.yaml.

Step 1 — Create K8s secrets from KeePassXC:

cd sso-mfa/k8s/privacyidea
bash create-secrets.sh   # reads from env vars; source from KeePassXC

Step 2 — Apply manifests:

kubectl apply -f pvc.yaml
kubectl apply -f configmap.yaml
kubectl apply -f middleware.yaml
kubectl apply -f deployment.yaml
kubectl apply -f ingress.yaml

Step 3 — Bootstrap key material (time-sensitive): Run immediately once the pod reaches Running state. This window must not be missed — if the pod is deleted before this runs, the enckey is lost.

bash enckey-bootstrap.sh   # extracts PI_ENCFILE + audit keys → K8s Secrets + KeePassXC

Step 4 — Create admin accounts:

bash bootstrap-admin.sh    # creates pi-admin + trigger-admin, sets policies
# store trigger-admin token in KeePassXC net-kingdom/privacyidea/trigger-admin

Verify: bash sso-mfa/k8s/verify-t04.sh

Expected: pod Running, TLS cert issued for pink.coulomb.social, admin accounts exist, enckey backed up.

T05 — Deploy LLDAP

id: NK-WP-0003-T05
status: todo
priority: high
state_hub_task_id: "82fc90f7-8eb4-4718-b02a-dfd5fa39e5bc"

Deploy LLDAP into the sso namespace.

cd sso-mfa/k8s/lldap
bash create-secrets.sh
kubectl apply -f deployment.yaml
kubectl apply -f ingress.yaml
kubectl apply -f middleware.yaml
bash bootstrap-users.sh   # creates base OU structure + initial admin user

Verify pod Running and LDAP bind works on ldap.coulomb.social.

T06 — Deploy Authelia

id: NK-WP-0003-T06
status: todo
priority: high
state_hub_task_id: "3a28ff10-fbfa-443b-a64d-bbfe6153c544"

Deploy Authelia into the sso namespace.

cd sso-mfa/k8s/authelia
bash create-secrets.sh
kubectl apply -f configmap.yaml
kubectl apply -f deployment.yaml
kubectl apply -f ingress.yaml

Verify: bash sso-mfa/k8s/verify-t05.sh (covers LLDAP + Authelia together)

T07 — Deploy KeyCape

id: NK-WP-0003-T07
status: todo
priority: high
state_hub_task_id: "496a97c9-3e2a-486e-ba62-18449868c6cf"

Deploy KeyCape into the sso namespace.

cd sso-mfa/k8s/keycape
bash create-secrets.sh       # includes privacyIDEA trigger-admin token
bash create-pi-token.sh      # registers KeyCape as a privacyIDEA application
kubectl apply -f deployment.yaml
kubectl apply -f ingress.yaml
kubectl apply -f middleware.yaml

Verify: OIDC discovery endpoint reachable at https://id.coulomb.social/.well-known/openid-configuration

T08 — End-to-end authentication test

id: NK-WP-0003-T08
status: todo
priority: high
state_hub_task_id: "0fba3392-c916-43fd-a2c1-24ce39481043"

Prove the full auth flow works:

1. OIDC discovery resolves at id.coulomb.social
2. Authelia password auth succeeds for a test user
3. privacyIDEA TOTP challenge issued and accepted
4. KeyCape issues a valid access token
5. Token introspection returns expected claims (sub, groups, email)

Use the KeyCape acceptance test suite:

cd /home/worsch/key-cape
go test ./tests/... -run TestProfileBaseline -v

T09 — Backup, DR, and monitoring

id: NK-WP-0003-T09
status: todo
priority: medium
state_hub_task_id: "a82751d8-4de8-4668-8568-8dc140a6322b"

Operational hardening:

1. Deploy backup CronJob for CloudNativePG → MinIO/S3

   kubectl apply -f sso-mfa/k8s/backup/
   

2. Execute DB restore drill (mandatory before production traffic): restore privacyidea_db from a backup into a test namespace, verify privacyIDEA starts cleanly with the restored data
3. Deploy break-glass admin access (disabled by default):

   bash sso-mfa/k8s/lldap/break-glass.sh setup
   

4. Verify Prometheus scraping for privacyIDEA and Authelia metrics
5. Confirm NetworkPolicies block all unexpected egress

Verify: bash sso-mfa/k8s/verify-t08.sh (if exists) or manual checklist from NK-WP-0001 T08 scope.

Done criteria

• All verify-t*.sh scripts exit 0
• KeyCape acceptance test suite passes
• DB restore drill completed
• All key material backed up in KeePassXC + ops bundle
• privacyIDEA enckey backed up (K8s Secret + KeePassXC)
• Monitoring active (Prometheus scraping all three services)