# Security Bootstrap Retrospective and Automation Gap Matrix

**Status:** initial draft (NET-WP-0018-T03)
**Date:** 2026-06-03
**Scope:** Captures the first successful NetKingdom bootstrap (through 0015-0017 "Reopen under custody" + 0019 T06-adjacent polish) and automation gaps. Input from T02 runtime architecture doc, prior workplan notes/evidence (0017/0019), console metadata/.local flags, operator experience, assessment, and pragmatic records (progress events, evidence.json, git, console validators). 

This feeds T05 (smooth guide with evidence per step), T06/T08 (control surface + validations), T09 (rebuild risk/rehearsal), and future optimization.

Not a full historical dump; focused on bumps, diagnosis, current automation status, and remaining fragile/manual assumptions.

## Executive Summary

The first bootstrap proved the stack (LLDAP + Authelia + privacyIDEA + KeyCape + OpenBao + local bootstrap console + State Hub) can deliver a working IAM/security bootstrap path with platform-root custody, MFA, OIDC admin bridging to OpenBao, and non-root user lifecycle.

It also proved the system is easy to derail interactively: realm drift, OIDC callback bridging, LLDAP lookup assumptions, OpenBao claim/policy shape, token expiry/revocation, operator-state persistence (.local metadata), secret taint hygiene, and audit correlation.

**Post-0017/0019 state (S6 Reopen under custody):**
- Many gates now durable/non-secret in .local/security-bootstrap.json + console (platform_reopened, cleanup_complete, oidc_login_verified, audit_core_bootstrap_risk_accepted with owner/review 2026-07-02, etc.).
- Control surface (console + make + web-ui) + evidence templates/validators provide repeatable safe checks and runbooks.
- 0019 added repeatable non-root dry-run orchestration ( /tmp hygiene + k8s fallback, create --test, verifs, lock/offboard, evidence with 12+ exact bools, validate) + claims helper + cleanup + runbook exposure. Makes T06 gate automatable and less manual/taint-prone.
- Pragmatic audit (local-identity TSV + OpenBao PVC+mock + State Hub/console evidence) in use; production Audit Core deferred (risk accepted).
- T02 arch doc now exists as specific-as-deployed baseline (including pragmatic audit paths and UE gaps).

**Key wins (now automated or evidenced):**
- S6 reopen + custody approval + MFA self-enroll + KeyCape OIDC admin path to OpenBao.
- Evidence discipline for cleanup (T03/T04 0017), lifecycle flow (T05), onboarding dry-run (T06 0017/0019) — all validate exact bools + no secrets.
- Dry-run user lifecycle repeatable/safe (0019 orchestrator + console/make).
- Non-secret progress in console metadata + State Hub /progress/ (used for tracking 0018 impl itself).

**Remaining gaps / fragile assumptions (see matrix):**
- Consolidated smooth bootstrap guide (T05; piecemeal docs + console guide exist but not one operator sequence with per-step evidence).
- Full automated tests/validations for all UI sections/runbooks (T07; 0019 pieces are ready for coverage; more gates needed in T08).
- UE integration (adapters, claims_enrichment routing, membership sync, audit correlation, bootstrap-to-governed transition) — biggest per assessment; current paths direct LLDAP/Keycloak (valid for bootstrap per contract but drift risk).
- Production Audit Core + full correlation (deferred; pragmatic sufficient for now).
- Scratch rebuild risk rehearsal plan (T09; needs T03/T05/T07/T08 complete).
- Some interactive repairs still required for drift cases (realm, callbacks, claims); console exposes repair but not fully preventive.

A scratch rebuild should only happen after the guide + validations + T09 say which human interactions are unavoidable.

## Bumps Encountered, Diagnosis, and Current Status

(Compiled from 0017/0019 notes, T02 doc, console, metadata, assessment, operator history. Each: bump, root cause/diagnosis, was it automated now?, remaining manual/fragile?)

1. **Realm drift / privacyIDEA repair (early bootstrap):**
   - Bump: Realm/resolver not correctly set for LLDAP users; self-enroll or admin MFA broken.
   - Diagnosis: Manual inspection of privacyIDEA admin + LLDAP queries; repair-realm-live.sh or similar.
   - Now automated?: Console has "privacyIDEA realm repair" runbook (template + attended steps); validate-t02 etc. check related. repair script exists.
   - Remaining: Still attended (not fully declarative/CI); needs live cluster + operator. Gap in T05 guide + T08 validator for "realm healthy for bootstrap users".

2. **OIDC callback bridging / KeyCape client registration:**
   - Bump: Redirects (localhost:8250 etc.) or client config not matching; login fails for console/OpenBao.
   - Diagnosis: KeyCape client definition in code (create-secrets.sh); apply + restart; verify via console OIDC login.
   - Now automated?: KeyCape OpenBao client definition (non-secret in source) + "KeyCape OpenBao client deployed" gate; console preflight + status checks.
   - Remaining: Manual apply/restart cycle for drift; documented in T02 but no preventive validator yet (T08 target).

3. **LLDAP lookup assumptions / group membership for claims:**
   - Bump: Groups (net-kingdom-admins/users) not reflected in OIDC claims or policy; platform-admin not granted or over-granted.
   - Diagnosis: Direct LLDAP queries (GraphQL or scripts); inventory scripts; console claims verification helper (0019).
   - Now automated?: 0019 dry-run-nonroot-user.sh + create-user.sh --test + verify-openbao-client + onboarding-dry-run-claims (infers from groups + T01 role; warns on root/admins); evidence "lldap_identity_verified", "keycape_oidc_claims_verified"; console status.
   - Remaining: Direct LLDAP in paths (not yet via UE claims_enrichment adapter per assessment gap 6); for non-dry-run, still some manual verification. UE adapters missing.

4. **OpenBao claim mapping / policy shape / OIDC admin binding:**
   - Bump: Claims from KeyCape not mapping to expected platform-admin policy; root token or unseal issues post-init.
   - Diagnosis: OpenBao status + token create with policy; manual claim inspection; attended init ceremony.
   - Now automated?: OpenBao OIDC auth configured gate + "OIDC admin login verified"; platform-admin via KeyCape/MFA proven; root token disposition (revoked); console openbao-preflight + status.
   - Remaining: Init/unseal still highly attended (human custody); claim mapping config in KeyCape source (declarative but apply manual). T02 documents current.

5. **Token expiry / revocation / short-lived handling:**
   - Bump: Tokens (OpenBao helper, sessions) expired or leaked; revocation needed without exposing values.
   - Diagnosis: Token lookup/revoke commands (accessor or self); manual in console or kubectl exec.
   - Now automated?: Runbook "OpenBao token revocation" (template + interactive but no plaintext on CLI); console helpers for revoke.
   - Remaining: Interactive for some cases; no fully non-interactive revocation in dry-run paths yet. Gap for T08.

6. **Operator-state persistence / .local metadata drift:**
   - Bump: Flags (e.g. oidc verified, cleanup) out of sync with reality; stage stuck; manual edits risky.
   - Diagnosis: cat .local/security-bootstrap.json; console status/approve/validate flows.
   - Now automated?: Console metadata-template + approve-custody-mode + save_progress_metadata; validate-* targets; .local updated only via console (non-secret); S6 "platform_reopened" + "cleanup_complete" set.
   - Remaining: Still file-based (not in cluster secret or State Hub durable for multi-op); risk of manual tamper. T08 should compute more from validators.

7. **Secret taint / hygiene during user lifecycle (pre-0019):**
   - Bump: Temporary secrets written to sso-mfa/bootstrap/secrets/ for dry-run/test users; not cleaned; plaintext exposure risk.
   - Diagnosis: Manual steps in early T06; inventory + evidence checks.
   - Now automated?: 0019 dry-run-nonroot-user.sh (/tmp WORKSPACE + trap EXIT rm; k8s fallback in create-user.sh never touches persistent bootstrap/secrets for --test); evidence "no_secret_material_recorded":true + validator; make security-bootstrap-onboarding-dry-run + cleanup targets; console subcmd; rm in guide updated conceptually.
   - Remaining: Old manual path in lifecycle_guide T06 section still lists secret-mkdir (minor doc staleness; prefer orchestrator). Good model for other secret handling.

8. **Audit / correlation gaps (ongoing):**
   - Bump: Audit events not correlated across bootstrap (local-identity TSV, OpenBao PVC, State Hub progress, console evidence) vs. UE audit records or flex-auth decisions.
   - Diagnosis: Separate systems; assessment gap 7; contract requires shared IDs (request/actor/decision/user_engine_audit/outbox).
   - Now automated?: Pragmatic layer working (progress events with workstream/task/decision correlation used for 0018 impl tracking itself; console evidence; local audit.py); audit_core posture gate in console (risk accepted).
   - Remaining: No production Audit Core sink (deferred per metadata 2026-07-02); no UE adapters for audit writer/outbox; bootstrap audit separate. T02/T03 document it; T09 risk item.

9. **Other (realm repair, image delivery/KeyCape config, escrow, restore drill, etc.):**
   - Many covered in T02 gates + 0017 evidence (restore drill passed, custody roster, etc.).
   - 0019 added dry-run evidence for user lifecycle (effective preview before save, prevents platform root grant, actor_class checks, reversible lock/offboard).
   - Realm repair, KeyCape delivery still have attended elements (runbooks exist).

## Automation Gap Matrix

| Area | Bump/Fragile Assumption | Current Automation (0017/0019/T02) | Remaining Manual/Fragile | Priority for T05/T08/T09 | Notes / Evidence |
|------|-------------------------|------------------------------------|---------------------------|---------------------------|------------------|
| State persistence | .local metadata drift; stage/flags out of sync | Console approve/validate/metadata flows; S6 flags (platform_reopened, cleanup_complete) set | File-based (tamper risk); not cluster-durable | High (T08 compute from validators) | .local/security-bootstrap.json; console save_progress |
| privacyIDEA realm repair | Realm/resolver drift; MFA self-enroll broken | Runbook + repair script; some validate-t02 | Attended apply; no full declarative gate | Medium | Console "privacyIDEA realm repair" |
| KeyCape image/config delivery + client | Client/redirect mismatch; OIDC login fails | Non-secret client def in source; "client deployed" gate; console verify | Manual apply/restart for drift | High (T08) | keycape/create-secrets.sh; T02 clients |
| OIDC callbacks / bridging | Redirect or client config issues | Documented in T02; console OIDC paths | Manual verification | Medium | kc.coulomb.social + localhost:8250 |
| OpenBao claim mapping / policy | Claims not granting expected policy | OIDC auth configured + admin login verified gates | Init/policy apply attended | Medium | T02 OpenBao OIDC section |
| Token revocation / expiry | Leaked/expired tokens hard to revoke safely | Runbook + console revoke helpers (no plaintext CLI) | Some cases still interactive | Medium (T08) | T02 token flows |
| Audit (pragmatic vs proper) | No correlation; separate systems | Pragmatic: local-identity/audit.py TSV, OpenBao PVC+mock, State Hub/progress/console evidence, audit_core gate (risk accepted) | Production tenant-aware sink + full UE/flex/platform correlation (gap 7) | High (T03/T09) | See T02 "Pragmatic Audit Paths", assessment, metadata audit_core_* |
| Secret taint / hygiene (user lifecycle) | Plaintext in bootstrap/secrets for tests | 0019 orchestrator (/tmp + trap + k8s fallback); evidence "no_secret..."; validate + cleanup make/console | Old manual path lingers in guide docs | Low (mostly done) | dry-run-nonroot-user.sh; 0019 evidence 12 bools |
| User lifecycle (onboard/lock/offboard) | Manual, no preview, no evidence, taint risk | 0019 dry-run + claims + cleanup + console + make + evidence (effective before save, actor checks, reversible) | Transition to UE-backed (adapters) | High (T05/T08 use as model) | T02 + 0019; prevents platform-root grant |
| Restore drill / escrow | No proof of recovery before trust | restore drill passed gate + evidence; custody roster (2of3 planned) | Attended; low-friction upgrade path to escrow | Medium (T09) | 0017 T02 evidence; T02 custody |
| UE integration (adapters, claims_enrichment, memberships, app onboarding, audit correlation) | Direct LLDAP in bootstrap paths; no adapters | Documented in T02 + assessment; 0019 dry-run proves IAM contract | Adapters missing (biggest gap); claims still direct; memberships not synced with owning semantics | High (T03/T09 classify; T07/T08 testbed) | assessment 7 gaps; T02 UE section; boundary contract |
| Consolidated guide + per-step evidence | Operator must rediscover sequence | Piecemeal docs + console lifecycle-guide (T05/T06 flows + 0019 dry-run) + evidence templates | No single "smooth bootstrap guide" with evidence per step + wrong-order hard | High (T05 primary) | T02 feeds it; link validate-* |
| Tests / validations for UI/runbooks | No coverage; sections can regress | Layered plan in T07; 0019 pieces (orchestrator, console cmds, claims, validators) ready | Most sections lack unit/fixture/live tests; live gated | High (T07) | Use T02 doc + 0019 artifacts as fixtures |
| Rebuild risk / rehearsal | Unknown residual human interactions | T02 specific doc + 0019 dry-run model + S6 evidence | Full T03/T05/T07/T08 needed before T09 assessment | High (T09 at end) | Recommend isolated/namespace/scripted first (non-goal: destructive) |

## Recommendations / Next Steps (from this retrospective)

- **T05 priority:** Consolidate into one smooth guide (update operator-journey or new) with explicit evidence per step (link the validate-* and 0019 templates). Update console lifecycle_guide T06 section to prefer orchestrator.
- **T07/T08:** Use 0019 dry-run + new T02 arch doc + evidence as concrete test cases/fixtures. Add validators for realm health, KeyCape client, audit_core posture (already partial), token revocation success, etc. Static tests for runbook presence.
- **T03 complete:** Expand this doc with any new bumps from T05-T08 work. Output the matrix as table in final.
- **T09:** After above, classify risks (esp. UE integration as high, with mitigation via adapters + updated dry-run). Recommend rehearsal strategy (scripted dry + namespace first).
- **Cross:** Feed pragmatic records (this retrospective process itself used progress events + file notes + T02 doc) back into T03. Document current audit in T02 (done).
- **UE:** Per assessment, do not block 0018 on adapters (NK orchestration role), but use T07/T08 + 0019 tooling to prepare integration tests. Create follow-up for UE-side adapter stub if needed.
- Continue pragmatic tracking for remaining T0x (progress + workplan notes).

## References / Inputs

- docs/NetkingdomRuntimeArchitecture.md (T02)
- NET-WP-0017 + 0019 workplans + evidence.json examples
- .local/security-bootstrap.json + console status (S6 + available actions)
- docs/platform-identity-security-architecture.md, responsibility-map.md, security-bootstrap-*.md (operator-journey, openbao-ceremony-ux, user-lifecycle, handover-cleanup, etc.), user-engine-netkingdom-integration-assessment.md, SCOPE.md, platform-root-custody.md
- tools/security-bootstrap-console/security_bootstrap_console.py + Makefile
- sso-mfa/k8s/lldap/dry-run-nonroot-user.sh + related
- State Hub /progress/ events for 0018 (pragmatic record of impl)
- canon/standards/* (iam-profile, user-engine-boundary-contract)
- Assessment gap 7 + contract audit correlation bundle

Update this doc as T03-T09 proceed. It is the "what went wrong / now fixed / still fragile" companion to the runtime architecture doc.