coulomb/net-kingdom
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4a99f8245de9c8620cf1f82d729f20bbd0a2951a
net-kingdom/docs/security-bootstrap-retrospective.md
tegwick 3466c431dd feat(NET-WP-0018-T03): initial retrospective + gap matrix (substantial draft) 
- Created docs/security-bootstrap-retrospective.md:
  - Exec summary (wins: S6, console/0019 automation, evidence discipline, T02 arch doc; gaps: UE adapters, consolidated guide, tests, proper audit)
  - 9 bumps with diagnosis/now-automated?/remaining (realm, OIDC callbacks, LLDAP claims, OpenBao mapping, tokens, operator-state, secret taint pre-0019, audit correlation, etc.)
  - Full gap matrix table (areas, current status incl. 0019/T02, remaining, priorities)
  - Recommendations (T05 consolidate guide, T07/T08 use 0019+T02 as fixtures, T09 classify UE risk, continue pragmatic)
  - Refs to T02 doc, 0017/0019 evidence, console, assessment gap 7, pragmatic records
- Updated workplan T03 with progress note (still in_progress for expansion)
- Pragmatic tracking: progress events (with task), file notes, commit
- Builds directly on T02 + prior 0017/0019 + Coordination pragmatic guidance

Feeds T05/T06/T08/T09. Next tasks can reference this + T02.
2026-06-03 16:31:08 +02:00

16 KiB
Raw Blame History

Security Bootstrap Retrospective and Automation Gap Matrix

Status: initial draft (NET-WP-0018-T03) Date: 2026-06-03 Scope: Captures the first successful NetKingdom bootstrap (through 0015-0017 "Reopen under custody" + 0019 T06-adjacent polish) and automation gaps. Input from T02 runtime architecture doc, prior workplan notes/evidence (0017/0019), console metadata/.local flags, operator experience, assessment, and pragmatic records (progress events, evidence.json, git, console validators).

This feeds T05 (smooth guide with evidence per step), T06/T08 (control surface + validations), T09 (rebuild risk/rehearsal), and future optimization.

Not a full historical dump; focused on bumps, diagnosis, current automation status, and remaining fragile/manual assumptions.

Executive Summary

The first bootstrap proved the stack (LLDAP + Authelia + privacyIDEA + KeyCape + OpenBao + local bootstrap console + State Hub) can deliver a working IAM/security bootstrap path with platform-root custody, MFA, OIDC admin bridging to OpenBao, and non-root user lifecycle.

It also proved the system is easy to derail interactively: realm drift, OIDC callback bridging, LLDAP lookup assumptions, OpenBao claim/policy shape, token expiry/revocation, operator-state persistence (.local metadata), secret taint hygiene, and audit correlation.

Post-0017/0019 state (S6 Reopen under custody):

  • Many gates now durable/non-secret in .local/security-bootstrap.json + console (platform_reopened, cleanup_complete, oidc_login_verified, audit_core_bootstrap_risk_accepted with owner/review 2026-07-02, etc.).
  • Control surface (console + make + web-ui) + evidence templates/validators provide repeatable safe checks and runbooks.
  • 0019 added repeatable non-root dry-run orchestration ( /tmp hygiene + k8s fallback, create --test, verifs, lock/offboard, evidence with 12+ exact bools, validate) + claims helper + cleanup + runbook exposure. Makes T06 gate automatable and less manual/taint-prone.
  • Pragmatic audit (local-identity TSV + OpenBao PVC+mock + State Hub/console evidence) in use; production Audit Core deferred (risk accepted).
  • T02 arch doc now exists as specific-as-deployed baseline (including pragmatic audit paths and UE gaps).

Key wins (now automated or evidenced):

  • S6 reopen + custody approval + MFA self-enroll + KeyCape OIDC admin path to OpenBao.
  • Evidence discipline for cleanup (T03/T04 0017), lifecycle flow (T05), onboarding dry-run (T06 0017/0019) — all validate exact bools + no secrets.
  • Dry-run user lifecycle repeatable/safe (0019 orchestrator + console/make).
  • Non-secret progress in console metadata + State Hub /progress/ (used for tracking 0018 impl itself).

Remaining gaps / fragile assumptions (see matrix):

  • Consolidated smooth bootstrap guide (T05; piecemeal docs + console guide exist but not one operator sequence with per-step evidence).
  • Full automated tests/validations for all UI sections/runbooks (T07; 0019 pieces are ready for coverage; more gates needed in T08).
  • UE integration (adapters, claims_enrichment routing, membership sync, audit correlation, bootstrap-to-governed transition) — biggest per assessment; current paths direct LLDAP/Keycloak (valid for bootstrap per contract but drift risk).
  • Production Audit Core + full correlation (deferred; pragmatic sufficient for now).
  • Scratch rebuild risk rehearsal plan (T09; needs T03/T05/T07/T08 complete).
  • Some interactive repairs still required for drift cases (realm, callbacks, claims); console exposes repair but not fully preventive.

A scratch rebuild should only happen after the guide + validations + T09 say which human interactions are unavoidable.

Bumps Encountered, Diagnosis, and Current Status

(Compiled from 0017/0019 notes, T02 doc, console, metadata, assessment, operator history. Each: bump, root cause/diagnosis, was it automated now?, remaining manual/fragile?)

  1. Realm drift / privacyIDEA repair (early bootstrap):

    • Bump: Realm/resolver not correctly set for LLDAP users; self-enroll or admin MFA broken.
    • Diagnosis: Manual inspection of privacyIDEA admin + LLDAP queries; repair-realm-live.sh or similar.
    • Now automated?: Console has "privacyIDEA realm repair" runbook (template + attended steps); validate-t02 etc. check related. repair script exists.
    • Remaining: Still attended (not fully declarative/CI); needs live cluster + operator. Gap in T05 guide + T08 validator for "realm healthy for bootstrap users".

  2. OIDC callback bridging / KeyCape client registration:

    • Bump: Redirects (localhost:8250 etc.) or client config not matching; login fails for console/OpenBao.
    • Diagnosis: KeyCape client definition in code (create-secrets.sh); apply + restart; verify via console OIDC login.
    • Now automated?: KeyCape OpenBao client definition (non-secret in source) + "KeyCape OpenBao client deployed" gate; console preflight + status checks.
    • Remaining: Manual apply/restart cycle for drift; documented in T02 but no preventive validator yet (T08 target).

  3. LLDAP lookup assumptions / group membership for claims:

    • Bump: Groups (net-kingdom-admins/users) not reflected in OIDC claims or policy; platform-admin not granted or over-granted.
    • Diagnosis: Direct LLDAP queries (GraphQL or scripts); inventory scripts; console claims verification helper (0019).
    • Now automated?: 0019 dry-run-nonroot-user.sh + create-user.sh --test + verify-openbao-client + onboarding-dry-run-claims (infers from groups + T01 role; warns on root/admins); evidence "lldap_identity_verified", "keycape_oidc_claims_verified"; console status.
    • Remaining: Direct LLDAP in paths (not yet via UE claims_enrichment adapter per assessment gap 6); for non-dry-run, still some manual verification. UE adapters missing.

  4. OpenBao claim mapping / policy shape / OIDC admin binding:

    • Bump: Claims from KeyCape not mapping to expected platform-admin policy; root token or unseal issues post-init.
    • Diagnosis: OpenBao status + token create with policy; manual claim inspection; attended init ceremony.
    • Now automated?: OpenBao OIDC auth configured gate + "OIDC admin login verified"; platform-admin via KeyCape/MFA proven; root token disposition (revoked); console openbao-preflight + status.
    • Remaining: Init/unseal still highly attended (human custody); claim mapping config in KeyCape source (declarative but apply manual). T02 documents current.

  5. Token expiry / revocation / short-lived handling:

    • Bump: Tokens (OpenBao helper, sessions) expired or leaked; revocation needed without exposing values.
    • Diagnosis: Token lookup/revoke commands (accessor or self); manual in console or kubectl exec.
    • Now automated?: Runbook "OpenBao token revocation" (template + interactive but no plaintext on CLI); console helpers for revoke.
    • Remaining: Interactive for some cases; no fully non-interactive revocation in dry-run paths yet. Gap for T08.

  6. Operator-state persistence / .local metadata drift:

    • Bump: Flags (e.g. oidc verified, cleanup) out of sync with reality; stage stuck; manual edits risky.
    • Diagnosis: cat .local/security-bootstrap.json; console status/approve/validate flows.
    • Now automated?: Console metadata-template + approve-custody-mode + save_progress_metadata; validate-* targets; .local updated only via console (non-secret); S6 "platform_reopened" + "cleanup_complete" set.
    • Remaining: Still file-based (not in cluster secret or State Hub durable for multi-op); risk of manual tamper. T08 should compute more from validators.

  7. Secret taint / hygiene during user lifecycle (pre-0019):

    • Bump: Temporary secrets written to sso-mfa/bootstrap/secrets/ for dry-run/test users; not cleaned; plaintext exposure risk.
    • Diagnosis: Manual steps in early T06; inventory + evidence checks.
    • Now automated?: 0019 dry-run-nonroot-user.sh (/tmp WORKSPACE + trap EXIT rm; k8s fallback in create-user.sh never touches persistent bootstrap/secrets for --test); evidence "no_secret_material_recorded":true + validator; make security-bootstrap-onboarding-dry-run + cleanup targets; console subcmd; rm in guide updated conceptually.
    • Remaining: Old manual path in lifecycle_guide T06 section still lists secret-mkdir (minor doc staleness; prefer orchestrator). Good model for other secret handling.

  8. Audit / correlation gaps (ongoing):

    • Bump: Audit events not correlated across bootstrap (local-identity TSV, OpenBao PVC, State Hub progress, console evidence) vs. UE audit records or flex-auth decisions.
    • Diagnosis: Separate systems; assessment gap 7; contract requires shared IDs (request/actor/decision/user_engine_audit/outbox).
    • Now automated?: Pragmatic layer working (progress events with workstream/task/decision correlation used for 0018 impl tracking itself; console evidence; local audit.py); audit_core posture gate in console (risk accepted).
    • Remaining: No production Audit Core sink (deferred per metadata 2026-07-02); no UE adapters for audit writer/outbox; bootstrap audit separate. T02/T03 document it; T09 risk item.

  9. Other (realm repair, image delivery/KeyCape config, escrow, restore drill, etc.):

    • Many covered in T02 gates + 0017 evidence (restore drill passed, custody roster, etc.).
    • 0019 added dry-run evidence for user lifecycle (effective preview before save, prevents platform root grant, actor_class checks, reversible lock/offboard).
    • Realm repair, KeyCape delivery still have attended elements (runbooks exist).

Automation Gap Matrix

Area Bump/Fragile Assumption Current Automation (0017/0019/T02) Remaining Manual/Fragile Priority for T05/T08/T09 Notes / Evidence
State persistence .local metadata drift; stage/flags out of sync Console approve/validate/metadata flows; S6 flags (platform_reopened, cleanup_complete) set File-based (tamper risk); not cluster-durable High (T08 compute from validators) .local/security-bootstrap.json; console save_progress
privacyIDEA realm repair Realm/resolver drift; MFA self-enroll broken Runbook + repair script; some validate-t02 Attended apply; no full declarative gate Medium Console "privacyIDEA realm repair"
KeyCape image/config delivery + client Client/redirect mismatch; OIDC login fails Non-secret client def in source; "client deployed" gate; console verify Manual apply/restart for drift High (T08) keycape/create-secrets.sh; T02 clients
OIDC callbacks / bridging Redirect or client config issues Documented in T02; console OIDC paths Manual verification Medium kc.coulomb.social + localhost:8250
OpenBao claim mapping / policy Claims not granting expected policy OIDC auth configured + admin login verified gates Init/policy apply attended Medium T02 OpenBao OIDC section
Token revocation / expiry Leaked/expired tokens hard to revoke safely Runbook + console revoke helpers (no plaintext CLI) Some cases still interactive Medium (T08) T02 token flows
Audit (pragmatic vs proper) No correlation; separate systems Pragmatic: local-identity/audit.py TSV, OpenBao PVC+mock, State Hub/progress/console evidence, audit_core gate (risk accepted) Production tenant-aware sink + full UE/flex/platform correlation (gap 7) High (T03/T09) See T02 "Pragmatic Audit Paths", assessment, metadata audit_core_*
Secret taint / hygiene (user lifecycle) Plaintext in bootstrap/secrets for tests 0019 orchestrator (/tmp + trap + k8s fallback); evidence "no_secret..."; validate + cleanup make/console Old manual path lingers in guide docs Low (mostly done) dry-run-nonroot-user.sh; 0019 evidence 12 bools
User lifecycle (onboard/lock/offboard) Manual, no preview, no evidence, taint risk 0019 dry-run + claims + cleanup + console + make + evidence (effective before save, actor checks, reversible) Transition to UE-backed (adapters) High (T05/T08 use as model) T02 + 0019; prevents platform-root grant
Restore drill / escrow No proof of recovery before trust restore drill passed gate + evidence; custody roster (2of3 planned) Attended; low-friction upgrade path to escrow Medium (T09) 0017 T02 evidence; T02 custody
UE integration (adapters, claims_enrichment, memberships, app onboarding, audit correlation) Direct LLDAP in bootstrap paths; no adapters Documented in T02 + assessment; 0019 dry-run proves IAM contract Adapters missing (biggest gap); claims still direct; memberships not synced with owning semantics High (T03/T09 classify; T07/T08 testbed) assessment 7 gaps; T02 UE section; boundary contract
Consolidated guide + per-step evidence Operator must rediscover sequence Piecemeal docs + console lifecycle-guide (T05/T06 flows + 0019 dry-run) + evidence templates No single "smooth bootstrap guide" with evidence per step + wrong-order hard High (T05 primary) T02 feeds it; link validate-*
Tests / validations for UI/runbooks No coverage; sections can regress Layered plan in T07; 0019 pieces (orchestrator, console cmds, claims, validators) ready Most sections lack unit/fixture/live tests; live gated High (T07) Use T02 doc + 0019 artifacts as fixtures
Rebuild risk / rehearsal Unknown residual human interactions T02 specific doc + 0019 dry-run model + S6 evidence Full T03/T05/T07/T08 needed before T09 assessment High (T09 at end) Recommend isolated/namespace/scripted first (non-goal: destructive)

Recommendations / Next Steps (from this retrospective)

  • T05 priority: Consolidate into one smooth guide (update operator-journey or new) with explicit evidence per step (link the validate-* and 0019 templates). Update console lifecycle_guide T06 section to prefer orchestrator.
  • T07/T08: Use 0019 dry-run + new T02 arch doc + evidence as concrete test cases/fixtures. Add validators for realm health, KeyCape client, audit_core posture (already partial), token revocation success, etc. Static tests for runbook presence.
  • T03 complete: Expand this doc with any new bumps from T05-T08 work. Output the matrix as table in final.
  • T09: After above, classify risks (esp. UE integration as high, with mitigation via adapters + updated dry-run). Recommend rehearsal strategy (scripted dry + namespace first).
  • Cross: Feed pragmatic records (this retrospective process itself used progress events + file notes + T02 doc) back into T03. Document current audit in T02 (done).
  • UE: Per assessment, do not block 0018 on adapters (NK orchestration role), but use T07/T08 + 0019 tooling to prepare integration tests. Create follow-up for UE-side adapter stub if needed.
  • Continue pragmatic tracking for remaining T0x (progress + workplan notes).

References / Inputs

  • docs/NetkingdomRuntimeArchitecture.md (T02)
  • NET-WP-0017 + 0019 workplans + evidence.json examples
  • .local/security-bootstrap.json + console status (S6 + available actions)
  • docs/platform-identity-security-architecture.md, responsibility-map.md, security-bootstrap-*.md (operator-journey, openbao-ceremony-ux, user-lifecycle, handover-cleanup, etc.), user-engine-netkingdom-integration-assessment.md, SCOPE.md, platform-root-custody.md
  • tools/security-bootstrap-console/security_bootstrap_console.py + Makefile
  • sso-mfa/k8s/lldap/dry-run-nonroot-user.sh + related
  • State Hub /progress/ events for 0018 (pragmatic record of impl)
  • canon/standards/* (iam-profile, user-engine-boundary-contract)
  • Assessment gap 7 + contract audit correlation bundle

Update this doc as T03-T09 proceed. It is the "what went wrong / now fixed / still fragile" companion to the runtime architecture doc.

Reference in New Issue View Git Blame Copy Permalink