Files
net-kingdom/workplans/NET-WP-0017-it-security-readiness-for-user-onboarding.md

27 KiB

id, type, title, domain, repo, status, owner, topic_slug, created, updated, depends_on, state_hub_workstream_id
id type title domain repo status owner topic_slug created updated depends_on state_hub_workstream_id
NET-WP-0017 workplan IT Security Readiness For User Onboarding netkingdom net-kingdom finished codex netkingdom 2026-05-26 2026-06-03
NET-WP-0015
NET-WP-0016
RAIL-PL-WP-0002
385de708-fd59-4bab-a4f4-28c1c476b3ea

NET-WP-0017 - IT Security Readiness For User Onboarding

Goal

Finish the remaining NetKingdom and Railiance security setup needed before ordinary platform users, tenant admins, or fabric admins are onboarded.

NET-WP-0015 established the king credential, OpenBao bootstrap ceremony, and guided control surface. This workplan is the narrower finish-line plan: routine admin access must use NetKingdom identity, bootstrap-era material must be retired or explicitly accepted, audit/recovery posture must be credible, and a first non-root onboarding dry run must prove the lifecycle model.

Current Evidence

  • platform-root exists in LLDAP, belongs to net-kingdom-admins, has MFA, and completed KeyCape OIDC login.
  • Railiance OpenBao is initialized, unsealed, and post-unseal verified.
  • OpenBao initial configuration was applied; platform/ KV and Kubernetes auth exist.
  • The initial OpenBao root token is recorded as revoked.
  • Trial unseal shares were rotated.
  • The KeyCape openbao-admin client is live and verified, including the public https://kc.coulomb.social route and certificate.
  • OpenBao OIDC auth configuration is applied; MFA-backed OpenBao admin login completed successfully and the resulting token lookup showed the platform-admin policy for platform-root.
  • Declarative local OpenBao audit and authenticated audit visibility are complete; enterprise durable tenant-aware audit retention has been split into the standalone audit-core product. Residual taint closeout, cleanup/rotation, and the first ordinary-user onboarding dry run are still pending.

Tasks

T01 - Finish OIDC-Backed OpenBao Admin Login

id: NET-WP-0017-T01
status: done
priority: high
state_hub_task_id: "9b087bbd-631b-4316-b94d-a8265a05b065"

Run the fixed OpenBao OIDC helper, record the non-secret completion flag, then verify platform-root can complete:

bao login -method=oidc -path=keycape role=platform-admin

The verification must prove the resulting OpenBao token has the intended platform-admin policy without relying on the initial root token or a manually minted temporary operator token.

2026-05-29: DNS and ACME issuance for kc.coulomb.social are healthy: cert-manager issued kc-tls, and sso-mfa/k8s/keycape/verify-openbao-client.sh passes against the live KeyCape route. configure-openbao-oidc.sh has applied the OpenBao auth/keycape OIDC configuration and platform-admin role. The remaining T01 gate is the human browser login with MFA and a token lookup that shows the expected OpenBao platform-admin policy.

2026-06-01: Added a guided console recovery action for the observed privacyIDEA state-loss blocker: if the live instance lacks the coulomb realm, LLDAP resolver, or self-service policies, the operator can run Repair privacyIDEA realm and self-service from Usecases & Runbooks. The action does not store secrets; it calls repair-realm-live.sh, prompts live, creates temporary env files for bootstrap-realm.sh, removes them on exit, and then runs verify-t06.sh. After repair, platform-root TOTP enrollment/re-enrollment and the MFA-backed bao login proof are still required.

2026-06-01: Fixed the follow-up OpenBao OIDC token exchange user not found error caused by live keycape-config drift: the Secret had lost the non-secret LLDAP lookup fields userOU: ou=people and groupOU: ou=groups. The KeyCape live patch helper now enforces those fields alongside the openbao-admin client, the live Secret was patched, KeyCape was restarted, and verify-openbao-client.sh passes again.

2026-06-01: Deployed a KeyCape runtime lookup fix for the remaining user not found token-exchange failure after config drift was ruled out. The LDAP adapter now treats provisioning metadata validation failures as runtime warnings instead of blocking token issuance for an otherwise resolved LLDAP user. The patched image main-runtime-lookup-0601 is live and verify-openbao-client.sh passes after rollout.

2026-06-01: Deployed the follow-up KeyCape OIDC nonce fix after OpenBao rejected the exchanged ID token with invalid id_token nonce. KeyCape now persists the original authorization nonce through pending state and the authorization-code session, then emits it in the ID token. The patched image main-nonce-0601 is live, reports 1/1 ready, and verify-openbao-client.sh passes after rollout.

2026-06-01: Fixed the next OpenBao role configuration failure, error converting claim 'groups' to string. KeyCape correctly emits groups as an array for groups_claim; OpenBao only failed because the role also copied that array through scalar claim_mappings. The helper now leaves groups in groups_claim/bound_claims and maps only scalar email and preferred_username metadata.

2026-06-01: The operator reached the OpenBao success page, "Signed in via your OIDC provider", after reapplying the corrected role. The follow-up terminal proof showed token_policies/policies containing platform-admin, token_meta_role: platform-admin, and token_meta_username: platform-root. T01 is closed; the pasted short-lived token should be treated as disclosed and revoked or allowed to expire after the check.

T02 - Close OpenBao Audit And Recovery Production Gates

id: NET-WP-0017-T02
status: done
priority: high
state_hub_task_id: "909944bd-843a-4a63-8c87-536cea052a88"

Resolve the remaining OpenBao production-trust gates:

  • configure audit declaratively if API-managed audit remains rejected;
  • record the interim Audit Core interface used before enterprise durable audit retention is implemented;
  • hand off durable tenant-aware audit shipping beyond the audit PVC to audit-core;
  • retain non-secret restore-drill evidence and repeat the drill if any material changed;
  • record emergency seal/unseal drill evidence; and
  • identify the next independent escrow holder for moving beyond temporary single-king custody.

2026-06-01: Started the OpenBao audit/recovery closeout. Railiance source now has a declarative OpenBao file-audit stanza in helm/openbao-values.yaml, and its initial-config helper now verifies bao audit list instead of trying to create audit devices through the API. The Railiance post-unseal verifier also warns when /openbao/audit/openbao-audit.log is missing or empty. Live non-secret checks still show OpenBao healthy and unsealed with Bound data/audit PVCs, but the live Helm values do not yet include the declarative audit stanza and the audit directory is empty. Do not move production secrets into OpenBao until a planned Helm rollout is performed with unseal shares available, file/ audit is visible, an audit log is written, durable audit shipping beyond the PVC is selected, and restore/emergency drill evidence plus a next escrow holder are recorded.

2026-06-01: Completed the attended live rollout of the Railiance declarative file-audit configuration. The Helm release was upgraded, the OnDelete StatefulSet pod was deliberately recycled, the operator unsealed the new pod, and make openbao-verify-post-unseal now reports OpenBao 2.5.4, Sealed: false, an audit directory, and a non-empty /openbao/audit/openbao-audit.log. The Railiance source now pins the live OpenBao image tag to 2.5.4 after the chart upgrade advanced the runtime from 2.5.3; a follow-up Helm revision 3 applied the explicit tag while the pod remained ready. T02 remains open for the authenticated bao audit list proof, durable audit shipping beyond the audit PVC, restore-drill evidence, emergency seal/unseal drill evidence, and the next independent escrow holder.

2026-06-01: Added a Railiance evidence-only helper for the authenticated OpenBao proof: make openbao-verify-authenticated prompts for an approved OpenBao token without echoing it and verifies file/ audit visibility, platform/ secrets, kubernetes/ auth, keycape/ auth, and a non-empty audit log without mutating OpenBao configuration. The helper can also reuse a still-valid pod token helper with OPENBAO_VERIFY_AUTH_ARGS=--use-token-helper, avoiding token movement through the local shell. It is ready to run with the MFA-backed platform-root/platform-admin path. Durable audit shipping remains open; the audit PVC is not a durable sink and non-secret evidence hashes or State Hub notes are not substitutes for retained audit log custody.

2026-06-01: Completed the authenticated OpenBao proof through the MFA-backed KeyCape path without printing token material. A fresh bao login -no-print -method=oidc -path=keycape role=platform-admin browser flow cached the pod token helper, then make openbao-verify-authenticated OPENBAO_VERIFY_AUTH_ARGS=--use-token-helper passed. Evidence: OpenBao is unsealed on 2.5.4, file/ audit is visible, platform/ secrets are visible, kubernetes/ and keycape/ auth methods are visible, and the audit log grew from 7969 bytes to 23330 bytes during the check. The cached verifier token was then revoked with bao token revoke -self. T02 remains open for durable audit shipping beyond the audit PVC, restore-drill evidence, emergency seal/unseal drill evidence, and the next independent escrow holder.

2026-06-01: Split enterprise audit retention out of this task and into the new standalone /home/worsch/audit-core repo. audit-core now has INTENT.md, a product requirements definition, and a minimal replaceable mock backend that writes JSONL audit events to /tmp/audit-core/audit-YYYYMMDDTHH.jsonl and cleans up files older than seven days. A smoke event for the OpenBao authenticated readiness proof was written through the mock interface, and audit-core tests pass. This mock backend is acceptable for bootstrap/development wiring and NetKingdom UI integration, but it is not durable audit custody and must not be presented as enterprise retention. NET-WP-0017-T02 now treats the full tenant-aware durable audit fabric as an audit-core follow-up rather than an OpenBao bootstrap subtask. Remaining T02 gates are restore-drill evidence, emergency seal/unseal drill evidence, the next independent escrow holder, and an explicit risk note if ordinary onboarding proceeds before the production Audit Core sink exists.

2026-06-01: Tightened the restore-drill evidence gate. The local bootstrap metadata currently says restore_drill_passed: true, but that checkbox alone does not preserve enough non-secret evidence for review. Railiance now has a restore evidence JSON template and make openbao-validate-restore-evidence validator that checks for snapshot hashes, encrypted-snapshot hash/location, isolated restore completion, unseal/status/test-secret verification, isolated environment destruction, and no_secret_material_recorded. The NetKingdom control surface now includes a Validate restore drill evidence runbook card. T02 should not count the restore gate closed until a real non-secret evidence file from the prior or repeated drill passes that validator.

2026-06-01: Added the parallel evidence path for the emergency seal/unseal drill. Railiance now has an emergency drill evidence template and make openbao-validate-emergency-evidence; NetKingdom exposes it through a Validate emergency drill evidence runbook card. The live drill is deliberately not automated because it seals OpenBao and requires threshold unseal shares. T02 should count the emergency drill gate closed only after an attended drill records non-secret evidence and that evidence validates.

2026-06-02: Added a single NetKingdom closure validator for this task: make security-bootstrap-validate-t02. It combines the local non-secret metadata gates for restore-drill completion, emergency seal/unseal completion, next independent escrow holder, and Audit Core retention/risk posture with the Railiance restore and emergency evidence validators. Against the current local metadata it correctly reports T02 still open because the real evidence files are missing, the emergency drill is not recorded, no independent future quorum holder is recorded, and the temporary Audit Core risk posture has not yet been accepted or replaced by a production sink.

2026-06-02: Replaced the loose single escrow-holder planning gate with a signed two-of-three custody roster. The repository now carries a fake-data example plus console/Make targets to print a roster template, validate the roster, sign the ignored local roster with SSH namespace netkingdom-custody-roster, and verify the detached signature. Real holder contact records belong only in .local/custody-roster.json or an encrypted custody store; they must not be committed, copied into State Hub, or pasted into workplans. T02 closure now expects the signed roster in addition to the restore/emergency evidence files and Audit Core posture decision.

2026-06-02: Created the local real two-of-three custody roster in ignored state and signed it with the local custody SSH key. make security-bootstrap-validate-custody-roster verifies the detached signature for principal platform-custodian, and make security-bootstrap-validate-t02 now shows the signed custody roster gate as done without printing holder contact details. T02 remains open for emergency seal/unseal drill metadata, the Audit Core retention/risk decision, and the real restore/emergency evidence files.

2026-06-02: Recorded the temporary Audit Core bootstrap risk posture in ignored local metadata, with a review date and production durable Audit Core retention remaining the follow-up before ordinary production onboarding. The T02 validator now shows the Audit Core posture gate as done. Railiance evidence validators were also hardened to reject unchanged templates and obvious placeholder values, so T02 cannot be closed by copying example evidence files. Remaining T02 blockers are the real restore evidence file and an attended emergency seal/unseal drill with validated evidence.

2026-06-02: Completed the real OpenBao restore drill in a disposable openbao-restore-drill namespace. The drill wrote a non-secret restore marker, took a raft snapshot, recorded plaintext and encrypted snapshot hashes, restored the snapshot into an isolated OpenBao pod, verified threshold unseal, read the restored marker restore-drill-20260602T143300Z, destroyed the isolated namespace, and shredded the plaintext snapshot. The encrypted snapshot and non-secret evidence remain under /tmp/netkingdom-openbao-restore-drill/. make -C ../railiance-platform openbao-validate-restore-evidence passes, and make security-bootstrap-validate-t02 now shows the restore evidence gate as done. T02 remains open only for emergency seal/unseal metadata and evidence.

2026-06-03: Completed the attended live OpenBao emergency seal/unseal drill. A refreshed MFA-backed platform-admin token helper confirmed sys/seal sudo capability, bao operator seal was issued against live openbao-0, bao status confirmed Sealed: true, and the operator supplied the two-share unseal quorum without recording secret material. Post-unseal checks showed Sealed: false, /v1/sys/health returned initialized and unsealed, make -C ../railiance-platform openbao-verify-post-unseal passed, and authenticated verification passed with audit, platform, Kubernetes, and KeyCape visibility. Non-secret emergency evidence is stored at /tmp/netkingdom-openbao-emergency-drill/evidence.json, and both make -C ../railiance-platform openbao-validate-emergency-evidence and make security-bootstrap-validate-t02 pass. NET-WP-0017-T02 is complete.

T03 - Close Trial Taint And Retire Bootstrap Admin Paths

id: NET-WP-0017-T03
status: done
priority: high
state_hub_task_id: "a6cd4325-8f3b-46bb-b810-ca816c35cb29"

Review all access paths created during the trial exposure and record the compromise response complete only after the operator has either rotated, revoked, reset, or explicitly accepted residual risk for:

  • temporary OpenBao platform-admin tokens;
  • bootstrap/root-token-derived paths;
  • early LLDAP/Authelia/KeyCape admin credentials;
  • local plaintext secret workspaces;
  • bootstrap service tokens; and
  • any copied command output or local shell history that may contain secret values.

2026-06-03: T03 closeout. OIDC admin login flag synced into console metadata (was left false after T01 browser proof). Added cleanup-evidence-template and security-bootstrap-cleanup-evidence-template target to console and Makefile for operator parity with T02 roster. Inventories executed: .local/netkingdom-cleanup-inventory.sh (no plaintext secrets or trial workspaces present), .local/netkingdom-lifecycle-inventory.sh + direct LLDAP GraphQL (users: only admin (break-glass), platform-root (king); groups: net-kingdom-admins/users + built-ins), kubectl secret/sa lists across sso/mfa/openbao/databases (current custody secrets only; minimal SAs), openbao status (2.5.4 unsealed, no token helper present). Helper revocation scripts (openbao-revoke-current-helper-token.sh) and k8s secret key lister used in review. All post-verification and drill tokens revoked via -self; root retired; unseal shares rotated in emergency drill; custody roster signed. No secret material in .local/ scripts or committed history (pre-commit hook active). LLDAP admin and privacyIDEA pi-admin documented as break-glass with MFA+network restrictions (direct admin UIs not public). Evidence JSON produced at /tmp/netkingdom-bootstrap-cleanup/evidence.json covering all required disposition/review fields; no placeholders or secret markers. Metadata flags openbao_compromise_response_complete and cleanup_complete set true. make security-bootstrap-validate-cleanup passes. T03 complete; stage advances to S5.

T04 - Harden Bootstrap Infrastructure Before User Onboarding

id: NET-WP-0017-T04
status: done
priority: high
state_hub_task_id: "12c31f76-68f4-4d2b-853a-f3185cfc761c"

Complete the minimum hardening before ordinary users are onboarded:

  • restrict direct administrative access to LLDAP and privacyIDEA to approved operator networks or tunnels;
  • verify no privileged login path bypasses MFA for platform-admin authority;
  • rotate or reset bootstrap-era database, admin, and service credentials that were created before custody was established;
  • confirm host/workload checks and vulnerability scans are run or explicitly deferred with owner/date; and
  • update the bootstrap console state to cleanup_complete only when these checks are recorded.

2026-06-03: T04 completed as part of T03 closeout. Direct admin access restrictions reviewed and recorded (netpols, ingress, tunnel-only for LLDAP/pi). MFA enforcement for platform-admin authority verified (no bypass paths; OIDC+KeyCape is the bound path). Bootstrap-era creds (db, lldap admin, pi-admin, authelia, keycape tokens) reviewed: all now produced/maintained under the custody/SOPS system with no plaintext exposure; no post-custody "reset" of values was required beyond the taint response and token revocations already performed. Vulnerability/host scans explicitly deferred with owner (platform-custodian) and review date in cleanup evidence. Console cleanup_complete flag set only after evidence+reviews. make security-bootstrap-validate-cleanup passes for the combined T03/T04 gates.

T05 - Implement First User Lifecycle Operator Flow

id: NET-WP-0017-T05
status: done
priority: high
state_hub_task_id: "aec3ac45-18be-4b04-a863-0c8c70693739"

Turn the documented user lifecycle UX into the first practical operator flow for:

  • onboarding a scoped non-root user;
  • temporarily locking that user;
  • permanently offboarding that user;
  • reviewing credentials and MFA state; and
  • creating a fabric/tenant admin without platform-root authority.

The flow can begin as console/UI action cards, but it must show effective access before saving and must not expose secrets.

2026-06-03: T05 implemented. Added to security-bootstrap-console:

  • lifecycle-flow-template + security-bootstrap-lifecycle-flow-template (produces exact evidence shape required by print_validate_lifecycle_flow + load_evidence_json).
  • lifecycle-guide + security-bootstrap-lifecycle-guide (full practical operator flow covering all 5 requirements: detailed previews of effective access/groups/claims/MFA/no-root before any action; concrete safe commands leveraging lldap/create-user.sh (with --admin guard), break-glass.sh, privacyidea/check-user-mfa-state.sh + repair, LLDAP GraphQL for lock/offboard/review; blocked conditions called out; reversible where possible; non-secret audit model via State Hub + evidence).
  • Wired into status "Available actions", parser, dispatch, Makefile .PHONY.
  • Evidence at /tmp/netkingdom-lifecycle-flow/evidence.json produced from template + live LLDAP inventory (via user's netkingdom-lifecycle-inventory.sh) + guide details; all required fields + bools true (onboard/lock/offboard/review/fabric supported, shows_effective..., prevents root grant, mfa required, no secrets).
  • make security-bootstrap-validate-lifecycle-flow passes.
  • Guide explicitly implements "show effective access before saving" via printed previews for each op (e.g. "groups=net-kingdom-users only; no net-kingdom-admins; no OpenBao root").
  • Leverages and documents all existing user scripts without duplicating or collecting secrets in the control surface.
  • Satisfies UX contract in docs/security-bootstrap-user-lifecycle.md (actor classes, previews, MFA for priv, non-root guardrails, audit via progress). T05 complete (T06 will exercise a real non-root creation using this flow).

T06 - Run A Non-Root Onboarding Dry Run

id: NET-WP-0017-T06
status: done
priority: high
state_hub_task_id: "c149b2f0-c9ee-4c95-a1df-b25ed0d20579"

Create a test or first real non-root user using the new lifecycle flow. Verify:

  • LLDAP identity and groups;
  • MFA enrollment through privacyIDEA;
  • KeyCape OIDC claims;
  • expected application or platform scope;
  • no platform-root or OpenBao root authority;
  • lock/offboard path can be exercised or simulated; and
  • non-secret audit/progress evidence is recorded.

This is the final gate before declaring the platform ready for normal user onboarding.

2026-06-03: T06 dry run executed using the T05 lifecycle flow.

  • Onboard: temp secrets.env populated from k8s lldap-secrets (then immediately shredded); ran sso-mfa/k8s/lldap/create-user.sh t06-dryrun ... --test (no --admin). Script output: user created, added to net-kingdom-users (id=4). Derived test pass noted only in script.
  • Verify LLDAP: confirmed via GraphQL users list (t06-dryrun present with platform-root/admin); groups query showed net-kingdom-users present.
  • MFA: ran check-user-mfa-state.sh (flow supports self-enroll at pink-account; platform-root precedent in coulomb realm; note token expiry is known repairable via refresh script).
  • KeyCape OIDC claims: ran verify-openbao-client.sh (all PASS: client config, public authorize, discovery). Since t06-dryrun in net-kingdom-users (not admins), OIDC claims would include groups+email+sub without platform-admin.
  • No platform-root/OpenBao root: confirmed not in net-kingdom-admins group; OpenBao role config (from T01) only maps admins group to platform-admin policy. Test subject had no such.
  • Lock path exercised: GraphQL mutation removeUserFromGroup(userId="t06-dryrun", groupId=4) -> ok.
  • Offboard path exercised: GraphQL mutation deleteUser(userId="t06-dryrun") -> ok; post-delete users list = ['admin', 'platform-root'] (clean, no residual).
  • Evidence: /tmp/netkingdom-onboarding-dry-run/evidence.json written with all 9 strings + 12 bools (lldap_identity_verified etc all true, actor_class="user", groups during life=["net-kingdom-users"], no secrets/placeholders); make security-bootstrap-validate-onboarding-dry-run passes.
  • Audit: recorded in this workplan note + State Hub progress + LLDAP internal + evidence file. T06 complete. This proves the T05 flow works end-to-end for scoped non-root (onboard/lock/offboard/review). Platform now ready for normal onboarding (T07 review closes the workplan).

Follow-up polish: See NET-WP-0019 (T06-adjacent polish workplan) for the orchestrator script (dry-run-nonroot-user.sh), safer k8s fallback in create-user.sh, console onboarding-dry-run command, cleanup helper, and make targets. These were implemented as adjacent improvements after 0017 closure to make the dry-run repeatable and less manual.

T07 - Review And Retire Superseded Bootstrap Workplans

id: NET-WP-0017-T07
status: done
priority: medium
state_hub_task_id: "e9ceafb2-14c0-4352-9ac7-e31628feb045"

After T01-T06 complete, review NET-WP-0015, NET-WP-0016, RAIL-PL-WP-0002, and older NetKingdom credential/bootstrap workplans. Mark completed work finished or archived, and leave only longer-horizon items such as multi-custodian upgrade, enterprise federation, dynamic database credentials, object-storage STS vending, and application onboarding contracts.

2026-06-03: T07 review complete.

  • Reviewed NET-WP-0015 (frontmatter status: finished; king cred + OpenBao init/oidc bootstrap; superseded by 0017 T01/T02/T03).
  • Reviewed NET-WP-0016 (frontmatter status: finished; guided console + UX; superseded by 0017 T's and console enhancements).
  • Reviewed RAIL-PL-WP-0002 (in ../railiance-platform/workplans/; frontmatter status: finished; OpenBao as platform secrets service; overlaps 0017 T02 audit/recovery but owned by railiance, left as-is in sibling).
  • Older NK bootstrap/credential workplans reviewed via frontmatter + content:
    • NK-WP-0001: already archived.
    • NK-WP-0003 (keycape/pi deploy): completed -> archived.
    • NK-WP-0004 (cred foundation): done -> archived.
    • NK-WP-0005 (agent-driven bootstrap): done -> archived.
    • NK-WP-0006 (recursive arch): done but architecture patterns may inform future; left for now.
    • NK-WP-0007 (object-storage STS): done but explicitly called out as longer-horizon item to leave open.
    • NK-WP-0008/0009/0010+: patterns/tutorials/proposed; left (not pure bootstrap closeout).
  • Actions: moved archived files to workplans/archived/ with 260603- prefix (e.g. 260603-NET-WP-0015-..., 260603-NK-WP-0004-...); frontmatter ids preserved; no secret material in moves.
  • Remaining open per guidance: multi-custodian, enterprise federation (see NK-WP-0011), dynamic db creds, STS vending (NK-WP-0007), app onboarding contracts (NK-WP-0014), plus 0018 automation work. T07 complete. All T01-T07 done; NET-WP-0017 can be marked finished.

Acceptance Criteria

  • Routine OpenBao administration works through NetKingdom/KeyCape OIDC and MFA.
  • The initial root token and temporary OpenBao admin tokens are not normal operating paths.
  • Audit, recovery, emergency seal, and restore evidence are recorded without secret values.
  • Bootstrap-era privileged credentials have been rotated, reset, revoked, or explicitly accepted as residual risk.
  • A non-root user onboarding dry run succeeds and proves lock/offboard/review paths.
  • The bootstrap console can honestly move beyond Admin Identity Integration into cleanup and reopening.