Add OpenBao emergency drill evidence validator
This commit is contained in:
7
Makefile
7
Makefile
@@ -15,6 +15,7 @@ OPENBAO_RELEASE ?= openbao
|
|||||||
OPENBAO_VALUES ?= helm/openbao-values.yaml
|
OPENBAO_VALUES ?= helm/openbao-values.yaml
|
||||||
OPENBAO_VERIFY_AUTH_ARGS ?=
|
OPENBAO_VERIFY_AUTH_ARGS ?=
|
||||||
OPENBAO_RESTORE_EVIDENCE ?= /tmp/netkingdom-openbao-restore-drill/evidence.json
|
OPENBAO_RESTORE_EVIDENCE ?= /tmp/netkingdom-openbao-restore-drill/evidence.json
|
||||||
|
OPENBAO_EMERGENCY_EVIDENCE ?= /tmp/netkingdom-openbao-emergency-drill/evidence.json
|
||||||
|
|
||||||
##@ CloudNative PG (cnpg) — primary database operator
|
##@ CloudNative PG (cnpg) — primary database operator
|
||||||
|
|
||||||
@@ -131,6 +132,10 @@ openbao-validate-restore-evidence: ## Validate non-secret OpenBao restore-drill
|
|||||||
OPENBAO_RESTORE_EVIDENCE='$(OPENBAO_RESTORE_EVIDENCE)' \
|
OPENBAO_RESTORE_EVIDENCE='$(OPENBAO_RESTORE_EVIDENCE)' \
|
||||||
scripts/openbao-validate-restore-evidence.sh
|
scripts/openbao-validate-restore-evidence.sh
|
||||||
|
|
||||||
|
openbao-validate-emergency-evidence: ## Validate non-secret OpenBao emergency seal/unseal drill evidence JSON
|
||||||
|
OPENBAO_EMERGENCY_EVIDENCE='$(OPENBAO_EMERGENCY_EVIDENCE)' \
|
||||||
|
scripts/openbao-validate-emergency-drill-evidence.sh
|
||||||
|
|
||||||
##@ Backup
|
##@ Backup
|
||||||
|
|
||||||
backup: ## Backup platform services (PostgreSQL logical dump) — age-encrypted to Nextcloud
|
backup: ## Backup platform services (PostgreSQL logical dump) — age-encrypted to Nextcloud
|
||||||
@@ -143,4 +148,4 @@ help: ## Show this help
|
|||||||
/^[a-zA-Z_-]+:.*?##/ { printf " \033[36m%-22s\033[0m %s\n", $$1, $$2 } \
|
/^[a-zA-Z_-]+:.*?##/ { printf " \033[36m%-22s\033[0m %s\n", $$1, $$2 } \
|
||||||
/^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) }' $(MAKEFILE_LIST)
|
/^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) }' $(MAKEFILE_LIST)
|
||||||
|
|
||||||
.PHONY: db-deploy db-status db-shell db-logs apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-verify-authenticated openbao-validate-restore-evidence backup help
|
.PHONY: db-deploy db-status db-shell db-logs apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-verify-authenticated openbao-validate-restore-evidence openbao-validate-emergency-evidence backup help
|
||||||
|
|||||||
21
docs/openbao-emergency-drill-evidence.example.json
Normal file
21
docs/openbao-emergency-drill-evidence.example.json
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
{
|
||||||
|
"drill_date": "2026-06-01",
|
||||||
|
"operator": "platform-root",
|
||||||
|
"source_cluster": "railiance01",
|
||||||
|
"source_namespace": "openbao",
|
||||||
|
"source_pod": "openbao-0",
|
||||||
|
"seal_started_at": "2026-06-01T22:00:00Z",
|
||||||
|
"seal_command_issued": true,
|
||||||
|
"sealed_status_confirmed": true,
|
||||||
|
"sealed_status_observed_at": "2026-06-01T22:00:30Z",
|
||||||
|
"unseal_quorum_available": true,
|
||||||
|
"unseal_started_at": "2026-06-01T22:01:00Z",
|
||||||
|
"unseal_completed": true,
|
||||||
|
"unseal_completed_at": "2026-06-01T22:02:30Z",
|
||||||
|
"post_unseal_status_verified": true,
|
||||||
|
"post_unseal_readiness_verified": true,
|
||||||
|
"post_unseal_verification": "bao status reported Sealed false and make openbao-verify-post-unseal passed after the drill",
|
||||||
|
"availability_window_minutes": 3,
|
||||||
|
"no_secret_material_recorded": true,
|
||||||
|
"notes": "Do not record OpenBao tokens, root tokens, unseal shares, private keys, passwords, OTP seeds, or recovery codes."
|
||||||
|
}
|
||||||
@@ -322,6 +322,16 @@ Audit Core backend that writes JSONL records under
|
|||||||
days. Use it only to wire interfaces and setup validation before the durable
|
days. Use it only to wire interfaces and setup validation before the durable
|
||||||
Audit Core archive exists.
|
Audit Core archive exists.
|
||||||
|
|
||||||
|
Emergency seal/unseal drills are disruptive and must only run in an attended
|
||||||
|
window with threshold unseal shares available. Record non-secret drill evidence
|
||||||
|
using `docs/openbao-emergency-drill-evidence.example.json` as a template, then
|
||||||
|
validate it with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make openbao-validate-emergency-evidence \
|
||||||
|
OPENBAO_EMERGENCY_EVIDENCE=/path/to/evidence.json
|
||||||
|
```
|
||||||
|
|
||||||
Monitoring baseline:
|
Monitoring baseline:
|
||||||
|
|
||||||
- pod readiness and liveness from Kubernetes probes
|
- pod readiness and liveness from Kubernetes probes
|
||||||
|
|||||||
107
scripts/openbao-validate-emergency-drill-evidence.sh
Executable file
107
scripts/openbao-validate-emergency-drill-evidence.sh
Executable file
@@ -0,0 +1,107 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
EVIDENCE="${OPENBAO_EMERGENCY_EVIDENCE:-${1:-/tmp/netkingdom-openbao-emergency-drill/evidence.json}}"
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<'USAGE'
|
||||||
|
Usage: scripts/openbao-validate-emergency-drill-evidence.sh [evidence.json]
|
||||||
|
|
||||||
|
Validates non-secret OpenBao emergency seal/unseal drill evidence. The evidence
|
||||||
|
file should record timing, status checks, verification flags, and operator
|
||||||
|
notes, but must not contain OpenBao tokens, unseal shares, root tokens, private
|
||||||
|
keys, passwords, OTP seeds, or recovery codes.
|
||||||
|
|
||||||
|
Environment:
|
||||||
|
OPENBAO_EMERGENCY_EVIDENCE Evidence JSON path. Default:
|
||||||
|
/tmp/netkingdom-openbao-emergency-drill/evidence.json
|
||||||
|
USAGE
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; then
|
||||||
|
usage
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 - "$EVIDENCE" <<'PY'
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
path = Path(sys.argv[1])
|
||||||
|
if not path.exists():
|
||||||
|
print(f"[FAIL] emergency drill evidence file is missing: {path}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
try:
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
print(f"[FAIL] emergency drill evidence is not valid JSON: {exc}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
print("[FAIL] emergency drill evidence root must be a JSON object", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
required_strings = [
|
||||||
|
"drill_date",
|
||||||
|
"operator",
|
||||||
|
"source_cluster",
|
||||||
|
"source_namespace",
|
||||||
|
"source_pod",
|
||||||
|
"seal_started_at",
|
||||||
|
"sealed_status_observed_at",
|
||||||
|
"unseal_started_at",
|
||||||
|
"unseal_completed_at",
|
||||||
|
"post_unseal_verification",
|
||||||
|
]
|
||||||
|
required_true = [
|
||||||
|
"seal_command_issued",
|
||||||
|
"sealed_status_confirmed",
|
||||||
|
"unseal_quorum_available",
|
||||||
|
"unseal_completed",
|
||||||
|
"post_unseal_status_verified",
|
||||||
|
"post_unseal_readiness_verified",
|
||||||
|
"no_secret_material_recorded",
|
||||||
|
]
|
||||||
|
|
||||||
|
errors: list[str] = []
|
||||||
|
for key in required_strings:
|
||||||
|
value = data.get(key)
|
||||||
|
if not isinstance(value, str) or not value.strip():
|
||||||
|
errors.append(f"missing non-empty string: {key}")
|
||||||
|
|
||||||
|
for key in required_true:
|
||||||
|
if data.get(key) is not True:
|
||||||
|
errors.append(f"must be true: {key}")
|
||||||
|
|
||||||
|
window = data.get("availability_window_minutes")
|
||||||
|
if not isinstance(window, int) or window < 0:
|
||||||
|
errors.append("availability_window_minutes must be a non-negative integer")
|
||||||
|
|
||||||
|
encoded = json.dumps(data, sort_keys=True)
|
||||||
|
secret_markers = [
|
||||||
|
"OPENBAO_ROOT_TOKEN",
|
||||||
|
"VAULT_TOKEN",
|
||||||
|
"BEGIN PRIVATE KEY",
|
||||||
|
"BEGIN OPENSSH PRIVATE KEY",
|
||||||
|
"AGE-SECRET-KEY-1",
|
||||||
|
"-----BEGIN",
|
||||||
|
"hvs.",
|
||||||
|
]
|
||||||
|
for marker in secret_markers:
|
||||||
|
if marker in encoded:
|
||||||
|
errors.append(f"secret-looking marker present: {marker}")
|
||||||
|
|
||||||
|
if errors:
|
||||||
|
for error in errors:
|
||||||
|
print(f"[FAIL] {error}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(f"[OK] emergency drill evidence is structurally valid: {path}")
|
||||||
|
print(f"[OK] drill_date: {data['drill_date']}")
|
||||||
|
print(f"[OK] source: {data['source_cluster']}/{data['source_namespace']}/{data['source_pod']}")
|
||||||
|
print(f"[OK] availability_window_minutes: {data['availability_window_minutes']}")
|
||||||
|
PY
|
||||||
@@ -301,6 +301,14 @@ restore completion, unseal/status/test-secret verification, isolated
|
|||||||
environment destruction, and a `no_secret_material_recorded` assertion. This
|
environment destruction, and a `no_secret_material_recorded` assertion. This
|
||||||
keeps `NET-WP-0017-T02` from relying on a bare UI checkbox for restore proof.
|
keeps `NET-WP-0017-T02` from relying on a bare UI checkbox for restore proof.
|
||||||
|
|
||||||
|
**2026-06-01:** Added the matching non-secret emergency seal/unseal drill
|
||||||
|
evidence template and `make openbao-validate-emergency-evidence`. The validator
|
||||||
|
requires an attended seal/unseal evidence file with timing, sealed-state proof,
|
||||||
|
unseal quorum availability, post-unseal verification, availability-window
|
||||||
|
duration, and `no_secret_material_recorded`. The validator does not run the
|
||||||
|
disruptive drill; it only checks the evidence captured after the attended
|
||||||
|
operation.
|
||||||
|
|
||||||
### T07 - Cross-Repo Transition Tasks
|
### T07 - Cross-Repo Transition Tasks
|
||||||
|
|
||||||
```task
|
```task
|
||||||
|
|||||||
Reference in New Issue
Block a user