ops-hub/docs/bootstrap-runbook.md

# Ops Hub Bootstrap Runbook

Date: 2026-06-17

## Purpose

This runbook is the operator-ready path for activating `ops-hub` in production
Inter-Hub. It covers the preferred API bootstrap, key custody expectations, the
ops-warden remote execution lane, and the explicit SQL fallback.

Use this when finishing `CUST-WP-0047-T05` or any later ops-hub Inter-Hub
activation task.

## Inputs

- Manifest draft: `seeds/ops-hub-manifest.draft.json`
- Widget seed: `seeds/ops-hub-widgets.seed.json`
- API helper: `scripts/ops-hub-bootstrap-api.py`
- Migration fallback: `seeds/ops-hub-bootstrap.sql`
- Production Inter-Hub base URL: `https://hub.coulomb.social`

## Secret Custody Rules

- Do not paste operator keys into Codex-visible chat, workplans, commits, or
  shell transcripts.
- Prefer `IHUB_OPERATOR_KEY_FILE` over `IHUB_OPERATOR_KEY`.
- Operator key temp files must be mode `0600` and removed after the run.
- The generated ops-hub runtime key is display-once material. Store it in
  OpenBao or the approved operator secret store immediately.
- The helper prints only non-secret ids, key prefixes, and file paths.

## Preferred API Bootstrap

First confirm the API surface:

```bash
make interhub-gate
```

Then materialize the Inter-Hub operator key into a temporary file. Example
local pattern:

```bash
umask 077
operator_key_file="$(mktemp)"
# Write the operator key into "$operator_key_file" from the approved secret
# source. Do not echo it into shared logs.
```

Run the attended bootstrap:

```bash
make interhub-bootstrap \
  IHUB_BASE=https://hub.coulomb.social \
  IHUB_OPERATOR_KEY_FILE="$operator_key_file"
```

The helper creates or reuses:

- the `ops-hub` hub row
- the active ops-hub capability manifest
- the `ops-hub` API consumer
- an ops-hub runtime API key, when one was not supplied
- the seed widgets from `seeds/ops-hub-widgets.seed.json`
- the initial Gitea readiness event, unless `--skip-event` is used directly

If the helper creates a runtime key, it writes the full value to a `0600` file
and prints the path plus key prefix. Store that key immediately, then remove the
file:

```bash
runtime_key_file="/path/printed/as/runtimeKey.keyFile"
# Example path only; use the approved OpenBao mount/path for the environment.
bao kv put secret/platform/operators/ops-hub/runtime \
  OPS_HUB_KEY="$(cat "$runtime_key_file")"

rm -f "$operator_key_file" "$runtime_key_file"
```

If an ops-hub runtime key already exists in the secret store, materialize it as
`OPS_HUB_KEY_FILE` before the run. The helper will reuse it instead of creating
a new display-once key.

## Remote Execution With Ops-Warden

When the operator key must stay on a trusted host, use the ops-warden access
lane instead of moving the key into the workstation session:

1. Add or reuse an `agt` actor such as `agt-codex-interhub-bootstrap` with a
   narrow principal such as `agt-interhub-bootstrap` and a short TTL.
2. Use `ops-ssh-wrapper` to acquire a short-lived SSH certificate.
3. On the target host, railiance-infra should map the principal to a narrow
   force-command or wrapper that runs only this bootstrap path.
4. The host wrapper reads the operator key from OpenBao or an attended temp
   file, runs `make interhub-bootstrap`, stores the generated ops-hub runtime
   key back into OpenBao, and emits non-secret JSON evidence.

See `ops-warden/wiki/InterHubBootstrapAccessLane.md` for the certificate
envelope and host-boundary rules.

## SQL Fallback Path

Prefer the API bootstrap whenever possible. Use SQL only when the operator
explicitly approves deployment-side migration/bootstrap execution.

From a shell with Railiance01 Kubernetes access:

```bash
kubectl exec -i -n databases net-kingdom-pg-1 -- \
  psql -U postgres -d interhub \
  < /home/worsch/ops-hub/seeds/ops-hub-bootstrap.sql
```

If using the HostEurope kubeconfig from the workstation:

```bash
KUBECONFIG=~/.kube/config-hosteurope \
kubectl exec -i -n databases net-kingdom-pg-1 -- \
  psql -U postgres -d interhub \
  < /home/worsch/ops-hub/seeds/ops-hub-bootstrap.sql
```

The SQL fallback creates the hub, active manifest, registry entries, API
consumer row, and seed widgets. It does not create the one-time visible static
API key. Generate that through the authenticated Inter-Hub UI or the supported
API helper and store it outside Git.

## Validation

After manifest activation:

```bash
curl -s https://hub.coulomb.social/api/v2/widget-types
curl -s https://hub.coulomb.social/api/v2/event-types
curl -s https://hub.coulomb.social/api/v2/annotation-categories
```

Expected: ops-owned vocabulary appears in the relevant registries.

After API key creation or reuse:

```bash
curl -s -X POST https://hub.coulomb.social/api/v2/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "grant_type=client_credentials" \
  --data-urlencode "client_id=<api-consumer-id>" \
  --data-urlencode "client_secret=<static-api-key>" \
  --data-urlencode "scope=framework:read hub:ops-hub:read hub:ops-hub:write"
```

Expected: a short-lived access token is returned.

After widget seeding:

```bash
curl -s https://hub.coulomb.social/api/v2/hub-registry
```

Expected: `ops-hub` is visible, and the operator can see the seeded widgets in
the authenticated UI.

## Current Live-Execution Blocker

The repo-side helper and runbook are ready. The remaining blocker is an
authenticated production execution lane:

- an operator-provided `IHUB_OPERATOR_KEY_FILE`,
- an OpenBao-materialized key on a trusted host, or
- an explicitly approved deployment-side migration/bootstrap path.

Until one of those is available, run only local validation and gate probes.