helix-forge/wiki/OpsHubBootstrapRunbook.md

Ops Hub Bootstrap Runbook

Date: 2026-06-14

Purpose

This runbook gives the operator-ready bootstrap path for ops-hub, the VSM Operations / System 1 extension of Inter-Hub.

Use this when an authenticated Inter-Hub admin session or deployment migration is available. The current public v2 API now exposes the supported bootstrap surface; creation and mutation calls require an Inter-Hub operator/admin API key.

As of 2026-06-06, implementation work for ops-hub belongs in the dedicated repo at /home/worsch/ops-hub with remote gitea-remote:coulomb/ops-hub.git. This runbook remains a HelixForge bootstrap handoff reference until the implementation repo ports or supersedes it.

Inputs

• Manifest draft: wiki/ops-hub-manifest.draft.json
• Widget seed: wiki/ops-hub-widgets.seed.json
• Migration fallback: wiki/ops-hub-bootstrap.sql
• Implementation repo: /home/worsch/ops-hub

Current Bootstrap Decision

Prefer the supported Inter-Hub bootstrap API for hub, manifest, API consumer, and runtime key creation. Use the idempotent SQL fallback for seed widgets and initial evidence only when the live API hits a known server-side bootstrap bug and the operator explicitly approves that fallback.

Latest check, 2026-06-14 after Inter-Hub deployment:

• https://hub.coulomb.social/api/v2/hubs returns 200 with a public, paginated hub list.
• OpenAPI lists /hubs, /hub-capability-manifests, /api-consumers, and /policy-scopes.
• Hub and policy-scope list reads are public. Hub creation, manifest creation/activation, API consumer/key creation, widget creation, and event submission require Authorization: Bearer <key>.
• The old ops-hub/scripts/interhub-gate-probe.py expectation that unauthenticated GET /api/v2/hubs must return 401 is stale. The deployed contract is public-read/authenticated-write.

VSM classification is stored in the manifest capability description for now:

• hub_family: vsm
• vsm_function: OPS
• vsm_system: S1

Newer Inter-Hub schemas have first-class hub metadata columns for these values. The SQL fallback sets those columns when present and still carries the same classification in the manifest description for older deployments.

Latest bootstrap result, 2026-06-15:

• scripts/ops-hub-bootstrap-api.py created/reused the ops-hub hub row, active manifest, runtime API consumer, and display-once runtime API key.
• POST /api/v2/widgets failed with an Inter-Hub COUNT(*) decode error in type-registry validation.
• The operator-approved SQL fallback seeded 14 widgets, 14 initial widget versions, and the first Gitea registry readiness event.
• The runtime key can be exchanged through POST /api/v2/token, can read all 14 widgets through GET /api/v2/widgets, and can read the first event through GET /api/v2/interaction-events.
• GET /api/v2/hub-registry currently returns HTTP 500 because Inter-Hub decodes COUNT(*) from api_request_log as Int while PostgreSQL returns bigint.

Inter-Hub Operator Key

IHUB_OPERATOR_KEY is an existing Inter-Hub API key or short-lived access token accepted by v2 endpoints as:

Authorization: Bearer <key>

It is needed only for privileged bootstrap writes:

• creating or reusing the ops-hub hub row;
• creating and activating the capability manifest;
• creating the ops-hub runtime ApiConsumer;
• minting the display-once runtime API key;
• creating widgets and submitting the first event.

The operator key is not the long-term ops-hub runtime credential. It should be used for the attended bootstrap only, then normal ops-hub traffic should use the narrower runtime key created during bootstrap.

Allowed sources:

1. Retrieve an existing Inter-Hub operator/admin key from the approved secret store, if one has already been created. After HF-WP-0002 is deployed, the intended browser path is https://bao.coulomb.social with KeyCape auth path netkingdom and role platform-admin. The older keycape auth path may remain available as a compatibility alias.
2. If no such key exists, log in to the Inter-Hub admin UI and create an operator/bootstrap ApiConsumer and API key. The full key is display-once; store it immediately in the approved secret store.
3. NetKingdom/OpenBao may provide the key only as the secret-custody path once the key exists and an appropriate policy/path is defined. The net-kingdom Git repository must not contain the key.

net-kingdom-pg-1 is only the PostgreSQL pod used by the SQL fallback. It is not by itself the source of an Inter-Hub operator key.

When using the OpenBao browser UI, inspect metadata/path presence first, for example under:

platform/operators/
platform/operators/inter-hub/

Do not copy secret values into Git, State Hub, chat, shell history, or this runbook. Store display-once Inter-Hub and ops-hub runtime keys directly in the approved secret path.

For an attended local run, enter the key in a trusted shell without echoing it:

export IHUB_BASE="https://hub.coulomb.social"
read -rsp "Inter-Hub operator key: " IHUB_OPERATOR_KEY
echo
export IHUB_OPERATOR_KEY

The preferred HelixForge helper uses the prepared manifest and widget seed artifacts instead of the smaller one-widget smoke fixture:

umask 077
IHUB_KEY_FILE=$(mktemp)
# Paste the OpenBao value into this file without printing it to the terminal.
read -rsp "Inter-Hub operator key: " IHUB_OPERATOR_KEY
printf '%s' "$IHUB_OPERATOR_KEY" > "$IHUB_KEY_FILE"
unset IHUB_OPERATOR_KEY
echo

IHUB_BASE="https://hub.coulomb.social" \
IHUB_OPERATOR_KEY_FILE="$IHUB_KEY_FILE" \
python3 scripts/ops-hub-bootstrap-api.py

The helper creates/reuses the ops-hub hub row, activates the full HelixForge manifest, creates the ops-hub API consumer, creates the runtime API key if needed, creates any missing seed widgets, and submits the first Gitea registry readiness event. If it creates a new ops-hub runtime API key, it writes the full key to a 0600 temp file and prints only the path and key prefix. Store that runtime key immediately in OpenBao, for example:

platform/operators/ops-hub/runtime
field: OPS_HUB_KEY

After storing both keys in OpenBao, remove local temp files:

rm -f "$IHUB_KEY_FILE" "<runtime-key-file-printed-by-helper>"
unset IHUB_KEY_FILE

/home/worsch/inter-hub/scripts/ops-hub-bootstrap-smoke.py remains useful as a narrow source-side smoke proof, but it creates a one-widget fixture rather than the full HF-WP-0001 Operations vocabulary.

As of the 2026-05-19 access check, the workstation kubeconfig only points at CoulombCore (92.205.130.254) and does not include the Railiance01 (92.205.62.239) cluster where hub.coulomb.social resolves. SSH key access to root, worsch, and ubuntu on Railiance01 was denied, so the SQL fallback must be run from a host/session that already has Railiance01 cluster access.

UI Path

1. Log in to Inter-Hub at https://hub.coulomb.social/NewSession.
2. Open /Hubs/new.
3. Create the hub:
   • Name: Ops Hub
   • Slug: ops-hub
   • Domain: ops.coulomb.social
   • Kind: domain
4. Open /HubCapabilityManifests/new?hubId=<ops-hub-id>.
5. Create a draft manifest with:
   • Version: 1.0
   • Capability description from wiki/ops-hub-manifest.draft.json
   • Contact: operator/team contact
6. Edit the manifest and copy in:
   • declaredWidgetTypes
   • declaredEventTypes
   • declaredAnnotationCategories
   • declaredPolicyScopes
7. Activate the manifest.
8. Open /ApiConsumers/new.
9. Create an API consumer bound to the active ops manifest:
   • Name: ops-hub
   • Description: API consumer for the VSM Operations hub
   • Scopes for the key: framework:read hub:ops-hub:read hub:ops-hub:write
10. Generate an API key and store it only in the operator secret store or local environment. Do not commit it to Git.
11. Seed the widgets from wiki/ops-hub-widgets.seed.json through the UI or migration fallback.

SQL Fallback Path

From a shell with Railiance01 Kubernetes access, run:

kubectl exec -i -n databases net-kingdom-pg-1 -- \
  psql -U postgres -d interhub \
  < /home/worsch/helix-forge/wiki/ops-hub-bootstrap.sql

If using the HostEurope kubeconfig from the workstation, first restore ~/.kube/config-hosteurope as documented by railiance-cluster, then run:

KUBECONFIG=~/.kube/config-hosteurope \
kubectl exec -i -n databases net-kingdom-pg-1 -- \
  psql -U postgres -d interhub \
  < /home/worsch/helix-forge/wiki/ops-hub-bootstrap.sql

The SQL fallback creates the hub, active manifest, registry entries, API consumer row, seed widgets, initial widget version rows, and the first Gitea registry readiness event. It does not create the one-time visible static API key; generate that through the authenticated Inter-Hub API/UI helper and store it outside Git.

Validation

After manifest activation:

curl -s https://hub.coulomb.social/api/v2/widget-types
curl -s https://hub.coulomb.social/api/v2/event-types
curl -s https://hub.coulomb.social/api/v2/annotation-categories

Expected: ops-owned vocabulary appears in the relevant registries.

After API key creation:

curl -s -X POST https://hub.coulomb.social/api/v2/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "grant_type=client_credentials" \
  --data-urlencode "client_id=<api-consumer-id>" \
  --data-urlencode "client_secret=<static-api-key>" \
  --data-urlencode "scope=framework:read hub:ops-hub:read hub:ops-hub:write"

Expected: a short-lived access token is returned.

After widget seeding:

curl -s https://hub.coulomb.social/api/v2/widgets
curl -s https://hub.coulomb.social/api/v2/interaction-events

Expected: the 14 seeded ops-hub widgets are readable, and the Gitea registry readiness event is visible. GET /api/v2/hub-registry is the desired final registry validation, but it currently returns HTTP 500 on production because of the api_request_log COUNT(*) decode issue.

Known Blockers

• POST /api/v2/widgets currently fails on production because Inter-Hub decodes COUNT(*) from widget_type_registry as Int while PostgreSQL returns bigint.
• GET /api/v2/hub-registry currently fails on production for the same bug class in the api_request_log rate-limit query.
• The generated ops-hub runtime key must be moved from the local temp file into OpenBao at platform/operators/ops-hub/runtime, field OPS_HUB_KEY, then the temp file must be removed.

These are tracked by HF-WP-0001 T10 for Inter-Hub hardening.