generated from coulomb/repo-seed
tegwick
215e62a221
Document the 2026-06-15 attended Inter-Hub bootstrap: hub row, active manifest, widget seeding, and runtime API key creation. Add scripts/ops-hub-bootstrap-api.py, extend the SQL fallback for widget versions and the first Gitea event, refresh OpsHubBootstrapRunbook, and inline credential-routing guidance for agents.
282 lines
10 KiB
Markdown
282 lines
10 KiB
Markdown
# Ops Hub Bootstrap Runbook
|
|
|
|
Date: 2026-06-14
|
|
|
|
## Purpose
|
|
|
|
This runbook gives the operator-ready bootstrap path for `ops-hub`, the VSM
|
|
Operations / System 1 extension of Inter-Hub.
|
|
|
|
Use this when an authenticated Inter-Hub admin session or deployment migration
|
|
is available. The current public v2 API now exposes the supported bootstrap
|
|
surface; creation and mutation calls require an Inter-Hub operator/admin API
|
|
key.
|
|
|
|
As of 2026-06-06, implementation work for `ops-hub` belongs in the dedicated
|
|
repo at `/home/worsch/ops-hub` with remote `gitea-remote:coulomb/ops-hub.git`.
|
|
This runbook remains a HelixForge bootstrap handoff reference until the
|
|
implementation repo ports or supersedes it.
|
|
|
|
## Inputs
|
|
|
|
- Manifest draft: `wiki/ops-hub-manifest.draft.json`
|
|
- Widget seed: `wiki/ops-hub-widgets.seed.json`
|
|
- Migration fallback: `wiki/ops-hub-bootstrap.sql`
|
|
- Implementation repo: `/home/worsch/ops-hub`
|
|
|
|
## Current Bootstrap Decision
|
|
|
|
Prefer the supported Inter-Hub bootstrap API for hub, manifest, API consumer,
|
|
and runtime key creation. Use the idempotent SQL fallback for seed widgets and
|
|
initial evidence only when the live API hits a known server-side bootstrap bug
|
|
and the operator explicitly approves that fallback.
|
|
|
|
Latest check, 2026-06-14 after Inter-Hub deployment:
|
|
|
|
- `https://hub.coulomb.social/api/v2/hubs` returns `200` with a public,
|
|
paginated hub list.
|
|
- OpenAPI lists `/hubs`, `/hub-capability-manifests`, `/api-consumers`, and
|
|
`/policy-scopes`.
|
|
- Hub and policy-scope list reads are public. Hub creation, manifest
|
|
creation/activation, API consumer/key creation, widget creation, and event
|
|
submission require `Authorization: Bearer <key>`.
|
|
- The old `ops-hub/scripts/interhub-gate-probe.py` expectation that
|
|
unauthenticated `GET /api/v2/hubs` must return `401` is stale. The deployed
|
|
contract is public-read/authenticated-write.
|
|
|
|
VSM classification is stored in the manifest capability description for now:
|
|
|
|
- `hub_family`: `vsm`
|
|
- `vsm_function`: `OPS`
|
|
- `vsm_system`: `S1`
|
|
|
|
Newer Inter-Hub schemas have first-class hub metadata columns for these values.
|
|
The SQL fallback sets those columns when present and still carries the same
|
|
classification in the manifest description for older deployments.
|
|
|
|
Latest bootstrap result, 2026-06-15:
|
|
|
|
- `scripts/ops-hub-bootstrap-api.py` created/reused the `ops-hub` hub row,
|
|
active manifest, runtime API consumer, and display-once runtime API key.
|
|
- `POST /api/v2/widgets` failed with an Inter-Hub `COUNT(*)` decode error in
|
|
type-registry validation.
|
|
- The operator-approved SQL fallback seeded 14 widgets, 14 initial widget
|
|
versions, and the first Gitea registry readiness event.
|
|
- The runtime key can be exchanged through `POST /api/v2/token`, can read all
|
|
14 widgets through `GET /api/v2/widgets`, and can read the first event
|
|
through `GET /api/v2/interaction-events`.
|
|
- `GET /api/v2/hub-registry` currently returns HTTP 500 because Inter-Hub
|
|
decodes `COUNT(*)` from `api_request_log` as `Int` while PostgreSQL returns
|
|
`bigint`.
|
|
|
|
## Inter-Hub Operator Key
|
|
|
|
`IHUB_OPERATOR_KEY` is an existing Inter-Hub API key or short-lived access
|
|
token accepted by v2 endpoints as:
|
|
|
|
```text
|
|
Authorization: Bearer <key>
|
|
```
|
|
|
|
It is needed only for privileged bootstrap writes:
|
|
|
|
- creating or reusing the `ops-hub` hub row;
|
|
- creating and activating the capability manifest;
|
|
- creating the `ops-hub` runtime `ApiConsumer`;
|
|
- minting the display-once runtime API key;
|
|
- creating widgets and submitting the first event.
|
|
|
|
The operator key is not the long-term `ops-hub` runtime credential. It should
|
|
be used for the attended bootstrap only, then normal `ops-hub` traffic should
|
|
use the narrower runtime key created during bootstrap.
|
|
|
|
Allowed sources:
|
|
|
|
1. Retrieve an existing Inter-Hub operator/admin key from the approved secret
|
|
store, if one has already been created. After `HF-WP-0002` is deployed,
|
|
the intended browser path is `https://bao.coulomb.social` with KeyCape auth
|
|
path `netkingdom` and role `platform-admin`. The older `keycape` auth path
|
|
may remain available as a compatibility alias.
|
|
2. If no such key exists, log in to the Inter-Hub admin UI and create an
|
|
operator/bootstrap `ApiConsumer` and API key. The full key is display-once;
|
|
store it immediately in the approved secret store.
|
|
3. NetKingdom/OpenBao may provide the key only as the secret-custody path once
|
|
the key exists and an appropriate policy/path is defined. The
|
|
`net-kingdom` Git repository must not contain the key.
|
|
|
|
`net-kingdom-pg-1` is only the PostgreSQL pod used by the SQL fallback. It is
|
|
not by itself the source of an Inter-Hub operator key.
|
|
|
|
When using the OpenBao browser UI, inspect metadata/path presence first, for
|
|
example under:
|
|
|
|
```text
|
|
platform/operators/
|
|
platform/operators/inter-hub/
|
|
```
|
|
|
|
Do not copy secret values into Git, State Hub, chat, shell history, or this
|
|
runbook. Store display-once Inter-Hub and `ops-hub` runtime keys directly in
|
|
the approved secret path.
|
|
|
|
For an attended local run, enter the key in a trusted shell without echoing it:
|
|
|
|
```bash
|
|
export IHUB_BASE="https://hub.coulomb.social"
|
|
read -rsp "Inter-Hub operator key: " IHUB_OPERATOR_KEY
|
|
echo
|
|
export IHUB_OPERATOR_KEY
|
|
```
|
|
|
|
The preferred HelixForge helper uses the prepared manifest and widget seed
|
|
artifacts instead of the smaller one-widget smoke fixture:
|
|
|
|
```bash
|
|
umask 077
|
|
IHUB_KEY_FILE=$(mktemp)
|
|
# Paste the OpenBao value into this file without printing it to the terminal.
|
|
read -rsp "Inter-Hub operator key: " IHUB_OPERATOR_KEY
|
|
printf '%s' "$IHUB_OPERATOR_KEY" > "$IHUB_KEY_FILE"
|
|
unset IHUB_OPERATOR_KEY
|
|
echo
|
|
|
|
IHUB_BASE="https://hub.coulomb.social" \
|
|
IHUB_OPERATOR_KEY_FILE="$IHUB_KEY_FILE" \
|
|
python3 scripts/ops-hub-bootstrap-api.py
|
|
```
|
|
|
|
The helper creates/reuses the `ops-hub` hub row, activates the full
|
|
HelixForge manifest, creates the `ops-hub` API consumer, creates the runtime
|
|
API key if needed, creates any missing seed widgets, and submits the first
|
|
Gitea registry readiness event. If it creates a new `ops-hub` runtime API key,
|
|
it writes the full key to a 0600 temp file and prints only the path and key
|
|
prefix. Store that runtime key immediately in OpenBao, for example:
|
|
|
|
```text
|
|
platform/operators/ops-hub/runtime
|
|
field: OPS_HUB_KEY
|
|
```
|
|
|
|
After storing both keys in OpenBao, remove local temp files:
|
|
|
|
```bash
|
|
rm -f "$IHUB_KEY_FILE" "<runtime-key-file-printed-by-helper>"
|
|
unset IHUB_KEY_FILE
|
|
```
|
|
|
|
`/home/worsch/inter-hub/scripts/ops-hub-bootstrap-smoke.py` remains useful as
|
|
a narrow source-side smoke proof, but it creates a one-widget fixture rather
|
|
than the full HF-WP-0001 Operations vocabulary.
|
|
|
|
As of the 2026-05-19 access check, the workstation kubeconfig only points at
|
|
CoulombCore (`92.205.130.254`) and does not include the Railiance01
|
|
(`92.205.62.239`) cluster where `hub.coulomb.social` resolves. SSH key access
|
|
to `root`, `worsch`, and `ubuntu` on Railiance01 was denied, so the SQL fallback
|
|
must be run from a host/session that already has Railiance01 cluster access.
|
|
|
|
## UI Path
|
|
|
|
1. Log in to Inter-Hub at `https://hub.coulomb.social/NewSession`.
|
|
2. Open `/Hubs/new`.
|
|
3. Create the hub:
|
|
- Name: `Ops Hub`
|
|
- Slug: `ops-hub`
|
|
- Domain: `ops.coulomb.social`
|
|
- Kind: `domain`
|
|
4. Open `/HubCapabilityManifests/new?hubId=<ops-hub-id>`.
|
|
5. Create a draft manifest with:
|
|
- Version: `1.0`
|
|
- Capability description from `wiki/ops-hub-manifest.draft.json`
|
|
- Contact: operator/team contact
|
|
6. Edit the manifest and copy in:
|
|
- `declaredWidgetTypes`
|
|
- `declaredEventTypes`
|
|
- `declaredAnnotationCategories`
|
|
- `declaredPolicyScopes`
|
|
7. Activate the manifest.
|
|
8. Open `/ApiConsumers/new`.
|
|
9. Create an API consumer bound to the active ops manifest:
|
|
- Name: `ops-hub`
|
|
- Description: `API consumer for the VSM Operations hub`
|
|
- Scopes for the key: `framework:read hub:ops-hub:read hub:ops-hub:write`
|
|
10. Generate an API key and store it only in the operator secret store or local
|
|
environment. Do not commit it to Git.
|
|
11. Seed the widgets from `wiki/ops-hub-widgets.seed.json` through the UI or
|
|
migration fallback.
|
|
|
|
## SQL Fallback Path
|
|
|
|
From a shell with Railiance01 Kubernetes access, run:
|
|
|
|
```bash
|
|
kubectl exec -i -n databases net-kingdom-pg-1 -- \
|
|
psql -U postgres -d interhub \
|
|
< /home/worsch/helix-forge/wiki/ops-hub-bootstrap.sql
|
|
```
|
|
|
|
If using the HostEurope kubeconfig from the workstation, first restore
|
|
`~/.kube/config-hosteurope` as documented by `railiance-cluster`, then run:
|
|
|
|
```bash
|
|
KUBECONFIG=~/.kube/config-hosteurope \
|
|
kubectl exec -i -n databases net-kingdom-pg-1 -- \
|
|
psql -U postgres -d interhub \
|
|
< /home/worsch/helix-forge/wiki/ops-hub-bootstrap.sql
|
|
```
|
|
|
|
The SQL fallback creates the hub, active manifest, registry entries, API
|
|
consumer row, seed widgets, initial widget version rows, and the first Gitea
|
|
registry readiness event. It does not create the one-time visible static API
|
|
key; generate that through the authenticated Inter-Hub API/UI helper and store
|
|
it outside Git.
|
|
|
|
## Validation
|
|
|
|
After manifest activation:
|
|
|
|
```bash
|
|
curl -s https://hub.coulomb.social/api/v2/widget-types
|
|
curl -s https://hub.coulomb.social/api/v2/event-types
|
|
curl -s https://hub.coulomb.social/api/v2/annotation-categories
|
|
```
|
|
|
|
Expected: ops-owned vocabulary appears in the relevant registries.
|
|
|
|
After API key creation:
|
|
|
|
```bash
|
|
curl -s -X POST https://hub.coulomb.social/api/v2/token \
|
|
-H "Content-Type: application/x-www-form-urlencoded" \
|
|
--data-urlencode "grant_type=client_credentials" \
|
|
--data-urlencode "client_id=<api-consumer-id>" \
|
|
--data-urlencode "client_secret=<static-api-key>" \
|
|
--data-urlencode "scope=framework:read hub:ops-hub:read hub:ops-hub:write"
|
|
```
|
|
|
|
Expected: a short-lived access token is returned.
|
|
|
|
After widget seeding:
|
|
|
|
```bash
|
|
curl -s https://hub.coulomb.social/api/v2/widgets
|
|
curl -s https://hub.coulomb.social/api/v2/interaction-events
|
|
```
|
|
|
|
Expected: the 14 seeded `ops-hub` widgets are readable, and the Gitea registry
|
|
readiness event is visible. `GET /api/v2/hub-registry` is the desired final
|
|
registry validation, but it currently returns HTTP 500 on production because
|
|
of the `api_request_log` `COUNT(*)` decode issue.
|
|
|
|
## Known Blockers
|
|
|
|
- `POST /api/v2/widgets` currently fails on production because Inter-Hub
|
|
decodes `COUNT(*)` from `widget_type_registry` as `Int` while PostgreSQL
|
|
returns `bigint`.
|
|
- `GET /api/v2/hub-registry` currently fails on production for the same bug
|
|
class in the `api_request_log` rate-limit query.
|
|
- The generated `ops-hub` runtime key must be moved from the local temp file
|
|
into OpenBao at `platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`,
|
|
then the temp file must be removed.
|
|
|
|
These are tracked by HF-WP-0001 T10 for Inter-Hub hardening.
|