Record ops-hub bootstrap progress and add API bootstrap helper

Document the 2026-06-15 attended Inter-Hub bootstrap: hub row, active manifest, widget seeding, and runtime API key creation. Add scripts/ops-hub-bootstrap-api.py, extend the SQL fallback for widget versions and the first Gitea event, refresh OpsHubBootstrapRunbook, and inline credential-routing guidance for agents.
2026-06-19 19:09:21 +02:00
parent 3638ee14ad
commit 215e62a221
7 changed files with 1005 additions and 43 deletions
--- a/workplans/HF-WP-0001-establish-ops-hub-first-extension.md
+++ b/workplans/HF-WP-0001-establish-ops-hub-first-extension.md
@@ -7,7 +7,7 @@ repo: helix-forge
 status: active
 owner: worsch
 created: "2026-05-16"
-updated: "2026-06-14"
+updated: "2026-06-15"
 planning_priority: high
 planning_order: 1
 related_repos:
@@ -370,7 +370,7 @@ Output: `Confirmed Bootstrap Path` section in this workplan.

 ```task
 id: HF-WP-0001-T02
-status: wait
+status: done
 priority: high
 state_hub_task_id: "8e9bd9b2-54fc-49a4-8bb8-11c8577be48d"
 ```
@@ -393,8 +393,9 @@ fields as an Inter-Hub API/model gap.
 Done when: `ops-hub` appears in `/Hubs` and `/api/v2/hub-registry` after
 authentication, and a human can tell that it is the VSM Operations hub.

-Blocked until: an authenticated Inter-Hub admin session or deployment-side
-migration is available.
+Ready when: the operator loads the `IHUB_OPERATOR_KEY` from OpenBao into a
+trusted shell without echoing it. The key exists in approved custody, but the
+hub row has not been created yet.

 Prepared artifacts:

@@ -402,13 +403,37 @@ Prepared artifacts:
 - `wiki/ops-hub-manifest.draft.json`
 - `wiki/ops-hub-bootstrap.sql`

+Dependency update on 2026-06-15: a temporary Inter-Hub bootstrap operator key
+was minted directly in the `interhub` database after the web admin seed
+credential failed. OpenBao audit confirms the key was stored at
+`platform/operators/inter-hub/bootstrap-operator`, and Inter-Hub DB metadata
+shows active key prefix `8fab0bef` for `inter-hub-bootstrap-operator`.
+
+Implementation progress on 2026-06-15: added
+`scripts/ops-hub-bootstrap-api.py`, which uses
+`wiki/ops-hub-manifest.draft.json` and `wiki/ops-hub-widgets.seed.json` to
+create/reuse the hub, activate the full manifest, create the runtime
+`ops-hub` API consumer/key, seed widgets, and submit the first Gitea readiness
+event through the supported Inter-Hub API. The helper requires
+`IHUB_OPERATOR_KEY` or `IHUB_OPERATOR_KEY_FILE` and does not print full key
+values.
+
+Live result on 2026-06-15: the attended helper run used the OpenBao-custodied
+operator key from a 0600 temp file and created/reused the Inter-Hub hub row.
+Verified non-secret state:
+
+- Hub id `4f6e4cf7-6a96-4ff2-8a37-08c9f9e405d2`
+- Slug `ops-hub`
+- VSM metadata `hub_family=vsm`, `vsm_function=OPS`, `vsm_system=1`
+- Active manifest id `00aaf90a-8e76-4b0e-892d-33b162862f38`
+
 ---

 ### T03 — Activate the ops-hub capability manifest

 ```task
 id: HF-WP-0001-T03
-status: wait
+status: done
 priority: high
 state_hub_task_id: "55f5aeed-21c3-4a83-bc78-f90f92c7d597"
 ```
@@ -440,6 +465,15 @@ operator or migration can create and activate the manifest.

 Prepared artifact: `wiki/ops-hub-manifest.draft.json`.

+Live result on 2026-06-15: the full manifest is active in production. Public
+registry checks found all expected ops vocabulary values:
+
+- 14 of 14 widget types present.
+- 15 of 15 event types present.
+- 10 of 10 annotation categories present.
+- Policy scopes are present in the production database and still lack a
+  dedicated public validation check in this repo's helper.
+
 ---

 ### T04 — Create ops-hub API consumer and key
@@ -467,13 +501,46 @@ Blocked until: an authenticated operator creates the API key and stores the
 full static key outside Git. The SQL fallback intentionally creates only the
 consumer row, not the one-time visible secret.

+Progress on 2026-06-15:
+
+- The `ops-hub` API consumer exists with id
+  `f9e595c6-4e1d-41fd-86cb-c1830bd7ec81`.
+- The bootstrap helper created the display-once runtime key and wrote it to a
+  local 0600 temp file without printing the key.
+- `POST /api/v2/token` returns a short-lived Bearer token for the runtime key
+  with `expires_in=3600`.
+- `GET /api/v2/widgets` works with the runtime key and returns the 14
+  `ops-hub` widgets.
+
+Remaining before closing:
+
+- Store the runtime key from the temporary file in OpenBao at
+  `platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, then remove the
+  temp file.
+- `GET /api/v2/hub-registry` currently returns HTTP 500 because Inter-Hub
+  decodes `COUNT(*)` from `api_request_log` as `Int` while PostgreSQL returns
+  `bigint`. Track/fix this under T10 before treating hub-registry as a clean
+  acceptance signal.
+
+Custody attempt on 2026-06-15: copied the generated runtime key into the
+OpenBao pod as a temporary file and attempted to write it to the approved KV
+path using the pod token helper. OpenBao denied the request with `403
+permission denied` while resolving the KV mount through
+`sys/internal/ui/mounts/platform/operators/ops-hub/runtime`. The temporary
+in-pod key file was removed and verified absent. The local 0600 runtime-key
+file remains because it has not yet been successfully stored in OpenBao.
+
+Current blocker: requires an attended OpenBao root/sudo token handoff, or the
+operator storing the local runtime key manually through the browser UI, before
+the temp file can be removed and this task can close.
+
 ---

 ### T05 — Seed first governed ops widgets

 ```task
 id: HF-WP-0001-T05
-status: wait
+status: done
 priority: high
 state_hub_task_id: "d303884d-d1f6-4fd0-a4ec-97afe6162164"
 ```
@@ -508,6 +575,13 @@ Prepared artifacts:
 - `wiki/ops-hub-widgets.seed.json`
 - `wiki/ops-hub-bootstrap.sql`

+Live result on 2026-06-15: widget creation through `POST /api/v2/widgets`
+failed because Inter-Hub decodes `COUNT(*)` from the type-registry validation
+query as `Int` while PostgreSQL returns `bigint`. The operator-approved SQL
+fallback was then applied idempotently and created the 14 governed widgets plus
+14 initial widget version rows. A runtime-key API smoke check verified all 14
+widgets are readable through `GET /api/v2/widgets`.
+
 ---

 ### T06 — Build the first ops inventory artifact
@@ -547,7 +621,7 @@ Output: `wiki/OpsHubInventory.md`.

 ```task
 id: HF-WP-0001-T07
-status: wait
+status: done
 priority: medium
 state_hub_task_id: "ed3e0396-b16d-40c2-9519-e755ad6241eb"
 ```
@@ -578,6 +652,17 @@ Blocked until: the `ops-endpoint-gitea-registry` widget exists, the
 `ops-endpoint-verified` event type is active, and an ops-hub API key is
 available to the operator.

+Live result on 2026-06-15: the first Gitea registry readiness event was
+inserted by the SQL fallback and is visible through
+`GET /api/v2/interaction-events` with the runtime key:
+
+- Event id `4af73b21-75a9-4814-b6df-62083cfda15f`
+- Event type `ops-endpoint-verified`
+- View context `railiance-apps/workplans/RAIL-AP-WP-0001`
+- Metadata records endpoint `https://gitea.coulomb.social/v2/`, expected
+  status `401`, and observed header
+  `Docker-Distribution-Api-Version: registry/2.0`
+
 ---

 ### T08 — Define the ops-hub readiness gate model for ThreePhoenix migration
@@ -728,6 +813,102 @@ Production gate recheck on 2026-06-14:
  on deployment of the current Inter-Hub API or an explicit operator decision
  to use the manual SQL fallback.

+Production gate recheck after Inter-Hub deployment on 2026-06-14:
+
+- `https://hub.coulomb.social/api/v2/hubs` now returns `200` with an empty
+  paginated hub list instead of `404`.
+- The live OpenAPI now lists `/hubs`, `/hub-capability-manifests`,
+  `/api-consumers`, and `/policy-scopes`.
+- OpenAPI shows public reads for `GET /hubs` and `GET /policy-scopes`, and
+  authenticated `BearerAuth` writes for hub, manifest, API consumer/key,
+  widget, and interaction-event creation.
+- `ops-hub/scripts/interhub-gate-probe.py` still exits nonzero because it
+  expects unauthenticated `GET /api/v2/hubs` to return `401`; that expectation
+  is stale relative to the deployed public-read/authenticated-write contract.
+- Result: the Inter-Hub bootstrap API hardening and production deployment gate
+  are complete. HF-WP-0001 now waits on an attended bootstrap run with
+  `IHUB_OPERATOR_KEY` or equivalent authenticated operator session, not on
+  manual SQL fallback or API deployment.
+
+Live bootstrap follow-up on 2026-06-15:
+
+- Hub, manifest, API consumer, and runtime key creation work through the live
+  API.
+- `POST /api/v2/widgets` fails with
+  `UnexpectedColumnTypeStatementError 0 23 20` from
+  `SELECT COUNT(*) FROM widget_type_registry ...`; the query result needs to be
+  decoded as `Int64` or cast to `int`.
+- `GET /api/v2/hub-registry` with either the static runtime key or a
+  short-lived token fails with the same class of error from
+  `SELECT COUNT(*) FROM api_request_log ...`.
+- The operator-approved SQL fallback was used for seed widgets and the first
+  event so HF-WP-0001 could keep moving, but the next VSM hub is not yet fully
+  scriptable without direct DB access.
+
+Source fix on 2026-06-15: patched Inter-Hub
+`Application/Helper/TypeRegistry.hs` and
+`Application/Helper/ApiRateLimit.hs` to cast the affected `COUNT(*)` queries
+to `int`, committed as `5101eb5 Fix API count decoding`, and pushed to
+`origin/main`.
+
+Deployment status on 2026-06-15:
+
+- Local `git diff --check` passed in `inter-hub`.
+- `nix develop ... scripts/compile-check` could not run because the checkout
+  lacks `.devenv/root` and plain `nix develop` cannot determine the current
+  directory in this shell.
+- A local `nix build .#docker` was attempted, but after more than 20 minutes it
+  was still compiling dependencies and could not be used for deployment from
+  this session because registry/deploy credentials are not available here.
+- Railiance01 release inspection through `railiance-apps` shows the live
+  `hub.coulomb.social` Deployment still uses
+  `92.205.130.254:32166/coulomb/inter-hub:790b5e5`, while Helm values report
+  `image.tag: 11ff61c`; in either view, the pushed `5101eb5` fix is not live.
+
+Railiance deploy-surface recheck on 2026-06-15:
+
+- `railiance-apps` commit `c7d49d3` adds local Makefile targets for
+  `INTER_HUB_IMAGE_TAG=<sha>` dry-run, deploy, status, release-info, logs, and
+  smoke checks.
+- `INTER_HUB_IMAGE_TAG=5101eb5 make inter-hub-dry-run` renders
+  `gitea.coulomb.social/coulomb/inter-hub:5101eb5` and preserves the live
+  immutable selector label `app=inter-hub`.
+- A Helm `--dry-run=server` upgrade against Railiance01 with tag `5101eb5`
+  succeeds, so the chart shape is acceptable to the cluster.
+- The Gitea registry currently returns `MANIFEST_UNKNOWN` for
+  `coulomb/inter-hub:5101eb5`, so the deployment must not be run until the
+  Inter-Hub image is built and published.
+- `make inter-hub-smoke` is stale: it expects public
+  `GET /api/v2/hubs` to return `401` and checks `/openapi.json`, while the
+  live contract is `200` for public hub discovery, `401` for protected
+  resources such as `/api/v2/widgets`, and OpenAPI at `/api/v2/openapi.json`.
+- `railiance-apps` still has only the pull-request manifest dry-run workflow;
+  it does not yet provide a `workflow_dispatch` production deploy trigger.
+
+Railiance deploy-surface follow-up review on 2026-06-15:
+
+- `railiance-apps` commit `6abf753` adds `RAILIANCE-WP-0011` and addresses the
+  requested hardening: OCI image preflight, explicit production dry-runs,
+  `inter-hub-server-dry-run`, current public-read/authenticated-write smoke
+  contract, and `.gitea/workflows/inter-hub-production-deploy.yaml` with
+  `workflow_dispatch`.
+- `make inter-hub-smoke` passes against live `https://hub.coulomb.social`:
+  public `GET /api/v2/hubs` returns discovery JSON, protected widgets and
+  hub-registry routes return `401 invalid_api_key` without a key, and
+  `/api/v2/openapi.json` lists the expected v2 resources.
+- `INTER_HUB_IMAGE_TAG=5101eb5 make check-inter-hub-image` fails before Helm
+  with `manifest unknown`, which is the desired safe behavior while the image
+  is absent.
+- Inter-Hub confirmed via State Hub message
+  `269d0ace-5b8e-4fec-a1d0-11a52ad23cc5` that the remaining deploy-side
+  blocker is to build and publish `gitea.coulomb.social/coulomb/inter-hub:5101eb5`
+  or another tag containing the same `COUNT(*)` decode fix.
+
+Current blocker: publish a Gitea registry image for Inter-Hub commit
+`5101eb5` or an equivalent fix tag, then deploy it through the approved
+Railiance path and rerun the authenticated widget-create and hub-registry
+smoke checks. Railiance-apps no longer appears to be the blocking surface.
+
 ## Initial Acceptance Criteria

 This workplan is complete when:
@@ -820,6 +1001,109 @@ Interpretation:
 - HF-WP-0001 T10 is moved to `wait` until the production API is deployed or the
  operator explicitly chooses the manual SQL fallback.

+### 2026-06-14 — production API gate opened
+
+Rechecked the live Inter-Hub API after deployment moved forward:
+
+- `GET https://hub.coulomb.social/api/v2/hubs` now returns `200` with
+  `{"data":[],"meta":{"page":1,"per_page":50,"total":0}}`.
+- The live OpenAPI includes `/hubs`, `/hub-capability-manifests`,
+  `/api-consumers`, and `/policy-scopes`.
+- The deployed contract exposes public reads for hub and policy-scope lists,
+  while creation and mutation paths require `BearerAuth`.
+- `IHUB_OPERATOR_KEY` was not present in this Codex shell, so the authenticated
+  bootstrap/smoke script was not run.
+
+Result:
+
+- T10 is closed as `done`.
+- T02 is ready to run once the operator loads the OpenBao-custodied
+  `IHUB_OPERATOR_KEY` into a trusted shell.
+- T03-T05 remain `wait` until the `ops-hub` row and active manifest exist.
+- The next action is to run the supported bootstrap path with
+  `IHUB_BASE=https://hub.coulomb.social` and the operator key, while preserving
+  the one-time API key secret outside Git and State Hub.
+
+### 2026-06-15 — bootstrap operator key created and custodied
+
+The seeded Inter-Hub web admin credential was not usable on production, so the
+operator explicitly chose a controlled database bootstrap for the first
+temporary API key.
+
+Non-secret evidence:
+
+- `net-kingdom-pg-1` is healthy and contains the `interhub` database.
+- `api_consumers` initially had no rows.
+- A static `inter-hub-bootstrap-operator` key now exists with prefix
+  `8fab0bef`.
+- OpenBao audit shows successful create/read activity for
+  `platform/data/operators/inter-hub/bootstrap-operator`.
+
+The full key value was not printed into Git, State Hub, or chat. It was stored
+manually by the operator in OpenBao. This unblocks the next attended step:
+retrieve the key from OpenBao into a trusted shell and create the `ops-hub`
+hub row through the supported Inter-Hub API.
+
+### 2026-06-15 — full API bootstrap helper prepared
+
+Added `scripts/ops-hub-bootstrap-api.py` as the HF-WP-0001 production helper.
+Unlike the source-side Inter-Hub smoke script, it consumes the full
+HelixForge manifest and widget seed artifacts. It supports
+`IHUB_OPERATOR_KEY_FILE` and `OPS_HUB_KEY_FILE` so keys can be passed through
+0600 temp files rather than shell history or chat.
+
+Validation performed:
+
+- `python3 -m py_compile scripts/ops-hub-bootstrap-api.py`
+- `python3 scripts/ops-hub-bootstrap-api.py --help`
+- JSON seed parsing for the manifest and 14 widget seeds
+
+### 2026-06-15 — attended Ops Hub bootstrap applied
+
+The operator provided the Inter-Hub bootstrap key through a local temp file.
+The helper used that key without printing it and successfully created/reused:
+
+- `ops-hub` hub row with VSM metadata `vsm/OPS/1`.
+- Active capability manifest `00aaf90a-8e76-4b0e-892d-33b162862f38`.
+- Runtime API consumer `f9e595c6-4e1d-41fd-86cb-c1830bd7ec81`.
+- Display-once runtime API key, written only to a 0600 local temp file.
+
+The helper then exposed a live Inter-Hub bug while creating widgets:
+
+```text
+UnexpectedColumnTypeStatementError 0 23 20
+SELECT COUNT(*) FROM widget_type_registry WHERE name = $1 AND status = 'active'
+```
+
+Because PostgreSQL returns `COUNT(*)` as `bigint`, the Inter-Hub validation
+code must decode this as `Int64` or cast the query result. The same bug class
+also affects authenticated `GET /api/v2/hub-registry` through the
+`api_request_log` rate-limit query.
+
+To keep the approved bootstrap moving, the idempotent SQL fallback was applied
+for the remaining data. It created:
+
+- 14 governed `ops-hub` widgets.
+- 14 initial widget version rows.
+- The first `ops-endpoint-verified` Gitea registry readiness event.
+
+Validation:
+
+- All 14 widget types, 15 event types, and 10 annotation categories from the
+  manifest are present in public registries.
+- `GET /api/v2/widgets` with the runtime key returns all 14 `ops-hub` widgets.
+- `POST /api/v2/token` returns a short-lived Bearer token for the runtime key.
+- `GET /api/v2/interaction-events` with the runtime key returns the Gitea
+  registry readiness event and metadata.
+
+Remaining operator action:
+
+- Store the generated runtime key from the local temp file in OpenBao at
+  `platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, then remove the
+  temp file.
+- Track/fix the Inter-Hub `COUNT(*)` decode issues before declaring the next
+  VSM hub fully scriptable through the public API.
+
 ## Notes

 `ops-hub` should complement State Hub during the transition: