--- id: HF-WP-0001 type: workplan title: "Establish ops-hub as the First VSM Inter-Hub Extension" domain: helix_forge repo: helix-forge status: finished owner: worsch created: "2026-05-16" updated: "2026-06-19" planning_priority: high planning_order: 1 related_repos: - inter-hub - ops-hub - railiance-infra - railiance-cluster - railiance-platform - railiance-apps state_hub_workstream_id: "48d91935-197e-4ad4-be07-7bbcd535847c" --- # Establish ops-hub as the First VSM Inter-Hub Extension ## Goal Use Inter-Hub as the generic hub framework and establish `ops-hub` as the first VSM-oriented domain hub extension: the Operations / System 1 hub. `ops-hub` should professionalize Railiance operations while the current CoulombCore environment transitions toward the future ThreePhoenix production setup. Just as importantly, it should prove the repeatable extension pattern for the later VSM hubs: - `ops-hub` — Operations and Activities / System 1 - `syn-hub` — Synchronization and Coordination / System 2 - `ctl-hub` — Internal Control and Regulation / System 3 - `aud-hub` — Audit and Monitoring / System 3* - `int-hub` — Intelligence and Adaptation / System 4 - `pol-hub` — Policy and Identity / System 5 - `env-hub` — Boundary and Environment The first increment should not replace State Hub. It establishes the operational model, the VSM hub vocabulary, and the smallest governed integration with Inter-Hub. As of 2026-06-06, `ops-hub` has its own repository at `/home/worsch/ops-hub` with remote `gitea-remote:coulomb/ops-hub.git`; use that repo for implementation from now on while treating it as an extension of Inter-Hub rather than a fork of the generic hub framework. ## VSM Hub Extension Strategy Inter-Hub is the framework. HelixForge should extend it with specific hubs that map to the viable-system functions already named in `INTENT.md`. | Hub | VSM function | First responsibility | |---|---|---| | `ops-hub` | Operations and Activities / System 1 | Operational truth surface for environments, hosts, clusters, services, endpoints, releases, backups, incidents, risks, runbooks, and migration waves. | | `syn-hub` | Synchronization and Coordination / System 2 | Coordination between operational units, repos, workstreams, and service handoffs so local actions do not conflict. | | `ctl-hub` | Internal Control and Regulation / System 3 | Current-state control, resource constraints, readiness gates, priorities, and operational governance. | | `aud-hub` | Audit and Monitoring / System 3* | Independent evidence, checks, observations, drift detection, and verification trails. | | `int-hub` | Intelligence and Adaptation / System 4 | Future sensing, migration analysis, forecasting, recommendations, and adaptation planning. | | `pol-hub` | Policy and Identity / System 5 | Identity, values, ultimate constraints, policy decisions, and acceptable operating posture. | | `env-hub` | Boundary and Environment | External actors, surfaces, endpoints, users, markets, partner systems, and environmental signals. | This workplan starts only with `ops-hub`, but every bootstrap choice should be judged by whether it can become the template for the next hub. ## Context `wiki/CurrentOperationsSituation.md` captures the immediate operational background as of 2026-05-15. The short version: the operational platform is real, useful, and already carrying production-like responsibilities, but its state is spread across live systems, repo workplans, shell knowledge, and operator memory. There is no central service catalog or operational registry yet. Current operational reality: - `coulombcore` / `92.205.130.254` is the live production-like server. It runs the current Gitea deployment and other hands-on experimental services. - The local workstation still hosts important services such as State Hub and local build/runtime pieces. - `railiance01` / `92.205.62.239` is the first server of the intended future ThreePhoenix production environment. - The Railiance repo stack already separates operational responsibility: `railiance-infra` (S1), `railiance-cluster` (S2), `railiance-platform` (S3), `railiance-enablement` (S4), and `railiance-apps` (S5). - Gitea is live on the CoulombCore Kubernetes cluster as Helm release `gitea` in namespace `default`, exposed through NodePort `32166`, with its database in namespace `databases` and shared data on PVC `default/gitea-shared-storage`. - The Gitea OCI registry route at `https://gitea.coulomb.social/v2/` now returns the expected registry auth challenge, but publishing still needs to be proven with encrypted Helm values, a package token, `docker login`, push, and pull. - Ops Bridge can help reveal which servers are connected and reachable, but it is not itself a full operational service catalog. `ops-hub` should become the operational truth surface across those realities: environments, hosts, clusters, services, releases, endpoints, backups, readiness gates, incidents, risks, service discovery, and migration waves. ## Inter-Hub API Findings Checked live and local Inter-Hub evidence on 2026-05-16. Live API: - `https://hub.coulomb.social/api/v2/openapi.json` is available. - `https://hub.coulomb.social/api/v2/docs` is available. - Public UI route `https://hub.coulomb.social/Hubs` redirects to `/NewSession`, so hub creation is currently an authenticated UI flow. Live OpenAPI paths: - `/widgets`, `/widgets/{id}` - `/interaction-events` - `/annotations` - `/requirement-candidates`, `/requirement-candidates/{id}` - `/decision-records`, `/decision-records/{id}` - `/deployment-records`, `/deployment-records/{id}` - `/outcome-signals`, `/outcome-signals/{id}` - `/widget-types`, `/event-types`, `/annotation-categories` - `/hub-registry`, `/hub-registry/{hubId}` - `/widget-patterns`, `/widget-patterns/{id}`, `/widget-patterns/{id}/adopt` - `/token` Useful local Inter-Hub docs: - `inter-hub/docs/domain-hub-extension-guide.md` - `inter-hub/docs/new-hub-quickstart.md` - `inter-hub/contracts/extensions/hub-capability-manifest-v1.md` - `inter-hub/contracts/functional/interaction-reporting-v1.md` Assessment: - Inter-Hub provides enough guidance to start `ops-hub` as an API consumer pattern or as a manually registered domain hub. - Inter-Hub does not yet provide enough API surface to fully automate first hub bootstrap. Hub creation, capability manifest creation/activation, API consumer creation, API key issuance, and widget creation are primarily UI or internal-controller workflows. - The quickstart mentions `POST /api/v2/hubs` and `POST /api/v2/widgets`, but the live OpenAPI and local routes do not expose those create endpoints. Treat the quickstart as aspirational for bootstrap automation until Inter-Hub is hardened. ## Confirmed Bootstrap Path Checked against the live API and local Inter-Hub source on 2026-05-16. Decision: - Bootstrap `ops-hub` through the authenticated Inter-Hub admin UI where possible, with a migration-backed fallback when a repeatable bootstrap is needed before the public API is hardened. - Treat Pattern A, API Consumer Hub, as the first implementation pattern. - Store VSM classification in first-class hub metadata where the target Inter-Hub schema has `hub_family`, `vsm_function`, and `vsm_system`; keep the same values in the manifest capability description for compatibility with older deployments and API consumers. - Use `hub_kind = domain` for `ops-hub`. - Record missing first-class VSM metadata fields and create endpoints as Inter-Hub hardening work under T10. Confirmed current support: - Live `/Hubs` redirects to `/NewSession`, so hub creation is an authenticated UI flow. - Local `HubsController` supports creating and editing hub rows through the UI. - Local `HubCapabilityManifestsController` supports draft creation, JSON-array vocabulary editing, activation, and registry upsert. - Local `ApiConsumersController` and `ApiKeysController` support authenticated UI creation of consumers and static keys. - Local `/api/v2/interaction-events` supports `POST` and validates event types. Confirmed gaps: - Live OpenAPI has no `POST /api/v2/hubs`. - Live OpenAPI has no `POST /api/v2/widgets`; widgets are read-only in v2. - There are no v2 endpoints for manifest draft creation, manifest activation, API consumer creation, or API key creation. - There is no `/api/v2/policy-scopes` endpoint. - Live type registries currently return empty arrays, so the ops vocabulary still needs manifest activation before events can be accepted. - Event metadata is exposed in the response schema, but the v2 interaction event create controller currently does not persist submitted metadata. - Webhook dispatch still uses the hard-coded `"clicked"` event name. ## Architectural Decision Start with **Pattern A: API Consumer Hub** for `ops-hub`, plus a manual or migration-backed Inter-Hub registration. Treat `ops-hub` as the first VSM hub instance rather than a one-off operational dashboard: 1. Register `ops-hub` as a domain hub in Inter-Hub. 2. Classify it as the Operations / System 1 hub in hub metadata or manifest metadata, depending on what Inter-Hub currently supports. 3. Activate a `HubCapabilityManifest` for its operational vocabulary. 4. Create an `ApiConsumer` and API key for `ops-hub`. 5. Seed a small set of governed widgets representing operational surfaces. 6. Emit interaction events and annotations from lightweight scripts or a prototype UI. The first reusable contract to prove is: ```text Hub identity + VSM function + manifest vocabulary + API consumer + seed widgets + evidence events ``` The next hubs should be able to follow the same shape with their own vocabularies: ```text syn-hub / ctl-hub / aud-hub / int-hub / pol-hub / env-hub ``` Use the separate `ops-hub` repository for implementation work from now on. `inter-hub` remains the framework, registry, authentication, manifest, widget, and event substrate. `helix-forge` remains the architecture/workplan home and keeps these bootstrap artifacts as handoff references until they are ported or superseded by `ops-hub` repo workplans. Railiance repos remain the desired state and evidence owners for their operational systems. ## Initial ops-hub Vocabulary This vocabulary is deliberately scoped to Operations / System 1. Coordination, control, audit, intelligence, policy, and environment concerns should be represented only where they touch operational evidence; their own hubs will own the broader semantics later. Suggested manifest values: ### Widget Types ```json [ "ops-environment", "ops-host", "ops-cluster", "ops-service", "ops-service-catalog", "ops-endpoint", "ops-release", "ops-backup-set", "ops-secret-set", "ops-runbook", "ops-incident", "ops-readiness-gate", "ops-migration-wave", "ops-risk" ] ``` ### Event Types ```json [ "ops-inventory-registered", "ops-inventory-updated", "ops-service-discovered", "ops-health-checked", "ops-release-observed", "ops-endpoint-verified", "ops-backup-verified", "ops-restore-tested", "ops-runbook-executed", "ops-drift-detected", "ops-risk-raised", "ops-risk-accepted", "ops-readiness-gate-updated", "ops-migration-gate-passed", "ops-migration-gate-failed" ] ``` ### Annotation Categories ```json [ "ops-drift", "ops-service-catalog-gap", "ops-backup-gap", "ops-security-gap", "ops-routing-gap", "ops-secret-gap", "ops-readiness-blocker", "ops-migration-risk", "ops-observability-gap", "ops-recovery-gap" ] ``` ### Policy Scopes ```json [ "ops-local", "ops-transitional-prod", "ops-production", "ops-threephoenix", "ops-registry", "ops-secrets", "ops-backup-retention" ] ``` ## Initial Operational Inventory The first ops-hub inventory should cover: | Environment | Role | Current notes | |---|---|---| | `local` | Workstation services and development runtime | State Hub and local build/runtime pieces currently live here. | | `coulombcore` | Live transitional production | Public IP `92.205.130.254`; hosts current Gitea and hand-built experimental production services. | | `railiance01` | Future production foundation | Public IP `92.205.62.239`; first server of the intended ThreePhoenix setup. | | `threephoenix-prod` | Target production topology | Future three-node Railiance production environment. | The first services to model: - Gitea / container registry - State Hub and underlying services - Inter-Hub itself - PostgreSQL/CNPG services used by Gitea and State Hub - Ingress/DNS/TLS endpoints for the above - Backup and restore coverage for each persistent data store - Ops Bridge connectivity as reachability evidence, not as the catalog itself The first explicit service-catalog gap: - There is no central place that answers "what runs where, why, who owns it, how it is reached, and what evidence proves it is healthy." `ops-hub` should make that question answerable before the ThreePhoenix migration becomes more complicated. ## Tasks ### T01 — Confirm the VSM hub extension bootstrap path ```task id: HF-WP-0001-T01 status: done priority: high state_hub_task_id: "2587a3b8-3b9b-4948-acaf-1547644e4563" ``` Confirm whether `ops-hub` should be registered through the Inter-Hub UI, through a migration, or through new API endpoints. Capture the result as the first repeatable VSM hub bootstrap path, not just as a local workaround for Operations. Checks: - Confirm the active Inter-Hub deployment URL and authentication path. - Confirm whether `/Hubs/new`, `/HubCapabilityManifests`, `/ApiConsumers`, and `/ApiKeys` are accessible to the operator. - Confirm whether direct DB migration is acceptable for initial bootstrap. - Confirm where hub metadata can carry the VSM function (`OPS`, `SYN`, `CTL`, `AUD`, `INT`, `POL`, `ENV`) and VSM system mapping. - Record the chosen bootstrap path in this workplan so `syn-hub` can reuse it. Done when: there is a concrete, repeatable path to create the `ops-hub` row, manifest, API consumer, and API key, with enough metadata to classify it as the Operations / System 1 hub. Output: `Confirmed Bootstrap Path` section in this workplan. --- ### T02 — Register ops-hub in Inter-Hub as the Operations hub ```task id: HF-WP-0001-T02 status: done priority: high state_hub_task_id: "8e9bd9b2-54fc-49a4-8bb8-11c8577be48d" ``` Create the Hub row: - `name`: `Ops Hub` - `slug`: `ops-hub` - `domain`: `ops.coulomb.social` or another explicit domain chosen by the operator - `hub_kind`: `domain` - VSM function metadata: `OPS` - VSM system metadata: `S1` - Hub family metadata: `vsm` If Inter-Hub does not yet have explicit fields for VSM function, system, or hub family, store them in manifest metadata and record the missing first-class fields as an Inter-Hub API/model gap. Done when: `ops-hub` appears in `/Hubs` and `/api/v2/hub-registry` after authentication, and a human can tell that it is the VSM Operations hub. Ready when: the operator loads the `IHUB_OPERATOR_KEY` from OpenBao into a trusted shell without echoing it. The key exists in approved custody, but the hub row has not been created yet. Prepared artifacts: - `wiki/OpsHubBootstrapRunbook.md` - `wiki/ops-hub-manifest.draft.json` - `wiki/ops-hub-bootstrap.sql` Dependency update on 2026-06-15: a temporary Inter-Hub bootstrap operator key was minted directly in the `interhub` database after the web admin seed credential failed. OpenBao audit confirms the key was stored at `platform/operators/inter-hub/bootstrap-operator`, and Inter-Hub DB metadata shows active key prefix `8fab0bef` for `inter-hub-bootstrap-operator`. Implementation progress on 2026-06-15: added `scripts/ops-hub-bootstrap-api.py`, which uses `wiki/ops-hub-manifest.draft.json` and `wiki/ops-hub-widgets.seed.json` to create/reuse the hub, activate the full manifest, create the runtime `ops-hub` API consumer/key, seed widgets, and submit the first Gitea readiness event through the supported Inter-Hub API. The helper requires `IHUB_OPERATOR_KEY` or `IHUB_OPERATOR_KEY_FILE` and does not print full key values. Live result on 2026-06-15: the attended helper run used the OpenBao-custodied operator key from a 0600 temp file and created/reused the Inter-Hub hub row. Verified non-secret state: - Hub id `4f6e4cf7-6a96-4ff2-8a37-08c9f9e405d2` - Slug `ops-hub` - VSM metadata `hub_family=vsm`, `vsm_function=OPS`, `vsm_system=1` - Active manifest id `00aaf90a-8e76-4b0e-892d-33b162862f38` --- ### T03 — Activate the ops-hub capability manifest ```task id: HF-WP-0001-T03 status: done priority: high state_hub_task_id: "55f5aeed-21c3-4a83-bc78-f90f92c7d597" ``` Create and activate a `HubCapabilityManifest` for `ops-hub` using the vocabulary in this workplan. The manifest should make the VSM classification explicit: - `hub_family`: `vsm` - `vsm_function`: `OPS` - `vsm_system`: `S1` - `scope`: operational truth and evidence, not coordination/control/audit ownership Validation: - Declared widget types appear in `/api/v2/widget-types`. - Declared event types appear in `/api/v2/event-types`. - Declared annotation categories appear in `/api/v2/annotation-categories`. - Policy scopes are visible in the Inter-Hub registry UI or DB, even though the public v2 API currently lacks `/policy-scopes`. - Future VSM hub values can be added by changing manifest vocabulary, not by inventing a different bootstrap mechanism. Done when: the manifest status is `active` and no type conflicts remain. Blocked until: the `ops-hub` row exists in Inter-Hub and an authenticated operator or migration can create and activate the manifest. Prepared artifact: `wiki/ops-hub-manifest.draft.json`. Live result on 2026-06-15: the full manifest is active in production. Public registry checks found all expected ops vocabulary values: - 14 of 14 widget types present. - 15 of 15 event types present. - 10 of 10 annotation categories present. - Policy scopes are present in the production database and still lack a dedicated public validation check in this repo's helper. --- ### T04 — Create ops-hub API consumer and key ```task id: HF-WP-0001-T04 status: done priority: high state_hub_task_id: "ad08e729-8562-4a02-8bf6-dcdfebe430c8" ``` Create an `ApiConsumer` associated with the active `ops-hub` manifest, then create a static API key with at least: - `framework:read` - `hub:ops-hub:read` - `hub:ops-hub:write` Store the key only in the operator secret store or local env file, never in Git. Done when: `POST /api/v2/token` can exchange the static key for a short-lived access token and `GET /api/v2/hub-registry` works with that token. Blocked until: an authenticated operator creates the API key and stores the full static key outside Git. The SQL fallback intentionally creates only the consumer row, not the one-time visible secret. Progress on 2026-06-15: - The `ops-hub` API consumer exists with id `f9e595c6-4e1d-41fd-86cb-c1830bd7ec81`. - The bootstrap helper created the display-once runtime key and wrote it to a local 0600 temp file without printing the key. - `POST /api/v2/token` returns a short-lived Bearer token for the runtime key with `expires_in=3600`. - `GET /api/v2/widgets` works with the runtime key and returns the 14 `ops-hub` widgets. Remaining before closing: - Store the runtime key from the temporary file in OpenBao at `platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, then remove the temp file. - `GET /api/v2/hub-registry` currently returns HTTP 500 because Inter-Hub decodes `COUNT(*)` from `api_request_log` as `Int` while PostgreSQL returns `bigint`. Track/fix this under T10 before treating hub-registry as a clean acceptance signal. Custody attempt on 2026-06-15: copied the generated runtime key into the OpenBao pod as a temporary file and attempted to write it to the approved KV path using the pod token helper. OpenBao denied the request with `403 permission denied` while resolving the KV mount through `sys/internal/ui/mounts/platform/operators/ops-hub/runtime`. The temporary in-pod key file was removed and verified absent. The local 0600 runtime-key file remains because it has not yet been successfully stored in OpenBao. Current blocker: requires an attended OpenBao root/sudo token handoff, or the operator storing the local runtime key manually through the browser UI, before the temp file can be removed and this task can close. Completed on 2026-06-19: - Regenerated the display-once runtime key through `scripts/ops-hub-bootstrap-api.py` after the earlier 0600 temp file was no longer present. - Stored the runtime key in OpenBao at `platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, using an approved operator token. No key values were copied into Git, State Hub, or chat. - Removed the local runtime-key temp file after successful OpenBao write. - Verified non-secret acceptance evidence with the custodied runtime key: - `POST /api/v2/token` exchanges the static key for a short-lived Bearer token (`expires_in=3600`). - `GET /api/v2/hub-registry` returns HTTP `200` with the exchanged token. - `GET /api/v2/widgets` returns all 14 `ops-hub` widgets with the exchanged token. - Current runtime key prefix: `c1f3ac3a`. --- ### T05 — Seed first governed ops widgets ```task id: HF-WP-0001-T05 status: done priority: high state_hub_task_id: "d303884d-d1f6-4fd0-a4ec-97afe6162164" ``` Create initial widgets for the operational surfaces: - `ops-env-local` - `ops-env-coulombcore` - `ops-env-railiance01` - `ops-env-threephoenix-prod` - `ops-host-coulombcore` - `ops-host-railiance01` - `ops-service-catalog` - `ops-service-gitea` - `ops-service-state-hub` - `ops-service-inter-hub` - `ops-endpoint-gitea-registry` - `ops-readiness-gitea-registry` - `ops-readiness-state-hub-cluster-deploy` - `ops-migration-coulombcore-to-threephoenix` If Inter-Hub still lacks a widget creation API, seed these through the UI or a migration and record that as an API gap. Done when: the widgets appear under `ops-hub` and can accept interaction events and annotations. Blocked until: `ops-hub` and its active manifest exist in Inter-Hub. Prepared artifacts: - `wiki/ops-hub-widgets.seed.json` - `wiki/ops-hub-bootstrap.sql` Live result on 2026-06-15: widget creation through `POST /api/v2/widgets` failed because Inter-Hub decodes `COUNT(*)` from the type-registry validation query as `Int` while PostgreSQL returns `bigint`. The operator-approved SQL fallback was then applied idempotently and created the 14 governed widgets plus 14 initial widget version rows. A runtime-key API smoke check verified all 14 widgets are readable through `GET /api/v2/widgets`. --- ### T06 — Build the first ops inventory artifact ```task id: HF-WP-0001-T06 status: done priority: medium state_hub_task_id: "2a0b2f69-5a3d-433c-9cbd-85fd868b63d8" ``` Create an ops inventory document in `helix-forge` that expresses the current state of: - environments - hosts - clusters - services - endpoints - service discovery and service-catalog gaps - storage and backup coverage - migration readiness gates Use `wiki/CurrentOperationsSituation.md` as the seed background, then turn it into a more structured inventory artifact. Use this as the working model before creating a separate `ops-hub` repository. Done when: a human can see the CoulombCore, local, railiance01, and ThreePhoenix relationship, including the current Gitea registry state, without reading multiple repo workplans or relying on shell history. Output: `wiki/OpsHubInventory.md`. --- ### T07 — Instrument the current Gitea registry work as the first ops-hub signal ```task id: HF-WP-0001-T07 status: done priority: medium state_hub_task_id: "ed3e0396-b16d-40c2-9519-e755ad6241eb" ``` Use the recently fixed Gitea `/v2` route as the first real operational signal. Suggested event: ```json { "widgetId": "", "eventType": "ops-endpoint-verified", "viewContext": "railiance-apps/workplans/RAIL-AP-WP-0001", "metadata": { "vsmFunction": "OPS", "vsmSystem": "S1", "endpoint": "https://gitea.coulomb.social/v2/", "expectedStatus": 401, "observedHeader": "Docker-Distribution-Api-Version: registry/2.0" } } ``` Done when: the Gitea registry readiness event is visible in Inter-Hub and traceable back to the Railiance workplan. Blocked until: the `ops-endpoint-gitea-registry` widget exists, the `ops-endpoint-verified` event type is active, and an ops-hub API key is available to the operator. Live result on 2026-06-15: the first Gitea registry readiness event was inserted by the SQL fallback and is visible through `GET /api/v2/interaction-events` with the runtime key: - Event id `4af73b21-75a9-4814-b6df-62083cfda15f` - Event type `ops-endpoint-verified` - View context `railiance-apps/workplans/RAIL-AP-WP-0001` - Metadata records endpoint `https://gitea.coulomb.social/v2/`, expected status `401`, and observed header `Docker-Distribution-Api-Version: registry/2.0` --- ### T08 — Define the ops-hub readiness gate model for ThreePhoenix migration ```task id: HF-WP-0001-T08 status: done priority: medium state_hub_task_id: "72a58622-c3ac-4765-8026-5c2489af2058" ``` Define readiness gates that must be green before moving production responsibility from CoulombCore to ThreePhoenix: - DNS and TLS are codified. - Service catalog entries exist for the live and target production services. - Git hosting and container registry are reproducible. - Persistent data stores have backup and restore evidence. - Secrets and SOPS/age keys are available through governed operator paths. - Cluster runtime and platform services are recreated through Railiance repos. - Rollback path is documented. - Operator runbooks exist for deploy, restore, rotate, and incident response. Done when: each gate has an owner repo, evidence requirement, and status. Output: `wiki/OpsHubReadinessGates.md`. --- ### T09 — Decide whether to create a separate ops-hub repository ```task id: HF-WP-0001-T09 status: done priority: medium state_hub_task_id: "0e5842fd-1d33-4e2a-9701-07f623a2b901" ``` Decision recorded on 2026-06-06: create and use a separate `ops-hub` repository. The repo is present locally at `/home/worsch/ops-hub` and tracks `gitea-remote:coulomb/ops-hub.git`. Rationale: - The operator has provided the repo and decided that `ops-hub` should live outside `helix-forge`. - `ops-hub` is expected to need implementation assets such as collectors, adapters, scheduled probes, bootstrap smoke tooling, and possibly UI beyond Inter-Hub's generic hub dashboards. - `ops-hub` needs its own workplan prefix, release lifecycle, and repository agent instructions once implementation begins. - Keeping deployable/runtime code out of `inter-hub` preserves Inter-Hub as the generic hub framework while still allowing `ops-hub` to be included as a VSM Operations / System 1 extension. Repository boundary: - `ops-hub`: implementation home for the Operations hub extension, including collectors, adapters, scheduled probes, runtime packaging, UI/extensions, tests, and Inter-Hub bootstrap/smoke clients. - `inter-hub`: generic framework, extension registry, hub manifests, API consumers, authentication, widgets, event persistence, and bootstrap API. - `helix-forge`: product/architecture intent, VSM extension pattern, coordination workplan, and current bootstrap handoff artifacts until they are ported or retired. - Railiance repos: desired state, deployment, backup/restore evidence, and operational facts for their owned infrastructure and services. Done: the decision is recorded with rationale and boundary; future implementation should happen in `ops-hub`. --- ### T10 — Inter-Hub API hardening for VSM hub bootstrap ```task id: HF-WP-0001-T10 status: done priority: high target_repo: inter-hub state_hub_task_id: "7fa54508-7add-4885-8913-12edaadc4d92" ``` Create or link an `inter-hub` workplan to make VSM domain hub bootstrapping machine-repeatable. Recommended Inter-Hub improvements: 1. Add `POST /api/v2/hubs` and include it in OpenAPI. 2. Add `POST /api/v2/widgets` and include it in OpenAPI. 3. Add API endpoints for `HubCapabilityManifest` draft creation, update, and activation. 4. Add a documented place for hub-family metadata such as `hub_family`, `vsm_function`, and `vsm_system`. 5. Add API endpoints for `ApiConsumer` and API key creation, or a clearly documented admin-only bootstrap command if API key creation remains UI-only. 6. Add `/api/v2/policy-scopes` to match the policy scope registry already used by manifests. 7. Add distinct OpenAPI request schemas for create requests instead of reusing response schemas. 8. Align `docs/new-hub-quickstart.md` with the actual live API until the create endpoints exist. 9. Fix `Web.Controller.Api.V2.InteractionEvents` so manifest-declared event types are actually decoded and enforced. 10. Fix webhook dispatch so it uses the submitted event type instead of the hard-coded `"clicked"` event name. 11. Decide whether event `metadata` is part of the v2 create contract; if yes, persist it in the controller and test it. 12. Document the bootstrap recipe as a template for `syn-hub`, `ctl-hub`, `aud-hub`, `int-hub`, `pol-hub`, and `env-hub`. Done when: the next VSM hub can be created from a script using documented API calls and without direct DB access. Linked Inter-Hub workplan: `inter-hub/workplans/IHUB-WP-0019-vsm-hub-bootstrap-api.md`. Source implementation status as of 2026-06-14: - `IHUB-WP-0019` is finished in the local `inter-hub` repo, and `main` is aligned with `origin/main`. - The documented smoke path exists at `/home/worsch/inter-hub/scripts/ops-hub-bootstrap-smoke.py`. - The `ops-hub` implementation repo has the handoff track `workplans/OPS-WP-0002-interhub-extension-bootstrap.md`; its production gate probe is `scripts/interhub-gate-probe.py`. Current production gate as of 2026-06-06: - Do not proceed with manual DB seeding unless the operator explicitly chooses that fallback. - Wait for `https://hub.coulomb.social/api/v2/hubs` to return `401` unauthenticated instead of `404`. - Confirm OpenAPI lists `/hubs`, `/hub-capability-manifests`, `/api-consumers`, and `/policy-scopes`. - After the gate passes, run the supported bootstrap/smoke path from the relevant `inter-hub` or `ops-hub` tooling with `IHUB_BASE` and an operator key. Production gate recheck on 2026-06-14: - `ops-hub/scripts/interhub-gate-probe.py` against `https://hub.coulomb.social` still fails. - `GET /api/v2/hubs` returns `404`, not `401`. - The live OpenAPI still omits `/hubs`, `/hub-capability-manifests`, `/api-consumers`, and `/policy-scopes`. - Result: source-side API hardening is complete, but HF-WP-0001 remains gated on deployment of the current Inter-Hub API or an explicit operator decision to use the manual SQL fallback. Production gate recheck after Inter-Hub deployment on 2026-06-14: - `https://hub.coulomb.social/api/v2/hubs` now returns `200` with an empty paginated hub list instead of `404`. - The live OpenAPI now lists `/hubs`, `/hub-capability-manifests`, `/api-consumers`, and `/policy-scopes`. - OpenAPI shows public reads for `GET /hubs` and `GET /policy-scopes`, and authenticated `BearerAuth` writes for hub, manifest, API consumer/key, widget, and interaction-event creation. - `ops-hub/scripts/interhub-gate-probe.py` still exits nonzero because it expects unauthenticated `GET /api/v2/hubs` to return `401`; that expectation is stale relative to the deployed public-read/authenticated-write contract. - Result: the Inter-Hub bootstrap API hardening and production deployment gate are complete. HF-WP-0001 now waits on an attended bootstrap run with `IHUB_OPERATOR_KEY` or equivalent authenticated operator session, not on manual SQL fallback or API deployment. Live bootstrap follow-up on 2026-06-15: - Hub, manifest, API consumer, and runtime key creation work through the live API. - `POST /api/v2/widgets` fails with `UnexpectedColumnTypeStatementError 0 23 20` from `SELECT COUNT(*) FROM widget_type_registry ...`; the query result needs to be decoded as `Int64` or cast to `int`. - `GET /api/v2/hub-registry` with either the static runtime key or a short-lived token fails with the same class of error from `SELECT COUNT(*) FROM api_request_log ...`. - The operator-approved SQL fallback was used for seed widgets and the first event so HF-WP-0001 could keep moving, but the next VSM hub is not yet fully scriptable without direct DB access. Source fix on 2026-06-15: patched Inter-Hub `Application/Helper/TypeRegistry.hs` and `Application/Helper/ApiRateLimit.hs` to cast the affected `COUNT(*)` queries to `int`, committed as `5101eb5 Fix API count decoding`, and pushed to `origin/main`. Deployment status on 2026-06-15: - Local `git diff --check` passed in `inter-hub`. - `nix develop ... scripts/compile-check` could not run because the checkout lacks `.devenv/root` and plain `nix develop` cannot determine the current directory in this shell. - A local `nix build .#docker` was attempted, but after more than 20 minutes it was still compiling dependencies and could not be used for deployment from this session because registry/deploy credentials are not available here. - Railiance01 release inspection through `railiance-apps` shows the live `hub.coulomb.social` Deployment still uses `92.205.130.254:32166/coulomb/inter-hub:790b5e5`, while Helm values report `image.tag: 11ff61c`; in either view, the pushed `5101eb5` fix is not live. Railiance deploy-surface recheck on 2026-06-15: - `railiance-apps` commit `c7d49d3` adds local Makefile targets for `INTER_HUB_IMAGE_TAG=` dry-run, deploy, status, release-info, logs, and smoke checks. - `INTER_HUB_IMAGE_TAG=5101eb5 make inter-hub-dry-run` renders `gitea.coulomb.social/coulomb/inter-hub:5101eb5` and preserves the live immutable selector label `app=inter-hub`. - A Helm `--dry-run=server` upgrade against Railiance01 with tag `5101eb5` succeeds, so the chart shape is acceptable to the cluster. - The Gitea registry currently returns `MANIFEST_UNKNOWN` for `coulomb/inter-hub:5101eb5`, so the deployment must not be run until the Inter-Hub image is built and published. - `make inter-hub-smoke` is stale: it expects public `GET /api/v2/hubs` to return `401` and checks `/openapi.json`, while the live contract is `200` for public hub discovery, `401` for protected resources such as `/api/v2/widgets`, and OpenAPI at `/api/v2/openapi.json`. - `railiance-apps` still has only the pull-request manifest dry-run workflow; it does not yet provide a `workflow_dispatch` production deploy trigger. Railiance deploy-surface follow-up review on 2026-06-15: - `railiance-apps` commit `6abf753` adds `RAILIANCE-WP-0011` and addresses the requested hardening: OCI image preflight, explicit production dry-runs, `inter-hub-server-dry-run`, current public-read/authenticated-write smoke contract, and `.gitea/workflows/inter-hub-production-deploy.yaml` with `workflow_dispatch`. - `make inter-hub-smoke` passes against live `https://hub.coulomb.social`: public `GET /api/v2/hubs` returns discovery JSON, protected widgets and hub-registry routes return `401 invalid_api_key` without a key, and `/api/v2/openapi.json` lists the expected v2 resources. - `INTER_HUB_IMAGE_TAG=5101eb5 make check-inter-hub-image` fails before Helm with `manifest unknown`, which is the desired safe behavior while the image is absent. - Inter-Hub confirmed via State Hub message `269d0ace-5b8e-4fec-a1d0-11a52ad23cc5` that the remaining deploy-side blocker is to build and publish `gitea.coulomb.social/coulomb/inter-hub:5101eb5` or another tag containing the same `COUNT(*)` decode fix. Current blocker: publish a Gitea registry image for Inter-Hub commit `5101eb5` or an equivalent fix tag, then deploy it through the approved Railiance path and rerun the authenticated widget-create and hub-registry smoke checks. Railiance-apps no longer appears to be the blocking surface. Completed on 2026-06-19: - Production Inter-Hub now runs image `gitea.coulomb.social/coulomb/inter-hub:eed4322`, which is ahead of the `5101eb5` COUNT-decode fix commit. - Authenticated `GET /api/v2/hub-registry` returns HTTP `200` with the bootstrap operator key and with a runtime key exchanged through `POST /api/v2/token`. - Authenticated `POST /api/v2/widgets` succeeds through the public API; a smoke widget was created and deleted without using direct DB access. - Result: the next VSM hub can bootstrap through the documented v2 API surface without the earlier `COUNT(*)` decode failure class blocking widget creation or hub-registry reads. ## Initial Acceptance Criteria This workplan is complete when: 1. `ops-hub` is registered in Inter-Hub as the VSM Operations / System 1 hub. 2. Its capability manifest is active. 3. It has an API consumer and key. 4. Initial ops widgets exist for environments, services, readiness gates, and migration waves. 5. At least one real operational event has been submitted. 6. The CoulombCore-to-ThreePhoenix readiness model is documented. 7. A decision has been made whether to create a separate `ops-hub` repository. 8. Inter-Hub bootstrap API gaps are either fixed or tracked in an Inter-Hub workplan. 9. The bootstrap path is reusable enough that `syn-hub` can be created next without rediscovering the whole process. ## Implementation Log ### 2026-05-19 — Live bootstrap access check Attempted to execute the prepared bootstrap path. Findings: - `hub.coulomb.social` resolves to Railiance01 (`92.205.62.239`). - The current workstation kubeconfig has only the `default` context and points at CoulombCore (`92.205.130.254`). - `~/.kube/config-hosteurope` is missing, so the documented Railiance01 kubeconfig path is not available locally. - `kubectl` against the current context has no `inter-hub` deployment because it is the CoulombCore cluster. - SSH key auth to `root@92.205.62.239`, `worsch@92.205.62.239`, and `ubuntu@92.205.62.239` was denied. Result: - Live `ops-hub` bootstrap was not applied from this session. - `wiki/ops-hub-bootstrap.sql` was strengthened to set first-class VSM hub metadata when the target Inter-Hub schema supports it. - `wiki/OpsHubBootstrapRunbook.md` now records the exact SQL fallback command to run from a Railiance01-capable session. Next required operator action: - Restore/provide Railiance01 Kubernetes access, or run `wiki/ops-hub-bootstrap.sql` from a host that already has access to the `net-kingdom-pg-1` pod in the `databases` namespace. ### 2026-06-06 — ops-hub repo decision The operator decided that `ops-hub` should live in its own repository and be included as an extension of Inter-Hub. The repo is available locally at `/home/worsch/ops-hub` with remote `gitea-remote:coulomb/ops-hub.git`. Result: - HF-WP-0001 T09 is closed as `done`. - `ops-hub` implementation work moves to the `ops-hub` repo. - `helix-forge` keeps architecture, vocabulary, readiness, and bootstrap handoff references. - Inter-Hub remains the generic extension framework and still owns the bootstrap API hardening tracked by T10. - Live bootstrap remains gated on current Inter-Hub production API availability unless the operator explicitly chooses the manual SQL fallback. ### 2026-06-14 — production API gate recheck Rechecked the live Inter-Hub bootstrap gate from the dedicated `ops-hub` implementation repo: ```text python3 scripts/interhub-gate-probe.py ``` Result: - `https://hub.coulomb.social/api/v2/hubs` returns `404`. - `https://hub.coulomb.social/api/v2/openapi.json` returns `200`, but the required bootstrap paths are absent. - Missing paths: `/hubs`, `/hub-capability-manifests`, `/api-consumers`, and `/policy-scopes`. Interpretation: - Inter-Hub source work for `IHUB-WP-0019` is complete and pushed to `origin/main`. - Production is still serving an older API surface, so the preferred ops-hub bootstrap path cannot run yet. - HF-WP-0001 T10 is moved to `wait` until the production API is deployed or the operator explicitly chooses the manual SQL fallback. ### 2026-06-14 — production API gate opened Rechecked the live Inter-Hub API after deployment moved forward: - `GET https://hub.coulomb.social/api/v2/hubs` now returns `200` with `{"data":[],"meta":{"page":1,"per_page":50,"total":0}}`. - The live OpenAPI includes `/hubs`, `/hub-capability-manifests`, `/api-consumers`, and `/policy-scopes`. - The deployed contract exposes public reads for hub and policy-scope lists, while creation and mutation paths require `BearerAuth`. - `IHUB_OPERATOR_KEY` was not present in this Codex shell, so the authenticated bootstrap/smoke script was not run. Result: - T10 is closed as `done`. - T02 is ready to run once the operator loads the OpenBao-custodied `IHUB_OPERATOR_KEY` into a trusted shell. - T03-T05 remain `wait` until the `ops-hub` row and active manifest exist. - The next action is to run the supported bootstrap path with `IHUB_BASE=https://hub.coulomb.social` and the operator key, while preserving the one-time API key secret outside Git and State Hub. ### 2026-06-15 — bootstrap operator key created and custodied The seeded Inter-Hub web admin credential was not usable on production, so the operator explicitly chose a controlled database bootstrap for the first temporary API key. Non-secret evidence: - `net-kingdom-pg-1` is healthy and contains the `interhub` database. - `api_consumers` initially had no rows. - A static `inter-hub-bootstrap-operator` key now exists with prefix `8fab0bef`. - OpenBao audit shows successful create/read activity for `platform/data/operators/inter-hub/bootstrap-operator`. The full key value was not printed into Git, State Hub, or chat. It was stored manually by the operator in OpenBao. This unblocks the next attended step: retrieve the key from OpenBao into a trusted shell and create the `ops-hub` hub row through the supported Inter-Hub API. ### 2026-06-15 — full API bootstrap helper prepared Added `scripts/ops-hub-bootstrap-api.py` as the HF-WP-0001 production helper. Unlike the source-side Inter-Hub smoke script, it consumes the full HelixForge manifest and widget seed artifacts. It supports `IHUB_OPERATOR_KEY_FILE` and `OPS_HUB_KEY_FILE` so keys can be passed through 0600 temp files rather than shell history or chat. Validation performed: - `python3 -m py_compile scripts/ops-hub-bootstrap-api.py` - `python3 scripts/ops-hub-bootstrap-api.py --help` - JSON seed parsing for the manifest and 14 widget seeds ### 2026-06-15 — attended Ops Hub bootstrap applied The operator provided the Inter-Hub bootstrap key through a local temp file. The helper used that key without printing it and successfully created/reused: - `ops-hub` hub row with VSM metadata `vsm/OPS/1`. - Active capability manifest `00aaf90a-8e76-4b0e-892d-33b162862f38`. - Runtime API consumer `f9e595c6-4e1d-41fd-86cb-c1830bd7ec81`. - Display-once runtime API key, written only to a 0600 local temp file. The helper then exposed a live Inter-Hub bug while creating widgets: ```text UnexpectedColumnTypeStatementError 0 23 20 SELECT COUNT(*) FROM widget_type_registry WHERE name = $1 AND status = 'active' ``` Because PostgreSQL returns `COUNT(*)` as `bigint`, the Inter-Hub validation code must decode this as `Int64` or cast the query result. The same bug class also affects authenticated `GET /api/v2/hub-registry` through the `api_request_log` rate-limit query. To keep the approved bootstrap moving, the idempotent SQL fallback was applied for the remaining data. It created: - 14 governed `ops-hub` widgets. - 14 initial widget version rows. - The first `ops-endpoint-verified` Gitea registry readiness event. Validation: - All 14 widget types, 15 event types, and 10 annotation categories from the manifest are present in public registries. - `GET /api/v2/widgets` with the runtime key returns all 14 `ops-hub` widgets. - `POST /api/v2/token` returns a short-lived Bearer token for the runtime key. - `GET /api/v2/interaction-events` with the runtime key returns the Gitea registry readiness event and metadata. Remaining operator action: - Store the generated runtime key from the local temp file in OpenBao at `platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, then remove the temp file. - Track/fix the Inter-Hub `COUNT(*)` decode issues before declaring the next VSM hub fully scriptable through the public API. ### 2026-06-19 — HF-WP-0001 closed out Closed the remaining bootstrap custody and production verification gaps: - Stored the `ops-hub` runtime key in OpenBao at `platform/operators/ops-hub/runtime` and removed the local temp file. - Verified runtime-key token exchange, hub-registry reads, and widget listing through the public Inter-Hub API. - Confirmed production Inter-Hub image `eed4322` includes the deployed COUNT-decode fix path; authenticated widget creation and hub-registry reads now succeed without SQL fallback. No API keys, OpenBao tokens, or secret values were copied into Git, State Hub, chat, or workplan text. ## Notes `ops-hub` should complement State Hub during the transition: - State Hub continues to track workstreams, decisions, and progress events. - `ops-hub` tracks operational reality and readiness evidence. - `syn-hub`, `ctl-hub`, and `aud-hub` can later absorb coordination, control, and evidence responsibilities once the broader hub constellation is established.