helix-forge/workplans/HF-WP-0001-establish-ops-hub-first-extension.md

id, type, title, domain, repo, status, owner, created, updated, planning_priority, planning_order, related_repos, state_hub_workstream_id

id	type	title	domain	repo	status	owner	created	updated	planning_priority	planning_order	related_repos	state_hub_workstream_id
HF-WP-0001	workplan	Establish ops-hub as the First VSM Inter-Hub Extension	helix_forge	helix-forge	finished	worsch	2026-05-16	2026-06-19	high	1	inter-hub ops-hub railiance-infra railiance-cluster railiance-platform railiance-apps	48d91935-197e-4ad4-be07-7bbcd535847c

Establish ops-hub as the First VSM Inter-Hub Extension

Goal

Use Inter-Hub as the generic hub framework and establish ops-hub as the first VSM-oriented domain hub extension: the Operations / System 1 hub.

ops-hub should professionalize Railiance operations while the current CoulombCore environment transitions toward the future ThreePhoenix production setup. Just as importantly, it should prove the repeatable extension pattern for the later VSM hubs:

• ops-hub — Operations and Activities / System 1
• syn-hub — Synchronization and Coordination / System 2
• ctl-hub — Internal Control and Regulation / System 3
• aud-hub — Audit and Monitoring / System 3*
• int-hub — Intelligence and Adaptation / System 4
• pol-hub — Policy and Identity / System 5
• env-hub — Boundary and Environment

The first increment should not replace State Hub. It establishes the operational model, the VSM hub vocabulary, and the smallest governed integration with Inter-Hub. As of 2026-06-06, ops-hub has its own repository at /home/worsch/ops-hub with remote gitea-remote:coulomb/ops-hub.git; use that repo for implementation from now on while treating it as an extension of Inter-Hub rather than a fork of the generic hub framework.

VSM Hub Extension Strategy

Inter-Hub is the framework. HelixForge should extend it with specific hubs that map to the viable-system functions already named in INTENT.md.

Hub	VSM function	First responsibility
ops-hub	Operations and Activities / System 1	Operational truth surface for environments, hosts, clusters, services, endpoints, releases, backups, incidents, risks, runbooks, and migration waves.
syn-hub	Synchronization and Coordination / System 2	Coordination between operational units, repos, workstreams, and service handoffs so local actions do not conflict.
ctl-hub	Internal Control and Regulation / System 3	Current-state control, resource constraints, readiness gates, priorities, and operational governance.
aud-hub	Audit and Monitoring / System 3*	Independent evidence, checks, observations, drift detection, and verification trails.
int-hub	Intelligence and Adaptation / System 4	Future sensing, migration analysis, forecasting, recommendations, and adaptation planning.
pol-hub	Policy and Identity / System 5	Identity, values, ultimate constraints, policy decisions, and acceptable operating posture.
env-hub	Boundary and Environment	External actors, surfaces, endpoints, users, markets, partner systems, and environmental signals.

This workplan starts only with ops-hub, but every bootstrap choice should be judged by whether it can become the template for the next hub.

Context

wiki/CurrentOperationsSituation.md captures the immediate operational background as of 2026-05-15. The short version: the operational platform is real, useful, and already carrying production-like responsibilities, but its state is spread across live systems, repo workplans, shell knowledge, and operator memory. There is no central service catalog or operational registry yet.

Current operational reality:

• coulombcore / 92.205.130.254 is the live production-like server. It runs the current Gitea deployment and other hands-on experimental services.
• The local workstation still hosts important services such as State Hub and local build/runtime pieces.
• railiance01 / 92.205.62.239 is the first server of the intended future ThreePhoenix production environment.
• The Railiance repo stack already separates operational responsibility: railiance-infra (S1), railiance-cluster (S2), railiance-platform (S3), railiance-enablement (S4), and railiance-apps (S5).
• Gitea is live on the CoulombCore Kubernetes cluster as Helm release gitea in namespace default, exposed through NodePort 32166, with its database in namespace databases and shared data on PVC default/gitea-shared-storage.
• The Gitea OCI registry route at https://gitea.coulomb.social/v2/ now returns the expected registry auth challenge, but publishing still needs to be proven with encrypted Helm values, a package token, docker login, push, and pull.
• Ops Bridge can help reveal which servers are connected and reachable, but it is not itself a full operational service catalog.

ops-hub should become the operational truth surface across those realities: environments, hosts, clusters, services, releases, endpoints, backups, readiness gates, incidents, risks, service discovery, and migration waves.

Inter-Hub API Findings

Checked live and local Inter-Hub evidence on 2026-05-16.

Live API:

• https://hub.coulomb.social/api/v2/openapi.json is available.
• https://hub.coulomb.social/api/v2/docs is available.
• Public UI route https://hub.coulomb.social/Hubs redirects to /NewSession, so hub creation is currently an authenticated UI flow.

Live OpenAPI paths:

• /widgets, /widgets/{id}
• /interaction-events
• /annotations
• /requirement-candidates, /requirement-candidates/{id}
• /decision-records, /decision-records/{id}
• /deployment-records, /deployment-records/{id}
• /outcome-signals, /outcome-signals/{id}
• /widget-types, /event-types, /annotation-categories
• /hub-registry, /hub-registry/{hubId}
• /widget-patterns, /widget-patterns/{id}, /widget-patterns/{id}/adopt
• /token

Useful local Inter-Hub docs:

• inter-hub/docs/domain-hub-extension-guide.md
• inter-hub/docs/new-hub-quickstart.md
• inter-hub/contracts/extensions/hub-capability-manifest-v1.md
• inter-hub/contracts/functional/interaction-reporting-v1.md

Assessment:

• Inter-Hub provides enough guidance to start ops-hub as an API consumer pattern or as a manually registered domain hub.
• Inter-Hub does not yet provide enough API surface to fully automate first hub bootstrap. Hub creation, capability manifest creation/activation, API consumer creation, API key issuance, and widget creation are primarily UI or internal-controller workflows.
• The quickstart mentions POST /api/v2/hubs and POST /api/v2/widgets, but the live OpenAPI and local routes do not expose those create endpoints. Treat the quickstart as aspirational for bootstrap automation until Inter-Hub is hardened.

Confirmed Bootstrap Path

Checked against the live API and local Inter-Hub source on 2026-05-16.

Decision:

• Bootstrap ops-hub through the authenticated Inter-Hub admin UI where possible, with a migration-backed fallback when a repeatable bootstrap is needed before the public API is hardened.
• Treat Pattern A, API Consumer Hub, as the first implementation pattern.
• Store VSM classification in first-class hub metadata where the target Inter-Hub schema has hub_family, vsm_function, and vsm_system; keep the same values in the manifest capability description for compatibility with older deployments and API consumers.
• Use hub_kind = domain for ops-hub.
• Record missing first-class VSM metadata fields and create endpoints as Inter-Hub hardening work under T10.

Confirmed current support:

• Live /Hubs redirects to /NewSession, so hub creation is an authenticated UI flow.
• Local HubsController supports creating and editing hub rows through the UI.
• Local HubCapabilityManifestsController supports draft creation, JSON-array vocabulary editing, activation, and registry upsert.
• Local ApiConsumersController and ApiKeysController support authenticated UI creation of consumers and static keys.
• Local /api/v2/interaction-events supports POST and validates event types.

Confirmed gaps:

• Live OpenAPI has no POST /api/v2/hubs.
• Live OpenAPI has no POST /api/v2/widgets; widgets are read-only in v2.
• There are no v2 endpoints for manifest draft creation, manifest activation, API consumer creation, or API key creation.
• There is no /api/v2/policy-scopes endpoint.
• Live type registries currently return empty arrays, so the ops vocabulary still needs manifest activation before events can be accepted.
• Event metadata is exposed in the response schema, but the v2 interaction event create controller currently does not persist submitted metadata.
• Webhook dispatch still uses the hard-coded "clicked" event name.

Architectural Decision

Start with Pattern A: API Consumer Hub for ops-hub, plus a manual or migration-backed Inter-Hub registration. Treat ops-hub as the first VSM hub instance rather than a one-off operational dashboard:

1. Register ops-hub as a domain hub in Inter-Hub.
2. Classify it as the Operations / System 1 hub in hub metadata or manifest metadata, depending on what Inter-Hub currently supports.
3. Activate a HubCapabilityManifest for its operational vocabulary.
4. Create an ApiConsumer and API key for ops-hub.
5. Seed a small set of governed widgets representing operational surfaces.
6. Emit interaction events and annotations from lightweight scripts or a prototype UI.

The first reusable contract to prove is:

Hub identity + VSM function + manifest vocabulary + API consumer + seed widgets + evidence events

The next hubs should be able to follow the same shape with their own vocabularies:

syn-hub / ctl-hub / aud-hub / int-hub / pol-hub / env-hub

Use the separate ops-hub repository for implementation work from now on. inter-hub remains the framework, registry, authentication, manifest, widget, and event substrate. helix-forge remains the architecture/workplan home and keeps these bootstrap artifacts as handoff references until they are ported or superseded by ops-hub repo workplans. Railiance repos remain the desired state and evidence owners for their operational systems.

Initial ops-hub Vocabulary

This vocabulary is deliberately scoped to Operations / System 1. Coordination, control, audit, intelligence, policy, and environment concerns should be represented only where they touch operational evidence; their own hubs will own the broader semantics later.

Suggested manifest values:

Widget Types

[
  "ops-environment",
  "ops-host",
  "ops-cluster",
  "ops-service",
  "ops-service-catalog",
  "ops-endpoint",
  "ops-release",
  "ops-backup-set",
  "ops-secret-set",
  "ops-runbook",
  "ops-incident",
  "ops-readiness-gate",
  "ops-migration-wave",
  "ops-risk"
]

Event Types

[
  "ops-inventory-registered",
  "ops-inventory-updated",
  "ops-service-discovered",
  "ops-health-checked",
  "ops-release-observed",
  "ops-endpoint-verified",
  "ops-backup-verified",
  "ops-restore-tested",
  "ops-runbook-executed",
  "ops-drift-detected",
  "ops-risk-raised",
  "ops-risk-accepted",
  "ops-readiness-gate-updated",
  "ops-migration-gate-passed",
  "ops-migration-gate-failed"
]

Annotation Categories

[
  "ops-drift",
  "ops-service-catalog-gap",
  "ops-backup-gap",
  "ops-security-gap",
  "ops-routing-gap",
  "ops-secret-gap",
  "ops-readiness-blocker",
  "ops-migration-risk",
  "ops-observability-gap",
  "ops-recovery-gap"
]

Policy Scopes

[
  "ops-local",
  "ops-transitional-prod",
  "ops-production",
  "ops-threephoenix",
  "ops-registry",
  "ops-secrets",
  "ops-backup-retention"
]

Initial Operational Inventory

The first ops-hub inventory should cover:

Environment	Role	Current notes
local	Workstation services and development runtime	State Hub and local build/runtime pieces currently live here.
coulombcore	Live transitional production	Public IP 92.205.130.254; hosts current Gitea and hand-built experimental production services.
railiance01	Future production foundation	Public IP 92.205.62.239; first server of the intended ThreePhoenix setup.
threephoenix-prod	Target production topology	Future three-node Railiance production environment.

The first services to model:

• Gitea / container registry
• State Hub and underlying services
• Inter-Hub itself
• PostgreSQL/CNPG services used by Gitea and State Hub
• Ingress/DNS/TLS endpoints for the above
• Backup and restore coverage for each persistent data store
• Ops Bridge connectivity as reachability evidence, not as the catalog itself

The first explicit service-catalog gap:

• There is no central place that answers "what runs where, why, who owns it, how it is reached, and what evidence proves it is healthy." ops-hub should make that question answerable before the ThreePhoenix migration becomes more complicated.

Tasks

T01 — Confirm the VSM hub extension bootstrap path

id: HF-WP-0001-T01
status: done
priority: high
state_hub_task_id: "2587a3b8-3b9b-4948-acaf-1547644e4563"

Confirm whether ops-hub should be registered through the Inter-Hub UI, through a migration, or through new API endpoints. Capture the result as the first repeatable VSM hub bootstrap path, not just as a local workaround for Operations.

Checks:

• Confirm the active Inter-Hub deployment URL and authentication path.
• Confirm whether /Hubs/new, /HubCapabilityManifests, /ApiConsumers, and /ApiKeys are accessible to the operator.
• Confirm whether direct DB migration is acceptable for initial bootstrap.
• Confirm where hub metadata can carry the VSM function (OPS, SYN, CTL, AUD, INT, POL, ENV) and VSM system mapping.
• Record the chosen bootstrap path in this workplan so syn-hub can reuse it.

Done when: there is a concrete, repeatable path to create the ops-hub row, manifest, API consumer, and API key, with enough metadata to classify it as the Operations / System 1 hub.

Output: Confirmed Bootstrap Path section in this workplan.

---

T02 — Register ops-hub in Inter-Hub as the Operations hub

id: HF-WP-0001-T02
status: done
priority: high
state_hub_task_id: "8e9bd9b2-54fc-49a4-8bb8-11c8577be48d"

Create the Hub row:

• name: Ops Hub
• slug: ops-hub
• domain: ops.coulomb.social or another explicit domain chosen by the operator
• hub_kind: domain
• VSM function metadata: OPS
• VSM system metadata: S1
• Hub family metadata: vsm

If Inter-Hub does not yet have explicit fields for VSM function, system, or hub family, store them in manifest metadata and record the missing first-class fields as an Inter-Hub API/model gap.

Done when: ops-hub appears in /Hubs and /api/v2/hub-registry after authentication, and a human can tell that it is the VSM Operations hub.

Ready when: the operator loads the IHUB_OPERATOR_KEY from OpenBao into a trusted shell without echoing it. The key exists in approved custody, but the hub row has not been created yet.

Prepared artifacts:

• wiki/OpsHubBootstrapRunbook.md
• wiki/ops-hub-manifest.draft.json
• wiki/ops-hub-bootstrap.sql

Dependency update on 2026-06-15: a temporary Inter-Hub bootstrap operator key was minted directly in the interhub database after the web admin seed credential failed. OpenBao audit confirms the key was stored at platform/operators/inter-hub/bootstrap-operator, and Inter-Hub DB metadata shows active key prefix 8fab0bef for inter-hub-bootstrap-operator.

Implementation progress on 2026-06-15: added scripts/ops-hub-bootstrap-api.py, which uses wiki/ops-hub-manifest.draft.json and wiki/ops-hub-widgets.seed.json to create/reuse the hub, activate the full manifest, create the runtime ops-hub API consumer/key, seed widgets, and submit the first Gitea readiness event through the supported Inter-Hub API. The helper requires IHUB_OPERATOR_KEY or IHUB_OPERATOR_KEY_FILE and does not print full key values.

Live result on 2026-06-15: the attended helper run used the OpenBao-custodied operator key from a 0600 temp file and created/reused the Inter-Hub hub row. Verified non-secret state:

• Hub id 4f6e4cf7-6a96-4ff2-8a37-08c9f9e405d2
• Slug ops-hub
• VSM metadata hub_family=vsm, vsm_function=OPS, vsm_system=1
• Active manifest id 00aaf90a-8e76-4b0e-892d-33b162862f38

---

T03 — Activate the ops-hub capability manifest

id: HF-WP-0001-T03
status: done
priority: high
state_hub_task_id: "55f5aeed-21c3-4a83-bc78-f90f92c7d597"

Create and activate a HubCapabilityManifest for ops-hub using the vocabulary in this workplan. The manifest should make the VSM classification explicit:

• hub_family: vsm
• vsm_function: OPS
• vsm_system: S1
• scope: operational truth and evidence, not coordination/control/audit ownership

Validation:

• Declared widget types appear in /api/v2/widget-types.
• Declared event types appear in /api/v2/event-types.
• Declared annotation categories appear in /api/v2/annotation-categories.
• Policy scopes are visible in the Inter-Hub registry UI or DB, even though the public v2 API currently lacks /policy-scopes.
• Future VSM hub values can be added by changing manifest vocabulary, not by inventing a different bootstrap mechanism.

Done when: the manifest status is active and no type conflicts remain.

Blocked until: the ops-hub row exists in Inter-Hub and an authenticated operator or migration can create and activate the manifest.

Prepared artifact: wiki/ops-hub-manifest.draft.json.

Live result on 2026-06-15: the full manifest is active in production. Public registry checks found all expected ops vocabulary values:

• 14 of 14 widget types present.
• 15 of 15 event types present.
• 10 of 10 annotation categories present.
• Policy scopes are present in the production database and still lack a dedicated public validation check in this repo's helper.

---

T04 — Create ops-hub API consumer and key

id: HF-WP-0001-T04
status: done
priority: high
state_hub_task_id: "ad08e729-8562-4a02-8bf6-dcdfebe430c8"

Create an ApiConsumer associated with the active ops-hub manifest, then create a static API key with at least:

• framework:read
• hub:ops-hub:read
• hub:ops-hub:write

Store the key only in the operator secret store or local env file, never in Git.

Done when: POST /api/v2/token can exchange the static key for a short-lived access token and GET /api/v2/hub-registry works with that token.

Blocked until: an authenticated operator creates the API key and stores the full static key outside Git. The SQL fallback intentionally creates only the consumer row, not the one-time visible secret.

Progress on 2026-06-15:

• The ops-hub API consumer exists with id f9e595c6-4e1d-41fd-86cb-c1830bd7ec81.
• The bootstrap helper created the display-once runtime key and wrote it to a local 0600 temp file without printing the key.
• POST /api/v2/token returns a short-lived Bearer token for the runtime key with expires_in=3600.
• GET /api/v2/widgets works with the runtime key and returns the 14 ops-hub widgets.

Remaining before closing:

• Store the runtime key from the temporary file in OpenBao at platform/operators/ops-hub/runtime, field OPS_HUB_KEY, then remove the temp file.
• GET /api/v2/hub-registry currently returns HTTP 500 because Inter-Hub decodes COUNT(*) from api_request_log as Int while PostgreSQL returns bigint. Track/fix this under T10 before treating hub-registry as a clean acceptance signal.

Custody attempt on 2026-06-15: copied the generated runtime key into the OpenBao pod as a temporary file and attempted to write it to the approved KV path using the pod token helper. OpenBao denied the request with 403 permission denied while resolving the KV mount through sys/internal/ui/mounts/platform/operators/ops-hub/runtime. The temporary in-pod key file was removed and verified absent. The local 0600 runtime-key file remains because it has not yet been successfully stored in OpenBao.

Current blocker: requires an attended OpenBao root/sudo token handoff, or the operator storing the local runtime key manually through the browser UI, before the temp file can be removed and this task can close.

Completed on 2026-06-19:

• Regenerated the display-once runtime key through scripts/ops-hub-bootstrap-api.py after the earlier 0600 temp file was no longer present.
• Stored the runtime key in OpenBao at platform/operators/ops-hub/runtime, field OPS_HUB_KEY, using an approved operator token. No key values were copied into Git, State Hub, or chat.
• Removed the local runtime-key temp file after successful OpenBao write.
• Verified non-secret acceptance evidence with the custodied runtime key:
  • POST /api/v2/token exchanges the static key for a short-lived Bearer token (expires_in=3600).
  • GET /api/v2/hub-registry returns HTTP 200 with the exchanged token.
  • GET /api/v2/widgets returns all 14 ops-hub widgets with the exchanged token.
• Current runtime key prefix: c1f3ac3a.

---

T05 — Seed first governed ops widgets

id: HF-WP-0001-T05
status: done
priority: high
state_hub_task_id: "d303884d-d1f6-4fd0-a4ec-97afe6162164"

Create initial widgets for the operational surfaces:

• ops-env-local
• ops-env-coulombcore
• ops-env-railiance01
• ops-env-threephoenix-prod
• ops-host-coulombcore
• ops-host-railiance01
• ops-service-catalog
• ops-service-gitea
• ops-service-state-hub
• ops-service-inter-hub
• ops-endpoint-gitea-registry
• ops-readiness-gitea-registry
• ops-readiness-state-hub-cluster-deploy
• ops-migration-coulombcore-to-threephoenix

If Inter-Hub still lacks a widget creation API, seed these through the UI or a migration and record that as an API gap.

Done when: the widgets appear under ops-hub and can accept interaction events and annotations.

Blocked until: ops-hub and its active manifest exist in Inter-Hub.

Prepared artifacts:

• wiki/ops-hub-widgets.seed.json
• wiki/ops-hub-bootstrap.sql

Live result on 2026-06-15: widget creation through POST /api/v2/widgets failed because Inter-Hub decodes COUNT(*) from the type-registry validation query as Int while PostgreSQL returns bigint. The operator-approved SQL fallback was then applied idempotently and created the 14 governed widgets plus 14 initial widget version rows. A runtime-key API smoke check verified all 14 widgets are readable through GET /api/v2/widgets.

---

T06 — Build the first ops inventory artifact

id: HF-WP-0001-T06
status: done
priority: medium
state_hub_task_id: "2a0b2f69-5a3d-433c-9cbd-85fd868b63d8"

Create an ops inventory document in helix-forge that expresses the current state of:

• environments
• hosts
• clusters
• services
• endpoints
• service discovery and service-catalog gaps
• storage and backup coverage
• migration readiness gates

Use wiki/CurrentOperationsSituation.md as the seed background, then turn it into a more structured inventory artifact. Use this as the working model before creating a separate ops-hub repository.

Done when: a human can see the CoulombCore, local, railiance01, and ThreePhoenix relationship, including the current Gitea registry state, without reading multiple repo workplans or relying on shell history.

Output: wiki/OpsHubInventory.md.

---

T07 — Instrument the current Gitea registry work as the first ops-hub signal

id: HF-WP-0001-T07
status: done
priority: medium
state_hub_task_id: "ed3e0396-b16d-40c2-9519-e755ad6241eb"

Use the recently fixed Gitea /v2 route as the first real operational signal.

Suggested event:

{
  "widgetId": "<ops-readiness-gitea-registry-widget-id>",
  "eventType": "ops-endpoint-verified",
  "viewContext": "railiance-apps/workplans/RAIL-AP-WP-0001",
  "metadata": {
    "vsmFunction": "OPS",
    "vsmSystem": "S1",
    "endpoint": "https://gitea.coulomb.social/v2/",
    "expectedStatus": 401,
    "observedHeader": "Docker-Distribution-Api-Version: registry/2.0"
  }
}

Done when: the Gitea registry readiness event is visible in Inter-Hub and traceable back to the Railiance workplan.

Blocked until: the ops-endpoint-gitea-registry widget exists, the ops-endpoint-verified event type is active, and an ops-hub API key is available to the operator.

Live result on 2026-06-15: the first Gitea registry readiness event was inserted by the SQL fallback and is visible through GET /api/v2/interaction-events with the runtime key:

• Event id 4af73b21-75a9-4814-b6df-62083cfda15f
• Event type ops-endpoint-verified
• View context railiance-apps/workplans/RAIL-AP-WP-0001
• Metadata records endpoint https://gitea.coulomb.social/v2/, expected status 401, and observed header Docker-Distribution-Api-Version: registry/2.0

---

T08 — Define the ops-hub readiness gate model for ThreePhoenix migration

id: HF-WP-0001-T08
status: done
priority: medium
state_hub_task_id: "72a58622-c3ac-4765-8026-5c2489af2058"

Define readiness gates that must be green before moving production responsibility from CoulombCore to ThreePhoenix:

• DNS and TLS are codified.
• Service catalog entries exist for the live and target production services.
• Git hosting and container registry are reproducible.
• Persistent data stores have backup and restore evidence.
• Secrets and SOPS/age keys are available through governed operator paths.
• Cluster runtime and platform services are recreated through Railiance repos.
• Rollback path is documented.
• Operator runbooks exist for deploy, restore, rotate, and incident response.

Done when: each gate has an owner repo, evidence requirement, and status.

Output: wiki/OpsHubReadinessGates.md.

---

T09 — Decide whether to create a separate ops-hub repository

id: HF-WP-0001-T09
status: done
priority: medium
state_hub_task_id: "0e5842fd-1d33-4e2a-9701-07f623a2b901"

Decision recorded on 2026-06-06: create and use a separate ops-hub repository. The repo is present locally at /home/worsch/ops-hub and tracks gitea-remote:coulomb/ops-hub.git.

Rationale:

• The operator has provided the repo and decided that ops-hub should live outside helix-forge.
• ops-hub is expected to need implementation assets such as collectors, adapters, scheduled probes, bootstrap smoke tooling, and possibly UI beyond Inter-Hub's generic hub dashboards.
• ops-hub needs its own workplan prefix, release lifecycle, and repository agent instructions once implementation begins.
• Keeping deployable/runtime code out of inter-hub preserves Inter-Hub as the generic hub framework while still allowing ops-hub to be included as a VSM Operations / System 1 extension.

Repository boundary:

• ops-hub: implementation home for the Operations hub extension, including collectors, adapters, scheduled probes, runtime packaging, UI/extensions, tests, and Inter-Hub bootstrap/smoke clients.
• inter-hub: generic framework, extension registry, hub manifests, API consumers, authentication, widgets, event persistence, and bootstrap API.
• helix-forge: product/architecture intent, VSM extension pattern, coordination workplan, and current bootstrap handoff artifacts until they are ported or retired.
• Railiance repos: desired state, deployment, backup/restore evidence, and operational facts for their owned infrastructure and services.

Done: the decision is recorded with rationale and boundary; future implementation should happen in ops-hub.

---

T10 — Inter-Hub API hardening for VSM hub bootstrap

id: HF-WP-0001-T10
status: done
priority: high
target_repo: inter-hub
state_hub_task_id: "7fa54508-7add-4885-8913-12edaadc4d92"

Create or link an inter-hub workplan to make VSM domain hub bootstrapping machine-repeatable.

Recommended Inter-Hub improvements:

1. Add POST /api/v2/hubs and include it in OpenAPI.
2. Add POST /api/v2/widgets and include it in OpenAPI.
3. Add API endpoints for HubCapabilityManifest draft creation, update, and activation.
4. Add a documented place for hub-family metadata such as hub_family, vsm_function, and vsm_system.
5. Add API endpoints for ApiConsumer and API key creation, or a clearly documented admin-only bootstrap command if API key creation remains UI-only.
6. Add /api/v2/policy-scopes to match the policy scope registry already used by manifests.
7. Add distinct OpenAPI request schemas for create requests instead of reusing response schemas.
8. Align docs/new-hub-quickstart.md with the actual live API until the create endpoints exist.
9. Fix Web.Controller.Api.V2.InteractionEvents so manifest-declared event types are actually decoded and enforced.
10. Fix webhook dispatch so it uses the submitted event type instead of the hard-coded "clicked" event name.
11. Decide whether event metadata is part of the v2 create contract; if yes, persist it in the controller and test it.
12. Document the bootstrap recipe as a template for syn-hub, ctl-hub, aud-hub, int-hub, pol-hub, and env-hub.

Done when: the next VSM hub can be created from a script using documented API calls and without direct DB access.

Linked Inter-Hub workplan: inter-hub/workplans/IHUB-WP-0019-vsm-hub-bootstrap-api.md.

Source implementation status as of 2026-06-14:

• IHUB-WP-0019 is finished in the local inter-hub repo, and main is aligned with origin/main.
• The documented smoke path exists at /home/worsch/inter-hub/scripts/ops-hub-bootstrap-smoke.py.
• The ops-hub implementation repo has the handoff track workplans/OPS-WP-0002-interhub-extension-bootstrap.md; its production gate probe is scripts/interhub-gate-probe.py.

Current production gate as of 2026-06-06:

• Do not proceed with manual DB seeding unless the operator explicitly chooses that fallback.
• Wait for https://hub.coulomb.social/api/v2/hubs to return 401 unauthenticated instead of 404.
• Confirm OpenAPI lists /hubs, /hub-capability-manifests, /api-consumers, and /policy-scopes.
• After the gate passes, run the supported bootstrap/smoke path from the relevant inter-hub or ops-hub tooling with IHUB_BASE and an operator key.

Production gate recheck on 2026-06-14:

• ops-hub/scripts/interhub-gate-probe.py against https://hub.coulomb.social still fails.
• GET /api/v2/hubs returns 404, not 401.
• The live OpenAPI still omits /hubs, /hub-capability-manifests, /api-consumers, and /policy-scopes.
• Result: source-side API hardening is complete, but HF-WP-0001 remains gated on deployment of the current Inter-Hub API or an explicit operator decision to use the manual SQL fallback.

Production gate recheck after Inter-Hub deployment on 2026-06-14:

• https://hub.coulomb.social/api/v2/hubs now returns 200 with an empty paginated hub list instead of 404.
• The live OpenAPI now lists /hubs, /hub-capability-manifests, /api-consumers, and /policy-scopes.
• OpenAPI shows public reads for GET /hubs and GET /policy-scopes, and authenticated BearerAuth writes for hub, manifest, API consumer/key, widget, and interaction-event creation.
• ops-hub/scripts/interhub-gate-probe.py still exits nonzero because it expects unauthenticated GET /api/v2/hubs to return 401; that expectation is stale relative to the deployed public-read/authenticated-write contract.
• Result: the Inter-Hub bootstrap API hardening and production deployment gate are complete. HF-WP-0001 now waits on an attended bootstrap run with IHUB_OPERATOR_KEY or equivalent authenticated operator session, not on manual SQL fallback or API deployment.

Live bootstrap follow-up on 2026-06-15:

• Hub, manifest, API consumer, and runtime key creation work through the live API.
• POST /api/v2/widgets fails with UnexpectedColumnTypeStatementError 0 23 20 from SELECT COUNT(*) FROM widget_type_registry ...; the query result needs to be decoded as Int64 or cast to int.
• GET /api/v2/hub-registry with either the static runtime key or a short-lived token fails with the same class of error from SELECT COUNT(*) FROM api_request_log ....
• The operator-approved SQL fallback was used for seed widgets and the first event so HF-WP-0001 could keep moving, but the next VSM hub is not yet fully scriptable without direct DB access.

Source fix on 2026-06-15: patched Inter-Hub Application/Helper/TypeRegistry.hs and Application/Helper/ApiRateLimit.hs to cast the affected COUNT(*) queries to int, committed as 5101eb5 Fix API count decoding, and pushed to origin/main.

Deployment status on 2026-06-15:

• Local git diff --check passed in inter-hub.
• nix develop ... scripts/compile-check could not run because the checkout lacks .devenv/root and plain nix develop cannot determine the current directory in this shell.
• A local nix build .#docker was attempted, but after more than 20 minutes it was still compiling dependencies and could not be used for deployment from this session because registry/deploy credentials are not available here.
• Railiance01 release inspection through railiance-apps shows the live hub.coulomb.social Deployment still uses 92.205.130.254:32166/coulomb/inter-hub:790b5e5, while Helm values report image.tag: 11ff61c; in either view, the pushed 5101eb5 fix is not live.

Railiance deploy-surface recheck on 2026-06-15:

• railiance-apps commit c7d49d3 adds local Makefile targets for INTER_HUB_IMAGE_TAG=<sha> dry-run, deploy, status, release-info, logs, and smoke checks.
• INTER_HUB_IMAGE_TAG=5101eb5 make inter-hub-dry-run renders gitea.coulomb.social/coulomb/inter-hub:5101eb5 and preserves the live immutable selector label app=inter-hub.
• A Helm --dry-run=server upgrade against Railiance01 with tag 5101eb5 succeeds, so the chart shape is acceptable to the cluster.
• The Gitea registry currently returns MANIFEST_UNKNOWN for coulomb/inter-hub:5101eb5, so the deployment must not be run until the Inter-Hub image is built and published.
• make inter-hub-smoke is stale: it expects public GET /api/v2/hubs to return 401 and checks /openapi.json, while the live contract is 200 for public hub discovery, 401 for protected resources such as /api/v2/widgets, and OpenAPI at /api/v2/openapi.json.
• railiance-apps still has only the pull-request manifest dry-run workflow; it does not yet provide a workflow_dispatch production deploy trigger.

Railiance deploy-surface follow-up review on 2026-06-15:

• railiance-apps commit 6abf753 adds RAILIANCE-WP-0011 and addresses the requested hardening: OCI image preflight, explicit production dry-runs, inter-hub-server-dry-run, current public-read/authenticated-write smoke contract, and .gitea/workflows/inter-hub-production-deploy.yaml with workflow_dispatch.
• make inter-hub-smoke passes against live https://hub.coulomb.social: public GET /api/v2/hubs returns discovery JSON, protected widgets and hub-registry routes return 401 invalid_api_key without a key, and /api/v2/openapi.json lists the expected v2 resources.
• INTER_HUB_IMAGE_TAG=5101eb5 make check-inter-hub-image fails before Helm with manifest unknown, which is the desired safe behavior while the image is absent.
• Inter-Hub confirmed via State Hub message 269d0ace-5b8e-4fec-a1d0-11a52ad23cc5 that the remaining deploy-side blocker is to build and publish gitea.coulomb.social/coulomb/inter-hub:5101eb5 or another tag containing the same COUNT(*) decode fix.

Current blocker: publish a Gitea registry image for Inter-Hub commit 5101eb5 or an equivalent fix tag, then deploy it through the approved Railiance path and rerun the authenticated widget-create and hub-registry smoke checks. Railiance-apps no longer appears to be the blocking surface.

Completed on 2026-06-19:

• Production Inter-Hub now runs image gitea.coulomb.social/coulomb/inter-hub:eed4322, which is ahead of the 5101eb5 COUNT-decode fix commit.
• Authenticated GET /api/v2/hub-registry returns HTTP 200 with the bootstrap operator key and with a runtime key exchanged through POST /api/v2/token.
• Authenticated POST /api/v2/widgets succeeds through the public API; a smoke widget was created and deleted without using direct DB access.
• Result: the next VSM hub can bootstrap through the documented v2 API surface without the earlier COUNT(*) decode failure class blocking widget creation or hub-registry reads.

Initial Acceptance Criteria

This workplan is complete when:

1. ops-hub is registered in Inter-Hub as the VSM Operations / System 1 hub.
2. Its capability manifest is active.
3. It has an API consumer and key.
4. Initial ops widgets exist for environments, services, readiness gates, and migration waves.
5. At least one real operational event has been submitted.
6. The CoulombCore-to-ThreePhoenix readiness model is documented.
7. A decision has been made whether to create a separate ops-hub repository.
8. Inter-Hub bootstrap API gaps are either fixed or tracked in an Inter-Hub workplan.
9. The bootstrap path is reusable enough that syn-hub can be created next without rediscovering the whole process.

Implementation Log

2026-05-19 — Live bootstrap access check

Attempted to execute the prepared bootstrap path.

Findings:

• hub.coulomb.social resolves to Railiance01 (92.205.62.239).
• The current workstation kubeconfig has only the default context and points at CoulombCore (92.205.130.254).
• ~/.kube/config-hosteurope is missing, so the documented Railiance01 kubeconfig path is not available locally.
• kubectl against the current context has no inter-hub deployment because it is the CoulombCore cluster.
• SSH key auth to root@92.205.62.239, worsch@92.205.62.239, and ubuntu@92.205.62.239 was denied.

Result:

• Live ops-hub bootstrap was not applied from this session.
• wiki/ops-hub-bootstrap.sql was strengthened to set first-class VSM hub metadata when the target Inter-Hub schema supports it.
• wiki/OpsHubBootstrapRunbook.md now records the exact SQL fallback command to run from a Railiance01-capable session.

Next required operator action:

• Restore/provide Railiance01 Kubernetes access, or run wiki/ops-hub-bootstrap.sql from a host that already has access to the net-kingdom-pg-1 pod in the databases namespace.

2026-06-06 — ops-hub repo decision

The operator decided that ops-hub should live in its own repository and be included as an extension of Inter-Hub. The repo is available locally at /home/worsch/ops-hub with remote gitea-remote:coulomb/ops-hub.git.

Result:

• HF-WP-0001 T09 is closed as done.
• ops-hub implementation work moves to the ops-hub repo.
• helix-forge keeps architecture, vocabulary, readiness, and bootstrap handoff references.
• Inter-Hub remains the generic extension framework and still owns the bootstrap API hardening tracked by T10.
• Live bootstrap remains gated on current Inter-Hub production API availability unless the operator explicitly chooses the manual SQL fallback.

2026-06-14 — production API gate recheck

Rechecked the live Inter-Hub bootstrap gate from the dedicated ops-hub implementation repo:

python3 scripts/interhub-gate-probe.py

Result:

• https://hub.coulomb.social/api/v2/hubs returns 404.
• https://hub.coulomb.social/api/v2/openapi.json returns 200, but the required bootstrap paths are absent.
• Missing paths: /hubs, /hub-capability-manifests, /api-consumers, and /policy-scopes.

Interpretation:

• Inter-Hub source work for IHUB-WP-0019 is complete and pushed to origin/main.
• Production is still serving an older API surface, so the preferred ops-hub bootstrap path cannot run yet.
• HF-WP-0001 T10 is moved to wait until the production API is deployed or the operator explicitly chooses the manual SQL fallback.

2026-06-14 — production API gate opened

Rechecked the live Inter-Hub API after deployment moved forward:

• GET https://hub.coulomb.social/api/v2/hubs now returns 200 with {"data":[],"meta":{"page":1,"per_page":50,"total":0}}.
• The live OpenAPI includes /hubs, /hub-capability-manifests, /api-consumers, and /policy-scopes.
• The deployed contract exposes public reads for hub and policy-scope lists, while creation and mutation paths require BearerAuth.
• IHUB_OPERATOR_KEY was not present in this Codex shell, so the authenticated bootstrap/smoke script was not run.

Result:

• T10 is closed as done.
• T02 is ready to run once the operator loads the OpenBao-custodied IHUB_OPERATOR_KEY into a trusted shell.
• T03-T05 remain wait until the ops-hub row and active manifest exist.
• The next action is to run the supported bootstrap path with IHUB_BASE=https://hub.coulomb.social and the operator key, while preserving the one-time API key secret outside Git and State Hub.

2026-06-15 — bootstrap operator key created and custodied

The seeded Inter-Hub web admin credential was not usable on production, so the operator explicitly chose a controlled database bootstrap for the first temporary API key.

Non-secret evidence:

• net-kingdom-pg-1 is healthy and contains the interhub database.
• api_consumers initially had no rows.
• A static inter-hub-bootstrap-operator key now exists with prefix 8fab0bef.
• OpenBao audit shows successful create/read activity for platform/data/operators/inter-hub/bootstrap-operator.

The full key value was not printed into Git, State Hub, or chat. It was stored manually by the operator in OpenBao. This unblocks the next attended step: retrieve the key from OpenBao into a trusted shell and create the ops-hub hub row through the supported Inter-Hub API.

2026-06-15 — full API bootstrap helper prepared

Added scripts/ops-hub-bootstrap-api.py as the HF-WP-0001 production helper. Unlike the source-side Inter-Hub smoke script, it consumes the full HelixForge manifest and widget seed artifacts. It supports IHUB_OPERATOR_KEY_FILE and OPS_HUB_KEY_FILE so keys can be passed through 0600 temp files rather than shell history or chat.

Validation performed:

• python3 -m py_compile scripts/ops-hub-bootstrap-api.py
• python3 scripts/ops-hub-bootstrap-api.py --help
• JSON seed parsing for the manifest and 14 widget seeds

2026-06-15 — attended Ops Hub bootstrap applied

The operator provided the Inter-Hub bootstrap key through a local temp file. The helper used that key without printing it and successfully created/reused:

• ops-hub hub row with VSM metadata vsm/OPS/1.
• Active capability manifest 00aaf90a-8e76-4b0e-892d-33b162862f38.
• Runtime API consumer f9e595c6-4e1d-41fd-86cb-c1830bd7ec81.
• Display-once runtime API key, written only to a 0600 local temp file.

The helper then exposed a live Inter-Hub bug while creating widgets:

UnexpectedColumnTypeStatementError 0 23 20
SELECT COUNT(*) FROM widget_type_registry WHERE name = $1 AND status = 'active'

Because PostgreSQL returns COUNT(*) as bigint, the Inter-Hub validation code must decode this as Int64 or cast the query result. The same bug class also affects authenticated GET /api/v2/hub-registry through the api_request_log rate-limit query.

To keep the approved bootstrap moving, the idempotent SQL fallback was applied for the remaining data. It created:

• 14 governed ops-hub widgets.
• 14 initial widget version rows.
• The first ops-endpoint-verified Gitea registry readiness event.

Validation:

• All 14 widget types, 15 event types, and 10 annotation categories from the manifest are present in public registries.
• GET /api/v2/widgets with the runtime key returns all 14 ops-hub widgets.
• POST /api/v2/token returns a short-lived Bearer token for the runtime key.
• GET /api/v2/interaction-events with the runtime key returns the Gitea registry readiness event and metadata.

Remaining operator action:

• Store the generated runtime key from the local temp file in OpenBao at platform/operators/ops-hub/runtime, field OPS_HUB_KEY, then remove the temp file.
• Track/fix the Inter-Hub COUNT(*) decode issues before declaring the next VSM hub fully scriptable through the public API.

2026-06-19 — HF-WP-0001 closed out

Closed the remaining bootstrap custody and production verification gaps:

• Stored the ops-hub runtime key in OpenBao at platform/operators/ops-hub/runtime and removed the local temp file.
• Verified runtime-key token exchange, hub-registry reads, and widget listing through the public Inter-Hub API.
• Confirmed production Inter-Hub image eed4322 includes the deployed COUNT-decode fix path; authenticated widget creation and hub-registry reads now succeed without SQL fallback.

No API keys, OpenBao tokens, or secret values were copied into Git, State Hub, chat, or workplan text.

Notes

ops-hub should complement State Hub during the transition:

• State Hub continues to track workstreams, decisions, and progress events.
• ops-hub tracks operational reality and readiness evidence.
• syn-hub, ctl-hub, and aud-hub can later absorb coordination, control, and evidence responsibilities once the broader hub constellation is established.