coulomb/helix-forge
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
02f11f8a9b85520da047daf1458cfbab26896e68
helix-forge/workplans/HF-WP-0001-establish-ops-hub-first-extension.md
tegwick 023eb20a14 Finish HF-WP-0001: custodied runtime key and production API verification 
Close T04 after storing the ops-hub runtime key in OpenBao and verifying token
exchange plus hub-registry access. Close T10 after confirming production
Inter-Hub image eed4322 fixes COUNT decode failures for widget creation and
hub-registry reads.
2026-06-19 19:11:28 +02:00

1162 lines
45 KiB
Markdown
Raw Blame History

---
id: HF-WP-0001
type: workplan
title: "Establish ops-hub as the First VSM Inter-Hub Extension"
domain: helix_forge
repo: helix-forge
status: finished
owner: worsch
created: "2026-05-16"
updated: "2026-06-19"
planning_priority: high
planning_order: 1
related_repos:
- inter-hub
- ops-hub
- railiance-infra
- railiance-cluster
- railiance-platform
- railiance-apps
state_hub_workstream_id: "48d91935-197e-4ad4-be07-7bbcd535847c"
---
# Establish ops-hub as the First VSM Inter-Hub Extension
## Goal
Use Inter-Hub as the generic hub framework and establish `ops-hub` as the
first VSM-oriented domain hub extension: the Operations / System 1 hub.
`ops-hub` should professionalize Railiance operations while the current
CoulombCore environment transitions toward the future ThreePhoenix production
setup. Just as importantly, it should prove the repeatable extension pattern
for the later VSM hubs:
- `ops-hub` — Operations and Activities / System 1
- `syn-hub` — Synchronization and Coordination / System 2
- `ctl-hub` — Internal Control and Regulation / System 3
- `aud-hub` — Audit and Monitoring / System 3*
- `int-hub` — Intelligence and Adaptation / System 4
- `pol-hub` — Policy and Identity / System 5
- `env-hub` — Boundary and Environment
The first increment should not replace State Hub. It establishes the
operational model, the VSM hub vocabulary, and the smallest governed
integration with Inter-Hub. As of 2026-06-06, `ops-hub` has its own repository
at `/home/worsch/ops-hub` with remote `gitea-remote:coulomb/ops-hub.git`; use
that repo for implementation from now on while treating it as an extension of
Inter-Hub rather than a fork of the generic hub framework.
## VSM Hub Extension Strategy
Inter-Hub is the framework. HelixForge should extend it with specific hubs that
map to the viable-system functions already named in `INTENT.md`.
| Hub | VSM function | First responsibility |
|---|---|---|
| `ops-hub` | Operations and Activities / System 1 | Operational truth surface for environments, hosts, clusters, services, endpoints, releases, backups, incidents, risks, runbooks, and migration waves. |
| `syn-hub` | Synchronization and Coordination / System 2 | Coordination between operational units, repos, workstreams, and service handoffs so local actions do not conflict. |
| `ctl-hub` | Internal Control and Regulation / System 3 | Current-state control, resource constraints, readiness gates, priorities, and operational governance. |
| `aud-hub` | Audit and Monitoring / System 3* | Independent evidence, checks, observations, drift detection, and verification trails. |
| `int-hub` | Intelligence and Adaptation / System 4 | Future sensing, migration analysis, forecasting, recommendations, and adaptation planning. |
| `pol-hub` | Policy and Identity / System 5 | Identity, values, ultimate constraints, policy decisions, and acceptable operating posture. |
| `env-hub` | Boundary and Environment | External actors, surfaces, endpoints, users, markets, partner systems, and environmental signals. |
This workplan starts only with `ops-hub`, but every bootstrap choice should be
judged by whether it can become the template for the next hub.
## Context
`wiki/CurrentOperationsSituation.md` captures the immediate operational
background as of 2026-05-15. The short version: the operational platform is
real, useful, and already carrying production-like responsibilities, but its
state is spread across live systems, repo workplans, shell knowledge, and
operator memory. There is no central service catalog or operational registry
yet.
Current operational reality:
- `coulombcore` / `92.205.130.254` is the live production-like server. It runs
the current Gitea deployment and other hands-on experimental services.
- The local workstation still hosts important services such as State Hub and
local build/runtime pieces.
- `railiance01` / `92.205.62.239` is the first server of the intended future
ThreePhoenix production environment.
- The Railiance repo stack already separates operational responsibility:
`railiance-infra` (S1), `railiance-cluster` (S2), `railiance-platform` (S3),
`railiance-enablement` (S4), and `railiance-apps` (S5).
- Gitea is live on the CoulombCore Kubernetes cluster as Helm release `gitea`
in namespace `default`, exposed through NodePort `32166`, with its database
in namespace `databases` and shared data on PVC `default/gitea-shared-storage`.
- The Gitea OCI registry route at `https://gitea.coulomb.social/v2/` now
returns the expected registry auth challenge, but publishing still needs to
be proven with encrypted Helm values, a package token, `docker login`, push,
and pull.
- Ops Bridge can help reveal which servers are connected and reachable, but it
is not itself a full operational service catalog.
`ops-hub` should become the operational truth surface across those realities:
environments, hosts, clusters, services, releases, endpoints, backups,
readiness gates, incidents, risks, service discovery, and migration waves.
## Inter-Hub API Findings
Checked live and local Inter-Hub evidence on 2026-05-16.
Live API:
- `https://hub.coulomb.social/api/v2/openapi.json` is available.
- `https://hub.coulomb.social/api/v2/docs` is available.
- Public UI route `https://hub.coulomb.social/Hubs` redirects to
`/NewSession`, so hub creation is currently an authenticated UI flow.
Live OpenAPI paths:
- `/widgets`, `/widgets/{id}`
- `/interaction-events`
- `/annotations`
- `/requirement-candidates`, `/requirement-candidates/{id}`
- `/decision-records`, `/decision-records/{id}`
- `/deployment-records`, `/deployment-records/{id}`
- `/outcome-signals`, `/outcome-signals/{id}`
- `/widget-types`, `/event-types`, `/annotation-categories`
- `/hub-registry`, `/hub-registry/{hubId}`
- `/widget-patterns`, `/widget-patterns/{id}`, `/widget-patterns/{id}/adopt`
- `/token`
Useful local Inter-Hub docs:
- `inter-hub/docs/domain-hub-extension-guide.md`
- `inter-hub/docs/new-hub-quickstart.md`
- `inter-hub/contracts/extensions/hub-capability-manifest-v1.md`
- `inter-hub/contracts/functional/interaction-reporting-v1.md`
Assessment:
- Inter-Hub provides enough guidance to start `ops-hub` as an API consumer
pattern or as a manually registered domain hub.
- Inter-Hub does not yet provide enough API surface to fully automate first hub
bootstrap. Hub creation, capability manifest creation/activation, API
consumer creation, API key issuance, and widget creation are primarily UI or
internal-controller workflows.
- The quickstart mentions `POST /api/v2/hubs` and `POST /api/v2/widgets`, but
the live OpenAPI and local routes do not expose those create endpoints. Treat
the quickstart as aspirational for bootstrap automation until Inter-Hub is
hardened.
## Confirmed Bootstrap Path
Checked against the live API and local Inter-Hub source on 2026-05-16.
Decision:
- Bootstrap `ops-hub` through the authenticated Inter-Hub admin UI where
possible, with a migration-backed fallback when a repeatable bootstrap is
needed before the public API is hardened.
- Treat Pattern A, API Consumer Hub, as the first implementation pattern.
- Store VSM classification in first-class hub metadata where the target
Inter-Hub schema has `hub_family`, `vsm_function`, and `vsm_system`; keep the
same values in the manifest capability description for compatibility with
older deployments and API consumers.
- Use `hub_kind = domain` for `ops-hub`.
- Record missing first-class VSM metadata fields and create endpoints as
Inter-Hub hardening work under T10.
Confirmed current support:
- Live `/Hubs` redirects to `/NewSession`, so hub creation is an authenticated
UI flow.
- Local `HubsController` supports creating and editing hub rows through the UI.
- Local `HubCapabilityManifestsController` supports draft creation, JSON-array
vocabulary editing, activation, and registry upsert.
- Local `ApiConsumersController` and `ApiKeysController` support authenticated
UI creation of consumers and static keys.
- Local `/api/v2/interaction-events` supports `POST` and validates event types.
Confirmed gaps:
- Live OpenAPI has no `POST /api/v2/hubs`.
- Live OpenAPI has no `POST /api/v2/widgets`; widgets are read-only in v2.
- There are no v2 endpoints for manifest draft creation, manifest activation,
API consumer creation, or API key creation.
- There is no `/api/v2/policy-scopes` endpoint.
- Live type registries currently return empty arrays, so the ops vocabulary
still needs manifest activation before events can be accepted.
- Event metadata is exposed in the response schema, but the v2 interaction
event create controller currently does not persist submitted metadata.
- Webhook dispatch still uses the hard-coded `"clicked"` event name.
## Architectural Decision
Start with **Pattern A: API Consumer Hub** for `ops-hub`, plus a manual or
migration-backed Inter-Hub registration. Treat `ops-hub` as the first VSM hub
instance rather than a one-off operational dashboard:
1. Register `ops-hub` as a domain hub in Inter-Hub.
2. Classify it as the Operations / System 1 hub in hub metadata or manifest
metadata, depending on what Inter-Hub currently supports.
3. Activate a `HubCapabilityManifest` for its operational vocabulary.
4. Create an `ApiConsumer` and API key for `ops-hub`.
5. Seed a small set of governed widgets representing operational surfaces.
6. Emit interaction events and annotations from lightweight scripts or a
prototype UI.
The first reusable contract to prove is:
```text
Hub identity + VSM function + manifest vocabulary + API consumer + seed widgets + evidence events
```
The next hubs should be able to follow the same shape with their own
vocabularies:
```text
syn-hub / ctl-hub / aud-hub / int-hub / pol-hub / env-hub
```
Use the separate `ops-hub` repository for implementation work from now on.
`inter-hub` remains the framework, registry, authentication, manifest, widget,
and event substrate. `helix-forge` remains the architecture/workplan home and
keeps these bootstrap artifacts as handoff references until they are ported or
superseded by `ops-hub` repo workplans. Railiance repos remain the desired
state and evidence owners for their operational systems.
## Initial ops-hub Vocabulary
This vocabulary is deliberately scoped to Operations / System 1. Coordination,
control, audit, intelligence, policy, and environment concerns should be
represented only where they touch operational evidence; their own hubs will
own the broader semantics later.
Suggested manifest values:
### Widget Types
```json
[
"ops-environment",
"ops-host",
"ops-cluster",
"ops-service",
"ops-service-catalog",
"ops-endpoint",
"ops-release",
"ops-backup-set",
"ops-secret-set",
"ops-runbook",
"ops-incident",
"ops-readiness-gate",
"ops-migration-wave",
"ops-risk"
]
```
### Event Types
```json
[
"ops-inventory-registered",
"ops-inventory-updated",
"ops-service-discovered",
"ops-health-checked",
"ops-release-observed",
"ops-endpoint-verified",
"ops-backup-verified",
"ops-restore-tested",
"ops-runbook-executed",
"ops-drift-detected",
"ops-risk-raised",
"ops-risk-accepted",
"ops-readiness-gate-updated",
"ops-migration-gate-passed",
"ops-migration-gate-failed"
]
```
### Annotation Categories
```json
[
"ops-drift",
"ops-service-catalog-gap",
"ops-backup-gap",
"ops-security-gap",
"ops-routing-gap",
"ops-secret-gap",
"ops-readiness-blocker",
"ops-migration-risk",
"ops-observability-gap",
"ops-recovery-gap"
]
```
### Policy Scopes
```json
[
"ops-local",
"ops-transitional-prod",
"ops-production",
"ops-threephoenix",
"ops-registry",
"ops-secrets",
"ops-backup-retention"
]
```
## Initial Operational Inventory
The first ops-hub inventory should cover:
| Environment | Role | Current notes |
|---|---|---|
| `local` | Workstation services and development runtime | State Hub and local build/runtime pieces currently live here. |
| `coulombcore` | Live transitional production | Public IP `92.205.130.254`; hosts current Gitea and hand-built experimental production services. |
| `railiance01` | Future production foundation | Public IP `92.205.62.239`; first server of the intended ThreePhoenix setup. |
| `threephoenix-prod` | Target production topology | Future three-node Railiance production environment. |
The first services to model:
- Gitea / container registry
- State Hub and underlying services
- Inter-Hub itself
- PostgreSQL/CNPG services used by Gitea and State Hub
- Ingress/DNS/TLS endpoints for the above
- Backup and restore coverage for each persistent data store
- Ops Bridge connectivity as reachability evidence, not as the catalog itself
The first explicit service-catalog gap:
- There is no central place that answers "what runs where, why, who owns it,
how it is reached, and what evidence proves it is healthy." `ops-hub` should
make that question answerable before the ThreePhoenix migration becomes more
complicated.
## Tasks
### T01 — Confirm the VSM hub extension bootstrap path
```task
id: HF-WP-0001-T01
status: done
priority: high
state_hub_task_id: "2587a3b8-3b9b-4948-acaf-1547644e4563"
```
Confirm whether `ops-hub` should be registered through the Inter-Hub UI,
through a migration, or through new API endpoints. Capture the result as the
first repeatable VSM hub bootstrap path, not just as a local workaround for
Operations.
Checks:
- Confirm the active Inter-Hub deployment URL and authentication path.
- Confirm whether `/Hubs/new`, `/HubCapabilityManifests`, `/ApiConsumers`, and
`/ApiKeys` are accessible to the operator.
- Confirm whether direct DB migration is acceptable for initial bootstrap.
- Confirm where hub metadata can carry the VSM function (`OPS`, `SYN`, `CTL`,
`AUD`, `INT`, `POL`, `ENV`) and VSM system mapping.
- Record the chosen bootstrap path in this workplan so `syn-hub` can reuse it.
Done when: there is a concrete, repeatable path to create the `ops-hub` row,
manifest, API consumer, and API key, with enough metadata to classify it as the
Operations / System 1 hub.
Output: `Confirmed Bootstrap Path` section in this workplan.
---
### T02 — Register ops-hub in Inter-Hub as the Operations hub
```task
id: HF-WP-0001-T02
status: done
priority: high
state_hub_task_id: "8e9bd9b2-54fc-49a4-8bb8-11c8577be48d"
```
Create the Hub row:
- `name`: `Ops Hub`
- `slug`: `ops-hub`
- `domain`: `ops.coulomb.social` or another explicit domain chosen by the
operator
- `hub_kind`: `domain`
- VSM function metadata: `OPS`
- VSM system metadata: `S1`
- Hub family metadata: `vsm`
If Inter-Hub does not yet have explicit fields for VSM function, system, or hub
family, store them in manifest metadata and record the missing first-class
fields as an Inter-Hub API/model gap.
Done when: `ops-hub` appears in `/Hubs` and `/api/v2/hub-registry` after
authentication, and a human can tell that it is the VSM Operations hub.
Ready when: the operator loads the `IHUB_OPERATOR_KEY` from OpenBao into a
trusted shell without echoing it. The key exists in approved custody, but the
hub row has not been created yet.
Prepared artifacts:
- `wiki/OpsHubBootstrapRunbook.md`
- `wiki/ops-hub-manifest.draft.json`
- `wiki/ops-hub-bootstrap.sql`
Dependency update on 2026-06-15: a temporary Inter-Hub bootstrap operator key
was minted directly in the `interhub` database after the web admin seed
credential failed. OpenBao audit confirms the key was stored at
`platform/operators/inter-hub/bootstrap-operator`, and Inter-Hub DB metadata
shows active key prefix `8fab0bef` for `inter-hub-bootstrap-operator`.
Implementation progress on 2026-06-15: added
`scripts/ops-hub-bootstrap-api.py`, which uses
`wiki/ops-hub-manifest.draft.json` and `wiki/ops-hub-widgets.seed.json` to
create/reuse the hub, activate the full manifest, create the runtime
`ops-hub` API consumer/key, seed widgets, and submit the first Gitea readiness
event through the supported Inter-Hub API. The helper requires
`IHUB_OPERATOR_KEY` or `IHUB_OPERATOR_KEY_FILE` and does not print full key
values.
Live result on 2026-06-15: the attended helper run used the OpenBao-custodied
operator key from a 0600 temp file and created/reused the Inter-Hub hub row.
Verified non-secret state:
- Hub id `4f6e4cf7-6a96-4ff2-8a37-08c9f9e405d2`
- Slug `ops-hub`
- VSM metadata `hub_family=vsm`, `vsm_function=OPS`, `vsm_system=1`
- Active manifest id `00aaf90a-8e76-4b0e-892d-33b162862f38`
---
### T03 — Activate the ops-hub capability manifest
```task
id: HF-WP-0001-T03
status: done
priority: high
state_hub_task_id: "55f5aeed-21c3-4a83-bc78-f90f92c7d597"
```
Create and activate a `HubCapabilityManifest` for `ops-hub` using the
vocabulary in this workplan. The manifest should make the VSM classification
explicit:
- `hub_family`: `vsm`
- `vsm_function`: `OPS`
- `vsm_system`: `S1`
- `scope`: operational truth and evidence, not coordination/control/audit
ownership
Validation:
- Declared widget types appear in `/api/v2/widget-types`.
- Declared event types appear in `/api/v2/event-types`.
- Declared annotation categories appear in `/api/v2/annotation-categories`.
- Policy scopes are visible in the Inter-Hub registry UI or DB, even though the
public v2 API currently lacks `/policy-scopes`.
- Future VSM hub values can be added by changing manifest vocabulary, not by
inventing a different bootstrap mechanism.
Done when: the manifest status is `active` and no type conflicts remain.
Blocked until: the `ops-hub` row exists in Inter-Hub and an authenticated
operator or migration can create and activate the manifest.
Prepared artifact: `wiki/ops-hub-manifest.draft.json`.
Live result on 2026-06-15: the full manifest is active in production. Public
registry checks found all expected ops vocabulary values:
- 14 of 14 widget types present.
- 15 of 15 event types present.
- 10 of 10 annotation categories present.
- Policy scopes are present in the production database and still lack a
dedicated public validation check in this repo's helper.
---
### T04 — Create ops-hub API consumer and key
```task
id: HF-WP-0001-T04
status: done
priority: high
state_hub_task_id: "ad08e729-8562-4a02-8bf6-dcdfebe430c8"
```
Create an `ApiConsumer` associated with the active `ops-hub` manifest, then
create a static API key with at least:
- `framework:read`
- `hub:ops-hub:read`
- `hub:ops-hub:write`
Store the key only in the operator secret store or local env file, never in Git.
Done when: `POST /api/v2/token` can exchange the static key for a short-lived
access token and `GET /api/v2/hub-registry` works with that token.
Blocked until: an authenticated operator creates the API key and stores the
full static key outside Git. The SQL fallback intentionally creates only the
consumer row, not the one-time visible secret.
Progress on 2026-06-15:
- The `ops-hub` API consumer exists with id
`f9e595c6-4e1d-41fd-86cb-c1830bd7ec81`.
- The bootstrap helper created the display-once runtime key and wrote it to a
local 0600 temp file without printing the key.
- `POST /api/v2/token` returns a short-lived Bearer token for the runtime key
with `expires_in=3600`.
- `GET /api/v2/widgets` works with the runtime key and returns the 14
`ops-hub` widgets.
Remaining before closing:
- Store the runtime key from the temporary file in OpenBao at
`platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, then remove the
temp file.
- `GET /api/v2/hub-registry` currently returns HTTP 500 because Inter-Hub
decodes `COUNT(*)` from `api_request_log` as `Int` while PostgreSQL returns
`bigint`. Track/fix this under T10 before treating hub-registry as a clean
acceptance signal.
Custody attempt on 2026-06-15: copied the generated runtime key into the
OpenBao pod as a temporary file and attempted to write it to the approved KV
path using the pod token helper. OpenBao denied the request with `403
permission denied` while resolving the KV mount through
`sys/internal/ui/mounts/platform/operators/ops-hub/runtime`. The temporary
in-pod key file was removed and verified absent. The local 0600 runtime-key
file remains because it has not yet been successfully stored in OpenBao.
Current blocker: requires an attended OpenBao root/sudo token handoff, or the
operator storing the local runtime key manually through the browser UI, before
the temp file can be removed and this task can close.
Completed on 2026-06-19:
- Regenerated the display-once runtime key through
`scripts/ops-hub-bootstrap-api.py` after the earlier 0600 temp file was no
longer present.
- Stored the runtime key in OpenBao at
`platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, using an approved
operator token. No key values were copied into Git, State Hub, or chat.
- Removed the local runtime-key temp file after successful OpenBao write.
- Verified non-secret acceptance evidence with the custodied runtime key:
- `POST /api/v2/token` exchanges the static key for a short-lived Bearer
token (`expires_in=3600`).
- `GET /api/v2/hub-registry` returns HTTP `200` with the exchanged token.
- `GET /api/v2/widgets` returns all 14 `ops-hub` widgets with the exchanged
token.
- Current runtime key prefix: `c1f3ac3a`.
---
### T05 — Seed first governed ops widgets
```task
id: HF-WP-0001-T05
status: done
priority: high
state_hub_task_id: "d303884d-d1f6-4fd0-a4ec-97afe6162164"
```
Create initial widgets for the operational surfaces:
- `ops-env-local`
- `ops-env-coulombcore`
- `ops-env-railiance01`
- `ops-env-threephoenix-prod`
- `ops-host-coulombcore`
- `ops-host-railiance01`
- `ops-service-catalog`
- `ops-service-gitea`
- `ops-service-state-hub`
- `ops-service-inter-hub`
- `ops-endpoint-gitea-registry`
- `ops-readiness-gitea-registry`
- `ops-readiness-state-hub-cluster-deploy`
- `ops-migration-coulombcore-to-threephoenix`
If Inter-Hub still lacks a widget creation API, seed these through the UI or a
migration and record that as an API gap.
Done when: the widgets appear under `ops-hub` and can accept interaction events
and annotations.
Blocked until: `ops-hub` and its active manifest exist in Inter-Hub.
Prepared artifacts:
- `wiki/ops-hub-widgets.seed.json`
- `wiki/ops-hub-bootstrap.sql`
Live result on 2026-06-15: widget creation through `POST /api/v2/widgets`
failed because Inter-Hub decodes `COUNT(*)` from the type-registry validation
query as `Int` while PostgreSQL returns `bigint`. The operator-approved SQL
fallback was then applied idempotently and created the 14 governed widgets plus
14 initial widget version rows. A runtime-key API smoke check verified all 14
widgets are readable through `GET /api/v2/widgets`.
---
### T06 — Build the first ops inventory artifact
```task
id: HF-WP-0001-T06
status: done
priority: medium
state_hub_task_id: "2a0b2f69-5a3d-433c-9cbd-85fd868b63d8"
```
Create an ops inventory document in `helix-forge` that expresses the current
state of:
- environments
- hosts
- clusters
- services
- endpoints
- service discovery and service-catalog gaps
- storage and backup coverage
- migration readiness gates
Use `wiki/CurrentOperationsSituation.md` as the seed background, then turn it
into a more structured inventory artifact. Use this as the working model before
creating a separate `ops-hub` repository.
Done when: a human can see the CoulombCore, local, railiance01, and
ThreePhoenix relationship, including the current Gitea registry state, without
reading multiple repo workplans or relying on shell history.
Output: `wiki/OpsHubInventory.md`.
---
### T07 — Instrument the current Gitea registry work as the first ops-hub signal
```task
id: HF-WP-0001-T07
status: done
priority: medium
state_hub_task_id: "ed3e0396-b16d-40c2-9519-e755ad6241eb"
```
Use the recently fixed Gitea `/v2` route as the first real operational signal.
Suggested event:
```json
{
"widgetId": "<ops-readiness-gitea-registry-widget-id>",
"eventType": "ops-endpoint-verified",
"viewContext": "railiance-apps/workplans/RAIL-AP-WP-0001",
"metadata": {
"vsmFunction": "OPS",
"vsmSystem": "S1",
"endpoint": "https://gitea.coulomb.social/v2/",
"expectedStatus": 401,
"observedHeader": "Docker-Distribution-Api-Version: registry/2.0"
}
}
```
Done when: the Gitea registry readiness event is visible in Inter-Hub and
traceable back to the Railiance workplan.
Blocked until: the `ops-endpoint-gitea-registry` widget exists, the
`ops-endpoint-verified` event type is active, and an ops-hub API key is
available to the operator.
Live result on 2026-06-15: the first Gitea registry readiness event was
inserted by the SQL fallback and is visible through
`GET /api/v2/interaction-events` with the runtime key:
- Event id `4af73b21-75a9-4814-b6df-62083cfda15f`
- Event type `ops-endpoint-verified`
- View context `railiance-apps/workplans/RAIL-AP-WP-0001`
- Metadata records endpoint `https://gitea.coulomb.social/v2/`, expected
status `401`, and observed header
`Docker-Distribution-Api-Version: registry/2.0`
---
### T08 — Define the ops-hub readiness gate model for ThreePhoenix migration
```task
id: HF-WP-0001-T08
status: done
priority: medium
state_hub_task_id: "72a58622-c3ac-4765-8026-5c2489af2058"
```
Define readiness gates that must be green before moving production
responsibility from CoulombCore to ThreePhoenix:
- DNS and TLS are codified.
- Service catalog entries exist for the live and target production services.
- Git hosting and container registry are reproducible.
- Persistent data stores have backup and restore evidence.
- Secrets and SOPS/age keys are available through governed operator paths.
- Cluster runtime and platform services are recreated through Railiance repos.
- Rollback path is documented.
- Operator runbooks exist for deploy, restore, rotate, and incident response.
Done when: each gate has an owner repo, evidence requirement, and status.
Output: `wiki/OpsHubReadinessGates.md`.
---
### T09 — Decide whether to create a separate ops-hub repository
```task
id: HF-WP-0001-T09
status: done
priority: medium
state_hub_task_id: "0e5842fd-1d33-4e2a-9701-07f623a2b901"
```
Decision recorded on 2026-06-06: create and use a separate `ops-hub`
repository. The repo is present locally at `/home/worsch/ops-hub` and tracks
`gitea-remote:coulomb/ops-hub.git`.
Rationale:
- The operator has provided the repo and decided that `ops-hub` should live
outside `helix-forge`.
- `ops-hub` is expected to need implementation assets such as collectors,
adapters, scheduled probes, bootstrap smoke tooling, and possibly UI beyond
Inter-Hub's generic hub dashboards.
- `ops-hub` needs its own workplan prefix, release lifecycle, and repository
agent instructions once implementation begins.
- Keeping deployable/runtime code out of `inter-hub` preserves Inter-Hub as the
generic hub framework while still allowing `ops-hub` to be included as a
VSM Operations / System 1 extension.
Repository boundary:
- `ops-hub`: implementation home for the Operations hub extension, including
collectors, adapters, scheduled probes, runtime packaging, UI/extensions,
tests, and Inter-Hub bootstrap/smoke clients.
- `inter-hub`: generic framework, extension registry, hub manifests, API
consumers, authentication, widgets, event persistence, and bootstrap API.
- `helix-forge`: product/architecture intent, VSM extension pattern,
coordination workplan, and current bootstrap handoff artifacts until they
are ported or retired.
- Railiance repos: desired state, deployment, backup/restore evidence, and
operational facts for their owned infrastructure and services.
Done: the decision is recorded with rationale and boundary; future
implementation should happen in `ops-hub`.
---
### T10 — Inter-Hub API hardening for VSM hub bootstrap
```task
id: HF-WP-0001-T10
status: done
priority: high
target_repo: inter-hub
state_hub_task_id: "7fa54508-7add-4885-8913-12edaadc4d92"
```
Create or link an `inter-hub` workplan to make VSM domain hub bootstrapping
machine-repeatable.
Recommended Inter-Hub improvements:
1. Add `POST /api/v2/hubs` and include it in OpenAPI.
2. Add `POST /api/v2/widgets` and include it in OpenAPI.
3. Add API endpoints for `HubCapabilityManifest` draft creation, update, and
activation.
4. Add a documented place for hub-family metadata such as `hub_family`,
`vsm_function`, and `vsm_system`.
5. Add API endpoints for `ApiConsumer` and API key creation, or a clearly
documented admin-only bootstrap command if API key creation remains UI-only.
6. Add `/api/v2/policy-scopes` to match the policy scope registry already used
by manifests.
7. Add distinct OpenAPI request schemas for create requests instead of reusing
response schemas.
8. Align `docs/new-hub-quickstart.md` with the actual live API until the create
endpoints exist.
9. Fix `Web.Controller.Api.V2.InteractionEvents` so manifest-declared event
types are actually decoded and enforced.
10. Fix webhook dispatch so it uses the submitted event type instead of the
hard-coded `"clicked"` event name.
11. Decide whether event `metadata` is part of the v2 create contract; if yes,
persist it in the controller and test it.
12. Document the bootstrap recipe as a template for `syn-hub`, `ctl-hub`,
`aud-hub`, `int-hub`, `pol-hub`, and `env-hub`.
Done when: the next VSM hub can be created from a script using documented API
calls and without direct DB access.
Linked Inter-Hub workplan:
`inter-hub/workplans/IHUB-WP-0019-vsm-hub-bootstrap-api.md`.
Source implementation status as of 2026-06-14:
- `IHUB-WP-0019` is finished in the local `inter-hub` repo, and
`main` is aligned with `origin/main`.
- The documented smoke path exists at
`/home/worsch/inter-hub/scripts/ops-hub-bootstrap-smoke.py`.
- The `ops-hub` implementation repo has the handoff track
`workplans/OPS-WP-0002-interhub-extension-bootstrap.md`; its production gate
probe is `scripts/interhub-gate-probe.py`.
Current production gate as of 2026-06-06:
- Do not proceed with manual DB seeding unless the operator explicitly chooses
that fallback.
- Wait for `https://hub.coulomb.social/api/v2/hubs` to return `401`
unauthenticated instead of `404`.
- Confirm OpenAPI lists `/hubs`, `/hub-capability-manifests`,
`/api-consumers`, and `/policy-scopes`.
- After the gate passes, run the supported bootstrap/smoke path from the
relevant `inter-hub` or `ops-hub` tooling with `IHUB_BASE` and an operator
key.
Production gate recheck on 2026-06-14:
- `ops-hub/scripts/interhub-gate-probe.py` against
`https://hub.coulomb.social` still fails.
- `GET /api/v2/hubs` returns `404`, not `401`.
- The live OpenAPI still omits `/hubs`, `/hub-capability-manifests`,
`/api-consumers`, and `/policy-scopes`.
- Result: source-side API hardening is complete, but HF-WP-0001 remains gated
on deployment of the current Inter-Hub API or an explicit operator decision
to use the manual SQL fallback.
Production gate recheck after Inter-Hub deployment on 2026-06-14:
- `https://hub.coulomb.social/api/v2/hubs` now returns `200` with an empty
paginated hub list instead of `404`.
- The live OpenAPI now lists `/hubs`, `/hub-capability-manifests`,
`/api-consumers`, and `/policy-scopes`.
- OpenAPI shows public reads for `GET /hubs` and `GET /policy-scopes`, and
authenticated `BearerAuth` writes for hub, manifest, API consumer/key,
widget, and interaction-event creation.
- `ops-hub/scripts/interhub-gate-probe.py` still exits nonzero because it
expects unauthenticated `GET /api/v2/hubs` to return `401`; that expectation
is stale relative to the deployed public-read/authenticated-write contract.
- Result: the Inter-Hub bootstrap API hardening and production deployment gate
are complete. HF-WP-0001 now waits on an attended bootstrap run with
`IHUB_OPERATOR_KEY` or equivalent authenticated operator session, not on
manual SQL fallback or API deployment.
Live bootstrap follow-up on 2026-06-15:
- Hub, manifest, API consumer, and runtime key creation work through the live
API.
- `POST /api/v2/widgets` fails with
`UnexpectedColumnTypeStatementError 0 23 20` from
`SELECT COUNT(*) FROM widget_type_registry ...`; the query result needs to be
decoded as `Int64` or cast to `int`.
- `GET /api/v2/hub-registry` with either the static runtime key or a
short-lived token fails with the same class of error from
`SELECT COUNT(*) FROM api_request_log ...`.
- The operator-approved SQL fallback was used for seed widgets and the first
event so HF-WP-0001 could keep moving, but the next VSM hub is not yet fully
scriptable without direct DB access.
Source fix on 2026-06-15: patched Inter-Hub
`Application/Helper/TypeRegistry.hs` and
`Application/Helper/ApiRateLimit.hs` to cast the affected `COUNT(*)` queries
to `int`, committed as `5101eb5 Fix API count decoding`, and pushed to
`origin/main`.
Deployment status on 2026-06-15:
- Local `git diff --check` passed in `inter-hub`.
- `nix develop ... scripts/compile-check` could not run because the checkout
lacks `.devenv/root` and plain `nix develop` cannot determine the current
directory in this shell.
- A local `nix build .#docker` was attempted, but after more than 20 minutes it
was still compiling dependencies and could not be used for deployment from
this session because registry/deploy credentials are not available here.
- Railiance01 release inspection through `railiance-apps` shows the live
`hub.coulomb.social` Deployment still uses
`92.205.130.254:32166/coulomb/inter-hub:790b5e5`, while Helm values report
`image.tag: 11ff61c`; in either view, the pushed `5101eb5` fix is not live.
Railiance deploy-surface recheck on 2026-06-15:
- `railiance-apps` commit `c7d49d3` adds local Makefile targets for
`INTER_HUB_IMAGE_TAG=<sha>` dry-run, deploy, status, release-info, logs, and
smoke checks.
- `INTER_HUB_IMAGE_TAG=5101eb5 make inter-hub-dry-run` renders
`gitea.coulomb.social/coulomb/inter-hub:5101eb5` and preserves the live
immutable selector label `app=inter-hub`.
- A Helm `--dry-run=server` upgrade against Railiance01 with tag `5101eb5`
succeeds, so the chart shape is acceptable to the cluster.
- The Gitea registry currently returns `MANIFEST_UNKNOWN` for
`coulomb/inter-hub:5101eb5`, so the deployment must not be run until the
Inter-Hub image is built and published.
- `make inter-hub-smoke` is stale: it expects public
`GET /api/v2/hubs` to return `401` and checks `/openapi.json`, while the
live contract is `200` for public hub discovery, `401` for protected
resources such as `/api/v2/widgets`, and OpenAPI at `/api/v2/openapi.json`.
- `railiance-apps` still has only the pull-request manifest dry-run workflow;
it does not yet provide a `workflow_dispatch` production deploy trigger.
Railiance deploy-surface follow-up review on 2026-06-15:
- `railiance-apps` commit `6abf753` adds `RAILIANCE-WP-0011` and addresses the
requested hardening: OCI image preflight, explicit production dry-runs,
`inter-hub-server-dry-run`, current public-read/authenticated-write smoke
contract, and `.gitea/workflows/inter-hub-production-deploy.yaml` with
`workflow_dispatch`.
- `make inter-hub-smoke` passes against live `https://hub.coulomb.social`:
public `GET /api/v2/hubs` returns discovery JSON, protected widgets and
hub-registry routes return `401 invalid_api_key` without a key, and
`/api/v2/openapi.json` lists the expected v2 resources.
- `INTER_HUB_IMAGE_TAG=5101eb5 make check-inter-hub-image` fails before Helm
with `manifest unknown`, which is the desired safe behavior while the image
is absent.
- Inter-Hub confirmed via State Hub message
`269d0ace-5b8e-4fec-a1d0-11a52ad23cc5` that the remaining deploy-side
blocker is to build and publish `gitea.coulomb.social/coulomb/inter-hub:5101eb5`
or another tag containing the same `COUNT(*)` decode fix.
Current blocker: publish a Gitea registry image for Inter-Hub commit
`5101eb5` or an equivalent fix tag, then deploy it through the approved
Railiance path and rerun the authenticated widget-create and hub-registry
smoke checks. Railiance-apps no longer appears to be the blocking surface.
Completed on 2026-06-19:
- Production Inter-Hub now runs image
`gitea.coulomb.social/coulomb/inter-hub:eed4322`, which is ahead of the
`5101eb5` COUNT-decode fix commit.
- Authenticated `GET /api/v2/hub-registry` returns HTTP `200` with the
bootstrap operator key and with a runtime key exchanged through
`POST /api/v2/token`.
- Authenticated `POST /api/v2/widgets` succeeds through the public API; a smoke
widget was created and deleted without using direct DB access.
- Result: the next VSM hub can bootstrap through the documented v2 API surface
without the earlier `COUNT(*)` decode failure class blocking widget creation
or hub-registry reads.
## Initial Acceptance Criteria
This workplan is complete when:
1. `ops-hub` is registered in Inter-Hub as the VSM Operations / System 1 hub.
2. Its capability manifest is active.
3. It has an API consumer and key.
4. Initial ops widgets exist for environments, services, readiness gates, and
migration waves.
5. At least one real operational event has been submitted.
6. The CoulombCore-to-ThreePhoenix readiness model is documented.
7. A decision has been made whether to create a separate `ops-hub` repository.
8. Inter-Hub bootstrap API gaps are either fixed or tracked in an Inter-Hub
workplan.
9. The bootstrap path is reusable enough that `syn-hub` can be created next
without rediscovering the whole process.
## Implementation Log
### 2026-05-19 — Live bootstrap access check
Attempted to execute the prepared bootstrap path.
Findings:
- `hub.coulomb.social` resolves to Railiance01 (`92.205.62.239`).
- The current workstation kubeconfig has only the `default` context and points
at CoulombCore (`92.205.130.254`).
- `~/.kube/config-hosteurope` is missing, so the documented Railiance01
kubeconfig path is not available locally.
- `kubectl` against the current context has no `inter-hub` deployment because
it is the CoulombCore cluster.
- SSH key auth to `root@92.205.62.239`, `worsch@92.205.62.239`, and
`ubuntu@92.205.62.239` was denied.
Result:
- Live `ops-hub` bootstrap was not applied from this session.
- `wiki/ops-hub-bootstrap.sql` was strengthened to set first-class VSM hub
metadata when the target Inter-Hub schema supports it.
- `wiki/OpsHubBootstrapRunbook.md` now records the exact SQL fallback command
to run from a Railiance01-capable session.
Next required operator action:
- Restore/provide Railiance01 Kubernetes access, or run
`wiki/ops-hub-bootstrap.sql` from a host that already has access to the
`net-kingdom-pg-1` pod in the `databases` namespace.
### 2026-06-06 — ops-hub repo decision
The operator decided that `ops-hub` should live in its own repository and be
included as an extension of Inter-Hub. The repo is available locally at
`/home/worsch/ops-hub` with remote `gitea-remote:coulomb/ops-hub.git`.
Result:
- HF-WP-0001 T09 is closed as `done`.
- `ops-hub` implementation work moves to the `ops-hub` repo.
- `helix-forge` keeps architecture, vocabulary, readiness, and bootstrap
handoff references.
- Inter-Hub remains the generic extension framework and still owns the
bootstrap API hardening tracked by T10.
- Live bootstrap remains gated on current Inter-Hub production API availability
unless the operator explicitly chooses the manual SQL fallback.
### 2026-06-14 — production API gate recheck
Rechecked the live Inter-Hub bootstrap gate from the dedicated `ops-hub`
implementation repo:
```text
python3 scripts/interhub-gate-probe.py
```
Result:
- `https://hub.coulomb.social/api/v2/hubs` returns `404`.
- `https://hub.coulomb.social/api/v2/openapi.json` returns `200`, but the
required bootstrap paths are absent.
- Missing paths: `/hubs`, `/hub-capability-manifests`, `/api-consumers`, and
`/policy-scopes`.
Interpretation:
- Inter-Hub source work for `IHUB-WP-0019` is complete and pushed to
`origin/main`.
- Production is still serving an older API surface, so the preferred
ops-hub bootstrap path cannot run yet.
- HF-WP-0001 T10 is moved to `wait` until the production API is deployed or the
operator explicitly chooses the manual SQL fallback.
### 2026-06-14 — production API gate opened
Rechecked the live Inter-Hub API after deployment moved forward:
- `GET https://hub.coulomb.social/api/v2/hubs` now returns `200` with
`{"data":[],"meta":{"page":1,"per_page":50,"total":0}}`.
- The live OpenAPI includes `/hubs`, `/hub-capability-manifests`,
`/api-consumers`, and `/policy-scopes`.
- The deployed contract exposes public reads for hub and policy-scope lists,
while creation and mutation paths require `BearerAuth`.
- `IHUB_OPERATOR_KEY` was not present in this Codex shell, so the authenticated
bootstrap/smoke script was not run.
Result:
- T10 is closed as `done`.
- T02 is ready to run once the operator loads the OpenBao-custodied
`IHUB_OPERATOR_KEY` into a trusted shell.
- T03-T05 remain `wait` until the `ops-hub` row and active manifest exist.
- The next action is to run the supported bootstrap path with
`IHUB_BASE=https://hub.coulomb.social` and the operator key, while preserving
the one-time API key secret outside Git and State Hub.
### 2026-06-15 — bootstrap operator key created and custodied
The seeded Inter-Hub web admin credential was not usable on production, so the
operator explicitly chose a controlled database bootstrap for the first
temporary API key.
Non-secret evidence:
- `net-kingdom-pg-1` is healthy and contains the `interhub` database.
- `api_consumers` initially had no rows.
- A static `inter-hub-bootstrap-operator` key now exists with prefix
`8fab0bef`.
- OpenBao audit shows successful create/read activity for
`platform/data/operators/inter-hub/bootstrap-operator`.
The full key value was not printed into Git, State Hub, or chat. It was stored
manually by the operator in OpenBao. This unblocks the next attended step:
retrieve the key from OpenBao into a trusted shell and create the `ops-hub`
hub row through the supported Inter-Hub API.
### 2026-06-15 — full API bootstrap helper prepared
Added `scripts/ops-hub-bootstrap-api.py` as the HF-WP-0001 production helper.
Unlike the source-side Inter-Hub smoke script, it consumes the full
HelixForge manifest and widget seed artifacts. It supports
`IHUB_OPERATOR_KEY_FILE` and `OPS_HUB_KEY_FILE` so keys can be passed through
0600 temp files rather than shell history or chat.
Validation performed:
- `python3 -m py_compile scripts/ops-hub-bootstrap-api.py`
- `python3 scripts/ops-hub-bootstrap-api.py --help`
- JSON seed parsing for the manifest and 14 widget seeds
### 2026-06-15 — attended Ops Hub bootstrap applied
The operator provided the Inter-Hub bootstrap key through a local temp file.
The helper used that key without printing it and successfully created/reused:
- `ops-hub` hub row with VSM metadata `vsm/OPS/1`.
- Active capability manifest `00aaf90a-8e76-4b0e-892d-33b162862f38`.
- Runtime API consumer `f9e595c6-4e1d-41fd-86cb-c1830bd7ec81`.
- Display-once runtime API key, written only to a 0600 local temp file.
The helper then exposed a live Inter-Hub bug while creating widgets:
```text
UnexpectedColumnTypeStatementError 0 23 20
SELECT COUNT(*) FROM widget_type_registry WHERE name = $1 AND status = 'active'
```
Because PostgreSQL returns `COUNT(*)` as `bigint`, the Inter-Hub validation
code must decode this as `Int64` or cast the query result. The same bug class
also affects authenticated `GET /api/v2/hub-registry` through the
`api_request_log` rate-limit query.
To keep the approved bootstrap moving, the idempotent SQL fallback was applied
for the remaining data. It created:
- 14 governed `ops-hub` widgets.
- 14 initial widget version rows.
- The first `ops-endpoint-verified` Gitea registry readiness event.
Validation:
- All 14 widget types, 15 event types, and 10 annotation categories from the
manifest are present in public registries.
- `GET /api/v2/widgets` with the runtime key returns all 14 `ops-hub` widgets.
- `POST /api/v2/token` returns a short-lived Bearer token for the runtime key.
- `GET /api/v2/interaction-events` with the runtime key returns the Gitea
registry readiness event and metadata.
Remaining operator action:
- Store the generated runtime key from the local temp file in OpenBao at
`platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, then remove the
temp file.
- Track/fix the Inter-Hub `COUNT(*)` decode issues before declaring the next
VSM hub fully scriptable through the public API.
### 2026-06-19 — HF-WP-0001 closed out
Closed the remaining bootstrap custody and production verification gaps:
- Stored the `ops-hub` runtime key in OpenBao at
`platform/operators/ops-hub/runtime` and removed the local temp file.
- Verified runtime-key token exchange, hub-registry reads, and widget listing
through the public Inter-Hub API.
- Confirmed production Inter-Hub image `eed4322` includes the deployed
COUNT-decode fix path; authenticated widget creation and hub-registry reads
now succeed without SQL fallback.
No API keys, OpenBao tokens, or secret values were copied into Git, State Hub,
chat, or workplan text.
## Notes
`ops-hub` should complement State Hub during the transition:
- State Hub continues to track workstreams, decisions, and progress events.
- `ops-hub` tracks operational reality and readiness evidence.
- `syn-hub`, `ctl-hub`, and `aud-hub` can later absorb coordination, control,
and evidence responsibilities once the broader hub constellation is
established.
Reference in New Issue View Git Blame Copy Permalink