generated from coulomb/repo-seed
Record ops-hub bootstrap progress and add API bootstrap helper
Document the 2026-06-15 attended Inter-Hub bootstrap: hub row, active manifest, widget seeding, and runtime API key creation. Add scripts/ops-hub-bootstrap-api.py, extend the SQL fallback for widget versions and the first Gitea event, refresh OpsHubBootstrapRunbook, and inline credential-routing guidance for agents.
This commit is contained in:
50
.claude/rules/credential-routing.md
Normal file
50
.claude/rules/credential-routing.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=helix-forge` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
52
AGENTS.md
52
AGENTS.md
@@ -101,6 +101,58 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
|
||||
---
|
||||
|
||||
## Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=helix-forge` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
---
|
||||
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
Work items originate as files in this repo — not in the hub. The hub is a
|
||||
|
||||
@@ -8,4 +8,5 @@
|
||||
@.claude/rules/stack-and-commands.md
|
||||
@.claude/rules/architecture.md
|
||||
@.claude/rules/repo-boundary.md
|
||||
@.claude/rules/credential-routing.md
|
||||
@.claude/rules/agents.md
|
||||
|
||||
368
scripts/ops-hub-bootstrap-api.py
Executable file
368
scripts/ops-hub-bootstrap-api.py
Executable file
@@ -0,0 +1,368 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Bootstrap ops-hub in Inter-Hub using the prepared HelixForge seeds.
|
||||
|
||||
The script never prints full API keys. The operator key is read from
|
||||
IHUB_OPERATOR_KEY or IHUB_OPERATOR_KEY_FILE. If an ops-hub runtime key is
|
||||
created, the full key is written to a 0600 temp file and only the file path and
|
||||
key prefix are printed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
DEFAULT_BASE = "https://hub.coulomb.social"
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MANIFEST_PATH = ROOT / "wiki" / "ops-hub-manifest.draft.json"
|
||||
WIDGETS_PATH = ROOT / "wiki" / "ops-hub-widgets.seed.json"
|
||||
|
||||
|
||||
class BootstrapError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def load_secret(name: str, file_name: str) -> str:
|
||||
value = os.environ.get(name, "").strip()
|
||||
if value:
|
||||
return value
|
||||
file_path = os.environ.get(file_name, "").strip()
|
||||
if file_path:
|
||||
return Path(file_path).read_text(encoding="utf-8").strip()
|
||||
return ""
|
||||
|
||||
|
||||
def request_json(
|
||||
base_url: str,
|
||||
method: str,
|
||||
path: str,
|
||||
token: str | None,
|
||||
body: dict[str, Any] | None,
|
||||
*,
|
||||
expected: set[int],
|
||||
) -> dict[str, Any]:
|
||||
data = json.dumps(body).encode("utf-8") if body is not None else None
|
||||
request = urllib.request.Request(base_url + path, data=data, method=method)
|
||||
request.add_header("Accept", "application/json")
|
||||
request.add_header("User-Agent", "helix-forge-ops-hub-bootstrap/0.1")
|
||||
if token:
|
||||
request.add_header("Authorization", f"Bearer {token}")
|
||||
if body is not None:
|
||||
request.add_header("Content-Type", "application/json")
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
status = response.status
|
||||
payload = response.read().decode("utf-8")
|
||||
except urllib.error.HTTPError as error:
|
||||
payload = error.read().decode("utf-8", errors="replace")
|
||||
raise BootstrapError(f"{method} {path} failed with HTTP {error.code}: {payload}") from error
|
||||
|
||||
if status not in expected:
|
||||
raise BootstrapError(f"{method} {path} returned HTTP {status}, expected {sorted(expected)}")
|
||||
if not payload:
|
||||
return {}
|
||||
return json.loads(payload)
|
||||
|
||||
|
||||
def list_items(base_url: str, path: str, token: str | None) -> list[dict[str, Any]]:
|
||||
response = request_json(base_url, "GET", path, token, None, expected={200})
|
||||
data = response.get("data", [])
|
||||
if not isinstance(data, list):
|
||||
raise BootstrapError(f"expected paginated data array from {path}")
|
||||
return data
|
||||
|
||||
|
||||
def first_by(items: list[dict[str, Any]], key: str, value: Any) -> dict[str, Any] | None:
|
||||
return next((item for item in items if item.get(key) == value), None)
|
||||
|
||||
|
||||
def load_manifest() -> dict[str, Any]:
|
||||
manifest = json.loads(MANIFEST_PATH.read_text(encoding="utf-8"))
|
||||
required = [
|
||||
"hub",
|
||||
"manifestVersion",
|
||||
"declaredWidgetTypes",
|
||||
"declaredEventTypes",
|
||||
"declaredAnnotationCategories",
|
||||
"declaredPolicyScopes",
|
||||
]
|
||||
missing = [key for key in required if key not in manifest]
|
||||
if missing:
|
||||
raise BootstrapError(f"{MANIFEST_PATH} missing required key(s): {', '.join(missing)}")
|
||||
return manifest
|
||||
|
||||
|
||||
def load_widgets() -> list[dict[str, Any]]:
|
||||
widgets = json.loads(WIDGETS_PATH.read_text(encoding="utf-8"))
|
||||
if not isinstance(widgets, list):
|
||||
raise BootstrapError(f"{WIDGETS_PATH} must contain a JSON array")
|
||||
return widgets
|
||||
|
||||
|
||||
def ensure_hub(base_url: str, operator_key: str, manifest: dict[str, Any]) -> dict[str, Any]:
|
||||
hub_seed = manifest["hub"]
|
||||
slug = hub_seed["slug"]
|
||||
existing = first_by(list_items(base_url, "/api/v2/hubs", None), "slug", slug)
|
||||
if existing:
|
||||
return {"record": existing, "created": False}
|
||||
|
||||
body = {
|
||||
"slug": slug,
|
||||
"name": hub_seed["name"],
|
||||
"domain": hub_seed["domain"],
|
||||
"hubKind": hub_seed.get("hubKind", "domain"),
|
||||
"hubFamily": "vsm",
|
||||
"vsmFunction": "OPS",
|
||||
"vsmSystem": "1",
|
||||
}
|
||||
record = request_json(base_url, "POST", "/api/v2/hubs", operator_key, body, expected={201})
|
||||
return {"record": record, "created": True}
|
||||
|
||||
|
||||
def manifest_body(manifest: dict[str, Any], hub_id: str | None = None) -> dict[str, Any]:
|
||||
body: dict[str, Any] = {
|
||||
"manifestVersion": manifest["manifestVersion"],
|
||||
"declaredWidgetTypes": manifest["declaredWidgetTypes"],
|
||||
"declaredEventTypes": manifest["declaredEventTypes"],
|
||||
"declaredAnnotationCategories": manifest["declaredAnnotationCategories"],
|
||||
"declaredPolicyScopes": manifest["declaredPolicyScopes"],
|
||||
"capabilityDescription": manifest.get("capabilityDescription", ""),
|
||||
"contact": manifest.get("contact", "operator"),
|
||||
}
|
||||
if hub_id:
|
||||
body["hubId"] = hub_id
|
||||
return body
|
||||
|
||||
|
||||
def ensure_manifest(base_url: str, operator_key: str, hub_id: str, manifest: dict[str, Any]) -> dict[str, Any]:
|
||||
query = urllib.parse.urlencode({"hubId": hub_id})
|
||||
manifests = list_items(base_url, f"/api/v2/hub-capability-manifests?{query}", operator_key)
|
||||
active = first_by(manifests, "status", "active")
|
||||
if active:
|
||||
return {"record": active, "created": False, "activated": False}
|
||||
|
||||
draft = first_by(manifests, "status", "draft")
|
||||
if draft:
|
||||
record = request_json(
|
||||
base_url,
|
||||
"PATCH",
|
||||
f"/api/v2/hub-capability-manifests/{draft['id']}",
|
||||
operator_key,
|
||||
manifest_body(manifest),
|
||||
expected={200},
|
||||
)
|
||||
created = False
|
||||
else:
|
||||
record = request_json(
|
||||
base_url,
|
||||
"POST",
|
||||
"/api/v2/hub-capability-manifests",
|
||||
operator_key,
|
||||
manifest_body(manifest, hub_id),
|
||||
expected={201},
|
||||
)
|
||||
created = True
|
||||
|
||||
activated = request_json(
|
||||
base_url,
|
||||
"POST",
|
||||
f"/api/v2/hub-capability-manifests/{record['id']}/activate",
|
||||
operator_key,
|
||||
None,
|
||||
expected={200},
|
||||
)
|
||||
return {"record": activated, "created": created, "activated": True}
|
||||
|
||||
|
||||
def ensure_api_consumer(base_url: str, operator_key: str, manifest_id: str) -> dict[str, Any]:
|
||||
consumers = list_items(base_url, "/api/v2/api-consumers", operator_key)
|
||||
existing = first_by(consumers, "name", "ops-hub")
|
||||
if existing:
|
||||
return {"record": existing, "created": False}
|
||||
|
||||
record = request_json(
|
||||
base_url,
|
||||
"POST",
|
||||
"/api/v2/api-consumers",
|
||||
operator_key,
|
||||
{
|
||||
"name": "ops-hub",
|
||||
"description": "API consumer for the VSM Operations hub",
|
||||
"hubCapabilityManifestId": manifest_id,
|
||||
"rateLimitPerMinute": 120,
|
||||
"quotaPerDay": 50000,
|
||||
},
|
||||
expected={201},
|
||||
)
|
||||
return {"record": record, "created": True}
|
||||
|
||||
|
||||
def write_runtime_key(full_key: str, output_path: str | None) -> Path:
|
||||
if output_path:
|
||||
path = Path(output_path)
|
||||
path.write_text(full_key, encoding="utf-8")
|
||||
else:
|
||||
fd, raw_path = tempfile.mkstemp(prefix="ops-hub-runtime-key-", text=True)
|
||||
path = Path(raw_path)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
handle.write(full_key)
|
||||
path.chmod(stat.S_IRUSR | stat.S_IWUSR)
|
||||
return path
|
||||
|
||||
|
||||
def ensure_runtime_key(
|
||||
base_url: str,
|
||||
operator_key: str,
|
||||
api_consumer_id: str,
|
||||
output_path: str | None,
|
||||
) -> dict[str, Any]:
|
||||
existing_runtime_key = load_secret("OPS_HUB_KEY", "OPS_HUB_KEY_FILE")
|
||||
if existing_runtime_key:
|
||||
return {
|
||||
"created": False,
|
||||
"token": existing_runtime_key,
|
||||
"keyPrefix": existing_runtime_key[:8],
|
||||
"keyFile": os.environ.get("OPS_HUB_KEY_FILE"),
|
||||
}
|
||||
|
||||
response = request_json(
|
||||
base_url,
|
||||
"POST",
|
||||
f"/api/v2/api-consumers/{api_consumer_id}/api-keys",
|
||||
operator_key,
|
||||
{"scopes": "framework:read hub:ops-hub:read hub:ops-hub:write"},
|
||||
expected={201},
|
||||
)
|
||||
full_key = response.get("fullKey")
|
||||
if not full_key:
|
||||
raise BootstrapError("api key creation did not return display-once fullKey")
|
||||
key_file = write_runtime_key(full_key, output_path)
|
||||
api_key = response.get("apiKey") or {}
|
||||
return {
|
||||
"created": True,
|
||||
"token": full_key,
|
||||
"keyPrefix": api_key.get("keyPrefix", full_key[:8]),
|
||||
"keyFile": str(key_file),
|
||||
}
|
||||
|
||||
|
||||
def ensure_widgets(base_url: str, runtime_key: str, hub_id: str, widget_seeds: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
existing_widgets = list_items(base_url, "/api/v2/widgets", runtime_key)
|
||||
existing_by_ref = {
|
||||
widget.get("capabilityRef"): widget
|
||||
for widget in existing_widgets
|
||||
if widget.get("hubId") == hub_id and widget.get("capabilityRef")
|
||||
}
|
||||
created: list[dict[str, Any]] = []
|
||||
reused: list[dict[str, Any]] = []
|
||||
for seed in widget_seeds:
|
||||
existing = existing_by_ref.get(seed.get("capabilityRef"))
|
||||
if existing:
|
||||
reused.append(existing)
|
||||
continue
|
||||
body = {"hubId": hub_id, "status": "active", **seed}
|
||||
created.append(request_json(base_url, "POST", "/api/v2/widgets", runtime_key, body, expected={201}))
|
||||
return {"created": created, "reused": reused}
|
||||
|
||||
|
||||
def submit_gitea_event(base_url: str, runtime_key: str, widgets: dict[str, Any]) -> dict[str, Any] | None:
|
||||
all_widgets = widgets["created"] + widgets["reused"]
|
||||
readiness = next(
|
||||
(widget for widget in all_widgets if widget.get("capabilityRef") == "ops:readiness:gitea-registry"),
|
||||
None,
|
||||
)
|
||||
if not readiness:
|
||||
return None
|
||||
return request_json(
|
||||
base_url,
|
||||
"POST",
|
||||
"/api/v2/interaction-events",
|
||||
runtime_key,
|
||||
{
|
||||
"widgetId": readiness["id"],
|
||||
"eventType": "ops-endpoint-verified",
|
||||
"viewContext": "railiance-apps/workplans/RAIL-AP-WP-0001",
|
||||
"metadata": {
|
||||
"vsmFunction": "OPS",
|
||||
"vsmSystem": "S1",
|
||||
"endpoint": "https://gitea.coulomb.social/v2/",
|
||||
"expectedStatus": 401,
|
||||
"observedHeader": "Docker-Distribution-Api-Version: registry/2.0",
|
||||
"recordedBy": "helix-forge/scripts/ops-hub-bootstrap-api.py",
|
||||
"recordedAt": int(time.time()),
|
||||
},
|
||||
},
|
||||
expected={201},
|
||||
)
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--base", default=os.environ.get("IHUB_BASE", DEFAULT_BASE).rstrip("/"))
|
||||
parser.add_argument("--runtime-key-output", default=os.environ.get("OPS_HUB_RUNTIME_KEY_OUTPUT"))
|
||||
parser.add_argument("--skip-event", action="store_true", help="Do not submit the initial Gitea readiness event")
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = build_parser().parse_args()
|
||||
operator_key = load_secret("IHUB_OPERATOR_KEY", "IHUB_OPERATOR_KEY_FILE")
|
||||
if not operator_key:
|
||||
print("ERROR: set IHUB_OPERATOR_KEY or IHUB_OPERATOR_KEY_FILE", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
manifest = load_manifest()
|
||||
widget_seeds = load_widgets()
|
||||
|
||||
hub = ensure_hub(args.base, operator_key, manifest)
|
||||
hub_record = hub["record"]
|
||||
manifest_result = ensure_manifest(args.base, operator_key, hub_record["id"], manifest)
|
||||
manifest_record = manifest_result["record"]
|
||||
consumer = ensure_api_consumer(args.base, operator_key, manifest_record["id"])
|
||||
runtime_key = ensure_runtime_key(args.base, operator_key, consumer["record"]["id"], args.runtime_key_output)
|
||||
widgets = ensure_widgets(args.base, runtime_key["token"], hub_record["id"], widget_seeds)
|
||||
event = None if args.skip_event else submit_gitea_event(args.base, runtime_key["token"], widgets)
|
||||
|
||||
summary = {
|
||||
"ok": True,
|
||||
"base": args.base,
|
||||
"hub": {"id": hub_record["id"], "slug": hub_record["slug"], "created": hub["created"]},
|
||||
"manifest": {
|
||||
"id": manifest_record["id"],
|
||||
"status": manifest_record["status"],
|
||||
"created": manifest_result["created"],
|
||||
"activated": manifest_result["activated"],
|
||||
},
|
||||
"apiConsumer": {"id": consumer["record"]["id"], "name": consumer["record"]["name"], "created": consumer["created"]},
|
||||
"runtimeKey": {
|
||||
"created": runtime_key["created"],
|
||||
"keyPrefix": runtime_key["keyPrefix"],
|
||||
"keyFile": runtime_key["keyFile"],
|
||||
"storeImmediatelyInOpenBao": "platform/operators/ops-hub/runtime",
|
||||
"field": "OPS_HUB_KEY",
|
||||
},
|
||||
"widgets": {"created": len(widgets["created"]), "reused": len(widgets["reused"])},
|
||||
"event": None if event is None else {"id": event["id"], "eventType": event["eventType"], "widgetId": event["widgetId"]},
|
||||
}
|
||||
print(json.dumps(summary, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except BootstrapError as exc:
|
||||
print(f"ERROR: {exc}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -8,8 +8,9 @@ This runbook gives the operator-ready bootstrap path for `ops-hub`, the VSM
|
||||
Operations / System 1 extension of Inter-Hub.
|
||||
|
||||
Use this when an authenticated Inter-Hub admin session or deployment migration
|
||||
is available. The current public v2 API is not sufficient to create the hub,
|
||||
manifest, API consumer, API key, or seed widgets by itself.
|
||||
is available. The current public v2 API now exposes the supported bootstrap
|
||||
surface; creation and mutation calls require an Inter-Hub operator/admin API
|
||||
key.
|
||||
|
||||
As of 2026-06-06, implementation work for `ops-hub` belongs in the dedicated
|
||||
repo at `/home/worsch/ops-hub` with remote `gitea-remote:coulomb/ops-hub.git`.
|
||||
@@ -25,31 +26,23 @@ implementation repo ports or supersedes it.
|
||||
|
||||
## Current Bootstrap Decision
|
||||
|
||||
Prefer the supported Inter-Hub bootstrap API once production exposes the
|
||||
current API surface. Do not proceed with manual DB seeding unless the operator
|
||||
explicitly chooses that fallback. Until the production API gate passes, the
|
||||
authenticated Inter-Hub admin UI and SQL migration remain fallback paths for
|
||||
attended operator use only.
|
||||
Prefer the supported Inter-Hub bootstrap API for hub, manifest, API consumer,
|
||||
and runtime key creation. Use the idempotent SQL fallback for seed widgets and
|
||||
initial evidence only when the live API hits a known server-side bootstrap bug
|
||||
and the operator explicitly approves that fallback.
|
||||
|
||||
Production API gate:
|
||||
Latest check, 2026-06-14 after Inter-Hub deployment:
|
||||
|
||||
- `https://hub.coulomb.social/api/v2/hubs` returns `401` unauthenticated, not
|
||||
`404`.
|
||||
- `https://hub.coulomb.social/api/v2/hubs` returns `200` with a public,
|
||||
paginated hub list.
|
||||
- OpenAPI lists `/hubs`, `/hub-capability-manifests`, `/api-consumers`, and
|
||||
`/policy-scopes`.
|
||||
- After the gate passes, run the supported bootstrap/smoke path from the
|
||||
relevant `inter-hub` or `ops-hub` tooling with `IHUB_BASE` and an operator
|
||||
key.
|
||||
|
||||
Latest check, 2026-06-14:
|
||||
|
||||
- `ops-hub/scripts/interhub-gate-probe.py` still reports the production gate
|
||||
closed.
|
||||
- `GET /api/v2/hubs` returns `404`.
|
||||
- Live OpenAPI still omits `/hubs`, `/hub-capability-manifests`,
|
||||
`/api-consumers`, and `/policy-scopes`.
|
||||
- Do not run the preferred API bootstrap path until the current Inter-Hub API
|
||||
is deployed, unless the operator explicitly chooses the SQL fallback.
|
||||
- Hub and policy-scope list reads are public. Hub creation, manifest
|
||||
creation/activation, API consumer/key creation, widget creation, and event
|
||||
submission require `Authorization: Bearer <key>`.
|
||||
- The old `ops-hub/scripts/interhub-gate-probe.py` expectation that
|
||||
unauthenticated `GET /api/v2/hubs` must return `401` is stale. The deployed
|
||||
contract is public-read/authenticated-write.
|
||||
|
||||
VSM classification is stored in the manifest capability description for now:
|
||||
|
||||
@@ -61,6 +54,120 @@ Newer Inter-Hub schemas have first-class hub metadata columns for these values.
|
||||
The SQL fallback sets those columns when present and still carries the same
|
||||
classification in the manifest description for older deployments.
|
||||
|
||||
Latest bootstrap result, 2026-06-15:
|
||||
|
||||
- `scripts/ops-hub-bootstrap-api.py` created/reused the `ops-hub` hub row,
|
||||
active manifest, runtime API consumer, and display-once runtime API key.
|
||||
- `POST /api/v2/widgets` failed with an Inter-Hub `COUNT(*)` decode error in
|
||||
type-registry validation.
|
||||
- The operator-approved SQL fallback seeded 14 widgets, 14 initial widget
|
||||
versions, and the first Gitea registry readiness event.
|
||||
- The runtime key can be exchanged through `POST /api/v2/token`, can read all
|
||||
14 widgets through `GET /api/v2/widgets`, and can read the first event
|
||||
through `GET /api/v2/interaction-events`.
|
||||
- `GET /api/v2/hub-registry` currently returns HTTP 500 because Inter-Hub
|
||||
decodes `COUNT(*)` from `api_request_log` as `Int` while PostgreSQL returns
|
||||
`bigint`.
|
||||
|
||||
## Inter-Hub Operator Key
|
||||
|
||||
`IHUB_OPERATOR_KEY` is an existing Inter-Hub API key or short-lived access
|
||||
token accepted by v2 endpoints as:
|
||||
|
||||
```text
|
||||
Authorization: Bearer <key>
|
||||
```
|
||||
|
||||
It is needed only for privileged bootstrap writes:
|
||||
|
||||
- creating or reusing the `ops-hub` hub row;
|
||||
- creating and activating the capability manifest;
|
||||
- creating the `ops-hub` runtime `ApiConsumer`;
|
||||
- minting the display-once runtime API key;
|
||||
- creating widgets and submitting the first event.
|
||||
|
||||
The operator key is not the long-term `ops-hub` runtime credential. It should
|
||||
be used for the attended bootstrap only, then normal `ops-hub` traffic should
|
||||
use the narrower runtime key created during bootstrap.
|
||||
|
||||
Allowed sources:
|
||||
|
||||
1. Retrieve an existing Inter-Hub operator/admin key from the approved secret
|
||||
store, if one has already been created. After `HF-WP-0002` is deployed,
|
||||
the intended browser path is `https://bao.coulomb.social` with KeyCape auth
|
||||
path `netkingdom` and role `platform-admin`. The older `keycape` auth path
|
||||
may remain available as a compatibility alias.
|
||||
2. If no such key exists, log in to the Inter-Hub admin UI and create an
|
||||
operator/bootstrap `ApiConsumer` and API key. The full key is display-once;
|
||||
store it immediately in the approved secret store.
|
||||
3. NetKingdom/OpenBao may provide the key only as the secret-custody path once
|
||||
the key exists and an appropriate policy/path is defined. The
|
||||
`net-kingdom` Git repository must not contain the key.
|
||||
|
||||
`net-kingdom-pg-1` is only the PostgreSQL pod used by the SQL fallback. It is
|
||||
not by itself the source of an Inter-Hub operator key.
|
||||
|
||||
When using the OpenBao browser UI, inspect metadata/path presence first, for
|
||||
example under:
|
||||
|
||||
```text
|
||||
platform/operators/
|
||||
platform/operators/inter-hub/
|
||||
```
|
||||
|
||||
Do not copy secret values into Git, State Hub, chat, shell history, or this
|
||||
runbook. Store display-once Inter-Hub and `ops-hub` runtime keys directly in
|
||||
the approved secret path.
|
||||
|
||||
For an attended local run, enter the key in a trusted shell without echoing it:
|
||||
|
||||
```bash
|
||||
export IHUB_BASE="https://hub.coulomb.social"
|
||||
read -rsp "Inter-Hub operator key: " IHUB_OPERATOR_KEY
|
||||
echo
|
||||
export IHUB_OPERATOR_KEY
|
||||
```
|
||||
|
||||
The preferred HelixForge helper uses the prepared manifest and widget seed
|
||||
artifacts instead of the smaller one-widget smoke fixture:
|
||||
|
||||
```bash
|
||||
umask 077
|
||||
IHUB_KEY_FILE=$(mktemp)
|
||||
# Paste the OpenBao value into this file without printing it to the terminal.
|
||||
read -rsp "Inter-Hub operator key: " IHUB_OPERATOR_KEY
|
||||
printf '%s' "$IHUB_OPERATOR_KEY" > "$IHUB_KEY_FILE"
|
||||
unset IHUB_OPERATOR_KEY
|
||||
echo
|
||||
|
||||
IHUB_BASE="https://hub.coulomb.social" \
|
||||
IHUB_OPERATOR_KEY_FILE="$IHUB_KEY_FILE" \
|
||||
python3 scripts/ops-hub-bootstrap-api.py
|
||||
```
|
||||
|
||||
The helper creates/reuses the `ops-hub` hub row, activates the full
|
||||
HelixForge manifest, creates the `ops-hub` API consumer, creates the runtime
|
||||
API key if needed, creates any missing seed widgets, and submits the first
|
||||
Gitea registry readiness event. If it creates a new `ops-hub` runtime API key,
|
||||
it writes the full key to a 0600 temp file and prints only the path and key
|
||||
prefix. Store that runtime key immediately in OpenBao, for example:
|
||||
|
||||
```text
|
||||
platform/operators/ops-hub/runtime
|
||||
field: OPS_HUB_KEY
|
||||
```
|
||||
|
||||
After storing both keys in OpenBao, remove local temp files:
|
||||
|
||||
```bash
|
||||
rm -f "$IHUB_KEY_FILE" "<runtime-key-file-printed-by-helper>"
|
||||
unset IHUB_KEY_FILE
|
||||
```
|
||||
|
||||
`/home/worsch/inter-hub/scripts/ops-hub-bootstrap-smoke.py` remains useful as
|
||||
a narrow source-side smoke proof, but it creates a one-widget fixture rather
|
||||
than the full HF-WP-0001 Operations vocabulary.
|
||||
|
||||
As of the 2026-05-19 access check, the workstation kubeconfig only points at
|
||||
CoulombCore (`92.205.130.254`) and does not include the Railiance01
|
||||
(`92.205.62.239`) cluster where `hub.coulomb.social` resolves. SSH key access
|
||||
@@ -118,9 +225,10 @@ kubectl exec -i -n databases net-kingdom-pg-1 -- \
|
||||
```
|
||||
|
||||
The SQL fallback creates the hub, active manifest, registry entries, API
|
||||
consumer row, and seed widgets. It does not create the one-time visible static
|
||||
API key; generate that in the authenticated Inter-Hub UI and store it outside
|
||||
Git.
|
||||
consumer row, seed widgets, initial widget version rows, and the first Gitea
|
||||
registry readiness event. It does not create the one-time visible static API
|
||||
key; generate that through the authenticated Inter-Hub API/UI helper and store
|
||||
it outside Git.
|
||||
|
||||
## Validation
|
||||
|
||||
@@ -150,20 +258,24 @@ Expected: a short-lived access token is returned.
|
||||
After widget seeding:
|
||||
|
||||
```bash
|
||||
curl -s https://hub.coulomb.social/api/v2/hub-registry
|
||||
curl -s https://hub.coulomb.social/api/v2/widgets
|
||||
curl -s https://hub.coulomb.social/api/v2/interaction-events
|
||||
```
|
||||
|
||||
Expected: `ops-hub` is visible, and the operator can see the seeded widgets in
|
||||
the authenticated UI.
|
||||
Expected: the 14 seeded `ops-hub` widgets are readable, and the Gitea registry
|
||||
readiness event is visible. `GET /api/v2/hub-registry` is the desired final
|
||||
registry validation, but it currently returns HTTP 500 on production because
|
||||
of the `api_request_log` `COUNT(*)` decode issue.
|
||||
|
||||
## Known Blockers
|
||||
|
||||
- The live public v2 API has no `POST /api/v2/hubs`.
|
||||
- The live public v2 API has no `POST /api/v2/widgets`.
|
||||
- There are no v2 endpoints for manifest creation/activation.
|
||||
- There are no v2 endpoints for API consumer or key creation.
|
||||
- There is no `/api/v2/policy-scopes`.
|
||||
- Interaction event create currently does not persist submitted metadata.
|
||||
- Webhook dispatch currently uses the hard-coded `"clicked"` event type.
|
||||
- `POST /api/v2/widgets` currently fails on production because Inter-Hub
|
||||
decodes `COUNT(*)` from `widget_type_registry` as `Int` while PostgreSQL
|
||||
returns `bigint`.
|
||||
- `GET /api/v2/hub-registry` currently fails on production for the same bug
|
||||
class in the `api_request_log` rate-limit query.
|
||||
- The generated `ops-hub` runtime key must be moved from the local temp file
|
||||
into OpenBao at `platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`,
|
||||
then the temp file must be removed.
|
||||
|
||||
These are tracked by HF-WP-0001 T10 for Inter-Hub hardening.
|
||||
|
||||
@@ -9,6 +9,8 @@
|
||||
-- - Owned type registry entries
|
||||
-- - ApiConsumer row
|
||||
-- - Seed widgets
|
||||
-- - Initial widget versions for the seed widgets
|
||||
-- - First Gitea registry readiness event
|
||||
--
|
||||
-- It intentionally does not create an ApiKey. Generate the key through the
|
||||
-- authenticated Inter-Hub UI so the full static key can be shown once and
|
||||
@@ -284,4 +286,97 @@ WHERE NOT EXISTS (
|
||||
AND capability_ref = seed.capability_ref
|
||||
);
|
||||
|
||||
WITH hub AS (
|
||||
SELECT id FROM hubs WHERE slug = 'ops-hub'
|
||||
), seeded_widgets AS (
|
||||
SELECT
|
||||
w.id,
|
||||
w.name,
|
||||
w.widget_type,
|
||||
w.hub_id,
|
||||
w.capability_ref,
|
||||
w.view_context,
|
||||
w.policy_scope,
|
||||
w.status,
|
||||
w.version
|
||||
FROM widgets w
|
||||
JOIN hub ON hub.id = w.hub_id
|
||||
WHERE w.capability_ref IN (
|
||||
'ops:environment:local',
|
||||
'ops:environment:coulombcore',
|
||||
'ops:environment:railiance01',
|
||||
'ops:environment:threephoenix-prod',
|
||||
'ops:host:coulombcore',
|
||||
'ops:host:railiance01',
|
||||
'ops:service-catalog',
|
||||
'ops:service:gitea',
|
||||
'ops:service:state-hub',
|
||||
'ops:service:inter-hub',
|
||||
'ops:endpoint:gitea-registry',
|
||||
'ops:readiness:gitea-registry',
|
||||
'ops:readiness:state-hub-cluster-deploy',
|
||||
'ops:migration:coulombcore-to-threephoenix'
|
||||
)
|
||||
)
|
||||
INSERT INTO widget_versions (
|
||||
widget_id,
|
||||
version,
|
||||
schema_snapshot
|
||||
)
|
||||
SELECT
|
||||
seeded_widgets.id,
|
||||
1,
|
||||
jsonb_build_object(
|
||||
'name', seeded_widgets.name,
|
||||
'widget_type', seeded_widgets.widget_type,
|
||||
'hub_id', seeded_widgets.hub_id,
|
||||
'capability_ref', seeded_widgets.capability_ref,
|
||||
'view_context', seeded_widgets.view_context,
|
||||
'policy_scope', seeded_widgets.policy_scope,
|
||||
'status', seeded_widgets.status,
|
||||
'version', seeded_widgets.version
|
||||
)
|
||||
FROM seeded_widgets
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM widget_versions
|
||||
WHERE widget_id = seeded_widgets.id
|
||||
AND version = 1
|
||||
);
|
||||
|
||||
WITH readiness_widget AS (
|
||||
SELECT id
|
||||
FROM widgets
|
||||
WHERE capability_ref = 'ops:readiness:gitea-registry'
|
||||
AND hub_id = (SELECT id FROM hubs WHERE slug = 'ops-hub')
|
||||
)
|
||||
INSERT INTO interaction_events (
|
||||
widget_id,
|
||||
event_type,
|
||||
actor_type,
|
||||
view_context_ref,
|
||||
metadata
|
||||
)
|
||||
SELECT
|
||||
readiness_widget.id,
|
||||
'ops-endpoint-verified',
|
||||
'api',
|
||||
'railiance-apps/workplans/RAIL-AP-WP-0001',
|
||||
jsonb_build_object(
|
||||
'vsmFunction', 'OPS',
|
||||
'vsmSystem', 'S1',
|
||||
'endpoint', 'https://gitea.coulomb.social/v2/',
|
||||
'expectedStatus', 401,
|
||||
'observedHeader', 'Docker-Distribution-Api-Version: registry/2.0',
|
||||
'recordedBy', 'helix-forge/wiki/ops-hub-bootstrap.sql'
|
||||
)
|
||||
FROM readiness_widget
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM interaction_events
|
||||
WHERE widget_id = readiness_widget.id
|
||||
AND event_type = 'ops-endpoint-verified'
|
||||
AND view_context_ref = 'railiance-apps/workplans/RAIL-AP-WP-0001'
|
||||
);
|
||||
|
||||
COMMIT;
|
||||
|
||||
@@ -7,7 +7,7 @@ repo: helix-forge
|
||||
status: active
|
||||
owner: worsch
|
||||
created: "2026-05-16"
|
||||
updated: "2026-06-14"
|
||||
updated: "2026-06-15"
|
||||
planning_priority: high
|
||||
planning_order: 1
|
||||
related_repos:
|
||||
@@ -370,7 +370,7 @@ Output: `Confirmed Bootstrap Path` section in this workplan.
|
||||
|
||||
```task
|
||||
id: HF-WP-0001-T02
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "8e9bd9b2-54fc-49a4-8bb8-11c8577be48d"
|
||||
```
|
||||
@@ -393,8 +393,9 @@ fields as an Inter-Hub API/model gap.
|
||||
Done when: `ops-hub` appears in `/Hubs` and `/api/v2/hub-registry` after
|
||||
authentication, and a human can tell that it is the VSM Operations hub.
|
||||
|
||||
Blocked until: an authenticated Inter-Hub admin session or deployment-side
|
||||
migration is available.
|
||||
Ready when: the operator loads the `IHUB_OPERATOR_KEY` from OpenBao into a
|
||||
trusted shell without echoing it. The key exists in approved custody, but the
|
||||
hub row has not been created yet.
|
||||
|
||||
Prepared artifacts:
|
||||
|
||||
@@ -402,13 +403,37 @@ Prepared artifacts:
|
||||
- `wiki/ops-hub-manifest.draft.json`
|
||||
- `wiki/ops-hub-bootstrap.sql`
|
||||
|
||||
Dependency update on 2026-06-15: a temporary Inter-Hub bootstrap operator key
|
||||
was minted directly in the `interhub` database after the web admin seed
|
||||
credential failed. OpenBao audit confirms the key was stored at
|
||||
`platform/operators/inter-hub/bootstrap-operator`, and Inter-Hub DB metadata
|
||||
shows active key prefix `8fab0bef` for `inter-hub-bootstrap-operator`.
|
||||
|
||||
Implementation progress on 2026-06-15: added
|
||||
`scripts/ops-hub-bootstrap-api.py`, which uses
|
||||
`wiki/ops-hub-manifest.draft.json` and `wiki/ops-hub-widgets.seed.json` to
|
||||
create/reuse the hub, activate the full manifest, create the runtime
|
||||
`ops-hub` API consumer/key, seed widgets, and submit the first Gitea readiness
|
||||
event through the supported Inter-Hub API. The helper requires
|
||||
`IHUB_OPERATOR_KEY` or `IHUB_OPERATOR_KEY_FILE` and does not print full key
|
||||
values.
|
||||
|
||||
Live result on 2026-06-15: the attended helper run used the OpenBao-custodied
|
||||
operator key from a 0600 temp file and created/reused the Inter-Hub hub row.
|
||||
Verified non-secret state:
|
||||
|
||||
- Hub id `4f6e4cf7-6a96-4ff2-8a37-08c9f9e405d2`
|
||||
- Slug `ops-hub`
|
||||
- VSM metadata `hub_family=vsm`, `vsm_function=OPS`, `vsm_system=1`
|
||||
- Active manifest id `00aaf90a-8e76-4b0e-892d-33b162862f38`
|
||||
|
||||
---
|
||||
|
||||
### T03 — Activate the ops-hub capability manifest
|
||||
|
||||
```task
|
||||
id: HF-WP-0001-T03
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "55f5aeed-21c3-4a83-bc78-f90f92c7d597"
|
||||
```
|
||||
@@ -440,6 +465,15 @@ operator or migration can create and activate the manifest.
|
||||
|
||||
Prepared artifact: `wiki/ops-hub-manifest.draft.json`.
|
||||
|
||||
Live result on 2026-06-15: the full manifest is active in production. Public
|
||||
registry checks found all expected ops vocabulary values:
|
||||
|
||||
- 14 of 14 widget types present.
|
||||
- 15 of 15 event types present.
|
||||
- 10 of 10 annotation categories present.
|
||||
- Policy scopes are present in the production database and still lack a
|
||||
dedicated public validation check in this repo's helper.
|
||||
|
||||
---
|
||||
|
||||
### T04 — Create ops-hub API consumer and key
|
||||
@@ -467,13 +501,46 @@ Blocked until: an authenticated operator creates the API key and stores the
|
||||
full static key outside Git. The SQL fallback intentionally creates only the
|
||||
consumer row, not the one-time visible secret.
|
||||
|
||||
Progress on 2026-06-15:
|
||||
|
||||
- The `ops-hub` API consumer exists with id
|
||||
`f9e595c6-4e1d-41fd-86cb-c1830bd7ec81`.
|
||||
- The bootstrap helper created the display-once runtime key and wrote it to a
|
||||
local 0600 temp file without printing the key.
|
||||
- `POST /api/v2/token` returns a short-lived Bearer token for the runtime key
|
||||
with `expires_in=3600`.
|
||||
- `GET /api/v2/widgets` works with the runtime key and returns the 14
|
||||
`ops-hub` widgets.
|
||||
|
||||
Remaining before closing:
|
||||
|
||||
- Store the runtime key from the temporary file in OpenBao at
|
||||
`platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, then remove the
|
||||
temp file.
|
||||
- `GET /api/v2/hub-registry` currently returns HTTP 500 because Inter-Hub
|
||||
decodes `COUNT(*)` from `api_request_log` as `Int` while PostgreSQL returns
|
||||
`bigint`. Track/fix this under T10 before treating hub-registry as a clean
|
||||
acceptance signal.
|
||||
|
||||
Custody attempt on 2026-06-15: copied the generated runtime key into the
|
||||
OpenBao pod as a temporary file and attempted to write it to the approved KV
|
||||
path using the pod token helper. OpenBao denied the request with `403
|
||||
permission denied` while resolving the KV mount through
|
||||
`sys/internal/ui/mounts/platform/operators/ops-hub/runtime`. The temporary
|
||||
in-pod key file was removed and verified absent. The local 0600 runtime-key
|
||||
file remains because it has not yet been successfully stored in OpenBao.
|
||||
|
||||
Current blocker: requires an attended OpenBao root/sudo token handoff, or the
|
||||
operator storing the local runtime key manually through the browser UI, before
|
||||
the temp file can be removed and this task can close.
|
||||
|
||||
---
|
||||
|
||||
### T05 — Seed first governed ops widgets
|
||||
|
||||
```task
|
||||
id: HF-WP-0001-T05
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "d303884d-d1f6-4fd0-a4ec-97afe6162164"
|
||||
```
|
||||
@@ -508,6 +575,13 @@ Prepared artifacts:
|
||||
- `wiki/ops-hub-widgets.seed.json`
|
||||
- `wiki/ops-hub-bootstrap.sql`
|
||||
|
||||
Live result on 2026-06-15: widget creation through `POST /api/v2/widgets`
|
||||
failed because Inter-Hub decodes `COUNT(*)` from the type-registry validation
|
||||
query as `Int` while PostgreSQL returns `bigint`. The operator-approved SQL
|
||||
fallback was then applied idempotently and created the 14 governed widgets plus
|
||||
14 initial widget version rows. A runtime-key API smoke check verified all 14
|
||||
widgets are readable through `GET /api/v2/widgets`.
|
||||
|
||||
---
|
||||
|
||||
### T06 — Build the first ops inventory artifact
|
||||
@@ -547,7 +621,7 @@ Output: `wiki/OpsHubInventory.md`.
|
||||
|
||||
```task
|
||||
id: HF-WP-0001-T07
|
||||
status: wait
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "ed3e0396-b16d-40c2-9519-e755ad6241eb"
|
||||
```
|
||||
@@ -578,6 +652,17 @@ Blocked until: the `ops-endpoint-gitea-registry` widget exists, the
|
||||
`ops-endpoint-verified` event type is active, and an ops-hub API key is
|
||||
available to the operator.
|
||||
|
||||
Live result on 2026-06-15: the first Gitea registry readiness event was
|
||||
inserted by the SQL fallback and is visible through
|
||||
`GET /api/v2/interaction-events` with the runtime key:
|
||||
|
||||
- Event id `4af73b21-75a9-4814-b6df-62083cfda15f`
|
||||
- Event type `ops-endpoint-verified`
|
||||
- View context `railiance-apps/workplans/RAIL-AP-WP-0001`
|
||||
- Metadata records endpoint `https://gitea.coulomb.social/v2/`, expected
|
||||
status `401`, and observed header
|
||||
`Docker-Distribution-Api-Version: registry/2.0`
|
||||
|
||||
---
|
||||
|
||||
### T08 — Define the ops-hub readiness gate model for ThreePhoenix migration
|
||||
@@ -728,6 +813,102 @@ Production gate recheck on 2026-06-14:
|
||||
on deployment of the current Inter-Hub API or an explicit operator decision
|
||||
to use the manual SQL fallback.
|
||||
|
||||
Production gate recheck after Inter-Hub deployment on 2026-06-14:
|
||||
|
||||
- `https://hub.coulomb.social/api/v2/hubs` now returns `200` with an empty
|
||||
paginated hub list instead of `404`.
|
||||
- The live OpenAPI now lists `/hubs`, `/hub-capability-manifests`,
|
||||
`/api-consumers`, and `/policy-scopes`.
|
||||
- OpenAPI shows public reads for `GET /hubs` and `GET /policy-scopes`, and
|
||||
authenticated `BearerAuth` writes for hub, manifest, API consumer/key,
|
||||
widget, and interaction-event creation.
|
||||
- `ops-hub/scripts/interhub-gate-probe.py` still exits nonzero because it
|
||||
expects unauthenticated `GET /api/v2/hubs` to return `401`; that expectation
|
||||
is stale relative to the deployed public-read/authenticated-write contract.
|
||||
- Result: the Inter-Hub bootstrap API hardening and production deployment gate
|
||||
are complete. HF-WP-0001 now waits on an attended bootstrap run with
|
||||
`IHUB_OPERATOR_KEY` or equivalent authenticated operator session, not on
|
||||
manual SQL fallback or API deployment.
|
||||
|
||||
Live bootstrap follow-up on 2026-06-15:
|
||||
|
||||
- Hub, manifest, API consumer, and runtime key creation work through the live
|
||||
API.
|
||||
- `POST /api/v2/widgets` fails with
|
||||
`UnexpectedColumnTypeStatementError 0 23 20` from
|
||||
`SELECT COUNT(*) FROM widget_type_registry ...`; the query result needs to be
|
||||
decoded as `Int64` or cast to `int`.
|
||||
- `GET /api/v2/hub-registry` with either the static runtime key or a
|
||||
short-lived token fails with the same class of error from
|
||||
`SELECT COUNT(*) FROM api_request_log ...`.
|
||||
- The operator-approved SQL fallback was used for seed widgets and the first
|
||||
event so HF-WP-0001 could keep moving, but the next VSM hub is not yet fully
|
||||
scriptable without direct DB access.
|
||||
|
||||
Source fix on 2026-06-15: patched Inter-Hub
|
||||
`Application/Helper/TypeRegistry.hs` and
|
||||
`Application/Helper/ApiRateLimit.hs` to cast the affected `COUNT(*)` queries
|
||||
to `int`, committed as `5101eb5 Fix API count decoding`, and pushed to
|
||||
`origin/main`.
|
||||
|
||||
Deployment status on 2026-06-15:
|
||||
|
||||
- Local `git diff --check` passed in `inter-hub`.
|
||||
- `nix develop ... scripts/compile-check` could not run because the checkout
|
||||
lacks `.devenv/root` and plain `nix develop` cannot determine the current
|
||||
directory in this shell.
|
||||
- A local `nix build .#docker` was attempted, but after more than 20 minutes it
|
||||
was still compiling dependencies and could not be used for deployment from
|
||||
this session because registry/deploy credentials are not available here.
|
||||
- Railiance01 release inspection through `railiance-apps` shows the live
|
||||
`hub.coulomb.social` Deployment still uses
|
||||
`92.205.130.254:32166/coulomb/inter-hub:790b5e5`, while Helm values report
|
||||
`image.tag: 11ff61c`; in either view, the pushed `5101eb5` fix is not live.
|
||||
|
||||
Railiance deploy-surface recheck on 2026-06-15:
|
||||
|
||||
- `railiance-apps` commit `c7d49d3` adds local Makefile targets for
|
||||
`INTER_HUB_IMAGE_TAG=<sha>` dry-run, deploy, status, release-info, logs, and
|
||||
smoke checks.
|
||||
- `INTER_HUB_IMAGE_TAG=5101eb5 make inter-hub-dry-run` renders
|
||||
`gitea.coulomb.social/coulomb/inter-hub:5101eb5` and preserves the live
|
||||
immutable selector label `app=inter-hub`.
|
||||
- A Helm `--dry-run=server` upgrade against Railiance01 with tag `5101eb5`
|
||||
succeeds, so the chart shape is acceptable to the cluster.
|
||||
- The Gitea registry currently returns `MANIFEST_UNKNOWN` for
|
||||
`coulomb/inter-hub:5101eb5`, so the deployment must not be run until the
|
||||
Inter-Hub image is built and published.
|
||||
- `make inter-hub-smoke` is stale: it expects public
|
||||
`GET /api/v2/hubs` to return `401` and checks `/openapi.json`, while the
|
||||
live contract is `200` for public hub discovery, `401` for protected
|
||||
resources such as `/api/v2/widgets`, and OpenAPI at `/api/v2/openapi.json`.
|
||||
- `railiance-apps` still has only the pull-request manifest dry-run workflow;
|
||||
it does not yet provide a `workflow_dispatch` production deploy trigger.
|
||||
|
||||
Railiance deploy-surface follow-up review on 2026-06-15:
|
||||
|
||||
- `railiance-apps` commit `6abf753` adds `RAILIANCE-WP-0011` and addresses the
|
||||
requested hardening: OCI image preflight, explicit production dry-runs,
|
||||
`inter-hub-server-dry-run`, current public-read/authenticated-write smoke
|
||||
contract, and `.gitea/workflows/inter-hub-production-deploy.yaml` with
|
||||
`workflow_dispatch`.
|
||||
- `make inter-hub-smoke` passes against live `https://hub.coulomb.social`:
|
||||
public `GET /api/v2/hubs` returns discovery JSON, protected widgets and
|
||||
hub-registry routes return `401 invalid_api_key` without a key, and
|
||||
`/api/v2/openapi.json` lists the expected v2 resources.
|
||||
- `INTER_HUB_IMAGE_TAG=5101eb5 make check-inter-hub-image` fails before Helm
|
||||
with `manifest unknown`, which is the desired safe behavior while the image
|
||||
is absent.
|
||||
- Inter-Hub confirmed via State Hub message
|
||||
`269d0ace-5b8e-4fec-a1d0-11a52ad23cc5` that the remaining deploy-side
|
||||
blocker is to build and publish `gitea.coulomb.social/coulomb/inter-hub:5101eb5`
|
||||
or another tag containing the same `COUNT(*)` decode fix.
|
||||
|
||||
Current blocker: publish a Gitea registry image for Inter-Hub commit
|
||||
`5101eb5` or an equivalent fix tag, then deploy it through the approved
|
||||
Railiance path and rerun the authenticated widget-create and hub-registry
|
||||
smoke checks. Railiance-apps no longer appears to be the blocking surface.
|
||||
|
||||
## Initial Acceptance Criteria
|
||||
|
||||
This workplan is complete when:
|
||||
@@ -820,6 +1001,109 @@ Interpretation:
|
||||
- HF-WP-0001 T10 is moved to `wait` until the production API is deployed or the
|
||||
operator explicitly chooses the manual SQL fallback.
|
||||
|
||||
### 2026-06-14 — production API gate opened
|
||||
|
||||
Rechecked the live Inter-Hub API after deployment moved forward:
|
||||
|
||||
- `GET https://hub.coulomb.social/api/v2/hubs` now returns `200` with
|
||||
`{"data":[],"meta":{"page":1,"per_page":50,"total":0}}`.
|
||||
- The live OpenAPI includes `/hubs`, `/hub-capability-manifests`,
|
||||
`/api-consumers`, and `/policy-scopes`.
|
||||
- The deployed contract exposes public reads for hub and policy-scope lists,
|
||||
while creation and mutation paths require `BearerAuth`.
|
||||
- `IHUB_OPERATOR_KEY` was not present in this Codex shell, so the authenticated
|
||||
bootstrap/smoke script was not run.
|
||||
|
||||
Result:
|
||||
|
||||
- T10 is closed as `done`.
|
||||
- T02 is ready to run once the operator loads the OpenBao-custodied
|
||||
`IHUB_OPERATOR_KEY` into a trusted shell.
|
||||
- T03-T05 remain `wait` until the `ops-hub` row and active manifest exist.
|
||||
- The next action is to run the supported bootstrap path with
|
||||
`IHUB_BASE=https://hub.coulomb.social` and the operator key, while preserving
|
||||
the one-time API key secret outside Git and State Hub.
|
||||
|
||||
### 2026-06-15 — bootstrap operator key created and custodied
|
||||
|
||||
The seeded Inter-Hub web admin credential was not usable on production, so the
|
||||
operator explicitly chose a controlled database bootstrap for the first
|
||||
temporary API key.
|
||||
|
||||
Non-secret evidence:
|
||||
|
||||
- `net-kingdom-pg-1` is healthy and contains the `interhub` database.
|
||||
- `api_consumers` initially had no rows.
|
||||
- A static `inter-hub-bootstrap-operator` key now exists with prefix
|
||||
`8fab0bef`.
|
||||
- OpenBao audit shows successful create/read activity for
|
||||
`platform/data/operators/inter-hub/bootstrap-operator`.
|
||||
|
||||
The full key value was not printed into Git, State Hub, or chat. It was stored
|
||||
manually by the operator in OpenBao. This unblocks the next attended step:
|
||||
retrieve the key from OpenBao into a trusted shell and create the `ops-hub`
|
||||
hub row through the supported Inter-Hub API.
|
||||
|
||||
### 2026-06-15 — full API bootstrap helper prepared
|
||||
|
||||
Added `scripts/ops-hub-bootstrap-api.py` as the HF-WP-0001 production helper.
|
||||
Unlike the source-side Inter-Hub smoke script, it consumes the full
|
||||
HelixForge manifest and widget seed artifacts. It supports
|
||||
`IHUB_OPERATOR_KEY_FILE` and `OPS_HUB_KEY_FILE` so keys can be passed through
|
||||
0600 temp files rather than shell history or chat.
|
||||
|
||||
Validation performed:
|
||||
|
||||
- `python3 -m py_compile scripts/ops-hub-bootstrap-api.py`
|
||||
- `python3 scripts/ops-hub-bootstrap-api.py --help`
|
||||
- JSON seed parsing for the manifest and 14 widget seeds
|
||||
|
||||
### 2026-06-15 — attended Ops Hub bootstrap applied
|
||||
|
||||
The operator provided the Inter-Hub bootstrap key through a local temp file.
|
||||
The helper used that key without printing it and successfully created/reused:
|
||||
|
||||
- `ops-hub` hub row with VSM metadata `vsm/OPS/1`.
|
||||
- Active capability manifest `00aaf90a-8e76-4b0e-892d-33b162862f38`.
|
||||
- Runtime API consumer `f9e595c6-4e1d-41fd-86cb-c1830bd7ec81`.
|
||||
- Display-once runtime API key, written only to a 0600 local temp file.
|
||||
|
||||
The helper then exposed a live Inter-Hub bug while creating widgets:
|
||||
|
||||
```text
|
||||
UnexpectedColumnTypeStatementError 0 23 20
|
||||
SELECT COUNT(*) FROM widget_type_registry WHERE name = $1 AND status = 'active'
|
||||
```
|
||||
|
||||
Because PostgreSQL returns `COUNT(*)` as `bigint`, the Inter-Hub validation
|
||||
code must decode this as `Int64` or cast the query result. The same bug class
|
||||
also affects authenticated `GET /api/v2/hub-registry` through the
|
||||
`api_request_log` rate-limit query.
|
||||
|
||||
To keep the approved bootstrap moving, the idempotent SQL fallback was applied
|
||||
for the remaining data. It created:
|
||||
|
||||
- 14 governed `ops-hub` widgets.
|
||||
- 14 initial widget version rows.
|
||||
- The first `ops-endpoint-verified` Gitea registry readiness event.
|
||||
|
||||
Validation:
|
||||
|
||||
- All 14 widget types, 15 event types, and 10 annotation categories from the
|
||||
manifest are present in public registries.
|
||||
- `GET /api/v2/widgets` with the runtime key returns all 14 `ops-hub` widgets.
|
||||
- `POST /api/v2/token` returns a short-lived Bearer token for the runtime key.
|
||||
- `GET /api/v2/interaction-events` with the runtime key returns the Gitea
|
||||
registry readiness event and metadata.
|
||||
|
||||
Remaining operator action:
|
||||
|
||||
- Store the generated runtime key from the local temp file in OpenBao at
|
||||
`platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, then remove the
|
||||
temp file.
|
||||
- Track/fix the Inter-Hub `COUNT(*)` decode issues before declaring the next
|
||||
VSM hub fully scriptable through the public API.
|
||||
|
||||
## Notes
|
||||
|
||||
`ops-hub` should complement State Hub during the transition:
|
||||
|
||||
Reference in New Issue
Block a user