generated from coulomb/repo-seed
461 lines
17 KiB
Markdown
461 lines
17 KiB
Markdown
---
|
|
id: IHUB-WP-0022
|
|
type: workplan
|
|
title: "Ops Hub Evidence Intake for Activity Core"
|
|
domain: infotech
|
|
repo: inter-hub
|
|
status: active
|
|
owner: codex
|
|
topic_slug: inter_hub
|
|
created: "2026-06-15"
|
|
updated: "2026-06-27"
|
|
planning_priority: high
|
|
planning_order: 22
|
|
related_repos:
|
|
- activity-core
|
|
- helix-forge
|
|
related_workplans:
|
|
- ACTIVITY-WP-0007
|
|
- IHUB-WP-0019
|
|
- HF-WP-0001
|
|
state_hub_workstream_id: "bd086c41-287d-4a4e-8ac5-9ab270f14d72"
|
|
---
|
|
|
|
# Ops Hub Evidence Intake for Activity Core
|
|
|
|
## Goal
|
|
|
|
Prepare the Inter-Hub `ops-hub` intake side for activity-core operational
|
|
evidence events so `ACTIVITY-WP-0007` can move from State Hub fallback
|
|
summaries to governed Inter-Hub submissions without ad hoc database access or
|
|
ungoverned secrets.
|
|
|
|
This workplan comes from the activity-core suggestion:
|
|
|
|
- Message `18b4bf54-6fae-422b-ab29-8586bfc094e8`, created 2026-06-05:
|
|
prepare the ops-hub intake side for `ops-service-observed`,
|
|
`ops-endpoint-verified`, `ops-access-path-checked`,
|
|
`ops-backup-verified`, and `ops-inventory-drift`.
|
|
- Related closure-gate message `f3ec4a36-6abf-4550-be92-39f5709863de`,
|
|
created 2026-06-07: activity-core can fall back to State Hub
|
|
`ops_inventory_probe` summaries, but final activation waits on the Inter-Hub
|
|
path or an explicit deferral decision.
|
|
|
|
Numbering note: `IHUB-WP-0021` is intentionally left available for the
|
|
personal-dashboard implementation workplan already named by
|
|
`IHUB-WP-0020-T04`.
|
|
|
|
## Background
|
|
|
|
Inter-Hub already has the generic bootstrap surface from `IHUB-WP-0019`:
|
|
|
|
- `POST /api/v2/hubs`
|
|
- `POST /api/v2/hub-capability-manifests`
|
|
- manifest activation with declared widget, event, annotation, and policy
|
|
vocabulary
|
|
- `POST /api/v2/api-consumers`
|
|
- one-time API key creation
|
|
- `POST /api/v2/widgets`
|
|
- `POST /api/v2/interaction-events`
|
|
|
|
The quickstart in `docs/new-hub-quickstart.md` shows this path for `ops-hub`.
|
|
However, the activity-core evidence stream needs a concrete, durable contract:
|
|
which ops-hub widgets receive which event types, how `OPS_HUB_WIDGET_MAPPING`
|
|
is shaped, where the runtime key is provisioned, and which payload fields
|
|
activity-core may rely on.
|
|
|
|
Current production caveat as of 2026-06-15: the source fix for COUNT decoding
|
|
is committed as `5101eb5`, but production image publication/deployment is
|
|
still tracked in `ADHOC-2026-06-15`. Do not treat widget-create or
|
|
hub-registry smoke checks as production-ready until that deployment gate is
|
|
closed.
|
|
|
|
## Scope
|
|
|
|
### In Scope
|
|
|
|
- Define the `ops-hub` evidence vocabulary and target widget mapping for
|
|
activity-core.
|
|
- Document `OPS_HUB_WIDGET_MAPPING` and the expected event payload shape.
|
|
- Provision or hand off the `OPS_HUB_KEY` secret through an approved
|
|
operator-owned secret store outside Git.
|
|
- Validate the State Hub fallback-first path through `ops_inventory_probe`.
|
|
- Enable per-entity Inter-Hub submissions only after the widget/API-key path
|
|
is live and smoke-tested.
|
|
- Produce closure guidance for `ACTIVITY-WP-0007/T06`.
|
|
|
|
### Out of Scope
|
|
|
|
- Building activity-core's evidence sink implementation.
|
|
- Storing static API keys in Git, State Hub, workplans, logs, or chat.
|
|
- Manual production DB seeding except under explicit operator approval.
|
|
- Expanding ops-hub beyond the five activity-core evidence event types.
|
|
- Changing the public/private authentication contract for existing v2 reads.
|
|
|
|
## Proposed Evidence Vocabulary
|
|
|
|
The original activity-core suggestion used five evidence names. Production
|
|
ops-hub now exposes a live seed vocabulary, so the Inter-Hub activation target
|
|
is the compatibility mapping below rather than the early names verbatim:
|
|
|
|
| Original evidence intent | Live Inter-Hub target | Widget family | Purpose |
|
|
|---|---|---|---|
|
|
| `ops-service-observed` | `ops-service-discovered` | `ops-service` / `ops-service-catalog` | Record that a service exists and was discovered. |
|
|
| `ops-endpoint-verified` | `ops-endpoint-verified` | `ops-endpoint` | Record endpoint reachability, auth challenge, or health verification. |
|
|
| `ops-access-path-checked` | deferred | State Hub fallback until ops-hub adds access-path vocabulary or maps it to readiness/risk | Record operator or service access path verification without inventing unsupported registry names. |
|
|
| `ops-backup-verified` | `ops-backup-verified` | `ops-backup-set` | Record backup presence, recency, or restore-drill evidence after a backup-set widget exists. |
|
|
| `ops-inventory-drift` | `ops-drift-detected` | `ops-readiness-gate` / `ops-risk` | Record drift between expected and observed operations inventory. |
|
|
|
|
The first implementation should use existing seeded widgets where possible and
|
|
seed only the missing backup/risk widgets that are needed for the attended
|
|
smoke. If activity-core cannot know entity identity reliably, keep State Hub
|
|
fallback evidence rather than submitting to a made-up aggregate widget.
|
|
|
|
## Tasks
|
|
|
|
### T01 - Audit current ops-hub bootstrap and activity-core contracts
|
|
|
|
```task
|
|
id: IHUB-WP-0022-T01
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "f9006504-e5f5-465f-9588-3f4279d12b84"
|
|
```
|
|
|
|
Review the current Inter-Hub API, `docs/new-hub-quickstart.md`, the latest
|
|
activity-core evidence sink contract, and State Hub messages related to
|
|
`ACTIVITY-WP-0007`.
|
|
|
|
Answer:
|
|
|
|
- Which event types are already declared by activity-core?
|
|
- Which widget types and policy scopes should ops-hub declare?
|
|
- Does the live Inter-Hub deployment include the `5101eb5` COUNT decode fix?
|
|
- Is `ops-hub` already present in the target environment, and is its manifest
|
|
active?
|
|
- Is there an existing API consumer/key that should be reused, rotated, or
|
|
replaced?
|
|
|
|
Exit criteria: `docs/research/ops-hub-evidence-intake-current-state.md`
|
|
exists with non-secret findings and open gates.
|
|
|
|
Implementation note (2026-06-15): completed
|
|
`docs/research/ops-hub-evidence-intake-current-state.md`. The audit confirms
|
|
that Inter-Hub has the required v2 widget and interaction-event primitives,
|
|
activity-core has the five event definitions and a tested State Hub fallback
|
|
sink, but the Inter-Hub sink is still deferred and no live
|
|
`ops_inventory_probe` progress event exists in State Hub yet.
|
|
|
|
---
|
|
|
|
### T02 - Define the ops-hub evidence mapping contract
|
|
|
|
```task
|
|
id: IHUB-WP-0022-T02
|
|
status: done
|
|
priority: high
|
|
depends_on: T01
|
|
state_hub_task_id: "4f8a98b9-0d01-4333-b847-f83b8c85a5ab"
|
|
```
|
|
|
|
Define the durable mapping that activity-core should receive as
|
|
`OPS_HUB_WIDGET_MAPPING`.
|
|
|
|
The contract must specify:
|
|
|
|
- Map shape and version field.
|
|
- Event type keys.
|
|
- Widget identifiers or stable logical names for each event family.
|
|
- Entity selector shape for per-service, per-endpoint, per-access-path, and
|
|
per-backup submissions.
|
|
- Fallback aggregate widgets for uncertain entity mapping.
|
|
- Policy scope for operational evidence writes.
|
|
- Rotation and compatibility expectations when widgets are renamed or split.
|
|
|
|
Exit criteria: `docs/contracts/ops-hub-activity-core-mapping.md` documents the
|
|
mapping in copyable JSON examples without containing secrets.
|
|
|
|
Implementation note (2026-06-15): completed
|
|
`docs/contracts/ops-hub-activity-core-mapping.md`. The contract defines a
|
|
versioned non-secret `OPS_HUB_WIDGET_MAPPING` JSON shape, aggregate-first
|
|
fallback widgets, per-entity selector rules, stable `widgetRef` values, and
|
|
Secret-only handling for `OPS_HUB_KEY`.
|
|
|
|
Compatibility note (2026-06-27): revised the mapping contract to match the
|
|
live ops-hub seed vocabulary. The mapping now aliases `ops-service-observed`
|
|
to `ops-service-discovered`, aliases `ops-inventory-drift` to
|
|
`ops-drift-detected`, uses declared ops policy scopes instead of the old
|
|
`ops-evidence` proposal, and defers access-path evidence until ops-hub has a
|
|
supported event/widget target.
|
|
|
|
---
|
|
|
|
### T03 - Prepare manifest vocabulary and seed widgets
|
|
|
|
```task
|
|
id: IHUB-WP-0022-T03
|
|
status: wait
|
|
priority: high
|
|
depends_on: T02
|
|
state_hub_task_id: "94fc9806-781c-45f6-a43c-a6bce13da47b"
|
|
```
|
|
|
|
Use the supported v2 bootstrap surface, or a documented operator-approved
|
|
bootstrap script, to ensure `ops-hub` declares and activates the required
|
|
vocabulary.
|
|
|
|
Required vocabulary:
|
|
|
|
- Widget type or types for service, endpoint, access path, backup, and drift
|
|
evidence.
|
|
- Event types listed in the activity-core suggestion.
|
|
- Annotation category for operational risk or follow-up review.
|
|
- Policy scope for ops evidence writes.
|
|
|
|
Seed the initial widgets named by the mapping contract.
|
|
|
|
Exit criteria:
|
|
|
|
- The active ops-hub manifest declares all required event types.
|
|
- The widgets named in `OPS_HUB_WIDGET_MAPPING` exist.
|
|
- `POST /api/v2/widgets` no longer fails with COUNT decode errors in the
|
|
target environment.
|
|
- Non-secret widget IDs or logical names are recorded in the mapping contract.
|
|
|
|
Current wait reason (2026-06-15): manifest/widget activation needs a target
|
|
environment with the `5101eb5` COUNT decode fix live and an authenticated
|
|
operator/runtime key path. The required vocabulary is documented, but no live
|
|
manifest or widget seed was performed in this implementation slice.
|
|
|
|
Progress note (2026-06-27): public production probes show an `ops-hub` row and
|
|
the live seed registry vocabulary, and the contract docs now target that live
|
|
vocabulary. T03 remains waiting because protected widget lookup, widget ids,
|
|
any missing backup/risk seed widgets, and authenticated smoke evidence still
|
|
require the operator/runtime key path.
|
|
|
|
---
|
|
|
|
### T04 - Provision the runtime API key outside Git
|
|
|
|
```task
|
|
id: IHUB-WP-0022-T04
|
|
status: wait
|
|
priority: high
|
|
depends_on: T03
|
|
state_hub_task_id: "267db6a7-67d2-48af-b3e8-7588f8684957"
|
|
```
|
|
|
|
Create or confirm the ops-hub runtime API consumer and static key, then store
|
|
the full key only in the approved operator-owned secret store.
|
|
|
|
Rules:
|
|
|
|
- Never print, commit, or paste the full static key into Git, State Hub, or
|
|
chat.
|
|
- If a key already exists in a temporary local file, move it into the approved
|
|
secret path and remove the temp file after verification.
|
|
- Prefer OpenBao path `platform/operators/ops-hub/runtime`, field
|
|
`OPS_HUB_KEY`, unless the operator chooses a different approved path.
|
|
- Record only non-secret evidence: consumer id, key creation time, storage
|
|
path, and verification result.
|
|
|
|
Exit criteria:
|
|
|
|
- Activity-core has an approved way to receive `OPS_HUB_KEY`.
|
|
- `POST /api/v2/token` succeeds for the runtime key or the selected auth path.
|
|
- A protected ops-hub read/write smoke can run without exposing the key.
|
|
|
|
Current wait reason: storing the runtime key in OpenBao requires an attended
|
|
root/sudo-capable token handoff or operator UI action.
|
|
|
|
---
|
|
|
|
### T05 - Document event payload shape and validation rules
|
|
|
|
```task
|
|
id: IHUB-WP-0022-T05
|
|
status: done
|
|
priority: high
|
|
depends_on: T02
|
|
state_hub_task_id: "4eb04a83-8eea-4cab-861c-a39f312a5bb9"
|
|
```
|
|
|
|
Document the payload shape activity-core should send to
|
|
`POST /api/v2/interaction-events`.
|
|
|
|
The document must cover:
|
|
|
|
- Required Inter-Hub fields: `widgetId`, `eventType`, `viewContext`, and
|
|
`metadata`.
|
|
- Per-event metadata fields activity-core should include.
|
|
- Entity identity fields and how to handle unknown values.
|
|
- Timestamp semantics: observed time versus submitted time.
|
|
- Severity/result vocabulary, if used.
|
|
- Idempotency or duplicate tolerance expectations.
|
|
- Privacy and secret redaction rules.
|
|
- Expected API errors and recovery behavior.
|
|
|
|
Exit criteria: `docs/contracts/ops-hub-activity-core-event-payloads.md`
|
|
exists with one example payload for each of the five event types.
|
|
|
|
Implementation note (2026-06-15): completed
|
|
`docs/contracts/ops-hub-activity-core-event-payloads.md`. It documents the
|
|
Inter-Hub request envelope, shared validation rules, idempotency expectations,
|
|
forbidden payload material, expected API errors, and one example for each
|
|
activity-core event type.
|
|
|
|
Compatibility note (2026-06-27): revised payload examples to submit only live
|
|
ops-hub event types. Access-path payloads are documented as deferred fallback
|
|
evidence, and old event names are treated as aliases that should not be posted
|
|
to Inter-Hub.
|
|
|
|
---
|
|
|
|
### T06 - Validate fallback-first intake
|
|
|
|
```task
|
|
id: IHUB-WP-0022-T06
|
|
status: done
|
|
priority: medium
|
|
depends_on: T01
|
|
state_hub_task_id: "38b54991-bed2-4f9d-bede-bea35821b1ef"
|
|
```
|
|
|
|
Before enabling per-entity Inter-Hub submission, accept and review the State
|
|
Hub fallback evidence path that activity-core already supports.
|
|
|
|
Validation path:
|
|
|
|
- Trigger or review an `ops_inventory_probe` fallback summary.
|
|
- Confirm it contains enough non-secret evidence to preserve operating
|
|
continuity while Inter-Hub submission is gated.
|
|
- Record which evidence cannot be represented well in fallback summaries and
|
|
should move to Inter-Hub first.
|
|
- Decide whether `ACTIVITY-WP-0007/T06` may close with Inter-Hub submission
|
|
explicitly deferred, or whether live Inter-Hub submission is a hard closure
|
|
gate.
|
|
|
|
Exit criteria: `docs/evidence/ops-hub-activity-core-fallback-validation.md`
|
|
records fallback evidence, gaps, and the closure recommendation.
|
|
|
|
Implementation note (2026-06-15): created
|
|
`docs/evidence/ops-hub-activity-core-fallback-validation.md`. Activity-core's
|
|
fallback sink is implemented and locally tested, but a direct State Hub query
|
|
for `event_type=ops_inventory_probe` returned no live events. This task remains
|
|
waiting on a disabled/manual activity-core probe or other live fallback
|
|
evidence before it can close.
|
|
|
|
Implementation note (2026-06-16): completed fallback-first validation using
|
|
Railiance cluster-owned verifier evidence. State Hub progress
|
|
`db408146-0310-4ac3-ac77-f73c5a41e070` records a live
|
|
`ops_inventory_probe` summary from activity-core:
|
|
`0 ok, 4 degraded, 0 down, 5 skipped`. Railiance evidence note
|
|
`60256e9a-9d1b-44db-8999-738cf03bca2e` proves the progress event matched the
|
|
manual trigger run id `90e3b112-d1e3-51af-8fb2-cb61f26add17` and includes the
|
|
live `actcore-api` image digest. Updated the validation document with the
|
|
evidence, gaps, and closure recommendation. Inter-Hub per-entity submission
|
|
remains deferred to T03/T04/T07.
|
|
|
|
---
|
|
|
|
### T07 - Run end-to-end Inter-Hub submission smoke
|
|
|
|
```task
|
|
id: IHUB-WP-0022-T07
|
|
status: wait
|
|
priority: high
|
|
depends_on: T03,T04,T05
|
|
state_hub_task_id: "23baee9b-d710-42c8-9a19-f936bd237444"
|
|
```
|
|
|
|
Run a controlled submission from activity-core or a fixture that uses the same
|
|
environment variables and mapping shape:
|
|
|
|
- `INTER_HUB_URL`
|
|
- `OPS_HUB_KEY`
|
|
- `OPS_HUB_WIDGET_MAPPING`
|
|
|
|
Smoke checks:
|
|
|
|
- One event per evidence type is accepted by
|
|
`POST /api/v2/interaction-events`.
|
|
- Event type enforcement rejects an undeclared event type.
|
|
- Metadata is persisted and returned by the relevant list/show endpoint.
|
|
- API rate-limit and hub-registry reads do not hit COUNT decode failures.
|
|
- Failure mode is clean when config is absent, matching activity-core's current
|
|
gated behavior.
|
|
|
|
Exit criteria: non-secret smoke evidence is recorded and activity-core can
|
|
enable the Inter-Hub sink in its controlled environment.
|
|
|
|
Current wait reason (2026-06-15): depends on live manifest/widgets from T03,
|
|
runtime key provisioning from T04, and activity-core implementing actual
|
|
Inter-Hub submission beyond its current deferred sink.
|
|
|
|
---
|
|
|
|
### T08 - Coordinate ACTIVITY-WP-0007 closure handoff
|
|
|
|
```task
|
|
id: IHUB-WP-0022-T08
|
|
status: done
|
|
priority: medium
|
|
depends_on: T06
|
|
state_hub_task_id: "4a7ed0ed-552e-42d3-a90f-1efd52b8851e"
|
|
```
|
|
|
|
Send a closure decision or handoff back to activity-core.
|
|
|
|
The handoff must state:
|
|
|
|
- Whether `ACTIVITY-WP-0007/T06` can close on fallback evidence with explicit
|
|
Inter-Hub deferral.
|
|
- Or whether live Inter-Hub submission is now ready and should be required
|
|
before closure.
|
|
- Which config values activity-core needs, naming only secret references and
|
|
never secret values.
|
|
- Which widgets/event types are active.
|
|
- Which smoke evidence was collected.
|
|
|
|
Exit criteria: activity-core has enough non-secret evidence and configuration
|
|
shape to close or unblock `ACTIVITY-WP-0007/T06`.
|
|
|
|
Current wait reason (2026-06-15): closure handoff depends on either a live
|
|
State Hub fallback event plus an explicit Inter-Hub deferral decision, or a
|
|
successful Inter-Hub submission smoke.
|
|
|
|
Implementation note (2026-06-16): completed the activity-core closure handoff
|
|
on the fallback-deferred path. `ACTIVITY-WP-0007/T06` is already closed in
|
|
activity-core and State Hub. Inter-Hub accepts that closure on live State Hub
|
|
fallback evidence (`ops_inventory_probe`
|
|
`db408146-0310-4ac3-ac77-f73c5a41e070`) with explicit deferral of governed
|
|
Inter-Hub submissions until the ops-hub manifest/widget path, runtime key, and
|
|
end-to-end smoke are complete under T03, T04, and T07. No secret values or
|
|
runtime key material are required for this handoff.
|
|
|
|
## Exit Criteria Summary
|
|
|
|
| Task | Deliverable | Status |
|
|
|---|---|---|
|
|
| T01 | `docs/research/ops-hub-evidence-intake-current-state.md` | done |
|
|
| T02 | `docs/contracts/ops-hub-activity-core-mapping.md` | done |
|
|
| T03 | Active ops-hub manifest and seed widgets | wait |
|
|
| T04 | `OPS_HUB_KEY` stored outside Git and smokeable | wait |
|
|
| T05 | `docs/contracts/ops-hub-activity-core-event-payloads.md` | done |
|
|
| T06 | `docs/evidence/ops-hub-activity-core-fallback-validation.md` | done |
|
|
| T07 | End-to-end Inter-Hub submission smoke evidence | wait |
|
|
| T08 | activity-core closure handoff | done |
|
|
|
|
## Binding Principles
|
|
|
|
- Governed vocabulary first: every event type must come from an active
|
|
manifest before activity-core sends it.
|
|
- Secret custody stays out of Git: workplans may name paths and fields, never
|
|
key values.
|
|
- Fallback before activation: State Hub fallback evidence remains the safety
|
|
path until the Inter-Hub widget/API-key path is verified.
|
|
- Aggregate first, split later: use aggregate widgets when entity identity is
|
|
ambiguous, then move to per-entity widgets only when stable.
|
|
- No manual DB access by default: use the documented v2 API unless the operator
|
|
explicitly approves a fallback.
|