--- id: IHUB-WP-0022 type: workplan title: "Ops Hub Evidence Intake for Activity Core" domain: inter_hub repo: inter-hub status: active owner: codex topic_slug: inter_hub created: "2026-06-15" updated: "2026-06-15" planning_priority: high planning_order: 22 related_repos: - activity-core - helix-forge related_workplans: - ACTIVITY-WP-0007 - IHUB-WP-0019 - HF-WP-0001 state_hub_workstream_id: "bd086c41-287d-4a4e-8ac5-9ab270f14d72" --- # Ops Hub Evidence Intake for Activity Core ## Goal Prepare the Inter-Hub `ops-hub` intake side for activity-core operational evidence events so `ACTIVITY-WP-0007` can move from State Hub fallback summaries to governed Inter-Hub submissions without ad hoc database access or ungoverned secrets. This workplan comes from the activity-core suggestion: - Message `18b4bf54-6fae-422b-ab29-8586bfc094e8`, created 2026-06-05: prepare the ops-hub intake side for `ops-service-observed`, `ops-endpoint-verified`, `ops-access-path-checked`, `ops-backup-verified`, and `ops-inventory-drift`. - Related closure-gate message `f3ec4a36-6abf-4550-be92-39f5709863de`, created 2026-06-07: activity-core can fall back to State Hub `ops_inventory_probe` summaries, but final activation waits on the Inter-Hub path or an explicit deferral decision. Numbering note: `IHUB-WP-0021` is intentionally left available for the personal-dashboard implementation workplan already named by `IHUB-WP-0020-T04`. ## Background Inter-Hub already has the generic bootstrap surface from `IHUB-WP-0019`: - `POST /api/v2/hubs` - `POST /api/v2/hub-capability-manifests` - manifest activation with declared widget, event, annotation, and policy vocabulary - `POST /api/v2/api-consumers` - one-time API key creation - `POST /api/v2/widgets` - `POST /api/v2/interaction-events` The quickstart in `docs/new-hub-quickstart.md` shows this path for `ops-hub`. However, the activity-core evidence stream needs a concrete, durable contract: which ops-hub widgets receive which event types, how `OPS_HUB_WIDGET_MAPPING` is shaped, where the runtime key is provisioned, and which payload fields activity-core may rely on. Current production caveat as of 2026-06-15: the source fix for COUNT decoding is committed as `5101eb5`, but production image publication/deployment is still tracked in `ADHOC-2026-06-15`. Do not treat widget-create or hub-registry smoke checks as production-ready until that deployment gate is closed. ## Scope ### In Scope - Define the `ops-hub` evidence vocabulary and target widget mapping for activity-core. - Document `OPS_HUB_WIDGET_MAPPING` and the expected event payload shape. - Provision or hand off the `OPS_HUB_KEY` secret through an approved operator-owned secret store outside Git. - Validate the State Hub fallback-first path through `ops_inventory_probe`. - Enable per-entity Inter-Hub submissions only after the widget/API-key path is live and smoke-tested. - Produce closure guidance for `ACTIVITY-WP-0007/T06`. ### Out of Scope - Building activity-core's evidence sink implementation. - Storing static API keys in Git, State Hub, workplans, logs, or chat. - Manual production DB seeding except under explicit operator approval. - Expanding ops-hub beyond the five activity-core evidence event types. - Changing the public/private authentication contract for existing v2 reads. ## Proposed Evidence Vocabulary Activity-core has already declared the event contracts it wants to send: | Event type | Suggested widget family | Purpose | |---|---|---| | `ops-service-observed` | service inventory | Record that a service exists and was observed. | | `ops-endpoint-verified` | endpoint inventory | Record endpoint reachability, auth challenge, or health verification. | | `ops-access-path-checked` | access path inventory | Record operator or service access path verification. | | `ops-backup-verified` | backup inventory | Record backup presence, recency, or restore-drill evidence. | | `ops-inventory-drift` | drift inventory | Record drift between expected and observed operations inventory. | The first implementation should keep one stable widget per entity and evidence family where possible. If activity-core cannot know entity identity reliably, use one aggregate intake widget per family as a conservative first slice, then split into per-entity widgets after payload evidence proves stable. ## Tasks ### T01 - Audit current ops-hub bootstrap and activity-core contracts ```task id: IHUB-WP-0022-T01 status: done priority: high state_hub_task_id: "f9006504-e5f5-465f-9588-3f4279d12b84" ``` Review the current Inter-Hub API, `docs/new-hub-quickstart.md`, the latest activity-core evidence sink contract, and State Hub messages related to `ACTIVITY-WP-0007`. Answer: - Which event types are already declared by activity-core? - Which widget types and policy scopes should ops-hub declare? - Does the live Inter-Hub deployment include the `5101eb5` COUNT decode fix? - Is `ops-hub` already present in the target environment, and is its manifest active? - Is there an existing API consumer/key that should be reused, rotated, or replaced? Exit criteria: `docs/research/ops-hub-evidence-intake-current-state.md` exists with non-secret findings and open gates. Implementation note (2026-06-15): completed `docs/research/ops-hub-evidence-intake-current-state.md`. The audit confirms that Inter-Hub has the required v2 widget and interaction-event primitives, activity-core has the five event definitions and a tested State Hub fallback sink, but the Inter-Hub sink is still deferred and no live `ops_inventory_probe` progress event exists in State Hub yet. --- ### T02 - Define the ops-hub evidence mapping contract ```task id: IHUB-WP-0022-T02 status: done priority: high depends_on: T01 state_hub_task_id: "4f8a98b9-0d01-4333-b847-f83b8c85a5ab" ``` Define the durable mapping that activity-core should receive as `OPS_HUB_WIDGET_MAPPING`. The contract must specify: - Map shape and version field. - Event type keys. - Widget identifiers or stable logical names for each event family. - Entity selector shape for per-service, per-endpoint, per-access-path, and per-backup submissions. - Fallback aggregate widgets for uncertain entity mapping. - Policy scope for operational evidence writes. - Rotation and compatibility expectations when widgets are renamed or split. Exit criteria: `docs/contracts/ops-hub-activity-core-mapping.md` documents the mapping in copyable JSON examples without containing secrets. Implementation note (2026-06-15): completed `docs/contracts/ops-hub-activity-core-mapping.md`. The contract defines a versioned non-secret `OPS_HUB_WIDGET_MAPPING` JSON shape, aggregate-first fallback widgets, per-entity selector rules, stable `widgetRef` values, and Secret-only handling for `OPS_HUB_KEY`. --- ### T03 - Prepare manifest vocabulary and seed widgets ```task id: IHUB-WP-0022-T03 status: wait priority: high depends_on: T02 state_hub_task_id: "94fc9806-781c-45f6-a43c-a6bce13da47b" ``` Use the supported v2 bootstrap surface, or a documented operator-approved bootstrap script, to ensure `ops-hub` declares and activates the required vocabulary. Required vocabulary: - Widget type or types for service, endpoint, access path, backup, and drift evidence. - Event types listed in the activity-core suggestion. - Annotation category for operational risk or follow-up review. - Policy scope for ops evidence writes. Seed the initial widgets named by the mapping contract. Exit criteria: - The active ops-hub manifest declares all required event types. - The widgets named in `OPS_HUB_WIDGET_MAPPING` exist. - `POST /api/v2/widgets` no longer fails with COUNT decode errors in the target environment. - Non-secret widget IDs or logical names are recorded in the mapping contract. Current wait reason (2026-06-15): manifest/widget activation needs a target environment with the `5101eb5` COUNT decode fix live and an authenticated operator/runtime key path. The required vocabulary is documented, but no live manifest or widget seed was performed in this implementation slice. --- ### T04 - Provision the runtime API key outside Git ```task id: IHUB-WP-0022-T04 status: wait priority: high depends_on: T03 state_hub_task_id: "267db6a7-67d2-48af-b3e8-7588f8684957" ``` Create or confirm the ops-hub runtime API consumer and static key, then store the full key only in the approved operator-owned secret store. Rules: - Never print, commit, or paste the full static key into Git, State Hub, or chat. - If a key already exists in a temporary local file, move it into the approved secret path and remove the temp file after verification. - Prefer OpenBao path `platform/operators/ops-hub/runtime`, field `OPS_HUB_KEY`, unless the operator chooses a different approved path. - Record only non-secret evidence: consumer id, key creation time, storage path, and verification result. Exit criteria: - Activity-core has an approved way to receive `OPS_HUB_KEY`. - `POST /api/v2/token` succeeds for the runtime key or the selected auth path. - A protected ops-hub read/write smoke can run without exposing the key. Current wait reason: storing the runtime key in OpenBao requires an attended root/sudo-capable token handoff or operator UI action. --- ### T05 - Document event payload shape and validation rules ```task id: IHUB-WP-0022-T05 status: done priority: high depends_on: T02 state_hub_task_id: "4eb04a83-8eea-4cab-861c-a39f312a5bb9" ``` Document the payload shape activity-core should send to `POST /api/v2/interaction-events`. The document must cover: - Required Inter-Hub fields: `widgetId`, `eventType`, `viewContext`, and `metadata`. - Per-event metadata fields activity-core should include. - Entity identity fields and how to handle unknown values. - Timestamp semantics: observed time versus submitted time. - Severity/result vocabulary, if used. - Idempotency or duplicate tolerance expectations. - Privacy and secret redaction rules. - Expected API errors and recovery behavior. Exit criteria: `docs/contracts/ops-hub-activity-core-event-payloads.md` exists with one example payload for each of the five event types. Implementation note (2026-06-15): completed `docs/contracts/ops-hub-activity-core-event-payloads.md`. It documents the Inter-Hub request envelope, shared validation rules, idempotency expectations, forbidden payload material, expected API errors, and one example for each activity-core event type. --- ### T06 - Validate fallback-first intake ```task id: IHUB-WP-0022-T06 status: wait priority: medium depends_on: T01 state_hub_task_id: "38b54991-bed2-4f9d-bede-bea35821b1ef" ``` Before enabling per-entity Inter-Hub submission, accept and review the State Hub fallback evidence path that activity-core already supports. Validation path: - Trigger or review an `ops_inventory_probe` fallback summary. - Confirm it contains enough non-secret evidence to preserve operating continuity while Inter-Hub submission is gated. - Record which evidence cannot be represented well in fallback summaries and should move to Inter-Hub first. - Decide whether `ACTIVITY-WP-0007/T06` may close with Inter-Hub submission explicitly deferred, or whether live Inter-Hub submission is a hard closure gate. Exit criteria: `docs/evidence/ops-hub-activity-core-fallback-validation.md` records fallback evidence, gaps, and the closure recommendation. Implementation note (2026-06-15): created `docs/evidence/ops-hub-activity-core-fallback-validation.md`. Activity-core's fallback sink is implemented and locally tested, but a direct State Hub query for `event_type=ops_inventory_probe` returned no live events. This task remains waiting on a disabled/manual activity-core probe or other live fallback evidence before it can close. --- ### T07 - Run end-to-end Inter-Hub submission smoke ```task id: IHUB-WP-0022-T07 status: wait priority: high depends_on: T03,T04,T05 state_hub_task_id: "23baee9b-d710-42c8-9a19-f936bd237444" ``` Run a controlled submission from activity-core or a fixture that uses the same environment variables and mapping shape: - `INTER_HUB_URL` - `OPS_HUB_KEY` - `OPS_HUB_WIDGET_MAPPING` Smoke checks: - One event per evidence type is accepted by `POST /api/v2/interaction-events`. - Event type enforcement rejects an undeclared event type. - Metadata is persisted and returned by the relevant list/show endpoint. - API rate-limit and hub-registry reads do not hit COUNT decode failures. - Failure mode is clean when config is absent, matching activity-core's current gated behavior. Exit criteria: non-secret smoke evidence is recorded and activity-core can enable the Inter-Hub sink in its controlled environment. Current wait reason (2026-06-15): depends on live manifest/widgets from T03, runtime key provisioning from T04, and activity-core implementing actual Inter-Hub submission beyond its current deferred sink. --- ### T08 - Coordinate ACTIVITY-WP-0007 closure handoff ```task id: IHUB-WP-0022-T08 status: wait priority: medium depends_on: T06,T07 state_hub_task_id: "4a7ed0ed-552e-42d3-a90f-1efd52b8851e" ``` Send a closure decision or handoff back to activity-core. The handoff must state: - Whether `ACTIVITY-WP-0007/T06` can close on fallback evidence with explicit Inter-Hub deferral. - Or whether live Inter-Hub submission is now ready and should be required before closure. - Which config values activity-core needs, naming only secret references and never secret values. - Which widgets/event types are active. - Which smoke evidence was collected. Exit criteria: activity-core has enough non-secret evidence and configuration shape to close or unblock `ACTIVITY-WP-0007/T06`. Current wait reason (2026-06-15): closure handoff depends on either a live State Hub fallback event plus an explicit Inter-Hub deferral decision, or a successful Inter-Hub submission smoke. ## Exit Criteria Summary | Task | Deliverable | Status | |---|---|---| | T01 | `docs/research/ops-hub-evidence-intake-current-state.md` | done | | T02 | `docs/contracts/ops-hub-activity-core-mapping.md` | done | | T03 | Active ops-hub manifest and seed widgets | wait | | T04 | `OPS_HUB_KEY` stored outside Git and smokeable | wait | | T05 | `docs/contracts/ops-hub-activity-core-event-payloads.md` | done | | T06 | `docs/evidence/ops-hub-activity-core-fallback-validation.md` | wait | | T07 | End-to-end Inter-Hub submission smoke evidence | wait | | T08 | activity-core closure handoff | wait | ## Binding Principles - Governed vocabulary first: every event type must come from an active manifest before activity-core sends it. - Secret custody stays out of Git: workplans may name paths and fields, never key values. - Fallback before activation: State Hub fallback evidence remains the safety path until the Inter-Hub widget/API-key path is verified. - Aggregate first, split later: use aggregate widgets when entity identity is ambiguous, then move to per-entity widgets only when stable. - No manual DB access by default: use the documented v2 API unless the operator explicitly approves a fallback.