inter-hub/workplans/IHUB-WP-0022-ops-hub-evidence-intake.md

id, type, title, domain, repo, status, owner, topic_slug, created, updated, planning_priority, planning_order, related_repos, related_workplans, state_hub_workstream_id

id	type	title	domain	repo	status	owner	topic_slug	created	updated	planning_priority	planning_order	related_repos	related_workplans	state_hub_workstream_id
IHUB-WP-0022	workplan	Ops Hub Evidence Intake for Activity Core	inter_hub	inter-hub	active	codex	inter_hub	2026-06-15	2026-06-15	high	22	activity-core helix-forge	ACTIVITY-WP-0007 IHUB-WP-0019 HF-WP-0001	bd086c41-287d-4a4e-8ac5-9ab270f14d72

Ops Hub Evidence Intake for Activity Core

Goal

Prepare the Inter-Hub ops-hub intake side for activity-core operational evidence events so ACTIVITY-WP-0007 can move from State Hub fallback summaries to governed Inter-Hub submissions without ad hoc database access or ungoverned secrets.

This workplan comes from the activity-core suggestion:

• Message 18b4bf54-6fae-422b-ab29-8586bfc094e8, created 2026-06-05: prepare the ops-hub intake side for ops-service-observed, ops-endpoint-verified, ops-access-path-checked, ops-backup-verified, and ops-inventory-drift.
• Related closure-gate message f3ec4a36-6abf-4550-be92-39f5709863de, created 2026-06-07: activity-core can fall back to State Hub ops_inventory_probe summaries, but final activation waits on the Inter-Hub path or an explicit deferral decision.

Numbering note: IHUB-WP-0021 is intentionally left available for the personal-dashboard implementation workplan already named by IHUB-WP-0020-T04.

Background

Inter-Hub already has the generic bootstrap surface from IHUB-WP-0019:

• POST /api/v2/hubs
• POST /api/v2/hub-capability-manifests
• manifest activation with declared widget, event, annotation, and policy vocabulary
• POST /api/v2/api-consumers
• one-time API key creation
• POST /api/v2/widgets
• POST /api/v2/interaction-events

The quickstart in docs/new-hub-quickstart.md shows this path for ops-hub. However, the activity-core evidence stream needs a concrete, durable contract: which ops-hub widgets receive which event types, how OPS_HUB_WIDGET_MAPPING is shaped, where the runtime key is provisioned, and which payload fields activity-core may rely on.

Current production caveat as of 2026-06-15: the source fix for COUNT decoding is committed as 5101eb5, but production image publication/deployment is still tracked in ADHOC-2026-06-15. Do not treat widget-create or hub-registry smoke checks as production-ready until that deployment gate is closed.

Scope

In Scope

• Define the ops-hub evidence vocabulary and target widget mapping for activity-core.
• Document OPS_HUB_WIDGET_MAPPING and the expected event payload shape.
• Provision or hand off the OPS_HUB_KEY secret through an approved operator-owned secret store outside Git.
• Validate the State Hub fallback-first path through ops_inventory_probe.
• Enable per-entity Inter-Hub submissions only after the widget/API-key path is live and smoke-tested.
• Produce closure guidance for ACTIVITY-WP-0007/T06.

Out of Scope

• Building activity-core's evidence sink implementation.
• Storing static API keys in Git, State Hub, workplans, logs, or chat.
• Manual production DB seeding except under explicit operator approval.
• Expanding ops-hub beyond the five activity-core evidence event types.
• Changing the public/private authentication contract for existing v2 reads.

Proposed Evidence Vocabulary

Activity-core has already declared the event contracts it wants to send:

Event type	Suggested widget family	Purpose
ops-service-observed	service inventory	Record that a service exists and was observed.
ops-endpoint-verified	endpoint inventory	Record endpoint reachability, auth challenge, or health verification.
ops-access-path-checked	access path inventory	Record operator or service access path verification.
ops-backup-verified	backup inventory	Record backup presence, recency, or restore-drill evidence.
ops-inventory-drift	drift inventory	Record drift between expected and observed operations inventory.

The first implementation should keep one stable widget per entity and evidence family where possible. If activity-core cannot know entity identity reliably, use one aggregate intake widget per family as a conservative first slice, then split into per-entity widgets after payload evidence proves stable.

Tasks

T01 - Audit current ops-hub bootstrap and activity-core contracts

id: IHUB-WP-0022-T01
status: done
priority: high
state_hub_task_id: "f9006504-e5f5-465f-9588-3f4279d12b84"

Review the current Inter-Hub API, docs/new-hub-quickstart.md, the latest activity-core evidence sink contract, and State Hub messages related to ACTIVITY-WP-0007.

Answer:

• Which event types are already declared by activity-core?
• Which widget types and policy scopes should ops-hub declare?
• Does the live Inter-Hub deployment include the 5101eb5 COUNT decode fix?
• Is ops-hub already present in the target environment, and is its manifest active?
• Is there an existing API consumer/key that should be reused, rotated, or replaced?

Exit criteria: docs/research/ops-hub-evidence-intake-current-state.md exists with non-secret findings and open gates.

Implementation note (2026-06-15): completed docs/research/ops-hub-evidence-intake-current-state.md. The audit confirms that Inter-Hub has the required v2 widget and interaction-event primitives, activity-core has the five event definitions and a tested State Hub fallback sink, but the Inter-Hub sink is still deferred and no live ops_inventory_probe progress event exists in State Hub yet.

---

T02 - Define the ops-hub evidence mapping contract

id: IHUB-WP-0022-T02
status: done
priority: high
depends_on: T01
state_hub_task_id: "4f8a98b9-0d01-4333-b847-f83b8c85a5ab"

Define the durable mapping that activity-core should receive as OPS_HUB_WIDGET_MAPPING.

The contract must specify:

• Map shape and version field.
• Event type keys.
• Widget identifiers or stable logical names for each event family.
• Entity selector shape for per-service, per-endpoint, per-access-path, and per-backup submissions.
• Fallback aggregate widgets for uncertain entity mapping.
• Policy scope for operational evidence writes.
• Rotation and compatibility expectations when widgets are renamed or split.

Exit criteria: docs/contracts/ops-hub-activity-core-mapping.md documents the mapping in copyable JSON examples without containing secrets.

Implementation note (2026-06-15): completed docs/contracts/ops-hub-activity-core-mapping.md. The contract defines a versioned non-secret OPS_HUB_WIDGET_MAPPING JSON shape, aggregate-first fallback widgets, per-entity selector rules, stable widgetRef values, and Secret-only handling for OPS_HUB_KEY.

---

T03 - Prepare manifest vocabulary and seed widgets

id: IHUB-WP-0022-T03
status: wait
priority: high
depends_on: T02
state_hub_task_id: "94fc9806-781c-45f6-a43c-a6bce13da47b"

Use the supported v2 bootstrap surface, or a documented operator-approved bootstrap script, to ensure ops-hub declares and activates the required vocabulary.

Required vocabulary:

• Widget type or types for service, endpoint, access path, backup, and drift evidence.
• Event types listed in the activity-core suggestion.
• Annotation category for operational risk or follow-up review.
• Policy scope for ops evidence writes.

Seed the initial widgets named by the mapping contract.

Exit criteria:

• The active ops-hub manifest declares all required event types.
• The widgets named in OPS_HUB_WIDGET_MAPPING exist.
• POST /api/v2/widgets no longer fails with COUNT decode errors in the target environment.
• Non-secret widget IDs or logical names are recorded in the mapping contract.

Current wait reason (2026-06-15): manifest/widget activation needs a target environment with the 5101eb5 COUNT decode fix live and an authenticated operator/runtime key path. The required vocabulary is documented, but no live manifest or widget seed was performed in this implementation slice.

---

T04 - Provision the runtime API key outside Git

id: IHUB-WP-0022-T04
status: wait
priority: high
depends_on: T03
state_hub_task_id: "267db6a7-67d2-48af-b3e8-7588f8684957"

Create or confirm the ops-hub runtime API consumer and static key, then store the full key only in the approved operator-owned secret store.

Rules:

• Never print, commit, or paste the full static key into Git, State Hub, or chat.
• If a key already exists in a temporary local file, move it into the approved secret path and remove the temp file after verification.
• Prefer OpenBao path platform/operators/ops-hub/runtime, field OPS_HUB_KEY, unless the operator chooses a different approved path.
• Record only non-secret evidence: consumer id, key creation time, storage path, and verification result.

Exit criteria:

• Activity-core has an approved way to receive OPS_HUB_KEY.
• POST /api/v2/token succeeds for the runtime key or the selected auth path.
• A protected ops-hub read/write smoke can run without exposing the key.

Current wait reason: storing the runtime key in OpenBao requires an attended root/sudo-capable token handoff or operator UI action.

---

T05 - Document event payload shape and validation rules

id: IHUB-WP-0022-T05
status: done
priority: high
depends_on: T02
state_hub_task_id: "4eb04a83-8eea-4cab-861c-a39f312a5bb9"

Document the payload shape activity-core should send to POST /api/v2/interaction-events.

The document must cover:

• Required Inter-Hub fields: widgetId, eventType, viewContext, and metadata.
• Per-event metadata fields activity-core should include.
• Entity identity fields and how to handle unknown values.
• Timestamp semantics: observed time versus submitted time.
• Severity/result vocabulary, if used.
• Idempotency or duplicate tolerance expectations.
• Privacy and secret redaction rules.
• Expected API errors and recovery behavior.

Exit criteria: docs/contracts/ops-hub-activity-core-event-payloads.md exists with one example payload for each of the five event types.

Implementation note (2026-06-15): completed docs/contracts/ops-hub-activity-core-event-payloads.md. It documents the Inter-Hub request envelope, shared validation rules, idempotency expectations, forbidden payload material, expected API errors, and one example for each activity-core event type.

---

T06 - Validate fallback-first intake

id: IHUB-WP-0022-T06
status: wait
priority: medium
depends_on: T01
state_hub_task_id: "38b54991-bed2-4f9d-bede-bea35821b1ef"

Before enabling per-entity Inter-Hub submission, accept and review the State Hub fallback evidence path that activity-core already supports.

Validation path:

• Trigger or review an ops_inventory_probe fallback summary.
• Confirm it contains enough non-secret evidence to preserve operating continuity while Inter-Hub submission is gated.
• Record which evidence cannot be represented well in fallback summaries and should move to Inter-Hub first.
• Decide whether ACTIVITY-WP-0007/T06 may close with Inter-Hub submission explicitly deferred, or whether live Inter-Hub submission is a hard closure gate.

Exit criteria: docs/evidence/ops-hub-activity-core-fallback-validation.md records fallback evidence, gaps, and the closure recommendation.

Implementation note (2026-06-15): created docs/evidence/ops-hub-activity-core-fallback-validation.md. Activity-core's fallback sink is implemented and locally tested, but a direct State Hub query for event_type=ops_inventory_probe returned no live events. This task remains waiting on a disabled/manual activity-core probe or other live fallback evidence before it can close.

---

T07 - Run end-to-end Inter-Hub submission smoke

id: IHUB-WP-0022-T07
status: wait
priority: high
depends_on: T03,T04,T05
state_hub_task_id: "23baee9b-d710-42c8-9a19-f936bd237444"

Run a controlled submission from activity-core or a fixture that uses the same environment variables and mapping shape:

• INTER_HUB_URL
• OPS_HUB_KEY
• OPS_HUB_WIDGET_MAPPING

Smoke checks:

• One event per evidence type is accepted by POST /api/v2/interaction-events.
• Event type enforcement rejects an undeclared event type.
• Metadata is persisted and returned by the relevant list/show endpoint.
• API rate-limit and hub-registry reads do not hit COUNT decode failures.
• Failure mode is clean when config is absent, matching activity-core's current gated behavior.

Exit criteria: non-secret smoke evidence is recorded and activity-core can enable the Inter-Hub sink in its controlled environment.

Current wait reason (2026-06-15): depends on live manifest/widgets from T03, runtime key provisioning from T04, and activity-core implementing actual Inter-Hub submission beyond its current deferred sink.

---

T08 - Coordinate ACTIVITY-WP-0007 closure handoff

id: IHUB-WP-0022-T08
status: wait
priority: medium
depends_on: T06,T07
state_hub_task_id: "4a7ed0ed-552e-42d3-a90f-1efd52b8851e"

Send a closure decision or handoff back to activity-core.

The handoff must state:

• Whether ACTIVITY-WP-0007/T06 can close on fallback evidence with explicit Inter-Hub deferral.
• Or whether live Inter-Hub submission is now ready and should be required before closure.
• Which config values activity-core needs, naming only secret references and never secret values.
• Which widgets/event types are active.
• Which smoke evidence was collected.

Exit criteria: activity-core has enough non-secret evidence and configuration shape to close or unblock ACTIVITY-WP-0007/T06.

Current wait reason (2026-06-15): closure handoff depends on either a live State Hub fallback event plus an explicit Inter-Hub deferral decision, or a successful Inter-Hub submission smoke.

Exit Criteria Summary

Task	Deliverable	Status
T01	docs/research/ops-hub-evidence-intake-current-state.md	done
T02	docs/contracts/ops-hub-activity-core-mapping.md	done
T03	Active ops-hub manifest and seed widgets	wait
T04	OPS_HUB_KEY stored outside Git and smokeable	wait
T05	docs/contracts/ops-hub-activity-core-event-payloads.md	done
T06	docs/evidence/ops-hub-activity-core-fallback-validation.md	wait
T07	End-to-end Inter-Hub submission smoke evidence	wait
T08	activity-core closure handoff	wait

Binding Principles

• Governed vocabulary first: every event type must come from an active manifest before activity-core sends it.
• Secret custody stays out of Git: workplans may name paths and fields, never key values.
• Fallback before activation: State Hub fallback evidence remains the safety path until the Inter-Hub widget/API-key path is verified.
• Aggregate first, split later: use aggregate widgets when entity identity is ambiguous, then move to per-entity widgets only when stable.
• No manual DB access by default: use the documented v2 API unless the operator explicitly approves a fallback.