Files
reuse-surface/workplans/REUSE-WP-0018-plan-check-consumption-loop.md
tegwick 0c76ccb28b
Some checks failed
ci / validate-registry (push) Has been cancelled
REUSE-WP-0018-T03: LLM semantic rerank for plan-check
llm-connect came up locally (mock provider, 127.0.0.1:8080), unblocking
this task.

reuse_surface/plan_check.py: build_rerank_prompt/request_rerank/apply_rerank,
reusing llm_bridge.execute_prompt/extract_json_object (same pattern as
maintain_llm.py's request_maintain_patches). New schema
schemas/plan-check-rerank.schema.json rejects malformed responses (missing
fields, non-JSON, invented candidate ids outside the input set) rather than
guessing.

Per design principle 3 (deterministic matches always rank first),
apply_rerank appends LLM-scored entries after the deterministic list
(kind: 'llm') instead of reordering it -- the trusted base result is
identical with or without the rerank pass. Graceful skip via a 'notes'
field when LLM_CONNECT_URL is unset or the response is malformed, mirroring
maintain.py's no_llm/skip pattern. New --llm-url/--no-llm CLI flags.

Also extended plan-check-result.schema.json for 'notes' and the
(pre-existing, previously unschema'd) 'filed_capability_request' field.

9 new pytest cases; 89 total pass. Live-verified against the running
llm-connect instance: it correctly rejected the mock provider's non-JSON
response and surfaced the skip note, with the deterministic verdict and
match order completely unaffected.

REUSE-WP-0018 is now fully done (T01-T06). Updated
docs/IntentScopeGapAnalysis.md priority 29 to Closed and the workplan's
own frontmatter status to finished.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-07 17:27:39 +02:00

11 KiB
Raw Blame History

id, type, title, domain, repo, status, owner, topic_slug, created, updated, state_hub_workstream_id, reuse_check
id type title domain repo status owner topic_slug created updated state_hub_workstream_id reuse_check
REUSE-WP-0018 workplan plan-check: close the consumption loop for capability reuse infotech reuse-surface finished claude-code helix-forge 2026-07-06 2026-07-07 cd8683ff-6e6c-4f6c-a62f-565bd55113ea new — dogfooded 2026-07-07 via reuse-surface plan-check against the full 61-capability federated index; no existing capability covers query-before-build matching

plan-check: close the consumption loop for capability reuse

The registry is write-mostly: nothing nudges an agent planning work in a sibling repo to query the federated index before building. The State Hub has a request_capability / list_capability_requests flow, but it is not bridged to reuse-surface data. This workplan adds the query-before-build step to the ecosystem: a plan-check command that matches a draft workplan or intent text against federated capabilities, wired into every repo's session protocol, with a two-way bridge to State Hub capability requests.

This is the highest-leverage coherence mechanism: it is what stops 61 repos from independently reinventing the same thing.

Depends on REUSE-WP-0017 for meaningful match rates — plan-check against a mostly-empty index produces noise and erodes trust. T01T03 (design + implementation) can proceed in parallel with the coverage campaign; the ecosystem rollout (T05) waits for coverage.

Proposed CLI surface

# Match a draft workplan against the federated index
reuse-surface plan-check workplans/XXX-WP-0042-something.md

# Free-text intent
reuse-surface plan-check --intent "parse invoices from email and file evidence"

# Agent-consumable output
reuse-surface plan-check --format json workplans/XXX-WP-0042.md

# Record the outcome (feeds R-axis evidence, WP-0019)
reuse-surface plan-check ... --record-outcome reused|extended|new

Verdict classes per match: reuse (capability covers the need — link it), extend (close scope — extend instead of duplicating), new (no match — proceed, optionally file a capability request). Output includes capability id, maturity vector, owning repo, and consumer guidance.

Design principles

  1. Deterministic core, LLM assist optional — keyword/scope/relation matching against index rows works without llm-connect; semantic matching via llm-connect improves recall when available (same optional-backend pattern as WP-0016).
  2. Advisory, not blocking — plan-check informs; it never vetoes a workplan. Adoption comes from usefulness plus protocol convention, not gates.
  3. One line in the session protocol — sibling integration is a single instruction in the shared .claude/rules/ template: run plan-check before creating a workplan. No per-repo bespoke wiring.
  4. Every check is a data point — plan-check invocations and recorded outcomes are the raw material for reuse telemetry (WP-0019-T04).

Dependencies

Dependency Owner Notes
REUSE-WP-0017 coverage reuse-surface rollout gate for T05
federated.yaml freshness reuse-surface improved by WP-0019 auto-recompose
llm-connect llm-connect optional semantic matching backend
State Hub capability-request API state-hub request_capability, list_capability_requests
.claude/rules template propagation the-custodian / repo-seed mechanism for T05 rollout

Design Matching Semantics And Verdict Model

id: REUSE-WP-0018-T01
status: done
priority: high
state_hub_task_id: "c7b3407a-2b9d-46ba-ae9c-7ab12235f0ba"

Design doc specs/PlanCheck.md:

  • Input model: workplan file (frontmatter + body) vs free intent text
  • Match pipeline: token/keyword scoring over id, name, scope, guidance → relation expansion → optional llm-connect semantic rerank
  • Verdict thresholds and tie-breaking (maturity vector as rank signal: prefer higher D/A when scopes tie)
  • JSON output schema (schemas/plan-check-result.schema.json)
  • Outcome-recording format (append-only JSONL under registry/telemetry/, schema shared with WP-0019-T04)

Implement Deterministic plan-check

id: REUSE-WP-0018-T02
status: done
priority: high
state_hub_task_id: "24911d0e-41bc-4c8c-a39a-7dc8ec65d8c6"

Implemented in reuse_surface/plan_check.py + plan-check CLI command:

  • Reuses overlaps.py's TOKEN_RE/Jaccard approach rather than a second scoring method (per spec §4)
  • Reads registry/indexes/federated.yaml, falls back to the local index, warns when the composed index is >14 days stale
  • Markdown (TTY default) and --format json (validated against schemas/plan-check-result.schema.json)
  • 11 pytest cases: verdict thresholds, workplan-file parsing, intent-text parsing, tie-window vector ranking, staleness warning, CLI wiring
  • --record-outcome appends to registry/telemetry/plan-check-events.jsonl per the shared schemas/reuse-event.schema.json (WP-0019-T04 will post the same shape to the hub)

Add llm-connect Semantic Rerank

id: REUSE-WP-0018-T03
status: done
priority: medium
state_hub_task_id: "9040505a-238e-4815-ae1a-8c5f8c12afa9"

Unblocked 2026-07-07 — llm-connect running locally (mock provider, 127.0.0.1:8080). Implemented in reuse_surface/plan_check.py:

  • build_rerank_prompt/request_rerank/apply_rerank, reusing llm_bridge.execute_prompt/extract_json_object (same pattern as maintain_llm.py's request_maintain_patches)
  • New schema schemas/plan-check-rerank.schema.json; malformed responses (missing fields, non-JSON, invented candidate ids not in the input set) are rejected via schema validation + an id-membership filter, never guessed at
  • apply_rerank appends LLM-scored entries after the deterministic list (kind: "llm") rather than reordering it — deterministic matches keep their original order/scores untouched, per design principle 3
  • Graceful skip (ValueError → a notes field in the result, mirroring maintain.py's no_llm/skip pattern) when LLM_CONNECT_URL is unset or the LLM response is malformed; --no-llm skips without attempting a call at all
  • New CLI flags: --llm-url, --no-llm
  • schemas/plan-check-result.schema.json extended for notes and the (pre-existing, previously unschema'd) filed_capability_request field
  • 9 new pytest cases (valid, malformed, non-JSON, invented-id-filtering, append-not-reorder, graceful skip, --no-llm short-circuit, successful integration) — 89 total pass
  • Live-verified, not just mocked: real HTTP round-trip against the running mock llm-connect instance correctly rejected its non-JSON mock response and surfaced the skip note, while the deterministic verdict and match order stayed completely unaffected

Bridge State Hub Capability Requests

id: REUSE-WP-0018-T04
status: done
priority: medium
state_hub_task_id: "8249dca7-18ca-4150-9671-ca87e6a5ad88"

Implemented in reuse_surface/statehub_bridge.py against the live local State Hub HTTP API (http://127.0.0.1:8000, confirmed reachable):

  • new → request: plan-check --file-request files a State Hub capability request (POST /capability-requests/) on a new verdict
  • request → gap: report gaps --check-capability-requests lists open requests (GET /capability-requests/, filtered on status — a request can carry a catalog_entry_id while still requested/routed, so status is the correct open/closed signal, not catalog_entry_id presence). Opt-in flag so report gaps stays fast and offline-safe by default
  • Both degrade to None/graceful-skip when the hub is unreachable; neither ever blocks the primary command
  • List endpoint observed taking ~7s in production with only 5 rows; uses a 20s timeout distinct from the 5s health-check timeout
  • 5 pytest cases with the hub client fully mocked (never hits the network in the default test run)
  • Not done: marking matched requests resolved with a pointer to the capability id — deferred, no current caller needs it yet

Ecosystem Rollout: Session-Protocol Integration

id: REUSE-WP-0018-T05
status: done
priority: high
state_hub_task_id: "c232a669-b263-4978-a8b1-3721b8bfaa68"

Gate cleared 2026-07-07: WP-0017 finished, federated.yaml composed with 61 real capabilities (up from the original 24). Dogfooded immediately — plan-check against "unified interface for issue tracking across Gitea GitHub GitLab" now correctly returns extend pointing at capability.infotech.issue-tracking, where before WP-0017 published it would have returned new. That's the whole point of sequencing T05 after WP-0017 landed.

  • Drafted the convention in .claude/rules/workplan-convention.md ("Query before building" section): run plan-check, record reuse_check: reuse|extend|new — <capability-id?> in frontmatter, advisory only
  • Sent State Hub message to custodian-agent proposing propagation via the fix-consistency/repo-seed shared rules template (message 13cecd08-14e8-46d9-880c-65b8fef857cb) — propagation itself depends on the-custodian's own template mechanism, not something reuse-surface can push directly into sibling repos
  • Added reuse_check frontmatter to this repo's own REUSE-WP-0018 and REUSE-WP-0019 as the reference implementation, both dogfooded for real (both correctly verdict new)

Docs, CI, SCOPE

id: REUSE-WP-0018-T06
status: done
priority: low
state_hub_task_id: "ff0a91eb-cc41-487b-b726-8c2f84860399"
  • tools/README.md command reference for plan-check and report gaps --check-capability-requests; docs/RegistryFederation.md "Query before building" consumer section; SCOPE.md "What Is Possible Now"
  • CI: informational plan-check --intent "smoke test" --format json run added to .gitea/workflows/ci.yml
  • docs/IntentScopeGapAnalysis.md: priority 29 added, marked Partial (T02/T04 shipped; T03/T05 remain open — not flipped fully to "shipped" since the ecosystem rollout hasn't happened yet)

Acceptance

  • plan-check returns reuse/extend/new verdicts for workplan files and intent text, with and without llm-connect (T03 shipped 2026-07-07, live-verified against a running instance)
  • JSON output validates against the published schema
  • new verdicts can file State Hub capability requests; report gaps lists unmatched open requests
  • Session-protocol convention drafted; propagation proposed to custodian-agent 2026-07-07 (message 13cecd08-14e8-46d9-880c-65b8fef857cb) — actual adoption into the shared rules template is the-custodian's call, not something reuse-surface can complete unilaterally
  • reuse-surface itself records reuse_check in its own workplans (REUSE-WP-0018, REUSE-WP-0019) as the reference implementation

Out of scope

  • Blocking/gating workplan creation on plan-check results
  • Reuse telemetry aggregation and R-axis evidence (WP-0019)
  • Embedding-based matching (llm-connect rerank only)
  • Editing sibling repos' rules files directly (owned by template propagation)