generated from coulomb/repo-seed
Some checks failed
ci / validate-registry (push) Has been cancelled
llm-connect came up locally (mock provider, 127.0.0.1:8080), unblocking this task. reuse_surface/plan_check.py: build_rerank_prompt/request_rerank/apply_rerank, reusing llm_bridge.execute_prompt/extract_json_object (same pattern as maintain_llm.py's request_maintain_patches). New schema schemas/plan-check-rerank.schema.json rejects malformed responses (missing fields, non-JSON, invented candidate ids outside the input set) rather than guessing. Per design principle 3 (deterministic matches always rank first), apply_rerank appends LLM-scored entries after the deterministic list (kind: 'llm') instead of reordering it -- the trusted base result is identical with or without the rerank pass. Graceful skip via a 'notes' field when LLM_CONNECT_URL is unset or the response is malformed, mirroring maintain.py's no_llm/skip pattern. New --llm-url/--no-llm CLI flags. Also extended plan-check-result.schema.json for 'notes' and the (pre-existing, previously unschema'd) 'filed_capability_request' field. 9 new pytest cases; 89 total pass. Live-verified against the running llm-connect instance: it correctly rejected the mock provider's non-JSON response and surfaced the skip note, with the deterministic verdict and match order completely unaffected. REUSE-WP-0018 is now fully done (T01-T06). Updated docs/IntentScopeGapAnalysis.md priority 29 to Closed and the workplan's own frontmatter status to finished. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
258 lines
11 KiB
Markdown
258 lines
11 KiB
Markdown
---
|
||
id: REUSE-WP-0018
|
||
type: workplan
|
||
title: "plan-check: close the consumption loop for capability reuse"
|
||
domain: infotech
|
||
repo: reuse-surface
|
||
status: finished
|
||
owner: claude-code
|
||
topic_slug: helix-forge
|
||
created: "2026-07-06"
|
||
updated: "2026-07-07"
|
||
state_hub_workstream_id: "cd8683ff-6e6c-4f6c-a62f-565bd55113ea"
|
||
reuse_check: "new — dogfooded 2026-07-07 via reuse-surface plan-check against the full 61-capability federated index; no existing capability covers query-before-build matching"
|
||
---
|
||
|
||
# plan-check: close the consumption loop for capability reuse
|
||
|
||
The registry is write-mostly: nothing nudges an agent planning work in a
|
||
sibling repo to query the federated index before building. The State Hub has a
|
||
`request_capability` / `list_capability_requests` flow, but it is not bridged
|
||
to reuse-surface data. This workplan adds the **query-before-build** step to
|
||
the ecosystem: a `plan-check` command that matches a draft workplan or intent
|
||
text against federated capabilities, wired into every repo's session protocol,
|
||
with a two-way bridge to State Hub capability requests.
|
||
|
||
This is the highest-leverage coherence mechanism: it is what stops 61 repos
|
||
from independently reinventing the same thing.
|
||
|
||
**Depends on REUSE-WP-0017** for meaningful match rates — plan-check against a
|
||
mostly-empty index produces noise and erodes trust. T01–T03 (design +
|
||
implementation) can proceed in parallel with the coverage campaign; the
|
||
ecosystem rollout (T05) waits for coverage.
|
||
|
||
## Proposed CLI surface
|
||
|
||
```bash
|
||
# Match a draft workplan against the federated index
|
||
reuse-surface plan-check workplans/XXX-WP-0042-something.md
|
||
|
||
# Free-text intent
|
||
reuse-surface plan-check --intent "parse invoices from email and file evidence"
|
||
|
||
# Agent-consumable output
|
||
reuse-surface plan-check --format json workplans/XXX-WP-0042.md
|
||
|
||
# Record the outcome (feeds R-axis evidence, WP-0019)
|
||
reuse-surface plan-check ... --record-outcome reused|extended|new
|
||
```
|
||
|
||
**Verdict classes per match:** `reuse` (capability covers the need — link it),
|
||
`extend` (close scope — extend instead of duplicating), `new` (no match —
|
||
proceed, optionally file a capability request). Output includes capability id,
|
||
maturity vector, owning repo, and consumer guidance.
|
||
|
||
## Design principles
|
||
|
||
1. **Deterministic core, LLM assist optional** — keyword/scope/relation
|
||
matching against index rows works without llm-connect; semantic matching
|
||
via llm-connect improves recall when available (same optional-backend
|
||
pattern as WP-0016).
|
||
2. **Advisory, not blocking** — plan-check informs; it never vetoes a
|
||
workplan. Adoption comes from usefulness plus protocol convention, not
|
||
gates.
|
||
3. **One line in the session protocol** — sibling integration is a single
|
||
instruction in the shared `.claude/rules/` template: run plan-check before
|
||
creating a workplan. No per-repo bespoke wiring.
|
||
4. **Every check is a data point** — plan-check invocations and recorded
|
||
outcomes are the raw material for reuse telemetry (WP-0019-T04).
|
||
|
||
## Dependencies
|
||
|
||
| Dependency | Owner | Notes |
|
||
|---|---|---|
|
||
| REUSE-WP-0017 coverage | reuse-surface | rollout gate for T05 |
|
||
| federated.yaml freshness | reuse-surface | improved by WP-0019 auto-recompose |
|
||
| llm-connect | llm-connect | optional semantic matching backend |
|
||
| State Hub capability-request API | state-hub | `request_capability`, `list_capability_requests` |
|
||
| .claude/rules template propagation | the-custodian / repo-seed | mechanism for T05 rollout |
|
||
|
||
---
|
||
|
||
## Design Matching Semantics And Verdict Model
|
||
|
||
```task
|
||
id: REUSE-WP-0018-T01
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "c7b3407a-2b9d-46ba-ae9c-7ab12235f0ba"
|
||
```
|
||
|
||
Design doc `specs/PlanCheck.md`:
|
||
|
||
- Input model: workplan file (frontmatter + body) vs free intent text
|
||
- Match pipeline: token/keyword scoring over id, name, scope, guidance →
|
||
relation expansion → optional llm-connect semantic rerank
|
||
- Verdict thresholds and tie-breaking (maturity vector as rank signal:
|
||
prefer higher D/A when scopes tie)
|
||
- JSON output schema (`schemas/plan-check-result.schema.json`)
|
||
- Outcome-recording format (append-only JSONL under `registry/telemetry/`,
|
||
schema shared with WP-0019-T04)
|
||
|
||
## Implement Deterministic plan-check
|
||
|
||
```task
|
||
id: REUSE-WP-0018-T02
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "24911d0e-41bc-4c8c-a39a-7dc8ec65d8c6"
|
||
```
|
||
|
||
Implemented in `reuse_surface/plan_check.py` + `plan-check` CLI command:
|
||
|
||
- Reuses `overlaps.py`'s `TOKEN_RE`/Jaccard approach rather than a second
|
||
scoring method (per spec §4)
|
||
- Reads `registry/indexes/federated.yaml`, falls back to the local index,
|
||
warns when the composed index is >14 days stale
|
||
- Markdown (TTY default) and `--format json` (validated against
|
||
`schemas/plan-check-result.schema.json`)
|
||
- 11 pytest cases: verdict thresholds, workplan-file parsing, intent-text
|
||
parsing, tie-window vector ranking, staleness warning, CLI wiring
|
||
- `--record-outcome` appends to `registry/telemetry/plan-check-events.jsonl`
|
||
per the shared `schemas/reuse-event.schema.json` (WP-0019-T04 will post the
|
||
same shape to the hub)
|
||
|
||
## Add llm-connect Semantic Rerank
|
||
|
||
```task
|
||
id: REUSE-WP-0018-T03
|
||
status: done
|
||
priority: medium
|
||
state_hub_task_id: "9040505a-238e-4815-ae1a-8c5f8c12afa9"
|
||
```
|
||
|
||
Unblocked 2026-07-07 — `llm-connect` running locally (mock provider,
|
||
`127.0.0.1:8080`). Implemented in `reuse_surface/plan_check.py`:
|
||
|
||
- `build_rerank_prompt`/`request_rerank`/`apply_rerank`, reusing
|
||
`llm_bridge.execute_prompt`/`extract_json_object` (same pattern as
|
||
`maintain_llm.py`'s `request_maintain_patches`)
|
||
- New schema `schemas/plan-check-rerank.schema.json`; malformed responses
|
||
(missing fields, non-JSON, invented candidate ids not in the input set)
|
||
are rejected via schema validation + an id-membership filter, never
|
||
guessed at
|
||
- `apply_rerank` **appends** LLM-scored entries after the deterministic
|
||
list (`kind: "llm"`) rather than reordering it — deterministic matches
|
||
keep their original order/scores untouched, per design principle 3
|
||
- Graceful skip (`ValueError` → a `notes` field in the result, mirroring
|
||
`maintain.py`'s `no_llm`/skip pattern) when `LLM_CONNECT_URL` is unset
|
||
or the LLM response is malformed; `--no-llm` skips without attempting a
|
||
call at all
|
||
- New CLI flags: `--llm-url`, `--no-llm`
|
||
- `schemas/plan-check-result.schema.json` extended for `notes` and the
|
||
(pre-existing, previously unschema'd) `filed_capability_request` field
|
||
- 9 new pytest cases (valid, malformed, non-JSON, invented-id-filtering,
|
||
append-not-reorder, graceful skip, `--no-llm` short-circuit, successful
|
||
integration) — 89 total pass
|
||
- **Live-verified**, not just mocked: real HTTP round-trip against the
|
||
running mock `llm-connect` instance correctly rejected its non-JSON mock
|
||
response and surfaced the skip note, while the deterministic verdict and
|
||
match order stayed completely unaffected
|
||
|
||
## Bridge State Hub Capability Requests
|
||
|
||
```task
|
||
id: REUSE-WP-0018-T04
|
||
status: done
|
||
priority: medium
|
||
state_hub_task_id: "8249dca7-18ca-4150-9671-ca87e6a5ad88"
|
||
```
|
||
|
||
Implemented in `reuse_surface/statehub_bridge.py` against the live local
|
||
State Hub HTTP API (`http://127.0.0.1:8000`, confirmed reachable):
|
||
|
||
- **new → request:** `plan-check --file-request` files a State Hub
|
||
capability request (`POST /capability-requests/`) on a `new` verdict
|
||
- **request → gap:** `report gaps --check-capability-requests` lists open
|
||
requests (`GET /capability-requests/`, filtered on `status` — a request
|
||
can carry a `catalog_entry_id` while still `requested`/routed, so status
|
||
is the correct open/closed signal, not `catalog_entry_id` presence).
|
||
Opt-in flag so `report gaps` stays fast and offline-safe by default
|
||
- Both degrade to `None`/graceful-skip when the hub is unreachable; neither
|
||
ever blocks the primary command
|
||
- List endpoint observed taking ~7s in production with only 5 rows; uses a
|
||
20s timeout distinct from the 5s health-check timeout
|
||
- 5 pytest cases with the hub client fully mocked (never hits the network
|
||
in the default test run)
|
||
- **Not done:** marking matched requests resolved with a pointer to the
|
||
capability id — deferred, no current caller needs it yet
|
||
|
||
## Ecosystem Rollout: Session-Protocol Integration
|
||
|
||
```task
|
||
id: REUSE-WP-0018-T05
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "c232a669-b263-4978-a8b1-3721b8bfaa68"
|
||
```
|
||
|
||
Gate cleared 2026-07-07: WP-0017 finished, `federated.yaml` composed with
|
||
61 real capabilities (up from the original 24). Dogfooded immediately —
|
||
`plan-check` against "unified interface for issue tracking across Gitea
|
||
GitHub GitLab" now correctly returns `extend` pointing at
|
||
`capability.infotech.issue-tracking`, where before WP-0017 published it
|
||
would have returned `new`. That's the whole point of sequencing T05 after
|
||
WP-0017 landed.
|
||
|
||
- [x] Drafted the convention in `.claude/rules/workplan-convention.md`
|
||
("Query before building" section): run `plan-check`, record
|
||
`reuse_check: reuse|extend|new — <capability-id?>` in frontmatter,
|
||
advisory only
|
||
- [x] Sent State Hub message to `custodian-agent` proposing propagation
|
||
via the fix-consistency/repo-seed shared rules template (message
|
||
`13cecd08-14e8-46d9-880c-65b8fef857cb`) — propagation itself depends on
|
||
the-custodian's own template mechanism, not something reuse-surface
|
||
can push directly into sibling repos
|
||
- [x] Added `reuse_check` frontmatter to this repo's own REUSE-WP-0018 and
|
||
REUSE-WP-0019 as the reference implementation, both dogfooded for real
|
||
(both correctly verdict `new`)
|
||
|
||
## Docs, CI, SCOPE
|
||
|
||
```task
|
||
id: REUSE-WP-0018-T06
|
||
status: done
|
||
priority: low
|
||
state_hub_task_id: "ff0a91eb-cc41-487b-b726-8c2f84860399"
|
||
```
|
||
|
||
- `tools/README.md` command reference for `plan-check` and
|
||
`report gaps --check-capability-requests`; `docs/RegistryFederation.md`
|
||
"Query before building" consumer section; `SCOPE.md` "What Is Possible Now"
|
||
- CI: informational `plan-check --intent "smoke test" --format json` run
|
||
added to `.gitea/workflows/ci.yml`
|
||
- `docs/IntentScopeGapAnalysis.md`: priority 29 added, marked **Partial**
|
||
(T02/T04 shipped; T03/T05 remain open — not flipped fully to "shipped"
|
||
since the ecosystem rollout hasn't happened yet)
|
||
|
||
---
|
||
|
||
## Acceptance
|
||
|
||
- [x] `plan-check` returns reuse/extend/new verdicts for workplan files and intent text, with and without llm-connect (T03 shipped 2026-07-07, live-verified against a running instance)
|
||
- [x] JSON output validates against the published schema
|
||
- [x] `new` verdicts can file State Hub capability requests; `report gaps` lists unmatched open requests
|
||
- [x] Session-protocol convention drafted; propagation *proposed* to custodian-agent
|
||
2026-07-07 (message `13cecd08-14e8-46d9-880c-65b8fef857cb`) — actual
|
||
adoption into the shared rules template is the-custodian's call, not
|
||
something reuse-surface can complete unilaterally
|
||
- [x] reuse-surface itself records `reuse_check` in its own workplans
|
||
(REUSE-WP-0018, REUSE-WP-0019) as the reference implementation
|
||
|
||
## Out of scope
|
||
|
||
- Blocking/gating workplan creation on plan-check results
|
||
- Reuse telemetry aggregation and R-axis evidence (WP-0019)
|
||
- Embedding-based matching (llm-connect rerank only)
|
||
- Editing sibling repos' rules files directly (owned by template propagation)
|