generated from coulomb/repo-seed
tegwick
0c76ccb28b
Some checks failed
ci / validate-registry (push) Has been cancelled
llm-connect came up locally (mock provider, 127.0.0.1:8080), unblocking this task. reuse_surface/plan_check.py: build_rerank_prompt/request_rerank/apply_rerank, reusing llm_bridge.execute_prompt/extract_json_object (same pattern as maintain_llm.py's request_maintain_patches). New schema schemas/plan-check-rerank.schema.json rejects malformed responses (missing fields, non-JSON, invented candidate ids outside the input set) rather than guessing. Per design principle 3 (deterministic matches always rank first), apply_rerank appends LLM-scored entries after the deterministic list (kind: 'llm') instead of reordering it -- the trusted base result is identical with or without the rerank pass. Graceful skip via a 'notes' field when LLM_CONNECT_URL is unset or the response is malformed, mirroring maintain.py's no_llm/skip pattern. New --llm-url/--no-llm CLI flags. Also extended plan-check-result.schema.json for 'notes' and the (pre-existing, previously unschema'd) 'filed_capability_request' field. 9 new pytest cases; 89 total pass. Live-verified against the running llm-connect instance: it correctly rejected the mock provider's non-JSON response and surfaced the skip note, with the deterministic verdict and match order completely unaffected. REUSE-WP-0018 is now fully done (T01-T06). Updated docs/IntentScopeGapAnalysis.md priority 29 to Closed and the workplan's own frontmatter status to finished. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
171 lines
7.6 KiB
Markdown
171 lines
7.6 KiB
Markdown
# Plan Check
|
||
|
||
**Repository:** `reuse-surface`
|
||
**Artifact:** `specs/PlanCheck.md`
|
||
**Status:** Implemented (T02 deterministic matching, T03 LLM rerank, T04
|
||
State Hub bridge, T05 ecosystem convention, T06 docs/CI all shipped)
|
||
**Schema:** `schemas/plan-check-result.schema.json`
|
||
|
||
---
|
||
|
||
## 1. Purpose
|
||
|
||
`plan-check` closes the consumption loop the registry has lacked since
|
||
inception: nothing today nudges an agent planning work in a sibling repo to
|
||
query the federated capability index before building. The registry has been
|
||
write-mostly. `plan-check` is the query-before-build step — advisory, not a
|
||
gate — that matches a draft workplan or a free-text intent against the
|
||
federated index and returns a verdict: reuse it, extend it, or it's genuinely
|
||
new.
|
||
|
||
This spec covers the deterministic matching core (T02) and its interfaces
|
||
with the optional LLM rerank (T03) and the State Hub capability-request
|
||
bridge (T04). It does not cover the ecosystem rollout (T05) — the
|
||
session-protocol convention is designed once this spec is implemented and
|
||
dogfooded in this repo.
|
||
|
||
## 2. Design principles
|
||
|
||
1. **Deterministic core, LLM assist optional.** Keyword/scope/relation
|
||
matching against `registry/indexes/federated.yaml` works with no external
|
||
dependency. `LLM_CONNECT_URL` unlocks a semantic rerank pass; its absence
|
||
degrades gracefully, never blocks.
|
||
2. **Advisory, not blocking.** `plan-check` never fails a workplan into
|
||
existence or refuses to let one be created. Adoption comes from the tool
|
||
being useful and from convention (REUSE-WP-0018-T05), not from a gate.
|
||
3. **Deterministic matches always rank first.** Whatever the LLM rerank
|
||
proposes, it is listed after — and clearly labeled apart from — the
|
||
deterministic candidates, so a human or agent can trust the base result
|
||
even with `--no-llm`.
|
||
4. **Every invocation is a data point.** `--record-outcome` writes an
|
||
append-only fact. This is the raw material for REUSE-WP-0019's reuse
|
||
telemetry — `plan-check` and telemetry share one event schema from day
|
||
one so nothing needs migrating later.
|
||
|
||
## 3. Input model
|
||
|
||
Two input shapes, mutually exclusive:
|
||
|
||
| Input | How it's read |
|
||
|---|---|
|
||
| Workplan file | Parsed like `registry_update.py`'s git-diff signal collector: YAML frontmatter (`---`-delimited) plus the Markdown body. `title`, the one-liner in the intro paragraph, and any `## Problem statement` / `## Core Idea` heading content feed the match blob. |
|
||
| `--intent "free text"` | Used verbatim as the match blob. |
|
||
|
||
Both normalize to the same internal `MatchQuery { text: str, tokens: set[str] }`
|
||
before matching — the matcher does not care which input shape it came from.
|
||
|
||
## 4. Matching pipeline
|
||
|
||
Reuses the token-Jaccard approach already proven in `overlaps.py`
|
||
(`TOKEN_RE`, `_tokens`) rather than inventing a second scoring method in the
|
||
same codebase:
|
||
|
||
1. **Tokenize** the query blob with the existing `TOKEN_RE`.
|
||
2. **Score** against every federated capability's blob (`name` + `summary` +
|
||
`tags` + `discovery.intent` + `discovery.includes`, mirroring
|
||
`overlaps._entry_blob`) via Jaccard similarity.
|
||
3. **Rank signal, not just tie-break:** among candidates within
|
||
`--tie-window` (default 0.05) of the top score, prefer the entry with the
|
||
higher discovery/availability vector (a more mature capability is a safer
|
||
reuse bet at equal textual match).
|
||
4. **Relation expansion:** if a top candidate has `relations.supports` or
|
||
`relations.related_to` entries, surface them as secondary candidates
|
||
labeled `related` rather than silently dropped.
|
||
|
||
## 5. Verdict model
|
||
|
||
| Verdict | Condition | Meaning |
|
||
|---|---|---|
|
||
| `reuse` | top score ≥ `--reuse-threshold` (default 0.45) | An existing capability already covers this need — link it, don't rebuild it. |
|
||
| `extend` | top score in `[--extend-threshold, --reuse-threshold)` (default 0.22–0.45) | Scope overlaps a capability closely enough that extending it is very likely cheaper than a new one — surfaced with an explicit "consider extending" framing. |
|
||
| `new` | top score < `--extend-threshold`, or no federated capabilities exist | No good match. Proceed; optionally file a capability request (T04). |
|
||
|
||
Thresholds are CLI flags, not hardcoded, because the right cutoff will drift
|
||
as coverage and entry quality improve (see REUSE-WP-0017). Defaults come from
|
||
the same threshold reasoning as `overlaps.py`'s `--threshold 0.28` default,
|
||
shifted since plan-check matches short intent text against short summaries
|
||
rather than long entry-to-entry blobs.
|
||
|
||
## 6. Output
|
||
|
||
### Markdown (TTY default)
|
||
|
||
```text
|
||
# Plan check: reuse | extend | new
|
||
|
||
**Query:** <workplan title or intent text>
|
||
|
||
## Top match
|
||
- `capability.infotech.issue-tracking` (score 0.52, D4/A2/C2/R1) — issue-core
|
||
> Unified Python/CLI interface for issue tracking across Gitea, GitHub, and GitLab...
|
||
|
||
## Other candidates
|
||
- ...
|
||
|
||
## Related (via relations.*)
|
||
- ...
|
||
|
||
Verdict: REUSE — link this capability instead of building new.
|
||
```
|
||
|
||
### JSON (`--format json`, agent-consumable)
|
||
|
||
Validated against `schemas/plan-check-result.schema.json`:
|
||
|
||
```json
|
||
{
|
||
"query": {"source": "workplan|intent", "text": "...", "workplan_id": "..."},
|
||
"verdict": "reuse|extend|new",
|
||
"top_score": 0.52,
|
||
"matches": [
|
||
{"id": "capability.infotech.issue-tracking", "score": 0.52, "vector": "D4/A2/C2/R1", "owner": "issue-core", "kind": "deterministic|llm|related"}
|
||
],
|
||
"federated_index_updated": "2026-07-06",
|
||
"federated_index_stale_warning": null
|
||
}
|
||
```
|
||
|
||
`federated_index_stale_warning` is set when `federated.yaml`'s `updated`
|
||
field is older than 14 days — a cheap freshness signal until
|
||
REUSE-WP-0019's automatic recompose lands.
|
||
|
||
## 7. Outcome recording (`--record-outcome`)
|
||
|
||
Append-only JSONL under `registry/telemetry/plan-check-events.jsonl`
|
||
(reuse-surface's own copy; sibling repos get their own via the same schema
|
||
when they adopt T05). One line per invocation:
|
||
|
||
```json
|
||
{"ts": "2026-07-07T10:00:00Z", "consumer_repo": "reuse-surface", "capability_id": "capability.infotech.issue-tracking", "verdict": "reuse", "outcome": "reused|extended|new|skipped", "source": "plan-check"}
|
||
```
|
||
|
||
This schema is deliberately identical to the reuse-event schema
|
||
REUSE-WP-0019-T04 will implement server-side — `plan-check`'s local JSONL
|
||
file is the fallback path when the hub is unreachable, and the same record
|
||
shape posts to `POST /v1/reuse-events` once T04/WP-0019 exist. No local
|
||
telemetry write ever blocks the primary command.
|
||
|
||
## 8. Relationship to T03 (LLM rerank) and T04 (State Hub bridge)
|
||
|
||
- **T03** adds an optional post-pass: send the deterministic top-N candidates
|
||
plus the query text to `llm-connect` for a semantic confidence score and
|
||
possible reordering *within* the deterministic candidate set. It does not
|
||
invent new candidates outside what deterministic matching already found —
|
||
keeps the "deterministic matches always rank first" guarantee simple to
|
||
reason about. Schema-constrained JSON response, mirrors the
|
||
`maintain_llm.py` pattern (graceful skip on missing `LLM_CONNECT_URL`,
|
||
reject malformed responses rather than guess).
|
||
- **T04** wires `new` verdicts to `POST /messages/` (State Hub
|
||
`request_capability`-equivalent) and reads back open capability requests
|
||
for `report gaps` to list unmatched. Deferred to its own task since it
|
||
depends on State Hub API shapes this spec does not need to pin down yet.
|
||
|
||
## 9. Non-goals
|
||
|
||
- Blocking or gating workplan creation (§2.2).
|
||
- Embedding-based / vector-similarity matching — token-Jaccard is the
|
||
deterministic core for consistency with `overlaps.py`; embeddings are an
|
||
LLM-rerank concern (T03), not a second deterministic path.
|
||
- Editing sibling repos' rules files directly — that's the template
|
||
propagation mechanism's job (T05), not this tool's.
|