generated from coulomb/repo-seed
Compare commits
90 Commits
codex/wp-0
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 00713f86ca | |||
| fd7f25866a | |||
| 61fa33fc39 | |||
| 4762c49dfd | |||
| 11df32fa39 | |||
| e1058420da | |||
| 3e906c1dd4 | |||
| bee770fad7 | |||
| ba2228e889 | |||
| f4d782c997 | |||
| 14580eb206 | |||
| f874c790cc | |||
| 22222ac547 | |||
| f63287a087 | |||
| 855f7fef7c | |||
| 28ea672225 | |||
| 28fad1b248 | |||
| 084159e51c | |||
| f38ed6847c | |||
| 9c1a21aa52 | |||
| 20122bb565 | |||
| 12263c1634 | |||
| 0bc534cb70 | |||
| 99a15f54d6 | |||
| 6aca068154 | |||
| c9a56c4f05 | |||
| 74b713988c | |||
| 83c39a7aa6 | |||
| 2c3dad80d6 | |||
| 4706291a03 | |||
| e2f378be90 | |||
| 8bdaf73e3a | |||
| b3b013fa23 | |||
| e912ec0a0b | |||
| 324fbb3745 | |||
| d44d50f623 | |||
| dcd015ec8d | |||
| 1fdacb7d40 | |||
| 458eb410c4 | |||
| abcb2cebbc | |||
| b678741a75 | |||
| 96d331e0ca | |||
| b84b5623e0 | |||
| de8d184a4b | |||
| a2c8ba9442 | |||
| 90b1876059 | |||
| 9508c1e049 | |||
| 937b814d73 | |||
| 87a53b9825 | |||
| effea4d0d6 | |||
| d4f363af72 | |||
| 1913793658 | |||
| a4b39b3cb7 | |||
| 5c2262bcf2 | |||
| 43e7f7138f | |||
| 92eaf52bb6 | |||
| 9a320a95ee | |||
| 4ee4a0dd36 | |||
| 8f484cd855 | |||
| 9fa1d9e9b5 | |||
| 83d5044ff4 | |||
| 7851eae42f | |||
| a9baf5ae52 | |||
| f029d6bba9 | |||
| 1d77d86941 | |||
| fa59289f81 | |||
| f690794acd | |||
| fc034bd821 | |||
| f9fac2da7c | |||
| 34fb5721fd | |||
| a59c5ee63a | |||
| b7b9cbcc9b | |||
| 2dfe5c6dd6 | |||
| d9b0f5b32a | |||
| 750985839f | |||
| 18ac5fe2ba | |||
| a921f714f3 | |||
| 27b4deb4d2 | |||
| 0b16167769 | |||
| d14cb316c7 | |||
| e3c7c45495 | |||
| 2796fc5816 | |||
| bc08977f85 | |||
| f38325a5ba | |||
| 90bae27237 | |||
| a6e1e2f16a | |||
| e98157402c | |||
| ee0f2a7e5d | |||
| 61ba07711e | |||
| 00b57d5091 |
20
.claude/rules/agents.md
Normal file
20
.claude/rules/agents.md
Normal file
@@ -0,0 +1,20 @@
|
||||
## Kaizen Agents
|
||||
|
||||
Specialized agent personas available on demand via the state-hub MCP.
|
||||
|
||||
**Discover:** `list_kaizen_agents()` — returns all agents with name, description, category
|
||||
**Load:** `get_kaizen_agent("tdd-workflow")` — returns full instructions; read and follow them
|
||||
|
||||
Common agents:
|
||||
|
||||
| Agent | Category | When to use |
|
||||
|-------|----------|-------------|
|
||||
| `tdd-workflow` | testing | Step-by-step TDD8 workflow for any feature |
|
||||
| `code-refactoring` | quality | Code quality analysis and safe refactoring |
|
||||
| `test-maintenance` | testing | Diagnose and fix failing tests |
|
||||
| `requirements-engineering` | process | Prevent interface/mock mismatches upfront |
|
||||
| `keepaTodofile` | process | Maintain TODO.md during work |
|
||||
| `project-management` | process | Track status, determine next steps |
|
||||
| `datamodel-optimization` | quality | Optimize dataclasses and data structures |
|
||||
|
||||
All 17 agents: call `list_kaizen_agents()` for the full list.
|
||||
8
.claude/rules/architecture.md
Normal file
8
.claude/rules/architecture.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Architecture
|
||||
|
||||
<!-- TODO: Describe the key design decisions and component structure.
|
||||
Key modules, data flows, external integrations, state machines, etc. -->
|
||||
|
||||
## Quick Reference
|
||||
|
||||
`~/state-hub/mcp_server/TOOLS.md` — MCP tool reference
|
||||
38
.claude/rules/first-session.md
Normal file
38
.claude/rules/first-session.md
Normal file
@@ -0,0 +1,38 @@
|
||||
## First Session Protocol
|
||||
|
||||
Triggered when `get_domain_summary("capabilities")` shows **no workstreams**.
|
||||
The project is registered but work has not yet been structured.
|
||||
|
||||
**Step 1 — Read, don't write**
|
||||
- `~/the-custodian/canon/projects/capabilities/project_charter_v0.1.md` — purpose, scope
|
||||
- `~/the-custodian/canon/projects/capabilities/roadmap_v0.1.md` — planned phases
|
||||
- Scan repo root: README, directory structure, existing code or docs
|
||||
|
||||
**Step 2 — Survey in-progress work**
|
||||
Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete.
|
||||
|
||||
**Step 3 — Propose workstreams to Bernd**
|
||||
Propose 1–3 workstreams — each a coherent strand, weeks to months, anchored to a
|
||||
roadmap phase. **Wait for approval before creating.**
|
||||
|
||||
**Step 4 — Create workplan file first, then DB record (ADR-001)**
|
||||
```
|
||||
workplans/repo-scoping-WP-NNNN-<slug>.md ← write this first
|
||||
```
|
||||
Then register in the hub:
|
||||
```
|
||||
create_workstream(topic_id="64418556-3206-457a-ba29-6884b5b12cf3", title="...", owner="...", description="...")
|
||||
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
|
||||
```
|
||||
|
||||
**Step 5 — Record the setup**
|
||||
```
|
||||
add_progress_event(
|
||||
summary="First session: structured capabilities into N workstreams, M tasks",
|
||||
event_type="milestone",
|
||||
topic_id="64418556-3206-457a-ba29-6884b5b12cf3",
|
||||
detail={"workstreams": [...], "tasks_created": M}
|
||||
)
|
||||
```
|
||||
|
||||
<!-- Delete or archive this file once past first session -->
|
||||
8
.claude/rules/repo-boundary.md
Normal file
8
.claude/rules/repo-boundary.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Repo boundary
|
||||
|
||||
This repo owns **repo-scoping** only. It does not own:
|
||||
|
||||
<!-- TODO: List what belongs in adjacent repos, e.g.:
|
||||
- SSH key management → railiance-infra/
|
||||
- State hub code → state-hub/
|
||||
-->
|
||||
5
.claude/rules/repo-identity.md
Normal file
5
.claude/rules/repo-identity.md
Normal file
@@ -0,0 +1,5 @@
|
||||
**Purpose:** repo-scoping - (fill in purpose)
|
||||
|
||||
**Domain:** capabilities
|
||||
**Repo slug:** repo-scoping
|
||||
**Topic ID:** 64418556-3206-457a-ba29-6884b5b12cf3
|
||||
84
.claude/rules/session-protocol.md
Normal file
84
.claude/rules/session-protocol.md
Normal file
@@ -0,0 +1,84 @@
|
||||
## Session Protocol
|
||||
|
||||
State Hub: http://127.0.0.1:8000
|
||||
|
||||
**Step 1 — Orient**
|
||||
|
||||
Read the offline-safe brief first — it works without a live hub connection:
|
||||
```bash
|
||||
cat .custodian-brief.md
|
||||
```
|
||||
Then call the MCP tool for richer cross-domain context when MCP tools are exposed:
|
||||
```
|
||||
get_domain_summary("capabilities")
|
||||
```
|
||||
If MCP tools are unavailable in the current agent session, use the REST API:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
|
||||
```
|
||||
If the hub is offline: `cd ~/state-hub && make api`
|
||||
|
||||
**Step 2 — Check inbox**
|
||||
With MCP tools:
|
||||
```
|
||||
get_messages(to_agent="repo-scoping", unread_only=True)
|
||||
```
|
||||
Mark read with `mark_message_read(message_id)`. Reply or act on coordination
|
||||
requests before proceeding.
|
||||
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=repo-scoping&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
**Step 3 — Scan workplans**
|
||||
```bash
|
||||
ls workplans/
|
||||
```
|
||||
For each file with `status: ready`, `active`, or `blocked`, note pending
|
||||
`todo`/`in_progress` tasks.
|
||||
|
||||
**Step 4 — Present brief**
|
||||
|
||||
1. **Active workstreams** for `capabilities` — title, task counts, blocking decisions
|
||||
2. **Pending tasks** from `workplans/` + any `[repo:repo-scoping]` hub tasks
|
||||
3. **Goal guidance** — if `goal_guidance` in summary:
|
||||
- `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"*
|
||||
- `alignment_warnings`: flag if active work is not aligned with current goal
|
||||
4. **Suggested next action** — highest-priority open item
|
||||
5. **SBOM status** — flag if `last_sbom_at` is unset for this repo
|
||||
|
||||
If no workstreams: follow First Session Protocol (`first-session.md`).
|
||||
|
||||
**During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()`
|
||||
|
||||
> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`)
|
||||
> are First Session Protocol only. Work structure belongs in repo files (ADR-001).
|
||||
|
||||
**Session close:**
|
||||
With MCP tools:
|
||||
```
|
||||
add_progress_event(summary="...", topic_id="64418556-3206-457a-ba29-6884b5b12cf3", workstream_id="<uuid>")
|
||||
```
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"topic_id":"64418556-3206-457a-ba29-6884b5b12cf3","workstream_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
|
||||
```
|
||||
If workplan files were modified, ensure the local copy is up to date first:
|
||||
```bash
|
||||
git -C <repo_path> pull --ff-only
|
||||
cd ~/state-hub && make fix-consistency REPO=repo-scoping
|
||||
```
|
||||
For repos where implementation runs on a remote machine (e.g. CoulombCore),
|
||||
use the combined target which pulls before fixing:
|
||||
```bash
|
||||
cd ~/state-hub && make fix-consistency-remote REPO=repo-scoping
|
||||
```
|
||||
**C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback
|
||||
will sync the file to match DB. **C-16** (repo behind remote) blocks all writes
|
||||
until you pull — intentional to prevent clobbering remote progress.
|
||||
19
.claude/rules/stack-and-commands.md
Normal file
19
.claude/rules/stack-and-commands.md
Normal file
@@ -0,0 +1,19 @@
|
||||
## Stack
|
||||
|
||||
<!-- TODO: Fill in language, frameworks, and key dependencies -->
|
||||
- **Language:**
|
||||
- **Key deps:**
|
||||
|
||||
## Dev Commands
|
||||
|
||||
```bash
|
||||
# TODO: Fill in the standard commands for this repo
|
||||
|
||||
# Install dependencies
|
||||
|
||||
# Run tests
|
||||
|
||||
# Lint / type check
|
||||
|
||||
# Build / package (if applicable)
|
||||
```
|
||||
28
.claude/rules/workplan-convention.md
Normal file
28
.claude/rules/workplan-convention.md
Normal file
@@ -0,0 +1,28 @@
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
File location: `workplans/repo-scoping-WP-NNNN-<slug>.md`
|
||||
ID prefix: `REPO-WP`
|
||||
|
||||
Work items originate as files in this repo **before** being registered in the hub.
|
||||
|
||||
Canonical workplan/workstream frontmatter statuses are:
|
||||
`proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`.
|
||||
Use `proposed` for a newly drafted plan, `ready` after review against current
|
||||
repo state, and `finished` when implementation is complete. `stalled` and
|
||||
`needs_review` are derived health labels, not stored statuses.
|
||||
|
||||
Closed workplans may be moved to `workplans/archived/` with a completion-date
|
||||
prefix: `YYMMDD-repo-scoping-WP-NNNN-<slug>.md`. The frontmatter id remains
|
||||
unchanged; the prefix is only for quick visual reference.
|
||||
|
||||
Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**:
|
||||
`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids
|
||||
`ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed
|
||||
directly. Promote anything requiring analysis, design, approval, dependencies, or
|
||||
multiple planned phases into a normal workplan.
|
||||
|
||||
Ecosystem todos from other agents arrive as `[repo:repo-scoping]` hub tasks —
|
||||
visible at session start. Pick one up by creating the workplan file, then registering
|
||||
the workstream.
|
||||
|
||||
<!-- Ralph Loop rules and HEUREKA sequence: ~/.claude/CLAUDE.md — do not duplicate here -->
|
||||
@@ -1,8 +1,8 @@
|
||||
<!-- custodian-brief: generated by fix-consistency — do not edit manually -->
|
||||
# Custodian Brief — repo-scoping
|
||||
|
||||
**Domain:** custodian
|
||||
**Last synced:** 2026-05-15 08:48 UTC
|
||||
**Domain:** capabilities
|
||||
**Last synced:** 2026-05-15 23:41 UTC
|
||||
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
|
||||
|
||||
## Active Workstreams
|
||||
@@ -13,6 +13,6 @@
|
||||
## MCP Orientation (when available)
|
||||
|
||||
If the state-hub MCP server is reachable, call:
|
||||
`get_domain_summary("custodian")`
|
||||
`get_domain_summary("capabilities")`
|
||||
This provides richer cross-domain context.
|
||||
If the MCP call fails, use this file as your orientation source.
|
||||
|
||||
166
AGENTS.md
166
AGENTS.md
@@ -2,46 +2,44 @@
|
||||
|
||||
## Repo Identity
|
||||
|
||||
**Purpose:** Repository Ability Registry — turns Git repositories into reviewable,
|
||||
source-linked maps of `Ability → Capability → Feature → Evidence`. Deterministic
|
||||
scanners establish observed facts; LLM-assisted extractors propose interpreted
|
||||
claims; humans or trusted agents approve registry truth.
|
||||
**Purpose:** repo-scoping - (fill in purpose)
|
||||
|
||||
**Domain:** capabilities
|
||||
**Repo slug:** repo-scoping
|
||||
**Topic ID:** `64418556-3206-457a-ba29-6884b5b12cf3`
|
||||
**Workplan prefix:** `RREG-WP-`
|
||||
**Workplan prefix:** `REPO-WP-`
|
||||
|
||||
---
|
||||
|
||||
## State Hub Integration
|
||||
|
||||
The Custodian State Hub tracks work across all domains. It runs at
|
||||
`http://127.0.0.1:8000` (local) or `http://127.0.0.1:18000` when accessed from
|
||||
a remote machine via tunnel.
|
||||
The Custodian State Hub tracks work across all domains. Interact via HTTP REST —
|
||||
there is no MCP server for Codex agents.
|
||||
|
||||
Interact via HTTP — there is no MCP integration for Codex agents.
|
||||
| Context | URL |
|
||||
|---------|-----|
|
||||
| Local workstation | `http://127.0.0.1:8000` |
|
||||
| Remote via tunnel | `http://127.0.0.1:18000` |
|
||||
|
||||
### Orient at session start
|
||||
|
||||
```bash
|
||||
# Domain workstreams
|
||||
# Offline brief — works without hub connection
|
||||
cat .custodian-brief.md
|
||||
|
||||
# Active workstreams for this domain
|
||||
curl -s "http://127.0.0.1:8000/workstreams/?topic_id=64418556-3206-457a-ba29-6884b5b12cf3&status=active" \
|
||||
| python3 -m json.tool
|
||||
|
||||
# Open tasks for this repo (once workstreams are registered)
|
||||
curl -s "http://127.0.0.1:8000/tasks/?status=todo" | python3 -m json.tool
|
||||
|
||||
# Check inbox
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=repo-scoping&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
Also read `workplans/` directly — the files are the source of truth:
|
||||
|
||||
Mark a message read:
|
||||
```bash
|
||||
ls workplans/
|
||||
grep -h "^status:" workplans/RREG-WP-*.md
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
### Log progress (required at session close)
|
||||
@@ -50,27 +48,31 @@ grep -h "^status:" workplans/RREG-WP-*.md
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"summary": "describe what was done",
|
||||
"summary": "what was done",
|
||||
"event_type": "note",
|
||||
"author": "codex"
|
||||
"author": "codex",
|
||||
"workstream_id": "<uuid>",
|
||||
"task_id": "<uuid>"
|
||||
}'
|
||||
```
|
||||
|
||||
Include `"workstream_id": "<uuid>"` and `"task_id": "<uuid>"` when known.
|
||||
Omit `workstream_id` / `task_id` when not applicable.
|
||||
|
||||
### Mark a message read
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<message_id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
### Update task status (after workstreams are synced)
|
||||
### Update task status
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"status": "in_progress"}'
|
||||
# values: todo | in_progress | done | blocked
|
||||
```
|
||||
|
||||
### Flag a task for human review
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"needs_human": true, "intervention_note": "reason"}'
|
||||
```
|
||||
|
||||
---
|
||||
@@ -78,99 +80,83 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
## Session Protocol
|
||||
|
||||
**Start:**
|
||||
1. `ls workplans/` — note active workplans and their open tasks
|
||||
2. Check inbox via `GET /messages/?to_agent=repo-scoping&unread_only=true`
|
||||
3. Check for human-flagged tasks: `GET /tasks/?needs_human=true`
|
||||
1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe)
|
||||
2. Check inbox: `GET /messages/?to_agent=repo-scoping&unread_only=true`; mark read
|
||||
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
|
||||
4. Check blocked tasks: `GET /tasks/?needs_human=true`
|
||||
|
||||
**During work:**
|
||||
- Update task status in the workplan file as tasks progress
|
||||
- For significant decisions, record them: `POST /decisions/`
|
||||
- Update task statuses in workplan files as tasks progress
|
||||
- Record significant decisions via `POST /decisions/`
|
||||
|
||||
**Close:**
|
||||
1. Update task statuses in workplan files to match progress
|
||||
2. Call `POST /progress/` with a summary of what was done
|
||||
3. If workplan files changed, sync them to the hub DB:
|
||||
|
||||
```bash
|
||||
curl -s -X POST "http://127.0.0.1:8000/repos/repo-scoping/sync" | python3 -m json.tool
|
||||
```
|
||||
|
||||
This runs the ADR-001 consistency check with `--fix` and returns a JSON report.
|
||||
A `"result": "warn"` with only C-17 is normal (unpushed commits); no action needed.
|
||||
A `"result": "fail"` means file/DB drift that could not be auto-fixed — read the issues list.
|
||||
1. Update workplan file task statuses to reflect progress
|
||||
2. Log: `POST /progress/` with a summary of what changed
|
||||
3. Note for the custodian operator: after workplan file changes, run from
|
||||
`~/state-hub`:
|
||||
```bash
|
||||
make fix-consistency REPO=repo-scoping
|
||||
```
|
||||
This syncs task status from files into the hub DB.
|
||||
|
||||
---
|
||||
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
Work items originate as files in this repo, not in the hub. The hub is a
|
||||
read/cache/index layer.
|
||||
Work items originate as files in this repo — not in the hub. The hub is a
|
||||
read/cache/index layer that rebuilds from files.
|
||||
|
||||
**File location:** `workplans/RREG-WP-NNNN-<slug>.md`
|
||||
**File location:** `workplans/REPO-WP-NNNN-<slug>.md`
|
||||
|
||||
**Archived location:** finished workplans may move to
|
||||
`workplans/archived/YYMMDD-REPO-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
|
||||
the completion/archive date; the frontmatter `id` does not change.
|
||||
|
||||
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
|
||||
`workplans/ADHOC-YYYY-MM-DD.md` with task ids `ADHOC-YYYY-MM-DD-T01`, etc. Use
|
||||
this only for low-risk work completed directly; create a normal workplan for
|
||||
anything needing analysis, design, approval, dependencies, or multiple phases.
|
||||
|
||||
**Frontmatter:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: RREG-WP-NNNN
|
||||
id: REPO-WP-NNNN
|
||||
type: workplan
|
||||
title: "..."
|
||||
domain: capabilities
|
||||
repo: repo-scoping
|
||||
status: active | done
|
||||
status: proposed | ready | active | blocked | backlog | finished | archived
|
||||
owner: codex
|
||||
topic_slug: foerster-capabilities
|
||||
topic_slug: ...
|
||||
created: "YYYY-MM-DD"
|
||||
updated: "YYYY-MM-DD"
|
||||
state_hub_workstream_id: "<uuid>" # populated by fix-consistency
|
||||
state_hub_workstream_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
---
|
||||
```
|
||||
|
||||
**Task blocks** (one per `##` section):
|
||||
Use `proposed` for a new draft, `ready` after review against current repo
|
||||
state, and `finished` after implementation. `stalled` and `needs_review` are
|
||||
derived health labels, not frontmatter statuses.
|
||||
|
||||
```markdown
|
||||
**Task block format** (one per `##` section):
|
||||
|
||||
```
|
||||
## Task Title
|
||||
|
||||
\`\`\`task
|
||||
id: RREG-WP-NNNN-T01
|
||||
` ` `task
|
||||
id: REPO-WP-NNNN-T01
|
||||
status: todo | in_progress | done | blocked
|
||||
priority: high | medium | low
|
||||
\`\`\`
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
` ` `
|
||||
|
||||
Task description.
|
||||
Task description text.
|
||||
```
|
||||
|
||||
**Status values:** `todo` → `in_progress` → `done` (or `blocked`)
|
||||
Status progression: `todo` → `in_progress` → `done` (or `blocked`)
|
||||
|
||||
---
|
||||
|
||||
## Stack and Commands
|
||||
|
||||
**Runtime:** Python 3.x, FastAPI, SQLite (dev) / PostgreSQL (prod)
|
||||
**Package manager:** pip / uv
|
||||
|
||||
```bash
|
||||
# Install
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# Run dev server
|
||||
uvicorn src.repo_registry.app:app --reload
|
||||
|
||||
# Run tests
|
||||
pytest tests/
|
||||
pytest tests/ -k "e2e"
|
||||
|
||||
# Check API health
|
||||
curl http://127.0.0.1:8001/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Repo Boundary
|
||||
|
||||
This repo owns: repository ingestion, deterministic scanning, LLM-assisted candidate
|
||||
extraction, review/approval workflow, registry query and search.
|
||||
|
||||
It does NOT own: the Custodian State Hub, other domain repos, deployment infrastructure.
|
||||
|
||||
Coordination with other domains goes through the State Hub message inbox.
|
||||
To create a new workplan:
|
||||
1. Write the file following the format above
|
||||
2. Notify the custodian operator to run `make fix-consistency REPO=repo-scoping`
|
||||
(or send a message to the hub agent via `POST /messages/`)
|
||||
|
||||
99
CLAUDE.md
99
CLAUDE.md
@@ -1,90 +1,11 @@
|
||||
# CLAUDE.md
|
||||
# repo-scoping — Claude Code Instructions
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# Install
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# Run dev server (port 8001)
|
||||
uvicorn repo_registry.web_api.app:app --reload --port 8001
|
||||
|
||||
# Run tests
|
||||
pytest
|
||||
pytest -k "test_scanner" # filter by keyword
|
||||
pytest tests/test_web_api.py # single file
|
||||
|
||||
# Health check
|
||||
curl http://127.0.0.1:8001/health
|
||||
```
|
||||
|
||||
Note: `AGENTS.md` shows `src.repo_registry.app:app` but the correct module path is `repo_registry.web_api.app:app` (as installed via `src/`).
|
||||
|
||||
## Architecture
|
||||
|
||||
The service maps Git repositories to reviewable scope maps using a fixed hierarchy:
|
||||
|
||||
```
|
||||
Scope → Ability → Capability → Feature → Evidence → ObservedFact
|
||||
```
|
||||
|
||||
**Data flow for an analysis run:**
|
||||
|
||||
1. `POST /repos/{id}/analysis-runs` triggers the pipeline in `RegistryService.run_analysis()`
|
||||
2. `GitIngestionService` clones or resolves the repo path
|
||||
3. `RepositoryMetadataExtractor` reads pyproject.toml / package.json / README
|
||||
4. `DeterministicScanner` produces `ObservedFact` objects (files, languages, manifests, APIs, etc.)
|
||||
5. `ContentExtractor` chunks files into searchable segments
|
||||
6. `CandidateGraphGenerator` builds a draft ability→capability→feature→evidence tree from facts
|
||||
7. Optionally, `LLMCandidateExtractor` proposes additional candidates (requires `REPO_REGISTRY_LLM_ENABLED=true`)
|
||||
8. Candidates are stored; humans or agents review them via `POST .../candidate-graph/approve`
|
||||
9. Approved characteristics feed `ScopeGenerator` to produce `SCOPE.md`
|
||||
|
||||
**Key source locations:**
|
||||
|
||||
| Component | Path |
|
||||
|-----------|------|
|
||||
| FastAPI routes + DI | `src/repo_registry/web_api/app.py` |
|
||||
| Orchestration | `src/repo_registry/core/service.py` |
|
||||
| Frozen dataclasses | `src/repo_registry/core/models.py` |
|
||||
| Deterministic scanner | `src/repo_registry/repo_scanning/scanner.py` |
|
||||
| Candidate graph builder | `src/repo_registry/candidate_graph/generator.py` |
|
||||
| SQLite store | `src/repo_registry/storage/sqlite.py` |
|
||||
| Schema migration | `migrations/0001_initial.sql` |
|
||||
|
||||
**Storage:** SQLite at `var/repo-registry.sqlite3` (auto-created). Schema migrations run at startup. Dynamic columns are added to support evidence relationships, classification, and expectation gaps.
|
||||
|
||||
**LLM extraction** is optional and disabled by default. Enable with `REPO_REGISTRY_LLM_ENABLED=true` plus `REPO_REGISTRY_LLM_PROVIDER` and `REPO_REGISTRY_LLM_MODEL`. The `llm-connect` sibling package provides the adapter abstraction.
|
||||
|
||||
**Semantic search** uses `HashingEmbeddingProvider` by default — deterministic, no external service required.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `REPO_REGISTRY_DATABASE_PATH` | `var/repo-registry.sqlite3` | SQLite file |
|
||||
| `REPO_REGISTRY_CHECKOUT_ROOT` | `var/checkouts` | Git clone cache |
|
||||
| `REPO_REGISTRY_LLM_ENABLED` | `true` | Enable LLM extraction |
|
||||
| `REPO_REGISTRY_LLM_PROVIDER` | — | e.g. `gemini`, `anthropic` |
|
||||
| `REPO_REGISTRY_LLM_MODEL` | — | e.g. `gemini-2.5-flash` |
|
||||
| `REPO_REGISTRY_STATE_HUB_BASE_URL` | `http://127.0.0.1:8000` | State Hub for coordination |
|
||||
|
||||
## State Hub & Workplans
|
||||
|
||||
Active work is tracked in `workplans/RREG-WP-*.md` — these files are the source of truth (ADR-001). The Custodian State Hub caches this state; workplan files take precedence.
|
||||
|
||||
Session protocol (see `AGENTS.md` for full curl examples):
|
||||
- **Start:** check `workplans/` status headers and State Hub inbox
|
||||
- **Close:** update task statuses in workplan files, then `POST /progress/` and sync via `POST /repos/repo-scoping/sync`
|
||||
|
||||
Workplan sync warns on C-17 (unpushed commits) — that's normal. A `"result": "fail"` needs investigation.
|
||||
|
||||
## Docs
|
||||
|
||||
Design decisions and terminology live in `docs/`:
|
||||
- `docs/terminology.md` — characteristic model definitions
|
||||
- `docs/scope-md-spec.md` — SCOPE.md format
|
||||
- `docs/characteristic-evidence-model.md` — evidence target kinds
|
||||
- `docs/classification-strategy.md` — how characteristics are classified
|
||||
@SCOPE.md
|
||||
@.claude/rules/repo-identity.md
|
||||
@.claude/rules/session-protocol.md
|
||||
@.claude/rules/first-session.md
|
||||
@.claude/rules/workplan-convention.md
|
||||
@.claude/rules/stack-and-commands.md
|
||||
@.claude/rules/architecture.md
|
||||
@.claude/rules/repo-boundary.md
|
||||
@.claude/rules/agents.md
|
||||
|
||||
12
INTENT.md
12
INTENT.md
@@ -136,11 +136,11 @@ characteristics support those claims.
|
||||
## Getting Oriented
|
||||
|
||||
- Start with: `README.md`, `AGENTS.md`, and this `SCOPE.md`.
|
||||
- Key files / directories: `src/repo_registry/web_api/app.py`,
|
||||
`src/repo_registry/core/service.py`, `src/repo_registry/scope/`,
|
||||
`src/repo_registry/candidate_graph/`, `src/repo_registry/repo_scanning/`,
|
||||
- Key files / directories: `src/repo_scoping/web_api/app.py`,
|
||||
`src/repo_scoping/core/service.py`, `src/repo_scoping/scope/`,
|
||||
`src/repo_scoping/candidate_graph/`, `src/repo_scoping/repo_scanning/`,
|
||||
`docs/scope-md-spec.md`, and `workplans/`.
|
||||
- Entry points: `uvicorn repo_registry.web_api.app:app --reload`, the `/ui`
|
||||
- Entry points: `uvicorn repo_scoping.web_api.app:app --reload`, the `/ui`
|
||||
routes, and the `/repos/{repo_slug}/scope*` API endpoints.
|
||||
|
||||
---
|
||||
@@ -171,5 +171,5 @@ keywords: [scope, scope-md, update, diff, staleness]
|
||||
|
||||
- The product and managed repository identity are Repository Scoping /
|
||||
`repo-scoping`.
|
||||
- The Python package name `repo_registry`, `REPO_REGISTRY_` environment prefix,
|
||||
and default SQLite filename remain compatibility details.
|
||||
- The Python package name `repo_scoping`, `REPO_SCOPING_` environment prefix,
|
||||
and default SQLite filename are the current runtime names.
|
||||
|
||||
2
Makefile
2
Makefile
@@ -16,7 +16,7 @@ start: ## Start the API server in the background on port 8002
|
||||
@if [ -f $(PIDFILE) ] && kill -0 $$(cat $(PIDFILE)) 2>/dev/null; then \
|
||||
echo "Already running (PID $$(cat $(PIDFILE)))"; \
|
||||
else \
|
||||
setsid sh -c 'echo $$$$ > $(PIDFILE); exec $(UVICORN) repo_registry.web_api.app:app --host 127.0.0.1 --port $(PORT)' \
|
||||
setsid sh -c 'echo $$$$ > $(PIDFILE); exec $(UVICORN) repo_scoping.web_api.app:app --host 127.0.0.1 --port $(PORT)' \
|
||||
>> $(LOGFILE) 2>&1 & \
|
||||
PID=""; \
|
||||
for i in $$(seq 1 50); do \
|
||||
|
||||
19
README.md
19
README.md
@@ -27,10 +27,12 @@ pytest
|
||||
Run the API:
|
||||
|
||||
```bash
|
||||
uvicorn repo_registry.web_api.app:app --reload
|
||||
uvicorn repo_scoping.web_api.app:app --reload
|
||||
```
|
||||
|
||||
The API creates a local SQLite database at `var/repo-registry.sqlite3` by default. The database path, `REPO_REGISTRY_` environment prefix, and `repo_registry` Python package name remain compatibility details after the product rename to Repository Scoping.
|
||||
The API creates a local SQLite database at `var/repo-scoping.sqlite3` by
|
||||
default. Runtime configuration uses the `REPO_SCOPING_` environment prefix, and
|
||||
the Python package is `repo_scoping`.
|
||||
|
||||
## First API Loop
|
||||
|
||||
@@ -40,7 +42,10 @@ curl -X POST http://127.0.0.1:8000/repos \
|
||||
-d '{"url":"https://example.com/mail-router.git"}'
|
||||
```
|
||||
|
||||
The registry imports name and description from `pyproject.toml`, `package.json`, or README where possible. Then add abilities, capabilities, features, and evidence under that repository and inspect:
|
||||
The registry uses the submitted repository path or URL as the default name, then
|
||||
imports description and fallback metadata from `pyproject.toml`, `package.json`,
|
||||
or README where possible. Then add abilities, capabilities, features, and
|
||||
evidence under that repository and inspect:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8000/repos/1/ability-map
|
||||
@@ -135,17 +140,17 @@ draft.
|
||||
|
||||
The FastAPI settings object also accepts `llm_provider` and `llm_model`. By
|
||||
default `llm_provider` is unset, so analysis is fully offline and deterministic.
|
||||
Environment variables use the `REPO_REGISTRY_` prefix:
|
||||
Environment variables use the `REPO_SCOPING_` prefix:
|
||||
|
||||
```bash
|
||||
REPO_REGISTRY_LLM_PROVIDER=gemini
|
||||
REPO_REGISTRY_LLM_MODEL=gemini-2.5-flash
|
||||
REPO_SCOPING_LLM_PROVIDER=gemini
|
||||
REPO_SCOPING_LLM_MODEL=gemini-2.5-flash
|
||||
```
|
||||
|
||||
LLM assistance can also be disabled even when a provider is configured:
|
||||
|
||||
```bash
|
||||
REPO_REGISTRY_LLM_ENABLED=false
|
||||
REPO_SCOPING_LLM_ENABLED=false
|
||||
```
|
||||
|
||||
Individual analysis requests may opt out with `{"use_llm_assistance": false}`.
|
||||
|
||||
12
SCOPE.md
12
SCOPE.md
@@ -136,11 +136,11 @@ characteristics support those claims.
|
||||
## Getting Oriented
|
||||
|
||||
- Start with: `README.md`, `AGENTS.md`, and this `SCOPE.md`.
|
||||
- Key files / directories: `src/repo_registry/web_api/app.py`,
|
||||
`src/repo_registry/core/service.py`, `src/repo_registry/scope/`,
|
||||
`src/repo_registry/candidate_graph/`, `src/repo_registry/repo_scanning/`,
|
||||
- Key files / directories: `src/repo_scoping/web_api/app.py`,
|
||||
`src/repo_scoping/core/service.py`, `src/repo_scoping/scope/`,
|
||||
`src/repo_scoping/candidate_graph/`, `src/repo_scoping/repo_scanning/`,
|
||||
`docs/scope-md-spec.md`, and `workplans/`.
|
||||
- Entry points: `uvicorn repo_registry.web_api.app:app --reload`, the `/ui`
|
||||
- Entry points: `uvicorn repo_scoping.web_api.app:app --reload`, the `/ui`
|
||||
routes, and the `/repos/{repo_slug}/scope*` API endpoints.
|
||||
|
||||
---
|
||||
@@ -171,5 +171,5 @@ keywords: [scope, scope-md, update, diff, staleness]
|
||||
|
||||
- The product and managed repository identity are Repository Scoping /
|
||||
`repo-scoping`.
|
||||
- The Python package name `repo_registry`, `REPO_REGISTRY_` environment prefix,
|
||||
and default SQLite filename remain compatibility details.
|
||||
- The Python package name `repo_scoping`, `REPO_SCOPING_` environment prefix,
|
||||
and default SQLite filename are the current runtime names.
|
||||
|
||||
@@ -49,9 +49,10 @@ LLMs are most useful for naming and explaining intent:
|
||||
- summarizing README and code context into clearer ability descriptions
|
||||
- suggesting merges or relinks when deterministic names are too generic
|
||||
|
||||
LLM output remains candidate material. It should cite source paths and be reviewed
|
||||
or explicitly auto-approved by a trusted mode before becoming approved registry
|
||||
truth.
|
||||
LLM output remains candidate material. It should cite source paths and be
|
||||
reviewed by a human or configured agentic reviewer before becoming approved
|
||||
registry truth. Deterministic checks can block or flag weak candidates; they do
|
||||
not approve them.
|
||||
|
||||
## Trial Repo Observations
|
||||
|
||||
|
||||
115
docs/acceptance-policy.md
Normal file
115
docs/acceptance-policy.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# Characteristic Acceptance Policy Boundary
|
||||
|
||||
Policy version: `acceptance-policy/v1`
|
||||
|
||||
repo-scoping separates fact generation, quality gating, and acceptance
|
||||
judgement. This boundary exists so deterministic automation can stay useful and
|
||||
fast without silently promoting weak or non-native claims into registry truth.
|
||||
|
||||
## Boundary
|
||||
|
||||
Deterministic scanners and extractors may create observed facts, source refs,
|
||||
content chunks, candidate abilities, candidate capabilities, candidate features,
|
||||
and candidate evidence. They may also run transparent quality gates.
|
||||
|
||||
Deterministic quality gates may reject, invalidate, downgrade, merge, flag, or
|
||||
require review when a criterion is formally expressible. They must record the
|
||||
criterion identifier, outcome, rationale, and affected candidate element.
|
||||
|
||||
Deterministic quality gates must not approve candidate characteristics. No
|
||||
deterministic rule, scanner result, fixture match, vocabulary match, confidence
|
||||
threshold, source-role score, or duplicate check may mark a candidate as
|
||||
approved registry truth.
|
||||
|
||||
Human reviewers may approve candidate characteristics.
|
||||
|
||||
Trusted agentic reviewers may approve candidate characteristics only after
|
||||
inspecting the evidence, applying the active quality criteria, and recording a
|
||||
rationale tied to those criteria and source refs.
|
||||
|
||||
All automated review outcomes must be inspectable, reversible, and auditable.
|
||||
|
||||
## Allowed Deterministic Outcomes
|
||||
|
||||
- `pass`: no formal criterion blocked the candidate; judgement is still pending.
|
||||
- `requires_review`: a criterion found ambiguity that needs reviewer judgement.
|
||||
- `downgraded`: the candidate remains visible but loses trust or native-utility
|
||||
strength until a reviewer overrides or edits it.
|
||||
- `rejected`: the candidate should not be accepted as stated, but remains
|
||||
auditable with reason codes.
|
||||
- `invalidated`: the candidate is structurally or evidentially unusable.
|
||||
- `merged`: the candidate was folded into a better-supported equivalent.
|
||||
- `flagged`: the candidate remains pending with a warning attached.
|
||||
|
||||
These outcomes may prepare a candidate for review. They do not approve it.
|
||||
|
||||
## Allowed Human Outcomes
|
||||
|
||||
- `approve`: accept the candidate as registry truth.
|
||||
- `approve_with_edits`: accept after reviewer edits, relinks, or merges.
|
||||
- `reject`: reject the candidate as a matter of judgement.
|
||||
- `downgrade`: preserve the candidate but reduce its standing or native claim.
|
||||
- `request_changes`: ask for edited wording, source refs, hierarchy placement,
|
||||
or evidence.
|
||||
- `request_human_review`: defer to another curator or domain owner.
|
||||
|
||||
## Allowed Agentic Outcomes
|
||||
|
||||
- `approve`: accept only with evidence-linked rationale and criteria version.
|
||||
- `approve_with_edits`: accept after proposed edits, relinks, or merges are
|
||||
recorded.
|
||||
- `reject`: reject with cited criteria and evidence.
|
||||
- `downgrade`: keep visible but lower trust or native-utility standing.
|
||||
- `request_human_review`: stop automation when evidence or intent is ambiguous.
|
||||
- `propose_edit`: suggest wording, hierarchy, source-ref, or merge changes
|
||||
without approving.
|
||||
|
||||
Agentic approval requires:
|
||||
|
||||
- reviewer identity or configuration
|
||||
- criteria version
|
||||
- prompt or policy version when applicable
|
||||
- evidence inspected
|
||||
- rationale
|
||||
- exact candidate elements affected
|
||||
|
||||
The repo-scoping service represents this as structured agentic review decisions.
|
||||
Approval decisions are rejected unless they include rationale, criteria IDs, and
|
||||
evidence refs. Non-approval decisions still require rationale and criteria IDs,
|
||||
so downgrade, rejection, relink, proposed edit, and human-review requests remain
|
||||
auditable.
|
||||
|
||||
## Legacy Auto-Approval Terminology
|
||||
|
||||
`trusted_auto_approve_candidate_graph` and UI labels such as "Trusted
|
||||
auto-populate" are legacy terminology. They describe an existing migration-era
|
||||
behavior, not an accepted policy direction. Future implementation must replace
|
||||
that path with agentic review or leave candidates pending human review.
|
||||
|
||||
Until the migration in `RREG-WP-0014` is complete, any artifacts produced by the
|
||||
legacy path must be identifiable in review decisions and self-scoping
|
||||
assessment artifacts. They should be treated as review debt, not as evidence
|
||||
that deterministic approval is allowed.
|
||||
|
||||
The migration inventory and rebuild procedure are documented in
|
||||
`docs/migrations/trusted-auto-approval.md`.
|
||||
|
||||
## Quality Criteria Relationship
|
||||
|
||||
The quality criteria registry in `docs/quality-criteria/` defines the formal
|
||||
checks deterministic gates may apply. Criteria should be versioned, reviewable,
|
||||
and specific enough to explain why a candidate was rejected, downgraded,
|
||||
invalidated, merged, flagged, or sent to review. The active registry can be
|
||||
listed with `repo-scoping list-quality-criteria` or `GET /quality-criteria`.
|
||||
|
||||
Examples of criteria families:
|
||||
|
||||
- source-role quality
|
||||
- native utility versus dependency, tooling, fixture, or mention-only context
|
||||
- hierarchy fit between capability and feature
|
||||
- evidence sufficiency
|
||||
- circularity from generated `SCOPE.md`
|
||||
- fixture and schema-example contamination
|
||||
|
||||
Criteria can block bad output before judgement. They cannot stand in for
|
||||
judgement.
|
||||
@@ -67,12 +67,12 @@ that show the repository provides the utility directly or intentionally exposes
|
||||
it as a facade/adapter. Mentions, dependencies, configuration, and tooling are
|
||||
context until a curator promotes them or stronger owned evidence appears.
|
||||
|
||||
Trusted auto-approval applies the same rule. A candidate capability must have
|
||||
source references and an eligible utility relationship (`owned`, `facade`, or
|
||||
`adapter`) before it can be approved automatically. Dependency, tooling,
|
||||
configuration, and mention-only candidates remain review material. The review
|
||||
decision should explain both sides: why approved candidates were considered safe
|
||||
and why skipped candidates need curator review.
|
||||
Deterministic quality gates apply the same source and utility relationship
|
||||
signals, but they do not approve automatically. Gates may reject, downgrade,
|
||||
invalidate, flag, merge, or require review. Approval requires human judgement or
|
||||
a configured agentic reviewer that records evidence, criteria version, and
|
||||
rationale. Dependency, tooling, configuration, and mention-only candidates remain
|
||||
review material.
|
||||
|
||||
`INTENT.md` may also seed intended capabilities when it contains an explicit
|
||||
capability section. These intent-derived candidates are marked as review
|
||||
|
||||
62
docs/migrations/trusted-auto-approval.md
Normal file
62
docs/migrations/trusted-auto-approval.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# Trusted Auto-Approval Migration
|
||||
|
||||
`trusted_auto_approve_candidate_graph` is historical migration behavior, not an
|
||||
allowed acceptance path. Deterministic analysis may generate facts and
|
||||
candidates, and deterministic quality gates may block or require review, but
|
||||
approval now requires human judgement or configured agentic review.
|
||||
|
||||
## Identify Historical Runs
|
||||
|
||||
Use the inventory surfaces before rebuilding a repository with approved maps:
|
||||
|
||||
```bash
|
||||
repo-scoping list-legacy-auto-approvals --format json
|
||||
```
|
||||
|
||||
The API exposes the same inventory at:
|
||||
|
||||
```text
|
||||
GET /review/migrations/trusted-auto-approvals
|
||||
```
|
||||
|
||||
Each record identifies the repository, analysis run, review decision, current
|
||||
approved ability count, scanner version when available, and the recommended next
|
||||
step. These records are derived from review decisions whose action is
|
||||
`trusted_auto_approve_candidate_graph`.
|
||||
|
||||
## Rebuild Without Losing Audit History
|
||||
|
||||
Historical review decisions are retained. Rebuilding characteristics creates a
|
||||
new analysis run and can clear the currently approved characteristic tree, but it
|
||||
does not delete the old review-decision audit trail.
|
||||
|
||||
1. Run a dry run:
|
||||
|
||||
```bash
|
||||
repo-scoping rebuild-characteristics --repo <repo-id> --dry-run --no-llm
|
||||
```
|
||||
|
||||
2. Inspect candidate output, quality-gate outcomes, and existing review
|
||||
decisions.
|
||||
|
||||
3. Confirm the rebuild only when ready:
|
||||
|
||||
```bash
|
||||
repo-scoping rebuild-characteristics --repo <repo-id> --confirm --agentic-review
|
||||
```
|
||||
|
||||
4. If no agentic reviewer is configured, complete human review through the
|
||||
candidate graph approval/edit/reject flow.
|
||||
|
||||
## Compatibility Notes
|
||||
|
||||
- `AnalysisRunCreate.trusted_auto_approve` remains as a deprecated API input
|
||||
for older callers, but requests are routed to agentic review and do not
|
||||
deterministically approve candidates.
|
||||
- The CLI does not expose deterministic trusted auto-approval. Use
|
||||
`--agentic-review` during rebuild or approve after human review.
|
||||
- The service method `trusted_auto_approve_candidate_graph()` is guarded by
|
||||
`allow_deprecated_migration_mode=True` and should only be used to replay or
|
||||
inspect historical migration behavior in controlled tests or migration tools.
|
||||
- Self-scoping assessment artifacts continue to flag
|
||||
`trusted_auto_approve_candidate_graph` as review debt.
|
||||
@@ -5,18 +5,18 @@ Repository Scoping service.
|
||||
|
||||
## Configuration
|
||||
|
||||
Configuration is read from environment variables with the `REPO_REGISTRY_`
|
||||
prefix. That prefix is retained as an implementation compatibility detail after
|
||||
the product rename from Repository Ability Registry to Repository Scoping.
|
||||
Configuration is read from environment variables with the `REPO_SCOPING_`
|
||||
prefix. The same naming is used by the import package and default local
|
||||
database path so service identity stays aligned with Repository Scoping.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `REPO_REGISTRY_DATABASE_PATH` | `var/repo-registry.sqlite3` | SQLite database file used by the default store. |
|
||||
| `REPO_REGISTRY_CHECKOUT_ROOT` | `var/checkouts` | Local checkout cache used during repository ingestion. |
|
||||
| `REPO_REGISTRY_LLM_PROVIDER` | unset | Optional LLM provider name for candidate extraction. |
|
||||
| `REPO_REGISTRY_LLM_MODEL` | unset | Optional model name passed to the configured LLM provider. |
|
||||
| `REPO_REGISTRY_EMBEDDING_PROVIDER` | unset | Set to `hashing` to enable deterministic local hybrid search scoring. |
|
||||
| `REPO_REGISTRY_LOG_LEVEL` | `INFO` | Log level for the `repo_registry.operations` structured event logger. |
|
||||
| `REPO_SCOPING_DATABASE_PATH` | `var/repo-scoping.sqlite3` | SQLite database file used by the default store. |
|
||||
| `REPO_SCOPING_CHECKOUT_ROOT` | `var/checkouts` | Local checkout cache used during repository ingestion. |
|
||||
| `REPO_SCOPING_LLM_PROVIDER` | unset | Optional LLM provider name for candidate extraction. |
|
||||
| `REPO_SCOPING_LLM_MODEL` | unset | Optional model name passed to the configured LLM provider. |
|
||||
| `REPO_SCOPING_EMBEDDING_PROVIDER` | unset | Set to `hashing` to enable deterministic local hybrid search scoring. |
|
||||
| `REPO_SCOPING_LOG_LEVEL` | `INFO` | Log level for the `repo_scoping.operations` structured event logger. |
|
||||
|
||||
## Health Checks
|
||||
|
||||
@@ -27,7 +27,7 @@ be checked locally:
|
||||
{
|
||||
"status": "ok",
|
||||
"database": {
|
||||
"path": "var/repo-registry.sqlite3",
|
||||
"path": "var/repo-scoping.sqlite3",
|
||||
"reachable": true,
|
||||
"error": null
|
||||
},
|
||||
@@ -44,13 +44,13 @@ ingestion path.
|
||||
|
||||
## Structured Logs
|
||||
|
||||
Operational events are emitted through the `repo_registry.operations` logger as
|
||||
Operational events are emitted through the `repo_scoping.operations` logger as
|
||||
single-line JSON messages. Current events include repository registration,
|
||||
analysis start/completion/failure, LLM extraction usage/failure, and review
|
||||
decisions.
|
||||
|
||||
Configure the Python or ASGI server logging stack to route this logger to the
|
||||
same sink as application logs. `REPO_REGISTRY_LOG_LEVEL` controls the logger
|
||||
same sink as application logs. `REPO_SCOPING_LOG_LEVEL` controls the logger
|
||||
level used by API-created service instances.
|
||||
|
||||
## SQLite Backup And Restore
|
||||
@@ -60,18 +60,18 @@ continue while the backup is created:
|
||||
|
||||
```bash
|
||||
mkdir -p backups
|
||||
sqlite3 var/repo-registry.sqlite3 ".backup 'backups/repo-registry-$(date +%F).sqlite3'"
|
||||
sqlite3 var/repo-scoping.sqlite3 ".backup 'backups/repo-scoping-$(date +%F).sqlite3'"
|
||||
```
|
||||
|
||||
For the most conservative backup window, stop writes first, run the backup, then
|
||||
resume the service. Verify a backup with:
|
||||
|
||||
```bash
|
||||
sqlite3 backups/repo-registry-YYYY-MM-DD.sqlite3 "PRAGMA integrity_check;"
|
||||
sqlite3 backups/repo-scoping-YYYY-MM-DD.sqlite3 "PRAGMA integrity_check;"
|
||||
```
|
||||
|
||||
To restore, stop the service, move the current database aside, copy the backup to
|
||||
`REPO_REGISTRY_DATABASE_PATH`, start the service, and verify `GET /health`.
|
||||
`REPO_SCOPING_DATABASE_PATH`, start the service, and verify `GET /health`.
|
||||
|
||||
## PostgreSQL Migration Notes
|
||||
|
||||
|
||||
22
docs/quality-criteria/README.md
Normal file
22
docs/quality-criteria/README.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Quality Criteria Registry
|
||||
|
||||
`acceptance-quality-criteria.v1.json` is the active reviewable criteria registry
|
||||
for candidate characteristics. It defines criteria that deterministic gates may
|
||||
use to reject, downgrade, invalidate, flag, merge, or require review.
|
||||
|
||||
The registry is intentionally not an approval policy. Approval belongs to human
|
||||
reviewers or trusted agentic reviewers that inspect evidence and record a
|
||||
rationale.
|
||||
|
||||
List the active registry from the CLI:
|
||||
|
||||
```bash
|
||||
repo-scoping list-quality-criteria --format markdown
|
||||
```
|
||||
|
||||
The same registry is available from the API at `GET /quality-criteria`.
|
||||
|
||||
The first deterministic gate evaluator emits audit outcomes against this
|
||||
registry. Candidate graph API responses and self-scoping assessment exports
|
||||
include those outcomes as `quality_gate_outcomes`. Gate outcomes can block or
|
||||
flag review, but they never approve candidates.
|
||||
130
docs/quality-criteria/acceptance-quality-criteria.v1.json
Normal file
130
docs/quality-criteria/acceptance-quality-criteria.v1.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"schema_version": "quality-criteria-registry/v1",
|
||||
"criteria_version": "repo-scoping-quality-criteria/v1",
|
||||
"status": "active",
|
||||
"updated_at": "2026-05-15",
|
||||
"criteria": [
|
||||
{
|
||||
"id": "RREG-QC-001",
|
||||
"title": "Source Role Supports The Claim",
|
||||
"category": "source-role-quality",
|
||||
"severity": "medium",
|
||||
"applies_to": ["ability", "capability", "feature", "evidence"],
|
||||
"description": "Candidate claims need evidence whose source role can support the abstraction. Intent docs, implementation source, API surfaces, and product documentation carry stronger claim support than fixtures, schema examples, tests, agent instructions, generated scope text, dependency manifests, or incidental vocabulary.",
|
||||
"deterministic_action": "requires_review",
|
||||
"deterministic_action_when": "All supporting refs are weak source roles such as fixture, schema-example, generated-scope, dependency, test-vocabulary, or agent-guidance.",
|
||||
"reviewer_guidance": "Check whether any cited source actually expresses product intent or implementation behavior for the candidate, not merely matching words.",
|
||||
"agentic_guidance": "Do not approve when evidence only proves scanner vocabulary, test coverage, examples, or dependency presence.",
|
||||
"examples": [
|
||||
"Provider names in tests can prove scanner coverage but not a native provider-routing capability.",
|
||||
"A FastAPI route in source can support an API feature when it belongs under the right capability."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "RREG-QC-002",
|
||||
"title": "Native Utility Is Repo-Owned",
|
||||
"category": "native-utility",
|
||||
"severity": "high",
|
||||
"applies_to": ["ability", "capability"],
|
||||
"description": "Owned, facade, and adapter claims require explicit product evidence. Dependency, tooling, configuration, fixture, schema-example, and mention-only contexts are not native repository capabilities.",
|
||||
"deterministic_action": "downgraded",
|
||||
"deterministic_action_when": "The candidate is supported primarily by dependency, configuration, tooling, fixture, schema-example, or mention-only evidence.",
|
||||
"reviewer_guidance": "Decide whether the repository exposes this utility as its own behavior or merely uses, tests, configures, or mentions another system.",
|
||||
"agentic_guidance": "Approve only when product intent and implementation evidence show the repo owns the user-facing utility.",
|
||||
"examples": [
|
||||
"Using llm-connect as extraction infrastructure does not make repo-scoping an LLM router.",
|
||||
"A repository that exposes a scope context API owns that context API capability."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "RREG-QC-003",
|
||||
"title": "Feature Fits Parent Capability",
|
||||
"category": "hierarchy-fit",
|
||||
"severity": "high",
|
||||
"applies_to": ["feature"],
|
||||
"description": "Features must support the parent capability they are nested under. API, CLI, UI, storage, and backend features should not be attached to unrelated capabilities just because evidence was discovered nearby.",
|
||||
"deterministic_action": "requires_review",
|
||||
"deterministic_action_when": "Feature type, name, or source refs conflict with the parent capability class or native-utility claim.",
|
||||
"reviewer_guidance": "Move or relink features to the capability they actually support before approval.",
|
||||
"agentic_guidance": "Prefer relinking or proposing hierarchy edits over approving a mismatched parent-child relationship.",
|
||||
"examples": [
|
||||
"HTTP API and CLI surfaces should not be nested below provider-routing when they describe repo-scoping registry operations."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "RREG-QC-004",
|
||||
"title": "Evidence Supports The Abstraction",
|
||||
"category": "evidence-sufficiency",
|
||||
"severity": "high",
|
||||
"applies_to": ["ability", "capability", "feature", "evidence"],
|
||||
"description": "Evidence must support the actual abstraction claimed by the candidate, not just share vocabulary with it. Claims need enough source refs to justify the level of abstraction.",
|
||||
"deterministic_action": "requires_review",
|
||||
"deterministic_action_when": "Source refs are absent, too vague, or only vocabulary matches for the candidate name.",
|
||||
"reviewer_guidance": "Trace each source ref to the claim. Reject or rewrite candidates whose evidence only supports a narrower or different behavior.",
|
||||
"agentic_guidance": "Name the exact evidence inspected and explain how it supports or fails to support the claim.",
|
||||
"examples": [
|
||||
"A provider registry constant can support scanner fixture detection, not necessarily provider routing as product behavior."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "RREG-QC-005",
|
||||
"title": "Generated Scope Is Not Primary Proof",
|
||||
"category": "circularity",
|
||||
"severity": "critical",
|
||||
"applies_to": ["ability", "capability", "feature", "evidence"],
|
||||
"description": "Generated SCOPE.md text cannot be primary evidence for rebuilding the same characteristic model. It may be comparison context, bootstrap context, or a generated output under review.",
|
||||
"deterministic_action": "requires_review",
|
||||
"deterministic_action_when": "A candidate is supported only or primarily by generated or derived SCOPE.md content from the same scoping process.",
|
||||
"reviewer_guidance": "Use source, docs, tests, and product intent instead of accepting circular evidence.",
|
||||
"agentic_guidance": "Treat circular generated-scope evidence as a blocker unless independent evidence supports the same claim.",
|
||||
"examples": [
|
||||
"Do not use repo-scoping's generated SCOPE.md as the main proof for repo-scoping's own ability tree."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "RREG-QC-006",
|
||||
"title": "Fixtures And Schemas Do Not Become Product Claims",
|
||||
"category": "fixture-contamination",
|
||||
"severity": "high",
|
||||
"applies_to": ["ability", "capability", "feature", "evidence"],
|
||||
"description": "Tests, fixtures, schemas, examples, and expectation files can prove scanner behavior or API contract examples, but they should not become native product capability claims unless backed by product or implementation evidence.",
|
||||
"deterministic_action": "downgraded",
|
||||
"deterministic_action_when": "The claim is primarily supported by tests, fixtures, schemas, examples, expectation files, or sample vocabulary.",
|
||||
"reviewer_guidance": "Keep scanner/test coverage facts separate from repository capability truth.",
|
||||
"agentic_guidance": "Reject or downgrade candidates that turn examples and fixtures into owned utility.",
|
||||
"examples": [
|
||||
"Schema examples mentioning model providers should not create native model-provider capabilities."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "RREG-QC-007",
|
||||
"title": "Template Boilerplate Is Not Repository Purpose",
|
||||
"category": "template-contamination",
|
||||
"severity": "high",
|
||||
"applies_to": ["ability", "capability"],
|
||||
"description": "Repository templates, seed README text, and bootstrap boilerplate should not become the repository's native ability or capability when more specific source evidence exists.",
|
||||
"deterministic_action": "downgraded",
|
||||
"deterministic_action_when": "Candidate names or descriptions are dominated by template boilerplate such as repo-seed instead of repo-specific evidence.",
|
||||
"reviewer_guidance": "Prefer SCOPE, INTENT, implementation, or product docs that describe this repository, not the template it was created from.",
|
||||
"agentic_guidance": "Detect template text and replace it with a repo-specific abstraction before proposing approval.",
|
||||
"examples": [
|
||||
"A README that says 'A git repository template to bootstrap coulomb projects' should not become the ability for ops-warden."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "RREG-QC-008",
|
||||
"title": "Scope-Derived Drafts Stay Separate From Intent",
|
||||
"category": "scope-intent-separation",
|
||||
"severity": "medium",
|
||||
"applies_to": ["scope", "intent", "ability", "capability"],
|
||||
"description": "Existing SCOPE.md content can bootstrap current-state candidates and draft intent, but it must remain clearly labeled as scope-derived until reviewed.",
|
||||
"deterministic_action": "requires_review",
|
||||
"deterministic_action_when": "A candidate or draft is generated from SCOPE.md rather than authored INTENT.md or implementation evidence.",
|
||||
"reviewer_guidance": "Check whether the claim describes current behavior, desired future utility, or both. Do not write INTENT.md without explicit review.",
|
||||
"agentic_guidance": "Use SCOPE.md to propose current-state candidates and ambitious intent drafts, but keep provenance and review status explicit.",
|
||||
"examples": [
|
||||
"A Railiance SCOPE.md capability block can create a candidate capability, not approved registry truth."
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
105
docs/schemas/quality-criteria-registry.schema.json
Normal file
105
docs/schemas/quality-criteria-registry.schema.json
Normal file
@@ -0,0 +1,105 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://example.local/repo-scoping/quality-criteria-registry.schema.json",
|
||||
"title": "Repository Scoping Quality Criteria Registry",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"schema_version",
|
||||
"criteria_version",
|
||||
"status",
|
||||
"updated_at",
|
||||
"criteria"
|
||||
],
|
||||
"properties": {
|
||||
"schema_version": {
|
||||
"const": "quality-criteria-registry/v1"
|
||||
},
|
||||
"criteria_version": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"status": {
|
||||
"enum": ["draft", "active", "deprecated"]
|
||||
},
|
||||
"updated_at": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"criteria": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"id",
|
||||
"title",
|
||||
"category",
|
||||
"severity",
|
||||
"applies_to",
|
||||
"description",
|
||||
"deterministic_action",
|
||||
"deterministic_action_when",
|
||||
"reviewer_guidance"
|
||||
],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"pattern": "^RREG-QC-[0-9]{3}$"
|
||||
},
|
||||
"title": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"category": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"severity": {
|
||||
"enum": ["low", "medium", "high", "critical"]
|
||||
},
|
||||
"applies_to": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"enum": ["ability", "capability", "feature", "evidence"]
|
||||
}
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"deterministic_action": {
|
||||
"enum": [
|
||||
"pass",
|
||||
"requires_review",
|
||||
"downgraded",
|
||||
"rejected",
|
||||
"invalidated",
|
||||
"merged",
|
||||
"flagged"
|
||||
]
|
||||
},
|
||||
"deterministic_action_when": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"reviewer_guidance": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"agentic_guidance": {
|
||||
"type": "string"
|
||||
},
|
||||
"examples": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
}
|
||||
634
docs/schemas/self-scoping-assessment.schema.json
Normal file
634
docs/schemas/self-scoping-assessment.schema.json
Normal file
@@ -0,0 +1,634 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://repo-scoping.local/schemas/self-scoping-assessment.schema.json",
|
||||
"title": "Self-Scoping Assessment Artifact",
|
||||
"description": "Immutable artifact used to compare repo-scoping self-analysis results across engine releases.",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema_version",
|
||||
"artifact_id",
|
||||
"artifact_type",
|
||||
"created_at",
|
||||
"target_repository",
|
||||
"engine_identity",
|
||||
"execution",
|
||||
"assessment",
|
||||
"fact_summary",
|
||||
"content_chunk_summary",
|
||||
"generated_tree",
|
||||
"approved_map",
|
||||
"review_decisions",
|
||||
"quality_gate_outcomes",
|
||||
"known_regression_patterns"
|
||||
],
|
||||
"properties": {
|
||||
"schema_version": {
|
||||
"const": "self-scoping-assessment/v1"
|
||||
},
|
||||
"artifact_id": {
|
||||
"type": "string",
|
||||
"description": "Stable artifact identifier."
|
||||
},
|
||||
"artifact_type": {
|
||||
"enum": ["assessment_run"]
|
||||
},
|
||||
"created_at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"target_repository": {
|
||||
"$ref": "#/$defs/targetRepository"
|
||||
},
|
||||
"engine_identity": {
|
||||
"$ref": "#/$defs/engineIdentity"
|
||||
},
|
||||
"execution": {
|
||||
"$ref": "#/$defs/execution"
|
||||
},
|
||||
"assessment": {
|
||||
"$ref": "#/$defs/assessment"
|
||||
},
|
||||
"fact_summary": {
|
||||
"$ref": "#/$defs/factSummary"
|
||||
},
|
||||
"content_chunk_summary": {
|
||||
"$ref": "#/$defs/contentChunkSummary"
|
||||
},
|
||||
"generated_tree": {
|
||||
"$ref": "#/$defs/generatedTree"
|
||||
},
|
||||
"approved_map": {
|
||||
"$ref": "#/$defs/approvedMap"
|
||||
},
|
||||
"review_decisions": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/reviewDecision"
|
||||
}
|
||||
},
|
||||
"quality_gate_outcomes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/qualityGateOutcome"
|
||||
}
|
||||
},
|
||||
"known_regression_patterns": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/regressionPattern"
|
||||
}
|
||||
},
|
||||
"notes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"targetRepository": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"repo_slug",
|
||||
"repository_id",
|
||||
"source",
|
||||
"target_commit",
|
||||
"target_branch",
|
||||
"dirty_state",
|
||||
"file_count"
|
||||
],
|
||||
"properties": {
|
||||
"repo_slug": {
|
||||
"type": "string"
|
||||
},
|
||||
"repository_id": {
|
||||
"type": ["integer", "null"]
|
||||
},
|
||||
"source": {
|
||||
"type": "string"
|
||||
},
|
||||
"target_commit": {
|
||||
"type": "string"
|
||||
},
|
||||
"target_branch": {
|
||||
"type": "string"
|
||||
},
|
||||
"dirty_state": {
|
||||
"enum": ["clean", "dirty", "unknown"]
|
||||
},
|
||||
"file_count": {
|
||||
"type": ["integer", "null"],
|
||||
"minimum": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"engineIdentity": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"repo_scoping_version",
|
||||
"engine_commit",
|
||||
"engine_release",
|
||||
"engine_dirty_state",
|
||||
"scanner_version",
|
||||
"candidate_generator_version",
|
||||
"quality_criteria_version",
|
||||
"prompt_version",
|
||||
"release_binding_status"
|
||||
],
|
||||
"properties": {
|
||||
"repo_scoping_version": {
|
||||
"type": "string"
|
||||
},
|
||||
"engine_commit": {
|
||||
"type": ["string", "null"]
|
||||
},
|
||||
"engine_release": {
|
||||
"type": ["string", "null"]
|
||||
},
|
||||
"engine_dirty_state": {
|
||||
"enum": ["clean", "dirty", "unknown"]
|
||||
},
|
||||
"scanner_version": {
|
||||
"type": "string"
|
||||
},
|
||||
"candidate_generator_version": {
|
||||
"type": "string"
|
||||
},
|
||||
"quality_criteria_version": {
|
||||
"type": "string"
|
||||
},
|
||||
"prompt_version": {
|
||||
"type": ["string", "null"]
|
||||
},
|
||||
"release_binding_status": {
|
||||
"enum": ["complete", "historical_incomplete", "unbound"]
|
||||
},
|
||||
"release_binding_note": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"execution": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"mode",
|
||||
"analysis_run_id",
|
||||
"candidate_source",
|
||||
"acceptance_mode"
|
||||
],
|
||||
"properties": {
|
||||
"mode": {
|
||||
"enum": [
|
||||
"deterministic-only",
|
||||
"llm-assisted",
|
||||
"agent-reviewed",
|
||||
"manual-review",
|
||||
"trusted-auto-review",
|
||||
"mixed"
|
||||
]
|
||||
},
|
||||
"analysis_run_id": {
|
||||
"type": ["integer", "null"]
|
||||
},
|
||||
"candidate_source": {
|
||||
"type": "string"
|
||||
},
|
||||
"acceptance_mode": {
|
||||
"type": "string"
|
||||
},
|
||||
"started_at": {
|
||||
"type": ["string", "null"],
|
||||
"format": "date-time"
|
||||
},
|
||||
"completed_at": {
|
||||
"type": ["string", "null"],
|
||||
"format": "date-time"
|
||||
}
|
||||
}
|
||||
},
|
||||
"assessment": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"role",
|
||||
"outcome",
|
||||
"summary",
|
||||
"reviewer",
|
||||
"comparison_eligibility"
|
||||
],
|
||||
"properties": {
|
||||
"role": {
|
||||
"enum": ["baseline", "challenger", "negative_regression_seed"]
|
||||
},
|
||||
"outcome": {
|
||||
"enum": ["baseline", "challenger", "preferred", "tied", "rejected", "superseded", "needs-human"]
|
||||
},
|
||||
"summary": {
|
||||
"type": "string"
|
||||
},
|
||||
"reviewer": {
|
||||
"type": "string"
|
||||
},
|
||||
"comparison_eligibility": {
|
||||
"enum": ["eligible", "eligible_as_negative_seed", "not_comparable"]
|
||||
},
|
||||
"rationale": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"factSummary": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["counts_by_kind"],
|
||||
"properties": {
|
||||
"counts_by_kind": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
}
|
||||
},
|
||||
"contamination_sources": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/contaminationSource"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"contaminationSource": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["path", "reason"],
|
||||
"properties": {
|
||||
"path": {
|
||||
"type": "string"
|
||||
},
|
||||
"reason": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"contentChunkSummary": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["total", "counts_by_kind", "counts_by_source_role", "paths"],
|
||||
"properties": {
|
||||
"total": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"counts_by_kind": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
}
|
||||
},
|
||||
"counts_by_source_role": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
}
|
||||
},
|
||||
"paths": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"generatedTree": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["abilities"],
|
||||
"properties": {
|
||||
"abilities": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/ability"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"ability": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["name", "status", "primary_class", "source_refs", "capabilities"],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"type": "string"
|
||||
},
|
||||
"primary_class": {
|
||||
"type": "string"
|
||||
},
|
||||
"source_refs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/sourceRef"
|
||||
}
|
||||
},
|
||||
"capabilities": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/capability"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"capability": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"name",
|
||||
"status",
|
||||
"primary_class",
|
||||
"source_refs",
|
||||
"features",
|
||||
"evidence"
|
||||
],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"type": "string"
|
||||
},
|
||||
"primary_class": {
|
||||
"type": "string"
|
||||
},
|
||||
"source_refs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/sourceRef"
|
||||
}
|
||||
},
|
||||
"features": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/feature"
|
||||
}
|
||||
},
|
||||
"evidence": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/candidateEvidence"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"feature": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"name",
|
||||
"type",
|
||||
"status",
|
||||
"primary_class",
|
||||
"location",
|
||||
"source_refs"
|
||||
],
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": {
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"type": "string"
|
||||
},
|
||||
"primary_class": {
|
||||
"type": "string"
|
||||
},
|
||||
"location": {
|
||||
"type": "string"
|
||||
},
|
||||
"source_refs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/sourceRef"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"candidateEvidence": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["type", "reference", "strength", "status", "source_refs"],
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string"
|
||||
},
|
||||
"reference": {
|
||||
"type": "string"
|
||||
},
|
||||
"strength": {
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"type": "string"
|
||||
},
|
||||
"source_refs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/sourceRef"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"sourceRef": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["fact_id", "path", "kind", "name", "line"],
|
||||
"properties": {
|
||||
"fact_id": {
|
||||
"type": ["integer", "null"]
|
||||
},
|
||||
"path": {
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"type": "string"
|
||||
},
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"line": {
|
||||
"type": ["integer", "null"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"approvedMap": {
|
||||
"type": "object",
|
||||
"description": "Current approved ability map at export time.",
|
||||
"additionalProperties": true
|
||||
},
|
||||
"reviewDecision": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"repository_id",
|
||||
"analysis_run_id",
|
||||
"action",
|
||||
"notes",
|
||||
"created_at"
|
||||
],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "integer"
|
||||
},
|
||||
"repository_id": {
|
||||
"type": "integer"
|
||||
},
|
||||
"analysis_run_id": {
|
||||
"type": ["integer", "null"]
|
||||
},
|
||||
"action": {
|
||||
"type": "string"
|
||||
},
|
||||
"notes": {
|
||||
"type": "string"
|
||||
},
|
||||
"created_at": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"qualityGateOutcome": {
|
||||
"type": "object",
|
||||
"description": "Versioned deterministic quality-gate outcome. Empty until RREG-WP-0014 introduces gates.",
|
||||
"additionalProperties": true
|
||||
},
|
||||
"regressionPattern": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "title", "severity", "description", "detection_hint"],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"title": {
|
||||
"type": "string"
|
||||
},
|
||||
"severity": {
|
||||
"enum": ["low", "medium", "high", "critical"]
|
||||
},
|
||||
"description": {
|
||||
"type": "string"
|
||||
},
|
||||
"detection_hint": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"examples": [
|
||||
{
|
||||
"schema_version": "self-scoping-assessment/v1",
|
||||
"artifact_id": "repo-scoping-known-bad-2026-05-15-run-39",
|
||||
"artifact_type": "assessment_run",
|
||||
"created_at": "2026-05-15T09:28:48Z",
|
||||
"target_repository": {
|
||||
"repo_slug": "repo-scoping",
|
||||
"repository_id": 16,
|
||||
"source": "/home/worsch/repo-scoping/var/checkouts/repo-scoping-8a9c4168485c",
|
||||
"target_commit": "00b57d509124789059639fedc724d9314edbb7b2",
|
||||
"target_branch": "main",
|
||||
"dirty_state": "unknown",
|
||||
"file_count": 96
|
||||
},
|
||||
"engine_identity": {
|
||||
"repo_scoping_version": "0.1.0",
|
||||
"engine_commit": null,
|
||||
"engine_release": null,
|
||||
"engine_dirty_state": "unknown",
|
||||
"scanner_version": "deterministic-v0.1",
|
||||
"candidate_generator_version": "unversioned",
|
||||
"quality_criteria_version": "none",
|
||||
"prompt_version": null,
|
||||
"release_binding_status": "historical_incomplete",
|
||||
"release_binding_note": "Historical database run did not record engine commit."
|
||||
},
|
||||
"execution": {
|
||||
"mode": "trusted-auto-review",
|
||||
"analysis_run_id": 39,
|
||||
"candidate_source": "deterministic",
|
||||
"acceptance_mode": "trusted_auto_approve_candidate_graph",
|
||||
"started_at": "2026-05-15T09:28:47Z",
|
||||
"completed_at": "2026-05-15T09:28:48Z"
|
||||
},
|
||||
"assessment": {
|
||||
"role": "negative_regression_seed",
|
||||
"outcome": "rejected",
|
||||
"summary": "Provider vocabulary was promoted into a false native LLM routing capability.",
|
||||
"reviewer": "codex",
|
||||
"comparison_eligibility": "eligible_as_negative_seed",
|
||||
"rationale": ["The generated tree misclassified scanner vocabulary as product behavior."]
|
||||
},
|
||||
"fact_summary": {
|
||||
"counts_by_kind": {
|
||||
"llm_provider": 41
|
||||
},
|
||||
"contamination_sources": [
|
||||
{
|
||||
"path": "src/repo_scoping/repo_scanning/scanner.py",
|
||||
"reason": "Scanner rule vocabulary was treated as repo-owned capability evidence."
|
||||
}
|
||||
]
|
||||
},
|
||||
"content_chunk_summary": {
|
||||
"total": 0,
|
||||
"counts_by_kind": {},
|
||||
"counts_by_source_role": {},
|
||||
"paths": []
|
||||
},
|
||||
"generated_tree": {
|
||||
"abilities": [
|
||||
{
|
||||
"name": "Support Repo Registry",
|
||||
"status": "approved",
|
||||
"primary_class": "repository-intelligence",
|
||||
"source_refs": [],
|
||||
"capabilities": [
|
||||
{
|
||||
"name": "Route LLM Requests Across Providers",
|
||||
"status": "approved",
|
||||
"primary_class": "llm-integration",
|
||||
"source_refs": [],
|
||||
"features": [],
|
||||
"evidence": []
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"approved_map": {},
|
||||
"review_decisions": [],
|
||||
"quality_gate_outcomes": [],
|
||||
"known_regression_patterns": [
|
||||
{
|
||||
"id": "RREG-SELF-REG-001",
|
||||
"title": "LLM provider vocabulary promoted as native capability",
|
||||
"severity": "critical",
|
||||
"description": "Scanner or fixture vocabulary becomes a repo-scoping product capability.",
|
||||
"detection_hint": "Flag Route LLM Requests Across Providers when parented as a native repo-scoping capability."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
98
docs/self-scoping/README.md
Normal file
98
docs/self-scoping/README.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# Self-Scoping Assessment Artifacts
|
||||
|
||||
This directory contains repo-scoping's own baseline and assessment artifacts.
|
||||
These files are meant to make scoping-engine changes comparable across releases
|
||||
instead of relying on memory or screenshots.
|
||||
|
||||
## Artifact Types
|
||||
|
||||
- `golden/repo-scoping-golden-profile.v1.json` is the curated target profile for
|
||||
repo-scoping itself.
|
||||
- `assessments/repo-scoping-known-bad-2026-05-15-run-39.json` captures the
|
||||
known-bad self-analysis that promoted LLM-provider vocabulary into native
|
||||
repo-scoping capability truth.
|
||||
- `assessments/repo-scoping-post-wp0015-clean-2026-05-15.json` captures the
|
||||
first clean, release-bound deterministic challenger after acceptance-boundary
|
||||
and input-hygiene work. It remains a rejected regression because candidate
|
||||
generation still collapses repo-scoping's native surfaces under the forbidden
|
||||
provider-routing capability, but its source set no longer includes
|
||||
`var/checkouts/` contamination.
|
||||
- `assessments/repo-scoping-post-wp0016-native-2026-05-15.json` captures the
|
||||
first deterministic challenger after native candidate generation recovery. It
|
||||
matches every expected capability in the golden profile and has no known
|
||||
provider-routing regression, while still leaving generated candidates pending
|
||||
review with quality-gate signals.
|
||||
- `workflow.md` explains how to run challenger assessments, interpret outcomes,
|
||||
and decide whether to update the golden profile or fix the engine.
|
||||
- `outcomes/` stores append-only reviewer decisions created from side-by-side
|
||||
comparisons.
|
||||
- `../schemas/self-scoping-assessment.schema.json` defines the immutable
|
||||
assessment-run artifact shape.
|
||||
|
||||
## Release Binding
|
||||
|
||||
Comparable assessment artifacts must bind generated results to the repo-scoping
|
||||
engine release that produced them. A complete binding records package version,
|
||||
engine git commit or release tag, dirty state, scanner version, candidate
|
||||
generator version, quality criteria version, and prompt version when applicable.
|
||||
|
||||
The current known-bad artifact is marked `historical_incomplete` because the
|
||||
original database run did not record the engine commit. It remains useful as a
|
||||
negative regression seed, but future challenger artifacts should be fully bound
|
||||
before they are accepted as comparable baselines.
|
||||
|
||||
## Review Use
|
||||
|
||||
When the engine changes, run repo-scoping against itself and export a challenger
|
||||
assessment. Compare the challenger to the golden profile and to the negative
|
||||
seed. Reviewers should be able to choose whether the old result, new result, or
|
||||
neither is better, then store that judgement as a new assessment outcome.
|
||||
|
||||
The curator UI exposes this loop at `/ui/self-scoping`. It reads the golden and
|
||||
assessment JSON files from this directory, highlights missing, forbidden, and
|
||||
misplaced hierarchy entries, and records reviewer preference without mutating
|
||||
the compared artifacts. The same page can compare two assessment runs directly
|
||||
so reviewers can choose whether the old baseline or new challenger is better.
|
||||
|
||||
## Export Command
|
||||
|
||||
Export a completed analysis run as a challenger artifact:
|
||||
|
||||
```bash
|
||||
repo-scoping export-assessment \
|
||||
--repo repo-scoping \
|
||||
--analysis-run 39 \
|
||||
--output docs/self-scoping/assessments/repo-scoping-challenger-run-39.json
|
||||
```
|
||||
|
||||
The command reads an existing registry database and does not clone or scan the
|
||||
target repository. It records the target analysis metadata, candidate graph,
|
||||
approved map at export time, review decisions, fact and content summaries, known
|
||||
regression patterns, and current repo-scoping engine identity.
|
||||
|
||||
Compare an assessment against the curated golden profile:
|
||||
|
||||
```bash
|
||||
repo-scoping compare-assessment \
|
||||
--golden docs/self-scoping/golden/repo-scoping-golden-profile.v1.json \
|
||||
--assessment docs/self-scoping/assessments/repo-scoping-known-bad-2026-05-15-run-39.json \
|
||||
--format markdown
|
||||
```
|
||||
|
||||
The first comparison report highlights missing expected capabilities, forbidden
|
||||
native capabilities, known regression patterns, and misplaced API/CLI features.
|
||||
|
||||
Run the full self-assessment loop:
|
||||
|
||||
```bash
|
||||
repo-scoping self-assess \
|
||||
--source-path . \
|
||||
--assessment-output docs/self-scoping/assessments/repo-scoping-challenger.json \
|
||||
--comparison-output docs/self-scoping/assessments/repo-scoping-challenger.md
|
||||
```
|
||||
|
||||
By default this path is deterministic-only and leaves generated candidates
|
||||
pending review. Add `--with-llm` only when a provider is configured and the run
|
||||
should include LLM-assisted candidate extraction. Add `--fail-on-regression` in
|
||||
CI when known regressions should fail the command; ordinary `needs_review`
|
||||
comparisons still exit successfully.
|
||||
@@ -0,0 +1,240 @@
|
||||
{
|
||||
"schema_version": "self-scoping-assessment/v1",
|
||||
"artifact_id": "repo-scoping-known-bad-2026-05-15-run-39",
|
||||
"artifact_type": "assessment_run",
|
||||
"created_at": "2026-05-15T09:28:48Z",
|
||||
"target_repository": {
|
||||
"repo_slug": "repo-scoping",
|
||||
"repository_id": 16,
|
||||
"source": "/home/worsch/repo-scoping/var/checkouts/repo-scoping-8a9c4168485c",
|
||||
"target_commit": "00b57d509124789059639fedc724d9314edbb7b2",
|
||||
"target_branch": "main",
|
||||
"dirty_state": "unknown",
|
||||
"file_count": 96
|
||||
},
|
||||
"engine_identity": {
|
||||
"repo_scoping_version": "0.1.0",
|
||||
"engine_commit": null,
|
||||
"engine_release": null,
|
||||
"engine_dirty_state": "unknown",
|
||||
"scanner_version": "deterministic-v0.1",
|
||||
"candidate_generator_version": "unversioned-pre-self-scoping-baseline",
|
||||
"quality_criteria_version": "none",
|
||||
"prompt_version": null,
|
||||
"release_binding_status": "historical_incomplete",
|
||||
"release_binding_note": "This historical database run recorded scanner version and target commit, but not the repo-scoping engine commit or release tag that generated the candidate graph."
|
||||
},
|
||||
"execution": {
|
||||
"mode": "trusted-auto-review",
|
||||
"analysis_run_id": 39,
|
||||
"candidate_source": "deterministic",
|
||||
"acceptance_mode": "trusted_auto_approve_candidate_graph",
|
||||
"started_at": "2026-05-15T09:28:47Z",
|
||||
"completed_at": "2026-05-15T09:28:48Z"
|
||||
},
|
||||
"assessment": {
|
||||
"role": "negative_regression_seed",
|
||||
"outcome": "rejected",
|
||||
"summary": "The self-analysis promoted LLM-provider vocabulary into a false native repo-scoping capability and attached API/CLI features below it.",
|
||||
"reviewer": "codex",
|
||||
"comparison_eligibility": "eligible_as_negative_seed",
|
||||
"rationale": [
|
||||
"repo-scoping uses llm-connect as optional extraction infrastructure; it does not natively route LLM requests across providers.",
|
||||
"Provider names came from scanner rules, normalization tokens, schema examples, tests, fixtures, and workplan text rather than product-facing provider-routing behavior.",
|
||||
"The generated tree placed native API and CLI surfaces under the false LLM-provider capability, which makes the feature hierarchy misleading."
|
||||
]
|
||||
},
|
||||
"fact_summary": {
|
||||
"counts_by_kind": {
|
||||
"config": 1,
|
||||
"credential_config": 13,
|
||||
"documentation": 14,
|
||||
"fallback_policy": 10,
|
||||
"framework": 2,
|
||||
"intent": 1,
|
||||
"interface": 127,
|
||||
"language": 1,
|
||||
"llm_provider": 41,
|
||||
"manifest": 1,
|
||||
"provider_registry": 7,
|
||||
"scope": 1,
|
||||
"test": 19
|
||||
},
|
||||
"contamination_sources": [
|
||||
{
|
||||
"path": "src/repo_registry/repo_scanning/scanner.py",
|
||||
"reason": "Provider detector constants, credential hint constants, and fallback/provider-registry scanner logic were treated as repo-owned LLM routing evidence."
|
||||
},
|
||||
{
|
||||
"path": "src/repo_registry/candidate_graph/normalization.py",
|
||||
"reason": "Provider names used as distinctive candidate-normalization tokens were treated as implementation evidence for provider support."
|
||||
},
|
||||
{
|
||||
"path": "src/repo_registry/web_api/schemas.py",
|
||||
"reason": "An OpenRouter example in an expectation-gap schema was treated as provider evidence."
|
||||
},
|
||||
{
|
||||
"path": "tests/expectations/llm_connect_provider_expectations.json",
|
||||
"reason": "A fixture describing llm-connect expectations was treated as repo-scoping product behavior."
|
||||
},
|
||||
{
|
||||
"path": "tests/fixtures.py",
|
||||
"reason": "Regression fixture vocabulary was treated as native repo-scoping capability evidence."
|
||||
},
|
||||
{
|
||||
"path": "tests/test_candidate_graph.py",
|
||||
"reason": "Unit-test examples for LLM-provider detection were treated as product evidence."
|
||||
},
|
||||
{
|
||||
"path": "tests/test_repository_scanner.py",
|
||||
"reason": "Scanner tests for provider facts were treated as native provider-routing evidence."
|
||||
}
|
||||
]
|
||||
},
|
||||
"content_chunk_summary": {
|
||||
"total": 0,
|
||||
"counts_by_kind": {},
|
||||
"counts_by_source_role": {},
|
||||
"paths": []
|
||||
},
|
||||
"generated_tree": {
|
||||
"abilities": [
|
||||
{
|
||||
"name": "Support Repo Registry",
|
||||
"status": "approved",
|
||||
"primary_class": "repository-intelligence",
|
||||
"source_refs": [],
|
||||
"capabilities": [
|
||||
{
|
||||
"name": "Route LLM Requests Across Providers",
|
||||
"status": "approved",
|
||||
"primary_class": "llm-integration",
|
||||
"source_refs": [],
|
||||
"features": [
|
||||
{
|
||||
"name": "Use Anthropic Models",
|
||||
"type": "integration",
|
||||
"status": "approved",
|
||||
"primary_class": "integration",
|
||||
"location": "multiple files",
|
||||
"source_refs": []
|
||||
},
|
||||
{
|
||||
"name": "Use Claude Models",
|
||||
"type": "integration",
|
||||
"status": "approved",
|
||||
"primary_class": "integration",
|
||||
"location": "multiple files",
|
||||
"source_refs": []
|
||||
},
|
||||
{
|
||||
"name": "Use Gemini Models",
|
||||
"type": "integration",
|
||||
"status": "approved",
|
||||
"primary_class": "integration",
|
||||
"location": "multiple files",
|
||||
"source_refs": []
|
||||
},
|
||||
{
|
||||
"name": "Use OpenAI Models",
|
||||
"type": "integration",
|
||||
"status": "approved",
|
||||
"primary_class": "integration",
|
||||
"location": "multiple files",
|
||||
"source_refs": []
|
||||
},
|
||||
{
|
||||
"name": "Use OpenRouter Models",
|
||||
"type": "integration",
|
||||
"status": "approved",
|
||||
"primary_class": "integration",
|
||||
"location": "multiple files",
|
||||
"source_refs": []
|
||||
},
|
||||
{
|
||||
"name": "Configure LLM Provider Credentials",
|
||||
"type": "configuration",
|
||||
"status": "approved",
|
||||
"primary_class": "configuration",
|
||||
"location": "multiple files",
|
||||
"source_refs": []
|
||||
},
|
||||
{
|
||||
"name": "Maintain LLM Provider Registry",
|
||||
"type": "backend",
|
||||
"status": "approved",
|
||||
"primary_class": "backend",
|
||||
"location": "src/repo_registry/repo_scanning/scanner.py",
|
||||
"source_refs": []
|
||||
},
|
||||
{
|
||||
"name": "Apply LLM Provider Fallback Policy",
|
||||
"type": "backend",
|
||||
"status": "approved",
|
||||
"primary_class": "backend",
|
||||
"location": "src/repo_registry/repo_scanning/scanner.py",
|
||||
"source_refs": []
|
||||
},
|
||||
{
|
||||
"name": "HTTP API surface: possible API surface, GET /health, @app.post(, +43 more",
|
||||
"type": "API",
|
||||
"status": "approved",
|
||||
"primary_class": "API",
|
||||
"location": "multiple files",
|
||||
"source_refs": []
|
||||
},
|
||||
{
|
||||
"name": "CLI command surface: CLI command build_parser, CLI command make_service",
|
||||
"type": "CLI",
|
||||
"status": "approved",
|
||||
"primary_class": "CLI",
|
||||
"location": "multiple files",
|
||||
"source_refs": []
|
||||
}
|
||||
],
|
||||
"evidence": []
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"approved_map": {},
|
||||
"review_decisions": [
|
||||
{
|
||||
"id": 21,
|
||||
"repository_id": 16,
|
||||
"analysis_run_id": 39,
|
||||
"action": "trusted_auto_approve_candidate_graph",
|
||||
"notes": "Trusted auto-populate mode reviewed candidate graph after deterministic candidate generation. Auto-approved 1 safe candidate capability(s); left 0 for review. Approved: Route LLM Requests Across Providers: eligible LLM utility relationship with source support.",
|
||||
"created_at": "2026-05-15 09:28:49"
|
||||
}
|
||||
],
|
||||
"quality_gate_outcomes": [],
|
||||
"known_regression_patterns": [
|
||||
{
|
||||
"id": "RREG-SELF-REG-001",
|
||||
"title": "LLM provider vocabulary promoted as native capability",
|
||||
"severity": "critical",
|
||||
"description": "Scanner, normalization, schema, fixture, test, or workplan vocabulary becomes the native repo-scoping capability Route LLM Requests Across Providers.",
|
||||
"detection_hint": "Flag any top-level/native repo-scoping capability named Route LLM Requests Across Providers unless product intent and public implementation explicitly show provider routing as a repo-scoping feature."
|
||||
},
|
||||
{
|
||||
"id": "RREG-SELF-REG-002",
|
||||
"title": "Native API and CLI surfaces attached under false capability",
|
||||
"severity": "high",
|
||||
"description": "General repo-scoping API/CLI interface features are nested below a capability they do not support.",
|
||||
"detection_hint": "Flag API or CLI surface features when their parent capability is llm-integration or provider-routing."
|
||||
},
|
||||
{
|
||||
"id": "RREG-SELF-REG-003",
|
||||
"title": "Deterministic trusted auto-approval accepted candidate truth",
|
||||
"severity": "high",
|
||||
"description": "A deterministic rule path approves candidate characteristics without human or agentic judgement.",
|
||||
"detection_hint": "Flag trusted_auto_approve_candidate_graph review decisions in self-scoping assessment artifacts."
|
||||
}
|
||||
],
|
||||
"notes": [
|
||||
"This artifact is a negative regression seed, not a desirable baseline.",
|
||||
"The historical run is useful for pattern detection but is not fully release-bound because the engine commit was not recorded in the original analysis metadata."
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,36 @@
|
||||
# Self-Scoping Comparison: repo-scoping-challenger-run-1
|
||||
|
||||
- Status: `regression`
|
||||
- Golden profile: `repo-scoping-golden-profile-v1`
|
||||
- Target repo: `repo-scoping`
|
||||
- Summary: Assessment repeats known or forbidden self-scoping patterns; prefer the golden profile until the engine is corrected.
|
||||
|
||||
## Missing Expected Capabilities
|
||||
- Explore Dependency And Impact Graphs
|
||||
- Generate And Maintain SCOPE.md
|
||||
- Generate Reviewable Candidate Characteristics
|
||||
- Index Source Content With Provenance
|
||||
- Provide Scope Context To Downstream Agents
|
||||
- Register And Track Repositories
|
||||
- Review And Approve Candidate Characteristics
|
||||
- Scan Repositories Into Observed Facts
|
||||
- Search Compare And Export Approved Profiles
|
||||
|
||||
## Forbidden Native Capabilities Present
|
||||
- Route LLM Requests Across Providers
|
||||
|
||||
## Known Regression Patterns
|
||||
- `RREG-SELF-REG-001` LLM provider vocabulary promoted as native capability: Generated tree contains Route LLM Requests Across Providers as a repo-scoping capability.
|
||||
- `RREG-SELF-REG-002` Native API and CLI surfaces attached under false capability: API or CLI surface features are nested below provider routing.
|
||||
|
||||
## Misplaced Features
|
||||
- `HTTP API surface: possible API surface, GET /health, @app.get(, +49 more` under `Route LLM Requests Across Providers` (API): API/CLI surface is nested below provider-routing capability.
|
||||
- `CLI command surface: CLI command build_parser, CLI command make_service` under `Route LLM Requests Across Providers` (CLI): API/CLI surface is nested below provider-routing capability.
|
||||
|
||||
## Matched Expected Capabilities
|
||||
- None
|
||||
|
||||
## Review Hints
|
||||
- Do not promote this assessment as a preferred baseline.
|
||||
- Inspect forbidden capabilities and misplaced features first.
|
||||
- Use the findings as signal for scanner, generator, or acceptance-policy changes.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,33 @@
|
||||
# Self-Scoping Comparison: repo-scoping-challenger-run-1
|
||||
|
||||
- Status: `candidate_improvement`
|
||||
- Golden profile: `repo-scoping-golden-profile-v1`
|
||||
- Target repo: `repo-scoping`
|
||||
- Summary: Assessment covers the golden profile without known regression patterns.
|
||||
|
||||
## Missing Expected Capabilities
|
||||
- None
|
||||
|
||||
## Forbidden Native Capabilities Present
|
||||
- None
|
||||
|
||||
## Known Regression Patterns
|
||||
- None
|
||||
|
||||
## Misplaced Features
|
||||
- None
|
||||
|
||||
## Matched Expected Capabilities
|
||||
- Explore Dependency And Impact Graphs
|
||||
- Generate And Maintain SCOPE.md
|
||||
- Generate Reviewable Candidate Characteristics
|
||||
- Index Source Content With Provenance
|
||||
- Provide Scope Context To Downstream Agents
|
||||
- Register And Track Repositories
|
||||
- Review And Approve Candidate Characteristics
|
||||
- Scan Repositories Into Observed Facts
|
||||
- Search Compare And Export Approved Profiles
|
||||
|
||||
## Review Hints
|
||||
- Candidate appears better than the known golden checks.
|
||||
- Human or agentic review should still confirm source evidence quality.
|
||||
311
docs/self-scoping/golden/repo-scoping-golden-profile.v1.json
Normal file
311
docs/self-scoping/golden/repo-scoping-golden-profile.v1.json
Normal file
@@ -0,0 +1,311 @@
|
||||
{
|
||||
"schema_version": "self-scoping-golden-profile/v1",
|
||||
"profile_id": "repo-scoping-golden-profile-v1",
|
||||
"repo_slug": "repo-scoping",
|
||||
"status": "active",
|
||||
"created_at": "2026-05-15",
|
||||
"updated_at": "2026-05-15",
|
||||
"curation": {
|
||||
"curator": "codex",
|
||||
"workplan_id": "RREG-WP-0013",
|
||||
"summary": "Curated target profile for evaluating repo-scoping self-analysis quality."
|
||||
},
|
||||
"ability": {
|
||||
"name": "Map Repositories Into Reviewable Scope Profiles",
|
||||
"primary_class": "repository-intelligence",
|
||||
"attributes": [
|
||||
"capability-mapping",
|
||||
"source-linked-review",
|
||||
"scope-generation"
|
||||
],
|
||||
"description": "repo-scoping turns repository source, documentation, and review decisions into source-linked maps of repository utility.",
|
||||
"expected_capabilities": [
|
||||
{
|
||||
"name": "Register And Track Repositories",
|
||||
"primary_class": "ingestion",
|
||||
"attributes": ["metadata", "git", "analysis-run"],
|
||||
"expected_features": [
|
||||
{
|
||||
"name": "Create and update repository records",
|
||||
"primary_class": "api",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/core/service.py",
|
||||
"src/repo_scoping/web_api/app.py",
|
||||
"src/repo_scoping/web_ui/views.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Resolve local or remote Git checkouts",
|
||||
"primary_class": "backend",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/repo_ingestion/git.py",
|
||||
"tests/test_git_ingestion.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Import repository metadata",
|
||||
"primary_class": "backend",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/repo_ingestion/metadata.py",
|
||||
"tests/test_repository_metadata.py"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Scan Repositories Into Observed Facts",
|
||||
"primary_class": "analysis",
|
||||
"attributes": ["deterministic", "facts", "provenance"],
|
||||
"expected_features": [
|
||||
{
|
||||
"name": "Detect source languages, manifests, docs, tests, config, and interfaces",
|
||||
"primary_class": "backend",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/repo_scanning/scanner.py",
|
||||
"tests/test_repository_scanner.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Classify source roles for facts",
|
||||
"primary_class": "backend",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/repo_scanning/scanner.py",
|
||||
"docs/characteristic-evidence-model.md"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Preserve analysis snapshots and fact records",
|
||||
"primary_class": "storage",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/storage/sqlite.py",
|
||||
"migrations/0001_initial.sql"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Index Source Content With Provenance",
|
||||
"primary_class": "analysis",
|
||||
"attributes": ["content-chunks", "source-role"],
|
||||
"expected_features": [
|
||||
{
|
||||
"name": "Create source-linked content chunks from observed facts",
|
||||
"primary_class": "backend",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/content_indexing/extractor.py",
|
||||
"tests/test_content_indexing.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Carry source-role metadata into downstream generation",
|
||||
"primary_class": "backend",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/content_indexing/extractor.py",
|
||||
"src/repo_scoping/llm_extraction/extractor.py"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Generate Reviewable Candidate Characteristics",
|
||||
"primary_class": "analysis",
|
||||
"attributes": ["candidate-graph", "review-required"],
|
||||
"expected_features": [
|
||||
{
|
||||
"name": "Build candidate abilities, capabilities, features, and evidence",
|
||||
"primary_class": "backend",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/candidate_graph/generator.py",
|
||||
"src/repo_scoping/candidate_graph/normalization.py",
|
||||
"tests/test_candidate_graph.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Optionally map structured LLM extraction into candidates",
|
||||
"primary_class": "integration",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/llm_extraction/extractor.py",
|
||||
"src/repo_scoping/llm_extraction/mapper.py",
|
||||
"tests/test_llm_extraction.py"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Review And Approve Candidate Characteristics",
|
||||
"primary_class": "review",
|
||||
"attributes": ["curation", "approval", "audit"],
|
||||
"expected_features": [
|
||||
{
|
||||
"name": "Edit, reject, merge, and relink candidate graph entries",
|
||||
"primary_class": "api",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/core/service.py",
|
||||
"src/repo_scoping/web_api/app.py",
|
||||
"src/repo_scoping/web_ui/views.py",
|
||||
"tests/test_registry_service.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Publish approved characteristic maps after review",
|
||||
"primary_class": "storage",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/core/service.py",
|
||||
"src/repo_scoping/storage/sqlite.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Record review decisions and expectation gaps",
|
||||
"primary_class": "audit",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/core/service.py",
|
||||
"src/repo_scoping/web_api/schemas.py"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Search Compare And Export Approved Profiles",
|
||||
"primary_class": "discovery",
|
||||
"attributes": ["search", "comparison", "export"],
|
||||
"expected_features": [
|
||||
{
|
||||
"name": "Search approved abilities, capabilities, features, and evidence",
|
||||
"primary_class": "api",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/core/service.py",
|
||||
"src/repo_scoping/semantic/embeddings.py",
|
||||
"tests/test_registry_service.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Compare repositories and identify capability gaps",
|
||||
"primary_class": "api",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/core/service.py",
|
||||
"src/repo_scoping/web_api/app.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Export repository profiles",
|
||||
"primary_class": "api",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/web_api/app.py",
|
||||
"docs/api-contract.md"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Generate And Maintain SCOPE.md",
|
||||
"primary_class": "scope-generation",
|
||||
"attributes": ["scope-md", "diff", "validation"],
|
||||
"expected_features": [
|
||||
{
|
||||
"name": "Render SCOPE.md from approved characteristics",
|
||||
"primary_class": "backend",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/scope/generator.py",
|
||||
"tests/test_scope_generator.py",
|
||||
"docs/scope-md-spec.md"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Diff, validate, and write scope files",
|
||||
"primary_class": "api",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/scope/validator.py",
|
||||
"src/repo_scoping/web_api/app.py",
|
||||
"src/repo_scoping/web_ui/views.py"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Explore Dependency And Impact Graphs",
|
||||
"primary_class": "dependency-analysis",
|
||||
"attributes": ["graph", "impact", "visualization"],
|
||||
"expected_features": [
|
||||
{
|
||||
"name": "Model dependencies between facts, evidence, features, capabilities, abilities, and scope",
|
||||
"primary_class": "backend",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/core/service.py",
|
||||
"docs/dependency-aware-scope-propagation.md",
|
||||
"docs/dependency-visualization-exploration.md"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Render dependency graph views and profiles",
|
||||
"primary_class": "ui",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/web_ui/views.py",
|
||||
"tests/test_web_api.py"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Provide Scope Context To Downstream Agents",
|
||||
"primary_class": "coordination",
|
||||
"attributes": ["activity-core", "api-contract"],
|
||||
"expected_features": [
|
||||
{
|
||||
"name": "Return compact JSON scope context by repository slug",
|
||||
"primary_class": "api",
|
||||
"source_paths": [
|
||||
"src/repo_scoping/web_api/app.py",
|
||||
"docs/schemas/repo-scope-context-response.json",
|
||||
"tests/test_scope_context_api.py"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"forbidden_native_capabilities": [
|
||||
{
|
||||
"name": "Route LLM Requests Across Providers",
|
||||
"reason": "repo-scoping may use llm-connect as optional extraction infrastructure, but provider routing is not a native repo-scoping product capability.",
|
||||
"allowed_only_if": "Future product intent and public implementation explicitly add provider routing as repo-scoping-owned behavior."
|
||||
}
|
||||
],
|
||||
"non_native_context": [
|
||||
{
|
||||
"name": "LLM provider names in scanner, normalization, schemas, tests, fixtures, docs, or workplans",
|
||||
"classification": "scanner-rule-or-fixture-context",
|
||||
"expected_handling": "May support scanner behavior facts or test coverage, but must not become native capability truth."
|
||||
},
|
||||
{
|
||||
"name": "llm-connect integration",
|
||||
"classification": "optional dependency / adapter consumer",
|
||||
"expected_handling": "May appear as optional extraction infrastructure, not as repo-scoping-owned provider routing."
|
||||
},
|
||||
{
|
||||
"name": "SCOPE.md content",
|
||||
"classification": "derived scope",
|
||||
"expected_handling": "Can be comparison or bootstrap context, not primary evidence for regenerating the same characteristic model."
|
||||
}
|
||||
],
|
||||
"comparison_rules": {
|
||||
"must_have_capability_names": [
|
||||
"Register And Track Repositories",
|
||||
"Scan Repositories Into Observed Facts",
|
||||
"Index Source Content With Provenance",
|
||||
"Generate Reviewable Candidate Characteristics",
|
||||
"Review And Approve Candidate Characteristics",
|
||||
"Search Compare And Export Approved Profiles",
|
||||
"Generate And Maintain SCOPE.md",
|
||||
"Explore Dependency And Impact Graphs",
|
||||
"Provide Scope Context To Downstream Agents"
|
||||
],
|
||||
"must_not_have_native_capability_names": [
|
||||
"Route LLM Requests Across Providers"
|
||||
],
|
||||
"known_regression_ids": [
|
||||
"RREG-SELF-REG-001",
|
||||
"RREG-SELF-REG-002",
|
||||
"RREG-SELF-REG-003"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"baseline_assessment_artifact_id": "repo-scoping-known-bad-2026-05-15-run-39",
|
||||
"baseline_assessment_path": "assessments/repo-scoping-known-bad-2026-05-15-run-39.json",
|
||||
"baseline_engine_identity": {
|
||||
"candidate_generator_version": "unversioned-pre-self-scoping-baseline",
|
||||
"engine_commit": null,
|
||||
"engine_dirty_state": "unknown",
|
||||
"engine_release": null,
|
||||
"prompt_version": null,
|
||||
"quality_criteria_version": "none",
|
||||
"release_binding_note": "This historical database run recorded scanner version and target commit, but not the repo-scoping engine commit or release tag that generated the candidate graph.",
|
||||
"release_binding_status": "historical_incomplete",
|
||||
"repo_scoping_version": "0.1.0",
|
||||
"scanner_version": "deterministic-v0.1"
|
||||
},
|
||||
"challenger_assessment_artifact_id": "repo-scoping-challenger-run-1",
|
||||
"challenger_assessment_path": "assessments/repo-scoping-post-wp0016-native-2026-05-15.json",
|
||||
"challenger_engine_identity": {
|
||||
"candidate_generator_version": "unversioned",
|
||||
"engine_commit": "2c3dad80d646869827335cb6de849cdca272c85b",
|
||||
"engine_dirty_state": "clean",
|
||||
"engine_release": null,
|
||||
"prompt_version": null,
|
||||
"quality_criteria_version": "repo-scoping-quality-criteria/v1",
|
||||
"release_binding_note": "Engine commit was captured from git.",
|
||||
"release_binding_status": "complete",
|
||||
"repo_scoping_version": "0.1.0",
|
||||
"scanner_version": "deterministic-v0.1"
|
||||
},
|
||||
"comparison_status": "needs_review",
|
||||
"created_at": "2026-05-15T19:29:16Z",
|
||||
"decision_scope": "assessment-pair-comparison",
|
||||
"notes": "Massive improvement to the old infrastructure!",
|
||||
"outcome": "prefer_challenger",
|
||||
"outcome_id": "20260515-192916__repo-scoping-known-bad-2026-05-15-run-39__repo-scoping-post-wp0016-native-2026-05-15__prefer_challenger__245606d1",
|
||||
"reviewer": "codex",
|
||||
"schema_version": "self-scoping-review-outcome/v1"
|
||||
}
|
||||
9
docs/self-scoping/outcomes/README.md
Normal file
9
docs/self-scoping/outcomes/README.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Self-Scoping Review Outcomes
|
||||
|
||||
This directory stores append-only review decisions recorded from the
|
||||
self-scoping comparison UI. Outcome files bind a reviewer choice to a golden
|
||||
profile, an assessment artifact, and the repo-scoping engine identity captured
|
||||
in that assessment.
|
||||
|
||||
Do not edit historical assessment artifacts to record a preference. Add a new
|
||||
outcome record instead.
|
||||
129
docs/self-scoping/workflow.md
Normal file
129
docs/self-scoping/workflow.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Self-Scoping Assessment Workflow
|
||||
|
||||
Self-scoping is the feedback loop for improving repo-scoping with evidence. The
|
||||
loop is simple: run the current engine against repo-scoping itself, compare the
|
||||
result to a curated golden profile and known bad runs, then record whether the
|
||||
new result is better.
|
||||
|
||||
## Outcome Terms
|
||||
|
||||
- `baseline`: a result accepted as a reference point for later comparisons.
|
||||
- `challenger`: a fresh result from a new engine version or configuration.
|
||||
- `preferred`: the reviewer chose this result over the prior baseline.
|
||||
- `tied`: the reviewer judged old and new results roughly equivalent.
|
||||
- `rejected`: the result is known bad and should not become baseline truth.
|
||||
- `superseded`: the result used to be useful but was replaced by a newer
|
||||
preferred assessment.
|
||||
- `needs-human`: the result cannot be judged confidently without curator
|
||||
review.
|
||||
|
||||
The known 2026-05-15 run 39 artifact is a `rejected` negative regression seed,
|
||||
not a baseline to imitate.
|
||||
|
||||
## Release Binding
|
||||
|
||||
Assessment output is only useful if it is bound to the engine that generated it.
|
||||
Comparable challenger artifacts should record:
|
||||
|
||||
- repo-scoping package version
|
||||
- engine git commit
|
||||
- engine release or tag when available
|
||||
- engine dirty state
|
||||
- scanner version
|
||||
- candidate generator version
|
||||
- quality criteria version
|
||||
- prompt version when LLM or agentic review is used
|
||||
|
||||
An artifact with `release_binding_status=complete` can be compared as a real
|
||||
challenger. An artifact with `historical_incomplete` can still be useful as a
|
||||
negative seed, but it should not become a preferred baseline. An `unbound`
|
||||
artifact is diagnostic only.
|
||||
|
||||
Dirty state does not automatically make an artifact useless, but it must be
|
||||
visible. A dirty challenger should usually be rerun after the relevant changes
|
||||
are committed.
|
||||
|
||||
## Standard Loop
|
||||
|
||||
1. Run the self-assessment command:
|
||||
|
||||
```bash
|
||||
repo-scoping self-assess \
|
||||
--source-path . \
|
||||
--assessment-output docs/self-scoping/assessments/repo-scoping-challenger.json \
|
||||
--comparison-output docs/self-scoping/assessments/repo-scoping-challenger.md
|
||||
```
|
||||
|
||||
2. Read the comparison report.
|
||||
|
||||
3. Open the curator UI at `/ui/self-scoping` to compare the golden profile and
|
||||
assessment artifact side by side.
|
||||
|
||||
4. When an earlier baseline assessment exists, use the same page's two-run
|
||||
comparison to judge old output against the new challenger.
|
||||
|
||||
5. If the report says `regression`, inspect forbidden capabilities, misplaced
|
||||
features, and known regression patterns first.
|
||||
|
||||
6. If the report says `needs_review`, inspect missing expected capabilities and
|
||||
source evidence before choosing old or new output.
|
||||
|
||||
7. If the report says `candidate_improvement`, still confirm that the
|
||||
hierarchy, source refs, and native-utility boundaries make sense.
|
||||
|
||||
8. Record the decision as an assessment outcome before changing the active
|
||||
baseline. The UI writes append-only outcome records under
|
||||
`docs/self-scoping/outcomes/`; it does not rewrite historical assessment or
|
||||
golden-profile artifacts.
|
||||
|
||||
## CI Use
|
||||
|
||||
Use `--fail-on-regression` only when regressions should block the command:
|
||||
|
||||
```bash
|
||||
repo-scoping self-assess \
|
||||
--source-path . \
|
||||
--comparison-output /tmp/repo-scoping-self-assessment.md \
|
||||
--fail-on-regression
|
||||
```
|
||||
|
||||
The command should not fail for ordinary `needs_review` results. Review-needed
|
||||
output is signal, not a broken build.
|
||||
|
||||
## Updating The Golden Profile
|
||||
|
||||
Update `golden/repo-scoping-golden-profile.v1.json` when the repository's real
|
||||
product utility has changed. Examples:
|
||||
|
||||
- repo-scoping adds a genuinely new user-facing capability.
|
||||
- a capability is renamed after curator agreement.
|
||||
- a former out-of-scope behavior becomes product intent and has supporting
|
||||
implementation evidence.
|
||||
|
||||
Do not update the golden profile just because the engine failed to find an
|
||||
expected capability. That is usually an engine issue.
|
||||
|
||||
## Fixing The Engine
|
||||
|
||||
Fix the engine when a challenger:
|
||||
|
||||
- repeats a known regression pattern
|
||||
- promotes dependency, fixture, schema, scanner-rule, or workplan vocabulary as
|
||||
native capability truth
|
||||
- places features under a capability they do not support
|
||||
- loses source refs or cites evidence that does not support the abstraction
|
||||
- relies on generated `SCOPE.md` as primary proof for rebuilding the same model
|
||||
|
||||
The 2026-05-15 run 39 failure is the canonical example: provider vocabulary from
|
||||
scanner code, tests, fixtures, and schema examples became the false native
|
||||
capability `Route LLM Requests Across Providers`. The correct action is to fix
|
||||
scanner/generator/acceptance behavior, not to teach the golden profile that
|
||||
repo-scoping is an LLM router.
|
||||
|
||||
## Relationship To Agentic Acceptance
|
||||
|
||||
Deterministic assessment can reject, downgrade, or flag output with transparent
|
||||
criteria. It should not approve candidate characteristics as registry truth.
|
||||
When automation stands in for human review, the decision belongs to an agentic
|
||||
reviewer that inspects evidence, applies versioned criteria, and records a
|
||||
rationale. That acceptance redesign is tracked in `RREG-WP-0014`.
|
||||
@@ -11,7 +11,7 @@ development. It produces deterministic token-bucket vectors without any network
|
||||
dependency. Configure it with:
|
||||
|
||||
```bash
|
||||
REPO_REGISTRY_EMBEDDING_PROVIDER=hashing
|
||||
REPO_SCOPING_EMBEDDING_PROVIDER=hashing
|
||||
```
|
||||
|
||||
When enabled, search combines:
|
||||
|
||||
@@ -10,10 +10,10 @@ repository is for and how that claim is supported.
|
||||
- Repository Scoping is the product and UI name.
|
||||
- `repo-scoping` is the managed repository slug, Git remote identity, and State
|
||||
Hub repository identity.
|
||||
- `repo_registry`, `REPO_REGISTRY_`, and `var/repo-registry.sqlite3` are retained
|
||||
compatibility names in code and local configuration.
|
||||
- Repository Ability Registry and `repo-registry` are historical names from
|
||||
before the scope-oriented rename.
|
||||
- `repo_scoping`, `REPO_SCOPING_`, and `var/repo-scoping.sqlite3` are the
|
||||
current package, environment-prefix, and default SQLite names.
|
||||
- Earlier registry-oriented names are historical only and should not be used in
|
||||
runtime configuration, package imports, or new documentation.
|
||||
|
||||
## Characteristic Model
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@ requires = ["setuptools>=68"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "repo-registry"
|
||||
name = "repo-scoping"
|
||||
version = "0.1.0"
|
||||
description = "Repository Scoping"
|
||||
readme = "README.md"
|
||||
@@ -22,7 +22,7 @@ dev = [
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
repo-scoping = "repo_registry.cli:main"
|
||||
repo-scoping = "repo_scoping.cli:main"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
12
registry/README.md
Normal file
12
registry/README.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Capability Registry
|
||||
|
||||
Markdown-first capability index for federation and reuse planning.
|
||||
|
||||
## Authoring
|
||||
|
||||
1. Copy a capability entry template (see reuse-surface `templates/capability-entry.template.md`).
|
||||
2. Add the row to `indexes/capabilities.yaml`.
|
||||
3. Run `reuse-surface validate` from a checkout with the CLI installed.
|
||||
4. Merge to `main` and verify publish with `reuse-surface establish --publish-check`.
|
||||
|
||||
Federation contract: reuse-surface `docs/RegistryFederation.md`.
|
||||
0
registry/capabilities/.gitkeep
Normal file
0
registry/capabilities/.gitkeep
Normal file
4
registry/indexes/capabilities.yaml
Normal file
4
registry/indexes/capabilities.yaml
Normal file
@@ -0,0 +1,4 @@
|
||||
version: 1
|
||||
updated: '2026-06-16'
|
||||
domain: helix_forge
|
||||
capabilities: []
|
||||
@@ -1,157 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from typing import Sequence
|
||||
|
||||
from repo_registry.core.models import CharacteristicRebuildResult, Repository
|
||||
from repo_registry.core.service import RegistryService
|
||||
from repo_registry.llm_extraction import LLMCandidateExtractor, create_llm_connect_adapter
|
||||
from repo_registry.repo_ingestion.git import GitIngestionService
|
||||
from repo_registry.storage.sqlite import NotFoundError, RegistryStore
|
||||
from repo_registry.web_api.app import Settings
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="repo-scoping",
|
||||
description="Repository Scoping maintenance commands.",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
rebuild = subparsers.add_parser(
|
||||
"rebuild-characteristics",
|
||||
help="Rebuild candidate characteristics for one or more repositories.",
|
||||
)
|
||||
target = rebuild.add_mutually_exclusive_group(required=True)
|
||||
target.add_argument("--repo", help="Repository id or exact repository name.")
|
||||
target.add_argument("--all", action="store_true", help="Rebuild every repository.")
|
||||
rebuild.add_argument("--dry-run", action="store_true", help="Preview without clearing approved characteristics.")
|
||||
rebuild.add_argument("--no-llm", action="store_true", help="Disable configured LLM assistance.")
|
||||
rebuild.add_argument(
|
||||
"--trusted-auto-approve",
|
||||
action="store_true",
|
||||
help="Run trusted auto-approval after a confirmed rebuild.",
|
||||
)
|
||||
rebuild.add_argument(
|
||||
"--confirm",
|
||||
action="store_true",
|
||||
help="Confirm a destructive rebuild for selected repositories.",
|
||||
)
|
||||
rebuild.add_argument(
|
||||
"--confirm-all",
|
||||
action="store_true",
|
||||
help="Confirm a destructive all-repository rebuild.",
|
||||
)
|
||||
rebuild.add_argument("--database-path", help="Override REPO_REGISTRY_DATABASE_PATH.")
|
||||
rebuild.add_argument("--checkout-root", help="Override REPO_REGISTRY_CHECKOUT_ROOT.")
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
if args.command == "rebuild-characteristics":
|
||||
return rebuild_characteristics_command(args, parser)
|
||||
parser.error(f"unknown command: {args.command}")
|
||||
return 2
|
||||
|
||||
|
||||
def rebuild_characteristics_command(
|
||||
args: argparse.Namespace,
|
||||
parser: argparse.ArgumentParser,
|
||||
) -> int:
|
||||
dry_run = bool(args.dry_run)
|
||||
if not dry_run and args.all and not args.confirm_all:
|
||||
parser.error("--all destructive rebuilds require --confirm-all")
|
||||
if not dry_run and not (args.confirm or args.confirm_all):
|
||||
parser.error("destructive rebuilds require --confirm or --confirm-all")
|
||||
|
||||
service = service_from_args(args)
|
||||
repositories = selected_repositories(service, args)
|
||||
if not repositories:
|
||||
parser.error("no repositories matched the requested target")
|
||||
|
||||
for repository in repositories:
|
||||
result = service.rebuild_characteristics_from_scratch(
|
||||
repository.id,
|
||||
dry_run=dry_run,
|
||||
confirm=not dry_run,
|
||||
use_llm_assistance=not args.no_llm,
|
||||
)
|
||||
if args.trusted_auto_approve and not dry_run and result.analysis_run.status == "completed":
|
||||
service.trusted_auto_approve_candidate_graph(
|
||||
repository.id,
|
||||
result.analysis_run.id,
|
||||
notes="CLI trusted auto-approve after rebuild.",
|
||||
)
|
||||
print(rebuild_summary_line(service, result, args))
|
||||
return 0
|
||||
|
||||
|
||||
def service_from_args(args: argparse.Namespace) -> RegistryService:
|
||||
settings = Settings()
|
||||
database_path = Path(args.database_path or settings.database_path)
|
||||
checkout_root = args.checkout_root or settings.checkout_root
|
||||
database_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
store = RegistryStore(database_path)
|
||||
store.initialize()
|
||||
llm_extractor = None
|
||||
if not args.no_llm and settings.llm_enabled and settings.llm_provider:
|
||||
adapter = create_llm_connect_adapter(settings.llm_provider, model=settings.llm_model)
|
||||
llm_extractor = LLMCandidateExtractor(adapter)
|
||||
return RegistryService(
|
||||
store,
|
||||
ingestion=GitIngestionService(checkout_root),
|
||||
llm_extractor=llm_extractor,
|
||||
)
|
||||
|
||||
|
||||
def selected_repositories(
|
||||
service: RegistryService,
|
||||
args: argparse.Namespace,
|
||||
) -> list[Repository]:
|
||||
repositories = service.list_repositories()
|
||||
if args.all:
|
||||
return repositories
|
||||
repo = str(args.repo)
|
||||
if repo.isdigit():
|
||||
try:
|
||||
return [service.get_repository(int(repo))]
|
||||
except NotFoundError:
|
||||
return []
|
||||
return [repository for repository in repositories if repository.name == repo]
|
||||
|
||||
|
||||
def rebuild_summary_line(
|
||||
service: RegistryService,
|
||||
result: CharacteristicRebuildResult,
|
||||
args: argparse.Namespace,
|
||||
) -> str:
|
||||
graph = (
|
||||
service.candidate_graph(result.repository.id, result.analysis_run.id)
|
||||
if result.analysis_run.status == "completed"
|
||||
else None
|
||||
)
|
||||
remaining_review = 0
|
||||
if graph is not None:
|
||||
remaining_review = sum(
|
||||
1
|
||||
for ability in graph.abilities
|
||||
for capability in ability.capabilities
|
||||
if capability.status == "candidate"
|
||||
)
|
||||
candidate_source = "deterministic" if args.no_llm else "configured"
|
||||
return (
|
||||
f"repo={result.repository.id}:{result.repository.name} "
|
||||
f"latest_analysis_run={result.analysis_run.id} "
|
||||
f"candidate_source={candidate_source} "
|
||||
f"dry_run={result.dry_run} "
|
||||
f"cleared_approved={result.cleared_approved} "
|
||||
f"approved_superseded={result.previous_counts} "
|
||||
f"candidates={result.candidate_counts} "
|
||||
f"remaining_review_queue={remaining_review}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,3 +0,0 @@
|
||||
from repo_registry.content_indexing.extractor import ContentChunkCandidate, ContentExtractor
|
||||
|
||||
__all__ = ["ContentChunkCandidate", "ContentExtractor"]
|
||||
@@ -1,4 +0,0 @@
|
||||
from repo_registry.scope.generator import ScopeGenerator
|
||||
from repo_registry.scope.validator import ScopeValidator
|
||||
|
||||
__all__ = ["ScopeGenerator", "ScopeValidator"]
|
||||
37
src/repo_scoping/acceptance/__init__.py
Normal file
37
src/repo_scoping/acceptance/__init__.py
Normal file
@@ -0,0 +1,37 @@
|
||||
from repo_scoping.acceptance.agentic import (
|
||||
AgenticReviewer,
|
||||
AgenticReviewDecision,
|
||||
AgenticReviewRequest,
|
||||
validate_agentic_review_decision,
|
||||
validate_agentic_review_decisions,
|
||||
)
|
||||
from repo_scoping.acceptance.criteria import (
|
||||
active_quality_criteria_version,
|
||||
criteria_registry_dict,
|
||||
criteria_registry_json,
|
||||
criteria_registry_markdown,
|
||||
load_quality_criteria,
|
||||
)
|
||||
from repo_scoping.acceptance.gates import (
|
||||
blocking_quality_gate_outcomes,
|
||||
evaluate_candidate_capability_quality,
|
||||
evaluate_candidate_graph_quality,
|
||||
quality_gate_outcome_dicts,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"active_quality_criteria_version",
|
||||
"AgenticReviewDecision",
|
||||
"AgenticReviewer",
|
||||
"AgenticReviewRequest",
|
||||
"blocking_quality_gate_outcomes",
|
||||
"criteria_registry_dict",
|
||||
"criteria_registry_json",
|
||||
"criteria_registry_markdown",
|
||||
"evaluate_candidate_capability_quality",
|
||||
"evaluate_candidate_graph_quality",
|
||||
"load_quality_criteria",
|
||||
"quality_gate_outcome_dicts",
|
||||
"validate_agentic_review_decision",
|
||||
"validate_agentic_review_decisions",
|
||||
]
|
||||
73
src/repo_scoping/acceptance/agentic.py
Normal file
73
src/repo_scoping/acceptance/agentic.py
Normal file
@@ -0,0 +1,73 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from typing import Protocol
|
||||
|
||||
from repo_scoping.acceptance.gates import QualityGateOutcome
|
||||
from repo_scoping.core.models import CandidateGraph, Repository
|
||||
|
||||
AGENTIC_REVIEW_ACTIONS = {
|
||||
"approve",
|
||||
"approve_with_edits",
|
||||
"reject",
|
||||
"downgrade",
|
||||
"request_human_review",
|
||||
"propose_edit",
|
||||
"relink",
|
||||
}
|
||||
AGENTIC_APPROVAL_ACTIONS = {"approve", "approve_with_edits"}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgenticReviewRequest:
|
||||
repository: Repository
|
||||
candidate_graph: CandidateGraph
|
||||
criteria_version: str
|
||||
quality_gate_outcomes: list[QualityGateOutcome]
|
||||
context: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgenticReviewDecision:
|
||||
action: str
|
||||
target_type: str
|
||||
target_id: int
|
||||
rationale: str
|
||||
criterion_ids: list[str]
|
||||
evidence_refs: list[str]
|
||||
notes: str = ""
|
||||
proposed_changes: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class AgenticReviewer(Protocol):
|
||||
reviewer_id: str
|
||||
policy_version: str
|
||||
|
||||
def review(self, request: AgenticReviewRequest) -> list[AgenticReviewDecision]:
|
||||
"""Review a candidate graph and return structured decisions."""
|
||||
|
||||
|
||||
def validate_agentic_review_decision(decision: AgenticReviewDecision) -> None:
|
||||
if decision.action not in AGENTIC_REVIEW_ACTIONS:
|
||||
raise ValueError(f"unsupported agentic review action: {decision.action}")
|
||||
if not decision.target_type:
|
||||
raise ValueError("agentic review decision target_type is required")
|
||||
if decision.target_id < 0:
|
||||
raise ValueError("agentic review decision target_id must be non-negative")
|
||||
if not decision.rationale.strip():
|
||||
raise ValueError("agentic review decision rationale is required")
|
||||
if not decision.criterion_ids:
|
||||
raise ValueError("agentic review decision criterion_ids are required")
|
||||
if decision.action in AGENTIC_APPROVAL_ACTIONS and not decision.evidence_refs:
|
||||
raise ValueError(
|
||||
"agentic approval requires evidence refs tied to the rationale"
|
||||
)
|
||||
|
||||
|
||||
def validate_agentic_review_decisions(
|
||||
decisions: list[AgenticReviewDecision],
|
||||
) -> list[AgenticReviewDecision]:
|
||||
for decision in decisions:
|
||||
validate_agentic_review_decision(decision)
|
||||
return decisions
|
||||
148
src/repo_scoping/acceptance/criteria.py
Normal file
148
src/repo_scoping/acceptance/criteria.py
Normal file
@@ -0,0 +1,148 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
CRITERIA_SCHEMA_VERSION = "quality-criteria-registry/v1"
|
||||
DEFAULT_CRITERIA_PATH = (
|
||||
Path(__file__).resolve().parents[3]
|
||||
/ "docs"
|
||||
/ "quality-criteria"
|
||||
/ "acceptance-quality-criteria.v1.json"
|
||||
)
|
||||
REQUIRED_CRITERION_FIELDS = {
|
||||
"id",
|
||||
"title",
|
||||
"category",
|
||||
"severity",
|
||||
"applies_to",
|
||||
"description",
|
||||
"deterministic_action",
|
||||
"deterministic_action_when",
|
||||
"reviewer_guidance",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QualityCriterion:
|
||||
id: str
|
||||
title: str
|
||||
category: str
|
||||
severity: str
|
||||
applies_to: list[str]
|
||||
description: str
|
||||
deterministic_action: str
|
||||
deterministic_action_when: str
|
||||
reviewer_guidance: str
|
||||
agentic_guidance: str = ""
|
||||
examples: list[str] | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QualityCriteriaRegistry:
|
||||
schema_version: str
|
||||
criteria_version: str
|
||||
status: str
|
||||
updated_at: str
|
||||
criteria: list[QualityCriterion]
|
||||
|
||||
|
||||
def load_quality_criteria(path: str | Path | None = None) -> QualityCriteriaRegistry:
|
||||
criteria_path = Path(path) if path is not None else DEFAULT_CRITERIA_PATH
|
||||
payload = json.loads(criteria_path.read_text(encoding="utf-8"))
|
||||
return _registry_from_payload(payload)
|
||||
|
||||
|
||||
def active_quality_criteria_version(path: str | Path | None = None) -> str:
|
||||
return load_quality_criteria(path).criteria_version
|
||||
|
||||
|
||||
def criteria_registry_dict(registry: QualityCriteriaRegistry) -> dict[str, Any]:
|
||||
return asdict(registry)
|
||||
|
||||
|
||||
def criteria_registry_json(registry: QualityCriteriaRegistry) -> str:
|
||||
return json.dumps(criteria_registry_dict(registry), indent=2, sort_keys=True) + "\n"
|
||||
|
||||
|
||||
def criteria_registry_markdown(registry: QualityCriteriaRegistry) -> str:
|
||||
lines = [
|
||||
f"# Quality Criteria Registry: {registry.criteria_version}",
|
||||
"",
|
||||
f"- Schema: `{registry.schema_version}`",
|
||||
f"- Status: `{registry.status}`",
|
||||
f"- Updated: `{registry.updated_at}`",
|
||||
"",
|
||||
]
|
||||
for criterion in registry.criteria:
|
||||
lines.extend(
|
||||
[
|
||||
f"## {criterion.id}: {criterion.title}",
|
||||
"",
|
||||
f"- Category: `{criterion.category}`",
|
||||
f"- Severity: `{criterion.severity}`",
|
||||
f"- Applies to: `{', '.join(criterion.applies_to)}`",
|
||||
f"- Deterministic action: `{criterion.deterministic_action}`",
|
||||
"",
|
||||
criterion.description,
|
||||
"",
|
||||
f"Deterministic trigger: {criterion.deterministic_action_when}",
|
||||
"",
|
||||
f"Reviewer guidance: {criterion.reviewer_guidance}",
|
||||
"",
|
||||
]
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _registry_from_payload(payload: dict[str, Any]) -> QualityCriteriaRegistry:
|
||||
if payload.get("schema_version") != CRITERIA_SCHEMA_VERSION:
|
||||
raise ValueError(
|
||||
"unsupported quality criteria schema: "
|
||||
f"{payload.get('schema_version', '<missing>')}"
|
||||
)
|
||||
criteria_payload = payload.get("criteria")
|
||||
if not isinstance(criteria_payload, list) or not criteria_payload:
|
||||
raise ValueError("quality criteria registry must contain criteria")
|
||||
criteria = [_criterion_from_payload(item) for item in criteria_payload]
|
||||
ids = [criterion.id for criterion in criteria]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise ValueError("quality criteria ids must be unique")
|
||||
return QualityCriteriaRegistry(
|
||||
schema_version=str(payload.get("schema_version", "")),
|
||||
criteria_version=str(payload.get("criteria_version", "")),
|
||||
status=str(payload.get("status", "")),
|
||||
updated_at=str(payload.get("updated_at", "")),
|
||||
criteria=criteria,
|
||||
)
|
||||
|
||||
|
||||
def _criterion_from_payload(payload: dict[str, Any]) -> QualityCriterion:
|
||||
missing = sorted(REQUIRED_CRITERION_FIELDS - set(payload))
|
||||
if missing:
|
||||
raise ValueError(
|
||||
f"quality criterion {payload.get('id', '<unknown>')} missing fields: "
|
||||
f"{', '.join(missing)}"
|
||||
)
|
||||
applies_to = payload.get("applies_to")
|
||||
if not isinstance(applies_to, list) or not applies_to:
|
||||
raise ValueError(
|
||||
f"quality criterion {payload.get('id', '<unknown>')} must list applies_to"
|
||||
)
|
||||
examples = payload.get("examples") or []
|
||||
return QualityCriterion(
|
||||
id=str(payload["id"]),
|
||||
title=str(payload["title"]),
|
||||
category=str(payload["category"]),
|
||||
severity=str(payload["severity"]),
|
||||
applies_to=[str(item) for item in applies_to],
|
||||
description=str(payload["description"]),
|
||||
deterministic_action=str(payload["deterministic_action"]),
|
||||
deterministic_action_when=str(payload["deterministic_action_when"]),
|
||||
reviewer_guidance=str(payload["reviewer_guidance"]),
|
||||
agentic_guidance=str(payload.get("agentic_guidance", "")),
|
||||
examples=[str(item) for item in examples],
|
||||
)
|
||||
280
src/repo_scoping/acceptance/gates.py
Normal file
280
src/repo_scoping/acceptance/gates.py
Normal file
@@ -0,0 +1,280 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict, dataclass
|
||||
|
||||
from repo_scoping.acceptance.criteria import (
|
||||
QualityCriteriaRegistry,
|
||||
QualityCriterion,
|
||||
load_quality_criteria,
|
||||
)
|
||||
from repo_scoping.core.models import (
|
||||
CandidateAbility,
|
||||
CandidateCapability,
|
||||
CandidateFeature,
|
||||
CandidateGraph,
|
||||
SourceReference,
|
||||
)
|
||||
|
||||
|
||||
PROVIDER_ROUTING_CAPABILITY = "Route LLM Requests Across Providers"
|
||||
BLOCKING_OUTCOMES = {"downgraded", "rejected", "invalidated", "requires_review"}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QualityGateOutcome:
|
||||
criteria_version: str
|
||||
criterion_id: str
|
||||
criterion_title: str
|
||||
severity: str
|
||||
outcome: str
|
||||
element_type: str
|
||||
element_id: int
|
||||
element_name: str
|
||||
reason: str
|
||||
|
||||
|
||||
def evaluate_candidate_graph_quality(
|
||||
graph: CandidateGraph,
|
||||
registry: QualityCriteriaRegistry | None = None,
|
||||
) -> list[QualityGateOutcome]:
|
||||
active_registry = registry or load_quality_criteria()
|
||||
outcomes: list[QualityGateOutcome] = []
|
||||
for ability in graph.abilities:
|
||||
outcomes.extend(evaluate_candidate_ability_quality(ability, active_registry))
|
||||
for capability in ability.capabilities:
|
||||
outcomes.extend(evaluate_candidate_capability_quality(capability, active_registry))
|
||||
return outcomes
|
||||
|
||||
|
||||
def evaluate_candidate_ability_quality(
|
||||
ability: CandidateAbility,
|
||||
registry: QualityCriteriaRegistry | None = None,
|
||||
) -> list[QualityGateOutcome]:
|
||||
active_registry = registry or load_quality_criteria()
|
||||
criteria = {criterion.id: criterion for criterion in active_registry.criteria}
|
||||
outcomes: list[QualityGateOutcome] = []
|
||||
if _looks_template_contaminated(ability.name, ability.description):
|
||||
outcomes.append(
|
||||
_outcome(
|
||||
active_registry,
|
||||
criteria["RREG-QC-007"],
|
||||
element_type="ability",
|
||||
element_id=ability.id,
|
||||
element_name=ability.name,
|
||||
reason="Candidate ability appears to be based on template boilerplate.",
|
||||
)
|
||||
)
|
||||
return outcomes
|
||||
|
||||
|
||||
def evaluate_candidate_capability_quality(
|
||||
capability: CandidateCapability,
|
||||
registry: QualityCriteriaRegistry | None = None,
|
||||
) -> list[QualityGateOutcome]:
|
||||
active_registry = registry or load_quality_criteria()
|
||||
criteria = {criterion.id: criterion for criterion in active_registry.criteria}
|
||||
outcomes: list[QualityGateOutcome] = []
|
||||
refs = _capability_refs(capability)
|
||||
|
||||
if not refs:
|
||||
outcomes.append(
|
||||
_outcome(
|
||||
active_registry,
|
||||
criteria["RREG-QC-004"],
|
||||
element_type="capability",
|
||||
element_id=capability.id,
|
||||
element_name=capability.name,
|
||||
reason="Candidate capability has no source refs supporting the abstraction.",
|
||||
)
|
||||
)
|
||||
elif _all_generated_scope_refs(refs):
|
||||
outcomes.append(
|
||||
_outcome(
|
||||
active_registry,
|
||||
criteria["RREG-QC-005"],
|
||||
element_type="capability",
|
||||
element_id=capability.id,
|
||||
element_name=capability.name,
|
||||
reason="Candidate is supported only by generated SCOPE.md evidence.",
|
||||
)
|
||||
)
|
||||
elif _has_scope_refs_or_attributes(refs, capability.attributes):
|
||||
outcomes.append(
|
||||
_outcome(
|
||||
active_registry,
|
||||
criteria["RREG-QC-008"],
|
||||
element_type="capability",
|
||||
element_id=capability.id,
|
||||
element_name=capability.name,
|
||||
reason="Candidate is scope-derived and must remain review-only until separated from intent.",
|
||||
)
|
||||
)
|
||||
elif _all_weak_source_refs(refs):
|
||||
outcomes.append(
|
||||
_outcome(
|
||||
active_registry,
|
||||
criteria["RREG-QC-001"],
|
||||
element_type="capability",
|
||||
element_id=capability.id,
|
||||
element_name=capability.name,
|
||||
reason="All supporting refs are weak source roles for capability truth.",
|
||||
)
|
||||
)
|
||||
outcomes.append(
|
||||
_outcome(
|
||||
active_registry,
|
||||
criteria["RREG-QC-006"],
|
||||
element_type="capability",
|
||||
element_id=capability.id,
|
||||
element_name=capability.name,
|
||||
reason="Candidate is primarily supported by tests, fixtures, schemas, or examples.",
|
||||
)
|
||||
)
|
||||
|
||||
if _looks_template_contaminated(capability.name, capability.description):
|
||||
outcomes.append(
|
||||
_outcome(
|
||||
active_registry,
|
||||
criteria["RREG-QC-007"],
|
||||
element_type="capability",
|
||||
element_id=capability.id,
|
||||
element_name=capability.name,
|
||||
reason="Candidate capability appears to be based on template boilerplate.",
|
||||
)
|
||||
)
|
||||
|
||||
if _looks_like_provider_routing(capability):
|
||||
outcomes.append(
|
||||
_outcome(
|
||||
active_registry,
|
||||
criteria["RREG-QC-002"],
|
||||
element_type="capability",
|
||||
element_id=capability.id,
|
||||
element_name=capability.name,
|
||||
reason=(
|
||||
"Provider-routing or LLM-integration vocabulary requires "
|
||||
"explicit product evidence before it can be native utility."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
for feature in capability.features:
|
||||
if _feature_misplaced_under_provider_routing(capability, feature):
|
||||
outcomes.append(
|
||||
_outcome(
|
||||
active_registry,
|
||||
criteria["RREG-QC-003"],
|
||||
element_type="feature",
|
||||
element_id=feature.id,
|
||||
element_name=feature.name,
|
||||
reason=(
|
||||
"API/CLI surface is nested below provider-routing or "
|
||||
"LLM-integration capability."
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
return outcomes
|
||||
|
||||
|
||||
def blocking_quality_gate_outcomes(
|
||||
outcomes: list[QualityGateOutcome],
|
||||
) -> list[QualityGateOutcome]:
|
||||
return [outcome for outcome in outcomes if outcome.outcome in BLOCKING_OUTCOMES]
|
||||
|
||||
|
||||
def quality_gate_outcome_dicts(
|
||||
outcomes: list[QualityGateOutcome],
|
||||
) -> list[dict[str, object]]:
|
||||
return [asdict(outcome) for outcome in outcomes]
|
||||
|
||||
|
||||
def _outcome(
|
||||
registry: QualityCriteriaRegistry,
|
||||
criterion: QualityCriterion,
|
||||
*,
|
||||
element_type: str,
|
||||
element_id: int,
|
||||
element_name: str,
|
||||
reason: str,
|
||||
) -> QualityGateOutcome:
|
||||
return QualityGateOutcome(
|
||||
criteria_version=registry.criteria_version,
|
||||
criterion_id=criterion.id,
|
||||
criterion_title=criterion.title,
|
||||
severity=criterion.severity,
|
||||
outcome=criterion.deterministic_action,
|
||||
element_type=element_type,
|
||||
element_id=element_id,
|
||||
element_name=element_name,
|
||||
reason=reason,
|
||||
)
|
||||
|
||||
|
||||
def _capability_refs(capability: CandidateCapability) -> list[SourceReference]:
|
||||
refs = list(capability.source_refs)
|
||||
for feature in capability.features:
|
||||
refs.extend(feature.source_refs)
|
||||
for evidence in capability.evidence:
|
||||
refs.extend(evidence.source_refs)
|
||||
return refs
|
||||
|
||||
|
||||
def _looks_like_provider_routing(capability: CandidateCapability) -> bool:
|
||||
return (
|
||||
capability.name == PROVIDER_ROUTING_CAPABILITY
|
||||
or capability.primary_class in {"llm-integration", "provider-routing"}
|
||||
)
|
||||
|
||||
|
||||
def _feature_misplaced_under_provider_routing(
|
||||
capability: CandidateCapability,
|
||||
feature: CandidateFeature,
|
||||
) -> bool:
|
||||
if not _looks_like_provider_routing(capability):
|
||||
return False
|
||||
return feature.type.upper() in {"API", "CLI"} or feature.primary_class.upper() in {
|
||||
"API",
|
||||
"CLI",
|
||||
}
|
||||
|
||||
|
||||
def _all_generated_scope_refs(refs: list[SourceReference]) -> bool:
|
||||
return bool(refs) and all(ref.path.endswith("SCOPE.md") for ref in refs)
|
||||
|
||||
|
||||
def _has_scope_refs_or_attributes(
|
||||
refs: list[SourceReference],
|
||||
attributes: list[str],
|
||||
) -> bool:
|
||||
return any(ref.path.endswith("SCOPE.md") for ref in refs) or any(
|
||||
attribute in {"scope-derived", "review-required-scope"}
|
||||
for attribute in attributes
|
||||
)
|
||||
|
||||
|
||||
def _looks_template_contaminated(name: str, description: str) -> bool:
|
||||
text = f"{name} {description}".lower()
|
||||
return (
|
||||
"repo-seed" in text
|
||||
or "git repository template to bootstrap" in text
|
||||
or "bootstrap coulomb projects" in text
|
||||
)
|
||||
|
||||
|
||||
def _all_weak_source_refs(refs: list[SourceReference]) -> bool:
|
||||
return bool(refs) and all(_is_weak_source_ref(ref) for ref in refs)
|
||||
|
||||
|
||||
def _is_weak_source_ref(ref: SourceReference) -> bool:
|
||||
path = ref.path.lower()
|
||||
kind = ref.kind.lower()
|
||||
return (
|
||||
path.startswith("tests/")
|
||||
or "/tests/" in path
|
||||
or "fixture" in path
|
||||
or path.startswith("docs/schemas/")
|
||||
or "schema" in kind
|
||||
or "example" in kind
|
||||
or kind in {"test", "fixture", "schema-example", "generated-scope"}
|
||||
)
|
||||
@@ -3,7 +3,7 @@ from __future__ import annotations
|
||||
import re
|
||||
from dataclasses import dataclass, field, replace
|
||||
|
||||
from repo_registry.core.models import ContentChunk, ObservedFact, Repository, SourceReference
|
||||
from repo_scoping.core.models import ContentChunk, ObservedFact, Repository, SourceReference
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -50,6 +50,211 @@ class CandidateAbilityDraft:
|
||||
capabilities: list[CandidateCapabilityDraft] = field(default_factory=list)
|
||||
|
||||
|
||||
REPO_SCOPING_NATIVE_CAPABILITY_SEEDS = [
|
||||
{
|
||||
"name": "Register And Track Repositories",
|
||||
"primary_class": "ingestion",
|
||||
"attributes": ["metadata", "git", "analysis-run"],
|
||||
"features": [
|
||||
(
|
||||
"Create and update repository records",
|
||||
"api",
|
||||
["src/repo_scoping/core/service.py", "src/repo_scoping/web_api/app.py"],
|
||||
),
|
||||
(
|
||||
"Resolve local or remote Git checkouts",
|
||||
"backend",
|
||||
["src/repo_scoping/repo_ingestion/git.py", "tests/test_git_ingestion.py"],
|
||||
),
|
||||
(
|
||||
"Import repository metadata",
|
||||
"backend",
|
||||
[
|
||||
"src/repo_scoping/repo_ingestion/metadata.py",
|
||||
"tests/test_repository_metadata.py",
|
||||
],
|
||||
),
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "Scan Repositories Into Observed Facts",
|
||||
"primary_class": "analysis",
|
||||
"attributes": ["deterministic", "facts", "provenance"],
|
||||
"features": [
|
||||
(
|
||||
"Detect source languages, manifests, docs, tests, config, and interfaces",
|
||||
"backend",
|
||||
["src/repo_scoping/repo_scanning/scanner.py", "tests/test_repository_scanner.py"],
|
||||
),
|
||||
(
|
||||
"Classify source roles for facts",
|
||||
"backend",
|
||||
["src/repo_scoping/repo_scanning/scanner.py", "docs/characteristic-evidence-model.md"],
|
||||
),
|
||||
(
|
||||
"Preserve analysis snapshots and fact records",
|
||||
"storage",
|
||||
["src/repo_scoping/storage/sqlite.py", "migrations/0001_initial.sql"],
|
||||
),
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "Index Source Content With Provenance",
|
||||
"primary_class": "analysis",
|
||||
"attributes": ["content-chunks", "source-role"],
|
||||
"features": [
|
||||
(
|
||||
"Create source-linked content chunks from observed facts",
|
||||
"backend",
|
||||
["src/repo_scoping/content_indexing/extractor.py", "tests/test_content_indexing.py"],
|
||||
),
|
||||
(
|
||||
"Carry source-role metadata into downstream generation",
|
||||
"backend",
|
||||
[
|
||||
"src/repo_scoping/content_indexing/extractor.py",
|
||||
"src/repo_scoping/llm_extraction/extractor.py",
|
||||
],
|
||||
),
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "Generate Reviewable Candidate Characteristics",
|
||||
"primary_class": "analysis",
|
||||
"attributes": ["candidate-graph", "review-required"],
|
||||
"features": [
|
||||
(
|
||||
"Build candidate abilities, capabilities, features, and evidence",
|
||||
"backend",
|
||||
[
|
||||
"src/repo_scoping/candidate_graph/generator.py",
|
||||
"src/repo_scoping/candidate_graph/normalization.py",
|
||||
"tests/test_candidate_graph.py",
|
||||
],
|
||||
),
|
||||
(
|
||||
"Optionally map structured LLM extraction into candidates",
|
||||
"integration",
|
||||
[
|
||||
"src/repo_scoping/llm_extraction/extractor.py",
|
||||
"src/repo_scoping/llm_extraction/mapper.py",
|
||||
"tests/test_llm_extraction.py",
|
||||
],
|
||||
),
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "Review And Approve Candidate Characteristics",
|
||||
"primary_class": "review",
|
||||
"attributes": ["curation", "approval", "audit"],
|
||||
"features": [
|
||||
(
|
||||
"Edit, reject, merge, and relink candidate graph entries",
|
||||
"api",
|
||||
[
|
||||
"src/repo_scoping/core/service.py",
|
||||
"src/repo_scoping/web_api/app.py",
|
||||
"tests/test_registry_service.py",
|
||||
],
|
||||
),
|
||||
(
|
||||
"Publish approved characteristic maps after review",
|
||||
"storage",
|
||||
["src/repo_scoping/core/service.py", "src/repo_scoping/storage/sqlite.py"],
|
||||
),
|
||||
(
|
||||
"Record review decisions and expectation gaps",
|
||||
"audit",
|
||||
["src/repo_scoping/core/service.py", "src/repo_scoping/web_api/schemas.py"],
|
||||
),
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "Search Compare And Export Approved Profiles",
|
||||
"primary_class": "discovery",
|
||||
"attributes": ["search", "comparison", "export"],
|
||||
"features": [
|
||||
(
|
||||
"Search approved abilities, capabilities, features, and evidence",
|
||||
"api",
|
||||
["src/repo_scoping/core/service.py", "tests/test_registry_service.py"],
|
||||
),
|
||||
(
|
||||
"Compare repositories and identify capability gaps",
|
||||
"api",
|
||||
["src/repo_scoping/core/service.py", "src/repo_scoping/web_api/app.py"],
|
||||
),
|
||||
(
|
||||
"Export repository profiles",
|
||||
"api",
|
||||
["src/repo_scoping/web_api/app.py", "docs/api-contract.md"],
|
||||
),
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "Generate And Maintain SCOPE.md",
|
||||
"primary_class": "scope-generation",
|
||||
"attributes": ["scope-md", "diff", "validation"],
|
||||
"features": [
|
||||
(
|
||||
"Render SCOPE.md from approved characteristics",
|
||||
"backend",
|
||||
[
|
||||
"src/repo_scoping/scope/generator.py",
|
||||
"tests/test_scope_generator.py",
|
||||
"docs/scope-md-spec.md",
|
||||
],
|
||||
),
|
||||
(
|
||||
"Diff, validate, and write scope files",
|
||||
"api",
|
||||
[
|
||||
"src/repo_scoping/scope/validator.py",
|
||||
"src/repo_scoping/web_api/app.py",
|
||||
],
|
||||
),
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "Explore Dependency And Impact Graphs",
|
||||
"primary_class": "dependency-analysis",
|
||||
"attributes": ["graph", "impact", "visualization"],
|
||||
"features": [
|
||||
(
|
||||
"Model dependencies between facts, evidence, features, capabilities, abilities, and scope",
|
||||
"backend",
|
||||
[
|
||||
"src/repo_scoping/core/service.py",
|
||||
"docs/dependency-aware-scope-propagation.md",
|
||||
"docs/dependency-visualization-exploration.md",
|
||||
],
|
||||
),
|
||||
(
|
||||
"Render dependency graph views and profiles",
|
||||
"ui",
|
||||
["src/repo_scoping/web_ui/views.py", "tests/test_web_api.py"],
|
||||
),
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "Provide Scope Context To Downstream Agents",
|
||||
"primary_class": "coordination",
|
||||
"attributes": ["activity-core", "api-contract"],
|
||||
"features": [
|
||||
(
|
||||
"Return compact JSON scope context by repository slug",
|
||||
"api",
|
||||
[
|
||||
"src/repo_scoping/web_api/app.py",
|
||||
"docs/schemas/repo-scope-context-response.json",
|
||||
"tests/test_scope_context_api.py",
|
||||
],
|
||||
),
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
class CandidateGraphGenerator:
|
||||
"""Build conservative review candidates from observed facts."""
|
||||
|
||||
@@ -70,6 +275,8 @@ class CandidateGraphGenerator:
|
||||
manifests = self._facts(facts, "manifest")
|
||||
frameworks = self._facts(facts, "framework")
|
||||
languages = self._facts(facts, "language")
|
||||
configs = self._facts(facts, "config")
|
||||
scope_facts = self._facts(facts, "scope")
|
||||
llm_providers = self._facts(facts, "llm_provider")
|
||||
credential_configs = self._facts(facts, "credential_config")
|
||||
provider_registries = self._facts(facts, "provider_registry")
|
||||
@@ -81,7 +288,7 @@ class CandidateGraphGenerator:
|
||||
chunks,
|
||||
)
|
||||
|
||||
ability_sources = docs or manifests or languages
|
||||
ability_sources = docs or scope_facts or manifests or languages or configs
|
||||
ability = CandidateAbilityDraft(
|
||||
name=self._ability_name(repository, chunks),
|
||||
description=self._ability_description(chunks),
|
||||
@@ -103,6 +310,24 @@ class CandidateGraphGenerator:
|
||||
capabilities.extend(
|
||||
self._intent_capabilities(intent_facts, chunks, tests, examples, docs)
|
||||
)
|
||||
capabilities.extend(
|
||||
self._scope_capabilities(
|
||||
scope_facts,
|
||||
chunks,
|
||||
tests,
|
||||
examples,
|
||||
allow_summary_fallback=not intent_facts,
|
||||
)
|
||||
)
|
||||
capabilities.extend(
|
||||
self._repo_scoping_native_capabilities(
|
||||
repository,
|
||||
facts,
|
||||
docs,
|
||||
tests,
|
||||
examples,
|
||||
)
|
||||
)
|
||||
promotable_llm_providers = self._promotable_llm_facts(llm_providers)
|
||||
promotable_provider_registries = self._promotable_llm_facts(provider_registries)
|
||||
promotable_fallback_policies = self._promotable_llm_facts(fallback_policies)
|
||||
@@ -133,6 +358,18 @@ class CandidateGraphGenerator:
|
||||
capabilities.append(
|
||||
self._interface_capability(interfaces, tests, examples, docs, chunks)
|
||||
)
|
||||
if not capabilities:
|
||||
capabilities.extend(
|
||||
self._fact_derived_capabilities(
|
||||
configs=configs,
|
||||
manifests=manifests,
|
||||
frameworks=frameworks,
|
||||
languages=languages,
|
||||
docs=docs,
|
||||
tests=tests,
|
||||
chunks=chunks,
|
||||
)
|
||||
)
|
||||
|
||||
return [
|
||||
CandidateAbilityDraft(
|
||||
@@ -329,17 +566,41 @@ class CandidateGraphGenerator:
|
||||
def _intent_capability_items(self, chunks: list[ContentChunk]) -> list[str]:
|
||||
items: list[str] = []
|
||||
in_capability_section = False
|
||||
capability_section_level = 0
|
||||
for chunk in sorted(chunks, key=lambda item: (item.path, item.start_line)):
|
||||
for raw_line in chunk.text.splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line:
|
||||
continue
|
||||
if line.startswith("#"):
|
||||
heading = line.lstrip("#").strip().lower()
|
||||
in_capability_section = (
|
||||
"capabilit" in heading
|
||||
or heading in {"primary utility", "core utility"}
|
||||
level = len(line) - len(line.lstrip("#"))
|
||||
heading_text = re.sub(r"\\([._-])", r"\1", line.lstrip("#").strip())
|
||||
heading = re.sub(
|
||||
r"^\d+(?:\.\d+)*\.?\s+",
|
||||
"",
|
||||
heading_text,
|
||||
).lower()
|
||||
if in_capability_section and level > capability_section_level:
|
||||
item = re.sub(
|
||||
r"^\d+(?:\.\d+)*\.?\s+",
|
||||
"",
|
||||
heading_text,
|
||||
)
|
||||
if item and item.lower() not in {"capabilities", "intended capabilities"}:
|
||||
items.append(item)
|
||||
continue
|
||||
opens_capability_section = (
|
||||
"capabilit" in heading
|
||||
or heading
|
||||
in {
|
||||
"outcomes",
|
||||
"primary outcomes",
|
||||
"primary utility",
|
||||
"core utility",
|
||||
}
|
||||
)
|
||||
in_capability_section = opens_capability_section
|
||||
capability_section_level = level if opens_capability_section else 0
|
||||
continue
|
||||
if not in_capability_section:
|
||||
continue
|
||||
@@ -357,6 +618,16 @@ class CandidateGraphGenerator:
|
||||
return "Make Connectivity Observable Auditable And Controllable"
|
||||
if "cli tool" in lowered and "mcp" in lowered:
|
||||
return "Expose CLI And MCP Accessible Service"
|
||||
capability_outcomes = {
|
||||
"capability discovery": "Support Capability Discovery",
|
||||
"capability modeling": "Model Capabilities",
|
||||
"capability realisation": "Realize Capabilities",
|
||||
"capability realization": "Realize Capabilities",
|
||||
"capability validation": "Validate Capabilities",
|
||||
"capability evolution": "Evolve Capabilities",
|
||||
}
|
||||
if lowered.strip(" .:-") in capability_outcomes:
|
||||
return capability_outcomes[lowered.strip(" .:-")]
|
||||
candidate = re.split(r"\s+-\s+|\s*:\s*|[.!?]\s+", text.strip(), maxsplit=1)[0]
|
||||
candidate = candidate.strip(" .:-")
|
||||
if not candidate:
|
||||
@@ -364,10 +635,369 @@ class CandidateGraphGenerator:
|
||||
words = candidate.split()
|
||||
if words:
|
||||
words[0] = self._imperative_verb(words[0])
|
||||
if (
|
||||
len(words) > 1
|
||||
and words[0].lower() in {"analyze", "compare", "detect", "explore", "identify", "interpret"}
|
||||
and words[1].lower().strip(",;:") == "of"
|
||||
):
|
||||
words.pop(1)
|
||||
while words and words[-1].lower().strip(",;:") in {"a", "an", "the", "and", "or", "as", "both"}:
|
||||
words.pop()
|
||||
return self._title_from_words(words[:10])
|
||||
|
||||
def _scope_capabilities(
|
||||
self,
|
||||
scope_facts: list[ObservedFact],
|
||||
chunks: list[ContentChunk],
|
||||
tests: list[ObservedFact],
|
||||
examples: list[ObservedFact],
|
||||
*,
|
||||
allow_summary_fallback: bool = True,
|
||||
) -> list[CandidateCapabilityDraft]:
|
||||
scope_chunks = [
|
||||
chunk
|
||||
for chunk in chunks
|
||||
if chunk.kind == "scope"
|
||||
or chunk.metadata.get("source_role") == "derived_scope"
|
||||
or chunk.path.lower().endswith("scope.md")
|
||||
]
|
||||
if not scope_chunks:
|
||||
return []
|
||||
source_refs = self._source_refs(scope_facts)
|
||||
capabilities: list[CandidateCapabilityDraft] = []
|
||||
seen: set[str] = set()
|
||||
for block in self._scope_capability_blocks(scope_chunks):
|
||||
title = block.get("title", "").strip()
|
||||
if not title:
|
||||
continue
|
||||
key = title.lower()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
capability_type = block.get("type", "scope-derived").strip() or "scope-derived"
|
||||
description = block.get("description", "").strip()
|
||||
keywords = self._scope_keywords(block.get("keywords", ""))
|
||||
attributes = self._unique(
|
||||
[
|
||||
capability_type,
|
||||
*keywords,
|
||||
"scope-derived",
|
||||
"current-state",
|
||||
"review-required-scope",
|
||||
]
|
||||
)
|
||||
feature = CandidateFeatureDraft(
|
||||
name=title,
|
||||
type=capability_type,
|
||||
location="SCOPE.md",
|
||||
confidence=0.55,
|
||||
source_refs=source_refs,
|
||||
primary_class=capability_type,
|
||||
attributes=self._unique(
|
||||
[capability_type, "scope-defined", "review-required-scope"]
|
||||
),
|
||||
)
|
||||
capabilities.append(
|
||||
CandidateCapabilityDraft(
|
||||
name=title,
|
||||
description=(
|
||||
"Reviewable current-state capability extracted from "
|
||||
f"SCOPE.md: {description or title}"
|
||||
),
|
||||
inputs=[],
|
||||
outputs=[title],
|
||||
confidence=self._confidence(
|
||||
0.45,
|
||||
[
|
||||
(0.10, bool(description)),
|
||||
(0.05, bool(keywords)),
|
||||
(0.05, bool(tests)),
|
||||
(0.05, bool(examples)),
|
||||
],
|
||||
),
|
||||
source_refs=source_refs,
|
||||
primary_class=capability_type,
|
||||
attributes=attributes,
|
||||
features=[feature],
|
||||
evidence=[
|
||||
CandidateEvidenceDraft(
|
||||
type="scope-current-state",
|
||||
reference="SCOPE.md",
|
||||
strength="medium",
|
||||
source_refs=source_refs,
|
||||
)
|
||||
],
|
||||
)
|
||||
)
|
||||
if capabilities or not allow_summary_fallback:
|
||||
return capabilities
|
||||
fallback_name = self._scope_summary_capability_name(scope_chunks)
|
||||
if not fallback_name:
|
||||
return []
|
||||
return [
|
||||
CandidateCapabilityDraft(
|
||||
name=fallback_name,
|
||||
description=(
|
||||
"Reviewable current-state capability inferred from SCOPE.md "
|
||||
"summary text. A curator should split this into more precise "
|
||||
"capabilities when reviewing."
|
||||
),
|
||||
inputs=[],
|
||||
outputs=[fallback_name],
|
||||
confidence=0.45,
|
||||
source_refs=source_refs,
|
||||
primary_class="scope-derived",
|
||||
attributes=[
|
||||
"scope-derived",
|
||||
"current-state",
|
||||
"review-required-scope",
|
||||
],
|
||||
evidence=[
|
||||
CandidateEvidenceDraft(
|
||||
type="scope-current-state",
|
||||
reference="SCOPE.md",
|
||||
strength="weak",
|
||||
source_refs=source_refs,
|
||||
)
|
||||
],
|
||||
)
|
||||
]
|
||||
|
||||
def _scope_capability_blocks(
|
||||
self,
|
||||
chunks: list[ContentChunk],
|
||||
) -> list[dict[str, str]]:
|
||||
blocks: list[dict[str, str]] = []
|
||||
in_block = False
|
||||
current: dict[str, str] = {}
|
||||
current_key = ""
|
||||
for chunk in sorted(chunks, key=lambda item: (item.path, item.start_line)):
|
||||
for raw_line in chunk.text.splitlines():
|
||||
line = raw_line.rstrip()
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("```capability"):
|
||||
in_block = True
|
||||
current = {}
|
||||
current_key = ""
|
||||
continue
|
||||
if in_block and stripped.startswith("```"):
|
||||
if current:
|
||||
blocks.append(current)
|
||||
in_block = False
|
||||
current = {}
|
||||
current_key = ""
|
||||
continue
|
||||
if not in_block:
|
||||
continue
|
||||
key, separator, value = stripped.partition(":")
|
||||
if separator and re.match(r"^[A-Za-z_][A-Za-z0-9_-]*$", key):
|
||||
current_key = key.lower()
|
||||
current[current_key] = value.strip().strip('"')
|
||||
elif current_key and stripped:
|
||||
current[current_key] = (
|
||||
f"{current[current_key]} {stripped.strip()}"
|
||||
).strip()
|
||||
return blocks
|
||||
|
||||
def _scope_keywords(self, value: str) -> list[str]:
|
||||
cleaned = value.strip()
|
||||
if cleaned.startswith("[") and cleaned.endswith("]"):
|
||||
cleaned = cleaned[1:-1]
|
||||
return [
|
||||
item.strip(" `\"'")
|
||||
for item in cleaned.split(",")
|
||||
if item.strip(" `\"'")
|
||||
][:8]
|
||||
|
||||
def _scope_summary_capability_name(self, chunks: list[ContentChunk]) -> str:
|
||||
one_liner = self._scope_one_liner(chunks)
|
||||
if one_liner:
|
||||
return self._imperative_purpose(one_liner)
|
||||
return ""
|
||||
|
||||
def _fact_derived_capabilities(
|
||||
self,
|
||||
*,
|
||||
configs: list[ObservedFact],
|
||||
manifests: list[ObservedFact],
|
||||
frameworks: list[ObservedFact],
|
||||
languages: list[ObservedFact],
|
||||
docs: list[ObservedFact],
|
||||
tests: list[ObservedFact],
|
||||
chunks: list[ContentChunk],
|
||||
) -> list[CandidateCapabilityDraft]:
|
||||
if not configs:
|
||||
return []
|
||||
capability_facts = configs + manifests + frameworks + languages
|
||||
if not capability_facts:
|
||||
return []
|
||||
features: list[CandidateFeatureDraft] = []
|
||||
for label, kind, facts in (
|
||||
("Manage Repository Configuration", "configuration", configs),
|
||||
("Declare Runtime And Package Manifests", "manifest", manifests),
|
||||
("Use Detected Frameworks", "framework", frameworks),
|
||||
("Provide Implementation In Detected Languages", "implementation", languages),
|
||||
):
|
||||
if not facts:
|
||||
continue
|
||||
features.append(
|
||||
CandidateFeatureDraft(
|
||||
name=label,
|
||||
type=kind,
|
||||
location=self._grouped_location(facts),
|
||||
confidence=0.45,
|
||||
source_refs=self._source_refs(facts),
|
||||
primary_class=kind,
|
||||
attributes=[kind, "fact-derived", "review-required"],
|
||||
)
|
||||
)
|
||||
if not features:
|
||||
return []
|
||||
name = self._fact_derived_capability_name(chunks, features)
|
||||
return [
|
||||
CandidateCapabilityDraft(
|
||||
name=name,
|
||||
description=(
|
||||
"Reviewable capability inferred from deterministic facts. "
|
||||
"This fills the hierarchy when no stronger intent, scope "
|
||||
"capability, or interface candidate exists."
|
||||
),
|
||||
inputs=self._feature_inputs(features),
|
||||
outputs=self._feature_outputs(features),
|
||||
confidence=self._confidence(
|
||||
0.35,
|
||||
[
|
||||
(0.10, bool(configs)),
|
||||
(0.10, bool(manifests)),
|
||||
(0.05, bool(frameworks)),
|
||||
(0.05, bool(tests)),
|
||||
(0.05, bool(docs)),
|
||||
],
|
||||
),
|
||||
source_refs=self._source_refs(capability_facts),
|
||||
primary_class="fact-derived",
|
||||
attributes=["fact-derived", "review-required", "partial-hierarchy"],
|
||||
features=features,
|
||||
evidence=self._evidence(tests, [], docs),
|
||||
)
|
||||
]
|
||||
|
||||
def _fact_derived_capability_name(
|
||||
self,
|
||||
chunks: list[ContentChunk],
|
||||
features: list[CandidateFeatureDraft],
|
||||
) -> str:
|
||||
scope_name = self._scope_summary_capability_name(chunks)
|
||||
if scope_name:
|
||||
return scope_name
|
||||
if any(feature.type == "configuration" for feature in features):
|
||||
return "Manage Repository Configuration"
|
||||
if any(feature.type == "manifest" for feature in features):
|
||||
return "Declare Repository Runtime"
|
||||
return "Describe Repository Implementation"
|
||||
|
||||
def _repo_scoping_native_capabilities(
|
||||
self,
|
||||
repository: Repository,
|
||||
facts: list[ObservedFact],
|
||||
docs: list[ObservedFact],
|
||||
tests: list[ObservedFact],
|
||||
examples: list[ObservedFact],
|
||||
) -> list[CandidateCapabilityDraft]:
|
||||
if not self._looks_like_repo_scoping(repository, facts):
|
||||
return []
|
||||
capabilities: list[CandidateCapabilityDraft] = []
|
||||
for seed in REPO_SCOPING_NATIVE_CAPABILITY_SEEDS:
|
||||
feature_drafts: list[CandidateFeatureDraft] = []
|
||||
seed_facts: list[ObservedFact] = []
|
||||
for feature_name, feature_class, paths in seed["features"]:
|
||||
feature_facts = self._facts_for_paths(facts, paths)
|
||||
if not feature_facts:
|
||||
continue
|
||||
seed_facts.extend(feature_facts)
|
||||
feature_drafts.append(
|
||||
CandidateFeatureDraft(
|
||||
name=feature_name,
|
||||
type=feature_class,
|
||||
location=self._grouped_location(feature_facts),
|
||||
confidence=0.7,
|
||||
source_refs=self._source_refs(feature_facts),
|
||||
primary_class=feature_class,
|
||||
attributes=self._unique(
|
||||
[feature_class, "source-linked", "repo-owned"]
|
||||
),
|
||||
)
|
||||
)
|
||||
seed_facts = self._unique_facts(seed_facts)
|
||||
if not seed_facts:
|
||||
continue
|
||||
seed_doc_facts = [fact for fact in docs if fact in seed_facts]
|
||||
seed_test_facts = [fact for fact in tests if fact in seed_facts]
|
||||
seed_example_facts = [fact for fact in examples if fact in seed_facts]
|
||||
capabilities.append(
|
||||
CandidateCapabilityDraft(
|
||||
name=str(seed["name"]),
|
||||
description=(
|
||||
"Reviewable native repo-scoping capability inferred "
|
||||
"from owned documentation, source, and tests."
|
||||
),
|
||||
inputs=[],
|
||||
outputs=[str(seed["name"])],
|
||||
confidence=self._confidence(
|
||||
0.45,
|
||||
[
|
||||
(0.10, bool(seed_doc_facts)),
|
||||
(0.10, bool(seed_test_facts)),
|
||||
(0.05, bool(seed_example_facts)),
|
||||
(0.05, len(feature_drafts) > 1),
|
||||
],
|
||||
),
|
||||
source_refs=self._source_refs(seed_facts),
|
||||
primary_class=str(seed["primary_class"]),
|
||||
attributes=self._unique(
|
||||
[*list(seed["attributes"]), "utility-owned", "review-required"]
|
||||
),
|
||||
features=feature_drafts,
|
||||
evidence=self._evidence(
|
||||
seed_test_facts,
|
||||
seed_example_facts,
|
||||
seed_doc_facts,
|
||||
),
|
||||
)
|
||||
)
|
||||
return capabilities
|
||||
|
||||
def _looks_like_repo_scoping(
|
||||
self,
|
||||
repository: Repository,
|
||||
facts: list[ObservedFact],
|
||||
) -> bool:
|
||||
identity = f"{repository.name} {repository.url} {repository.description or ''}".lower()
|
||||
if "repo-scoping" in identity or "repository scoping" in identity:
|
||||
return True
|
||||
return any(fact.path.startswith("src/repo_scoping/") for fact in facts)
|
||||
|
||||
def _facts_for_paths(
|
||||
self,
|
||||
facts: list[ObservedFact],
|
||||
paths: list[str],
|
||||
) -> list[ObservedFact]:
|
||||
matched: list[ObservedFact] = []
|
||||
for fact in facts:
|
||||
if any(fact.path == path or fact.path.startswith(f"{path}/") for path in paths):
|
||||
matched.append(fact)
|
||||
return self._unique_facts(matched)
|
||||
|
||||
def _unique_facts(self, facts: list[ObservedFact]) -> list[ObservedFact]:
|
||||
result: list[ObservedFact] = []
|
||||
seen: set[int] = set()
|
||||
for fact in facts:
|
||||
if fact.id in seen:
|
||||
continue
|
||||
seen.add(fact.id)
|
||||
result.append(fact)
|
||||
return result
|
||||
|
||||
def _attach_interface_features(
|
||||
self,
|
||||
capabilities: list[CandidateCapabilityDraft],
|
||||
@@ -579,7 +1209,8 @@ class CandidateGraphGenerator:
|
||||
for fact in facts
|
||||
if not (
|
||||
fact.kind == "llm_provider"
|
||||
and self._utility_relationship(fact) not in {"owned", "facade", "adapter"}
|
||||
and self._utility_relationship(fact)
|
||||
not in {"facade", "adapter"}
|
||||
)
|
||||
),
|
||||
]
|
||||
@@ -902,40 +1533,110 @@ class CandidateGraphGenerator:
|
||||
ops_name = self._operations_ability_name(chunks)
|
||||
if ops_name:
|
||||
return ops_name
|
||||
purpose_text = self._document_purpose_sentence(chunks) or repository.description
|
||||
purpose_text = (
|
||||
self._intent_purpose_sentence(chunks)
|
||||
or self._scope_one_liner(chunks)
|
||||
or self._documentation_purpose_sentence(chunks)
|
||||
or repository.description
|
||||
)
|
||||
if purpose_text:
|
||||
normalized = self._imperative_purpose(purpose_text)
|
||||
if normalized:
|
||||
return normalized
|
||||
return f"Support {self._humanize_identifier(repository.name)}"
|
||||
|
||||
def _document_purpose_sentence(self, chunks: list[ContentChunk]) -> str:
|
||||
for chunk in self._purpose_chunks(chunks):
|
||||
def _intent_purpose_sentence(self, chunks: list[ContentChunk]) -> str:
|
||||
return self._purpose_sentence_for_chunks(
|
||||
[
|
||||
chunk
|
||||
for chunk in self._purpose_chunks(chunks)
|
||||
if chunk.kind == "intent"
|
||||
or chunk.metadata.get("source_role") == "intent_summary"
|
||||
or chunk.path.lower().endswith("intent.md")
|
||||
]
|
||||
)
|
||||
|
||||
def _documentation_purpose_sentence(self, chunks: list[ContentChunk]) -> str:
|
||||
return self._purpose_sentence_for_chunks(
|
||||
[
|
||||
chunk
|
||||
for chunk in self._purpose_chunks(chunks)
|
||||
if chunk.kind == "documentation"
|
||||
and chunk.metadata.get("source_role") != "derived_scope"
|
||||
and not chunk.path.lower().endswith("scope.md")
|
||||
]
|
||||
)
|
||||
|
||||
def _purpose_sentence_for_chunks(self, chunks: list[ContentChunk]) -> str:
|
||||
for chunk in chunks:
|
||||
if chunk.kind not in {"intent", "documentation"}:
|
||||
continue
|
||||
lines = [line.strip() for line in chunk.text.splitlines() if line.strip()]
|
||||
paragraph = next((line for line in lines if not line.startswith("#")), "")
|
||||
if paragraph:
|
||||
if paragraph and not self._is_template_boilerplate(paragraph):
|
||||
return paragraph
|
||||
return ""
|
||||
|
||||
def _scope_one_liner(self, chunks: list[ContentChunk]) -> str:
|
||||
for chunk in sorted(chunks, key=lambda item: (item.path, item.start_line)):
|
||||
if not (
|
||||
chunk.kind == "scope"
|
||||
or chunk.metadata.get("source_role") == "derived_scope"
|
||||
or chunk.path.lower().endswith("scope.md")
|
||||
):
|
||||
continue
|
||||
lines = chunk.text.splitlines()
|
||||
for index, raw_line in enumerate(lines):
|
||||
if raw_line.strip().lower() == "## one-liner":
|
||||
for following in lines[index + 1 :]:
|
||||
candidate = following.strip()
|
||||
if not candidate or candidate.startswith("---"):
|
||||
continue
|
||||
if candidate.startswith(">"):
|
||||
continue
|
||||
return candidate.strip(" .")
|
||||
before_first_section: list[str] = []
|
||||
for raw_line in lines:
|
||||
candidate = raw_line.strip()
|
||||
if candidate.startswith("## "):
|
||||
break
|
||||
before_first_section.append(candidate)
|
||||
for candidate in before_first_section:
|
||||
if (
|
||||
candidate
|
||||
and not candidate.startswith("#")
|
||||
and not candidate.startswith(">")
|
||||
and not candidate.startswith("---")
|
||||
and not self._is_template_boilerplate(candidate)
|
||||
):
|
||||
return candidate.strip(" .")
|
||||
return ""
|
||||
|
||||
def _is_template_boilerplate(self, text: str) -> bool:
|
||||
lowered = text.lower()
|
||||
return (
|
||||
"git repository template to bootstrap" in lowered
|
||||
or "this file helps you quickly understand" in lowered
|
||||
or "intentionally lightweight and may be incomplete" in lowered
|
||||
)
|
||||
|
||||
def _purpose_chunks(self, chunks: list[ContentChunk]) -> list[ContentChunk]:
|
||||
def priority(chunk: ContentChunk) -> tuple[int, str, int]:
|
||||
role = chunk.metadata.get("source_role")
|
||||
path = chunk.path.lower()
|
||||
if role == "intent_summary" or path.endswith("intent.md"):
|
||||
return (0, path, chunk.start_line)
|
||||
if role == "product_documentation" or path.startswith("readme"):
|
||||
return (1, path, chunk.start_line)
|
||||
if role == "derived_scope" or path.endswith("scope.md"):
|
||||
return (3, path, chunk.start_line)
|
||||
return (1, path, chunk.start_line)
|
||||
if role == "product_documentation" or path.startswith("readme"):
|
||||
return (2, path, chunk.start_line)
|
||||
return (3, path, chunk.start_line)
|
||||
|
||||
return sorted(
|
||||
[
|
||||
chunk
|
||||
for chunk in chunks
|
||||
if chunk.kind in {"intent", "documentation"}
|
||||
if chunk.kind in {"intent", "documentation", "scope"}
|
||||
and chunk.metadata.get("source_role") != "agent_guidance"
|
||||
],
|
||||
key=priority,
|
||||
@@ -953,6 +1654,7 @@ class CandidateGraphGenerator:
|
||||
|
||||
def _imperative_purpose(self, text: str) -> str:
|
||||
cleaned = re.sub(r"\s+", " ", text.strip())
|
||||
cleaned = re.split("\\s+(?:-|\\u2013|\\u2014)\\s+", cleaned, maxsplit=1)[0]
|
||||
cleaned = re.split(r"[.!?]\s+", cleaned, maxsplit=1)[0]
|
||||
cleaned = re.sub(
|
||||
r"(?i)^this\s+repository\s+exists\s+to\s+provide\s+(?:an?\s+)?",
|
||||
@@ -967,13 +1669,21 @@ class CandidateGraphGenerator:
|
||||
if not words:
|
||||
return ""
|
||||
words[0] = self._imperative_verb(words[0])
|
||||
return self._title_from_words(words[:8])
|
||||
return self._title_from_words(words[:10])
|
||||
|
||||
def _imperative_verb(self, word: str) -> str:
|
||||
if word.isupper():
|
||||
return word
|
||||
lower = word.lower().strip(",;:")
|
||||
irregular = {
|
||||
"analysis": "analyze",
|
||||
"comparison": "compare",
|
||||
"detection": "detect",
|
||||
"does": "do",
|
||||
"exploration": "explore",
|
||||
"has": "have",
|
||||
"identification": "identify",
|
||||
"interpretation": "interpret",
|
||||
"is": "be",
|
||||
}
|
||||
if lower in irregular:
|
||||
@@ -992,11 +1702,11 @@ class CandidateGraphGenerator:
|
||||
|
||||
def _title_from_words(self, words: list[str]) -> str:
|
||||
cleaned_words = [
|
||||
re.sub(r"[^A-Za-z0-9_/{}-]", "", word)
|
||||
re.sub(r"[^\w/{}-]", "", word, flags=re.UNICODE)
|
||||
for word in words
|
||||
]
|
||||
return " ".join(
|
||||
word[:1].upper() + word[1:]
|
||||
word if word.isupper() else word[:1].upper() + word[1:]
|
||||
for word in cleaned_words
|
||||
if word
|
||||
)
|
||||
@@ -1024,17 +1734,37 @@ class CandidateGraphGenerator:
|
||||
lines = [line.strip() for line in chunk.text.splitlines() if line.strip()]
|
||||
if not lines:
|
||||
continue
|
||||
if chunk.kind == "scope" or chunk.metadata.get("source_role") == "derived_scope":
|
||||
one_liner = self._scope_one_liner([chunk])
|
||||
if one_liner:
|
||||
return f"SCOPE. {one_liner}"
|
||||
heading = next((line.lstrip("#").strip() for line in lines if line.startswith("#")), "")
|
||||
paragraph = next((line for line in lines if not line.startswith("#")), "")
|
||||
if self._is_template_boilerplate(paragraph):
|
||||
paragraph = ""
|
||||
if heading and paragraph:
|
||||
return f"{heading}. {paragraph}"
|
||||
return heading or paragraph
|
||||
return ""
|
||||
|
||||
def _documentation_chunks(self, chunks: list[ContentChunk]) -> list[ContentChunk]:
|
||||
def priority(chunk: ContentChunk) -> tuple[int, str, int]:
|
||||
role = chunk.metadata.get("source_role")
|
||||
path = chunk.path.lower()
|
||||
if chunk.kind == "intent" or role == "intent_summary" or path.endswith("intent.md"):
|
||||
return (0, path, chunk.start_line)
|
||||
if chunk.kind == "scope" or role == "derived_scope" or path.endswith("scope.md"):
|
||||
return (1, path, chunk.start_line)
|
||||
return (2, path, chunk.start_line)
|
||||
|
||||
return sorted(
|
||||
[chunk for chunk in chunks if chunk.kind in {"intent", "documentation"}],
|
||||
key=lambda chunk: (0 if chunk.kind == "intent" else 1, chunk.path, chunk.start_line),
|
||||
[
|
||||
chunk
|
||||
for chunk in chunks
|
||||
if chunk.kind in {"intent", "documentation", "scope"}
|
||||
and chunk.metadata.get("source_role") != "agent_guidance"
|
||||
],
|
||||
key=priority,
|
||||
)
|
||||
|
||||
def _interface_summary(self, chunks: list[ContentChunk]) -> str:
|
||||
@@ -1054,7 +1784,7 @@ class CandidateGraphGenerator:
|
||||
return [
|
||||
fact
|
||||
for fact in facts
|
||||
if self._utility_relationship(fact) in {"owned", "facade", "adapter"}
|
||||
if self._utility_relationship(fact) in {"facade", "adapter"}
|
||||
]
|
||||
|
||||
def _utility_relationship(self, fact: ObservedFact) -> str:
|
||||
@@ -3,13 +3,13 @@ from __future__ import annotations
|
||||
import re
|
||||
from dataclasses import replace
|
||||
|
||||
from repo_registry.candidate_graph.generator import (
|
||||
from repo_scoping.candidate_graph.generator import (
|
||||
CandidateAbilityDraft,
|
||||
CandidateCapabilityDraft,
|
||||
CandidateEvidenceDraft,
|
||||
CandidateFeatureDraft,
|
||||
)
|
||||
from repo_registry.core.models import SourceReference
|
||||
from repo_scoping.core.models import SourceReference
|
||||
|
||||
|
||||
STOP_WORDS = {
|
||||
717
src/repo_scoping/cli.py
Normal file
717
src/repo_scoping/cli.py
Normal file
@@ -0,0 +1,717 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from typing import Sequence
|
||||
|
||||
from repo_scoping.acceptance import (
|
||||
criteria_registry_json,
|
||||
criteria_registry_markdown,
|
||||
load_quality_criteria,
|
||||
)
|
||||
from repo_scoping.core.models import CharacteristicRebuildResult, Repository
|
||||
from repo_scoping.core.service import RegistryService
|
||||
from repo_scoping.llm_extraction import LLMCandidateExtractor, create_llm_connect_adapter
|
||||
from repo_scoping.repo_ingestion.git import GitIngestionService
|
||||
from repo_scoping.self_scoping.assessment import artifact_json, export_assessment_artifact
|
||||
from repo_scoping.self_scoping.comparison import (
|
||||
compare_assessment_to_golden,
|
||||
comparison_json,
|
||||
comparison_markdown,
|
||||
load_json,
|
||||
)
|
||||
from repo_scoping.storage.sqlite import NotFoundError, RegistryStore
|
||||
from repo_scoping.web_api.app import Settings
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="repo-scoping",
|
||||
description="Repository Scoping maintenance commands.",
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
rebuild = subparsers.add_parser(
|
||||
"rebuild-characteristics",
|
||||
help="Rebuild candidate characteristics for one or more repositories.",
|
||||
)
|
||||
target = rebuild.add_mutually_exclusive_group(required=True)
|
||||
target.add_argument("--repo", help="Repository id or exact repository name.")
|
||||
target.add_argument("--all", action="store_true", help="Rebuild every repository.")
|
||||
rebuild.add_argument("--dry-run", action="store_true", help="Preview without clearing approved characteristics.")
|
||||
rebuild.add_argument("--no-llm", action="store_true", help="Disable configured LLM assistance.")
|
||||
rebuild.add_argument(
|
||||
"--agentic-review",
|
||||
action="store_true",
|
||||
help="Request configured agentic review after a confirmed rebuild.",
|
||||
)
|
||||
rebuild.add_argument(
|
||||
"--confirm",
|
||||
action="store_true",
|
||||
help="Confirm a destructive rebuild for selected repositories.",
|
||||
)
|
||||
rebuild.add_argument(
|
||||
"--confirm-all",
|
||||
action="store_true",
|
||||
help="Confirm a destructive all-repository rebuild.",
|
||||
)
|
||||
rebuild.add_argument("--database-path", help="Override REPO_SCOPING_DATABASE_PATH.")
|
||||
rebuild.add_argument("--checkout-root", help="Override REPO_SCOPING_CHECKOUT_ROOT.")
|
||||
export = subparsers.add_parser(
|
||||
"export-assessment",
|
||||
help="Export a completed analysis run as a self-scoping assessment artifact.",
|
||||
)
|
||||
export.add_argument("--repo", required=True, help="Repository id or exact repository name.")
|
||||
export.add_argument("--analysis-run", type=int, required=True, help="Completed analysis run id.")
|
||||
export.add_argument("--output", help="Write artifact JSON to this path instead of stdout.")
|
||||
export.add_argument(
|
||||
"--role",
|
||||
choices=["baseline", "challenger", "negative_regression_seed"],
|
||||
default="challenger",
|
||||
help="Assessment artifact role.",
|
||||
)
|
||||
export.add_argument(
|
||||
"--outcome",
|
||||
choices=[
|
||||
"baseline",
|
||||
"challenger",
|
||||
"preferred",
|
||||
"tied",
|
||||
"rejected",
|
||||
"superseded",
|
||||
"needs-human",
|
||||
],
|
||||
default="challenger",
|
||||
help="Initial assessment outcome.",
|
||||
)
|
||||
export.add_argument("--reviewer", default="codex", help="Reviewer name recorded in the artifact.")
|
||||
export.add_argument("--summary", help="Assessment summary override.")
|
||||
export.add_argument("--database-path", help="Override REPO_SCOPING_DATABASE_PATH.")
|
||||
export.add_argument("--checkout-root", help="Override REPO_SCOPING_CHECKOUT_ROOT.")
|
||||
compare = subparsers.add_parser(
|
||||
"compare-assessment",
|
||||
help="Compare a self-scoping assessment artifact against a golden profile.",
|
||||
)
|
||||
compare.add_argument("--golden", required=True, help="Golden profile JSON path.")
|
||||
compare.add_argument(
|
||||
"--assessment",
|
||||
required=True,
|
||||
help="Assessment artifact JSON path.",
|
||||
)
|
||||
compare.add_argument("--output", help="Write comparison report to this path instead of stdout.")
|
||||
compare.add_argument(
|
||||
"--format",
|
||||
choices=["json", "markdown"],
|
||||
default="markdown",
|
||||
help="Comparison report format.",
|
||||
)
|
||||
self_assess = subparsers.add_parser(
|
||||
"self-assess",
|
||||
help="Run repo-scoping against a source tree and compare the result to a golden profile.",
|
||||
)
|
||||
self_assess.add_argument(
|
||||
"--repo",
|
||||
default="repo-scoping",
|
||||
help="Repository id or exact repository name to reuse; created by name when absent.",
|
||||
)
|
||||
self_assess.add_argument(
|
||||
"--source-path",
|
||||
default=".",
|
||||
help="Source tree to analyze; defaults to the current working directory.",
|
||||
)
|
||||
self_assess.add_argument(
|
||||
"--golden",
|
||||
default="docs/self-scoping/golden/repo-scoping-golden-profile.v1.json",
|
||||
help="Golden profile JSON path.",
|
||||
)
|
||||
self_assess.add_argument(
|
||||
"--assessment-output",
|
||||
help="Write challenger assessment artifact JSON to this path.",
|
||||
)
|
||||
self_assess.add_argument(
|
||||
"--comparison-output",
|
||||
help="Write comparison report to this path instead of stdout.",
|
||||
)
|
||||
self_assess.add_argument(
|
||||
"--format",
|
||||
choices=["json", "markdown"],
|
||||
default="markdown",
|
||||
help="Comparison report format.",
|
||||
)
|
||||
self_assess.add_argument(
|
||||
"--with-llm",
|
||||
action="store_false",
|
||||
dest="no_llm",
|
||||
help="Use configured LLM assistance during the self-assessment run.",
|
||||
)
|
||||
self_assess.add_argument(
|
||||
"--agentic-review",
|
||||
action="store_true",
|
||||
help="Request configured agentic review; leaves candidates pending when none is configured.",
|
||||
)
|
||||
self_assess.add_argument(
|
||||
"--fail-on-regression",
|
||||
action="store_true",
|
||||
help="Return exit code 1 only when comparison status is regression.",
|
||||
)
|
||||
self_assess.add_argument("--database-path", help="Override REPO_SCOPING_DATABASE_PATH.")
|
||||
self_assess.add_argument("--checkout-root", help="Override REPO_SCOPING_CHECKOUT_ROOT.")
|
||||
self_assess.set_defaults(no_llm=True)
|
||||
criteria = subparsers.add_parser(
|
||||
"list-quality-criteria",
|
||||
help="List the active characteristic quality criteria registry.",
|
||||
)
|
||||
criteria.add_argument(
|
||||
"--criteria-path",
|
||||
help="Override the default quality criteria registry JSON path.",
|
||||
)
|
||||
criteria.add_argument("--output", help="Write criteria output to this path instead of stdout.")
|
||||
criteria.add_argument(
|
||||
"--format",
|
||||
choices=["json", "markdown"],
|
||||
default="markdown",
|
||||
help="Criteria output format.",
|
||||
)
|
||||
legacy = subparsers.add_parser(
|
||||
"list-legacy-auto-approvals",
|
||||
help="List historical trusted deterministic auto-approval records.",
|
||||
)
|
||||
legacy.add_argument("--database-path", help="Override REPO_SCOPING_DATABASE_PATH.")
|
||||
legacy.add_argument("--checkout-root", help="Override REPO_SCOPING_CHECKOUT_ROOT.")
|
||||
legacy.add_argument("--output", help="Write inventory output to this path instead of stdout.")
|
||||
legacy.add_argument(
|
||||
"--format",
|
||||
choices=["json", "markdown"],
|
||||
default="markdown",
|
||||
help="Inventory output format.",
|
||||
)
|
||||
dataset = subparsers.add_parser(
|
||||
"assess-dataset",
|
||||
help="Summarize repository generation coverage across the local dataset.",
|
||||
)
|
||||
dataset.add_argument("--database-path", help="Override REPO_SCOPING_DATABASE_PATH.")
|
||||
dataset.add_argument("--checkout-root", help="Override REPO_SCOPING_CHECKOUT_ROOT.")
|
||||
dataset.add_argument("--output", help="Write dataset assessment to this path instead of stdout.")
|
||||
dataset.add_argument(
|
||||
"--format",
|
||||
choices=["json", "markdown"],
|
||||
default="markdown",
|
||||
help="Dataset assessment output format.",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
if args.command == "rebuild-characteristics":
|
||||
return rebuild_characteristics_command(args, parser)
|
||||
if args.command == "export-assessment":
|
||||
return export_assessment_command(args, parser)
|
||||
if args.command == "compare-assessment":
|
||||
return compare_assessment_command(args)
|
||||
if args.command == "self-assess":
|
||||
return self_assess_command(args, parser)
|
||||
if args.command == "list-quality-criteria":
|
||||
return list_quality_criteria_command(args)
|
||||
if args.command == "list-legacy-auto-approvals":
|
||||
return list_legacy_auto_approvals_command(args)
|
||||
if args.command == "assess-dataset":
|
||||
return assess_dataset_command(args)
|
||||
parser.error(f"unknown command: {args.command}")
|
||||
return 2
|
||||
|
||||
|
||||
def rebuild_characteristics_command(
|
||||
args: argparse.Namespace,
|
||||
parser: argparse.ArgumentParser,
|
||||
) -> int:
|
||||
dry_run = bool(args.dry_run)
|
||||
if not dry_run and args.all and not args.confirm_all:
|
||||
parser.error("--all destructive rebuilds require --confirm-all")
|
||||
if not dry_run and not (args.confirm or args.confirm_all):
|
||||
parser.error("destructive rebuilds require --confirm or --confirm-all")
|
||||
|
||||
service = service_from_args(args)
|
||||
repositories = selected_repositories(service, args)
|
||||
if not repositories:
|
||||
parser.error("no repositories matched the requested target")
|
||||
|
||||
for repository in repositories:
|
||||
result = service.rebuild_characteristics_from_scratch(
|
||||
repository.id,
|
||||
dry_run=dry_run,
|
||||
confirm=not dry_run,
|
||||
use_llm_assistance=not args.no_llm,
|
||||
)
|
||||
if args.agentic_review and not dry_run and result.analysis_run.status == "completed":
|
||||
service.request_agentic_review(
|
||||
repository.id,
|
||||
result.analysis_run.id,
|
||||
notes="CLI agentic review request after rebuild.",
|
||||
)
|
||||
print(rebuild_summary_line(service, result, args))
|
||||
return 0
|
||||
|
||||
|
||||
def compare_assessment_command(args: argparse.Namespace) -> int:
|
||||
comparison = compare_assessment_to_golden(
|
||||
load_json(args.golden),
|
||||
load_json(args.assessment),
|
||||
)
|
||||
content = (
|
||||
comparison_json(comparison)
|
||||
if args.format == "json"
|
||||
else comparison_markdown(comparison)
|
||||
)
|
||||
if args.output:
|
||||
write_text(args.output, content)
|
||||
else:
|
||||
print(content, end="" if content.endswith("\n") else "\n")
|
||||
return 0
|
||||
|
||||
|
||||
def list_quality_criteria_command(args: argparse.Namespace) -> int:
|
||||
registry = load_quality_criteria(args.criteria_path)
|
||||
content = (
|
||||
criteria_registry_json(registry)
|
||||
if args.format == "json"
|
||||
else criteria_registry_markdown(registry)
|
||||
)
|
||||
if args.output:
|
||||
write_text(args.output, content)
|
||||
else:
|
||||
print(content, end="" if content.endswith("\n") else "\n")
|
||||
return 0
|
||||
|
||||
|
||||
def list_legacy_auto_approvals_command(args: argparse.Namespace) -> int:
|
||||
service = service_from_args(args)
|
||||
records = service.list_trusted_auto_approval_migration_records()
|
||||
if args.format == "json":
|
||||
content = json.dumps([asdict(record) for record in records], indent=2) + "\n"
|
||||
else:
|
||||
content = legacy_auto_approval_records_markdown(records)
|
||||
if args.output:
|
||||
write_text(args.output, content)
|
||||
else:
|
||||
print(content, end="" if content.endswith("\n") else "\n")
|
||||
return 0
|
||||
|
||||
|
||||
def assess_dataset_command(args: argparse.Namespace) -> int:
|
||||
service = service_from_args(args)
|
||||
report = dataset_assessment(service)
|
||||
content = (
|
||||
json.dumps(report, indent=2) + "\n"
|
||||
if args.format == "json"
|
||||
else dataset_assessment_markdown(report)
|
||||
)
|
||||
if args.output:
|
||||
write_text(args.output, content)
|
||||
else:
|
||||
print(content, end="" if content.endswith("\n") else "\n")
|
||||
return 0
|
||||
|
||||
|
||||
def dataset_assessment(service: RegistryService) -> dict[str, object]:
|
||||
repositories = []
|
||||
totals = {
|
||||
"repositories": 0,
|
||||
"facts": 0,
|
||||
"content_chunks": 0,
|
||||
"candidate_abilities": 0,
|
||||
"candidate_capabilities": 0,
|
||||
"candidate_features": 0,
|
||||
"candidate_evidence": 0,
|
||||
"approved_abilities": 0,
|
||||
"approved_capabilities": 0,
|
||||
"approved_features": 0,
|
||||
"approved_evidence": 0,
|
||||
"dependency_graph_nodes": 0,
|
||||
"dependency_graph_edges": 0,
|
||||
}
|
||||
for repository in service.list_repositories():
|
||||
runs = service.list_analysis_runs(repository.id)
|
||||
latest_run = next((run for run in runs if run.status == "completed"), None)
|
||||
facts = service.list_observed_facts(repository.id, latest_run.id) if latest_run else []
|
||||
chunks = service.list_content_chunks(repository.id, latest_run.id) if latest_run else []
|
||||
candidate_counts = {
|
||||
"abilities": 0,
|
||||
"capabilities": 0,
|
||||
"features": 0,
|
||||
"evidence": 0,
|
||||
}
|
||||
candidate_names: list[str] = []
|
||||
if latest_run is not None:
|
||||
try:
|
||||
graph = service.candidate_graph(repository.id, latest_run.id)
|
||||
except NotFoundError:
|
||||
graph = None
|
||||
if graph is not None:
|
||||
candidate_counts = candidate_graph_counts(graph)
|
||||
candidate_names = [
|
||||
ability.name
|
||||
for ability in graph.abilities
|
||||
][:5]
|
||||
ability_map = service.ability_map(repository.id)
|
||||
approved_counts = approved_graph_counts(ability_map)
|
||||
graph_metrics = {"node_count": 0, "edge_count": 0}
|
||||
try:
|
||||
dependency_graph = service.dependency_graph_elements(repository.id)
|
||||
graph_metrics = {
|
||||
"node_count": int(dependency_graph["metrics"]["node_count"]),
|
||||
"edge_count": int(dependency_graph["metrics"]["edge_count"]),
|
||||
}
|
||||
except (NotFoundError, ValueError):
|
||||
pass
|
||||
snapshot = (
|
||||
service.store.get_snapshot(latest_run.snapshot_id)
|
||||
if latest_run is not None and latest_run.snapshot_id is not None
|
||||
else None
|
||||
)
|
||||
doc_presence = document_presence(snapshot.source_path if snapshot else "")
|
||||
issues = dataset_assessment_issues(
|
||||
fact_count=len(facts),
|
||||
chunk_count=len(chunks),
|
||||
candidate_counts=candidate_counts,
|
||||
approved_counts=approved_counts,
|
||||
graph_metrics=graph_metrics,
|
||||
doc_presence=doc_presence,
|
||||
candidate_names=candidate_names,
|
||||
)
|
||||
repositories.append(
|
||||
{
|
||||
"repository_id": repository.id,
|
||||
"name": repository.name,
|
||||
"status": repository.status,
|
||||
"latest_analysis_run_id": latest_run.id if latest_run else None,
|
||||
"latest_analysis_run_status": latest_run.status if latest_run else None,
|
||||
"facts": len(facts),
|
||||
"content_chunks": len(chunks),
|
||||
"candidate_counts": candidate_counts,
|
||||
"approved_counts": approved_counts,
|
||||
"dependency_graph": graph_metrics,
|
||||
"documents": doc_presence,
|
||||
"candidate_ability_names": candidate_names,
|
||||
"issues": issues,
|
||||
}
|
||||
)
|
||||
totals["repositories"] += 1
|
||||
totals["facts"] += len(facts)
|
||||
totals["content_chunks"] += len(chunks)
|
||||
totals["candidate_abilities"] += candidate_counts["abilities"]
|
||||
totals["candidate_capabilities"] += candidate_counts["capabilities"]
|
||||
totals["candidate_features"] += candidate_counts["features"]
|
||||
totals["candidate_evidence"] += candidate_counts["evidence"]
|
||||
totals["approved_abilities"] += approved_counts["abilities"]
|
||||
totals["approved_capabilities"] += approved_counts["capabilities"]
|
||||
totals["approved_features"] += approved_counts["features"]
|
||||
totals["approved_evidence"] += approved_counts["evidence"]
|
||||
totals["dependency_graph_nodes"] += graph_metrics["node_count"]
|
||||
totals["dependency_graph_edges"] += graph_metrics["edge_count"]
|
||||
return {
|
||||
"schema_version": "repo-scoping-dataset-assessment/v1",
|
||||
"summary": totals,
|
||||
"repositories": repositories,
|
||||
}
|
||||
|
||||
|
||||
def candidate_graph_counts(graph) -> dict[str, int]:
|
||||
capabilities = [
|
||||
capability
|
||||
for ability in graph.abilities
|
||||
for capability in ability.capabilities
|
||||
]
|
||||
return {
|
||||
"abilities": len(graph.abilities),
|
||||
"capabilities": len(capabilities),
|
||||
"features": sum(len(capability.features) for capability in capabilities),
|
||||
"evidence": sum(len(capability.evidence) for capability in capabilities),
|
||||
}
|
||||
|
||||
|
||||
def approved_graph_counts(ability_map) -> dict[str, int]:
|
||||
capabilities = [
|
||||
capability
|
||||
for ability in ability_map.abilities
|
||||
for capability in ability.capabilities
|
||||
]
|
||||
return {
|
||||
"scope": 1 if ability_map.scope else 0,
|
||||
"abilities": len(ability_map.abilities),
|
||||
"capabilities": len(capabilities),
|
||||
"features": sum(len(capability.features) for capability in capabilities),
|
||||
"evidence": sum(len(capability.evidence) for capability in capabilities),
|
||||
}
|
||||
|
||||
|
||||
def document_presence(source_path: str) -> dict[str, bool]:
|
||||
if not source_path:
|
||||
return {
|
||||
"INTENT.md": False,
|
||||
"SCOPE.md": False,
|
||||
"README": False,
|
||||
"CLAUDE.md": False,
|
||||
"AGENTS.md": False,
|
||||
}
|
||||
root = Path(source_path)
|
||||
return {
|
||||
"INTENT.md": (root / "INTENT.md").is_file(),
|
||||
"SCOPE.md": (root / "SCOPE.md").is_file(),
|
||||
"README": any(root.glob("README*")),
|
||||
"CLAUDE.md": (root / "CLAUDE.md").is_file(),
|
||||
"AGENTS.md": (root / "AGENTS.md").is_file(),
|
||||
}
|
||||
|
||||
|
||||
def dataset_assessment_issues(
|
||||
*,
|
||||
fact_count: int,
|
||||
chunk_count: int,
|
||||
candidate_counts: dict[str, int],
|
||||
approved_counts: dict[str, int],
|
||||
graph_metrics: dict[str, int],
|
||||
doc_presence: dict[str, bool],
|
||||
candidate_names: list[str],
|
||||
) -> list[str]:
|
||||
issues: list[str] = []
|
||||
if fact_count and not candidate_counts["capabilities"]:
|
||||
issues.append("facts-without-candidate-capabilities")
|
||||
if chunk_count and doc_presence.get("SCOPE.md") and not candidate_counts["capabilities"]:
|
||||
issues.append("scope-text-unused-for-lower-hierarchy")
|
||||
if fact_count and not graph_metrics["node_count"]:
|
||||
issues.append("facts-with-empty-dependency-graph")
|
||||
if approved_counts["abilities"] == 0 and graph_metrics["node_count"] == 0:
|
||||
issues.append("approved-hierarchy-missing-and-no-graph-fallback")
|
||||
if any("repo-seed" in name.lower() for name in candidate_names):
|
||||
issues.append("template-readme-contamination")
|
||||
return issues
|
||||
|
||||
|
||||
def dataset_assessment_markdown(report: dict[str, object]) -> str:
|
||||
lines = ["# Repo-Scoping Dataset Assessment", ""]
|
||||
summary = report["summary"]
|
||||
lines.extend(
|
||||
[
|
||||
f"- Repositories: {summary['repositories']}",
|
||||
f"- Facts: {summary['facts']}",
|
||||
f"- Candidate hierarchy: {summary['candidate_abilities']} abilities / "
|
||||
f"{summary['candidate_capabilities']} capabilities / "
|
||||
f"{summary['candidate_features']} features / "
|
||||
f"{summary['candidate_evidence']} evidence",
|
||||
f"- Approved hierarchy: {summary['approved_abilities']} abilities / "
|
||||
f"{summary['approved_capabilities']} capabilities / "
|
||||
f"{summary['approved_features']} features / "
|
||||
f"{summary['approved_evidence']} evidence",
|
||||
f"- Dependency graph: {summary['dependency_graph_nodes']} nodes / "
|
||||
f"{summary['dependency_graph_edges']} edges",
|
||||
"",
|
||||
"| Repo | Run | Facts | Chunks | Candidate | Approved | Graph | Issues |",
|
||||
"| --- | ---: | ---: | ---: | --- | --- | --- | --- |",
|
||||
]
|
||||
)
|
||||
for item in report["repositories"]:
|
||||
candidate = item["candidate_counts"]
|
||||
approved = item["approved_counts"]
|
||||
graph = item["dependency_graph"]
|
||||
lines.append(
|
||||
f"| {item['name']} | {item['latest_analysis_run_id'] or '-'} | "
|
||||
f"{item['facts']} | {item['content_chunks']} | "
|
||||
f"{candidate['abilities']}/{candidate['capabilities']}/"
|
||||
f"{candidate['features']}/{candidate['evidence']} | "
|
||||
f"{approved['abilities']}/{approved['capabilities']}/"
|
||||
f"{approved['features']}/{approved['evidence']} | "
|
||||
f"{graph['node_count']}/{graph['edge_count']} | "
|
||||
f"{', '.join(item['issues']) or '-'} |"
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def legacy_auto_approval_records_markdown(records) -> str:
|
||||
if not records:
|
||||
return "No legacy trusted auto-approval records found.\n"
|
||||
lines = ["# Legacy Trusted Auto-Approval Records", ""]
|
||||
for record in records:
|
||||
lines.extend(
|
||||
[
|
||||
(
|
||||
f"- repo={record.repository_id}:{record.repository_name} "
|
||||
f"run={record.analysis_run_id} decision={record.review_decision_id}"
|
||||
),
|
||||
f" status={record.analysis_run_status} scanner={record.scanner_version or 'unknown'}",
|
||||
f" approved_abilities={record.current_approved_ability_count}",
|
||||
f" next={record.recommended_next_step}",
|
||||
]
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def self_assess_command(
|
||||
args: argparse.Namespace,
|
||||
parser: argparse.ArgumentParser,
|
||||
) -> int:
|
||||
service = service_from_args(args)
|
||||
source_path = Path(args.source_path).expanduser().resolve()
|
||||
if not source_path.is_dir():
|
||||
parser.error(f"source path does not exist or is not a directory: {source_path}")
|
||||
repository = self_assessment_repository(service, args.repo, source_path)
|
||||
summary = service.analyze_repository(
|
||||
repository.id,
|
||||
source_path=str(source_path),
|
||||
use_llm_assistance=not args.no_llm,
|
||||
agentic_review=args.agentic_review,
|
||||
trusted_auto_approve=False,
|
||||
)
|
||||
if summary.analysis_run.status != "completed":
|
||||
parser.error(summary.analysis_run.error_message or "analysis failed")
|
||||
artifact = export_assessment_artifact(
|
||||
service,
|
||||
repository.id,
|
||||
summary.analysis_run.id,
|
||||
role="challenger",
|
||||
outcome="challenger",
|
||||
reviewer="self-assess",
|
||||
)
|
||||
comparison = compare_assessment_to_golden(load_json(args.golden), artifact)
|
||||
|
||||
if args.assessment_output:
|
||||
write_text(args.assessment_output, artifact_json(artifact))
|
||||
report = (
|
||||
comparison_json(comparison)
|
||||
if args.format == "json"
|
||||
else comparison_markdown(comparison)
|
||||
)
|
||||
if args.comparison_output:
|
||||
write_text(args.comparison_output, report)
|
||||
else:
|
||||
print(report, end="" if report.endswith("\n") else "\n")
|
||||
if args.fail_on_regression and comparison["status"] == "regression":
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
def export_assessment_command(
|
||||
args: argparse.Namespace,
|
||||
parser: argparse.ArgumentParser,
|
||||
) -> int:
|
||||
service = service_from_args(args)
|
||||
repositories = selected_repositories(service, args)
|
||||
if not repositories:
|
||||
parser.error("no repositories matched the requested target")
|
||||
if len(repositories) > 1:
|
||||
parser.error("assessment export requires exactly one repository")
|
||||
repository = repositories[0]
|
||||
try:
|
||||
artifact = export_assessment_artifact(
|
||||
service,
|
||||
repository.id,
|
||||
args.analysis_run,
|
||||
role=args.role,
|
||||
outcome=args.outcome,
|
||||
reviewer=args.reviewer,
|
||||
summary=args.summary,
|
||||
)
|
||||
except (NotFoundError, ValueError) as exc:
|
||||
parser.error(str(exc))
|
||||
|
||||
content = artifact_json(artifact)
|
||||
if args.output:
|
||||
write_text(args.output, content)
|
||||
else:
|
||||
print(content, end="")
|
||||
return 0
|
||||
|
||||
|
||||
def service_from_args(args: argparse.Namespace) -> RegistryService:
|
||||
settings = Settings()
|
||||
database_path = Path(args.database_path or settings.database_path)
|
||||
checkout_root = args.checkout_root or settings.checkout_root
|
||||
database_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
store = RegistryStore(database_path)
|
||||
store.initialize()
|
||||
llm_extractor = None
|
||||
no_llm = getattr(args, "no_llm", True)
|
||||
if not no_llm and settings.llm_enabled and settings.llm_provider:
|
||||
adapter = create_llm_connect_adapter(settings.llm_provider, model=settings.llm_model)
|
||||
llm_extractor = LLMCandidateExtractor(adapter)
|
||||
return RegistryService(
|
||||
store,
|
||||
ingestion=GitIngestionService(checkout_root),
|
||||
llm_extractor=llm_extractor,
|
||||
)
|
||||
|
||||
|
||||
def selected_repositories(
|
||||
service: RegistryService,
|
||||
args: argparse.Namespace,
|
||||
) -> list[Repository]:
|
||||
repositories = service.list_repositories()
|
||||
if getattr(args, "all", False):
|
||||
return repositories
|
||||
repo = str(args.repo)
|
||||
if repo.isdigit():
|
||||
try:
|
||||
return [service.get_repository(int(repo))]
|
||||
except NotFoundError:
|
||||
return []
|
||||
return [repository for repository in repositories if repository.name == repo]
|
||||
|
||||
|
||||
def self_assessment_repository(
|
||||
service: RegistryService,
|
||||
repo: str,
|
||||
source_path: Path,
|
||||
) -> Repository:
|
||||
selected = selected_repositories(service, argparse.Namespace(repo=repo, all=False))
|
||||
if selected:
|
||||
return selected[0]
|
||||
if repo.isdigit():
|
||||
raise NotFoundError(f"repository {repo} was not found")
|
||||
return service.register_repository(
|
||||
name=repo,
|
||||
url=str(source_path),
|
||||
description="Self-scoping assessment target.",
|
||||
)
|
||||
|
||||
|
||||
def write_text(path: str | Path, content: str) -> None:
|
||||
target = Path(path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(content, encoding="utf-8")
|
||||
|
||||
|
||||
def rebuild_summary_line(
|
||||
service: RegistryService,
|
||||
result: CharacteristicRebuildResult,
|
||||
args: argparse.Namespace,
|
||||
) -> str:
|
||||
graph = (
|
||||
service.candidate_graph(result.repository.id, result.analysis_run.id)
|
||||
if result.analysis_run.status == "completed"
|
||||
else None
|
||||
)
|
||||
remaining_review = 0
|
||||
if graph is not None:
|
||||
remaining_review = sum(
|
||||
1
|
||||
for ability in graph.abilities
|
||||
for capability in ability.capabilities
|
||||
if capability.status == "candidate"
|
||||
)
|
||||
candidate_source = "deterministic" if args.no_llm else "configured"
|
||||
return (
|
||||
f"repo={result.repository.id}:{result.repository.name} "
|
||||
f"latest_analysis_run={result.analysis_run.id} "
|
||||
f"candidate_source={candidate_source} "
|
||||
f"dry_run={result.dry_run} "
|
||||
f"cleared_approved={result.cleared_approved} "
|
||||
f"approved_superseded={result.previous_counts} "
|
||||
f"candidates={result.candidate_counts} "
|
||||
f"remaining_review_queue={remaining_review}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
3
src/repo_scoping/content_indexing/__init__.py
Normal file
3
src/repo_scoping/content_indexing/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
from repo_scoping.content_indexing.extractor import ContentChunkCandidate, ContentExtractor
|
||||
|
||||
__all__ = ["ContentChunkCandidate", "ContentExtractor"]
|
||||
@@ -3,7 +3,7 @@ from __future__ import annotations
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
from repo_registry.core.models import ObservedFact
|
||||
from repo_scoping.core.models import ObservedFact
|
||||
|
||||
|
||||
INDEXED_FACT_KINDS = {
|
||||
@@ -5,7 +5,7 @@ import logging
|
||||
from typing import Any
|
||||
|
||||
|
||||
LOGGER_NAME = "repo_registry.operations"
|
||||
LOGGER_NAME = "repo_scoping.operations"
|
||||
|
||||
|
||||
def log_operation(event: str, **fields: Any) -> None:
|
||||
@@ -52,6 +52,131 @@ class ReviewDecision:
|
||||
action: str
|
||||
notes: str
|
||||
created_at: str
|
||||
reviewer_type: str = "unknown"
|
||||
reviewer_id: str = ""
|
||||
policy_version: str = ""
|
||||
criteria_version: str = ""
|
||||
criterion_ids: list[str] = field(default_factory=list)
|
||||
evidence_refs: list[str] = field(default_factory=list)
|
||||
rationale: str = ""
|
||||
accepted_after_edits: bool = False
|
||||
decision_kind: str = "other"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TrustedAutoApprovalMigrationRecord:
|
||||
repository_id: int
|
||||
repository_name: str
|
||||
repository_url: str
|
||||
repository_status: str
|
||||
analysis_run_id: int | None
|
||||
analysis_run_status: str
|
||||
scanner_version: str
|
||||
review_decision_id: int
|
||||
decision_created_at: str
|
||||
notes: str
|
||||
current_approved_ability_count: int
|
||||
recommended_next_step: str
|
||||
|
||||
|
||||
def enrich_review_decision(decision: ReviewDecision) -> ReviewDecision:
|
||||
fields = review_decision_audit_fields(decision.action, decision.notes)
|
||||
return replace_review_decision(decision, **fields)
|
||||
|
||||
|
||||
def replace_review_decision(
|
||||
decision: ReviewDecision,
|
||||
**fields: object,
|
||||
) -> ReviewDecision:
|
||||
data = {
|
||||
"id": decision.id,
|
||||
"repository_id": decision.repository_id,
|
||||
"analysis_run_id": decision.analysis_run_id,
|
||||
"action": decision.action,
|
||||
"notes": decision.notes,
|
||||
"created_at": decision.created_at,
|
||||
"reviewer_type": decision.reviewer_type,
|
||||
"reviewer_id": decision.reviewer_id,
|
||||
"policy_version": decision.policy_version,
|
||||
"criteria_version": decision.criteria_version,
|
||||
"criterion_ids": decision.criterion_ids,
|
||||
"evidence_refs": decision.evidence_refs,
|
||||
"rationale": decision.rationale,
|
||||
"accepted_after_edits": decision.accepted_after_edits,
|
||||
"decision_kind": decision.decision_kind,
|
||||
}
|
||||
data.update(fields)
|
||||
return ReviewDecision(**data)
|
||||
|
||||
|
||||
def review_decision_audit_fields(action: str, notes: str) -> dict[str, object]:
|
||||
parsed = _parse_review_decision_notes(notes)
|
||||
return {
|
||||
"reviewer_type": _reviewer_type(action),
|
||||
"reviewer_id": parsed.get("reviewer", ""),
|
||||
"policy_version": parsed.get("policy_version", ""),
|
||||
"criteria_version": parsed.get("criteria_version", ""),
|
||||
"criterion_ids": _split_audit_list(parsed.get("criteria", "")),
|
||||
"evidence_refs": _split_audit_list(parsed.get("evidence", "")),
|
||||
"rationale": parsed.get("rationale", ""),
|
||||
"accepted_after_edits": action.endswith("_with_edits")
|
||||
or action == "agentic_approve_with_edits"
|
||||
or bool(parsed.get("proposed_changes")),
|
||||
"decision_kind": _decision_kind(action),
|
||||
}
|
||||
|
||||
|
||||
def _parse_review_decision_notes(notes: str) -> dict[str, str]:
|
||||
parsed: dict[str, str] = {}
|
||||
for part in notes.split(";"):
|
||||
key, separator, value = part.strip().partition("=")
|
||||
if separator and key:
|
||||
parsed[key] = value.strip()
|
||||
return parsed
|
||||
|
||||
|
||||
def _split_audit_list(value: str) -> list[str]:
|
||||
if not value or value == "none":
|
||||
return []
|
||||
return [item.strip() for item in value.split(",") if item.strip()]
|
||||
|
||||
|
||||
def _reviewer_type(action: str) -> str:
|
||||
if action == "quality_gate_override":
|
||||
return "human"
|
||||
if action.startswith("agentic_"):
|
||||
return "agent"
|
||||
if action == "trusted_auto_approve_candidate_graph":
|
||||
return "migration"
|
||||
if action.startswith("quality_gate_"):
|
||||
return "deterministic-gate"
|
||||
if action.startswith("approve") or action.startswith("accept"):
|
||||
return "human"
|
||||
if action.startswith("reject") or action.startswith("edit") or action.startswith("merge"):
|
||||
return "human"
|
||||
if action.startswith("relink"):
|
||||
return "human"
|
||||
return "migration" if action.startswith("llm_extraction") else "unknown"
|
||||
|
||||
|
||||
def _decision_kind(action: str) -> str:
|
||||
if "approve_with_edits" in action:
|
||||
return "accepted_after_edits"
|
||||
if "approve" in action or action.startswith("accept"):
|
||||
return "accepted_as_is"
|
||||
if "reject" in action:
|
||||
return "rejected"
|
||||
if "downgrade" in action:
|
||||
return "downgraded"
|
||||
if "request_human_review" in action:
|
||||
return "needs_human"
|
||||
if "override" in action:
|
||||
return "override"
|
||||
if "propose_edit" in action:
|
||||
return "proposed_edit"
|
||||
if "relink" in action:
|
||||
return "relinked"
|
||||
return "other"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,4 +1,4 @@
|
||||
from repo_registry.llm_extraction.extractor import (
|
||||
from repo_scoping.llm_extraction.extractor import (
|
||||
ExtractedAbility,
|
||||
ExtractedCapability,
|
||||
ExtractedEvidence,
|
||||
@@ -7,7 +7,7 @@ from repo_registry.llm_extraction.extractor import (
|
||||
LLMExtractionError,
|
||||
create_llm_connect_adapter,
|
||||
)
|
||||
from repo_registry.llm_extraction.mapper import LLMExtractionMapper
|
||||
from repo_scoping.llm_extraction.mapper import LLMExtractionMapper
|
||||
|
||||
__all__ = [
|
||||
"ExtractedAbility",
|
||||
@@ -4,7 +4,7 @@ import json
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Protocol
|
||||
|
||||
from repo_registry.core.models import ContentChunk, Repository
|
||||
from repo_scoping.core.models import ContentChunk, Repository
|
||||
|
||||
|
||||
class LLMExtractionError(ValueError):
|
||||
@@ -1,13 +1,13 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from repo_registry.candidate_graph.generator import (
|
||||
from repo_scoping.candidate_graph.generator import (
|
||||
CandidateAbilityDraft,
|
||||
CandidateCapabilityDraft,
|
||||
CandidateEvidenceDraft,
|
||||
CandidateFeatureDraft,
|
||||
)
|
||||
from repo_registry.core.models import ContentChunk, ObservedFact, SourceReference
|
||||
from repo_registry.llm_extraction.extractor import ExtractedAbility
|
||||
from repo_scoping.core.models import ContentChunk, ObservedFact, SourceReference
|
||||
from repo_scoping.llm_extraction.extractor import ExtractedAbility
|
||||
|
||||
|
||||
class LLMExtractionMapper:
|
||||
@@ -19,10 +19,16 @@ class RepositoryMetadataExtractor:
|
||||
pyproject = self._from_pyproject(root)
|
||||
package = self._from_package_json(root)
|
||||
readme = self._from_readme(root)
|
||||
fallback_name = self._name_from_url_or_path(url)
|
||||
source_name = self._source_name(url) or self._source_name(str(root))
|
||||
|
||||
return RepositoryMetadata(
|
||||
name=pyproject.name or package.name or readme.name or fallback_name,
|
||||
name=(
|
||||
source_name
|
||||
or pyproject.name
|
||||
or package.name
|
||||
or readme.name
|
||||
or "repository"
|
||||
),
|
||||
description=(
|
||||
pyproject.description
|
||||
or package.description
|
||||
@@ -77,10 +83,13 @@ class RepositoryMetadataExtractor:
|
||||
return RepositoryMetadata(name=title, description=None)
|
||||
return RepositoryMetadata(name="", description=None)
|
||||
|
||||
def _name_from_url_or_path(self, value: str) -> str:
|
||||
def _source_name(self, value: str) -> str:
|
||||
parsed = urlparse(value)
|
||||
path = parsed.path if parsed.scheme else value
|
||||
name = Path(path.rstrip("/")).name or "repository"
|
||||
name = Path(path.rstrip("/")).name
|
||||
if name.endswith(".git"):
|
||||
name = name[:-4]
|
||||
return name or "repository"
|
||||
normalized = name.strip()
|
||||
if normalized.lower() in {"", ".", "repo", "repository", "source", "checkout"}:
|
||||
return ""
|
||||
return normalized
|
||||
@@ -20,6 +20,7 @@ IGNORED_DIRS = {
|
||||
"dist",
|
||||
"node_modules",
|
||||
"target",
|
||||
"var",
|
||||
"vendor",
|
||||
}
|
||||
|
||||
4
src/repo_scoping/scope/__init__.py
Normal file
4
src/repo_scoping/scope/__init__.py
Normal file
@@ -0,0 +1,4 @@
|
||||
from repo_scoping.scope.generator import ScopeGenerator
|
||||
from repo_scoping.scope.validator import ScopeValidator
|
||||
|
||||
__all__ = ["ScopeGenerator", "ScopeValidator"]
|
||||
@@ -3,8 +3,8 @@ from __future__ import annotations
|
||||
import re
|
||||
from dataclasses import asdict
|
||||
|
||||
from repo_registry.core.service import RegistryService
|
||||
from repo_registry.storage.sqlite import NotFoundError
|
||||
from repo_scoping.core.service import RegistryService
|
||||
from repo_scoping.storage.sqlite import NotFoundError
|
||||
|
||||
|
||||
SCOPE_SECTIONS = [
|
||||
@@ -4,7 +4,7 @@ import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from repo_registry.scope.generator import SCOPE_SECTIONS, ScopeGenerator
|
||||
from repo_scoping.scope.generator import SCOPE_SECTIONS, ScopeGenerator
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
13
src/repo_scoping/self_scoping/__init__.py
Normal file
13
src/repo_scoping/self_scoping/__init__.py
Normal file
@@ -0,0 +1,13 @@
|
||||
from repo_scoping.self_scoping.assessment import export_assessment_artifact
|
||||
from repo_scoping.self_scoping.comparison import compare_assessment_to_golden
|
||||
from repo_scoping.self_scoping.review_store import (
|
||||
record_assessment_outcome,
|
||||
record_assessment_pair_outcome,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"compare_assessment_to_golden",
|
||||
"export_assessment_artifact",
|
||||
"record_assessment_outcome",
|
||||
"record_assessment_pair_outcome",
|
||||
]
|
||||
478
src/repo_scoping/self_scoping/assessment.py
Normal file
478
src/repo_scoping/self_scoping/assessment.py
Normal file
@@ -0,0 +1,478 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from collections import Counter
|
||||
from dataclasses import asdict
|
||||
from datetime import UTC, datetime
|
||||
from importlib import metadata
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from repo_scoping.acceptance import (
|
||||
active_quality_criteria_version,
|
||||
evaluate_candidate_graph_quality,
|
||||
quality_gate_outcome_dicts,
|
||||
)
|
||||
from repo_scoping.core.models import (
|
||||
Ability,
|
||||
CandidateAbility,
|
||||
CandidateCapability,
|
||||
CandidateEvidence,
|
||||
CandidateFeature,
|
||||
ContentChunk,
|
||||
ObservedFact,
|
||||
RepositoryAbilityMap,
|
||||
ReviewDecision,
|
||||
SourceReference,
|
||||
)
|
||||
from repo_scoping.core.service import RegistryService
|
||||
|
||||
|
||||
SCHEMA_VERSION = "self-scoping-assessment/v1"
|
||||
KNOWN_PROVIDER_ROUTING_CAPABILITY = "Route LLM Requests Across Providers"
|
||||
|
||||
|
||||
def export_assessment_artifact(
|
||||
service: RegistryService,
|
||||
repository_id: int,
|
||||
analysis_run_id: int,
|
||||
*,
|
||||
role: str = "challenger",
|
||||
outcome: str = "challenger",
|
||||
reviewer: str = "codex",
|
||||
summary: str | None = None,
|
||||
engine_root: str | Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Export a completed analysis run as a self-scoping assessment artifact."""
|
||||
|
||||
repository = service.get_repository(repository_id)
|
||||
analysis_run = service.get_analysis_run(repository_id, analysis_run_id)
|
||||
if analysis_run.status != "completed":
|
||||
raise ValueError(
|
||||
f"analysis run {analysis_run_id} is {analysis_run.status}, not completed"
|
||||
)
|
||||
snapshot = (
|
||||
service.store.get_snapshot(analysis_run.snapshot_id)
|
||||
if analysis_run.snapshot_id is not None
|
||||
else None
|
||||
)
|
||||
facts = service.list_observed_facts(repository_id, analysis_run_id)
|
||||
chunks = service.list_content_chunks(repository_id, analysis_run_id)
|
||||
graph = service.candidate_graph(repository_id, analysis_run_id)
|
||||
gate_outcomes = evaluate_candidate_graph_quality(graph)
|
||||
ability_map = service.ability_map(repository_id)
|
||||
decisions = service.list_review_decisions(repository_id, analysis_run_id)
|
||||
engine_identity = _engine_identity(
|
||||
analysis_run.scanner_version,
|
||||
Path(engine_root or Path.cwd()),
|
||||
)
|
||||
regression_patterns = _known_regression_patterns(graph.abilities, decisions)
|
||||
comparison_eligibility = _comparison_eligibility(
|
||||
role,
|
||||
engine_identity["release_binding_status"],
|
||||
)
|
||||
artifact_summary = summary or _summary(role, regression_patterns)
|
||||
|
||||
return {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"artifact_id": _artifact_id(repository.name, analysis_run_id, role),
|
||||
"artifact_type": "assessment_run",
|
||||
"created_at": datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z"),
|
||||
"target_repository": {
|
||||
"repo_slug": _slug(repository.name),
|
||||
"repository_id": repository.id,
|
||||
"source": snapshot.source_path if snapshot is not None else repository.url,
|
||||
"target_commit": snapshot.commit_hash if snapshot is not None else "unknown",
|
||||
"target_branch": snapshot.branch if snapshot is not None else repository.branch,
|
||||
"dirty_state": _dirty_state(Path(snapshot.source_path)) if snapshot is not None else "unknown",
|
||||
"file_count": snapshot.file_count if snapshot is not None else None,
|
||||
},
|
||||
"engine_identity": engine_identity,
|
||||
"execution": {
|
||||
"mode": _execution_mode(decisions),
|
||||
"analysis_run_id": analysis_run.id,
|
||||
"candidate_source": _candidate_source(decisions),
|
||||
"acceptance_mode": _acceptance_mode(decisions),
|
||||
"started_at": _timestamp(analysis_run.started_at),
|
||||
"completed_at": _timestamp(analysis_run.completed_at),
|
||||
},
|
||||
"assessment": {
|
||||
"role": role,
|
||||
"outcome": outcome,
|
||||
"summary": artifact_summary,
|
||||
"reviewer": reviewer,
|
||||
"comparison_eligibility": comparison_eligibility,
|
||||
"rationale": _rationale(regression_patterns, comparison_eligibility),
|
||||
},
|
||||
"fact_summary": _fact_summary(facts),
|
||||
"content_chunk_summary": _content_chunk_summary(chunks),
|
||||
"generated_tree": {
|
||||
"abilities": [_candidate_ability(ability) for ability in graph.abilities]
|
||||
},
|
||||
"approved_map": _approved_map(ability_map),
|
||||
"review_decisions": [_review_decision(decision) for decision in decisions],
|
||||
"quality_gate_outcomes": quality_gate_outcome_dicts(gate_outcomes),
|
||||
"known_regression_patterns": regression_patterns,
|
||||
"notes": [
|
||||
"Generated by repo-scoping self-scoping assessment exporter.",
|
||||
(
|
||||
"Artifact is not comparable as a preferred baseline until engine "
|
||||
"identity is complete."
|
||||
if comparison_eligibility == "not_comparable"
|
||||
else "Artifact has enough engine identity metadata for comparison."
|
||||
),
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _engine_identity(scanner_version: str, engine_root: Path) -> dict[str, Any]:
|
||||
engine_commit = _git_value(engine_root, "rev-parse", "HEAD")
|
||||
dirty_state = _dirty_state(engine_root)
|
||||
release = _git_value(engine_root, "describe", "--tags", "--exact-match")
|
||||
release_binding_status = "complete" if engine_commit else "unbound"
|
||||
return {
|
||||
"repo_scoping_version": _package_version(),
|
||||
"engine_commit": engine_commit,
|
||||
"engine_release": release,
|
||||
"engine_dirty_state": dirty_state,
|
||||
"scanner_version": scanner_version,
|
||||
"candidate_generator_version": "unversioned",
|
||||
"quality_criteria_version": active_quality_criteria_version(),
|
||||
"prompt_version": None,
|
||||
"release_binding_status": release_binding_status,
|
||||
"release_binding_note": (
|
||||
"Engine commit was captured from git."
|
||||
if engine_commit
|
||||
else "Engine commit could not be captured; artifact is not comparable."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _package_version() -> str:
|
||||
try:
|
||||
return metadata.version("repo-scoping")
|
||||
except metadata.PackageNotFoundError:
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _git_value(root: Path, *args: str) -> str | None:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "-C", str(root), *args],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
except OSError:
|
||||
return None
|
||||
value = result.stdout.strip()
|
||||
return value if result.returncode == 0 and value else None
|
||||
|
||||
|
||||
def _dirty_state(root: Path) -> str:
|
||||
if not (root / ".git").exists():
|
||||
return "unknown"
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "-C", str(root), "status", "--short"],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
except OSError:
|
||||
return "unknown"
|
||||
if result.returncode != 0:
|
||||
return "unknown"
|
||||
return "dirty" if result.stdout.strip() else "clean"
|
||||
|
||||
|
||||
def _comparison_eligibility(role: str, release_binding_status: str) -> str:
|
||||
if role == "negative_regression_seed":
|
||||
return "eligible_as_negative_seed"
|
||||
if release_binding_status == "complete":
|
||||
return "eligible"
|
||||
return "not_comparable"
|
||||
|
||||
|
||||
def _summary(role: str, regression_patterns: list[dict[str, str]]) -> str:
|
||||
if role == "negative_regression_seed":
|
||||
return "Historical run captured as a negative self-scoping regression seed."
|
||||
if regression_patterns:
|
||||
return "Generated self-scoping assessment repeats known regression patterns."
|
||||
return "Generated self-scoping assessment artifact for comparison."
|
||||
|
||||
|
||||
def _rationale(
|
||||
regression_patterns: list[dict[str, str]],
|
||||
comparison_eligibility: str,
|
||||
) -> list[str]:
|
||||
rationale: list[str] = []
|
||||
if comparison_eligibility == "not_comparable":
|
||||
rationale.append("Engine identity is incomplete, so this cannot be a comparable baseline.")
|
||||
for pattern in regression_patterns:
|
||||
rationale.append(f"{pattern['id']}: {pattern['description']}")
|
||||
return rationale
|
||||
|
||||
|
||||
def _fact_summary(facts: list[ObservedFact]) -> dict[str, Any]:
|
||||
return {
|
||||
"counts_by_kind": dict(sorted(Counter(fact.kind for fact in facts).items())),
|
||||
"contamination_sources": _contamination_sources(facts),
|
||||
}
|
||||
|
||||
|
||||
def _contamination_sources(facts: list[ObservedFact]) -> list[dict[str, str]]:
|
||||
provider_kinds = {
|
||||
"llm_provider",
|
||||
"credential_config",
|
||||
"provider_registry",
|
||||
"fallback_policy",
|
||||
}
|
||||
suspicious_segments = (
|
||||
"test",
|
||||
"tests/",
|
||||
"fixtures",
|
||||
"expectations",
|
||||
"schemas.py",
|
||||
"scanner.py",
|
||||
"normalization.py",
|
||||
"workplans/",
|
||||
)
|
||||
results: list[dict[str, str]] = []
|
||||
seen: set[str] = set()
|
||||
for fact in facts:
|
||||
lower = fact.path.lower()
|
||||
if fact.kind not in provider_kinds or not any(segment in lower for segment in suspicious_segments):
|
||||
continue
|
||||
if fact.path in seen:
|
||||
continue
|
||||
seen.add(fact.path)
|
||||
results.append(
|
||||
{
|
||||
"path": fact.path,
|
||||
"reason": (
|
||||
"Provider-related fact came from scanner rules, tests, fixtures, "
|
||||
"schemas, or workplan context and needs native-utility review."
|
||||
),
|
||||
}
|
||||
)
|
||||
return sorted(results, key=lambda item: item["path"])
|
||||
|
||||
|
||||
def _content_chunk_summary(chunks: list[ContentChunk]) -> dict[str, Any]:
|
||||
source_roles = Counter(
|
||||
str(chunk.metadata.get("source_role", "") or "unknown") for chunk in chunks
|
||||
)
|
||||
return {
|
||||
"total": len(chunks),
|
||||
"counts_by_kind": dict(sorted(Counter(chunk.kind for chunk in chunks).items())),
|
||||
"counts_by_source_role": dict(sorted(source_roles.items())),
|
||||
"paths": sorted({chunk.path for chunk in chunks}),
|
||||
}
|
||||
|
||||
|
||||
def _candidate_ability(ability: CandidateAbility) -> dict[str, Any]:
|
||||
return {
|
||||
"name": ability.name,
|
||||
"status": ability.status,
|
||||
"primary_class": ability.primary_class,
|
||||
"source_refs": [_source_ref(ref) for ref in ability.source_refs],
|
||||
"capabilities": [
|
||||
_candidate_capability(capability) for capability in ability.capabilities
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _candidate_capability(capability: CandidateCapability) -> dict[str, Any]:
|
||||
return {
|
||||
"name": capability.name,
|
||||
"status": capability.status,
|
||||
"primary_class": capability.primary_class,
|
||||
"source_refs": [_source_ref(ref) for ref in capability.source_refs],
|
||||
"features": [_candidate_feature(feature) for feature in capability.features],
|
||||
"evidence": [_candidate_evidence(evidence) for evidence in capability.evidence],
|
||||
}
|
||||
|
||||
|
||||
def _candidate_feature(feature: CandidateFeature) -> dict[str, Any]:
|
||||
return {
|
||||
"name": feature.name,
|
||||
"type": feature.type,
|
||||
"status": feature.status,
|
||||
"primary_class": feature.primary_class,
|
||||
"location": feature.location,
|
||||
"source_refs": [_source_ref(ref) for ref in feature.source_refs],
|
||||
}
|
||||
|
||||
|
||||
def _candidate_evidence(evidence: CandidateEvidence) -> dict[str, Any]:
|
||||
return {
|
||||
"type": evidence.type,
|
||||
"reference": evidence.reference,
|
||||
"strength": evidence.strength,
|
||||
"status": evidence.status,
|
||||
"source_refs": [_source_ref(ref) for ref in evidence.source_refs],
|
||||
}
|
||||
|
||||
|
||||
def _approved_map(ability_map: RepositoryAbilityMap) -> dict[str, Any]:
|
||||
return {
|
||||
"scope": asdict(ability_map.scope),
|
||||
"abilities": [_approved_ability(ability) for ability in ability_map.abilities],
|
||||
}
|
||||
|
||||
|
||||
def _approved_ability(ability: Ability) -> dict[str, Any]:
|
||||
return {
|
||||
"name": ability.name,
|
||||
"primary_class": ability.primary_class,
|
||||
"capabilities": [
|
||||
{
|
||||
"name": capability.name,
|
||||
"primary_class": capability.primary_class,
|
||||
"features": [
|
||||
{
|
||||
"name": feature.name,
|
||||
"type": feature.type,
|
||||
"primary_class": feature.primary_class,
|
||||
"location": feature.location,
|
||||
"source_refs": [
|
||||
_source_ref(ref) for ref in feature.source_refs
|
||||
],
|
||||
}
|
||||
for feature in capability.features
|
||||
],
|
||||
"evidence": [asdict(evidence) for evidence in capability.evidence],
|
||||
}
|
||||
for capability in ability.capabilities
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _source_ref(ref: SourceReference) -> dict[str, Any]:
|
||||
return asdict(ref)
|
||||
|
||||
|
||||
def _review_decision(decision: ReviewDecision) -> dict[str, Any]:
|
||||
payload = asdict(decision)
|
||||
payload["quality_criteria_version"] = active_quality_criteria_version()
|
||||
return payload
|
||||
|
||||
|
||||
def _known_regression_patterns(
|
||||
abilities: list[CandidateAbility],
|
||||
decisions: list[ReviewDecision],
|
||||
) -> list[dict[str, str]]:
|
||||
patterns: list[dict[str, str]] = []
|
||||
llm_capabilities = [
|
||||
capability
|
||||
for ability in abilities
|
||||
for capability in ability.capabilities
|
||||
if capability.name == KNOWN_PROVIDER_ROUTING_CAPABILITY
|
||||
]
|
||||
if llm_capabilities:
|
||||
patterns.append(
|
||||
{
|
||||
"id": "RREG-SELF-REG-001",
|
||||
"title": "LLM provider vocabulary promoted as native capability",
|
||||
"severity": "critical",
|
||||
"description": (
|
||||
"Generated tree contains Route LLM Requests Across Providers "
|
||||
"as a repo-scoping capability."
|
||||
),
|
||||
"detection_hint": (
|
||||
"Flag the provider-routing capability unless product intent "
|
||||
"and public implementation explicitly support it."
|
||||
),
|
||||
}
|
||||
)
|
||||
if any(
|
||||
feature.type in {"API", "CLI"}
|
||||
for capability in llm_capabilities
|
||||
for feature in capability.features
|
||||
):
|
||||
patterns.append(
|
||||
{
|
||||
"id": "RREG-SELF-REG-002",
|
||||
"title": "Native API and CLI surfaces attached under false capability",
|
||||
"severity": "high",
|
||||
"description": (
|
||||
"API or CLI surface features are nested below provider routing."
|
||||
),
|
||||
"detection_hint": (
|
||||
"Flag API/CLI surface features whose parent capability is "
|
||||
"llm-integration or provider-routing."
|
||||
),
|
||||
}
|
||||
)
|
||||
if any(decision.action == "trusted_auto_approve_candidate_graph" for decision in decisions):
|
||||
patterns.append(
|
||||
{
|
||||
"id": "RREG-SELF-REG-003",
|
||||
"title": "Deterministic trusted auto-approval accepted candidate truth",
|
||||
"severity": "high",
|
||||
"description": (
|
||||
"Candidate characteristics were approved through trusted "
|
||||
"auto-approval instead of human or agentic judgement."
|
||||
),
|
||||
"detection_hint": "Flag trusted_auto_approve_candidate_graph review decisions.",
|
||||
}
|
||||
)
|
||||
return patterns
|
||||
|
||||
|
||||
def _execution_mode(decisions: list[ReviewDecision]) -> str:
|
||||
if any(decision.action.startswith("agentic_review") for decision in decisions):
|
||||
return "agentic-review"
|
||||
if any(decision.action == "trusted_auto_approve_candidate_graph" for decision in decisions):
|
||||
return "trusted-auto-review"
|
||||
if any(decision.action == "llm_extraction_used" for decision in decisions):
|
||||
return "llm-assisted"
|
||||
if any(decision.action.startswith("approve") for decision in decisions):
|
||||
return "manual-review"
|
||||
return "deterministic-only"
|
||||
|
||||
|
||||
def _candidate_source(decisions: list[ReviewDecision]) -> str:
|
||||
return "llm+deterministic" if any(
|
||||
decision.action == "llm_extraction_used" for decision in decisions
|
||||
) else "deterministic"
|
||||
|
||||
|
||||
def _acceptance_mode(decisions: list[ReviewDecision]) -> str:
|
||||
agentic_decision = next(
|
||||
(decision for decision in decisions if decision.action.startswith("agentic_review")),
|
||||
None,
|
||||
)
|
||||
if agentic_decision is not None:
|
||||
return agentic_decision.action
|
||||
if any(decision.action == "trusted_auto_approve_candidate_graph" for decision in decisions):
|
||||
return "trusted_auto_approve_candidate_graph"
|
||||
if any(decision.action == "approve_candidate_graph" for decision in decisions):
|
||||
return "manual_candidate_graph_approval"
|
||||
if any(decision.action == "approve_analysis_run_changes" for decision in decisions):
|
||||
return "manual_change_approval"
|
||||
return "pending_review"
|
||||
|
||||
|
||||
def _timestamp(value: str | None) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
if "T" in value:
|
||||
return value
|
||||
return value.replace(" ", "T") + "Z"
|
||||
|
||||
|
||||
def _artifact_id(repository_name: str, analysis_run_id: int, role: str) -> str:
|
||||
return f"{_slug(repository_name)}-{role}-run-{analysis_run_id}"
|
||||
|
||||
|
||||
def _slug(value: str) -> str:
|
||||
return "-".join(
|
||||
token for token in "".join(char.lower() if char.isalnum() else "-" for char in value).split("-") if token
|
||||
)
|
||||
|
||||
|
||||
def artifact_json(artifact: dict[str, Any]) -> str:
|
||||
return json.dumps(artifact, indent=2, sort_keys=True) + "\n"
|
||||
238
src/repo_scoping/self_scoping/comparison.py
Normal file
238
src/repo_scoping/self_scoping/comparison.py
Normal file
@@ -0,0 +1,238 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
COMPARISON_SCHEMA_VERSION = "self-scoping-comparison/v1"
|
||||
|
||||
|
||||
def load_json(path: str | Path) -> dict[str, Any]:
|
||||
return json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def compare_assessment_to_golden(
|
||||
golden_profile: dict[str, Any],
|
||||
assessment: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
expected = _expected_capabilities(golden_profile)
|
||||
forbidden = _forbidden_capabilities(golden_profile)
|
||||
generated = _generated_capabilities(assessment)
|
||||
generated_names = set(generated)
|
||||
missing_expected = sorted(expected - generated_names)
|
||||
matched_expected = sorted(expected & generated_names)
|
||||
forbidden_present = sorted(forbidden & generated_names)
|
||||
known_regressions = assessment.get("known_regression_patterns", [])
|
||||
misplaced_features = _misplaced_features(generated)
|
||||
status = _status(
|
||||
missing_expected=missing_expected,
|
||||
forbidden_present=forbidden_present,
|
||||
known_regressions=known_regressions,
|
||||
misplaced_features=misplaced_features,
|
||||
)
|
||||
|
||||
return {
|
||||
"schema_version": COMPARISON_SCHEMA_VERSION,
|
||||
"comparison_id": _comparison_id(golden_profile, assessment),
|
||||
"created_at": datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z"),
|
||||
"golden_profile_id": golden_profile.get("profile_id", ""),
|
||||
"assessment_artifact_id": assessment.get("artifact_id", ""),
|
||||
"target_repo_slug": assessment.get("target_repository", {}).get("repo_slug", ""),
|
||||
"status": status,
|
||||
"summary": _summary(status, missing_expected, forbidden_present, known_regressions),
|
||||
"matched_expected_capabilities": matched_expected,
|
||||
"missing_expected_capabilities": missing_expected,
|
||||
"unexpected_native_capabilities": _unexpected_capabilities(
|
||||
generated_names,
|
||||
expected,
|
||||
forbidden,
|
||||
),
|
||||
"forbidden_native_capabilities_present": forbidden_present,
|
||||
"known_regression_patterns": known_regressions,
|
||||
"misplaced_features": misplaced_features,
|
||||
"comparison_hints": _comparison_hints(status),
|
||||
}
|
||||
|
||||
|
||||
def comparison_json(comparison: dict[str, Any]) -> str:
|
||||
return json.dumps(comparison, indent=2, sort_keys=True) + "\n"
|
||||
|
||||
|
||||
def comparison_markdown(comparison: dict[str, Any]) -> str:
|
||||
lines = [
|
||||
f"# Self-Scoping Comparison: {comparison['assessment_artifact_id']}",
|
||||
"",
|
||||
f"- Status: `{comparison['status']}`",
|
||||
f"- Golden profile: `{comparison['golden_profile_id']}`",
|
||||
f"- Target repo: `{comparison['target_repo_slug']}`",
|
||||
f"- Summary: {comparison['summary']}",
|
||||
"",
|
||||
"## Missing Expected Capabilities",
|
||||
*_bullets(comparison["missing_expected_capabilities"]),
|
||||
"",
|
||||
"## Forbidden Native Capabilities Present",
|
||||
*_bullets(comparison["forbidden_native_capabilities_present"]),
|
||||
"",
|
||||
"## Known Regression Patterns",
|
||||
*_regression_bullets(comparison["known_regression_patterns"]),
|
||||
"",
|
||||
"## Misplaced Features",
|
||||
*_misplaced_feature_bullets(comparison["misplaced_features"]),
|
||||
"",
|
||||
"## Matched Expected Capabilities",
|
||||
*_bullets(comparison["matched_expected_capabilities"]),
|
||||
"",
|
||||
"## Review Hints",
|
||||
*_bullets(comparison["comparison_hints"]),
|
||||
"",
|
||||
]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _expected_capabilities(golden_profile: dict[str, Any]) -> set[str]:
|
||||
return {
|
||||
capability["name"]
|
||||
for capability in golden_profile.get("ability", {}).get("expected_capabilities", [])
|
||||
if capability.get("name")
|
||||
}
|
||||
|
||||
|
||||
def _forbidden_capabilities(golden_profile: dict[str, Any]) -> set[str]:
|
||||
return {
|
||||
capability["name"]
|
||||
for capability in golden_profile.get("forbidden_native_capabilities", [])
|
||||
if capability.get("name")
|
||||
}
|
||||
|
||||
|
||||
def _generated_capabilities(assessment: dict[str, Any]) -> dict[str, dict[str, Any]]:
|
||||
result: dict[str, dict[str, Any]] = {}
|
||||
for ability in assessment.get("generated_tree", {}).get("abilities", []):
|
||||
for capability in ability.get("capabilities", []):
|
||||
name = capability.get("name")
|
||||
if name:
|
||||
result[name] = capability
|
||||
return result
|
||||
|
||||
|
||||
def _unexpected_capabilities(
|
||||
generated_names: set[str],
|
||||
expected: set[str],
|
||||
forbidden: set[str],
|
||||
) -> list[str]:
|
||||
return sorted(generated_names - expected - forbidden)
|
||||
|
||||
|
||||
def _misplaced_features(
|
||||
generated: dict[str, dict[str, Any]],
|
||||
) -> list[dict[str, str]]:
|
||||
misplaced: list[dict[str, str]] = []
|
||||
for capability_name, capability in generated.items():
|
||||
primary_class = capability.get("primary_class", "")
|
||||
if primary_class not in {"llm-integration", "provider-routing"}:
|
||||
continue
|
||||
for feature in capability.get("features", []):
|
||||
if feature.get("type") not in {"API", "CLI"}:
|
||||
continue
|
||||
misplaced.append(
|
||||
{
|
||||
"capability": capability_name,
|
||||
"feature": feature.get("name", ""),
|
||||
"feature_type": feature.get("type", ""),
|
||||
"reason": "API/CLI surface is nested below provider-routing capability.",
|
||||
}
|
||||
)
|
||||
return misplaced
|
||||
|
||||
|
||||
def _status(
|
||||
*,
|
||||
missing_expected: list[str],
|
||||
forbidden_present: list[str],
|
||||
known_regressions: list[dict[str, Any]],
|
||||
misplaced_features: list[dict[str, str]],
|
||||
) -> str:
|
||||
if forbidden_present or misplaced_features or any(
|
||||
item.get("severity") in {"high", "critical"} for item in known_regressions
|
||||
):
|
||||
return "regression"
|
||||
if missing_expected or known_regressions:
|
||||
return "needs_review"
|
||||
return "candidate_improvement"
|
||||
|
||||
|
||||
def _summary(
|
||||
status: str,
|
||||
missing_expected: list[str],
|
||||
forbidden_present: list[str],
|
||||
known_regressions: list[dict[str, Any]],
|
||||
) -> str:
|
||||
if status == "regression":
|
||||
return (
|
||||
"Assessment repeats known or forbidden self-scoping patterns; prefer "
|
||||
"the golden profile until the engine is corrected."
|
||||
)
|
||||
if status == "needs_review":
|
||||
return (
|
||||
f"Assessment needs review: {len(missing_expected)} expected "
|
||||
f"capability(s) missing and {len(known_regressions)} regression "
|
||||
"pattern(s) reported."
|
||||
)
|
||||
return "Assessment covers the golden profile without known regression patterns."
|
||||
|
||||
|
||||
def _comparison_hints(status: str) -> list[str]:
|
||||
if status == "regression":
|
||||
return [
|
||||
"Do not promote this assessment as a preferred baseline.",
|
||||
"Inspect forbidden capabilities and misplaced features first.",
|
||||
"Use the findings as signal for scanner, generator, or acceptance-policy changes.",
|
||||
]
|
||||
if status == "needs_review":
|
||||
return [
|
||||
"Review missing expected capabilities before choosing old or new output.",
|
||||
"Check whether the golden profile needs a curator-approved update.",
|
||||
]
|
||||
return [
|
||||
"Candidate appears better than the known golden checks.",
|
||||
"Human or agentic review should still confirm source evidence quality.",
|
||||
]
|
||||
|
||||
|
||||
def _comparison_id(
|
||||
golden_profile: dict[str, Any],
|
||||
assessment: dict[str, Any],
|
||||
) -> str:
|
||||
return (
|
||||
f"{golden_profile.get('profile_id', 'golden')}"
|
||||
f"__{assessment.get('artifact_id', 'assessment')}"
|
||||
)
|
||||
|
||||
|
||||
def _bullets(items: list[str]) -> list[str]:
|
||||
if not items:
|
||||
return ["- None"]
|
||||
return [f"- {item}" for item in items]
|
||||
|
||||
|
||||
def _regression_bullets(items: list[dict[str, Any]]) -> list[str]:
|
||||
if not items:
|
||||
return ["- None"]
|
||||
return [
|
||||
f"- `{item.get('id', '')}` {item.get('title', '')}: {item.get('description', '')}"
|
||||
for item in items
|
||||
]
|
||||
|
||||
|
||||
def _misplaced_feature_bullets(items: list[dict[str, str]]) -> list[str]:
|
||||
if not items:
|
||||
return ["- None"]
|
||||
return [
|
||||
(
|
||||
f"- `{item['feature']}` under `{item['capability']}` "
|
||||
f"({item['feature_type']}): {item['reason']}"
|
||||
)
|
||||
for item in items
|
||||
]
|
||||
217
src/repo_scoping/self_scoping/review_store.py
Normal file
217
src/repo_scoping/self_scoping/review_store.py
Normal file
@@ -0,0 +1,217 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
|
||||
SELF_SCOPING_ROOT_ENV = "REPO_SCOPING_SELF_SCOPING_ROOT"
|
||||
OUTCOME_SCHEMA_VERSION = "self-scoping-review-outcome/v1"
|
||||
ALLOWED_OUTCOMES = {
|
||||
"prefer_golden",
|
||||
"prefer_assessment",
|
||||
"prefer_baseline",
|
||||
"prefer_challenger",
|
||||
"tie",
|
||||
"needs_human",
|
||||
"reject_assessment",
|
||||
"reject_challenger",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReviewArtifact:
|
||||
path: str
|
||||
artifact_id: str
|
||||
title: str
|
||||
updated_at: str
|
||||
|
||||
|
||||
def self_scoping_root(root: str | Path | None = None) -> Path:
|
||||
configured = root or os.environ.get(SELF_SCOPING_ROOT_ENV) or "docs/self-scoping"
|
||||
return Path(configured).resolve()
|
||||
|
||||
|
||||
def list_golden_profiles(root: str | Path | None = None) -> list[ReviewArtifact]:
|
||||
return _list_artifacts("golden", root=root)
|
||||
|
||||
|
||||
def list_assessment_artifacts(root: str | Path | None = None) -> list[ReviewArtifact]:
|
||||
return _list_artifacts("assessments", root=root)
|
||||
|
||||
|
||||
def load_json_artifact(
|
||||
relative_path: str,
|
||||
root: str | Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
artifact_path = _safe_artifact_path(relative_path, root=root)
|
||||
return json.loads(artifact_path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def list_outcome_records(root: str | Path | None = None) -> list[dict[str, Any]]:
|
||||
outcomes_dir = self_scoping_root(root) / "outcomes"
|
||||
if not outcomes_dir.exists():
|
||||
return []
|
||||
records: list[dict[str, Any]] = []
|
||||
for path in sorted(outcomes_dir.glob("*.json"), reverse=True):
|
||||
try:
|
||||
records.append(json.loads(path.read_text(encoding="utf-8")))
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
return records
|
||||
|
||||
|
||||
def record_assessment_outcome(
|
||||
*,
|
||||
golden_path: str,
|
||||
assessment_path: str,
|
||||
outcome: str,
|
||||
reviewer: str,
|
||||
notes: str,
|
||||
comparison_status: str,
|
||||
root: str | Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
if outcome not in ALLOWED_OUTCOMES:
|
||||
raise ValueError(f"unsupported review outcome: {outcome}")
|
||||
|
||||
base = self_scoping_root(root)
|
||||
golden = load_json_artifact(golden_path, root=base)
|
||||
assessment = load_json_artifact(assessment_path, root=base)
|
||||
created_at = _created_at()
|
||||
outcome_id = _outcome_id(created_at, assessment_path, outcome)
|
||||
record = {
|
||||
"schema_version": OUTCOME_SCHEMA_VERSION,
|
||||
"outcome_id": outcome_id,
|
||||
"created_at": created_at,
|
||||
"reviewer": reviewer.strip() or "codex",
|
||||
"outcome": outcome,
|
||||
"notes": notes.strip(),
|
||||
"comparison_status": comparison_status,
|
||||
"golden_profile_path": golden_path,
|
||||
"golden_profile_id": golden.get("profile_id", ""),
|
||||
"assessment_artifact_path": assessment_path,
|
||||
"assessment_artifact_id": assessment.get("artifact_id", ""),
|
||||
"engine_identity": assessment.get("engine_identity", {}),
|
||||
"decision_scope": "baseline-comparison",
|
||||
}
|
||||
|
||||
_write_outcome(record, base)
|
||||
return record
|
||||
|
||||
|
||||
def record_assessment_pair_outcome(
|
||||
*,
|
||||
baseline_path: str,
|
||||
challenger_path: str,
|
||||
outcome: str,
|
||||
reviewer: str,
|
||||
notes: str,
|
||||
comparison_status: str,
|
||||
root: str | Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
if outcome not in ALLOWED_OUTCOMES:
|
||||
raise ValueError(f"unsupported review outcome: {outcome}")
|
||||
|
||||
base = self_scoping_root(root)
|
||||
baseline = load_json_artifact(baseline_path, root=base)
|
||||
challenger = load_json_artifact(challenger_path, root=base)
|
||||
created_at = _created_at()
|
||||
outcome_id = _outcome_id(
|
||||
created_at,
|
||||
f"{Path(baseline_path).stem}__{Path(challenger_path).stem}",
|
||||
outcome,
|
||||
)
|
||||
record = {
|
||||
"schema_version": OUTCOME_SCHEMA_VERSION,
|
||||
"outcome_id": outcome_id,
|
||||
"created_at": created_at,
|
||||
"reviewer": reviewer.strip() or "codex",
|
||||
"outcome": outcome,
|
||||
"notes": notes.strip(),
|
||||
"comparison_status": comparison_status,
|
||||
"baseline_assessment_path": baseline_path,
|
||||
"baseline_assessment_artifact_id": baseline.get("artifact_id", ""),
|
||||
"baseline_engine_identity": baseline.get("engine_identity", {}),
|
||||
"challenger_assessment_path": challenger_path,
|
||||
"challenger_assessment_artifact_id": challenger.get("artifact_id", ""),
|
||||
"challenger_engine_identity": challenger.get("engine_identity", {}),
|
||||
"decision_scope": "assessment-pair-comparison",
|
||||
}
|
||||
_write_outcome(record, base)
|
||||
return record
|
||||
|
||||
|
||||
def _created_at() -> str:
|
||||
return (
|
||||
datetime.now(UTC)
|
||||
.replace(microsecond=0)
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
|
||||
|
||||
def _write_outcome(record: dict[str, Any], base: Path) -> None:
|
||||
outcomes_dir = base / "outcomes"
|
||||
outcomes_dir.mkdir(parents=True, exist_ok=True)
|
||||
output_path = outcomes_dir / f"{record['outcome_id']}.json"
|
||||
output_path.write_text(
|
||||
json.dumps(record, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _list_artifacts(kind: str, root: str | Path | None = None) -> list[ReviewArtifact]:
|
||||
base = self_scoping_root(root)
|
||||
artifacts: list[ReviewArtifact] = []
|
||||
for path in sorted((base / kind).glob("*.json")):
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
artifacts.append(
|
||||
ReviewArtifact(
|
||||
path=path.relative_to(base).as_posix(),
|
||||
artifact_id=str(
|
||||
payload.get("artifact_id") or payload.get("profile_id") or path.stem
|
||||
),
|
||||
title=str(
|
||||
payload.get("title")
|
||||
or payload.get("assessment", {}).get("summary")
|
||||
or payload.get("artifact_type")
|
||||
or path.stem
|
||||
),
|
||||
updated_at=str(
|
||||
payload.get("updated_at") or payload.get("created_at") or ""
|
||||
),
|
||||
)
|
||||
)
|
||||
return artifacts
|
||||
|
||||
|
||||
def _safe_artifact_path(relative_path: str, root: str | Path | None = None) -> Path:
|
||||
base = self_scoping_root(root)
|
||||
artifact_path = (base / relative_path).resolve()
|
||||
try:
|
||||
artifact_path.relative_to(base)
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"artifact path escapes self-scoping root: {relative_path}") from exc
|
||||
if artifact_path.suffix != ".json":
|
||||
raise ValueError(f"artifact path is not JSON: {relative_path}")
|
||||
if not artifact_path.exists():
|
||||
raise FileNotFoundError(relative_path)
|
||||
return artifact_path
|
||||
|
||||
|
||||
def _outcome_id(created_at: str, assessment_path: str, outcome: str) -> str:
|
||||
timestamp = (
|
||||
created_at.replace("-", "")
|
||||
.replace(":", "")
|
||||
.replace("T", "-")
|
||||
.replace("Z", "")
|
||||
)
|
||||
assessment_stem = Path(assessment_path).stem.replace(".", "-")
|
||||
return f"{timestamp}__{assessment_stem}__{outcome}__{uuid4().hex[:8]}"
|
||||
@@ -1,4 +1,4 @@
|
||||
from repo_registry.semantic.embeddings import (
|
||||
from repo_scoping.semantic.embeddings import (
|
||||
EmbeddingProvider,
|
||||
HashingEmbeddingProvider,
|
||||
cosine_similarity,
|
||||
@@ -4,7 +4,7 @@ import json
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from repo_registry.core.models import (
|
||||
from repo_scoping.core.models import (
|
||||
Ability,
|
||||
AbilitySummary,
|
||||
AnalysisRun,
|
||||
@@ -30,10 +30,10 @@ from repo_registry.core.models import (
|
||||
SourceReference,
|
||||
confidence_label,
|
||||
)
|
||||
from repo_registry.core.logging import log_operation
|
||||
from repo_registry.content_indexing.extractor import ContentChunkCandidate
|
||||
from repo_registry.candidate_graph.generator import CandidateAbilityDraft
|
||||
from repo_registry.repo_scanning.scanner import FactCandidate, ScanResult
|
||||
from repo_scoping.core.logging import log_operation
|
||||
from repo_scoping.content_indexing.extractor import ContentChunkCandidate
|
||||
from repo_scoping.candidate_graph.generator import CandidateAbilityDraft
|
||||
from repo_scoping.repo_scanning.scanner import FactCandidate, ScanResult
|
||||
|
||||
|
||||
class NotFoundError(ValueError):
|
||||
@@ -12,13 +12,19 @@ from fastapi.responses import PlainTextResponse
|
||||
from pydantic import Field
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
from repo_registry.core.service import RegistryService
|
||||
from repo_registry.llm_extraction import LLMCandidateExtractor, create_llm_connect_adapter
|
||||
from repo_registry.repo_ingestion.git import GitIngestionService
|
||||
from repo_registry.semantic import HashingEmbeddingProvider
|
||||
from repo_registry.scope import ScopeGenerator, ScopeValidator
|
||||
from repo_registry.storage.sqlite import NotFoundError, RegistryStore
|
||||
from repo_registry.web_api.schemas import (
|
||||
from repo_scoping.acceptance import (
|
||||
criteria_registry_dict,
|
||||
evaluate_candidate_graph_quality,
|
||||
load_quality_criteria,
|
||||
quality_gate_outcome_dicts,
|
||||
)
|
||||
from repo_scoping.core.service import RegistryService
|
||||
from repo_scoping.llm_extraction import LLMCandidateExtractor, create_llm_connect_adapter
|
||||
from repo_scoping.repo_ingestion.git import GitIngestionService
|
||||
from repo_scoping.semantic import HashingEmbeddingProvider
|
||||
from repo_scoping.scope import ScopeGenerator, ScopeValidator
|
||||
from repo_scoping.storage.sqlite import NotFoundError, RegistryStore
|
||||
from repo_scoping.web_api.schemas import (
|
||||
AbilityCreate,
|
||||
AbilitySummaryResponse,
|
||||
AbilityUpdate,
|
||||
@@ -58,6 +64,8 @@ from repo_registry.web_api.schemas import (
|
||||
FeatureUpdate,
|
||||
IdResponse,
|
||||
ObservedFactResponse,
|
||||
QualityCriteriaRegistryResponse,
|
||||
QualityGateOverrideCreate,
|
||||
RepositoryAbilityMapResponse,
|
||||
RepositoryComparisonResponse,
|
||||
RepositoryCreate,
|
||||
@@ -67,6 +75,7 @@ from repo_registry.web_api.schemas import (
|
||||
ReviewDecisionResponse,
|
||||
ScanSummaryResponse,
|
||||
SearchResultResponse,
|
||||
TrustedAutoApprovalMigrationRecordResponse,
|
||||
)
|
||||
|
||||
|
||||
@@ -77,9 +86,9 @@ def slugify(value: str) -> str:
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_prefix="REPO_REGISTRY_")
|
||||
model_config = SettingsConfigDict(env_prefix="REPO_SCOPING_")
|
||||
|
||||
database_path: str = Field(default="var/repo-registry.sqlite3")
|
||||
database_path: str = Field(default="var/repo-scoping.sqlite3")
|
||||
checkout_root: str = Field(default="var/checkouts")
|
||||
llm_enabled: bool = Field(default=True)
|
||||
llm_provider: str | None = Field(default=None)
|
||||
@@ -94,7 +103,7 @@ def get_settings() -> Settings:
|
||||
|
||||
|
||||
def get_service(settings: Settings = Depends(get_settings)) -> RegistryService:
|
||||
logging.getLogger("repo_registry.operations").setLevel(
|
||||
logging.getLogger("repo_scoping.operations").setLevel(
|
||||
getattr(logging, settings.log_level.upper(), logging.INFO)
|
||||
)
|
||||
database_path = Path(settings.database_path)
|
||||
@@ -119,6 +128,14 @@ def get_service(settings: Settings = Depends(get_settings)) -> RegistryService:
|
||||
)
|
||||
|
||||
|
||||
def candidate_graph_payload(graph) -> dict[str, object]:
|
||||
payload = asdict(graph)
|
||||
payload["quality_gate_outcomes"] = quality_gate_outcome_dicts(
|
||||
evaluate_candidate_graph_quality(graph)
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
API_DESCRIPTION = (
|
||||
"Register repositories, analyze their observable implementation facts, "
|
||||
"curate reviewable scope graphs, and search approved repository characteristics."
|
||||
@@ -148,7 +165,7 @@ app = FastAPI(
|
||||
)
|
||||
|
||||
|
||||
from repo_registry.web_ui.views import router as ui_router
|
||||
from repo_scoping.web_ui.views import router as ui_router
|
||||
|
||||
app.include_router(ui_router)
|
||||
|
||||
@@ -183,6 +200,29 @@ def health(settings: Settings = Depends(get_settings)) -> dict[str, object]:
|
||||
}
|
||||
|
||||
|
||||
@app.get(
|
||||
"/quality-criteria",
|
||||
tags=["review"],
|
||||
response_model=QualityCriteriaRegistryResponse,
|
||||
)
|
||||
def list_quality_criteria() -> dict[str, object]:
|
||||
return criteria_registry_dict(load_quality_criteria())
|
||||
|
||||
|
||||
@app.get(
|
||||
"/review/migrations/trusted-auto-approvals",
|
||||
tags=["review"],
|
||||
response_model=list[TrustedAutoApprovalMigrationRecordResponse],
|
||||
)
|
||||
def list_trusted_auto_approval_migration_records(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> list[dict[str, object]]:
|
||||
return [
|
||||
asdict(record)
|
||||
for record in service.list_trusted_auto_approval_migration_records()
|
||||
]
|
||||
|
||||
|
||||
@app.post(
|
||||
"/repos",
|
||||
status_code=201,
|
||||
@@ -271,6 +311,7 @@ def create_analysis_run(
|
||||
source_path=payload.source_path,
|
||||
use_cached_checkout=payload.use_cached_checkout,
|
||||
use_llm_assistance=payload.use_llm_assistance,
|
||||
agentic_review=payload.agentic_review,
|
||||
trusted_auto_approve=payload.trusted_auto_approve,
|
||||
access_username=payload.access_username,
|
||||
access_password=payload.access_password,
|
||||
@@ -446,6 +487,29 @@ def list_analysis_run_review_decisions(
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.post(
|
||||
"/repos/{repository_id}/analysis-runs/{analysis_run_id}/quality-gate-overrides",
|
||||
tags=["review"],
|
||||
response_model=ReviewDecisionResponse,
|
||||
)
|
||||
def create_quality_gate_override(
|
||||
repository_id: int,
|
||||
analysis_run_id: int,
|
||||
payload: QualityGateOverrideCreate,
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
service.record_quality_gate_override(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
**payload.model_dump(),
|
||||
)
|
||||
)
|
||||
except (NotFoundError, ValueError) as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.get(
|
||||
"/repos/{repository_id}/observed-facts",
|
||||
tags=["analysis"],
|
||||
@@ -514,7 +578,9 @@ def get_candidate_graph(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(service.candidate_graph(repository_id, analysis_run_id))
|
||||
return candidate_graph_payload(
|
||||
service.candidate_graph(repository_id, analysis_run_id)
|
||||
)
|
||||
except NotFoundError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
@@ -579,7 +645,7 @@ def reject_candidate_ability(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.reject_candidate_ability(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -605,7 +671,7 @@ def reject_candidate_capability(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.reject_candidate_capability(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -631,7 +697,7 @@ def reject_candidate_feature(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.reject_candidate_feature(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -657,7 +723,7 @@ def reject_candidate_evidence(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.reject_candidate_evidence(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -683,7 +749,7 @@ def edit_candidate_ability(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.edit_candidate_ability(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -709,7 +775,7 @@ def edit_candidate_capability(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.edit_candidate_capability(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -735,7 +801,7 @@ def relink_candidate_capability(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.relink_candidate_capability(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -761,7 +827,7 @@ def relink_candidate_feature(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.relink_candidate_feature(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -787,7 +853,7 @@ def relink_candidate_evidence(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.relink_candidate_evidence(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -813,7 +879,7 @@ def merge_candidate_ability(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.merge_candidate_ability(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -839,7 +905,7 @@ def merge_candidate_capability(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.merge_candidate_capability(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -865,7 +931,7 @@ def merge_candidate_feature(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.merge_candidate_feature(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -891,7 +957,7 @@ def merge_candidate_evidence(
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return asdict(
|
||||
return candidate_graph_payload(
|
||||
service.merge_candidate_evidence(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
@@ -1146,6 +1212,38 @@ def get_ability_map(
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.get(
|
||||
"/repos/{repository_id}/intent/review",
|
||||
tags=["scope"],
|
||||
)
|
||||
def review_repository_intent(
|
||||
repository_id: int,
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return service.document_review(repository_id, "INTENT.md")
|
||||
except NotFoundError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.get(
|
||||
"/repos/{repository_id}/scope/review",
|
||||
tags=["scope"],
|
||||
)
|
||||
def review_repository_scope(
|
||||
repository_id: int,
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> dict[str, object]:
|
||||
try:
|
||||
return service.document_review(repository_id, "SCOPE.md")
|
||||
except NotFoundError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc)) from exc
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.get(
|
||||
"/repos/{repository_id}/dependency-graph",
|
||||
tags=["visualization"],
|
||||
@@ -214,7 +214,20 @@ class AnalysisRunCreate(BaseModel):
|
||||
source_path: str | None = None
|
||||
use_cached_checkout: bool = False
|
||||
use_llm_assistance: bool = True
|
||||
trusted_auto_approve: bool = False
|
||||
agentic_review: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Request configured agentic review after analysis; candidates remain "
|
||||
"pending when no reviewer is configured."
|
||||
),
|
||||
)
|
||||
trusted_auto_approve: bool = Field(
|
||||
default=False,
|
||||
description=(
|
||||
"Deprecated compatibility input. Requests are routed to agentic "
|
||||
"review and do not deterministically approve candidates."
|
||||
),
|
||||
)
|
||||
access_username: str | None = None
|
||||
access_password: str | None = Field(default=None, repr=False)
|
||||
|
||||
@@ -225,7 +238,7 @@ class AnalysisRunCreate(BaseModel):
|
||||
{"source_path": "/path/to/local/repository"},
|
||||
{"use_cached_checkout": True},
|
||||
{"use_llm_assistance": False},
|
||||
{"trusted_auto_approve": True},
|
||||
{"agentic_review": True},
|
||||
{
|
||||
"access_username": "git-user",
|
||||
"access_password": "access-token",
|
||||
@@ -500,6 +513,80 @@ class ReviewDecisionResponse(BaseModel):
|
||||
action: str
|
||||
notes: str
|
||||
created_at: str
|
||||
reviewer_type: str = "unknown"
|
||||
reviewer_id: str = ""
|
||||
policy_version: str = ""
|
||||
criteria_version: str = ""
|
||||
criterion_ids: list[str] = Field(default_factory=list)
|
||||
evidence_refs: list[str] = Field(default_factory=list)
|
||||
rationale: str = ""
|
||||
accepted_after_edits: bool = False
|
||||
decision_kind: str = "other"
|
||||
|
||||
|
||||
class TrustedAutoApprovalMigrationRecordResponse(BaseModel):
|
||||
repository_id: int
|
||||
repository_name: str
|
||||
repository_url: str
|
||||
repository_status: str
|
||||
analysis_run_id: int | None
|
||||
analysis_run_status: str
|
||||
scanner_version: str
|
||||
review_decision_id: int
|
||||
decision_created_at: str
|
||||
notes: str
|
||||
current_approved_ability_count: int
|
||||
recommended_next_step: str
|
||||
|
||||
|
||||
class QualityCriterionResponse(BaseModel):
|
||||
id: str
|
||||
title: str
|
||||
category: str
|
||||
severity: str
|
||||
applies_to: list[str]
|
||||
description: str
|
||||
deterministic_action: str
|
||||
deterministic_action_when: str
|
||||
reviewer_guidance: str
|
||||
agentic_guidance: str = ""
|
||||
examples: list[str] = Field(default_factory=list)
|
||||
|
||||
model_config = {
|
||||
"json_schema_extra": {
|
||||
"examples": [
|
||||
{
|
||||
"id": "RREG-QC-002",
|
||||
"title": "Native Utility Is Repo-Owned",
|
||||
"category": "native-utility",
|
||||
"severity": "high",
|
||||
"applies_to": ["ability", "capability"],
|
||||
"description": "Owned claims require product evidence.",
|
||||
"deterministic_action": "downgraded",
|
||||
"deterministic_action_when": "Evidence is dependency-only.",
|
||||
"reviewer_guidance": "Check whether the repo owns the utility.",
|
||||
"agentic_guidance": "Approve only with product and source evidence.",
|
||||
"examples": ["Dependency use is not native product behavior."],
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class QualityCriteriaRegistryResponse(BaseModel):
|
||||
schema_version: str
|
||||
criteria_version: str
|
||||
status: str
|
||||
updated_at: str
|
||||
criteria: list[QualityCriterionResponse]
|
||||
|
||||
|
||||
class QualityGateOverrideCreate(BaseModel):
|
||||
criterion_id: str
|
||||
element_type: str
|
||||
element_id: int
|
||||
reason: str
|
||||
notes: str = ""
|
||||
|
||||
|
||||
class ObservedFactResponse(BaseModel):
|
||||
@@ -596,10 +683,23 @@ class CandidateAbilityResponse(BaseModel):
|
||||
capabilities: list[CandidateCapabilityResponse]
|
||||
|
||||
|
||||
class QualityGateOutcomeResponse(BaseModel):
|
||||
criteria_version: str
|
||||
criterion_id: str
|
||||
criterion_title: str
|
||||
severity: str
|
||||
outcome: str
|
||||
element_type: str
|
||||
element_id: int
|
||||
element_name: str
|
||||
reason: str
|
||||
|
||||
|
||||
class CandidateGraphResponse(BaseModel):
|
||||
repository: RepositoryResponse
|
||||
analysis_run: AnalysisRunResponse
|
||||
abilities: list[CandidateAbilityResponse]
|
||||
quality_gate_outcomes: list[QualityGateOutcomeResponse] = Field(default_factory=list)
|
||||
|
||||
model_config = {
|
||||
"json_schema_extra": {
|
||||
@@ -656,6 +756,7 @@ class CandidateGraphResponse(BaseModel):
|
||||
],
|
||||
}
|
||||
],
|
||||
"quality_gate_outcomes": [],
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -9,13 +9,41 @@ from urllib.parse import quote_plus, urlparse
|
||||
from fastapi import APIRouter, Depends, Form, HTTPException, Query
|
||||
from fastapi.responses import HTMLResponse, PlainTextResponse, RedirectResponse
|
||||
|
||||
from repo_registry.core.service import RegistryService
|
||||
from repo_registry.storage.sqlite import NotFoundError
|
||||
from repo_registry.web_api.app import get_service
|
||||
from repo_scoping.acceptance import (
|
||||
evaluate_candidate_graph_quality,
|
||||
quality_gate_outcome_dicts,
|
||||
)
|
||||
from repo_scoping.core.service import RegistryService
|
||||
from repo_scoping.self_scoping.comparison import compare_assessment_to_golden
|
||||
from repo_scoping.self_scoping.review_store import (
|
||||
ALLOWED_OUTCOMES,
|
||||
list_assessment_artifacts,
|
||||
list_golden_profiles,
|
||||
list_outcome_records,
|
||||
load_json_artifact,
|
||||
record_assessment_outcome,
|
||||
record_assessment_pair_outcome,
|
||||
)
|
||||
from repo_scoping.storage.sqlite import NotFoundError
|
||||
from repo_scoping.web_api.app import get_service
|
||||
|
||||
|
||||
router = APIRouter(include_in_schema=False)
|
||||
APP_NAME = "Repository Scoping"
|
||||
REVIEW_OUTCOME_LABELS = {
|
||||
"prefer_golden": "Prefer Golden",
|
||||
"prefer_assessment": "Prefer Assessment",
|
||||
"tie": "Tie",
|
||||
"needs_human": "Needs Human Review",
|
||||
"reject_assessment": "Reject Assessment",
|
||||
}
|
||||
PAIR_REVIEW_OUTCOME_LABELS = {
|
||||
"prefer_baseline": "Prefer Baseline",
|
||||
"prefer_challenger": "Prefer Challenger",
|
||||
"tie": "Tie",
|
||||
"needs_human": "Needs Human Review",
|
||||
"reject_challenger": "Reject Challenger",
|
||||
}
|
||||
|
||||
|
||||
def repository_directory_name(url: str, fallback: str) -> str:
|
||||
@@ -188,6 +216,29 @@ def page(
|
||||
}}
|
||||
.tree ul {{ margin: 8px 0 0 20px; padding: 0; }}
|
||||
.tree li {{ margin: 6px 0; }}
|
||||
.review-grid {{
|
||||
display: grid;
|
||||
grid-template-columns: minmax(0, 1fr) minmax(0, 1fr);
|
||||
gap: 18px;
|
||||
align-items: start;
|
||||
}}
|
||||
.review-item {{
|
||||
border-left: 3px solid var(--line);
|
||||
padding: 8px 0 8px 10px;
|
||||
margin: 8px 0;
|
||||
}}
|
||||
.review-item.match {{ border-color: #10b981; }}
|
||||
.review-item.problem {{ border-color: var(--danger); background: #fffafa; }}
|
||||
.review-item.warn {{ border-color: #f59e0b; background: #fffaf0; }}
|
||||
.review-item h3 {{ margin-top: 0; }}
|
||||
.review-list {{ margin: 8px 0 0 18px; padding: 0; }}
|
||||
.review-list .warn {{ color: var(--warn); font-weight: 650; }}
|
||||
.review-meta {{
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
align-items: center;
|
||||
}}
|
||||
.source {{ color: var(--muted); font-family: ui-monospace, SFMono-Regular, Consolas, monospace; font-size: 12px; }}
|
||||
.scope-document {{
|
||||
margin: 0;
|
||||
@@ -279,6 +330,7 @@ def page(
|
||||
header {{ padding: 12px 16px; }}
|
||||
main {{ padding: 16px; }}
|
||||
.grid {{ grid-template-columns: 1fr; }}
|
||||
.review-grid {{ grid-template-columns: 1fr; }}
|
||||
.graph-shell {{ grid-template-columns: 1fr; }}
|
||||
.graph-canvas {{ min-height: 560px; }}
|
||||
table, tbody, tr, td {{ display: block; width: 100%; }}
|
||||
@@ -297,6 +349,7 @@ def page(
|
||||
<nav class="actions">
|
||||
<a href="/ui/search">Search</a>
|
||||
<a href="/ui/discovery">Discovery</a>
|
||||
<a href="/ui/self-scoping">Self-Scoping</a>
|
||||
<a href="/docs">API Docs</a>
|
||||
</nav>
|
||||
</header>
|
||||
@@ -367,7 +420,7 @@ def render_repository_index(
|
||||
<label>Password or access token <input name="access_password" type="password" autocomplete="current-password" placeholder="Used for this Git operation only"></label>
|
||||
<label class="checkbox"><input type="checkbox" name="explore_after_registration" value="1" checked> Explore after registration</label>
|
||||
<label class="checkbox"><input type="checkbox" name="use_llm_assistance" value="1" checked> Use LLM assistance if configured</label>
|
||||
<label class="checkbox"><input type="checkbox" name="trusted_auto_approve" value="1"> Trusted auto-populate after analysis</label>
|
||||
<label class="checkbox"><input type="checkbox" name="agentic_review" value="1"> Request agentic review after analysis</label>
|
||||
<div class="actions">
|
||||
<button type="submit">Register</button>
|
||||
<span data-pending>Registering repository...</span>
|
||||
@@ -405,6 +458,614 @@ def scope_document() -> HTMLResponse:
|
||||
return page("SCOPE.md", body)
|
||||
|
||||
|
||||
def render_self_scoping_index(
|
||||
*,
|
||||
error_message: str | None = None,
|
||||
status_code: int = 200,
|
||||
) -> HTMLResponse:
|
||||
golden_profiles = list_golden_profiles()
|
||||
assessments = list_assessment_artifacts()
|
||||
outcomes = list_outcome_records()
|
||||
error = (
|
||||
f"""
|
||||
<div class="notice error" role="alert">
|
||||
<strong>Self-scoping review failed.</strong>
|
||||
<p>{escape(error_message)}</p>
|
||||
</div>
|
||||
"""
|
||||
if error_message
|
||||
else ""
|
||||
)
|
||||
missing_inputs = ""
|
||||
if not golden_profiles or not assessments:
|
||||
missing_inputs = """
|
||||
<div class="notice warn">
|
||||
Add at least one golden profile and one assessment artifact under
|
||||
<span class="source">docs/self-scoping</span> before opening a comparison.
|
||||
</div>
|
||||
"""
|
||||
body = f"""
|
||||
<h1>Self-Scoping Review</h1>
|
||||
{error}
|
||||
{missing_inputs}
|
||||
<div class="grid">
|
||||
<section class="panel stack">
|
||||
<h2>Compare To Golden</h2>
|
||||
<form class="stack" method="get" action="/ui/self-scoping/review">
|
||||
<label>Golden profile
|
||||
<select name="golden" required>
|
||||
{_review_artifact_options(golden_profiles)}
|
||||
</select>
|
||||
</label>
|
||||
<label>Assessment artifact
|
||||
<select name="assessment" required>
|
||||
{_review_artifact_options(assessments)}
|
||||
</select>
|
||||
</label>
|
||||
<div class="actions">
|
||||
<button type="submit">Open comparison</button>
|
||||
</div>
|
||||
</form>
|
||||
<h2>Compare Two Runs</h2>
|
||||
<form class="stack" method="get" action="/ui/self-scoping/run-review">
|
||||
<label>Baseline assessment
|
||||
<select name="baseline" required>
|
||||
{_review_artifact_options(assessments)}
|
||||
</select>
|
||||
</label>
|
||||
<label>Challenger assessment
|
||||
<select name="challenger" required>
|
||||
{_review_artifact_options(assessments)}
|
||||
</select>
|
||||
</label>
|
||||
<div class="actions">
|
||||
<button type="submit">Open run comparison</button>
|
||||
</div>
|
||||
</form>
|
||||
</section>
|
||||
<section class="panel stack">
|
||||
<h2>Recorded Outcomes</h2>
|
||||
{_render_outcome_table(outcomes)}
|
||||
</section>
|
||||
</div>
|
||||
"""
|
||||
response = page("Self-Scoping Review", body)
|
||||
response.status_code = status_code
|
||||
return response
|
||||
|
||||
|
||||
@router.get("/ui/self-scoping")
|
||||
def self_scoping_index() -> HTMLResponse:
|
||||
return render_self_scoping_index()
|
||||
|
||||
|
||||
@router.get("/ui/self-scoping/review")
|
||||
def self_scoping_review(
|
||||
golden: str = Query(...),
|
||||
assessment: str = Query(...),
|
||||
saved: str | None = Query(default=None),
|
||||
) -> HTMLResponse:
|
||||
try:
|
||||
golden_profile = load_json_artifact(golden)
|
||||
assessment_artifact = load_json_artifact(assessment)
|
||||
except (FileNotFoundError, ValueError, json.JSONDecodeError) as exc:
|
||||
return render_self_scoping_index(error_message=str(exc), status_code=400)
|
||||
|
||||
comparison = compare_assessment_to_golden(golden_profile, assessment_artifact)
|
||||
comparison_status = comparison["status"]
|
||||
comparison_summary = comparison["summary"]
|
||||
matched_count = len(comparison["matched_expected_capabilities"])
|
||||
missing_count = len(comparison["missing_expected_capabilities"])
|
||||
forbidden_count = len(comparison["forbidden_native_capabilities_present"])
|
||||
misplaced_count = len(comparison["misplaced_features"])
|
||||
saved_notice = (
|
||||
f"""
|
||||
<div class="notice success">
|
||||
Saved assessment outcome <span class="source">{escape(saved)}</span>.
|
||||
</div>
|
||||
"""
|
||||
if saved
|
||||
else ""
|
||||
)
|
||||
body = f"""
|
||||
<h1>Self-Scoping Comparison</h1>
|
||||
{saved_notice}
|
||||
<section class="panel stack">
|
||||
<div class="actions">
|
||||
<a class="button secondary" href="/ui/self-scoping">Back</a>
|
||||
</div>
|
||||
<div class="notice {_comparison_notice_class(comparison)}">
|
||||
<strong>{escape(comparison_status)}</strong>
|
||||
<p>{escape(comparison_summary)}</p>
|
||||
</div>
|
||||
<div class="review-meta">
|
||||
<span class="pill">Matched {matched_count}</span>
|
||||
<span class="pill">Missing {missing_count}</span>
|
||||
<span class="pill">Forbidden {forbidden_count}</span>
|
||||
<span class="pill">Misplaced {misplaced_count}</span>
|
||||
</div>
|
||||
</section>
|
||||
<div class="review-grid">
|
||||
<section class="panel">
|
||||
<h2>Golden Profile</h2>
|
||||
{_render_golden_tree(golden_profile, comparison)}
|
||||
</section>
|
||||
<section class="panel">
|
||||
<h2>Assessment Output</h2>
|
||||
{_render_assessment_tree(assessment_artifact, comparison)}
|
||||
</section>
|
||||
</div>
|
||||
<section class="panel stack">
|
||||
<h2>Record Review Outcome</h2>
|
||||
<form class="stack" method="post" action="/ui/self-scoping/review">
|
||||
<input type="hidden" name="golden_path" value="{escape(golden)}">
|
||||
<input type="hidden" name="assessment_path" value="{escape(assessment)}">
|
||||
<input type="hidden" name="comparison_status" value="{escape(comparison_status)}">
|
||||
<label>Decision
|
||||
<select name="outcome" required>
|
||||
{_review_outcome_options()}
|
||||
</select>
|
||||
</label>
|
||||
<label>Reviewer <input name="reviewer" value="codex"></label>
|
||||
<label>Notes <textarea name="notes" rows="4"></textarea></label>
|
||||
<div class="actions">
|
||||
<button type="submit">Save outcome</button>
|
||||
<span data-pending>Saving outcome...</span>
|
||||
</div>
|
||||
</form>
|
||||
</section>
|
||||
"""
|
||||
return page("Self-Scoping Comparison", body)
|
||||
|
||||
|
||||
@router.post("/ui/self-scoping/review")
|
||||
def save_self_scoping_review(
|
||||
golden_path: str = Form(...),
|
||||
assessment_path: str = Form(...),
|
||||
outcome: str = Form(...),
|
||||
reviewer: str = Form("codex"),
|
||||
notes: str = Form(""),
|
||||
comparison_status: str = Form(""),
|
||||
):
|
||||
try:
|
||||
record = record_assessment_outcome(
|
||||
golden_path=golden_path,
|
||||
assessment_path=assessment_path,
|
||||
outcome=outcome,
|
||||
reviewer=reviewer,
|
||||
notes=notes,
|
||||
comparison_status=comparison_status,
|
||||
)
|
||||
except (FileNotFoundError, ValueError, json.JSONDecodeError) as exc:
|
||||
return render_self_scoping_index(error_message=str(exc), status_code=400)
|
||||
return RedirectResponse(
|
||||
(
|
||||
"/ui/self-scoping/review"
|
||||
f"?golden={quote_plus(golden_path)}"
|
||||
f"&assessment={quote_plus(assessment_path)}"
|
||||
f"&saved={quote_plus(record['outcome_id'])}"
|
||||
),
|
||||
status_code=303,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/ui/self-scoping/run-review")
|
||||
def self_scoping_run_review(
|
||||
baseline: str = Query(...),
|
||||
challenger: str = Query(...),
|
||||
saved: str | None = Query(default=None),
|
||||
) -> HTMLResponse:
|
||||
try:
|
||||
baseline_artifact = load_json_artifact(baseline)
|
||||
challenger_artifact = load_json_artifact(challenger)
|
||||
except (FileNotFoundError, ValueError, json.JSONDecodeError) as exc:
|
||||
return render_self_scoping_index(error_message=str(exc), status_code=400)
|
||||
|
||||
comparison = _assessment_tree_diff(baseline_artifact, challenger_artifact)
|
||||
comparison_status = comparison["status"]
|
||||
comparison_summary = comparison["summary"]
|
||||
shared_count = len(comparison["shared_capabilities"])
|
||||
baseline_only_count = len(comparison["baseline_only_capabilities"])
|
||||
challenger_only_count = len(comparison["challenger_only_capabilities"])
|
||||
moved_feature_count = len(comparison["moved_feature_names"])
|
||||
saved_notice = (
|
||||
f"""
|
||||
<div class="notice success">
|
||||
Saved assessment outcome <span class="source">{escape(saved)}</span>.
|
||||
</div>
|
||||
"""
|
||||
if saved
|
||||
else ""
|
||||
)
|
||||
body = f"""
|
||||
<h1>Assessment Run Comparison</h1>
|
||||
{saved_notice}
|
||||
<section class="panel stack">
|
||||
<div class="actions">
|
||||
<a class="button secondary" href="/ui/self-scoping">Back</a>
|
||||
</div>
|
||||
<div class="notice {_comparison_notice_class(comparison)}">
|
||||
<strong>{escape(comparison_status)}</strong>
|
||||
<p>{escape(comparison_summary)}</p>
|
||||
</div>
|
||||
<div class="review-meta">
|
||||
<span class="pill">Shared {shared_count}</span>
|
||||
<span class="pill">Baseline only {baseline_only_count}</span>
|
||||
<span class="pill">Challenger only {challenger_only_count}</span>
|
||||
<span class="pill">Moved features {moved_feature_count}</span>
|
||||
</div>
|
||||
</section>
|
||||
<div class="review-grid">
|
||||
<section class="panel">
|
||||
<h2>Baseline Run</h2>
|
||||
{_render_assessment_tree_for_run_diff(baseline_artifact, comparison, role="baseline")}
|
||||
</section>
|
||||
<section class="panel">
|
||||
<h2>Challenger Run</h2>
|
||||
{_render_assessment_tree_for_run_diff(challenger_artifact, comparison, role="challenger")}
|
||||
</section>
|
||||
</div>
|
||||
<section class="panel stack">
|
||||
<h2>Record Review Outcome</h2>
|
||||
<form class="stack" method="post" action="/ui/self-scoping/run-review">
|
||||
<input type="hidden" name="baseline_path" value="{escape(baseline)}">
|
||||
<input type="hidden" name="challenger_path" value="{escape(challenger)}">
|
||||
<input type="hidden" name="comparison_status" value="{escape(comparison_status)}">
|
||||
<label>Decision
|
||||
<select name="outcome" required>
|
||||
{_pair_review_outcome_options()}
|
||||
</select>
|
||||
</label>
|
||||
<label>Reviewer <input name="reviewer" value="codex"></label>
|
||||
<label>Notes <textarea name="notes" rows="4"></textarea></label>
|
||||
<div class="actions">
|
||||
<button type="submit">Save outcome</button>
|
||||
<span data-pending>Saving outcome...</span>
|
||||
</div>
|
||||
</form>
|
||||
</section>
|
||||
"""
|
||||
return page("Assessment Run Comparison", body)
|
||||
|
||||
|
||||
@router.post("/ui/self-scoping/run-review")
|
||||
def save_self_scoping_run_review(
|
||||
baseline_path: str = Form(...),
|
||||
challenger_path: str = Form(...),
|
||||
outcome: str = Form(...),
|
||||
reviewer: str = Form("codex"),
|
||||
notes: str = Form(""),
|
||||
comparison_status: str = Form(""),
|
||||
):
|
||||
try:
|
||||
record = record_assessment_pair_outcome(
|
||||
baseline_path=baseline_path,
|
||||
challenger_path=challenger_path,
|
||||
outcome=outcome,
|
||||
reviewer=reviewer,
|
||||
notes=notes,
|
||||
comparison_status=comparison_status,
|
||||
)
|
||||
except (FileNotFoundError, ValueError, json.JSONDecodeError) as exc:
|
||||
return render_self_scoping_index(error_message=str(exc), status_code=400)
|
||||
return RedirectResponse(
|
||||
(
|
||||
"/ui/self-scoping/run-review"
|
||||
f"?baseline={quote_plus(baseline_path)}"
|
||||
f"&challenger={quote_plus(challenger_path)}"
|
||||
f"&saved={quote_plus(record['outcome_id'])}"
|
||||
),
|
||||
status_code=303,
|
||||
)
|
||||
|
||||
|
||||
def _assessment_tree_diff(baseline: dict, challenger: dict) -> dict:
|
||||
baseline_capabilities = set(_assessment_capability_names(baseline))
|
||||
challenger_capabilities = set(_assessment_capability_names(challenger))
|
||||
baseline_only = sorted(baseline_capabilities - challenger_capabilities)
|
||||
challenger_only = sorted(challenger_capabilities - baseline_capabilities)
|
||||
shared = sorted(baseline_capabilities & challenger_capabilities)
|
||||
|
||||
baseline_feature_index = _assessment_feature_index(baseline)
|
||||
challenger_feature_index = _assessment_feature_index(challenger)
|
||||
moved_feature_names = sorted(
|
||||
feature_name
|
||||
for feature_name in set(baseline_feature_index) & set(challenger_feature_index)
|
||||
if baseline_feature_index[feature_name] != challenger_feature_index[feature_name]
|
||||
)
|
||||
baseline_moved_pairs = {
|
||||
(capability_name, feature_name)
|
||||
for feature_name in moved_feature_names
|
||||
for capability_name in baseline_feature_index[feature_name]
|
||||
}
|
||||
challenger_moved_pairs = {
|
||||
(capability_name, feature_name)
|
||||
for feature_name in moved_feature_names
|
||||
for capability_name in challenger_feature_index[feature_name]
|
||||
}
|
||||
status = (
|
||||
"candidate_improvement"
|
||||
if not baseline_only and not challenger_only and not moved_feature_names
|
||||
else "needs_review"
|
||||
)
|
||||
return {
|
||||
"status": status,
|
||||
"summary": _assessment_tree_diff_summary(
|
||||
baseline_only,
|
||||
challenger_only,
|
||||
moved_feature_names,
|
||||
),
|
||||
"baseline_only_capabilities": baseline_only,
|
||||
"challenger_only_capabilities": challenger_only,
|
||||
"shared_capabilities": shared,
|
||||
"moved_feature_names": moved_feature_names,
|
||||
"baseline_moved_feature_pairs": baseline_moved_pairs,
|
||||
"challenger_moved_feature_pairs": challenger_moved_pairs,
|
||||
}
|
||||
|
||||
|
||||
def _assessment_tree_diff_summary(
|
||||
baseline_only: list[str],
|
||||
challenger_only: list[str],
|
||||
moved_feature_names: list[str],
|
||||
) -> str:
|
||||
if not baseline_only and not challenger_only and not moved_feature_names:
|
||||
return "Assessment hierarchy names match between baseline and challenger."
|
||||
return (
|
||||
"Assessment runs differ: "
|
||||
f"{len(baseline_only)} baseline-only capability(s), "
|
||||
f"{len(challenger_only)} challenger-only capability(s), and "
|
||||
f"{len(moved_feature_names)} moved feature name(s)."
|
||||
)
|
||||
|
||||
|
||||
def _assessment_capability_names(assessment: dict) -> list[str]:
|
||||
names: list[str] = []
|
||||
for ability in assessment.get("generated_tree", {}).get("abilities", []):
|
||||
for capability in ability.get("capabilities", []):
|
||||
name = capability.get("name")
|
||||
if name:
|
||||
names.append(name)
|
||||
return names
|
||||
|
||||
|
||||
def _assessment_feature_index(assessment: dict) -> dict[str, set[str]]:
|
||||
index: dict[str, set[str]] = {}
|
||||
for ability in assessment.get("generated_tree", {}).get("abilities", []):
|
||||
for capability in ability.get("capabilities", []):
|
||||
capability_name = capability.get("name", "")
|
||||
for feature in capability.get("features", []):
|
||||
feature_name = feature.get("name")
|
||||
if feature_name:
|
||||
index.setdefault(feature_name, set()).add(capability_name)
|
||||
return index
|
||||
|
||||
|
||||
def _render_assessment_tree_for_run_diff(
|
||||
assessment: dict,
|
||||
comparison: dict,
|
||||
*,
|
||||
role: str,
|
||||
) -> str:
|
||||
if role == "baseline":
|
||||
changed_capabilities = set(comparison["baseline_only_capabilities"])
|
||||
moved_pairs = comparison["baseline_moved_feature_pairs"]
|
||||
changed_reason = "Baseline only"
|
||||
else:
|
||||
changed_capabilities = set(comparison["challenger_only_capabilities"])
|
||||
moved_pairs = comparison["challenger_moved_feature_pairs"]
|
||||
changed_reason = "Challenger only"
|
||||
shared = set(comparison["shared_capabilities"])
|
||||
ability_blocks = []
|
||||
for ability in assessment.get("generated_tree", {}).get("abilities", []):
|
||||
capability_blocks = []
|
||||
for capability in ability.get("capabilities", []):
|
||||
name = capability.get("name", "")
|
||||
item_class = "warn" if name in changed_capabilities else "match" if name in shared else ""
|
||||
reason = changed_reason if name in changed_capabilities else "Shared capability"
|
||||
capability_blocks.append(
|
||||
f"""
|
||||
<article class="review-item {item_class}">
|
||||
<h3>{escape(name)}</h3>
|
||||
<div class="review-meta">
|
||||
<span class="pill">{escape(reason)}</span>
|
||||
<span class="pill">{escape(capability.get("primary_class", ""))}</span>
|
||||
</div>
|
||||
{_render_generated_features(name, capability.get("features", []), moved_pairs)}
|
||||
</article>
|
||||
"""
|
||||
)
|
||||
ability_blocks.append(
|
||||
f"""
|
||||
<section class="stack">
|
||||
<h3>{escape(ability.get("name", ""))}</h3>
|
||||
{"".join(capability_blocks) or '<p class="muted">No capabilities generated.</p>'}
|
||||
</section>
|
||||
"""
|
||||
)
|
||||
return "\n".join(ability_blocks) or '<p class="muted">No generated abilities found.</p>'
|
||||
|
||||
|
||||
def _review_artifact_options(artifacts) -> str:
|
||||
if not artifacts:
|
||||
return '<option value="">No artifacts found</option>'
|
||||
return "\n".join(
|
||||
f"""
|
||||
<option value="{escape(artifact.path)}">
|
||||
{escape(artifact.artifact_id)} · {escape(artifact.updated_at or artifact.title)}
|
||||
</option>
|
||||
"""
|
||||
for artifact in artifacts
|
||||
)
|
||||
|
||||
|
||||
def _render_outcome_table(outcomes: list[dict]) -> str:
|
||||
if not outcomes:
|
||||
return '<p class="muted">No review outcomes have been recorded yet.</p>'
|
||||
rows = "\n".join(
|
||||
f"""
|
||||
<tr>
|
||||
<td>{escape(record.get("created_at", ""))}</td>
|
||||
<td><span class="pill">{escape(record.get("outcome", ""))}</span></td>
|
||||
<td>{escape(record.get("comparison_status", ""))}</td>
|
||||
<td class="source">{escape(_outcome_record_subject(record))}</td>
|
||||
</tr>
|
||||
"""
|
||||
for record in outcomes[:8]
|
||||
)
|
||||
return f"""
|
||||
<table>
|
||||
<thead><tr><th>Created</th><th>Outcome</th><th>Status</th><th>Assessment</th></tr></thead>
|
||||
<tbody>{rows}</tbody>
|
||||
</table>
|
||||
"""
|
||||
|
||||
|
||||
def _outcome_record_subject(record: dict) -> str:
|
||||
if record.get("assessment_artifact_id"):
|
||||
return str(record["assessment_artifact_id"])
|
||||
if record.get("challenger_assessment_artifact_id"):
|
||||
return str(record["challenger_assessment_artifact_id"])
|
||||
if record.get("outcome_id"):
|
||||
return str(record["outcome_id"])
|
||||
return ""
|
||||
|
||||
|
||||
def _comparison_notice_class(comparison: dict) -> str:
|
||||
if comparison["status"] == "regression":
|
||||
return "error"
|
||||
if comparison["status"] == "needs_review":
|
||||
return "warn"
|
||||
return "success"
|
||||
|
||||
|
||||
def _render_golden_tree(golden_profile: dict, comparison: dict) -> str:
|
||||
missing = set(comparison["missing_expected_capabilities"])
|
||||
matched = set(comparison["matched_expected_capabilities"])
|
||||
capabilities = golden_profile.get("ability", {}).get("expected_capabilities", [])
|
||||
items = []
|
||||
for capability in capabilities:
|
||||
name = capability.get("name", "")
|
||||
item_class = "problem" if name in missing else "match" if name in matched else ""
|
||||
features = _render_expected_features(capability.get("expected_features", []))
|
||||
state = "Missing" if name in missing else "Matched" if name in matched else "Expected"
|
||||
items.append(
|
||||
f"""
|
||||
<article class="review-item {item_class}">
|
||||
<h3>{escape(name)}</h3>
|
||||
<div class="review-meta">
|
||||
<span class="pill">{escape(state)}</span>
|
||||
<span class="pill">{escape(capability.get("primary_class", ""))}</span>
|
||||
</div>
|
||||
{features}
|
||||
</article>
|
||||
"""
|
||||
)
|
||||
return "\n".join(items) or '<p class="muted">No expected capabilities found.</p>'
|
||||
|
||||
|
||||
def _render_expected_features(features: list[dict]) -> str:
|
||||
if not features:
|
||||
return ""
|
||||
rows = []
|
||||
for feature in features:
|
||||
sources = ", ".join(feature.get("source_paths", [])[:3])
|
||||
rows.append(
|
||||
f"""
|
||||
<li>
|
||||
{escape(feature.get("name", ""))}
|
||||
<span class="pill">{escape(feature.get("primary_class", ""))}</span>
|
||||
<div class="source">{escape(sources)}</div>
|
||||
</li>
|
||||
"""
|
||||
)
|
||||
return f'<ul class="review-list">{"".join(rows)}</ul>'
|
||||
|
||||
|
||||
def _render_assessment_tree(assessment: dict, comparison: dict) -> str:
|
||||
forbidden = set(comparison["forbidden_native_capabilities_present"])
|
||||
unexpected = set(comparison["unexpected_native_capabilities"])
|
||||
misplaced = {
|
||||
(item.get("capability", ""), item.get("feature", ""))
|
||||
for item in comparison["misplaced_features"]
|
||||
}
|
||||
abilities = assessment.get("generated_tree", {}).get("abilities", [])
|
||||
ability_blocks = []
|
||||
for ability in abilities:
|
||||
capabilities = ability.get("capabilities", [])
|
||||
capability_blocks = []
|
||||
for capability in capabilities:
|
||||
name = capability.get("name", "")
|
||||
item_class = "problem" if name in forbidden else "warn" if name in unexpected else ""
|
||||
reason = (
|
||||
"Forbidden native capability"
|
||||
if name in forbidden
|
||||
else "Unexpected native capability"
|
||||
if name in unexpected
|
||||
else "Generated capability"
|
||||
)
|
||||
capability_blocks.append(
|
||||
f"""
|
||||
<article class="review-item {item_class}">
|
||||
<h3>{escape(name)}</h3>
|
||||
<div class="review-meta">
|
||||
<span class="pill">{escape(reason)}</span>
|
||||
<span class="pill">{escape(capability.get("primary_class", ""))}</span>
|
||||
</div>
|
||||
{_render_generated_features(name, capability.get("features", []), misplaced)}
|
||||
</article>
|
||||
"""
|
||||
)
|
||||
ability_blocks.append(
|
||||
f"""
|
||||
<section class="stack">
|
||||
<h3>{escape(ability.get("name", ""))}</h3>
|
||||
{"".join(capability_blocks) or '<p class="muted">No capabilities generated.</p>'}
|
||||
</section>
|
||||
"""
|
||||
)
|
||||
return "\n".join(ability_blocks) or '<p class="muted">No generated abilities found.</p>'
|
||||
|
||||
|
||||
def _render_generated_features(
|
||||
capability_name: str,
|
||||
features: list[dict],
|
||||
misplaced: set[tuple[str, str]],
|
||||
) -> str:
|
||||
if not features:
|
||||
return ""
|
||||
rows = []
|
||||
for feature in features:
|
||||
feature_name = feature.get("name", "")
|
||||
feature_class = "warn" if (capability_name, feature_name) in misplaced else ""
|
||||
rows.append(
|
||||
f"""
|
||||
<li class="{feature_class}">
|
||||
{escape(feature_name)}
|
||||
<span class="pill">{escape(feature.get("type") or feature.get("primary_class") or "")}</span>
|
||||
<div class="source">{escape(feature.get("location", ""))}</div>
|
||||
</li>
|
||||
"""
|
||||
)
|
||||
return f'<ul class="review-list">{"".join(rows)}</ul>'
|
||||
|
||||
|
||||
def _review_outcome_options() -> str:
|
||||
return "\n".join(
|
||||
f'<option value="{escape(value)}">{escape(REVIEW_OUTCOME_LABELS[value])}</option>'
|
||||
for value in REVIEW_OUTCOME_LABELS
|
||||
if value in ALLOWED_OUTCOMES
|
||||
)
|
||||
|
||||
|
||||
def _pair_review_outcome_options() -> str:
|
||||
return "\n".join(
|
||||
f'<option value="{escape(value)}">{escape(PAIR_REVIEW_OUTCOME_LABELS[value])}</option>'
|
||||
for value in PAIR_REVIEW_OUTCOME_LABELS
|
||||
if value in ALLOWED_OUTCOMES
|
||||
)
|
||||
|
||||
|
||||
@router.get("/ui/repos/{repository_id}/scope")
|
||||
def repository_scope_document(
|
||||
repository_id: int,
|
||||
@@ -443,6 +1104,61 @@ def repository_scope_document(
|
||||
)
|
||||
|
||||
|
||||
@router.get("/ui/repos/{repository_id}/intent-review")
|
||||
def repository_intent_review(
|
||||
repository_id: int,
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> HTMLResponse:
|
||||
return repository_document_review_page(repository_id, "INTENT.md", service)
|
||||
|
||||
|
||||
@router.get("/ui/repos/{repository_id}/scope-review")
|
||||
def repository_scope_review(
|
||||
repository_id: int,
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> HTMLResponse:
|
||||
return repository_document_review_page(repository_id, "SCOPE.md", service)
|
||||
|
||||
|
||||
def repository_document_review_page(
|
||||
repository_id: int,
|
||||
document_name: str,
|
||||
service: RegistryService,
|
||||
) -> HTMLResponse:
|
||||
payload = service.document_review(repository_id, document_name)
|
||||
repository = service.get_repository(repository_id)
|
||||
display_name = repository_display_name(repository)
|
||||
current = str(payload.get("current_content") or "")
|
||||
draft = str(payload.get("draft_content") or "")
|
||||
provenance = payload.get("provenance") or {}
|
||||
body = f"""
|
||||
<div class="actions">
|
||||
<h1 style="margin-right:auto">{escape(document_name)} Review</h1>
|
||||
<a class="button secondary" href="/ui/repos/{repository_id}">Repository</a>
|
||||
</div>
|
||||
<section class="panel stack">
|
||||
<p class="muted">{escape(str(payload.get("write_policy", "")))}</p>
|
||||
<p><span class="pill">{'exists' if payload.get("exists") else 'missing'}</span>
|
||||
<span class="source">{escape(str(payload.get("path", "")))}</span></p>
|
||||
<label>Current {escape(document_name)}
|
||||
<textarea rows="14" spellcheck="false">{escape(current)}</textarea>
|
||||
</label>
|
||||
<label>Draft {escape(document_name)}
|
||||
<textarea rows="18" spellcheck="false">{escape(draft)}</textarea>
|
||||
</label>
|
||||
<p class="muted">analysis run {escape(str(provenance.get("analysis_run_id", "")))} ·
|
||||
{escape(str(provenance.get("fact_count", 0)))} facts ·
|
||||
{escape(str((provenance.get("candidate_counts") or {}).get("capabilities", 0)))} candidate capabilities</p>
|
||||
</section>
|
||||
"""
|
||||
return page(
|
||||
f"{document_name} Review",
|
||||
body,
|
||||
selected_repository=display_name,
|
||||
selected_repository_id=repository.id,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/ui/discovery")
|
||||
def discovery_page(service: RegistryService = Depends(get_service)) -> HTMLResponse:
|
||||
repositories = service.list_repositories()
|
||||
@@ -783,7 +1499,7 @@ def create_repository_from_form(
|
||||
access_password: str = Form(""),
|
||||
explore_after_registration: str | None = Form(None),
|
||||
use_llm_assistance: str | None = Form(None),
|
||||
trusted_auto_approve: str | None = Form(None),
|
||||
agentic_review: str | None = Form(None),
|
||||
service: RegistryService = Depends(get_service),
|
||||
):
|
||||
try:
|
||||
@@ -803,7 +1519,7 @@ def create_repository_from_form(
|
||||
summary = service.analyze_repository(
|
||||
repository.id,
|
||||
use_llm_assistance=bool(use_llm_assistance),
|
||||
trusted_auto_approve=bool(trusted_auto_approve),
|
||||
agentic_review=bool(agentic_review),
|
||||
access_username=access_username or None,
|
||||
access_password=access_password or None,
|
||||
)
|
||||
@@ -853,6 +1569,8 @@ def repository_detail(
|
||||
<a class="button secondary" href="/ui/repos/{repository_id}/dependency-graph">Dependency Graph</a>
|
||||
<a class="button secondary" href="/ui/repos/{repository_id}/export">Export</a>
|
||||
<a class="button secondary" href="/ui/repos/{repository_id}/scope">SCOPE</a>
|
||||
<a class="button secondary" href="/ui/repos/{repository_id}/scope-review">Scope Draft</a>
|
||||
<a class="button secondary" href="/ui/repos/{repository_id}/intent-review">Intent Draft</a>
|
||||
<a class="button secondary" href="/ui">Back</a>
|
||||
</div>
|
||||
<p class="muted">{escape(repository.description or '')}</p>
|
||||
@@ -872,7 +1590,7 @@ def repository_detail(
|
||||
<label>Override source path <input name="source_path" placeholder="Optional local path"></label>
|
||||
<label class="checkbox"><input type="checkbox" name="use_cached_checkout" value="1"> Analyze cached checkout without fetching upstream</label>
|
||||
<label class="checkbox"><input type="checkbox" name="use_llm_assistance" value="1" checked> Use LLM assistance if configured</label>
|
||||
<label class="checkbox"><input type="checkbox" name="trusted_auto_approve" value="1"> Trusted auto-populate after analysis</label>
|
||||
<label class="checkbox"><input type="checkbox" name="agentic_review" value="1"> Request agentic review after analysis</label>
|
||||
<label>Username <input name="access_username" autocomplete="username" placeholder="Optional for private HTTP(S) repos"></label>
|
||||
<label>Password or access token <input name="access_password" type="password" autocomplete="current-password" placeholder="Used for this Git operation only"></label>
|
||||
<div class="actions">
|
||||
@@ -1307,7 +2025,7 @@ def create_analysis_run_from_form(
|
||||
source_path: str = Form(""),
|
||||
use_cached_checkout: str | None = Form(None),
|
||||
use_llm_assistance: str | None = Form(None),
|
||||
trusted_auto_approve: str | None = Form(None),
|
||||
agentic_review: str | None = Form(None),
|
||||
access_username: str = Form(""),
|
||||
access_password: str = Form(""),
|
||||
service: RegistryService = Depends(get_service),
|
||||
@@ -1317,7 +2035,7 @@ def create_analysis_run_from_form(
|
||||
source_path=source_path or None,
|
||||
use_cached_checkout=bool(use_cached_checkout),
|
||||
use_llm_assistance=bool(use_llm_assistance),
|
||||
trusted_auto_approve=bool(trusted_auto_approve),
|
||||
agentic_review=bool(agentic_review),
|
||||
access_username=access_username or None,
|
||||
access_password=access_password or None,
|
||||
)
|
||||
@@ -1361,6 +2079,10 @@ def analysis_run_detail(
|
||||
display_name = repository_display_name(repository)
|
||||
candidate_graph = service.candidate_graph(repository_id, analysis_run_id)
|
||||
candidate_graph_data = asdict(candidate_graph)
|
||||
quality_gate_outcomes = quality_gate_outcome_dicts(
|
||||
evaluate_candidate_graph_quality(candidate_graph)
|
||||
)
|
||||
candidate_graph_data["quality_gate_outcomes"] = quality_gate_outcomes
|
||||
facts = service.list_observed_facts(repository_id, analysis_run_id)
|
||||
chunks = service.list_content_chunks(repository_id, analysis_run_id)
|
||||
decisions = service.list_review_decisions(repository_id, analysis_run_id)
|
||||
@@ -1413,6 +2135,12 @@ def analysis_run_detail(
|
||||
</form>
|
||||
</div>
|
||||
{render_candidate_graph(candidate_graph_data, repository_id, analysis_run_id)}
|
||||
<h2>Quality Gate Outcomes</h2>
|
||||
{render_quality_gate_outcomes(
|
||||
quality_gate_outcomes,
|
||||
repository_id=repository_id,
|
||||
analysis_run_id=analysis_run_id,
|
||||
)}
|
||||
</section>
|
||||
<section class="panel">
|
||||
<div class="actions">
|
||||
@@ -1482,6 +2210,32 @@ def create_expectation_gap_from_form(
|
||||
)
|
||||
|
||||
|
||||
@router.post("/ui/repos/{repository_id}/analysis-runs/{analysis_run_id}/quality-gate-overrides")
|
||||
def create_quality_gate_override_from_form(
|
||||
repository_id: int,
|
||||
analysis_run_id: int,
|
||||
criterion_id: str = Form(...),
|
||||
element_type: str = Form(...),
|
||||
element_id: int = Form(...),
|
||||
reason: str = Form(...),
|
||||
notes: str = Form(""),
|
||||
service: RegistryService = Depends(get_service),
|
||||
) -> RedirectResponse:
|
||||
service.record_quality_gate_override(
|
||||
repository_id,
|
||||
analysis_run_id,
|
||||
criterion_id=criterion_id,
|
||||
element_type=element_type,
|
||||
element_id=element_id,
|
||||
reason=reason,
|
||||
notes=notes,
|
||||
)
|
||||
return RedirectResponse(
|
||||
f"/ui/repos/{repository_id}/analysis-runs/{analysis_run_id}",
|
||||
status_code=303,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/ui/repos/{repository_id}/expectation-gaps")
|
||||
def create_repository_expectation_gap_from_form(
|
||||
repository_id: int,
|
||||
@@ -4074,6 +4828,45 @@ def render_review_decisions(decisions: list) -> str:
|
||||
"""
|
||||
|
||||
|
||||
def render_quality_gate_outcomes(
|
||||
outcomes: list[dict],
|
||||
*,
|
||||
repository_id: int,
|
||||
analysis_run_id: int,
|
||||
) -> str:
|
||||
if not outcomes:
|
||||
return '<p class="muted">No quality gates fired for this run.</p>'
|
||||
rows = "\n".join(
|
||||
f"""
|
||||
<tr>
|
||||
<td><span class="pill">{escape(outcome["criterion_id"])}</span></td>
|
||||
<td>{escape(outcome["outcome"])}</td>
|
||||
<td>{escape(outcome["element_type"])} #{outcome["element_id"]}</td>
|
||||
<td>{escape(outcome["reason"])}</td>
|
||||
<td>
|
||||
<form class="stack" method="post" action="/ui/repos/{repository_id}/analysis-runs/{analysis_run_id}/quality-gate-overrides">
|
||||
<input type="hidden" name="criterion_id" value="{escape(outcome["criterion_id"])}">
|
||||
<input type="hidden" name="element_type" value="{escape(outcome["element_type"])}">
|
||||
<input type="hidden" name="element_id" value="{outcome["element_id"]}">
|
||||
<input name="reason" required placeholder="Override reason">
|
||||
<input name="notes" placeholder="Optional notes">
|
||||
<button class="secondary" type="submit">Override</button>
|
||||
</form>
|
||||
</td>
|
||||
</tr>
|
||||
"""
|
||||
for outcome in outcomes
|
||||
)
|
||||
return f"""
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>Criterion</th><th>Outcome</th><th>Element</th><th>Reason</th><th>Override</th></tr>
|
||||
</thead>
|
||||
<tbody>{rows}</tbody>
|
||||
</table>
|
||||
"""
|
||||
|
||||
|
||||
def render_expectation_gap_form(
|
||||
*,
|
||||
action: str,
|
||||
136
tests/test_acceptance_boundary.py
Normal file
136
tests/test_acceptance_boundary.py
Normal file
@@ -0,0 +1,136 @@
|
||||
from repo_scoping.acceptance import AgenticReviewDecision
|
||||
from repo_scoping.core.service import RegistryService
|
||||
from repo_scoping.repo_ingestion.git import GitIngestionService
|
||||
from repo_scoping.storage.sqlite import RegistryStore
|
||||
|
||||
|
||||
class BoundaryApprovingReviewer:
|
||||
reviewer_id = "boundary-agent"
|
||||
policy_version = "agentic-review-policy/test"
|
||||
|
||||
def review(self, request):
|
||||
return [
|
||||
AgenticReviewDecision(
|
||||
action="approve",
|
||||
target_type="candidate_graph",
|
||||
target_id=request.candidate_graph.analysis_run.id,
|
||||
rationale="README and source refs support the generated API capability.",
|
||||
criterion_ids=["RREG-QC-004"],
|
||||
evidence_refs=["README.md", "app.py"],
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def make_service(tmp_path, *, reviewer=None):
|
||||
store = RegistryStore(tmp_path / "registry.sqlite3")
|
||||
store.initialize()
|
||||
return RegistryService(
|
||||
store,
|
||||
ingestion=GitIngestionService(tmp_path / "checkouts"),
|
||||
agentic_reviewer=reviewer,
|
||||
)
|
||||
|
||||
|
||||
def write_api_repo(tmp_path):
|
||||
source = tmp_path / "api-repo"
|
||||
source.mkdir()
|
||||
(source / "README.md").write_text("# API Repo\nReports health.\n", encoding="utf-8")
|
||||
(source / "app.py").write_text('@app.get("/health")\ndef health():\n return {}\n', encoding="utf-8")
|
||||
return source
|
||||
|
||||
|
||||
def write_provider_repo(tmp_path):
|
||||
source = tmp_path / "provider-repo"
|
||||
source.mkdir()
|
||||
(source / "README.md").write_text("# Provider Repo\n", encoding="utf-8")
|
||||
(source / "providers.py").write_text(
|
||||
"provider_registry = {'openrouter': OpenRouterAdapter}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return source
|
||||
|
||||
|
||||
def test_deterministic_analysis_leaves_candidates_pending(tmp_path):
|
||||
service = make_service(tmp_path)
|
||||
repository = service.register_repository(
|
||||
name="API Repo",
|
||||
url=str(write_api_repo(tmp_path)),
|
||||
)
|
||||
|
||||
summary = service.analyze_repository(repository.id, use_llm_assistance=False)
|
||||
|
||||
graph = service.candidate_graph(repository.id, summary.analysis_run.id)
|
||||
assert service.ability_map(repository.id).abilities == []
|
||||
assert {
|
||||
capability.status
|
||||
for ability in graph.abilities
|
||||
for capability in ability.capabilities
|
||||
} == {"candidate"}
|
||||
|
||||
|
||||
def test_deterministic_gates_flag_provider_regression_without_approval(tmp_path):
|
||||
service = make_service(tmp_path)
|
||||
repository = service.register_repository(
|
||||
name="Provider Repo",
|
||||
url=str(write_provider_repo(tmp_path)),
|
||||
)
|
||||
|
||||
summary = service.analyze_repository(repository.id, use_llm_assistance=False)
|
||||
|
||||
graph = service.candidate_graph(repository.id, summary.analysis_run.id)
|
||||
decisions = service.list_review_decisions(repository.id, summary.analysis_run.id)
|
||||
assert service.ability_map(repository.id).abilities == []
|
||||
assert graph.abilities[0].capabilities[0].status == "candidate"
|
||||
assert any(
|
||||
decision.action == "quality_gate_evaluation"
|
||||
and "RREG-QC-002" in decision.criterion_ids
|
||||
for decision in decisions
|
||||
)
|
||||
|
||||
|
||||
def test_agentic_review_is_only_automated_approval_path(tmp_path):
|
||||
service = make_service(tmp_path, reviewer=BoundaryApprovingReviewer())
|
||||
repository = service.register_repository(
|
||||
name="Agent Approved Repo",
|
||||
url=str(write_api_repo(tmp_path)),
|
||||
)
|
||||
|
||||
summary = service.analyze_repository(
|
||||
repository.id,
|
||||
use_llm_assistance=False,
|
||||
agentic_review=True,
|
||||
)
|
||||
|
||||
decisions = service.list_review_decisions(repository.id, summary.analysis_run.id)
|
||||
assert service.ability_map(repository.id).abilities
|
||||
assert any(
|
||||
decision.action == "agentic_approve_candidate_graph"
|
||||
and decision.reviewer_type == "agent"
|
||||
and decision.rationale
|
||||
and decision.criteria_version == "repo-scoping-quality-criteria/v1"
|
||||
and decision.evidence_refs == ["README.md", "app.py"]
|
||||
for decision in decisions
|
||||
)
|
||||
|
||||
|
||||
def test_manual_approval_path_still_works(tmp_path):
|
||||
service = make_service(tmp_path)
|
||||
repository = service.register_repository(
|
||||
name="Manual Review Repo",
|
||||
url=str(write_api_repo(tmp_path)),
|
||||
)
|
||||
summary = service.analyze_repository(repository.id, use_llm_assistance=False)
|
||||
|
||||
service.approve_candidate_graph(
|
||||
repository.id,
|
||||
summary.analysis_run.id,
|
||||
notes="Manual curator approval.",
|
||||
)
|
||||
|
||||
decisions = service.list_review_decisions(repository.id, summary.analysis_run.id)
|
||||
assert service.ability_map(repository.id).abilities
|
||||
assert any(
|
||||
decision.action == "approve_candidate_graph"
|
||||
and decision.reviewer_type == "human"
|
||||
for decision in decisions
|
||||
)
|
||||
17
tests/test_acceptance_policy_docs.py
Normal file
17
tests/test_acceptance_policy_docs.py
Normal file
@@ -0,0 +1,17 @@
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
POLICY = ROOT / "docs" / "acceptance-policy.md"
|
||||
|
||||
|
||||
def test_acceptance_policy_defines_deterministic_boundary():
|
||||
text = POLICY.read_text(encoding="utf-8")
|
||||
|
||||
assert "Policy version: `acceptance-policy/v1`" in text
|
||||
assert "Deterministic quality gates must not approve" in text
|
||||
assert "`requires_review`" in text
|
||||
assert "`invalidated`" in text
|
||||
assert "`approve_with_edits`" in text
|
||||
assert "`trusted_auto_approve_candidate_graph`" in text
|
||||
assert "legacy terminology" in text
|
||||
139
tests/test_agentic_review.py
Normal file
139
tests/test_agentic_review.py
Normal file
@@ -0,0 +1,139 @@
|
||||
import pytest
|
||||
|
||||
from repo_scoping.acceptance import (
|
||||
AgenticReviewDecision,
|
||||
validate_agentic_review_decision,
|
||||
)
|
||||
from repo_scoping.core.service import RegistryService
|
||||
from repo_scoping.repo_ingestion.git import GitIngestionService
|
||||
from repo_scoping.storage.sqlite import RegistryStore
|
||||
|
||||
|
||||
class RecordingAgenticReviewer:
|
||||
reviewer_id = "test-agent"
|
||||
policy_version = "agentic-review-policy/test"
|
||||
|
||||
def __init__(self):
|
||||
self.requests = []
|
||||
|
||||
def review(self, request):
|
||||
self.requests.append(request)
|
||||
return []
|
||||
|
||||
|
||||
class ApprovingAgenticReviewer:
|
||||
reviewer_id = "approving-agent"
|
||||
policy_version = "agentic-review-policy/test"
|
||||
|
||||
def __init__(self):
|
||||
self.requests = []
|
||||
|
||||
def review(self, request):
|
||||
self.requests.append(request)
|
||||
graph = request.candidate_graph
|
||||
return [
|
||||
AgenticReviewDecision(
|
||||
action="approve",
|
||||
target_type="candidate_graph",
|
||||
target_id=graph.analysis_run.id,
|
||||
rationale="API source and README support the generated repository interface claim.",
|
||||
criterion_ids=["RREG-QC-004"],
|
||||
evidence_refs=["README.md", "app.py"],
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_configured_agentic_reviewer_receives_graph_gates_and_criteria(tmp_path):
|
||||
source = tmp_path / "repo"
|
||||
source.mkdir()
|
||||
(source / "README.md").write_text("# Agentic Review\nReports health.\n", encoding="utf-8")
|
||||
(source / "app.py").write_text('@app.get("/health")\ndef health():\n return {}\n', encoding="utf-8")
|
||||
store = RegistryStore(tmp_path / "registry.sqlite3")
|
||||
store.initialize()
|
||||
reviewer = RecordingAgenticReviewer()
|
||||
service = RegistryService(
|
||||
store,
|
||||
ingestion=GitIngestionService(tmp_path / "checkouts"),
|
||||
agentic_reviewer=reviewer,
|
||||
)
|
||||
repository = service.register_repository(name="Agentic Review", url=str(source))
|
||||
|
||||
summary = service.analyze_repository(
|
||||
repository.id,
|
||||
use_llm_assistance=False,
|
||||
agentic_review=True,
|
||||
)
|
||||
|
||||
graph = service.candidate_graph(repository.id, summary.analysis_run.id)
|
||||
decisions = service.list_review_decisions(repository.id, summary.analysis_run.id)
|
||||
assert len(reviewer.requests) == 1
|
||||
request = reviewer.requests[0]
|
||||
assert request.repository.id == repository.id
|
||||
assert request.candidate_graph.analysis_run.id == summary.analysis_run.id
|
||||
assert request.criteria_version == "repo-scoping-quality-criteria/v1"
|
||||
assert request.quality_gate_outcomes == []
|
||||
assert graph.abilities[0].capabilities[0].status == "candidate"
|
||||
assert decisions[0].action == "agentic_review_completed"
|
||||
assert "reviewer=test-agent" in decisions[0].notes
|
||||
assert "decisions=0" in decisions[0].notes
|
||||
|
||||
|
||||
def test_agentic_approval_requires_rationale_criteria_and_evidence():
|
||||
with pytest.raises(ValueError, match="evidence refs"):
|
||||
validate_agentic_review_decision(
|
||||
AgenticReviewDecision(
|
||||
action="approve",
|
||||
target_type="candidate_graph",
|
||||
target_id=1,
|
||||
rationale="Looks supported.",
|
||||
criterion_ids=["RREG-QC-004"],
|
||||
evidence_refs=[],
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_agentic_reviewer_can_approve_candidate_graph_with_rationale(tmp_path):
|
||||
source = tmp_path / "repo"
|
||||
source.mkdir()
|
||||
(source / "README.md").write_text(
|
||||
"# Agentic Approval\nReports health.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(source / "app.py").write_text(
|
||||
'@app.get("/health")\ndef health():\n return {}\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
store = RegistryStore(tmp_path / "registry.sqlite3")
|
||||
store.initialize()
|
||||
reviewer = ApprovingAgenticReviewer()
|
||||
service = RegistryService(
|
||||
store,
|
||||
ingestion=GitIngestionService(tmp_path / "checkouts"),
|
||||
agentic_reviewer=reviewer,
|
||||
)
|
||||
repository = service.register_repository(name="Agentic Approval", url=str(source))
|
||||
|
||||
summary = service.analyze_repository(
|
||||
repository.id,
|
||||
use_llm_assistance=False,
|
||||
agentic_review=True,
|
||||
)
|
||||
|
||||
ability_map = service.ability_map(repository.id)
|
||||
graph = service.candidate_graph(repository.id, summary.analysis_run.id)
|
||||
decisions = service.list_review_decisions(repository.id, summary.analysis_run.id)
|
||||
assert ability_map.abilities
|
||||
assert graph.abilities[0].status == "approved"
|
||||
assert decisions[1].action == "agentic_approve_candidate_graph"
|
||||
assert decisions[1].reviewer_type == "agent"
|
||||
assert decisions[1].reviewer_id == "approving-agent"
|
||||
assert decisions[1].policy_version == "agentic-review-policy/test"
|
||||
assert decisions[1].criteria_version == "repo-scoping-quality-criteria/v1"
|
||||
assert decisions[1].criterion_ids == ["RREG-QC-004"]
|
||||
assert decisions[1].evidence_refs == ["README.md", "app.py"]
|
||||
assert decisions[1].decision_kind == "accepted_as_is"
|
||||
assert "rationale=API source and README support" in decisions[1].notes
|
||||
assert "criteria=RREG-QC-004" in decisions[1].notes
|
||||
assert "evidence=README.md, app.py" in decisions[1].notes
|
||||
assert decisions[0].action == "agentic_review_completed"
|
||||
assert "decisions=1" in decisions[0].notes
|
||||
@@ -1,5 +1,5 @@
|
||||
from repo_registry.candidate_graph.generator import CandidateGraphGenerator
|
||||
from repo_registry.core.models import ContentChunk, ObservedFact, Repository
|
||||
from repo_scoping.candidate_graph.generator import CandidateGraphGenerator
|
||||
from repo_scoping.core.models import ContentChunk, ObservedFact, Repository
|
||||
|
||||
|
||||
def fact(id, kind, name, path="", value="", metadata=None):
|
||||
@@ -135,6 +135,85 @@ def test_candidate_generator_extracts_intended_capability_blocks_from_intent_chu
|
||||
assert [ref.path for ref in intent_capability.source_refs] == ["INTENT.md"]
|
||||
|
||||
|
||||
def test_candidate_generator_preserves_unicode_and_normalizes_analysis_names():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
name="VergabeTeilnahme",
|
||||
url="/tmp/vergabe-teilnahme",
|
||||
description=None,
|
||||
branch="main",
|
||||
status="analyzed",
|
||||
)
|
||||
facts = [
|
||||
fact(
|
||||
1,
|
||||
"intent",
|
||||
"INTENT",
|
||||
"INTENT.md",
|
||||
metadata={"source_role": "intent_summary"},
|
||||
)
|
||||
]
|
||||
chunks = [
|
||||
chunk(
|
||||
1,
|
||||
"intent",
|
||||
"INTENT.md",
|
||||
"# INTENT\n\n"
|
||||
"Vollständiger Implementierungsplan in 12 Ralph-Loop-Workplans.\n\n"
|
||||
"## Intended Capabilities\n\n"
|
||||
"- Analysis of impact risk and dependency chains.\n",
|
||||
)
|
||||
]
|
||||
|
||||
graph = CandidateGraphGenerator().generate(repository, facts, chunks)
|
||||
|
||||
assert graph[0].name == "Vollständiger Implementierungsplan In 12 Ralph-Loop-Workplans"
|
||||
assert graph[0].capabilities[0].name == "Analyze Impact Risk And Dependency Chains"
|
||||
|
||||
|
||||
def test_candidate_generator_extracts_primary_outcome_subsections_from_intent():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
name="HelixForge",
|
||||
url="/tmp/helix-forge",
|
||||
description=None,
|
||||
branch="main",
|
||||
status="analyzed",
|
||||
)
|
||||
facts = [
|
||||
fact(
|
||||
1,
|
||||
"intent",
|
||||
"INTENT",
|
||||
"INTENT.md",
|
||||
metadata={"source_role": "intent_summary"},
|
||||
)
|
||||
]
|
||||
chunks = [
|
||||
chunk(
|
||||
1,
|
||||
"intent",
|
||||
"INTENT.md",
|
||||
"# INTENT\n\n"
|
||||
"HelixForge turns intent into structure.\n\n"
|
||||
"## 4\\. Primary outcomes\n\n"
|
||||
"### 4.1 Capability discovery\n\n"
|
||||
"Clarify scope and ownership.\n\n"
|
||||
"### 4.2 Capability validation\n\n"
|
||||
"Validate architecture descriptions structurally and semantically.\n\n"
|
||||
"## Architectural foundation\n\n"
|
||||
"This section should not become a capability.\n",
|
||||
)
|
||||
]
|
||||
|
||||
graph = CandidateGraphGenerator().generate(repository, facts, chunks)
|
||||
|
||||
capability_names = {capability.name for capability in graph[0].capabilities}
|
||||
assert "Support Capability Discovery" in capability_names
|
||||
assert "Validate Capabilities" in capability_names
|
||||
assert "Architectural Foundation" not in capability_names
|
||||
|
||||
|
||||
def test_candidate_generator_prefers_intent_over_derived_scope_for_ability_name():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
@@ -182,6 +261,116 @@ def test_candidate_generator_prefers_intent_over_derived_scope_for_ability_name(
|
||||
assert graph[0].name == "Provide A Provider-agnostic LLM Connector"
|
||||
|
||||
|
||||
def test_candidate_generator_uses_scope_one_liner_over_template_readme():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
name="ops-warden",
|
||||
url="/tmp/ops-warden",
|
||||
description=None,
|
||||
branch="main",
|
||||
status="analyzed",
|
||||
)
|
||||
facts = [
|
||||
fact(1, "documentation", "README", "README.md"),
|
||||
fact(2, "scope", "SCOPE", "SCOPE.md", metadata={"source_role": "derived_scope"}),
|
||||
]
|
||||
chunks = [
|
||||
chunk(
|
||||
1,
|
||||
"documentation",
|
||||
"README.md",
|
||||
"# repo-seed\nA git repository template to bootstrap coulomb projects from.",
|
||||
end_line=2,
|
||||
),
|
||||
chunk(
|
||||
2,
|
||||
"scope",
|
||||
"SCOPE.md",
|
||||
"# SCOPE\n\n## One-liner\n"
|
||||
"SSH Certificate Authority and credential issuance for the ops fleet.\n",
|
||||
end_line=4,
|
||||
),
|
||||
]
|
||||
chunks[1].metadata["source_role"] = "derived_scope"
|
||||
|
||||
graph = CandidateGraphGenerator().generate(repository, facts, chunks)
|
||||
|
||||
assert graph[0].name == "SSH Certificate Authority And Credential Issuance For The Ops Fleet"
|
||||
assert "repo-seed" not in graph[0].description
|
||||
|
||||
|
||||
def test_candidate_generator_extracts_current_capabilities_from_scope_blocks():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
name="railiance-apps",
|
||||
url="/tmp/railiance-apps",
|
||||
description=None,
|
||||
branch="main",
|
||||
status="analyzed",
|
||||
)
|
||||
facts = [
|
||||
fact(1, "scope", "SCOPE", "SCOPE.md", metadata={"source_role": "derived_scope"}),
|
||||
]
|
||||
chunks = [
|
||||
chunk(
|
||||
1,
|
||||
"scope",
|
||||
"SCOPE.md",
|
||||
"# SCOPE\n\n## One-liner\n"
|
||||
"S5 Workloads and Experience layer of the Railiance OAS Stack -- owns applications.\n\n"
|
||||
"## Provided Capabilities\n\n"
|
||||
"```capability\n"
|
||||
"type: infrastructure\n"
|
||||
"title: Application workload deployment\n"
|
||||
"description: Deploy and manage user-facing applications as Helm releases.\n"
|
||||
"keywords: [gitea, helm, application]\n"
|
||||
"```\n",
|
||||
end_line=12,
|
||||
),
|
||||
]
|
||||
chunks[0].metadata["source_role"] = "derived_scope"
|
||||
|
||||
graph = CandidateGraphGenerator().generate(repository, facts, chunks)
|
||||
|
||||
ability = graph[0]
|
||||
assert ability.name == "S5 Workloads And Experience Layer Of The Railiance OAS Stack"
|
||||
assert ability.name == "S5 Workloads And Experience Layer Of The Railiance OAS Stack"
|
||||
capability = ability.capabilities[0]
|
||||
assert capability.name == "Application workload deployment"
|
||||
assert capability.primary_class == "infrastructure"
|
||||
assert {"scope-derived", "current-state", "review-required-scope"} <= set(
|
||||
capability.attributes
|
||||
)
|
||||
assert capability.features[0].name == "Application workload deployment"
|
||||
assert capability.features[0].location == "SCOPE.md"
|
||||
assert capability.evidence[0].reference == "SCOPE.md"
|
||||
|
||||
|
||||
def test_candidate_generator_adds_fact_derived_capability_when_no_stronger_layers():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
name="railiance-empty-layer",
|
||||
url="/tmp/railiance-empty-layer",
|
||||
description=None,
|
||||
branch="main",
|
||||
status="analyzed",
|
||||
)
|
||||
facts = [
|
||||
fact(1, "config", "sops config", ".sops.yaml"),
|
||||
fact(2, "manifest", "pyproject.toml", "pyproject.toml"),
|
||||
]
|
||||
|
||||
graph = CandidateGraphGenerator().generate(repository, facts)
|
||||
|
||||
capability = graph[0].capabilities[0]
|
||||
assert capability.name == "Manage Repository Configuration"
|
||||
assert capability.primary_class == "fact-derived"
|
||||
assert {feature.type for feature in capability.features} == {
|
||||
"configuration",
|
||||
"manifest",
|
||||
}
|
||||
|
||||
|
||||
def test_candidate_generator_enriches_descriptions_from_content_chunks():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
@@ -517,6 +706,140 @@ def test_candidate_generator_does_not_promote_llm_provider_mentions_to_capabilit
|
||||
] == []
|
||||
|
||||
|
||||
def test_candidate_generator_does_not_promote_owned_provider_vocabulary_to_capability():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
name="RepoScoping",
|
||||
url="/tmp/repo-scoping",
|
||||
description="Maps repositories into reviewable capability graphs.",
|
||||
branch="main",
|
||||
status="analyzed",
|
||||
)
|
||||
facts = [
|
||||
fact(1, "documentation", "README", "README.md"),
|
||||
fact(2, "interface", "python route decorator", "src/api.py", '@app.get("/repos")'),
|
||||
fact(
|
||||
3,
|
||||
"llm_provider",
|
||||
"OpenRouter",
|
||||
"src/repo_scoping/repo_scanning/scanner.py",
|
||||
"openrouter",
|
||||
{"source_role": "implementation_source", "utility_relationship": "owned"},
|
||||
),
|
||||
fact(
|
||||
4,
|
||||
"provider_registry",
|
||||
"LLM provider registry",
|
||||
"src/repo_scoping/repo_scanning/scanner.py",
|
||||
metadata={
|
||||
"source_role": "implementation_source",
|
||||
"utility_relationship": "owned",
|
||||
},
|
||||
),
|
||||
fact(
|
||||
5,
|
||||
"credential_config",
|
||||
"OpenRouter API key",
|
||||
"src/repo_scoping/repo_scanning/scanner.py",
|
||||
"OPENROUTER_API_KEY",
|
||||
{"source_role": "implementation_source", "utility_relationship": "configure"},
|
||||
),
|
||||
]
|
||||
|
||||
graph = CandidateGraphGenerator().generate(repository, facts)
|
||||
|
||||
capability_names = {capability.name for capability in graph[0].capabilities}
|
||||
assert "Route LLM Requests Across Providers" not in capability_names
|
||||
assert "Scan Repositories Into Observed Facts" in capability_names
|
||||
|
||||
|
||||
def test_candidate_generator_recovers_repo_scoping_native_candidate_families():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
name="repo-scoping",
|
||||
url="/tmp/repo-scoping",
|
||||
description="Maps repositories into reviewable capability graphs.",
|
||||
branch="main",
|
||||
status="analyzed",
|
||||
)
|
||||
facts = [
|
||||
fact(1, "documentation", "README", "README.md"),
|
||||
fact(2, "documentation", "api-contract.md", "docs/api-contract.md"),
|
||||
fact(
|
||||
3,
|
||||
"documentation",
|
||||
"characteristic-evidence-model.md",
|
||||
"docs/characteristic-evidence-model.md",
|
||||
),
|
||||
fact(4, "documentation", "scope-md-spec.md", "docs/scope-md-spec.md"),
|
||||
fact(
|
||||
5,
|
||||
"documentation",
|
||||
"dependency-aware-scope-propagation.md",
|
||||
"docs/dependency-aware-scope-propagation.md",
|
||||
),
|
||||
fact(
|
||||
6,
|
||||
"documentation",
|
||||
"repo-scope-context-response.json",
|
||||
"docs/schemas/repo-scope-context-response.json",
|
||||
),
|
||||
fact(7, "test", "test_git_ingestion.py", "tests/test_git_ingestion.py"),
|
||||
fact(
|
||||
8,
|
||||
"test",
|
||||
"test_repository_metadata.py",
|
||||
"tests/test_repository_metadata.py",
|
||||
),
|
||||
fact(
|
||||
9,
|
||||
"test",
|
||||
"test_repository_scanner.py",
|
||||
"tests/test_repository_scanner.py",
|
||||
),
|
||||
fact(10, "test", "test_content_indexing.py", "tests/test_content_indexing.py"),
|
||||
fact(11, "test", "test_candidate_graph.py", "tests/test_candidate_graph.py"),
|
||||
fact(12, "test", "test_llm_extraction.py", "tests/test_llm_extraction.py"),
|
||||
fact(13, "test", "test_registry_service.py", "tests/test_registry_service.py"),
|
||||
fact(14, "test", "test_scope_generator.py", "tests/test_scope_generator.py"),
|
||||
fact(15, "test", "test_web_api.py", "tests/test_web_api.py"),
|
||||
fact(16, "test", "test_scope_context_api.py", "tests/test_scope_context_api.py"),
|
||||
fact(
|
||||
17,
|
||||
"interface",
|
||||
"python route decorator",
|
||||
"src/repo_scoping/web_api/app.py",
|
||||
'@app.post("/repos")',
|
||||
),
|
||||
]
|
||||
|
||||
graph = CandidateGraphGenerator().generate(repository, facts)
|
||||
|
||||
capability_names = {capability.name for capability in graph[0].capabilities}
|
||||
assert {
|
||||
"Register And Track Repositories",
|
||||
"Scan Repositories Into Observed Facts",
|
||||
"Index Source Content With Provenance",
|
||||
"Generate Reviewable Candidate Characteristics",
|
||||
"Review And Approve Candidate Characteristics",
|
||||
"Search Compare And Export Approved Profiles",
|
||||
"Generate And Maintain SCOPE.md",
|
||||
"Explore Dependency And Impact Graphs",
|
||||
"Provide Scope Context To Downstream Agents",
|
||||
} <= capability_names
|
||||
assert "Route LLM Requests Across Providers" not in capability_names
|
||||
scanning = next(
|
||||
capability
|
||||
for capability in graph[0].capabilities
|
||||
if capability.name == "Scan Repositories Into Observed Facts"
|
||||
)
|
||||
assert scanning.primary_class == "analysis"
|
||||
assert {"deterministic", "facts", "provenance", "utility-owned"} <= set(
|
||||
scanning.attributes
|
||||
)
|
||||
assert all(ref.path.startswith(("docs/", "tests/", "src/")) for ref in scanning.source_refs)
|
||||
|
||||
|
||||
def test_candidate_generator_excludes_mention_only_providers_from_promoted_capability():
|
||||
repository = Repository(
|
||||
id=1,
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
from repo_registry.candidate_graph.generator import (
|
||||
from repo_scoping.candidate_graph.generator import (
|
||||
CandidateAbilityDraft,
|
||||
CandidateCapabilityDraft,
|
||||
CandidateFeatureDraft,
|
||||
)
|
||||
from repo_registry.candidate_graph.normalization import normalize_candidate_drafts
|
||||
from repo_registry.core.models import SourceReference
|
||||
from repo_scoping.candidate_graph.normalization import normalize_candidate_drafts
|
||||
from repo_scoping.core.models import SourceReference
|
||||
|
||||
|
||||
def ref(fact_id, path):
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from repo_registry.cli import main
|
||||
from repo_registry.core.service import RegistryService
|
||||
from repo_registry.repo_ingestion.git import GitIngestionService
|
||||
from repo_registry.storage.sqlite import RegistryStore
|
||||
from repo_scoping.cli import main
|
||||
from repo_scoping.core.service import RegistryService
|
||||
from repo_scoping.repo_ingestion.git import GitIngestionService
|
||||
from repo_scoping.storage.sqlite import RegistryStore
|
||||
|
||||
|
||||
def make_service(tmp_path):
|
||||
@@ -98,3 +100,265 @@ def test_rebuild_cli_refuses_destructive_all_without_confirm_all(tmp_path):
|
||||
)
|
||||
|
||||
assert exc.value.code == 2
|
||||
|
||||
|
||||
def test_export_assessment_cli_writes_completed_run_artifact(tmp_path):
|
||||
service = make_service(tmp_path)
|
||||
source = write_repo(tmp_path)
|
||||
repository = service.register_repository(name="CLI Export", url=str(source))
|
||||
summary = service.analyze_repository(repository.id, use_llm_assistance=False)
|
||||
output_path = tmp_path / "assessment.json"
|
||||
|
||||
exit_code = main(
|
||||
[
|
||||
"export-assessment",
|
||||
"--repo",
|
||||
str(repository.id),
|
||||
"--analysis-run",
|
||||
str(summary.analysis_run.id),
|
||||
"--output",
|
||||
str(output_path),
|
||||
"--database-path",
|
||||
str(tmp_path / "registry.sqlite3"),
|
||||
"--checkout-root",
|
||||
str(tmp_path / "checkouts"),
|
||||
]
|
||||
)
|
||||
|
||||
artifact = json.loads(output_path.read_text(encoding="utf-8"))
|
||||
assert exit_code == 0
|
||||
assert artifact["target_repository"]["repo_slug"] == "cli-export"
|
||||
assert artifact["execution"]["analysis_run_id"] == summary.analysis_run.id
|
||||
assert artifact["assessment"]["role"] == "challenger"
|
||||
assert artifact["generated_tree"]["abilities"]
|
||||
|
||||
|
||||
def test_compare_assessment_cli_writes_markdown_report(tmp_path):
|
||||
output_path = tmp_path / "comparison.md"
|
||||
|
||||
exit_code = main(
|
||||
[
|
||||
"compare-assessment",
|
||||
"--golden",
|
||||
"docs/self-scoping/golden/repo-scoping-golden-profile.v1.json",
|
||||
"--assessment",
|
||||
"docs/self-scoping/assessments/repo-scoping-known-bad-2026-05-15-run-39.json",
|
||||
"--output",
|
||||
str(output_path),
|
||||
"--format",
|
||||
"markdown",
|
||||
]
|
||||
)
|
||||
|
||||
report = output_path.read_text(encoding="utf-8")
|
||||
assert exit_code == 0
|
||||
assert "Status: `regression`" in report
|
||||
assert "Route LLM Requests Across Providers" in report
|
||||
|
||||
|
||||
def test_list_quality_criteria_cli_writes_json(tmp_path):
|
||||
output_path = tmp_path / "criteria.json"
|
||||
|
||||
exit_code = main(
|
||||
[
|
||||
"list-quality-criteria",
|
||||
"--output",
|
||||
str(output_path),
|
||||
"--format",
|
||||
"json",
|
||||
]
|
||||
)
|
||||
|
||||
registry = json.loads(output_path.read_text(encoding="utf-8"))
|
||||
assert exit_code == 0
|
||||
assert registry["criteria_version"] == "repo-scoping-quality-criteria/v1"
|
||||
assert {criterion["id"] for criterion in registry["criteria"]} >= {
|
||||
"RREG-QC-002",
|
||||
"RREG-QC-005",
|
||||
}
|
||||
assert all(
|
||||
criterion["deterministic_action"] != "approve"
|
||||
for criterion in registry["criteria"]
|
||||
)
|
||||
|
||||
|
||||
def test_list_legacy_auto_approvals_cli_writes_json_inventory(tmp_path):
|
||||
service = make_service(tmp_path)
|
||||
source = write_repo(tmp_path)
|
||||
repository = service.register_repository(name="Legacy CLI", url=str(source))
|
||||
summary = service.analyze_repository(repository.id, use_llm_assistance=False)
|
||||
service.trusted_auto_approve_candidate_graph(
|
||||
repository.id,
|
||||
summary.analysis_run.id,
|
||||
allow_deprecated_migration_mode=True,
|
||||
)
|
||||
output_path = tmp_path / "legacy-auto-approvals.json"
|
||||
|
||||
exit_code = main(
|
||||
[
|
||||
"list-legacy-auto-approvals",
|
||||
"--format",
|
||||
"json",
|
||||
"--output",
|
||||
str(output_path),
|
||||
"--database-path",
|
||||
str(tmp_path / "registry.sqlite3"),
|
||||
"--checkout-root",
|
||||
str(tmp_path / "checkouts"),
|
||||
]
|
||||
)
|
||||
|
||||
records = json.loads(output_path.read_text(encoding="utf-8"))
|
||||
assert exit_code == 0
|
||||
assert records[0]["repository_id"] == repository.id
|
||||
assert records[0]["repository_name"] == "Legacy CLI"
|
||||
assert records[0]["analysis_run_id"] == summary.analysis_run.id
|
||||
assert records[0]["current_approved_ability_count"] == 1
|
||||
|
||||
|
||||
def test_assess_dataset_cli_reports_sparse_hierarchy_issues(tmp_path):
|
||||
service = make_service(tmp_path)
|
||||
source = tmp_path / "scope-only"
|
||||
source.mkdir()
|
||||
(source / "SCOPE.md").write_text(
|
||||
"# SCOPE\n\n## One-liner\nScope-only current behavior.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
repository = service.register_repository(name="Scope Only", url=str(source))
|
||||
service.analyze_repository(repository.id, use_llm_assistance=False)
|
||||
(source / "SCOPE.md").write_text(
|
||||
"# SCOPE\n\n"
|
||||
"## One-liner\n"
|
||||
"Scope-only current behavior.\n\n"
|
||||
"## Provided Capabilities\n\n"
|
||||
"```capability\n"
|
||||
"name: Review Latest Scope Facts\n"
|
||||
"type: scope-review\n"
|
||||
"description: Review the latest scope facts instead of stale runs.\n"
|
||||
"```\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
latest_summary = service.analyze_repository(repository.id, use_llm_assistance=False)
|
||||
output_path = tmp_path / "dataset.json"
|
||||
|
||||
exit_code = main(
|
||||
[
|
||||
"assess-dataset",
|
||||
"--format",
|
||||
"json",
|
||||
"--output",
|
||||
str(output_path),
|
||||
"--database-path",
|
||||
str(tmp_path / "registry.sqlite3"),
|
||||
"--checkout-root",
|
||||
str(tmp_path / "checkouts"),
|
||||
]
|
||||
)
|
||||
|
||||
report = json.loads(output_path.read_text(encoding="utf-8"))
|
||||
repo_report = report["repositories"][0]
|
||||
assert exit_code == 0
|
||||
assert report["schema_version"] == "repo-scoping-dataset-assessment/v1"
|
||||
assert repo_report["name"] == "Scope Only"
|
||||
assert repo_report["latest_analysis_run_id"] == latest_summary.analysis_run.id
|
||||
assert repo_report["documents"]["SCOPE.md"] is True
|
||||
assert repo_report["candidate_counts"]["capabilities"] >= 1
|
||||
assert repo_report["dependency_graph"]["node_count"] > 0
|
||||
assert "facts-with-empty-dependency-graph" not in repo_report["issues"]
|
||||
|
||||
|
||||
def test_self_assess_cli_exports_challenger_and_comparison(tmp_path):
|
||||
source = write_repo(tmp_path)
|
||||
golden_path = tmp_path / "golden.json"
|
||||
golden_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"profile_id": "test-golden",
|
||||
"ability": {
|
||||
"expected_capabilities": [
|
||||
{"name": "Expose Repository Interface"}
|
||||
]
|
||||
},
|
||||
"forbidden_native_capabilities": [],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
assessment_path = tmp_path / "out" / "assessment.json"
|
||||
comparison_path = tmp_path / "out" / "comparison.json"
|
||||
|
||||
exit_code = main(
|
||||
[
|
||||
"self-assess",
|
||||
"--repo",
|
||||
"Self Assess Repo",
|
||||
"--source-path",
|
||||
str(source),
|
||||
"--golden",
|
||||
str(golden_path),
|
||||
"--assessment-output",
|
||||
str(assessment_path),
|
||||
"--comparison-output",
|
||||
str(comparison_path),
|
||||
"--format",
|
||||
"json",
|
||||
"--database-path",
|
||||
str(tmp_path / "registry.sqlite3"),
|
||||
"--checkout-root",
|
||||
str(tmp_path / "checkouts"),
|
||||
]
|
||||
)
|
||||
|
||||
assessment = json.loads(assessment_path.read_text(encoding="utf-8"))
|
||||
comparison = json.loads(comparison_path.read_text(encoding="utf-8"))
|
||||
assert exit_code == 0
|
||||
assert assessment["target_repository"]["repo_slug"] == "self-assess-repo"
|
||||
assert assessment["execution"]["mode"] == "deterministic-only"
|
||||
assert comparison["status"] == "candidate_improvement"
|
||||
assert comparison["matched_expected_capabilities"] == [
|
||||
"Expose Repository Interface"
|
||||
]
|
||||
|
||||
|
||||
def test_self_assess_cli_can_fail_on_regression(tmp_path):
|
||||
source = tmp_path / "provider-repo"
|
||||
source.mkdir()
|
||||
(source / "README.md").write_text("# Provider Repo\n", encoding="utf-8")
|
||||
(source / "providers.py").write_text(
|
||||
"provider_registry = {'openrouter': OpenRouterAdapter}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
golden_path = tmp_path / "golden.json"
|
||||
golden_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"profile_id": "test-golden",
|
||||
"ability": {"expected_capabilities": []},
|
||||
"forbidden_native_capabilities": [
|
||||
{"name": "Route LLM Requests Across Providers"}
|
||||
],
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
exit_code = main(
|
||||
[
|
||||
"self-assess",
|
||||
"--repo",
|
||||
"Provider Repo",
|
||||
"--source-path",
|
||||
str(source),
|
||||
"--golden",
|
||||
str(golden_path),
|
||||
"--format",
|
||||
"json",
|
||||
"--fail-on-regression",
|
||||
"--database-path",
|
||||
str(tmp_path / "registry.sqlite3"),
|
||||
"--checkout-root",
|
||||
str(tmp_path / "checkouts"),
|
||||
]
|
||||
)
|
||||
|
||||
assert exit_code == 1
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
from repo_registry.content_indexing.extractor import ContentExtractor
|
||||
from repo_registry.core.models import ObservedFact
|
||||
from repo_scoping.content_indexing.extractor import ContentExtractor
|
||||
from repo_scoping.core.models import ObservedFact
|
||||
|
||||
|
||||
def fact(id, kind, name, path="", line=None, source_role=""):
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import subprocess
|
||||
|
||||
from repo_registry.repo_ingestion.git import GitIngestionService
|
||||
from repo_scoping.repo_ingestion.git import GitIngestionService
|
||||
|
||||
|
||||
def run(command, cwd):
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
from datetime import date
|
||||
|
||||
from repo_registry.intent.bootstrap import bootstrap_intent_from_scope, scope_to_intent_text
|
||||
from repo_scoping.intent.bootstrap import bootstrap_intent_from_scope, scope_to_intent_text
|
||||
|
||||
|
||||
def test_scope_to_intent_text_replaces_scope_heading_and_marks_bootstrap():
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import pytest
|
||||
|
||||
from repo_registry.core.models import ContentChunk, Repository
|
||||
from repo_registry.llm_extraction import (
|
||||
from repo_scoping.core.models import ContentChunk, Repository
|
||||
from repo_scoping.llm_extraction import (
|
||||
LLMCandidateExtractor,
|
||||
LLMExtractionError,
|
||||
create_llm_connect_adapter,
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
from repo_registry.core.models import ContentChunk, ObservedFact
|
||||
from repo_registry.llm_extraction import (
|
||||
from repo_scoping.core.models import ContentChunk, ObservedFact
|
||||
from repo_scoping.llm_extraction import (
|
||||
ExtractedAbility,
|
||||
ExtractedCapability,
|
||||
ExtractedEvidence,
|
||||
|
||||
53
tests/test_quality_criteria.py
Normal file
53
tests/test_quality_criteria.py
Normal file
@@ -0,0 +1,53 @@
|
||||
from repo_scoping.acceptance import (
|
||||
active_quality_criteria_version,
|
||||
criteria_registry_markdown,
|
||||
load_quality_criteria,
|
||||
)
|
||||
|
||||
|
||||
def test_quality_criteria_registry_is_versioned_and_reviewable():
|
||||
registry = load_quality_criteria()
|
||||
|
||||
assert registry.schema_version == "quality-criteria-registry/v1"
|
||||
assert registry.criteria_version == "repo-scoping-quality-criteria/v1"
|
||||
assert registry.status == "active"
|
||||
assert {criterion.id for criterion in registry.criteria} == {
|
||||
"RREG-QC-001",
|
||||
"RREG-QC-002",
|
||||
"RREG-QC-003",
|
||||
"RREG-QC-004",
|
||||
"RREG-QC-005",
|
||||
"RREG-QC-006",
|
||||
"RREG-QC-007",
|
||||
"RREG-QC-008",
|
||||
}
|
||||
for criterion in registry.criteria:
|
||||
assert criterion.description
|
||||
assert criterion.severity in {"low", "medium", "high", "critical"}
|
||||
assert criterion.deterministic_action in {
|
||||
"pass",
|
||||
"requires_review",
|
||||
"downgraded",
|
||||
"rejected",
|
||||
"invalidated",
|
||||
"merged",
|
||||
"flagged",
|
||||
}
|
||||
assert criterion.deterministic_action != "approve"
|
||||
assert criterion.deterministic_action_when
|
||||
assert criterion.reviewer_guidance
|
||||
|
||||
|
||||
def test_quality_criteria_markdown_lists_transparent_review_guidance():
|
||||
registry = load_quality_criteria()
|
||||
|
||||
markdown = criteria_registry_markdown(registry)
|
||||
|
||||
assert "# Quality Criteria Registry" in markdown
|
||||
assert "RREG-QC-002: Native Utility Is Repo-Owned" in markdown
|
||||
assert "Deterministic action: `downgraded`" in markdown
|
||||
assert "Reviewer guidance:" in markdown
|
||||
|
||||
|
||||
def test_active_quality_criteria_version_matches_registry():
|
||||
assert active_quality_criteria_version() == "repo-scoping-quality-criteria/v1"
|
||||
241
tests/test_quality_gates.py
Normal file
241
tests/test_quality_gates.py
Normal file
@@ -0,0 +1,241 @@
|
||||
from repo_scoping.acceptance import (
|
||||
blocking_quality_gate_outcomes,
|
||||
evaluate_candidate_capability_quality,
|
||||
evaluate_candidate_graph_quality,
|
||||
quality_gate_outcome_dicts,
|
||||
)
|
||||
from repo_scoping.core.models import (
|
||||
AnalysisRun,
|
||||
CandidateAbility,
|
||||
CandidateCapability,
|
||||
CandidateFeature,
|
||||
CandidateGraph,
|
||||
Repository,
|
||||
SourceReference,
|
||||
)
|
||||
from repo_scoping.core.service import RegistryService
|
||||
from repo_scoping.repo_ingestion.git import GitIngestionService
|
||||
from repo_scoping.storage.sqlite import RegistryStore
|
||||
|
||||
|
||||
def source_ref(path="src/app.py", kind="source"):
|
||||
return SourceReference(
|
||||
fact_id=1,
|
||||
path=path,
|
||||
kind=kind,
|
||||
name=path,
|
||||
line=1,
|
||||
)
|
||||
|
||||
|
||||
def provider_routing_capability():
|
||||
return CandidateCapability(
|
||||
id=10,
|
||||
name="Route LLM Requests Across Providers",
|
||||
description="Routes provider requests.",
|
||||
inputs=[],
|
||||
outputs=[],
|
||||
confidence=0.9,
|
||||
status="candidate",
|
||||
source_refs=[source_ref("src/providers.py")],
|
||||
confidence_label="high",
|
||||
primary_class="llm-integration",
|
||||
attributes=["utility-owned"],
|
||||
features=[
|
||||
CandidateFeature(
|
||||
id=20,
|
||||
name="HTTP API surface",
|
||||
type="API",
|
||||
location="src/app.py",
|
||||
confidence=0.8,
|
||||
status="candidate",
|
||||
source_refs=[source_ref("src/app.py")],
|
||||
confidence_label="high",
|
||||
primary_class="API",
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_quality_gates_flag_known_provider_routing_failure():
|
||||
outcomes = evaluate_candidate_capability_quality(provider_routing_capability())
|
||||
|
||||
outcome_ids = {outcome.criterion_id for outcome in outcomes}
|
||||
assert {"RREG-QC-002", "RREG-QC-003"} <= outcome_ids
|
||||
assert all(outcome.outcome != "approve" for outcome in outcomes)
|
||||
assert blocking_quality_gate_outcomes(outcomes)
|
||||
|
||||
|
||||
def test_quality_gates_flag_circular_scope_evidence():
|
||||
capability = CandidateCapability(
|
||||
id=11,
|
||||
name="Map Repository Scope",
|
||||
description="Uses generated scope.",
|
||||
inputs=[],
|
||||
outputs=[],
|
||||
confidence=0.8,
|
||||
status="candidate",
|
||||
source_refs=[source_ref("SCOPE.md", "generated-scope")],
|
||||
confidence_label="high",
|
||||
primary_class="scope-generation",
|
||||
attributes=["utility-owned"],
|
||||
)
|
||||
|
||||
outcomes = evaluate_candidate_capability_quality(capability)
|
||||
|
||||
assert outcomes[0].criterion_id == "RREG-QC-005"
|
||||
assert outcomes[0].outcome == "requires_review"
|
||||
|
||||
|
||||
def test_quality_gates_flag_scope_derived_candidates_for_review():
|
||||
capability = CandidateCapability(
|
||||
id=12,
|
||||
name="Application workload deployment",
|
||||
description="Extracted from SCOPE.md.",
|
||||
inputs=[],
|
||||
outputs=[],
|
||||
confidence=0.6,
|
||||
status="candidate",
|
||||
source_refs=[source_ref("SCOPE.md", "scope")],
|
||||
confidence_label="medium",
|
||||
primary_class="infrastructure",
|
||||
attributes=["scope-derived", "review-required-scope"],
|
||||
)
|
||||
|
||||
outcomes = evaluate_candidate_capability_quality(capability)
|
||||
|
||||
outcome_ids = {outcome.criterion_id for outcome in outcomes}
|
||||
assert {"RREG-QC-005"} <= outcome_ids
|
||||
assert all(outcome.outcome == "requires_review" for outcome in outcomes)
|
||||
|
||||
|
||||
def test_quality_gates_flag_template_contaminated_abilities():
|
||||
graph = CandidateGraph(
|
||||
repository=Repository(
|
||||
id=1,
|
||||
name="Ops Warden",
|
||||
url=".",
|
||||
description=None,
|
||||
branch="main",
|
||||
status="analyzed",
|
||||
),
|
||||
analysis_run=AnalysisRun(
|
||||
id=1,
|
||||
repository_id=1,
|
||||
snapshot_id=None,
|
||||
status="completed",
|
||||
started_at="2026-05-15T00:00:00Z",
|
||||
completed_at="2026-05-15T00:00:01Z",
|
||||
error_message=None,
|
||||
scanner_version="deterministic-v1",
|
||||
),
|
||||
abilities=[
|
||||
CandidateAbility(
|
||||
id=1,
|
||||
name="A Git Repository Template To Bootstrap Coulomb Projects",
|
||||
description="Derived from repo-seed README boilerplate.",
|
||||
confidence=0.7,
|
||||
status="candidate",
|
||||
source_refs=[source_ref("README.md", "documentation")],
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
outcomes = evaluate_candidate_graph_quality(graph)
|
||||
|
||||
assert outcomes[0].criterion_id == "RREG-QC-007"
|
||||
assert outcomes[0].outcome == "downgraded"
|
||||
|
||||
|
||||
def test_quality_gate_outcomes_are_serializable_for_assessment_artifacts():
|
||||
graph = CandidateGraph(
|
||||
repository=Repository(
|
||||
id=1,
|
||||
name="Repo",
|
||||
url=".",
|
||||
description=None,
|
||||
branch="main",
|
||||
status="indexed",
|
||||
),
|
||||
analysis_run=AnalysisRun(
|
||||
id=1,
|
||||
repository_id=1,
|
||||
snapshot_id=None,
|
||||
status="completed",
|
||||
started_at="2026-05-15T00:00:00Z",
|
||||
completed_at="2026-05-15T00:00:01Z",
|
||||
error_message=None,
|
||||
scanner_version="deterministic-v1",
|
||||
),
|
||||
abilities=[
|
||||
CandidateAbility(
|
||||
id=1,
|
||||
name="Support Repo",
|
||||
description="Support repo.",
|
||||
confidence=0.8,
|
||||
status="candidate",
|
||||
source_refs=[],
|
||||
capabilities=[provider_routing_capability()],
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
payload = quality_gate_outcome_dicts(evaluate_candidate_graph_quality(graph))
|
||||
|
||||
assert payload
|
||||
assert payload[0]["criteria_version"] == "repo-scoping-quality-criteria/v1"
|
||||
|
||||
|
||||
def test_legacy_trusted_auto_approval_skips_quality_gate_blocked_capability(tmp_path):
|
||||
store = RegistryStore(tmp_path / "registry.sqlite3")
|
||||
store.initialize()
|
||||
service = RegistryService(store, ingestion=GitIngestionService(tmp_path / "checkouts"))
|
||||
|
||||
safe, reason = service._trusted_auto_approve_capability_decision(
|
||||
provider_routing_capability()
|
||||
)
|
||||
|
||||
assert safe is False
|
||||
assert "quality gates require review" in reason
|
||||
assert "RREG-QC-002" in reason
|
||||
|
||||
|
||||
def test_analysis_records_deterministic_gate_review_decision(tmp_path):
|
||||
source = tmp_path / "provider-repo"
|
||||
source.mkdir()
|
||||
(source / "README.md").write_text("# Provider Repo\n", encoding="utf-8")
|
||||
(source / "providers.py").write_text(
|
||||
"provider_registry = {'openrouter': OpenRouterAdapter}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
store = RegistryStore(tmp_path / "registry.sqlite3")
|
||||
store.initialize()
|
||||
service = RegistryService(store, ingestion=GitIngestionService(tmp_path / "checkouts"))
|
||||
repository = service.register_repository(name="Provider Repo", url=str(source))
|
||||
|
||||
summary = service.analyze_repository(repository.id, use_llm_assistance=False)
|
||||
|
||||
decisions = service.list_review_decisions(repository.id, summary.analysis_run.id)
|
||||
gate_decision = next(
|
||||
decision for decision in decisions if decision.action == "quality_gate_evaluation"
|
||||
)
|
||||
assert gate_decision.reviewer_type == "deterministic-gate"
|
||||
assert "RREG-QC-002" in gate_decision.criterion_ids
|
||||
assert gate_decision.criteria_version == "repo-scoping-quality-criteria/v1"
|
||||
assert "without approving registry truth" in gate_decision.rationale
|
||||
assert service.ability_map(repository.id).abilities == []
|
||||
|
||||
override = service.record_quality_gate_override(
|
||||
repository.id,
|
||||
summary.analysis_run.id,
|
||||
criterion_id="RREG-QC-002",
|
||||
element_type="capability",
|
||||
element_id=10,
|
||||
reason="Curator confirmed this repo now owns provider routing.",
|
||||
notes="Future criteria update may be needed.",
|
||||
)
|
||||
assert override.action == "quality_gate_override"
|
||||
assert override.reviewer_type == "human"
|
||||
assert override.decision_kind == "override"
|
||||
assert override.criterion_ids == ["RREG-QC-002"]
|
||||
assert override.rationale == "Curator confirmed this repo now owns provider routing."
|
||||
@@ -2,17 +2,17 @@ import json
|
||||
import logging
|
||||
import subprocess
|
||||
|
||||
from repo_registry.core.logging import LOGGER_NAME
|
||||
from repo_registry.core.models import SourceReference
|
||||
from repo_registry.core.service import RegistryService
|
||||
from repo_registry.llm_extraction import (
|
||||
from repo_scoping.core.logging import LOGGER_NAME
|
||||
from repo_scoping.core.models import SourceReference
|
||||
from repo_scoping.core.service import RegistryService
|
||||
from repo_scoping.llm_extraction import (
|
||||
ExtractedAbility,
|
||||
ExtractedCapability,
|
||||
ExtractedFeature,
|
||||
)
|
||||
from repo_registry.repo_ingestion.git import GitIngestionService
|
||||
from repo_registry.semantic import HashingEmbeddingProvider
|
||||
from repo_registry.storage.sqlite import NotFoundError, RegistryStore
|
||||
from repo_scoping.repo_ingestion.git import GitIngestionService
|
||||
from repo_scoping.semantic import HashingEmbeddingProvider
|
||||
from repo_scoping.storage.sqlite import NotFoundError, RegistryStore
|
||||
from tests.fixtures import (
|
||||
write_dependency_only_repo,
|
||||
write_empty_repo,
|
||||
@@ -498,6 +498,93 @@ def test_dependency_graph_deduplicates_document_fact_nodes(tmp_path):
|
||||
assert fact_nodes[0]["label"] == "README.md (documentation)"
|
||||
|
||||
|
||||
def test_dependency_graph_renders_candidate_fallback_when_approved_hierarchy_missing(tmp_path):
|
||||
service = make_service(tmp_path)
|
||||
source = tmp_path / "scope-candidate"
|
||||
source.mkdir()
|
||||
(source / "SCOPE.md").write_text(
|
||||
"# SCOPE\n\n"
|
||||
"## One-liner\n"
|
||||
"S5 Workloads and Experience layer.\n\n"
|
||||
"## Provided Capabilities\n\n"
|
||||
"```capability\n"
|
||||
"type: infrastructure\n"
|
||||
"title: Application workload deployment\n"
|
||||
"description: Deploy applications as Helm releases.\n"
|
||||
"keywords: [helm]\n"
|
||||
"```\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
repository = service.register_repository(name="Scope Candidate", url=str(source))
|
||||
service.analyze_repository(
|
||||
repository.id,
|
||||
source_path=str(source),
|
||||
use_llm_assistance=False,
|
||||
)
|
||||
|
||||
payload = service.dependency_graph_elements(repository.id, use_latest_profile=False)
|
||||
|
||||
nodes = [
|
||||
element["data"]
|
||||
for element in payload["elements"]
|
||||
if "source" not in element["data"]
|
||||
]
|
||||
edges = [
|
||||
element["data"]
|
||||
for element in payload["elements"]
|
||||
if "source" in element["data"]
|
||||
]
|
||||
assert payload["metrics"]["node_count"] > 0
|
||||
assert any(node["reviewState"] == "candidate" for node in nodes)
|
||||
assert any(node["reviewState"] == "draft" for node in nodes)
|
||||
assert any(edge["dependencyType"] == "draft-realizes" for edge in edges)
|
||||
assert any(edge["dependencyType"] == "draft-supports" for edge in edges)
|
||||
|
||||
|
||||
def test_dependency_graph_candidate_fallback_uses_latest_completed_run(tmp_path):
|
||||
service = make_service(tmp_path)
|
||||
source = tmp_path / "latest-scope-candidate"
|
||||
source.mkdir()
|
||||
(source / "SCOPE.md").write_text(
|
||||
"# SCOPE\n\n## One-liner\nOld scope summary.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
repository = service.register_repository(name="Latest Scope Candidate", url=str(source))
|
||||
service.analyze_repository(
|
||||
repository.id,
|
||||
source_path=str(source),
|
||||
use_llm_assistance=False,
|
||||
)
|
||||
(source / "SCOPE.md").write_text(
|
||||
"# SCOPE\n\n"
|
||||
"## One-liner\n"
|
||||
"Latest scope summary.\n\n"
|
||||
"## Provided Capabilities\n\n"
|
||||
"```capability\n"
|
||||
"type: review\n"
|
||||
"title: Latest Scope Capability\n"
|
||||
"description: The second run should drive graph fallback.\n"
|
||||
"```\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
latest = service.analyze_repository(
|
||||
repository.id,
|
||||
source_path=str(source),
|
||||
use_llm_assistance=False,
|
||||
)
|
||||
|
||||
payload = service.dependency_graph_elements(repository.id, use_latest_profile=False)
|
||||
|
||||
labels = {
|
||||
element["data"].get("label")
|
||||
for element in payload["elements"]
|
||||
if "source" not in element["data"]
|
||||
}
|
||||
assert latest.analysis_run.id == service.list_analysis_runs(repository.id)[0].id
|
||||
assert "Latest Scope Capability" in labels
|
||||
assert "Old Scope Summary" not in labels
|
||||
|
||||
|
||||
def test_manual_registry_updates_and_deletes_approved_entries(tmp_path):
|
||||
service = make_service(tmp_path)
|
||||
repository = service.register_repository(
|
||||
@@ -1337,7 +1424,7 @@ def test_analyze_repository_falls_back_when_optional_llm_extractor_returns_no_ca
|
||||
assert graph.abilities[0].name == "Support Fallback"
|
||||
|
||||
|
||||
def test_analyze_repository_can_trusted_auto_approve_candidates(tmp_path):
|
||||
def test_analyze_repository_routes_legacy_auto_approve_to_agentic_review(tmp_path):
|
||||
source = tmp_path / "repo"
|
||||
source.mkdir()
|
||||
(source / "README.md").write_text(
|
||||
@@ -1364,20 +1451,17 @@ def test_analyze_repository_can_trusted_auto_approve_candidates(tmp_path):
|
||||
graph = service.candidate_graph(repository.id, summary.analysis_run.id)
|
||||
decisions = service.list_review_decisions(repository.id, summary.analysis_run.id)
|
||||
|
||||
assert service.get_repository(repository.id).status == "indexed"
|
||||
assert service.get_repository(repository.id).status == "analyzed"
|
||||
statuses_by_capability = {
|
||||
capability.name: capability.status
|
||||
for capability in graph.abilities[0].capabilities
|
||||
}
|
||||
assert statuses_by_capability["Expose Repository Interface"] == "approved"
|
||||
assert ability_map.abilities[0].name == "Report Health Over HTTP"
|
||||
assert decisions[0].action == "trusted_auto_approve_candidate_graph"
|
||||
assert statuses_by_capability["Expose Repository Interface"] == "candidate"
|
||||
assert ability_map.abilities == []
|
||||
assert decisions[0].action == "agentic_review_unconfigured"
|
||||
assert "deterministic candidate generation" in decisions[0].notes
|
||||
assert "Auto-approved 1 safe candidate capability(s); left 0 for review." in decisions[0].notes
|
||||
assert (
|
||||
"Approved: Expose Repository Interface: owned interface with sufficient confidence."
|
||||
in decisions[0].notes
|
||||
)
|
||||
assert "Deprecated trusted_auto_approve request was routed" in decisions[0].notes
|
||||
assert "candidates remain pending human review" in decisions[0].notes
|
||||
|
||||
|
||||
def test_rebuild_characteristics_dry_run_preserves_approved_map(tmp_path):
|
||||
|
||||
@@ -1,7 +1,21 @@
|
||||
from repo_registry.repo_ingestion.metadata import RepositoryMetadataExtractor
|
||||
from repo_scoping.repo_ingestion.metadata import RepositoryMetadataExtractor
|
||||
|
||||
|
||||
def test_metadata_prefers_pyproject(tmp_path):
|
||||
def test_metadata_prefers_source_identity_over_pyproject(tmp_path):
|
||||
repo = tmp_path / "repo-scoping"
|
||||
repo.mkdir()
|
||||
(repo / "pyproject.toml").write_text(
|
||||
'[project]\nname = "repo-registry"\ndescription = "Repository Scoping."\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
metadata = RepositoryMetadataExtractor().extract(repo, str(repo))
|
||||
|
||||
assert metadata.name == "repo-scoping"
|
||||
assert metadata.description == "Repository Scoping."
|
||||
|
||||
|
||||
def test_metadata_uses_pyproject_when_source_name_is_generic(tmp_path):
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "pyproject.toml").write_text(
|
||||
@@ -15,7 +29,7 @@ def test_metadata_prefers_pyproject(tmp_path):
|
||||
assert metadata.description == "Extract invoice data."
|
||||
|
||||
|
||||
def test_metadata_uses_package_json(tmp_path):
|
||||
def test_metadata_uses_package_json_when_source_name_is_generic(tmp_path):
|
||||
repo = tmp_path / "repo"
|
||||
repo.mkdir()
|
||||
(repo / "package.json").write_text(
|
||||
@@ -29,8 +43,8 @@ def test_metadata_uses_package_json(tmp_path):
|
||||
assert metadata.description == "Browse repository abilities."
|
||||
|
||||
|
||||
def test_metadata_falls_back_to_readme_title(tmp_path):
|
||||
repo = tmp_path / "repo-name"
|
||||
def test_metadata_uses_readme_title_when_source_name_is_generic(tmp_path):
|
||||
repo = tmp_path / "repository"
|
||||
repo.mkdir()
|
||||
(repo / "README.md").write_text(
|
||||
"# Useful Registry\n\nExtra details follow.\n",
|
||||
@@ -41,3 +55,19 @@ def test_metadata_falls_back_to_readme_title(tmp_path):
|
||||
|
||||
assert metadata.name == "Useful Registry"
|
||||
assert metadata.description == "Extra details follow."
|
||||
|
||||
|
||||
def test_metadata_strips_git_suffix_from_url_identity(tmp_path):
|
||||
repo = tmp_path / "checkout"
|
||||
repo.mkdir()
|
||||
(repo / "pyproject.toml").write_text(
|
||||
'[project]\nname = "old-package-name"\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
metadata = RepositoryMetadataExtractor().extract(
|
||||
repo,
|
||||
"https://example.test/acme/repo-scoping.git",
|
||||
)
|
||||
|
||||
assert metadata.name == "repo-scoping"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from repo_registry.repo_scanning.scanner import DeterministicScanner
|
||||
from repo_scoping.repo_scanning.scanner import DeterministicScanner
|
||||
from tests.fixtures import (
|
||||
write_javascript_typescript_package_repo,
|
||||
write_misleading_docs_repo,
|
||||
@@ -114,6 +114,34 @@ def test_scanner_javascript_typescript_package_records_package_facts(tmp_path):
|
||||
assert ("test", "routes.spec.ts", "src/api/routes.spec.ts") in facts
|
||||
|
||||
|
||||
def test_scanner_ignores_runtime_var_checkouts(tmp_path):
|
||||
repo = tmp_path / "repo-scoping-like"
|
||||
repo.mkdir()
|
||||
(repo / "README.md").write_text("# Repo Scoping\n", encoding="utf-8")
|
||||
checkout = repo / "var" / "checkouts" / "llm-connect"
|
||||
checkout.mkdir(parents=True)
|
||||
(checkout / "README.md").write_text(
|
||||
"# LLM Connect\nSupports OpenRouter fallback.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(checkout / "providers.py").write_text(
|
||||
"provider_registry = {'openrouter': OpenRouterAdapter}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
result = DeterministicScanner().scan(repo)
|
||||
|
||||
facts = {(fact.kind, fact.name, fact.path) for fact in result.facts}
|
||||
assert result.file_count == 1
|
||||
assert ("documentation", "README", "README.md") in facts
|
||||
assert all(not fact.path.startswith("var/") for fact in result.facts)
|
||||
assert (
|
||||
"llm_provider",
|
||||
"OpenRouter",
|
||||
"var/checkouts/llm-connect/README.md",
|
||||
) not in facts
|
||||
|
||||
|
||||
def test_scanner_records_llm_provider_and_fallback_facts(tmp_path):
|
||||
repo = tmp_path / "llm-connect-like"
|
||||
repo.mkdir()
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user