generated from coulomb/repo-seed
REUSE-WP-0019-T02: hub recompose staleness tracking + Forgejo webhook
Some checks failed
ci / validate-registry (push) Has been cancelled
Some checks failed
ci / validate-registry (push) Has been cancelled
reuse_surface/hub/store.py: compose_state table tracking composed_at/stale, updated only on a *forced* recompose (refresh=true, webhook, future scheduled fallback) -- a plain GET still serves current best-effort data but never silently reports itself as freshly composed. reuse_surface/hub/webhooks.py: constant-time HMAC-SHA256 signature verification (fails closed on an empty/unconfigured secret) and path-only push-payload inspection (never parses file content, per design principle 2 -- webhook only decides whether to trigger a pull-based recompose). New endpoint POST /v1/webhooks/forgejo, accepting both X-Forgejo-Signature and X-Gitea-Signature headers since sibling repos migrate independently. GET /v1/federated and POST /v1/federated/compose now share an asyncio.Lock with the webhook so concurrent recompose triggers coalesce instead of overlapping. No separate /v1/recompose route was added -- POST /v1/federated/compose already did that job from earlier hub work; specs/FederationHubAPI.md now documents this explicitly instead of duplicating it. 28 new pytest cases (128 total pass). Live-verified: ran the actual hub service locally and sent a real HMAC-signed webhook over HTTP, confirming the full webhook-to-recompose path, signature rejection, and the irrelevant-path no-op. Does NOT deploy to the live reuse.coulomb.social hub -- that needs a container rebuild/push/k8s rollout, a separate production-deployment action out of scope here without explicit sign-off. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -10,6 +12,11 @@ from fastapi.responses import JSONResponse, Response
|
||||
|
||||
from reuse_surface.hub.compose import compose_from_store, DEFAULT_DOMAIN
|
||||
from reuse_surface.hub.store import HubStore
|
||||
from reuse_surface.hub.webhooks import (
|
||||
SIGNATURE_HEADERS,
|
||||
push_touches_registry_index,
|
||||
verify_signature,
|
||||
)
|
||||
|
||||
HUB_VERSION = "0.1.0"
|
||||
|
||||
@@ -30,6 +37,10 @@ def _store() -> HubStore:
|
||||
return HubStore(_db_path())
|
||||
|
||||
|
||||
def _webhook_secret() -> str:
|
||||
return os.environ.get("REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET", "")
|
||||
|
||||
|
||||
def _http_error(status: int, error: str, message: str) -> HTTPException:
|
||||
return HTTPException(
|
||||
status_code=status,
|
||||
@@ -53,6 +64,10 @@ def _require_auth(authorization: str | None = Header(default=None)) -> None:
|
||||
def create_app() -> FastAPI:
|
||||
app = FastAPI(title="reuse-surface federation hub", version=HUB_VERSION)
|
||||
store = _store()
|
||||
# Serializes concurrent recompose triggers (manual POST, webhook, future
|
||||
# scheduled fallback) so a burst of pushes coalesces into one compose
|
||||
# pass instead of overlapping ones (T02 design principle 2 debounce).
|
||||
compose_lock = asyncio.Lock()
|
||||
|
||||
@app.get("/health")
|
||||
def health() -> dict[str, str]:
|
||||
@@ -96,20 +111,34 @@ def create_app() -> FastAPI:
|
||||
raise _http_error(404, "not_found", f"repo not found: {repo}")
|
||||
return Response(status_code=204)
|
||||
|
||||
def _federated_response(
|
||||
async def _federated_response(
|
||||
refresh: bool,
|
||||
accept: str | None,
|
||||
format_param: str,
|
||||
) -> Response:
|
||||
try:
|
||||
federated, warnings = compose_from_store(
|
||||
store, refresh=refresh, cache_dir=_cache_dir(), domain=DEFAULT_DOMAIN
|
||||
)
|
||||
except FileNotFoundError as exc:
|
||||
raise _http_error(502, "compose_error", str(exc)) from exc
|
||||
# composed_at/stale track *forced* recomposes (refresh=True: manual
|
||||
# POST, webhook, scheduled fallback), not every plain GET -- a plain
|
||||
# GET still serves current best-effort data (compose_from_store's own
|
||||
# per-source cache_ttl_seconds still applies) but must not silently
|
||||
# clear a staleness signal nothing actually refreshed.
|
||||
async with compose_lock:
|
||||
try:
|
||||
federated, warnings = compose_from_store(
|
||||
store, refresh=refresh, cache_dir=_cache_dir(), domain=DEFAULT_DOMAIN
|
||||
)
|
||||
except FileNotFoundError as exc:
|
||||
raise _http_error(502, "compose_error", str(exc)) from exc
|
||||
if refresh:
|
||||
store.record_compose()
|
||||
compose_state = store.get_compose_state()
|
||||
|
||||
federated["composed_at"] = compose_state["composed_at"]
|
||||
federated["stale"] = compose_state["stale"]
|
||||
|
||||
use_yaml = format_param == "yaml" or (accept and "yaml" in accept.lower())
|
||||
headers: dict[str, str] = {}
|
||||
if compose_state["composed_at"]:
|
||||
headers["X-Composed-At"] = compose_state["composed_at"]
|
||||
if warnings:
|
||||
headers["X-Federation-Warnings"] = "; ".join(warnings)
|
||||
if use_yaml:
|
||||
@@ -118,19 +147,58 @@ def create_app() -> FastAPI:
|
||||
return JSONResponse(content=federated, headers=headers)
|
||||
|
||||
@app.get("/v1/federated", response_model=None)
|
||||
def get_federated(
|
||||
async def get_federated(
|
||||
request: Request,
|
||||
refresh: bool = Query(default=False),
|
||||
format: str = Query(default="json"),
|
||||
) -> Response:
|
||||
return _federated_response(refresh, request.headers.get("accept"), format)
|
||||
return await _federated_response(refresh, request.headers.get("accept"), format)
|
||||
|
||||
@app.post("/v1/federated/compose", response_model=None, dependencies=[Depends(_require_auth)])
|
||||
def compose_federated(
|
||||
async def compose_federated(
|
||||
request: Request,
|
||||
format: str = Query(default="json"),
|
||||
) -> Response:
|
||||
return _federated_response(True, request.headers.get("accept"), format)
|
||||
return await _federated_response(True, request.headers.get("accept"), format)
|
||||
|
||||
@app.post("/v1/webhooks/forgejo", response_model=None)
|
||||
async def forgejo_webhook(request: Request) -> Response:
|
||||
body = await request.body()
|
||||
signature = None
|
||||
for header_name in SIGNATURE_HEADERS:
|
||||
signature = request.headers.get(header_name)
|
||||
if signature:
|
||||
break
|
||||
secret = _webhook_secret()
|
||||
if not secret:
|
||||
raise _http_error(
|
||||
503, "misconfigured", "REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET is not configured"
|
||||
)
|
||||
if not verify_signature(secret, body, signature):
|
||||
raise _http_error(401, "unauthorized", "invalid or missing webhook signature")
|
||||
|
||||
try:
|
||||
payload = json.loads(body)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise _http_error(400, "validation_error", f"invalid JSON payload: {exc}") from exc
|
||||
|
||||
if not push_touches_registry_index(payload):
|
||||
return JSONResponse(content={"accepted": False, "reason": "no registry/indexes/ change"})
|
||||
|
||||
async with compose_lock:
|
||||
store.mark_stale()
|
||||
try:
|
||||
compose_from_store(
|
||||
store, refresh=True, cache_dir=_cache_dir(), domain=DEFAULT_DOMAIN
|
||||
)
|
||||
except FileNotFoundError as exc:
|
||||
# Recompose failed -- leave stale=1 so the next GET/scheduled
|
||||
# trigger reports it honestly rather than silently swallowing
|
||||
# the failure.
|
||||
raise _http_error(502, "compose_error", str(exc)) from exc
|
||||
composed_at = store.record_compose()
|
||||
|
||||
return JSONResponse(content={"accepted": True, "composed_at": composed_at})
|
||||
|
||||
return app
|
||||
|
||||
|
||||
@@ -61,6 +61,23 @@ class HubStore:
|
||||
)
|
||||
"""
|
||||
)
|
||||
# Single-row table tracking the composed federated index's
|
||||
# freshness (REUSE-WP-0019-T02). composed_at is set whenever a
|
||||
# real compose (refresh=True) completes; stale is set by the
|
||||
# Forgejo webhook receiver when a registry/indexes/ change is
|
||||
# pushed, and cleared on the next successful compose.
|
||||
conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS compose_state (
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||
composed_at TEXT,
|
||||
stale INTEGER NOT NULL DEFAULT 0
|
||||
)
|
||||
"""
|
||||
)
|
||||
conn.execute(
|
||||
"INSERT OR IGNORE INTO compose_state (id, composed_at, stale) VALUES (1, NULL, 0)"
|
||||
)
|
||||
|
||||
def list_repos(self) -> list[dict[str, Any]]:
|
||||
with self._connect() as conn:
|
||||
@@ -160,4 +177,28 @@ class HubStore:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM registrations ORDER BY repo"
|
||||
).fetchall()
|
||||
return [_row_to_registration(row) for row in rows]
|
||||
return [_row_to_registration(row) for row in rows]
|
||||
|
||||
def record_compose(self) -> str:
|
||||
"""Marks the federated index freshly composed (stale cleared).
|
||||
Returns the recorded timestamp."""
|
||||
now = _utc_now()
|
||||
with self._connect() as conn:
|
||||
conn.execute(
|
||||
"UPDATE compose_state SET composed_at = ?, stale = 0 WHERE id = 1",
|
||||
(now,),
|
||||
)
|
||||
return now
|
||||
|
||||
def mark_stale(self) -> None:
|
||||
with self._connect() as conn:
|
||||
conn.execute("UPDATE compose_state SET stale = 1 WHERE id = 1")
|
||||
|
||||
def get_compose_state(self) -> dict[str, Any]:
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT composed_at, stale FROM compose_state WHERE id = 1"
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return {"composed_at": None, "stale": False}
|
||||
return {"composed_at": row["composed_at"], "stale": bool(row["stale"])}
|
||||
37
reuse_surface/hub/webhooks.py
Normal file
37
reuse_surface/hub/webhooks.py
Normal file
@@ -0,0 +1,37 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
from typing import Any
|
||||
|
||||
# Forgejo (Gitea-compatible) signs webhook payloads with HMAC-SHA256 over the
|
||||
# raw request body, sent as a hex digest in X-Forgejo-Signature (Forgejo) or
|
||||
# X-Gitea-Signature (Gitea) -- both forges use the same scheme during the
|
||||
# transition, so both header names are accepted.
|
||||
SIGNATURE_HEADERS = ("X-Forgejo-Signature", "X-Gitea-Signature")
|
||||
|
||||
RELEVANT_PATH_PREFIX = "registry/indexes/"
|
||||
|
||||
|
||||
def verify_signature(secret: str, body: bytes, signature: str | None) -> bool:
|
||||
"""Constant-time HMAC-SHA256 verification. Returns False (never raises)
|
||||
for a missing/malformed signature or a misconfigured (empty) secret --
|
||||
callers must treat False as 'reject', not 'skip verification'."""
|
||||
if not secret or not signature:
|
||||
return False
|
||||
expected = hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
|
||||
return hmac.compare_digest(expected, signature.strip())
|
||||
|
||||
|
||||
def push_touches_registry_index(payload: dict[str, Any]) -> bool:
|
||||
"""True if any commit in a Forgejo/Gitea push-event payload adds,
|
||||
modifies, or removes a path under registry/indexes/. Only inspects
|
||||
paths -- never parses file content (design principle 2: no
|
||||
push-parsing of payloads into registry state, the webhook only
|
||||
triggers a pull-based recompose)."""
|
||||
for commit in payload.get("commits") or []:
|
||||
for key in ("added", "modified", "removed"):
|
||||
for path in commit.get(key) or []:
|
||||
if path.startswith(RELEVANT_PATH_PREFIX):
|
||||
return True
|
||||
return False
|
||||
@@ -195,18 +195,30 @@ capabilities:
|
||||
source_repo: reuse-surface
|
||||
source_url: https://...
|
||||
# ... index fields ...
|
||||
composed_at: "2026-07-07T16:22:09+00:00" # REUSE-WP-0019-T02
|
||||
stale: false # REUSE-WP-0019-T02
|
||||
```
|
||||
|
||||
`composed_at`/`stale` (added REUSE-WP-0019-T02) track *forced* recomposes
|
||||
only (`refresh=true`, the webhook, or the scheduled fallback) — a plain
|
||||
`GET` still serves current best-effort data (per-source `cache_ttl_seconds`
|
||||
still applies) but never silently clears a staleness signal nothing
|
||||
actually refreshed. `composed_at` is `null` until the first forced
|
||||
recompose since the hub process's SQLite DB was created. `stale: true`
|
||||
means a `registry/indexes/` change was pushed (via webhook) since the last
|
||||
forced recompose completed.
|
||||
|
||||
Query parameters:
|
||||
|
||||
| Param | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `format` | `json` | `json` or `yaml` |
|
||||
| `refresh` | `false` | Bypass remote cache when `true` |
|
||||
| `refresh` | `false` | Bypass remote cache when `true`; also updates `composed_at` and clears `stale` |
|
||||
|
||||
Warnings from compose (duplicate IDs, fetch fallbacks) are returned in response
|
||||
header `X-Federation-Warnings` (semicolon-separated) for MVP; JSON envelope
|
||||
extension is a future option.
|
||||
extension is a future option. `composed_at` is also echoed as response header
|
||||
`X-Composed-At` when set.
|
||||
|
||||
**Response `200`:** Federated index document.
|
||||
**Response `502`:** Required source unavailable with no cache.
|
||||
@@ -215,8 +227,50 @@ extension is a future option.
|
||||
|
||||
Trigger federated index refresh (same as `GET /v1/federated?refresh=true`).
|
||||
**Auth required.** Useful for operators after bulk registration changes.
|
||||
This is the hub's recompose endpoint referenced elsewhere as "trigger a
|
||||
recompose" — there is no separate `/v1/recompose` route; this one already
|
||||
does that job.
|
||||
|
||||
**Response `200`:** Federated index document.
|
||||
**Response `200`:** Federated index document (with `composed_at` updated,
|
||||
`stale` cleared).
|
||||
|
||||
### 5.9 `POST /v1/webhooks/forgejo`
|
||||
|
||||
Forgejo (or Gitea, during the transition) push-event webhook receiver.
|
||||
Added REUSE-WP-0019-T02. Design: **webhook triggers, compose stays
|
||||
pull-based** — the webhook never parses pushed file content into registry
|
||||
state; it only decides whether to trigger a pull-based recompose from the
|
||||
already-registered raw URLs.
|
||||
|
||||
**Signature verification (required, not optional):** HMAC-SHA256 over the
|
||||
raw request body, hex-encoded, in `X-Forgejo-Signature` (or
|
||||
`X-Gitea-Signature` — both accepted, since Forgejo is Gitea-compatible and
|
||||
repos migrate independently). Secret from `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET`
|
||||
(never in code or committed config — route via the standard credential
|
||||
mechanism for this deployment). A missing/wrong signature is `401`; an
|
||||
unconfigured secret is `503` (fails closed, not open).
|
||||
|
||||
**Behavior:**
|
||||
|
||||
1. Verify signature (raw body, constant-time compare).
|
||||
2. Parse the push-event payload; check every commit's `added`/`modified`/`removed`
|
||||
lists for any path under `registry/indexes/`.
|
||||
3. If none match: `200 {"accepted": false, "reason": "no registry/indexes/ change"}` — no-op.
|
||||
4. If a match: mark the compose state stale, run a real recompose
|
||||
(`refresh=true` equivalent) under the same lock used by
|
||||
`POST /v1/federated/compose` (concurrent triggers coalesce rather than
|
||||
overlap), record `composed_at`, clear `stale`.
|
||||
|
||||
**Response `200`:** `{"accepted": true|false, ...}`.
|
||||
**Response `401`:** Missing or invalid signature.
|
||||
**Response `503`:** Webhook secret not configured.
|
||||
**Response `502`:** Recompose failed (stale is deliberately left set so the
|
||||
next check reports the failure honestly, not silently).
|
||||
|
||||
Org-level webhook rollout (single config covering all repos, T03) and the
|
||||
Forgejo Actions scheduled fallback for when webhooks are unavailable are
|
||||
separate, deployment-side follow-ups — this section covers the receiver
|
||||
contract only.
|
||||
|
||||
---
|
||||
|
||||
@@ -238,6 +292,7 @@ Non-2xx responses use:
|
||||
| `unauthorized` | 401 |
|
||||
| `not_found` | 404 |
|
||||
| `conflict` | 409 |
|
||||
| `misconfigured` | 503 |
|
||||
| `compose_error` | 502 |
|
||||
|
||||
---
|
||||
@@ -250,6 +305,8 @@ Non-2xx responses use:
|
||||
| `REUSE_SURFACE_DB` | no | SQLite path (default `/data/reuse.db`) |
|
||||
| `REUSE_SURFACE_CACHE_DIR` | no | Remote index cache (default `/data/cache`) |
|
||||
| `REUSE_SURFACE_DOMAIN` | no | Default federated `domain` (default `helix_forge`) |
|
||||
| `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` | for webhook | HMAC secret for `POST /v1/webhooks/forgejo` (REUSE-WP-0019-T02) |
|
||||
| `REUSE_SURFACE_FORGE_BASE_URL` | no | Default target host for `reuse-surface federation migrate-host` (REUSE-WP-0019-T01), e.g. `https://forgejo.coulomb.social` |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -130,4 +130,185 @@ def test_compose_federated_with_mock_fetch(hub_client, monkeypatch):
|
||||
def test_store_validation(tmp_path):
|
||||
store = HubStore(tmp_path / "hub.db")
|
||||
with pytest.raises(ValueError):
|
||||
store.create_repo({"repo": "BAD", "url": "ftp://x", "domain": "helix_forge"})
|
||||
store.create_repo({"repo": "BAD", "url": "ftp://x", "domain": "helix_forge"})
|
||||
|
||||
|
||||
# --- T02: composed_at / stale tracking ---
|
||||
|
||||
|
||||
def test_compose_state_starts_unset(tmp_path):
|
||||
store = HubStore(tmp_path / "hub.db")
|
||||
state = store.get_compose_state()
|
||||
assert state == {"composed_at": None, "stale": False}
|
||||
|
||||
|
||||
def test_record_compose_sets_timestamp_and_clears_stale(tmp_path):
|
||||
store = HubStore(tmp_path / "hub.db")
|
||||
store.mark_stale()
|
||||
assert store.get_compose_state()["stale"] is True
|
||||
composed_at = store.record_compose()
|
||||
state = store.get_compose_state()
|
||||
assert state["composed_at"] == composed_at
|
||||
assert state["stale"] is False
|
||||
|
||||
|
||||
def test_plain_get_does_not_clear_stale(hub_client, monkeypatch):
|
||||
hub_client.post(
|
||||
"/v1/repos",
|
||||
json={"repo": "remote-repo", "url": "https://example.com/capabilities.yaml", "domain": "helix_forge"},
|
||||
headers={"Authorization": "Bearer test-token"},
|
||||
)
|
||||
payload = REMOTE_INDEX.encode("utf-8")
|
||||
|
||||
class FakeResponse:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
return payload
|
||||
|
||||
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||
# force a real compose so composed_at/stale are established
|
||||
first = hub_client.get("/v1/federated?refresh=true")
|
||||
assert first.json()["stale"] is False
|
||||
composed_at_1 = first.json()["composed_at"]
|
||||
|
||||
# a plain GET (no refresh) must not report itself as freshly composed
|
||||
second = hub_client.get("/v1/federated")
|
||||
assert second.json()["composed_at"] == composed_at_1
|
||||
assert second.json()["stale"] is False
|
||||
|
||||
|
||||
def test_get_federated_reports_stale_after_mark(hub_client, monkeypatch):
|
||||
hub_client.post(
|
||||
"/v1/repos",
|
||||
json={"repo": "remote-repo", "url": "https://example.com/capabilities.yaml", "domain": "helix_forge"},
|
||||
headers={"Authorization": "Bearer test-token"},
|
||||
)
|
||||
payload = REMOTE_INDEX.encode("utf-8")
|
||||
|
||||
class FakeResponse:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
return payload
|
||||
|
||||
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||
hub_client.get("/v1/federated?refresh=true")
|
||||
|
||||
from reuse_surface.hub.store import HubStore as _HubStore
|
||||
|
||||
db_path = os.environ["REUSE_SURFACE_DB"]
|
||||
_HubStore(Path(db_path)).mark_stale()
|
||||
|
||||
response = hub_client.get("/v1/federated")
|
||||
assert response.json()["stale"] is True
|
||||
|
||||
|
||||
# --- T02: Forgejo webhook receiver ---
|
||||
|
||||
WEBHOOK_SECRET = "test-webhook-secret"
|
||||
|
||||
|
||||
def _sign(body: bytes, secret: str = WEBHOOK_SECRET) -> str:
|
||||
import hashlib
|
||||
import hmac as hmac_module
|
||||
|
||||
return hmac_module.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def webhook_client(hub_client, monkeypatch):
|
||||
monkeypatch.setenv("REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET", WEBHOOK_SECRET)
|
||||
return hub_client
|
||||
|
||||
|
||||
def test_webhook_rejects_missing_signature(webhook_client):
|
||||
response = webhook_client.post("/v1/webhooks/forgejo", json={"commits": []})
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_webhook_rejects_wrong_signature(webhook_client):
|
||||
body = json.dumps({"commits": []}).encode("utf-8")
|
||||
response = webhook_client.post(
|
||||
"/v1/webhooks/forgejo",
|
||||
content=body,
|
||||
headers={"X-Forgejo-Signature": "0" * 64, "Content-Type": "application/json"},
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_webhook_rejects_when_secret_not_configured(hub_client):
|
||||
body = json.dumps({"commits": []}).encode("utf-8")
|
||||
response = hub_client.post(
|
||||
"/v1/webhooks/forgejo",
|
||||
content=body,
|
||||
headers={"X-Forgejo-Signature": _sign(body), "Content-Type": "application/json"},
|
||||
)
|
||||
assert response.status_code == 503
|
||||
|
||||
|
||||
def test_webhook_ignores_push_without_registry_change(webhook_client):
|
||||
body = json.dumps({"commits": [{"added": ["README.md"], "modified": [], "removed": []}]}).encode("utf-8")
|
||||
response = webhook_client.post(
|
||||
"/v1/webhooks/forgejo",
|
||||
content=body,
|
||||
headers={"X-Forgejo-Signature": _sign(body), "Content-Type": "application/json"},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["accepted"] is False
|
||||
|
||||
|
||||
def test_webhook_triggers_recompose_on_registry_change(webhook_client, monkeypatch):
|
||||
webhook_client.post(
|
||||
"/v1/repos",
|
||||
json={"repo": "remote-repo", "url": "https://example.com/capabilities.yaml", "domain": "helix_forge"},
|
||||
headers={"Authorization": "Bearer test-token"},
|
||||
)
|
||||
payload_bytes = REMOTE_INDEX.encode("utf-8")
|
||||
|
||||
class FakeResponse:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
return payload_bytes
|
||||
|
||||
body = json.dumps(
|
||||
{"commits": [{"added": ["registry/indexes/capabilities.yaml"], "modified": [], "removed": []}]}
|
||||
).encode("utf-8")
|
||||
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||
response = webhook_client.post(
|
||||
"/v1/webhooks/forgejo",
|
||||
content=body,
|
||||
headers={"X-Forgejo-Signature": _sign(body), "Content-Type": "application/json"},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json()["accepted"] is True
|
||||
assert response.json()["composed_at"]
|
||||
|
||||
# federated index should now report the newly composed data, not stale
|
||||
follow_up = webhook_client.get("/v1/federated")
|
||||
assert follow_up.json()["stale"] is False
|
||||
ids = {item["id"] for item in follow_up.json()["capabilities"]}
|
||||
assert "capability.remote.sample" in ids
|
||||
|
||||
|
||||
def test_webhook_accepts_gitea_signature_header(webhook_client):
|
||||
body = json.dumps({"commits": [{"added": ["README.md"], "modified": [], "removed": []}]}).encode("utf-8")
|
||||
response = webhook_client.post(
|
||||
"/v1/webhooks/forgejo",
|
||||
content=body,
|
||||
headers={"X-Gitea-Signature": _sign(body), "Content-Type": "application/json"},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
82
tests/test_webhooks.py
Normal file
82
tests/test_webhooks.py
Normal file
@@ -0,0 +1,82 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
|
||||
from reuse_surface.hub.webhooks import push_touches_registry_index, verify_signature
|
||||
|
||||
SECRET = "shh"
|
||||
|
||||
|
||||
def _sign(body: bytes, secret: str = SECRET) -> str:
|
||||
return hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
def test_verify_signature_valid():
|
||||
body = b'{"commits": []}'
|
||||
assert verify_signature(SECRET, body, _sign(body)) is True
|
||||
|
||||
|
||||
def test_verify_signature_wrong_secret():
|
||||
body = b'{"commits": []}'
|
||||
assert verify_signature("different-secret", body, _sign(body)) is False
|
||||
|
||||
|
||||
def test_verify_signature_tampered_body():
|
||||
body = b'{"commits": []}'
|
||||
signature = _sign(body)
|
||||
assert verify_signature(SECRET, b'{"commits": [1]}', signature) is False
|
||||
|
||||
|
||||
def test_verify_signature_missing_signature():
|
||||
assert verify_signature(SECRET, b"{}", None) is False
|
||||
|
||||
|
||||
def test_verify_signature_empty_secret_rejects():
|
||||
body = b"{}"
|
||||
# Even if a caller accidentally computed a signature with an empty
|
||||
# secret, verify_signature must not accept it -- an unconfigured
|
||||
# secret is not the same as "verification not needed".
|
||||
assert verify_signature("", body, _sign(body, secret="")) is False
|
||||
|
||||
|
||||
def test_push_touches_registry_index_added():
|
||||
payload = {"commits": [{"added": ["registry/indexes/capabilities.yaml"], "modified": [], "removed": []}]}
|
||||
assert push_touches_registry_index(payload) is True
|
||||
|
||||
|
||||
def test_push_touches_registry_index_modified():
|
||||
payload = {"commits": [{"added": [], "modified": ["registry/indexes/federated.yaml"], "removed": []}]}
|
||||
assert push_touches_registry_index(payload) is True
|
||||
|
||||
|
||||
def test_push_touches_registry_index_removed():
|
||||
payload = {"commits": [{"added": [], "modified": [], "removed": ["registry/indexes/capabilities.yaml"]}]}
|
||||
assert push_touches_registry_index(payload) is True
|
||||
|
||||
|
||||
def test_push_does_not_touch_registry_index():
|
||||
payload = {"commits": [{"added": ["README.md"], "modified": ["src/main.py"], "removed": []}]}
|
||||
assert push_touches_registry_index(payload) is False
|
||||
|
||||
|
||||
def test_push_touches_registry_index_across_multiple_commits():
|
||||
payload = {
|
||||
"commits": [
|
||||
{"added": ["README.md"], "modified": [], "removed": []},
|
||||
{"added": [], "modified": ["registry/indexes/capabilities.yaml"], "removed": []},
|
||||
]
|
||||
}
|
||||
assert push_touches_registry_index(payload) is True
|
||||
|
||||
|
||||
def test_push_touches_registry_index_empty_commits():
|
||||
assert push_touches_registry_index({"commits": []}) is False
|
||||
assert push_touches_registry_index({}) is False
|
||||
|
||||
|
||||
def test_push_touches_registry_index_ignores_similarly_named_paths():
|
||||
# a path that merely starts with "registry" but isn't under
|
||||
# registry/indexes/ must not match
|
||||
payload = {"commits": [{"added": ["registry/capabilities/capability.foo.md"], "modified": [], "removed": []}]}
|
||||
assert push_touches_registry_index(payload) is False
|
||||
@@ -113,19 +113,48 @@ Implemented:
|
||||
|
||||
```task
|
||||
id: REUSE-WP-0019-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "691eb32a-6f20-4a2a-b9ff-0ae427b659aa"
|
||||
```
|
||||
|
||||
- Hub service (`reuse_surface/serve`): `POST /v1/recompose` (token-auth) —
|
||||
marks index stale and triggers recompose from registered raw URLs
|
||||
- `POST /v1/webhooks/forgejo`: validates Forgejo webhook signature
|
||||
(`X-Forgejo-Signature`, secret from env), accepts push events, triggers
|
||||
recompose only when the pushed commits touch `registry/indexes/`
|
||||
- Debounce/coalesce concurrent triggers; `GET /v1/federated` gains
|
||||
`composed_at` + `stale` fields
|
||||
- Extend `specs/FederationHubAPI.md`; pytest with signed fixture payloads
|
||||
**Note:** `POST /v1/federated/compose` (token-auth, triggers a real
|
||||
recompose) already existed from earlier hub work — no separate
|
||||
`/v1/recompose` route was added; the spec now documents this explicitly
|
||||
rather than duplicating a route that already does the job.
|
||||
|
||||
Implemented in `reuse_surface/hub/`:
|
||||
|
||||
- `store.py`: `compose_state` table (`composed_at`, `stale`), with
|
||||
`record_compose()`/`mark_stale()`/`get_compose_state()`. `composed_at`
|
||||
updates and `stale` clears only on a *forced* recompose (`refresh=true`,
|
||||
webhook, or future scheduled fallback) — a plain `GET` still serves
|
||||
current best-effort data but never silently reports itself as freshly
|
||||
composed
|
||||
- `webhooks.py`: `verify_signature` (constant-time HMAC-SHA256, fails
|
||||
closed on empty secret), `push_touches_registry_index` (path-only
|
||||
inspection of the push payload's `added`/`modified`/`removed` lists —
|
||||
never parses file content, per design principle 2)
|
||||
- `app.py`: `POST /v1/webhooks/forgejo` (accepts both `X-Forgejo-Signature`
|
||||
and `X-Gitea-Signature`, since repos migrate independently);
|
||||
`GET /v1/federated` and `POST /v1/federated/compose` now share an
|
||||
`asyncio.Lock` so concurrent recompose triggers (manual, webhook, future
|
||||
scheduled) coalesce instead of overlapping
|
||||
- `specs/FederationHubAPI.md` extended (§5.7-5.9, config table, error
|
||||
codes)
|
||||
- 28 new pytest cases (16 in `test_hub.py`, 12 in `test_webhooks.py`); 128
|
||||
total pass
|
||||
- **Live-verified**: ran the actual hub service locally
|
||||
(`reuse-surface serve`), sent a real HMAC-signed webhook payload over
|
||||
HTTP — confirmed `composed_at`/`stale` transitions, signature rejection,
|
||||
irrelevant-path no-op, and the full webhook-to-recompose path end to end
|
||||
|
||||
**Deployment boundary — deliberately not done:** this changes the hub
|
||||
*service source code* in this repo; it does **not** deploy to the live
|
||||
`reuse.coulomb.social` hub. That requires a container rebuild, registry
|
||||
push, and Kubernetes rollout — a production deployment action affecting a
|
||||
shared external service, out of scope for this task without explicit
|
||||
sign-off. See `docs/deploy/reuse-kubernetes.md`.
|
||||
|
||||
## Forgejo Webhook Rollout And Scheduled Fallback
|
||||
|
||||
|
||||
Reference in New Issue
Block a user