generated from coulomb/repo-seed
REUSE-WP-0019-T02: hub recompose staleness tracking + Forgejo webhook
Some checks failed
ci / validate-registry (push) Has been cancelled
Some checks failed
ci / validate-registry (push) Has been cancelled
reuse_surface/hub/store.py: compose_state table tracking composed_at/stale, updated only on a *forced* recompose (refresh=true, webhook, future scheduled fallback) -- a plain GET still serves current best-effort data but never silently reports itself as freshly composed. reuse_surface/hub/webhooks.py: constant-time HMAC-SHA256 signature verification (fails closed on an empty/unconfigured secret) and path-only push-payload inspection (never parses file content, per design principle 2 -- webhook only decides whether to trigger a pull-based recompose). New endpoint POST /v1/webhooks/forgejo, accepting both X-Forgejo-Signature and X-Gitea-Signature headers since sibling repos migrate independently. GET /v1/federated and POST /v1/federated/compose now share an asyncio.Lock with the webhook so concurrent recompose triggers coalesce instead of overlapping. No separate /v1/recompose route was added -- POST /v1/federated/compose already did that job from earlier hub work; specs/FederationHubAPI.md now documents this explicitly instead of duplicating it. 28 new pytest cases (128 total pass). Live-verified: ran the actual hub service locally and sent a real HMAC-signed webhook over HTTP, confirming the full webhook-to-recompose path, signature rejection, and the irrelevant-path no-op. Does NOT deploy to the live reuse.coulomb.social hub -- that needs a container rebuild/push/k8s rollout, a separate production-deployment action out of scope here without explicit sign-off. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,7 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
import os
|
import os
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -10,6 +12,11 @@ from fastapi.responses import JSONResponse, Response
|
|||||||
|
|
||||||
from reuse_surface.hub.compose import compose_from_store, DEFAULT_DOMAIN
|
from reuse_surface.hub.compose import compose_from_store, DEFAULT_DOMAIN
|
||||||
from reuse_surface.hub.store import HubStore
|
from reuse_surface.hub.store import HubStore
|
||||||
|
from reuse_surface.hub.webhooks import (
|
||||||
|
SIGNATURE_HEADERS,
|
||||||
|
push_touches_registry_index,
|
||||||
|
verify_signature,
|
||||||
|
)
|
||||||
|
|
||||||
HUB_VERSION = "0.1.0"
|
HUB_VERSION = "0.1.0"
|
||||||
|
|
||||||
@@ -30,6 +37,10 @@ def _store() -> HubStore:
|
|||||||
return HubStore(_db_path())
|
return HubStore(_db_path())
|
||||||
|
|
||||||
|
|
||||||
|
def _webhook_secret() -> str:
|
||||||
|
return os.environ.get("REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET", "")
|
||||||
|
|
||||||
|
|
||||||
def _http_error(status: int, error: str, message: str) -> HTTPException:
|
def _http_error(status: int, error: str, message: str) -> HTTPException:
|
||||||
return HTTPException(
|
return HTTPException(
|
||||||
status_code=status,
|
status_code=status,
|
||||||
@@ -53,6 +64,10 @@ def _require_auth(authorization: str | None = Header(default=None)) -> None:
|
|||||||
def create_app() -> FastAPI:
|
def create_app() -> FastAPI:
|
||||||
app = FastAPI(title="reuse-surface federation hub", version=HUB_VERSION)
|
app = FastAPI(title="reuse-surface federation hub", version=HUB_VERSION)
|
||||||
store = _store()
|
store = _store()
|
||||||
|
# Serializes concurrent recompose triggers (manual POST, webhook, future
|
||||||
|
# scheduled fallback) so a burst of pushes coalesces into one compose
|
||||||
|
# pass instead of overlapping ones (T02 design principle 2 debounce).
|
||||||
|
compose_lock = asyncio.Lock()
|
||||||
|
|
||||||
@app.get("/health")
|
@app.get("/health")
|
||||||
def health() -> dict[str, str]:
|
def health() -> dict[str, str]:
|
||||||
@@ -96,20 +111,34 @@ def create_app() -> FastAPI:
|
|||||||
raise _http_error(404, "not_found", f"repo not found: {repo}")
|
raise _http_error(404, "not_found", f"repo not found: {repo}")
|
||||||
return Response(status_code=204)
|
return Response(status_code=204)
|
||||||
|
|
||||||
def _federated_response(
|
async def _federated_response(
|
||||||
refresh: bool,
|
refresh: bool,
|
||||||
accept: str | None,
|
accept: str | None,
|
||||||
format_param: str,
|
format_param: str,
|
||||||
) -> Response:
|
) -> Response:
|
||||||
|
# composed_at/stale track *forced* recomposes (refresh=True: manual
|
||||||
|
# POST, webhook, scheduled fallback), not every plain GET -- a plain
|
||||||
|
# GET still serves current best-effort data (compose_from_store's own
|
||||||
|
# per-source cache_ttl_seconds still applies) but must not silently
|
||||||
|
# clear a staleness signal nothing actually refreshed.
|
||||||
|
async with compose_lock:
|
||||||
try:
|
try:
|
||||||
federated, warnings = compose_from_store(
|
federated, warnings = compose_from_store(
|
||||||
store, refresh=refresh, cache_dir=_cache_dir(), domain=DEFAULT_DOMAIN
|
store, refresh=refresh, cache_dir=_cache_dir(), domain=DEFAULT_DOMAIN
|
||||||
)
|
)
|
||||||
except FileNotFoundError as exc:
|
except FileNotFoundError as exc:
|
||||||
raise _http_error(502, "compose_error", str(exc)) from exc
|
raise _http_error(502, "compose_error", str(exc)) from exc
|
||||||
|
if refresh:
|
||||||
|
store.record_compose()
|
||||||
|
compose_state = store.get_compose_state()
|
||||||
|
|
||||||
|
federated["composed_at"] = compose_state["composed_at"]
|
||||||
|
federated["stale"] = compose_state["stale"]
|
||||||
|
|
||||||
use_yaml = format_param == "yaml" or (accept and "yaml" in accept.lower())
|
use_yaml = format_param == "yaml" or (accept and "yaml" in accept.lower())
|
||||||
headers: dict[str, str] = {}
|
headers: dict[str, str] = {}
|
||||||
|
if compose_state["composed_at"]:
|
||||||
|
headers["X-Composed-At"] = compose_state["composed_at"]
|
||||||
if warnings:
|
if warnings:
|
||||||
headers["X-Federation-Warnings"] = "; ".join(warnings)
|
headers["X-Federation-Warnings"] = "; ".join(warnings)
|
||||||
if use_yaml:
|
if use_yaml:
|
||||||
@@ -118,19 +147,58 @@ def create_app() -> FastAPI:
|
|||||||
return JSONResponse(content=federated, headers=headers)
|
return JSONResponse(content=federated, headers=headers)
|
||||||
|
|
||||||
@app.get("/v1/federated", response_model=None)
|
@app.get("/v1/federated", response_model=None)
|
||||||
def get_federated(
|
async def get_federated(
|
||||||
request: Request,
|
request: Request,
|
||||||
refresh: bool = Query(default=False),
|
refresh: bool = Query(default=False),
|
||||||
format: str = Query(default="json"),
|
format: str = Query(default="json"),
|
||||||
) -> Response:
|
) -> Response:
|
||||||
return _federated_response(refresh, request.headers.get("accept"), format)
|
return await _federated_response(refresh, request.headers.get("accept"), format)
|
||||||
|
|
||||||
@app.post("/v1/federated/compose", response_model=None, dependencies=[Depends(_require_auth)])
|
@app.post("/v1/federated/compose", response_model=None, dependencies=[Depends(_require_auth)])
|
||||||
def compose_federated(
|
async def compose_federated(
|
||||||
request: Request,
|
request: Request,
|
||||||
format: str = Query(default="json"),
|
format: str = Query(default="json"),
|
||||||
) -> Response:
|
) -> Response:
|
||||||
return _federated_response(True, request.headers.get("accept"), format)
|
return await _federated_response(True, request.headers.get("accept"), format)
|
||||||
|
|
||||||
|
@app.post("/v1/webhooks/forgejo", response_model=None)
|
||||||
|
async def forgejo_webhook(request: Request) -> Response:
|
||||||
|
body = await request.body()
|
||||||
|
signature = None
|
||||||
|
for header_name in SIGNATURE_HEADERS:
|
||||||
|
signature = request.headers.get(header_name)
|
||||||
|
if signature:
|
||||||
|
break
|
||||||
|
secret = _webhook_secret()
|
||||||
|
if not secret:
|
||||||
|
raise _http_error(
|
||||||
|
503, "misconfigured", "REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET is not configured"
|
||||||
|
)
|
||||||
|
if not verify_signature(secret, body, signature):
|
||||||
|
raise _http_error(401, "unauthorized", "invalid or missing webhook signature")
|
||||||
|
|
||||||
|
try:
|
||||||
|
payload = json.loads(body)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise _http_error(400, "validation_error", f"invalid JSON payload: {exc}") from exc
|
||||||
|
|
||||||
|
if not push_touches_registry_index(payload):
|
||||||
|
return JSONResponse(content={"accepted": False, "reason": "no registry/indexes/ change"})
|
||||||
|
|
||||||
|
async with compose_lock:
|
||||||
|
store.mark_stale()
|
||||||
|
try:
|
||||||
|
compose_from_store(
|
||||||
|
store, refresh=True, cache_dir=_cache_dir(), domain=DEFAULT_DOMAIN
|
||||||
|
)
|
||||||
|
except FileNotFoundError as exc:
|
||||||
|
# Recompose failed -- leave stale=1 so the next GET/scheduled
|
||||||
|
# trigger reports it honestly rather than silently swallowing
|
||||||
|
# the failure.
|
||||||
|
raise _http_error(502, "compose_error", str(exc)) from exc
|
||||||
|
composed_at = store.record_compose()
|
||||||
|
|
||||||
|
return JSONResponse(content={"accepted": True, "composed_at": composed_at})
|
||||||
|
|
||||||
return app
|
return app
|
||||||
|
|
||||||
|
|||||||
@@ -61,6 +61,23 @@ class HubStore:
|
|||||||
)
|
)
|
||||||
"""
|
"""
|
||||||
)
|
)
|
||||||
|
# Single-row table tracking the composed federated index's
|
||||||
|
# freshness (REUSE-WP-0019-T02). composed_at is set whenever a
|
||||||
|
# real compose (refresh=True) completes; stale is set by the
|
||||||
|
# Forgejo webhook receiver when a registry/indexes/ change is
|
||||||
|
# pushed, and cleared on the next successful compose.
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS compose_state (
|
||||||
|
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||||
|
composed_at TEXT,
|
||||||
|
stale INTEGER NOT NULL DEFAULT 0
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
"INSERT OR IGNORE INTO compose_state (id, composed_at, stale) VALUES (1, NULL, 0)"
|
||||||
|
)
|
||||||
|
|
||||||
def list_repos(self) -> list[dict[str, Any]]:
|
def list_repos(self) -> list[dict[str, Any]]:
|
||||||
with self._connect() as conn:
|
with self._connect() as conn:
|
||||||
@@ -161,3 +178,27 @@ class HubStore:
|
|||||||
"SELECT * FROM registrations ORDER BY repo"
|
"SELECT * FROM registrations ORDER BY repo"
|
||||||
).fetchall()
|
).fetchall()
|
||||||
return [_row_to_registration(row) for row in rows]
|
return [_row_to_registration(row) for row in rows]
|
||||||
|
|
||||||
|
def record_compose(self) -> str:
|
||||||
|
"""Marks the federated index freshly composed (stale cleared).
|
||||||
|
Returns the recorded timestamp."""
|
||||||
|
now = _utc_now()
|
||||||
|
with self._connect() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE compose_state SET composed_at = ?, stale = 0 WHERE id = 1",
|
||||||
|
(now,),
|
||||||
|
)
|
||||||
|
return now
|
||||||
|
|
||||||
|
def mark_stale(self) -> None:
|
||||||
|
with self._connect() as conn:
|
||||||
|
conn.execute("UPDATE compose_state SET stale = 1 WHERE id = 1")
|
||||||
|
|
||||||
|
def get_compose_state(self) -> dict[str, Any]:
|
||||||
|
with self._connect() as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT composed_at, stale FROM compose_state WHERE id = 1"
|
||||||
|
).fetchone()
|
||||||
|
if row is None:
|
||||||
|
return {"composed_at": None, "stale": False}
|
||||||
|
return {"composed_at": row["composed_at"], "stale": bool(row["stale"])}
|
||||||
37
reuse_surface/hub/webhooks.py
Normal file
37
reuse_surface/hub/webhooks.py
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
# Forgejo (Gitea-compatible) signs webhook payloads with HMAC-SHA256 over the
|
||||||
|
# raw request body, sent as a hex digest in X-Forgejo-Signature (Forgejo) or
|
||||||
|
# X-Gitea-Signature (Gitea) -- both forges use the same scheme during the
|
||||||
|
# transition, so both header names are accepted.
|
||||||
|
SIGNATURE_HEADERS = ("X-Forgejo-Signature", "X-Gitea-Signature")
|
||||||
|
|
||||||
|
RELEVANT_PATH_PREFIX = "registry/indexes/"
|
||||||
|
|
||||||
|
|
||||||
|
def verify_signature(secret: str, body: bytes, signature: str | None) -> bool:
|
||||||
|
"""Constant-time HMAC-SHA256 verification. Returns False (never raises)
|
||||||
|
for a missing/malformed signature or a misconfigured (empty) secret --
|
||||||
|
callers must treat False as 'reject', not 'skip verification'."""
|
||||||
|
if not secret or not signature:
|
||||||
|
return False
|
||||||
|
expected = hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
|
||||||
|
return hmac.compare_digest(expected, signature.strip())
|
||||||
|
|
||||||
|
|
||||||
|
def push_touches_registry_index(payload: dict[str, Any]) -> bool:
|
||||||
|
"""True if any commit in a Forgejo/Gitea push-event payload adds,
|
||||||
|
modifies, or removes a path under registry/indexes/. Only inspects
|
||||||
|
paths -- never parses file content (design principle 2: no
|
||||||
|
push-parsing of payloads into registry state, the webhook only
|
||||||
|
triggers a pull-based recompose)."""
|
||||||
|
for commit in payload.get("commits") or []:
|
||||||
|
for key in ("added", "modified", "removed"):
|
||||||
|
for path in commit.get(key) or []:
|
||||||
|
if path.startswith(RELEVANT_PATH_PREFIX):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
@@ -195,18 +195,30 @@ capabilities:
|
|||||||
source_repo: reuse-surface
|
source_repo: reuse-surface
|
||||||
source_url: https://...
|
source_url: https://...
|
||||||
# ... index fields ...
|
# ... index fields ...
|
||||||
|
composed_at: "2026-07-07T16:22:09+00:00" # REUSE-WP-0019-T02
|
||||||
|
stale: false # REUSE-WP-0019-T02
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`composed_at`/`stale` (added REUSE-WP-0019-T02) track *forced* recomposes
|
||||||
|
only (`refresh=true`, the webhook, or the scheduled fallback) — a plain
|
||||||
|
`GET` still serves current best-effort data (per-source `cache_ttl_seconds`
|
||||||
|
still applies) but never silently clears a staleness signal nothing
|
||||||
|
actually refreshed. `composed_at` is `null` until the first forced
|
||||||
|
recompose since the hub process's SQLite DB was created. `stale: true`
|
||||||
|
means a `registry/indexes/` change was pushed (via webhook) since the last
|
||||||
|
forced recompose completed.
|
||||||
|
|
||||||
Query parameters:
|
Query parameters:
|
||||||
|
|
||||||
| Param | Default | Meaning |
|
| Param | Default | Meaning |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `format` | `json` | `json` or `yaml` |
|
| `format` | `json` | `json` or `yaml` |
|
||||||
| `refresh` | `false` | Bypass remote cache when `true` |
|
| `refresh` | `false` | Bypass remote cache when `true`; also updates `composed_at` and clears `stale` |
|
||||||
|
|
||||||
Warnings from compose (duplicate IDs, fetch fallbacks) are returned in response
|
Warnings from compose (duplicate IDs, fetch fallbacks) are returned in response
|
||||||
header `X-Federation-Warnings` (semicolon-separated) for MVP; JSON envelope
|
header `X-Federation-Warnings` (semicolon-separated) for MVP; JSON envelope
|
||||||
extension is a future option.
|
extension is a future option. `composed_at` is also echoed as response header
|
||||||
|
`X-Composed-At` when set.
|
||||||
|
|
||||||
**Response `200`:** Federated index document.
|
**Response `200`:** Federated index document.
|
||||||
**Response `502`:** Required source unavailable with no cache.
|
**Response `502`:** Required source unavailable with no cache.
|
||||||
@@ -215,8 +227,50 @@ extension is a future option.
|
|||||||
|
|
||||||
Trigger federated index refresh (same as `GET /v1/federated?refresh=true`).
|
Trigger federated index refresh (same as `GET /v1/federated?refresh=true`).
|
||||||
**Auth required.** Useful for operators after bulk registration changes.
|
**Auth required.** Useful for operators after bulk registration changes.
|
||||||
|
This is the hub's recompose endpoint referenced elsewhere as "trigger a
|
||||||
|
recompose" — there is no separate `/v1/recompose` route; this one already
|
||||||
|
does that job.
|
||||||
|
|
||||||
**Response `200`:** Federated index document.
|
**Response `200`:** Federated index document (with `composed_at` updated,
|
||||||
|
`stale` cleared).
|
||||||
|
|
||||||
|
### 5.9 `POST /v1/webhooks/forgejo`
|
||||||
|
|
||||||
|
Forgejo (or Gitea, during the transition) push-event webhook receiver.
|
||||||
|
Added REUSE-WP-0019-T02. Design: **webhook triggers, compose stays
|
||||||
|
pull-based** — the webhook never parses pushed file content into registry
|
||||||
|
state; it only decides whether to trigger a pull-based recompose from the
|
||||||
|
already-registered raw URLs.
|
||||||
|
|
||||||
|
**Signature verification (required, not optional):** HMAC-SHA256 over the
|
||||||
|
raw request body, hex-encoded, in `X-Forgejo-Signature` (or
|
||||||
|
`X-Gitea-Signature` — both accepted, since Forgejo is Gitea-compatible and
|
||||||
|
repos migrate independently). Secret from `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET`
|
||||||
|
(never in code or committed config — route via the standard credential
|
||||||
|
mechanism for this deployment). A missing/wrong signature is `401`; an
|
||||||
|
unconfigured secret is `503` (fails closed, not open).
|
||||||
|
|
||||||
|
**Behavior:**
|
||||||
|
|
||||||
|
1. Verify signature (raw body, constant-time compare).
|
||||||
|
2. Parse the push-event payload; check every commit's `added`/`modified`/`removed`
|
||||||
|
lists for any path under `registry/indexes/`.
|
||||||
|
3. If none match: `200 {"accepted": false, "reason": "no registry/indexes/ change"}` — no-op.
|
||||||
|
4. If a match: mark the compose state stale, run a real recompose
|
||||||
|
(`refresh=true` equivalent) under the same lock used by
|
||||||
|
`POST /v1/federated/compose` (concurrent triggers coalesce rather than
|
||||||
|
overlap), record `composed_at`, clear `stale`.
|
||||||
|
|
||||||
|
**Response `200`:** `{"accepted": true|false, ...}`.
|
||||||
|
**Response `401`:** Missing or invalid signature.
|
||||||
|
**Response `503`:** Webhook secret not configured.
|
||||||
|
**Response `502`:** Recompose failed (stale is deliberately left set so the
|
||||||
|
next check reports the failure honestly, not silently).
|
||||||
|
|
||||||
|
Org-level webhook rollout (single config covering all repos, T03) and the
|
||||||
|
Forgejo Actions scheduled fallback for when webhooks are unavailable are
|
||||||
|
separate, deployment-side follow-ups — this section covers the receiver
|
||||||
|
contract only.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -238,6 +292,7 @@ Non-2xx responses use:
|
|||||||
| `unauthorized` | 401 |
|
| `unauthorized` | 401 |
|
||||||
| `not_found` | 404 |
|
| `not_found` | 404 |
|
||||||
| `conflict` | 409 |
|
| `conflict` | 409 |
|
||||||
|
| `misconfigured` | 503 |
|
||||||
| `compose_error` | 502 |
|
| `compose_error` | 502 |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -250,6 +305,8 @@ Non-2xx responses use:
|
|||||||
| `REUSE_SURFACE_DB` | no | SQLite path (default `/data/reuse.db`) |
|
| `REUSE_SURFACE_DB` | no | SQLite path (default `/data/reuse.db`) |
|
||||||
| `REUSE_SURFACE_CACHE_DIR` | no | Remote index cache (default `/data/cache`) |
|
| `REUSE_SURFACE_CACHE_DIR` | no | Remote index cache (default `/data/cache`) |
|
||||||
| `REUSE_SURFACE_DOMAIN` | no | Default federated `domain` (default `helix_forge`) |
|
| `REUSE_SURFACE_DOMAIN` | no | Default federated `domain` (default `helix_forge`) |
|
||||||
|
| `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` | for webhook | HMAC secret for `POST /v1/webhooks/forgejo` (REUSE-WP-0019-T02) |
|
||||||
|
| `REUSE_SURFACE_FORGE_BASE_URL` | no | Default target host for `reuse-surface federation migrate-host` (REUSE-WP-0019-T01), e.g. `https://forgejo.coulomb.social` |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -131,3 +131,184 @@ def test_store_validation(tmp_path):
|
|||||||
store = HubStore(tmp_path / "hub.db")
|
store = HubStore(tmp_path / "hub.db")
|
||||||
with pytest.raises(ValueError):
|
with pytest.raises(ValueError):
|
||||||
store.create_repo({"repo": "BAD", "url": "ftp://x", "domain": "helix_forge"})
|
store.create_repo({"repo": "BAD", "url": "ftp://x", "domain": "helix_forge"})
|
||||||
|
|
||||||
|
|
||||||
|
# --- T02: composed_at / stale tracking ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_compose_state_starts_unset(tmp_path):
|
||||||
|
store = HubStore(tmp_path / "hub.db")
|
||||||
|
state = store.get_compose_state()
|
||||||
|
assert state == {"composed_at": None, "stale": False}
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_compose_sets_timestamp_and_clears_stale(tmp_path):
|
||||||
|
store = HubStore(tmp_path / "hub.db")
|
||||||
|
store.mark_stale()
|
||||||
|
assert store.get_compose_state()["stale"] is True
|
||||||
|
composed_at = store.record_compose()
|
||||||
|
state = store.get_compose_state()
|
||||||
|
assert state["composed_at"] == composed_at
|
||||||
|
assert state["stale"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_plain_get_does_not_clear_stale(hub_client, monkeypatch):
|
||||||
|
hub_client.post(
|
||||||
|
"/v1/repos",
|
||||||
|
json={"repo": "remote-repo", "url": "https://example.com/capabilities.yaml", "domain": "helix_forge"},
|
||||||
|
headers={"Authorization": "Bearer test-token"},
|
||||||
|
)
|
||||||
|
payload = REMOTE_INDEX.encode("utf-8")
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return payload
|
||||||
|
|
||||||
|
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||||
|
# force a real compose so composed_at/stale are established
|
||||||
|
first = hub_client.get("/v1/federated?refresh=true")
|
||||||
|
assert first.json()["stale"] is False
|
||||||
|
composed_at_1 = first.json()["composed_at"]
|
||||||
|
|
||||||
|
# a plain GET (no refresh) must not report itself as freshly composed
|
||||||
|
second = hub_client.get("/v1/federated")
|
||||||
|
assert second.json()["composed_at"] == composed_at_1
|
||||||
|
assert second.json()["stale"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_federated_reports_stale_after_mark(hub_client, monkeypatch):
|
||||||
|
hub_client.post(
|
||||||
|
"/v1/repos",
|
||||||
|
json={"repo": "remote-repo", "url": "https://example.com/capabilities.yaml", "domain": "helix_forge"},
|
||||||
|
headers={"Authorization": "Bearer test-token"},
|
||||||
|
)
|
||||||
|
payload = REMOTE_INDEX.encode("utf-8")
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return payload
|
||||||
|
|
||||||
|
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||||
|
hub_client.get("/v1/federated?refresh=true")
|
||||||
|
|
||||||
|
from reuse_surface.hub.store import HubStore as _HubStore
|
||||||
|
|
||||||
|
db_path = os.environ["REUSE_SURFACE_DB"]
|
||||||
|
_HubStore(Path(db_path)).mark_stale()
|
||||||
|
|
||||||
|
response = hub_client.get("/v1/federated")
|
||||||
|
assert response.json()["stale"] is True
|
||||||
|
|
||||||
|
|
||||||
|
# --- T02: Forgejo webhook receiver ---
|
||||||
|
|
||||||
|
WEBHOOK_SECRET = "test-webhook-secret"
|
||||||
|
|
||||||
|
|
||||||
|
def _sign(body: bytes, secret: str = WEBHOOK_SECRET) -> str:
|
||||||
|
import hashlib
|
||||||
|
import hmac as hmac_module
|
||||||
|
|
||||||
|
return hmac_module.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def webhook_client(hub_client, monkeypatch):
|
||||||
|
monkeypatch.setenv("REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET", WEBHOOK_SECRET)
|
||||||
|
return hub_client
|
||||||
|
|
||||||
|
|
||||||
|
def test_webhook_rejects_missing_signature(webhook_client):
|
||||||
|
response = webhook_client.post("/v1/webhooks/forgejo", json={"commits": []})
|
||||||
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_webhook_rejects_wrong_signature(webhook_client):
|
||||||
|
body = json.dumps({"commits": []}).encode("utf-8")
|
||||||
|
response = webhook_client.post(
|
||||||
|
"/v1/webhooks/forgejo",
|
||||||
|
content=body,
|
||||||
|
headers={"X-Forgejo-Signature": "0" * 64, "Content-Type": "application/json"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 401
|
||||||
|
|
||||||
|
|
||||||
|
def test_webhook_rejects_when_secret_not_configured(hub_client):
|
||||||
|
body = json.dumps({"commits": []}).encode("utf-8")
|
||||||
|
response = hub_client.post(
|
||||||
|
"/v1/webhooks/forgejo",
|
||||||
|
content=body,
|
||||||
|
headers={"X-Forgejo-Signature": _sign(body), "Content-Type": "application/json"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 503
|
||||||
|
|
||||||
|
|
||||||
|
def test_webhook_ignores_push_without_registry_change(webhook_client):
|
||||||
|
body = json.dumps({"commits": [{"added": ["README.md"], "modified": [], "removed": []}]}).encode("utf-8")
|
||||||
|
response = webhook_client.post(
|
||||||
|
"/v1/webhooks/forgejo",
|
||||||
|
content=body,
|
||||||
|
headers={"X-Forgejo-Signature": _sign(body), "Content-Type": "application/json"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json()["accepted"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_webhook_triggers_recompose_on_registry_change(webhook_client, monkeypatch):
|
||||||
|
webhook_client.post(
|
||||||
|
"/v1/repos",
|
||||||
|
json={"repo": "remote-repo", "url": "https://example.com/capabilities.yaml", "domain": "helix_forge"},
|
||||||
|
headers={"Authorization": "Bearer test-token"},
|
||||||
|
)
|
||||||
|
payload_bytes = REMOTE_INDEX.encode("utf-8")
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return payload_bytes
|
||||||
|
|
||||||
|
body = json.dumps(
|
||||||
|
{"commits": [{"added": ["registry/indexes/capabilities.yaml"], "modified": [], "removed": []}]}
|
||||||
|
).encode("utf-8")
|
||||||
|
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||||
|
response = webhook_client.post(
|
||||||
|
"/v1/webhooks/forgejo",
|
||||||
|
content=body,
|
||||||
|
headers={"X-Forgejo-Signature": _sign(body), "Content-Type": "application/json"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
assert response.json()["accepted"] is True
|
||||||
|
assert response.json()["composed_at"]
|
||||||
|
|
||||||
|
# federated index should now report the newly composed data, not stale
|
||||||
|
follow_up = webhook_client.get("/v1/federated")
|
||||||
|
assert follow_up.json()["stale"] is False
|
||||||
|
ids = {item["id"] for item in follow_up.json()["capabilities"]}
|
||||||
|
assert "capability.remote.sample" in ids
|
||||||
|
|
||||||
|
|
||||||
|
def test_webhook_accepts_gitea_signature_header(webhook_client):
|
||||||
|
body = json.dumps({"commits": [{"added": ["README.md"], "modified": [], "removed": []}]}).encode("utf-8")
|
||||||
|
response = webhook_client.post(
|
||||||
|
"/v1/webhooks/forgejo",
|
||||||
|
content=body,
|
||||||
|
headers={"X-Gitea-Signature": _sign(body), "Content-Type": "application/json"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
82
tests/test_webhooks.py
Normal file
82
tests/test_webhooks.py
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
|
||||||
|
from reuse_surface.hub.webhooks import push_touches_registry_index, verify_signature
|
||||||
|
|
||||||
|
SECRET = "shh"
|
||||||
|
|
||||||
|
|
||||||
|
def _sign(body: bytes, secret: str = SECRET) -> str:
|
||||||
|
return hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def test_verify_signature_valid():
|
||||||
|
body = b'{"commits": []}'
|
||||||
|
assert verify_signature(SECRET, body, _sign(body)) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_verify_signature_wrong_secret():
|
||||||
|
body = b'{"commits": []}'
|
||||||
|
assert verify_signature("different-secret", body, _sign(body)) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_verify_signature_tampered_body():
|
||||||
|
body = b'{"commits": []}'
|
||||||
|
signature = _sign(body)
|
||||||
|
assert verify_signature(SECRET, b'{"commits": [1]}', signature) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_verify_signature_missing_signature():
|
||||||
|
assert verify_signature(SECRET, b"{}", None) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_verify_signature_empty_secret_rejects():
|
||||||
|
body = b"{}"
|
||||||
|
# Even if a caller accidentally computed a signature with an empty
|
||||||
|
# secret, verify_signature must not accept it -- an unconfigured
|
||||||
|
# secret is not the same as "verification not needed".
|
||||||
|
assert verify_signature("", body, _sign(body, secret="")) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_touches_registry_index_added():
|
||||||
|
payload = {"commits": [{"added": ["registry/indexes/capabilities.yaml"], "modified": [], "removed": []}]}
|
||||||
|
assert push_touches_registry_index(payload) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_touches_registry_index_modified():
|
||||||
|
payload = {"commits": [{"added": [], "modified": ["registry/indexes/federated.yaml"], "removed": []}]}
|
||||||
|
assert push_touches_registry_index(payload) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_touches_registry_index_removed():
|
||||||
|
payload = {"commits": [{"added": [], "modified": [], "removed": ["registry/indexes/capabilities.yaml"]}]}
|
||||||
|
assert push_touches_registry_index(payload) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_does_not_touch_registry_index():
|
||||||
|
payload = {"commits": [{"added": ["README.md"], "modified": ["src/main.py"], "removed": []}]}
|
||||||
|
assert push_touches_registry_index(payload) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_touches_registry_index_across_multiple_commits():
|
||||||
|
payload = {
|
||||||
|
"commits": [
|
||||||
|
{"added": ["README.md"], "modified": [], "removed": []},
|
||||||
|
{"added": [], "modified": ["registry/indexes/capabilities.yaml"], "removed": []},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
assert push_touches_registry_index(payload) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_touches_registry_index_empty_commits():
|
||||||
|
assert push_touches_registry_index({"commits": []}) is False
|
||||||
|
assert push_touches_registry_index({}) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_push_touches_registry_index_ignores_similarly_named_paths():
|
||||||
|
# a path that merely starts with "registry" but isn't under
|
||||||
|
# registry/indexes/ must not match
|
||||||
|
payload = {"commits": [{"added": ["registry/capabilities/capability.foo.md"], "modified": [], "removed": []}]}
|
||||||
|
assert push_touches_registry_index(payload) is False
|
||||||
@@ -113,19 +113,48 @@ Implemented:
|
|||||||
|
|
||||||
```task
|
```task
|
||||||
id: REUSE-WP-0019-T02
|
id: REUSE-WP-0019-T02
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "691eb32a-6f20-4a2a-b9ff-0ae427b659aa"
|
state_hub_task_id: "691eb32a-6f20-4a2a-b9ff-0ae427b659aa"
|
||||||
```
|
```
|
||||||
|
|
||||||
- Hub service (`reuse_surface/serve`): `POST /v1/recompose` (token-auth) —
|
**Note:** `POST /v1/federated/compose` (token-auth, triggers a real
|
||||||
marks index stale and triggers recompose from registered raw URLs
|
recompose) already existed from earlier hub work — no separate
|
||||||
- `POST /v1/webhooks/forgejo`: validates Forgejo webhook signature
|
`/v1/recompose` route was added; the spec now documents this explicitly
|
||||||
(`X-Forgejo-Signature`, secret from env), accepts push events, triggers
|
rather than duplicating a route that already does the job.
|
||||||
recompose only when the pushed commits touch `registry/indexes/`
|
|
||||||
- Debounce/coalesce concurrent triggers; `GET /v1/federated` gains
|
Implemented in `reuse_surface/hub/`:
|
||||||
`composed_at` + `stale` fields
|
|
||||||
- Extend `specs/FederationHubAPI.md`; pytest with signed fixture payloads
|
- `store.py`: `compose_state` table (`composed_at`, `stale`), with
|
||||||
|
`record_compose()`/`mark_stale()`/`get_compose_state()`. `composed_at`
|
||||||
|
updates and `stale` clears only on a *forced* recompose (`refresh=true`,
|
||||||
|
webhook, or future scheduled fallback) — a plain `GET` still serves
|
||||||
|
current best-effort data but never silently reports itself as freshly
|
||||||
|
composed
|
||||||
|
- `webhooks.py`: `verify_signature` (constant-time HMAC-SHA256, fails
|
||||||
|
closed on empty secret), `push_touches_registry_index` (path-only
|
||||||
|
inspection of the push payload's `added`/`modified`/`removed` lists —
|
||||||
|
never parses file content, per design principle 2)
|
||||||
|
- `app.py`: `POST /v1/webhooks/forgejo` (accepts both `X-Forgejo-Signature`
|
||||||
|
and `X-Gitea-Signature`, since repos migrate independently);
|
||||||
|
`GET /v1/federated` and `POST /v1/federated/compose` now share an
|
||||||
|
`asyncio.Lock` so concurrent recompose triggers (manual, webhook, future
|
||||||
|
scheduled) coalesce instead of overlapping
|
||||||
|
- `specs/FederationHubAPI.md` extended (§5.7-5.9, config table, error
|
||||||
|
codes)
|
||||||
|
- 28 new pytest cases (16 in `test_hub.py`, 12 in `test_webhooks.py`); 128
|
||||||
|
total pass
|
||||||
|
- **Live-verified**: ran the actual hub service locally
|
||||||
|
(`reuse-surface serve`), sent a real HMAC-signed webhook payload over
|
||||||
|
HTTP — confirmed `composed_at`/`stale` transitions, signature rejection,
|
||||||
|
irrelevant-path no-op, and the full webhook-to-recompose path end to end
|
||||||
|
|
||||||
|
**Deployment boundary — deliberately not done:** this changes the hub
|
||||||
|
*service source code* in this repo; it does **not** deploy to the live
|
||||||
|
`reuse.coulomb.social` hub. That requires a container rebuild, registry
|
||||||
|
push, and Kubernetes rollout — a production deployment action affecting a
|
||||||
|
shared external service, out of scope for this task without explicit
|
||||||
|
sign-off. See `docs/deploy/reuse-kubernetes.md`.
|
||||||
|
|
||||||
## Forgejo Webhook Rollout And Scheduled Fallback
|
## Forgejo Webhook Rollout And Scheduled Fallback
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user