Complete WP-0010: HTTP remote federation with cache

Extend federation manifest schema for url sources with auth and TTL metadata. Fetch remote capability indexes over HTTP(S), cache under registry/federation/cache/, and fall back to stale cache on fetch failure. Add --refresh flag, seven federation tests, and updated federation docs.
2026-06-15 02:28:44 +02:00
parent c9b957d398
commit e8797b2e91
13 changed files with 487 additions and 45 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,7 @@
+# Federation remote index cache (keep directory via .gitkeep)
+registry/federation/cache/*
+!registry/federation/cache/.gitkeep
+
 # ---> Python
 # Byte-compiled / optimized / DLL files
 __pycache__/
--- a/SCOPE.md
+++ b/SCOPE.md
@@ -55,6 +55,7 @@ and agents can:
 - **Browse a searchable catalog** at `docs/catalog/search.html` (client-side
  filter over `registry.json`)
 - **Compose federated indexes** with `reuse-surface federation compose`
+  (local paths and remote HTTP URLs with cache)
 - **Generate relation graphs** with `reuse-surface graph`
 - **Explore relations interactively** at `docs/graph/index.html`
 - **Avoid duplicates** by querying the index and checking overlaps before adding entries
@@ -65,7 +66,7 @@ the index, and CLI automation.

 ## What Is Not Possible Yet

- Network-based federation or cross-org index sync
+- Live cross-org index sync (polling/webhooks) beyond on-demand HTTP fetch
 - Packaged releases beyond local `pip install -e .` and Gitea CI validation

 See `tools/README.md` for command reference.
@@ -84,7 +85,8 @@ See `tools/README.md` for command reference.
  `docs/graph/index.html`.
 - Searchable catalog: `docs/catalog/search.html`.
 - Test suite: `tests/` (pytest).
- Finished workplans: `REUSE-WP-0001` through `REUSE-WP-0009`.
+- Remote federation: HTTP URL sources with cache in `registry/federation/cache/`.
+- Finished workplans: `REUSE-WP-0001` through `REUSE-WP-0010`.
 - **Self-assessed vector:** `D5 / A3 / C4 / R3` (see gap analysis).

 ## Repository Layout
--- a/docs/IntentScopeGapAnalysis.md
+++ b/docs/IntentScopeGapAnalysis.md
@@ -18,13 +18,12 @@ with **A3 CLI tooling** (`validate`, `query`, `export`) atop Markdown-first
 authoring.

 The two documents are **directionally aligned** on registry-first reuse, four
-maturity dimensions, and human/agent consumers. REUSE-WP-0003 through
-REUSE-WP-0009 closed the priority gaps from section 8 except network
-federation. Remaining gaps are primarily scale and cross-org sync:
+maturity dimensions, and human/agent consumers. REUSE-WP-0003 through REUSE-WP-0010 closed the priority gaps from section 8.
+Remaining gaps are primarily document cross-coverage and operational polish:

-1. **Network federation** — local compose only; no remote index fetch.
-2. **Document cross-coverage** — SCOPE still carries operational detail INTENT
+1. **Document cross-coverage** — SCOPE still carries operational detail INTENT
   omits; INTENT success criteria are not fully enumerated in SCOPE.
+2. **Live federation sync** — on-demand HTTP fetch only; no polling or webhooks.

 **Current reuse-surface vector (self-assessment):** `D5 / A3 / C4 / R3`

@@ -276,7 +275,7 @@ core commands. Individual registered capabilities may carry their own evidence
 | Priority | Gap | Outcome | Status |
 |---|---|---|---|
 | 13 | Interactive catalog | `docs/catalog/search.html` + `registry.json` | Closed (WP-0007) |
-| 15 | Network federation | Remote index fetch and cross-org sync | Open (WP-0010) |
+| 15 | Network federation | HTTP URL sources + cache in `federation compose` | Closed (WP-0010) |
 | 16 | Graph UI | `docs/graph/index.html` explorer | Closed (WP-0008) |

 ---
@@ -303,3 +302,4 @@ core commands. Individual registered capabilities may carry their own evidence
 | 2026-06-15 | REUSE-WP-0007 closed priority 13 (searchable catalog UI) |
 | 2026-06-15 | REUSE-WP-0008 closed priority 16 (graph explorer) |
 | 2026-06-15 | REUSE-WP-0009 added pytest suite and CI fail-on-warnings; vector R3 |
+| 2026-06-15 | REUSE-WP-0010 closed priority 15 (HTTP remote federation + cache) |
--- a/docs/RegistryFederation.md
+++ b/docs/RegistryFederation.md
@@ -11,6 +11,10 @@ helix_forge capabilities may be registered in multiple repositories. Federation
 composes capability indexes from configured sources into a single discovery
 surface without silently merging duplicate IDs.

+Sources may be **local filesystem paths** or **remote HTTP(S) URLs** (git raw
+endpoints, published index artifacts, etc.). Remote indexes are cached under
+`registry/federation/cache/` for offline reuse and faster compose.
+
 ## Manifest

 `registry/federation/sources.yaml` lists index sources:
@@ -24,6 +28,14 @@ sources:
    index: registry/indexes/capabilities.yaml
    enabled: true
    required: true
+
+  - repo: sibling-repo
+    url: https://git.example.com/org/sibling-repo/raw/main/registry/indexes/capabilities.yaml
+    enabled: false
+    required: false
+    cache_ttl_seconds: 86400
+    auth_env: FEDERATION_TOKEN
+    auth_header: Authorization
 ```

 Schema: `schemas/federation.schema.yaml`
@@ -33,26 +45,44 @@ Schema: `schemas/federation.schema.yaml`
 | Field | Meaning |
 |---|---|
 | `repo` | Source repository slug |
-| `index` | Path to `capabilities.yaml` (repo-relative or `~/...`) |
+| `index` | Local path to `capabilities.yaml` (repo-relative or `~/...`) |
+| `url` | Remote HTTP(S) URL to a `capabilities.yaml` index |
 | `enabled` | Include this source in compose |
-| `required` | Fail compose if index missing when enabled |
+| `required` | Fail compose if index missing or remote fetch fails with no cache |
 | `domain` | Optional domain label |
+| `cache_ttl_seconds` | Reuse cached remote index for this many seconds (`0` = always refetch) |
+| `auth_env` | Environment variable holding token or full header value for `url` sources |
+| `auth_header` | HTTP header for `auth_env` (default `Authorization`) |
+
+Each source must specify **either** `index` **or** `url`, not both.

 Sibling repos (`state-hub`, `feature-control`, `identity-canon`) are listed as
-disabled placeholders until they publish registry indexes.
+disabled local placeholders until they publish registry indexes. A disabled
+`example-remote` URL source illustrates HTTP federation.

 ## Compose workflow

 ```bash
 reuse-surface federation compose
+reuse-surface federation compose --refresh   # bypass remote cache
 ```

 Writes `registry/indexes/federated.yaml` with:

 - Merged `capabilities` from all enabled sources
 - `source_repo` and `source_index` on every row
+- `source_url` when the row came from a remote source
 - `collision_policy` and per-source counts

+### Remote cache
+
+Fetched URL indexes are stored at `registry/federation/cache/<repo>.yaml` with
+metadata in `<repo>.meta.yaml`. The cache directory is gitignored; only
+`.gitkeep` is tracked.
+
+When a refetch fails, compose reuses a stale cache and emits a warning. Required
+remote sources without cache fail compose with a clear error.
+
 ### Collision policy

 `warn` (default): duplicate IDs across sources are kept but reported as
@@ -62,9 +92,17 @@ warnings. Consumers must inspect `source_repo` before choosing an entry.

 1. Run `reuse-surface federation compose` after manifest or sibling index changes.
 2. Read `registry/indexes/federated.yaml` for cross-repo discovery.
-3. Open `path` in the source repo for full entry detail when local.
+3. Open `path` in the source repo for full entry detail when local; follow
+   `source_url` / `source_index` when remote.
 4. Run `reuse-surface graph --check` before relying on relation navigation.

+### Cross-repo discovery without local checkout
+
+Enable a `url` source pointing at a published raw index (Gitea, GitHub, static
+host). Set `auth_env` when the endpoint requires a token. Agents on machines
+without sibling repo clones can still compose a federated view from HTTP sources
+plus the local `reuse-surface` index.
+
 ## Relation graphs

 ```bash
@@ -82,9 +120,12 @@ federated ID set.
 Gitea CI runs:

 ```bash
-reuse-surface validate --relations
+reuse-surface validate --relations --fail-on-warnings
 reuse-surface federation compose
+reuse-surface catalog
+reuse-surface graph --check --fail-on-warnings
+pytest -q
 ```

-Warnings on broken relations or missing optional sibling indexes do not fail CI;
-schema validation errors do.
+CI uses local sources only (remote examples are disabled). Warnings on missing
+optional sibling indexes do not fail CI; schema validation errors do.
--- a/registry/federation/cache/.gitkeep
+++ b/registry/federation/cache/.gitkeep
--- a/registry/federation/sources.yaml
+++ b/registry/federation/sources.yaml
@@ -33,3 +33,12 @@ sources:
    required: false
    domain: helix_forge
    description: Identity canon research capabilities
+
+  # Remote index example — enable when a repo publishes a raw capabilities.yaml URL
+  - repo: example-remote
+    url: https://example.com/registry/indexes/capabilities.yaml
+    enabled: false
+    required: false
+    domain: helix_forge
+    cache_ttl_seconds: 86400
+    description: Illustrative HTTP federation source (disabled)
--- a/registry/indexes/federated.yaml
+++ b/registry/indexes/federated.yaml
@@ -6,8 +6,8 @@ domain: helix_forge
 collision_policy: warn
 sources:
 - repo: reuse-surface
-  index: registry/indexes/capabilities.yaml
  count: 12
+  index: registry/indexes/capabilities.yaml
 capabilities:
 - id: capability.activity.event-coordinate
  name: Organizational Event Coordination
--- a/reuse_surface/cli.py
+++ b/reuse_surface/cli.py
@@ -146,7 +146,7 @@ def cmd_overlaps(args: argparse.Namespace) -> int:

 def cmd_federation_compose(args: argparse.Namespace) -> int:
    try:
-        target, warnings = write_federated_index()
+        target, warnings = write_federated_index(refresh=args.refresh)
    except (FileNotFoundError, ValueError) as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 1
@@ -258,6 +258,11 @@ def main(argv: list[str] | None = None) -> int:
    )
    federation_sub = federation.add_subparsers(dest="federation_command", required=True)
    compose = federation_sub.add_parser("compose", help="compose federated index")
+    compose.add_argument(
+        "--refresh",
+        action="store_true",
+        help="bypass remote index cache and refetch URL sources",
+    )
    compose.set_defaults(func=cmd_federation_compose)

    query = subparsers.add_parser("query", help="query capability index")
--- a/reuse_surface/federation.py
+++ b/reuse_surface/federation.py
@@ -1,7 +1,10 @@
 from __future__ import annotations

-import sys
-from datetime import date
+import os
+import re
+import urllib.error
+import urllib.request
+from datetime import date, datetime, timezone
 from pathlib import Path
 from typing import Any

@@ -13,6 +16,9 @@ from reuse_surface.registry import ROOT
 MANIFEST_PATH = ROOT / "registry" / "federation" / "sources.yaml"
 SCHEMA_PATH = ROOT / "schemas" / "federation.schema.yaml"
 FEDERATED_INDEX_PATH = ROOT / "registry" / "indexes" / "federated.yaml"
+CACHE_DIR = ROOT / "registry" / "federation" / "cache"
+DEFAULT_CACHE_TTL_SECONDS = 86400
+USER_AGENT = "reuse-surface/0.1 federation-compose"


 def _expand_path(index_path: str) -> Path:
@@ -39,8 +45,139 @@ def _resolve_index_path(index_value: str) -> Path:
    return path


+def _safe_repo_slug(repo: str) -> str:
+    return re.sub(r"[^\w.-]", "_", repo)
+
+
+def _path_label(path: Path) -> str:
+    try:
+        return str(path.relative_to(ROOT))
+    except ValueError:
+        return str(path)
+
+
+def _cache_paths(repo: str) -> tuple[Path, Path]:
+    slug = _safe_repo_slug(repo)
+    return CACHE_DIR / f"{slug}.yaml", CACHE_DIR / f"{slug}.meta.yaml"
+
+
+def _read_cache_meta(meta_path: Path) -> dict[str, Any] | None:
+    if not meta_path.exists():
+        return None
+    data = yaml.safe_load(meta_path.read_text(encoding="utf-8"))
+    return data if isinstance(data, dict) else None
+
+
+def _cache_is_fresh(meta: dict[str, Any], ttl_seconds: int) -> bool:
+    if ttl_seconds <= 0:
+        return False
+    fetched_at = meta.get("fetched_at")
+    if not fetched_at:
+        return False
+    fetched = datetime.fromisoformat(str(fetched_at))
+    if fetched.tzinfo is None:
+        fetched = fetched.replace(tzinfo=timezone.utc)
+    age = (datetime.now(timezone.utc) - fetched).total_seconds()
+    return age < ttl_seconds
+
+
+def _auth_headers(source: dict[str, Any]) -> dict[str, str]:
+    auth_env = source.get("auth_env")
+    if not auth_env:
+        return {}
+    token = os.environ.get(auth_env)
+    if not token:
+        raise ValueError(
+            f"auth env {auth_env} is not set for remote source {source['repo']}"
+        )
+    header_name = source.get("auth_header", "Authorization")
+    if header_name.lower() == "authorization" and not token.lower().startswith(
+        ("bearer ", "basic ")
+    ):
+        token = f"Bearer {token}"
+    return {header_name: token}
+
+
+def fetch_remote_index_text(url: str, source: dict[str, Any]) -> str:
+    headers = {"User-Agent": USER_AGENT, **_auth_headers(source)}
+    request = urllib.request.Request(url, headers=headers)
+    try:
+        with urllib.request.urlopen(request, timeout=30) as response:
+            return response.read().decode("utf-8")
+    except urllib.error.HTTPError as exc:
+        raise ConnectionError(
+            f"HTTP {exc.code} fetching {url} for {source['repo']}"
+        ) from exc
+    except urllib.error.URLError as exc:
+        raise ConnectionError(
+            f"failed to fetch {url} for {source['repo']}: {exc.reason}"
+        ) from exc
+
+
+def _write_remote_cache(repo: str, url: str, content: str) -> Path:
+    CACHE_DIR.mkdir(parents=True, exist_ok=True)
+    index_path, meta_path = _cache_paths(repo)
+    index_path.write_text(content, encoding="utf-8")
+    meta = {
+        "fetched_at": datetime.now(timezone.utc).isoformat(),
+        "url": url,
+        "repo": repo,
+    }
+    meta_path.write_text(yaml.safe_dump(meta, sort_keys=False), encoding="utf-8")
+    return index_path
+
+
+def resolve_source_index_path(
+    source: dict[str, Any],
+    *,
+    refresh: bool = False,
+) -> tuple[Path | None, list[str]]:
+    warnings: list[str] = []
+    if "index" in source:
+        path = _resolve_index_path(source["index"])
+        if not path.exists():
+            message = f"missing index for {source['repo']}: {path}"
+            if source.get("required", False):
+                raise FileNotFoundError(message)
+            warnings.append(message)
+            return None, warnings
+        return path, warnings
+
+    url = source["url"]
+    ttl_seconds = int(source.get("cache_ttl_seconds", DEFAULT_CACHE_TTL_SECONDS))
+    index_path, meta_path = _cache_paths(source["repo"])
+    meta = _read_cache_meta(meta_path)
+    use_cache = (
+        index_path.exists()
+        and meta is not None
+        and meta.get("url") == url
+        and _cache_is_fresh(meta, ttl_seconds)
+        and not refresh
+    )
+    if use_cache:
+        return index_path, warnings
+
+    try:
+        content = fetch_remote_index_text(url, source)
+    except (ConnectionError, ValueError) as exc:
+        if index_path.exists():
+            warnings.append(
+                f"remote fetch failed for {source['repo']}, using stale cache: {exc}"
+            )
+            return index_path, warnings
+        message = f"remote index unavailable for {source['repo']}: {exc}"
+        if source.get("required", False):
+            raise FileNotFoundError(message) from exc
+        warnings.append(message)
+        return None, warnings
+
+    return _write_remote_cache(source["repo"], url, content), warnings
+
+
 def compose_federated_index(
    manifest: dict[str, Any] | None = None,
+    *,
+    refresh: bool = False,
 ) -> tuple[dict[str, Any], list[str]]:
    manifest = manifest or load_federation_manifest()
    warnings: list[str] = []
@@ -51,12 +188,11 @@ def compose_federated_index(
    for source in manifest["sources"]:
        if not source.get("enabled", False):
            continue
-        index_path = _resolve_index_path(source["index"])
-        if not index_path.exists():
-            message = f"missing index for {source['repo']}: {index_path}"
-            if source.get("required", False):
-                raise FileNotFoundError(message)
-            warnings.append(message)
+        index_path, source_warnings = resolve_source_index_path(
+            source, refresh=refresh
+        )
+        warnings.extend(source_warnings)
+        if index_path is None:
            continue
        with index_path.open(encoding="utf-8") as handle:
            index_data = yaml.safe_load(handle)
@@ -71,16 +207,23 @@ def compose_federated_index(
                seen_ids[cap_id] = source["repo"]
            federated_item = dict(item)
            federated_item["source_repo"] = source["repo"]
+            if "url" in source:
+                federated_item["source_url"] = source["url"]
+                federated_item["source_index"] = _path_label(index_path)
+            else:
                federated_item["source_index"] = source["index"]
            merged.append(federated_item)
            count += 1
-        source_summaries.append(
-            {
+        summary: dict[str, Any] = {
            "repo": source["repo"],
-                "index": source["index"],
            "count": count,
        }
-        )
+        if "url" in source:
+            summary["url"] = source["url"]
+            summary["cache"] = _path_label(index_path)
+        else:
+            summary["index"] = source["index"]
+        source_summaries.append(summary)

    federated = {
        "version": manifest.get("version", 1),
@@ -96,8 +239,10 @@ def compose_federated_index(
 def write_federated_index(
    output_path: Path | None = None,
    manifest: dict[str, Any] | None = None,
+    *,
+    refresh: bool = False,
 ) -> tuple[Path, list[str]]:
-    federated, warnings = compose_federated_index(manifest)
+    federated, warnings = compose_federated_index(manifest, refresh=refresh)
    target = output_path or FEDERATED_INDEX_PATH
    target.parent.mkdir(parents=True, exist_ok=True)
    header = (
--- a/schemas/federation.schema.yaml
+++ b/schemas/federation.schema.yaml
@@ -2,8 +2,8 @@ $schema: https://json-schema.org/draft/2020-12/schema
 $id: https://reuse-surface.local/schemas/federation.schema.yaml
 title: Registry Federation Manifest
 description: >
-  Schema for registry/federation/sources.yaml. Describes local and sibling
-  capability index sources to compose into a federated index.
+  Schema for registry/federation/sources.yaml. Describes local filesystem and
+  remote HTTP capability index sources to compose into a federated index.
 type: object
 additionalProperties: false
 required: [version, domain, collision_policy, sources]
@@ -25,7 +25,7 @@ $defs:
  source:
    type: object
    additionalProperties: false
-    required: [repo, index, enabled]
+    required: [repo, enabled]
    properties:
      repo:
        type: string
@@ -33,6 +33,12 @@ $defs:
      index:
        type: string
        minLength: 1
+        description: Local path to capabilities.yaml (repo-relative or ~/...)
+      url:
+        type: string
+        format: uri
+        pattern: '^https?://'
+        description: Remote HTTP(S) URL to a capabilities.yaml index
      enabled:
        type: boolean
      required:
@@ -42,3 +48,22 @@ $defs:
        type: string
      description:
        type: string
+      cache_ttl_seconds:
+        type: integer
+        minimum: 0
+        default: 86400
+        description: >
+          Seconds to reuse a cached remote index. 0 always refetches when
+          compose runs (stale cache still used as fallback on fetch failure).
+      auth_env:
+        type: string
+        minLength: 1
+        description: Environment variable holding an Authorization token or full header value
+      auth_header:
+        type: string
+        minLength: 1
+        default: Authorization
+        description: HTTP header name for auth_env value when fetching url sources
+    oneOf:
+      - required: [index]
+      - required: [url]
--- a/tests/test_federation.py
+++ b/tests/test_federation.py
@@ -0,0 +1,208 @@
+from __future__ import annotations
+
+from pathlib import Path
+from unittest.mock import patch
+
+import urllib.error
+import yaml
+
+from reuse_surface.federation import (
+    compose_federated_index,
+    fetch_remote_index_text,
+    load_federation_manifest,
+    resolve_source_index_path,
+)
+
+ROOT = Path(__file__).resolve().parent.parent
+
+REMOTE_INDEX = """
+version: 1
+domain: helix_forge
+updated: "2026-06-15"
+capabilities:
+  - id: capability.remote.sample
+    name: Remote Sample
+    domain: helix_forge
+    vector: D2/A0/C0/R0
+    owner: example
+    path: registry/capabilities/capability.remote.sample.md
+    summary: Sample capability from a remote index
+    tags: [sample]
+    consumption_modes: [planning]
+"""
+
+
+def _remote_manifest() -> dict:
+    return {
+        "version": 1,
+        "domain": "helix_forge",
+        "collision_policy": "warn",
+        "sources": [
+            {
+                "repo": "local",
+                "index": "registry/indexes/capabilities.yaml",
+                "enabled": True,
+                "required": True,
+            },
+            {
+                "repo": "remote-repo",
+                "url": "https://example.com/capabilities.yaml",
+                "enabled": True,
+                "required": False,
+                "cache_ttl_seconds": 3600,
+            },
+        ],
+    }
+
+
+def test_manifest_with_url_source_validates():
+    manifest = _remote_manifest()
+    schema_path = ROOT / "schemas" / "federation.schema.yaml"
+    from jsonschema import Draft202012Validator
+
+    schema = yaml.safe_load(schema_path.read_text(encoding="utf-8"))
+    errors = list(Draft202012Validator(schema).iter_errors(manifest))
+    assert errors == []
+
+
+def test_fetch_remote_index_text():
+    source = {"repo": "remote-repo", "url": "https://example.com/capabilities.yaml"}
+    payload = REMOTE_INDEX.encode("utf-8")
+
+    class FakeResponse:
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *args):
+            return False
+
+        def read(self):
+            return payload
+
+    with patch("urllib.request.urlopen", return_value=FakeResponse()):
+        text = fetch_remote_index_text(source["url"], source)
+    assert "capability.remote.sample" in text
+
+
+def test_remote_fetch_writes_cache(tmp_path, monkeypatch):
+    monkeypatch.setattr("reuse_surface.federation.CACHE_DIR", tmp_path / "cache")
+    source = {
+        "repo": "remote-repo",
+        "url": "https://example.com/capabilities.yaml",
+        "cache_ttl_seconds": 3600,
+    }
+    payload = REMOTE_INDEX.encode("utf-8")
+
+    class FakeResponse:
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *args):
+            return False
+
+        def read(self):
+            return payload
+
+    with patch("urllib.request.urlopen", return_value=FakeResponse()):
+        path, warnings = resolve_source_index_path(source)
+    assert path is not None
+    assert warnings == []
+    assert path.exists()
+    assert "capability.remote.sample" in path.read_text(encoding="utf-8")
+
+
+def test_remote_fetch_uses_fresh_cache_without_refetch(tmp_path, monkeypatch):
+    monkeypatch.setattr("reuse_surface.federation.CACHE_DIR", tmp_path / "cache")
+    source = {
+        "repo": "remote-repo",
+        "url": "https://example.com/capabilities.yaml",
+        "cache_ttl_seconds": 3600,
+    }
+    payload = REMOTE_INDEX.encode("utf-8")
+
+    class FakeResponse:
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *args):
+            return False
+
+        def read(self):
+            return payload
+
+    with patch("urllib.request.urlopen", return_value=FakeResponse()) as urlopen:
+        resolve_source_index_path(source)
+        path, warnings = resolve_source_index_path(source)
+    assert warnings == []
+    assert path is not None
+    urlopen.assert_called_once()
+
+
+def test_remote_fetch_falls_back_to_stale_cache(tmp_path, monkeypatch):
+    monkeypatch.setattr("reuse_surface.federation.CACHE_DIR", tmp_path / "cache")
+    source = {
+        "repo": "remote-repo",
+        "url": "https://example.com/capabilities.yaml",
+        "cache_ttl_seconds": 0,
+    }
+    payload = REMOTE_INDEX.encode("utf-8")
+
+    class FakeResponse:
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *args):
+            return False
+
+        def read(self):
+            return payload
+
+    with patch("urllib.request.urlopen", return_value=FakeResponse()):
+        resolve_source_index_path(source)
+
+    with patch(
+        "urllib.request.urlopen",
+        side_effect=urllib.error.URLError("network down"),
+    ):
+        path, warnings = resolve_source_index_path(source)
+    assert path is not None
+    assert any("stale cache" in warning for warning in warnings)
+
+
+def test_compose_merges_remote_capabilities(tmp_path, monkeypatch):
+    monkeypatch.setattr("reuse_surface.federation.CACHE_DIR", tmp_path / "cache")
+    source = {
+        "repo": "remote-repo",
+        "url": "https://example.com/capabilities.yaml",
+        "enabled": True,
+        "cache_ttl_seconds": 3600,
+    }
+    payload = REMOTE_INDEX.encode("utf-8")
+
+    class FakeResponse:
+        def __enter__(self):
+            return self
+
+        def __exit__(self, *args):
+            return False
+
+        def read(self):
+            return payload
+
+    manifest = _remote_manifest()
+    with patch("urllib.request.urlopen", return_value=FakeResponse()):
+        federated, warnings = compose_federated_index(manifest)
+    ids = {item["id"] for item in federated["capabilities"]}
+    assert "capability.remote.sample" in ids
+    assert "capability.registry.register" in ids
+    remote = next(
+        item for item in federated["capabilities"] if item["id"] == "capability.remote.sample"
+    )
+    assert remote["source_repo"] == "remote-repo"
+    assert remote["source_url"] == "https://example.com/capabilities.yaml"
+
+
+def test_load_production_manifest_still_validates():
+    manifest = load_federation_manifest()
+    assert manifest["domain"] == "helix_forge"
+    assert any(source["repo"] == "reuse-surface" for source in manifest["sources"])
--- a/tools/README.md
+++ b/tools/README.md
@@ -68,9 +68,12 @@ Compose a federated index from `registry/federation/sources.yaml`.

 ```bash
 reuse-surface federation compose
+reuse-surface federation compose --refresh
 ```

-Writes `registry/indexes/federated.yaml` with `source_repo` attribution.
+Composes local and remote HTTP index sources. Writes
+`registry/indexes/federated.yaml` with `source_repo` attribution. Remote indexes
+cache under `registry/federation/cache/`.

 ### graph

--- a/workplans/REUSE-WP-0010-network-federation.md
+++ b/workplans/REUSE-WP-0010-network-federation.md
@@ -4,7 +4,7 @@ type: workplan
 title: "Network federation for remote indexes"
 domain: helix_forge
 repo: reuse-surface
-status: backlog
+status: finished
 owner: codex
 topic_slug: helix-forge
 created: "2026-06-15"
@@ -21,7 +21,7 @@ fetch capability indexes from HTTP URLs or git raw endpoints.

 ```task
 id: REUSE-WP-0010-T01
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "6f181057-e0f7-4879-9eb9-928a527a01ee"
 ```
@@ -33,7 +33,7 @@ sources alongside `index` file paths, with optional auth and TTL metadata.

 ```task
 id: REUSE-WP-0010-T02
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "a2fac7d5-9383-4a42-bd23-3e8dbc7d550b"
 ```
@@ -45,7 +45,7 @@ Add HTTP fetch to `federation compose` with local cache under

 ```task
 id: REUSE-WP-0010-T03
-status: todo
+status: done
 priority: low
 state_hub_task_id: "73996193-ecae-4fb4-84f7-fe84a5cd8898"
 ```