generated from coulomb/repo-seed
Complete WP-0010: HTTP remote federation with cache
Some checks failed
ci / validate-registry (push) Has been cancelled
Some checks failed
ci / validate-registry (push) Has been cancelled
Extend federation manifest schema for url sources with auth and TTL metadata. Fetch remote capability indexes over HTTP(S), cache under registry/federation/cache/, and fall back to stale cache on fetch failure. Add --refresh flag, seven federation tests, and updated federation docs.
This commit is contained in:
4
.gitignore
vendored
4
.gitignore
vendored
@@ -1,3 +1,7 @@
|
|||||||
|
# Federation remote index cache (keep directory via .gitkeep)
|
||||||
|
registry/federation/cache/*
|
||||||
|
!registry/federation/cache/.gitkeep
|
||||||
|
|
||||||
# ---> Python
|
# ---> Python
|
||||||
# Byte-compiled / optimized / DLL files
|
# Byte-compiled / optimized / DLL files
|
||||||
__pycache__/
|
__pycache__/
|
||||||
|
|||||||
6
SCOPE.md
6
SCOPE.md
@@ -55,6 +55,7 @@ and agents can:
|
|||||||
- **Browse a searchable catalog** at `docs/catalog/search.html` (client-side
|
- **Browse a searchable catalog** at `docs/catalog/search.html` (client-side
|
||||||
filter over `registry.json`)
|
filter over `registry.json`)
|
||||||
- **Compose federated indexes** with `reuse-surface federation compose`
|
- **Compose federated indexes** with `reuse-surface federation compose`
|
||||||
|
(local paths and remote HTTP URLs with cache)
|
||||||
- **Generate relation graphs** with `reuse-surface graph`
|
- **Generate relation graphs** with `reuse-surface graph`
|
||||||
- **Explore relations interactively** at `docs/graph/index.html`
|
- **Explore relations interactively** at `docs/graph/index.html`
|
||||||
- **Avoid duplicates** by querying the index and checking overlaps before adding entries
|
- **Avoid duplicates** by querying the index and checking overlaps before adding entries
|
||||||
@@ -65,7 +66,7 @@ the index, and CLI automation.
|
|||||||
|
|
||||||
## What Is Not Possible Yet
|
## What Is Not Possible Yet
|
||||||
|
|
||||||
- Network-based federation or cross-org index sync
|
- Live cross-org index sync (polling/webhooks) beyond on-demand HTTP fetch
|
||||||
- Packaged releases beyond local `pip install -e .` and Gitea CI validation
|
- Packaged releases beyond local `pip install -e .` and Gitea CI validation
|
||||||
|
|
||||||
See `tools/README.md` for command reference.
|
See `tools/README.md` for command reference.
|
||||||
@@ -84,7 +85,8 @@ See `tools/README.md` for command reference.
|
|||||||
`docs/graph/index.html`.
|
`docs/graph/index.html`.
|
||||||
- Searchable catalog: `docs/catalog/search.html`.
|
- Searchable catalog: `docs/catalog/search.html`.
|
||||||
- Test suite: `tests/` (pytest).
|
- Test suite: `tests/` (pytest).
|
||||||
- Finished workplans: `REUSE-WP-0001` through `REUSE-WP-0009`.
|
- Remote federation: HTTP URL sources with cache in `registry/federation/cache/`.
|
||||||
|
- Finished workplans: `REUSE-WP-0001` through `REUSE-WP-0010`.
|
||||||
- **Self-assessed vector:** `D5 / A3 / C4 / R3` (see gap analysis).
|
- **Self-assessed vector:** `D5 / A3 / C4 / R3` (see gap analysis).
|
||||||
|
|
||||||
## Repository Layout
|
## Repository Layout
|
||||||
|
|||||||
@@ -18,13 +18,12 @@ with **A3 CLI tooling** (`validate`, `query`, `export`) atop Markdown-first
|
|||||||
authoring.
|
authoring.
|
||||||
|
|
||||||
The two documents are **directionally aligned** on registry-first reuse, four
|
The two documents are **directionally aligned** on registry-first reuse, four
|
||||||
maturity dimensions, and human/agent consumers. REUSE-WP-0003 through
|
maturity dimensions, and human/agent consumers. REUSE-WP-0003 through REUSE-WP-0010 closed the priority gaps from section 8.
|
||||||
REUSE-WP-0009 closed the priority gaps from section 8 except network
|
Remaining gaps are primarily document cross-coverage and operational polish:
|
||||||
federation. Remaining gaps are primarily scale and cross-org sync:
|
|
||||||
|
|
||||||
1. **Network federation** — local compose only; no remote index fetch.
|
1. **Document cross-coverage** — SCOPE still carries operational detail INTENT
|
||||||
2. **Document cross-coverage** — SCOPE still carries operational detail INTENT
|
|
||||||
omits; INTENT success criteria are not fully enumerated in SCOPE.
|
omits; INTENT success criteria are not fully enumerated in SCOPE.
|
||||||
|
2. **Live federation sync** — on-demand HTTP fetch only; no polling or webhooks.
|
||||||
|
|
||||||
**Current reuse-surface vector (self-assessment):** `D5 / A3 / C4 / R3`
|
**Current reuse-surface vector (self-assessment):** `D5 / A3 / C4 / R3`
|
||||||
|
|
||||||
@@ -276,7 +275,7 @@ core commands. Individual registered capabilities may carry their own evidence
|
|||||||
| Priority | Gap | Outcome | Status |
|
| Priority | Gap | Outcome | Status |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| 13 | Interactive catalog | `docs/catalog/search.html` + `registry.json` | Closed (WP-0007) |
|
| 13 | Interactive catalog | `docs/catalog/search.html` + `registry.json` | Closed (WP-0007) |
|
||||||
| 15 | Network federation | Remote index fetch and cross-org sync | Open (WP-0010) |
|
| 15 | Network federation | HTTP URL sources + cache in `federation compose` | Closed (WP-0010) |
|
||||||
| 16 | Graph UI | `docs/graph/index.html` explorer | Closed (WP-0008) |
|
| 16 | Graph UI | `docs/graph/index.html` explorer | Closed (WP-0008) |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -303,3 +302,4 @@ core commands. Individual registered capabilities may carry their own evidence
|
|||||||
| 2026-06-15 | REUSE-WP-0007 closed priority 13 (searchable catalog UI) |
|
| 2026-06-15 | REUSE-WP-0007 closed priority 13 (searchable catalog UI) |
|
||||||
| 2026-06-15 | REUSE-WP-0008 closed priority 16 (graph explorer) |
|
| 2026-06-15 | REUSE-WP-0008 closed priority 16 (graph explorer) |
|
||||||
| 2026-06-15 | REUSE-WP-0009 added pytest suite and CI fail-on-warnings; vector R3 |
|
| 2026-06-15 | REUSE-WP-0009 added pytest suite and CI fail-on-warnings; vector R3 |
|
||||||
|
| 2026-06-15 | REUSE-WP-0010 closed priority 15 (HTTP remote federation + cache) |
|
||||||
@@ -11,6 +11,10 @@ helix_forge capabilities may be registered in multiple repositories. Federation
|
|||||||
composes capability indexes from configured sources into a single discovery
|
composes capability indexes from configured sources into a single discovery
|
||||||
surface without silently merging duplicate IDs.
|
surface without silently merging duplicate IDs.
|
||||||
|
|
||||||
|
Sources may be **local filesystem paths** or **remote HTTP(S) URLs** (git raw
|
||||||
|
endpoints, published index artifacts, etc.). Remote indexes are cached under
|
||||||
|
`registry/federation/cache/` for offline reuse and faster compose.
|
||||||
|
|
||||||
## Manifest
|
## Manifest
|
||||||
|
|
||||||
`registry/federation/sources.yaml` lists index sources:
|
`registry/federation/sources.yaml` lists index sources:
|
||||||
@@ -24,6 +28,14 @@ sources:
|
|||||||
index: registry/indexes/capabilities.yaml
|
index: registry/indexes/capabilities.yaml
|
||||||
enabled: true
|
enabled: true
|
||||||
required: true
|
required: true
|
||||||
|
|
||||||
|
- repo: sibling-repo
|
||||||
|
url: https://git.example.com/org/sibling-repo/raw/main/registry/indexes/capabilities.yaml
|
||||||
|
enabled: false
|
||||||
|
required: false
|
||||||
|
cache_ttl_seconds: 86400
|
||||||
|
auth_env: FEDERATION_TOKEN
|
||||||
|
auth_header: Authorization
|
||||||
```
|
```
|
||||||
|
|
||||||
Schema: `schemas/federation.schema.yaml`
|
Schema: `schemas/federation.schema.yaml`
|
||||||
@@ -33,26 +45,44 @@ Schema: `schemas/federation.schema.yaml`
|
|||||||
| Field | Meaning |
|
| Field | Meaning |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `repo` | Source repository slug |
|
| `repo` | Source repository slug |
|
||||||
| `index` | Path to `capabilities.yaml` (repo-relative or `~/...`) |
|
| `index` | Local path to `capabilities.yaml` (repo-relative or `~/...`) |
|
||||||
|
| `url` | Remote HTTP(S) URL to a `capabilities.yaml` index |
|
||||||
| `enabled` | Include this source in compose |
|
| `enabled` | Include this source in compose |
|
||||||
| `required` | Fail compose if index missing when enabled |
|
| `required` | Fail compose if index missing or remote fetch fails with no cache |
|
||||||
| `domain` | Optional domain label |
|
| `domain` | Optional domain label |
|
||||||
|
| `cache_ttl_seconds` | Reuse cached remote index for this many seconds (`0` = always refetch) |
|
||||||
|
| `auth_env` | Environment variable holding token or full header value for `url` sources |
|
||||||
|
| `auth_header` | HTTP header for `auth_env` (default `Authorization`) |
|
||||||
|
|
||||||
|
Each source must specify **either** `index` **or** `url`, not both.
|
||||||
|
|
||||||
Sibling repos (`state-hub`, `feature-control`, `identity-canon`) are listed as
|
Sibling repos (`state-hub`, `feature-control`, `identity-canon`) are listed as
|
||||||
disabled placeholders until they publish registry indexes.
|
disabled local placeholders until they publish registry indexes. A disabled
|
||||||
|
`example-remote` URL source illustrates HTTP federation.
|
||||||
|
|
||||||
## Compose workflow
|
## Compose workflow
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
reuse-surface federation compose
|
reuse-surface federation compose
|
||||||
|
reuse-surface federation compose --refresh # bypass remote cache
|
||||||
```
|
```
|
||||||
|
|
||||||
Writes `registry/indexes/federated.yaml` with:
|
Writes `registry/indexes/federated.yaml` with:
|
||||||
|
|
||||||
- Merged `capabilities` from all enabled sources
|
- Merged `capabilities` from all enabled sources
|
||||||
- `source_repo` and `source_index` on every row
|
- `source_repo` and `source_index` on every row
|
||||||
|
- `source_url` when the row came from a remote source
|
||||||
- `collision_policy` and per-source counts
|
- `collision_policy` and per-source counts
|
||||||
|
|
||||||
|
### Remote cache
|
||||||
|
|
||||||
|
Fetched URL indexes are stored at `registry/federation/cache/<repo>.yaml` with
|
||||||
|
metadata in `<repo>.meta.yaml`. The cache directory is gitignored; only
|
||||||
|
`.gitkeep` is tracked.
|
||||||
|
|
||||||
|
When a refetch fails, compose reuses a stale cache and emits a warning. Required
|
||||||
|
remote sources without cache fail compose with a clear error.
|
||||||
|
|
||||||
### Collision policy
|
### Collision policy
|
||||||
|
|
||||||
`warn` (default): duplicate IDs across sources are kept but reported as
|
`warn` (default): duplicate IDs across sources are kept but reported as
|
||||||
@@ -62,9 +92,17 @@ warnings. Consumers must inspect `source_repo` before choosing an entry.
|
|||||||
|
|
||||||
1. Run `reuse-surface federation compose` after manifest or sibling index changes.
|
1. Run `reuse-surface federation compose` after manifest or sibling index changes.
|
||||||
2. Read `registry/indexes/federated.yaml` for cross-repo discovery.
|
2. Read `registry/indexes/federated.yaml` for cross-repo discovery.
|
||||||
3. Open `path` in the source repo for full entry detail when local.
|
3. Open `path` in the source repo for full entry detail when local; follow
|
||||||
|
`source_url` / `source_index` when remote.
|
||||||
4. Run `reuse-surface graph --check` before relying on relation navigation.
|
4. Run `reuse-surface graph --check` before relying on relation navigation.
|
||||||
|
|
||||||
|
### Cross-repo discovery without local checkout
|
||||||
|
|
||||||
|
Enable a `url` source pointing at a published raw index (Gitea, GitHub, static
|
||||||
|
host). Set `auth_env` when the endpoint requires a token. Agents on machines
|
||||||
|
without sibling repo clones can still compose a federated view from HTTP sources
|
||||||
|
plus the local `reuse-surface` index.
|
||||||
|
|
||||||
## Relation graphs
|
## Relation graphs
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -82,9 +120,12 @@ federated ID set.
|
|||||||
Gitea CI runs:
|
Gitea CI runs:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
reuse-surface validate --relations
|
reuse-surface validate --relations --fail-on-warnings
|
||||||
reuse-surface federation compose
|
reuse-surface federation compose
|
||||||
|
reuse-surface catalog
|
||||||
|
reuse-surface graph --check --fail-on-warnings
|
||||||
|
pytest -q
|
||||||
```
|
```
|
||||||
|
|
||||||
Warnings on broken relations or missing optional sibling indexes do not fail CI;
|
CI uses local sources only (remote examples are disabled). Warnings on missing
|
||||||
schema validation errors do.
|
optional sibling indexes do not fail CI; schema validation errors do.
|
||||||
0
registry/federation/cache/.gitkeep
vendored
Normal file
0
registry/federation/cache/.gitkeep
vendored
Normal file
@@ -33,3 +33,12 @@ sources:
|
|||||||
required: false
|
required: false
|
||||||
domain: helix_forge
|
domain: helix_forge
|
||||||
description: Identity canon research capabilities
|
description: Identity canon research capabilities
|
||||||
|
|
||||||
|
# Remote index example — enable when a repo publishes a raw capabilities.yaml URL
|
||||||
|
- repo: example-remote
|
||||||
|
url: https://example.com/registry/indexes/capabilities.yaml
|
||||||
|
enabled: false
|
||||||
|
required: false
|
||||||
|
domain: helix_forge
|
||||||
|
cache_ttl_seconds: 86400
|
||||||
|
description: Illustrative HTTP federation source (disabled)
|
||||||
@@ -6,8 +6,8 @@ domain: helix_forge
|
|||||||
collision_policy: warn
|
collision_policy: warn
|
||||||
sources:
|
sources:
|
||||||
- repo: reuse-surface
|
- repo: reuse-surface
|
||||||
index: registry/indexes/capabilities.yaml
|
|
||||||
count: 12
|
count: 12
|
||||||
|
index: registry/indexes/capabilities.yaml
|
||||||
capabilities:
|
capabilities:
|
||||||
- id: capability.activity.event-coordinate
|
- id: capability.activity.event-coordinate
|
||||||
name: Organizational Event Coordination
|
name: Organizational Event Coordination
|
||||||
|
|||||||
@@ -146,7 +146,7 @@ def cmd_overlaps(args: argparse.Namespace) -> int:
|
|||||||
|
|
||||||
def cmd_federation_compose(args: argparse.Namespace) -> int:
|
def cmd_federation_compose(args: argparse.Namespace) -> int:
|
||||||
try:
|
try:
|
||||||
target, warnings = write_federated_index()
|
target, warnings = write_federated_index(refresh=args.refresh)
|
||||||
except (FileNotFoundError, ValueError) as exc:
|
except (FileNotFoundError, ValueError) as exc:
|
||||||
print(f"error: {exc}", file=sys.stderr)
|
print(f"error: {exc}", file=sys.stderr)
|
||||||
return 1
|
return 1
|
||||||
@@ -258,6 +258,11 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
)
|
)
|
||||||
federation_sub = federation.add_subparsers(dest="federation_command", required=True)
|
federation_sub = federation.add_subparsers(dest="federation_command", required=True)
|
||||||
compose = federation_sub.add_parser("compose", help="compose federated index")
|
compose = federation_sub.add_parser("compose", help="compose federated index")
|
||||||
|
compose.add_argument(
|
||||||
|
"--refresh",
|
||||||
|
action="store_true",
|
||||||
|
help="bypass remote index cache and refetch URL sources",
|
||||||
|
)
|
||||||
compose.set_defaults(func=cmd_federation_compose)
|
compose.set_defaults(func=cmd_federation_compose)
|
||||||
|
|
||||||
query = subparsers.add_parser("query", help="query capability index")
|
query = subparsers.add_parser("query", help="query capability index")
|
||||||
|
|||||||
@@ -1,7 +1,10 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import sys
|
import os
|
||||||
from datetime import date
|
import re
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from datetime import date, datetime, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
@@ -13,6 +16,9 @@ from reuse_surface.registry import ROOT
|
|||||||
MANIFEST_PATH = ROOT / "registry" / "federation" / "sources.yaml"
|
MANIFEST_PATH = ROOT / "registry" / "federation" / "sources.yaml"
|
||||||
SCHEMA_PATH = ROOT / "schemas" / "federation.schema.yaml"
|
SCHEMA_PATH = ROOT / "schemas" / "federation.schema.yaml"
|
||||||
FEDERATED_INDEX_PATH = ROOT / "registry" / "indexes" / "federated.yaml"
|
FEDERATED_INDEX_PATH = ROOT / "registry" / "indexes" / "federated.yaml"
|
||||||
|
CACHE_DIR = ROOT / "registry" / "federation" / "cache"
|
||||||
|
DEFAULT_CACHE_TTL_SECONDS = 86400
|
||||||
|
USER_AGENT = "reuse-surface/0.1 federation-compose"
|
||||||
|
|
||||||
|
|
||||||
def _expand_path(index_path: str) -> Path:
|
def _expand_path(index_path: str) -> Path:
|
||||||
@@ -39,8 +45,139 @@ def _resolve_index_path(index_value: str) -> Path:
|
|||||||
return path
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_repo_slug(repo: str) -> str:
|
||||||
|
return re.sub(r"[^\w.-]", "_", repo)
|
||||||
|
|
||||||
|
|
||||||
|
def _path_label(path: Path) -> str:
|
||||||
|
try:
|
||||||
|
return str(path.relative_to(ROOT))
|
||||||
|
except ValueError:
|
||||||
|
return str(path)
|
||||||
|
|
||||||
|
|
||||||
|
def _cache_paths(repo: str) -> tuple[Path, Path]:
|
||||||
|
slug = _safe_repo_slug(repo)
|
||||||
|
return CACHE_DIR / f"{slug}.yaml", CACHE_DIR / f"{slug}.meta.yaml"
|
||||||
|
|
||||||
|
|
||||||
|
def _read_cache_meta(meta_path: Path) -> dict[str, Any] | None:
|
||||||
|
if not meta_path.exists():
|
||||||
|
return None
|
||||||
|
data = yaml.safe_load(meta_path.read_text(encoding="utf-8"))
|
||||||
|
return data if isinstance(data, dict) else None
|
||||||
|
|
||||||
|
|
||||||
|
def _cache_is_fresh(meta: dict[str, Any], ttl_seconds: int) -> bool:
|
||||||
|
if ttl_seconds <= 0:
|
||||||
|
return False
|
||||||
|
fetched_at = meta.get("fetched_at")
|
||||||
|
if not fetched_at:
|
||||||
|
return False
|
||||||
|
fetched = datetime.fromisoformat(str(fetched_at))
|
||||||
|
if fetched.tzinfo is None:
|
||||||
|
fetched = fetched.replace(tzinfo=timezone.utc)
|
||||||
|
age = (datetime.now(timezone.utc) - fetched).total_seconds()
|
||||||
|
return age < ttl_seconds
|
||||||
|
|
||||||
|
|
||||||
|
def _auth_headers(source: dict[str, Any]) -> dict[str, str]:
|
||||||
|
auth_env = source.get("auth_env")
|
||||||
|
if not auth_env:
|
||||||
|
return {}
|
||||||
|
token = os.environ.get(auth_env)
|
||||||
|
if not token:
|
||||||
|
raise ValueError(
|
||||||
|
f"auth env {auth_env} is not set for remote source {source['repo']}"
|
||||||
|
)
|
||||||
|
header_name = source.get("auth_header", "Authorization")
|
||||||
|
if header_name.lower() == "authorization" and not token.lower().startswith(
|
||||||
|
("bearer ", "basic ")
|
||||||
|
):
|
||||||
|
token = f"Bearer {token}"
|
||||||
|
return {header_name: token}
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_remote_index_text(url: str, source: dict[str, Any]) -> str:
|
||||||
|
headers = {"User-Agent": USER_AGENT, **_auth_headers(source)}
|
||||||
|
request = urllib.request.Request(url, headers=headers)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(request, timeout=30) as response:
|
||||||
|
return response.read().decode("utf-8")
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
raise ConnectionError(
|
||||||
|
f"HTTP {exc.code} fetching {url} for {source['repo']}"
|
||||||
|
) from exc
|
||||||
|
except urllib.error.URLError as exc:
|
||||||
|
raise ConnectionError(
|
||||||
|
f"failed to fetch {url} for {source['repo']}: {exc.reason}"
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
|
||||||
|
def _write_remote_cache(repo: str, url: str, content: str) -> Path:
|
||||||
|
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
index_path, meta_path = _cache_paths(repo)
|
||||||
|
index_path.write_text(content, encoding="utf-8")
|
||||||
|
meta = {
|
||||||
|
"fetched_at": datetime.now(timezone.utc).isoformat(),
|
||||||
|
"url": url,
|
||||||
|
"repo": repo,
|
||||||
|
}
|
||||||
|
meta_path.write_text(yaml.safe_dump(meta, sort_keys=False), encoding="utf-8")
|
||||||
|
return index_path
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_source_index_path(
|
||||||
|
source: dict[str, Any],
|
||||||
|
*,
|
||||||
|
refresh: bool = False,
|
||||||
|
) -> tuple[Path | None, list[str]]:
|
||||||
|
warnings: list[str] = []
|
||||||
|
if "index" in source:
|
||||||
|
path = _resolve_index_path(source["index"])
|
||||||
|
if not path.exists():
|
||||||
|
message = f"missing index for {source['repo']}: {path}"
|
||||||
|
if source.get("required", False):
|
||||||
|
raise FileNotFoundError(message)
|
||||||
|
warnings.append(message)
|
||||||
|
return None, warnings
|
||||||
|
return path, warnings
|
||||||
|
|
||||||
|
url = source["url"]
|
||||||
|
ttl_seconds = int(source.get("cache_ttl_seconds", DEFAULT_CACHE_TTL_SECONDS))
|
||||||
|
index_path, meta_path = _cache_paths(source["repo"])
|
||||||
|
meta = _read_cache_meta(meta_path)
|
||||||
|
use_cache = (
|
||||||
|
index_path.exists()
|
||||||
|
and meta is not None
|
||||||
|
and meta.get("url") == url
|
||||||
|
and _cache_is_fresh(meta, ttl_seconds)
|
||||||
|
and not refresh
|
||||||
|
)
|
||||||
|
if use_cache:
|
||||||
|
return index_path, warnings
|
||||||
|
|
||||||
|
try:
|
||||||
|
content = fetch_remote_index_text(url, source)
|
||||||
|
except (ConnectionError, ValueError) as exc:
|
||||||
|
if index_path.exists():
|
||||||
|
warnings.append(
|
||||||
|
f"remote fetch failed for {source['repo']}, using stale cache: {exc}"
|
||||||
|
)
|
||||||
|
return index_path, warnings
|
||||||
|
message = f"remote index unavailable for {source['repo']}: {exc}"
|
||||||
|
if source.get("required", False):
|
||||||
|
raise FileNotFoundError(message) from exc
|
||||||
|
warnings.append(message)
|
||||||
|
return None, warnings
|
||||||
|
|
||||||
|
return _write_remote_cache(source["repo"], url, content), warnings
|
||||||
|
|
||||||
|
|
||||||
def compose_federated_index(
|
def compose_federated_index(
|
||||||
manifest: dict[str, Any] | None = None,
|
manifest: dict[str, Any] | None = None,
|
||||||
|
*,
|
||||||
|
refresh: bool = False,
|
||||||
) -> tuple[dict[str, Any], list[str]]:
|
) -> tuple[dict[str, Any], list[str]]:
|
||||||
manifest = manifest or load_federation_manifest()
|
manifest = manifest or load_federation_manifest()
|
||||||
warnings: list[str] = []
|
warnings: list[str] = []
|
||||||
@@ -51,12 +188,11 @@ def compose_federated_index(
|
|||||||
for source in manifest["sources"]:
|
for source in manifest["sources"]:
|
||||||
if not source.get("enabled", False):
|
if not source.get("enabled", False):
|
||||||
continue
|
continue
|
||||||
index_path = _resolve_index_path(source["index"])
|
index_path, source_warnings = resolve_source_index_path(
|
||||||
if not index_path.exists():
|
source, refresh=refresh
|
||||||
message = f"missing index for {source['repo']}: {index_path}"
|
)
|
||||||
if source.get("required", False):
|
warnings.extend(source_warnings)
|
||||||
raise FileNotFoundError(message)
|
if index_path is None:
|
||||||
warnings.append(message)
|
|
||||||
continue
|
continue
|
||||||
with index_path.open(encoding="utf-8") as handle:
|
with index_path.open(encoding="utf-8") as handle:
|
||||||
index_data = yaml.safe_load(handle)
|
index_data = yaml.safe_load(handle)
|
||||||
@@ -71,16 +207,23 @@ def compose_federated_index(
|
|||||||
seen_ids[cap_id] = source["repo"]
|
seen_ids[cap_id] = source["repo"]
|
||||||
federated_item = dict(item)
|
federated_item = dict(item)
|
||||||
federated_item["source_repo"] = source["repo"]
|
federated_item["source_repo"] = source["repo"]
|
||||||
federated_item["source_index"] = source["index"]
|
if "url" in source:
|
||||||
|
federated_item["source_url"] = source["url"]
|
||||||
|
federated_item["source_index"] = _path_label(index_path)
|
||||||
|
else:
|
||||||
|
federated_item["source_index"] = source["index"]
|
||||||
merged.append(federated_item)
|
merged.append(federated_item)
|
||||||
count += 1
|
count += 1
|
||||||
source_summaries.append(
|
summary: dict[str, Any] = {
|
||||||
{
|
"repo": source["repo"],
|
||||||
"repo": source["repo"],
|
"count": count,
|
||||||
"index": source["index"],
|
}
|
||||||
"count": count,
|
if "url" in source:
|
||||||
}
|
summary["url"] = source["url"]
|
||||||
)
|
summary["cache"] = _path_label(index_path)
|
||||||
|
else:
|
||||||
|
summary["index"] = source["index"]
|
||||||
|
source_summaries.append(summary)
|
||||||
|
|
||||||
federated = {
|
federated = {
|
||||||
"version": manifest.get("version", 1),
|
"version": manifest.get("version", 1),
|
||||||
@@ -96,8 +239,10 @@ def compose_federated_index(
|
|||||||
def write_federated_index(
|
def write_federated_index(
|
||||||
output_path: Path | None = None,
|
output_path: Path | None = None,
|
||||||
manifest: dict[str, Any] | None = None,
|
manifest: dict[str, Any] | None = None,
|
||||||
|
*,
|
||||||
|
refresh: bool = False,
|
||||||
) -> tuple[Path, list[str]]:
|
) -> tuple[Path, list[str]]:
|
||||||
federated, warnings = compose_federated_index(manifest)
|
federated, warnings = compose_federated_index(manifest, refresh=refresh)
|
||||||
target = output_path or FEDERATED_INDEX_PATH
|
target = output_path or FEDERATED_INDEX_PATH
|
||||||
target.parent.mkdir(parents=True, exist_ok=True)
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
header = (
|
header = (
|
||||||
|
|||||||
@@ -2,8 +2,8 @@ $schema: https://json-schema.org/draft/2020-12/schema
|
|||||||
$id: https://reuse-surface.local/schemas/federation.schema.yaml
|
$id: https://reuse-surface.local/schemas/federation.schema.yaml
|
||||||
title: Registry Federation Manifest
|
title: Registry Federation Manifest
|
||||||
description: >
|
description: >
|
||||||
Schema for registry/federation/sources.yaml. Describes local and sibling
|
Schema for registry/federation/sources.yaml. Describes local filesystem and
|
||||||
capability index sources to compose into a federated index.
|
remote HTTP capability index sources to compose into a federated index.
|
||||||
type: object
|
type: object
|
||||||
additionalProperties: false
|
additionalProperties: false
|
||||||
required: [version, domain, collision_policy, sources]
|
required: [version, domain, collision_policy, sources]
|
||||||
@@ -25,7 +25,7 @@ $defs:
|
|||||||
source:
|
source:
|
||||||
type: object
|
type: object
|
||||||
additionalProperties: false
|
additionalProperties: false
|
||||||
required: [repo, index, enabled]
|
required: [repo, enabled]
|
||||||
properties:
|
properties:
|
||||||
repo:
|
repo:
|
||||||
type: string
|
type: string
|
||||||
@@ -33,6 +33,12 @@ $defs:
|
|||||||
index:
|
index:
|
||||||
type: string
|
type: string
|
||||||
minLength: 1
|
minLength: 1
|
||||||
|
description: Local path to capabilities.yaml (repo-relative or ~/...)
|
||||||
|
url:
|
||||||
|
type: string
|
||||||
|
format: uri
|
||||||
|
pattern: '^https?://'
|
||||||
|
description: Remote HTTP(S) URL to a capabilities.yaml index
|
||||||
enabled:
|
enabled:
|
||||||
type: boolean
|
type: boolean
|
||||||
required:
|
required:
|
||||||
@@ -42,3 +48,22 @@ $defs:
|
|||||||
type: string
|
type: string
|
||||||
description:
|
description:
|
||||||
type: string
|
type: string
|
||||||
|
cache_ttl_seconds:
|
||||||
|
type: integer
|
||||||
|
minimum: 0
|
||||||
|
default: 86400
|
||||||
|
description: >
|
||||||
|
Seconds to reuse a cached remote index. 0 always refetches when
|
||||||
|
compose runs (stale cache still used as fallback on fetch failure).
|
||||||
|
auth_env:
|
||||||
|
type: string
|
||||||
|
minLength: 1
|
||||||
|
description: Environment variable holding an Authorization token or full header value
|
||||||
|
auth_header:
|
||||||
|
type: string
|
||||||
|
minLength: 1
|
||||||
|
default: Authorization
|
||||||
|
description: HTTP header name for auth_env value when fetching url sources
|
||||||
|
oneOf:
|
||||||
|
- required: [index]
|
||||||
|
- required: [url]
|
||||||
208
tests/test_federation.py
Normal file
208
tests/test_federation.py
Normal file
@@ -0,0 +1,208 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import urllib.error
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
from reuse_surface.federation import (
|
||||||
|
compose_federated_index,
|
||||||
|
fetch_remote_index_text,
|
||||||
|
load_federation_manifest,
|
||||||
|
resolve_source_index_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
|
||||||
|
REMOTE_INDEX = """
|
||||||
|
version: 1
|
||||||
|
domain: helix_forge
|
||||||
|
updated: "2026-06-15"
|
||||||
|
capabilities:
|
||||||
|
- id: capability.remote.sample
|
||||||
|
name: Remote Sample
|
||||||
|
domain: helix_forge
|
||||||
|
vector: D2/A0/C0/R0
|
||||||
|
owner: example
|
||||||
|
path: registry/capabilities/capability.remote.sample.md
|
||||||
|
summary: Sample capability from a remote index
|
||||||
|
tags: [sample]
|
||||||
|
consumption_modes: [planning]
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _remote_manifest() -> dict:
|
||||||
|
return {
|
||||||
|
"version": 1,
|
||||||
|
"domain": "helix_forge",
|
||||||
|
"collision_policy": "warn",
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"repo": "local",
|
||||||
|
"index": "registry/indexes/capabilities.yaml",
|
||||||
|
"enabled": True,
|
||||||
|
"required": True,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"repo": "remote-repo",
|
||||||
|
"url": "https://example.com/capabilities.yaml",
|
||||||
|
"enabled": True,
|
||||||
|
"required": False,
|
||||||
|
"cache_ttl_seconds": 3600,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_manifest_with_url_source_validates():
|
||||||
|
manifest = _remote_manifest()
|
||||||
|
schema_path = ROOT / "schemas" / "federation.schema.yaml"
|
||||||
|
from jsonschema import Draft202012Validator
|
||||||
|
|
||||||
|
schema = yaml.safe_load(schema_path.read_text(encoding="utf-8"))
|
||||||
|
errors = list(Draft202012Validator(schema).iter_errors(manifest))
|
||||||
|
assert errors == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_remote_index_text():
|
||||||
|
source = {"repo": "remote-repo", "url": "https://example.com/capabilities.yaml"}
|
||||||
|
payload = REMOTE_INDEX.encode("utf-8")
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return payload
|
||||||
|
|
||||||
|
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||||
|
text = fetch_remote_index_text(source["url"], source)
|
||||||
|
assert "capability.remote.sample" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_remote_fetch_writes_cache(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr("reuse_surface.federation.CACHE_DIR", tmp_path / "cache")
|
||||||
|
source = {
|
||||||
|
"repo": "remote-repo",
|
||||||
|
"url": "https://example.com/capabilities.yaml",
|
||||||
|
"cache_ttl_seconds": 3600,
|
||||||
|
}
|
||||||
|
payload = REMOTE_INDEX.encode("utf-8")
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return payload
|
||||||
|
|
||||||
|
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||||
|
path, warnings = resolve_source_index_path(source)
|
||||||
|
assert path is not None
|
||||||
|
assert warnings == []
|
||||||
|
assert path.exists()
|
||||||
|
assert "capability.remote.sample" in path.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def test_remote_fetch_uses_fresh_cache_without_refetch(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr("reuse_surface.federation.CACHE_DIR", tmp_path / "cache")
|
||||||
|
source = {
|
||||||
|
"repo": "remote-repo",
|
||||||
|
"url": "https://example.com/capabilities.yaml",
|
||||||
|
"cache_ttl_seconds": 3600,
|
||||||
|
}
|
||||||
|
payload = REMOTE_INDEX.encode("utf-8")
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return payload
|
||||||
|
|
||||||
|
with patch("urllib.request.urlopen", return_value=FakeResponse()) as urlopen:
|
||||||
|
resolve_source_index_path(source)
|
||||||
|
path, warnings = resolve_source_index_path(source)
|
||||||
|
assert warnings == []
|
||||||
|
assert path is not None
|
||||||
|
urlopen.assert_called_once()
|
||||||
|
|
||||||
|
|
||||||
|
def test_remote_fetch_falls_back_to_stale_cache(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr("reuse_surface.federation.CACHE_DIR", tmp_path / "cache")
|
||||||
|
source = {
|
||||||
|
"repo": "remote-repo",
|
||||||
|
"url": "https://example.com/capabilities.yaml",
|
||||||
|
"cache_ttl_seconds": 0,
|
||||||
|
}
|
||||||
|
payload = REMOTE_INDEX.encode("utf-8")
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return payload
|
||||||
|
|
||||||
|
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||||
|
resolve_source_index_path(source)
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"urllib.request.urlopen",
|
||||||
|
side_effect=urllib.error.URLError("network down"),
|
||||||
|
):
|
||||||
|
path, warnings = resolve_source_index_path(source)
|
||||||
|
assert path is not None
|
||||||
|
assert any("stale cache" in warning for warning in warnings)
|
||||||
|
|
||||||
|
|
||||||
|
def test_compose_merges_remote_capabilities(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr("reuse_surface.federation.CACHE_DIR", tmp_path / "cache")
|
||||||
|
source = {
|
||||||
|
"repo": "remote-repo",
|
||||||
|
"url": "https://example.com/capabilities.yaml",
|
||||||
|
"enabled": True,
|
||||||
|
"cache_ttl_seconds": 3600,
|
||||||
|
}
|
||||||
|
payload = REMOTE_INDEX.encode("utf-8")
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return payload
|
||||||
|
|
||||||
|
manifest = _remote_manifest()
|
||||||
|
with patch("urllib.request.urlopen", return_value=FakeResponse()):
|
||||||
|
federated, warnings = compose_federated_index(manifest)
|
||||||
|
ids = {item["id"] for item in federated["capabilities"]}
|
||||||
|
assert "capability.remote.sample" in ids
|
||||||
|
assert "capability.registry.register" in ids
|
||||||
|
remote = next(
|
||||||
|
item for item in federated["capabilities"] if item["id"] == "capability.remote.sample"
|
||||||
|
)
|
||||||
|
assert remote["source_repo"] == "remote-repo"
|
||||||
|
assert remote["source_url"] == "https://example.com/capabilities.yaml"
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_production_manifest_still_validates():
|
||||||
|
manifest = load_federation_manifest()
|
||||||
|
assert manifest["domain"] == "helix_forge"
|
||||||
|
assert any(source["repo"] == "reuse-surface" for source in manifest["sources"])
|
||||||
@@ -68,9 +68,12 @@ Compose a federated index from `registry/federation/sources.yaml`.
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
reuse-surface federation compose
|
reuse-surface federation compose
|
||||||
|
reuse-surface federation compose --refresh
|
||||||
```
|
```
|
||||||
|
|
||||||
Writes `registry/indexes/federated.yaml` with `source_repo` attribution.
|
Composes local and remote HTTP index sources. Writes
|
||||||
|
`registry/indexes/federated.yaml` with `source_repo` attribution. Remote indexes
|
||||||
|
cache under `registry/federation/cache/`.
|
||||||
|
|
||||||
### graph
|
### graph
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ type: workplan
|
|||||||
title: "Network federation for remote indexes"
|
title: "Network federation for remote indexes"
|
||||||
domain: helix_forge
|
domain: helix_forge
|
||||||
repo: reuse-surface
|
repo: reuse-surface
|
||||||
status: backlog
|
status: finished
|
||||||
owner: codex
|
owner: codex
|
||||||
topic_slug: helix-forge
|
topic_slug: helix-forge
|
||||||
created: "2026-06-15"
|
created: "2026-06-15"
|
||||||
@@ -21,7 +21,7 @@ fetch capability indexes from HTTP URLs or git raw endpoints.
|
|||||||
|
|
||||||
```task
|
```task
|
||||||
id: REUSE-WP-0010-T01
|
id: REUSE-WP-0010-T01
|
||||||
status: todo
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "6f181057-e0f7-4879-9eb9-928a527a01ee"
|
state_hub_task_id: "6f181057-e0f7-4879-9eb9-928a527a01ee"
|
||||||
```
|
```
|
||||||
@@ -33,7 +33,7 @@ sources alongside `index` file paths, with optional auth and TTL metadata.
|
|||||||
|
|
||||||
```task
|
```task
|
||||||
id: REUSE-WP-0010-T02
|
id: REUSE-WP-0010-T02
|
||||||
status: todo
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "a2fac7d5-9383-4a42-bd23-3e8dbc7d550b"
|
state_hub_task_id: "a2fac7d5-9383-4a42-bd23-3e8dbc7d550b"
|
||||||
```
|
```
|
||||||
@@ -45,7 +45,7 @@ Add HTTP fetch to `federation compose` with local cache under
|
|||||||
|
|
||||||
```task
|
```task
|
||||||
id: REUSE-WP-0010-T03
|
id: REUSE-WP-0010-T03
|
||||||
status: todo
|
status: done
|
||||||
priority: low
|
priority: low
|
||||||
state_hub_task_id: "73996193-ecae-4fb4-84f7-fe84a5cd8898"
|
state_hub_task_id: "73996193-ecae-4fb4-84f7-fe84a5cd8898"
|
||||||
```
|
```
|
||||||
|
|||||||
Reference in New Issue
Block a user