Complete WP-0010: HTTP remote federation with cache
Some checks failed
ci / validate-registry (push) Has been cancelled

Extend federation manifest schema for url sources with auth and TTL metadata.
Fetch remote capability indexes over HTTP(S), cache under
registry/federation/cache/, and fall back to stale cache on fetch failure.
Add --refresh flag, seven federation tests, and updated federation docs.
This commit is contained in:
2026-06-15 02:28:44 +02:00
parent c9b957d398
commit e8797b2e91
13 changed files with 487 additions and 45 deletions

View File

@@ -11,6 +11,10 @@ helix_forge capabilities may be registered in multiple repositories. Federation
composes capability indexes from configured sources into a single discovery
surface without silently merging duplicate IDs.
Sources may be **local filesystem paths** or **remote HTTP(S) URLs** (git raw
endpoints, published index artifacts, etc.). Remote indexes are cached under
`registry/federation/cache/` for offline reuse and faster compose.
## Manifest
`registry/federation/sources.yaml` lists index sources:
@@ -24,6 +28,14 @@ sources:
index: registry/indexes/capabilities.yaml
enabled: true
required: true
- repo: sibling-repo
url: https://git.example.com/org/sibling-repo/raw/main/registry/indexes/capabilities.yaml
enabled: false
required: false
cache_ttl_seconds: 86400
auth_env: FEDERATION_TOKEN
auth_header: Authorization
```
Schema: `schemas/federation.schema.yaml`
@@ -33,26 +45,44 @@ Schema: `schemas/federation.schema.yaml`
| Field | Meaning |
|---|---|
| `repo` | Source repository slug |
| `index` | Path to `capabilities.yaml` (repo-relative or `~/...`) |
| `index` | Local path to `capabilities.yaml` (repo-relative or `~/...`) |
| `url` | Remote HTTP(S) URL to a `capabilities.yaml` index |
| `enabled` | Include this source in compose |
| `required` | Fail compose if index missing when enabled |
| `required` | Fail compose if index missing or remote fetch fails with no cache |
| `domain` | Optional domain label |
| `cache_ttl_seconds` | Reuse cached remote index for this many seconds (`0` = always refetch) |
| `auth_env` | Environment variable holding token or full header value for `url` sources |
| `auth_header` | HTTP header for `auth_env` (default `Authorization`) |
Each source must specify **either** `index` **or** `url`, not both.
Sibling repos (`state-hub`, `feature-control`, `identity-canon`) are listed as
disabled placeholders until they publish registry indexes.
disabled local placeholders until they publish registry indexes. A disabled
`example-remote` URL source illustrates HTTP federation.
## Compose workflow
```bash
reuse-surface federation compose
reuse-surface federation compose --refresh # bypass remote cache
```
Writes `registry/indexes/federated.yaml` with:
- Merged `capabilities` from all enabled sources
- `source_repo` and `source_index` on every row
- `source_url` when the row came from a remote source
- `collision_policy` and per-source counts
### Remote cache
Fetched URL indexes are stored at `registry/federation/cache/<repo>.yaml` with
metadata in `<repo>.meta.yaml`. The cache directory is gitignored; only
`.gitkeep` is tracked.
When a refetch fails, compose reuses a stale cache and emits a warning. Required
remote sources without cache fail compose with a clear error.
### Collision policy
`warn` (default): duplicate IDs across sources are kept but reported as
@@ -62,9 +92,17 @@ warnings. Consumers must inspect `source_repo` before choosing an entry.
1. Run `reuse-surface federation compose` after manifest or sibling index changes.
2. Read `registry/indexes/federated.yaml` for cross-repo discovery.
3. Open `path` in the source repo for full entry detail when local.
3. Open `path` in the source repo for full entry detail when local; follow
`source_url` / `source_index` when remote.
4. Run `reuse-surface graph --check` before relying on relation navigation.
### Cross-repo discovery without local checkout
Enable a `url` source pointing at a published raw index (Gitea, GitHub, static
host). Set `auth_env` when the endpoint requires a token. Agents on machines
without sibling repo clones can still compose a federated view from HTTP sources
plus the local `reuse-surface` index.
## Relation graphs
```bash
@@ -82,9 +120,12 @@ federated ID set.
Gitea CI runs:
```bash
reuse-surface validate --relations
reuse-surface validate --relations --fail-on-warnings
reuse-surface federation compose
reuse-surface catalog
reuse-surface graph --check --fail-on-warnings
pytest -q
```
Warnings on broken relations or missing optional sibling indexes do not fail CI;
schema validation errors do.
CI uses local sources only (remote examples are disabled). Warnings on missing
optional sibling indexes do not fail CI; schema validation errors do.