generated from coulomb/repo-seed
Complete WP-0010: HTTP remote federation with cache
Some checks failed
ci / validate-registry (push) Has been cancelled
Some checks failed
ci / validate-registry (push) Has been cancelled
Extend federation manifest schema for url sources with auth and TTL metadata. Fetch remote capability indexes over HTTP(S), cache under registry/federation/cache/, and fall back to stale cache on fetch failure. Add --refresh flag, seven federation tests, and updated federation docs.
This commit is contained in:
@@ -11,6 +11,10 @@ helix_forge capabilities may be registered in multiple repositories. Federation
|
||||
composes capability indexes from configured sources into a single discovery
|
||||
surface without silently merging duplicate IDs.
|
||||
|
||||
Sources may be **local filesystem paths** or **remote HTTP(S) URLs** (git raw
|
||||
endpoints, published index artifacts, etc.). Remote indexes are cached under
|
||||
`registry/federation/cache/` for offline reuse and faster compose.
|
||||
|
||||
## Manifest
|
||||
|
||||
`registry/federation/sources.yaml` lists index sources:
|
||||
@@ -24,6 +28,14 @@ sources:
|
||||
index: registry/indexes/capabilities.yaml
|
||||
enabled: true
|
||||
required: true
|
||||
|
||||
- repo: sibling-repo
|
||||
url: https://git.example.com/org/sibling-repo/raw/main/registry/indexes/capabilities.yaml
|
||||
enabled: false
|
||||
required: false
|
||||
cache_ttl_seconds: 86400
|
||||
auth_env: FEDERATION_TOKEN
|
||||
auth_header: Authorization
|
||||
```
|
||||
|
||||
Schema: `schemas/federation.schema.yaml`
|
||||
@@ -33,26 +45,44 @@ Schema: `schemas/federation.schema.yaml`
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `repo` | Source repository slug |
|
||||
| `index` | Path to `capabilities.yaml` (repo-relative or `~/...`) |
|
||||
| `index` | Local path to `capabilities.yaml` (repo-relative or `~/...`) |
|
||||
| `url` | Remote HTTP(S) URL to a `capabilities.yaml` index |
|
||||
| `enabled` | Include this source in compose |
|
||||
| `required` | Fail compose if index missing when enabled |
|
||||
| `required` | Fail compose if index missing or remote fetch fails with no cache |
|
||||
| `domain` | Optional domain label |
|
||||
| `cache_ttl_seconds` | Reuse cached remote index for this many seconds (`0` = always refetch) |
|
||||
| `auth_env` | Environment variable holding token or full header value for `url` sources |
|
||||
| `auth_header` | HTTP header for `auth_env` (default `Authorization`) |
|
||||
|
||||
Each source must specify **either** `index` **or** `url`, not both.
|
||||
|
||||
Sibling repos (`state-hub`, `feature-control`, `identity-canon`) are listed as
|
||||
disabled placeholders until they publish registry indexes.
|
||||
disabled local placeholders until they publish registry indexes. A disabled
|
||||
`example-remote` URL source illustrates HTTP federation.
|
||||
|
||||
## Compose workflow
|
||||
|
||||
```bash
|
||||
reuse-surface federation compose
|
||||
reuse-surface federation compose --refresh # bypass remote cache
|
||||
```
|
||||
|
||||
Writes `registry/indexes/federated.yaml` with:
|
||||
|
||||
- Merged `capabilities` from all enabled sources
|
||||
- `source_repo` and `source_index` on every row
|
||||
- `source_url` when the row came from a remote source
|
||||
- `collision_policy` and per-source counts
|
||||
|
||||
### Remote cache
|
||||
|
||||
Fetched URL indexes are stored at `registry/federation/cache/<repo>.yaml` with
|
||||
metadata in `<repo>.meta.yaml`. The cache directory is gitignored; only
|
||||
`.gitkeep` is tracked.
|
||||
|
||||
When a refetch fails, compose reuses a stale cache and emits a warning. Required
|
||||
remote sources without cache fail compose with a clear error.
|
||||
|
||||
### Collision policy
|
||||
|
||||
`warn` (default): duplicate IDs across sources are kept but reported as
|
||||
@@ -62,9 +92,17 @@ warnings. Consumers must inspect `source_repo` before choosing an entry.
|
||||
|
||||
1. Run `reuse-surface federation compose` after manifest or sibling index changes.
|
||||
2. Read `registry/indexes/federated.yaml` for cross-repo discovery.
|
||||
3. Open `path` in the source repo for full entry detail when local.
|
||||
3. Open `path` in the source repo for full entry detail when local; follow
|
||||
`source_url` / `source_index` when remote.
|
||||
4. Run `reuse-surface graph --check` before relying on relation navigation.
|
||||
|
||||
### Cross-repo discovery without local checkout
|
||||
|
||||
Enable a `url` source pointing at a published raw index (Gitea, GitHub, static
|
||||
host). Set `auth_env` when the endpoint requires a token. Agents on machines
|
||||
without sibling repo clones can still compose a federated view from HTTP sources
|
||||
plus the local `reuse-surface` index.
|
||||
|
||||
## Relation graphs
|
||||
|
||||
```bash
|
||||
@@ -82,9 +120,12 @@ federated ID set.
|
||||
Gitea CI runs:
|
||||
|
||||
```bash
|
||||
reuse-surface validate --relations
|
||||
reuse-surface validate --relations --fail-on-warnings
|
||||
reuse-surface federation compose
|
||||
reuse-surface catalog
|
||||
reuse-surface graph --check --fail-on-warnings
|
||||
pytest -q
|
||||
```
|
||||
|
||||
Warnings on broken relations or missing optional sibling indexes do not fail CI;
|
||||
schema validation errors do.
|
||||
CI uses local sources only (remote examples are disabled). Warnings on missing
|
||||
optional sibling indexes do not fail CI; schema validation errors do.
|
||||
Reference in New Issue
Block a user