Complete WP-0010: HTTP remote federation with cache
Some checks failed
ci / validate-registry (push) Has been cancelled

Extend federation manifest schema for url sources with auth and TTL metadata.
Fetch remote capability indexes over HTTP(S), cache under
registry/federation/cache/, and fall back to stale cache on fetch failure.
Add --refresh flag, seven federation tests, and updated federation docs.
This commit is contained in:
2026-06-15 02:28:44 +02:00
parent c9b957d398
commit e8797b2e91
13 changed files with 487 additions and 45 deletions

View File

@@ -18,13 +18,12 @@ with **A3 CLI tooling** (`validate`, `query`, `export`) atop Markdown-first
authoring.
The two documents are **directionally aligned** on registry-first reuse, four
maturity dimensions, and human/agent consumers. REUSE-WP-0003 through
REUSE-WP-0009 closed the priority gaps from section 8 except network
federation. Remaining gaps are primarily scale and cross-org sync:
maturity dimensions, and human/agent consumers. REUSE-WP-0003 through REUSE-WP-0010 closed the priority gaps from section 8.
Remaining gaps are primarily document cross-coverage and operational polish:
1. **Network federation** — local compose only; no remote index fetch.
2. **Document cross-coverage** — SCOPE still carries operational detail INTENT
1. **Document cross-coverage** — SCOPE still carries operational detail INTENT
omits; INTENT success criteria are not fully enumerated in SCOPE.
2. **Live federation sync** — on-demand HTTP fetch only; no polling or webhooks.
**Current reuse-surface vector (self-assessment):** `D5 / A3 / C4 / R3`
@@ -276,7 +275,7 @@ core commands. Individual registered capabilities may carry their own evidence
| Priority | Gap | Outcome | Status |
|---|---|---|---|
| 13 | Interactive catalog | `docs/catalog/search.html` + `registry.json` | Closed (WP-0007) |
| 15 | Network federation | Remote index fetch and cross-org sync | Open (WP-0010) |
| 15 | Network federation | HTTP URL sources + cache in `federation compose` | Closed (WP-0010) |
| 16 | Graph UI | `docs/graph/index.html` explorer | Closed (WP-0008) |
---
@@ -302,4 +301,5 @@ core commands. Individual registered capabilities may carry their own evidence
| 2026-06-15 | REUSE-WP-0006 expanded registry to 12 capabilities; relation hygiene clean |
| 2026-06-15 | REUSE-WP-0007 closed priority 13 (searchable catalog UI) |
| 2026-06-15 | REUSE-WP-0008 closed priority 16 (graph explorer) |
| 2026-06-15 | REUSE-WP-0009 added pytest suite and CI fail-on-warnings; vector R3 |
| 2026-06-15 | REUSE-WP-0009 added pytest suite and CI fail-on-warnings; vector R3 |
| 2026-06-15 | REUSE-WP-0010 closed priority 15 (HTTP remote federation + cache) |

View File

@@ -11,6 +11,10 @@ helix_forge capabilities may be registered in multiple repositories. Federation
composes capability indexes from configured sources into a single discovery
surface without silently merging duplicate IDs.
Sources may be **local filesystem paths** or **remote HTTP(S) URLs** (git raw
endpoints, published index artifacts, etc.). Remote indexes are cached under
`registry/federation/cache/` for offline reuse and faster compose.
## Manifest
`registry/federation/sources.yaml` lists index sources:
@@ -24,6 +28,14 @@ sources:
index: registry/indexes/capabilities.yaml
enabled: true
required: true
- repo: sibling-repo
url: https://git.example.com/org/sibling-repo/raw/main/registry/indexes/capabilities.yaml
enabled: false
required: false
cache_ttl_seconds: 86400
auth_env: FEDERATION_TOKEN
auth_header: Authorization
```
Schema: `schemas/federation.schema.yaml`
@@ -33,26 +45,44 @@ Schema: `schemas/federation.schema.yaml`
| Field | Meaning |
|---|---|
| `repo` | Source repository slug |
| `index` | Path to `capabilities.yaml` (repo-relative or `~/...`) |
| `index` | Local path to `capabilities.yaml` (repo-relative or `~/...`) |
| `url` | Remote HTTP(S) URL to a `capabilities.yaml` index |
| `enabled` | Include this source in compose |
| `required` | Fail compose if index missing when enabled |
| `required` | Fail compose if index missing or remote fetch fails with no cache |
| `domain` | Optional domain label |
| `cache_ttl_seconds` | Reuse cached remote index for this many seconds (`0` = always refetch) |
| `auth_env` | Environment variable holding token or full header value for `url` sources |
| `auth_header` | HTTP header for `auth_env` (default `Authorization`) |
Each source must specify **either** `index` **or** `url`, not both.
Sibling repos (`state-hub`, `feature-control`, `identity-canon`) are listed as
disabled placeholders until they publish registry indexes.
disabled local placeholders until they publish registry indexes. A disabled
`example-remote` URL source illustrates HTTP federation.
## Compose workflow
```bash
reuse-surface federation compose
reuse-surface federation compose --refresh # bypass remote cache
```
Writes `registry/indexes/federated.yaml` with:
- Merged `capabilities` from all enabled sources
- `source_repo` and `source_index` on every row
- `source_url` when the row came from a remote source
- `collision_policy` and per-source counts
### Remote cache
Fetched URL indexes are stored at `registry/federation/cache/<repo>.yaml` with
metadata in `<repo>.meta.yaml`. The cache directory is gitignored; only
`.gitkeep` is tracked.
When a refetch fails, compose reuses a stale cache and emits a warning. Required
remote sources without cache fail compose with a clear error.
### Collision policy
`warn` (default): duplicate IDs across sources are kept but reported as
@@ -62,9 +92,17 @@ warnings. Consumers must inspect `source_repo` before choosing an entry.
1. Run `reuse-surface federation compose` after manifest or sibling index changes.
2. Read `registry/indexes/federated.yaml` for cross-repo discovery.
3. Open `path` in the source repo for full entry detail when local.
3. Open `path` in the source repo for full entry detail when local; follow
`source_url` / `source_index` when remote.
4. Run `reuse-surface graph --check` before relying on relation navigation.
### Cross-repo discovery without local checkout
Enable a `url` source pointing at a published raw index (Gitea, GitHub, static
host). Set `auth_env` when the endpoint requires a token. Agents on machines
without sibling repo clones can still compose a federated view from HTTP sources
plus the local `reuse-surface` index.
## Relation graphs
```bash
@@ -82,9 +120,12 @@ federated ID set.
Gitea CI runs:
```bash
reuse-surface validate --relations
reuse-surface validate --relations --fail-on-warnings
reuse-surface federation compose
reuse-surface catalog
reuse-surface graph --check --fail-on-warnings
pytest -q
```
Warnings on broken relations or missing optional sibling indexes do not fail CI;
schema validation errors do.
CI uses local sources only (remote examples are disabled). Warnings on missing
optional sibling indexes do not fail CI; schema validation errors do.