generated from coulomb/repo-seed
Complete WP-0010: HTTP remote federation with cache
Some checks failed
ci / validate-registry (push) Has been cancelled
Some checks failed
ci / validate-registry (push) Has been cancelled
Extend federation manifest schema for url sources with auth and TTL metadata. Fetch remote capability indexes over HTTP(S), cache under registry/federation/cache/, and fall back to stale cache on fetch failure. Add --refresh flag, seven federation tests, and updated federation docs.
This commit is contained in:
@@ -18,13 +18,12 @@ with **A3 CLI tooling** (`validate`, `query`, `export`) atop Markdown-first
|
||||
authoring.
|
||||
|
||||
The two documents are **directionally aligned** on registry-first reuse, four
|
||||
maturity dimensions, and human/agent consumers. REUSE-WP-0003 through
|
||||
REUSE-WP-0009 closed the priority gaps from section 8 except network
|
||||
federation. Remaining gaps are primarily scale and cross-org sync:
|
||||
maturity dimensions, and human/agent consumers. REUSE-WP-0003 through REUSE-WP-0010 closed the priority gaps from section 8.
|
||||
Remaining gaps are primarily document cross-coverage and operational polish:
|
||||
|
||||
1. **Network federation** — local compose only; no remote index fetch.
|
||||
2. **Document cross-coverage** — SCOPE still carries operational detail INTENT
|
||||
1. **Document cross-coverage** — SCOPE still carries operational detail INTENT
|
||||
omits; INTENT success criteria are not fully enumerated in SCOPE.
|
||||
2. **Live federation sync** — on-demand HTTP fetch only; no polling or webhooks.
|
||||
|
||||
**Current reuse-surface vector (self-assessment):** `D5 / A3 / C4 / R3`
|
||||
|
||||
@@ -276,7 +275,7 @@ core commands. Individual registered capabilities may carry their own evidence
|
||||
| Priority | Gap | Outcome | Status |
|
||||
|---|---|---|---|
|
||||
| 13 | Interactive catalog | `docs/catalog/search.html` + `registry.json` | Closed (WP-0007) |
|
||||
| 15 | Network federation | Remote index fetch and cross-org sync | Open (WP-0010) |
|
||||
| 15 | Network federation | HTTP URL sources + cache in `federation compose` | Closed (WP-0010) |
|
||||
| 16 | Graph UI | `docs/graph/index.html` explorer | Closed (WP-0008) |
|
||||
|
||||
---
|
||||
@@ -302,4 +301,5 @@ core commands. Individual registered capabilities may carry their own evidence
|
||||
| 2026-06-15 | REUSE-WP-0006 expanded registry to 12 capabilities; relation hygiene clean |
|
||||
| 2026-06-15 | REUSE-WP-0007 closed priority 13 (searchable catalog UI) |
|
||||
| 2026-06-15 | REUSE-WP-0008 closed priority 16 (graph explorer) |
|
||||
| 2026-06-15 | REUSE-WP-0009 added pytest suite and CI fail-on-warnings; vector R3 |
|
||||
| 2026-06-15 | REUSE-WP-0009 added pytest suite and CI fail-on-warnings; vector R3 |
|
||||
| 2026-06-15 | REUSE-WP-0010 closed priority 15 (HTTP remote federation + cache) |
|
||||
@@ -11,6 +11,10 @@ helix_forge capabilities may be registered in multiple repositories. Federation
|
||||
composes capability indexes from configured sources into a single discovery
|
||||
surface without silently merging duplicate IDs.
|
||||
|
||||
Sources may be **local filesystem paths** or **remote HTTP(S) URLs** (git raw
|
||||
endpoints, published index artifacts, etc.). Remote indexes are cached under
|
||||
`registry/federation/cache/` for offline reuse and faster compose.
|
||||
|
||||
## Manifest
|
||||
|
||||
`registry/federation/sources.yaml` lists index sources:
|
||||
@@ -24,6 +28,14 @@ sources:
|
||||
index: registry/indexes/capabilities.yaml
|
||||
enabled: true
|
||||
required: true
|
||||
|
||||
- repo: sibling-repo
|
||||
url: https://git.example.com/org/sibling-repo/raw/main/registry/indexes/capabilities.yaml
|
||||
enabled: false
|
||||
required: false
|
||||
cache_ttl_seconds: 86400
|
||||
auth_env: FEDERATION_TOKEN
|
||||
auth_header: Authorization
|
||||
```
|
||||
|
||||
Schema: `schemas/federation.schema.yaml`
|
||||
@@ -33,26 +45,44 @@ Schema: `schemas/federation.schema.yaml`
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `repo` | Source repository slug |
|
||||
| `index` | Path to `capabilities.yaml` (repo-relative or `~/...`) |
|
||||
| `index` | Local path to `capabilities.yaml` (repo-relative or `~/...`) |
|
||||
| `url` | Remote HTTP(S) URL to a `capabilities.yaml` index |
|
||||
| `enabled` | Include this source in compose |
|
||||
| `required` | Fail compose if index missing when enabled |
|
||||
| `required` | Fail compose if index missing or remote fetch fails with no cache |
|
||||
| `domain` | Optional domain label |
|
||||
| `cache_ttl_seconds` | Reuse cached remote index for this many seconds (`0` = always refetch) |
|
||||
| `auth_env` | Environment variable holding token or full header value for `url` sources |
|
||||
| `auth_header` | HTTP header for `auth_env` (default `Authorization`) |
|
||||
|
||||
Each source must specify **either** `index` **or** `url`, not both.
|
||||
|
||||
Sibling repos (`state-hub`, `feature-control`, `identity-canon`) are listed as
|
||||
disabled placeholders until they publish registry indexes.
|
||||
disabled local placeholders until they publish registry indexes. A disabled
|
||||
`example-remote` URL source illustrates HTTP federation.
|
||||
|
||||
## Compose workflow
|
||||
|
||||
```bash
|
||||
reuse-surface federation compose
|
||||
reuse-surface federation compose --refresh # bypass remote cache
|
||||
```
|
||||
|
||||
Writes `registry/indexes/federated.yaml` with:
|
||||
|
||||
- Merged `capabilities` from all enabled sources
|
||||
- `source_repo` and `source_index` on every row
|
||||
- `source_url` when the row came from a remote source
|
||||
- `collision_policy` and per-source counts
|
||||
|
||||
### Remote cache
|
||||
|
||||
Fetched URL indexes are stored at `registry/federation/cache/<repo>.yaml` with
|
||||
metadata in `<repo>.meta.yaml`. The cache directory is gitignored; only
|
||||
`.gitkeep` is tracked.
|
||||
|
||||
When a refetch fails, compose reuses a stale cache and emits a warning. Required
|
||||
remote sources without cache fail compose with a clear error.
|
||||
|
||||
### Collision policy
|
||||
|
||||
`warn` (default): duplicate IDs across sources are kept but reported as
|
||||
@@ -62,9 +92,17 @@ warnings. Consumers must inspect `source_repo` before choosing an entry.
|
||||
|
||||
1. Run `reuse-surface federation compose` after manifest or sibling index changes.
|
||||
2. Read `registry/indexes/federated.yaml` for cross-repo discovery.
|
||||
3. Open `path` in the source repo for full entry detail when local.
|
||||
3. Open `path` in the source repo for full entry detail when local; follow
|
||||
`source_url` / `source_index` when remote.
|
||||
4. Run `reuse-surface graph --check` before relying on relation navigation.
|
||||
|
||||
### Cross-repo discovery without local checkout
|
||||
|
||||
Enable a `url` source pointing at a published raw index (Gitea, GitHub, static
|
||||
host). Set `auth_env` when the endpoint requires a token. Agents on machines
|
||||
without sibling repo clones can still compose a federated view from HTTP sources
|
||||
plus the local `reuse-surface` index.
|
||||
|
||||
## Relation graphs
|
||||
|
||||
```bash
|
||||
@@ -82,9 +120,12 @@ federated ID set.
|
||||
Gitea CI runs:
|
||||
|
||||
```bash
|
||||
reuse-surface validate --relations
|
||||
reuse-surface validate --relations --fail-on-warnings
|
||||
reuse-surface federation compose
|
||||
reuse-surface catalog
|
||||
reuse-surface graph --check --fail-on-warnings
|
||||
pytest -q
|
||||
```
|
||||
|
||||
Warnings on broken relations or missing optional sibling indexes do not fail CI;
|
||||
schema validation errors do.
|
||||
CI uses local sources only (remote examples are disabled). Warnings on missing
|
||||
optional sibling indexes do not fail CI; schema validation errors do.
|
||||
Reference in New Issue
Block a user