From a2f15806f5f952263a169a9cfc44ce8a5728348a Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 15 Jun 2026 02:19:43 +0200 Subject: [PATCH] spec(SHARD-WP-0001): yawex-derived federation requirements (ADR-01..06); workplan done MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Delivers spec/FederationRequirements.md — six ADR-ready design notes mapping the yawex prior art onto the hardened CoreArchitectureBlueprint: ADR-01 cross-shard resolution (yawex PageLookUp states as a checklist, keyed on identity), ADR-02 namespace/path + shard roles (path=placement not identity), ADR-03 derived views core-vs-adapter (BackLinks/RecentChanges/AllPages/SiteMap core, Search hybrid), ADR-04 concrete provenance envelope fields, ADR-05 overlay/ patch/append lifecycle, ADR-06 wikilink+red-link extension (model core, render UI). Resolves findings open-Qs #1-#3; access-model thread ratified. Flips WP-0001 done. Co-Authored-By: Claude Opus 4.8 --- SCOPE.md | 2 +- spec/FederationRequirements.md | 214 ++++++++++++++++++ spec/README.md | 1 + workplans/SHARD-WP-0001-yawex-requirements.md | 16 +- 4 files changed, 224 insertions(+), 9 deletions(-) create mode 100644 spec/FederationRequirements.md diff --git a/SCOPE.md b/SCOPE.md index 41823a3..d526bd1 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -22,7 +22,7 @@ Learnings update both SCOPE and INTENT where necessary. | Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) | | Demand | NetKingdom integration asks captured, not yet negotiated | | Spec | CoreArchitectureBlueprint (whole-system architecture, hardened via SHARD-WP-0005) + ArchitectureBlueprint (auth/history) drafted; UseCaseCatalog 84 UCs from research; PRD/TSD scaffolds | -| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (18 tasks: T1–T10 federation + T11–T16 adapter contract + T17 federation-model taxonomy + T18 computational content, re-folded from synthesis v3 + the computational page model); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` **done** (all 8 computational-knowledge dives T1–T8 complete + "computational page model" synthesis); `SHARD-WP-0005` **done** (9 tasks: CoreArchitectureBlueprint hardened against the 260615 review); `SHARD-WP-0006` **done** (5 tasks: round-2 hardening — overview reconciled, event-sourced coordination + append authority, adapter conformance, incremental correctness + I-2 verification) | +| Work | `SHARD-WP-0001` **done** (6 ADRs: yawex-derived federation requirements → `spec/FederationRequirements.md`); `SHARD-WP-0002` active (18 tasks: T1–T10 federation + T11–T16 adapter contract + T17 federation-model taxonomy + T18 computational content, re-folded from synthesis v3 + the computational page model); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` **done** (all 8 computational-knowledge dives T1–T8 complete + "computational page model" synthesis); `SHARD-WP-0005` **done** (9 tasks: CoreArchitectureBlueprint hardened against the 260615 review); `SHARD-WP-0006` **done** (5 tasks: round-2 hardening — overview reconciled, event-sourced coordination + append authority, adapter conformance, incremental correctness + I-2 verification) | ## In Scope (today) diff --git a/spec/FederationRequirements.md b/spec/FederationRequirements.md new file mode 100644 index 0000000..72142cf --- /dev/null +++ b/spec/FederationRequirements.md @@ -0,0 +1,214 @@ +# FederationRequirements — yawex-derived design notes + +Status: **draft for review** · Date: 2026-06-15 · Deliverable of **SHARD-WP-0001** + +Concrete, ADR-ready design decisions for the union/federation layer, distilled from the yawex +prior-art review (`research/260608-yawex-prior-art/findings.md`) and made **consistent with** +`spec/CoreArchitectureBlueprint.md` (the whole-system architecture) and `INTENT.md`. yawex is +*inspiration and a case checklist*, never an interface to inherit (decision 2026-06-08). + +These are **requirements** (what the union must do); `spec/FederationArchitecture.md` +(SHARD-WP-0002) is the architecture that realises them. UC references are to +`spec/UseCaseCatalog.md`. + +**Resolved precondition.** The yawex "minimal access model in core" thread is **settled**: +authorization-in-core / authentication-delegated, the L0–L4 ladder — INTENT amended, +`ArchitectureBlueprint.md` ratified. Auth is therefore *not* re-litigated here; these ADRs +assume it (the resolver and overlay paths run behind the L5 PEP/PDP). + +--- + +## ADR-01 — Cross-shard page resolution + +**Status:** accepted · **CoreArchitectureBlueprint:** §8.4 (union), §7.2 (identity) + +**Context.** yawex's most-tested component was `PageLookUp`, a context-relative resolver with +ordered states `LOCAL, CLIMB, DROP, GLOBAL, REMOTE, SWITCH, JUMP, VIRTUAL, FAILED`. Decision +(2026-06-08): **inspiration only** — use the states as a *checklist of cases*, design the +federation resolver fresh. + +**Decision.** Resolution is a pure function over the derived union: +`resolve(name, from_context, policy) → ResolutionResult`, evaluated in a defined order and +keyed on **page identity** (the stable handle), not path (I-9): + +1. **Same-namespace, same-shard** exact identity match — *(yawex LOCAL)*. +2. **Namespace walk** — climb/descend ancestor namespaces of `from_context` — *(CLIMB/DROP)*. +3. **Union lookup** — match identity / alias-table entry across all attached shards — *(GLOBAL)*. +4. **Equivalence set** — if several shards hold an equivalent page, return the **chorus set**; + the *canonical-source* policy preset (chorus / designated-canonical) decides presentation — + never silently pick one (union without erasure, I-4). *(implicit in yawex; explicit here.)* +5. **Projection / virtual** — a page whose content lives in a remote shard (lazy replication, + *REMOTE*) or is computed/query-defined (derivation, *VIRTUAL*). +6. **Explicit address** — a shard- or space-qualified reference resolves directly — *(SWITCH/JUMP)*. +7. **Not found** → **red-link**: a resolvable *target* with no page yet, createable (UC-23) — + *(FAILED)*. + +`ResolutionResult` carries the resolved identity (or red-link), the placement(s), the +provenance envelope (ADR-04), and — on ambiguity — the full chorus set. Resolution is a +**derived-tier read** (union graph + equivalence index, §8.4); it is incremental-maintained and +respects the L5 authz filter (a principal never resolves to content it can't see). + +**Consequences.** Deterministic, policy-driven, no privileged shard; every yawex state has a +home without inheriting its structure. **Open:** cross-space `JUMP` addressing syntax ties to +the portable-address scheme (CoreArchitectureBlueprint O-6). + +--- + +## ADR-02 — Namespace / path model & shard roles + +**Status:** accepted · **CoreArchitectureBlueprint:** §7.2 (identity≠placement), §4 (shard mode) + +**Context.** yawex modelled **topics as directories** (a topic's gateway page shares the dir +name) with relative (`../`) / absolute (`/`) paths and normalization, and page classes +`local / global / virtual`. + +**Decision.** +- The union exposes a **namespace tree**; a **path is a placement coordinate, not an identity** + (I-9). One identity may have **N placements** (paths/shards) → a DAG, no single canonical path. +- **Path grammar:** relative (`../`, `./`) resolved against `from_context`; absolute (`/`) + against the space root; **normalization** (collapse `.`/`..`; case & separator handling) is a + per-space **policy** knob (§10) — defaulting to case-preserving, `/`-separated. +- **yawex page classes → shard roles / modes:** `local → canonical`, `global → cross-cutting` + (a shard whose pages augment/overlay across namespaces), `virtual → projected/computed`. These + are the INTENT shard modes (read-only · write-through · mirrored · projected · cached · + canonical), selected by policy, constrained by the capability profile. +- Cross-shard name collisions are resolved by ADR-01's order + equivalence, not by a global + flat namespace. + +**Consequences.** The namespace is a navigable view over placements (a dimension of the union), +not a storage layout. **Open:** per-space normalization policy defaults (case-sensitivity, +unicode) — a policy preset, bundled per persona (O-8). + +--- + +## ADR-03 — Union-level derived views: core vs adapter + +**Status:** accepted · **CoreArchitectureBlueprint:** §8.4 (derived query index), §8.7 +(incremental) · **resolves findings open-Q #3** + +**Context.** yawex shipped BackLinks, RecentChanges, AllPages, SiteMap, full-text Search. + +**Decision.** Each is a **derived-tier** view (rebuildable, incrementally maintained); classify +by where it is computed: + +| View | Tier | Rationale | +|------|------|-----------| +| **BackLinks** | **core** | the link-graph over the union — squarely shard-wiki's concern (UC-18); the strongest core view | +| **RecentChanges** | **core** | merge the coordination journal (§8.1) with shard change events (notify/poll, §8.8) across the union (UC-17) | +| **AllPages / SiteMap** | **core** | an enumeration/projection of the union graph; cheap | +| **Search (full-text)** | **hybrid** | **delegate** to a shard's native search where the native-query axis allows; **else** build a derived index over projections (the Logseq DataScript-over-files pattern, UC-19/UC-63) | + +All are computed in the derived tier and carry provenance; **presentation is L6 (UI)**, never +hard-coded in core (mechanism over policy, I-7). + +**Consequences.** BackLinks/RecentChanges/AllPages/SiteMap are core deliverables; Search is a +capability-gated delegate-or-derive. **Open:** cross-shard search ranking/relevance is a policy +concern, not core mechanism. + +--- + +## ADR-04 — Provenance & freshness model (concrete fields) + +**Status:** accepted · **CoreArchitectureBlueprint:** §7.3 (layered provenance), §8.8 (freshness) + +**Context.** yawex `Page::info` exposed modtime (and TODO'd last-editor / hits / edits). INTENT +mandates explicit provenance & freshness (I-4). + +**Decision.** Concretise the **provenance envelope** (layered: page-level + span-level deltas): + +``` +ProvenanceEnvelope (page-level): + source_shard # which shard this came from + source_rev? # shard-native revision id, if the shard exposes one + observed_at # when shard-wiki last read it + liveness # static | captured-snapshot | live-over-files | view-time | irreducibly-live + staleness_state # live | fresh | stale | unavailable + authz_context # the L5 context under which it was read (no-leak, §9) + overlay_state # none | draft | patch-pending | applied + divergence[] # equivalence-set peers that differ (the chorus, ADR-01 step 4) + derivation_lineage? # for derived/derivation-projection content (source → this view) +Span-level: only the fields that differ from the page envelope (effective = page ⊕ delta). +``` + +`freshness = (observed_at, source_rev?, staleness_state)`; `unavailable` is the dead-shard +state (feeds ADR-03 RecentChanges and the union-under-unavailability open item O-11). This +envelope is the data behind union-without-erasure and the input to conflict display. + +**Consequences.** Provenance is queryable, layered (cheap per-span), and drives views/conflict +UI. **Open / DROP:** hits/edits *analytics* are an L6/analytics concern, out of core (dropped +from the yawex feature set). + +--- + +## ADR-05 — Overlay / lightweight-patch model + +**Status:** accepted · **CoreArchitectureBlueprint:** §8.2 (overlay engine), §8.1 (decision +log), §8.6 (apply-under-drift) + +**Context.** yawex had an append + threaded-comment workflow (edit a page without a full +rewrite). INTENT mandates overlay-before-mutation (I-5). + +**Decision.** The overlay lifecycle for any shard below write-through capability: + +1. **Draft** — an edit becomes a draft recorded as a **coordination-canonical event** in the + space's decision log (§8.1); the unapplied overlay is the **local truth**, fully attributed + (`overlay_state`, ADR-04). +2. **Patch / MR** — rendered in the shard's **native syntax (lossless)** or **Markdown + (lossy-with-fidelity-report)** per the translation axis. +3. **Apply** — only on explicit intent, only where profile + policy permit, with + **apply-under-drift** semantics (§8.6: fast-forward / three-way / refuse-and-re-present). +4. **Append / comment** = a **constrained overlay subtype** (purely additive, low-conflict) — + the direct generalisation of yawex's workflow; safe even on the most limited shards. + +This makes read-only, rate-limited, and lossy shards **editable** without silent remote +mutation (graceful degradation, I-8; no silent mutation). + +**Consequences.** One overlay mechanism spans drafts, patches, MRs, and comments; storage is the +decision log (no separate store). **Open:** whether threaded-comment threading is a first-class +overlay subtype or a generic structured-append — defer to implementation. + +--- + +## ADR-06 — Markdown link semantics (wikilink + red-link) + +**Status:** accepted · **CoreArchitectureBlueprint:** §7 (page model), ADR-01 (resolution) · +**resolves findings open-Q #2** + +**Context.** yawex had CamelCase WikiLinks, `[[free links]]`, red-`?` links for nonexistent +pages, `::` labels. INTENT mandates Markdown-first; TRANSFORM the *semantics*, drop the bespoke +syntax. + +**Decision.** +- Adopt a **CommonMark wikilink extension**: `[[Target]]` / `[[Target|label]]`, resolved via + ADR-01 (by identity/name across the union). +- **Red-link** = a wikilink whose target resolves to FAILED (ADR-01 step 7): a valid, + *createable* target with no page yet (the soft-create affordance, UC-23). +- **CamelCase auto-linking is OFF by default** (a legacy affordance); opt-in **per space** for + migrating CamelCase wikis (UseModWiki/c2 lineage, UC-25). +- Links are stored **as text** (git-diffable; structure/links federate iff in-text). +- **Boundary (resolves open-Q #2):** the link **model + resolution is core**; the **visual + rendering** of wikilinks/red-links is a **reference-UI (L6)** concern. Core decides *what a + link resolves to and whether it's a red-link*; the UI decides how it looks. + +**Consequences.** Markdown-first, backend-neutral, resolution-unified with ADR-01. **Open:** +cross-shard wikilink target disambiguation syntax ties to the portable-address scheme (O-6). + +--- + +## Coverage & open questions + +- **Findings open-Q #1** (per-page vs per-shard ACL) — answered by the settled access model: + per-shard / per-namespace default, per-page ACL opt-in at L4 (`ArchitectureBlueprint.md`). +- **Findings open-Q #2** (wikilink core vs UI) — ADR-06: model/resolution core, presentation UI. +- **Findings open-Q #3** (derived views core vs adapter) — ADR-03. +- Carried to CoreArchitectureBlueprint §12: cross-space addressing syntax (O-6), namespace + normalization presets (O-8), cross-shard search ranking (policy), union-under-unavailability + (O-11). + +## Acceptance (SHARD-WP-0001) + +Each task T1–T6 has an ADR (T1→ADR-01, T2→ADR-02, T3→ADR-03, T4→ADR-04, T5→ADR-05, T6→ADR-06); +all honour INTENT (mechanism over policy, union without erasure, overlay before mutation, +capability-aware adapters) and are consistent with `CoreArchitectureBlueprint.md`. The +access-model thread is ratified (not deferred). The next implementation workplan (domain model / +adapter contract) can proceed without unresolved yawex-derived design gaps. diff --git a/spec/README.md b/spec/README.md index 7723110..a4134e6 100644 --- a/spec/README.md +++ b/spec/README.md @@ -7,6 +7,7 @@ Background on document types: InfoTechPrimers on coulomb.social. | File | Status | Role | |------|--------|------| | `CoreArchitectureBlueprint.md` | draft for review | **Whole-system architecture** — layers, abstractions, load-bearing decisions (synthesised from all research) | +| `FederationRequirements.md` | draft for review | yawex-derived union/federation design notes — resolution, namespace, derived views, provenance, overlay, links (ADR-01…06; SHARD-WP-0001) | | `ProductRequirementsDocument.md` | draft scaffold | What the product must deliver | | `TechnicalSpecificationDocument.md` | draft scaffold | How the system is built | | `UseCaseCatalog.md` | draft | 84 use cases promoted from c2 + yawex + ~23-system research | diff --git a/workplans/SHARD-WP-0001-yawex-requirements.md b/workplans/SHARD-WP-0001-yawex-requirements.md index c2df110..83c6b77 100644 --- a/workplans/SHARD-WP-0001-yawex-requirements.md +++ b/workplans/SHARD-WP-0001-yawex-requirements.md @@ -4,11 +4,11 @@ type: workplan title: "shard-wiki requirements from yawex prior art" domain: whynot repo: shard-wiki -status: active +status: done owner: tegwick topic_slug: whynot created: "2026-06-08" -updated: "2026-06-08" +updated: "2026-06-15" state_hub_workstream_id: "0ed023a2-760b-4990-b931-8ee1f41ea08f" --- @@ -35,7 +35,7 @@ is treated as settled. ```task id: SHARD-WP-0001-T1 -status: todo +status: done priority: high state_hub_task_id: "ebc036e4-1c3c-4e39-959b-1e37bc1049ad" ``` @@ -49,7 +49,7 @@ projections), not as the interface. Decision 2026-06-08: inspiration only. ```task id: SHARD-WP-0001-T2 -status: todo +status: done priority: high state_hub_task_id: "431b4d28-3768-468f-b76f-4767f5aa1888" ``` @@ -62,7 +62,7 @@ canonical/cross-cutting/projected-computed. ```task id: SHARD-WP-0001-T3 -status: todo +status: done priority: medium state_hub_task_id: "564545ec-84f7-4c3d-a779-5b9c356919f4" ``` @@ -75,7 +75,7 @@ federated union) is the strongest core candidate. ```task id: SHARD-WP-0001-T4 -status: todo +status: done priority: medium state_hub_task_id: "738326f5-b76c-4ecf-8843-7c762c3647d8" ``` @@ -88,7 +88,7 @@ diverges-from?. ```task id: SHARD-WP-0001-T5 -status: todo +status: done priority: medium state_hub_task_id: "a268de6a-d1ef-4ed9-9828-c70991e1a772" ``` @@ -101,7 +101,7 @@ threaded-comment workflow. ```task id: SHARD-WP-0001-T6 -status: todo +status: done priority: low state_hub_task_id: "a7499f3e-911c-455b-afc3-eefc401eb60d" ```