From fae3cbde8b64e6c09cecb6a768b6c616227139a6 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 14 Jun 2026 14:01:49 +0200 Subject: [PATCH] synthesis: shard spectrum across nine systems; fold into SHARD-WP-0002 (T11 eleven spectra, new T16) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-dive synthesis (research/260614-shard-spectrum-synthesis/) reading the two Nelson conceptual systems, four engines, and three modern tools across each other: a shard family matrix and eleven capability spectra (addressing, content identity, structure, history, native query, translation, attachment mode, operational envelope, access grant, write granularity, content types) — positions anchored at both ends by a real system, federation ops degrading by position. Through-lines: files-canonical/index-derived wins; fine-grained addressing is adoptable; transclusion=clone=embed is one primitive; structure/history federate iff in-text; attach mode is a per-binding choice; Notion proves the platform can enforce no-silent-mutation. Folded into SHARD-WP-0002: T11 reframed around the eleven spectra; T12 page model stretched four ways (prose + typed records + non-Markdown assets + query-defined); T13 adds Notion supplement; T14 rewritten as the three-mode attachment taxonomy (file-store / in-engine-host / external-API); T15 adds lossy-with-fidelity-report; new T16 (addressing, content identity, dimensional/query navigation). UC coverage UC-34-43 -> UC-34-59. Workplan now 16 tasks. No new UCs (synthesis only). Co-Authored-By: Claude Opus 4.8 --- SCOPE.md | 2 +- .../260614-shard-spectrum-synthesis/README.md | 47 ++++ .../findings.md | 220 ++++++++++++++++++ research/README.md | 3 +- .../SHARD-WP-0002-federation-architecture.md | 200 ++++++++++++---- 5 files changed, 431 insertions(+), 41 deletions(-) create mode 100644 research/260614-shard-spectrum-synthesis/README.md create mode 100644 research/260614-shard-spectrum-synthesis/findings.md diff --git a/SCOPE.md b/SCOPE.md index 7c3e1ce..dba591c 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -22,7 +22,7 @@ Learnings update both SCOPE and INTENT where necessary. | Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion deep dives (`research/260614-*/`) | | Demand | NetKingdom integration asks captured, not yet negotiated | | Spec | Architecture blueprint drafted; UseCaseCatalog 59 UCs from research; PRD/TSD scaffolds | -| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) | +| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract) | ## In Scope (today) diff --git a/research/260614-shard-spectrum-synthesis/README.md b/research/260614-shard-spectrum-synthesis/README.md new file mode 100644 index 0000000..1379c13 --- /dev/null +++ b/research/260614-shard-spectrum-synthesis/README.md @@ -0,0 +1,47 @@ +# 260614 — Shard spectrum synthesis (one capability model across nine systems) + +Date: 2026-06-14 + +## What this is + +A **synthesis** (no new external research) that reads the nine studied systems *across* +each other and distills them into a single comparative model feeding the **shard adapter +contract** (`SHARD-WP-0002`). + +The nine: two conceptual ancestors (**Xanadu**, **ZigZag**), four engines (**XWiki, +TWiki, Foswiki** + the wiki-engines landscape), and three modern tools (**Roam, +Obsidian, Notion**), against the federation/origin research. + +Centerpieces: +- **The shard family matrix** — every candidate backend × {substrate, attach mode, + addressing, structure, history, query, transclusion, →Markdown, write granularity}, + with Xanadu/ZigZag as the ideal anchors. +- **The eleven capability spectra** — the claim that the adapter contract should model + *positions on spectra* (addressing, content identity, structure, history, query, + translation, attachment mode, operational envelope, access grant, write granularity, + content types), each anchored at both ends by a real system, with federation ops + degrading by position. +- **UC-44–UC-59 → workplan task mapping**, including one genuinely new thread (T16: + addressing, content identity, dimensional/query navigation). + +## Contents + +| Path | Role | +|------|------| +| `findings.md` | Family matrix, the eleven spectra, cross-cutting through-lines, UC→task fold-in, recommendations/decisions, escalated open questions | + +## Status + +Synthesis complete. No new use cases (consolidation only). Feeds `SHARD-WP-0002`: T11 +reframed around the eleven spectra; T12 page-model breadth (prose + typed records + +non-Markdown assets + query-defined); T13 history confirmed; T14 three-mode attachment +taxonomy; T15 lossy-with-fidelity-report; **new T16** (addressing, content identity, +dimensional/query navigation). UC coverage in the workplan extended UC-34–UC-43 → +UC-34–UC-59. + +**Through-lines recorded:** files-canonical/index-derived is the winning architecture; +fine-grained addressing is real and adoptable (adopt native, fingerprint otherwise); +transclusion ⇄ clone ⇄ embed is one primitive; structure/history federate iff in text; +attach mode is a per-binding capability-gated choice; Notion proves the platform can +*enforce* no-silent-mutation via scoped grants. + diff --git a/research/260614-shard-spectrum-synthesis/findings.md b/research/260614-shard-spectrum-synthesis/findings.md new file mode 100644 index 0000000..a20b50e --- /dev/null +++ b/research/260614-shard-spectrum-synthesis/findings.md @@ -0,0 +1,220 @@ +# Synthesis — the shard spectrum: one capability model across nine systems + +Date: 2026-06-14 +Source kind: **synthesis** — consolidates the engine and tool deep dives plus the two +Nelson conceptual systems into a single comparative model that feeds the **shard adapter +contract** (`SHARD-WP-0002` T11–T16) +Lens: shard-wiki — what a backend must expose to participate, expressed as *spectra* of +capability rather than a yes/no checklist + +> Purpose. Nine systems have been studied: two conceptual ancestors (**Xanadu**, +> **ZigZag**), four engines (**XWiki, TWiki, Foswiki**, + the wiki-engines landscape), +> and three modern tools (**Roam, Obsidian, Notion**), against the federation and +> origin research (federation-concepts, c2, yawex). They were studied one at a time; +> this document reads them *across* each other. The payoff is a small set of +> **capability spectra** — addressing, structure, history, query, translation, +> attachment mode, operational envelope, access grant — each anchored at both ends by a +> real system. That spectrum *is* the adapter contract's design surface. It also folds +> the post-engine use cases (**UC-44–UC-59**) into the existing `SHARD-WP-0002` tasks. + +Inputs: `research/260608-federation-concepts`, `research/260608-wikiengines-overview`, +`research/260613-{xwiki,twiki,foswiki}-deep-dive`, +`research/260614-{xanadu,zigzag,roam,obsidian,notion}-deep-dive`. Output target: +`spec/TechnicalSpecificationDocument.md` (adapter contract) via `SHARD-WP-0002`. + +--- + +## 1. The shard family matrix + +Candidate-shard backends across the dimensions that matter to the contract. (Xanadu and +ZigZag are *not* shards — they are the conceptual ideals each column aspires to; listed +last as reference anchors.) + +| Backend | Substrate | Attach mode | Addressing | Structure | History | Native query | Transclusion | →Markdown | Write granularity | +|---------|-----------|-------------|-----------|-----------|---------|--------------|-------------|-----------|-------------------| +| **Git folder / repo** | files | file-store | path (+ commit) | flat MD + frontmatter | **git-native** | no | no | native | per-file | +| **Obsidian vault** | files | file-store **or** in-app plugin | path + **in-file `^id`** | frontmatter (in-file) | none (Git plugin) | plugin (Dataview) | in-file `![[ ]]` | native (OFM) | per-file | +| **TWiki** | files + RCS | file-store / API | path | `%META%` in-file | **open file (RCS)** | no | include macro | lossless (TML) | per-topic | +| **Foswiki** | pluggable store | file-store / API | path | `%META%` in-file, N records | open file / PlainFile | no | include macro | lossless (TML) | per-topic | +| **XWiki** | DB (Hibernate) | in-engine host / REST | path | **XObjects/XClass** | internal (`xwikircs`) | yes (XWQL) | macro | engine syntax | per-page / per-object | +| **Roam** | client DataScript | **in-app host only** | **store UUID** | `key::` attrs | internal (txn log) | **yes (Datalog)** | **block embed** | (Roam MD) | **per-block** | +| **Notion** | hosted Postgres | **external API only** | **store UUID** | **DB schema + relations + rollups** | internal, not portable | **yes (DB query)** | synced block | **lossy** | **per-block** | +| **TiddlyWiki** | single file | file-store | path | typed tiddlers | none | no | transclusion | varies | **whole-file** | +| **MediaWiki / Confluence** | DB | API | path | wikitext / macros | internal-only | limited | templates | lossy | per-page | +| — *Xanadu (ideal)* | permascroll | — | **tumbler (span)** | spans + links | permanent | — | **content-identity** | — | span | +| — *ZigZag (ideal)* | cells/dims | — | cell id | **N dimensions** | — | dimension walk | clone | — | cell | + +Reading the matrix top to bottom is reading shard-wiki's difficulty gradient: **Git +folder → Obsidian** are the friction-free, INTENT-native cases; **TWiki/Foswiki** add +syntax translation but stay file/diffable; **XWiki/Roam/Notion** add DB structure, store +addressing, and (for Roam/Notion) non-file substrates; **Notion** is the hardest +(closed, hosted, lossy, rate-limited). Xanadu/ZigZag mark where the *ideals* live. + +--- + +## 2. The capability spectra (the contract's real shape) + +Each capability is **not boolean** — it is a position on a spectrum with a real anchor at +each end. This is the core synthesis claim: the adapter contract should model *positions*, +and federation operations should degrade by position. + +1. **Addressing granularity** — `none → whole-page(path) → in-file span(Obsidian ^id) → + store-minted span(Roam/Notion UUID) → portable tumbler(Xanadu ideal)`. (UC-51, UC-44/45.) +2. **Content identity** — `none → path/title match → content fingerprint(hash) → + span-set/equivalence(Xanadu)`. Drives equivalence + reverse-transclusion. (UC-46, UC-27.) +3. **Structure** — `flat Markdown → in-file frontmatter → in-file %META% → DB + objects(XWiki) → DB schema+relations+rollups(Notion)`. The git-diffable forms federate + easily; DB-locked forms need sidecar + fidelity reporting. (UC-34, UC-39, UC-58.) +4. **History** — `none → internal-only/not-portable(Notion/Confluence) → open file + format(TWiki RCS) → git-native(Git/Obsidian-with-Git)`. Internal ⇒ *supplement*; + open-file ⇒ *import*; git-native ⇒ *adopt*. (UC-36, UC-41.) +5. **Native query** — `none → text search → datalog(Roam) → DB query(Notion/XWiki)`. + Where present, **delegate** derived-view computation; else compute over the projection. + (UC-52, UC-05, UC-54.) +6. **Translation to Markdown** — `native → lossless round-trip(Foswiki TML↔HTML) → + lossy-with-fidelity-report(Notion)`. Lossless ⇒ writable; lossy ⇒ read/projection floor + + visible fidelity loss. (UC-42, UC-59, UC-03.) +7. **Attachment mode** — `file-store direct(Obsidian/TWiki) → in-engine hosted + adapter(Roam/XWiki) → external-API only(Notion)`. A backend may offer **more than one** + (Obsidian offers both; TWiki file-or-API). (UC-40, UC-38, UC-50, UC-57.) +8. **Operational envelope** — `local/unbounded → rate-limited+eventually-consistent+ + paginated(Notion ~3 rps)`. Determines whether projection can be live or must be + cache/poll/webhook. (UC-57, UC-31.) +9. **Access grant** — `open(L0) → token → OAuth scoped + revocable(Notion)`. The backend + may *enforce* no-silent-mutation (Notion scoped grants). Ties the authz-in-core ladder. + (UC-57, [[shard-wiki-auth-in-core-decision]].) +10. **Write granularity** — `whole-file(TiddlyWiki) → per-page(Git/Obsidian/TWiki) → + per-block(Roam/Notion)`. Sets overlay/patch/lock/conflict scope. (UC-35.) +11. **Content types** — `Markdown-only → typed records(XWiki/Notion DB) → non-Markdown + assets(Excalidraw/Canvas/attachments)`. The page model must stretch to all three. + (UC-55, UC-58.) + +Design consequence: **T11's capability vocabulary should be these eleven spectra**, not +the original flat `read/write/diff/...` list. The flat verbs remain as the *operations*; +the spectra are the *profile* that says how well each verb is supported and how it +degrades. + +--- + +## 3. Cross-cutting findings (the through-lines) + +- **Files-canonical, index-derived is the winning architecture.** Obsidian's + MetadataCache, Git's working tree, and shard-wiki's own projection model all agree: + the link graph / backlinks / structured index are **derived and rebuildable**, never a + second source of truth. Roam/Notion invert this (DB canonical) and pay for it in + portability. shard-wiki keeps files+journal canonical; everything else is projection. + (Reinforces UC-05, UC-17–20; INTENT.) +- **Fine-grained addressing is real and adoptable** — Roam (`:block/uid`) and Notion + (UUID) ship it store-minted; Obsidian ships it in-file (`^id`). The Xanadu tumbler is + the unreached ideal (portable, transfinite). shard-wiki should **adopt native span IDs + where present, fingerprint where not** (UC-51 + UC-46), and treat a portable span + address as the open hard problem (Xanadu §11 Q1). +- **Transclusion ⇄ clone ⇄ embed is one primitive.** Xanadu transclusion, ZigZag clones, + Roam block embeds, Obsidian `![[ ]]`, Notion synced blocks are the same idea at + different fidelities. A single internal "reference-not-copy" primitive over an + addressable union serves UC-32/44/45/51. (Convergence first noted in the ZigZag dive.) +- **Structure and history federate iff they live in text.** The sharpest recurring + engine insight: `%META%`-in-file (TWiki/Foswiki) and frontmatter (Obsidian) diff and + travel; XObjects and Notion DBs lock in. The contract should *prefer* in-text + structure and *tolerate* DB structure via sidecar + fidelity report. +- **Three attachment modes, and a backend may span them.** The clean taxonomy + (file-store / in-engine-host / external-API) plus the observation that one backend can + offer several (Obsidian: file *or* plugin; TWiki: file *or* API) means **attach mode is + a per-binding choice, capability-gated**, not a fixed property of the backend. +- **Notion proves the platform can enforce INTENT.** Scoped, revocable grants are + no-silent-mutation made mechanical — evidence the principle is implementable, and a + model for how shard-wiki represents partial/consented visibility. +- **The page model must stretch four ways at once:** prose Markdown, typed records + (with N-per-page and inter-record relations), non-Markdown assets, and + reference/query-defined pages. This is the heaviest single demand on T12. + +--- + +## 4. How the post-engine use cases fold into the workplan + +UC-44–UC-59 (from the five 260614 dives) map onto the existing adapter-contract tasks, +with one genuinely new thread needing a home (T16): + +| UCs | Theme | Lands in | +|-----|-------|----------| +| UC-35, UC-50, UC-53, UC-57 | attachment mode + operational envelope + write granularity | **T11** (capabilities) + **T14** (binding) | +| UC-34, UC-39, UC-55, UC-58 | structured/typed payload, non-Markdown assets, DB schema+relations | **T12** (page model) | +| UC-36 | internal-only history (Notion) = supplement | **T13** (history) | +| UC-42, UC-59 | translation: lossless vs lossy-with-fidelity-report | **T15** (translation) | +| UC-31 | webhooks / push-vs-poll under eventual consistency | **T6** / **T11** envelope | +| UC-57 §6 | scoped, revocable grant; no silent mutation | **T11** access-grant + authz decision | +| **UC-44, UC-45, UC-46, UC-51** | span addressing, content identity, transclusion-as-reference | **T8** + **new T16** | +| **UC-47, UC-48, UC-52, UC-54** | dimensional navigation, query delegation, query-defined pages | **T5/T10** + **new T16** | + +**New thread (T16): Addressing, content identity, and dimensional/query navigation.** No +existing task owns (a) the portable span-address scheme + content-identity model that +transclusion/overlay/reverse-lookup need, nor (b) the dimensional/query view model +(ZigZag dimensions, Roam/Notion query, Dataview query-defined pages). These are the +Nelson-ideals-meet-modern-tools thread and warrant their own design task. + +--- + +## 5. Recommendations (decisions to record under SHARD-WP-0002) + +1. **Model capabilities as the eleven spectra (§2), not flat verbs.** Keep verbs as + operations; add the spectra as the profile. (T11.) +2. **Adopt native span IDs; fingerprint otherwise; keep portable-span-address open.** + (T16, UC-51/46.) +3. **One reference-not-copy primitive** for transclusion/clone/embed over the addressable + union. (T16/T8, UC-32/44/45.) +4. **Prefer in-text structure; tolerate DB structure via sidecar + fidelity report.** + (T12/T15, UC-34/58/59.) +5. **Attach mode is a per-binding, capability-gated choice** across the three modes; a + backend may offer several. (T14, UC-40/38/50/57.) +6. **Add operational-envelope and access-grant to the profile** as first-class fields. + (T11, UC-57.) +7. **Delegate derived views to native query engines where present; else compute over the + projection.** (T16/T5, UC-52/54.) +8. **Keep files + coordination journal canonical; all indexes/projections derived and + rebuildable.** (Architecture invariant, INTENT.) + +These honor INTENT: mechanism over policy (spectra not hard-coded behaviors), union +without erasure (fidelity reporting makes loss visible), graceful degradation (every +capability has a read/projection floor), no silent mutation (access-grant + overlay), +shard sovereignty (no backend forced to change substrate), Markdown-first/backend-neutral +(in-text preferred, DB tolerated). + +--- + +## 6. Open questions escalated by the synthesis + +1. **Portable span address** — still unsolved across heterogeneous backends (Xanadu's + unbuilt part). Wrap native IDs in a shard-scoped address? (T16.) +2. **External-API write-through at scale** — is Notion-class write viable under ~3 rps, + or read/projection/overlay/backup by default? (T14, UC-57.) +3. **Inter-record relations in the union** (UC-58) — typed links in the link graph, a + separate relation index (ZigZag many-to-many), or both? (T16/T5.) +4. **Page model breadth** — can one model carry prose + typed records + non-Markdown + assets + query-defined pages without becoming incoherent? (T12.) +5. **Dimensional navigation surface** — public API/UI paradigm or internal organizing + concept? (T16, UC-47/48.) + +--- + +## 7. Sources + +This is a synthesis; primary sources are the nine dives' `findings.md` files plus +`research/260608-federation-concepts`, `research/260608-wikiengines-overview`, +`research/260608-c2-wiki-origins`, `research/260608-yawex-prior-art`. No new external +research was performed. + +Cross-references: `spec/UseCaseCatalog.md` (UC-26–UC-59), +`workplans/SHARD-WP-0002-federation-architecture.md` (T1–T16), +`INTENT.md` (constraints), [[shard-wiki-auth-in-core-decision]]. + +--- + +## 8. Traceability + +- Consolidates: nine deep dives + federation/origin research into one capability model. +- Feeds: `SHARD-WP-0002` T11 (eleven-spectra vocabulary), T12 (page-model breadth), T13 + (history confirmed), T14 (three-mode attachment taxonomy), T15 (lossy fidelity report), + and a **new T16** (addressing, content identity, dimensional/query navigation). +- UC coverage extended in the workplan from UC-34–UC-43 to **UC-34–UC-59**. +- No UCs added (synthesis only); no boundary changes (INTENT Stability Note untouched). + diff --git a/research/README.md b/research/README.md index ad620e9..fd0f71b 100644 --- a/research/README.md +++ b/research/README.md @@ -21,4 +21,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and | 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 | | 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 | | 2026-06-14 | `260614-obsidian-deep-dive/` | Obsidian — file-over-app vaults, plugin API, ecosystem-popularity → UC signal; UC-53/54/55/56 | -| 2026-06-14 | `260614-notion-deep-dive/` | Notion — closed block-DB SaaS, external REST API only, database-as-pages; UC-57/58/59 | \ No newline at end of file +| 2026-06-14 | `260614-notion-deep-dive/` | Notion — closed block-DB SaaS, external REST API only, database-as-pages; UC-57/58/59 | +| 2026-06-14 | `260614-shard-spectrum-synthesis/` | Synthesis — shard family matrix + eleven capability spectra across nine systems; feeds SHARD-WP-0002 T11–T16 | \ No newline at end of file diff --git a/workplans/SHARD-WP-0002-federation-architecture.md b/workplans/SHARD-WP-0002-federation-architecture.md index 33496e4..4ce2186 100644 --- a/workplans/SHARD-WP-0002-federation-architecture.md +++ b/workplans/SHARD-WP-0002-federation-architecture.md @@ -27,13 +27,21 @@ UC-26–UC-33. Primary deliverable: `spec/FederationArchitecture.md` (created and filled by this workplan's tasks). -Secondary deliverable (T11–T15, added 2026-06-13): the **shard adapter contract** -constraints distilled from the engine deep dives, feeding +Secondary deliverable (T11–T16): the **shard adapter contract** constraints distilled +from the engine deep dives (T11–T15, added 2026-06-13) and the modern-tool dives + +cross-dive synthesis (T16 and scope extensions, added 2026-06-14), feeding `spec/TechnicalSpecificationDocument.md`. Federation architecture decides *what the union does*; the adapter contract decides *what a backend must expose to participate*. The two are co-dependent (the capability matrix T10 consumes the contract vocabulary T11 defines), so they live in one workplan. +The cross-dive synthesis (`research/260614-shard-spectrum-synthesis/findings.md`) +reframes the contract around **eleven capability spectra** (addressing, content +identity, structure, history, native query, translation, attachment mode, operational +envelope, access grant, write granularity, content types) — positions on a spectrum +anchored at both ends by a real system, with federation ops degrading by position — +rather than a flat verb checklist. + ## Context - Federation research: `research/260608-federation-concepts/findings.md` @@ -44,8 +52,21 @@ T11 defines), so they live in one workplan. - `research/260613-twiki-deep-dive/findings.md` (file+RCS app-platform) - `research/260613-foswiki-deep-dive/findings.md` (`Foswiki::Store` versioned store interface = adapter-contract prior art) -- Use cases: `spec/UseCaseCatalog.md` § A (UC-26–UC-43; UC-34–43 are the - engine-attachment / adapter-contract cases) +- Conceptual + modern-tool dives (T16 + scope extensions, 2026-06-14): + - `research/260614-xanadu-deep-dive/findings.md` (reference-not-copy, transclusion, + tumbler span addressing — ideal) + - `research/260614-zigzag-deep-dive/findings.md` (information space as co-equal + dimensions; clone = transclusion — ideal) + - `research/260614-roam-deep-dive/findings.md` (block-DB, store UUIDs, Datalog, + in-app host) + - `research/260614-obsidian-deep-dive/findings.md` (file-over-app, in-file `^id`, + MetadataCache derived index, dual attach, ecosystem popularity) + - `research/260614-notion-deep-dive/findings.md` (closed SaaS, external-API-only, + DB schema+relations, operational envelope, scoped grant) + - `research/260614-shard-spectrum-synthesis/findings.md` (**synthesis**: the eleven + capability spectra, shard family matrix, UC→task fold-in) +- Use cases: `spec/UseCaseCatalog.md` § A (UC-26–UC-59; UC-34–UC-43 are the + engine-attachment cases, UC-44–UC-59 the conceptual + modern-tool cases) - Aspiration: `INTENT.md` (orchestrator, not engine; mechanism over policy; capability-aware adapters; Markdown-first, backend-neutral) - Related workplan: `SHARD-WP-0001` (page resolution, namespaces, overlays, @@ -74,11 +95,12 @@ decision records only. | Topic | Key tradeoff | Primary UCs | |-------|--------------|-------------| -| Adapter contract & capabilities | Versioned interface + capability vocabulary; write granularity | UC-02, UC-35, UC-38 | -| Structured page model | Typed frontmatter vs sidecar vs blob; bodiless / multi-record pages | UC-34, UC-39 | -| History portability | Supplement (DB-internal) vs import (open file history) | UC-36, UC-41 | -| Adapter binding | Runtime API vs on-disk store; engine-hosted vs external; backend-swap | UC-38, UC-40, UC-43 | -| Syntax translation | Lossless non-Markdown round-trip vs read-only degradation | UC-42, UC-03 | +| Adapter contract & capabilities | Eleven spectra (not flat verbs); write granularity; operational envelope; access grant | UC-02, UC-35, UC-38, UC-57 | +| Structured page model | Typed frontmatter vs sidecar vs blob; bodiless / multi-record; DB schema+relations; non-Markdown assets; query-defined pages | UC-34, UC-39, UC-55, UC-58, UC-54 | +| History portability | Supplement (DB-internal, incl. Notion) vs import (open file history) | UC-36, UC-41 | +| Adapter binding | Three modes (file-store / in-engine-host / external-API); per-binding choice; backend-swap; scoped grant | UC-38, UC-40, UC-43, UC-50, UC-57 | +| Syntax translation | Lossless round-trip vs **lossy-with-fidelity-report** vs read-only degradation | UC-42, UC-59, UC-03 | +| Addressing, identity & navigation (T16) | Span addressing; content identity; transclusion-as-reference; dimensional/query views | UC-44, UC-45, UC-46, UC-47, UC-48, UC-51, UC-52, UC-54 | --- @@ -338,14 +360,27 @@ Define the **capability-aware shard adapter contract** as a *versioned* interfac taking `Foswiki::Store` + `Foswiki::Meta` as direct prior art (a real engine that already separates store backend from core behind a stable interface). -- Capability vocabulary: `read, write, diff, merge, lock, version, publish, notify, +- **Operation verbs:** `read, write, diff, merge, lock, version, publish, notify, transclude-source, translate-syntax, structured-payload` (reconcile with T10). -- **Write granularity** as a first-class capability dimension: per-page / per-file / - per-space / append-only (TiddlyWiki single-file vs DB whole-space vs TWiki per-topic). -- Versioning/compatibility rules for the contract itself; capability discovery. +- **Capability profile = the eleven spectra** (synthesis §2), each a *position* not a + boolean, so federation ops degrade by position: + 1. addressing granularity (none → path → in-file span → store-UUID → portable tumbler) + 2. content identity (none → path/title → fingerprint → span-set) + 3. structure (flat MD → frontmatter → %META% → DB objects → DB schema+relations) + 4. history (none → internal-only → open-file → git-native) + 5. native query (none → text → datalog → DB query) + 6. translation (native → lossless → lossy-with-fidelity-report) + 7. attachment mode (file-store → in-engine-host → external-API) + 8. operational envelope (local/unbounded → rate-limited/eventually-consistent/paginated) + 9. access grant (open → token → OAuth scoped+revocable) + 10. write granularity (whole-file → per-page → per-block) + 11. content types (Markdown-only → typed records → non-Markdown assets) +- Versioning/compatibility rules for the contract itself; capability discovery + (static profile vs runtime negotiation). -**Inputs:** `260608-wikiengines-overview` §3, §5; `260613-foswiki-deep-dive` §2, §6. -**Provides** the vocabulary T10's matrix consumes. **UCs:** UC-02, UC-35, UC-38. +**Inputs:** `260608-wikiengines-overview` §3, §5; `260613-foswiki-deep-dive` §2, §6; +`260614-shard-spectrum-synthesis` §2, §5. **Provides** the vocabulary T10's matrix +consumes. **UCs:** UC-02, UC-35, UC-38, UC-57. **Tradeoffs:** Rich capability set (precise degradation) vs contract complexity; static profile vs runtime capability negotiation. @@ -364,17 +399,32 @@ state_hub_task_id: "7334a4a4-ba75-4fac-a8b4-8350d342b299" Decide how the **backend-neutral, Markdown-first page model carries structured data without flattening**: +The model must stretch **four ways at once** (synthesis §3): prose Markdown, typed +records, non-Markdown assets, and reference/query-defined pages. + - Source shapes: XWiki XObjects against an XClass; TWiki/Foswiki DataForms stored as `%META:FIELD%` in text; Foswiki MetaDataPlugin **multiple records per topic**; - Semantic MediaWiki annotations. + Semantic MediaWiki annotations; **Notion databases** (schema + typed properties + + inter-record **relations** + rollups/formulas) — the apex of database-as-pages. - Representation choice: typed frontmatter vs sidecar file vs opaque provenance blob. + Prefer **in-text structure** (frontmatter/`%META%`, git-diffable); tolerate DB + structure via sidecar + fidelity report (T15). - **Bodiless pages** (UC-39): may a page be purely structured, no Markdown body? - **N typed records per page** (not one form) must be representable. +- **Non-Markdown content types** (UC-55): drawings (Excalidraw), spatial canvases + (JSON Canvas), images, attachments — typed asset vs opaque blob vs content-type + registry, with provenance, not flattened away. +- **Inter-record relations** (UC-58): represented as typed links in the union link + graph, a separate relation index (cf. ZigZag many-to-many), or both (T16). +- **Query-defined / reference pages** (UC-44, UC-54): a page whose canonical form is a + span manifest (Xanadu EDL) or a saved query (Dataview/Notion linked DB) — coordinate + with T16. -**Inputs:** `xwiki` §2.3; `twiki` §2.3; `foswiki` §4. **UCs:** UC-34, UC-39. +**Inputs:** `xwiki` §2.3; `twiki` §2.3; `foswiki` §4; `notion` §2–§3; `obsidian` §3; +`shard-spectrum-synthesis` §3. **UCs:** UC-34, UC-39, UC-55, UC-58, UC-54, UC-44. **Tradeoffs:** Lossless fidelity vs a uniform queryable model; diff/search over -structured fields vs prose. +structured fields vs prose; one coherent model vs four content shapes. --- @@ -390,15 +440,18 @@ state_hub_task_id: "6837862a-8f57-410d-9200-a6a5dcf1a7b9" Define how the **coordination journal** relates to an engine's native history (aligns with and refines T4): +History is a spectrum (synthesis §2): `none → internal-only → open-file → git-native`. + +- **Adopt** — already git-native (Git folder/repo, Obsidian-with-Git): use as-is. - **Supplement** — engine history is DB-internal and non-portable - (Confluence, MediaWiki, XWiki `xwikircs`): the journal begins now / mirrors forward - (UC-36). + (Confluence, MediaWiki, XWiki `xwikircs`, **Notion** internal page history): the + journal begins now / mirrors forward (UC-36). - **Import** — engine history is an open file format (TWiki/yawex RCS `.txt,v`, Foswiki PlainFile timestamped copies): backfill into Git preserving author/timestamp (UC-41). - Authoritative journal vs mirror-reconciled-back; one-time vs continuous. -**Inputs:** `xwiki` §2.4; `twiki` §2.1; `foswiki` §2.2. **UCs:** UC-36, UC-41. +**Inputs:** `xwiki` §2.4; `twiki` §2.1; `foswiki` §2.2; `notion` §4. **UCs:** UC-36, UC-41. **Tradeoffs:** Import fidelity vs effort; duplicated history vs single source of truth. @@ -415,17 +468,31 @@ state_hub_task_id: "f8835969-d118-4738-952a-5e67e5209f3d" Decide **how and where an adapter binds to a backend**: -- **Dual-path attach** — runtime API vs reading the on-disk store directly - (consistency vs offline/fidelity); capability-gate the choice (UC-40). -- **Hosting** — engine-side adapter via native extension API (XWiki components/REST/ - UIX; TWiki/Foswiki plugin handlers + REST + `registerTagHandler`) vs external - adapter (no engine change, lower fidelity) (UC-38). +**Three attachment modes** (synthesis §2, §3), with a backend possibly offering more +than one — so attach mode is a **per-binding, capability-gated choice**, not a fixed +property: + +- **File-store direct** — read the on-disk store as a folder shard (Obsidian vault, + TWiki `data/`, DokuWiki, Git repo): offline, git-native, but risks inconsistent reads + under a running engine (UC-40, UC-53). +- **In-engine hosted adapter** — adapter runs *inside* the engine via its extension API + (XWiki components/REST/UIX; TWiki/Foswiki plugin handlers; **Roam Depot** + `onload/onunload` over `roamAlphaAPI`; **Obsidian plugin** over `App.vault`): high + fidelity, needs deploy access (UC-38, UC-50). +- **External-API only** — attach from outside via the backend's REST API, no in-engine + hosting (**Notion**): full write-through without deploy access, but bounded by the + **operational envelope** (rate limit, eventual consistency, payload caps) and a + **scoped, revocable access grant** that enforces no-silent-mutation (UC-57). +- **Dual attach** — a backend offering several modes (Obsidian: file-store *or* plugin; + TWiki: file *or* API): pick per binding; declare which is authoritative (UC-53). - **Backend-swap tolerance** — shard identity/provenance survives a storage-format change (Foswiki RCS↔PlainFile; folder→Git) (UC-43). -**Inputs:** `twiki` §5, §6; `xwiki` §3; `foswiki` §2. **UCs:** UC-38, UC-40, UC-43. +**Inputs:** `twiki` §5, §6; `xwiki` §3; `foswiki` §2; `roam` §5; `obsidian` §4; +`notion` §4, §6; `shard-spectrum-synthesis` §3. **UCs:** UC-38, UC-40, UC-43, UC-50, UC-53, UC-57. -**Tradeoffs:** Fidelity vs deploy access; consistency vs offline capability. +**Tradeoffs:** Fidelity vs deploy access; consistency vs offline capability; external +write-through vs rate-limit/eventual-consistency ceiling. --- @@ -440,18 +507,71 @@ state_hub_task_id: "22b57b3a-b06b-4ff0-a34a-667a0386bf25" Specify how **non-Markdown shards** participate in the Markdown-first model: +Translation is a spectrum (synthesis §2): `native → lossless → lossy-with-fidelity-report`. + - Read native markup (TWiki/Foswiki TML, XWiki syntax) into the page model and accept Markdown overlays back via **bidirectional, lossless translation** (UC-42). - Feasibility proof: Foswiki **WysiwygPlugin** (TML→HTML for editing, HTML→TML losslessly on save). -- **Graceful degradation:** when lossless translation is unavailable, the shard is a - read-only/projection participant (UC-03), never silently corrupted. +- **Lossy-with-fidelity-report** (UC-59) — for proprietary models that do *not* + round-trip (**Notion** blocks/rich-text/databases; its own Markdown export is lossy): + translate lossily but **make fidelity loss visible** — a per-shard/per-page report of + what projects cleanly vs. degrades, with non-mappable elements preserved as + provenance/sidecar. Fidelity becomes data (union without erasure). Distinct from the + lossless case (UC-42). +- **Graceful degradation:** when neither lossless nor acceptable lossy translation is + available, the shard is a read-only/projection participant (UC-03), never silently + corrupted. - Open: may overlays be stored in Markdown, or must they round-trip in native syntax? -**Inputs:** `foswiki` §5; `twiki` §2.4; `xwiki` §2.5. **UCs:** UC-42, UC-03. +**Inputs:** `foswiki` §5; `twiki` §2.4; `xwiki` §2.5; `notion` §3. **UCs:** UC-42, UC-59, UC-03. **Tradeoffs:** Translation coverage vs read-only floor; Markdown overlays (portable) -vs native-syntax overlays (safe round-trip). +vs native-syntax overlays (safe round-trip); lossy-but-usable vs read-only-but-faithful. + +--- + +## Addressing, content identity, and dimensional / query navigation + +```task +id: SHARD-WP-0002-T16 +status: todo +priority: medium +``` + +New thread from the conceptual (Xanadu, ZigZag) + modern-tool (Roam, Notion, Obsidian/ +Dataview) dives and the synthesis (`260614-shard-spectrum-synthesis` §4). No existing +task owns the addressing/identity/navigation layer that transclusion, overlay, +equivalence, and dimensional views depend on. Refines T8 (transclusion/projection depth) +and T5 (union composition); consumes T11 capabilities. + +- **Span addressing** — a portable sub-page address. Adopt **native span IDs** where the + backend mints them (Roam `:block/uid`, Notion UUID) or carries them in-file (Obsidian + `^id`); fall back to **content fingerprint** or path+range otherwise. Open: wrap native + IDs in a shard-scoped address so they survive projection and don't collide across + shards (Xanadu tumbler is the unreached ideal). (UC-51, UC-44.) +- **Content identity** — detect that two pages are the same / derived content by + fingerprint or span-set overlap, **path-independently** — the equivalence mechanism + UC-27 left open. Enables reverse transclusion (UC-45). (UC-46.) +- **Transclusion as one reference-not-copy primitive** — unify Xanadu transclusion, + ZigZag clone, Roam/Obsidian embed, Notion synced block over the addressable union + (synthesis §3). Compose-by-reference pages (UC-44) and reverse lookup (UC-45) are + views over it. +- **Dimensional / query navigation** — model the union so each relationship (namespace, + genealogy, version, shard, equivalence, links) is a navigable **dimension** (ZigZag), + and a page can be **query-defined** (Dataview/Notion linked DB, UC-54). **Delegate** + view computation to a shard's native query engine where present (Roam Datalog, Notion + DB query, XWiki XWQL); else compute over the projection. (UC-47, UC-48, UC-52, UC-54.) +- **Boundary:** this is a **derived lens over canonical files+journal**, never a new + store (ZigZag dive §6); the many-to-many link graph stays a graph index, not a rank. + +**Inputs:** `xanadu` §2–§5; `zigzag` §2–§6; `roam` §2–§4, §7; `notion` §2; `obsidian` +§2–§3; `shard-spectrum-synthesis` §3–§5. **UCs:** UC-44, UC-45, UC-46, UC-47, UC-48, +UC-51, UC-52, UC-54. + +**Tradeoffs:** Native-ID adoption (cheap, backend-coupled) vs a portable address scheme +(harder, uniform); core navigation API vs internal organizing concept; query delegation +(fast, backend-dependent) vs orchestrator-computed (uniform, costly). --- @@ -460,13 +580,14 @@ vs native-syntax overlays (safe round-trip). - `spec/FederationArchitecture.md` exists with all ten federation topic sections (T1–T10) and an explicit **decisions / deferred / open** table per topic. - The **adapter contract** section of `spec/TechnicalSpecificationDocument.md` exists, - populated by T11–T15, with the capability vocabulary (T11) reconciled against the - T10 federation-ops matrix. + populated by T11–T16, with the capability vocabulary (T11, the **eleven spectra**) + reconciled against the T10 federation-ops matrix. - Each decision honors INTENT: mechanism over policy, union without erasure, overlay before mutation, no silent remote mutation, shard sovereignty, capability-aware adapters, Markdown-first / backend-neutral. -- UC-26–UC-43 are traceable to architecture / adapter-contract sections - (UC-34–UC-43 to T11–T15 specifically). +- UC-26–UC-59 are traceable to architecture / adapter-contract sections + (UC-34–UC-43 to T11–T15; UC-44–UC-59 to T11–T16 per + `research/260614-shard-spectrum-synthesis/findings.md` §4). - Conflicts or dependencies on `SHARD-WP-0001` outputs are listed (e.g. namespace model affects equivalent-page identity; page model T12 affects resolution/overlays). @@ -484,8 +605,9 @@ vs native-syntax overlays (safe round-trip). 5. T7 lifecycle + T8 transclusion (parallel) 6. T9 policy catalog + T10 capability matrix (parallel, finalize doc) -**Adapter contract (T11–T15)** — can start in parallel with T1; T11 first as it +**Adapter contract (T11–T16)** — can start in parallel with T1; T11 first as it frames the rest, and pair it with T10 (shared capability vocabulary): -7. T11 contract & capabilities (frames T12–T15; sync with T10) +7. T11 contract & the eleven capability spectra (frames T12–T16; sync with T10) 8. T12 page model + T13 history portability (parallel; T13 aligns with T4) -9. T14 binding + T15 syntax translation (parallel, finalize adapter-contract spec) \ No newline at end of file +9. T14 binding + T15 syntax translation (parallel) +10. T16 addressing, identity & navigation (refines T8/T5; finalize adapter-contract spec) \ No newline at end of file