coulomb/shard-wiki
Archived
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

synthesis v2: extend shard spectrum to full dive set (13 spectra); re-fold into SHARD-WP-0002

Browse Source 
Extend the cross-dive synthesis from nine systems to the full fourteen
(added Joplin, Logseq, the CRDT cohort Anytype/AFFiNE/AppFlowy, Trilium).
Spectra grown 11 -> 13: new merge-model (none/git/conflict-notes/
native-CRDT) and content-opacity (plaintext/whole-shard-E2EE/per-item),
plus identity-vs-placement emphasis (Trilium note/branch DAG), Logseq's
in-file id:: addressing sweet spot, build-your-own derived query index,
and a six-mode attachment taxonomy (file-store native/interchange-mirror,
in-engine-host, local-REST, external-API, CRDT-replica, P2P). New
through-lines: CRDT changes the merge math; identity != placement;
metadata can be computed; attach surface != native store (+ substrate
migration). UC fold-in extended UC-44-59 -> UC-44-67.

Folded into SHARD-WP-0002: T11 thirteen spectra; T12 computed metadata +
multi-placement/DAG identity; T13 CRDT-log supplement; T14 full six-mode
attachment taxonomy; T15 HTML source model; T16 identity!=placement +
derived index. Context inputs += four dives; acceptance UC-26-67. No new
tasks, no new UCs (synthesis only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-06-14 17:25:32 +02:00
parent 5697669063
commit 714eaf6b5a
3 changed files with 378 additions and 265 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
research/260614-shard-spectrum-synthesis/README.md
View File

@@ -1,47 +1,55 @@
# 260614 — Shard spectrum synthesis (one capability model across nine systems) # 260614 — Shard spectrum synthesis (one capability model across the full dive set)
Date: 2026-06-14 Date: 2026-06-14 · **revised 2026-06-14 (v2)**
## What this is ## What this is
A **synthesis** (no new external research) that reads the nine studied systems *across* A **synthesis** (no new external research) that reads every studied system *across* the
each other and distills them into a single comparative model feeding the **shard adapter others and distills them into a single comparative model feeding the **shard adapter
contract** (`SHARD-WP-0002`). contract** (`SHARD-WP-0002`).
The nine: two conceptual ancestors (**Xanadu**, **ZigZag**), four engines (**XWiki, **v2** extends from nine systems to the full set and grows the spectra from eleven to
TWiki, Foswiki** + the wiki-engines landscape), and three modern tools (**Roam, **thirteen**. The systems: two conceptual ancestors (**Xanadu, ZigZag**), four engines
Obsidian, Notion**), against the federation/origin research. (**XWiki, TWiki, Foswiki** + the wiki-engines landscape), and the modern note/PKB tools
(**Roam, Obsidian, Notion, Joplin, Logseq, Anytype, AFFiNE, AppFlowy, Trilium**), against
the federation/origin research.
Centerpieces: Centerpieces:
- **The shard family matrix** — every candidate backend × {substrate, attach mode, - **The shard family matrix** — every candidate backend × {substrate, attach mode(s),
addressing, structure, history, query, transclusion, →Markdown, write granularity}, addressing, structure, history, **merge**, query, →Markdown, **opacity**}, with Xanadu/
with Xanadu/ZigZag as the ideal anchors. ZigZag as the ideal anchors.
- **The eleven capability spectra** — the claim that the adapter contract should model - **The thirteen capability spectra** — the claim that the adapter contract should model
*positions on spectra* (addressing, content identity, structure, history, query, *positions on spectra*, each anchored at both ends by a real system, with federation
translation, attachment mode, operational envelope, access grant, write granularity, ops degrading by position. v2 adds **content opacity** (12th) and **merge model**
content types), each anchored at both ends by a real system, with federation ops (13th), plus emphasis on **identity vs placement**, and expands the attachment-mode
degrading by position. spectrum (file-store native/interchange-mirror, in-engine-host, local-REST,
- **UC-44–UC-59 → workplan task mapping**, including one genuinely new thread (T16: external-API, CRDT-replica, P2P/no-central-endpoint).
addressing, content identity, dimensional/query navigation). - **UC-44–UC-67 → workplan task mapping** (T11–T16).
## Contents ## Contents
| Path | Role | | Path | Role |
|------|------| |------|------|
| `findings.md` | Family matrix, the eleven spectra, cross-cutting through-lines, UC→task fold-in, recommendations/decisions, escalated open questions | | `findings.md` | Family matrix, the thirteen spectra, cross-cutting through-lines, UC→task fold-in, recommendations/decisions, escalated open questions |
## Status ## Status
Synthesis complete. No new use cases (consolidation only). Feeds `SHARD-WP-0002`: T11 Synthesis v2 complete. No new use cases (consolidation only). Feeds `SHARD-WP-0002`: T11
reframed around the eleven spectra; T12 page-model breadth (prose + typed records + reframed around the **thirteen spectra** (incl. content-opacity + merge-model); T12
non-Markdown assets + query-defined); T13 history confirmed; T14 three-mode attachment page-model breadth (prose + typed/**computed** records + non-Markdown assets +
taxonomy; T15 lossy-with-fidelity-report; **new T16** (addressing, content identity, query-defined + **multi-placement/DAG identity**); T13 history (incl. CRDT-log =
dimensional/query navigation). UC coverage in the workplan extended UC-34–UC-43 → supplement); T14 **full attachment-mode taxonomy** (CRDT-replica + P2P + interchange-mirror
UC-34–UC-59. + local-REST added); T15 lossy-with-fidelity-report (incl. HTML); T16 (addressing, content
identity, **identity≠placement**, derived index, dimensional/query). UC coverage extended
UC-34–UC-59 → **UC-34–UC-67**.
**Through-lines recorded:** files-canonical/index-derived is the winning architecture; **New through-lines (v2):** CRDT changes the merge math (native merge — never impose git
fine-grained addressing is real and adoptable (adopt native, fingerprint otherwise); merge); identity ≠ placement (Trilium note/branch) is the model for multi-location/
transclusion ⇄ clone ⇄ embed is one primitive; structure/history federate iff in text; multi-shard pages; metadata can be computed (inherited/templated), not just stored;
attach mode is a per-binding capability-gated choice; Notion proves the platform can content opacity is per-item, not only whole-shard; the attach surface is not always the
*enforce* no-silent-mutation via scoped grants. native store (Joplin sync mirror; Logseq file-vs-DB and its substrate migration); the
block-graph-on-files sweet spot (Logseq `id::`) resolves the addressing tension.
**Carried from v1:** files-canonical/index-derived; fine-grained addressing is adoptable;
transclusion=clone=embed is one primitive; structure/history federate iff in-text; attach
mode is per-binding; Notion proves the platform can enforce no-silent-mutation.
</content> </content>

356
research/260614-shard-spectrum-synthesis/findings.md
View File

@@ -1,26 +1,28 @@
# Synthesis — the shard spectrum: one capability model across nine systems # Synthesis — the shard spectrum: one capability model across the full dive set
Date: 2026-06-14 Date: 2026-06-14 · **revised 2026-06-14 (v2)** — extended from nine systems to the full
Source kind: **synthesis** — consolidates the engine and tool deep dives plus the two set (added Joplin, Logseq, the CRDT local-first cohort — Anytype/AFFiNE/AppFlowy — and
Nelson conceptual systems into a single comparative model that feeds the **shard adapter Trilium); spectra grown from eleven to **thirteen**.
contract** (`SHARD-WP-0002` T11–T16) Source kind: **synthesis** — consolidates every deep dive into a single comparative model
feeding the **shard adapter contract** (`SHARD-WP-0002` T11–T16)
Lens: shard-wiki — what a backend must expose to participate, expressed as *spectra* of Lens: shard-wiki — what a backend must expose to participate, expressed as *spectra* of
capability rather than a yes/no checklist capability rather than a yes/no checklist
> Purpose. Nine systems have been studied: two conceptual ancestors (**Xanadu**, > Purpose. Fourteen tools/systems have now been studied (plus the wiki-engines
> **ZigZag**), four engines (**XWiki, TWiki, Foswiki**, + the wiki-engines landscape), > landscape): two conceptual ancestors (**Xanadu, ZigZag**); four engines (**XWiki,
> and three modern tools (**Roam, Obsidian, Notion**), against the federation and > TWiki, Foswiki** + landscape); and the modern note/PKB tools (**Roam, Obsidian,
> origin research (federation-concepts, c2, yawex). They were studied one at a time; > Notion, Joplin, Logseq, Anytype, AFFiNE, AppFlowy, Trilium**), against the federation
> this document reads them *across* each other. The payoff is a small set of > and origin research. This document reads them *across* each other. The payoff is a
> **capability spectra** — addressing, structure, history, query, translation, > small set of **capability spectra** — now **thirteen** — each anchored at both ends by
> attachment mode, operational envelope, access grant — each anchored at both ends by a > a real system, with federation operations degrading by position. That spectrum *is*
> real system. That spectrum *is* the adapter contract's design surface. It also folds > the adapter contract's design surface. v2 folds the post-engine use cases (**UC-44–
> the post-engine use cases (**UC-44–UC-59**) into the existing `SHARD-WP-0002` tasks. > UC-67**) into the `SHARD-WP-0002` tasks.
Inputs: `research/260608-federation-concepts`, `research/260608-wikiengines-overview`, Inputs: `research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`,
`research/260613-{xwiki,twiki,foswiki}-deep-dive`, `research/260613-{xwiki,twiki,foswiki}-deep-dive`,
`research/260614-{xanadu,zigzag,roam,obsidian,notion}-deep-dive`. Output target: `research/260614-{xanadu,zigzag,roam,obsidian,notion,joplin,logseq,localfirst-workspaces,trilium}-deep-dive`.
`spec/TechnicalSpecificationDocument.md` (adapter contract) via `SHARD-WP-0002`. Output target: `spec/TechnicalSpecificationDocument.md` (adapter contract) via
`SHARD-WP-0002`.
--- ---
@@ -30,180 +32,221 @@ Candidate-shard backends across the dimensions that matter to the contract. (Xan
ZigZag are *not* shards — they are the conceptual ideals each column aspires to; listed ZigZag are *not* shards — they are the conceptual ideals each column aspires to; listed
last as reference anchors.) last as reference anchors.)
| Backend | Substrate | Attach mode | Addressing | Structure | History | Native query | Transclusion | →Markdown | Write granularity | | Backend | Substrate | Attach mode(s) | Addressing | Structure | History | Merge | Query | →Markdown | Opacity |
|---------|-----------|-------------|-----------|-----------|---------|--------------|-------------|-----------|-------------------| |---------|-----------|----------------|-----------|-----------|---------|-------|-------|-----------|---------|
| **Git folder / repo** | files | file-store | path (+ commit) | flat MD + frontmatter | **git-native** | no | no | native | per-file | | **Git folder / repo** | files | file-store | path (+ commit) | flat MD + frontmatter | **git-native** | git/text | no | native | none |
| **Obsidian vault** | files | file-store **or** in-app plugin | path + **in-file `^id`** | frontmatter (in-file) | none (Git plugin) | plugin (Dataview) | in-file `![[ ]]` | native (OFM) | per-file | | **Obsidian** | files | file-store **/** plugin | path + in-file `^id` | frontmatter (in-file) | none (Git plugin) | git/text | plugin (Dataview) | native (OFM) | none |
| **TWiki** | files + RCS | file-store / API | path | `%META%` in-file | **open file (RCS)** | no | include macro | lossless (TML) | per-topic | | **Logseq** | files (MD/Org) → SQLite (DB-graph) | file-store **/** plugin | **in-file block `id::`** (sweet spot) | `key::` props (in-file) | none (git) | git/text | **Datalog (derived)** | native-ish | none |
| **Foswiki** | pluggable store | file-store / API | path | `%META%` in-file, N records | open file / PlainFile | no | include macro | lossless (TML) | per-topic | | **TWiki / Foswiki** | files + RCS / pluggable | file-store **/** API | path | `%META%` in-file (N records) | **open file (RCS/PlainFile)** | git/text | no | lossless (TML) | none |
| **XWiki** | DB (Hibernate) | in-engine host / REST | path | **XObjects/XClass** | internal (`xwikircs`) | yes (XWQL) | macro | engine syntax | per-page / per-object | | **Trilium** | SQLite (one file) | **ETAPI** / scripting / DB | `noteId` + **`branchId`** (id≠place) | labels+relations, **inherited+templated**; **DAG** | internal revisions | conflict-res | attr search | **lossy (HTML)** | **per-note** |
| **Roam** | client DataScript | **in-app host only** | **store UUID** | `key::` attrs | internal (txn log) | **yes (Datalog)** | **block embed** | (Roam MD) | **per-block** | | **Joplin** | SQLite-local | **sync-mirror** / local-REST / plugin | page-level `:/id` (store) | notebooks + tags | internal revisions | conflict-notes | search | lossy | E2EE (whole) |
| **Notion** | hosted Postgres | **external API only** | **store UUID** | **DB schema + relations + rollups** | internal, not portable | **yes (DB query)** | synced block | **lossy** | **per-block** | | **XWiki** | DB (Hibernate) | in-engine host / REST | path | **XObjects/XClass** | internal (`xwikircs`) | git/text | yes (XWQL) | engine syntax | none |
| **TiddlyWiki** | single file | file-store | path | typed tiddlers | none | no | transclusion | varies | **whole-file** | | **Roam** | client DataScript | **in-app host only** | **store UUID** | `key::` attrs | internal (txn log) | (in-app) | yes (Datalog) | (Roam MD) | none |
| **MediaWiki / Confluence** | DB | API | path | wikitext / macros | internal-only | limited | templates | lossy | per-page | | **Notion** | hosted Postgres | **external API only** | store UUID | **DB schema + relations + rollups** | internal, not portable | internal | yes (DB query) | **lossy** | none |
| — *Xanadu (ideal)* | permascroll | — | **tumbler (span)** | spans + links | permanent | — | **content-identity** | — | span | | **Anytype** | **CRDT (any-sync)** | **replica / P2P node** | object id (store) | **typed object graph (ontology)** | CRDT log | **native-CRDT** | graph | lossy | **E2EE (whole)** |
| — *ZigZag (ideal)* | cells/dims | — | cell id | **N dimensions** | — | dimension walk | clone | — | cell | | **AFFiNE** | **CRDT (Yjs)** | replica / self-host | block id (store) | blocks; **one-data-many-views** | CRDT log | **native-CRDT** | DB filters | lossy | optional |
| **AppFlowy** | **CRDT (Yrs)** | replica / self-host | block id (store) | **Notion-style DB + views** | CRDT log | **native-CRDT** | DB query | lossy | self-host |
| **TiddlyWiki** | single file | file-store | path | typed tiddlers | none | git/text | no | varies | none |
| **MediaWiki / Confluence** | DB | API | path | wikitext / macros | internal-only | internal | limited | lossy | none |
| — *Xanadu (ideal)* | permascroll | — | **tumbler (span)** | spans + links | permanent | content-merge | — | — | — |
| — *ZigZag (ideal)* | cells/dims | — | cell id | **N dimensions** | — | — | dimension walk | — | — |
Reading the matrix top to bottom is reading shard-wiki's difficulty gradient: **Git Reading top to bottom is roughly shard-wiki's difficulty gradient: **Git/Obsidian/
folder → Obsidian** are the friction-free, INTENT-native cases; **TWiki/Foswiki** add Logseq** are friction-free file-store cases (Logseq adding block addressing on files);
syntax translation but stay file/diffable; **XWiki/Roam/Notion** add DB structure, store **TWiki/Foswiki/Trilium** add translation and (Trilium) a DAG + computed metadata;
addressing, and (for Roam/Notion) non-file substrates; **Notion** is the hardest **XWiki/Roam/Notion** add DB structure and store addressing; the **CRDT cohort
(closed, hosted, lossy, rate-limited). Xanadu/ZigZag mark where the *ideals* live. (Anytype/AFFiNE/AppFlowy)** adds a new merge model and (Anytype) P2P+E2EE; **Notion** is
the hardest hosted case. Xanadu/ZigZag mark the ideals.
--- ---
## 2. The capability spectra (the contract's real shape) ## 2. The capability spectra (the contract's real shape) — thirteen
Each capability is **not boolean** — it is a position on a spectrum with a real anchor at Each capability is **not boolean** — it is a position on a spectrum anchored at each end
each end. This is the core synthesis claim: the adapter contract should model *positions*, by a real system. The contract should model *positions*; federation ops degrade by
and federation operations should degrade by position. position.
1. **Addressing granularity** — `none → whole-page(path) → in-file span(Obsidian ^id) → 1. **Addressing granularity** — `none → whole-page(path) → page-level store id(Joplin
store-minted span(Roam/Notion UUID) → portable tumbler(Xanadu ideal)`. (UC-51, UC-44/45.) :/id, Trilium noteId) → in-file span(Obsidian ^id) → in-file block(Logseq id::, the
2. **Content identity** — `none → path/title match → content fingerprint(hash) → sweet spot: block-level AND git-diffable) → store-minted span(Roam/Notion/CRDT UUID) →
span-set/equivalence(Xanadu)`. Drives equivalence + reverse-transclusion. (UC-46, UC-27.) portable tumbler(Xanadu ideal)`. (UC-51, UC-44/45.)
3. **Structure** — `flat Markdown → in-file frontmatter → in-file %META% → DB 2. **Content identity** — `none → path/title → fingerprint(hash) → span-set/equivalence
objects(XWiki) → DB schema+relations+rollups(Notion)`. The git-diffable forms federate (Xanadu)`. Drives equivalence + reverse-transclusion. (UC-46, UC-27.)
easily; DB-locked forms need sidecar + fidelity reporting. (UC-34, UC-39, UC-58.) 3. **Identity vs placement** *(new emphasis)* — `path = identity(most) → identity
4. **History** — `none → internal-only/not-portable(Notion/Confluence) → open file separated from placement(Trilium note/branch; a page in many locations = a DAG)`. The
format(TWiki RCS) → git-native(Git/Obsidian-with-Git)`. Internal ⇒ *supplement*; clean model for a page under multiple paths/shards. (UC-66, UC-22.)
open-file ⇒ *import*; git-native ⇒ *adopt*. (UC-36, UC-41.) 4. **Structure** — `flat Markdown → in-file frontmatter/key::(Obsidian/Logseq) → in-file
5. **Native query** — `none → text search → datalog(Roam) → DB query(Notion/XWiki)`. %META%(TWiki) → typed objects(XWiki) → DB schema+relations+rollups(Notion/AppFlowy) →
Where present, **delegate** derived-view computation; else compute over the projection. typed object graph/ontology(Anytype) → computed: inherited+templated(Trilium)`.
(UC-52, UC-05, UC-54.) In-text federates; DB-locked needs sidecar+fidelity; **computed** needs effective-vs-
6. **Translation to Markdown** — `native → lossless round-trip(Foswiki TML↔HTML) → own provenance. (UC-34, UC-39, UC-58, UC-67.)
lossy-with-fidelity-report(Notion)`. Lossless ⇒ writable; lossy ⇒ read/projection floor 5. **History** — `none → internal-only/not-portable(Notion/Joplin/Trilium) → CRDT update
+ visible fidelity loss. (UC-42, UC-59, UC-03.) log(Anytype/AFFiNE/AppFlowy) → open file format(TWiki RCS) → git-native(Git/Obsidian/
7. **Attachment mode** — `file-store direct(Obsidian/TWiki) → in-engine hosted Logseq)`. Internal/CRDT-log ⇒ *supplement*; open-file ⇒ *import*; git ⇒ *adopt*.
adapter(Roam/XWiki) → external-API only(Notion)`. A backend may offer **more than one** (UC-36, UC-41.)
(Obsidian offers both; TWiki file-or-API). (UC-40, UC-38, UC-50, UC-57.) 6. **Merge model** *(new, 13th)* — `none → git/text merge → conflict-notes/keep-both
8. **Operational envelope** — `local/unbounded → rate-limited+eventually-consistent+ (Joplin) → native-CRDT conflict-free(Anytype/AFFiNE/AppFlowy)`. A CRDT shard merges
paginated(Notion ~3 rps)`. Determines whether projection can be live or must be itself — **never impose git/text merge**; speak the CRDT (replica) or stay projection/
cache/poll/webhook. (UC-57, UC-31.) overlay. (UC-64.)
9. **Access grant** — `open(L0) → token → OAuth scoped + revocable(Notion)`. The backend 7. **Native query** — `none → text search → build-your-own derived index(Logseq DataScript
may *enforce* no-silent-mutation (Notion scoped grants). Ties the authz-in-core ladder. over files; shard-wiki can do likewise) → datalog/graph(Roam/Anytype) → DB query
(UC-57, [[shard-wiki-auth-in-core-decision]].) (Notion/AppFlowy/XWiki)`. Delegate where present; **build an index over the projection**
10. **Write granularity** — `whole-file(TiddlyWiki) → per-page(Git/Obsidian/TWiki) → where not. (UC-52, UC-63, UC-05, UC-54.)
per-block(Roam/Notion)`. Sets overlay/patch/lock/conflict scope. (UC-35.) 8. **Translation to Markdown** — `native → lossless round-trip(Foswiki TML↔HTML) → lossy-
11. **Content types** — `Markdown-only → typed records(XWiki/Notion DB) → non-Markdown with-fidelity-report(HTML/CKEditor Trilium; Notion blocks; CRDT/object models)`.
assets(Excalidraw/Canvas/attachments)`. The page model must stretch to all three. Lossless ⇒ writable; lossy ⇒ read/projection floor + visible fidelity loss. (UC-42,
(UC-55, UC-58.) UC-59, UC-03.)
9. **Attachment mode** *(expanded)* — a **per-binding, capability-gated** choice; a
backend may offer several:
- **file-store** — *native on-disk store*(Obsidian/Logseq/TWiki) **or** *interchange/
sync mirror*(Joplin items on WebDAV/S3) (UC-40, UC-53, UC-60, UC-62)
- **in-engine host** — adapter inside the app via its API (Roam/Obsidian/Logseq/
Trilium scripting, XWiki components) (UC-38, UC-50)
- **local-REST** — localhost API, app-running (Joplin Data API; Trilium ETAPI) (UC-38)
- **external-API** — remote REST from outside (Notion) (UC-57)
- **CRDT replica** — hold a local CRDT replica (Anytype/AFFiNE/AppFlowy) (UC-64)
- **P2P / no-central-endpoint** — replica or peer/node, not a URL (Anytype) (UC-65)
10. **Operational envelope** — `local/unbounded → realtime CRDT/WebSocket → rate-limited+
eventually-consistent+paginated(Notion ~3 rps)`. Sets live vs cache/poll/webhook.
(UC-57, UC-31.)
11. **Access grant** — `open(L0) → token → OAuth scoped+revocable(Notion) → P2P key/invite
(Anytype)`. The backend may *enforce* no-silent-mutation. (UC-57, UC-65,
[[shard-wiki-auth-in-core-decision]].)
12. **Content opacity** *(new, 12th)* — `plaintext → encrypted-at-rest whole-shard(Joplin/
Anytype E2EE) → per-item(Trilium protected notes — encrypted and plaintext coexist)`.
Opaque content ⇒ backup/structure-shell only; never present ciphertext as readable.
(UC-61.)
13. **Write granularity** — `whole-file(TiddlyWiki) → per-page/note(Git/Obsidian/Joplin/
Trilium) → per-block(Roam/Notion/Logseq/CRDT)`. Sets overlay/patch/lock/conflict
scope. (UC-35.)
Design consequence: **T11's capability vocabulary should be these eleven spectra**, not *(Content types — Markdown-only → typed records → non-Markdown assets (Excalidraw/Canvas/
the original flat `read/write/diff/...` list. The flat verbs remain as the *operations*; whiteboards/objects) — remains a cross-cutting page-model demand, tracked under structure
the spectra are the *profile* that says how well each verb is supported and how it + T12 rather than as a standalone capability. UC-55.)*
degrades.
Design consequence: **T11's capability vocabulary = these thirteen spectra**, not a flat
`read/write/diff/...` list. The flat verbs remain the *operations*; the spectra are the
*profile* saying how well each verb is supported and how it degrades.
--- ---
## 3. Cross-cutting findings (the through-lines) ## 3. Cross-cutting findings (the through-lines)
- **Files-canonical, index-derived is the winning architecture.** Obsidian's - **Files-canonical, index-derived is the winning architecture.** Obsidian's
MetadataCache, Git's working tree, and shard-wiki's own projection model all agree: MetadataCache, **Logseq's DataScript-over-files**, Git's working tree, and shard-wiki's
the link graph / backlinks / structured index are **derived and rebuildable**, never a projection model agree: the graph/backlinks/structured index are **derived and
second source of truth. Roam/Notion invert this (DB canonical) and pay for it in rebuildable**, never a second source of truth. Roam/Notion/CRDT invert this (store
portability. shard-wiki keeps files+journal canonical; everything else is projection. canonical) and pay in portability. shard-wiki keeps files+journal canonical.
(Reinforces UC-05, UC-17–20; INTENT.) - **Fine-grained addressing is real, adoptable, and can be git-diffable.** Roam/Notion/
- **Fine-grained addressing is real and adoptable** — Roam (`:block/uid`) and Notion CRDT mint store UUIDs; Obsidian uses in-file `^id`; **Logseq resolves the tension —
(UUID) ship it store-minted; Obsidian ships it in-file (`^id`). The Xanadu tumbler is block-level `id::` that is also git-diffable text**. Adopt native IDs where present,
the unreached ideal (portable, transfinite). shard-wiki should **adopt native span IDs fingerprint where not; portable span address stays the open ideal (Xanadu).
where present, fingerprint where not** (UC-51 + UC-46), and treat a portable span - **Identity ≠ placement.** *(new)* Trilium's **note vs branch** (a note cloned into many
address as the open hard problem (Xanadu §11 Q1). locations = a DAG) is the clean model for a page under multiple paths or across shards —
- **Transclusion ⇄ clone ⇄ embed is one primitive.** Xanadu transclusion, ZigZag clones, separate *what a page is* from *where it sits*. The namespace-level form of the clone/
Roam block embeds, Obsidian `![[ ]]`, Notion synced blocks are the same idea at reference primitive.
different fidelities. A single internal "reference-not-copy" primitive over an - **Transclusion ⇄ clone ⇄ embed ⇄ cloned-note is one primitive.** Xanadu transclusion,
addressable union serves UC-32/44/45/51. (Convergence first noted in the ZigZag dive.) ZigZag clone, Roam/Logseq embed, Obsidian `![[ ]]`, Notion synced block, **Trilium note
- **Structure and history federate iff they live in text.** The sharpest recurring cloning** — one "reference-not-copy" primitive over an addressable union (UC-32/44/45/
engine insight: `%META%`-in-file (TWiki/Foswiki) and frontmatter (Obsidian) diff and 51/66).
travel; XObjects and Notion DBs lock in. The contract should *prefer* in-text - **CRDT changes the merge math.** *(new)* The CRDT cohort merges concurrent edits itself;
structure and *tolerate* DB structure via sidecar + fidelity report. shard-wiki must **not impose git/text merge** — speak the CRDT (replica) or stay a
- **Three attachment modes, and a backend may span them.** The clean taxonomy projection/overlay. A new **merge-model** capability (spectrum 6).
(file-store / in-engine-host / external-API) plus the observation that one backend can - **Structure & history federate iff in text; metadata can be computed.** `%META%`/
offer several (Obsidian: file *or* plugin; TWiki: file *or* API) means **attach mode is frontmatter/`key::` diff and travel; XObjects/Notion-DB/CRDT lock in. *(new)* Trilium
a per-binding choice, capability-gated**, not a fixed property of the backend. adds **computed metadata** (inherited+templated) — represent effective-vs-own, don't
- **Notion proves the platform can enforce INTENT.** Scoped, revocable grants are flatten.
no-silent-mutation made mechanical — evidence the principle is implementable, and a - **Content opacity is per-item, not only whole-shard.** *(new)* Joplin/Anytype encrypt
model for how shard-wiki represents partial/consented visibility. whole-shard; **Trilium encrypts per-note** — the opacity capability must be granular.
- **The page model must stretch four ways at once:** prose Markdown, typed records - **The attach surface is not always the native store.** *(new)* Joplin's best surface is
(with N-per-page and inter-record relations), non-Markdown assets, and its **sync mirror** (not its SQLite); Logseq offers file-graph **or** DB-graph, and is
reference/query-defined pages. This is the heaviest single demand on T12. **migrating substrate** (file→SQLite). Bind to capabilities, not to "it's files."
- **The page model must stretch many ways at once:** prose Markdown, typed/computed
records (N-per-page, relations, inheritance), non-Markdown assets, reference/query-
defined pages, and **multi-placement (DAG) identity**. The heaviest demand on T12.
--- ---
## 4. How the post-engine use cases fold into the workplan ## 4. How the post-engine use cases fold into the workplan
UC-44–UC-59 (from the five 260614 dives) map onto the existing adapter-contract tasks, UC-44–UC-67 (from the 260614 dives) map onto the adapter-contract tasks:
with one genuinely new thread needing a home (T16):
| UCs | Theme | Lands in | | UCs | Theme | Lands in |
|-----|-------|----------| |-----|-------|----------|
| UC-35, UC-50, UC-53, UC-57 | attachment mode + operational envelope + write granularity | **T11** (capabilities) + **T14** (binding) | | UC-35, UC-50, UC-53, UC-57, **UC-60, UC-62, UC-64, UC-65** | attachment modes (file-store native/mirror, in-engine, local-REST, external-API, CRDT-replica, P2P) + operational envelope + merge model | **T11** + **T14** |
| UC-34, UC-39, UC-55, UC-58 | structured/typed payload, non-Markdown assets, DB schema+relations | **T12** (page model) | | UC-34, UC-39, UC-55, UC-58, **UC-67** | structured/typed/**computed** payload, non-MD assets, DB schema+relations | **T12** |
| UC-36 | internal-only history (Notion) = supplement | **T13** (history) | | UC-36 | internal/CRDT-log history = supplement | **T13** |
| UC-42, UC-59 | translation: lossless vs lossy-with-fidelity-report | **T15** (translation) | | UC-42, UC-59 | translation: lossless vs lossy-with-fidelity-report (HTML/blocks/CRDT) | **T15** |
| UC-31 | webhooks / push-vs-poll under eventual consistency | **T6** / **T11** envelope | | UC-31 | webhooks / realtime / push-vs-poll | **T6** / **T11** envelope |
| UC-57 §6 | scoped, revocable grant; no silent mutation | **T11** access-grant + authz decision | | UC-57 §6, **UC-61, UC-65** | scoped grant; **content opacity** (whole/ per-item); P2P key | **T11** (access-grant + opacity) |
| **UC-44, UC-45, UC-46, UC-51** | span addressing, content identity, transclusion-as-reference | **T8** + **new T16** | | UC-44, UC-45, UC-46, UC-51, **UC-63, UC-66** | span addressing, content identity, **identity≠placement**, transclusion-as-reference, derived index | **T16** (+ T12) |
| **UC-47, UC-48, UC-52, UC-54** | dimensional navigation, query delegation, query-defined pages | **T5/T10** + **new T16** | | UC-47, UC-48, UC-52, UC-54 | dimensional navigation, query delegation/build, query-defined pages | **T16** (+ T5/T10) |
**New thread (T16): Addressing, content identity, and dimensional/query navigation.** No No new task needed — **T16** already owns the addressing/identity/navigation thread; v2
existing task owns (a) the portable span-address scheme + content-identity model that adds **identity≠placement (UC-66)** and **build-your-own derived index (UC-63)** to it,
transclusion/overlay/reverse-lookup need, nor (b) the dimensional/query view model and **computed metadata (UC-67)** + **non-MD/multi-placement** to **T12**.
(ZigZag dimensions, Roam/Notion query, Dataview query-defined pages). These are the
Nelson-ideals-meet-modern-tools thread and warrant their own design task.
--- ---
## 5. Recommendations (decisions to record under SHARD-WP-0002) ## 5. Recommendations (decisions to record under SHARD-WP-0002)
1. **Model capabilities as the eleven spectra (§2), not flat verbs.** Keep verbs as 1. **Model capabilities as the thirteen spectra (§2)**, not flat verbs. (T11.)
operations; add the spectra as the profile. (T11.) 2. **Two new spectra: content opacity (per-item) and merge model.** A CRDT shard's merge
2. **Adopt native span IDs; fingerprint otherwise; keep portable-span-address open.** is native — never git-merge it; an encrypted shard degrades to structure-shell. (T11.)
(T16, UC-51/46.) 3. **Adopt native span IDs (prefer in-file/git-diffable, à la Logseq `id::`); fingerprint
3. **One reference-not-copy primitive** for transclusion/clone/embed over the addressable otherwise; keep portable-span-address open.** (T16/T11, UC-51/46.)
union. (T16/T8, UC-32/44/45.) 4. **Separate page identity from placement** (Trilium note/branch) in the page/namespace
4. **Prefer in-text structure; tolerate DB structure via sidecar + fidelity report.** model — a page is one entity with N placements (paths/shards). (T12/T16, UC-66.)
(T12/T15, UC-34/58/59.) 5. **One reference-not-copy primitive** for transclusion/clone/embed/cloned-note over the
5. **Attach mode is a per-binding, capability-gated choice** across the three modes; a addressable union. (T16, UC-32/44/45/66.)
backend may offer several. (T14, UC-40/38/50/57.) 6. **Prefer in-text structure; tolerate DB structure via sidecar+fidelity; represent
6. **Add operational-envelope and access-grant to the profile** as first-class fields. computed (inherited/templated) metadata as effective-vs-own.** (T12/T15, UC-34/58/67.)
(T11, UC-57.) 7. **Attachment mode is a per-binding, capability-gated choice** across the full set:
7. **Delegate derived views to native query engines where present; else compute over the file-store (native *or* interchange-mirror), in-engine-host, local-REST, external-API,
projection.** (T16/T5, UC-52/54.) CRDT-replica, P2P/no-central-endpoint. Bind to capabilities — substrate can migrate
8. **Keep files + coordination journal canonical; all indexes/projections derived and (Logseq). (T14, UC-40/38/50/57/60/62/64/65 + UC-43.)
8. **Query: delegate to a native engine where present; build a derived index over the
projection where not** (Logseq pattern). (T16/T5, UC-52/63/54.)
9. **Keep files + coordination journal canonical; all indexes/projections derived and
rebuildable.** (Architecture invariant, INTENT.) rebuildable.** (Architecture invariant, INTENT.)
These honor INTENT: mechanism over policy (spectra not hard-coded behaviors), union These honor INTENT: mechanism over policy (spectra not hard-coded behaviors), union
without erasure (fidelity reporting makes loss visible), graceful degradation (every without erasure (fidelity + effective-vs-own + identity/placement all preserve
capability has a read/projection floor), no silent mutation (access-grant + overlay), information), graceful degradation (every capability has a read/projection floor; opaque/
shard sovereignty (no backend forced to change substrate), Markdown-first/backend-neutral CRDT shards degrade to backup/projection), no silent mutation (access-grant + opacity +
(in-text preferred, DB tolerated). overlay + respect native merge), shard sovereignty (no backend forced to change
substrate; substrate may migrate), Markdown-first/backend-neutral (in-text preferred, DB/
CRDT/HTML tolerated).
--- ---
## 6. Open questions escalated by the synthesis ## 6. Open questions escalated by the synthesis
1. **Portable span address** — still unsolved across heterogeneous backends (Xanadu's 1. **Portable span address** across heterogeneous backends — wrap native IDs (Roam/Notion/
unbuilt part). Wrap native IDs in a shard-scoped address? (T16.) CRDT UUID, Logseq `id::`, Trilium noteId) in a shard-scoped address? (T16.)
2. **External-API write-through at scale** — is Notion-class write viable under ~3 rps, 2. **CRDT shards** — embed a CRDT client (Yjs/Yrs) for a live replica, or consume
or read/projection/overlay/backup by default? (T14, UC-57.) snapshots? Overlays as CRDT ops or out-of-band patches? (T14, UC-64.)
3. **Inter-record relations in the union** (UC-58) — typed links in the link graph, a 3. **Identity vs placement** — does shard-wiki adopt the note/branch split as its own page
separate relation index (ZigZag many-to-many), or both? (T16/T5.) model, and is a cloned page one page with N placements or a transclusion? (T12/T16,
4. **Page model breadth** — can one model carry prose + typed records + non-Markdown UC-66.)
assets + query-defined pages without becoming incoherent? (T12.) 4. **Computed metadata** — materialize effective attributes (snapshot) or compute live
5. **Dimensional navigation surface** — public API/UI paradigm or internal organizing from the shard's tree/templates, with per-attribute provenance? (T12, UC-67.)
concept? (T16, UC-47/48.) 5. **Page model breadth** — can one model carry prose + typed/computed records + non-MD
assets + query-defined pages + multi-placement identity coherently? (T12.)
6. **External-API write-through at scale** (Notion ~3 rps) vs read/projection default.
(T14, UC-57.)
7. **Content opacity** — what is visible for an opaque shard/item (IDs/structure vs
nothing); does shard-wiki ever hold keys? (T11, UC-61.)
--- ---
## 7. Sources ## 7. Sources
This is a synthesis; primary sources are the nine dives' `findings.md` files plus A synthesis; primary sources are the fourteen dives' `findings.md` files plus
`research/260608-federation-concepts`, `research/260608-wikiengines-overview`, `research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`.
`research/260608-c2-wiki-origins`, `research/260608-yawex-prior-art`. No new external No new external research was performed in v2.
research was performed.
Cross-references: `spec/UseCaseCatalog.md` (UC-26–UC-59), Cross-references: `spec/UseCaseCatalog.md` (UC-26–UC-67),
`workplans/SHARD-WP-0002-federation-architecture.md` (T1–T16), `workplans/SHARD-WP-0002-federation-architecture.md` (T1–T16),
`INTENT.md` (constraints), [[shard-wiki-auth-in-core-decision]]. `INTENT.md` (constraints), [[shard-wiki-auth-in-core-decision]].
@@ -211,10 +254,13 @@ Cross-references: `spec/UseCaseCatalog.md` (UC-26–UC-59),
## 8. Traceability ## 8. Traceability
- Consolidates: nine deep dives + federation/origin research into one capability model. - Consolidates: all fourteen deep dives + federation/origin research into one capability
- Feeds: `SHARD-WP-0002` T11 (eleven-spectra vocabulary), T12 (page-model breadth), T13 model (v2).
(history confirmed), T14 (three-mode attachment taxonomy), T15 (lossy fidelity report), - Feeds: `SHARD-WP-0002` T11 (**thirteen-spectra** vocabulary, incl. content-opacity +
and a **new T16** (addressing, content identity, dimensional/query navigation). merge-model), T12 (page-model breadth: computed metadata, identity≠placement, non-MD),
- UC coverage extended in the workplan from UC-34–UC-43 to **UC-34–UC-59**. T13 (history incl. CRDT-log = supplement), T14 (full attachment-mode taxonomy incl.
CRDT-replica + P2P + interchange-mirror + local-REST), T15 (lossy incl. HTML), T16
(addressing, content identity, identity≠placement, derived index, dimensional/query).
- UC coverage extended in the workplan from UC-34–UC-59 to **UC-34–UC-67**.
- No UCs added (synthesis only); no boundary changes (INTENT Stability Note untouched). - No UCs added (synthesis only); no boundary changes (INTENT Stability Note untouched).
</content> </content>

221
workplans/SHARD-WP-0002-federation-architecture.md
View File

@@ -8,7 +8,7 @@ status: active
owner: tegwick owner: tegwick
topic_slug: whynot topic_slug: whynot
created: "2026-06-08" created: "2026-06-08"
updated: "2026-06-13" updated: "2026-06-14"
depends_on: depends_on:
- SHARD-WP-0001 - SHARD-WP-0001
state_hub_workstream_id: "2af4c46d-cbfd-40ea-a94b-d9e60b0f9945" state_hub_workstream_id: "2af4c46d-cbfd-40ea-a94b-d9e60b0f9945"
@@ -35,12 +35,14 @@ union does*; the adapter contract decides *what a backend must expose to partici
The two are co-dependent (the capability matrix T10 consumes the contract vocabulary The two are co-dependent (the capability matrix T10 consumes the contract vocabulary
T11 defines), so they live in one workplan. T11 defines), so they live in one workplan.
The cross-dive synthesis (`research/260614-shard-spectrum-synthesis/findings.md`) The cross-dive synthesis (`research/260614-shard-spectrum-synthesis/findings.md`, **v2
reframes the contract around **eleven capability spectra** (addressing, content 2026-06-14** — extended to the full dive set incl. Joplin, Logseq, the CRDT cohort
identity, structure, history, native query, translation, attachment mode, operational Anytype/AFFiNE/AppFlowy, and Trilium) reframes the contract around **thirteen capability
envelope, access grant, write granularity, content types) — positions on a spectrum spectra** (addressing, content identity, **identity-vs-placement**, structure, history,
anchored at both ends by a real system, with federation ops degrading by position — **merge model**, native query, translation, attachment mode, operational envelope, access
rather than a flat verb checklist. grant, **content opacity**, write granularity) — positions on a spectrum anchored at both
ends by a real system, with federation ops degrading by position — rather than a flat verb
checklist.
## Context ## Context
@@ -63,10 +65,18 @@ rather than a flat verb checklist.
MetadataCache derived index, dual attach, ecosystem popularity) MetadataCache derived index, dual attach, ecosystem popularity)
- `research/260614-notion-deep-dive/findings.md` (closed SaaS, external-API-only, - `research/260614-notion-deep-dive/findings.md` (closed SaaS, external-API-only,
DB schema+relations, operational envelope, scoped grant) DB schema+relations, operational envelope, scoped grant)
- `research/260614-shard-spectrum-synthesis/findings.md` (**synthesis**: the eleven - `research/260614-joplin-deep-dive/findings.md` (SQLite-local, interchange/sync-mirror
attach, E2EE content opacity, local-REST)
- `research/260614-logseq-deep-dive/findings.md` (block-graph on plain files, in-file
`id::`, derived Datalog index, file→SQLite substrate migration)
- `research/260614-localfirst-workspaces-deep-dive/findings.md` (Anytype/AFFiNE/AppFlowy:
CRDT substrate, native merge, P2P/E2EE, CRDT-replica attach)
- `research/260614-trilium-deep-dive/findings.md` (note cloning → DAG / identity≠placement,
inherited+templated computed metadata, HTML-native, scripting+ETAPI, per-item opacity)
- `research/260614-shard-spectrum-synthesis/findings.md` (**synthesis v2**: the thirteen
capability spectra, shard family matrix, UC→task fold-in) capability spectra, shard family matrix, UC→task fold-in)
- Use cases: `spec/UseCaseCatalog.md` § A (UC-26–UC-59; UC-34–UC-43 are the - Use cases: `spec/UseCaseCatalog.md` § A (UC-26–UC-67; UC-34–UC-43 are the
engine-attachment cases, UC-44–UC-59 the conceptual + modern-tool cases) engine-attachment cases, UC-44–UC-67 the conceptual + modern-tool cases)
- Aspiration: `INTENT.md` (orchestrator, not engine; mechanism over policy; - Aspiration: `INTENT.md` (orchestrator, not engine; mechanism over policy;
capability-aware adapters; Markdown-first, backend-neutral) capability-aware adapters; Markdown-first, backend-neutral)
- Related workplan: `SHARD-WP-0001` (page resolution, namespaces, overlays, - Related workplan: `SHARD-WP-0001` (page resolution, namespaces, overlays,
@@ -91,16 +101,16 @@ decision records only.
| Consensus presets | Spread, merge, designated canonical — policy not core | UC-07, UC-27 | | Consensus presets | Spread, merge, designated canonical — policy not core | UC-07, UC-27 |
| Capability matrix | Which federation ops require which adapter capabilities | UC-02–UC-07 | | Capability matrix | Which federation ops require which adapter capabilities | UC-02–UC-07 |
### Adapter-contract topics (T11–T15, from engine deep dives) ### Adapter-contract topics (T11–T16, from engine + modern-tool dives)
| Topic | Key tradeoff | Primary UCs | | Topic | Key tradeoff | Primary UCs |
|-------|--------------|-------------| |-------|--------------|-------------|
| Adapter contract & capabilities | Eleven spectra (not flat verbs); write granularity; operational envelope; access grant | UC-02, UC-35, UC-38, UC-57 | | Adapter contract & capabilities | **Thirteen spectra** (not flat verbs); write granularity; operational envelope; access grant; **merge model**; **content opacity** | UC-02, UC-35, UC-38, UC-57, UC-61, UC-64 |
| Structured page model | Typed frontmatter vs sidecar vs blob; bodiless / multi-record; DB schema+relations; non-Markdown assets; query-defined pages | UC-34, UC-39, UC-55, UC-58, UC-54 | | Structured page model | Typed frontmatter vs sidecar vs blob; bodiless / multi-record; DB schema+relations; **computed (inherited/templated) metadata**; non-Markdown assets; query-defined pages; **multi-placement (DAG) identity** | UC-34, UC-39, UC-55, UC-58, UC-54, UC-66, UC-67 |
| History portability | Supplement (DB-internal, incl. Notion) vs import (open file history) | UC-36, UC-41 | | History portability | Supplement (DB-internal incl. Notion/Joplin/Trilium; **CRDT-log**) vs import (open file history) | UC-36, UC-41 |
| Adapter binding | Three modes (file-store / in-engine-host / external-API); per-binding choice; backend-swap; scoped grant | UC-38, UC-40, UC-43, UC-50, UC-57 | | Adapter binding | **Full mode taxonomy**: file-store (native *or* interchange/sync-mirror) / in-engine-host / local-REST / external-API / **CRDT-replica** / **P2P-no-central-endpoint**; per-binding choice; backend-swap / substrate-migration | UC-38, UC-40, UC-43, UC-50, UC-57, UC-60, UC-62, UC-64, UC-65 |
| Syntax translation | Lossless round-trip vs **lossy-with-fidelity-report** vs read-only degradation | UC-42, UC-59, UC-03 | | Syntax translation | Lossless round-trip vs **lossy-with-fidelity-report** (HTML/blocks/CRDT) vs read-only degradation | UC-42, UC-59, UC-03 |
| Addressing, identity & navigation (T16) | Span addressing; content identity; transclusion-as-reference; dimensional/query views | UC-44, UC-45, UC-46, UC-47, UC-48, UC-51, UC-52, UC-54 | | Addressing, identity & navigation (T16) | Span addressing; content identity; **identity≠placement**; transclusion-as-reference; **derived query index**; dimensional/query views | UC-44, UC-45, UC-46, UC-47, UC-48, UC-51, UC-52, UC-54, UC-63, UC-66 |
--- ---
@@ -362,25 +372,32 @@ already separates store backend from core behind a stable interface).
- **Operation verbs:** `read, write, diff, merge, lock, version, publish, notify, - **Operation verbs:** `read, write, diff, merge, lock, version, publish, notify,
transclude-source, translate-syntax, structured-payload` (reconcile with T10). transclude-source, translate-syntax, structured-payload` (reconcile with T10).
- **Capability profile = the eleven spectra** (synthesis §2), each a *position* not a - **Capability profile = the thirteen spectra** (synthesis v2 §2), each a *position* not a
boolean, so federation ops degrade by position: boolean, so federation ops degrade by position:
1. addressing granularity (none → path → in-file span → store-UUID → portable tumbler) 1. addressing granularity (none → path → page-level store id → in-file span → in-file
block `id::` → store-UUID → portable tumbler)
2. content identity (none → path/title → fingerprint → span-set) 2. content identity (none → path/title → fingerprint → span-set)
3. structure (flat MD → frontmatter → %META% → DB objects → DB schema+relations) 3. identity vs placement (path=identity → **identity separated from placement**:
4. history (none → internal-only → open-file → git-native) Trilium note/branch; a page in many locations = a DAG)
5. native query (none → text → datalog → DB query) 4. structure (flat MD → frontmatter/`key::` → %META% → typed objects → DB
6. translation (native → lossless → lossy-with-fidelity-report) schema+relations → object-graph/ontology → **computed: inherited+templated**)
7. attachment mode (file-store → in-engine-host → external-API) 5. history (none → internal-only / **CRDT-log** → open-file → git-native)
8. operational envelope (local/unbounded → rate-limited/eventually-consistent/paginated) 6. **merge model** (none → git/text → conflict-notes/keep-both → **native-CRDT**)
9. access grant (open → token → OAuth scoped+revocable) 7. native query (none → text → **build-your-own derived index** → datalog/graph → DB query)
10. write granularity (whole-file → per-page → per-block) 8. translation (native → lossless → lossy-with-fidelity-report; incl. HTML)
11. content types (Markdown-only → typed records → non-Markdown assets) 9. attachment mode (file-store [native *or* interchange/sync-mirror] → in-engine-host →
local-REST → external-API → CRDT-replica → P2P/no-central-endpoint)
10. operational envelope (local/unbounded → realtime CRDT/WebSocket → rate-limited/
eventually-consistent/paginated)
11. access grant (open → token → OAuth scoped+revocable → P2P key/invite)
12. **content opacity** (plaintext → encrypted-at-rest whole-shard → **per-item**)
13. write granularity (whole-file → per-page/note → per-block)
- Versioning/compatibility rules for the contract itself; capability discovery - Versioning/compatibility rules for the contract itself; capability discovery
(static profile vs runtime negotiation). (static profile vs runtime negotiation).
**Inputs:** `260608-wikiengines-overview` §3, §5; `260613-foswiki-deep-dive` §2, §6; **Inputs:** `260608-wikiengines-overview` §3, §5; `260613-foswiki-deep-dive` §2, §6;
`260614-shard-spectrum-synthesis` §2, §5. **Provides** the vocabulary T10's matrix `260614-shard-spectrum-synthesis` §2, §5 (v2). **Provides** the vocabulary T10's matrix
consumes. **UCs:** UC-02, UC-35, UC-38, UC-57. consumes. **UCs:** UC-02, UC-35, UC-38, UC-57, UC-61, UC-64.
**Tradeoffs:** Rich capability set (precise degradation) vs contract complexity; **Tradeoffs:** Rich capability set (precise degradation) vs contract complexity;
static profile vs runtime capability negotiation. static profile vs runtime capability negotiation.
@@ -399,8 +416,9 @@ state_hub_task_id: "7334a4a4-ba75-4fac-a8b4-8350d342b299"
Decide how the **backend-neutral, Markdown-first page model carries structured data Decide how the **backend-neutral, Markdown-first page model carries structured data
without flattening**: without flattening**:
The model must stretch **four ways at once** (synthesis §3): prose Markdown, typed The model must stretch **many ways at once** (synthesis v2 §3): prose Markdown, typed
records, non-Markdown assets, and reference/query-defined pages. *and computed* records, non-Markdown assets, reference/query-defined pages, and
**multi-placement (DAG) identity**.
- Source shapes: XWiki XObjects against an XClass; TWiki/Foswiki DataForms stored as - Source shapes: XWiki XObjects against an XClass; TWiki/Foswiki DataForms stored as
`%META:FIELD%` in text; Foswiki MetaDataPlugin **multiple records per topic**; `%META:FIELD%` in text; Foswiki MetaDataPlugin **multiple records per topic**;
@@ -419,12 +437,22 @@ records, non-Markdown assets, and reference/query-defined pages.
- **Query-defined / reference pages** (UC-44, UC-54): a page whose canonical form is a - **Query-defined / reference pages** (UC-44, UC-54): a page whose canonical form is a
span manifest (Xanadu EDL) or a saved query (Dataview/Notion linked DB) — coordinate span manifest (Xanadu EDL) or a saved query (Dataview/Notion linked DB) — coordinate
with T16. with T16.
- **Computed metadata** (UC-67): Trilium attributes are **inherited (down the tree) +
templated**, so effective metadata = own + inherited + templated. Represent
**effective-vs-own with per-attribute provenance** (own / inherited-from / template);
decide materialize (snapshot) vs compute-live.
- **Multi-placement / DAG identity** (UC-66): Trilium **note (identity) vs branch
(placement)** — a page may occupy several locations at once. Separate **page identity
from placement** (one entity, N placements/paths/shards); no single canonical path —
coordinate with T16 (the namespace-level clone/reference primitive).
**Inputs:** `xwiki` §2.3; `twiki` §2.3; `foswiki` §4; `notion` §2–§3; `obsidian` §3; **Inputs:** `xwiki` §2.3; `twiki` §2.3; `foswiki` §4; `notion` §2–§3; `obsidian` §3;
`shard-spectrum-synthesis` §3. **UCs:** UC-34, UC-39, UC-55, UC-58, UC-54, UC-44. `trilium` §2–§3; `localfirst-workspaces` §1–§3; `shard-spectrum-synthesis` §3 (v2).
**UCs:** UC-34, UC-39, UC-55, UC-58, UC-54, UC-44, UC-66, UC-67.
**Tradeoffs:** Lossless fidelity vs a uniform queryable model; diff/search over **Tradeoffs:** Lossless fidelity vs a uniform queryable model; diff/search over
structured fields vs prose; one coherent model vs four content shapes. structured fields vs prose; one coherent model vs many content shapes; effective-vs-own
metadata snapshot vs live computation.
--- ---
@@ -440,18 +468,22 @@ state_hub_task_id: "6837862a-8f57-410d-9200-a6a5dcf1a7b9"
Define how the **coordination journal** relates to an engine's native history (aligns Define how the **coordination journal** relates to an engine's native history (aligns
with and refines T4): with and refines T4):
History is a spectrum (synthesis §2): `none → internal-only → open-file → git-native`. History is a spectrum (synthesis v2 §2): `none → internal-only / CRDT-log → open-file →
git-native`.
- **Adopt** — already git-native (Git folder/repo, Obsidian-with-Git): use as-is. - **Adopt** — already git-native (Git folder/repo, Obsidian-with-Git, Logseq files): use
- **Supplement** — engine history is DB-internal and non-portable as-is.
(Confluence, MediaWiki, XWiki `xwikircs`, **Notion** internal page history): the - **Supplement** — engine history is non-portable: DB-internal (Confluence, MediaWiki,
journal begins now / mirrors forward (UC-36). XWiki `xwikircs`, **Notion**, **Joplin/Trilium** revisions) or a **CRDT update log**
(Anytype/AFFiNE/AppFlowy): the journal begins now / mirrors forward / snapshots the
replica (UC-36).
- **Import** — engine history is an open file format (TWiki/yawex RCS `.txt,v`, - **Import** — engine history is an open file format (TWiki/yawex RCS `.txt,v`,
Foswiki PlainFile timestamped copies): backfill into Git preserving author/timestamp Foswiki PlainFile timestamped copies): backfill into Git preserving author/timestamp
(UC-41). (UC-41).
- Authoritative journal vs mirror-reconciled-back; one-time vs continuous. - Authoritative journal vs mirror-reconciled-back; one-time vs continuous.
**Inputs:** `xwiki` §2.4; `twiki` §2.1; `foswiki` §2.2; `notion` §4. **UCs:** UC-36, UC-41. **Inputs:** `xwiki` §2.4; `twiki` §2.1; `foswiki` §2.2; `notion` §4; `joplin` §1;
`trilium` §1; `localfirst-workspaces` §4. **UCs:** UC-36, UC-41.
**Tradeoffs:** Import fidelity vs effort; duplicated history vs single source of truth. **Tradeoffs:** Import fidelity vs effort; duplicated history vs single source of truth.
@@ -468,31 +500,47 @@ state_hub_task_id: "f8835969-d118-4738-952a-5e67e5209f3d"
Decide **how and where an adapter binds to a backend**: Decide **how and where an adapter binds to a backend**:
**Three attachment modes** (synthesis §2, §3), with a backend possibly offering more **Full attachment-mode taxonomy** (synthesis v2 §2, §3), with a backend possibly
than one — so attach mode is a **per-binding, capability-gated choice**, not a fixed offering more than one — so attach mode is a **per-binding, capability-gated choice**,
property: not a fixed property:
- **File-store direct** — read the on-disk store as a folder shard (Obsidian vault, - **File-store direct** — read the on-disk store as a folder shard, in two sub-kinds:
TWiki `data/`, DokuWiki, Git repo): offline, git-native, but risks inconsistent reads - *native store* — Obsidian vault, **Logseq** `pages/`+`journals/` (with a format
under a running engine (UC-40, UC-53). profile parsing `id::`/`key::`), TWiki `data/`, Git repo: offline, git-native, but
- **In-engine hosted adapter** — adapter runs *inside* the engine via its extension API risks inconsistent reads under a running engine (UC-40, UC-53, UC-62).
(XWiki components/REST/UIX; TWiki/Foswiki plugin handlers; **Roam Depot** - *interchange / sync mirror* — a tool's documented sync representation on a third-party
`onload/onunload` over `roamAlphaAPI`; **Obsidian plugin** over `App.vault`): high target (**Joplin** items on WebDAV/Nextcloud/S3): app-independent, read/projection/
fidelity, needs deploy access (UC-38, UC-50). overlay-mostly (the tool owns the format; never re-sync) (UC-60).
- **External-API only** — attach from outside via the backend's REST API, no in-engine - **In-engine hosted adapter** — adapter runs *inside* the app via its extension API
(XWiki components; TWiki/Foswiki handlers; **Roam Depot** over `roamAlphaAPI`;
**Obsidian plugin** over `App.vault`; **Logseq** plugin; **Trilium** scripting code
notes): high fidelity, needs deploy access (UC-38, UC-50).
- **Local-REST** — a localhost API served by the running app (**Joplin** Data API;
**Trilium** ETAPI): app-must-run, token auth (UC-38, links UC-57).
- **External-API only** — attach from outside via a remote REST API, no in-engine
hosting (**Notion**): full write-through without deploy access, but bounded by the hosting (**Notion**): full write-through without deploy access, but bounded by the
**operational envelope** (rate limit, eventual consistency, payload caps) and a **operational envelope** (rate limit, eventual consistency, payload caps) and a
**scoped, revocable access grant** that enforces no-silent-mutation (UC-57). **scoped, revocable access grant** that enforces no-silent-mutation (UC-57).
- **Dual attach** — a backend offering several modes (Obsidian: file-store *or* plugin; - **CRDT replica** — hold a local CRDT replica (**Anytype/AFFiNE/AppFlowy**: any-sync/
TWiki: file *or* API): pick per binding; declare which is authoritative (UC-53). Yjs/Yrs): the backend merges natively — respect CRDT semantics, don't git-merge; or a
- **Backend-swap tolerance** — shard identity/provenance survives a storage-format self-host sync endpoint (AFFiNE/AppFlowy Cloud) (UC-64).
change (Foswiki RCS↔PlainFile; folder→Git) (UC-43). - **P2P / no-central-endpoint** — bind a replica or a named peer/node, not a URL
(**Anytype** any-sync), content possibly E2EE (UC-65).
- **Dual / multi attach** — a backend offering several modes (Obsidian/Logseq: file *or*
plugin; TWiki: file *or* API; Joplin: mirror *or* Data API): pick per binding; declare
which is authoritative (UC-53).
- **Backend-swap / substrate-migration tolerance** — shard identity/provenance survives a
storage change (Foswiki RCS↔PlainFile; folder→Git; **Logseq file→SQLite DB-graph**) —
bind to capabilities, not to "it's files" (UC-43).
**Inputs:** `twiki` §5, §6; `xwiki` §3; `foswiki` §2; `roam` §5; `obsidian` §4; **Inputs:** `twiki` §5–§6; `xwiki` §3; `foswiki` §2; `roam` §5; `obsidian` §4;
`notion` §4, §6; `shard-spectrum-synthesis` §3. **UCs:** UC-38, UC-40, UC-43, UC-50, UC-53, UC-57. `notion` §4, §6; `joplin` §2, §4; `logseq` §4–§5; `localfirst-workspaces` §4; `trilium`
§5; `shard-spectrum-synthesis` §3 (v2). **UCs:** UC-38, UC-40, UC-43, UC-50, UC-53,
UC-57, UC-60, UC-62, UC-64, UC-65.
**Tradeoffs:** Fidelity vs deploy access; consistency vs offline capability; external **Tradeoffs:** Fidelity vs deploy access; consistency vs offline capability; external
write-through vs rate-limit/eventual-consistency ceiling. write-through vs rate-limit/eventual-consistency ceiling; CRDT-replica liveness vs
snapshot simplicity.
--- ---
@@ -513,18 +561,21 @@ Translation is a spectrum (synthesis §2): `native → lossless → lossy-with-f
Markdown overlays back via **bidirectional, lossless translation** (UC-42). Markdown overlays back via **bidirectional, lossless translation** (UC-42).
- Feasibility proof: Foswiki **WysiwygPlugin** (TML→HTML for editing, HTML→TML - Feasibility proof: Foswiki **WysiwygPlugin** (TML→HTML for editing, HTML→TML
losslessly on save). losslessly on save).
- **Lossy-with-fidelity-report** (UC-59) — for proprietary models that do *not* - **Lossy-with-fidelity-report** (UC-59) — for models that do *not* round-trip: **Notion**
round-trip (**Notion** blocks/rich-text/databases; its own Markdown export is lossy): blocks/rich-text/databases (export is lossy); **Trilium** **HTML** (CKEditor — more
translate lossily but **make fidelity loss visible** — a per-shard/per-page report of tractable than blocks but still lossy); CRDT/object models. Translate lossily but **make
what projects cleanly vs. degrades, with non-mappable elements preserved as fidelity loss visible** — a per-shard/per-page report of what projects cleanly vs.
provenance/sidecar. Fidelity becomes data (union without erasure). Distinct from the degrades, with non-mappable elements preserved as provenance/sidecar. Fidelity becomes
lossless case (UC-42). data (union without erasure). Distinct from the lossless case (UC-42).
- **HTML as a source model** (Trilium) joins TML/Notion-blocks/CRDT in the translation
capability — HTML↔Markdown, lossy-aware.
- **Graceful degradation:** when neither lossless nor acceptable lossy translation is - **Graceful degradation:** when neither lossless nor acceptable lossy translation is
available, the shard is a read-only/projection participant (UC-03), never silently available, the shard is a read-only/projection participant (UC-03), never silently
corrupted. corrupted.
- Open: may overlays be stored in Markdown, or must they round-trip in native syntax? - Open: may overlays be stored in Markdown, or must they round-trip in native syntax?
**Inputs:** `foswiki` §5; `twiki` §2.4; `xwiki` §2.5; `notion` §3. **UCs:** UC-42, UC-59, UC-03. **Inputs:** `foswiki` §5; `twiki` §2.4; `xwiki` §2.5; `notion` §3; `trilium` §4.
**UCs:** UC-42, UC-59, UC-03.
**Tradeoffs:** Translation coverage vs read-only floor; Markdown overlays (portable) **Tradeoffs:** Translation coverage vs read-only floor; Markdown overlays (portable)
vs native-syntax overlays (safe round-trip); lossy-but-usable vs read-only-but-faithful. vs native-syntax overlays (safe round-trip); lossy-but-usable vs read-only-but-faithful.
@@ -537,6 +588,7 @@ vs native-syntax overlays (safe round-trip); lossy-but-usable vs read-only-but-f
id: SHARD-WP-0002-T16 id: SHARD-WP-0002-T16
status: todo status: todo
priority: medium priority: medium
state_hub_task_id: "b00ca669-59d6-454a-8d6e-f34694e35192"
``` ```
New thread from the conceptual (Xanadu, ZigZag) + modern-tool (Roam, Notion, Obsidian/ New thread from the conceptual (Xanadu, ZigZag) + modern-tool (Roam, Notion, Obsidian/
@@ -546,28 +598,35 @@ equivalence, and dimensional views depend on. Refines T8 (transclusion/projectio
and T5 (union composition); consumes T11 capabilities. and T5 (union composition); consumes T11 capabilities.
- **Span addressing** — a portable sub-page address. Adopt **native span IDs** where the - **Span addressing** — a portable sub-page address. Adopt **native span IDs** where the
backend mints them (Roam `:block/uid`, Notion UUID) or carries them in-file (Obsidian backend mints them (Roam `:block/uid`, Notion/CRDT UUID, Joplin/Trilium page-level id)
`^id`); fall back to **content fingerprint** or path+range otherwise. Open: wrap native or carries them in-file (Obsidian `^id`, **Logseq `id::` — block-level AND git-diffable,
IDs in a shard-scoped address so they survive projection and don't collide across the sweet spot**); fall back to **content fingerprint** or path+range otherwise. Open:
shards (Xanadu tumbler is the unreached ideal). (UC-51, UC-44.) wrap native IDs in a shard-scoped address so they survive projection and don't collide
across shards (Xanadu tumbler is the unreached ideal). (UC-51, UC-44.)
- **Content identity** — detect that two pages are the same / derived content by - **Content identity** — detect that two pages are the same / derived content by
fingerprint or span-set overlap, **path-independently** — the equivalence mechanism fingerprint or span-set overlap, **path-independently** — the equivalence mechanism
UC-27 left open. Enables reverse transclusion (UC-45). (UC-46.) UC-27 left open. Enables reverse transclusion (UC-45). (UC-46.)
- **Identity vs placement** — separate **page identity from placement** (Trilium **note vs
branch**: a page cloned into many locations = a DAG). One entity, N placements (paths/
shards); no single canonical path — the namespace-level form of the clone/reference
primitive. (UC-66, UC-22.)
- **Transclusion as one reference-not-copy primitive** — unify Xanadu transclusion, - **Transclusion as one reference-not-copy primitive** — unify Xanadu transclusion,
ZigZag clone, Roam/Obsidian embed, Notion synced block over the addressable union ZigZag clone, Roam/Obsidian/Logseq embed, Notion synced block, **Trilium note cloning**
(synthesis §3). Compose-by-reference pages (UC-44) and reverse lookup (UC-45) are over the addressable union (synthesis v2 §3). Compose-by-reference pages (UC-44) and
views over it. reverse lookup (UC-45) are views over it.
- **Dimensional / query navigation** — model the union so each relationship (namespace, - **Dimensional / query navigation** — model the union so each relationship (namespace,
genealogy, version, shard, equivalence, links) is a navigable **dimension** (ZigZag), genealogy, version, shard, equivalence, links) is a navigable **dimension** (ZigZag),
and a page can be **query-defined** (Dataview/Notion linked DB, UC-54). **Delegate** and a page can be **query-defined** (Dataview/Notion linked DB, UC-54). **Delegate**
view computation to a shard's native query engine where present (Roam Datalog, Notion view computation to a shard's native query engine where present (Roam/Logseq Datalog,
DB query, XWiki XWQL); else compute over the projection. (UC-47, UC-48, UC-52, UC-54.) Notion/AppFlowy DB query, XWiki XWQL); **else build a derived index over the
projection** (Logseq DataScript-over-files pattern, UC-63). (UC-47, UC-48, UC-52,
UC-54, UC-63.)
- **Boundary:** this is a **derived lens over canonical files+journal**, never a new - **Boundary:** this is a **derived lens over canonical files+journal**, never a new
store (ZigZag dive §6); the many-to-many link graph stays a graph index, not a rank. store (ZigZag dive §6); the many-to-many link graph stays a graph index, not a rank.
**Inputs:** `xanadu` §2–§5; `zigzag` §2–§6; `roam` §2–§4, §7; `notion` §2; `obsidian` **Inputs:** `xanadu` §2–§5; `zigzag` §2–§6; `roam` §2–§4, §7; `notion` §2; `obsidian`
§2–§3; `shard-spectrum-synthesis` §3–§5. **UCs:** UC-44, UC-45, UC-46, UC-47, UC-48, §2–§3; `logseq` §1–§3; `trilium` §2–§3; `shard-spectrum-synthesis` §3–§5 (v2).
UC-51, UC-52, UC-54. **UCs:** UC-44, UC-45, UC-46, UC-47, UC-48, UC-51, UC-52, UC-54, UC-63, UC-66.
**Tradeoffs:** Native-ID adoption (cheap, backend-coupled) vs a portable address scheme **Tradeoffs:** Native-ID adoption (cheap, backend-coupled) vs a portable address scheme
(harder, uniform); core navigation API vs internal organizing concept; query delegation (harder, uniform); core navigation API vs internal organizing concept; query delegation
@@ -580,14 +639,14 @@ UC-51, UC-52, UC-54.
- `spec/FederationArchitecture.md` exists with all ten federation topic sections - `spec/FederationArchitecture.md` exists with all ten federation topic sections
(T1–T10) and an explicit **decisions / deferred / open** table per topic. (T1–T10) and an explicit **decisions / deferred / open** table per topic.
- The **adapter contract** section of `spec/TechnicalSpecificationDocument.md` exists, - The **adapter contract** section of `spec/TechnicalSpecificationDocument.md` exists,
populated by T11–T16, with the capability vocabulary (T11, the **eleven spectra**) populated by T11–T16, with the capability vocabulary (T11, the **thirteen spectra**)
reconciled against the T10 federation-ops matrix. reconciled against the T10 federation-ops matrix.
- Each decision honors INTENT: mechanism over policy, union without erasure, - Each decision honors INTENT: mechanism over policy, union without erasure,
overlay before mutation, no silent remote mutation, shard sovereignty, overlay before mutation, no silent remote mutation, shard sovereignty,
capability-aware adapters, Markdown-first / backend-neutral. capability-aware adapters, Markdown-first / backend-neutral.
- UC-26–UC-59 are traceable to architecture / adapter-contract sections - UC-26–UC-67 are traceable to architecture / adapter-contract sections
(UC-34–UC-43 to T11–T15; UC-44–UC-59 to T11–T16 per (UC-34–UC-43 to T11–T15; UC-44–UC-67 to T11–T16 per
`research/260614-shard-spectrum-synthesis/findings.md` §4). `research/260614-shard-spectrum-synthesis/findings.md` §4, v2).
- Conflicts or dependencies on `SHARD-WP-0001` outputs are listed (e.g. - Conflicts or dependencies on `SHARD-WP-0001` outputs are listed (e.g.
namespace model affects equivalent-page identity; page model T12 affects namespace model affects equivalent-page identity; page model T12 affects
resolution/overlays). resolution/overlays).
@@ -607,7 +666,7 @@ UC-51, UC-52, UC-54.
**Adapter contract (T11–T16)** — can start in parallel with T1; T11 first as it **Adapter contract (T11–T16)** — can start in parallel with T1; T11 first as it
frames the rest, and pair it with T10 (shared capability vocabulary): frames the rest, and pair it with T10 (shared capability vocabulary):
7. T11 contract & the eleven capability spectra (frames T12–T16; sync with T10) 7. T11 contract & the thirteen capability spectra (frames T12–T16; sync with T10)
8. T12 page model + T13 history portability (parallel; T13 aligns with T4) 8. T12 page model + T13 history portability (parallel; T13 aligns with T4)
9. T14 binding + T15 syntax translation (parallel) 9. T14 binding + T15 syntax translation (parallel)
10. T16 addressing, identity & navigation (refines T8/T5; finalize adapter-contract spec) 10. T16 addressing, identity & navigation (refines T8/T5; finalize adapter-contract spec)