shard-wiki/research/260614-shard-spectrum-synthesis/findings.md

Synthesis — the shard spectrum: one capability model across nine systems

Date: 2026-06-14 Source kind: synthesis — consolidates the engine and tool deep dives plus the two Nelson conceptual systems into a single comparative model that feeds the shard adapter contract (SHARD-WP-0002 T11–T16) Lens: shard-wiki — what a backend must expose to participate, expressed as spectra of capability rather than a yes/no checklist

Purpose. Nine systems have been studied: two conceptual ancestors (Xanadu, ZigZag), four engines (XWiki, TWiki, Foswiki, + the wiki-engines landscape), and three modern tools (Roam, Obsidian, Notion), against the federation and origin research (federation-concepts, c2, yawex). They were studied one at a time; this document reads them across each other. The payoff is a small set of capability spectra — addressing, structure, history, query, translation, attachment mode, operational envelope, access grant — each anchored at both ends by a real system. That spectrum is the adapter contract's design surface. It also folds the post-engine use cases (UC-44–UC-59) into the existing SHARD-WP-0002 tasks.

Inputs: research/260608-federation-concepts, research/260608-wikiengines-overview, research/260613-{xwiki,twiki,foswiki}-deep-dive, research/260614-{xanadu,zigzag,roam,obsidian,notion}-deep-dive. Output target: spec/TechnicalSpecificationDocument.md (adapter contract) via SHARD-WP-0002.

---

1. The shard family matrix

Candidate-shard backends across the dimensions that matter to the contract. (Xanadu and ZigZag are not shards — they are the conceptual ideals each column aspires to; listed last as reference anchors.)

Backend	Substrate	Attach mode	Addressing	Structure	History	Native query	Transclusion	→Markdown	Write granularity
Git folder / repo	files	file-store	path (+ commit)	flat MD + frontmatter	git-native	no	no	native	per-file
Obsidian vault	files	file-store or in-app plugin	path + in-file ^id	frontmatter (in-file)	none (Git plugin)	plugin (Dataview)	in-file ![[ ]]	native (OFM)	per-file
TWiki	files + RCS	file-store / API	path	%META% in-file	open file (RCS)	no	include macro	lossless (TML)	per-topic
Foswiki	pluggable store	file-store / API	path	%META% in-file, N records	open file / PlainFile	no	include macro	lossless (TML)	per-topic
XWiki	DB (Hibernate)	in-engine host / REST	path	XObjects/XClass	internal (xwikircs)	yes (XWQL)	macro	engine syntax	per-page / per-object
Roam	client DataScript	in-app host only	store UUID	key:: attrs	internal (txn log)	yes (Datalog)	block embed	(Roam MD)	per-block
Notion	hosted Postgres	external API only	store UUID	DB schema + relations + rollups	internal, not portable	yes (DB query)	synced block	lossy	per-block
TiddlyWiki	single file	file-store	path	typed tiddlers	none	no	transclusion	varies	whole-file
MediaWiki / Confluence	DB	API	path	wikitext / macros	internal-only	limited	templates	lossy	per-page
— Xanadu (ideal)	permascroll	—	tumbler (span)	spans + links	permanent	—	content-identity	—	span
— ZigZag (ideal)	cells/dims	—	cell id	N dimensions	—	dimension walk	clone	—	cell

Reading the matrix top to bottom is reading shard-wiki's difficulty gradient: Git folder → Obsidian are the friction-free, INTENT-native cases; TWiki/Foswiki add syntax translation but stay file/diffable; XWiki/Roam/Notion add DB structure, store addressing, and (for Roam/Notion) non-file substrates; Notion is the hardest (closed, hosted, lossy, rate-limited). Xanadu/ZigZag mark where the ideals live.

---

2. The capability spectra (the contract's real shape)

Each capability is not boolean — it is a position on a spectrum with a real anchor at each end. This is the core synthesis claim: the adapter contract should model positions, and federation operations should degrade by position.

1. Addressing granularity — none → whole-page(path) → in-file span(Obsidian ^id) → store-minted span(Roam/Notion UUID) → portable tumbler(Xanadu ideal). (UC-51, UC-44/45.)
2. Content identity — none → path/title match → content fingerprint(hash) → span-set/equivalence(Xanadu). Drives equivalence + reverse-transclusion. (UC-46, UC-27.)
3. Structure — flat Markdown → in-file frontmatter → in-file %META% → DB objects(XWiki) → DB schema+relations+rollups(Notion). The git-diffable forms federate easily; DB-locked forms need sidecar + fidelity reporting. (UC-34, UC-39, UC-58.)
4. History — none → internal-only/not-portable(Notion/Confluence) → open file format(TWiki RCS) → git-native(Git/Obsidian-with-Git). Internal ⇒ supplement; open-file ⇒ import; git-native ⇒ adopt. (UC-36, UC-41.)
5. Native query — none → text search → datalog(Roam) → DB query(Notion/XWiki). Where present, delegate derived-view computation; else compute over the projection. (UC-52, UC-05, UC-54.)
6. Translation to Markdown — native → lossless round-trip(Foswiki TML↔HTML) → lossy-with-fidelity-report(Notion). Lossless ⇒ writable; lossy ⇒ read/projection floor
   • visible fidelity loss. (UC-42, UC-59, UC-03.)
7. Attachment mode — file-store direct(Obsidian/TWiki) → in-engine hosted adapter(Roam/XWiki) → external-API only(Notion). A backend may offer more than one (Obsidian offers both; TWiki file-or-API). (UC-40, UC-38, UC-50, UC-57.)
8. Operational envelope — local/unbounded → rate-limited+eventually-consistent+ paginated(Notion ~3 rps). Determines whether projection can be live or must be cache/poll/webhook. (UC-57, UC-31.)
9. Access grant — open(L0) → token → OAuth scoped + revocable(Notion). The backend may enforce no-silent-mutation (Notion scoped grants). Ties the authz-in-core ladder. (UC-57, shard-wiki-auth-in-core-decision.)
10. Write granularity — whole-file(TiddlyWiki) → per-page(Git/Obsidian/TWiki) → per-block(Roam/Notion). Sets overlay/patch/lock/conflict scope. (UC-35.)
11. Content types — Markdown-only → typed records(XWiki/Notion DB) → non-Markdown assets(Excalidraw/Canvas/attachments). The page model must stretch to all three. (UC-55, UC-58.)

Design consequence: T11's capability vocabulary should be these eleven spectra, not the original flat read/write/diff/... list. The flat verbs remain as the operations; the spectra are the profile that says how well each verb is supported and how it degrades.

---

3. Cross-cutting findings (the through-lines)

• Files-canonical, index-derived is the winning architecture. Obsidian's MetadataCache, Git's working tree, and shard-wiki's own projection model all agree: the link graph / backlinks / structured index are derived and rebuildable, never a second source of truth. Roam/Notion invert this (DB canonical) and pay for it in portability. shard-wiki keeps files+journal canonical; everything else is projection. (Reinforces UC-05, UC-17–20; INTENT.)
• Fine-grained addressing is real and adoptable — Roam (:block/uid) and Notion (UUID) ship it store-minted; Obsidian ships it in-file (^id). The Xanadu tumbler is the unreached ideal (portable, transfinite). shard-wiki should adopt native span IDs where present, fingerprint where not (UC-51 + UC-46), and treat a portable span address as the open hard problem (Xanadu §11 Q1).
• Transclusion ⇄ clone ⇄ embed is one primitive. Xanadu transclusion, ZigZag clones, Roam block embeds, Obsidian ![[ ]], Notion synced blocks are the same idea at different fidelities. A single internal "reference-not-copy" primitive over an addressable union serves UC-32/44/45/51. (Convergence first noted in the ZigZag dive.)
• Structure and history federate iff they live in text. The sharpest recurring engine insight: %META%-in-file (TWiki/Foswiki) and frontmatter (Obsidian) diff and travel; XObjects and Notion DBs lock in. The contract should prefer in-text structure and tolerate DB structure via sidecar + fidelity report.
• Three attachment modes, and a backend may span them. The clean taxonomy (file-store / in-engine-host / external-API) plus the observation that one backend can offer several (Obsidian: file or plugin; TWiki: file or API) means attach mode is a per-binding choice, capability-gated, not a fixed property of the backend.
• Notion proves the platform can enforce INTENT. Scoped, revocable grants are no-silent-mutation made mechanical — evidence the principle is implementable, and a model for how shard-wiki represents partial/consented visibility.
• The page model must stretch four ways at once: prose Markdown, typed records (with N-per-page and inter-record relations), non-Markdown assets, and reference/query-defined pages. This is the heaviest single demand on T12.

---

4. How the post-engine use cases fold into the workplan

UC-44–UC-59 (from the five 260614 dives) map onto the existing adapter-contract tasks, with one genuinely new thread needing a home (T16):

UCs	Theme	Lands in
UC-35, UC-50, UC-53, UC-57	attachment mode + operational envelope + write granularity	T11 (capabilities) + T14 (binding)
UC-34, UC-39, UC-55, UC-58	structured/typed payload, non-Markdown assets, DB schema+relations	T12 (page model)
UC-36	internal-only history (Notion) = supplement	T13 (history)
UC-42, UC-59	translation: lossless vs lossy-with-fidelity-report	T15 (translation)
UC-31	webhooks / push-vs-poll under eventual consistency	T6 / T11 envelope
UC-57 §6	scoped, revocable grant; no silent mutation	T11 access-grant + authz decision
UC-44, UC-45, UC-46, UC-51	span addressing, content identity, transclusion-as-reference	T8 + new T16
UC-47, UC-48, UC-52, UC-54	dimensional navigation, query delegation, query-defined pages	T5/T10 + new T16

New thread (T16): Addressing, content identity, and dimensional/query navigation. No existing task owns (a) the portable span-address scheme + content-identity model that transclusion/overlay/reverse-lookup need, nor (b) the dimensional/query view model (ZigZag dimensions, Roam/Notion query, Dataview query-defined pages). These are the Nelson-ideals-meet-modern-tools thread and warrant their own design task.

---

5. Recommendations (decisions to record under SHARD-WP-0002)

1. Model capabilities as the eleven spectra (§2), not flat verbs. Keep verbs as operations; add the spectra as the profile. (T11.)
2. Adopt native span IDs; fingerprint otherwise; keep portable-span-address open. (T16, UC-51/46.)
3. One reference-not-copy primitive for transclusion/clone/embed over the addressable union. (T16/T8, UC-32/44/45.)
4. Prefer in-text structure; tolerate DB structure via sidecar + fidelity report. (T12/T15, UC-34/58/59.)
5. Attach mode is a per-binding, capability-gated choice across the three modes; a backend may offer several. (T14, UC-40/38/50/57.)
6. Add operational-envelope and access-grant to the profile as first-class fields. (T11, UC-57.)
7. Delegate derived views to native query engines where present; else compute over the projection. (T16/T5, UC-52/54.)
8. Keep files + coordination journal canonical; all indexes/projections derived and rebuildable. (Architecture invariant, INTENT.)

These honor INTENT: mechanism over policy (spectra not hard-coded behaviors), union without erasure (fidelity reporting makes loss visible), graceful degradation (every capability has a read/projection floor), no silent mutation (access-grant + overlay), shard sovereignty (no backend forced to change substrate), Markdown-first/backend-neutral (in-text preferred, DB tolerated).

---

6. Open questions escalated by the synthesis

1. Portable span address — still unsolved across heterogeneous backends (Xanadu's unbuilt part). Wrap native IDs in a shard-scoped address? (T16.)
2. External-API write-through at scale — is Notion-class write viable under ~3 rps, or read/projection/overlay/backup by default? (T14, UC-57.)
3. Inter-record relations in the union (UC-58) — typed links in the link graph, a separate relation index (ZigZag many-to-many), or both? (T16/T5.)
4. Page model breadth — can one model carry prose + typed records + non-Markdown assets + query-defined pages without becoming incoherent? (T12.)
5. Dimensional navigation surface — public API/UI paradigm or internal organizing concept? (T16, UC-47/48.)

---

7. Sources

This is a synthesis; primary sources are the nine dives' findings.md files plus research/260608-federation-concepts, research/260608-wikiengines-overview, research/260608-c2-wiki-origins, research/260608-yawex-prior-art. No new external research was performed.

Cross-references: spec/UseCaseCatalog.md (UC-26–UC-59), workplans/SHARD-WP-0002-federation-architecture.md (T1–T16), INTENT.md (constraints), shard-wiki-auth-in-core-decision.

---

8. Traceability

• Consolidates: nine deep dives + federation/origin research into one capability model.
• Feeds: SHARD-WP-0002 T11 (eleven-spectra vocabulary), T12 (page-model breadth), T13 (history confirmed), T14 (three-mode attachment taxonomy), T15 (lossy fidelity report), and a new T16 (addressing, content identity, dimensional/query navigation).
• UC coverage extended in the workplan from UC-34–UC-43 to UC-34–UC-59.
• No UCs added (synthesis only); no boundary changes (INTENT Stability Note untouched).