shard-wiki/research/260614-roam-deep-dive/findings.md

Findings — Roam Research: the block graph as a queryable database, and its extension API

Date: 2026-06-14 Source kind: modern shipped product — a block-graph note tool; a candidate shard backend (DB-backed / API-attached, like XWiki), and the place where several Nelson ideas actually shipped Lens: shard-wiki — fine-grained addressing, transclusion, bidirectional links, a structured/queryable page space, and the engine-hosts-adapter extension path

Why Roam, and why now. The previous two dives were Ted Nelson's unbuilt ideals — Xanadu (research/260614-xanadu-deep-dive/: reference-not-copy, transclusion, stable fine-grained addresses) and ZigZag (research/260614-zigzag-deep-dive/: an information space as many co-equal dimensions over a queryable structure). Roam Research is where a large slice of both actually shipped to a mainstream audience. Every block has a stable address; block references and embeds are working transclusion; the whole graph is a queryable database; links are bidirectional. So Roam is the natural modern bookend: it lets us check shard-wiki's open questions (portable span addressing, transclusion, structured pages) against a system that solved them in production — and study its extension architecture as a concrete adapter-host surface.

This dive treats Roam as a candidate shard (capability profile in §6) and as evidence on the open questions the Nelson dives left (§7), not as something to copy wholesale — Roam is block-first and DB-backed, shard-wiki is Markdown-page-first and Git-backed (divergences in §8.2).

---

1. What Roam is, in architecture terms

Roam Research is a networked-thought outliner whose entire graph is a client-side DataScript database (DataScript = a ClojureScript reimplementation of Datomic's data model and Datalog query engine). The store is not files; it is a set of immutable datoms.

• A datom is one fact: [entity, attribute, value, transaction] — the EAV(T) model. The graph is the accumulation of datoms; edits are transactions.
• Synchronization and Roam's deep undo fall out of the transaction log — a parallel worth noting to Git's immutable commit history (but it is not Git; see §8.2).

This makes Roam a structured, queryable wiki-shaped store — the same family as XWiki (objects + DB) rather than TWiki/Foswiki (flat files), which matters for how an adapter attaches (§6).

---

2. The block model — everything is a block, every block is addressable

Every paragraph and every page is a block (an entity). Key attributes:

Attribute	Meaning
:block/uid	nine-character public ID — the block's stable, referenceable address (e.g. ((GGv3cyL6Y)))
:block/string	the block's text content
:block/order	position among siblings
:block/children	immediate child entity-ids
:block/parents	full ancestor chain (incl. the page)
:block/refs	edges to blocks/pages this block references
:block/page	the owning page entity
:create/time, :edit/time, :create/email, :edit/email	provenance metadata
:node/title	page-only — the page's title (this attribute is what distinguishes a page from an ordinary block)

Structurally the DB is a forest: each page is a tree of nested blocks; a "page" is just the top-level block that carries :node/title. The single most shard-wiki-relevant fact: :block/uid gives every block — i.e. every sub-page span — a stable, first-class, public address. This is the shipped form of the Xanadu tumbler / the shard-wiki "portable span address" open question (§7).

---

3. References, transclusion, bidirectional links

• Page references [[Page]] and #tags, and block references ((uid)), all create :block/refs edges. The link graph is therefore queryable, not parsed-on-read.
• Block embeds ({{embed: ((uid))}}) are working transclusion: a block's content rendered live in another location by reference, not copied — when the source changes, the embed reflects it. This is Xanadu transclusion / shard-wiki UC-32/UC-44, shipped at block granularity.
• Bidirectional links come in two flavours, exactly shard-wiki's BackLinks problem:
  • Linked references — explicit :block/refs edges (queryable backlinks).
  • Unlinked references — bare text mentions with no edge, surfaced by text search.
• Attributes key:: value are a convention on top of blocks/refs that turns a page into a lightweight record — Roam's answer to structured/typed pages (UC-34/UC-39), queryable via Datalog.

---

4. Querying — Datalog over the graph

Roam exposes Datalog (:find / :where) plus pull expressions:

[:find ?uid :where [?b :block/string ?s] [?b :block/uid ?uid] ...]

This means Roam's "derived views" (a page's linked references, a {{query}} block, a table) are just queries over the datom graph — not bespoke features. For shard-wiki this is the strongest evidence yet that derived views = queries over a structured space (the ZigZag "dimensions + rasters" insight, made executable), and that an adapter could delegate view computation to a shard's native query engine instead of scanning (UC-52).

---

5. Extension architecture — Roam Depot and window.roamAlphaAPI

Roam's official extension system (Roam Depot) is a concrete model of the engine-hosts-adapter path (UC-38):

• Distribution: an extension is a GitHub repo (README.md, extension.js, optional extension.css / CHANGELOG.md / build.sh), cataloged in the roam-depot repo via a metadata JSON (name, author, tags, source_repo, source_commit, optional stripe_account for paid extensions).
• Lifecycle: extension.js default-exports a map with onload and onunload functions; everything created on load must be torn down on unload. An extensionAPI object provides a settings panel and settings.get/set.
• Data API — window.roamAlphaAPI (the surface an adapter would target):
  • Read: q (Datalog query → entity ids), pull / pull-many (fetch entity attributes), full graph traversal.
  • Write: data.block.create / update / move / delete, data.page.create.
  • UI: command palette, blocks, settings.

Crucially, this API runs inside the Roam client (global window.roamAlphaAPI). There is no first-class external REST write API; programmatic external access uses graph export (EDN / JSON / Markdown) or unofficial private APIs. So a live, write-through Roam adapter must be hosted inside Roam as a Depot extension — which is exactly the UC-38 engine-side direction (cf. XWiki components, TWiki plugin handlers).

---

6. Roam as a shard — capability profile

Capability	Roam	Notes for the adapter contract
Read	yes	q/pull live (in-app) or EDN/JSON/Markdown export (snapshot)
Write	yes, in-app	block.create/update/move/delete, page.create — but only via the in-client API → needs a Depot-hosted adapter (UC-38) or unofficial private API
Write granularity	block-level (fine)	the opposite extreme from TiddlyWiki's whole-file writes — sharpens UC-35
Identity / addressing	block :block/uid + page :node/title	shipped sub-page stable addressing (UC-51)
Transclusion	yes	block embeds by uid (UC-32/44/45)
Backlinks	yes	linked (:block/refs) + unlinked (text) (UC-05/18)
Structured data	yes	attributes + Datalog (UC-34/39)
Native query	yes (Datalog)	adapter could delegate views (UC-52)
Diff / merge	no native	transaction log exists but not exposed as page diffs
Version history	hosted, internal-only	not portable Git history → needs supplement/import (UC-36, like Confluence/MediaWiki)
Lock	no	—
Syntax	Roam-flavored Markdown	not CommonMark; block-outline structure → translation/mapping needed (cf. UC-42)

Verdict: Roam is a legitimate DB-backed, API-attached shard — the XWiki family, not the file-store family. Its standout offerings are block-level addressing and shipped transclusion; its costs are block↔page impedance and no portable history.

---

7. Evidence on shard-wiki's open questions (the payoff)

The Nelson dives left open questions; Roam answers several empirically:

1. "What is a portable fine-grained span address?" (Xanadu §11 Q1, the tumbler problem). Roam's answer: a short opaque per-block UID, minted by the store, public and referenceable. shard-wiki lesson — fine-grained addressing is tractable when the backend mints stable block/span IDs; the adapter should adopt native block IDs as span addresses where they exist (UC-51), and fall back to content-fingerprint or path+range where they do not (cf. Xanadu content-identity, UC-46).
2. "Does transclusion belong in core or adapter/UI?" (catalog Q6). Roam shows transclusion is cheap and natural when the store is addressable and queryable; it lives at the data layer (refs/embeds), surfaced by UI. Argues for transclusion as a core capability over an addressable union, not a UI-only trick.
3. "Derived views — core or adapter?" Roam shows them as queries over a structured space. Where a shard exposes a native query engine, delegate (UC-52); where it does not, the orchestrator computes them over the projection.
4. Block↔page mapping is the new question Roam raises: in a block graph, what is a "page"? Roam's own answer (:node/title node = page; nested blocks = spans) is a clean mapping rule for the adapter (UC-50).

---

8. Mapping to shard-wiki INTENT (compare, do not equate)

8.1 Reinforcements

• Fine-grained addressing is achievable (block UIDs) — de-risks UC-44/45/51 and the Xanadu tumbler worry.
• Transclusion + bidirectional links + structured data are not exotic; a shipped tool does all three over one addressable, queryable store — validating shard-wiki's ambitions for UC-32/34/39/05.
• Engine-hosts-adapter (UC-38) has a clean modern template: Depot's onload/onunload + roamAlphaAPI read/write surface.

8.2 Deliberate divergences (design bugs if conflated)

1. Block-DB substrate vs. Markdown-file page model. Roam's "everything is a block in a DataScript DB" must not tempt shard-wiki away from its Markdown-first, backend-neutral page model (INTENT Stability Note). Roam is one shard shape, mapped into the page model, not the model itself.
2. Client-side proprietary DB vs. Git coordination journal. Roam's transaction log is not portable history; shard-wiki keeps Git as the coordination layer. A Roam shard contributes snapshots/projections, and its history needs supplement/import (UC-36), not adoption as canonical.
3. Single graph vs. federation. A Roam graph is one sovereign shard, never the federation layer. Do not model the union as "a big Roam graph."
4. Roam-flavored Markdown + outline structure ≠ CommonMark pages; the adapter owns a lossy-aware translation (block outline ↔ page + headings/lists), cf. UC-42.

8.3 What Roam teaches that shard-wiki should keep

• Mint/adopt stable sub-page IDs. The cheapest path to transclusion, overlay, and reverse-lookup at span granularity is a backend that already addresses spans — lean on it (UC-51), degrade gracefully otherwise.
• Treat a structured shard's query engine as a capability, and delegate derived views to it when present (UC-52).
• A clean block↔page rule (:node/title node = page) keeps a block backend usable without flattening (UC-50, complementing UC-34's no-lossy-flatten rule).

---

9. Use-case seeds → catalog (promoted 2026-06-14)

Last existing UC is UC-49. New UCs UC-50–UC-52 added; existing UCs enriched.

Seed	Catalog action
Attach a block-graph DB wiki (Roam-style) as a shard via its query/CRUD API, mapping blocks to the page model (:node/title node = page; nested blocks = spans)	UC-50 (new)
Adopt a shard's native block/span IDs as portable span addresses for transclusion/overlay (the shipped answer to the span-address question)	UC-51 (new)
Delegate derived views to a shard's native query engine (Datalog) where advertised	UC-52 (new)
Shipped transclusion via block embeds + shipped fine-grained addressing	enriches UC-32 (and UC-44/45)
Attributes + Datalog = shipped structured/typed pages	enriches UC-34 (and UC-39)
Roam Depot onload/onunload + roamAlphaAPI = modern engine-hosts-adapter template	enriches UC-38
Block-level write granularity = the fine extreme opposite TiddlyWiki	enriches UC-35
Hosted history is internal-only, not portable Git	links UC-36

---

10. Architecture notes for SHARD-WP-0002 (no UC)

• The adapter contract should model "native span/block IDs" as a capability: a shard advertises whether it mints stable sub-page addresses; transclusion/overlay/reverse- lookup capabilities key off it (ties UC-44/45/46/51).
• The contract should model a "native query" capability so the orchestrator can delegate derived-view computation (UC-52) vs. scanning the projection.
• A block↔page mapping belongs in the adapter, with the no-lossy-flatten rule of UC-34: page = titled node, blocks = addressable spans, attributes = sidecar metadata.
• Roam joins XWiki as a DB-backed / API-attached exemplar (vs. TWiki/Foswiki file stores) and as a second engine-hosts-adapter template (vs. XWiki components / TWiki handlers) for the T14 adapter-binding task.

---

11. Open questions (for spec / workplans)

1. When a backend mints native span IDs (Roam UIDs), are they used directly as the shard-wiki span address, or wrapped in a shard-scoped address so they survive projection/overlay and don't collide across shards?
2. For a write-through Roam shard the adapter must run inside Roam (Depot extension). Is in-engine hosting an accepted adapter deployment mode generally, or do we restrict Roam to read/projection/overlay-target (graceful degradation) when we cannot deploy inside it?
3. How lossy is the block-outline ↔ Markdown-page mapping, and is round-trip fidelity required or best-effort (cf. UC-42 Foswiki TML↔HTML)?
4. Do we consume Roam's export (EDN/JSON) as the projection source, or the live roamAlphaAPI? Snapshot vs. live freshness trade-off (cf. UC-31).

---

12. Sources

Source	Used for
zsolt.blog — "Deep Dive Into Roam's Data Structure" (https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html)	Datom/EAV model, block attributes (:block/uid etc.), refs/embeds, linked vs unlinked references, Datalog/pull
Roam Research Datalog Cheatsheet (https://gist.github.com/2b3pro/231e4f230ed41e3f52e8a89ebf49848b)	Attribute names, :q/:pull query shapes
GitHub — Roam-Research/roam-depot (https://github.com/Roam-Research/roam-depot)	Extension repo/manifest structure, extension.js onload/onunload, extensionAPI settings, roamAlphaAPI read/write/ui surface
"Introduction to the Roam Alpha API" (https://www.putyourleftfoot.in/introduction-to-the-roam-alpha-api)	window.roamAlphaAPI q/pull semantics, global-scope dependency
David Vargas — "How To Create Your Own Roam Extensions" (https://davidvargas.me/blog/how_to_create_your_own_roam_extensions)	Extension authoring lifecycle, write methods
Wikipedia / general — Roam Research, DataScript, Datomic	Datalog/Datomic lineage, client-side DB framing

Cross-references: research/260614-xanadu-deep-dive/findings.md (span addressing, transclusion), research/260614-zigzag-deep-dive/findings.md (queryable dimensions), research/260613-xwiki-deep-dive/findings.md (DB-backed engine + extension host), spec/UseCaseCatalog.md (UC-05, UC-18, UC-32, UC-34/35/36/38/39, UC-44/45/46), workplans/SHARD-WP-0002-federation-architecture.md (T14 adapter binding).

---

13. Traceability

• New UCs: UC-50, UC-51, UC-52 → spec/UseCaseCatalog.md.
• Enriched UCs: UC-32, UC-34, UC-35, UC-38 (and links UC-36, UC-39, UC-44/45/46).
• Architecture (no UC): native-span-ID capability; native-query capability; block↔page mapping; Roam as DB-backed/API-attached + engine-hosts-adapter exemplar → SHARD-WP-0002 (T14).
• Boundary recorded: Roam is one candidate shard, mapped into the Markdown-first page model; not a substrate, not the federation layer (INTENT Stability Note).