shard-wiki/research/260614-roam-deep-dive/findings.md

# Findings — Roam Research: the block graph as a queryable database, and its extension API

Date: 2026-06-14
Source kind: **modern shipped product** — a block-graph note tool; a *candidate shard
backend* (DB-backed / API-attached, like XWiki), and the place where several Nelson
ideas actually shipped
Lens: shard-wiki — fine-grained addressing, transclusion, bidirectional links, a
structured/queryable page space, and the engine-hosts-adapter extension path

> Why Roam, and why now. The previous two dives were Ted Nelson's *unbuilt* ideals —
> Xanadu (`research/260614-xanadu-deep-dive/`: reference-not-copy, transclusion,
> stable fine-grained addresses) and ZigZag (`research/260614-zigzag-deep-dive/`: an
> information space as many co-equal dimensions over a queryable structure). **Roam
> Research is where a large slice of both actually shipped to a mainstream audience.**
> Every block has a stable address; block references and embeds are working
> transclusion; the whole graph is a queryable database; links are bidirectional. So
> Roam is the natural modern bookend: it lets us check shard-wiki's open questions
> (portable span addressing, transclusion, structured pages) against a system that
> solved them *in production* — and study its **extension architecture** as a concrete
> adapter-host surface.

This dive treats Roam as a *candidate shard* (capability profile in §6) and as
*evidence* on the open questions the Nelson dives left (§7), not as something to copy
wholesale — Roam is block-first and DB-backed, shard-wiki is Markdown-page-first and
Git-backed (divergences in §8.2).

---

## 1. What Roam is, in architecture terms

Roam Research is a networked-thought outliner whose **entire graph is a client-side
DataScript database** (DataScript = a ClojureScript reimplementation of Datomic's
data model and Datalog query engine). The store is not files; it is a set of immutable
**datoms**.

- A **datom** is one fact: `[entity, attribute, value, transaction]` — the EAV(T)
  model. The graph is the accumulation of datoms; edits are transactions.
- Synchronization and Roam's deep undo fall out of the transaction log — a parallel
  worth noting to Git's immutable commit history (but it is *not* Git; see §8.2).

This makes Roam a **structured, queryable** wiki-shaped store — the same family as
XWiki (objects + DB) rather than TWiki/Foswiki (flat files), which matters for how an
adapter attaches (§6).

---

## 2. The block model — everything is a block, every block is addressable

Every paragraph *and* every page is a **block** (an entity). Key attributes:

| Attribute | Meaning |
|-----------|---------|
| `:block/uid` | **nine-character public ID** — the block's stable, referenceable address (e.g. `((GGv3cyL6Y))`) |
| `:block/string` | the block's text content |
| `:block/order` | position among siblings |
| `:block/children` | immediate child entity-ids |
| `:block/parents` | full ancestor chain (incl. the page) |
| `:block/refs` | edges to blocks/pages this block references |
| `:block/page` | the owning page entity |
| `:create/time`, `:edit/time`, `:create/email`, `:edit/email` | provenance metadata |
| `:node/title` | **page-only** — the page's title (this attribute is what distinguishes a page from an ordinary block) |

Structurally the DB is a **forest**: each page is a tree of nested blocks; a "page" is
just the top-level block that carries `:node/title`. The single most shard-wiki-relevant
fact: **`:block/uid` gives every block — i.e. every sub-page span — a stable,
first-class, public address.** This is the *shipped* form of the Xanadu tumbler / the
shard-wiki "portable span address" open question (§7).

---

## 3. References, transclusion, bidirectional links

- **Page references** `[[Page]]` and `#tags`, and **block references** `((uid))`, all
  create `:block/refs` edges. The link graph is therefore queryable, not parsed-on-read.
- **Block embeds** (`{{embed: ((uid))}}`) are **working transclusion**: a block's
  content rendered live in another location by reference, not copied — when the source
  changes, the embed reflects it. This is Xanadu transclusion / shard-wiki UC-32/UC-44,
  shipped at block granularity.
- **Bidirectional links** come in two flavours, exactly shard-wiki's BackLinks problem:
  - **Linked references** — explicit `:block/refs` edges (queryable backlinks).
  - **Unlinked references** — bare text mentions with no edge, surfaced by text search.
- **Attributes** `key:: value` are a convention on top of blocks/refs that turns a page
  into a lightweight record — Roam's answer to structured/typed pages (UC-34/UC-39),
  queryable via Datalog.

---

## 4. Querying — Datalog over the graph

Roam exposes **Datalog** (`:find` / `:where`) plus **pull** expressions:

```
[:find ?uid :where [?b :block/string ?s] [?b :block/uid ?uid] ...]
```

This means Roam's "derived views" (a page's linked references, a `{{query}}` block, a
table) are **just queries over the datom graph** — not bespoke features. For shard-wiki
this is the strongest evidence yet that *derived views = queries over a structured
space* (the ZigZag "dimensions + rasters" insight, made executable), and that an
adapter could **delegate view computation to a shard's native query engine** instead of
scanning (UC-52).

---

## 5. Extension architecture — Roam Depot and `window.roamAlphaAPI`

Roam's official extension system (**Roam Depot**) is a concrete model of the
*engine-hosts-adapter* path (UC-38):

- **Distribution:** an extension is a GitHub repo (`README.md`, `extension.js`, optional
  `extension.css` / `CHANGELOG.md` / `build.sh`), cataloged in the `roam-depot` repo via
  a metadata JSON (`name`, `author`, `tags`, `source_repo`, `source_commit`, optional
  `stripe_account` for paid extensions).
- **Lifecycle:** `extension.js` default-exports a map with **`onload`** and
  **`onunload`** functions; everything created on load must be torn down on unload. An
  `extensionAPI` object provides a **settings panel** and `settings.get/set`.
- **Data API — `window.roamAlphaAPI`** (the surface an adapter would target):
  - **Read:** `q` (Datalog query → entity ids), `pull` / `pull-many` (fetch entity
    attributes), full graph traversal.
  - **Write:** `data.block.create` / `update` / `move` / `delete`, `data.page.create`.
  - **UI:** command palette, blocks, settings.

Crucially, this API runs **inside the Roam client** (global `window.roamAlphaAPI`).
There is no first-class external REST write API; programmatic external access uses
graph **export** (EDN / JSON / Markdown) or unofficial private APIs. So a *live,
write-through* Roam adapter must be **hosted inside Roam as a Depot extension** — which
is exactly the UC-38 engine-side direction (cf. XWiki components, TWiki plugin handlers).

---

## 6. Roam as a shard — capability profile

| Capability | Roam | Notes for the adapter contract |
|------------|------|--------------------------------|
| Read | **yes** | `q`/`pull` live (in-app) or EDN/JSON/Markdown export (snapshot) |
| Write | **yes, in-app** | `block.create/update/move/delete`, `page.create` — but only via the in-client API → needs a Depot-hosted adapter (UC-38) or unofficial private API |
| Write granularity | **block-level (fine)** | the opposite extreme from TiddlyWiki's whole-file writes — sharpens UC-35 |
| Identity / addressing | **block `:block/uid` + page `:node/title`** | shipped sub-page stable addressing (UC-51) |
| Transclusion | **yes** | block embeds by uid (UC-32/44/45) |
| Backlinks | **yes** | linked (`:block/refs`) + unlinked (text) (UC-05/18) |
| Structured data | **yes** | attributes + Datalog (UC-34/39) |
| Native query | **yes (Datalog)** | adapter could delegate views (UC-52) |
| Diff / merge | **no native** | transaction log exists but not exposed as page diffs |
| Version history | **hosted, internal-only** | not portable Git history → needs supplement/import (UC-36, like Confluence/MediaWiki) |
| Lock | **no** | — |
| Syntax | **Roam-flavored Markdown** | not CommonMark; block-outline structure → translation/mapping needed (cf. UC-42) |

Verdict: Roam is a legitimate **DB-backed, API-attached shard** — the XWiki family, not
the file-store family. Its standout offerings are **block-level addressing** and
**shipped transclusion**; its costs are **block↔page impedance** and **no portable
history**.

---

## 7. Evidence on shard-wiki's open questions (the payoff)

The Nelson dives left open questions; Roam answers several *empirically*:

1. **"What is a portable fine-grained span address?"** (Xanadu §11 Q1, the tumbler
   problem). Roam's answer: **a short opaque per-block UID, minted by the store, public
   and referenceable.** shard-wiki lesson — fine-grained addressing is *tractable* when
   the backend mints stable block/span IDs; the adapter should **adopt native block IDs
   as span addresses where they exist** (UC-51), and fall back to content-fingerprint or
   path+range where they do not (cf. Xanadu content-identity, UC-46).
2. **"Does transclusion belong in core or adapter/UI?"** (catalog Q6). Roam shows
   transclusion is cheap and natural *when the store is addressable and queryable*; it
   lives at the **data layer** (refs/embeds), surfaced by UI. Argues for transclusion as
   a **core capability over an addressable union**, not a UI-only trick.
3. **"Derived views — core or adapter?"** Roam shows them as **queries over a
   structured space**. Where a shard exposes a native query engine, **delegate**
   (UC-52); where it does not, the orchestrator computes them over the projection.
4. **Block↔page mapping** is the new question Roam *raises*: in a block graph, what is a
   "page"? Roam's own answer (`:node/title` node = page; nested blocks = spans) is a
   clean mapping rule for the adapter (UC-50).

---

## 8. Mapping to shard-wiki INTENT (compare, do not equate)

### 8.1 Reinforcements

- **Fine-grained addressing is achievable** (block UIDs) — de-risks UC-44/45/51 and the
  Xanadu tumbler worry.
- **Transclusion + bidirectional links + structured data** are not exotic; a shipped
  tool does all three over one addressable, queryable store — validating shard-wiki's
  ambitions for UC-32/34/39/05.
- **Engine-hosts-adapter (UC-38)** has a clean modern template: Depot's
  `onload/onunload` + `roamAlphaAPI` read/write surface.

### 8.2 Deliberate divergences (design bugs if conflated)

1. **Block-DB substrate vs. Markdown-file page model.** Roam's "everything is a block in
   a DataScript DB" must **not** tempt shard-wiki away from its Markdown-first,
   backend-neutral page model (INTENT Stability Note). Roam is *one shard shape*, mapped
   *into* the page model, not the model itself.
2. **Client-side proprietary DB vs. Git coordination journal.** Roam's transaction log
   is not portable history; shard-wiki keeps Git as the coordination layer. A Roam shard
   contributes snapshots/projections, and its history needs **supplement/import**
   (UC-36), not adoption as canonical.
3. **Single graph vs. federation.** A Roam graph is **one sovereign shard**, never the
   federation layer. Do not model the union as "a big Roam graph."
4. **Roam-flavored Markdown + outline structure** ≠ CommonMark pages; the adapter owns a
   **lossy-aware translation** (block outline ↔ page + headings/lists), cf. UC-42.

### 8.3 What Roam teaches that shard-wiki should keep

- **Mint/adopt stable sub-page IDs.** The cheapest path to transclusion, overlay, and
  reverse-lookup at span granularity is a backend that already addresses spans — lean on
  it (UC-51), degrade gracefully otherwise.
- **Treat a structured shard's query engine as a capability**, and delegate derived
  views to it when present (UC-52).
- **A clean block↔page rule** (`:node/title` node = page) keeps a block backend usable
  without flattening (UC-50, complementing UC-34's no-lossy-flatten rule).

---

## 9. Use-case seeds → catalog (promoted 2026-06-14)

Last existing UC is **UC-49**. New UCs **UC-50–UC-52** added; existing UCs enriched.

| Seed | Catalog action |
|------|----------------|
| **Attach a block-graph DB wiki (Roam-style) as a shard via its query/CRUD API, mapping blocks to the page model** (`:node/title` node = page; nested blocks = spans) | **UC-50 (new)** |
| **Adopt a shard's native block/span IDs as portable span addresses** for transclusion/overlay (the shipped answer to the span-address question) | **UC-51 (new)** |
| **Delegate derived views to a shard's native query engine** (Datalog) where advertised | **UC-52 (new)** |
| Shipped transclusion via block embeds + shipped fine-grained addressing | **enriches UC-32** (and UC-44/45) |
| Attributes + Datalog = shipped structured/typed pages | **enriches UC-34** (and UC-39) |
| Roam Depot `onload/onunload` + `roamAlphaAPI` = modern engine-hosts-adapter template | **enriches UC-38** |
| Block-level write granularity = the fine extreme opposite TiddlyWiki | **enriches UC-35** |
| Hosted history is internal-only, not portable Git | links **UC-36** |

---

## 10. Architecture notes for SHARD-WP-0002 (no UC)

- The adapter contract should model **"native span/block IDs"** as a capability: a shard
  advertises whether it mints stable sub-page addresses; transclusion/overlay/reverse-
  lookup capabilities key off it (ties UC-44/45/46/51).
- The contract should model a **"native query"** capability so the orchestrator can
  delegate derived-view computation (UC-52) vs. scanning the projection.
- A **block↔page mapping** belongs in the adapter, with the no-lossy-flatten rule of
  UC-34: page = titled node, blocks = addressable spans, attributes = sidecar metadata.
- Roam joins XWiki as a **DB-backed / API-attached** exemplar (vs. TWiki/Foswiki file
  stores) and as a **second engine-hosts-adapter** template (vs. XWiki components / TWiki
  handlers) for the T14 adapter-binding task.

---

## 11. Open questions (for spec / workplans)

1. When a backend mints native span IDs (Roam UIDs), are they used **directly** as the
   shard-wiki span address, or **wrapped** in a shard-scoped address so they survive
   projection/overlay and don't collide across shards?
2. For a **write-through** Roam shard the adapter must run *inside* Roam (Depot
   extension). Is in-engine hosting an accepted adapter deployment mode generally, or do
   we restrict Roam to **read/projection/overlay-target** (graceful degradation) when we
   cannot deploy inside it?
3. How lossy is the **block-outline ↔ Markdown-page** mapping, and is round-trip
   fidelity required or best-effort (cf. UC-42 Foswiki TML↔HTML)?
4. Do we consume Roam's **export (EDN/JSON)** as the projection source, or the live
   `roamAlphaAPI`? Snapshot vs. live freshness trade-off (cf. UC-31).

---

## 12. Sources

| Source | Used for |
|--------|----------|
| zsolt.blog — "Deep Dive Into Roam's Data Structure" (https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html) | Datom/EAV model, block attributes (`:block/uid` etc.), refs/embeds, linked vs unlinked references, Datalog/pull |
| Roam Research Datalog Cheatsheet (https://gist.github.com/2b3pro/231e4f230ed41e3f52e8a89ebf49848b) | Attribute names, `:q`/`:pull` query shapes |
| GitHub — Roam-Research/roam-depot (https://github.com/Roam-Research/roam-depot) | Extension repo/manifest structure, `extension.js` `onload`/`onunload`, `extensionAPI` settings, `roamAlphaAPI` read/write/ui surface |
| "Introduction to the Roam Alpha API" (https://www.putyourleftfoot.in/introduction-to-the-roam-alpha-api) | `window.roamAlphaAPI` `q`/`pull` semantics, global-scope dependency |
| David Vargas — "How To Create Your Own Roam Extensions" (https://davidvargas.me/blog/how_to_create_your_own_roam_extensions) | Extension authoring lifecycle, write methods |
| Wikipedia / general — Roam Research, DataScript, Datomic | Datalog/Datomic lineage, client-side DB framing |

Cross-references: `research/260614-xanadu-deep-dive/findings.md` (span addressing,
transclusion), `research/260614-zigzag-deep-dive/findings.md` (queryable dimensions),
`research/260613-xwiki-deep-dive/findings.md` (DB-backed engine + extension host),
`spec/UseCaseCatalog.md` (UC-05, UC-18, UC-32, UC-34/35/36/38/39, UC-44/45/46),
`workplans/SHARD-WP-0002-federation-architecture.md` (T14 adapter binding).

---

## 13. Traceability

- New UCs: **UC-50, UC-51, UC-52** → `spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-32, UC-34, UC-35, UC-38** (and links UC-36, UC-39, UC-44/45/46).
- Architecture (no UC): native-span-ID capability; native-query capability; block↔page
  mapping; Roam as DB-backed/API-attached + engine-hosts-adapter exemplar → `SHARD-WP-0002` (T14).
- Boundary recorded: Roam is **one candidate shard**, mapped into the Markdown-first
  page model; not a substrate, not the federation layer (INTENT Stability Note).
</content>