From dfff9ab42e0c8917d58854ce6ad05d76ec9fee1f Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 14 Jun 2026 11:36:50 +0200 Subject: [PATCH] research: Roam Research deep dive (block-graph DataScript DB, transclusion, datalog, Roam Depot extension API); UC-50/51/52 The modern bookend to the Xanadu/ZigZag dives: where those are unbuilt ideals, Roam shipped fine-grained addressing (:block/uid), live transclusion (block embeds), bidirectional links, and a queryable structured space (DataScript datoms + Datalog). Studied as a candidate DB-backed/API-attached shard (XWiki family) and as a concrete engine-hosts-adapter surface (Roam Depot onload/onunload over window.roamAlphaAPI). Added UC-50 (attach block-graph DB shard, block<->page mapping), UC-51 (adopt native span IDs as portable span addresses), UC-52 (delegate derived views to a shard's native query engine); enriched UC-32/34/35/38. Boundary: Roam is one candidate shard mapped into the Markdown-first page model, not a substrate and not the federation layer. Co-Authored-By: Claude Opus 4.8 --- SCOPE.md | 4 +- research/260614-roam-deep-dive/README.md | 49 ++++ research/260614-roam-deep-dive/findings.md | 292 +++++++++++++++++++++ research/README.md | 3 +- spec/UseCaseCatalog.md | 108 +++++++- 5 files changed, 447 insertions(+), 9 deletions(-) create mode 100644 research/260614-roam-deep-dive/README.md create mode 100644 research/260614-roam-deep-dive/findings.md diff --git a/SCOPE.md b/SCOPE.md index a26a6fc..2e0f58a 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -19,9 +19,9 @@ Learnings update both SCOPE and INTENT where necessary. |-------|-------| | Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) | | Intent | `INTENT.md` established; authorization-in-core amendments drafted | -| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag deep dives (`research/260614-*/`) | +| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam deep dives (`research/260614-*/`) | | Demand | NetKingdom integration asks captured, not yet negotiated | -| Spec | Architecture blueprint drafted; UseCaseCatalog 49 UCs from research; PRD/TSD scaffolds | +| Spec | Architecture blueprint drafted; UseCaseCatalog 52 UCs from research; PRD/TSD scaffolds | | Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) | ## In Scope (today) diff --git a/research/260614-roam-deep-dive/README.md b/research/260614-roam-deep-dive/README.md new file mode 100644 index 0000000..93b11cb --- /dev/null +++ b/research/260614-roam-deep-dive/README.md @@ -0,0 +1,49 @@ +# 260614 — Roam Research deep dive (the block graph as a queryable database + extension API) + +Date: 2026-06-14 + +## What this is + +A focused study of **Roam Research** — core architecture and extension structure — +read through shard-wiki's lens. Roam is the **modern bookend** to the two Nelson dives: +where Xanadu (`research/260614-xanadu-deep-dive/`) and ZigZag +(`research/260614-zigzag-deep-dive/`) are *unbuilt ideals*, Roam **shipped** stable +fine-grained addressing, transclusion, bidirectional links, and a queryable structured +space to a mainstream audience. + +Distinctive material: +- **Data model** — the entire graph is a client-side **DataScript** (Datomic-like) DB + of **datoms** (EAV); every paragraph and page is a **block** entity with a stable + nine-character **`:block/uid`**, `:block/string`, `:block/refs`, `:block/page`, + pages marked by `:node/title` +- **References & transclusion** — `[[Page]]`/`#tag`/`((uid))` create `:block/refs` + edges; **block embeds** are working transclusion; linked vs unlinked references +- **Querying** — Datalog (`:q`/`:pull`); derived views are queries over the graph +- **Extension architecture** — **Roam Depot** (`extension.js` `onload`/`onunload`, + manifest, `extensionAPI` settings) over **`window.roamAlphaAPI`** read (`q`/`pull`) + + write (`block.create/update/move/delete`, `page.create`) — a concrete + *engine-hosts-adapter* (UC-38) surface + +## Contents + +| Path | Role | +|------|------| +| `findings.md` | Data model, references/transclusion, Datalog, extension API, Roam-as-shard capability profile, evidence on shard-wiki's open questions, INTENT mapping, UC seeds, architecture notes, sources | + +## Status + +Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md` +(UC-50 attach a block-graph DB shard with block↔page mapping, UC-51 adopt native +block/span IDs as portable span addresses, UC-52 delegate derived views to a shard's +native query engine); UC-32/34/35/38 enriched. Logged for +`workplans/SHARD-WP-0002-federation-architecture.md` (T14): native-span-ID and +native-query capabilities, block↔page mapping, and Roam as a second DB-backed / +engine-hosts-adapter exemplar alongside XWiki. + +**Key payoff recorded:** Roam empirically answers the Nelson dives' open questions — +fine-grained addressing is tractable (block UIDs), transclusion is a data-layer +capability over an addressable union, and derived views are queries over a structured +space. **Boundary:** Roam is *one candidate shard* (DB-backed/API-attached, block-first, +no portable Git history), mapped into shard-wiki's Markdown-first page model — not a +substrate and not the federation layer. + diff --git a/research/260614-roam-deep-dive/findings.md b/research/260614-roam-deep-dive/findings.md new file mode 100644 index 0000000..6d06352 --- /dev/null +++ b/research/260614-roam-deep-dive/findings.md @@ -0,0 +1,292 @@ +# Findings — Roam Research: the block graph as a queryable database, and its extension API + +Date: 2026-06-14 +Source kind: **modern shipped product** — a block-graph note tool; a *candidate shard +backend* (DB-backed / API-attached, like XWiki), and the place where several Nelson +ideas actually shipped +Lens: shard-wiki — fine-grained addressing, transclusion, bidirectional links, a +structured/queryable page space, and the engine-hosts-adapter extension path + +> Why Roam, and why now. The previous two dives were Ted Nelson's *unbuilt* ideals — +> Xanadu (`research/260614-xanadu-deep-dive/`: reference-not-copy, transclusion, +> stable fine-grained addresses) and ZigZag (`research/260614-zigzag-deep-dive/`: an +> information space as many co-equal dimensions over a queryable structure). **Roam +> Research is where a large slice of both actually shipped to a mainstream audience.** +> Every block has a stable address; block references and embeds are working +> transclusion; the whole graph is a queryable database; links are bidirectional. So +> Roam is the natural modern bookend: it lets us check shard-wiki's open questions +> (portable span addressing, transclusion, structured pages) against a system that +> solved them *in production* — and study its **extension architecture** as a concrete +> adapter-host surface. + +This dive treats Roam as a *candidate shard* (capability profile in §6) and as +*evidence* on the open questions the Nelson dives left (§7), not as something to copy +wholesale — Roam is block-first and DB-backed, shard-wiki is Markdown-page-first and +Git-backed (divergences in §8.2). + +--- + +## 1. What Roam is, in architecture terms + +Roam Research is a networked-thought outliner whose **entire graph is a client-side +DataScript database** (DataScript = a ClojureScript reimplementation of Datomic's +data model and Datalog query engine). The store is not files; it is a set of immutable +**datoms**. + +- A **datom** is one fact: `[entity, attribute, value, transaction]` — the EAV(T) + model. The graph is the accumulation of datoms; edits are transactions. +- Synchronization and Roam's deep undo fall out of the transaction log — a parallel + worth noting to Git's immutable commit history (but it is *not* Git; see §8.2). + +This makes Roam a **structured, queryable** wiki-shaped store — the same family as +XWiki (objects + DB) rather than TWiki/Foswiki (flat files), which matters for how an +adapter attaches (§6). + +--- + +## 2. The block model — everything is a block, every block is addressable + +Every paragraph *and* every page is a **block** (an entity). Key attributes: + +| Attribute | Meaning | +|-----------|---------| +| `:block/uid` | **nine-character public ID** — the block's stable, referenceable address (e.g. `((GGv3cyL6Y))`) | +| `:block/string` | the block's text content | +| `:block/order` | position among siblings | +| `:block/children` | immediate child entity-ids | +| `:block/parents` | full ancestor chain (incl. the page) | +| `:block/refs` | edges to blocks/pages this block references | +| `:block/page` | the owning page entity | +| `:create/time`, `:edit/time`, `:create/email`, `:edit/email` | provenance metadata | +| `:node/title` | **page-only** — the page's title (this attribute is what distinguishes a page from an ordinary block) | + +Structurally the DB is a **forest**: each page is a tree of nested blocks; a "page" is +just the top-level block that carries `:node/title`. The single most shard-wiki-relevant +fact: **`:block/uid` gives every block — i.e. every sub-page span — a stable, +first-class, public address.** This is the *shipped* form of the Xanadu tumbler / the +shard-wiki "portable span address" open question (§7). + +--- + +## 3. References, transclusion, bidirectional links + +- **Page references** `[[Page]]` and `#tags`, and **block references** `((uid))`, all + create `:block/refs` edges. The link graph is therefore queryable, not parsed-on-read. +- **Block embeds** (`{{embed: ((uid))}}`) are **working transclusion**: a block's + content rendered live in another location by reference, not copied — when the source + changes, the embed reflects it. This is Xanadu transclusion / shard-wiki UC-32/UC-44, + shipped at block granularity. +- **Bidirectional links** come in two flavours, exactly shard-wiki's BackLinks problem: + - **Linked references** — explicit `:block/refs` edges (queryable backlinks). + - **Unlinked references** — bare text mentions with no edge, surfaced by text search. +- **Attributes** `key:: value` are a convention on top of blocks/refs that turns a page + into a lightweight record — Roam's answer to structured/typed pages (UC-34/UC-39), + queryable via Datalog. + +--- + +## 4. Querying — Datalog over the graph + +Roam exposes **Datalog** (`:find` / `:where`) plus **pull** expressions: + +``` +[:find ?uid :where [?b :block/string ?s] [?b :block/uid ?uid] ...] +``` + +This means Roam's "derived views" (a page's linked references, a `{{query}}` block, a +table) are **just queries over the datom graph** — not bespoke features. For shard-wiki +this is the strongest evidence yet that *derived views = queries over a structured +space* (the ZigZag "dimensions + rasters" insight, made executable), and that an +adapter could **delegate view computation to a shard's native query engine** instead of +scanning (UC-52). + +--- + +## 5. Extension architecture — Roam Depot and `window.roamAlphaAPI` + +Roam's official extension system (**Roam Depot**) is a concrete model of the +*engine-hosts-adapter* path (UC-38): + +- **Distribution:** an extension is a GitHub repo (`README.md`, `extension.js`, optional + `extension.css` / `CHANGELOG.md` / `build.sh`), cataloged in the `roam-depot` repo via + a metadata JSON (`name`, `author`, `tags`, `source_repo`, `source_commit`, optional + `stripe_account` for paid extensions). +- **Lifecycle:** `extension.js` default-exports a map with **`onload`** and + **`onunload`** functions; everything created on load must be torn down on unload. An + `extensionAPI` object provides a **settings panel** and `settings.get/set`. +- **Data API — `window.roamAlphaAPI`** (the surface an adapter would target): + - **Read:** `q` (Datalog query → entity ids), `pull` / `pull-many` (fetch entity + attributes), full graph traversal. + - **Write:** `data.block.create` / `update` / `move` / `delete`, `data.page.create`. + - **UI:** command palette, blocks, settings. + +Crucially, this API runs **inside the Roam client** (global `window.roamAlphaAPI`). +There is no first-class external REST write API; programmatic external access uses +graph **export** (EDN / JSON / Markdown) or unofficial private APIs. So a *live, +write-through* Roam adapter must be **hosted inside Roam as a Depot extension** — which +is exactly the UC-38 engine-side direction (cf. XWiki components, TWiki plugin handlers). + +--- + +## 6. Roam as a shard — capability profile + +| Capability | Roam | Notes for the adapter contract | +|------------|------|--------------------------------| +| Read | **yes** | `q`/`pull` live (in-app) or EDN/JSON/Markdown export (snapshot) | +| Write | **yes, in-app** | `block.create/update/move/delete`, `page.create` — but only via the in-client API → needs a Depot-hosted adapter (UC-38) or unofficial private API | +| Write granularity | **block-level (fine)** | the opposite extreme from TiddlyWiki's whole-file writes — sharpens UC-35 | +| Identity / addressing | **block `:block/uid` + page `:node/title`** | shipped sub-page stable addressing (UC-51) | +| Transclusion | **yes** | block embeds by uid (UC-32/44/45) | +| Backlinks | **yes** | linked (`:block/refs`) + unlinked (text) (UC-05/18) | +| Structured data | **yes** | attributes + Datalog (UC-34/39) | +| Native query | **yes (Datalog)** | adapter could delegate views (UC-52) | +| Diff / merge | **no native** | transaction log exists but not exposed as page diffs | +| Version history | **hosted, internal-only** | not portable Git history → needs supplement/import (UC-36, like Confluence/MediaWiki) | +| Lock | **no** | — | +| Syntax | **Roam-flavored Markdown** | not CommonMark; block-outline structure → translation/mapping needed (cf. UC-42) | + +Verdict: Roam is a legitimate **DB-backed, API-attached shard** — the XWiki family, not +the file-store family. Its standout offerings are **block-level addressing** and +**shipped transclusion**; its costs are **block↔page impedance** and **no portable +history**. + +--- + +## 7. Evidence on shard-wiki's open questions (the payoff) + +The Nelson dives left open questions; Roam answers several *empirically*: + +1. **"What is a portable fine-grained span address?"** (Xanadu §11 Q1, the tumbler + problem). Roam's answer: **a short opaque per-block UID, minted by the store, public + and referenceable.** shard-wiki lesson — fine-grained addressing is *tractable* when + the backend mints stable block/span IDs; the adapter should **adopt native block IDs + as span addresses where they exist** (UC-51), and fall back to content-fingerprint or + path+range where they do not (cf. Xanadu content-identity, UC-46). +2. **"Does transclusion belong in core or adapter/UI?"** (catalog Q6). Roam shows + transclusion is cheap and natural *when the store is addressable and queryable*; it + lives at the **data layer** (refs/embeds), surfaced by UI. Argues for transclusion as + a **core capability over an addressable union**, not a UI-only trick. +3. **"Derived views — core or adapter?"** Roam shows them as **queries over a + structured space**. Where a shard exposes a native query engine, **delegate** + (UC-52); where it does not, the orchestrator computes them over the projection. +4. **Block↔page mapping** is the new question Roam *raises*: in a block graph, what is a + "page"? Roam's own answer (`:node/title` node = page; nested blocks = spans) is a + clean mapping rule for the adapter (UC-50). + +--- + +## 8. Mapping to shard-wiki INTENT (compare, do not equate) + +### 8.1 Reinforcements + +- **Fine-grained addressing is achievable** (block UIDs) — de-risks UC-44/45/51 and the + Xanadu tumbler worry. +- **Transclusion + bidirectional links + structured data** are not exotic; a shipped + tool does all three over one addressable, queryable store — validating shard-wiki's + ambitions for UC-32/34/39/05. +- **Engine-hosts-adapter (UC-38)** has a clean modern template: Depot's + `onload/onunload` + `roamAlphaAPI` read/write surface. + +### 8.2 Deliberate divergences (design bugs if conflated) + +1. **Block-DB substrate vs. Markdown-file page model.** Roam's "everything is a block in + a DataScript DB" must **not** tempt shard-wiki away from its Markdown-first, + backend-neutral page model (INTENT Stability Note). Roam is *one shard shape*, mapped + *into* the page model, not the model itself. +2. **Client-side proprietary DB vs. Git coordination journal.** Roam's transaction log + is not portable history; shard-wiki keeps Git as the coordination layer. A Roam shard + contributes snapshots/projections, and its history needs **supplement/import** + (UC-36), not adoption as canonical. +3. **Single graph vs. federation.** A Roam graph is **one sovereign shard**, never the + federation layer. Do not model the union as "a big Roam graph." +4. **Roam-flavored Markdown + outline structure** ≠ CommonMark pages; the adapter owns a + **lossy-aware translation** (block outline ↔ page + headings/lists), cf. UC-42. + +### 8.3 What Roam teaches that shard-wiki should keep + +- **Mint/adopt stable sub-page IDs.** The cheapest path to transclusion, overlay, and + reverse-lookup at span granularity is a backend that already addresses spans — lean on + it (UC-51), degrade gracefully otherwise. +- **Treat a structured shard's query engine as a capability**, and delegate derived + views to it when present (UC-52). +- **A clean block↔page rule** (`:node/title` node = page) keeps a block backend usable + without flattening (UC-50, complementing UC-34's no-lossy-flatten rule). + +--- + +## 9. Use-case seeds → catalog (promoted 2026-06-14) + +Last existing UC is **UC-49**. New UCs **UC-50–UC-52** added; existing UCs enriched. + +| Seed | Catalog action | +|------|----------------| +| **Attach a block-graph DB wiki (Roam-style) as a shard via its query/CRUD API, mapping blocks to the page model** (`:node/title` node = page; nested blocks = spans) | **UC-50 (new)** | +| **Adopt a shard's native block/span IDs as portable span addresses** for transclusion/overlay (the shipped answer to the span-address question) | **UC-51 (new)** | +| **Delegate derived views to a shard's native query engine** (Datalog) where advertised | **UC-52 (new)** | +| Shipped transclusion via block embeds + shipped fine-grained addressing | **enriches UC-32** (and UC-44/45) | +| Attributes + Datalog = shipped structured/typed pages | **enriches UC-34** (and UC-39) | +| Roam Depot `onload/onunload` + `roamAlphaAPI` = modern engine-hosts-adapter template | **enriches UC-38** | +| Block-level write granularity = the fine extreme opposite TiddlyWiki | **enriches UC-35** | +| Hosted history is internal-only, not portable Git | links **UC-36** | + +--- + +## 10. Architecture notes for SHARD-WP-0002 (no UC) + +- The adapter contract should model **"native span/block IDs"** as a capability: a shard + advertises whether it mints stable sub-page addresses; transclusion/overlay/reverse- + lookup capabilities key off it (ties UC-44/45/46/51). +- The contract should model a **"native query"** capability so the orchestrator can + delegate derived-view computation (UC-52) vs. scanning the projection. +- A **block↔page mapping** belongs in the adapter, with the no-lossy-flatten rule of + UC-34: page = titled node, blocks = addressable spans, attributes = sidecar metadata. +- Roam joins XWiki as a **DB-backed / API-attached** exemplar (vs. TWiki/Foswiki file + stores) and as a **second engine-hosts-adapter** template (vs. XWiki components / TWiki + handlers) for the T14 adapter-binding task. + +--- + +## 11. Open questions (for spec / workplans) + +1. When a backend mints native span IDs (Roam UIDs), are they used **directly** as the + shard-wiki span address, or **wrapped** in a shard-scoped address so they survive + projection/overlay and don't collide across shards? +2. For a **write-through** Roam shard the adapter must run *inside* Roam (Depot + extension). Is in-engine hosting an accepted adapter deployment mode generally, or do + we restrict Roam to **read/projection/overlay-target** (graceful degradation) when we + cannot deploy inside it? +3. How lossy is the **block-outline ↔ Markdown-page** mapping, and is round-trip + fidelity required or best-effort (cf. UC-42 Foswiki TML↔HTML)? +4. Do we consume Roam's **export (EDN/JSON)** as the projection source, or the live + `roamAlphaAPI`? Snapshot vs. live freshness trade-off (cf. UC-31). + +--- + +## 12. Sources + +| Source | Used for | +|--------|----------| +| zsolt.blog — "Deep Dive Into Roam's Data Structure" (https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html) | Datom/EAV model, block attributes (`:block/uid` etc.), refs/embeds, linked vs unlinked references, Datalog/pull | +| Roam Research Datalog Cheatsheet (https://gist.github.com/2b3pro/231e4f230ed41e3f52e8a89ebf49848b) | Attribute names, `:q`/`:pull` query shapes | +| GitHub — Roam-Research/roam-depot (https://github.com/Roam-Research/roam-depot) | Extension repo/manifest structure, `extension.js` `onload`/`onunload`, `extensionAPI` settings, `roamAlphaAPI` read/write/ui surface | +| "Introduction to the Roam Alpha API" (https://www.putyourleftfoot.in/introduction-to-the-roam-alpha-api) | `window.roamAlphaAPI` `q`/`pull` semantics, global-scope dependency | +| David Vargas — "How To Create Your Own Roam Extensions" (https://davidvargas.me/blog/how_to_create_your_own_roam_extensions) | Extension authoring lifecycle, write methods | +| Wikipedia / general — Roam Research, DataScript, Datomic | Datalog/Datomic lineage, client-side DB framing | + +Cross-references: `research/260614-xanadu-deep-dive/findings.md` (span addressing, +transclusion), `research/260614-zigzag-deep-dive/findings.md` (queryable dimensions), +`research/260613-xwiki-deep-dive/findings.md` (DB-backed engine + extension host), +`spec/UseCaseCatalog.md` (UC-05, UC-18, UC-32, UC-34/35/36/38/39, UC-44/45/46), +`workplans/SHARD-WP-0002-federation-architecture.md` (T14 adapter binding). + +--- + +## 13. Traceability + +- New UCs: **UC-50, UC-51, UC-52** → `spec/UseCaseCatalog.md`. +- Enriched UCs: **UC-32, UC-34, UC-35, UC-38** (and links UC-36, UC-39, UC-44/45/46). +- Architecture (no UC): native-span-ID capability; native-query capability; block↔page + mapping; Roam as DB-backed/API-attached + engine-hosts-adapter exemplar → `SHARD-WP-0002` (T14). +- Boundary recorded: Roam is **one candidate shard**, mapped into the Markdown-first + page model; not a substrate, not the federation layer (INTENT Stability Note). + diff --git a/research/README.md b/research/README.md index cb40c6a..c6bb3b8 100644 --- a/research/README.md +++ b/research/README.md @@ -18,4 +18,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and | 2026-06-13 | `260613-twiki-deep-dive/` | TWiki impl, plugin API, ecosystem; UC-40/41 | | 2026-06-13 | `260613-foswiki-deep-dive/` | Foswiki store abstraction, extension API; UC-42/43 | | 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 | -| 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 | \ No newline at end of file +| 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 | +| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 | \ No newline at end of file diff --git a/spec/UseCaseCatalog.md b/spec/UseCaseCatalog.md index 8a352cf..fb1c31e 100644 --- a/spec/UseCaseCatalog.md +++ b/spec/UseCaseCatalog.md @@ -6,7 +6,8 @@ Promoted from `research/260608-c2-wiki-origins/`, `research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`, `research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`, `research/260613-twiki-deep-dive/`, `research/260613-foswiki-deep-dive/`, -`research/260614-xanadu-deep-dive/`, and `research/260614-zigzag-deep-dive/`. +`research/260614-xanadu-deep-dive/`, `research/260614-zigzag-deep-dive/`, and +`research/260614-roam-deep-dive/`. See InfoTechPrimers on coulomb.social for use-case catalog conventions. ## Conventions @@ -188,7 +189,11 @@ projection. Provenance and staleness must be explicit. Xanadu deep dive sharpens transclusion should be **content-identity based and bidirectional** (content is "knowably in more than one place" and aware of its appearances), not a one-way path fetch — enables UC-45 reverse lookup (`research/260614-xanadu-deep-dive/findings.md` §4). See -UC-44 for the whole-page composition-manifest form. +UC-44 for the whole-page composition-manifest form. **Roam ships this**: block embeds +transclude by `:block/uid` live (not copied), proving transclusion is a cheap data-layer +capability over an addressable store (`research/260614-roam-deep-dive/findings.md` §3, +§7) — argues for transclusion in core over the union, surfaced by UI. Depends on span +addressing (UC-51). **Priority:** Later ### UC-33 — Git-branch an information space @@ -217,7 +222,10 @@ XObjects against an XClass schema (`research/260613-xwiki-deep-dive/findings.md` §2.3); UC-39 covers the extreme where the page is *only* structure. TWiki shows the **git-friendly** variant: TWiki Forms store fields as `%META:FIELD%` *inside the topic text file*, so structure is diffable rather than locked in a DB -(`research/260613-twiki-deep-dive/findings.md` §2.3). +(`research/260613-twiki-deep-dive/findings.md` §2.3). Roam is the modern exemplar: +`key:: value` attributes over a datom graph, queryable via Datalog +(`research/260614-roam-deep-dive/findings.md` §3, §4) — structure to preserve via +sidecar metadata, and a candidate for query delegation (UC-52). **Priority:** Later ### UC-35 — Attach a shard with coarse write granularity @@ -229,7 +237,10 @@ respect that granularity instead of assuming per-page writes. **Source:** wikiengines, intent **Notes:** Capability-aware adapters: the **capability profile must encode write granularity** (per-page / per-file / per-space / append-only). Overlay and -patch flows must adapt (findings §5 #1). Affects conflict scope and locking. +patch flows must adapt (findings §5 #1). Affects conflict scope and locking. Roam marks +the **fine extreme** — block-level writes (`block.create/update/move/delete`), the +opposite of TiddlyWiki's whole-file granularity +(`research/260614-roam-deep-dive/findings.md` §6). **Priority:** Later ### UC-36 — Supply a git-addressable history to an internal-history engine @@ -278,7 +289,11 @@ surfacing — enough to host a high-fidelity adapter adapter (needs deploy access, higher fidelity) is open (findings §8 Q3). Generalizes beyond XWiki: TWiki's plugin handler API (`beforeSaveHandler`, `afterRenameHandler`, REST handlers) is an equivalent host surface -(`research/260613-twiki-deep-dive/findings.md` §3). +(`research/260613-twiki-deep-dive/findings.md` §3). **Roam Depot** is the modern +template: an extension default-exports `onload`/`onunload` and drives +`window.roamAlphaAPI` read (`q`/`pull`) + write (`block.*`/`page.create`) +(`research/260614-roam-deep-dive/findings.md` §5) — note Roam's API runs *inside* the +client, so write-through requires in-engine hosting (findings §11 Q2). **Priority:** Later ### UC-39 — Attach a wiki-as-application-platform shard (pages as typed records) @@ -443,6 +458,51 @@ genealogy edge recorded at fork/remix/import time (UC-26, UC-29) in the coordina journal. Makes provenance a traversal, not a buried footer (complements UC-24). **Priority:** Later +### UC-50 — Attach a block-graph database wiki as a shard, mapping blocks to the page model + +**Actor:** Maintainer +**Goal:** Attach a block-first, DB-backed graph tool (Roam-style) as a shard via its +query/CRUD API, mapping its blocks onto shard-wiki's Markdown page model without +flattening the structure. +**Source:** roam, intent +**Notes:** Roam's graph is a DataScript datom DB where a *page* is the node carrying +`:node/title` and nested *blocks* are addressable spans +(`research/260614-roam-deep-dive/findings.md` §2, §6). Clean mapping rule: titled node += page, nested blocks = spans, attributes = sidecar metadata (honors UC-34 no-lossy- +flatten). DB-backed/API-attached path (like XWiki UC-38), **not** the file-store path +(UC-40). Write-through requires hosting the adapter inside the engine (Roam Depot), +else degrade to read/projection/overlay-target (findings §11 Q2). +**Priority:** Later + +### UC-51 — Adopt a shard's native span IDs as portable span addresses + +**Actor:** Orchestrator / adapter +**Goal:** Where a backend mints stable sub-page identifiers (Roam `:block/uid`), adopt +them as the span address for transclusion, overlay, and reverse-lookup, rather than +inventing one. +**Source:** roam, xanadu, intent +**Notes:** Roam ships the fine-grained stable address that Xanadu's tumblers +(`research/260614-xanadu-deep-dive/findings.md` §3) and the catalog left open — a short +opaque per-block UID, public and referenceable (`research/260614-roam-deep-dive/findings.md` +§2, §7). Enables UC-44/45 at sub-page granularity; falls back to content-fingerprint +(UC-46) or path+range where no native ID exists. Open: use native ID directly or wrap in +a shard-scoped address to avoid cross-shard collision and survive projection (findings +§11 Q1). +**Priority:** Later + +### UC-52 — Delegate derived views to a shard's native query engine + +**Actor:** Orchestrator +**Goal:** When a shard exposes a native query engine, compute derived views (backlinks, +recent changes, typed queries) by delegating to it instead of scanning the projection. +**Source:** roam, intent +**Notes:** In Roam, derived views *are* Datalog queries over the datom graph +(`research/260614-roam-deep-dive/findings.md` §4). Makes "native query" an adapter +capability the orchestrator can key off (vs. computing views itself over the +projection). Operationalizes the ZigZag "dimensions + rasters" insight +(`research/260614-zigzag-deep-dive/findings.md` §5) and relates to UC-05/UC-34. +**Priority:** Later + --- ## B. Knowledge work and collaboration @@ -678,6 +738,9 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD. | UC-47 | | | ‡ | | ✓ | | UC-48 | | | ‡ | | | | UC-49 | | | ‡ | | ✓ | +| UC-50 | | | | § | ✓ | +| UC-51 | | | | § | ✓ | +| UC-52 | | | | § | ✓ | | UC-08 | ✓ | | | | UC-09 | ✓ | | | | UC-10 | ✓ | | | @@ -823,6 +886,36 @@ Architecture logged for `SHARD-WP-0002`: a dimensional projection layer; genealo edges in the coordination journal; clone↔transclusion as a possible shared primitive (findings §9). Pairs with the Xanadu dive (clone ↔ transclusion convergence). +### roam mapping + +(§ UC-50–UC-52 are placed in the **wikiengines** matrix column as the nearest existing +source — Roam is a shipped tool — but their true lineage is the **Roam deep dive**, +`research/260614-roam-deep-dive/findings.md`.) + +| Roam mechanism (findings §) | Catalog UC | +|-----------------------------|------------| +| Block-graph DataScript DB; page = `:node/title` node, blocks = spans (§2, §6) | UC-50 | +| `:block/uid` — shipped stable sub-page address (§2, §7) | UC-51 | +| Derived views = Datalog queries over the graph (§4) | UC-52 | +| Block embeds = shipped live transclusion by uid (§3) | UC-32 (enriched; + UC-44/45) | +| `key:: value` attributes + Datalog = structured/typed pages (§3, §4) | UC-34 (enriched; + UC-39) | +| Roam Depot `onload/onunload` + `roamAlphaAPI` = engine-hosts-adapter template (§5) | UC-38 (enriched) | +| Block-level write granularity = the fine extreme (§6) | UC-35 (enriched) | +| Hosted history is internal-only, not portable Git (§6, §8.2) | links UC-36 | + +Note: Roam is the **modern bookend** to the Nelson dives — where Xanadu and ZigZag are +unbuilt ideals, Roam *shipped* fine-grained addressing, transclusion, bidirectional +links, and a queryable structured space. Its payoff is **empirical evidence** on +shard-wiki's open questions (span addressing is tractable via native block IDs; +transclusion is a data-layer capability; derived views are queries — +`research/260614-roam-deep-dive/findings.md` §7). **Boundary recorded:** Roam is *one +candidate shard* — DB-backed / API-attached (the XWiki family, not the file-store +family), block-first, with no portable Git history — **mapped into** shard-wiki's +Markdown-first page model, not adopted as a substrate and never the federation layer +(findings §8.2). Architecture logged for `SHARD-WP-0002` (T14): native-span-ID and +native-query capabilities, block↔page mapping, and Roam as a second +engine-hosts-adapter exemplar alongside XWiki. + --- ## Open questions @@ -840,4 +933,7 @@ edges in the coordination journal; clone↔transclusion as a possible shared pri 7. Is the **dimensional/zzstructure model** (UC-47/48) a public navigation API and UI paradigm, or only an internal organizing concept for the derived views? 8. How is the **many-to-many link graph** reconciled with the rank-based dimensional - model — clones, a parallel graph index, or both? (ZigZag dive §10.) \ No newline at end of file + model — clones, a parallel graph index, or both? (ZigZag dive §10.) +9. Is **in-engine adapter hosting** (e.g. a Roam Depot extension for write-through) an + accepted deployment mode generally, or do we restrict API-only backends to + read/projection/overlay-target? (Roam dive §11 Q2; ties UC-38, UC-50.) \ No newline at end of file