coulomb/shard-wiki
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research: Roam Research deep dive (block-graph DataScript DB, transclusion, datalog, Roam Depot extension API); UC-50/51/52

Browse Source 
The modern bookend to the Xanadu/ZigZag dives: where those are unbuilt
ideals, Roam shipped fine-grained addressing (:block/uid), live
transclusion (block embeds), bidirectional links, and a queryable
structured space (DataScript datoms + Datalog). Studied as a candidate
DB-backed/API-attached shard (XWiki family) and as a concrete
engine-hosts-adapter surface (Roam Depot onload/onunload over
window.roamAlphaAPI). Added UC-50 (attach block-graph DB shard, block<->page
mapping), UC-51 (adopt native span IDs as portable span addresses), UC-52
(delegate derived views to a shard's native query engine); enriched
UC-32/34/35/38. Boundary: Roam is one candidate shard mapped into the
Markdown-first page model, not a substrate and not the federation layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-06-14 11:36:50 +02:00
parent 7c22c468af
commit dfff9ab42e
5 changed files with 447 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
SCOPE.md
View File

@@ -19,9 +19,9 @@ Learnings update both SCOPE and INTENT where necessary.
|-------|-------| |-------|-------|
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) | | Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
| Intent | `INTENT.md` established; authorization-in-core amendments drafted | | Intent | `INTENT.md` established; authorization-in-core amendments drafted |
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag deep dives (`research/260614-*/`) | | Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam deep dives (`research/260614-*/`) |
| Demand | NetKingdom integration asks captured, not yet negotiated | | Demand | NetKingdom integration asks captured, not yet negotiated |
| Spec | Architecture blueprint drafted; UseCaseCatalog 49 UCs from research; PRD/TSD scaffolds | | Spec | Architecture blueprint drafted; UseCaseCatalog 52 UCs from research; PRD/TSD scaffolds |
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) | | Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) |
## In Scope (today) ## In Scope (today)

49
research/260614-roam-deep-dive/README.md Normal file
View File

@@ -0,0 +1,49 @@
# 260614 — Roam Research deep dive (the block graph as a queryable database + extension API)
Date: 2026-06-14
## What this is
A focused study of **Roam Research** — core architecture and extension structure —
read through shard-wiki's lens. Roam is the **modern bookend** to the two Nelson dives:
where Xanadu (`research/260614-xanadu-deep-dive/`) and ZigZag
(`research/260614-zigzag-deep-dive/`) are *unbuilt ideals*, Roam **shipped** stable
fine-grained addressing, transclusion, bidirectional links, and a queryable structured
space to a mainstream audience.
Distinctive material:
- **Data model** — the entire graph is a client-side **DataScript** (Datomic-like) DB
of **datoms** (EAV); every paragraph and page is a **block** entity with a stable
nine-character **`:block/uid`**, `:block/string`, `:block/refs`, `:block/page`,
pages marked by `:node/title`
- **References & transclusion** — `[[Page]]`/`#tag`/`((uid))` create `:block/refs`
edges; **block embeds** are working transclusion; linked vs unlinked references
- **Querying** — Datalog (`:q`/`:pull`); derived views are queries over the graph
- **Extension architecture** — **Roam Depot** (`extension.js` `onload`/`onunload`,
manifest, `extensionAPI` settings) over **`window.roamAlphaAPI`** read (`q`/`pull`) +
write (`block.create/update/move/delete`, `page.create`) — a concrete
*engine-hosts-adapter* (UC-38) surface
## Contents
| Path | Role |
|------|------|
| `findings.md` | Data model, references/transclusion, Datalog, extension API, Roam-as-shard capability profile, evidence on shard-wiki's open questions, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
(UC-50 attach a block-graph DB shard with block↔page mapping, UC-51 adopt native
block/span IDs as portable span addresses, UC-52 delegate derived views to a shard's
native query engine); UC-32/34/35/38 enriched. Logged for
`workplans/SHARD-WP-0002-federation-architecture.md` (T14): native-span-ID and
native-query capabilities, block↔page mapping, and Roam as a second DB-backed /
engine-hosts-adapter exemplar alongside XWiki.
**Key payoff recorded:** Roam empirically answers the Nelson dives' open questions —
fine-grained addressing is tractable (block UIDs), transclusion is a data-layer
capability over an addressable union, and derived views are queries over a structured
space. **Boundary:** Roam is *one candidate shard* (DB-backed/API-attached, block-first,
no portable Git history), mapped into shard-wiki's Markdown-first page model — not a
substrate and not the federation layer.
</content>

292
research/260614-roam-deep-dive/findings.md Normal file
View File

@@ -0,0 +1,292 @@
# Findings — Roam Research: the block graph as a queryable database, and its extension API
Date: 2026-06-14
Source kind: **modern shipped product** — a block-graph note tool; a *candidate shard
backend* (DB-backed / API-attached, like XWiki), and the place where several Nelson
ideas actually shipped
Lens: shard-wiki — fine-grained addressing, transclusion, bidirectional links, a
structured/queryable page space, and the engine-hosts-adapter extension path
> Why Roam, and why now. The previous two dives were Ted Nelson's *unbuilt* ideals —
> Xanadu (`research/260614-xanadu-deep-dive/`: reference-not-copy, transclusion,
> stable fine-grained addresses) and ZigZag (`research/260614-zigzag-deep-dive/`: an
> information space as many co-equal dimensions over a queryable structure). **Roam
> Research is where a large slice of both actually shipped to a mainstream audience.**
> Every block has a stable address; block references and embeds are working
> transclusion; the whole graph is a queryable database; links are bidirectional. So
> Roam is the natural modern bookend: it lets us check shard-wiki's open questions
> (portable span addressing, transclusion, structured pages) against a system that
> solved them *in production* — and study its **extension architecture** as a concrete
> adapter-host surface.
This dive treats Roam as a *candidate shard* (capability profile in §6) and as
*evidence* on the open questions the Nelson dives left (§7), not as something to copy
wholesale — Roam is block-first and DB-backed, shard-wiki is Markdown-page-first and
Git-backed (divergences in §8.2).
---
## 1. What Roam is, in architecture terms
Roam Research is a networked-thought outliner whose **entire graph is a client-side
DataScript database** (DataScript = a ClojureScript reimplementation of Datomic's
data model and Datalog query engine). The store is not files; it is a set of immutable
**datoms**.
- A **datom** is one fact: `[entity, attribute, value, transaction]` — the EAV(T)
model. The graph is the accumulation of datoms; edits are transactions.
- Synchronization and Roam's deep undo fall out of the transaction log — a parallel
worth noting to Git's immutable commit history (but it is *not* Git; see §8.2).
This makes Roam a **structured, queryable** wiki-shaped store — the same family as
XWiki (objects + DB) rather than TWiki/Foswiki (flat files), which matters for how an
adapter attaches (§6).
---
## 2. The block model — everything is a block, every block is addressable
Every paragraph *and* every page is a **block** (an entity). Key attributes:
| Attribute | Meaning |
|-----------|---------|
| `:block/uid` | **nine-character public ID** — the block's stable, referenceable address (e.g. `((GGv3cyL6Y))`) |
| `:block/string` | the block's text content |
| `:block/order` | position among siblings |
| `:block/children` | immediate child entity-ids |
| `:block/parents` | full ancestor chain (incl. the page) |
| `:block/refs` | edges to blocks/pages this block references |
| `:block/page` | the owning page entity |
| `:create/time`, `:edit/time`, `:create/email`, `:edit/email` | provenance metadata |
| `:node/title` | **page-only** — the page's title (this attribute is what distinguishes a page from an ordinary block) |
Structurally the DB is a **forest**: each page is a tree of nested blocks; a "page" is
just the top-level block that carries `:node/title`. The single most shard-wiki-relevant
fact: **`:block/uid` gives every block — i.e. every sub-page span — a stable,
first-class, public address.** This is the *shipped* form of the Xanadu tumbler / the
shard-wiki "portable span address" open question (§7).
---
## 3. References, transclusion, bidirectional links
- **Page references** `[[Page]]` and `#tags`, and **block references** `((uid))`, all
create `:block/refs` edges. The link graph is therefore queryable, not parsed-on-read.
- **Block embeds** (`{{embed: ((uid))}}`) are **working transclusion**: a block's
content rendered live in another location by reference, not copied — when the source
changes, the embed reflects it. This is Xanadu transclusion / shard-wiki UC-32/UC-44,
shipped at block granularity.
- **Bidirectional links** come in two flavours, exactly shard-wiki's BackLinks problem:
- **Linked references** — explicit `:block/refs` edges (queryable backlinks).
- **Unlinked references** — bare text mentions with no edge, surfaced by text search.
- **Attributes** `key:: value` are a convention on top of blocks/refs that turns a page
into a lightweight record — Roam's answer to structured/typed pages (UC-34/UC-39),
queryable via Datalog.
---
## 4. Querying — Datalog over the graph
Roam exposes **Datalog** (`:find` / `:where`) plus **pull** expressions:
```
[:find ?uid :where [?b :block/string ?s] [?b :block/uid ?uid] ...]
```
This means Roam's "derived views" (a page's linked references, a `{{query}}` block, a
table) are **just queries over the datom graph** — not bespoke features. For shard-wiki
this is the strongest evidence yet that *derived views = queries over a structured
space* (the ZigZag "dimensions + rasters" insight, made executable), and that an
adapter could **delegate view computation to a shard's native query engine** instead of
scanning (UC-52).
---
## 5. Extension architecture — Roam Depot and `window.roamAlphaAPI`
Roam's official extension system (**Roam Depot**) is a concrete model of the
*engine-hosts-adapter* path (UC-38):
- **Distribution:** an extension is a GitHub repo (`README.md`, `extension.js`, optional
`extension.css` / `CHANGELOG.md` / `build.sh`), cataloged in the `roam-depot` repo via
a metadata JSON (`name`, `author`, `tags`, `source_repo`, `source_commit`, optional
`stripe_account` for paid extensions).
- **Lifecycle:** `extension.js` default-exports a map with **`onload`** and
**`onunload`** functions; everything created on load must be torn down on unload. An
`extensionAPI` object provides a **settings panel** and `settings.get/set`.
- **Data API — `window.roamAlphaAPI`** (the surface an adapter would target):
- **Read:** `q` (Datalog query → entity ids), `pull` / `pull-many` (fetch entity
attributes), full graph traversal.
- **Write:** `data.block.create` / `update` / `move` / `delete`, `data.page.create`.
- **UI:** command palette, blocks, settings.
Crucially, this API runs **inside the Roam client** (global `window.roamAlphaAPI`).
There is no first-class external REST write API; programmatic external access uses
graph **export** (EDN / JSON / Markdown) or unofficial private APIs. So a *live,
write-through* Roam adapter must be **hosted inside Roam as a Depot extension** — which
is exactly the UC-38 engine-side direction (cf. XWiki components, TWiki plugin handlers).
---
## 6. Roam as a shard — capability profile
| Capability | Roam | Notes for the adapter contract |
|------------|------|--------------------------------|
| Read | **yes** | `q`/`pull` live (in-app) or EDN/JSON/Markdown export (snapshot) |
| Write | **yes, in-app** | `block.create/update/move/delete`, `page.create` — but only via the in-client API → needs a Depot-hosted adapter (UC-38) or unofficial private API |
| Write granularity | **block-level (fine)** | the opposite extreme from TiddlyWiki's whole-file writes — sharpens UC-35 |
| Identity / addressing | **block `:block/uid` + page `:node/title`** | shipped sub-page stable addressing (UC-51) |
| Transclusion | **yes** | block embeds by uid (UC-32/44/45) |
| Backlinks | **yes** | linked (`:block/refs`) + unlinked (text) (UC-05/18) |
| Structured data | **yes** | attributes + Datalog (UC-34/39) |
| Native query | **yes (Datalog)** | adapter could delegate views (UC-52) |
| Diff / merge | **no native** | transaction log exists but not exposed as page diffs |
| Version history | **hosted, internal-only** | not portable Git history → needs supplement/import (UC-36, like Confluence/MediaWiki) |
| Lock | **no** | — |
| Syntax | **Roam-flavored Markdown** | not CommonMark; block-outline structure → translation/mapping needed (cf. UC-42) |
Verdict: Roam is a legitimate **DB-backed, API-attached shard** — the XWiki family, not
the file-store family. Its standout offerings are **block-level addressing** and
**shipped transclusion**; its costs are **block↔page impedance** and **no portable
history**.
---
## 7. Evidence on shard-wiki's open questions (the payoff)
The Nelson dives left open questions; Roam answers several *empirically*:
1. **"What is a portable fine-grained span address?"** (Xanadu §11 Q1, the tumbler
problem). Roam's answer: **a short opaque per-block UID, minted by the store, public
and referenceable.** shard-wiki lesson — fine-grained addressing is *tractable* when
the backend mints stable block/span IDs; the adapter should **adopt native block IDs
as span addresses where they exist** (UC-51), and fall back to content-fingerprint or
path+range where they do not (cf. Xanadu content-identity, UC-46).
2. **"Does transclusion belong in core or adapter/UI?"** (catalog Q6). Roam shows
transclusion is cheap and natural *when the store is addressable and queryable*; it
lives at the **data layer** (refs/embeds), surfaced by UI. Argues for transclusion as
a **core capability over an addressable union**, not a UI-only trick.
3. **"Derived views — core or adapter?"** Roam shows them as **queries over a
structured space**. Where a shard exposes a native query engine, **delegate**
(UC-52); where it does not, the orchestrator computes them over the projection.
4. **Block↔page mapping** is the new question Roam *raises*: in a block graph, what is a
"page"? Roam's own answer (`:node/title` node = page; nested blocks = spans) is a
clean mapping rule for the adapter (UC-50).
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **Fine-grained addressing is achievable** (block UIDs) — de-risks UC-44/45/51 and the
Xanadu tumbler worry.
- **Transclusion + bidirectional links + structured data** are not exotic; a shipped
tool does all three over one addressable, queryable store — validating shard-wiki's
ambitions for UC-32/34/39/05.
- **Engine-hosts-adapter (UC-38)** has a clean modern template: Depot's
`onload/onunload` + `roamAlphaAPI` read/write surface.
### 8.2 Deliberate divergences (design bugs if conflated)
1. **Block-DB substrate vs. Markdown-file page model.** Roam's "everything is a block in
a DataScript DB" must **not** tempt shard-wiki away from its Markdown-first,
backend-neutral page model (INTENT Stability Note). Roam is *one shard shape*, mapped
*into* the page model, not the model itself.
2. **Client-side proprietary DB vs. Git coordination journal.** Roam's transaction log
is not portable history; shard-wiki keeps Git as the coordination layer. A Roam shard
contributes snapshots/projections, and its history needs **supplement/import**
(UC-36), not adoption as canonical.
3. **Single graph vs. federation.** A Roam graph is **one sovereign shard**, never the
federation layer. Do not model the union as "a big Roam graph."
4. **Roam-flavored Markdown + outline structure** ≠ CommonMark pages; the adapter owns a
**lossy-aware translation** (block outline ↔ page + headings/lists), cf. UC-42.
### 8.3 What Roam teaches that shard-wiki should keep
- **Mint/adopt stable sub-page IDs.** The cheapest path to transclusion, overlay, and
reverse-lookup at span granularity is a backend that already addresses spans — lean on
it (UC-51), degrade gracefully otherwise.
- **Treat a structured shard's query engine as a capability**, and delegate derived
views to it when present (UC-52).
- **A clean block↔page rule** (`:node/title` node = page) keeps a block backend usable
without flattening (UC-50, complementing UC-34's no-lossy-flatten rule).
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-49**. New UCs **UC-50–UC-52** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a block-graph DB wiki (Roam-style) as a shard via its query/CRUD API, mapping blocks to the page model** (`:node/title` node = page; nested blocks = spans) | **UC-50 (new)** |
| **Adopt a shard's native block/span IDs as portable span addresses** for transclusion/overlay (the shipped answer to the span-address question) | **UC-51 (new)** |
| **Delegate derived views to a shard's native query engine** (Datalog) where advertised | **UC-52 (new)** |
| Shipped transclusion via block embeds + shipped fine-grained addressing | **enriches UC-32** (and UC-44/45) |
| Attributes + Datalog = shipped structured/typed pages | **enriches UC-34** (and UC-39) |
| Roam Depot `onload/onunload` + `roamAlphaAPI` = modern engine-hosts-adapter template | **enriches UC-38** |
| Block-level write granularity = the fine extreme opposite TiddlyWiki | **enriches UC-35** |
| Hosted history is internal-only, not portable Git | links **UC-36** |
---
## 10. Architecture notes for SHARD-WP-0002 (no UC)
- The adapter contract should model **"native span/block IDs"** as a capability: a shard
advertises whether it mints stable sub-page addresses; transclusion/overlay/reverse-
lookup capabilities key off it (ties UC-44/45/46/51).
- The contract should model a **"native query"** capability so the orchestrator can
delegate derived-view computation (UC-52) vs. scanning the projection.
- A **block↔page mapping** belongs in the adapter, with the no-lossy-flatten rule of
UC-34: page = titled node, blocks = addressable spans, attributes = sidecar metadata.
- Roam joins XWiki as a **DB-backed / API-attached** exemplar (vs. TWiki/Foswiki file
stores) and as a **second engine-hosts-adapter** template (vs. XWiki components / TWiki
handlers) for the T14 adapter-binding task.
---
## 11. Open questions (for spec / workplans)
1. When a backend mints native span IDs (Roam UIDs), are they used **directly** as the
shard-wiki span address, or **wrapped** in a shard-scoped address so they survive
projection/overlay and don't collide across shards?
2. For a **write-through** Roam shard the adapter must run *inside* Roam (Depot
extension). Is in-engine hosting an accepted adapter deployment mode generally, or do
we restrict Roam to **read/projection/overlay-target** (graceful degradation) when we
cannot deploy inside it?
3. How lossy is the **block-outline ↔ Markdown-page** mapping, and is round-trip
fidelity required or best-effort (cf. UC-42 Foswiki TML↔HTML)?
4. Do we consume Roam's **export (EDN/JSON)** as the projection source, or the live
`roamAlphaAPI`? Snapshot vs. live freshness trade-off (cf. UC-31).
---
## 12. Sources
| Source | Used for |
|--------|----------|
| zsolt.blog — "Deep Dive Into Roam's Data Structure" (https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html) | Datom/EAV model, block attributes (`:block/uid` etc.), refs/embeds, linked vs unlinked references, Datalog/pull |
| Roam Research Datalog Cheatsheet (https://gist.github.com/2b3pro/231e4f230ed41e3f52e8a89ebf49848b) | Attribute names, `:q`/`:pull` query shapes |
| GitHub — Roam-Research/roam-depot (https://github.com/Roam-Research/roam-depot) | Extension repo/manifest structure, `extension.js` `onload`/`onunload`, `extensionAPI` settings, `roamAlphaAPI` read/write/ui surface |
| "Introduction to the Roam Alpha API" (https://www.putyourleftfoot.in/introduction-to-the-roam-alpha-api) | `window.roamAlphaAPI` `q`/`pull` semantics, global-scope dependency |
| David Vargas — "How To Create Your Own Roam Extensions" (https://davidvargas.me/blog/how_to_create_your_own_roam_extensions) | Extension authoring lifecycle, write methods |
| Wikipedia / general — Roam Research, DataScript, Datomic | Datalog/Datomic lineage, client-side DB framing |
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (span addressing,
transclusion), `research/260614-zigzag-deep-dive/findings.md` (queryable dimensions),
`research/260613-xwiki-deep-dive/findings.md` (DB-backed engine + extension host),
`spec/UseCaseCatalog.md` (UC-05, UC-18, UC-32, UC-34/35/36/38/39, UC-44/45/46),
`workplans/SHARD-WP-0002-federation-architecture.md` (T14 adapter binding).
---
## 13. Traceability
- New UCs: **UC-50, UC-51, UC-52** → `spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-32, UC-34, UC-35, UC-38** (and links UC-36, UC-39, UC-44/45/46).
- Architecture (no UC): native-span-ID capability; native-query capability; block↔page
mapping; Roam as DB-backed/API-attached + engine-hosts-adapter exemplar → `SHARD-WP-0002` (T14).
- Boundary recorded: Roam is **one candidate shard**, mapped into the Markdown-first
page model; not a substrate, not the federation layer (INTENT Stability Note).
</content>

3
research/README.md
View File

@@ -18,4 +18,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and
| 2026-06-13 | `260613-twiki-deep-dive/` | TWiki impl, plugin API, ecosystem; UC-40/41 | | 2026-06-13 | `260613-twiki-deep-dive/` | TWiki impl, plugin API, ecosystem; UC-40/41 |
| 2026-06-13 | `260613-foswiki-deep-dive/` | Foswiki store abstraction, extension API; UC-42/43 | | 2026-06-13 | `260613-foswiki-deep-dive/` | Foswiki store abstraction, extension API; UC-42/43 |
| 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 | | 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 |
| 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 | | 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 |
| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 |

108
spec/UseCaseCatalog.md
View File

@@ -6,7 +6,8 @@ Promoted from `research/260608-c2-wiki-origins/`,
`research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`, `research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`,
`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`, `research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`,
`research/260613-twiki-deep-dive/`, `research/260613-foswiki-deep-dive/`, `research/260613-twiki-deep-dive/`, `research/260613-foswiki-deep-dive/`,
`research/260614-xanadu-deep-dive/`, and `research/260614-zigzag-deep-dive/`. `research/260614-xanadu-deep-dive/`, `research/260614-zigzag-deep-dive/`, and
`research/260614-roam-deep-dive/`.
See InfoTechPrimers on coulomb.social for use-case catalog conventions. See InfoTechPrimers on coulomb.social for use-case catalog conventions.
## Conventions ## Conventions
@@ -188,7 +189,11 @@ projection. Provenance and staleness must be explicit. Xanadu deep dive sharpens
transclusion should be **content-identity based and bidirectional** (content is "knowably transclusion should be **content-identity based and bidirectional** (content is "knowably
in more than one place" and aware of its appearances), not a one-way path fetch — in more than one place" and aware of its appearances), not a one-way path fetch —
enables UC-45 reverse lookup (`research/260614-xanadu-deep-dive/findings.md` §4). See enables UC-45 reverse lookup (`research/260614-xanadu-deep-dive/findings.md` §4). See
UC-44 for the whole-page composition-manifest form. UC-44 for the whole-page composition-manifest form. **Roam ships this**: block embeds
transclude by `:block/uid` live (not copied), proving transclusion is a cheap data-layer
capability over an addressable store (`research/260614-roam-deep-dive/findings.md` §3,
§7) — argues for transclusion in core over the union, surfaced by UI. Depends on span
addressing (UC-51).
**Priority:** Later **Priority:** Later
### UC-33 — Git-branch an information space ### UC-33 — Git-branch an information space
@@ -217,7 +222,10 @@ XObjects against an XClass schema (`research/260613-xwiki-deep-dive/findings.md`
§2.3); UC-39 covers the extreme where the page is *only* structure. TWiki shows the §2.3); UC-39 covers the extreme where the page is *only* structure. TWiki shows the
**git-friendly** variant: TWiki Forms store fields as `%META:FIELD%` *inside the **git-friendly** variant: TWiki Forms store fields as `%META:FIELD%` *inside the
topic text file*, so structure is diffable rather than locked in a DB topic text file*, so structure is diffable rather than locked in a DB
(`research/260613-twiki-deep-dive/findings.md` §2.3). (`research/260613-twiki-deep-dive/findings.md` §2.3). Roam is the modern exemplar:
`key:: value` attributes over a datom graph, queryable via Datalog
(`research/260614-roam-deep-dive/findings.md` §3, §4) — structure to preserve via
sidecar metadata, and a candidate for query delegation (UC-52).
**Priority:** Later **Priority:** Later
### UC-35 — Attach a shard with coarse write granularity ### UC-35 — Attach a shard with coarse write granularity
@@ -229,7 +237,10 @@ respect that granularity instead of assuming per-page writes.
**Source:** wikiengines, intent **Source:** wikiengines, intent
**Notes:** Capability-aware adapters: the **capability profile must encode write **Notes:** Capability-aware adapters: the **capability profile must encode write
granularity** (per-page / per-file / per-space / append-only). Overlay and granularity** (per-page / per-file / per-space / append-only). Overlay and
patch flows must adapt (findings §5 #1). Affects conflict scope and locking. patch flows must adapt (findings §5 #1). Affects conflict scope and locking. Roam marks
the **fine extreme** — block-level writes (`block.create/update/move/delete`), the
opposite of TiddlyWiki's whole-file granularity
(`research/260614-roam-deep-dive/findings.md` §6).
**Priority:** Later **Priority:** Later
### UC-36 — Supply a git-addressable history to an internal-history engine ### UC-36 — Supply a git-addressable history to an internal-history engine
@@ -278,7 +289,11 @@ surfacing — enough to host a high-fidelity adapter
adapter (needs deploy access, higher fidelity) is open (findings §8 Q3). adapter (needs deploy access, higher fidelity) is open (findings §8 Q3).
Generalizes beyond XWiki: TWiki's plugin handler API (`beforeSaveHandler`, Generalizes beyond XWiki: TWiki's plugin handler API (`beforeSaveHandler`,
`afterRenameHandler`, REST handlers) is an equivalent host surface `afterRenameHandler`, REST handlers) is an equivalent host surface
(`research/260613-twiki-deep-dive/findings.md` §3). (`research/260613-twiki-deep-dive/findings.md` §3). **Roam Depot** is the modern
template: an extension default-exports `onload`/`onunload` and drives
`window.roamAlphaAPI` read (`q`/`pull`) + write (`block.*`/`page.create`)
(`research/260614-roam-deep-dive/findings.md` §5) — note Roam's API runs *inside* the
client, so write-through requires in-engine hosting (findings §11 Q2).
**Priority:** Later **Priority:** Later
### UC-39 — Attach a wiki-as-application-platform shard (pages as typed records) ### UC-39 — Attach a wiki-as-application-platform shard (pages as typed records)
@@ -443,6 +458,51 @@ genealogy edge recorded at fork/remix/import time (UC-26, UC-29) in the coordina
journal. Makes provenance a traversal, not a buried footer (complements UC-24). journal. Makes provenance a traversal, not a buried footer (complements UC-24).
**Priority:** Later **Priority:** Later
### UC-50 — Attach a block-graph database wiki as a shard, mapping blocks to the page model
**Actor:** Maintainer
**Goal:** Attach a block-first, DB-backed graph tool (Roam-style) as a shard via its
query/CRUD API, mapping its blocks onto shard-wiki's Markdown page model without
flattening the structure.
**Source:** roam, intent
**Notes:** Roam's graph is a DataScript datom DB where a *page* is the node carrying
`:node/title` and nested *blocks* are addressable spans
(`research/260614-roam-deep-dive/findings.md` §2, §6). Clean mapping rule: titled node
= page, nested blocks = spans, attributes = sidecar metadata (honors UC-34 no-lossy-
flatten). DB-backed/API-attached path (like XWiki UC-38), **not** the file-store path
(UC-40). Write-through requires hosting the adapter inside the engine (Roam Depot),
else degrade to read/projection/overlay-target (findings §11 Q2).
**Priority:** Later
### UC-51 — Adopt a shard's native span IDs as portable span addresses
**Actor:** Orchestrator / adapter
**Goal:** Where a backend mints stable sub-page identifiers (Roam `:block/uid`), adopt
them as the span address for transclusion, overlay, and reverse-lookup, rather than
inventing one.
**Source:** roam, xanadu, intent
**Notes:** Roam ships the fine-grained stable address that Xanadu's tumblers
(`research/260614-xanadu-deep-dive/findings.md` §3) and the catalog left open — a short
opaque per-block UID, public and referenceable (`research/260614-roam-deep-dive/findings.md`
§2, §7). Enables UC-44/45 at sub-page granularity; falls back to content-fingerprint
(UC-46) or path+range where no native ID exists. Open: use native ID directly or wrap in
a shard-scoped address to avoid cross-shard collision and survive projection (findings
§11 Q1).
**Priority:** Later
### UC-52 — Delegate derived views to a shard's native query engine
**Actor:** Orchestrator
**Goal:** When a shard exposes a native query engine, compute derived views (backlinks,
recent changes, typed queries) by delegating to it instead of scanning the projection.
**Source:** roam, intent
**Notes:** In Roam, derived views *are* Datalog queries over the datom graph
(`research/260614-roam-deep-dive/findings.md` §4). Makes "native query" an adapter
capability the orchestrator can key off (vs. computing views itself over the
projection). Operationalizes the ZigZag "dimensions + rasters" insight
(`research/260614-zigzag-deep-dive/findings.md` §5) and relates to UC-05/UC-34.
**Priority:** Later
--- ---
## B. Knowledge work and collaboration ## B. Knowledge work and collaboration
@@ -678,6 +738,9 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD.
| UC-47 | | | ‡ | | ✓ | | UC-47 | | | ‡ | | ✓ |
| UC-48 | | | ‡ | | | | UC-48 | | | ‡ | | |
| UC-49 | | | ‡ | | ✓ | | UC-49 | | | ‡ | | ✓ |
| UC-50 | | | | § | ✓ |
| UC-51 | | | | § | ✓ |
| UC-52 | | | | § | ✓ |
| UC-08 | ✓ | | | | UC-08 | ✓ | | |
| UC-09 | ✓ | | | | UC-09 | ✓ | | |
| UC-10 | ✓ | | | | UC-10 | ✓ | | |
@@ -823,6 +886,36 @@ Architecture logged for `SHARD-WP-0002`: a dimensional projection layer; genealo
edges in the coordination journal; clone↔transclusion as a possible shared primitive edges in the coordination journal; clone↔transclusion as a possible shared primitive
(findings §9). Pairs with the Xanadu dive (clone ↔ transclusion convergence). (findings §9). Pairs with the Xanadu dive (clone ↔ transclusion convergence).
### roam mapping
(§ UC-50–UC-52 are placed in the **wikiengines** matrix column as the nearest existing
source — Roam is a shipped tool — but their true lineage is the **Roam deep dive**,
`research/260614-roam-deep-dive/findings.md`.)
| Roam mechanism (findings §) | Catalog UC |
|-----------------------------|------------|
| Block-graph DataScript DB; page = `:node/title` node, blocks = spans (§2, §6) | UC-50 |
| `:block/uid` — shipped stable sub-page address (§2, §7) | UC-51 |
| Derived views = Datalog queries over the graph (§4) | UC-52 |
| Block embeds = shipped live transclusion by uid (§3) | UC-32 (enriched; + UC-44/45) |
| `key:: value` attributes + Datalog = structured/typed pages (§3, §4) | UC-34 (enriched; + UC-39) |
| Roam Depot `onload/onunload` + `roamAlphaAPI` = engine-hosts-adapter template (§5) | UC-38 (enriched) |
| Block-level write granularity = the fine extreme (§6) | UC-35 (enriched) |
| Hosted history is internal-only, not portable Git (§6, §8.2) | links UC-36 |
Note: Roam is the **modern bookend** to the Nelson dives — where Xanadu and ZigZag are
unbuilt ideals, Roam *shipped* fine-grained addressing, transclusion, bidirectional
links, and a queryable structured space. Its payoff is **empirical evidence** on
shard-wiki's open questions (span addressing is tractable via native block IDs;
transclusion is a data-layer capability; derived views are queries —
`research/260614-roam-deep-dive/findings.md` §7). **Boundary recorded:** Roam is *one
candidate shard* — DB-backed / API-attached (the XWiki family, not the file-store
family), block-first, with no portable Git history — **mapped into** shard-wiki's
Markdown-first page model, not adopted as a substrate and never the federation layer
(findings §8.2). Architecture logged for `SHARD-WP-0002` (T14): native-span-ID and
native-query capabilities, block↔page mapping, and Roam as a second
engine-hosts-adapter exemplar alongside XWiki.
--- ---
## Open questions ## Open questions
@@ -840,4 +933,7 @@ edges in the coordination journal; clone↔transclusion as a possible shared pri
7. Is the **dimensional/zzstructure model** (UC-47/48) a public navigation API and UI 7. Is the **dimensional/zzstructure model** (UC-47/48) a public navigation API and UI
paradigm, or only an internal organizing concept for the derived views? paradigm, or only an internal organizing concept for the derived views?
8. How is the **many-to-many link graph** reconciled with the rank-based dimensional 8. How is the **many-to-many link graph** reconciled with the rank-based dimensional
model — clones, a parallel graph index, or both? (ZigZag dive §10.) model — clones, a parallel graph index, or both? (ZigZag dive §10.)
9. Is **in-engine adapter hosting** (e.g. a Roam Depot extension for write-through) an
accepted deployment mode generally, or do we restrict API-only backends to
read/projection/overlay-target? (Roam dive §11 Q2; ties UC-38, UC-50.)