generated from coulomb/repo-seed
research: Roam Research deep dive (block-graph DataScript DB, transclusion, datalog, Roam Depot extension API); UC-50/51/52
The modern bookend to the Xanadu/ZigZag dives: where those are unbuilt ideals, Roam shipped fine-grained addressing (:block/uid), live transclusion (block embeds), bidirectional links, and a queryable structured space (DataScript datoms + Datalog). Studied as a candidate DB-backed/API-attached shard (XWiki family) and as a concrete engine-hosts-adapter surface (Roam Depot onload/onunload over window.roamAlphaAPI). Added UC-50 (attach block-graph DB shard, block<->page mapping), UC-51 (adopt native span IDs as portable span addresses), UC-52 (delegate derived views to a shard's native query engine); enriched UC-32/34/35/38. Boundary: Roam is one candidate shard mapped into the Markdown-first page model, not a substrate and not the federation layer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
4
SCOPE.md
4
SCOPE.md
@@ -19,9 +19,9 @@ Learnings update both SCOPE and INTENT where necessary.
|
||||
|-------|-------|
|
||||
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
|
||||
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
|
||||
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag deep dives (`research/260614-*/`) |
|
||||
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam deep dives (`research/260614-*/`) |
|
||||
| Demand | NetKingdom integration asks captured, not yet negotiated |
|
||||
| Spec | Architecture blueprint drafted; UseCaseCatalog 49 UCs from research; PRD/TSD scaffolds |
|
||||
| Spec | Architecture blueprint drafted; UseCaseCatalog 52 UCs from research; PRD/TSD scaffolds |
|
||||
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) |
|
||||
|
||||
## In Scope (today)
|
||||
|
||||
49
research/260614-roam-deep-dive/README.md
Normal file
49
research/260614-roam-deep-dive/README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# 260614 — Roam Research deep dive (the block graph as a queryable database + extension API)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Roam Research** — core architecture and extension structure —
|
||||
read through shard-wiki's lens. Roam is the **modern bookend** to the two Nelson dives:
|
||||
where Xanadu (`research/260614-xanadu-deep-dive/`) and ZigZag
|
||||
(`research/260614-zigzag-deep-dive/`) are *unbuilt ideals*, Roam **shipped** stable
|
||||
fine-grained addressing, transclusion, bidirectional links, and a queryable structured
|
||||
space to a mainstream audience.
|
||||
|
||||
Distinctive material:
|
||||
- **Data model** — the entire graph is a client-side **DataScript** (Datomic-like) DB
|
||||
of **datoms** (EAV); every paragraph and page is a **block** entity with a stable
|
||||
nine-character **`:block/uid`**, `:block/string`, `:block/refs`, `:block/page`,
|
||||
pages marked by `:node/title`
|
||||
- **References & transclusion** — `[[Page]]`/`#tag`/`((uid))` create `:block/refs`
|
||||
edges; **block embeds** are working transclusion; linked vs unlinked references
|
||||
- **Querying** — Datalog (`:q`/`:pull`); derived views are queries over the graph
|
||||
- **Extension architecture** — **Roam Depot** (`extension.js` `onload`/`onunload`,
|
||||
manifest, `extensionAPI` settings) over **`window.roamAlphaAPI`** read (`q`/`pull`) +
|
||||
write (`block.create/update/move/delete`, `page.create`) — a concrete
|
||||
*engine-hosts-adapter* (UC-38) surface
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Data model, references/transclusion, Datalog, extension API, Roam-as-shard capability profile, evidence on shard-wiki's open questions, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-50 attach a block-graph DB shard with block↔page mapping, UC-51 adopt native
|
||||
block/span IDs as portable span addresses, UC-52 delegate derived views to a shard's
|
||||
native query engine); UC-32/34/35/38 enriched. Logged for
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T14): native-span-ID and
|
||||
native-query capabilities, block↔page mapping, and Roam as a second DB-backed /
|
||||
engine-hosts-adapter exemplar alongside XWiki.
|
||||
|
||||
**Key payoff recorded:** Roam empirically answers the Nelson dives' open questions —
|
||||
fine-grained addressing is tractable (block UIDs), transclusion is a data-layer
|
||||
capability over an addressable union, and derived views are queries over a structured
|
||||
space. **Boundary:** Roam is *one candidate shard* (DB-backed/API-attached, block-first,
|
||||
no portable Git history), mapped into shard-wiki's Markdown-first page model — not a
|
||||
substrate and not the federation layer.
|
||||
</content>
|
||||
292
research/260614-roam-deep-dive/findings.md
Normal file
292
research/260614-roam-deep-dive/findings.md
Normal file
@@ -0,0 +1,292 @@
|
||||
# Findings — Roam Research: the block graph as a queryable database, and its extension API
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — a block-graph note tool; a *candidate shard
|
||||
backend* (DB-backed / API-attached, like XWiki), and the place where several Nelson
|
||||
ideas actually shipped
|
||||
Lens: shard-wiki — fine-grained addressing, transclusion, bidirectional links, a
|
||||
structured/queryable page space, and the engine-hosts-adapter extension path
|
||||
|
||||
> Why Roam, and why now. The previous two dives were Ted Nelson's *unbuilt* ideals —
|
||||
> Xanadu (`research/260614-xanadu-deep-dive/`: reference-not-copy, transclusion,
|
||||
> stable fine-grained addresses) and ZigZag (`research/260614-zigzag-deep-dive/`: an
|
||||
> information space as many co-equal dimensions over a queryable structure). **Roam
|
||||
> Research is where a large slice of both actually shipped to a mainstream audience.**
|
||||
> Every block has a stable address; block references and embeds are working
|
||||
> transclusion; the whole graph is a queryable database; links are bidirectional. So
|
||||
> Roam is the natural modern bookend: it lets us check shard-wiki's open questions
|
||||
> (portable span addressing, transclusion, structured pages) against a system that
|
||||
> solved them *in production* — and study its **extension architecture** as a concrete
|
||||
> adapter-host surface.
|
||||
|
||||
This dive treats Roam as a *candidate shard* (capability profile in §6) and as
|
||||
*evidence* on the open questions the Nelson dives left (§7), not as something to copy
|
||||
wholesale — Roam is block-first and DB-backed, shard-wiki is Markdown-page-first and
|
||||
Git-backed (divergences in §8.2).
|
||||
|
||||
---
|
||||
|
||||
## 1. What Roam is, in architecture terms
|
||||
|
||||
Roam Research is a networked-thought outliner whose **entire graph is a client-side
|
||||
DataScript database** (DataScript = a ClojureScript reimplementation of Datomic's
|
||||
data model and Datalog query engine). The store is not files; it is a set of immutable
|
||||
**datoms**.
|
||||
|
||||
- A **datom** is one fact: `[entity, attribute, value, transaction]` — the EAV(T)
|
||||
model. The graph is the accumulation of datoms; edits are transactions.
|
||||
- Synchronization and Roam's deep undo fall out of the transaction log — a parallel
|
||||
worth noting to Git's immutable commit history (but it is *not* Git; see §8.2).
|
||||
|
||||
This makes Roam a **structured, queryable** wiki-shaped store — the same family as
|
||||
XWiki (objects + DB) rather than TWiki/Foswiki (flat files), which matters for how an
|
||||
adapter attaches (§6).
|
||||
|
||||
---
|
||||
|
||||
## 2. The block model — everything is a block, every block is addressable
|
||||
|
||||
Every paragraph *and* every page is a **block** (an entity). Key attributes:
|
||||
|
||||
| Attribute | Meaning |
|
||||
|-----------|---------|
|
||||
| `:block/uid` | **nine-character public ID** — the block's stable, referenceable address (e.g. `((GGv3cyL6Y))`) |
|
||||
| `:block/string` | the block's text content |
|
||||
| `:block/order` | position among siblings |
|
||||
| `:block/children` | immediate child entity-ids |
|
||||
| `:block/parents` | full ancestor chain (incl. the page) |
|
||||
| `:block/refs` | edges to blocks/pages this block references |
|
||||
| `:block/page` | the owning page entity |
|
||||
| `:create/time`, `:edit/time`, `:create/email`, `:edit/email` | provenance metadata |
|
||||
| `:node/title` | **page-only** — the page's title (this attribute is what distinguishes a page from an ordinary block) |
|
||||
|
||||
Structurally the DB is a **forest**: each page is a tree of nested blocks; a "page" is
|
||||
just the top-level block that carries `:node/title`. The single most shard-wiki-relevant
|
||||
fact: **`:block/uid` gives every block — i.e. every sub-page span — a stable,
|
||||
first-class, public address.** This is the *shipped* form of the Xanadu tumbler / the
|
||||
shard-wiki "portable span address" open question (§7).
|
||||
|
||||
---
|
||||
|
||||
## 3. References, transclusion, bidirectional links
|
||||
|
||||
- **Page references** `[[Page]]` and `#tags`, and **block references** `((uid))`, all
|
||||
create `:block/refs` edges. The link graph is therefore queryable, not parsed-on-read.
|
||||
- **Block embeds** (`{{embed: ((uid))}}`) are **working transclusion**: a block's
|
||||
content rendered live in another location by reference, not copied — when the source
|
||||
changes, the embed reflects it. This is Xanadu transclusion / shard-wiki UC-32/UC-44,
|
||||
shipped at block granularity.
|
||||
- **Bidirectional links** come in two flavours, exactly shard-wiki's BackLinks problem:
|
||||
- **Linked references** — explicit `:block/refs` edges (queryable backlinks).
|
||||
- **Unlinked references** — bare text mentions with no edge, surfaced by text search.
|
||||
- **Attributes** `key:: value` are a convention on top of blocks/refs that turns a page
|
||||
into a lightweight record — Roam's answer to structured/typed pages (UC-34/UC-39),
|
||||
queryable via Datalog.
|
||||
|
||||
---
|
||||
|
||||
## 4. Querying — Datalog over the graph
|
||||
|
||||
Roam exposes **Datalog** (`:find` / `:where`) plus **pull** expressions:
|
||||
|
||||
```
|
||||
[:find ?uid :where [?b :block/string ?s] [?b :block/uid ?uid] ...]
|
||||
```
|
||||
|
||||
This means Roam's "derived views" (a page's linked references, a `{{query}}` block, a
|
||||
table) are **just queries over the datom graph** — not bespoke features. For shard-wiki
|
||||
this is the strongest evidence yet that *derived views = queries over a structured
|
||||
space* (the ZigZag "dimensions + rasters" insight, made executable), and that an
|
||||
adapter could **delegate view computation to a shard's native query engine** instead of
|
||||
scanning (UC-52).
|
||||
|
||||
---
|
||||
|
||||
## 5. Extension architecture — Roam Depot and `window.roamAlphaAPI`
|
||||
|
||||
Roam's official extension system (**Roam Depot**) is a concrete model of the
|
||||
*engine-hosts-adapter* path (UC-38):
|
||||
|
||||
- **Distribution:** an extension is a GitHub repo (`README.md`, `extension.js`, optional
|
||||
`extension.css` / `CHANGELOG.md` / `build.sh`), cataloged in the `roam-depot` repo via
|
||||
a metadata JSON (`name`, `author`, `tags`, `source_repo`, `source_commit`, optional
|
||||
`stripe_account` for paid extensions).
|
||||
- **Lifecycle:** `extension.js` default-exports a map with **`onload`** and
|
||||
**`onunload`** functions; everything created on load must be torn down on unload. An
|
||||
`extensionAPI` object provides a **settings panel** and `settings.get/set`.
|
||||
- **Data API — `window.roamAlphaAPI`** (the surface an adapter would target):
|
||||
- **Read:** `q` (Datalog query → entity ids), `pull` / `pull-many` (fetch entity
|
||||
attributes), full graph traversal.
|
||||
- **Write:** `data.block.create` / `update` / `move` / `delete`, `data.page.create`.
|
||||
- **UI:** command palette, blocks, settings.
|
||||
|
||||
Crucially, this API runs **inside the Roam client** (global `window.roamAlphaAPI`).
|
||||
There is no first-class external REST write API; programmatic external access uses
|
||||
graph **export** (EDN / JSON / Markdown) or unofficial private APIs. So a *live,
|
||||
write-through* Roam adapter must be **hosted inside Roam as a Depot extension** — which
|
||||
is exactly the UC-38 engine-side direction (cf. XWiki components, TWiki plugin handlers).
|
||||
|
||||
---
|
||||
|
||||
## 6. Roam as a shard — capability profile
|
||||
|
||||
| Capability | Roam | Notes for the adapter contract |
|
||||
|------------|------|--------------------------------|
|
||||
| Read | **yes** | `q`/`pull` live (in-app) or EDN/JSON/Markdown export (snapshot) |
|
||||
| Write | **yes, in-app** | `block.create/update/move/delete`, `page.create` — but only via the in-client API → needs a Depot-hosted adapter (UC-38) or unofficial private API |
|
||||
| Write granularity | **block-level (fine)** | the opposite extreme from TiddlyWiki's whole-file writes — sharpens UC-35 |
|
||||
| Identity / addressing | **block `:block/uid` + page `:node/title`** | shipped sub-page stable addressing (UC-51) |
|
||||
| Transclusion | **yes** | block embeds by uid (UC-32/44/45) |
|
||||
| Backlinks | **yes** | linked (`:block/refs`) + unlinked (text) (UC-05/18) |
|
||||
| Structured data | **yes** | attributes + Datalog (UC-34/39) |
|
||||
| Native query | **yes (Datalog)** | adapter could delegate views (UC-52) |
|
||||
| Diff / merge | **no native** | transaction log exists but not exposed as page diffs |
|
||||
| Version history | **hosted, internal-only** | not portable Git history → needs supplement/import (UC-36, like Confluence/MediaWiki) |
|
||||
| Lock | **no** | — |
|
||||
| Syntax | **Roam-flavored Markdown** | not CommonMark; block-outline structure → translation/mapping needed (cf. UC-42) |
|
||||
|
||||
Verdict: Roam is a legitimate **DB-backed, API-attached shard** — the XWiki family, not
|
||||
the file-store family. Its standout offerings are **block-level addressing** and
|
||||
**shipped transclusion**; its costs are **block↔page impedance** and **no portable
|
||||
history**.
|
||||
|
||||
---
|
||||
|
||||
## 7. Evidence on shard-wiki's open questions (the payoff)
|
||||
|
||||
The Nelson dives left open questions; Roam answers several *empirically*:
|
||||
|
||||
1. **"What is a portable fine-grained span address?"** (Xanadu §11 Q1, the tumbler
|
||||
problem). Roam's answer: **a short opaque per-block UID, minted by the store, public
|
||||
and referenceable.** shard-wiki lesson — fine-grained addressing is *tractable* when
|
||||
the backend mints stable block/span IDs; the adapter should **adopt native block IDs
|
||||
as span addresses where they exist** (UC-51), and fall back to content-fingerprint or
|
||||
path+range where they do not (cf. Xanadu content-identity, UC-46).
|
||||
2. **"Does transclusion belong in core or adapter/UI?"** (catalog Q6). Roam shows
|
||||
transclusion is cheap and natural *when the store is addressable and queryable*; it
|
||||
lives at the **data layer** (refs/embeds), surfaced by UI. Argues for transclusion as
|
||||
a **core capability over an addressable union**, not a UI-only trick.
|
||||
3. **"Derived views — core or adapter?"** Roam shows them as **queries over a
|
||||
structured space**. Where a shard exposes a native query engine, **delegate**
|
||||
(UC-52); where it does not, the orchestrator computes them over the projection.
|
||||
4. **Block↔page mapping** is the new question Roam *raises*: in a block graph, what is a
|
||||
"page"? Roam's own answer (`:node/title` node = page; nested blocks = spans) is a
|
||||
clean mapping rule for the adapter (UC-50).
|
||||
|
||||
---
|
||||
|
||||
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 8.1 Reinforcements
|
||||
|
||||
- **Fine-grained addressing is achievable** (block UIDs) — de-risks UC-44/45/51 and the
|
||||
Xanadu tumbler worry.
|
||||
- **Transclusion + bidirectional links + structured data** are not exotic; a shipped
|
||||
tool does all three over one addressable, queryable store — validating shard-wiki's
|
||||
ambitions for UC-32/34/39/05.
|
||||
- **Engine-hosts-adapter (UC-38)** has a clean modern template: Depot's
|
||||
`onload/onunload` + `roamAlphaAPI` read/write surface.
|
||||
|
||||
### 8.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Block-DB substrate vs. Markdown-file page model.** Roam's "everything is a block in
|
||||
a DataScript DB" must **not** tempt shard-wiki away from its Markdown-first,
|
||||
backend-neutral page model (INTENT Stability Note). Roam is *one shard shape*, mapped
|
||||
*into* the page model, not the model itself.
|
||||
2. **Client-side proprietary DB vs. Git coordination journal.** Roam's transaction log
|
||||
is not portable history; shard-wiki keeps Git as the coordination layer. A Roam shard
|
||||
contributes snapshots/projections, and its history needs **supplement/import**
|
||||
(UC-36), not adoption as canonical.
|
||||
3. **Single graph vs. federation.** A Roam graph is **one sovereign shard**, never the
|
||||
federation layer. Do not model the union as "a big Roam graph."
|
||||
4. **Roam-flavored Markdown + outline structure** ≠ CommonMark pages; the adapter owns a
|
||||
**lossy-aware translation** (block outline ↔ page + headings/lists), cf. UC-42.
|
||||
|
||||
### 8.3 What Roam teaches that shard-wiki should keep
|
||||
|
||||
- **Mint/adopt stable sub-page IDs.** The cheapest path to transclusion, overlay, and
|
||||
reverse-lookup at span granularity is a backend that already addresses spans — lean on
|
||||
it (UC-51), degrade gracefully otherwise.
|
||||
- **Treat a structured shard's query engine as a capability**, and delegate derived
|
||||
views to it when present (UC-52).
|
||||
- **A clean block↔page rule** (`:node/title` node = page) keeps a block backend usable
|
||||
without flattening (UC-50, complementing UC-34's no-lossy-flatten rule).
|
||||
|
||||
---
|
||||
|
||||
## 9. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-49**. New UCs **UC-50–UC-52** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a block-graph DB wiki (Roam-style) as a shard via its query/CRUD API, mapping blocks to the page model** (`:node/title` node = page; nested blocks = spans) | **UC-50 (new)** |
|
||||
| **Adopt a shard's native block/span IDs as portable span addresses** for transclusion/overlay (the shipped answer to the span-address question) | **UC-51 (new)** |
|
||||
| **Delegate derived views to a shard's native query engine** (Datalog) where advertised | **UC-52 (new)** |
|
||||
| Shipped transclusion via block embeds + shipped fine-grained addressing | **enriches UC-32** (and UC-44/45) |
|
||||
| Attributes + Datalog = shipped structured/typed pages | **enriches UC-34** (and UC-39) |
|
||||
| Roam Depot `onload/onunload` + `roamAlphaAPI` = modern engine-hosts-adapter template | **enriches UC-38** |
|
||||
| Block-level write granularity = the fine extreme opposite TiddlyWiki | **enriches UC-35** |
|
||||
| Hosted history is internal-only, not portable Git | links **UC-36** |
|
||||
|
||||
---
|
||||
|
||||
## 10. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- The adapter contract should model **"native span/block IDs"** as a capability: a shard
|
||||
advertises whether it mints stable sub-page addresses; transclusion/overlay/reverse-
|
||||
lookup capabilities key off it (ties UC-44/45/46/51).
|
||||
- The contract should model a **"native query"** capability so the orchestrator can
|
||||
delegate derived-view computation (UC-52) vs. scanning the projection.
|
||||
- A **block↔page mapping** belongs in the adapter, with the no-lossy-flatten rule of
|
||||
UC-34: page = titled node, blocks = addressable spans, attributes = sidecar metadata.
|
||||
- Roam joins XWiki as a **DB-backed / API-attached** exemplar (vs. TWiki/Foswiki file
|
||||
stores) and as a **second engine-hosts-adapter** template (vs. XWiki components / TWiki
|
||||
handlers) for the T14 adapter-binding task.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions (for spec / workplans)
|
||||
|
||||
1. When a backend mints native span IDs (Roam UIDs), are they used **directly** as the
|
||||
shard-wiki span address, or **wrapped** in a shard-scoped address so they survive
|
||||
projection/overlay and don't collide across shards?
|
||||
2. For a **write-through** Roam shard the adapter must run *inside* Roam (Depot
|
||||
extension). Is in-engine hosting an accepted adapter deployment mode generally, or do
|
||||
we restrict Roam to **read/projection/overlay-target** (graceful degradation) when we
|
||||
cannot deploy inside it?
|
||||
3. How lossy is the **block-outline ↔ Markdown-page** mapping, and is round-trip
|
||||
fidelity required or best-effort (cf. UC-42 Foswiki TML↔HTML)?
|
||||
4. Do we consume Roam's **export (EDN/JSON)** as the projection source, or the live
|
||||
`roamAlphaAPI`? Snapshot vs. live freshness trade-off (cf. UC-31).
|
||||
|
||||
---
|
||||
|
||||
## 12. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| zsolt.blog — "Deep Dive Into Roam's Data Structure" (https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html) | Datom/EAV model, block attributes (`:block/uid` etc.), refs/embeds, linked vs unlinked references, Datalog/pull |
|
||||
| Roam Research Datalog Cheatsheet (https://gist.github.com/2b3pro/231e4f230ed41e3f52e8a89ebf49848b) | Attribute names, `:q`/`:pull` query shapes |
|
||||
| GitHub — Roam-Research/roam-depot (https://github.com/Roam-Research/roam-depot) | Extension repo/manifest structure, `extension.js` `onload`/`onunload`, `extensionAPI` settings, `roamAlphaAPI` read/write/ui surface |
|
||||
| "Introduction to the Roam Alpha API" (https://www.putyourleftfoot.in/introduction-to-the-roam-alpha-api) | `window.roamAlphaAPI` `q`/`pull` semantics, global-scope dependency |
|
||||
| David Vargas — "How To Create Your Own Roam Extensions" (https://davidvargas.me/blog/how_to_create_your_own_roam_extensions) | Extension authoring lifecycle, write methods |
|
||||
| Wikipedia / general — Roam Research, DataScript, Datomic | Datalog/Datomic lineage, client-side DB framing |
|
||||
|
||||
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (span addressing,
|
||||
transclusion), `research/260614-zigzag-deep-dive/findings.md` (queryable dimensions),
|
||||
`research/260613-xwiki-deep-dive/findings.md` (DB-backed engine + extension host),
|
||||
`spec/UseCaseCatalog.md` (UC-05, UC-18, UC-32, UC-34/35/36/38/39, UC-44/45/46),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T14 adapter binding).
|
||||
|
||||
---
|
||||
|
||||
## 13. Traceability
|
||||
|
||||
- New UCs: **UC-50, UC-51, UC-52** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-32, UC-34, UC-35, UC-38** (and links UC-36, UC-39, UC-44/45/46).
|
||||
- Architecture (no UC): native-span-ID capability; native-query capability; block↔page
|
||||
mapping; Roam as DB-backed/API-attached + engine-hosts-adapter exemplar → `SHARD-WP-0002` (T14).
|
||||
- Boundary recorded: Roam is **one candidate shard**, mapped into the Markdown-first
|
||||
page model; not a substrate, not the federation layer (INTENT Stability Note).
|
||||
</content>
|
||||
@@ -18,4 +18,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and
|
||||
| 2026-06-13 | `260613-twiki-deep-dive/` | TWiki impl, plugin API, ecosystem; UC-40/41 |
|
||||
| 2026-06-13 | `260613-foswiki-deep-dive/` | Foswiki store abstraction, extension API; UC-42/43 |
|
||||
| 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 |
|
||||
| 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 |
|
||||
| 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 |
|
||||
| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 |
|
||||
@@ -6,7 +6,8 @@ Promoted from `research/260608-c2-wiki-origins/`,
|
||||
`research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`,
|
||||
`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`,
|
||||
`research/260613-twiki-deep-dive/`, `research/260613-foswiki-deep-dive/`,
|
||||
`research/260614-xanadu-deep-dive/`, and `research/260614-zigzag-deep-dive/`.
|
||||
`research/260614-xanadu-deep-dive/`, `research/260614-zigzag-deep-dive/`, and
|
||||
`research/260614-roam-deep-dive/`.
|
||||
See InfoTechPrimers on coulomb.social for use-case catalog conventions.
|
||||
|
||||
## Conventions
|
||||
@@ -188,7 +189,11 @@ projection. Provenance and staleness must be explicit. Xanadu deep dive sharpens
|
||||
transclusion should be **content-identity based and bidirectional** (content is "knowably
|
||||
in more than one place" and aware of its appearances), not a one-way path fetch —
|
||||
enables UC-45 reverse lookup (`research/260614-xanadu-deep-dive/findings.md` §4). See
|
||||
UC-44 for the whole-page composition-manifest form.
|
||||
UC-44 for the whole-page composition-manifest form. **Roam ships this**: block embeds
|
||||
transclude by `:block/uid` live (not copied), proving transclusion is a cheap data-layer
|
||||
capability over an addressable store (`research/260614-roam-deep-dive/findings.md` §3,
|
||||
§7) — argues for transclusion in core over the union, surfaced by UI. Depends on span
|
||||
addressing (UC-51).
|
||||
**Priority:** Later
|
||||
|
||||
### UC-33 — Git-branch an information space
|
||||
@@ -217,7 +222,10 @@ XObjects against an XClass schema (`research/260613-xwiki-deep-dive/findings.md`
|
||||
§2.3); UC-39 covers the extreme where the page is *only* structure. TWiki shows the
|
||||
**git-friendly** variant: TWiki Forms store fields as `%META:FIELD%` *inside the
|
||||
topic text file*, so structure is diffable rather than locked in a DB
|
||||
(`research/260613-twiki-deep-dive/findings.md` §2.3).
|
||||
(`research/260613-twiki-deep-dive/findings.md` §2.3). Roam is the modern exemplar:
|
||||
`key:: value` attributes over a datom graph, queryable via Datalog
|
||||
(`research/260614-roam-deep-dive/findings.md` §3, §4) — structure to preserve via
|
||||
sidecar metadata, and a candidate for query delegation (UC-52).
|
||||
**Priority:** Later
|
||||
|
||||
### UC-35 — Attach a shard with coarse write granularity
|
||||
@@ -229,7 +237,10 @@ respect that granularity instead of assuming per-page writes.
|
||||
**Source:** wikiengines, intent
|
||||
**Notes:** Capability-aware adapters: the **capability profile must encode write
|
||||
granularity** (per-page / per-file / per-space / append-only). Overlay and
|
||||
patch flows must adapt (findings §5 #1). Affects conflict scope and locking.
|
||||
patch flows must adapt (findings §5 #1). Affects conflict scope and locking. Roam marks
|
||||
the **fine extreme** — block-level writes (`block.create/update/move/delete`), the
|
||||
opposite of TiddlyWiki's whole-file granularity
|
||||
(`research/260614-roam-deep-dive/findings.md` §6).
|
||||
**Priority:** Later
|
||||
|
||||
### UC-36 — Supply a git-addressable history to an internal-history engine
|
||||
@@ -278,7 +289,11 @@ surfacing — enough to host a high-fidelity adapter
|
||||
adapter (needs deploy access, higher fidelity) is open (findings §8 Q3).
|
||||
Generalizes beyond XWiki: TWiki's plugin handler API (`beforeSaveHandler`,
|
||||
`afterRenameHandler`, REST handlers) is an equivalent host surface
|
||||
(`research/260613-twiki-deep-dive/findings.md` §3).
|
||||
(`research/260613-twiki-deep-dive/findings.md` §3). **Roam Depot** is the modern
|
||||
template: an extension default-exports `onload`/`onunload` and drives
|
||||
`window.roamAlphaAPI` read (`q`/`pull`) + write (`block.*`/`page.create`)
|
||||
(`research/260614-roam-deep-dive/findings.md` §5) — note Roam's API runs *inside* the
|
||||
client, so write-through requires in-engine hosting (findings §11 Q2).
|
||||
**Priority:** Later
|
||||
|
||||
### UC-39 — Attach a wiki-as-application-platform shard (pages as typed records)
|
||||
@@ -443,6 +458,51 @@ genealogy edge recorded at fork/remix/import time (UC-26, UC-29) in the coordina
|
||||
journal. Makes provenance a traversal, not a buried footer (complements UC-24).
|
||||
**Priority:** Later
|
||||
|
||||
### UC-50 — Attach a block-graph database wiki as a shard, mapping blocks to the page model
|
||||
|
||||
**Actor:** Maintainer
|
||||
**Goal:** Attach a block-first, DB-backed graph tool (Roam-style) as a shard via its
|
||||
query/CRUD API, mapping its blocks onto shard-wiki's Markdown page model without
|
||||
flattening the structure.
|
||||
**Source:** roam, intent
|
||||
**Notes:** Roam's graph is a DataScript datom DB where a *page* is the node carrying
|
||||
`:node/title` and nested *blocks* are addressable spans
|
||||
(`research/260614-roam-deep-dive/findings.md` §2, §6). Clean mapping rule: titled node
|
||||
= page, nested blocks = spans, attributes = sidecar metadata (honors UC-34 no-lossy-
|
||||
flatten). DB-backed/API-attached path (like XWiki UC-38), **not** the file-store path
|
||||
(UC-40). Write-through requires hosting the adapter inside the engine (Roam Depot),
|
||||
else degrade to read/projection/overlay-target (findings §11 Q2).
|
||||
**Priority:** Later
|
||||
|
||||
### UC-51 — Adopt a shard's native span IDs as portable span addresses
|
||||
|
||||
**Actor:** Orchestrator / adapter
|
||||
**Goal:** Where a backend mints stable sub-page identifiers (Roam `:block/uid`), adopt
|
||||
them as the span address for transclusion, overlay, and reverse-lookup, rather than
|
||||
inventing one.
|
||||
**Source:** roam, xanadu, intent
|
||||
**Notes:** Roam ships the fine-grained stable address that Xanadu's tumblers
|
||||
(`research/260614-xanadu-deep-dive/findings.md` §3) and the catalog left open — a short
|
||||
opaque per-block UID, public and referenceable (`research/260614-roam-deep-dive/findings.md`
|
||||
§2, §7). Enables UC-44/45 at sub-page granularity; falls back to content-fingerprint
|
||||
(UC-46) or path+range where no native ID exists. Open: use native ID directly or wrap in
|
||||
a shard-scoped address to avoid cross-shard collision and survive projection (findings
|
||||
§11 Q1).
|
||||
**Priority:** Later
|
||||
|
||||
### UC-52 — Delegate derived views to a shard's native query engine
|
||||
|
||||
**Actor:** Orchestrator
|
||||
**Goal:** When a shard exposes a native query engine, compute derived views (backlinks,
|
||||
recent changes, typed queries) by delegating to it instead of scanning the projection.
|
||||
**Source:** roam, intent
|
||||
**Notes:** In Roam, derived views *are* Datalog queries over the datom graph
|
||||
(`research/260614-roam-deep-dive/findings.md` §4). Makes "native query" an adapter
|
||||
capability the orchestrator can key off (vs. computing views itself over the
|
||||
projection). Operationalizes the ZigZag "dimensions + rasters" insight
|
||||
(`research/260614-zigzag-deep-dive/findings.md` §5) and relates to UC-05/UC-34.
|
||||
**Priority:** Later
|
||||
|
||||
---
|
||||
|
||||
## B. Knowledge work and collaboration
|
||||
@@ -678,6 +738,9 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD.
|
||||
| UC-47 | | | ‡ | | ✓ |
|
||||
| UC-48 | | | ‡ | | |
|
||||
| UC-49 | | | ‡ | | ✓ |
|
||||
| UC-50 | | | | § | ✓ |
|
||||
| UC-51 | | | | § | ✓ |
|
||||
| UC-52 | | | | § | ✓ |
|
||||
| UC-08 | ✓ | | |
|
||||
| UC-09 | ✓ | | |
|
||||
| UC-10 | ✓ | | |
|
||||
@@ -823,6 +886,36 @@ Architecture logged for `SHARD-WP-0002`: a dimensional projection layer; genealo
|
||||
edges in the coordination journal; clone↔transclusion as a possible shared primitive
|
||||
(findings §9). Pairs with the Xanadu dive (clone ↔ transclusion convergence).
|
||||
|
||||
### roam mapping
|
||||
|
||||
(§ UC-50–UC-52 are placed in the **wikiengines** matrix column as the nearest existing
|
||||
source — Roam is a shipped tool — but their true lineage is the **Roam deep dive**,
|
||||
`research/260614-roam-deep-dive/findings.md`.)
|
||||
|
||||
| Roam mechanism (findings §) | Catalog UC |
|
||||
|-----------------------------|------------|
|
||||
| Block-graph DataScript DB; page = `:node/title` node, blocks = spans (§2, §6) | UC-50 |
|
||||
| `:block/uid` — shipped stable sub-page address (§2, §7) | UC-51 |
|
||||
| Derived views = Datalog queries over the graph (§4) | UC-52 |
|
||||
| Block embeds = shipped live transclusion by uid (§3) | UC-32 (enriched; + UC-44/45) |
|
||||
| `key:: value` attributes + Datalog = structured/typed pages (§3, §4) | UC-34 (enriched; + UC-39) |
|
||||
| Roam Depot `onload/onunload` + `roamAlphaAPI` = engine-hosts-adapter template (§5) | UC-38 (enriched) |
|
||||
| Block-level write granularity = the fine extreme (§6) | UC-35 (enriched) |
|
||||
| Hosted history is internal-only, not portable Git (§6, §8.2) | links UC-36 |
|
||||
|
||||
Note: Roam is the **modern bookend** to the Nelson dives — where Xanadu and ZigZag are
|
||||
unbuilt ideals, Roam *shipped* fine-grained addressing, transclusion, bidirectional
|
||||
links, and a queryable structured space. Its payoff is **empirical evidence** on
|
||||
shard-wiki's open questions (span addressing is tractable via native block IDs;
|
||||
transclusion is a data-layer capability; derived views are queries —
|
||||
`research/260614-roam-deep-dive/findings.md` §7). **Boundary recorded:** Roam is *one
|
||||
candidate shard* — DB-backed / API-attached (the XWiki family, not the file-store
|
||||
family), block-first, with no portable Git history — **mapped into** shard-wiki's
|
||||
Markdown-first page model, not adopted as a substrate and never the federation layer
|
||||
(findings §8.2). Architecture logged for `SHARD-WP-0002` (T14): native-span-ID and
|
||||
native-query capabilities, block↔page mapping, and Roam as a second
|
||||
engine-hosts-adapter exemplar alongside XWiki.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
@@ -840,4 +933,7 @@ edges in the coordination journal; clone↔transclusion as a possible shared pri
|
||||
7. Is the **dimensional/zzstructure model** (UC-47/48) a public navigation API and UI
|
||||
paradigm, or only an internal organizing concept for the derived views?
|
||||
8. How is the **many-to-many link graph** reconciled with the rank-based dimensional
|
||||
model — clones, a parallel graph index, or both? (ZigZag dive §10.)
|
||||
model — clones, a parallel graph index, or both? (ZigZag dive §10.)
|
||||
9. Is **in-engine adapter hosting** (e.g. a Roam Depot extension for write-through) an
|
||||
accepted deployment mode generally, or do we restrict API-only backends to
|
||||
read/projection/overlay-target? (Roam dive §11 Q2; ties UC-38, UC-50.)
|
||||
Reference in New Issue
Block a user