coulomb/shard-wiki
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research: Obsidian deep dive (file-over-app vaults, plugin API, ecosystem-popularity signal); UC-53/54/55/56

Browse Source 
The most INTENT-aligned tool yet and the file-backed counterpart to Roam:
file-over-app vaults (plain .md folders, files canonical, MetadataCache a
derived index), in-file git-diffable addressing/structure (^block-id,
wikilink embeds, YAML frontmatter), and a plugin API (Plugin onload/onunload
over App.vault/metadataCache/workspace) that doubles as an adapter host —
so a vault is dual-attachable (file-store direct or in-app plugin). Mined
the plugin download rankings as demand evidence per the research brief:
#1 is drawings (Excalidraw, non-Markdown content), query-as-DB
(Dataview/Tasks) is top-tier but an add-on, Git is top-7 (bolt-on history),
Remotely Save shows sync-to-anywhere demand. Added UC-53 (attach local
vault w/ live concurrent native editor), UC-54 (query-defined dynamic
page), UC-55 (non-Markdown content types), UC-56 (outbound publish of a
projection); enriched UC-15/28/34/36/40/51/52. Boundary: a vault is one
file-backed candidate shard, not the federation layer and not a file-sync
target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-06-14 12:51:22 +02:00
parent dfff9ab42e
commit f0be5799aa
5 changed files with 518 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
SCOPE.md
View File

@@ -19,9 +19,9 @@ Learnings update both SCOPE and INTENT where necessary.
|-------|-------| |-------|-------|
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) | | Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
| Intent | `INTENT.md` established; authorization-in-core amendments drafted | | Intent | `INTENT.md` established; authorization-in-core amendments drafted |
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam deep dives (`research/260614-*/`) | | Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian deep dives (`research/260614-*/`) |
| Demand | NetKingdom integration asks captured, not yet negotiated | | Demand | NetKingdom integration asks captured, not yet negotiated |
| Spec | Architecture blueprint drafted; UseCaseCatalog 52 UCs from research; PRD/TSD scaffolds | | Spec | Architecture blueprint drafted; UseCaseCatalog 56 UCs from research; PRD/TSD scaffolds |
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) | | Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) |
## In Scope (today) ## In Scope (today)

51
research/260614-obsidian-deep-dive/README.md Normal file
View File

@@ -0,0 +1,51 @@
# 260614 — Obsidian deep dive (file-over-app vaults, plugin API, ecosystem signal)
Date: 2026-06-14
## What this is
A focused study of **Obsidian** — core/extension architecture and the plugin
ecosystem — read through shard-wiki's lens. Obsidian is the **most INTENT-aligned tool
in the research set**: INTENT names Obsidian vaults as a shard participant, and its
**"file over app"** model (notes are plain Markdown files in a folder you own; the app
is a lens) is the closest cousin to shard-wiki's Markdown-first, sovereignty-preserving
thesis. It is the **file-backed counterpart to the Roam dive** (same tool category,
opposite storage model: Roam = client DB + API; Obsidian = files on disk).
Distinctive material:
- **Architecture** — vault = folder of `.md` + `.obsidian/` config; files canonical;
**MetadataCache** is a *derived*, async-rebuilt index (links, embeds, tags,
frontmatter, block IDs, resolved/unresolved links, backlinks)
- **In-file, git-diffable** addressing/structure — `[[wikilinks]]`, `![[embeds]]`
(transclusion), `^block-id` span refs, YAML frontmatter/properties
- **Plugin API** — `Plugin` `onload/onunload`; `App` modules `vault` (CRUD + file
events), `metadataCache`, `workspace` (CM6, views), `fileManager.renameFile`;
`manifest.json`; community plugins, Restricted Mode
- **Ecosystem popularity → UC signal** (§7) — ranked download data mined for use cases,
per the research brief
## Contents
| Path | Role |
|------|------|
| `findings.md` | File-over-app architecture, in-file addressing, plugin API as adapter host, capability profile, **plugin-popularity → UC mapping**, derived-index lesson, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Four new use cases promoted to `spec/UseCaseCatalog.md`
(UC-53 attach a local vault with a live concurrent native editor, UC-54 query-defined
dynamic page, UC-55 carry non-Markdown content types, UC-56 outbound publish of a
projection); UC-15/28/34/36/40/51/52 enriched. Logged for `SHARD-WP-0002` (T14):
**dual attachment mode** (file-store *or* in-app plugin host), consume-native-derived-
index capability, non-Markdown content types in the page model, outbound publish,
external-writer tolerance.
**Key takeaways recorded:** (1) Obsidian validates *files-canonical / index-derived* —
shard-wiki's projection model. (2) Fine-grained addressing & structure can be portable
**in-file text** (`^block-id`, frontmatter), not DB state (contrast Roam). (3) The
ecosystem is demand evidence — the #1 plugin is **drawings** (non-Markdown content),
query-as-DB (Dataview/Tasks) is top-tier but an *add-on*, and **Git is top-7** (users
bolt on the portable history shard-wiki provides natively). **Boundary:** a vault is one
file-backed candidate shard, not the federation layer and not a file-sync target
(not-a-sync-daemon).
</content>

324
research/260614-obsidian-deep-dive/findings.md Normal file
View File

@@ -0,0 +1,324 @@
# Findings — Obsidian: file-over-app vaults, the plugin API, and what the ecosystem reveals
Date: 2026-06-14
Source kind: **modern shipped product** — a local-first Markdown vault tool; the
*most INTENT-aligned candidate shard yet* (file-backed, Markdown-first); plus an
ecosystem whose popularity is direct evidence of what users actually do
Lens: shard-wiki — file-store attachment, in-file fine-grained addressing, derived
index vs. canonical files, the engine-hosts-adapter path, and ecosystem-driven UCs
> Why Obsidian matters most. INTENT names **Obsidian vaults** explicitly as a shard
> participant, and Obsidian's **"file over app"** philosophy — your notes are plain
> Markdown files in a folder you own, the app is just a lens — is the closest cousin
> to shard-wiki's own Markdown-first, sovereignty-preserving thesis. It is also the
> **file-backed counterpart to the Roam dive**: same "personal knowledge tool"
> category, opposite storage model (Roam = client DB + API; Obsidian = files on disk).
> And uniquely among the dives, Obsidian has a **large, ranked plugin ecosystem** — so
> the user's instruction to let *plugin popularity inform the use-case catalog* is
> actionable: what people install tells us what a federated wiki must support (§7).
This dive treats Obsidian as a candidate shard (capability profile §5), studies its
plugin API as an adapter-host surface (§4), and mines the download rankings for UCs
(§7). Pairs with the Roam dive (the DB/API contrast) and the TWiki/Foswiki dives (the
file-store attachment path, UC-40).
---
## 1. Core architecture — file over app
- A **vault** is a local folder of plain **Markdown files** plus a `.obsidian/` config
directory (settings, enabled plugins, themes, workspace layout). **The files are the
source of truth**; the app holds no separate canonical database.
- Markdown is **Obsidian-flavored**: CommonMark + `[[wikilinks]]`, `![[embeds]]`
(transclusion), `^block-id` block references, `#tags`, YAML **frontmatter /
properties** (typed metadata), and callouts. Note: all addressing and structure live
**in the file text** — so they are git-diffable and portable.
- **MetadataCache** is a *derived* index: Obsidian pre-parses every file into cached
metadata (headings, links, embeds, tags, frontmatter, **blocks/block IDs**,
`resolvedLinks` / `unresolvedLinks`, backlinks). It is **rebuilt from files** and
updated **asynchronously** ("don't rely on immediate updates after a modify"). Files
canonical; index derived.
- **Canvas** files (`.canvas`, the open **JSON Canvas** format) hold spatial
arrangements — a non-Markdown content type living in the vault.
- Sync is *not* built into the format: users add Git, Obsidian Sync, or a file-sync
plugin (see §7). The vault is just files.
For shard-wiki this is the **ideal file-backed shard**: Markdown-first, on disk,
git-friendly, with a derived index model that mirrors shard-wiki's own
projection/derived-view philosophy (the cache is to Obsidian what projections and union
BackLinks are to shard-wiki — computed, not canonical).
---
## 2. Addressing, links, transclusion — in-file and git-diffable
- **Links:** `[[Page]]` and `[[Page#Heading]]` / `[[Page#^block-id]]`; the resolved/
unresolved link sets and **backlinks** are computed in MetadataCache — Obsidian's
shipped answer to union BackLinks (UC-05/UC-18), but per-vault.
- **Fine-grained addressing:** a **`^block-id`** suffix on a paragraph (and heading
anchors) gives a stable sub-page address — *but it lives in the Markdown text*, not in
a database. This is the **git-diffable, portable variant** of Roam's `:block/uid`
(which lives in a DataScript DB). Important contrast for UC-51: native span IDs can be
**text-embedded** (Obsidian) or **store-minted** (Roam); the text-embedded form is
more portable and survives a file copy, but is opt-in (the user must add the `^id`).
- **Transclusion:** `![[Page]]`, `![[Page#Heading]]`, `![[Page#^block-id]]` embed live
content by reference — shipped transclusion at page/section/block granularity (UC-32),
again stored as plain text.
---
## 3. Structured data — frontmatter as git-diffable records
YAML **frontmatter / properties** make each note a lightweight typed record
(`tags`, `aliases`, arbitrary keys). This is the **git-diffable structured-data**
variant (cf. TWiki `%META%` inside the file, UC-34) — structure lives in the file, not a
DB. The **Dataview** plugin (§7) turns frontmatter + inline fields + the link graph into
a queryable database — but that query power is an *ecosystem add-on*, not core.
---
## 4. Extension architecture — the plugin API as an adapter host
Obsidian plugins are TypeScript, shipped as `manifest.json` + `main.js` (+ optional
`styles.css`):
- **Manifest:** `id`, `name`, `version`, `minAppVersion` (required); `author`,
`description`, `isDesktopOnly`, `fundingUrl` (optional). Community plugins are
GitHub-hosted and listed in the `obsidian-releases` repo; installed from inside the
app; gated by **Restricted Mode** (plugins run arbitrary code — Node/Electron on
desktop).
- **Plugin lifecycle:** extend `Plugin`, implement **`onload()` / `onunload()`**; use
`registerEvent` / `registerDomEvent` / `registerInterval` for auto-cleanup.
- **The `App` object** (global singleton, `this.app`) exposes four modules:
- **`vault`** — file CRUD: `read` / `cachedRead`, `create`, `modify`, `process`,
`delete`, `rename`, `getFiles`, low-level `adapter`, and **events**
(`on('create'|'modify'|'delete'|'rename')`) for live file-watching.
- **`metadataCache`** — `getFileCache`, `frontmatter`, `links`, `embeds`, `tags`,
`headings`, `blocks`, `resolvedLinks` / `unresolvedLinks`, `getBacklinksForFile`.
- **`workspace`** — panes/leaves/views; `registerView`, `registerEditorExtension`
(CodeMirror 6), `registerMarkdownPostProcessor`.
- **`fileManager`** — high-level ops; **`renameFile`** updates inbound links (use it
instead of raw vault rename to keep link integrity).
- **Registrations:** `addCommand`, `addRibbonIcon`, `addStatusBarItem`,
`addSettingTab`, `registerView`, `registerEditorExtension`,
`registerMarkdownPostProcessor`.
Two consequences for shard-wiki:
1. **A vault is attachable two ways.** (a) **File-store direct attach** (UC-40) — read
the folder as a folder shard; no plugin, offline-capable, git-native; good for
read/projection/overlay. (b) **In-app plugin host** (UC-38) — a shard-wiki adapter
*as an Obsidian plugin* drives `vault` write + `metadataCache` reads + live file
events for high-fidelity write-through. This dual mode is a cleaner story than Roam
(which only offers the in-app path).
2. The **MetadataCache API is a ready-made derived-view source** — backlinks, tags,
block IDs, resolved links — an adapter can consume instead of reparsing.
---
## 5. Obsidian as a shard — capability profile
| Capability | Obsidian | Notes for the adapter contract |
|------------|----------|--------------------------------|
| Read | **yes** | direct file read (folder shard) or `vault.read`/`cachedRead` in-app |
| Write | **yes** | direct file write, or `vault.modify`/`create`/`process` in-app; per-file granularity |
| Write granularity | **per-file (page)** | the natural shard-wiki granularity — contrast Roam (block) and TiddlyWiki (whole-file-single-vault) (UC-35) |
| Identity / addressing | **path + in-file `^block-id` / headings** | git-diffable, portable, **opt-in** span IDs (UC-51 text-embedded variant) |
| Transclusion | **yes** | `![[...]]` page/section/block embeds (UC-32) |
| Backlinks / links | **yes (derived)** | MetadataCache resolved/unresolved links + backlinks (UC-05/18) |
| Structured data | **yes (in-file)** | YAML frontmatter/properties; queryable via Dataview plugin (UC-34) |
| Native query | **plugin only** | Dataview — *not core*; informs UC-52 (query is adapter/plugin-provided, not assumable) |
| Version history | **none native** | users add the Git plugin (top-7!) → validates the coordination journal (UC-36) |
| Diff / merge | **none native** | git-level if Git plugin / repo-backed |
| Lock | **no** | local-first, single-user assumption |
| Publish | **via Obsidian Publish / Quartz / Digital Garden** | outbound publish of a projection (UC-56) |
| Syntax | **Obsidian-flavored Markdown** | close to CommonMark; wikilinks/embeds/callouts/`^id` need adapter awareness (cf. UC-42) |
| Non-Markdown content | **Canvas (JSON Canvas), attachments, Excalidraw** | typed/opaque assets in the vault (UC-55) |
Verdict: Obsidian is the **cleanest file-backed, Markdown-first shard** — the
file-store family (TWiki/Foswiki/DokuWiki, UC-40), and the *reference* personal-vault
shard INTENT already names. Its history/query/sync gaps are exactly what users fill with
plugins (§7) — i.e. exactly what shard-wiki offers as orchestration.
---
## 6. The derived-index lesson (architecture)
Obsidian's **files-canonical / MetadataCache-derived** split is independent validation
of shard-wiki's core stance: the link graph, backlinks, tags, and block index are
**computed projections over canonical files**, rebuilt on change, eventually consistent
("updated asynchronously"). shard-wiki's union BackLinks, projections, and derived views
(UC-05, UC-17–UC-20) should likewise be **derived and rebuildable**, never a second
source of truth. The cache-invalidation/async-update caveat is a real design note for
projection freshness (UC-31).
---
## 7. Ecosystem popularity → use-case signal (the user's ask)
All-time download ranks (obsidianstats.com), read as *demand evidence*:
| Plugin (downloads) | What users do with a vault | shard-wiki signal → UC |
|--------------------|----------------------------|------------------------|
| **Excalidraw** (6.4M) | drawings/diagrams as first-class content | non-Markdown content types in a shard → **UC-55 (new)** |
| **Templater** (4.6M), **QuickAdd** (1.9M) | templated note creation | blueprint pages → **UC-15 (enriched)** |
| **Dataview** (4.4M) | query the vault (files+frontmatter+links) as a DB | query-defined dynamic pages → **UC-54 (new)**; query is a *plugin* → **UC-52 (enriched)** |
| **Tasks** (3.6M) | aggregate to-dos across all notes | cross-page typed-item aggregation → **UC-54** |
| **Advanced Tables** (2.9M) | structured tables in Markdown | in-file structured data → UC-34 |
| **Calendar** (2.8M) | daily-note navigation | temporal dimension (ZigZag `d.recent`) → UC-17 |
| **Git** (2.7M, top-7) | version control + sync on the vault | **history is a bolt-on, not native** → validates coordination journal → **UC-36 (enriched)** |
| **Kanban** (2.4M) | board view of Markdown cards | alternate projection/view of pages → UC-47/48 |
| **Remotely Save** (2.0M) | sync vault to S3 / Dropbox / OneDrive / GDrive / **WebDAV** | demand to federate a vault to heterogeneous remote stores → reinforces INTENT WebDAV/Nextcloud shards (but shard-wiki is *not* a file-sync daemon — attach as shards, don't mirror) |
| **Omnisearch** (1.6M) | search across notes, PDFs, images | union full-text search over mixed content → UC-20 |
| **Importer** (1.4M) | import from Evernote / Notion / Bear / Apple Notes | carry-forward/import from foreign tools → **UC-28 (enriched)** |
| **Smart Connections** (1.0M), **Copilot** (1.5M), **Claudian** (938K) | AI over the vault (related notes, chat) | mostly out of scope; "related notes" ≈ link/equivalence discovery (UC-46) — note the trend, don't build it into core |
| **Tag Wrangler** (981K) | rename/merge tags vault-wide | namespace/tag refactor across the union → UC-22 |
| **Linter** (927K) | normalize frontmatter/formatting | content normalization → fingerprint basis for equivalence (UC-46) |
| **Admonition** (922K) | callouts | Obsidian-flavored syntax → translation awareness (UC-42) |
Headline reads:
- The **#1 plugin is drawings** (Excalidraw) — users keep **non-Markdown content** in
"Markdown" vaults. A wiki orchestrator that assumes pure Markdown will mishandle the
most popular real-world use → **UC-55**.
- **Query-the-vault-as-a-DB** (Dataview #3 + Tasks #4) is a top use — but it is an
*add-on*, confirming query is an **adapter/plugin capability** (UC-52), and motivating
**query-defined dynamic pages** (UC-54).
- **Git is top-7** — users manually bolt portable version control onto their vault. This
is *direct demand* for what shard-wiki provides natively as the coordination journal
(UC-36).
- **Sync-to-anywhere** (Remotely Save #11) shows demand to connect a vault to S3/WebDAV/
cloud — the heterogeneous backends INTENT targets — while reminding us of the
*not-a-file-sync-daemon* boundary.
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **File-over-app == shard sovereignty + Markdown-first.** Obsidian is the living proof
that a serious tool can be "just files you own"; it is the model shard for INTENT's
Obsidian/local-folder participants.
- **Files canonical, index derived** validates shard-wiki's projection/derived-view
architecture (§6).
- **In-file, git-diffable structure & addressing** (frontmatter, `^block-id`, embeds)
shows fine-grained addressing and structured data can be **portable text**, not DB
state — friendlier to the coordination journal than Roam's DB.
### 8.2 Deliberate divergences (design bugs if conflated)
1. **A vault is one shard, not the federation.** Local-first, single-vault,
single-user; do not model the union as "one big vault."
2. **Not a file-sync daemon.** The Remotely Save popularity is tempting; shard-wiki must
stay **wiki-page-semantic** — attach remote stores as shards, never generic file
mirroring (INTENT constraint).
3. **`.obsidian/` is opaque app config, not page content.** The adapter must exclude/
treat it as shard-local config, not project it as pages.
4. **Obsidian-flavored Markdown ≠ CommonMark.** Wikilinks/embeds/callouts/`^id` need
adapter awareness; closer than Roam's outline, but still a translation surface
(UC-42).
5. **Plugins run arbitrary code.** If a shard-wiki adapter is hosted *as* an Obsidian
plugin (UC-38 path), it inherits Electron/Node trust — a deployment/security note,
not a core concern.
### 8.3 What Obsidian teaches that shard-wiki should keep
- Offer **both attachment modes** for a vault: zero-config **file-store direct attach**
(read/projection/overlay) and an optional **in-app adapter** (write-through, live
events). Graceful degradation by default, fidelity on opt-in.
- **Consume a backend's existing derived index** (MetadataCache) instead of reparsing,
when the shard exposes one.
- **Embrace non-Markdown content types** (Canvas/attachments/drawings) as typed or
opaque assets with provenance — don't flatten them away (UC-55, extends UC-34).
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-52**. New UCs **UC-53–UC-56** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a local-first Markdown vault as a file-backed shard with a live concurrent native editor** — file-watch, tolerate the vault's own app writing concurrently, treat `.obsidian/` as opaque | **UC-53 (new)** |
| **Define a page as a live query over the union** (Dataview/Tasks pattern — query-defined dynamic page) | **UC-54 (new)** |
| **Carry non-Markdown content types** (Canvas/JSON Canvas, drawings, attachments) as typed/opaque assets with provenance | **UC-55 (new)** |
| **Publish a curated projection of the union or a shard to an external read-only target** (Obsidian Publish / Quartz / Digital Garden) | **UC-56 (new)** |
| Local Markdown vault = cleanest file-backed direct-attach; dual attach (file-store *or* in-app plugin host) | **enriches UC-40** (and UC-02, UC-38) |
| In-file `^block-id` / heading anchors = git-diffable, portable, opt-in span IDs | **enriches UC-51** |
| YAML frontmatter/properties = git-diffable in-file structured data | **enriches UC-34** |
| Git plugin (top-7) = users bolt on portable history → demand for the coordination journal | **enriches UC-36** |
| Query is a popular plugin (Dataview), not core → query is adapter/plugin-provided | **enriches UC-52** |
| Importer plugin = import from foreign tools | **enriches UC-28** |
| Templater/QuickAdd = templated creation | **enriches UC-15** |
| `![[...]]` embeds = in-file transclusion | links UC-32 |
| MetadataCache backlinks/links | links UC-05/UC-18 |
---
## 10. Architecture notes for SHARD-WP-0002 (no UC)
- **Dual attachment mode** for a single backend (file-store *and* in-engine adapter)
should be first-class in the adapter contract / T14 binding — Obsidian is the clean
example (Roam was in-app-only; TWiki had file-vs-API but not a personal vault).
- **"Consume native derived index"** as a capability: a shard may expose its own link/
backlink/block index (Obsidian MetadataCache, Roam Datalog) the orchestrator can read
instead of reparsing (ties UC-52).
- **Non-Markdown content types** (UC-55) push on the wiki page model: pages vs. typed
assets vs. opaque blobs — a page-model spec decision, not just adapter config.
- **Outbound publish** (UC-56) formalizes the `publish` capability from INTENT's list as
a projection target, complementing inbound static-export attach (UC-37).
- Concurrent-native-editor (UC-53) needs the contract to express **external-writer
tolerance** (file-watching, re-projection, conflict-with-live-app) — distinct from
multi-user write conflicts.
---
## 11. Open questions (for spec / workplans)
1. For a vault attached **both** ways (file-store + in-app plugin), which is
authoritative, and how do they reconcile (the in-app adapter sees MetadataCache; the
file-store path sees raw bytes)?
2. Is **UC-54 (query-defined dynamic page)** a core page type, an adapter feature, or a
reference-UI/plugin concern? (Mirrors the Roam/Dataview "views are queries" question
and catalog Q7.)
3. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a
Markdown stub, opaque blob with provenance, or a pluggable content-type registry?
4. Does **outbound publish** (UC-56) belong in core or as a publish-adapter family, and
how does it interact with overlays/projection freshness?
5. How is **concurrent native editing** (UC-53) surfaced — optimistic re-projection,
advisory lock, or overlay-onto-moving-target?
---
## 12. Sources
| Source | Used for |
|--------|----------|
| Obsidian API — App Architecture (https://www.mintlify.com/obsidianmd/obsidian-api/concepts/app-architecture) | App singleton, four modules, this.app |
| obsidianmd/obsidian-api README (https://github.com/obsidianmd/obsidian-api/blob/master/README.md) | Plugin lifecycle, Vault/MetadataCache/Workspace/FileManager APIs, registrations, manifest fields |
| DeepWiki — Vault and File System; MetadataCache and Link Resolution (https://deepwiki.com/obsidianmd/obsidian-api) | Vault CRUD + events; MetadataCache parsed elements, resolved/unresolved links, backlinks |
| obsidianstats.com — Most Downloaded Plugins (https://www.obsidianstats.com/most-downloaded) | All-time download ranks + counts for the popularity → UC mapping (§7) |
| Obsidian — "The future of plugins" (https://obsidian.md/blog/future-of-plugins/) | Plugin distribution model, restricted mode context |
| Obsidian Help / JSON Canvas (jsoncanvas.org) | Vault format, `.obsidian/` config, Canvas open format |
Cross-references: `research/260614-roam-deep-dive/findings.md` (DB/API contrast, native
span IDs, query delegation), `research/260613-twiki-deep-dive/findings.md` (file-store
attach UC-40), `spec/UseCaseCatalog.md` (UC-02, UC-05, UC-15, UC-18, UC-28, UC-31,
UC-32, UC-34, UC-36, UC-38, UC-40, UC-51, UC-52), `workplans/SHARD-WP-0002-federation-architecture.md` (T14).
---
## 13. Traceability
- New UCs: **UC-53, UC-54, UC-55, UC-56** → `spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-15, UC-28, UC-34, UC-36, UC-40, UC-51, UC-52** (and links UC-02,
UC-05, UC-18, UC-32, UC-38).
- Architecture (no UC): dual attachment mode; consume-native-derived-index capability;
non-Markdown content types in the page model; outbound publish; external-writer
tolerance → `SHARD-WP-0002` (T14).
- Distinctive artifact: **plugin-popularity → UC mapping** (§7) — ecosystem demand
evidence, per the research brief.
- Boundary recorded: an Obsidian vault is **one file-backed candidate shard** (the
cleanest, INTENT-named), mapped into the Markdown-first page model; not the federation
layer and not a file-sync target (INTENT Stability Note + not-a-sync-daemon).
</content>

3
research/README.md
View File

@@ -19,4 +19,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and
| 2026-06-13 | `260613-foswiki-deep-dive/` | Foswiki store abstraction, extension API; UC-42/43 | | 2026-06-13 | `260613-foswiki-deep-dive/` | Foswiki store abstraction, extension API; UC-42/43 |
| 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 | | 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 |
| 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 | | 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 |
| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 | | 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 |
| 2026-06-14 | `260614-obsidian-deep-dive/` | Obsidian — file-over-app vaults, plugin API, ecosystem-popularity → UC signal; UC-53/54/55/56 |

149
spec/UseCaseCatalog.md
View File

@@ -6,8 +6,8 @@ Promoted from `research/260608-c2-wiki-origins/`,
`research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`, `research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`,
`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`, `research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`,
`research/260613-twiki-deep-dive/`, `research/260613-foswiki-deep-dive/`, `research/260613-twiki-deep-dive/`, `research/260613-foswiki-deep-dive/`,
`research/260614-xanadu-deep-dive/`, `research/260614-zigzag-deep-dive/`, and `research/260614-xanadu-deep-dive/`, `research/260614-zigzag-deep-dive/`,
`research/260614-roam-deep-dive/`. `research/260614-roam-deep-dive/`, and `research/260614-obsidian-deep-dive/`.
See InfoTechPrimers on coulomb.social for use-case catalog conventions. See InfoTechPrimers on coulomb.social for use-case catalog conventions.
## Conventions ## Conventions
@@ -137,7 +137,9 @@ detection.
read-only, or retired shard into an active information space. read-only, or retired shard into an active information space.
**Source:** federation **Source:** federation
**Notes:** Caulfield bounded conversation / reverse bit-rot. Optional space **Notes:** Caulfield bounded conversation / reverse bit-rot. Optional space
lifecycle policy. Complements UC-02 attach and UC-26 fork at scale. lifecycle policy. Complements UC-02 attach and UC-26 fork at scale. Obsidian's
**Importer** (1.4M) plugin (Evernote/Notion/Bear/Apple Notes → Markdown) shows demand
for importing from foreign tools (`research/260614-obsidian-deep-dive/findings.md` §7).
**Priority:** Later **Priority:** Later
### UC-29 — Remix with portable attribution ### UC-29 — Remix with portable attribution
@@ -225,7 +227,10 @@ topic text file*, so structure is diffable rather than locked in a DB
(`research/260613-twiki-deep-dive/findings.md` §2.3). Roam is the modern exemplar: (`research/260613-twiki-deep-dive/findings.md` §2.3). Roam is the modern exemplar:
`key:: value` attributes over a datom graph, queryable via Datalog `key:: value` attributes over a datom graph, queryable via Datalog
(`research/260614-roam-deep-dive/findings.md` §3, §4) — structure to preserve via (`research/260614-roam-deep-dive/findings.md` §3, §4) — structure to preserve via
sidecar metadata, and a candidate for query delegation (UC-52). sidecar metadata, and a candidate for query delegation (UC-52). Obsidian shows the
**git-diffable in-file** variant: YAML frontmatter/properties live in the Markdown text
and are queried by the Dataview plugin (`research/260614-obsidian-deep-dive/findings.md`
§3) — structure as portable text, not DB state.
**Priority:** Later **Priority:** Later
### UC-35 — Attach a shard with coarse write granularity ### UC-35 — Attach a shard with coarse write granularity
@@ -257,7 +262,10 @@ is the journal authoritative or a mirror reconciled back to the engine
internal RCS table (`xwikircs`) is a concrete instance internal RCS table (`xwikircs`) is a concrete instance
(`research/260613-xwiki-deep-dive/findings.md` §2.4). This UC is *supplementation* (`research/260613-xwiki-deep-dive/findings.md` §2.4). This UC is *supplementation*
(the engine's past is DB-locked); where history is already in an open file format (the engine's past is DB-locked); where history is already in an open file format
(TWiki RCS `.txt,v`) it can instead be **imported** — see UC-41. (TWiki RCS `.txt,v`) it can instead be **imported** — see UC-41. Demand evidence:
**Obsidian Git is a top-7 plugin (2.7M)** — users manually bolt portable git history
onto a vault that has none natively (`research/260614-obsidian-deep-dive/findings.md`
§5, §7), exactly the gap the coordination journal fills as core.
**Priority:** Later **Priority:** Later
### UC-37 — Attach a static engine export as a read-only backup shard ### UC-37 — Attach a static engine export as a read-only backup shard
@@ -328,7 +336,10 @@ the choice (`research/260613-twiki-deep-dive/findings.md` §5–§6, §8 Q1). Di
from UC-37 (no live origin) and broader than UC-02's generic attach. Foswiki's from UC-37 (no live origin) and broader than UC-02's generic attach. Foswiki's
**PlainFile** store (timestamped whole-version copies, no RCS) is a cleaner **PlainFile** store (timestamped whole-version copies, no RCS) is a cleaner
direct-attach target than RCS diffs (`research/260613-foswiki-deep-dive/findings.md` direct-attach target than RCS diffs (`research/260613-foswiki-deep-dive/findings.md`
§2.2). §2.2). An **Obsidian vault** is the cleanest case of all — a plain Markdown folder,
Markdown-first and git-friendly — and uniquely **dual-attachable**: file-store direct
*or* via an in-app plugin adapter for write-through + live file events (UC-53,
`research/260614-obsidian-deep-dive/findings.md` §4, §10).
**Priority:** Later **Priority:** Later
### UC-41 — Import an engine's native file history into the coordination journal ### UC-41 — Import an engine's native file history into the coordination journal
@@ -487,7 +498,10 @@ opaque per-block UID, public and referenceable (`research/260614-roam-deep-dive/
§2, §7). Enables UC-44/45 at sub-page granularity; falls back to content-fingerprint §2, §7). Enables UC-44/45 at sub-page granularity; falls back to content-fingerprint
(UC-46) or path+range where no native ID exists. Open: use native ID directly or wrap in (UC-46) or path+range where no native ID exists. Open: use native ID directly or wrap in
a shard-scoped address to avoid cross-shard collision and survive projection (findings a shard-scoped address to avoid cross-shard collision and survive projection (findings
§11 Q1). §11 Q1). Obsidian shows the **text-embedded** variant: `^block-id` and heading anchors
live in the Markdown text — git-diffable and portable (survives a file copy) but opt-in
(`research/260614-obsidian-deep-dive/findings.md` §2), vs Roam's mandatory DB-minted
UID.
**Priority:** Later **Priority:** Later
### UC-52 — Delegate derived views to a shard's native query engine ### UC-52 — Delegate derived views to a shard's native query engine
@@ -500,7 +514,68 @@ recent changes, typed queries) by delegating to it instead of scanning the proje
(`research/260614-roam-deep-dive/findings.md` §4). Makes "native query" an adapter (`research/260614-roam-deep-dive/findings.md` §4). Makes "native query" an adapter
capability the orchestrator can key off (vs. computing views itself over the capability the orchestrator can key off (vs. computing views itself over the
projection). Operationalizes the ZigZag "dimensions + rasters" insight projection). Operationalizes the ZigZag "dimensions + rasters" insight
(`research/260614-zigzag-deep-dive/findings.md` §5) and relates to UC-05/UC-34. (`research/260614-zigzag-deep-dive/findings.md` §5) and relates to UC-05/UC-34. Obsidian
nuance: query can be an *ecosystem plugin* (Dataview), **not core** — so "native query"
is adapter/plugin-provided and must not be assumed
(`research/260614-obsidian-deep-dive/findings.md` §7).
**Priority:** Later
### UC-53 — Attach a local-first Markdown vault with a live concurrent native editor
**Actor:** Maintainer
**Goal:** Attach an Obsidian-style local Markdown vault as a file-backed shard while its
own app continues to edit the files concurrently — watching for external changes,
re-projecting, and treating the app's config dir as opaque.
**Source:** obsidian, intent
**Notes:** Obsidian "file over app": vault = folder of `.md` + `.obsidian/` config;
files are canonical (`research/260614-obsidian-deep-dive/findings.md` §1). Cleanest
file-store attach (UC-40) — and uniquely **dual-mode**: zero-config direct attach
(read/projection/overlay) *or* an in-app adapter (Obsidian plugin) for write-through +
live `vault` events (UC-38, findings §4). Needs **external-writer tolerance**
(file-watch, re-project, conflict-with-live-app) distinct from multi-user write
conflicts. Exclude `.obsidian/` as shard-local config, not page content.
**Priority:** Later
### UC-54 — Define a page as a live query over the union
**Actor:** Author
**Goal:** Author a page whose body is materialized from a saved query over the union
(e.g. "all open tasks", "all pages tagged X"), refreshed as the union changes.
**Source:** obsidian, roam, intent
**Notes:** The Obsidian **Dataview** (4.4M downloads) / **Tasks** (3.6M) pattern, and
Roam `{{query}}` — query-defined dynamic pages
(`research/260614-obsidian-deep-dive/findings.md` §7). Distinct from UC-44 (reference/
span manifest) and UC-52 (delegating *computation* of an existing view): here the *page
itself* is a query. Open: core page type vs. adapter vs. reference-UI/plugin (findings
§11 Q2). May delegate execution to a shard's native query engine (UC-52).
**Priority:** Later
### UC-55 — Carry non-Markdown content types as typed or opaque assets
**Actor:** Maintainer or author
**Goal:** Attach and present a shard's non-Markdown content — drawings (Excalidraw),
spatial canvases (JSON Canvas), images, attachments — as typed or opaque assets with
provenance, without flattening them into Markdown or dropping them.
**Source:** obsidian, intent
**Notes:** The **#1 Obsidian plugin is Excalidraw (6.4M)** — users keep non-Markdown
content in "Markdown" vaults (`research/260614-obsidian-deep-dive/findings.md` §7).
Extends UC-34's no-lossy-flatten rule from structured-text to binary/spatial content.
Pushes the **wiki page model**: page vs. typed asset vs. opaque blob — a page-model
decision, not just adapter config (findings §10). JSON Canvas is an open format worth
first-class support.
**Priority:** Later
### UC-56 — Publish a curated projection to an external read-only target
**Actor:** Maintainer
**Goal:** Publish a selected projection of the union (or a shard) to an external
read-only target — a static site or hosted page set — preserving provenance.
**Source:** obsidian, intent
**Notes:** The Obsidian Publish / **Quartz** / Digital Garden pattern — outbound
publish (`research/260614-obsidian-deep-dive/findings.md` §7, §10). Formalizes the
`publish` capability from INTENT's adapter-capability list; **complements UC-37**
(inbound static-export *attach*) as its outbound mirror. Open: core vs. publish-adapter
family; interaction with overlays and projection freshness (findings §11 Q4).
**Priority:** Later **Priority:** Later
--- ---
@@ -586,7 +661,9 @@ shard-wiki adds Git history as structural safety net beyond social norms.
an agreed layout or subtree. an agreed layout or subtree.
**Source:** yawex **Source:** yawex
**Notes:** yawex blueprint pages. Distinct from UC-08 (blank capture) and UC-10 **Notes:** yawex blueprint pages. Distinct from UC-08 (blank capture) and UC-10
(refactoring existing discussion). (refactoring existing discussion). Obsidian's popular **Templater** (4.6M) / **QuickAdd**
(1.9M) plugins show templated creation is a top real-world need
(`research/260614-obsidian-deep-dive/findings.md` §7).
**Priority:** Later **Priority:** Later
### UC-16 — Append or comment without full edit ### UC-16 — Append or comment without full edit
@@ -741,6 +818,10 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD.
| UC-50 | | | | § | ✓ | | UC-50 | | | | § | ✓ |
| UC-51 | | | | § | ✓ | | UC-51 | | | | § | ✓ |
| UC-52 | | | | § | ✓ | | UC-52 | | | | § | ✓ |
| UC-53 | | | | ¶ | ✓ |
| UC-54 | | | | ¶ | ✓ |
| UC-55 | | | | ¶ | ✓ |
| UC-56 | | | | ¶ | ✓ |
| UC-08 | ✓ | | | | UC-08 | ✓ | | |
| UC-09 | ✓ | | | | UC-09 | ✓ | | |
| UC-10 | ✓ | | | | UC-10 | ✓ | | |
@@ -916,6 +997,50 @@ Markdown-first page model, not adopted as a substrate and never the federation l
native-query capabilities, block↔page mapping, and Roam as a second native-query capabilities, block↔page mapping, and Roam as a second
engine-hosts-adapter exemplar alongside XWiki. engine-hosts-adapter exemplar alongside XWiki.
### obsidian mapping
(¶ UC-53–UC-56 are placed in the **wikiengines** matrix column as the nearest existing
source — Obsidian is a shipped tool — but their true lineage is the **Obsidian deep
dive**, `research/260614-obsidian-deep-dive/findings.md`.)
| Obsidian mechanism (findings §) | Catalog UC |
|---------------------------------|------------|
| File-over-app vault; cleanest file-backed Markdown shard, dual-attachable (§1, §4) | UC-53 (new); enriches UC-40, UC-02, UC-38 |
| Dataview/Tasks: a page materialized from a query (§7) | UC-54 (new) |
| Excalidraw/Canvas/attachments: non-Markdown content (§7) | UC-55 (new) |
| Obsidian Publish/Quartz: outbound publish (§7) | UC-56 (new) |
| In-file `^block-id` / heading anchors = git-diffable opt-in span IDs (§2) | UC-51 (enriched) |
| YAML frontmatter/properties = git-diffable in-file structured data (§3) | UC-34 (enriched) |
| Git plugin (top-7) = bolt-on portable history (§5, §7) | UC-36 (enriched) |
| Query is a plugin (Dataview), not core (§7) | UC-52 (enriched) |
| Importer plugin = import from foreign tools (§7) | UC-28 (enriched) |
| Templater/QuickAdd = templated creation (§7) | UC-15 (enriched) |
| MetadataCache: files-canonical / index-derived (§1, §6) | reinforces UC-05/17–20 derived-view model |
#### Plugin-popularity → use-case signal
Per the research brief (let ecosystem popularity inform the catalog). All-time download
ranks read as **demand evidence** — full table in
`research/260614-obsidian-deep-dive/findings.md` §7. Headlines:
- **#1 is drawings** (Excalidraw 6.4M) → non-Markdown content is a *primary* real-world
use, not an edge case → **UC-55**.
- **Query-as-DB** (Dataview 4.4M, Tasks 3.6M) is top-tier but an *add-on* → query is an
adapter/plugin capability (**UC-52**) and motivates query-defined pages (**UC-54**).
- **Git is top-7** (2.7M) → users manually bolt on portable history → direct demand for
the coordination journal (**UC-36**).
- **Sync-to-anywhere** (Remotely Save 2.0M: S3/Dropbox/OneDrive/GDrive/WebDAV) →
demand to connect a vault to the heterogeneous backends INTENT targets — while
reaffirming the **not-a-file-sync-daemon** boundary (attach as shards, don't mirror).
Note: an Obsidian vault is **one file-backed candidate shard** — the cleanest,
INTENT-named, and the file-store counterpart to the DB-backed Roam shard — mapped into
the Markdown-first page model; **not** the federation layer and **not** a file-sync
target (`research/260614-obsidian-deep-dive/findings.md` §8.2). Architecture logged for
`SHARD-WP-0002` (T14): dual attachment mode, consume-native-derived-index capability,
non-Markdown content types in the page model, outbound publish, external-writer
tolerance.
--- ---
## Open questions ## Open questions
@@ -936,4 +1061,8 @@ engine-hosts-adapter exemplar alongside XWiki.
model — clones, a parallel graph index, or both? (ZigZag dive §10.) model — clones, a parallel graph index, or both? (ZigZag dive §10.)
9. Is **in-engine adapter hosting** (e.g. a Roam Depot extension for write-through) an 9. Is **in-engine adapter hosting** (e.g. a Roam Depot extension for write-through) an
accepted deployment mode generally, or do we restrict API-only backends to accepted deployment mode generally, or do we restrict API-only backends to
read/projection/overlay-target? (Roam dive §11 Q2; ties UC-38, UC-50.) read/projection/overlay-target? (Roam dive §11 Q2; ties UC-38, UC-50.)
10. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a
Markdown stub, opaque blob with provenance, or a pluggable content-type registry?
11. Is a **query-defined page** (UC-54) a core page type, an adapter feature, or a
reference-UI/plugin concern? (Shared with catalog Q7.)