shard-wiki/research/260614-obsidian-deep-dive/findings.md

# Findings — Obsidian: file-over-app vaults, the plugin API, and what the ecosystem reveals

Date: 2026-06-14
Source kind: **modern shipped product** — a local-first Markdown vault tool; the
*most INTENT-aligned candidate shard yet* (file-backed, Markdown-first); plus an
ecosystem whose popularity is direct evidence of what users actually do
Lens: shard-wiki — file-store attachment, in-file fine-grained addressing, derived
index vs. canonical files, the engine-hosts-adapter path, and ecosystem-driven UCs

> Why Obsidian matters most. INTENT names **Obsidian vaults** explicitly as a shard
> participant, and Obsidian's **"file over app"** philosophy — your notes are plain
> Markdown files in a folder you own, the app is just a lens — is the closest cousin
> to shard-wiki's own Markdown-first, sovereignty-preserving thesis. It is also the
> **file-backed counterpart to the Roam dive**: same "personal knowledge tool"
> category, opposite storage model (Roam = client DB + API; Obsidian = files on disk).
> And uniquely among the dives, Obsidian has a **large, ranked plugin ecosystem** — so
> the user's instruction to let *plugin popularity inform the use-case catalog* is
> actionable: what people install tells us what a federated wiki must support (§7).

This dive treats Obsidian as a candidate shard (capability profile §5), studies its
plugin API as an adapter-host surface (§4), and mines the download rankings for UCs
(§7). Pairs with the Roam dive (the DB/API contrast) and the TWiki/Foswiki dives (the
file-store attachment path, UC-40).

---

## 1. Core architecture — file over app

- A **vault** is a local folder of plain **Markdown files** plus a `.obsidian/` config
  directory (settings, enabled plugins, themes, workspace layout). **The files are the
  source of truth**; the app holds no separate canonical database.
- Markdown is **Obsidian-flavored**: CommonMark + `[[wikilinks]]`, `![[embeds]]`
  (transclusion), `^block-id` block references, `#tags`, YAML **frontmatter /
  properties** (typed metadata), and callouts. Note: all addressing and structure live
  **in the file text** — so they are git-diffable and portable.
- **MetadataCache** is a *derived* index: Obsidian pre-parses every file into cached
  metadata (headings, links, embeds, tags, frontmatter, **blocks/block IDs**,
  `resolvedLinks` / `unresolvedLinks`, backlinks). It is **rebuilt from files** and
  updated **asynchronously** ("don't rely on immediate updates after a modify"). Files
  canonical; index derived.
- **Canvas** files (`.canvas`, the open **JSON Canvas** format) hold spatial
  arrangements — a non-Markdown content type living in the vault.
- Sync is *not* built into the format: users add Git, Obsidian Sync, or a file-sync
  plugin (see §7). The vault is just files.

For shard-wiki this is the **ideal file-backed shard**: Markdown-first, on disk,
git-friendly, with a derived index model that mirrors shard-wiki's own
projection/derived-view philosophy (the cache is to Obsidian what projections and union
BackLinks are to shard-wiki — computed, not canonical).

---

## 2. Addressing, links, transclusion — in-file and git-diffable

- **Links:** `[[Page]]` and `[[Page#Heading]]` / `[[Page#^block-id]]`; the resolved/
  unresolved link sets and **backlinks** are computed in MetadataCache — Obsidian's
  shipped answer to union BackLinks (UC-05/UC-18), but per-vault.
- **Fine-grained addressing:** a **`^block-id`** suffix on a paragraph (and heading
  anchors) gives a stable sub-page address — *but it lives in the Markdown text*, not in
  a database. This is the **git-diffable, portable variant** of Roam's `:block/uid`
  (which lives in a DataScript DB). Important contrast for UC-51: native span IDs can be
  **text-embedded** (Obsidian) or **store-minted** (Roam); the text-embedded form is
  more portable and survives a file copy, but is opt-in (the user must add the `^id`).
- **Transclusion:** `![[Page]]`, `![[Page#Heading]]`, `![[Page#^block-id]]` embed live
  content by reference — shipped transclusion at page/section/block granularity (UC-32),
  again stored as plain text.

---

## 3. Structured data — frontmatter as git-diffable records

YAML **frontmatter / properties** make each note a lightweight typed record
(`tags`, `aliases`, arbitrary keys). This is the **git-diffable structured-data**
variant (cf. TWiki `%META%` inside the file, UC-34) — structure lives in the file, not a
DB. The **Dataview** plugin (§7) turns frontmatter + inline fields + the link graph into
a queryable database — but that query power is an *ecosystem add-on*, not core.

---

## 4. Extension architecture — the plugin API as an adapter host

Obsidian plugins are TypeScript, shipped as `manifest.json` + `main.js` (+ optional
`styles.css`):

- **Manifest:** `id`, `name`, `version`, `minAppVersion` (required); `author`,
  `description`, `isDesktopOnly`, `fundingUrl` (optional). Community plugins are
  GitHub-hosted and listed in the `obsidian-releases` repo; installed from inside the
  app; gated by **Restricted Mode** (plugins run arbitrary code — Node/Electron on
  desktop).
- **Plugin lifecycle:** extend `Plugin`, implement **`onload()` / `onunload()`**; use
  `registerEvent` / `registerDomEvent` / `registerInterval` for auto-cleanup.
- **The `App` object** (global singleton, `this.app`) exposes four modules:
  - **`vault`** — file CRUD: `read` / `cachedRead`, `create`, `modify`, `process`,
    `delete`, `rename`, `getFiles`, low-level `adapter`, and **events**
    (`on('create'|'modify'|'delete'|'rename')`) for live file-watching.
  - **`metadataCache`** — `getFileCache`, `frontmatter`, `links`, `embeds`, `tags`,
    `headings`, `blocks`, `resolvedLinks` / `unresolvedLinks`, `getBacklinksForFile`.
  - **`workspace`** — panes/leaves/views; `registerView`, `registerEditorExtension`
    (CodeMirror 6), `registerMarkdownPostProcessor`.
  - **`fileManager`** — high-level ops; **`renameFile`** updates inbound links (use it
    instead of raw vault rename to keep link integrity).
- **Registrations:** `addCommand`, `addRibbonIcon`, `addStatusBarItem`,
  `addSettingTab`, `registerView`, `registerEditorExtension`,
  `registerMarkdownPostProcessor`.

Two consequences for shard-wiki:

1. **A vault is attachable two ways.** (a) **File-store direct attach** (UC-40) — read
   the folder as a folder shard; no plugin, offline-capable, git-native; good for
   read/projection/overlay. (b) **In-app plugin host** (UC-38) — a shard-wiki adapter
   *as an Obsidian plugin* drives `vault` write + `metadataCache` reads + live file
   events for high-fidelity write-through. This dual mode is a cleaner story than Roam
   (which only offers the in-app path).
2. The **MetadataCache API is a ready-made derived-view source** — backlinks, tags,
   block IDs, resolved links — an adapter can consume instead of reparsing.

---

## 5. Obsidian as a shard — capability profile

| Capability | Obsidian | Notes for the adapter contract |
|------------|----------|--------------------------------|
| Read | **yes** | direct file read (folder shard) or `vault.read`/`cachedRead` in-app |
| Write | **yes** | direct file write, or `vault.modify`/`create`/`process` in-app; per-file granularity |
| Write granularity | **per-file (page)** | the natural shard-wiki granularity — contrast Roam (block) and TiddlyWiki (whole-file-single-vault) (UC-35) |
| Identity / addressing | **path + in-file `^block-id` / headings** | git-diffable, portable, **opt-in** span IDs (UC-51 text-embedded variant) |
| Transclusion | **yes** | `![[...]]` page/section/block embeds (UC-32) |
| Backlinks / links | **yes (derived)** | MetadataCache resolved/unresolved links + backlinks (UC-05/18) |
| Structured data | **yes (in-file)** | YAML frontmatter/properties; queryable via Dataview plugin (UC-34) |
| Native query | **plugin only** | Dataview — *not core*; informs UC-52 (query is adapter/plugin-provided, not assumable) |
| Version history | **none native** | users add the Git plugin (top-7!) → validates the coordination journal (UC-36) |
| Diff / merge | **none native** | git-level if Git plugin / repo-backed |
| Lock | **no** | local-first, single-user assumption |
| Publish | **via Obsidian Publish / Quartz / Digital Garden** | outbound publish of a projection (UC-56) |
| Syntax | **Obsidian-flavored Markdown** | close to CommonMark; wikilinks/embeds/callouts/`^id` need adapter awareness (cf. UC-42) |
| Non-Markdown content | **Canvas (JSON Canvas), attachments, Excalidraw** | typed/opaque assets in the vault (UC-55) |

Verdict: Obsidian is the **cleanest file-backed, Markdown-first shard** — the
file-store family (TWiki/Foswiki/DokuWiki, UC-40), and the *reference* personal-vault
shard INTENT already names. Its history/query/sync gaps are exactly what users fill with
plugins (§7) — i.e. exactly what shard-wiki offers as orchestration.

---

## 6. The derived-index lesson (architecture)

Obsidian's **files-canonical / MetadataCache-derived** split is independent validation
of shard-wiki's core stance: the link graph, backlinks, tags, and block index are
**computed projections over canonical files**, rebuilt on change, eventually consistent
("updated asynchronously"). shard-wiki's union BackLinks, projections, and derived views
(UC-05, UC-17–UC-20) should likewise be **derived and rebuildable**, never a second
source of truth. The cache-invalidation/async-update caveat is a real design note for
projection freshness (UC-31).

---

## 7. Ecosystem popularity → use-case signal (the user's ask)

All-time download ranks (obsidianstats.com), read as *demand evidence*:

| Plugin (downloads) | What users do with a vault | shard-wiki signal → UC |
|--------------------|----------------------------|------------------------|
| **Excalidraw** (6.4M) | drawings/diagrams as first-class content | non-Markdown content types in a shard → **UC-55 (new)** |
| **Templater** (4.6M), **QuickAdd** (1.9M) | templated note creation | blueprint pages → **UC-15 (enriched)** |
| **Dataview** (4.4M) | query the vault (files+frontmatter+links) as a DB | query-defined dynamic pages → **UC-54 (new)**; query is a *plugin* → **UC-52 (enriched)** |
| **Tasks** (3.6M) | aggregate to-dos across all notes | cross-page typed-item aggregation → **UC-54** |
| **Advanced Tables** (2.9M) | structured tables in Markdown | in-file structured data → UC-34 |
| **Calendar** (2.8M) | daily-note navigation | temporal dimension (ZigZag `d.recent`) → UC-17 |
| **Git** (2.7M, top-7) | version control + sync on the vault | **history is a bolt-on, not native** → validates coordination journal → **UC-36 (enriched)** |
| **Kanban** (2.4M) | board view of Markdown cards | alternate projection/view of pages → UC-47/48 |
| **Remotely Save** (2.0M) | sync vault to S3 / Dropbox / OneDrive / GDrive / **WebDAV** | demand to federate a vault to heterogeneous remote stores → reinforces INTENT WebDAV/Nextcloud shards (but shard-wiki is *not* a file-sync daemon — attach as shards, don't mirror) |
| **Omnisearch** (1.6M) | search across notes, PDFs, images | union full-text search over mixed content → UC-20 |
| **Importer** (1.4M) | import from Evernote / Notion / Bear / Apple Notes | carry-forward/import from foreign tools → **UC-28 (enriched)** |
| **Smart Connections** (1.0M), **Copilot** (1.5M), **Claudian** (938K) | AI over the vault (related notes, chat) | mostly out of scope; "related notes" ≈ link/equivalence discovery (UC-46) — note the trend, don't build it into core |
| **Tag Wrangler** (981K) | rename/merge tags vault-wide | namespace/tag refactor across the union → UC-22 |
| **Linter** (927K) | normalize frontmatter/formatting | content normalization → fingerprint basis for equivalence (UC-46) |
| **Admonition** (922K) | callouts | Obsidian-flavored syntax → translation awareness (UC-42) |

Headline reads:
- The **#1 plugin is drawings** (Excalidraw) — users keep **non-Markdown content** in
  "Markdown" vaults. A wiki orchestrator that assumes pure Markdown will mishandle the
  most popular real-world use → **UC-55**.
- **Query-the-vault-as-a-DB** (Dataview #3 + Tasks #4) is a top use — but it is an
  *add-on*, confirming query is an **adapter/plugin capability** (UC-52), and motivating
  **query-defined dynamic pages** (UC-54).
- **Git is top-7** — users manually bolt portable version control onto their vault. This
  is *direct demand* for what shard-wiki provides natively as the coordination journal
  (UC-36).
- **Sync-to-anywhere** (Remotely Save #11) shows demand to connect a vault to S3/WebDAV/
  cloud — the heterogeneous backends INTENT targets — while reminding us of the
  *not-a-file-sync-daemon* boundary.

---

## 8. Mapping to shard-wiki INTENT (compare, do not equate)

### 8.1 Reinforcements

- **File-over-app == shard sovereignty + Markdown-first.** Obsidian is the living proof
  that a serious tool can be "just files you own"; it is the model shard for INTENT's
  Obsidian/local-folder participants.
- **Files canonical, index derived** validates shard-wiki's projection/derived-view
  architecture (§6).
- **In-file, git-diffable structure & addressing** (frontmatter, `^block-id`, embeds)
  shows fine-grained addressing and structured data can be **portable text**, not DB
  state — friendlier to the coordination journal than Roam's DB.

### 8.2 Deliberate divergences (design bugs if conflated)

1. **A vault is one shard, not the federation.** Local-first, single-vault,
   single-user; do not model the union as "one big vault."
2. **Not a file-sync daemon.** The Remotely Save popularity is tempting; shard-wiki must
   stay **wiki-page-semantic** — attach remote stores as shards, never generic file
   mirroring (INTENT constraint).
3. **`.obsidian/` is opaque app config, not page content.** The adapter must exclude/
   treat it as shard-local config, not project it as pages.
4. **Obsidian-flavored Markdown ≠ CommonMark.** Wikilinks/embeds/callouts/`^id` need
   adapter awareness; closer than Roam's outline, but still a translation surface
   (UC-42).
5. **Plugins run arbitrary code.** If a shard-wiki adapter is hosted *as* an Obsidian
   plugin (UC-38 path), it inherits Electron/Node trust — a deployment/security note,
   not a core concern.

### 8.3 What Obsidian teaches that shard-wiki should keep

- Offer **both attachment modes** for a vault: zero-config **file-store direct attach**
  (read/projection/overlay) and an optional **in-app adapter** (write-through, live
  events). Graceful degradation by default, fidelity on opt-in.
- **Consume a backend's existing derived index** (MetadataCache) instead of reparsing,
  when the shard exposes one.
- **Embrace non-Markdown content types** (Canvas/attachments/drawings) as typed or
  opaque assets with provenance — don't flatten them away (UC-55, extends UC-34).

---

## 9. Use-case seeds → catalog (promoted 2026-06-14)

Last existing UC is **UC-52**. New UCs **UC-53–UC-56** added; existing UCs enriched.

| Seed | Catalog action |
|------|----------------|
| **Attach a local-first Markdown vault as a file-backed shard with a live concurrent native editor** — file-watch, tolerate the vault's own app writing concurrently, treat `.obsidian/` as opaque | **UC-53 (new)** |
| **Define a page as a live query over the union** (Dataview/Tasks pattern — query-defined dynamic page) | **UC-54 (new)** |
| **Carry non-Markdown content types** (Canvas/JSON Canvas, drawings, attachments) as typed/opaque assets with provenance | **UC-55 (new)** |
| **Publish a curated projection of the union or a shard to an external read-only target** (Obsidian Publish / Quartz / Digital Garden) | **UC-56 (new)** |
| Local Markdown vault = cleanest file-backed direct-attach; dual attach (file-store *or* in-app plugin host) | **enriches UC-40** (and UC-02, UC-38) |
| In-file `^block-id` / heading anchors = git-diffable, portable, opt-in span IDs | **enriches UC-51** |
| YAML frontmatter/properties = git-diffable in-file structured data | **enriches UC-34** |
| Git plugin (top-7) = users bolt on portable history → demand for the coordination journal | **enriches UC-36** |
| Query is a popular plugin (Dataview), not core → query is adapter/plugin-provided | **enriches UC-52** |
| Importer plugin = import from foreign tools | **enriches UC-28** |
| Templater/QuickAdd = templated creation | **enriches UC-15** |
| `![[...]]` embeds = in-file transclusion | links UC-32 |
| MetadataCache backlinks/links | links UC-05/UC-18 |

---

## 10. Architecture notes for SHARD-WP-0002 (no UC)

- **Dual attachment mode** for a single backend (file-store *and* in-engine adapter)
  should be first-class in the adapter contract / T14 binding — Obsidian is the clean
  example (Roam was in-app-only; TWiki had file-vs-API but not a personal vault).
- **"Consume native derived index"** as a capability: a shard may expose its own link/
  backlink/block index (Obsidian MetadataCache, Roam Datalog) the orchestrator can read
  instead of reparsing (ties UC-52).
- **Non-Markdown content types** (UC-55) push on the wiki page model: pages vs. typed
  assets vs. opaque blobs — a page-model spec decision, not just adapter config.
- **Outbound publish** (UC-56) formalizes the `publish` capability from INTENT's list as
  a projection target, complementing inbound static-export attach (UC-37).
- Concurrent-native-editor (UC-53) needs the contract to express **external-writer
  tolerance** (file-watching, re-projection, conflict-with-live-app) — distinct from
  multi-user write conflicts.

---

## 11. Open questions (for spec / workplans)

1. For a vault attached **both** ways (file-store + in-app plugin), which is
   authoritative, and how do they reconcile (the in-app adapter sees MetadataCache; the
   file-store path sees raw bytes)?
2. Is **UC-54 (query-defined dynamic page)** a core page type, an adapter feature, or a
   reference-UI/plugin concern? (Mirrors the Roam/Dataview "views are queries" question
   and catalog Q7.)
3. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a
   Markdown stub, opaque blob with provenance, or a pluggable content-type registry?
4. Does **outbound publish** (UC-56) belong in core or as a publish-adapter family, and
   how does it interact with overlays/projection freshness?
5. How is **concurrent native editing** (UC-53) surfaced — optimistic re-projection,
   advisory lock, or overlay-onto-moving-target?

---

## 12. Sources

| Source | Used for |
|--------|----------|
| Obsidian API — App Architecture (https://www.mintlify.com/obsidianmd/obsidian-api/concepts/app-architecture) | App singleton, four modules, this.app |
| obsidianmd/obsidian-api README (https://github.com/obsidianmd/obsidian-api/blob/master/README.md) | Plugin lifecycle, Vault/MetadataCache/Workspace/FileManager APIs, registrations, manifest fields |
| DeepWiki — Vault and File System; MetadataCache and Link Resolution (https://deepwiki.com/obsidianmd/obsidian-api) | Vault CRUD + events; MetadataCache parsed elements, resolved/unresolved links, backlinks |
| obsidianstats.com — Most Downloaded Plugins (https://www.obsidianstats.com/most-downloaded) | All-time download ranks + counts for the popularity → UC mapping (§7) |
| Obsidian — "The future of plugins" (https://obsidian.md/blog/future-of-plugins/) | Plugin distribution model, restricted mode context |
| Obsidian Help / JSON Canvas (jsoncanvas.org) | Vault format, `.obsidian/` config, Canvas open format |

Cross-references: `research/260614-roam-deep-dive/findings.md` (DB/API contrast, native
span IDs, query delegation), `research/260613-twiki-deep-dive/findings.md` (file-store
attach UC-40), `spec/UseCaseCatalog.md` (UC-02, UC-05, UC-15, UC-18, UC-28, UC-31,
UC-32, UC-34, UC-36, UC-38, UC-40, UC-51, UC-52), `workplans/SHARD-WP-0002-federation-architecture.md` (T14).

---

## 13. Traceability

- New UCs: **UC-53, UC-54, UC-55, UC-56** → `spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-15, UC-28, UC-34, UC-36, UC-40, UC-51, UC-52** (and links UC-02,
  UC-05, UC-18, UC-32, UC-38).
- Architecture (no UC): dual attachment mode; consume-native-derived-index capability;
  non-Markdown content types in the page model; outbound publish; external-writer
  tolerance → `SHARD-WP-0002` (T14).
- Distinctive artifact: **plugin-popularity → UC mapping** (§7) — ecosystem demand
  evidence, per the research brief.
- Boundary recorded: an Obsidian vault is **one file-backed candidate shard** (the
  cleanest, INTENT-named), mapped into the Markdown-first page model; not the federation
  layer and not a file-sync target (INTENT Stability Note + not-a-sync-daemon).
</content>