# Findings — Obsidian: file-over-app vaults, the plugin API, and what the ecosystem reveals Date: 2026-06-14 Source kind: **modern shipped product** — a local-first Markdown vault tool; the *most INTENT-aligned candidate shard yet* (file-backed, Markdown-first); plus an ecosystem whose popularity is direct evidence of what users actually do Lens: shard-wiki — file-store attachment, in-file fine-grained addressing, derived index vs. canonical files, the engine-hosts-adapter path, and ecosystem-driven UCs > Why Obsidian matters most. INTENT names **Obsidian vaults** explicitly as a shard > participant, and Obsidian's **"file over app"** philosophy — your notes are plain > Markdown files in a folder you own, the app is just a lens — is the closest cousin > to shard-wiki's own Markdown-first, sovereignty-preserving thesis. It is also the > **file-backed counterpart to the Roam dive**: same "personal knowledge tool" > category, opposite storage model (Roam = client DB + API; Obsidian = files on disk). > And uniquely among the dives, Obsidian has a **large, ranked plugin ecosystem** — so > the user's instruction to let *plugin popularity inform the use-case catalog* is > actionable: what people install tells us what a federated wiki must support (§7). This dive treats Obsidian as a candidate shard (capability profile §5), studies its plugin API as an adapter-host surface (§4), and mines the download rankings for UCs (§7). Pairs with the Roam dive (the DB/API contrast) and the TWiki/Foswiki dives (the file-store attachment path, UC-40). --- ## 1. Core architecture — file over app - A **vault** is a local folder of plain **Markdown files** plus a `.obsidian/` config directory (settings, enabled plugins, themes, workspace layout). **The files are the source of truth**; the app holds no separate canonical database. - Markdown is **Obsidian-flavored**: CommonMark + `[[wikilinks]]`, `![[embeds]]` (transclusion), `^block-id` block references, `#tags`, YAML **frontmatter / properties** (typed metadata), and callouts. Note: all addressing and structure live **in the file text** — so they are git-diffable and portable. - **MetadataCache** is a *derived* index: Obsidian pre-parses every file into cached metadata (headings, links, embeds, tags, frontmatter, **blocks/block IDs**, `resolvedLinks` / `unresolvedLinks`, backlinks). It is **rebuilt from files** and updated **asynchronously** ("don't rely on immediate updates after a modify"). Files canonical; index derived. - **Canvas** files (`.canvas`, the open **JSON Canvas** format) hold spatial arrangements — a non-Markdown content type living in the vault. - Sync is *not* built into the format: users add Git, Obsidian Sync, or a file-sync plugin (see §7). The vault is just files. For shard-wiki this is the **ideal file-backed shard**: Markdown-first, on disk, git-friendly, with a derived index model that mirrors shard-wiki's own projection/derived-view philosophy (the cache is to Obsidian what projections and union BackLinks are to shard-wiki — computed, not canonical). --- ## 2. Addressing, links, transclusion — in-file and git-diffable - **Links:** `[[Page]]` and `[[Page#Heading]]` / `[[Page#^block-id]]`; the resolved/ unresolved link sets and **backlinks** are computed in MetadataCache — Obsidian's shipped answer to union BackLinks (UC-05/UC-18), but per-vault. - **Fine-grained addressing:** a **`^block-id`** suffix on a paragraph (and heading anchors) gives a stable sub-page address — *but it lives in the Markdown text*, not in a database. This is the **git-diffable, portable variant** of Roam's `:block/uid` (which lives in a DataScript DB). Important contrast for UC-51: native span IDs can be **text-embedded** (Obsidian) or **store-minted** (Roam); the text-embedded form is more portable and survives a file copy, but is opt-in (the user must add the `^id`). - **Transclusion:** `![[Page]]`, `![[Page#Heading]]`, `![[Page#^block-id]]` embed live content by reference — shipped transclusion at page/section/block granularity (UC-32), again stored as plain text. --- ## 3. Structured data — frontmatter as git-diffable records YAML **frontmatter / properties** make each note a lightweight typed record (`tags`, `aliases`, arbitrary keys). This is the **git-diffable structured-data** variant (cf. TWiki `%META%` inside the file, UC-34) — structure lives in the file, not a DB. The **Dataview** plugin (§7) turns frontmatter + inline fields + the link graph into a queryable database — but that query power is an *ecosystem add-on*, not core. --- ## 4. Extension architecture — the plugin API as an adapter host Obsidian plugins are TypeScript, shipped as `manifest.json` + `main.js` (+ optional `styles.css`): - **Manifest:** `id`, `name`, `version`, `minAppVersion` (required); `author`, `description`, `isDesktopOnly`, `fundingUrl` (optional). Community plugins are GitHub-hosted and listed in the `obsidian-releases` repo; installed from inside the app; gated by **Restricted Mode** (plugins run arbitrary code — Node/Electron on desktop). - **Plugin lifecycle:** extend `Plugin`, implement **`onload()` / `onunload()`**; use `registerEvent` / `registerDomEvent` / `registerInterval` for auto-cleanup. - **The `App` object** (global singleton, `this.app`) exposes four modules: - **`vault`** — file CRUD: `read` / `cachedRead`, `create`, `modify`, `process`, `delete`, `rename`, `getFiles`, low-level `adapter`, and **events** (`on('create'|'modify'|'delete'|'rename')`) for live file-watching. - **`metadataCache`** — `getFileCache`, `frontmatter`, `links`, `embeds`, `tags`, `headings`, `blocks`, `resolvedLinks` / `unresolvedLinks`, `getBacklinksForFile`. - **`workspace`** — panes/leaves/views; `registerView`, `registerEditorExtension` (CodeMirror 6), `registerMarkdownPostProcessor`. - **`fileManager`** — high-level ops; **`renameFile`** updates inbound links (use it instead of raw vault rename to keep link integrity). - **Registrations:** `addCommand`, `addRibbonIcon`, `addStatusBarItem`, `addSettingTab`, `registerView`, `registerEditorExtension`, `registerMarkdownPostProcessor`. Two consequences for shard-wiki: 1. **A vault is attachable two ways.** (a) **File-store direct attach** (UC-40) — read the folder as a folder shard; no plugin, offline-capable, git-native; good for read/projection/overlay. (b) **In-app plugin host** (UC-38) — a shard-wiki adapter *as an Obsidian plugin* drives `vault` write + `metadataCache` reads + live file events for high-fidelity write-through. This dual mode is a cleaner story than Roam (which only offers the in-app path). 2. The **MetadataCache API is a ready-made derived-view source** — backlinks, tags, block IDs, resolved links — an adapter can consume instead of reparsing. --- ## 5. Obsidian as a shard — capability profile | Capability | Obsidian | Notes for the adapter contract | |------------|----------|--------------------------------| | Read | **yes** | direct file read (folder shard) or `vault.read`/`cachedRead` in-app | | Write | **yes** | direct file write, or `vault.modify`/`create`/`process` in-app; per-file granularity | | Write granularity | **per-file (page)** | the natural shard-wiki granularity — contrast Roam (block) and TiddlyWiki (whole-file-single-vault) (UC-35) | | Identity / addressing | **path + in-file `^block-id` / headings** | git-diffable, portable, **opt-in** span IDs (UC-51 text-embedded variant) | | Transclusion | **yes** | `![[...]]` page/section/block embeds (UC-32) | | Backlinks / links | **yes (derived)** | MetadataCache resolved/unresolved links + backlinks (UC-05/18) | | Structured data | **yes (in-file)** | YAML frontmatter/properties; queryable via Dataview plugin (UC-34) | | Native query | **plugin only** | Dataview — *not core*; informs UC-52 (query is adapter/plugin-provided, not assumable) | | Version history | **none native** | users add the Git plugin (top-7!) → validates the coordination journal (UC-36) | | Diff / merge | **none native** | git-level if Git plugin / repo-backed | | Lock | **no** | local-first, single-user assumption | | Publish | **via Obsidian Publish / Quartz / Digital Garden** | outbound publish of a projection (UC-56) | | Syntax | **Obsidian-flavored Markdown** | close to CommonMark; wikilinks/embeds/callouts/`^id` need adapter awareness (cf. UC-42) | | Non-Markdown content | **Canvas (JSON Canvas), attachments, Excalidraw** | typed/opaque assets in the vault (UC-55) | Verdict: Obsidian is the **cleanest file-backed, Markdown-first shard** — the file-store family (TWiki/Foswiki/DokuWiki, UC-40), and the *reference* personal-vault shard INTENT already names. Its history/query/sync gaps are exactly what users fill with plugins (§7) — i.e. exactly what shard-wiki offers as orchestration. --- ## 6. The derived-index lesson (architecture) Obsidian's **files-canonical / MetadataCache-derived** split is independent validation of shard-wiki's core stance: the link graph, backlinks, tags, and block index are **computed projections over canonical files**, rebuilt on change, eventually consistent ("updated asynchronously"). shard-wiki's union BackLinks, projections, and derived views (UC-05, UC-17–UC-20) should likewise be **derived and rebuildable**, never a second source of truth. The cache-invalidation/async-update caveat is a real design note for projection freshness (UC-31). --- ## 7. Ecosystem popularity → use-case signal (the user's ask) All-time download ranks (obsidianstats.com), read as *demand evidence*: | Plugin (downloads) | What users do with a vault | shard-wiki signal → UC | |--------------------|----------------------------|------------------------| | **Excalidraw** (6.4M) | drawings/diagrams as first-class content | non-Markdown content types in a shard → **UC-55 (new)** | | **Templater** (4.6M), **QuickAdd** (1.9M) | templated note creation | blueprint pages → **UC-15 (enriched)** | | **Dataview** (4.4M) | query the vault (files+frontmatter+links) as a DB | query-defined dynamic pages → **UC-54 (new)**; query is a *plugin* → **UC-52 (enriched)** | | **Tasks** (3.6M) | aggregate to-dos across all notes | cross-page typed-item aggregation → **UC-54** | | **Advanced Tables** (2.9M) | structured tables in Markdown | in-file structured data → UC-34 | | **Calendar** (2.8M) | daily-note navigation | temporal dimension (ZigZag `d.recent`) → UC-17 | | **Git** (2.7M, top-7) | version control + sync on the vault | **history is a bolt-on, not native** → validates coordination journal → **UC-36 (enriched)** | | **Kanban** (2.4M) | board view of Markdown cards | alternate projection/view of pages → UC-47/48 | | **Remotely Save** (2.0M) | sync vault to S3 / Dropbox / OneDrive / GDrive / **WebDAV** | demand to federate a vault to heterogeneous remote stores → reinforces INTENT WebDAV/Nextcloud shards (but shard-wiki is *not* a file-sync daemon — attach as shards, don't mirror) | | **Omnisearch** (1.6M) | search across notes, PDFs, images | union full-text search over mixed content → UC-20 | | **Importer** (1.4M) | import from Evernote / Notion / Bear / Apple Notes | carry-forward/import from foreign tools → **UC-28 (enriched)** | | **Smart Connections** (1.0M), **Copilot** (1.5M), **Claudian** (938K) | AI over the vault (related notes, chat) | mostly out of scope; "related notes" ≈ link/equivalence discovery (UC-46) — note the trend, don't build it into core | | **Tag Wrangler** (981K) | rename/merge tags vault-wide | namespace/tag refactor across the union → UC-22 | | **Linter** (927K) | normalize frontmatter/formatting | content normalization → fingerprint basis for equivalence (UC-46) | | **Admonition** (922K) | callouts | Obsidian-flavored syntax → translation awareness (UC-42) | Headline reads: - The **#1 plugin is drawings** (Excalidraw) — users keep **non-Markdown content** in "Markdown" vaults. A wiki orchestrator that assumes pure Markdown will mishandle the most popular real-world use → **UC-55**. - **Query-the-vault-as-a-DB** (Dataview #3 + Tasks #4) is a top use — but it is an *add-on*, confirming query is an **adapter/plugin capability** (UC-52), and motivating **query-defined dynamic pages** (UC-54). - **Git is top-7** — users manually bolt portable version control onto their vault. This is *direct demand* for what shard-wiki provides natively as the coordination journal (UC-36). - **Sync-to-anywhere** (Remotely Save #11) shows demand to connect a vault to S3/WebDAV/ cloud — the heterogeneous backends INTENT targets — while reminding us of the *not-a-file-sync-daemon* boundary. --- ## 8. Mapping to shard-wiki INTENT (compare, do not equate) ### 8.1 Reinforcements - **File-over-app == shard sovereignty + Markdown-first.** Obsidian is the living proof that a serious tool can be "just files you own"; it is the model shard for INTENT's Obsidian/local-folder participants. - **Files canonical, index derived** validates shard-wiki's projection/derived-view architecture (§6). - **In-file, git-diffable structure & addressing** (frontmatter, `^block-id`, embeds) shows fine-grained addressing and structured data can be **portable text**, not DB state — friendlier to the coordination journal than Roam's DB. ### 8.2 Deliberate divergences (design bugs if conflated) 1. **A vault is one shard, not the federation.** Local-first, single-vault, single-user; do not model the union as "one big vault." 2. **Not a file-sync daemon.** The Remotely Save popularity is tempting; shard-wiki must stay **wiki-page-semantic** — attach remote stores as shards, never generic file mirroring (INTENT constraint). 3. **`.obsidian/` is opaque app config, not page content.** The adapter must exclude/ treat it as shard-local config, not project it as pages. 4. **Obsidian-flavored Markdown ≠ CommonMark.** Wikilinks/embeds/callouts/`^id` need adapter awareness; closer than Roam's outline, but still a translation surface (UC-42). 5. **Plugins run arbitrary code.** If a shard-wiki adapter is hosted *as* an Obsidian plugin (UC-38 path), it inherits Electron/Node trust — a deployment/security note, not a core concern. ### 8.3 What Obsidian teaches that shard-wiki should keep - Offer **both attachment modes** for a vault: zero-config **file-store direct attach** (read/projection/overlay) and an optional **in-app adapter** (write-through, live events). Graceful degradation by default, fidelity on opt-in. - **Consume a backend's existing derived index** (MetadataCache) instead of reparsing, when the shard exposes one. - **Embrace non-Markdown content types** (Canvas/attachments/drawings) as typed or opaque assets with provenance — don't flatten them away (UC-55, extends UC-34). --- ## 9. Use-case seeds → catalog (promoted 2026-06-14) Last existing UC is **UC-52**. New UCs **UC-53–UC-56** added; existing UCs enriched. | Seed | Catalog action | |------|----------------| | **Attach a local-first Markdown vault as a file-backed shard with a live concurrent native editor** — file-watch, tolerate the vault's own app writing concurrently, treat `.obsidian/` as opaque | **UC-53 (new)** | | **Define a page as a live query over the union** (Dataview/Tasks pattern — query-defined dynamic page) | **UC-54 (new)** | | **Carry non-Markdown content types** (Canvas/JSON Canvas, drawings, attachments) as typed/opaque assets with provenance | **UC-55 (new)** | | **Publish a curated projection of the union or a shard to an external read-only target** (Obsidian Publish / Quartz / Digital Garden) | **UC-56 (new)** | | Local Markdown vault = cleanest file-backed direct-attach; dual attach (file-store *or* in-app plugin host) | **enriches UC-40** (and UC-02, UC-38) | | In-file `^block-id` / heading anchors = git-diffable, portable, opt-in span IDs | **enriches UC-51** | | YAML frontmatter/properties = git-diffable in-file structured data | **enriches UC-34** | | Git plugin (top-7) = users bolt on portable history → demand for the coordination journal | **enriches UC-36** | | Query is a popular plugin (Dataview), not core → query is adapter/plugin-provided | **enriches UC-52** | | Importer plugin = import from foreign tools | **enriches UC-28** | | Templater/QuickAdd = templated creation | **enriches UC-15** | | `![[...]]` embeds = in-file transclusion | links UC-32 | | MetadataCache backlinks/links | links UC-05/UC-18 | --- ## 10. Architecture notes for SHARD-WP-0002 (no UC) - **Dual attachment mode** for a single backend (file-store *and* in-engine adapter) should be first-class in the adapter contract / T14 binding — Obsidian is the clean example (Roam was in-app-only; TWiki had file-vs-API but not a personal vault). - **"Consume native derived index"** as a capability: a shard may expose its own link/ backlink/block index (Obsidian MetadataCache, Roam Datalog) the orchestrator can read instead of reparsing (ties UC-52). - **Non-Markdown content types** (UC-55) push on the wiki page model: pages vs. typed assets vs. opaque blobs — a page-model spec decision, not just adapter config. - **Outbound publish** (UC-56) formalizes the `publish` capability from INTENT's list as a projection target, complementing inbound static-export attach (UC-37). - Concurrent-native-editor (UC-53) needs the contract to express **external-writer tolerance** (file-watching, re-projection, conflict-with-live-app) — distinct from multi-user write conflicts. --- ## 11. Open questions (for spec / workplans) 1. For a vault attached **both** ways (file-store + in-app plugin), which is authoritative, and how do they reconcile (the in-app adapter sees MetadataCache; the file-store path sees raw bytes)? 2. Is **UC-54 (query-defined dynamic page)** a core page type, an adapter feature, or a reference-UI/plugin concern? (Mirrors the Roam/Dataview "views are queries" question and catalog Q7.) 3. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a Markdown stub, opaque blob with provenance, or a pluggable content-type registry? 4. Does **outbound publish** (UC-56) belong in core or as a publish-adapter family, and how does it interact with overlays/projection freshness? 5. How is **concurrent native editing** (UC-53) surfaced — optimistic re-projection, advisory lock, or overlay-onto-moving-target? --- ## 12. Sources | Source | Used for | |--------|----------| | Obsidian API — App Architecture (https://www.mintlify.com/obsidianmd/obsidian-api/concepts/app-architecture) | App singleton, four modules, this.app | | obsidianmd/obsidian-api README (https://github.com/obsidianmd/obsidian-api/blob/master/README.md) | Plugin lifecycle, Vault/MetadataCache/Workspace/FileManager APIs, registrations, manifest fields | | DeepWiki — Vault and File System; MetadataCache and Link Resolution (https://deepwiki.com/obsidianmd/obsidian-api) | Vault CRUD + events; MetadataCache parsed elements, resolved/unresolved links, backlinks | | obsidianstats.com — Most Downloaded Plugins (https://www.obsidianstats.com/most-downloaded) | All-time download ranks + counts for the popularity → UC mapping (§7) | | Obsidian — "The future of plugins" (https://obsidian.md/blog/future-of-plugins/) | Plugin distribution model, restricted mode context | | Obsidian Help / JSON Canvas (jsoncanvas.org) | Vault format, `.obsidian/` config, Canvas open format | Cross-references: `research/260614-roam-deep-dive/findings.md` (DB/API contrast, native span IDs, query delegation), `research/260613-twiki-deep-dive/findings.md` (file-store attach UC-40), `spec/UseCaseCatalog.md` (UC-02, UC-05, UC-15, UC-18, UC-28, UC-31, UC-32, UC-34, UC-36, UC-38, UC-40, UC-51, UC-52), `workplans/SHARD-WP-0002-federation-architecture.md` (T14). --- ## 13. Traceability - New UCs: **UC-53, UC-54, UC-55, UC-56** → `spec/UseCaseCatalog.md`. - Enriched UCs: **UC-15, UC-28, UC-34, UC-36, UC-40, UC-51, UC-52** (and links UC-02, UC-05, UC-18, UC-32, UC-38). - Architecture (no UC): dual attachment mode; consume-native-derived-index capability; non-Markdown content types in the page model; outbound publish; external-writer tolerance → `SHARD-WP-0002` (T14). - Distinctive artifact: **plugin-popularity → UC mapping** (§7) — ecosystem demand evidence, per the research brief. - Boundary recorded: an Obsidian vault is **one file-backed candidate shard** (the cleanest, INTENT-named), mapped into the Markdown-first page model; not the federation layer and not a file-sync target (INTENT Stability Note + not-a-sync-daemon).