shard-wiki/research/260614-obsidian-deep-dive/findings.md

Findings — Obsidian: file-over-app vaults, the plugin API, and what the ecosystem reveals

Date: 2026-06-14 Source kind: modern shipped product — a local-first Markdown vault tool; the most INTENT-aligned candidate shard yet (file-backed, Markdown-first); plus an ecosystem whose popularity is direct evidence of what users actually do Lens: shard-wiki — file-store attachment, in-file fine-grained addressing, derived index vs. canonical files, the engine-hosts-adapter path, and ecosystem-driven UCs

Why Obsidian matters most. INTENT names Obsidian vaults explicitly as a shard participant, and Obsidian's "file over app" philosophy — your notes are plain Markdown files in a folder you own, the app is just a lens — is the closest cousin to shard-wiki's own Markdown-first, sovereignty-preserving thesis. It is also the file-backed counterpart to the Roam dive: same "personal knowledge tool" category, opposite storage model (Roam = client DB + API; Obsidian = files on disk). And uniquely among the dives, Obsidian has a large, ranked plugin ecosystem — so the user's instruction to let plugin popularity inform the use-case catalog is actionable: what people install tells us what a federated wiki must support (§7).

This dive treats Obsidian as a candidate shard (capability profile §5), studies its plugin API as an adapter-host surface (§4), and mines the download rankings for UCs (§7). Pairs with the Roam dive (the DB/API contrast) and the TWiki/Foswiki dives (the file-store attachment path, UC-40).

---

1. Core architecture — file over app

• A vault is a local folder of plain Markdown files plus a .obsidian/ config directory (settings, enabled plugins, themes, workspace layout). The files are the source of truth; the app holds no separate canonical database.
• Markdown is Obsidian-flavored: CommonMark + [[wikilinks]], ![[embeds]] (transclusion), ^block-id block references, #tags, YAML frontmatter / properties (typed metadata), and callouts. Note: all addressing and structure live in the file text — so they are git-diffable and portable.
• MetadataCache is a derived index: Obsidian pre-parses every file into cached metadata (headings, links, embeds, tags, frontmatter, blocks/block IDs, resolvedLinks / unresolvedLinks, backlinks). It is rebuilt from files and updated asynchronously ("don't rely on immediate updates after a modify"). Files canonical; index derived.
• Canvas files (.canvas, the open JSON Canvas format) hold spatial arrangements — a non-Markdown content type living in the vault.
• Sync is not built into the format: users add Git, Obsidian Sync, or a file-sync plugin (see §7). The vault is just files.

For shard-wiki this is the ideal file-backed shard: Markdown-first, on disk, git-friendly, with a derived index model that mirrors shard-wiki's own projection/derived-view philosophy (the cache is to Obsidian what projections and union BackLinks are to shard-wiki — computed, not canonical).

---

2. Addressing, links, transclusion — in-file and git-diffable

• Links: [[Page]] and [[Page#Heading]] / [[Page#^block-id]]; the resolved/ unresolved link sets and backlinks are computed in MetadataCache — Obsidian's shipped answer to union BackLinks (UC-05/UC-18), but per-vault.
• Fine-grained addressing: a ^block-id suffix on a paragraph (and heading anchors) gives a stable sub-page address — but it lives in the Markdown text, not in a database. This is the git-diffable, portable variant of Roam's :block/uid (which lives in a DataScript DB). Important contrast for UC-51: native span IDs can be text-embedded (Obsidian) or store-minted (Roam); the text-embedded form is more portable and survives a file copy, but is opt-in (the user must add the ^id).
• Transclusion: ![[Page]], ![[Page#Heading]], ![[Page#^block-id]] embed live content by reference — shipped transclusion at page/section/block granularity (UC-32), again stored as plain text.

---

3. Structured data — frontmatter as git-diffable records

YAML frontmatter / properties make each note a lightweight typed record (tags, aliases, arbitrary keys). This is the git-diffable structured-data variant (cf. TWiki %META% inside the file, UC-34) — structure lives in the file, not a DB. The Dataview plugin (§7) turns frontmatter + inline fields + the link graph into a queryable database — but that query power is an ecosystem add-on, not core.

---

4. Extension architecture — the plugin API as an adapter host

Obsidian plugins are TypeScript, shipped as manifest.json + main.js (+ optional styles.css):

• Manifest: id, name, version, minAppVersion (required); author, description, isDesktopOnly, fundingUrl (optional). Community plugins are GitHub-hosted and listed in the obsidian-releases repo; installed from inside the app; gated by Restricted Mode (plugins run arbitrary code — Node/Electron on desktop).
• Plugin lifecycle: extend Plugin, implement onload() / onunload(); use registerEvent / registerDomEvent / registerInterval for auto-cleanup.
• The App object (global singleton, this.app) exposes four modules:
  • vault — file CRUD: read / cachedRead, create, modify, process, delete, rename, getFiles, low-level adapter, and events (on('create'|'modify'|'delete'|'rename')) for live file-watching.
  • metadataCache — getFileCache, frontmatter, links, embeds, tags, headings, blocks, resolvedLinks / unresolvedLinks, getBacklinksForFile.
  • workspace — panes/leaves/views; registerView, registerEditorExtension (CodeMirror 6), registerMarkdownPostProcessor.
  • fileManager — high-level ops; renameFile updates inbound links (use it instead of raw vault rename to keep link integrity).
• Registrations: addCommand, addRibbonIcon, addStatusBarItem, addSettingTab, registerView, registerEditorExtension, registerMarkdownPostProcessor.

Two consequences for shard-wiki:

1. A vault is attachable two ways. (a) File-store direct attach (UC-40) — read the folder as a folder shard; no plugin, offline-capable, git-native; good for read/projection/overlay. (b) In-app plugin host (UC-38) — a shard-wiki adapter as an Obsidian plugin drives vault write + metadataCache reads + live file events for high-fidelity write-through. This dual mode is a cleaner story than Roam (which only offers the in-app path).
2. The MetadataCache API is a ready-made derived-view source — backlinks, tags, block IDs, resolved links — an adapter can consume instead of reparsing.

---

5. Obsidian as a shard — capability profile

Capability	Obsidian	Notes for the adapter contract
Read	yes	direct file read (folder shard) or vault.read/cachedRead in-app
Write	yes	direct file write, or vault.modify/create/process in-app; per-file granularity
Write granularity	per-file (page)	the natural shard-wiki granularity — contrast Roam (block) and TiddlyWiki (whole-file-single-vault) (UC-35)
Identity / addressing	path + in-file ^block-id / headings	git-diffable, portable, opt-in span IDs (UC-51 text-embedded variant)
Transclusion	yes	![[...]] page/section/block embeds (UC-32)
Backlinks / links	yes (derived)	MetadataCache resolved/unresolved links + backlinks (UC-05/18)
Structured data	yes (in-file)	YAML frontmatter/properties; queryable via Dataview plugin (UC-34)
Native query	plugin only	Dataview — not core; informs UC-52 (query is adapter/plugin-provided, not assumable)
Version history	none native	users add the Git plugin (top-7!) → validates the coordination journal (UC-36)
Diff / merge	none native	git-level if Git plugin / repo-backed
Lock	no	local-first, single-user assumption
Publish	via Obsidian Publish / Quartz / Digital Garden	outbound publish of a projection (UC-56)
Syntax	Obsidian-flavored Markdown	close to CommonMark; wikilinks/embeds/callouts/^id need adapter awareness (cf. UC-42)
Non-Markdown content	Canvas (JSON Canvas), attachments, Excalidraw	typed/opaque assets in the vault (UC-55)

Verdict: Obsidian is the cleanest file-backed, Markdown-first shard — the file-store family (TWiki/Foswiki/DokuWiki, UC-40), and the reference personal-vault shard INTENT already names. Its history/query/sync gaps are exactly what users fill with plugins (§7) — i.e. exactly what shard-wiki offers as orchestration.

---

6. The derived-index lesson (architecture)

Obsidian's files-canonical / MetadataCache-derived split is independent validation of shard-wiki's core stance: the link graph, backlinks, tags, and block index are computed projections over canonical files, rebuilt on change, eventually consistent ("updated asynchronously"). shard-wiki's union BackLinks, projections, and derived views (UC-05, UC-17–UC-20) should likewise be derived and rebuildable, never a second source of truth. The cache-invalidation/async-update caveat is a real design note for projection freshness (UC-31).

---

7. Ecosystem popularity → use-case signal (the user's ask)

All-time download ranks (obsidianstats.com), read as demand evidence:

Plugin (downloads)	What users do with a vault	shard-wiki signal → UC
Excalidraw (6.4M)	drawings/diagrams as first-class content	non-Markdown content types in a shard → UC-55 (new)
Templater (4.6M), QuickAdd (1.9M)	templated note creation	blueprint pages → UC-15 (enriched)
Dataview (4.4M)	query the vault (files+frontmatter+links) as a DB	query-defined dynamic pages → UC-54 (new); query is a plugin → UC-52 (enriched)
Tasks (3.6M)	aggregate to-dos across all notes	cross-page typed-item aggregation → UC-54
Advanced Tables (2.9M)	structured tables in Markdown	in-file structured data → UC-34
Calendar (2.8M)	daily-note navigation	temporal dimension (ZigZag d.recent) → UC-17
Git (2.7M, top-7)	version control + sync on the vault	history is a bolt-on, not native → validates coordination journal → UC-36 (enriched)
Kanban (2.4M)	board view of Markdown cards	alternate projection/view of pages → UC-47/48
Remotely Save (2.0M)	sync vault to S3 / Dropbox / OneDrive / GDrive / WebDAV	demand to federate a vault to heterogeneous remote stores → reinforces INTENT WebDAV/Nextcloud shards (but shard-wiki is not a file-sync daemon — attach as shards, don't mirror)
Omnisearch (1.6M)	search across notes, PDFs, images	union full-text search over mixed content → UC-20
Importer (1.4M)	import from Evernote / Notion / Bear / Apple Notes	carry-forward/import from foreign tools → UC-28 (enriched)
Smart Connections (1.0M), Copilot (1.5M), Claudian (938K)	AI over the vault (related notes, chat)	mostly out of scope; "related notes" ≈ link/equivalence discovery (UC-46) — note the trend, don't build it into core
Tag Wrangler (981K)	rename/merge tags vault-wide	namespace/tag refactor across the union → UC-22
Linter (927K)	normalize frontmatter/formatting	content normalization → fingerprint basis for equivalence (UC-46)
Admonition (922K)	callouts	Obsidian-flavored syntax → translation awareness (UC-42)

Headline reads:

• The #1 plugin is drawings (Excalidraw) — users keep non-Markdown content in "Markdown" vaults. A wiki orchestrator that assumes pure Markdown will mishandle the most popular real-world use → UC-55.
• Query-the-vault-as-a-DB (Dataview #3 + Tasks #4) is a top use — but it is an add-on, confirming query is an adapter/plugin capability (UC-52), and motivating query-defined dynamic pages (UC-54).
• Git is top-7 — users manually bolt portable version control onto their vault. This is direct demand for what shard-wiki provides natively as the coordination journal (UC-36).
• Sync-to-anywhere (Remotely Save #11) shows demand to connect a vault to S3/WebDAV/ cloud — the heterogeneous backends INTENT targets — while reminding us of the not-a-file-sync-daemon boundary.

---

8. Mapping to shard-wiki INTENT (compare, do not equate)

8.1 Reinforcements

• File-over-app == shard sovereignty + Markdown-first. Obsidian is the living proof that a serious tool can be "just files you own"; it is the model shard for INTENT's Obsidian/local-folder participants.
• Files canonical, index derived validates shard-wiki's projection/derived-view architecture (§6).
• In-file, git-diffable structure & addressing (frontmatter, ^block-id, embeds) shows fine-grained addressing and structured data can be portable text, not DB state — friendlier to the coordination journal than Roam's DB.

8.2 Deliberate divergences (design bugs if conflated)

1. A vault is one shard, not the federation. Local-first, single-vault, single-user; do not model the union as "one big vault."
2. Not a file-sync daemon. The Remotely Save popularity is tempting; shard-wiki must stay wiki-page-semantic — attach remote stores as shards, never generic file mirroring (INTENT constraint).
3. .obsidian/ is opaque app config, not page content. The adapter must exclude/ treat it as shard-local config, not project it as pages.
4. Obsidian-flavored Markdown ≠ CommonMark. Wikilinks/embeds/callouts/^id need adapter awareness; closer than Roam's outline, but still a translation surface (UC-42).
5. Plugins run arbitrary code. If a shard-wiki adapter is hosted as an Obsidian plugin (UC-38 path), it inherits Electron/Node trust — a deployment/security note, not a core concern.

8.3 What Obsidian teaches that shard-wiki should keep

• Offer both attachment modes for a vault: zero-config file-store direct attach (read/projection/overlay) and an optional in-app adapter (write-through, live events). Graceful degradation by default, fidelity on opt-in.
• Consume a backend's existing derived index (MetadataCache) instead of reparsing, when the shard exposes one.
• Embrace non-Markdown content types (Canvas/attachments/drawings) as typed or opaque assets with provenance — don't flatten them away (UC-55, extends UC-34).

---

9. Use-case seeds → catalog (promoted 2026-06-14)

Last existing UC is UC-52. New UCs UC-53–UC-56 added; existing UCs enriched.

Seed	Catalog action
Attach a local-first Markdown vault as a file-backed shard with a live concurrent native editor — file-watch, tolerate the vault's own app writing concurrently, treat .obsidian/ as opaque	UC-53 (new)
Define a page as a live query over the union (Dataview/Tasks pattern — query-defined dynamic page)	UC-54 (new)
Carry non-Markdown content types (Canvas/JSON Canvas, drawings, attachments) as typed/opaque assets with provenance	UC-55 (new)
Publish a curated projection of the union or a shard to an external read-only target (Obsidian Publish / Quartz / Digital Garden)	UC-56 (new)
Local Markdown vault = cleanest file-backed direct-attach; dual attach (file-store or in-app plugin host)	enriches UC-40 (and UC-02, UC-38)
In-file ^block-id / heading anchors = git-diffable, portable, opt-in span IDs	enriches UC-51
YAML frontmatter/properties = git-diffable in-file structured data	enriches UC-34
Git plugin (top-7) = users bolt on portable history → demand for the coordination journal	enriches UC-36
Query is a popular plugin (Dataview), not core → query is adapter/plugin-provided	enriches UC-52
Importer plugin = import from foreign tools	enriches UC-28
Templater/QuickAdd = templated creation	enriches UC-15
![[...]] embeds = in-file transclusion	links UC-32
MetadataCache backlinks/links	links UC-05/UC-18

---

10. Architecture notes for SHARD-WP-0002 (no UC)

• Dual attachment mode for a single backend (file-store and in-engine adapter) should be first-class in the adapter contract / T14 binding — Obsidian is the clean example (Roam was in-app-only; TWiki had file-vs-API but not a personal vault).
• "Consume native derived index" as a capability: a shard may expose its own link/ backlink/block index (Obsidian MetadataCache, Roam Datalog) the orchestrator can read instead of reparsing (ties UC-52).
• Non-Markdown content types (UC-55) push on the wiki page model: pages vs. typed assets vs. opaque blobs — a page-model spec decision, not just adapter config.
• Outbound publish (UC-56) formalizes the publish capability from INTENT's list as a projection target, complementing inbound static-export attach (UC-37).
• Concurrent-native-editor (UC-53) needs the contract to express external-writer tolerance (file-watching, re-projection, conflict-with-live-app) — distinct from multi-user write conflicts.

---

11. Open questions (for spec / workplans)

1. For a vault attached both ways (file-store + in-app plugin), which is authoritative, and how do they reconcile (the in-app adapter sees MetadataCache; the file-store path sees raw bytes)?
2. Is UC-54 (query-defined dynamic page) a core page type, an adapter feature, or a reference-UI/plugin concern? (Mirrors the Roam/Dataview "views are queries" question and catalog Q7.)
3. How does shard-wiki represent non-Markdown content (UC-55) — typed asset with a Markdown stub, opaque blob with provenance, or a pluggable content-type registry?
4. Does outbound publish (UC-56) belong in core or as a publish-adapter family, and how does it interact with overlays/projection freshness?
5. How is concurrent native editing (UC-53) surfaced — optimistic re-projection, advisory lock, or overlay-onto-moving-target?

---

12. Sources

Source	Used for
Obsidian API — App Architecture (https://www.mintlify.com/obsidianmd/obsidian-api/concepts/app-architecture)	App singleton, four modules, this.app
obsidianmd/obsidian-api README (https://github.com/obsidianmd/obsidian-api/blob/master/README.md)	Plugin lifecycle, Vault/MetadataCache/Workspace/FileManager APIs, registrations, manifest fields
DeepWiki — Vault and File System; MetadataCache and Link Resolution (https://deepwiki.com/obsidianmd/obsidian-api)	Vault CRUD + events; MetadataCache parsed elements, resolved/unresolved links, backlinks
obsidianstats.com — Most Downloaded Plugins (https://www.obsidianstats.com/most-downloaded)	All-time download ranks + counts for the popularity → UC mapping (§7)
Obsidian — "The future of plugins" (https://obsidian.md/blog/future-of-plugins/)	Plugin distribution model, restricted mode context
Obsidian Help / JSON Canvas (jsoncanvas.org)	Vault format, .obsidian/ config, Canvas open format

Cross-references: research/260614-roam-deep-dive/findings.md (DB/API contrast, native span IDs, query delegation), research/260613-twiki-deep-dive/findings.md (file-store attach UC-40), spec/UseCaseCatalog.md (UC-02, UC-05, UC-15, UC-18, UC-28, UC-31, UC-32, UC-34, UC-36, UC-38, UC-40, UC-51, UC-52), workplans/SHARD-WP-0002-federation-architecture.md (T14).

---

13. Traceability

• New UCs: UC-53, UC-54, UC-55, UC-56 → spec/UseCaseCatalog.md.
• Enriched UCs: UC-15, UC-28, UC-34, UC-36, UC-40, UC-51, UC-52 (and links UC-02, UC-05, UC-18, UC-32, UC-38).
• Architecture (no UC): dual attachment mode; consume-native-derived-index capability; non-Markdown content types in the page model; outbound publish; external-writer tolerance → SHARD-WP-0002 (T14).
• Distinctive artifact: plugin-popularity → UC mapping (§7) — ecosystem demand evidence, per the research brief.
• Boundary recorded: an Obsidian vault is one file-backed candidate shard (the cleanest, INTENT-named), mapped into the Markdown-first page model; not the federation layer and not a file-sync target (INTENT Stability Note + not-a-sync-daemon).