diff --git a/SCOPE.md b/SCOPE.md index 2e0f58a..08ea96a 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -19,9 +19,9 @@ Learnings update both SCOPE and INTENT where necessary. |-------|-------| | Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) | | Intent | `INTENT.md` established; authorization-in-core amendments drafted | -| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam deep dives (`research/260614-*/`) | +| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian deep dives (`research/260614-*/`) | | Demand | NetKingdom integration asks captured, not yet negotiated | -| Spec | Architecture blueprint drafted; UseCaseCatalog 52 UCs from research; PRD/TSD scaffolds | +| Spec | Architecture blueprint drafted; UseCaseCatalog 56 UCs from research; PRD/TSD scaffolds | | Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) | ## In Scope (today) diff --git a/research/260614-obsidian-deep-dive/README.md b/research/260614-obsidian-deep-dive/README.md new file mode 100644 index 0000000..d994de1 --- /dev/null +++ b/research/260614-obsidian-deep-dive/README.md @@ -0,0 +1,51 @@ +# 260614 — Obsidian deep dive (file-over-app vaults, plugin API, ecosystem signal) + +Date: 2026-06-14 + +## What this is + +A focused study of **Obsidian** — core/extension architecture and the plugin +ecosystem — read through shard-wiki's lens. Obsidian is the **most INTENT-aligned tool +in the research set**: INTENT names Obsidian vaults as a shard participant, and its +**"file over app"** model (notes are plain Markdown files in a folder you own; the app +is a lens) is the closest cousin to shard-wiki's Markdown-first, sovereignty-preserving +thesis. It is the **file-backed counterpart to the Roam dive** (same tool category, +opposite storage model: Roam = client DB + API; Obsidian = files on disk). + +Distinctive material: +- **Architecture** — vault = folder of `.md` + `.obsidian/` config; files canonical; + **MetadataCache** is a *derived*, async-rebuilt index (links, embeds, tags, + frontmatter, block IDs, resolved/unresolved links, backlinks) +- **In-file, git-diffable** addressing/structure — `[[wikilinks]]`, `![[embeds]]` + (transclusion), `^block-id` span refs, YAML frontmatter/properties +- **Plugin API** — `Plugin` `onload/onunload`; `App` modules `vault` (CRUD + file + events), `metadataCache`, `workspace` (CM6, views), `fileManager.renameFile`; + `manifest.json`; community plugins, Restricted Mode +- **Ecosystem popularity → UC signal** (§7) — ranked download data mined for use cases, + per the research brief + +## Contents + +| Path | Role | +|------|------| +| `findings.md` | File-over-app architecture, in-file addressing, plugin API as adapter host, capability profile, **plugin-popularity → UC mapping**, derived-index lesson, INTENT mapping, UC seeds, sources | + +## Status + +Initial deep dive complete. Four new use cases promoted to `spec/UseCaseCatalog.md` +(UC-53 attach a local vault with a live concurrent native editor, UC-54 query-defined +dynamic page, UC-55 carry non-Markdown content types, UC-56 outbound publish of a +projection); UC-15/28/34/36/40/51/52 enriched. Logged for `SHARD-WP-0002` (T14): +**dual attachment mode** (file-store *or* in-app plugin host), consume-native-derived- +index capability, non-Markdown content types in the page model, outbound publish, +external-writer tolerance. + +**Key takeaways recorded:** (1) Obsidian validates *files-canonical / index-derived* — +shard-wiki's projection model. (2) Fine-grained addressing & structure can be portable +**in-file text** (`^block-id`, frontmatter), not DB state (contrast Roam). (3) The +ecosystem is demand evidence — the #1 plugin is **drawings** (non-Markdown content), +query-as-DB (Dataview/Tasks) is top-tier but an *add-on*, and **Git is top-7** (users +bolt on the portable history shard-wiki provides natively). **Boundary:** a vault is one +file-backed candidate shard, not the federation layer and not a file-sync target +(not-a-sync-daemon). + diff --git a/research/260614-obsidian-deep-dive/findings.md b/research/260614-obsidian-deep-dive/findings.md new file mode 100644 index 0000000..19c8baf --- /dev/null +++ b/research/260614-obsidian-deep-dive/findings.md @@ -0,0 +1,324 @@ +# Findings — Obsidian: file-over-app vaults, the plugin API, and what the ecosystem reveals + +Date: 2026-06-14 +Source kind: **modern shipped product** — a local-first Markdown vault tool; the +*most INTENT-aligned candidate shard yet* (file-backed, Markdown-first); plus an +ecosystem whose popularity is direct evidence of what users actually do +Lens: shard-wiki — file-store attachment, in-file fine-grained addressing, derived +index vs. canonical files, the engine-hosts-adapter path, and ecosystem-driven UCs + +> Why Obsidian matters most. INTENT names **Obsidian vaults** explicitly as a shard +> participant, and Obsidian's **"file over app"** philosophy — your notes are plain +> Markdown files in a folder you own, the app is just a lens — is the closest cousin +> to shard-wiki's own Markdown-first, sovereignty-preserving thesis. It is also the +> **file-backed counterpart to the Roam dive**: same "personal knowledge tool" +> category, opposite storage model (Roam = client DB + API; Obsidian = files on disk). +> And uniquely among the dives, Obsidian has a **large, ranked plugin ecosystem** — so +> the user's instruction to let *plugin popularity inform the use-case catalog* is +> actionable: what people install tells us what a federated wiki must support (§7). + +This dive treats Obsidian as a candidate shard (capability profile §5), studies its +plugin API as an adapter-host surface (§4), and mines the download rankings for UCs +(§7). Pairs with the Roam dive (the DB/API contrast) and the TWiki/Foswiki dives (the +file-store attachment path, UC-40). + +--- + +## 1. Core architecture — file over app + +- A **vault** is a local folder of plain **Markdown files** plus a `.obsidian/` config + directory (settings, enabled plugins, themes, workspace layout). **The files are the + source of truth**; the app holds no separate canonical database. +- Markdown is **Obsidian-flavored**: CommonMark + `[[wikilinks]]`, `![[embeds]]` + (transclusion), `^block-id` block references, `#tags`, YAML **frontmatter / + properties** (typed metadata), and callouts. Note: all addressing and structure live + **in the file text** — so they are git-diffable and portable. +- **MetadataCache** is a *derived* index: Obsidian pre-parses every file into cached + metadata (headings, links, embeds, tags, frontmatter, **blocks/block IDs**, + `resolvedLinks` / `unresolvedLinks`, backlinks). It is **rebuilt from files** and + updated **asynchronously** ("don't rely on immediate updates after a modify"). Files + canonical; index derived. +- **Canvas** files (`.canvas`, the open **JSON Canvas** format) hold spatial + arrangements — a non-Markdown content type living in the vault. +- Sync is *not* built into the format: users add Git, Obsidian Sync, or a file-sync + plugin (see §7). The vault is just files. + +For shard-wiki this is the **ideal file-backed shard**: Markdown-first, on disk, +git-friendly, with a derived index model that mirrors shard-wiki's own +projection/derived-view philosophy (the cache is to Obsidian what projections and union +BackLinks are to shard-wiki — computed, not canonical). + +--- + +## 2. Addressing, links, transclusion — in-file and git-diffable + +- **Links:** `[[Page]]` and `[[Page#Heading]]` / `[[Page#^block-id]]`; the resolved/ + unresolved link sets and **backlinks** are computed in MetadataCache — Obsidian's + shipped answer to union BackLinks (UC-05/UC-18), but per-vault. +- **Fine-grained addressing:** a **`^block-id`** suffix on a paragraph (and heading + anchors) gives a stable sub-page address — *but it lives in the Markdown text*, not in + a database. This is the **git-diffable, portable variant** of Roam's `:block/uid` + (which lives in a DataScript DB). Important contrast for UC-51: native span IDs can be + **text-embedded** (Obsidian) or **store-minted** (Roam); the text-embedded form is + more portable and survives a file copy, but is opt-in (the user must add the `^id`). +- **Transclusion:** `![[Page]]`, `![[Page#Heading]]`, `![[Page#^block-id]]` embed live + content by reference — shipped transclusion at page/section/block granularity (UC-32), + again stored as plain text. + +--- + +## 3. Structured data — frontmatter as git-diffable records + +YAML **frontmatter / properties** make each note a lightweight typed record +(`tags`, `aliases`, arbitrary keys). This is the **git-diffable structured-data** +variant (cf. TWiki `%META%` inside the file, UC-34) — structure lives in the file, not a +DB. The **Dataview** plugin (§7) turns frontmatter + inline fields + the link graph into +a queryable database — but that query power is an *ecosystem add-on*, not core. + +--- + +## 4. Extension architecture — the plugin API as an adapter host + +Obsidian plugins are TypeScript, shipped as `manifest.json` + `main.js` (+ optional +`styles.css`): + +- **Manifest:** `id`, `name`, `version`, `minAppVersion` (required); `author`, + `description`, `isDesktopOnly`, `fundingUrl` (optional). Community plugins are + GitHub-hosted and listed in the `obsidian-releases` repo; installed from inside the + app; gated by **Restricted Mode** (plugins run arbitrary code — Node/Electron on + desktop). +- **Plugin lifecycle:** extend `Plugin`, implement **`onload()` / `onunload()`**; use + `registerEvent` / `registerDomEvent` / `registerInterval` for auto-cleanup. +- **The `App` object** (global singleton, `this.app`) exposes four modules: + - **`vault`** — file CRUD: `read` / `cachedRead`, `create`, `modify`, `process`, + `delete`, `rename`, `getFiles`, low-level `adapter`, and **events** + (`on('create'|'modify'|'delete'|'rename')`) for live file-watching. + - **`metadataCache`** — `getFileCache`, `frontmatter`, `links`, `embeds`, `tags`, + `headings`, `blocks`, `resolvedLinks` / `unresolvedLinks`, `getBacklinksForFile`. + - **`workspace`** — panes/leaves/views; `registerView`, `registerEditorExtension` + (CodeMirror 6), `registerMarkdownPostProcessor`. + - **`fileManager`** — high-level ops; **`renameFile`** updates inbound links (use it + instead of raw vault rename to keep link integrity). +- **Registrations:** `addCommand`, `addRibbonIcon`, `addStatusBarItem`, + `addSettingTab`, `registerView`, `registerEditorExtension`, + `registerMarkdownPostProcessor`. + +Two consequences for shard-wiki: + +1. **A vault is attachable two ways.** (a) **File-store direct attach** (UC-40) — read + the folder as a folder shard; no plugin, offline-capable, git-native; good for + read/projection/overlay. (b) **In-app plugin host** (UC-38) — a shard-wiki adapter + *as an Obsidian plugin* drives `vault` write + `metadataCache` reads + live file + events for high-fidelity write-through. This dual mode is a cleaner story than Roam + (which only offers the in-app path). +2. The **MetadataCache API is a ready-made derived-view source** — backlinks, tags, + block IDs, resolved links — an adapter can consume instead of reparsing. + +--- + +## 5. Obsidian as a shard — capability profile + +| Capability | Obsidian | Notes for the adapter contract | +|------------|----------|--------------------------------| +| Read | **yes** | direct file read (folder shard) or `vault.read`/`cachedRead` in-app | +| Write | **yes** | direct file write, or `vault.modify`/`create`/`process` in-app; per-file granularity | +| Write granularity | **per-file (page)** | the natural shard-wiki granularity — contrast Roam (block) and TiddlyWiki (whole-file-single-vault) (UC-35) | +| Identity / addressing | **path + in-file `^block-id` / headings** | git-diffable, portable, **opt-in** span IDs (UC-51 text-embedded variant) | +| Transclusion | **yes** | `![[...]]` page/section/block embeds (UC-32) | +| Backlinks / links | **yes (derived)** | MetadataCache resolved/unresolved links + backlinks (UC-05/18) | +| Structured data | **yes (in-file)** | YAML frontmatter/properties; queryable via Dataview plugin (UC-34) | +| Native query | **plugin only** | Dataview — *not core*; informs UC-52 (query is adapter/plugin-provided, not assumable) | +| Version history | **none native** | users add the Git plugin (top-7!) → validates the coordination journal (UC-36) | +| Diff / merge | **none native** | git-level if Git plugin / repo-backed | +| Lock | **no** | local-first, single-user assumption | +| Publish | **via Obsidian Publish / Quartz / Digital Garden** | outbound publish of a projection (UC-56) | +| Syntax | **Obsidian-flavored Markdown** | close to CommonMark; wikilinks/embeds/callouts/`^id` need adapter awareness (cf. UC-42) | +| Non-Markdown content | **Canvas (JSON Canvas), attachments, Excalidraw** | typed/opaque assets in the vault (UC-55) | + +Verdict: Obsidian is the **cleanest file-backed, Markdown-first shard** — the +file-store family (TWiki/Foswiki/DokuWiki, UC-40), and the *reference* personal-vault +shard INTENT already names. Its history/query/sync gaps are exactly what users fill with +plugins (§7) — i.e. exactly what shard-wiki offers as orchestration. + +--- + +## 6. The derived-index lesson (architecture) + +Obsidian's **files-canonical / MetadataCache-derived** split is independent validation +of shard-wiki's core stance: the link graph, backlinks, tags, and block index are +**computed projections over canonical files**, rebuilt on change, eventually consistent +("updated asynchronously"). shard-wiki's union BackLinks, projections, and derived views +(UC-05, UC-17–UC-20) should likewise be **derived and rebuildable**, never a second +source of truth. The cache-invalidation/async-update caveat is a real design note for +projection freshness (UC-31). + +--- + +## 7. Ecosystem popularity → use-case signal (the user's ask) + +All-time download ranks (obsidianstats.com), read as *demand evidence*: + +| Plugin (downloads) | What users do with a vault | shard-wiki signal → UC | +|--------------------|----------------------------|------------------------| +| **Excalidraw** (6.4M) | drawings/diagrams as first-class content | non-Markdown content types in a shard → **UC-55 (new)** | +| **Templater** (4.6M), **QuickAdd** (1.9M) | templated note creation | blueprint pages → **UC-15 (enriched)** | +| **Dataview** (4.4M) | query the vault (files+frontmatter+links) as a DB | query-defined dynamic pages → **UC-54 (new)**; query is a *plugin* → **UC-52 (enriched)** | +| **Tasks** (3.6M) | aggregate to-dos across all notes | cross-page typed-item aggregation → **UC-54** | +| **Advanced Tables** (2.9M) | structured tables in Markdown | in-file structured data → UC-34 | +| **Calendar** (2.8M) | daily-note navigation | temporal dimension (ZigZag `d.recent`) → UC-17 | +| **Git** (2.7M, top-7) | version control + sync on the vault | **history is a bolt-on, not native** → validates coordination journal → **UC-36 (enriched)** | +| **Kanban** (2.4M) | board view of Markdown cards | alternate projection/view of pages → UC-47/48 | +| **Remotely Save** (2.0M) | sync vault to S3 / Dropbox / OneDrive / GDrive / **WebDAV** | demand to federate a vault to heterogeneous remote stores → reinforces INTENT WebDAV/Nextcloud shards (but shard-wiki is *not* a file-sync daemon — attach as shards, don't mirror) | +| **Omnisearch** (1.6M) | search across notes, PDFs, images | union full-text search over mixed content → UC-20 | +| **Importer** (1.4M) | import from Evernote / Notion / Bear / Apple Notes | carry-forward/import from foreign tools → **UC-28 (enriched)** | +| **Smart Connections** (1.0M), **Copilot** (1.5M), **Claudian** (938K) | AI over the vault (related notes, chat) | mostly out of scope; "related notes" ≈ link/equivalence discovery (UC-46) — note the trend, don't build it into core | +| **Tag Wrangler** (981K) | rename/merge tags vault-wide | namespace/tag refactor across the union → UC-22 | +| **Linter** (927K) | normalize frontmatter/formatting | content normalization → fingerprint basis for equivalence (UC-46) | +| **Admonition** (922K) | callouts | Obsidian-flavored syntax → translation awareness (UC-42) | + +Headline reads: +- The **#1 plugin is drawings** (Excalidraw) — users keep **non-Markdown content** in + "Markdown" vaults. A wiki orchestrator that assumes pure Markdown will mishandle the + most popular real-world use → **UC-55**. +- **Query-the-vault-as-a-DB** (Dataview #3 + Tasks #4) is a top use — but it is an + *add-on*, confirming query is an **adapter/plugin capability** (UC-52), and motivating + **query-defined dynamic pages** (UC-54). +- **Git is top-7** — users manually bolt portable version control onto their vault. This + is *direct demand* for what shard-wiki provides natively as the coordination journal + (UC-36). +- **Sync-to-anywhere** (Remotely Save #11) shows demand to connect a vault to S3/WebDAV/ + cloud — the heterogeneous backends INTENT targets — while reminding us of the + *not-a-file-sync-daemon* boundary. + +--- + +## 8. Mapping to shard-wiki INTENT (compare, do not equate) + +### 8.1 Reinforcements + +- **File-over-app == shard sovereignty + Markdown-first.** Obsidian is the living proof + that a serious tool can be "just files you own"; it is the model shard for INTENT's + Obsidian/local-folder participants. +- **Files canonical, index derived** validates shard-wiki's projection/derived-view + architecture (§6). +- **In-file, git-diffable structure & addressing** (frontmatter, `^block-id`, embeds) + shows fine-grained addressing and structured data can be **portable text**, not DB + state — friendlier to the coordination journal than Roam's DB. + +### 8.2 Deliberate divergences (design bugs if conflated) + +1. **A vault is one shard, not the federation.** Local-first, single-vault, + single-user; do not model the union as "one big vault." +2. **Not a file-sync daemon.** The Remotely Save popularity is tempting; shard-wiki must + stay **wiki-page-semantic** — attach remote stores as shards, never generic file + mirroring (INTENT constraint). +3. **`.obsidian/` is opaque app config, not page content.** The adapter must exclude/ + treat it as shard-local config, not project it as pages. +4. **Obsidian-flavored Markdown ≠ CommonMark.** Wikilinks/embeds/callouts/`^id` need + adapter awareness; closer than Roam's outline, but still a translation surface + (UC-42). +5. **Plugins run arbitrary code.** If a shard-wiki adapter is hosted *as* an Obsidian + plugin (UC-38 path), it inherits Electron/Node trust — a deployment/security note, + not a core concern. + +### 8.3 What Obsidian teaches that shard-wiki should keep + +- Offer **both attachment modes** for a vault: zero-config **file-store direct attach** + (read/projection/overlay) and an optional **in-app adapter** (write-through, live + events). Graceful degradation by default, fidelity on opt-in. +- **Consume a backend's existing derived index** (MetadataCache) instead of reparsing, + when the shard exposes one. +- **Embrace non-Markdown content types** (Canvas/attachments/drawings) as typed or + opaque assets with provenance — don't flatten them away (UC-55, extends UC-34). + +--- + +## 9. Use-case seeds → catalog (promoted 2026-06-14) + +Last existing UC is **UC-52**. New UCs **UC-53–UC-56** added; existing UCs enriched. + +| Seed | Catalog action | +|------|----------------| +| **Attach a local-first Markdown vault as a file-backed shard with a live concurrent native editor** — file-watch, tolerate the vault's own app writing concurrently, treat `.obsidian/` as opaque | **UC-53 (new)** | +| **Define a page as a live query over the union** (Dataview/Tasks pattern — query-defined dynamic page) | **UC-54 (new)** | +| **Carry non-Markdown content types** (Canvas/JSON Canvas, drawings, attachments) as typed/opaque assets with provenance | **UC-55 (new)** | +| **Publish a curated projection of the union or a shard to an external read-only target** (Obsidian Publish / Quartz / Digital Garden) | **UC-56 (new)** | +| Local Markdown vault = cleanest file-backed direct-attach; dual attach (file-store *or* in-app plugin host) | **enriches UC-40** (and UC-02, UC-38) | +| In-file `^block-id` / heading anchors = git-diffable, portable, opt-in span IDs | **enriches UC-51** | +| YAML frontmatter/properties = git-diffable in-file structured data | **enriches UC-34** | +| Git plugin (top-7) = users bolt on portable history → demand for the coordination journal | **enriches UC-36** | +| Query is a popular plugin (Dataview), not core → query is adapter/plugin-provided | **enriches UC-52** | +| Importer plugin = import from foreign tools | **enriches UC-28** | +| Templater/QuickAdd = templated creation | **enriches UC-15** | +| `![[...]]` embeds = in-file transclusion | links UC-32 | +| MetadataCache backlinks/links | links UC-05/UC-18 | + +--- + +## 10. Architecture notes for SHARD-WP-0002 (no UC) + +- **Dual attachment mode** for a single backend (file-store *and* in-engine adapter) + should be first-class in the adapter contract / T14 binding — Obsidian is the clean + example (Roam was in-app-only; TWiki had file-vs-API but not a personal vault). +- **"Consume native derived index"** as a capability: a shard may expose its own link/ + backlink/block index (Obsidian MetadataCache, Roam Datalog) the orchestrator can read + instead of reparsing (ties UC-52). +- **Non-Markdown content types** (UC-55) push on the wiki page model: pages vs. typed + assets vs. opaque blobs — a page-model spec decision, not just adapter config. +- **Outbound publish** (UC-56) formalizes the `publish` capability from INTENT's list as + a projection target, complementing inbound static-export attach (UC-37). +- Concurrent-native-editor (UC-53) needs the contract to express **external-writer + tolerance** (file-watching, re-projection, conflict-with-live-app) — distinct from + multi-user write conflicts. + +--- + +## 11. Open questions (for spec / workplans) + +1. For a vault attached **both** ways (file-store + in-app plugin), which is + authoritative, and how do they reconcile (the in-app adapter sees MetadataCache; the + file-store path sees raw bytes)? +2. Is **UC-54 (query-defined dynamic page)** a core page type, an adapter feature, or a + reference-UI/plugin concern? (Mirrors the Roam/Dataview "views are queries" question + and catalog Q7.) +3. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a + Markdown stub, opaque blob with provenance, or a pluggable content-type registry? +4. Does **outbound publish** (UC-56) belong in core or as a publish-adapter family, and + how does it interact with overlays/projection freshness? +5. How is **concurrent native editing** (UC-53) surfaced — optimistic re-projection, + advisory lock, or overlay-onto-moving-target? + +--- + +## 12. Sources + +| Source | Used for | +|--------|----------| +| Obsidian API — App Architecture (https://www.mintlify.com/obsidianmd/obsidian-api/concepts/app-architecture) | App singleton, four modules, this.app | +| obsidianmd/obsidian-api README (https://github.com/obsidianmd/obsidian-api/blob/master/README.md) | Plugin lifecycle, Vault/MetadataCache/Workspace/FileManager APIs, registrations, manifest fields | +| DeepWiki — Vault and File System; MetadataCache and Link Resolution (https://deepwiki.com/obsidianmd/obsidian-api) | Vault CRUD + events; MetadataCache parsed elements, resolved/unresolved links, backlinks | +| obsidianstats.com — Most Downloaded Plugins (https://www.obsidianstats.com/most-downloaded) | All-time download ranks + counts for the popularity → UC mapping (§7) | +| Obsidian — "The future of plugins" (https://obsidian.md/blog/future-of-plugins/) | Plugin distribution model, restricted mode context | +| Obsidian Help / JSON Canvas (jsoncanvas.org) | Vault format, `.obsidian/` config, Canvas open format | + +Cross-references: `research/260614-roam-deep-dive/findings.md` (DB/API contrast, native +span IDs, query delegation), `research/260613-twiki-deep-dive/findings.md` (file-store +attach UC-40), `spec/UseCaseCatalog.md` (UC-02, UC-05, UC-15, UC-18, UC-28, UC-31, +UC-32, UC-34, UC-36, UC-38, UC-40, UC-51, UC-52), `workplans/SHARD-WP-0002-federation-architecture.md` (T14). + +--- + +## 13. Traceability + +- New UCs: **UC-53, UC-54, UC-55, UC-56** → `spec/UseCaseCatalog.md`. +- Enriched UCs: **UC-15, UC-28, UC-34, UC-36, UC-40, UC-51, UC-52** (and links UC-02, + UC-05, UC-18, UC-32, UC-38). +- Architecture (no UC): dual attachment mode; consume-native-derived-index capability; + non-Markdown content types in the page model; outbound publish; external-writer + tolerance → `SHARD-WP-0002` (T14). +- Distinctive artifact: **plugin-popularity → UC mapping** (§7) — ecosystem demand + evidence, per the research brief. +- Boundary recorded: an Obsidian vault is **one file-backed candidate shard** (the + cleanest, INTENT-named), mapped into the Markdown-first page model; not the federation + layer and not a file-sync target (INTENT Stability Note + not-a-sync-daemon). + diff --git a/research/README.md b/research/README.md index c6bb3b8..d61f27b 100644 --- a/research/README.md +++ b/research/README.md @@ -19,4 +19,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and | 2026-06-13 | `260613-foswiki-deep-dive/` | Foswiki store abstraction, extension API; UC-42/43 | | 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 | | 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 | -| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 | \ No newline at end of file +| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 | +| 2026-06-14 | `260614-obsidian-deep-dive/` | Obsidian — file-over-app vaults, plugin API, ecosystem-popularity → UC signal; UC-53/54/55/56 | \ No newline at end of file diff --git a/spec/UseCaseCatalog.md b/spec/UseCaseCatalog.md index fb1c31e..1a64ae2 100644 --- a/spec/UseCaseCatalog.md +++ b/spec/UseCaseCatalog.md @@ -6,8 +6,8 @@ Promoted from `research/260608-c2-wiki-origins/`, `research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`, `research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`, `research/260613-twiki-deep-dive/`, `research/260613-foswiki-deep-dive/`, -`research/260614-xanadu-deep-dive/`, `research/260614-zigzag-deep-dive/`, and -`research/260614-roam-deep-dive/`. +`research/260614-xanadu-deep-dive/`, `research/260614-zigzag-deep-dive/`, +`research/260614-roam-deep-dive/`, and `research/260614-obsidian-deep-dive/`. See InfoTechPrimers on coulomb.social for use-case catalog conventions. ## Conventions @@ -137,7 +137,9 @@ detection. read-only, or retired shard into an active information space. **Source:** federation **Notes:** Caulfield bounded conversation / reverse bit-rot. Optional space -lifecycle policy. Complements UC-02 attach and UC-26 fork at scale. +lifecycle policy. Complements UC-02 attach and UC-26 fork at scale. Obsidian's +**Importer** (1.4M) plugin (Evernote/Notion/Bear/Apple Notes → Markdown) shows demand +for importing from foreign tools (`research/260614-obsidian-deep-dive/findings.md` §7). **Priority:** Later ### UC-29 — Remix with portable attribution @@ -225,7 +227,10 @@ topic text file*, so structure is diffable rather than locked in a DB (`research/260613-twiki-deep-dive/findings.md` §2.3). Roam is the modern exemplar: `key:: value` attributes over a datom graph, queryable via Datalog (`research/260614-roam-deep-dive/findings.md` §3, §4) — structure to preserve via -sidecar metadata, and a candidate for query delegation (UC-52). +sidecar metadata, and a candidate for query delegation (UC-52). Obsidian shows the +**git-diffable in-file** variant: YAML frontmatter/properties live in the Markdown text +and are queried by the Dataview plugin (`research/260614-obsidian-deep-dive/findings.md` +§3) — structure as portable text, not DB state. **Priority:** Later ### UC-35 — Attach a shard with coarse write granularity @@ -257,7 +262,10 @@ is the journal authoritative or a mirror reconciled back to the engine internal RCS table (`xwikircs`) is a concrete instance (`research/260613-xwiki-deep-dive/findings.md` §2.4). This UC is *supplementation* (the engine's past is DB-locked); where history is already in an open file format -(TWiki RCS `.txt,v`) it can instead be **imported** — see UC-41. +(TWiki RCS `.txt,v`) it can instead be **imported** — see UC-41. Demand evidence: +**Obsidian Git is a top-7 plugin (2.7M)** — users manually bolt portable git history +onto a vault that has none natively (`research/260614-obsidian-deep-dive/findings.md` +§5, §7), exactly the gap the coordination journal fills as core. **Priority:** Later ### UC-37 — Attach a static engine export as a read-only backup shard @@ -328,7 +336,10 @@ the choice (`research/260613-twiki-deep-dive/findings.md` §5–§6, §8 Q1). Di from UC-37 (no live origin) and broader than UC-02's generic attach. Foswiki's **PlainFile** store (timestamped whole-version copies, no RCS) is a cleaner direct-attach target than RCS diffs (`research/260613-foswiki-deep-dive/findings.md` -§2.2). +§2.2). An **Obsidian vault** is the cleanest case of all — a plain Markdown folder, +Markdown-first and git-friendly — and uniquely **dual-attachable**: file-store direct +*or* via an in-app plugin adapter for write-through + live file events (UC-53, +`research/260614-obsidian-deep-dive/findings.md` §4, §10). **Priority:** Later ### UC-41 — Import an engine's native file history into the coordination journal @@ -487,7 +498,10 @@ opaque per-block UID, public and referenceable (`research/260614-roam-deep-dive/ §2, §7). Enables UC-44/45 at sub-page granularity; falls back to content-fingerprint (UC-46) or path+range where no native ID exists. Open: use native ID directly or wrap in a shard-scoped address to avoid cross-shard collision and survive projection (findings -§11 Q1). +§11 Q1). Obsidian shows the **text-embedded** variant: `^block-id` and heading anchors +live in the Markdown text — git-diffable and portable (survives a file copy) but opt-in +(`research/260614-obsidian-deep-dive/findings.md` §2), vs Roam's mandatory DB-minted +UID. **Priority:** Later ### UC-52 — Delegate derived views to a shard's native query engine @@ -500,7 +514,68 @@ recent changes, typed queries) by delegating to it instead of scanning the proje (`research/260614-roam-deep-dive/findings.md` §4). Makes "native query" an adapter capability the orchestrator can key off (vs. computing views itself over the projection). Operationalizes the ZigZag "dimensions + rasters" insight -(`research/260614-zigzag-deep-dive/findings.md` §5) and relates to UC-05/UC-34. +(`research/260614-zigzag-deep-dive/findings.md` §5) and relates to UC-05/UC-34. Obsidian +nuance: query can be an *ecosystem plugin* (Dataview), **not core** — so "native query" +is adapter/plugin-provided and must not be assumed +(`research/260614-obsidian-deep-dive/findings.md` §7). +**Priority:** Later + +### UC-53 — Attach a local-first Markdown vault with a live concurrent native editor + +**Actor:** Maintainer +**Goal:** Attach an Obsidian-style local Markdown vault as a file-backed shard while its +own app continues to edit the files concurrently — watching for external changes, +re-projecting, and treating the app's config dir as opaque. +**Source:** obsidian, intent +**Notes:** Obsidian "file over app": vault = folder of `.md` + `.obsidian/` config; +files are canonical (`research/260614-obsidian-deep-dive/findings.md` §1). Cleanest +file-store attach (UC-40) — and uniquely **dual-mode**: zero-config direct attach +(read/projection/overlay) *or* an in-app adapter (Obsidian plugin) for write-through + +live `vault` events (UC-38, findings §4). Needs **external-writer tolerance** +(file-watch, re-project, conflict-with-live-app) distinct from multi-user write +conflicts. Exclude `.obsidian/` as shard-local config, not page content. +**Priority:** Later + +### UC-54 — Define a page as a live query over the union + +**Actor:** Author +**Goal:** Author a page whose body is materialized from a saved query over the union +(e.g. "all open tasks", "all pages tagged X"), refreshed as the union changes. +**Source:** obsidian, roam, intent +**Notes:** The Obsidian **Dataview** (4.4M downloads) / **Tasks** (3.6M) pattern, and +Roam `{{query}}` — query-defined dynamic pages +(`research/260614-obsidian-deep-dive/findings.md` §7). Distinct from UC-44 (reference/ +span manifest) and UC-52 (delegating *computation* of an existing view): here the *page +itself* is a query. Open: core page type vs. adapter vs. reference-UI/plugin (findings +§11 Q2). May delegate execution to a shard's native query engine (UC-52). +**Priority:** Later + +### UC-55 — Carry non-Markdown content types as typed or opaque assets + +**Actor:** Maintainer or author +**Goal:** Attach and present a shard's non-Markdown content — drawings (Excalidraw), +spatial canvases (JSON Canvas), images, attachments — as typed or opaque assets with +provenance, without flattening them into Markdown or dropping them. +**Source:** obsidian, intent +**Notes:** The **#1 Obsidian plugin is Excalidraw (6.4M)** — users keep non-Markdown +content in "Markdown" vaults (`research/260614-obsidian-deep-dive/findings.md` §7). +Extends UC-34's no-lossy-flatten rule from structured-text to binary/spatial content. +Pushes the **wiki page model**: page vs. typed asset vs. opaque blob — a page-model +decision, not just adapter config (findings §10). JSON Canvas is an open format worth +first-class support. +**Priority:** Later + +### UC-56 — Publish a curated projection to an external read-only target + +**Actor:** Maintainer +**Goal:** Publish a selected projection of the union (or a shard) to an external +read-only target — a static site or hosted page set — preserving provenance. +**Source:** obsidian, intent +**Notes:** The Obsidian Publish / **Quartz** / Digital Garden pattern — outbound +publish (`research/260614-obsidian-deep-dive/findings.md` §7, §10). Formalizes the +`publish` capability from INTENT's adapter-capability list; **complements UC-37** +(inbound static-export *attach*) as its outbound mirror. Open: core vs. publish-adapter +family; interaction with overlays and projection freshness (findings §11 Q4). **Priority:** Later --- @@ -586,7 +661,9 @@ shard-wiki adds Git history as structural safety net beyond social norms. an agreed layout or subtree. **Source:** yawex **Notes:** yawex blueprint pages. Distinct from UC-08 (blank capture) and UC-10 -(refactoring existing discussion). +(refactoring existing discussion). Obsidian's popular **Templater** (4.6M) / **QuickAdd** +(1.9M) plugins show templated creation is a top real-world need +(`research/260614-obsidian-deep-dive/findings.md` §7). **Priority:** Later ### UC-16 — Append or comment without full edit @@ -741,6 +818,10 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD. | UC-50 | | | | § | ✓ | | UC-51 | | | | § | ✓ | | UC-52 | | | | § | ✓ | +| UC-53 | | | | ¶ | ✓ | +| UC-54 | | | | ¶ | ✓ | +| UC-55 | | | | ¶ | ✓ | +| UC-56 | | | | ¶ | ✓ | | UC-08 | ✓ | | | | UC-09 | ✓ | | | | UC-10 | ✓ | | | @@ -916,6 +997,50 @@ Markdown-first page model, not adopted as a substrate and never the federation l native-query capabilities, block↔page mapping, and Roam as a second engine-hosts-adapter exemplar alongside XWiki. +### obsidian mapping + +(¶ UC-53–UC-56 are placed in the **wikiengines** matrix column as the nearest existing +source — Obsidian is a shipped tool — but their true lineage is the **Obsidian deep +dive**, `research/260614-obsidian-deep-dive/findings.md`.) + +| Obsidian mechanism (findings §) | Catalog UC | +|---------------------------------|------------| +| File-over-app vault; cleanest file-backed Markdown shard, dual-attachable (§1, §4) | UC-53 (new); enriches UC-40, UC-02, UC-38 | +| Dataview/Tasks: a page materialized from a query (§7) | UC-54 (new) | +| Excalidraw/Canvas/attachments: non-Markdown content (§7) | UC-55 (new) | +| Obsidian Publish/Quartz: outbound publish (§7) | UC-56 (new) | +| In-file `^block-id` / heading anchors = git-diffable opt-in span IDs (§2) | UC-51 (enriched) | +| YAML frontmatter/properties = git-diffable in-file structured data (§3) | UC-34 (enriched) | +| Git plugin (top-7) = bolt-on portable history (§5, §7) | UC-36 (enriched) | +| Query is a plugin (Dataview), not core (§7) | UC-52 (enriched) | +| Importer plugin = import from foreign tools (§7) | UC-28 (enriched) | +| Templater/QuickAdd = templated creation (§7) | UC-15 (enriched) | +| MetadataCache: files-canonical / index-derived (§1, §6) | reinforces UC-05/17–20 derived-view model | + +#### Plugin-popularity → use-case signal + +Per the research brief (let ecosystem popularity inform the catalog). All-time download +ranks read as **demand evidence** — full table in +`research/260614-obsidian-deep-dive/findings.md` §7. Headlines: + +- **#1 is drawings** (Excalidraw 6.4M) → non-Markdown content is a *primary* real-world + use, not an edge case → **UC-55**. +- **Query-as-DB** (Dataview 4.4M, Tasks 3.6M) is top-tier but an *add-on* → query is an + adapter/plugin capability (**UC-52**) and motivates query-defined pages (**UC-54**). +- **Git is top-7** (2.7M) → users manually bolt on portable history → direct demand for + the coordination journal (**UC-36**). +- **Sync-to-anywhere** (Remotely Save 2.0M: S3/Dropbox/OneDrive/GDrive/WebDAV) → + demand to connect a vault to the heterogeneous backends INTENT targets — while + reaffirming the **not-a-file-sync-daemon** boundary (attach as shards, don't mirror). + +Note: an Obsidian vault is **one file-backed candidate shard** — the cleanest, +INTENT-named, and the file-store counterpart to the DB-backed Roam shard — mapped into +the Markdown-first page model; **not** the federation layer and **not** a file-sync +target (`research/260614-obsidian-deep-dive/findings.md` §8.2). Architecture logged for +`SHARD-WP-0002` (T14): dual attachment mode, consume-native-derived-index capability, +non-Markdown content types in the page model, outbound publish, external-writer +tolerance. + --- ## Open questions @@ -936,4 +1061,8 @@ engine-hosts-adapter exemplar alongside XWiki. model — clones, a parallel graph index, or both? (ZigZag dive §10.) 9. Is **in-engine adapter hosting** (e.g. a Roam Depot extension for write-through) an accepted deployment mode generally, or do we restrict API-only backends to - read/projection/overlay-target? (Roam dive §11 Q2; ties UC-38, UC-50.) \ No newline at end of file + read/projection/overlay-target? (Roam dive §11 Q2; ties UC-38, UC-50.) +10. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a + Markdown stub, opaque blob with provenance, or a pluggable content-type registry? +11. Is a **query-defined page** (UC-54) a core page type, an adapter feature, or a + reference-UI/plugin concern? (Shared with catalog Q7.) \ No newline at end of file