generated from coulomb/repo-seed
research: Joplin deep dive (SQLite-local/Markdown-on-sync, interchange-format attach, E2EE content opacity); UC-60/61
Architecturally distinct from the Obsidian/Roam/Notion trio: content is Markdown, the local store is SQLite, and the sync representation is a folder of per-item Markdown+metadata files on a backend of choice (filesystem, WebDAV, Nextcloud, Dropbox, OneDrive, S3, Joplin Server/Cloud). Two new findings: (1) attach the sync/interchange mirror, not the app or the SQLite store, offline and app-independent, landing on INTENT's WebDAV/Nextcloud/S3 backends (UC-60, distinct from native on-disk store UC-40 and app-files UC-53); (2) E2EE => content opacity, a proposed twelfth capability spectrum for encrypted-at-rest shards (UC-61). Also: Joplin is the file-sync-daemon boundary case (attach the mirror as pages, never re-sync); store-minted page-level :/id links (addressing spectrum middle); dual surface (plugin host + local Data API on localhost:41184). Enriched UC-31/36/38/40/51/55. Catalog now 61 UCs. Architecture for SHARD-WP-0002 T11/T14: interchange-format attach, content-opacity field, local-REST sub-mode, format-aware file-store profiles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
4
SCOPE.md
4
SCOPE.md
@@ -19,9 +19,9 @@ Learnings update both SCOPE and INTENT where necessary.
|
|||||||
|-------|-------|
|
|-------|-------|
|
||||||
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
|
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
|
||||||
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
|
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
|
||||||
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion deep dives (`research/260614-*/`) |
|
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin deep dives & shard-spectrum synthesis (`research/260614-*/`) |
|
||||||
| Demand | NetKingdom integration asks captured, not yet negotiated |
|
| Demand | NetKingdom integration asks captured, not yet negotiated |
|
||||||
| Spec | Architecture blueprint drafted; UseCaseCatalog 59 UCs from research; PRD/TSD scaffolds |
|
| Spec | Architecture blueprint drafted; UseCaseCatalog 61 UCs from research; PRD/TSD scaffolds |
|
||||||
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract) |
|
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract) |
|
||||||
|
|
||||||
## In Scope (today)
|
## In Scope (today)
|
||||||
|
|||||||
49
research/260614-joplin-deep-dive/README.md
Normal file
49
research/260614-joplin-deep-dive/README.md
Normal file
@@ -0,0 +1,49 @@
|
|||||||
|
# 260614 — Joplin deep dive (SQLite-local, Markdown-on-sync, interchange-format attach)
|
||||||
|
|
||||||
|
Date: 2026-06-14
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
A focused study of **Joplin** — the open-source Markdown note app — read through
|
||||||
|
shard-wiki's lens. Joplin looks like "open-source Obsidian" but is a genuinely different
|
||||||
|
point in the space: **content is Markdown, the local store is SQLite, and the *sync
|
||||||
|
representation* is a folder of per-item Markdown+metadata files on a backend you choose**
|
||||||
|
(filesystem, WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). It
|
||||||
|
also ships **end-to-end encryption** and is itself a **sync layer over heterogeneous
|
||||||
|
storage**.
|
||||||
|
|
||||||
|
Distinctive material:
|
||||||
|
- **Architecture** — SQLite local store (documented schema); Markdown content;
|
||||||
|
notebooks → notes → resources + tags + to-dos; 32-char item IDs; `:/<id>` note links
|
||||||
|
(rename-stable, page-level)
|
||||||
|
- **Sync** — serialized to plain-text items on a chosen target; tombstones; **conflict
|
||||||
|
notes** (keep-both) → enables the **"attach the sync mirror, not the app"** pattern on
|
||||||
|
WebDAV/Nextcloud/S3
|
||||||
|
- **E2EE** — items encrypted before leaving the device → **content opacity** (ciphertext
|
||||||
|
at rest)
|
||||||
|
- **Extension surfaces** — a **plugin API** (`joplin.data`/`workspace`/`contentScripts`/
|
||||||
|
views) *and* a **local Data API** (REST on `localhost:41184`, token) used by the Web
|
||||||
|
Clipper; plus JEX/Markdown/RAW export
|
||||||
|
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
| Path | Role |
|
||||||
|
|------|------|
|
||||||
|
| `findings.md` | Architecture, sync/interchange attach, E2EE/content-opacity, plugin + Data API surfaces, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md`
|
||||||
|
(UC-60 attach a tool's documented sync/interchange representation on third-party storage
|
||||||
|
without the app; UC-61 attach an encrypted-at-rest shard with content opacity);
|
||||||
|
UC-31/36/38/40/51/55 enriched. Logged for `SHARD-WP-0002` (T11, T14): the
|
||||||
|
interchange/sync-representation attach surface, **content opacity** as a proposed twelfth
|
||||||
|
capability spectrum, a local-REST attach sub-mode, and format-aware file-store profiles.
|
||||||
|
|
||||||
|
**Key takeaways recorded:** the best attach surface is often a tool's **interchange/sync
|
||||||
|
representation** (offline, app-independent — Joplin items on Nextcloud/WebDAV/S3, the
|
||||||
|
INTENT-named backends), not its live store or API; **encryption-at-rest** demands a
|
||||||
|
content-opacity capability so encrypted shards degrade to backup/structure-shell; and
|
||||||
|
Joplin is the **file-sync-daemon boundary case** — shard-wiki attaches its mirror as
|
||||||
|
pages and never re-syncs (not-a-sync-daemon).
|
||||||
|
</content>
|
||||||
269
research/260614-joplin-deep-dive/findings.md
Normal file
269
research/260614-joplin-deep-dive/findings.md
Normal file
@@ -0,0 +1,269 @@
|
|||||||
|
# Findings — Joplin: SQLite locally, Markdown-on-sync, and the interchange-format attach
|
||||||
|
|
||||||
|
Date: 2026-06-14
|
||||||
|
Source kind: **modern shipped product** — an open-source Markdown note app; a *candidate
|
||||||
|
shard* whose distinctive trait is a **documented file-based sync/interchange format on
|
||||||
|
third-party storage** plus optional **end-to-end encryption**
|
||||||
|
Lens: shard-wiki — attachment surfaces, the "attach the sync mirror not the app"
|
||||||
|
pattern, content opacity (encryption), note-level stable IDs, and the file-sync-daemon
|
||||||
|
boundary
|
||||||
|
|
||||||
|
> Why Joplin earns a dive after Obsidian/Roam/Notion. It looks like "Obsidian but
|
||||||
|
> open-source," yet its architecture is a genuinely different point in the space:
|
||||||
|
> **content is Markdown but the local store is SQLite, and the *sync representation* is
|
||||||
|
> a folder of per-item Markdown+metadata files on a backend you choose** (filesystem,
|
||||||
|
> WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). That sync
|
||||||
|
> mirror — not the app, not the SQLite DB — is the interesting attach surface, and it
|
||||||
|
> lands squarely on the **WebDAV/Nextcloud backends INTENT already names**. Joplin also
|
||||||
|
> ships **E2EE**, forcing the question of a shard whose content is *opaque ciphertext*,
|
||||||
|
> and it is itself a **sync layer over heterogeneous storage** — the very "file-sync
|
||||||
|
> daemon" shard-wiki says it is *not*, making Joplin a sharp boundary case.
|
||||||
|
|
||||||
|
Contrast set: Obsidian (file-over-app: native store *is* files), Roam (client DB,
|
||||||
|
in-app API only), Notion (hosted DB, external API only), TWiki/Foswiki (file+RCS native
|
||||||
|
store). Joplin is none of these exactly — **DB-local, files-on-sync** — which is the
|
||||||
|
whole point.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Core architecture — SQLite local, Markdown content
|
||||||
|
|
||||||
|
- **Local store:** a **SQLite database** (`~/.config/joplin-desktop/database.sqlite`),
|
||||||
|
schema documented. So, unlike Obsidian, the *native* local store is **not** plain
|
||||||
|
files — you cannot point shard-wiki at a folder of `.md` and read the live store
|
||||||
|
(you'd read SQLite, fragile while the app runs).
|
||||||
|
- **Content:** notes are **Markdown** (Joplin-flavored: `:/imageId` resource links,
|
||||||
|
internal `:/<id>` note links, checkboxes/to-dos). Markdown-first — INTENT-aligned at
|
||||||
|
the *content* layer even though the *storage* layer is a DB.
|
||||||
|
- **Model:** `folders` (notebooks, hierarchical) → `notes` → `resources` (attachments);
|
||||||
|
cross-cutting `tags`; notes can be **to-dos**. Every item has a **32-char hex ID**;
|
||||||
|
notes link by ID (`:/<id>`), so links survive rename/move (store-minted, page-level).
|
||||||
|
- **Clients:** desktop (Electron), mobile, and a **terminal/CLI** app — multiple clients
|
||||||
|
edit one logical store, reconciled by sync (§2).
|
||||||
|
- **Revisions:** Joplin keeps **internal note history** locally — not git, not portable
|
||||||
|
→ a *supplement* case (UC-36), like Notion/Confluence.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Sync — the distinctive layer (and the attach opportunity)
|
||||||
|
|
||||||
|
Joplin's headline feature is **sync to a backend of your choice**. On sync, the SQLite
|
||||||
|
store is serialized to a **folder of plain-text items** — one file per note / notebook /
|
||||||
|
tag / resource-metadata, each carrying the **Markdown body plus a metadata block**
|
||||||
|
(`id`, `parent_id`, `type_`, `created_time`, `updated_time`, …) — written to the chosen
|
||||||
|
target: **filesystem, WebDAV, Nextcloud, Dropbox, OneDrive, S3, Joplin Server, Joplin
|
||||||
|
Cloud**. Deletions propagate via tombstones; concurrent edits across clients produce
|
||||||
|
**conflict notes** (keep-both, not silent overwrite).
|
||||||
|
|
||||||
|
Two consequences for shard-wiki:
|
||||||
|
|
||||||
|
1. **Attach the sync mirror, not the app (UC-60).** A Nextcloud/WebDAV/S3/filesystem
|
||||||
|
directory full of Joplin sync items is a **file-store shard** — readable *without
|
||||||
|
Joplin running and without touching SQLite* — provided the adapter understands the
|
||||||
|
**documented item format** (body + metadata footer, `:/id` links, resource items).
|
||||||
|
This is a new attach pattern: the backend's **interchange/sync representation** as
|
||||||
|
the attach surface, distinct from a native on-disk store (UC-40) or app files (UC-53).
|
||||||
|
It also realizes INTENT's WebDAV/Nextcloud participants concretely.
|
||||||
|
2. **Joplin is a file-sync layer over heterogeneous storage** — exactly what shard-wiki
|
||||||
|
says it is *not* (INTENT: "not a file-sync daemon"). Joplin syncs **one** logical
|
||||||
|
store across backends; shard-wiki **federates many** logical stores with wiki-page
|
||||||
|
semantics. Attaching a Joplin sync mirror means **reading its file representation as
|
||||||
|
pages**, never re-driving Joplin's sync — and never becoming a second sync engine
|
||||||
|
over the same target (don't double-sync).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. End-to-end encryption — content opacity
|
||||||
|
|
||||||
|
Joplin offers **E2EE**: items are encrypted **before** leaving the device, so the sync
|
||||||
|
target holds **ciphertext** regardless of provider. (Security wart, noted: the master
|
||||||
|
key is stored *in clear* in the local SQLite — irrelevant to attaching the mirror, but
|
||||||
|
a reason never to treat the local DB as trusted.)
|
||||||
|
|
||||||
|
For shard-wiki this introduces a capability dimension the prior dives did not:
|
||||||
|
**content opacity**. An encrypted Joplin sync target can be attached only as a
|
||||||
|
**backup/mirror/structure-shell** participant — item IDs, counts, and change events may
|
||||||
|
be visible, but **bodies are undecryptable without keys**. The adapter must **never
|
||||||
|
present ciphertext as readable content** and must degrade to "present-but-opaque" with
|
||||||
|
provenance. This extends the synthesis capability spectra with a proposed twelfth field,
|
||||||
|
**content opacity** (`plaintext → encrypted-at-rest/opaque`), feeding `SHARD-WP-0002`
|
||||||
|
T11 (§9).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Extension surfaces — a plugin host *and* a local REST API
|
||||||
|
|
||||||
|
Joplin exposes **two** programmatic surfaces (plus export):
|
||||||
|
|
||||||
|
**A. Plugin API (in-app host).** TypeScript plugins, distributed via **npm + a plugin
|
||||||
|
repository**, loaded by the app:
|
||||||
|
- `joplin.data` — CRUD over the data model, shaped like REST (GET/POST/PUT/DELETE) on
|
||||||
|
`notes`/`folders`/`tags`/`resources`.
|
||||||
|
- `joplin.workspace` — selection + events (note selected, content changed, sync events).
|
||||||
|
- `joplin.contentScripts` — **markdown-it** render plugins and **CodeMirror** editor
|
||||||
|
extensions (the syntax/rendering extension point).
|
||||||
|
- `joplin.views` — panels, dialogs, toolbar/menu items, custom editors;
|
||||||
|
`joplin.commands`, `joplin.settings`.
|
||||||
|
|
||||||
|
**B. Data API (local REST).** A REST service the desktop app serves on
|
||||||
|
**`localhost:41184`** (the "clipper server"), **token-authenticated**, used by the Web
|
||||||
|
Clipper and any local integration: endpoints `/notes`, `/folders`, `/tags`,
|
||||||
|
`/resources` (incl. `/resources/:id/file`), `/revisions`, `/search`; verbs
|
||||||
|
GET/POST/PUT/DELETE; pagination (`page`, `limit`≤100, `order_by`, `order_dir`). Notably
|
||||||
|
**plugins can use this API even when the clipper server is off**.
|
||||||
|
|
||||||
|
**C. Export** — **JEX** (tar of raw items), **Markdown directory**, **RAW** (Joplin
|
||||||
|
export directory), **ENEX** import (Evernote). Snapshot import/attach surface (UC-28/37).
|
||||||
|
|
||||||
|
So Joplin spans attachment modes: **in-app plugin host** (like Roam/Obsidian-plugin),
|
||||||
|
**local-REST** (like Notion's external API but *localhost + app-must-run* — a sub-mode),
|
||||||
|
the **sync-mirror file-store** (§2, the novel one), and **export snapshots**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Joplin as a shard — capability profile
|
||||||
|
|
||||||
|
| Capability | Joplin | Notes for the adapter contract |
|
||||||
|
|------------|--------|--------------------------------|
|
||||||
|
| Read | **yes** | sync-mirror files (no app) · local Data API (app running) · plugin · export |
|
||||||
|
| Write | **yes** | Data API / plugin (`joplin.data`); writing the sync mirror directly is risky (Joplin owns that format) |
|
||||||
|
| Write granularity | **per-note (page)** | notebooks/notes/resources; not block-level |
|
||||||
|
| Identity / addressing | **32-char item ID; `:/<id>` links** | store-minted, **page-level**, survives rename (between Obsidian path and Roam block-UUID) (UC-51) |
|
||||||
|
| Structure | notebooks (hierarchy) + tags + to-dos | modest; metadata in the sync item footer (in-text on the mirror) |
|
||||||
|
| History | **internal revisions, not portable** | supplement via coordination journal (UC-36) |
|
||||||
|
| Native query | search API (FTS) | delegate text search; no datalog/DB-query (weaker than Roam/Notion) |
|
||||||
|
| Subscribe | sync is **poll-based**; conflict notes | poll; conflicts surface as keep-both items (UC-31, UC-07) |
|
||||||
|
| Content opacity | **optional E2EE → ciphertext** | **new dimension**: opaque-at-rest; mirror usable only as backup/structure without keys (UC-61) |
|
||||||
|
| Transclusion | none (note-level links only) | — |
|
||||||
|
| Diff / merge | none native | git-level if mirror is on a git-backed target |
|
||||||
|
| Publish | via export / static-site pipelines | outbound (UC-56) |
|
||||||
|
| Content types | Markdown + **resources** (attachments) | non-Markdown assets with IDs (UC-55) |
|
||||||
|
| Attach modes | sync-mirror file-store · local-REST · plugin · export | multi-mode, per-binding (UC-60, UC-57, UC-38) |
|
||||||
|
|
||||||
|
Verdict: a solid **Markdown-first candidate shard** whose best attach surface is the
|
||||||
|
**sync mirror on a file/WebDAV/S3 target** (offline, app-independent, INTENT-named),
|
||||||
|
with the local Data API or a plugin for live write-through. Standout: the
|
||||||
|
interchange-format attach (UC-60) and the encryption/opacity case (UC-61). Limits: DB
|
||||||
|
local store, note-level (not block) addressing, internal-only history, weak query.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||||
|
|
||||||
|
### 6.1 Reinforcements
|
||||||
|
|
||||||
|
- **Interchange-format attach** validates that a shard's *documented sync/export
|
||||||
|
representation* — not just its live store or API — is a legitimate, often *best*,
|
||||||
|
attach surface (offline, app-independent). Generalizes beyond Joplin (UC-60).
|
||||||
|
- **WebDAV/Nextcloud/S3 become concrete** participants via Joplin's sync targets — the
|
||||||
|
backends INTENT names, now with a real format to parse.
|
||||||
|
- **Conflict-as-data** (Joplin conflict notes = keep-both) is union-without-erasure in
|
||||||
|
the wild (reinforces UC-07, divergence; consensus policy T9).
|
||||||
|
- **Store-minted, rename-stable note IDs** (`:/id`) confirm the addressing spectrum's
|
||||||
|
middle (page-level stable ID) between path (Obsidian) and block-UUID (Roam/Notion)
|
||||||
|
(UC-51).
|
||||||
|
|
||||||
|
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||||
|
|
||||||
|
1. **Joplin is a sync daemon; shard-wiki is not.** Attach the **mirror as pages**; never
|
||||||
|
re-implement Joplin's sync or run a competing sync over the same target. (INTENT
|
||||||
|
not-a-file-sync-daemon.)
|
||||||
|
2. **DB-local; don't read SQLite live.** The native store is a DB; treat it as opaque.
|
||||||
|
Use the sync mirror, the Data API, or a plugin — not the live `database.sqlite`.
|
||||||
|
3. **Joplin owns the sync format; treat the mirror read-mostly.** Writing items into the
|
||||||
|
sync folder behind Joplin's back risks corruption/conflict; prefer overlay/projection
|
||||||
|
or write through the Data API/plugin (overlay before mutation, no silent mutation).
|
||||||
|
4. **Encrypted shards are opaque.** Never present ciphertext as content; degrade to
|
||||||
|
backup/structure with provenance (UC-61, graceful degradation).
|
||||||
|
|
||||||
|
### 6.3 What Joplin teaches that shard-wiki should keep
|
||||||
|
|
||||||
|
- Add **"interchange/sync representation"** as a recognized attach surface in the
|
||||||
|
contract — sometimes preferable to the live store/API (UC-60).
|
||||||
|
- Add **content opacity** (encryption-at-rest) as a capability-profile field so encrypted
|
||||||
|
shards degrade correctly (UC-61).
|
||||||
|
- Expect **multi-client concurrent editing reconciled by the backend's own sync**
|
||||||
|
(desktop/mobile/CLI) — a different concurrency than single-app file edits (extends
|
||||||
|
UC-53) or multi-user server writes; conflicts may pre-exist as keep-both items.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Use-case seeds → catalog (promoted 2026-06-14)
|
||||||
|
|
||||||
|
Last existing UC is **UC-59**. New UCs **UC-60, UC-61** added; existing UCs enriched.
|
||||||
|
|
||||||
|
| Seed | Catalog action |
|
||||||
|
|------|----------------|
|
||||||
|
| **Attach a tool's documented sync/interchange representation on a third-party storage target** (Joplin items on Nextcloud/WebDAV/S3) as a file-store shard, without the app | **UC-60 (new)** |
|
||||||
|
| **Attach an encrypted-at-rest shard** (E2EE sync target): content opaque without keys; participate as backup/mirror/structure-shell, never presenting ciphertext as readable | **UC-61 (new)** |
|
||||||
|
| Store-minted, rename-stable **note-level** IDs (`:/id`) — the middle of the addressing spectrum | **enriches UC-51** |
|
||||||
|
| Native store can be a **DB while the sync representation is files** → attach surface ≠ native store | **enriches UC-40** |
|
||||||
|
| Internal revisions, not portable → supplement | **enriches UC-36** |
|
||||||
|
| Dual surfaces: in-app plugin host + **local Data API** (localhost, app-running) | **enriches UC-38** (links UC-57) |
|
||||||
|
| Resources/attachments = non-Markdown assets with IDs | **enriches UC-55** |
|
||||||
|
| Poll-based sync; conflict notes (keep-both) | **enriches UC-31** (links UC-07) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Architecture notes for SHARD-WP-0002 (no UC)
|
||||||
|
|
||||||
|
- **Attachment-mode taxonomy gains a refinement:** within file-store, distinguish
|
||||||
|
*native on-disk store* (Obsidian/TWiki, UC-40) from *interchange/sync representation*
|
||||||
|
(Joplin mirror, UC-60). Both are "files," but format ownership and write-safety differ.
|
||||||
|
(T14.)
|
||||||
|
- **Add a "content opacity" capability field** (proposed **twelfth spectrum**:
|
||||||
|
`plaintext → encrypted-at-rest/opaque`) so encrypted shards degrade to
|
||||||
|
backup/structure-shell. Feeds T11 the next time the eleven spectra
|
||||||
|
(`research/260614-shard-spectrum-synthesis/findings.md` §2) are revised. (T11/T14.)
|
||||||
|
- **Local-REST as an attach sub-mode** (localhost, app-must-run, token) sits between
|
||||||
|
in-engine-host (UC-38) and external-API (UC-57) — note in T14.
|
||||||
|
- **Format-aware file-store adapters:** the contract should let a file-store adapter
|
||||||
|
declare a *format profile* (plain Markdown vs Joplin-item vs Foswiki-PlainFile) so the
|
||||||
|
same "folder of files" attach can parse tool-specific item formats. (T11/T14.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Open questions (for spec / workplans)
|
||||||
|
|
||||||
|
1. For an **encrypted shard** (UC-61), what *is* visible without keys — item IDs/counts/
|
||||||
|
timestamps (structure shell) or nothing? Does shard-wiki ever hold keys, or only ever
|
||||||
|
treat such shards as opaque backups?
|
||||||
|
2. Is **writing to a tool's sync mirror** (UC-60) ever safe, or are interchange-format
|
||||||
|
shards **read/projection/overlay-only** by policy (Joplin owns the format)?
|
||||||
|
3. Does shard-wiki parse Joplin's **item format** via a dedicated format profile, or
|
||||||
|
only attach the **export** (JEX/Markdown dir) as a cleaner snapshot (UC-37)?
|
||||||
|
4. How do we avoid **double-sync** when a shard's storage target is itself driven by a
|
||||||
|
sync daemon (Joplin) we don't control?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Sources
|
||||||
|
|
||||||
|
| Source | Used for |
|
||||||
|
|--------|----------|
|
||||||
|
| Joplin Data API reference (https://joplinapp.org/help/api/references/rest_api/) | Local REST on `localhost:41184`, token auth, endpoints, 32-char IDs, item fields, pagination |
|
||||||
|
| Joplin Plugin API docs (https://joplinapp.org/api/references/plugin_api/classes/joplin.html) | `joplin.data` / `joplin.workspace` / `joplin.contentScripts` / views / settings |
|
||||||
|
| Joplin — Extending Joplin (https://joplinapp.org/help/api/) | Plugin distribution (npm + repository), Data API vs plugin access |
|
||||||
|
| Joplin — WebDAV / Nextcloud sync (https://joplinapp.org/help/apps/sync/nextcloud/) | Sync targets; items as plain-text files on the target |
|
||||||
|
| Joplin FAQ + community (https://joplinapp.org/help/faq/) | SQLite local store; Markdown content; E2EE before leaving device |
|
||||||
|
| Obsidian vs Joplin comparison (https://petronellatech.com/blog/obsidian-vs-joplin/) | Architecture contrast (SQLite+sync vs file-over-app), E2EE framing |
|
||||||
|
|
||||||
|
Cross-references: `research/260614-obsidian-deep-dive/findings.md` (file-over-app
|
||||||
|
contrast), `research/260614-notion-deep-dive/findings.md` (external-API, scoped grant),
|
||||||
|
`research/260614-shard-spectrum-synthesis/findings.md` (the spectra this extends),
|
||||||
|
`spec/UseCaseCatalog.md` (UC-31, UC-36, UC-38, UC-40, UC-51, UC-55, UC-57),
|
||||||
|
`workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Traceability
|
||||||
|
|
||||||
|
- New UCs: **UC-60, UC-61** → `spec/UseCaseCatalog.md`.
|
||||||
|
- Enriched UCs: **UC-31, UC-36, UC-38, UC-40, UC-51, UC-55** (links UC-07, UC-57).
|
||||||
|
- Architecture (no UC): interchange/sync-representation attach surface; **content
|
||||||
|
opacity** as a proposed twelfth capability spectrum; local-REST sub-mode; format-aware
|
||||||
|
file-store profiles → `SHARD-WP-0002` (T11, T14).
|
||||||
|
- Boundary recorded: Joplin is a **sync daemon over one store**; shard-wiki attaches its
|
||||||
|
**mirror as pages** and does not re-sync — and treats DB-local store and encrypted
|
||||||
|
content as opaque (INTENT not-a-sync-daemon, graceful degradation, no silent mutation).
|
||||||
|
</content>
|
||||||
@@ -22,4 +22,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and
|
|||||||
| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 |
|
| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 |
|
||||||
| 2026-06-14 | `260614-obsidian-deep-dive/` | Obsidian — file-over-app vaults, plugin API, ecosystem-popularity → UC signal; UC-53/54/55/56 |
|
| 2026-06-14 | `260614-obsidian-deep-dive/` | Obsidian — file-over-app vaults, plugin API, ecosystem-popularity → UC signal; UC-53/54/55/56 |
|
||||||
| 2026-06-14 | `260614-notion-deep-dive/` | Notion — closed block-DB SaaS, external REST API only, database-as-pages; UC-57/58/59 |
|
| 2026-06-14 | `260614-notion-deep-dive/` | Notion — closed block-DB SaaS, external REST API only, database-as-pages; UC-57/58/59 |
|
||||||
| 2026-06-14 | `260614-shard-spectrum-synthesis/` | Synthesis — shard family matrix + eleven capability spectra across nine systems; feeds SHARD-WP-0002 T11–T16 |
|
| 2026-06-14 | `260614-shard-spectrum-synthesis/` | Synthesis — shard family matrix + eleven capability spectra across nine systems; feeds SHARD-WP-0002 T11–T16 |
|
||||||
|
| 2026-06-14 | `260614-joplin-deep-dive/` | Joplin — SQLite-local/Markdown-on-sync, interchange-format attach, E2EE content opacity; UC-60/61 |
|
||||||
@@ -7,8 +7,8 @@ Promoted from `research/260608-c2-wiki-origins/`,
|
|||||||
`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`,
|
`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`,
|
||||||
`research/260613-twiki-deep-dive/`, `research/260613-foswiki-deep-dive/`,
|
`research/260613-twiki-deep-dive/`, `research/260613-foswiki-deep-dive/`,
|
||||||
`research/260614-xanadu-deep-dive/`, `research/260614-zigzag-deep-dive/`,
|
`research/260614-xanadu-deep-dive/`, `research/260614-zigzag-deep-dive/`,
|
||||||
`research/260614-roam-deep-dive/`, `research/260614-obsidian-deep-dive/`, and
|
`research/260614-roam-deep-dive/`, `research/260614-obsidian-deep-dive/`,
|
||||||
`research/260614-notion-deep-dive/`.
|
`research/260614-notion-deep-dive/`, and `research/260614-joplin-deep-dive/`.
|
||||||
See InfoTechPrimers on coulomb.social for use-case catalog conventions.
|
See InfoTechPrimers on coulomb.social for use-case catalog conventions.
|
||||||
|
|
||||||
## Conventions
|
## Conventions
|
||||||
@@ -181,7 +181,9 @@ concrete push transport: an engine event bus (`ObservationManager`
|
|||||||
(`research/260613-xwiki-deep-dive/findings.md` §2.5, §8 Q4). Notion adds **webhooks**
|
(`research/260613-xwiki-deep-dive/findings.md` §2.5, §8 Q4). Notion adds **webhooks**
|
||||||
(2026) delivering page/database change events — a SaaS push transport, weighed against
|
(2026) delivering page/database change events — a SaaS push transport, weighed against
|
||||||
eventual consistency and the rate limit (`research/260614-notion-deep-dive/findings.md`
|
eventual consistency and the rate limit (`research/260614-notion-deep-dive/findings.md`
|
||||||
§4).
|
§4). **Joplin** is poll-based and turns concurrent edits into **conflict notes**
|
||||||
|
(keep-both, not silent overwrite) — conflict-as-data, links UC-07
|
||||||
|
(`research/260614-joplin-deep-dive/findings.md` §2).
|
||||||
**Priority:** Later
|
**Priority:** Later
|
||||||
|
|
||||||
### UC-32 — Transclude remote span with live freshness
|
### UC-32 — Transclude remote span with live freshness
|
||||||
@@ -274,7 +276,9 @@ onto a vault that has none natively (`research/260614-obsidian-deep-dive/finding
|
|||||||
§5, §7), exactly the gap the coordination journal fills as core. **Notion** is the
|
§5, §7), exactly the gap the coordination journal fills as core. **Notion** is the
|
||||||
closed-SaaS instance: internal page history bounded by plan, **not portable** and not
|
closed-SaaS instance: internal page history bounded by plan, **not portable** and not
|
||||||
exported as git (`research/260614-notion-deep-dive/findings.md` §4) — a supplementation
|
exported as git (`research/260614-notion-deep-dive/findings.md` §4) — a supplementation
|
||||||
case like Confluence/MediaWiki, not an import case.
|
case like Confluence/MediaWiki, not an import case. **Joplin** keeps internal note
|
||||||
|
revisions locally, likewise not portable (`research/260614-joplin-deep-dive/findings.md`
|
||||||
|
§1) — supplement.
|
||||||
**Priority:** Later
|
**Priority:** Later
|
||||||
|
|
||||||
### UC-37 — Attach a static engine export as a read-only backup shard
|
### UC-37 — Attach a static engine export as a read-only backup shard
|
||||||
@@ -310,7 +314,11 @@ Generalizes beyond XWiki: TWiki's plugin handler API (`beforeSaveHandler`,
|
|||||||
template: an extension default-exports `onload`/`onunload` and drives
|
template: an extension default-exports `onload`/`onunload` and drives
|
||||||
`window.roamAlphaAPI` read (`q`/`pull`) + write (`block.*`/`page.create`)
|
`window.roamAlphaAPI` read (`q`/`pull`) + write (`block.*`/`page.create`)
|
||||||
(`research/260614-roam-deep-dive/findings.md` §5) — note Roam's API runs *inside* the
|
(`research/260614-roam-deep-dive/findings.md` §5) — note Roam's API runs *inside* the
|
||||||
client, so write-through requires in-engine hosting (findings §11 Q2).
|
client, so write-through requires in-engine hosting (findings §11 Q2). **Joplin** offers
|
||||||
|
*two* surfaces: an in-app plugin host (`joplin.data`/`workspace`/`contentScripts`) **and**
|
||||||
|
a **local Data API** (REST on `localhost:41184`, token, app-running) — a local-REST attach
|
||||||
|
sub-mode between in-engine host and external API (UC-57,
|
||||||
|
`research/260614-joplin-deep-dive/findings.md` §4).
|
||||||
**Priority:** Later
|
**Priority:** Later
|
||||||
|
|
||||||
### UC-39 — Attach a wiki-as-application-platform shard (pages as typed records)
|
### UC-39 — Attach a wiki-as-application-platform shard (pages as typed records)
|
||||||
@@ -350,7 +358,10 @@ direct-attach target than RCS diffs (`research/260613-foswiki-deep-dive/findings
|
|||||||
§2.2). An **Obsidian vault** is the cleanest case of all — a plain Markdown folder,
|
§2.2). An **Obsidian vault** is the cleanest case of all — a plain Markdown folder,
|
||||||
Markdown-first and git-friendly — and uniquely **dual-attachable**: file-store direct
|
Markdown-first and git-friendly — and uniquely **dual-attachable**: file-store direct
|
||||||
*or* via an in-app plugin adapter for write-through + live file events (UC-53,
|
*or* via an in-app plugin adapter for write-through + live file events (UC-53,
|
||||||
`research/260614-obsidian-deep-dive/findings.md` §4, §10).
|
`research/260614-obsidian-deep-dive/findings.md` §4, §10). Counter-example: **Joplin**'s
|
||||||
|
native store is a **DB** (SQLite), so its file-attach surface is the *sync/interchange
|
||||||
|
mirror* on a WebDAV/S3 target, not the native store — see UC-60
|
||||||
|
(`research/260614-joplin-deep-dive/findings.md` §2).
|
||||||
**Priority:** Later
|
**Priority:** Later
|
||||||
|
|
||||||
### UC-41 — Import an engine's native file history into the coordination journal
|
### UC-41 — Import an engine's native file history into the coordination journal
|
||||||
@@ -517,7 +528,10 @@ a shard-scoped address to avoid cross-shard collision and survive projection (fi
|
|||||||
live in the Markdown text — git-diffable and portable (survives a file copy) but opt-in
|
live in the Markdown text — git-diffable and portable (survives a file copy) but opt-in
|
||||||
(`research/260614-obsidian-deep-dive/findings.md` §2), vs Roam's mandatory DB-minted
|
(`research/260614-obsidian-deep-dive/findings.md` §2), vs Roam's mandatory DB-minted
|
||||||
UID. Notion is a second store-minted case: per-block **UUID v4**, exposed via the REST
|
UID. Notion is a second store-minted case: per-block **UUID v4**, exposed via the REST
|
||||||
API (`research/260614-notion-deep-dive/findings.md` §1).
|
API (`research/260614-notion-deep-dive/findings.md` §1). **Joplin** marks the spectrum's
|
||||||
|
middle: a store-minted **page-level** 32-char ID with `:/<id>` links that survive
|
||||||
|
rename/move (`research/260614-joplin-deep-dive/findings.md` §1) — coarser than a block
|
||||||
|
UUID, more stable than a path.
|
||||||
**Priority:** Later
|
**Priority:** Later
|
||||||
|
|
||||||
### UC-52 — Delegate derived views to a shard's native query engine
|
### UC-52 — Delegate derived views to a shard's native query engine
|
||||||
@@ -582,7 +596,9 @@ content in "Markdown" vaults (`research/260614-obsidian-deep-dive/findings.md`
|
|||||||
Extends UC-34's no-lossy-flatten rule from structured-text to binary/spatial content.
|
Extends UC-34's no-lossy-flatten rule from structured-text to binary/spatial content.
|
||||||
Pushes the **wiki page model**: page vs. typed asset vs. opaque blob — a page-model
|
Pushes the **wiki page model**: page vs. typed asset vs. opaque blob — a page-model
|
||||||
decision, not just adapter config (findings §10). JSON Canvas is an open format worth
|
decision, not just adapter config (findings §10). JSON Canvas is an open format worth
|
||||||
first-class support.
|
first-class support. **Joplin** resources (attachments, each with an ID, linked `:/<id>`)
|
||||||
|
are the same demand in a sync-mirror shard (`research/260614-joplin-deep-dive/findings.md`
|
||||||
|
§5).
|
||||||
**Priority:** Later
|
**Priority:** Later
|
||||||
|
|
||||||
### UC-56 — Publish a curated projection to an external read-only target
|
### UC-56 — Publish a curated projection to an external read-only target
|
||||||
@@ -652,6 +668,42 @@ provenance/sidecar. Embodies union-without-erasure by making *loss of fidelity*
|
|||||||
Open: report format and where it surfaces (findings §10 Q3).
|
Open: report format and where it surfaces (findings §10 Q3).
|
||||||
**Priority:** Later
|
**Priority:** Later
|
||||||
|
|
||||||
|
### UC-60 — Attach a tool's documented sync / interchange representation
|
||||||
|
|
||||||
|
**Actor:** Maintainer
|
||||||
|
**Goal:** Attach a backend's documented **sync or interchange representation** — a folder
|
||||||
|
of per-item files a tool writes to a third-party storage target — as a file-store shard,
|
||||||
|
without running the tool and without touching its native store.
|
||||||
|
**Source:** joplin, intent
|
||||||
|
**Notes:** Joplin's native store is **SQLite**, but on sync it serializes to per-item
|
||||||
|
**Markdown+metadata files** on filesystem / WebDAV / **Nextcloud** / Dropbox / OneDrive /
|
||||||
|
**S3** (`research/260614-joplin-deep-dive/findings.md` §2). That mirror is attachable
|
||||||
|
offline and app-independently *if* the adapter understands the documented item format
|
||||||
|
(body + metadata footer, `:/id` links, resources). **Distinct from UC-40** (native
|
||||||
|
on-disk store) and **UC-53** (app files *are* the store): here the attach surface is the
|
||||||
|
**interchange/sync representation**, and the tool **owns the format** → treat
|
||||||
|
read/projection/overlay-mostly, never re-sync (INTENT not-a-file-sync-daemon). Realizes
|
||||||
|
INTENT's WebDAV/Nextcloud/S3 participants. Needs a **format profile** in the adapter
|
||||||
|
(findings §8).
|
||||||
|
**Priority:** Later
|
||||||
|
|
||||||
|
### UC-61 — Attach an encrypted-at-rest shard with content opacity
|
||||||
|
|
||||||
|
**Actor:** Maintainer
|
||||||
|
**Goal:** Attach a shard whose stored content is encrypted (E2EE sync target) and
|
||||||
|
participate correctly when bodies are undecryptable — as backup / mirror / structure-
|
||||||
|
shell — without ever presenting ciphertext as readable content.
|
||||||
|
**Source:** joplin, intent
|
||||||
|
**Notes:** Joplin encrypts items **before** they leave the device, so the sync target
|
||||||
|
holds ciphertext (`research/260614-joplin-deep-dive/findings.md` §3). Introduces a new
|
||||||
|
capability dimension — **content opacity** (`plaintext → encrypted-at-rest/opaque`) —
|
||||||
|
proposed as a twelfth spectrum for the adapter contract (findings §8; extends
|
||||||
|
`research/260614-shard-spectrum-synthesis/findings.md` §2). Graceful degradation extreme:
|
||||||
|
IDs/counts/timestamps may be visible while bodies are opaque; never silently surface
|
||||||
|
undecryptable content. Open: does shard-wiki ever hold keys, or only treat such shards as
|
||||||
|
opaque backups (findings §9 Q1)? Ties [[shard-wiki-auth-in-core-decision]].
|
||||||
|
**Priority:** Later
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## B. Knowledge work and collaboration
|
## B. Knowledge work and collaboration
|
||||||
@@ -899,6 +951,8 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD.
|
|||||||
| UC-57 | | | | ◊ | ✓ |
|
| UC-57 | | | | ◊ | ✓ |
|
||||||
| UC-58 | | | | ◊ | ✓ |
|
| UC-58 | | | | ◊ | ✓ |
|
||||||
| UC-59 | | | | ◊ | ✓ |
|
| UC-59 | | | | ◊ | ✓ |
|
||||||
|
| UC-60 | | | | ‖ | ✓ |
|
||||||
|
| UC-61 | | | | ‖ | ✓ |
|
||||||
| UC-08 | ✓ | | |
|
| UC-08 | ✓ | | |
|
||||||
| UC-09 | ✓ | | |
|
| UC-09 | ✓ | | |
|
||||||
| UC-10 | ✓ | | |
|
| UC-10 | ✓ | | |
|
||||||
@@ -1151,6 +1205,36 @@ an operational-envelope capability section (rate limit, consistency class, paylo
|
|||||||
push-vs-poll), access-grant semantics, a translation-fidelity capability, and the
|
push-vs-poll), access-grant semantics, a translation-fidelity capability, and the
|
||||||
three-way attachment-mode taxonomy.
|
three-way attachment-mode taxonomy.
|
||||||
|
|
||||||
|
### joplin mapping
|
||||||
|
|
||||||
|
(‖ UC-60–UC-61 are placed in the **wikiengines** matrix column as the nearest existing
|
||||||
|
source — Joplin is a shipped tool — but their true lineage is the **Joplin deep dive**,
|
||||||
|
`research/260614-joplin-deep-dive/findings.md`.)
|
||||||
|
|
||||||
|
| Joplin mechanism (findings §) | Catalog UC |
|
||||||
|
|-------------------------------|------------|
|
||||||
|
| SQLite-local, Markdown-on-sync; per-item files on WebDAV/Nextcloud/S3 (§2) | UC-60 (new) |
|
||||||
|
| E2EE → ciphertext at rest; content opacity (§3) | UC-61 (new) |
|
||||||
|
| Store-minted page-level 32-char ID, rename-stable `:/id` links (§1) | UC-51 (enriched) |
|
||||||
|
| Native store is a DB; attach surface is the sync mirror, not the store (§2) | UC-40 (enriched) |
|
||||||
|
| Internal revisions, not portable (§1) | UC-36 (enriched) |
|
||||||
|
| Plugin host + local Data API (localhost:41184, token) (§4) | UC-38 (enriched; links UC-57) |
|
||||||
|
| Resources (attachments) with IDs (§5) | UC-55 (enriched) |
|
||||||
|
| Poll sync; conflict notes (keep-both) (§2) | UC-31 (enriched; links UC-07) |
|
||||||
|
|
||||||
|
Note: Joplin is the **SQLite-local / Markdown-on-sync** case — its best attach surface is
|
||||||
|
the **interchange/sync representation** (per-item files on a WebDAV/Nextcloud/S3 target),
|
||||||
|
offline and app-independent, landing on INTENT's named backends. It adds two findings for
|
||||||
|
the adapter contract: the **interchange/sync-representation attach surface** (distinct
|
||||||
|
from native on-disk store UC-40 and app-files UC-53), and **content opacity** (a proposed
|
||||||
|
twelfth capability spectrum for encrypted-at-rest shards, extending
|
||||||
|
`research/260614-shard-spectrum-synthesis/findings.md` §2). **Boundary recorded:** Joplin
|
||||||
|
is a **file-sync daemon over one store**; shard-wiki attaches its **mirror as pages** and
|
||||||
|
never re-syncs, and treats the DB-local store and encrypted content as opaque
|
||||||
|
(`research/260614-joplin-deep-dive/findings.md` §6). Architecture logged for
|
||||||
|
`SHARD-WP-0002` (T11/T14): interchange-format attach, content-opacity field, local-REST
|
||||||
|
sub-mode, format-aware file-store profiles.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
@@ -1180,4 +1264,8 @@ three-way attachment-mode taxonomy.
|
|||||||
wiki scale, or do we default such shards to read/projection/overlay/backup? (Notion
|
wiki scale, or do we default such shards to read/projection/overlay/backup? (Notion
|
||||||
dive §10 Q1; ties UC-57.)
|
dive §10 Q1; ties UC-57.)
|
||||||
13. How are **inter-database relations** (UC-58) represented in the union — typed links
|
13. How are **inter-database relations** (UC-58) represented in the union — typed links
|
||||||
in the link graph, a separate relation index (cf. ZigZag many-to-many), or both?
|
in the link graph, a separate relation index (cf. ZigZag many-to-many), or both?
|
||||||
|
14. For an **encrypted shard** (UC-61), what is visible without keys (IDs/structure vs
|
||||||
|
nothing), and does shard-wiki ever hold keys or only treat it as an opaque backup?
|
||||||
|
15. Is writing to a tool's **sync mirror** (UC-60) ever safe, or are interchange-format
|
||||||
|
shards read/projection/overlay-only by policy? (Joplin dive §9.)
|
||||||
Reference in New Issue
Block a user