research: Notion deep dive (closed block-DB SaaS, external REST API only, database-as-pages); UC-57/58/59

The closed/hosted/schema-rich extreme: everything is a block (UUID id,
type, properties, ordered child content, single parent), pages are blocks
and database rows are pages with a schema, Postgres-backed hosted SaaS.
Databases add typed properties + relations + rollups + formulas across
many views = the apex of wiki-page-as-structured-record. Extension model
has no in-app plugin runtime; the only extensibility is the external REST
API (+ webhooks 2026) inside a tight envelope (~3 rps, eventual
consistency, recursive child fetch, scoped/revocable per-page grants).
Adds the third attachment mode (external-API-only) alongside file-store
(Obsidian/TWiki) and in-engine host (Roam/XWiki); Notion enforces no
silent remote mutation via scoped grants. Added UC-57 (attach closed
external-API-only shard w/ operational envelope + scoped grant), UC-58
(typed database w/ schema+relations+views, no flattening), UC-59
(lossy-aware translation w/ fidelity report); enriched
UC-31/34/36/39/50/51/52/54/56. Boundary: one external-API candidate shard,
best as projection/mirror/overlay/backup, not a substrate and not the
federation layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

research/260614-notion-deep-dive/README.md

@@ -0,0 +1,56 @@
+# 260614 — Notion deep dive (closed block-database SaaS, attached only through its API)
+
+Date: 2026-06-14
+
+## What this is
+
+A focused study of **Notion** — architecture and extension model — read through
+shard-wiki's lens. Notion is the **closed, hosted, schema-rich** end of the spectrum and
+the dive that stress-tests shard-wiki's hardest constraints (*graceful degradation*,
+*no silent remote mutation*, *Markdown-first that must degrade*).
+
+It contrasts sharply with its neighbours:
+- vs **Roam** — both are block databases with per-block UUIDs, but Roam is a *client*
+  DataScript DB reached only via an *in-app* JS API, while Notion is a *server* Postgres
+  store reached via a real *external* REST API (no in-engine hosting needed, but
+  rate-limited, eventually consistent, scoped by per-page grants).
+- vs **Obsidian** — the mirror image of "file over app": Notion is **app over file**, a
+  closed hosted store with no portable files.
+- vs **XWiki** — both are structured wiki-as-app-platforms; Notion is the **apex of
+  database-as-pages** (schema + relations + rollups + views) but closed and REST-only.
+
+Distinctive material:
+- **Data model** — everything is a **block** (UUID id, type, properties, ordered child
+  `content`, single `parent`); pages are blocks, **database rows are pages with a
+  schema**; Postgres-backed hosted SaaS
+- **Databases** — typed properties, **relations** (bidirectional), **rollups**,
+  **formulas**, and many **views** (table/board/calendar/gallery) of one row-set
+- **Extension model** — **no in-app plugin runtime**; extension = **external REST API**
+  integrations (+ webhooks since 2026), with ~3 rps rate limit, eventual consistency,
+  recursive child fetch, payload caps, and **scoped/revocable per-page access grants**
+
+## Contents
+
+| Path | Role |
+|------|------|
+| `findings.md` | Block model, databases, page-model impedance, the API-only extension model, capability profile, scoped consent, INTENT mapping, UC seeds, architecture notes, sources |
+
+## Status
+
+Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
+(UC-57 attach a closed external-API-only hosted shard with rate-limit/eventual-
+consistency/scoped-grant constraints, UC-58 attach a typed database with schema +
+relations + views without flattening, UC-59 lossy-aware translation with a fidelity
+report); UC-31/34/36/39/50/51/52/54/56 enriched. Logged for `SHARD-WP-0002` (T14):
+an **operational-envelope** capability section, **access-grant semantics**, a
+**translation-fidelity** capability, and a three-way **attachment-mode taxonomy**
+(file-store / in-engine-hosted / external-API-only).
+
+**Key takeaways recorded:** Notion *enforces* "no silent remote mutation" via scoped,
+revocable per-integration grants (ties the authz-in-core decision); it is the heaviest
+**lossy-translation** case (proprietary blocks/rich-text, lossy Markdown export); and it
+adds the **external-API-only** attachment mode (full write-through without in-engine
+hosting, but bounded by rate limits and eventual consistency). **Boundary:** one
+external-API candidate shard — best as projection/mirror/overlay/backup — not a
+substrate and not the federation layer.
+</content>

research/260614-notion-deep-dive/findings.md

@@ -0,0 +1,302 @@
+# Findings — Notion: a closed block-database SaaS, attached only through its API
+
+Date: 2026-06-14
+Source kind: **modern shipped product** — a hosted block-database SaaS; a *candidate
+shard* of a distinct family (closed, external-REST-API-only, schema-rich, heaviest
+translation cost); has **no in-app plugin runtime**
+Lens: shard-wiki — the external-API attachment mode, database-as-pages, lossy
+translation/fidelity, scoped consent, and graceful degradation against a sovereign
+closed backend
+
+> Where Notion sits in the set. Roam and Notion are both **block databases with
+> per-block UUIDs**, but they are opposite on access: Roam is a *client-side*
+> DataScript DB reachable only through an *in-app* JS API (write-through needs an
+> adapter hosted inside Roam); **Notion is a *server-side* Postgres store reachable
+> through a real *external* REST API** — attachable from outside with no in-engine
+> hosting, but rate-limited, eventually consistent, and **scoped by explicit
+> per-integration page grants**. And against Obsidian's "file over app," Notion is the
+> mirror image: **app over file** — a closed hosted store with no portable files at
+> all. So Notion is the dive that stress-tests shard-wiki's hardest constraints:
+> *graceful degradation*, *no silent remote mutation*, *union without erasure*, and
+> *Markdown-first that must degrade* against a proprietary, schema-rich, non-Markdown
+> backend.
+
+Pairs with — and contrasts against — the Roam dive (block-DB, in-app vs external API),
+the Obsidian dive (closed-hosted vs file-over-app), and the XWiki dive (both are
+structured wiki-as-app-platforms; Notion is the apex of database-as-pages but closed
+and REST-only).
+
+---
+
+## 1. Core data model — everything is a block
+
+Notion's data model is uniform: **"Everything you see in Notion is a block. Text,
+images, lists, a row in a database, even pages themselves — these are all blocks."**
+
+Each block record holds:
+
+| Field | Meaning |
+|-------|---------|
+| **id** | a randomly generated **UUID v4**, visible in the page URL — the block's stable address |
+| **type** | determines rendering + which properties apply (text, to-do, heading, **page**, **child_database**, …) |
+| **properties** | type-specific attributes (e.g. `title` text; for database rows, the typed property values) |
+| **content** | an **ordered array of child block IDs** — nesting |
+| **parent** | a single upward pointer — used for **permission inheritance** |
+
+Blocks form a **render tree** (content + parent pointers); indentation is *structural*,
+not cosmetic — it changes block relationships. **Pages are blocks**; **databases are
+blocks whose children are pages**; a database **row is a page** with typed properties.
+Stored in **Postgres** on Notion's servers (hosted SaaS — not local, not files).
+
+The shard-wiki-relevant facts: (a) **per-block UUIDs** give native sub-page addressing
+(UC-51); (b) the **page = block, database-row = page-with-schema** identity is a real
+page-model impedance (§3); (c) the store is **closed and server-side** — reachable only
+through the API (§4).
+
+---
+
+## 2. Databases — schema, typed properties, relations, views
+
+A Notion **database** is a collection of pages sharing a **schema** of typed
+**properties**: `title`, `rich_text`, `select` / `multi_select`, `number`, `date`,
+`checkbox`, `person`, `files`, **`relation`** (a typed link to rows in another
+database — shown on *both* sides, i.e. bidirectional), **`rollup`** (an aggregate over a
+relation), and **`formula`** (computed). The same row-set is shown through multiple
+**views** — **table, board (kanban), calendar, gallery, list, timeline** — each a
+filtered/sorted/grouped projection of the same data.
+
+For shard-wiki this is the **apex of "wiki page as structured record"** (stronger than
+XWiki XObjects, UC-39): a page can be *bodiless typed data*, pages are joined by
+**typed inter-database relations** (a graph of typed links), and "the page" is routinely
+**a row in a schema** rather than prose. Notion databases also ship the ZigZag insight
+commercially: **one row-set, many views/dimensions** (UC-47/48) and **filtered/linked
+databases = query-defined pages** (UC-54).
+
+---
+
+## 3. Page-model impedance
+
+Mapping Notion to a Markdown-first page model is the **heaviest translation case** in
+the research set:
+
+- **Block/rich-text ≠ Markdown.** Notion's rich text is an annotated-span model; many
+  block types (synced blocks, columns, callouts, embeds, database views) have no clean
+  Markdown equivalent. Notion's own **export to Markdown/CSV is lossy** (databases →
+  CSV, relations/rollups/formulas flatten or drop).
+- **Database-row-as-page + schema** must map onto pages + sidecar metadata without
+  discarding the schema or the relations (extends UC-34/UC-39).
+- Therefore translation must be **lossy-aware with a fidelity report** — surface what
+  did *not* round-trip rather than silently flattening (UC-59). This is *different from*
+  UC-42 (Foswiki TML↔HTML *lossless* round-trip); Notion is fundamentally lossy.
+
+---
+
+## 4. Extension model — there is no plugin runtime; only the API
+
+This is the defining architectural fact for shard-wiki: **Notion has no third-party
+in-app plugin/extension system.** There is no marketplace of code that runs inside
+Notion (the only "in-app modding" is the *unofficial* Notion Enhancer desktop patcher,
+out of scope). **Extension = external integration via the public REST API** (plus
+embeds and, recently, webhooks).
+
+The **public REST API** (`api.notion.com/v1`):
+
+- **Resources:** `pages`, `blocks` (retrieve / append / update / delete children),
+  `databases` (retrieve, **query** with filters+sorts, create), `users`, `search`,
+  `comments`.
+- **Authorization:** an **internal integration token** (workspace-owned) or **OAuth
+  2.0** (public integrations). Critically, **an integration only sees pages a user has
+  explicitly connected to it** — the user "approves the app and connects specific
+  pages" via *Add connections*. Access is a **scoped, revocable, per-page grant** (§6).
+- **Operational constraints the adapter must encode:**
+  - **Rate limit** ~**3 requests/second** average (429 `rate_limited`); **no paid
+    increase**.
+  - **Eventual consistency / no live read model** — the API is not real-time; reads can
+    lag.
+  - **Recursive retrieval** — a block returns only **first-level children**; full-page
+    reconstruction requires walking `has_children` recursively.
+  - **Payload caps** — 1000 blocks / 500 KB per request; child arrays ≤ 100.
+  - **Webhooks** (added 2026) deliver page/database change events — a push transport for
+    UC-31, replacing pure polling.
+- **Version history:** Notion keeps **internal page history** (retention bounded by
+  plan); it is **not portable** and not exposed as git — a UC-36 *supplementation* case
+  (like Confluence/MediaWiki), not an import case.
+- **Publish:** pages can be **published to the web** as read-only public pages — an
+  outbound publish surface (UC-56).
+
+Consequence: Notion is attachable **only** as an **external-API shard** — but, unlike
+Roam, **no in-engine adapter hosting is needed** (the REST API is external). The cost is
+the operational envelope (rate-limit, eventual consistency, scoped grant) that the
+capability profile must model (UC-57).
+
+---
+
+## 5. Notion as a shard — capability profile
+
+| Capability | Notion | Notes for the adapter contract |
+|------------|--------|--------------------------------|
+| Read | **yes (external REST)** | rate-limited (~3 rps), eventually consistent, recursive child fetch |
+| Write | **yes (external REST)** | append/update/delete blocks, create pages — no in-engine host needed (vs Roam) |
+| Write granularity | **block-level (fine)** | like Roam; per-block ops |
+| Identity / addressing | **block UUID v4** | native sub-page addressing (UC-51), store-minted (like Roam, not in-file) |
+| Structured data | **yes (apex)** | databases: schema + typed properties + relations + rollups + formulas (UC-34/39/58) |
+| Native query | **yes** | database query API (filters/sorts) → delegate views (UC-52) |
+| Views / dimensions | **yes** | table/board/calendar/gallery = many views of one row-set (UC-47/48/54) |
+| Subscribe | **webhooks (2026)** | push events; else poll (UC-31) |
+| Version history | **internal, not portable** | supplement via coordination journal (UC-36) |
+| Diff / merge | **no native** | — |
+| Lock | **no** | — |
+| Publish | **publish-to-web** | outbound read-only (UC-56) |
+| Access model | **scoped per-page grant (OAuth/token)** | explicit consent; revocable (UC-57, authz) |
+| Syntax / content | **proprietary block + rich text** | **lossy** to Markdown; needs fidelity-aware translation (UC-59) |
+
+Verdict: Notion is a legitimate but **demanding** shard — **external-API-attached,
+schema-rich, fine-grained, closed**. It behaves best as a **projected / mirrored /
+overlay / backup** participant; full write-through is possible but bounded by rate limits
+and eventual consistency. The strongest reasons to attach it: structured databases
+(UC-58) and block-UUID addressing (UC-51); the strongest cautions: lossy translation
+(UC-59) and no portable history (UC-36).
+
+---
+
+## 6. Scoped consent and "no silent remote mutation"
+
+Notion *enforces* one of shard-wiki's INTENT constraints at the platform level: an
+integration can touch **only** the pages a user has explicitly **connected** to it, and
+the grant is **revocable**. This is a clean, real-world model of **no silent remote
+mutation** and of a shard granting the orchestrator **scoped, consented access** — and
+it ties directly to the settled **authz-in-core / authn-delegated** decision
+([[shard-wiki-auth-in-core-decision]]): authentication to Notion is delegated (OAuth /
+integration token), while shard-wiki's *own* authorization decides what to do with the
+granted scope. The adapter contract should treat **"scoped, revocable grant"** as a
+first-class attachment property (UC-57), not an afterthought.
+
+---
+
+## 7. Mapping to shard-wiki INTENT (compare, do not equate)
+
+### 7.1 Reinforcements
+
+- **Graceful degradation** has its sharpest test here: a closed SaaS with no files, rate
+  limits, eventual consistency, and lossy export must still be usable as
+  read/projection/overlay/backup. If the adapter contract handles Notion, it handles
+  most things.
+- **No silent remote mutation** is *modeled by the platform* (scoped grants) — Notion
+  validates the principle (§6).
+- **Database-as-pages** validates that "wiki page" must stretch to **typed records with
+  relations**, not just prose (UC-34/39/58).
+- **Block UUIDs** reconfirm (with Roam) that native sub-page addressing is real and
+  adoptable (UC-51).
+
+### 7.2 Deliberate divergences (design bugs if conflated)
+
+1. **Closed hosted store; no sovereignty over bytes.** shard-wiki cannot make Notion
+   git-native or local. It can mirror/project/overlay/back-up and **supply** a
+   git-addressable history (UC-36) — never claim to own Notion's store.
+2. **Lossy, proprietary content.** Do **not** pretend Notion round-trips to Markdown.
+   Translate lossily *with a fidelity report* and preserve non-mappable elements as
+   provenance/sidecar (UC-59) — union without erasure includes erasure *of fidelity*
+   being made visible.
+3. **External-API-only, rate-limited, eventually consistent.** Projection must be
+   **cache/poll/webhook**, not a live read model; sync is bounded — encode this in the
+   capability profile (UC-57). Do not design flows that assume cheap, instant, unlimited
+   reads.
+4. **One workspace (or its granted page set) = one shard**, never the federation layer.
+
+### 7.3 What Notion teaches that shard-wiki should keep
+
+- Model **operational envelope** (rate limit, consistency class, payload caps,
+  pagination) as explicit capability-profile fields — Notion makes them unignorable.
+- Model **scoped, revocable consent** as a first-class attachment property (UC-57, §6).
+- Treat **translation fidelity as data**: a per-shard, per-page report of what projects
+  cleanly vs. degrades (UC-59) — applies beyond Notion.
+- Recognize **external-REST attach** as a distinct, *preferred-where-available* mode:
+  full write-through without in-engine hosting (contrast Roam) — but pay the operational
+  envelope.
+
+---
+
+## 8. Use-case seeds → catalog (promoted 2026-06-14)
+
+Last existing UC is **UC-56**. New UCs **UC-57–UC-59** added; existing UCs enriched.
+
+| Seed | Catalog action |
+|------|----------------|
+| **Attach a closed hosted shard via its external REST API only** — no file store, no in-app runtime — honoring rate limits, eventual consistency, payload caps, and a scoped/revocable access grant | **UC-57 (new)** |
+| **Attach a typed database (schema + relations + rollups + multiple views) as a shard** without flattening the schema or the inter-record relations | **UC-58 (new)** |
+| **Translate a proprietary block/rich-text model to/from Markdown with an explicit fidelity report**, preserving non-mappable elements rather than silently dropping them | **UC-59 (new)** |
+| Block UUIDs = store-minted native span addresses (external-API variant) | **enriches UC-51** |
+| External-API block-DB attach (no in-engine host) — contrast Roam's in-app-only | **enriches UC-50** |
+| Database query API + filtered/linked DBs | **enriches UC-52, UC-54** |
+| Database-as-pages apex; typed records + relations | **enriches UC-34, UC-39** |
+| Webhooks (2026) as a push transport | **enriches UC-31** |
+| Internal-only page history, not portable | **enriches UC-36** |
+| Publish-to-web outbound | **enriches UC-56** |
+| Scoped, revocable per-integration grant; no silent mutation | links **UC-57** + [[shard-wiki-auth-in-core-decision]] |
+
+---
+
+## 9. Architecture notes for SHARD-WP-0002 (no UC)
+
+- Add an **operational-envelope** section to the adapter capability profile: rate limit,
+  consistency class (live / eventually-consistent / snapshot), payload/pagination caps,
+  recursive-fetch requirement, push-vs-poll transport. Notion is the forcing example.
+- Add **access-grant semantics**: scope (which pages), revocability, auth mode
+  (delegated token/OAuth) — ties the authz-in-core decision and "no silent mutation".
+- Add a **translation-fidelity capability**: adapters declare and report what content
+  round-trips vs. degrades (UC-59); generalizes UC-42 (lossless) to the lossy case.
+- **Attachment-mode taxonomy** now spans: file-store direct (Obsidian/TWiki, UC-40),
+  in-engine hosted adapter (Roam/XWiki, UC-38/50), and **external-API-only**
+  (Notion, UC-57). T14 binding should enumerate all three.
+- **Database/schema/relations** as a unit (UC-58) presses the page-model spec: collection
+  + schema + typed relations, not just a page.
+
+---
+
+## 10. Open questions (for spec / workplans)
+
+1. Is **external-API-only with a tight rate limit** (Notion) viable for write-through at
+   wiki scale, or do we cap Notion at read/projection/overlay/backup by default?
+2. How are **inter-database relations** (UC-58) represented in the union — as typed links
+   in the link graph, as a separate relation index (cf. ZigZag many-to-many), or both?
+3. What is the **fidelity report** format (UC-59), and where does it surface — provenance
+   panel, projection metadata, reconciliation review?
+4. For **scoped grants** (§6), how does shard-wiki represent partial visibility (only
+   *some* of a workspace's pages granted) without misrepresenting the shard as complete?
+5. Do we consume Notion **webhooks** (push) or poll, given eventual consistency and the
+   rate limit (UC-31)?
+
+---
+
+## 11. Sources
+
+| Source | Used for |
+|--------|----------|
+| Notion — "The data model behind Notion's flexibility" (https://www.notion.com/blog/data-model-behind-notion) | Everything-is-a-block; block record (id/type/properties/content/parent); render tree; pages/databases as blocks; Postgres |
+| Notion Docs — Request limits (https://developers.notion.com/reference/request-limits) | ~3 rps rate limit, 429, payload caps, recursive first-level children |
+| Notion Docs — Authorization (https://developers.notion.com/docs/authorization) | Internal token vs OAuth; integration connected to specific pages; scoped grant |
+| Hookdeck / ClickUp — Notion webhooks guides (https://hookdeck.com/webhooks/platforms/guide-to-notion-webhooks-features-and-best-practices) | Webhook support (2026), page/database change events |
+| Truto / Rollout — Notion API architecture & essentials (https://truto.one/blog/how-to-integrate-with-the-notion-api-architecture-guide-for-b2b-saas/) | REST endpoints (pages/blocks/databases/search), integration patterns, no in-app plugin model |
+| General API knowledge — database property types, views, relations/rollups/formulas, export-to-Markdown lossiness | §2, §3 |
+
+Cross-references: `research/260614-roam-deep-dive/findings.md` (block-DB/UUID, in-app vs
+external API), `research/260614-obsidian-deep-dive/findings.md` (closed-hosted vs
+file-over-app), `research/260613-xwiki-deep-dive/findings.md` (structured wiki-app-
+platform), `spec/UseCaseCatalog.md` (UC-31, UC-34, UC-36, UC-39, UC-50/51/52, UC-54,
+UC-56), `workplans/SHARD-WP-0002-federation-architecture.md` (T14), and the authz
+decision [[shard-wiki-auth-in-core-decision]].
+
+---
+
+## 12. Traceability
+
+- New UCs: **UC-57, UC-58, UC-59** → `spec/UseCaseCatalog.md`.
+- Enriched UCs: **UC-31, UC-34, UC-36, UC-39, UC-50, UC-51, UC-52, UC-54, UC-56**.
+- Architecture (no UC): operational-envelope + access-grant + translation-fidelity
+  capability fields; three-way attachment-mode taxonomy; database/schema/relations in the
+  page model → `SHARD-WP-0002` (T14).
+- Decision link: scoped/revocable grant + no-silent-mutation → [[shard-wiki-auth-in-core-decision]].
+- Boundary recorded: Notion is **one external-API candidate shard** — closed, hosted,
+  schema-rich, lossy-to-Markdown — best as projection/mirror/overlay/backup; not a
+  substrate, not the federation layer (INTENT graceful-degradation + no-silent-mutation).
+</content>

research/README.md

@@ -20,4 +20,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and
 | 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 |
 | 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 |
 | 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 |
-| 2026-06-14 | `260614-obsidian-deep-dive/` | Obsidian — file-over-app vaults, plugin API, ecosystem-popularity → UC signal; UC-53/54/55/56 |
+| 2026-06-14 | `260614-obsidian-deep-dive/` | Obsidian — file-over-app vaults, plugin API, ecosystem-popularity → UC signal; UC-53/54/55/56 |
+| 2026-06-14 | `260614-notion-deep-dive/` | Notion — closed block-DB SaaS, external REST API only, database-as-pages; UC-57/58/59 |