research: Jupyter Notebooks deep dive; UC-84 (SHARD-WP-0004 T3)

.ipynb JSON cells + embedded computed outputs with fragile execution provenance; derived output stored inside the source. Non-Markdown/lossy; kernel = capability, default = present snapshot + static render. Enriches UC-54/55/59/35; links UC-32/83/79. Marks T3 done. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:08:13 +02:00
parent 3eb026139f
commit 25a714efa7
6 changed files with 286 additions and 5 deletions
--- a/SCOPE.md
+++ b/SCOPE.md
@@ -21,8 +21,8 @@ Learnings update both SCOPE and INTENT where necessary.
 | Intent | `INTENT.md` established; authorization-in-core amendments drafted |
 | Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) |
 | Demand | NetKingdom integration asks captured, not yet negotiated |
-| Spec | Architecture blueprint drafted; UseCaseCatalog 83 UCs from research; PRD/TSD scaffolds |
-| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` active (8 computational-knowledge dives: T1 literate programming done) |
+| Spec | Architecture blueprint drafted; UseCaseCatalog 84 UCs from research; PRD/TSD scaffolds |
+| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` active (8 computational-knowledge dives: T1 literate programming, T3 Jupyter done) |

 ## In Scope (today)

--- a/research/260614-jupyter-deep-dive/README.md
+++ b/research/260614-jupyter-deep-dive/README.md
@@ -0,0 +1,35 @@
+# 260614 — Jupyter Notebooks deep dive
+
+Date: 2026-06-14 · Source: **SHARD-WP-0004 T3**
+
+## What this is
+
+A deep dive into **Jupyter Notebooks**: the **`.ipynb` JSON** document (ordered cells:
+markdown / code+outputs / raw), **kernels**, **embedded computed outputs** (MIME bundles),
+and **execution-count provenance**. The dominant modern computational document and the
+concrete case where the **derived output is captured and stored *inside* the source** with
+real-but-fragile provenance.
+
+## Why it matters
+
+- Tests the T1 **replication- vs derivation-projection** split on the dominant real artifact
+  and adds the wrinkle that **outputs are stored back inside the source** — the source/
+  projection line runs *through* the document.
+- The page model (T12) must carry a **notebook shape**: ordered cells with code cells owning
+  embedded computed outputs that have **weak execution provenance** (run order, environment
+  not captured).
+- Non-Markdown + lossy translation (T15): JSON+MIME bundles; nbconvert→Markdown is lossy and
+  directional. JSON diffs are noisy → Jupytext text-pairing / nbdime (T13).
+
+## Yield
+
+- **UC-84** (new): attach/project a computational notebook preserving cell structure +
+  embedded outputs, surfacing outputs as **snapshots with weak execution provenance**;
+  re-execution **capability-gated**, default = present snapshot + static render.
+- Enrich **UC-54, UC-55, UC-59, UC-35**; links **UC-32, UC-83, UC-79**.
+
+## Contents
+
+| Path | Role |
+|------|------|
+| `findings.md` | `.ipynb` model, kernels/execution-count fragility, ecosystem (nbconvert/Jupytext/papermill/nbdime/nbstripout), capability profile, INTENT mapping, UC seed, architecture notes, open questions |
--- a/research/260614-jupyter-deep-dive/findings.md
+++ b/research/260614-jupyter-deep-dive/findings.md
@@ -0,0 +1,185 @@
+# Jupyter Notebooks — deep dive (findings)
+
+**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T3 · **Subject:** Jupyter Notebooks — the
+`.ipynb` JSON document, kernels, embedded computed outputs, execution provenance.
+
+## Why this dive
+
+T1 (literate programming) established **one source → derived projections** and split
+**replication-projection** from **derivation-projection**. Jupyter is the *dominant modern
+computational document* and the concrete case where the **derived output is captured and
+stored inside the source** — a non-Markdown, partially-executable content type whose
+provenance is real but **fragile**. It is the most plausible concrete "computational shard"
+content type, so it tests the page model (T12), lossy translation (T15), and the
+output-provenance question head-on.
+
+## 1. The `.ipynb` document model
+
+A notebook is a single **JSON document** (`nbformat`), not Markdown:
+
+- **`cells[]`** — an ordered list. Each cell has a `cell_type`:
+  - `markdown` — prose (Markdown + LaTeX), the human-readable part.
+  - `code` — source text (`source`), plus an **`execution_count`** and an **`outputs[]`**
+    array captured from the last run.
+  - `raw` — passthrough.
+- **`outputs[]`** (per code cell) carry results inline: `stream` (stdout/stderr),
+  `execute_result` / `display_data` (a **MIME bundle** — `text/plain`, `text/html`,
+  `image/png` base64, `application/json`, vendor MIME types), and `error` (traceback).
+- **`metadata`** at notebook and cell level (`kernelspec`, `language_info`, tags like
+  `hide-input`, `scrolled`, slideshow roles).
+
+So an `.ipynb` is **source + last-run computed outputs + environment metadata, fused in one
+JSON file**. The Markdown cells are an *island* inside a JSON envelope — relevant to how
+shard-wiki extracts/round-trips content.
+
+## 2. Kernels and execution
+
+- A **kernel** is a separate language process (IPython, IRkernel, IJulia, …) speaking the
+  Jupyter messaging protocol (ZeroMQ). The document is **decoupled from the kernel**: the
+  `.ipynb` persists *captured* outputs; re-running requires a live kernel + the right
+  environment.
+- **`execution_count`** numbers the order cells were *run*, which **need not match document
+  order** — the infamous **hidden-state / out-of-order execution** problem: stored outputs
+  may reflect a run sequence that no longer corresponds to top-to-bottom reading.
+- Reproducibility therefore depends on **out-of-band state**: package versions, data files,
+  environment, random seeds — none captured by `nbformat` itself.
+
+**Consequence for shard-wiki:** the captured outputs are a **snapshot projection with weak
+provenance** — honest treatment must mark them as "computed at run N, environment not
+guaranteed," never as live or authoritative truth.
+
+## 3. The ecosystem (relevant to attach/project/translate)
+
+- **nbconvert** — derives other forms from a notebook: HTML, Markdown, LaTeX/PDF, slides,
+  script. This is **derivation-projection** (T1): notebook source → rendered view, lossy in
+  both directions (HTML keeps outputs; `--to script` keeps only code, like `tangle`).
+- **Jupytext** — represents a notebook **as** a `.py`/`.md` text file (pairing), making it
+  **git-diffable plain text** and round-trippable — directly relevant to storing notebooks
+  in a git shard without JSON-diff noise.
+- **papermill** — parameterize + execute a notebook to produce a new output notebook
+  (notebook as a runnable template — a *derivation with inputs*).
+- **JupyterLab / Notebook / nbviewer / Colab** — front-ends; nbviewer renders a static
+  read-only projection from a URL (a natural projection target).
+- **`nbstripout`** — strips outputs before commit: teams treat **outputs as derived noise**,
+  keeping only source under version control — an explicit "source canonical, outputs
+  derived" stance mirroring T1.
+
+## 4. Capability profile (as a shard / content type)
+
+| Dimension (synthesis spectrum) | Jupyter notebook |
+|--------------------------------|------------------|
+| Attachment mode | file-store (`.ipynb` JSON in a repo) or via Jupyter Server REST API |
+| Addressing granularity | document; **cell** as sub-address (by index / id; `nbformat 4.5+` adds stable cell `id`) |
+| Content identity | file path; cell `id` (4.5+) else positional |
+| Structure | **ordered cell list** (markdown / code+outputs / raw); MIME-bundle outputs |
+| History | VCS on the file; **JSON diffs are noisy** unless paired (Jupytext) or stripped |
+| Merge model | git on JSON (poor) → **paired text** (good) or nbdime (cell-aware diff/merge) |
+| Native query | none |
+| Translation | nbconvert → HTML/MD/script/PDF (lossy, directional); Jupytext text pairing |
+| Write granularity | file / **cell** |
+| Operational envelope | a kernel + environment to (re)execute; static render needs none |
+| Content opacity | **mixed**: source transparent; outputs = MIME blobs (some opaque, e.g. base64 PNG) |
+| Provenance | `execution_count` (weak, out-of-order); environment **not** captured |
+| **Computed-output** | **stored inline**, snapshot, reproducibility out-of-band |
+
+## 5. INTENT mapping
+
+### Reinforcements
+
+- **Replication- vs derivation-projection (T1) confirmed and extended.** nbconvert (→HTML/
+  script) and nbviewer are derivation-projections; `--to script` is literally `tangle`.
+  Jupyter adds a third wrinkle: **the derived output is also stored back inside the source**
+  (captured outputs), so the "source vs projection" line runs *through* the document.
+- **Union without erasure / provenance honesty.** Captured outputs must be surfaced **as
+  snapshots with weak provenance** (run N, environment unguaranteed) — a concrete instance
+  of "never hide freshness/authorship." The out-of-order `execution_count` is exactly the
+  kind of fragility shard-wiki must *show*, not paper over.
+- **Non-Markdown content + lossy translation (UC-55/UC-59).** `.ipynb` is JSON with embedded
+  MIME-bundle outputs; any Markdown projection is **lossy** (loses live outputs, kernel,
+  rich MIME). Surface the lossiness; keep the JSON as canonical payload (T12/T15).
+- **Markdown island.** Markdown cells fit the text-first model, but only as *fragments
+  inside* a JSON envelope — the adapter extracts/round-trips them, it does not pretend the
+  notebook is a Markdown page.
+
+### Divergences / boundaries
+
+- **shard-wiki is not a kernel host.** Re-execution (driving a kernel) is out of scope/
+  capability-gated; default treatment is **attach + present captured outputs as a snapshot
+  projection** + offer nbconvert-style static render. Executing/parameterizing (papermill)
+  is an optional capability, never assumed.
+- **Outputs-in-source is an anti-pattern to respect, not adopt.** Teams strip/pair outputs
+  precisely because mixing derived data into the source breaks diffs. shard-wiki should
+  prefer the **source-canonical, outputs-as-derived** reading (Jupytext pairing / nbstripout
+  ethos) and treat stored outputs as a capturable projection.
+
+### What to keep
+
+1. **Computational-notebook as a first-class content type** with cell structure + inline
+   **computed outputs carrying (weak) execution provenance** — UC-84.
+2. **Outputs = derivation-projection snapshot** (T1 vocabulary): regenerable only with a
+   kernel+environment; degrade gracefully to the stored snapshot / static render.
+3. **Cell-level addressing** (stable cell `id`, nbformat 4.5+) as the sub-page granularity
+   for transclusion/anchoring (UC-32/UC-35).
+4. **Text-pairing (Jupytext)** as the git-friendly storage strategy — feeds the
+   history-portability thread (poor JSON diffs → paired text / nbdime).
+
+## 6. UC seed
+
+| # | Seed | Disposition |
+|---|------|-------------|
+| UC-84 | Attach/project a **computational notebook** (`.ipynb`): preserve **cell structure** (markdown / code / output) and **embedded computed outputs**, surfacing each output **as a snapshot with its (weak) execution provenance** (run count, environment not guaranteed) — re-execution is **capability-gated**, default is present-the-snapshot + offer a static rendered projection | **new** |
+| — | Notebook JSON / MIME-bundle outputs = non-Markdown content; Markdown projection is lossy | enrich **UC-55**, **UC-59** |
+| — | Computed/evaluated cell = computation-defined content | enrich **UC-54** |
+| — | Cell `id` (nbformat 4.5+) = sub-page address for anchor/transclusion | enrich **UC-35**, links **UC-32** |
+| — | Stored outputs as derived snapshot (nbstripout/Jupytext ethos) = source-canonical/outputs-derived | links **UC-83**, **UC-79** |
+
+## 7. Architecture notes for SHARD-WP-0002
+
+- **T12 (page model):** add **computational-notebook** as a page shape — an **ordered cell
+  list** where code cells own **embedded computed outputs** (MIME bundles) with weak
+  execution provenance. Distinct from prose, typed records, query-defined, inline-embedded
+  objects (Quip/Notion), typed-graph (Wikibase), and the literate one-source-many-projection
+  shape (UC-83). The defining new attribute: **derived output stored *inside* the source**.
+- **T15 (translation / fidelity):** `.ipynb` is non-Markdown; nbconvert→Markdown is **lossy
+  and directional** (drops live outputs/kernel/rich MIME). Keep JSON canonical; any Markdown
+  is a projection. MIME-bundle outputs map to the content-opacity spectrum (text→html→base64
+  image = transparent→opaque).
+- **T13 (history):** JSON diffs are **noisy**; record **text-pairing (Jupytext)** and
+  **cell-aware diff/merge (nbdime)** as history-portability strategies for embedded-output
+  documents. Reinforces "source-canonical, outputs-derived."
+- **T16 (projection):** captured outputs are a **derivation-projection snapshot**;
+  re-execution (kernel) and parameterized execution (papermill) are **capabilities**, not
+  assumptions; degrade to the stored snapshot / nbviewer-style static render.
+
+## 8. Open questions
+
+1. Does shard-wiki ever **re-execute** a notebook (host/broker a kernel), or strictly
+   attach + present captured outputs + static render? (Same scope boundary as UC-83/UC-56
+   "do we ever drive the derivation.")
+2. Is **UC-84** distinct from **UC-83**, or is a notebook just the "outputs-stored-in-source"
+   special case of the literate one-source-many-projection pattern? (Kept separate: UC-84's
+   defining trait is *captured derived output embedded in the canonical source with weak
+   provenance* — a page-model attribute UC-83 doesn't carry.)
+3. How are **MIME-bundle outputs** represented in the page model — opaque provenance-tagged
+   blobs, a typed-asset registry (UC-55 open question #10), or selected-MIME projection?
+4. Default storage: attach `.ipynb` **as-is** (JSON, noisy diffs) or prefer a **paired text
+   representation** when the shard is a git repo? (Policy → configurable.)
+
+## 9. Sources
+
+- Jupyter `nbformat` reference (cells, outputs, MIME bundles, cell `id` 4.5+);
+  Jupyter messaging protocol / kernels docs.
+- **nbconvert**, **nbviewer**, **JupyterLab**, **Colab** docs.
+- **Jupytext**, **papermill**, **nbdime**, **nbstripout** project docs.
+- prior: `research/260614-literate-programming-deep-dive/` (replication- vs
+  derivation-projection, UC-83); `research/260614-notion-deep-dive/` (block-JSON,
+  external-API), `research/260614-quip-deep-dive/` (inline embedded objects, UC-55/58/59).
+
+## 10. Traceability
+
+New UC **UC-84** carries the marker **⊜** in the wikiengines column of
+`spec/UseCaseCatalog.md` (true lineage = this dive). Enriched: UC-54, UC-55, UC-59, UC-35;
+links UC-32, UC-83, UC-79. Architecture cross-refs: SHARD-WP-0002 T12 (notebook page shape:
+outputs embedded in source), T15 (lossy non-Markdown translation; MIME opacity), T13
+(paired-text / nbdime history), T16 (output = derivation-projection snapshot; kernel =
+capability).
--- a/research/README.md
+++ b/research/README.md
@@ -37,4 +37,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and
 | 2026-06-14 | `260614-mojomojo-deep-dive/` | MojoMojo — Perl Catalyst/DBIx::Class DB-backed wiki; pages + history in SQL tables, no file store/API → direct-DB-read binding; DB version rows as history source; UC-81 |
 | 2026-06-14 | `260614-oddmuse-deep-dive/` | Oddmuse — single Perl CGI, plain-text `page/` files + `keep/` revisions, no DB; the minimal file-store floor / graceful-degradation baseline; partial-history honesty; UC-82 |
 | 2026-06-14 | `260614-usemodwiki-deep-dive/` | UseModWiki — flat-file ancestor (Wikipedia's MediaWiki Phase I); CamelCase linking; lineage grounding for the minimal file-store floor; enrichment-only (reinforces UC-82, UC-25) |
-| 2026-06-14 | `260614-literate-programming-deep-dive/` | Literate programming (Knuth's WEB / weave / tangle) — one source → N co-equal derived projections (docs + code); named-chunk transclusion; splits replication- vs derivation-projection; SHARD-WP-0004 T1; UC-83 |
+| 2026-06-14 | `260614-literate-programming-deep-dive/` | Literate programming (Knuth's WEB / weave / tangle) — one source → N co-equal derived projections (docs + code); named-chunk transclusion; splits replication- vs derivation-projection; SHARD-WP-0004 T1; UC-83 |
+| 2026-06-14 | `260614-jupyter-deep-dive/` | Jupyter Notebooks — `.ipynb` JSON cells + embedded computed outputs with fragile execution provenance; derived output stored *inside* the source; non-Markdown/lossy; kernel = capability; SHARD-WP-0004 T3; UC-84 |
--- a/spec/UseCaseCatalog.md
+++ b/spec/UseCaseCatalog.md
@@ -1082,6 +1082,34 @@ build system (driving derivation is capability-gated). Feeds SHARD-WP-0002 T12
 named-chunk transclusion).  
 **Priority:** Later

+---
+
+### UC-84 — Attach or project a computational notebook with computed-output provenance
+
+**Actor:** Orchestrator / adapter  
+**Goal:** Attach/project a **computational notebook** (`.ipynb`) preserving its **cell
+structure** (markdown / code / output) and its **embedded computed outputs**, surfacing each
+output **as a snapshot carrying its (weak) execution provenance** (run count, environment
+not guaranteed); re-execution is **capability-gated** — the default is to present the stored
+snapshot plus an offered static rendered projection.  
+**Source:** wikiengines, intent  
+**Notes:** A Jupyter notebook is a **JSON document** of ordered cells where code cells own
+**MIME-bundle outputs captured from the last run**
+(`research/260614-jupyter-deep-dive/findings.md`). Its defining trait is that the **derived
+output is stored *inside* the canonical source** — the source/projection line runs *through*
+the document — and its provenance is **fragile** (`execution_count` may be out-of-order;
+environment/data/versions are not captured by `nbformat`). Treatment: keep the JSON as
+canonical payload (non-Markdown; any Markdown projection is lossy and directional, UC-55/
+UC-59); present captured outputs as **derivation-projection snapshots** (UC-83) marked "run
+N, environment unguaranteed" (never as live/authoritative, union-without-erasure); offer
+nbconvert/nbviewer static render; **re-execute or parameterize (papermill) only as a
+capability**, never assumed (shard-wiki is not a kernel host). Cell `id` (nbformat 4.5+) is
+the sub-page address for anchoring/transclusion (UC-32/UC-35); JSON diffs are noisy →
+text-pairing (Jupytext) / cell-aware merge (nbdime) for history (T13). Feeds SHARD-WP-0002
+T12 (notebook page shape: outputs embedded in source), T15 (lossy non-Markdown translation;
+MIME opacity), T13 (paired-text history), T16 (output = derivation-projection snapshot).  
+**Priority:** Later
+
 ## D. Addressing, identity & query

 *Span / identity addressing, transclusion-as-reference, dimensional navigation, and query over the union.*
@@ -1583,7 +1611,7 @@ marker's true lineage is its per-source mapping subsection below):
 | ⚓ | wiki.js | ⊞ | federated-wiki | ⬡ | wikibase |
 | ⎇ | forge-wikis (Gitea/GitLab/GitHub) | ⊡ | tiddlywiki | ⊟ | ikiwiki |
 | ◧ | quip | ⊙ | mojomojo | ⊚ | oddmuse (UseModWiki reinforces, no marker) |
-| ⊛ | literate-programming (WEB/weave/tangle) | | | | |
+| ⊛ | literate-programming (WEB/weave/tangle) | ⊜ | jupyter (notebooks) | | |

 | UC | c2 research | yawex research | federation research | wikiengines research | INTENT |
 |----|-------------|----------------|---------------------|----------------------|--------|
@@ -1652,6 +1680,7 @@ marker's true lineage is its per-source mapping subsection below):
 | UC-81 | | | | ⊙ | ✓ |
 | UC-82 | ✓ | | | ⊚ | ✓ |
 | UC-83 | | | ✓⊛ | | ✓ |
+| UC-84 | | | | ⊜ | ✓ |
 | UC-08 | ✓ | | |
 | UC-09 | ✓ | | |
 | UC-10 | ✓ | | |
@@ -2341,6 +2370,37 @@ snapshot** — never present a derivation without its source link (union-without
 Architecture logged for `SHARD-WP-0002` (T12/T16): one-source-many-projections page shape,
 replication- vs derivation-projection, named-chunk transclusion.

+### jupyter mapping
+
+(⊜ UC-84 lineage = the **Jupyter deep dive**, `research/260614-jupyter-deep-dive/findings.md`;
+a concrete computational content type, not a wiki engine.)
+
+| Jupyter mechanism (findings §) | Catalog UC |
+|--------------------------------|------------|
+| `.ipynb` JSON: ordered cells, code cells own embedded MIME-bundle outputs (§1) | UC-84 (new) |
+| Captured outputs stored *inside* the source; weak `execution_count` provenance (§2) | UC-84; links UC-83 (derivation-projection snapshot) |
+| JSON + MIME bundles = non-Markdown; nbconvert→MD lossy & directional (§3) | UC-55 / UC-59 (enriched) |
+| Code cell = computation-defined content (§1) | UC-54 (enriched) |
+| Cell `id` (nbformat 4.5+) = sub-page address (§4) | UC-35 (enriched); links UC-32 |
+| Noisy JSON diffs → Jupytext pairing / nbdime cell-merge (§3) | links UC-43, T13 |
+| nbstripout/pairing = source-canonical, outputs-derived (§3) | links UC-79, UC-83 |
+
+Note: Jupyter is the **dominant computational document** and the concrete test of T1's
+projection split: nbconvert/nbviewer are **derivation-projections**, but the twist is that
+**captured outputs are stored back inside the canonical source** (the source/projection line
+runs *through* the document) with **fragile provenance** (out-of-order `execution_count`;
+environment uncaptured). The page model (T12) gains a **notebook shape** — ordered cells with
+code cells owning embedded computed outputs — distinct from prose, typed records,
+query-defined, inline-embedded objects (Quip/Notion), typed-graph (Wikibase), and the
+literate one-source-many-projection shape (UC-83); its new attribute is *derived output
+embedded in source*. **Boundary recorded:** shard-wiki is **not a kernel host** — default to
+attach + present captured outputs as snapshots marked "run N, environment unguaranteed"
+(never live, UC-84) + offer static render; re-execution/papermill are **capabilities**. Keep
+JSON canonical (Markdown projection is lossy, UC-55/59); prefer paired-text storage / nbdime
+for history when the shard is git. Architecture logged for `SHARD-WP-0002` (T12/T15/T13/T16):
+notebook page shape, lossy non-Markdown + MIME opacity, paired-text history, output =
+derivation-projection snapshot.
+
 ---

 ## Open questions
--- a/workplans/SHARD-WP-0004-computational-knowledge-systems.md
+++ b/workplans/SHARD-WP-0004-computational-knowledge-systems.md
@@ -92,7 +92,7 @@ provenance, and reproducibility. Enrich UC-54 (query/computation-defined content

 ```task
 id: SHARD-WP-0004-T3
-status: todo
+status: done
 priority: high
 state_hub_task_id: "3d35d58f-a9ea-44f9-89ea-d89205da8d8b"
 ```