shard-wiki/research/260614-jupyter-deep-dive/findings.md

# Jupyter Notebooks — deep dive (findings)

**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T3 · **Subject:** Jupyter Notebooks — the
`.ipynb` JSON document, kernels, embedded computed outputs, execution provenance.

## Why this dive

T1 (literate programming) established **one source → derived projections** and split
**replication-projection** from **derivation-projection**. Jupyter is the *dominant modern
computational document* and the concrete case where the **derived output is captured and
stored inside the source** — a non-Markdown, partially-executable content type whose
provenance is real but **fragile**. It is the most plausible concrete "computational shard"
content type, so it tests the page model (T12), lossy translation (T15), and the
output-provenance question head-on.

## 1. The `.ipynb` document model

A notebook is a single **JSON document** (`nbformat`), not Markdown:

- **`cells[]`** — an ordered list. Each cell has a `cell_type`:
  - `markdown` — prose (Markdown + LaTeX), the human-readable part.
  - `code` — source text (`source`), plus an **`execution_count`** and an **`outputs[]`**
    array captured from the last run.
  - `raw` — passthrough.
- **`outputs[]`** (per code cell) carry results inline: `stream` (stdout/stderr),
  `execute_result` / `display_data` (a **MIME bundle** — `text/plain`, `text/html`,
  `image/png` base64, `application/json`, vendor MIME types), and `error` (traceback).
- **`metadata`** at notebook and cell level (`kernelspec`, `language_info`, tags like
  `hide-input`, `scrolled`, slideshow roles).

So an `.ipynb` is **source + last-run computed outputs + environment metadata, fused in one
JSON file**. The Markdown cells are an *island* inside a JSON envelope — relevant to how
shard-wiki extracts/round-trips content.

## 2. Kernels and execution

- A **kernel** is a separate language process (IPython, IRkernel, IJulia, …) speaking the
  Jupyter messaging protocol (ZeroMQ). The document is **decoupled from the kernel**: the
  `.ipynb` persists *captured* outputs; re-running requires a live kernel + the right
  environment.
- **`execution_count`** numbers the order cells were *run*, which **need not match document
  order** — the infamous **hidden-state / out-of-order execution** problem: stored outputs
  may reflect a run sequence that no longer corresponds to top-to-bottom reading.
- Reproducibility therefore depends on **out-of-band state**: package versions, data files,
  environment, random seeds — none captured by `nbformat` itself.

**Consequence for shard-wiki:** the captured outputs are a **snapshot projection with weak
provenance** — honest treatment must mark them as "computed at run N, environment not
guaranteed," never as live or authoritative truth.

## 3. The ecosystem (relevant to attach/project/translate)

- **nbconvert** — derives other forms from a notebook: HTML, Markdown, LaTeX/PDF, slides,
  script. This is **derivation-projection** (T1): notebook source → rendered view, lossy in
  both directions (HTML keeps outputs; `--to script` keeps only code, like `tangle`).
- **Jupytext** — represents a notebook **as** a `.py`/`.md` text file (pairing), making it
  **git-diffable plain text** and round-trippable — directly relevant to storing notebooks
  in a git shard without JSON-diff noise.
- **papermill** — parameterize + execute a notebook to produce a new output notebook
  (notebook as a runnable template — a *derivation with inputs*).
- **JupyterLab / Notebook / nbviewer / Colab** — front-ends; nbviewer renders a static
  read-only projection from a URL (a natural projection target).
- **`nbstripout`** — strips outputs before commit: teams treat **outputs as derived noise**,
  keeping only source under version control — an explicit "source canonical, outputs
  derived" stance mirroring T1.

## 4. Capability profile (as a shard / content type)

| Dimension (synthesis spectrum) | Jupyter notebook |
|--------------------------------|------------------|
| Attachment mode | file-store (`.ipynb` JSON in a repo) or via Jupyter Server REST API |
| Addressing granularity | document; **cell** as sub-address (by index / id; `nbformat 4.5+` adds stable cell `id`) |
| Content identity | file path; cell `id` (4.5+) else positional |
| Structure | **ordered cell list** (markdown / code+outputs / raw); MIME-bundle outputs |
| History | VCS on the file; **JSON diffs are noisy** unless paired (Jupytext) or stripped |
| Merge model | git on JSON (poor) → **paired text** (good) or nbdime (cell-aware diff/merge) |
| Native query | none |
| Translation | nbconvert → HTML/MD/script/PDF (lossy, directional); Jupytext text pairing |
| Write granularity | file / **cell** |
| Operational envelope | a kernel + environment to (re)execute; static render needs none |
| Content opacity | **mixed**: source transparent; outputs = MIME blobs (some opaque, e.g. base64 PNG) |
| Provenance | `execution_count` (weak, out-of-order); environment **not** captured |
| **Computed-output** | **stored inline**, snapshot, reproducibility out-of-band |

## 5. INTENT mapping

### Reinforcements

- **Replication- vs derivation-projection (T1) confirmed and extended.** nbconvert (→HTML/
  script) and nbviewer are derivation-projections; `--to script` is literally `tangle`.
  Jupyter adds a third wrinkle: **the derived output is also stored back inside the source**
  (captured outputs), so the "source vs projection" line runs *through* the document.
- **Union without erasure / provenance honesty.** Captured outputs must be surfaced **as
  snapshots with weak provenance** (run N, environment unguaranteed) — a concrete instance
  of "never hide freshness/authorship." The out-of-order `execution_count` is exactly the
  kind of fragility shard-wiki must *show*, not paper over.
- **Non-Markdown content + lossy translation (UC-55/UC-59).** `.ipynb` is JSON with embedded
  MIME-bundle outputs; any Markdown projection is **lossy** (loses live outputs, kernel,
  rich MIME). Surface the lossiness; keep the JSON as canonical payload (T12/T15).
- **Markdown island.** Markdown cells fit the text-first model, but only as *fragments
  inside* a JSON envelope — the adapter extracts/round-trips them, it does not pretend the
  notebook is a Markdown page.

### Divergences / boundaries

- **shard-wiki is not a kernel host.** Re-execution (driving a kernel) is out of scope/
  capability-gated; default treatment is **attach + present captured outputs as a snapshot
  projection** + offer nbconvert-style static render. Executing/parameterizing (papermill)
  is an optional capability, never assumed.
- **Outputs-in-source is an anti-pattern to respect, not adopt.** Teams strip/pair outputs
  precisely because mixing derived data into the source breaks diffs. shard-wiki should
  prefer the **source-canonical, outputs-as-derived** reading (Jupytext pairing / nbstripout
  ethos) and treat stored outputs as a capturable projection.

### What to keep

1. **Computational-notebook as a first-class content type** with cell structure + inline
   **computed outputs carrying (weak) execution provenance** — UC-84.
2. **Outputs = derivation-projection snapshot** (T1 vocabulary): regenerable only with a
   kernel+environment; degrade gracefully to the stored snapshot / static render.
3. **Cell-level addressing** (stable cell `id`, nbformat 4.5+) as the sub-page granularity
   for transclusion/anchoring (UC-32/UC-35).
4. **Text-pairing (Jupytext)** as the git-friendly storage strategy — feeds the
   history-portability thread (poor JSON diffs → paired text / nbdime).

## 6. UC seed

| # | Seed | Disposition |
|---|------|-------------|
| UC-84 | Attach/project a **computational notebook** (`.ipynb`): preserve **cell structure** (markdown / code / output) and **embedded computed outputs**, surfacing each output **as a snapshot with its (weak) execution provenance** (run count, environment not guaranteed) — re-execution is **capability-gated**, default is present-the-snapshot + offer a static rendered projection | **new** |
| — | Notebook JSON / MIME-bundle outputs = non-Markdown content; Markdown projection is lossy | enrich **UC-55**, **UC-59** |
| — | Computed/evaluated cell = computation-defined content | enrich **UC-54** |
| — | Cell `id` (nbformat 4.5+) = sub-page address for anchor/transclusion | enrich **UC-35**, links **UC-32** |
| — | Stored outputs as derived snapshot (nbstripout/Jupytext ethos) = source-canonical/outputs-derived | links **UC-83**, **UC-79** |

## 7. Architecture notes for SHARD-WP-0002

- **T12 (page model):** add **computational-notebook** as a page shape — an **ordered cell
  list** where code cells own **embedded computed outputs** (MIME bundles) with weak
  execution provenance. Distinct from prose, typed records, query-defined, inline-embedded
  objects (Quip/Notion), typed-graph (Wikibase), and the literate one-source-many-projection
  shape (UC-83). The defining new attribute: **derived output stored *inside* the source**.
- **T15 (translation / fidelity):** `.ipynb` is non-Markdown; nbconvert→Markdown is **lossy
  and directional** (drops live outputs/kernel/rich MIME). Keep JSON canonical; any Markdown
  is a projection. MIME-bundle outputs map to the content-opacity spectrum (text→html→base64
  image = transparent→opaque).
- **T13 (history):** JSON diffs are **noisy**; record **text-pairing (Jupytext)** and
  **cell-aware diff/merge (nbdime)** as history-portability strategies for embedded-output
  documents. Reinforces "source-canonical, outputs-derived."
- **T16 (projection):** captured outputs are a **derivation-projection snapshot**;
  re-execution (kernel) and parameterized execution (papermill) are **capabilities**, not
  assumptions; degrade to the stored snapshot / nbviewer-style static render.

## 8. Open questions

1. Does shard-wiki ever **re-execute** a notebook (host/broker a kernel), or strictly
   attach + present captured outputs + static render? (Same scope boundary as UC-83/UC-56
   "do we ever drive the derivation.")
2. Is **UC-84** distinct from **UC-83**, or is a notebook just the "outputs-stored-in-source"
   special case of the literate one-source-many-projection pattern? (Kept separate: UC-84's
   defining trait is *captured derived output embedded in the canonical source with weak
   provenance* — a page-model attribute UC-83 doesn't carry.)
3. How are **MIME-bundle outputs** represented in the page model — opaque provenance-tagged
   blobs, a typed-asset registry (UC-55 open question #10), or selected-MIME projection?
4. Default storage: attach `.ipynb` **as-is** (JSON, noisy diffs) or prefer a **paired text
   representation** when the shard is a git repo? (Policy → configurable.)

## 9. Sources

- Jupyter `nbformat` reference (cells, outputs, MIME bundles, cell `id` 4.5+);
  Jupyter messaging protocol / kernels docs.
- **nbconvert**, **nbviewer**, **JupyterLab**, **Colab** docs.
- **Jupytext**, **papermill**, **nbdime**, **nbstripout** project docs.
- prior: `research/260614-literate-programming-deep-dive/` (replication- vs
  derivation-projection, UC-83); `research/260614-notion-deep-dive/` (block-JSON,
  external-API), `research/260614-quip-deep-dive/` (inline embedded objects, UC-55/58/59).

## 10. Traceability

New UC **UC-84** carries the marker **⊜** in the wikiengines column of
`spec/UseCaseCatalog.md` (true lineage = this dive). Enriched: UC-54, UC-55, UC-59, UC-35;
links UC-32, UC-83, UC-79. Architecture cross-refs: SHARD-WP-0002 T12 (notebook page shape:
outputs embedded in source), T15 (lossy non-Markdown translation; MIME opacity), T13
(paired-text / nbdime history), T16 (output = derivation-projection snapshot; kernel =
capability).