diff --git a/SCOPE.md b/SCOPE.md index e665d88..225bed1 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -21,8 +21,8 @@ Learnings update both SCOPE and INTENT where necessary. | Intent | `INTENT.md` established; authorization-in-core amendments drafted | | Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) | | Demand | NetKingdom integration asks captured, not yet negotiated | -| Spec | Architecture blueprint drafted; UseCaseCatalog 83 UCs from research; PRD/TSD scaffolds | -| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` active (8 computational-knowledge dives: T1 literate programming done) | +| Spec | Architecture blueprint drafted; UseCaseCatalog 84 UCs from research; PRD/TSD scaffolds | +| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` active (8 computational-knowledge dives: T1 literate programming, T3 Jupyter done) | ## In Scope (today) diff --git a/research/260614-jupyter-deep-dive/README.md b/research/260614-jupyter-deep-dive/README.md new file mode 100644 index 0000000..75c1b82 --- /dev/null +++ b/research/260614-jupyter-deep-dive/README.md @@ -0,0 +1,35 @@ +# 260614 — Jupyter Notebooks deep dive + +Date: 2026-06-14 · Source: **SHARD-WP-0004 T3** + +## What this is + +A deep dive into **Jupyter Notebooks**: the **`.ipynb` JSON** document (ordered cells: +markdown / code+outputs / raw), **kernels**, **embedded computed outputs** (MIME bundles), +and **execution-count provenance**. The dominant modern computational document and the +concrete case where the **derived output is captured and stored *inside* the source** with +real-but-fragile provenance. + +## Why it matters + +- Tests the T1 **replication- vs derivation-projection** split on the dominant real artifact + and adds the wrinkle that **outputs are stored back inside the source** — the source/ + projection line runs *through* the document. +- The page model (T12) must carry a **notebook shape**: ordered cells with code cells owning + embedded computed outputs that have **weak execution provenance** (run order, environment + not captured). +- Non-Markdown + lossy translation (T15): JSON+MIME bundles; nbconvert→Markdown is lossy and + directional. JSON diffs are noisy → Jupytext text-pairing / nbdime (T13). + +## Yield + +- **UC-84** (new): attach/project a computational notebook preserving cell structure + + embedded outputs, surfacing outputs as **snapshots with weak execution provenance**; + re-execution **capability-gated**, default = present snapshot + static render. +- Enrich **UC-54, UC-55, UC-59, UC-35**; links **UC-32, UC-83, UC-79**. + +## Contents + +| Path | Role | +|------|------| +| `findings.md` | `.ipynb` model, kernels/execution-count fragility, ecosystem (nbconvert/Jupytext/papermill/nbdime/nbstripout), capability profile, INTENT mapping, UC seed, architecture notes, open questions | diff --git a/research/260614-jupyter-deep-dive/findings.md b/research/260614-jupyter-deep-dive/findings.md new file mode 100644 index 0000000..59521c0 --- /dev/null +++ b/research/260614-jupyter-deep-dive/findings.md @@ -0,0 +1,185 @@ +# Jupyter Notebooks — deep dive (findings) + +**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T3 · **Subject:** Jupyter Notebooks — the +`.ipynb` JSON document, kernels, embedded computed outputs, execution provenance. + +## Why this dive + +T1 (literate programming) established **one source → derived projections** and split +**replication-projection** from **derivation-projection**. Jupyter is the *dominant modern +computational document* and the concrete case where the **derived output is captured and +stored inside the source** — a non-Markdown, partially-executable content type whose +provenance is real but **fragile**. It is the most plausible concrete "computational shard" +content type, so it tests the page model (T12), lossy translation (T15), and the +output-provenance question head-on. + +## 1. The `.ipynb` document model + +A notebook is a single **JSON document** (`nbformat`), not Markdown: + +- **`cells[]`** — an ordered list. Each cell has a `cell_type`: + - `markdown` — prose (Markdown + LaTeX), the human-readable part. + - `code` — source text (`source`), plus an **`execution_count`** and an **`outputs[]`** + array captured from the last run. + - `raw` — passthrough. +- **`outputs[]`** (per code cell) carry results inline: `stream` (stdout/stderr), + `execute_result` / `display_data` (a **MIME bundle** — `text/plain`, `text/html`, + `image/png` base64, `application/json`, vendor MIME types), and `error` (traceback). +- **`metadata`** at notebook and cell level (`kernelspec`, `language_info`, tags like + `hide-input`, `scrolled`, slideshow roles). + +So an `.ipynb` is **source + last-run computed outputs + environment metadata, fused in one +JSON file**. The Markdown cells are an *island* inside a JSON envelope — relevant to how +shard-wiki extracts/round-trips content. + +## 2. Kernels and execution + +- A **kernel** is a separate language process (IPython, IRkernel, IJulia, …) speaking the + Jupyter messaging protocol (ZeroMQ). The document is **decoupled from the kernel**: the + `.ipynb` persists *captured* outputs; re-running requires a live kernel + the right + environment. +- **`execution_count`** numbers the order cells were *run*, which **need not match document + order** — the infamous **hidden-state / out-of-order execution** problem: stored outputs + may reflect a run sequence that no longer corresponds to top-to-bottom reading. +- Reproducibility therefore depends on **out-of-band state**: package versions, data files, + environment, random seeds — none captured by `nbformat` itself. + +**Consequence for shard-wiki:** the captured outputs are a **snapshot projection with weak +provenance** — honest treatment must mark them as "computed at run N, environment not +guaranteed," never as live or authoritative truth. + +## 3. The ecosystem (relevant to attach/project/translate) + +- **nbconvert** — derives other forms from a notebook: HTML, Markdown, LaTeX/PDF, slides, + script. This is **derivation-projection** (T1): notebook source → rendered view, lossy in + both directions (HTML keeps outputs; `--to script` keeps only code, like `tangle`). +- **Jupytext** — represents a notebook **as** a `.py`/`.md` text file (pairing), making it + **git-diffable plain text** and round-trippable — directly relevant to storing notebooks + in a git shard without JSON-diff noise. +- **papermill** — parameterize + execute a notebook to produce a new output notebook + (notebook as a runnable template — a *derivation with inputs*). +- **JupyterLab / Notebook / nbviewer / Colab** — front-ends; nbviewer renders a static + read-only projection from a URL (a natural projection target). +- **`nbstripout`** — strips outputs before commit: teams treat **outputs as derived noise**, + keeping only source under version control — an explicit "source canonical, outputs + derived" stance mirroring T1. + +## 4. Capability profile (as a shard / content type) + +| Dimension (synthesis spectrum) | Jupyter notebook | +|--------------------------------|------------------| +| Attachment mode | file-store (`.ipynb` JSON in a repo) or via Jupyter Server REST API | +| Addressing granularity | document; **cell** as sub-address (by index / id; `nbformat 4.5+` adds stable cell `id`) | +| Content identity | file path; cell `id` (4.5+) else positional | +| Structure | **ordered cell list** (markdown / code+outputs / raw); MIME-bundle outputs | +| History | VCS on the file; **JSON diffs are noisy** unless paired (Jupytext) or stripped | +| Merge model | git on JSON (poor) → **paired text** (good) or nbdime (cell-aware diff/merge) | +| Native query | none | +| Translation | nbconvert → HTML/MD/script/PDF (lossy, directional); Jupytext text pairing | +| Write granularity | file / **cell** | +| Operational envelope | a kernel + environment to (re)execute; static render needs none | +| Content opacity | **mixed**: source transparent; outputs = MIME blobs (some opaque, e.g. base64 PNG) | +| Provenance | `execution_count` (weak, out-of-order); environment **not** captured | +| **Computed-output** | **stored inline**, snapshot, reproducibility out-of-band | + +## 5. INTENT mapping + +### Reinforcements + +- **Replication- vs derivation-projection (T1) confirmed and extended.** nbconvert (→HTML/ + script) and nbviewer are derivation-projections; `--to script` is literally `tangle`. + Jupyter adds a third wrinkle: **the derived output is also stored back inside the source** + (captured outputs), so the "source vs projection" line runs *through* the document. +- **Union without erasure / provenance honesty.** Captured outputs must be surfaced **as + snapshots with weak provenance** (run N, environment unguaranteed) — a concrete instance + of "never hide freshness/authorship." The out-of-order `execution_count` is exactly the + kind of fragility shard-wiki must *show*, not paper over. +- **Non-Markdown content + lossy translation (UC-55/UC-59).** `.ipynb` is JSON with embedded + MIME-bundle outputs; any Markdown projection is **lossy** (loses live outputs, kernel, + rich MIME). Surface the lossiness; keep the JSON as canonical payload (T12/T15). +- **Markdown island.** Markdown cells fit the text-first model, but only as *fragments + inside* a JSON envelope — the adapter extracts/round-trips them, it does not pretend the + notebook is a Markdown page. + +### Divergences / boundaries + +- **shard-wiki is not a kernel host.** Re-execution (driving a kernel) is out of scope/ + capability-gated; default treatment is **attach + present captured outputs as a snapshot + projection** + offer nbconvert-style static render. Executing/parameterizing (papermill) + is an optional capability, never assumed. +- **Outputs-in-source is an anti-pattern to respect, not adopt.** Teams strip/pair outputs + precisely because mixing derived data into the source breaks diffs. shard-wiki should + prefer the **source-canonical, outputs-as-derived** reading (Jupytext pairing / nbstripout + ethos) and treat stored outputs as a capturable projection. + +### What to keep + +1. **Computational-notebook as a first-class content type** with cell structure + inline + **computed outputs carrying (weak) execution provenance** — UC-84. +2. **Outputs = derivation-projection snapshot** (T1 vocabulary): regenerable only with a + kernel+environment; degrade gracefully to the stored snapshot / static render. +3. **Cell-level addressing** (stable cell `id`, nbformat 4.5+) as the sub-page granularity + for transclusion/anchoring (UC-32/UC-35). +4. **Text-pairing (Jupytext)** as the git-friendly storage strategy — feeds the + history-portability thread (poor JSON diffs → paired text / nbdime). + +## 6. UC seed + +| # | Seed | Disposition | +|---|------|-------------| +| UC-84 | Attach/project a **computational notebook** (`.ipynb`): preserve **cell structure** (markdown / code / output) and **embedded computed outputs**, surfacing each output **as a snapshot with its (weak) execution provenance** (run count, environment not guaranteed) — re-execution is **capability-gated**, default is present-the-snapshot + offer a static rendered projection | **new** | +| — | Notebook JSON / MIME-bundle outputs = non-Markdown content; Markdown projection is lossy | enrich **UC-55**, **UC-59** | +| — | Computed/evaluated cell = computation-defined content | enrich **UC-54** | +| — | Cell `id` (nbformat 4.5+) = sub-page address for anchor/transclusion | enrich **UC-35**, links **UC-32** | +| — | Stored outputs as derived snapshot (nbstripout/Jupytext ethos) = source-canonical/outputs-derived | links **UC-83**, **UC-79** | + +## 7. Architecture notes for SHARD-WP-0002 + +- **T12 (page model):** add **computational-notebook** as a page shape — an **ordered cell + list** where code cells own **embedded computed outputs** (MIME bundles) with weak + execution provenance. Distinct from prose, typed records, query-defined, inline-embedded + objects (Quip/Notion), typed-graph (Wikibase), and the literate one-source-many-projection + shape (UC-83). The defining new attribute: **derived output stored *inside* the source**. +- **T15 (translation / fidelity):** `.ipynb` is non-Markdown; nbconvert→Markdown is **lossy + and directional** (drops live outputs/kernel/rich MIME). Keep JSON canonical; any Markdown + is a projection. MIME-bundle outputs map to the content-opacity spectrum (text→html→base64 + image = transparent→opaque). +- **T13 (history):** JSON diffs are **noisy**; record **text-pairing (Jupytext)** and + **cell-aware diff/merge (nbdime)** as history-portability strategies for embedded-output + documents. Reinforces "source-canonical, outputs-derived." +- **T16 (projection):** captured outputs are a **derivation-projection snapshot**; + re-execution (kernel) and parameterized execution (papermill) are **capabilities**, not + assumptions; degrade to the stored snapshot / nbviewer-style static render. + +## 8. Open questions + +1. Does shard-wiki ever **re-execute** a notebook (host/broker a kernel), or strictly + attach + present captured outputs + static render? (Same scope boundary as UC-83/UC-56 + "do we ever drive the derivation.") +2. Is **UC-84** distinct from **UC-83**, or is a notebook just the "outputs-stored-in-source" + special case of the literate one-source-many-projection pattern? (Kept separate: UC-84's + defining trait is *captured derived output embedded in the canonical source with weak + provenance* — a page-model attribute UC-83 doesn't carry.) +3. How are **MIME-bundle outputs** represented in the page model — opaque provenance-tagged + blobs, a typed-asset registry (UC-55 open question #10), or selected-MIME projection? +4. Default storage: attach `.ipynb` **as-is** (JSON, noisy diffs) or prefer a **paired text + representation** when the shard is a git repo? (Policy → configurable.) + +## 9. Sources + +- Jupyter `nbformat` reference (cells, outputs, MIME bundles, cell `id` 4.5+); + Jupyter messaging protocol / kernels docs. +- **nbconvert**, **nbviewer**, **JupyterLab**, **Colab** docs. +- **Jupytext**, **papermill**, **nbdime**, **nbstripout** project docs. +- prior: `research/260614-literate-programming-deep-dive/` (replication- vs + derivation-projection, UC-83); `research/260614-notion-deep-dive/` (block-JSON, + external-API), `research/260614-quip-deep-dive/` (inline embedded objects, UC-55/58/59). + +## 10. Traceability + +New UC **UC-84** carries the marker **⊜** in the wikiengines column of +`spec/UseCaseCatalog.md` (true lineage = this dive). Enriched: UC-54, UC-55, UC-59, UC-35; +links UC-32, UC-83, UC-79. Architecture cross-refs: SHARD-WP-0002 T12 (notebook page shape: +outputs embedded in source), T15 (lossy non-Markdown translation; MIME opacity), T13 +(paired-text / nbdime history), T16 (output = derivation-projection snapshot; kernel = +capability). diff --git a/research/README.md b/research/README.md index 31c6128..e53eb28 100644 --- a/research/README.md +++ b/research/README.md @@ -37,4 +37,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and | 2026-06-14 | `260614-mojomojo-deep-dive/` | MojoMojo — Perl Catalyst/DBIx::Class DB-backed wiki; pages + history in SQL tables, no file store/API → direct-DB-read binding; DB version rows as history source; UC-81 | | 2026-06-14 | `260614-oddmuse-deep-dive/` | Oddmuse — single Perl CGI, plain-text `page/` files + `keep/` revisions, no DB; the minimal file-store floor / graceful-degradation baseline; partial-history honesty; UC-82 | | 2026-06-14 | `260614-usemodwiki-deep-dive/` | UseModWiki — flat-file ancestor (Wikipedia's MediaWiki Phase I); CamelCase linking; lineage grounding for the minimal file-store floor; enrichment-only (reinforces UC-82, UC-25) | -| 2026-06-14 | `260614-literate-programming-deep-dive/` | Literate programming (Knuth's WEB / weave / tangle) — one source → N co-equal derived projections (docs + code); named-chunk transclusion; splits replication- vs derivation-projection; SHARD-WP-0004 T1; UC-83 | \ No newline at end of file +| 2026-06-14 | `260614-literate-programming-deep-dive/` | Literate programming (Knuth's WEB / weave / tangle) — one source → N co-equal derived projections (docs + code); named-chunk transclusion; splits replication- vs derivation-projection; SHARD-WP-0004 T1; UC-83 | +| 2026-06-14 | `260614-jupyter-deep-dive/` | Jupyter Notebooks — `.ipynb` JSON cells + embedded computed outputs with fragile execution provenance; derived output stored *inside* the source; non-Markdown/lossy; kernel = capability; SHARD-WP-0004 T3; UC-84 | \ No newline at end of file diff --git a/spec/UseCaseCatalog.md b/spec/UseCaseCatalog.md index c4faefb..024df59 100644 --- a/spec/UseCaseCatalog.md +++ b/spec/UseCaseCatalog.md @@ -1082,6 +1082,34 @@ build system (driving derivation is capability-gated). Feeds SHARD-WP-0002 T12 named-chunk transclusion). **Priority:** Later +--- + +### UC-84 — Attach or project a computational notebook with computed-output provenance + +**Actor:** Orchestrator / adapter +**Goal:** Attach/project a **computational notebook** (`.ipynb`) preserving its **cell +structure** (markdown / code / output) and its **embedded computed outputs**, surfacing each +output **as a snapshot carrying its (weak) execution provenance** (run count, environment +not guaranteed); re-execution is **capability-gated** — the default is to present the stored +snapshot plus an offered static rendered projection. +**Source:** wikiengines, intent +**Notes:** A Jupyter notebook is a **JSON document** of ordered cells where code cells own +**MIME-bundle outputs captured from the last run** +(`research/260614-jupyter-deep-dive/findings.md`). Its defining trait is that the **derived +output is stored *inside* the canonical source** — the source/projection line runs *through* +the document — and its provenance is **fragile** (`execution_count` may be out-of-order; +environment/data/versions are not captured by `nbformat`). Treatment: keep the JSON as +canonical payload (non-Markdown; any Markdown projection is lossy and directional, UC-55/ +UC-59); present captured outputs as **derivation-projection snapshots** (UC-83) marked "run +N, environment unguaranteed" (never as live/authoritative, union-without-erasure); offer +nbconvert/nbviewer static render; **re-execute or parameterize (papermill) only as a +capability**, never assumed (shard-wiki is not a kernel host). Cell `id` (nbformat 4.5+) is +the sub-page address for anchoring/transclusion (UC-32/UC-35); JSON diffs are noisy → +text-pairing (Jupytext) / cell-aware merge (nbdime) for history (T13). Feeds SHARD-WP-0002 +T12 (notebook page shape: outputs embedded in source), T15 (lossy non-Markdown translation; +MIME opacity), T13 (paired-text history), T16 (output = derivation-projection snapshot). +**Priority:** Later + ## D. Addressing, identity & query *Span / identity addressing, transclusion-as-reference, dimensional navigation, and query over the union.* @@ -1583,7 +1611,7 @@ marker's true lineage is its per-source mapping subsection below): | ⚓ | wiki.js | ⊞ | federated-wiki | ⬡ | wikibase | | ⎇ | forge-wikis (Gitea/GitLab/GitHub) | ⊡ | tiddlywiki | ⊟ | ikiwiki | | ◧ | quip | ⊙ | mojomojo | ⊚ | oddmuse (UseModWiki reinforces, no marker) | -| ⊛ | literate-programming (WEB/weave/tangle) | | | | | +| ⊛ | literate-programming (WEB/weave/tangle) | ⊜ | jupyter (notebooks) | | | | UC | c2 research | yawex research | federation research | wikiengines research | INTENT | |----|-------------|----------------|---------------------|----------------------|--------| @@ -1652,6 +1680,7 @@ marker's true lineage is its per-source mapping subsection below): | UC-81 | | | | ⊙ | ✓ | | UC-82 | ✓ | | | ⊚ | ✓ | | UC-83 | | | ✓⊛ | | ✓ | +| UC-84 | | | | ⊜ | ✓ | | UC-08 | ✓ | | | | UC-09 | ✓ | | | | UC-10 | ✓ | | | @@ -2341,6 +2370,37 @@ snapshot** — never present a derivation without its source link (union-without Architecture logged for `SHARD-WP-0002` (T12/T16): one-source-many-projections page shape, replication- vs derivation-projection, named-chunk transclusion. +### jupyter mapping + +(⊜ UC-84 lineage = the **Jupyter deep dive**, `research/260614-jupyter-deep-dive/findings.md`; +a concrete computational content type, not a wiki engine.) + +| Jupyter mechanism (findings §) | Catalog UC | +|--------------------------------|------------| +| `.ipynb` JSON: ordered cells, code cells own embedded MIME-bundle outputs (§1) | UC-84 (new) | +| Captured outputs stored *inside* the source; weak `execution_count` provenance (§2) | UC-84; links UC-83 (derivation-projection snapshot) | +| JSON + MIME bundles = non-Markdown; nbconvert→MD lossy & directional (§3) | UC-55 / UC-59 (enriched) | +| Code cell = computation-defined content (§1) | UC-54 (enriched) | +| Cell `id` (nbformat 4.5+) = sub-page address (§4) | UC-35 (enriched); links UC-32 | +| Noisy JSON diffs → Jupytext pairing / nbdime cell-merge (§3) | links UC-43, T13 | +| nbstripout/pairing = source-canonical, outputs-derived (§3) | links UC-79, UC-83 | + +Note: Jupyter is the **dominant computational document** and the concrete test of T1's +projection split: nbconvert/nbviewer are **derivation-projections**, but the twist is that +**captured outputs are stored back inside the canonical source** (the source/projection line +runs *through* the document) with **fragile provenance** (out-of-order `execution_count`; +environment uncaptured). The page model (T12) gains a **notebook shape** — ordered cells with +code cells owning embedded computed outputs — distinct from prose, typed records, +query-defined, inline-embedded objects (Quip/Notion), typed-graph (Wikibase), and the +literate one-source-many-projection shape (UC-83); its new attribute is *derived output +embedded in source*. **Boundary recorded:** shard-wiki is **not a kernel host** — default to +attach + present captured outputs as snapshots marked "run N, environment unguaranteed" +(never live, UC-84) + offer static render; re-execution/papermill are **capabilities**. Keep +JSON canonical (Markdown projection is lossy, UC-55/59); prefer paired-text storage / nbdime +for history when the shard is git. Architecture logged for `SHARD-WP-0002` (T12/T15/T13/T16): +notebook page shape, lossy non-Markdown + MIME opacity, paired-text history, output = +derivation-projection snapshot. + --- ## Open questions diff --git a/workplans/SHARD-WP-0004-computational-knowledge-systems.md b/workplans/SHARD-WP-0004-computational-knowledge-systems.md index fc8b87b..a246ab0 100644 --- a/workplans/SHARD-WP-0004-computational-knowledge-systems.md +++ b/workplans/SHARD-WP-0004-computational-knowledge-systems.md @@ -92,7 +92,7 @@ provenance, and reproducibility. Enrich UC-54 (query/computation-defined content ```task id: SHARD-WP-0004-T3 -status: todo +status: done priority: high state_hub_task_id: "3d35d58f-a9ea-44f9-89ea-d89205da8d8b" ```