diff --git a/.claude/ralph-loop.local.md b/.claude/ralph-loop.local.md new file mode 100644 index 0000000..e7a68d8 --- /dev/null +++ b/.claude/ralph-loop.local.md @@ -0,0 +1,21 @@ +--- +active: true +iteration: 1 +session_id: b24b53ca-a25b-4a18-8331-925c4100d39f +max_iterations: 20 +completion_promise: "HEUREKA" +workplan_id: SHARD-WP-0004 +workplan_file: workplans/SHARD-WP-0004-computational-knowledge-systems.md +started_at: "2026-06-14T20:35:07Z" +--- + +Read the workplan at `workplans/SHARD-WP-0004-computational-knowledge-systems.md`. + +If every task has `status: done` AND frontmatter `status: done`: +run `rm -f .claude/ralph-loop.local.md` first (deactivates the loop so the stop hook exits cleanly), +then output HEUREKA. + +Otherwise implement the next `todo` task as described in the workplan. +Set task `in_progress` when starting, `done` when complete. +When all tasks are done set frontmatter `status: done`. + diff --git a/SCOPE.md b/SCOPE.md index 566a522..e665d88 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -21,8 +21,8 @@ Learnings update both SCOPE and INTENT where necessary. | Intent | `INTENT.md` established; authorization-in-core amendments drafted | | Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) | | Demand | NetKingdom integration asks captured, not yet negotiated | -| Spec | Architecture blueprint drafted; UseCaseCatalog 82 UCs from research; PRD/TSD scaffolds | -| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` active (8 computational-knowledge dives) | +| Spec | Architecture blueprint drafted; UseCaseCatalog 83 UCs from research; PRD/TSD scaffolds | +| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` active (8 computational-knowledge dives: T1 literate programming done) | ## In Scope (today) diff --git a/research/260614-literate-programming-deep-dive/README.md b/research/260614-literate-programming-deep-dive/README.md new file mode 100644 index 0000000..900a625 --- /dev/null +++ b/research/260614-literate-programming-deep-dive/README.md @@ -0,0 +1,35 @@ +# 260614 — Literate Programming (Knuth's WEB / weave / tangle) deep dive + +Date: 2026-06-14 · Source: **SHARD-WP-0004 T1** + +## What this is + +A deep dive into **literate programming** — Knuth's WEB and the **`weave`/`tangle`** model +— as the deepest ancestor of shard-wiki's **projection** and **transclusion** ideas +applied to *executable* content. The keystone: **one canonical source → two co-derived +projections** (typeset docs via `weave`, compilable code via `tangle`), plus **named code +chunks** assembled by reference (transclusion). + +## Why it matters + +- Establishes **one-source-many-projections** as a page-model + projection pattern that + *generalizes* compile-to-static (UC-79, single output) to **N co-equal, semantically + different** derived views — feeds SHARD-WP-0002 **T12/T16**. +- Splits projection into **replication-projection** (lazy cache, current default) vs + **derivation-projection** (transform/compile/weave/evaluate) — a distinction the rest of + the computational batch (notebooks, REPLs) extends. +- Named chunks are the **executable-content face of transclusion / compose-by-reference** + (UC-32 / UC-44). + +## Yield + +- **UC-83** (new): attach a single-source-multiple-projection (literate) artifact; present + each derived view with output→source provenance; edits target the source. +- Enrich **UC-32**, **UC-44**, **UC-79**; links **UC-54** (computed/evaluated projection, + → T3 Jupyter). + +## Contents + +| Path | Role | +|------|------| +| `findings.md` | WEB model, named-chunk transclusion, descendants (noweb/org-babel/knitr/Jupytext), capability profile, INTENT mapping, UC seed, architecture notes, open questions | diff --git a/research/260614-literate-programming-deep-dive/findings.md b/research/260614-literate-programming-deep-dive/findings.md new file mode 100644 index 0000000..30ffaa5 --- /dev/null +++ b/research/260614-literate-programming-deep-dive/findings.md @@ -0,0 +1,180 @@ +# Literate Programming (Knuth's WEB / weave / tangle) — deep dive (findings) + +**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T1 · **Subject:** Donald Knuth's WEB +system and the literate-programming model (`weave`/`tangle`, named chunks). + +## Why this dive + +SHARD-WP-0004 asks the carried question: *can a shard-wiki page be a source that is +woven/evaluated into rendered forms, and how do projection/transclusion/provenance treat +the source vs the output?* Literate programming is the **deepest ancestor** of that idea. +Knuth (1984) inverted the program/comment relationship: you write a **document** for +humans whose fragments happen to also be the program. From the **one WEB source** two +tools derive two artifacts: **`weave` → typeset documentation** (TeX) and **`tangle` → +compilable code** (Pascal/C/…). This is *one source, two projections* in its purest, +oldest form — the conceptual root of shard-wiki's **projection** and **transclusion**. + +## 1. The WEB model: one source, two tools + +A WEB file interleaves prose and code in author-chosen order (the order that best +*explains*, not the order the compiler needs). Two programs read it: + +- **`weave`** produces a `.tex` file → typeset documentation: prose, pretty-printed code, + a cross-reference index of where each chunk and identifier is defined/used. +- **`tangle`** produces a compilable source file → reorders and expands the code chunks + into the sequence the compiler demands, strips the prose, macro-expands references. + +The crucial property: **the two outputs are co-derived from one canonical source and are +semantically different audiences** (human reader vs compiler). Neither output is the +source; both are **regenerable, disposable derivations**. Editing happens on the WEB; you +never edit the woven `.tex` or the tangled `.c`. + +## 2. Named chunks = transclusion of code fragments + +The organizing primitive is the **named section / code chunk**: + +- A chunk is declared `<>=` and **referenced** elsewhere as `<>`. `tangle` + expands references in place (recursively) to assemble the final program. +- A chunk name can be **defined in multiple places** and is *accreted* (later `+=` + additions append) — so one logical unit is authored across scattered locations. +- References can appear before definitions; resolution is by name, not by position. + +This is **transclusion by reference** (UC-32) and **compose-by-reference / manifest** +(UC-44, the EDL/Xanadu lineage): the document is a graph of named fragments assembled at +derivation time. Knuth's chunk graph is the same shape as Xanadu's reference-not-copy and +shard-wiki's "compose a page from referenced parts" — applied to *executable* content and +resolved by a build tool rather than a viewer. + +## 3. The descendants (noweb, CWEB, org-babel, Sweave/knitr, Jupytext) + +- **CWEB** (Knuth/Levy): WEB for C. **noweb** (Ramsey): language-agnostic, minimal markup + (`<>`), `notangle`/`noweave` — proof that the *model* (chunks + two projections) + is separable from any one language or typesetter. +- **org-babel** (Emacs Org-mode): named source blocks, `:noweb` references, **tangle** to + files **and evaluate** blocks inline (results captured back into the document) — literate + programming fused with notebook execution (bridges to T2/T3). +- **Sweave / knitr** (R): weave prose + R, executing code and **interleaving computed + results/figures** into the woven document — adds the *computed-output* dimension that + Jupyter (T3) centers on. +- **Jupytext**: represents a Jupyter notebook **as** a literate text/Markdown source — + closing the loop: the notebook (T3) becomes a weave/tangle-style plain-text source. + +The throughline: **one canonical source → N derived projections**, where projections may +be (a) reformatted (weave), (b) reordered/extracted (tangle), or (c) **evaluated** +(org-babel/knitr) — the evaluated case is exactly the computational-page question. + +## 4. Capability profile (as a would-be shard / page type) + +| Dimension (synthesis spectrum) | Literate-programming source | +|--------------------------------|-----------------------------| +| Attachment mode | file-store (a WEB/`.nw`/`.org` text source in a repo) | +| Addressing granularity | document; **named chunk** as sub-page address | +| Content identity | source file path + chunk name (name-resolved, not position) | +| Structure | **graph of named chunks** assembled by reference | +| History | whatever VCS holds the source (git) — text, diffable | +| Merge model | text/git merge on the source | +| Native query | none; `weave` emits a cross-reference index (derived) | +| Translation | source → woven docs **and** tangled code (build-time) | +| Write granularity | file / **chunk** (text region) | +| Operational envelope | a build tool (`tangle`/`weave`/`noweb`/babel) | +| Content opacity | transparent text | +| Provenance | VCS author/time; chunk cross-ref maps output→source location | +| **Projection model** | **one source → many co-equal derived projections** (new emphasis) | + +## 5. INTENT mapping + +### Reinforcements + +- **Projection (canonical vs derived).** Literate programming is the archetype of + "canonical source, regenerable derived view" — the same principle as ikiwiki + compile-to-static (UC-79), but generalized: **two-plus co-equal projections** from one + source (docs *and* code), not a single output. Strengthens the rule *never confuse a + projection for the source*. +- **Transclusion / compose-by-reference.** Named chunks are transclusion (UC-32) and a + manifest of referenced parts (UC-44) — resolved at derivation time. Confirms + transclusion=clone=embed=reference as one primitive that also covers *fragment assembly + of executable content*. +- **Markdown-first but backend-neutral page model.** noweb/org/Jupytext show the literate + source can *be* Markdown-ish plain text — so a "literate page" fits the text-first model; + the *derivations* are the non-text part. +- **Mechanism over policy.** weave/tangle are mechanisms; *which* projections to + materialize, when to regenerate, and where outputs go stay configurable. + +### Divergences / boundaries + +- **shard-wiki is not a build system.** It should *recognize and present* a + source-with-projections (attach the source, surface derived views with provenance), not + re-implement `tangle`/kernels. Materializing a projection may delegate to the source's + own tool or be capability-gated to "snapshot only." +- **The interesting projection is derivation, not caching.** shard-wiki's base projection + is cache-like (lazy copy of remote content, UC-53/57). Weave/tangle is a *different* + projection species: **transform/derive** one source into rendered forms. The contract + should model projection as having (at least) two kinds: **replication-projection** and + **derivation-projection** (compile/weave/evaluate). + +### What to keep + +1. **One-source-many-projections** as a first-class page-model + projection pattern + (generalizes UC-79's single output) — see UC-83. +2. **Named-chunk transclusion** as the executable-content face of UC-32/UC-44 (assembly by + reference at derivation time). +3. **Output→source provenance** (the woven cross-ref index): every derived view must point + back to the exact source location it came from — never present derived output without + that link (union-without-erasure for derivations). + +## 6. UC seed + +| # | Seed | Disposition | +|---|------|-------------| +| UC-83 | Attach a **single-source-multiple-projection** artifact (a literate/woven source): treat the source as canonical, present each **derived projection** (e.g. a documentation view *and* a code view) with **provenance back to the one source**, edits target the source and projections regenerate (delegated to the source's tool or degraded to a static snapshot) | **new** | +| — | Named chunks `<>` = transclusion / compose-by-reference of (executable) fragments | enrich **UC-32**, **UC-44** | +| — | Generalize compile-to-static (single output) to **N co-equal projections** from one source | enrich **UC-79** | +| — | Computed/evaluated projection (org-babel/knitr) = derivation-projection with results | links **UC-54**, foreshadows **T3** | + +## 7. Architecture notes for SHARD-WP-0002 + +- **T12 (page model):** add **one-source-many-projections** as a page-model shape — a page + may be a *source* whose presented forms are **derivations** (woven docs, tangled code, + evaluated results), each carrying output→source provenance. Distinct from prose, typed + records, query-defined, and inline-embedded objects already logged. +- **T16 (projection):** split projection into **replication-projection** (lazy cache of + remote content — current default) vs **derivation-projection** (transform/compile/weave/ + evaluate a source into rendered forms). Derivation-projection is regenerable, may + delegate to an external tool, and degrades to a captured snapshot when the tool is + absent (graceful degradation). +- **Transclusion (T16):** named-chunk-by-name resolution is a transclusion variant where + the *target is a fragment resolved by name across the source*, assembled at derivation + time — a concrete shape for UC-32/UC-44 mechanics. +- **Capability gating:** "can derive projection X" is a capability; a shard that can't run + the tool still exposes the source + any pre-built/snapshot projections (UC-83 degrade). + +## 8. Open questions + +1. Does shard-wiki ever *drive* a derivation (run weave/tangle/evaluate), or only attach + sources and surface pre-built projections + snapshots? (Same scope question as UC-56 + "do we ever compile-to-static ourselves," now for literate sources.) +2. Is **UC-83** distinct enough from UC-79 (compile-to-static) to stand alone, or should + UC-79 be re-read as the single-output special case of UC-83's N-projection general case? + (Recorded as a possible later consolidation; kept separate now because UC-83's + projections are *co-equal and semantically different audiences*, not one publish target.) +3. How is **output→source provenance** represented when a derived line came from a chunk + accreted across several source locations (the cross-ref is many-to-one)? + +## 9. Sources + +- Knuth, *Literate Programming* (1984, *Computer Journal*); the WEB user manual; *TeX: The + Program* / *MMIX* as canonical WEB exemplars. +- Ramsey, *noweb* (a simple, extensible literate-programming tool). +- CWEB (Knuth & Levy); Emacs **Org-mode babel** (tangle + evaluate); **Sweave**/**knitr** + (R); **Jupytext** (notebook-as-text). +- prior: `research/260614-ikiwiki-deep-dive/` (compile-to-static, canonical-vs-derived); + `research/260614-xanadu-deep-dive/` (compose-by-reference / EDL, UC-44). + +## 10. Traceability + +New UC **UC-83** carries the marker **⊛** in the federation column of +`spec/UseCaseCatalog.md` (true lineage = this dive; literate programming is design prior +art, not a candidate shard, so the marker sits with the projection/compose-by-reference +family). Enriched: UC-32, UC-44, UC-79; links UC-54. Architecture cross-refs: SHARD-WP-0002 +T12 (one-source-many-projections page shape), T16 (replication- vs derivation-projection; +named-chunk transclusion). diff --git a/research/README.md b/research/README.md index 6139a6a..31c6128 100644 --- a/research/README.md +++ b/research/README.md @@ -36,4 +36,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and | 2026-06-14 | `260614-quip-deep-dive/` | Salesforce Quip — closed-SaaS live docs + embedded spreadsheets/live apps; REST + lossy HTML import/export; Salesforce enterprise ACL; external-API payload-format facet + inline-object page model; UC-80 | | 2026-06-14 | `260614-mojomojo-deep-dive/` | MojoMojo — Perl Catalyst/DBIx::Class DB-backed wiki; pages + history in SQL tables, no file store/API → direct-DB-read binding; DB version rows as history source; UC-81 | | 2026-06-14 | `260614-oddmuse-deep-dive/` | Oddmuse — single Perl CGI, plain-text `page/` files + `keep/` revisions, no DB; the minimal file-store floor / graceful-degradation baseline; partial-history honesty; UC-82 | -| 2026-06-14 | `260614-usemodwiki-deep-dive/` | UseModWiki — flat-file ancestor (Wikipedia's MediaWiki Phase I); CamelCase linking; lineage grounding for the minimal file-store floor; enrichment-only (reinforces UC-82, UC-25) | \ No newline at end of file +| 2026-06-14 | `260614-usemodwiki-deep-dive/` | UseModWiki — flat-file ancestor (Wikipedia's MediaWiki Phase I); CamelCase linking; lineage grounding for the minimal file-store floor; enrichment-only (reinforces UC-82, UC-25) | +| 2026-06-14 | `260614-literate-programming-deep-dive/` | Literate programming (Knuth's WEB / weave / tangle) — one source → N co-equal derived projections (docs + code); named-chunk transclusion; splits replication- vs derivation-projection; SHARD-WP-0004 T1; UC-83 | \ No newline at end of file diff --git a/spec/UseCaseCatalog.md b/spec/UseCaseCatalog.md index f3557e0..c4faefb 100644 --- a/spec/UseCaseCatalog.md +++ b/spec/UseCaseCatalog.md @@ -1056,6 +1056,32 @@ lossy export. Feeds SHARD-WP-0002 T11 (payload-format + content-opacity), T12 (i objects), T14. **Priority:** Later +--- + +### UC-83 — Attach a single-source-multiple-projection (literate) artifact + +**Actor:** Orchestrator / adapter +**Goal:** Attach a **literate / woven source** (a Knuth-WEB / noweb / org-babel-style +artifact) as a shard whose canonical content is **one source** that derives **multiple +co-equal projections** (e.g. a documentation view *and* a code view), presenting each +derived form **with provenance back to the one source** and targeting edits at the source. +**Source:** wikiengines, intent +**Notes:** Literate programming is the deepest ancestor of shard-wiki's projection + +transclusion (`research/260614-literate-programming-deep-dive/findings.md`). It generalizes +**compile-to-static** (UC-79, *single* derived output) to **N co-equal, semantically +different** derivations (`weave`→docs, `tangle`→code), and its **named chunks** are +transclusion / compose-by-reference of executable fragments (UC-32, UC-44). Distinguishes +**derivation-projection** (transform/compile/weave/evaluate a source) from the default +**replication-projection** (lazy cache of remote content) — derivation-projection is +regenerable, may delegate to the source's own tool, and **degrades to a captured static +snapshot** when the tool is absent (graceful degradation). Every derived view must carry +**output→source provenance** (never present derivation without the link, union-without- +erasure). shard-wiki **recognizes and presents** such sources; it does not re-implement a +build system (driving derivation is capability-gated). Feeds SHARD-WP-0002 T12 +(one-source-many-projections page shape), T16 (replication- vs derivation-projection; +named-chunk transclusion). +**Priority:** Later + ## D. Addressing, identity & query *Span / identity addressing, transclusion-as-reference, dimensional navigation, and query over the union.* @@ -1557,6 +1583,7 @@ marker's true lineage is its per-source mapping subsection below): | ⚓ | wiki.js | ⊞ | federated-wiki | ⬡ | wikibase | | ⎇ | forge-wikis (Gitea/GitLab/GitHub) | ⊡ | tiddlywiki | ⊟ | ikiwiki | | ◧ | quip | ⊙ | mojomojo | ⊚ | oddmuse (UseModWiki reinforces, no marker) | +| ⊛ | literate-programming (WEB/weave/tangle) | | | | | | UC | c2 research | yawex research | federation research | wikiengines research | INTENT | |----|-------------|----------------|---------------------|----------------------|--------| @@ -1624,6 +1651,7 @@ marker's true lineage is its per-source mapping subsection below): | UC-80 | | | | ◧ | ✓ | | UC-81 | | | | ⊙ | ✓ | | UC-82 | ✓ | | | ⊚ | ✓ | +| UC-83 | | | ✓⊛ | | ✓ | | UC-08 | ✓ | | | | UC-09 | ✓ | | | | UC-10 | ✓ | | | @@ -2284,6 +2312,35 @@ attach-compatible with that ancestor (flat files + CamelCase identities + partia Architecture: a second instance of the **minimal/floor profile** (T11) and the CamelCase identity surface (UC-25) the adapter must parse/translate. +### literate-programming mapping + +(⊛ UC-83 lineage = the **literate-programming deep dive**, +`research/260614-literate-programming-deep-dive/findings.md`; design prior art, not a +candidate shard — marked in the federation/projection family column.) + +| Literate-programming mechanism (findings §) | Catalog UC | +|---------------------------------------------|------------| +| One WEB source → `weave` (docs) + `tangle` (code): N co-equal derived projections (§1) | UC-83 (new) | +| Generalizes compile-to-static (single output) to many co-equal projections (§1, §5) | UC-79 (enriched) | +| Named chunks `<>` = transclusion / compose-by-reference of executable fragments (§2) | UC-32 / UC-44 (enriched) | +| org-babel/knitr **evaluated** projection = computed/derivation projection (§3) | links UC-54 (→ T3 Jupyter) | +| Output→source cross-reference index = derivation provenance (§5) | links UC-24 | + +Note: Literate programming (Knuth's WEB; noweb/CWEB/org-babel/Sweave/knitr/Jupytext) is the +**deepest ancestor** of shard-wiki's projection + transclusion, applied to *executable* +content. Its defining contribution is **one canonical source → multiple co-equal, +semantically different derived projections** (`weave`→human docs, `tangle`→compiler code), +which **generalizes** ikiwiki's single compile-to-static output (UC-79) and splits +projection into **replication-projection** (lazy cache — the current default) vs +**derivation-projection** (transform/compile/weave/evaluate a source). **Named chunks** are +the executable-content face of transclusion / compose-by-reference (UC-32/UC-44), resolved +by name at derivation time. **Boundary recorded:** shard-wiki *recognizes and presents* +such sources (attach the source, surface derived views with output→source provenance); +driving the derivation is **capability-gated** and **degrades to a captured static +snapshot** — never present a derivation without its source link (union-without-erasure). +Architecture logged for `SHARD-WP-0002` (T12/T16): one-source-many-projections page shape, +replication- vs derivation-projection, named-chunk transclusion. + --- ## Open questions