shard-wiki/research/260614-literate-programming-deep-dive/findings.md

Literate Programming (Knuth's WEB / weave / tangle) — deep dive (findings)

Date: 2026-06-14 · Source: SHARD-WP-0004 T1 · Subject: Donald Knuth's WEB system and the literate-programming model (weave/tangle, named chunks).

Why this dive

SHARD-WP-0004 asks the carried question: can a shard-wiki page be a source that is woven/evaluated into rendered forms, and how do projection/transclusion/provenance treat the source vs the output? Literate programming is the deepest ancestor of that idea. Knuth (1984) inverted the program/comment relationship: you write a document for humans whose fragments happen to also be the program. From the one WEB source two tools derive two artifacts: weave → typeset documentation (TeX) and tangle → compilable code (Pascal/C/…). This is one source, two projections in its purest, oldest form — the conceptual root of shard-wiki's projection and transclusion.

1. The WEB model: one source, two tools

A WEB file interleaves prose and code in author-chosen order (the order that best explains, not the order the compiler needs). Two programs read it:

• weave produces a .tex file → typeset documentation: prose, pretty-printed code, a cross-reference index of where each chunk and identifier is defined/used.
• tangle produces a compilable source file → reorders and expands the code chunks into the sequence the compiler demands, strips the prose, macro-expands references.

The crucial property: the two outputs are co-derived from one canonical source and are semantically different audiences (human reader vs compiler). Neither output is the source; both are regenerable, disposable derivations. Editing happens on the WEB; you never edit the woven .tex or the tangled .c.

2. Named chunks = transclusion of code fragments

The organizing primitive is the named section / code chunk:

• A chunk is declared <<name>>= and referenced elsewhere as <<name>>. tangle expands references in place (recursively) to assemble the final program.
• A chunk name can be defined in multiple places and is accreted (later += additions append) — so one logical unit is authored across scattered locations.
• References can appear before definitions; resolution is by name, not by position.

This is transclusion by reference (UC-32) and compose-by-reference / manifest (UC-44, the EDL/Xanadu lineage): the document is a graph of named fragments assembled at derivation time. Knuth's chunk graph is the same shape as Xanadu's reference-not-copy and shard-wiki's "compose a page from referenced parts" — applied to executable content and resolved by a build tool rather than a viewer.

3. The descendants (noweb, CWEB, org-babel, Sweave/knitr, Jupytext)

• CWEB (Knuth/Levy): WEB for C. noweb (Ramsey): language-agnostic, minimal markup (<<chunk>>), notangle/noweave — proof that the model (chunks + two projections) is separable from any one language or typesetter.
• org-babel (Emacs Org-mode): named source blocks, :noweb references, tangle to files and evaluate blocks inline (results captured back into the document) — literate programming fused with notebook execution (bridges to T2/T3).
• Sweave / knitr (R): weave prose + R, executing code and interleaving computed results/figures into the woven document — adds the computed-output dimension that Jupyter (T3) centers on.
• Jupytext: represents a Jupyter notebook as a literate text/Markdown source — closing the loop: the notebook (T3) becomes a weave/tangle-style plain-text source.

The throughline: one canonical source → N derived projections, where projections may be (a) reformatted (weave), (b) reordered/extracted (tangle), or (c) evaluated (org-babel/knitr) — the evaluated case is exactly the computational-page question.

4. Capability profile (as a would-be shard / page type)

Dimension (synthesis spectrum)	Literate-programming source
Attachment mode	file-store (a WEB/.nw/.org text source in a repo)
Addressing granularity	document; named chunk as sub-page address
Content identity	source file path + chunk name (name-resolved, not position)
Structure	graph of named chunks assembled by reference
History	whatever VCS holds the source (git) — text, diffable
Merge model	text/git merge on the source
Native query	none; weave emits a cross-reference index (derived)
Translation	source → woven docs and tangled code (build-time)
Write granularity	file / chunk (text region)
Operational envelope	a build tool (tangle/weave/noweb/babel)
Content opacity	transparent text
Provenance	VCS author/time; chunk cross-ref maps output→source location
Projection model	one source → many co-equal derived projections (new emphasis)

5. INTENT mapping

Reinforcements

• Projection (canonical vs derived). Literate programming is the archetype of "canonical source, regenerable derived view" — the same principle as ikiwiki compile-to-static (UC-79), but generalized: two-plus co-equal projections from one source (docs and code), not a single output. Strengthens the rule never confuse a projection for the source.
• Transclusion / compose-by-reference. Named chunks are transclusion (UC-32) and a manifest of referenced parts (UC-44) — resolved at derivation time. Confirms transclusion=clone=embed=reference as one primitive that also covers fragment assembly of executable content.
• Markdown-first but backend-neutral page model. noweb/org/Jupytext show the literate source can be Markdown-ish plain text — so a "literate page" fits the text-first model; the derivations are the non-text part.
• Mechanism over policy. weave/tangle are mechanisms; which projections to materialize, when to regenerate, and where outputs go stay configurable.

Divergences / boundaries

• shard-wiki is not a build system. It should recognize and present a source-with-projections (attach the source, surface derived views with provenance), not re-implement tangle/kernels. Materializing a projection may delegate to the source's own tool or be capability-gated to "snapshot only."
• The interesting projection is derivation, not caching. shard-wiki's base projection is cache-like (lazy copy of remote content, UC-53/57). Weave/tangle is a different projection species: transform/derive one source into rendered forms. The contract should model projection as having (at least) two kinds: replication-projection and derivation-projection (compile/weave/evaluate).

What to keep

1. One-source-many-projections as a first-class page-model + projection pattern (generalizes UC-79's single output) — see UC-83.
2. Named-chunk transclusion as the executable-content face of UC-32/UC-44 (assembly by reference at derivation time).
3. Output→source provenance (the woven cross-ref index): every derived view must point back to the exact source location it came from — never present derived output without that link (union-without-erasure for derivations).

6. UC seed

#	Seed	Disposition
UC-83	Attach a single-source-multiple-projection artifact (a literate/woven source): treat the source as canonical, present each derived projection (e.g. a documentation view and a code view) with provenance back to the one source, edits target the source and projections regenerate (delegated to the source's tool or degraded to a static snapshot)	new
—	Named chunks <<name>> = transclusion / compose-by-reference of (executable) fragments	enrich UC-32, UC-44
—	Generalize compile-to-static (single output) to N co-equal projections from one source	enrich UC-79
—	Computed/evaluated projection (org-babel/knitr) = derivation-projection with results	links UC-54, foreshadows T3

7. Architecture notes for SHARD-WP-0002

• T12 (page model): add one-source-many-projections as a page-model shape — a page may be a source whose presented forms are derivations (woven docs, tangled code, evaluated results), each carrying output→source provenance. Distinct from prose, typed records, query-defined, and inline-embedded objects already logged.
• T16 (projection): split projection into replication-projection (lazy cache of remote content — current default) vs derivation-projection (transform/compile/weave/ evaluate a source into rendered forms). Derivation-projection is regenerable, may delegate to an external tool, and degrades to a captured snapshot when the tool is absent (graceful degradation).
• Transclusion (T16): named-chunk-by-name resolution is a transclusion variant where the target is a fragment resolved by name across the source, assembled at derivation time — a concrete shape for UC-32/UC-44 mechanics.
• Capability gating: "can derive projection X" is a capability; a shard that can't run the tool still exposes the source + any pre-built/snapshot projections (UC-83 degrade).

8. Open questions

1. Does shard-wiki ever drive a derivation (run weave/tangle/evaluate), or only attach sources and surface pre-built projections + snapshots? (Same scope question as UC-56 "do we ever compile-to-static ourselves," now for literate sources.)
2. Is UC-83 distinct enough from UC-79 (compile-to-static) to stand alone, or should UC-79 be re-read as the single-output special case of UC-83's N-projection general case? (Recorded as a possible later consolidation; kept separate now because UC-83's projections are co-equal and semantically different audiences, not one publish target.)
3. How is output→source provenance represented when a derived line came from a chunk accreted across several source locations (the cross-ref is many-to-one)?

9. Sources

• Knuth, Literate Programming (1984, Computer Journal); the WEB user manual; TeX: The Program / MMIX as canonical WEB exemplars.
• Ramsey, noweb (a simple, extensible literate-programming tool).
• CWEB (Knuth & Levy); Emacs Org-mode babel (tangle + evaluate); Sweave/knitr (R); Jupytext (notebook-as-text).
• prior: research/260614-ikiwiki-deep-dive/ (compile-to-static, canonical-vs-derived); research/260614-xanadu-deep-dive/ (compose-by-reference / EDL, UC-44).

10. Traceability

New UC UC-83 carries the marker ⊛ in the federation column of spec/UseCaseCatalog.md (true lineage = this dive; literate programming is design prior art, not a candidate shard, so the marker sits with the projection/compose-by-reference family). Enriched: UC-32, UC-44, UC-79; links UC-54. Architecture cross-refs: SHARD-WP-0002 T12 (one-source-many-projections page shape), T16 (replication- vs derivation-projection; named-chunk transclusion).