diff --git a/SCOPE.md b/SCOPE.md index 773ac7e..4a90955 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -19,9 +19,9 @@ Learnings update both SCOPE and INTENT where necessary. |-------|-------| | Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) | | Intent | `INTENT.md` established; authorization-in-core amendments drafted | -| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) | +| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki deep dives & shard-spectrum synthesis (`research/260614-*/`) | | Demand | NetKingdom integration asks captured, not yet negotiated | -| Spec | Architecture blueprint drafted; UseCaseCatalog 78 UCs from research; PRD/TSD scaffolds | +| Spec | Architecture blueprint drafted; UseCaseCatalog 79 UCs from research; PRD/TSD scaffolds | | Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract); `SHARD-WP-0003` active (9 engine dives); `SHARD-WP-0004` active (8 computational-knowledge dives) | ## In Scope (today) diff --git a/research/260614-ikiwiki-deep-dive/README.md b/research/260614-ikiwiki-deep-dive/README.md new file mode 100644 index 0000000..e36cd72 --- /dev/null +++ b/research/260614-ikiwiki-deep-dive/README.md @@ -0,0 +1,15 @@ +# 260614 — ikiwiki deep dive + +Deep dive on **ikiwiki** (Joey Hess): a **wiki compiler** — it compiles a **VCS-backed** +(usually **git**) tree of Markdown source into **static HTML**, supports **distributed** +operation (clone/edit/push between wiki instances, change **pings**), and treats web edits as +commits to the same repo. + +- `findings.md` — the compiler model, git-distributed federation + pinger, static output, + capability profile, INTENT mapping, UC seed (UC-79), architecture notes for SHARD-WP-0002, + open questions, sources, traceability. + +Catalog yield: UC-79 (attach a git-backed **compile-to-static** wiki — git Markdown source +is the shard, compiled static HTML is a derived publish/projection; participate in +git-distributed clone federation with change-pings). Enriched UC-31/56/37/33. Feeds +SHARD-WP-0002 T4 (federation), T6 (publish/projection). diff --git a/research/260614-ikiwiki-deep-dive/findings.md b/research/260614-ikiwiki-deep-dive/findings.md new file mode 100644 index 0000000..d62c287 --- /dev/null +++ b/research/260614-ikiwiki-deep-dive/findings.md @@ -0,0 +1,153 @@ +# ikiwiki — deep dive (findings) + +**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T4 · **Subject:** ikiwiki, Joey Hess's +VCS-backed wiki compiler. + +## Why this dive + +The forge-wiki dive (T5) established *git-IS-the-store* for hosted Markdown wikis. ikiwiki +takes the same git-canonical source but adds two ideas shard-wiki cares about directly: +**compile-to-static** (the wiki is *built*, not served from a DB) and **git-distributed +federation** (wiki instances clone, pull, push, and **ping** each other). It is referenced +in `research/260608-federation-concepts/`; here we go into the model. + +## 1. The wiki-compiler model + +ikiwiki is fundamentally a **compiler**: input is a directory of **source pages** (Markdown +by default; also other formats) held in a **version-control repo**; output is a tree of +**static HTML**. A rebuild is triggered by a **VCS post-commit/post-update hook**, so: + +- **The VCS repo is canonical**; the HTML is *derived build output* (regenerable, disposable). +- **Web edits are commits.** The CGI edit interface writes the change *into the repo* (a + commit) and triggers a rebuild — so browser edits and `git push` edits converge on one + history. (Same convergence as forge wikis, but here the canonical store is *your* git repo, + not a forge's.) +- **VCS-agnostic**: git is usual, but svn/bzr/mercurial/darcs are supported via a VCS + plugin layer — an early "pluggable backend behind a stable interface" (adapter-contract + echo). +- **Plugins** (Perl) provide directives, feeds, auth (`openid`), and the federation hooks + below. + +## 2. Git-distributed federation + the pinger + +Because the source is an ordinary VCS repo, ikiwiki instances federate the way *git* does: + +- **Clone-and-diverge**: you can `git clone` a wiki, edit offline, and `push`/`pull` between + instances — **multiple wiki clones that reconcile via git merge**. A wiki is a branch-space + (UC-33). +- **`pinger` / `pingee` plugins**: an instance can send an **XML-RPC ping** to another + ikiwiki when it changes, prompting the other to **pull and rebuild** — a lightweight + **subscribe/notify** primitive over the git-distributed mesh (UC-31). +- **`aggregate` plugin**: pulls external **RSS/Atom feeds** into the wiki as pages — an + inbound projection of remote content. + +So ikiwiki is *federation by git plus a ping* — distinct from fedwiki's fork/journal +(UC-72) and from Wikibase's query-time `SERVICE` (UC-74): a **third federation flavor**, +*VCS-replication federation with change notification*. + +## 3. Static output as a publish/projection target + +The compiled static HTML is a **read-only, regenerable projection** of the source: + +- It is a natural **outbound publish target** (UC-56): render the union (or a shard) to a + static site for hosting/backup, no server needed. +- It is also the **read-only backup** end (UC-37): a static snapshot that survives the engine. + +The key shard-wiki framing: **source (git Markdown) is the attachable shard; static HTML is a +derived projection** — never confuse the build output for the canonical content. + +## 4. Capability profile + +| Dimension (synthesis spectrum) | ikiwiki | +|--------------------------------|---------| +| Attachment mode | **file-store** (VCS-backed git Markdown source) | +| Addressing granularity | page = source file; path = identity | +| Content identity | path/filename (placement-bound) | +| Structure | directory tree of Markdown source + directives | +| History | **native VCS (git) history** | +| Merge model | **git** (clone/pull/push/merge across instances) | +| Native query | none; directives + plugins compute derived pages at build | +| Translation | Markdown source → static HTML (build-time render) | +| Write granularity | **file (page)** per commit | +| Operational envelope | a compiler + VCS hook; static hosting for output | +| Access grant | VCS/file perms; `openid` for web edits | +| Content opacity | transparent Markdown | +| Provenance | git author/timestamp; aggregated feeds carry source | +| Federation | **git replication + XML-RPC ping**; RSS aggregation | + +## 5. INTENT mapping + +### Reinforcements + +- **Git-canonical Markdown source** = the home case (shared with forge wikis UC-76/40): + attach the source repo, adopt its git log as the journal, write by commit. +- **Coordination layer is git** (INTENT): ikiwiki's whole federation *is* git replication + + a ping — the most literal realization of "Git-addressable coordination layer." +- **Projection vs canonical**: compile-to-static cleanly separates **canonical source** from + **derived output** — exactly shard-wiki's projection principle (static HTML = a lazy/cache + projection that is regenerable, never the source of truth). +- **Graceful degradation / publish**: static output is the trivial read-only backup and + outbound publish target (UC-37/UC-56). +- **Subscribe/notify mechanism, not policy**: the pinger is a *mechanism* (notify a peer to + pull); *which* peers, when, and conflict policy stay configurable. + +### Divergences (boundaries / notes) + +- ikiwiki is mostly a **reinforcement** of git-canonical-Markdown (UC-76) — its *new* + contributions are (a) **compile-to-static** as a distinct projection/publish direction and + (b) the **git-distributed-clone + ping** federation flavor. The static output is *not* a + shard to attach (it's derived); attach the **source repo**. +- VCS-agnostic backend layer is an interesting adapter-contract echo, but for shard-wiki the + git case dominates. + +### What to keep + +1. **Source-repo-as-shard, static-output-as-derived-projection** — never attach the build + output as canonical (UC-79; relates UC-37/UC-56). +2. **Git-replication + change-ping** as a named federation flavor beside fork/journal + (fedwiki) and query-SERVICE (Wikibase) — a **subscribe/notify over a git mesh** (UC-31/UC-33). +3. **Inbound feed aggregation** (RSS/Atom → pages) as an inbound projection pattern. + +## 6. UC seed + +| # | Seed | Disposition | +|---|------|-------------| +| UC-79 | Attach a git-backed **compile-to-static** wiki — the **git Markdown source is the shard**, compiled **static HTML is a derived publish/projection**; participate in **git-distributed clone federation** with change-**pings** | **new** | +| — | XML-RPC pinger = subscribe/notify over a git mesh | enrich **UC-31** | +| — | render union/shard to a static site (publish) | enrich **UC-56** | +| — | static HTML as read-only regenerable backup | enrich **UC-37** | +| — | wiki as a git branch-space; clones reconcile via merge | enrich **UC-33** | + +## 7. Architecture notes for SHARD-WP-0002 + +- **T4 (federation):** record **git-replication + change-ping** as a federation flavor — peers + hold git clones, reconcile via merge, and notify with a ping to pull/rebuild. Distinct from + fedwiki fork/journal and Wikibase `SERVICE`; complements the "Git-addressable coordination + layer" mandate most literally. +- **T6 (publish/projection):** **compile-to-static** is the canonical *outbound* projection — + source (git) is canonical, static HTML is a regenerable derived view (publish/backup + target). Reinforces projection-vs-canonical separation. +- **Attach binding (T14):** attach the **source repo** (git Markdown), not the build output; + shares the forge-wiki / `wiki/`-subdir git+Markdown adapter (parameterized by repo/path). + +## 8. Open questions + +1. Is the **pinger** (notify-peer-to-pull) modeled as shard-wiki's own subscribe/notify + primitive, or only recognized when bridging two ikiwiki instances? +2. Does shard-wiki ever *drive* a compile-to-static **publish** of the union (act as the + ikiwiki-like compiler), or only attach existing ikiwiki source repos? (UC-56 scope.) +3. Is **feed aggregation** (RSS→pages) an inbound projection mode shard-wiki offers generally + (a feed-shard), or an ikiwiki-internal feature? + +## 9. Sources + +- ikiwiki.info — *ikiwiki* overview, *rcs* (VCS backends), *plugins* (`pinger`/`pingee`, + `aggregate`, `openid`), *setup* / post-commit hooks +- prior: `research/260608-federation-concepts/` (ikiwiki reference); + `research/260614-forge-wikis-deep-dive/` (git-canonical Markdown contrast) + +## 10. Traceability + +New UC **UC-79** carries the marker **⊟** in the wikiengines column of +`spec/UseCaseCatalog.md`. Enriched: UC-31, UC-56, UC-37, UC-33. Architecture cross-refs: +SHARD-WP-0002 T4 (git-replication+ping federation), T6 (compile-to-static publish), T14. diff --git a/research/README.md b/research/README.md index b613b85..a54721f 100644 --- a/research/README.md +++ b/research/README.md @@ -31,4 +31,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and | 2026-06-14 | `260614-federated-wiki-deep-dive/` | Federated Wiki — fork-with-provenance, per-page semantic-action journal (story=replay), neighborhood/roster + chorus; prior art for our coordination journal / overlay / union pillars; UC-70/71/72 | | 2026-06-14 | `260614-wikibase-deep-dive/` | Wikibase/Wikidata — typed entity-statement knowledge graph (claim+qualifiers+refs+rank), SPARQL/RDF + federated SERVICE, opaque stable IDs, statement-level provenance; structure & query far-end; UC-73/74/75 | | 2026-06-14 | `260614-forge-wikis-deep-dive/` | Gitea · GitLab · GitHub wikis — a wiki is a separate `.wiki.git` of Markdown; git-clone universal, wiki API capability-varying (GitHub git-only); git IS the store (resolves UC-68 race); the home case; UC-76/77 | -| 2026-06-14 | `260614-tiddlywiki-deep-dive/` | TiddlyWiki — entire wiki (content + app) in one self-contained HTML file = whole-file write-granularity extreme; Node `.tid` file-per-tiddler substrate (git-diffable); tiddler/field records, filter-expression query; UC-78 | \ No newline at end of file +| 2026-06-14 | `260614-tiddlywiki-deep-dive/` | TiddlyWiki — entire wiki (content + app) in one self-contained HTML file = whole-file write-granularity extreme; Node `.tid` file-per-tiddler substrate (git-diffable); tiddler/field records, filter-expression query; UC-78 | +| 2026-06-14 | `260614-ikiwiki-deep-dive/` | ikiwiki — wiki compiler: git-canonical Markdown source → static HTML (derived publish/projection); git-distributed clone federation + XML-RPC pinger (third federation flavor); UC-79 | \ No newline at end of file diff --git a/spec/UseCaseCatalog.md b/spec/UseCaseCatalog.md index 69faac6..5825e0a 100644 --- a/spec/UseCaseCatalog.md +++ b/spec/UseCaseCatalog.md @@ -15,7 +15,8 @@ Promoted from `research/260608-c2-wiki-origins/`, `research/260614-federated-wiki-deep-dive/`, and `research/260614-wikibase-deep-dive/`, and `research/260614-forge-wikis-deep-dive/`, and -`research/260614-tiddlywiki-deep-dive/`. +`research/260614-tiddlywiki-deep-dive/`, and +`research/260614-ikiwiki-deep-dive/`. See InfoTechPrimers on coulomb.social for use-case catalog conventions. ## Conventions @@ -197,7 +198,11 @@ concrete push transport: an engine event bus (`ObservationManager` eventual consistency and the rate limit (`research/260614-notion-deep-dive/findings.md` §4). **Joplin** is poll-based and turns concurrent edits into **conflict notes** (keep-both, not silent overwrite) — conflict-as-data, links UC-07 -(`research/260614-joplin-deep-dive/findings.md` §2). +(`research/260614-joplin-deep-dive/findings.md` §2). The **ikiwiki** dive details the +canonical pinger: an instance sends an **XML-RPC ping** to a peer when it changes, prompting +the peer to **pull and rebuild** — a subscribe/notify *mechanism* over a **git-distributed +mesh** (UC-79, `research/260614-ikiwiki-deep-dive/findings.md` §2), the federation flavor +beside fedwiki fork/journal (UC-72) and Wikibase `SERVICE` (UC-74). **Priority:** Later ### UC-32 — Transclude remote span with live freshness @@ -1075,6 +1080,23 @@ ignore the embedded engine). Feeds SHARD-WP-0002 T11 (whole-file tier), T14 (sin `.tid` binding; cf. UC-43/UC-62). **Priority:** Later +### UC-79 — Attach a git-backed compile-to-static wiki (source is the shard) + +**Actor:** Orchestrator / adapter +**Goal:** Attach a **compile-to-static** wiki (ikiwiki) where the **git Markdown source is +the shard** and the **compiled static HTML is a derived publish/projection**; optionally +participate in **git-distributed clone federation** with change-**pings**. +**Source:** wikiengines, federation, intent +**Notes:** ikiwiki compiles a VCS-backed (git) Markdown tree to static HTML; web edits are +commits, so git is canonical and HTML is regenerable build output +(`research/260614-ikiwiki-deep-dive/findings.md` §1, §3). **Attach the source repo, never +the build output** (the static site is a projection, UC-37/UC-56). Its federation is **git +replication + an XML-RPC pinger** (notify a peer to pull/rebuild, UC-31) — a third +federation flavor beside fedwiki fork/journal (UC-72) and Wikibase `SERVICE` (UC-74). Shares +the git+Markdown adapter with forge wikis (UC-76). Feeds SHARD-WP-0002 T4 (federation), T6 +(publish/projection). +**Priority:** Later + --- ## B. Knowledge work and collaboration @@ -1350,6 +1372,7 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD. | UC-76 | | | | ⎇ | ✓ | | UC-77 | | | | ⎇ | ✓ | | UC-78 | | | | ⊡ | ✓ | +| UC-79 | | | ✓ | ⊟ | ✓ | | UC-08 | ✓ | | | | UC-09 | ✓ | | | | UC-10 | ✓ | | | @@ -1881,6 +1904,33 @@ are a real native-query tier (UC-52). **Boundary recorded:** treat the single HT Architecture logged for `SHARD-WP-0002` (T11/T14): whole-file as the coarsest write tier, single-file-container vs `.tid`-dir dual binding. +### ikiwiki mapping + +(⊟ UC-79 lineage = the **ikiwiki deep dive**, `research/260614-ikiwiki-deep-dive/findings.md`; +marked in both federation and wikiengines columns.) + +| ikiwiki mechanism (findings §) | Catalog UC | +|--------------------------------|------------| +| VCS(git)-backed Markdown source compiled to static HTML; web edits = commits (§1, §3) | UC-79 (new) | +| `pinger`/`pingee` XML-RPC = notify a peer to pull/rebuild (§2) | UC-31 (enriched) | +| Compile union/shard to a static site = outbound publish (§3) | UC-56 (enriched) | +| Static HTML = read-only regenerable backup (§3) | UC-37 (enriched) | +| Clone/pull/push between instances; wiki = git branch-space (§2) | UC-33 (enriched) | +| `aggregate` (RSS/Atom → pages) = inbound feed projection (§2) | links UC-03 | +| VCS-agnostic backend behind a plugin layer (§1) | links UC-43 | + +Note: ikiwiki is a **wiki compiler** — git-canonical Markdown **source** built into **static +HTML** — so it mostly **reinforces** the git-IS-store home case (UC-76/40), but adds two +genuinely new things: **compile-to-static** as a clean *canonical-source vs derived-output* +separation (attach the **source repo**, treat static HTML as a regenerable +projection/publish target, UC-37/UC-56), and a **third federation flavor** — **git +replication + an XML-RPC change-ping** (peers hold clones, merge to reconcile, notify to +pull) beside fedwiki fork/journal (UC-72) and Wikibase `SERVICE` (UC-74). **Boundary +recorded:** never attach the build output as canonical; the pinger is a subscribe/notify +*mechanism* (peers/timing/conflict stay policy). Architecture logged for `SHARD-WP-0002` +(T4/T6/T14): git-replication+ping federation, compile-to-static publish, source-repo attach +sharing the git+Markdown adapter. + --- ## Open questions @@ -1951,5 +2001,9 @@ single-file-container vs `.tid`-dir dual binding. 26. For a **single-file wiki** (UC-78), does shard-wiki support per-page overlays by buffering and re-serializing the whole file, or require the Node `.tid` substrate for write-through and treat single-file as read/projection/backup only? (TiddlyWiki dive §9.) +27. For a **compile-to-static** wiki (UC-79), does shard-wiki ever *drive* the build to + **publish the union** as a static site (act as the compiler), or only attach existing + source repos — and is the **pinger** modeled as shard-wiki's own subscribe/notify + primitive or only recognized when bridging two ikiwiki instances? (ikiwiki dive §8.) 23. How does shard-wiki **honor/surface a shard's path-based access rules** (UC-06) in a projection without re-implementing its ACL engine? (Wiki.js dive §9.) \ No newline at end of file diff --git a/workplans/SHARD-WP-0003-engine-dives-batch.md b/workplans/SHARD-WP-0003-engine-dives-batch.md index 266bf09..983f7f4 100644 --- a/workplans/SHARD-WP-0003-engine-dives-batch.md +++ b/workplans/SHARD-WP-0003-engine-dives-batch.md @@ -100,7 +100,7 @@ model. The portability/granularity extreme. Enrich UC-35 (write granularity), UC ```task id: SHARD-WP-0003-T4 -status: todo +status: done priority: medium state_hub_task_id: "5748d246-66a8-4cf2-9b22-427cc4507b63" ```