generated from coulomb/repo-seed
research: ikiwiki deep dive (compile-to-static, git-distributed); UC-79
SHARD-WP-0003 T4. Wiki compiler: git-canonical Markdown source -> static HTML (derived publish/projection, not the canonical store). Two new insights atop the git-IS-store home case: compile-to-static = clean canonical-source vs derived-output separation (attach source repo, UC-37/UC-56), and a third federation flavor -- git replication + XML-RPC change-ping (pinger), beside fedwiki fork/journal (UC-72) and Wikibase SERVICE (UC-74). UC-79. Enriched UC-31/56/37/33. Marks T4 done. Feeds SHARD-WP-0002 T4/T6/T14. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
4
SCOPE.md
4
SCOPE.md
@@ -19,9 +19,9 @@ Learnings update both SCOPE and INTENT where necessary.
|
||||
|-------|-------|
|
||||
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
|
||||
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
|
||||
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) |
|
||||
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki deep dives & shard-spectrum synthesis (`research/260614-*/`) |
|
||||
| Demand | NetKingdom integration asks captured, not yet negotiated |
|
||||
| Spec | Architecture blueprint drafted; UseCaseCatalog 78 UCs from research; PRD/TSD scaffolds |
|
||||
| Spec | Architecture blueprint drafted; UseCaseCatalog 79 UCs from research; PRD/TSD scaffolds |
|
||||
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (16 tasks: T1–T10 federation + T11–T16 adapter contract); `SHARD-WP-0003` active (9 engine dives); `SHARD-WP-0004` active (8 computational-knowledge dives) |
|
||||
|
||||
## In Scope (today)
|
||||
|
||||
15
research/260614-ikiwiki-deep-dive/README.md
Normal file
15
research/260614-ikiwiki-deep-dive/README.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# 260614 — ikiwiki deep dive
|
||||
|
||||
Deep dive on **ikiwiki** (Joey Hess): a **wiki compiler** — it compiles a **VCS-backed**
|
||||
(usually **git**) tree of Markdown source into **static HTML**, supports **distributed**
|
||||
operation (clone/edit/push between wiki instances, change **pings**), and treats web edits as
|
||||
commits to the same repo.
|
||||
|
||||
- `findings.md` — the compiler model, git-distributed federation + pinger, static output,
|
||||
capability profile, INTENT mapping, UC seed (UC-79), architecture notes for SHARD-WP-0002,
|
||||
open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-79 (attach a git-backed **compile-to-static** wiki — git Markdown source
|
||||
is the shard, compiled static HTML is a derived publish/projection; participate in
|
||||
git-distributed clone federation with change-pings). Enriched UC-31/56/37/33. Feeds
|
||||
SHARD-WP-0002 T4 (federation), T6 (publish/projection).
|
||||
153
research/260614-ikiwiki-deep-dive/findings.md
Normal file
153
research/260614-ikiwiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# ikiwiki — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T4 · **Subject:** ikiwiki, Joey Hess's
|
||||
VCS-backed wiki compiler.
|
||||
|
||||
## Why this dive
|
||||
|
||||
The forge-wiki dive (T5) established *git-IS-the-store* for hosted Markdown wikis. ikiwiki
|
||||
takes the same git-canonical source but adds two ideas shard-wiki cares about directly:
|
||||
**compile-to-static** (the wiki is *built*, not served from a DB) and **git-distributed
|
||||
federation** (wiki instances clone, pull, push, and **ping** each other). It is referenced
|
||||
in `research/260608-federation-concepts/`; here we go into the model.
|
||||
|
||||
## 1. The wiki-compiler model
|
||||
|
||||
ikiwiki is fundamentally a **compiler**: input is a directory of **source pages** (Markdown
|
||||
by default; also other formats) held in a **version-control repo**; output is a tree of
|
||||
**static HTML**. A rebuild is triggered by a **VCS post-commit/post-update hook**, so:
|
||||
|
||||
- **The VCS repo is canonical**; the HTML is *derived build output* (regenerable, disposable).
|
||||
- **Web edits are commits.** The CGI edit interface writes the change *into the repo* (a
|
||||
commit) and triggers a rebuild — so browser edits and `git push` edits converge on one
|
||||
history. (Same convergence as forge wikis, but here the canonical store is *your* git repo,
|
||||
not a forge's.)
|
||||
- **VCS-agnostic**: git is usual, but svn/bzr/mercurial/darcs are supported via a VCS
|
||||
plugin layer — an early "pluggable backend behind a stable interface" (adapter-contract
|
||||
echo).
|
||||
- **Plugins** (Perl) provide directives, feeds, auth (`openid`), and the federation hooks
|
||||
below.
|
||||
|
||||
## 2. Git-distributed federation + the pinger
|
||||
|
||||
Because the source is an ordinary VCS repo, ikiwiki instances federate the way *git* does:
|
||||
|
||||
- **Clone-and-diverge**: you can `git clone` a wiki, edit offline, and `push`/`pull` between
|
||||
instances — **multiple wiki clones that reconcile via git merge**. A wiki is a branch-space
|
||||
(UC-33).
|
||||
- **`pinger` / `pingee` plugins**: an instance can send an **XML-RPC ping** to another
|
||||
ikiwiki when it changes, prompting the other to **pull and rebuild** — a lightweight
|
||||
**subscribe/notify** primitive over the git-distributed mesh (UC-31).
|
||||
- **`aggregate` plugin**: pulls external **RSS/Atom feeds** into the wiki as pages — an
|
||||
inbound projection of remote content.
|
||||
|
||||
So ikiwiki is *federation by git plus a ping* — distinct from fedwiki's fork/journal
|
||||
(UC-72) and from Wikibase's query-time `SERVICE` (UC-74): a **third federation flavor**,
|
||||
*VCS-replication federation with change notification*.
|
||||
|
||||
## 3. Static output as a publish/projection target
|
||||
|
||||
The compiled static HTML is a **read-only, regenerable projection** of the source:
|
||||
|
||||
- It is a natural **outbound publish target** (UC-56): render the union (or a shard) to a
|
||||
static site for hosting/backup, no server needed.
|
||||
- It is also the **read-only backup** end (UC-37): a static snapshot that survives the engine.
|
||||
|
||||
The key shard-wiki framing: **source (git Markdown) is the attachable shard; static HTML is a
|
||||
derived projection** — never confuse the build output for the canonical content.
|
||||
|
||||
## 4. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | ikiwiki |
|
||||
|--------------------------------|---------|
|
||||
| Attachment mode | **file-store** (VCS-backed git Markdown source) |
|
||||
| Addressing granularity | page = source file; path = identity |
|
||||
| Content identity | path/filename (placement-bound) |
|
||||
| Structure | directory tree of Markdown source + directives |
|
||||
| History | **native VCS (git) history** |
|
||||
| Merge model | **git** (clone/pull/push/merge across instances) |
|
||||
| Native query | none; directives + plugins compute derived pages at build |
|
||||
| Translation | Markdown source → static HTML (build-time render) |
|
||||
| Write granularity | **file (page)** per commit |
|
||||
| Operational envelope | a compiler + VCS hook; static hosting for output |
|
||||
| Access grant | VCS/file perms; `openid` for web edits |
|
||||
| Content opacity | transparent Markdown |
|
||||
| Provenance | git author/timestamp; aggregated feeds carry source |
|
||||
| Federation | **git replication + XML-RPC ping**; RSS aggregation |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Git-canonical Markdown source** = the home case (shared with forge wikis UC-76/40):
|
||||
attach the source repo, adopt its git log as the journal, write by commit.
|
||||
- **Coordination layer is git** (INTENT): ikiwiki's whole federation *is* git replication +
|
||||
a ping — the most literal realization of "Git-addressable coordination layer."
|
||||
- **Projection vs canonical**: compile-to-static cleanly separates **canonical source** from
|
||||
**derived output** — exactly shard-wiki's projection principle (static HTML = a lazy/cache
|
||||
projection that is regenerable, never the source of truth).
|
||||
- **Graceful degradation / publish**: static output is the trivial read-only backup and
|
||||
outbound publish target (UC-37/UC-56).
|
||||
- **Subscribe/notify mechanism, not policy**: the pinger is a *mechanism* (notify a peer to
|
||||
pull); *which* peers, when, and conflict policy stay configurable.
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- ikiwiki is mostly a **reinforcement** of git-canonical-Markdown (UC-76) — its *new*
|
||||
contributions are (a) **compile-to-static** as a distinct projection/publish direction and
|
||||
(b) the **git-distributed-clone + ping** federation flavor. The static output is *not* a
|
||||
shard to attach (it's derived); attach the **source repo**.
|
||||
- VCS-agnostic backend layer is an interesting adapter-contract echo, but for shard-wiki the
|
||||
git case dominates.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Source-repo-as-shard, static-output-as-derived-projection** — never attach the build
|
||||
output as canonical (UC-79; relates UC-37/UC-56).
|
||||
2. **Git-replication + change-ping** as a named federation flavor beside fork/journal
|
||||
(fedwiki) and query-SERVICE (Wikibase) — a **subscribe/notify over a git mesh** (UC-31/UC-33).
|
||||
3. **Inbound feed aggregation** (RSS/Atom → pages) as an inbound projection pattern.
|
||||
|
||||
## 6. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-79 | Attach a git-backed **compile-to-static** wiki — the **git Markdown source is the shard**, compiled **static HTML is a derived publish/projection**; participate in **git-distributed clone federation** with change-**pings** | **new** |
|
||||
| — | XML-RPC pinger = subscribe/notify over a git mesh | enrich **UC-31** |
|
||||
| — | render union/shard to a static site (publish) | enrich **UC-56** |
|
||||
| — | static HTML as read-only regenerable backup | enrich **UC-37** |
|
||||
| — | wiki as a git branch-space; clones reconcile via merge | enrich **UC-33** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T4 (federation):** record **git-replication + change-ping** as a federation flavor — peers
|
||||
hold git clones, reconcile via merge, and notify with a ping to pull/rebuild. Distinct from
|
||||
fedwiki fork/journal and Wikibase `SERVICE`; complements the "Git-addressable coordination
|
||||
layer" mandate most literally.
|
||||
- **T6 (publish/projection):** **compile-to-static** is the canonical *outbound* projection —
|
||||
source (git) is canonical, static HTML is a regenerable derived view (publish/backup
|
||||
target). Reinforces projection-vs-canonical separation.
|
||||
- **Attach binding (T14):** attach the **source repo** (git Markdown), not the build output;
|
||||
shares the forge-wiki / `wiki/`-subdir git+Markdown adapter (parameterized by repo/path).
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Is the **pinger** (notify-peer-to-pull) modeled as shard-wiki's own subscribe/notify
|
||||
primitive, or only recognized when bridging two ikiwiki instances?
|
||||
2. Does shard-wiki ever *drive* a compile-to-static **publish** of the union (act as the
|
||||
ikiwiki-like compiler), or only attach existing ikiwiki source repos? (UC-56 scope.)
|
||||
3. Is **feed aggregation** (RSS→pages) an inbound projection mode shard-wiki offers generally
|
||||
(a feed-shard), or an ikiwiki-internal feature?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- ikiwiki.info — *ikiwiki* overview, *rcs* (VCS backends), *plugins* (`pinger`/`pingee`,
|
||||
`aggregate`, `openid`), *setup* / post-commit hooks
|
||||
- prior: `research/260608-federation-concepts/` (ikiwiki reference);
|
||||
`research/260614-forge-wikis-deep-dive/` (git-canonical Markdown contrast)
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UC **UC-79** carries the marker **⊟** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-31, UC-56, UC-37, UC-33. Architecture cross-refs:
|
||||
SHARD-WP-0002 T4 (git-replication+ping federation), T6 (compile-to-static publish), T14.
|
||||
@@ -31,4 +31,5 @@ when multiple files or sources are involved. Findings here inform `spec/` and
|
||||
| 2026-06-14 | `260614-federated-wiki-deep-dive/` | Federated Wiki — fork-with-provenance, per-page semantic-action journal (story=replay), neighborhood/roster + chorus; prior art for our coordination journal / overlay / union pillars; UC-70/71/72 |
|
||||
| 2026-06-14 | `260614-wikibase-deep-dive/` | Wikibase/Wikidata — typed entity-statement knowledge graph (claim+qualifiers+refs+rank), SPARQL/RDF + federated SERVICE, opaque stable IDs, statement-level provenance; structure & query far-end; UC-73/74/75 |
|
||||
| 2026-06-14 | `260614-forge-wikis-deep-dive/` | Gitea · GitLab · GitHub wikis — a wiki is a separate `.wiki.git` of Markdown; git-clone universal, wiki API capability-varying (GitHub git-only); git IS the store (resolves UC-68 race); the home case; UC-76/77 |
|
||||
| 2026-06-14 | `260614-tiddlywiki-deep-dive/` | TiddlyWiki — entire wiki (content + app) in one self-contained HTML file = whole-file write-granularity extreme; Node `.tid` file-per-tiddler substrate (git-diffable); tiddler/field records, filter-expression query; UC-78 |
|
||||
| 2026-06-14 | `260614-tiddlywiki-deep-dive/` | TiddlyWiki — entire wiki (content + app) in one self-contained HTML file = whole-file write-granularity extreme; Node `.tid` file-per-tiddler substrate (git-diffable); tiddler/field records, filter-expression query; UC-78 |
|
||||
| 2026-06-14 | `260614-ikiwiki-deep-dive/` | ikiwiki — wiki compiler: git-canonical Markdown source → static HTML (derived publish/projection); git-distributed clone federation + XML-RPC pinger (third federation flavor); UC-79 |
|
||||
@@ -15,7 +15,8 @@ Promoted from `research/260608-c2-wiki-origins/`,
|
||||
`research/260614-federated-wiki-deep-dive/`, and
|
||||
`research/260614-wikibase-deep-dive/`, and
|
||||
`research/260614-forge-wikis-deep-dive/`, and
|
||||
`research/260614-tiddlywiki-deep-dive/`.
|
||||
`research/260614-tiddlywiki-deep-dive/`, and
|
||||
`research/260614-ikiwiki-deep-dive/`.
|
||||
See InfoTechPrimers on coulomb.social for use-case catalog conventions.
|
||||
|
||||
## Conventions
|
||||
@@ -197,7 +198,11 @@ concrete push transport: an engine event bus (`ObservationManager`
|
||||
eventual consistency and the rate limit (`research/260614-notion-deep-dive/findings.md`
|
||||
§4). **Joplin** is poll-based and turns concurrent edits into **conflict notes**
|
||||
(keep-both, not silent overwrite) — conflict-as-data, links UC-07
|
||||
(`research/260614-joplin-deep-dive/findings.md` §2).
|
||||
(`research/260614-joplin-deep-dive/findings.md` §2). The **ikiwiki** dive details the
|
||||
canonical pinger: an instance sends an **XML-RPC ping** to a peer when it changes, prompting
|
||||
the peer to **pull and rebuild** — a subscribe/notify *mechanism* over a **git-distributed
|
||||
mesh** (UC-79, `research/260614-ikiwiki-deep-dive/findings.md` §2), the federation flavor
|
||||
beside fedwiki fork/journal (UC-72) and Wikibase `SERVICE` (UC-74).
|
||||
**Priority:** Later
|
||||
|
||||
### UC-32 — Transclude remote span with live freshness
|
||||
@@ -1075,6 +1080,23 @@ ignore the embedded engine). Feeds SHARD-WP-0002 T11 (whole-file tier), T14 (sin
|
||||
`.tid` binding; cf. UC-43/UC-62).
|
||||
**Priority:** Later
|
||||
|
||||
### UC-79 — Attach a git-backed compile-to-static wiki (source is the shard)
|
||||
|
||||
**Actor:** Orchestrator / adapter
|
||||
**Goal:** Attach a **compile-to-static** wiki (ikiwiki) where the **git Markdown source is
|
||||
the shard** and the **compiled static HTML is a derived publish/projection**; optionally
|
||||
participate in **git-distributed clone federation** with change-**pings**.
|
||||
**Source:** wikiengines, federation, intent
|
||||
**Notes:** ikiwiki compiles a VCS-backed (git) Markdown tree to static HTML; web edits are
|
||||
commits, so git is canonical and HTML is regenerable build output
|
||||
(`research/260614-ikiwiki-deep-dive/findings.md` §1, §3). **Attach the source repo, never
|
||||
the build output** (the static site is a projection, UC-37/UC-56). Its federation is **git
|
||||
replication + an XML-RPC pinger** (notify a peer to pull/rebuild, UC-31) — a third
|
||||
federation flavor beside fedwiki fork/journal (UC-72) and Wikibase `SERVICE` (UC-74). Shares
|
||||
the git+Markdown adapter with forge wikis (UC-76). Feeds SHARD-WP-0002 T4 (federation), T6
|
||||
(publish/projection).
|
||||
**Priority:** Later
|
||||
|
||||
---
|
||||
|
||||
## B. Knowledge work and collaboration
|
||||
@@ -1350,6 +1372,7 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD.
|
||||
| UC-76 | | | | ⎇ | ✓ |
|
||||
| UC-77 | | | | ⎇ | ✓ |
|
||||
| UC-78 | | | | ⊡ | ✓ |
|
||||
| UC-79 | | | ✓ | ⊟ | ✓ |
|
||||
| UC-08 | ✓ | | |
|
||||
| UC-09 | ✓ | | |
|
||||
| UC-10 | ✓ | | |
|
||||
@@ -1881,6 +1904,33 @@ are a real native-query tier (UC-52). **Boundary recorded:** treat the single HT
|
||||
Architecture logged for `SHARD-WP-0002` (T11/T14): whole-file as the coarsest write tier,
|
||||
single-file-container vs `.tid`-dir dual binding.
|
||||
|
||||
### ikiwiki mapping
|
||||
|
||||
(⊟ UC-79 lineage = the **ikiwiki deep dive**, `research/260614-ikiwiki-deep-dive/findings.md`;
|
||||
marked in both federation and wikiengines columns.)
|
||||
|
||||
| ikiwiki mechanism (findings §) | Catalog UC |
|
||||
|--------------------------------|------------|
|
||||
| VCS(git)-backed Markdown source compiled to static HTML; web edits = commits (§1, §3) | UC-79 (new) |
|
||||
| `pinger`/`pingee` XML-RPC = notify a peer to pull/rebuild (§2) | UC-31 (enriched) |
|
||||
| Compile union/shard to a static site = outbound publish (§3) | UC-56 (enriched) |
|
||||
| Static HTML = read-only regenerable backup (§3) | UC-37 (enriched) |
|
||||
| Clone/pull/push between instances; wiki = git branch-space (§2) | UC-33 (enriched) |
|
||||
| `aggregate` (RSS/Atom → pages) = inbound feed projection (§2) | links UC-03 |
|
||||
| VCS-agnostic backend behind a plugin layer (§1) | links UC-43 |
|
||||
|
||||
Note: ikiwiki is a **wiki compiler** — git-canonical Markdown **source** built into **static
|
||||
HTML** — so it mostly **reinforces** the git-IS-store home case (UC-76/40), but adds two
|
||||
genuinely new things: **compile-to-static** as a clean *canonical-source vs derived-output*
|
||||
separation (attach the **source repo**, treat static HTML as a regenerable
|
||||
projection/publish target, UC-37/UC-56), and a **third federation flavor** — **git
|
||||
replication + an XML-RPC change-ping** (peers hold clones, merge to reconcile, notify to
|
||||
pull) beside fedwiki fork/journal (UC-72) and Wikibase `SERVICE` (UC-74). **Boundary
|
||||
recorded:** never attach the build output as canonical; the pinger is a subscribe/notify
|
||||
*mechanism* (peers/timing/conflict stay policy). Architecture logged for `SHARD-WP-0002`
|
||||
(T4/T6/T14): git-replication+ping federation, compile-to-static publish, source-repo attach
|
||||
sharing the git+Markdown adapter.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
@@ -1951,5 +2001,9 @@ single-file-container vs `.tid`-dir dual binding.
|
||||
26. For a **single-file wiki** (UC-78), does shard-wiki support per-page overlays by
|
||||
buffering and re-serializing the whole file, or require the Node `.tid` substrate for
|
||||
write-through and treat single-file as read/projection/backup only? (TiddlyWiki dive §9.)
|
||||
27. For a **compile-to-static** wiki (UC-79), does shard-wiki ever *drive* the build to
|
||||
**publish the union** as a static site (act as the compiler), or only attach existing
|
||||
source repos — and is the **pinger** modeled as shard-wiki's own subscribe/notify
|
||||
primitive or only recognized when bridging two ikiwiki instances? (ikiwiki dive §8.)
|
||||
23. How does shard-wiki **honor/surface a shard's path-based access rules** (UC-06) in a
|
||||
projection without re-implementing its ACL engine? (Wiki.js dive §9.)
|
||||
@@ -100,7 +100,7 @@ model. The portability/granularity extreme. Enrich UC-35 (write granularity), UC
|
||||
|
||||
```task
|
||||
id: SHARD-WP-0003-T4
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "5748d246-66a8-4cf2-9b22-427cc4507b63"
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user