From ffd5459b3e480838c0ba5050e485f6f72496396d Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 13 Jun 2026 23:20:11 +0200 Subject: [PATCH] research: TWiki deep dive (impl, plugin API, ecosystem); UC-40/41 Deep dive into TWiki as the file-based Perl counterpoint to XWiki: flat-file + RCS store (data//.txt), Webs/Topics, TWiki Forms storing fields as %META% in the topic text, TWikiML/variables, TWiki::Func API; the plugin handler callback surface (initPlugin, commonTagsHandler, before/afterSaveHandler, afterRenameHandler, REST handlers) and package types (Plugin/Skin/AddOn/Contrib); per-topic ALLOW/DENY access control (origin of yawex's model); Foswiki fork. Catalog (now 41 UCs): - UC-40 attach a file-backed engine's on-disk store directly (dual-path attach) - UC-41 import an engine's native file history (RCS .txt,v) into the journal - enrich UC-06 (TWiki per-topic ACL lineage), UC-34 (file-embedded %META%), UC-36 (RCS-import vs DB-supplement), UC-38 (TWiki handlers as adapter host) Co-Authored-By: Claude Opus 4.8 --- research/260613-twiki-deep-dive/README.md | 39 ++++ research/260613-twiki-deep-dive/findings.md | 210 ++++++++++++++++++++ spec/UseCaseCatalog.md | 61 +++++- 3 files changed, 303 insertions(+), 7 deletions(-) create mode 100644 research/260613-twiki-deep-dive/README.md create mode 100644 research/260613-twiki-deep-dive/findings.md diff --git a/research/260613-twiki-deep-dive/README.md b/research/260613-twiki-deep-dive/README.md new file mode 100644 index 0000000..0965046 --- /dev/null +++ b/research/260613-twiki-deep-dive/README.md @@ -0,0 +1,39 @@ +# 260613 — TWiki deep dive (implementation, plugin API, ecosystem) + +Date: 2026-06-13 + +## What this is + +A focused study of **TWiki** — the original "structured enterprise wiki / wiki +application platform" (Peter Thoeny, ~1998), and the **file-based Perl counterpoint** +to XWiki's Java/DB platform. Covers TWiki's **implementation** (Perl + flat-file + +RCS store, Webs/Topics, TWiki Forms, TWikiML/variables), the **plugin API and +handler callbacks** that extend the core, the **extension package types** (Plugin / +Skin / AddOn / Contrib) and repository, and TWiki's **per-topic access control** +(the prior art behind yawex's `AccessControl`). + +Read through shard-wiki's lens, paired with the XWiki dive: TWiki shows the same +"wiki-as-application-platform" goal reached with **text files + RCS** instead of a +database — which changes the adapter story (direct on-disk attach, git-convertible +history). + +Complements: +- `research/260613-xwiki-deep-dive/` — the DB/Java app-platform sibling +- `research/260608-wikiengines-overview/` — the landscape scan +- `research/260608-yawex-prior-art/` — yawex borrowed TWiki's per-topic AccessControl + +Note: most of TWiki's community forked to **Foswiki** in Oct 2008 (from TWiki +4.2.4); findings flag where TWiki/Foswiki differ. + +## Contents + +| Path | Role | +|------|------| +| `findings.md` | Architecture, plugin-handler API, ecosystem, capability profile, INTENT mapping, UC seeds, sources | + +## Status + +Initial deep dive complete. Two new use cases promoted to +`spec/UseCaseCatalog.md` (UC-40, UC-41); UC-06/34/36/38/39 enriched with TWiki +specifics. Adapter-contract implications feed +`workplans/SHARD-WP-0002-federation-architecture.md`. diff --git a/research/260613-twiki-deep-dive/findings.md b/research/260613-twiki-deep-dive/findings.md new file mode 100644 index 0000000..c3b40d1 --- /dev/null +++ b/research/260613-twiki-deep-dive/findings.md @@ -0,0 +1,210 @@ +# Findings — TWiki: implementation, plugin API, ecosystem + +Date: 2026-06-13 · Status: research draft + +Scope: TWiki as prior art for the **file-based** wiki-application-platform — the +counterpoint to XWiki's database/component platform. Two shard-wiki concerns: +(1) attaching a structured, file-backed engine as a shard (including attaching its +**on-disk store directly** vs through its runtime), and (2) how TWiki's plugin +handler API exposes hooks rich enough to host a federation adapter +(INTENT *composable integration*). Sources: twiki.org dev docs (TWikiPlugins, +TWikiForms, TWiki::Func, TWikiAccessControl), Foswiki docs, Wikipedia. + +**Complements:** `research/260613-xwiki-deep-dive/` (DB/Java sibling), +`research/260608-wikiengines-overview/` (landscape), and +`research/260608-yawex-prior-art/` (yawex borrowed TWiki's per-topic AccessControl). + +--- + +## 1. What TWiki is + +A **structured enterprise wiki** and **wiki application platform** (Peter Thoeny, +~1998), written in **Perl 5** (5.10.1+), classically run as CGI. Like XWiki it lets +pages become forms/records/apps — but it reaches that with **flat text files + +RCS**, no database. Most of the community forked to **Foswiki** in October 2008 +(from TWiki 4.2.4; Foswiki 1.0 in January 2009); the plugin API and data model below +are shared lineage, with minor divergences noted. + +## 2. Implementation architecture + +### 2.1 Storage — flat files + RCS (no DB) +- Content lives on disk as `data//.txt`; the directory tree **mirrors + the logical Web/Topic hierarchy**. Attachments live under `pub///`. +- **History is RCS** (`.txt,v` companion files) — the same GNU RCS yawex used. + History is real, per-file, and in an **open, git-convertible format** (unlike + XWiki's DB-internal `xwikircs`). +- Scales to 300k+ topics on a single server (Yahoo) — file store is not a toy. + +### 2.2 Webs and Topics +- **Topic** = page (the unit of content). **Web** = namespace / collection of topics + (can be nested). Webs are the natural shard / root-entity boundary. + +### 2.3 TWiki Forms — structured data, embedded in text +- A **form template** is a topic defining fields as a table (one row per field: + name, type, size, values). A topic can *attach* a form; the field values are + stored as **TWikiMetaData inside the topic `.txt`** (`%META:FIELD{...}%`), shown as + a table on view and edited via fields/radios/checkboxes/lists. +- **Forms + formatted `%SEARCH%`** = TWiki's "database applications" — structured + records queried by the SEARCH variable. This is the XWiki XObject/XClass idea, but + **file-embedded and git-diffable** rather than rows in a DB. + +### 2.4 TWikiML / variables (macros) +- Markup is **TWikiML** (not Markdown). Dynamic behavior comes from **variables** + `%VAR%` / `%VAR{...}%` expanded at render time; `%SEARCH%` is the query engine. +- Custom variables are provided by plugins (see §3). + +### 2.5 TWiki::Func — the official API boundary +- `lib/TWiki/Func.pm` ("TWikiFuncDotPm") documents **all** interfaces available to + plugins, deliberately abstracting the flat-file assumption: `readTopic`, + `saveTopic`, `getTopicList`, `getListOfWebs`, `getWorkArea`, `saveAttachment`, … + Plugins are told *not* to assume flat files and to go through Func. + +## 3. Plugin API — interfaces to extend the core + +TWiki extends "without altering core code" by **registering Perl handlers that the +core calls at defined points** in the request/render/save pipeline. A plugin is +`lib/TWiki/Plugins/.pm` + a documentation topic + an optional `Config.spec`. + +### 3.1 Handler callbacks (the hook surface) +| Phase | Handlers | +|-------|----------| +| Init / users | `initPlugin` (must return 1), `registrationHandler` | +| Tag/variable expansion | `commonTagsHandler`, `beforeCommonTagsHandler`, `afterCommonTagsHandler` | +| Rendering | `preRenderingHandler`, `postRenderingHandler` (`start/endRenderingHandler` deprecated) | +| Edit lifecycle | `beforeEditHandler`, `afterEditHandler` | +| Save lifecycle | `beforeSaveHandler`, `afterSaveHandler`, `beforeMergeHandler` | +| Attachments | `beforeAttachmentSaveHandler`, `afterAttachmentSaveHandler` | +| Topic mgmt | `afterRenameHandler`, `completePageHandler` | + +These are the TWiki analogue of XWiki's `ObservationManager` events — but +**synchronous pipeline hooks** rather than an async event bus. The save/rename/ +attachment handlers are exactly the interception points a federation adapter needs. + +### 3.2 Other extension mechanisms +- **Custom variables/macros** — registered via `commonTagsHandler` (Foswiki adds + `registerTagHandler`); convention `%PLUGINNAME_SETTING%`. +- **REST handlers** — a plugin registers REST handlers invoked via the `rest` + script, for transactions outside the standard `view/edit/save` scripts. This is the + natural remote adapter transport. +- **`Config.spec`** — declares configuration items (BOOLEAN, STRING, SELECT, PATH, + PERL) surfaced in the `configure` web UI. +- **Work area** — `getWorkArea()` gives a plugin a persistent, non-web-accessible + data directory. + +### 3.3 Extension package types +| Type | Role | +|------|------| +| **Plugin** | handler-based behavior, no core change (`lib/TWiki/Plugins/.pm`) | +| **Skin** | visual appearance only; topic content unchanged (e.g. PrintSkin) | +| **AddOn** | a script in `bin/` callable from a topic | +| **Contrib** | shared library code, or **alternative implementations of core sections** (e.g. user management) when something "can't be a plugin because it needs close core access" | + +## 4. Access control (origin of yawex's model) + +TWiki authorizes via **preference settings in topics**: `ALLOWWEBVIEW` / +`ALLOWWEBCHANGE` at web level and `ALLOWTOPICVIEW` / `ALLOWTOPICCHANGE` / +`ALLOWTOPICRENAME` (and `DENY*`) at topic level, with **VIEW / CHANGE / RENAME +grantable separately**. Topic-level settings apply only to that topic. This +per-topic ACL is the **direct prior art** behind yawex's `AccessControl` +(denied/view/form/edit/admin) and a concrete reference for shard-wiki's optional +**per-page ACL at L4** (`spec/ArchitectureBlueprint.md` §5). + +## 5. TWiki as a shard — capability profile + +| Capability | TWiki | Note | +|------------|-------|------| +| Read | ✓ | TWiki::Func / `view` script / `rest`; or read `data/*.txt` directly | +| Write | ✓ | saveTopic / `save` / `rest`; **per-topic** granularity | +| Structured payload | ✓ | TWiki Forms → `%META:FIELD%` **in the text file** (git-diffable) | +| Version/diff | ✓ | RCS `.txt,v` — **open format, git-convertible** (cf. XWiki DB) | +| Merge | partial | `beforeMergeHandler`; no git 3-way | +| Change hooks | ✓ | synchronous save/rename/attachment handlers | +| Auth model | per-web/topic ACL | authorize through shard-wiki core; engine ACL is advisory | +| Direct-store attach | ✓✓ | `data//.txt` + RCS is a folder shard on its own | +| Federation hooks | ✓ | plugin handlers + REST handlers can host an adapter (UC-38) | + +## 6. Mapping to shard-wiki INTENT (compare, do not equate) + +### 6.1 Reinforcements +| Observation | INTENT principle | +|-------------|------------------| +| Forms store data as `%META%` **in the topic file** | structured pages can be **git-diffable**, strengthening UC-34/39 vs XWiki's DB | +| RCS `.txt,v` history | **git-addressable coordination** — here history is *convertible*, not just supplementable (UC-36 → UC-41) | +| `data//.txt` mirrors logic | an engine's **on-disk store is itself attachable as a folder shard** (UC-40) | +| Plugin save/rename/REST handlers | **composable integration** — TWiki can host an adapter too, generalizing UC-38 beyond XWiki | +| Per-topic `ALLOW/DENY` ACL | prior art for **per-page ACL at L4**; lineage TWiki→yawex (UC-06) | +| Webs as namespaces / shards | namespace navigation (UC-22); webs as roots | + +### 6.2 Deliberate divergences (design bugs if conflated) +| TWiki assumption | shard-wiki correction | +|------------------|----------------------| +| TWikiML + variable expansion is the page | **Markdown-first**; TWikiML render stays out of core | +| ACL lives in topic preference settings | **authorize in core**; topic ALLOW/DENY is advisory provenance | +| History = RCS files next to content | **coordination journal** is the space-level git layer (may *import* RCS) | +| Apps = Forms + `%SEARCH%` rendered by TWiki | represent records **without** depending on TWiki to render/search them | +| Attach = run TWiki and call it | shard-wiki may attach the **bare data dir** with TWiki offline (UC-40) | + +### 6.3 What TWiki teaches that shard-wiki should not lose +1. **File-backed engines are the easy, high-value case** — their store is already a + folder of text + open-format history; the adapter can often skip the runtime. +2. **Structured data in text beats structured data in a DB** for federation — + diffable, portable, journal-friendly. Prefer file-embedded metadata where offered. +3. **One backend, two attachment paths** (runtime API vs on-disk store) is a real + capability/consistency trade-off the adapter contract must express. + +## 7. Use-case seeds → catalog (promoted 2026-06-13) + +| Seed | Catalog UC | Disposition | +|------|------------|-------------| +| Attach a live engine's **on-disk store directly** (vs its runtime API) | **UC-40** (new) | dual-path attach; fidelity/consistency trade-off | +| **Import** an engine's native file-based history (RCS) into the journal | **UC-41** (new) | history *migration* with fidelity, vs UC-36 *supplementation* | +| Engine hosts adapter via its plugin/handler API | UC-38 | enriched: TWiki save/rename/REST handlers generalize it beyond XWiki | +| Structured page carries typed data | UC-34 / UC-39 | enriched: TWiki Forms `%META%`, git-diffable | +| Internal-history engine | UC-36 | enriched: contrast DB (supplement) vs RCS (import, UC-41) | +| Authenticated team wiki / per-page ACL | UC-06 | enriched: TWiki per-topic ACL is the origin of yawex's model | + +## 8. Open questions (for spec / workplans) + +1. **Dual-path adapters** — when does shard-wiki attach an engine's on-disk store + directly vs go through its API? Consistency risk of reading live files under a + running engine; capability-gate it (UC-40)? +2. **History import fidelity** — can RCS `.txt,v` be converted to git commits + preserving author/timestamp, and is that authoritative or a one-time backfill + feeding the coordination journal (UC-41, cf. UC-36 Q3)? +3. **File-embedded metadata mapping** — map TWiki `%META:FIELD%` (and XWiki XObjects) + onto one structured-metadata page-model representation (shared with xwiki §8 Q1). +4. **TWiki vs Foswiki** — target both via one adapter (shared API) or treat as + distinct shard types? +5. **ACL provenance** — surface a topic's `ALLOW/DENY` as read-only provenance even + though authorization is decided in core? + +## 9. Sources + +| Source | URL | +|--------|-----| +| TWiki — TWikiPlugins (handlers, package types) | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins | +| TWiki — Developing Plugins / EmptyPlugin | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins | +| TWiki — TWiki::Func module | https://twiki.org/cgi-bin/view/TWiki/TWikiFuncModule | +| TWiki — TWiki Forms | https://twiki.org/cgi-bin/view/TWiki/TWikiForms | +| TWiki — TWiki Access Control | https://twiki.org/cgi-bin/view/TWiki/TWikiAccessControl | +| TWiki — System Requirements (Perl/RCS) | https://twiki.org/cgi-bin/view/TWiki/TWikiSystemRequirements | +| TWiki — Source Code | https://twiki.org/cgi-bin/view/TWiki/SourceCode | +| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork | +| Wikipedia — TWiki | https://en.wikipedia.org/wiki/TWiki | +| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki | +| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` | +| shard-wiki — yawex prior art | `research/260608-yawex-prior-art/findings.md` | + +--- + +## 10. Traceability + +| This document section | Informs (future) | +|-----------------------|------------------| +| §2 architecture | adapter design for file-backed engines | +| §3 plugin handlers | UC-38 engine-side adapter; composable-integration API shape | +| §4 access control | per-page ACL at L4 (`spec/ArchitectureBlueprint.md`), UC-06 | +| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) | +| §6 INTENT mapping | architecture-blueprint guardrails | +| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-40, UC-41; UC-06/34/36/38/39 enrichment) | +| §8 open questions | spec — dual-path adapters, history import, structured-metadata model | diff --git a/spec/UseCaseCatalog.md b/spec/UseCaseCatalog.md index cc96811..6559c48 100644 --- a/spec/UseCaseCatalog.md +++ b/spec/UseCaseCatalog.md @@ -4,7 +4,8 @@ Status: **draft** · Date: 2026-06-08 · Updated: 2026-06-13 Promoted from `research/260608-c2-wiki-origins/`, `research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`, -`research/260608-wikiengines-overview/`, and `research/260613-xwiki-deep-dive/`. +`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`, and +`research/260613-twiki-deep-dive/`. See InfoTechPrimers on coulomb.social for use-case catalog conventions. ## Conventions @@ -81,7 +82,9 @@ external provider. **Source:** intent, yawex **Notes:** yawex htpasswd + per-topic `AccessControl` is prior art only; shard-wiki delegates authentication, owns authorization -(`spec/ArchitectureBlueprint.md`). +(`spec/ArchitectureBlueprint.md`). yawex's model traces to **TWiki's per-topic +`ALLOW/DENY TOPICVIEW/CHANGE/RENAME`** — the origin reference for shard-wiki's +optional per-page ACL at L4 (`research/260613-twiki-deep-dive/findings.md` §4). **Priority:** Later ### UC-07 — Detect and reconcile cross-shard divergence @@ -192,7 +195,10 @@ needs a passthrough / sidecar-metadata / provenance escape hatch. Page-model question deferred to `SHARD-WP-0002` (findings §6 Q1). Distinct from UC-02 (assumes Markdown-shaped pages). XWiki is the concrete exemplar: pages carry typed XObjects against an XClass schema (`research/260613-xwiki-deep-dive/findings.md` -§2.3); UC-39 covers the extreme where the page is *only* structure. +§2.3); UC-39 covers the extreme where the page is *only* structure. TWiki shows the +**git-friendly** variant: TWiki Forms store fields as `%META:FIELD%` *inside the +topic text file*, so structure is diffable rather than locked in a DB +(`research/260613-twiki-deep-dive/findings.md` §2.3). **Priority:** Later ### UC-35 — Attach a shard with coarse write granularity @@ -219,7 +225,9 @@ The **coordination journal** supplies the Git-addressable layer (INTENT). Open: is the journal authoritative or a mirror reconciled back to the engine (findings §6 Q3)? Complements UC-07 divergence and UC-24 provenance. XWiki's internal RCS table (`xwikircs`) is a concrete instance -(`research/260613-xwiki-deep-dive/findings.md` §2.4). +(`research/260613-xwiki-deep-dive/findings.md` §2.4). This UC is *supplementation* +(the engine's past is DB-locked); where history is already in an open file format +(TWiki RCS `.txt,v`) it can instead be **imported** — see UC-41. **Priority:** Later ### UC-37 — Attach a static engine export as a read-only backup shard @@ -248,7 +256,10 @@ component model (`@Component` + role hint) can replace auth/storage/rights, its REST API and `ObservationManager` give transport + change events, and UIX adds surfacing — enough to host a high-fidelity adapter (`research/260613-xwiki-deep-dive/findings.md` §3). Trade-off vs an external REST -adapter (needs deploy access, higher fidelity) is open (findings §8 Q3). +adapter (needs deploy access, higher fidelity) is open (findings §8 Q3). +Generalizes beyond XWiki: TWiki's plugin handler API (`beforeSaveHandler`, +`afterRenameHandler`, REST handlers) is an equivalent host surface +(`research/260613-twiki-deep-dive/findings.md` §3). **Priority:** Later ### UC-39 — Attach a wiki-as-application-platform shard (pages as typed records) @@ -264,6 +275,36 @@ or can a page be purely typed data (`research/260613-xwiki-deep-dive/findings.md §8 Q2)? Affects union views, diff, and search. Deferred to `SHARD-WP-0002`. **Priority:** Later +### UC-40 — Attach a file-backed engine's on-disk store directly + +**Actor:** Maintainer +**Goal:** Attach a live engine whose content is plain files on disk — TWiki +`data//.txt`, DokuWiki pages, a Gollum/ikiwiki repo — by reading its +**backing store directly** as a folder shard, instead of (or alongside) going +through its runtime API. +**Source:** wikiengines, intent +**Notes:** One backend, **two attachment paths** with different fidelity/consistency: +the on-disk store (offline-capable, git-native for repo-backed engines, but risks +reading inconsistent state under a running engine) vs the runtime API (consistent, +respects engine logic, but requires the engine up). Capability profile must express +the choice (`research/260613-twiki-deep-dive/findings.md` §5–§6, §8 Q1). Distinct +from UC-37 (no live origin) and broader than UC-02's generic attach. +**Priority:** Later + +### UC-41 — Import an engine's native file history into the coordination journal + +**Actor:** Maintainer +**Goal:** When an engine already keeps history in an open file format (TWiki/yawex +RCS `.txt,v`), **import that history** into the Git-backed coordination journal +preserving authorship and timestamps — not just start a fresh journal going forward. +**Source:** wikiengines, intent +**Notes:** History *migration with fidelity*, distinct from UC-36 *supplementation* +(where the engine's past is locked in a DB and the journal can only begin now). Open: +is the imported history authoritative or a one-time backfill +(`research/260613-twiki-deep-dive/findings.md` §8 Q2)? Strengthens recoverability +(INTENT *History as the safety net*). +**Priority:** Later + --- ## B. Knowledge work and collaboration @@ -482,6 +523,8 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD. | UC-37 | | | | ✓ | ✓ | | UC-38 | | | | ✓ | ✓ | | UC-39 | | | | ✓ | ✓ | +| UC-40 | | | | ✓ | ✓ | +| UC-41 | | | | ✓ | ✓ | | UC-08 | ✓ | | | | UC-09 | ✓ | | | | UC-10 | ✓ | | | @@ -553,14 +596,18 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD. | Static exports / read-only archives (MediaWiki XML dump, archived C2, HTML mirror) | UC-37 | | Engine hosts adapter via native extension API (XWiki components/REST/UIX) | UC-38 | | Wiki-as-application-platform, bodiless typed pages (XWiki AppWithinMinutes/XObjects) | UC-39 | +| Attach a file-backed engine's on-disk store directly (TWiki data dir, DokuWiki) | UC-40 | +| Import an engine's native file history (TWiki RCS `.txt,v`) into the journal | UC-41 | Note: the landscape scan mostly **reinforced** existing UCs and the L0→L4 ladder rather than adding scenarios — its primary yield is adapter-contract constraints (`research/260608-wikiengines-overview/findings.md` §3, §5), tracked in `workplans/SHARD-WP-0002-federation-architecture.md`. UC-34–UC-37 are the attachment scenarios it surfaced. UC-38–UC-39 come from the **XWiki deep dive** -(`research/260613-xwiki-deep-dive/findings.md` §7), which also enriched UC-31 -(event-driven sync), UC-34 (XObject model), and UC-36 (`xwikircs` history). +(`research/260613-xwiki-deep-dive/findings.md` §7); UC-40–UC-41 from the **TWiki +deep dive** (`research/260613-twiki-deep-dive/findings.md` §7), which also enriched +UC-06 (TWiki per-topic ACL → yawex), UC-34 (file-embedded `%META%`), UC-36 (RCS vs +DB history), and UC-38 (TWiki plugin handlers as an adapter host). ---