shard-wiki/research/260613-twiki-deep-dive/findings.md

Findings — TWiki: implementation, plugin API, ecosystem

Date: 2026-06-13 · Status: research draft

Scope: TWiki as prior art for the file-based wiki-application-platform — the counterpoint to XWiki's database/component platform. Two shard-wiki concerns: (1) attaching a structured, file-backed engine as a shard (including attaching its on-disk store directly vs through its runtime), and (2) how TWiki's plugin handler API exposes hooks rich enough to host a federation adapter (INTENT composable integration). Sources: twiki.org dev docs (TWikiPlugins, TWikiForms, TWiki::Func, TWikiAccessControl), Foswiki docs, Wikipedia.

Complements: research/260613-xwiki-deep-dive/ (DB/Java sibling), research/260608-wikiengines-overview/ (landscape), and research/260608-yawex-prior-art/ (yawex borrowed TWiki's per-topic AccessControl).

---

1. What TWiki is

A structured enterprise wiki and wiki application platform (Peter Thoeny, ~1998), written in Perl 5 (5.10.1+), classically run as CGI. Like XWiki it lets pages become forms/records/apps — but it reaches that with flat text files + RCS, no database. Most of the community forked to Foswiki in October 2008 (from TWiki 4.2.4; Foswiki 1.0 in January 2009); the plugin API and data model below are shared lineage, with minor divergences noted.

2. Implementation architecture

2.1 Storage — flat files + RCS (no DB)

• Content lives on disk as data/<Web>/<Topic>.txt; the directory tree mirrors the logical Web/Topic hierarchy. Attachments live under pub/<Web>/<Topic>/.
• History is RCS (<Topic>.txt,v companion files) — the same GNU RCS yawex used. History is real, per-file, and in an open, git-convertible format (unlike XWiki's DB-internal xwikircs).
• Scales to 300k+ topics on a single server (Yahoo) — file store is not a toy.

2.2 Webs and Topics

• Topic = page (the unit of content). Web = namespace / collection of topics (can be nested). Webs are the natural shard / root-entity boundary.

2.3 TWiki Forms — structured data, embedded in text

• A form template is a topic defining fields as a table (one row per field: name, type, size, values). A topic can attach a form; the field values are stored as TWikiMetaData inside the topic .txt (%META:FIELD{...}%), shown as a table on view and edited via fields/radios/checkboxes/lists.
• Forms + formatted %SEARCH% = TWiki's "database applications" — structured records queried by the SEARCH variable. This is the XWiki XObject/XClass idea, but file-embedded and git-diffable rather than rows in a DB.

2.4 TWikiML / variables (macros)

• Markup is TWikiML (not Markdown). Dynamic behavior comes from variables %VAR% / %VAR{...}% expanded at render time; %SEARCH% is the query engine.
• Custom variables are provided by plugins (see §3).

2.5 TWiki::Func — the official API boundary

• lib/TWiki/Func.pm ("TWikiFuncDotPm") documents all interfaces available to plugins, deliberately abstracting the flat-file assumption: readTopic, saveTopic, getTopicList, getListOfWebs, getWorkArea, saveAttachment, … Plugins are told not to assume flat files and to go through Func.

3. Plugin API — interfaces to extend the core

TWiki extends "without altering core code" by registering Perl handlers that the core calls at defined points in the request/render/save pipeline. A plugin is lib/TWiki/Plugins/<Name>.pm + a documentation topic + an optional Config.spec.

3.1 Handler callbacks (the hook surface)

Phase	Handlers
Init / users	initPlugin (must return 1), registrationHandler
Tag/variable expansion	commonTagsHandler, beforeCommonTagsHandler, afterCommonTagsHandler
Rendering	preRenderingHandler, postRenderingHandler (start/endRenderingHandler deprecated)
Edit lifecycle	beforeEditHandler, afterEditHandler
Save lifecycle	beforeSaveHandler, afterSaveHandler, beforeMergeHandler
Attachments	beforeAttachmentSaveHandler, afterAttachmentSaveHandler
Topic mgmt	afterRenameHandler, completePageHandler

These are the TWiki analogue of XWiki's ObservationManager events — but synchronous pipeline hooks rather than an async event bus. The save/rename/ attachment handlers are exactly the interception points a federation adapter needs.

3.2 Other extension mechanisms

• Custom variables/macros — registered via commonTagsHandler (Foswiki adds registerTagHandler); convention %PLUGINNAME_SETTING%.
• REST handlers — a plugin registers REST handlers invoked via the rest script, for transactions outside the standard view/edit/save scripts. This is the natural remote adapter transport.
• Config.spec — declares configuration items (BOOLEAN, STRING, SELECT, PATH, PERL) surfaced in the configure web UI.
• Work area — getWorkArea() gives a plugin a persistent, non-web-accessible data directory.

3.3 Extension package types

Type	Role
Plugin	handler-based behavior, no core change (lib/TWiki/Plugins/<Name>.pm)
Skin	visual appearance only; topic content unchanged (e.g. PrintSkin)
AddOn	a script in bin/ callable from a topic
Contrib	shared library code, or alternative implementations of core sections (e.g. user management) when something "can't be a plugin because it needs close core access"

4. Access control (origin of yawex's model)

TWiki authorizes via preference settings in topics: ALLOWWEBVIEW / ALLOWWEBCHANGE at web level and ALLOWTOPICVIEW / ALLOWTOPICCHANGE / ALLOWTOPICRENAME (and DENY*) at topic level, with VIEW / CHANGE / RENAME grantable separately. Topic-level settings apply only to that topic. This per-topic ACL is the direct prior art behind yawex's AccessControl (denied/view/form/edit/admin) and a concrete reference for shard-wiki's optional per-page ACL at L4 (spec/ArchitectureBlueprint.md §5).

5. TWiki as a shard — capability profile

Capability	TWiki	Note
Read	✓	TWiki::Func / view script / rest; or read data/*.txt directly
Write	✓	saveTopic / save / rest; per-topic granularity
Structured payload	✓	TWiki Forms → %META:FIELD% in the text file (git-diffable)
Version/diff	✓	RCS .txt,v — open format, git-convertible (cf. XWiki DB)
Merge	partial	beforeMergeHandler; no git 3-way
Change hooks	✓	synchronous save/rename/attachment handlers
Auth model	per-web/topic ACL	authorize through shard-wiki core; engine ACL is advisory
Direct-store attach	✓✓	data/<Web>/<Topic>.txt + RCS is a folder shard on its own
Federation hooks	✓	plugin handlers + REST handlers can host an adapter (UC-38)

6. Mapping to shard-wiki INTENT (compare, do not equate)

6.1 Reinforcements

Observation	INTENT principle
Forms store data as %META% in the topic file	structured pages can be git-diffable, strengthening UC-34/39 vs XWiki's DB
RCS .txt,v history	git-addressable coordination — here history is convertible, not just supplementable (UC-36 → UC-41)
data/<Web>/<Topic>.txt mirrors logic	an engine's on-disk store is itself attachable as a folder shard (UC-40)
Plugin save/rename/REST handlers	composable integration — TWiki can host an adapter too, generalizing UC-38 beyond XWiki
Per-topic ALLOW/DENY ACL	prior art for per-page ACL at L4; lineage TWiki→yawex (UC-06)
Webs as namespaces / shards	namespace navigation (UC-22); webs as roots

6.2 Deliberate divergences (design bugs if conflated)

TWiki assumption	shard-wiki correction
TWikiML + variable expansion is the page	Markdown-first; TWikiML render stays out of core
ACL lives in topic preference settings	authorize in core; topic ALLOW/DENY is advisory provenance
History = RCS files next to content	coordination journal is the space-level git layer (may import RCS)
Apps = Forms + %SEARCH% rendered by TWiki	represent records without depending on TWiki to render/search them
Attach = run TWiki and call it	shard-wiki may attach the bare data dir with TWiki offline (UC-40)

6.3 What TWiki teaches that shard-wiki should not lose

1. File-backed engines are the easy, high-value case — their store is already a folder of text + open-format history; the adapter can often skip the runtime.
2. Structured data in text beats structured data in a DB for federation — diffable, portable, journal-friendly. Prefer file-embedded metadata where offered.
3. One backend, two attachment paths (runtime API vs on-disk store) is a real capability/consistency trade-off the adapter contract must express.

7. Use-case seeds → catalog (promoted 2026-06-13)

Seed	Catalog UC	Disposition
Attach a live engine's on-disk store directly (vs its runtime API)	UC-40 (new)	dual-path attach; fidelity/consistency trade-off
Import an engine's native file-based history (RCS) into the journal	UC-41 (new)	history migration with fidelity, vs UC-36 supplementation
Engine hosts adapter via its plugin/handler API	UC-38	enriched: TWiki save/rename/REST handlers generalize it beyond XWiki
Structured page carries typed data	UC-34 / UC-39	enriched: TWiki Forms %META%, git-diffable
Internal-history engine	UC-36	enriched: contrast DB (supplement) vs RCS (import, UC-41)
Authenticated team wiki / per-page ACL	UC-06	enriched: TWiki per-topic ACL is the origin of yawex's model

8. Open questions (for spec / workplans)

1. Dual-path adapters — when does shard-wiki attach an engine's on-disk store directly vs go through its API? Consistency risk of reading live files under a running engine; capability-gate it (UC-40)?
2. History import fidelity — can RCS .txt,v be converted to git commits preserving author/timestamp, and is that authoritative or a one-time backfill feeding the coordination journal (UC-41, cf. UC-36 Q3)?
3. File-embedded metadata mapping — map TWiki %META:FIELD% (and XWiki XObjects) onto one structured-metadata page-model representation (shared with xwiki §8 Q1).
4. TWiki vs Foswiki — target both via one adapter (shared API) or treat as distinct shard types?
5. ACL provenance — surface a topic's ALLOW/DENY as read-only provenance even though authorization is decided in core?

9. Sources

Source	URL
TWiki — TWikiPlugins (handlers, package types)	https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins
TWiki — Developing Plugins / EmptyPlugin	https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins
TWiki — TWiki::Func module	https://twiki.org/cgi-bin/view/TWiki/TWikiFuncModule
TWiki — TWiki Forms	https://twiki.org/cgi-bin/view/TWiki/TWikiForms
TWiki — TWiki Access Control	https://twiki.org/cgi-bin/view/TWiki/TWikiAccessControl
TWiki — System Requirements (Perl/RCS)	https://twiki.org/cgi-bin/view/TWiki/TWikiSystemRequirements
TWiki — Source Code	https://twiki.org/cgi-bin/view/TWiki/SourceCode
Foswiki — Why this fork	https://foswiki.org/Home/WhyThisFork
Wikipedia — TWiki	https://en.wikipedia.org/wiki/TWiki
Wikipedia — Foswiki	https://en.wikipedia.org/wiki/Foswiki
shard-wiki — XWiki deep dive	research/260613-xwiki-deep-dive/findings.md
shard-wiki — yawex prior art	research/260608-yawex-prior-art/findings.md

---

10. Traceability

This document section	Informs (future)
§2 architecture	adapter design for file-backed engines
§3 plugin handlers	UC-38 engine-side adapter; composable-integration API shape
§4 access control	per-page ACL at L4 (spec/ArchitectureBlueprint.md), UC-06
§5 capability profile	adapter capability-profile vocabulary (SHARD-WP-0002)
§6 INTENT mapping	architecture-blueprint guardrails
§7 UC seeds	spec/UseCaseCatalog.md (UC-40, UC-41; UC-06/34/36/38/39 enrichment)
§8 open questions	spec — dual-path adapters, history import, structured-metadata model