# Findings — TWiki: implementation, plugin API, ecosystem

Date: 2026-06-13 · Status: research draft

Scope: TWiki as prior art for the **file-based** wiki-application-platform — the
counterpoint to XWiki's database/component platform. Two shard-wiki concerns:
(1) attaching a structured, file-backed engine as a shard (including attaching its
**on-disk store directly** vs through its runtime), and (2) how TWiki's plugin
handler API exposes hooks rich enough to host a federation adapter
(INTENT *composable integration*). Sources: twiki.org dev docs (TWikiPlugins,
TWikiForms, TWiki::Func, TWikiAccessControl), Foswiki docs, Wikipedia.

**Complements:** `research/260613-xwiki-deep-dive/` (DB/Java sibling),
`research/260608-wikiengines-overview/` (landscape), and
`research/260608-yawex-prior-art/` (yawex borrowed TWiki's per-topic AccessControl).

---

## 1. What TWiki is

A **structured enterprise wiki** and **wiki application platform** (Peter Thoeny,
~1998), written in **Perl 5** (5.10.1+), classically run as CGI. Like XWiki it lets
pages become forms/records/apps — but it reaches that with **flat text files +
RCS**, no database. Most of the community forked to **Foswiki** in October 2008
(from TWiki 4.2.4; Foswiki 1.0 in January 2009); the plugin API and data model below
are shared lineage, with minor divergences noted.

## 2. Implementation architecture

### 2.1 Storage — flat files + RCS (no DB)
- Content lives on disk as `data/<Web>/<Topic>.txt`; the directory tree **mirrors
  the logical Web/Topic hierarchy**. Attachments live under `pub/<Web>/<Topic>/`.
- **History is RCS** (`<Topic>.txt,v` companion files) — the same GNU RCS yawex used.
  History is real, per-file, and in an **open, git-convertible format** (unlike
  XWiki's DB-internal `xwikircs`).
- Scales to 300k+ topics on a single server (Yahoo) — file store is not a toy.

### 2.2 Webs and Topics
- **Topic** = page (the unit of content). **Web** = namespace / collection of topics
  (can be nested). Webs are the natural shard / root-entity boundary.

### 2.3 TWiki Forms — structured data, embedded in text
- A **form template** is a topic defining fields as a table (one row per field:
  name, type, size, values). A topic can *attach* a form; the field values are
  stored as **TWikiMetaData inside the topic `.txt`** (`%META:FIELD{...}%`), shown as
  a table on view and edited via fields/radios/checkboxes/lists.
- **Forms + formatted `%SEARCH%`** = TWiki's "database applications" — structured
  records queried by the SEARCH variable. This is the XWiki XObject/XClass idea, but
  **file-embedded and git-diffable** rather than rows in a DB.

### 2.4 TWikiML / variables (macros)
- Markup is **TWikiML** (not Markdown). Dynamic behavior comes from **variables**
  `%VAR%` / `%VAR{...}%` expanded at render time; `%SEARCH%` is the query engine.
- Custom variables are provided by plugins (see §3).

### 2.5 TWiki::Func — the official API boundary
- `lib/TWiki/Func.pm` ("TWikiFuncDotPm") documents **all** interfaces available to
  plugins, deliberately abstracting the flat-file assumption: `readTopic`,
  `saveTopic`, `getTopicList`, `getListOfWebs`, `getWorkArea`, `saveAttachment`, …
  Plugins are told *not* to assume flat files and to go through Func.

## 3. Plugin API — interfaces to extend the core

TWiki extends "without altering core code" by **registering Perl handlers that the
core calls at defined points** in the request/render/save pipeline. A plugin is
`lib/TWiki/Plugins/<Name>.pm` + a documentation topic + an optional `Config.spec`.

### 3.1 Handler callbacks (the hook surface)
| Phase | Handlers |
|-------|----------|
| Init / users | `initPlugin` (must return 1), `registrationHandler` |
| Tag/variable expansion | `commonTagsHandler`, `beforeCommonTagsHandler`, `afterCommonTagsHandler` |
| Rendering | `preRenderingHandler`, `postRenderingHandler` (`start/endRenderingHandler` deprecated) |
| Edit lifecycle | `beforeEditHandler`, `afterEditHandler` |
| Save lifecycle | `beforeSaveHandler`, `afterSaveHandler`, `beforeMergeHandler` |
| Attachments | `beforeAttachmentSaveHandler`, `afterAttachmentSaveHandler` |
| Topic mgmt | `afterRenameHandler`, `completePageHandler` |

These are the TWiki analogue of XWiki's `ObservationManager` events — but
**synchronous pipeline hooks** rather than an async event bus. The save/rename/
attachment handlers are exactly the interception points a federation adapter needs.

### 3.2 Other extension mechanisms
- **Custom variables/macros** — registered via `commonTagsHandler` (Foswiki adds
  `registerTagHandler`); convention `%PLUGINNAME_SETTING%`.
- **REST handlers** — a plugin registers REST handlers invoked via the `rest`
  script, for transactions outside the standard `view/edit/save` scripts. This is the
  natural remote adapter transport.
- **`Config.spec`** — declares configuration items (BOOLEAN, STRING, SELECT, PATH,
  PERL) surfaced in the `configure` web UI.
- **Work area** — `getWorkArea()` gives a plugin a persistent, non-web-accessible
  data directory.

### 3.3 Extension package types
| Type | Role |
|------|------|
| **Plugin** | handler-based behavior, no core change (`lib/TWiki/Plugins/<Name>.pm`) |
| **Skin** | visual appearance only; topic content unchanged (e.g. PrintSkin) |
| **AddOn** | a script in `bin/` callable from a topic |
| **Contrib** | shared library code, or **alternative implementations of core sections** (e.g. user management) when something "can't be a plugin because it needs close core access" |

## 4. Access control (origin of yawex's model)

TWiki authorizes via **preference settings in topics**: `ALLOWWEBVIEW` /
`ALLOWWEBCHANGE` at web level and `ALLOWTOPICVIEW` / `ALLOWTOPICCHANGE` /
`ALLOWTOPICRENAME` (and `DENY*`) at topic level, with **VIEW / CHANGE / RENAME
grantable separately**. Topic-level settings apply only to that topic. This
per-topic ACL is the **direct prior art** behind yawex's `AccessControl`
(denied/view/form/edit/admin) and a concrete reference for shard-wiki's optional
**per-page ACL at L4** (`spec/ArchitectureBlueprint.md` §5).

## 5. TWiki as a shard — capability profile

| Capability | TWiki | Note |
|------------|-------|------|
| Read | ✓ | TWiki::Func / `view` script / `rest`; or read `data/*.txt` directly |
| Write | ✓ | saveTopic / `save` / `rest`; **per-topic** granularity |
| Structured payload | ✓ | TWiki Forms → `%META:FIELD%` **in the text file** (git-diffable) |
| Version/diff | ✓ | RCS `.txt,v` — **open format, git-convertible** (cf. XWiki DB) |
| Merge | partial | `beforeMergeHandler`; no git 3-way |
| Change hooks | ✓ | synchronous save/rename/attachment handlers |
| Auth model | per-web/topic ACL | authorize through shard-wiki core; engine ACL is advisory |
| Direct-store attach | ✓✓ | `data/<Web>/<Topic>.txt` + RCS is a folder shard on its own |
| Federation hooks | ✓ | plugin handlers + REST handlers can host an adapter (UC-38) |

## 6. Mapping to shard-wiki INTENT (compare, do not equate)

### 6.1 Reinforcements
| Observation | INTENT principle |
|-------------|------------------|
| Forms store data as `%META%` **in the topic file** | structured pages can be **git-diffable**, strengthening UC-34/39 vs XWiki's DB |
| RCS `.txt,v` history | **git-addressable coordination** — here history is *convertible*, not just supplementable (UC-36 → UC-41) |
| `data/<Web>/<Topic>.txt` mirrors logic | an engine's **on-disk store is itself attachable as a folder shard** (UC-40) |
| Plugin save/rename/REST handlers | **composable integration** — TWiki can host an adapter too, generalizing UC-38 beyond XWiki |
| Per-topic `ALLOW/DENY` ACL | prior art for **per-page ACL at L4**; lineage TWiki→yawex (UC-06) |
| Webs as namespaces / shards | namespace navigation (UC-22); webs as roots |

### 6.2 Deliberate divergences (design bugs if conflated)
| TWiki assumption | shard-wiki correction |
|------------------|----------------------|
| TWikiML + variable expansion is the page | **Markdown-first**; TWikiML render stays out of core |
| ACL lives in topic preference settings | **authorize in core**; topic ALLOW/DENY is advisory provenance |
| History = RCS files next to content | **coordination journal** is the space-level git layer (may *import* RCS) |
| Apps = Forms + `%SEARCH%` rendered by TWiki | represent records **without** depending on TWiki to render/search them |
| Attach = run TWiki and call it | shard-wiki may attach the **bare data dir** with TWiki offline (UC-40) |

### 6.3 What TWiki teaches that shard-wiki should not lose
1. **File-backed engines are the easy, high-value case** — their store is already a
   folder of text + open-format history; the adapter can often skip the runtime.
2. **Structured data in text beats structured data in a DB** for federation —
   diffable, portable, journal-friendly. Prefer file-embedded metadata where offered.
3. **One backend, two attachment paths** (runtime API vs on-disk store) is a real
   capability/consistency trade-off the adapter contract must express.

## 7. Use-case seeds → catalog (promoted 2026-06-13)

| Seed | Catalog UC | Disposition |
|------|------------|-------------|
| Attach a live engine's **on-disk store directly** (vs its runtime API) | **UC-40** (new) | dual-path attach; fidelity/consistency trade-off |
| **Import** an engine's native file-based history (RCS) into the journal | **UC-41** (new) | history *migration* with fidelity, vs UC-36 *supplementation* |
| Engine hosts adapter via its plugin/handler API | UC-38 | enriched: TWiki save/rename/REST handlers generalize it beyond XWiki |
| Structured page carries typed data | UC-34 / UC-39 | enriched: TWiki Forms `%META%`, git-diffable |
| Internal-history engine | UC-36 | enriched: contrast DB (supplement) vs RCS (import, UC-41) |
| Authenticated team wiki / per-page ACL | UC-06 | enriched: TWiki per-topic ACL is the origin of yawex's model |

## 8. Open questions (for spec / workplans)

1. **Dual-path adapters** — when does shard-wiki attach an engine's on-disk store
   directly vs go through its API? Consistency risk of reading live files under a
   running engine; capability-gate it (UC-40)?
2. **History import fidelity** — can RCS `.txt,v` be converted to git commits
   preserving author/timestamp, and is that authoritative or a one-time backfill
   feeding the coordination journal (UC-41, cf. UC-36 Q3)?
3. **File-embedded metadata mapping** — map TWiki `%META:FIELD%` (and XWiki XObjects)
   onto one structured-metadata page-model representation (shared with xwiki §8 Q1).
4. **TWiki vs Foswiki** — target both via one adapter (shared API) or treat as
   distinct shard types?
5. **ACL provenance** — surface a topic's `ALLOW/DENY` as read-only provenance even
   though authorization is decided in core?

## 9. Sources

| Source | URL |
|--------|-----|
| TWiki — TWikiPlugins (handlers, package types) | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
| TWiki — Developing Plugins / EmptyPlugin | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
| TWiki — TWiki::Func module | https://twiki.org/cgi-bin/view/TWiki/TWikiFuncModule |
| TWiki — TWiki Forms | https://twiki.org/cgi-bin/view/TWiki/TWikiForms |
| TWiki — TWiki Access Control | https://twiki.org/cgi-bin/view/TWiki/TWikiAccessControl |
| TWiki — System Requirements (Perl/RCS) | https://twiki.org/cgi-bin/view/TWiki/TWikiSystemRequirements |
| TWiki — Source Code | https://twiki.org/cgi-bin/view/TWiki/SourceCode |
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
| Wikipedia — TWiki | https://en.wikipedia.org/wiki/TWiki |
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
| shard-wiki — yawex prior art | `research/260608-yawex-prior-art/findings.md` |

---

## 10. Traceability

| This document section | Informs (future) |
|-----------------------|------------------|
| §2 architecture | adapter design for file-backed engines |
| §3 plugin handlers | UC-38 engine-side adapter; composable-integration API shape |
| §4 access control | per-page ACL at L4 (`spec/ArchitectureBlueprint.md`), UC-06 |
| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) |
| §6 INTENT mapping | architecture-blueprint guardrails |
| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-40, UC-41; UC-06/34/36/38/39 enrichment) |
| §8 open questions | spec — dual-path adapters, history import, structured-metadata model |