shard-wiki/research/260613-foswiki-deep-dive/findings.md

Findings — Foswiki: store abstraction, extension API, ecosystem

Date: 2026-06-13 · Status: research draft

Scope: Foswiki as distinct from TWiki — only the deltas, since the two share a data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to shard-wiki: a pluggable store backend behind a versioned interface (Foswiki::Store + Foswiki::Meta), an OO/MVC core rewrite, a cleaner extension API (Foswiki::Func::registerTagHandler), enhanced DataForms, and a TML↔HTML round-trip editor framework. Sources: foswiki.org Development + System docs, foswiki/distro on GitHub.

Read first for shared lineage: research/260613-twiki-deep-dive/ (flat-file + RCS store, Webs/Topics, handler callbacks, per-topic ALLOW/DENY ACL). This file does not repeat those.

---

1. Origin and compatibility

Forked from TWiki 4.2.x in October 2008 by most of the TWiki community (sources cite 4.2.3–4.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content compatibility — same TML markup, same plugin API — via a TWikiCompatibilityPlugin that maps the TWiki:: namespace and legacy variables. So everything in the TWiki dive still applies; Foswiki then grew the features below.

2. The headline delta — a pluggable store behind a versioned interface

This is the reason Foswiki earns its own dive.

2.1 Foswiki::Meta indirection

After Foswiki 1.0.4 the core was changed to delegate almost all store operations to a topic object, Foswiki::Meta. Callers manipulate a Foswiki::Meta object representing the item they're changing and never touch the store implementation directly; Foswiki::Meta talks to the real store through the well-defined Foswiki::Store interface. This is a clean MVC/repository separation.

2.2 Swappable backends

Foswiki::Store:: ships multiple low-level back-ends behind that one interface:

• RcsWrap — RCS via the external rcs binaries (classic TWiki behavior)
• RcsLite — a pure-Perl RCS implementation
• PlainFileStoreContrib — a newer store that saves topics/attachments as timestamped copies instead of RCS diffs: more disk, but no RCS dependency and much higher performance.

So in Foswiki the storage format is a configuration choice, not a fixed property of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core.

2.3 OO/MVC core rewrite

Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving backward compatibility through tests. The store abstraction above is the most load-bearing result.

3. Extension API deltas

Shared with TWiki: handler callbacks (initPlugin, commonTagsHandler, before/afterSaveHandler, afterRenameHandler, attachment handlers), REST handlers, Config.spec, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements:

• Foswiki::Func is the blessed API — "if there's a Foswiki::Func way and another way, the Foswiki::Func way is almost always right." Direct use of internal packages is discouraged and governed by PluginsApiPolicies.
• Foswiki::Func::registerTagHandler($tag, \&sub) — register a custom macro/variable programmatically (cleaner than TWiki's commonTagsHandler string-matching convention).
• Contrib-defined APIs carry an explicit "use at your own risk / awaiting merge" policy — a maturity signal on extension interfaces.
• EmptyPlugin (lib/Foswiki/Plugins/EmptyPlugin.pm) is the canonical handler reference/skeleton.

4. DataForms and structured data (delta)

DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi value, label) stored as %META:FIELD% in the topic text. Foswiki extends the model via popular extensions:

• MoreFormfieldsPlugin — additional special-purpose field types.
• FlexFormPlugin — render DataForm interfaces from custom templates (Foswiki::Form classes).
• MetaDataPlugin — multiple structured data records per topic (beyond the classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model while keeping data in the text file.

5. Editing / syntax (delta relevant to "Markdown-first")

• WysiwygPlugin — a generic framework that transforms TML → HTML for a browser editor and HTML → TML on save (lossless round-trip).
• TinyMCEPlugin — pre-installed WYSIWYG editor built on WysiwygPlugin.

This proves bidirectional, lossless translation between an engine's native markup and another representation is feasible — directly relevant to shard-wiki reading a non-Markdown shard and writing Markdown overlays back (UC-42).

6. Foswiki as a shard — capability profile (delta from TWiki)

Capability	Foswiki	Note vs TWiki
Store backend	pluggable	RCS or PlainFile behind Foswiki::Store — direct-attach target is cleaner on PlainFile (UC-40)
History	RCS or timestamped copies	PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41)
Structured payload	DataForms %META%, multi-record via MetaDataPlugin	richer than classic TWiki Forms (UC-34/39)
Syntax translation	WysiwygPlugin TML↔HTML	bidirectional round-trip exists (UC-42)
Extension host	Foswiki::Func + registerTagHandler + REST	cleaner adapter-host surface (UC-38)
Store as contract	Foswiki::Store versioned interface	architectural prior art for shard-wiki's adapter contract

7. Mapping to shard-wiki INTENT (compare, do not equate)

7.1 Reinforcements

Observation	INTENT principle
Foswiki::Store versioned interface + swappable backends	capability-aware adapters / shard adapter contract — an engine that already did exactly this separation
Foswiki::Meta mediates all store access	clean mechanism over policy boundary worth mirroring in the adapter contract
PlainFile store = timestamped text copies	git-addressable coordination / direct-attach (UC-40), history import (UC-41)
WysiwygPlugin TML↔HTML round-trip	Markdown-first, backend-neutral is achievable for prose via translation (UC-42)
MetaDataPlugin multi-record topics	structured pages beyond one form (UC-34/39)

7.2 Deliberate divergences (design bugs if conflated)

Foswiki assumption	shard-wiki correction
One pluggable store per wiki, chosen at config	shard-wiki federates many heterogeneous stores at once
TML is the canonical syntax; HTML is a render target	shard-wiki is Markdown-first; TML↔Markdown is an adapter translation, not core
Store interface is Foswiki-internal Perl	shard-wiki's adapter contract is cross-language, cross-backend, versioned
ACL in topic preferences	authorize in core; engine ACL is advisory provenance

7.3 What Foswiki teaches that shard-wiki should not lose

1. The store-abstraction pattern works in practice — Foswiki::Store is a real-world proof that a stable interface + swappable backends is viable; the shard adapter contract is the same idea generalized across engines.
2. Backend format should be swappable under a stable identity — Foswiki migrates RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's backend changing under it (UC-43).
3. Lossless syntax round-trip is a solved problem — don't treat non-Markdown prose as read-only; translate it (UC-42).

8. Use-case seeds → catalog (promoted 2026-06-13)

Seed	Catalog UC	Disposition
Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown)	UC-42 (new)	realizes Markdown-first for prose; WysiwygPlugin is proof
Tolerate a shard's storage-backend swap without losing identity/provenance	UC-43 (new)	Foswiki RCS↔PlainFile; orchestration robustness
Structured / multi-record pages	UC-34 / UC-39	enriched: DataForms + MetaDataPlugin multi-record
Attach a file-backed engine's on-disk store directly	UC-40	enriched: PlainFile store is a cleaner direct-attach target than RCS
Foswiki::Store versioned interface	(no UC)	architecture prior art → SHARD-WP-0002 adapter contract

9. Open questions (for spec / workplans)

1. Adapter-contract shape — how much of Foswiki::Store's method set (read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's capability-aware adapter contract? (architecture, SHARD-WP-0002)
2. Syntax-translation fidelity — is TML↔Markdown lossless enough for overlay round-trips, or must overlays be stored in TML to be safe (UC-42)?
3. Backend-swap detection — how does shard-wiki notice and tolerate a shard changing store format underneath (RCS→PlainFile), and does provenance/history survive it (UC-43)?
4. Multi-record metadata — represent MetaDataPlugin multi-record topics and XWiki multi-XObject pages in one structured-metadata page model (shared open question).

10. Sources

Source	URL
Foswiki — Technical Overview	https://foswiki.org/Development/TechnicalOverview
Foswiki — Core Internals	https://foswiki.org/Development/CoreInternals
Foswiki — PlainFileStoreContrib	https://foswiki.org/Extensions/PlainFileStoreContrib
Foswiki — RCSStoreContrib	https://foswiki.org/Extensions/RCSStoreContrib
Foswiki::Store::PlainFile (source)	https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm
Foswiki — Developing Plugins	https://foswiki.org/System/DevelopingPlugins
Foswiki — Plugins API Policies	https://foswiki.org/Development/PluginsApiPolicies
Foswiki — DataForms	https://foswiki.org/System/DataForms
Foswiki — MetaDataPlugin	https://foswiki.org/Extensions/MetaDataPlugin
Foswiki — WysiwygPlugin	https://foswiki.org/System/WysiwygPlugin
Foswiki — Why this fork	https://foswiki.org/Home/WhyThisFork
Wikipedia — Foswiki	https://en.wikipedia.org/wiki/Foswiki
shard-wiki — TWiki deep dive	research/260613-twiki-deep-dive/findings.md
shard-wiki — XWiki deep dive	research/260613-xwiki-deep-dive/findings.md

---

11. Traceability

This document section	Informs (future)
§2 store abstraction	adapter contract design (SHARD-WP-0002) — Foswiki::Store as prior art
§3 extension API	UC-38 engine-side adapter host
§4 DataForms	structured-metadata page model (UC-34/39)
§5 WysiwygPlugin	UC-42 syntax-translation adapter
§6 capability profile	adapter capability-profile vocabulary
§7 INTENT mapping	architecture-blueprint guardrails
§8 UC seeds	spec/UseCaseCatalog.md (UC-42, UC-43; UC-34/39/40 enrichment)