shard-wiki/research/260613-foswiki-deep-dive/findings.md

# Findings — Foswiki: store abstraction, extension API, ecosystem

Date: 2026-06-13 · Status: research draft

Scope: **Foswiki as distinct from TWiki** — only the deltas, since the two share a
data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to
shard-wiki: a **pluggable store backend behind a versioned interface**
(`Foswiki::Store` + `Foswiki::Meta`), an OO/MVC core rewrite, a cleaner extension
API (`Foswiki::Func::registerTagHandler`), enhanced DataForms, and a TML↔HTML
round-trip editor framework. Sources: foswiki.org Development + System docs,
`foswiki/distro` on GitHub.

**Read first for shared lineage:** `research/260613-twiki-deep-dive/` (flat-file +
RCS store, Webs/Topics, handler callbacks, per-topic `ALLOW/DENY` ACL). This file
does **not** repeat those.

---

## 1. Origin and compatibility

Forked from **TWiki 4.2.x** in October 2008 by most of the TWiki community (sources
cite 4.2.3–4.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content
compatibility — same TML markup, same plugin API — via a **`TWikiCompatibilityPlugin`**
that maps the `TWiki::` namespace and legacy variables. So everything in the TWiki
dive still applies; Foswiki then grew the features below.

## 2. The headline delta — a pluggable store behind a versioned interface

This is the reason Foswiki earns its own dive.

### 2.1 `Foswiki::Meta` indirection
After Foswiki 1.0.4 the core was changed to **delegate almost all store operations
to a topic object, `Foswiki::Meta`**. Callers manipulate a `Foswiki::Meta` object
representing the item they're changing and **never touch the store implementation
directly**; `Foswiki::Meta` talks to the real store through the well-defined
**`Foswiki::Store`** interface. This is a clean MVC/repository separation.

### 2.2 Swappable backends
`Foswiki::Store::` ships multiple low-level back-ends behind that one interface:
- **`RcsWrap`** — RCS via the external `rcs` binaries (classic TWiki behavior)
- **`RcsLite`** — a pure-Perl RCS implementation
- **`PlainFileStoreContrib`** — a newer store that saves topics/attachments as
  **timestamped copies** instead of RCS diffs: more disk, but **no RCS dependency**
  and much higher performance.

So in Foswiki the **storage format is a configuration choice**, not a fixed property
of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core.

### 2.3 OO/MVC core rewrite
Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward
object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving
backward compatibility through tests. The store abstraction above is the most
load-bearing result.

## 3. Extension API deltas

Shared with TWiki: handler callbacks (`initPlugin`, `commonTagsHandler`,
`before/afterSaveHandler`, `afterRenameHandler`, attachment handlers), REST handlers,
`Config.spec`, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements:

- **`Foswiki::Func` is the blessed API** — "if there's a `Foswiki::Func` way and
  another way, the `Foswiki::Func` way is almost always right." Direct use of
  internal packages is discouraged and governed by **`PluginsApiPolicies`**.
- **`Foswiki::Func::registerTagHandler($tag, \&sub)`** — register a custom
  macro/variable **programmatically** (cleaner than TWiki's `commonTagsHandler`
  string-matching convention).
- **Contrib-defined APIs** carry an explicit "use at your own risk / awaiting merge"
  policy — a maturity signal on extension interfaces.
- `EmptyPlugin` (`lib/Foswiki/Plugins/EmptyPlugin.pm`) is the canonical handler
  reference/skeleton.

## 4. DataForms and structured data (delta)

DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi
value, label) stored as `%META:FIELD%` in the topic text. Foswiki extends the model
via popular extensions:
- **MoreFormfieldsPlugin** — additional special-purpose field types.
- **FlexFormPlugin** — render DataForm interfaces from custom templates
  (`Foswiki::Form` classes).
- **MetaDataPlugin** — **multiple structured data records per topic** (beyond the
  classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model
  while keeping data in the text file.

## 5. Editing / syntax (delta relevant to "Markdown-first")

- **WysiwygPlugin** — a generic framework that transforms **TML → HTML** for a
  browser editor and **HTML → TML on save** (lossless round-trip).
- **TinyMCEPlugin** — pre-installed WYSIWYG editor built on WysiwygPlugin.

This proves **bidirectional, lossless translation between an engine's native markup
and another representation is feasible** — directly relevant to shard-wiki reading a
non-Markdown shard and writing Markdown overlays back (UC-42).

## 6. Foswiki as a shard — capability profile (delta from TWiki)

| Capability | Foswiki | Note vs TWiki |
|------------|---------|---------------|
| Store backend | **pluggable** | RCS *or* PlainFile behind `Foswiki::Store` — direct-attach target is cleaner on PlainFile (UC-40) |
| History | RCS *or* timestamped copies | PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41) |
| Structured payload | DataForms `%META%`, **multi-record** via MetaDataPlugin | richer than classic TWiki Forms (UC-34/39) |
| Syntax translation | **WysiwygPlugin TML↔HTML** | bidirectional round-trip exists (UC-42) |
| Extension host | `Foswiki::Func` + `registerTagHandler` + REST | cleaner adapter-host surface (UC-38) |
| Store as contract | **`Foswiki::Store` versioned interface** | architectural prior art for shard-wiki's adapter contract |

## 7. Mapping to shard-wiki INTENT (compare, do not equate)

### 7.1 Reinforcements
| Observation | INTENT principle |
|-------------|------------------|
| `Foswiki::Store` versioned interface + swappable backends | **capability-aware adapters** / **shard adapter contract** — an engine that already did exactly this separation |
| `Foswiki::Meta` mediates all store access | clean **mechanism over policy** boundary worth mirroring in the adapter contract |
| PlainFile store = timestamped text copies | **git-addressable coordination** / direct-attach (UC-40), history import (UC-41) |
| WysiwygPlugin TML↔HTML round-trip | **Markdown-first, backend-neutral** is achievable for prose via translation (UC-42) |
| MetaDataPlugin multi-record topics | structured pages beyond one form (UC-34/39) |

### 7.2 Deliberate divergences (design bugs if conflated)
| Foswiki assumption | shard-wiki correction |
|--------------------|----------------------|
| One pluggable store per wiki, chosen at config | shard-wiki federates **many** heterogeneous stores at once |
| TML is the canonical syntax; HTML is a render target | shard-wiki is **Markdown-first**; TML↔Markdown is an adapter translation, not core |
| Store interface is Foswiki-internal Perl | shard-wiki's adapter contract is **cross-language, cross-backend, versioned** |
| ACL in topic preferences | **authorize in core**; engine ACL is advisory provenance |

### 7.3 What Foswiki teaches that shard-wiki should not lose
1. **The store-abstraction pattern works in practice** — `Foswiki::Store` is a
   real-world proof that a stable interface + swappable backends is viable; the shard
   adapter contract is the same idea generalized across engines.
2. **Backend format should be swappable under a stable identity** — Foswiki migrates
   RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's
   backend changing under it (UC-43).
3. **Lossless syntax round-trip is a solved problem** — don't treat non-Markdown
   prose as read-only; translate it (UC-42).

## 8. Use-case seeds → catalog (promoted 2026-06-13)

| Seed | Catalog UC | Disposition |
|------|------------|-------------|
| Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown) | **UC-42** (new) | realizes Markdown-first for prose; WysiwygPlugin is proof |
| Tolerate a shard's storage-backend swap without losing identity/provenance | **UC-43** (new) | Foswiki RCS↔PlainFile; orchestration robustness |
| Structured / multi-record pages | UC-34 / UC-39 | enriched: DataForms + MetaDataPlugin multi-record |
| Attach a file-backed engine's on-disk store directly | UC-40 | enriched: PlainFile store is a cleaner direct-attach target than RCS |
| `Foswiki::Store` versioned interface | (no UC) | architecture prior art → `SHARD-WP-0002` adapter contract |

## 9. Open questions (for spec / workplans)

1. **Adapter-contract shape** — how much of `Foswiki::Store`'s method set
   (read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's
   capability-aware adapter contract? (architecture, `SHARD-WP-0002`)
2. **Syntax-translation fidelity** — is TML↔Markdown lossless enough for overlay
   round-trips, or must overlays be stored in TML to be safe (UC-42)?
3. **Backend-swap detection** — how does shard-wiki notice and tolerate a shard
   changing store format underneath (RCS→PlainFile), and does provenance/history
   survive it (UC-43)?
4. **Multi-record metadata** — represent MetaDataPlugin multi-record topics and XWiki
   multi-XObject pages in one structured-metadata page model (shared open question).

## 10. Sources

| Source | URL |
|--------|-----|
| Foswiki — Technical Overview | https://foswiki.org/Development/TechnicalOverview |
| Foswiki — Core Internals | https://foswiki.org/Development/CoreInternals |
| Foswiki — PlainFileStoreContrib | https://foswiki.org/Extensions/PlainFileStoreContrib |
| Foswiki — RCSStoreContrib | https://foswiki.org/Extensions/RCSStoreContrib |
| Foswiki::Store::PlainFile (source) | https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm |
| Foswiki — Developing Plugins | https://foswiki.org/System/DevelopingPlugins |
| Foswiki — Plugins API Policies | https://foswiki.org/Development/PluginsApiPolicies |
| Foswiki — DataForms | https://foswiki.org/System/DataForms |
| Foswiki — MetaDataPlugin | https://foswiki.org/Extensions/MetaDataPlugin |
| Foswiki — WysiwygPlugin | https://foswiki.org/System/WysiwygPlugin |
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
| shard-wiki — TWiki deep dive | `research/260613-twiki-deep-dive/findings.md` |
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |

---

## 11. Traceability

| This document section | Informs (future) |
|-----------------------|------------------|
| §2 store abstraction | **adapter contract** design (`SHARD-WP-0002`) — Foswiki::Store as prior art |
| §3 extension API | UC-38 engine-side adapter host |
| §4 DataForms | structured-metadata page model (UC-34/39) |
| §5 WysiwygPlugin | UC-42 syntax-translation adapter |
| §6 capability profile | adapter capability-profile vocabulary |
| §7 INTENT mapping | architecture-blueprint guardrails |
| §8 UC seeds | `spec/UseCaseCatalog.md` (UC-42, UC-43; UC-34/39/40 enrichment) |