research: Foswiki deep dive (store abstraction, ext API); UC-42/43

Deep dive into Foswiki focused on its deltas from TWiki (not the shared lineage): the pluggable Foswiki::Store backend (RcsWrap/RcsLite/PlainFile) behind a versioned interface via Foswiki::Meta, the OO/MVC core rewrite, Foswiki::Func + registerTagHandler, DataForms + MetaDataPlugin multi-record, and WysiwygPlugin TML<->HTML round-trip. The store abstraction is logged as prior art for shard-wiki's own adapter contract (SHARD-WP-0002). Catalog (now 43 UCs): - UC-42 read/write a non-Markdown shard via lossless syntax translation (Markdown-first for prose; Foswiki WysiwygPlugin is proof) - UC-43 tolerate a shard's storage-backend swap (RCS<->PlainFile) under a stable identity - enrich UC-39 (multi-record metadata) and UC-40 (PlainFile direct-attach) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:27:24 +02:00
parent ffd5459b3e
commit fa7adab239
3 changed files with 274 additions and 5 deletions
--- a/research/260613-foswiki-deep-dive/README.md
+++ b/research/260613-foswiki-deep-dive/README.md
@@ -0,0 +1,36 @@
+# 260613 — Foswiki deep dive (store abstraction, extension API, ecosystem)
+
+Date: 2026-06-13
+
+## What this is
+
+A focused study of **Foswiki** — the community fork of TWiki (2008, from TWiki
+4.2.x) — concentrating on **what Foswiki changed**, not the lineage it shares with
+TWiki. The distinctive material: a **pluggable `Foswiki::Store` backend**
+(RcsWrap / RcsLite / **PlainFileStoreContrib**) behind a versioned interface, the
+`Foswiki::Meta` topic-object indirection, the OO/MVC core rewrite, the cleaner
+`Foswiki::Func` + `registerTagHandler` extension API, **DataForms** (+ MetaDataPlugin,
+FlexFormPlugin), and the **WysiwygPlugin** TML↔HTML round-trip.
+
+Read through shard-wiki's lens: Foswiki is the engine that **already separates its
+store backend from its core via a versioned interface** — i.e. it is concrete prior
+art for shard-wiki's own *shard adapter contract* and *capability-aware adapters*.
+
+Pairs with — and deliberately does not repeat — the shared TWiki material:
+- `research/260613-twiki-deep-dive/` — flat-file/RCS store, Webs/Topics, plugin
+  handler callbacks, per-topic ACL (all inherited; read there for the shared parts)
+- `research/260613-xwiki-deep-dive/` — the Java/DB app-platform sibling
+- `research/260608-wikiengines-overview/` — landscape scan
+
+## Contents
+
+| Path | Role |
+|------|------|
+| `findings.md` | Distinctives vs TWiki, store abstraction, extension API, DataForms, capability profile, INTENT mapping, UC seeds, sources |
+
+## Status
+
+Initial deep dive complete. Two new use cases promoted to
+`spec/UseCaseCatalog.md` (UC-42, UC-43); UC-34/39/40 enriched. The
+`Foswiki::Store`/`Foswiki::Meta` abstraction is logged as adapter-contract prior art
+for `workplans/SHARD-WP-0002-federation-architecture.md`.
--- a/research/260613-foswiki-deep-dive/findings.md
+++ b/research/260613-foswiki-deep-dive/findings.md
@@ -0,0 +1,188 @@
+# Findings — Foswiki: store abstraction, extension API, ecosystem
+
+Date: 2026-06-13 · Status: research draft
+
+Scope: **Foswiki as distinct from TWiki** — only the deltas, since the two share a
+data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to
+shard-wiki: a **pluggable store backend behind a versioned interface**
+(`Foswiki::Store` + `Foswiki::Meta`), an OO/MVC core rewrite, a cleaner extension
+API (`Foswiki::Func::registerTagHandler`), enhanced DataForms, and a TML↔HTML
+round-trip editor framework. Sources: foswiki.org Development + System docs,
+`foswiki/distro` on GitHub.
+
+**Read first for shared lineage:** `research/260613-twiki-deep-dive/` (flat-file +
+RCS store, Webs/Topics, handler callbacks, per-topic `ALLOW/DENY` ACL). This file
+does **not** repeat those.
+
+---
+
+## 1. Origin and compatibility
+
+Forked from **TWiki 4.2.x** in October 2008 by most of the TWiki community (sources
+cite 4.2.3–4.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content
+compatibility — same TML markup, same plugin API — via a **`TWikiCompatibilityPlugin`**
+that maps the `TWiki::` namespace and legacy variables. So everything in the TWiki
+dive still applies; Foswiki then grew the features below.
+
+## 2. The headline delta — a pluggable store behind a versioned interface
+
+This is the reason Foswiki earns its own dive.
+
+### 2.1 `Foswiki::Meta` indirection
+After Foswiki 1.0.4 the core was changed to **delegate almost all store operations
+to a topic object, `Foswiki::Meta`**. Callers manipulate a `Foswiki::Meta` object
+representing the item they're changing and **never touch the store implementation
+directly**; `Foswiki::Meta` talks to the real store through the well-defined
+**`Foswiki::Store`** interface. This is a clean MVC/repository separation.
+
+### 2.2 Swappable backends
+`Foswiki::Store::` ships multiple low-level back-ends behind that one interface:
+- **`RcsWrap`** — RCS via the external `rcs` binaries (classic TWiki behavior)
+- **`RcsLite`** — a pure-Perl RCS implementation
+- **`PlainFileStoreContrib`** — a newer store that saves topics/attachments as
+  **timestamped copies** instead of RCS diffs: more disk, but **no RCS dependency**
+  and much higher performance.
+
+So in Foswiki the **storage format is a configuration choice**, not a fixed property
+of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core.
+
+### 2.3 OO/MVC core rewrite
+Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward
+object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving
+backward compatibility through tests. The store abstraction above is the most
+load-bearing result.
+
+## 3. Extension API deltas
+
+Shared with TWiki: handler callbacks (`initPlugin`, `commonTagsHandler`,
+`before/afterSaveHandler`, `afterRenameHandler`, attachment handlers), REST handlers,
+`Config.spec`, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements:
+
+- **`Foswiki::Func` is the blessed API** — "if there's a `Foswiki::Func` way and
+  another way, the `Foswiki::Func` way is almost always right." Direct use of
+  internal packages is discouraged and governed by **`PluginsApiPolicies`**.
+- **`Foswiki::Func::registerTagHandler($tag, \&sub)`** — register a custom
+  macro/variable **programmatically** (cleaner than TWiki's `commonTagsHandler`
+  string-matching convention).
+- **Contrib-defined APIs** carry an explicit "use at your own risk / awaiting merge"
+  policy — a maturity signal on extension interfaces.
+- `EmptyPlugin` (`lib/Foswiki/Plugins/EmptyPlugin.pm`) is the canonical handler
+  reference/skeleton.
+
+## 4. DataForms and structured data (delta)
+
+DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi
+value, label) stored as `%META:FIELD%` in the topic text. Foswiki extends the model
+via popular extensions:
+- **MoreFormfieldsPlugin** — additional special-purpose field types.
+- **FlexFormPlugin** — render DataForm interfaces from custom templates
+  (`Foswiki::Form` classes).
+- **MetaDataPlugin** — **multiple structured data records per topic** (beyond the
+  classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model
+  while keeping data in the text file.
+
+## 5. Editing / syntax (delta relevant to "Markdown-first")
+
+- **WysiwygPlugin** — a generic framework that transforms **TML → HTML** for a
+  browser editor and **HTML → TML on save** (lossless round-trip).
+- **TinyMCEPlugin** — pre-installed WYSIWYG editor built on WysiwygPlugin.
+
+This proves **bidirectional, lossless translation between an engine's native markup
+and another representation is feasible** — directly relevant to shard-wiki reading a
+non-Markdown shard and writing Markdown overlays back (UC-42).
+
+## 6. Foswiki as a shard — capability profile (delta from TWiki)
+
+| Capability | Foswiki | Note vs TWiki |
+|------------|---------|---------------|
+| Store backend | **pluggable** | RCS *or* PlainFile behind `Foswiki::Store` — direct-attach target is cleaner on PlainFile (UC-40) |
+| History | RCS *or* timestamped copies | PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41) |
+| Structured payload | DataForms `%META%`, **multi-record** via MetaDataPlugin | richer than classic TWiki Forms (UC-34/39) |
+| Syntax translation | **WysiwygPlugin TML↔HTML** | bidirectional round-trip exists (UC-42) |
+| Extension host | `Foswiki::Func` + `registerTagHandler` + REST | cleaner adapter-host surface (UC-38) |
+| Store as contract | **`Foswiki::Store` versioned interface** | architectural prior art for shard-wiki's adapter contract |
+
+## 7. Mapping to shard-wiki INTENT (compare, do not equate)
+
+### 7.1 Reinforcements
+| Observation | INTENT principle |
+|-------------|------------------|
+| `Foswiki::Store` versioned interface + swappable backends | **capability-aware adapters** / **shard adapter contract** — an engine that already did exactly this separation |
+| `Foswiki::Meta` mediates all store access | clean **mechanism over policy** boundary worth mirroring in the adapter contract |
+| PlainFile store = timestamped text copies | **git-addressable coordination** / direct-attach (UC-40), history import (UC-41) |
+| WysiwygPlugin TML↔HTML round-trip | **Markdown-first, backend-neutral** is achievable for prose via translation (UC-42) |
+| MetaDataPlugin multi-record topics | structured pages beyond one form (UC-34/39) |
+
+### 7.2 Deliberate divergences (design bugs if conflated)
+| Foswiki assumption | shard-wiki correction |
+|--------------------|----------------------|
+| One pluggable store per wiki, chosen at config | shard-wiki federates **many** heterogeneous stores at once |
+| TML is the canonical syntax; HTML is a render target | shard-wiki is **Markdown-first**; TML↔Markdown is an adapter translation, not core |
+| Store interface is Foswiki-internal Perl | shard-wiki's adapter contract is **cross-language, cross-backend, versioned** |
+| ACL in topic preferences | **authorize in core**; engine ACL is advisory provenance |
+
+### 7.3 What Foswiki teaches that shard-wiki should not lose
+1. **The store-abstraction pattern works in practice** — `Foswiki::Store` is a
+   real-world proof that a stable interface + swappable backends is viable; the shard
+   adapter contract is the same idea generalized across engines.
+2. **Backend format should be swappable under a stable identity** — Foswiki migrates
+   RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's
+   backend changing under it (UC-43).
+3. **Lossless syntax round-trip is a solved problem** — don't treat non-Markdown
+   prose as read-only; translate it (UC-42).
+
+## 8. Use-case seeds → catalog (promoted 2026-06-13)
+
+| Seed | Catalog UC | Disposition |
+|------|------------|-------------|
+| Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown) | **UC-42** (new) | realizes Markdown-first for prose; WysiwygPlugin is proof |
+| Tolerate a shard's storage-backend swap without losing identity/provenance | **UC-43** (new) | Foswiki RCS↔PlainFile; orchestration robustness |
+| Structured / multi-record pages | UC-34 / UC-39 | enriched: DataForms + MetaDataPlugin multi-record |
+| Attach a file-backed engine's on-disk store directly | UC-40 | enriched: PlainFile store is a cleaner direct-attach target than RCS |
+| `Foswiki::Store` versioned interface | (no UC) | architecture prior art → `SHARD-WP-0002` adapter contract |
+
+## 9. Open questions (for spec / workplans)
+
+1. **Adapter-contract shape** — how much of `Foswiki::Store`'s method set
+   (read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's
+   capability-aware adapter contract? (architecture, `SHARD-WP-0002`)
+2. **Syntax-translation fidelity** — is TML↔Markdown lossless enough for overlay
+   round-trips, or must overlays be stored in TML to be safe (UC-42)?
+3. **Backend-swap detection** — how does shard-wiki notice and tolerate a shard
+   changing store format underneath (RCS→PlainFile), and does provenance/history
+   survive it (UC-43)?
+4. **Multi-record metadata** — represent MetaDataPlugin multi-record topics and XWiki
+   multi-XObject pages in one structured-metadata page model (shared open question).
+
+## 10. Sources
+
+| Source | URL |
+|--------|-----|
+| Foswiki — Technical Overview | https://foswiki.org/Development/TechnicalOverview |
+| Foswiki — Core Internals | https://foswiki.org/Development/CoreInternals |
+| Foswiki — PlainFileStoreContrib | https://foswiki.org/Extensions/PlainFileStoreContrib |
+| Foswiki — RCSStoreContrib | https://foswiki.org/Extensions/RCSStoreContrib |
+| Foswiki::Store::PlainFile (source) | https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm |
+| Foswiki — Developing Plugins | https://foswiki.org/System/DevelopingPlugins |
+| Foswiki — Plugins API Policies | https://foswiki.org/Development/PluginsApiPolicies |
+| Foswiki — DataForms | https://foswiki.org/System/DataForms |
+| Foswiki — MetaDataPlugin | https://foswiki.org/Extensions/MetaDataPlugin |
+| Foswiki — WysiwygPlugin | https://foswiki.org/System/WysiwygPlugin |
+| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
+| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
+| shard-wiki — TWiki deep dive | `research/260613-twiki-deep-dive/findings.md` |
+| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
+
+---
+
+## 11. Traceability
+
+| This document section | Informs (future) |
+|-----------------------|------------------|
+| §2 store abstraction | **adapter contract** design (`SHARD-WP-0002`) — Foswiki::Store as prior art |
+| §3 extension API | UC-38 engine-side adapter host |
+| §4 DataForms | structured-metadata page model (UC-34/39) |
+| §5 WysiwygPlugin | UC-42 syntax-translation adapter |
+| §6 capability profile | adapter capability-profile vocabulary |
+| §7 INTENT mapping | architecture-blueprint guardrails |
+| §8 UC seeds | `spec/UseCaseCatalog.md` (UC-42, UC-43; UC-34/39/40 enrichment) |