research: Foswiki deep dive (store abstraction, ext API); UC-42/43

Deep dive into Foswiki focused on its deltas from TWiki (not the shared lineage): the pluggable Foswiki::Store backend (RcsWrap/RcsLite/PlainFile) behind a versioned interface via Foswiki::Meta, the OO/MVC core rewrite, Foswiki::Func + registerTagHandler, DataForms + MetaDataPlugin multi-record, and WysiwygPlugin TML<->HTML round-trip. The store abstraction is logged as prior art for shard-wiki's own adapter contract (SHARD-WP-0002). Catalog (now 43 UCs): - UC-42 read/write a non-Markdown shard via lossless syntax translation (Markdown-first for prose; Foswiki WysiwygPlugin is proof) - UC-43 tolerate a shard's storage-backend swap (RCS<->PlainFile) under a stable identity - enrich UC-39 (multi-record metadata) and UC-40 (PlainFile direct-attach) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:27:24 +02:00
parent ffd5459b3e
commit fa7adab239
3 changed files with 274 additions and 5 deletions
--- a/research/260613-foswiki-deep-dive/README.md
+++ b/research/260613-foswiki-deep-dive/README.md
@@ -0,0 +1,36 @@
+# 260613 — Foswiki deep dive (store abstraction, extension API, ecosystem)
+
+Date: 2026-06-13
+
+## What this is
+
+A focused study of **Foswiki** — the community fork of TWiki (2008, from TWiki
+4.2.x) — concentrating on **what Foswiki changed**, not the lineage it shares with
+TWiki. The distinctive material: a **pluggable `Foswiki::Store` backend**
+(RcsWrap / RcsLite / **PlainFileStoreContrib**) behind a versioned interface, the
+`Foswiki::Meta` topic-object indirection, the OO/MVC core rewrite, the cleaner
+`Foswiki::Func` + `registerTagHandler` extension API, **DataForms** (+ MetaDataPlugin,
+FlexFormPlugin), and the **WysiwygPlugin** TML↔HTML round-trip.
+
+Read through shard-wiki's lens: Foswiki is the engine that **already separates its
+store backend from its core via a versioned interface** — i.e. it is concrete prior
+art for shard-wiki's own *shard adapter contract* and *capability-aware adapters*.
+
+Pairs with — and deliberately does not repeat — the shared TWiki material:
+- `research/260613-twiki-deep-dive/` — flat-file/RCS store, Webs/Topics, plugin
+  handler callbacks, per-topic ACL (all inherited; read there for the shared parts)
+- `research/260613-xwiki-deep-dive/` — the Java/DB app-platform sibling
+- `research/260608-wikiengines-overview/` — landscape scan
+
+## Contents
+
+| Path | Role |
+|------|------|
+| `findings.md` | Distinctives vs TWiki, store abstraction, extension API, DataForms, capability profile, INTENT mapping, UC seeds, sources |
+
+## Status
+
+Initial deep dive complete. Two new use cases promoted to
+`spec/UseCaseCatalog.md` (UC-42, UC-43); UC-34/39/40 enriched. The
+`Foswiki::Store`/`Foswiki::Meta` abstraction is logged as adapter-contract prior art
+for `workplans/SHARD-WP-0002-federation-architecture.md`.
--- a/research/260613-foswiki-deep-dive/findings.md
+++ b/research/260613-foswiki-deep-dive/findings.md
@@ -0,0 +1,188 @@
+# Findings — Foswiki: store abstraction, extension API, ecosystem
+
+Date: 2026-06-13 · Status: research draft
+
+Scope: **Foswiki as distinct from TWiki** — only the deltas, since the two share a
+data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to
+shard-wiki: a **pluggable store backend behind a versioned interface**
+(`Foswiki::Store` + `Foswiki::Meta`), an OO/MVC core rewrite, a cleaner extension
+API (`Foswiki::Func::registerTagHandler`), enhanced DataForms, and a TML↔HTML
+round-trip editor framework. Sources: foswiki.org Development + System docs,
+`foswiki/distro` on GitHub.
+
+**Read first for shared lineage:** `research/260613-twiki-deep-dive/` (flat-file +
+RCS store, Webs/Topics, handler callbacks, per-topic `ALLOW/DENY` ACL). This file
+does **not** repeat those.
+
+---
+
+## 1. Origin and compatibility
+
+Forked from **TWiki 4.2.x** in October 2008 by most of the TWiki community (sources
+cite 4.2.3–4.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content
+compatibility — same TML markup, same plugin API — via a **`TWikiCompatibilityPlugin`**
+that maps the `TWiki::` namespace and legacy variables. So everything in the TWiki
+dive still applies; Foswiki then grew the features below.
+
+## 2. The headline delta — a pluggable store behind a versioned interface
+
+This is the reason Foswiki earns its own dive.
+
+### 2.1 `Foswiki::Meta` indirection
+After Foswiki 1.0.4 the core was changed to **delegate almost all store operations
+to a topic object, `Foswiki::Meta`**. Callers manipulate a `Foswiki::Meta` object
+representing the item they're changing and **never touch the store implementation
+directly**; `Foswiki::Meta` talks to the real store through the well-defined
+**`Foswiki::Store`** interface. This is a clean MVC/repository separation.
+
+### 2.2 Swappable backends
+`Foswiki::Store::` ships multiple low-level back-ends behind that one interface:
+- **`RcsWrap`** — RCS via the external `rcs` binaries (classic TWiki behavior)
+- **`RcsLite`** — a pure-Perl RCS implementation
+- **`PlainFileStoreContrib`** — a newer store that saves topics/attachments as
+  **timestamped copies** instead of RCS diffs: more disk, but **no RCS dependency**
+  and much higher performance.
+
+So in Foswiki the **storage format is a configuration choice**, not a fixed property
+of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core.
+
+### 2.3 OO/MVC core rewrite
+Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward
+object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving
+backward compatibility through tests. The store abstraction above is the most
+load-bearing result.
+
+## 3. Extension API deltas
+
+Shared with TWiki: handler callbacks (`initPlugin`, `commonTagsHandler`,
+`before/afterSaveHandler`, `afterRenameHandler`, attachment handlers), REST handlers,
+`Config.spec`, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements:
+
+- **`Foswiki::Func` is the blessed API** — "if there's a `Foswiki::Func` way and
+  another way, the `Foswiki::Func` way is almost always right." Direct use of
+  internal packages is discouraged and governed by **`PluginsApiPolicies`**.
+- **`Foswiki::Func::registerTagHandler($tag, \&sub)`** — register a custom
+  macro/variable **programmatically** (cleaner than TWiki's `commonTagsHandler`
+  string-matching convention).
+- **Contrib-defined APIs** carry an explicit "use at your own risk / awaiting merge"
+  policy — a maturity signal on extension interfaces.
+- `EmptyPlugin` (`lib/Foswiki/Plugins/EmptyPlugin.pm`) is the canonical handler
+  reference/skeleton.
+
+## 4. DataForms and structured data (delta)
+
+DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi
+value, label) stored as `%META:FIELD%` in the topic text. Foswiki extends the model
+via popular extensions:
+- **MoreFormfieldsPlugin** — additional special-purpose field types.
+- **FlexFormPlugin** — render DataForm interfaces from custom templates
+  (`Foswiki::Form` classes).
+- **MetaDataPlugin** — **multiple structured data records per topic** (beyond the
+  classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model
+  while keeping data in the text file.
+
+## 5. Editing / syntax (delta relevant to "Markdown-first")
+
+- **WysiwygPlugin** — a generic framework that transforms **TML → HTML** for a
+  browser editor and **HTML → TML on save** (lossless round-trip).
+- **TinyMCEPlugin** — pre-installed WYSIWYG editor built on WysiwygPlugin.
+
+This proves **bidirectional, lossless translation between an engine's native markup
+and another representation is feasible** — directly relevant to shard-wiki reading a
+non-Markdown shard and writing Markdown overlays back (UC-42).
+
+## 6. Foswiki as a shard — capability profile (delta from TWiki)
+
+| Capability | Foswiki | Note vs TWiki |
+|------------|---------|---------------|
+| Store backend | **pluggable** | RCS *or* PlainFile behind `Foswiki::Store` — direct-attach target is cleaner on PlainFile (UC-40) |
+| History | RCS *or* timestamped copies | PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41) |
+| Structured payload | DataForms `%META%`, **multi-record** via MetaDataPlugin | richer than classic TWiki Forms (UC-34/39) |
+| Syntax translation | **WysiwygPlugin TML↔HTML** | bidirectional round-trip exists (UC-42) |
+| Extension host | `Foswiki::Func` + `registerTagHandler` + REST | cleaner adapter-host surface (UC-38) |
+| Store as contract | **`Foswiki::Store` versioned interface** | architectural prior art for shard-wiki's adapter contract |
+
+## 7. Mapping to shard-wiki INTENT (compare, do not equate)
+
+### 7.1 Reinforcements
+| Observation | INTENT principle |
+|-------------|------------------|
+| `Foswiki::Store` versioned interface + swappable backends | **capability-aware adapters** / **shard adapter contract** — an engine that already did exactly this separation |
+| `Foswiki::Meta` mediates all store access | clean **mechanism over policy** boundary worth mirroring in the adapter contract |
+| PlainFile store = timestamped text copies | **git-addressable coordination** / direct-attach (UC-40), history import (UC-41) |
+| WysiwygPlugin TML↔HTML round-trip | **Markdown-first, backend-neutral** is achievable for prose via translation (UC-42) |
+| MetaDataPlugin multi-record topics | structured pages beyond one form (UC-34/39) |
+
+### 7.2 Deliberate divergences (design bugs if conflated)
+| Foswiki assumption | shard-wiki correction |
+|--------------------|----------------------|
+| One pluggable store per wiki, chosen at config | shard-wiki federates **many** heterogeneous stores at once |
+| TML is the canonical syntax; HTML is a render target | shard-wiki is **Markdown-first**; TML↔Markdown is an adapter translation, not core |
+| Store interface is Foswiki-internal Perl | shard-wiki's adapter contract is **cross-language, cross-backend, versioned** |
+| ACL in topic preferences | **authorize in core**; engine ACL is advisory provenance |
+
+### 7.3 What Foswiki teaches that shard-wiki should not lose
+1. **The store-abstraction pattern works in practice** — `Foswiki::Store` is a
+   real-world proof that a stable interface + swappable backends is viable; the shard
+   adapter contract is the same idea generalized across engines.
+2. **Backend format should be swappable under a stable identity** — Foswiki migrates
+   RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's
+   backend changing under it (UC-43).
+3. **Lossless syntax round-trip is a solved problem** — don't treat non-Markdown
+   prose as read-only; translate it (UC-42).
+
+## 8. Use-case seeds → catalog (promoted 2026-06-13)
+
+| Seed | Catalog UC | Disposition |
+|------|------------|-------------|
+| Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown) | **UC-42** (new) | realizes Markdown-first for prose; WysiwygPlugin is proof |
+| Tolerate a shard's storage-backend swap without losing identity/provenance | **UC-43** (new) | Foswiki RCS↔PlainFile; orchestration robustness |
+| Structured / multi-record pages | UC-34 / UC-39 | enriched: DataForms + MetaDataPlugin multi-record |
+| Attach a file-backed engine's on-disk store directly | UC-40 | enriched: PlainFile store is a cleaner direct-attach target than RCS |
+| `Foswiki::Store` versioned interface | (no UC) | architecture prior art → `SHARD-WP-0002` adapter contract |
+
+## 9. Open questions (for spec / workplans)
+
+1. **Adapter-contract shape** — how much of `Foswiki::Store`'s method set
+   (read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's
+   capability-aware adapter contract? (architecture, `SHARD-WP-0002`)
+2. **Syntax-translation fidelity** — is TML↔Markdown lossless enough for overlay
+   round-trips, or must overlays be stored in TML to be safe (UC-42)?
+3. **Backend-swap detection** — how does shard-wiki notice and tolerate a shard
+   changing store format underneath (RCS→PlainFile), and does provenance/history
+   survive it (UC-43)?
+4. **Multi-record metadata** — represent MetaDataPlugin multi-record topics and XWiki
+   multi-XObject pages in one structured-metadata page model (shared open question).
+
+## 10. Sources
+
+| Source | URL |
+|--------|-----|
+| Foswiki — Technical Overview | https://foswiki.org/Development/TechnicalOverview |
+| Foswiki — Core Internals | https://foswiki.org/Development/CoreInternals |
+| Foswiki — PlainFileStoreContrib | https://foswiki.org/Extensions/PlainFileStoreContrib |
+| Foswiki — RCSStoreContrib | https://foswiki.org/Extensions/RCSStoreContrib |
+| Foswiki::Store::PlainFile (source) | https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm |
+| Foswiki — Developing Plugins | https://foswiki.org/System/DevelopingPlugins |
+| Foswiki — Plugins API Policies | https://foswiki.org/Development/PluginsApiPolicies |
+| Foswiki — DataForms | https://foswiki.org/System/DataForms |
+| Foswiki — MetaDataPlugin | https://foswiki.org/Extensions/MetaDataPlugin |
+| Foswiki — WysiwygPlugin | https://foswiki.org/System/WysiwygPlugin |
+| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
+| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
+| shard-wiki — TWiki deep dive | `research/260613-twiki-deep-dive/findings.md` |
+| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
+
+---
+
+## 11. Traceability
+
+| This document section | Informs (future) |
+|-----------------------|------------------|
+| §2 store abstraction | **adapter contract** design (`SHARD-WP-0002`) — Foswiki::Store as prior art |
+| §3 extension API | UC-38 engine-side adapter host |
+| §4 DataForms | structured-metadata page model (UC-34/39) |
+| §5 WysiwygPlugin | UC-42 syntax-translation adapter |
+| §6 capability profile | adapter capability-profile vocabulary |
+| §7 INTENT mapping | architecture-blueprint guardrails |
+| §8 UC seeds | `spec/UseCaseCatalog.md` (UC-42, UC-43; UC-34/39/40 enrichment) |
--- a/spec/UseCaseCatalog.md
+++ b/spec/UseCaseCatalog.md
@@ -4,8 +4,8 @@ Status: **draft** · Date: 2026-06-08 · Updated: 2026-06-13

 Promoted from `research/260608-c2-wiki-origins/`,
 `research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`,
-`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`, and
-`research/260613-twiki-deep-dive/`.
+`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`,
+`research/260613-twiki-deep-dive/`, and `research/260613-foswiki-deep-dive/`.
 See InfoTechPrimers on coulomb.social for use-case catalog conventions.

 ## Conventions
@@ -272,7 +272,10 @@ union without inventing a fake Markdown body.
 **Notes:** The extreme of UC-34 — not *annotations on prose* but **bodiless
 structured pages**. Forces the question: does the page model require a Markdown body,
 or can a page be purely typed data (`research/260613-xwiki-deep-dive/findings.md`
-§8 Q2)? Affects union views, diff, and search. Deferred to `SHARD-WP-0002`.  
+§8 Q2)? Affects union views, diff, and search. Deferred to `SHARD-WP-0002`.
+Foswiki shows a middle point: MetaDataPlugin stores **multiple** structured records
+per topic, still as `%META%` in text (`research/260613-foswiki-deep-dive/findings.md`
+§4) — the page model must allow N typed records, not one form.  
 **Priority:** Later

 ### UC-40 — Attach a file-backed engine's on-disk store directly
@@ -288,7 +291,10 @@ the on-disk store (offline-capable, git-native for repo-backed engines, but risk
 reading inconsistent state under a running engine) vs the runtime API (consistent,
 respects engine logic, but requires the engine up). Capability profile must express
 the choice (`research/260613-twiki-deep-dive/findings.md` §5–§6, §8 Q1). Distinct
-from UC-37 (no live origin) and broader than UC-02's generic attach.  
+from UC-37 (no live origin) and broader than UC-02's generic attach. Foswiki's
+**PlainFile** store (timestamped whole-version copies, no RCS) is a cleaner
+direct-attach target than RCS diffs (`research/260613-foswiki-deep-dive/findings.md`
+§2.2).  
 **Priority:** Later

 ### UC-41 — Import an engine's native file history into the coordination journal
@@ -305,6 +311,37 @@ is the imported history authoritative or a one-time backfill
 (INTENT *History as the safety net*).  
 **Priority:** Later

+### UC-42 — Read and write a non-Markdown shard via lossless syntax translation
+
+**Actor:** Author or maintainer  
+**Goal:** Attach a shard whose native markup is not Markdown (TWiki/Foswiki TML,
+XWiki syntax) and read it into the Markdown-first model — and write Markdown overlays
+back — through **bidirectional, lossless syntax translation**, instead of treating it
+as read-only.  
+**Source:** wikiengines, intent  
+**Notes:** Realizes *Markdown-first, backend-neutral* for **prose** (UC-34/39 cover
+*structure*). Feasibility proven by Foswiki's **WysiwygPlugin** (TML→HTML for editing,
+HTML→TML losslessly on save) (`research/260613-foswiki-deep-dive/findings.md` §5).
+Open: is round-trip lossless enough, or must overlays be stored in the shard's native
+syntax to be safe (findings §9 Q2)? Without this, non-Markdown shards degrade to
+read-only (UC-03) — a graceful-degradation floor, not the goal.  
+**Priority:** Later
+
+### UC-43 — Tolerate a shard's storage-backend swap without losing identity
+
+**Actor:** Maintainer  
+**Goal:** Keep a shard attached, with its provenance and history intact, when its
+underlying storage backend changes — e.g. a Foswiki wiki migrated RCS→PlainFile, or a
+folder shard promoted into Git.  
+**Source:** wikiengines, intent  
+**Notes:** Foswiki proves backends are swappable under a stable identity via its
+versioned `Foswiki::Store` interface (`research/260613-foswiki-deep-dive/findings.md`
+§2). shard-wiki orchestration robustness: a shard's identity/attachment is **not** its
+storage format. Open: detect the swap and preserve history across it (findings §9 Q3).
+Relates to UC-40 (attach path) and UC-41 (history import) but is about *change under a
+live attachment*, not initial attach.  
+**Priority:** Later
+
 ---

 ## B. Knowledge work and collaboration
@@ -525,6 +562,8 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD.
 | UC-39 | | | | ✓ | ✓ |
 | UC-40 | | | | ✓ | ✓ |
 | UC-41 | | | | ✓ | ✓ |
+| UC-42 | | | | ✓ | ✓ |
+| UC-43 | | | | ✓ | ✓ |
 | UC-08 | ✓ | | |
 | UC-09 | ✓ | | |
 | UC-10 | ✓ | | |
@@ -598,6 +637,8 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD.
 | Wiki-as-application-platform, bodiless typed pages (XWiki AppWithinMinutes/XObjects) | UC-39 |
 | Attach a file-backed engine's on-disk store directly (TWiki data dir, DokuWiki) | UC-40 |
 | Import an engine's native file history (TWiki RCS `.txt,v`) into the journal | UC-41 |
+| Lossless syntax translation for a non-Markdown shard (Foswiki WysiwygPlugin TML↔HTML) | UC-42 |
+| Tolerate a shard's storage-backend swap (Foswiki RCS↔PlainFile) under stable identity | UC-43 |

 Note: the landscape scan mostly **reinforced** existing UCs and the L0→L4 ladder
 rather than adding scenarios — its primary yield is adapter-contract constraints
@@ -607,7 +648,11 @@ attachment scenarios it surfaced. UC-38–UC-39 come from the **XWiki deep dive*
 (`research/260613-xwiki-deep-dive/findings.md` §7); UC-40–UC-41 from the **TWiki
 deep dive** (`research/260613-twiki-deep-dive/findings.md` §7), which also enriched
 UC-06 (TWiki per-topic ACL → yawex), UC-34 (file-embedded `%META%`), UC-36 (RCS vs
-DB history), and UC-38 (TWiki plugin handlers as an adapter host).
+DB history), and UC-38 (TWiki plugin handlers as an adapter host). UC-42–UC-43 come
+from the **Foswiki deep dive** (`research/260613-foswiki-deep-dive/findings.md` §8),
+which also enriched UC-39 (MetaDataPlugin multi-record), UC-40 (PlainFile store), and
+logged the `Foswiki::Store` versioned interface as **adapter-contract prior art** for
+`SHARD-WP-0002` (no UC — architecture).

 ---