coulomb/shard-wiki
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

research: Foswiki deep dive (store abstraction, ext API); UC-42/43

Browse Source 
Deep dive into Foswiki focused on its deltas from TWiki (not the shared
lineage): the pluggable Foswiki::Store backend (RcsWrap/RcsLite/PlainFile)
behind a versioned interface via Foswiki::Meta, the OO/MVC core rewrite,
Foswiki::Func + registerTagHandler, DataForms + MetaDataPlugin multi-record,
and WysiwygPlugin TML<->HTML round-trip. The store abstraction is logged as
prior art for shard-wiki's own adapter contract (SHARD-WP-0002).

Catalog (now 43 UCs):
- UC-42 read/write a non-Markdown shard via lossless syntax translation
  (Markdown-first for prose; Foswiki WysiwygPlugin is proof)
- UC-43 tolerate a shard's storage-backend swap (RCS<->PlainFile) under a
  stable identity
- enrich UC-39 (multi-record metadata) and UC-40 (PlainFile direct-attach)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-06-13 23:27:24 +02:00
parent ffd5459b3e
commit fa7adab239
3 changed files with 274 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
research/260613-foswiki-deep-dive/README.md Normal file
View File

@@ -0,0 +1,36 @@
# 260613 — Foswiki deep dive (store abstraction, extension API, ecosystem)
Date: 2026-06-13
## What this is
A focused study of **Foswiki** — the community fork of TWiki (2008, from TWiki
4.2.x) — concentrating on **what Foswiki changed**, not the lineage it shares with
TWiki. The distinctive material: a **pluggable `Foswiki::Store` backend**
(RcsWrap / RcsLite / **PlainFileStoreContrib**) behind a versioned interface, the
`Foswiki::Meta` topic-object indirection, the OO/MVC core rewrite, the cleaner
`Foswiki::Func` + `registerTagHandler` extension API, **DataForms** (+ MetaDataPlugin,
FlexFormPlugin), and the **WysiwygPlugin** TML↔HTML round-trip.
Read through shard-wiki's lens: Foswiki is the engine that **already separates its
store backend from its core via a versioned interface** — i.e. it is concrete prior
art for shard-wiki's own *shard adapter contract* and *capability-aware adapters*.
Pairs with — and deliberately does not repeat — the shared TWiki material:
- `research/260613-twiki-deep-dive/` — flat-file/RCS store, Webs/Topics, plugin
handler callbacks, per-topic ACL (all inherited; read there for the shared parts)
- `research/260613-xwiki-deep-dive/` — the Java/DB app-platform sibling
- `research/260608-wikiengines-overview/` — landscape scan
## Contents
| Path | Role |
|------|------|
| `findings.md` | Distinctives vs TWiki, store abstraction, extension API, DataForms, capability profile, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Two new use cases promoted to
`spec/UseCaseCatalog.md` (UC-42, UC-43); UC-34/39/40 enriched. The
`Foswiki::Store`/`Foswiki::Meta` abstraction is logged as adapter-contract prior art
for `workplans/SHARD-WP-0002-federation-architecture.md`.

188
research/260613-foswiki-deep-dive/findings.md Normal file
View File

@@ -0,0 +1,188 @@
# Findings — Foswiki: store abstraction, extension API, ecosystem
Date: 2026-06-13 · Status: research draft
Scope: **Foswiki as distinct from TWiki** — only the deltas, since the two share a
data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to
shard-wiki: a **pluggable store backend behind a versioned interface**
(`Foswiki::Store` + `Foswiki::Meta`), an OO/MVC core rewrite, a cleaner extension
API (`Foswiki::Func::registerTagHandler`), enhanced DataForms, and a TML↔HTML
round-trip editor framework. Sources: foswiki.org Development + System docs,
`foswiki/distro` on GitHub.
**Read first for shared lineage:** `research/260613-twiki-deep-dive/` (flat-file +
RCS store, Webs/Topics, handler callbacks, per-topic `ALLOW/DENY` ACL). This file
does **not** repeat those.
---
## 1. Origin and compatibility
Forked from **TWiki 4.2.x** in October 2008 by most of the TWiki community (sources
cite 4.2.3–4.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content
compatibility — same TML markup, same plugin API — via a **`TWikiCompatibilityPlugin`**
that maps the `TWiki::` namespace and legacy variables. So everything in the TWiki
dive still applies; Foswiki then grew the features below.
## 2. The headline delta — a pluggable store behind a versioned interface
This is the reason Foswiki earns its own dive.
### 2.1 `Foswiki::Meta` indirection
After Foswiki 1.0.4 the core was changed to **delegate almost all store operations
to a topic object, `Foswiki::Meta`**. Callers manipulate a `Foswiki::Meta` object
representing the item they're changing and **never touch the store implementation
directly**; `Foswiki::Meta` talks to the real store through the well-defined
**`Foswiki::Store`** interface. This is a clean MVC/repository separation.
### 2.2 Swappable backends
`Foswiki::Store::` ships multiple low-level back-ends behind that one interface:
- **`RcsWrap`** — RCS via the external `rcs` binaries (classic TWiki behavior)
- **`RcsLite`** — a pure-Perl RCS implementation
- **`PlainFileStoreContrib`** — a newer store that saves topics/attachments as
**timestamped copies** instead of RCS diffs: more disk, but **no RCS dependency**
and much higher performance.
So in Foswiki the **storage format is a configuration choice**, not a fixed property
of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core.
### 2.3 OO/MVC core rewrite
Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward
object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving
backward compatibility through tests. The store abstraction above is the most
load-bearing result.
## 3. Extension API deltas
Shared with TWiki: handler callbacks (`initPlugin`, `commonTagsHandler`,
`before/afterSaveHandler`, `afterRenameHandler`, attachment handlers), REST handlers,
`Config.spec`, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements:
- **`Foswiki::Func` is the blessed API** — "if there's a `Foswiki::Func` way and
another way, the `Foswiki::Func` way is almost always right." Direct use of
internal packages is discouraged and governed by **`PluginsApiPolicies`**.
- **`Foswiki::Func::registerTagHandler($tag, \&sub)`** — register a custom
macro/variable **programmatically** (cleaner than TWiki's `commonTagsHandler`
string-matching convention).
- **Contrib-defined APIs** carry an explicit "use at your own risk / awaiting merge"
policy — a maturity signal on extension interfaces.
- `EmptyPlugin` (`lib/Foswiki/Plugins/EmptyPlugin.pm`) is the canonical handler
reference/skeleton.
## 4. DataForms and structured data (delta)
DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi
value, label) stored as `%META:FIELD%` in the topic text. Foswiki extends the model
via popular extensions:
- **MoreFormfieldsPlugin** — additional special-purpose field types.
- **FlexFormPlugin** — render DataForm interfaces from custom templates
(`Foswiki::Form` classes).
- **MetaDataPlugin** — **multiple structured data records per topic** (beyond the
classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model
while keeping data in the text file.
## 5. Editing / syntax (delta relevant to "Markdown-first")
- **WysiwygPlugin** — a generic framework that transforms **TML → HTML** for a
browser editor and **HTML → TML on save** (lossless round-trip).
- **TinyMCEPlugin** — pre-installed WYSIWYG editor built on WysiwygPlugin.
This proves **bidirectional, lossless translation between an engine's native markup
and another representation is feasible** — directly relevant to shard-wiki reading a
non-Markdown shard and writing Markdown overlays back (UC-42).
## 6. Foswiki as a shard — capability profile (delta from TWiki)
| Capability | Foswiki | Note vs TWiki |
|------------|---------|---------------|
| Store backend | **pluggable** | RCS *or* PlainFile behind `Foswiki::Store` — direct-attach target is cleaner on PlainFile (UC-40) |
| History | RCS *or* timestamped copies | PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41) |
| Structured payload | DataForms `%META%`, **multi-record** via MetaDataPlugin | richer than classic TWiki Forms (UC-34/39) |
| Syntax translation | **WysiwygPlugin TML↔HTML** | bidirectional round-trip exists (UC-42) |
| Extension host | `Foswiki::Func` + `registerTagHandler` + REST | cleaner adapter-host surface (UC-38) |
| Store as contract | **`Foswiki::Store` versioned interface** | architectural prior art for shard-wiki's adapter contract |
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
| Observation | INTENT principle |
|-------------|------------------|
| `Foswiki::Store` versioned interface + swappable backends | **capability-aware adapters** / **shard adapter contract** — an engine that already did exactly this separation |
| `Foswiki::Meta` mediates all store access | clean **mechanism over policy** boundary worth mirroring in the adapter contract |
| PlainFile store = timestamped text copies | **git-addressable coordination** / direct-attach (UC-40), history import (UC-41) |
| WysiwygPlugin TML↔HTML round-trip | **Markdown-first, backend-neutral** is achievable for prose via translation (UC-42) |
| MetaDataPlugin multi-record topics | structured pages beyond one form (UC-34/39) |
### 7.2 Deliberate divergences (design bugs if conflated)
| Foswiki assumption | shard-wiki correction |
|--------------------|----------------------|
| One pluggable store per wiki, chosen at config | shard-wiki federates **many** heterogeneous stores at once |
| TML is the canonical syntax; HTML is a render target | shard-wiki is **Markdown-first**; TML↔Markdown is an adapter translation, not core |
| Store interface is Foswiki-internal Perl | shard-wiki's adapter contract is **cross-language, cross-backend, versioned** |
| ACL in topic preferences | **authorize in core**; engine ACL is advisory provenance |
### 7.3 What Foswiki teaches that shard-wiki should not lose
1. **The store-abstraction pattern works in practice** — `Foswiki::Store` is a
real-world proof that a stable interface + swappable backends is viable; the shard
adapter contract is the same idea generalized across engines.
2. **Backend format should be swappable under a stable identity** — Foswiki migrates
RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's
backend changing under it (UC-43).
3. **Lossless syntax round-trip is a solved problem** — don't treat non-Markdown
prose as read-only; translate it (UC-42).
## 8. Use-case seeds → catalog (promoted 2026-06-13)
| Seed | Catalog UC | Disposition |
|------|------------|-------------|
| Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown) | **UC-42** (new) | realizes Markdown-first for prose; WysiwygPlugin is proof |
| Tolerate a shard's storage-backend swap without losing identity/provenance | **UC-43** (new) | Foswiki RCS↔PlainFile; orchestration robustness |
| Structured / multi-record pages | UC-34 / UC-39 | enriched: DataForms + MetaDataPlugin multi-record |
| Attach a file-backed engine's on-disk store directly | UC-40 | enriched: PlainFile store is a cleaner direct-attach target than RCS |
| `Foswiki::Store` versioned interface | (no UC) | architecture prior art → `SHARD-WP-0002` adapter contract |
## 9. Open questions (for spec / workplans)
1. **Adapter-contract shape** — how much of `Foswiki::Store`'s method set
(read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's
capability-aware adapter contract? (architecture, `SHARD-WP-0002`)
2. **Syntax-translation fidelity** — is TML↔Markdown lossless enough for overlay
round-trips, or must overlays be stored in TML to be safe (UC-42)?
3. **Backend-swap detection** — how does shard-wiki notice and tolerate a shard
changing store format underneath (RCS→PlainFile), and does provenance/history
survive it (UC-43)?
4. **Multi-record metadata** — represent MetaDataPlugin multi-record topics and XWiki
multi-XObject pages in one structured-metadata page model (shared open question).
## 10. Sources
| Source | URL |
|--------|-----|
| Foswiki — Technical Overview | https://foswiki.org/Development/TechnicalOverview |
| Foswiki — Core Internals | https://foswiki.org/Development/CoreInternals |
| Foswiki — PlainFileStoreContrib | https://foswiki.org/Extensions/PlainFileStoreContrib |
| Foswiki — RCSStoreContrib | https://foswiki.org/Extensions/RCSStoreContrib |
| Foswiki::Store::PlainFile (source) | https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm |
| Foswiki — Developing Plugins | https://foswiki.org/System/DevelopingPlugins |
| Foswiki — Plugins API Policies | https://foswiki.org/Development/PluginsApiPolicies |
| Foswiki — DataForms | https://foswiki.org/System/DataForms |
| Foswiki — MetaDataPlugin | https://foswiki.org/Extensions/MetaDataPlugin |
| Foswiki — WysiwygPlugin | https://foswiki.org/System/WysiwygPlugin |
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
| shard-wiki — TWiki deep dive | `research/260613-twiki-deep-dive/findings.md` |
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
---
## 11. Traceability
| This document section | Informs (future) |
|-----------------------|------------------|
| §2 store abstraction | **adapter contract** design (`SHARD-WP-0002`) — Foswiki::Store as prior art |
| §3 extension API | UC-38 engine-side adapter host |
| §4 DataForms | structured-metadata page model (UC-34/39) |
| §5 WysiwygPlugin | UC-42 syntax-translation adapter |
| §6 capability profile | adapter capability-profile vocabulary |
| §7 INTENT mapping | architecture-blueprint guardrails |
| §8 UC seeds | `spec/UseCaseCatalog.md` (UC-42, UC-43; UC-34/39/40 enrichment) |

53
spec/UseCaseCatalog.md
View File

@@ -4,8 +4,8 @@ Status: **draft** · Date: 2026-06-08 · Updated: 2026-06-13
Promoted from `research/260608-c2-wiki-origins/`, Promoted from `research/260608-c2-wiki-origins/`,
`research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`, `research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`,
`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`, and `research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`,
`research/260613-twiki-deep-dive/`. `research/260613-twiki-deep-dive/`, and `research/260613-foswiki-deep-dive/`.
See InfoTechPrimers on coulomb.social for use-case catalog conventions. See InfoTechPrimers on coulomb.social for use-case catalog conventions.
## Conventions ## Conventions
@@ -273,6 +273,9 @@ union without inventing a fake Markdown body.
structured pages**. Forces the question: does the page model require a Markdown body, structured pages**. Forces the question: does the page model require a Markdown body,
or can a page be purely typed data (`research/260613-xwiki-deep-dive/findings.md` or can a page be purely typed data (`research/260613-xwiki-deep-dive/findings.md`
§8 Q2)? Affects union views, diff, and search. Deferred to `SHARD-WP-0002`. §8 Q2)? Affects union views, diff, and search. Deferred to `SHARD-WP-0002`.
Foswiki shows a middle point: MetaDataPlugin stores **multiple** structured records
per topic, still as `%META%` in text (`research/260613-foswiki-deep-dive/findings.md`
§4) — the page model must allow N typed records, not one form.
**Priority:** Later **Priority:** Later
### UC-40 — Attach a file-backed engine's on-disk store directly ### UC-40 — Attach a file-backed engine's on-disk store directly
@@ -288,7 +291,10 @@ the on-disk store (offline-capable, git-native for repo-backed engines, but risk
reading inconsistent state under a running engine) vs the runtime API (consistent, reading inconsistent state under a running engine) vs the runtime API (consistent,
respects engine logic, but requires the engine up). Capability profile must express respects engine logic, but requires the engine up). Capability profile must express
the choice (`research/260613-twiki-deep-dive/findings.md` §5–§6, §8 Q1). Distinct the choice (`research/260613-twiki-deep-dive/findings.md` §5–§6, §8 Q1). Distinct
from UC-37 (no live origin) and broader than UC-02's generic attach. from UC-37 (no live origin) and broader than UC-02's generic attach. Foswiki's
**PlainFile** store (timestamped whole-version copies, no RCS) is a cleaner
direct-attach target than RCS diffs (`research/260613-foswiki-deep-dive/findings.md`
§2.2).
**Priority:** Later **Priority:** Later
### UC-41 — Import an engine's native file history into the coordination journal ### UC-41 — Import an engine's native file history into the coordination journal
@@ -305,6 +311,37 @@ is the imported history authoritative or a one-time backfill
(INTENT *History as the safety net*). (INTENT *History as the safety net*).
**Priority:** Later **Priority:** Later
### UC-42 — Read and write a non-Markdown shard via lossless syntax translation
**Actor:** Author or maintainer
**Goal:** Attach a shard whose native markup is not Markdown (TWiki/Foswiki TML,
XWiki syntax) and read it into the Markdown-first model — and write Markdown overlays
back — through **bidirectional, lossless syntax translation**, instead of treating it
as read-only.
**Source:** wikiengines, intent
**Notes:** Realizes *Markdown-first, backend-neutral* for **prose** (UC-34/39 cover
*structure*). Feasibility proven by Foswiki's **WysiwygPlugin** (TML→HTML for editing,
HTML→TML losslessly on save) (`research/260613-foswiki-deep-dive/findings.md` §5).
Open: is round-trip lossless enough, or must overlays be stored in the shard's native
syntax to be safe (findings §9 Q2)? Without this, non-Markdown shards degrade to
read-only (UC-03) — a graceful-degradation floor, not the goal.
**Priority:** Later
### UC-43 — Tolerate a shard's storage-backend swap without losing identity
**Actor:** Maintainer
**Goal:** Keep a shard attached, with its provenance and history intact, when its
underlying storage backend changes — e.g. a Foswiki wiki migrated RCS→PlainFile, or a
folder shard promoted into Git.
**Source:** wikiengines, intent
**Notes:** Foswiki proves backends are swappable under a stable identity via its
versioned `Foswiki::Store` interface (`research/260613-foswiki-deep-dive/findings.md`
§2). shard-wiki orchestration robustness: a shard's identity/attachment is **not** its
storage format. Open: detect the swap and preserve history across it (findings §9 Q3).
Relates to UC-40 (attach path) and UC-41 (history import) but is about *change under a
live attachment*, not initial attach.
**Priority:** Later
--- ---
## B. Knowledge work and collaboration ## B. Knowledge work and collaboration
@@ -525,6 +562,8 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD.
| UC-39 | | | | ✓ | ✓ | | UC-39 | | | | ✓ | ✓ |
| UC-40 | | | | ✓ | ✓ | | UC-40 | | | | ✓ | ✓ |
| UC-41 | | | | ✓ | ✓ | | UC-41 | | | | ✓ | ✓ |
| UC-42 | | | | ✓ | ✓ |
| UC-43 | | | | ✓ | ✓ |
| UC-08 | ✓ | | | | UC-08 | ✓ | | |
| UC-09 | ✓ | | | | UC-09 | ✓ | | |
| UC-10 | ✓ | | | | UC-10 | ✓ | | |
@@ -598,6 +637,8 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD.
| Wiki-as-application-platform, bodiless typed pages (XWiki AppWithinMinutes/XObjects) | UC-39 | | Wiki-as-application-platform, bodiless typed pages (XWiki AppWithinMinutes/XObjects) | UC-39 |
| Attach a file-backed engine's on-disk store directly (TWiki data dir, DokuWiki) | UC-40 | | Attach a file-backed engine's on-disk store directly (TWiki data dir, DokuWiki) | UC-40 |
| Import an engine's native file history (TWiki RCS `.txt,v`) into the journal | UC-41 | | Import an engine's native file history (TWiki RCS `.txt,v`) into the journal | UC-41 |
| Lossless syntax translation for a non-Markdown shard (Foswiki WysiwygPlugin TML↔HTML) | UC-42 |
| Tolerate a shard's storage-backend swap (Foswiki RCS↔PlainFile) under stable identity | UC-43 |
Note: the landscape scan mostly **reinforced** existing UCs and the L0→L4 ladder Note: the landscape scan mostly **reinforced** existing UCs and the L0→L4 ladder
rather than adding scenarios — its primary yield is adapter-contract constraints rather than adding scenarios — its primary yield is adapter-contract constraints
@@ -607,7 +648,11 @@ attachment scenarios it surfaced. UC-38–UC-39 come from the **XWiki deep dive*
(`research/260613-xwiki-deep-dive/findings.md` §7); UC-40–UC-41 from the **TWiki (`research/260613-xwiki-deep-dive/findings.md` §7); UC-40–UC-41 from the **TWiki
deep dive** (`research/260613-twiki-deep-dive/findings.md` §7), which also enriched deep dive** (`research/260613-twiki-deep-dive/findings.md` §7), which also enriched
UC-06 (TWiki per-topic ACL → yawex), UC-34 (file-embedded `%META%`), UC-36 (RCS vs UC-06 (TWiki per-topic ACL → yawex), UC-34 (file-embedded `%META%`), UC-36 (RCS vs
DB history), and UC-38 (TWiki plugin handlers as an adapter host). DB history), and UC-38 (TWiki plugin handlers as an adapter host). UC-42–UC-43 come
from the **Foswiki deep dive** (`research/260613-foswiki-deep-dive/findings.md` §8),
which also enriched UC-39 (MetaDataPlugin multi-record), UC-40 (PlainFile store), and
logged the `Foswiki::Store` versioned interface as **adapter-contract prior art** for
`SHARD-WP-0002` (no UC — architecture).
--- ---