diff --git a/research/260613-foswiki-deep-dive/README.md b/research/260613-foswiki-deep-dive/README.md new file mode 100644 index 0000000..eab4bdf --- /dev/null +++ b/research/260613-foswiki-deep-dive/README.md @@ -0,0 +1,36 @@ +# 260613 — Foswiki deep dive (store abstraction, extension API, ecosystem) + +Date: 2026-06-13 + +## What this is + +A focused study of **Foswiki** — the community fork of TWiki (2008, from TWiki +4.2.x) — concentrating on **what Foswiki changed**, not the lineage it shares with +TWiki. The distinctive material: a **pluggable `Foswiki::Store` backend** +(RcsWrap / RcsLite / **PlainFileStoreContrib**) behind a versioned interface, the +`Foswiki::Meta` topic-object indirection, the OO/MVC core rewrite, the cleaner +`Foswiki::Func` + `registerTagHandler` extension API, **DataForms** (+ MetaDataPlugin, +FlexFormPlugin), and the **WysiwygPlugin** TML↔HTML round-trip. + +Read through shard-wiki's lens: Foswiki is the engine that **already separates its +store backend from its core via a versioned interface** — i.e. it is concrete prior +art for shard-wiki's own *shard adapter contract* and *capability-aware adapters*. + +Pairs with — and deliberately does not repeat — the shared TWiki material: +- `research/260613-twiki-deep-dive/` — flat-file/RCS store, Webs/Topics, plugin + handler callbacks, per-topic ACL (all inherited; read there for the shared parts) +- `research/260613-xwiki-deep-dive/` — the Java/DB app-platform sibling +- `research/260608-wikiengines-overview/` — landscape scan + +## Contents + +| Path | Role | +|------|------| +| `findings.md` | Distinctives vs TWiki, store abstraction, extension API, DataForms, capability profile, INTENT mapping, UC seeds, sources | + +## Status + +Initial deep dive complete. Two new use cases promoted to +`spec/UseCaseCatalog.md` (UC-42, UC-43); UC-34/39/40 enriched. The +`Foswiki::Store`/`Foswiki::Meta` abstraction is logged as adapter-contract prior art +for `workplans/SHARD-WP-0002-federation-architecture.md`. diff --git a/research/260613-foswiki-deep-dive/findings.md b/research/260613-foswiki-deep-dive/findings.md new file mode 100644 index 0000000..b9e245d --- /dev/null +++ b/research/260613-foswiki-deep-dive/findings.md @@ -0,0 +1,188 @@ +# Findings — Foswiki: store abstraction, extension API, ecosystem + +Date: 2026-06-13 · Status: research draft + +Scope: **Foswiki as distinct from TWiki** — only the deltas, since the two share a +data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to +shard-wiki: a **pluggable store backend behind a versioned interface** +(`Foswiki::Store` + `Foswiki::Meta`), an OO/MVC core rewrite, a cleaner extension +API (`Foswiki::Func::registerTagHandler`), enhanced DataForms, and a TML↔HTML +round-trip editor framework. Sources: foswiki.org Development + System docs, +`foswiki/distro` on GitHub. + +**Read first for shared lineage:** `research/260613-twiki-deep-dive/` (flat-file + +RCS store, Webs/Topics, handler callbacks, per-topic `ALLOW/DENY` ACL). This file +does **not** repeat those. + +--- + +## 1. Origin and compatibility + +Forked from **TWiki 4.2.x** in October 2008 by most of the TWiki community (sources +cite 4.2.3–4.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content +compatibility — same TML markup, same plugin API — via a **`TWikiCompatibilityPlugin`** +that maps the `TWiki::` namespace and legacy variables. So everything in the TWiki +dive still applies; Foswiki then grew the features below. + +## 2. The headline delta — a pluggable store behind a versioned interface + +This is the reason Foswiki earns its own dive. + +### 2.1 `Foswiki::Meta` indirection +After Foswiki 1.0.4 the core was changed to **delegate almost all store operations +to a topic object, `Foswiki::Meta`**. Callers manipulate a `Foswiki::Meta` object +representing the item they're changing and **never touch the store implementation +directly**; `Foswiki::Meta` talks to the real store through the well-defined +**`Foswiki::Store`** interface. This is a clean MVC/repository separation. + +### 2.2 Swappable backends +`Foswiki::Store::` ships multiple low-level back-ends behind that one interface: +- **`RcsWrap`** — RCS via the external `rcs` binaries (classic TWiki behavior) +- **`RcsLite`** — a pure-Perl RCS implementation +- **`PlainFileStoreContrib`** — a newer store that saves topics/attachments as + **timestamped copies** instead of RCS diffs: more disk, but **no RCS dependency** + and much higher performance. + +So in Foswiki the **storage format is a configuration choice**, not a fixed property +of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core. + +### 2.3 OO/MVC core rewrite +Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward +object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving +backward compatibility through tests. The store abstraction above is the most +load-bearing result. + +## 3. Extension API deltas + +Shared with TWiki: handler callbacks (`initPlugin`, `commonTagsHandler`, +`before/afterSaveHandler`, `afterRenameHandler`, attachment handlers), REST handlers, +`Config.spec`, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements: + +- **`Foswiki::Func` is the blessed API** — "if there's a `Foswiki::Func` way and + another way, the `Foswiki::Func` way is almost always right." Direct use of + internal packages is discouraged and governed by **`PluginsApiPolicies`**. +- **`Foswiki::Func::registerTagHandler($tag, \&sub)`** — register a custom + macro/variable **programmatically** (cleaner than TWiki's `commonTagsHandler` + string-matching convention). +- **Contrib-defined APIs** carry an explicit "use at your own risk / awaiting merge" + policy — a maturity signal on extension interfaces. +- `EmptyPlugin` (`lib/Foswiki/Plugins/EmptyPlugin.pm`) is the canonical handler + reference/skeleton. + +## 4. DataForms and structured data (delta) + +DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi +value, label) stored as `%META:FIELD%` in the topic text. Foswiki extends the model +via popular extensions: +- **MoreFormfieldsPlugin** — additional special-purpose field types. +- **FlexFormPlugin** — render DataForm interfaces from custom templates + (`Foswiki::Form` classes). +- **MetaDataPlugin** — **multiple structured data records per topic** (beyond the + classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model + while keeping data in the text file. + +## 5. Editing / syntax (delta relevant to "Markdown-first") + +- **WysiwygPlugin** — a generic framework that transforms **TML → HTML** for a + browser editor and **HTML → TML on save** (lossless round-trip). +- **TinyMCEPlugin** — pre-installed WYSIWYG editor built on WysiwygPlugin. + +This proves **bidirectional, lossless translation between an engine's native markup +and another representation is feasible** — directly relevant to shard-wiki reading a +non-Markdown shard and writing Markdown overlays back (UC-42). + +## 6. Foswiki as a shard — capability profile (delta from TWiki) + +| Capability | Foswiki | Note vs TWiki | +|------------|---------|---------------| +| Store backend | **pluggable** | RCS *or* PlainFile behind `Foswiki::Store` — direct-attach target is cleaner on PlainFile (UC-40) | +| History | RCS *or* timestamped copies | PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41) | +| Structured payload | DataForms `%META%`, **multi-record** via MetaDataPlugin | richer than classic TWiki Forms (UC-34/39) | +| Syntax translation | **WysiwygPlugin TML↔HTML** | bidirectional round-trip exists (UC-42) | +| Extension host | `Foswiki::Func` + `registerTagHandler` + REST | cleaner adapter-host surface (UC-38) | +| Store as contract | **`Foswiki::Store` versioned interface** | architectural prior art for shard-wiki's adapter contract | + +## 7. Mapping to shard-wiki INTENT (compare, do not equate) + +### 7.1 Reinforcements +| Observation | INTENT principle | +|-------------|------------------| +| `Foswiki::Store` versioned interface + swappable backends | **capability-aware adapters** / **shard adapter contract** — an engine that already did exactly this separation | +| `Foswiki::Meta` mediates all store access | clean **mechanism over policy** boundary worth mirroring in the adapter contract | +| PlainFile store = timestamped text copies | **git-addressable coordination** / direct-attach (UC-40), history import (UC-41) | +| WysiwygPlugin TML↔HTML round-trip | **Markdown-first, backend-neutral** is achievable for prose via translation (UC-42) | +| MetaDataPlugin multi-record topics | structured pages beyond one form (UC-34/39) | + +### 7.2 Deliberate divergences (design bugs if conflated) +| Foswiki assumption | shard-wiki correction | +|--------------------|----------------------| +| One pluggable store per wiki, chosen at config | shard-wiki federates **many** heterogeneous stores at once | +| TML is the canonical syntax; HTML is a render target | shard-wiki is **Markdown-first**; TML↔Markdown is an adapter translation, not core | +| Store interface is Foswiki-internal Perl | shard-wiki's adapter contract is **cross-language, cross-backend, versioned** | +| ACL in topic preferences | **authorize in core**; engine ACL is advisory provenance | + +### 7.3 What Foswiki teaches that shard-wiki should not lose +1. **The store-abstraction pattern works in practice** — `Foswiki::Store` is a + real-world proof that a stable interface + swappable backends is viable; the shard + adapter contract is the same idea generalized across engines. +2. **Backend format should be swappable under a stable identity** — Foswiki migrates + RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's + backend changing under it (UC-43). +3. **Lossless syntax round-trip is a solved problem** — don't treat non-Markdown + prose as read-only; translate it (UC-42). + +## 8. Use-case seeds → catalog (promoted 2026-06-13) + +| Seed | Catalog UC | Disposition | +|------|------------|-------------| +| Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown) | **UC-42** (new) | realizes Markdown-first for prose; WysiwygPlugin is proof | +| Tolerate a shard's storage-backend swap without losing identity/provenance | **UC-43** (new) | Foswiki RCS↔PlainFile; orchestration robustness | +| Structured / multi-record pages | UC-34 / UC-39 | enriched: DataForms + MetaDataPlugin multi-record | +| Attach a file-backed engine's on-disk store directly | UC-40 | enriched: PlainFile store is a cleaner direct-attach target than RCS | +| `Foswiki::Store` versioned interface | (no UC) | architecture prior art → `SHARD-WP-0002` adapter contract | + +## 9. Open questions (for spec / workplans) + +1. **Adapter-contract shape** — how much of `Foswiki::Store`'s method set + (read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's + capability-aware adapter contract? (architecture, `SHARD-WP-0002`) +2. **Syntax-translation fidelity** — is TML↔Markdown lossless enough for overlay + round-trips, or must overlays be stored in TML to be safe (UC-42)? +3. **Backend-swap detection** — how does shard-wiki notice and tolerate a shard + changing store format underneath (RCS→PlainFile), and does provenance/history + survive it (UC-43)? +4. **Multi-record metadata** — represent MetaDataPlugin multi-record topics and XWiki + multi-XObject pages in one structured-metadata page model (shared open question). + +## 10. Sources + +| Source | URL | +|--------|-----| +| Foswiki — Technical Overview | https://foswiki.org/Development/TechnicalOverview | +| Foswiki — Core Internals | https://foswiki.org/Development/CoreInternals | +| Foswiki — PlainFileStoreContrib | https://foswiki.org/Extensions/PlainFileStoreContrib | +| Foswiki — RCSStoreContrib | https://foswiki.org/Extensions/RCSStoreContrib | +| Foswiki::Store::PlainFile (source) | https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm | +| Foswiki — Developing Plugins | https://foswiki.org/System/DevelopingPlugins | +| Foswiki — Plugins API Policies | https://foswiki.org/Development/PluginsApiPolicies | +| Foswiki — DataForms | https://foswiki.org/System/DataForms | +| Foswiki — MetaDataPlugin | https://foswiki.org/Extensions/MetaDataPlugin | +| Foswiki — WysiwygPlugin | https://foswiki.org/System/WysiwygPlugin | +| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork | +| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki | +| shard-wiki — TWiki deep dive | `research/260613-twiki-deep-dive/findings.md` | +| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` | + +--- + +## 11. Traceability + +| This document section | Informs (future) | +|-----------------------|------------------| +| §2 store abstraction | **adapter contract** design (`SHARD-WP-0002`) — Foswiki::Store as prior art | +| §3 extension API | UC-38 engine-side adapter host | +| §4 DataForms | structured-metadata page model (UC-34/39) | +| §5 WysiwygPlugin | UC-42 syntax-translation adapter | +| §6 capability profile | adapter capability-profile vocabulary | +| §7 INTENT mapping | architecture-blueprint guardrails | +| §8 UC seeds | `spec/UseCaseCatalog.md` (UC-42, UC-43; UC-34/39/40 enrichment) | diff --git a/spec/UseCaseCatalog.md b/spec/UseCaseCatalog.md index 6559c48..53d7902 100644 --- a/spec/UseCaseCatalog.md +++ b/spec/UseCaseCatalog.md @@ -4,8 +4,8 @@ Status: **draft** · Date: 2026-06-08 · Updated: 2026-06-13 Promoted from `research/260608-c2-wiki-origins/`, `research/260608-yawex-prior-art/`, `research/260608-federation-concepts/`, -`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`, and -`research/260613-twiki-deep-dive/`. +`research/260608-wikiengines-overview/`, `research/260613-xwiki-deep-dive/`, +`research/260613-twiki-deep-dive/`, and `research/260613-foswiki-deep-dive/`. See InfoTechPrimers on coulomb.social for use-case catalog conventions. ## Conventions @@ -272,7 +272,10 @@ union without inventing a fake Markdown body. **Notes:** The extreme of UC-34 — not *annotations on prose* but **bodiless structured pages**. Forces the question: does the page model require a Markdown body, or can a page be purely typed data (`research/260613-xwiki-deep-dive/findings.md` -§8 Q2)? Affects union views, diff, and search. Deferred to `SHARD-WP-0002`. +§8 Q2)? Affects union views, diff, and search. Deferred to `SHARD-WP-0002`. +Foswiki shows a middle point: MetaDataPlugin stores **multiple** structured records +per topic, still as `%META%` in text (`research/260613-foswiki-deep-dive/findings.md` +§4) — the page model must allow N typed records, not one form. **Priority:** Later ### UC-40 — Attach a file-backed engine's on-disk store directly @@ -288,7 +291,10 @@ the on-disk store (offline-capable, git-native for repo-backed engines, but risk reading inconsistent state under a running engine) vs the runtime API (consistent, respects engine logic, but requires the engine up). Capability profile must express the choice (`research/260613-twiki-deep-dive/findings.md` §5–§6, §8 Q1). Distinct -from UC-37 (no live origin) and broader than UC-02's generic attach. +from UC-37 (no live origin) and broader than UC-02's generic attach. Foswiki's +**PlainFile** store (timestamped whole-version copies, no RCS) is a cleaner +direct-attach target than RCS diffs (`research/260613-foswiki-deep-dive/findings.md` +§2.2). **Priority:** Later ### UC-41 — Import an engine's native file history into the coordination journal @@ -305,6 +311,37 @@ is the imported history authoritative or a one-time backfill (INTENT *History as the safety net*). **Priority:** Later +### UC-42 — Read and write a non-Markdown shard via lossless syntax translation + +**Actor:** Author or maintainer +**Goal:** Attach a shard whose native markup is not Markdown (TWiki/Foswiki TML, +XWiki syntax) and read it into the Markdown-first model — and write Markdown overlays +back — through **bidirectional, lossless syntax translation**, instead of treating it +as read-only. +**Source:** wikiengines, intent +**Notes:** Realizes *Markdown-first, backend-neutral* for **prose** (UC-34/39 cover +*structure*). Feasibility proven by Foswiki's **WysiwygPlugin** (TML→HTML for editing, +HTML→TML losslessly on save) (`research/260613-foswiki-deep-dive/findings.md` §5). +Open: is round-trip lossless enough, or must overlays be stored in the shard's native +syntax to be safe (findings §9 Q2)? Without this, non-Markdown shards degrade to +read-only (UC-03) — a graceful-degradation floor, not the goal. +**Priority:** Later + +### UC-43 — Tolerate a shard's storage-backend swap without losing identity + +**Actor:** Maintainer +**Goal:** Keep a shard attached, with its provenance and history intact, when its +underlying storage backend changes — e.g. a Foswiki wiki migrated RCS→PlainFile, or a +folder shard promoted into Git. +**Source:** wikiengines, intent +**Notes:** Foswiki proves backends are swappable under a stable identity via its +versioned `Foswiki::Store` interface (`research/260613-foswiki-deep-dive/findings.md` +§2). shard-wiki orchestration robustness: a shard's identity/attachment is **not** its +storage format. Open: detect the swap and preserve history across it (findings §9 Q3). +Relates to UC-40 (attach path) and UC-41 (history import) but is about *change under a +live attachment*, not initial attach. +**Priority:** Later + --- ## B. Knowledge work and collaboration @@ -525,6 +562,8 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD. | UC-39 | | | | ✓ | ✓ | | UC-40 | | | | ✓ | ✓ | | UC-41 | | | | ✓ | ✓ | +| UC-42 | | | | ✓ | ✓ | +| UC-43 | | | | ✓ | ✓ | | UC-08 | ✓ | | | | UC-09 | ✓ | | | | UC-10 | ✓ | | | @@ -598,6 +637,8 @@ CamelCase and `[[free links]]`. Markdown-first link semantics TBD. | Wiki-as-application-platform, bodiless typed pages (XWiki AppWithinMinutes/XObjects) | UC-39 | | Attach a file-backed engine's on-disk store directly (TWiki data dir, DokuWiki) | UC-40 | | Import an engine's native file history (TWiki RCS `.txt,v`) into the journal | UC-41 | +| Lossless syntax translation for a non-Markdown shard (Foswiki WysiwygPlugin TML↔HTML) | UC-42 | +| Tolerate a shard's storage-backend swap (Foswiki RCS↔PlainFile) under stable identity | UC-43 | Note: the landscape scan mostly **reinforced** existing UCs and the L0→L4 ladder rather than adding scenarios — its primary yield is adapter-contract constraints @@ -607,7 +648,11 @@ attachment scenarios it surfaced. UC-38–UC-39 come from the **XWiki deep dive* (`research/260613-xwiki-deep-dive/findings.md` §7); UC-40–UC-41 from the **TWiki deep dive** (`research/260613-twiki-deep-dive/findings.md` §7), which also enriched UC-06 (TWiki per-topic ACL → yawex), UC-34 (file-embedded `%META%`), UC-36 (RCS vs -DB history), and UC-38 (TWiki plugin handlers as an adapter host). +DB history), and UC-38 (TWiki plugin handlers as an adapter host). UC-42–UC-43 come +from the **Foswiki deep dive** (`research/260613-foswiki-deep-dive/findings.md` §8), +which also enriched UC-39 (MetaDataPlugin multi-record), UC-40 (PlainFile store), and +logged the `Foswiki::Store` versioned interface as **adapter-contract prior art** for +`SHARD-WP-0002` (no UC — architecture). ---