generated from coulomb/repo-seed
Compare commits
170 Commits
1c0c0efc56
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 776f5af5a7 | |||
| fd961c83b4 | |||
| cca5bf83c3 | |||
| def699c1eb | |||
| a4e0f52ec1 | |||
| 4231daf94f | |||
| 37681d89b6 | |||
| a8e65235a8 | |||
| d7d046cac0 | |||
| 0b3ab2086f | |||
| d85d019543 | |||
| 3a5acdcb28 | |||
| 34b0c539f3 | |||
| da540d4eea | |||
| 951b24300d | |||
| c731c96634 | |||
| f0fee65cc0 | |||
| 34432c2e15 | |||
| 45a858ead0 | |||
| b31e9bc337 | |||
| e50dcc6b5c | |||
| a165cced33 | |||
| 8393a9c55d | |||
| ff96ee0c48 | |||
| 8b353f1077 | |||
| b9bb1f7d10 | |||
| c40fa3c934 | |||
| 54c2bf2ae5 | |||
| 6d8bd837a4 | |||
| b48a99d3c2 | |||
| 9b7f86ba69 | |||
| 74142096d0 | |||
| 2100e956aa | |||
| e62560eb5a | |||
| b147d3e831 | |||
| cdcf4b09aa | |||
| b21efe307b | |||
| e18397272a | |||
| 0ee972f2e2 | |||
| bb1b54e0af | |||
| b70f1c9acc | |||
| 8de044bbde | |||
| 67d851be0b | |||
| 46ec6f2a5f | |||
| ad4a2dbf5a | |||
| 23bc597343 | |||
| ec7a1ec946 | |||
| d6d7cc555f | |||
| 1dd1def40f | |||
| 6d341cd4e6 | |||
| 1b3d4aaa39 | |||
| 6e49bd3a4b | |||
| d2c73b02d9 | |||
| 042e198286 | |||
| 3ea0cc1226 | |||
| 4be2f190a0 | |||
| 7d00ae758e | |||
| 715ab1ca00 | |||
| d797bc5ee4 | |||
| 92d5774baf | |||
| e24f0034a0 | |||
| b2ea276c00 | |||
| 9fedb31d8b | |||
| 517cf1d282 | |||
| b44b2a74a4 | |||
| 24108b65aa | |||
| 6d48a1d3e6 | |||
| 9a4e00a05a | |||
| 5a77ea879c | |||
| aca9bf30f9 | |||
| 3b4f10a349 | |||
| f2dd2e124a | |||
| 1f3aba7aad | |||
| d65f9e21f3 | |||
| 802a80231a | |||
| 40575045ca | |||
| a2f15806f5 | |||
| 9b5fceb451 | |||
| d39e7f7765 | |||
| 1ad70a9c8a | |||
| 3a753a6f3b | |||
| 08a2148079 | |||
| cbd29e0a32 | |||
| 079652b61f | |||
| 84ab25eeb9 | |||
| 229d24d1cf | |||
| f21b7b5259 | |||
| c895d33091 | |||
| 59c36ac9d1 | |||
| 012d151fe8 | |||
| af840576e9 | |||
| 0383d78440 | |||
| dc451b0f4e | |||
| 04be66161e | |||
| c8dbe9b573 | |||
| b5d2cbc330 | |||
| ee2449d987 | |||
| f1dc7aff61 | |||
| dd812abb81 | |||
| 9b5b393519 | |||
| 6a88eb7710 | |||
| d898307b7e | |||
| 90a854f841 | |||
| 3d137c96b6 | |||
| f2f9f31df8 | |||
| 2c978d71f0 | |||
| b676579407 | |||
| 7fb90219b5 | |||
| 56b6cdd110 | |||
| 4acadacfee | |||
| 1546ca09bf | |||
| a3ca3e975b | |||
| 25091dbd2e | |||
| 1e3aabc143 | |||
| 25a714efa7 | |||
| 3eb026139f | |||
| 5ba6975573 | |||
| d4afce3699 | |||
| 2397c2c4b3 | |||
| a7084ec74b | |||
| 700566b1e2 | |||
| af5841a0d7 | |||
| 8941e131e8 | |||
| ce4b0cb9e4 | |||
| 6878a0c184 | |||
| 87f30e7c62 | |||
| e0c3d0699c | |||
| 1e962a0aef | |||
| 54468b60bc | |||
| 9f0fc57428 | |||
| ee805d5af7 | |||
| 188643b0f5 | |||
| fee47514a2 | |||
| 1281ca689b | |||
| dbc42f05db | |||
| 31ec8ab205 | |||
| a6629bdb29 | |||
| ecb7d0154e | |||
| d955fa239c | |||
| 036dbad816 | |||
| 9acd4e2841 | |||
| 5ada333bb4 | |||
| 9b5defa6d8 | |||
| 4543f5dc43 | |||
| 714eaf6b5a | |||
| 5697669063 | |||
| 3e0c05a2b3 | |||
| 6ccf349209 | |||
| e00c160a43 | |||
| 26bee2cc84 | |||
| fae3cbde8b | |||
| 64796f7b3a | |||
| f0be5799aa | |||
| dfff9ab42e | |||
| 7c22c468af | |||
| c1424e8863 | |||
| 0a335eca8b | |||
| 60e1aa661c | |||
| 316fdeedc2 | |||
| fa7adab239 | |||
| ffd5459b3e | |||
| 27da940df1 | |||
| 0473c3f043 | |||
| 7d1fb97968 | |||
| 81cfce10d2 | |||
| 969dae23b8 | |||
| f1384144eb | |||
| 3160648b61 | |||
| 5d8608aeae | |||
| 64b77f30c1 |
20
.claude/rules/agents.md
Normal file
20
.claude/rules/agents.md
Normal file
@@ -0,0 +1,20 @@
|
||||
## Kaizen Agents
|
||||
|
||||
Specialized agent personas available on demand via the state-hub MCP.
|
||||
|
||||
**Discover:** `list_kaizen_agents()` — returns all agents with name, description, category
|
||||
**Load:** `get_kaizen_agent("tdd-workflow")` — returns full instructions; read and follow them
|
||||
|
||||
Common agents:
|
||||
|
||||
| Agent | Category | When to use |
|
||||
|-------|----------|-------------|
|
||||
| `tdd-workflow` | testing | Step-by-step TDD8 workflow for any feature |
|
||||
| `code-refactoring` | quality | Code quality analysis and safe refactoring |
|
||||
| `test-maintenance` | testing | Diagnose and fix failing tests |
|
||||
| `requirements-engineering` | process | Prevent interface/mock mismatches upfront |
|
||||
| `keepaTodofile` | process | Maintain TODO.md during work |
|
||||
| `project-management` | process | Track status, determine next steps |
|
||||
| `datamodel-optimization` | quality | Optimize dataclasses and data structures |
|
||||
|
||||
All 17 agents: call `list_kaizen_agents()` for the full list.
|
||||
8
.claude/rules/architecture.md
Normal file
8
.claude/rules/architecture.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Architecture
|
||||
|
||||
<!-- TODO: Describe the key design decisions and component structure.
|
||||
Key modules, data flows, external integrations, state machines, etc. -->
|
||||
|
||||
## Quick Reference
|
||||
|
||||
`~/state-hub/mcp_server/TOOLS.md` — MCP tool reference
|
||||
50
.claude/rules/credential-routing.md
Normal file
50
.claude/rules/credential-routing.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=shard-wiki` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
38
.claude/rules/first-session.md
Normal file
38
.claude/rules/first-session.md
Normal file
@@ -0,0 +1,38 @@
|
||||
## First Session Protocol
|
||||
|
||||
Triggered when `get_domain_summary("consumer")` shows **no workstreams**.
|
||||
The project is registered but work has not yet been structured.
|
||||
|
||||
**Step 1 — Read, don't write**
|
||||
- `~/the-custodian/canon/projects/consumer/project_charter_v0.1.md` — purpose, scope
|
||||
- `~/the-custodian/canon/projects/consumer/roadmap_v0.1.md` — planned phases
|
||||
- Scan repo root: README, directory structure, existing code or docs
|
||||
|
||||
**Step 2 — Survey in-progress work**
|
||||
Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete.
|
||||
|
||||
**Step 3 — Propose workstreams to Bernd**
|
||||
Propose 1–3 workstreams — each a coherent strand, weeks to months, anchored to a
|
||||
roadmap phase. **Wait for approval before creating.**
|
||||
|
||||
**Step 4 — Create workplan file first, then DB record (ADR-001)**
|
||||
```
|
||||
workplans/SHARD-WP-NNNN-<slug>.md ← write this first
|
||||
```
|
||||
Then register in the hub:
|
||||
```
|
||||
create_workstream(topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0", title="...", owner="...", description="...")
|
||||
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
|
||||
```
|
||||
|
||||
**Step 5 — Record the setup**
|
||||
```
|
||||
add_progress_event(
|
||||
summary="First session: structured consumer into N workstreams, M tasks",
|
||||
event_type="milestone",
|
||||
topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0",
|
||||
detail={"workstreams": [...], "tasks_created": M}
|
||||
)
|
||||
```
|
||||
|
||||
<!-- Delete or archive this file once past first session -->
|
||||
8
.claude/rules/repo-boundary.md
Normal file
8
.claude/rules/repo-boundary.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Repo boundary
|
||||
|
||||
This repo owns **shard-wiki** only. It does not own:
|
||||
|
||||
<!-- TODO: List what belongs in adjacent repos, e.g.:
|
||||
- SSH key management → railiance-infra/
|
||||
- State hub code → state-hub/
|
||||
-->
|
||||
5
.claude/rules/repo-identity.md
Normal file
5
.claude/rules/repo-identity.md
Normal file
@@ -0,0 +1,5 @@
|
||||
**Purpose:** Git-based Markdown wiki orchestrator and federation layer. Python (src/ layout, hatchling, pytest). Early-stage: scaffold + INTENT.md defined, domain model not yet implemented. See INTENT.md for authoritative scope.
|
||||
|
||||
**Domain:** consumer
|
||||
**Repo slug:** shard-wiki
|
||||
**Topic ID:** 4c2e5315-2cb9-447c-9d16-a39bdb0aabd0
|
||||
85
.claude/rules/session-protocol.md
Normal file
85
.claude/rules/session-protocol.md
Normal file
@@ -0,0 +1,85 @@
|
||||
## Session Protocol
|
||||
|
||||
Dev Hub (State Hub API): http://127.0.0.1:8000
|
||||
MCP server name in `~/.claude.json`: `dev-hub`
|
||||
|
||||
**Step 1 — Orient**
|
||||
|
||||
Read the offline-safe brief first — it works without a live hub connection:
|
||||
```bash
|
||||
cat .custodian-brief.md
|
||||
```
|
||||
Then call the MCP tool for richer cross-domain context when MCP tools are exposed:
|
||||
```
|
||||
get_domain_summary("consumer")
|
||||
```
|
||||
If MCP tools are unavailable in the current agent session, use the REST API:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
|
||||
```
|
||||
If the hub is offline: `cd ~/state-hub && make api`
|
||||
|
||||
**Step 2 — Check inbox**
|
||||
With MCP tools:
|
||||
```
|
||||
get_messages(to_agent="shard-wiki", unread_only=True)
|
||||
```
|
||||
Mark read with `mark_message_read(message_id)`. Reply or act on coordination
|
||||
requests before proceeding.
|
||||
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=shard-wiki&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
**Step 3 — Scan workplans**
|
||||
```bash
|
||||
ls workplans/
|
||||
```
|
||||
For each file with `status: ready`, `active`, or `blocked`, note pending
|
||||
`wait`/`todo`/`progress` tasks.
|
||||
|
||||
**Step 4 — Present brief**
|
||||
|
||||
1. **Active workstreams** for `consumer` — title, task counts, blocking decisions
|
||||
2. **Pending tasks** from `workplans/` + any `[repo:shard-wiki]` hub tasks
|
||||
3. **Goal guidance** — if `goal_guidance` in summary:
|
||||
- `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"*
|
||||
- `alignment_warnings`: flag if active work is not aligned with current goal
|
||||
4. **Suggested next action** — highest-priority open item
|
||||
5. **SBOM status** — flag if `last_sbom_at` is unset for this repo
|
||||
|
||||
If no workstreams: follow First Session Protocol (`first-session.md`).
|
||||
|
||||
**During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()`
|
||||
|
||||
> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`)
|
||||
> are First Session Protocol only. Work structure belongs in repo files (ADR-001).
|
||||
|
||||
**Session close:**
|
||||
With MCP tools:
|
||||
```
|
||||
add_progress_event(summary="...", topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0", workstream_id="<uuid>")
|
||||
```
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"topic_id":"4c2e5315-2cb9-447c-9d16-a39bdb0aabd0","workstream_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
|
||||
```
|
||||
If workplan files were modified, ensure the local copy is up to date first:
|
||||
```bash
|
||||
git -C <repo_path> pull --ff-only
|
||||
cd ~/state-hub && make fix-consistency REPO=shard-wiki
|
||||
```
|
||||
For repos where implementation runs on a remote machine (e.g. CoulombCore),
|
||||
use the combined target which pulls before fixing:
|
||||
```bash
|
||||
cd ~/state-hub && make fix-consistency-remote REPO=shard-wiki
|
||||
```
|
||||
**C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback
|
||||
will sync the file to match DB. **C-16** (repo behind remote) blocks all writes
|
||||
until you pull — intentional to prevent clobbering remote progress.
|
||||
19
.claude/rules/stack-and-commands.md
Normal file
19
.claude/rules/stack-and-commands.md
Normal file
@@ -0,0 +1,19 @@
|
||||
## Stack
|
||||
|
||||
<!-- TODO: Fill in language, frameworks, and key dependencies -->
|
||||
- **Language:**
|
||||
- **Key deps:**
|
||||
|
||||
## Dev Commands
|
||||
|
||||
```bash
|
||||
# TODO: Fill in the standard commands for this repo
|
||||
|
||||
# Install dependencies
|
||||
|
||||
# Run tests
|
||||
|
||||
# Lint / type check
|
||||
|
||||
# Build / package (if applicable)
|
||||
```
|
||||
40
.claude/rules/workplan-convention.md
Normal file
40
.claude/rules/workplan-convention.md
Normal file
@@ -0,0 +1,40 @@
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
File location: `workplans/SHARD-WP-NNNN-<slug>.md`
|
||||
ID prefix: `SHARD-WP-`
|
||||
|
||||
Work items originate as files in this repo **before** being registered in the hub.
|
||||
|
||||
Canonical workplan/workstream frontmatter statuses are:
|
||||
`proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`.
|
||||
Use `proposed` for a newly drafted plan, `ready` after review against current
|
||||
repo state, and `finished` when implementation is complete. `stalled` and
|
||||
`needs_review` are derived health labels, not stored statuses.
|
||||
|
||||
Closed workplans may be moved to `workplans/archived/` with a completion-date
|
||||
prefix: `YYMMDD-SHARD-WP-NNNN-<slug>.md`. The frontmatter id remains
|
||||
unchanged; the prefix is only for quick visual reference.
|
||||
|
||||
Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**:
|
||||
`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids
|
||||
`ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed
|
||||
directly. Promote anything requiring analysis, design, approval, dependencies, or
|
||||
multiple planned phases into a normal workplan.
|
||||
|
||||
Ecosystem todos from other agents arrive as `[repo:shard-wiki]` hub tasks —
|
||||
visible at session start. Pick one up by creating the workplan file, then registering
|
||||
the workstream.
|
||||
|
||||
Task blocks use this shape:
|
||||
|
||||
```task
|
||||
id: SHARD-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
```
|
||||
|
||||
Status progression is `todo` → `progress` → `done`; use `wait` for waiting or
|
||||
blocked work and `cancel` for stopped work.
|
||||
|
||||
<!-- Ralph Loop rules and HEUREKA sequence: ~/.claude/CLAUDE.md — do not duplicate here -->
|
||||
@@ -2,21 +2,46 @@
|
||||
# Custodian Brief — shard-wiki
|
||||
|
||||
**Domain:** whynot
|
||||
**Last synced:** 2026-06-08 10:24 UTC
|
||||
**Last synced:** 2026-06-15 22:57 UTC
|
||||
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
|
||||
|
||||
## Active Workstreams
|
||||
|
||||
### shard-wiki requirements from yawex prior art
|
||||
Progress: 0/6 done | workstream_id: `0ed023a2-760b-4990-b931-8ee1f41ea08f`
|
||||
### second adapter — git-IS-store shard (contract validation on a new substrate)
|
||||
Progress: 0/3 done | workstream_id: `9e24eeb0-c0f0-41e6-a1ca-88d71e4139ea`
|
||||
|
||||
**Open tasks:**
|
||||
- · Design federation page-resolution model (yawex state space as inspiration) `ebc036e4`
|
||||
- · Define namespace/path model and page+shard roles `431b4d28`
|
||||
- · Specify union-level derived views (BackLinks, RecentChanges, AllPages, SiteMap, Search) `564545ec`
|
||||
- · Provenance & freshness model for pages/revisions/projections `738326f5`
|
||||
- · Overlay / lightweight-patch model (from yawex append/comment) `a268de6a`
|
||||
- · Markdown link semantics: wikilink + red-link extension `a7499f3e`
|
||||
- · GitShardAdapter — read over a git working tree/repo `8a1c7c80`
|
||||
- · Write = commit; current_rev = sha (drift) `b47dfb86`
|
||||
- · History adopt + integration with union/overlay `4c895f42`
|
||||
|
||||
### incremental union maintenance + equivalence index + I-2 verification
|
||||
Progress: 0/4 done | workstream_id: `78d48bcf-6482-4266-bc81-084b7ec1cd80`
|
||||
|
||||
**Open tasks:**
|
||||
- · Equivalence index: blocking + verify `842f480b`
|
||||
- · Incremental maintenance (delta, not additive) `2da4e0b8`
|
||||
- · I-2 verification: digest + consistency-checker `b602ce31`
|
||||
- · Wire incremental tier behind resolution + views `2f3d083c`
|
||||
|
||||
### derived views — wikilinks, BackLinks, RecentChanges, AllPages/SiteMap
|
||||
Progress: 0/5 done | workstream_id: `2fe15330-ddf6-4b0f-8e55-ada341375d35`
|
||||
|
||||
**Open tasks:**
|
||||
- · Wikilink + red-link model `792660c3`
|
||||
- · BackLinks (core) `431a54c3`
|
||||
- · RecentChanges (core) `270c1c31`
|
||||
- · AllPages / SiteMap (core) `898ba43e`
|
||||
- · Wiring + integration `7157544b`
|
||||
|
||||
### git-backed DecisionLog + per-space append authority
|
||||
Progress: 0/4 done | workstream_id: `4fb5b29b-955c-4f37-85cf-58b4643ab1ca`
|
||||
|
||||
**Open tasks:**
|
||||
- · Git event-store backend (append = commit/object) `a8fcbb3e`
|
||||
- · Per-space append authority (lease) `62abd162`
|
||||
- · Fold over the git log + read-your-writes across processes `8cc3691e`
|
||||
- · Migration + wiring `281e1db4`
|
||||
|
||||
---
|
||||
## MCP Orientation (when available)
|
||||
|
||||
17
.repo-classification.yaml
Normal file
17
.repo-classification.yaml
Normal file
@@ -0,0 +1,17 @@
|
||||
repo_classification:
|
||||
standard: Repo Classification Standard
|
||||
version: '1.0'
|
||||
classified_at: '2026-06-22'
|
||||
classified_by: agent
|
||||
category: project
|
||||
domain: consumer
|
||||
secondary_domains: []
|
||||
capability_tags:
|
||||
- knowledge
|
||||
- documentation
|
||||
business_stake:
|
||||
- product
|
||||
- experience
|
||||
business_mechanics:
|
||||
- coordination
|
||||
- operation
|
||||
243
AGENTS.md
243
AGENTS.md
@@ -1,62 +1,219 @@
|
||||
# AGENTS.md
|
||||
# shard-wiki — Agent Instructions
|
||||
|
||||
Guidance for agents working in `shard-wiki`.
|
||||
## Repo Identity
|
||||
|
||||
## Read First
|
||||
**Purpose:** Git-based Markdown wiki orchestrator and federation layer. Python (src/ layout, hatchling, pytest). Early-stage: scaffold + INTENT.md defined, domain model not yet implemented. See INTENT.md for authoritative scope.
|
||||
|
||||
1. `INTENT.md` — aspiration and boundaries (stable; architectural changes are rare).
|
||||
2. `SCOPE.md` — what we are achieving now and current maturity.
|
||||
3. `.custodian-brief.md` — State Hub snapshot (generated; do not edit manually).
|
||||
**Domain:** consumer
|
||||
**Repo slug:** shard-wiki
|
||||
**Topic ID:** `4c2e5315-2cb9-447c-9d16-a39bdb0aabd0`
|
||||
**Workplan prefix:** `SHARD-WP-`
|
||||
|
||||
## Documentation Layout
|
||||
---
|
||||
|
||||
This repo follows the CoulombSocial / HelixForge / MarkiTect documentation
|
||||
layout (recommendation, not strict law). Efficient retrieval by purpose:
|
||||
## State Hub Integration
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `INTENT.md` | Aspiration and boundaries |
|
||||
| `SCOPE.md` | Top-level view of current achievement; closes gap to INTENT |
|
||||
| `research/` | Exploration results (`yymmdd-` prefix on files or subdirs) |
|
||||
| `demand/` | Inbound requests not yet reviewed into spec or workplans |
|
||||
| `spec/` | Implementation guardrails (PRD, TSD, use cases, architecture) |
|
||||
| `workplans/` | State Hub–registered implementation tasks |
|
||||
| `docs/` | Stakeholder documentation (users, developers, humans, agents) |
|
||||
| `wiki/` | Perspective-free interconnected knowledge (wiki UI when connected) |
|
||||
| `issues/` | Mirror of relevant open tickets when ticket systems are in use |
|
||||
| `history/` | Archived material (`yymmdd-` prefix); out of scope for daily work |
|
||||
The Custodian State Hub tracks work across all domains. Interact via HTTP REST —
|
||||
there is no MCP server for Codex agents.
|
||||
|
||||
**Mode of operation:** close SCOPE → INTENT while learning; refine both as needed.
|
||||
| Context | URL |
|
||||
|---------|-----|
|
||||
| Local workstation | `http://127.0.0.1:8000` |
|
||||
| Remote via tunnel | `http://127.0.0.1:18000` |
|
||||
|
||||
## Domain Vocabulary
|
||||
|
||||
Honor terms from `INTENT.md`: shard, root entity, adapter contract, projection,
|
||||
overlay, coordination journal, shard modes. Do not invent parallel vocabulary.
|
||||
|
||||
## Build And Test
|
||||
### Orient at session start
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]"
|
||||
pytest
|
||||
ruff check
|
||||
ruff format
|
||||
# Offline brief — works without hub connection
|
||||
cat .custodian-brief.md
|
||||
|
||||
# Active workstreams for this domain
|
||||
curl -s "http://127.0.0.1:8000/workstreams/?topic_id=4c2e5315-2cb9-447c-9d16-a39bdb0aabd0&status=active" \
|
||||
| python3 -m json.tool
|
||||
|
||||
# Check inbox
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=shard-wiki&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
## State Hub
|
||||
Mark a message read:
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
Workplans register with State Hub. After workplan changes:
|
||||
### Log progress (required at session close)
|
||||
|
||||
```bash
|
||||
cd ~/state-hub && make fix-consistency REPO=shard-wiki
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"summary": "what was done",
|
||||
"event_type": "note",
|
||||
"author": "codex",
|
||||
"workstream_id": "<uuid>",
|
||||
"task_id": "<uuid>"
|
||||
}'
|
||||
```
|
||||
|
||||
Finished or canceled workplans move to `history/` with a `yymmdd-` archive prefix.
|
||||
Omit `workstream_id` / `task_id` when not applicable.
|
||||
|
||||
## Where To Put New Material
|
||||
### Update task status
|
||||
|
||||
- Exploratory analysis → `research/yymmdd-<topic>/`
|
||||
- Raw feature ask or external requirement → `demand/`
|
||||
- Reviewed design ready to guide code → `spec/`
|
||||
- Implementation tasks → `workplans/`
|
||||
- User/dev/agent how-to → `docs/`
|
||||
- Collaborative unstructured notes → `wiki/`
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"status": "progress"}'
|
||||
# values: wait | todo | progress | done | cancel
|
||||
```
|
||||
|
||||
### Flag a task for human review
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"needs_human": true, "intervention_note": "reason"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Session Protocol
|
||||
|
||||
**Start:**
|
||||
1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe)
|
||||
2. Check inbox: `GET /messages/?to_agent=shard-wiki&unread_only=true`; mark read
|
||||
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
|
||||
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
|
||||
|
||||
**During work:**
|
||||
- Update task statuses in workplan files as tasks progress
|
||||
- Record significant decisions via `POST /decisions/`
|
||||
|
||||
**Close:**
|
||||
1. Update workplan file task statuses to reflect progress
|
||||
2. Log: `POST /progress/` with a summary of what changed
|
||||
3. Note for the custodian operator: after workplan file changes, run from
|
||||
`~/state-hub`:
|
||||
```bash
|
||||
make fix-consistency REPO=shard-wiki
|
||||
```
|
||||
This syncs task status from files into the hub DB.
|
||||
|
||||
---
|
||||
|
||||
## Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=shard-wiki` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
|
||||
<!-- REPO-AGENTS-EXTENSIONS -->
|
||||
<!-- Append repo-specific agent instructions below this marker.
|
||||
The state-hub template sync preserves content after this line. -->
|
||||
|
||||
---
|
||||
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
Work items originate as files in this repo — not in the hub. The hub is a
|
||||
read/cache/index layer that rebuilds from files.
|
||||
|
||||
**File location:** `workplans/SHARD-WP-NNNN-<slug>.md`
|
||||
|
||||
**Archived location:** finished workplans may move to
|
||||
`workplans/archived/YYMMDD-SHARD-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
|
||||
the completion/archive date; the frontmatter `id` does not change.
|
||||
|
||||
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
|
||||
`workplans/ADHOC-YYYY-MM-DD.md` with task ids `ADHOC-YYYY-MM-DD-T01`, etc. Use
|
||||
this only for low-risk work completed directly; create a normal workplan for
|
||||
anything needing analysis, design, approval, dependencies, or multiple phases.
|
||||
|
||||
**Frontmatter:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: SHARD-WP-NNNN
|
||||
type: workplan
|
||||
title: "..."
|
||||
domain: consumer
|
||||
repo: shard-wiki
|
||||
status: proposed | ready | active | blocked | backlog | finished | archived
|
||||
owner: codex
|
||||
topic_slug: ...
|
||||
created: "YYYY-MM-DD"
|
||||
updated: "YYYY-MM-DD"
|
||||
state_hub_workstream_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
---
|
||||
```
|
||||
|
||||
Use `proposed` for a new draft, `ready` after review against current repo
|
||||
state, and `finished` after implementation. `stalled` and `needs_review` are
|
||||
derived health labels, not frontmatter statuses.
|
||||
|
||||
**Task block format** (one per `##` section):
|
||||
|
||||
```
|
||||
## Task Title
|
||||
|
||||
` ` `task
|
||||
id: SHARD-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
` ` `
|
||||
|
||||
Task description text.
|
||||
```
|
||||
|
||||
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
|
||||
|
||||
To create a new workplan:
|
||||
1. Write the file following the format above
|
||||
2. Notify the custodian operator to run `make fix-consistency REPO=shard-wiki`
|
||||
(or send a message to the hub agent via `POST /messages/`)
|
||||
|
||||
63
CLAUDE.md
63
CLAUDE.md
@@ -1,53 +1,12 @@
|
||||
# CLAUDE.md
|
||||
# shard-wiki — Claude Code Instructions
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Repository status
|
||||
|
||||
This is an **early-stage Python repository**. The package scaffold (`src/shard_wiki/`, `tests/`, `pyproject.toml`) exists with only smoke tests — the domain model is not yet implemented. Read `INTENT.md` (aspiration), `SCOPE.md` (current achievement), and `AGENTS.md` (layout and conventions) before designing anything. Close the gap from SCOPE to INTENT via `research/`, `spec/`, and `workplans/`.
|
||||
|
||||
## What this project is
|
||||
|
||||
`shard-wiki` is a **Git-based Markdown wiki orchestrator and federation layer**, not a wiki engine. It lets multiple heterogeneous wiki-shaped page stores (**shards**) attach to a shared root entity and be presented as a **union of pages**, while preserving each shard's separate storage, provenance, capabilities, and history.
|
||||
|
||||
The core job is orchestration across backends — Git repos, repo subdirectories (`wiki/`), Gitea wikis, local folders, Obsidian vaults, WebDAV/Nextcloud directories, Coulomb spaces — never replacing or homogenizing them.
|
||||
|
||||
## Core domain model (the concepts code must honor)
|
||||
|
||||
These abstractions come from `INTENT.md` and define the architecture. New code should map onto them rather than inventing parallel vocabulary:
|
||||
|
||||
- **Shard** — an independently meaningful page store attached to a root entity. Shards have *sovereignty*: their own backend, capabilities, limits, history, and identity model. Not all shards are Git-native.
|
||||
- **Root entity / information space** — the joined space that shards attach to. Each information space should have a **Git-addressable coordination layer** (history, patches, review, backup, reconciliation) even when individual shards are not Git-native.
|
||||
- **Shard adapter contract** — the versioned interface a backend implements to participate. Adapters are **capability-aware**: the core must model explicitly which operations a shard supports (read, write, diff, merge, lock, version, publish, accept patches) rather than assuming uniformity.
|
||||
- **Wiki page model** — a stable, versioned, Markdown-first but backend-neutral representation of pages, paths, links, metadata, revisions.
|
||||
- **Projection** — a lazy, cache-like local view of remote/external shard content. Prefer lazy projection over eager copying.
|
||||
- **Overlay** — a non-destructive local edit against a remote, read-only, or capability-limited shard, representable as drafts/patches/commits/merge requests *before* destructive application ("overlay before mutation").
|
||||
- **Coordination journal** — the Git-backed record of change flows for an information space.
|
||||
- **Shard modes** — read-only, write-through, mirrored, projected, cached, canonical.
|
||||
|
||||
## Design constraints to enforce in code
|
||||
|
||||
These are hard boundaries from `INTENT.md`; treat violations as design bugs:
|
||||
|
||||
- **Mechanism over policy.** Provide primitives for federation, sync, overlays, patching, conflict detection, projection, reconciliation. Do *not* hard-code one editorial/sync/conflict/canonical-source policy — keep those configurable.
|
||||
- **Union without erasure.** Always preserve provenance: which shard a page came from, its freshness, whether it is cached, whether it has overlays, whether it diverges from an equivalent page elsewhere. Never hide authorship, conflicts, freshness, or backend limitations.
|
||||
- **No silent remote mutation.** Do not mutate remote systems without explicit adapter support and user intent.
|
||||
- **Graceful degradation.** Limited backends must still be usable as read-only/cache/projection/backup/patch targets.
|
||||
- **Not a file-sync daemon.** Synchronization is wiki-page-semantic, not generic file mirroring.
|
||||
|
||||
`INTENT.md` has a "Stability Note": changes that redefine what a shard is, Git's role, how root entities are modeled, or whether this is an orchestrator vs. an engine are **architectural changes** and should be rare and deliberate.
|
||||
|
||||
## Build, test, run
|
||||
|
||||
Python with a `src/` layout, built via hatchling, tested with pytest. Tests run against the source tree directly (`pythonpath = ["src"]` in `pyproject.toml`), so no install/editable step is required to run them.
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]" # one-time: install dev tooling (pytest, pytest-cov, ruff)
|
||||
pytest # run the full test suite
|
||||
pytest tests/test_package.py::test_version_is_exposed # run a single test
|
||||
pytest --cov # run with coverage
|
||||
ruff check # lint
|
||||
ruff format # format
|
||||
```
|
||||
|
||||
Note: the system `pytest` is 7.4.x; `minversion` in `pyproject.toml` is pinned to `7.0` to match. Bump it if a newer pytest is installed into the dev environment.
|
||||
@SCOPE.md
|
||||
@.claude/rules/repo-identity.md
|
||||
@.claude/rules/session-protocol.md
|
||||
@.claude/rules/first-session.md
|
||||
@.claude/rules/workplan-convention.md
|
||||
@.claude/rules/stack-and-commands.md
|
||||
@.claude/rules/architecture.md
|
||||
@.claude/rules/repo-boundary.md
|
||||
@.claude/rules/credential-routing.md
|
||||
@.claude/rules/agents.md
|
||||
|
||||
10
INTENT.md
10
INTENT.md
@@ -16,6 +16,8 @@ The goal is to allow independently stored and differently implemented wikis, pag
|
||||
|
||||
The repository provides a **shard orchestration layer** for interconnected Markdown and markup-based wiki content.
|
||||
|
||||
Equivalently, shard-wiki can be used as a **headless, API-first wiki engine** — optimized for **integrating heterogeneous data sources** and for **efficient access by agents and automation** — that ships its own native engine as one (canonical-mode) shard among many. There is no bundled UI: presentation and rendering are consumer concerns.
|
||||
|
||||
It allows wiki-like systems to:
|
||||
|
||||
* Attach heterogeneous page stores as shards of a shared information space
|
||||
@@ -30,6 +32,7 @@ It allows wiki-like systems to:
|
||||
* Run fully standalone with open read/write access and complete change history, then progressively layer multi-tenant enterprise access control through external identity integration
|
||||
* Allow existing wiki engines to become federation-capable through a shared API
|
||||
* Allow non-federation-aware systems to participate through adapters and projections
|
||||
* Serve as a **headless, API-first wiki engine** (a small typed-extension core) that integrates heterogeneous data sources and is consumed efficiently by agents and automation
|
||||
|
||||
It transforms disconnected wiki engines, Git repositories, local folders, WebDAV directories, application-specific content stores, and desktop editing workflows into a **composable federated wiki space**.
|
||||
|
||||
@@ -85,7 +88,7 @@ A mature `shard-wiki` should allow each participating shard to see the others as
|
||||
|
||||
This repository is **not** intended to:
|
||||
|
||||
* Replace all wiki engines with a single canonical wiki implementation
|
||||
* Replace all wiki engines with a single canonical wiki implementation *(shard-wiki MAY still provide its own native, headless, API-first engine as one optional shard backend — see Design Principles — but never as a mandated or universal replacement)*
|
||||
* Force every shard to use the same backend, database, directory layout, or storage format
|
||||
* Require every participating system to become federation-aware
|
||||
* Require every participating shard to be Git-native
|
||||
@@ -148,6 +151,9 @@ Policy decisions such as conflict preference, canonical source selection, public
|
||||
* **Composable integration**
|
||||
Wiki engines should be able to use the `shard-wiki` API to become federation-enabled without reimplementing federation internally.
|
||||
|
||||
* **Native reference engine (additive, headless & API-first)**
|
||||
shard-wiki MAY provide its own native wiki-engine as a **canonical-mode shard backend** — a **small core** with a **typed-extension framework**, activated **per shard** (only what you need). It is **headless and API-first** (no bundled UI; presentation/rendering are consumer concerns) and tuned for **integrating heterogeneous data sources** and **efficient agent/automation access**. It is *one shard type among many*, implemented against shard-wiki's own adapter contract; it does **not** replace other engines, mandate a single implementation, or change shard-wiki's role as an orchestrator. Shard sovereignty and union-without-erasure are preserved.
|
||||
|
||||
* **Open by default, progressively governed**
|
||||
A standalone `shard-wiki` must be runnable with zero external dependencies in a classic Ward Cunningham / c2-style open read/write-for-all mode. Access control is an *additive capability*, not a precondition: the same core progresses — without re-architecture — to authenticated single-user, to group/role-based, to multi-tenant enterprise access control, mirroring the NetKingdom capability ladder (lightweight → expanded).
|
||||
|
||||
@@ -201,3 +207,5 @@ Such changes should be rare, because they affect all downstream systems relying
|
||||
|
||||
In particular, changes that redefine what counts as a shard, what role Git plays, how root entities are modeled, or whether `shard-wiki` is an orchestrator rather than a wiki engine should be treated as architectural changes.
|
||||
|
||||
**Amendment — 2026-06-15 (SHARD-WP-0013 T4, decision `84ffdb48`):** admits an **additive** native reference wiki-engine — **headless, API-first**, a small typed-extension core — as a **canonical-mode shard backend** optimized for data-source integration and agent access. Deliberate, narrow scope change; shard-wiki remains an orchestrator and neither mandates nor replaces other engines. (Mirrors the earlier auth-in-core amendment precedent.)
|
||||
|
||||
|
||||
28
SCOPE.md
28
SCOPE.md
@@ -17,12 +17,12 @@ Learnings update both SCOPE and INTENT where necessary.
|
||||
|
||||
| Layer | State |
|
||||
|-------|-------|
|
||||
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
|
||||
| Code | Foundation slice implemented (SHARD-WP-0007): `provenance` + `policy` leaves, `model` (Identity/Placement/Span/Page/CapabilityProfile), `adapters` (contract + FolderAdapter + conformance suite), `coordination` (event-sourced DecisionLog), `union` (resolution + chorus, overlay-aware), `InformationSpace` orchestrator. Write path added (SHARD-WP-0008): writable adapter, overlay engine (draft→patch→apply-under-drift), edit() unifies write-through + overlay-before-mutation. Native engine implemented (SHARD-WP-0014): `engine` (kernel + typed-extension runtime + per-shard activation [ADR-0001] + capability-profile-from-extensions + EngineShardAdapter + the `ext.struct` built-in) — an engine shard attaches to an InformationSpace as a canonical-mode shard. Git-backed coordination log (SHARD-WP-0009): `DecisionLog` storage factored behind an `EventStore`; `GitEventStore` makes the log git-addressable (each space a ref, append = immutable CAS-guarded commit), a per-space `AppendAuthority` (lease) gives a single-writer total order with re-grantable HA hand-off, cross-process read-your-writes verified, and a verbatim one-time importer (`migrate_space`/JSONL) replays in-memory logs into git; `InformationSpace.git_backed(...)` wires it. Derived views (SHARD-WP-0010): `views` (wikilink + red-link model, BackLinks, RecentChanges, AllPages/SiteMap) — recomputable, provenance-carrying, presentation-free, exposed via `InformationSpace.backlinks/recent_changes/all_pages/site_map`. Incremental-first derived tier (SHARD-WP-0011): `incremental` (indexed equivalence via MinHash/LSH blocking + verify, change-driven delta maintenance with retraction/propagation, Merkle-style digest + self-healing I-2 consistency-checker, `UnionIndex` routed behind `InformationSpace.all_pages` with rebuild as explicit fallback). Second adapter (SHARD-WP-0012): `GitShardAdapter` — git-IS-store substrate (read=tracked *.md, write=commit, current_rev=per-path sha for drift, adopted git-native history), passes conformance, works across folder+git shards in union/overlay/edit with no core change (capability-as-data proven on a second substrate). 196 tests green, ~97% coverage |
|
||||
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
|
||||
| Research | yawex prior-art exploration (`research/260608-yawex-prior-art/`) |
|
||||
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) |
|
||||
| Demand | NetKingdom integration asks captured, not yet negotiated |
|
||||
| Spec | Access/history architecture blueprint drafted; PRD/TSD/UCC scaffolds |
|
||||
| Work | `SHARD-WP-0001` active — 6 design tasks, all todo |
|
||||
| Spec | CoreArchitectureBlueprint (whole-system, hardened via SHARD-WP-0005/0006) + FederationArchitecture + FederationRequirements + TSD §A adapter contract + ArchitectureBlueprint (auth/history) + WikiEngineCoreArchitecture (headless API-first engine, SHARD-WP-0013) drafted; UseCaseCatalog 84 UCs (+ engine capability-structure layer); PRD scaffold |
|
||||
| Work | `SHARD-WP-0001` **done** (6 ADRs: yawex-derived federation requirements → `spec/FederationRequirements.md`); `SHARD-WP-0002` **done** (18 tasks → `FederationArchitecture.md` [T1–T10, T17] + `TechnicalSpecificationDocument.md` §A adapter contract [T11–T16, T18]); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` **done** (all 8 computational-knowledge dives T1–T8 complete + "computational page model" synthesis); `SHARD-WP-0005` **done** (9 tasks: CoreArchitectureBlueprint hardened against the 260615 review); `SHARD-WP-0006` **done** (5 tasks: round-2 hardening — overview reconciled, event-sourced coordination + append authority, adapter conformance, incremental correctness + I-2 verification) |
|
||||
|
||||
## In Scope (today)
|
||||
|
||||
@@ -32,11 +32,15 @@ Learnings update both SCOPE and INTENT where necessary.
|
||||
- Authorization model design (delegated authentication, core authorization).
|
||||
- Shard adapter contract and wiki page model (to be specified, then implemented).
|
||||
- Git-backed coordination journal for information spaces.
|
||||
- A **native, headless, API-first wiki-engine core** (small typed-extension core, as a
|
||||
canonical-mode shard backend) — design via SHARD-WP-0013; optimized for data-source
|
||||
integration and agent access.
|
||||
- State Hub workplan registration and consistency sync.
|
||||
|
||||
## Out Of Scope (today)
|
||||
|
||||
- A standalone wiki engine UI or rendering pipeline.
|
||||
- A wiki-engine **UI or rendering pipeline** (the engine is headless/API-first; presentation
|
||||
is a consumer concern). A bundled standalone UI is not provided.
|
||||
- Authentication, credential storage, or user directory implementation.
|
||||
- Hard-coded editorial, sync, or conflict-resolution policy.
|
||||
- Generic file mirroring independent of wiki-page semantics.
|
||||
@@ -52,6 +56,14 @@ live in core.
|
||||
|
||||
## Current Planning
|
||||
|
||||
Design work is tracked in `workplans/SHARD-WP-0001-yawex-requirements.md`.
|
||||
Specification outputs from that workplan land in `spec/`. Inbound integration
|
||||
asks remain in `demand/` until reviewed and promoted into spec or workplans.
|
||||
Design work is tracked in `workplans/SHARD-WP-0001-yawex-requirements.md`
|
||||
(yawex-derived resolution, namespaces, overlays) and
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (federation architecture,
|
||||
decisions, tradeoffs). Research continues under
|
||||
`workplans/SHARD-WP-0003-engine-dives-batch.md` (remaining new-insight wiki
|
||||
engines + git-forge wikis + classic engines) and
|
||||
`workplans/SHARD-WP-0004-computational-knowledge-systems.md` (literate
|
||||
programming, computational notebooks, live-coding REPLs, image-based/moldable
|
||||
environments — the executable/computational page-model thread). Specification
|
||||
outputs land in `spec/`. Inbound integration asks remain in `demand/` until
|
||||
reviewed and promoted into spec or workplans.
|
||||
@@ -82,8 +82,10 @@ shard-wiki authorizes; it never authenticates or stores identity. Everything bel
|
||||
|
||||
## Next actions
|
||||
|
||||
- [ ] Send this as a **capability request / interface negotiation** to the netkingdom domain
|
||||
(user-engine + net-kingdom owners) via the state hub.
|
||||
- [x] Send this as a **capability request / interface negotiation** to the netkingdom domain
|
||||
(user-engine + net-kingdom owners) via the state hub. — *sent 2026-06-13, capability
|
||||
request `3136a9e2-0c1a-4b3f-8a02-ef47b27071fe` (auto-routed to `netkingdom`), inbox
|
||||
message `46c60f17`.*
|
||||
- [ ] Lock the **token claim contract** (A2–A4) and the **membership read API** (B1–B5) first
|
||||
— they gate everything L2+.
|
||||
- [ ] Define shard-wiki's **audit event schema** (C1) against user-engine's existing audit model.
|
||||
|
||||
91
history/260615-core-architecture-blueprint-review-2.md
Normal file
91
history/260615-core-architecture-blueprint-review-2.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# Critical review (round 2) — CoreArchitectureBlueprint.md (hardened)
|
||||
|
||||
Date: 2026-06-15 · Reviewer: tegwick (with Claude) · Subject:
|
||||
`spec/CoreArchitectureBlueprint.md` after **SHARD-WP-0005** (commit f21b7b5) · Feeds:
|
||||
**SHARD-WP-0006**
|
||||
|
||||
A second hostile pass over the *hardened* blueprint. Round 1 found design bugs; this round
|
||||
finds (a) self-consistency regressions the surgical hardening introduced, and (b) deeper
|
||||
second-order gaps — some of which the hardening *sharpened*. Verdict: the architecture is now
|
||||
substantially sound; what remains in §B/§C are hard distributed-systems/operational questions,
|
||||
not design smells — except §A (a real regression) and three foundational gaps in §B.
|
||||
|
||||
---
|
||||
|
||||
## A. Self-consistency regressions introduced by surgical hardening
|
||||
|
||||
The 9 edits deepened §6–§9 but did not propagate to the **overview surfaces**, so the document
|
||||
now contradicts itself between its summary and its body (and readers trust the summary).
|
||||
|
||||
- **A-1 (real contradiction).** §4 still says "Addressing, **equivalence**, and transclusion
|
||||
key on identity" — the exact conflation T2 fixed in §7.2 (equivalence keys on *content
|
||||
fingerprint across distinct identities*). → **WP-0006 T1**
|
||||
- **A-2.** §4 "Projection — typed on two axes" and §4 "Provenance envelope … every artifact
|
||||
carries [full wrapper]" are stale vs T7's §8.4 (two-axis = extension point; trivial default)
|
||||
and §7.3 (layered effective-vs-own). → **T1**
|
||||
- **A-3.** §10 policy surface omits knobs the hardening added (freshness/staleness §8.8,
|
||||
squash-compaction §8.1, conflict-resolution preset §8.6, tenant-partition) — yet §11 defines
|
||||
`policy/` as "owns the §10 surface." The module contract points at a stale list. → **T1**
|
||||
- **A-4 (cosmetic).** §3 diagram + §11 header still say "L4 rebuildable cache" / "15 spectra,"
|
||||
advertising the pre-hardening model (§8.7 incremental-first; §6.5 orthogonal-core). → **T1**
|
||||
|
||||
Meta-point: surgical editing hardened the body but regressed whole-document coherence; v2
|
||||
needs an **overview-reconciliation pass**.
|
||||
|
||||
## B. Foundational gaps (serious; some sharpened by the hardening)
|
||||
|
||||
- **B-1 — The journal is now a concurrent-write DB, but it's single-writer Git.** §8.6's
|
||||
consistency model assumes "the journal is local Git, read-your-writes." L4 multi-tenant + the
|
||||
L6 Orchestrator API imply a server; HA/scale implies *multiple* instances. Concurrent commits
|
||||
of coordination-canonical state to one git journal = lock contention / merge races; Git is not
|
||||
a concurrent-write store. Either single-writer-per-space (an unstated HA ceiling) or a real
|
||||
concurrent coordination store with Git as an *export*. **T1 of WP-0005 worsened this** by
|
||||
loading more canonical state into the journal. The keystone unanswered question. → **T2**
|
||||
- **B-2 — Capability-as-data trusts self-reported profiles with no conformance check.** I-3 +
|
||||
§6.5's degradation contract assume the profile tells the truth. A buggy adapter (claims
|
||||
`merge=git/text`, corrupts; claims `notify`, never emits) silently poisons every degradation
|
||||
decision. No **adapter conformance suite** (declared profile == observed behavior) exists.
|
||||
Foundational for an architecture whose correctness rests on profile accuracy. → **T3**
|
||||
- **B-3 — "Coordination-canonical state in the journal" has no representation design.** T1
|
||||
relocated overlays/bindings/aliases/equivalence-sets/merges into "the journal" without saying
|
||||
*how* Git stores structured mutable state. "All equivalences touching X" over a git-of-files
|
||||
is O(scan) unless indexed — and an index is L4/derived. The new central concept is a black
|
||||
box; resolve *with* B-1. → **T2**
|
||||
- **B-4 — Incremental equivalence is under-specified/likely incorrect; I-2 only eventually
|
||||
true.** §8.7 re-verifies a changed page's *new* candidate set but not the pairs it *leaves*
|
||||
(a page exiting an LSH bucket can break an existing equivalence edge); the delta is not
|
||||
additive. Deeper: incremental maintenance drifts from `f(canonical)`, so I-2 holds only
|
||||
eventually, guaranteed solely by an expensive reconcile-against-rebuild. Needs a stated
|
||||
verification mechanism (background checker / digest-vs-sampled-rebuild). → **T4**
|
||||
|
||||
## C. Real but second-tier (track as open problems O-8…O-11)
|
||||
|
||||
- **C-1 — Mechanism-over-policy → operator burden; no preset bundles.** ~7 knob families with
|
||||
sub-modes and interactions; only authz (L0–L4) bundles into personas. Need named bundles
|
||||
("personal vault" / "team wiki" / "enterprise federation"). → **O-8 / T5**
|
||||
- **C-2 — Tenant partitioning (I-13) vs shard sharing + lazy projection.** A shard in two roots
|
||||
is cached twice → duplicate storage + double refresh on rate-limited backends. Shard
|
||||
exclusive-to-one-root or shareable? Unresolved. → **O-9 / T5**
|
||||
- **C-3 — Span-level authz + transclusion is an unmodeled leak path.** Authz is per
|
||||
page/shard/tenant; transclusion crosses shards at span granularity → a page can leak a span
|
||||
past its ACL (aggregation/inference). §7.3's ⊕ also stops being simple two-level inheritance
|
||||
across a transclusion boundary. → **O-10 / T5**
|
||||
- **C-4 — Union-under-unavailability undefined.** Freshness covers *stale*, nothing covers
|
||||
*down*. The dead-shard read path (partial union? error? last-known?) is unspecified though
|
||||
it's the commonest real failure. → **O-11 / T5**
|
||||
|
||||
## D. Recommended resolution (→ SHARD-WP-0006)
|
||||
|
||||
1. **§A reconciliation** (T1) — make the overview match the hardened body.
|
||||
2. **Journal & coordination-state model** (T2) — settle single-vs-multi-writer and separate the
|
||||
**coordination-state store** from the **content-history journal**. Likely resolution:
|
||||
**event-sourced coordination** — an append-only *decision log* is the coordination-canonical
|
||||
tier (git-addressable, I-6 preserved); the queryable current state (alias table, equivalence
|
||||
set) is a *derived fold* of the log (disposable). Append-logs tolerate concurrency far better
|
||||
than mutable-file Git; state a concurrency model. Resolves **B-1 + B-3** together.
|
||||
3. **Adapter conformance suite** (T3) — make a passing conformance run part of the contract
|
||||
(B-2): every adapter proves declared profile == observed behavior.
|
||||
4. **Incremental correctness + I-2 verification** (T4) — fix the leaving-bucket re-verification
|
||||
and propagation; add a background consistency-checker / derived-tier digest so I-2 is
|
||||
verifiable, not merely asserted (B-4).
|
||||
5. **Track §C** (T5) — O-8…O-11 with chosen direction + revisit trigger; close-out.
|
||||
122
history/260615-core-architecture-blueprint-review.md
Normal file
122
history/260615-core-architecture-blueprint-review.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# Critical review — CoreArchitectureBlueprint.md
|
||||
|
||||
Date: 2026-06-15 · Reviewer: tegwick (with Claude) · Subject:
|
||||
`spec/CoreArchitectureBlueprint.md` @ commit **9b5b393** · Feeds: **SHARD-WP-0005**
|
||||
|
||||
A deliberately hostile review of the first whole-system architecture, to find where it
|
||||
**breaks (correctness)**, **fails to scale**, and **could be more elegant/efficient** before
|
||||
any implementation. Findings are prioritised; each is the input to a SHARD-WP-0005 task.
|
||||
|
||||
## Verdict in one line
|
||||
|
||||
The **layering and the dual narrow waist are sound and stay**. The **thesis is ~90% right**;
|
||||
the missing 10% (curatorial / coordination-canonical state) breaks its clean story. There are
|
||||
**two genuine bugs**, **two large unaddressed scaling risks**, and several **elegance/efficiency
|
||||
debts** — all fixable without touching INTENT.
|
||||
|
||||
---
|
||||
|
||||
## A. The framing crack (fix resolves three issues)
|
||||
|
||||
**A-1 — Two buckets hide a third.** The thesis "canonical at the edges, derived in the middle"
|
||||
omits **born-in-the-middle-but-canonical** state: overlays that are the local truth against a
|
||||
read-only shard (Flow C), manual **curator equivalence bindings**, alias tables, merge
|
||||
decisions. These encode human judgment or local-only content and **cannot be rebuilt** from
|
||||
shards+journal.
|
||||
|
||||
**Contradiction:** I-2 declares L4 rebuildable, yet §8.4 puts "alias table, curator binding"
|
||||
in L4. You cannot rebuild a curator's manual binding.
|
||||
|
||||
**Fix:** three states — **sharded-canonical**, **coordination-canonical** (journal: overlays,
|
||||
bindings, aliases, merges — durable, born in the middle), **derived-disposable** (union graph,
|
||||
indexes, projections). Re-frame §1 as **canonical (sharded + coordination) vs derived
|
||||
(disposable)**; `derived = f(canonical)` then becomes actually true. → **T1**
|
||||
|
||||
---
|
||||
|
||||
## B. Where it breaks (correctness)
|
||||
|
||||
**B-1 — Identity conflated with content-fingerprint (BUG).** §7.2 derives page identity from
|
||||
content fingerprint. That makes **editing a page change its identity**, breaking every
|
||||
reference. Fingerprints identify *versions/equivalence*, not *identity*. Page identity must be
|
||||
a **stable handle (uid)** surviving edits; fingerprints belong to the **equivalence** mechanism
|
||||
(§8.4). One word, two concepts, wrong implementation for the stable one. → **T2**
|
||||
|
||||
**B-2 — No concurrency/consistency model.** Concurrent overlays on one page, overlay applied
|
||||
after source drift, journal-commit vs shard-native-write ordering — all undefined. Conflict
|
||||
handling is deferred to "policy presets," but **conflict *detection + representation* is core
|
||||
mechanism**; only *resolution* is policy. The union's consistency guarantee is unstated
|
||||
(eventually-consistent? read-your-writes? causal-via-journal?). → **T3**
|
||||
|
||||
**B-3 — Persisted union cache + multi-tenant = leak surface.** §13 recommends a persisted L4
|
||||
cache; §9 protects content by *read-time* filtering on the provenance envelope. A persisted
|
||||
cross-tenant union cache guarded only by read-time filtering is an L4 attack surface. Tension
|
||||
between I-2 (persisted rebuildable cache), scale, and L5 isolation is unacknowledged. → **T8**
|
||||
|
||||
---
|
||||
|
||||
## C. Where it fails to scale
|
||||
|
||||
**C-1 — Equivalence detection is O(N²), no indexing/incremental story.** Fingerprint /
|
||||
span-set-overlap across all pages of all shards is combinatorial (10 shards × 100k pages ≈
|
||||
10¹² comparisons). No blocking/LSH/indexing, no incremental maintenance. Biggest scaling
|
||||
hazard in the document. → **T4**
|
||||
|
||||
**C-2 — "Rebuildable cache" collides with the operational-envelope axis.** A byte-exact
|
||||
rebuild requires reading *every page of every shard*, including rate-limited/paginated
|
||||
external APIs (Notion) and irreducibly-live sources — hours-to-days. I-2 contradicts axis-10.
|
||||
**Incremental, change-driven maintenance must be primary** (notify→delta), rebuild a rare
|
||||
fallback. Cache invalidation — the actual hard problem — is named once and never designed. →
|
||||
**T4, T5**
|
||||
|
||||
**C-3 — Unbounded history at open L0 = DoS/perf.** "Every write a commit" + "open for all" ⇒
|
||||
the git journal grows without bound under bots/vandalism and git degrades on huge histories.
|
||||
"History is the floor" has an unacknowledged cost: packing, compaction, per-shard offload. →
|
||||
**T8**
|
||||
|
||||
---
|
||||
|
||||
## D. Elegance / efficiency debts
|
||||
|
||||
**D-1 — The 15 spectra assert a clean degradation function never demonstrated.** Either most
|
||||
axes are irrelevant to most ops (then the 15-D profile is ceremony), or behavior depends on
|
||||
several axes *jointly* (then "no per-backend code" becomes a sprawling axis-interaction matrix
|
||||
— the flat-checklist problem in higher dimensions). And the axes **aren't orthogonal**
|
||||
(git-native history ⟺ git-IS-store ⟺ git/text merge; encrypted opacity ⟹ query/translation
|
||||
collapse). Model a **smaller orthogonal core** + **derived/implied** positions, and state the
|
||||
**axis-interaction subset** the degradation logic truly uses. → **T6**
|
||||
|
||||
**D-2 — Provenance envelope isn't inherited; it'll dwarf the content.** Per-span envelopes at
|
||||
block granularity = 10k near-identical envelopes for a 10k-block graph. The doc already
|
||||
invented the right pattern for Trilium ("effective-vs-own with per-attribute provenance") and
|
||||
failed to apply it to its own envelope. Make provenance **layered (page envelope + span
|
||||
deltas)**. → **T7**
|
||||
|
||||
**D-3 — Projection machinery over-fit to the exotic tail.** Two-axis model + three facets +
|
||||
view registry exist mostly for UC-83/84 (2 of 84 UCs); the 95% case (markdown in git) pays the
|
||||
weight. Make the **common case trivial** (default = plain lazy replication) and
|
||||
derivation/liveness an **extension point**, not a taxonomy every projection instantiates. →
|
||||
**T7**
|
||||
|
||||
**D-4 — Cross-cutting rails are the highest-coupling components, presented as clean.**
|
||||
`provenance/` and capability types are imported by every layer (god-modules); an envelope
|
||||
change ripples everywhere. And **policy has no module** (§10 enumerates it; §11 omits it)
|
||||
despite being consulted by L3/L4/L5. Give policy a home; pin the rails behind stable narrow
|
||||
interfaces. → **T7**
|
||||
|
||||
---
|
||||
|
||||
## E. What explicitly stays
|
||||
|
||||
- The 6-layer model + the dual narrow waist (adapter contract / page model).
|
||||
- Capability-as-data (I-3), union-without-erasure (I-4), overlay-before-mutation (I-5),
|
||||
Git-addressable coordination (I-6), mechanism-over-policy (I-7), graceful degradation (I-8).
|
||||
- The federation-model taxonomy and the auth ladder (ArchitectureBlueprint.md).
|
||||
|
||||
## F. Disposition
|
||||
|
||||
Some findings are **solvable now** (A-1, B-1, D-2, D-3, D-4, C-3); some are **partially open**
|
||||
and should be tracked honestly rather than pretend-solved (B-2 consistency model: pick a
|
||||
guarantee; C-1 equivalence-at-scale: pick a blocking strategy; D-1 axis interactions: enumerate
|
||||
the real subset). SHARD-WP-0005 closes the solvable ones and records the open ones in a new
|
||||
"Known scaling risks & open problems" section of the blueprint. → **T9**
|
||||
45
history/260615-reuse-surface-contributions.md
Normal file
45
history/260615-reuse-surface-contributions.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# reuse-surface contributions — shard-wiki (SHARD-WP-0013 T3)
|
||||
|
||||
Date: 2026-06-15 · From: shard-wiki (whynot) · To: reuse-surface (helix_forge) · Tracked list
|
||||
of (A) capabilities shard-wiki registered, (B) **gaps** shard-wiki proposes the reuse surface
|
||||
add, and (C) capabilities shard-wiki will **consume** rather than rebuild. Communicated to the
|
||||
reuse-surface agent via state-hub `send_message`.
|
||||
|
||||
## A. Registered (T1 + T3) — 8 entries
|
||||
|
||||
`capability.wiki.{shard-orchestration, adapter-contract, page-model, coordination-journal,
|
||||
overlay, federation-models, engine-typed-extensions, derived-views}` — committed in
|
||||
reuse-surface, `validate` ok (20 entries total).
|
||||
|
||||
## B. Proposed gaps (cross-cutting; not shard-wiki-internal) → reuse-surface should own/define
|
||||
|
||||
- **G1 — `capability.platform.typed-extension-framework`** *(suggested)*
|
||||
A reusable pattern: a **small core + a stringent typed-extension framework** where extensions
|
||||
declare typed contracts, compose, and are **activated per context**. shard-wiki's wiki engine
|
||||
(`capability.wiki.engine-typed-extensions`) is one instance, but the *pattern* is
|
||||
cross-domain (any HelixForge capability platform). Evidence: shard-wiki UseCaseCatalog
|
||||
"Capability structure" layer (core + 10 typed extensions + conflict-mediation map).
|
||||
Suggested owner: helix_forge / reuse-surface. Relation: would `generalize`
|
||||
`capability.wiki.engine-typed-extensions`.
|
||||
|
||||
- **G2 — `capability.content.translation-fidelity`** *(suggested)*
|
||||
Lossless/lossy **content translation with an explicit fidelity report** (what round-trips
|
||||
cleanly vs degrades, non-mappable elements preserved as provenance). Reusable well beyond
|
||||
wiki (any format-bridging consumer). Evidence: shard-wiki TSD §A.6, UC-42/UC-59.
|
||||
|
||||
## C. Consumptions (reuse, do not rebuild)
|
||||
|
||||
- **`capability.feature-control.evaluate`** (helix_forge/feature-control) → shard-wiki's
|
||||
**per-shard extension/feature activation** (the "activate only what you need" mechanism).
|
||||
Already recorded as a relation on `capability.wiki.engine-typed-extensions`.
|
||||
- **`capability.authorization.policy-evaluate`** (flex-auth) → shard-wiki's **X-AUTHZ** policy
|
||||
decisions. shard-wiki owns the authz *model* (authz-in-core) but can reuse this evaluation
|
||||
engine rather than building one.
|
||||
- **`capability.statehub.{progress-log, workstream-coordinate}`** → already in use for
|
||||
coordination across this work.
|
||||
|
||||
## Status
|
||||
|
||||
Gaps G1/G2 are **suggestions** to the reuse-surface owner (not unilateral registrations, since
|
||||
they are cross-cutting, not shard-wiki-internal). Consumptions are recorded for the engine
|
||||
architecture (T5) so it reuses rather than reinvents.
|
||||
@@ -1,8 +1,19 @@
|
||||
# history/
|
||||
|
||||
Archived material that is no longer needed for daily work but should be kept.
|
||||
Archived material and the project's **meta-history**: finished/canceled workplans kept for
|
||||
the record, plus durable **reviews, critical assessments, and decision records** — the
|
||||
reasoning behind the specs, captured at a point in time.
|
||||
|
||||
Use a `yymmdd-` prefix when archiving files or directories. Content here is
|
||||
**out of scope** for regular tasks — consult only for research or diagnostics.
|
||||
Use a `yymmdd-` prefix. Archived material is **out of scope** for regular tasks (consult only
|
||||
for research or diagnostics); assessment/review records are point-in-time and may seed active
|
||||
workplans, but are not edited after the fact — supersede with a new dated record and link back.
|
||||
|
||||
Finished or canceled workplans from `workplans/` are archived here.
|
||||
Distinct from the **coordination journal** (a runtime Git-backed record of *content* change
|
||||
flows inside an information space, an INTENT domain concept); `history/` is the *project's own*
|
||||
design evolution.
|
||||
|
||||
| Date | Record | Subject |
|
||||
|------|--------|---------|
|
||||
| 2026-06-15 | `260615-core-architecture-blueprint-review.md` | Critical review of `spec/CoreArchitectureBlueprint.md` (commit 9b5b393); inputs to `SHARD-WP-0005` |
|
||||
| 2026-06-15 | `260615-core-architecture-blueprint-review-2.md` | Round-2 review of the hardened blueprint (post-`SHARD-WP-0005`, f21b7b5); inputs to `SHARD-WP-0006` |
|
||||
| 2026-06-15 | `260615-reuse-surface-contributions.md` | shard-wiki's reuse-surface registrations, proposed gaps (G1/G2), and consumptions (`SHARD-WP-0013` T3) |
|
||||
@@ -36,6 +36,11 @@ pythonpath = ["src"]
|
||||
branch = true
|
||||
source = ["shard_wiki"]
|
||||
|
||||
[tool.coverage.report]
|
||||
show_missing = true
|
||||
# Quality floor for `pytest --cov` / `coverage report` (not forced on a bare `pytest` run).
|
||||
fail_under = 90
|
||||
|
||||
[tool.ruff]
|
||||
src = ["src", "tests"]
|
||||
target-version = "py311"
|
||||
|
||||
12
registry/README.md
Normal file
12
registry/README.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Capability Registry
|
||||
|
||||
Markdown-first capability index for federation and reuse planning.
|
||||
|
||||
## Authoring
|
||||
|
||||
1. Copy a capability entry template (see reuse-surface `templates/capability-entry.template.md`).
|
||||
2. Add the row to `indexes/capabilities.yaml`.
|
||||
3. Run `reuse-surface validate` from a checkout with the CLI installed.
|
||||
4. Merge to `main` and verify publish with `reuse-surface establish --publish-check`.
|
||||
|
||||
Federation contract: reuse-surface `docs/RegistryFederation.md`.
|
||||
103
registry/capabilities/capability.wiki.adapter-contract.md
Normal file
103
registry/capabilities/capability.wiki.adapter-contract.md
Normal file
@@ -0,0 +1,103 @@
|
||||
---
|
||||
id: capability.wiki.adapter-contract
|
||||
name: Capability-Aware Shard Adapter Contract
|
||||
summary: A versioned backend interface where each binding declares a verified capability profile (positions on capability spectra), so federation ops degrade by capability.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, adapter, capability, contract, conformance, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Fifteen capability spectra with an orthogonal core + implication rules, plus
|
||||
a normative contract spec (TSD Section A); derived from a ~23-system synthesis.
|
||||
availability:
|
||||
current: A2
|
||||
target: A5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
AdapterContract + a read/write FolderAdapter + a conformance suite that
|
||||
verifies declared profile == observed behaviour exist as a source module.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- versioned interface with declared, conformance-verified capability profiles
|
||||
- one concrete adapter (file-store) passes the conformance suite
|
||||
broken_expectations:
|
||||
- only one substrate implemented (git-IS-store, REST, CRDT adapters planned)
|
||||
out_of_scope_expectations:
|
||||
- hosting backends
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- single adapter implemented so far
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Mediate heterogeneity at one narrow waist: a backend participates by implementing a
|
||||
versioned interface and declaring a verified position on each capability spectrum.
|
||||
includes:
|
||||
- capability profile as data (orthogonal-core spectra + implied positions)
|
||||
- operation verbs (read/write/diff/merge/notify/.../derive-projection/execute)
|
||||
- a conformance suite (profiles verified, not self-asserted)
|
||||
excludes:
|
||||
- assuming uniform backend capabilities
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-34..UC-43, UC-50, UC-57, UC-60..UC-69 (shard attachment & adapter binding)"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A5
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/adapters/"
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.page-model
|
||||
supports:
|
||||
- capability.wiki.shard-orchestration
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/TechnicalSpecificationDocument.md (Section A)"
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 6)"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_folder_adapter.py"
|
||||
- "shard-wiki/tests/test_conformance.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- exposing any page store as a capability-described, conformance-checked shard
|
||||
not_recommended_for:
|
||||
- backends that cannot honestly describe their capabilities
|
||||
known_limitations:
|
||||
- reference implementation covers the file-store substrate only so far
|
||||
---
|
||||
|
||||
# Capability-Aware Shard Adapter Contract
|
||||
|
||||
The bottom narrow waist of shard-wiki: a versioned interface plus a **verified** capability
|
||||
profile per binding. Core logic is written once against capabilities (not per-backend), and
|
||||
the conformance suite rejects profiles whose declared abilities don't match observed behaviour.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Fifteen spectra reduced to an orthogonal core with implication rules (CoreArchitectureBlueprint
|
||||
Section 6.5); normative in TSD Section A.
|
||||
|
||||
### Availability
|
||||
`adapters/` ships the contract, a folder adapter, and `assert_conformant`.
|
||||
103
registry/capabilities/capability.wiki.coordination-journal.md
Normal file
103
registry/capabilities/capability.wiki.coordination-journal.md
Normal file
@@ -0,0 +1,103 @@
|
||||
---
|
||||
id: capability.wiki.coordination-journal
|
||||
name: Event-Sourced Coordination Journal
|
||||
summary: An append-only, totally-ordered-per-space decision log (overlays, bindings, aliases, merges, forks) whose current state is a derived fold; git-addressable history.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, event-sourcing, coordination, git, journal, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Keystone resolved across two architecture reviews: coordination-canonical state
|
||||
as an append-only decision log with a per-space append authority; current state
|
||||
is a derived fold (derived = f(log)).
|
||||
availability:
|
||||
current: A2
|
||||
target: A4
|
||||
confidence: medium
|
||||
rationale: >
|
||||
In-memory DecisionLog + fold work as a source module; the git-backed store with a
|
||||
per-space lease (the production backing) is planned.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- append-only, totally-ordered-per-space log with read-your-writes
|
||||
- derived fold to aliases + transitively-merged equivalence groups
|
||||
broken_expectations:
|
||||
- git-backed storage and per-space lease/append-authority not yet implemented
|
||||
out_of_scope_expectations:
|
||||
- general-purpose event bus
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- in-memory backing only; cross-process durability pending
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Make coordination-canonical decisions durable and git-addressable as events, with the
|
||||
queryable current state always recomputable by replay.
|
||||
includes:
|
||||
- append-only decision log, totally ordered per information space
|
||||
- derived fold to current coordination state (aliases, equivalence groups, overlays)
|
||||
- per-space append authority (concurrency model)
|
||||
excludes:
|
||||
- storing derived/disposable union state
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-29, UC-33 (history, attribution, coordination journal)"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/coordination/decision_log.py"
|
||||
target_artifacts:
|
||||
- git-backed log store with per-space lease
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
supports:
|
||||
- capability.wiki.shard-orchestration
|
||||
- capability.wiki.overlay
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 8.1)"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_decision_log.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- durable, replayable, git-addressable coordination state for a federated space
|
||||
not_recommended_for:
|
||||
- high-frequency general event streaming
|
||||
known_limitations:
|
||||
- production git backing + lease are still on the roadmap (SHARD-WP-0009)
|
||||
---
|
||||
|
||||
# Event-Sourced Coordination Journal
|
||||
|
||||
The keystone: coordination-canonical state (overlays, equivalence bindings, aliases, merges,
|
||||
forks) is an append-only **decision log**, totally ordered per information space; the queryable
|
||||
current state is a derived **fold** of the log (`derived = f(log)`). The log is git-addressable,
|
||||
giving history/patch/review/backup for coordination decisions for free.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Resolved across the round-1/round-2 architecture reviews (CoreArchitectureBlueprint Section 8.1).
|
||||
|
||||
### Availability
|
||||
`decision_log.py` ships an in-memory, totally-ordered log + fold; git+lease backing is planned.
|
||||
87
registry/capabilities/capability.wiki.derived-views.md
Normal file
87
registry/capabilities/capability.wiki.derived-views.md
Normal file
@@ -0,0 +1,87 @@
|
||||
---
|
||||
id: capability.wiki.derived-views
|
||||
name: Wiki Derived Views
|
||||
summary: Recomputable views over a wiki union — BackLinks, RecentChanges, AllPages, SiteMap, and (delegate-or-derive) Search — carrying provenance.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, derived-views, backlinks, recentchanges, search, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D3
|
||||
target: D5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Core-vs-adapter classification and behaviours are decided (FederationRequirements ADR-03);
|
||||
implementation is planned (SHARD-WP-0010), not built.
|
||||
availability:
|
||||
current: A0
|
||||
target: A4
|
||||
confidence: low
|
||||
rationale: >
|
||||
Designed; no implementation yet. Informational/planning reuse only today.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C0
|
||||
name: Absent
|
||||
confidence: low
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations: []
|
||||
broken_expectations:
|
||||
- no derived view is implemented yet
|
||||
out_of_scope_expectations:
|
||||
- presentation / rendering of views
|
||||
reliability:
|
||||
level: R0
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- planning-stage
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Provide recomputable, provenance-carrying views over the union (link graph, change feed,
|
||||
enumeration, search) without introducing canonical state.
|
||||
includes:
|
||||
- BackLinks (link graph), RecentChanges (journal + shard signals), AllPages, SiteMap
|
||||
- Search as delegate-to-native-or-derive-index
|
||||
excludes:
|
||||
- view presentation / UI
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-17..UC-21, UC-63"
|
||||
|
||||
availability:
|
||||
current_level: A0
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/workplans/SHARD-WP-0010-derived-views.md"
|
||||
consumption_modes:
|
||||
- informational
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.shard-orchestration
|
||||
- capability.wiki.page-model
|
||||
related_to:
|
||||
- capability.wiki.engine-typed-extensions
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/FederationRequirements.md (ADR-03)"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- planning derived navigation/discovery over a federated wiki union
|
||||
not_recommended_for:
|
||||
- implementation reuse today (planning-stage)
|
||||
known_limitations:
|
||||
- not implemented; Search ranking policy undecided
|
||||
---
|
||||
|
||||
# Wiki Derived Views
|
||||
|
||||
Recomputable views over the union (BackLinks, RecentChanges, AllPages, SiteMap, Search). All
|
||||
are derived/disposable (no canonical state) and carry provenance; Search is delegate-to-native
|
||||
where a shard's query capability allows, else a derived index. Planned in SHARD-WP-0010.
|
||||
115
registry/capabilities/capability.wiki.engine-typed-extensions.md
Normal file
115
registry/capabilities/capability.wiki.engine-typed-extensions.md
Normal file
@@ -0,0 +1,115 @@
|
||||
---
|
||||
id: capability.wiki.engine-typed-extensions
|
||||
name: Wiki Engine with Typed Extensions
|
||||
summary: A small-core wiki engine realizing a stringent typed-extension framework that addresses all wiki use cases and lets each shard activate only the features it needs.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, engine, typed-extensions, feature-activation, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D3
|
||||
target: D5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Architecture authored (shard-wiki/spec/WikiEngineCoreArchitecture.md): small page-store
|
||||
kernel + typed-extension framework, per-shard activation, engine-as-canonical-mode-shard,
|
||||
and a conflict-mediation realization are explored. Detailed extension SDK/ABI and the API
|
||||
protocol remain (so D3 Explored, not yet D4/D5).
|
||||
availability:
|
||||
current: A0
|
||||
target: A4
|
||||
confidence: low
|
||||
rationale: >
|
||||
Planned. No engine kernel or extensions exist yet; informational/planning reuse only.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C0
|
||||
name: Absent
|
||||
confidence: low
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations: []
|
||||
broken_expectations:
|
||||
- engine core and typed-extension mechanism not yet designed in detail
|
||||
out_of_scope_expectations:
|
||||
- replacing other wiki engines or mandating one implementation
|
||||
reliability:
|
||||
level: R0
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- planning-stage capability
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Provide shard-wiki's reference first-party shard backend: a small core + a stringent
|
||||
typed-extension framework covering all collected use cases, mediating conflicting
|
||||
requirements into an integrated whole, with per-shard activation (only what you need).
|
||||
includes:
|
||||
- a minimal engine kernel (page lifecycle, storage via the adapter contract, the typing mechanism)
|
||||
- typed extensions that declare contracts and compose
|
||||
- per-shard feature activation
|
||||
excludes:
|
||||
- replacing or mandating other wiki engines (it is one shard type among many)
|
||||
- a single canonical implementation for all wikis
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-08..UC-25 and the full catalog (the engine must cover all)"
|
||||
|
||||
availability:
|
||||
current_level: A0
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/workplans/SHARD-WP-0013-wiki-engine-prep.md"
|
||||
- "shard-wiki/spec/WikiEngineCoreArchitecture.md"
|
||||
consumption_modes:
|
||||
- informational
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.adapter-contract
|
||||
- capability.wiki.page-model
|
||||
related_to:
|
||||
- capability.feature-control.evaluate
|
||||
- capability.authorization.policy-evaluate
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/workplans/SHARD-WP-0013-wiki-engine-prep.md"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- planning a composable, feature-activatable native wiki engine
|
||||
not_recommended_for:
|
||||
- implementation reuse today (planning-stage)
|
||||
known_limitations:
|
||||
- architecture authored; extension SDK/ABI + API protocol still to design; not yet built
|
||||
|
||||
promotion_history:
|
||||
- date: "2026-06-15"
|
||||
dimension: discovery
|
||||
from: D2
|
||||
to: D3
|
||||
rationale: WikiEngineCoreArchitecture.md authored (kernel + typed-extension framework explored); INTENT amendment ratified.
|
||||
author: shard-wiki
|
||||
---
|
||||
|
||||
# Wiki Engine with Typed Extensions
|
||||
|
||||
shard-wiki's planned reference first-party shard backend — a *canonical-mode shard* it
|
||||
implements natively: a small core plus a stringent typed-extension framework addressing all
|
||||
collected use cases, mediating conflicting requirements into a consistent whole, with per-shard
|
||||
activation (activate only what you need). It is one shard type among many — not a replacement
|
||||
for other engines. Per-shard activation is a candidate consumer of
|
||||
`capability.feature-control.evaluate`.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Architecture authored: `shard-wiki/spec/WikiEngineCoreArchitecture.md` (small kernel +
|
||||
typed-extension framework; engine = canonical-mode shard). INTENT amendment ratified
|
||||
(2026-06-15, decision 84ffdb48). Extension SDK/ABI + API protocol are the next deliverables.
|
||||
|
||||
### Availability
|
||||
Planning-stage; informational reuse only.
|
||||
97
registry/capabilities/capability.wiki.federation-models.md
Normal file
97
registry/capabilities/capability.wiki.federation-models.md
Normal file
@@ -0,0 +1,97 @@
|
||||
---
|
||||
id: capability.wiki.federation-models
|
||||
name: Selectable Federation-Model Taxonomy
|
||||
summary: Federation as a plural, composable coordination axis (fork+journal, VCS-replication+ping, query-time graph-join, feed, activity-streams, engine-mirror) selected per space.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, federation, taxonomy, composable, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D4
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
A six-model taxonomy distilled from a ~23-system synthesis, each model anchored in a
|
||||
real system, with capability prerequisites and per-space/per-shard composition rules.
|
||||
availability:
|
||||
current: A0
|
||||
target: A4
|
||||
confidence: low
|
||||
rationale: >
|
||||
Designed and specified (FederationArchitecture T17) but not implemented; informational
|
||||
reuse only today.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C1
|
||||
name: Sparse
|
||||
confidence: low
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- the model taxonomy and selection/composition rules are documented
|
||||
broken_expectations:
|
||||
- no federation transport is implemented yet
|
||||
out_of_scope_expectations:
|
||||
- mandating a single federation mechanism
|
||||
reliability:
|
||||
level: R0
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- design-stage; no runtime evidence
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Treat federation as selectable and composable rather than one mechanism, so each space
|
||||
picks fork+journal, VCS-replication, query-join, feed, activity-streams, or engine-mirror.
|
||||
includes:
|
||||
- the six federation models + their capability floors
|
||||
- per-space selection and per-shard composition
|
||||
excludes:
|
||||
- imposing one homogeneous federation network
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-26, UC-31, UC-33, UC-71, UC-72, UC-74, UC-79"
|
||||
|
||||
availability:
|
||||
current_level: A0
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/spec/FederationArchitecture.md (T17)"
|
||||
consumption_modes:
|
||||
- informational
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.shard-orchestration
|
||||
- capability.wiki.coordination-journal
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/FederationArchitecture.md"
|
||||
- "shard-wiki/research/260614-shard-spectrum-synthesis/findings.md"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- planning a federation strategy that mixes models per source
|
||||
not_recommended_for:
|
||||
- implementation reuse today (design-stage)
|
||||
known_limitations:
|
||||
- no transport implemented; informational planning reuse only
|
||||
---
|
||||
|
||||
# Selectable Federation-Model Taxonomy
|
||||
|
||||
Federation is plural and composable: fork+journal (Federated Wiki), VCS-replication+ping
|
||||
(ikiwiki), query-time graph-join (Wikibase SERVICE), feed aggregation, activity streams
|
||||
(ActivityPub), and engine-mirror (Wiki.js). A space selects a model and composes per shard;
|
||||
the default is fork+journal over git. Design-stage capability — strong for planning reuse.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
FederationArchitecture T17, distilled from the shard-spectrum synthesis (v3).
|
||||
|
||||
### Availability
|
||||
Specified, not implemented — informational reuse only.
|
||||
102
registry/capabilities/capability.wiki.overlay.md
Normal file
102
registry/capabilities/capability.wiki.overlay.md
Normal file
@@ -0,0 +1,102 @@
|
||||
---
|
||||
id: capability.wiki.overlay
|
||||
name: Overlay-Before-Mutation Write Path
|
||||
summary: Non-destructive edits (draft -> patch -> apply-under-drift) that let read-only, rate-limited, or lossy backends be edited safely without silent remote mutation.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, overlay, patch, write-path, conflict, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Overlay lifecycle and apply-under-drift semantics are specified (ADR-05, blueprint
|
||||
Section 8.6) and implemented as a single principled write path.
|
||||
availability:
|
||||
current: A2
|
||||
target: A4
|
||||
confidence: medium
|
||||
rationale: >
|
||||
OverlayEngine (draft/patch/apply), writable adapter, and InformationSpace.edit
|
||||
exist as a source module; three-way merge is not (refuse-on-drift only).
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- draft -> patch -> apply with fast-forward / refuse-on-drift / keep-draft outcomes
|
||||
- no silent remote mutation; overlay_state surfaced in provenance
|
||||
broken_expectations:
|
||||
- three-way / auto merge not implemented (refuse-on-conflict only)
|
||||
out_of_scope_expectations:
|
||||
- federation propagation of applied overlays
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- early implementation; conflict handling is detect-and-refuse only
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Make any sub-write-through backend editable safely: an edit is an overlay first, applied
|
||||
only on explicit intent and only when the source has not drifted.
|
||||
includes:
|
||||
- overlay drafts recorded as coordination-canonical events
|
||||
- patch rendering (unified diff)
|
||||
- apply-under-drift (fast-forward / refuse / keep-draft)
|
||||
excludes:
|
||||
- destructive write without drift check
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-04, UC-26, UC-29 (remix primitives, overlay)"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/coordination/overlay.py"
|
||||
- "shard-wiki/src/shard_wiki/coordination/patch.py"
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.coordination-journal
|
||||
- capability.wiki.adapter-contract
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/FederationRequirements.md (ADR-05)"
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 8.2, 8.6)"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_apply.py"
|
||||
- "shard-wiki/tests/test_write_path_integration.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- safe editing over read-only / rate-limited / lossy backends
|
||||
not_recommended_for:
|
||||
- workflows needing automatic conflict resolution today
|
||||
known_limitations:
|
||||
- merge is detect-and-refuse; three-way merge is future work
|
||||
---
|
||||
|
||||
# Overlay-Before-Mutation Write Path
|
||||
|
||||
One principled write path: every edit drafts an overlay (a coordination-canonical event),
|
||||
renders as a patch, and applies under drift checks — fast-forwarding a writable target,
|
||||
keeping a local draft on a read-only target, and refusing (never clobbering) on external drift.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Specified in FederationRequirements ADR-05 and CoreArchitectureBlueprint Section 8.2/8.6.
|
||||
|
||||
### Availability
|
||||
`overlay.py` + `patch.py` + `InformationSpace.edit` ship the path; built in SHARD-WP-0008.
|
||||
104
registry/capabilities/capability.wiki.page-model.md
Normal file
104
registry/capabilities/capability.wiki.page-model.md
Normal file
@@ -0,0 +1,104 @@
|
||||
---
|
||||
id: capability.wiki.page-model
|
||||
name: Backend-Neutral Wiki Page Model
|
||||
summary: A Markdown-first but stretchable page model with stable identity separate from placement and layered provenance, spanning prose to typed-graph and computational shapes.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, page-model, identity, provenance, markdown, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Page shapes (prose, typed records, typed-graph, inline-embedded, non-Markdown,
|
||||
and four computational shapes) plus identity != placement and layered provenance
|
||||
are specified and grounded in the dive research.
|
||||
availability:
|
||||
current: A2
|
||||
target: A5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Identity/Placement/Span/Page and layered ProvenanceEnvelope exist as a source
|
||||
module; richer shapes (typed-graph, notebook) are modeled but not all built.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- stable identity distinct from placement and from content fingerprint
|
||||
- layered (effective-vs-own) provenance with near-zero per-span cost
|
||||
broken_expectations:
|
||||
- non-prose shapes (typed-graph, notebook, inline-embedded) not fully realized
|
||||
out_of_scope_expectations:
|
||||
- rendering / presentation
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- prose shape is the only exercised path so far
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
One backend-neutral lingua franca every consumer sees; every shape reduces to
|
||||
(content|source, structure, provenance envelope, optional derivation rule).
|
||||
includes:
|
||||
- page identity (stable handle) vs placement (N paths/shards) vs equivalence (fingerprint)
|
||||
- layered provenance envelope (page + span deltas)
|
||||
- page-shape taxonomy incl. computational shapes
|
||||
excludes:
|
||||
- deriving identity from content (a fingerprint identifies a version, not a page)
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-34, UC-39, UC-44..UC-49, UC-55, UC-73, UC-83, UC-84"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A5
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/model/"
|
||||
- "shard-wiki/src/shard_wiki/provenance/"
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
supports:
|
||||
- capability.wiki.adapter-contract
|
||||
- capability.wiki.shard-orchestration
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 7)"
|
||||
- "shard-wiki/spec/FederationRequirements.md (ADR-02, ADR-04)"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_model.py"
|
||||
- "shard-wiki/tests/test_provenance.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- a portable, provenance-carrying representation of wiki pages across backends
|
||||
not_recommended_for:
|
||||
- cases needing a single canonical path per page (use identity, not path)
|
||||
known_limitations:
|
||||
- non-prose shapes specified ahead of implementation
|
||||
---
|
||||
|
||||
# Backend-Neutral Wiki Page Model
|
||||
|
||||
The top narrow waist: a Markdown-first model that stretches to typed records, typed-graph
|
||||
statements, inline-embedded objects, non-Markdown assets, and computational shapes. Identity
|
||||
is a stable handle; placement and equivalence are separate mechanisms; provenance is layered
|
||||
(effective = page envelope + span delta).
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Specified in CoreArchitectureBlueprint Section 7 and FederationRequirements ADR-02/04.
|
||||
|
||||
### Availability
|
||||
`model/` + `provenance/` ship the prose path and the layered envelope today.
|
||||
114
registry/capabilities/capability.wiki.shard-orchestration.md
Normal file
114
registry/capabilities/capability.wiki.shard-orchestration.md
Normal file
@@ -0,0 +1,114 @@
|
||||
---
|
||||
id: capability.wiki.shard-orchestration
|
||||
name: Wiki Shard Orchestration
|
||||
summary: Present a union of pages across heterogeneous wiki-shaped shards while preserving each shard's provenance, capabilities, and history.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, federation, orchestration, union, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Grounded in 84 documented use cases and a twice-reviewed whole-system
|
||||
architecture (CoreArchitectureBlueprint) derived from ~23 prior-art systems.
|
||||
availability:
|
||||
current: A2
|
||||
target: A5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
InformationSpace orchestrator (attach -> resolve -> read, chorus on
|
||||
ambiguity) works as a Python source module; network API and incremental
|
||||
union are planned.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- attach folder shards and read a union page with layered provenance
|
||||
- chorus presentation of equivalent-but-divergent pages (union without erasure)
|
||||
broken_expectations:
|
||||
- incremental union maintenance and equivalence index not yet built
|
||||
- write-through federation transports not yet built
|
||||
out_of_scope_expectations:
|
||||
- hosting or replacing the underlying wiki engines
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- early implementation; 64 tests but no production exposure
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Let independently stored, differently implemented wikis behave as one
|
||||
coherent, versionable, inspectable information space without homogenizing them.
|
||||
includes:
|
||||
- union resolution across shards (identity-keyed)
|
||||
- chorus / designated-canonical presentation of equivalent pages
|
||||
- lazy replication projection of remote content with freshness
|
||||
excludes:
|
||||
- implementing a backend wiki engine (see capability.wiki.engine-typed-extensions)
|
||||
- silent remote mutation
|
||||
assumptions:
|
||||
- canonical truth lives in shards + a git coordination journal; the union is derived
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-01..UC-07, UC-26..UC-33 (information space, federation, coordination)"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A5
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/union/"
|
||||
- "shard-wiki/src/shard_wiki/space.py"
|
||||
target_artifacts:
|
||||
- orchestrator network API
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.adapter-contract
|
||||
- capability.wiki.page-model
|
||||
- capability.wiki.coordination-journal
|
||||
supports:
|
||||
- capability.wiki.federation-models
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md"
|
||||
- "shard-wiki/spec/FederationArchitecture.md"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_union.py"
|
||||
- "shard-wiki/tests/test_integration.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- composing multiple Markdown/wiki stores into one provenance-preserving view
|
||||
not_recommended_for:
|
||||
- replacing a single wiki engine
|
||||
known_limitations:
|
||||
- resolution is recompute-on-read until the incremental tier lands
|
||||
---
|
||||
|
||||
# Wiki Shard Orchestration
|
||||
|
||||
shard-wiki's core capability: orchestrate wiki-shaped content across heterogeneous *shards*
|
||||
as a union of pages, preserving provenance, capabilities, and history per shard. Canonical
|
||||
truth stays at the edges (shards + the git coordination journal); the union is a derived,
|
||||
recomputable view (orchestrator, not engine).
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Grounded by `UseCaseCatalog.md` (84 UCs) and the hardened `CoreArchitectureBlueprint.md`.
|
||||
|
||||
### Availability
|
||||
`InformationSpace` provides attach/resolve/read today (source module); a network API is the
|
||||
target availability step.
|
||||
145
registry/indexes/capabilities.yaml
Normal file
145
registry/indexes/capabilities.yaml
Normal file
@@ -0,0 +1,145 @@
|
||||
version: 1
|
||||
updated: '2026-06-16'
|
||||
domain: helix_forge
|
||||
capabilities:
|
||||
- id: capability.wiki.shard-orchestration
|
||||
name: Wiki Shard Orchestration
|
||||
summary: Present a union of pages across heterogeneous wiki-shaped shards while
|
||||
preserving each shard's provenance, capabilities, and history.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.shard-orchestration.md
|
||||
tags:
|
||||
- wiki
|
||||
- federation
|
||||
- orchestration
|
||||
- union
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.adapter-contract
|
||||
name: Capability-Aware Shard Adapter Contract
|
||||
summary: A versioned backend interface where each binding declares a verified capability
|
||||
profile, so federation ops degrade by capability.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.adapter-contract.md
|
||||
tags:
|
||||
- wiki
|
||||
- adapter
|
||||
- capability
|
||||
- contract
|
||||
- conformance
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.page-model
|
||||
name: Backend-Neutral Wiki Page Model
|
||||
summary: A Markdown-first but stretchable page model with stable identity separate
|
||||
from placement and layered provenance.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.page-model.md
|
||||
tags:
|
||||
- wiki
|
||||
- page-model
|
||||
- identity
|
||||
- provenance
|
||||
- markdown
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.coordination-journal
|
||||
name: Event-Sourced Coordination Journal
|
||||
summary: An append-only, totally-ordered-per-space decision log whose current state
|
||||
is a derived fold; git-addressable history.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.coordination-journal.md
|
||||
tags:
|
||||
- wiki
|
||||
- event-sourcing
|
||||
- coordination
|
||||
- git
|
||||
- journal
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.overlay
|
||||
name: Overlay-Before-Mutation Write Path
|
||||
summary: Non-destructive edits (draft -> patch -> apply-under-drift) that let read-only
|
||||
or limited backends be edited safely without silent remote mutation.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.overlay.md
|
||||
tags:
|
||||
- wiki
|
||||
- overlay
|
||||
- patch
|
||||
- write-path
|
||||
- conflict
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.federation-models
|
||||
name: Selectable Federation-Model Taxonomy
|
||||
summary: Federation as a plural, composable coordination axis (fork+journal, VCS-replication,
|
||||
query-join, feed, activity-streams, engine-mirror) selected per space.
|
||||
vector: D4 / A0 / C1 / R0
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.federation-models.md
|
||||
tags:
|
||||
- wiki
|
||||
- federation
|
||||
- taxonomy
|
||||
- composable
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- informational
|
||||
- id: capability.wiki.engine-typed-extensions
|
||||
name: Wiki Engine with Typed Extensions
|
||||
summary: A small-core wiki engine realizing a typed-extension framework that addresses
|
||||
all wiki use cases and lets each shard activate only the features it needs.
|
||||
vector: D3 / A0 / C0 / R0
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.engine-typed-extensions.md
|
||||
tags:
|
||||
- wiki
|
||||
- engine
|
||||
- typed-extensions
|
||||
- feature-activation
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- informational
|
||||
- id: capability.wiki.derived-views
|
||||
name: Wiki Derived Views
|
||||
summary: Recomputable views over a wiki union — BackLinks, RecentChanges, AllPages,
|
||||
SiteMap, and (delegate-or-derive) Search — carrying provenance.
|
||||
vector: D3 / A0 / C0 / R0
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.derived-views.md
|
||||
tags:
|
||||
- wiki
|
||||
- derived-views
|
||||
- backlinks
|
||||
- recentchanges
|
||||
- search
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- informational
|
||||
23
research/260608-c2-wiki-origins/README.md
Normal file
23
research/260608-c2-wiki-origins/README.md
Normal file
@@ -0,0 +1,23 @@
|
||||
# 260608 — c2 wiki origins (Ward Cunningham)
|
||||
|
||||
Date: 2026-06-08
|
||||
|
||||
## What this is
|
||||
|
||||
Research into the beginnings of the wiki movement through Ward Cunningham,
|
||||
the Portland Pattern Repository, and the original **WikiWikiWeb** at c2.com.
|
||||
|
||||
Focus: popular terms, social conventions, and documented use cases from and
|
||||
about the c2 wiki. **Federation is explicitly out of scope** for this
|
||||
exploration (separate research planned).
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Terms glossary, use-case catalog, design principles, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial exploration complete. Findings may inform `spec/` and future
|
||||
workplans; not yet promoted to specification.
|
||||
327
research/260608-c2-wiki-origins/findings.md
Normal file
327
research/260608-c2-wiki-origins/findings.md
Normal file
@@ -0,0 +1,327 @@
|
||||
# Findings — Ward Cunningham, c2 wiki, and wiki movement origins
|
||||
|
||||
Date: 2026-06-08 · Status: research draft
|
||||
|
||||
Scope: beginnings of the wiki movement via the **WikiWikiWeb** (c2.com) and
|
||||
its creator **Ward Cunningham**. Emphasis on terms and use cases documented
|
||||
in and about the c2 wiki. **Federation concepts are out of scope here.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Historical anchor
|
||||
|
||||
| Fact | Detail |
|
||||
|------|--------|
|
||||
| First wiki | WikiWikiWeb — launched **25 March 1995** on c2.com |
|
||||
| Creator | Ward Cunningham (Cunningham & Cunningham, Portland, Oregon) |
|
||||
| Companion site | [Portland Pattern Repository](http://c2.com/ppr/) (PPR) |
|
||||
| PPR motto | **"People, Projects & Patterns"** |
|
||||
| Original software | Perl CGI script named `wiki`; later called **WikiBase** |
|
||||
| Intellectual roots | HyperCard stacks (late 1980s); pattern languages (Cunningham & Beck, OOPSLA 1987, inspired by Christopher Alexander) |
|
||||
| Name origin | Hawaiian *wiki* = "quick"; *wiki wiki* = "very quick" — from Honolulu airport Wiki Wiki Shuttle |
|
||||
| Intended name alternative | "QuickWeb" rejected in favor of **WikiWikiWeb** as more fun to say |
|
||||
| Capital-W convention | **Wiki** (capital W) = this original site; lowercase *wiki* = the technology genre |
|
||||
|
||||
Ward's stated purpose (WikiHistory, 2002): make exchange of ideas between
|
||||
programmers easier. The wiki was an **automated supplement** to the Portland
|
||||
Pattern Repository, not a replacement encyclopedia.
|
||||
|
||||
By 2015 the site went read-only after vandalism; Ward migrated toward
|
||||
Federated Wiki. That later history is **not** covered here.
|
||||
|
||||
---
|
||||
|
||||
## 2. What c2 said it was for (primary use cases)
|
||||
|
||||
From **WelcomeVisitors**, **WikiHistory**, and community self-description:
|
||||
|
||||
### Core mission use cases
|
||||
|
||||
1. **Pattern exchange** — publish, discuss, and refine software design patterns
|
||||
(the PPR mission).
|
||||
2. **People + projects + patterns** — connect practitioners, project experience,
|
||||
and reusable design knowledge (`PeopleProjectsAndPatterns`).
|
||||
3. **Idea exchange among programmers** — fast, lightweight collaborative
|
||||
writing; "a forum where people share ideas."
|
||||
4. **Informal history of programming ideas** — accumulated narrative, not a
|
||||
formal reference work (`InformalHistoryOfProgrammingIdeas`).
|
||||
5. **Distillation of experience into patterns** — thread discussions refined
|
||||
into document-mode consensus, sometimes culminating in patterns
|
||||
(`ThreadMode` → `DocumentMode` → `PatternMode`).
|
||||
|
||||
### Community / onboarding use cases
|
||||
|
||||
6. **First wiki experience** — deliberate onboarding path for newcomers
|
||||
(`WelcomeVisitors`, `NewUserPages`, `WikiWikiSandbox`).
|
||||
7. **Sandbox experimentation** — safe place to learn editing mechanics.
|
||||
8. **Visitor presence** — sign the guest book (`RecentVisitors`).
|
||||
9. **Culture acclimation** — expect `CultureShock`; learn local conventions
|
||||
before adding pages.
|
||||
|
||||
### Knowledge maintenance use cases
|
||||
|
||||
10. **Self-indexing** — wiki indexes itself; community maintains structure
|
||||
(`VolunteerHousekeeper`, categories, road maps).
|
||||
11. **Activity awareness** — follow what changed (`RecentChanges`,
|
||||
`QuickChanges`, `RecentChangesJunkie`).
|
||||
12. **Discovery** — browse by search, categories, random pages, visual tour,
|
||||
like-pages, starting points.
|
||||
13. **Refactoring pages** — merge thread into document, split tangents,
|
||||
improve clarity (`ReFactoring`, `WikiMaster` role).
|
||||
14. **Work-in-progress knowledge** — all content explicitly provisional
|
||||
(`WorkInProgress`).
|
||||
|
||||
### What c2 explicitly said it was *not*
|
||||
|
||||
- **Not Wikipedia** — subjective, conversational, not a dedicated reference
|
||||
site (`WikiIsNotWikipedia`, `WikiPedia` as pointer to the real Wikipedia).
|
||||
- **Not a polished encyclopedia** — "Most of all, this is a forum where
|
||||
people share ideas!"
|
||||
|
||||
### Subject areas the community grew into (approximate eras, WikiHistory)
|
||||
|
||||
| Era | Dominant topics on c2 |
|
||||
|-----|----------------------|
|
||||
| 1994+ | Patterns, sources, application |
|
||||
| 1996+ | General design, architecture, methods |
|
||||
| 1997+ | People and organizational aspects of programming |
|
||||
| 1998+ | Extreme Programming |
|
||||
| 2000+ | Wiki itself (meta-discussion) |
|
||||
|
||||
Popular page clusters noted by observers: **DesignPatterns**, **ExtremeProgramming**, **TestDrivenDevelopment**, **AgileManifesto**, **UnitTest**, **AntiPattern**, build tools, languages, organizational culture.
|
||||
|
||||
---
|
||||
|
||||
## 3. Ward's WikiDesignPrinciples (1995 intent)
|
||||
|
||||
From Ward Cunningham's reconstructed **WikiDesignPrinciples** page:
|
||||
|
||||
| Principle | Meaning |
|
||||
|-----------|---------|
|
||||
| **Open** | Incomplete or poorly organized pages may be edited by any reader |
|
||||
| **Incremental** | Pages cite other pages, including ones not yet written |
|
||||
| **Organic** | Site structure is editable and evolves like page text |
|
||||
| **Mundane** | Small number of irregular text conventions for markup |
|
||||
| **Universal** | Editing and organizing use the same mechanisms as writing |
|
||||
| **Overt** | Formatted output suggests the input needed to reproduce it |
|
||||
| **Unified** | Page names from a flat namespace — no extra context to interpret |
|
||||
| **Precise** | Titles are precise noun phrases to avoid name clashes |
|
||||
| **Tolerant** | Interpretable behavior preferred over error messages |
|
||||
| **Observable** | Site activity can be watched and reviewed by any visitor |
|
||||
| **Convergent** | Duplication discouraged by finding and citing related content |
|
||||
|
||||
Ward later noted additional forced principles (e.g. server robustness) beyond
|
||||
the original eleven.
|
||||
|
||||
---
|
||||
|
||||
## 4. Popular terms glossary
|
||||
|
||||
Terms below are **CamelCase page names** as used on c2 unless noted.
|
||||
|
||||
### Wiki mechanics & naming
|
||||
|
||||
| Term | Meaning on c2 |
|
||||
|------|---------------|
|
||||
| **Wiki** / **WikiWikiWeb** | The first wiki site; capital-W Wiki = this site specifically |
|
||||
| **WikiBase** | The Perl wiki engine behind c2 |
|
||||
| **WikiWord** / **WikiWords** | MixedCase token the engine treats as a hyperlink |
|
||||
| **WikiName** / **WikiNames** | Page title / link target (often built from WikiWords) |
|
||||
| **MixedCase** / **CamelCase** | CapitalizedWordsRunTogether → automatic link |
|
||||
| **TextFormattingRules** | Minimal markup conventions (paragraphs, bold, links) |
|
||||
| **EditText** / **EditCopy** | Core edit operations |
|
||||
| **Undefined page** / **red link** | `PageName?` with `?` — page does not exist yet; click to create |
|
||||
| **Flat namespace** | All page names in one space (no hierarchical paths) |
|
||||
|
||||
### Content modes (social conventions, not software features)
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **DocumentMode** | Third-person, unsigned, community-owned text; consensus artifact |
|
||||
| **ThreadMode** | Signed, first-person discussion; conversation thread |
|
||||
| **PatternMode** | Distilled pattern statements from converged discussion |
|
||||
| **OpeningStatement** | Strong initial claim/question that frames a page |
|
||||
| **SignedDocumentMode** | Hybrid: document-like but attributed |
|
||||
| **ThreadModeConsideredHarmful** | Community norm pushing toward document mode |
|
||||
| **InFavorOfDissertation** | Ward's preference for essay-like pages over dialog |
|
||||
|
||||
### Roles & community behavior
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **WikiMaster** | Person who refactors threads into documents (anyone can be one) |
|
||||
| **VolunteerHousekeeper** | Community members who maintain indexes and hygiene |
|
||||
| **RecentChangesJunkie** | Person who obsessively monitors RecentChanges |
|
||||
| **InvitedAuthors** | Early contributors who bootstrapped site culture |
|
||||
| **GoodStyle** | Ward's editing advice: factual, concrete, civil, flow over chronology |
|
||||
| **ReFactoring** / **RefactorMercilessly** | Rewrite pages for clarity; merge/split threads |
|
||||
| **RefactorDontDelete** | Prefer refactoring over deletion |
|
||||
| **CultureShock** | Newcomer disorientation from wiki norms |
|
||||
|
||||
### Navigation & derived views
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **RecentChanges** | Chronological edit log — central community nerve center |
|
||||
| **QuickChanges** | Short-form recent changes |
|
||||
| **RecentEdits** | Minor edits stream |
|
||||
| **NotSoRecentChanges** | Older change archives |
|
||||
| **FindPage** | Search |
|
||||
| **LikePages** | Similar-title suggestions |
|
||||
| **RandomPages** | Serendipitous browse |
|
||||
| **VisualTour** | Graphical browse aid |
|
||||
| **StartingPoints** | Curated entry paths |
|
||||
| **WikiCategories** / **RoadMaps** | Community-maintained indexes |
|
||||
| **PeopleIndex** / **RecentVisitors** | Who participates |
|
||||
| **BackLinks** | Pages linking here (classic derived view) |
|
||||
| **AllPages** / **SiteMap** / **SearchPage** | Core derived pages (also present in yawex prior art) |
|
||||
| **SisterSites** | Links to other wikis |
|
||||
|
||||
### Pattern & programming vocabulary (c2's dominant content)
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **Pattern** / **DesignPatterns** | Reusable design solution in context |
|
||||
| **AntiPattern** | Common bad solution |
|
||||
| **ProtoPattern** | Pattern-in-formation |
|
||||
| **PortlandPatternRepository** | Pattern submission/distribution site |
|
||||
| **PeopleProjectsAndPatterns** | c2's thematic focus |
|
||||
| **ExtremeProgramming** / **AgileManifesto** | Major c2 discourse topics (late 1990s–2000s) |
|
||||
| **CrcCards** | Design technique; also spawned from Ward's HyperCard work |
|
||||
|
||||
### Meta & identity pages
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **WorkInProgress** | Nothing is finished; everything editable |
|
||||
| **WikiIsNotWikipedia** | Identity boundary vs encyclopedic wikis |
|
||||
| **WhyWikiWorks** / **WhyWikiWorksNot** | Community theory of success/failure |
|
||||
| **WikiZen** | Cultural / philosophical reflection |
|
||||
| **WikiOnWiki** | Meta-discussion about wiki itself (sometimes controversial) |
|
||||
|
||||
---
|
||||
|
||||
## 5. Documented editing & collaboration patterns
|
||||
|
||||
From **ThreadMode**, **DocumentMode**, **GoodStyle**, **WhyWikiWorks**:
|
||||
|
||||
### ThreadMode contributions (ADD / EDIT / SPLIT / CAPTURE)
|
||||
|
||||
1. **ADD** — append signed comment to continue conversation.
|
||||
2. **EDIT** — improve flow of others' signed comments (with care).
|
||||
3. **SPLIT** — separate tangents onto new pages with summary links.
|
||||
4. **CAPTURE** — distill converging ideas into anonymous pattern paragraphs.
|
||||
|
||||
### DocumentMode lifecycle
|
||||
|
||||
- Page opens with strong **OpeningStatement**.
|
||||
- Feedback arrives in ThreadMode.
|
||||
- On consensus, a **WikiMaster** (or anyone) replaces thread with
|
||||
DocumentMode synthesis.
|
||||
- Newcomers are expected to **rewrite** unclear passages rather than stack
|
||||
clarifying replies.
|
||||
|
||||
### WhyWikiWorks (community theory)
|
||||
|
||||
Paradoxical strengths cited on c2:
|
||||
|
||||
- Anyone can delete anything → only meaningful content survives curation.
|
||||
- Low WYSIWYG appeal → filters out drive-by noise; participants self-select.
|
||||
- Slow, considered edits → pages evolve over days/weeks.
|
||||
- Pedantic community → shared professional camaraderie.
|
||||
- "Insecure, indiscriminate, user-hostile, slow" — yet it worked *because*
|
||||
other online communities optimized differently.
|
||||
|
||||
---
|
||||
|
||||
## 6. Technical & formatting conventions (original c2)
|
||||
|
||||
From Ward's etymology correspondence and c2 pages:
|
||||
|
||||
- **Double conventions as formatting clues:**
|
||||
- double carriage-return → new paragraph
|
||||
- double single-quote → italic
|
||||
- double capitalized-word (CamelCase) → hyperlink
|
||||
- **Flat page namespace** — precise noun-phrase titles.
|
||||
- **Minimal markup** — "mundane" rules; overt correspondence between input
|
||||
and rendered output.
|
||||
- **Optional attribution** — `UserName` (2000+) to attach edits to a name
|
||||
instead of IP; signing in ThreadMode remained common.
|
||||
- **Deletion** — supported but culturally discouraged vs refactoring
|
||||
(`RefactorDontDelete`).
|
||||
|
||||
---
|
||||
|
||||
## 7. Use-case patterns worth naming (synthesis)
|
||||
|
||||
Grouping c2-documented uses into reusable patterns:
|
||||
|
||||
| ID | Use-case pattern | c2 evidence |
|
||||
|----|------------------|-------------|
|
||||
| UC-C2-01 | **Quick idea capture** | Wiki = quick web; incremental linking to unwritten pages |
|
||||
| UC-C2-02 | **Collaborative glossary** | Flat namespace of precise terms; WikiWords as vocabulary |
|
||||
| UC-C2-03 | **Discussion → consensus doc** | ThreadMode → DocumentMode refactoring |
|
||||
| UC-C2-04 | **Pattern mining** | Thread → PatternMode distillation |
|
||||
| UC-C2-05 | **Community guest book** | RecentVisitors, people pages |
|
||||
| UC-C2-06 | **Change radar** | RecentChanges / QuickChanges monitoring |
|
||||
| UC-C2-07 | **Self-curating knowledge base** | Open editing + convergent deduplication |
|
||||
| UC-C2-08 | **Sandbox learning** | WikiWikiSandbox for safe first edits |
|
||||
| UC-C2-09 | **Serendipitous browse** | RandomPages, VisualTour, LikePages |
|
||||
| UC-C2-10 | **Practitioner field notes** | InformalHistoryOfProgrammingIdeas, not encyclopedia |
|
||||
| UC-C2-11 | **Team memory for methods** | XP, TDD, patterns, tools discourse |
|
||||
| UC-C2-12 | **Soft creation of missing topics** | Red-link `?` pages created on first write |
|
||||
|
||||
---
|
||||
|
||||
## 8. Boundaries for shard-wiki (light notes, not federation)
|
||||
|
||||
Items from c2 origins that align with existing `INTENT.md` themes **without**
|
||||
entering federation design:
|
||||
|
||||
- **Open editing + recoverable history** — c2 trusted the community; shard-wiki
|
||||
INTENT makes Git history the safety net (stronger than c2's soft norms).
|
||||
- **WorkInProgress** — matches overlay/provenance/freshness thinking.
|
||||
- **Mechanism over policy** — c2 used social conventions (DocumentMode,
|
||||
GoodStyle) rather than hard gates; shard-wiki keeps policy configurable.
|
||||
- **Not an encyclopedia** — c2's `WikiIsNotWikipedia` parallels shard-wiki not
|
||||
owning universal ontology.
|
||||
- **Derived views** — BackLinks, RecentChanges, AllPages, SiteMap, Search were
|
||||
first-class on c2; already flagged in yawex research as union-view candidates.
|
||||
- **Flat namespace + CamelCase links** — original c2 model; shard-wiki is
|
||||
Markdown-first (wikilink extension question remains open).
|
||||
|
||||
**Deferred:** Federated Wiki, sister sites, multi-wiki configuration — separate
|
||||
research track.
|
||||
|
||||
---
|
||||
|
||||
## 9. Sources
|
||||
|
||||
| Source | URL / location |
|
||||
|--------|----------------|
|
||||
| Ward — WikiHistory | https://wiki.c2.com/?WikiHistory (archive 2002) |
|
||||
| Ward — WikiDesignPrinciples | https://wiki.c2.com/?WikiDesignPrinciples (archive 2002) |
|
||||
| Ward — etymology correspondence | https://c2.com/doc/etymology.html |
|
||||
| WelcomeVisitors | https://wiki.c2.com/?WelcomeVisitors (archive 2002) |
|
||||
| DocumentMode | https://wiki.c2.com/?DocumentMode (archive 2002) |
|
||||
| ThreadMode | https://wiki.c2.com/?ThreadMode (archive 2002) |
|
||||
| GoodStyle | https://wiki.c2.com/?GoodStyle (archive 2002) |
|
||||
| WhyWikiWorks | https://wiki.c2.com/?WhyWikiWorks (archive 2002) |
|
||||
| RecentChanges | https://wiki.c2.com/?RecentChanges (archive 2002) |
|
||||
| Wikipedia — WikiWikiWeb | https://en.wikipedia.org/wiki/WikiWikiWeb |
|
||||
| Wikipedia — Portland Pattern Repository | https://en.wikipedia.org/wiki/Portland_Pattern_Repository |
|
||||
| Observer summary — T.J. Maher (2016) | https://www.tjmaher.com/2016/06/time-capsule-ward-cunninghams-wiki-wiki.html |
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions (for later spec work)
|
||||
|
||||
1. Which c2 **social conventions** (DocumentMode, GoodStyle, WikiMaster
|
||||
refactoring) belong in shard-wiki core vs reference UI vs `wiki/` content?
|
||||
2. How much of the **flat CamelCase namespace** survives in a Markdown-first,
|
||||
path-oriented federation model?
|
||||
3. Which **derived views** from c2 are MVP for an orchestrator vs adapter-provided?
|
||||
4. Does shard-wiki want an explicit **WorkInProgress / WikiIsNotWikipedia**
|
||||
stance in `spec/ProductRequirementsDocument.md`?
|
||||
5. How does c2's **open-by-default** community norm relate to shard-wiki's
|
||||
L0 mode — coincidence of values or intentional lineage?
|
||||
28
research/260608-federation-concepts/README.md
Normal file
28
research/260608-federation-concepts/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# 260608 — Federation concepts for unified information spaces
|
||||
|
||||
Date: 2026-06-08
|
||||
|
||||
## What this is
|
||||
|
||||
Research into **federation models** for decentralized but unified knowledge
|
||||
spaces — with emphasis on **Federated Wiki** (Ward Cunningham, ~2011+) and
|
||||
comparable approaches (git-backed wikis, ActivityPub, Xanadu patterns, yawex
|
||||
REMOTE/VIRTUAL).
|
||||
|
||||
Complements `research/260608-c2-wiki-origins/` (pre-federation wiki culture)
|
||||
and `research/260608-yawex-prior-art/` (Perl engine resolution states).
|
||||
|
||||
Focus: terms, architectural patterns, and **mapping to `shard-wiki` INTENT**
|
||||
— not implementation specification.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Glossary, model comparison, shard-wiki mapping, open questions, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial exploration complete. Use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-26–UC-33). Federation architecture design tracked in
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
409
research/260608-federation-concepts/findings.md
Normal file
409
research/260608-federation-concepts/findings.md
Normal file
@@ -0,0 +1,409 @@
|
||||
# Findings — Federation concepts for decentralized, unified information spaces
|
||||
|
||||
Date: 2026-06-08 · Status: research draft
|
||||
|
||||
Scope: federation models that let independently stored knowledge participate in
|
||||
a **joined view** without erasing provenance, sovereignty, or backend
|
||||
differences. Primary anchor: **Federated Wiki** and Ward Cunningham's
|
||||
post-c2 inversion. Secondary: git-backed wikis, ActivityPub wiki extensions,
|
||||
Xanadu hypertext patterns, and yawex multi-wiki resolution.
|
||||
|
||||
**Complements:** `research/260608-c2-wiki-origins/` (federation explicitly
|
||||
deferred there) and `research/260608-yawex-prior-art/` (REMOTE/VIRTUAL as
|
||||
adapter foreshadowing).
|
||||
|
||||
---
|
||||
|
||||
## 1. Why federation (the problem statement)
|
||||
|
||||
Centralized wikis concentrate **meaning** and **records** on one server:
|
||||
|
||||
| Problem | Manifestation |
|
||||
|---------|---------------|
|
||||
| **Ownership** | Contributors build on someone else's host; site shutdown = loss risk |
|
||||
| **Consensus pressure** | Early edits must survive notability/deletion fights (Wikipedia model) |
|
||||
| **Heat death** | Bounded communities exhaust their charter; maintenance mode → colony collapse |
|
||||
| **Reuse friction** | Copy-paste HTML/DB content across sites is slow, lossy, attribution-prone |
|
||||
| **Backend lock-in** | One engine, one storage model, one permission model |
|
||||
|
||||
Federation responses split along two axes:
|
||||
|
||||
1. **Where records live** — per-user site, per-instance repo, personal pod, mirrored git clone.
|
||||
2. **How unity is achieved** — fork + spread, git push/pull, protocol activities, projection/cache, orchestrated union.
|
||||
|
||||
`shard-wiki` (per `INTENT.md`) targets axis (2) via an **orchestration layer**
|
||||
over heterogeneous **shards**, with Git as coordination substrate — not a
|
||||
single federation mechanism.
|
||||
|
||||
---
|
||||
|
||||
## 2. Federated Wiki (Smallest Federated Wiki / SFW)
|
||||
|
||||
### 2.1 Historical anchor
|
||||
|
||||
| Fact | Detail |
|
||||
|------|--------|
|
||||
| Origin | Ward Cunningham; launched at IndieWebCamp 2011 |
|
||||
| Motivation | Regret that c2's 35,000+ pages lived on Ward's server, not contributors' |
|
||||
| c2 migration | WikiWikiWeb moved to Federated Wiki ~2015 after read-only period |
|
||||
| GitHub lineage | Fork button inspired by GitHub; "radical code sharing" as model |
|
||||
| Current code | [github.com/fedwiki/wiki](https://github.com/fedwiki/wiki) (CoffeeScript; active releases through 2025) |
|
||||
|
||||
Ward's framing (Wired, 2012): the wiki's radical idea was an **edit button on
|
||||
every page**; federated wiki's radical idea is a **fork button on every page**.
|
||||
|
||||
### 2.2 Inverted architecture
|
||||
|
||||
Traditional wiki vs federated wiki (Caulfield, NWACC 2014, summarizing Ward):
|
||||
|
||||
```
|
||||
Traditional: many people → one server → server owns records → consensus on server
|
||||
Federated: many people → many servers → each owns records → browser composes union
|
||||
```
|
||||
|
||||
**Meaning is made in the browser.** The client pulls JSON page records from
|
||||
multiple origins (CORS) and presents them as one navigable space. Consensus
|
||||
emerges through **which versions spread** across the network, not through a
|
||||
single server arbiter.
|
||||
|
||||
### 2.3 Page model (JSON + journal)
|
||||
|
||||
From [viki.wiki JSON schema](https://viki.wiki/json-schema.html):
|
||||
|
||||
```
|
||||
page = { title, story, journal }
|
||||
story = [ item ] # paragraph-like plugin items
|
||||
journal = [ action ] # edit history that reconstructs story
|
||||
action.type = create|add|move|edit|remove|fork
|
||||
action.site = origin host # present when content came from elsewhere
|
||||
```
|
||||
|
||||
Key properties:
|
||||
|
||||
| Property | Implication |
|
||||
|----------|-------------|
|
||||
| **Data files, not DB render** | Fork copies JSON, not scraped HTML — reuse stays structured |
|
||||
| **JSON items, not HTML** | Target site re-renders via plugins; display differences don't break remix |
|
||||
| **Journal travels with page** | Attribution/history portable; no separate blame log |
|
||||
| **Plugin architecture** | Item `type` selects renderer; missing plugins surfaced to user |
|
||||
| **Fork action** | Whole-page copy from remote site into local site with provenance |
|
||||
| **Item IDs preserved** | Stable anchors through edits; aliasing guards duplicate IDs |
|
||||
|
||||
### 2.4 Interaction patterns
|
||||
|
||||
| Pattern | Behavior |
|
||||
|---------|----------|
|
||||
| **Fork** | Copy remote page to your site; edit locally; original owner may merge back |
|
||||
| **Drag-and-drop fork** | Cross-site page transfer in ~15 seconds (OER case) |
|
||||
| **Chorus of voices** | Multiple versions of "same" topic coexist; linked, not merged by default |
|
||||
| **Neighborhood / river** | Discovery via recent changes across federated sites (e.g. fedwikiriver.com) |
|
||||
| **Happening** | Time-bounded collaborative event on a topic-specific subdomain/site |
|
||||
|
||||
### 2.5 Mike Caulfield — lifecycle and pedagogy
|
||||
|
||||
Caulfield articulated fedwiki's social model beyond Ward's mechanics:
|
||||
|
||||
**Kinneavy triangle / knowledge lifecycle** (composition theory):
|
||||
|
||||
| Phase | Mode | Typical medium | Fedwiki fit |
|
||||
|-------|------|----------------|-------------|
|
||||
| **I** | Personal capture | Notes, journal | Each person's site |
|
||||
| **You** | Dialogic | Conversation, annotation | Fork chains, cross-site references |
|
||||
| **It** | Expository | Stable shared article | Spread of refined forks |
|
||||
|
||||
Fedwiki supports **I → You → It** in one system; centralized wikis force **It**
|
||||
(consensus) from the first edit.
|
||||
|
||||
**Bounded conversations / expected heat death** (2015):
|
||||
|
||||
- Wiki sites are **bounded conversations**, not permanent monuments.
|
||||
- Sites are **expected to die**; valuable material is **forked into the next happening**.
|
||||
- **Reverse bit-rot** — Smalltalk-era term Ward revived: objects/pages improve each
|
||||
time reused across systems (refactor on fork), defying entropy of abandoned forums.
|
||||
- Contrast: colony collapse on centralized forums (Blue Hampshire example) vs
|
||||
humane site retirement with selective carry-forward.
|
||||
|
||||
**OER reuse** (2015): WordPress copy-paste ~10–15 min/page vs fedwiki fork ~15 sec;
|
||||
bottleneck is software, not human willingness to remix.
|
||||
|
||||
**Dissent as feature** (Ogden via Wired): Wikipedia forces one perspective;
|
||||
fedwiki enables multiple controversial pages, still linked for exploration.
|
||||
|
||||
### 2.6 Strengths and limits
|
||||
|
||||
| Strengths | Limits |
|
||||
|-----------|--------|
|
||||
| Personal sovereignty over pages | Requires running a site (mitigated by one-click AWS deploy) |
|
||||
| Low-friction remix with provenance | Small niche ecosystem |
|
||||
| Multi-perspective knowledge | Weak team-governance / permissions story vs enterprise wikis |
|
||||
| JSON portability | Plugin fragmentation; non-Markdown item model |
|
||||
| Strong conceptual clarity | Not a general adapter layer for arbitrary backends |
|
||||
|
||||
---
|
||||
|
||||
## 3. Other federation models
|
||||
|
||||
### 3.1 Git-backed distributed wikis (ikiwiki pattern)
|
||||
|
||||
[ikiwiki distributed wikis](https://ikiwiki.info/tips/distributed_wikis/) documents
|
||||
a **spectrum of decentralization**:
|
||||
|
||||
| Level | Setup | Federation mechanism |
|
||||
|-------|-------|---------------------|
|
||||
| 0 | Single server | None |
|
||||
| 1 | HTML mirror | Read-only copy |
|
||||
| 2 | Split web + git host | Clone/pull |
|
||||
| 3 | Pinger mirrors | Central bare repo + ping-on-edit propagation |
|
||||
| 4 | Fully decentralized | Independent wikis push/pull via git to peers |
|
||||
|
||||
**Branching a wiki** = clone origin read-only (`git://`), edit locally without
|
||||
push-back — explicit fork. Conflicts surface as markers in rendered pages.
|
||||
|
||||
**Relevance to shard-wiki:** Strong alignment with **Git-addressable
|
||||
coordination** and **coordination journal** in `INTENT.md`. ikiwiki federates
|
||||
*homogeneous* git-backed wikis; shard-wiki must also attach non-git shards
|
||||
(Obsidian, WebDAV, Gitea wiki, Coulomb, etc.).
|
||||
|
||||
**Gollum / GitHub wikis:** Same backing-store idea — pages as files in a repo;
|
||||
federation is git sync, not a wiki protocol. Multi-repo Gollum setups are
|
||||
operational, not semantic union.
|
||||
|
||||
### 3.2 ActivityPub wiki federation (XWiki)
|
||||
|
||||
[XWiki ActivityPub Application](https://extensions.xwiki.org/xwiki/bin/view/Extension/ActivityPub%20Application/)
|
||||
(v1.7.x) connects XWiki instances to the **fediverse**:
|
||||
|
||||
| Capability | Notes |
|
||||
|------------|-------|
|
||||
| Follow user / wiki actor | Receive Create/Update notifications |
|
||||
| Share documents | Push page content to followers; view remote in modal |
|
||||
| Messaging / discussions | Note objects threaded as discussions |
|
||||
| Like, mention | Social engagement primitives |
|
||||
| Webfinger + ActivityPub endpoints | Standard discovery |
|
||||
|
||||
**Caveats (documented by XWiki):** Federation "not well supported yet" for
|
||||
full activity routing; Mastodon mention of Page entities incomplete; security
|
||||
signing partial.
|
||||
|
||||
**Contrast with fedwiki:** ActivityPub federates **activities and notifications**
|
||||
between **compatible app instances**; content may be copied on share, but the
|
||||
model is social-graph + event stream, not per-user fork sovereignty. Closer to
|
||||
**fediverse** than **fedwiki** UX.
|
||||
|
||||
**Contrast with shard-wiki:** ActivityPub could be an **adapter transport** for
|
||||
some shards, but shard-wiki's core is wiki-page semantics (paths, links,
|
||||
overlays, provenance), not activity streams.
|
||||
|
||||
### 3.3 Xanadu / transclusion lineage
|
||||
|
||||
Ted Nelson's [Project Xanadu](https://en.wikipedia.org/wiki/Project_Xanadu)
|
||||
(1960s–) pursued **serious electronic literature** with patterns still relevant:
|
||||
|
||||
| Pattern | Intent | Modern partial implementations |
|
||||
|---------|--------|-------------------------------|
|
||||
| **Visible links** | Show destination context, not opaque jumps | Hover previews, Open Graph unfurls |
|
||||
| **Parallel documents** | Side-by-side source + derivative | Multi-pane editors (Obsidian, Roam) |
|
||||
| **Transclusion** | Live inclusion of remote span with origin pointer | Embeds, block references (weak) |
|
||||
| **Transcopyright** | Permissioning for reuse | Mostly unsolved at web scale |
|
||||
| **Stable addresses** | Fine-grained, durable pointers | Block UIDs (Roam); URL rot elsewhere |
|
||||
| **Bi-directional links** | Backlinks as first-class | Obsidian, Roam, shard-wiki BackLinks UC |
|
||||
|
||||
Xanadu is **speculative design / pattern language**, not deployable federation.
|
||||
Useful for shard-wiki **provenance, transclusion, and link resolution** thinking —
|
||||
especially union BackLinks and projection freshness.
|
||||
|
||||
### 3.4 Personal / local-first wikis (TiddlyWiki, Obsidian)
|
||||
|
||||
Not federation protocols, but **sovereignty** models:
|
||||
|
||||
- **TiddlyWiki** — single-file portable wiki; complete user ownership.
|
||||
- **Obsidian** — local vault + optional sync; community plugins for publish/remix.
|
||||
|
||||
Shard-wiki INTENT explicitly lists these as shard participants via adapters.
|
||||
Federation is **attachment to a root information space**, not replacing the
|
||||
local tool.
|
||||
|
||||
### 3.5 yawex multi-wiki resolution (cross-reference)
|
||||
|
||||
From `research/260608-yawex-prior-art/findings.md`:
|
||||
|
||||
| yawex state | Federation reading |
|
||||
|-------------|-------------------|
|
||||
| `REMOTE` | Jump to another wiki's page — remote shard navigation |
|
||||
| `VIRTUAL` | Local processing, remote content — projection |
|
||||
| Multi-wiki config | Multiple named wikis = multiple shards |
|
||||
|
||||
**Decision already recorded:** inspiration only; shard-wiki designs resolution
|
||||
fresh for heterogeneous adapters.
|
||||
|
||||
### 3.6 Solid / personal data pods (brief)
|
||||
|
||||
Solid aims at **user-controlled pods** + linked data apps. Distributed wiki
|
||||
discussions (forum threads, TiddlyWiki-on-Solid experiments) explore **personal
|
||||
wikis on pods** vs **team commons**.
|
||||
|
||||
| Axis | Solid-style | Fedwiki | shard-wiki |
|
||||
|------|-------------|---------|------------|
|
||||
| Unit of sovereignty | Pod / user | Site / person | Shard |
|
||||
| Unity mechanism | Linked data queries | Browser composition + fork | Orchestrated union + Git journal |
|
||||
| Team wiki | Secondary; app-dependent | Weak | Supported via shards + authz |
|
||||
|
||||
No mature Solid wiki federation standard emerged; treat as **adjacent personal-
|
||||
data sovereignty** research, not direct prior art.
|
||||
|
||||
---
|
||||
|
||||
## 4. Comparison matrix
|
||||
|
||||
| Model | Unit of ownership | Unity mechanism | Consensus model | History / provenance | Heterogeneous backends |
|
||||
|-------|-------------------|-----------------|-----------------|----------------------|------------------------|
|
||||
| **Centralized wiki** (c2, MediaWiki) | Site operator | Single namespace | On-page consensus | Server revision log | No |
|
||||
| **Federated Wiki** | Per-site (per person) | Browser pulls JSON; fork | Spread of versions | Journal per page | No (all SFW-shaped) |
|
||||
| **ikiwiki distributed** | Per git clone | git push/pull, pingers | Merge conflicts in git | Git commits | No (all ikiwiki) |
|
||||
| **ActivityPub (XWiki)** | Per instance | Activity stream | Social follow/share | Activity + page store | No (XWiki only) |
|
||||
| **Xanadu** | Document address | Transclusion | Parallel versions | Version trails (concept) | N/A (unbuilt) |
|
||||
| **shard-wiki (INTENT)** | Per shard | Orchestrator union | Configurable policy | Git coordination journal + shard history | **Yes** (adapter contract) |
|
||||
|
||||
---
|
||||
|
||||
## 5. Glossary (federation track)
|
||||
|
||||
| Term | Meaning | Primary source |
|
||||
|------|---------|----------------|
|
||||
| **Fork** | Copy a page (or site branch) to a new home; divergent editing | Fedwiki, git |
|
||||
| **Journal** | Ordered actions reconstructing page state; travels with page | Fedwiki JSON schema |
|
||||
| **Story** | Array of typed items (paragraph, image, …) forming page body | Fedwiki |
|
||||
| **Happening** | Time-bounded fedwiki collaboration on a topic site | Caulfield |
|
||||
| **Bounded conversation** | Site with expected end; not infinite commons | Caulfield |
|
||||
| **Reverse bit-rot** | Content improves through reuse across contexts | Caulfield / Ward |
|
||||
| **Chorus of voices** | Multiple perspectives on same topic, linked | Ward |
|
||||
| **Inverted model** | Records distributed; meaning composed client-side | Ward |
|
||||
| **Projection** | Local view of remote content without claiming ownership | shard-wiki / yawex VIRTUAL |
|
||||
| **Overlay** | Local edit against remote/canonical without immediate mutation | shard-wiki |
|
||||
| **Shard** | Independently meaningful page store with own capabilities | shard-wiki |
|
||||
| **Coordination journal** | Git-backed change record for an information space | shard-wiki |
|
||||
| **Transclusion** | Include live span from elsewhere with origin visible | Xanadu |
|
||||
| **Fediverse** | Network of ActivityPub-speaking apps | XWiki AP, Mastodon |
|
||||
| **Pinger / pingee** | Edit-triggered mirror refresh between ikiwiki sites | ikiwiki |
|
||||
| **Wiki actor** | Entire wiki as ActivityPub actor | XWiki AP |
|
||||
|
||||
---
|
||||
|
||||
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 6.1 Strong resonances
|
||||
|
||||
| Fedwiki / federation idea | shard-wiki concept | Notes |
|
||||
|---------------------------|-------------------|-------|
|
||||
| Personal ownership of edits | Shard sovereignty | shard-wiki generalizes beyond per-person sites |
|
||||
| Fork with provenance | Overlay + patch flows | shard-wiki separates draft from destructive apply |
|
||||
| Journal / history with page | Coordination journal + shard revision | Git journal is space-level; shard retains own history |
|
||||
| Browser-composed union | Union of pages across shards | Orchestrator presents coherent graph |
|
||||
| JSON not HTML for remix | Markdown-first page model | Different format; same structural intent |
|
||||
| Lazy pull from remote | Projection | yawex VIRTUAL; explicit freshness |
|
||||
| Chorus of voices | Union without erasure | Show provenance, divergence, equivalents |
|
||||
| Bounded conversation | Configurable policy | Mechanism not policy — space may be permanent or ephemeral |
|
||||
| Reverse bit-rot on reuse | Reconciliation / sync | Policy-driven; not automatic fork |
|
||||
| Derived views (recent, links) | UC-05 union views | BackLinks, RecentChanges across shards |
|
||||
|
||||
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
| Fedwiki assumption | shard-wiki correction |
|
||||
|--------------------|----------------------|
|
||||
| Every participant runs SFW-shaped JSON site | **Adapter contract** for Gitea, folders, Obsidian, WebDAV, engines |
|
||||
| Fork = default edit primitive | **Capability-aware**: read-only shards need overlay, not fork |
|
||||
| Client composes in browser only | Orchestrator may serve CLI, agents, CI — not browser-only |
|
||||
| Per-site = per-person | Shards may be team repos, org wikis, app databases |
|
||||
| No central coordination | **Git-addressable coordination layer** per information space |
|
||||
| Plugin item types | Markdown-first; engine-specific render is out of core scope |
|
||||
| Implicit spread-as-consensus | **Mechanism over policy** — canonical source is explicit config |
|
||||
|
||||
### 6.3 What shard-wiki adds beyond fedwiki
|
||||
|
||||
1. **Heterogeneous attachment** — not all shards speak the same page JSON.
|
||||
2. **Capability matrix** — read/write/diff/merge/lock/publish per shard.
|
||||
3. **Overlay-before-mutation** — respect remote sovereignty and limited backends.
|
||||
4. **Semantic wiki sync** — not generic file mirroring.
|
||||
5. **Enterprise authz ladder** — L0 open → delegated IAM (without owning identity).
|
||||
6. **Divergence detection** — equivalent pages across shards, reconcilable.
|
||||
|
||||
### 6.4 What fedwiki teaches that shard-wiki should not lose
|
||||
|
||||
1. **Frictionless reuse** — if remix takes 15 minutes, federation fails socially.
|
||||
2. **Provenance by default** — history must travel or be reconstructible.
|
||||
3. **Multi-perspective truth** — union ≠ single canonical article.
|
||||
4. **Expected lifecycle** — information spaces may be ephemeral; plan for carry-forward.
|
||||
5. **Personal sovereignty** — contributors must not be hostage to one operator's server.
|
||||
|
||||
---
|
||||
|
||||
## 7. Use-case promotion (done 2026-06-08)
|
||||
|
||||
Promoted to `spec/UseCaseCatalog.md` as UC-26–UC-33. Architecture decisions
|
||||
tracked in `workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
|
||||
| Research ID | Catalog UC |
|
||||
|-------------|------------|
|
||||
| UC-FED-01 Fork page from remote shard | UC-26 |
|
||||
| UC-FED-02 View multiple versions of equivalent page | UC-27 |
|
||||
| UC-FED-03 Carry forward from closed/archived shard | UC-28 |
|
||||
| UC-FED-04 Remix with portable attribution | UC-29 |
|
||||
| UC-FED-05 Time-bounded collaboration space | UC-30 |
|
||||
| UC-FED-06 Subscribe to remote shard changes | UC-31 |
|
||||
| UC-FED-07 Transclude remote span | UC-32 |
|
||||
| UC-FED-08 Git-branch information space | UC-33 |
|
||||
|
||||
---
|
||||
|
||||
## 8. Open questions (for spec / workplans)
|
||||
|
||||
1. **Fork vs overlay vs import** — When does shard-wiki copy content into a
|
||||
writable shard vs keep an overlay vs link-only reference?
|
||||
2. **Equivalent page identity** — How are "same topic" pages matched across
|
||||
shards (title, path, link graph, explicit alias)?
|
||||
3. **Journal format** — Does shard-wiki adopt fedwiki-like action journals,
|
||||
Git commits only, or both per shard type?
|
||||
4. **Browser vs server composition** — Where does union merging run for agents
|
||||
and non-UI consumers?
|
||||
5. **ActivityPub as adapter** — Optional transport for change notification, or
|
||||
out of scope?
|
||||
6. **Ephemeral spaces** — Should information spaces have first-class lifecycle
|
||||
(archived, read-only, merged-into-successor)?
|
||||
7. **Plugin/item extensibility** — Markdown extensions vs structured blocks for
|
||||
remixable assessments (fedwiki OER lesson)?
|
||||
8. **Consensus without erasure** — Policy presets: fedwiki-spread, git-merge,
|
||||
designated canonical shard, vote-to-merge?
|
||||
|
||||
---
|
||||
|
||||
## 9. Sources
|
||||
|
||||
| Source | URL |
|
||||
|--------|-----|
|
||||
| Wikipedia — Federated Wiki | https://en.wikipedia.org/wiki/Federated_Wiki |
|
||||
| Wired — Wiki Inventor Sticks a Fork | https://www.wired.com/2012/07/wiki-inventor/ |
|
||||
| viki.wiki — JSON Schema | https://viki.wiki/json-schema.html |
|
||||
| Fedwiki GitHub | https://github.com/fedwiki/wiki |
|
||||
| Caulfield — Federated Education (NWACC 2014) | https://hapgood.us/2014/11/06/federated-education-new-directions-in-digital-collaboration/ |
|
||||
| Caulfield — Bounded Conversations | https://hapgood.us/2015/01/21/rethinking-wiki-lifecycle-sites-as-bounded-conversations/ |
|
||||
| Caulfield — OER Case for Federated Wiki | https://hapgood.us/2015/05/05/the-oer-case-for-federated-wiki/ |
|
||||
| ikiwiki — distributed wikis | https://ikiwiki.info/tips/distributed_wikis/ |
|
||||
| XWiki — ActivityPub Application | https://extensions.xwiki.org/xwiki/bin/view/Extension/ActivityPub%20Application/ |
|
||||
| Maggie Appleton — Xanadu Patterns | https://maggieappleton.com/xanadu-patterns |
|
||||
| Wikipedia — Project Xanadu | https://en.wikipedia.org/wiki/Project_Xanadu |
|
||||
| shard-wiki — yawex prior art | `research/260608-yawex-prior-art/findings.md` |
|
||||
| shard-wiki — c2 origins | `research/260608-c2-wiki-origins/findings.md` |
|
||||
| shard-wiki — INTENT | `INTENT.md` |
|
||||
|
||||
---
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
| This document section | Informs (future) |
|
||||
|-----------------------|------------------|
|
||||
| §2 Federated Wiki | Adapter design, page model, overlay/fork policy |
|
||||
| §3.1 ikiwiki | Git coordination journal, mirror/branch modes |
|
||||
| §3.2 ActivityPub | Optional notification adapter |
|
||||
| §3.3 Xanadu | Link resolution, transclusion, provenance UI |
|
||||
| §6 Mapping | Architecture blueprint guardrails |
|
||||
| §7 UC seeds | `spec/UseCaseCatalog.md` promotion pass |
|
||||
@@ -0,0 +1,397 @@
|
||||
# Perplexity Research Wiki Engines
|
||||
|
||||
Source: https://www.perplexity.ai/search/9b76560e-31b7-4465-87a4-01a9df9cdbbe#1
|
||||
|
||||
## What are the most technically interesting wiki engines out there?
|
||||
|
||||
Below is an expanded, more technical map of the wiki landscape, including your additions: C2, Federated Wiki, TWiki, Confluence, Semantic MediaWiki, TiddlyWiki, Fossil, and the Tcl-based option I found. I’ve kept the focus on what each system is *for*, what makes it distinct, and what kind of adoption it seems to have had. [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
|
||||
|
||||
## Wiki.js
|
||||
Wiki.js is a modern, extensible, self-hosted wiki aimed at polished documentation and flexible deployment. It feels like the “modern web app” interpretation of a wiki. [js](https://js.wiki)
|
||||
|
||||
**USPs**
|
||||
- Modern UI and authoring experience.
|
||||
- Extensible architecture.
|
||||
- Self-hosted and open source. [wikimatrix](https://www.wikimatrix.org/compare/wiki-js+dokuwiki+xwiki+mediawiki+tiki-wiki-cms-groupware+tiddlywiki)
|
||||
|
||||
**Strengths**
|
||||
- Strong for documentation teams that want low-friction editing.
|
||||
- Good fit when you want a contemporary stack and a clean UX.
|
||||
- Broadly positioned as “powerful and extensible.” [js](https://js.wiki)
|
||||
|
||||
**Weaknesses**
|
||||
- Less historically proven than MediaWiki or TWiki in huge legacy enterprise environments.
|
||||
- Smaller ecosystem than the oldest wiki engines.
|
||||
- Not as “structured application platform” oriented as Foswiki/XWiki. [wikimatrix](https://www.wikimatrix.org/compare/wiki-js+dokuwiki+xwiki+mediawiki+tiki-wiki-cms-groupware+tiddlywiki)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Internal docs.
|
||||
- Product documentation.
|
||||
- Team knowledge bases.
|
||||
- Self-hosted alternative to SaaS doc tools. [js](https://js.wiki)
|
||||
|
||||
**Userbase estimate**
|
||||
- Likely **tens of thousands of active deployments/users**, but public numbers are not well standardized. That makes it a strong product choice, but not a giant legacy ecosystem like MediaWiki or TWiki. [wikimatrix](https://www.wikimatrix.org/compare/wiki-js+dokuwiki+xwiki+mediawiki+tiki-wiki-cms-groupware+tiddlywiki)
|
||||
|
||||
## XWiki
|
||||
XWiki is a programmable wiki platform that blurs the line between wiki and lightweight application platform. It is technically interesting because it supports scripts, macros, and application-style pages. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**USPs**
|
||||
- Scriptable pages and apps.
|
||||
- Strong extension model.
|
||||
- Syntax compatibility with other wiki styles. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Strengths**
|
||||
- Great for building structured internal tools.
|
||||
- More “platform-like” than plain wiki software.
|
||||
- Good fit for teams that need dynamic content and forms. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Weaknesses**
|
||||
- More complexity than simpler wikis.
|
||||
- Can feel heavyweight if you only want plain documentation.
|
||||
- Requires more governance and administration. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Enterprise knowledge bases.
|
||||
- Internal business applications.
|
||||
- Structured reporting and collaborative workflows.
|
||||
- Documentation that needs embedded logic. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Userbase estimate**
|
||||
- Probably **tens of thousands to low hundreds of thousands** of users across deployments, but public install counts are not clearly disclosed. [wikimatrix](https://www.wikimatrix.org/compare/wiki-js+dokuwiki+xwiki+mediawiki+tiki-wiki-cms-groupware+tiddlywiki)
|
||||
|
||||
## MediaWiki
|
||||
MediaWiki is the canonical large-scale wiki engine: boring in the best possible way, because it works at enormous scale and has an enormous extension ecosystem. It remains technically important because it set the pattern for high-volume collaborative editing. [herothemes](https://herothemes.com/blog/wiki-software/)
|
||||
|
||||
**USPs**
|
||||
- Proven at internet scale.
|
||||
- Huge extension ecosystem.
|
||||
- Strong revision/history and collaboration model. [herothemes](https://herothemes.com/blog/wiki-software/)
|
||||
|
||||
**Strengths**
|
||||
- Best-in-class for public, high-traffic, multi-author wikis.
|
||||
- Extremely mature.
|
||||
- Broad tooling and community support. [herothemes](https://herothemes.com/blog/wiki-software/)
|
||||
|
||||
**Weaknesses**
|
||||
- Can be operationally heavy.
|
||||
- Editing experience is not the smoothest by default.
|
||||
- Often overkill for small internal knowledge bases. [herothemes](https://herothemes.com/blog/wiki-software/)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Wikipedia-style public encyclopedias.
|
||||
- Community knowledge bases.
|
||||
- Large multilingual collaborative sites.
|
||||
- Semantic layering via extensions. [herothemes](https://herothemes.com/blog/wiki-software/)
|
||||
|
||||
**Userbase estimate**
|
||||
- By far the largest in this group; think **millions of contributors and readers across many thousands of sites**. [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
|
||||
|
||||
## Semantic MediaWiki
|
||||
Semantic MediaWiki is technically interesting because it turns wiki pages into queryable knowledge objects. It is less “just a wiki” and more a semantic knowledge base on top of MediaWiki. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
|
||||
|
||||
**USPs**
|
||||
- Semantic annotations.
|
||||
- Queryable structured content.
|
||||
- Strong fit for knowledge management with ontology-like behavior. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Strengths**
|
||||
- Very powerful for structured knowledge capture.
|
||||
- Supports reporting and retrieval beyond full-text search.
|
||||
- Bridges freeform wiki editing and data-driven retrieval. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Weaknesses**
|
||||
- Steeper learning curve.
|
||||
- Requires discipline in content modeling.
|
||||
- More complex than plain wiki editing. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Research knowledge bases.
|
||||
- Reference systems with structured facts.
|
||||
- Enterprise knowledge graphs and catalogs.
|
||||
- Areas where pages should behave like data records. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Userbase estimate**
|
||||
- Smaller than MediaWiki, but meaningful in specialist knowledge-management circles: likely **thousands of sites, not millions**. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
|
||||
|
||||
## DokuWiki
|
||||
DokuWiki is the classic lightweight, file-based wiki: simple, pragmatic, and still technically elegant because it avoids a database. It is a workhorse for teams that want low operational overhead. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
|
||||
|
||||
**USPs**
|
||||
- No database required.
|
||||
- Simple deployment.
|
||||
- Large plugin ecosystem. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
|
||||
|
||||
**Strengths**
|
||||
- Easy to run and back up.
|
||||
- Good text/wiki syntax experience.
|
||||
- Reliable for small-to-medium teams. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
|
||||
|
||||
**Weaknesses**
|
||||
- Less suited to highly structured app-like workflows.
|
||||
- Not the strongest choice for huge public communities.
|
||||
- Interface feels older than modern doc platforms. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Internal docs.
|
||||
- Departmental knowledge bases.
|
||||
- Small team collaboration.
|
||||
- Simple intranets and technical notes. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
|
||||
|
||||
**Userbase estimate**
|
||||
- Likely **tens of thousands of installations**; broad but not dominant. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
|
||||
|
||||
## TiddlyWiki
|
||||
TiddlyWiki is one of the most technically unusual wiki engines because it is often a single self-contained HTML file that acts like a personal knowledge system. That makes it deeply portable and hackable. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**USPs**
|
||||
- Single-file architecture.
|
||||
- Highly personal and portable.
|
||||
- Strong customization via plugins/macros/themes. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Strengths**
|
||||
- Great for personal knowledge management.
|
||||
- Easy to carry around and version-control.
|
||||
- Extremely flexible for power users. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Weaknesses**
|
||||
- Not a conventional multi-user server wiki.
|
||||
- Collaboration and permissions are not its core strength.
|
||||
- Can become messy without personal discipline. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Personal notes.
|
||||
- Research notebooks.
|
||||
- Offline-first knowledge bases.
|
||||
- Individual knowledge workflows. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Userbase estimate**
|
||||
- Probably **tens of thousands to a few hundred thousand users**, with a strong enthusiast niche. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
## TWiki
|
||||
TWiki is a structured enterprise wiki that emerged from enterprise demand rather than minimalist wiki ideals. It is interesting because it became a platform for forms, workflows, and application-like pages. [junauza](https://www.junauza.com/2008/05/five-of-best-freeopen-source-wiki.html)
|
||||
|
||||
**USPs**
|
||||
- Enterprise collaboration platform.
|
||||
- Strong extensibility, with hundreds of extensions.
|
||||
- Structured content and application building. [sourceforge](https://sourceforge.net/projects/twiki/)
|
||||
|
||||
**Strengths**
|
||||
- Proven in corporate environments.
|
||||
- Great for structured team workflows.
|
||||
- Strong fit for intranets and process-heavy use. [sourceforge](https://sourceforge.net/projects/twiki/)
|
||||
|
||||
**Weaknesses**
|
||||
- Older design philosophy.
|
||||
- Can feel complex and procedural.
|
||||
- Less attractive for lightweight modern docs. [sourceforge](https://sourceforge.net/projects/twiki/)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Corporate intranets.
|
||||
- Team collaboration.
|
||||
- Project tracking.
|
||||
- Enterprise knowledge management. [junauza](https://www.junauza.com/2008/05/five-of-best-freeopen-source-wiki.html)
|
||||
|
||||
**Userbase estimate**
|
||||
- SourceForge’s description claims **50,000 small businesses, many Fortune 500 companies, and millions of people** use TWiki, though that should be treated as a vendor-style estimate rather than audited usage data. [sourceforge](https://sourceforge.net/projects/twiki/)
|
||||
|
||||
## Foswiki
|
||||
Foswiki is the continuation of the structured enterprise-wiki tradition, with strong emphasis on collaboration, plugins, and business use. It is technically interesting because it treats the wiki as an application platform with macros, forms, and reporting. [sourceforge](https://sourceforge.net/projects/foswiki/)
|
||||
|
||||
**USPs**
|
||||
- Enterprise wiki and collaboration platform.
|
||||
- Application-building via markup and macros.
|
||||
- Fine-grained access control and auditability. [sourceforge](https://sourceforge.net/projects/foswiki/)
|
||||
|
||||
**Strengths**
|
||||
- Strong fit for controlled enterprise environments.
|
||||
- Very flexible for internal business applications.
|
||||
- Deep extension model and workflow-style usage. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
|
||||
|
||||
**Weaknesses**
|
||||
- Heavier and more specialized than lightweight wikis.
|
||||
- UX is functional rather than modern-first.
|
||||
- Best when governance and structure matter more than simplicity. [sourceforge](https://sourceforge.net/projects/foswiki/)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Knowledge bases.
|
||||
- Team collaboration portals.
|
||||
- Workflow and project tracking.
|
||||
- Document management and structured reporting. [hpcsupport.utsa](http://hpcsupport.utsa.edu/foswiki/bin/view/System/WikiCulture)
|
||||
|
||||
**Userbase estimate**
|
||||
- Likely **thousands to tens of thousands of active users/deployments**, with a strong enterprise niche rather than mass-market scale. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
|
||||
|
||||
## Confluence
|
||||
Confluence is the enterprise commercial benchmark for team wikis, but technically it is more a knowledge collaboration platform than a pure wiki engine. It matters because many organizations chose it as the default workplace knowledge layer. [herothemes](https://herothemes.com/blog/wiki-software/)
|
||||
|
||||
**USPs**
|
||||
- Tight enterprise integration.
|
||||
- Mature permissions and collaboration.
|
||||
- Large marketplace ecosystem. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Strengths**
|
||||
- Strong adoption in enterprises.
|
||||
- Good for cross-team documentation and workflows.
|
||||
- Familiar to many business users. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Weaknesses**
|
||||
- Commercial lock-in.
|
||||
- Less transparent/inspectable than open-source engines.
|
||||
- Can become sprawling and hard to govern at scale. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Enterprise documentation.
|
||||
- Project spaces.
|
||||
- Meeting notes and cross-functional knowledge sharing.
|
||||
- Regulated orgs that need access control and audit trails. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Userbase estimate**
|
||||
- Very large enterprise footprint, plausibly **millions of users** across customers, but exact counts are not public. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
## C2
|
||||
C2 is historically important because it sits close to the origin story of wiki culture and especially “wiki engine as a social process,” not merely software. It is less a mainstream product choice today and more a foundational technical-cultural reference point. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
**USPs**
|
||||
- Foundational wiki culture.
|
||||
- Minimalist and historically influential.
|
||||
- Strong association with Ward Cunningham’s original ideas. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
**Strengths**
|
||||
- Very important conceptually.
|
||||
- Captures the original wiki ethos.
|
||||
- Useful as a reference model for design thinking. [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
|
||||
|
||||
**Weaknesses**
|
||||
- Not a modern enterprise product.
|
||||
- Limited contemporary ecosystem.
|
||||
- Not typically chosen for new production deployments. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Early collaborative documentation.
|
||||
- Idea exploration.
|
||||
- Community knowledge sharing.
|
||||
- Historical origin of wiki practice. [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
|
||||
|
||||
**Userbase estimate**
|
||||
- Modern active userbase is likely **very small**, mostly historical and enthusiast usage. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
## Federated Wiki
|
||||
Federated Wiki is one of the most intellectually interesting designs because Ward Cunningham pushed the idea beyond a central shared wiki toward a network of interoperating personal sites. It is a reaction against the assumptions of centralized knowledge ownership. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
**USPs**
|
||||
- Federated, distributed model.
|
||||
- Personal control with sharing across sites.
|
||||
- Strong conceptual innovation. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
**Strengths**
|
||||
- Excellent for decentralization-minded users.
|
||||
- Encourages personal ownership of notes and pages.
|
||||
- Very distinctive architecture. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
**Weaknesses**
|
||||
- Smaller ecosystem.
|
||||
- Less mainstream operational tooling.
|
||||
- Can be harder for teams expecting centralized permissions and governance. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Personal knowledge networks.
|
||||
- Distributed collaboration.
|
||||
- Experimental knowledge sharing.
|
||||
- Communities interested in networked authorship. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
**Userbase estimate**
|
||||
- Probably **small niche adoption**, more influential than widely deployed. [wiki.c2](https://wiki.c2.com/?WikiEngines)
|
||||
|
||||
## TiddlyWiki
|
||||
TiddlyWiki is already covered above as one of the core additions, but it deserves emphasis: it is the most “personal-computing” oriented wiki in this list. Its architecture is a big reason it remains technically admired. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**USPs**
|
||||
- Portable single-file personal wiki.
|
||||
- Deep customization.
|
||||
- Offline-friendly. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Strengths**
|
||||
- Great for individual workflows.
|
||||
- Easy to archive and transport.
|
||||
- Powerful for note structuring and metadata. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Weaknesses**
|
||||
- Not a natural fit for large teams.
|
||||
- Sync/conflict handling is user-managed.
|
||||
- Can be too freeform for rigid organizations. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Zettelkasten-like note systems.
|
||||
- Research and personal PKM.
|
||||
- Portable offline notes.
|
||||
- Solo technical planning. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
**Userbase estimate**
|
||||
- Niche but durable: **tens of thousands to low hundreds of thousands**. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
|
||||
|
||||
## Fossil
|
||||
Fossil is technically interesting because it bundles version control, bug tracking, and wiki/documentation into one system. Its wiki is part of a developer workflow, not a standalone knowledge product. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
|
||||
|
||||
**USPs**
|
||||
- Integrated VCS, issue tracker, and wiki.
|
||||
- Single-tool developer workflow.
|
||||
- Simple, self-contained project hosting. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
|
||||
|
||||
**Strengths**
|
||||
- Excellent for small to medium software projects.
|
||||
- Tight coupling between code, tickets, and docs.
|
||||
- Strong for maintaining project history. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
|
||||
|
||||
**Weaknesses**
|
||||
- Not a general-purpose wiki platform.
|
||||
- Smaller ecosystem than GitHub/GitLab-style stacks.
|
||||
- More dev-centric than business-user-centric. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
|
||||
|
||||
**Characteristic use cases**
|
||||
- Software project documentation.
|
||||
- Developer-run issue tracking and wiki pages.
|
||||
- Integrated team repositories.
|
||||
- Small tooling-centric open-source projects. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
|
||||
|
||||
**Userbase estimate**
|
||||
- Likely **tens of thousands of users**, mostly among developers and small project teams. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
|
||||
|
||||
## Tcl wiki option
|
||||
The Tcl ecosystem does have a wiki presence, but the result I found is mainly **Tcler's Wiki**, which is a wiki for the Tcl community rather than clear evidence of a widely used Tcl-native standalone engine. So I would not confidently label it as a major separate engine without more evidence. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
|
||||
|
||||
**USPs**
|
||||
- Community knowledge base for Tcl.
|
||||
- Demonstrates Tcl community use of wiki patterns. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
|
||||
|
||||
**Strengths**
|
||||
- Niche relevance for Tcl users.
|
||||
- Good example of a community-maintained knowledge base. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
|
||||
|
||||
**Weaknesses**
|
||||
- Not clearly a mainstream, distinct wiki engine product.
|
||||
- Limited evidence of broad adoption beyond the Tcl community. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
|
||||
|
||||
**Characteristic use cases**
|
||||
- Tcl community documentation.
|
||||
- Shared project knowledge.
|
||||
- Language-specific community support. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
|
||||
|
||||
**Userbase estimate**
|
||||
- Very small, niche community usage. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
|
||||
|
||||
## What stands out most
|
||||
If you want the most technically distinctive group, I’d rank them this way: **Federated Wiki** for distributed architecture, **TiddlyWiki** for single-file personal knowledge, **Semantic MediaWiki** for queryable structure, **Fossil** for integrated dev workflow, and **Foswiki/TWiki** for enterprise wiki-as-application platforms. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
|
||||
|
||||
The older engines are still interesting because they reflect different answers to the same design question: should a wiki be a simple page editor, a structured app platform, or a knowledge graph with a browser front end? [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
|
||||
|
||||
|
||||
# Possible Follow Ups
|
||||
- How do the underlying data architectures of these wiki engines impact long-term knowledge curation and data longeevity?
|
||||
- How does TiddlyWiki manage non-linear storage in a single file?
|
||||
- How do Semantic MediaWiki graph capabilities compare to modern graph DBs?
|
||||
- What architectural challenges led to the creation of Federated Wiki?
|
||||
- How does Fossil provide version control and wiki in one binary?
|
||||
|
||||
|
||||
|
||||
32
research/260608-wikiengines-overview/README.md
Normal file
32
research/260608-wikiengines-overview/README.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# 260608 — Wiki engines overview (landscape scan)
|
||||
|
||||
Date: 2026-06-08
|
||||
|
||||
## What this is
|
||||
|
||||
A breadth-first **landscape scan of notable wiki engines** — Wiki.js, XWiki,
|
||||
MediaWiki, Semantic MediaWiki, DokuWiki, TiddlyWiki, TWiki, Foswiki, Confluence,
|
||||
C2, Federated Wiki, Fossil — captured to understand the design space
|
||||
`shard-wiki` orchestrates over (and deliberately does **not** re-implement).
|
||||
|
||||
Complements the deeper-dive research:
|
||||
- `research/260608-c2-wiki-origins/` — pre-federation wiki culture
|
||||
- `research/260608-federation-concepts/` — Federated Wiki, git/AP federation models
|
||||
- `research/260608-yawex-prior-art/` — the specific Perl engine being superseded
|
||||
|
||||
Focus: what each engine is *for*, its storage/permission model, and what its
|
||||
existence implies for shard-wiki's **adapter contract** — not engine selection.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Synthesis: storage/permission axes, shard-wiki mapping, adapter implications, open questions |
|
||||
| `260608-perplexity-wikiengines-research.md` | Raw source dump (Perplexity), kept as provenance |
|
||||
|
||||
## Status
|
||||
|
||||
Initial scan complete. Reinforces the INTENT stance that shard-wiki is an
|
||||
**orchestrator, not an engine**: the heterogeneity below is the problem space,
|
||||
not a menu to pick from. Adapter-contract implications feed
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
147
research/260608-wikiengines-overview/findings.md
Normal file
147
research/260608-wikiengines-overview/findings.md
Normal file
@@ -0,0 +1,147 @@
|
||||
# Findings — Wiki engine landscape and what it implies for shard-wiki
|
||||
|
||||
Date: 2026-06-08 · Status: research draft
|
||||
|
||||
Scope: a breadth-first scan of notable wiki engines, read through one question —
|
||||
**what does this engine's storage and permission model demand of a shard adapter,
|
||||
and what must shard-wiki therefore NOT assume is uniform?** This is landscape, not
|
||||
selection: shard-wiki orchestrates over these, it does not pick one.
|
||||
|
||||
**Complements:** `research/260608-federation-concepts/` (federation mechanics) and
|
||||
`research/260608-c2-wiki-origins/` (origin culture). Raw source kept alongside as
|
||||
`260608-perplexity-wikiengines-research.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1. The engines, by what they actually are
|
||||
|
||||
The "wiki" label spans at least four distinct system shapes. Conflating them is
|
||||
exactly the mistake shard-wiki's adapter contract exists to prevent.
|
||||
|
||||
| Shape | Engines | Storage | Permission model | Page format |
|
||||
|-------|---------|---------|------------------|-------------|
|
||||
| **Simple page editor** | C2 (origin), DokuWiki | flat files (DokuWiki: no DB) | open / htaccess / ACL plugin | wiki markup |
|
||||
| **App platform** | XWiki, TWiki, Foswiki | DB + structured objects | fine-grained, per-space, forms/workflow | macro markup |
|
||||
| **Internet-scale collaborative** | MediaWiki, Semantic MediaWiki | DB + revision tables | role/group, extension-driven | wikitext (+ semantic annotations) |
|
||||
| **Personal / single-file** | TiddlyWiki, (Obsidian) | one HTML file / local vault | user-managed, not multi-user | tiddlers / Markdown |
|
||||
| **Federated / distributed** | Federated Wiki, Fossil, ikiwiki | per-site JSON / repo | per-site sovereignty / VCS auth | JSON story+journal / files |
|
||||
| **Commercial benchmark** | Confluence | proprietary store | mature enterprise ACL + SSO | proprietary |
|
||||
|
||||
The same word, six storage substrates and at least five permission philosophies.
|
||||
|
||||
## 2. The two axes that matter for orchestration
|
||||
|
||||
Stripping the marketing, engines differ on two axes shard-wiki must model
|
||||
explicitly rather than flatten:
|
||||
|
||||
1. **Storage substrate** — flat files (DokuWiki, Fossil), single file (TiddlyWiki),
|
||||
relational DB + revision tables (MediaWiki, XWiki, Confluence), per-site JSON
|
||||
journals (Federated Wiki), git repo (Fossil, ikiwiki, Gitea wikis).
|
||||
2. **Permission philosophy** — open-by-default (C2), file/htaccess gating
|
||||
(DokuWiki, yawex), per-space/per-page enterprise ACL (Foswiki, XWiki,
|
||||
Confluence), VCS-level auth (Fossil), or *none / user-managed* (TiddlyWiki).
|
||||
|
||||
These map directly onto two INTENT concepts:
|
||||
- Storage substrate → the **shard adapter contract** must not assume one backend.
|
||||
- Permission philosophy → validates the **L0→L4 access ladder**: real engines
|
||||
already span "open C2" to "enterprise ACL + SSO". shard-wiki's ladder is not
|
||||
invented; it's the union of where these engines actually sit.
|
||||
|
||||
## 3. Capability heterogeneity (why "capability-aware" is non-negotiable)
|
||||
|
||||
The scan concretely demonstrates the INTENT principle *capability-aware adapters*:
|
||||
|
||||
| Operation | Always available? | Counter-example |
|
||||
|-----------|-------------------|-----------------|
|
||||
| Read | ~yes | Confluence behind SSO: not without auth |
|
||||
| Write | no | C2 today is read-only/archival; MediaWiki dumps are read-only |
|
||||
| Structured diff/merge | no | DB-backed engines expose revisions but not git-style 3-way merge |
|
||||
| Semantic query | rarely | only Semantic MediaWiki / XWiki |
|
||||
| Single-file atomicity | special | TiddlyWiki is whole-file, not per-page, writes |
|
||||
| Native VCS history | sometimes | Fossil/ikiwiki yes; Confluence no (internal only) |
|
||||
|
||||
No adapter can assume a peer supports its own operation set — the capability
|
||||
profile per shard is load-bearing, not decorative.
|
||||
|
||||
## 4. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 4.1 Reinforcements (engines confirm an INTENT stance)
|
||||
|
||||
| Observation | INTENT principle confirmed |
|
||||
|-------------|----------------------------|
|
||||
| Six storage substrates under one word | Markdown-first, **backend-neutral**; adapter contract |
|
||||
| Permission philosophies span open→enterprise | **Open by default, progressively governed** (L0→L4) |
|
||||
| DB engines expose revisions but not portable history | **Git-addressable coordination** adds the missing portable journal |
|
||||
| TiddlyWiki/Obsidian = sovereign personal stores | **Local-first usefulness**; shards, not replacements |
|
||||
| Federated Wiki = union composed from many origins | **Union without erasure** |
|
||||
| Confluence/Foswiki = mature ACL but lock-in | shard-wiki **delegates** IAM, never owns identity |
|
||||
|
||||
### 4.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
| Engine assumption | shard-wiki correction |
|
||||
|-------------------|----------------------|
|
||||
| "Pick the one engine for your team" | shard-wiki attaches *many* as shards; not a single choice |
|
||||
| App-platform macros define the page | **Markdown-first**; engine-specific render stays out of core |
|
||||
| Permissions live in the engine's DB | **Authorization in core**, identity delegated — uniform across shards |
|
||||
| History = the engine's revision table | **Coordination journal** is space-level, git-backed, portable |
|
||||
| Federation = everyone runs the same engine | Adapter contract spans heterogeneous, non-git, capability-limited backends |
|
||||
|
||||
### 4.3 What the landscape teaches that shard-wiki should not lose
|
||||
|
||||
1. **Don't homogenize** — the heterogeneity is the user's reality, not a defect to fix.
|
||||
2. **Openness and governance are a spectrum, not a toggle** — engines live at every
|
||||
point C2→Confluence; the ladder must too.
|
||||
3. **Structured/semantic pages exist** (Semantic MediaWiki, XWiki) — Markdown-first
|
||||
must degrade gracefully when a shard carries structure shard-wiki can't model yet.
|
||||
4. **Single-file/whole-vault write granularity** (TiddlyWiki) breaks per-page write
|
||||
assumptions — the capability profile must express write granularity.
|
||||
|
||||
## 5. Implications for the adapter contract (carry to SHARD-WP-0002)
|
||||
|
||||
1. **Capability profile must encode write granularity** (per-page vs whole-file vs
|
||||
whole-space) — TiddlyWiki and DB engines differ here.
|
||||
2. **History portability is an adapter concern** — engines with internal-only
|
||||
revisions (Confluence, MediaWiki) need the coordination journal to supply the
|
||||
git-addressable layer they lack.
|
||||
3. **Structured/semantic payloads need a passthrough/provenance escape hatch** so
|
||||
Semantic MediaWiki / XWiki content isn't silently flattened to lossy Markdown.
|
||||
4. **Permission model is never trusted from the shard** — even engines with rich
|
||||
internal ACLs are authorized through shard-wiki's core, not their DB.
|
||||
|
||||
## 6. Open questions (for spec / workplans)
|
||||
|
||||
1. How does shard-wiki represent **structured/semantic** pages (Semantic MediaWiki,
|
||||
XWiki objects) — lossy Markdown projection, sidecar metadata, or opaque blob?
|
||||
2. **Write granularity** in the capability profile — what's the vocabulary
|
||||
(per-page / per-file / per-space / append-only)?
|
||||
3. For engines with **internal-only history** (Confluence), is the git coordination
|
||||
journal authoritative, or a mirror with reconciliation back?
|
||||
4. Which of these engines get **first-class adapters** vs are reachable only via a
|
||||
generic file/WebDAV adapter?
|
||||
|
||||
## 7. Sources
|
||||
|
||||
| Source | URL |
|
||||
|--------|-----|
|
||||
| Perplexity — wiki engines scan (raw, this dir) | `260608-perplexity-wikiengines-research.md` |
|
||||
| C2 — Top Ten Wiki Engines | http://c2.com/cgi/wiki?TopTenWikiEngines |
|
||||
| C2 — WikiEngines | https://wiki.c2.com/?WikiEngines |
|
||||
| Comparison of wiki software | https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software |
|
||||
| WikiMatrix engine comparison | https://www.wikimatrix.org/ |
|
||||
| Wiki.js | https://js.wiki |
|
||||
| Foswiki | https://en.wikipedia.org/wiki/Foswiki |
|
||||
| Fossil (software) | https://en.wikipedia.org/wiki/Fossil_(software) |
|
||||
| shard-wiki — federation concepts | `research/260608-federation-concepts/findings.md` |
|
||||
| shard-wiki — INTENT | `INTENT.md` |
|
||||
|
||||
---
|
||||
|
||||
## 8. Traceability
|
||||
|
||||
| This document section | Informs (future) |
|
||||
|-----------------------|------------------|
|
||||
| §2 two axes | Access ladder (INTENT) + adapter contract guardrails |
|
||||
| §3 capability heterogeneity | Adapter capability profile design |
|
||||
| §4 INTENT mapping | Architecture blueprint guardrails |
|
||||
| §5 adapter implications | `workplans/SHARD-WP-0002-federation-architecture.md` |
|
||||
| §6 open questions | spec — page model for structured/semantic content |
|
||||
36
research/260613-foswiki-deep-dive/README.md
Normal file
36
research/260613-foswiki-deep-dive/README.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# 260613 — Foswiki deep dive (store abstraction, extension API, ecosystem)
|
||||
|
||||
Date: 2026-06-13
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Foswiki** — the community fork of TWiki (2008, from TWiki
|
||||
4.2.x) — concentrating on **what Foswiki changed**, not the lineage it shares with
|
||||
TWiki. The distinctive material: a **pluggable `Foswiki::Store` backend**
|
||||
(RcsWrap / RcsLite / **PlainFileStoreContrib**) behind a versioned interface, the
|
||||
`Foswiki::Meta` topic-object indirection, the OO/MVC core rewrite, the cleaner
|
||||
`Foswiki::Func` + `registerTagHandler` extension API, **DataForms** (+ MetaDataPlugin,
|
||||
FlexFormPlugin), and the **WysiwygPlugin** TML↔HTML round-trip.
|
||||
|
||||
Read through shard-wiki's lens: Foswiki is the engine that **already separates its
|
||||
store backend from its core via a versioned interface** — i.e. it is concrete prior
|
||||
art for shard-wiki's own *shard adapter contract* and *capability-aware adapters*.
|
||||
|
||||
Pairs with — and deliberately does not repeat — the shared TWiki material:
|
||||
- `research/260613-twiki-deep-dive/` — flat-file/RCS store, Webs/Topics, plugin
|
||||
handler callbacks, per-topic ACL (all inherited; read there for the shared parts)
|
||||
- `research/260613-xwiki-deep-dive/` — the Java/DB app-platform sibling
|
||||
- `research/260608-wikiengines-overview/` — landscape scan
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Distinctives vs TWiki, store abstraction, extension API, DataForms, capability profile, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to
|
||||
`spec/UseCaseCatalog.md` (UC-42, UC-43); UC-34/39/40 enriched. The
|
||||
`Foswiki::Store`/`Foswiki::Meta` abstraction is logged as adapter-contract prior art
|
||||
for `workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
188
research/260613-foswiki-deep-dive/findings.md
Normal file
188
research/260613-foswiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,188 @@
|
||||
# Findings — Foswiki: store abstraction, extension API, ecosystem
|
||||
|
||||
Date: 2026-06-13 · Status: research draft
|
||||
|
||||
Scope: **Foswiki as distinct from TWiki** — only the deltas, since the two share a
|
||||
data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to
|
||||
shard-wiki: a **pluggable store backend behind a versioned interface**
|
||||
(`Foswiki::Store` + `Foswiki::Meta`), an OO/MVC core rewrite, a cleaner extension
|
||||
API (`Foswiki::Func::registerTagHandler`), enhanced DataForms, and a TML↔HTML
|
||||
round-trip editor framework. Sources: foswiki.org Development + System docs,
|
||||
`foswiki/distro` on GitHub.
|
||||
|
||||
**Read first for shared lineage:** `research/260613-twiki-deep-dive/` (flat-file +
|
||||
RCS store, Webs/Topics, handler callbacks, per-topic `ALLOW/DENY` ACL). This file
|
||||
does **not** repeat those.
|
||||
|
||||
---
|
||||
|
||||
## 1. Origin and compatibility
|
||||
|
||||
Forked from **TWiki 4.2.x** in October 2008 by most of the TWiki community (sources
|
||||
cite 4.2.3–4.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content
|
||||
compatibility — same TML markup, same plugin API — via a **`TWikiCompatibilityPlugin`**
|
||||
that maps the `TWiki::` namespace and legacy variables. So everything in the TWiki
|
||||
dive still applies; Foswiki then grew the features below.
|
||||
|
||||
## 2. The headline delta — a pluggable store behind a versioned interface
|
||||
|
||||
This is the reason Foswiki earns its own dive.
|
||||
|
||||
### 2.1 `Foswiki::Meta` indirection
|
||||
After Foswiki 1.0.4 the core was changed to **delegate almost all store operations
|
||||
to a topic object, `Foswiki::Meta`**. Callers manipulate a `Foswiki::Meta` object
|
||||
representing the item they're changing and **never touch the store implementation
|
||||
directly**; `Foswiki::Meta` talks to the real store through the well-defined
|
||||
**`Foswiki::Store`** interface. This is a clean MVC/repository separation.
|
||||
|
||||
### 2.2 Swappable backends
|
||||
`Foswiki::Store::` ships multiple low-level back-ends behind that one interface:
|
||||
- **`RcsWrap`** — RCS via the external `rcs` binaries (classic TWiki behavior)
|
||||
- **`RcsLite`** — a pure-Perl RCS implementation
|
||||
- **`PlainFileStoreContrib`** — a newer store that saves topics/attachments as
|
||||
**timestamped copies** instead of RCS diffs: more disk, but **no RCS dependency**
|
||||
and much higher performance.
|
||||
|
||||
So in Foswiki the **storage format is a configuration choice**, not a fixed property
|
||||
of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core.
|
||||
|
||||
### 2.3 OO/MVC core rewrite
|
||||
Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward
|
||||
object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving
|
||||
backward compatibility through tests. The store abstraction above is the most
|
||||
load-bearing result.
|
||||
|
||||
## 3. Extension API deltas
|
||||
|
||||
Shared with TWiki: handler callbacks (`initPlugin`, `commonTagsHandler`,
|
||||
`before/afterSaveHandler`, `afterRenameHandler`, attachment handlers), REST handlers,
|
||||
`Config.spec`, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements:
|
||||
|
||||
- **`Foswiki::Func` is the blessed API** — "if there's a `Foswiki::Func` way and
|
||||
another way, the `Foswiki::Func` way is almost always right." Direct use of
|
||||
internal packages is discouraged and governed by **`PluginsApiPolicies`**.
|
||||
- **`Foswiki::Func::registerTagHandler($tag, \&sub)`** — register a custom
|
||||
macro/variable **programmatically** (cleaner than TWiki's `commonTagsHandler`
|
||||
string-matching convention).
|
||||
- **Contrib-defined APIs** carry an explicit "use at your own risk / awaiting merge"
|
||||
policy — a maturity signal on extension interfaces.
|
||||
- `EmptyPlugin` (`lib/Foswiki/Plugins/EmptyPlugin.pm`) is the canonical handler
|
||||
reference/skeleton.
|
||||
|
||||
## 4. DataForms and structured data (delta)
|
||||
|
||||
DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi
|
||||
value, label) stored as `%META:FIELD%` in the topic text. Foswiki extends the model
|
||||
via popular extensions:
|
||||
- **MoreFormfieldsPlugin** — additional special-purpose field types.
|
||||
- **FlexFormPlugin** — render DataForm interfaces from custom templates
|
||||
(`Foswiki::Form` classes).
|
||||
- **MetaDataPlugin** — **multiple structured data records per topic** (beyond the
|
||||
classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model
|
||||
while keeping data in the text file.
|
||||
|
||||
## 5. Editing / syntax (delta relevant to "Markdown-first")
|
||||
|
||||
- **WysiwygPlugin** — a generic framework that transforms **TML → HTML** for a
|
||||
browser editor and **HTML → TML on save** (lossless round-trip).
|
||||
- **TinyMCEPlugin** — pre-installed WYSIWYG editor built on WysiwygPlugin.
|
||||
|
||||
This proves **bidirectional, lossless translation between an engine's native markup
|
||||
and another representation is feasible** — directly relevant to shard-wiki reading a
|
||||
non-Markdown shard and writing Markdown overlays back (UC-42).
|
||||
|
||||
## 6. Foswiki as a shard — capability profile (delta from TWiki)
|
||||
|
||||
| Capability | Foswiki | Note vs TWiki |
|
||||
|------------|---------|---------------|
|
||||
| Store backend | **pluggable** | RCS *or* PlainFile behind `Foswiki::Store` — direct-attach target is cleaner on PlainFile (UC-40) |
|
||||
| History | RCS *or* timestamped copies | PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41) |
|
||||
| Structured payload | DataForms `%META%`, **multi-record** via MetaDataPlugin | richer than classic TWiki Forms (UC-34/39) |
|
||||
| Syntax translation | **WysiwygPlugin TML↔HTML** | bidirectional round-trip exists (UC-42) |
|
||||
| Extension host | `Foswiki::Func` + `registerTagHandler` + REST | cleaner adapter-host surface (UC-38) |
|
||||
| Store as contract | **`Foswiki::Store` versioned interface** | architectural prior art for shard-wiki's adapter contract |
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
| Observation | INTENT principle |
|
||||
|-------------|------------------|
|
||||
| `Foswiki::Store` versioned interface + swappable backends | **capability-aware adapters** / **shard adapter contract** — an engine that already did exactly this separation |
|
||||
| `Foswiki::Meta` mediates all store access | clean **mechanism over policy** boundary worth mirroring in the adapter contract |
|
||||
| PlainFile store = timestamped text copies | **git-addressable coordination** / direct-attach (UC-40), history import (UC-41) |
|
||||
| WysiwygPlugin TML↔HTML round-trip | **Markdown-first, backend-neutral** is achievable for prose via translation (UC-42) |
|
||||
| MetaDataPlugin multi-record topics | structured pages beyond one form (UC-34/39) |
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
| Foswiki assumption | shard-wiki correction |
|
||||
|--------------------|----------------------|
|
||||
| One pluggable store per wiki, chosen at config | shard-wiki federates **many** heterogeneous stores at once |
|
||||
| TML is the canonical syntax; HTML is a render target | shard-wiki is **Markdown-first**; TML↔Markdown is an adapter translation, not core |
|
||||
| Store interface is Foswiki-internal Perl | shard-wiki's adapter contract is **cross-language, cross-backend, versioned** |
|
||||
| ACL in topic preferences | **authorize in core**; engine ACL is advisory provenance |
|
||||
|
||||
### 7.3 What Foswiki teaches that shard-wiki should not lose
|
||||
1. **The store-abstraction pattern works in practice** — `Foswiki::Store` is a
|
||||
real-world proof that a stable interface + swappable backends is viable; the shard
|
||||
adapter contract is the same idea generalized across engines.
|
||||
2. **Backend format should be swappable under a stable identity** — Foswiki migrates
|
||||
RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's
|
||||
backend changing under it (UC-43).
|
||||
3. **Lossless syntax round-trip is a solved problem** — don't treat non-Markdown
|
||||
prose as read-only; translate it (UC-42).
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-13)
|
||||
|
||||
| Seed | Catalog UC | Disposition |
|
||||
|------|------------|-------------|
|
||||
| Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown) | **UC-42** (new) | realizes Markdown-first for prose; WysiwygPlugin is proof |
|
||||
| Tolerate a shard's storage-backend swap without losing identity/provenance | **UC-43** (new) | Foswiki RCS↔PlainFile; orchestration robustness |
|
||||
| Structured / multi-record pages | UC-34 / UC-39 | enriched: DataForms + MetaDataPlugin multi-record |
|
||||
| Attach a file-backed engine's on-disk store directly | UC-40 | enriched: PlainFile store is a cleaner direct-attach target than RCS |
|
||||
| `Foswiki::Store` versioned interface | (no UC) | architecture prior art → `SHARD-WP-0002` adapter contract |
|
||||
|
||||
## 9. Open questions (for spec / workplans)
|
||||
|
||||
1. **Adapter-contract shape** — how much of `Foswiki::Store`'s method set
|
||||
(read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's
|
||||
capability-aware adapter contract? (architecture, `SHARD-WP-0002`)
|
||||
2. **Syntax-translation fidelity** — is TML↔Markdown lossless enough for overlay
|
||||
round-trips, or must overlays be stored in TML to be safe (UC-42)?
|
||||
3. **Backend-swap detection** — how does shard-wiki notice and tolerate a shard
|
||||
changing store format underneath (RCS→PlainFile), and does provenance/history
|
||||
survive it (UC-43)?
|
||||
4. **Multi-record metadata** — represent MetaDataPlugin multi-record topics and XWiki
|
||||
multi-XObject pages in one structured-metadata page model (shared open question).
|
||||
|
||||
## 10. Sources
|
||||
|
||||
| Source | URL |
|
||||
|--------|-----|
|
||||
| Foswiki — Technical Overview | https://foswiki.org/Development/TechnicalOverview |
|
||||
| Foswiki — Core Internals | https://foswiki.org/Development/CoreInternals |
|
||||
| Foswiki — PlainFileStoreContrib | https://foswiki.org/Extensions/PlainFileStoreContrib |
|
||||
| Foswiki — RCSStoreContrib | https://foswiki.org/Extensions/RCSStoreContrib |
|
||||
| Foswiki::Store::PlainFile (source) | https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm |
|
||||
| Foswiki — Developing Plugins | https://foswiki.org/System/DevelopingPlugins |
|
||||
| Foswiki — Plugins API Policies | https://foswiki.org/Development/PluginsApiPolicies |
|
||||
| Foswiki — DataForms | https://foswiki.org/System/DataForms |
|
||||
| Foswiki — MetaDataPlugin | https://foswiki.org/Extensions/MetaDataPlugin |
|
||||
| Foswiki — WysiwygPlugin | https://foswiki.org/System/WysiwygPlugin |
|
||||
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
|
||||
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
|
||||
| shard-wiki — TWiki deep dive | `research/260613-twiki-deep-dive/findings.md` |
|
||||
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
|
||||
|
||||
---
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
| This document section | Informs (future) |
|
||||
|-----------------------|------------------|
|
||||
| §2 store abstraction | **adapter contract** design (`SHARD-WP-0002`) — Foswiki::Store as prior art |
|
||||
| §3 extension API | UC-38 engine-side adapter host |
|
||||
| §4 DataForms | structured-metadata page model (UC-34/39) |
|
||||
| §5 WysiwygPlugin | UC-42 syntax-translation adapter |
|
||||
| §6 capability profile | adapter capability-profile vocabulary |
|
||||
| §7 INTENT mapping | architecture-blueprint guardrails |
|
||||
| §8 UC seeds | `spec/UseCaseCatalog.md` (UC-42, UC-43; UC-34/39/40 enrichment) |
|
||||
39
research/260613-twiki-deep-dive/README.md
Normal file
39
research/260613-twiki-deep-dive/README.md
Normal file
@@ -0,0 +1,39 @@
|
||||
# 260613 — TWiki deep dive (implementation, plugin API, ecosystem)
|
||||
|
||||
Date: 2026-06-13
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **TWiki** — the original "structured enterprise wiki / wiki
|
||||
application platform" (Peter Thoeny, ~1998), and the **file-based Perl counterpoint**
|
||||
to XWiki's Java/DB platform. Covers TWiki's **implementation** (Perl + flat-file +
|
||||
RCS store, Webs/Topics, TWiki Forms, TWikiML/variables), the **plugin API and
|
||||
handler callbacks** that extend the core, the **extension package types** (Plugin /
|
||||
Skin / AddOn / Contrib) and repository, and TWiki's **per-topic access control**
|
||||
(the prior art behind yawex's `AccessControl`).
|
||||
|
||||
Read through shard-wiki's lens, paired with the XWiki dive: TWiki shows the same
|
||||
"wiki-as-application-platform" goal reached with **text files + RCS** instead of a
|
||||
database — which changes the adapter story (direct on-disk attach, git-convertible
|
||||
history).
|
||||
|
||||
Complements:
|
||||
- `research/260613-xwiki-deep-dive/` — the DB/Java app-platform sibling
|
||||
- `research/260608-wikiengines-overview/` — the landscape scan
|
||||
- `research/260608-yawex-prior-art/` — yawex borrowed TWiki's per-topic AccessControl
|
||||
|
||||
Note: most of TWiki's community forked to **Foswiki** in Oct 2008 (from TWiki
|
||||
4.2.4); findings flag where TWiki/Foswiki differ.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, plugin-handler API, ecosystem, capability profile, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to
|
||||
`spec/UseCaseCatalog.md` (UC-40, UC-41); UC-06/34/36/38/39 enriched with TWiki
|
||||
specifics. Adapter-contract implications feed
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
210
research/260613-twiki-deep-dive/findings.md
Normal file
210
research/260613-twiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# Findings — TWiki: implementation, plugin API, ecosystem
|
||||
|
||||
Date: 2026-06-13 · Status: research draft
|
||||
|
||||
Scope: TWiki as prior art for the **file-based** wiki-application-platform — the
|
||||
counterpoint to XWiki's database/component platform. Two shard-wiki concerns:
|
||||
(1) attaching a structured, file-backed engine as a shard (including attaching its
|
||||
**on-disk store directly** vs through its runtime), and (2) how TWiki's plugin
|
||||
handler API exposes hooks rich enough to host a federation adapter
|
||||
(INTENT *composable integration*). Sources: twiki.org dev docs (TWikiPlugins,
|
||||
TWikiForms, TWiki::Func, TWikiAccessControl), Foswiki docs, Wikipedia.
|
||||
|
||||
**Complements:** `research/260613-xwiki-deep-dive/` (DB/Java sibling),
|
||||
`research/260608-wikiengines-overview/` (landscape), and
|
||||
`research/260608-yawex-prior-art/` (yawex borrowed TWiki's per-topic AccessControl).
|
||||
|
||||
---
|
||||
|
||||
## 1. What TWiki is
|
||||
|
||||
A **structured enterprise wiki** and **wiki application platform** (Peter Thoeny,
|
||||
~1998), written in **Perl 5** (5.10.1+), classically run as CGI. Like XWiki it lets
|
||||
pages become forms/records/apps — but it reaches that with **flat text files +
|
||||
RCS**, no database. Most of the community forked to **Foswiki** in October 2008
|
||||
(from TWiki 4.2.4; Foswiki 1.0 in January 2009); the plugin API and data model below
|
||||
are shared lineage, with minor divergences noted.
|
||||
|
||||
## 2. Implementation architecture
|
||||
|
||||
### 2.1 Storage — flat files + RCS (no DB)
|
||||
- Content lives on disk as `data/<Web>/<Topic>.txt`; the directory tree **mirrors
|
||||
the logical Web/Topic hierarchy**. Attachments live under `pub/<Web>/<Topic>/`.
|
||||
- **History is RCS** (`<Topic>.txt,v` companion files) — the same GNU RCS yawex used.
|
||||
History is real, per-file, and in an **open, git-convertible format** (unlike
|
||||
XWiki's DB-internal `xwikircs`).
|
||||
- Scales to 300k+ topics on a single server (Yahoo) — file store is not a toy.
|
||||
|
||||
### 2.2 Webs and Topics
|
||||
- **Topic** = page (the unit of content). **Web** = namespace / collection of topics
|
||||
(can be nested). Webs are the natural shard / root-entity boundary.
|
||||
|
||||
### 2.3 TWiki Forms — structured data, embedded in text
|
||||
- A **form template** is a topic defining fields as a table (one row per field:
|
||||
name, type, size, values). A topic can *attach* a form; the field values are
|
||||
stored as **TWikiMetaData inside the topic `.txt`** (`%META:FIELD{...}%`), shown as
|
||||
a table on view and edited via fields/radios/checkboxes/lists.
|
||||
- **Forms + formatted `%SEARCH%`** = TWiki's "database applications" — structured
|
||||
records queried by the SEARCH variable. This is the XWiki XObject/XClass idea, but
|
||||
**file-embedded and git-diffable** rather than rows in a DB.
|
||||
|
||||
### 2.4 TWikiML / variables (macros)
|
||||
- Markup is **TWikiML** (not Markdown). Dynamic behavior comes from **variables**
|
||||
`%VAR%` / `%VAR{...}%` expanded at render time; `%SEARCH%` is the query engine.
|
||||
- Custom variables are provided by plugins (see §3).
|
||||
|
||||
### 2.5 TWiki::Func — the official API boundary
|
||||
- `lib/TWiki/Func.pm` ("TWikiFuncDotPm") documents **all** interfaces available to
|
||||
plugins, deliberately abstracting the flat-file assumption: `readTopic`,
|
||||
`saveTopic`, `getTopicList`, `getListOfWebs`, `getWorkArea`, `saveAttachment`, …
|
||||
Plugins are told *not* to assume flat files and to go through Func.
|
||||
|
||||
## 3. Plugin API — interfaces to extend the core
|
||||
|
||||
TWiki extends "without altering core code" by **registering Perl handlers that the
|
||||
core calls at defined points** in the request/render/save pipeline. A plugin is
|
||||
`lib/TWiki/Plugins/<Name>.pm` + a documentation topic + an optional `Config.spec`.
|
||||
|
||||
### 3.1 Handler callbacks (the hook surface)
|
||||
| Phase | Handlers |
|
||||
|-------|----------|
|
||||
| Init / users | `initPlugin` (must return 1), `registrationHandler` |
|
||||
| Tag/variable expansion | `commonTagsHandler`, `beforeCommonTagsHandler`, `afterCommonTagsHandler` |
|
||||
| Rendering | `preRenderingHandler`, `postRenderingHandler` (`start/endRenderingHandler` deprecated) |
|
||||
| Edit lifecycle | `beforeEditHandler`, `afterEditHandler` |
|
||||
| Save lifecycle | `beforeSaveHandler`, `afterSaveHandler`, `beforeMergeHandler` |
|
||||
| Attachments | `beforeAttachmentSaveHandler`, `afterAttachmentSaveHandler` |
|
||||
| Topic mgmt | `afterRenameHandler`, `completePageHandler` |
|
||||
|
||||
These are the TWiki analogue of XWiki's `ObservationManager` events — but
|
||||
**synchronous pipeline hooks** rather than an async event bus. The save/rename/
|
||||
attachment handlers are exactly the interception points a federation adapter needs.
|
||||
|
||||
### 3.2 Other extension mechanisms
|
||||
- **Custom variables/macros** — registered via `commonTagsHandler` (Foswiki adds
|
||||
`registerTagHandler`); convention `%PLUGINNAME_SETTING%`.
|
||||
- **REST handlers** — a plugin registers REST handlers invoked via the `rest`
|
||||
script, for transactions outside the standard `view/edit/save` scripts. This is the
|
||||
natural remote adapter transport.
|
||||
- **`Config.spec`** — declares configuration items (BOOLEAN, STRING, SELECT, PATH,
|
||||
PERL) surfaced in the `configure` web UI.
|
||||
- **Work area** — `getWorkArea()` gives a plugin a persistent, non-web-accessible
|
||||
data directory.
|
||||
|
||||
### 3.3 Extension package types
|
||||
| Type | Role |
|
||||
|------|------|
|
||||
| **Plugin** | handler-based behavior, no core change (`lib/TWiki/Plugins/<Name>.pm`) |
|
||||
| **Skin** | visual appearance only; topic content unchanged (e.g. PrintSkin) |
|
||||
| **AddOn** | a script in `bin/` callable from a topic |
|
||||
| **Contrib** | shared library code, or **alternative implementations of core sections** (e.g. user management) when something "can't be a plugin because it needs close core access" |
|
||||
|
||||
## 4. Access control (origin of yawex's model)
|
||||
|
||||
TWiki authorizes via **preference settings in topics**: `ALLOWWEBVIEW` /
|
||||
`ALLOWWEBCHANGE` at web level and `ALLOWTOPICVIEW` / `ALLOWTOPICCHANGE` /
|
||||
`ALLOWTOPICRENAME` (and `DENY*`) at topic level, with **VIEW / CHANGE / RENAME
|
||||
grantable separately**. Topic-level settings apply only to that topic. This
|
||||
per-topic ACL is the **direct prior art** behind yawex's `AccessControl`
|
||||
(denied/view/form/edit/admin) and a concrete reference for shard-wiki's optional
|
||||
**per-page ACL at L4** (`spec/ArchitectureBlueprint.md` §5).
|
||||
|
||||
## 5. TWiki as a shard — capability profile
|
||||
|
||||
| Capability | TWiki | Note |
|
||||
|------------|-------|------|
|
||||
| Read | ✓ | TWiki::Func / `view` script / `rest`; or read `data/*.txt` directly |
|
||||
| Write | ✓ | saveTopic / `save` / `rest`; **per-topic** granularity |
|
||||
| Structured payload | ✓ | TWiki Forms → `%META:FIELD%` **in the text file** (git-diffable) |
|
||||
| Version/diff | ✓ | RCS `.txt,v` — **open format, git-convertible** (cf. XWiki DB) |
|
||||
| Merge | partial | `beforeMergeHandler`; no git 3-way |
|
||||
| Change hooks | ✓ | synchronous save/rename/attachment handlers |
|
||||
| Auth model | per-web/topic ACL | authorize through shard-wiki core; engine ACL is advisory |
|
||||
| Direct-store attach | ✓✓ | `data/<Web>/<Topic>.txt` + RCS is a folder shard on its own |
|
||||
| Federation hooks | ✓ | plugin handlers + REST handlers can host an adapter (UC-38) |
|
||||
|
||||
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 6.1 Reinforcements
|
||||
| Observation | INTENT principle |
|
||||
|-------------|------------------|
|
||||
| Forms store data as `%META%` **in the topic file** | structured pages can be **git-diffable**, strengthening UC-34/39 vs XWiki's DB |
|
||||
| RCS `.txt,v` history | **git-addressable coordination** — here history is *convertible*, not just supplementable (UC-36 → UC-41) |
|
||||
| `data/<Web>/<Topic>.txt` mirrors logic | an engine's **on-disk store is itself attachable as a folder shard** (UC-40) |
|
||||
| Plugin save/rename/REST handlers | **composable integration** — TWiki can host an adapter too, generalizing UC-38 beyond XWiki |
|
||||
| Per-topic `ALLOW/DENY` ACL | prior art for **per-page ACL at L4**; lineage TWiki→yawex (UC-06) |
|
||||
| Webs as namespaces / shards | namespace navigation (UC-22); webs as roots |
|
||||
|
||||
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||
| TWiki assumption | shard-wiki correction |
|
||||
|------------------|----------------------|
|
||||
| TWikiML + variable expansion is the page | **Markdown-first**; TWikiML render stays out of core |
|
||||
| ACL lives in topic preference settings | **authorize in core**; topic ALLOW/DENY is advisory provenance |
|
||||
| History = RCS files next to content | **coordination journal** is the space-level git layer (may *import* RCS) |
|
||||
| Apps = Forms + `%SEARCH%` rendered by TWiki | represent records **without** depending on TWiki to render/search them |
|
||||
| Attach = run TWiki and call it | shard-wiki may attach the **bare data dir** with TWiki offline (UC-40) |
|
||||
|
||||
### 6.3 What TWiki teaches that shard-wiki should not lose
|
||||
1. **File-backed engines are the easy, high-value case** — their store is already a
|
||||
folder of text + open-format history; the adapter can often skip the runtime.
|
||||
2. **Structured data in text beats structured data in a DB** for federation —
|
||||
diffable, portable, journal-friendly. Prefer file-embedded metadata where offered.
|
||||
3. **One backend, two attachment paths** (runtime API vs on-disk store) is a real
|
||||
capability/consistency trade-off the adapter contract must express.
|
||||
|
||||
## 7. Use-case seeds → catalog (promoted 2026-06-13)
|
||||
|
||||
| Seed | Catalog UC | Disposition |
|
||||
|------|------------|-------------|
|
||||
| Attach a live engine's **on-disk store directly** (vs its runtime API) | **UC-40** (new) | dual-path attach; fidelity/consistency trade-off |
|
||||
| **Import** an engine's native file-based history (RCS) into the journal | **UC-41** (new) | history *migration* with fidelity, vs UC-36 *supplementation* |
|
||||
| Engine hosts adapter via its plugin/handler API | UC-38 | enriched: TWiki save/rename/REST handlers generalize it beyond XWiki |
|
||||
| Structured page carries typed data | UC-34 / UC-39 | enriched: TWiki Forms `%META%`, git-diffable |
|
||||
| Internal-history engine | UC-36 | enriched: contrast DB (supplement) vs RCS (import, UC-41) |
|
||||
| Authenticated team wiki / per-page ACL | UC-06 | enriched: TWiki per-topic ACL is the origin of yawex's model |
|
||||
|
||||
## 8. Open questions (for spec / workplans)
|
||||
|
||||
1. **Dual-path adapters** — when does shard-wiki attach an engine's on-disk store
|
||||
directly vs go through its API? Consistency risk of reading live files under a
|
||||
running engine; capability-gate it (UC-40)?
|
||||
2. **History import fidelity** — can RCS `.txt,v` be converted to git commits
|
||||
preserving author/timestamp, and is that authoritative or a one-time backfill
|
||||
feeding the coordination journal (UC-41, cf. UC-36 Q3)?
|
||||
3. **File-embedded metadata mapping** — map TWiki `%META:FIELD%` (and XWiki XObjects)
|
||||
onto one structured-metadata page-model representation (shared with xwiki §8 Q1).
|
||||
4. **TWiki vs Foswiki** — target both via one adapter (shared API) or treat as
|
||||
distinct shard types?
|
||||
5. **ACL provenance** — surface a topic's `ALLOW/DENY` as read-only provenance even
|
||||
though authorization is decided in core?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
| Source | URL |
|
||||
|--------|-----|
|
||||
| TWiki — TWikiPlugins (handlers, package types) | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
|
||||
| TWiki — Developing Plugins / EmptyPlugin | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
|
||||
| TWiki — TWiki::Func module | https://twiki.org/cgi-bin/view/TWiki/TWikiFuncModule |
|
||||
| TWiki — TWiki Forms | https://twiki.org/cgi-bin/view/TWiki/TWikiForms |
|
||||
| TWiki — TWiki Access Control | https://twiki.org/cgi-bin/view/TWiki/TWikiAccessControl |
|
||||
| TWiki — System Requirements (Perl/RCS) | https://twiki.org/cgi-bin/view/TWiki/TWikiSystemRequirements |
|
||||
| TWiki — Source Code | https://twiki.org/cgi-bin/view/TWiki/SourceCode |
|
||||
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
|
||||
| Wikipedia — TWiki | https://en.wikipedia.org/wiki/TWiki |
|
||||
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
|
||||
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
|
||||
| shard-wiki — yawex prior art | `research/260608-yawex-prior-art/findings.md` |
|
||||
|
||||
---
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
| This document section | Informs (future) |
|
||||
|-----------------------|------------------|
|
||||
| §2 architecture | adapter design for file-backed engines |
|
||||
| §3 plugin handlers | UC-38 engine-side adapter; composable-integration API shape |
|
||||
| §4 access control | per-page ACL at L4 (`spec/ArchitectureBlueprint.md`), UC-06 |
|
||||
| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) |
|
||||
| §6 INTENT mapping | architecture-blueprint guardrails |
|
||||
| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-40, UC-41; UC-06/34/36/38/39 enrichment) |
|
||||
| §8 open questions | spec — dual-path adapters, history import, structured-metadata model |
|
||||
35
research/260613-xwiki-deep-dive/README.md
Normal file
35
research/260613-xwiki-deep-dive/README.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# 260613 — XWiki deep dive (implementation, extension interfaces, ecosystem)
|
||||
|
||||
Date: 2026-06-13
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **XWiki** — the canonical "wiki-as-application-platform"
|
||||
engine flagged in `research/260608-wikiengines-overview/`. Goes past the landscape
|
||||
scan into XWiki's **implementation** (component/DI architecture, document data
|
||||
model, storage, rendering, events), the **interfaces it exposes to extend the core
|
||||
engine** (components, macros, UI extension points, script services, REST), and the
|
||||
**extension ecosystem** (extensions.xwiki.org repository, XAR/JAR/WebJar types,
|
||||
notable extensions).
|
||||
|
||||
Read through one lens: **what does XWiki teach shard-wiki about (a) adapting a
|
||||
structured app-platform engine as a shard, and (b) how a host engine can become
|
||||
federation-capable through its own extension mechanism** (INTENT *composable
|
||||
integration*).
|
||||
|
||||
Complements:
|
||||
- `research/260608-wikiengines-overview/` — the landscape scan that surfaced XWiki
|
||||
- `research/260608-federation-concepts/` — already covered XWiki's ActivityPub extension
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, extension-interface catalog, ecosystem, capability profile, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to
|
||||
`spec/UseCaseCatalog.md` (UC-38, UC-39); existing UC-34/UC-36/UC-31 enriched with
|
||||
XWiki specifics. Adapter-contract implications feed
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
217
research/260613-xwiki-deep-dive/findings.md
Normal file
217
research/260613-xwiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# Findings — XWiki: implementation, extension interfaces, ecosystem
|
||||
|
||||
Date: 2026-06-13 · Status: research draft
|
||||
|
||||
Scope: XWiki as **prior art for two shard-wiki concerns** — (1) attaching a
|
||||
structured "wiki-as-application-platform" engine as a shard, and (2) how an engine
|
||||
exposes extension interfaces rich enough to *host* a federation adapter
|
||||
(INTENT *composable integration*). Sources are XWiki's own dev docs, the
|
||||
DeepWiki code analysis of `xwiki/xwiki-platform`, and the extensions repository.
|
||||
|
||||
**Complements:** `research/260608-wikiengines-overview/` (landscape) and
|
||||
`research/260608-federation-concepts/` §3.2 (XWiki ActivityPub).
|
||||
|
||||
---
|
||||
|
||||
## 1. What XWiki is
|
||||
|
||||
A **programmable, component-based wiki platform** (Java/Jakarta EE, Hibernate ORM)
|
||||
that blurs wiki and lightweight application platform. Pages are not just prose:
|
||||
they carry **typed structured objects**, so an XWiki page can be a form, a record,
|
||||
or a small application. This is the defining property the landscape scan flagged —
|
||||
and the reason XWiki is the hardest case for shard-wiki's Markdown-first page model.
|
||||
|
||||
## 2. Implementation architecture
|
||||
|
||||
### 2.1 Component system (the spine)
|
||||
- **Dependency-injection component model.** Components are interfaces (roles)
|
||||
discovered via `META-INF/components.txt` and instantiated by a `ComponentManager`.
|
||||
Implementations declare `@Component`, an optional `@Named` **role hint**, and
|
||||
`@Singleton`. No interface→impl dependency — impls can ship in separate JARs and
|
||||
**replace defaults at runtime**.
|
||||
- Core replaceable roles include `XWikiStoreInterface` (storage), `ObservationManager`
|
||||
(events), `XWikiRightService` (authorization), `XWikiAuthService` (authentication).
|
||||
|
||||
### 2.2 oldcore vs modular platform
|
||||
- `xwiki-platform-oldcore` holds the foundational document model, persistence, and
|
||||
legacy APIs (`com.xpn.xwiki.*`). A bridge connects it to the newer component world
|
||||
to avoid circular dependencies.
|
||||
- Newer modules layer on top: `model` (entity references), `rendering`, `security`,
|
||||
`web` (HTTP actions/templates), `search-solr`.
|
||||
|
||||
### 2.3 Data model — document-centric + class/object
|
||||
- Central entity: **`XWikiDocument`**, addressed by `DocumentReference`
|
||||
(wiki / space / page). Content is parsed into an **XDOM** (rendering tree).
|
||||
- **Structured data via a class-object pattern** (OOP-like):
|
||||
- `BaseClass` / **XClass** — schema (field definitions), like a class declaration
|
||||
- `BaseObject` / **XObject** — an instance conforming to a class, attached to a page
|
||||
- `PropertyClass` / `BaseProperty` — field definition / value
|
||||
- Property types: String, LargeString, Date, Number, List, …
|
||||
- A single page can hold content **plus N typed objects** plus attachments. This is
|
||||
the structural fact shard-wiki must represent without flattening.
|
||||
|
||||
### 2.4 Storage & history
|
||||
- Two-tier: **`XWikiCacheStore`** over **`XWikiHibernateStore`** (Hibernate, multi-DB).
|
||||
- Tables: `xwikidoc`, `xwikiobject`, `xwikiproperties`, `xwikiattachment`, and
|
||||
**`xwikircs`** — internal **RCS-style version history**. History is real but
|
||||
**engine-internal and non-portable** (directly relevant to shard-wiki UC-36).
|
||||
|
||||
### 2.5 Events & rendering
|
||||
- **`ObservationManager`** event bus: cancelable (`DocumentCreatingEvent`,
|
||||
`DocumentUpdatingEvent`, `DocumentDeletingEvent`) and post-facto
|
||||
(`DocumentCreatedEvent`, `DocumentUpdatedEvent`, …). Listeners are components —
|
||||
a natural **change-notification source for incremental sync**.
|
||||
- **Rendering pipeline:** source syntax → parse → XDOM → transformations (macros) →
|
||||
serialize (HTML, PDF, …). Pluggable parsers/renderers; XWiki can parse **Markdown**
|
||||
as an input syntax even though its native syntax differs.
|
||||
|
||||
### 2.6 Multi-wiki
|
||||
- One installation hosts many wikis (`WikiReference`), each with separate documents
|
||||
and configuration — XWiki's own multi-tenancy, conceptually adjacent to
|
||||
shard-wiki's root-entity/tenant boundary.
|
||||
|
||||
### 2.7 APIs & security
|
||||
- **Dual API surface:** internal `com.xpn.xwiki.*` (full power) and script
|
||||
`com.xpn.xwiki.api.*` (wrapped, enforces **programming rights** and view rights).
|
||||
Privilege is enforced at the script boundary, not the storage boundary.
|
||||
|
||||
## 3. Interfaces to extend the core engine
|
||||
|
||||
XWiki's extensibility is unusually layered. From lowest/most-powerful to highest:
|
||||
|
||||
| # | Mechanism | What it extends | How |
|
||||
|---|-----------|-----------------|-----|
|
||||
| 1 | **Components** | anything | implement a role interface, `@Component` + `@Named` hint; ship in a JAR with `components.txt`; can override defaults (store, auth, rights) |
|
||||
| 2 | **Rendering macros (Java)** | content processing | register `Macro.class` role + hint; participates in the XDOM transformation pipeline |
|
||||
| 3 | **Wiki macros** | content processing | a wiki page following a spec — **no Java, no compile**; in-wiki authoring |
|
||||
| 4 | **Script services** | scripting API | implement `org.xwiki.script.service.ScriptService`; exposed to Velocity/Groovy as `$services.<hint>` |
|
||||
| 5 | **UI Extensions (UIX)** at **UI Extension Points (UIXP)** | the interface, no skin edit | UIX stored as XObjects of `XWiki.UIExtensionClass` (label/target/icon); injected at named hooks (since v4.2), e.g. `org.xwiki.platform.panels.Applications` |
|
||||
| 6 | **REST API (JAX-RS)** | remote integration | resources for pages, spaces, **objects**, classes, attachments, comments, tags, annotations, users/groups, **history**; custom JAX-RS resources can be added |
|
||||
| 7 | **Event listeners** | reactive behavior | register a component on `ObservationManager` to react to document lifecycle events |
|
||||
| 8 | **Resource reference handlers** | new URL/resource types | implement `ResourceReferenceHandler` on the root component manager |
|
||||
| 9 | **Skins / templates / Velocity** | presentation | Flamingo skin macros, templates |
|
||||
|
||||
Two things stand out for shard-wiki: (a) the **component model lets an extension
|
||||
replace authentication, authorization, and storage** — i.e. an XWiki instance could
|
||||
delegate auth exactly as shard-wiki's blueprint prescribes; (b) the **REST API plus
|
||||
event bus** give a federation adapter both a transport (read/write pages+objects+
|
||||
history) and a push-based freshness signal.
|
||||
|
||||
## 4. Extension ecosystem
|
||||
|
||||
- **`extensions.xwiki.org`** repository: **900+ extensions** (applications, macros,
|
||||
skins, panels, themes, legacy plugins). Browsed/installed/upgraded at runtime via
|
||||
the **Extension Manager Application** from the admin UI.
|
||||
- **Extension types:**
|
||||
- **XAR** — a package of wiki pages (an application, a wiki macro, or a snippet)
|
||||
- **JAR** — server-side Java: **components and script services**
|
||||
- **WebJar** — JAR of web resources (JS/CSS/Less)
|
||||
- **"Plugin" (legacy) vs Component (modern):** the old plugin API is superseded by
|
||||
the component system; new code is components.
|
||||
- **Notable extensions** (relevance to shard-wiki in parentheses):
|
||||
- **AppWithinMinutes** — build structured apps (XClass+XObjects) from the UI
|
||||
(→ pages-as-records, the UC-39 case)
|
||||
- **LDAP / Active Directory** and **Entra ID (Azure AD)** authentication
|
||||
(→ confirms pluggable `XWikiAuthService`; mirrors shard-wiki's delegated authn)
|
||||
- **ActivityPub** — fediverse federation (→ adapter transport, cf. UC-31)
|
||||
- **LiveData / LiveTable** — queryable tabular views over XObjects
|
||||
- draw.io diagrams, PDF viewer, Task Manager, Blog — content/app breadth
|
||||
|
||||
## 5. XWiki as a shard — capability profile
|
||||
|
||||
Read through shard-wiki's capability-aware adapter lens:
|
||||
|
||||
| Capability | XWiki | Note |
|
||||
|------------|-------|------|
|
||||
| Read | ✓ | REST + render to HTML/Markdown |
|
||||
| Write | ✓ | REST PUT page/object; **per-document** granularity (page + its objects) |
|
||||
| Structured payload | ✓✓ | XObjects/XClass — *richer* than Markdown; must not be flattened (UC-34/UC-39) |
|
||||
| Version/diff | ✓ (internal) | `xwikircs`; **not git-portable** → coordination journal supplies that (UC-36) |
|
||||
| Merge | partial | no native git-style 3-way merge |
|
||||
| Change events | ✓ | `ObservationManager` → push-based sync/freshness (enriches UC-31) |
|
||||
| Auth model | pluggable | `XWikiAuthService`/`XWikiRightService` replaceable — authorize through shard-wiki core, don't trust engine ACLs |
|
||||
| Federation hooks | ✓✓ | component + REST + UIX make XWiki able to **host** a shard-wiki adapter (UC-38) |
|
||||
| Multi-tenancy | ✓ | `WikiReference` farm → multiple shards / root entities |
|
||||
|
||||
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 6.1 Reinforcements
|
||||
| Observation | INTENT principle |
|
||||
|-------------|------------------|
|
||||
| Pages = content + typed XObjects | **Markdown-first, backend-neutral** must carry structure, not flatten (UC-34/39) |
|
||||
| `xwikircs` internal-only history | **Git-addressable coordination** adds the portable layer (UC-36) |
|
||||
| Pluggable `XWikiAuthService` | **authorization in core, authentication delegated** — XWiki already does exactly this |
|
||||
| Component + REST + UIX extension surface | **composable integration** — engine becomes federation-capable via its own API (UC-38) |
|
||||
| `ObservationManager` events | **explicit provenance/freshness** via push, not poll (UC-31, UC-24) |
|
||||
| `WikiReference` multi-wiki | root-entity / tenant boundary has prior art |
|
||||
|
||||
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||
| XWiki assumption | shard-wiki correction |
|
||||
|------------------|----------------------|
|
||||
| Native XWiki syntax + macro render is the page | shard-wiki keeps **Markdown-first**; engine render stays out of core, structure travels as metadata |
|
||||
| ACLs/rights live in the engine DB | **authorize in core**; engine rights are advisory, not trusted |
|
||||
| History is the engine's RCS table | **coordination journal** is space-level and git-backed |
|
||||
| Federation = an XWiki extension talking to other XWikis | shard-wiki federates **heterogeneous** backends; XWiki is one shard among many |
|
||||
| App = XObjects rendered by XWiki | shard-wiki must represent records **without** depending on XWiki to render them |
|
||||
|
||||
### 6.3 What XWiki teaches that shard-wiki should not lose
|
||||
1. **A clean role/hint component model** is what makes auth/store/rights swappable —
|
||||
shard-wiki's adapter contract should be similarly role-based and override-friendly.
|
||||
2. **Structured objects are first-class**, not an afterthought — the page model needs
|
||||
a typed-metadata escape hatch from day one.
|
||||
3. **An engine with a real extension API can host the adapter itself** — the cheapest
|
||||
path to a high-fidelity shard is engine-side, not screen-scraping.
|
||||
|
||||
## 7. Use-case seeds → catalog (promoted 2026-06-13)
|
||||
|
||||
| Seed | Catalog UC | Disposition |
|
||||
|------|------------|-------------|
|
||||
| Engine hosts a federation adapter via its native extension API | **UC-38** (new) | composable integration — first UC for the engine-side direction |
|
||||
| Attach a wiki-as-application-platform shard (pages are typed records/forms) | **UC-39** (new) | extends UC-34 toward bodiless structured pages |
|
||||
| Structured page carries semantic data | UC-34 | enriched with XObject/XClass concretes |
|
||||
| Internal-only revision history | UC-36 | enriched with `xwikircs` example |
|
||||
| Subscribe to remote shard changes | UC-31 | enriched with `ObservationManager` event-driven transport |
|
||||
|
||||
## 8. Open questions (for spec / workplans)
|
||||
|
||||
1. **Page-model representation of XObjects** — typed frontmatter, sidecar file, or
|
||||
opaque provenance blob? (shared with `wikiengines` §6 Q1; XWiki makes it urgent)
|
||||
2. **Bodiless record pages** — does shard-wiki's page model require a Markdown body,
|
||||
or can a page be purely structured (AppWithinMinutes apps)?
|
||||
3. **Adapter placement** — engine-side extension (high fidelity, needs deploy access)
|
||||
vs external REST adapter (zero engine change, lower fidelity). Both? Capability-gated?
|
||||
4. **Event transport** — consume `ObservationManager` via a JAR listener, or poll REST
|
||||
history? Affects freshness guarantees (UC-31/UC-24).
|
||||
5. **Rights mapping** — translate XWiki `XWikiRightService` rights into shard-wiki
|
||||
actions (read/write/patch/merge/administer), or ignore and authorize purely in core?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
| Source | URL |
|
||||
|--------|-----|
|
||||
| XWiki — Developer Guide | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/ |
|
||||
| XWiki — Creating XWiki Components | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/WritingComponents/ |
|
||||
| XWiki — Available Extension Points | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/ExtensionPoint/ |
|
||||
| XWiki — UI Extension Point Tutorial | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/UIXTutorial/ |
|
||||
| XWiki — Wiki Macro Tutorial | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/WritingMacros/WikiMacroTutorial/ |
|
||||
| XWiki — RESTful API | https://www.xwiki.org/xwiki/bin/view/Documentation/UserGuide/Features/XWikiRESTfulAPI |
|
||||
| XWiki Rendering — Writing a Macro | https://rendering.xwiki.org/xwiki/bin/view/Main/ExtendingMacro |
|
||||
| Extension Manager Application | https://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Manager%20Application |
|
||||
| Extension Module — Extensions/types | https://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Module/Extensions/ |
|
||||
| DeepWiki — xwiki/xwiki-platform (code analysis) | https://deepwiki.com/xwiki/xwiki-platform |
|
||||
| Mancinelli — Overview of XWiki's RESTful API | http://blog.fabio.mancinelli.me/2011/03/07/XWikis_RESTful_API.html |
|
||||
| shard-wiki — wiki engines overview | `research/260608-wikiengines-overview/findings.md` |
|
||||
| shard-wiki — federation concepts | `research/260608-federation-concepts/findings.md` |
|
||||
|
||||
---
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
| This document section | Informs (future) |
|
||||
|-----------------------|------------------|
|
||||
| §2 architecture | adapter design for app-platform engines |
|
||||
| §3 extension interfaces | UC-38 engine-side adapter; composable-integration API shape |
|
||||
| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) |
|
||||
| §6 INTENT mapping | architecture-blueprint guardrails |
|
||||
| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-38, UC-39; UC-31/34/36 enrichment) |
|
||||
| §8 open questions | spec — page model for structured/bodiless pages |
|
||||
38
research/260614-computational-page-model-synthesis/README.md
Normal file
38
research/260614-computational-page-model-synthesis/README.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# 260614 — The computational page model (SHARD-WP-0004 synthesis)
|
||||
|
||||
Date: 2026-06-15 · Source: **SHARD-WP-0004** post-batch synthesis (T1–T8)
|
||||
|
||||
## What this is
|
||||
|
||||
The acceptance-criteria synthesis for SHARD-WP-0004 — *"the computational page model"* —
|
||||
reading the eight computational/interactive-knowledge dives across each other and distilling
|
||||
them into **one model**: **source is canonical; everything rendered/computed is a
|
||||
projection**, placed on **two axes** (projection-kind: replication vs derivation; liveness:
|
||||
live↔snapshot), with a recommendation on an executable-content capability.
|
||||
|
||||
## The answer to the carried question
|
||||
|
||||
*Can a shard-wiki page be a live computational artifact?* **Yes — as a page-model +
|
||||
projection concern, not as an execution platform.** Every system externalizes to a canonical
|
||||
source and treats the live/computed form as derived; shard-wiki **recognizes** computational
|
||||
content, **attaches the source**, and **presents derivations as provenance- and
|
||||
liveness-marked projections**, with **execution as a gated capability (off by default,
|
||||
degrade to snapshot)**. No INTENT amendment required.
|
||||
|
||||
## Key contributions
|
||||
|
||||
- **One model:** `(source, derivation rule, projection with provenance + liveness)` covers
|
||||
all four computational page shapes (one-source-many-projections UC-83; notebook UC-84;
|
||||
program-as-page; live/temporal content).
|
||||
- **Two axes for T16:** replication vs **derivation-projection** (timing / multiplicity /
|
||||
continuity facets) × the **live↔snapshot** axis (bounded at the irreducibly-live far end by
|
||||
Strudel).
|
||||
- **One snapshot-provenance record** reused for notebook outputs, renders, recordings.
|
||||
- **Hard boundaries:** never host a kernel/runtime as store; **image-is-not-a-store**; never
|
||||
present a derivation without output→source provenance.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | The source/derivation/projection model, the two axes, the four page shapes, provenance/reproducibility, the recommendation, the SHARD-WP-0002 fold-in, escalated open questions |
|
||||
178
research/260614-computational-page-model-synthesis/findings.md
Normal file
178
research/260614-computational-page-model-synthesis/findings.md
Normal file
@@ -0,0 +1,178 @@
|
||||
# The computational page model — synthesis (SHARD-WP-0004)
|
||||
|
||||
**Date:** 2026-06-15 · **Source:** SHARD-WP-0004 (post-batch synthesis, T1–T8) · **Kind:**
|
||||
synthesis (no new external research) reading the eight computational/interactive-knowledge
|
||||
dives *across* each other.
|
||||
|
||||
## What this is
|
||||
|
||||
The acceptance-criteria synthesis for SHARD-WP-0004. The batch asked one carried question:
|
||||
*can a shard-wiki page be a live computational artifact — a source woven/evaluated into
|
||||
rendered forms — and if so, how do projection, transclusion, provenance, and the adapter
|
||||
contract treat the source, the environment, and the computed output?* This memo answers it
|
||||
by distilling the eight dives into **one model**: the **source / derivation / projection**
|
||||
view of computational content, anchored on two axes, plus a recommendation on whether
|
||||
shard-wiki needs an executable-content capability.
|
||||
|
||||
Dives consolidated: **T1** literate programming (WEB/weave/tangle), **T2** Mathematica,
|
||||
**T3** Jupyter, **T4** Processing/p5.js, **T5** Strudel, **T6** Squeak, **T7** Glamorous
|
||||
Toolkit, **T8** Pharo. Catalog yield: **UC-83** (literate one-source-many-projections),
|
||||
**UC-84** (notebook with computed-output provenance); the other six are **enrichment/boundary**
|
||||
dives (UC-54/55/47/48 + the projection model).
|
||||
|
||||
## 1. The one finding: source is canonical, everything rendered is a projection
|
||||
|
||||
Every system in the batch, however "live," **externalizes to a canonical artifact** and
|
||||
treats the rendered/computed form as **derived**:
|
||||
|
||||
| System | Canonical (attach this) | Derived (a projection) |
|
||||
|--------|-------------------------|------------------------|
|
||||
| Literate (WEB) | the WEB/`.nw`/`.org` **source** | woven docs **and** tangled code |
|
||||
| Mathematica | the `.nb` (a Wolfram expression) | cached `Output` cells; CDF; `Dynamic` widgets |
|
||||
| Jupyter | `.ipynb` source (ideally paired text) | embedded outputs; nbconvert/nbviewer renders |
|
||||
| Processing | the **sketch source** | the view-time canvas render |
|
||||
| Strudel | the **pattern source** | the live audio performance / a recording |
|
||||
| GT / Lepiter | git-versionable **page/Tonel files** | moldable `gtView`s; live snippet results |
|
||||
| Squeak/Pharo | exported **files** (Tonel/git) | the live image / running objects |
|
||||
|
||||
**This is the same principle shard-wiki already holds** (files-canonical, index/render
|
||||
derived — ikiwiki UC-79, Logseq UC-62, nbstripout). The computational batch did not break it;
|
||||
it **stress-tested it to the live extreme and it held**. *The image (Squeak/Pharo) is the
|
||||
only would-be exception, and it is a boundary, not a counterexample: an image is not a store.*
|
||||
|
||||
## 2. Two axes the contract must add
|
||||
|
||||
The batch refines the **projection model** (SHARD-WP-0002 T16) with two orthogonal axes:
|
||||
|
||||
### 2.1 Projection kind: replication vs derivation
|
||||
- **Replication-projection** (the existing default): a lazy **cache/copy** of remote content
|
||||
(Obsidian/Notion mirrors, UC-53/57).
|
||||
- **Derivation-projection** (new, from T1): a **transform/compile/weave/evaluate** of a
|
||||
source into rendered forms — regenerable, may delegate to the source's tool, **degrades to
|
||||
a captured snapshot** when the tool is absent. Covers weave/tangle, nbconvert, CDF, sketch
|
||||
render, audio recording, `gtView`.
|
||||
|
||||
Sub-facets of derivation (from T4): **materialization timing** = *ahead-of-time* (CDF,
|
||||
nbconvert, static HTML) vs *view-time* (Processing/Strudel); **multiplicity** = one output
|
||||
(UC-79) vs **N co-equal projections** (UC-83 weave+tangle; UC-47/48/54 moldable views).
|
||||
|
||||
### 2.2 Liveness: the live↔snapshot axis (from T6, bounded by T5)
|
||||
Every derived view sits on a spectrum, and the more live the source, the more its static form
|
||||
is a clearly-marked degrading snapshot:
|
||||
|
||||
```
|
||||
static source ── captured output ── live-over-files ── view-time one-shot ── continuous/
|
||||
(literate) (notebook UC-84) (GT/Lepiter) (Processing) interactive
|
||||
── irreducibly
|
||||
live/temporal
|
||||
(Strudel: source
|
||||
+ recording only)
|
||||
```
|
||||
|
||||
**Honesty rule (union-without-erasure):** a computed/live view must always declare *what it
|
||||
is* — "captured at run N, environment unguaranteed" (UC-84), "one performance, time T, source
|
||||
rev R" (UC-83/Strudel), "live render needs the runtime." Never present a snapshot as live or a
|
||||
static page as capturing a live artifact.
|
||||
|
||||
## 3. The computational page-model shapes (T12)
|
||||
|
||||
The batch adds these page shapes (beyond prose / typed records / query-defined / inline-
|
||||
embedded objects / typed-graph already catalogued):
|
||||
|
||||
1. **One-source-many-projections** (UC-83) — a source whose presented forms are co-equal
|
||||
derivations (docs + code), each with output→source provenance; **named-chunk transclusion**
|
||||
assembles fragments by name at derivation time (UC-32/44).
|
||||
2. **Notebook** (UC-84) — **ordered/nestable cells** (Mathematica adds the outline tree)
|
||||
where code cells own **embedded computed outputs** (the derived output is stored *inside*
|
||||
the source) with **weak execution provenance**; outputs may be MIME blobs or **structured
|
||||
re-evaluable values** (Mathematica) — a new point on the content-opacity spectrum.
|
||||
3. **Program-as-page** (Processing) — canonical content = **source text**, presentation = an
|
||||
**executable render** with **no cached output**; non-Markdown executable content.
|
||||
4. **Live/temporal/generative content** (Strudel) — source canonical, render irreducibly
|
||||
live; static = source + a marked recording.
|
||||
|
||||
All four reduce to **(source, derivation rule, projection with provenance + liveness)** — one
|
||||
model, four positions.
|
||||
|
||||
## 4. Provenance & reproducibility (the honest weakness)
|
||||
|
||||
Computed output provenance is **real but fragile** everywhere: Jupyter/Mathematica
|
||||
`execution_count`/`In`-`Out` can be **out-of-order**; **environment/versions/data are not
|
||||
captured**; Strudel/Processing may be **non-deterministic**. Implication for the contract:
|
||||
treat a computed output as a **snapshot with declared, incomplete provenance** (run id, source
|
||||
rev, timestamp; environment "unguaranteed"), reusing **one snapshot-provenance machinery**
|
||||
across notebooks, recordings, and renders (UC-84 is the template). This is consistent with
|
||||
shard-wiki's existing "surface freshness/completeness, never imply more than you have"
|
||||
(Oddmuse partial-history UC-82).
|
||||
|
||||
## 5. The recommendation
|
||||
|
||||
**Does shard-wiki need an executable/computational content capability? — Yes, but only as
|
||||
recognition + projection + capability-gating, never as an execution engine.**
|
||||
|
||||
1. **Adopt the source/derivation/projection model** (no execution required). shard-wiki
|
||||
**recognizes** computational content types (literate source, notebook, sketch, pattern),
|
||||
**attaches the canonical source**, and **presents derived forms as projections with
|
||||
provenance + liveness markers**. This alone delivers UC-83/UC-84 and the enrichment of
|
||||
UC-54/55 — and needs **no kernel, no sandbox, no runtime**.
|
||||
2. **Make execution a capability, off by default.** "Drive a derivation" (run tangle/weave,
|
||||
re-execute a notebook, render a sketch, evaluate a pattern in the viewer) is a **gated
|
||||
capability** with a **trust/sandbox** sub-concern (T11). Absent it, **degrade to the
|
||||
captured snapshot / static render / recording** — the graceful-degradation rule, which the
|
||||
batch shows always has an honest fallback (source is tiny and diffable everywhere).
|
||||
3. **One projection model, two axes** (T16): projection-kind (replication vs derivation; with
|
||||
timing + multiplicity facets) × liveness (live↔snapshot). The **moldable view registry**
|
||||
(T7) is the unifying structure — an open, type-keyed set of co-equal projections, none
|
||||
canonical-by-fact (display-canonical is policy).
|
||||
4. **One snapshot-provenance record** reused for notebook outputs, renders, and recordings
|
||||
(run id, source rev, timestamp, environment "unguaranteed").
|
||||
5. **Hard boundaries** (design-bugs if violated): never host a kernel/runtime as the store;
|
||||
**image-is-not-a-store** (attach exported files); never present a derivation without
|
||||
output→source provenance; never imply a static view captures a live artifact.
|
||||
|
||||
**Net:** computational content is **in scope as a page-model + projection concern**, **out of
|
||||
scope as an execution platform** — exactly the mechanism-over-policy, capability-aware,
|
||||
degradable posture INTENT already mandates. No INTENT amendment is required; this extends the
|
||||
page model and projection model within existing constraints.
|
||||
|
||||
## 6. Fold into SHARD-WP-0002
|
||||
|
||||
- **T12 (page model):** add the four computational shapes (§3); allow nestable cells and
|
||||
structured re-evaluable outputs; "derived output may live inside the source" (notebook).
|
||||
- **T16 (projection):** the **two-axis model** (§2) + the **moldable view registry** (§3/T7);
|
||||
materialization-timing, multiplicity, continuity, and the live↔snapshot far end as explicit
|
||||
projection metadata.
|
||||
- **T11 (capabilities):** "derive/execute/render/evaluate" as gated capabilities with trust/
|
||||
sandbox; default off → snapshot.
|
||||
- **T15 (fidelity):** non-Markdown executable/computed content; lossy renders; the
|
||||
structured-re-evaluable-value point on the content-opacity spectrum.
|
||||
- **T13 (history):** paired-text (Jupytext) / cell-aware (nbdime) strategies for embedded-
|
||||
output documents; outputs-as-derived (nbstripout ethos).
|
||||
- **T14 (binding):** **image-is-not-a-store** boundary (export→files only).
|
||||
|
||||
## 7. Open questions (escalated)
|
||||
|
||||
1. Is **liveness** (and "irreducibly live / no faithful static form") an explicit first-class
|
||||
metadata flag on every projection, so the union renders the honest fallback automatically?
|
||||
(T5/T6 far-end question.)
|
||||
2. Does shard-wiki **ever drive a derivation** (sandboxed), or strictly attach + present
|
||||
snapshots? (Recurs UC-56/UC-83/UC-84/T4/T5 — a single capability/trust policy decision.)
|
||||
3. Is a computed output's **structured re-evaluable value** (Mathematica/Wolfram) modeled as a
|
||||
typed value or stored opaquely with provenance? (UC-55 open-Q #10; UC-84 Q3.)
|
||||
4. Should **UC-83** and **UC-84** eventually merge as two positions of one "source +
|
||||
derivations" shape, or stay distinct? (Kept distinct: UC-84's defining trait is *output
|
||||
embedded in source with weak provenance*; UC-83's is *N co-equal external derivations*.)
|
||||
|
||||
## 8. Sources
|
||||
|
||||
The eight SHARD-WP-0004 dives: `research/260614-{literate-programming,mathematica,jupyter,
|
||||
processing,strudel,squeak-pharo,glamorous-toolkit}-deep-dive/`. Prior projection/structure
|
||||
anchors: `research/260614-{ikiwiki,logseq,zigzag}-deep-dive/`,
|
||||
`research/260614-shard-spectrum-synthesis/`.
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
No new UC (consolidation). Consolidates **UC-83, UC-84** and enrichments to **UC-32, UC-44,
|
||||
UC-47, UC-48, UC-54, UC-55, UC-79, UC-37, UC-35**. Feeds **SHARD-WP-0002 T11/T12/T13/T14/T15/
|
||||
T16** (see §6). Recommendation: computational content is **in scope as page-model + projection
|
||||
mechanism, out of scope as an execution platform**; no INTENT amendment required.
|
||||
20
research/260614-federated-wiki-deep-dive/README.md
Normal file
20
research/260614-federated-wiki-deep-dive/README.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# 260614 — Federated Wiki deep dive
|
||||
|
||||
Deep dive on Ward Cunningham's **Federated Wiki** (Smallest Federated Wiki / SFW,
|
||||
2011 →) as a **federation model** rather than a single shard: fork-with-provenance,
|
||||
the per-page **JSON journal** of semantic actions, the **story** of typed items, the
|
||||
**neighborhood/roster** discovery model, and time-bounded **happenings**.
|
||||
|
||||
This is prior art for shard-wiki's **coordination layer itself** — the closest existing
|
||||
system to "a union of pages preserving provenance, assembled non-destructively." It
|
||||
extends `research/260608-federation-concepts/` §3 with the concrete data model + protocol.
|
||||
|
||||
- `findings.md` — full writeup: data model, journal/action types, federation protocol,
|
||||
capability profile, INTENT mapping, UC seeds (UC-70–UC-72), architecture notes for
|
||||
SHARD-WP-0002, open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-70 (attach a fedwiki site via page-JSON + CORS), UC-71 (append-only
|
||||
semantic action journal with site provenance as a coordination-journal model), UC-72
|
||||
(fork-with-site-provenance federation across a neighborhood / chorus). Enriched
|
||||
UC-26/28/30/05/27. Feeds SHARD-WP-0002 T1–T5 (federation) and T11/T13/T16 (write
|
||||
granularity, log-based merge, identity≠placement).
|
||||
240
research/260614-federated-wiki-deep-dive/findings.md
Normal file
240
research/260614-federated-wiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,240 @@
|
||||
# Federated Wiki — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T1 · **Subject:** Ward Cunningham's
|
||||
Smallest Federated Wiki (SFW) / Federated Wiki (fedwiki ecosystem).
|
||||
|
||||
## Why this dive
|
||||
|
||||
Every prior dive has been a *shard candidate* — a store we might attach. Federated Wiki
|
||||
is different: it is a **federation model**, the one piece of public prior art whose core
|
||||
job is the same as shard-wiki's coordination layer — *present a union of pages from many
|
||||
independent sites while preserving where each came from, and let people copy and edit
|
||||
non-destructively*. Ward Cunningham (inventor of the wiki) built SFW in 2011 precisely to
|
||||
fix the original wiki's single-canonical-page weakness with **fork + provenance**. We go
|
||||
past the surface (`260608-federation-concepts/` §3) into the data model and protocol, then
|
||||
ask what shard-wiki should adopt.
|
||||
|
||||
**Framing:** fedwiki is not just "a shard we attach" — it is a *worked example of the
|
||||
coordination journal, overlay-before-mutation, and union-without-erasure*, three of our
|
||||
own design pillars, shipped and running.
|
||||
|
||||
---
|
||||
|
||||
## 1. The data model — page = title + story + journal
|
||||
|
||||
A fedwiki page is a small JSON object with three core fields (plus optional decoration):
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Welcome Visitors",
|
||||
"story": [
|
||||
{ "type": "paragraph", "id": "7b56f22a4b9ee974",
|
||||
"text": "Welcome to this [[Federated Wiki]] site." },
|
||||
{ "type": "image", "id": "a1c0e3...", "url": "...", "caption": "..." }
|
||||
],
|
||||
"journal": [
|
||||
{ "type": "create", "id": "7b56f22a4b9ee974", "item": {...}, "date": 1310000000000 },
|
||||
{ "type": "add", "id": "a1c0e3...", "item": {...}, "after": "7b56f22a4b9ee974",
|
||||
"date": 1310000100000 },
|
||||
{ "type": "edit", "id": "7b56f22a4b9ee974", "item": {...}, "date": 1310000200000 },
|
||||
{ "type": "fork", "site": "ward.fed.wiki.org", "date": 1310000300000 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- **story** — an *ordered array of typed items* ("paragraph-like" items). Each item is
|
||||
`{ type, id, text, ...type-specific }`. The **`id`** is a random 16-hex string,
|
||||
**stable across edits** (it is the unit of identity within a page). The **`type`** names
|
||||
the **plugin** that renders/edits the item (`paragraph`, `image`, `html`, `markdown`,
|
||||
`code`, `method`, `pagefold`, chart plugins, …). *Data lives in the item; behavior lives
|
||||
in the plugin* — the item is portable JSON; the plugin is the renderer.
|
||||
- **journal** — an *ordered, append-only array of action objects* that, when replayed,
|
||||
**reconstructs the story**. The story is a materialized view of the journal. This is the
|
||||
key architectural choice: **the journal is the source of truth, the story is derived.**
|
||||
|
||||
## 2. Journal action types — a semantic op-log
|
||||
|
||||
Each journal entry is an action with `{ type, ... , date }` (epoch-ms). The action types:
|
||||
|
||||
| action | fields | meaning |
|
||||
|---------|--------|---------|
|
||||
| `create`| `id, item, date` | first item — page born |
|
||||
| `add` | `id, item, after, date` | insert an item after another |
|
||||
| `edit` | `id, item, date` | replace an item's content (id preserved) |
|
||||
| `move` | `order, date` | reorder items |
|
||||
| `remove`| `id, date` | delete an item |
|
||||
| `fork` | `site, date` | **mark that the page was copied from `site` at this point** |
|
||||
|
||||
Two things matter for us:
|
||||
|
||||
1. **These are *semantic* operations** (add/move/edit/remove a paragraph), not text diffs
|
||||
and not character-level CRDT ops. The write granularity is the **story item
|
||||
(paragraph)** — a *middle* granularity between whole-file (TiddlyWiki) and
|
||||
block/character (Logseq/CRDT). It is an **op-log** like a CRDT, but the ops are
|
||||
coarse-grained and **applied by humans via fork**, not auto-merged.
|
||||
2. **`fork` is the provenance primitive.** When you copy a remote page to your own site,
|
||||
a `fork` entry is appended recording the **source site** and time. The journal of a
|
||||
forked page therefore **serializes a directed acyclic graph (DAG)** of where content
|
||||
came from — "the journal of a forked page is detailed enough to recognize where in the
|
||||
journal of the original the fork took place" (CouchDB-style per-entry sequence numbers
|
||||
make the cut-point identifiable). History visualization highlights the forked entry.
|
||||
|
||||
## 3. The federation protocol — sites, neighborhood, roster
|
||||
|
||||
- **Site** = an independent server (originally Node.js; also static-file and serverless
|
||||
variants). A site owns a set of pages, each served as **page JSON over HTTP** at
|
||||
`/<slug>.json`, with **CORS headers** so a *browser-side* client can fetch pages from
|
||||
**any** site. Page identity within a site is the **slug** (a title-derived kebab name).
|
||||
- **The client assembles the union, not the server.** The fedwiki client ("the lineup")
|
||||
renders pages **side by side**: clicking a link opens that page *from whatever site it
|
||||
resolves against*, appended to the right. Browsing literally builds a left-to-right
|
||||
trail across sites.
|
||||
- **Neighborhood** = the dynamic set of sites encountered in the current session (from the
|
||||
sites of pages you've opened, links, and forks). **Search runs across the neighborhood**
|
||||
— a federated search over exactly the sites you've touched.
|
||||
- **Roster** = an explicit, authored list of sites to include (a curated neighborhood);
|
||||
"sister sites" are peers you watch. There is **no central registry** — discovery is by
|
||||
link, fork, and roster.
|
||||
- **Happenings** = time-bounded collaborative events where many participants fork around a
|
||||
topic for a period, producing a burst of related forks (a bounded collaboration that
|
||||
leaves a durable forked record on each participant's own site).
|
||||
|
||||
## 4. The editorial model — fork, don't edit-in-place
|
||||
|
||||
You can only write to **your own** site. To change someone else's page you **fork** it
|
||||
(copy into your site, journal records the source), then edit your copy. Many forks of the
|
||||
same page coexist across sites — Cunningham's **"chorus of voices"**: *no canonical
|
||||
version*, divergence is normal and visible, and you choose whose changes to pull by forking
|
||||
them. There is **no automatic merge** — reconciliation is human: compare journals, fork the
|
||||
version you prefer, optionally re-fork upstream changes.
|
||||
|
||||
---
|
||||
|
||||
## 5. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | Federated Wiki |
|
||||
|--------------------------------|----------------|
|
||||
| Attachment mode | **REST/file-store hybrid** — page JSON over HTTP+CORS; also static files |
|
||||
| Addressing granularity | **story item (paragraph)** via stable 16-hex `id` |
|
||||
| Content identity | item `id` random+stable; page id = site + slug |
|
||||
| Identity vs placement | **placement-bound**: identity = `site` + `slug`; forks are *new* identities linked by journal provenance |
|
||||
| Structure | ordered array of **typed items** (plugin-typed) |
|
||||
| History | **per-page append-only journal** of semantic actions (op-log) |
|
||||
| Merge model | **fork + manual journal compare** — a *third model* beside git 3-way and CRDT auto-merge |
|
||||
| Native query | none built-in; **neighborhood search** (federated full-text across touched sites) |
|
||||
| Translation | item `text` is wiki/Markdown-ish; plugins own their formats |
|
||||
| Attachment/write granularity | **story-item level** (add/edit/move/remove one item) |
|
||||
| Operational envelope | tiny servers, browser-driven; CORS is the whole API surface |
|
||||
| Access grant | **own-site-only writes**; reads open via CORS |
|
||||
| Content opacity | transparent JSON (no E2EE); plugin-typed but inspectable |
|
||||
| Provenance | **first-class** — `fork` records source site; journal = provenance DAG |
|
||||
|
||||
## 6. INTENT mapping
|
||||
|
||||
### Reinforcements (fedwiki validates our pillars)
|
||||
|
||||
- **Coordination journal** (INTENT) ≈ fedwiki **journal**. Our journal idea is *exactly*
|
||||
fedwiki's per-page append-only action log — and fedwiki proves the story-as-derived-view
|
||||
pattern works. Strong reinforcement; adopt the **semantic-op + provenance-entry** shape.
|
||||
- **Overlay before mutation** ≈ **fork**. Fork *is* the canonical overlay: a
|
||||
non-destructive copy onto a writable surface, recording provenance, before any change.
|
||||
- **Union without erasure** ≈ **neighborhood + chorus**. The union is assembled from many
|
||||
sovereign sites; provenance (which site, forked-from) is never hidden; divergence is
|
||||
surfaced, not resolved away.
|
||||
- **No silent remote mutation** ≈ **own-site-only writes**. You structurally *cannot*
|
||||
mutate a remote; you fork to your own site. This is our rule, enforced by architecture.
|
||||
- **Mechanism over policy** ≈ **no canonical source**. Fedwiki ships the mechanism (fork,
|
||||
journal, neighborhood) and leaves "which version wins" entirely to people.
|
||||
- **Graceful degradation** ≈ static-file sites — a fedwiki site can be a read-only pile of
|
||||
JSON files; still forkable, still in the neighborhood.
|
||||
|
||||
### Divergences (boundaries / design notes, not bugs)
|
||||
|
||||
- **Identity = placement.** Fedwiki page identity is `site` + `slug`; a fork is a *new*
|
||||
page whose only tie to the origin is a journal `fork` entry. shard-wiki wants
|
||||
**identity ≠ placement** (the "same" page across shards under a stable identity, T16) —
|
||||
so we treat fedwiki's journal-linked forks as *provenance edges*, and layer our own
|
||||
cross-shard identity over them rather than adopting slug-as-identity.
|
||||
- **No query / no typed-record model.** Fedwiki is paragraphs+plugins, not a typed DB
|
||||
(contrast Notion/Wikibase). Fine — it sits at the *coordination* end, not the structure
|
||||
end. We don't ask fedwiki to provide query; the neighborhood search is the model for
|
||||
*federated* search across shards (T-federation), not in-shard query.
|
||||
- **Browser-assembles-union.** Fedwiki pushes union assembly to the client. shard-wiki
|
||||
assembles server/orchestrator-side. Adopt the *model* (union from sovereign sources +
|
||||
provenance), not the client-only locus.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Journal = append-only semantic-op log with provenance entries**, story = derived
|
||||
replay view. This is the concrete shape for our coordination journal (T13).
|
||||
2. **Fork-with-source-attribution** as the overlay/adopt primitive across shards.
|
||||
3. **Neighborhood** as the model for a *dynamic, link-and-fork-discovered* federated set +
|
||||
search, with **roster** as the curated/explicit variant.
|
||||
4. **Chorus of forks** — represent divergent versions across shards as co-equal, linked by
|
||||
provenance, with reconciliation as an explicit human/policy step (mechanism over policy).
|
||||
|
||||
---
|
||||
|
||||
## 7. UC seeds
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-70 | Attach a Federated Wiki site as a shard via its **page JSON + CORS** (REST/file-store hybrid); project pages, fork to overlay | **new** |
|
||||
| UC-71 | Adopt a **per-page append-only semantic-action journal with provenance entries** (fork=source site) as the coordination-journal model — replay to materialize, compare to locate divergence | **new** |
|
||||
| UC-72 | **Fork-with-site-provenance federation across a neighborhood** of peer shards — assemble a union from links/forks, search across it, preserve the chorus without forcing a canonical | **new** |
|
||||
| — | fork-with-provenance as overlay/adopt | enrich **UC-26** (fork) |
|
||||
| — | carry-forward of forked content + upstream re-fork | enrich **UC-28** (carry-forward) |
|
||||
| — | happenings = time-bounded collaboration leaving durable forks | enrich **UC-30** (time-bounded space) |
|
||||
| — | union/chorus of co-equal versions, provenance-linked | enrich **UC-05 / UC-27** |
|
||||
|
||||
## 8. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T1–T5 (federation):** fedwiki is the reference design. The **journal** (append-only,
|
||||
semantic ops, fork-provenance) is the concrete coordination-journal shape; **neighborhood
|
||||
+ roster** is the discovery/membership model (dynamic vs curated); **fork** is the
|
||||
overlay/adopt op. Model the union as an assembly over sovereign sources with provenance
|
||||
edges, reconciliation left to policy.
|
||||
- **T11 (capability/write-granularity):** add **story-item / paragraph** as a named
|
||||
write-granularity tier between whole-file and block/character.
|
||||
- **T13 (history portability / merge model):** record fedwiki's **journal-replay op-log**
|
||||
as a *third merge model* beside git 3-way and CRDT auto-merge — a **coarse semantic
|
||||
op-log applied manually via fork**. A shard whose history *is* such a journal can supply
|
||||
our coordination journal almost directly (vs git-commit import or CRDT-update import).
|
||||
- **T16 (identity ≠ placement):** fedwiki's `fork` journal entries are **provenance edges**
|
||||
between same-named pages on different sites — exactly the cross-shard "same page,
|
||||
different placement" relation we must model. Use them as edges; keep our own identity
|
||||
layer above slug.
|
||||
|
||||
## 9. Open questions
|
||||
|
||||
1. Should shard-wiki's coordination journal adopt fedwiki's **exact action vocabulary**
|
||||
(create/add/edit/move/remove/fork) at the page-item level, or a more granular/abstract
|
||||
op set that other shards can also emit?
|
||||
2. Is **neighborhood** (dynamic, link/fork-discovered) a first-class membership mode for an
|
||||
information space, or only a *view* over an explicitly-configured shard set (roster)?
|
||||
3. How do we reconcile fedwiki's **slug-as-identity + fork-DAG** with our intended
|
||||
**stable cross-shard identity** (T16) — promote fork edges into the identity graph, or
|
||||
keep them as provenance-only annotations?
|
||||
4. Does the **chorus / no-canonical** stance compose with shards that *do* assert a
|
||||
canonical (Notion, an upstream git main)? (policy-selectable canonical over a
|
||||
mechanism that permits chorus.)
|
||||
|
||||
## 10. Sources
|
||||
|
||||
- Smallest Federated Wiki wiki: **Story JSON**, **Federation Details** —
|
||||
github.com/WardCunningham/Smallest-Federated-Wiki/wiki
|
||||
- JSON Schema notes — song.fed.wiki.org/json-schema.html
|
||||
- "Smallest Federated Wiki" — home.c2.com/smallest-federated-wiki.html
|
||||
- Federated Wiki — federated.wiki (Visualizing Page History)
|
||||
- Mike Caulfield, "The OER Case for Federated Wiki" — hapgood.us (2015)
|
||||
- Jon Udell, "A federated Wikipedia" — blog.jonudell.net (2015)
|
||||
- Wikipedia: *Federated Wiki*; IndieWeb: *Smallest Federated Wiki*
|
||||
- fedwiki/wiki-plugin-transport (plugin/transport reference)
|
||||
- prior: `research/260608-federation-concepts/` §3
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
New UCs **UC-70–UC-72** carry the marker **⊞** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md` (true lineage = this dive; placed in the nearest existing column).
|
||||
Enriched: UC-26, UC-28, UC-30, UC-05, UC-27. Architecture cross-refs: SHARD-WP-0002
|
||||
T1–T5, T11, T13, T16.
|
||||
17
research/260614-forge-wikis-deep-dive/README.md
Normal file
17
research/260614-forge-wikis-deep-dive/README.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# 260614 — git-forge wikis deep dive (Gitea · GitLab · GitHub)
|
||||
|
||||
Deep dive on the **git-forge-hosted Markdown wikis** as one grouped memo: each is a
|
||||
**dedicated git repository of Markdown** exposed through a forge, attachable by **cloning
|
||||
the wiki repo directly** *or* (where offered) through the **forge's wiki API**. INTENT names
|
||||
**Gitea wikis** explicitly as a shard participant — this dive confirms the **git-native
|
||||
file-store** as a first-class, and the *simplest*, backend.
|
||||
|
||||
- `findings.md` — the three forges compared, the `.wiki.git` model, API matrix, capability
|
||||
profile, INTENT mapping, UC seeds (UC-76/77), architecture notes for SHARD-WP-0002, open
|
||||
questions (resolves the UC-68 source-of-truth question for this case), sources,
|
||||
traceability.
|
||||
|
||||
Catalog yield: UC-76 (attach a forge wiki by cloning its dedicated `.wiki.git` — git **is**
|
||||
the native store and the coordination journal), UC-77 (attach/write via the forge wiki API
|
||||
where git-clone is unavailable/undesired — capability varies by forge). Enriched
|
||||
UC-40/02/68/38. Feeds SHARD-WP-0002 T14 (attachment binding).
|
||||
173
research/260614-forge-wikis-deep-dive/findings.md
Normal file
173
research/260614-forge-wikis-deep-dive/findings.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# git-forge wikis (Gitea · GitLab · GitHub) — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T5 · **Subject:** the Markdown wikis
|
||||
hosted by the three major git forges — **Gitea**, **GitLab**, **GitHub** — treated as one
|
||||
family because they share one architecture: *a wiki is a separate git repo of Markdown.*
|
||||
|
||||
## Why this dive
|
||||
|
||||
INTENT names **Gitea wikis** as a shard participant, and the whole project is "a **Git-based
|
||||
Markdown** wiki orchestrator." The forge wikis are therefore the **least exotic, highest-
|
||||
fit** backend in the entire study: the page store is *literally a git repository of Markdown
|
||||
files*. After fourteen dives into DBs, CRDTs, graphs and SaaS, this one confirms the
|
||||
**home case** — and sharpens it by contrasting *git-IS-the-store* (forge wikis) against
|
||||
*git-is-a-mirror* (Wiki.js, UC-68).
|
||||
|
||||
## 1. The shared architecture — a wiki is a `.wiki.git` repo
|
||||
|
||||
All three forges implement a project/repo wiki as a **second, dedicated git repository**
|
||||
alongside the code repo, addressable as `<repo>.wiki.git`:
|
||||
|
||||
- `git@host:owner/project.wiki.git` (GitLab), `…/owner/repo.wiki.git` (Gitea/GitHub).
|
||||
- **Pages are Markdown files** (`Home.md`, `Some-Page.md`), one file per page; the **page
|
||||
title ↔ filename** (spaces ↔ hyphens by convention). Other markups are accepted
|
||||
(AsciiDoc, Textile, reStructuredText, Org) — GitHub/Gitea via **Gollum** (the Ruby
|
||||
git-backed wiki library), GitLab via its own renderer.
|
||||
- **History is git history** — every page edit (web or pushed) is a git commit with
|
||||
author/timestamp/message. *The wiki's revision history is a real git log.*
|
||||
- **Special pages** by convention: `_Sidebar`, `_Footer`, `_Header` (GitHub/Gitea),
|
||||
`_sidebar` (GitLab) — engine-rendered chrome stored as ordinary files.
|
||||
- **Subdirectories / nested pages**: GitLab and Gitea support directory structure; GitHub
|
||||
wikis are historically flat (Gollum supports paths but the GitHub UI is shallow).
|
||||
|
||||
The decisive property: **you can `git clone` the wiki repo, edit files, commit, and push**,
|
||||
and the forge UI reflects it — *and vice versa*. Git is **a** (often **the**) first-class
|
||||
write path. This is exactly shard-wiki's native medium with no impedance.
|
||||
|
||||
## 2. Where they differ — the API matrix
|
||||
|
||||
| | git clone/push of `.wiki.git` | wiki content **API** | nested dirs | markups |
|
||||
|--|--|--|--|--|
|
||||
| **Gitea** | ✅ yes | ✅ **REST wiki endpoints** (list/get/create/edit/delete pages) | ✅ | Markdown (+Gollum-style) |
|
||||
| **GitLab** | ✅ yes | ✅ **REST Wikis API** (project & group wikis) | ✅ | Markdown/AsciiDoc/RDoc/Org |
|
||||
| **GitHub** | ✅ yes | ❌ **no wiki REST API** — wiki is **git-only** (Gollum) | ⚠️ flat UI | Markdown + Gollum markups |
|
||||
|
||||
The key asymmetry: **GitHub exposes wiki content *only* through git** (the REST/GraphQL API
|
||||
covers issues/PRs/code but **not** wiki pages); **GitLab and Gitea offer both** a wiki API
|
||||
*and* git access. So the **git-clone path is the universal one** (works for all three); the
|
||||
API path is an *optional, capability-varying* alternative.
|
||||
|
||||
## 3. git-IS-the-store vs git-is-a-mirror (the UC-68 contrast)
|
||||
|
||||
Wiki.js (UC-68) keeps a **DB as canonical** and *maintains a git mirror* — so writing by
|
||||
commit risks **racing the engine's DB↔git sync** (catalog open-Q22). Forge wikis are the
|
||||
opposite: **the git repo IS the canonical store**; there is *no* separate DB of record for
|
||||
wiki content. Therefore:
|
||||
|
||||
- **The source-of-truth question (Q22) is resolved for this case:** the `.wiki.git` repo is
|
||||
authoritative. shard-wiki can **write by commit/push directly** with no engine to race —
|
||||
the forge merely *renders* what git holds.
|
||||
- The forge **API** (GitLab/Gitea), where present, is a *convenience over the same git
|
||||
repo*, not a competing store — so API-write and git-write converge on one history.
|
||||
|
||||
This makes forge wikis the **cleanest possible write-through file-store shard**: clone =
|
||||
projection/mirror, commit = overlay-applied/write, git log = the coordination journal *as
|
||||
is*.
|
||||
|
||||
## 4. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | Gitea / GitLab / GitHub wiki |
|
||||
|--------------------------------|------------------------------|
|
||||
| Attachment mode | **file-store (native: git clone)** + optional **external-API** (GitLab/Gitea wiki REST) |
|
||||
| Addressing granularity | **page = file**; sub-page = path (GitLab/Gitea) |
|
||||
| Content identity | path/filename within the wiki repo (title-derived) |
|
||||
| Identity vs placement | placement-bound (path = identity), like a plain git repo |
|
||||
| Structure | flat or directory tree of Markdown files; `_Sidebar`/`_Footer` chrome |
|
||||
| History | **native git history** (real commits, authors, messages) |
|
||||
| Merge model | **git** (3-way merge, branches) — though wiki repos are usually single-branch |
|
||||
| Native query | none (it's files); forge full-text search over the wiki |
|
||||
| Translation | **Markdown-native** (+ AsciiDoc/Org via renderer) — minimal/no translation needed |
|
||||
| Attachment/write granularity | **file (page)** per commit |
|
||||
| Operational envelope | ordinary git + forge; clone is cheap; API rate limits apply to API path |
|
||||
| Access grant | **forge repo permissions** (delegated auth; per-repo/role ACL) |
|
||||
| Content opacity | transparent Markdown in git |
|
||||
| Provenance | git author/committer/timestamp per commit — native |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements (this is the home case)
|
||||
|
||||
- **Git-based Markdown orchestrator** (INTENT core): forge wikis *are* git repos of
|
||||
Markdown. The **wiki page model** (Markdown-first, path-addressed, git-versioned) maps 1:1
|
||||
— minimal adapter, maximal fit.
|
||||
- **Coordination journal = git** (INTENT): the wiki repo's **git log is already the
|
||||
coordination journal** — no synthesis needed; adopt it directly.
|
||||
- **Overlay before mutation**: overlays are **branches/commits** on the cloned wiki repo;
|
||||
applying = push (or open an MR/PR where the forge supports wiki MRs — GitLab does not for
|
||||
wikis, so push-to-branch + manual is the path).
|
||||
- **Graceful degradation**: even GitHub (no wiki API) is fully usable via git-clone — the
|
||||
*universal* path means a limited forge is still a first-class read/write shard.
|
||||
- **No silent remote mutation**: writes are explicit git pushes (or explicit API calls)
|
||||
under the user's forge credentials and repo permissions.
|
||||
|
||||
### Divergences (boundaries / notes — minor)
|
||||
|
||||
- **Capability varies by forge**: GitHub = git-only (no content API); GitLab/Gitea = git +
|
||||
API. The adapter must **model the API as an optional capability**, defaulting to the
|
||||
universal git path (T11/T14). Not a bug — exactly the capability-awareness INTENT mandates.
|
||||
- **Wiki repos rarely use branches/MRs for review**: forge wikis usually edit a single
|
||||
branch directly; the rich PR-review flow is on the *code* repo, not the wiki. So
|
||||
"overlay → review → merge" needs shard-wiki to provide the review layer, not the forge.
|
||||
- **Identity = path** (like any git repo) — cross-shard identity (T16) is layered above, as
|
||||
for plain git/`wiki/` subdir shards.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **git-clone as the universal, canonical file-store attach** for forge wikis — Markdown +
|
||||
git history directly as page model + coordination journal (UC-76). The reference
|
||||
easy-case backend.
|
||||
2. **Forge wiki API as an optional capability** (GitLab/Gitea), with **git-only fallback**
|
||||
(GitHub) — capability-aware binding (UC-77).
|
||||
3. **git-IS-store ⇒ write-by-commit is safe** (no engine race) — record this as the
|
||||
resolution of the Wiki.js mirror dilemma (Q22) for forge wikis.
|
||||
|
||||
## 6. UC seeds
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-76 | Attach a **git-forge wiki** by **cloning its dedicated `.wiki.git`** — git is the native store; Markdown files = pages, git log = coordination journal; commit/push = write (no engine to race) | **new** |
|
||||
| UC-77 | Attach/write a forge wiki via the **forge's wiki API** (GitLab/Gitea REST) where git-clone is unavailable or API-write is preferred; **git-only fallback** for GitHub — capability varies by forge | **new** |
|
||||
| — | git-native file-store as the *canonical store* (not mirror) | enrich **UC-40** |
|
||||
| — | dual-path attach (git clone vs forge API) | enrich **UC-02** |
|
||||
| — | git-IS-store vs engine-maintained mirror (resolves Q22) | enrich **UC-68** |
|
||||
| — | forge as an API host for the wiki resource | enrich **UC-38** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T14 (adapter binding / attach path):** forge wikis are the canonical **file-store
|
||||
attach** — bind to the `.wiki.git` clone as the universal path; model the **wiki API as an
|
||||
optional, forge-specific capability** (present: GitLab, Gitea; absent: GitHub). One shard,
|
||||
two possible bindings converging on the same git history.
|
||||
- **T11 (capability model):** "has-content-API" is a **per-forge capability flag**; git
|
||||
clone/push is the baseline every forge satisfies. Minimal adapter profile — near the
|
||||
Oddmuse-simple end but Markdown-native and git-versioned.
|
||||
- **Coordination journal:** adopt the wiki repo's **git log directly** — the one backend
|
||||
where INTENT's git-backed journal needs *zero* synthesis.
|
||||
- **Resolves Q22 (UC-68):** because git **is** the store (not a mirror), **write-by-commit
|
||||
is safe** — no engine DB↔git sync to race. Record the distinction *engine-mirror*
|
||||
(Wiki.js: DB canonical, careful) vs *git-canonical* (forge wikis: commit freely).
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. For overlay → **review** → apply, does shard-wiki supply the review layer over a forge
|
||||
wiki (which lacks wiki-MRs), e.g. via a branch + its own diff/approve, or push directly?
|
||||
2. When a forge offers **both** git and a wiki API (GitLab/Gitea), which does the adapter
|
||||
prefer by default — git (universal, full history) with API as a fallback for hosts where
|
||||
clone is disabled? (cf. UC-43 backend-swap under stable binding.)
|
||||
3. Should the **code-repo `wiki/` subdir** shard and the **forge wiki repo** shard share one
|
||||
adapter (both git+Markdown) with a "which repo / which path" parameter, or stay distinct?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- GitLab Docs — *Wiki* (separate git repo; web/git/API; `.wiki.git`) — docs.gitlab.com
|
||||
- Gitea — wiki via git clone + repository **wiki API**; forum/issue threads on
|
||||
`.wiki.git` clone (go-gitea/gitea #1426, #15420) — gitea.com / github.com/go-gitea
|
||||
- GitHub — wiki = Gollum git repo (`<repo>.wiki.git`), no wiki REST API — docs.github.com
|
||||
- Gollum (git-based wiki library) — github.com/gollum/gollum
|
||||
- prior: `research/260614-wikijs-deep-dive/` (engine-maintained mirror contrast, UC-68)
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UCs **UC-76–UC-77** carry the marker **⎇** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-40, UC-02, UC-68, UC-38. Architecture cross-refs:
|
||||
SHARD-WP-0002 T14, T11; coordination-journal-from-git; resolves catalog open-Q22.
|
||||
33
research/260614-glamorous-toolkit-deep-dive/README.md
Normal file
33
research/260614-glamorous-toolkit-deep-dive/README.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# 260614 — Glamorous Toolkit (moldable development) deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T7**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Glamorous Toolkit** (GT, on Pharo): **moldable development** — cheap,
|
||||
custom, **domain-specific views** (`gtView` methods) so any object explains itself through an
|
||||
**open set of co-equal projections, none canonical** — plus **Lepiter**, GT's live notebook/
|
||||
knowledge base (git-versionable JSON page files with live, inspectable code results).
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Strongest prior art for **moldable, multi-view projection**: projection is not *a* view
|
||||
but an **open, type-keyed set of co-equal, possibly-computed views, none privileged** —
|
||||
refines SHARD-WP-0002 **T16** and unifies replication-/derivation-/dimensional-/query-
|
||||
projection under "many co-equal views."
|
||||
- Generalizes ZigZag dimensional views (UC-47/48) and query/computed views (UC-54) into a
|
||||
**pluggable view registry** keyed by content type (answers UC-55's open question on a
|
||||
content-type registry).
|
||||
- Reinforces **files-canonical, liveness-above, degrade-to-snapshot** (Lepiter files vs the
|
||||
Pharo image; same boundary as Jupyter UC-84 / Squeak T6).
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (design prior art, not a candidate shard — like the UseModWiki lineage dive).
|
||||
- Enrich **UC-47, UC-48, UC-54**; links **UC-55, UC-83, UC-84, UC-79**.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Moldable development & `gtView`, the Moldable Inspector, Lepiter, relation to ZigZag/query/derivation projection, INTENT mapping, UC disposition (enrichment-only), architecture notes, open questions |
|
||||
161
research/260614-glamorous-toolkit-deep-dive/findings.md
Normal file
161
research/260614-glamorous-toolkit-deep-dive/findings.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Glamorous Toolkit — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T7 · **Subject:** Glamorous Toolkit (GT) on
|
||||
Pharo — moldable development, custom views/inspectors, the live notebook (Lepiter).
|
||||
|
||||
## Why this dive
|
||||
|
||||
T1/T3 gave us **derivation-projection** (one source → rendered/computed forms). GT comes at
|
||||
projection from the *other* side: **many co-equal, domain-specific views over the same live
|
||||
content**, where the *environment molds itself to the knowledge* rather than forcing the
|
||||
knowledge into one fixed rendering. This is the strongest prior art for **moldable,
|
||||
multi-view projection** and a striking parallel to ZigZag's dimensional model (UC-47/48) and
|
||||
to query/computed views (UC-54). It is *design* prior art — GT is not a candidate shard —
|
||||
so the yield is **enrichment + projection design notes**, not a new shard UC.
|
||||
|
||||
## 1. Moldable development
|
||||
|
||||
GT's thesis: **systems should be explainable through custom tools that are cheap to build.**
|
||||
Instead of a single generic object inspector/renderer, a developer adds **small,
|
||||
domain-specific views** to a class so any instance explains itself in the terms that matter:
|
||||
|
||||
- **`gtView`-annotated methods** — a class declares extra inspector views by writing methods
|
||||
tagged `<gtView>` that return a view (tree, list, table, chart, source, raw, a custom
|
||||
diagram…). Each is a **co-equal projection of the same object**; none is privileged.
|
||||
- The **Moldable Inspector** shows these views as **switchable tabs** over one object, and
|
||||
lets you **dive** into sub-objects (each with its own custom views) — navigation *is*
|
||||
moving across projections.
|
||||
- Views are **cheap and local**: a view is just a method, versioned with the code, added
|
||||
incrementally as understanding grows. The environment **adapts to the domain**.
|
||||
|
||||
The key abstraction for us: **the same underlying content carries an open, extensible set of
|
||||
views, selected at inspection time, none canonical.**
|
||||
|
||||
## 2. Lepiter — the live notebook / knowledge base
|
||||
|
||||
GT ships **Lepiter**, a notebook/knowledge base where pages mix prose, **live code
|
||||
snippets** (evaluated in-image, results inspectable with the moldable views above), and
|
||||
links. Notebook pages are stored as **JSON "database" files** on disk (a Lepiter DB =
|
||||
directory of page files), so the knowledge base is **file-backed and git-versionable** while
|
||||
remaining live in the image.
|
||||
|
||||
This is the literate/notebook pattern (T1/T3) fused with moldable views: a snippet's result
|
||||
is not a static captured output but a **live object you can open into any of its views** —
|
||||
the *anti-snapshot*. (Boundary: that liveness is exactly what shard-wiki must degrade to a
|
||||
snapshot when the image isn't present — see §4.)
|
||||
|
||||
## 3. Relationship to ZigZag, query-views, and the projection model
|
||||
|
||||
- **vs ZigZag (UC-47/48):** ZigZag gives **dimensional** views — the *same cells* seen along
|
||||
different orthogonal axes. GT gives **moldable** views — the *same object* seen through
|
||||
different *purpose-built lenses*. Both reject a single privileged rendering; both make
|
||||
**multi-view, none-canonical** the norm. GT generalizes the idea from fixed dimensions to
|
||||
an **open, code-defined view set**.
|
||||
- **vs query/computed views (UC-54):** a `gtView` is a **computed projection** (it runs code
|
||||
to build the view) — like a query-defined page, but keyed to a **content type / domain**
|
||||
rather than a stored query. Strengthens "a view can be computed, not stored."
|
||||
- **vs derivation-projection (T1/UC-83):** GT confirms projections are **plural and
|
||||
co-equal**; UC-83/UC-84 had *few* well-known derivations (docs/code/outputs), GT has an
|
||||
**open registry** of them keyed by type.
|
||||
|
||||
So GT's contribution to the contract is: **projection is not one view but an open set of
|
||||
co-equal, type-keyed, possibly-computed views, none canonical** — a *moldable projection
|
||||
registry*. That is a refinement of T16's projection model, not a new shard.
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Union without erasure / no privileged rendering.** GT's "many co-equal views, none
|
||||
canonical" is the same ethos as showing provenance/freshness without hiding any: a page's
|
||||
presentation is **plural**, and shard-wiki should be able to offer **multiple co-equal
|
||||
projections of the same content** (raw, rendered, structured, domain-specific) rather than
|
||||
one flattened view.
|
||||
- **Computed views (UC-54) keyed by content type.** A moldable view = a **computed
|
||||
projection registered against a content type** — directly supports a *pluggable view/
|
||||
projection registry* in the contract (the answer shape for UC-55's "pluggable content-type
|
||||
registry" open question).
|
||||
- **Files-canonical, live-on-top.** Lepiter stores pages as **git-versionable JSON files**
|
||||
while being live in the image — reinforcing "the durable artifact is files; liveness is a
|
||||
layer above," consistent with shard-wiki's git-canonical stance.
|
||||
- **Mechanism over policy.** Which views to show, and whether to compute or snapshot them,
|
||||
stay configurable; GT provides the *mechanism* (open view set), not a fixed presentation.
|
||||
|
||||
### Divergences / boundaries
|
||||
|
||||
- **The image is not a store (shared boundary with T6 Squeak).** GT's *liveness* lives in a
|
||||
Pharo image; shard-wiki must not treat the image as a shard. Attach the **Lepiter DB files**
|
||||
(git-versionable) as the durable content; treat live/computed views as **derivation-
|
||||
projections that degrade to captured snapshots** when no image/kernel is present (same rule
|
||||
as Jupyter UC-84).
|
||||
- **Not a view engine.** shard-wiki *models* "this content type has these co-equal views and
|
||||
one may be canonical-for-display"; it does not implement GT's rendering. Domain-specific
|
||||
view code stays with the source/adapter, surfaced as a capability.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Moldable projection = an open, type-keyed set of co-equal, possibly-computed views,
|
||||
none canonical** — refine T16's projection model and the UC-47/48/54 cluster toward a
|
||||
**pluggable view/projection registry**.
|
||||
2. **Lepiter** = literate/notebook (T1/T3) with **live results** stored as git-versionable
|
||||
files — another "files-canonical, liveness-above, degrade-to-snapshot" instance.
|
||||
3. **Navigation = moving across projections** (dive into sub-objects) — a navigation idea
|
||||
beside dimensional movement (UC-47/48) for the derived-views thread.
|
||||
|
||||
## 5. UC disposition (enrichment-only — no new shard UC)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC |
|
||||
|------------------------|------------|
|
||||
| Many co-equal, domain-specific views over the same content; none canonical (§1, §3) | UC-47 / UC-48 (enriched) |
|
||||
| `gtView` = a **computed projection registered against a content type** (§1, §3) | UC-54 (enriched) |
|
||||
| Open, pluggable view set keyed by type = the shape of a content-type/view registry (§3) | links UC-55 (open-Q #10) |
|
||||
| Lepiter live-snippet results = live objects → degrade to snapshot when no image (§2, §4) | links UC-84, UC-83 |
|
||||
| Lepiter pages = git-versionable JSON files; image is not the store (§2, §4) | links UC-79 (files-canonical) |
|
||||
| Dive-into-sub-object navigation = moving across projections (§1) | links UC-17–UC-20 (derived views) |
|
||||
|
||||
GT is **design prior art, not a candidate shard**, so it yields **no new UC** (like the
|
||||
UseModWiki lineage dive). Its value is sharpening the **projection model** for SHARD-WP-0002
|
||||
T16: projection is an **open set of co-equal, type-keyed, possibly-computed views, none
|
||||
privileged** — a moldable view registry.
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T16 (projection):** generalize projection from "a view" to a **moldable view registry** —
|
||||
an open, extensible set of **co-equal, type-keyed, possibly-computed** projections of the
|
||||
same content, none canonical (display-canonical is a policy choice, not a fact). This
|
||||
unifies replication-projection (UC-53/57), derivation-projection (UC-83/84), dimensional
|
||||
views (UC-47/48), and query/computed views (UC-54) under one "many co-equal views" model.
|
||||
- **T12 (page model):** a content type may **carry/declare its own views** (a capability);
|
||||
the page model should allow attaching view definitions to a type (the registry's entries),
|
||||
answering UC-55's "pluggable content-type registry" question.
|
||||
- **Boundary (T14):** attach the **Lepiter DB files** (durable, git-versionable), never the
|
||||
Pharo image; live/computed views are derivation-projections degrading to snapshots — same
|
||||
rule as Jupyter (no kernel/image host).
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. Does shard-wiki expose a **view/projection registry** as a first-class public concept
|
||||
(content type → its co-equal views), or keep "moldable" as an adapter-internal idea?
|
||||
2. When multiple co-equal views exist, is **"canonical-for-display"** a per-shard policy, a
|
||||
user preference, or unset (always show the chooser)? (Mechanism-over-policy says
|
||||
configurable.)
|
||||
3. How does a **computed view** (UC-54/`gtView`) declare its **freshness/provenance** so the
|
||||
union doesn't present a stale computed projection as current? (Ties UC-84's snapshot
|
||||
marking.)
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- Glamorous Toolkit docs (`gtoolkit.com`): moldable development, the Moldable Inspector,
|
||||
`gtView` methods, Lepiter knowledge base; Feenk essays on moldable development.
|
||||
- Pharo (substrate — see T8 dive): live image, reflective environment.
|
||||
- prior: `research/260614-zigzag-deep-dive/` (dimensional multi-view UC-47/48);
|
||||
`research/260614-jupyter-deep-dive/` (live vs snapshot, UC-84);
|
||||
`research/260614-literate-programming-deep-dive/` (derivation-projection, UC-83).
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
**No new UC** (GT is design prior art, not a candidate shard). Enriched: UC-47, UC-48,
|
||||
UC-54; links UC-55 (content-type/view registry), UC-83/UC-84 (live→snapshot), UC-79
|
||||
(files-canonical), UC-17–UC-20 (derived-view navigation). Architecture cross-refs:
|
||||
SHARD-WP-0002 T16 (moldable view registry: open set of co-equal type-keyed computed views),
|
||||
T12 (content type declares its views), T14 (attach Lepiter files, not the image).
|
||||
15
research/260614-ikiwiki-deep-dive/README.md
Normal file
15
research/260614-ikiwiki-deep-dive/README.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# 260614 — ikiwiki deep dive
|
||||
|
||||
Deep dive on **ikiwiki** (Joey Hess): a **wiki compiler** — it compiles a **VCS-backed**
|
||||
(usually **git**) tree of Markdown source into **static HTML**, supports **distributed**
|
||||
operation (clone/edit/push between wiki instances, change **pings**), and treats web edits as
|
||||
commits to the same repo.
|
||||
|
||||
- `findings.md` — the compiler model, git-distributed federation + pinger, static output,
|
||||
capability profile, INTENT mapping, UC seed (UC-79), architecture notes for SHARD-WP-0002,
|
||||
open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-79 (attach a git-backed **compile-to-static** wiki — git Markdown source
|
||||
is the shard, compiled static HTML is a derived publish/projection; participate in
|
||||
git-distributed clone federation with change-pings). Enriched UC-31/56/37/33. Feeds
|
||||
SHARD-WP-0002 T4 (federation), T6 (publish/projection).
|
||||
153
research/260614-ikiwiki-deep-dive/findings.md
Normal file
153
research/260614-ikiwiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# ikiwiki — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T4 · **Subject:** ikiwiki, Joey Hess's
|
||||
VCS-backed wiki compiler.
|
||||
|
||||
## Why this dive
|
||||
|
||||
The forge-wiki dive (T5) established *git-IS-the-store* for hosted Markdown wikis. ikiwiki
|
||||
takes the same git-canonical source but adds two ideas shard-wiki cares about directly:
|
||||
**compile-to-static** (the wiki is *built*, not served from a DB) and **git-distributed
|
||||
federation** (wiki instances clone, pull, push, and **ping** each other). It is referenced
|
||||
in `research/260608-federation-concepts/`; here we go into the model.
|
||||
|
||||
## 1. The wiki-compiler model
|
||||
|
||||
ikiwiki is fundamentally a **compiler**: input is a directory of **source pages** (Markdown
|
||||
by default; also other formats) held in a **version-control repo**; output is a tree of
|
||||
**static HTML**. A rebuild is triggered by a **VCS post-commit/post-update hook**, so:
|
||||
|
||||
- **The VCS repo is canonical**; the HTML is *derived build output* (regenerable, disposable).
|
||||
- **Web edits are commits.** The CGI edit interface writes the change *into the repo* (a
|
||||
commit) and triggers a rebuild — so browser edits and `git push` edits converge on one
|
||||
history. (Same convergence as forge wikis, but here the canonical store is *your* git repo,
|
||||
not a forge's.)
|
||||
- **VCS-agnostic**: git is usual, but svn/bzr/mercurial/darcs are supported via a VCS
|
||||
plugin layer — an early "pluggable backend behind a stable interface" (adapter-contract
|
||||
echo).
|
||||
- **Plugins** (Perl) provide directives, feeds, auth (`openid`), and the federation hooks
|
||||
below.
|
||||
|
||||
## 2. Git-distributed federation + the pinger
|
||||
|
||||
Because the source is an ordinary VCS repo, ikiwiki instances federate the way *git* does:
|
||||
|
||||
- **Clone-and-diverge**: you can `git clone` a wiki, edit offline, and `push`/`pull` between
|
||||
instances — **multiple wiki clones that reconcile via git merge**. A wiki is a branch-space
|
||||
(UC-33).
|
||||
- **`pinger` / `pingee` plugins**: an instance can send an **XML-RPC ping** to another
|
||||
ikiwiki when it changes, prompting the other to **pull and rebuild** — a lightweight
|
||||
**subscribe/notify** primitive over the git-distributed mesh (UC-31).
|
||||
- **`aggregate` plugin**: pulls external **RSS/Atom feeds** into the wiki as pages — an
|
||||
inbound projection of remote content.
|
||||
|
||||
So ikiwiki is *federation by git plus a ping* — distinct from fedwiki's fork/journal
|
||||
(UC-72) and from Wikibase's query-time `SERVICE` (UC-74): a **third federation flavor**,
|
||||
*VCS-replication federation with change notification*.
|
||||
|
||||
## 3. Static output as a publish/projection target
|
||||
|
||||
The compiled static HTML is a **read-only, regenerable projection** of the source:
|
||||
|
||||
- It is a natural **outbound publish target** (UC-56): render the union (or a shard) to a
|
||||
static site for hosting/backup, no server needed.
|
||||
- It is also the **read-only backup** end (UC-37): a static snapshot that survives the engine.
|
||||
|
||||
The key shard-wiki framing: **source (git Markdown) is the attachable shard; static HTML is a
|
||||
derived projection** — never confuse the build output for the canonical content.
|
||||
|
||||
## 4. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | ikiwiki |
|
||||
|--------------------------------|---------|
|
||||
| Attachment mode | **file-store** (VCS-backed git Markdown source) |
|
||||
| Addressing granularity | page = source file; path = identity |
|
||||
| Content identity | path/filename (placement-bound) |
|
||||
| Structure | directory tree of Markdown source + directives |
|
||||
| History | **native VCS (git) history** |
|
||||
| Merge model | **git** (clone/pull/push/merge across instances) |
|
||||
| Native query | none; directives + plugins compute derived pages at build |
|
||||
| Translation | Markdown source → static HTML (build-time render) |
|
||||
| Write granularity | **file (page)** per commit |
|
||||
| Operational envelope | a compiler + VCS hook; static hosting for output |
|
||||
| Access grant | VCS/file perms; `openid` for web edits |
|
||||
| Content opacity | transparent Markdown |
|
||||
| Provenance | git author/timestamp; aggregated feeds carry source |
|
||||
| Federation | **git replication + XML-RPC ping**; RSS aggregation |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Git-canonical Markdown source** = the home case (shared with forge wikis UC-76/40):
|
||||
attach the source repo, adopt its git log as the journal, write by commit.
|
||||
- **Coordination layer is git** (INTENT): ikiwiki's whole federation *is* git replication +
|
||||
a ping — the most literal realization of "Git-addressable coordination layer."
|
||||
- **Projection vs canonical**: compile-to-static cleanly separates **canonical source** from
|
||||
**derived output** — exactly shard-wiki's projection principle (static HTML = a lazy/cache
|
||||
projection that is regenerable, never the source of truth).
|
||||
- **Graceful degradation / publish**: static output is the trivial read-only backup and
|
||||
outbound publish target (UC-37/UC-56).
|
||||
- **Subscribe/notify mechanism, not policy**: the pinger is a *mechanism* (notify a peer to
|
||||
pull); *which* peers, when, and conflict policy stay configurable.
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- ikiwiki is mostly a **reinforcement** of git-canonical-Markdown (UC-76) — its *new*
|
||||
contributions are (a) **compile-to-static** as a distinct projection/publish direction and
|
||||
(b) the **git-distributed-clone + ping** federation flavor. The static output is *not* a
|
||||
shard to attach (it's derived); attach the **source repo**.
|
||||
- VCS-agnostic backend layer is an interesting adapter-contract echo, but for shard-wiki the
|
||||
git case dominates.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Source-repo-as-shard, static-output-as-derived-projection** — never attach the build
|
||||
output as canonical (UC-79; relates UC-37/UC-56).
|
||||
2. **Git-replication + change-ping** as a named federation flavor beside fork/journal
|
||||
(fedwiki) and query-SERVICE (Wikibase) — a **subscribe/notify over a git mesh** (UC-31/UC-33).
|
||||
3. **Inbound feed aggregation** (RSS/Atom → pages) as an inbound projection pattern.
|
||||
|
||||
## 6. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-79 | Attach a git-backed **compile-to-static** wiki — the **git Markdown source is the shard**, compiled **static HTML is a derived publish/projection**; participate in **git-distributed clone federation** with change-**pings** | **new** |
|
||||
| — | XML-RPC pinger = subscribe/notify over a git mesh | enrich **UC-31** |
|
||||
| — | render union/shard to a static site (publish) | enrich **UC-56** |
|
||||
| — | static HTML as read-only regenerable backup | enrich **UC-37** |
|
||||
| — | wiki as a git branch-space; clones reconcile via merge | enrich **UC-33** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T4 (federation):** record **git-replication + change-ping** as a federation flavor — peers
|
||||
hold git clones, reconcile via merge, and notify with a ping to pull/rebuild. Distinct from
|
||||
fedwiki fork/journal and Wikibase `SERVICE`; complements the "Git-addressable coordination
|
||||
layer" mandate most literally.
|
||||
- **T6 (publish/projection):** **compile-to-static** is the canonical *outbound* projection —
|
||||
source (git) is canonical, static HTML is a regenerable derived view (publish/backup
|
||||
target). Reinforces projection-vs-canonical separation.
|
||||
- **Attach binding (T14):** attach the **source repo** (git Markdown), not the build output;
|
||||
shares the forge-wiki / `wiki/`-subdir git+Markdown adapter (parameterized by repo/path).
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Is the **pinger** (notify-peer-to-pull) modeled as shard-wiki's own subscribe/notify
|
||||
primitive, or only recognized when bridging two ikiwiki instances?
|
||||
2. Does shard-wiki ever *drive* a compile-to-static **publish** of the union (act as the
|
||||
ikiwiki-like compiler), or only attach existing ikiwiki source repos? (UC-56 scope.)
|
||||
3. Is **feed aggregation** (RSS→pages) an inbound projection mode shard-wiki offers generally
|
||||
(a feed-shard), or an ikiwiki-internal feature?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- ikiwiki.info — *ikiwiki* overview, *rcs* (VCS backends), *plugins* (`pinger`/`pingee`,
|
||||
`aggregate`, `openid`), *setup* / post-commit hooks
|
||||
- prior: `research/260608-federation-concepts/` (ikiwiki reference);
|
||||
`research/260614-forge-wikis-deep-dive/` (git-canonical Markdown contrast)
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UC **UC-79** carries the marker **⊟** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-31, UC-56, UC-37, UC-33. Architecture cross-refs:
|
||||
SHARD-WP-0002 T4 (git-replication+ping federation), T6 (compile-to-static publish), T14.
|
||||
49
research/260614-joplin-deep-dive/README.md
Normal file
49
research/260614-joplin-deep-dive/README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# 260614 — Joplin deep dive (SQLite-local, Markdown-on-sync, interchange-format attach)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Joplin** — the open-source Markdown note app — read through
|
||||
shard-wiki's lens. Joplin looks like "open-source Obsidian" but is a genuinely different
|
||||
point in the space: **content is Markdown, the local store is SQLite, and the *sync
|
||||
representation* is a folder of per-item Markdown+metadata files on a backend you choose**
|
||||
(filesystem, WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). It
|
||||
also ships **end-to-end encryption** and is itself a **sync layer over heterogeneous
|
||||
storage**.
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — SQLite local store (documented schema); Markdown content;
|
||||
notebooks → notes → resources + tags + to-dos; 32-char item IDs; `:/<id>` note links
|
||||
(rename-stable, page-level)
|
||||
- **Sync** — serialized to plain-text items on a chosen target; tombstones; **conflict
|
||||
notes** (keep-both) → enables the **"attach the sync mirror, not the app"** pattern on
|
||||
WebDAV/Nextcloud/S3
|
||||
- **E2EE** — items encrypted before leaving the device → **content opacity** (ciphertext
|
||||
at rest)
|
||||
- **Extension surfaces** — a **plugin API** (`joplin.data`/`workspace`/`contentScripts`/
|
||||
views) *and* a **local Data API** (REST on `localhost:41184`, token) used by the Web
|
||||
Clipper; plus JEX/Markdown/RAW export
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, sync/interchange attach, E2EE/content-opacity, plugin + Data API surfaces, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-60 attach a tool's documented sync/interchange representation on third-party storage
|
||||
without the app; UC-61 attach an encrypted-at-rest shard with content opacity);
|
||||
UC-31/36/38/40/51/55 enriched. Logged for `SHARD-WP-0002` (T11, T14): the
|
||||
interchange/sync-representation attach surface, **content opacity** as a proposed twelfth
|
||||
capability spectrum, a local-REST attach sub-mode, and format-aware file-store profiles.
|
||||
|
||||
**Key takeaways recorded:** the best attach surface is often a tool's **interchange/sync
|
||||
representation** (offline, app-independent — Joplin items on Nextcloud/WebDAV/S3, the
|
||||
INTENT-named backends), not its live store or API; **encryption-at-rest** demands a
|
||||
content-opacity capability so encrypted shards degrade to backup/structure-shell; and
|
||||
Joplin is the **file-sync-daemon boundary case** — shard-wiki attaches its mirror as
|
||||
pages and never re-syncs (not-a-sync-daemon).
|
||||
</content>
|
||||
269
research/260614-joplin-deep-dive/findings.md
Normal file
269
research/260614-joplin-deep-dive/findings.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# Findings — Joplin: SQLite locally, Markdown-on-sync, and the interchange-format attach
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — an open-source Markdown note app; a *candidate
|
||||
shard* whose distinctive trait is a **documented file-based sync/interchange format on
|
||||
third-party storage** plus optional **end-to-end encryption**
|
||||
Lens: shard-wiki — attachment surfaces, the "attach the sync mirror not the app"
|
||||
pattern, content opacity (encryption), note-level stable IDs, and the file-sync-daemon
|
||||
boundary
|
||||
|
||||
> Why Joplin earns a dive after Obsidian/Roam/Notion. It looks like "Obsidian but
|
||||
> open-source," yet its architecture is a genuinely different point in the space:
|
||||
> **content is Markdown but the local store is SQLite, and the *sync representation* is
|
||||
> a folder of per-item Markdown+metadata files on a backend you choose** (filesystem,
|
||||
> WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). That sync
|
||||
> mirror — not the app, not the SQLite DB — is the interesting attach surface, and it
|
||||
> lands squarely on the **WebDAV/Nextcloud backends INTENT already names**. Joplin also
|
||||
> ships **E2EE**, forcing the question of a shard whose content is *opaque ciphertext*,
|
||||
> and it is itself a **sync layer over heterogeneous storage** — the very "file-sync
|
||||
> daemon" shard-wiki says it is *not*, making Joplin a sharp boundary case.
|
||||
|
||||
Contrast set: Obsidian (file-over-app: native store *is* files), Roam (client DB,
|
||||
in-app API only), Notion (hosted DB, external API only), TWiki/Foswiki (file+RCS native
|
||||
store). Joplin is none of these exactly — **DB-local, files-on-sync** — which is the
|
||||
whole point.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture — SQLite local, Markdown content
|
||||
|
||||
- **Local store:** a **SQLite database** (`~/.config/joplin-desktop/database.sqlite`),
|
||||
schema documented. So, unlike Obsidian, the *native* local store is **not** plain
|
||||
files — you cannot point shard-wiki at a folder of `.md` and read the live store
|
||||
(you'd read SQLite, fragile while the app runs).
|
||||
- **Content:** notes are **Markdown** (Joplin-flavored: `:/imageId` resource links,
|
||||
internal `:/<id>` note links, checkboxes/to-dos). Markdown-first — INTENT-aligned at
|
||||
the *content* layer even though the *storage* layer is a DB.
|
||||
- **Model:** `folders` (notebooks, hierarchical) → `notes` → `resources` (attachments);
|
||||
cross-cutting `tags`; notes can be **to-dos**. Every item has a **32-char hex ID**;
|
||||
notes link by ID (`:/<id>`), so links survive rename/move (store-minted, page-level).
|
||||
- **Clients:** desktop (Electron), mobile, and a **terminal/CLI** app — multiple clients
|
||||
edit one logical store, reconciled by sync (§2).
|
||||
- **Revisions:** Joplin keeps **internal note history** locally — not git, not portable
|
||||
→ a *supplement* case (UC-36), like Notion/Confluence.
|
||||
|
||||
---
|
||||
|
||||
## 2. Sync — the distinctive layer (and the attach opportunity)
|
||||
|
||||
Joplin's headline feature is **sync to a backend of your choice**. On sync, the SQLite
|
||||
store is serialized to a **folder of plain-text items** — one file per note / notebook /
|
||||
tag / resource-metadata, each carrying the **Markdown body plus a metadata block**
|
||||
(`id`, `parent_id`, `type_`, `created_time`, `updated_time`, …) — written to the chosen
|
||||
target: **filesystem, WebDAV, Nextcloud, Dropbox, OneDrive, S3, Joplin Server, Joplin
|
||||
Cloud**. Deletions propagate via tombstones; concurrent edits across clients produce
|
||||
**conflict notes** (keep-both, not silent overwrite).
|
||||
|
||||
Two consequences for shard-wiki:
|
||||
|
||||
1. **Attach the sync mirror, not the app (UC-60).** A Nextcloud/WebDAV/S3/filesystem
|
||||
directory full of Joplin sync items is a **file-store shard** — readable *without
|
||||
Joplin running and without touching SQLite* — provided the adapter understands the
|
||||
**documented item format** (body + metadata footer, `:/id` links, resource items).
|
||||
This is a new attach pattern: the backend's **interchange/sync representation** as
|
||||
the attach surface, distinct from a native on-disk store (UC-40) or app files (UC-53).
|
||||
It also realizes INTENT's WebDAV/Nextcloud participants concretely.
|
||||
2. **Joplin is a file-sync layer over heterogeneous storage** — exactly what shard-wiki
|
||||
says it is *not* (INTENT: "not a file-sync daemon"). Joplin syncs **one** logical
|
||||
store across backends; shard-wiki **federates many** logical stores with wiki-page
|
||||
semantics. Attaching a Joplin sync mirror means **reading its file representation as
|
||||
pages**, never re-driving Joplin's sync — and never becoming a second sync engine
|
||||
over the same target (don't double-sync).
|
||||
|
||||
---
|
||||
|
||||
## 3. End-to-end encryption — content opacity
|
||||
|
||||
Joplin offers **E2EE**: items are encrypted **before** leaving the device, so the sync
|
||||
target holds **ciphertext** regardless of provider. (Security wart, noted: the master
|
||||
key is stored *in clear* in the local SQLite — irrelevant to attaching the mirror, but
|
||||
a reason never to treat the local DB as trusted.)
|
||||
|
||||
For shard-wiki this introduces a capability dimension the prior dives did not:
|
||||
**content opacity**. An encrypted Joplin sync target can be attached only as a
|
||||
**backup/mirror/structure-shell** participant — item IDs, counts, and change events may
|
||||
be visible, but **bodies are undecryptable without keys**. The adapter must **never
|
||||
present ciphertext as readable content** and must degrade to "present-but-opaque" with
|
||||
provenance. This extends the synthesis capability spectra with a proposed twelfth field,
|
||||
**content opacity** (`plaintext → encrypted-at-rest/opaque`), feeding `SHARD-WP-0002`
|
||||
T11 (§9).
|
||||
|
||||
---
|
||||
|
||||
## 4. Extension surfaces — a plugin host *and* a local REST API
|
||||
|
||||
Joplin exposes **two** programmatic surfaces (plus export):
|
||||
|
||||
**A. Plugin API (in-app host).** TypeScript plugins, distributed via **npm + a plugin
|
||||
repository**, loaded by the app:
|
||||
- `joplin.data` — CRUD over the data model, shaped like REST (GET/POST/PUT/DELETE) on
|
||||
`notes`/`folders`/`tags`/`resources`.
|
||||
- `joplin.workspace` — selection + events (note selected, content changed, sync events).
|
||||
- `joplin.contentScripts` — **markdown-it** render plugins and **CodeMirror** editor
|
||||
extensions (the syntax/rendering extension point).
|
||||
- `joplin.views` — panels, dialogs, toolbar/menu items, custom editors;
|
||||
`joplin.commands`, `joplin.settings`.
|
||||
|
||||
**B. Data API (local REST).** A REST service the desktop app serves on
|
||||
**`localhost:41184`** (the "clipper server"), **token-authenticated**, used by the Web
|
||||
Clipper and any local integration: endpoints `/notes`, `/folders`, `/tags`,
|
||||
`/resources` (incl. `/resources/:id/file`), `/revisions`, `/search`; verbs
|
||||
GET/POST/PUT/DELETE; pagination (`page`, `limit`≤100, `order_by`, `order_dir`). Notably
|
||||
**plugins can use this API even when the clipper server is off**.
|
||||
|
||||
**C. Export** — **JEX** (tar of raw items), **Markdown directory**, **RAW** (Joplin
|
||||
export directory), **ENEX** import (Evernote). Snapshot import/attach surface (UC-28/37).
|
||||
|
||||
So Joplin spans attachment modes: **in-app plugin host** (like Roam/Obsidian-plugin),
|
||||
**local-REST** (like Notion's external API but *localhost + app-must-run* — a sub-mode),
|
||||
the **sync-mirror file-store** (§2, the novel one), and **export snapshots**.
|
||||
|
||||
---
|
||||
|
||||
## 5. Joplin as a shard — capability profile
|
||||
|
||||
| Capability | Joplin | Notes for the adapter contract |
|
||||
|------------|--------|--------------------------------|
|
||||
| Read | **yes** | sync-mirror files (no app) · local Data API (app running) · plugin · export |
|
||||
| Write | **yes** | Data API / plugin (`joplin.data`); writing the sync mirror directly is risky (Joplin owns that format) |
|
||||
| Write granularity | **per-note (page)** | notebooks/notes/resources; not block-level |
|
||||
| Identity / addressing | **32-char item ID; `:/<id>` links** | store-minted, **page-level**, survives rename (between Obsidian path and Roam block-UUID) (UC-51) |
|
||||
| Structure | notebooks (hierarchy) + tags + to-dos | modest; metadata in the sync item footer (in-text on the mirror) |
|
||||
| History | **internal revisions, not portable** | supplement via coordination journal (UC-36) |
|
||||
| Native query | search API (FTS) | delegate text search; no datalog/DB-query (weaker than Roam/Notion) |
|
||||
| Subscribe | sync is **poll-based**; conflict notes | poll; conflicts surface as keep-both items (UC-31, UC-07) |
|
||||
| Content opacity | **optional E2EE → ciphertext** | **new dimension**: opaque-at-rest; mirror usable only as backup/structure without keys (UC-61) |
|
||||
| Transclusion | none (note-level links only) | — |
|
||||
| Diff / merge | none native | git-level if mirror is on a git-backed target |
|
||||
| Publish | via export / static-site pipelines | outbound (UC-56) |
|
||||
| Content types | Markdown + **resources** (attachments) | non-Markdown assets with IDs (UC-55) |
|
||||
| Attach modes | sync-mirror file-store · local-REST · plugin · export | multi-mode, per-binding (UC-60, UC-57, UC-38) |
|
||||
|
||||
Verdict: a solid **Markdown-first candidate shard** whose best attach surface is the
|
||||
**sync mirror on a file/WebDAV/S3 target** (offline, app-independent, INTENT-named),
|
||||
with the local Data API or a plugin for live write-through. Standout: the
|
||||
interchange-format attach (UC-60) and the encryption/opacity case (UC-61). Limits: DB
|
||||
local store, note-level (not block) addressing, internal-only history, weak query.
|
||||
|
||||
---
|
||||
|
||||
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 6.1 Reinforcements
|
||||
|
||||
- **Interchange-format attach** validates that a shard's *documented sync/export
|
||||
representation* — not just its live store or API — is a legitimate, often *best*,
|
||||
attach surface (offline, app-independent). Generalizes beyond Joplin (UC-60).
|
||||
- **WebDAV/Nextcloud/S3 become concrete** participants via Joplin's sync targets — the
|
||||
backends INTENT names, now with a real format to parse.
|
||||
- **Conflict-as-data** (Joplin conflict notes = keep-both) is union-without-erasure in
|
||||
the wild (reinforces UC-07, divergence; consensus policy T9).
|
||||
- **Store-minted, rename-stable note IDs** (`:/id`) confirm the addressing spectrum's
|
||||
middle (page-level stable ID) between path (Obsidian) and block-UUID (Roam/Notion)
|
||||
(UC-51).
|
||||
|
||||
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Joplin is a sync daemon; shard-wiki is not.** Attach the **mirror as pages**; never
|
||||
re-implement Joplin's sync or run a competing sync over the same target. (INTENT
|
||||
not-a-file-sync-daemon.)
|
||||
2. **DB-local; don't read SQLite live.** The native store is a DB; treat it as opaque.
|
||||
Use the sync mirror, the Data API, or a plugin — not the live `database.sqlite`.
|
||||
3. **Joplin owns the sync format; treat the mirror read-mostly.** Writing items into the
|
||||
sync folder behind Joplin's back risks corruption/conflict; prefer overlay/projection
|
||||
or write through the Data API/plugin (overlay before mutation, no silent mutation).
|
||||
4. **Encrypted shards are opaque.** Never present ciphertext as content; degrade to
|
||||
backup/structure with provenance (UC-61, graceful degradation).
|
||||
|
||||
### 6.3 What Joplin teaches that shard-wiki should keep
|
||||
|
||||
- Add **"interchange/sync representation"** as a recognized attach surface in the
|
||||
contract — sometimes preferable to the live store/API (UC-60).
|
||||
- Add **content opacity** (encryption-at-rest) as a capability-profile field so encrypted
|
||||
shards degrade correctly (UC-61).
|
||||
- Expect **multi-client concurrent editing reconciled by the backend's own sync**
|
||||
(desktop/mobile/CLI) — a different concurrency than single-app file edits (extends
|
||||
UC-53) or multi-user server writes; conflicts may pre-exist as keep-both items.
|
||||
|
||||
---
|
||||
|
||||
## 7. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-59**. New UCs **UC-60, UC-61** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a tool's documented sync/interchange representation on a third-party storage target** (Joplin items on Nextcloud/WebDAV/S3) as a file-store shard, without the app | **UC-60 (new)** |
|
||||
| **Attach an encrypted-at-rest shard** (E2EE sync target): content opaque without keys; participate as backup/mirror/structure-shell, never presenting ciphertext as readable | **UC-61 (new)** |
|
||||
| Store-minted, rename-stable **note-level** IDs (`:/id`) — the middle of the addressing spectrum | **enriches UC-51** |
|
||||
| Native store can be a **DB while the sync representation is files** → attach surface ≠ native store | **enriches UC-40** |
|
||||
| Internal revisions, not portable → supplement | **enriches UC-36** |
|
||||
| Dual surfaces: in-app plugin host + **local Data API** (localhost, app-running) | **enriches UC-38** (links UC-57) |
|
||||
| Resources/attachments = non-Markdown assets with IDs | **enriches UC-55** |
|
||||
| Poll-based sync; conflict notes (keep-both) | **enriches UC-31** (links UC-07) |
|
||||
|
||||
---
|
||||
|
||||
## 8. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Attachment-mode taxonomy gains a refinement:** within file-store, distinguish
|
||||
*native on-disk store* (Obsidian/TWiki, UC-40) from *interchange/sync representation*
|
||||
(Joplin mirror, UC-60). Both are "files," but format ownership and write-safety differ.
|
||||
(T14.)
|
||||
- **Add a "content opacity" capability field** (proposed **twelfth spectrum**:
|
||||
`plaintext → encrypted-at-rest/opaque`) so encrypted shards degrade to
|
||||
backup/structure-shell. Feeds T11 the next time the eleven spectra
|
||||
(`research/260614-shard-spectrum-synthesis/findings.md` §2) are revised. (T11/T14.)
|
||||
- **Local-REST as an attach sub-mode** (localhost, app-must-run, token) sits between
|
||||
in-engine-host (UC-38) and external-API (UC-57) — note in T14.
|
||||
- **Format-aware file-store adapters:** the contract should let a file-store adapter
|
||||
declare a *format profile* (plain Markdown vs Joplin-item vs Foswiki-PlainFile) so the
|
||||
same "folder of files" attach can parse tool-specific item formats. (T11/T14.)
|
||||
|
||||
---
|
||||
|
||||
## 9. Open questions (for spec / workplans)
|
||||
|
||||
1. For an **encrypted shard** (UC-61), what *is* visible without keys — item IDs/counts/
|
||||
timestamps (structure shell) or nothing? Does shard-wiki ever hold keys, or only ever
|
||||
treat such shards as opaque backups?
|
||||
2. Is **writing to a tool's sync mirror** (UC-60) ever safe, or are interchange-format
|
||||
shards **read/projection/overlay-only** by policy (Joplin owns the format)?
|
||||
3. Does shard-wiki parse Joplin's **item format** via a dedicated format profile, or
|
||||
only attach the **export** (JEX/Markdown dir) as a cleaner snapshot (UC-37)?
|
||||
4. How do we avoid **double-sync** when a shard's storage target is itself driven by a
|
||||
sync daemon (Joplin) we don't control?
|
||||
|
||||
---
|
||||
|
||||
## 10. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Joplin Data API reference (https://joplinapp.org/help/api/references/rest_api/) | Local REST on `localhost:41184`, token auth, endpoints, 32-char IDs, item fields, pagination |
|
||||
| Joplin Plugin API docs (https://joplinapp.org/api/references/plugin_api/classes/joplin.html) | `joplin.data` / `joplin.workspace` / `joplin.contentScripts` / views / settings |
|
||||
| Joplin — Extending Joplin (https://joplinapp.org/help/api/) | Plugin distribution (npm + repository), Data API vs plugin access |
|
||||
| Joplin — WebDAV / Nextcloud sync (https://joplinapp.org/help/apps/sync/nextcloud/) | Sync targets; items as plain-text files on the target |
|
||||
| Joplin FAQ + community (https://joplinapp.org/help/faq/) | SQLite local store; Markdown content; E2EE before leaving device |
|
||||
| Obsidian vs Joplin comparison (https://petronellatech.com/blog/obsidian-vs-joplin/) | Architecture contrast (SQLite+sync vs file-over-app), E2EE framing |
|
||||
|
||||
Cross-references: `research/260614-obsidian-deep-dive/findings.md` (file-over-app
|
||||
contrast), `research/260614-notion-deep-dive/findings.md` (external-API, scoped grant),
|
||||
`research/260614-shard-spectrum-synthesis/findings.md` (the spectra this extends),
|
||||
`spec/UseCaseCatalog.md` (UC-31, UC-36, UC-38, UC-40, UC-51, UC-55, UC-57),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
|
||||
|
||||
---
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
- New UCs: **UC-60, UC-61** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-31, UC-36, UC-38, UC-40, UC-51, UC-55** (links UC-07, UC-57).
|
||||
- Architecture (no UC): interchange/sync-representation attach surface; **content
|
||||
opacity** as a proposed twelfth capability spectrum; local-REST sub-mode; format-aware
|
||||
file-store profiles → `SHARD-WP-0002` (T11, T14).
|
||||
- Boundary recorded: Joplin is a **sync daemon over one store**; shard-wiki attaches its
|
||||
**mirror as pages** and does not re-sync — and treats DB-local store and encrypted
|
||||
content as opaque (INTENT not-a-sync-daemon, graceful degradation, no silent mutation).
|
||||
</content>
|
||||
35
research/260614-jupyter-deep-dive/README.md
Normal file
35
research/260614-jupyter-deep-dive/README.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# 260614 — Jupyter Notebooks deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T3**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Jupyter Notebooks**: the **`.ipynb` JSON** document (ordered cells:
|
||||
markdown / code+outputs / raw), **kernels**, **embedded computed outputs** (MIME bundles),
|
||||
and **execution-count provenance**. The dominant modern computational document and the
|
||||
concrete case where the **derived output is captured and stored *inside* the source** with
|
||||
real-but-fragile provenance.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Tests the T1 **replication- vs derivation-projection** split on the dominant real artifact
|
||||
and adds the wrinkle that **outputs are stored back inside the source** — the source/
|
||||
projection line runs *through* the document.
|
||||
- The page model (T12) must carry a **notebook shape**: ordered cells with code cells owning
|
||||
embedded computed outputs that have **weak execution provenance** (run order, environment
|
||||
not captured).
|
||||
- Non-Markdown + lossy translation (T15): JSON+MIME bundles; nbconvert→Markdown is lossy and
|
||||
directional. JSON diffs are noisy → Jupytext text-pairing / nbdime (T13).
|
||||
|
||||
## Yield
|
||||
|
||||
- **UC-84** (new): attach/project a computational notebook preserving cell structure +
|
||||
embedded outputs, surfacing outputs as **snapshots with weak execution provenance**;
|
||||
re-execution **capability-gated**, default = present snapshot + static render.
|
||||
- Enrich **UC-54, UC-55, UC-59, UC-35**; links **UC-32, UC-83, UC-79**.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | `.ipynb` model, kernels/execution-count fragility, ecosystem (nbconvert/Jupytext/papermill/nbdime/nbstripout), capability profile, INTENT mapping, UC seed, architecture notes, open questions |
|
||||
185
research/260614-jupyter-deep-dive/findings.md
Normal file
185
research/260614-jupyter-deep-dive/findings.md
Normal file
@@ -0,0 +1,185 @@
|
||||
# Jupyter Notebooks — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T3 · **Subject:** Jupyter Notebooks — the
|
||||
`.ipynb` JSON document, kernels, embedded computed outputs, execution provenance.
|
||||
|
||||
## Why this dive
|
||||
|
||||
T1 (literate programming) established **one source → derived projections** and split
|
||||
**replication-projection** from **derivation-projection**. Jupyter is the *dominant modern
|
||||
computational document* and the concrete case where the **derived output is captured and
|
||||
stored inside the source** — a non-Markdown, partially-executable content type whose
|
||||
provenance is real but **fragile**. It is the most plausible concrete "computational shard"
|
||||
content type, so it tests the page model (T12), lossy translation (T15), and the
|
||||
output-provenance question head-on.
|
||||
|
||||
## 1. The `.ipynb` document model
|
||||
|
||||
A notebook is a single **JSON document** (`nbformat`), not Markdown:
|
||||
|
||||
- **`cells[]`** — an ordered list. Each cell has a `cell_type`:
|
||||
- `markdown` — prose (Markdown + LaTeX), the human-readable part.
|
||||
- `code` — source text (`source`), plus an **`execution_count`** and an **`outputs[]`**
|
||||
array captured from the last run.
|
||||
- `raw` — passthrough.
|
||||
- **`outputs[]`** (per code cell) carry results inline: `stream` (stdout/stderr),
|
||||
`execute_result` / `display_data` (a **MIME bundle** — `text/plain`, `text/html`,
|
||||
`image/png` base64, `application/json`, vendor MIME types), and `error` (traceback).
|
||||
- **`metadata`** at notebook and cell level (`kernelspec`, `language_info`, tags like
|
||||
`hide-input`, `scrolled`, slideshow roles).
|
||||
|
||||
So an `.ipynb` is **source + last-run computed outputs + environment metadata, fused in one
|
||||
JSON file**. The Markdown cells are an *island* inside a JSON envelope — relevant to how
|
||||
shard-wiki extracts/round-trips content.
|
||||
|
||||
## 2. Kernels and execution
|
||||
|
||||
- A **kernel** is a separate language process (IPython, IRkernel, IJulia, …) speaking the
|
||||
Jupyter messaging protocol (ZeroMQ). The document is **decoupled from the kernel**: the
|
||||
`.ipynb` persists *captured* outputs; re-running requires a live kernel + the right
|
||||
environment.
|
||||
- **`execution_count`** numbers the order cells were *run*, which **need not match document
|
||||
order** — the infamous **hidden-state / out-of-order execution** problem: stored outputs
|
||||
may reflect a run sequence that no longer corresponds to top-to-bottom reading.
|
||||
- Reproducibility therefore depends on **out-of-band state**: package versions, data files,
|
||||
environment, random seeds — none captured by `nbformat` itself.
|
||||
|
||||
**Consequence for shard-wiki:** the captured outputs are a **snapshot projection with weak
|
||||
provenance** — honest treatment must mark them as "computed at run N, environment not
|
||||
guaranteed," never as live or authoritative truth.
|
||||
|
||||
## 3. The ecosystem (relevant to attach/project/translate)
|
||||
|
||||
- **nbconvert** — derives other forms from a notebook: HTML, Markdown, LaTeX/PDF, slides,
|
||||
script. This is **derivation-projection** (T1): notebook source → rendered view, lossy in
|
||||
both directions (HTML keeps outputs; `--to script` keeps only code, like `tangle`).
|
||||
- **Jupytext** — represents a notebook **as** a `.py`/`.md` text file (pairing), making it
|
||||
**git-diffable plain text** and round-trippable — directly relevant to storing notebooks
|
||||
in a git shard without JSON-diff noise.
|
||||
- **papermill** — parameterize + execute a notebook to produce a new output notebook
|
||||
(notebook as a runnable template — a *derivation with inputs*).
|
||||
- **JupyterLab / Notebook / nbviewer / Colab** — front-ends; nbviewer renders a static
|
||||
read-only projection from a URL (a natural projection target).
|
||||
- **`nbstripout`** — strips outputs before commit: teams treat **outputs as derived noise**,
|
||||
keeping only source under version control — an explicit "source canonical, outputs
|
||||
derived" stance mirroring T1.
|
||||
|
||||
## 4. Capability profile (as a shard / content type)
|
||||
|
||||
| Dimension (synthesis spectrum) | Jupyter notebook |
|
||||
|--------------------------------|------------------|
|
||||
| Attachment mode | file-store (`.ipynb` JSON in a repo) or via Jupyter Server REST API |
|
||||
| Addressing granularity | document; **cell** as sub-address (by index / id; `nbformat 4.5+` adds stable cell `id`) |
|
||||
| Content identity | file path; cell `id` (4.5+) else positional |
|
||||
| Structure | **ordered cell list** (markdown / code+outputs / raw); MIME-bundle outputs |
|
||||
| History | VCS on the file; **JSON diffs are noisy** unless paired (Jupytext) or stripped |
|
||||
| Merge model | git on JSON (poor) → **paired text** (good) or nbdime (cell-aware diff/merge) |
|
||||
| Native query | none |
|
||||
| Translation | nbconvert → HTML/MD/script/PDF (lossy, directional); Jupytext text pairing |
|
||||
| Write granularity | file / **cell** |
|
||||
| Operational envelope | a kernel + environment to (re)execute; static render needs none |
|
||||
| Content opacity | **mixed**: source transparent; outputs = MIME blobs (some opaque, e.g. base64 PNG) |
|
||||
| Provenance | `execution_count` (weak, out-of-order); environment **not** captured |
|
||||
| **Computed-output** | **stored inline**, snapshot, reproducibility out-of-band |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Replication- vs derivation-projection (T1) confirmed and extended.** nbconvert (→HTML/
|
||||
script) and nbviewer are derivation-projections; `--to script` is literally `tangle`.
|
||||
Jupyter adds a third wrinkle: **the derived output is also stored back inside the source**
|
||||
(captured outputs), so the "source vs projection" line runs *through* the document.
|
||||
- **Union without erasure / provenance honesty.** Captured outputs must be surfaced **as
|
||||
snapshots with weak provenance** (run N, environment unguaranteed) — a concrete instance
|
||||
of "never hide freshness/authorship." The out-of-order `execution_count` is exactly the
|
||||
kind of fragility shard-wiki must *show*, not paper over.
|
||||
- **Non-Markdown content + lossy translation (UC-55/UC-59).** `.ipynb` is JSON with embedded
|
||||
MIME-bundle outputs; any Markdown projection is **lossy** (loses live outputs, kernel,
|
||||
rich MIME). Surface the lossiness; keep the JSON as canonical payload (T12/T15).
|
||||
- **Markdown island.** Markdown cells fit the text-first model, but only as *fragments
|
||||
inside* a JSON envelope — the adapter extracts/round-trips them, it does not pretend the
|
||||
notebook is a Markdown page.
|
||||
|
||||
### Divergences / boundaries
|
||||
|
||||
- **shard-wiki is not a kernel host.** Re-execution (driving a kernel) is out of scope/
|
||||
capability-gated; default treatment is **attach + present captured outputs as a snapshot
|
||||
projection** + offer nbconvert-style static render. Executing/parameterizing (papermill)
|
||||
is an optional capability, never assumed.
|
||||
- **Outputs-in-source is an anti-pattern to respect, not adopt.** Teams strip/pair outputs
|
||||
precisely because mixing derived data into the source breaks diffs. shard-wiki should
|
||||
prefer the **source-canonical, outputs-as-derived** reading (Jupytext pairing / nbstripout
|
||||
ethos) and treat stored outputs as a capturable projection.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Computational-notebook as a first-class content type** with cell structure + inline
|
||||
**computed outputs carrying (weak) execution provenance** — UC-84.
|
||||
2. **Outputs = derivation-projection snapshot** (T1 vocabulary): regenerable only with a
|
||||
kernel+environment; degrade gracefully to the stored snapshot / static render.
|
||||
3. **Cell-level addressing** (stable cell `id`, nbformat 4.5+) as the sub-page granularity
|
||||
for transclusion/anchoring (UC-32/UC-35).
|
||||
4. **Text-pairing (Jupytext)** as the git-friendly storage strategy — feeds the
|
||||
history-portability thread (poor JSON diffs → paired text / nbdime).
|
||||
|
||||
## 6. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-84 | Attach/project a **computational notebook** (`.ipynb`): preserve **cell structure** (markdown / code / output) and **embedded computed outputs**, surfacing each output **as a snapshot with its (weak) execution provenance** (run count, environment not guaranteed) — re-execution is **capability-gated**, default is present-the-snapshot + offer a static rendered projection | **new** |
|
||||
| — | Notebook JSON / MIME-bundle outputs = non-Markdown content; Markdown projection is lossy | enrich **UC-55**, **UC-59** |
|
||||
| — | Computed/evaluated cell = computation-defined content | enrich **UC-54** |
|
||||
| — | Cell `id` (nbformat 4.5+) = sub-page address for anchor/transclusion | enrich **UC-35**, links **UC-32** |
|
||||
| — | Stored outputs as derived snapshot (nbstripout/Jupytext ethos) = source-canonical/outputs-derived | links **UC-83**, **UC-79** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12 (page model):** add **computational-notebook** as a page shape — an **ordered cell
|
||||
list** where code cells own **embedded computed outputs** (MIME bundles) with weak
|
||||
execution provenance. Distinct from prose, typed records, query-defined, inline-embedded
|
||||
objects (Quip/Notion), typed-graph (Wikibase), and the literate one-source-many-projection
|
||||
shape (UC-83). The defining new attribute: **derived output stored *inside* the source**.
|
||||
- **T15 (translation / fidelity):** `.ipynb` is non-Markdown; nbconvert→Markdown is **lossy
|
||||
and directional** (drops live outputs/kernel/rich MIME). Keep JSON canonical; any Markdown
|
||||
is a projection. MIME-bundle outputs map to the content-opacity spectrum (text→html→base64
|
||||
image = transparent→opaque).
|
||||
- **T13 (history):** JSON diffs are **noisy**; record **text-pairing (Jupytext)** and
|
||||
**cell-aware diff/merge (nbdime)** as history-portability strategies for embedded-output
|
||||
documents. Reinforces "source-canonical, outputs-derived."
|
||||
- **T16 (projection):** captured outputs are a **derivation-projection snapshot**;
|
||||
re-execution (kernel) and parameterized execution (papermill) are **capabilities**, not
|
||||
assumptions; degrade to the stored snapshot / nbviewer-style static render.
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Does shard-wiki ever **re-execute** a notebook (host/broker a kernel), or strictly
|
||||
attach + present captured outputs + static render? (Same scope boundary as UC-83/UC-56
|
||||
"do we ever drive the derivation.")
|
||||
2. Is **UC-84** distinct from **UC-83**, or is a notebook just the "outputs-stored-in-source"
|
||||
special case of the literate one-source-many-projection pattern? (Kept separate: UC-84's
|
||||
defining trait is *captured derived output embedded in the canonical source with weak
|
||||
provenance* — a page-model attribute UC-83 doesn't carry.)
|
||||
3. How are **MIME-bundle outputs** represented in the page model — opaque provenance-tagged
|
||||
blobs, a typed-asset registry (UC-55 open question #10), or selected-MIME projection?
|
||||
4. Default storage: attach `.ipynb` **as-is** (JSON, noisy diffs) or prefer a **paired text
|
||||
representation** when the shard is a git repo? (Policy → configurable.)
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- Jupyter `nbformat` reference (cells, outputs, MIME bundles, cell `id` 4.5+);
|
||||
Jupyter messaging protocol / kernels docs.
|
||||
- **nbconvert**, **nbviewer**, **JupyterLab**, **Colab** docs.
|
||||
- **Jupytext**, **papermill**, **nbdime**, **nbstripout** project docs.
|
||||
- prior: `research/260614-literate-programming-deep-dive/` (replication- vs
|
||||
derivation-projection, UC-83); `research/260614-notion-deep-dive/` (block-JSON,
|
||||
external-API), `research/260614-quip-deep-dive/` (inline embedded objects, UC-55/58/59).
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UC **UC-84** carries the marker **⊜** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md` (true lineage = this dive). Enriched: UC-54, UC-55, UC-59, UC-35;
|
||||
links UC-32, UC-83, UC-79. Architecture cross-refs: SHARD-WP-0002 T12 (notebook page shape:
|
||||
outputs embedded in source), T15 (lossy non-Markdown translation; MIME opacity), T13
|
||||
(paired-text / nbdime history), T16 (output = derivation-projection snapshot; kernel =
|
||||
capability).
|
||||
35
research/260614-literate-programming-deep-dive/README.md
Normal file
35
research/260614-literate-programming-deep-dive/README.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# 260614 — Literate Programming (Knuth's WEB / weave / tangle) deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T1**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **literate programming** — Knuth's WEB and the **`weave`/`tangle`** model
|
||||
— as the deepest ancestor of shard-wiki's **projection** and **transclusion** ideas
|
||||
applied to *executable* content. The keystone: **one canonical source → two co-derived
|
||||
projections** (typeset docs via `weave`, compilable code via `tangle`), plus **named code
|
||||
chunks** assembled by reference (transclusion).
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Establishes **one-source-many-projections** as a page-model + projection pattern that
|
||||
*generalizes* compile-to-static (UC-79, single output) to **N co-equal, semantically
|
||||
different** derived views — feeds SHARD-WP-0002 **T12/T16**.
|
||||
- Splits projection into **replication-projection** (lazy cache, current default) vs
|
||||
**derivation-projection** (transform/compile/weave/evaluate) — a distinction the rest of
|
||||
the computational batch (notebooks, REPLs) extends.
|
||||
- Named chunks are the **executable-content face of transclusion / compose-by-reference**
|
||||
(UC-32 / UC-44).
|
||||
|
||||
## Yield
|
||||
|
||||
- **UC-83** (new): attach a single-source-multiple-projection (literate) artifact; present
|
||||
each derived view with output→source provenance; edits target the source.
|
||||
- Enrich **UC-32**, **UC-44**, **UC-79**; links **UC-54** (computed/evaluated projection,
|
||||
→ T3 Jupyter).
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | WEB model, named-chunk transclusion, descendants (noweb/org-babel/knitr/Jupytext), capability profile, INTENT mapping, UC seed, architecture notes, open questions |
|
||||
180
research/260614-literate-programming-deep-dive/findings.md
Normal file
180
research/260614-literate-programming-deep-dive/findings.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# Literate Programming (Knuth's WEB / weave / tangle) — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T1 · **Subject:** Donald Knuth's WEB
|
||||
system and the literate-programming model (`weave`/`tangle`, named chunks).
|
||||
|
||||
## Why this dive
|
||||
|
||||
SHARD-WP-0004 asks the carried question: *can a shard-wiki page be a source that is
|
||||
woven/evaluated into rendered forms, and how do projection/transclusion/provenance treat
|
||||
the source vs the output?* Literate programming is the **deepest ancestor** of that idea.
|
||||
Knuth (1984) inverted the program/comment relationship: you write a **document** for
|
||||
humans whose fragments happen to also be the program. From the **one WEB source** two
|
||||
tools derive two artifacts: **`weave` → typeset documentation** (TeX) and **`tangle` →
|
||||
compilable code** (Pascal/C/…). This is *one source, two projections* in its purest,
|
||||
oldest form — the conceptual root of shard-wiki's **projection** and **transclusion**.
|
||||
|
||||
## 1. The WEB model: one source, two tools
|
||||
|
||||
A WEB file interleaves prose and code in author-chosen order (the order that best
|
||||
*explains*, not the order the compiler needs). Two programs read it:
|
||||
|
||||
- **`weave`** produces a `.tex` file → typeset documentation: prose, pretty-printed code,
|
||||
a cross-reference index of where each chunk and identifier is defined/used.
|
||||
- **`tangle`** produces a compilable source file → reorders and expands the code chunks
|
||||
into the sequence the compiler demands, strips the prose, macro-expands references.
|
||||
|
||||
The crucial property: **the two outputs are co-derived from one canonical source and are
|
||||
semantically different audiences** (human reader vs compiler). Neither output is the
|
||||
source; both are **regenerable, disposable derivations**. Editing happens on the WEB; you
|
||||
never edit the woven `.tex` or the tangled `.c`.
|
||||
|
||||
## 2. Named chunks = transclusion of code fragments
|
||||
|
||||
The organizing primitive is the **named section / code chunk**:
|
||||
|
||||
- A chunk is declared `<<name>>=` and **referenced** elsewhere as `<<name>>`. `tangle`
|
||||
expands references in place (recursively) to assemble the final program.
|
||||
- A chunk name can be **defined in multiple places** and is *accreted* (later `+=`
|
||||
additions append) — so one logical unit is authored across scattered locations.
|
||||
- References can appear before definitions; resolution is by name, not by position.
|
||||
|
||||
This is **transclusion by reference** (UC-32) and **compose-by-reference / manifest**
|
||||
(UC-44, the EDL/Xanadu lineage): the document is a graph of named fragments assembled at
|
||||
derivation time. Knuth's chunk graph is the same shape as Xanadu's reference-not-copy and
|
||||
shard-wiki's "compose a page from referenced parts" — applied to *executable* content and
|
||||
resolved by a build tool rather than a viewer.
|
||||
|
||||
## 3. The descendants (noweb, CWEB, org-babel, Sweave/knitr, Jupytext)
|
||||
|
||||
- **CWEB** (Knuth/Levy): WEB for C. **noweb** (Ramsey): language-agnostic, minimal markup
|
||||
(`<<chunk>>`), `notangle`/`noweave` — proof that the *model* (chunks + two projections)
|
||||
is separable from any one language or typesetter.
|
||||
- **org-babel** (Emacs Org-mode): named source blocks, `:noweb` references, **tangle** to
|
||||
files **and evaluate** blocks inline (results captured back into the document) — literate
|
||||
programming fused with notebook execution (bridges to T2/T3).
|
||||
- **Sweave / knitr** (R): weave prose + R, executing code and **interleaving computed
|
||||
results/figures** into the woven document — adds the *computed-output* dimension that
|
||||
Jupyter (T3) centers on.
|
||||
- **Jupytext**: represents a Jupyter notebook **as** a literate text/Markdown source —
|
||||
closing the loop: the notebook (T3) becomes a weave/tangle-style plain-text source.
|
||||
|
||||
The throughline: **one canonical source → N derived projections**, where projections may
|
||||
be (a) reformatted (weave), (b) reordered/extracted (tangle), or (c) **evaluated**
|
||||
(org-babel/knitr) — the evaluated case is exactly the computational-page question.
|
||||
|
||||
## 4. Capability profile (as a would-be shard / page type)
|
||||
|
||||
| Dimension (synthesis spectrum) | Literate-programming source |
|
||||
|--------------------------------|-----------------------------|
|
||||
| Attachment mode | file-store (a WEB/`.nw`/`.org` text source in a repo) |
|
||||
| Addressing granularity | document; **named chunk** as sub-page address |
|
||||
| Content identity | source file path + chunk name (name-resolved, not position) |
|
||||
| Structure | **graph of named chunks** assembled by reference |
|
||||
| History | whatever VCS holds the source (git) — text, diffable |
|
||||
| Merge model | text/git merge on the source |
|
||||
| Native query | none; `weave` emits a cross-reference index (derived) |
|
||||
| Translation | source → woven docs **and** tangled code (build-time) |
|
||||
| Write granularity | file / **chunk** (text region) |
|
||||
| Operational envelope | a build tool (`tangle`/`weave`/`noweb`/babel) |
|
||||
| Content opacity | transparent text |
|
||||
| Provenance | VCS author/time; chunk cross-ref maps output→source location |
|
||||
| **Projection model** | **one source → many co-equal derived projections** (new emphasis) |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Projection (canonical vs derived).** Literate programming is the archetype of
|
||||
"canonical source, regenerable derived view" — the same principle as ikiwiki
|
||||
compile-to-static (UC-79), but generalized: **two-plus co-equal projections** from one
|
||||
source (docs *and* code), not a single output. Strengthens the rule *never confuse a
|
||||
projection for the source*.
|
||||
- **Transclusion / compose-by-reference.** Named chunks are transclusion (UC-32) and a
|
||||
manifest of referenced parts (UC-44) — resolved at derivation time. Confirms
|
||||
transclusion=clone=embed=reference as one primitive that also covers *fragment assembly
|
||||
of executable content*.
|
||||
- **Markdown-first but backend-neutral page model.** noweb/org/Jupytext show the literate
|
||||
source can *be* Markdown-ish plain text — so a "literate page" fits the text-first model;
|
||||
the *derivations* are the non-text part.
|
||||
- **Mechanism over policy.** weave/tangle are mechanisms; *which* projections to
|
||||
materialize, when to regenerate, and where outputs go stay configurable.
|
||||
|
||||
### Divergences / boundaries
|
||||
|
||||
- **shard-wiki is not a build system.** It should *recognize and present* a
|
||||
source-with-projections (attach the source, surface derived views with provenance), not
|
||||
re-implement `tangle`/kernels. Materializing a projection may delegate to the source's
|
||||
own tool or be capability-gated to "snapshot only."
|
||||
- **The interesting projection is derivation, not caching.** shard-wiki's base projection
|
||||
is cache-like (lazy copy of remote content, UC-53/57). Weave/tangle is a *different*
|
||||
projection species: **transform/derive** one source into rendered forms. The contract
|
||||
should model projection as having (at least) two kinds: **replication-projection** and
|
||||
**derivation-projection** (compile/weave/evaluate).
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **One-source-many-projections** as a first-class page-model + projection pattern
|
||||
(generalizes UC-79's single output) — see UC-83.
|
||||
2. **Named-chunk transclusion** as the executable-content face of UC-32/UC-44 (assembly by
|
||||
reference at derivation time).
|
||||
3. **Output→source provenance** (the woven cross-ref index): every derived view must point
|
||||
back to the exact source location it came from — never present derived output without
|
||||
that link (union-without-erasure for derivations).
|
||||
|
||||
## 6. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-83 | Attach a **single-source-multiple-projection** artifact (a literate/woven source): treat the source as canonical, present each **derived projection** (e.g. a documentation view *and* a code view) with **provenance back to the one source**, edits target the source and projections regenerate (delegated to the source's tool or degraded to a static snapshot) | **new** |
|
||||
| — | Named chunks `<<name>>` = transclusion / compose-by-reference of (executable) fragments | enrich **UC-32**, **UC-44** |
|
||||
| — | Generalize compile-to-static (single output) to **N co-equal projections** from one source | enrich **UC-79** |
|
||||
| — | Computed/evaluated projection (org-babel/knitr) = derivation-projection with results | links **UC-54**, foreshadows **T3** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12 (page model):** add **one-source-many-projections** as a page-model shape — a page
|
||||
may be a *source* whose presented forms are **derivations** (woven docs, tangled code,
|
||||
evaluated results), each carrying output→source provenance. Distinct from prose, typed
|
||||
records, query-defined, and inline-embedded objects already logged.
|
||||
- **T16 (projection):** split projection into **replication-projection** (lazy cache of
|
||||
remote content — current default) vs **derivation-projection** (transform/compile/weave/
|
||||
evaluate a source into rendered forms). Derivation-projection is regenerable, may
|
||||
delegate to an external tool, and degrades to a captured snapshot when the tool is
|
||||
absent (graceful degradation).
|
||||
- **Transclusion (T16):** named-chunk-by-name resolution is a transclusion variant where
|
||||
the *target is a fragment resolved by name across the source*, assembled at derivation
|
||||
time — a concrete shape for UC-32/UC-44 mechanics.
|
||||
- **Capability gating:** "can derive projection X" is a capability; a shard that can't run
|
||||
the tool still exposes the source + any pre-built/snapshot projections (UC-83 degrade).
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Does shard-wiki ever *drive* a derivation (run weave/tangle/evaluate), or only attach
|
||||
sources and surface pre-built projections + snapshots? (Same scope question as UC-56
|
||||
"do we ever compile-to-static ourselves," now for literate sources.)
|
||||
2. Is **UC-83** distinct enough from UC-79 (compile-to-static) to stand alone, or should
|
||||
UC-79 be re-read as the single-output special case of UC-83's N-projection general case?
|
||||
(Recorded as a possible later consolidation; kept separate now because UC-83's
|
||||
projections are *co-equal and semantically different audiences*, not one publish target.)
|
||||
3. How is **output→source provenance** represented when a derived line came from a chunk
|
||||
accreted across several source locations (the cross-ref is many-to-one)?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- Knuth, *Literate Programming* (1984, *Computer Journal*); the WEB user manual; *TeX: The
|
||||
Program* / *MMIX* as canonical WEB exemplars.
|
||||
- Ramsey, *noweb* (a simple, extensible literate-programming tool).
|
||||
- CWEB (Knuth & Levy); Emacs **Org-mode babel** (tangle + evaluate); **Sweave**/**knitr**
|
||||
(R); **Jupytext** (notebook-as-text).
|
||||
- prior: `research/260614-ikiwiki-deep-dive/` (compile-to-static, canonical-vs-derived);
|
||||
`research/260614-xanadu-deep-dive/` (compose-by-reference / EDL, UC-44).
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UC **UC-83** carries the marker **⊛** in the federation column of
|
||||
`spec/UseCaseCatalog.md` (true lineage = this dive; literate programming is design prior
|
||||
art, not a candidate shard, so the marker sits with the projection/compose-by-reference
|
||||
family). Enriched: UC-32, UC-44, UC-79; links UC-54. Architecture cross-refs: SHARD-WP-0002
|
||||
T12 (one-source-many-projections page shape), T16 (replication- vs derivation-projection;
|
||||
named-chunk transclusion).
|
||||
46
research/260614-localfirst-workspaces-deep-dive/README.md
Normal file
46
research/260614-localfirst-workspaces-deep-dive/README.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# 260614 — Local-first workspaces cohort deep dive (Anytype · AFFiNE · AppFlowy)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A **combined cohort deep dive** of three open-source, local-first "all-in-one workspace"
|
||||
tools — **Anytype**, **AFFiNE**, **AppFlowy** — studied together because they share the
|
||||
one trait none of the prior eleven systems had: **the store is a CRDT** (Anytype's
|
||||
*any-sync*, AFFiNE's *Yjs/y-octo*, AppFlowy's *Yrs*). The substrate is the contribution;
|
||||
studying them together isolates it from each tool's surface features.
|
||||
|
||||
Per-tool highlights:
|
||||
- **Anytype** — `any-sync`: **P2P + E2EE** CRDT (sync/file/consensus/coordinator nodes;
|
||||
encrypted backup), a **typed object graph** (user-editable types + relations), IPFS
|
||||
files, an **open API + MCP**.
|
||||
- **AFFiNE** — **Yjs/y-octo** over **OctoBase**; **BlockSuite** editor framework; docs,
|
||||
whiteboards, and databases as **views of the same block set**; WebSocket sync,
|
||||
self-host (Docker, MPL-2.0).
|
||||
- **AppFlowy** — **Yrs** via **AppFlowy-Collab** (Rust); Flutter UI; Notion-style
|
||||
**databases + views**; pluggable storage/sync (RocksDB/IndexedDB; AppFlowy Cloud/
|
||||
Supabase).
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Per-tool sections, the CRDT cohort thesis, comparative matrix, combined capability profile, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Cohort dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-64
|
||||
attach a CRDT-synced local-first shard with native conflict-free merge; UC-65 attach a
|
||||
peer-to-peer / decentralized shard with no single canonical endpoint); UC-36/47/48/51/55/
|
||||
57/58/61 enriched. Logged for `SHARD-WP-0002` (T11/T14): a **merge-model** capability
|
||||
(proposed thirteenth spectrum: `none / git / native-CRDT`), a **CRDT-replica** substrate +
|
||||
**P2P/no-central-endpoint** attach mode, an **endpoint-model** capability, and MCP as an
|
||||
integration surface.
|
||||
|
||||
**Cohort thesis recorded:** CRDT changes the adapter math — the backend performs
|
||||
conflict-free merge itself (so shard-wiki must not impose git/text merge), history is a
|
||||
CRDT update log (supplement, not git), and the attach surface is a **replica or sync
|
||||
endpoint** (Anytype adds **P2P + E2EE**). **Boundary:** CRDT local-first candidate shards
|
||||
— attach a replica/projection as pages, respect native merge, never re-drive their sync;
|
||||
not substrates and not the federation layer.
|
||||
</content>
|
||||
261
research/260614-localfirst-workspaces-deep-dive/findings.md
Normal file
261
research/260614-localfirst-workspaces-deep-dive/findings.md
Normal file
@@ -0,0 +1,261 @@
|
||||
# Findings — Anytype, AFFiNE, AppFlowy: the CRDT local-first workspace cohort
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped products (cohort dive)** — three open-source, local-first
|
||||
"all-in-one workspace" tools studied together; candidate shards of a **new substrate
|
||||
family: CRDT-synced local-first stores**
|
||||
Lens: shard-wiki — the CRDT substrate and its native-merge / P2P / E2EE implications for
|
||||
the adapter contract
|
||||
|
||||
> Why one combined memo. Anytype, AFFiNE, and AppFlowy are the open-source local-first
|
||||
> Notion/Miro alternatives, and they share the one trait the prior eleven systems never
|
||||
> had: **the store is a CRDT** (Anytype's *any-sync*, AFFiNE's *Yjs/y-octo*, AppFlowy's
|
||||
> *Yrs*). That single fact changes the adapter math — **the backend performs conflict-
|
||||
> free merge itself**, history is a CRDT update log (not git), and content arrives as a
|
||||
> replica/export, not files or rows. Studying them together isolates the *substrate*
|
||||
> contribution from each tool's surface features and avoids triplicating boilerplate.
|
||||
> Per-tool sections (§1–§3), then the cohort thesis and what it means for shard-wiki
|
||||
> (§4 onward).
|
||||
|
||||
Contrast set: file-store (Obsidian/Logseq), client-DB (Roam), hosted-DB (Notion),
|
||||
SQLite-local+file-sync (Joplin). This cohort adds **CRDT-replica** as a substrate.
|
||||
|
||||
---
|
||||
|
||||
## 1. Anytype — P2P, E2EE, typed object graph (any-sync)
|
||||
|
||||
- **Substrate:** the open **any-sync** protocol — **local-first, peer-to-peer,
|
||||
end-to-end encrypted** CRDT. Infrastructure is four node types: **sync, file,
|
||||
consensus, coordinator**; **backup nodes store ciphertext they cannot read**. Every
|
||||
change is **encrypted and signed with the user's private key**, verified via a shared
|
||||
public key. Files via **IPFS**.
|
||||
- **Data model:** **spaces** = graph-based databases; users define **custom object
|
||||
types with relations** ("everything is an object"; a user-editable **ontology**).
|
||||
Closest to Notion's typed model but **local-first + E2EE + user-owned schema**.
|
||||
- **Extension:** an **open API** for integrations, **plus MCP support** (AI-agent
|
||||
collaboration) — a notable modern surface (a shard speaking MCP).
|
||||
- shard-wiki read: the **most decentralized + encrypted** shard yet — no single
|
||||
canonical endpoint (attach a local replica or a node), content opaque without keys
|
||||
(UC-61), typed object graph (UC-58), CRDT merge (UC-64).
|
||||
|
||||
## 2. AFFiNE — docs + whiteboard + DB as views of one block set (Yjs/BlockSuite)
|
||||
|
||||
- **Substrate:** **Yjs CRDT** via **y-octo** (native Rust Yjs) over **OctoBase** (Rust
|
||||
local-first engine). Local-first by default; optional **WebSocket realtime** sync to
|
||||
AFFiNE Cloud or self-host (MPL-2.0, Docker).
|
||||
- **Editor:** **BlockSuite** — an open block-editor framework providing the **page
|
||||
editor, the edgeless canvas, and block types (incl. databases/embeds)**.
|
||||
- **Headline:** **docs, whiteboards, and databases are *views of the same underlying
|
||||
data*** — one block set rendered as page mode, edgeless canvas, or DB views. CRDT
|
||||
guarantees offline edits "merge mathematically perfectly" on reconnect.
|
||||
- shard-wiki read: the **one-data-many-renderings** model shipped (ZigZag dimensions,
|
||||
UC-47/48), with **non-Markdown spatial content** (edgeless canvas, UC-55) and CRDT
|
||||
merge (UC-64).
|
||||
|
||||
## 3. AppFlowy — Rust/Flutter, Notion-style DBs over Yrs CRDT
|
||||
|
||||
- **Substrate:** **Yrs** (Rust Yjs) wrapped as **AppFlowy-Collab** — CRDT primitives +
|
||||
persistence + domain objects (documents, **databases**, folders, importers, plugins,
|
||||
user state). **Pluggable storage/sync:** RocksDB (native) / IndexedDB (wasm) locally;
|
||||
**Supabase or AppFlowy Cloud** (self-hostable) for sync, via **trait-based local-only
|
||||
vs cloud**.
|
||||
- **Data model:** Notion-like **databases with views** (grid/board/calendar), typed
|
||||
fields. Flutter UI + Rust core, modular.
|
||||
- shard-wiki read: a **self-hostable CRDT Notion** — typed DBs + views (UC-58, UC-54),
|
||||
CRDT merge (UC-64), self-host sync endpoint (UC-57).
|
||||
|
||||
---
|
||||
|
||||
## 4. The cohort thesis — CRDT changes the adapter math
|
||||
|
||||
All three converge on **CRDT-synced, local-first, open-source, self-hostable** stores
|
||||
with **typed/database object models** and **non-Markdown content** (canvas/DB). The
|
||||
*substrate* is the new thing, and it has four consequences shard-wiki must absorb:
|
||||
|
||||
1. **Native, conflict-free merge (UC-64).** The backend resolves concurrent edits
|
||||
itself (Yjs/Yrs/any-sync). shard-wiki must **not impose git/text merge** on a CRDT
|
||||
shard; it either speaks the CRDT (replica) or stays a **projection/overlay** that
|
||||
respects CRDT semantics. This adds a **merge-model** capability:
|
||||
`none → git/text → native-CRDT` — a candidate **thirteenth spectrum** for the
|
||||
contract (after Joplin's content-opacity twelfth).
|
||||
2. **History is a CRDT update log, not git (UC-36).** No portable commit history; the
|
||||
coordination journal **supplements** (like Notion/Joplin), or shard-wiki snapshots
|
||||
the replica over time.
|
||||
3. **The attach surface is a replica or a sync endpoint, not files/rows.** Attach a
|
||||
**local CRDT replica** (offline, full state) or the **self-host sync server's API**
|
||||
(AFFiNE Cloud / AppFlowy Cloud / any-sync node). Anytype adds **P2P with no single
|
||||
canonical endpoint** (UC-65) and **E2EE content opacity** (UC-61).
|
||||
4. **Typed object graphs + multi-view (UC-58, UC-47/48, UC-55).** User-editable
|
||||
ontologies (Anytype types/relations), Notion-style DBs (AppFlowy), and one-block-set-
|
||||
many-renderings (AFFiNE page/canvas/DB) — the structured + dimensional demands the
|
||||
prior dives raised, now local-first.
|
||||
|
||||
These are **sync layers** (any-sync, Yjs sync, AppFlowy Cloud) — like Joplin, the
|
||||
**not-a-file-sync-daemon** boundary applies: shard-wiki attaches a **replica/projection
|
||||
as pages** and does **not** re-implement or re-drive their CRDT sync.
|
||||
|
||||
---
|
||||
|
||||
## 5. Comparative matrix
|
||||
|
||||
| | Anytype | AFFiNE | AppFlowy |
|
||||
|---|---------|--------|----------|
|
||||
| CRDT | any-sync (custom) | Yjs (y-octo) | Yrs |
|
||||
| Sync | **P2P + nodes, E2EE** | WebSocket, cloud/self-host | RocksDB local + cloud/Supabase |
|
||||
| Encryption | **E2EE (default)** | optional (transport/self-host) | self-host boundary |
|
||||
| Local engine | local + IPFS files | OctoBase (Rust) | RocksDB / IndexedDB |
|
||||
| Data model | **typed object graph (ontology)** | blocks; **doc/canvas/DB = one data, many views** | Notion-style **DBs + views** |
|
||||
| Non-MD content | objects, files | **edgeless canvas** | board/calendar/grid |
|
||||
| Extension | **open API + MCP** | BlockSuite framework | Flutter/Rust modules; plugins (roadmap) |
|
||||
| Self-host | backup/sync nodes | Docker (MPL-2.0) | AppFlowy Cloud |
|
||||
| Markdown | export (lossy) | import/export (lossy) | import/export (lossy) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Combined capability profile (as a shard)
|
||||
|
||||
| Capability | This cohort | Notes for the adapter contract |
|
||||
|------------|-------------|--------------------------------|
|
||||
| Read | yes (replica or sync API) | local CRDT replica (offline) or self-host/P2P endpoint |
|
||||
| Write | yes, **CRDT-mediated** | writes are CRDT ops; respect merge — don't overwrite |
|
||||
| Merge | **native CRDT (conflict-free)** | **new capability**: backend owns merge; no external git merge (UC-64) |
|
||||
| Write granularity | block/object | fine-grained |
|
||||
| Identity / addressing | object/block IDs (CRDT-assigned) | fine-grained, store-minted (UC-51) |
|
||||
| Structure | **typed object graph / DBs + relations** | user-editable ontology (Anytype); DB views (AppFlowy) (UC-58) |
|
||||
| History | **CRDT update log, not git** | supplement / snapshot (UC-36) |
|
||||
| Native query | varies (graph/DB filters) | delegate where present (UC-52) |
|
||||
| Subscribe | realtime CRDT sync / P2P | push by nature; attach as replica (UC-31) |
|
||||
| Content opacity | **Anytype E2EE → ciphertext** | opaque without keys (UC-61) |
|
||||
| Endpoint model | **replica / self-host server / P2P (Anytype)** | no single canonical URL for P2P (UC-65) |
|
||||
| Content types | docs + **canvas/DB/objects** | non-Markdown (UC-55); multi-view (UC-47/48) |
|
||||
| Markdown | lossy export | fidelity report (UC-59) |
|
||||
| Attach modes | CRDT replica · self-host API · (Anytype) MCP/open API | new substrate row in the taxonomy |
|
||||
|
||||
Verdict: a **demanding but coherent new family** — best attached as a **local replica
|
||||
(offline, full CRDT state)** projected to pages, or via a **self-host sync endpoint**;
|
||||
Anytype additionally **P2P + E2EE** (replica-only/opaque). Standout demands:
|
||||
**native-CRDT merge** (UC-64) and **P2P/no-central-endpoint + E2EE** (UC-65/UC-61).
|
||||
|
||||
---
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
|
||||
- **Local-first sovereignty** is the cohort's whole pitch — strong INTENT alignment;
|
||||
each is a legitimate sovereign shard with user-owned data.
|
||||
- **Self-hostable + open-source** means shard-wiki can attach within a user's trust
|
||||
boundary (good for the L0→L4 authz ladder).
|
||||
- **One-data-many-views** (AFFiNE) and **typed object graphs** (Anytype/AppFlowy) are
|
||||
the dimensional (UC-47/48) and structured (UC-58) demands, now validated local-first.
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Do not impose git/text merge on a CRDT shard.** The backend merges; shard-wiki
|
||||
respects CRDT semantics or stays a projection/overlay (UC-64). Treating a CRDT shard
|
||||
like a file shard corrupts it.
|
||||
2. **Not a sync daemon.** any-sync/Yjs/AppFlowy Cloud already sync; shard-wiki attaches
|
||||
a **replica as pages**, never re-drives their sync (INTENT not-a-file-sync-daemon).
|
||||
3. **P2P has no canonical endpoint.** Don't model the shard as a URL; attach a replica
|
||||
or a named peer/node (UC-65).
|
||||
4. **E2EE shards are opaque** without keys — backup/structure-shell only (UC-61).
|
||||
5. **Proprietary CRDT/object formats are lossy to Markdown** — translate with a fidelity
|
||||
report (UC-59), one workspace = one shard, not the federation layer.
|
||||
|
||||
### 7.3 What the cohort teaches that shard-wiki should keep
|
||||
|
||||
- Add a **merge-model** capability (`none / git / native-CRDT`) so CRDT shards are
|
||||
handled correctly (UC-64) — a thirteenth capability spectrum.
|
||||
- Add **CRDT-replica** to the substrate/attachment taxonomy, and **P2P/no-central-
|
||||
endpoint** as a binding mode (UC-64/65).
|
||||
- **MCP as an integration surface** (Anytype) is worth noting — a shard may expose an
|
||||
AI-agent protocol shard-wiki could consume or coexist with.
|
||||
|
||||
---
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-63**. New UCs **UC-64, UC-65** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a CRDT-synced local-first shard** whose backend performs native conflict-free merge; respect CRDT semantics (no git merge), consume a replica/export, CRDT-log history (supplement) | **UC-64 (new)** |
|
||||
| **Attach a peer-to-peer / decentralized shard with no single canonical endpoint** (Anytype any-sync: replica or peer/backup node; E2EE) | **UC-65 (new)** |
|
||||
| Typed object graph / user-editable ontology + Notion-style DBs, local-first | **enriches UC-58** |
|
||||
| E2EE content opacity (Anytype) reinforced; self-host trust boundary | **enriches UC-61** |
|
||||
| CRDT update log ≠ portable git history → supplement | **enriches UC-36** |
|
||||
| One block set rendered as page / canvas / DB views (AFFiNE) | **enriches UC-47, UC-48** |
|
||||
| Edgeless canvas / board / objects = non-Markdown content | **enriches UC-55** |
|
||||
| Self-host sync server (AFFiNE/AppFlowy Cloud) + Anytype open API/MCP as endpoints | **enriches UC-57** |
|
||||
| Object/block IDs CRDT-assigned (fine-grained) | **enriches UC-51** |
|
||||
| Lossy Markdown export | links UC-59 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Merge-model capability** (`none / git-or-text / native-CRDT`) — proposed
|
||||
**thirteenth spectrum** (with Joplin's content-opacity twelfth). CRDT shards reject
|
||||
external merge; shard-wiki overlays must be CRDT-aware or remain projections. (T11.)
|
||||
- **Substrate taxonomy gains CRDT-replica**; **attach modes gain "local CRDT replica"
|
||||
and "P2P/no-central-endpoint"** beside file-store / in-engine-host / external-API.
|
||||
(T14.)
|
||||
- **Endpoint model** as a capability: single URL vs self-host server vs **P2P peer set**
|
||||
(Anytype) — affects how a binding is addressed and how freshness/sync work. (T14.)
|
||||
- **MCP / AI-agent protocol** as a possible shard integration surface (Anytype). Note
|
||||
for the extension-surface catalogue. (T14.)
|
||||
- These three plus Joplin and Logseq all argue the contract must treat **substrate as a
|
||||
spectrum that tools migrate along and that shard-wiki must not hard-code**.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions (for spec / workplans)
|
||||
|
||||
1. For a **CRDT shard** (UC-64), does shard-wiki embed a CRDT client (Yjs/Yrs) to hold a
|
||||
live replica, or only consume periodic **exports/snapshots**? The former gives
|
||||
live/writable; the latter is simpler but lossy on merge timing.
|
||||
2. How are **overlays** expressed against a CRDT shard — as CRDT ops (requires the
|
||||
client) or as out-of-band patches reconciled later (projection-only)?
|
||||
3. For **P2P shards** (UC-65), what is the "address" of the shard — a space ID + a peer/
|
||||
node, a local replica path, or an invite/key? How does this fit shard attachment
|
||||
config?
|
||||
4. Does shard-wiki ever hold **E2EE keys** (Anytype) to project content, or only ever
|
||||
treat such shards as opaque replicas/backups (UC-61)?
|
||||
5. Is the **self-host sync server** (AFFiNE/AppFlowy Cloud) attached as an external API
|
||||
(UC-57) or by running a co-located replica?
|
||||
|
||||
---
|
||||
|
||||
## 11. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Anytype any-sync — Protocol overview (https://tech.anytype.io/any-sync/overview) | Local-first/P2P/E2EE CRDT; sync/file/consensus/coordinator nodes; encrypted backup |
|
||||
| anyproto/any-sync (https://github.com/anyproto/any-sync) | Protocol scope; keys signed/verified; IPFS files |
|
||||
| Anytype Docs (https://doc.anytype.io/anytype-docs) | Spaces, object types + relations (ontology); open API + MCP |
|
||||
| toeverything/AFFiNE + DeepWiki (https://deepwiki.com/toeverything/AFFiNE) | BlockSuite (page/edgeless/DB); y-octo Yjs; OctoBase; one-data-many-views; self-host |
|
||||
| AFFiNE — interactive whiteboard / blog (https://affine.pro/blog/interactive-whiteboard-app) | Edgeless canvas; CRDT offline merge |
|
||||
| AppFlowy-IO/AppFlowy-Collab (https://github.com/AppFlowy-IO/AppFlowy-Collab) | Yrs CRDT; collab domain objects (documents/databases/folders); persistence |
|
||||
| AppFlowy Architecture Overview — DeepWiki (https://deepwiki.com/AppFlowy-IO/AppFlowy/2-architecture-overview) | Flutter/Rust layers; pluggable storage/sync (RocksDB/IndexedDB; Supabase/AppFlowy Cloud); local vs cloud trait |
|
||||
| AFFiNE vs AppFlowy vs Anytype (https://affine.pro/blog/affine-vs-appflowy-vs-anytype) | Cohort comparison framing |
|
||||
|
||||
Cross-references: `research/260614-notion-deep-dive/findings.md` (typed DB, hosted
|
||||
contrast), `research/260614-joplin-deep-dive/findings.md` (E2EE/content opacity, sync-
|
||||
daemon boundary), `research/260614-shard-spectrum-synthesis/findings.md` (the spectra
|
||||
this extends), `spec/UseCaseCatalog.md` (UC-36, UC-47, UC-48, UC-51, UC-55, UC-57,
|
||||
UC-58, UC-61), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
|
||||
|
||||
---
|
||||
|
||||
## 12. Traceability
|
||||
|
||||
- New UCs: **UC-64, UC-65** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-36, UC-47, UC-48, UC-51, UC-55, UC-57, UC-58, UC-61** (links UC-59).
|
||||
- Architecture (no UC): **merge-model** capability (proposed thirteenth spectrum);
|
||||
CRDT-replica substrate + P2P/no-central-endpoint attach mode; endpoint-model capability;
|
||||
MCP integration surface → `SHARD-WP-0002` (T11, T14).
|
||||
- Boundary recorded: Anytype/AFFiNE/AppFlowy are **CRDT local-first candidate shards**
|
||||
(attach a replica/projection as pages; respect native CRDT merge; never re-drive their
|
||||
sync; P2P/E2EE shards are replica-only/opaque) — not substrates, not the federation
|
||||
layer (INTENT not-a-sync-daemon, graceful degradation, no silent mutation).
|
||||
</content>
|
||||
50
research/260614-logseq-deep-dive/README.md
Normal file
50
research/260614-logseq-deep-dive/README.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# 260614 — Logseq deep dive (block-graph semantics on plain Markdown files)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Logseq** — the open-source, local-first outliner — read through
|
||||
shard-wiki's lens. Logseq occupies the point the other modern tools leave empty:
|
||||
**block-graph semantics (UUID-addressable, embeddable, queryable blocks) stored as plain
|
||||
Markdown/Org files on disk**, with a DataScript graph **derived** from those files. It is
|
||||
the bridge between **Roam** (block-graph, client-DB) and **Obsidian** (file-over-app,
|
||||
page-level), and it resolves a tension the synthesis flagged: block-level addressing
|
||||
*that is also git-diffable text*.
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — files canonical (`pages/`, `journals/` MD/Org); a DataScript graph
|
||||
**derived** via an `mldoc` AST parse; a DB Worker now manages **both DataScript and
|
||||
SQLite** (the file→DB migration)
|
||||
- **Block-graph in the file text** — block IDs as in-file `id:: <uuid>` properties,
|
||||
`((uuid))` refs, `key:: value` properties, `{{embed}}` transclusion, the outline tree
|
||||
(indent/zoom/move-subtree) — all git-diffable
|
||||
- **Queries** — `{{query}}` + advanced **Datalog** over the derived graph
|
||||
(`logseq.DB.datascriptQuery`)
|
||||
- **Extension** — plugin API (`logseq.App/Editor/DB/Git/UI/Assets/FileStorage`),
|
||||
marketplace (~486 plugins); dual-attachable (file-store direct *or* in-app plugin)
|
||||
- **Trajectory** — migrating from Markdown-files to a SQLite "DB graph" (a live UC-43)
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, in-file block-graph, Datalog queries, plugin API, file→DB migration, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-62 attach a block-graph-on-plain-files shard with in-file block IDs/properties +
|
||||
outline tree + derivable query index; UC-63 serve structured queries over a file-backed
|
||||
shard via a derived index the orchestrator/adapter builds); UC-32/34/43/50/51/52/55
|
||||
enriched. Logged for `SHARD-WP-0002` (T11/T14/T16): a Logseq **format profile** for
|
||||
file-store adapters, a **derived-query-index** capability, **substrate-migration
|
||||
tolerance**, and **in-file block addressing** as the concrete T16 span-address target.
|
||||
|
||||
**Key takeaway recorded:** Logseq resolves the addressing-spectrum tension — block-level
|
||||
addressing **and** git-diffable in-file text (`id::`) — and proves a file-backed shard
|
||||
can support rich Datalog queries via a **derived** index (files canonical, graph
|
||||
derived). **Boundary:** one block-graph-on-files candidate shard (the addressing sweet
|
||||
spot), best attached file-store-direct with a format profile; not the federation layer;
|
||||
substrate is migrating file→SQLite.
|
||||
</content>
|
||||
246
research/260614-logseq-deep-dive/findings.md
Normal file
246
research/260614-logseq-deep-dive/findings.md
Normal file
@@ -0,0 +1,246 @@
|
||||
# Findings — Logseq: block-graph semantics on plain Markdown files
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — an open-source, local-first outliner; a
|
||||
*candidate shard* occupying the point none of the prior tools do: **block-graph + file-
|
||||
over-app**
|
||||
Lens: shard-wiki — the convergence of block-level addressing with git-diffable files, a
|
||||
derived query index over canonical files, and a live file→DB storage migration
|
||||
|
||||
> Why Logseq matters distinctly. The modern-tool dives staked out the corners: **Roam**
|
||||
> = block-graph but client-DB / in-app-API / store-minted UUID; **Obsidian** =
|
||||
> file-over-app but page-level / path-addressed (block `^id` opt-in); **Notion** =
|
||||
> block-graph but closed hosted DB / external-API. **Logseq sits in the middle they
|
||||
> leave empty: block-graph *semantics* (UUID-addressable, embeddable, queryable blocks)
|
||||
> stored as *plain Markdown/Org files on disk you own*, with a DataScript index
|
||||
> **derived** from those files.** It is the system that proves you do not have to choose
|
||||
> between Roam's fine-grained addressing and Obsidian's file sovereignty — you can have
|
||||
> block-level addressing *that is also git-diffable text*. That resolves a tension the
|
||||
> synthesis flagged (addressing spectrum) and is the reason it earns a memo rather than
|
||||
> a footnote.
|
||||
|
||||
Contrast set: Roam (block-DB, in-app), Obsidian (file, page-level), Notion (hosted DB),
|
||||
Joplin (SQLite-local, files-on-sync). Logseq = **block-graph, files-canonical,
|
||||
index-derived** — and now *also* migrating toward SQLite (§5), a live UC-43 case.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture — files canonical, DataScript derived
|
||||
|
||||
- **Storage:** plain-text **Markdown or Org-mode files on disk** — `pages/<Page>.md` and
|
||||
`journals/<date>.md` — fully usable even if the app disappears (local-first, open
|
||||
source, ClojureScript).
|
||||
- **Index:** a **DataScript** in-memory graph database, **derived** from the files: an
|
||||
`mldoc` parser turns Markdown into an AST, extracts references/properties, and
|
||||
transforms it into **DataScript entities** with tree-position attributes. Files are the
|
||||
source of truth; the graph DB is a rebuildable projection. (In current versions a **DB
|
||||
Worker** manages **both DataScript and SQLite** connections — see §5.)
|
||||
- This is the **files-canonical / index-derived** architecture (Obsidian's MetadataCache,
|
||||
Git's working tree) — but here the derived index is a *full Datalog-queryable graph*,
|
||||
not just a metadata cache. Logseq is the strongest evidence that **a file-backed shard
|
||||
can support rich structured queries via a derived index** (UC-52, UC-63).
|
||||
|
||||
---
|
||||
|
||||
## 2. The block-graph, in the file text
|
||||
|
||||
Everything is an **atomic block** (a bullet), individually referenceable, embeddable, and
|
||||
queryable — Roam's model — but the addressing lives **in the Markdown**:
|
||||
|
||||
- **Block IDs are in-file properties.** When a block is referenced, Logseq writes an
|
||||
`id:: <uuid>` property line into the Markdown. So the block's stable address is
|
||||
**git-diffable text that survives a file copy**, not a DB-minted hidden key. This is the
|
||||
**sweet spot of the addressing spectrum**: block-level (like Roam) *and* in-file/
|
||||
portable (like Obsidian's `^id`), without choosing.
|
||||
- **References:** `[[page]]`, **`((block-uuid))`**, `#tag` — all extracted into the graph;
|
||||
block embeds `{{embed ((uuid))}}` are transclusion at block granularity (UC-32).
|
||||
- **Properties:** `key:: value` lines attach typed metadata to blocks (block properties)
|
||||
and pages (first-block/page properties) — **git-diffable structured data at block
|
||||
granularity** (UC-34), queried by Datalog.
|
||||
- **Outline tree:** a page is a **nested tree of blocks** (indent = structure), with
|
||||
outliner operations — indent/outdent, move subtree, **zoom/narrow** into a block. The
|
||||
page is not flat prose; it is an addressable, reorderable block tree (UC-63).
|
||||
|
||||
---
|
||||
|
||||
## 3. Queries — Datalog over the derived graph
|
||||
|
||||
Logseq exposes **`{{query}}`** (simple) and **advanced Datalog queries** over the
|
||||
DataScript graph (and via the plugin API, `logseq.DB.datascriptQuery`). Because the graph
|
||||
is derived from files, **query-defined pages** (UC-54) and structured aggregation (tasks,
|
||||
tagged blocks) run over a *file-backed* store. Key lesson: query capability here is
|
||||
**neither native-DB (Notion) nor a third-party plugin (Obsidian Dataview)** — it is
|
||||
**built into the tool over its own files**, demonstrating that *a derived query index is a
|
||||
viable adapter/orchestrator capability for file shards* (UC-63).
|
||||
|
||||
---
|
||||
|
||||
## 4. Extension surface — plugin API + marketplace
|
||||
|
||||
Logseq has a JS **plugin API** (marketplace ~486 plugins, `logseq/marketplace`):
|
||||
|
||||
- `logseq.Editor` — block/page CRUD, insert/move/update/delete, get current block/page.
|
||||
- `logseq.DB` — **`datascriptQuery`** (Datalog), plus DB change subscriptions.
|
||||
- `logseq.App` — app-level ops/events; `logseq.UI`; `logseq.Git` (git ops on the graph);
|
||||
`logseq.Assets`; `logseq.FileStorage`.
|
||||
- Slash commands, block/page context menus, `provideModel`/`provideStyle`, hooks/events.
|
||||
|
||||
So, like Obsidian, Logseq is **dual-attachable**: (a) **file-store direct** — read the
|
||||
`pages/`+`journals/` Markdown with a **Logseq format profile** (parse `id::`, `((uuid))`,
|
||||
`key::`, the outline tree); (b) **in-app plugin host** — `logseq.Editor` write +
|
||||
`logseq.DB` query + change events for live write-through. A notable extra: a built-in
|
||||
**`logseq.Git`** surface — the tool treats git as a first-class companion to the file
|
||||
graph (validating the coordination journal).
|
||||
|
||||
---
|
||||
|
||||
## 5. The file→DB migration — a live UC-43
|
||||
|
||||
Logseq is **migrating its storage model from Markdown-files to a SQLite "DB graph"**
|
||||
(the DB Worker already manages SQLite alongside DataScript; the plugin API has a distinct
|
||||
"DB graph" mode with tags/classes/typed properties). This is a real-world instance of
|
||||
**UC-43 (backend-swap under stable identity)**: the *same tool and graph identity* moving
|
||||
from a file substrate to a DB substrate, trading git-diffability for richer typed
|
||||
structure (toward the Notion/XWiki end). For shard-wiki it is both a caution (a shard's
|
||||
substrate can change under it) and a confirmation that the **addressing/structure/history
|
||||
spectra** are trajectories tools actually travel — an adapter keyed to a fixed substrate
|
||||
will break.
|
||||
|
||||
---
|
||||
|
||||
## 6. Logseq as a shard — capability profile
|
||||
|
||||
| Capability | Logseq (MD-file graph) | Notes for the adapter contract |
|
||||
|------------|------------------------|--------------------------------|
|
||||
| Read | **yes** | file-store direct (pages/journals MD) or `logseq.Editor`/`logseq.DB` in-app |
|
||||
| Write | **yes** | direct file write (format-aware) or plugin; block-level |
|
||||
| Write granularity | **per-block** (in a per-file page) | finer than Obsidian (per-file), like Roam — but on files |
|
||||
| Identity / addressing | **in-file block `id:: uuid` + `((uuid))`** | block-level **and** git-diffable — the addressing sweet spot (UC-51) |
|
||||
| Structure | `key:: value` block/page properties; outline tree | git-diffable structured data at block granularity (UC-34) |
|
||||
| History | none native; **`logseq.Git`** + files = git-native | git-friendly out of the box (adopt, not supplement) |
|
||||
| Native query | **Datalog over derived DataScript** | derived index over files → delegate or rebuild (UC-52, UC-63) |
|
||||
| Transclusion | **block embeds `{{embed ((uuid))}}`** | in-file, addressable (UC-32) |
|
||||
| Backlinks | linked + unlinked references | derived (UC-05/18) |
|
||||
| Content types | Markdown/Org + **whiteboards** (tldraw JSON) | non-Markdown spatial content (UC-55) |
|
||||
| Substrate | **files now, SQLite "DB graph" emerging** | live backend-swap (UC-43) |
|
||||
| Attach modes | file-store direct (format profile) · in-app plugin | dual, per-binding (UC-62) |
|
||||
|
||||
Verdict: **the most shard-wiki-friendly block tool** — block-graph power with file-over-
|
||||
app sovereignty and git-diffable addressing/structure. Best attach: **file-store direct
|
||||
with a Logseq format profile** (offline, git-native), with the plugin for live write-
|
||||
through. Watch the **DB-graph migration** (UC-43).
|
||||
|
||||
---
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
|
||||
- **Resolves the addressing tension.** Block-level addressing *can* be git-diffable
|
||||
in-file (`id::`), not forced to choose Roam-DB vs Obsidian-page (UC-51, synthesis §2).
|
||||
- **Confirms files-canonical / index-derived at full power** — a Datalog graph derived
|
||||
from files, not just a metadata cache (UC-52, UC-63).
|
||||
- **Structure as git-diffable text** (`key::`) reinforces "prefer in-text structure"
|
||||
(UC-34, synthesis through-line).
|
||||
- **Outliner block tree** validates the page-as-addressable-block-tree demand (UC-50,
|
||||
UC-63) on a *file* backend.
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **One graph = one shard.** Local-first single graph; not the federation layer.
|
||||
2. **The MD files carry Logseq-specific syntax** (`id::`, `((uuid))`, `key::`, outline
|
||||
semantics) — a **format profile**, not plain CommonMark. A naïve Markdown reader will
|
||||
mangle block IDs/properties (cf. UC-42; lighter than Notion's lossy case).
|
||||
3. **The substrate is moving (file→SQLite).** Don't hard-code the file model; gate on the
|
||||
substrate capability and tolerate the swap (UC-43).
|
||||
4. **Whiteboards are not Markdown** — typed/opaque assets, not flattened (UC-55).
|
||||
|
||||
### 7.3 What Logseq teaches that shard-wiki should keep
|
||||
|
||||
- **In-file block addressing is the target shape** for a portable span address where the
|
||||
backend cooperates — adopt `id::`-style schemes; they are git-diffable and survive copy.
|
||||
- **A derived query index over files is a first-class capability** — shard-wiki can build
|
||||
one over a file shard (UC-63) when the backend exposes none, or delegate when it does.
|
||||
- **Expect substrate migration** — bind to capabilities, not to "it's files."
|
||||
|
||||
---
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-61**. New UCs **UC-62, UC-63** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a block-graph-on-plain-files shard** (Logseq-style): file-over-app Markdown carrying in-file block IDs (`id::`), block refs, and `key::` properties, with the outline tree and a derivable query index | **UC-62 (new)** |
|
||||
| **Serve structured queries over a file-backed shard via a derived index** the orchestrator/adapter builds (Logseq DataScript-over-files) when the backend exposes none | **UC-63 (new)** |
|
||||
| In-file block `id::` = block-level **and** git-diffable addressing (the spectrum sweet spot) | **enriches UC-51** |
|
||||
| Live **file→SQLite substrate migration** under stable graph identity | **enriches UC-43** |
|
||||
| Block-graph that is **files, not a DB** — the file-store variant of the block tool family | **enriches UC-50** |
|
||||
| Datalog over a **derived** index built from files | **enriches UC-52** |
|
||||
| `key:: value` block/page properties in-text | **enriches UC-34** |
|
||||
| Whiteboards (tldraw JSON) = non-Markdown content | **enriches UC-55** |
|
||||
| Block embeds `{{embed ((uuid))}}` = in-file transclusion | links UC-32 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Format-aware file-store profiles** (already flagged by Joplin, UC-60) gain a strong
|
||||
case here: a Logseq profile parses `id::`/`((uuid))`/`key::`/outline from otherwise-
|
||||
plain Markdown. The contract should let a file-store adapter declare its **format
|
||||
profile** (plain MD / Obsidian / Logseq / Joplin-item / Foswiki-PlainFile). (T11/T14.)
|
||||
- **Derived-query-index as an orchestrator capability** (UC-63): when a file shard has no
|
||||
native query engine, build a DataScript-like index over the projection; when it does
|
||||
(Roam/Notion/XWiki), delegate (UC-52). Both belong in T16's navigation layer + T11.
|
||||
- **Substrate-migration tolerance** (UC-43, Logseq file→SQLite): T14 binding should treat
|
||||
substrate as a capability that can change under a live attachment, preserving identity.
|
||||
- **In-file block addressing** is the concrete realization of T16's span-address thread —
|
||||
prefer `id::`-style git-diffable IDs over DB-minted where the backend allows.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions (for spec / workplans)
|
||||
|
||||
1. When the same tool offers **file** and **DB** substrates (Logseq now), does shard-wiki
|
||||
prefer the file graph (git-diffable, UC-62) or the DB graph (richer typed structure),
|
||||
and can one binding follow the migration (UC-43)?
|
||||
2. Is the **derived query index** (UC-63) built by the **adapter** (per-shard) or the
|
||||
**core orchestrator** (over the union), and is it persisted or rebuilt?
|
||||
3. How much **Logseq outline semantics** (zoom, subtree move) must shard-wiki preserve vs.
|
||||
present as a flat page (UC-63 vs. Markdown-first page model)?
|
||||
4. Does the **Logseq format profile** round-trip overlays (write `id::`/`key::` back) or
|
||||
only read them? (cf. UC-42 round-trip question.)
|
||||
|
||||
---
|
||||
|
||||
## 11. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| DeepWiki — logseq/logseq (https://deepwiki.com/logseq/logseq) | DataScript+Datalog graph, mldoc AST parse, block entities/tree, DB Worker managing DataScript **and** SQLite |
|
||||
| logseq/docs (https://deepwiki.com/logseq/docs) | Plain-text MD/Org files; pages/journals; references ([[ ]], (( )), #tag); properties |
|
||||
| Logseq Plugin API docs (https://plugins-doc.logseq.com/) | `logseq.App/Editor/DB/Git/UI/Assets/FileStorage`; `DB.datascriptQuery` |
|
||||
| logseq/marketplace (https://github.com/logseq/marketplace) | Plugin distribution; ~486 plugins |
|
||||
| kerim/logseq-db-plugin-api-skill (https://github.com/kerim/logseq-db-plugin-api-skill) | DB-graph version: tags/classes, typed properties, file→DB migration |
|
||||
| pangea.app glossary — Logseq (https://pangea.app/glossary/logseq) | Local-first framing, outliner, plain-text control |
|
||||
|
||||
Cross-references: `research/260614-roam-deep-dive/findings.md` (block-DB contrast),
|
||||
`research/260614-obsidian-deep-dive/findings.md` (file-over-app, derived index),
|
||||
`research/260614-joplin-deep-dive/findings.md` (format-aware file-store profiles),
|
||||
`research/260614-shard-spectrum-synthesis/findings.md` (addressing spectrum this
|
||||
resolves), `spec/UseCaseCatalog.md` (UC-32, UC-34, UC-43, UC-50, UC-51, UC-52, UC-55),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14, T16).
|
||||
|
||||
---
|
||||
|
||||
## 12. Traceability
|
||||
|
||||
- New UCs: **UC-62, UC-63** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-32, UC-34, UC-43, UC-50, UC-51, UC-52, UC-55**.
|
||||
- Architecture (no UC): format-aware file-store profiles (Logseq profile); derived-query-
|
||||
index capability; substrate-migration tolerance; in-file block addressing as the T16
|
||||
span-address target → `SHARD-WP-0002` (T11, T14, T16).
|
||||
- Boundary recorded: Logseq is **one block-graph-on-files candidate shard** (the
|
||||
addressing sweet spot), best attached file-store-direct with a format profile; not the
|
||||
federation layer; substrate is migrating file→SQLite (UC-43).
|
||||
</content>
|
||||
32
research/260614-mathematica-deep-dive/README.md
Normal file
32
research/260614-mathematica-deep-dive/README.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# 260614 — Mathematica Notebooks deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T2**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Wolfram Mathematica Notebooks** — the **original computational notebook**
|
||||
(1988), the ancestor Jupyter descends from. The `.nb` document **is itself a Wolfram Language
|
||||
expression** (`Notebook[{Cell[…], …}]`) with **nested cell groups**, kernel evaluation
|
||||
(`In`/`Out`), **structured (symbolic/graphics) results**, live `Manipulate`/`Dynamic`
|
||||
widgets, and CDF as a reduced-runtime distribution projection.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Confirms the **notebook page shape (UC-84) is a genus**, not a Jupyter quirk: cells +
|
||||
cached outputs + fragile counter provenance + kernel-gated re-execution predate `.ipynb`.
|
||||
- Two refinements: notebooks have **nestable cell groups (an outline tree)**, not just an
|
||||
ordered list; outputs can be **structured re-evaluable values**, not only MIME blobs →
|
||||
adds a "structured re-evaluable value" point to the content-opacity spectrum.
|
||||
- `Manipulate`/`Dynamic` join the **static-projection-impossible** end (snapshot-only) —
|
||||
foreshadows Strudel (T5).
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (lineage/reinforcement). Reinforces **UC-84**; enriches **UC-54, UC-55**;
|
||||
links UC-34, UC-83.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | `.nb` expression/cell model, evaluation & provenance, CDF, capability delta vs Jupyter, INTENT mapping, UC disposition (enrichment-only), architecture notes |
|
||||
139
research/260614-mathematica-deep-dive/findings.md
Normal file
139
research/260614-mathematica-deep-dive/findings.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# Mathematica Notebooks — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T2 · **Subject:** Wolfram Mathematica
|
||||
Notebooks — the `.nb` format, the cell/expression model, symbolic evaluation.
|
||||
|
||||
## Why this dive
|
||||
|
||||
Mathematica (1988) is the **original computational notebook** — the ancestor Jupyter (T3)
|
||||
descends from. This dive checks whether the notebook page-model conclusions from UC-84 hold
|
||||
at the genus level or need extension. It is **medium priority / lineage**: a candidate
|
||||
*content type* like Jupyter, but closed/proprietary, so the expected yield is **reinforcement
|
||||
of UC-84** plus a couple of distinct wrinkles (the document *is itself* a Wolfram-language
|
||||
expression; symbolic, not just textual, results).
|
||||
|
||||
## 1. The `.nb` document model
|
||||
|
||||
A Mathematica notebook is itself a **Wolfram Language expression** — `Notebook[{Cell[...],
|
||||
Cell[...], ...}, opts]`. So the document format and the language are the *same* substrate:
|
||||
|
||||
- **Cells** are typed: `Input`, `Output`, `Text`, `Title`/`Section` (structure), `Code`,
|
||||
with **cell groups** nesting into an **outline tree** (the document has real hierarchy, not
|
||||
just a flat list like `.ipynb`).
|
||||
- An **`Input` cell** holds an expression; evaluating it produces a linked **`Output` cell`**
|
||||
containing the **result expression** (symbolic, graphical via `Graphics[...]`, or
|
||||
typeset). The result is a **first-class expression**, not a MIME blob — it can be
|
||||
re-evaluated, edited, transcluded.
|
||||
- The whole `.nb` is **plain-text-serializable** (it's an expression) but verbose and
|
||||
proprietary in conventions; output cells are **cached results** stored in the file.
|
||||
|
||||
So like Jupyter: **source + cached computed output fused in one document**, with **out-of-
|
||||
band reproducibility** (kernel + package/version state). Unlike Jupyter: **nested cell
|
||||
groups (a tree)** and **results that are structured Wolfram expressions** rather than MIME
|
||||
bundles.
|
||||
|
||||
## 2. Evaluation, provenance, reproducibility
|
||||
|
||||
- A **kernel** evaluates input cells; `In[n]`/`Out[n]` counters mirror Jupyter's
|
||||
`execution_count` and carry the **same fragility** (out-of-order evaluation, hidden
|
||||
symbol/global state, kernel-version dependence). No environment capture in the file.
|
||||
- **Dynamic/interactive output** (`Manipulate`, `Dynamic`) embeds **live interactive
|
||||
widgets** whose state is computed on view — these have **no faithful static form** beyond a
|
||||
snapshot frame (echoes Strudel T5's "live, time/interaction-based content" limit).
|
||||
- **CDF (Computable Document Format)** is Wolfram's *projection-for-distribution*: a notebook
|
||||
rendered with a free runtime so readers can interact without a full license — a
|
||||
derivation-projection (T1) with a reduced-capability viewer.
|
||||
|
||||
## 3. Capability profile (delta vs Jupyter UC-84)
|
||||
|
||||
| Dimension | Mathematica `.nb` (delta from Jupyter) |
|
||||
|-----------|----------------------------------------|
|
||||
| Structure | **nested cell groups → outline tree** (richer than `.ipynb`'s flat cell list) |
|
||||
| Output type | **structured Wolfram expressions** (symbolic/graphics), not MIME blobs |
|
||||
| Document = language | the `.nb` **is** a Wolfram expression (format ≡ language) |
|
||||
| Liveness | `Manipulate`/`Dynamic` = **interactive widgets**, snapshot-only when static |
|
||||
| Opacity | proprietary serialization; results re-evaluable only with a Wolfram kernel |
|
||||
| Projection-for-distribution | **CDF** = reduced-runtime interactive projection |
|
||||
| Otherwise | same as UC-84: cells, cached outputs, fragile `In/Out` provenance, kernel-gated |
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Reinforcements (mostly confirms UC-84)
|
||||
|
||||
- **Notebook page shape (UC-84) is a genus, not a Jupyter quirk.** Mathematica predates and
|
||||
matches it: cells + cached computed outputs + fragile counter provenance + kernel-gated
|
||||
re-execution. Confirms the page model should carry a **notebook shape** generically (T12),
|
||||
not a `.ipynb`-specific one.
|
||||
- **Outputs as derivation-projection snapshots (UC-83/84).** Cached `Output` cells are
|
||||
snapshots; honest treatment marks them "evaluated run N, kernel/env unguaranteed."
|
||||
- **Derivation-projection for distribution (T1).** CDF is a clean "reduced-capability
|
||||
interactive projection of a source" — a real-world instance of degrade-by-capability.
|
||||
|
||||
### Distinct wrinkles (extend the notes, not new UCs)
|
||||
|
||||
- **Nested cell-group outline** — the notebook page model should allow **hierarchical cell
|
||||
grouping**, not just an ordered list (generalize UC-84's "ordered cells" to "ordered/
|
||||
*nestable* cells"). Feeds T12.
|
||||
- **Structured (non-MIME) results** — outputs can be **typed structured values** (symbolic
|
||||
expressions), not only MIME blobs; reinforces UC-55's "typed asset" reading over "opaque
|
||||
blob," and links the typed-record page model (UC-34) — the content-opacity spectrum needs a
|
||||
"structured re-evaluable value" point, not just text↔blob.
|
||||
- **Format ≡ language** — a curiosity, not actionable for us beyond noting that some shards'
|
||||
document format is *the same artifact* as their computation (don't assume doc/code split).
|
||||
- **Live interactive widgets** — `Manipulate`/`Dynamic` join Strudel (T5) at the
|
||||
**static-projection-impossible** end: capture a snapshot frame, never imply interactivity.
|
||||
|
||||
### Boundaries
|
||||
|
||||
- Proprietary + kernel-gated → default **read/projection/snapshot**; attach the `.nb`
|
||||
(or an exported form), present cached outputs as snapshots, offer a static/CDF projection;
|
||||
**no kernel host** (same rule as Jupyter UC-84, GT T7).
|
||||
|
||||
## 5. UC disposition (enrichment-only — no new UC)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC |
|
||||
|------------------------|------------|
|
||||
| Cells + cached computed outputs + fragile In/Out provenance; kernel-gated (§1, §2) | UC-84 (reinforced) |
|
||||
| Nested cell groups → outline tree (richer than flat `.ipynb`) (§1) | UC-84 (enriched: nestable cells); links UC-34 |
|
||||
| Output = structured re-evaluable Wolfram expression, not MIME blob (§1) | UC-55 (enriched: structured value point on opacity spectrum) |
|
||||
| Input cell = computation-defined content (§1) | UC-54 (enriched) |
|
||||
| `Manipulate`/`Dynamic` interactive output = snapshot-only (§2) | links UC-55, foreshadows T5 |
|
||||
| CDF = reduced-runtime interactive distribution projection (§2) | links UC-83 (derivation-projection) |
|
||||
|
||||
Mathematica is a **lineage/reinforcement** dive — it **adds no new UC**, confirming UC-84's
|
||||
notebook page shape as a genus and contributing two refinements (nestable cells; a
|
||||
**structured re-evaluable value** point on the content-opacity spectrum).
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12 (page model):** generalize the notebook shape from "ordered cells" (UC-84) to
|
||||
**ordered/nestable cell groups (an outline tree)**; allow code-cell outputs to be **typed
|
||||
structured values**, not only MIME blobs.
|
||||
- **T11/T15 (content opacity / fidelity):** add a **"structured re-evaluable value"** point
|
||||
to the content-opacity spectrum between transparent-text and opaque-blob (Wolfram
|
||||
expression, symbolic result) — relevant to how outputs are stored/surfaced.
|
||||
- **T16 (projection):** CDF is a **reduced-capability interactive projection**; with
|
||||
`Manipulate`/`Dynamic`, static projection is a **snapshot frame only** (join the
|
||||
live-content limit recorded for Strudel T5).
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. Is a **structured re-evaluable result** (Wolfram expression) modeled as a typed value in
|
||||
the page model, or stored opaquely with provenance like other computed outputs? (Ties
|
||||
UC-55 open-Q #10 and UC-84 Q3.)
|
||||
2. Do interactive outputs (`Manipulate`, `Dynamic`, and Jupyter widgets) deserve a shared
|
||||
**"interactive, snapshot-only" content marker** in the contract? (Recurs at T4/T5.)
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- Wolfram documentation: notebook format (`Notebook`/`Cell` expressions), cell types &
|
||||
groups, evaluation (`In`/`Out`), `Manipulate`/`Dynamic`, CDF.
|
||||
- prior: `research/260614-jupyter-deep-dive/` (UC-84 notebook shape; the descendant);
|
||||
`research/260614-literate-programming-deep-dive/` (derivation-projection, UC-83).
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
**No new UC** (lineage/reinforcement). Reinforced: UC-84; enriched: UC-54, UC-55; links
|
||||
UC-34, UC-83. Architecture cross-refs: SHARD-WP-0002 T12 (nestable cell-group outline; typed
|
||||
structured outputs), T11/T15 (structured-re-evaluable-value point on the opacity spectrum),
|
||||
T16 (CDF reduced-capability projection; interactive = snapshot-only).
|
||||
15
research/260614-mojomojo-deep-dive/README.md
Normal file
15
research/260614-mojomojo-deep-dive/README.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# 260614 — MojoMojo deep dive
|
||||
|
||||
Deep dive on **MojoMojo**: a **Perl Catalyst / DBIx::Class** DB-backed wiki — hierarchical
|
||||
pages, attachments, inline (AJAX) editing, Markdown content, and **page history in
|
||||
relational version tables**. The classic **MVC DB-backed** contrast to the file-store
|
||||
classics: no file store, no real content API → attach by reading the **relational store
|
||||
directly**.
|
||||
|
||||
- `findings.md` — architecture, the DB schema shape, capability profile, INTENT mapping, UC
|
||||
seed (UC-81), architecture notes for SHARD-WP-0002, open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-81 (attach a **DB-backed wiki with no file store/API** by reading its
|
||||
relational store directly — pages + version tables — and importing DB-resident history to the
|
||||
journal). Enriched UC-02/40/36/34. Feeds SHARD-WP-0002 T13 (history portability), T14
|
||||
(direct-DB binding).
|
||||
133
research/260614-mojomojo-deep-dive/findings.md
Normal file
133
research/260614-mojomojo-deep-dive/findings.md
Normal file
@@ -0,0 +1,133 @@
|
||||
# MojoMojo — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T8 · **Subject:** MojoMojo, a Perl
|
||||
Catalyst wiki/CMS.
|
||||
|
||||
## Why this dive
|
||||
|
||||
The file-store classics (TWiki, Foswiki, Oddmuse, UseMod) keep pages as files; the modern
|
||||
SaaS keep them behind APIs. MojoMojo is the **classic relational-DB-backed** wiki — a
|
||||
Catalyst MVC app over **DBIx::Class** with pages and their history in **SQL tables**, and
|
||||
**no file store and no first-class content API**. It anchors the *"attach by reading the
|
||||
database directly"* hard case the adapter contract must account for (T13/T14).
|
||||
|
||||
## 1. Architecture
|
||||
|
||||
- **Stack:** Perl **Catalyst** (MVC web framework) + **DBIx::Class** (ORM) over a relational
|
||||
DB (SQLite / PostgreSQL / MySQL). Templating via Template Toolkit.
|
||||
- **Content:** **Markdown** (Text::MultiMarkdown) is the page markup — so the *body* is
|
||||
Markdown, but it lives **in a DB column**, not a file.
|
||||
- **Pages are hierarchical:** a **path tree** (`/parent/child`) modeled as rows with
|
||||
parent/lineage relations — structure is relational, not directory-based.
|
||||
- **Versioning:** each page edit creates a **new version row** (a `page_version`-style
|
||||
table) — full revision history lives in **DB version tables**, with author/timestamp.
|
||||
- **Features:** inline **AJAX editing**, **attachments** (stored as DB rows / blobs +
|
||||
metadata), diffs, RSS feeds, full-text search, per-page permissions.
|
||||
|
||||
## 2. The attach problem — DB or nothing
|
||||
|
||||
MojoMojo exposes its content through the **web app** (HTML) and the **database**; there is
|
||||
**no clean REST/GraphQL content API** and **no file store**. So a shard adapter has two
|
||||
realistic paths:
|
||||
|
||||
1. **Direct relational read** (preferred): read the `page` + `page_version` (+ `content`,
|
||||
`attachment`) tables via DBIx::Class schema — pages, the path tree, and **full history**
|
||||
are all there, importable to the coordination journal (UC-41-style history import, but
|
||||
from **DB version rows** rather than RCS/git).
|
||||
2. **HTML scrape** (fallback): parse rendered pages — lossy, last resort.
|
||||
|
||||
This makes MojoMojo the **direct-DB-read** binding archetype: the canonical store is a
|
||||
relational schema, and the adapter's job is to map that schema to the wiki page model +
|
||||
journal.
|
||||
|
||||
## 3. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | MojoMojo |
|
||||
|--------------------------------|----------|
|
||||
| Attachment mode | **direct DB read** (relational); HTML-scrape fallback; no file store, no API |
|
||||
| Addressing granularity | page (row); path tree via lineage rows |
|
||||
| Content identity | DB page id; path as human key |
|
||||
| Identity vs placement | row id vs path lineage (separable) |
|
||||
| Structure | **relational**: page rows + parent/lineage; attachments as rows |
|
||||
| History | **DB version tables** (per-edit version rows, author/timestamp) |
|
||||
| Merge model | app-level last-writer; DB transactions |
|
||||
| Native query | SQL over the schema (not a wiki query language) |
|
||||
| Translation | **Markdown body in a DB column** — minimal translation, but extraction needed |
|
||||
| Write granularity | page (row) per save |
|
||||
| Operational envelope | a Perl app + its DB; direct DB access needs credentials |
|
||||
| Access grant | per-page permissions in DB; app auth |
|
||||
| Content opacity | transparent if you can read the DB |
|
||||
| Provenance | author/timestamp on version rows |
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Backend-neutral page model**: the body is **Markdown** — once extracted from the DB
|
||||
column it maps directly; the adapter's work is **schema→page-model**, not format
|
||||
translation.
|
||||
- **History portability** (T13): DB **version rows** are a third history-source shape beside
|
||||
git commits and RCS files — importable to the journal as discrete revisions with
|
||||
author/timestamp.
|
||||
- **Graceful degradation**: even with only DB read (no API), MojoMojo is a usable
|
||||
read/projection/backup shard; with DB write it could be write-through, but carefully
|
||||
(app invariants).
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- **No file store, no API** ⇒ the **direct-DB-read** binding is a first-class attach mode the
|
||||
contract must name (alongside file-store, in-engine host, external-API, CRDT, P2P) — or a
|
||||
sub-mode of "external store" where the medium is **a relational schema** (T14). Reading a
|
||||
third-party app's DB is **coupling to its schema** (versioned, may drift across MojoMojo
|
||||
versions) — a stated risk (UC-43 backend-swap analogue at the schema level).
|
||||
- **Writing by direct DB** risks violating app invariants (lineage, version counters,
|
||||
search index) — default to **read/projection/overlay**; write-through only with the app's
|
||||
cooperation.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Direct-DB-read as a named binding** for DB-backed engines with no file/API (UC-81),
|
||||
mapping a **relational schema → wiki page model + journal**.
|
||||
2. **DB version rows as a history source** for the journal (T13), beside git and RCS.
|
||||
3. **Schema-coupling caution** — treat the schema as a versioned interface that can drift
|
||||
(relates UC-43).
|
||||
|
||||
## 5. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-81 | Attach a **DB-backed wiki with no file store / no API** (MojoMojo) by reading its **relational store directly** (page + version tables), mapping schema → page model and **importing DB-resident history** to the journal | **new** |
|
||||
| — | DB attach vs file attach | enrich **UC-02** / **UC-40** |
|
||||
| — | DB version-table history import | enrich **UC-36** |
|
||||
| — | relational page rows / lineage as structure | enrich **UC-34** |
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T14 (binding):** add **direct relational read** as a binding (or external-store sub-mode
|
||||
whose medium is a SQL schema) for DB-backed engines lacking a file store or API; HTML
|
||||
scrape is the lossy fallback. Schema is a **versioned coupling** (drift risk, UC-43).
|
||||
- **T13 (history portability):** **DB version rows** = a history source alongside git commits
|
||||
and RCS revisions — import as discrete journal entries (author/timestamp).
|
||||
- **T11 (capability):** "has-file-store" / "has-API" are **absent** here; "has-readable-DB"
|
||||
is the capability — a sparse profile relying on schema knowledge.
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. Does shard-wiki sanction **direct third-party DB reads** as a binding, or restrict them
|
||||
(schema coupling/drift) to a documented best-effort mode? How is schema drift across
|
||||
MojoMojo versions handled (UC-43)?
|
||||
2. Is **write-through by direct DB** ever allowed (risking app invariants), or are DB-backed
|
||||
no-API engines read/projection/overlay/backup only?
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- MojoMojo — github.com/mojomojo/mojomojo; metacpan MojoMojo (Catalyst app, DBIx::Class
|
||||
schema: Page / PageVersion / Content / Attachment)
|
||||
- Catalyst + DBIx::Class framework docs (architecture context)
|
||||
- prior: `research/260613-twiki-deep-dive/` (file-store classic contrast, UC-40/41)
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
New UC **UC-81** carries the marker **⊙** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-02, UC-40, UC-36, UC-34. Architecture cross-refs:
|
||||
SHARD-WP-0002 T14 (direct-DB binding), T13 (DB version-row history), T11.
|
||||
56
research/260614-notion-deep-dive/README.md
Normal file
56
research/260614-notion-deep-dive/README.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# 260614 — Notion deep dive (closed block-database SaaS, attached only through its API)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Notion** — architecture and extension model — read through
|
||||
shard-wiki's lens. Notion is the **closed, hosted, schema-rich** end of the spectrum and
|
||||
the dive that stress-tests shard-wiki's hardest constraints (*graceful degradation*,
|
||||
*no silent remote mutation*, *Markdown-first that must degrade*).
|
||||
|
||||
It contrasts sharply with its neighbours:
|
||||
- vs **Roam** — both are block databases with per-block UUIDs, but Roam is a *client*
|
||||
DataScript DB reached only via an *in-app* JS API, while Notion is a *server* Postgres
|
||||
store reached via a real *external* REST API (no in-engine hosting needed, but
|
||||
rate-limited, eventually consistent, scoped by per-page grants).
|
||||
- vs **Obsidian** — the mirror image of "file over app": Notion is **app over file**, a
|
||||
closed hosted store with no portable files.
|
||||
- vs **XWiki** — both are structured wiki-as-app-platforms; Notion is the **apex of
|
||||
database-as-pages** (schema + relations + rollups + views) but closed and REST-only.
|
||||
|
||||
Distinctive material:
|
||||
- **Data model** — everything is a **block** (UUID id, type, properties, ordered child
|
||||
`content`, single `parent`); pages are blocks, **database rows are pages with a
|
||||
schema**; Postgres-backed hosted SaaS
|
||||
- **Databases** — typed properties, **relations** (bidirectional), **rollups**,
|
||||
**formulas**, and many **views** (table/board/calendar/gallery) of one row-set
|
||||
- **Extension model** — **no in-app plugin runtime**; extension = **external REST API**
|
||||
integrations (+ webhooks since 2026), with ~3 rps rate limit, eventual consistency,
|
||||
recursive child fetch, payload caps, and **scoped/revocable per-page access grants**
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Block model, databases, page-model impedance, the API-only extension model, capability profile, scoped consent, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-57 attach a closed external-API-only hosted shard with rate-limit/eventual-
|
||||
consistency/scoped-grant constraints, UC-58 attach a typed database with schema +
|
||||
relations + views without flattening, UC-59 lossy-aware translation with a fidelity
|
||||
report); UC-31/34/36/39/50/51/52/54/56 enriched. Logged for `SHARD-WP-0002` (T14):
|
||||
an **operational-envelope** capability section, **access-grant semantics**, a
|
||||
**translation-fidelity** capability, and a three-way **attachment-mode taxonomy**
|
||||
(file-store / in-engine-hosted / external-API-only).
|
||||
|
||||
**Key takeaways recorded:** Notion *enforces* "no silent remote mutation" via scoped,
|
||||
revocable per-integration grants (ties the authz-in-core decision); it is the heaviest
|
||||
**lossy-translation** case (proprietary blocks/rich-text, lossy Markdown export); and it
|
||||
adds the **external-API-only** attachment mode (full write-through without in-engine
|
||||
hosting, but bounded by rate limits and eventual consistency). **Boundary:** one
|
||||
external-API candidate shard — best as projection/mirror/overlay/backup — not a
|
||||
substrate and not the federation layer.
|
||||
</content>
|
||||
302
research/260614-notion-deep-dive/findings.md
Normal file
302
research/260614-notion-deep-dive/findings.md
Normal file
@@ -0,0 +1,302 @@
|
||||
# Findings — Notion: a closed block-database SaaS, attached only through its API
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — a hosted block-database SaaS; a *candidate
|
||||
shard* of a distinct family (closed, external-REST-API-only, schema-rich, heaviest
|
||||
translation cost); has **no in-app plugin runtime**
|
||||
Lens: shard-wiki — the external-API attachment mode, database-as-pages, lossy
|
||||
translation/fidelity, scoped consent, and graceful degradation against a sovereign
|
||||
closed backend
|
||||
|
||||
> Where Notion sits in the set. Roam and Notion are both **block databases with
|
||||
> per-block UUIDs**, but they are opposite on access: Roam is a *client-side*
|
||||
> DataScript DB reachable only through an *in-app* JS API (write-through needs an
|
||||
> adapter hosted inside Roam); **Notion is a *server-side* Postgres store reachable
|
||||
> through a real *external* REST API** — attachable from outside with no in-engine
|
||||
> hosting, but rate-limited, eventually consistent, and **scoped by explicit
|
||||
> per-integration page grants**. And against Obsidian's "file over app," Notion is the
|
||||
> mirror image: **app over file** — a closed hosted store with no portable files at
|
||||
> all. So Notion is the dive that stress-tests shard-wiki's hardest constraints:
|
||||
> *graceful degradation*, *no silent remote mutation*, *union without erasure*, and
|
||||
> *Markdown-first that must degrade* against a proprietary, schema-rich, non-Markdown
|
||||
> backend.
|
||||
|
||||
Pairs with — and contrasts against — the Roam dive (block-DB, in-app vs external API),
|
||||
the Obsidian dive (closed-hosted vs file-over-app), and the XWiki dive (both are
|
||||
structured wiki-as-app-platforms; Notion is the apex of database-as-pages but closed
|
||||
and REST-only).
|
||||
|
||||
---
|
||||
|
||||
## 1. Core data model — everything is a block
|
||||
|
||||
Notion's data model is uniform: **"Everything you see in Notion is a block. Text,
|
||||
images, lists, a row in a database, even pages themselves — these are all blocks."**
|
||||
|
||||
Each block record holds:
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| **id** | a randomly generated **UUID v4**, visible in the page URL — the block's stable address |
|
||||
| **type** | determines rendering + which properties apply (text, to-do, heading, **page**, **child_database**, …) |
|
||||
| **properties** | type-specific attributes (e.g. `title` text; for database rows, the typed property values) |
|
||||
| **content** | an **ordered array of child block IDs** — nesting |
|
||||
| **parent** | a single upward pointer — used for **permission inheritance** |
|
||||
|
||||
Blocks form a **render tree** (content + parent pointers); indentation is *structural*,
|
||||
not cosmetic — it changes block relationships. **Pages are blocks**; **databases are
|
||||
blocks whose children are pages**; a database **row is a page** with typed properties.
|
||||
Stored in **Postgres** on Notion's servers (hosted SaaS — not local, not files).
|
||||
|
||||
The shard-wiki-relevant facts: (a) **per-block UUIDs** give native sub-page addressing
|
||||
(UC-51); (b) the **page = block, database-row = page-with-schema** identity is a real
|
||||
page-model impedance (§3); (c) the store is **closed and server-side** — reachable only
|
||||
through the API (§4).
|
||||
|
||||
---
|
||||
|
||||
## 2. Databases — schema, typed properties, relations, views
|
||||
|
||||
A Notion **database** is a collection of pages sharing a **schema** of typed
|
||||
**properties**: `title`, `rich_text`, `select` / `multi_select`, `number`, `date`,
|
||||
`checkbox`, `person`, `files`, **`relation`** (a typed link to rows in another
|
||||
database — shown on *both* sides, i.e. bidirectional), **`rollup`** (an aggregate over a
|
||||
relation), and **`formula`** (computed). The same row-set is shown through multiple
|
||||
**views** — **table, board (kanban), calendar, gallery, list, timeline** — each a
|
||||
filtered/sorted/grouped projection of the same data.
|
||||
|
||||
For shard-wiki this is the **apex of "wiki page as structured record"** (stronger than
|
||||
XWiki XObjects, UC-39): a page can be *bodiless typed data*, pages are joined by
|
||||
**typed inter-database relations** (a graph of typed links), and "the page" is routinely
|
||||
**a row in a schema** rather than prose. Notion databases also ship the ZigZag insight
|
||||
commercially: **one row-set, many views/dimensions** (UC-47/48) and **filtered/linked
|
||||
databases = query-defined pages** (UC-54).
|
||||
|
||||
---
|
||||
|
||||
## 3. Page-model impedance
|
||||
|
||||
Mapping Notion to a Markdown-first page model is the **heaviest translation case** in
|
||||
the research set:
|
||||
|
||||
- **Block/rich-text ≠ Markdown.** Notion's rich text is an annotated-span model; many
|
||||
block types (synced blocks, columns, callouts, embeds, database views) have no clean
|
||||
Markdown equivalent. Notion's own **export to Markdown/CSV is lossy** (databases →
|
||||
CSV, relations/rollups/formulas flatten or drop).
|
||||
- **Database-row-as-page + schema** must map onto pages + sidecar metadata without
|
||||
discarding the schema or the relations (extends UC-34/UC-39).
|
||||
- Therefore translation must be **lossy-aware with a fidelity report** — surface what
|
||||
did *not* round-trip rather than silently flattening (UC-59). This is *different from*
|
||||
UC-42 (Foswiki TML↔HTML *lossless* round-trip); Notion is fundamentally lossy.
|
||||
|
||||
---
|
||||
|
||||
## 4. Extension model — there is no plugin runtime; only the API
|
||||
|
||||
This is the defining architectural fact for shard-wiki: **Notion has no third-party
|
||||
in-app plugin/extension system.** There is no marketplace of code that runs inside
|
||||
Notion (the only "in-app modding" is the *unofficial* Notion Enhancer desktop patcher,
|
||||
out of scope). **Extension = external integration via the public REST API** (plus
|
||||
embeds and, recently, webhooks).
|
||||
|
||||
The **public REST API** (`api.notion.com/v1`):
|
||||
|
||||
- **Resources:** `pages`, `blocks` (retrieve / append / update / delete children),
|
||||
`databases` (retrieve, **query** with filters+sorts, create), `users`, `search`,
|
||||
`comments`.
|
||||
- **Authorization:** an **internal integration token** (workspace-owned) or **OAuth
|
||||
2.0** (public integrations). Critically, **an integration only sees pages a user has
|
||||
explicitly connected to it** — the user "approves the app and connects specific
|
||||
pages" via *Add connections*. Access is a **scoped, revocable, per-page grant** (§6).
|
||||
- **Operational constraints the adapter must encode:**
|
||||
- **Rate limit** ~**3 requests/second** average (429 `rate_limited`); **no paid
|
||||
increase**.
|
||||
- **Eventual consistency / no live read model** — the API is not real-time; reads can
|
||||
lag.
|
||||
- **Recursive retrieval** — a block returns only **first-level children**; full-page
|
||||
reconstruction requires walking `has_children` recursively.
|
||||
- **Payload caps** — 1000 blocks / 500 KB per request; child arrays ≤ 100.
|
||||
- **Webhooks** (added 2026) deliver page/database change events — a push transport for
|
||||
UC-31, replacing pure polling.
|
||||
- **Version history:** Notion keeps **internal page history** (retention bounded by
|
||||
plan); it is **not portable** and not exposed as git — a UC-36 *supplementation* case
|
||||
(like Confluence/MediaWiki), not an import case.
|
||||
- **Publish:** pages can be **published to the web** as read-only public pages — an
|
||||
outbound publish surface (UC-56).
|
||||
|
||||
Consequence: Notion is attachable **only** as an **external-API shard** — but, unlike
|
||||
Roam, **no in-engine adapter hosting is needed** (the REST API is external). The cost is
|
||||
the operational envelope (rate-limit, eventual consistency, scoped grant) that the
|
||||
capability profile must model (UC-57).
|
||||
|
||||
---
|
||||
|
||||
## 5. Notion as a shard — capability profile
|
||||
|
||||
| Capability | Notion | Notes for the adapter contract |
|
||||
|------------|--------|--------------------------------|
|
||||
| Read | **yes (external REST)** | rate-limited (~3 rps), eventually consistent, recursive child fetch |
|
||||
| Write | **yes (external REST)** | append/update/delete blocks, create pages — no in-engine host needed (vs Roam) |
|
||||
| Write granularity | **block-level (fine)** | like Roam; per-block ops |
|
||||
| Identity / addressing | **block UUID v4** | native sub-page addressing (UC-51), store-minted (like Roam, not in-file) |
|
||||
| Structured data | **yes (apex)** | databases: schema + typed properties + relations + rollups + formulas (UC-34/39/58) |
|
||||
| Native query | **yes** | database query API (filters/sorts) → delegate views (UC-52) |
|
||||
| Views / dimensions | **yes** | table/board/calendar/gallery = many views of one row-set (UC-47/48/54) |
|
||||
| Subscribe | **webhooks (2026)** | push events; else poll (UC-31) |
|
||||
| Version history | **internal, not portable** | supplement via coordination journal (UC-36) |
|
||||
| Diff / merge | **no native** | — |
|
||||
| Lock | **no** | — |
|
||||
| Publish | **publish-to-web** | outbound read-only (UC-56) |
|
||||
| Access model | **scoped per-page grant (OAuth/token)** | explicit consent; revocable (UC-57, authz) |
|
||||
| Syntax / content | **proprietary block + rich text** | **lossy** to Markdown; needs fidelity-aware translation (UC-59) |
|
||||
|
||||
Verdict: Notion is a legitimate but **demanding** shard — **external-API-attached,
|
||||
schema-rich, fine-grained, closed**. It behaves best as a **projected / mirrored /
|
||||
overlay / backup** participant; full write-through is possible but bounded by rate limits
|
||||
and eventual consistency. The strongest reasons to attach it: structured databases
|
||||
(UC-58) and block-UUID addressing (UC-51); the strongest cautions: lossy translation
|
||||
(UC-59) and no portable history (UC-36).
|
||||
|
||||
---
|
||||
|
||||
## 6. Scoped consent and "no silent remote mutation"
|
||||
|
||||
Notion *enforces* one of shard-wiki's INTENT constraints at the platform level: an
|
||||
integration can touch **only** the pages a user has explicitly **connected** to it, and
|
||||
the grant is **revocable**. This is a clean, real-world model of **no silent remote
|
||||
mutation** and of a shard granting the orchestrator **scoped, consented access** — and
|
||||
it ties directly to the settled **authz-in-core / authn-delegated** decision
|
||||
([[shard-wiki-auth-in-core-decision]]): authentication to Notion is delegated (OAuth /
|
||||
integration token), while shard-wiki's *own* authorization decides what to do with the
|
||||
granted scope. The adapter contract should treat **"scoped, revocable grant"** as a
|
||||
first-class attachment property (UC-57), not an afterthought.
|
||||
|
||||
---
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
|
||||
- **Graceful degradation** has its sharpest test here: a closed SaaS with no files, rate
|
||||
limits, eventual consistency, and lossy export must still be usable as
|
||||
read/projection/overlay/backup. If the adapter contract handles Notion, it handles
|
||||
most things.
|
||||
- **No silent remote mutation** is *modeled by the platform* (scoped grants) — Notion
|
||||
validates the principle (§6).
|
||||
- **Database-as-pages** validates that "wiki page" must stretch to **typed records with
|
||||
relations**, not just prose (UC-34/39/58).
|
||||
- **Block UUIDs** reconfirm (with Roam) that native sub-page addressing is real and
|
||||
adoptable (UC-51).
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Closed hosted store; no sovereignty over bytes.** shard-wiki cannot make Notion
|
||||
git-native or local. It can mirror/project/overlay/back-up and **supply** a
|
||||
git-addressable history (UC-36) — never claim to own Notion's store.
|
||||
2. **Lossy, proprietary content.** Do **not** pretend Notion round-trips to Markdown.
|
||||
Translate lossily *with a fidelity report* and preserve non-mappable elements as
|
||||
provenance/sidecar (UC-59) — union without erasure includes erasure *of fidelity*
|
||||
being made visible.
|
||||
3. **External-API-only, rate-limited, eventually consistent.** Projection must be
|
||||
**cache/poll/webhook**, not a live read model; sync is bounded — encode this in the
|
||||
capability profile (UC-57). Do not design flows that assume cheap, instant, unlimited
|
||||
reads.
|
||||
4. **One workspace (or its granted page set) = one shard**, never the federation layer.
|
||||
|
||||
### 7.3 What Notion teaches that shard-wiki should keep
|
||||
|
||||
- Model **operational envelope** (rate limit, consistency class, payload caps,
|
||||
pagination) as explicit capability-profile fields — Notion makes them unignorable.
|
||||
- Model **scoped, revocable consent** as a first-class attachment property (UC-57, §6).
|
||||
- Treat **translation fidelity as data**: a per-shard, per-page report of what projects
|
||||
cleanly vs. degrades (UC-59) — applies beyond Notion.
|
||||
- Recognize **external-REST attach** as a distinct, *preferred-where-available* mode:
|
||||
full write-through without in-engine hosting (contrast Roam) — but pay the operational
|
||||
envelope.
|
||||
|
||||
---
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-56**. New UCs **UC-57–UC-59** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a closed hosted shard via its external REST API only** — no file store, no in-app runtime — honoring rate limits, eventual consistency, payload caps, and a scoped/revocable access grant | **UC-57 (new)** |
|
||||
| **Attach a typed database (schema + relations + rollups + multiple views) as a shard** without flattening the schema or the inter-record relations | **UC-58 (new)** |
|
||||
| **Translate a proprietary block/rich-text model to/from Markdown with an explicit fidelity report**, preserving non-mappable elements rather than silently dropping them | **UC-59 (new)** |
|
||||
| Block UUIDs = store-minted native span addresses (external-API variant) | **enriches UC-51** |
|
||||
| External-API block-DB attach (no in-engine host) — contrast Roam's in-app-only | **enriches UC-50** |
|
||||
| Database query API + filtered/linked DBs | **enriches UC-52, UC-54** |
|
||||
| Database-as-pages apex; typed records + relations | **enriches UC-34, UC-39** |
|
||||
| Webhooks (2026) as a push transport | **enriches UC-31** |
|
||||
| Internal-only page history, not portable | **enriches UC-36** |
|
||||
| Publish-to-web outbound | **enriches UC-56** |
|
||||
| Scoped, revocable per-integration grant; no silent mutation | links **UC-57** + [[shard-wiki-auth-in-core-decision]] |
|
||||
|
||||
---
|
||||
|
||||
## 9. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- Add an **operational-envelope** section to the adapter capability profile: rate limit,
|
||||
consistency class (live / eventually-consistent / snapshot), payload/pagination caps,
|
||||
recursive-fetch requirement, push-vs-poll transport. Notion is the forcing example.
|
||||
- Add **access-grant semantics**: scope (which pages), revocability, auth mode
|
||||
(delegated token/OAuth) — ties the authz-in-core decision and "no silent mutation".
|
||||
- Add a **translation-fidelity capability**: adapters declare and report what content
|
||||
round-trips vs. degrades (UC-59); generalizes UC-42 (lossless) to the lossy case.
|
||||
- **Attachment-mode taxonomy** now spans: file-store direct (Obsidian/TWiki, UC-40),
|
||||
in-engine hosted adapter (Roam/XWiki, UC-38/50), and **external-API-only**
|
||||
(Notion, UC-57). T14 binding should enumerate all three.
|
||||
- **Database/schema/relations** as a unit (UC-58) presses the page-model spec: collection
|
||||
+ schema + typed relations, not just a page.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions (for spec / workplans)
|
||||
|
||||
1. Is **external-API-only with a tight rate limit** (Notion) viable for write-through at
|
||||
wiki scale, or do we cap Notion at read/projection/overlay/backup by default?
|
||||
2. How are **inter-database relations** (UC-58) represented in the union — as typed links
|
||||
in the link graph, as a separate relation index (cf. ZigZag many-to-many), or both?
|
||||
3. What is the **fidelity report** format (UC-59), and where does it surface — provenance
|
||||
panel, projection metadata, reconciliation review?
|
||||
4. For **scoped grants** (§6), how does shard-wiki represent partial visibility (only
|
||||
*some* of a workspace's pages granted) without misrepresenting the shard as complete?
|
||||
5. Do we consume Notion **webhooks** (push) or poll, given eventual consistency and the
|
||||
rate limit (UC-31)?
|
||||
|
||||
---
|
||||
|
||||
## 11. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Notion — "The data model behind Notion's flexibility" (https://www.notion.com/blog/data-model-behind-notion) | Everything-is-a-block; block record (id/type/properties/content/parent); render tree; pages/databases as blocks; Postgres |
|
||||
| Notion Docs — Request limits (https://developers.notion.com/reference/request-limits) | ~3 rps rate limit, 429, payload caps, recursive first-level children |
|
||||
| Notion Docs — Authorization (https://developers.notion.com/docs/authorization) | Internal token vs OAuth; integration connected to specific pages; scoped grant |
|
||||
| Hookdeck / ClickUp — Notion webhooks guides (https://hookdeck.com/webhooks/platforms/guide-to-notion-webhooks-features-and-best-practices) | Webhook support (2026), page/database change events |
|
||||
| Truto / Rollout — Notion API architecture & essentials (https://truto.one/blog/how-to-integrate-with-the-notion-api-architecture-guide-for-b2b-saas/) | REST endpoints (pages/blocks/databases/search), integration patterns, no in-app plugin model |
|
||||
| General API knowledge — database property types, views, relations/rollups/formulas, export-to-Markdown lossiness | §2, §3 |
|
||||
|
||||
Cross-references: `research/260614-roam-deep-dive/findings.md` (block-DB/UUID, in-app vs
|
||||
external API), `research/260614-obsidian-deep-dive/findings.md` (closed-hosted vs
|
||||
file-over-app), `research/260613-xwiki-deep-dive/findings.md` (structured wiki-app-
|
||||
platform), `spec/UseCaseCatalog.md` (UC-31, UC-34, UC-36, UC-39, UC-50/51/52, UC-54,
|
||||
UC-56), `workplans/SHARD-WP-0002-federation-architecture.md` (T14), and the authz
|
||||
decision [[shard-wiki-auth-in-core-decision]].
|
||||
|
||||
---
|
||||
|
||||
## 12. Traceability
|
||||
|
||||
- New UCs: **UC-57, UC-58, UC-59** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-31, UC-34, UC-36, UC-39, UC-50, UC-51, UC-52, UC-54, UC-56**.
|
||||
- Architecture (no UC): operational-envelope + access-grant + translation-fidelity
|
||||
capability fields; three-way attachment-mode taxonomy; database/schema/relations in the
|
||||
page model → `SHARD-WP-0002` (T14).
|
||||
- Decision link: scoped/revocable grant + no-silent-mutation → [[shard-wiki-auth-in-core-decision]].
|
||||
- Boundary recorded: Notion is **one external-API candidate shard** — closed, hosted,
|
||||
schema-rich, lossy-to-Markdown — best as projection/mirror/overlay/backup; not a
|
||||
substrate, not the federation layer (INTENT graceful-degradation + no-silent-mutation).
|
||||
</content>
|
||||
51
research/260614-obsidian-deep-dive/README.md
Normal file
51
research/260614-obsidian-deep-dive/README.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# 260614 — Obsidian deep dive (file-over-app vaults, plugin API, ecosystem signal)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Obsidian** — core/extension architecture and the plugin
|
||||
ecosystem — read through shard-wiki's lens. Obsidian is the **most INTENT-aligned tool
|
||||
in the research set**: INTENT names Obsidian vaults as a shard participant, and its
|
||||
**"file over app"** model (notes are plain Markdown files in a folder you own; the app
|
||||
is a lens) is the closest cousin to shard-wiki's Markdown-first, sovereignty-preserving
|
||||
thesis. It is the **file-backed counterpart to the Roam dive** (same tool category,
|
||||
opposite storage model: Roam = client DB + API; Obsidian = files on disk).
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — vault = folder of `.md` + `.obsidian/` config; files canonical;
|
||||
**MetadataCache** is a *derived*, async-rebuilt index (links, embeds, tags,
|
||||
frontmatter, block IDs, resolved/unresolved links, backlinks)
|
||||
- **In-file, git-diffable** addressing/structure — `[[wikilinks]]`, `![[embeds]]`
|
||||
(transclusion), `^block-id` span refs, YAML frontmatter/properties
|
||||
- **Plugin API** — `Plugin` `onload/onunload`; `App` modules `vault` (CRUD + file
|
||||
events), `metadataCache`, `workspace` (CM6, views), `fileManager.renameFile`;
|
||||
`manifest.json`; community plugins, Restricted Mode
|
||||
- **Ecosystem popularity → UC signal** (§7) — ranked download data mined for use cases,
|
||||
per the research brief
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | File-over-app architecture, in-file addressing, plugin API as adapter host, capability profile, **plugin-popularity → UC mapping**, derived-index lesson, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Four new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-53 attach a local vault with a live concurrent native editor, UC-54 query-defined
|
||||
dynamic page, UC-55 carry non-Markdown content types, UC-56 outbound publish of a
|
||||
projection); UC-15/28/34/36/40/51/52 enriched. Logged for `SHARD-WP-0002` (T14):
|
||||
**dual attachment mode** (file-store *or* in-app plugin host), consume-native-derived-
|
||||
index capability, non-Markdown content types in the page model, outbound publish,
|
||||
external-writer tolerance.
|
||||
|
||||
**Key takeaways recorded:** (1) Obsidian validates *files-canonical / index-derived* —
|
||||
shard-wiki's projection model. (2) Fine-grained addressing & structure can be portable
|
||||
**in-file text** (`^block-id`, frontmatter), not DB state (contrast Roam). (3) The
|
||||
ecosystem is demand evidence — the #1 plugin is **drawings** (non-Markdown content),
|
||||
query-as-DB (Dataview/Tasks) is top-tier but an *add-on*, and **Git is top-7** (users
|
||||
bolt on the portable history shard-wiki provides natively). **Boundary:** a vault is one
|
||||
file-backed candidate shard, not the federation layer and not a file-sync target
|
||||
(not-a-sync-daemon).
|
||||
</content>
|
||||
324
research/260614-obsidian-deep-dive/findings.md
Normal file
324
research/260614-obsidian-deep-dive/findings.md
Normal file
@@ -0,0 +1,324 @@
|
||||
# Findings — Obsidian: file-over-app vaults, the plugin API, and what the ecosystem reveals
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — a local-first Markdown vault tool; the
|
||||
*most INTENT-aligned candidate shard yet* (file-backed, Markdown-first); plus an
|
||||
ecosystem whose popularity is direct evidence of what users actually do
|
||||
Lens: shard-wiki — file-store attachment, in-file fine-grained addressing, derived
|
||||
index vs. canonical files, the engine-hosts-adapter path, and ecosystem-driven UCs
|
||||
|
||||
> Why Obsidian matters most. INTENT names **Obsidian vaults** explicitly as a shard
|
||||
> participant, and Obsidian's **"file over app"** philosophy — your notes are plain
|
||||
> Markdown files in a folder you own, the app is just a lens — is the closest cousin
|
||||
> to shard-wiki's own Markdown-first, sovereignty-preserving thesis. It is also the
|
||||
> **file-backed counterpart to the Roam dive**: same "personal knowledge tool"
|
||||
> category, opposite storage model (Roam = client DB + API; Obsidian = files on disk).
|
||||
> And uniquely among the dives, Obsidian has a **large, ranked plugin ecosystem** — so
|
||||
> the user's instruction to let *plugin popularity inform the use-case catalog* is
|
||||
> actionable: what people install tells us what a federated wiki must support (§7).
|
||||
|
||||
This dive treats Obsidian as a candidate shard (capability profile §5), studies its
|
||||
plugin API as an adapter-host surface (§4), and mines the download rankings for UCs
|
||||
(§7). Pairs with the Roam dive (the DB/API contrast) and the TWiki/Foswiki dives (the
|
||||
file-store attachment path, UC-40).
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture — file over app
|
||||
|
||||
- A **vault** is a local folder of plain **Markdown files** plus a `.obsidian/` config
|
||||
directory (settings, enabled plugins, themes, workspace layout). **The files are the
|
||||
source of truth**; the app holds no separate canonical database.
|
||||
- Markdown is **Obsidian-flavored**: CommonMark + `[[wikilinks]]`, `![[embeds]]`
|
||||
(transclusion), `^block-id` block references, `#tags`, YAML **frontmatter /
|
||||
properties** (typed metadata), and callouts. Note: all addressing and structure live
|
||||
**in the file text** — so they are git-diffable and portable.
|
||||
- **MetadataCache** is a *derived* index: Obsidian pre-parses every file into cached
|
||||
metadata (headings, links, embeds, tags, frontmatter, **blocks/block IDs**,
|
||||
`resolvedLinks` / `unresolvedLinks`, backlinks). It is **rebuilt from files** and
|
||||
updated **asynchronously** ("don't rely on immediate updates after a modify"). Files
|
||||
canonical; index derived.
|
||||
- **Canvas** files (`.canvas`, the open **JSON Canvas** format) hold spatial
|
||||
arrangements — a non-Markdown content type living in the vault.
|
||||
- Sync is *not* built into the format: users add Git, Obsidian Sync, or a file-sync
|
||||
plugin (see §7). The vault is just files.
|
||||
|
||||
For shard-wiki this is the **ideal file-backed shard**: Markdown-first, on disk,
|
||||
git-friendly, with a derived index model that mirrors shard-wiki's own
|
||||
projection/derived-view philosophy (the cache is to Obsidian what projections and union
|
||||
BackLinks are to shard-wiki — computed, not canonical).
|
||||
|
||||
---
|
||||
|
||||
## 2. Addressing, links, transclusion — in-file and git-diffable
|
||||
|
||||
- **Links:** `[[Page]]` and `[[Page#Heading]]` / `[[Page#^block-id]]`; the resolved/
|
||||
unresolved link sets and **backlinks** are computed in MetadataCache — Obsidian's
|
||||
shipped answer to union BackLinks (UC-05/UC-18), but per-vault.
|
||||
- **Fine-grained addressing:** a **`^block-id`** suffix on a paragraph (and heading
|
||||
anchors) gives a stable sub-page address — *but it lives in the Markdown text*, not in
|
||||
a database. This is the **git-diffable, portable variant** of Roam's `:block/uid`
|
||||
(which lives in a DataScript DB). Important contrast for UC-51: native span IDs can be
|
||||
**text-embedded** (Obsidian) or **store-minted** (Roam); the text-embedded form is
|
||||
more portable and survives a file copy, but is opt-in (the user must add the `^id`).
|
||||
- **Transclusion:** `![[Page]]`, `![[Page#Heading]]`, `![[Page#^block-id]]` embed live
|
||||
content by reference — shipped transclusion at page/section/block granularity (UC-32),
|
||||
again stored as plain text.
|
||||
|
||||
---
|
||||
|
||||
## 3. Structured data — frontmatter as git-diffable records
|
||||
|
||||
YAML **frontmatter / properties** make each note a lightweight typed record
|
||||
(`tags`, `aliases`, arbitrary keys). This is the **git-diffable structured-data**
|
||||
variant (cf. TWiki `%META%` inside the file, UC-34) — structure lives in the file, not a
|
||||
DB. The **Dataview** plugin (§7) turns frontmatter + inline fields + the link graph into
|
||||
a queryable database — but that query power is an *ecosystem add-on*, not core.
|
||||
|
||||
---
|
||||
|
||||
## 4. Extension architecture — the plugin API as an adapter host
|
||||
|
||||
Obsidian plugins are TypeScript, shipped as `manifest.json` + `main.js` (+ optional
|
||||
`styles.css`):
|
||||
|
||||
- **Manifest:** `id`, `name`, `version`, `minAppVersion` (required); `author`,
|
||||
`description`, `isDesktopOnly`, `fundingUrl` (optional). Community plugins are
|
||||
GitHub-hosted and listed in the `obsidian-releases` repo; installed from inside the
|
||||
app; gated by **Restricted Mode** (plugins run arbitrary code — Node/Electron on
|
||||
desktop).
|
||||
- **Plugin lifecycle:** extend `Plugin`, implement **`onload()` / `onunload()`**; use
|
||||
`registerEvent` / `registerDomEvent` / `registerInterval` for auto-cleanup.
|
||||
- **The `App` object** (global singleton, `this.app`) exposes four modules:
|
||||
- **`vault`** — file CRUD: `read` / `cachedRead`, `create`, `modify`, `process`,
|
||||
`delete`, `rename`, `getFiles`, low-level `adapter`, and **events**
|
||||
(`on('create'|'modify'|'delete'|'rename')`) for live file-watching.
|
||||
- **`metadataCache`** — `getFileCache`, `frontmatter`, `links`, `embeds`, `tags`,
|
||||
`headings`, `blocks`, `resolvedLinks` / `unresolvedLinks`, `getBacklinksForFile`.
|
||||
- **`workspace`** — panes/leaves/views; `registerView`, `registerEditorExtension`
|
||||
(CodeMirror 6), `registerMarkdownPostProcessor`.
|
||||
- **`fileManager`** — high-level ops; **`renameFile`** updates inbound links (use it
|
||||
instead of raw vault rename to keep link integrity).
|
||||
- **Registrations:** `addCommand`, `addRibbonIcon`, `addStatusBarItem`,
|
||||
`addSettingTab`, `registerView`, `registerEditorExtension`,
|
||||
`registerMarkdownPostProcessor`.
|
||||
|
||||
Two consequences for shard-wiki:
|
||||
|
||||
1. **A vault is attachable two ways.** (a) **File-store direct attach** (UC-40) — read
|
||||
the folder as a folder shard; no plugin, offline-capable, git-native; good for
|
||||
read/projection/overlay. (b) **In-app plugin host** (UC-38) — a shard-wiki adapter
|
||||
*as an Obsidian plugin* drives `vault` write + `metadataCache` reads + live file
|
||||
events for high-fidelity write-through. This dual mode is a cleaner story than Roam
|
||||
(which only offers the in-app path).
|
||||
2. The **MetadataCache API is a ready-made derived-view source** — backlinks, tags,
|
||||
block IDs, resolved links — an adapter can consume instead of reparsing.
|
||||
|
||||
---
|
||||
|
||||
## 5. Obsidian as a shard — capability profile
|
||||
|
||||
| Capability | Obsidian | Notes for the adapter contract |
|
||||
|------------|----------|--------------------------------|
|
||||
| Read | **yes** | direct file read (folder shard) or `vault.read`/`cachedRead` in-app |
|
||||
| Write | **yes** | direct file write, or `vault.modify`/`create`/`process` in-app; per-file granularity |
|
||||
| Write granularity | **per-file (page)** | the natural shard-wiki granularity — contrast Roam (block) and TiddlyWiki (whole-file-single-vault) (UC-35) |
|
||||
| Identity / addressing | **path + in-file `^block-id` / headings** | git-diffable, portable, **opt-in** span IDs (UC-51 text-embedded variant) |
|
||||
| Transclusion | **yes** | `![[...]]` page/section/block embeds (UC-32) |
|
||||
| Backlinks / links | **yes (derived)** | MetadataCache resolved/unresolved links + backlinks (UC-05/18) |
|
||||
| Structured data | **yes (in-file)** | YAML frontmatter/properties; queryable via Dataview plugin (UC-34) |
|
||||
| Native query | **plugin only** | Dataview — *not core*; informs UC-52 (query is adapter/plugin-provided, not assumable) |
|
||||
| Version history | **none native** | users add the Git plugin (top-7!) → validates the coordination journal (UC-36) |
|
||||
| Diff / merge | **none native** | git-level if Git plugin / repo-backed |
|
||||
| Lock | **no** | local-first, single-user assumption |
|
||||
| Publish | **via Obsidian Publish / Quartz / Digital Garden** | outbound publish of a projection (UC-56) |
|
||||
| Syntax | **Obsidian-flavored Markdown** | close to CommonMark; wikilinks/embeds/callouts/`^id` need adapter awareness (cf. UC-42) |
|
||||
| Non-Markdown content | **Canvas (JSON Canvas), attachments, Excalidraw** | typed/opaque assets in the vault (UC-55) |
|
||||
|
||||
Verdict: Obsidian is the **cleanest file-backed, Markdown-first shard** — the
|
||||
file-store family (TWiki/Foswiki/DokuWiki, UC-40), and the *reference* personal-vault
|
||||
shard INTENT already names. Its history/query/sync gaps are exactly what users fill with
|
||||
plugins (§7) — i.e. exactly what shard-wiki offers as orchestration.
|
||||
|
||||
---
|
||||
|
||||
## 6. The derived-index lesson (architecture)
|
||||
|
||||
Obsidian's **files-canonical / MetadataCache-derived** split is independent validation
|
||||
of shard-wiki's core stance: the link graph, backlinks, tags, and block index are
|
||||
**computed projections over canonical files**, rebuilt on change, eventually consistent
|
||||
("updated asynchronously"). shard-wiki's union BackLinks, projections, and derived views
|
||||
(UC-05, UC-17–UC-20) should likewise be **derived and rebuildable**, never a second
|
||||
source of truth. The cache-invalidation/async-update caveat is a real design note for
|
||||
projection freshness (UC-31).
|
||||
|
||||
---
|
||||
|
||||
## 7. Ecosystem popularity → use-case signal (the user's ask)
|
||||
|
||||
All-time download ranks (obsidianstats.com), read as *demand evidence*:
|
||||
|
||||
| Plugin (downloads) | What users do with a vault | shard-wiki signal → UC |
|
||||
|--------------------|----------------------------|------------------------|
|
||||
| **Excalidraw** (6.4M) | drawings/diagrams as first-class content | non-Markdown content types in a shard → **UC-55 (new)** |
|
||||
| **Templater** (4.6M), **QuickAdd** (1.9M) | templated note creation | blueprint pages → **UC-15 (enriched)** |
|
||||
| **Dataview** (4.4M) | query the vault (files+frontmatter+links) as a DB | query-defined dynamic pages → **UC-54 (new)**; query is a *plugin* → **UC-52 (enriched)** |
|
||||
| **Tasks** (3.6M) | aggregate to-dos across all notes | cross-page typed-item aggregation → **UC-54** |
|
||||
| **Advanced Tables** (2.9M) | structured tables in Markdown | in-file structured data → UC-34 |
|
||||
| **Calendar** (2.8M) | daily-note navigation | temporal dimension (ZigZag `d.recent`) → UC-17 |
|
||||
| **Git** (2.7M, top-7) | version control + sync on the vault | **history is a bolt-on, not native** → validates coordination journal → **UC-36 (enriched)** |
|
||||
| **Kanban** (2.4M) | board view of Markdown cards | alternate projection/view of pages → UC-47/48 |
|
||||
| **Remotely Save** (2.0M) | sync vault to S3 / Dropbox / OneDrive / GDrive / **WebDAV** | demand to federate a vault to heterogeneous remote stores → reinforces INTENT WebDAV/Nextcloud shards (but shard-wiki is *not* a file-sync daemon — attach as shards, don't mirror) |
|
||||
| **Omnisearch** (1.6M) | search across notes, PDFs, images | union full-text search over mixed content → UC-20 |
|
||||
| **Importer** (1.4M) | import from Evernote / Notion / Bear / Apple Notes | carry-forward/import from foreign tools → **UC-28 (enriched)** |
|
||||
| **Smart Connections** (1.0M), **Copilot** (1.5M), **Claudian** (938K) | AI over the vault (related notes, chat) | mostly out of scope; "related notes" ≈ link/equivalence discovery (UC-46) — note the trend, don't build it into core |
|
||||
| **Tag Wrangler** (981K) | rename/merge tags vault-wide | namespace/tag refactor across the union → UC-22 |
|
||||
| **Linter** (927K) | normalize frontmatter/formatting | content normalization → fingerprint basis for equivalence (UC-46) |
|
||||
| **Admonition** (922K) | callouts | Obsidian-flavored syntax → translation awareness (UC-42) |
|
||||
|
||||
Headline reads:
|
||||
- The **#1 plugin is drawings** (Excalidraw) — users keep **non-Markdown content** in
|
||||
"Markdown" vaults. A wiki orchestrator that assumes pure Markdown will mishandle the
|
||||
most popular real-world use → **UC-55**.
|
||||
- **Query-the-vault-as-a-DB** (Dataview #3 + Tasks #4) is a top use — but it is an
|
||||
*add-on*, confirming query is an **adapter/plugin capability** (UC-52), and motivating
|
||||
**query-defined dynamic pages** (UC-54).
|
||||
- **Git is top-7** — users manually bolt portable version control onto their vault. This
|
||||
is *direct demand* for what shard-wiki provides natively as the coordination journal
|
||||
(UC-36).
|
||||
- **Sync-to-anywhere** (Remotely Save #11) shows demand to connect a vault to S3/WebDAV/
|
||||
cloud — the heterogeneous backends INTENT targets — while reminding us of the
|
||||
*not-a-file-sync-daemon* boundary.
|
||||
|
||||
---
|
||||
|
||||
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 8.1 Reinforcements
|
||||
|
||||
- **File-over-app == shard sovereignty + Markdown-first.** Obsidian is the living proof
|
||||
that a serious tool can be "just files you own"; it is the model shard for INTENT's
|
||||
Obsidian/local-folder participants.
|
||||
- **Files canonical, index derived** validates shard-wiki's projection/derived-view
|
||||
architecture (§6).
|
||||
- **In-file, git-diffable structure & addressing** (frontmatter, `^block-id`, embeds)
|
||||
shows fine-grained addressing and structured data can be **portable text**, not DB
|
||||
state — friendlier to the coordination journal than Roam's DB.
|
||||
|
||||
### 8.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **A vault is one shard, not the federation.** Local-first, single-vault,
|
||||
single-user; do not model the union as "one big vault."
|
||||
2. **Not a file-sync daemon.** The Remotely Save popularity is tempting; shard-wiki must
|
||||
stay **wiki-page-semantic** — attach remote stores as shards, never generic file
|
||||
mirroring (INTENT constraint).
|
||||
3. **`.obsidian/` is opaque app config, not page content.** The adapter must exclude/
|
||||
treat it as shard-local config, not project it as pages.
|
||||
4. **Obsidian-flavored Markdown ≠ CommonMark.** Wikilinks/embeds/callouts/`^id` need
|
||||
adapter awareness; closer than Roam's outline, but still a translation surface
|
||||
(UC-42).
|
||||
5. **Plugins run arbitrary code.** If a shard-wiki adapter is hosted *as* an Obsidian
|
||||
plugin (UC-38 path), it inherits Electron/Node trust — a deployment/security note,
|
||||
not a core concern.
|
||||
|
||||
### 8.3 What Obsidian teaches that shard-wiki should keep
|
||||
|
||||
- Offer **both attachment modes** for a vault: zero-config **file-store direct attach**
|
||||
(read/projection/overlay) and an optional **in-app adapter** (write-through, live
|
||||
events). Graceful degradation by default, fidelity on opt-in.
|
||||
- **Consume a backend's existing derived index** (MetadataCache) instead of reparsing,
|
||||
when the shard exposes one.
|
||||
- **Embrace non-Markdown content types** (Canvas/attachments/drawings) as typed or
|
||||
opaque assets with provenance — don't flatten them away (UC-55, extends UC-34).
|
||||
|
||||
---
|
||||
|
||||
## 9. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-52**. New UCs **UC-53–UC-56** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a local-first Markdown vault as a file-backed shard with a live concurrent native editor** — file-watch, tolerate the vault's own app writing concurrently, treat `.obsidian/` as opaque | **UC-53 (new)** |
|
||||
| **Define a page as a live query over the union** (Dataview/Tasks pattern — query-defined dynamic page) | **UC-54 (new)** |
|
||||
| **Carry non-Markdown content types** (Canvas/JSON Canvas, drawings, attachments) as typed/opaque assets with provenance | **UC-55 (new)** |
|
||||
| **Publish a curated projection of the union or a shard to an external read-only target** (Obsidian Publish / Quartz / Digital Garden) | **UC-56 (new)** |
|
||||
| Local Markdown vault = cleanest file-backed direct-attach; dual attach (file-store *or* in-app plugin host) | **enriches UC-40** (and UC-02, UC-38) |
|
||||
| In-file `^block-id` / heading anchors = git-diffable, portable, opt-in span IDs | **enriches UC-51** |
|
||||
| YAML frontmatter/properties = git-diffable in-file structured data | **enriches UC-34** |
|
||||
| Git plugin (top-7) = users bolt on portable history → demand for the coordination journal | **enriches UC-36** |
|
||||
| Query is a popular plugin (Dataview), not core → query is adapter/plugin-provided | **enriches UC-52** |
|
||||
| Importer plugin = import from foreign tools | **enriches UC-28** |
|
||||
| Templater/QuickAdd = templated creation | **enriches UC-15** |
|
||||
| `![[...]]` embeds = in-file transclusion | links UC-32 |
|
||||
| MetadataCache backlinks/links | links UC-05/UC-18 |
|
||||
|
||||
---
|
||||
|
||||
## 10. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Dual attachment mode** for a single backend (file-store *and* in-engine adapter)
|
||||
should be first-class in the adapter contract / T14 binding — Obsidian is the clean
|
||||
example (Roam was in-app-only; TWiki had file-vs-API but not a personal vault).
|
||||
- **"Consume native derived index"** as a capability: a shard may expose its own link/
|
||||
backlink/block index (Obsidian MetadataCache, Roam Datalog) the orchestrator can read
|
||||
instead of reparsing (ties UC-52).
|
||||
- **Non-Markdown content types** (UC-55) push on the wiki page model: pages vs. typed
|
||||
assets vs. opaque blobs — a page-model spec decision, not just adapter config.
|
||||
- **Outbound publish** (UC-56) formalizes the `publish` capability from INTENT's list as
|
||||
a projection target, complementing inbound static-export attach (UC-37).
|
||||
- Concurrent-native-editor (UC-53) needs the contract to express **external-writer
|
||||
tolerance** (file-watching, re-projection, conflict-with-live-app) — distinct from
|
||||
multi-user write conflicts.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions (for spec / workplans)
|
||||
|
||||
1. For a vault attached **both** ways (file-store + in-app plugin), which is
|
||||
authoritative, and how do they reconcile (the in-app adapter sees MetadataCache; the
|
||||
file-store path sees raw bytes)?
|
||||
2. Is **UC-54 (query-defined dynamic page)** a core page type, an adapter feature, or a
|
||||
reference-UI/plugin concern? (Mirrors the Roam/Dataview "views are queries" question
|
||||
and catalog Q7.)
|
||||
3. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a
|
||||
Markdown stub, opaque blob with provenance, or a pluggable content-type registry?
|
||||
4. Does **outbound publish** (UC-56) belong in core or as a publish-adapter family, and
|
||||
how does it interact with overlays/projection freshness?
|
||||
5. How is **concurrent native editing** (UC-53) surfaced — optimistic re-projection,
|
||||
advisory lock, or overlay-onto-moving-target?
|
||||
|
||||
---
|
||||
|
||||
## 12. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Obsidian API — App Architecture (https://www.mintlify.com/obsidianmd/obsidian-api/concepts/app-architecture) | App singleton, four modules, this.app |
|
||||
| obsidianmd/obsidian-api README (https://github.com/obsidianmd/obsidian-api/blob/master/README.md) | Plugin lifecycle, Vault/MetadataCache/Workspace/FileManager APIs, registrations, manifest fields |
|
||||
| DeepWiki — Vault and File System; MetadataCache and Link Resolution (https://deepwiki.com/obsidianmd/obsidian-api) | Vault CRUD + events; MetadataCache parsed elements, resolved/unresolved links, backlinks |
|
||||
| obsidianstats.com — Most Downloaded Plugins (https://www.obsidianstats.com/most-downloaded) | All-time download ranks + counts for the popularity → UC mapping (§7) |
|
||||
| Obsidian — "The future of plugins" (https://obsidian.md/blog/future-of-plugins/) | Plugin distribution model, restricted mode context |
|
||||
| Obsidian Help / JSON Canvas (jsoncanvas.org) | Vault format, `.obsidian/` config, Canvas open format |
|
||||
|
||||
Cross-references: `research/260614-roam-deep-dive/findings.md` (DB/API contrast, native
|
||||
span IDs, query delegation), `research/260613-twiki-deep-dive/findings.md` (file-store
|
||||
attach UC-40), `spec/UseCaseCatalog.md` (UC-02, UC-05, UC-15, UC-18, UC-28, UC-31,
|
||||
UC-32, UC-34, UC-36, UC-38, UC-40, UC-51, UC-52), `workplans/SHARD-WP-0002-federation-architecture.md` (T14).
|
||||
|
||||
---
|
||||
|
||||
## 13. Traceability
|
||||
|
||||
- New UCs: **UC-53, UC-54, UC-55, UC-56** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-15, UC-28, UC-34, UC-36, UC-40, UC-51, UC-52** (and links UC-02,
|
||||
UC-05, UC-18, UC-32, UC-38).
|
||||
- Architecture (no UC): dual attachment mode; consume-native-derived-index capability;
|
||||
non-Markdown content types in the page model; outbound publish; external-writer
|
||||
tolerance → `SHARD-WP-0002` (T14).
|
||||
- Distinctive artifact: **plugin-popularity → UC mapping** (§7) — ecosystem demand
|
||||
evidence, per the research brief.
|
||||
- Boundary recorded: an Obsidian vault is **one file-backed candidate shard** (the
|
||||
cleanest, INTENT-named), mapped into the Markdown-first page model; not the federation
|
||||
layer and not a file-sync target (INTENT Stability Note + not-a-sync-daemon).
|
||||
</content>
|
||||
13
research/260614-oddmuse-deep-dive/README.md
Normal file
13
research/260614-oddmuse-deep-dive/README.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# 260614 — Oddmuse deep dive
|
||||
|
||||
Deep dive on **Oddmuse**: a deliberately minimal wiki — a **single Perl CGI script** over
|
||||
**plain-text page files** (one file per page, old revisions in a `keep/` dir), no database.
|
||||
The low-complexity **file-store floor** — useful as the **minimal-adapter / graceful-
|
||||
degradation baseline** of the capability profile.
|
||||
|
||||
- `findings.md` — the minimal model, storage layout, capability profile, INTENT mapping, UC
|
||||
seed (UC-82), architecture notes for SHARD-WP-0002, open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-82 (attach a **minimal flat-file wiki** — plain-text page files + a simple
|
||||
revision dir — as the **graceful-degradation baseline / minimal capability-profile floor**).
|
||||
Enriched UC-40/01/36/41. Feeds SHARD-WP-0002 T11 (minimal capability profile).
|
||||
124
research/260614-oddmuse-deep-dive/findings.md
Normal file
124
research/260614-oddmuse-deep-dive/findings.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# Oddmuse — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T7 · **Subject:** Oddmuse, Alex Schroeder's
|
||||
minimal single-script wiki (EmacsWiki, Community Wiki).
|
||||
|
||||
## Why this dive
|
||||
|
||||
After the structure/graph/SaaS far-ends, Oddmuse anchors the **opposite** corner: the
|
||||
*minimal* file-store wiki. It is the reference for shard-wiki's **graceful-degradation**
|
||||
promise — *a limited backend must still be usable* — and defines the **floor** of the
|
||||
capability profile (T11): what the simplest possible real wiki looks like as a shard.
|
||||
|
||||
## 1. The minimal model
|
||||
|
||||
- **One Perl CGI script** (`wiki.pl`) is the whole engine — drop it on any CGI host. No
|
||||
framework, no database, minimal dependencies.
|
||||
- **Plain-text page files**: each page is a text file in a **page directory** (`page/`),
|
||||
with a small header of metadata and the body; **old revisions** are kept in a **`keep/`**
|
||||
directory (recent history retained, older optionally expired).
|
||||
- **Locking** via lock files; edits append a new keep-revision.
|
||||
- **Markup:** simple wiki markup; **CamelCase** and **free links** `[[Like This]]`;
|
||||
InterWiki/near-links; tags and "clusters."
|
||||
- **No DB, no API** (beyond the CGI itself); content *is* the files on disk.
|
||||
|
||||
## 2. The shard view — the file-store floor
|
||||
|
||||
Because pages are **plain-text files on disk**, Oddmuse is **trivially attachable** as a
|
||||
**file-store shard** even though the engine offers nothing fancy:
|
||||
|
||||
- Read the `page/` files → pages (parse the tiny header + body).
|
||||
- Read `keep/` → recent revision history (import to the journal; note it may be **truncated**
|
||||
— older revisions can be expired, so history is *partial*).
|
||||
- Write = write a page file + a keep-revision (respecting the lock) — but the engine's own
|
||||
invariants (indexes) mean **write-through is best done via the engine or carefully**.
|
||||
|
||||
This is the **minimal capability profile**: file-store, page-granularity, plain-text,
|
||||
possibly-truncated history, no query, no structured fields, open editing. Everything richer
|
||||
in the synthesis matrix is measured *against this floor*.
|
||||
|
||||
## 3. Capability profile (the floor)
|
||||
|
||||
| Dimension (synthesis spectrum) | Oddmuse |
|
||||
|--------------------------------|---------|
|
||||
| Attachment mode | **file-store** (plain-text files); CGI, no API |
|
||||
| Addressing granularity | page = file |
|
||||
| Content identity | page name = filename |
|
||||
| Identity vs placement | name-bound |
|
||||
| Structure | none beyond tags/clusters; flat page space |
|
||||
| History | **`keep/` revisions — recent, possibly truncated** |
|
||||
| Merge model | lock-file; last-writer |
|
||||
| Native query | none |
|
||||
| Translation | simple wiki markup (→ Markdown translation needed) |
|
||||
| Write granularity | page (file) |
|
||||
| Operational envelope | a CGI script; tiny |
|
||||
| Access grant | open by default (optional password) |
|
||||
| Content opacity | transparent text |
|
||||
| Provenance | minimal (timestamp, optional username) |
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Graceful degradation** (INTENT): Oddmuse is the *definition* of the limited-backend case
|
||||
— still a perfectly good read/projection/overlay/backup shard via its files.
|
||||
- **Union without erasure**: even a minimal shard contributes pages with provenance; its
|
||||
**truncated history** must be surfaced honestly (don't imply full history when `keep/` is
|
||||
partial).
|
||||
- **Open wiki** (UC-01): Oddmuse is open-editing by default — the c2-era ethos.
|
||||
- **Markdown-first but backend-neutral**: its wiki markup needs translation to the
|
||||
Markdown-first page model (UC-42-style), a small lossy step.
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- **Partial history**: `keep/` may expire old revisions — the journal import must record that
|
||||
history is **truncated/partial**, not complete (a freshness/provenance honesty point).
|
||||
- **Minimal profile** means many capabilities are simply **absent** — the adapter advertises
|
||||
a sparse profile; the orchestrator must not assume query/structure/locking semantics it
|
||||
doesn't have (T11 capability-awareness in its purest form).
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Minimal flat-file wiki as the graceful-degradation baseline** (UC-82): plain-text files
|
||||
+ simple revision dir = the floor every richer profile extends.
|
||||
2. **Honest partial-history reporting** when a shard's revision store is truncated.
|
||||
3. **Sparse capability profile** handling — absence of a capability is first-class (T11).
|
||||
|
||||
## 5. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-82 | Attach a **minimal flat-file wiki** (plain-text page files + a simple revision dir, Oddmuse) as the **graceful-degradation baseline / minimal capability-profile floor**, surfacing **partial history** honestly | **new** |
|
||||
| — | plain-text file-store at the simple end | enrich **UC-40** |
|
||||
| — | open-editing wiki | enrich **UC-01** |
|
||||
| — | `keep/` plain-text revision history (possibly truncated) | enrich **UC-36** / **UC-41** |
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T11 (capability model):** Oddmuse defines the **minimal/floor profile** — file-store,
|
||||
page granularity, plain-text, **partial** history, no query/structure. Validate that the
|
||||
capability vocabulary can express **absence** cleanly and that the orchestrator degrades
|
||||
to read/projection/overlay/backup against it.
|
||||
- **History portability (T13):** `keep/` revisions import as journal entries but may be
|
||||
**truncated** — record completeness as metadata (full vs partial history).
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. How does shard-wiki represent a shard with **partial/truncated history** in the journal
|
||||
and provenance UI (UC-24) — explicit "history begins at" marker?
|
||||
2. Is write-through to a minimal CGI wiki (write page + keep-revision under its lock) ever
|
||||
sanctioned, or read/projection/overlay/backup only by default?
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- oddmuse.org — Oddmuse wiki (single-script install, `page/` + `keep/` storage, markup,
|
||||
CamelCase/free links, clusters/tags)
|
||||
- EmacsWiki / Community Wiki (Oddmuse in production)
|
||||
- prior: `research/260608-c2-wiki-origins/` (open-wiki ethos); `research/260613-twiki-deep-dive/`
|
||||
(file-store + RCS contrast)
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
New UC **UC-82** carries the marker **⊚** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-40, UC-01, UC-36, UC-41. Architecture cross-refs:
|
||||
SHARD-WP-0002 T11 (minimal/floor profile), T13 (partial-history import).
|
||||
30
research/260614-processing-deep-dive/README.md
Normal file
30
research/260614-processing-deep-dive/README.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# 260614 — Processing & Processing.js deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T4**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Processing** (creative coding) and **Processing.js / p5.js** (browser-run
|
||||
sketches): **the sketch *is* the document** — a program whose presentation is **live visual
|
||||
output rendered at view time** in the browser, with **no cached output** by default.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- The cleanest **program-as-page** case: canonical content = **source text**, presentation =
|
||||
**executable render** (no input/output cells, no prose envelope) — sharpens the page model
|
||||
(T12/T15) and UC-54/55.
|
||||
- Adds a **view-time** variant to derivation-projection (the render runs **in the viewer,
|
||||
continuously**) and a **continuity** facet (one-shot vs continuous/interactive); continuous
|
||||
→ static is a **snapshot frame/recording** on the live↔snapshot axis (T6).
|
||||
- "Execute/render in the viewer" = an explicit **capability + trust/sandbox** surface.
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (enrichment / design prior art). Enriches **UC-54, UC-55**; links UC-83,
|
||||
UC-84, UC-35.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Program-as-page, view-time/live render, no-cached-output, capability+trust, INTENT mapping, UC disposition (enrichment-only), architecture notes |
|
||||
114
research/260614-processing-deep-dive/findings.md
Normal file
114
research/260614-processing-deep-dive/findings.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# Processing & Processing.js — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T4 · **Subject:** Processing (creative
|
||||
coding) and Processing.js / p5.js — **the sketch *is* the document**, rendered live at view
|
||||
time in the browser.
|
||||
|
||||
## Why this dive
|
||||
|
||||
Low-priority, enrichment-focused. Processing tests a page-model edge the notebooks (T2/T3)
|
||||
didn't: a page that is **wholly a program whose presentation *is* its running output** — no
|
||||
separate input/output cells, no prose envelope. The defining question: what is a "page" when
|
||||
**the rendered form is a live computation evaluated at view time**? It feeds UC-54
|
||||
(computation-defined content) and UC-55 (non-Markdown / executable content) and sharpens the
|
||||
**live↔snapshot** axis named at T6.
|
||||
|
||||
## 1. Program-as-page
|
||||
|
||||
- A **Processing sketch** is a program (`setup()` + `draw()`) whose output is a **canvas of
|
||||
visual/animated/interactive content**. There are no "output cells" — **the program's
|
||||
execution is the content**.
|
||||
- **Processing.js** (Resig, 2008; now largely **p5.js**) runs sketches **client-side in the
|
||||
browser** on `<canvas>`. The page ships the **source**; the **rendering happens at view
|
||||
time** in the reader's browser — no server, no pre-rendered artifact required.
|
||||
- So the durable artifact is **text (the sketch source)**; the *presentation* is a **live,
|
||||
view-time derivation** of that source (a derivation-projection, T1) — with the twist that
|
||||
the derivation runs **in the viewer**, continuously (animation/interaction), not once.
|
||||
|
||||
## 2. The view-time / live-render dimension
|
||||
|
||||
This adds a wrinkle beyond Jupyter's *captured* outputs (UC-84) and Mathematica's CDF:
|
||||
|
||||
- **No captured output at all by default** — unlike a notebook, a sketch typically stores
|
||||
**only source**; there is nothing cached. The output exists only when executed.
|
||||
- **Continuous & interactive** — `draw()` loops; mouse/keyboard drive it. The output is
|
||||
**time-based and interaction-based**, so any static capture is a **single frame** (or a
|
||||
recording) — it cannot represent the artifact faithfully (shared limit with Mathematica
|
||||
`Dynamic` and, more extremely, Strudel T5).
|
||||
- **Client-side execution = a capability + trust surface** — running arbitrary sketch code in
|
||||
the viewer is an execution capability with sandboxing concerns; shard-wiki must treat
|
||||
"render live in the viewer" as an explicit, gated capability, never an automatic behavior.
|
||||
|
||||
## 3. INTENT mapping (enrichment-only — no new UC)
|
||||
|
||||
### Reinforcements / refinements
|
||||
|
||||
- **Executable-as-page (UC-54/55).** Processing is the cleanest **"the whole page is the
|
||||
program"** case: content = source text, presentation = view-time live render. Strengthens
|
||||
the page model's need to represent **executable content whose rendered form is derived at
|
||||
view time**, distinct from notebooks (which interleave cells + captured outputs).
|
||||
- **Derivation-projection, view-time variant (T1/UC-83).** The render is a derivation-
|
||||
projection that runs **in the viewer, continuously** — extends the projection model: a
|
||||
derivation may be *materialized ahead* (CDF/nbconvert) or *run at view time* (sketch).
|
||||
- **Live↔snapshot axis (T6).** With **no cached output** and **continuous/interactive**
|
||||
rendering, the only static form is a **snapshot frame or recording** — a concrete point on
|
||||
the live↔snapshot axis; honest treatment offers the snapshot and **marks it as a frame of a
|
||||
live artifact**, never as the artifact.
|
||||
- **Capability + trust gating.** "Execute in the viewer" is an explicit capability with a
|
||||
sandbox/trust boundary — mechanism-over-policy: whether to run, sandbox, or only snapshot
|
||||
stays configurable.
|
||||
|
||||
### Boundaries
|
||||
|
||||
- shard-wiki is **not a sketch runtime**; default is **attach the source + offer a captured
|
||||
snapshot/recording**; live in-viewer rendering is a gated capability. Source is canonical;
|
||||
the render is a degrading, view-time projection.
|
||||
|
||||
## 4. UC disposition (enrichment-only)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC / thread |
|
||||
|------------------------|---------------------|
|
||||
| Sketch = program-as-page; presentation = view-time live render (§1) | UC-54 (enriched: executable/view-time content), UC-55 (enriched: non-Markdown executable) |
|
||||
| Render = derivation-projection run **in the viewer, continuously** (§1, §2) | links UC-83 (view-time variant) |
|
||||
| No cached output; continuous/interactive → static = snapshot frame only (§2) | links UC-84, live↔snapshot axis (T6) |
|
||||
| Client-side execution = capability + trust/sandbox surface (§2) | links UC-35 (capability-awareness) |
|
||||
|
||||
**No new UC** — Processing is design prior art reinforcing executable-as-page; its
|
||||
contribution is the **view-time, no-cached-output, continuous-render** point on the
|
||||
projection/liveness model.
|
||||
|
||||
## 5. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12/T15 (page model):** represent **program-as-page** — a page whose canonical content is
|
||||
**source text** and whose presentation is an **executable render**; no inherent cached
|
||||
output (contrast notebooks). Non-Markdown executable content type.
|
||||
- **T16 (projection):** add a **materialization timing** facet to derivation-projection:
|
||||
**ahead-of-time** (CDF/nbconvert/static HTML) vs **at view time** (sketch render); and a
|
||||
**continuity** facet: one-shot vs continuous/interactive. Continuous → static is a snapshot
|
||||
frame/recording on the live↔snapshot axis (T6).
|
||||
- **T11 (capabilities):** "execute/render in the viewer" is a capability with a **trust/
|
||||
sandbox** sub-concern; default off → snapshot.
|
||||
|
||||
## 6. Open questions
|
||||
|
||||
1. Is **view-time execution** ever offered (sandboxed in-viewer render), or does shard-wiki
|
||||
always degrade program-as-page to a captured snapshot/recording? (Capability/trust policy.)
|
||||
2. Should **materialization timing** (ahead-of-time vs view-time) and **continuity**
|
||||
(one-shot vs continuous) be explicit projection metadata, alongside the live↔snapshot
|
||||
axis? (Recurs at T5.)
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- `processing.org`; **p5.js** (`p5js.org`); Processing.js (Resig) history; `<canvas>` /
|
||||
client-side rendering model.
|
||||
- prior: `research/260614-jupyter-deep-dive/` (captured vs no-cached output, UC-84);
|
||||
`research/260614-mathematica-deep-dive/` (`Dynamic`/CDF, snapshot-only);
|
||||
`research/260614-squeak-pharo-deep-dive/` (live↔snapshot axis).
|
||||
|
||||
## 8. Traceability
|
||||
|
||||
**No new UC** (enrichment / design prior art). Enriched: UC-54, UC-55; links UC-83 (view-time
|
||||
derivation-projection), UC-84 (no-cached-output contrast), UC-35 (execute capability +
|
||||
trust). Architecture cross-refs: SHARD-WP-0002 T12/T15 (program-as-page, source-canonical/
|
||||
render-derived), T16 (materialization-timing + continuity facets on the live↔snapshot axis),
|
||||
T11 (view-time-execute capability + sandbox).
|
||||
15
research/260614-quip-deep-dive/README.md
Normal file
15
research/260614-quip-deep-dive/README.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# 260614 — Salesforce Quip deep dive
|
||||
|
||||
Deep dive on **Salesforce Quip**: a closed SaaS of **live collaborative documents** with
|
||||
**embedded spreadsheets and live apps**, a REST API (HTML import/export), and
|
||||
**Salesforce-tied identity/permissions**. A hosted-collab contrast to Notion: the
|
||||
document+spreadsheet hybrid under enterprise auth.
|
||||
|
||||
- `findings.md` — the doc/spreadsheet model, REST API, enterprise auth, capability profile,
|
||||
INTENT mapping, UC seed (UC-80), architecture notes for SHARD-WP-0002, open questions,
|
||||
sources, traceability.
|
||||
|
||||
Catalog yield: UC-80 (attach a SaaS live-doc shard whose pages **mix prose + embedded live
|
||||
structured objects** — spreadsheets/apps — via REST with lossy HTML import/export, under
|
||||
enterprise identity). Enriched UC-57/55/58/36/06. Feeds SHARD-WP-0002 T11 (capability /
|
||||
content opacity), T14 (external-API binding).
|
||||
148
research/260614-quip-deep-dive/findings.md
Normal file
148
research/260614-quip-deep-dive/findings.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Salesforce Quip — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T6 · **Subject:** Quip (Salesforce), a
|
||||
collaborative-document SaaS.
|
||||
|
||||
## Why this dive
|
||||
|
||||
Notion (UC-57/58/59) gave us the closed-SaaS, external-REST-only, database-as-pages model.
|
||||
Quip is the **enterprise document-collab** contrast: a **document+spreadsheet hybrid** where
|
||||
a page is **prose interleaved with embedded live structured objects** (spreadsheets,
|
||||
calendars, kanban "live apps"), reachable only through a REST API, gated by **Salesforce
|
||||
identity**. The question: how does shard-wiki attach a shard whose "page" is a *mixed
|
||||
prose+live-object document* behind enterprise auth?
|
||||
|
||||
## 1. The document + live-object model
|
||||
|
||||
- A Quip **document** is a real-time collaborative doc (concurrent editing). Its body is
|
||||
rich content — headings, lists, prose — **interleaved with embedded objects**:
|
||||
- **Spreadsheets** are *first-class, inline* — a doc can contain live spreadsheet sections
|
||||
with formulas, not just static tables.
|
||||
- **Live apps** (calendars, kanban boards, project trackers, polls) embed interactive
|
||||
structured widgets inside the document.
|
||||
- **Folders** organize documents; **threads** (docs and chat are unified — every doc is also
|
||||
a conversation thread) carry **messages/comments** inline.
|
||||
- So a Quip "page" is a **hybrid**: prose + embedded structured/live content in one
|
||||
document, with conversation attached. Not Markdown; a proprietary rich model.
|
||||
|
||||
## 2. The REST API (the only door)
|
||||
|
||||
Quip exposes a **REST API**: threads/documents (get, create, **edit-document** with an HTML
|
||||
fragment + a location/section anchor), folders, messages, users, and spreadsheet cell
|
||||
access. Content crosses the API as **HTML** (import and export) — there is **no native
|
||||
Markdown** and no file/git access. Implications:
|
||||
|
||||
- **Import/export is HTML → lossy** to Markdown (like Notion's export, UC-59): embedded
|
||||
spreadsheets and live apps **do not round-trip** to Markdown cleanly — they degrade to
|
||||
tables or links.
|
||||
- Editing is **section/anchor-scoped HTML splice** (`edit-document` targets a location) — a
|
||||
mid-granularity write (a section, not the whole doc, not a character).
|
||||
- **Rate-limited** like any SaaS API; history is internal (revisions exist in-product, with
|
||||
limited API exposure).
|
||||
|
||||
## 3. Enterprise identity / permissions
|
||||
|
||||
Quip is tied to **Salesforce** (acquired 2016): authentication and access run through the
|
||||
Salesforce org / Quip site; **folder and document sharing ACLs** govern visibility, with
|
||||
enterprise SSO. So this is an **enterprise-ACL, authn-delegated** shard (UC-06) — shard-wiki
|
||||
honors Salesforce-side permissions and never bypasses them.
|
||||
|
||||
## 4. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | Quip |
|
||||
|--------------------------------|------|
|
||||
| Attachment mode | **external-API** (REST; HTML import/export) — closed SaaS |
|
||||
| Addressing granularity | document; **section/anchor** for edits; spreadsheet cell |
|
||||
| Content identity | Quip thread/document ID (opaque) |
|
||||
| Identity vs placement | API-id identity; folder placement separate |
|
||||
| Structure | **hybrid doc**: prose + embedded spreadsheets + live apps + thread |
|
||||
| History | **internal** revisions (limited API exposure) |
|
||||
| Merge model | server-side real-time collab (OT-like); no external merge |
|
||||
| Native query | none exposed (spreadsheet formulas internal) |
|
||||
| Translation | **HTML** in/out → **lossy** to Markdown (spreadsheets/apps degrade) |
|
||||
| Write granularity | **section/anchor HTML splice** (mid) |
|
||||
| Operational envelope | rate-limited SaaS REST |
|
||||
| Access grant | **Salesforce identity + folder/doc ACL** (enterprise SSO) |
|
||||
| Content opacity | proprietary rich model; not E2EE but not transparent files |
|
||||
| Provenance | author/edit metadata in-product; API-limited |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **External-API attachment** (UC-57): Quip is a second concrete instance beside Notion of
|
||||
the closed-SaaS REST-only shard — generalizes the external-API mode (REST + HTML payload,
|
||||
vs Notion's REST + block-JSON, vs Wiki.js GraphQL).
|
||||
- **Union without erasure / no silent flatten**: embedded spreadsheets and live apps must be
|
||||
**surfaced as what they are** with provenance, and the **HTML→Markdown lossiness made
|
||||
explicit** (UC-59) — never silently drop a live spreadsheet to a static table.
|
||||
- **Authz-in-core, authn-delegated** (settled decision): Quip's Salesforce-tied ACL is the
|
||||
enterprise case — honor delegated identity and the shard's ACL (UC-06).
|
||||
- **Graceful degradation**: with only a lossy HTML export, Quip is still usable as a
|
||||
read/projection/overlay-target/backup shard.
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- **Mixed prose+live-object page** is a content shape beyond "Markdown body + frontmatter":
|
||||
the page model must allow **embedded typed/live objects within a prose page** (not just a
|
||||
whole-page-is-a-record like Notion DB, but *inline* structured content) — feeds T12 and the
|
||||
non-Markdown-content question (UC-55).
|
||||
- **HTML is the only interchange** — no Markdown, no files, no git. Content opacity is
|
||||
"proprietary-but-exportable": transparent-ish via lossy HTML, not via files (T11
|
||||
content-opacity tier between transparent-files and E2EE-opaque).
|
||||
- **Write granularity = section-anchor HTML splice** — a mid tier (between whole-file and
|
||||
block/character) realized over an API, distinct from fedwiki's story-item op-log.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **External-API mode generalized** to carry an **HTML payload** variant (Quip) beside
|
||||
block-JSON (Notion) and GraphQL (Wiki.js) — capability/payload-format is part of the
|
||||
adapter profile (UC-57/UC-80).
|
||||
2. **Inline embedded live/structured objects** as a page-model element (prose + embedded
|
||||
spreadsheet/app), with **explicit lossy projection** to Markdown (UC-55/UC-58/UC-59).
|
||||
3. **Enterprise-ACL + delegated identity** honored, not bypassed (UC-06).
|
||||
|
||||
## 6. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-80 | Attach a **SaaS live-doc shard** whose pages **mix prose + embedded live structured objects** (spreadsheets / live apps) via **REST with lossy HTML import/export**, under **enterprise (Salesforce) identity** | **new** |
|
||||
| — | external-API mode w/ HTML payload variant | enrich **UC-57** |
|
||||
| — | inline embedded spreadsheet/live-app = non-Markdown content | enrich **UC-55** |
|
||||
| — | embedded structured objects within a prose page | enrich **UC-58** |
|
||||
| — | internal (API-limited) revision history | enrich **UC-36** |
|
||||
| — | Salesforce-tied enterprise ACL + SSO | enrich **UC-06** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T11 (capability / content opacity):** add a **payload-format** facet to external-API
|
||||
shards (HTML / block-JSON / GraphQL) and a **content-opacity tier** "proprietary but
|
||||
lossy-exportable" between transparent-files and E2EE-opaque. Write granularity =
|
||||
section-anchor splice.
|
||||
- **T12 (page model):** support **inline embedded structured/live objects** within a prose
|
||||
page (not only whole-page-as-record) — with explicit lossy render to Markdown.
|
||||
- **T14 (binding):** external-API binding with **HTML import/export**; honor Salesforce
|
||||
identity/ACL; default to read/projection/overlay given rate limits + lossy export.
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Does shard-wiki represent an **embedded live spreadsheet** as a typed sub-object with
|
||||
provenance (preferred) or flatten it to a static Markdown table (lossy) — and can overlays
|
||||
target a spreadsheet cell via the API, or only the prose?
|
||||
2. Given **HTML-only, lossy** interchange and rate limits, is Quip ever a **write-through**
|
||||
shard, or read/projection/overlay/backup by default? (cf. Notion Q1, UC-57.)
|
||||
3. How is **Salesforce identity** mapped to shard-wiki's delegated-authn model — pass-through
|
||||
token, service account, or per-user OAuth?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- Quip Automation API (REST) docs — quip.com / Salesforce developer docs (threads/documents,
|
||||
`edit-document`, folders, messages, spreadsheets)
|
||||
- Salesforce Quip product docs (live apps, spreadsheets, sharing/permissions)
|
||||
- prior: `research/260614-notion-deep-dive/` (closed-SaaS external-API contrast, UC-57/59)
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UC **UC-80** carries the marker **◧** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-57, UC-55, UC-58, UC-36, UC-06. Architecture
|
||||
cross-refs: SHARD-WP-0002 T11 (payload-format + content-opacity), T12 (inline objects), T14.
|
||||
49
research/260614-roam-deep-dive/README.md
Normal file
49
research/260614-roam-deep-dive/README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# 260614 — Roam Research deep dive (the block graph as a queryable database + extension API)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Roam Research** — core architecture and extension structure —
|
||||
read through shard-wiki's lens. Roam is the **modern bookend** to the two Nelson dives:
|
||||
where Xanadu (`research/260614-xanadu-deep-dive/`) and ZigZag
|
||||
(`research/260614-zigzag-deep-dive/`) are *unbuilt ideals*, Roam **shipped** stable
|
||||
fine-grained addressing, transclusion, bidirectional links, and a queryable structured
|
||||
space to a mainstream audience.
|
||||
|
||||
Distinctive material:
|
||||
- **Data model** — the entire graph is a client-side **DataScript** (Datomic-like) DB
|
||||
of **datoms** (EAV); every paragraph and page is a **block** entity with a stable
|
||||
nine-character **`:block/uid`**, `:block/string`, `:block/refs`, `:block/page`,
|
||||
pages marked by `:node/title`
|
||||
- **References & transclusion** — `[[Page]]`/`#tag`/`((uid))` create `:block/refs`
|
||||
edges; **block embeds** are working transclusion; linked vs unlinked references
|
||||
- **Querying** — Datalog (`:q`/`:pull`); derived views are queries over the graph
|
||||
- **Extension architecture** — **Roam Depot** (`extension.js` `onload`/`onunload`,
|
||||
manifest, `extensionAPI` settings) over **`window.roamAlphaAPI`** read (`q`/`pull`) +
|
||||
write (`block.create/update/move/delete`, `page.create`) — a concrete
|
||||
*engine-hosts-adapter* (UC-38) surface
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Data model, references/transclusion, Datalog, extension API, Roam-as-shard capability profile, evidence on shard-wiki's open questions, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-50 attach a block-graph DB shard with block↔page mapping, UC-51 adopt native
|
||||
block/span IDs as portable span addresses, UC-52 delegate derived views to a shard's
|
||||
native query engine); UC-32/34/35/38 enriched. Logged for
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T14): native-span-ID and
|
||||
native-query capabilities, block↔page mapping, and Roam as a second DB-backed /
|
||||
engine-hosts-adapter exemplar alongside XWiki.
|
||||
|
||||
**Key payoff recorded:** Roam empirically answers the Nelson dives' open questions —
|
||||
fine-grained addressing is tractable (block UIDs), transclusion is a data-layer
|
||||
capability over an addressable union, and derived views are queries over a structured
|
||||
space. **Boundary:** Roam is *one candidate shard* (DB-backed/API-attached, block-first,
|
||||
no portable Git history), mapped into shard-wiki's Markdown-first page model — not a
|
||||
substrate and not the federation layer.
|
||||
</content>
|
||||
292
research/260614-roam-deep-dive/findings.md
Normal file
292
research/260614-roam-deep-dive/findings.md
Normal file
@@ -0,0 +1,292 @@
|
||||
# Findings — Roam Research: the block graph as a queryable database, and its extension API
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — a block-graph note tool; a *candidate shard
|
||||
backend* (DB-backed / API-attached, like XWiki), and the place where several Nelson
|
||||
ideas actually shipped
|
||||
Lens: shard-wiki — fine-grained addressing, transclusion, bidirectional links, a
|
||||
structured/queryable page space, and the engine-hosts-adapter extension path
|
||||
|
||||
> Why Roam, and why now. The previous two dives were Ted Nelson's *unbuilt* ideals —
|
||||
> Xanadu (`research/260614-xanadu-deep-dive/`: reference-not-copy, transclusion,
|
||||
> stable fine-grained addresses) and ZigZag (`research/260614-zigzag-deep-dive/`: an
|
||||
> information space as many co-equal dimensions over a queryable structure). **Roam
|
||||
> Research is where a large slice of both actually shipped to a mainstream audience.**
|
||||
> Every block has a stable address; block references and embeds are working
|
||||
> transclusion; the whole graph is a queryable database; links are bidirectional. So
|
||||
> Roam is the natural modern bookend: it lets us check shard-wiki's open questions
|
||||
> (portable span addressing, transclusion, structured pages) against a system that
|
||||
> solved them *in production* — and study its **extension architecture** as a concrete
|
||||
> adapter-host surface.
|
||||
|
||||
This dive treats Roam as a *candidate shard* (capability profile in §6) and as
|
||||
*evidence* on the open questions the Nelson dives left (§7), not as something to copy
|
||||
wholesale — Roam is block-first and DB-backed, shard-wiki is Markdown-page-first and
|
||||
Git-backed (divergences in §8.2).
|
||||
|
||||
---
|
||||
|
||||
## 1. What Roam is, in architecture terms
|
||||
|
||||
Roam Research is a networked-thought outliner whose **entire graph is a client-side
|
||||
DataScript database** (DataScript = a ClojureScript reimplementation of Datomic's
|
||||
data model and Datalog query engine). The store is not files; it is a set of immutable
|
||||
**datoms**.
|
||||
|
||||
- A **datom** is one fact: `[entity, attribute, value, transaction]` — the EAV(T)
|
||||
model. The graph is the accumulation of datoms; edits are transactions.
|
||||
- Synchronization and Roam's deep undo fall out of the transaction log — a parallel
|
||||
worth noting to Git's immutable commit history (but it is *not* Git; see §8.2).
|
||||
|
||||
This makes Roam a **structured, queryable** wiki-shaped store — the same family as
|
||||
XWiki (objects + DB) rather than TWiki/Foswiki (flat files), which matters for how an
|
||||
adapter attaches (§6).
|
||||
|
||||
---
|
||||
|
||||
## 2. The block model — everything is a block, every block is addressable
|
||||
|
||||
Every paragraph *and* every page is a **block** (an entity). Key attributes:
|
||||
|
||||
| Attribute | Meaning |
|
||||
|-----------|---------|
|
||||
| `:block/uid` | **nine-character public ID** — the block's stable, referenceable address (e.g. `((GGv3cyL6Y))`) |
|
||||
| `:block/string` | the block's text content |
|
||||
| `:block/order` | position among siblings |
|
||||
| `:block/children` | immediate child entity-ids |
|
||||
| `:block/parents` | full ancestor chain (incl. the page) |
|
||||
| `:block/refs` | edges to blocks/pages this block references |
|
||||
| `:block/page` | the owning page entity |
|
||||
| `:create/time`, `:edit/time`, `:create/email`, `:edit/email` | provenance metadata |
|
||||
| `:node/title` | **page-only** — the page's title (this attribute is what distinguishes a page from an ordinary block) |
|
||||
|
||||
Structurally the DB is a **forest**: each page is a tree of nested blocks; a "page" is
|
||||
just the top-level block that carries `:node/title`. The single most shard-wiki-relevant
|
||||
fact: **`:block/uid` gives every block — i.e. every sub-page span — a stable,
|
||||
first-class, public address.** This is the *shipped* form of the Xanadu tumbler / the
|
||||
shard-wiki "portable span address" open question (§7).
|
||||
|
||||
---
|
||||
|
||||
## 3. References, transclusion, bidirectional links
|
||||
|
||||
- **Page references** `[[Page]]` and `#tags`, and **block references** `((uid))`, all
|
||||
create `:block/refs` edges. The link graph is therefore queryable, not parsed-on-read.
|
||||
- **Block embeds** (`{{embed: ((uid))}}`) are **working transclusion**: a block's
|
||||
content rendered live in another location by reference, not copied — when the source
|
||||
changes, the embed reflects it. This is Xanadu transclusion / shard-wiki UC-32/UC-44,
|
||||
shipped at block granularity.
|
||||
- **Bidirectional links** come in two flavours, exactly shard-wiki's BackLinks problem:
|
||||
- **Linked references** — explicit `:block/refs` edges (queryable backlinks).
|
||||
- **Unlinked references** — bare text mentions with no edge, surfaced by text search.
|
||||
- **Attributes** `key:: value` are a convention on top of blocks/refs that turns a page
|
||||
into a lightweight record — Roam's answer to structured/typed pages (UC-34/UC-39),
|
||||
queryable via Datalog.
|
||||
|
||||
---
|
||||
|
||||
## 4. Querying — Datalog over the graph
|
||||
|
||||
Roam exposes **Datalog** (`:find` / `:where`) plus **pull** expressions:
|
||||
|
||||
```
|
||||
[:find ?uid :where [?b :block/string ?s] [?b :block/uid ?uid] ...]
|
||||
```
|
||||
|
||||
This means Roam's "derived views" (a page's linked references, a `{{query}}` block, a
|
||||
table) are **just queries over the datom graph** — not bespoke features. For shard-wiki
|
||||
this is the strongest evidence yet that *derived views = queries over a structured
|
||||
space* (the ZigZag "dimensions + rasters" insight, made executable), and that an
|
||||
adapter could **delegate view computation to a shard's native query engine** instead of
|
||||
scanning (UC-52).
|
||||
|
||||
---
|
||||
|
||||
## 5. Extension architecture — Roam Depot and `window.roamAlphaAPI`
|
||||
|
||||
Roam's official extension system (**Roam Depot**) is a concrete model of the
|
||||
*engine-hosts-adapter* path (UC-38):
|
||||
|
||||
- **Distribution:** an extension is a GitHub repo (`README.md`, `extension.js`, optional
|
||||
`extension.css` / `CHANGELOG.md` / `build.sh`), cataloged in the `roam-depot` repo via
|
||||
a metadata JSON (`name`, `author`, `tags`, `source_repo`, `source_commit`, optional
|
||||
`stripe_account` for paid extensions).
|
||||
- **Lifecycle:** `extension.js` default-exports a map with **`onload`** and
|
||||
**`onunload`** functions; everything created on load must be torn down on unload. An
|
||||
`extensionAPI` object provides a **settings panel** and `settings.get/set`.
|
||||
- **Data API — `window.roamAlphaAPI`** (the surface an adapter would target):
|
||||
- **Read:** `q` (Datalog query → entity ids), `pull` / `pull-many` (fetch entity
|
||||
attributes), full graph traversal.
|
||||
- **Write:** `data.block.create` / `update` / `move` / `delete`, `data.page.create`.
|
||||
- **UI:** command palette, blocks, settings.
|
||||
|
||||
Crucially, this API runs **inside the Roam client** (global `window.roamAlphaAPI`).
|
||||
There is no first-class external REST write API; programmatic external access uses
|
||||
graph **export** (EDN / JSON / Markdown) or unofficial private APIs. So a *live,
|
||||
write-through* Roam adapter must be **hosted inside Roam as a Depot extension** — which
|
||||
is exactly the UC-38 engine-side direction (cf. XWiki components, TWiki plugin handlers).
|
||||
|
||||
---
|
||||
|
||||
## 6. Roam as a shard — capability profile
|
||||
|
||||
| Capability | Roam | Notes for the adapter contract |
|
||||
|------------|------|--------------------------------|
|
||||
| Read | **yes** | `q`/`pull` live (in-app) or EDN/JSON/Markdown export (snapshot) |
|
||||
| Write | **yes, in-app** | `block.create/update/move/delete`, `page.create` — but only via the in-client API → needs a Depot-hosted adapter (UC-38) or unofficial private API |
|
||||
| Write granularity | **block-level (fine)** | the opposite extreme from TiddlyWiki's whole-file writes — sharpens UC-35 |
|
||||
| Identity / addressing | **block `:block/uid` + page `:node/title`** | shipped sub-page stable addressing (UC-51) |
|
||||
| Transclusion | **yes** | block embeds by uid (UC-32/44/45) |
|
||||
| Backlinks | **yes** | linked (`:block/refs`) + unlinked (text) (UC-05/18) |
|
||||
| Structured data | **yes** | attributes + Datalog (UC-34/39) |
|
||||
| Native query | **yes (Datalog)** | adapter could delegate views (UC-52) |
|
||||
| Diff / merge | **no native** | transaction log exists but not exposed as page diffs |
|
||||
| Version history | **hosted, internal-only** | not portable Git history → needs supplement/import (UC-36, like Confluence/MediaWiki) |
|
||||
| Lock | **no** | — |
|
||||
| Syntax | **Roam-flavored Markdown** | not CommonMark; block-outline structure → translation/mapping needed (cf. UC-42) |
|
||||
|
||||
Verdict: Roam is a legitimate **DB-backed, API-attached shard** — the XWiki family, not
|
||||
the file-store family. Its standout offerings are **block-level addressing** and
|
||||
**shipped transclusion**; its costs are **block↔page impedance** and **no portable
|
||||
history**.
|
||||
|
||||
---
|
||||
|
||||
## 7. Evidence on shard-wiki's open questions (the payoff)
|
||||
|
||||
The Nelson dives left open questions; Roam answers several *empirically*:
|
||||
|
||||
1. **"What is a portable fine-grained span address?"** (Xanadu §11 Q1, the tumbler
|
||||
problem). Roam's answer: **a short opaque per-block UID, minted by the store, public
|
||||
and referenceable.** shard-wiki lesson — fine-grained addressing is *tractable* when
|
||||
the backend mints stable block/span IDs; the adapter should **adopt native block IDs
|
||||
as span addresses where they exist** (UC-51), and fall back to content-fingerprint or
|
||||
path+range where they do not (cf. Xanadu content-identity, UC-46).
|
||||
2. **"Does transclusion belong in core or adapter/UI?"** (catalog Q6). Roam shows
|
||||
transclusion is cheap and natural *when the store is addressable and queryable*; it
|
||||
lives at the **data layer** (refs/embeds), surfaced by UI. Argues for transclusion as
|
||||
a **core capability over an addressable union**, not a UI-only trick.
|
||||
3. **"Derived views — core or adapter?"** Roam shows them as **queries over a
|
||||
structured space**. Where a shard exposes a native query engine, **delegate**
|
||||
(UC-52); where it does not, the orchestrator computes them over the projection.
|
||||
4. **Block↔page mapping** is the new question Roam *raises*: in a block graph, what is a
|
||||
"page"? Roam's own answer (`:node/title` node = page; nested blocks = spans) is a
|
||||
clean mapping rule for the adapter (UC-50).
|
||||
|
||||
---
|
||||
|
||||
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 8.1 Reinforcements
|
||||
|
||||
- **Fine-grained addressing is achievable** (block UIDs) — de-risks UC-44/45/51 and the
|
||||
Xanadu tumbler worry.
|
||||
- **Transclusion + bidirectional links + structured data** are not exotic; a shipped
|
||||
tool does all three over one addressable, queryable store — validating shard-wiki's
|
||||
ambitions for UC-32/34/39/05.
|
||||
- **Engine-hosts-adapter (UC-38)** has a clean modern template: Depot's
|
||||
`onload/onunload` + `roamAlphaAPI` read/write surface.
|
||||
|
||||
### 8.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Block-DB substrate vs. Markdown-file page model.** Roam's "everything is a block in
|
||||
a DataScript DB" must **not** tempt shard-wiki away from its Markdown-first,
|
||||
backend-neutral page model (INTENT Stability Note). Roam is *one shard shape*, mapped
|
||||
*into* the page model, not the model itself.
|
||||
2. **Client-side proprietary DB vs. Git coordination journal.** Roam's transaction log
|
||||
is not portable history; shard-wiki keeps Git as the coordination layer. A Roam shard
|
||||
contributes snapshots/projections, and its history needs **supplement/import**
|
||||
(UC-36), not adoption as canonical.
|
||||
3. **Single graph vs. federation.** A Roam graph is **one sovereign shard**, never the
|
||||
federation layer. Do not model the union as "a big Roam graph."
|
||||
4. **Roam-flavored Markdown + outline structure** ≠ CommonMark pages; the adapter owns a
|
||||
**lossy-aware translation** (block outline ↔ page + headings/lists), cf. UC-42.
|
||||
|
||||
### 8.3 What Roam teaches that shard-wiki should keep
|
||||
|
||||
- **Mint/adopt stable sub-page IDs.** The cheapest path to transclusion, overlay, and
|
||||
reverse-lookup at span granularity is a backend that already addresses spans — lean on
|
||||
it (UC-51), degrade gracefully otherwise.
|
||||
- **Treat a structured shard's query engine as a capability**, and delegate derived
|
||||
views to it when present (UC-52).
|
||||
- **A clean block↔page rule** (`:node/title` node = page) keeps a block backend usable
|
||||
without flattening (UC-50, complementing UC-34's no-lossy-flatten rule).
|
||||
|
||||
---
|
||||
|
||||
## 9. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-49**. New UCs **UC-50–UC-52** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a block-graph DB wiki (Roam-style) as a shard via its query/CRUD API, mapping blocks to the page model** (`:node/title` node = page; nested blocks = spans) | **UC-50 (new)** |
|
||||
| **Adopt a shard's native block/span IDs as portable span addresses** for transclusion/overlay (the shipped answer to the span-address question) | **UC-51 (new)** |
|
||||
| **Delegate derived views to a shard's native query engine** (Datalog) where advertised | **UC-52 (new)** |
|
||||
| Shipped transclusion via block embeds + shipped fine-grained addressing | **enriches UC-32** (and UC-44/45) |
|
||||
| Attributes + Datalog = shipped structured/typed pages | **enriches UC-34** (and UC-39) |
|
||||
| Roam Depot `onload/onunload` + `roamAlphaAPI` = modern engine-hosts-adapter template | **enriches UC-38** |
|
||||
| Block-level write granularity = the fine extreme opposite TiddlyWiki | **enriches UC-35** |
|
||||
| Hosted history is internal-only, not portable Git | links **UC-36** |
|
||||
|
||||
---
|
||||
|
||||
## 10. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- The adapter contract should model **"native span/block IDs"** as a capability: a shard
|
||||
advertises whether it mints stable sub-page addresses; transclusion/overlay/reverse-
|
||||
lookup capabilities key off it (ties UC-44/45/46/51).
|
||||
- The contract should model a **"native query"** capability so the orchestrator can
|
||||
delegate derived-view computation (UC-52) vs. scanning the projection.
|
||||
- A **block↔page mapping** belongs in the adapter, with the no-lossy-flatten rule of
|
||||
UC-34: page = titled node, blocks = addressable spans, attributes = sidecar metadata.
|
||||
- Roam joins XWiki as a **DB-backed / API-attached** exemplar (vs. TWiki/Foswiki file
|
||||
stores) and as a **second engine-hosts-adapter** template (vs. XWiki components / TWiki
|
||||
handlers) for the T14 adapter-binding task.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions (for spec / workplans)
|
||||
|
||||
1. When a backend mints native span IDs (Roam UIDs), are they used **directly** as the
|
||||
shard-wiki span address, or **wrapped** in a shard-scoped address so they survive
|
||||
projection/overlay and don't collide across shards?
|
||||
2. For a **write-through** Roam shard the adapter must run *inside* Roam (Depot
|
||||
extension). Is in-engine hosting an accepted adapter deployment mode generally, or do
|
||||
we restrict Roam to **read/projection/overlay-target** (graceful degradation) when we
|
||||
cannot deploy inside it?
|
||||
3. How lossy is the **block-outline ↔ Markdown-page** mapping, and is round-trip
|
||||
fidelity required or best-effort (cf. UC-42 Foswiki TML↔HTML)?
|
||||
4. Do we consume Roam's **export (EDN/JSON)** as the projection source, or the live
|
||||
`roamAlphaAPI`? Snapshot vs. live freshness trade-off (cf. UC-31).
|
||||
|
||||
---
|
||||
|
||||
## 12. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| zsolt.blog — "Deep Dive Into Roam's Data Structure" (https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html) | Datom/EAV model, block attributes (`:block/uid` etc.), refs/embeds, linked vs unlinked references, Datalog/pull |
|
||||
| Roam Research Datalog Cheatsheet (https://gist.github.com/2b3pro/231e4f230ed41e3f52e8a89ebf49848b) | Attribute names, `:q`/`:pull` query shapes |
|
||||
| GitHub — Roam-Research/roam-depot (https://github.com/Roam-Research/roam-depot) | Extension repo/manifest structure, `extension.js` `onload`/`onunload`, `extensionAPI` settings, `roamAlphaAPI` read/write/ui surface |
|
||||
| "Introduction to the Roam Alpha API" (https://www.putyourleftfoot.in/introduction-to-the-roam-alpha-api) | `window.roamAlphaAPI` `q`/`pull` semantics, global-scope dependency |
|
||||
| David Vargas — "How To Create Your Own Roam Extensions" (https://davidvargas.me/blog/how_to_create_your_own_roam_extensions) | Extension authoring lifecycle, write methods |
|
||||
| Wikipedia / general — Roam Research, DataScript, Datomic | Datalog/Datomic lineage, client-side DB framing |
|
||||
|
||||
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (span addressing,
|
||||
transclusion), `research/260614-zigzag-deep-dive/findings.md` (queryable dimensions),
|
||||
`research/260613-xwiki-deep-dive/findings.md` (DB-backed engine + extension host),
|
||||
`spec/UseCaseCatalog.md` (UC-05, UC-18, UC-32, UC-34/35/36/38/39, UC-44/45/46),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T14 adapter binding).
|
||||
|
||||
---
|
||||
|
||||
## 13. Traceability
|
||||
|
||||
- New UCs: **UC-50, UC-51, UC-52** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-32, UC-34, UC-35, UC-38** (and links UC-36, UC-39, UC-44/45/46).
|
||||
- Architecture (no UC): native-span-ID capability; native-query capability; block↔page
|
||||
mapping; Roam as DB-backed/API-attached + engine-hosts-adapter exemplar → `SHARD-WP-0002` (T14).
|
||||
- Boundary recorded: Roam is **one candidate shard**, mapped into the Markdown-first
|
||||
page model; not a substrate, not the federation layer (INTENT Stability Note).
|
||||
</content>
|
||||
69
research/260614-shard-spectrum-synthesis/README.md
Normal file
69
research/260614-shard-spectrum-synthesis/README.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# 260614 — Shard spectrum synthesis (one capability model across the full dive set)
|
||||
|
||||
Date: 2026-06-14 · **revised 2026-06-14 (v3)**
|
||||
|
||||
## What this is
|
||||
|
||||
A **synthesis** (no new external research) that reads every studied system *across* the
|
||||
others and distills them into a single comparative model feeding the **shard adapter
|
||||
contract** and the **federation track** (`SHARD-WP-0002`).
|
||||
|
||||
**v3** extends the model to the **SHARD-WP-0003 engine batch** (Federated Wiki, Wikibase/
|
||||
Wikidata, the git-forge wikis Gitea/GitLab/GitHub, TiddlyWiki, ikiwiki, Salesforce Quip,
|
||||
MojoMojo, Oddmuse, UseModWiki) — ~23 systems in all. Per-shard spectra grow from thirteen to
|
||||
**fourteen** (adds **provenance granularity**), and a **new coordination-layer axis** is
|
||||
introduced: the **federation-model taxonomy**.
|
||||
|
||||
Centerpieces:
|
||||
- **The shard family matrix** — every candidate backend × {substrate, attach mode(s),
|
||||
addressing, structure, history, merge, query, →Markdown, opacity}, with Xanadu/ZigZag as
|
||||
ideal anchors and Federated Wiki as the federation-model anchor. v3 adds the flat-file
|
||||
floor (Oddmuse/UseModWiki), git-IS-store (forge wikis/ikiwiki), TiddlyWiki, MojoMojo
|
||||
(direct-DB), Quip (external-API/HTML), and Wikibase (typed-graph far-end).
|
||||
- **The fourteen capability spectra** — the claim that the adapter contract should model
|
||||
*positions on spectra*, each anchored at both ends by a real system, with federation ops
|
||||
degrading by position. v3 adds **provenance granularity** (14th: per-shard → per-page →
|
||||
per-edit → per-statement/value), and refines merge-model (+fork/journal-replay,
|
||||
+coexist-with-rank), attachment-mode (+git-IS-store, +container, +direct-DB,
|
||||
+REST/file-store-hybrid, +external-API payload-format facet), native-query (+SPARQL/RDF
|
||||
far-end, +filter mid), history (+DB-version-rows, +partial/truncated), structure
|
||||
(+typed-graph, +inline-embedded objects), content-opacity (+proprietary-lossy-exportable),
|
||||
write-granularity (+story-item, +section-anchor).
|
||||
- **The federation-model taxonomy (§2.5, new)** — federation itself is plural: fork+journal
|
||||
(Federated Wiki), VCS-replication+ping (ikiwiki), query-time graph join (Wikibase SPARQL
|
||||
`SERVICE`), feed aggregation, activity streams, engine-mirror (Wiki.js). A
|
||||
selectable/composable coordination-layer axis feeding **T1–T6**.
|
||||
- **UC-44–UC-82 → workplan task mapping** (T1–T6 + T11–T16).
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Family matrix, the fourteen spectra, the federation-model taxonomy, cross-cutting through-lines, UC→task fold-in, recommendations/decisions, escalated open questions |
|
||||
|
||||
## Status
|
||||
|
||||
Synthesis v3 complete. No new use cases (consolidation only). Feeds `SHARD-WP-0002`:
|
||||
**T1–T6** federation-model taxonomy (selectable/composable federation); **T11** reframed
|
||||
around the **fourteen spectra** (incl. provenance granularity + expanded attachment modes +
|
||||
external-API payload-format facet); T12 page-model breadth (prose + typed/computed records +
|
||||
**inline-embedded objects** + **typed-graph statements** + non-Markdown assets +
|
||||
query-defined + multi-placement/DAG identity); T13 history (DB-rows + partial-flat-file =
|
||||
supplement; completeness metadata; **journal-shaped** coordination journal); T14 **full
|
||||
attachment taxonomy** (git-IS-store / container / direct-DB / REST-file-hybrid +
|
||||
source-of-truth per binding); T15 lossy + **not-Markdown** (graph/HTML); T16 (addressing
|
||||
incl. statement GUID + **opaque stable identity**, identity≠placement, derived index,
|
||||
**graph query / federated SERVICE**). UC coverage extended UC-34–UC-67 → **UC-34–UC-82**.
|
||||
|
||||
**New through-lines (v3):** federation is plural (the federation-model taxonomy); provenance
|
||||
has a granularity spectrum (down to per-statement, Wikibase rank/references); git is both the
|
||||
home store and the home journal (forge wikis make git *the* store, resolving the engine-
|
||||
mirror write-race); the flat-file floor (Oddmuse/UseModWiki, Wikipedia's MediaWiki Phase I)
|
||||
is the field's common root and the minimal capability profile; the page model must also carry
|
||||
typed-graph statements and inline-embedded objects.
|
||||
**Carried from v2:** CRDT changes the merge math; identity ≠ placement; metadata can be
|
||||
computed; content opacity is per-item; the attach surface is not always the native store; the
|
||||
block-graph-on-files sweet spot (Logseq `id::`).
|
||||
**Carried from v1:** files-canonical/index-derived; fine-grained addressing is adoptable;
|
||||
transclusion=clone=embed=reference is one primitive; structure/history federate iff in-text;
|
||||
attach mode is per-binding; the platform can enforce no-silent-mutation.
|
||||
406
research/260614-shard-spectrum-synthesis/findings.md
Normal file
406
research/260614-shard-spectrum-synthesis/findings.md
Normal file
@@ -0,0 +1,406 @@
|
||||
# Synthesis — the shard spectrum: one capability model across the full dive set
|
||||
|
||||
Date: 2026-06-14 · **revised 2026-06-14 (v3)** — extended from the fourteen-system set to
|
||||
include the **SHARD-WP-0003 engine batch** (Federated Wiki, Wikibase/Wikidata, the git-forge
|
||||
wikis Gitea/GitLab/GitHub, TiddlyWiki, ikiwiki, Salesforce Quip, MojoMojo, Oddmuse,
|
||||
UseModWiki). Per-shard spectra grown from thirteen to **fourteen** (added **provenance
|
||||
granularity**), with refinements to merge-model, attachment-mode, native-query, structure,
|
||||
history, content-opacity and write-granularity; and a **new coordination-layer axis — the
|
||||
federation-model taxonomy (§2.5)** — the headline v3 contribution.
|
||||
Source kind: **synthesis** — consolidates every deep dive into a single comparative model
|
||||
feeding the **shard adapter contract** and the **federation track** (`SHARD-WP-0002`).
|
||||
Lens: shard-wiki — what a backend must expose to participate, expressed as *spectra* of
|
||||
capability rather than a yes/no checklist.
|
||||
|
||||
> Purpose. ~23 tools/systems have now been studied. Two conceptual ancestors (**Xanadu,
|
||||
> ZigZag**); the file/DB engines (**XWiki, TWiki, Foswiki**, + landscape); the modern
|
||||
> note/PKB tools (**Roam, Obsidian, Notion, Joplin, Logseq, Anytype, AFFiNE, AppFlowy,
|
||||
> Trilium**); and the WP-0003 batch (**Federated Wiki, Wikibase, forge wikis, TiddlyWiki,
|
||||
> ikiwiki, Quip, MojoMojo, Oddmuse, UseModWiki**), against the federation and origin
|
||||
> research. This document reads them *across* each other. The payoff is a small set of
|
||||
> **capability spectra** — now **fourteen** per-shard, plus a **federation-model taxonomy**
|
||||
> for the coordination layer — each anchored at both ends by a real system, with federation
|
||||
> operations degrading by position. That spectrum *is* the adapter contract's design
|
||||
> surface. v3 folds the WP-0003 use cases (**UC-68–UC-82**) into the `SHARD-WP-0002` tasks.
|
||||
|
||||
Inputs: `research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`,
|
||||
`research/260613-{xwiki,twiki,foswiki}-deep-dive`,
|
||||
`research/260614-{xanadu,zigzag,roam,obsidian,notion,joplin,logseq,localfirst-workspaces,trilium,wikijs,federated-wiki,wikibase,forge-wikis,tiddlywiki,ikiwiki,quip,mojomojo,oddmuse,usemodwiki}-deep-dive`.
|
||||
Output target: `spec/TechnicalSpecificationDocument.md` (adapter contract) via
|
||||
`SHARD-WP-0002`.
|
||||
|
||||
---
|
||||
|
||||
## 1. The shard family matrix
|
||||
|
||||
Candidate-shard backends across the dimensions that matter to the contract. (Xanadu and
|
||||
ZigZag are *not* shards — they are the conceptual ideals each column aspires to; Federated
|
||||
Wiki is *barely* a shard — it is mostly a **federation model**, see §2.5; both listed apart.)
|
||||
|
||||
| Backend | Substrate | Attach mode(s) | Addressing | Structure | History | Merge | Query | →Markdown | Opacity |
|
||||
|---------|-----------|----------------|-----------|-----------|---------|-------|-------|-----------|---------|
|
||||
| **Git folder / forge wiki** | files (`.wiki.git`) | **file-store (git IS store)** | path (+ commit) | flat MD + frontmatter | **git-native** | git/text | no | native | none |
|
||||
| **ikiwiki** | files (git) → static HTML | file-store (source repo) | path | flat MD + directives | **git-native** | git/text | no | native (src) | none |
|
||||
| **Obsidian** | files | file-store **/** plugin | path + in-file `^id` | frontmatter (in-file) | none (Git plugin) | git/text | plugin (Dataview) | native (OFM) | none |
|
||||
| **Logseq** | files (MD/Org) → SQLite | file-store **/** plugin | **in-file block `id::`** (sweet spot) | `key::` props (in-file) | none (git) | git/text | **Datalog (derived)** | native-ish | none |
|
||||
| **TiddlyWiki** | single HTML **/** `.tid` dir | **file-store (container/`.tid`)** | tiddler `title` | typed tiddler **fields** | none / git (.tid) | git/text | **filters** | varies (`type`) | none |
|
||||
| **TWiki / Foswiki** | files + RCS / pluggable | file-store **/** API | path | `%META%` in-file (N records) | **open file (RCS)** | git/text | no | lossless (TML) | none |
|
||||
| **Oddmuse / UseModWiki** | flat files (CGI) | file-store (**minimal floor**) | CamelCase/path | none (flat) | flat-file, **partial** | last-writer | no | lossy | none |
|
||||
| **MojoMojo** | relational DB | **direct-DB-read** | page row / path | relational rows+lineage | **DB version rows** | last-writer | SQL | MD-in-column | none |
|
||||
| **Trilium** | SQLite (one file) | ETAPI / scripting / DB | `noteId` + **`branchId`** (id≠place) | labels+relations, **inherited+templated**; **DAG** | internal revisions | conflict-res | attr search | **lossy (HTML)** | **per-note** |
|
||||
| **Joplin** | SQLite-local | **sync-mirror** / local-REST / plugin | page-level `:/id` | notebooks + tags | internal revisions | conflict-notes | search | lossy | E2EE (whole) |
|
||||
| **Wiki.js** | DB ↔ git mirror | file-store(mirror) / **GraphQL** | path | frontmatter (mirror) | git (via mirror) | git/text | GraphQL | native (mirror) | none |
|
||||
| **XWiki** | DB (Hibernate) | in-engine host / REST | path | **XObjects/XClass** | internal (`xwikircs`) | git/text | yes (XWQL) | engine syntax | none |
|
||||
| **Quip** | hosted (SaaS) | **external-API (HTML)** | doc id + section anchor | **prose + inline spreadsheets/live-apps** | internal | server (OT) | no | **lossy (HTML)** | proprietary |
|
||||
| **Roam** | client DataScript | **in-app host only** | **store UUID** | `key::` attrs | internal (txn log) | (in-app) | yes (Datalog) | (Roam MD) | none |
|
||||
| **Notion** | hosted Postgres | **external-API (block-JSON)** | store UUID | **DB schema + relations + rollups** | internal, not portable | internal | yes (DB query) | **lossy** | none |
|
||||
| **Wikibase / Wikidata** | MediaWiki + RDF index | external-API + **SPARQL** | **opaque Q/P id + stmt GUID** | **typed entity-statement graph** | page revisions (JSON) | last-writer + **rank** | **SPARQL (+SERVICE)** | **lossy (not MD)** | none |
|
||||
| **Anytype** | **CRDT (any-sync)** | **replica / P2P node** | object id (store) | **typed object graph (ontology)** | CRDT log | **native-CRDT** | graph | lossy | **E2EE (whole)** |
|
||||
| **AFFiNE** | **CRDT (Yjs)** | replica / self-host | block id (store) | blocks; **one-data-many-views** | CRDT log | **native-CRDT** | DB filters | lossy | optional |
|
||||
| **AppFlowy** | **CRDT (Yrs)** | replica / self-host | block id (store) | **Notion-style DB + views** | CRDT log | **native-CRDT** | DB query | lossy | self-host |
|
||||
| — *Federated Wiki (mostly a federation model, §2.5)* | per-page JSON | **REST/file-store hybrid** | story-item `id` | typed story items | **append-only journal** | **fork + journal-replay** | neighborhood search | wiki-ish | none |
|
||||
| — *Xanadu (ideal)* | permascroll | — | **tumbler (span)** | spans + links | permanent | content-merge | — | — | — |
|
||||
| — *ZigZag (ideal)* | cells/dims | — | cell id | **N dimensions** | — | — | dimension walk | — | — |
|
||||
|
||||
Reading top to bottom is roughly shard-wiki's difficulty gradient, now with both ends
|
||||
extended: the **flat-file floor (Oddmuse/UseModWiki)** anchors the bottom — minimal, partial
|
||||
history, the field's common ancestor; **git/forge wikis, ikiwiki, Obsidian, Logseq,
|
||||
TiddlyWiki** are friction-free file-store cases (forge wikis make *git the store itself*);
|
||||
**TWiki/Foswiki/Trilium/Wiki.js** add translation, DAG/computed metadata, or a git mirror;
|
||||
**MojoMojo** needs direct-DB reads; **XWiki/Roam/Notion/Quip** add DB/SaaS structure and
|
||||
store/API addressing; **Wikibase** is the **typed-knowledge-graph far-end** (SPARQL,
|
||||
statement-level provenance); the **CRDT cohort** adds native merge + (Anytype) P2P/E2EE.
|
||||
Xanadu/ZigZag mark the ideals; Federated Wiki marks the federation-model ideal (§2.5).
|
||||
|
||||
---
|
||||
|
||||
## 2. The capability spectra (the contract's real shape) — fourteen
|
||||
|
||||
Each capability is **not boolean** — it is a position on a spectrum anchored at each end by
|
||||
a real system. The contract models *positions*; federation ops degrade by position.
|
||||
|
||||
1. **Addressing granularity** — `none → whole-page(path) → page-level store id(Joplin :/id,
|
||||
Trilium noteId) → in-file span(Obsidian ^id) → in-file block(Logseq id::, the sweet spot:
|
||||
block-level AND git-diffable) → store-minted span(Roam/Notion/CRDT UUID) → statement GUID
|
||||
(Wikibase) → portable tumbler(Xanadu ideal)`. Story-item `id` (Federated Wiki) is a
|
||||
mid-tier within-page handle. (UC-51, UC-44/45, UC-73.)
|
||||
2. **Content identity** — `none → path/title → opaque stable id, labels-as-annotation
|
||||
(Wikibase Q/P) → fingerprint(hash) → span-set/equivalence(Xanadu)`. Wikibase is the
|
||||
cleanest real instance of **stable, language-neutral identity**. (UC-46, UC-27, UC-73.)
|
||||
3. **Identity vs placement** — `path = identity(most) → identity separated from placement
|
||||
(Trilium note/branch; a page in many locations = a DAG) → provenance-edge links across
|
||||
sites (Federated Wiki fork-DAG; forge wiki = path, identity layered above)`. The clean
|
||||
model for a page under multiple paths/shards. (UC-66, UC-22, UC-71.)
|
||||
4. **Structure** — `flat Markdown(Oddmuse) → in-file frontmatter/key::(Obsidian/Logseq) →
|
||||
in-file %META%(TWiki) → tiddler fields(TiddlyWiki) → relational rows(MojoMojo) → typed
|
||||
objects(XWiki) → DB schema+relations+rollups(Notion/AppFlowy) → prose+inline-embedded
|
||||
objects(Quip) → typed object graph/ontology(Anytype) → computed inherited+templated
|
||||
(Trilium) → typed entity-statement knowledge graph(Wikibase)`. In-text federates;
|
||||
DB-locked needs sidecar+fidelity; **graph/computed** needs effective-vs-own +
|
||||
render-without-flatten. (UC-34, UC-39, UC-58, UC-67, UC-73, UC-80.)
|
||||
5. **History** — `none → partial/truncated flat-file(Oddmuse keep/) → internal-only/
|
||||
not-portable(Notion/Joplin/Trilium) → DB version rows(MojoMojo) → CRDT update log
|
||||
(Anytype/AFFiNE/AppFlowy) → append-only semantic journal(Federated Wiki) → open file
|
||||
format(TWiki RCS) → git-native(Git/forge/ikiwiki/Obsidian/Logseq/Wiki.js mirror)`.
|
||||
Internal/CRDT/DB-rows ⇒ *supplement*; open-file/journal ⇒ *import*; git ⇒ *adopt*.
|
||||
**Completeness is metadata** (Oddmuse history is partial — never imply complete). (UC-36,
|
||||
UC-41, UC-71, UC-81, UC-82.)
|
||||
6. **Merge model** — `none → last-writer → git/text 3-way merge → conflict-notes/keep-both
|
||||
(Joplin) → fork + manual journal-replay(Federated Wiki) → coexist-with-rank(Wikibase:
|
||||
contradictory values kept, curated) → native-CRDT conflict-free(Anytype/AFFiNE/AppFlowy)`.
|
||||
Four+ distinct models — never impose git/text merge on a CRDT or a journal shard; speak
|
||||
the shard's model or stay projection/overlay. (UC-64, UC-71, UC-75.)
|
||||
7. **Native query** — `none → text search → filter expressions(TiddlyWiki) → build-your-own
|
||||
derived index(Logseq DataScript over files; shard-wiki can do likewise) → datalog/graph
|
||||
(Roam/Anytype) → DB query(Notion/AppFlowy/XWiki) → SPARQL/RDF + federated SERVICE
|
||||
(Wikibase, the far-end + query-time cross-shard join)`. Delegate where present; **build an
|
||||
index over the projection** where not. (UC-52, UC-63, UC-05, UC-54, UC-74.)
|
||||
8. **Translation to Markdown** — `native → lossless round-trip(Foswiki TML↔HTML) → lossy-
|
||||
with-fidelity-report(HTML/CKEditor Trilium; Notion blocks; Quip HTML+embedded objects;
|
||||
CRDT/object models) → not-Markdown-at-all(Wikibase statements → lossy render or keep
|
||||
graph)`. Lossless ⇒ writable; lossy ⇒ read/projection floor + visible fidelity loss;
|
||||
not-MD ⇒ structured payload + optional rendered view. (UC-42, UC-59, UC-03, UC-73, UC-80.)
|
||||
9. **Attachment mode** — a **per-binding, capability-gated** choice; a backend may offer
|
||||
several. The full taxonomy:
|
||||
- **file-store** — *native on-disk store* (Obsidian/Logseq/TWiki), *git IS the store*
|
||||
(forge `.wiki.git`, ikiwiki source), *single-file container* (TiddlyWiki HTML),
|
||||
*flat-file floor* (Oddmuse), **or** *interchange/sync mirror* (Joplin; Wiki.js git
|
||||
mirror) (UC-40, UC-53, UC-60, UC-62, UC-76, UC-78, UC-79, UC-82, UC-68)
|
||||
- **in-engine host** — adapter inside the app via its API (Roam/Obsidian/Logseq/Trilium
|
||||
scripting, XWiki components) (UC-38, UC-50)
|
||||
- **local-REST** — localhost API, app-running (Joplin Data API; Trilium ETAPI) (UC-38)
|
||||
- **external-API** — remote API from outside, with a **payload-format facet**:
|
||||
*block-JSON* (Notion, UC-57), *GraphQL* (Wiki.js, UC-69), *HTML* (Quip, UC-80),
|
||||
*forge wiki REST* (GitLab/Gitea, UC-77), *MediaWiki/SPARQL* (Wikibase, UC-73/74)
|
||||
- **direct-DB-read** — read the engine's relational store (MojoMojo) when no file/API
|
||||
exists; schema = a versioned coupling (UC-81)
|
||||
- **CRDT replica** — hold a local CRDT replica (Anytype/AFFiNE/AppFlowy) (UC-64)
|
||||
- **P2P / no-central-endpoint** — replica or peer/node, not a URL (Anytype) (UC-65)
|
||||
- **REST/file-store hybrid** — page JSON over HTTP+CORS or static files (Federated Wiki)
|
||||
(UC-70)
|
||||
10. **Operational envelope** — `local/unbounded → realtime CRDT/WebSocket → rate-limited+
|
||||
eventually-consistent+paginated(Notion ~3 rps, Quip, Wikibase public endpoints)`. Sets
|
||||
live vs cache/poll/webhook. (UC-57, UC-31.)
|
||||
11. **Access grant** — `open(L0; Oddmuse) → token → OAuth scoped+revocable(Notion) →
|
||||
enterprise SSO + ACL(Quip/Salesforce, Wiki.js path rules) → P2P key/invite(Anytype) →
|
||||
own-site-only writes(Federated Wiki)`. The backend may *enforce* no-silent-mutation.
|
||||
(UC-57, UC-06, UC-65, [[shard-wiki-auth-in-core-decision]].)
|
||||
12. **Content opacity** — `plaintext(files) → proprietary-but-lossy-exportable(Quip HTML;
|
||||
Notion) → encrypted-at-rest whole-shard(Joplin/Anytype E2EE) → per-item(Trilium
|
||||
protected notes)`. Opaque ⇒ backup/structure-shell; never present ciphertext (or imply
|
||||
a lossy export is faithful) as readable. (UC-61, UC-80.)
|
||||
13. **Write granularity** — `whole-file(TiddlyWiki single-file) → per-page/note(Git/forge/
|
||||
Obsidian/Joplin/Trilium/Oddmuse) → story-item/paragraph(Federated Wiki) → section-anchor
|
||||
splice(Quip) → per-statement(Wikibase API) → per-block(Roam/Notion/Logseq/CRDT)`. Sets
|
||||
overlay/patch/lock/conflict scope; whole-file ⇒ **no per-page atomicity**. (UC-35,
|
||||
UC-78.)
|
||||
14. **Provenance granularity** *(new, 14th)* — `none → per-shard → per-page(most; author/
|
||||
time) → per-commit(git) → per-edit(journal entry, Federated Wiki) → per-statement/value
|
||||
(Wikibase references + rank)`. How finely the union can attribute and source content;
|
||||
Wikibase pushes provenance below the page (sourced, contradictory values coexist with a
|
||||
curation signal). The page model + journal should *allow* sub-page provenance even if MVP
|
||||
records per page. (UC-24, UC-71, UC-75.)
|
||||
|
||||
*(Content types — Markdown-only → typed records → inline-embedded objects (Quip
|
||||
spreadsheets) → non-Markdown assets (Excalidraw/Canvas/whiteboards) → typed-graph statements
|
||||
(Wikibase) — remains a cross-cutting page-model demand, tracked under structure + T12 rather
|
||||
than as a standalone capability. UC-55.)*
|
||||
|
||||
Design consequence: **T11's capability vocabulary = these fourteen spectra**, not a flat
|
||||
`read/write/diff/...` list. The flat verbs remain the *operations*; the spectra are the
|
||||
*profile* saying how well each verb is supported and how it degrades. The **floor**
|
||||
(Oddmuse/UseModWiki) and **far-ends** (Wikibase graph/query/provenance; CRDT merge; Notion
|
||||
hosted) bound every spectrum with a real system.
|
||||
|
||||
---
|
||||
|
||||
## 2.5. The federation-model taxonomy (the coordination-layer axis) — new in v3
|
||||
|
||||
The fourteen spectra above describe a *single shard's* capabilities. The WP-0003 batch
|
||||
revealed a second, orthogonal axis the v2 synthesis under-modelled: **federation itself is
|
||||
plural.** "Attach many shards and present a union" can be realized by several distinct
|
||||
coordination models, each a real system, each with different reconciliation semantics. This
|
||||
axis lives at shard-wiki's **coordination layer** (`SHARD-WP-0002` T1–T6), not in a single
|
||||
adapter.
|
||||
|
||||
| Federation model | Exemplar | Mechanism | Reconciliation | Discovery |
|
||||
|------------------|----------|-----------|----------------|-----------|
|
||||
| **Fork + journal** | Federated Wiki | copy page to own site; append-only semantic **journal** records `fork`-with-source | **manual**: compare journals, fork the version you prefer; **chorus**, no canonical | link + fork (**neighborhood**) / curated **roster** |
|
||||
| **VCS replication + ping** | ikiwiki | git clone/pull/push between instances; **XML-RPC pinger** notifies peers to pull/rebuild | **git merge** across clones | configured peers + pings |
|
||||
| **Query-time graph join** | Wikibase | **SPARQL `SERVICE`** runs a sub-query on another endpoint and joins | none (read-time join; rank curates conflicts) | endpoint URLs |
|
||||
| **Feed aggregation** | ikiwiki `aggregate`, RSS/Atom | pull remote feeds in as pages | one-way inbound projection | feed URLs |
|
||||
| **Activity streams** | ActivityPub (federation research) | actor/inbox/outbox Create/Update | per-actor; eventual | actor handles |
|
||||
| **Engine-maintained mirror** | Wiki.js git mirror | DB-canonical engine syncs to a git mirror | engine owns DB↔git sync (don't double-sync) | the mirror repo |
|
||||
|
||||
Two cross-cutting lessons:
|
||||
|
||||
- **git-IS-store vs engine-mirror resolves the write-race.** A **forge wiki** (`.wiki.git`)
|
||||
and **ikiwiki source** make *git the canonical store*, so **write-by-commit is safe** — no
|
||||
engine to race. Wiki.js (engine-mirror, DB canonical) is the opposite and needs care. The
|
||||
contract must record *which side is source of truth* per binding. (Resolves UC-68's open
|
||||
race for the git-canonical case; UC-76.)
|
||||
- **shard-wiki's own coordination journal should be journal-shaped.** Federated Wiki proves
|
||||
the **append-only semantic-op log with provenance entries, page-state-as-derived-replay**
|
||||
pattern in production — the concrete shape for INTENT's coordination journal (UC-71), and a
|
||||
superset that can *ingest* git history, CRDT logs, DB version rows, and partial flat-file
|
||||
histories as differently-grained inputs.
|
||||
|
||||
Design consequence: **T1–T6 should model federation as a selectable model (or composition of
|
||||
models), not a single hard-coded flow** — mechanism over policy at the coordination layer,
|
||||
mirroring how T11 models per-shard capability as spectra. A given information space may use
|
||||
fork+journal for human-curated shards, VCS-replication for git shards, query-join for graph
|
||||
shards, and feed-aggregation for read-only sources — concurrently.
|
||||
|
||||
---
|
||||
|
||||
## 3. Cross-cutting findings (the through-lines)
|
||||
|
||||
- **Files-canonical, index-derived is the winning architecture.** Obsidian's MetadataCache,
|
||||
**Logseq's DataScript-over-files**, Git's working tree, **Wikibase's WDQS (SPARQL index
|
||||
rebuilt from canonical JSON entities)**, and shard-wiki's projection model agree: the
|
||||
graph/backlinks/query index is **derived and rebuildable**, never a second source of truth.
|
||||
Roam/Notion/CRDT invert this (store canonical) and pay in portability. shard-wiki keeps
|
||||
files + journal canonical.
|
||||
- **Git is both the home store and the home journal.** *(sharpened in v3)* Forge wikis make
|
||||
git *the* store; ikiwiki and Wiki.js make git the source or mirror; for all of them the
|
||||
**git log is the coordination journal with zero synthesis**. The git-canonical cases are
|
||||
the friction-free core; everything else is measured as deviation.
|
||||
- **The flat-file floor is the field's common root.** *(new)* Oddmuse and UseModWiki
|
||||
(Wikipedia's MediaWiki Phase I) show the minimal plain-text page+history wiki is the
|
||||
ancestor every richer engine elaborates — so the **minimal/floor capability profile** is
|
||||
the right baseline, and shard-wiki's page model must stay attach-compatible with it
|
||||
(flat files, CamelCase identities, **partial** history). (UC-82.)
|
||||
- **Federation is plural** *(new — §2.5).* Fork+journal, VCS-replication+ping, query-time
|
||||
graph join, feed aggregation, activity streams, engine-mirror — distinct coordination
|
||||
models, selectable and composable, not one flow.
|
||||
- **Provenance has a granularity spectrum** *(new — spectrum 14).* From per-shard down to
|
||||
**per-statement/value** (Wikibase references + rank). Union-without-erasure includes
|
||||
*attribution at the right grain* and letting sourced contradictions coexist with curation.
|
||||
- **Identity ≠ placement.** Trilium's **note vs branch** (a note cloned into many locations =
|
||||
a DAG) and Wikibase's **opaque stable IDs (labels-as-annotation)** are the clean models:
|
||||
separate *what a page is* from *where it sits* and from *what it's called*. Federated
|
||||
Wiki's fork entries are **provenance edges** between same-named pages across sites.
|
||||
- **Transclusion ⇄ clone ⇄ embed ⇄ cloned-note ⇄ reference is one primitive** over an
|
||||
addressable union (Xanadu, ZigZag, Roam/Logseq, Obsidian `![[ ]]`, Notion synced block,
|
||||
Trilium cloning, TiddlyWiki `{{ }}`). (UC-32/44/45/51/66.)
|
||||
- **CRDT changes the merge math; journals and rank add more models.** *(extended)* The merge
|
||||
spectrum now spans last-writer → git 3-way → conflict-notes → **fork+journal-replay** →
|
||||
**coexist-with-rank** → native-CRDT. Never impose git/text merge across that range.
|
||||
- **Structure & history federate iff in text; metadata can be computed; the far-end is a
|
||||
graph.** `%META%`/frontmatter/`key::`/tiddler-fields diff and travel; XObjects/Notion-DB/
|
||||
CRDT/Quip-objects lock in; Trilium computes metadata (inherited+templated); **Wikibase is
|
||||
a full typed knowledge graph** — render to Markdown is lossy, so keep the graph and offer a
|
||||
view, never silent-flatten.
|
||||
- **The attach surface is rarely "just files," and source-of-truth varies.** *(extended)*
|
||||
Joplin's best surface is its sync mirror; Logseq offers file-graph or DB-graph and is
|
||||
migrating substrate; **forge wikis = git-canonical (write freely)** vs **Wiki.js =
|
||||
DB-canonical mirror (write carefully)** vs **MojoMojo = DB-only (direct read)**. Bind to
|
||||
capabilities *and* record the source of truth.
|
||||
- **The page model must stretch many ways at once:** prose Markdown, typed/computed records
|
||||
(N-per-page, relations, inheritance), **inline-embedded objects** (Quip), **typed-graph
|
||||
statements** (Wikibase), non-Markdown assets, reference/query-defined pages, and
|
||||
**multi-placement (DAG) identity**. The heaviest demand on T12.
|
||||
|
||||
---
|
||||
|
||||
## 4. How the use cases fold into the workplan
|
||||
|
||||
The 260614 + WP-0003 use cases (UC-44–UC-82) map onto the adapter-contract and federation
|
||||
tasks:
|
||||
|
||||
| UCs | Theme | Lands in |
|
||||
|-----|-------|----------|
|
||||
| UC-35, UC-50, UC-53, UC-57, UC-60, UC-62, UC-64, UC-65, **UC-68, UC-70, UC-76, UC-77, UC-78, UC-79, UC-80, UC-81, UC-82** | attachment modes (file-store native/git-IS-store/container/mirror, in-engine, local-REST, external-API w/ payload-format, direct-DB, CRDT-replica, P2P, REST/file-store-hybrid) + operational envelope | **T11** + **T14** |
|
||||
| UC-34, UC-39, UC-55, UC-58, UC-67, **UC-73, UC-80** | structured/typed/computed/**graph** payload, **inline-embedded objects**, non-MD assets | **T12** |
|
||||
| UC-36, UC-41, **UC-81, UC-82** | history: internal/CRDT-log/**DB-rows**/**partial-flat-file** = supplement; **completeness metadata** | **T13** |
|
||||
| UC-42, UC-59, **UC-73, UC-80** | translation: lossless vs lossy-with-fidelity vs **not-Markdown** (graph/HTML/objects) | **T15** |
|
||||
| UC-31, **UC-79** | webhooks / realtime / push-vs-poll / **VCS ping** | **T6** / **T11** envelope |
|
||||
| UC-57 §6, UC-61, UC-65, **UC-06, UC-80** | scoped grant; **enterprise ACL/SSO**; content opacity (whole/per-item/**lossy-exportable**); P2P key | **T11** (access-grant + opacity) |
|
||||
| UC-44, UC-45, UC-46, UC-51, UC-63, UC-66, **UC-73, UC-74** | span/statement addressing, **opaque stable identity**, identity≠placement, transclusion-as-reference, derived index, **graph query** | **T16** (+ T12) |
|
||||
| UC-47, UC-48, UC-52, UC-54, **UC-74** | dimensional navigation, query delegation/build, query-defined pages, **SPARQL/federated SERVICE** | **T16** (+ T5/T10) |
|
||||
| UC-24, **UC-71, UC-75** | **provenance granularity** (per-edit / per-statement); coordination journal shape | **T13/T16** + journal |
|
||||
| UC-26, UC-27, UC-28, UC-30, UC-05, **UC-71, UC-72, UC-79** | **federation-model taxonomy** (fork+journal, VCS-replication+ping, query-join, chorus/neighborhood/roster) | **T1–T6** |
|
||||
|
||||
The one **structural** v3 change: the **federation-model taxonomy (§2.5)** is a new design
|
||||
surface for **T1–T6** — federation becomes a *selectable, composable model* rather than a
|
||||
single flow. Per-shard, **T11** grows to **fourteen spectra** (adds provenance granularity)
|
||||
and **T14** absorbs the expanded attachment-mode taxonomy (git-IS-store, container,
|
||||
direct-DB, REST/file-store-hybrid, external-API payload-format facet).
|
||||
|
||||
---
|
||||
|
||||
## 5. Recommendations (decisions to record under SHARD-WP-0002)
|
||||
|
||||
1. **Model per-shard capabilities as the fourteen spectra (§2)**, not flat verbs. (T11.)
|
||||
2. **Model federation as a selectable/composable taxonomy of models (§2.5)** — fork+journal,
|
||||
VCS-replication+ping, query-time graph join, feed aggregation, activity streams,
|
||||
engine-mirror — at the coordination layer. (T1–T6.)
|
||||
3. **Make the coordination journal journal-shaped** (Federated Wiki): append-only semantic
|
||||
ops + provenance entries, page state = derived replay; able to *ingest* git history, CRDT
|
||||
logs, DB version rows, and partial flat-file histories as differently-grained inputs.
|
||||
(T13 + journal, UC-71.)
|
||||
4. **Add provenance granularity as a spectrum** (per-shard → per-page → per-edit →
|
||||
per-statement/value); allow sub-page provenance and **coexist-with-rank** for sourced
|
||||
contradictions. (T11/T13, UC-24/75.)
|
||||
5. **Record source-of-truth per binding** and resolve the write-race accordingly:
|
||||
git-canonical (forge wiki / ikiwiki) ⇒ write-by-commit; engine-mirror (Wiki.js) ⇒ careful;
|
||||
direct-DB (MojoMojo) ⇒ read/projection/overlay default. (T14, UC-68/76/79/81.)
|
||||
6. **Expand attachment mode (§2 #9 / T14):** file-store (native / **git-IS-store** /
|
||||
**container** / mirror / **flat-file floor**), in-engine, local-REST, **external-API with
|
||||
a payload-format facet** (block-JSON/GraphQL/HTML/forge-REST/SPARQL), **direct-DB-read**,
|
||||
CRDT-replica, P2P, **REST/file-store-hybrid**. Bind to capabilities; substrate can
|
||||
migrate. (T14, UC-40/68/70/76/77/78/79/80/81/82 + UC-43.)
|
||||
7. **Page model: support a typed-graph payload and inline-embedded objects, not only
|
||||
typed records** — render to Markdown lossily, keep the graph/objects canonical; preserve
|
||||
computed (inherited/templated) metadata as effective-vs-own. (T12/T15, UC-73/80/67.)
|
||||
8. **Capability profile must express absence and partiality cleanly** (Oddmuse floor):
|
||||
sparse profiles, **partial/truncated history reported honestly**, graceful degradation to
|
||||
read/projection/overlay/backup. (T11/T13, UC-82.)
|
||||
9. **Query: SPARQL/RDF + federated `SERVICE` is the graph far-end** — delegate to native
|
||||
engines (filters/datalog/DB-query/SPARQL) where present; build a derived index over the
|
||||
projection where not. (T16/T5, UC-52/63/74.)
|
||||
10. **Adopt opaque stable identity with labels-as-annotation** (Wikibase) and **separate
|
||||
identity from placement** (Trilium note/branch); fork entries are provenance edges.
|
||||
(T16, UC-73/66/71.)
|
||||
11. **Keep files + coordination journal canonical; all indexes/projections derived and
|
||||
rebuildable.** (Architecture invariant, INTENT.)
|
||||
|
||||
These honor INTENT: mechanism over policy (capability *and now federation* modelled as
|
||||
spectra/taxonomy, not hard-coded behaviors), union without erasure (fidelity +
|
||||
effective-vs-own + identity/placement + **provenance granularity** all preserve
|
||||
information), graceful degradation (every spectrum has a read/projection floor; the
|
||||
Oddmuse floor is explicit; opaque/CRDT/graph shards degrade to backup/projection/lossy-view),
|
||||
no silent mutation (access-grant + opacity + overlay + respect native merge + **source-of-
|
||||
truth per binding**), shard sovereignty (no backend forced to change substrate),
|
||||
Markdown-first/backend-neutral (in-text preferred; DB/CRDT/HTML/**graph** tolerated with a
|
||||
view).
|
||||
|
||||
---
|
||||
|
||||
## 6. Open questions escalated by the synthesis
|
||||
|
||||
1. **Federation composition** — can one information space run several federation models
|
||||
(§2.5) concurrently over different shards, and how does the union reconcile a chorus
|
||||
(fork+journal) with a canonical-asserting shard (Notion / upstream main)? (T1–T6,
|
||||
UC-72.)
|
||||
2. **Coordination-journal op vocabulary** — adopt Federated Wiki's exact ops
|
||||
(create/add/edit/move/remove/fork) at item grain, or an abstract op set other shards can
|
||||
emit, with git/CRDT/DB-row/flat-file histories ingested as inputs? (T13, UC-71.)
|
||||
3. **Provenance granularity in the model** — does the journal/page model carry per-statement
|
||||
provenance + rank (Wikibase), per-edit (journal), or per-page (MVP), configurable? (T13,
|
||||
UC-75.)
|
||||
4. **Typed-graph page** — model a Wikibase entity natively (statements) or always project to
|
||||
a lossy Markdown/table view, or both (canonical graph + view)? Is SPARQL a union-level
|
||||
capability or pass-through? (T12/T16, UC-73/74.)
|
||||
5. **Source-of-truth + write-race** — formalize git-canonical vs engine-mirror vs direct-DB
|
||||
per binding; sanction direct third-party DB reads (schema drift) and write-by-commit
|
||||
timing. (T14, UC-68/76/79/81.)
|
||||
6. **Portable span address** across heterogeneous backends — wrap native IDs (Roam/Notion/
|
||||
CRDT UUID, Logseq `id::`, Trilium noteId, **Wikibase Q/P + stmt GUID**, fedwiki story-item
|
||||
id) in a shard-scoped address? (T16.)
|
||||
7. **CRDT shards** — embed a CRDT client (Yjs/Yrs) for a live replica, or consume snapshots?
|
||||
Overlays as CRDT ops or out-of-band patches? (T14, UC-64.)
|
||||
8. **Page model breadth** — can one model carry prose + typed/computed records +
|
||||
inline-embedded objects + typed-graph statements + non-MD assets + query-defined pages +
|
||||
multi-placement identity coherently? (T12.)
|
||||
9. **Whole-file & partial-history shards** — overlays against a whole-file shard
|
||||
(TiddlyWiki single-file: buffer+re-serialize vs require `.tid`); representing
|
||||
partial/truncated history (Oddmuse) with completeness metadata. (T11/T13, UC-78/82.)
|
||||
10. **Content opacity & lossy-exportable** — what is visible for an opaque/proprietary shard
|
||||
(IDs/structure vs nothing vs lossy HTML); never present a lossy export as faithful; does
|
||||
shard-wiki ever hold keys? (T11, UC-61/80.)
|
||||
|
||||
---
|
||||
|
||||
## 7. Sources
|
||||
|
||||
A synthesis; primary sources are the ~23 dives' `findings.md` files plus
|
||||
`research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`.
|
||||
No new external research was performed in v3 (it consolidates the WP-0003 batch dives
|
||||
260614-{federated-wiki,wikibase,forge-wikis,tiddlywiki,ikiwiki,quip,mojomojo,oddmuse,
|
||||
usemodwiki}).
|
||||
|
||||
Cross-references: `spec/UseCaseCatalog.md` (UC-26–UC-82),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T1–T16),
|
||||
`workplans/SHARD-WP-0003-engine-dives-batch.md` (the engine batch, done),
|
||||
`INTENT.md` (constraints), [[shard-wiki-auth-in-core-decision]].
|
||||
|
||||
---
|
||||
|
||||
## 8. Traceability
|
||||
|
||||
- Consolidates: all ~23 deep dives + federation/origin research into one capability model
|
||||
(v3 adds the WP-0003 batch).
|
||||
- Feeds: `SHARD-WP-0002` **T1–T6** (federation-model taxonomy §2.5 — selectable/composable
|
||||
federation), **T11** (**fourteen-spectra** vocabulary, incl. provenance granularity +
|
||||
expanded attachment modes + payload-format facet), T12 (page-model breadth: typed-graph,
|
||||
inline-embedded objects, computed metadata, identity≠placement, non-MD), T13 (history incl.
|
||||
DB-rows + partial-flat-file = supplement, completeness metadata, journal-shaped
|
||||
coordination journal), T14 (full attachment taxonomy incl. git-IS-store / container /
|
||||
direct-DB / REST-file-hybrid + source-of-truth per binding), T15 (lossy + not-Markdown
|
||||
graph/HTML), T16 (addressing incl. statement GUID + opaque stable identity, graph query /
|
||||
federated SERVICE).
|
||||
- UC coverage extended in the workplan from UC-34–UC-67 to **UC-34–UC-82**.
|
||||
- No UCs added (synthesis only); no boundary changes (INTENT Stability Note untouched —
|
||||
the federation-model taxonomy is a refinement of *how* the coordination layer works, not a
|
||||
redefinition of shard / Git's role / orchestrator-vs-engine).
|
||||
33
research/260614-squeak-pharo-deep-dive/README.md
Normal file
33
research/260614-squeak-pharo-deep-dive/README.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# 260614 — Squeak & Pharo (image-based Smalltalk) deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T6 (Squeak) + T8 (Pharo)** — combined memo
|
||||
(justified merge: both are image-based Smalltalks; Pharo is T8's thin "context for T6/T7").
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into the **image-based live-object** environment — Squeak and Pharo (the
|
||||
substrate Glamorous Toolkit T7 runs on): the **image** as a persistent world of live objects
|
||||
with **no file/document/app boundary**, the live **inspector**, and Pharo's retreat to
|
||||
**code-as-text in git** (Tonel/Iceberg).
|
||||
|
||||
## Why it matters
|
||||
|
||||
- The **purest "live" end** of the batch's spectrum (literate source → notebook snapshot →
|
||||
GT/Lepiter live-over-files → **image: everything live**). Names the **live↔snapshot** axis
|
||||
the projection model (T16) must carry.
|
||||
- Hardens the **image-is-not-a-store** boundary (opaque monolithic non-diffable blob; no
|
||||
page identity/history/provenance) — generalizes "attach files, not the kernel/image"
|
||||
(UC-84, T7) into a named binding boundary (T14).
|
||||
- Pharo **confirms** the resolution: even image traditions externalize to **git-versionable
|
||||
text** (Tonel) to version/collaborate — files-canonical from the Smalltalk side.
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (boundary / design prior art; covers T6 and T8). Boundary for UC-34/35/79;
|
||||
links UC-83/84 (live→snapshot), UC-54/47/48 (live-object inspection), UC-76/79 (Tonel/git).
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | The image & live objects, the image-as-store anti-pattern, Pharo Tonel/Iceberg→git, INTENT mapping, UC disposition (enrichment-only), architecture notes (T14 boundary, T16 live↔snapshot axis) |
|
||||
141
research/260614-squeak-pharo-deep-dive/findings.md
Normal file
141
research/260614-squeak-pharo-deep-dive/findings.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# Squeak & Pharo (image-based Smalltalk) — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 **T6 (Squeak)** + **T8 (Pharo)** — combined
|
||||
(justified merge: both are image-based Smalltalks; Pharo is T8's "context for T6/T7," kept
|
||||
brief per the workplan). · **Subject:** the **image-based live-object** environment and what
|
||||
it teaches (and warns) about shard-wiki's page model.
|
||||
|
||||
## Why this dive (and why merged)
|
||||
|
||||
Squeak (1996, the Alan Kay/Ingalls lineage) and Pharo (2008 fork, the substrate **Glamorous
|
||||
Toolkit** T7 runs on) are the same idea: **a persistent world of live objects, the "image,"
|
||||
with no file/document/application boundary.** They are **not candidate shards** — they are
|
||||
the **anti-pattern boundary** for shard-wiki's files-canonical stance *and* the inspiration
|
||||
behind moldable inspection (T7). T8 (Pharo) is folded here as the workplan allows: its
|
||||
distinct contribution over Squeak (Tonel/Iceberg file-based code → git) is the one piece that
|
||||
*does* touch our concerns, covered in §3.
|
||||
|
||||
## 1. The image: knowledge-as-live-objects
|
||||
|
||||
- A Smalltalk **image** is a serialized snapshot of the **entire object memory** — every
|
||||
object, class, tool, window, and the running program state — persisted as one binary file
|
||||
(`.image` + `.changes` log). You resume exactly where you left off.
|
||||
- **"Everything is a live object"**: code, data, UI (Morphic), the debugger, the inspector —
|
||||
all are objects you can open, message, and modify *in place*, while running. There is **no
|
||||
edit/compile/run cycle** and **no document-vs-app distinction**.
|
||||
- The **inspector** lets you open any object and explore/modify its state live — the direct
|
||||
ancestor of GT's moldable inspector (T7), but generic rather than domain-molded.
|
||||
|
||||
This is the **purest "live" end** of the spectrum the whole batch traverses: literate source
|
||||
(static) → notebook captured output (snapshot) → GT/Lepiter (live results over files) →
|
||||
**image (everything live, nothing inherently a file).**
|
||||
|
||||
## 2. The boundary: image-as-store is the anti-pattern
|
||||
|
||||
The image directly **contradicts** shard-wiki's design constraints, which is exactly why it's
|
||||
worth recording as a hard boundary:
|
||||
|
||||
- **Opaque, monolithic, non-diffable.** The image is one big binary blob of entangled live
|
||||
state — no per-page identity, no text diff, no mergeable history, no provenance per unit.
|
||||
It violates *union-without-erasure granularity*, *Markdown-first*, and *git-addressable
|
||||
coordination*.
|
||||
- **No stable addressable "page."** Knowledge is an object graph in memory, not addressable
|
||||
documents — there is nothing to attach at page granularity without an export step.
|
||||
- **History is a `.changes` log**, a serial source-change stream, not a content history.
|
||||
|
||||
**Conclusion (boundary recorded):** an image is **not a shard and not a store**. This is the
|
||||
generalized form of the rule already hit at Jupyter (UC-84) and GT (T7): *attach the
|
||||
exported files, never the live image/kernel.* The image can only participate via an **export
|
||||
projection** (objects/code → files), which is a **derivation-projection** (T1) that
|
||||
**degrades the liveness to a static snapshot**.
|
||||
|
||||
## 3. Pharo's twist: code-as-files (Tonel) → git (the one actionable bit)
|
||||
|
||||
Pharo (T8) matters precisely where it **retreats from pure-image**:
|
||||
|
||||
- **Tonel / FileTree** serialize each class/method as **plain-text files** in a directory,
|
||||
and **Iceberg** manages those files as a **git repository** — so Pharo code lives in git as
|
||||
text, diffable and mergeable, *outside* the image.
|
||||
- This is the **same move** as Lepiter (T7), nbstripout/Jupytext (T3), and ikiwiki source
|
||||
(UC-79): **the durable, attachable artifact is the file representation; the live
|
||||
environment is a layer above it.** It confirms our stance from the *Smalltalk* side: even
|
||||
the most image-centric tradition externalizes to **git-versionable text** to collaborate.
|
||||
|
||||
So Pharo adds **no new page-model idea** beyond "image-based environments still externalize
|
||||
to git text" — exactly the "keep brief / fold" expectation. Its value is **confirming the
|
||||
boundary resolution**: attach the **Tonel/git source**, treat the image as live-only.
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Inspiration (keep)
|
||||
|
||||
- **Live-object inspection** is the seed of moldable views (T7/UC-54): the *idea* that any
|
||||
unit can be opened and explained interactively. shard-wiki adopts this as **projection/
|
||||
view**, not as a storage model.
|
||||
- **Resume-where-you-left-off** liveness names the far end of the **live↔snapshot** axis the
|
||||
contract must place every computed/projected view on (UC-83/84): the more live the source,
|
||||
the more its attached form is a **degrading snapshot**.
|
||||
|
||||
### Boundary (enforce — design-bug if violated)
|
||||
|
||||
- **Image-as-store is a design-bug boundary.** Never model an image (or any monolithic live
|
||||
memory blob) as a shard/store; participate only via **export → files** (a degrading
|
||||
derivation-projection). Generalizes "attach files, not the kernel/image" (UC-84, T7).
|
||||
|
||||
### Confirmation (Pharo)
|
||||
|
||||
- Even pure-image traditions externalize to **git-versionable text** (Tonel/Iceberg) to
|
||||
version and collaborate — reinforcing **files-canonical + git coordination** as the
|
||||
durable substrate; the live environment sits above it.
|
||||
|
||||
## 5. UC disposition (enrichment-only — no new UC)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC / thread |
|
||||
|------------------------|---------------------|
|
||||
| Live-object inspector = generic ancestor of moldable views (§1) | links UC-54, UC-47/48 (T7) |
|
||||
| Image = opaque monolithic non-diffable blob; not a page/store (§2) | **boundary** for UC-34/UC-35/UC-79 (granularity, identity, files-canonical) |
|
||||
| Image participates only via export→files = degrading derivation-projection (§2) | links UC-83, UC-84 (live→snapshot) |
|
||||
| Pharo Tonel/Iceberg: code-as-text in git (§3) | links UC-79, UC-76 (git-canonical text) |
|
||||
| `.changes` = serial source-change log, not content history (§1) | links UC-36 (history shape) |
|
||||
|
||||
Both Squeak and Pharo are **design prior art / boundary**, not candidate shards → **no new
|
||||
UC**. They sharpen the **live↔snapshot** axis and harden the **image-is-not-a-store**
|
||||
boundary; Pharo confirms even image traditions externalize to git text.
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T14 (attach binding):** record the **image-is-not-a-store** boundary explicitly — a
|
||||
monolithic live-memory blob is never an attach target; participation is via **export→files**
|
||||
only. Generalize the "attach files, not the kernel/image" rule (UC-84, GT T7) to a named
|
||||
boundary in the binding taxonomy.
|
||||
- **T16 (projection):** add the **live↔snapshot** axis to the projection model — every
|
||||
computed/projected view sits somewhere between "live (re-derivable on demand)" and "static
|
||||
snapshot," and the more live the source, the more its attached form must be a clearly-
|
||||
marked degrading snapshot.
|
||||
- **T11/T12:** the live-object inspector is the *inspiration* for the moldable view registry
|
||||
(T7), not a storage shape; nothing new to the page model itself.
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. Is **live↔snapshot** an explicit, first-class metadata axis on every projection (so the
|
||||
union can label "this view was live / is a snapshot from time T"), or implicit per
|
||||
capability? (Recurs across UC-83/84, GT, Mathematica `Dynamic`, Strudel T5.)
|
||||
2. Do we ever ingest a Smalltalk project by attaching its **Tonel/git** repo as an ordinary
|
||||
git-text shard (no Smalltalk-specific adapter needed), confirming the boundary resolution?
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- Squeak: `squeak.org`, the *Back to the Future* (Squeak) paper (Ingalls et al.), Morphic;
|
||||
image/`.changes` model.
|
||||
- Pharo: `pharo.org`, **Tonel** format, **Iceberg** (git integration), FileTree.
|
||||
- prior: `research/260614-glamorous-toolkit-deep-dive/` (moldable inspector, Lepiter, T7);
|
||||
`research/260614-jupyter-deep-dive/` (live→snapshot boundary, UC-84).
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
**No new UC** (boundary / design prior art; covers **T6 Squeak** and **T8 Pharo** in one
|
||||
justified-merge memo). Boundary hardened for: UC-34/UC-35/UC-79 (image-is-not-a-store);
|
||||
links UC-83/UC-84 (live→snapshot), UC-54/UC-47/UC-48 (live-object inspection → moldable
|
||||
views), UC-76/UC-79 (Pharo Tonel/git text), UC-36 (`.changes` history shape). Architecture
|
||||
cross-refs: SHARD-WP-0002 T14 (image-is-not-a-store boundary; export→files only), T16
|
||||
(live↔snapshot projection axis).
|
||||
30
research/260614-strudel-deep-dive/README.md
Normal file
30
research/260614-strudel-deep-dive/README.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# 260614 — Strudel.cc (live-coding REPL) deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T5**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Strudel.cc** (the JavaScript port of **TidalCycles**): a browser
|
||||
**live-coding REPL** where terse **pattern source** is **evaluated live into time-based
|
||||
audio** — code as a running musical performance, with **no document, no output cell, no file
|
||||
of results**.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- The **extreme of the live↔snapshot axis** (named at T6): output is **temporal, generative,
|
||||
performative**, so there is **no faithful static form** — the best static projection is
|
||||
**source (canonical, diffable) + an optional audio recording snapshot**, marked as one
|
||||
performance. The **honesty test** for union-without-erasure + graceful degradation.
|
||||
- Bounds the projection model's live end: ahead-of-time → view-time one-shot → continuous →
|
||||
**irreducibly live/temporal (recording-only)**.
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (enrichment / design prior art; far live end). Enriches **UC-54, UC-55**;
|
||||
links UC-83/84, UC-37, UC-35.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Code-as-live-performance, the limit of static projection, INTENT mapping, UC disposition (enrichment-only), architecture notes (T16 far end of live↔snapshot axis) |
|
||||
118
research/260614-strudel-deep-dive/findings.md
Normal file
118
research/260614-strudel-deep-dive/findings.md
Normal file
@@ -0,0 +1,118 @@
|
||||
# Strudel.cc — live-coding REPL — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T5 · **Subject:** Strudel.cc (the
|
||||
JavaScript port of TidalCycles) — a browser **live-coding REPL** where code is a **running
|
||||
musical performance**.
|
||||
|
||||
## Why this dive
|
||||
|
||||
The closing, lowest-priority dive — and the **extreme** of the live↔snapshot axis. Where
|
||||
Processing (T4) renders *visual* output at view time, Strudel renders **time-based audio**
|
||||
that only exists *while running and evolving*. It is the limit case for "can a page be a live
|
||||
computation?" — the answer where **static projection is least faithful**. Enrichment-only;
|
||||
feeds UC-54/55 and hardens the snapshot-degradation rule.
|
||||
|
||||
## 1. Code as a live, evaluated performance
|
||||
|
||||
- **Strudel** ports **TidalCycles**' pattern language to JavaScript, running entirely in the
|
||||
**browser** (Web Audio). You write **pattern expressions** (e.g. `note("c e g")`,
|
||||
`sound("bd sd")` with transformations) and **evaluate them live**; the sound updates
|
||||
**without stopping** — the essence of *live coding*.
|
||||
- The artifact is **terse source text** (a pattern); the "content" is the **sound it
|
||||
produces over time**. There is **no document, no output cell, no file of results** —
|
||||
output is **ephemeral, temporal, and performative**.
|
||||
- A Strudel "page" (a shared REPL link / snippet) is **source + the implicit promise of a
|
||||
running evaluation**. The source is tiny and diffable; the experience is not capturable as
|
||||
text.
|
||||
|
||||
## 2. The limit of static projection
|
||||
|
||||
Strudel pushes past Processing on every "live" dimension:
|
||||
|
||||
- **Temporal & generative** — output unfolds over time and may be **non-deterministic**
|
||||
(randomness, evolving state). There is no single "frame"; the faithful capture is a
|
||||
**recording (audio), itself just one performance**, not the artifact.
|
||||
- **Performative** — the value is partly the **act of live editing**; even a recording loses
|
||||
the live-coding dimension.
|
||||
- So on the **live↔snapshot axis** (named at T6), Strudel sits at the **far live end**: the
|
||||
best static projection is **(a) the source** (canonical, diffable) **+ (b) an optional
|
||||
audio recording snapshot**, explicitly marked as one rendering of a live/temporal artifact.
|
||||
|
||||
This makes Strudel the **honesty test** for the contract: shard-wiki must be able to attach
|
||||
such a source, present it truthfully (here is the source; a live render needs the runtime; a
|
||||
recording is one performance), and **never imply a static page captures it**.
|
||||
|
||||
## 3. INTENT mapping (enrichment-only — no new UC)
|
||||
|
||||
### Reinforcements / refinements
|
||||
|
||||
- **Live-evaluated, time-based content (UC-54/55).** Strudel is the extreme executable-as-
|
||||
page: **source canonical, presentation = a temporal live evaluation**. Confirms the page
|
||||
model must represent content whose rendered form is **time-based / generative / performative**.
|
||||
- **live↔snapshot axis (T6) — far end.** Establishes the **upper bound**: some content is
|
||||
**irreducibly live**; static projection degrades to **source + a recording snapshot**, with
|
||||
honesty about what's lost. Generalizes Processing's "snapshot frame" to "recording of one
|
||||
performance."
|
||||
- **Graceful degradation (INTENT).** A backend that can't run the REPL still serves the
|
||||
**source** (tiny, diffable) and any **recording** as read/projection/backup — the
|
||||
limited-backend-still-usable rule, at the hardest content type.
|
||||
- **Union without erasure.** Presenting a Strudel shard must surface "this is a **live
|
||||
temporal artifact**; what you see/hear statically is **source / one recording**" — never
|
||||
hide the liveness or imply completeness.
|
||||
|
||||
### Boundaries
|
||||
|
||||
- shard-wiki is **not an audio/REPL runtime**; default = attach the **source** + offer/store
|
||||
a **recording** with provenance; live in-viewer evaluation is a gated capability (trust/
|
||||
sandbox, like Processing T4). Source is canonical; everything rendered is a degrading,
|
||||
view-time/temporal projection.
|
||||
|
||||
## 4. UC disposition (enrichment-only)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC / thread |
|
||||
|------------------------|---------------------|
|
||||
| Pattern source = live, evaluated, time-based performance (§1) | UC-54 / UC-55 (enriched: time-based/generative executable content) |
|
||||
| Output ephemeral/temporal/non-deterministic → no faithful static form (§2) | links live↔snapshot axis (T6), far end |
|
||||
| Best static projection = source + audio recording snapshot, marked as one performance (§2) | links UC-83/UC-84 (degrading projection), UC-37 (recording as backup) |
|
||||
| Limited backend still serves source + recording (§3) | links UC-37 graceful degradation |
|
||||
| Live in-viewer evaluation = capability + trust/sandbox (§3) | links UC-35 |
|
||||
|
||||
**No new UC** — Strudel is design prior art marking the **far live end** of the projection/
|
||||
liveness model; it adds no orchestration scenario, it bounds one.
|
||||
|
||||
## 5. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T16 (projection):** anchor the **far end of the live↔snapshot axis** — content can be
|
||||
**irreducibly live/temporal/generative**; the contract must allow a projection to declare
|
||||
"no faithful static form; static = source + a marked recording." Combined with T4's
|
||||
**materialization-timing** and **continuity** facets, the projection model now spans:
|
||||
ahead-of-time materialized → view-time one-shot → view-time continuous/interactive →
|
||||
**temporal/generative/performative (recording-only snapshot)**.
|
||||
- **T12/T15 (page model):** **time-based / generative executable content** as a page-model
|
||||
edge; source canonical, render temporal.
|
||||
- **T11 (capabilities):** "live-evaluate (audio/REPL) in viewer" capability + trust/sandbox;
|
||||
default off → source + recording.
|
||||
|
||||
## 6. Open questions
|
||||
|
||||
1. Does the contract carry an explicit **"irreducibly live / no faithful static form"** flag
|
||||
on a projection (so the union renders the honest fallback automatically)? (The far-end
|
||||
resolution of the live↔snapshot axis open question from T6.)
|
||||
2. Is a **recording** modeled as a stored **derivation-projection snapshot with provenance**
|
||||
(one performance, time T, source rev R), reusing the UC-84 snapshot machinery?
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- `strudel.cc` (docs, REPL); **TidalCycles** (`tidalcycles.org`) — the pattern language
|
||||
Strudel ports; Web Audio.
|
||||
- prior: `research/260614-processing-deep-dive/` (view-time render, continuity, UC-54/55);
|
||||
`research/260614-squeak-pharo-deep-dive/` (live↔snapshot axis);
|
||||
`research/260614-mathematica-deep-dive/` (`Dynamic` interactive, snapshot-only).
|
||||
|
||||
## 8. Traceability
|
||||
|
||||
**No new UC** (enrichment / design prior art; the far live end). Enriched: UC-54, UC-55;
|
||||
links UC-83/UC-84 (degrading projection), UC-37 (recording = backup / graceful degradation),
|
||||
UC-35 (live-evaluate capability + trust). Architecture cross-refs: SHARD-WP-0002 T16 (far end
|
||||
of live↔snapshot axis: irreducibly-live content, static = source + marked recording), T12/T15
|
||||
(time-based/generative executable content), T11 (live-evaluate capability + sandbox).
|
||||
16
research/260614-tiddlywiki-deep-dive/README.md
Normal file
16
research/260614-tiddlywiki-deep-dive/README.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# 260614 — TiddlyWiki deep dive
|
||||
|
||||
Deep dive on **TiddlyWiki** (TW5): an entire wiki — tiddlers **plus** the app engine — in
|
||||
**one self-contained HTML file**, the **whole-file write-granularity** anchor of the
|
||||
synthesis matrix, with a Node.js **file-per-tiddler** (`.tid`) substrate as the git-diffable
|
||||
alternative, a tiddler/field record model, and **filter expressions** as the native query
|
||||
language.
|
||||
|
||||
- `findings.md` — the single-file model, tiddler data model, dual substrate, filters,
|
||||
capability profile, INTENT mapping, UC seed (UC-78), architecture notes for SHARD-WP-0002,
|
||||
open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-78 (attach a single-file self-contained wiki as one shard — parse tiddlers
|
||||
out, project; write = rewrite the whole file, the coarsest write-granularity anchor).
|
||||
Enriched UC-35/40/34/52/43. Feeds SHARD-WP-0002 T11 (write-granularity extreme) and T14
|
||||
(single-file vs file-per-tiddler binding).
|
||||
164
research/260614-tiddlywiki-deep-dive/findings.md
Normal file
164
research/260614-tiddlywiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,164 @@
|
||||
# TiddlyWiki — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T3 · **Subject:** TiddlyWiki / TiddlyWiki5
|
||||
(TW5), Jeremy Ruston's self-contained personal wiki.
|
||||
|
||||
## Why this dive
|
||||
|
||||
The synthesis matrix names **whole-file write granularity** as one extreme of the
|
||||
write-granularity spectrum, anchored by TiddlyWiki. This dive confirms the anchor and the
|
||||
*portability extreme* it implies — a wiki that is **a single HTML file you can email** — and
|
||||
finds the twist: TiddlyWiki also has a **Node.js file-per-tiddler** substrate, so it spans
|
||||
the granularity spectrum the way Logseq spans file/DB (UC-62). The question: how does
|
||||
shard-wiki attach a backend whose *entire content is one file*?
|
||||
|
||||
## 1. The single-file model
|
||||
|
||||
Classic TiddlyWiki ships as **one `.html` file** that contains **both**:
|
||||
|
||||
1. the **TiddlyWiki core** (the JavaScript engine, parser, renderer, UI), and
|
||||
2. **every tiddler** (all content), serialized into the file.
|
||||
|
||||
Open it in a browser and the file *is* the running application. There is **no server and no
|
||||
build step** — the app reconstitutes itself from the file it was loaded from. This is the
|
||||
portability extreme: a complete, self-hosting wiki in a single, emailable, USB-stick-able
|
||||
artifact that runs offline anywhere.
|
||||
|
||||
**Saving** is the catch: a browser page cannot normally overwrite the file it came from, so
|
||||
TiddlyWiki uses **"savers"** — TiddlyFox/browser extension, the File System Access API, a
|
||||
Node.js server, TiddlySpot/put-savers, or "download a new copy." Crucially, **a save
|
||||
rewrites the *entire* HTML file** (core + all tiddlers re-serialized). Hence **whole-file
|
||||
write granularity**: there is no concept of writing one page in isolation in the
|
||||
single-file mode — every save touches the whole artifact.
|
||||
|
||||
## 2. The tiddler data model
|
||||
|
||||
The atomic unit is the **tiddler** — a named record with **fields**:
|
||||
|
||||
- Core fields: **`title`** (the identity), **`text`** (the body), **`tags`**, **`created`**,
|
||||
**`modified`**, **`type`** (content type, e.g. `text/vnd.tiddlywiki`, `text/markdown`),
|
||||
plus **arbitrary custom fields** (any key→value). A tiddler is effectively a **flexible
|
||||
flat record** — closer to a typed-field record (UC-34) than to prose-with-frontmatter.
|
||||
- **Everything is a tiddler**: not just pages, but tags, macros, templates, themes, plugins,
|
||||
and the wiki's own configuration are all tiddlers. A **plugin is a bundle of tiddlers**.
|
||||
- Content markup is **WikiText** (TW5's own), though `type` can mark a tiddler as Markdown,
|
||||
JSON, image, etc. **Transclusion** is native: `{{SomeTiddler}}` embeds another tiddler;
|
||||
`{{SomeTiddler!!field}}` embeds a field.
|
||||
|
||||
## 3. The dual substrate — single-file vs file-per-tiddler
|
||||
|
||||
TiddlyWiki on **Node.js** stores each tiddler as a **separate `.tid` file** on disk: a small
|
||||
text file with a header of `field: value` lines, a blank line, then the body. The Node
|
||||
server assembles these into the same wiki at serve time. This substrate is:
|
||||
|
||||
- **git-diffable and fine-grained** — one file per tiddler, line-level diffs, per-tiddler
|
||||
history — the *opposite* end of the granularity spectrum from the single HTML file.
|
||||
- the natural attach surface for a *versioned, multi-author* TiddlyWiki.
|
||||
|
||||
So TiddlyWiki **spans the write-granularity spectrum by substrate** (single-file = whole-file
|
||||
write; Node = file-per-tiddler write), exactly as Logseq spans file/DB (UC-62) and as the
|
||||
backend-swap question (UC-43) anticipates.
|
||||
|
||||
## 4. Native query — filter expressions
|
||||
|
||||
TiddlyWiki's query language is **filter expressions** over tiddler fields, e.g.
|
||||
`[tag[todo]!tag[done]sort[modified]]` — a compact DSL that selects/orders tiddlers by field
|
||||
and tag. Lists, tables, and dynamic views are built from filters. This is a **native-query
|
||||
capability** (UC-52 tier) — less expressive than SPARQL/datalog but real, and computed over
|
||||
the tiddler store.
|
||||
|
||||
## 5. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | TiddlyWiki (single-file) | TiddlyWiki (Node `.tid`) |
|
||||
|--------------------------------|--------------------------|--------------------------|
|
||||
| Attachment mode | **file-store: one HTML file** | **file-store: dir of `.tid` files** |
|
||||
| Addressing granularity | tiddler (`title`) within the file | tiddler = one file |
|
||||
| Content identity | **`title`** (placement-bound) | title ↔ filename |
|
||||
| Structure | flat record store w/ arbitrary **fields** + tags | same |
|
||||
| History | none in-file (whole-file save) | **per-file git history** |
|
||||
| Merge model | whole-file replace (no merge) | git 3-way per tiddler |
|
||||
| Native query | **filter expressions** | filter expressions |
|
||||
| Translation | WikiText (or per-tiddler `type`: markdown/json/…) | same |
|
||||
| **Write granularity** | **whole file** (the anchor) | **file per tiddler** |
|
||||
| Operational envelope | trivial — a browser; no server | a Node server |
|
||||
| Access grant | file access = full access | server/file perms |
|
||||
| Content opacity | transparent (parse the HTML store) | transparent text |
|
||||
| Provenance | created/modified fields | git + fields |
|
||||
|
||||
## 6. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Graceful degradation**: a single-file TiddlyWiki is a *trivial* read-only / projection /
|
||||
backup shard — parse the tiddlers out of the HTML, project pages; no server needed. The
|
||||
limited-backend-still-usable principle at its simplest.
|
||||
- **Markdown-first but backend-neutral**: tiddlers carry a `type`, so Markdown tiddlers
|
||||
coexist with WikiText — the page model's content-type field maps directly.
|
||||
- **Typed fields** (UC-34): arbitrary tiddler fields are a flexible record model the page
|
||||
model already accommodates.
|
||||
- **Backend-swap under stable identity** (UC-43): single-file ↔ Node `.tid` is the same
|
||||
logical wiki on two substrates — the migration UC-43 anticipates, within one engine.
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- **Whole-file write granularity** is a real constraint: in single-file mode shard-wiki
|
||||
cannot write one page atomically — an overlay applied "to one page" still **rewrites the
|
||||
whole file** (T11). This is the coarsest write tier; model it explicitly so overlays/locks
|
||||
account for it (a write to any page conflicts with any concurrent write).
|
||||
- **Identity = title**, file-local; cross-shard identity (T16) layered above.
|
||||
- **The app is in the file**: when parsing a single-file TiddlyWiki, shard-wiki must extract
|
||||
the **tiddler store** and ignore the embedded engine — i.e. treat the HTML as a *container
|
||||
format*, not as page content (don't mistake the app for content).
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Single-file self-contained wiki as a first-class file-store shard** — container-format
|
||||
parse, whole-file write granularity (UC-78); the portability/granularity extreme.
|
||||
2. **Whole-file write granularity as a named tier** (T11) with overlay/lock implications.
|
||||
3. **Dual-substrate binding** (single-file vs `.tid` dir) as another instance of
|
||||
substrate-choice under one identity (UC-43/UC-62).
|
||||
|
||||
## 7. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-78 | Attach a **single-file self-contained wiki** (TiddlyWiki HTML) as one shard — parse tiddlers out of the container, project pages; **write = rewrite the whole file** (whole-file write granularity, the coarsest tier) | **new** |
|
||||
| — | whole-file write granularity anchor + overlay/lock implications | enrich **UC-35** |
|
||||
| — | single HTML file as a file-store shard (container format) | enrich **UC-40** |
|
||||
| — | tiddler arbitrary fields = flexible record | enrich **UC-34** |
|
||||
| — | filter expressions as a native-query tier | enrich **UC-52** |
|
||||
| — | single-file ↔ Node `.tid` substrate swap | enrich **UC-43** |
|
||||
|
||||
## 8. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T11 (capability / write granularity):** confirm **whole-file** as the coarsest named
|
||||
write tier (anchored by single-file TiddlyWiki), with the implication that an overlay to
|
||||
*any* page conflicts with concurrent writes (no per-page atomicity). File-per-tiddler is
|
||||
the fine tier on the same engine.
|
||||
- **T14 (attach binding):** a single-file wiki binds as a **container-format file-store**
|
||||
(parse tiddler store, ignore embedded engine); a Node TiddlyWiki binds as a **dir of
|
||||
`.tid` files** (git-diffable). One engine, two bindings — parameterize like UC-43.
|
||||
- **Native query:** filter expressions are a low-mid native-query tier between "none" and
|
||||
datalog/SPARQL — delegate where present (UC-52).
|
||||
|
||||
## 9. Open questions
|
||||
|
||||
1. In single-file mode, how does shard-wiki represent **per-page overlays** when writes are
|
||||
whole-file — buffer overlays and re-serialize, or require the Node `.tid` substrate for
|
||||
write-through and treat single-file as read/projection/backup only?
|
||||
2. Is a single-file TiddlyWiki's **embedded plugins/config** ever relevant to the union, or
|
||||
strictly ignored as app-internals (parse only content tiddlers)?
|
||||
3. Does shard-wiki expose tiddler **filter expressions** as a delegated query, or only its
|
||||
own union query over projected tiddlers?
|
||||
|
||||
## 10. Sources
|
||||
|
||||
- TiddlyWiki.com — *Tiddlers*, *TiddlerFields*, *Filters*, *Saving*, *Node.js* docs
|
||||
- *TiddlyWiki5* GitHub (Jermolene/TiddlyWiki5) — `.tid` file format, store structure
|
||||
- prior: `research/260614-logseq-deep-dive/` (file/DB dual substrate, UC-62)
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
New UC **UC-78** carries the marker **⊡** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-35, UC-40, UC-34, UC-52, UC-43. Architecture
|
||||
cross-refs: SHARD-WP-0002 T11 (whole-file tier), T14 (dual binding).
|
||||
53
research/260614-trilium-deep-dive/README.md
Normal file
53
research/260614-trilium-deep-dive/README.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# 260614 — Trilium (TriliumNext) deep dive (note cloning, attribute inheritance, HTML-native)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Trilium / TriliumNext** — the open-source hierarchical personal
|
||||
knowledge base — read through shard-wiki's lens. It is another SQLite-local note app, but
|
||||
it brings a structural feature **none of the prior twelve systems had: note cloning** —
|
||||
a single note can sit in **multiple places in the tree at once**, so the hierarchy is a
|
||||
**DAG, not a tree**, with **note identity cleanly separated from placement** (a note has
|
||||
many "branches"). It also has an **attribute system with inheritance + templates**
|
||||
(computed metadata) and is one of the few **HTML-native** (not Markdown) tools.
|
||||
|
||||
Lineage: **TriliumNext** is the community fork of the original **Trilium** (zadam, now
|
||||
maintenance-only) — the same pattern as TWiki→Foswiki.
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — single **SQLite** file (`document.db`); desktop or self-host server;
|
||||
multi-instance **sync protocol w/ conflict resolution** + WebSocket; 12-char IDs
|
||||
(`noteId`/`branchId`/`attributeId`/`attachmentId`)
|
||||
- **Note cloning** — note vs **branch** (identity vs placement); DAG hierarchy; no single
|
||||
canonical path
|
||||
- **Attributes** — labels (`#tag`) + typed relations (`~relation`), **inheritable** +
|
||||
**templates** (`~template`) → effective vs own metadata
|
||||
- **Content** — **HTML** (CKEditor5), plus canvas/mindmap/spreadsheet/geo/render note
|
||||
types; export to MD/HTML (lossy)
|
||||
- **Extension** — **scripting** (frontend/backend code notes, Script API) + **ETAPI**
|
||||
(token REST); **per-note encryption** (protected notes)
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, note cloning/DAG, attributes+inheritance, HTML content, scripting/ETAPI, per-note encryption, capability profile, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-66
|
||||
attach a shard with a DAG hierarchy / note cloning, identity separated from placement;
|
||||
UC-67 preserve inherited/templated attributes — effective vs own metadata); UC-15/22/34/
|
||||
38/42/61 enriched. Logged for `SHARD-WP-0002` (T11/T12/T14/T15/T16): DAG namespace +
|
||||
identity/placement (note/branch) split, computed/inherited metadata in the page model,
|
||||
**per-item** content opacity (refining the 12th spectrum), HTML as a source content model,
|
||||
and scripting + ETAPI host surfaces.
|
||||
|
||||
**Key takeaways recorded:** Trilium's **note/branch split (identity ≠ placement)** is the
|
||||
clean model for pages that live in multiple locations/shards and the namespace-level form
|
||||
of the clone/reference primitive; metadata is **computed** (own + inherited + template),
|
||||
not a flat bag; and **content opacity is per-item** (per-note encryption), not only
|
||||
whole-shard. **Boundary:** one SQLite-local candidate shard (DAG hierarchy, HTML-native),
|
||||
best attached via ETAPI; not a substrate and not the federation layer.
|
||||
</content>
|
||||
274
research/260614-trilium-deep-dive/findings.md
Normal file
274
research/260614-trilium-deep-dive/findings.md
Normal file
@@ -0,0 +1,274 @@
|
||||
# Findings — Trilium (TriliumNext): note cloning, attribute inheritance, HTML-native
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — an open-source hierarchical PKB; a *candidate
|
||||
shard* whose distinctive traits are **note cloning (a DAG hierarchy)**, an
|
||||
**attribute/relation system with inheritance + templates**, and **HTML-native** content
|
||||
Lens: shard-wiki — the namespace model (identity vs placement), inherited/computed
|
||||
metadata, HTML translation, per-note encryption, and the scripting/ETAPI surfaces
|
||||
|
||||
> Why Trilium earns a dive. It is another SQLite-local note app, but it brings a
|
||||
> structural feature **none of the prior twelve systems had: note cloning** — a single
|
||||
> note can sit in **multiple places in the tree at once**, so the hierarchy is a **DAG,
|
||||
> not a tree**, and **note identity is cleanly separated from placement** (a note has
|
||||
> many "branches"). That directly challenges shard-wiki's namespace model (UC-22 assumes
|
||||
> a path) and is the clone/reference primitive made concrete at the *namespace* level.
|
||||
> Trilium also has an **attribute system with inheritance and templates** (effective
|
||||
> metadata is computed, not just stored), and it is one of the few **HTML-native** (not
|
||||
> Markdown) tools — a useful stress on "Markdown-first must degrade."
|
||||
|
||||
Lineage note (like TWiki→Foswiki): **TriliumNext** is the community fork of the original
|
||||
**Trilium** (zadam) after it went maintenance-only; the active project is TriliumNext.
|
||||
|
||||
Contrast set: Joplin (SQLite-local, files-on-sync, page-level), Logseq (block-graph on
|
||||
files), Notion (hosted DB, schema+relations). Trilium = **SQLite-local, DAG hierarchy,
|
||||
inherited attributes, HTML content, self-host sync + ETAPI**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture — one SQLite file, server-syncable
|
||||
|
||||
- **Storage:** a single **SQLite** file (`document.db`, via better-sqlite3) holding
|
||||
notes, attachments, history, and settings. Local DB store (like Joplin; not files).
|
||||
- **Clients/sync:** desktop (Electron) or a self-hosted **server**; multi-instance
|
||||
**sync protocol** with **conflict resolution + entity change tracking**, plus
|
||||
**WebSocket** realtime updates. Attach via the server, the ETAPI (§5), or the DB.
|
||||
- **IDs:** 12-char IDs — **`noteId`, `branchId`, `attributeId`, `attachmentId`**. The
|
||||
**note vs branch** split is the key idea (§2).
|
||||
- **Export:** to Markdown/HTML (lossy — content is HTML, §4).
|
||||
|
||||
---
|
||||
|
||||
## 2. Note cloning — a DAG hierarchy, identity separated from placement
|
||||
|
||||
Notes form an arbitrarily deep tree, **but a single note can be placed in multiple
|
||||
locations** — "cloning," implemented via the **branches** model:
|
||||
|
||||
- A **note** (`noteId`) is the content + identity. A **branch** (`branchId`) is *one
|
||||
placement* of that note under a parent (with an optional **branch prefix** giving
|
||||
per-location context). A note with several branches is **cloned** — it appears in
|
||||
several places at once; editing it anywhere edits the one note.
|
||||
- So the hierarchy is a **DAG**, not a tree, and **identity (note) is separated from
|
||||
location (branch)**. There is no single canonical path for a cloned note.
|
||||
|
||||
This is the most shard-wiki-relevant feature in the dive:
|
||||
|
||||
- It breaks the assumption behind **UC-22** (resolve a page by *a* path). shard-wiki's
|
||||
namespace model must allow **a page in multiple namespace locations** with no single
|
||||
canonical path (UC-66).
|
||||
- The **note/branch split** is a clean model shard-wiki should borrow: **page identity ≠
|
||||
page placement**. A page is one entity; its locations are separate placement records —
|
||||
exactly how shard-wiki should treat a page that appears under multiple paths/shards.
|
||||
- It is the **clone/reference-not-copy primitive** (Xanadu clone, ZigZag clone, UC-44/45,
|
||||
T16) realized at the *namespace* level: same note, many positions, one source of truth.
|
||||
|
||||
---
|
||||
|
||||
## 3. Attributes — labels + relations, with inheritance and templates
|
||||
|
||||
Trilium's metadata system is **attributes**:
|
||||
|
||||
- **Labels** (`#tag`, optionally `#key=value`) — typed metadata on a note.
|
||||
- **Relations** (`~relation`) — **typed links to other notes** (a knowledge graph).
|
||||
- **Inheritance** — attributes can be **inherited down the subtree** (inheritable
|
||||
attributes), and **templates** (a note whose attributes/structure are applied to
|
||||
instances via a `~template` relation) inject attribute sets. So a note's **effective
|
||||
metadata = its own attributes + inherited + templated** — computed, not just stored.
|
||||
- Promoted attributes give a form-like UI; attributes drive **search/queries** and
|
||||
**scripting**.
|
||||
|
||||
shard-wiki read: this is structured data (UC-34) and typed relations (UC-58), but with a
|
||||
**new wrinkle — inheritance/templates make metadata *computed***. Projecting such a shard
|
||||
must distinguish **effective vs own** attributes (and record their provenance: own /
|
||||
inherited-from / template), not flatten them (UC-67). Templates also reinforce blueprint
|
||||
pages (UC-15).
|
||||
|
||||
---
|
||||
|
||||
## 4. Content model — HTML-native, many note types
|
||||
|
||||
- **Text notes are HTML** (WYSIWYG via **CKEditor5**), **not Markdown**. Trilium is one
|
||||
of the few HTML-first tools studied. Markdown participation therefore needs **HTML↔
|
||||
Markdown translation** — more tractable than Notion's block model but still **lossy**
|
||||
for some constructs (CKEditor features, includes) → a fidelity-aware case bridging
|
||||
UC-42 (lossless ideal) and UC-59 (lossy-with-report).
|
||||
- **Note types:** text (HTML), code (CodeMirror), **canvas (Excalidraw)**, relation
|
||||
maps, mind maps (Mind Elixir), spreadsheets (Univer), geo maps (Leaflet), **render
|
||||
notes**, file/image, book — i.e. lots of **non-Markdown content** (UC-55) and
|
||||
**script/render-generated dynamic notes** (UC-54).
|
||||
|
||||
---
|
||||
|
||||
## 5. Extension surfaces — scripting (in-app) + ETAPI (external)
|
||||
|
||||
Two surfaces, like Joplin:
|
||||
|
||||
- **Scripting (in-engine host).** **Code notes** in JS run as **frontend** (UI widgets,
|
||||
buttons) or **backend** (server-side automation) scripts against a **Script API**
|
||||
(create/query notes, attributes, etc.). Executable, in-app — the adapter-host path
|
||||
(UC-38). "Render notes" + scripts produce **dynamic, generated content** (UC-54).
|
||||
- **ETAPI (external REST API).** Trilium's public REST API (since v0.50), **token auth**
|
||||
(`Bearer ETAPITOKEN`), with client libs (`trilium-py`): CRUD over notes/branches/
|
||||
attributes/attachments, search, import/export — an external attach surface (UC-57)
|
||||
that, unlike Joplin's localhost-only Data API, is the **designed external integration
|
||||
API** for a (often self-hosted) server.
|
||||
|
||||
---
|
||||
|
||||
## 6. Security — per-note (partial) encryption
|
||||
|
||||
Trilium offers **strong per-note encryption** ("protected notes") unlocked in a
|
||||
**protected session** (password; TOTP/OpenID for login). Crucially this is **per-note**:
|
||||
a shard can hold **some encrypted and some plaintext notes at once**. This refines the
|
||||
**content-opacity** dimension (UC-61, proposed twelfth spectrum from Joplin/Anytype):
|
||||
opacity is **per-item, not only whole-shard** — the adapter must handle a shard where
|
||||
*part* of the content is opaque without a key (protected notes projectable only as
|
||||
structure-shell; unprotected notes fully).
|
||||
|
||||
---
|
||||
|
||||
## 7. Trilium as a shard — capability profile
|
||||
|
||||
| Capability | Trilium | Notes for the adapter contract |
|
||||
|------------|---------|--------------------------------|
|
||||
| Read | yes (ETAPI / sync / DB) | ETAPI REST (server) is the clean surface |
|
||||
| Write | yes (ETAPI / script) | per-note; via ETAPI or backend code notes |
|
||||
| Write granularity | per-note (page) | — |
|
||||
| Identity / addressing | **`noteId` + `branchId`** (12-char) | **identity ≠ placement**; cloned notes have many branches (UC-51, UC-66) |
|
||||
| Hierarchy | **DAG (cloning)** | a note in multiple locations; no single canonical path (UC-66) |
|
||||
| Structure | **labels + relations, inherited + templated** | effective vs own metadata (UC-34, UC-58, UC-67) |
|
||||
| Content | **HTML (CKEditor)** + many types | HTML↔MD lossy translation (UC-42/59); non-MD types (UC-55) |
|
||||
| History | internal revisions in the DB | not portable git → supplement (UC-36) |
|
||||
| Native query | attribute search / script queries | delegate or build index (UC-52/63) |
|
||||
| Subscribe | WebSocket + sync protocol | push; conflict resolution built in (UC-31) |
|
||||
| Content opacity | **per-note encryption (partial)** | per-item opacity (UC-61 refined) |
|
||||
| Extension | **scripting (in-app) + ETAPI (external)** | dual surface (UC-38, UC-57) |
|
||||
| Templates | yes (`~template`) | blueprint pages (UC-15) |
|
||||
|
||||
Verdict: a capable **SQLite-local, server-syncable** shard best attached via **ETAPI**.
|
||||
Standout demands: the **DAG hierarchy / note-branch identity model** (UC-66) and
|
||||
**inherited/templated attributes** (UC-67); plus HTML-native translation, per-note
|
||||
opacity, and the scripting host.
|
||||
|
||||
---
|
||||
|
||||
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 8.1 Reinforcements
|
||||
|
||||
- **Identity ≠ placement** (note/branch) is a model shard-wiki should adopt for pages
|
||||
that appear under multiple paths or in multiple shards — provenance and union without
|
||||
erasure depend on separating "what a page is" from "where it sits."
|
||||
- **Typed relations + templates** validate the structured/relation demand (UC-58) and
|
||||
blueprints (UC-15) on a local-first, self-hostable backend.
|
||||
- **ETAPI** is a clean example of a **designed external REST surface** for a self-hosted
|
||||
server (cf. Notion external API, but self-hosted/trust-boundary friendly).
|
||||
|
||||
### 8.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Don't force a single canonical path.** A cloned note has many placements; modeling
|
||||
it with one path loses information (UC-66). Use the note/branch separation.
|
||||
2. **Don't flatten computed metadata.** Effective attributes include inherited/templated
|
||||
values; record provenance (own vs inherited vs template), don't collapse (UC-67).
|
||||
3. **HTML-native, not Markdown.** Translate HTML↔Markdown with a fidelity report; degrade
|
||||
to read-only where lossy (UC-42/59/03). Markdown-first must degrade gracefully.
|
||||
4. **Per-note opacity.** Some notes are encrypted; never surface protected-note
|
||||
ciphertext; project them as structure-shell (UC-61).
|
||||
5. **One Trilium instance = one shard**, not the federation layer; its sync protocol is
|
||||
its own — attach via ETAPI/replica, don't re-drive it (not-a-sync-daemon).
|
||||
|
||||
### 8.3 What Trilium teaches that shard-wiki should keep
|
||||
|
||||
- **Separate page identity from placement** (note vs branch) — the cleanest model for
|
||||
multi-location / multi-shard pages and for the clone/reference primitive (T16).
|
||||
- **Model metadata as computed** (own + inherited + templated) with per-attribute
|
||||
provenance (UC-67) — not a flat key/value bag.
|
||||
- **Content opacity is per-item**, not only whole-shard (UC-61 refinement).
|
||||
|
||||
---
|
||||
|
||||
## 9. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-65**. New UCs **UC-66, UC-67** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a shard with a DAG hierarchy / note cloning** — a page may occupy multiple namespace locations at once (note identity separated from placement via branches); no single canonical path | **UC-66 (new)** |
|
||||
| **Preserve inherited / templated attributes** — project a structured shard whose metadata is computed (own + inherited + template), distinguishing effective vs own with per-attribute provenance | **UC-67 (new)** |
|
||||
| DAG/multi-parent vs tree; note/branch (identity vs placement) | **enriches UC-22** |
|
||||
| Labels (`#tag`) + typed relations (`~relation`); HTML-native | **enriches UC-34** |
|
||||
| Templates (`~template`) inject attribute sets | **enriches UC-15** |
|
||||
| HTML (CKEditor) content → HTML↔Markdown lossy translation | **enriches UC-42** (links UC-59) |
|
||||
| Per-note encryption (protected notes) = **partial** content opacity | **enriches UC-61** |
|
||||
| Scripting (frontend/backend code notes, Script API) = in-app host | **enriches UC-38** |
|
||||
| ETAPI (token REST) = designed external surface for a self-host server | links UC-57 |
|
||||
| Render/script notes + attribute queries = dynamic content | links UC-54; typed relations link UC-58 |
|
||||
| `noteId`/`branchId` 12-char IDs | links UC-51 |
|
||||
|
||||
---
|
||||
|
||||
## 10. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Namespace model must support a DAG** and **separate page identity from placement**
|
||||
(note/branch). A page is one entity with N placements (paths/shards). Feeds the page/
|
||||
namespace model and the clone/reference primitive. (T12, T16.)
|
||||
- **Computed/inherited metadata** (UC-67): the page model's structured-data
|
||||
representation must carry **effective vs own** with per-attribute provenance
|
||||
(own/inherited/template), not a flat bag. (T12.)
|
||||
- **Content opacity is per-item** (UC-61 refinement): the content-opacity capability
|
||||
(proposed twelfth spectrum) should be **granular** (per-note), not only whole-shard.
|
||||
(T11.)
|
||||
- **HTML as a source content model** joins TML/Notion-blocks in the translation
|
||||
capability (HTML↔Markdown, lossy-aware). (T15.)
|
||||
- **Scripting as an in-engine host** + **ETAPI external REST** are two more adapter-host
|
||||
exemplars (with Roam/Obsidian/Joplin). (T14.)
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions (for spec / workplans)
|
||||
|
||||
1. How does shard-wiki represent a **cloned note** in the union — one page with multiple
|
||||
path placements, or a page transcluded into multiple locations? (UC-66 vs UC-44/45.)
|
||||
2. When projecting **inherited attributes** (UC-67), does shard-wiki materialize
|
||||
effective values (snapshot) or compute them live from the shard's tree/templates?
|
||||
3. Is **HTML↔Markdown** round-trip lossless enough for write-back overlays, or are
|
||||
Trilium overlays read-only/native-HTML (cf. UC-42 Q2)?
|
||||
4. For **per-note encryption** (UC-61), is a partially-opaque shard projected with
|
||||
protected notes as visible-but-opaque placeholders, or hidden entirely?
|
||||
|
||||
---
|
||||
|
||||
## 12. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| TriliumNext/Trilium — DeepWiki (https://deepwiki.com/TriliumNext/Trilium) | SQLite `document.db`/better-sqlite3; tree + cloning via branches; note types; sync |
|
||||
| Cloning notes — TriliumNext wiki (https://github.com/TriliumNext/Trilium/wiki/Cloning-notes) | Note in multiple locations; branches; branch prefixes |
|
||||
| TriliumNext — DeepWiki API & Synchronization (https://deepwiki.com/TriliumNext/Trilium/4.3-api-and-synchronization) | WebSocket + sync protocol w/ conflict resolution; 12-char IDs (note/branch/attribute/attachment) |
|
||||
| ETAPI (REST API) — Trilium docs (https://docs.triliumnotes.org/User%20Guide/User%20Guide/Advanced%20Usage/ETAPI%20(REST%20API)/) | Public REST API since v0.50; token/Bearer auth; trilium-py |
|
||||
| Script API — TriliumNext wiki (https://github.com/TriliumNext/Trilium/wiki/Script-API) | Frontend/backend code notes; Script API |
|
||||
| BrightCoding — TriliumNext overview (https://www.blog.brightcoding.dev/2025/09/20/triliumnext-notes...) | Attributes (labels/relations), inheritance, templates, per-note encryption, note types |
|
||||
|
||||
Cross-references: `research/260614-joplin-deep-dive/findings.md` (SQLite-local, dual
|
||||
surface, content opacity), `research/260614-notion-deep-dive/findings.md` (typed
|
||||
relations, external API), `research/260614-zigzag-deep-dive/findings.md` (clone /
|
||||
dimensions), `research/260614-shard-spectrum-synthesis/findings.md` (spectra this
|
||||
refines), `spec/UseCaseCatalog.md` (UC-15, UC-22, UC-34, UC-38, UC-42, UC-51, UC-54,
|
||||
UC-57, UC-58, UC-61), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T12,
|
||||
T14, T15, T16).
|
||||
|
||||
---
|
||||
|
||||
## 13. Traceability
|
||||
|
||||
- New UCs: **UC-66, UC-67** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-15, UC-22, UC-34, UC-38, UC-42, UC-61** (links UC-51, UC-54, UC-57,
|
||||
UC-58, UC-59).
|
||||
- Architecture (no UC): DAG namespace + identity/placement (note/branch) split;
|
||||
computed/inherited metadata; per-item content opacity; HTML source model; scripting +
|
||||
ETAPI host surfaces → `SHARD-WP-0002` (T11, T12, T14, T15, T16).
|
||||
- Boundary recorded: Trilium (TriliumNext) is **one SQLite-local candidate shard** with a
|
||||
DAG hierarchy and computed metadata, best attached via **ETAPI**; HTML-native (lossy to
|
||||
Markdown), per-note opacity; not a substrate, not the federation layer (INTENT
|
||||
graceful-degradation, no-silent-mutation, not-a-sync-daemon).
|
||||
</content>
|
||||
14
research/260614-usemodwiki-deep-dive/README.md
Normal file
14
research/260614-usemodwiki-deep-dive/README.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# 260614 — UseModWiki deep dive
|
||||
|
||||
Deep dive on **UseModWiki**: the **flat-file ancestor** — a Perl CGI wiki (Clifford Adams,
|
||||
2000) descended from AtisWiki/CvWiki, **CamelCase** linking, plain flat-file page + history
|
||||
storage, and the **engine Wikipedia originally ran on** (MediaWiki Phase I). Origins/lineage
|
||||
value: the minimal flat-file page+history model the whole field descends from.
|
||||
|
||||
- `findings.md` — the model, lineage, capability profile, INTENT mapping, enrichments (no new
|
||||
UC — reinforces the minimal flat-file baseline UC-82), architecture notes, sources,
|
||||
traceability.
|
||||
|
||||
Catalog yield: **enrichment-only** (reinforces UC-82) — enriched UC-01 (open wiki), UC-40
|
||||
(flat-file store), UC-25 (CamelCase naming), UC-36/41 (flat-file history). Lineage noted for
|
||||
the origins record. Feeds SHARD-WP-0002 T11 (minimal profile lineage).
|
||||
87
research/260614-usemodwiki-deep-dive/findings.md
Normal file
87
research/260614-usemodwiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# UseModWiki — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T9 · **Subject:** UseModWiki, Clifford
|
||||
Adams's flat-file Perl wiki — the lineage root of much of the field.
|
||||
|
||||
## Why this dive
|
||||
|
||||
This is a **lineage** dive, not a new-capability one. UseModWiki is the **ancestor**: the
|
||||
minimal flat-file page+history wiki the c2-era engines and **MediaWiki Phase I** descend
|
||||
from (Wikipedia ran on UseModWiki, 2001–2002). It pairs with
|
||||
`research/260608-c2-wiki-origins/` to record *where the page+history model came from*. It
|
||||
adds no new shard capability beyond the minimal flat-file floor (UC-82, Oddmuse) — its value
|
||||
is **historical grounding** and confirming the floor is genuinely the field's common root.
|
||||
|
||||
## 1. The model
|
||||
|
||||
- **Perl CGI** (`wiki.pl`), single-script, descended from AtisWiki → CvWiki → UseModWiki
|
||||
(c. 2000), by Clifford Adams.
|
||||
- **Flat-file storage**: each page stored as a text file (under a `db/` data directory),
|
||||
with **page history** kept (older revisions retained as files / in the page record). No
|
||||
database.
|
||||
- **Linking**: **CamelCase** (`WikiWord`) auto-links by default; later added **free links**
|
||||
`[[Like This]]`. This is the canonical CamelCase-naming lineage (UC-25).
|
||||
- **Open editing**, recent-changes, simple markup — the c2 ethos in a portable script.
|
||||
|
||||
## 2. Lineage
|
||||
|
||||
- **MediaWiki Phase I** *was* UseModWiki — early Wikipedia ran on it before the PHP rewrite
|
||||
(Phase II/III). So UseModWiki is the **direct ancestor of MediaWiki/Wikibase** (T2) and a
|
||||
sibling-root of the c2 tradition.
|
||||
- The **flat-file page+history model** here is the shape Oddmuse (UC-82), TWiki's file store
|
||||
(UC-40), and others elaborate — confirming the **minimal file-store floor** is the field's
|
||||
common origin, not a modern simplification.
|
||||
|
||||
## 3. Capability profile
|
||||
|
||||
Essentially **identical to the minimal floor** (UC-82, Oddmuse):
|
||||
|
||||
| Dimension | UseModWiki |
|
||||
|-----------|------------|
|
||||
| Attachment mode | **file-store** (flat files under `db/`); CGI, no API |
|
||||
| Addressing | page = file; **CamelCase** name = identity |
|
||||
| Structure | flat page space; CamelCase link graph |
|
||||
| History | flat-file retained revisions (may be limited) |
|
||||
| Native query | none |
|
||||
| Translation | simple wiki markup → Markdown (lossy) |
|
||||
| Write granularity | page (file) |
|
||||
| Access | open editing (optional admin password) |
|
||||
| Provenance | timestamp, optional username |
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
- **Reinforces the minimal flat-file baseline** (UC-82): UseModWiki is the *historical*
|
||||
instance of the graceful-degradation floor — attach via its flat files; partial history
|
||||
surfaced honestly.
|
||||
- **CamelCase naming** (UC-25): the canonical source of WikiWord auto-linking — the page
|
||||
model's name/identity and link-resolution must accommodate CamelCase-derived identities.
|
||||
- **Open wiki** (UC-01): the c2 open-editing ethos at the root.
|
||||
- **Lineage grounding**: confirms shard-wiki's "Git-based Markdown" page model descends from
|
||||
(and must remain attach-compatible with) the flat-file ancestor.
|
||||
|
||||
**No new UC** — UseModWiki adds historical grounding, not a new orchestration scenario;
|
||||
its capabilities are subsumed by UC-82 (minimal flat-file baseline) plus UC-25 (CamelCase).
|
||||
|
||||
## 5. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T11:** UseModWiki = a second instance of the **minimal/floor profile** (with Oddmuse);
|
||||
confirms the floor is the field's common root, so the floor profile is the right baseline.
|
||||
- **Naming/identity:** CamelCase-derived page identities (UC-25) are part of the legacy
|
||||
identity surface the adapter must parse (and translate to the page model's identity).
|
||||
|
||||
## 6. Open questions
|
||||
|
||||
(None new — covered by UC-82 partial-history honesty, catalog open-Q30, and the c2 origins
|
||||
record. CamelCase resolution is UC-25.)
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- UseModWiki — usemod.com/cgi-bin/wiki.pl (the wiki about itself); Wikipedia: *UseModWiki*,
|
||||
*MediaWiki* (Phase I history)
|
||||
- prior: `research/260608-c2-wiki-origins/`; `research/260614-oddmuse-deep-dive/` (minimal
|
||||
floor, UC-82)
|
||||
|
||||
## 8. Traceability
|
||||
|
||||
**No new UC** (reinforces UC-82). Enriched: UC-01, UC-40, UC-25, UC-36, UC-41. Lineage noted
|
||||
for the origins record. Architecture cross-ref: SHARD-WP-0002 T11 (minimal-profile lineage).
|
||||
16
research/260614-wikibase-deep-dive/README.md
Normal file
16
research/260614-wikibase-deep-dive/README.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# 260614 — Wikibase / Wikidata deep dive
|
||||
|
||||
Deep dive on **Wikibase** (the MediaWiki extension behind **Wikidata**) as the
|
||||
**entity-statement (RDF) far end** of the structure and native-query spectra: items and
|
||||
properties, statements = claim + qualifiers + references + rank, stable opaque IDs, and
|
||||
**SPARQL** (Wikidata Query Service / Blazegraph) including **federated** queries.
|
||||
|
||||
- `findings.md` — data model, RDF/SPARQL surface, storage & history, capability profile,
|
||||
INTENT mapping, UC seeds (UC-73–UC-75), architecture notes for SHARD-WP-0002, open
|
||||
questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-73 (attach a typed entity-statement / RDF shard, lossy render to a page),
|
||||
UC-74 (graph query the union via SPARQL + federated `SERVICE` cross-endpoint query), UC-75
|
||||
(statement-level provenance — references + rank per assertion). Enriched UC-34/58/52/24.
|
||||
Feeds SHARD-WP-0002 T12 (structured/typed page model) and T16 (stable language-neutral
|
||||
identity ≠ label/placement).
|
||||
200
research/260614-wikibase-deep-dive/findings.md
Normal file
200
research/260614-wikibase-deep-dive/findings.md
Normal file
@@ -0,0 +1,200 @@
|
||||
# Wikibase / Wikidata — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T2 · **Subject:** Wikibase (MediaWiki
|
||||
extension) and its flagship instance **Wikidata**, incl. the Wikidata Query Service (SPARQL).
|
||||
|
||||
## Why this dive
|
||||
|
||||
Every structured shard so far tops out at *typed records in a database*: Notion's
|
||||
database-pages, XWiki's XObjects/classes, Trilium's typed relations, Roam/Logseq's
|
||||
attribute blocks. Wikibase is a *different kind of structure altogether* — a **typed
|
||||
knowledge graph of entities and provenance-bearing statements**, queried with **SPARQL**
|
||||
over an **RDF** projection. It is the **far end of the structure spectrum** and of the
|
||||
**native-query spectrum**, and it pushes **provenance down to the individual assertion**.
|
||||
The question for shard-wiki: what does a shard look like when its "page" is *not prose but a
|
||||
set of statements*, and what does the page model / adapter contract owe such a shard?
|
||||
|
||||
## 1. The data model — entities, statements, snaks
|
||||
|
||||
**Entities** are the top-level objects, each on its own MediaWiki page with a **stable
|
||||
opaque ID**:
|
||||
|
||||
- **Item** — `Q42`. Has multilingual **labels / descriptions / aliases**, a set of
|
||||
**statements**, and **sitelinks** (links to wiki articles). The label is *annotation*,
|
||||
not identity — `Q42` is the identity, "Douglas Adams" is just its English label.
|
||||
- **Property** — `P31` ("instance of"). Also has labels/descriptions/aliases, plus a
|
||||
**fixed datatype** constraining its values (item-reference, string, time,
|
||||
globe-coordinate, quantity, monolingual-text, url, external-id, commons-media, …).
|
||||
|
||||
**Statement** = the unit of assertion on an item. Structure:
|
||||
|
||||
```
|
||||
statement = claim + references[] + rank
|
||||
claim = mainSnak + qualifiers[]
|
||||
snak = property + snaktype + (value) # snaktype ∈ value | somevalue | novalue
|
||||
```
|
||||
|
||||
- **Main snak** — the core property→value assertion (e.g. `P31` → `Q5` "human").
|
||||
- **Qualifiers** — snaks that *contextualize* the claim without being the subject (validity
|
||||
time, "as of", determination method, units). E.g. *population (P1082) = 8.4M, **point in
|
||||
time (P585) = 2020***.
|
||||
- **References** — lists of snaks citing **where the claim comes from** (a source item, a
|
||||
URL, a page number). **Provenance attached to the individual statement, not the page.**
|
||||
- **Rank** — `preferred` | `normal` | `deprecated`: relative importance among same-property
|
||||
statements (lets multiple, even contradictory, values coexist with a curation signal —
|
||||
the structured analogue of fedwiki's "chorus").
|
||||
- Each statement carries a **stable GUID** (`Q42$<uuid>`), so statements are individually
|
||||
addressable.
|
||||
|
||||
`somevalue` (known to exist, value unknown) and `novalue` (known *not* to have a value) are
|
||||
**first-class** — the model represents *known-unknowns* explicitly, which prose and most
|
||||
DBs cannot.
|
||||
|
||||
## 2. The RDF / SPARQL surface
|
||||
|
||||
Wikibase **projects entities to RDF**; the **Wikidata Query Service (WDQS)** is a
|
||||
**Blazegraph** triple store exposing a **SPARQL** endpoint. The projection is deliberately
|
||||
multi-layered:
|
||||
|
||||
- **Truthy** triples (`wdt:` prefix) — the simple "best" value, for easy queries:
|
||||
`wd:Q42 wdt:P31 wd:Q5`.
|
||||
- **Full** statements — reified so qualifiers/references/rank survive: `wd:Q42 p:P31
|
||||
?stmt . ?stmt ps:P31 wd:Q5 ; pq:P585 ?time ; prov:wasDerivedFrom ?ref`. (`p:`=statement
|
||||
node, `ps:`=statement value, `pq:`=qualifier, `pr:`/`prov:`=reference.)
|
||||
- **Federated SPARQL** — the `SERVICE <endpoint> { … }` keyword runs a sub-query against
|
||||
*another* SPARQL endpoint and joins the results. **Query-level federation is built into
|
||||
the query language** — a different federation primitive from fedwiki's fork/neighborhood.
|
||||
- **EntitySchemas / ShEx** — schemas (`E`-ids) that *validate* an entity's shape (Shape
|
||||
Expressions). Optional, declarative structure validation over the open graph.
|
||||
|
||||
## 3. Storage, identity, history
|
||||
|
||||
- **Storage:** each entity is a **JSON blob stored as a MediaWiki page** (`Item:` /
|
||||
`Property:` content model). The RDF/SPARQL store is a **derived index** rebuilt from these
|
||||
canonical JSON entities (an *update stream* feeds WDQS) — exactly shard-wiki's
|
||||
"derived query index over a canonical store" pattern (UC-63), at planet scale.
|
||||
- **Identity:** the **opaque Q/P/L IDs are the identity**, fully decoupled from
|
||||
human-readable labels and from language. This is the cleanest real-world instance of
|
||||
**stable, language-neutral identity ≠ label/placement** — a strong reinforcement of our
|
||||
identity model (T16).
|
||||
- **History:** because each entity is one MediaWiki page, history is **page-level MediaWiki
|
||||
revisions** — every edit is a full-entity JSON snapshot with author/timestamp/comment.
|
||||
*Coarse* history granularity (whole entity per revision), but the **edit API is
|
||||
fine-grained** (`wbsetclaim`, `wbeditentity` patch individual statements). So: **fine
|
||||
write API over a coarse history unit** — a distinct point on the write/history spectra.
|
||||
|
||||
## 4. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | Wikibase / Wikidata |
|
||||
|--------------------------------|---------------------|
|
||||
| Attachment mode | **external-API** (MediaWiki Action API + REST) **and** a derived **SPARQL endpoint**; self-hostable |
|
||||
| Addressing granularity | **statement** (each has a GUID) within an **entity** (Q/P id) |
|
||||
| Content identity | **stable opaque ID** (Q/P/L); labels are multilingual annotations |
|
||||
| Identity vs placement | **fully separated** — identity is language- and label-neutral |
|
||||
| Structure | **typed knowledge graph**: entities + statements (claim+qualifiers+refs+rank) |
|
||||
| History | **page-level revisions** (whole-entity JSON snapshots); fine-grained edit API |
|
||||
| Merge model | MediaWiki last-writer / edit-conflict; rank lets contradictory values coexist |
|
||||
| Native query | **SPARQL** (RDF) + **federated `SERVICE`** cross-endpoint join — the far end |
|
||||
| Translation | **not Markdown** — content *is* statements; render to prose is a lossy projection |
|
||||
| Attachment/write granularity | **statement-level writes** via API; coarse history unit |
|
||||
| Operational envelope | huge derived index (Blazegraph), rate-limited public endpoints |
|
||||
| Access grant | open read; MediaWiki user/permission model for write; self-host = own ACL |
|
||||
| Content opacity | transparent (public JSON + RDF); not encrypted |
|
||||
| Provenance | **statement-level** — references + rank per assertion (new far end) |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Stable identity ≠ placement** (T16): Q/P IDs decoupled from labels/language are the
|
||||
textbook case — adopt the principle that a page's *identity* is an opaque stable handle,
|
||||
display names are annotations.
|
||||
- **Derived index over canonical store** (UC-63): WDQS is exactly a SPARQL index rebuilt
|
||||
from canonical JSON entities via an update stream — validates the projection pattern.
|
||||
- **Union without erasure / chorus**: **rank** lets multiple (even contradictory) statements
|
||||
coexist with a curation signal rather than forcing one truth — the *structured* analogue
|
||||
of fedwiki's chorus (UC-72) and our "view multiple versions" (UC-27).
|
||||
- **Mechanism over policy**: references + rank are *mechanism* for representing disagreement
|
||||
and sourcing; which statement "wins" is left to the consumer/query.
|
||||
|
||||
### Divergences (boundaries / design notes)
|
||||
|
||||
- **Content is not Markdown.** A Wikibase "page" is a set of statements; there is no prose
|
||||
body. This is the **structure far-end**: shard-wiki must either (a) treat such a shard as
|
||||
a **structured/typed shard** projected to a *lossy* Markdown/table rendering (UC-55/UC-73),
|
||||
or (b) model a page whose payload is typed statements (T12). Forcing it into Markdown-first
|
||||
erases the graph — a design-bug if done silently; render-with-provenance instead.
|
||||
- **Provenance granularity is finer than ours.** Our provenance is per-page/per-shard;
|
||||
Wikibase is **per-statement** (references) and even per-value (rank). The page model and
|
||||
coordination journal should *allow* sub-page provenance (UC-75) even if MVP records it per
|
||||
page.
|
||||
- **Query is graph, not text/datalog.** SPARQL over RDF (with federated `SERVICE`) is a
|
||||
richer query far-end than Roam/Logseq datalog or Notion filters (UC-52) — and its
|
||||
`SERVICE` federation is a *query-time* cross-shard join, distinct from fedwiki structural
|
||||
federation. Note both as native-query tiers.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Opaque stable identity, labels-as-annotations** as the identity model (T16).
|
||||
2. **Statement/assertion-level provenance** (references) and a **coexistence-with-rank**
|
||||
model as the structured form of union-without-erasure (UC-75).
|
||||
3. **Derived SPARQL/graph index over a canonical entity store** as a projection pattern
|
||||
(UC-63/UC-74), incl. **federated query** as a first-class federation mode.
|
||||
4. A **typed-graph page payload** option in the page model (T12), with **lossy
|
||||
render-to-Markdown** as the projection (never silent flattening).
|
||||
|
||||
## 6. UC seeds
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-73 | Attach a **Wikibase** as a **typed entity-statement (RDF) shard** (items/properties/statements w/ qualifiers); project to a rendered page view, lossy to Markdown, preserving the graph | **new** |
|
||||
| UC-74 | **Graph-query the union** via **SPARQL** and **federate queries across endpoints** (`SERVICE`) — graph query as a native-query tier + query-time cross-shard join | **new** |
|
||||
| UC-75 | Preserve **statement-level provenance** — references + rank attached to each assertion (sub-page provenance granularity) | **new** |
|
||||
| — | typed records → typed *graph* entities | enrich **UC-34** |
|
||||
| — | inter-record relations → typed graph edges with qualifiers | enrich **UC-58** |
|
||||
| — | native query → SPARQL/RDF + federated SERVICE | enrich **UC-52** |
|
||||
| — | provenance → statement/assertion granularity | enrich **UC-24** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12 (structured/typed page model):** add a **typed-graph payload** tier above
|
||||
typed-records — a page whose content is **entities + statements (claim + qualifiers +
|
||||
references + rank)**, with `somevalue`/`novalue` known-unknowns. Render-to-Markdown is a
|
||||
**lossy projection**, not the canonical form.
|
||||
- **T16 (identity / addressing):** adopt **opaque stable identity with labels-as-annotation**
|
||||
(Q/P model); record **statement GUIDs** as an example of *sub-page addressable units*.
|
||||
- **Native-query tiering:** SPARQL/RDF + federated `SERVICE` is the **graph far-end** of the
|
||||
query spectrum (above datalog/filters); `SERVICE` is also a **query-time federation**
|
||||
mode to sit beside fedwiki's structural federation.
|
||||
- **Provenance model:** allow **per-statement references + rank** (sub-page provenance,
|
||||
coexistence-with-curation) in the union, even if MVP collapses to per-page.
|
||||
- **Derived index:** WDQS = canonical JSON entities → update stream → Blazegraph SPARQL
|
||||
index; the reference implementation of UC-63 at scale (per-shard or core-built index, Q16).
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Does shard-wiki model a **typed-graph page** natively (T12), or always treat Wikibase as
|
||||
a structured shard **projected to a Markdown/table rendering** (UC-55), or both
|
||||
(canonical graph + lossy view)?
|
||||
2. Is **SPARQL/graph query** exposed as a union-level capability (translate to a common
|
||||
query layer) or only as a **pass-through** to graph-capable shards? How does federated
|
||||
`SERVICE` relate to shard-wiki's own cross-shard query?
|
||||
3. At what granularity does the coordination journal record **provenance** — per page
|
||||
(MVP), per statement (Wikibase-native), or configurable?
|
||||
4. Is **rank** (coexisting contradictory values w/ curation) representable in the union as a
|
||||
first-class "chorus of statements," unifying with fedwiki's page-level chorus (UC-72/27)?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- Wikibase/DataModel and **DataModel/Primer** — mediawiki.org
|
||||
- Help:Qualifiers; Wikidata SPARQL query service + Query Help; SPARQL tutorial — wikidata.org
|
||||
- Wikidata Query Service / User Manual — mediawiki.org; Wikitech (Blazegraph, updater)
|
||||
- "The wikibase model" — Vanderbilt Libraries Digital Lab (heardlibrary.github.io)
|
||||
- RaiseWikibase — Wikibase Data Model functions (ub-mannheim.github.io)
|
||||
- WShEx / EntitySchemas (ShEx) — arxiv.org/abs/2208.02697, ceur-ws.org Vol-3262
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UCs **UC-73–UC-75** carry the marker **⬡** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-34, UC-58, UC-52, UC-24. Architecture cross-refs:
|
||||
SHARD-WP-0002 T12, T16, native-query tiering, provenance model, UC-63 derived index.
|
||||
49
research/260614-wikijs-deep-dive/README.md
Normal file
49
research/260614-wikijs-deep-dive/README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# 260614 — Wiki.js deep dive (the storage-module engine that already speaks Git + Markdown)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Wiki.js** — the modern Node.js server wiki — read through
|
||||
shard-wiki's lens. Wiki.js is the **closest existing engine to shard-wiki's own design**:
|
||||
it is DB-canonical (Postgres/MySQL/SQLite) **but abstracts persistence behind pluggable
|
||||
"storage modules"** — Git, local filesystem, S3, Azure Blob — each acting as **backup or
|
||||
source of truth**, with the **Git module bidirectionally syncing clean `.md` files (+ YAML
|
||||
frontmatter)** to a repo. That storage-module interface is, in effect, a shipped version
|
||||
of shard-wiki's adapter contract — the second concrete prior art after **Foswiki::Store**,
|
||||
and the closer one (medium = Markdown in Git).
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — Node.js/Vue; **GraphQL API** over all resources; DB-canonical;
|
||||
Markdown (primary) + HTML + AsciiDoc; path-based hierarchy; modular
|
||||
(storage/auth/search/render/editor)
|
||||
- **Storage modules** — provider connector (Git/FS/S3/Azure) with properties + create/
|
||||
update/delete/sync methods, **backup-or-source-of-truth**, `injectMetadata()` YAML
|
||||
frontmatter; **Git module commits every change → version-controlled clean Markdown**,
|
||||
bidirectional
|
||||
- **GraphQL** — typed, **introspectable** (capability/schema discovery), **selective-field**
|
||||
projection
|
||||
- **Auth modules** (delegated authn) + **path-based rule ACL**
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, storage modules (≈ adapter contract), Git mirror as attach+write surface, GraphQL, access control, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-68
|
||||
attach an engine-maintained bidirectional Git mirror of clean Markdown, incl. write-by-
|
||||
commit; UC-69 attach via a typed, introspectable API — GraphQL — for schema discovery +
|
||||
selective projection); UC-06/36/38/40/42/57 enriched. Logged for `SHARD-WP-0002`
|
||||
(T11/T14): the **storage-module abstraction as a second adapter-contract prior art** (with
|
||||
Foswiki::Store), the **engine-maintained Git mirror** as preferred attach+write surface,
|
||||
and **GraphQL introspection** for capability discovery + selective projection.
|
||||
|
||||
**Key takeaway recorded:** Wiki.js is the most shard-wiki-shaped engine yet — its
|
||||
storage-module interface is near-isomorphic to the adapter contract, and its
|
||||
Markdown-in-Git mirror is the ideal file-store attach (clean MD, git history,
|
||||
write-by-commit). **Boundary:** one server-engine candidate shard, DB-canonical with a
|
||||
maintained Git mirror; don't double-sync; honor its access rules; not the federation layer.
|
||||
</content>
|
||||
249
research/260614-wikijs-deep-dive/findings.md
Normal file
249
research/260614-wikijs-deep-dive/findings.md
Normal file
@@ -0,0 +1,249 @@
|
||||
# Findings — Wiki.js: the storage-module engine that already speaks Git + Markdown
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — a server wiki engine; a *candidate shard* and
|
||||
the closest existing engine to shard-wiki's own design (DB-canonical, but with a
|
||||
**pluggable storage-module abstraction** that bidirectionally syncs clean Markdown to
|
||||
**Git**/FS/S3/Azure)
|
||||
Lens: shard-wiki — storage-module-as-adapter-contract prior art, the engine-maintained
|
||||
Git mirror as an attach + write surface, GraphQL as a typed/introspectable API, and
|
||||
path-based access rules
|
||||
|
||||
> Why Wiki.js is special in this set. Every prior engine/tool stored content in *one*
|
||||
> substrate and exposed *one or two* surfaces. Wiki.js is DB-canonical (Postgres/MySQL/
|
||||
> SQLite) **but abstracts persistence behind pluggable "storage modules"** — Git, local
|
||||
> filesystem, S3, Azure Blob — each of which can act as **backup *or* source of truth**,
|
||||
> and the **Git module bidirectionally syncs clean `.md` files (with YAML frontmatter)**
|
||||
> to a repo. In other words, Wiki.js already does, internally, much of what shard-wiki's
|
||||
> adapter contract is meant to do: a **versioned, capability-bearing, multi-provider
|
||||
> persistence interface over Markdown**. It is the second concrete prior art for the
|
||||
> contract after **Foswiki::Store** — and arguably the closer one, because its content
|
||||
> is **Markdown in Git**, exactly shard-wiki's native shape.
|
||||
|
||||
Contrast set: the engine dives (XWiki DB/components, TWiki/Foswiki file+RCS) and the
|
||||
modern tools. Wiki.js = **modern server engine, DB-canonical, Git-storage-of-Markdown,
|
||||
GraphQL API, pluggable modules**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture
|
||||
|
||||
- **Stack:** Node.js backend, Vue front end; **everything is accessible via a GraphQL
|
||||
API** (`/graphql`, with a Playground). The DB is **Postgres** (primary; MySQL/SQLite
|
||||
supported), read-replica-capable.
|
||||
- **Content:** **Markdown (primary)**, plus **HTML** and **AsciiDoc**, via pluggable
|
||||
**editors** (Markdown, WYSIWYG/visual, raw HTML). Pages live on **paths** (a path-based
|
||||
hierarchy) with tags.
|
||||
- **Modular architecture:** typed pluggable **modules** — **storage**, **auth**,
|
||||
**search**, **rendering**, **editor**. The contract-relevant ones are *storage* (§2)
|
||||
and *auth* (§4).
|
||||
- **3.0:** a large rewrite is in development; storage remains a headline subsystem.
|
||||
|
||||
---
|
||||
|
||||
## 2. Storage modules — the headline (≈ shard-wiki's adapter contract)
|
||||
|
||||
A **storage module** "connects Wiki.js with a local or remote storage provider, to act as
|
||||
**backup or source of truth** for content. It consists of **properties** (user-set) and
|
||||
**methods called on events** — create, update, delete content, and sync." Providers
|
||||
include **Git, local filesystem, S3, Azure Blob**.
|
||||
|
||||
The **Git module** is the most-used: **every page change commits to a Git repository**,
|
||||
turning the wiki into **version-controlled `.md` files**. Content is written as **clean
|
||||
Markdown**, and `injectMetadata()` prepends **YAML frontmatter** (for Markdown) or HTML
|
||||
comments (for HTML) carrying page metadata. The sync is **bidirectional** with the remote
|
||||
repo, and access to the synced files honors Wiki.js **access rules** (§4).
|
||||
|
||||
Why this matters to shard-wiki — three distinct payoffs:
|
||||
|
||||
1. **Prior art for the adapter contract.** Wiki.js's storage-module interface is a
|
||||
shipped instance of what `SHARD-WP-0002` T11 specifies: a *versioned, capability-
|
||||
bearing, multi-provider* persistence abstraction, with a backup-vs-source-of-truth
|
||||
choice and lifecycle methods (create/update/delete/sync). It joins **Foswiki::Store**
|
||||
as concrete prior art — and is closer, because its medium is **Markdown in Git**.
|
||||
2. **The ideal file-store attach.** A Wiki.js instance with the Git module **is a git
|
||||
repo of clean Markdown + YAML frontmatter**, maintained by the engine. shard-wiki can
|
||||
attach that repo **directly** (UC-40) — better than reading the DB — and get **git
|
||||
history for free** (UC-36, *adopt* not supplement, via the mirror).
|
||||
3. **A write path via Git, no API needed.** Because the Git sync is **bidirectional**,
|
||||
shard-wiki can **write-through by committing Markdown** to the repo; Wiki.js **ingests
|
||||
git→DB**. This is overlay/patch application as a *git commit*, not an API call (UC-68).
|
||||
Caution: Wiki.js owns the DB↔git sync — **don't double-sync**, and coordinate which
|
||||
side is **source of truth** (configurable).
|
||||
|
||||
---
|
||||
|
||||
## 3. GraphQL — a typed, introspectable external API
|
||||
|
||||
Unlike the REST/local-REST tools (Joplin, Trilium, Notion), Wiki.js exposes a **GraphQL
|
||||
API** over all resources. Two shard-wiki-relevant properties:
|
||||
|
||||
- **Introspection = capability/schema discovery.** A GraphQL schema is self-describing;
|
||||
an adapter can introspect the content/page schema rather than hard-coding it — feeding
|
||||
T11's *capability discovery*.
|
||||
- **Selective-field projection.** GraphQL fetches **exactly the fields a projection
|
||||
needs** (page body vs. metadata vs. tags), reducing over-fetch — relevant to projection
|
||||
efficiency and the operational envelope. (UC-69; an external-API sub-mode beside
|
||||
Notion's REST, UC-57.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Auth modules + path-based access rules
|
||||
|
||||
- **Auth modules** (local, LDAP, OAuth, SAML, …) — **authentication is delegated to
|
||||
pluggable providers**, exactly shard-wiki's **authn-delegated** stance
|
||||
([[shard-wiki-auth-in-core-decision]]).
|
||||
- **Access rules** are **path-based / rule-based ACL**: allow/deny access to all or
|
||||
specific sections of the wiki per group; the Git-synced files honor the same rules. For
|
||||
shard-wiki this is a richer ACL than TWiki's per-topic flags (UC-06) — **rule-based by
|
||||
path pattern** — and informs how a projection should **honor/surface a shard's
|
||||
restricted regions** (union without erasure: show that a region is restricted, don't
|
||||
silently drop or expose it).
|
||||
|
||||
---
|
||||
|
||||
## 5. Wiki.js as a shard — capability profile
|
||||
|
||||
| Capability | Wiki.js | Notes for the adapter contract |
|
||||
|------------|---------|--------------------------------|
|
||||
| Read | yes (Git mirror / GraphQL / DB) | **Git mirror = clean Markdown** is the best surface |
|
||||
| Write | yes (GraphQL **or git commit**) | bidirectional Git ingest = write-by-commit (UC-68) |
|
||||
| Write granularity | per-page | — |
|
||||
| Identity / addressing | path (+ page id) | path-based; YAML frontmatter carries metadata/id |
|
||||
| Structure | YAML frontmatter (in-file via Git module) | git-diffable in-text structure (good) |
|
||||
| History | **git-native via Git module** (else DB versions) | adopt the mirror's git history (UC-36) |
|
||||
| Native query | GraphQL; pluggable search (DB/Elastic/Algolia) | delegate search; GraphQL selective fetch (UC-69) |
|
||||
| Translation | Markdown primary; HTML/AsciiDoc too | mostly native; multi-format = light translation (UC-42) |
|
||||
| Attach modes | **file-store (engine-maintained Git mirror)** / external-API (GraphQL) | clean MD in git is the ideal file-store (UC-40, UC-68) |
|
||||
| Access control | **path-based rule ACL**; delegated auth modules | authn-delegated; honor restricted regions (UC-06) |
|
||||
| Storage abstraction | **pluggable storage modules (Git/FS/S3/Azure)** | **adapter-contract prior art** (T11) |
|
||||
| Content types | pages + media/assets | assets via storage modules |
|
||||
|
||||
Verdict: **the most shard-wiki-shaped engine yet.** Best attached via its **Git storage
|
||||
mirror** (clean Markdown + frontmatter + git history, with write-by-commit), or via
|
||||
**GraphQL** for live/selective access. Its standout gift is **architectural**: a shipped
|
||||
storage-module abstraction that is near-isomorphic to shard-wiki's adapter contract.
|
||||
|
||||
---
|
||||
|
||||
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 6.1 Reinforcements
|
||||
|
||||
- **Storage modules = the adapter contract, shipped.** A versioned, capability-bearing,
|
||||
multi-provider persistence interface with backup-vs-source-of-truth — exactly T11's
|
||||
shape, in production (with Foswiki::Store as the other prior art).
|
||||
- **Markdown-in-Git is the native shape.** Wiki.js validates that a serious engine can
|
||||
keep content as **clean `.md` + frontmatter in git** — shard-wiki's preferred medium —
|
||||
while still offering a rich app over a DB.
|
||||
- **authn-delegated** (auth modules) and **git coordination** (Git storage) are both core
|
||||
shard-wiki stances, here shipped.
|
||||
|
||||
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **DB is Wiki.js's canonical store; the Git repo is a maintained mirror.** Treat the
|
||||
mirror as the attach surface, but respect that **Wiki.js runs the DB↔git sync** —
|
||||
don't double-sync; coordinate source-of-truth (UC-68).
|
||||
2. **Bidirectional write needs care.** Committing Markdown is a real write path, but
|
||||
races with the engine's own sync; apply overlays as commits the engine can ingest
|
||||
cleanly, never out-of-band DB edits (overlay before mutation, no silent mutation).
|
||||
3. **One Wiki.js instance = one shard**, not the federation layer; its access rules are
|
||||
the shard's, to be honored/surfaced, not re-implemented.
|
||||
4. **HTML/AsciiDoc pages** need light translation; Markdown pages are native.
|
||||
|
||||
### 6.3 What Wiki.js teaches that shard-wiki should keep
|
||||
|
||||
- **Model the contract on a real storage-module interface** (Wiki.js + Foswiki::Store):
|
||||
properties + lifecycle methods (create/update/delete/sync) + a **backup-vs-source-of-
|
||||
truth** flag + a provider list. (T11.)
|
||||
- **Prefer the engine-maintained Git mirror** as the attach surface for DB-canonical
|
||||
engines that offer one — clean Markdown, git history, and write-by-commit (UC-68).
|
||||
- **Use GraphQL introspection for capability discovery and selective projection** where a
|
||||
shard offers it (UC-69).
|
||||
|
||||
---
|
||||
|
||||
## 7. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-67**. New UCs **UC-68, UC-69** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach an engine-maintained bidirectional Git mirror of clean Markdown** (Wiki.js Git storage module): attach the repo as a file-store shard (clean MD + YAML frontmatter + git history) and optionally **write-through by committing Markdown the engine ingests**; coordinate source-of-truth, don't double-sync | **UC-68 (new)** |
|
||||
| **Attach via a typed, introspectable API (GraphQL)**: discover the shard's schema by introspection (capability discovery) and fetch only the fields a projection needs (selective projection) | **UC-69 (new)** |
|
||||
| Pluggable **storage modules** (Git/FS/S3/Azure) = adapter-contract prior art; modular auth/search/render/editor | **enriches UC-38** |
|
||||
| Git storage = **clean Markdown in git**, the ideal engine-maintained file-store attach | **enriches UC-40** |
|
||||
| Git history natively via the storage module (**adopt** via mirror) | **enriches UC-36** |
|
||||
| GraphQL external-API variant (typed/introspectable/selective) | **enriches UC-57** |
|
||||
| Multi-format (Markdown primary; HTML/AsciiDoc) | **enriches UC-42** |
|
||||
| Path-based rule ACL; delegated auth modules | **enriches UC-06** (links authz decision) |
|
||||
| Engine-maintained clean git shard | links UC-02 |
|
||||
|
||||
---
|
||||
|
||||
## 8. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Second adapter-contract prior art (T11).** Wiki.js's **storage-module interface**
|
||||
(properties + create/update/delete/sync methods + backup-vs-source-of-truth + multi-
|
||||
provider Git/FS/S3/Azure) is, with **Foswiki::Store**, the model to base the contract
|
||||
on — and the closer one (medium = Markdown in Git). Cite both in T11.
|
||||
- **Engine-maintained Git mirror (T14).** Add to the attachment taxonomy: for a
|
||||
DB-canonical engine that offers a bidirectional Git-of-Markdown mirror, the **mirror is
|
||||
the preferred attach + write surface** (clean MD, git history, write-by-commit). A
|
||||
cleaner sibling of Joplin's proprietary interchange mirror (UC-60) and the file-store
|
||||
native-store kind (UC-40).
|
||||
- **GraphQL → capability discovery + selective projection (T11/T14).** Typed/
|
||||
introspectable APIs give schema discovery (capability negotiation) and reduce over-fetch
|
||||
— note alongside Notion's REST external-API mode.
|
||||
- **authn-delegated + path-based ACL** reinforce the settled authz decision and inform how
|
||||
projections honor/surface restricted regions (union without erasure).
|
||||
|
||||
---
|
||||
|
||||
## 9. Open questions (for spec / workplans)
|
||||
|
||||
1. For an engine with a bidirectional Git mirror (UC-68), is the **mirror or the DB the
|
||||
source of truth** from shard-wiki's view, and how do we avoid races with the engine's
|
||||
own sync (write-by-commit cadence, locking)?
|
||||
2. When both a **Git mirror and GraphQL** are available, which does a binding prefer —
|
||||
mirror for content/history, GraphQL for live/selective metadata? Can one binding use
|
||||
both?
|
||||
3. How does shard-wiki **honor/surface path-based access rules** (UC-06) in a projection
|
||||
without re-implementing the shard's ACL engine?
|
||||
4. Should the adapter contract's **storage-module shape** be standardized directly on
|
||||
Wiki.js + Foswiki::Store, or generalized further?
|
||||
|
||||
---
|
||||
|
||||
## 10. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Wiki.js — GraphQL API docs (https://docs.requarks.io/dev/api) | Everything via GraphQL; `/graphql` Playground; access/modify all resources |
|
||||
| Wiki.js — Git storage (https://docs.requarks.io/storage/git) | Per-change commit to git; clean `.md`; bidirectional sync with remote repo |
|
||||
| Wiki.js — Storage overview + dev/storage.md (https://docs.requarks.io/storage ; https://github.com/requarks/wiki-docs/blob/master/dev/storage.md) | Storage module = provider connector; backup-or-source-of-truth; properties + create/update/delete/sync methods; `injectMetadata()` YAML frontmatter; providers Git/FS/S3/Azure |
|
||||
| Wiki.js 3.0 storage preview (https://beta.js.wiki/blog/2021-wiki-js-3-feature-preview-storage) | Storage subsystem direction (3.0) |
|
||||
| Wikipedia — Wiki.js (https://en.wikipedia.org/wiki/Wiki.js) | Node.js/Vue, DB options, formats, modular architecture |
|
||||
| BrightCoding / Grokipedia overviews | Auth providers, access rules, editors, read replicas |
|
||||
|
||||
Cross-references: `research/260613-foswiki-deep-dive/findings.md` (Foswiki::Store, the
|
||||
other storage-interface prior art), `research/260614-joplin-deep-dive/findings.md`
|
||||
(interchange/sync-mirror contrast), `research/260614-notion-deep-dive/findings.md`
|
||||
(external-API/REST contrast), `research/260614-shard-spectrum-synthesis/findings.md`
|
||||
(attachment-mode spectrum), `spec/UseCaseCatalog.md` (UC-02, UC-06, UC-36, UC-38, UC-40,
|
||||
UC-42, UC-57), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
|
||||
|
||||
---
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
- New UCs: **UC-68, UC-69** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-06, UC-36, UC-38, UC-40, UC-42, UC-57** (links UC-02, authz decision).
|
||||
- Architecture (no UC): **storage-module abstraction = 2nd adapter-contract prior art**
|
||||
(with Foswiki::Store); engine-maintained Git mirror as attach+write surface; GraphQL
|
||||
introspection → capability discovery + selective projection; path-based ACL + authn-
|
||||
delegated → `SHARD-WP-0002` (T11, T14).
|
||||
- Boundary recorded: Wiki.js is **one server-engine candidate shard**, DB-canonical with a
|
||||
**bidirectional Git-of-Markdown mirror** that is the preferred attach surface; not a
|
||||
substrate and not the federation layer (don't double-sync; honor its access rules).
|
||||
</content>
|
||||
47
research/260614-xanadu-deep-dive/README.md
Normal file
47
research/260614-xanadu-deep-dive/README.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# 260614 — Project Xanadu deep dive (the docuverse, the EDL, reference-not-copy)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Project Xanadu** (Ted Nelson) read through shard-wiki's lens.
|
||||
Unlike the engine dives (`xwiki`, `twiki`, `foswiki`), Xanadu is **not a candidate
|
||||
shard backend** — it never shipped at scale and there is nothing to attach. It is
|
||||
studied as the **deepest conceptual ancestor** of shard-wiki's own model: several
|
||||
shard-wiki primitives turn out to be Xanadu mechanisms under different names.
|
||||
|
||||
The distinctive material:
|
||||
- the **EDL / xanadoc** — a document that contains *no content*, only an ordered list
|
||||
of **spans** (content references) plus **xanalinks** (separate link tables); the
|
||||
client assembles the page by reference → this is shard-wiki **projection** + **union**
|
||||
- the storage substrate — **tumblers** (stable fine-grained addresses), **istream**
|
||||
(invariant content pool), **enfilades / spanfilade** (virtual↔content mapping,
|
||||
version comparison by span-set intersection)
|
||||
- **content-identity, bidirectional transclusion** — content "knowably in more than
|
||||
one place" that remembers all its appearances → stronger than UC-32
|
||||
- **transcopyright / micropayment** — a baked-in rights policy shard-wiki must keep
|
||||
*configurable*, not inherit
|
||||
|
||||
Goes deliberately **underneath** the surface pattern table in
|
||||
`research/260608-federation-concepts/findings.md` §3.3, which it extends rather than
|
||||
repeats.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | EDL/xanadoc, addressing substrate, transclusion, versioning, rights, INTENT mapping (reinforcements + design-bug divergences), UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-44 compose-by-reference, UC-45 reverse transclusion, UC-46 content-identity
|
||||
equivalence); UC-24/27/29/32 enriched. Span-addressing as an adapter capability,
|
||||
content identity, composition manifests, and reuse-terms metadata logged as
|
||||
architecture for `workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
|
||||
Key boundary recorded: shard-wiki inherits Xanadu's **reference-not-copy** mechanisms
|
||||
but **rejects** the single-global-docuverse premise, the single-canonical-instance
|
||||
model, and the baked-in economic policy — those would violate shard sovereignty,
|
||||
parallel-version support (UC-27), and mechanism-over-policy.
|
||||
</content>
|
||||
322
research/260614-xanadu-deep-dive/findings.md
Normal file
322
research/260614-xanadu-deep-dive/findings.md
Normal file
@@ -0,0 +1,322 @@
|
||||
# Findings — Project Xanadu: the docuverse, the EDL, and reference-not-copy
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **conceptual / architectural prior art** (not a deployable engine, not a
|
||||
candidate shard backend)
|
||||
Lens: shard-wiki orchestration — projection, overlay, transclusion, provenance,
|
||||
addressing, coordination journal
|
||||
|
||||
> Reading guide. Every previous deep dive (`xwiki`, `twiki`, `foswiki`) studied a
|
||||
> *shippable engine* as a candidate **shard**. Xanadu is different: it is the
|
||||
> **deepest conceptual ancestor** of shard-wiki's own model and is *not* a backend
|
||||
> we would ever attach. It is studied here for its **mechanisms** —
|
||||
> reference-not-copy documents, separated link tables, content-identity transclusion,
|
||||
> stable fine-grained addressing — several of which shard-wiki already reinvents
|
||||
> under different names (projection, overlay, union BackLinks, coordination journal).
|
||||
> The job of this file is to (a) name what shard-wiki inherits, (b) flag what would
|
||||
> be a **design bug** to inherit, and (c) extract use cases that are genuinely new
|
||||
> versus the federation track already in the catalog.
|
||||
|
||||
Pairs with — and deliberately extends, does not repeat —
|
||||
`research/260608-federation-concepts/findings.md` §3.3, which treats Xanadu only as a
|
||||
six-row pattern table and (correctly) labels it "speculative design / pattern
|
||||
language, not deployable federation." This dive goes underneath those patterns into
|
||||
the actual data architecture (EDL, istream, enfilades, tumblers) because *that* is
|
||||
where the resonance with shard-wiki's storage-neutral page model lives.
|
||||
|
||||
---
|
||||
|
||||
## 1. Origin and status
|
||||
|
||||
Project Xanadu (Ted Nelson, 1960– ; the term *hypertext* and *transclusion* are his)
|
||||
is the original electronic-literature system: a global **docuverse** of permanent,
|
||||
addressable documents joined by visible, two-way links. It famously never shipped at
|
||||
scale; partial artifacts exist — **Udanax Green / Udanax Gold** (source released 1999
|
||||
to establish prior art against patents), **XanaduSpace**, and **OpenXanadu** (2014, a
|
||||
small working browser demo built on the EDL format). Treat all of these as
|
||||
*reference designs*, not running software to integrate.
|
||||
|
||||
Why it matters to shard-wiki: Nelson's critique of the Web — **"one-way,
|
||||
ever-breaking links and no management of version or contents"** — is precisely the
|
||||
gap shard-wiki addresses for wikis. shard-wiki is, in effect, a *pragmatic, federated,
|
||||
Git-backed, heterogeneous-backend* descendant of the Xanadu idea, scoped to
|
||||
Markdown wiki pages and explicitly **refusing** Xanadu's one-global-store premise.
|
||||
|
||||
---
|
||||
|
||||
## 2. The headline insight — a document is a manifest of spans, not content
|
||||
|
||||
The single most shard-wiki-relevant Xanadu artifact is the **EDL (Edit Decision
|
||||
List)**, also called a **xanadoc**:
|
||||
|
||||
- A xanadoc is a *connected document that contains no original content*. It is a file
|
||||
listing (a) **spans** — "portions of content to bring in" addressed in remote
|
||||
sources — and (b) **xanalinks** — tables saying what to connect to what.
|
||||
- The client reads the EDL, **fetches each span from its source**, and **assembles**
|
||||
the document locally per the xanalinks. Content is referenced, never embedded.
|
||||
- Structure is strictly separated: **spans first** (content references), **xanalinks
|
||||
after** (relationships). Content and link structure live apart.
|
||||
|
||||
This is *the same move* shard-wiki makes:
|
||||
|
||||
| Xanadu (xanadoc / EDL) | shard-wiki concept |
|
||||
|------------------------|--------------------|
|
||||
| Document = ordered list of span references, no embedded content | **Projection** — lazy, cache-like view assembled from shard storage, not a copy |
|
||||
| Spans pulled from multiple remote sources at view time | **Union of pages** across heterogeneous shards |
|
||||
| Xanalinks stored separately from content | **Overlay** — non-destructive edits/annotations as separate objects, "overlay before mutation" |
|
||||
| "All media should be permanized and addressable" | **Coordination journal** + Git content-addressable storage |
|
||||
|
||||
shard-wiki's INTENT lines — *"prefer lazy projection over eager copying"* and *"union
|
||||
without erasure"* — are the xanadoc principle restated for wikis.
|
||||
|
||||
---
|
||||
|
||||
## 3. Addressing and the storage substrate
|
||||
|
||||
Xanadu's machinery underneath the EDL:
|
||||
|
||||
- **Tumblers** (Miller / Gregory): a transfinite-number addressing scheme where one
|
||||
address simultaneously encodes **machine, author, document version, byte span, and
|
||||
links**. A single tumbler is a stable, fine-grained, hierarchical pointer into the
|
||||
whole docuverse.
|
||||
- **istream (invariant stream):** a growing pool of *shareable content pieces*.
|
||||
Documents do not store text; they reference istream content by virtual address. A
|
||||
family of **enfilades** (tree structures using *dsps* = relative displacements and
|
||||
*wids* = ranges) maps **virtual addresses ↔ istream addresses** bidirectionally.
|
||||
- **Spanfilade:** indexes which istream spans each document uses. Other filades:
|
||||
**granfilade** (storage across disks/network), **POOMfilade** (permutation-of-order
|
||||
matrix mapping document position → istream location).
|
||||
- Implicit substrate: content pieces are **invariant** (append-only, never rewritten);
|
||||
versions are new arrangements of shared, permanent pieces. Nothing is deleted, so
|
||||
references never break.
|
||||
|
||||
shard-wiki mapping:
|
||||
|
||||
- **istream + invariance ≈ Git's content-addressable, immutable blob store**, and the
|
||||
**coordination journal** as the append-only record. shard-wiki gets "references
|
||||
never break" *for Git-native shards for free*; for non-Git shards it must
|
||||
approximate via the journal/projection cache.
|
||||
- **Tumbler ≈ the still-open question of a portable, stable, fine-grained span
|
||||
address** that survives projection, overlay, and versioning across heterogeneous
|
||||
backends. shard-wiki has no such addressing scheme yet; this is the hardest part
|
||||
Xanadu solved on paper and the part that never shipped. (See §10, §11.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Links and transclusion — separate, two-way, content-identity
|
||||
|
||||
- **Links are first-class objects stored apart from content** (the xanalinks tables),
|
||||
and are **visible and followable from all endpoints** (rule 7) — i.e. inherently
|
||||
**bidirectional**. Backlinks are not derived after the fact; they are the same
|
||||
object seen from the other end.
|
||||
- **Transclusion** = Nelson's "the same content *knowably* in more than one place."
|
||||
The content piece **remembers its identity and can trace back to all its
|
||||
appearances**. This is the crucial delta from every weak modern form (server-side
|
||||
includes, HTML embeds, Roam block-refs): those *copy or cache*; Xanadu keeps one
|
||||
permanent instance that is **aware of where it is reused**.
|
||||
|
||||
shard-wiki mapping:
|
||||
|
||||
- **Separate link tables ≈ overlays and union BackLinks as first-class.** An overlay
|
||||
is shard-wiki's xanalink: a non-destructive object spanning content it does not own.
|
||||
- **Content-identity, bidirectional transclusion** is *stronger* than UC-32 as
|
||||
currently written (which is path/span fetch with freshness). Xanadu says the
|
||||
*content itself* knows its appearances → enables **reverse transclusion** ("where is
|
||||
this paragraph used across all shards?") and **content-identity equivalence** ("these
|
||||
two pages in different shards are versions of the same content"). These are new
|
||||
capabilities, surfaced as UC-45 and UC-46 below.
|
||||
|
||||
---
|
||||
|
||||
## 5. Versioning and comparison — by span-set intersection
|
||||
|
||||
Because a document is a set of spans over an invariant pool, **comparing two documents
|
||||
or two versions = intersecting their span-sets** (the spanfilade operation). Shared
|
||||
spans are literally shared subtrees; differences fall out of the set arithmetic. The
|
||||
same operation surfaces links between documents.
|
||||
|
||||
shard-wiki mapping: this is a **content-identity diff/merge** model. shard-wiki's
|
||||
diff/merge capability (an adapter capability in the contract) is today implicitly
|
||||
path/title- and text-based. Xanadu shows a path-independent alternative: detect that
|
||||
page A in shard X and page B in shard Y are *the same or derived content* by span
|
||||
overlap, **without relying on matching titles or paths**. This directly serves UC-27
|
||||
(view parallel versions of equivalent pages) by giving an *equivalence-detection
|
||||
mechanism* rather than assuming naming conventions align across sovereign shards.
|
||||
|
||||
---
|
||||
|
||||
## 6. Rights and economics — transcopyright, micropayment, implicit permission
|
||||
|
||||
Three of the 17 rules are an economic/rights layer:
|
||||
|
||||
- Rule 8: **publication grants implicit permission to link/transclude.**
|
||||
- Rule 9: **granular royalty / micropayment** on any accessed portion.
|
||||
- **Transcopyright:** pre-granted permission for virtual republication *by reference*,
|
||||
with the attribution chain to origin preserved automatically because the content is
|
||||
transcluded, not copied.
|
||||
|
||||
shard-wiki mapping — handle with the **mechanism-over-policy** rule:
|
||||
|
||||
- The *mechanism* shard-wiki should provide: a reference (overlay/transclusion) that
|
||||
**carries provenance and reuse terms with it**, so attribution and permission travel
|
||||
with the span. This strengthens UC-29 (remix with portable attribution).
|
||||
- The *policy* (whether reuse requires permission, whether anything is metered) must
|
||||
**not be hard-coded**. Xanadu baked one global economic policy into the substrate;
|
||||
shard-wiki keeps editorial/economic policy configurable. Baking in a payments or
|
||||
permission model would be a design-bug per INTENT.
|
||||
- This intersects the **settled authz-in-core / authn-delegated decision**
|
||||
([[shard-wiki-auth-in-core-decision]]): "publication grants implicit linking
|
||||
permission" is exactly a *policy* that an information space might or might not adopt
|
||||
on its L0→L4 ladder. shard-wiki must be able to *represent* the permission on a
|
||||
transclusion; it must not *assume* Xanadu's answer.
|
||||
|
||||
---
|
||||
|
||||
## 7. Xanadu through shard-wiki's lens — it is architecture, not a shard
|
||||
|
||||
Unlike the engine dives, there is **no capability profile to fill in** — Xanadu is not
|
||||
a backend you attach. The useful framing is the inverse: *which shard-wiki primitives
|
||||
are Xanadu mechanisms in disguise?*
|
||||
|
||||
| shard-wiki primitive | Xanadu mechanism | Inheritance verdict |
|
||||
|----------------------|------------------|---------------------|
|
||||
| Projection (lazy view, no copy) | xanadoc / EDL span assembly | **Inherit** — same idea, scope to wiki pages |
|
||||
| Overlay (non-destructive edit) | xanalinks as separate objects | **Inherit** — overlays are xanalinks |
|
||||
| Union BackLinks | two-way links visible from all endpoints | **Inherit** — generalize to sub-page spans |
|
||||
| Transclusion (UC-32) | content-identity, content-aware reuse | **Inherit, strengthen** — add reverse + equivalence |
|
||||
| Coordination journal | invariant istream / permanent storage | **Inherit** (Git gives this for Git-native shards) |
|
||||
| Stable span address | tumblers | **Aspire** — hard, partly unshippable; degrade gracefully |
|
||||
| Editorial/economic policy | transcopyright + micropayment | **Reject as substrate** — keep configurable |
|
||||
| One global docuverse | the docuverse premise | **Reject** — see §8.2 |
|
||||
|
||||
---
|
||||
|
||||
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 8.1 Reinforcements
|
||||
|
||||
- **Reference-not-copy** is the core of both. Xanadu validates shard-wiki's
|
||||
"lazy projection over eager copying" and "union without erasure" as a coherent,
|
||||
decades-deep design lineage, not an ad-hoc preference.
|
||||
- **Provenance is structural, not decorative.** In Xanadu, knowing where content came
|
||||
from and where it is reused is *built into the storage model*. shard-wiki's "never
|
||||
hide authorship, conflicts, freshness, backend limits" is the same commitment.
|
||||
- **Links/edits as separate objects** validates overlays-before-mutation and
|
||||
first-class BackLinks.
|
||||
|
||||
### 8.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **No single docuverse / no universal address space.** Xanadu requires one global
|
||||
permanent store, universal addressing, and its own published client–server
|
||||
protocol (rules 12, 17). shard-wiki's whole reason to exist is the opposite:
|
||||
**federate heterogeneous, sovereign backends** (Git repos, Gitea, Obsidian,
|
||||
WebDAV, Coulomb), each keeping its own storage, history, identity, and limits.
|
||||
Adopting Xanadu's universality would violate **shard sovereignty** and **graceful
|
||||
degradation**. shard-wiki's addressing must tolerate backends that *cannot* offer
|
||||
stable span addresses, treating fine-grained addressing as an **adapter
|
||||
capability**, not a precondition.
|
||||
2. **One canonical content instance vs. parallel divergent versions.** Xanadu
|
||||
transclusion centers a *single source of truth*. shard-wiki must support
|
||||
**equivalent-but-divergent** pages across shards with conflicts *visible* (UC-27),
|
||||
not collapsed into one instance. Content-identity is a *detection* tool here, not a
|
||||
mandate to unify.
|
||||
3. **Baked-in economic/permission policy.** See §6 — reject as substrate, keep as
|
||||
configurable policy.
|
||||
4. **Permanence as a hard requirement.** Xanadu forbids deletion globally. shard-wiki
|
||||
cannot impose that on sovereign shards; permanence holds for the **coordination
|
||||
journal** and Git-native shards, and is *approximated* (cache/projection/snapshot)
|
||||
for backends that delete. This is graceful degradation, not a weaker promise.
|
||||
|
||||
### 8.3 What Xanadu teaches that shard-wiki should not lose
|
||||
|
||||
- A page view can legitimately be a **composition manifest** (a list of references),
|
||||
not a file. Build the page model so a page *can* be authored as references into
|
||||
other shards (UC-44), even if the common case is a plain Markdown file.
|
||||
- **Content identity** (not path/title) is the durable basis for transclusion,
|
||||
equivalence, and reverse lookup across sovereign shards (UC-45, UC-46). Lean on
|
||||
Git blob hashes where available; define a content-fingerprint fallback elsewhere.
|
||||
- Getting **stable fine-grained addressing** right is the hard, valuable, historically
|
||||
*unshippable* part. Scope it as an adapter capability with explicit degradation
|
||||
rather than promising tumbler-grade universality.
|
||||
|
||||
---
|
||||
|
||||
## 9. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-43**. New UCs **UC-44–UC-46** added; several existing UCs
|
||||
enriched (no new scenario, stronger mechanism/notes).
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Compose-by-reference page** — author a page whose body is an ordered list of spans pulled from multiple shards, stored as a manifest (xanadoc/EDL), not a copy | **UC-44 (new)** |
|
||||
| **Reverse transclusion** — find every page/shard where a given span (paragraph/section) appears or is transcluded | **UC-45 (new)** |
|
||||
| **Content-identity equivalence** — detect that two pages in different shards are the same or derived content via span/content overlap, without matching titles or paths | **UC-46 (new)** |
|
||||
| Content-identity, content-aware (bidirectional) transclusion | **enriches UC-32** |
|
||||
| Equivalence detection mechanism for parallel versions | **enriches UC-27** |
|
||||
| Reference carries provenance + reuse terms (transcopyright as representable policy) | **enriches UC-29**; links [[shard-wiki-auth-in-core-decision]] |
|
||||
| Provenance: content remembers its appearances | **enriches UC-24** |
|
||||
|
||||
---
|
||||
|
||||
## 10. Adapter-contract / architecture notes for SHARD-WP-0002
|
||||
|
||||
Logged as architecture (no UC):
|
||||
|
||||
- **Fine-grained span addressing is an adapter capability**, not a core assumption.
|
||||
The shard adapter contract should model whether a shard can mint a *stable address
|
||||
for a sub-page span* that survives edits/versions (tumbler-grade), down through
|
||||
whole-page-only, down to path-only. Transclusion/overlay capabilities depend on it.
|
||||
- **Content identity** should be a contract-level concept: a shard advertises how it
|
||||
fingerprints content (Git blob hash, normalized-text hash, none). UC-45/UC-46 and
|
||||
cross-shard diff/merge consume it.
|
||||
- **Composition manifests** (UC-44) imply the wiki page model must permit a page whose
|
||||
canonical form is a reference list. This is an INTENT-level page-model decision —
|
||||
flag for the page-model spec, not just the adapter contract.
|
||||
- Reuse-terms metadata on a reference (UC-29 / §6) is **policy data the core carries
|
||||
but does not interpret** — consistent with mechanism-over-policy and the L0→L4
|
||||
authz ladder.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions (for spec / workplans)
|
||||
|
||||
1. What is shard-wiki's **portable span address**? Git blob+range works for Git-native
|
||||
shards; what is the fallback for Obsidian/WebDAV/Gitea-wiki, and how does it
|
||||
survive a shard's storage swap (cf. UC-43 Foswiki RCS↔PlainFile)?
|
||||
2. Is **compose-by-reference (UC-44)** core orchestrator, adapter-provided, or
|
||||
reference-UI — and is it MVP or deferred with UC-32?
|
||||
3. Does **content-identity equivalence (UC-46)** belong in core (cross-shard union
|
||||
logic) or as an adapter-provided index? How expensive is span-set intersection at
|
||||
wiki scale without enfilades?
|
||||
4. How far do we take **reverse transclusion (UC-45)** — exact span identity only, or
|
||||
fuzzy/derived-content tracking? The latter is research-grade.
|
||||
5. Where does **reuse-terms/transcopyright metadata** sit on the L0→L4 ladder, and
|
||||
does any tier ever *enforce* it, or is it always advisory provenance?
|
||||
|
||||
---
|
||||
|
||||
## 12. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Wikipedia — Project Xanadu (https://en.wikipedia.org/wiki/Project_Xanadu) | 17 rules, tumblers, history, Web critique |
|
||||
| xanadu.com — The Edit Decision List / Xanadoc File (https://xanadu.com/xuEDL.html) | EDL/xanadoc structure: spans + xanalinks, client assembly |
|
||||
| Wikipedia — Enfilade (Xanadu) (https://en.wikipedia.org/wiki/Enfilade_(Xanadu)) | istream, enfilades (dsp/wid), spanfilade, granfilade, POOMfilade, version comparison by span-set intersection |
|
||||
| Wikipedia — Transclusion (https://en.wikipedia.org/wiki/Transclusion) | "same content knowably in more than one place," content-aware reuse, transcopyright, micropayment vs. weak modern forms |
|
||||
| Maggie Appleton — Xanadu Patterns (https://maggieappleton.com/xanadu-patterns) | pattern naming: visible links, parallel documents, transpointing windows, modular blocks, stable addresses, annotation |
|
||||
|
||||
Cross-references: `research/260608-federation-concepts/findings.md` §3.3 (prior
|
||||
surface treatment), `spec/UseCaseCatalog.md` (UC-24, UC-27, UC-29, UC-32),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (adapter contract).
|
||||
|
||||
---
|
||||
|
||||
## 13. Traceability
|
||||
|
||||
- New UCs: **UC-44, UC-45, UC-46** → `spec/UseCaseCatalog.md` (section F, federation).
|
||||
- Enriched UCs: **UC-24, UC-27, UC-29, UC-32**.
|
||||
- Architecture (no UC): span-addressing capability, content-identity, composition
|
||||
manifest, reuse-terms metadata → `SHARD-WP-0002`.
|
||||
- Decision link: transcopyright-as-policy → [[shard-wiki-auth-in-core-decision]].
|
||||
</content>
|
||||
</invoke>
|
||||
43
research/260614-zigzag-deep-dive/README.md
Normal file
43
research/260614-zigzag-deep-dive/README.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# 260614 — ZigZag / zzstructure deep dive (an information space as orthogonal dimensions)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Ted Nelson's ZigZag** and its data model **zzstructure**, asked
|
||||
through a specific shard-wiki question: a wiki page is linked in many independent ways
|
||||
at once — a **namespace** tree, a **created-from / genealogy** lineage, **hyperlinks**,
|
||||
plus shard-wiki's own **shard / overlay / version / transclusion / equivalence**
|
||||
relations. Could the information space be modelled as a zzstructure, where each
|
||||
relationship is a first-class **dimension** and none is privileged?
|
||||
|
||||
Distinctive material:
|
||||
- the zzstructure model — **cells**, **dimensions** (edge colours), **ranks**,
|
||||
**posward/negward**, and the one rule that defines it: *at most one posward and one
|
||||
negward neighbour per dimension per cell* (a directed coloured multigraph)
|
||||
- **views** — I-view (one dimension), **H-view** (pivot two dimensions into a cross),
|
||||
raster-vs-view
|
||||
- trees/lists/tables as *spent dimensions*; many-to-many graphs as the hard case
|
||||
(handled by **clones**)
|
||||
- **clone ↔ transclusion** convergence with the Xanadu dive
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | zzstructure model, views, fit analysis, a proposed dimension map for a shard-wiki space, recommendation (lens not store), INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-47 navigate-by-dimension, UC-48 H-view pivot/cross-tab, UC-49 genealogy
|
||||
dimension); UC-05/17–22/26/29 enriched. A **dimensional projection layer** is logged
|
||||
as architecture for `workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
|
||||
**Recommendation recorded:** adopt zzstructure as a **navigation / visualization /
|
||||
indexing lens** (a derived dimensional projection over the union), **not** as the
|
||||
storage substrate — Git and sovereign shards remain canonical (INTENT Stability Note).
|
||||
The many-to-many hyperlink graph does not fit zzstructure's rank constraint and stays a
|
||||
separate graph index. Pairs with `research/260614-xanadu-deep-dive/` (clone ↔
|
||||
transclusion).
|
||||
</content>
|
||||
294
research/260614-zigzag-deep-dive/findings.md
Normal file
294
research/260614-zigzag-deep-dive/findings.md
Normal file
@@ -0,0 +1,294 @@
|
||||
# Findings — ZigZag / zzstructure: an information space as orthogonal dimensions
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **conceptual / data-model prior art** (a data model + visualization
|
||||
paradigm, not a deployable wiki engine, not a candidate shard backend)
|
||||
Lens: shard-wiki — modelling a wiki information space where a page participates in
|
||||
*many* relationship structures at once (namespace, genealogy, links, versions, shards)
|
||||
|
||||
> Motivating question (from the user). A wiki page is linked in several independent
|
||||
> ways at once: a **primary namespace** association giving a tree-like hierarchy; a
|
||||
> **created-from / genealogy** association (fork/derivation lineage); **hyperlinks**
|
||||
> to other pages; and — in shard-wiki — *which shard* it lives in, its **overlays**,
|
||||
> its **versions**, its **transclusions**, and its **equivalents** elsewhere. Could
|
||||
> the information space be modelled as Ted Nelson's **ZigZag / zzstructure**, where
|
||||
> each kind of relationship is a separate, first-class **dimension** and no single
|
||||
> relationship is privileged as *the* structure?
|
||||
>
|
||||
> Short answer: **yes as a navigation / visualization / indexing lens; no as the
|
||||
> storage substrate.** zzstructure is an unusually good *conceptual* match for
|
||||
> shard-wiki's "union without erasure" thesis — it makes every relationship
|
||||
> co-equal and separately navigable — but its one-neighbour-per-direction
|
||||
> constraint makes it a poor *direct* encoding of the many-to-many hyperlink graph,
|
||||
> and Git/heterogeneous shards must remain the canonical store. Details below.
|
||||
|
||||
Companion to the Xanadu dive (`research/260614-xanadu-deep-dive/`): ZigZag is
|
||||
Nelson's *other* lifelong project, the structural/visual counterpart to Xanadu's
|
||||
literary/addressing one. Where Xanadu gives shard-wiki **reference-not-copy**, ZigZag
|
||||
gives it **many co-equal structures over the same cells**.
|
||||
|
||||
---
|
||||
|
||||
## 1. What ZigZag / zzstructure is
|
||||
|
||||
ZigZag (Ted Nelson, conceived 1981 at Datapoint; first Perl prototype by Andrew Pam,
|
||||
1997; the **GZigZag / Gzz** Java project 2000–2003; US patent 6,262,736, expired
|
||||
2019) is **"a data model that deconstructs the spreadsheet to allow irregular
|
||||
relations, generalizing the idea to multiple dimensions."** The generic, vendor-free
|
||||
name for the data model is **zzstructure**.
|
||||
|
||||
Core vocabulary:
|
||||
|
||||
- **Cell (zzcell)** — an atom of data (a value, a record, a page). Cells are the only
|
||||
content; everything else is connection.
|
||||
- **Dimension** — a *named kind of connection* (an edge colour). Dimensions are
|
||||
**first-class** — in ZigZag, dimensions are themselves cells.
|
||||
- **Link (zzlink)** — a connection between two cells *along one dimension*, with a
|
||||
**posward** (positive) and **negward** (negative) direction.
|
||||
- **Rank** — the maximal chain of cells along one dimension (a path, or a cycle). A
|
||||
rank is ZigZag's equivalent of "a row" / "a list" — but it exists only along one
|
||||
named dimension.
|
||||
- **Cursor** — the current focus cell from which views are drawn.
|
||||
- **Clone** — a copy of a cell that shares identity with its head cell (along a
|
||||
special clone dimension), used to let one logical item appear in several places /
|
||||
participate in many-to-one relations.
|
||||
|
||||
---
|
||||
|
||||
## 2. The one rule that defines everything
|
||||
|
||||
Formally (McGuffin & schraefel, HT2004; McGuffin's graph-theoretic intro), a
|
||||
zzstructure is:
|
||||
|
||||
> a **directed multigraph whose edges are coloured (typed)**, subject to the
|
||||
> restriction that **each node has at most one incoming edge of each colour and at
|
||||
> most one outgoing edge of each colour.**
|
||||
|
||||
Equivalently: **along any single dimension, a cell has at most one posward and one
|
||||
negward neighbour.** That single constraint is the whole design:
|
||||
|
||||
- Each dimension decomposes the cell set into disjoint **ranks** (paths and cycles) —
|
||||
never a branching tree, never a fan-out, *within one dimension*.
|
||||
- Many dimensions coexist over the **same** cells, each imposing its own independent
|
||||
rank structure. A cell sits at the intersection of as many dimensions as it
|
||||
participates in.
|
||||
- There is **no inherent hierarchy.** A tree is not the substrate; it is *one
|
||||
possible dimension* (encoded left-child/right-sibling — see §4).
|
||||
|
||||
This is the property that matters for shard-wiki: **the same set of pages can carry
|
||||
many orthogonal structures simultaneously, with none privileged.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Views — how multiple dimensions are seen
|
||||
|
||||
Because no screen can show all dimensions, ZigZag *rotates* two into view at a time:
|
||||
|
||||
- **I-view** — cells along **one** dimension from the cursor, drawn as a line.
|
||||
- **H-view** — a 2-D cross: **two** dimensions through the cursor, one drawn
|
||||
horizontally and one vertically; only cells actually connected to the cursor along
|
||||
those two dimensions appear (empty spreadsheet cells simply do not exist).
|
||||
- **I+/H+ (augmented) views** — fill remaining space with further neighbours.
|
||||
- **Raster vs. view (terminology)** — a *raster* selects which cells to take from the
|
||||
structure; a *view* places them on screen. "Pivoting" rotates a different dimension
|
||||
into the horizontal or vertical slot.
|
||||
|
||||
The H-view "pivot" is the genuinely interesting UX idea for a *federated* wiki: put
|
||||
**pages of a namespace** on one axis and **shards** (or **versions across shards**)
|
||||
on the other, then rotate to re-cross-tabulate the same union by a different pair of
|
||||
relationships.
|
||||
|
||||
---
|
||||
|
||||
## 4. Trees, lists, tables, graphs — what fits and what does not
|
||||
|
||||
zzstructure subsumes simpler structures by spending dimensions:
|
||||
|
||||
- **List** = one rank along one dimension.
|
||||
- **Table** = two dimensions (rows / columns), shown as an H-view.
|
||||
- **Tree** = encoded as a zzstructure "analogous to the left-child, right-sibling
|
||||
pointer implementation" — i.e. a *parent→firstchild* dimension plus a
|
||||
*sibling→sibling* dimension. So a hierarchy is **two dimensions**, not a primitive.
|
||||
- **Many-to-one / many-to-many** = NOT directly expressible (the one-edge-per-colour
|
||||
rule forbids fan-out). It is simulated with **clones**: the head cell links to many
|
||||
clone cells along the clone dimension, and each clone participates in a different
|
||||
rank.
|
||||
|
||||
**This is the critical caveat for wikis.** A namespace hierarchy (each page has *one*
|
||||
parent) and a created-from genealogy (each page has *one* origin) fit zzstructure
|
||||
*natively* and beautifully. But a wiki's **hyperlink graph is many-to-many** (a page
|
||||
links to many pages and is linked from many) — it is *not* a rank and cannot be one
|
||||
dimension. Representing links/backlinks in zzstructure requires cloning or treating
|
||||
the link set as data, which is awkward. zzstructure is excellent for the
|
||||
**functional/sequential/hierarchical** relationships and weak for the **arbitrary
|
||||
graph** ones.
|
||||
|
||||
---
|
||||
|
||||
## 5. Mapping a shard-wiki information space onto dimensions
|
||||
|
||||
Taking the user's framing literally — model the union of pages as cells, and each
|
||||
relationship as a dimension:
|
||||
|
||||
| Dimension (proposed) | Rank meaning | Fits zzstructure? | shard-wiki source |
|
||||
|----------------------|--------------|-------------------|-------------------|
|
||||
| `d.namespace-child` + `d.namespace-sibling` | namespace tree (left-child/right-sibling) | **Yes** (2 dims) | UC-22, yawex topics |
|
||||
| `d.genealogy` | created-from / fork lineage (each page one origin) | **Yes** (functional) | UC-26 fork, UC-29 remix |
|
||||
| `d.version` | revision order of a page | **Yes** (a rank = a history) | coordination journal |
|
||||
| `d.shard` | pages grouped by owning shard | **Yes** | shard sovereignty |
|
||||
| `d.overlay` | page → its overlays | **Yes** (or clones) | overlay model |
|
||||
| `d.equivalence` | parallel versions of the same topic across shards | **Yes** (a rank of equivalents) | UC-27, UC-46 |
|
||||
| `d.recent` | temporal order of changes | **Yes** | UC-17 RecentChanges |
|
||||
| `d.alphabetical` | AllPages ordering | **Yes** | UC-19 |
|
||||
| `d.links` / `d.backlinks` | hyperlink graph | **No** (many-to-many) — needs clones / stays a graph index | UC-05, UC-18 |
|
||||
| `d.transclusion` | content reused in many places | **No** as rank — but maps to **clones** (= "same content in many places") | UC-32, UC-44/45 |
|
||||
|
||||
Observation: shard-wiki's existing **derived views** (UC-05, UC-17–UC-20) are, in
|
||||
zzstructure terms, *dimensions + rasters*: RecentChanges is a raster along `d.recent`,
|
||||
AllPages along `d.alphabetical`, SiteMap is an H-view over the two namespace
|
||||
dimensions. zzstructure offers a **single unifying vocabulary** for what shard-wiki
|
||||
currently treats as a grab-bag of separate derived views.
|
||||
|
||||
Observation 2: ZigZag **clones** are conceptually the same move as Xanadu
|
||||
transclusion ("the same content knowably in more than one place") — the two Nelson
|
||||
projects converge here. A transcluded span / equivalent page is a clone that
|
||||
participates in multiple ranks.
|
||||
|
||||
---
|
||||
|
||||
## 6. Is zzstructure the right model for shard-wiki? (recommendation)
|
||||
|
||||
**Adopt as a lens, not a substrate.**
|
||||
|
||||
- **As a navigation & visualization model: promising.** Letting a reader pick which
|
||||
*dimension* to traverse (namespace vs. genealogy vs. version vs. shard) and pivot
|
||||
two dimensions into an H-view is a real, differentiated UX for a federated,
|
||||
provenance-preserving wiki. It operationalizes "union without erasure": every
|
||||
relationship is co-equal and visible, none hidden. (UC-47, UC-48.)
|
||||
- **As a conceptual model for the page graph: useful and clarifying.** Reframing the
|
||||
scattered derived views as *dimensions over one cell set* is a clean mental model
|
||||
and a candidate internal index/API shape. It also de-privileges the namespace tree
|
||||
(UC-49) — consistent with **mechanism over policy** (the hierarchy is one
|
||||
configurable dimension, not the canonical structure).
|
||||
- **As the storage substrate: reject.** Git and heterogeneous sovereign shards remain
|
||||
canonical (per INTENT Stability Note — Git's role is architectural). A zzstructure
|
||||
is a **derived, projected index** computed over the union, like BackLinks today —
|
||||
not a database that replaces the shards. The many-to-many hyperlink graph does not
|
||||
fit the rank constraint, and forcing the wiki into a zz-DB would violate shard
|
||||
sovereignty and the orchestrator-not-engine boundary.
|
||||
|
||||
Net: zzstructure is a strong answer to *"how do we present a page's many
|
||||
simultaneous relationships without picking a winner?"* — a **dimensional projection**
|
||||
layer over the existing union, not a new store.
|
||||
|
||||
---
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
|
||||
- **Union without erasure ↔ co-equal dimensions.** zzstructure's refusal to privilege
|
||||
any one structure is the data-model expression of shard-wiki's refusal to hide
|
||||
provenance, conflicts, or alternative arrangements.
|
||||
- **Mechanism over policy ↔ hierarchy-as-just-a-dimension.** The namespace tree being
|
||||
one dimension among many is exactly "do not hard-code one canonical structure."
|
||||
- **Provenance ↔ genealogy dimension.** Created-from lineage as a navigable rank makes
|
||||
provenance a first-class traversal, not metadata buried in a footer.
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **zzstructure as store.** Replacing Git/shards with a zz-database would break the
|
||||
Stability-Note boundary on Git's role and on orchestrator-vs-engine. zzstructure is
|
||||
a projection, full stop.
|
||||
2. **Forcing the hyperlink graph into ranks.** The link/backlink graph is many-to-many
|
||||
and must stay a graph index; do not contort it into a single dimension. Use clones
|
||||
or keep it as a separate graph projection.
|
||||
3. **One global zz-space.** Like Xanadu's docuverse, a single universal zz-space would
|
||||
fight shard sovereignty. Each information space gets its *own* dimensional
|
||||
projection over *its* attached shards.
|
||||
|
||||
### 7.3 What ZigZag teaches that shard-wiki should keep
|
||||
|
||||
- A page is best understood as **a cell at the intersection of many independent
|
||||
relationships**, not as a node in one tree. Build the page model and the navigation
|
||||
API so *adding a new relationship is adding a dimension*, not special-casing a view.
|
||||
- **Pivoting two relationships** (H-view) is a powerful, under-used exploration
|
||||
primitive for federation (namespace × shard, page × version, topic × equivalent).
|
||||
|
||||
---
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-46**. New UCs **UC-47–UC-49** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Navigate along a chosen relationship dimension** — pick namespace / genealogy / links / version / shard as the active traversal axis | **UC-47 (new)** |
|
||||
| **Pivot two dimensions as a cross-tab (H-view)** — e.g. namespace × shard, page × version-across-shards | **UC-48 (new)** |
|
||||
| **First-class created-from / fork genealogy dimension** — navigate derivation lineage as a rank | **UC-49 (new)** |
|
||||
| Namespace hierarchy is *one dimension among many*, not the privileged structure | **enriches UC-22** |
|
||||
| Derived views (RecentChanges, AllPages, SiteMap, BackLinks) reframed as dimensions + rasters | **enriches UC-05** (anchor for the UC-17–UC-20 view reframing) |
|
||||
| Fork/remix produce a genealogy edge consumed by UC-49 | **enriches UC-26, UC-29** |
|
||||
| Clone = "same content in many places" = transclusion convergence | links UC-32, UC-44/45 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- A **dimensional projection layer** over the union is a candidate internal model:
|
||||
pages = cells; relationships (namespace, genealogy, version, shard, equivalence,
|
||||
overlay, recent, alphabetical) = dimensions; existing derived views = rasters over
|
||||
them. It is **derived/lazy**, never canonical store.
|
||||
- The **link/backlink graph stays a separate many-to-many index**; do not model it as
|
||||
a rank. Reconcile with the dimensional model via clones or by keeping graph and
|
||||
dimensional projections side by side.
|
||||
- **Genealogy** (created-from) should be recorded as an edge when fork/remix/import
|
||||
happens (UC-26, UC-29) so the `d.genealogy` rank can be reconstructed — ties to the
|
||||
coordination journal.
|
||||
- Whether the dimensional model is exposed in a **public navigation API** or stays an
|
||||
internal organizing concept is open (see §10).
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions (for spec / workplans)
|
||||
|
||||
1. Is the dimensional/zzstructure model a **public navigation API** and UI paradigm,
|
||||
or only an *internal* organizing concept for the existing derived views?
|
||||
2. How is the **many-to-many link graph** reconciled with the rank-based dimensional
|
||||
model — clones, a parallel graph index, or both?
|
||||
3. Which dimensions are **core** (namespace, version, shard, genealogy, recent) vs.
|
||||
**adapter-provided** vs. **computed on demand** (equivalence, backlinks)?
|
||||
4. Is **H-view pivoting** worth a reference-UI investment, or is it research-only?
|
||||
5. Does the **clone = transclusion** convergence justify a single internal primitive
|
||||
shared by UC-44/45 (transclusion) and UC-47/48 (dimensional navigation)?
|
||||
|
||||
---
|
||||
|
||||
## 11. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Wikipedia — ZigZag (software) (https://en.wikipedia.org/wiki/ZigZag_(software)) | History, spreadsheet-deconstruction framing, one-connection-per-dimension rule, pivoting |
|
||||
| Wikipedia — Zzstructure (https://en.wikipedia.org/wiki/Zzstructure) | Directed-coloured-multigraph definition, dimensions-are-cells, raster vs. view |
|
||||
| McGuffin & schraefel, "A Comparison of Hyperstructures: Zzstructures, mSpaces, and Polyarchies," ACM Hypertext 2004 (https://www.dgp.toronto.edu/papers/mmcguffin_HT2004.pdf) | Graph-theoretic formal definition; taxonomy vs. mSpaces/polyarchies |
|
||||
| McGuffin, "A Graph-Theoretic Introduction to Ted Nelson's Zzstructures" (https://www.dgp.toronto.edu/~mjmcguff/research/zigzag/) | Ranks, I-view/H-view/augmented views, clones, tree-as-left-child/right-sibling, "multiple arrangements coexist via dimensions" |
|
||||
| Moore & Brailsford, "Unified Hyperstructures for Bioinformatics," JoDI (https://jodi-ojs-tdl.tdl.org/jodi/article/download/jodi-138/127) | Applied zzstructure as a unifying model over heterogeneous structures |
|
||||
| Nelson, "A Cosmology for a Different Computer Universe" (zzstructure context) | Nelson's own framing of cells/dimensions/visualization |
|
||||
| Gzz — "A Gentle Introduction to Ted Nelson's ZigZag Structure" (https://www.nongnu.org/gzz/gi/gi.html) | Practical cell/dimension/rank examples (rate-limited; corroborated by above) |
|
||||
|
||||
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (clone↔transclusion
|
||||
convergence), `spec/UseCaseCatalog.md` (UC-05, UC-17–UC-22, UC-26, UC-27, UC-29,
|
||||
UC-32, UC-44/45/46), `workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
|
||||
---
|
||||
|
||||
## 12. Traceability
|
||||
|
||||
- New UCs: **UC-47, UC-48, UC-49** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-05** (anchors the UC-17–UC-20 derived-view reframing), **UC-22,
|
||||
UC-26, UC-29**.
|
||||
- Architecture (no UC): dimensional projection layer; link-graph-stays-graph;
|
||||
genealogy edges in the journal; clone↔transclusion primitive → `SHARD-WP-0002`.
|
||||
- Decision boundary recorded: zzstructure is a **derived lens, not the store**;
|
||||
Git/shards remain canonical (INTENT Stability Note).
|
||||
</content>
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user