generated from coulomb/repo-seed
Compare commits
163 Commits
7d1fb97968
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 776f5af5a7 | |||
| fd961c83b4 | |||
| cca5bf83c3 | |||
| def699c1eb | |||
| a4e0f52ec1 | |||
| 4231daf94f | |||
| 37681d89b6 | |||
| a8e65235a8 | |||
| d7d046cac0 | |||
| 0b3ab2086f | |||
| d85d019543 | |||
| 3a5acdcb28 | |||
| 34b0c539f3 | |||
| da540d4eea | |||
| 951b24300d | |||
| c731c96634 | |||
| f0fee65cc0 | |||
| 34432c2e15 | |||
| 45a858ead0 | |||
| b31e9bc337 | |||
| e50dcc6b5c | |||
| a165cced33 | |||
| 8393a9c55d | |||
| ff96ee0c48 | |||
| 8b353f1077 | |||
| b9bb1f7d10 | |||
| c40fa3c934 | |||
| 54c2bf2ae5 | |||
| 6d8bd837a4 | |||
| b48a99d3c2 | |||
| 9b7f86ba69 | |||
| 74142096d0 | |||
| 2100e956aa | |||
| e62560eb5a | |||
| b147d3e831 | |||
| cdcf4b09aa | |||
| b21efe307b | |||
| e18397272a | |||
| 0ee972f2e2 | |||
| bb1b54e0af | |||
| b70f1c9acc | |||
| 8de044bbde | |||
| 67d851be0b | |||
| 46ec6f2a5f | |||
| ad4a2dbf5a | |||
| 23bc597343 | |||
| ec7a1ec946 | |||
| d6d7cc555f | |||
| 1dd1def40f | |||
| 6d341cd4e6 | |||
| 1b3d4aaa39 | |||
| 6e49bd3a4b | |||
| d2c73b02d9 | |||
| 042e198286 | |||
| 3ea0cc1226 | |||
| 4be2f190a0 | |||
| 7d00ae758e | |||
| 715ab1ca00 | |||
| d797bc5ee4 | |||
| 92d5774baf | |||
| e24f0034a0 | |||
| b2ea276c00 | |||
| 9fedb31d8b | |||
| 517cf1d282 | |||
| b44b2a74a4 | |||
| 24108b65aa | |||
| 6d48a1d3e6 | |||
| 9a4e00a05a | |||
| 5a77ea879c | |||
| aca9bf30f9 | |||
| 3b4f10a349 | |||
| f2dd2e124a | |||
| 1f3aba7aad | |||
| d65f9e21f3 | |||
| 802a80231a | |||
| 40575045ca | |||
| a2f15806f5 | |||
| 9b5fceb451 | |||
| d39e7f7765 | |||
| 1ad70a9c8a | |||
| 3a753a6f3b | |||
| 08a2148079 | |||
| cbd29e0a32 | |||
| 079652b61f | |||
| 84ab25eeb9 | |||
| 229d24d1cf | |||
| f21b7b5259 | |||
| c895d33091 | |||
| 59c36ac9d1 | |||
| 012d151fe8 | |||
| af840576e9 | |||
| 0383d78440 | |||
| dc451b0f4e | |||
| 04be66161e | |||
| c8dbe9b573 | |||
| b5d2cbc330 | |||
| ee2449d987 | |||
| f1dc7aff61 | |||
| dd812abb81 | |||
| 9b5b393519 | |||
| 6a88eb7710 | |||
| d898307b7e | |||
| 90a854f841 | |||
| 3d137c96b6 | |||
| f2f9f31df8 | |||
| 2c978d71f0 | |||
| b676579407 | |||
| 7fb90219b5 | |||
| 56b6cdd110 | |||
| 4acadacfee | |||
| 1546ca09bf | |||
| a3ca3e975b | |||
| 25091dbd2e | |||
| 1e3aabc143 | |||
| 25a714efa7 | |||
| 3eb026139f | |||
| 5ba6975573 | |||
| d4afce3699 | |||
| 2397c2c4b3 | |||
| a7084ec74b | |||
| 700566b1e2 | |||
| af5841a0d7 | |||
| 8941e131e8 | |||
| ce4b0cb9e4 | |||
| 6878a0c184 | |||
| 87f30e7c62 | |||
| e0c3d0699c | |||
| 1e962a0aef | |||
| 54468b60bc | |||
| 9f0fc57428 | |||
| ee805d5af7 | |||
| 188643b0f5 | |||
| fee47514a2 | |||
| 1281ca689b | |||
| dbc42f05db | |||
| 31ec8ab205 | |||
| a6629bdb29 | |||
| ecb7d0154e | |||
| d955fa239c | |||
| 036dbad816 | |||
| 9acd4e2841 | |||
| 5ada333bb4 | |||
| 9b5defa6d8 | |||
| 4543f5dc43 | |||
| 714eaf6b5a | |||
| 5697669063 | |||
| 3e0c05a2b3 | |||
| 6ccf349209 | |||
| e00c160a43 | |||
| 26bee2cc84 | |||
| fae3cbde8b | |||
| 64796f7b3a | |||
| f0be5799aa | |||
| dfff9ab42e | |||
| 7c22c468af | |||
| c1424e8863 | |||
| 0a335eca8b | |||
| 60e1aa661c | |||
| 316fdeedc2 | |||
| fa7adab239 | |||
| ffd5459b3e | |||
| 27da940df1 | |||
| 0473c3f043 |
20
.claude/rules/agents.md
Normal file
20
.claude/rules/agents.md
Normal file
@@ -0,0 +1,20 @@
|
||||
## Kaizen Agents
|
||||
|
||||
Specialized agent personas available on demand via the state-hub MCP.
|
||||
|
||||
**Discover:** `list_kaizen_agents()` — returns all agents with name, description, category
|
||||
**Load:** `get_kaizen_agent("tdd-workflow")` — returns full instructions; read and follow them
|
||||
|
||||
Common agents:
|
||||
|
||||
| Agent | Category | When to use |
|
||||
|-------|----------|-------------|
|
||||
| `tdd-workflow` | testing | Step-by-step TDD8 workflow for any feature |
|
||||
| `code-refactoring` | quality | Code quality analysis and safe refactoring |
|
||||
| `test-maintenance` | testing | Diagnose and fix failing tests |
|
||||
| `requirements-engineering` | process | Prevent interface/mock mismatches upfront |
|
||||
| `keepaTodofile` | process | Maintain TODO.md during work |
|
||||
| `project-management` | process | Track status, determine next steps |
|
||||
| `datamodel-optimization` | quality | Optimize dataclasses and data structures |
|
||||
|
||||
All 17 agents: call `list_kaizen_agents()` for the full list.
|
||||
8
.claude/rules/architecture.md
Normal file
8
.claude/rules/architecture.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Architecture
|
||||
|
||||
<!-- TODO: Describe the key design decisions and component structure.
|
||||
Key modules, data flows, external integrations, state machines, etc. -->
|
||||
|
||||
## Quick Reference
|
||||
|
||||
`~/state-hub/mcp_server/TOOLS.md` — MCP tool reference
|
||||
50
.claude/rules/credential-routing.md
Normal file
50
.claude/rules/credential-routing.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=shard-wiki` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
38
.claude/rules/first-session.md
Normal file
38
.claude/rules/first-session.md
Normal file
@@ -0,0 +1,38 @@
|
||||
## First Session Protocol
|
||||
|
||||
Triggered when `get_domain_summary("consumer")` shows **no workstreams**.
|
||||
The project is registered but work has not yet been structured.
|
||||
|
||||
**Step 1 — Read, don't write**
|
||||
- `~/the-custodian/canon/projects/consumer/project_charter_v0.1.md` — purpose, scope
|
||||
- `~/the-custodian/canon/projects/consumer/roadmap_v0.1.md` — planned phases
|
||||
- Scan repo root: README, directory structure, existing code or docs
|
||||
|
||||
**Step 2 — Survey in-progress work**
|
||||
Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete.
|
||||
|
||||
**Step 3 — Propose workstreams to Bernd**
|
||||
Propose 1–3 workstreams — each a coherent strand, weeks to months, anchored to a
|
||||
roadmap phase. **Wait for approval before creating.**
|
||||
|
||||
**Step 4 — Create workplan file first, then DB record (ADR-001)**
|
||||
```
|
||||
workplans/SHARD-WP-NNNN-<slug>.md ← write this first
|
||||
```
|
||||
Then register in the hub:
|
||||
```
|
||||
create_workstream(topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0", title="...", owner="...", description="...")
|
||||
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
|
||||
```
|
||||
|
||||
**Step 5 — Record the setup**
|
||||
```
|
||||
add_progress_event(
|
||||
summary="First session: structured consumer into N workstreams, M tasks",
|
||||
event_type="milestone",
|
||||
topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0",
|
||||
detail={"workstreams": [...], "tasks_created": M}
|
||||
)
|
||||
```
|
||||
|
||||
<!-- Delete or archive this file once past first session -->
|
||||
8
.claude/rules/repo-boundary.md
Normal file
8
.claude/rules/repo-boundary.md
Normal file
@@ -0,0 +1,8 @@
|
||||
## Repo boundary
|
||||
|
||||
This repo owns **shard-wiki** only. It does not own:
|
||||
|
||||
<!-- TODO: List what belongs in adjacent repos, e.g.:
|
||||
- SSH key management → railiance-infra/
|
||||
- State hub code → state-hub/
|
||||
-->
|
||||
5
.claude/rules/repo-identity.md
Normal file
5
.claude/rules/repo-identity.md
Normal file
@@ -0,0 +1,5 @@
|
||||
**Purpose:** Git-based Markdown wiki orchestrator and federation layer. Python (src/ layout, hatchling, pytest). Early-stage: scaffold + INTENT.md defined, domain model not yet implemented. See INTENT.md for authoritative scope.
|
||||
|
||||
**Domain:** consumer
|
||||
**Repo slug:** shard-wiki
|
||||
**Topic ID:** 4c2e5315-2cb9-447c-9d16-a39bdb0aabd0
|
||||
85
.claude/rules/session-protocol.md
Normal file
85
.claude/rules/session-protocol.md
Normal file
@@ -0,0 +1,85 @@
|
||||
## Session Protocol
|
||||
|
||||
Dev Hub (State Hub API): http://127.0.0.1:8000
|
||||
MCP server name in `~/.claude.json`: `dev-hub`
|
||||
|
||||
**Step 1 — Orient**
|
||||
|
||||
Read the offline-safe brief first — it works without a live hub connection:
|
||||
```bash
|
||||
cat .custodian-brief.md
|
||||
```
|
||||
Then call the MCP tool for richer cross-domain context when MCP tools are exposed:
|
||||
```
|
||||
get_domain_summary("consumer")
|
||||
```
|
||||
If MCP tools are unavailable in the current agent session, use the REST API:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
|
||||
```
|
||||
If the hub is offline: `cd ~/state-hub && make api`
|
||||
|
||||
**Step 2 — Check inbox**
|
||||
With MCP tools:
|
||||
```
|
||||
get_messages(to_agent="shard-wiki", unread_only=True)
|
||||
```
|
||||
Mark read with `mark_message_read(message_id)`. Reply or act on coordination
|
||||
requests before proceeding.
|
||||
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=shard-wiki&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
**Step 3 — Scan workplans**
|
||||
```bash
|
||||
ls workplans/
|
||||
```
|
||||
For each file with `status: ready`, `active`, or `blocked`, note pending
|
||||
`wait`/`todo`/`progress` tasks.
|
||||
|
||||
**Step 4 — Present brief**
|
||||
|
||||
1. **Active workstreams** for `consumer` — title, task counts, blocking decisions
|
||||
2. **Pending tasks** from `workplans/` + any `[repo:shard-wiki]` hub tasks
|
||||
3. **Goal guidance** — if `goal_guidance` in summary:
|
||||
- `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"*
|
||||
- `alignment_warnings`: flag if active work is not aligned with current goal
|
||||
4. **Suggested next action** — highest-priority open item
|
||||
5. **SBOM status** — flag if `last_sbom_at` is unset for this repo
|
||||
|
||||
If no workstreams: follow First Session Protocol (`first-session.md`).
|
||||
|
||||
**During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()`
|
||||
|
||||
> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`)
|
||||
> are First Session Protocol only. Work structure belongs in repo files (ADR-001).
|
||||
|
||||
**Session close:**
|
||||
With MCP tools:
|
||||
```
|
||||
add_progress_event(summary="...", topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0", workstream_id="<uuid>")
|
||||
```
|
||||
Without MCP tools:
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"topic_id":"4c2e5315-2cb9-447c-9d16-a39bdb0aabd0","workstream_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
|
||||
```
|
||||
If workplan files were modified, ensure the local copy is up to date first:
|
||||
```bash
|
||||
git -C <repo_path> pull --ff-only
|
||||
cd ~/state-hub && make fix-consistency REPO=shard-wiki
|
||||
```
|
||||
For repos where implementation runs on a remote machine (e.g. CoulombCore),
|
||||
use the combined target which pulls before fixing:
|
||||
```bash
|
||||
cd ~/state-hub && make fix-consistency-remote REPO=shard-wiki
|
||||
```
|
||||
**C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback
|
||||
will sync the file to match DB. **C-16** (repo behind remote) blocks all writes
|
||||
until you pull — intentional to prevent clobbering remote progress.
|
||||
19
.claude/rules/stack-and-commands.md
Normal file
19
.claude/rules/stack-and-commands.md
Normal file
@@ -0,0 +1,19 @@
|
||||
## Stack
|
||||
|
||||
<!-- TODO: Fill in language, frameworks, and key dependencies -->
|
||||
- **Language:**
|
||||
- **Key deps:**
|
||||
|
||||
## Dev Commands
|
||||
|
||||
```bash
|
||||
# TODO: Fill in the standard commands for this repo
|
||||
|
||||
# Install dependencies
|
||||
|
||||
# Run tests
|
||||
|
||||
# Lint / type check
|
||||
|
||||
# Build / package (if applicable)
|
||||
```
|
||||
40
.claude/rules/workplan-convention.md
Normal file
40
.claude/rules/workplan-convention.md
Normal file
@@ -0,0 +1,40 @@
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
File location: `workplans/SHARD-WP-NNNN-<slug>.md`
|
||||
ID prefix: `SHARD-WP-`
|
||||
|
||||
Work items originate as files in this repo **before** being registered in the hub.
|
||||
|
||||
Canonical workplan/workstream frontmatter statuses are:
|
||||
`proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`.
|
||||
Use `proposed` for a newly drafted plan, `ready` after review against current
|
||||
repo state, and `finished` when implementation is complete. `stalled` and
|
||||
`needs_review` are derived health labels, not stored statuses.
|
||||
|
||||
Closed workplans may be moved to `workplans/archived/` with a completion-date
|
||||
prefix: `YYMMDD-SHARD-WP-NNNN-<slug>.md`. The frontmatter id remains
|
||||
unchanged; the prefix is only for quick visual reference.
|
||||
|
||||
Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**:
|
||||
`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids
|
||||
`ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed
|
||||
directly. Promote anything requiring analysis, design, approval, dependencies, or
|
||||
multiple planned phases into a normal workplan.
|
||||
|
||||
Ecosystem todos from other agents arrive as `[repo:shard-wiki]` hub tasks —
|
||||
visible at session start. Pick one up by creating the workplan file, then registering
|
||||
the workstream.
|
||||
|
||||
Task blocks use this shape:
|
||||
|
||||
```task
|
||||
id: SHARD-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
```
|
||||
|
||||
Status progression is `todo` → `progress` → `done`; use `wait` for waiting or
|
||||
blocked work and `cancel` for stopped work.
|
||||
|
||||
<!-- Ralph Loop rules and HEUREKA sequence: ~/.claude/CLAUDE.md — do not duplicate here -->
|
||||
@@ -2,34 +2,46 @@
|
||||
# Custodian Brief — shard-wiki
|
||||
|
||||
**Domain:** whynot
|
||||
**Last synced:** 2026-06-08 12:55 UTC
|
||||
**Last synced:** 2026-06-15 22:57 UTC
|
||||
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
|
||||
|
||||
## Active Workstreams
|
||||
|
||||
### federation architecture design
|
||||
Progress: 0/10 done | workstream_id: `2af4c46d-cbfd-40ea-a94b-d9e60b0f9945`
|
||||
### second adapter — git-IS-store shard (contract validation on a new substrate)
|
||||
Progress: 0/3 done | workstream_id: `9e24eeb0-c0f0-41e6-a1ca-88d71e4139ea`
|
||||
|
||||
**Open tasks:**
|
||||
- · Architecture positioning and boundaries `ea8fdb22`
|
||||
- · Remix primitives: fork, overlay, import, reference `fb7d4bce`
|
||||
- · Equivalent page identity and multi-version presentation `8f2a333d`
|
||||
- · History, attribution, and coordination journal `5f39f48d`
|
||||
- · Union composition layer `3ff71e11`
|
||||
- · Change notification and subscription transports `9596e5e8`
|
||||
- · Information space lifecycle `38134064`
|
||||
- … and 3 more open tasks
|
||||
- · GitShardAdapter — read over a git working tree/repo `8a1c7c80`
|
||||
- · Write = commit; current_rev = sha (drift) `b47dfb86`
|
||||
- · History adopt + integration with union/overlay `4c895f42`
|
||||
|
||||
### shard-wiki requirements from yawex prior art
|
||||
Progress: 0/6 done | workstream_id: `0ed023a2-760b-4990-b931-8ee1f41ea08f`
|
||||
### incremental union maintenance + equivalence index + I-2 verification
|
||||
Progress: 0/4 done | workstream_id: `78d48bcf-6482-4266-bc81-084b7ec1cd80`
|
||||
|
||||
**Open tasks:**
|
||||
- · Design federation page-resolution model (yawex state space as inspiration) `ebc036e4`
|
||||
- · Define namespace/path model and page+shard roles `431b4d28`
|
||||
- · Specify union-level derived views (BackLinks, RecentChanges, AllPages, SiteMap, Search) `564545ec`
|
||||
- · Provenance & freshness model for pages/revisions/projections `738326f5`
|
||||
- · Overlay / lightweight-patch model (from yawex append/comment) `a268de6a`
|
||||
- · Markdown link semantics: wikilink + red-link extension `a7499f3e`
|
||||
- · Equivalence index: blocking + verify `842f480b`
|
||||
- · Incremental maintenance (delta, not additive) `2da4e0b8`
|
||||
- · I-2 verification: digest + consistency-checker `b602ce31`
|
||||
- · Wire incremental tier behind resolution + views `2f3d083c`
|
||||
|
||||
### derived views — wikilinks, BackLinks, RecentChanges, AllPages/SiteMap
|
||||
Progress: 0/5 done | workstream_id: `2fe15330-ddf6-4b0f-8e55-ada341375d35`
|
||||
|
||||
**Open tasks:**
|
||||
- · Wikilink + red-link model `792660c3`
|
||||
- · BackLinks (core) `431a54c3`
|
||||
- · RecentChanges (core) `270c1c31`
|
||||
- · AllPages / SiteMap (core) `898ba43e`
|
||||
- · Wiring + integration `7157544b`
|
||||
|
||||
### git-backed DecisionLog + per-space append authority
|
||||
Progress: 0/4 done | workstream_id: `4fb5b29b-955c-4f37-85cf-58b4643ab1ca`
|
||||
|
||||
**Open tasks:**
|
||||
- · Git event-store backend (append = commit/object) `a8fcbb3e`
|
||||
- · Per-space append authority (lease) `62abd162`
|
||||
- · Fold over the git log + read-your-writes across processes `8cc3691e`
|
||||
- · Migration + wiring `281e1db4`
|
||||
|
||||
---
|
||||
## MCP Orientation (when available)
|
||||
|
||||
17
.repo-classification.yaml
Normal file
17
.repo-classification.yaml
Normal file
@@ -0,0 +1,17 @@
|
||||
repo_classification:
|
||||
standard: Repo Classification Standard
|
||||
version: '1.0'
|
||||
classified_at: '2026-06-22'
|
||||
classified_by: agent
|
||||
category: project
|
||||
domain: consumer
|
||||
secondary_domains: []
|
||||
capability_tags:
|
||||
- knowledge
|
||||
- documentation
|
||||
business_stake:
|
||||
- product
|
||||
- experience
|
||||
business_mechanics:
|
||||
- coordination
|
||||
- operation
|
||||
243
AGENTS.md
243
AGENTS.md
@@ -1,62 +1,219 @@
|
||||
# AGENTS.md
|
||||
# shard-wiki — Agent Instructions
|
||||
|
||||
Guidance for agents working in `shard-wiki`.
|
||||
## Repo Identity
|
||||
|
||||
## Read First
|
||||
**Purpose:** Git-based Markdown wiki orchestrator and federation layer. Python (src/ layout, hatchling, pytest). Early-stage: scaffold + INTENT.md defined, domain model not yet implemented. See INTENT.md for authoritative scope.
|
||||
|
||||
1. `INTENT.md` — aspiration and boundaries (stable; architectural changes are rare).
|
||||
2. `SCOPE.md` — what we are achieving now and current maturity.
|
||||
3. `.custodian-brief.md` — State Hub snapshot (generated; do not edit manually).
|
||||
**Domain:** consumer
|
||||
**Repo slug:** shard-wiki
|
||||
**Topic ID:** `4c2e5315-2cb9-447c-9d16-a39bdb0aabd0`
|
||||
**Workplan prefix:** `SHARD-WP-`
|
||||
|
||||
## Documentation Layout
|
||||
---
|
||||
|
||||
This repo follows the CoulombSocial / HelixForge / MarkiTect documentation
|
||||
layout (recommendation, not strict law). Efficient retrieval by purpose:
|
||||
## State Hub Integration
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `INTENT.md` | Aspiration and boundaries |
|
||||
| `SCOPE.md` | Top-level view of current achievement; closes gap to INTENT |
|
||||
| `research/` | Exploration results (`yymmdd-` prefix on files or subdirs) |
|
||||
| `demand/` | Inbound requests not yet reviewed into spec or workplans |
|
||||
| `spec/` | Implementation guardrails (PRD, TSD, use cases, architecture) |
|
||||
| `workplans/` | State Hub–registered implementation tasks |
|
||||
| `docs/` | Stakeholder documentation (users, developers, humans, agents) |
|
||||
| `wiki/` | Perspective-free interconnected knowledge (wiki UI when connected) |
|
||||
| `issues/` | Mirror of relevant open tickets when ticket systems are in use |
|
||||
| `history/` | Archived material (`yymmdd-` prefix); out of scope for daily work |
|
||||
The Custodian State Hub tracks work across all domains. Interact via HTTP REST —
|
||||
there is no MCP server for Codex agents.
|
||||
|
||||
**Mode of operation:** close SCOPE → INTENT while learning; refine both as needed.
|
||||
| Context | URL |
|
||||
|---------|-----|
|
||||
| Local workstation | `http://127.0.0.1:8000` |
|
||||
| Remote via tunnel | `http://127.0.0.1:18000` |
|
||||
|
||||
## Domain Vocabulary
|
||||
|
||||
Honor terms from `INTENT.md`: shard, root entity, adapter contract, projection,
|
||||
overlay, coordination journal, shard modes. Do not invent parallel vocabulary.
|
||||
|
||||
## Build And Test
|
||||
### Orient at session start
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]"
|
||||
pytest
|
||||
ruff check
|
||||
ruff format
|
||||
# Offline brief — works without hub connection
|
||||
cat .custodian-brief.md
|
||||
|
||||
# Active workstreams for this domain
|
||||
curl -s "http://127.0.0.1:8000/workstreams/?topic_id=4c2e5315-2cb9-447c-9d16-a39bdb0aabd0&status=active" \
|
||||
| python3 -m json.tool
|
||||
|
||||
# Check inbox
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=shard-wiki&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
## State Hub
|
||||
Mark a message read:
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
Workplans register with State Hub. After workplan changes:
|
||||
### Log progress (required at session close)
|
||||
|
||||
```bash
|
||||
cd ~/state-hub && make fix-consistency REPO=shard-wiki
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"summary": "what was done",
|
||||
"event_type": "note",
|
||||
"author": "codex",
|
||||
"workstream_id": "<uuid>",
|
||||
"task_id": "<uuid>"
|
||||
}'
|
||||
```
|
||||
|
||||
Finished or canceled workplans move to `history/` with a `yymmdd-` archive prefix.
|
||||
Omit `workstream_id` / `task_id` when not applicable.
|
||||
|
||||
## Where To Put New Material
|
||||
### Update task status
|
||||
|
||||
- Exploratory analysis → `research/yymmdd-<topic>/`
|
||||
- Raw feature ask or external requirement → `demand/`
|
||||
- Reviewed design ready to guide code → `spec/`
|
||||
- Implementation tasks → `workplans/`
|
||||
- User/dev/agent how-to → `docs/`
|
||||
- Collaborative unstructured notes → `wiki/`
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"status": "progress"}'
|
||||
# values: wait | todo | progress | done | cancel
|
||||
```
|
||||
|
||||
### Flag a task for human review
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"needs_human": true, "intervention_note": "reason"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Session Protocol
|
||||
|
||||
**Start:**
|
||||
1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe)
|
||||
2. Check inbox: `GET /messages/?to_agent=shard-wiki&unread_only=true`; mark read
|
||||
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
|
||||
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
|
||||
|
||||
**During work:**
|
||||
- Update task statuses in workplan files as tasks progress
|
||||
- Record significant decisions via `POST /decisions/`
|
||||
|
||||
**Close:**
|
||||
1. Update workplan file task statuses to reflect progress
|
||||
2. Log: `POST /progress/` with a summary of what changed
|
||||
3. Note for the custodian operator: after workplan file changes, run from
|
||||
`~/state-hub`:
|
||||
```bash
|
||||
make fix-consistency REPO=shard-wiki
|
||||
```
|
||||
This syncs task status from files into the hub DB.
|
||||
|
||||
---
|
||||
|
||||
## Credential and access routing
|
||||
|
||||
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
|
||||
for inference. Run this check **before** requesting secrets, API keys, SSH access,
|
||||
login tokens, or database passwords — in any repo, not only `ops-warden`.
|
||||
|
||||
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
|
||||
other credential need belongs to another subsystem. **Do not** message
|
||||
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
|
||||
|
||||
### Lookup (do this first)
|
||||
|
||||
```bash
|
||||
warden route find "<describe your need>" --json
|
||||
warden route show <catalog-id> --json
|
||||
```
|
||||
|
||||
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
|
||||
|
||||
| Agent runtime | How to orient |
|
||||
| --- | --- |
|
||||
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=shard-wiki` is for coordination, not secret vending |
|
||||
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
|
||||
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
|
||||
|
||||
### Quick routing table
|
||||
|
||||
| I need… | Owner | ops-warden executes? |
|
||||
| --- | --- | --- |
|
||||
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
|
||||
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
|
||||
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
|
||||
| Authorization decision | flex-auth | No — route only |
|
||||
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
|
||||
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
|
||||
|
||||
### Anti-patterns (do not do these)
|
||||
|
||||
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
|
||||
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
|
||||
- Pasting secrets into Git, State Hub, workplans, logs, or chat
|
||||
|
||||
### Other capabilities (reuse-surface)
|
||||
|
||||
Non-credential capabilities are usually discovered through **reuse-surface** federation
|
||||
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
|
||||
every repo's agent instructions because it is high-frequency, high-risk, and easy to
|
||||
get wrong.
|
||||
|
||||
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
|
||||
|
||||
<!-- REPO-AGENTS-EXTENSIONS -->
|
||||
<!-- Append repo-specific agent instructions below this marker.
|
||||
The state-hub template sync preserves content after this line. -->
|
||||
|
||||
---
|
||||
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
Work items originate as files in this repo — not in the hub. The hub is a
|
||||
read/cache/index layer that rebuilds from files.
|
||||
|
||||
**File location:** `workplans/SHARD-WP-NNNN-<slug>.md`
|
||||
|
||||
**Archived location:** finished workplans may move to
|
||||
`workplans/archived/YYMMDD-SHARD-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
|
||||
the completion/archive date; the frontmatter `id` does not change.
|
||||
|
||||
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
|
||||
`workplans/ADHOC-YYYY-MM-DD.md` with task ids `ADHOC-YYYY-MM-DD-T01`, etc. Use
|
||||
this only for low-risk work completed directly; create a normal workplan for
|
||||
anything needing analysis, design, approval, dependencies, or multiple phases.
|
||||
|
||||
**Frontmatter:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: SHARD-WP-NNNN
|
||||
type: workplan
|
||||
title: "..."
|
||||
domain: consumer
|
||||
repo: shard-wiki
|
||||
status: proposed | ready | active | blocked | backlog | finished | archived
|
||||
owner: codex
|
||||
topic_slug: ...
|
||||
created: "YYYY-MM-DD"
|
||||
updated: "YYYY-MM-DD"
|
||||
state_hub_workstream_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
---
|
||||
```
|
||||
|
||||
Use `proposed` for a new draft, `ready` after review against current repo
|
||||
state, and `finished` after implementation. `stalled` and `needs_review` are
|
||||
derived health labels, not frontmatter statuses.
|
||||
|
||||
**Task block format** (one per `##` section):
|
||||
|
||||
```
|
||||
## Task Title
|
||||
|
||||
` ` `task
|
||||
id: SHARD-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
|
||||
` ` `
|
||||
|
||||
Task description text.
|
||||
```
|
||||
|
||||
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
|
||||
|
||||
To create a new workplan:
|
||||
1. Write the file following the format above
|
||||
2. Notify the custodian operator to run `make fix-consistency REPO=shard-wiki`
|
||||
(or send a message to the hub agent via `POST /messages/`)
|
||||
|
||||
63
CLAUDE.md
63
CLAUDE.md
@@ -1,53 +1,12 @@
|
||||
# CLAUDE.md
|
||||
# shard-wiki — Claude Code Instructions
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Repository status
|
||||
|
||||
This is an **early-stage Python repository**. The package scaffold (`src/shard_wiki/`, `tests/`, `pyproject.toml`) exists with only smoke tests — the domain model is not yet implemented. Read `INTENT.md` (aspiration), `SCOPE.md` (current achievement), and `AGENTS.md` (layout and conventions) before designing anything. Close the gap from SCOPE to INTENT via `research/`, `spec/`, and `workplans/`.
|
||||
|
||||
## What this project is
|
||||
|
||||
`shard-wiki` is a **Git-based Markdown wiki orchestrator and federation layer**, not a wiki engine. It lets multiple heterogeneous wiki-shaped page stores (**shards**) attach to a shared root entity and be presented as a **union of pages**, while preserving each shard's separate storage, provenance, capabilities, and history.
|
||||
|
||||
The core job is orchestration across backends — Git repos, repo subdirectories (`wiki/`), Gitea wikis, local folders, Obsidian vaults, WebDAV/Nextcloud directories, Coulomb spaces — never replacing or homogenizing them.
|
||||
|
||||
## Core domain model (the concepts code must honor)
|
||||
|
||||
These abstractions come from `INTENT.md` and define the architecture. New code should map onto them rather than inventing parallel vocabulary:
|
||||
|
||||
- **Shard** — an independently meaningful page store attached to a root entity. Shards have *sovereignty*: their own backend, capabilities, limits, history, and identity model. Not all shards are Git-native.
|
||||
- **Root entity / information space** — the joined space that shards attach to. Each information space should have a **Git-addressable coordination layer** (history, patches, review, backup, reconciliation) even when individual shards are not Git-native.
|
||||
- **Shard adapter contract** — the versioned interface a backend implements to participate. Adapters are **capability-aware**: the core must model explicitly which operations a shard supports (read, write, diff, merge, lock, version, publish, accept patches) rather than assuming uniformity.
|
||||
- **Wiki page model** — a stable, versioned, Markdown-first but backend-neutral representation of pages, paths, links, metadata, revisions.
|
||||
- **Projection** — a lazy, cache-like local view of remote/external shard content. Prefer lazy projection over eager copying.
|
||||
- **Overlay** — a non-destructive local edit against a remote, read-only, or capability-limited shard, representable as drafts/patches/commits/merge requests *before* destructive application ("overlay before mutation").
|
||||
- **Coordination journal** — the Git-backed record of change flows for an information space.
|
||||
- **Shard modes** — read-only, write-through, mirrored, projected, cached, canonical.
|
||||
|
||||
## Design constraints to enforce in code
|
||||
|
||||
These are hard boundaries from `INTENT.md`; treat violations as design bugs:
|
||||
|
||||
- **Mechanism over policy.** Provide primitives for federation, sync, overlays, patching, conflict detection, projection, reconciliation. Do *not* hard-code one editorial/sync/conflict/canonical-source policy — keep those configurable.
|
||||
- **Union without erasure.** Always preserve provenance: which shard a page came from, its freshness, whether it is cached, whether it has overlays, whether it diverges from an equivalent page elsewhere. Never hide authorship, conflicts, freshness, or backend limitations.
|
||||
- **No silent remote mutation.** Do not mutate remote systems without explicit adapter support and user intent.
|
||||
- **Graceful degradation.** Limited backends must still be usable as read-only/cache/projection/backup/patch targets.
|
||||
- **Not a file-sync daemon.** Synchronization is wiki-page-semantic, not generic file mirroring.
|
||||
|
||||
`INTENT.md` has a "Stability Note": changes that redefine what a shard is, Git's role, how root entities are modeled, or whether this is an orchestrator vs. an engine are **architectural changes** and should be rare and deliberate.
|
||||
|
||||
## Build, test, run
|
||||
|
||||
Python with a `src/` layout, built via hatchling, tested with pytest. Tests run against the source tree directly (`pythonpath = ["src"]` in `pyproject.toml`), so no install/editable step is required to run them.
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]" # one-time: install dev tooling (pytest, pytest-cov, ruff)
|
||||
pytest # run the full test suite
|
||||
pytest tests/test_package.py::test_version_is_exposed # run a single test
|
||||
pytest --cov # run with coverage
|
||||
ruff check # lint
|
||||
ruff format # format
|
||||
```
|
||||
|
||||
Note: the system `pytest` is 7.4.x; `minversion` in `pyproject.toml` is pinned to `7.0` to match. Bump it if a newer pytest is installed into the dev environment.
|
||||
@SCOPE.md
|
||||
@.claude/rules/repo-identity.md
|
||||
@.claude/rules/session-protocol.md
|
||||
@.claude/rules/first-session.md
|
||||
@.claude/rules/workplan-convention.md
|
||||
@.claude/rules/stack-and-commands.md
|
||||
@.claude/rules/architecture.md
|
||||
@.claude/rules/repo-boundary.md
|
||||
@.claude/rules/credential-routing.md
|
||||
@.claude/rules/agents.md
|
||||
|
||||
10
INTENT.md
10
INTENT.md
@@ -16,6 +16,8 @@ The goal is to allow independently stored and differently implemented wikis, pag
|
||||
|
||||
The repository provides a **shard orchestration layer** for interconnected Markdown and markup-based wiki content.
|
||||
|
||||
Equivalently, shard-wiki can be used as a **headless, API-first wiki engine** — optimized for **integrating heterogeneous data sources** and for **efficient access by agents and automation** — that ships its own native engine as one (canonical-mode) shard among many. There is no bundled UI: presentation and rendering are consumer concerns.
|
||||
|
||||
It allows wiki-like systems to:
|
||||
|
||||
* Attach heterogeneous page stores as shards of a shared information space
|
||||
@@ -30,6 +32,7 @@ It allows wiki-like systems to:
|
||||
* Run fully standalone with open read/write access and complete change history, then progressively layer multi-tenant enterprise access control through external identity integration
|
||||
* Allow existing wiki engines to become federation-capable through a shared API
|
||||
* Allow non-federation-aware systems to participate through adapters and projections
|
||||
* Serve as a **headless, API-first wiki engine** (a small typed-extension core) that integrates heterogeneous data sources and is consumed efficiently by agents and automation
|
||||
|
||||
It transforms disconnected wiki engines, Git repositories, local folders, WebDAV directories, application-specific content stores, and desktop editing workflows into a **composable federated wiki space**.
|
||||
|
||||
@@ -85,7 +88,7 @@ A mature `shard-wiki` should allow each participating shard to see the others as
|
||||
|
||||
This repository is **not** intended to:
|
||||
|
||||
* Replace all wiki engines with a single canonical wiki implementation
|
||||
* Replace all wiki engines with a single canonical wiki implementation *(shard-wiki MAY still provide its own native, headless, API-first engine as one optional shard backend — see Design Principles — but never as a mandated or universal replacement)*
|
||||
* Force every shard to use the same backend, database, directory layout, or storage format
|
||||
* Require every participating system to become federation-aware
|
||||
* Require every participating shard to be Git-native
|
||||
@@ -148,6 +151,9 @@ Policy decisions such as conflict preference, canonical source selection, public
|
||||
* **Composable integration**
|
||||
Wiki engines should be able to use the `shard-wiki` API to become federation-enabled without reimplementing federation internally.
|
||||
|
||||
* **Native reference engine (additive, headless & API-first)**
|
||||
shard-wiki MAY provide its own native wiki-engine as a **canonical-mode shard backend** — a **small core** with a **typed-extension framework**, activated **per shard** (only what you need). It is **headless and API-first** (no bundled UI; presentation/rendering are consumer concerns) and tuned for **integrating heterogeneous data sources** and **efficient agent/automation access**. It is *one shard type among many*, implemented against shard-wiki's own adapter contract; it does **not** replace other engines, mandate a single implementation, or change shard-wiki's role as an orchestrator. Shard sovereignty and union-without-erasure are preserved.
|
||||
|
||||
* **Open by default, progressively governed**
|
||||
A standalone `shard-wiki` must be runnable with zero external dependencies in a classic Ward Cunningham / c2-style open read/write-for-all mode. Access control is an *additive capability*, not a precondition: the same core progresses — without re-architecture — to authenticated single-user, to group/role-based, to multi-tenant enterprise access control, mirroring the NetKingdom capability ladder (lightweight → expanded).
|
||||
|
||||
@@ -201,3 +207,5 @@ Such changes should be rare, because they affect all downstream systems relying
|
||||
|
||||
In particular, changes that redefine what counts as a shard, what role Git plays, how root entities are modeled, or whether `shard-wiki` is an orchestrator rather than a wiki engine should be treated as architectural changes.
|
||||
|
||||
**Amendment — 2026-06-15 (SHARD-WP-0013 T4, decision `84ffdb48`):** admits an **additive** native reference wiki-engine — **headless, API-first**, a small typed-extension core — as a **canonical-mode shard backend** optimized for data-source integration and agent access. Deliberate, narrow scope change; shard-wiki remains an orchestrator and neither mandates nor replaces other engines. (Mirrors the earlier auth-in-core amendment precedent.)
|
||||
|
||||
|
||||
25
SCOPE.md
25
SCOPE.md
@@ -17,12 +17,12 @@ Learnings update both SCOPE and INTENT where necessary.
|
||||
|
||||
| Layer | State |
|
||||
|-------|-------|
|
||||
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
|
||||
| Code | Foundation slice implemented (SHARD-WP-0007): `provenance` + `policy` leaves, `model` (Identity/Placement/Span/Page/CapabilityProfile), `adapters` (contract + FolderAdapter + conformance suite), `coordination` (event-sourced DecisionLog), `union` (resolution + chorus, overlay-aware), `InformationSpace` orchestrator. Write path added (SHARD-WP-0008): writable adapter, overlay engine (draft→patch→apply-under-drift), edit() unifies write-through + overlay-before-mutation. Native engine implemented (SHARD-WP-0014): `engine` (kernel + typed-extension runtime + per-shard activation [ADR-0001] + capability-profile-from-extensions + EngineShardAdapter + the `ext.struct` built-in) — an engine shard attaches to an InformationSpace as a canonical-mode shard. Git-backed coordination log (SHARD-WP-0009): `DecisionLog` storage factored behind an `EventStore`; `GitEventStore` makes the log git-addressable (each space a ref, append = immutable CAS-guarded commit), a per-space `AppendAuthority` (lease) gives a single-writer total order with re-grantable HA hand-off, cross-process read-your-writes verified, and a verbatim one-time importer (`migrate_space`/JSONL) replays in-memory logs into git; `InformationSpace.git_backed(...)` wires it. Derived views (SHARD-WP-0010): `views` (wikilink + red-link model, BackLinks, RecentChanges, AllPages/SiteMap) — recomputable, provenance-carrying, presentation-free, exposed via `InformationSpace.backlinks/recent_changes/all_pages/site_map`. Incremental-first derived tier (SHARD-WP-0011): `incremental` (indexed equivalence via MinHash/LSH blocking + verify, change-driven delta maintenance with retraction/propagation, Merkle-style digest + self-healing I-2 consistency-checker, `UnionIndex` routed behind `InformationSpace.all_pages` with rebuild as explicit fallback). Second adapter (SHARD-WP-0012): `GitShardAdapter` — git-IS-store substrate (read=tracked *.md, write=commit, current_rev=per-path sha for drift, adopted git-native history), passes conformance, works across folder+git shards in union/overlay/edit with no core change (capability-as-data proven on a second substrate). 196 tests green, ~97% coverage |
|
||||
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
|
||||
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`) |
|
||||
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) |
|
||||
| Demand | NetKingdom integration asks captured, not yet negotiated |
|
||||
| Spec | Architecture blueprint drafted; UseCaseCatalog 25 UCs from research; PRD/TSD scaffolds |
|
||||
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) |
|
||||
| Spec | CoreArchitectureBlueprint (whole-system, hardened via SHARD-WP-0005/0006) + FederationArchitecture + FederationRequirements + TSD §A adapter contract + ArchitectureBlueprint (auth/history) + WikiEngineCoreArchitecture (headless API-first engine, SHARD-WP-0013) drafted; UseCaseCatalog 84 UCs (+ engine capability-structure layer); PRD scaffold |
|
||||
| Work | `SHARD-WP-0001` **done** (6 ADRs: yawex-derived federation requirements → `spec/FederationRequirements.md`); `SHARD-WP-0002` **done** (18 tasks → `FederationArchitecture.md` [T1–T10, T17] + `TechnicalSpecificationDocument.md` §A adapter contract [T11–T16, T18]); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` **done** (all 8 computational-knowledge dives T1–T8 complete + "computational page model" synthesis); `SHARD-WP-0005` **done** (9 tasks: CoreArchitectureBlueprint hardened against the 260615 review); `SHARD-WP-0006` **done** (5 tasks: round-2 hardening — overview reconciled, event-sourced coordination + append authority, adapter conformance, incremental correctness + I-2 verification) |
|
||||
|
||||
## In Scope (today)
|
||||
|
||||
@@ -32,11 +32,15 @@ Learnings update both SCOPE and INTENT where necessary.
|
||||
- Authorization model design (delegated authentication, core authorization).
|
||||
- Shard adapter contract and wiki page model (to be specified, then implemented).
|
||||
- Git-backed coordination journal for information spaces.
|
||||
- A **native, headless, API-first wiki-engine core** (small typed-extension core, as a
|
||||
canonical-mode shard backend) — design via SHARD-WP-0013; optimized for data-source
|
||||
integration and agent access.
|
||||
- State Hub workplan registration and consistency sync.
|
||||
|
||||
## Out Of Scope (today)
|
||||
|
||||
- A standalone wiki engine UI or rendering pipeline.
|
||||
- A wiki-engine **UI or rendering pipeline** (the engine is headless/API-first; presentation
|
||||
is a consumer concern). A bundled standalone UI is not provided.
|
||||
- Authentication, credential storage, or user directory implementation.
|
||||
- Hard-coded editorial, sync, or conflict-resolution policy.
|
||||
- Generic file mirroring independent of wiki-page semantics.
|
||||
@@ -55,6 +59,11 @@ live in core.
|
||||
Design work is tracked in `workplans/SHARD-WP-0001-yawex-requirements.md`
|
||||
(yawex-derived resolution, namespaces, overlays) and
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (federation architecture,
|
||||
decisions, tradeoffs). Specification outputs land in `spec/`. Inbound
|
||||
integration asks remain in `demand/` until reviewed and promoted into spec or
|
||||
workplans.
|
||||
decisions, tradeoffs). Research continues under
|
||||
`workplans/SHARD-WP-0003-engine-dives-batch.md` (remaining new-insight wiki
|
||||
engines + git-forge wikis + classic engines) and
|
||||
`workplans/SHARD-WP-0004-computational-knowledge-systems.md` (literate
|
||||
programming, computational notebooks, live-coding REPLs, image-based/moldable
|
||||
environments — the executable/computational page-model thread). Specification
|
||||
outputs land in `spec/`. Inbound integration asks remain in `demand/` until
|
||||
reviewed and promoted into spec or workplans.
|
||||
91
history/260615-core-architecture-blueprint-review-2.md
Normal file
91
history/260615-core-architecture-blueprint-review-2.md
Normal file
@@ -0,0 +1,91 @@
|
||||
# Critical review (round 2) — CoreArchitectureBlueprint.md (hardened)
|
||||
|
||||
Date: 2026-06-15 · Reviewer: tegwick (with Claude) · Subject:
|
||||
`spec/CoreArchitectureBlueprint.md` after **SHARD-WP-0005** (commit f21b7b5) · Feeds:
|
||||
**SHARD-WP-0006**
|
||||
|
||||
A second hostile pass over the *hardened* blueprint. Round 1 found design bugs; this round
|
||||
finds (a) self-consistency regressions the surgical hardening introduced, and (b) deeper
|
||||
second-order gaps — some of which the hardening *sharpened*. Verdict: the architecture is now
|
||||
substantially sound; what remains in §B/§C are hard distributed-systems/operational questions,
|
||||
not design smells — except §A (a real regression) and three foundational gaps in §B.
|
||||
|
||||
---
|
||||
|
||||
## A. Self-consistency regressions introduced by surgical hardening
|
||||
|
||||
The 9 edits deepened §6–§9 but did not propagate to the **overview surfaces**, so the document
|
||||
now contradicts itself between its summary and its body (and readers trust the summary).
|
||||
|
||||
- **A-1 (real contradiction).** §4 still says "Addressing, **equivalence**, and transclusion
|
||||
key on identity" — the exact conflation T2 fixed in §7.2 (equivalence keys on *content
|
||||
fingerprint across distinct identities*). → **WP-0006 T1**
|
||||
- **A-2.** §4 "Projection — typed on two axes" and §4 "Provenance envelope … every artifact
|
||||
carries [full wrapper]" are stale vs T7's §8.4 (two-axis = extension point; trivial default)
|
||||
and §7.3 (layered effective-vs-own). → **T1**
|
||||
- **A-3.** §10 policy surface omits knobs the hardening added (freshness/staleness §8.8,
|
||||
squash-compaction §8.1, conflict-resolution preset §8.6, tenant-partition) — yet §11 defines
|
||||
`policy/` as "owns the §10 surface." The module contract points at a stale list. → **T1**
|
||||
- **A-4 (cosmetic).** §3 diagram + §11 header still say "L4 rebuildable cache" / "15 spectra,"
|
||||
advertising the pre-hardening model (§8.7 incremental-first; §6.5 orthogonal-core). → **T1**
|
||||
|
||||
Meta-point: surgical editing hardened the body but regressed whole-document coherence; v2
|
||||
needs an **overview-reconciliation pass**.
|
||||
|
||||
## B. Foundational gaps (serious; some sharpened by the hardening)
|
||||
|
||||
- **B-1 — The journal is now a concurrent-write DB, but it's single-writer Git.** §8.6's
|
||||
consistency model assumes "the journal is local Git, read-your-writes." L4 multi-tenant + the
|
||||
L6 Orchestrator API imply a server; HA/scale implies *multiple* instances. Concurrent commits
|
||||
of coordination-canonical state to one git journal = lock contention / merge races; Git is not
|
||||
a concurrent-write store. Either single-writer-per-space (an unstated HA ceiling) or a real
|
||||
concurrent coordination store with Git as an *export*. **T1 of WP-0005 worsened this** by
|
||||
loading more canonical state into the journal. The keystone unanswered question. → **T2**
|
||||
- **B-2 — Capability-as-data trusts self-reported profiles with no conformance check.** I-3 +
|
||||
§6.5's degradation contract assume the profile tells the truth. A buggy adapter (claims
|
||||
`merge=git/text`, corrupts; claims `notify`, never emits) silently poisons every degradation
|
||||
decision. No **adapter conformance suite** (declared profile == observed behavior) exists.
|
||||
Foundational for an architecture whose correctness rests on profile accuracy. → **T3**
|
||||
- **B-3 — "Coordination-canonical state in the journal" has no representation design.** T1
|
||||
relocated overlays/bindings/aliases/equivalence-sets/merges into "the journal" without saying
|
||||
*how* Git stores structured mutable state. "All equivalences touching X" over a git-of-files
|
||||
is O(scan) unless indexed — and an index is L4/derived. The new central concept is a black
|
||||
box; resolve *with* B-1. → **T2**
|
||||
- **B-4 — Incremental equivalence is under-specified/likely incorrect; I-2 only eventually
|
||||
true.** §8.7 re-verifies a changed page's *new* candidate set but not the pairs it *leaves*
|
||||
(a page exiting an LSH bucket can break an existing equivalence edge); the delta is not
|
||||
additive. Deeper: incremental maintenance drifts from `f(canonical)`, so I-2 holds only
|
||||
eventually, guaranteed solely by an expensive reconcile-against-rebuild. Needs a stated
|
||||
verification mechanism (background checker / digest-vs-sampled-rebuild). → **T4**
|
||||
|
||||
## C. Real but second-tier (track as open problems O-8…O-11)
|
||||
|
||||
- **C-1 — Mechanism-over-policy → operator burden; no preset bundles.** ~7 knob families with
|
||||
sub-modes and interactions; only authz (L0–L4) bundles into personas. Need named bundles
|
||||
("personal vault" / "team wiki" / "enterprise federation"). → **O-8 / T5**
|
||||
- **C-2 — Tenant partitioning (I-13) vs shard sharing + lazy projection.** A shard in two roots
|
||||
is cached twice → duplicate storage + double refresh on rate-limited backends. Shard
|
||||
exclusive-to-one-root or shareable? Unresolved. → **O-9 / T5**
|
||||
- **C-3 — Span-level authz + transclusion is an unmodeled leak path.** Authz is per
|
||||
page/shard/tenant; transclusion crosses shards at span granularity → a page can leak a span
|
||||
past its ACL (aggregation/inference). §7.3's ⊕ also stops being simple two-level inheritance
|
||||
across a transclusion boundary. → **O-10 / T5**
|
||||
- **C-4 — Union-under-unavailability undefined.** Freshness covers *stale*, nothing covers
|
||||
*down*. The dead-shard read path (partial union? error? last-known?) is unspecified though
|
||||
it's the commonest real failure. → **O-11 / T5**
|
||||
|
||||
## D. Recommended resolution (→ SHARD-WP-0006)
|
||||
|
||||
1. **§A reconciliation** (T1) — make the overview match the hardened body.
|
||||
2. **Journal & coordination-state model** (T2) — settle single-vs-multi-writer and separate the
|
||||
**coordination-state store** from the **content-history journal**. Likely resolution:
|
||||
**event-sourced coordination** — an append-only *decision log* is the coordination-canonical
|
||||
tier (git-addressable, I-6 preserved); the queryable current state (alias table, equivalence
|
||||
set) is a *derived fold* of the log (disposable). Append-logs tolerate concurrency far better
|
||||
than mutable-file Git; state a concurrency model. Resolves **B-1 + B-3** together.
|
||||
3. **Adapter conformance suite** (T3) — make a passing conformance run part of the contract
|
||||
(B-2): every adapter proves declared profile == observed behavior.
|
||||
4. **Incremental correctness + I-2 verification** (T4) — fix the leaving-bucket re-verification
|
||||
and propagation; add a background consistency-checker / derived-tier digest so I-2 is
|
||||
verifiable, not merely asserted (B-4).
|
||||
5. **Track §C** (T5) — O-8…O-11 with chosen direction + revisit trigger; close-out.
|
||||
122
history/260615-core-architecture-blueprint-review.md
Normal file
122
history/260615-core-architecture-blueprint-review.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# Critical review — CoreArchitectureBlueprint.md
|
||||
|
||||
Date: 2026-06-15 · Reviewer: tegwick (with Claude) · Subject:
|
||||
`spec/CoreArchitectureBlueprint.md` @ commit **9b5b393** · Feeds: **SHARD-WP-0005**
|
||||
|
||||
A deliberately hostile review of the first whole-system architecture, to find where it
|
||||
**breaks (correctness)**, **fails to scale**, and **could be more elegant/efficient** before
|
||||
any implementation. Findings are prioritised; each is the input to a SHARD-WP-0005 task.
|
||||
|
||||
## Verdict in one line
|
||||
|
||||
The **layering and the dual narrow waist are sound and stay**. The **thesis is ~90% right**;
|
||||
the missing 10% (curatorial / coordination-canonical state) breaks its clean story. There are
|
||||
**two genuine bugs**, **two large unaddressed scaling risks**, and several **elegance/efficiency
|
||||
debts** — all fixable without touching INTENT.
|
||||
|
||||
---
|
||||
|
||||
## A. The framing crack (fix resolves three issues)
|
||||
|
||||
**A-1 — Two buckets hide a third.** The thesis "canonical at the edges, derived in the middle"
|
||||
omits **born-in-the-middle-but-canonical** state: overlays that are the local truth against a
|
||||
read-only shard (Flow C), manual **curator equivalence bindings**, alias tables, merge
|
||||
decisions. These encode human judgment or local-only content and **cannot be rebuilt** from
|
||||
shards+journal.
|
||||
|
||||
**Contradiction:** I-2 declares L4 rebuildable, yet §8.4 puts "alias table, curator binding"
|
||||
in L4. You cannot rebuild a curator's manual binding.
|
||||
|
||||
**Fix:** three states — **sharded-canonical**, **coordination-canonical** (journal: overlays,
|
||||
bindings, aliases, merges — durable, born in the middle), **derived-disposable** (union graph,
|
||||
indexes, projections). Re-frame §1 as **canonical (sharded + coordination) vs derived
|
||||
(disposable)**; `derived = f(canonical)` then becomes actually true. → **T1**
|
||||
|
||||
---
|
||||
|
||||
## B. Where it breaks (correctness)
|
||||
|
||||
**B-1 — Identity conflated with content-fingerprint (BUG).** §7.2 derives page identity from
|
||||
content fingerprint. That makes **editing a page change its identity**, breaking every
|
||||
reference. Fingerprints identify *versions/equivalence*, not *identity*. Page identity must be
|
||||
a **stable handle (uid)** surviving edits; fingerprints belong to the **equivalence** mechanism
|
||||
(§8.4). One word, two concepts, wrong implementation for the stable one. → **T2**
|
||||
|
||||
**B-2 — No concurrency/consistency model.** Concurrent overlays on one page, overlay applied
|
||||
after source drift, journal-commit vs shard-native-write ordering — all undefined. Conflict
|
||||
handling is deferred to "policy presets," but **conflict *detection + representation* is core
|
||||
mechanism**; only *resolution* is policy. The union's consistency guarantee is unstated
|
||||
(eventually-consistent? read-your-writes? causal-via-journal?). → **T3**
|
||||
|
||||
**B-3 — Persisted union cache + multi-tenant = leak surface.** §13 recommends a persisted L4
|
||||
cache; §9 protects content by *read-time* filtering on the provenance envelope. A persisted
|
||||
cross-tenant union cache guarded only by read-time filtering is an L4 attack surface. Tension
|
||||
between I-2 (persisted rebuildable cache), scale, and L5 isolation is unacknowledged. → **T8**
|
||||
|
||||
---
|
||||
|
||||
## C. Where it fails to scale
|
||||
|
||||
**C-1 — Equivalence detection is O(N²), no indexing/incremental story.** Fingerprint /
|
||||
span-set-overlap across all pages of all shards is combinatorial (10 shards × 100k pages ≈
|
||||
10¹² comparisons). No blocking/LSH/indexing, no incremental maintenance. Biggest scaling
|
||||
hazard in the document. → **T4**
|
||||
|
||||
**C-2 — "Rebuildable cache" collides with the operational-envelope axis.** A byte-exact
|
||||
rebuild requires reading *every page of every shard*, including rate-limited/paginated
|
||||
external APIs (Notion) and irreducibly-live sources — hours-to-days. I-2 contradicts axis-10.
|
||||
**Incremental, change-driven maintenance must be primary** (notify→delta), rebuild a rare
|
||||
fallback. Cache invalidation — the actual hard problem — is named once and never designed. →
|
||||
**T4, T5**
|
||||
|
||||
**C-3 — Unbounded history at open L0 = DoS/perf.** "Every write a commit" + "open for all" ⇒
|
||||
the git journal grows without bound under bots/vandalism and git degrades on huge histories.
|
||||
"History is the floor" has an unacknowledged cost: packing, compaction, per-shard offload. →
|
||||
**T8**
|
||||
|
||||
---
|
||||
|
||||
## D. Elegance / efficiency debts
|
||||
|
||||
**D-1 — The 15 spectra assert a clean degradation function never demonstrated.** Either most
|
||||
axes are irrelevant to most ops (then the 15-D profile is ceremony), or behavior depends on
|
||||
several axes *jointly* (then "no per-backend code" becomes a sprawling axis-interaction matrix
|
||||
— the flat-checklist problem in higher dimensions). And the axes **aren't orthogonal**
|
||||
(git-native history ⟺ git-IS-store ⟺ git/text merge; encrypted opacity ⟹ query/translation
|
||||
collapse). Model a **smaller orthogonal core** + **derived/implied** positions, and state the
|
||||
**axis-interaction subset** the degradation logic truly uses. → **T6**
|
||||
|
||||
**D-2 — Provenance envelope isn't inherited; it'll dwarf the content.** Per-span envelopes at
|
||||
block granularity = 10k near-identical envelopes for a 10k-block graph. The doc already
|
||||
invented the right pattern for Trilium ("effective-vs-own with per-attribute provenance") and
|
||||
failed to apply it to its own envelope. Make provenance **layered (page envelope + span
|
||||
deltas)**. → **T7**
|
||||
|
||||
**D-3 — Projection machinery over-fit to the exotic tail.** Two-axis model + three facets +
|
||||
view registry exist mostly for UC-83/84 (2 of 84 UCs); the 95% case (markdown in git) pays the
|
||||
weight. Make the **common case trivial** (default = plain lazy replication) and
|
||||
derivation/liveness an **extension point**, not a taxonomy every projection instantiates. →
|
||||
**T7**
|
||||
|
||||
**D-4 — Cross-cutting rails are the highest-coupling components, presented as clean.**
|
||||
`provenance/` and capability types are imported by every layer (god-modules); an envelope
|
||||
change ripples everywhere. And **policy has no module** (§10 enumerates it; §11 omits it)
|
||||
despite being consulted by L3/L4/L5. Give policy a home; pin the rails behind stable narrow
|
||||
interfaces. → **T7**
|
||||
|
||||
---
|
||||
|
||||
## E. What explicitly stays
|
||||
|
||||
- The 6-layer model + the dual narrow waist (adapter contract / page model).
|
||||
- Capability-as-data (I-3), union-without-erasure (I-4), overlay-before-mutation (I-5),
|
||||
Git-addressable coordination (I-6), mechanism-over-policy (I-7), graceful degradation (I-8).
|
||||
- The federation-model taxonomy and the auth ladder (ArchitectureBlueprint.md).
|
||||
|
||||
## F. Disposition
|
||||
|
||||
Some findings are **solvable now** (A-1, B-1, D-2, D-3, D-4, C-3); some are **partially open**
|
||||
and should be tracked honestly rather than pretend-solved (B-2 consistency model: pick a
|
||||
guarantee; C-1 equivalence-at-scale: pick a blocking strategy; D-1 axis interactions: enumerate
|
||||
the real subset). SHARD-WP-0005 closes the solvable ones and records the open ones in a new
|
||||
"Known scaling risks & open problems" section of the blueprint. → **T9**
|
||||
45
history/260615-reuse-surface-contributions.md
Normal file
45
history/260615-reuse-surface-contributions.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# reuse-surface contributions — shard-wiki (SHARD-WP-0013 T3)
|
||||
|
||||
Date: 2026-06-15 · From: shard-wiki (whynot) · To: reuse-surface (helix_forge) · Tracked list
|
||||
of (A) capabilities shard-wiki registered, (B) **gaps** shard-wiki proposes the reuse surface
|
||||
add, and (C) capabilities shard-wiki will **consume** rather than rebuild. Communicated to the
|
||||
reuse-surface agent via state-hub `send_message`.
|
||||
|
||||
## A. Registered (T1 + T3) — 8 entries
|
||||
|
||||
`capability.wiki.{shard-orchestration, adapter-contract, page-model, coordination-journal,
|
||||
overlay, federation-models, engine-typed-extensions, derived-views}` — committed in
|
||||
reuse-surface, `validate` ok (20 entries total).
|
||||
|
||||
## B. Proposed gaps (cross-cutting; not shard-wiki-internal) → reuse-surface should own/define
|
||||
|
||||
- **G1 — `capability.platform.typed-extension-framework`** *(suggested)*
|
||||
A reusable pattern: a **small core + a stringent typed-extension framework** where extensions
|
||||
declare typed contracts, compose, and are **activated per context**. shard-wiki's wiki engine
|
||||
(`capability.wiki.engine-typed-extensions`) is one instance, but the *pattern* is
|
||||
cross-domain (any HelixForge capability platform). Evidence: shard-wiki UseCaseCatalog
|
||||
"Capability structure" layer (core + 10 typed extensions + conflict-mediation map).
|
||||
Suggested owner: helix_forge / reuse-surface. Relation: would `generalize`
|
||||
`capability.wiki.engine-typed-extensions`.
|
||||
|
||||
- **G2 — `capability.content.translation-fidelity`** *(suggested)*
|
||||
Lossless/lossy **content translation with an explicit fidelity report** (what round-trips
|
||||
cleanly vs degrades, non-mappable elements preserved as provenance). Reusable well beyond
|
||||
wiki (any format-bridging consumer). Evidence: shard-wiki TSD §A.6, UC-42/UC-59.
|
||||
|
||||
## C. Consumptions (reuse, do not rebuild)
|
||||
|
||||
- **`capability.feature-control.evaluate`** (helix_forge/feature-control) → shard-wiki's
|
||||
**per-shard extension/feature activation** (the "activate only what you need" mechanism).
|
||||
Already recorded as a relation on `capability.wiki.engine-typed-extensions`.
|
||||
- **`capability.authorization.policy-evaluate`** (flex-auth) → shard-wiki's **X-AUTHZ** policy
|
||||
decisions. shard-wiki owns the authz *model* (authz-in-core) but can reuse this evaluation
|
||||
engine rather than building one.
|
||||
- **`capability.statehub.{progress-log, workstream-coordinate}`** → already in use for
|
||||
coordination across this work.
|
||||
|
||||
## Status
|
||||
|
||||
Gaps G1/G2 are **suggestions** to the reuse-surface owner (not unilateral registrations, since
|
||||
they are cross-cutting, not shard-wiki-internal). Consumptions are recorded for the engine
|
||||
architecture (T5) so it reuses rather than reinvents.
|
||||
@@ -1,8 +1,19 @@
|
||||
# history/
|
||||
|
||||
Archived material that is no longer needed for daily work but should be kept.
|
||||
Archived material and the project's **meta-history**: finished/canceled workplans kept for
|
||||
the record, plus durable **reviews, critical assessments, and decision records** — the
|
||||
reasoning behind the specs, captured at a point in time.
|
||||
|
||||
Use a `yymmdd-` prefix when archiving files or directories. Content here is
|
||||
**out of scope** for regular tasks — consult only for research or diagnostics.
|
||||
Use a `yymmdd-` prefix. Archived material is **out of scope** for regular tasks (consult only
|
||||
for research or diagnostics); assessment/review records are point-in-time and may seed active
|
||||
workplans, but are not edited after the fact — supersede with a new dated record and link back.
|
||||
|
||||
Finished or canceled workplans from `workplans/` are archived here.
|
||||
Distinct from the **coordination journal** (a runtime Git-backed record of *content* change
|
||||
flows inside an information space, an INTENT domain concept); `history/` is the *project's own*
|
||||
design evolution.
|
||||
|
||||
| Date | Record | Subject |
|
||||
|------|--------|---------|
|
||||
| 2026-06-15 | `260615-core-architecture-blueprint-review.md` | Critical review of `spec/CoreArchitectureBlueprint.md` (commit 9b5b393); inputs to `SHARD-WP-0005` |
|
||||
| 2026-06-15 | `260615-core-architecture-blueprint-review-2.md` | Round-2 review of the hardened blueprint (post-`SHARD-WP-0005`, f21b7b5); inputs to `SHARD-WP-0006` |
|
||||
| 2026-06-15 | `260615-reuse-surface-contributions.md` | shard-wiki's reuse-surface registrations, proposed gaps (G1/G2), and consumptions (`SHARD-WP-0013` T3) |
|
||||
@@ -36,6 +36,11 @@ pythonpath = ["src"]
|
||||
branch = true
|
||||
source = ["shard_wiki"]
|
||||
|
||||
[tool.coverage.report]
|
||||
show_missing = true
|
||||
# Quality floor for `pytest --cov` / `coverage report` (not forced on a bare `pytest` run).
|
||||
fail_under = 90
|
||||
|
||||
[tool.ruff]
|
||||
src = ["src", "tests"]
|
||||
target-version = "py311"
|
||||
|
||||
12
registry/README.md
Normal file
12
registry/README.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Capability Registry
|
||||
|
||||
Markdown-first capability index for federation and reuse planning.
|
||||
|
||||
## Authoring
|
||||
|
||||
1. Copy a capability entry template (see reuse-surface `templates/capability-entry.template.md`).
|
||||
2. Add the row to `indexes/capabilities.yaml`.
|
||||
3. Run `reuse-surface validate` from a checkout with the CLI installed.
|
||||
4. Merge to `main` and verify publish with `reuse-surface establish --publish-check`.
|
||||
|
||||
Federation contract: reuse-surface `docs/RegistryFederation.md`.
|
||||
103
registry/capabilities/capability.wiki.adapter-contract.md
Normal file
103
registry/capabilities/capability.wiki.adapter-contract.md
Normal file
@@ -0,0 +1,103 @@
|
||||
---
|
||||
id: capability.wiki.adapter-contract
|
||||
name: Capability-Aware Shard Adapter Contract
|
||||
summary: A versioned backend interface where each binding declares a verified capability profile (positions on capability spectra), so federation ops degrade by capability.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, adapter, capability, contract, conformance, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Fifteen capability spectra with an orthogonal core + implication rules, plus
|
||||
a normative contract spec (TSD Section A); derived from a ~23-system synthesis.
|
||||
availability:
|
||||
current: A2
|
||||
target: A5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
AdapterContract + a read/write FolderAdapter + a conformance suite that
|
||||
verifies declared profile == observed behaviour exist as a source module.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- versioned interface with declared, conformance-verified capability profiles
|
||||
- one concrete adapter (file-store) passes the conformance suite
|
||||
broken_expectations:
|
||||
- only one substrate implemented (git-IS-store, REST, CRDT adapters planned)
|
||||
out_of_scope_expectations:
|
||||
- hosting backends
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- single adapter implemented so far
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Mediate heterogeneity at one narrow waist: a backend participates by implementing a
|
||||
versioned interface and declaring a verified position on each capability spectrum.
|
||||
includes:
|
||||
- capability profile as data (orthogonal-core spectra + implied positions)
|
||||
- operation verbs (read/write/diff/merge/notify/.../derive-projection/execute)
|
||||
- a conformance suite (profiles verified, not self-asserted)
|
||||
excludes:
|
||||
- assuming uniform backend capabilities
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-34..UC-43, UC-50, UC-57, UC-60..UC-69 (shard attachment & adapter binding)"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A5
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/adapters/"
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.page-model
|
||||
supports:
|
||||
- capability.wiki.shard-orchestration
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/TechnicalSpecificationDocument.md (Section A)"
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 6)"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_folder_adapter.py"
|
||||
- "shard-wiki/tests/test_conformance.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- exposing any page store as a capability-described, conformance-checked shard
|
||||
not_recommended_for:
|
||||
- backends that cannot honestly describe their capabilities
|
||||
known_limitations:
|
||||
- reference implementation covers the file-store substrate only so far
|
||||
---
|
||||
|
||||
# Capability-Aware Shard Adapter Contract
|
||||
|
||||
The bottom narrow waist of shard-wiki: a versioned interface plus a **verified** capability
|
||||
profile per binding. Core logic is written once against capabilities (not per-backend), and
|
||||
the conformance suite rejects profiles whose declared abilities don't match observed behaviour.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Fifteen spectra reduced to an orthogonal core with implication rules (CoreArchitectureBlueprint
|
||||
Section 6.5); normative in TSD Section A.
|
||||
|
||||
### Availability
|
||||
`adapters/` ships the contract, a folder adapter, and `assert_conformant`.
|
||||
103
registry/capabilities/capability.wiki.coordination-journal.md
Normal file
103
registry/capabilities/capability.wiki.coordination-journal.md
Normal file
@@ -0,0 +1,103 @@
|
||||
---
|
||||
id: capability.wiki.coordination-journal
|
||||
name: Event-Sourced Coordination Journal
|
||||
summary: An append-only, totally-ordered-per-space decision log (overlays, bindings, aliases, merges, forks) whose current state is a derived fold; git-addressable history.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, event-sourcing, coordination, git, journal, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Keystone resolved across two architecture reviews: coordination-canonical state
|
||||
as an append-only decision log with a per-space append authority; current state
|
||||
is a derived fold (derived = f(log)).
|
||||
availability:
|
||||
current: A2
|
||||
target: A4
|
||||
confidence: medium
|
||||
rationale: >
|
||||
In-memory DecisionLog + fold work as a source module; the git-backed store with a
|
||||
per-space lease (the production backing) is planned.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- append-only, totally-ordered-per-space log with read-your-writes
|
||||
- derived fold to aliases + transitively-merged equivalence groups
|
||||
broken_expectations:
|
||||
- git-backed storage and per-space lease/append-authority not yet implemented
|
||||
out_of_scope_expectations:
|
||||
- general-purpose event bus
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- in-memory backing only; cross-process durability pending
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Make coordination-canonical decisions durable and git-addressable as events, with the
|
||||
queryable current state always recomputable by replay.
|
||||
includes:
|
||||
- append-only decision log, totally ordered per information space
|
||||
- derived fold to current coordination state (aliases, equivalence groups, overlays)
|
||||
- per-space append authority (concurrency model)
|
||||
excludes:
|
||||
- storing derived/disposable union state
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-29, UC-33 (history, attribution, coordination journal)"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/coordination/decision_log.py"
|
||||
target_artifacts:
|
||||
- git-backed log store with per-space lease
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
supports:
|
||||
- capability.wiki.shard-orchestration
|
||||
- capability.wiki.overlay
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 8.1)"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_decision_log.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- durable, replayable, git-addressable coordination state for a federated space
|
||||
not_recommended_for:
|
||||
- high-frequency general event streaming
|
||||
known_limitations:
|
||||
- production git backing + lease are still on the roadmap (SHARD-WP-0009)
|
||||
---
|
||||
|
||||
# Event-Sourced Coordination Journal
|
||||
|
||||
The keystone: coordination-canonical state (overlays, equivalence bindings, aliases, merges,
|
||||
forks) is an append-only **decision log**, totally ordered per information space; the queryable
|
||||
current state is a derived **fold** of the log (`derived = f(log)`). The log is git-addressable,
|
||||
giving history/patch/review/backup for coordination decisions for free.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Resolved across the round-1/round-2 architecture reviews (CoreArchitectureBlueprint Section 8.1).
|
||||
|
||||
### Availability
|
||||
`decision_log.py` ships an in-memory, totally-ordered log + fold; git+lease backing is planned.
|
||||
87
registry/capabilities/capability.wiki.derived-views.md
Normal file
87
registry/capabilities/capability.wiki.derived-views.md
Normal file
@@ -0,0 +1,87 @@
|
||||
---
|
||||
id: capability.wiki.derived-views
|
||||
name: Wiki Derived Views
|
||||
summary: Recomputable views over a wiki union — BackLinks, RecentChanges, AllPages, SiteMap, and (delegate-or-derive) Search — carrying provenance.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, derived-views, backlinks, recentchanges, search, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D3
|
||||
target: D5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Core-vs-adapter classification and behaviours are decided (FederationRequirements ADR-03);
|
||||
implementation is planned (SHARD-WP-0010), not built.
|
||||
availability:
|
||||
current: A0
|
||||
target: A4
|
||||
confidence: low
|
||||
rationale: >
|
||||
Designed; no implementation yet. Informational/planning reuse only today.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C0
|
||||
name: Absent
|
||||
confidence: low
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations: []
|
||||
broken_expectations:
|
||||
- no derived view is implemented yet
|
||||
out_of_scope_expectations:
|
||||
- presentation / rendering of views
|
||||
reliability:
|
||||
level: R0
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- planning-stage
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Provide recomputable, provenance-carrying views over the union (link graph, change feed,
|
||||
enumeration, search) without introducing canonical state.
|
||||
includes:
|
||||
- BackLinks (link graph), RecentChanges (journal + shard signals), AllPages, SiteMap
|
||||
- Search as delegate-to-native-or-derive-index
|
||||
excludes:
|
||||
- view presentation / UI
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-17..UC-21, UC-63"
|
||||
|
||||
availability:
|
||||
current_level: A0
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/workplans/SHARD-WP-0010-derived-views.md"
|
||||
consumption_modes:
|
||||
- informational
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.shard-orchestration
|
||||
- capability.wiki.page-model
|
||||
related_to:
|
||||
- capability.wiki.engine-typed-extensions
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/FederationRequirements.md (ADR-03)"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- planning derived navigation/discovery over a federated wiki union
|
||||
not_recommended_for:
|
||||
- implementation reuse today (planning-stage)
|
||||
known_limitations:
|
||||
- not implemented; Search ranking policy undecided
|
||||
---
|
||||
|
||||
# Wiki Derived Views
|
||||
|
||||
Recomputable views over the union (BackLinks, RecentChanges, AllPages, SiteMap, Search). All
|
||||
are derived/disposable (no canonical state) and carry provenance; Search is delegate-to-native
|
||||
where a shard's query capability allows, else a derived index. Planned in SHARD-WP-0010.
|
||||
115
registry/capabilities/capability.wiki.engine-typed-extensions.md
Normal file
115
registry/capabilities/capability.wiki.engine-typed-extensions.md
Normal file
@@ -0,0 +1,115 @@
|
||||
---
|
||||
id: capability.wiki.engine-typed-extensions
|
||||
name: Wiki Engine with Typed Extensions
|
||||
summary: A small-core wiki engine realizing a stringent typed-extension framework that addresses all wiki use cases and lets each shard activate only the features it needs.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, engine, typed-extensions, feature-activation, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D3
|
||||
target: D5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Architecture authored (shard-wiki/spec/WikiEngineCoreArchitecture.md): small page-store
|
||||
kernel + typed-extension framework, per-shard activation, engine-as-canonical-mode-shard,
|
||||
and a conflict-mediation realization are explored. Detailed extension SDK/ABI and the API
|
||||
protocol remain (so D3 Explored, not yet D4/D5).
|
||||
availability:
|
||||
current: A0
|
||||
target: A4
|
||||
confidence: low
|
||||
rationale: >
|
||||
Planned. No engine kernel or extensions exist yet; informational/planning reuse only.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C0
|
||||
name: Absent
|
||||
confidence: low
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations: []
|
||||
broken_expectations:
|
||||
- engine core and typed-extension mechanism not yet designed in detail
|
||||
out_of_scope_expectations:
|
||||
- replacing other wiki engines or mandating one implementation
|
||||
reliability:
|
||||
level: R0
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- planning-stage capability
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Provide shard-wiki's reference first-party shard backend: a small core + a stringent
|
||||
typed-extension framework covering all collected use cases, mediating conflicting
|
||||
requirements into an integrated whole, with per-shard activation (only what you need).
|
||||
includes:
|
||||
- a minimal engine kernel (page lifecycle, storage via the adapter contract, the typing mechanism)
|
||||
- typed extensions that declare contracts and compose
|
||||
- per-shard feature activation
|
||||
excludes:
|
||||
- replacing or mandating other wiki engines (it is one shard type among many)
|
||||
- a single canonical implementation for all wikis
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-08..UC-25 and the full catalog (the engine must cover all)"
|
||||
|
||||
availability:
|
||||
current_level: A0
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/workplans/SHARD-WP-0013-wiki-engine-prep.md"
|
||||
- "shard-wiki/spec/WikiEngineCoreArchitecture.md"
|
||||
consumption_modes:
|
||||
- informational
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.adapter-contract
|
||||
- capability.wiki.page-model
|
||||
related_to:
|
||||
- capability.feature-control.evaluate
|
||||
- capability.authorization.policy-evaluate
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/workplans/SHARD-WP-0013-wiki-engine-prep.md"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- planning a composable, feature-activatable native wiki engine
|
||||
not_recommended_for:
|
||||
- implementation reuse today (planning-stage)
|
||||
known_limitations:
|
||||
- architecture authored; extension SDK/ABI + API protocol still to design; not yet built
|
||||
|
||||
promotion_history:
|
||||
- date: "2026-06-15"
|
||||
dimension: discovery
|
||||
from: D2
|
||||
to: D3
|
||||
rationale: WikiEngineCoreArchitecture.md authored (kernel + typed-extension framework explored); INTENT amendment ratified.
|
||||
author: shard-wiki
|
||||
---
|
||||
|
||||
# Wiki Engine with Typed Extensions
|
||||
|
||||
shard-wiki's planned reference first-party shard backend — a *canonical-mode shard* it
|
||||
implements natively: a small core plus a stringent typed-extension framework addressing all
|
||||
collected use cases, mediating conflicting requirements into a consistent whole, with per-shard
|
||||
activation (activate only what you need). It is one shard type among many — not a replacement
|
||||
for other engines. Per-shard activation is a candidate consumer of
|
||||
`capability.feature-control.evaluate`.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Architecture authored: `shard-wiki/spec/WikiEngineCoreArchitecture.md` (small kernel +
|
||||
typed-extension framework; engine = canonical-mode shard). INTENT amendment ratified
|
||||
(2026-06-15, decision 84ffdb48). Extension SDK/ABI + API protocol are the next deliverables.
|
||||
|
||||
### Availability
|
||||
Planning-stage; informational reuse only.
|
||||
97
registry/capabilities/capability.wiki.federation-models.md
Normal file
97
registry/capabilities/capability.wiki.federation-models.md
Normal file
@@ -0,0 +1,97 @@
|
||||
---
|
||||
id: capability.wiki.federation-models
|
||||
name: Selectable Federation-Model Taxonomy
|
||||
summary: Federation as a plural, composable coordination axis (fork+journal, VCS-replication+ping, query-time graph-join, feed, activity-streams, engine-mirror) selected per space.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, federation, taxonomy, composable, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D4
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
A six-model taxonomy distilled from a ~23-system synthesis, each model anchored in a
|
||||
real system, with capability prerequisites and per-space/per-shard composition rules.
|
||||
availability:
|
||||
current: A0
|
||||
target: A4
|
||||
confidence: low
|
||||
rationale: >
|
||||
Designed and specified (FederationArchitecture T17) but not implemented; informational
|
||||
reuse only today.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C1
|
||||
name: Sparse
|
||||
confidence: low
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- the model taxonomy and selection/composition rules are documented
|
||||
broken_expectations:
|
||||
- no federation transport is implemented yet
|
||||
out_of_scope_expectations:
|
||||
- mandating a single federation mechanism
|
||||
reliability:
|
||||
level: R0
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- design-stage; no runtime evidence
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Treat federation as selectable and composable rather than one mechanism, so each space
|
||||
picks fork+journal, VCS-replication, query-join, feed, activity-streams, or engine-mirror.
|
||||
includes:
|
||||
- the six federation models + their capability floors
|
||||
- per-space selection and per-shard composition
|
||||
excludes:
|
||||
- imposing one homogeneous federation network
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-26, UC-31, UC-33, UC-71, UC-72, UC-74, UC-79"
|
||||
|
||||
availability:
|
||||
current_level: A0
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/spec/FederationArchitecture.md (T17)"
|
||||
consumption_modes:
|
||||
- informational
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.shard-orchestration
|
||||
- capability.wiki.coordination-journal
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/FederationArchitecture.md"
|
||||
- "shard-wiki/research/260614-shard-spectrum-synthesis/findings.md"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- planning a federation strategy that mixes models per source
|
||||
not_recommended_for:
|
||||
- implementation reuse today (design-stage)
|
||||
known_limitations:
|
||||
- no transport implemented; informational planning reuse only
|
||||
---
|
||||
|
||||
# Selectable Federation-Model Taxonomy
|
||||
|
||||
Federation is plural and composable: fork+journal (Federated Wiki), VCS-replication+ping
|
||||
(ikiwiki), query-time graph-join (Wikibase SERVICE), feed aggregation, activity streams
|
||||
(ActivityPub), and engine-mirror (Wiki.js). A space selects a model and composes per shard;
|
||||
the default is fork+journal over git. Design-stage capability — strong for planning reuse.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
FederationArchitecture T17, distilled from the shard-spectrum synthesis (v3).
|
||||
|
||||
### Availability
|
||||
Specified, not implemented — informational reuse only.
|
||||
102
registry/capabilities/capability.wiki.overlay.md
Normal file
102
registry/capabilities/capability.wiki.overlay.md
Normal file
@@ -0,0 +1,102 @@
|
||||
---
|
||||
id: capability.wiki.overlay
|
||||
name: Overlay-Before-Mutation Write Path
|
||||
summary: Non-destructive edits (draft -> patch -> apply-under-drift) that let read-only, rate-limited, or lossy backends be edited safely without silent remote mutation.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, overlay, patch, write-path, conflict, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Overlay lifecycle and apply-under-drift semantics are specified (ADR-05, blueprint
|
||||
Section 8.6) and implemented as a single principled write path.
|
||||
availability:
|
||||
current: A2
|
||||
target: A4
|
||||
confidence: medium
|
||||
rationale: >
|
||||
OverlayEngine (draft/patch/apply), writable adapter, and InformationSpace.edit
|
||||
exist as a source module; three-way merge is not (refuse-on-drift only).
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- draft -> patch -> apply with fast-forward / refuse-on-drift / keep-draft outcomes
|
||||
- no silent remote mutation; overlay_state surfaced in provenance
|
||||
broken_expectations:
|
||||
- three-way / auto merge not implemented (refuse-on-conflict only)
|
||||
out_of_scope_expectations:
|
||||
- federation propagation of applied overlays
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- early implementation; conflict handling is detect-and-refuse only
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Make any sub-write-through backend editable safely: an edit is an overlay first, applied
|
||||
only on explicit intent and only when the source has not drifted.
|
||||
includes:
|
||||
- overlay drafts recorded as coordination-canonical events
|
||||
- patch rendering (unified diff)
|
||||
- apply-under-drift (fast-forward / refuse / keep-draft)
|
||||
excludes:
|
||||
- destructive write without drift check
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-04, UC-26, UC-29 (remix primitives, overlay)"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A4
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/coordination/overlay.py"
|
||||
- "shard-wiki/src/shard_wiki/coordination/patch.py"
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.coordination-journal
|
||||
- capability.wiki.adapter-contract
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/FederationRequirements.md (ADR-05)"
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 8.2, 8.6)"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_apply.py"
|
||||
- "shard-wiki/tests/test_write_path_integration.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- safe editing over read-only / rate-limited / lossy backends
|
||||
not_recommended_for:
|
||||
- workflows needing automatic conflict resolution today
|
||||
known_limitations:
|
||||
- merge is detect-and-refuse; three-way merge is future work
|
||||
---
|
||||
|
||||
# Overlay-Before-Mutation Write Path
|
||||
|
||||
One principled write path: every edit drafts an overlay (a coordination-canonical event),
|
||||
renders as a patch, and applies under drift checks — fast-forwarding a writable target,
|
||||
keeping a local draft on a read-only target, and refusing (never clobbering) on external drift.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Specified in FederationRequirements ADR-05 and CoreArchitectureBlueprint Section 8.2/8.6.
|
||||
|
||||
### Availability
|
||||
`overlay.py` + `patch.py` + `InformationSpace.edit` ship the path; built in SHARD-WP-0008.
|
||||
104
registry/capabilities/capability.wiki.page-model.md
Normal file
104
registry/capabilities/capability.wiki.page-model.md
Normal file
@@ -0,0 +1,104 @@
|
||||
---
|
||||
id: capability.wiki.page-model
|
||||
name: Backend-Neutral Wiki Page Model
|
||||
summary: A Markdown-first but stretchable page model with stable identity separate from placement and layered provenance, spanning prose to typed-graph and computational shapes.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, page-model, identity, provenance, markdown, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Page shapes (prose, typed records, typed-graph, inline-embedded, non-Markdown,
|
||||
and four computational shapes) plus identity != placement and layered provenance
|
||||
are specified and grounded in the dive research.
|
||||
availability:
|
||||
current: A2
|
||||
target: A5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Identity/Placement/Span/Page and layered ProvenanceEnvelope exist as a source
|
||||
module; richer shapes (typed-graph, notebook) are modeled but not all built.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- stable identity distinct from placement and from content fingerprint
|
||||
- layered (effective-vs-own) provenance with near-zero per-span cost
|
||||
broken_expectations:
|
||||
- non-prose shapes (typed-graph, notebook, inline-embedded) not fully realized
|
||||
out_of_scope_expectations:
|
||||
- rendering / presentation
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- prose shape is the only exercised path so far
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
One backend-neutral lingua franca every consumer sees; every shape reduces to
|
||||
(content|source, structure, provenance envelope, optional derivation rule).
|
||||
includes:
|
||||
- page identity (stable handle) vs placement (N paths/shards) vs equivalence (fingerprint)
|
||||
- layered provenance envelope (page + span deltas)
|
||||
- page-shape taxonomy incl. computational shapes
|
||||
excludes:
|
||||
- deriving identity from content (a fingerprint identifies a version, not a page)
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-34, UC-39, UC-44..UC-49, UC-55, UC-73, UC-83, UC-84"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A5
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/model/"
|
||||
- "shard-wiki/src/shard_wiki/provenance/"
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
supports:
|
||||
- capability.wiki.adapter-contract
|
||||
- capability.wiki.shard-orchestration
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 7)"
|
||||
- "shard-wiki/spec/FederationRequirements.md (ADR-02, ADR-04)"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_model.py"
|
||||
- "shard-wiki/tests/test_provenance.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- a portable, provenance-carrying representation of wiki pages across backends
|
||||
not_recommended_for:
|
||||
- cases needing a single canonical path per page (use identity, not path)
|
||||
known_limitations:
|
||||
- non-prose shapes specified ahead of implementation
|
||||
---
|
||||
|
||||
# Backend-Neutral Wiki Page Model
|
||||
|
||||
The top narrow waist: a Markdown-first model that stretches to typed records, typed-graph
|
||||
statements, inline-embedded objects, non-Markdown assets, and computational shapes. Identity
|
||||
is a stable handle; placement and equivalence are separate mechanisms; provenance is layered
|
||||
(effective = page envelope + span delta).
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Specified in CoreArchitectureBlueprint Section 7 and FederationRequirements ADR-02/04.
|
||||
|
||||
### Availability
|
||||
`model/` + `provenance/` ship the prose path and the layered envelope today.
|
||||
114
registry/capabilities/capability.wiki.shard-orchestration.md
Normal file
114
registry/capabilities/capability.wiki.shard-orchestration.md
Normal file
@@ -0,0 +1,114 @@
|
||||
---
|
||||
id: capability.wiki.shard-orchestration
|
||||
name: Wiki Shard Orchestration
|
||||
summary: Present a union of pages across heterogeneous wiki-shaped shards while preserving each shard's provenance, capabilities, and history.
|
||||
owner: shard-wiki
|
||||
status: draft
|
||||
domain: helix_forge
|
||||
tags: [wiki, federation, orchestration, union, shard-wiki]
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D5
|
||||
target: D6
|
||||
confidence: high
|
||||
rationale: >
|
||||
Grounded in 84 documented use cases and a twice-reviewed whole-system
|
||||
architecture (CoreArchitectureBlueprint) derived from ~23 prior-art systems.
|
||||
availability:
|
||||
current: A2
|
||||
target: A5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
InformationSpace orchestrator (attach -> resolve -> read, chorus on
|
||||
ambiguity) works as a Python source module; network API and incremental
|
||||
union are planned.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C2
|
||||
name: Partial
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- attach folder shards and read a union page with layered provenance
|
||||
- chorus presentation of equivalent-but-divergent pages (union without erasure)
|
||||
broken_expectations:
|
||||
- incremental union maintenance and equivalence index not yet built
|
||||
- write-through federation transports not yet built
|
||||
out_of_scope_expectations:
|
||||
- hosting or replacing the underlying wiki engines
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: low
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- early implementation; 64 tests but no production exposure
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Let independently stored, differently implemented wikis behave as one
|
||||
coherent, versionable, inspectable information space without homogenizing them.
|
||||
includes:
|
||||
- union resolution across shards (identity-keyed)
|
||||
- chorus / designated-canonical presentation of equivalent pages
|
||||
- lazy replication projection of remote content with freshness
|
||||
excludes:
|
||||
- implementing a backend wiki engine (see capability.wiki.engine-typed-extensions)
|
||||
- silent remote mutation
|
||||
assumptions:
|
||||
- canonical truth lives in shards + a git coordination journal; the union is derived
|
||||
use_cases:
|
||||
- "shard-wiki UseCaseCatalog UC-01..UC-07, UC-26..UC-33 (information space, federation, coordination)"
|
||||
|
||||
availability:
|
||||
current_level: A2
|
||||
target_level: A5
|
||||
current_artifacts:
|
||||
- "shard-wiki/src/shard_wiki/union/"
|
||||
- "shard-wiki/src/shard_wiki/space.py"
|
||||
target_artifacts:
|
||||
- orchestrator network API
|
||||
consumption_modes:
|
||||
- source module
|
||||
|
||||
relations:
|
||||
depends_on:
|
||||
- capability.wiki.adapter-contract
|
||||
- capability.wiki.page-model
|
||||
- capability.wiki.coordination-journal
|
||||
supports:
|
||||
- capability.wiki.federation-models
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- "shard-wiki/spec/CoreArchitectureBlueprint.md"
|
||||
- "shard-wiki/spec/FederationArchitecture.md"
|
||||
tests:
|
||||
- "shard-wiki/tests/test_union.py"
|
||||
- "shard-wiki/tests/test_integration.py"
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- composing multiple Markdown/wiki stores into one provenance-preserving view
|
||||
not_recommended_for:
|
||||
- replacing a single wiki engine
|
||||
known_limitations:
|
||||
- resolution is recompute-on-read until the incremental tier lands
|
||||
---
|
||||
|
||||
# Wiki Shard Orchestration
|
||||
|
||||
shard-wiki's core capability: orchestrate wiki-shaped content across heterogeneous *shards*
|
||||
as a union of pages, preserving provenance, capabilities, and history per shard. Canonical
|
||||
truth stays at the edges (shards + the git coordination journal); the union is a derived,
|
||||
recomputable view (orchestrator, not engine).
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
Grounded by `UseCaseCatalog.md` (84 UCs) and the hardened `CoreArchitectureBlueprint.md`.
|
||||
|
||||
### Availability
|
||||
`InformationSpace` provides attach/resolve/read today (source module); a network API is the
|
||||
target availability step.
|
||||
145
registry/indexes/capabilities.yaml
Normal file
145
registry/indexes/capabilities.yaml
Normal file
@@ -0,0 +1,145 @@
|
||||
version: 1
|
||||
updated: '2026-06-16'
|
||||
domain: helix_forge
|
||||
capabilities:
|
||||
- id: capability.wiki.shard-orchestration
|
||||
name: Wiki Shard Orchestration
|
||||
summary: Present a union of pages across heterogeneous wiki-shaped shards while
|
||||
preserving each shard's provenance, capabilities, and history.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.shard-orchestration.md
|
||||
tags:
|
||||
- wiki
|
||||
- federation
|
||||
- orchestration
|
||||
- union
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.adapter-contract
|
||||
name: Capability-Aware Shard Adapter Contract
|
||||
summary: A versioned backend interface where each binding declares a verified capability
|
||||
profile, so federation ops degrade by capability.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.adapter-contract.md
|
||||
tags:
|
||||
- wiki
|
||||
- adapter
|
||||
- capability
|
||||
- contract
|
||||
- conformance
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.page-model
|
||||
name: Backend-Neutral Wiki Page Model
|
||||
summary: A Markdown-first but stretchable page model with stable identity separate
|
||||
from placement and layered provenance.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.page-model.md
|
||||
tags:
|
||||
- wiki
|
||||
- page-model
|
||||
- identity
|
||||
- provenance
|
||||
- markdown
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.coordination-journal
|
||||
name: Event-Sourced Coordination Journal
|
||||
summary: An append-only, totally-ordered-per-space decision log whose current state
|
||||
is a derived fold; git-addressable history.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.coordination-journal.md
|
||||
tags:
|
||||
- wiki
|
||||
- event-sourcing
|
||||
- coordination
|
||||
- git
|
||||
- journal
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.overlay
|
||||
name: Overlay-Before-Mutation Write Path
|
||||
summary: Non-destructive edits (draft -> patch -> apply-under-drift) that let read-only
|
||||
or limited backends be edited safely without silent remote mutation.
|
||||
vector: D5 / A2 / C2 / R1
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.overlay.md
|
||||
tags:
|
||||
- wiki
|
||||
- overlay
|
||||
- patch
|
||||
- write-path
|
||||
- conflict
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- source module
|
||||
- id: capability.wiki.federation-models
|
||||
name: Selectable Federation-Model Taxonomy
|
||||
summary: Federation as a plural, composable coordination axis (fork+journal, VCS-replication,
|
||||
query-join, feed, activity-streams, engine-mirror) selected per space.
|
||||
vector: D4 / A0 / C1 / R0
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.federation-models.md
|
||||
tags:
|
||||
- wiki
|
||||
- federation
|
||||
- taxonomy
|
||||
- composable
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- informational
|
||||
- id: capability.wiki.engine-typed-extensions
|
||||
name: Wiki Engine with Typed Extensions
|
||||
summary: A small-core wiki engine realizing a typed-extension framework that addresses
|
||||
all wiki use cases and lets each shard activate only the features it needs.
|
||||
vector: D3 / A0 / C0 / R0
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.engine-typed-extensions.md
|
||||
tags:
|
||||
- wiki
|
||||
- engine
|
||||
- typed-extensions
|
||||
- feature-activation
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- informational
|
||||
- id: capability.wiki.derived-views
|
||||
name: Wiki Derived Views
|
||||
summary: Recomputable views over a wiki union — BackLinks, RecentChanges, AllPages,
|
||||
SiteMap, and (delegate-or-derive) Search — carrying provenance.
|
||||
vector: D3 / A0 / C0 / R0
|
||||
domain: helix_forge
|
||||
status: draft
|
||||
owner: shard-wiki
|
||||
path: registry/capabilities/capability.wiki.derived-views.md
|
||||
tags:
|
||||
- wiki
|
||||
- derived-views
|
||||
- backlinks
|
||||
- recentchanges
|
||||
- search
|
||||
- shard-wiki
|
||||
consumption_modes:
|
||||
- informational
|
||||
36
research/260613-foswiki-deep-dive/README.md
Normal file
36
research/260613-foswiki-deep-dive/README.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# 260613 — Foswiki deep dive (store abstraction, extension API, ecosystem)
|
||||
|
||||
Date: 2026-06-13
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Foswiki** — the community fork of TWiki (2008, from TWiki
|
||||
4.2.x) — concentrating on **what Foswiki changed**, not the lineage it shares with
|
||||
TWiki. The distinctive material: a **pluggable `Foswiki::Store` backend**
|
||||
(RcsWrap / RcsLite / **PlainFileStoreContrib**) behind a versioned interface, the
|
||||
`Foswiki::Meta` topic-object indirection, the OO/MVC core rewrite, the cleaner
|
||||
`Foswiki::Func` + `registerTagHandler` extension API, **DataForms** (+ MetaDataPlugin,
|
||||
FlexFormPlugin), and the **WysiwygPlugin** TML↔HTML round-trip.
|
||||
|
||||
Read through shard-wiki's lens: Foswiki is the engine that **already separates its
|
||||
store backend from its core via a versioned interface** — i.e. it is concrete prior
|
||||
art for shard-wiki's own *shard adapter contract* and *capability-aware adapters*.
|
||||
|
||||
Pairs with — and deliberately does not repeat — the shared TWiki material:
|
||||
- `research/260613-twiki-deep-dive/` — flat-file/RCS store, Webs/Topics, plugin
|
||||
handler callbacks, per-topic ACL (all inherited; read there for the shared parts)
|
||||
- `research/260613-xwiki-deep-dive/` — the Java/DB app-platform sibling
|
||||
- `research/260608-wikiengines-overview/` — landscape scan
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Distinctives vs TWiki, store abstraction, extension API, DataForms, capability profile, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to
|
||||
`spec/UseCaseCatalog.md` (UC-42, UC-43); UC-34/39/40 enriched. The
|
||||
`Foswiki::Store`/`Foswiki::Meta` abstraction is logged as adapter-contract prior art
|
||||
for `workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
188
research/260613-foswiki-deep-dive/findings.md
Normal file
188
research/260613-foswiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,188 @@
|
||||
# Findings — Foswiki: store abstraction, extension API, ecosystem
|
||||
|
||||
Date: 2026-06-13 · Status: research draft
|
||||
|
||||
Scope: **Foswiki as distinct from TWiki** — only the deltas, since the two share a
|
||||
data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to
|
||||
shard-wiki: a **pluggable store backend behind a versioned interface**
|
||||
(`Foswiki::Store` + `Foswiki::Meta`), an OO/MVC core rewrite, a cleaner extension
|
||||
API (`Foswiki::Func::registerTagHandler`), enhanced DataForms, and a TML↔HTML
|
||||
round-trip editor framework. Sources: foswiki.org Development + System docs,
|
||||
`foswiki/distro` on GitHub.
|
||||
|
||||
**Read first for shared lineage:** `research/260613-twiki-deep-dive/` (flat-file +
|
||||
RCS store, Webs/Topics, handler callbacks, per-topic `ALLOW/DENY` ACL). This file
|
||||
does **not** repeat those.
|
||||
|
||||
---
|
||||
|
||||
## 1. Origin and compatibility
|
||||
|
||||
Forked from **TWiki 4.2.x** in October 2008 by most of the TWiki community (sources
|
||||
cite 4.2.3–4.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content
|
||||
compatibility — same TML markup, same plugin API — via a **`TWikiCompatibilityPlugin`**
|
||||
that maps the `TWiki::` namespace and legacy variables. So everything in the TWiki
|
||||
dive still applies; Foswiki then grew the features below.
|
||||
|
||||
## 2. The headline delta — a pluggable store behind a versioned interface
|
||||
|
||||
This is the reason Foswiki earns its own dive.
|
||||
|
||||
### 2.1 `Foswiki::Meta` indirection
|
||||
After Foswiki 1.0.4 the core was changed to **delegate almost all store operations
|
||||
to a topic object, `Foswiki::Meta`**. Callers manipulate a `Foswiki::Meta` object
|
||||
representing the item they're changing and **never touch the store implementation
|
||||
directly**; `Foswiki::Meta` talks to the real store through the well-defined
|
||||
**`Foswiki::Store`** interface. This is a clean MVC/repository separation.
|
||||
|
||||
### 2.2 Swappable backends
|
||||
`Foswiki::Store::` ships multiple low-level back-ends behind that one interface:
|
||||
- **`RcsWrap`** — RCS via the external `rcs` binaries (classic TWiki behavior)
|
||||
- **`RcsLite`** — a pure-Perl RCS implementation
|
||||
- **`PlainFileStoreContrib`** — a newer store that saves topics/attachments as
|
||||
**timestamped copies** instead of RCS diffs: more disk, but **no RCS dependency**
|
||||
and much higher performance.
|
||||
|
||||
So in Foswiki the **storage format is a configuration choice**, not a fixed property
|
||||
of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core.
|
||||
|
||||
### 2.3 OO/MVC core rewrite
|
||||
Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward
|
||||
object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving
|
||||
backward compatibility through tests. The store abstraction above is the most
|
||||
load-bearing result.
|
||||
|
||||
## 3. Extension API deltas
|
||||
|
||||
Shared with TWiki: handler callbacks (`initPlugin`, `commonTagsHandler`,
|
||||
`before/afterSaveHandler`, `afterRenameHandler`, attachment handlers), REST handlers,
|
||||
`Config.spec`, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements:
|
||||
|
||||
- **`Foswiki::Func` is the blessed API** — "if there's a `Foswiki::Func` way and
|
||||
another way, the `Foswiki::Func` way is almost always right." Direct use of
|
||||
internal packages is discouraged and governed by **`PluginsApiPolicies`**.
|
||||
- **`Foswiki::Func::registerTagHandler($tag, \&sub)`** — register a custom
|
||||
macro/variable **programmatically** (cleaner than TWiki's `commonTagsHandler`
|
||||
string-matching convention).
|
||||
- **Contrib-defined APIs** carry an explicit "use at your own risk / awaiting merge"
|
||||
policy — a maturity signal on extension interfaces.
|
||||
- `EmptyPlugin` (`lib/Foswiki/Plugins/EmptyPlugin.pm`) is the canonical handler
|
||||
reference/skeleton.
|
||||
|
||||
## 4. DataForms and structured data (delta)
|
||||
|
||||
DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi
|
||||
value, label) stored as `%META:FIELD%` in the topic text. Foswiki extends the model
|
||||
via popular extensions:
|
||||
- **MoreFormfieldsPlugin** — additional special-purpose field types.
|
||||
- **FlexFormPlugin** — render DataForm interfaces from custom templates
|
||||
(`Foswiki::Form` classes).
|
||||
- **MetaDataPlugin** — **multiple structured data records per topic** (beyond the
|
||||
classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model
|
||||
while keeping data in the text file.
|
||||
|
||||
## 5. Editing / syntax (delta relevant to "Markdown-first")
|
||||
|
||||
- **WysiwygPlugin** — a generic framework that transforms **TML → HTML** for a
|
||||
browser editor and **HTML → TML on save** (lossless round-trip).
|
||||
- **TinyMCEPlugin** — pre-installed WYSIWYG editor built on WysiwygPlugin.
|
||||
|
||||
This proves **bidirectional, lossless translation between an engine's native markup
|
||||
and another representation is feasible** — directly relevant to shard-wiki reading a
|
||||
non-Markdown shard and writing Markdown overlays back (UC-42).
|
||||
|
||||
## 6. Foswiki as a shard — capability profile (delta from TWiki)
|
||||
|
||||
| Capability | Foswiki | Note vs TWiki |
|
||||
|------------|---------|---------------|
|
||||
| Store backend | **pluggable** | RCS *or* PlainFile behind `Foswiki::Store` — direct-attach target is cleaner on PlainFile (UC-40) |
|
||||
| History | RCS *or* timestamped copies | PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41) |
|
||||
| Structured payload | DataForms `%META%`, **multi-record** via MetaDataPlugin | richer than classic TWiki Forms (UC-34/39) |
|
||||
| Syntax translation | **WysiwygPlugin TML↔HTML** | bidirectional round-trip exists (UC-42) |
|
||||
| Extension host | `Foswiki::Func` + `registerTagHandler` + REST | cleaner adapter-host surface (UC-38) |
|
||||
| Store as contract | **`Foswiki::Store` versioned interface** | architectural prior art for shard-wiki's adapter contract |
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
| Observation | INTENT principle |
|
||||
|-------------|------------------|
|
||||
| `Foswiki::Store` versioned interface + swappable backends | **capability-aware adapters** / **shard adapter contract** — an engine that already did exactly this separation |
|
||||
| `Foswiki::Meta` mediates all store access | clean **mechanism over policy** boundary worth mirroring in the adapter contract |
|
||||
| PlainFile store = timestamped text copies | **git-addressable coordination** / direct-attach (UC-40), history import (UC-41) |
|
||||
| WysiwygPlugin TML↔HTML round-trip | **Markdown-first, backend-neutral** is achievable for prose via translation (UC-42) |
|
||||
| MetaDataPlugin multi-record topics | structured pages beyond one form (UC-34/39) |
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
| Foswiki assumption | shard-wiki correction |
|
||||
|--------------------|----------------------|
|
||||
| One pluggable store per wiki, chosen at config | shard-wiki federates **many** heterogeneous stores at once |
|
||||
| TML is the canonical syntax; HTML is a render target | shard-wiki is **Markdown-first**; TML↔Markdown is an adapter translation, not core |
|
||||
| Store interface is Foswiki-internal Perl | shard-wiki's adapter contract is **cross-language, cross-backend, versioned** |
|
||||
| ACL in topic preferences | **authorize in core**; engine ACL is advisory provenance |
|
||||
|
||||
### 7.3 What Foswiki teaches that shard-wiki should not lose
|
||||
1. **The store-abstraction pattern works in practice** — `Foswiki::Store` is a
|
||||
real-world proof that a stable interface + swappable backends is viable; the shard
|
||||
adapter contract is the same idea generalized across engines.
|
||||
2. **Backend format should be swappable under a stable identity** — Foswiki migrates
|
||||
RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's
|
||||
backend changing under it (UC-43).
|
||||
3. **Lossless syntax round-trip is a solved problem** — don't treat non-Markdown
|
||||
prose as read-only; translate it (UC-42).
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-13)
|
||||
|
||||
| Seed | Catalog UC | Disposition |
|
||||
|------|------------|-------------|
|
||||
| Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown) | **UC-42** (new) | realizes Markdown-first for prose; WysiwygPlugin is proof |
|
||||
| Tolerate a shard's storage-backend swap without losing identity/provenance | **UC-43** (new) | Foswiki RCS↔PlainFile; orchestration robustness |
|
||||
| Structured / multi-record pages | UC-34 / UC-39 | enriched: DataForms + MetaDataPlugin multi-record |
|
||||
| Attach a file-backed engine's on-disk store directly | UC-40 | enriched: PlainFile store is a cleaner direct-attach target than RCS |
|
||||
| `Foswiki::Store` versioned interface | (no UC) | architecture prior art → `SHARD-WP-0002` adapter contract |
|
||||
|
||||
## 9. Open questions (for spec / workplans)
|
||||
|
||||
1. **Adapter-contract shape** — how much of `Foswiki::Store`'s method set
|
||||
(read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's
|
||||
capability-aware adapter contract? (architecture, `SHARD-WP-0002`)
|
||||
2. **Syntax-translation fidelity** — is TML↔Markdown lossless enough for overlay
|
||||
round-trips, or must overlays be stored in TML to be safe (UC-42)?
|
||||
3. **Backend-swap detection** — how does shard-wiki notice and tolerate a shard
|
||||
changing store format underneath (RCS→PlainFile), and does provenance/history
|
||||
survive it (UC-43)?
|
||||
4. **Multi-record metadata** — represent MetaDataPlugin multi-record topics and XWiki
|
||||
multi-XObject pages in one structured-metadata page model (shared open question).
|
||||
|
||||
## 10. Sources
|
||||
|
||||
| Source | URL |
|
||||
|--------|-----|
|
||||
| Foswiki — Technical Overview | https://foswiki.org/Development/TechnicalOverview |
|
||||
| Foswiki — Core Internals | https://foswiki.org/Development/CoreInternals |
|
||||
| Foswiki — PlainFileStoreContrib | https://foswiki.org/Extensions/PlainFileStoreContrib |
|
||||
| Foswiki — RCSStoreContrib | https://foswiki.org/Extensions/RCSStoreContrib |
|
||||
| Foswiki::Store::PlainFile (source) | https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm |
|
||||
| Foswiki — Developing Plugins | https://foswiki.org/System/DevelopingPlugins |
|
||||
| Foswiki — Plugins API Policies | https://foswiki.org/Development/PluginsApiPolicies |
|
||||
| Foswiki — DataForms | https://foswiki.org/System/DataForms |
|
||||
| Foswiki — MetaDataPlugin | https://foswiki.org/Extensions/MetaDataPlugin |
|
||||
| Foswiki — WysiwygPlugin | https://foswiki.org/System/WysiwygPlugin |
|
||||
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
|
||||
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
|
||||
| shard-wiki — TWiki deep dive | `research/260613-twiki-deep-dive/findings.md` |
|
||||
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
|
||||
|
||||
---
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
| This document section | Informs (future) |
|
||||
|-----------------------|------------------|
|
||||
| §2 store abstraction | **adapter contract** design (`SHARD-WP-0002`) — Foswiki::Store as prior art |
|
||||
| §3 extension API | UC-38 engine-side adapter host |
|
||||
| §4 DataForms | structured-metadata page model (UC-34/39) |
|
||||
| §5 WysiwygPlugin | UC-42 syntax-translation adapter |
|
||||
| §6 capability profile | adapter capability-profile vocabulary |
|
||||
| §7 INTENT mapping | architecture-blueprint guardrails |
|
||||
| §8 UC seeds | `spec/UseCaseCatalog.md` (UC-42, UC-43; UC-34/39/40 enrichment) |
|
||||
39
research/260613-twiki-deep-dive/README.md
Normal file
39
research/260613-twiki-deep-dive/README.md
Normal file
@@ -0,0 +1,39 @@
|
||||
# 260613 — TWiki deep dive (implementation, plugin API, ecosystem)
|
||||
|
||||
Date: 2026-06-13
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **TWiki** — the original "structured enterprise wiki / wiki
|
||||
application platform" (Peter Thoeny, ~1998), and the **file-based Perl counterpoint**
|
||||
to XWiki's Java/DB platform. Covers TWiki's **implementation** (Perl + flat-file +
|
||||
RCS store, Webs/Topics, TWiki Forms, TWikiML/variables), the **plugin API and
|
||||
handler callbacks** that extend the core, the **extension package types** (Plugin /
|
||||
Skin / AddOn / Contrib) and repository, and TWiki's **per-topic access control**
|
||||
(the prior art behind yawex's `AccessControl`).
|
||||
|
||||
Read through shard-wiki's lens, paired with the XWiki dive: TWiki shows the same
|
||||
"wiki-as-application-platform" goal reached with **text files + RCS** instead of a
|
||||
database — which changes the adapter story (direct on-disk attach, git-convertible
|
||||
history).
|
||||
|
||||
Complements:
|
||||
- `research/260613-xwiki-deep-dive/` — the DB/Java app-platform sibling
|
||||
- `research/260608-wikiengines-overview/` — the landscape scan
|
||||
- `research/260608-yawex-prior-art/` — yawex borrowed TWiki's per-topic AccessControl
|
||||
|
||||
Note: most of TWiki's community forked to **Foswiki** in Oct 2008 (from TWiki
|
||||
4.2.4); findings flag where TWiki/Foswiki differ.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, plugin-handler API, ecosystem, capability profile, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to
|
||||
`spec/UseCaseCatalog.md` (UC-40, UC-41); UC-06/34/36/38/39 enriched with TWiki
|
||||
specifics. Adapter-contract implications feed
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
210
research/260613-twiki-deep-dive/findings.md
Normal file
210
research/260613-twiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,210 @@
|
||||
# Findings — TWiki: implementation, plugin API, ecosystem
|
||||
|
||||
Date: 2026-06-13 · Status: research draft
|
||||
|
||||
Scope: TWiki as prior art for the **file-based** wiki-application-platform — the
|
||||
counterpoint to XWiki's database/component platform. Two shard-wiki concerns:
|
||||
(1) attaching a structured, file-backed engine as a shard (including attaching its
|
||||
**on-disk store directly** vs through its runtime), and (2) how TWiki's plugin
|
||||
handler API exposes hooks rich enough to host a federation adapter
|
||||
(INTENT *composable integration*). Sources: twiki.org dev docs (TWikiPlugins,
|
||||
TWikiForms, TWiki::Func, TWikiAccessControl), Foswiki docs, Wikipedia.
|
||||
|
||||
**Complements:** `research/260613-xwiki-deep-dive/` (DB/Java sibling),
|
||||
`research/260608-wikiengines-overview/` (landscape), and
|
||||
`research/260608-yawex-prior-art/` (yawex borrowed TWiki's per-topic AccessControl).
|
||||
|
||||
---
|
||||
|
||||
## 1. What TWiki is
|
||||
|
||||
A **structured enterprise wiki** and **wiki application platform** (Peter Thoeny,
|
||||
~1998), written in **Perl 5** (5.10.1+), classically run as CGI. Like XWiki it lets
|
||||
pages become forms/records/apps — but it reaches that with **flat text files +
|
||||
RCS**, no database. Most of the community forked to **Foswiki** in October 2008
|
||||
(from TWiki 4.2.4; Foswiki 1.0 in January 2009); the plugin API and data model below
|
||||
are shared lineage, with minor divergences noted.
|
||||
|
||||
## 2. Implementation architecture
|
||||
|
||||
### 2.1 Storage — flat files + RCS (no DB)
|
||||
- Content lives on disk as `data/<Web>/<Topic>.txt`; the directory tree **mirrors
|
||||
the logical Web/Topic hierarchy**. Attachments live under `pub/<Web>/<Topic>/`.
|
||||
- **History is RCS** (`<Topic>.txt,v` companion files) — the same GNU RCS yawex used.
|
||||
History is real, per-file, and in an **open, git-convertible format** (unlike
|
||||
XWiki's DB-internal `xwikircs`).
|
||||
- Scales to 300k+ topics on a single server (Yahoo) — file store is not a toy.
|
||||
|
||||
### 2.2 Webs and Topics
|
||||
- **Topic** = page (the unit of content). **Web** = namespace / collection of topics
|
||||
(can be nested). Webs are the natural shard / root-entity boundary.
|
||||
|
||||
### 2.3 TWiki Forms — structured data, embedded in text
|
||||
- A **form template** is a topic defining fields as a table (one row per field:
|
||||
name, type, size, values). A topic can *attach* a form; the field values are
|
||||
stored as **TWikiMetaData inside the topic `.txt`** (`%META:FIELD{...}%`), shown as
|
||||
a table on view and edited via fields/radios/checkboxes/lists.
|
||||
- **Forms + formatted `%SEARCH%`** = TWiki's "database applications" — structured
|
||||
records queried by the SEARCH variable. This is the XWiki XObject/XClass idea, but
|
||||
**file-embedded and git-diffable** rather than rows in a DB.
|
||||
|
||||
### 2.4 TWikiML / variables (macros)
|
||||
- Markup is **TWikiML** (not Markdown). Dynamic behavior comes from **variables**
|
||||
`%VAR%` / `%VAR{...}%` expanded at render time; `%SEARCH%` is the query engine.
|
||||
- Custom variables are provided by plugins (see §3).
|
||||
|
||||
### 2.5 TWiki::Func — the official API boundary
|
||||
- `lib/TWiki/Func.pm` ("TWikiFuncDotPm") documents **all** interfaces available to
|
||||
plugins, deliberately abstracting the flat-file assumption: `readTopic`,
|
||||
`saveTopic`, `getTopicList`, `getListOfWebs`, `getWorkArea`, `saveAttachment`, …
|
||||
Plugins are told *not* to assume flat files and to go through Func.
|
||||
|
||||
## 3. Plugin API — interfaces to extend the core
|
||||
|
||||
TWiki extends "without altering core code" by **registering Perl handlers that the
|
||||
core calls at defined points** in the request/render/save pipeline. A plugin is
|
||||
`lib/TWiki/Plugins/<Name>.pm` + a documentation topic + an optional `Config.spec`.
|
||||
|
||||
### 3.1 Handler callbacks (the hook surface)
|
||||
| Phase | Handlers |
|
||||
|-------|----------|
|
||||
| Init / users | `initPlugin` (must return 1), `registrationHandler` |
|
||||
| Tag/variable expansion | `commonTagsHandler`, `beforeCommonTagsHandler`, `afterCommonTagsHandler` |
|
||||
| Rendering | `preRenderingHandler`, `postRenderingHandler` (`start/endRenderingHandler` deprecated) |
|
||||
| Edit lifecycle | `beforeEditHandler`, `afterEditHandler` |
|
||||
| Save lifecycle | `beforeSaveHandler`, `afterSaveHandler`, `beforeMergeHandler` |
|
||||
| Attachments | `beforeAttachmentSaveHandler`, `afterAttachmentSaveHandler` |
|
||||
| Topic mgmt | `afterRenameHandler`, `completePageHandler` |
|
||||
|
||||
These are the TWiki analogue of XWiki's `ObservationManager` events — but
|
||||
**synchronous pipeline hooks** rather than an async event bus. The save/rename/
|
||||
attachment handlers are exactly the interception points a federation adapter needs.
|
||||
|
||||
### 3.2 Other extension mechanisms
|
||||
- **Custom variables/macros** — registered via `commonTagsHandler` (Foswiki adds
|
||||
`registerTagHandler`); convention `%PLUGINNAME_SETTING%`.
|
||||
- **REST handlers** — a plugin registers REST handlers invoked via the `rest`
|
||||
script, for transactions outside the standard `view/edit/save` scripts. This is the
|
||||
natural remote adapter transport.
|
||||
- **`Config.spec`** — declares configuration items (BOOLEAN, STRING, SELECT, PATH,
|
||||
PERL) surfaced in the `configure` web UI.
|
||||
- **Work area** — `getWorkArea()` gives a plugin a persistent, non-web-accessible
|
||||
data directory.
|
||||
|
||||
### 3.3 Extension package types
|
||||
| Type | Role |
|
||||
|------|------|
|
||||
| **Plugin** | handler-based behavior, no core change (`lib/TWiki/Plugins/<Name>.pm`) |
|
||||
| **Skin** | visual appearance only; topic content unchanged (e.g. PrintSkin) |
|
||||
| **AddOn** | a script in `bin/` callable from a topic |
|
||||
| **Contrib** | shared library code, or **alternative implementations of core sections** (e.g. user management) when something "can't be a plugin because it needs close core access" |
|
||||
|
||||
## 4. Access control (origin of yawex's model)
|
||||
|
||||
TWiki authorizes via **preference settings in topics**: `ALLOWWEBVIEW` /
|
||||
`ALLOWWEBCHANGE` at web level and `ALLOWTOPICVIEW` / `ALLOWTOPICCHANGE` /
|
||||
`ALLOWTOPICRENAME` (and `DENY*`) at topic level, with **VIEW / CHANGE / RENAME
|
||||
grantable separately**. Topic-level settings apply only to that topic. This
|
||||
per-topic ACL is the **direct prior art** behind yawex's `AccessControl`
|
||||
(denied/view/form/edit/admin) and a concrete reference for shard-wiki's optional
|
||||
**per-page ACL at L4** (`spec/ArchitectureBlueprint.md` §5).
|
||||
|
||||
## 5. TWiki as a shard — capability profile
|
||||
|
||||
| Capability | TWiki | Note |
|
||||
|------------|-------|------|
|
||||
| Read | ✓ | TWiki::Func / `view` script / `rest`; or read `data/*.txt` directly |
|
||||
| Write | ✓ | saveTopic / `save` / `rest`; **per-topic** granularity |
|
||||
| Structured payload | ✓ | TWiki Forms → `%META:FIELD%` **in the text file** (git-diffable) |
|
||||
| Version/diff | ✓ | RCS `.txt,v` — **open format, git-convertible** (cf. XWiki DB) |
|
||||
| Merge | partial | `beforeMergeHandler`; no git 3-way |
|
||||
| Change hooks | ✓ | synchronous save/rename/attachment handlers |
|
||||
| Auth model | per-web/topic ACL | authorize through shard-wiki core; engine ACL is advisory |
|
||||
| Direct-store attach | ✓✓ | `data/<Web>/<Topic>.txt` + RCS is a folder shard on its own |
|
||||
| Federation hooks | ✓ | plugin handlers + REST handlers can host an adapter (UC-38) |
|
||||
|
||||
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 6.1 Reinforcements
|
||||
| Observation | INTENT principle |
|
||||
|-------------|------------------|
|
||||
| Forms store data as `%META%` **in the topic file** | structured pages can be **git-diffable**, strengthening UC-34/39 vs XWiki's DB |
|
||||
| RCS `.txt,v` history | **git-addressable coordination** — here history is *convertible*, not just supplementable (UC-36 → UC-41) |
|
||||
| `data/<Web>/<Topic>.txt` mirrors logic | an engine's **on-disk store is itself attachable as a folder shard** (UC-40) |
|
||||
| Plugin save/rename/REST handlers | **composable integration** — TWiki can host an adapter too, generalizing UC-38 beyond XWiki |
|
||||
| Per-topic `ALLOW/DENY` ACL | prior art for **per-page ACL at L4**; lineage TWiki→yawex (UC-06) |
|
||||
| Webs as namespaces / shards | namespace navigation (UC-22); webs as roots |
|
||||
|
||||
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||
| TWiki assumption | shard-wiki correction |
|
||||
|------------------|----------------------|
|
||||
| TWikiML + variable expansion is the page | **Markdown-first**; TWikiML render stays out of core |
|
||||
| ACL lives in topic preference settings | **authorize in core**; topic ALLOW/DENY is advisory provenance |
|
||||
| History = RCS files next to content | **coordination journal** is the space-level git layer (may *import* RCS) |
|
||||
| Apps = Forms + `%SEARCH%` rendered by TWiki | represent records **without** depending on TWiki to render/search them |
|
||||
| Attach = run TWiki and call it | shard-wiki may attach the **bare data dir** with TWiki offline (UC-40) |
|
||||
|
||||
### 6.3 What TWiki teaches that shard-wiki should not lose
|
||||
1. **File-backed engines are the easy, high-value case** — their store is already a
|
||||
folder of text + open-format history; the adapter can often skip the runtime.
|
||||
2. **Structured data in text beats structured data in a DB** for federation —
|
||||
diffable, portable, journal-friendly. Prefer file-embedded metadata where offered.
|
||||
3. **One backend, two attachment paths** (runtime API vs on-disk store) is a real
|
||||
capability/consistency trade-off the adapter contract must express.
|
||||
|
||||
## 7. Use-case seeds → catalog (promoted 2026-06-13)
|
||||
|
||||
| Seed | Catalog UC | Disposition |
|
||||
|------|------------|-------------|
|
||||
| Attach a live engine's **on-disk store directly** (vs its runtime API) | **UC-40** (new) | dual-path attach; fidelity/consistency trade-off |
|
||||
| **Import** an engine's native file-based history (RCS) into the journal | **UC-41** (new) | history *migration* with fidelity, vs UC-36 *supplementation* |
|
||||
| Engine hosts adapter via its plugin/handler API | UC-38 | enriched: TWiki save/rename/REST handlers generalize it beyond XWiki |
|
||||
| Structured page carries typed data | UC-34 / UC-39 | enriched: TWiki Forms `%META%`, git-diffable |
|
||||
| Internal-history engine | UC-36 | enriched: contrast DB (supplement) vs RCS (import, UC-41) |
|
||||
| Authenticated team wiki / per-page ACL | UC-06 | enriched: TWiki per-topic ACL is the origin of yawex's model |
|
||||
|
||||
## 8. Open questions (for spec / workplans)
|
||||
|
||||
1. **Dual-path adapters** — when does shard-wiki attach an engine's on-disk store
|
||||
directly vs go through its API? Consistency risk of reading live files under a
|
||||
running engine; capability-gate it (UC-40)?
|
||||
2. **History import fidelity** — can RCS `.txt,v` be converted to git commits
|
||||
preserving author/timestamp, and is that authoritative or a one-time backfill
|
||||
feeding the coordination journal (UC-41, cf. UC-36 Q3)?
|
||||
3. **File-embedded metadata mapping** — map TWiki `%META:FIELD%` (and XWiki XObjects)
|
||||
onto one structured-metadata page-model representation (shared with xwiki §8 Q1).
|
||||
4. **TWiki vs Foswiki** — target both via one adapter (shared API) or treat as
|
||||
distinct shard types?
|
||||
5. **ACL provenance** — surface a topic's `ALLOW/DENY` as read-only provenance even
|
||||
though authorization is decided in core?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
| Source | URL |
|
||||
|--------|-----|
|
||||
| TWiki — TWikiPlugins (handlers, package types) | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
|
||||
| TWiki — Developing Plugins / EmptyPlugin | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
|
||||
| TWiki — TWiki::Func module | https://twiki.org/cgi-bin/view/TWiki/TWikiFuncModule |
|
||||
| TWiki — TWiki Forms | https://twiki.org/cgi-bin/view/TWiki/TWikiForms |
|
||||
| TWiki — TWiki Access Control | https://twiki.org/cgi-bin/view/TWiki/TWikiAccessControl |
|
||||
| TWiki — System Requirements (Perl/RCS) | https://twiki.org/cgi-bin/view/TWiki/TWikiSystemRequirements |
|
||||
| TWiki — Source Code | https://twiki.org/cgi-bin/view/TWiki/SourceCode |
|
||||
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
|
||||
| Wikipedia — TWiki | https://en.wikipedia.org/wiki/TWiki |
|
||||
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
|
||||
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
|
||||
| shard-wiki — yawex prior art | `research/260608-yawex-prior-art/findings.md` |
|
||||
|
||||
---
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
| This document section | Informs (future) |
|
||||
|-----------------------|------------------|
|
||||
| §2 architecture | adapter design for file-backed engines |
|
||||
| §3 plugin handlers | UC-38 engine-side adapter; composable-integration API shape |
|
||||
| §4 access control | per-page ACL at L4 (`spec/ArchitectureBlueprint.md`), UC-06 |
|
||||
| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) |
|
||||
| §6 INTENT mapping | architecture-blueprint guardrails |
|
||||
| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-40, UC-41; UC-06/34/36/38/39 enrichment) |
|
||||
| §8 open questions | spec — dual-path adapters, history import, structured-metadata model |
|
||||
35
research/260613-xwiki-deep-dive/README.md
Normal file
35
research/260613-xwiki-deep-dive/README.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# 260613 — XWiki deep dive (implementation, extension interfaces, ecosystem)
|
||||
|
||||
Date: 2026-06-13
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **XWiki** — the canonical "wiki-as-application-platform"
|
||||
engine flagged in `research/260608-wikiengines-overview/`. Goes past the landscape
|
||||
scan into XWiki's **implementation** (component/DI architecture, document data
|
||||
model, storage, rendering, events), the **interfaces it exposes to extend the core
|
||||
engine** (components, macros, UI extension points, script services, REST), and the
|
||||
**extension ecosystem** (extensions.xwiki.org repository, XAR/JAR/WebJar types,
|
||||
notable extensions).
|
||||
|
||||
Read through one lens: **what does XWiki teach shard-wiki about (a) adapting a
|
||||
structured app-platform engine as a shard, and (b) how a host engine can become
|
||||
federation-capable through its own extension mechanism** (INTENT *composable
|
||||
integration*).
|
||||
|
||||
Complements:
|
||||
- `research/260608-wikiengines-overview/` — the landscape scan that surfaced XWiki
|
||||
- `research/260608-federation-concepts/` — already covered XWiki's ActivityPub extension
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, extension-interface catalog, ecosystem, capability profile, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to
|
||||
`spec/UseCaseCatalog.md` (UC-38, UC-39); existing UC-34/UC-36/UC-31 enriched with
|
||||
XWiki specifics. Adapter-contract implications feed
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
217
research/260613-xwiki-deep-dive/findings.md
Normal file
217
research/260613-xwiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# Findings — XWiki: implementation, extension interfaces, ecosystem
|
||||
|
||||
Date: 2026-06-13 · Status: research draft
|
||||
|
||||
Scope: XWiki as **prior art for two shard-wiki concerns** — (1) attaching a
|
||||
structured "wiki-as-application-platform" engine as a shard, and (2) how an engine
|
||||
exposes extension interfaces rich enough to *host* a federation adapter
|
||||
(INTENT *composable integration*). Sources are XWiki's own dev docs, the
|
||||
DeepWiki code analysis of `xwiki/xwiki-platform`, and the extensions repository.
|
||||
|
||||
**Complements:** `research/260608-wikiengines-overview/` (landscape) and
|
||||
`research/260608-federation-concepts/` §3.2 (XWiki ActivityPub).
|
||||
|
||||
---
|
||||
|
||||
## 1. What XWiki is
|
||||
|
||||
A **programmable, component-based wiki platform** (Java/Jakarta EE, Hibernate ORM)
|
||||
that blurs wiki and lightweight application platform. Pages are not just prose:
|
||||
they carry **typed structured objects**, so an XWiki page can be a form, a record,
|
||||
or a small application. This is the defining property the landscape scan flagged —
|
||||
and the reason XWiki is the hardest case for shard-wiki's Markdown-first page model.
|
||||
|
||||
## 2. Implementation architecture
|
||||
|
||||
### 2.1 Component system (the spine)
|
||||
- **Dependency-injection component model.** Components are interfaces (roles)
|
||||
discovered via `META-INF/components.txt` and instantiated by a `ComponentManager`.
|
||||
Implementations declare `@Component`, an optional `@Named` **role hint**, and
|
||||
`@Singleton`. No interface→impl dependency — impls can ship in separate JARs and
|
||||
**replace defaults at runtime**.
|
||||
- Core replaceable roles include `XWikiStoreInterface` (storage), `ObservationManager`
|
||||
(events), `XWikiRightService` (authorization), `XWikiAuthService` (authentication).
|
||||
|
||||
### 2.2 oldcore vs modular platform
|
||||
- `xwiki-platform-oldcore` holds the foundational document model, persistence, and
|
||||
legacy APIs (`com.xpn.xwiki.*`). A bridge connects it to the newer component world
|
||||
to avoid circular dependencies.
|
||||
- Newer modules layer on top: `model` (entity references), `rendering`, `security`,
|
||||
`web` (HTTP actions/templates), `search-solr`.
|
||||
|
||||
### 2.3 Data model — document-centric + class/object
|
||||
- Central entity: **`XWikiDocument`**, addressed by `DocumentReference`
|
||||
(wiki / space / page). Content is parsed into an **XDOM** (rendering tree).
|
||||
- **Structured data via a class-object pattern** (OOP-like):
|
||||
- `BaseClass` / **XClass** — schema (field definitions), like a class declaration
|
||||
- `BaseObject` / **XObject** — an instance conforming to a class, attached to a page
|
||||
- `PropertyClass` / `BaseProperty` — field definition / value
|
||||
- Property types: String, LargeString, Date, Number, List, …
|
||||
- A single page can hold content **plus N typed objects** plus attachments. This is
|
||||
the structural fact shard-wiki must represent without flattening.
|
||||
|
||||
### 2.4 Storage & history
|
||||
- Two-tier: **`XWikiCacheStore`** over **`XWikiHibernateStore`** (Hibernate, multi-DB).
|
||||
- Tables: `xwikidoc`, `xwikiobject`, `xwikiproperties`, `xwikiattachment`, and
|
||||
**`xwikircs`** — internal **RCS-style version history**. History is real but
|
||||
**engine-internal and non-portable** (directly relevant to shard-wiki UC-36).
|
||||
|
||||
### 2.5 Events & rendering
|
||||
- **`ObservationManager`** event bus: cancelable (`DocumentCreatingEvent`,
|
||||
`DocumentUpdatingEvent`, `DocumentDeletingEvent`) and post-facto
|
||||
(`DocumentCreatedEvent`, `DocumentUpdatedEvent`, …). Listeners are components —
|
||||
a natural **change-notification source for incremental sync**.
|
||||
- **Rendering pipeline:** source syntax → parse → XDOM → transformations (macros) →
|
||||
serialize (HTML, PDF, …). Pluggable parsers/renderers; XWiki can parse **Markdown**
|
||||
as an input syntax even though its native syntax differs.
|
||||
|
||||
### 2.6 Multi-wiki
|
||||
- One installation hosts many wikis (`WikiReference`), each with separate documents
|
||||
and configuration — XWiki's own multi-tenancy, conceptually adjacent to
|
||||
shard-wiki's root-entity/tenant boundary.
|
||||
|
||||
### 2.7 APIs & security
|
||||
- **Dual API surface:** internal `com.xpn.xwiki.*` (full power) and script
|
||||
`com.xpn.xwiki.api.*` (wrapped, enforces **programming rights** and view rights).
|
||||
Privilege is enforced at the script boundary, not the storage boundary.
|
||||
|
||||
## 3. Interfaces to extend the core engine
|
||||
|
||||
XWiki's extensibility is unusually layered. From lowest/most-powerful to highest:
|
||||
|
||||
| # | Mechanism | What it extends | How |
|
||||
|---|-----------|-----------------|-----|
|
||||
| 1 | **Components** | anything | implement a role interface, `@Component` + `@Named` hint; ship in a JAR with `components.txt`; can override defaults (store, auth, rights) |
|
||||
| 2 | **Rendering macros (Java)** | content processing | register `Macro.class` role + hint; participates in the XDOM transformation pipeline |
|
||||
| 3 | **Wiki macros** | content processing | a wiki page following a spec — **no Java, no compile**; in-wiki authoring |
|
||||
| 4 | **Script services** | scripting API | implement `org.xwiki.script.service.ScriptService`; exposed to Velocity/Groovy as `$services.<hint>` |
|
||||
| 5 | **UI Extensions (UIX)** at **UI Extension Points (UIXP)** | the interface, no skin edit | UIX stored as XObjects of `XWiki.UIExtensionClass` (label/target/icon); injected at named hooks (since v4.2), e.g. `org.xwiki.platform.panels.Applications` |
|
||||
| 6 | **REST API (JAX-RS)** | remote integration | resources for pages, spaces, **objects**, classes, attachments, comments, tags, annotations, users/groups, **history**; custom JAX-RS resources can be added |
|
||||
| 7 | **Event listeners** | reactive behavior | register a component on `ObservationManager` to react to document lifecycle events |
|
||||
| 8 | **Resource reference handlers** | new URL/resource types | implement `ResourceReferenceHandler` on the root component manager |
|
||||
| 9 | **Skins / templates / Velocity** | presentation | Flamingo skin macros, templates |
|
||||
|
||||
Two things stand out for shard-wiki: (a) the **component model lets an extension
|
||||
replace authentication, authorization, and storage** — i.e. an XWiki instance could
|
||||
delegate auth exactly as shard-wiki's blueprint prescribes; (b) the **REST API plus
|
||||
event bus** give a federation adapter both a transport (read/write pages+objects+
|
||||
history) and a push-based freshness signal.
|
||||
|
||||
## 4. Extension ecosystem
|
||||
|
||||
- **`extensions.xwiki.org`** repository: **900+ extensions** (applications, macros,
|
||||
skins, panels, themes, legacy plugins). Browsed/installed/upgraded at runtime via
|
||||
the **Extension Manager Application** from the admin UI.
|
||||
- **Extension types:**
|
||||
- **XAR** — a package of wiki pages (an application, a wiki macro, or a snippet)
|
||||
- **JAR** — server-side Java: **components and script services**
|
||||
- **WebJar** — JAR of web resources (JS/CSS/Less)
|
||||
- **"Plugin" (legacy) vs Component (modern):** the old plugin API is superseded by
|
||||
the component system; new code is components.
|
||||
- **Notable extensions** (relevance to shard-wiki in parentheses):
|
||||
- **AppWithinMinutes** — build structured apps (XClass+XObjects) from the UI
|
||||
(→ pages-as-records, the UC-39 case)
|
||||
- **LDAP / Active Directory** and **Entra ID (Azure AD)** authentication
|
||||
(→ confirms pluggable `XWikiAuthService`; mirrors shard-wiki's delegated authn)
|
||||
- **ActivityPub** — fediverse federation (→ adapter transport, cf. UC-31)
|
||||
- **LiveData / LiveTable** — queryable tabular views over XObjects
|
||||
- draw.io diagrams, PDF viewer, Task Manager, Blog — content/app breadth
|
||||
|
||||
## 5. XWiki as a shard — capability profile
|
||||
|
||||
Read through shard-wiki's capability-aware adapter lens:
|
||||
|
||||
| Capability | XWiki | Note |
|
||||
|------------|-------|------|
|
||||
| Read | ✓ | REST + render to HTML/Markdown |
|
||||
| Write | ✓ | REST PUT page/object; **per-document** granularity (page + its objects) |
|
||||
| Structured payload | ✓✓ | XObjects/XClass — *richer* than Markdown; must not be flattened (UC-34/UC-39) |
|
||||
| Version/diff | ✓ (internal) | `xwikircs`; **not git-portable** → coordination journal supplies that (UC-36) |
|
||||
| Merge | partial | no native git-style 3-way merge |
|
||||
| Change events | ✓ | `ObservationManager` → push-based sync/freshness (enriches UC-31) |
|
||||
| Auth model | pluggable | `XWikiAuthService`/`XWikiRightService` replaceable — authorize through shard-wiki core, don't trust engine ACLs |
|
||||
| Federation hooks | ✓✓ | component + REST + UIX make XWiki able to **host** a shard-wiki adapter (UC-38) |
|
||||
| Multi-tenancy | ✓ | `WikiReference` farm → multiple shards / root entities |
|
||||
|
||||
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 6.1 Reinforcements
|
||||
| Observation | INTENT principle |
|
||||
|-------------|------------------|
|
||||
| Pages = content + typed XObjects | **Markdown-first, backend-neutral** must carry structure, not flatten (UC-34/39) |
|
||||
| `xwikircs` internal-only history | **Git-addressable coordination** adds the portable layer (UC-36) |
|
||||
| Pluggable `XWikiAuthService` | **authorization in core, authentication delegated** — XWiki already does exactly this |
|
||||
| Component + REST + UIX extension surface | **composable integration** — engine becomes federation-capable via its own API (UC-38) |
|
||||
| `ObservationManager` events | **explicit provenance/freshness** via push, not poll (UC-31, UC-24) |
|
||||
| `WikiReference` multi-wiki | root-entity / tenant boundary has prior art |
|
||||
|
||||
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||
| XWiki assumption | shard-wiki correction |
|
||||
|------------------|----------------------|
|
||||
| Native XWiki syntax + macro render is the page | shard-wiki keeps **Markdown-first**; engine render stays out of core, structure travels as metadata |
|
||||
| ACLs/rights live in the engine DB | **authorize in core**; engine rights are advisory, not trusted |
|
||||
| History is the engine's RCS table | **coordination journal** is space-level and git-backed |
|
||||
| Federation = an XWiki extension talking to other XWikis | shard-wiki federates **heterogeneous** backends; XWiki is one shard among many |
|
||||
| App = XObjects rendered by XWiki | shard-wiki must represent records **without** depending on XWiki to render them |
|
||||
|
||||
### 6.3 What XWiki teaches that shard-wiki should not lose
|
||||
1. **A clean role/hint component model** is what makes auth/store/rights swappable —
|
||||
shard-wiki's adapter contract should be similarly role-based and override-friendly.
|
||||
2. **Structured objects are first-class**, not an afterthought — the page model needs
|
||||
a typed-metadata escape hatch from day one.
|
||||
3. **An engine with a real extension API can host the adapter itself** — the cheapest
|
||||
path to a high-fidelity shard is engine-side, not screen-scraping.
|
||||
|
||||
## 7. Use-case seeds → catalog (promoted 2026-06-13)
|
||||
|
||||
| Seed | Catalog UC | Disposition |
|
||||
|------|------------|-------------|
|
||||
| Engine hosts a federation adapter via its native extension API | **UC-38** (new) | composable integration — first UC for the engine-side direction |
|
||||
| Attach a wiki-as-application-platform shard (pages are typed records/forms) | **UC-39** (new) | extends UC-34 toward bodiless structured pages |
|
||||
| Structured page carries semantic data | UC-34 | enriched with XObject/XClass concretes |
|
||||
| Internal-only revision history | UC-36 | enriched with `xwikircs` example |
|
||||
| Subscribe to remote shard changes | UC-31 | enriched with `ObservationManager` event-driven transport |
|
||||
|
||||
## 8. Open questions (for spec / workplans)
|
||||
|
||||
1. **Page-model representation of XObjects** — typed frontmatter, sidecar file, or
|
||||
opaque provenance blob? (shared with `wikiengines` §6 Q1; XWiki makes it urgent)
|
||||
2. **Bodiless record pages** — does shard-wiki's page model require a Markdown body,
|
||||
or can a page be purely structured (AppWithinMinutes apps)?
|
||||
3. **Adapter placement** — engine-side extension (high fidelity, needs deploy access)
|
||||
vs external REST adapter (zero engine change, lower fidelity). Both? Capability-gated?
|
||||
4. **Event transport** — consume `ObservationManager` via a JAR listener, or poll REST
|
||||
history? Affects freshness guarantees (UC-31/UC-24).
|
||||
5. **Rights mapping** — translate XWiki `XWikiRightService` rights into shard-wiki
|
||||
actions (read/write/patch/merge/administer), or ignore and authorize purely in core?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
| Source | URL |
|
||||
|--------|-----|
|
||||
| XWiki — Developer Guide | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/ |
|
||||
| XWiki — Creating XWiki Components | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/WritingComponents/ |
|
||||
| XWiki — Available Extension Points | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/ExtensionPoint/ |
|
||||
| XWiki — UI Extension Point Tutorial | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/UIXTutorial/ |
|
||||
| XWiki — Wiki Macro Tutorial | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/WritingMacros/WikiMacroTutorial/ |
|
||||
| XWiki — RESTful API | https://www.xwiki.org/xwiki/bin/view/Documentation/UserGuide/Features/XWikiRESTfulAPI |
|
||||
| XWiki Rendering — Writing a Macro | https://rendering.xwiki.org/xwiki/bin/view/Main/ExtendingMacro |
|
||||
| Extension Manager Application | https://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Manager%20Application |
|
||||
| Extension Module — Extensions/types | https://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Module/Extensions/ |
|
||||
| DeepWiki — xwiki/xwiki-platform (code analysis) | https://deepwiki.com/xwiki/xwiki-platform |
|
||||
| Mancinelli — Overview of XWiki's RESTful API | http://blog.fabio.mancinelli.me/2011/03/07/XWikis_RESTful_API.html |
|
||||
| shard-wiki — wiki engines overview | `research/260608-wikiengines-overview/findings.md` |
|
||||
| shard-wiki — federation concepts | `research/260608-federation-concepts/findings.md` |
|
||||
|
||||
---
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
| This document section | Informs (future) |
|
||||
|-----------------------|------------------|
|
||||
| §2 architecture | adapter design for app-platform engines |
|
||||
| §3 extension interfaces | UC-38 engine-side adapter; composable-integration API shape |
|
||||
| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) |
|
||||
| §6 INTENT mapping | architecture-blueprint guardrails |
|
||||
| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-38, UC-39; UC-31/34/36 enrichment) |
|
||||
| §8 open questions | spec — page model for structured/bodiless pages |
|
||||
38
research/260614-computational-page-model-synthesis/README.md
Normal file
38
research/260614-computational-page-model-synthesis/README.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# 260614 — The computational page model (SHARD-WP-0004 synthesis)
|
||||
|
||||
Date: 2026-06-15 · Source: **SHARD-WP-0004** post-batch synthesis (T1–T8)
|
||||
|
||||
## What this is
|
||||
|
||||
The acceptance-criteria synthesis for SHARD-WP-0004 — *"the computational page model"* —
|
||||
reading the eight computational/interactive-knowledge dives across each other and distilling
|
||||
them into **one model**: **source is canonical; everything rendered/computed is a
|
||||
projection**, placed on **two axes** (projection-kind: replication vs derivation; liveness:
|
||||
live↔snapshot), with a recommendation on an executable-content capability.
|
||||
|
||||
## The answer to the carried question
|
||||
|
||||
*Can a shard-wiki page be a live computational artifact?* **Yes — as a page-model +
|
||||
projection concern, not as an execution platform.** Every system externalizes to a canonical
|
||||
source and treats the live/computed form as derived; shard-wiki **recognizes** computational
|
||||
content, **attaches the source**, and **presents derivations as provenance- and
|
||||
liveness-marked projections**, with **execution as a gated capability (off by default,
|
||||
degrade to snapshot)**. No INTENT amendment required.
|
||||
|
||||
## Key contributions
|
||||
|
||||
- **One model:** `(source, derivation rule, projection with provenance + liveness)` covers
|
||||
all four computational page shapes (one-source-many-projections UC-83; notebook UC-84;
|
||||
program-as-page; live/temporal content).
|
||||
- **Two axes for T16:** replication vs **derivation-projection** (timing / multiplicity /
|
||||
continuity facets) × the **live↔snapshot** axis (bounded at the irreducibly-live far end by
|
||||
Strudel).
|
||||
- **One snapshot-provenance record** reused for notebook outputs, renders, recordings.
|
||||
- **Hard boundaries:** never host a kernel/runtime as store; **image-is-not-a-store**; never
|
||||
present a derivation without output→source provenance.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | The source/derivation/projection model, the two axes, the four page shapes, provenance/reproducibility, the recommendation, the SHARD-WP-0002 fold-in, escalated open questions |
|
||||
178
research/260614-computational-page-model-synthesis/findings.md
Normal file
178
research/260614-computational-page-model-synthesis/findings.md
Normal file
@@ -0,0 +1,178 @@
|
||||
# The computational page model — synthesis (SHARD-WP-0004)
|
||||
|
||||
**Date:** 2026-06-15 · **Source:** SHARD-WP-0004 (post-batch synthesis, T1–T8) · **Kind:**
|
||||
synthesis (no new external research) reading the eight computational/interactive-knowledge
|
||||
dives *across* each other.
|
||||
|
||||
## What this is
|
||||
|
||||
The acceptance-criteria synthesis for SHARD-WP-0004. The batch asked one carried question:
|
||||
*can a shard-wiki page be a live computational artifact — a source woven/evaluated into
|
||||
rendered forms — and if so, how do projection, transclusion, provenance, and the adapter
|
||||
contract treat the source, the environment, and the computed output?* This memo answers it
|
||||
by distilling the eight dives into **one model**: the **source / derivation / projection**
|
||||
view of computational content, anchored on two axes, plus a recommendation on whether
|
||||
shard-wiki needs an executable-content capability.
|
||||
|
||||
Dives consolidated: **T1** literate programming (WEB/weave/tangle), **T2** Mathematica,
|
||||
**T3** Jupyter, **T4** Processing/p5.js, **T5** Strudel, **T6** Squeak, **T7** Glamorous
|
||||
Toolkit, **T8** Pharo. Catalog yield: **UC-83** (literate one-source-many-projections),
|
||||
**UC-84** (notebook with computed-output provenance); the other six are **enrichment/boundary**
|
||||
dives (UC-54/55/47/48 + the projection model).
|
||||
|
||||
## 1. The one finding: source is canonical, everything rendered is a projection
|
||||
|
||||
Every system in the batch, however "live," **externalizes to a canonical artifact** and
|
||||
treats the rendered/computed form as **derived**:
|
||||
|
||||
| System | Canonical (attach this) | Derived (a projection) |
|
||||
|--------|-------------------------|------------------------|
|
||||
| Literate (WEB) | the WEB/`.nw`/`.org` **source** | woven docs **and** tangled code |
|
||||
| Mathematica | the `.nb` (a Wolfram expression) | cached `Output` cells; CDF; `Dynamic` widgets |
|
||||
| Jupyter | `.ipynb` source (ideally paired text) | embedded outputs; nbconvert/nbviewer renders |
|
||||
| Processing | the **sketch source** | the view-time canvas render |
|
||||
| Strudel | the **pattern source** | the live audio performance / a recording |
|
||||
| GT / Lepiter | git-versionable **page/Tonel files** | moldable `gtView`s; live snippet results |
|
||||
| Squeak/Pharo | exported **files** (Tonel/git) | the live image / running objects |
|
||||
|
||||
**This is the same principle shard-wiki already holds** (files-canonical, index/render
|
||||
derived — ikiwiki UC-79, Logseq UC-62, nbstripout). The computational batch did not break it;
|
||||
it **stress-tested it to the live extreme and it held**. *The image (Squeak/Pharo) is the
|
||||
only would-be exception, and it is a boundary, not a counterexample: an image is not a store.*
|
||||
|
||||
## 2. Two axes the contract must add
|
||||
|
||||
The batch refines the **projection model** (SHARD-WP-0002 T16) with two orthogonal axes:
|
||||
|
||||
### 2.1 Projection kind: replication vs derivation
|
||||
- **Replication-projection** (the existing default): a lazy **cache/copy** of remote content
|
||||
(Obsidian/Notion mirrors, UC-53/57).
|
||||
- **Derivation-projection** (new, from T1): a **transform/compile/weave/evaluate** of a
|
||||
source into rendered forms — regenerable, may delegate to the source's tool, **degrades to
|
||||
a captured snapshot** when the tool is absent. Covers weave/tangle, nbconvert, CDF, sketch
|
||||
render, audio recording, `gtView`.
|
||||
|
||||
Sub-facets of derivation (from T4): **materialization timing** = *ahead-of-time* (CDF,
|
||||
nbconvert, static HTML) vs *view-time* (Processing/Strudel); **multiplicity** = one output
|
||||
(UC-79) vs **N co-equal projections** (UC-83 weave+tangle; UC-47/48/54 moldable views).
|
||||
|
||||
### 2.2 Liveness: the live↔snapshot axis (from T6, bounded by T5)
|
||||
Every derived view sits on a spectrum, and the more live the source, the more its static form
|
||||
is a clearly-marked degrading snapshot:
|
||||
|
||||
```
|
||||
static source ── captured output ── live-over-files ── view-time one-shot ── continuous/
|
||||
(literate) (notebook UC-84) (GT/Lepiter) (Processing) interactive
|
||||
── irreducibly
|
||||
live/temporal
|
||||
(Strudel: source
|
||||
+ recording only)
|
||||
```
|
||||
|
||||
**Honesty rule (union-without-erasure):** a computed/live view must always declare *what it
|
||||
is* — "captured at run N, environment unguaranteed" (UC-84), "one performance, time T, source
|
||||
rev R" (UC-83/Strudel), "live render needs the runtime." Never present a snapshot as live or a
|
||||
static page as capturing a live artifact.
|
||||
|
||||
## 3. The computational page-model shapes (T12)
|
||||
|
||||
The batch adds these page shapes (beyond prose / typed records / query-defined / inline-
|
||||
embedded objects / typed-graph already catalogued):
|
||||
|
||||
1. **One-source-many-projections** (UC-83) — a source whose presented forms are co-equal
|
||||
derivations (docs + code), each with output→source provenance; **named-chunk transclusion**
|
||||
assembles fragments by name at derivation time (UC-32/44).
|
||||
2. **Notebook** (UC-84) — **ordered/nestable cells** (Mathematica adds the outline tree)
|
||||
where code cells own **embedded computed outputs** (the derived output is stored *inside*
|
||||
the source) with **weak execution provenance**; outputs may be MIME blobs or **structured
|
||||
re-evaluable values** (Mathematica) — a new point on the content-opacity spectrum.
|
||||
3. **Program-as-page** (Processing) — canonical content = **source text**, presentation = an
|
||||
**executable render** with **no cached output**; non-Markdown executable content.
|
||||
4. **Live/temporal/generative content** (Strudel) — source canonical, render irreducibly
|
||||
live; static = source + a marked recording.
|
||||
|
||||
All four reduce to **(source, derivation rule, projection with provenance + liveness)** — one
|
||||
model, four positions.
|
||||
|
||||
## 4. Provenance & reproducibility (the honest weakness)
|
||||
|
||||
Computed output provenance is **real but fragile** everywhere: Jupyter/Mathematica
|
||||
`execution_count`/`In`-`Out` can be **out-of-order**; **environment/versions/data are not
|
||||
captured**; Strudel/Processing may be **non-deterministic**. Implication for the contract:
|
||||
treat a computed output as a **snapshot with declared, incomplete provenance** (run id, source
|
||||
rev, timestamp; environment "unguaranteed"), reusing **one snapshot-provenance machinery**
|
||||
across notebooks, recordings, and renders (UC-84 is the template). This is consistent with
|
||||
shard-wiki's existing "surface freshness/completeness, never imply more than you have"
|
||||
(Oddmuse partial-history UC-82).
|
||||
|
||||
## 5. The recommendation
|
||||
|
||||
**Does shard-wiki need an executable/computational content capability? — Yes, but only as
|
||||
recognition + projection + capability-gating, never as an execution engine.**
|
||||
|
||||
1. **Adopt the source/derivation/projection model** (no execution required). shard-wiki
|
||||
**recognizes** computational content types (literate source, notebook, sketch, pattern),
|
||||
**attaches the canonical source**, and **presents derived forms as projections with
|
||||
provenance + liveness markers**. This alone delivers UC-83/UC-84 and the enrichment of
|
||||
UC-54/55 — and needs **no kernel, no sandbox, no runtime**.
|
||||
2. **Make execution a capability, off by default.** "Drive a derivation" (run tangle/weave,
|
||||
re-execute a notebook, render a sketch, evaluate a pattern in the viewer) is a **gated
|
||||
capability** with a **trust/sandbox** sub-concern (T11). Absent it, **degrade to the
|
||||
captured snapshot / static render / recording** — the graceful-degradation rule, which the
|
||||
batch shows always has an honest fallback (source is tiny and diffable everywhere).
|
||||
3. **One projection model, two axes** (T16): projection-kind (replication vs derivation; with
|
||||
timing + multiplicity facets) × liveness (live↔snapshot). The **moldable view registry**
|
||||
(T7) is the unifying structure — an open, type-keyed set of co-equal projections, none
|
||||
canonical-by-fact (display-canonical is policy).
|
||||
4. **One snapshot-provenance record** reused for notebook outputs, renders, and recordings
|
||||
(run id, source rev, timestamp, environment "unguaranteed").
|
||||
5. **Hard boundaries** (design-bugs if violated): never host a kernel/runtime as the store;
|
||||
**image-is-not-a-store** (attach exported files); never present a derivation without
|
||||
output→source provenance; never imply a static view captures a live artifact.
|
||||
|
||||
**Net:** computational content is **in scope as a page-model + projection concern**, **out of
|
||||
scope as an execution platform** — exactly the mechanism-over-policy, capability-aware,
|
||||
degradable posture INTENT already mandates. No INTENT amendment is required; this extends the
|
||||
page model and projection model within existing constraints.
|
||||
|
||||
## 6. Fold into SHARD-WP-0002
|
||||
|
||||
- **T12 (page model):** add the four computational shapes (§3); allow nestable cells and
|
||||
structured re-evaluable outputs; "derived output may live inside the source" (notebook).
|
||||
- **T16 (projection):** the **two-axis model** (§2) + the **moldable view registry** (§3/T7);
|
||||
materialization-timing, multiplicity, continuity, and the live↔snapshot far end as explicit
|
||||
projection metadata.
|
||||
- **T11 (capabilities):** "derive/execute/render/evaluate" as gated capabilities with trust/
|
||||
sandbox; default off → snapshot.
|
||||
- **T15 (fidelity):** non-Markdown executable/computed content; lossy renders; the
|
||||
structured-re-evaluable-value point on the content-opacity spectrum.
|
||||
- **T13 (history):** paired-text (Jupytext) / cell-aware (nbdime) strategies for embedded-
|
||||
output documents; outputs-as-derived (nbstripout ethos).
|
||||
- **T14 (binding):** **image-is-not-a-store** boundary (export→files only).
|
||||
|
||||
## 7. Open questions (escalated)
|
||||
|
||||
1. Is **liveness** (and "irreducibly live / no faithful static form") an explicit first-class
|
||||
metadata flag on every projection, so the union renders the honest fallback automatically?
|
||||
(T5/T6 far-end question.)
|
||||
2. Does shard-wiki **ever drive a derivation** (sandboxed), or strictly attach + present
|
||||
snapshots? (Recurs UC-56/UC-83/UC-84/T4/T5 — a single capability/trust policy decision.)
|
||||
3. Is a computed output's **structured re-evaluable value** (Mathematica/Wolfram) modeled as a
|
||||
typed value or stored opaquely with provenance? (UC-55 open-Q #10; UC-84 Q3.)
|
||||
4. Should **UC-83** and **UC-84** eventually merge as two positions of one "source +
|
||||
derivations" shape, or stay distinct? (Kept distinct: UC-84's defining trait is *output
|
||||
embedded in source with weak provenance*; UC-83's is *N co-equal external derivations*.)
|
||||
|
||||
## 8. Sources
|
||||
|
||||
The eight SHARD-WP-0004 dives: `research/260614-{literate-programming,mathematica,jupyter,
|
||||
processing,strudel,squeak-pharo,glamorous-toolkit}-deep-dive/`. Prior projection/structure
|
||||
anchors: `research/260614-{ikiwiki,logseq,zigzag}-deep-dive/`,
|
||||
`research/260614-shard-spectrum-synthesis/`.
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
No new UC (consolidation). Consolidates **UC-83, UC-84** and enrichments to **UC-32, UC-44,
|
||||
UC-47, UC-48, UC-54, UC-55, UC-79, UC-37, UC-35**. Feeds **SHARD-WP-0002 T11/T12/T13/T14/T15/
|
||||
T16** (see §6). Recommendation: computational content is **in scope as page-model + projection
|
||||
mechanism, out of scope as an execution platform**; no INTENT amendment required.
|
||||
20
research/260614-federated-wiki-deep-dive/README.md
Normal file
20
research/260614-federated-wiki-deep-dive/README.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# 260614 — Federated Wiki deep dive
|
||||
|
||||
Deep dive on Ward Cunningham's **Federated Wiki** (Smallest Federated Wiki / SFW,
|
||||
2011 →) as a **federation model** rather than a single shard: fork-with-provenance,
|
||||
the per-page **JSON journal** of semantic actions, the **story** of typed items, the
|
||||
**neighborhood/roster** discovery model, and time-bounded **happenings**.
|
||||
|
||||
This is prior art for shard-wiki's **coordination layer itself** — the closest existing
|
||||
system to "a union of pages preserving provenance, assembled non-destructively." It
|
||||
extends `research/260608-federation-concepts/` §3 with the concrete data model + protocol.
|
||||
|
||||
- `findings.md` — full writeup: data model, journal/action types, federation protocol,
|
||||
capability profile, INTENT mapping, UC seeds (UC-70–UC-72), architecture notes for
|
||||
SHARD-WP-0002, open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-70 (attach a fedwiki site via page-JSON + CORS), UC-71 (append-only
|
||||
semantic action journal with site provenance as a coordination-journal model), UC-72
|
||||
(fork-with-site-provenance federation across a neighborhood / chorus). Enriched
|
||||
UC-26/28/30/05/27. Feeds SHARD-WP-0002 T1–T5 (federation) and T11/T13/T16 (write
|
||||
granularity, log-based merge, identity≠placement).
|
||||
240
research/260614-federated-wiki-deep-dive/findings.md
Normal file
240
research/260614-federated-wiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,240 @@
|
||||
# Federated Wiki — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T1 · **Subject:** Ward Cunningham's
|
||||
Smallest Federated Wiki (SFW) / Federated Wiki (fedwiki ecosystem).
|
||||
|
||||
## Why this dive
|
||||
|
||||
Every prior dive has been a *shard candidate* — a store we might attach. Federated Wiki
|
||||
is different: it is a **federation model**, the one piece of public prior art whose core
|
||||
job is the same as shard-wiki's coordination layer — *present a union of pages from many
|
||||
independent sites while preserving where each came from, and let people copy and edit
|
||||
non-destructively*. Ward Cunningham (inventor of the wiki) built SFW in 2011 precisely to
|
||||
fix the original wiki's single-canonical-page weakness with **fork + provenance**. We go
|
||||
past the surface (`260608-federation-concepts/` §3) into the data model and protocol, then
|
||||
ask what shard-wiki should adopt.
|
||||
|
||||
**Framing:** fedwiki is not just "a shard we attach" — it is a *worked example of the
|
||||
coordination journal, overlay-before-mutation, and union-without-erasure*, three of our
|
||||
own design pillars, shipped and running.
|
||||
|
||||
---
|
||||
|
||||
## 1. The data model — page = title + story + journal
|
||||
|
||||
A fedwiki page is a small JSON object with three core fields (plus optional decoration):
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Welcome Visitors",
|
||||
"story": [
|
||||
{ "type": "paragraph", "id": "7b56f22a4b9ee974",
|
||||
"text": "Welcome to this [[Federated Wiki]] site." },
|
||||
{ "type": "image", "id": "a1c0e3...", "url": "...", "caption": "..." }
|
||||
],
|
||||
"journal": [
|
||||
{ "type": "create", "id": "7b56f22a4b9ee974", "item": {...}, "date": 1310000000000 },
|
||||
{ "type": "add", "id": "a1c0e3...", "item": {...}, "after": "7b56f22a4b9ee974",
|
||||
"date": 1310000100000 },
|
||||
{ "type": "edit", "id": "7b56f22a4b9ee974", "item": {...}, "date": 1310000200000 },
|
||||
{ "type": "fork", "site": "ward.fed.wiki.org", "date": 1310000300000 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- **story** — an *ordered array of typed items* ("paragraph-like" items). Each item is
|
||||
`{ type, id, text, ...type-specific }`. The **`id`** is a random 16-hex string,
|
||||
**stable across edits** (it is the unit of identity within a page). The **`type`** names
|
||||
the **plugin** that renders/edits the item (`paragraph`, `image`, `html`, `markdown`,
|
||||
`code`, `method`, `pagefold`, chart plugins, …). *Data lives in the item; behavior lives
|
||||
in the plugin* — the item is portable JSON; the plugin is the renderer.
|
||||
- **journal** — an *ordered, append-only array of action objects* that, when replayed,
|
||||
**reconstructs the story**. The story is a materialized view of the journal. This is the
|
||||
key architectural choice: **the journal is the source of truth, the story is derived.**
|
||||
|
||||
## 2. Journal action types — a semantic op-log
|
||||
|
||||
Each journal entry is an action with `{ type, ... , date }` (epoch-ms). The action types:
|
||||
|
||||
| action | fields | meaning |
|
||||
|---------|--------|---------|
|
||||
| `create`| `id, item, date` | first item — page born |
|
||||
| `add` | `id, item, after, date` | insert an item after another |
|
||||
| `edit` | `id, item, date` | replace an item's content (id preserved) |
|
||||
| `move` | `order, date` | reorder items |
|
||||
| `remove`| `id, date` | delete an item |
|
||||
| `fork` | `site, date` | **mark that the page was copied from `site` at this point** |
|
||||
|
||||
Two things matter for us:
|
||||
|
||||
1. **These are *semantic* operations** (add/move/edit/remove a paragraph), not text diffs
|
||||
and not character-level CRDT ops. The write granularity is the **story item
|
||||
(paragraph)** — a *middle* granularity between whole-file (TiddlyWiki) and
|
||||
block/character (Logseq/CRDT). It is an **op-log** like a CRDT, but the ops are
|
||||
coarse-grained and **applied by humans via fork**, not auto-merged.
|
||||
2. **`fork` is the provenance primitive.** When you copy a remote page to your own site,
|
||||
a `fork` entry is appended recording the **source site** and time. The journal of a
|
||||
forked page therefore **serializes a directed acyclic graph (DAG)** of where content
|
||||
came from — "the journal of a forked page is detailed enough to recognize where in the
|
||||
journal of the original the fork took place" (CouchDB-style per-entry sequence numbers
|
||||
make the cut-point identifiable). History visualization highlights the forked entry.
|
||||
|
||||
## 3. The federation protocol — sites, neighborhood, roster
|
||||
|
||||
- **Site** = an independent server (originally Node.js; also static-file and serverless
|
||||
variants). A site owns a set of pages, each served as **page JSON over HTTP** at
|
||||
`/<slug>.json`, with **CORS headers** so a *browser-side* client can fetch pages from
|
||||
**any** site. Page identity within a site is the **slug** (a title-derived kebab name).
|
||||
- **The client assembles the union, not the server.** The fedwiki client ("the lineup")
|
||||
renders pages **side by side**: clicking a link opens that page *from whatever site it
|
||||
resolves against*, appended to the right. Browsing literally builds a left-to-right
|
||||
trail across sites.
|
||||
- **Neighborhood** = the dynamic set of sites encountered in the current session (from the
|
||||
sites of pages you've opened, links, and forks). **Search runs across the neighborhood**
|
||||
— a federated search over exactly the sites you've touched.
|
||||
- **Roster** = an explicit, authored list of sites to include (a curated neighborhood);
|
||||
"sister sites" are peers you watch. There is **no central registry** — discovery is by
|
||||
link, fork, and roster.
|
||||
- **Happenings** = time-bounded collaborative events where many participants fork around a
|
||||
topic for a period, producing a burst of related forks (a bounded collaboration that
|
||||
leaves a durable forked record on each participant's own site).
|
||||
|
||||
## 4. The editorial model — fork, don't edit-in-place
|
||||
|
||||
You can only write to **your own** site. To change someone else's page you **fork** it
|
||||
(copy into your site, journal records the source), then edit your copy. Many forks of the
|
||||
same page coexist across sites — Cunningham's **"chorus of voices"**: *no canonical
|
||||
version*, divergence is normal and visible, and you choose whose changes to pull by forking
|
||||
them. There is **no automatic merge** — reconciliation is human: compare journals, fork the
|
||||
version you prefer, optionally re-fork upstream changes.
|
||||
|
||||
---
|
||||
|
||||
## 5. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | Federated Wiki |
|
||||
|--------------------------------|----------------|
|
||||
| Attachment mode | **REST/file-store hybrid** — page JSON over HTTP+CORS; also static files |
|
||||
| Addressing granularity | **story item (paragraph)** via stable 16-hex `id` |
|
||||
| Content identity | item `id` random+stable; page id = site + slug |
|
||||
| Identity vs placement | **placement-bound**: identity = `site` + `slug`; forks are *new* identities linked by journal provenance |
|
||||
| Structure | ordered array of **typed items** (plugin-typed) |
|
||||
| History | **per-page append-only journal** of semantic actions (op-log) |
|
||||
| Merge model | **fork + manual journal compare** — a *third model* beside git 3-way and CRDT auto-merge |
|
||||
| Native query | none built-in; **neighborhood search** (federated full-text across touched sites) |
|
||||
| Translation | item `text` is wiki/Markdown-ish; plugins own their formats |
|
||||
| Attachment/write granularity | **story-item level** (add/edit/move/remove one item) |
|
||||
| Operational envelope | tiny servers, browser-driven; CORS is the whole API surface |
|
||||
| Access grant | **own-site-only writes**; reads open via CORS |
|
||||
| Content opacity | transparent JSON (no E2EE); plugin-typed but inspectable |
|
||||
| Provenance | **first-class** — `fork` records source site; journal = provenance DAG |
|
||||
|
||||
## 6. INTENT mapping
|
||||
|
||||
### Reinforcements (fedwiki validates our pillars)
|
||||
|
||||
- **Coordination journal** (INTENT) ≈ fedwiki **journal**. Our journal idea is *exactly*
|
||||
fedwiki's per-page append-only action log — and fedwiki proves the story-as-derived-view
|
||||
pattern works. Strong reinforcement; adopt the **semantic-op + provenance-entry** shape.
|
||||
- **Overlay before mutation** ≈ **fork**. Fork *is* the canonical overlay: a
|
||||
non-destructive copy onto a writable surface, recording provenance, before any change.
|
||||
- **Union without erasure** ≈ **neighborhood + chorus**. The union is assembled from many
|
||||
sovereign sites; provenance (which site, forked-from) is never hidden; divergence is
|
||||
surfaced, not resolved away.
|
||||
- **No silent remote mutation** ≈ **own-site-only writes**. You structurally *cannot*
|
||||
mutate a remote; you fork to your own site. This is our rule, enforced by architecture.
|
||||
- **Mechanism over policy** ≈ **no canonical source**. Fedwiki ships the mechanism (fork,
|
||||
journal, neighborhood) and leaves "which version wins" entirely to people.
|
||||
- **Graceful degradation** ≈ static-file sites — a fedwiki site can be a read-only pile of
|
||||
JSON files; still forkable, still in the neighborhood.
|
||||
|
||||
### Divergences (boundaries / design notes, not bugs)
|
||||
|
||||
- **Identity = placement.** Fedwiki page identity is `site` + `slug`; a fork is a *new*
|
||||
page whose only tie to the origin is a journal `fork` entry. shard-wiki wants
|
||||
**identity ≠ placement** (the "same" page across shards under a stable identity, T16) —
|
||||
so we treat fedwiki's journal-linked forks as *provenance edges*, and layer our own
|
||||
cross-shard identity over them rather than adopting slug-as-identity.
|
||||
- **No query / no typed-record model.** Fedwiki is paragraphs+plugins, not a typed DB
|
||||
(contrast Notion/Wikibase). Fine — it sits at the *coordination* end, not the structure
|
||||
end. We don't ask fedwiki to provide query; the neighborhood search is the model for
|
||||
*federated* search across shards (T-federation), not in-shard query.
|
||||
- **Browser-assembles-union.** Fedwiki pushes union assembly to the client. shard-wiki
|
||||
assembles server/orchestrator-side. Adopt the *model* (union from sovereign sources +
|
||||
provenance), not the client-only locus.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Journal = append-only semantic-op log with provenance entries**, story = derived
|
||||
replay view. This is the concrete shape for our coordination journal (T13).
|
||||
2. **Fork-with-source-attribution** as the overlay/adopt primitive across shards.
|
||||
3. **Neighborhood** as the model for a *dynamic, link-and-fork-discovered* federated set +
|
||||
search, with **roster** as the curated/explicit variant.
|
||||
4. **Chorus of forks** — represent divergent versions across shards as co-equal, linked by
|
||||
provenance, with reconciliation as an explicit human/policy step (mechanism over policy).
|
||||
|
||||
---
|
||||
|
||||
## 7. UC seeds
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-70 | Attach a Federated Wiki site as a shard via its **page JSON + CORS** (REST/file-store hybrid); project pages, fork to overlay | **new** |
|
||||
| UC-71 | Adopt a **per-page append-only semantic-action journal with provenance entries** (fork=source site) as the coordination-journal model — replay to materialize, compare to locate divergence | **new** |
|
||||
| UC-72 | **Fork-with-site-provenance federation across a neighborhood** of peer shards — assemble a union from links/forks, search across it, preserve the chorus without forcing a canonical | **new** |
|
||||
| — | fork-with-provenance as overlay/adopt | enrich **UC-26** (fork) |
|
||||
| — | carry-forward of forked content + upstream re-fork | enrich **UC-28** (carry-forward) |
|
||||
| — | happenings = time-bounded collaboration leaving durable forks | enrich **UC-30** (time-bounded space) |
|
||||
| — | union/chorus of co-equal versions, provenance-linked | enrich **UC-05 / UC-27** |
|
||||
|
||||
## 8. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T1–T5 (federation):** fedwiki is the reference design. The **journal** (append-only,
|
||||
semantic ops, fork-provenance) is the concrete coordination-journal shape; **neighborhood
|
||||
+ roster** is the discovery/membership model (dynamic vs curated); **fork** is the
|
||||
overlay/adopt op. Model the union as an assembly over sovereign sources with provenance
|
||||
edges, reconciliation left to policy.
|
||||
- **T11 (capability/write-granularity):** add **story-item / paragraph** as a named
|
||||
write-granularity tier between whole-file and block/character.
|
||||
- **T13 (history portability / merge model):** record fedwiki's **journal-replay op-log**
|
||||
as a *third merge model* beside git 3-way and CRDT auto-merge — a **coarse semantic
|
||||
op-log applied manually via fork**. A shard whose history *is* such a journal can supply
|
||||
our coordination journal almost directly (vs git-commit import or CRDT-update import).
|
||||
- **T16 (identity ≠ placement):** fedwiki's `fork` journal entries are **provenance edges**
|
||||
between same-named pages on different sites — exactly the cross-shard "same page,
|
||||
different placement" relation we must model. Use them as edges; keep our own identity
|
||||
layer above slug.
|
||||
|
||||
## 9. Open questions
|
||||
|
||||
1. Should shard-wiki's coordination journal adopt fedwiki's **exact action vocabulary**
|
||||
(create/add/edit/move/remove/fork) at the page-item level, or a more granular/abstract
|
||||
op set that other shards can also emit?
|
||||
2. Is **neighborhood** (dynamic, link/fork-discovered) a first-class membership mode for an
|
||||
information space, or only a *view* over an explicitly-configured shard set (roster)?
|
||||
3. How do we reconcile fedwiki's **slug-as-identity + fork-DAG** with our intended
|
||||
**stable cross-shard identity** (T16) — promote fork edges into the identity graph, or
|
||||
keep them as provenance-only annotations?
|
||||
4. Does the **chorus / no-canonical** stance compose with shards that *do* assert a
|
||||
canonical (Notion, an upstream git main)? (policy-selectable canonical over a
|
||||
mechanism that permits chorus.)
|
||||
|
||||
## 10. Sources
|
||||
|
||||
- Smallest Federated Wiki wiki: **Story JSON**, **Federation Details** —
|
||||
github.com/WardCunningham/Smallest-Federated-Wiki/wiki
|
||||
- JSON Schema notes — song.fed.wiki.org/json-schema.html
|
||||
- "Smallest Federated Wiki" — home.c2.com/smallest-federated-wiki.html
|
||||
- Federated Wiki — federated.wiki (Visualizing Page History)
|
||||
- Mike Caulfield, "The OER Case for Federated Wiki" — hapgood.us (2015)
|
||||
- Jon Udell, "A federated Wikipedia" — blog.jonudell.net (2015)
|
||||
- Wikipedia: *Federated Wiki*; IndieWeb: *Smallest Federated Wiki*
|
||||
- fedwiki/wiki-plugin-transport (plugin/transport reference)
|
||||
- prior: `research/260608-federation-concepts/` §3
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
New UCs **UC-70–UC-72** carry the marker **⊞** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md` (true lineage = this dive; placed in the nearest existing column).
|
||||
Enriched: UC-26, UC-28, UC-30, UC-05, UC-27. Architecture cross-refs: SHARD-WP-0002
|
||||
T1–T5, T11, T13, T16.
|
||||
17
research/260614-forge-wikis-deep-dive/README.md
Normal file
17
research/260614-forge-wikis-deep-dive/README.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# 260614 — git-forge wikis deep dive (Gitea · GitLab · GitHub)
|
||||
|
||||
Deep dive on the **git-forge-hosted Markdown wikis** as one grouped memo: each is a
|
||||
**dedicated git repository of Markdown** exposed through a forge, attachable by **cloning
|
||||
the wiki repo directly** *or* (where offered) through the **forge's wiki API**. INTENT names
|
||||
**Gitea wikis** explicitly as a shard participant — this dive confirms the **git-native
|
||||
file-store** as a first-class, and the *simplest*, backend.
|
||||
|
||||
- `findings.md` — the three forges compared, the `.wiki.git` model, API matrix, capability
|
||||
profile, INTENT mapping, UC seeds (UC-76/77), architecture notes for SHARD-WP-0002, open
|
||||
questions (resolves the UC-68 source-of-truth question for this case), sources,
|
||||
traceability.
|
||||
|
||||
Catalog yield: UC-76 (attach a forge wiki by cloning its dedicated `.wiki.git` — git **is**
|
||||
the native store and the coordination journal), UC-77 (attach/write via the forge wiki API
|
||||
where git-clone is unavailable/undesired — capability varies by forge). Enriched
|
||||
UC-40/02/68/38. Feeds SHARD-WP-0002 T14 (attachment binding).
|
||||
173
research/260614-forge-wikis-deep-dive/findings.md
Normal file
173
research/260614-forge-wikis-deep-dive/findings.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# git-forge wikis (Gitea · GitLab · GitHub) — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T5 · **Subject:** the Markdown wikis
|
||||
hosted by the three major git forges — **Gitea**, **GitLab**, **GitHub** — treated as one
|
||||
family because they share one architecture: *a wiki is a separate git repo of Markdown.*
|
||||
|
||||
## Why this dive
|
||||
|
||||
INTENT names **Gitea wikis** as a shard participant, and the whole project is "a **Git-based
|
||||
Markdown** wiki orchestrator." The forge wikis are therefore the **least exotic, highest-
|
||||
fit** backend in the entire study: the page store is *literally a git repository of Markdown
|
||||
files*. After fourteen dives into DBs, CRDTs, graphs and SaaS, this one confirms the
|
||||
**home case** — and sharpens it by contrasting *git-IS-the-store* (forge wikis) against
|
||||
*git-is-a-mirror* (Wiki.js, UC-68).
|
||||
|
||||
## 1. The shared architecture — a wiki is a `.wiki.git` repo
|
||||
|
||||
All three forges implement a project/repo wiki as a **second, dedicated git repository**
|
||||
alongside the code repo, addressable as `<repo>.wiki.git`:
|
||||
|
||||
- `git@host:owner/project.wiki.git` (GitLab), `…/owner/repo.wiki.git` (Gitea/GitHub).
|
||||
- **Pages are Markdown files** (`Home.md`, `Some-Page.md`), one file per page; the **page
|
||||
title ↔ filename** (spaces ↔ hyphens by convention). Other markups are accepted
|
||||
(AsciiDoc, Textile, reStructuredText, Org) — GitHub/Gitea via **Gollum** (the Ruby
|
||||
git-backed wiki library), GitLab via its own renderer.
|
||||
- **History is git history** — every page edit (web or pushed) is a git commit with
|
||||
author/timestamp/message. *The wiki's revision history is a real git log.*
|
||||
- **Special pages** by convention: `_Sidebar`, `_Footer`, `_Header` (GitHub/Gitea),
|
||||
`_sidebar` (GitLab) — engine-rendered chrome stored as ordinary files.
|
||||
- **Subdirectories / nested pages**: GitLab and Gitea support directory structure; GitHub
|
||||
wikis are historically flat (Gollum supports paths but the GitHub UI is shallow).
|
||||
|
||||
The decisive property: **you can `git clone` the wiki repo, edit files, commit, and push**,
|
||||
and the forge UI reflects it — *and vice versa*. Git is **a** (often **the**) first-class
|
||||
write path. This is exactly shard-wiki's native medium with no impedance.
|
||||
|
||||
## 2. Where they differ — the API matrix
|
||||
|
||||
| | git clone/push of `.wiki.git` | wiki content **API** | nested dirs | markups |
|
||||
|--|--|--|--|--|
|
||||
| **Gitea** | ✅ yes | ✅ **REST wiki endpoints** (list/get/create/edit/delete pages) | ✅ | Markdown (+Gollum-style) |
|
||||
| **GitLab** | ✅ yes | ✅ **REST Wikis API** (project & group wikis) | ✅ | Markdown/AsciiDoc/RDoc/Org |
|
||||
| **GitHub** | ✅ yes | ❌ **no wiki REST API** — wiki is **git-only** (Gollum) | ⚠️ flat UI | Markdown + Gollum markups |
|
||||
|
||||
The key asymmetry: **GitHub exposes wiki content *only* through git** (the REST/GraphQL API
|
||||
covers issues/PRs/code but **not** wiki pages); **GitLab and Gitea offer both** a wiki API
|
||||
*and* git access. So the **git-clone path is the universal one** (works for all three); the
|
||||
API path is an *optional, capability-varying* alternative.
|
||||
|
||||
## 3. git-IS-the-store vs git-is-a-mirror (the UC-68 contrast)
|
||||
|
||||
Wiki.js (UC-68) keeps a **DB as canonical** and *maintains a git mirror* — so writing by
|
||||
commit risks **racing the engine's DB↔git sync** (catalog open-Q22). Forge wikis are the
|
||||
opposite: **the git repo IS the canonical store**; there is *no* separate DB of record for
|
||||
wiki content. Therefore:
|
||||
|
||||
- **The source-of-truth question (Q22) is resolved for this case:** the `.wiki.git` repo is
|
||||
authoritative. shard-wiki can **write by commit/push directly** with no engine to race —
|
||||
the forge merely *renders* what git holds.
|
||||
- The forge **API** (GitLab/Gitea), where present, is a *convenience over the same git
|
||||
repo*, not a competing store — so API-write and git-write converge on one history.
|
||||
|
||||
This makes forge wikis the **cleanest possible write-through file-store shard**: clone =
|
||||
projection/mirror, commit = overlay-applied/write, git log = the coordination journal *as
|
||||
is*.
|
||||
|
||||
## 4. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | Gitea / GitLab / GitHub wiki |
|
||||
|--------------------------------|------------------------------|
|
||||
| Attachment mode | **file-store (native: git clone)** + optional **external-API** (GitLab/Gitea wiki REST) |
|
||||
| Addressing granularity | **page = file**; sub-page = path (GitLab/Gitea) |
|
||||
| Content identity | path/filename within the wiki repo (title-derived) |
|
||||
| Identity vs placement | placement-bound (path = identity), like a plain git repo |
|
||||
| Structure | flat or directory tree of Markdown files; `_Sidebar`/`_Footer` chrome |
|
||||
| History | **native git history** (real commits, authors, messages) |
|
||||
| Merge model | **git** (3-way merge, branches) — though wiki repos are usually single-branch |
|
||||
| Native query | none (it's files); forge full-text search over the wiki |
|
||||
| Translation | **Markdown-native** (+ AsciiDoc/Org via renderer) — minimal/no translation needed |
|
||||
| Attachment/write granularity | **file (page)** per commit |
|
||||
| Operational envelope | ordinary git + forge; clone is cheap; API rate limits apply to API path |
|
||||
| Access grant | **forge repo permissions** (delegated auth; per-repo/role ACL) |
|
||||
| Content opacity | transparent Markdown in git |
|
||||
| Provenance | git author/committer/timestamp per commit — native |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements (this is the home case)
|
||||
|
||||
- **Git-based Markdown orchestrator** (INTENT core): forge wikis *are* git repos of
|
||||
Markdown. The **wiki page model** (Markdown-first, path-addressed, git-versioned) maps 1:1
|
||||
— minimal adapter, maximal fit.
|
||||
- **Coordination journal = git** (INTENT): the wiki repo's **git log is already the
|
||||
coordination journal** — no synthesis needed; adopt it directly.
|
||||
- **Overlay before mutation**: overlays are **branches/commits** on the cloned wiki repo;
|
||||
applying = push (or open an MR/PR where the forge supports wiki MRs — GitLab does not for
|
||||
wikis, so push-to-branch + manual is the path).
|
||||
- **Graceful degradation**: even GitHub (no wiki API) is fully usable via git-clone — the
|
||||
*universal* path means a limited forge is still a first-class read/write shard.
|
||||
- **No silent remote mutation**: writes are explicit git pushes (or explicit API calls)
|
||||
under the user's forge credentials and repo permissions.
|
||||
|
||||
### Divergences (boundaries / notes — minor)
|
||||
|
||||
- **Capability varies by forge**: GitHub = git-only (no content API); GitLab/Gitea = git +
|
||||
API. The adapter must **model the API as an optional capability**, defaulting to the
|
||||
universal git path (T11/T14). Not a bug — exactly the capability-awareness INTENT mandates.
|
||||
- **Wiki repos rarely use branches/MRs for review**: forge wikis usually edit a single
|
||||
branch directly; the rich PR-review flow is on the *code* repo, not the wiki. So
|
||||
"overlay → review → merge" needs shard-wiki to provide the review layer, not the forge.
|
||||
- **Identity = path** (like any git repo) — cross-shard identity (T16) is layered above, as
|
||||
for plain git/`wiki/` subdir shards.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **git-clone as the universal, canonical file-store attach** for forge wikis — Markdown +
|
||||
git history directly as page model + coordination journal (UC-76). The reference
|
||||
easy-case backend.
|
||||
2. **Forge wiki API as an optional capability** (GitLab/Gitea), with **git-only fallback**
|
||||
(GitHub) — capability-aware binding (UC-77).
|
||||
3. **git-IS-store ⇒ write-by-commit is safe** (no engine race) — record this as the
|
||||
resolution of the Wiki.js mirror dilemma (Q22) for forge wikis.
|
||||
|
||||
## 6. UC seeds
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-76 | Attach a **git-forge wiki** by **cloning its dedicated `.wiki.git`** — git is the native store; Markdown files = pages, git log = coordination journal; commit/push = write (no engine to race) | **new** |
|
||||
| UC-77 | Attach/write a forge wiki via the **forge's wiki API** (GitLab/Gitea REST) where git-clone is unavailable or API-write is preferred; **git-only fallback** for GitHub — capability varies by forge | **new** |
|
||||
| — | git-native file-store as the *canonical store* (not mirror) | enrich **UC-40** |
|
||||
| — | dual-path attach (git clone vs forge API) | enrich **UC-02** |
|
||||
| — | git-IS-store vs engine-maintained mirror (resolves Q22) | enrich **UC-68** |
|
||||
| — | forge as an API host for the wiki resource | enrich **UC-38** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T14 (adapter binding / attach path):** forge wikis are the canonical **file-store
|
||||
attach** — bind to the `.wiki.git` clone as the universal path; model the **wiki API as an
|
||||
optional, forge-specific capability** (present: GitLab, Gitea; absent: GitHub). One shard,
|
||||
two possible bindings converging on the same git history.
|
||||
- **T11 (capability model):** "has-content-API" is a **per-forge capability flag**; git
|
||||
clone/push is the baseline every forge satisfies. Minimal adapter profile — near the
|
||||
Oddmuse-simple end but Markdown-native and git-versioned.
|
||||
- **Coordination journal:** adopt the wiki repo's **git log directly** — the one backend
|
||||
where INTENT's git-backed journal needs *zero* synthesis.
|
||||
- **Resolves Q22 (UC-68):** because git **is** the store (not a mirror), **write-by-commit
|
||||
is safe** — no engine DB↔git sync to race. Record the distinction *engine-mirror*
|
||||
(Wiki.js: DB canonical, careful) vs *git-canonical* (forge wikis: commit freely).
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. For overlay → **review** → apply, does shard-wiki supply the review layer over a forge
|
||||
wiki (which lacks wiki-MRs), e.g. via a branch + its own diff/approve, or push directly?
|
||||
2. When a forge offers **both** git and a wiki API (GitLab/Gitea), which does the adapter
|
||||
prefer by default — git (universal, full history) with API as a fallback for hosts where
|
||||
clone is disabled? (cf. UC-43 backend-swap under stable binding.)
|
||||
3. Should the **code-repo `wiki/` subdir** shard and the **forge wiki repo** shard share one
|
||||
adapter (both git+Markdown) with a "which repo / which path" parameter, or stay distinct?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- GitLab Docs — *Wiki* (separate git repo; web/git/API; `.wiki.git`) — docs.gitlab.com
|
||||
- Gitea — wiki via git clone + repository **wiki API**; forum/issue threads on
|
||||
`.wiki.git` clone (go-gitea/gitea #1426, #15420) — gitea.com / github.com/go-gitea
|
||||
- GitHub — wiki = Gollum git repo (`<repo>.wiki.git`), no wiki REST API — docs.github.com
|
||||
- Gollum (git-based wiki library) — github.com/gollum/gollum
|
||||
- prior: `research/260614-wikijs-deep-dive/` (engine-maintained mirror contrast, UC-68)
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UCs **UC-76–UC-77** carry the marker **⎇** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-40, UC-02, UC-68, UC-38. Architecture cross-refs:
|
||||
SHARD-WP-0002 T14, T11; coordination-journal-from-git; resolves catalog open-Q22.
|
||||
33
research/260614-glamorous-toolkit-deep-dive/README.md
Normal file
33
research/260614-glamorous-toolkit-deep-dive/README.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# 260614 — Glamorous Toolkit (moldable development) deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T7**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Glamorous Toolkit** (GT, on Pharo): **moldable development** — cheap,
|
||||
custom, **domain-specific views** (`gtView` methods) so any object explains itself through an
|
||||
**open set of co-equal projections, none canonical** — plus **Lepiter**, GT's live notebook/
|
||||
knowledge base (git-versionable JSON page files with live, inspectable code results).
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Strongest prior art for **moldable, multi-view projection**: projection is not *a* view
|
||||
but an **open, type-keyed set of co-equal, possibly-computed views, none privileged** —
|
||||
refines SHARD-WP-0002 **T16** and unifies replication-/derivation-/dimensional-/query-
|
||||
projection under "many co-equal views."
|
||||
- Generalizes ZigZag dimensional views (UC-47/48) and query/computed views (UC-54) into a
|
||||
**pluggable view registry** keyed by content type (answers UC-55's open question on a
|
||||
content-type registry).
|
||||
- Reinforces **files-canonical, liveness-above, degrade-to-snapshot** (Lepiter files vs the
|
||||
Pharo image; same boundary as Jupyter UC-84 / Squeak T6).
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (design prior art, not a candidate shard — like the UseModWiki lineage dive).
|
||||
- Enrich **UC-47, UC-48, UC-54**; links **UC-55, UC-83, UC-84, UC-79**.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Moldable development & `gtView`, the Moldable Inspector, Lepiter, relation to ZigZag/query/derivation projection, INTENT mapping, UC disposition (enrichment-only), architecture notes, open questions |
|
||||
161
research/260614-glamorous-toolkit-deep-dive/findings.md
Normal file
161
research/260614-glamorous-toolkit-deep-dive/findings.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Glamorous Toolkit — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T7 · **Subject:** Glamorous Toolkit (GT) on
|
||||
Pharo — moldable development, custom views/inspectors, the live notebook (Lepiter).
|
||||
|
||||
## Why this dive
|
||||
|
||||
T1/T3 gave us **derivation-projection** (one source → rendered/computed forms). GT comes at
|
||||
projection from the *other* side: **many co-equal, domain-specific views over the same live
|
||||
content**, where the *environment molds itself to the knowledge* rather than forcing the
|
||||
knowledge into one fixed rendering. This is the strongest prior art for **moldable,
|
||||
multi-view projection** and a striking parallel to ZigZag's dimensional model (UC-47/48) and
|
||||
to query/computed views (UC-54). It is *design* prior art — GT is not a candidate shard —
|
||||
so the yield is **enrichment + projection design notes**, not a new shard UC.
|
||||
|
||||
## 1. Moldable development
|
||||
|
||||
GT's thesis: **systems should be explainable through custom tools that are cheap to build.**
|
||||
Instead of a single generic object inspector/renderer, a developer adds **small,
|
||||
domain-specific views** to a class so any instance explains itself in the terms that matter:
|
||||
|
||||
- **`gtView`-annotated methods** — a class declares extra inspector views by writing methods
|
||||
tagged `<gtView>` that return a view (tree, list, table, chart, source, raw, a custom
|
||||
diagram…). Each is a **co-equal projection of the same object**; none is privileged.
|
||||
- The **Moldable Inspector** shows these views as **switchable tabs** over one object, and
|
||||
lets you **dive** into sub-objects (each with its own custom views) — navigation *is*
|
||||
moving across projections.
|
||||
- Views are **cheap and local**: a view is just a method, versioned with the code, added
|
||||
incrementally as understanding grows. The environment **adapts to the domain**.
|
||||
|
||||
The key abstraction for us: **the same underlying content carries an open, extensible set of
|
||||
views, selected at inspection time, none canonical.**
|
||||
|
||||
## 2. Lepiter — the live notebook / knowledge base
|
||||
|
||||
GT ships **Lepiter**, a notebook/knowledge base where pages mix prose, **live code
|
||||
snippets** (evaluated in-image, results inspectable with the moldable views above), and
|
||||
links. Notebook pages are stored as **JSON "database" files** on disk (a Lepiter DB =
|
||||
directory of page files), so the knowledge base is **file-backed and git-versionable** while
|
||||
remaining live in the image.
|
||||
|
||||
This is the literate/notebook pattern (T1/T3) fused with moldable views: a snippet's result
|
||||
is not a static captured output but a **live object you can open into any of its views** —
|
||||
the *anti-snapshot*. (Boundary: that liveness is exactly what shard-wiki must degrade to a
|
||||
snapshot when the image isn't present — see §4.)
|
||||
|
||||
## 3. Relationship to ZigZag, query-views, and the projection model
|
||||
|
||||
- **vs ZigZag (UC-47/48):** ZigZag gives **dimensional** views — the *same cells* seen along
|
||||
different orthogonal axes. GT gives **moldable** views — the *same object* seen through
|
||||
different *purpose-built lenses*. Both reject a single privileged rendering; both make
|
||||
**multi-view, none-canonical** the norm. GT generalizes the idea from fixed dimensions to
|
||||
an **open, code-defined view set**.
|
||||
- **vs query/computed views (UC-54):** a `gtView` is a **computed projection** (it runs code
|
||||
to build the view) — like a query-defined page, but keyed to a **content type / domain**
|
||||
rather than a stored query. Strengthens "a view can be computed, not stored."
|
||||
- **vs derivation-projection (T1/UC-83):** GT confirms projections are **plural and
|
||||
co-equal**; UC-83/UC-84 had *few* well-known derivations (docs/code/outputs), GT has an
|
||||
**open registry** of them keyed by type.
|
||||
|
||||
So GT's contribution to the contract is: **projection is not one view but an open set of
|
||||
co-equal, type-keyed, possibly-computed views, none canonical** — a *moldable projection
|
||||
registry*. That is a refinement of T16's projection model, not a new shard.
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Union without erasure / no privileged rendering.** GT's "many co-equal views, none
|
||||
canonical" is the same ethos as showing provenance/freshness without hiding any: a page's
|
||||
presentation is **plural**, and shard-wiki should be able to offer **multiple co-equal
|
||||
projections of the same content** (raw, rendered, structured, domain-specific) rather than
|
||||
one flattened view.
|
||||
- **Computed views (UC-54) keyed by content type.** A moldable view = a **computed
|
||||
projection registered against a content type** — directly supports a *pluggable view/
|
||||
projection registry* in the contract (the answer shape for UC-55's "pluggable content-type
|
||||
registry" open question).
|
||||
- **Files-canonical, live-on-top.** Lepiter stores pages as **git-versionable JSON files**
|
||||
while being live in the image — reinforcing "the durable artifact is files; liveness is a
|
||||
layer above," consistent with shard-wiki's git-canonical stance.
|
||||
- **Mechanism over policy.** Which views to show, and whether to compute or snapshot them,
|
||||
stay configurable; GT provides the *mechanism* (open view set), not a fixed presentation.
|
||||
|
||||
### Divergences / boundaries
|
||||
|
||||
- **The image is not a store (shared boundary with T6 Squeak).** GT's *liveness* lives in a
|
||||
Pharo image; shard-wiki must not treat the image as a shard. Attach the **Lepiter DB files**
|
||||
(git-versionable) as the durable content; treat live/computed views as **derivation-
|
||||
projections that degrade to captured snapshots** when no image/kernel is present (same rule
|
||||
as Jupyter UC-84).
|
||||
- **Not a view engine.** shard-wiki *models* "this content type has these co-equal views and
|
||||
one may be canonical-for-display"; it does not implement GT's rendering. Domain-specific
|
||||
view code stays with the source/adapter, surfaced as a capability.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Moldable projection = an open, type-keyed set of co-equal, possibly-computed views,
|
||||
none canonical** — refine T16's projection model and the UC-47/48/54 cluster toward a
|
||||
**pluggable view/projection registry**.
|
||||
2. **Lepiter** = literate/notebook (T1/T3) with **live results** stored as git-versionable
|
||||
files — another "files-canonical, liveness-above, degrade-to-snapshot" instance.
|
||||
3. **Navigation = moving across projections** (dive into sub-objects) — a navigation idea
|
||||
beside dimensional movement (UC-47/48) for the derived-views thread.
|
||||
|
||||
## 5. UC disposition (enrichment-only — no new shard UC)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC |
|
||||
|------------------------|------------|
|
||||
| Many co-equal, domain-specific views over the same content; none canonical (§1, §3) | UC-47 / UC-48 (enriched) |
|
||||
| `gtView` = a **computed projection registered against a content type** (§1, §3) | UC-54 (enriched) |
|
||||
| Open, pluggable view set keyed by type = the shape of a content-type/view registry (§3) | links UC-55 (open-Q #10) |
|
||||
| Lepiter live-snippet results = live objects → degrade to snapshot when no image (§2, §4) | links UC-84, UC-83 |
|
||||
| Lepiter pages = git-versionable JSON files; image is not the store (§2, §4) | links UC-79 (files-canonical) |
|
||||
| Dive-into-sub-object navigation = moving across projections (§1) | links UC-17–UC-20 (derived views) |
|
||||
|
||||
GT is **design prior art, not a candidate shard**, so it yields **no new UC** (like the
|
||||
UseModWiki lineage dive). Its value is sharpening the **projection model** for SHARD-WP-0002
|
||||
T16: projection is an **open set of co-equal, type-keyed, possibly-computed views, none
|
||||
privileged** — a moldable view registry.
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T16 (projection):** generalize projection from "a view" to a **moldable view registry** —
|
||||
an open, extensible set of **co-equal, type-keyed, possibly-computed** projections of the
|
||||
same content, none canonical (display-canonical is a policy choice, not a fact). This
|
||||
unifies replication-projection (UC-53/57), derivation-projection (UC-83/84), dimensional
|
||||
views (UC-47/48), and query/computed views (UC-54) under one "many co-equal views" model.
|
||||
- **T12 (page model):** a content type may **carry/declare its own views** (a capability);
|
||||
the page model should allow attaching view definitions to a type (the registry's entries),
|
||||
answering UC-55's "pluggable content-type registry" question.
|
||||
- **Boundary (T14):** attach the **Lepiter DB files** (durable, git-versionable), never the
|
||||
Pharo image; live/computed views are derivation-projections degrading to snapshots — same
|
||||
rule as Jupyter (no kernel/image host).
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. Does shard-wiki expose a **view/projection registry** as a first-class public concept
|
||||
(content type → its co-equal views), or keep "moldable" as an adapter-internal idea?
|
||||
2. When multiple co-equal views exist, is **"canonical-for-display"** a per-shard policy, a
|
||||
user preference, or unset (always show the chooser)? (Mechanism-over-policy says
|
||||
configurable.)
|
||||
3. How does a **computed view** (UC-54/`gtView`) declare its **freshness/provenance** so the
|
||||
union doesn't present a stale computed projection as current? (Ties UC-84's snapshot
|
||||
marking.)
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- Glamorous Toolkit docs (`gtoolkit.com`): moldable development, the Moldable Inspector,
|
||||
`gtView` methods, Lepiter knowledge base; Feenk essays on moldable development.
|
||||
- Pharo (substrate — see T8 dive): live image, reflective environment.
|
||||
- prior: `research/260614-zigzag-deep-dive/` (dimensional multi-view UC-47/48);
|
||||
`research/260614-jupyter-deep-dive/` (live vs snapshot, UC-84);
|
||||
`research/260614-literate-programming-deep-dive/` (derivation-projection, UC-83).
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
**No new UC** (GT is design prior art, not a candidate shard). Enriched: UC-47, UC-48,
|
||||
UC-54; links UC-55 (content-type/view registry), UC-83/UC-84 (live→snapshot), UC-79
|
||||
(files-canonical), UC-17–UC-20 (derived-view navigation). Architecture cross-refs:
|
||||
SHARD-WP-0002 T16 (moldable view registry: open set of co-equal type-keyed computed views),
|
||||
T12 (content type declares its views), T14 (attach Lepiter files, not the image).
|
||||
15
research/260614-ikiwiki-deep-dive/README.md
Normal file
15
research/260614-ikiwiki-deep-dive/README.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# 260614 — ikiwiki deep dive
|
||||
|
||||
Deep dive on **ikiwiki** (Joey Hess): a **wiki compiler** — it compiles a **VCS-backed**
|
||||
(usually **git**) tree of Markdown source into **static HTML**, supports **distributed**
|
||||
operation (clone/edit/push between wiki instances, change **pings**), and treats web edits as
|
||||
commits to the same repo.
|
||||
|
||||
- `findings.md` — the compiler model, git-distributed federation + pinger, static output,
|
||||
capability profile, INTENT mapping, UC seed (UC-79), architecture notes for SHARD-WP-0002,
|
||||
open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-79 (attach a git-backed **compile-to-static** wiki — git Markdown source
|
||||
is the shard, compiled static HTML is a derived publish/projection; participate in
|
||||
git-distributed clone federation with change-pings). Enriched UC-31/56/37/33. Feeds
|
||||
SHARD-WP-0002 T4 (federation), T6 (publish/projection).
|
||||
153
research/260614-ikiwiki-deep-dive/findings.md
Normal file
153
research/260614-ikiwiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# ikiwiki — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T4 · **Subject:** ikiwiki, Joey Hess's
|
||||
VCS-backed wiki compiler.
|
||||
|
||||
## Why this dive
|
||||
|
||||
The forge-wiki dive (T5) established *git-IS-the-store* for hosted Markdown wikis. ikiwiki
|
||||
takes the same git-canonical source but adds two ideas shard-wiki cares about directly:
|
||||
**compile-to-static** (the wiki is *built*, not served from a DB) and **git-distributed
|
||||
federation** (wiki instances clone, pull, push, and **ping** each other). It is referenced
|
||||
in `research/260608-federation-concepts/`; here we go into the model.
|
||||
|
||||
## 1. The wiki-compiler model
|
||||
|
||||
ikiwiki is fundamentally a **compiler**: input is a directory of **source pages** (Markdown
|
||||
by default; also other formats) held in a **version-control repo**; output is a tree of
|
||||
**static HTML**. A rebuild is triggered by a **VCS post-commit/post-update hook**, so:
|
||||
|
||||
- **The VCS repo is canonical**; the HTML is *derived build output* (regenerable, disposable).
|
||||
- **Web edits are commits.** The CGI edit interface writes the change *into the repo* (a
|
||||
commit) and triggers a rebuild — so browser edits and `git push` edits converge on one
|
||||
history. (Same convergence as forge wikis, but here the canonical store is *your* git repo,
|
||||
not a forge's.)
|
||||
- **VCS-agnostic**: git is usual, but svn/bzr/mercurial/darcs are supported via a VCS
|
||||
plugin layer — an early "pluggable backend behind a stable interface" (adapter-contract
|
||||
echo).
|
||||
- **Plugins** (Perl) provide directives, feeds, auth (`openid`), and the federation hooks
|
||||
below.
|
||||
|
||||
## 2. Git-distributed federation + the pinger
|
||||
|
||||
Because the source is an ordinary VCS repo, ikiwiki instances federate the way *git* does:
|
||||
|
||||
- **Clone-and-diverge**: you can `git clone` a wiki, edit offline, and `push`/`pull` between
|
||||
instances — **multiple wiki clones that reconcile via git merge**. A wiki is a branch-space
|
||||
(UC-33).
|
||||
- **`pinger` / `pingee` plugins**: an instance can send an **XML-RPC ping** to another
|
||||
ikiwiki when it changes, prompting the other to **pull and rebuild** — a lightweight
|
||||
**subscribe/notify** primitive over the git-distributed mesh (UC-31).
|
||||
- **`aggregate` plugin**: pulls external **RSS/Atom feeds** into the wiki as pages — an
|
||||
inbound projection of remote content.
|
||||
|
||||
So ikiwiki is *federation by git plus a ping* — distinct from fedwiki's fork/journal
|
||||
(UC-72) and from Wikibase's query-time `SERVICE` (UC-74): a **third federation flavor**,
|
||||
*VCS-replication federation with change notification*.
|
||||
|
||||
## 3. Static output as a publish/projection target
|
||||
|
||||
The compiled static HTML is a **read-only, regenerable projection** of the source:
|
||||
|
||||
- It is a natural **outbound publish target** (UC-56): render the union (or a shard) to a
|
||||
static site for hosting/backup, no server needed.
|
||||
- It is also the **read-only backup** end (UC-37): a static snapshot that survives the engine.
|
||||
|
||||
The key shard-wiki framing: **source (git Markdown) is the attachable shard; static HTML is a
|
||||
derived projection** — never confuse the build output for the canonical content.
|
||||
|
||||
## 4. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | ikiwiki |
|
||||
|--------------------------------|---------|
|
||||
| Attachment mode | **file-store** (VCS-backed git Markdown source) |
|
||||
| Addressing granularity | page = source file; path = identity |
|
||||
| Content identity | path/filename (placement-bound) |
|
||||
| Structure | directory tree of Markdown source + directives |
|
||||
| History | **native VCS (git) history** |
|
||||
| Merge model | **git** (clone/pull/push/merge across instances) |
|
||||
| Native query | none; directives + plugins compute derived pages at build |
|
||||
| Translation | Markdown source → static HTML (build-time render) |
|
||||
| Write granularity | **file (page)** per commit |
|
||||
| Operational envelope | a compiler + VCS hook; static hosting for output |
|
||||
| Access grant | VCS/file perms; `openid` for web edits |
|
||||
| Content opacity | transparent Markdown |
|
||||
| Provenance | git author/timestamp; aggregated feeds carry source |
|
||||
| Federation | **git replication + XML-RPC ping**; RSS aggregation |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Git-canonical Markdown source** = the home case (shared with forge wikis UC-76/40):
|
||||
attach the source repo, adopt its git log as the journal, write by commit.
|
||||
- **Coordination layer is git** (INTENT): ikiwiki's whole federation *is* git replication +
|
||||
a ping — the most literal realization of "Git-addressable coordination layer."
|
||||
- **Projection vs canonical**: compile-to-static cleanly separates **canonical source** from
|
||||
**derived output** — exactly shard-wiki's projection principle (static HTML = a lazy/cache
|
||||
projection that is regenerable, never the source of truth).
|
||||
- **Graceful degradation / publish**: static output is the trivial read-only backup and
|
||||
outbound publish target (UC-37/UC-56).
|
||||
- **Subscribe/notify mechanism, not policy**: the pinger is a *mechanism* (notify a peer to
|
||||
pull); *which* peers, when, and conflict policy stay configurable.
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- ikiwiki is mostly a **reinforcement** of git-canonical-Markdown (UC-76) — its *new*
|
||||
contributions are (a) **compile-to-static** as a distinct projection/publish direction and
|
||||
(b) the **git-distributed-clone + ping** federation flavor. The static output is *not* a
|
||||
shard to attach (it's derived); attach the **source repo**.
|
||||
- VCS-agnostic backend layer is an interesting adapter-contract echo, but for shard-wiki the
|
||||
git case dominates.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Source-repo-as-shard, static-output-as-derived-projection** — never attach the build
|
||||
output as canonical (UC-79; relates UC-37/UC-56).
|
||||
2. **Git-replication + change-ping** as a named federation flavor beside fork/journal
|
||||
(fedwiki) and query-SERVICE (Wikibase) — a **subscribe/notify over a git mesh** (UC-31/UC-33).
|
||||
3. **Inbound feed aggregation** (RSS/Atom → pages) as an inbound projection pattern.
|
||||
|
||||
## 6. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-79 | Attach a git-backed **compile-to-static** wiki — the **git Markdown source is the shard**, compiled **static HTML is a derived publish/projection**; participate in **git-distributed clone federation** with change-**pings** | **new** |
|
||||
| — | XML-RPC pinger = subscribe/notify over a git mesh | enrich **UC-31** |
|
||||
| — | render union/shard to a static site (publish) | enrich **UC-56** |
|
||||
| — | static HTML as read-only regenerable backup | enrich **UC-37** |
|
||||
| — | wiki as a git branch-space; clones reconcile via merge | enrich **UC-33** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T4 (federation):** record **git-replication + change-ping** as a federation flavor — peers
|
||||
hold git clones, reconcile via merge, and notify with a ping to pull/rebuild. Distinct from
|
||||
fedwiki fork/journal and Wikibase `SERVICE`; complements the "Git-addressable coordination
|
||||
layer" mandate most literally.
|
||||
- **T6 (publish/projection):** **compile-to-static** is the canonical *outbound* projection —
|
||||
source (git) is canonical, static HTML is a regenerable derived view (publish/backup
|
||||
target). Reinforces projection-vs-canonical separation.
|
||||
- **Attach binding (T14):** attach the **source repo** (git Markdown), not the build output;
|
||||
shares the forge-wiki / `wiki/`-subdir git+Markdown adapter (parameterized by repo/path).
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Is the **pinger** (notify-peer-to-pull) modeled as shard-wiki's own subscribe/notify
|
||||
primitive, or only recognized when bridging two ikiwiki instances?
|
||||
2. Does shard-wiki ever *drive* a compile-to-static **publish** of the union (act as the
|
||||
ikiwiki-like compiler), or only attach existing ikiwiki source repos? (UC-56 scope.)
|
||||
3. Is **feed aggregation** (RSS→pages) an inbound projection mode shard-wiki offers generally
|
||||
(a feed-shard), or an ikiwiki-internal feature?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- ikiwiki.info — *ikiwiki* overview, *rcs* (VCS backends), *plugins* (`pinger`/`pingee`,
|
||||
`aggregate`, `openid`), *setup* / post-commit hooks
|
||||
- prior: `research/260608-federation-concepts/` (ikiwiki reference);
|
||||
`research/260614-forge-wikis-deep-dive/` (git-canonical Markdown contrast)
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UC **UC-79** carries the marker **⊟** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-31, UC-56, UC-37, UC-33. Architecture cross-refs:
|
||||
SHARD-WP-0002 T4 (git-replication+ping federation), T6 (compile-to-static publish), T14.
|
||||
49
research/260614-joplin-deep-dive/README.md
Normal file
49
research/260614-joplin-deep-dive/README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# 260614 — Joplin deep dive (SQLite-local, Markdown-on-sync, interchange-format attach)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Joplin** — the open-source Markdown note app — read through
|
||||
shard-wiki's lens. Joplin looks like "open-source Obsidian" but is a genuinely different
|
||||
point in the space: **content is Markdown, the local store is SQLite, and the *sync
|
||||
representation* is a folder of per-item Markdown+metadata files on a backend you choose**
|
||||
(filesystem, WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). It
|
||||
also ships **end-to-end encryption** and is itself a **sync layer over heterogeneous
|
||||
storage**.
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — SQLite local store (documented schema); Markdown content;
|
||||
notebooks → notes → resources + tags + to-dos; 32-char item IDs; `:/<id>` note links
|
||||
(rename-stable, page-level)
|
||||
- **Sync** — serialized to plain-text items on a chosen target; tombstones; **conflict
|
||||
notes** (keep-both) → enables the **"attach the sync mirror, not the app"** pattern on
|
||||
WebDAV/Nextcloud/S3
|
||||
- **E2EE** — items encrypted before leaving the device → **content opacity** (ciphertext
|
||||
at rest)
|
||||
- **Extension surfaces** — a **plugin API** (`joplin.data`/`workspace`/`contentScripts`/
|
||||
views) *and* a **local Data API** (REST on `localhost:41184`, token) used by the Web
|
||||
Clipper; plus JEX/Markdown/RAW export
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, sync/interchange attach, E2EE/content-opacity, plugin + Data API surfaces, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-60 attach a tool's documented sync/interchange representation on third-party storage
|
||||
without the app; UC-61 attach an encrypted-at-rest shard with content opacity);
|
||||
UC-31/36/38/40/51/55 enriched. Logged for `SHARD-WP-0002` (T11, T14): the
|
||||
interchange/sync-representation attach surface, **content opacity** as a proposed twelfth
|
||||
capability spectrum, a local-REST attach sub-mode, and format-aware file-store profiles.
|
||||
|
||||
**Key takeaways recorded:** the best attach surface is often a tool's **interchange/sync
|
||||
representation** (offline, app-independent — Joplin items on Nextcloud/WebDAV/S3, the
|
||||
INTENT-named backends), not its live store or API; **encryption-at-rest** demands a
|
||||
content-opacity capability so encrypted shards degrade to backup/structure-shell; and
|
||||
Joplin is the **file-sync-daemon boundary case** — shard-wiki attaches its mirror as
|
||||
pages and never re-syncs (not-a-sync-daemon).
|
||||
</content>
|
||||
269
research/260614-joplin-deep-dive/findings.md
Normal file
269
research/260614-joplin-deep-dive/findings.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# Findings — Joplin: SQLite locally, Markdown-on-sync, and the interchange-format attach
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — an open-source Markdown note app; a *candidate
|
||||
shard* whose distinctive trait is a **documented file-based sync/interchange format on
|
||||
third-party storage** plus optional **end-to-end encryption**
|
||||
Lens: shard-wiki — attachment surfaces, the "attach the sync mirror not the app"
|
||||
pattern, content opacity (encryption), note-level stable IDs, and the file-sync-daemon
|
||||
boundary
|
||||
|
||||
> Why Joplin earns a dive after Obsidian/Roam/Notion. It looks like "Obsidian but
|
||||
> open-source," yet its architecture is a genuinely different point in the space:
|
||||
> **content is Markdown but the local store is SQLite, and the *sync representation* is
|
||||
> a folder of per-item Markdown+metadata files on a backend you choose** (filesystem,
|
||||
> WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). That sync
|
||||
> mirror — not the app, not the SQLite DB — is the interesting attach surface, and it
|
||||
> lands squarely on the **WebDAV/Nextcloud backends INTENT already names**. Joplin also
|
||||
> ships **E2EE**, forcing the question of a shard whose content is *opaque ciphertext*,
|
||||
> and it is itself a **sync layer over heterogeneous storage** — the very "file-sync
|
||||
> daemon" shard-wiki says it is *not*, making Joplin a sharp boundary case.
|
||||
|
||||
Contrast set: Obsidian (file-over-app: native store *is* files), Roam (client DB,
|
||||
in-app API only), Notion (hosted DB, external API only), TWiki/Foswiki (file+RCS native
|
||||
store). Joplin is none of these exactly — **DB-local, files-on-sync** — which is the
|
||||
whole point.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture — SQLite local, Markdown content
|
||||
|
||||
- **Local store:** a **SQLite database** (`~/.config/joplin-desktop/database.sqlite`),
|
||||
schema documented. So, unlike Obsidian, the *native* local store is **not** plain
|
||||
files — you cannot point shard-wiki at a folder of `.md` and read the live store
|
||||
(you'd read SQLite, fragile while the app runs).
|
||||
- **Content:** notes are **Markdown** (Joplin-flavored: `:/imageId` resource links,
|
||||
internal `:/<id>` note links, checkboxes/to-dos). Markdown-first — INTENT-aligned at
|
||||
the *content* layer even though the *storage* layer is a DB.
|
||||
- **Model:** `folders` (notebooks, hierarchical) → `notes` → `resources` (attachments);
|
||||
cross-cutting `tags`; notes can be **to-dos**. Every item has a **32-char hex ID**;
|
||||
notes link by ID (`:/<id>`), so links survive rename/move (store-minted, page-level).
|
||||
- **Clients:** desktop (Electron), mobile, and a **terminal/CLI** app — multiple clients
|
||||
edit one logical store, reconciled by sync (§2).
|
||||
- **Revisions:** Joplin keeps **internal note history** locally — not git, not portable
|
||||
→ a *supplement* case (UC-36), like Notion/Confluence.
|
||||
|
||||
---
|
||||
|
||||
## 2. Sync — the distinctive layer (and the attach opportunity)
|
||||
|
||||
Joplin's headline feature is **sync to a backend of your choice**. On sync, the SQLite
|
||||
store is serialized to a **folder of plain-text items** — one file per note / notebook /
|
||||
tag / resource-metadata, each carrying the **Markdown body plus a metadata block**
|
||||
(`id`, `parent_id`, `type_`, `created_time`, `updated_time`, …) — written to the chosen
|
||||
target: **filesystem, WebDAV, Nextcloud, Dropbox, OneDrive, S3, Joplin Server, Joplin
|
||||
Cloud**. Deletions propagate via tombstones; concurrent edits across clients produce
|
||||
**conflict notes** (keep-both, not silent overwrite).
|
||||
|
||||
Two consequences for shard-wiki:
|
||||
|
||||
1. **Attach the sync mirror, not the app (UC-60).** A Nextcloud/WebDAV/S3/filesystem
|
||||
directory full of Joplin sync items is a **file-store shard** — readable *without
|
||||
Joplin running and without touching SQLite* — provided the adapter understands the
|
||||
**documented item format** (body + metadata footer, `:/id` links, resource items).
|
||||
This is a new attach pattern: the backend's **interchange/sync representation** as
|
||||
the attach surface, distinct from a native on-disk store (UC-40) or app files (UC-53).
|
||||
It also realizes INTENT's WebDAV/Nextcloud participants concretely.
|
||||
2. **Joplin is a file-sync layer over heterogeneous storage** — exactly what shard-wiki
|
||||
says it is *not* (INTENT: "not a file-sync daemon"). Joplin syncs **one** logical
|
||||
store across backends; shard-wiki **federates many** logical stores with wiki-page
|
||||
semantics. Attaching a Joplin sync mirror means **reading its file representation as
|
||||
pages**, never re-driving Joplin's sync — and never becoming a second sync engine
|
||||
over the same target (don't double-sync).
|
||||
|
||||
---
|
||||
|
||||
## 3. End-to-end encryption — content opacity
|
||||
|
||||
Joplin offers **E2EE**: items are encrypted **before** leaving the device, so the sync
|
||||
target holds **ciphertext** regardless of provider. (Security wart, noted: the master
|
||||
key is stored *in clear* in the local SQLite — irrelevant to attaching the mirror, but
|
||||
a reason never to treat the local DB as trusted.)
|
||||
|
||||
For shard-wiki this introduces a capability dimension the prior dives did not:
|
||||
**content opacity**. An encrypted Joplin sync target can be attached only as a
|
||||
**backup/mirror/structure-shell** participant — item IDs, counts, and change events may
|
||||
be visible, but **bodies are undecryptable without keys**. The adapter must **never
|
||||
present ciphertext as readable content** and must degrade to "present-but-opaque" with
|
||||
provenance. This extends the synthesis capability spectra with a proposed twelfth field,
|
||||
**content opacity** (`plaintext → encrypted-at-rest/opaque`), feeding `SHARD-WP-0002`
|
||||
T11 (§9).
|
||||
|
||||
---
|
||||
|
||||
## 4. Extension surfaces — a plugin host *and* a local REST API
|
||||
|
||||
Joplin exposes **two** programmatic surfaces (plus export):
|
||||
|
||||
**A. Plugin API (in-app host).** TypeScript plugins, distributed via **npm + a plugin
|
||||
repository**, loaded by the app:
|
||||
- `joplin.data` — CRUD over the data model, shaped like REST (GET/POST/PUT/DELETE) on
|
||||
`notes`/`folders`/`tags`/`resources`.
|
||||
- `joplin.workspace` — selection + events (note selected, content changed, sync events).
|
||||
- `joplin.contentScripts` — **markdown-it** render plugins and **CodeMirror** editor
|
||||
extensions (the syntax/rendering extension point).
|
||||
- `joplin.views` — panels, dialogs, toolbar/menu items, custom editors;
|
||||
`joplin.commands`, `joplin.settings`.
|
||||
|
||||
**B. Data API (local REST).** A REST service the desktop app serves on
|
||||
**`localhost:41184`** (the "clipper server"), **token-authenticated**, used by the Web
|
||||
Clipper and any local integration: endpoints `/notes`, `/folders`, `/tags`,
|
||||
`/resources` (incl. `/resources/:id/file`), `/revisions`, `/search`; verbs
|
||||
GET/POST/PUT/DELETE; pagination (`page`, `limit`≤100, `order_by`, `order_dir`). Notably
|
||||
**plugins can use this API even when the clipper server is off**.
|
||||
|
||||
**C. Export** — **JEX** (tar of raw items), **Markdown directory**, **RAW** (Joplin
|
||||
export directory), **ENEX** import (Evernote). Snapshot import/attach surface (UC-28/37).
|
||||
|
||||
So Joplin spans attachment modes: **in-app plugin host** (like Roam/Obsidian-plugin),
|
||||
**local-REST** (like Notion's external API but *localhost + app-must-run* — a sub-mode),
|
||||
the **sync-mirror file-store** (§2, the novel one), and **export snapshots**.
|
||||
|
||||
---
|
||||
|
||||
## 5. Joplin as a shard — capability profile
|
||||
|
||||
| Capability | Joplin | Notes for the adapter contract |
|
||||
|------------|--------|--------------------------------|
|
||||
| Read | **yes** | sync-mirror files (no app) · local Data API (app running) · plugin · export |
|
||||
| Write | **yes** | Data API / plugin (`joplin.data`); writing the sync mirror directly is risky (Joplin owns that format) |
|
||||
| Write granularity | **per-note (page)** | notebooks/notes/resources; not block-level |
|
||||
| Identity / addressing | **32-char item ID; `:/<id>` links** | store-minted, **page-level**, survives rename (between Obsidian path and Roam block-UUID) (UC-51) |
|
||||
| Structure | notebooks (hierarchy) + tags + to-dos | modest; metadata in the sync item footer (in-text on the mirror) |
|
||||
| History | **internal revisions, not portable** | supplement via coordination journal (UC-36) |
|
||||
| Native query | search API (FTS) | delegate text search; no datalog/DB-query (weaker than Roam/Notion) |
|
||||
| Subscribe | sync is **poll-based**; conflict notes | poll; conflicts surface as keep-both items (UC-31, UC-07) |
|
||||
| Content opacity | **optional E2EE → ciphertext** | **new dimension**: opaque-at-rest; mirror usable only as backup/structure without keys (UC-61) |
|
||||
| Transclusion | none (note-level links only) | — |
|
||||
| Diff / merge | none native | git-level if mirror is on a git-backed target |
|
||||
| Publish | via export / static-site pipelines | outbound (UC-56) |
|
||||
| Content types | Markdown + **resources** (attachments) | non-Markdown assets with IDs (UC-55) |
|
||||
| Attach modes | sync-mirror file-store · local-REST · plugin · export | multi-mode, per-binding (UC-60, UC-57, UC-38) |
|
||||
|
||||
Verdict: a solid **Markdown-first candidate shard** whose best attach surface is the
|
||||
**sync mirror on a file/WebDAV/S3 target** (offline, app-independent, INTENT-named),
|
||||
with the local Data API or a plugin for live write-through. Standout: the
|
||||
interchange-format attach (UC-60) and the encryption/opacity case (UC-61). Limits: DB
|
||||
local store, note-level (not block) addressing, internal-only history, weak query.
|
||||
|
||||
---
|
||||
|
||||
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 6.1 Reinforcements
|
||||
|
||||
- **Interchange-format attach** validates that a shard's *documented sync/export
|
||||
representation* — not just its live store or API — is a legitimate, often *best*,
|
||||
attach surface (offline, app-independent). Generalizes beyond Joplin (UC-60).
|
||||
- **WebDAV/Nextcloud/S3 become concrete** participants via Joplin's sync targets — the
|
||||
backends INTENT names, now with a real format to parse.
|
||||
- **Conflict-as-data** (Joplin conflict notes = keep-both) is union-without-erasure in
|
||||
the wild (reinforces UC-07, divergence; consensus policy T9).
|
||||
- **Store-minted, rename-stable note IDs** (`:/id`) confirm the addressing spectrum's
|
||||
middle (page-level stable ID) between path (Obsidian) and block-UUID (Roam/Notion)
|
||||
(UC-51).
|
||||
|
||||
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Joplin is a sync daemon; shard-wiki is not.** Attach the **mirror as pages**; never
|
||||
re-implement Joplin's sync or run a competing sync over the same target. (INTENT
|
||||
not-a-file-sync-daemon.)
|
||||
2. **DB-local; don't read SQLite live.** The native store is a DB; treat it as opaque.
|
||||
Use the sync mirror, the Data API, or a plugin — not the live `database.sqlite`.
|
||||
3. **Joplin owns the sync format; treat the mirror read-mostly.** Writing items into the
|
||||
sync folder behind Joplin's back risks corruption/conflict; prefer overlay/projection
|
||||
or write through the Data API/plugin (overlay before mutation, no silent mutation).
|
||||
4. **Encrypted shards are opaque.** Never present ciphertext as content; degrade to
|
||||
backup/structure with provenance (UC-61, graceful degradation).
|
||||
|
||||
### 6.3 What Joplin teaches that shard-wiki should keep
|
||||
|
||||
- Add **"interchange/sync representation"** as a recognized attach surface in the
|
||||
contract — sometimes preferable to the live store/API (UC-60).
|
||||
- Add **content opacity** (encryption-at-rest) as a capability-profile field so encrypted
|
||||
shards degrade correctly (UC-61).
|
||||
- Expect **multi-client concurrent editing reconciled by the backend's own sync**
|
||||
(desktop/mobile/CLI) — a different concurrency than single-app file edits (extends
|
||||
UC-53) or multi-user server writes; conflicts may pre-exist as keep-both items.
|
||||
|
||||
---
|
||||
|
||||
## 7. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-59**. New UCs **UC-60, UC-61** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a tool's documented sync/interchange representation on a third-party storage target** (Joplin items on Nextcloud/WebDAV/S3) as a file-store shard, without the app | **UC-60 (new)** |
|
||||
| **Attach an encrypted-at-rest shard** (E2EE sync target): content opaque without keys; participate as backup/mirror/structure-shell, never presenting ciphertext as readable | **UC-61 (new)** |
|
||||
| Store-minted, rename-stable **note-level** IDs (`:/id`) — the middle of the addressing spectrum | **enriches UC-51** |
|
||||
| Native store can be a **DB while the sync representation is files** → attach surface ≠ native store | **enriches UC-40** |
|
||||
| Internal revisions, not portable → supplement | **enriches UC-36** |
|
||||
| Dual surfaces: in-app plugin host + **local Data API** (localhost, app-running) | **enriches UC-38** (links UC-57) |
|
||||
| Resources/attachments = non-Markdown assets with IDs | **enriches UC-55** |
|
||||
| Poll-based sync; conflict notes (keep-both) | **enriches UC-31** (links UC-07) |
|
||||
|
||||
---
|
||||
|
||||
## 8. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Attachment-mode taxonomy gains a refinement:** within file-store, distinguish
|
||||
*native on-disk store* (Obsidian/TWiki, UC-40) from *interchange/sync representation*
|
||||
(Joplin mirror, UC-60). Both are "files," but format ownership and write-safety differ.
|
||||
(T14.)
|
||||
- **Add a "content opacity" capability field** (proposed **twelfth spectrum**:
|
||||
`plaintext → encrypted-at-rest/opaque`) so encrypted shards degrade to
|
||||
backup/structure-shell. Feeds T11 the next time the eleven spectra
|
||||
(`research/260614-shard-spectrum-synthesis/findings.md` §2) are revised. (T11/T14.)
|
||||
- **Local-REST as an attach sub-mode** (localhost, app-must-run, token) sits between
|
||||
in-engine-host (UC-38) and external-API (UC-57) — note in T14.
|
||||
- **Format-aware file-store adapters:** the contract should let a file-store adapter
|
||||
declare a *format profile* (plain Markdown vs Joplin-item vs Foswiki-PlainFile) so the
|
||||
same "folder of files" attach can parse tool-specific item formats. (T11/T14.)
|
||||
|
||||
---
|
||||
|
||||
## 9. Open questions (for spec / workplans)
|
||||
|
||||
1. For an **encrypted shard** (UC-61), what *is* visible without keys — item IDs/counts/
|
||||
timestamps (structure shell) or nothing? Does shard-wiki ever hold keys, or only ever
|
||||
treat such shards as opaque backups?
|
||||
2. Is **writing to a tool's sync mirror** (UC-60) ever safe, or are interchange-format
|
||||
shards **read/projection/overlay-only** by policy (Joplin owns the format)?
|
||||
3. Does shard-wiki parse Joplin's **item format** via a dedicated format profile, or
|
||||
only attach the **export** (JEX/Markdown dir) as a cleaner snapshot (UC-37)?
|
||||
4. How do we avoid **double-sync** when a shard's storage target is itself driven by a
|
||||
sync daemon (Joplin) we don't control?
|
||||
|
||||
---
|
||||
|
||||
## 10. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Joplin Data API reference (https://joplinapp.org/help/api/references/rest_api/) | Local REST on `localhost:41184`, token auth, endpoints, 32-char IDs, item fields, pagination |
|
||||
| Joplin Plugin API docs (https://joplinapp.org/api/references/plugin_api/classes/joplin.html) | `joplin.data` / `joplin.workspace` / `joplin.contentScripts` / views / settings |
|
||||
| Joplin — Extending Joplin (https://joplinapp.org/help/api/) | Plugin distribution (npm + repository), Data API vs plugin access |
|
||||
| Joplin — WebDAV / Nextcloud sync (https://joplinapp.org/help/apps/sync/nextcloud/) | Sync targets; items as plain-text files on the target |
|
||||
| Joplin FAQ + community (https://joplinapp.org/help/faq/) | SQLite local store; Markdown content; E2EE before leaving device |
|
||||
| Obsidian vs Joplin comparison (https://petronellatech.com/blog/obsidian-vs-joplin/) | Architecture contrast (SQLite+sync vs file-over-app), E2EE framing |
|
||||
|
||||
Cross-references: `research/260614-obsidian-deep-dive/findings.md` (file-over-app
|
||||
contrast), `research/260614-notion-deep-dive/findings.md` (external-API, scoped grant),
|
||||
`research/260614-shard-spectrum-synthesis/findings.md` (the spectra this extends),
|
||||
`spec/UseCaseCatalog.md` (UC-31, UC-36, UC-38, UC-40, UC-51, UC-55, UC-57),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
|
||||
|
||||
---
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
- New UCs: **UC-60, UC-61** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-31, UC-36, UC-38, UC-40, UC-51, UC-55** (links UC-07, UC-57).
|
||||
- Architecture (no UC): interchange/sync-representation attach surface; **content
|
||||
opacity** as a proposed twelfth capability spectrum; local-REST sub-mode; format-aware
|
||||
file-store profiles → `SHARD-WP-0002` (T11, T14).
|
||||
- Boundary recorded: Joplin is a **sync daemon over one store**; shard-wiki attaches its
|
||||
**mirror as pages** and does not re-sync — and treats DB-local store and encrypted
|
||||
content as opaque (INTENT not-a-sync-daemon, graceful degradation, no silent mutation).
|
||||
</content>
|
||||
35
research/260614-jupyter-deep-dive/README.md
Normal file
35
research/260614-jupyter-deep-dive/README.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# 260614 — Jupyter Notebooks deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T3**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Jupyter Notebooks**: the **`.ipynb` JSON** document (ordered cells:
|
||||
markdown / code+outputs / raw), **kernels**, **embedded computed outputs** (MIME bundles),
|
||||
and **execution-count provenance**. The dominant modern computational document and the
|
||||
concrete case where the **derived output is captured and stored *inside* the source** with
|
||||
real-but-fragile provenance.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Tests the T1 **replication- vs derivation-projection** split on the dominant real artifact
|
||||
and adds the wrinkle that **outputs are stored back inside the source** — the source/
|
||||
projection line runs *through* the document.
|
||||
- The page model (T12) must carry a **notebook shape**: ordered cells with code cells owning
|
||||
embedded computed outputs that have **weak execution provenance** (run order, environment
|
||||
not captured).
|
||||
- Non-Markdown + lossy translation (T15): JSON+MIME bundles; nbconvert→Markdown is lossy and
|
||||
directional. JSON diffs are noisy → Jupytext text-pairing / nbdime (T13).
|
||||
|
||||
## Yield
|
||||
|
||||
- **UC-84** (new): attach/project a computational notebook preserving cell structure +
|
||||
embedded outputs, surfacing outputs as **snapshots with weak execution provenance**;
|
||||
re-execution **capability-gated**, default = present snapshot + static render.
|
||||
- Enrich **UC-54, UC-55, UC-59, UC-35**; links **UC-32, UC-83, UC-79**.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | `.ipynb` model, kernels/execution-count fragility, ecosystem (nbconvert/Jupytext/papermill/nbdime/nbstripout), capability profile, INTENT mapping, UC seed, architecture notes, open questions |
|
||||
185
research/260614-jupyter-deep-dive/findings.md
Normal file
185
research/260614-jupyter-deep-dive/findings.md
Normal file
@@ -0,0 +1,185 @@
|
||||
# Jupyter Notebooks — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T3 · **Subject:** Jupyter Notebooks — the
|
||||
`.ipynb` JSON document, kernels, embedded computed outputs, execution provenance.
|
||||
|
||||
## Why this dive
|
||||
|
||||
T1 (literate programming) established **one source → derived projections** and split
|
||||
**replication-projection** from **derivation-projection**. Jupyter is the *dominant modern
|
||||
computational document* and the concrete case where the **derived output is captured and
|
||||
stored inside the source** — a non-Markdown, partially-executable content type whose
|
||||
provenance is real but **fragile**. It is the most plausible concrete "computational shard"
|
||||
content type, so it tests the page model (T12), lossy translation (T15), and the
|
||||
output-provenance question head-on.
|
||||
|
||||
## 1. The `.ipynb` document model
|
||||
|
||||
A notebook is a single **JSON document** (`nbformat`), not Markdown:
|
||||
|
||||
- **`cells[]`** — an ordered list. Each cell has a `cell_type`:
|
||||
- `markdown` — prose (Markdown + LaTeX), the human-readable part.
|
||||
- `code` — source text (`source`), plus an **`execution_count`** and an **`outputs[]`**
|
||||
array captured from the last run.
|
||||
- `raw` — passthrough.
|
||||
- **`outputs[]`** (per code cell) carry results inline: `stream` (stdout/stderr),
|
||||
`execute_result` / `display_data` (a **MIME bundle** — `text/plain`, `text/html`,
|
||||
`image/png` base64, `application/json`, vendor MIME types), and `error` (traceback).
|
||||
- **`metadata`** at notebook and cell level (`kernelspec`, `language_info`, tags like
|
||||
`hide-input`, `scrolled`, slideshow roles).
|
||||
|
||||
So an `.ipynb` is **source + last-run computed outputs + environment metadata, fused in one
|
||||
JSON file**. The Markdown cells are an *island* inside a JSON envelope — relevant to how
|
||||
shard-wiki extracts/round-trips content.
|
||||
|
||||
## 2. Kernels and execution
|
||||
|
||||
- A **kernel** is a separate language process (IPython, IRkernel, IJulia, …) speaking the
|
||||
Jupyter messaging protocol (ZeroMQ). The document is **decoupled from the kernel**: the
|
||||
`.ipynb` persists *captured* outputs; re-running requires a live kernel + the right
|
||||
environment.
|
||||
- **`execution_count`** numbers the order cells were *run*, which **need not match document
|
||||
order** — the infamous **hidden-state / out-of-order execution** problem: stored outputs
|
||||
may reflect a run sequence that no longer corresponds to top-to-bottom reading.
|
||||
- Reproducibility therefore depends on **out-of-band state**: package versions, data files,
|
||||
environment, random seeds — none captured by `nbformat` itself.
|
||||
|
||||
**Consequence for shard-wiki:** the captured outputs are a **snapshot projection with weak
|
||||
provenance** — honest treatment must mark them as "computed at run N, environment not
|
||||
guaranteed," never as live or authoritative truth.
|
||||
|
||||
## 3. The ecosystem (relevant to attach/project/translate)
|
||||
|
||||
- **nbconvert** — derives other forms from a notebook: HTML, Markdown, LaTeX/PDF, slides,
|
||||
script. This is **derivation-projection** (T1): notebook source → rendered view, lossy in
|
||||
both directions (HTML keeps outputs; `--to script` keeps only code, like `tangle`).
|
||||
- **Jupytext** — represents a notebook **as** a `.py`/`.md` text file (pairing), making it
|
||||
**git-diffable plain text** and round-trippable — directly relevant to storing notebooks
|
||||
in a git shard without JSON-diff noise.
|
||||
- **papermill** — parameterize + execute a notebook to produce a new output notebook
|
||||
(notebook as a runnable template — a *derivation with inputs*).
|
||||
- **JupyterLab / Notebook / nbviewer / Colab** — front-ends; nbviewer renders a static
|
||||
read-only projection from a URL (a natural projection target).
|
||||
- **`nbstripout`** — strips outputs before commit: teams treat **outputs as derived noise**,
|
||||
keeping only source under version control — an explicit "source canonical, outputs
|
||||
derived" stance mirroring T1.
|
||||
|
||||
## 4. Capability profile (as a shard / content type)
|
||||
|
||||
| Dimension (synthesis spectrum) | Jupyter notebook |
|
||||
|--------------------------------|------------------|
|
||||
| Attachment mode | file-store (`.ipynb` JSON in a repo) or via Jupyter Server REST API |
|
||||
| Addressing granularity | document; **cell** as sub-address (by index / id; `nbformat 4.5+` adds stable cell `id`) |
|
||||
| Content identity | file path; cell `id` (4.5+) else positional |
|
||||
| Structure | **ordered cell list** (markdown / code+outputs / raw); MIME-bundle outputs |
|
||||
| History | VCS on the file; **JSON diffs are noisy** unless paired (Jupytext) or stripped |
|
||||
| Merge model | git on JSON (poor) → **paired text** (good) or nbdime (cell-aware diff/merge) |
|
||||
| Native query | none |
|
||||
| Translation | nbconvert → HTML/MD/script/PDF (lossy, directional); Jupytext text pairing |
|
||||
| Write granularity | file / **cell** |
|
||||
| Operational envelope | a kernel + environment to (re)execute; static render needs none |
|
||||
| Content opacity | **mixed**: source transparent; outputs = MIME blobs (some opaque, e.g. base64 PNG) |
|
||||
| Provenance | `execution_count` (weak, out-of-order); environment **not** captured |
|
||||
| **Computed-output** | **stored inline**, snapshot, reproducibility out-of-band |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Replication- vs derivation-projection (T1) confirmed and extended.** nbconvert (→HTML/
|
||||
script) and nbviewer are derivation-projections; `--to script` is literally `tangle`.
|
||||
Jupyter adds a third wrinkle: **the derived output is also stored back inside the source**
|
||||
(captured outputs), so the "source vs projection" line runs *through* the document.
|
||||
- **Union without erasure / provenance honesty.** Captured outputs must be surfaced **as
|
||||
snapshots with weak provenance** (run N, environment unguaranteed) — a concrete instance
|
||||
of "never hide freshness/authorship." The out-of-order `execution_count` is exactly the
|
||||
kind of fragility shard-wiki must *show*, not paper over.
|
||||
- **Non-Markdown content + lossy translation (UC-55/UC-59).** `.ipynb` is JSON with embedded
|
||||
MIME-bundle outputs; any Markdown projection is **lossy** (loses live outputs, kernel,
|
||||
rich MIME). Surface the lossiness; keep the JSON as canonical payload (T12/T15).
|
||||
- **Markdown island.** Markdown cells fit the text-first model, but only as *fragments
|
||||
inside* a JSON envelope — the adapter extracts/round-trips them, it does not pretend the
|
||||
notebook is a Markdown page.
|
||||
|
||||
### Divergences / boundaries
|
||||
|
||||
- **shard-wiki is not a kernel host.** Re-execution (driving a kernel) is out of scope/
|
||||
capability-gated; default treatment is **attach + present captured outputs as a snapshot
|
||||
projection** + offer nbconvert-style static render. Executing/parameterizing (papermill)
|
||||
is an optional capability, never assumed.
|
||||
- **Outputs-in-source is an anti-pattern to respect, not adopt.** Teams strip/pair outputs
|
||||
precisely because mixing derived data into the source breaks diffs. shard-wiki should
|
||||
prefer the **source-canonical, outputs-as-derived** reading (Jupytext pairing / nbstripout
|
||||
ethos) and treat stored outputs as a capturable projection.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Computational-notebook as a first-class content type** with cell structure + inline
|
||||
**computed outputs carrying (weak) execution provenance** — UC-84.
|
||||
2. **Outputs = derivation-projection snapshot** (T1 vocabulary): regenerable only with a
|
||||
kernel+environment; degrade gracefully to the stored snapshot / static render.
|
||||
3. **Cell-level addressing** (stable cell `id`, nbformat 4.5+) as the sub-page granularity
|
||||
for transclusion/anchoring (UC-32/UC-35).
|
||||
4. **Text-pairing (Jupytext)** as the git-friendly storage strategy — feeds the
|
||||
history-portability thread (poor JSON diffs → paired text / nbdime).
|
||||
|
||||
## 6. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-84 | Attach/project a **computational notebook** (`.ipynb`): preserve **cell structure** (markdown / code / output) and **embedded computed outputs**, surfacing each output **as a snapshot with its (weak) execution provenance** (run count, environment not guaranteed) — re-execution is **capability-gated**, default is present-the-snapshot + offer a static rendered projection | **new** |
|
||||
| — | Notebook JSON / MIME-bundle outputs = non-Markdown content; Markdown projection is lossy | enrich **UC-55**, **UC-59** |
|
||||
| — | Computed/evaluated cell = computation-defined content | enrich **UC-54** |
|
||||
| — | Cell `id` (nbformat 4.5+) = sub-page address for anchor/transclusion | enrich **UC-35**, links **UC-32** |
|
||||
| — | Stored outputs as derived snapshot (nbstripout/Jupytext ethos) = source-canonical/outputs-derived | links **UC-83**, **UC-79** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12 (page model):** add **computational-notebook** as a page shape — an **ordered cell
|
||||
list** where code cells own **embedded computed outputs** (MIME bundles) with weak
|
||||
execution provenance. Distinct from prose, typed records, query-defined, inline-embedded
|
||||
objects (Quip/Notion), typed-graph (Wikibase), and the literate one-source-many-projection
|
||||
shape (UC-83). The defining new attribute: **derived output stored *inside* the source**.
|
||||
- **T15 (translation / fidelity):** `.ipynb` is non-Markdown; nbconvert→Markdown is **lossy
|
||||
and directional** (drops live outputs/kernel/rich MIME). Keep JSON canonical; any Markdown
|
||||
is a projection. MIME-bundle outputs map to the content-opacity spectrum (text→html→base64
|
||||
image = transparent→opaque).
|
||||
- **T13 (history):** JSON diffs are **noisy**; record **text-pairing (Jupytext)** and
|
||||
**cell-aware diff/merge (nbdime)** as history-portability strategies for embedded-output
|
||||
documents. Reinforces "source-canonical, outputs-derived."
|
||||
- **T16 (projection):** captured outputs are a **derivation-projection snapshot**;
|
||||
re-execution (kernel) and parameterized execution (papermill) are **capabilities**, not
|
||||
assumptions; degrade to the stored snapshot / nbviewer-style static render.
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Does shard-wiki ever **re-execute** a notebook (host/broker a kernel), or strictly
|
||||
attach + present captured outputs + static render? (Same scope boundary as UC-83/UC-56
|
||||
"do we ever drive the derivation.")
|
||||
2. Is **UC-84** distinct from **UC-83**, or is a notebook just the "outputs-stored-in-source"
|
||||
special case of the literate one-source-many-projection pattern? (Kept separate: UC-84's
|
||||
defining trait is *captured derived output embedded in the canonical source with weak
|
||||
provenance* — a page-model attribute UC-83 doesn't carry.)
|
||||
3. How are **MIME-bundle outputs** represented in the page model — opaque provenance-tagged
|
||||
blobs, a typed-asset registry (UC-55 open question #10), or selected-MIME projection?
|
||||
4. Default storage: attach `.ipynb` **as-is** (JSON, noisy diffs) or prefer a **paired text
|
||||
representation** when the shard is a git repo? (Policy → configurable.)
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- Jupyter `nbformat` reference (cells, outputs, MIME bundles, cell `id` 4.5+);
|
||||
Jupyter messaging protocol / kernels docs.
|
||||
- **nbconvert**, **nbviewer**, **JupyterLab**, **Colab** docs.
|
||||
- **Jupytext**, **papermill**, **nbdime**, **nbstripout** project docs.
|
||||
- prior: `research/260614-literate-programming-deep-dive/` (replication- vs
|
||||
derivation-projection, UC-83); `research/260614-notion-deep-dive/` (block-JSON,
|
||||
external-API), `research/260614-quip-deep-dive/` (inline embedded objects, UC-55/58/59).
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UC **UC-84** carries the marker **⊜** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md` (true lineage = this dive). Enriched: UC-54, UC-55, UC-59, UC-35;
|
||||
links UC-32, UC-83, UC-79. Architecture cross-refs: SHARD-WP-0002 T12 (notebook page shape:
|
||||
outputs embedded in source), T15 (lossy non-Markdown translation; MIME opacity), T13
|
||||
(paired-text / nbdime history), T16 (output = derivation-projection snapshot; kernel =
|
||||
capability).
|
||||
35
research/260614-literate-programming-deep-dive/README.md
Normal file
35
research/260614-literate-programming-deep-dive/README.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# 260614 — Literate Programming (Knuth's WEB / weave / tangle) deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T1**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **literate programming** — Knuth's WEB and the **`weave`/`tangle`** model
|
||||
— as the deepest ancestor of shard-wiki's **projection** and **transclusion** ideas
|
||||
applied to *executable* content. The keystone: **one canonical source → two co-derived
|
||||
projections** (typeset docs via `weave`, compilable code via `tangle`), plus **named code
|
||||
chunks** assembled by reference (transclusion).
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Establishes **one-source-many-projections** as a page-model + projection pattern that
|
||||
*generalizes* compile-to-static (UC-79, single output) to **N co-equal, semantically
|
||||
different** derived views — feeds SHARD-WP-0002 **T12/T16**.
|
||||
- Splits projection into **replication-projection** (lazy cache, current default) vs
|
||||
**derivation-projection** (transform/compile/weave/evaluate) — a distinction the rest of
|
||||
the computational batch (notebooks, REPLs) extends.
|
||||
- Named chunks are the **executable-content face of transclusion / compose-by-reference**
|
||||
(UC-32 / UC-44).
|
||||
|
||||
## Yield
|
||||
|
||||
- **UC-83** (new): attach a single-source-multiple-projection (literate) artifact; present
|
||||
each derived view with output→source provenance; edits target the source.
|
||||
- Enrich **UC-32**, **UC-44**, **UC-79**; links **UC-54** (computed/evaluated projection,
|
||||
→ T3 Jupyter).
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | WEB model, named-chunk transclusion, descendants (noweb/org-babel/knitr/Jupytext), capability profile, INTENT mapping, UC seed, architecture notes, open questions |
|
||||
180
research/260614-literate-programming-deep-dive/findings.md
Normal file
180
research/260614-literate-programming-deep-dive/findings.md
Normal file
@@ -0,0 +1,180 @@
|
||||
# Literate Programming (Knuth's WEB / weave / tangle) — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T1 · **Subject:** Donald Knuth's WEB
|
||||
system and the literate-programming model (`weave`/`tangle`, named chunks).
|
||||
|
||||
## Why this dive
|
||||
|
||||
SHARD-WP-0004 asks the carried question: *can a shard-wiki page be a source that is
|
||||
woven/evaluated into rendered forms, and how do projection/transclusion/provenance treat
|
||||
the source vs the output?* Literate programming is the **deepest ancestor** of that idea.
|
||||
Knuth (1984) inverted the program/comment relationship: you write a **document** for
|
||||
humans whose fragments happen to also be the program. From the **one WEB source** two
|
||||
tools derive two artifacts: **`weave` → typeset documentation** (TeX) and **`tangle` →
|
||||
compilable code** (Pascal/C/…). This is *one source, two projections* in its purest,
|
||||
oldest form — the conceptual root of shard-wiki's **projection** and **transclusion**.
|
||||
|
||||
## 1. The WEB model: one source, two tools
|
||||
|
||||
A WEB file interleaves prose and code in author-chosen order (the order that best
|
||||
*explains*, not the order the compiler needs). Two programs read it:
|
||||
|
||||
- **`weave`** produces a `.tex` file → typeset documentation: prose, pretty-printed code,
|
||||
a cross-reference index of where each chunk and identifier is defined/used.
|
||||
- **`tangle`** produces a compilable source file → reorders and expands the code chunks
|
||||
into the sequence the compiler demands, strips the prose, macro-expands references.
|
||||
|
||||
The crucial property: **the two outputs are co-derived from one canonical source and are
|
||||
semantically different audiences** (human reader vs compiler). Neither output is the
|
||||
source; both are **regenerable, disposable derivations**. Editing happens on the WEB; you
|
||||
never edit the woven `.tex` or the tangled `.c`.
|
||||
|
||||
## 2. Named chunks = transclusion of code fragments
|
||||
|
||||
The organizing primitive is the **named section / code chunk**:
|
||||
|
||||
- A chunk is declared `<<name>>=` and **referenced** elsewhere as `<<name>>`. `tangle`
|
||||
expands references in place (recursively) to assemble the final program.
|
||||
- A chunk name can be **defined in multiple places** and is *accreted* (later `+=`
|
||||
additions append) — so one logical unit is authored across scattered locations.
|
||||
- References can appear before definitions; resolution is by name, not by position.
|
||||
|
||||
This is **transclusion by reference** (UC-32) and **compose-by-reference / manifest**
|
||||
(UC-44, the EDL/Xanadu lineage): the document is a graph of named fragments assembled at
|
||||
derivation time. Knuth's chunk graph is the same shape as Xanadu's reference-not-copy and
|
||||
shard-wiki's "compose a page from referenced parts" — applied to *executable* content and
|
||||
resolved by a build tool rather than a viewer.
|
||||
|
||||
## 3. The descendants (noweb, CWEB, org-babel, Sweave/knitr, Jupytext)
|
||||
|
||||
- **CWEB** (Knuth/Levy): WEB for C. **noweb** (Ramsey): language-agnostic, minimal markup
|
||||
(`<<chunk>>`), `notangle`/`noweave` — proof that the *model* (chunks + two projections)
|
||||
is separable from any one language or typesetter.
|
||||
- **org-babel** (Emacs Org-mode): named source blocks, `:noweb` references, **tangle** to
|
||||
files **and evaluate** blocks inline (results captured back into the document) — literate
|
||||
programming fused with notebook execution (bridges to T2/T3).
|
||||
- **Sweave / knitr** (R): weave prose + R, executing code and **interleaving computed
|
||||
results/figures** into the woven document — adds the *computed-output* dimension that
|
||||
Jupyter (T3) centers on.
|
||||
- **Jupytext**: represents a Jupyter notebook **as** a literate text/Markdown source —
|
||||
closing the loop: the notebook (T3) becomes a weave/tangle-style plain-text source.
|
||||
|
||||
The throughline: **one canonical source → N derived projections**, where projections may
|
||||
be (a) reformatted (weave), (b) reordered/extracted (tangle), or (c) **evaluated**
|
||||
(org-babel/knitr) — the evaluated case is exactly the computational-page question.
|
||||
|
||||
## 4. Capability profile (as a would-be shard / page type)
|
||||
|
||||
| Dimension (synthesis spectrum) | Literate-programming source |
|
||||
|--------------------------------|-----------------------------|
|
||||
| Attachment mode | file-store (a WEB/`.nw`/`.org` text source in a repo) |
|
||||
| Addressing granularity | document; **named chunk** as sub-page address |
|
||||
| Content identity | source file path + chunk name (name-resolved, not position) |
|
||||
| Structure | **graph of named chunks** assembled by reference |
|
||||
| History | whatever VCS holds the source (git) — text, diffable |
|
||||
| Merge model | text/git merge on the source |
|
||||
| Native query | none; `weave` emits a cross-reference index (derived) |
|
||||
| Translation | source → woven docs **and** tangled code (build-time) |
|
||||
| Write granularity | file / **chunk** (text region) |
|
||||
| Operational envelope | a build tool (`tangle`/`weave`/`noweb`/babel) |
|
||||
| Content opacity | transparent text |
|
||||
| Provenance | VCS author/time; chunk cross-ref maps output→source location |
|
||||
| **Projection model** | **one source → many co-equal derived projections** (new emphasis) |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Projection (canonical vs derived).** Literate programming is the archetype of
|
||||
"canonical source, regenerable derived view" — the same principle as ikiwiki
|
||||
compile-to-static (UC-79), but generalized: **two-plus co-equal projections** from one
|
||||
source (docs *and* code), not a single output. Strengthens the rule *never confuse a
|
||||
projection for the source*.
|
||||
- **Transclusion / compose-by-reference.** Named chunks are transclusion (UC-32) and a
|
||||
manifest of referenced parts (UC-44) — resolved at derivation time. Confirms
|
||||
transclusion=clone=embed=reference as one primitive that also covers *fragment assembly
|
||||
of executable content*.
|
||||
- **Markdown-first but backend-neutral page model.** noweb/org/Jupytext show the literate
|
||||
source can *be* Markdown-ish plain text — so a "literate page" fits the text-first model;
|
||||
the *derivations* are the non-text part.
|
||||
- **Mechanism over policy.** weave/tangle are mechanisms; *which* projections to
|
||||
materialize, when to regenerate, and where outputs go stay configurable.
|
||||
|
||||
### Divergences / boundaries
|
||||
|
||||
- **shard-wiki is not a build system.** It should *recognize and present* a
|
||||
source-with-projections (attach the source, surface derived views with provenance), not
|
||||
re-implement `tangle`/kernels. Materializing a projection may delegate to the source's
|
||||
own tool or be capability-gated to "snapshot only."
|
||||
- **The interesting projection is derivation, not caching.** shard-wiki's base projection
|
||||
is cache-like (lazy copy of remote content, UC-53/57). Weave/tangle is a *different*
|
||||
projection species: **transform/derive** one source into rendered forms. The contract
|
||||
should model projection as having (at least) two kinds: **replication-projection** and
|
||||
**derivation-projection** (compile/weave/evaluate).
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **One-source-many-projections** as a first-class page-model + projection pattern
|
||||
(generalizes UC-79's single output) — see UC-83.
|
||||
2. **Named-chunk transclusion** as the executable-content face of UC-32/UC-44 (assembly by
|
||||
reference at derivation time).
|
||||
3. **Output→source provenance** (the woven cross-ref index): every derived view must point
|
||||
back to the exact source location it came from — never present derived output without
|
||||
that link (union-without-erasure for derivations).
|
||||
|
||||
## 6. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-83 | Attach a **single-source-multiple-projection** artifact (a literate/woven source): treat the source as canonical, present each **derived projection** (e.g. a documentation view *and* a code view) with **provenance back to the one source**, edits target the source and projections regenerate (delegated to the source's tool or degraded to a static snapshot) | **new** |
|
||||
| — | Named chunks `<<name>>` = transclusion / compose-by-reference of (executable) fragments | enrich **UC-32**, **UC-44** |
|
||||
| — | Generalize compile-to-static (single output) to **N co-equal projections** from one source | enrich **UC-79** |
|
||||
| — | Computed/evaluated projection (org-babel/knitr) = derivation-projection with results | links **UC-54**, foreshadows **T3** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12 (page model):** add **one-source-many-projections** as a page-model shape — a page
|
||||
may be a *source* whose presented forms are **derivations** (woven docs, tangled code,
|
||||
evaluated results), each carrying output→source provenance. Distinct from prose, typed
|
||||
records, query-defined, and inline-embedded objects already logged.
|
||||
- **T16 (projection):** split projection into **replication-projection** (lazy cache of
|
||||
remote content — current default) vs **derivation-projection** (transform/compile/weave/
|
||||
evaluate a source into rendered forms). Derivation-projection is regenerable, may
|
||||
delegate to an external tool, and degrades to a captured snapshot when the tool is
|
||||
absent (graceful degradation).
|
||||
- **Transclusion (T16):** named-chunk-by-name resolution is a transclusion variant where
|
||||
the *target is a fragment resolved by name across the source*, assembled at derivation
|
||||
time — a concrete shape for UC-32/UC-44 mechanics.
|
||||
- **Capability gating:** "can derive projection X" is a capability; a shard that can't run
|
||||
the tool still exposes the source + any pre-built/snapshot projections (UC-83 degrade).
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Does shard-wiki ever *drive* a derivation (run weave/tangle/evaluate), or only attach
|
||||
sources and surface pre-built projections + snapshots? (Same scope question as UC-56
|
||||
"do we ever compile-to-static ourselves," now for literate sources.)
|
||||
2. Is **UC-83** distinct enough from UC-79 (compile-to-static) to stand alone, or should
|
||||
UC-79 be re-read as the single-output special case of UC-83's N-projection general case?
|
||||
(Recorded as a possible later consolidation; kept separate now because UC-83's
|
||||
projections are *co-equal and semantically different audiences*, not one publish target.)
|
||||
3. How is **output→source provenance** represented when a derived line came from a chunk
|
||||
accreted across several source locations (the cross-ref is many-to-one)?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- Knuth, *Literate Programming* (1984, *Computer Journal*); the WEB user manual; *TeX: The
|
||||
Program* / *MMIX* as canonical WEB exemplars.
|
||||
- Ramsey, *noweb* (a simple, extensible literate-programming tool).
|
||||
- CWEB (Knuth & Levy); Emacs **Org-mode babel** (tangle + evaluate); **Sweave**/**knitr**
|
||||
(R); **Jupytext** (notebook-as-text).
|
||||
- prior: `research/260614-ikiwiki-deep-dive/` (compile-to-static, canonical-vs-derived);
|
||||
`research/260614-xanadu-deep-dive/` (compose-by-reference / EDL, UC-44).
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UC **UC-83** carries the marker **⊛** in the federation column of
|
||||
`spec/UseCaseCatalog.md` (true lineage = this dive; literate programming is design prior
|
||||
art, not a candidate shard, so the marker sits with the projection/compose-by-reference
|
||||
family). Enriched: UC-32, UC-44, UC-79; links UC-54. Architecture cross-refs: SHARD-WP-0002
|
||||
T12 (one-source-many-projections page shape), T16 (replication- vs derivation-projection;
|
||||
named-chunk transclusion).
|
||||
46
research/260614-localfirst-workspaces-deep-dive/README.md
Normal file
46
research/260614-localfirst-workspaces-deep-dive/README.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# 260614 — Local-first workspaces cohort deep dive (Anytype · AFFiNE · AppFlowy)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A **combined cohort deep dive** of three open-source, local-first "all-in-one workspace"
|
||||
tools — **Anytype**, **AFFiNE**, **AppFlowy** — studied together because they share the
|
||||
one trait none of the prior eleven systems had: **the store is a CRDT** (Anytype's
|
||||
*any-sync*, AFFiNE's *Yjs/y-octo*, AppFlowy's *Yrs*). The substrate is the contribution;
|
||||
studying them together isolates it from each tool's surface features.
|
||||
|
||||
Per-tool highlights:
|
||||
- **Anytype** — `any-sync`: **P2P + E2EE** CRDT (sync/file/consensus/coordinator nodes;
|
||||
encrypted backup), a **typed object graph** (user-editable types + relations), IPFS
|
||||
files, an **open API + MCP**.
|
||||
- **AFFiNE** — **Yjs/y-octo** over **OctoBase**; **BlockSuite** editor framework; docs,
|
||||
whiteboards, and databases as **views of the same block set**; WebSocket sync,
|
||||
self-host (Docker, MPL-2.0).
|
||||
- **AppFlowy** — **Yrs** via **AppFlowy-Collab** (Rust); Flutter UI; Notion-style
|
||||
**databases + views**; pluggable storage/sync (RocksDB/IndexedDB; AppFlowy Cloud/
|
||||
Supabase).
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Per-tool sections, the CRDT cohort thesis, comparative matrix, combined capability profile, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Cohort dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-64
|
||||
attach a CRDT-synced local-first shard with native conflict-free merge; UC-65 attach a
|
||||
peer-to-peer / decentralized shard with no single canonical endpoint); UC-36/47/48/51/55/
|
||||
57/58/61 enriched. Logged for `SHARD-WP-0002` (T11/T14): a **merge-model** capability
|
||||
(proposed thirteenth spectrum: `none / git / native-CRDT`), a **CRDT-replica** substrate +
|
||||
**P2P/no-central-endpoint** attach mode, an **endpoint-model** capability, and MCP as an
|
||||
integration surface.
|
||||
|
||||
**Cohort thesis recorded:** CRDT changes the adapter math — the backend performs
|
||||
conflict-free merge itself (so shard-wiki must not impose git/text merge), history is a
|
||||
CRDT update log (supplement, not git), and the attach surface is a **replica or sync
|
||||
endpoint** (Anytype adds **P2P + E2EE**). **Boundary:** CRDT local-first candidate shards
|
||||
— attach a replica/projection as pages, respect native merge, never re-drive their sync;
|
||||
not substrates and not the federation layer.
|
||||
</content>
|
||||
261
research/260614-localfirst-workspaces-deep-dive/findings.md
Normal file
261
research/260614-localfirst-workspaces-deep-dive/findings.md
Normal file
@@ -0,0 +1,261 @@
|
||||
# Findings — Anytype, AFFiNE, AppFlowy: the CRDT local-first workspace cohort
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped products (cohort dive)** — three open-source, local-first
|
||||
"all-in-one workspace" tools studied together; candidate shards of a **new substrate
|
||||
family: CRDT-synced local-first stores**
|
||||
Lens: shard-wiki — the CRDT substrate and its native-merge / P2P / E2EE implications for
|
||||
the adapter contract
|
||||
|
||||
> Why one combined memo. Anytype, AFFiNE, and AppFlowy are the open-source local-first
|
||||
> Notion/Miro alternatives, and they share the one trait the prior eleven systems never
|
||||
> had: **the store is a CRDT** (Anytype's *any-sync*, AFFiNE's *Yjs/y-octo*, AppFlowy's
|
||||
> *Yrs*). That single fact changes the adapter math — **the backend performs conflict-
|
||||
> free merge itself**, history is a CRDT update log (not git), and content arrives as a
|
||||
> replica/export, not files or rows. Studying them together isolates the *substrate*
|
||||
> contribution from each tool's surface features and avoids triplicating boilerplate.
|
||||
> Per-tool sections (§1–§3), then the cohort thesis and what it means for shard-wiki
|
||||
> (§4 onward).
|
||||
|
||||
Contrast set: file-store (Obsidian/Logseq), client-DB (Roam), hosted-DB (Notion),
|
||||
SQLite-local+file-sync (Joplin). This cohort adds **CRDT-replica** as a substrate.
|
||||
|
||||
---
|
||||
|
||||
## 1. Anytype — P2P, E2EE, typed object graph (any-sync)
|
||||
|
||||
- **Substrate:** the open **any-sync** protocol — **local-first, peer-to-peer,
|
||||
end-to-end encrypted** CRDT. Infrastructure is four node types: **sync, file,
|
||||
consensus, coordinator**; **backup nodes store ciphertext they cannot read**. Every
|
||||
change is **encrypted and signed with the user's private key**, verified via a shared
|
||||
public key. Files via **IPFS**.
|
||||
- **Data model:** **spaces** = graph-based databases; users define **custom object
|
||||
types with relations** ("everything is an object"; a user-editable **ontology**).
|
||||
Closest to Notion's typed model but **local-first + E2EE + user-owned schema**.
|
||||
- **Extension:** an **open API** for integrations, **plus MCP support** (AI-agent
|
||||
collaboration) — a notable modern surface (a shard speaking MCP).
|
||||
- shard-wiki read: the **most decentralized + encrypted** shard yet — no single
|
||||
canonical endpoint (attach a local replica or a node), content opaque without keys
|
||||
(UC-61), typed object graph (UC-58), CRDT merge (UC-64).
|
||||
|
||||
## 2. AFFiNE — docs + whiteboard + DB as views of one block set (Yjs/BlockSuite)
|
||||
|
||||
- **Substrate:** **Yjs CRDT** via **y-octo** (native Rust Yjs) over **OctoBase** (Rust
|
||||
local-first engine). Local-first by default; optional **WebSocket realtime** sync to
|
||||
AFFiNE Cloud or self-host (MPL-2.0, Docker).
|
||||
- **Editor:** **BlockSuite** — an open block-editor framework providing the **page
|
||||
editor, the edgeless canvas, and block types (incl. databases/embeds)**.
|
||||
- **Headline:** **docs, whiteboards, and databases are *views of the same underlying
|
||||
data*** — one block set rendered as page mode, edgeless canvas, or DB views. CRDT
|
||||
guarantees offline edits "merge mathematically perfectly" on reconnect.
|
||||
- shard-wiki read: the **one-data-many-renderings** model shipped (ZigZag dimensions,
|
||||
UC-47/48), with **non-Markdown spatial content** (edgeless canvas, UC-55) and CRDT
|
||||
merge (UC-64).
|
||||
|
||||
## 3. AppFlowy — Rust/Flutter, Notion-style DBs over Yrs CRDT
|
||||
|
||||
- **Substrate:** **Yrs** (Rust Yjs) wrapped as **AppFlowy-Collab** — CRDT primitives +
|
||||
persistence + domain objects (documents, **databases**, folders, importers, plugins,
|
||||
user state). **Pluggable storage/sync:** RocksDB (native) / IndexedDB (wasm) locally;
|
||||
**Supabase or AppFlowy Cloud** (self-hostable) for sync, via **trait-based local-only
|
||||
vs cloud**.
|
||||
- **Data model:** Notion-like **databases with views** (grid/board/calendar), typed
|
||||
fields. Flutter UI + Rust core, modular.
|
||||
- shard-wiki read: a **self-hostable CRDT Notion** — typed DBs + views (UC-58, UC-54),
|
||||
CRDT merge (UC-64), self-host sync endpoint (UC-57).
|
||||
|
||||
---
|
||||
|
||||
## 4. The cohort thesis — CRDT changes the adapter math
|
||||
|
||||
All three converge on **CRDT-synced, local-first, open-source, self-hostable** stores
|
||||
with **typed/database object models** and **non-Markdown content** (canvas/DB). The
|
||||
*substrate* is the new thing, and it has four consequences shard-wiki must absorb:
|
||||
|
||||
1. **Native, conflict-free merge (UC-64).** The backend resolves concurrent edits
|
||||
itself (Yjs/Yrs/any-sync). shard-wiki must **not impose git/text merge** on a CRDT
|
||||
shard; it either speaks the CRDT (replica) or stays a **projection/overlay** that
|
||||
respects CRDT semantics. This adds a **merge-model** capability:
|
||||
`none → git/text → native-CRDT` — a candidate **thirteenth spectrum** for the
|
||||
contract (after Joplin's content-opacity twelfth).
|
||||
2. **History is a CRDT update log, not git (UC-36).** No portable commit history; the
|
||||
coordination journal **supplements** (like Notion/Joplin), or shard-wiki snapshots
|
||||
the replica over time.
|
||||
3. **The attach surface is a replica or a sync endpoint, not files/rows.** Attach a
|
||||
**local CRDT replica** (offline, full state) or the **self-host sync server's API**
|
||||
(AFFiNE Cloud / AppFlowy Cloud / any-sync node). Anytype adds **P2P with no single
|
||||
canonical endpoint** (UC-65) and **E2EE content opacity** (UC-61).
|
||||
4. **Typed object graphs + multi-view (UC-58, UC-47/48, UC-55).** User-editable
|
||||
ontologies (Anytype types/relations), Notion-style DBs (AppFlowy), and one-block-set-
|
||||
many-renderings (AFFiNE page/canvas/DB) — the structured + dimensional demands the
|
||||
prior dives raised, now local-first.
|
||||
|
||||
These are **sync layers** (any-sync, Yjs sync, AppFlowy Cloud) — like Joplin, the
|
||||
**not-a-file-sync-daemon** boundary applies: shard-wiki attaches a **replica/projection
|
||||
as pages** and does **not** re-implement or re-drive their CRDT sync.
|
||||
|
||||
---
|
||||
|
||||
## 5. Comparative matrix
|
||||
|
||||
| | Anytype | AFFiNE | AppFlowy |
|
||||
|---|---------|--------|----------|
|
||||
| CRDT | any-sync (custom) | Yjs (y-octo) | Yrs |
|
||||
| Sync | **P2P + nodes, E2EE** | WebSocket, cloud/self-host | RocksDB local + cloud/Supabase |
|
||||
| Encryption | **E2EE (default)** | optional (transport/self-host) | self-host boundary |
|
||||
| Local engine | local + IPFS files | OctoBase (Rust) | RocksDB / IndexedDB |
|
||||
| Data model | **typed object graph (ontology)** | blocks; **doc/canvas/DB = one data, many views** | Notion-style **DBs + views** |
|
||||
| Non-MD content | objects, files | **edgeless canvas** | board/calendar/grid |
|
||||
| Extension | **open API + MCP** | BlockSuite framework | Flutter/Rust modules; plugins (roadmap) |
|
||||
| Self-host | backup/sync nodes | Docker (MPL-2.0) | AppFlowy Cloud |
|
||||
| Markdown | export (lossy) | import/export (lossy) | import/export (lossy) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Combined capability profile (as a shard)
|
||||
|
||||
| Capability | This cohort | Notes for the adapter contract |
|
||||
|------------|-------------|--------------------------------|
|
||||
| Read | yes (replica or sync API) | local CRDT replica (offline) or self-host/P2P endpoint |
|
||||
| Write | yes, **CRDT-mediated** | writes are CRDT ops; respect merge — don't overwrite |
|
||||
| Merge | **native CRDT (conflict-free)** | **new capability**: backend owns merge; no external git merge (UC-64) |
|
||||
| Write granularity | block/object | fine-grained |
|
||||
| Identity / addressing | object/block IDs (CRDT-assigned) | fine-grained, store-minted (UC-51) |
|
||||
| Structure | **typed object graph / DBs + relations** | user-editable ontology (Anytype); DB views (AppFlowy) (UC-58) |
|
||||
| History | **CRDT update log, not git** | supplement / snapshot (UC-36) |
|
||||
| Native query | varies (graph/DB filters) | delegate where present (UC-52) |
|
||||
| Subscribe | realtime CRDT sync / P2P | push by nature; attach as replica (UC-31) |
|
||||
| Content opacity | **Anytype E2EE → ciphertext** | opaque without keys (UC-61) |
|
||||
| Endpoint model | **replica / self-host server / P2P (Anytype)** | no single canonical URL for P2P (UC-65) |
|
||||
| Content types | docs + **canvas/DB/objects** | non-Markdown (UC-55); multi-view (UC-47/48) |
|
||||
| Markdown | lossy export | fidelity report (UC-59) |
|
||||
| Attach modes | CRDT replica · self-host API · (Anytype) MCP/open API | new substrate row in the taxonomy |
|
||||
|
||||
Verdict: a **demanding but coherent new family** — best attached as a **local replica
|
||||
(offline, full CRDT state)** projected to pages, or via a **self-host sync endpoint**;
|
||||
Anytype additionally **P2P + E2EE** (replica-only/opaque). Standout demands:
|
||||
**native-CRDT merge** (UC-64) and **P2P/no-central-endpoint + E2EE** (UC-65/UC-61).
|
||||
|
||||
---
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
|
||||
- **Local-first sovereignty** is the cohort's whole pitch — strong INTENT alignment;
|
||||
each is a legitimate sovereign shard with user-owned data.
|
||||
- **Self-hostable + open-source** means shard-wiki can attach within a user's trust
|
||||
boundary (good for the L0→L4 authz ladder).
|
||||
- **One-data-many-views** (AFFiNE) and **typed object graphs** (Anytype/AppFlowy) are
|
||||
the dimensional (UC-47/48) and structured (UC-58) demands, now validated local-first.
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Do not impose git/text merge on a CRDT shard.** The backend merges; shard-wiki
|
||||
respects CRDT semantics or stays a projection/overlay (UC-64). Treating a CRDT shard
|
||||
like a file shard corrupts it.
|
||||
2. **Not a sync daemon.** any-sync/Yjs/AppFlowy Cloud already sync; shard-wiki attaches
|
||||
a **replica as pages**, never re-drives their sync (INTENT not-a-file-sync-daemon).
|
||||
3. **P2P has no canonical endpoint.** Don't model the shard as a URL; attach a replica
|
||||
or a named peer/node (UC-65).
|
||||
4. **E2EE shards are opaque** without keys — backup/structure-shell only (UC-61).
|
||||
5. **Proprietary CRDT/object formats are lossy to Markdown** — translate with a fidelity
|
||||
report (UC-59), one workspace = one shard, not the federation layer.
|
||||
|
||||
### 7.3 What the cohort teaches that shard-wiki should keep
|
||||
|
||||
- Add a **merge-model** capability (`none / git / native-CRDT`) so CRDT shards are
|
||||
handled correctly (UC-64) — a thirteenth capability spectrum.
|
||||
- Add **CRDT-replica** to the substrate/attachment taxonomy, and **P2P/no-central-
|
||||
endpoint** as a binding mode (UC-64/65).
|
||||
- **MCP as an integration surface** (Anytype) is worth noting — a shard may expose an
|
||||
AI-agent protocol shard-wiki could consume or coexist with.
|
||||
|
||||
---
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-63**. New UCs **UC-64, UC-65** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a CRDT-synced local-first shard** whose backend performs native conflict-free merge; respect CRDT semantics (no git merge), consume a replica/export, CRDT-log history (supplement) | **UC-64 (new)** |
|
||||
| **Attach a peer-to-peer / decentralized shard with no single canonical endpoint** (Anytype any-sync: replica or peer/backup node; E2EE) | **UC-65 (new)** |
|
||||
| Typed object graph / user-editable ontology + Notion-style DBs, local-first | **enriches UC-58** |
|
||||
| E2EE content opacity (Anytype) reinforced; self-host trust boundary | **enriches UC-61** |
|
||||
| CRDT update log ≠ portable git history → supplement | **enriches UC-36** |
|
||||
| One block set rendered as page / canvas / DB views (AFFiNE) | **enriches UC-47, UC-48** |
|
||||
| Edgeless canvas / board / objects = non-Markdown content | **enriches UC-55** |
|
||||
| Self-host sync server (AFFiNE/AppFlowy Cloud) + Anytype open API/MCP as endpoints | **enriches UC-57** |
|
||||
| Object/block IDs CRDT-assigned (fine-grained) | **enriches UC-51** |
|
||||
| Lossy Markdown export | links UC-59 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Merge-model capability** (`none / git-or-text / native-CRDT`) — proposed
|
||||
**thirteenth spectrum** (with Joplin's content-opacity twelfth). CRDT shards reject
|
||||
external merge; shard-wiki overlays must be CRDT-aware or remain projections. (T11.)
|
||||
- **Substrate taxonomy gains CRDT-replica**; **attach modes gain "local CRDT replica"
|
||||
and "P2P/no-central-endpoint"** beside file-store / in-engine-host / external-API.
|
||||
(T14.)
|
||||
- **Endpoint model** as a capability: single URL vs self-host server vs **P2P peer set**
|
||||
(Anytype) — affects how a binding is addressed and how freshness/sync work. (T14.)
|
||||
- **MCP / AI-agent protocol** as a possible shard integration surface (Anytype). Note
|
||||
for the extension-surface catalogue. (T14.)
|
||||
- These three plus Joplin and Logseq all argue the contract must treat **substrate as a
|
||||
spectrum that tools migrate along and that shard-wiki must not hard-code**.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions (for spec / workplans)
|
||||
|
||||
1. For a **CRDT shard** (UC-64), does shard-wiki embed a CRDT client (Yjs/Yrs) to hold a
|
||||
live replica, or only consume periodic **exports/snapshots**? The former gives
|
||||
live/writable; the latter is simpler but lossy on merge timing.
|
||||
2. How are **overlays** expressed against a CRDT shard — as CRDT ops (requires the
|
||||
client) or as out-of-band patches reconciled later (projection-only)?
|
||||
3. For **P2P shards** (UC-65), what is the "address" of the shard — a space ID + a peer/
|
||||
node, a local replica path, or an invite/key? How does this fit shard attachment
|
||||
config?
|
||||
4. Does shard-wiki ever hold **E2EE keys** (Anytype) to project content, or only ever
|
||||
treat such shards as opaque replicas/backups (UC-61)?
|
||||
5. Is the **self-host sync server** (AFFiNE/AppFlowy Cloud) attached as an external API
|
||||
(UC-57) or by running a co-located replica?
|
||||
|
||||
---
|
||||
|
||||
## 11. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Anytype any-sync — Protocol overview (https://tech.anytype.io/any-sync/overview) | Local-first/P2P/E2EE CRDT; sync/file/consensus/coordinator nodes; encrypted backup |
|
||||
| anyproto/any-sync (https://github.com/anyproto/any-sync) | Protocol scope; keys signed/verified; IPFS files |
|
||||
| Anytype Docs (https://doc.anytype.io/anytype-docs) | Spaces, object types + relations (ontology); open API + MCP |
|
||||
| toeverything/AFFiNE + DeepWiki (https://deepwiki.com/toeverything/AFFiNE) | BlockSuite (page/edgeless/DB); y-octo Yjs; OctoBase; one-data-many-views; self-host |
|
||||
| AFFiNE — interactive whiteboard / blog (https://affine.pro/blog/interactive-whiteboard-app) | Edgeless canvas; CRDT offline merge |
|
||||
| AppFlowy-IO/AppFlowy-Collab (https://github.com/AppFlowy-IO/AppFlowy-Collab) | Yrs CRDT; collab domain objects (documents/databases/folders); persistence |
|
||||
| AppFlowy Architecture Overview — DeepWiki (https://deepwiki.com/AppFlowy-IO/AppFlowy/2-architecture-overview) | Flutter/Rust layers; pluggable storage/sync (RocksDB/IndexedDB; Supabase/AppFlowy Cloud); local vs cloud trait |
|
||||
| AFFiNE vs AppFlowy vs Anytype (https://affine.pro/blog/affine-vs-appflowy-vs-anytype) | Cohort comparison framing |
|
||||
|
||||
Cross-references: `research/260614-notion-deep-dive/findings.md` (typed DB, hosted
|
||||
contrast), `research/260614-joplin-deep-dive/findings.md` (E2EE/content opacity, sync-
|
||||
daemon boundary), `research/260614-shard-spectrum-synthesis/findings.md` (the spectra
|
||||
this extends), `spec/UseCaseCatalog.md` (UC-36, UC-47, UC-48, UC-51, UC-55, UC-57,
|
||||
UC-58, UC-61), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
|
||||
|
||||
---
|
||||
|
||||
## 12. Traceability
|
||||
|
||||
- New UCs: **UC-64, UC-65** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-36, UC-47, UC-48, UC-51, UC-55, UC-57, UC-58, UC-61** (links UC-59).
|
||||
- Architecture (no UC): **merge-model** capability (proposed thirteenth spectrum);
|
||||
CRDT-replica substrate + P2P/no-central-endpoint attach mode; endpoint-model capability;
|
||||
MCP integration surface → `SHARD-WP-0002` (T11, T14).
|
||||
- Boundary recorded: Anytype/AFFiNE/AppFlowy are **CRDT local-first candidate shards**
|
||||
(attach a replica/projection as pages; respect native CRDT merge; never re-drive their
|
||||
sync; P2P/E2EE shards are replica-only/opaque) — not substrates, not the federation
|
||||
layer (INTENT not-a-sync-daemon, graceful degradation, no silent mutation).
|
||||
</content>
|
||||
50
research/260614-logseq-deep-dive/README.md
Normal file
50
research/260614-logseq-deep-dive/README.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# 260614 — Logseq deep dive (block-graph semantics on plain Markdown files)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Logseq** — the open-source, local-first outliner — read through
|
||||
shard-wiki's lens. Logseq occupies the point the other modern tools leave empty:
|
||||
**block-graph semantics (UUID-addressable, embeddable, queryable blocks) stored as plain
|
||||
Markdown/Org files on disk**, with a DataScript graph **derived** from those files. It is
|
||||
the bridge between **Roam** (block-graph, client-DB) and **Obsidian** (file-over-app,
|
||||
page-level), and it resolves a tension the synthesis flagged: block-level addressing
|
||||
*that is also git-diffable text*.
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — files canonical (`pages/`, `journals/` MD/Org); a DataScript graph
|
||||
**derived** via an `mldoc` AST parse; a DB Worker now manages **both DataScript and
|
||||
SQLite** (the file→DB migration)
|
||||
- **Block-graph in the file text** — block IDs as in-file `id:: <uuid>` properties,
|
||||
`((uuid))` refs, `key:: value` properties, `{{embed}}` transclusion, the outline tree
|
||||
(indent/zoom/move-subtree) — all git-diffable
|
||||
- **Queries** — `{{query}}` + advanced **Datalog** over the derived graph
|
||||
(`logseq.DB.datascriptQuery`)
|
||||
- **Extension** — plugin API (`logseq.App/Editor/DB/Git/UI/Assets/FileStorage`),
|
||||
marketplace (~486 plugins); dual-attachable (file-store direct *or* in-app plugin)
|
||||
- **Trajectory** — migrating from Markdown-files to a SQLite "DB graph" (a live UC-43)
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, in-file block-graph, Datalog queries, plugin API, file→DB migration, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-62 attach a block-graph-on-plain-files shard with in-file block IDs/properties +
|
||||
outline tree + derivable query index; UC-63 serve structured queries over a file-backed
|
||||
shard via a derived index the orchestrator/adapter builds); UC-32/34/43/50/51/52/55
|
||||
enriched. Logged for `SHARD-WP-0002` (T11/T14/T16): a Logseq **format profile** for
|
||||
file-store adapters, a **derived-query-index** capability, **substrate-migration
|
||||
tolerance**, and **in-file block addressing** as the concrete T16 span-address target.
|
||||
|
||||
**Key takeaway recorded:** Logseq resolves the addressing-spectrum tension — block-level
|
||||
addressing **and** git-diffable in-file text (`id::`) — and proves a file-backed shard
|
||||
can support rich Datalog queries via a **derived** index (files canonical, graph
|
||||
derived). **Boundary:** one block-graph-on-files candidate shard (the addressing sweet
|
||||
spot), best attached file-store-direct with a format profile; not the federation layer;
|
||||
substrate is migrating file→SQLite.
|
||||
</content>
|
||||
246
research/260614-logseq-deep-dive/findings.md
Normal file
246
research/260614-logseq-deep-dive/findings.md
Normal file
@@ -0,0 +1,246 @@
|
||||
# Findings — Logseq: block-graph semantics on plain Markdown files
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — an open-source, local-first outliner; a
|
||||
*candidate shard* occupying the point none of the prior tools do: **block-graph + file-
|
||||
over-app**
|
||||
Lens: shard-wiki — the convergence of block-level addressing with git-diffable files, a
|
||||
derived query index over canonical files, and a live file→DB storage migration
|
||||
|
||||
> Why Logseq matters distinctly. The modern-tool dives staked out the corners: **Roam**
|
||||
> = block-graph but client-DB / in-app-API / store-minted UUID; **Obsidian** =
|
||||
> file-over-app but page-level / path-addressed (block `^id` opt-in); **Notion** =
|
||||
> block-graph but closed hosted DB / external-API. **Logseq sits in the middle they
|
||||
> leave empty: block-graph *semantics* (UUID-addressable, embeddable, queryable blocks)
|
||||
> stored as *plain Markdown/Org files on disk you own*, with a DataScript index
|
||||
> **derived** from those files.** It is the system that proves you do not have to choose
|
||||
> between Roam's fine-grained addressing and Obsidian's file sovereignty — you can have
|
||||
> block-level addressing *that is also git-diffable text*. That resolves a tension the
|
||||
> synthesis flagged (addressing spectrum) and is the reason it earns a memo rather than
|
||||
> a footnote.
|
||||
|
||||
Contrast set: Roam (block-DB, in-app), Obsidian (file, page-level), Notion (hosted DB),
|
||||
Joplin (SQLite-local, files-on-sync). Logseq = **block-graph, files-canonical,
|
||||
index-derived** — and now *also* migrating toward SQLite (§5), a live UC-43 case.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture — files canonical, DataScript derived
|
||||
|
||||
- **Storage:** plain-text **Markdown or Org-mode files on disk** — `pages/<Page>.md` and
|
||||
`journals/<date>.md` — fully usable even if the app disappears (local-first, open
|
||||
source, ClojureScript).
|
||||
- **Index:** a **DataScript** in-memory graph database, **derived** from the files: an
|
||||
`mldoc` parser turns Markdown into an AST, extracts references/properties, and
|
||||
transforms it into **DataScript entities** with tree-position attributes. Files are the
|
||||
source of truth; the graph DB is a rebuildable projection. (In current versions a **DB
|
||||
Worker** manages **both DataScript and SQLite** connections — see §5.)
|
||||
- This is the **files-canonical / index-derived** architecture (Obsidian's MetadataCache,
|
||||
Git's working tree) — but here the derived index is a *full Datalog-queryable graph*,
|
||||
not just a metadata cache. Logseq is the strongest evidence that **a file-backed shard
|
||||
can support rich structured queries via a derived index** (UC-52, UC-63).
|
||||
|
||||
---
|
||||
|
||||
## 2. The block-graph, in the file text
|
||||
|
||||
Everything is an **atomic block** (a bullet), individually referenceable, embeddable, and
|
||||
queryable — Roam's model — but the addressing lives **in the Markdown**:
|
||||
|
||||
- **Block IDs are in-file properties.** When a block is referenced, Logseq writes an
|
||||
`id:: <uuid>` property line into the Markdown. So the block's stable address is
|
||||
**git-diffable text that survives a file copy**, not a DB-minted hidden key. This is the
|
||||
**sweet spot of the addressing spectrum**: block-level (like Roam) *and* in-file/
|
||||
portable (like Obsidian's `^id`), without choosing.
|
||||
- **References:** `[[page]]`, **`((block-uuid))`**, `#tag` — all extracted into the graph;
|
||||
block embeds `{{embed ((uuid))}}` are transclusion at block granularity (UC-32).
|
||||
- **Properties:** `key:: value` lines attach typed metadata to blocks (block properties)
|
||||
and pages (first-block/page properties) — **git-diffable structured data at block
|
||||
granularity** (UC-34), queried by Datalog.
|
||||
- **Outline tree:** a page is a **nested tree of blocks** (indent = structure), with
|
||||
outliner operations — indent/outdent, move subtree, **zoom/narrow** into a block. The
|
||||
page is not flat prose; it is an addressable, reorderable block tree (UC-63).
|
||||
|
||||
---
|
||||
|
||||
## 3. Queries — Datalog over the derived graph
|
||||
|
||||
Logseq exposes **`{{query}}`** (simple) and **advanced Datalog queries** over the
|
||||
DataScript graph (and via the plugin API, `logseq.DB.datascriptQuery`). Because the graph
|
||||
is derived from files, **query-defined pages** (UC-54) and structured aggregation (tasks,
|
||||
tagged blocks) run over a *file-backed* store. Key lesson: query capability here is
|
||||
**neither native-DB (Notion) nor a third-party plugin (Obsidian Dataview)** — it is
|
||||
**built into the tool over its own files**, demonstrating that *a derived query index is a
|
||||
viable adapter/orchestrator capability for file shards* (UC-63).
|
||||
|
||||
---
|
||||
|
||||
## 4. Extension surface — plugin API + marketplace
|
||||
|
||||
Logseq has a JS **plugin API** (marketplace ~486 plugins, `logseq/marketplace`):
|
||||
|
||||
- `logseq.Editor` — block/page CRUD, insert/move/update/delete, get current block/page.
|
||||
- `logseq.DB` — **`datascriptQuery`** (Datalog), plus DB change subscriptions.
|
||||
- `logseq.App` — app-level ops/events; `logseq.UI`; `logseq.Git` (git ops on the graph);
|
||||
`logseq.Assets`; `logseq.FileStorage`.
|
||||
- Slash commands, block/page context menus, `provideModel`/`provideStyle`, hooks/events.
|
||||
|
||||
So, like Obsidian, Logseq is **dual-attachable**: (a) **file-store direct** — read the
|
||||
`pages/`+`journals/` Markdown with a **Logseq format profile** (parse `id::`, `((uuid))`,
|
||||
`key::`, the outline tree); (b) **in-app plugin host** — `logseq.Editor` write +
|
||||
`logseq.DB` query + change events for live write-through. A notable extra: a built-in
|
||||
**`logseq.Git`** surface — the tool treats git as a first-class companion to the file
|
||||
graph (validating the coordination journal).
|
||||
|
||||
---
|
||||
|
||||
## 5. The file→DB migration — a live UC-43
|
||||
|
||||
Logseq is **migrating its storage model from Markdown-files to a SQLite "DB graph"**
|
||||
(the DB Worker already manages SQLite alongside DataScript; the plugin API has a distinct
|
||||
"DB graph" mode with tags/classes/typed properties). This is a real-world instance of
|
||||
**UC-43 (backend-swap under stable identity)**: the *same tool and graph identity* moving
|
||||
from a file substrate to a DB substrate, trading git-diffability for richer typed
|
||||
structure (toward the Notion/XWiki end). For shard-wiki it is both a caution (a shard's
|
||||
substrate can change under it) and a confirmation that the **addressing/structure/history
|
||||
spectra** are trajectories tools actually travel — an adapter keyed to a fixed substrate
|
||||
will break.
|
||||
|
||||
---
|
||||
|
||||
## 6. Logseq as a shard — capability profile
|
||||
|
||||
| Capability | Logseq (MD-file graph) | Notes for the adapter contract |
|
||||
|------------|------------------------|--------------------------------|
|
||||
| Read | **yes** | file-store direct (pages/journals MD) or `logseq.Editor`/`logseq.DB` in-app |
|
||||
| Write | **yes** | direct file write (format-aware) or plugin; block-level |
|
||||
| Write granularity | **per-block** (in a per-file page) | finer than Obsidian (per-file), like Roam — but on files |
|
||||
| Identity / addressing | **in-file block `id:: uuid` + `((uuid))`** | block-level **and** git-diffable — the addressing sweet spot (UC-51) |
|
||||
| Structure | `key:: value` block/page properties; outline tree | git-diffable structured data at block granularity (UC-34) |
|
||||
| History | none native; **`logseq.Git`** + files = git-native | git-friendly out of the box (adopt, not supplement) |
|
||||
| Native query | **Datalog over derived DataScript** | derived index over files → delegate or rebuild (UC-52, UC-63) |
|
||||
| Transclusion | **block embeds `{{embed ((uuid))}}`** | in-file, addressable (UC-32) |
|
||||
| Backlinks | linked + unlinked references | derived (UC-05/18) |
|
||||
| Content types | Markdown/Org + **whiteboards** (tldraw JSON) | non-Markdown spatial content (UC-55) |
|
||||
| Substrate | **files now, SQLite "DB graph" emerging** | live backend-swap (UC-43) |
|
||||
| Attach modes | file-store direct (format profile) · in-app plugin | dual, per-binding (UC-62) |
|
||||
|
||||
Verdict: **the most shard-wiki-friendly block tool** — block-graph power with file-over-
|
||||
app sovereignty and git-diffable addressing/structure. Best attach: **file-store direct
|
||||
with a Logseq format profile** (offline, git-native), with the plugin for live write-
|
||||
through. Watch the **DB-graph migration** (UC-43).
|
||||
|
||||
---
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
|
||||
- **Resolves the addressing tension.** Block-level addressing *can* be git-diffable
|
||||
in-file (`id::`), not forced to choose Roam-DB vs Obsidian-page (UC-51, synthesis §2).
|
||||
- **Confirms files-canonical / index-derived at full power** — a Datalog graph derived
|
||||
from files, not just a metadata cache (UC-52, UC-63).
|
||||
- **Structure as git-diffable text** (`key::`) reinforces "prefer in-text structure"
|
||||
(UC-34, synthesis through-line).
|
||||
- **Outliner block tree** validates the page-as-addressable-block-tree demand (UC-50,
|
||||
UC-63) on a *file* backend.
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **One graph = one shard.** Local-first single graph; not the federation layer.
|
||||
2. **The MD files carry Logseq-specific syntax** (`id::`, `((uuid))`, `key::`, outline
|
||||
semantics) — a **format profile**, not plain CommonMark. A naïve Markdown reader will
|
||||
mangle block IDs/properties (cf. UC-42; lighter than Notion's lossy case).
|
||||
3. **The substrate is moving (file→SQLite).** Don't hard-code the file model; gate on the
|
||||
substrate capability and tolerate the swap (UC-43).
|
||||
4. **Whiteboards are not Markdown** — typed/opaque assets, not flattened (UC-55).
|
||||
|
||||
### 7.3 What Logseq teaches that shard-wiki should keep
|
||||
|
||||
- **In-file block addressing is the target shape** for a portable span address where the
|
||||
backend cooperates — adopt `id::`-style schemes; they are git-diffable and survive copy.
|
||||
- **A derived query index over files is a first-class capability** — shard-wiki can build
|
||||
one over a file shard (UC-63) when the backend exposes none, or delegate when it does.
|
||||
- **Expect substrate migration** — bind to capabilities, not to "it's files."
|
||||
|
||||
---
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-61**. New UCs **UC-62, UC-63** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a block-graph-on-plain-files shard** (Logseq-style): file-over-app Markdown carrying in-file block IDs (`id::`), block refs, and `key::` properties, with the outline tree and a derivable query index | **UC-62 (new)** |
|
||||
| **Serve structured queries over a file-backed shard via a derived index** the orchestrator/adapter builds (Logseq DataScript-over-files) when the backend exposes none | **UC-63 (new)** |
|
||||
| In-file block `id::` = block-level **and** git-diffable addressing (the spectrum sweet spot) | **enriches UC-51** |
|
||||
| Live **file→SQLite substrate migration** under stable graph identity | **enriches UC-43** |
|
||||
| Block-graph that is **files, not a DB** — the file-store variant of the block tool family | **enriches UC-50** |
|
||||
| Datalog over a **derived** index built from files | **enriches UC-52** |
|
||||
| `key:: value` block/page properties in-text | **enriches UC-34** |
|
||||
| Whiteboards (tldraw JSON) = non-Markdown content | **enriches UC-55** |
|
||||
| Block embeds `{{embed ((uuid))}}` = in-file transclusion | links UC-32 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Format-aware file-store profiles** (already flagged by Joplin, UC-60) gain a strong
|
||||
case here: a Logseq profile parses `id::`/`((uuid))`/`key::`/outline from otherwise-
|
||||
plain Markdown. The contract should let a file-store adapter declare its **format
|
||||
profile** (plain MD / Obsidian / Logseq / Joplin-item / Foswiki-PlainFile). (T11/T14.)
|
||||
- **Derived-query-index as an orchestrator capability** (UC-63): when a file shard has no
|
||||
native query engine, build a DataScript-like index over the projection; when it does
|
||||
(Roam/Notion/XWiki), delegate (UC-52). Both belong in T16's navigation layer + T11.
|
||||
- **Substrate-migration tolerance** (UC-43, Logseq file→SQLite): T14 binding should treat
|
||||
substrate as a capability that can change under a live attachment, preserving identity.
|
||||
- **In-file block addressing** is the concrete realization of T16's span-address thread —
|
||||
prefer `id::`-style git-diffable IDs over DB-minted where the backend allows.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions (for spec / workplans)
|
||||
|
||||
1. When the same tool offers **file** and **DB** substrates (Logseq now), does shard-wiki
|
||||
prefer the file graph (git-diffable, UC-62) or the DB graph (richer typed structure),
|
||||
and can one binding follow the migration (UC-43)?
|
||||
2. Is the **derived query index** (UC-63) built by the **adapter** (per-shard) or the
|
||||
**core orchestrator** (over the union), and is it persisted or rebuilt?
|
||||
3. How much **Logseq outline semantics** (zoom, subtree move) must shard-wiki preserve vs.
|
||||
present as a flat page (UC-63 vs. Markdown-first page model)?
|
||||
4. Does the **Logseq format profile** round-trip overlays (write `id::`/`key::` back) or
|
||||
only read them? (cf. UC-42 round-trip question.)
|
||||
|
||||
---
|
||||
|
||||
## 11. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| DeepWiki — logseq/logseq (https://deepwiki.com/logseq/logseq) | DataScript+Datalog graph, mldoc AST parse, block entities/tree, DB Worker managing DataScript **and** SQLite |
|
||||
| logseq/docs (https://deepwiki.com/logseq/docs) | Plain-text MD/Org files; pages/journals; references ([[ ]], (( )), #tag); properties |
|
||||
| Logseq Plugin API docs (https://plugins-doc.logseq.com/) | `logseq.App/Editor/DB/Git/UI/Assets/FileStorage`; `DB.datascriptQuery` |
|
||||
| logseq/marketplace (https://github.com/logseq/marketplace) | Plugin distribution; ~486 plugins |
|
||||
| kerim/logseq-db-plugin-api-skill (https://github.com/kerim/logseq-db-plugin-api-skill) | DB-graph version: tags/classes, typed properties, file→DB migration |
|
||||
| pangea.app glossary — Logseq (https://pangea.app/glossary/logseq) | Local-first framing, outliner, plain-text control |
|
||||
|
||||
Cross-references: `research/260614-roam-deep-dive/findings.md` (block-DB contrast),
|
||||
`research/260614-obsidian-deep-dive/findings.md` (file-over-app, derived index),
|
||||
`research/260614-joplin-deep-dive/findings.md` (format-aware file-store profiles),
|
||||
`research/260614-shard-spectrum-synthesis/findings.md` (addressing spectrum this
|
||||
resolves), `spec/UseCaseCatalog.md` (UC-32, UC-34, UC-43, UC-50, UC-51, UC-52, UC-55),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14, T16).
|
||||
|
||||
---
|
||||
|
||||
## 12. Traceability
|
||||
|
||||
- New UCs: **UC-62, UC-63** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-32, UC-34, UC-43, UC-50, UC-51, UC-52, UC-55**.
|
||||
- Architecture (no UC): format-aware file-store profiles (Logseq profile); derived-query-
|
||||
index capability; substrate-migration tolerance; in-file block addressing as the T16
|
||||
span-address target → `SHARD-WP-0002` (T11, T14, T16).
|
||||
- Boundary recorded: Logseq is **one block-graph-on-files candidate shard** (the
|
||||
addressing sweet spot), best attached file-store-direct with a format profile; not the
|
||||
federation layer; substrate is migrating file→SQLite (UC-43).
|
||||
</content>
|
||||
32
research/260614-mathematica-deep-dive/README.md
Normal file
32
research/260614-mathematica-deep-dive/README.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# 260614 — Mathematica Notebooks deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T2**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Wolfram Mathematica Notebooks** — the **original computational notebook**
|
||||
(1988), the ancestor Jupyter descends from. The `.nb` document **is itself a Wolfram Language
|
||||
expression** (`Notebook[{Cell[…], …}]`) with **nested cell groups**, kernel evaluation
|
||||
(`In`/`Out`), **structured (symbolic/graphics) results**, live `Manipulate`/`Dynamic`
|
||||
widgets, and CDF as a reduced-runtime distribution projection.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- Confirms the **notebook page shape (UC-84) is a genus**, not a Jupyter quirk: cells +
|
||||
cached outputs + fragile counter provenance + kernel-gated re-execution predate `.ipynb`.
|
||||
- Two refinements: notebooks have **nestable cell groups (an outline tree)**, not just an
|
||||
ordered list; outputs can be **structured re-evaluable values**, not only MIME blobs →
|
||||
adds a "structured re-evaluable value" point to the content-opacity spectrum.
|
||||
- `Manipulate`/`Dynamic` join the **static-projection-impossible** end (snapshot-only) —
|
||||
foreshadows Strudel (T5).
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (lineage/reinforcement). Reinforces **UC-84**; enriches **UC-54, UC-55**;
|
||||
links UC-34, UC-83.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | `.nb` expression/cell model, evaluation & provenance, CDF, capability delta vs Jupyter, INTENT mapping, UC disposition (enrichment-only), architecture notes |
|
||||
139
research/260614-mathematica-deep-dive/findings.md
Normal file
139
research/260614-mathematica-deep-dive/findings.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# Mathematica Notebooks — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T2 · **Subject:** Wolfram Mathematica
|
||||
Notebooks — the `.nb` format, the cell/expression model, symbolic evaluation.
|
||||
|
||||
## Why this dive
|
||||
|
||||
Mathematica (1988) is the **original computational notebook** — the ancestor Jupyter (T3)
|
||||
descends from. This dive checks whether the notebook page-model conclusions from UC-84 hold
|
||||
at the genus level or need extension. It is **medium priority / lineage**: a candidate
|
||||
*content type* like Jupyter, but closed/proprietary, so the expected yield is **reinforcement
|
||||
of UC-84** plus a couple of distinct wrinkles (the document *is itself* a Wolfram-language
|
||||
expression; symbolic, not just textual, results).
|
||||
|
||||
## 1. The `.nb` document model
|
||||
|
||||
A Mathematica notebook is itself a **Wolfram Language expression** — `Notebook[{Cell[...],
|
||||
Cell[...], ...}, opts]`. So the document format and the language are the *same* substrate:
|
||||
|
||||
- **Cells** are typed: `Input`, `Output`, `Text`, `Title`/`Section` (structure), `Code`,
|
||||
with **cell groups** nesting into an **outline tree** (the document has real hierarchy, not
|
||||
just a flat list like `.ipynb`).
|
||||
- An **`Input` cell** holds an expression; evaluating it produces a linked **`Output` cell`**
|
||||
containing the **result expression** (symbolic, graphical via `Graphics[...]`, or
|
||||
typeset). The result is a **first-class expression**, not a MIME blob — it can be
|
||||
re-evaluated, edited, transcluded.
|
||||
- The whole `.nb` is **plain-text-serializable** (it's an expression) but verbose and
|
||||
proprietary in conventions; output cells are **cached results** stored in the file.
|
||||
|
||||
So like Jupyter: **source + cached computed output fused in one document**, with **out-of-
|
||||
band reproducibility** (kernel + package/version state). Unlike Jupyter: **nested cell
|
||||
groups (a tree)** and **results that are structured Wolfram expressions** rather than MIME
|
||||
bundles.
|
||||
|
||||
## 2. Evaluation, provenance, reproducibility
|
||||
|
||||
- A **kernel** evaluates input cells; `In[n]`/`Out[n]` counters mirror Jupyter's
|
||||
`execution_count` and carry the **same fragility** (out-of-order evaluation, hidden
|
||||
symbol/global state, kernel-version dependence). No environment capture in the file.
|
||||
- **Dynamic/interactive output** (`Manipulate`, `Dynamic`) embeds **live interactive
|
||||
widgets** whose state is computed on view — these have **no faithful static form** beyond a
|
||||
snapshot frame (echoes Strudel T5's "live, time/interaction-based content" limit).
|
||||
- **CDF (Computable Document Format)** is Wolfram's *projection-for-distribution*: a notebook
|
||||
rendered with a free runtime so readers can interact without a full license — a
|
||||
derivation-projection (T1) with a reduced-capability viewer.
|
||||
|
||||
## 3. Capability profile (delta vs Jupyter UC-84)
|
||||
|
||||
| Dimension | Mathematica `.nb` (delta from Jupyter) |
|
||||
|-----------|----------------------------------------|
|
||||
| Structure | **nested cell groups → outline tree** (richer than `.ipynb`'s flat cell list) |
|
||||
| Output type | **structured Wolfram expressions** (symbolic/graphics), not MIME blobs |
|
||||
| Document = language | the `.nb` **is** a Wolfram expression (format ≡ language) |
|
||||
| Liveness | `Manipulate`/`Dynamic` = **interactive widgets**, snapshot-only when static |
|
||||
| Opacity | proprietary serialization; results re-evaluable only with a Wolfram kernel |
|
||||
| Projection-for-distribution | **CDF** = reduced-runtime interactive projection |
|
||||
| Otherwise | same as UC-84: cells, cached outputs, fragile `In/Out` provenance, kernel-gated |
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Reinforcements (mostly confirms UC-84)
|
||||
|
||||
- **Notebook page shape (UC-84) is a genus, not a Jupyter quirk.** Mathematica predates and
|
||||
matches it: cells + cached computed outputs + fragile counter provenance + kernel-gated
|
||||
re-execution. Confirms the page model should carry a **notebook shape** generically (T12),
|
||||
not a `.ipynb`-specific one.
|
||||
- **Outputs as derivation-projection snapshots (UC-83/84).** Cached `Output` cells are
|
||||
snapshots; honest treatment marks them "evaluated run N, kernel/env unguaranteed."
|
||||
- **Derivation-projection for distribution (T1).** CDF is a clean "reduced-capability
|
||||
interactive projection of a source" — a real-world instance of degrade-by-capability.
|
||||
|
||||
### Distinct wrinkles (extend the notes, not new UCs)
|
||||
|
||||
- **Nested cell-group outline** — the notebook page model should allow **hierarchical cell
|
||||
grouping**, not just an ordered list (generalize UC-84's "ordered cells" to "ordered/
|
||||
*nestable* cells"). Feeds T12.
|
||||
- **Structured (non-MIME) results** — outputs can be **typed structured values** (symbolic
|
||||
expressions), not only MIME blobs; reinforces UC-55's "typed asset" reading over "opaque
|
||||
blob," and links the typed-record page model (UC-34) — the content-opacity spectrum needs a
|
||||
"structured re-evaluable value" point, not just text↔blob.
|
||||
- **Format ≡ language** — a curiosity, not actionable for us beyond noting that some shards'
|
||||
document format is *the same artifact* as their computation (don't assume doc/code split).
|
||||
- **Live interactive widgets** — `Manipulate`/`Dynamic` join Strudel (T5) at the
|
||||
**static-projection-impossible** end: capture a snapshot frame, never imply interactivity.
|
||||
|
||||
### Boundaries
|
||||
|
||||
- Proprietary + kernel-gated → default **read/projection/snapshot**; attach the `.nb`
|
||||
(or an exported form), present cached outputs as snapshots, offer a static/CDF projection;
|
||||
**no kernel host** (same rule as Jupyter UC-84, GT T7).
|
||||
|
||||
## 5. UC disposition (enrichment-only — no new UC)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC |
|
||||
|------------------------|------------|
|
||||
| Cells + cached computed outputs + fragile In/Out provenance; kernel-gated (§1, §2) | UC-84 (reinforced) |
|
||||
| Nested cell groups → outline tree (richer than flat `.ipynb`) (§1) | UC-84 (enriched: nestable cells); links UC-34 |
|
||||
| Output = structured re-evaluable Wolfram expression, not MIME blob (§1) | UC-55 (enriched: structured value point on opacity spectrum) |
|
||||
| Input cell = computation-defined content (§1) | UC-54 (enriched) |
|
||||
| `Manipulate`/`Dynamic` interactive output = snapshot-only (§2) | links UC-55, foreshadows T5 |
|
||||
| CDF = reduced-runtime interactive distribution projection (§2) | links UC-83 (derivation-projection) |
|
||||
|
||||
Mathematica is a **lineage/reinforcement** dive — it **adds no new UC**, confirming UC-84's
|
||||
notebook page shape as a genus and contributing two refinements (nestable cells; a
|
||||
**structured re-evaluable value** point on the content-opacity spectrum).
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12 (page model):** generalize the notebook shape from "ordered cells" (UC-84) to
|
||||
**ordered/nestable cell groups (an outline tree)**; allow code-cell outputs to be **typed
|
||||
structured values**, not only MIME blobs.
|
||||
- **T11/T15 (content opacity / fidelity):** add a **"structured re-evaluable value"** point
|
||||
to the content-opacity spectrum between transparent-text and opaque-blob (Wolfram
|
||||
expression, symbolic result) — relevant to how outputs are stored/surfaced.
|
||||
- **T16 (projection):** CDF is a **reduced-capability interactive projection**; with
|
||||
`Manipulate`/`Dynamic`, static projection is a **snapshot frame only** (join the
|
||||
live-content limit recorded for Strudel T5).
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. Is a **structured re-evaluable result** (Wolfram expression) modeled as a typed value in
|
||||
the page model, or stored opaquely with provenance like other computed outputs? (Ties
|
||||
UC-55 open-Q #10 and UC-84 Q3.)
|
||||
2. Do interactive outputs (`Manipulate`, `Dynamic`, and Jupyter widgets) deserve a shared
|
||||
**"interactive, snapshot-only" content marker** in the contract? (Recurs at T4/T5.)
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- Wolfram documentation: notebook format (`Notebook`/`Cell` expressions), cell types &
|
||||
groups, evaluation (`In`/`Out`), `Manipulate`/`Dynamic`, CDF.
|
||||
- prior: `research/260614-jupyter-deep-dive/` (UC-84 notebook shape; the descendant);
|
||||
`research/260614-literate-programming-deep-dive/` (derivation-projection, UC-83).
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
**No new UC** (lineage/reinforcement). Reinforced: UC-84; enriched: UC-54, UC-55; links
|
||||
UC-34, UC-83. Architecture cross-refs: SHARD-WP-0002 T12 (nestable cell-group outline; typed
|
||||
structured outputs), T11/T15 (structured-re-evaluable-value point on the opacity spectrum),
|
||||
T16 (CDF reduced-capability projection; interactive = snapshot-only).
|
||||
15
research/260614-mojomojo-deep-dive/README.md
Normal file
15
research/260614-mojomojo-deep-dive/README.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# 260614 — MojoMojo deep dive
|
||||
|
||||
Deep dive on **MojoMojo**: a **Perl Catalyst / DBIx::Class** DB-backed wiki — hierarchical
|
||||
pages, attachments, inline (AJAX) editing, Markdown content, and **page history in
|
||||
relational version tables**. The classic **MVC DB-backed** contrast to the file-store
|
||||
classics: no file store, no real content API → attach by reading the **relational store
|
||||
directly**.
|
||||
|
||||
- `findings.md` — architecture, the DB schema shape, capability profile, INTENT mapping, UC
|
||||
seed (UC-81), architecture notes for SHARD-WP-0002, open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-81 (attach a **DB-backed wiki with no file store/API** by reading its
|
||||
relational store directly — pages + version tables — and importing DB-resident history to the
|
||||
journal). Enriched UC-02/40/36/34. Feeds SHARD-WP-0002 T13 (history portability), T14
|
||||
(direct-DB binding).
|
||||
133
research/260614-mojomojo-deep-dive/findings.md
Normal file
133
research/260614-mojomojo-deep-dive/findings.md
Normal file
@@ -0,0 +1,133 @@
|
||||
# MojoMojo — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T8 · **Subject:** MojoMojo, a Perl
|
||||
Catalyst wiki/CMS.
|
||||
|
||||
## Why this dive
|
||||
|
||||
The file-store classics (TWiki, Foswiki, Oddmuse, UseMod) keep pages as files; the modern
|
||||
SaaS keep them behind APIs. MojoMojo is the **classic relational-DB-backed** wiki — a
|
||||
Catalyst MVC app over **DBIx::Class** with pages and their history in **SQL tables**, and
|
||||
**no file store and no first-class content API**. It anchors the *"attach by reading the
|
||||
database directly"* hard case the adapter contract must account for (T13/T14).
|
||||
|
||||
## 1. Architecture
|
||||
|
||||
- **Stack:** Perl **Catalyst** (MVC web framework) + **DBIx::Class** (ORM) over a relational
|
||||
DB (SQLite / PostgreSQL / MySQL). Templating via Template Toolkit.
|
||||
- **Content:** **Markdown** (Text::MultiMarkdown) is the page markup — so the *body* is
|
||||
Markdown, but it lives **in a DB column**, not a file.
|
||||
- **Pages are hierarchical:** a **path tree** (`/parent/child`) modeled as rows with
|
||||
parent/lineage relations — structure is relational, not directory-based.
|
||||
- **Versioning:** each page edit creates a **new version row** (a `page_version`-style
|
||||
table) — full revision history lives in **DB version tables**, with author/timestamp.
|
||||
- **Features:** inline **AJAX editing**, **attachments** (stored as DB rows / blobs +
|
||||
metadata), diffs, RSS feeds, full-text search, per-page permissions.
|
||||
|
||||
## 2. The attach problem — DB or nothing
|
||||
|
||||
MojoMojo exposes its content through the **web app** (HTML) and the **database**; there is
|
||||
**no clean REST/GraphQL content API** and **no file store**. So a shard adapter has two
|
||||
realistic paths:
|
||||
|
||||
1. **Direct relational read** (preferred): read the `page` + `page_version` (+ `content`,
|
||||
`attachment`) tables via DBIx::Class schema — pages, the path tree, and **full history**
|
||||
are all there, importable to the coordination journal (UC-41-style history import, but
|
||||
from **DB version rows** rather than RCS/git).
|
||||
2. **HTML scrape** (fallback): parse rendered pages — lossy, last resort.
|
||||
|
||||
This makes MojoMojo the **direct-DB-read** binding archetype: the canonical store is a
|
||||
relational schema, and the adapter's job is to map that schema to the wiki page model +
|
||||
journal.
|
||||
|
||||
## 3. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | MojoMojo |
|
||||
|--------------------------------|----------|
|
||||
| Attachment mode | **direct DB read** (relational); HTML-scrape fallback; no file store, no API |
|
||||
| Addressing granularity | page (row); path tree via lineage rows |
|
||||
| Content identity | DB page id; path as human key |
|
||||
| Identity vs placement | row id vs path lineage (separable) |
|
||||
| Structure | **relational**: page rows + parent/lineage; attachments as rows |
|
||||
| History | **DB version tables** (per-edit version rows, author/timestamp) |
|
||||
| Merge model | app-level last-writer; DB transactions |
|
||||
| Native query | SQL over the schema (not a wiki query language) |
|
||||
| Translation | **Markdown body in a DB column** — minimal translation, but extraction needed |
|
||||
| Write granularity | page (row) per save |
|
||||
| Operational envelope | a Perl app + its DB; direct DB access needs credentials |
|
||||
| Access grant | per-page permissions in DB; app auth |
|
||||
| Content opacity | transparent if you can read the DB |
|
||||
| Provenance | author/timestamp on version rows |
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Backend-neutral page model**: the body is **Markdown** — once extracted from the DB
|
||||
column it maps directly; the adapter's work is **schema→page-model**, not format
|
||||
translation.
|
||||
- **History portability** (T13): DB **version rows** are a third history-source shape beside
|
||||
git commits and RCS files — importable to the journal as discrete revisions with
|
||||
author/timestamp.
|
||||
- **Graceful degradation**: even with only DB read (no API), MojoMojo is a usable
|
||||
read/projection/backup shard; with DB write it could be write-through, but carefully
|
||||
(app invariants).
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- **No file store, no API** ⇒ the **direct-DB-read** binding is a first-class attach mode the
|
||||
contract must name (alongside file-store, in-engine host, external-API, CRDT, P2P) — or a
|
||||
sub-mode of "external store" where the medium is **a relational schema** (T14). Reading a
|
||||
third-party app's DB is **coupling to its schema** (versioned, may drift across MojoMojo
|
||||
versions) — a stated risk (UC-43 backend-swap analogue at the schema level).
|
||||
- **Writing by direct DB** risks violating app invariants (lineage, version counters,
|
||||
search index) — default to **read/projection/overlay**; write-through only with the app's
|
||||
cooperation.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Direct-DB-read as a named binding** for DB-backed engines with no file/API (UC-81),
|
||||
mapping a **relational schema → wiki page model + journal**.
|
||||
2. **DB version rows as a history source** for the journal (T13), beside git and RCS.
|
||||
3. **Schema-coupling caution** — treat the schema as a versioned interface that can drift
|
||||
(relates UC-43).
|
||||
|
||||
## 5. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-81 | Attach a **DB-backed wiki with no file store / no API** (MojoMojo) by reading its **relational store directly** (page + version tables), mapping schema → page model and **importing DB-resident history** to the journal | **new** |
|
||||
| — | DB attach vs file attach | enrich **UC-02** / **UC-40** |
|
||||
| — | DB version-table history import | enrich **UC-36** |
|
||||
| — | relational page rows / lineage as structure | enrich **UC-34** |
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T14 (binding):** add **direct relational read** as a binding (or external-store sub-mode
|
||||
whose medium is a SQL schema) for DB-backed engines lacking a file store or API; HTML
|
||||
scrape is the lossy fallback. Schema is a **versioned coupling** (drift risk, UC-43).
|
||||
- **T13 (history portability):** **DB version rows** = a history source alongside git commits
|
||||
and RCS revisions — import as discrete journal entries (author/timestamp).
|
||||
- **T11 (capability):** "has-file-store" / "has-API" are **absent** here; "has-readable-DB"
|
||||
is the capability — a sparse profile relying on schema knowledge.
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. Does shard-wiki sanction **direct third-party DB reads** as a binding, or restrict them
|
||||
(schema coupling/drift) to a documented best-effort mode? How is schema drift across
|
||||
MojoMojo versions handled (UC-43)?
|
||||
2. Is **write-through by direct DB** ever allowed (risking app invariants), or are DB-backed
|
||||
no-API engines read/projection/overlay/backup only?
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- MojoMojo — github.com/mojomojo/mojomojo; metacpan MojoMojo (Catalyst app, DBIx::Class
|
||||
schema: Page / PageVersion / Content / Attachment)
|
||||
- Catalyst + DBIx::Class framework docs (architecture context)
|
||||
- prior: `research/260613-twiki-deep-dive/` (file-store classic contrast, UC-40/41)
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
New UC **UC-81** carries the marker **⊙** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-02, UC-40, UC-36, UC-34. Architecture cross-refs:
|
||||
SHARD-WP-0002 T14 (direct-DB binding), T13 (DB version-row history), T11.
|
||||
56
research/260614-notion-deep-dive/README.md
Normal file
56
research/260614-notion-deep-dive/README.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# 260614 — Notion deep dive (closed block-database SaaS, attached only through its API)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Notion** — architecture and extension model — read through
|
||||
shard-wiki's lens. Notion is the **closed, hosted, schema-rich** end of the spectrum and
|
||||
the dive that stress-tests shard-wiki's hardest constraints (*graceful degradation*,
|
||||
*no silent remote mutation*, *Markdown-first that must degrade*).
|
||||
|
||||
It contrasts sharply with its neighbours:
|
||||
- vs **Roam** — both are block databases with per-block UUIDs, but Roam is a *client*
|
||||
DataScript DB reached only via an *in-app* JS API, while Notion is a *server* Postgres
|
||||
store reached via a real *external* REST API (no in-engine hosting needed, but
|
||||
rate-limited, eventually consistent, scoped by per-page grants).
|
||||
- vs **Obsidian** — the mirror image of "file over app": Notion is **app over file**, a
|
||||
closed hosted store with no portable files.
|
||||
- vs **XWiki** — both are structured wiki-as-app-platforms; Notion is the **apex of
|
||||
database-as-pages** (schema + relations + rollups + views) but closed and REST-only.
|
||||
|
||||
Distinctive material:
|
||||
- **Data model** — everything is a **block** (UUID id, type, properties, ordered child
|
||||
`content`, single `parent`); pages are blocks, **database rows are pages with a
|
||||
schema**; Postgres-backed hosted SaaS
|
||||
- **Databases** — typed properties, **relations** (bidirectional), **rollups**,
|
||||
**formulas**, and many **views** (table/board/calendar/gallery) of one row-set
|
||||
- **Extension model** — **no in-app plugin runtime**; extension = **external REST API**
|
||||
integrations (+ webhooks since 2026), with ~3 rps rate limit, eventual consistency,
|
||||
recursive child fetch, payload caps, and **scoped/revocable per-page access grants**
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Block model, databases, page-model impedance, the API-only extension model, capability profile, scoped consent, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-57 attach a closed external-API-only hosted shard with rate-limit/eventual-
|
||||
consistency/scoped-grant constraints, UC-58 attach a typed database with schema +
|
||||
relations + views without flattening, UC-59 lossy-aware translation with a fidelity
|
||||
report); UC-31/34/36/39/50/51/52/54/56 enriched. Logged for `SHARD-WP-0002` (T14):
|
||||
an **operational-envelope** capability section, **access-grant semantics**, a
|
||||
**translation-fidelity** capability, and a three-way **attachment-mode taxonomy**
|
||||
(file-store / in-engine-hosted / external-API-only).
|
||||
|
||||
**Key takeaways recorded:** Notion *enforces* "no silent remote mutation" via scoped,
|
||||
revocable per-integration grants (ties the authz-in-core decision); it is the heaviest
|
||||
**lossy-translation** case (proprietary blocks/rich-text, lossy Markdown export); and it
|
||||
adds the **external-API-only** attachment mode (full write-through without in-engine
|
||||
hosting, but bounded by rate limits and eventual consistency). **Boundary:** one
|
||||
external-API candidate shard — best as projection/mirror/overlay/backup — not a
|
||||
substrate and not the federation layer.
|
||||
</content>
|
||||
302
research/260614-notion-deep-dive/findings.md
Normal file
302
research/260614-notion-deep-dive/findings.md
Normal file
@@ -0,0 +1,302 @@
|
||||
# Findings — Notion: a closed block-database SaaS, attached only through its API
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — a hosted block-database SaaS; a *candidate
|
||||
shard* of a distinct family (closed, external-REST-API-only, schema-rich, heaviest
|
||||
translation cost); has **no in-app plugin runtime**
|
||||
Lens: shard-wiki — the external-API attachment mode, database-as-pages, lossy
|
||||
translation/fidelity, scoped consent, and graceful degradation against a sovereign
|
||||
closed backend
|
||||
|
||||
> Where Notion sits in the set. Roam and Notion are both **block databases with
|
||||
> per-block UUIDs**, but they are opposite on access: Roam is a *client-side*
|
||||
> DataScript DB reachable only through an *in-app* JS API (write-through needs an
|
||||
> adapter hosted inside Roam); **Notion is a *server-side* Postgres store reachable
|
||||
> through a real *external* REST API** — attachable from outside with no in-engine
|
||||
> hosting, but rate-limited, eventually consistent, and **scoped by explicit
|
||||
> per-integration page grants**. And against Obsidian's "file over app," Notion is the
|
||||
> mirror image: **app over file** — a closed hosted store with no portable files at
|
||||
> all. So Notion is the dive that stress-tests shard-wiki's hardest constraints:
|
||||
> *graceful degradation*, *no silent remote mutation*, *union without erasure*, and
|
||||
> *Markdown-first that must degrade* against a proprietary, schema-rich, non-Markdown
|
||||
> backend.
|
||||
|
||||
Pairs with — and contrasts against — the Roam dive (block-DB, in-app vs external API),
|
||||
the Obsidian dive (closed-hosted vs file-over-app), and the XWiki dive (both are
|
||||
structured wiki-as-app-platforms; Notion is the apex of database-as-pages but closed
|
||||
and REST-only).
|
||||
|
||||
---
|
||||
|
||||
## 1. Core data model — everything is a block
|
||||
|
||||
Notion's data model is uniform: **"Everything you see in Notion is a block. Text,
|
||||
images, lists, a row in a database, even pages themselves — these are all blocks."**
|
||||
|
||||
Each block record holds:
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| **id** | a randomly generated **UUID v4**, visible in the page URL — the block's stable address |
|
||||
| **type** | determines rendering + which properties apply (text, to-do, heading, **page**, **child_database**, …) |
|
||||
| **properties** | type-specific attributes (e.g. `title` text; for database rows, the typed property values) |
|
||||
| **content** | an **ordered array of child block IDs** — nesting |
|
||||
| **parent** | a single upward pointer — used for **permission inheritance** |
|
||||
|
||||
Blocks form a **render tree** (content + parent pointers); indentation is *structural*,
|
||||
not cosmetic — it changes block relationships. **Pages are blocks**; **databases are
|
||||
blocks whose children are pages**; a database **row is a page** with typed properties.
|
||||
Stored in **Postgres** on Notion's servers (hosted SaaS — not local, not files).
|
||||
|
||||
The shard-wiki-relevant facts: (a) **per-block UUIDs** give native sub-page addressing
|
||||
(UC-51); (b) the **page = block, database-row = page-with-schema** identity is a real
|
||||
page-model impedance (§3); (c) the store is **closed and server-side** — reachable only
|
||||
through the API (§4).
|
||||
|
||||
---
|
||||
|
||||
## 2. Databases — schema, typed properties, relations, views
|
||||
|
||||
A Notion **database** is a collection of pages sharing a **schema** of typed
|
||||
**properties**: `title`, `rich_text`, `select` / `multi_select`, `number`, `date`,
|
||||
`checkbox`, `person`, `files`, **`relation`** (a typed link to rows in another
|
||||
database — shown on *both* sides, i.e. bidirectional), **`rollup`** (an aggregate over a
|
||||
relation), and **`formula`** (computed). The same row-set is shown through multiple
|
||||
**views** — **table, board (kanban), calendar, gallery, list, timeline** — each a
|
||||
filtered/sorted/grouped projection of the same data.
|
||||
|
||||
For shard-wiki this is the **apex of "wiki page as structured record"** (stronger than
|
||||
XWiki XObjects, UC-39): a page can be *bodiless typed data*, pages are joined by
|
||||
**typed inter-database relations** (a graph of typed links), and "the page" is routinely
|
||||
**a row in a schema** rather than prose. Notion databases also ship the ZigZag insight
|
||||
commercially: **one row-set, many views/dimensions** (UC-47/48) and **filtered/linked
|
||||
databases = query-defined pages** (UC-54).
|
||||
|
||||
---
|
||||
|
||||
## 3. Page-model impedance
|
||||
|
||||
Mapping Notion to a Markdown-first page model is the **heaviest translation case** in
|
||||
the research set:
|
||||
|
||||
- **Block/rich-text ≠ Markdown.** Notion's rich text is an annotated-span model; many
|
||||
block types (synced blocks, columns, callouts, embeds, database views) have no clean
|
||||
Markdown equivalent. Notion's own **export to Markdown/CSV is lossy** (databases →
|
||||
CSV, relations/rollups/formulas flatten or drop).
|
||||
- **Database-row-as-page + schema** must map onto pages + sidecar metadata without
|
||||
discarding the schema or the relations (extends UC-34/UC-39).
|
||||
- Therefore translation must be **lossy-aware with a fidelity report** — surface what
|
||||
did *not* round-trip rather than silently flattening (UC-59). This is *different from*
|
||||
UC-42 (Foswiki TML↔HTML *lossless* round-trip); Notion is fundamentally lossy.
|
||||
|
||||
---
|
||||
|
||||
## 4. Extension model — there is no plugin runtime; only the API
|
||||
|
||||
This is the defining architectural fact for shard-wiki: **Notion has no third-party
|
||||
in-app plugin/extension system.** There is no marketplace of code that runs inside
|
||||
Notion (the only "in-app modding" is the *unofficial* Notion Enhancer desktop patcher,
|
||||
out of scope). **Extension = external integration via the public REST API** (plus
|
||||
embeds and, recently, webhooks).
|
||||
|
||||
The **public REST API** (`api.notion.com/v1`):
|
||||
|
||||
- **Resources:** `pages`, `blocks` (retrieve / append / update / delete children),
|
||||
`databases` (retrieve, **query** with filters+sorts, create), `users`, `search`,
|
||||
`comments`.
|
||||
- **Authorization:** an **internal integration token** (workspace-owned) or **OAuth
|
||||
2.0** (public integrations). Critically, **an integration only sees pages a user has
|
||||
explicitly connected to it** — the user "approves the app and connects specific
|
||||
pages" via *Add connections*. Access is a **scoped, revocable, per-page grant** (§6).
|
||||
- **Operational constraints the adapter must encode:**
|
||||
- **Rate limit** ~**3 requests/second** average (429 `rate_limited`); **no paid
|
||||
increase**.
|
||||
- **Eventual consistency / no live read model** — the API is not real-time; reads can
|
||||
lag.
|
||||
- **Recursive retrieval** — a block returns only **first-level children**; full-page
|
||||
reconstruction requires walking `has_children` recursively.
|
||||
- **Payload caps** — 1000 blocks / 500 KB per request; child arrays ≤ 100.
|
||||
- **Webhooks** (added 2026) deliver page/database change events — a push transport for
|
||||
UC-31, replacing pure polling.
|
||||
- **Version history:** Notion keeps **internal page history** (retention bounded by
|
||||
plan); it is **not portable** and not exposed as git — a UC-36 *supplementation* case
|
||||
(like Confluence/MediaWiki), not an import case.
|
||||
- **Publish:** pages can be **published to the web** as read-only public pages — an
|
||||
outbound publish surface (UC-56).
|
||||
|
||||
Consequence: Notion is attachable **only** as an **external-API shard** — but, unlike
|
||||
Roam, **no in-engine adapter hosting is needed** (the REST API is external). The cost is
|
||||
the operational envelope (rate-limit, eventual consistency, scoped grant) that the
|
||||
capability profile must model (UC-57).
|
||||
|
||||
---
|
||||
|
||||
## 5. Notion as a shard — capability profile
|
||||
|
||||
| Capability | Notion | Notes for the adapter contract |
|
||||
|------------|--------|--------------------------------|
|
||||
| Read | **yes (external REST)** | rate-limited (~3 rps), eventually consistent, recursive child fetch |
|
||||
| Write | **yes (external REST)** | append/update/delete blocks, create pages — no in-engine host needed (vs Roam) |
|
||||
| Write granularity | **block-level (fine)** | like Roam; per-block ops |
|
||||
| Identity / addressing | **block UUID v4** | native sub-page addressing (UC-51), store-minted (like Roam, not in-file) |
|
||||
| Structured data | **yes (apex)** | databases: schema + typed properties + relations + rollups + formulas (UC-34/39/58) |
|
||||
| Native query | **yes** | database query API (filters/sorts) → delegate views (UC-52) |
|
||||
| Views / dimensions | **yes** | table/board/calendar/gallery = many views of one row-set (UC-47/48/54) |
|
||||
| Subscribe | **webhooks (2026)** | push events; else poll (UC-31) |
|
||||
| Version history | **internal, not portable** | supplement via coordination journal (UC-36) |
|
||||
| Diff / merge | **no native** | — |
|
||||
| Lock | **no** | — |
|
||||
| Publish | **publish-to-web** | outbound read-only (UC-56) |
|
||||
| Access model | **scoped per-page grant (OAuth/token)** | explicit consent; revocable (UC-57, authz) |
|
||||
| Syntax / content | **proprietary block + rich text** | **lossy** to Markdown; needs fidelity-aware translation (UC-59) |
|
||||
|
||||
Verdict: Notion is a legitimate but **demanding** shard — **external-API-attached,
|
||||
schema-rich, fine-grained, closed**. It behaves best as a **projected / mirrored /
|
||||
overlay / backup** participant; full write-through is possible but bounded by rate limits
|
||||
and eventual consistency. The strongest reasons to attach it: structured databases
|
||||
(UC-58) and block-UUID addressing (UC-51); the strongest cautions: lossy translation
|
||||
(UC-59) and no portable history (UC-36).
|
||||
|
||||
---
|
||||
|
||||
## 6. Scoped consent and "no silent remote mutation"
|
||||
|
||||
Notion *enforces* one of shard-wiki's INTENT constraints at the platform level: an
|
||||
integration can touch **only** the pages a user has explicitly **connected** to it, and
|
||||
the grant is **revocable**. This is a clean, real-world model of **no silent remote
|
||||
mutation** and of a shard granting the orchestrator **scoped, consented access** — and
|
||||
it ties directly to the settled **authz-in-core / authn-delegated** decision
|
||||
([[shard-wiki-auth-in-core-decision]]): authentication to Notion is delegated (OAuth /
|
||||
integration token), while shard-wiki's *own* authorization decides what to do with the
|
||||
granted scope. The adapter contract should treat **"scoped, revocable grant"** as a
|
||||
first-class attachment property (UC-57), not an afterthought.
|
||||
|
||||
---
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
|
||||
- **Graceful degradation** has its sharpest test here: a closed SaaS with no files, rate
|
||||
limits, eventual consistency, and lossy export must still be usable as
|
||||
read/projection/overlay/backup. If the adapter contract handles Notion, it handles
|
||||
most things.
|
||||
- **No silent remote mutation** is *modeled by the platform* (scoped grants) — Notion
|
||||
validates the principle (§6).
|
||||
- **Database-as-pages** validates that "wiki page" must stretch to **typed records with
|
||||
relations**, not just prose (UC-34/39/58).
|
||||
- **Block UUIDs** reconfirm (with Roam) that native sub-page addressing is real and
|
||||
adoptable (UC-51).
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Closed hosted store; no sovereignty over bytes.** shard-wiki cannot make Notion
|
||||
git-native or local. It can mirror/project/overlay/back-up and **supply** a
|
||||
git-addressable history (UC-36) — never claim to own Notion's store.
|
||||
2. **Lossy, proprietary content.** Do **not** pretend Notion round-trips to Markdown.
|
||||
Translate lossily *with a fidelity report* and preserve non-mappable elements as
|
||||
provenance/sidecar (UC-59) — union without erasure includes erasure *of fidelity*
|
||||
being made visible.
|
||||
3. **External-API-only, rate-limited, eventually consistent.** Projection must be
|
||||
**cache/poll/webhook**, not a live read model; sync is bounded — encode this in the
|
||||
capability profile (UC-57). Do not design flows that assume cheap, instant, unlimited
|
||||
reads.
|
||||
4. **One workspace (or its granted page set) = one shard**, never the federation layer.
|
||||
|
||||
### 7.3 What Notion teaches that shard-wiki should keep
|
||||
|
||||
- Model **operational envelope** (rate limit, consistency class, payload caps,
|
||||
pagination) as explicit capability-profile fields — Notion makes them unignorable.
|
||||
- Model **scoped, revocable consent** as a first-class attachment property (UC-57, §6).
|
||||
- Treat **translation fidelity as data**: a per-shard, per-page report of what projects
|
||||
cleanly vs. degrades (UC-59) — applies beyond Notion.
|
||||
- Recognize **external-REST attach** as a distinct, *preferred-where-available* mode:
|
||||
full write-through without in-engine hosting (contrast Roam) — but pay the operational
|
||||
envelope.
|
||||
|
||||
---
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-56**. New UCs **UC-57–UC-59** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a closed hosted shard via its external REST API only** — no file store, no in-app runtime — honoring rate limits, eventual consistency, payload caps, and a scoped/revocable access grant | **UC-57 (new)** |
|
||||
| **Attach a typed database (schema + relations + rollups + multiple views) as a shard** without flattening the schema or the inter-record relations | **UC-58 (new)** |
|
||||
| **Translate a proprietary block/rich-text model to/from Markdown with an explicit fidelity report**, preserving non-mappable elements rather than silently dropping them | **UC-59 (new)** |
|
||||
| Block UUIDs = store-minted native span addresses (external-API variant) | **enriches UC-51** |
|
||||
| External-API block-DB attach (no in-engine host) — contrast Roam's in-app-only | **enriches UC-50** |
|
||||
| Database query API + filtered/linked DBs | **enriches UC-52, UC-54** |
|
||||
| Database-as-pages apex; typed records + relations | **enriches UC-34, UC-39** |
|
||||
| Webhooks (2026) as a push transport | **enriches UC-31** |
|
||||
| Internal-only page history, not portable | **enriches UC-36** |
|
||||
| Publish-to-web outbound | **enriches UC-56** |
|
||||
| Scoped, revocable per-integration grant; no silent mutation | links **UC-57** + [[shard-wiki-auth-in-core-decision]] |
|
||||
|
||||
---
|
||||
|
||||
## 9. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- Add an **operational-envelope** section to the adapter capability profile: rate limit,
|
||||
consistency class (live / eventually-consistent / snapshot), payload/pagination caps,
|
||||
recursive-fetch requirement, push-vs-poll transport. Notion is the forcing example.
|
||||
- Add **access-grant semantics**: scope (which pages), revocability, auth mode
|
||||
(delegated token/OAuth) — ties the authz-in-core decision and "no silent mutation".
|
||||
- Add a **translation-fidelity capability**: adapters declare and report what content
|
||||
round-trips vs. degrades (UC-59); generalizes UC-42 (lossless) to the lossy case.
|
||||
- **Attachment-mode taxonomy** now spans: file-store direct (Obsidian/TWiki, UC-40),
|
||||
in-engine hosted adapter (Roam/XWiki, UC-38/50), and **external-API-only**
|
||||
(Notion, UC-57). T14 binding should enumerate all three.
|
||||
- **Database/schema/relations** as a unit (UC-58) presses the page-model spec: collection
|
||||
+ schema + typed relations, not just a page.
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions (for spec / workplans)
|
||||
|
||||
1. Is **external-API-only with a tight rate limit** (Notion) viable for write-through at
|
||||
wiki scale, or do we cap Notion at read/projection/overlay/backup by default?
|
||||
2. How are **inter-database relations** (UC-58) represented in the union — as typed links
|
||||
in the link graph, as a separate relation index (cf. ZigZag many-to-many), or both?
|
||||
3. What is the **fidelity report** format (UC-59), and where does it surface — provenance
|
||||
panel, projection metadata, reconciliation review?
|
||||
4. For **scoped grants** (§6), how does shard-wiki represent partial visibility (only
|
||||
*some* of a workspace's pages granted) without misrepresenting the shard as complete?
|
||||
5. Do we consume Notion **webhooks** (push) or poll, given eventual consistency and the
|
||||
rate limit (UC-31)?
|
||||
|
||||
---
|
||||
|
||||
## 11. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Notion — "The data model behind Notion's flexibility" (https://www.notion.com/blog/data-model-behind-notion) | Everything-is-a-block; block record (id/type/properties/content/parent); render tree; pages/databases as blocks; Postgres |
|
||||
| Notion Docs — Request limits (https://developers.notion.com/reference/request-limits) | ~3 rps rate limit, 429, payload caps, recursive first-level children |
|
||||
| Notion Docs — Authorization (https://developers.notion.com/docs/authorization) | Internal token vs OAuth; integration connected to specific pages; scoped grant |
|
||||
| Hookdeck / ClickUp — Notion webhooks guides (https://hookdeck.com/webhooks/platforms/guide-to-notion-webhooks-features-and-best-practices) | Webhook support (2026), page/database change events |
|
||||
| Truto / Rollout — Notion API architecture & essentials (https://truto.one/blog/how-to-integrate-with-the-notion-api-architecture-guide-for-b2b-saas/) | REST endpoints (pages/blocks/databases/search), integration patterns, no in-app plugin model |
|
||||
| General API knowledge — database property types, views, relations/rollups/formulas, export-to-Markdown lossiness | §2, §3 |
|
||||
|
||||
Cross-references: `research/260614-roam-deep-dive/findings.md` (block-DB/UUID, in-app vs
|
||||
external API), `research/260614-obsidian-deep-dive/findings.md` (closed-hosted vs
|
||||
file-over-app), `research/260613-xwiki-deep-dive/findings.md` (structured wiki-app-
|
||||
platform), `spec/UseCaseCatalog.md` (UC-31, UC-34, UC-36, UC-39, UC-50/51/52, UC-54,
|
||||
UC-56), `workplans/SHARD-WP-0002-federation-architecture.md` (T14), and the authz
|
||||
decision [[shard-wiki-auth-in-core-decision]].
|
||||
|
||||
---
|
||||
|
||||
## 12. Traceability
|
||||
|
||||
- New UCs: **UC-57, UC-58, UC-59** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-31, UC-34, UC-36, UC-39, UC-50, UC-51, UC-52, UC-54, UC-56**.
|
||||
- Architecture (no UC): operational-envelope + access-grant + translation-fidelity
|
||||
capability fields; three-way attachment-mode taxonomy; database/schema/relations in the
|
||||
page model → `SHARD-WP-0002` (T14).
|
||||
- Decision link: scoped/revocable grant + no-silent-mutation → [[shard-wiki-auth-in-core-decision]].
|
||||
- Boundary recorded: Notion is **one external-API candidate shard** — closed, hosted,
|
||||
schema-rich, lossy-to-Markdown — best as projection/mirror/overlay/backup; not a
|
||||
substrate, not the federation layer (INTENT graceful-degradation + no-silent-mutation).
|
||||
</content>
|
||||
51
research/260614-obsidian-deep-dive/README.md
Normal file
51
research/260614-obsidian-deep-dive/README.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# 260614 — Obsidian deep dive (file-over-app vaults, plugin API, ecosystem signal)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Obsidian** — core/extension architecture and the plugin
|
||||
ecosystem — read through shard-wiki's lens. Obsidian is the **most INTENT-aligned tool
|
||||
in the research set**: INTENT names Obsidian vaults as a shard participant, and its
|
||||
**"file over app"** model (notes are plain Markdown files in a folder you own; the app
|
||||
is a lens) is the closest cousin to shard-wiki's Markdown-first, sovereignty-preserving
|
||||
thesis. It is the **file-backed counterpart to the Roam dive** (same tool category,
|
||||
opposite storage model: Roam = client DB + API; Obsidian = files on disk).
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — vault = folder of `.md` + `.obsidian/` config; files canonical;
|
||||
**MetadataCache** is a *derived*, async-rebuilt index (links, embeds, tags,
|
||||
frontmatter, block IDs, resolved/unresolved links, backlinks)
|
||||
- **In-file, git-diffable** addressing/structure — `[[wikilinks]]`, `![[embeds]]`
|
||||
(transclusion), `^block-id` span refs, YAML frontmatter/properties
|
||||
- **Plugin API** — `Plugin` `onload/onunload`; `App` modules `vault` (CRUD + file
|
||||
events), `metadataCache`, `workspace` (CM6, views), `fileManager.renameFile`;
|
||||
`manifest.json`; community plugins, Restricted Mode
|
||||
- **Ecosystem popularity → UC signal** (§7) — ranked download data mined for use cases,
|
||||
per the research brief
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | File-over-app architecture, in-file addressing, plugin API as adapter host, capability profile, **plugin-popularity → UC mapping**, derived-index lesson, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Four new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-53 attach a local vault with a live concurrent native editor, UC-54 query-defined
|
||||
dynamic page, UC-55 carry non-Markdown content types, UC-56 outbound publish of a
|
||||
projection); UC-15/28/34/36/40/51/52 enriched. Logged for `SHARD-WP-0002` (T14):
|
||||
**dual attachment mode** (file-store *or* in-app plugin host), consume-native-derived-
|
||||
index capability, non-Markdown content types in the page model, outbound publish,
|
||||
external-writer tolerance.
|
||||
|
||||
**Key takeaways recorded:** (1) Obsidian validates *files-canonical / index-derived* —
|
||||
shard-wiki's projection model. (2) Fine-grained addressing & structure can be portable
|
||||
**in-file text** (`^block-id`, frontmatter), not DB state (contrast Roam). (3) The
|
||||
ecosystem is demand evidence — the #1 plugin is **drawings** (non-Markdown content),
|
||||
query-as-DB (Dataview/Tasks) is top-tier but an *add-on*, and **Git is top-7** (users
|
||||
bolt on the portable history shard-wiki provides natively). **Boundary:** a vault is one
|
||||
file-backed candidate shard, not the federation layer and not a file-sync target
|
||||
(not-a-sync-daemon).
|
||||
</content>
|
||||
324
research/260614-obsidian-deep-dive/findings.md
Normal file
324
research/260614-obsidian-deep-dive/findings.md
Normal file
@@ -0,0 +1,324 @@
|
||||
# Findings — Obsidian: file-over-app vaults, the plugin API, and what the ecosystem reveals
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — a local-first Markdown vault tool; the
|
||||
*most INTENT-aligned candidate shard yet* (file-backed, Markdown-first); plus an
|
||||
ecosystem whose popularity is direct evidence of what users actually do
|
||||
Lens: shard-wiki — file-store attachment, in-file fine-grained addressing, derived
|
||||
index vs. canonical files, the engine-hosts-adapter path, and ecosystem-driven UCs
|
||||
|
||||
> Why Obsidian matters most. INTENT names **Obsidian vaults** explicitly as a shard
|
||||
> participant, and Obsidian's **"file over app"** philosophy — your notes are plain
|
||||
> Markdown files in a folder you own, the app is just a lens — is the closest cousin
|
||||
> to shard-wiki's own Markdown-first, sovereignty-preserving thesis. It is also the
|
||||
> **file-backed counterpart to the Roam dive**: same "personal knowledge tool"
|
||||
> category, opposite storage model (Roam = client DB + API; Obsidian = files on disk).
|
||||
> And uniquely among the dives, Obsidian has a **large, ranked plugin ecosystem** — so
|
||||
> the user's instruction to let *plugin popularity inform the use-case catalog* is
|
||||
> actionable: what people install tells us what a federated wiki must support (§7).
|
||||
|
||||
This dive treats Obsidian as a candidate shard (capability profile §5), studies its
|
||||
plugin API as an adapter-host surface (§4), and mines the download rankings for UCs
|
||||
(§7). Pairs with the Roam dive (the DB/API contrast) and the TWiki/Foswiki dives (the
|
||||
file-store attachment path, UC-40).
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture — file over app
|
||||
|
||||
- A **vault** is a local folder of plain **Markdown files** plus a `.obsidian/` config
|
||||
directory (settings, enabled plugins, themes, workspace layout). **The files are the
|
||||
source of truth**; the app holds no separate canonical database.
|
||||
- Markdown is **Obsidian-flavored**: CommonMark + `[[wikilinks]]`, `![[embeds]]`
|
||||
(transclusion), `^block-id` block references, `#tags`, YAML **frontmatter /
|
||||
properties** (typed metadata), and callouts. Note: all addressing and structure live
|
||||
**in the file text** — so they are git-diffable and portable.
|
||||
- **MetadataCache** is a *derived* index: Obsidian pre-parses every file into cached
|
||||
metadata (headings, links, embeds, tags, frontmatter, **blocks/block IDs**,
|
||||
`resolvedLinks` / `unresolvedLinks`, backlinks). It is **rebuilt from files** and
|
||||
updated **asynchronously** ("don't rely on immediate updates after a modify"). Files
|
||||
canonical; index derived.
|
||||
- **Canvas** files (`.canvas`, the open **JSON Canvas** format) hold spatial
|
||||
arrangements — a non-Markdown content type living in the vault.
|
||||
- Sync is *not* built into the format: users add Git, Obsidian Sync, or a file-sync
|
||||
plugin (see §7). The vault is just files.
|
||||
|
||||
For shard-wiki this is the **ideal file-backed shard**: Markdown-first, on disk,
|
||||
git-friendly, with a derived index model that mirrors shard-wiki's own
|
||||
projection/derived-view philosophy (the cache is to Obsidian what projections and union
|
||||
BackLinks are to shard-wiki — computed, not canonical).
|
||||
|
||||
---
|
||||
|
||||
## 2. Addressing, links, transclusion — in-file and git-diffable
|
||||
|
||||
- **Links:** `[[Page]]` and `[[Page#Heading]]` / `[[Page#^block-id]]`; the resolved/
|
||||
unresolved link sets and **backlinks** are computed in MetadataCache — Obsidian's
|
||||
shipped answer to union BackLinks (UC-05/UC-18), but per-vault.
|
||||
- **Fine-grained addressing:** a **`^block-id`** suffix on a paragraph (and heading
|
||||
anchors) gives a stable sub-page address — *but it lives in the Markdown text*, not in
|
||||
a database. This is the **git-diffable, portable variant** of Roam's `:block/uid`
|
||||
(which lives in a DataScript DB). Important contrast for UC-51: native span IDs can be
|
||||
**text-embedded** (Obsidian) or **store-minted** (Roam); the text-embedded form is
|
||||
more portable and survives a file copy, but is opt-in (the user must add the `^id`).
|
||||
- **Transclusion:** `![[Page]]`, `![[Page#Heading]]`, `![[Page#^block-id]]` embed live
|
||||
content by reference — shipped transclusion at page/section/block granularity (UC-32),
|
||||
again stored as plain text.
|
||||
|
||||
---
|
||||
|
||||
## 3. Structured data — frontmatter as git-diffable records
|
||||
|
||||
YAML **frontmatter / properties** make each note a lightweight typed record
|
||||
(`tags`, `aliases`, arbitrary keys). This is the **git-diffable structured-data**
|
||||
variant (cf. TWiki `%META%` inside the file, UC-34) — structure lives in the file, not a
|
||||
DB. The **Dataview** plugin (§7) turns frontmatter + inline fields + the link graph into
|
||||
a queryable database — but that query power is an *ecosystem add-on*, not core.
|
||||
|
||||
---
|
||||
|
||||
## 4. Extension architecture — the plugin API as an adapter host
|
||||
|
||||
Obsidian plugins are TypeScript, shipped as `manifest.json` + `main.js` (+ optional
|
||||
`styles.css`):
|
||||
|
||||
- **Manifest:** `id`, `name`, `version`, `minAppVersion` (required); `author`,
|
||||
`description`, `isDesktopOnly`, `fundingUrl` (optional). Community plugins are
|
||||
GitHub-hosted and listed in the `obsidian-releases` repo; installed from inside the
|
||||
app; gated by **Restricted Mode** (plugins run arbitrary code — Node/Electron on
|
||||
desktop).
|
||||
- **Plugin lifecycle:** extend `Plugin`, implement **`onload()` / `onunload()`**; use
|
||||
`registerEvent` / `registerDomEvent` / `registerInterval` for auto-cleanup.
|
||||
- **The `App` object** (global singleton, `this.app`) exposes four modules:
|
||||
- **`vault`** — file CRUD: `read` / `cachedRead`, `create`, `modify`, `process`,
|
||||
`delete`, `rename`, `getFiles`, low-level `adapter`, and **events**
|
||||
(`on('create'|'modify'|'delete'|'rename')`) for live file-watching.
|
||||
- **`metadataCache`** — `getFileCache`, `frontmatter`, `links`, `embeds`, `tags`,
|
||||
`headings`, `blocks`, `resolvedLinks` / `unresolvedLinks`, `getBacklinksForFile`.
|
||||
- **`workspace`** — panes/leaves/views; `registerView`, `registerEditorExtension`
|
||||
(CodeMirror 6), `registerMarkdownPostProcessor`.
|
||||
- **`fileManager`** — high-level ops; **`renameFile`** updates inbound links (use it
|
||||
instead of raw vault rename to keep link integrity).
|
||||
- **Registrations:** `addCommand`, `addRibbonIcon`, `addStatusBarItem`,
|
||||
`addSettingTab`, `registerView`, `registerEditorExtension`,
|
||||
`registerMarkdownPostProcessor`.
|
||||
|
||||
Two consequences for shard-wiki:
|
||||
|
||||
1. **A vault is attachable two ways.** (a) **File-store direct attach** (UC-40) — read
|
||||
the folder as a folder shard; no plugin, offline-capable, git-native; good for
|
||||
read/projection/overlay. (b) **In-app plugin host** (UC-38) — a shard-wiki adapter
|
||||
*as an Obsidian plugin* drives `vault` write + `metadataCache` reads + live file
|
||||
events for high-fidelity write-through. This dual mode is a cleaner story than Roam
|
||||
(which only offers the in-app path).
|
||||
2. The **MetadataCache API is a ready-made derived-view source** — backlinks, tags,
|
||||
block IDs, resolved links — an adapter can consume instead of reparsing.
|
||||
|
||||
---
|
||||
|
||||
## 5. Obsidian as a shard — capability profile
|
||||
|
||||
| Capability | Obsidian | Notes for the adapter contract |
|
||||
|------------|----------|--------------------------------|
|
||||
| Read | **yes** | direct file read (folder shard) or `vault.read`/`cachedRead` in-app |
|
||||
| Write | **yes** | direct file write, or `vault.modify`/`create`/`process` in-app; per-file granularity |
|
||||
| Write granularity | **per-file (page)** | the natural shard-wiki granularity — contrast Roam (block) and TiddlyWiki (whole-file-single-vault) (UC-35) |
|
||||
| Identity / addressing | **path + in-file `^block-id` / headings** | git-diffable, portable, **opt-in** span IDs (UC-51 text-embedded variant) |
|
||||
| Transclusion | **yes** | `![[...]]` page/section/block embeds (UC-32) |
|
||||
| Backlinks / links | **yes (derived)** | MetadataCache resolved/unresolved links + backlinks (UC-05/18) |
|
||||
| Structured data | **yes (in-file)** | YAML frontmatter/properties; queryable via Dataview plugin (UC-34) |
|
||||
| Native query | **plugin only** | Dataview — *not core*; informs UC-52 (query is adapter/plugin-provided, not assumable) |
|
||||
| Version history | **none native** | users add the Git plugin (top-7!) → validates the coordination journal (UC-36) |
|
||||
| Diff / merge | **none native** | git-level if Git plugin / repo-backed |
|
||||
| Lock | **no** | local-first, single-user assumption |
|
||||
| Publish | **via Obsidian Publish / Quartz / Digital Garden** | outbound publish of a projection (UC-56) |
|
||||
| Syntax | **Obsidian-flavored Markdown** | close to CommonMark; wikilinks/embeds/callouts/`^id` need adapter awareness (cf. UC-42) |
|
||||
| Non-Markdown content | **Canvas (JSON Canvas), attachments, Excalidraw** | typed/opaque assets in the vault (UC-55) |
|
||||
|
||||
Verdict: Obsidian is the **cleanest file-backed, Markdown-first shard** — the
|
||||
file-store family (TWiki/Foswiki/DokuWiki, UC-40), and the *reference* personal-vault
|
||||
shard INTENT already names. Its history/query/sync gaps are exactly what users fill with
|
||||
plugins (§7) — i.e. exactly what shard-wiki offers as orchestration.
|
||||
|
||||
---
|
||||
|
||||
## 6. The derived-index lesson (architecture)
|
||||
|
||||
Obsidian's **files-canonical / MetadataCache-derived** split is independent validation
|
||||
of shard-wiki's core stance: the link graph, backlinks, tags, and block index are
|
||||
**computed projections over canonical files**, rebuilt on change, eventually consistent
|
||||
("updated asynchronously"). shard-wiki's union BackLinks, projections, and derived views
|
||||
(UC-05, UC-17–UC-20) should likewise be **derived and rebuildable**, never a second
|
||||
source of truth. The cache-invalidation/async-update caveat is a real design note for
|
||||
projection freshness (UC-31).
|
||||
|
||||
---
|
||||
|
||||
## 7. Ecosystem popularity → use-case signal (the user's ask)
|
||||
|
||||
All-time download ranks (obsidianstats.com), read as *demand evidence*:
|
||||
|
||||
| Plugin (downloads) | What users do with a vault | shard-wiki signal → UC |
|
||||
|--------------------|----------------------------|------------------------|
|
||||
| **Excalidraw** (6.4M) | drawings/diagrams as first-class content | non-Markdown content types in a shard → **UC-55 (new)** |
|
||||
| **Templater** (4.6M), **QuickAdd** (1.9M) | templated note creation | blueprint pages → **UC-15 (enriched)** |
|
||||
| **Dataview** (4.4M) | query the vault (files+frontmatter+links) as a DB | query-defined dynamic pages → **UC-54 (new)**; query is a *plugin* → **UC-52 (enriched)** |
|
||||
| **Tasks** (3.6M) | aggregate to-dos across all notes | cross-page typed-item aggregation → **UC-54** |
|
||||
| **Advanced Tables** (2.9M) | structured tables in Markdown | in-file structured data → UC-34 |
|
||||
| **Calendar** (2.8M) | daily-note navigation | temporal dimension (ZigZag `d.recent`) → UC-17 |
|
||||
| **Git** (2.7M, top-7) | version control + sync on the vault | **history is a bolt-on, not native** → validates coordination journal → **UC-36 (enriched)** |
|
||||
| **Kanban** (2.4M) | board view of Markdown cards | alternate projection/view of pages → UC-47/48 |
|
||||
| **Remotely Save** (2.0M) | sync vault to S3 / Dropbox / OneDrive / GDrive / **WebDAV** | demand to federate a vault to heterogeneous remote stores → reinforces INTENT WebDAV/Nextcloud shards (but shard-wiki is *not* a file-sync daemon — attach as shards, don't mirror) |
|
||||
| **Omnisearch** (1.6M) | search across notes, PDFs, images | union full-text search over mixed content → UC-20 |
|
||||
| **Importer** (1.4M) | import from Evernote / Notion / Bear / Apple Notes | carry-forward/import from foreign tools → **UC-28 (enriched)** |
|
||||
| **Smart Connections** (1.0M), **Copilot** (1.5M), **Claudian** (938K) | AI over the vault (related notes, chat) | mostly out of scope; "related notes" ≈ link/equivalence discovery (UC-46) — note the trend, don't build it into core |
|
||||
| **Tag Wrangler** (981K) | rename/merge tags vault-wide | namespace/tag refactor across the union → UC-22 |
|
||||
| **Linter** (927K) | normalize frontmatter/formatting | content normalization → fingerprint basis for equivalence (UC-46) |
|
||||
| **Admonition** (922K) | callouts | Obsidian-flavored syntax → translation awareness (UC-42) |
|
||||
|
||||
Headline reads:
|
||||
- The **#1 plugin is drawings** (Excalidraw) — users keep **non-Markdown content** in
|
||||
"Markdown" vaults. A wiki orchestrator that assumes pure Markdown will mishandle the
|
||||
most popular real-world use → **UC-55**.
|
||||
- **Query-the-vault-as-a-DB** (Dataview #3 + Tasks #4) is a top use — but it is an
|
||||
*add-on*, confirming query is an **adapter/plugin capability** (UC-52), and motivating
|
||||
**query-defined dynamic pages** (UC-54).
|
||||
- **Git is top-7** — users manually bolt portable version control onto their vault. This
|
||||
is *direct demand* for what shard-wiki provides natively as the coordination journal
|
||||
(UC-36).
|
||||
- **Sync-to-anywhere** (Remotely Save #11) shows demand to connect a vault to S3/WebDAV/
|
||||
cloud — the heterogeneous backends INTENT targets — while reminding us of the
|
||||
*not-a-file-sync-daemon* boundary.
|
||||
|
||||
---
|
||||
|
||||
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 8.1 Reinforcements
|
||||
|
||||
- **File-over-app == shard sovereignty + Markdown-first.** Obsidian is the living proof
|
||||
that a serious tool can be "just files you own"; it is the model shard for INTENT's
|
||||
Obsidian/local-folder participants.
|
||||
- **Files canonical, index derived** validates shard-wiki's projection/derived-view
|
||||
architecture (§6).
|
||||
- **In-file, git-diffable structure & addressing** (frontmatter, `^block-id`, embeds)
|
||||
shows fine-grained addressing and structured data can be **portable text**, not DB
|
||||
state — friendlier to the coordination journal than Roam's DB.
|
||||
|
||||
### 8.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **A vault is one shard, not the federation.** Local-first, single-vault,
|
||||
single-user; do not model the union as "one big vault."
|
||||
2. **Not a file-sync daemon.** The Remotely Save popularity is tempting; shard-wiki must
|
||||
stay **wiki-page-semantic** — attach remote stores as shards, never generic file
|
||||
mirroring (INTENT constraint).
|
||||
3. **`.obsidian/` is opaque app config, not page content.** The adapter must exclude/
|
||||
treat it as shard-local config, not project it as pages.
|
||||
4. **Obsidian-flavored Markdown ≠ CommonMark.** Wikilinks/embeds/callouts/`^id` need
|
||||
adapter awareness; closer than Roam's outline, but still a translation surface
|
||||
(UC-42).
|
||||
5. **Plugins run arbitrary code.** If a shard-wiki adapter is hosted *as* an Obsidian
|
||||
plugin (UC-38 path), it inherits Electron/Node trust — a deployment/security note,
|
||||
not a core concern.
|
||||
|
||||
### 8.3 What Obsidian teaches that shard-wiki should keep
|
||||
|
||||
- Offer **both attachment modes** for a vault: zero-config **file-store direct attach**
|
||||
(read/projection/overlay) and an optional **in-app adapter** (write-through, live
|
||||
events). Graceful degradation by default, fidelity on opt-in.
|
||||
- **Consume a backend's existing derived index** (MetadataCache) instead of reparsing,
|
||||
when the shard exposes one.
|
||||
- **Embrace non-Markdown content types** (Canvas/attachments/drawings) as typed or
|
||||
opaque assets with provenance — don't flatten them away (UC-55, extends UC-34).
|
||||
|
||||
---
|
||||
|
||||
## 9. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-52**. New UCs **UC-53–UC-56** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a local-first Markdown vault as a file-backed shard with a live concurrent native editor** — file-watch, tolerate the vault's own app writing concurrently, treat `.obsidian/` as opaque | **UC-53 (new)** |
|
||||
| **Define a page as a live query over the union** (Dataview/Tasks pattern — query-defined dynamic page) | **UC-54 (new)** |
|
||||
| **Carry non-Markdown content types** (Canvas/JSON Canvas, drawings, attachments) as typed/opaque assets with provenance | **UC-55 (new)** |
|
||||
| **Publish a curated projection of the union or a shard to an external read-only target** (Obsidian Publish / Quartz / Digital Garden) | **UC-56 (new)** |
|
||||
| Local Markdown vault = cleanest file-backed direct-attach; dual attach (file-store *or* in-app plugin host) | **enriches UC-40** (and UC-02, UC-38) |
|
||||
| In-file `^block-id` / heading anchors = git-diffable, portable, opt-in span IDs | **enriches UC-51** |
|
||||
| YAML frontmatter/properties = git-diffable in-file structured data | **enriches UC-34** |
|
||||
| Git plugin (top-7) = users bolt on portable history → demand for the coordination journal | **enriches UC-36** |
|
||||
| Query is a popular plugin (Dataview), not core → query is adapter/plugin-provided | **enriches UC-52** |
|
||||
| Importer plugin = import from foreign tools | **enriches UC-28** |
|
||||
| Templater/QuickAdd = templated creation | **enriches UC-15** |
|
||||
| `![[...]]` embeds = in-file transclusion | links UC-32 |
|
||||
| MetadataCache backlinks/links | links UC-05/UC-18 |
|
||||
|
||||
---
|
||||
|
||||
## 10. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Dual attachment mode** for a single backend (file-store *and* in-engine adapter)
|
||||
should be first-class in the adapter contract / T14 binding — Obsidian is the clean
|
||||
example (Roam was in-app-only; TWiki had file-vs-API but not a personal vault).
|
||||
- **"Consume native derived index"** as a capability: a shard may expose its own link/
|
||||
backlink/block index (Obsidian MetadataCache, Roam Datalog) the orchestrator can read
|
||||
instead of reparsing (ties UC-52).
|
||||
- **Non-Markdown content types** (UC-55) push on the wiki page model: pages vs. typed
|
||||
assets vs. opaque blobs — a page-model spec decision, not just adapter config.
|
||||
- **Outbound publish** (UC-56) formalizes the `publish` capability from INTENT's list as
|
||||
a projection target, complementing inbound static-export attach (UC-37).
|
||||
- Concurrent-native-editor (UC-53) needs the contract to express **external-writer
|
||||
tolerance** (file-watching, re-projection, conflict-with-live-app) — distinct from
|
||||
multi-user write conflicts.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions (for spec / workplans)
|
||||
|
||||
1. For a vault attached **both** ways (file-store + in-app plugin), which is
|
||||
authoritative, and how do they reconcile (the in-app adapter sees MetadataCache; the
|
||||
file-store path sees raw bytes)?
|
||||
2. Is **UC-54 (query-defined dynamic page)** a core page type, an adapter feature, or a
|
||||
reference-UI/plugin concern? (Mirrors the Roam/Dataview "views are queries" question
|
||||
and catalog Q7.)
|
||||
3. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a
|
||||
Markdown stub, opaque blob with provenance, or a pluggable content-type registry?
|
||||
4. Does **outbound publish** (UC-56) belong in core or as a publish-adapter family, and
|
||||
how does it interact with overlays/projection freshness?
|
||||
5. How is **concurrent native editing** (UC-53) surfaced — optimistic re-projection,
|
||||
advisory lock, or overlay-onto-moving-target?
|
||||
|
||||
---
|
||||
|
||||
## 12. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Obsidian API — App Architecture (https://www.mintlify.com/obsidianmd/obsidian-api/concepts/app-architecture) | App singleton, four modules, this.app |
|
||||
| obsidianmd/obsidian-api README (https://github.com/obsidianmd/obsidian-api/blob/master/README.md) | Plugin lifecycle, Vault/MetadataCache/Workspace/FileManager APIs, registrations, manifest fields |
|
||||
| DeepWiki — Vault and File System; MetadataCache and Link Resolution (https://deepwiki.com/obsidianmd/obsidian-api) | Vault CRUD + events; MetadataCache parsed elements, resolved/unresolved links, backlinks |
|
||||
| obsidianstats.com — Most Downloaded Plugins (https://www.obsidianstats.com/most-downloaded) | All-time download ranks + counts for the popularity → UC mapping (§7) |
|
||||
| Obsidian — "The future of plugins" (https://obsidian.md/blog/future-of-plugins/) | Plugin distribution model, restricted mode context |
|
||||
| Obsidian Help / JSON Canvas (jsoncanvas.org) | Vault format, `.obsidian/` config, Canvas open format |
|
||||
|
||||
Cross-references: `research/260614-roam-deep-dive/findings.md` (DB/API contrast, native
|
||||
span IDs, query delegation), `research/260613-twiki-deep-dive/findings.md` (file-store
|
||||
attach UC-40), `spec/UseCaseCatalog.md` (UC-02, UC-05, UC-15, UC-18, UC-28, UC-31,
|
||||
UC-32, UC-34, UC-36, UC-38, UC-40, UC-51, UC-52), `workplans/SHARD-WP-0002-federation-architecture.md` (T14).
|
||||
|
||||
---
|
||||
|
||||
## 13. Traceability
|
||||
|
||||
- New UCs: **UC-53, UC-54, UC-55, UC-56** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-15, UC-28, UC-34, UC-36, UC-40, UC-51, UC-52** (and links UC-02,
|
||||
UC-05, UC-18, UC-32, UC-38).
|
||||
- Architecture (no UC): dual attachment mode; consume-native-derived-index capability;
|
||||
non-Markdown content types in the page model; outbound publish; external-writer
|
||||
tolerance → `SHARD-WP-0002` (T14).
|
||||
- Distinctive artifact: **plugin-popularity → UC mapping** (§7) — ecosystem demand
|
||||
evidence, per the research brief.
|
||||
- Boundary recorded: an Obsidian vault is **one file-backed candidate shard** (the
|
||||
cleanest, INTENT-named), mapped into the Markdown-first page model; not the federation
|
||||
layer and not a file-sync target (INTENT Stability Note + not-a-sync-daemon).
|
||||
</content>
|
||||
13
research/260614-oddmuse-deep-dive/README.md
Normal file
13
research/260614-oddmuse-deep-dive/README.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# 260614 — Oddmuse deep dive
|
||||
|
||||
Deep dive on **Oddmuse**: a deliberately minimal wiki — a **single Perl CGI script** over
|
||||
**plain-text page files** (one file per page, old revisions in a `keep/` dir), no database.
|
||||
The low-complexity **file-store floor** — useful as the **minimal-adapter / graceful-
|
||||
degradation baseline** of the capability profile.
|
||||
|
||||
- `findings.md` — the minimal model, storage layout, capability profile, INTENT mapping, UC
|
||||
seed (UC-82), architecture notes for SHARD-WP-0002, open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-82 (attach a **minimal flat-file wiki** — plain-text page files + a simple
|
||||
revision dir — as the **graceful-degradation baseline / minimal capability-profile floor**).
|
||||
Enriched UC-40/01/36/41. Feeds SHARD-WP-0002 T11 (minimal capability profile).
|
||||
124
research/260614-oddmuse-deep-dive/findings.md
Normal file
124
research/260614-oddmuse-deep-dive/findings.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# Oddmuse — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T7 · **Subject:** Oddmuse, Alex Schroeder's
|
||||
minimal single-script wiki (EmacsWiki, Community Wiki).
|
||||
|
||||
## Why this dive
|
||||
|
||||
After the structure/graph/SaaS far-ends, Oddmuse anchors the **opposite** corner: the
|
||||
*minimal* file-store wiki. It is the reference for shard-wiki's **graceful-degradation**
|
||||
promise — *a limited backend must still be usable* — and defines the **floor** of the
|
||||
capability profile (T11): what the simplest possible real wiki looks like as a shard.
|
||||
|
||||
## 1. The minimal model
|
||||
|
||||
- **One Perl CGI script** (`wiki.pl`) is the whole engine — drop it on any CGI host. No
|
||||
framework, no database, minimal dependencies.
|
||||
- **Plain-text page files**: each page is a text file in a **page directory** (`page/`),
|
||||
with a small header of metadata and the body; **old revisions** are kept in a **`keep/`**
|
||||
directory (recent history retained, older optionally expired).
|
||||
- **Locking** via lock files; edits append a new keep-revision.
|
||||
- **Markup:** simple wiki markup; **CamelCase** and **free links** `[[Like This]]`;
|
||||
InterWiki/near-links; tags and "clusters."
|
||||
- **No DB, no API** (beyond the CGI itself); content *is* the files on disk.
|
||||
|
||||
## 2. The shard view — the file-store floor
|
||||
|
||||
Because pages are **plain-text files on disk**, Oddmuse is **trivially attachable** as a
|
||||
**file-store shard** even though the engine offers nothing fancy:
|
||||
|
||||
- Read the `page/` files → pages (parse the tiny header + body).
|
||||
- Read `keep/` → recent revision history (import to the journal; note it may be **truncated**
|
||||
— older revisions can be expired, so history is *partial*).
|
||||
- Write = write a page file + a keep-revision (respecting the lock) — but the engine's own
|
||||
invariants (indexes) mean **write-through is best done via the engine or carefully**.
|
||||
|
||||
This is the **minimal capability profile**: file-store, page-granularity, plain-text,
|
||||
possibly-truncated history, no query, no structured fields, open editing. Everything richer
|
||||
in the synthesis matrix is measured *against this floor*.
|
||||
|
||||
## 3. Capability profile (the floor)
|
||||
|
||||
| Dimension (synthesis spectrum) | Oddmuse |
|
||||
|--------------------------------|---------|
|
||||
| Attachment mode | **file-store** (plain-text files); CGI, no API |
|
||||
| Addressing granularity | page = file |
|
||||
| Content identity | page name = filename |
|
||||
| Identity vs placement | name-bound |
|
||||
| Structure | none beyond tags/clusters; flat page space |
|
||||
| History | **`keep/` revisions — recent, possibly truncated** |
|
||||
| Merge model | lock-file; last-writer |
|
||||
| Native query | none |
|
||||
| Translation | simple wiki markup (→ Markdown translation needed) |
|
||||
| Write granularity | page (file) |
|
||||
| Operational envelope | a CGI script; tiny |
|
||||
| Access grant | open by default (optional password) |
|
||||
| Content opacity | transparent text |
|
||||
| Provenance | minimal (timestamp, optional username) |
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Graceful degradation** (INTENT): Oddmuse is the *definition* of the limited-backend case
|
||||
— still a perfectly good read/projection/overlay/backup shard via its files.
|
||||
- **Union without erasure**: even a minimal shard contributes pages with provenance; its
|
||||
**truncated history** must be surfaced honestly (don't imply full history when `keep/` is
|
||||
partial).
|
||||
- **Open wiki** (UC-01): Oddmuse is open-editing by default — the c2-era ethos.
|
||||
- **Markdown-first but backend-neutral**: its wiki markup needs translation to the
|
||||
Markdown-first page model (UC-42-style), a small lossy step.
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- **Partial history**: `keep/` may expire old revisions — the journal import must record that
|
||||
history is **truncated/partial**, not complete (a freshness/provenance honesty point).
|
||||
- **Minimal profile** means many capabilities are simply **absent** — the adapter advertises
|
||||
a sparse profile; the orchestrator must not assume query/structure/locking semantics it
|
||||
doesn't have (T11 capability-awareness in its purest form).
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Minimal flat-file wiki as the graceful-degradation baseline** (UC-82): plain-text files
|
||||
+ simple revision dir = the floor every richer profile extends.
|
||||
2. **Honest partial-history reporting** when a shard's revision store is truncated.
|
||||
3. **Sparse capability profile** handling — absence of a capability is first-class (T11).
|
||||
|
||||
## 5. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-82 | Attach a **minimal flat-file wiki** (plain-text page files + a simple revision dir, Oddmuse) as the **graceful-degradation baseline / minimal capability-profile floor**, surfacing **partial history** honestly | **new** |
|
||||
| — | plain-text file-store at the simple end | enrich **UC-40** |
|
||||
| — | open-editing wiki | enrich **UC-01** |
|
||||
| — | `keep/` plain-text revision history (possibly truncated) | enrich **UC-36** / **UC-41** |
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T11 (capability model):** Oddmuse defines the **minimal/floor profile** — file-store,
|
||||
page granularity, plain-text, **partial** history, no query/structure. Validate that the
|
||||
capability vocabulary can express **absence** cleanly and that the orchestrator degrades
|
||||
to read/projection/overlay/backup against it.
|
||||
- **History portability (T13):** `keep/` revisions import as journal entries but may be
|
||||
**truncated** — record completeness as metadata (full vs partial history).
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. How does shard-wiki represent a shard with **partial/truncated history** in the journal
|
||||
and provenance UI (UC-24) — explicit "history begins at" marker?
|
||||
2. Is write-through to a minimal CGI wiki (write page + keep-revision under its lock) ever
|
||||
sanctioned, or read/projection/overlay/backup only by default?
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- oddmuse.org — Oddmuse wiki (single-script install, `page/` + `keep/` storage, markup,
|
||||
CamelCase/free links, clusters/tags)
|
||||
- EmacsWiki / Community Wiki (Oddmuse in production)
|
||||
- prior: `research/260608-c2-wiki-origins/` (open-wiki ethos); `research/260613-twiki-deep-dive/`
|
||||
(file-store + RCS contrast)
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
New UC **UC-82** carries the marker **⊚** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-40, UC-01, UC-36, UC-41. Architecture cross-refs:
|
||||
SHARD-WP-0002 T11 (minimal/floor profile), T13 (partial-history import).
|
||||
30
research/260614-processing-deep-dive/README.md
Normal file
30
research/260614-processing-deep-dive/README.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# 260614 — Processing & Processing.js deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T4**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Processing** (creative coding) and **Processing.js / p5.js** (browser-run
|
||||
sketches): **the sketch *is* the document** — a program whose presentation is **live visual
|
||||
output rendered at view time** in the browser, with **no cached output** by default.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- The cleanest **program-as-page** case: canonical content = **source text**, presentation =
|
||||
**executable render** (no input/output cells, no prose envelope) — sharpens the page model
|
||||
(T12/T15) and UC-54/55.
|
||||
- Adds a **view-time** variant to derivation-projection (the render runs **in the viewer,
|
||||
continuously**) and a **continuity** facet (one-shot vs continuous/interactive); continuous
|
||||
→ static is a **snapshot frame/recording** on the live↔snapshot axis (T6).
|
||||
- "Execute/render in the viewer" = an explicit **capability + trust/sandbox** surface.
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (enrichment / design prior art). Enriches **UC-54, UC-55**; links UC-83,
|
||||
UC-84, UC-35.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Program-as-page, view-time/live render, no-cached-output, capability+trust, INTENT mapping, UC disposition (enrichment-only), architecture notes |
|
||||
114
research/260614-processing-deep-dive/findings.md
Normal file
114
research/260614-processing-deep-dive/findings.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# Processing & Processing.js — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T4 · **Subject:** Processing (creative
|
||||
coding) and Processing.js / p5.js — **the sketch *is* the document**, rendered live at view
|
||||
time in the browser.
|
||||
|
||||
## Why this dive
|
||||
|
||||
Low-priority, enrichment-focused. Processing tests a page-model edge the notebooks (T2/T3)
|
||||
didn't: a page that is **wholly a program whose presentation *is* its running output** — no
|
||||
separate input/output cells, no prose envelope. The defining question: what is a "page" when
|
||||
**the rendered form is a live computation evaluated at view time**? It feeds UC-54
|
||||
(computation-defined content) and UC-55 (non-Markdown / executable content) and sharpens the
|
||||
**live↔snapshot** axis named at T6.
|
||||
|
||||
## 1. Program-as-page
|
||||
|
||||
- A **Processing sketch** is a program (`setup()` + `draw()`) whose output is a **canvas of
|
||||
visual/animated/interactive content**. There are no "output cells" — **the program's
|
||||
execution is the content**.
|
||||
- **Processing.js** (Resig, 2008; now largely **p5.js**) runs sketches **client-side in the
|
||||
browser** on `<canvas>`. The page ships the **source**; the **rendering happens at view
|
||||
time** in the reader's browser — no server, no pre-rendered artifact required.
|
||||
- So the durable artifact is **text (the sketch source)**; the *presentation* is a **live,
|
||||
view-time derivation** of that source (a derivation-projection, T1) — with the twist that
|
||||
the derivation runs **in the viewer**, continuously (animation/interaction), not once.
|
||||
|
||||
## 2. The view-time / live-render dimension
|
||||
|
||||
This adds a wrinkle beyond Jupyter's *captured* outputs (UC-84) and Mathematica's CDF:
|
||||
|
||||
- **No captured output at all by default** — unlike a notebook, a sketch typically stores
|
||||
**only source**; there is nothing cached. The output exists only when executed.
|
||||
- **Continuous & interactive** — `draw()` loops; mouse/keyboard drive it. The output is
|
||||
**time-based and interaction-based**, so any static capture is a **single frame** (or a
|
||||
recording) — it cannot represent the artifact faithfully (shared limit with Mathematica
|
||||
`Dynamic` and, more extremely, Strudel T5).
|
||||
- **Client-side execution = a capability + trust surface** — running arbitrary sketch code in
|
||||
the viewer is an execution capability with sandboxing concerns; shard-wiki must treat
|
||||
"render live in the viewer" as an explicit, gated capability, never an automatic behavior.
|
||||
|
||||
## 3. INTENT mapping (enrichment-only — no new UC)
|
||||
|
||||
### Reinforcements / refinements
|
||||
|
||||
- **Executable-as-page (UC-54/55).** Processing is the cleanest **"the whole page is the
|
||||
program"** case: content = source text, presentation = view-time live render. Strengthens
|
||||
the page model's need to represent **executable content whose rendered form is derived at
|
||||
view time**, distinct from notebooks (which interleave cells + captured outputs).
|
||||
- **Derivation-projection, view-time variant (T1/UC-83).** The render is a derivation-
|
||||
projection that runs **in the viewer, continuously** — extends the projection model: a
|
||||
derivation may be *materialized ahead* (CDF/nbconvert) or *run at view time* (sketch).
|
||||
- **Live↔snapshot axis (T6).** With **no cached output** and **continuous/interactive**
|
||||
rendering, the only static form is a **snapshot frame or recording** — a concrete point on
|
||||
the live↔snapshot axis; honest treatment offers the snapshot and **marks it as a frame of a
|
||||
live artifact**, never as the artifact.
|
||||
- **Capability + trust gating.** "Execute in the viewer" is an explicit capability with a
|
||||
sandbox/trust boundary — mechanism-over-policy: whether to run, sandbox, or only snapshot
|
||||
stays configurable.
|
||||
|
||||
### Boundaries
|
||||
|
||||
- shard-wiki is **not a sketch runtime**; default is **attach the source + offer a captured
|
||||
snapshot/recording**; live in-viewer rendering is a gated capability. Source is canonical;
|
||||
the render is a degrading, view-time projection.
|
||||
|
||||
## 4. UC disposition (enrichment-only)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC / thread |
|
||||
|------------------------|---------------------|
|
||||
| Sketch = program-as-page; presentation = view-time live render (§1) | UC-54 (enriched: executable/view-time content), UC-55 (enriched: non-Markdown executable) |
|
||||
| Render = derivation-projection run **in the viewer, continuously** (§1, §2) | links UC-83 (view-time variant) |
|
||||
| No cached output; continuous/interactive → static = snapshot frame only (§2) | links UC-84, live↔snapshot axis (T6) |
|
||||
| Client-side execution = capability + trust/sandbox surface (§2) | links UC-35 (capability-awareness) |
|
||||
|
||||
**No new UC** — Processing is design prior art reinforcing executable-as-page; its
|
||||
contribution is the **view-time, no-cached-output, continuous-render** point on the
|
||||
projection/liveness model.
|
||||
|
||||
## 5. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12/T15 (page model):** represent **program-as-page** — a page whose canonical content is
|
||||
**source text** and whose presentation is an **executable render**; no inherent cached
|
||||
output (contrast notebooks). Non-Markdown executable content type.
|
||||
- **T16 (projection):** add a **materialization timing** facet to derivation-projection:
|
||||
**ahead-of-time** (CDF/nbconvert/static HTML) vs **at view time** (sketch render); and a
|
||||
**continuity** facet: one-shot vs continuous/interactive. Continuous → static is a snapshot
|
||||
frame/recording on the live↔snapshot axis (T6).
|
||||
- **T11 (capabilities):** "execute/render in the viewer" is a capability with a **trust/
|
||||
sandbox** sub-concern; default off → snapshot.
|
||||
|
||||
## 6. Open questions
|
||||
|
||||
1. Is **view-time execution** ever offered (sandboxed in-viewer render), or does shard-wiki
|
||||
always degrade program-as-page to a captured snapshot/recording? (Capability/trust policy.)
|
||||
2. Should **materialization timing** (ahead-of-time vs view-time) and **continuity**
|
||||
(one-shot vs continuous) be explicit projection metadata, alongside the live↔snapshot
|
||||
axis? (Recurs at T5.)
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- `processing.org`; **p5.js** (`p5js.org`); Processing.js (Resig) history; `<canvas>` /
|
||||
client-side rendering model.
|
||||
- prior: `research/260614-jupyter-deep-dive/` (captured vs no-cached output, UC-84);
|
||||
`research/260614-mathematica-deep-dive/` (`Dynamic`/CDF, snapshot-only);
|
||||
`research/260614-squeak-pharo-deep-dive/` (live↔snapshot axis).
|
||||
|
||||
## 8. Traceability
|
||||
|
||||
**No new UC** (enrichment / design prior art). Enriched: UC-54, UC-55; links UC-83 (view-time
|
||||
derivation-projection), UC-84 (no-cached-output contrast), UC-35 (execute capability +
|
||||
trust). Architecture cross-refs: SHARD-WP-0002 T12/T15 (program-as-page, source-canonical/
|
||||
render-derived), T16 (materialization-timing + continuity facets on the live↔snapshot axis),
|
||||
T11 (view-time-execute capability + sandbox).
|
||||
15
research/260614-quip-deep-dive/README.md
Normal file
15
research/260614-quip-deep-dive/README.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# 260614 — Salesforce Quip deep dive
|
||||
|
||||
Deep dive on **Salesforce Quip**: a closed SaaS of **live collaborative documents** with
|
||||
**embedded spreadsheets and live apps**, a REST API (HTML import/export), and
|
||||
**Salesforce-tied identity/permissions**. A hosted-collab contrast to Notion: the
|
||||
document+spreadsheet hybrid under enterprise auth.
|
||||
|
||||
- `findings.md` — the doc/spreadsheet model, REST API, enterprise auth, capability profile,
|
||||
INTENT mapping, UC seed (UC-80), architecture notes for SHARD-WP-0002, open questions,
|
||||
sources, traceability.
|
||||
|
||||
Catalog yield: UC-80 (attach a SaaS live-doc shard whose pages **mix prose + embedded live
|
||||
structured objects** — spreadsheets/apps — via REST with lossy HTML import/export, under
|
||||
enterprise identity). Enriched UC-57/55/58/36/06. Feeds SHARD-WP-0002 T11 (capability /
|
||||
content opacity), T14 (external-API binding).
|
||||
148
research/260614-quip-deep-dive/findings.md
Normal file
148
research/260614-quip-deep-dive/findings.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Salesforce Quip — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T6 · **Subject:** Quip (Salesforce), a
|
||||
collaborative-document SaaS.
|
||||
|
||||
## Why this dive
|
||||
|
||||
Notion (UC-57/58/59) gave us the closed-SaaS, external-REST-only, database-as-pages model.
|
||||
Quip is the **enterprise document-collab** contrast: a **document+spreadsheet hybrid** where
|
||||
a page is **prose interleaved with embedded live structured objects** (spreadsheets,
|
||||
calendars, kanban "live apps"), reachable only through a REST API, gated by **Salesforce
|
||||
identity**. The question: how does shard-wiki attach a shard whose "page" is a *mixed
|
||||
prose+live-object document* behind enterprise auth?
|
||||
|
||||
## 1. The document + live-object model
|
||||
|
||||
- A Quip **document** is a real-time collaborative doc (concurrent editing). Its body is
|
||||
rich content — headings, lists, prose — **interleaved with embedded objects**:
|
||||
- **Spreadsheets** are *first-class, inline* — a doc can contain live spreadsheet sections
|
||||
with formulas, not just static tables.
|
||||
- **Live apps** (calendars, kanban boards, project trackers, polls) embed interactive
|
||||
structured widgets inside the document.
|
||||
- **Folders** organize documents; **threads** (docs and chat are unified — every doc is also
|
||||
a conversation thread) carry **messages/comments** inline.
|
||||
- So a Quip "page" is a **hybrid**: prose + embedded structured/live content in one
|
||||
document, with conversation attached. Not Markdown; a proprietary rich model.
|
||||
|
||||
## 2. The REST API (the only door)
|
||||
|
||||
Quip exposes a **REST API**: threads/documents (get, create, **edit-document** with an HTML
|
||||
fragment + a location/section anchor), folders, messages, users, and spreadsheet cell
|
||||
access. Content crosses the API as **HTML** (import and export) — there is **no native
|
||||
Markdown** and no file/git access. Implications:
|
||||
|
||||
- **Import/export is HTML → lossy** to Markdown (like Notion's export, UC-59): embedded
|
||||
spreadsheets and live apps **do not round-trip** to Markdown cleanly — they degrade to
|
||||
tables or links.
|
||||
- Editing is **section/anchor-scoped HTML splice** (`edit-document` targets a location) — a
|
||||
mid-granularity write (a section, not the whole doc, not a character).
|
||||
- **Rate-limited** like any SaaS API; history is internal (revisions exist in-product, with
|
||||
limited API exposure).
|
||||
|
||||
## 3. Enterprise identity / permissions
|
||||
|
||||
Quip is tied to **Salesforce** (acquired 2016): authentication and access run through the
|
||||
Salesforce org / Quip site; **folder and document sharing ACLs** govern visibility, with
|
||||
enterprise SSO. So this is an **enterprise-ACL, authn-delegated** shard (UC-06) — shard-wiki
|
||||
honors Salesforce-side permissions and never bypasses them.
|
||||
|
||||
## 4. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | Quip |
|
||||
|--------------------------------|------|
|
||||
| Attachment mode | **external-API** (REST; HTML import/export) — closed SaaS |
|
||||
| Addressing granularity | document; **section/anchor** for edits; spreadsheet cell |
|
||||
| Content identity | Quip thread/document ID (opaque) |
|
||||
| Identity vs placement | API-id identity; folder placement separate |
|
||||
| Structure | **hybrid doc**: prose + embedded spreadsheets + live apps + thread |
|
||||
| History | **internal** revisions (limited API exposure) |
|
||||
| Merge model | server-side real-time collab (OT-like); no external merge |
|
||||
| Native query | none exposed (spreadsheet formulas internal) |
|
||||
| Translation | **HTML** in/out → **lossy** to Markdown (spreadsheets/apps degrade) |
|
||||
| Write granularity | **section/anchor HTML splice** (mid) |
|
||||
| Operational envelope | rate-limited SaaS REST |
|
||||
| Access grant | **Salesforce identity + folder/doc ACL** (enterprise SSO) |
|
||||
| Content opacity | proprietary rich model; not E2EE but not transparent files |
|
||||
| Provenance | author/edit metadata in-product; API-limited |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **External-API attachment** (UC-57): Quip is a second concrete instance beside Notion of
|
||||
the closed-SaaS REST-only shard — generalizes the external-API mode (REST + HTML payload,
|
||||
vs Notion's REST + block-JSON, vs Wiki.js GraphQL).
|
||||
- **Union without erasure / no silent flatten**: embedded spreadsheets and live apps must be
|
||||
**surfaced as what they are** with provenance, and the **HTML→Markdown lossiness made
|
||||
explicit** (UC-59) — never silently drop a live spreadsheet to a static table.
|
||||
- **Authz-in-core, authn-delegated** (settled decision): Quip's Salesforce-tied ACL is the
|
||||
enterprise case — honor delegated identity and the shard's ACL (UC-06).
|
||||
- **Graceful degradation**: with only a lossy HTML export, Quip is still usable as a
|
||||
read/projection/overlay-target/backup shard.
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- **Mixed prose+live-object page** is a content shape beyond "Markdown body + frontmatter":
|
||||
the page model must allow **embedded typed/live objects within a prose page** (not just a
|
||||
whole-page-is-a-record like Notion DB, but *inline* structured content) — feeds T12 and the
|
||||
non-Markdown-content question (UC-55).
|
||||
- **HTML is the only interchange** — no Markdown, no files, no git. Content opacity is
|
||||
"proprietary-but-exportable": transparent-ish via lossy HTML, not via files (T11
|
||||
content-opacity tier between transparent-files and E2EE-opaque).
|
||||
- **Write granularity = section-anchor HTML splice** — a mid tier (between whole-file and
|
||||
block/character) realized over an API, distinct from fedwiki's story-item op-log.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **External-API mode generalized** to carry an **HTML payload** variant (Quip) beside
|
||||
block-JSON (Notion) and GraphQL (Wiki.js) — capability/payload-format is part of the
|
||||
adapter profile (UC-57/UC-80).
|
||||
2. **Inline embedded live/structured objects** as a page-model element (prose + embedded
|
||||
spreadsheet/app), with **explicit lossy projection** to Markdown (UC-55/UC-58/UC-59).
|
||||
3. **Enterprise-ACL + delegated identity** honored, not bypassed (UC-06).
|
||||
|
||||
## 6. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-80 | Attach a **SaaS live-doc shard** whose pages **mix prose + embedded live structured objects** (spreadsheets / live apps) via **REST with lossy HTML import/export**, under **enterprise (Salesforce) identity** | **new** |
|
||||
| — | external-API mode w/ HTML payload variant | enrich **UC-57** |
|
||||
| — | inline embedded spreadsheet/live-app = non-Markdown content | enrich **UC-55** |
|
||||
| — | embedded structured objects within a prose page | enrich **UC-58** |
|
||||
| — | internal (API-limited) revision history | enrich **UC-36** |
|
||||
| — | Salesforce-tied enterprise ACL + SSO | enrich **UC-06** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T11 (capability / content opacity):** add a **payload-format** facet to external-API
|
||||
shards (HTML / block-JSON / GraphQL) and a **content-opacity tier** "proprietary but
|
||||
lossy-exportable" between transparent-files and E2EE-opaque. Write granularity =
|
||||
section-anchor splice.
|
||||
- **T12 (page model):** support **inline embedded structured/live objects** within a prose
|
||||
page (not only whole-page-as-record) — with explicit lossy render to Markdown.
|
||||
- **T14 (binding):** external-API binding with **HTML import/export**; honor Salesforce
|
||||
identity/ACL; default to read/projection/overlay given rate limits + lossy export.
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Does shard-wiki represent an **embedded live spreadsheet** as a typed sub-object with
|
||||
provenance (preferred) or flatten it to a static Markdown table (lossy) — and can overlays
|
||||
target a spreadsheet cell via the API, or only the prose?
|
||||
2. Given **HTML-only, lossy** interchange and rate limits, is Quip ever a **write-through**
|
||||
shard, or read/projection/overlay/backup by default? (cf. Notion Q1, UC-57.)
|
||||
3. How is **Salesforce identity** mapped to shard-wiki's delegated-authn model — pass-through
|
||||
token, service account, or per-user OAuth?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- Quip Automation API (REST) docs — quip.com / Salesforce developer docs (threads/documents,
|
||||
`edit-document`, folders, messages, spreadsheets)
|
||||
- Salesforce Quip product docs (live apps, spreadsheets, sharing/permissions)
|
||||
- prior: `research/260614-notion-deep-dive/` (closed-SaaS external-API contrast, UC-57/59)
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UC **UC-80** carries the marker **◧** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-57, UC-55, UC-58, UC-36, UC-06. Architecture
|
||||
cross-refs: SHARD-WP-0002 T11 (payload-format + content-opacity), T12 (inline objects), T14.
|
||||
49
research/260614-roam-deep-dive/README.md
Normal file
49
research/260614-roam-deep-dive/README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# 260614 — Roam Research deep dive (the block graph as a queryable database + extension API)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Roam Research** — core architecture and extension structure —
|
||||
read through shard-wiki's lens. Roam is the **modern bookend** to the two Nelson dives:
|
||||
where Xanadu (`research/260614-xanadu-deep-dive/`) and ZigZag
|
||||
(`research/260614-zigzag-deep-dive/`) are *unbuilt ideals*, Roam **shipped** stable
|
||||
fine-grained addressing, transclusion, bidirectional links, and a queryable structured
|
||||
space to a mainstream audience.
|
||||
|
||||
Distinctive material:
|
||||
- **Data model** — the entire graph is a client-side **DataScript** (Datomic-like) DB
|
||||
of **datoms** (EAV); every paragraph and page is a **block** entity with a stable
|
||||
nine-character **`:block/uid`**, `:block/string`, `:block/refs`, `:block/page`,
|
||||
pages marked by `:node/title`
|
||||
- **References & transclusion** — `[[Page]]`/`#tag`/`((uid))` create `:block/refs`
|
||||
edges; **block embeds** are working transclusion; linked vs unlinked references
|
||||
- **Querying** — Datalog (`:q`/`:pull`); derived views are queries over the graph
|
||||
- **Extension architecture** — **Roam Depot** (`extension.js` `onload`/`onunload`,
|
||||
manifest, `extensionAPI` settings) over **`window.roamAlphaAPI`** read (`q`/`pull`) +
|
||||
write (`block.create/update/move/delete`, `page.create`) — a concrete
|
||||
*engine-hosts-adapter* (UC-38) surface
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Data model, references/transclusion, Datalog, extension API, Roam-as-shard capability profile, evidence on shard-wiki's open questions, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-50 attach a block-graph DB shard with block↔page mapping, UC-51 adopt native
|
||||
block/span IDs as portable span addresses, UC-52 delegate derived views to a shard's
|
||||
native query engine); UC-32/34/35/38 enriched. Logged for
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T14): native-span-ID and
|
||||
native-query capabilities, block↔page mapping, and Roam as a second DB-backed /
|
||||
engine-hosts-adapter exemplar alongside XWiki.
|
||||
|
||||
**Key payoff recorded:** Roam empirically answers the Nelson dives' open questions —
|
||||
fine-grained addressing is tractable (block UIDs), transclusion is a data-layer
|
||||
capability over an addressable union, and derived views are queries over a structured
|
||||
space. **Boundary:** Roam is *one candidate shard* (DB-backed/API-attached, block-first,
|
||||
no portable Git history), mapped into shard-wiki's Markdown-first page model — not a
|
||||
substrate and not the federation layer.
|
||||
</content>
|
||||
292
research/260614-roam-deep-dive/findings.md
Normal file
292
research/260614-roam-deep-dive/findings.md
Normal file
@@ -0,0 +1,292 @@
|
||||
# Findings — Roam Research: the block graph as a queryable database, and its extension API
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — a block-graph note tool; a *candidate shard
|
||||
backend* (DB-backed / API-attached, like XWiki), and the place where several Nelson
|
||||
ideas actually shipped
|
||||
Lens: shard-wiki — fine-grained addressing, transclusion, bidirectional links, a
|
||||
structured/queryable page space, and the engine-hosts-adapter extension path
|
||||
|
||||
> Why Roam, and why now. The previous two dives were Ted Nelson's *unbuilt* ideals —
|
||||
> Xanadu (`research/260614-xanadu-deep-dive/`: reference-not-copy, transclusion,
|
||||
> stable fine-grained addresses) and ZigZag (`research/260614-zigzag-deep-dive/`: an
|
||||
> information space as many co-equal dimensions over a queryable structure). **Roam
|
||||
> Research is where a large slice of both actually shipped to a mainstream audience.**
|
||||
> Every block has a stable address; block references and embeds are working
|
||||
> transclusion; the whole graph is a queryable database; links are bidirectional. So
|
||||
> Roam is the natural modern bookend: it lets us check shard-wiki's open questions
|
||||
> (portable span addressing, transclusion, structured pages) against a system that
|
||||
> solved them *in production* — and study its **extension architecture** as a concrete
|
||||
> adapter-host surface.
|
||||
|
||||
This dive treats Roam as a *candidate shard* (capability profile in §6) and as
|
||||
*evidence* on the open questions the Nelson dives left (§7), not as something to copy
|
||||
wholesale — Roam is block-first and DB-backed, shard-wiki is Markdown-page-first and
|
||||
Git-backed (divergences in §8.2).
|
||||
|
||||
---
|
||||
|
||||
## 1. What Roam is, in architecture terms
|
||||
|
||||
Roam Research is a networked-thought outliner whose **entire graph is a client-side
|
||||
DataScript database** (DataScript = a ClojureScript reimplementation of Datomic's
|
||||
data model and Datalog query engine). The store is not files; it is a set of immutable
|
||||
**datoms**.
|
||||
|
||||
- A **datom** is one fact: `[entity, attribute, value, transaction]` — the EAV(T)
|
||||
model. The graph is the accumulation of datoms; edits are transactions.
|
||||
- Synchronization and Roam's deep undo fall out of the transaction log — a parallel
|
||||
worth noting to Git's immutable commit history (but it is *not* Git; see §8.2).
|
||||
|
||||
This makes Roam a **structured, queryable** wiki-shaped store — the same family as
|
||||
XWiki (objects + DB) rather than TWiki/Foswiki (flat files), which matters for how an
|
||||
adapter attaches (§6).
|
||||
|
||||
---
|
||||
|
||||
## 2. The block model — everything is a block, every block is addressable
|
||||
|
||||
Every paragraph *and* every page is a **block** (an entity). Key attributes:
|
||||
|
||||
| Attribute | Meaning |
|
||||
|-----------|---------|
|
||||
| `:block/uid` | **nine-character public ID** — the block's stable, referenceable address (e.g. `((GGv3cyL6Y))`) |
|
||||
| `:block/string` | the block's text content |
|
||||
| `:block/order` | position among siblings |
|
||||
| `:block/children` | immediate child entity-ids |
|
||||
| `:block/parents` | full ancestor chain (incl. the page) |
|
||||
| `:block/refs` | edges to blocks/pages this block references |
|
||||
| `:block/page` | the owning page entity |
|
||||
| `:create/time`, `:edit/time`, `:create/email`, `:edit/email` | provenance metadata |
|
||||
| `:node/title` | **page-only** — the page's title (this attribute is what distinguishes a page from an ordinary block) |
|
||||
|
||||
Structurally the DB is a **forest**: each page is a tree of nested blocks; a "page" is
|
||||
just the top-level block that carries `:node/title`. The single most shard-wiki-relevant
|
||||
fact: **`:block/uid` gives every block — i.e. every sub-page span — a stable,
|
||||
first-class, public address.** This is the *shipped* form of the Xanadu tumbler / the
|
||||
shard-wiki "portable span address" open question (§7).
|
||||
|
||||
---
|
||||
|
||||
## 3. References, transclusion, bidirectional links
|
||||
|
||||
- **Page references** `[[Page]]` and `#tags`, and **block references** `((uid))`, all
|
||||
create `:block/refs` edges. The link graph is therefore queryable, not parsed-on-read.
|
||||
- **Block embeds** (`{{embed: ((uid))}}`) are **working transclusion**: a block's
|
||||
content rendered live in another location by reference, not copied — when the source
|
||||
changes, the embed reflects it. This is Xanadu transclusion / shard-wiki UC-32/UC-44,
|
||||
shipped at block granularity.
|
||||
- **Bidirectional links** come in two flavours, exactly shard-wiki's BackLinks problem:
|
||||
- **Linked references** — explicit `:block/refs` edges (queryable backlinks).
|
||||
- **Unlinked references** — bare text mentions with no edge, surfaced by text search.
|
||||
- **Attributes** `key:: value` are a convention on top of blocks/refs that turns a page
|
||||
into a lightweight record — Roam's answer to structured/typed pages (UC-34/UC-39),
|
||||
queryable via Datalog.
|
||||
|
||||
---
|
||||
|
||||
## 4. Querying — Datalog over the graph
|
||||
|
||||
Roam exposes **Datalog** (`:find` / `:where`) plus **pull** expressions:
|
||||
|
||||
```
|
||||
[:find ?uid :where [?b :block/string ?s] [?b :block/uid ?uid] ...]
|
||||
```
|
||||
|
||||
This means Roam's "derived views" (a page's linked references, a `{{query}}` block, a
|
||||
table) are **just queries over the datom graph** — not bespoke features. For shard-wiki
|
||||
this is the strongest evidence yet that *derived views = queries over a structured
|
||||
space* (the ZigZag "dimensions + rasters" insight, made executable), and that an
|
||||
adapter could **delegate view computation to a shard's native query engine** instead of
|
||||
scanning (UC-52).
|
||||
|
||||
---
|
||||
|
||||
## 5. Extension architecture — Roam Depot and `window.roamAlphaAPI`
|
||||
|
||||
Roam's official extension system (**Roam Depot**) is a concrete model of the
|
||||
*engine-hosts-adapter* path (UC-38):
|
||||
|
||||
- **Distribution:** an extension is a GitHub repo (`README.md`, `extension.js`, optional
|
||||
`extension.css` / `CHANGELOG.md` / `build.sh`), cataloged in the `roam-depot` repo via
|
||||
a metadata JSON (`name`, `author`, `tags`, `source_repo`, `source_commit`, optional
|
||||
`stripe_account` for paid extensions).
|
||||
- **Lifecycle:** `extension.js` default-exports a map with **`onload`** and
|
||||
**`onunload`** functions; everything created on load must be torn down on unload. An
|
||||
`extensionAPI` object provides a **settings panel** and `settings.get/set`.
|
||||
- **Data API — `window.roamAlphaAPI`** (the surface an adapter would target):
|
||||
- **Read:** `q` (Datalog query → entity ids), `pull` / `pull-many` (fetch entity
|
||||
attributes), full graph traversal.
|
||||
- **Write:** `data.block.create` / `update` / `move` / `delete`, `data.page.create`.
|
||||
- **UI:** command palette, blocks, settings.
|
||||
|
||||
Crucially, this API runs **inside the Roam client** (global `window.roamAlphaAPI`).
|
||||
There is no first-class external REST write API; programmatic external access uses
|
||||
graph **export** (EDN / JSON / Markdown) or unofficial private APIs. So a *live,
|
||||
write-through* Roam adapter must be **hosted inside Roam as a Depot extension** — which
|
||||
is exactly the UC-38 engine-side direction (cf. XWiki components, TWiki plugin handlers).
|
||||
|
||||
---
|
||||
|
||||
## 6. Roam as a shard — capability profile
|
||||
|
||||
| Capability | Roam | Notes for the adapter contract |
|
||||
|------------|------|--------------------------------|
|
||||
| Read | **yes** | `q`/`pull` live (in-app) or EDN/JSON/Markdown export (snapshot) |
|
||||
| Write | **yes, in-app** | `block.create/update/move/delete`, `page.create` — but only via the in-client API → needs a Depot-hosted adapter (UC-38) or unofficial private API |
|
||||
| Write granularity | **block-level (fine)** | the opposite extreme from TiddlyWiki's whole-file writes — sharpens UC-35 |
|
||||
| Identity / addressing | **block `:block/uid` + page `:node/title`** | shipped sub-page stable addressing (UC-51) |
|
||||
| Transclusion | **yes** | block embeds by uid (UC-32/44/45) |
|
||||
| Backlinks | **yes** | linked (`:block/refs`) + unlinked (text) (UC-05/18) |
|
||||
| Structured data | **yes** | attributes + Datalog (UC-34/39) |
|
||||
| Native query | **yes (Datalog)** | adapter could delegate views (UC-52) |
|
||||
| Diff / merge | **no native** | transaction log exists but not exposed as page diffs |
|
||||
| Version history | **hosted, internal-only** | not portable Git history → needs supplement/import (UC-36, like Confluence/MediaWiki) |
|
||||
| Lock | **no** | — |
|
||||
| Syntax | **Roam-flavored Markdown** | not CommonMark; block-outline structure → translation/mapping needed (cf. UC-42) |
|
||||
|
||||
Verdict: Roam is a legitimate **DB-backed, API-attached shard** — the XWiki family, not
|
||||
the file-store family. Its standout offerings are **block-level addressing** and
|
||||
**shipped transclusion**; its costs are **block↔page impedance** and **no portable
|
||||
history**.
|
||||
|
||||
---
|
||||
|
||||
## 7. Evidence on shard-wiki's open questions (the payoff)
|
||||
|
||||
The Nelson dives left open questions; Roam answers several *empirically*:
|
||||
|
||||
1. **"What is a portable fine-grained span address?"** (Xanadu §11 Q1, the tumbler
|
||||
problem). Roam's answer: **a short opaque per-block UID, minted by the store, public
|
||||
and referenceable.** shard-wiki lesson — fine-grained addressing is *tractable* when
|
||||
the backend mints stable block/span IDs; the adapter should **adopt native block IDs
|
||||
as span addresses where they exist** (UC-51), and fall back to content-fingerprint or
|
||||
path+range where they do not (cf. Xanadu content-identity, UC-46).
|
||||
2. **"Does transclusion belong in core or adapter/UI?"** (catalog Q6). Roam shows
|
||||
transclusion is cheap and natural *when the store is addressable and queryable*; it
|
||||
lives at the **data layer** (refs/embeds), surfaced by UI. Argues for transclusion as
|
||||
a **core capability over an addressable union**, not a UI-only trick.
|
||||
3. **"Derived views — core or adapter?"** Roam shows them as **queries over a
|
||||
structured space**. Where a shard exposes a native query engine, **delegate**
|
||||
(UC-52); where it does not, the orchestrator computes them over the projection.
|
||||
4. **Block↔page mapping** is the new question Roam *raises*: in a block graph, what is a
|
||||
"page"? Roam's own answer (`:node/title` node = page; nested blocks = spans) is a
|
||||
clean mapping rule for the adapter (UC-50).
|
||||
|
||||
---
|
||||
|
||||
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 8.1 Reinforcements
|
||||
|
||||
- **Fine-grained addressing is achievable** (block UIDs) — de-risks UC-44/45/51 and the
|
||||
Xanadu tumbler worry.
|
||||
- **Transclusion + bidirectional links + structured data** are not exotic; a shipped
|
||||
tool does all three over one addressable, queryable store — validating shard-wiki's
|
||||
ambitions for UC-32/34/39/05.
|
||||
- **Engine-hosts-adapter (UC-38)** has a clean modern template: Depot's
|
||||
`onload/onunload` + `roamAlphaAPI` read/write surface.
|
||||
|
||||
### 8.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Block-DB substrate vs. Markdown-file page model.** Roam's "everything is a block in
|
||||
a DataScript DB" must **not** tempt shard-wiki away from its Markdown-first,
|
||||
backend-neutral page model (INTENT Stability Note). Roam is *one shard shape*, mapped
|
||||
*into* the page model, not the model itself.
|
||||
2. **Client-side proprietary DB vs. Git coordination journal.** Roam's transaction log
|
||||
is not portable history; shard-wiki keeps Git as the coordination layer. A Roam shard
|
||||
contributes snapshots/projections, and its history needs **supplement/import**
|
||||
(UC-36), not adoption as canonical.
|
||||
3. **Single graph vs. federation.** A Roam graph is **one sovereign shard**, never the
|
||||
federation layer. Do not model the union as "a big Roam graph."
|
||||
4. **Roam-flavored Markdown + outline structure** ≠ CommonMark pages; the adapter owns a
|
||||
**lossy-aware translation** (block outline ↔ page + headings/lists), cf. UC-42.
|
||||
|
||||
### 8.3 What Roam teaches that shard-wiki should keep
|
||||
|
||||
- **Mint/adopt stable sub-page IDs.** The cheapest path to transclusion, overlay, and
|
||||
reverse-lookup at span granularity is a backend that already addresses spans — lean on
|
||||
it (UC-51), degrade gracefully otherwise.
|
||||
- **Treat a structured shard's query engine as a capability**, and delegate derived
|
||||
views to it when present (UC-52).
|
||||
- **A clean block↔page rule** (`:node/title` node = page) keeps a block backend usable
|
||||
without flattening (UC-50, complementing UC-34's no-lossy-flatten rule).
|
||||
|
||||
---
|
||||
|
||||
## 9. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-49**. New UCs **UC-50–UC-52** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a block-graph DB wiki (Roam-style) as a shard via its query/CRUD API, mapping blocks to the page model** (`:node/title` node = page; nested blocks = spans) | **UC-50 (new)** |
|
||||
| **Adopt a shard's native block/span IDs as portable span addresses** for transclusion/overlay (the shipped answer to the span-address question) | **UC-51 (new)** |
|
||||
| **Delegate derived views to a shard's native query engine** (Datalog) where advertised | **UC-52 (new)** |
|
||||
| Shipped transclusion via block embeds + shipped fine-grained addressing | **enriches UC-32** (and UC-44/45) |
|
||||
| Attributes + Datalog = shipped structured/typed pages | **enriches UC-34** (and UC-39) |
|
||||
| Roam Depot `onload/onunload` + `roamAlphaAPI` = modern engine-hosts-adapter template | **enriches UC-38** |
|
||||
| Block-level write granularity = the fine extreme opposite TiddlyWiki | **enriches UC-35** |
|
||||
| Hosted history is internal-only, not portable Git | links **UC-36** |
|
||||
|
||||
---
|
||||
|
||||
## 10. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- The adapter contract should model **"native span/block IDs"** as a capability: a shard
|
||||
advertises whether it mints stable sub-page addresses; transclusion/overlay/reverse-
|
||||
lookup capabilities key off it (ties UC-44/45/46/51).
|
||||
- The contract should model a **"native query"** capability so the orchestrator can
|
||||
delegate derived-view computation (UC-52) vs. scanning the projection.
|
||||
- A **block↔page mapping** belongs in the adapter, with the no-lossy-flatten rule of
|
||||
UC-34: page = titled node, blocks = addressable spans, attributes = sidecar metadata.
|
||||
- Roam joins XWiki as a **DB-backed / API-attached** exemplar (vs. TWiki/Foswiki file
|
||||
stores) and as a **second engine-hosts-adapter** template (vs. XWiki components / TWiki
|
||||
handlers) for the T14 adapter-binding task.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions (for spec / workplans)
|
||||
|
||||
1. When a backend mints native span IDs (Roam UIDs), are they used **directly** as the
|
||||
shard-wiki span address, or **wrapped** in a shard-scoped address so they survive
|
||||
projection/overlay and don't collide across shards?
|
||||
2. For a **write-through** Roam shard the adapter must run *inside* Roam (Depot
|
||||
extension). Is in-engine hosting an accepted adapter deployment mode generally, or do
|
||||
we restrict Roam to **read/projection/overlay-target** (graceful degradation) when we
|
||||
cannot deploy inside it?
|
||||
3. How lossy is the **block-outline ↔ Markdown-page** mapping, and is round-trip
|
||||
fidelity required or best-effort (cf. UC-42 Foswiki TML↔HTML)?
|
||||
4. Do we consume Roam's **export (EDN/JSON)** as the projection source, or the live
|
||||
`roamAlphaAPI`? Snapshot vs. live freshness trade-off (cf. UC-31).
|
||||
|
||||
---
|
||||
|
||||
## 12. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| zsolt.blog — "Deep Dive Into Roam's Data Structure" (https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html) | Datom/EAV model, block attributes (`:block/uid` etc.), refs/embeds, linked vs unlinked references, Datalog/pull |
|
||||
| Roam Research Datalog Cheatsheet (https://gist.github.com/2b3pro/231e4f230ed41e3f52e8a89ebf49848b) | Attribute names, `:q`/`:pull` query shapes |
|
||||
| GitHub — Roam-Research/roam-depot (https://github.com/Roam-Research/roam-depot) | Extension repo/manifest structure, `extension.js` `onload`/`onunload`, `extensionAPI` settings, `roamAlphaAPI` read/write/ui surface |
|
||||
| "Introduction to the Roam Alpha API" (https://www.putyourleftfoot.in/introduction-to-the-roam-alpha-api) | `window.roamAlphaAPI` `q`/`pull` semantics, global-scope dependency |
|
||||
| David Vargas — "How To Create Your Own Roam Extensions" (https://davidvargas.me/blog/how_to_create_your_own_roam_extensions) | Extension authoring lifecycle, write methods |
|
||||
| Wikipedia / general — Roam Research, DataScript, Datomic | Datalog/Datomic lineage, client-side DB framing |
|
||||
|
||||
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (span addressing,
|
||||
transclusion), `research/260614-zigzag-deep-dive/findings.md` (queryable dimensions),
|
||||
`research/260613-xwiki-deep-dive/findings.md` (DB-backed engine + extension host),
|
||||
`spec/UseCaseCatalog.md` (UC-05, UC-18, UC-32, UC-34/35/36/38/39, UC-44/45/46),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T14 adapter binding).
|
||||
|
||||
---
|
||||
|
||||
## 13. Traceability
|
||||
|
||||
- New UCs: **UC-50, UC-51, UC-52** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-32, UC-34, UC-35, UC-38** (and links UC-36, UC-39, UC-44/45/46).
|
||||
- Architecture (no UC): native-span-ID capability; native-query capability; block↔page
|
||||
mapping; Roam as DB-backed/API-attached + engine-hosts-adapter exemplar → `SHARD-WP-0002` (T14).
|
||||
- Boundary recorded: Roam is **one candidate shard**, mapped into the Markdown-first
|
||||
page model; not a substrate, not the federation layer (INTENT Stability Note).
|
||||
</content>
|
||||
69
research/260614-shard-spectrum-synthesis/README.md
Normal file
69
research/260614-shard-spectrum-synthesis/README.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# 260614 — Shard spectrum synthesis (one capability model across the full dive set)
|
||||
|
||||
Date: 2026-06-14 · **revised 2026-06-14 (v3)**
|
||||
|
||||
## What this is
|
||||
|
||||
A **synthesis** (no new external research) that reads every studied system *across* the
|
||||
others and distills them into a single comparative model feeding the **shard adapter
|
||||
contract** and the **federation track** (`SHARD-WP-0002`).
|
||||
|
||||
**v3** extends the model to the **SHARD-WP-0003 engine batch** (Federated Wiki, Wikibase/
|
||||
Wikidata, the git-forge wikis Gitea/GitLab/GitHub, TiddlyWiki, ikiwiki, Salesforce Quip,
|
||||
MojoMojo, Oddmuse, UseModWiki) — ~23 systems in all. Per-shard spectra grow from thirteen to
|
||||
**fourteen** (adds **provenance granularity**), and a **new coordination-layer axis** is
|
||||
introduced: the **federation-model taxonomy**.
|
||||
|
||||
Centerpieces:
|
||||
- **The shard family matrix** — every candidate backend × {substrate, attach mode(s),
|
||||
addressing, structure, history, merge, query, →Markdown, opacity}, with Xanadu/ZigZag as
|
||||
ideal anchors and Federated Wiki as the federation-model anchor. v3 adds the flat-file
|
||||
floor (Oddmuse/UseModWiki), git-IS-store (forge wikis/ikiwiki), TiddlyWiki, MojoMojo
|
||||
(direct-DB), Quip (external-API/HTML), and Wikibase (typed-graph far-end).
|
||||
- **The fourteen capability spectra** — the claim that the adapter contract should model
|
||||
*positions on spectra*, each anchored at both ends by a real system, with federation ops
|
||||
degrading by position. v3 adds **provenance granularity** (14th: per-shard → per-page →
|
||||
per-edit → per-statement/value), and refines merge-model (+fork/journal-replay,
|
||||
+coexist-with-rank), attachment-mode (+git-IS-store, +container, +direct-DB,
|
||||
+REST/file-store-hybrid, +external-API payload-format facet), native-query (+SPARQL/RDF
|
||||
far-end, +filter mid), history (+DB-version-rows, +partial/truncated), structure
|
||||
(+typed-graph, +inline-embedded objects), content-opacity (+proprietary-lossy-exportable),
|
||||
write-granularity (+story-item, +section-anchor).
|
||||
- **The federation-model taxonomy (§2.5, new)** — federation itself is plural: fork+journal
|
||||
(Federated Wiki), VCS-replication+ping (ikiwiki), query-time graph join (Wikibase SPARQL
|
||||
`SERVICE`), feed aggregation, activity streams, engine-mirror (Wiki.js). A
|
||||
selectable/composable coordination-layer axis feeding **T1–T6**.
|
||||
- **UC-44–UC-82 → workplan task mapping** (T1–T6 + T11–T16).
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Family matrix, the fourteen spectra, the federation-model taxonomy, cross-cutting through-lines, UC→task fold-in, recommendations/decisions, escalated open questions |
|
||||
|
||||
## Status
|
||||
|
||||
Synthesis v3 complete. No new use cases (consolidation only). Feeds `SHARD-WP-0002`:
|
||||
**T1–T6** federation-model taxonomy (selectable/composable federation); **T11** reframed
|
||||
around the **fourteen spectra** (incl. provenance granularity + expanded attachment modes +
|
||||
external-API payload-format facet); T12 page-model breadth (prose + typed/computed records +
|
||||
**inline-embedded objects** + **typed-graph statements** + non-Markdown assets +
|
||||
query-defined + multi-placement/DAG identity); T13 history (DB-rows + partial-flat-file =
|
||||
supplement; completeness metadata; **journal-shaped** coordination journal); T14 **full
|
||||
attachment taxonomy** (git-IS-store / container / direct-DB / REST-file-hybrid +
|
||||
source-of-truth per binding); T15 lossy + **not-Markdown** (graph/HTML); T16 (addressing
|
||||
incl. statement GUID + **opaque stable identity**, identity≠placement, derived index,
|
||||
**graph query / federated SERVICE**). UC coverage extended UC-34–UC-67 → **UC-34–UC-82**.
|
||||
|
||||
**New through-lines (v3):** federation is plural (the federation-model taxonomy); provenance
|
||||
has a granularity spectrum (down to per-statement, Wikibase rank/references); git is both the
|
||||
home store and the home journal (forge wikis make git *the* store, resolving the engine-
|
||||
mirror write-race); the flat-file floor (Oddmuse/UseModWiki, Wikipedia's MediaWiki Phase I)
|
||||
is the field's common root and the minimal capability profile; the page model must also carry
|
||||
typed-graph statements and inline-embedded objects.
|
||||
**Carried from v2:** CRDT changes the merge math; identity ≠ placement; metadata can be
|
||||
computed; content opacity is per-item; the attach surface is not always the native store; the
|
||||
block-graph-on-files sweet spot (Logseq `id::`).
|
||||
**Carried from v1:** files-canonical/index-derived; fine-grained addressing is adoptable;
|
||||
transclusion=clone=embed=reference is one primitive; structure/history federate iff in-text;
|
||||
attach mode is per-binding; the platform can enforce no-silent-mutation.
|
||||
406
research/260614-shard-spectrum-synthesis/findings.md
Normal file
406
research/260614-shard-spectrum-synthesis/findings.md
Normal file
@@ -0,0 +1,406 @@
|
||||
# Synthesis — the shard spectrum: one capability model across the full dive set
|
||||
|
||||
Date: 2026-06-14 · **revised 2026-06-14 (v3)** — extended from the fourteen-system set to
|
||||
include the **SHARD-WP-0003 engine batch** (Federated Wiki, Wikibase/Wikidata, the git-forge
|
||||
wikis Gitea/GitLab/GitHub, TiddlyWiki, ikiwiki, Salesforce Quip, MojoMojo, Oddmuse,
|
||||
UseModWiki). Per-shard spectra grown from thirteen to **fourteen** (added **provenance
|
||||
granularity**), with refinements to merge-model, attachment-mode, native-query, structure,
|
||||
history, content-opacity and write-granularity; and a **new coordination-layer axis — the
|
||||
federation-model taxonomy (§2.5)** — the headline v3 contribution.
|
||||
Source kind: **synthesis** — consolidates every deep dive into a single comparative model
|
||||
feeding the **shard adapter contract** and the **federation track** (`SHARD-WP-0002`).
|
||||
Lens: shard-wiki — what a backend must expose to participate, expressed as *spectra* of
|
||||
capability rather than a yes/no checklist.
|
||||
|
||||
> Purpose. ~23 tools/systems have now been studied. Two conceptual ancestors (**Xanadu,
|
||||
> ZigZag**); the file/DB engines (**XWiki, TWiki, Foswiki**, + landscape); the modern
|
||||
> note/PKB tools (**Roam, Obsidian, Notion, Joplin, Logseq, Anytype, AFFiNE, AppFlowy,
|
||||
> Trilium**); and the WP-0003 batch (**Federated Wiki, Wikibase, forge wikis, TiddlyWiki,
|
||||
> ikiwiki, Quip, MojoMojo, Oddmuse, UseModWiki**), against the federation and origin
|
||||
> research. This document reads them *across* each other. The payoff is a small set of
|
||||
> **capability spectra** — now **fourteen** per-shard, plus a **federation-model taxonomy**
|
||||
> for the coordination layer — each anchored at both ends by a real system, with federation
|
||||
> operations degrading by position. That spectrum *is* the adapter contract's design
|
||||
> surface. v3 folds the WP-0003 use cases (**UC-68–UC-82**) into the `SHARD-WP-0002` tasks.
|
||||
|
||||
Inputs: `research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`,
|
||||
`research/260613-{xwiki,twiki,foswiki}-deep-dive`,
|
||||
`research/260614-{xanadu,zigzag,roam,obsidian,notion,joplin,logseq,localfirst-workspaces,trilium,wikijs,federated-wiki,wikibase,forge-wikis,tiddlywiki,ikiwiki,quip,mojomojo,oddmuse,usemodwiki}-deep-dive`.
|
||||
Output target: `spec/TechnicalSpecificationDocument.md` (adapter contract) via
|
||||
`SHARD-WP-0002`.
|
||||
|
||||
---
|
||||
|
||||
## 1. The shard family matrix
|
||||
|
||||
Candidate-shard backends across the dimensions that matter to the contract. (Xanadu and
|
||||
ZigZag are *not* shards — they are the conceptual ideals each column aspires to; Federated
|
||||
Wiki is *barely* a shard — it is mostly a **federation model**, see §2.5; both listed apart.)
|
||||
|
||||
| Backend | Substrate | Attach mode(s) | Addressing | Structure | History | Merge | Query | →Markdown | Opacity |
|
||||
|---------|-----------|----------------|-----------|-----------|---------|-------|-------|-----------|---------|
|
||||
| **Git folder / forge wiki** | files (`.wiki.git`) | **file-store (git IS store)** | path (+ commit) | flat MD + frontmatter | **git-native** | git/text | no | native | none |
|
||||
| **ikiwiki** | files (git) → static HTML | file-store (source repo) | path | flat MD + directives | **git-native** | git/text | no | native (src) | none |
|
||||
| **Obsidian** | files | file-store **/** plugin | path + in-file `^id` | frontmatter (in-file) | none (Git plugin) | git/text | plugin (Dataview) | native (OFM) | none |
|
||||
| **Logseq** | files (MD/Org) → SQLite | file-store **/** plugin | **in-file block `id::`** (sweet spot) | `key::` props (in-file) | none (git) | git/text | **Datalog (derived)** | native-ish | none |
|
||||
| **TiddlyWiki** | single HTML **/** `.tid` dir | **file-store (container/`.tid`)** | tiddler `title` | typed tiddler **fields** | none / git (.tid) | git/text | **filters** | varies (`type`) | none |
|
||||
| **TWiki / Foswiki** | files + RCS / pluggable | file-store **/** API | path | `%META%` in-file (N records) | **open file (RCS)** | git/text | no | lossless (TML) | none |
|
||||
| **Oddmuse / UseModWiki** | flat files (CGI) | file-store (**minimal floor**) | CamelCase/path | none (flat) | flat-file, **partial** | last-writer | no | lossy | none |
|
||||
| **MojoMojo** | relational DB | **direct-DB-read** | page row / path | relational rows+lineage | **DB version rows** | last-writer | SQL | MD-in-column | none |
|
||||
| **Trilium** | SQLite (one file) | ETAPI / scripting / DB | `noteId` + **`branchId`** (id≠place) | labels+relations, **inherited+templated**; **DAG** | internal revisions | conflict-res | attr search | **lossy (HTML)** | **per-note** |
|
||||
| **Joplin** | SQLite-local | **sync-mirror** / local-REST / plugin | page-level `:/id` | notebooks + tags | internal revisions | conflict-notes | search | lossy | E2EE (whole) |
|
||||
| **Wiki.js** | DB ↔ git mirror | file-store(mirror) / **GraphQL** | path | frontmatter (mirror) | git (via mirror) | git/text | GraphQL | native (mirror) | none |
|
||||
| **XWiki** | DB (Hibernate) | in-engine host / REST | path | **XObjects/XClass** | internal (`xwikircs`) | git/text | yes (XWQL) | engine syntax | none |
|
||||
| **Quip** | hosted (SaaS) | **external-API (HTML)** | doc id + section anchor | **prose + inline spreadsheets/live-apps** | internal | server (OT) | no | **lossy (HTML)** | proprietary |
|
||||
| **Roam** | client DataScript | **in-app host only** | **store UUID** | `key::` attrs | internal (txn log) | (in-app) | yes (Datalog) | (Roam MD) | none |
|
||||
| **Notion** | hosted Postgres | **external-API (block-JSON)** | store UUID | **DB schema + relations + rollups** | internal, not portable | internal | yes (DB query) | **lossy** | none |
|
||||
| **Wikibase / Wikidata** | MediaWiki + RDF index | external-API + **SPARQL** | **opaque Q/P id + stmt GUID** | **typed entity-statement graph** | page revisions (JSON) | last-writer + **rank** | **SPARQL (+SERVICE)** | **lossy (not MD)** | none |
|
||||
| **Anytype** | **CRDT (any-sync)** | **replica / P2P node** | object id (store) | **typed object graph (ontology)** | CRDT log | **native-CRDT** | graph | lossy | **E2EE (whole)** |
|
||||
| **AFFiNE** | **CRDT (Yjs)** | replica / self-host | block id (store) | blocks; **one-data-many-views** | CRDT log | **native-CRDT** | DB filters | lossy | optional |
|
||||
| **AppFlowy** | **CRDT (Yrs)** | replica / self-host | block id (store) | **Notion-style DB + views** | CRDT log | **native-CRDT** | DB query | lossy | self-host |
|
||||
| — *Federated Wiki (mostly a federation model, §2.5)* | per-page JSON | **REST/file-store hybrid** | story-item `id` | typed story items | **append-only journal** | **fork + journal-replay** | neighborhood search | wiki-ish | none |
|
||||
| — *Xanadu (ideal)* | permascroll | — | **tumbler (span)** | spans + links | permanent | content-merge | — | — | — |
|
||||
| — *ZigZag (ideal)* | cells/dims | — | cell id | **N dimensions** | — | — | dimension walk | — | — |
|
||||
|
||||
Reading top to bottom is roughly shard-wiki's difficulty gradient, now with both ends
|
||||
extended: the **flat-file floor (Oddmuse/UseModWiki)** anchors the bottom — minimal, partial
|
||||
history, the field's common ancestor; **git/forge wikis, ikiwiki, Obsidian, Logseq,
|
||||
TiddlyWiki** are friction-free file-store cases (forge wikis make *git the store itself*);
|
||||
**TWiki/Foswiki/Trilium/Wiki.js** add translation, DAG/computed metadata, or a git mirror;
|
||||
**MojoMojo** needs direct-DB reads; **XWiki/Roam/Notion/Quip** add DB/SaaS structure and
|
||||
store/API addressing; **Wikibase** is the **typed-knowledge-graph far-end** (SPARQL,
|
||||
statement-level provenance); the **CRDT cohort** adds native merge + (Anytype) P2P/E2EE.
|
||||
Xanadu/ZigZag mark the ideals; Federated Wiki marks the federation-model ideal (§2.5).
|
||||
|
||||
---
|
||||
|
||||
## 2. The capability spectra (the contract's real shape) — fourteen
|
||||
|
||||
Each capability is **not boolean** — it is a position on a spectrum anchored at each end by
|
||||
a real system. The contract models *positions*; federation ops degrade by position.
|
||||
|
||||
1. **Addressing granularity** — `none → whole-page(path) → page-level store id(Joplin :/id,
|
||||
Trilium noteId) → in-file span(Obsidian ^id) → in-file block(Logseq id::, the sweet spot:
|
||||
block-level AND git-diffable) → store-minted span(Roam/Notion/CRDT UUID) → statement GUID
|
||||
(Wikibase) → portable tumbler(Xanadu ideal)`. Story-item `id` (Federated Wiki) is a
|
||||
mid-tier within-page handle. (UC-51, UC-44/45, UC-73.)
|
||||
2. **Content identity** — `none → path/title → opaque stable id, labels-as-annotation
|
||||
(Wikibase Q/P) → fingerprint(hash) → span-set/equivalence(Xanadu)`. Wikibase is the
|
||||
cleanest real instance of **stable, language-neutral identity**. (UC-46, UC-27, UC-73.)
|
||||
3. **Identity vs placement** — `path = identity(most) → identity separated from placement
|
||||
(Trilium note/branch; a page in many locations = a DAG) → provenance-edge links across
|
||||
sites (Federated Wiki fork-DAG; forge wiki = path, identity layered above)`. The clean
|
||||
model for a page under multiple paths/shards. (UC-66, UC-22, UC-71.)
|
||||
4. **Structure** — `flat Markdown(Oddmuse) → in-file frontmatter/key::(Obsidian/Logseq) →
|
||||
in-file %META%(TWiki) → tiddler fields(TiddlyWiki) → relational rows(MojoMojo) → typed
|
||||
objects(XWiki) → DB schema+relations+rollups(Notion/AppFlowy) → prose+inline-embedded
|
||||
objects(Quip) → typed object graph/ontology(Anytype) → computed inherited+templated
|
||||
(Trilium) → typed entity-statement knowledge graph(Wikibase)`. In-text federates;
|
||||
DB-locked needs sidecar+fidelity; **graph/computed** needs effective-vs-own +
|
||||
render-without-flatten. (UC-34, UC-39, UC-58, UC-67, UC-73, UC-80.)
|
||||
5. **History** — `none → partial/truncated flat-file(Oddmuse keep/) → internal-only/
|
||||
not-portable(Notion/Joplin/Trilium) → DB version rows(MojoMojo) → CRDT update log
|
||||
(Anytype/AFFiNE/AppFlowy) → append-only semantic journal(Federated Wiki) → open file
|
||||
format(TWiki RCS) → git-native(Git/forge/ikiwiki/Obsidian/Logseq/Wiki.js mirror)`.
|
||||
Internal/CRDT/DB-rows ⇒ *supplement*; open-file/journal ⇒ *import*; git ⇒ *adopt*.
|
||||
**Completeness is metadata** (Oddmuse history is partial — never imply complete). (UC-36,
|
||||
UC-41, UC-71, UC-81, UC-82.)
|
||||
6. **Merge model** — `none → last-writer → git/text 3-way merge → conflict-notes/keep-both
|
||||
(Joplin) → fork + manual journal-replay(Federated Wiki) → coexist-with-rank(Wikibase:
|
||||
contradictory values kept, curated) → native-CRDT conflict-free(Anytype/AFFiNE/AppFlowy)`.
|
||||
Four+ distinct models — never impose git/text merge on a CRDT or a journal shard; speak
|
||||
the shard's model or stay projection/overlay. (UC-64, UC-71, UC-75.)
|
||||
7. **Native query** — `none → text search → filter expressions(TiddlyWiki) → build-your-own
|
||||
derived index(Logseq DataScript over files; shard-wiki can do likewise) → datalog/graph
|
||||
(Roam/Anytype) → DB query(Notion/AppFlowy/XWiki) → SPARQL/RDF + federated SERVICE
|
||||
(Wikibase, the far-end + query-time cross-shard join)`. Delegate where present; **build an
|
||||
index over the projection** where not. (UC-52, UC-63, UC-05, UC-54, UC-74.)
|
||||
8. **Translation to Markdown** — `native → lossless round-trip(Foswiki TML↔HTML) → lossy-
|
||||
with-fidelity-report(HTML/CKEditor Trilium; Notion blocks; Quip HTML+embedded objects;
|
||||
CRDT/object models) → not-Markdown-at-all(Wikibase statements → lossy render or keep
|
||||
graph)`. Lossless ⇒ writable; lossy ⇒ read/projection floor + visible fidelity loss;
|
||||
not-MD ⇒ structured payload + optional rendered view. (UC-42, UC-59, UC-03, UC-73, UC-80.)
|
||||
9. **Attachment mode** — a **per-binding, capability-gated** choice; a backend may offer
|
||||
several. The full taxonomy:
|
||||
- **file-store** — *native on-disk store* (Obsidian/Logseq/TWiki), *git IS the store*
|
||||
(forge `.wiki.git`, ikiwiki source), *single-file container* (TiddlyWiki HTML),
|
||||
*flat-file floor* (Oddmuse), **or** *interchange/sync mirror* (Joplin; Wiki.js git
|
||||
mirror) (UC-40, UC-53, UC-60, UC-62, UC-76, UC-78, UC-79, UC-82, UC-68)
|
||||
- **in-engine host** — adapter inside the app via its API (Roam/Obsidian/Logseq/Trilium
|
||||
scripting, XWiki components) (UC-38, UC-50)
|
||||
- **local-REST** — localhost API, app-running (Joplin Data API; Trilium ETAPI) (UC-38)
|
||||
- **external-API** — remote API from outside, with a **payload-format facet**:
|
||||
*block-JSON* (Notion, UC-57), *GraphQL* (Wiki.js, UC-69), *HTML* (Quip, UC-80),
|
||||
*forge wiki REST* (GitLab/Gitea, UC-77), *MediaWiki/SPARQL* (Wikibase, UC-73/74)
|
||||
- **direct-DB-read** — read the engine's relational store (MojoMojo) when no file/API
|
||||
exists; schema = a versioned coupling (UC-81)
|
||||
- **CRDT replica** — hold a local CRDT replica (Anytype/AFFiNE/AppFlowy) (UC-64)
|
||||
- **P2P / no-central-endpoint** — replica or peer/node, not a URL (Anytype) (UC-65)
|
||||
- **REST/file-store hybrid** — page JSON over HTTP+CORS or static files (Federated Wiki)
|
||||
(UC-70)
|
||||
10. **Operational envelope** — `local/unbounded → realtime CRDT/WebSocket → rate-limited+
|
||||
eventually-consistent+paginated(Notion ~3 rps, Quip, Wikibase public endpoints)`. Sets
|
||||
live vs cache/poll/webhook. (UC-57, UC-31.)
|
||||
11. **Access grant** — `open(L0; Oddmuse) → token → OAuth scoped+revocable(Notion) →
|
||||
enterprise SSO + ACL(Quip/Salesforce, Wiki.js path rules) → P2P key/invite(Anytype) →
|
||||
own-site-only writes(Federated Wiki)`. The backend may *enforce* no-silent-mutation.
|
||||
(UC-57, UC-06, UC-65, [[shard-wiki-auth-in-core-decision]].)
|
||||
12. **Content opacity** — `plaintext(files) → proprietary-but-lossy-exportable(Quip HTML;
|
||||
Notion) → encrypted-at-rest whole-shard(Joplin/Anytype E2EE) → per-item(Trilium
|
||||
protected notes)`. Opaque ⇒ backup/structure-shell; never present ciphertext (or imply
|
||||
a lossy export is faithful) as readable. (UC-61, UC-80.)
|
||||
13. **Write granularity** — `whole-file(TiddlyWiki single-file) → per-page/note(Git/forge/
|
||||
Obsidian/Joplin/Trilium/Oddmuse) → story-item/paragraph(Federated Wiki) → section-anchor
|
||||
splice(Quip) → per-statement(Wikibase API) → per-block(Roam/Notion/Logseq/CRDT)`. Sets
|
||||
overlay/patch/lock/conflict scope; whole-file ⇒ **no per-page atomicity**. (UC-35,
|
||||
UC-78.)
|
||||
14. **Provenance granularity** *(new, 14th)* — `none → per-shard → per-page(most; author/
|
||||
time) → per-commit(git) → per-edit(journal entry, Federated Wiki) → per-statement/value
|
||||
(Wikibase references + rank)`. How finely the union can attribute and source content;
|
||||
Wikibase pushes provenance below the page (sourced, contradictory values coexist with a
|
||||
curation signal). The page model + journal should *allow* sub-page provenance even if MVP
|
||||
records per page. (UC-24, UC-71, UC-75.)
|
||||
|
||||
*(Content types — Markdown-only → typed records → inline-embedded objects (Quip
|
||||
spreadsheets) → non-Markdown assets (Excalidraw/Canvas/whiteboards) → typed-graph statements
|
||||
(Wikibase) — remains a cross-cutting page-model demand, tracked under structure + T12 rather
|
||||
than as a standalone capability. UC-55.)*
|
||||
|
||||
Design consequence: **T11's capability vocabulary = these fourteen spectra**, not a flat
|
||||
`read/write/diff/...` list. The flat verbs remain the *operations*; the spectra are the
|
||||
*profile* saying how well each verb is supported and how it degrades. The **floor**
|
||||
(Oddmuse/UseModWiki) and **far-ends** (Wikibase graph/query/provenance; CRDT merge; Notion
|
||||
hosted) bound every spectrum with a real system.
|
||||
|
||||
---
|
||||
|
||||
## 2.5. The federation-model taxonomy (the coordination-layer axis) — new in v3
|
||||
|
||||
The fourteen spectra above describe a *single shard's* capabilities. The WP-0003 batch
|
||||
revealed a second, orthogonal axis the v2 synthesis under-modelled: **federation itself is
|
||||
plural.** "Attach many shards and present a union" can be realized by several distinct
|
||||
coordination models, each a real system, each with different reconciliation semantics. This
|
||||
axis lives at shard-wiki's **coordination layer** (`SHARD-WP-0002` T1–T6), not in a single
|
||||
adapter.
|
||||
|
||||
| Federation model | Exemplar | Mechanism | Reconciliation | Discovery |
|
||||
|------------------|----------|-----------|----------------|-----------|
|
||||
| **Fork + journal** | Federated Wiki | copy page to own site; append-only semantic **journal** records `fork`-with-source | **manual**: compare journals, fork the version you prefer; **chorus**, no canonical | link + fork (**neighborhood**) / curated **roster** |
|
||||
| **VCS replication + ping** | ikiwiki | git clone/pull/push between instances; **XML-RPC pinger** notifies peers to pull/rebuild | **git merge** across clones | configured peers + pings |
|
||||
| **Query-time graph join** | Wikibase | **SPARQL `SERVICE`** runs a sub-query on another endpoint and joins | none (read-time join; rank curates conflicts) | endpoint URLs |
|
||||
| **Feed aggregation** | ikiwiki `aggregate`, RSS/Atom | pull remote feeds in as pages | one-way inbound projection | feed URLs |
|
||||
| **Activity streams** | ActivityPub (federation research) | actor/inbox/outbox Create/Update | per-actor; eventual | actor handles |
|
||||
| **Engine-maintained mirror** | Wiki.js git mirror | DB-canonical engine syncs to a git mirror | engine owns DB↔git sync (don't double-sync) | the mirror repo |
|
||||
|
||||
Two cross-cutting lessons:
|
||||
|
||||
- **git-IS-store vs engine-mirror resolves the write-race.** A **forge wiki** (`.wiki.git`)
|
||||
and **ikiwiki source** make *git the canonical store*, so **write-by-commit is safe** — no
|
||||
engine to race. Wiki.js (engine-mirror, DB canonical) is the opposite and needs care. The
|
||||
contract must record *which side is source of truth* per binding. (Resolves UC-68's open
|
||||
race for the git-canonical case; UC-76.)
|
||||
- **shard-wiki's own coordination journal should be journal-shaped.** Federated Wiki proves
|
||||
the **append-only semantic-op log with provenance entries, page-state-as-derived-replay**
|
||||
pattern in production — the concrete shape for INTENT's coordination journal (UC-71), and a
|
||||
superset that can *ingest* git history, CRDT logs, DB version rows, and partial flat-file
|
||||
histories as differently-grained inputs.
|
||||
|
||||
Design consequence: **T1–T6 should model federation as a selectable model (or composition of
|
||||
models), not a single hard-coded flow** — mechanism over policy at the coordination layer,
|
||||
mirroring how T11 models per-shard capability as spectra. A given information space may use
|
||||
fork+journal for human-curated shards, VCS-replication for git shards, query-join for graph
|
||||
shards, and feed-aggregation for read-only sources — concurrently.
|
||||
|
||||
---
|
||||
|
||||
## 3. Cross-cutting findings (the through-lines)
|
||||
|
||||
- **Files-canonical, index-derived is the winning architecture.** Obsidian's MetadataCache,
|
||||
**Logseq's DataScript-over-files**, Git's working tree, **Wikibase's WDQS (SPARQL index
|
||||
rebuilt from canonical JSON entities)**, and shard-wiki's projection model agree: the
|
||||
graph/backlinks/query index is **derived and rebuildable**, never a second source of truth.
|
||||
Roam/Notion/CRDT invert this (store canonical) and pay in portability. shard-wiki keeps
|
||||
files + journal canonical.
|
||||
- **Git is both the home store and the home journal.** *(sharpened in v3)* Forge wikis make
|
||||
git *the* store; ikiwiki and Wiki.js make git the source or mirror; for all of them the
|
||||
**git log is the coordination journal with zero synthesis**. The git-canonical cases are
|
||||
the friction-free core; everything else is measured as deviation.
|
||||
- **The flat-file floor is the field's common root.** *(new)* Oddmuse and UseModWiki
|
||||
(Wikipedia's MediaWiki Phase I) show the minimal plain-text page+history wiki is the
|
||||
ancestor every richer engine elaborates — so the **minimal/floor capability profile** is
|
||||
the right baseline, and shard-wiki's page model must stay attach-compatible with it
|
||||
(flat files, CamelCase identities, **partial** history). (UC-82.)
|
||||
- **Federation is plural** *(new — §2.5).* Fork+journal, VCS-replication+ping, query-time
|
||||
graph join, feed aggregation, activity streams, engine-mirror — distinct coordination
|
||||
models, selectable and composable, not one flow.
|
||||
- **Provenance has a granularity spectrum** *(new — spectrum 14).* From per-shard down to
|
||||
**per-statement/value** (Wikibase references + rank). Union-without-erasure includes
|
||||
*attribution at the right grain* and letting sourced contradictions coexist with curation.
|
||||
- **Identity ≠ placement.** Trilium's **note vs branch** (a note cloned into many locations =
|
||||
a DAG) and Wikibase's **opaque stable IDs (labels-as-annotation)** are the clean models:
|
||||
separate *what a page is* from *where it sits* and from *what it's called*. Federated
|
||||
Wiki's fork entries are **provenance edges** between same-named pages across sites.
|
||||
- **Transclusion ⇄ clone ⇄ embed ⇄ cloned-note ⇄ reference is one primitive** over an
|
||||
addressable union (Xanadu, ZigZag, Roam/Logseq, Obsidian `![[ ]]`, Notion synced block,
|
||||
Trilium cloning, TiddlyWiki `{{ }}`). (UC-32/44/45/51/66.)
|
||||
- **CRDT changes the merge math; journals and rank add more models.** *(extended)* The merge
|
||||
spectrum now spans last-writer → git 3-way → conflict-notes → **fork+journal-replay** →
|
||||
**coexist-with-rank** → native-CRDT. Never impose git/text merge across that range.
|
||||
- **Structure & history federate iff in text; metadata can be computed; the far-end is a
|
||||
graph.** `%META%`/frontmatter/`key::`/tiddler-fields diff and travel; XObjects/Notion-DB/
|
||||
CRDT/Quip-objects lock in; Trilium computes metadata (inherited+templated); **Wikibase is
|
||||
a full typed knowledge graph** — render to Markdown is lossy, so keep the graph and offer a
|
||||
view, never silent-flatten.
|
||||
- **The attach surface is rarely "just files," and source-of-truth varies.** *(extended)*
|
||||
Joplin's best surface is its sync mirror; Logseq offers file-graph or DB-graph and is
|
||||
migrating substrate; **forge wikis = git-canonical (write freely)** vs **Wiki.js =
|
||||
DB-canonical mirror (write carefully)** vs **MojoMojo = DB-only (direct read)**. Bind to
|
||||
capabilities *and* record the source of truth.
|
||||
- **The page model must stretch many ways at once:** prose Markdown, typed/computed records
|
||||
(N-per-page, relations, inheritance), **inline-embedded objects** (Quip), **typed-graph
|
||||
statements** (Wikibase), non-Markdown assets, reference/query-defined pages, and
|
||||
**multi-placement (DAG) identity**. The heaviest demand on T12.
|
||||
|
||||
---
|
||||
|
||||
## 4. How the use cases fold into the workplan
|
||||
|
||||
The 260614 + WP-0003 use cases (UC-44–UC-82) map onto the adapter-contract and federation
|
||||
tasks:
|
||||
|
||||
| UCs | Theme | Lands in |
|
||||
|-----|-------|----------|
|
||||
| UC-35, UC-50, UC-53, UC-57, UC-60, UC-62, UC-64, UC-65, **UC-68, UC-70, UC-76, UC-77, UC-78, UC-79, UC-80, UC-81, UC-82** | attachment modes (file-store native/git-IS-store/container/mirror, in-engine, local-REST, external-API w/ payload-format, direct-DB, CRDT-replica, P2P, REST/file-store-hybrid) + operational envelope | **T11** + **T14** |
|
||||
| UC-34, UC-39, UC-55, UC-58, UC-67, **UC-73, UC-80** | structured/typed/computed/**graph** payload, **inline-embedded objects**, non-MD assets | **T12** |
|
||||
| UC-36, UC-41, **UC-81, UC-82** | history: internal/CRDT-log/**DB-rows**/**partial-flat-file** = supplement; **completeness metadata** | **T13** |
|
||||
| UC-42, UC-59, **UC-73, UC-80** | translation: lossless vs lossy-with-fidelity vs **not-Markdown** (graph/HTML/objects) | **T15** |
|
||||
| UC-31, **UC-79** | webhooks / realtime / push-vs-poll / **VCS ping** | **T6** / **T11** envelope |
|
||||
| UC-57 §6, UC-61, UC-65, **UC-06, UC-80** | scoped grant; **enterprise ACL/SSO**; content opacity (whole/per-item/**lossy-exportable**); P2P key | **T11** (access-grant + opacity) |
|
||||
| UC-44, UC-45, UC-46, UC-51, UC-63, UC-66, **UC-73, UC-74** | span/statement addressing, **opaque stable identity**, identity≠placement, transclusion-as-reference, derived index, **graph query** | **T16** (+ T12) |
|
||||
| UC-47, UC-48, UC-52, UC-54, **UC-74** | dimensional navigation, query delegation/build, query-defined pages, **SPARQL/federated SERVICE** | **T16** (+ T5/T10) |
|
||||
| UC-24, **UC-71, UC-75** | **provenance granularity** (per-edit / per-statement); coordination journal shape | **T13/T16** + journal |
|
||||
| UC-26, UC-27, UC-28, UC-30, UC-05, **UC-71, UC-72, UC-79** | **federation-model taxonomy** (fork+journal, VCS-replication+ping, query-join, chorus/neighborhood/roster) | **T1–T6** |
|
||||
|
||||
The one **structural** v3 change: the **federation-model taxonomy (§2.5)** is a new design
|
||||
surface for **T1–T6** — federation becomes a *selectable, composable model* rather than a
|
||||
single flow. Per-shard, **T11** grows to **fourteen spectra** (adds provenance granularity)
|
||||
and **T14** absorbs the expanded attachment-mode taxonomy (git-IS-store, container,
|
||||
direct-DB, REST/file-store-hybrid, external-API payload-format facet).
|
||||
|
||||
---
|
||||
|
||||
## 5. Recommendations (decisions to record under SHARD-WP-0002)
|
||||
|
||||
1. **Model per-shard capabilities as the fourteen spectra (§2)**, not flat verbs. (T11.)
|
||||
2. **Model federation as a selectable/composable taxonomy of models (§2.5)** — fork+journal,
|
||||
VCS-replication+ping, query-time graph join, feed aggregation, activity streams,
|
||||
engine-mirror — at the coordination layer. (T1–T6.)
|
||||
3. **Make the coordination journal journal-shaped** (Federated Wiki): append-only semantic
|
||||
ops + provenance entries, page state = derived replay; able to *ingest* git history, CRDT
|
||||
logs, DB version rows, and partial flat-file histories as differently-grained inputs.
|
||||
(T13 + journal, UC-71.)
|
||||
4. **Add provenance granularity as a spectrum** (per-shard → per-page → per-edit →
|
||||
per-statement/value); allow sub-page provenance and **coexist-with-rank** for sourced
|
||||
contradictions. (T11/T13, UC-24/75.)
|
||||
5. **Record source-of-truth per binding** and resolve the write-race accordingly:
|
||||
git-canonical (forge wiki / ikiwiki) ⇒ write-by-commit; engine-mirror (Wiki.js) ⇒ careful;
|
||||
direct-DB (MojoMojo) ⇒ read/projection/overlay default. (T14, UC-68/76/79/81.)
|
||||
6. **Expand attachment mode (§2 #9 / T14):** file-store (native / **git-IS-store** /
|
||||
**container** / mirror / **flat-file floor**), in-engine, local-REST, **external-API with
|
||||
a payload-format facet** (block-JSON/GraphQL/HTML/forge-REST/SPARQL), **direct-DB-read**,
|
||||
CRDT-replica, P2P, **REST/file-store-hybrid**. Bind to capabilities; substrate can
|
||||
migrate. (T14, UC-40/68/70/76/77/78/79/80/81/82 + UC-43.)
|
||||
7. **Page model: support a typed-graph payload and inline-embedded objects, not only
|
||||
typed records** — render to Markdown lossily, keep the graph/objects canonical; preserve
|
||||
computed (inherited/templated) metadata as effective-vs-own. (T12/T15, UC-73/80/67.)
|
||||
8. **Capability profile must express absence and partiality cleanly** (Oddmuse floor):
|
||||
sparse profiles, **partial/truncated history reported honestly**, graceful degradation to
|
||||
read/projection/overlay/backup. (T11/T13, UC-82.)
|
||||
9. **Query: SPARQL/RDF + federated `SERVICE` is the graph far-end** — delegate to native
|
||||
engines (filters/datalog/DB-query/SPARQL) where present; build a derived index over the
|
||||
projection where not. (T16/T5, UC-52/63/74.)
|
||||
10. **Adopt opaque stable identity with labels-as-annotation** (Wikibase) and **separate
|
||||
identity from placement** (Trilium note/branch); fork entries are provenance edges.
|
||||
(T16, UC-73/66/71.)
|
||||
11. **Keep files + coordination journal canonical; all indexes/projections derived and
|
||||
rebuildable.** (Architecture invariant, INTENT.)
|
||||
|
||||
These honor INTENT: mechanism over policy (capability *and now federation* modelled as
|
||||
spectra/taxonomy, not hard-coded behaviors), union without erasure (fidelity +
|
||||
effective-vs-own + identity/placement + **provenance granularity** all preserve
|
||||
information), graceful degradation (every spectrum has a read/projection floor; the
|
||||
Oddmuse floor is explicit; opaque/CRDT/graph shards degrade to backup/projection/lossy-view),
|
||||
no silent mutation (access-grant + opacity + overlay + respect native merge + **source-of-
|
||||
truth per binding**), shard sovereignty (no backend forced to change substrate),
|
||||
Markdown-first/backend-neutral (in-text preferred; DB/CRDT/HTML/**graph** tolerated with a
|
||||
view).
|
||||
|
||||
---
|
||||
|
||||
## 6. Open questions escalated by the synthesis
|
||||
|
||||
1. **Federation composition** — can one information space run several federation models
|
||||
(§2.5) concurrently over different shards, and how does the union reconcile a chorus
|
||||
(fork+journal) with a canonical-asserting shard (Notion / upstream main)? (T1–T6,
|
||||
UC-72.)
|
||||
2. **Coordination-journal op vocabulary** — adopt Federated Wiki's exact ops
|
||||
(create/add/edit/move/remove/fork) at item grain, or an abstract op set other shards can
|
||||
emit, with git/CRDT/DB-row/flat-file histories ingested as inputs? (T13, UC-71.)
|
||||
3. **Provenance granularity in the model** — does the journal/page model carry per-statement
|
||||
provenance + rank (Wikibase), per-edit (journal), or per-page (MVP), configurable? (T13,
|
||||
UC-75.)
|
||||
4. **Typed-graph page** — model a Wikibase entity natively (statements) or always project to
|
||||
a lossy Markdown/table view, or both (canonical graph + view)? Is SPARQL a union-level
|
||||
capability or pass-through? (T12/T16, UC-73/74.)
|
||||
5. **Source-of-truth + write-race** — formalize git-canonical vs engine-mirror vs direct-DB
|
||||
per binding; sanction direct third-party DB reads (schema drift) and write-by-commit
|
||||
timing. (T14, UC-68/76/79/81.)
|
||||
6. **Portable span address** across heterogeneous backends — wrap native IDs (Roam/Notion/
|
||||
CRDT UUID, Logseq `id::`, Trilium noteId, **Wikibase Q/P + stmt GUID**, fedwiki story-item
|
||||
id) in a shard-scoped address? (T16.)
|
||||
7. **CRDT shards** — embed a CRDT client (Yjs/Yrs) for a live replica, or consume snapshots?
|
||||
Overlays as CRDT ops or out-of-band patches? (T14, UC-64.)
|
||||
8. **Page model breadth** — can one model carry prose + typed/computed records +
|
||||
inline-embedded objects + typed-graph statements + non-MD assets + query-defined pages +
|
||||
multi-placement identity coherently? (T12.)
|
||||
9. **Whole-file & partial-history shards** — overlays against a whole-file shard
|
||||
(TiddlyWiki single-file: buffer+re-serialize vs require `.tid`); representing
|
||||
partial/truncated history (Oddmuse) with completeness metadata. (T11/T13, UC-78/82.)
|
||||
10. **Content opacity & lossy-exportable** — what is visible for an opaque/proprietary shard
|
||||
(IDs/structure vs nothing vs lossy HTML); never present a lossy export as faithful; does
|
||||
shard-wiki ever hold keys? (T11, UC-61/80.)
|
||||
|
||||
---
|
||||
|
||||
## 7. Sources
|
||||
|
||||
A synthesis; primary sources are the ~23 dives' `findings.md` files plus
|
||||
`research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`.
|
||||
No new external research was performed in v3 (it consolidates the WP-0003 batch dives
|
||||
260614-{federated-wiki,wikibase,forge-wikis,tiddlywiki,ikiwiki,quip,mojomojo,oddmuse,
|
||||
usemodwiki}).
|
||||
|
||||
Cross-references: `spec/UseCaseCatalog.md` (UC-26–UC-82),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (T1–T16),
|
||||
`workplans/SHARD-WP-0003-engine-dives-batch.md` (the engine batch, done),
|
||||
`INTENT.md` (constraints), [[shard-wiki-auth-in-core-decision]].
|
||||
|
||||
---
|
||||
|
||||
## 8. Traceability
|
||||
|
||||
- Consolidates: all ~23 deep dives + federation/origin research into one capability model
|
||||
(v3 adds the WP-0003 batch).
|
||||
- Feeds: `SHARD-WP-0002` **T1–T6** (federation-model taxonomy §2.5 — selectable/composable
|
||||
federation), **T11** (**fourteen-spectra** vocabulary, incl. provenance granularity +
|
||||
expanded attachment modes + payload-format facet), T12 (page-model breadth: typed-graph,
|
||||
inline-embedded objects, computed metadata, identity≠placement, non-MD), T13 (history incl.
|
||||
DB-rows + partial-flat-file = supplement, completeness metadata, journal-shaped
|
||||
coordination journal), T14 (full attachment taxonomy incl. git-IS-store / container /
|
||||
direct-DB / REST-file-hybrid + source-of-truth per binding), T15 (lossy + not-Markdown
|
||||
graph/HTML), T16 (addressing incl. statement GUID + opaque stable identity, graph query /
|
||||
federated SERVICE).
|
||||
- UC coverage extended in the workplan from UC-34–UC-67 to **UC-34–UC-82**.
|
||||
- No UCs added (synthesis only); no boundary changes (INTENT Stability Note untouched —
|
||||
the federation-model taxonomy is a refinement of *how* the coordination layer works, not a
|
||||
redefinition of shard / Git's role / orchestrator-vs-engine).
|
||||
33
research/260614-squeak-pharo-deep-dive/README.md
Normal file
33
research/260614-squeak-pharo-deep-dive/README.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# 260614 — Squeak & Pharo (image-based Smalltalk) deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T6 (Squeak) + T8 (Pharo)** — combined memo
|
||||
(justified merge: both are image-based Smalltalks; Pharo is T8's thin "context for T6/T7").
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into the **image-based live-object** environment — Squeak and Pharo (the
|
||||
substrate Glamorous Toolkit T7 runs on): the **image** as a persistent world of live objects
|
||||
with **no file/document/app boundary**, the live **inspector**, and Pharo's retreat to
|
||||
**code-as-text in git** (Tonel/Iceberg).
|
||||
|
||||
## Why it matters
|
||||
|
||||
- The **purest "live" end** of the batch's spectrum (literate source → notebook snapshot →
|
||||
GT/Lepiter live-over-files → **image: everything live**). Names the **live↔snapshot** axis
|
||||
the projection model (T16) must carry.
|
||||
- Hardens the **image-is-not-a-store** boundary (opaque monolithic non-diffable blob; no
|
||||
page identity/history/provenance) — generalizes "attach files, not the kernel/image"
|
||||
(UC-84, T7) into a named binding boundary (T14).
|
||||
- Pharo **confirms** the resolution: even image traditions externalize to **git-versionable
|
||||
text** (Tonel) to version/collaborate — files-canonical from the Smalltalk side.
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (boundary / design prior art; covers T6 and T8). Boundary for UC-34/35/79;
|
||||
links UC-83/84 (live→snapshot), UC-54/47/48 (live-object inspection), UC-76/79 (Tonel/git).
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | The image & live objects, the image-as-store anti-pattern, Pharo Tonel/Iceberg→git, INTENT mapping, UC disposition (enrichment-only), architecture notes (T14 boundary, T16 live↔snapshot axis) |
|
||||
141
research/260614-squeak-pharo-deep-dive/findings.md
Normal file
141
research/260614-squeak-pharo-deep-dive/findings.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# Squeak & Pharo (image-based Smalltalk) — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 **T6 (Squeak)** + **T8 (Pharo)** — combined
|
||||
(justified merge: both are image-based Smalltalks; Pharo is T8's "context for T6/T7," kept
|
||||
brief per the workplan). · **Subject:** the **image-based live-object** environment and what
|
||||
it teaches (and warns) about shard-wiki's page model.
|
||||
|
||||
## Why this dive (and why merged)
|
||||
|
||||
Squeak (1996, the Alan Kay/Ingalls lineage) and Pharo (2008 fork, the substrate **Glamorous
|
||||
Toolkit** T7 runs on) are the same idea: **a persistent world of live objects, the "image,"
|
||||
with no file/document/application boundary.** They are **not candidate shards** — they are
|
||||
the **anti-pattern boundary** for shard-wiki's files-canonical stance *and* the inspiration
|
||||
behind moldable inspection (T7). T8 (Pharo) is folded here as the workplan allows: its
|
||||
distinct contribution over Squeak (Tonel/Iceberg file-based code → git) is the one piece that
|
||||
*does* touch our concerns, covered in §3.
|
||||
|
||||
## 1. The image: knowledge-as-live-objects
|
||||
|
||||
- A Smalltalk **image** is a serialized snapshot of the **entire object memory** — every
|
||||
object, class, tool, window, and the running program state — persisted as one binary file
|
||||
(`.image` + `.changes` log). You resume exactly where you left off.
|
||||
- **"Everything is a live object"**: code, data, UI (Morphic), the debugger, the inspector —
|
||||
all are objects you can open, message, and modify *in place*, while running. There is **no
|
||||
edit/compile/run cycle** and **no document-vs-app distinction**.
|
||||
- The **inspector** lets you open any object and explore/modify its state live — the direct
|
||||
ancestor of GT's moldable inspector (T7), but generic rather than domain-molded.
|
||||
|
||||
This is the **purest "live" end** of the spectrum the whole batch traverses: literate source
|
||||
(static) → notebook captured output (snapshot) → GT/Lepiter (live results over files) →
|
||||
**image (everything live, nothing inherently a file).**
|
||||
|
||||
## 2. The boundary: image-as-store is the anti-pattern
|
||||
|
||||
The image directly **contradicts** shard-wiki's design constraints, which is exactly why it's
|
||||
worth recording as a hard boundary:
|
||||
|
||||
- **Opaque, monolithic, non-diffable.** The image is one big binary blob of entangled live
|
||||
state — no per-page identity, no text diff, no mergeable history, no provenance per unit.
|
||||
It violates *union-without-erasure granularity*, *Markdown-first*, and *git-addressable
|
||||
coordination*.
|
||||
- **No stable addressable "page."** Knowledge is an object graph in memory, not addressable
|
||||
documents — there is nothing to attach at page granularity without an export step.
|
||||
- **History is a `.changes` log**, a serial source-change stream, not a content history.
|
||||
|
||||
**Conclusion (boundary recorded):** an image is **not a shard and not a store**. This is the
|
||||
generalized form of the rule already hit at Jupyter (UC-84) and GT (T7): *attach the
|
||||
exported files, never the live image/kernel.* The image can only participate via an **export
|
||||
projection** (objects/code → files), which is a **derivation-projection** (T1) that
|
||||
**degrades the liveness to a static snapshot**.
|
||||
|
||||
## 3. Pharo's twist: code-as-files (Tonel) → git (the one actionable bit)
|
||||
|
||||
Pharo (T8) matters precisely where it **retreats from pure-image**:
|
||||
|
||||
- **Tonel / FileTree** serialize each class/method as **plain-text files** in a directory,
|
||||
and **Iceberg** manages those files as a **git repository** — so Pharo code lives in git as
|
||||
text, diffable and mergeable, *outside* the image.
|
||||
- This is the **same move** as Lepiter (T7), nbstripout/Jupytext (T3), and ikiwiki source
|
||||
(UC-79): **the durable, attachable artifact is the file representation; the live
|
||||
environment is a layer above it.** It confirms our stance from the *Smalltalk* side: even
|
||||
the most image-centric tradition externalizes to **git-versionable text** to collaborate.
|
||||
|
||||
So Pharo adds **no new page-model idea** beyond "image-based environments still externalize
|
||||
to git text" — exactly the "keep brief / fold" expectation. Its value is **confirming the
|
||||
boundary resolution**: attach the **Tonel/git source**, treat the image as live-only.
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
### Inspiration (keep)
|
||||
|
||||
- **Live-object inspection** is the seed of moldable views (T7/UC-54): the *idea* that any
|
||||
unit can be opened and explained interactively. shard-wiki adopts this as **projection/
|
||||
view**, not as a storage model.
|
||||
- **Resume-where-you-left-off** liveness names the far end of the **live↔snapshot** axis the
|
||||
contract must place every computed/projected view on (UC-83/84): the more live the source,
|
||||
the more its attached form is a **degrading snapshot**.
|
||||
|
||||
### Boundary (enforce — design-bug if violated)
|
||||
|
||||
- **Image-as-store is a design-bug boundary.** Never model an image (or any monolithic live
|
||||
memory blob) as a shard/store; participate only via **export → files** (a degrading
|
||||
derivation-projection). Generalizes "attach files, not the kernel/image" (UC-84, T7).
|
||||
|
||||
### Confirmation (Pharo)
|
||||
|
||||
- Even pure-image traditions externalize to **git-versionable text** (Tonel/Iceberg) to
|
||||
version and collaborate — reinforcing **files-canonical + git coordination** as the
|
||||
durable substrate; the live environment sits above it.
|
||||
|
||||
## 5. UC disposition (enrichment-only — no new UC)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC / thread |
|
||||
|------------------------|---------------------|
|
||||
| Live-object inspector = generic ancestor of moldable views (§1) | links UC-54, UC-47/48 (T7) |
|
||||
| Image = opaque monolithic non-diffable blob; not a page/store (§2) | **boundary** for UC-34/UC-35/UC-79 (granularity, identity, files-canonical) |
|
||||
| Image participates only via export→files = degrading derivation-projection (§2) | links UC-83, UC-84 (live→snapshot) |
|
||||
| Pharo Tonel/Iceberg: code-as-text in git (§3) | links UC-79, UC-76 (git-canonical text) |
|
||||
| `.changes` = serial source-change log, not content history (§1) | links UC-36 (history shape) |
|
||||
|
||||
Both Squeak and Pharo are **design prior art / boundary**, not candidate shards → **no new
|
||||
UC**. They sharpen the **live↔snapshot** axis and harden the **image-is-not-a-store**
|
||||
boundary; Pharo confirms even image traditions externalize to git text.
|
||||
|
||||
## 6. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T14 (attach binding):** record the **image-is-not-a-store** boundary explicitly — a
|
||||
monolithic live-memory blob is never an attach target; participation is via **export→files**
|
||||
only. Generalize the "attach files, not the kernel/image" rule (UC-84, GT T7) to a named
|
||||
boundary in the binding taxonomy.
|
||||
- **T16 (projection):** add the **live↔snapshot** axis to the projection model — every
|
||||
computed/projected view sits somewhere between "live (re-derivable on demand)" and "static
|
||||
snapshot," and the more live the source, the more its attached form must be a clearly-
|
||||
marked degrading snapshot.
|
||||
- **T11/T12:** the live-object inspector is the *inspiration* for the moldable view registry
|
||||
(T7), not a storage shape; nothing new to the page model itself.
|
||||
|
||||
## 7. Open questions
|
||||
|
||||
1. Is **live↔snapshot** an explicit, first-class metadata axis on every projection (so the
|
||||
union can label "this view was live / is a snapshot from time T"), or implicit per
|
||||
capability? (Recurs across UC-83/84, GT, Mathematica `Dynamic`, Strudel T5.)
|
||||
2. Do we ever ingest a Smalltalk project by attaching its **Tonel/git** repo as an ordinary
|
||||
git-text shard (no Smalltalk-specific adapter needed), confirming the boundary resolution?
|
||||
|
||||
## 8. Sources
|
||||
|
||||
- Squeak: `squeak.org`, the *Back to the Future* (Squeak) paper (Ingalls et al.), Morphic;
|
||||
image/`.changes` model.
|
||||
- Pharo: `pharo.org`, **Tonel** format, **Iceberg** (git integration), FileTree.
|
||||
- prior: `research/260614-glamorous-toolkit-deep-dive/` (moldable inspector, Lepiter, T7);
|
||||
`research/260614-jupyter-deep-dive/` (live→snapshot boundary, UC-84).
|
||||
|
||||
## 9. Traceability
|
||||
|
||||
**No new UC** (boundary / design prior art; covers **T6 Squeak** and **T8 Pharo** in one
|
||||
justified-merge memo). Boundary hardened for: UC-34/UC-35/UC-79 (image-is-not-a-store);
|
||||
links UC-83/UC-84 (live→snapshot), UC-54/UC-47/UC-48 (live-object inspection → moldable
|
||||
views), UC-76/UC-79 (Pharo Tonel/git text), UC-36 (`.changes` history shape). Architecture
|
||||
cross-refs: SHARD-WP-0002 T14 (image-is-not-a-store boundary; export→files only), T16
|
||||
(live↔snapshot projection axis).
|
||||
30
research/260614-strudel-deep-dive/README.md
Normal file
30
research/260614-strudel-deep-dive/README.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# 260614 — Strudel.cc (live-coding REPL) deep dive
|
||||
|
||||
Date: 2026-06-14 · Source: **SHARD-WP-0004 T5**
|
||||
|
||||
## What this is
|
||||
|
||||
A deep dive into **Strudel.cc** (the JavaScript port of **TidalCycles**): a browser
|
||||
**live-coding REPL** where terse **pattern source** is **evaluated live into time-based
|
||||
audio** — code as a running musical performance, with **no document, no output cell, no file
|
||||
of results**.
|
||||
|
||||
## Why it matters
|
||||
|
||||
- The **extreme of the live↔snapshot axis** (named at T6): output is **temporal, generative,
|
||||
performative**, so there is **no faithful static form** — the best static projection is
|
||||
**source (canonical, diffable) + an optional audio recording snapshot**, marked as one
|
||||
performance. The **honesty test** for union-without-erasure + graceful degradation.
|
||||
- Bounds the projection model's live end: ahead-of-time → view-time one-shot → continuous →
|
||||
**irreducibly live/temporal (recording-only)**.
|
||||
|
||||
## Yield
|
||||
|
||||
- **No new UC** (enrichment / design prior art; far live end). Enriches **UC-54, UC-55**;
|
||||
links UC-83/84, UC-37, UC-35.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Code-as-live-performance, the limit of static projection, INTENT mapping, UC disposition (enrichment-only), architecture notes (T16 far end of live↔snapshot axis) |
|
||||
118
research/260614-strudel-deep-dive/findings.md
Normal file
118
research/260614-strudel-deep-dive/findings.md
Normal file
@@ -0,0 +1,118 @@
|
||||
# Strudel.cc — live-coding REPL — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T5 · **Subject:** Strudel.cc (the
|
||||
JavaScript port of TidalCycles) — a browser **live-coding REPL** where code is a **running
|
||||
musical performance**.
|
||||
|
||||
## Why this dive
|
||||
|
||||
The closing, lowest-priority dive — and the **extreme** of the live↔snapshot axis. Where
|
||||
Processing (T4) renders *visual* output at view time, Strudel renders **time-based audio**
|
||||
that only exists *while running and evolving*. It is the limit case for "can a page be a live
|
||||
computation?" — the answer where **static projection is least faithful**. Enrichment-only;
|
||||
feeds UC-54/55 and hardens the snapshot-degradation rule.
|
||||
|
||||
## 1. Code as a live, evaluated performance
|
||||
|
||||
- **Strudel** ports **TidalCycles**' pattern language to JavaScript, running entirely in the
|
||||
**browser** (Web Audio). You write **pattern expressions** (e.g. `note("c e g")`,
|
||||
`sound("bd sd")` with transformations) and **evaluate them live**; the sound updates
|
||||
**without stopping** — the essence of *live coding*.
|
||||
- The artifact is **terse source text** (a pattern); the "content" is the **sound it
|
||||
produces over time**. There is **no document, no output cell, no file of results** —
|
||||
output is **ephemeral, temporal, and performative**.
|
||||
- A Strudel "page" (a shared REPL link / snippet) is **source + the implicit promise of a
|
||||
running evaluation**. The source is tiny and diffable; the experience is not capturable as
|
||||
text.
|
||||
|
||||
## 2. The limit of static projection
|
||||
|
||||
Strudel pushes past Processing on every "live" dimension:
|
||||
|
||||
- **Temporal & generative** — output unfolds over time and may be **non-deterministic**
|
||||
(randomness, evolving state). There is no single "frame"; the faithful capture is a
|
||||
**recording (audio), itself just one performance**, not the artifact.
|
||||
- **Performative** — the value is partly the **act of live editing**; even a recording loses
|
||||
the live-coding dimension.
|
||||
- So on the **live↔snapshot axis** (named at T6), Strudel sits at the **far live end**: the
|
||||
best static projection is **(a) the source** (canonical, diffable) **+ (b) an optional
|
||||
audio recording snapshot**, explicitly marked as one rendering of a live/temporal artifact.
|
||||
|
||||
This makes Strudel the **honesty test** for the contract: shard-wiki must be able to attach
|
||||
such a source, present it truthfully (here is the source; a live render needs the runtime; a
|
||||
recording is one performance), and **never imply a static page captures it**.
|
||||
|
||||
## 3. INTENT mapping (enrichment-only — no new UC)
|
||||
|
||||
### Reinforcements / refinements
|
||||
|
||||
- **Live-evaluated, time-based content (UC-54/55).** Strudel is the extreme executable-as-
|
||||
page: **source canonical, presentation = a temporal live evaluation**. Confirms the page
|
||||
model must represent content whose rendered form is **time-based / generative / performative**.
|
||||
- **live↔snapshot axis (T6) — far end.** Establishes the **upper bound**: some content is
|
||||
**irreducibly live**; static projection degrades to **source + a recording snapshot**, with
|
||||
honesty about what's lost. Generalizes Processing's "snapshot frame" to "recording of one
|
||||
performance."
|
||||
- **Graceful degradation (INTENT).** A backend that can't run the REPL still serves the
|
||||
**source** (tiny, diffable) and any **recording** as read/projection/backup — the
|
||||
limited-backend-still-usable rule, at the hardest content type.
|
||||
- **Union without erasure.** Presenting a Strudel shard must surface "this is a **live
|
||||
temporal artifact**; what you see/hear statically is **source / one recording**" — never
|
||||
hide the liveness or imply completeness.
|
||||
|
||||
### Boundaries
|
||||
|
||||
- shard-wiki is **not an audio/REPL runtime**; default = attach the **source** + offer/store
|
||||
a **recording** with provenance; live in-viewer evaluation is a gated capability (trust/
|
||||
sandbox, like Processing T4). Source is canonical; everything rendered is a degrading,
|
||||
view-time/temporal projection.
|
||||
|
||||
## 4. UC disposition (enrichment-only)
|
||||
|
||||
| Mechanism (findings §) | Catalog UC / thread |
|
||||
|------------------------|---------------------|
|
||||
| Pattern source = live, evaluated, time-based performance (§1) | UC-54 / UC-55 (enriched: time-based/generative executable content) |
|
||||
| Output ephemeral/temporal/non-deterministic → no faithful static form (§2) | links live↔snapshot axis (T6), far end |
|
||||
| Best static projection = source + audio recording snapshot, marked as one performance (§2) | links UC-83/UC-84 (degrading projection), UC-37 (recording as backup) |
|
||||
| Limited backend still serves source + recording (§3) | links UC-37 graceful degradation |
|
||||
| Live in-viewer evaluation = capability + trust/sandbox (§3) | links UC-35 |
|
||||
|
||||
**No new UC** — Strudel is design prior art marking the **far live end** of the projection/
|
||||
liveness model; it adds no orchestration scenario, it bounds one.
|
||||
|
||||
## 5. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T16 (projection):** anchor the **far end of the live↔snapshot axis** — content can be
|
||||
**irreducibly live/temporal/generative**; the contract must allow a projection to declare
|
||||
"no faithful static form; static = source + a marked recording." Combined with T4's
|
||||
**materialization-timing** and **continuity** facets, the projection model now spans:
|
||||
ahead-of-time materialized → view-time one-shot → view-time continuous/interactive →
|
||||
**temporal/generative/performative (recording-only snapshot)**.
|
||||
- **T12/T15 (page model):** **time-based / generative executable content** as a page-model
|
||||
edge; source canonical, render temporal.
|
||||
- **T11 (capabilities):** "live-evaluate (audio/REPL) in viewer" capability + trust/sandbox;
|
||||
default off → source + recording.
|
||||
|
||||
## 6. Open questions
|
||||
|
||||
1. Does the contract carry an explicit **"irreducibly live / no faithful static form"** flag
|
||||
on a projection (so the union renders the honest fallback automatically)? (The far-end
|
||||
resolution of the live↔snapshot axis open question from T6.)
|
||||
2. Is a **recording** modeled as a stored **derivation-projection snapshot with provenance**
|
||||
(one performance, time T, source rev R), reusing the UC-84 snapshot machinery?
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- `strudel.cc` (docs, REPL); **TidalCycles** (`tidalcycles.org`) — the pattern language
|
||||
Strudel ports; Web Audio.
|
||||
- prior: `research/260614-processing-deep-dive/` (view-time render, continuity, UC-54/55);
|
||||
`research/260614-squeak-pharo-deep-dive/` (live↔snapshot axis);
|
||||
`research/260614-mathematica-deep-dive/` (`Dynamic` interactive, snapshot-only).
|
||||
|
||||
## 8. Traceability
|
||||
|
||||
**No new UC** (enrichment / design prior art; the far live end). Enriched: UC-54, UC-55;
|
||||
links UC-83/UC-84 (degrading projection), UC-37 (recording = backup / graceful degradation),
|
||||
UC-35 (live-evaluate capability + trust). Architecture cross-refs: SHARD-WP-0002 T16 (far end
|
||||
of live↔snapshot axis: irreducibly-live content, static = source + marked recording), T12/T15
|
||||
(time-based/generative executable content), T11 (live-evaluate capability + sandbox).
|
||||
16
research/260614-tiddlywiki-deep-dive/README.md
Normal file
16
research/260614-tiddlywiki-deep-dive/README.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# 260614 — TiddlyWiki deep dive
|
||||
|
||||
Deep dive on **TiddlyWiki** (TW5): an entire wiki — tiddlers **plus** the app engine — in
|
||||
**one self-contained HTML file**, the **whole-file write-granularity** anchor of the
|
||||
synthesis matrix, with a Node.js **file-per-tiddler** (`.tid`) substrate as the git-diffable
|
||||
alternative, a tiddler/field record model, and **filter expressions** as the native query
|
||||
language.
|
||||
|
||||
- `findings.md` — the single-file model, tiddler data model, dual substrate, filters,
|
||||
capability profile, INTENT mapping, UC seed (UC-78), architecture notes for SHARD-WP-0002,
|
||||
open questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-78 (attach a single-file self-contained wiki as one shard — parse tiddlers
|
||||
out, project; write = rewrite the whole file, the coarsest write-granularity anchor).
|
||||
Enriched UC-35/40/34/52/43. Feeds SHARD-WP-0002 T11 (write-granularity extreme) and T14
|
||||
(single-file vs file-per-tiddler binding).
|
||||
164
research/260614-tiddlywiki-deep-dive/findings.md
Normal file
164
research/260614-tiddlywiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,164 @@
|
||||
# TiddlyWiki — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T3 · **Subject:** TiddlyWiki / TiddlyWiki5
|
||||
(TW5), Jeremy Ruston's self-contained personal wiki.
|
||||
|
||||
## Why this dive
|
||||
|
||||
The synthesis matrix names **whole-file write granularity** as one extreme of the
|
||||
write-granularity spectrum, anchored by TiddlyWiki. This dive confirms the anchor and the
|
||||
*portability extreme* it implies — a wiki that is **a single HTML file you can email** — and
|
||||
finds the twist: TiddlyWiki also has a **Node.js file-per-tiddler** substrate, so it spans
|
||||
the granularity spectrum the way Logseq spans file/DB (UC-62). The question: how does
|
||||
shard-wiki attach a backend whose *entire content is one file*?
|
||||
|
||||
## 1. The single-file model
|
||||
|
||||
Classic TiddlyWiki ships as **one `.html` file** that contains **both**:
|
||||
|
||||
1. the **TiddlyWiki core** (the JavaScript engine, parser, renderer, UI), and
|
||||
2. **every tiddler** (all content), serialized into the file.
|
||||
|
||||
Open it in a browser and the file *is* the running application. There is **no server and no
|
||||
build step** — the app reconstitutes itself from the file it was loaded from. This is the
|
||||
portability extreme: a complete, self-hosting wiki in a single, emailable, USB-stick-able
|
||||
artifact that runs offline anywhere.
|
||||
|
||||
**Saving** is the catch: a browser page cannot normally overwrite the file it came from, so
|
||||
TiddlyWiki uses **"savers"** — TiddlyFox/browser extension, the File System Access API, a
|
||||
Node.js server, TiddlySpot/put-savers, or "download a new copy." Crucially, **a save
|
||||
rewrites the *entire* HTML file** (core + all tiddlers re-serialized). Hence **whole-file
|
||||
write granularity**: there is no concept of writing one page in isolation in the
|
||||
single-file mode — every save touches the whole artifact.
|
||||
|
||||
## 2. The tiddler data model
|
||||
|
||||
The atomic unit is the **tiddler** — a named record with **fields**:
|
||||
|
||||
- Core fields: **`title`** (the identity), **`text`** (the body), **`tags`**, **`created`**,
|
||||
**`modified`**, **`type`** (content type, e.g. `text/vnd.tiddlywiki`, `text/markdown`),
|
||||
plus **arbitrary custom fields** (any key→value). A tiddler is effectively a **flexible
|
||||
flat record** — closer to a typed-field record (UC-34) than to prose-with-frontmatter.
|
||||
- **Everything is a tiddler**: not just pages, but tags, macros, templates, themes, plugins,
|
||||
and the wiki's own configuration are all tiddlers. A **plugin is a bundle of tiddlers**.
|
||||
- Content markup is **WikiText** (TW5's own), though `type` can mark a tiddler as Markdown,
|
||||
JSON, image, etc. **Transclusion** is native: `{{SomeTiddler}}` embeds another tiddler;
|
||||
`{{SomeTiddler!!field}}` embeds a field.
|
||||
|
||||
## 3. The dual substrate — single-file vs file-per-tiddler
|
||||
|
||||
TiddlyWiki on **Node.js** stores each tiddler as a **separate `.tid` file** on disk: a small
|
||||
text file with a header of `field: value` lines, a blank line, then the body. The Node
|
||||
server assembles these into the same wiki at serve time. This substrate is:
|
||||
|
||||
- **git-diffable and fine-grained** — one file per tiddler, line-level diffs, per-tiddler
|
||||
history — the *opposite* end of the granularity spectrum from the single HTML file.
|
||||
- the natural attach surface for a *versioned, multi-author* TiddlyWiki.
|
||||
|
||||
So TiddlyWiki **spans the write-granularity spectrum by substrate** (single-file = whole-file
|
||||
write; Node = file-per-tiddler write), exactly as Logseq spans file/DB (UC-62) and as the
|
||||
backend-swap question (UC-43) anticipates.
|
||||
|
||||
## 4. Native query — filter expressions
|
||||
|
||||
TiddlyWiki's query language is **filter expressions** over tiddler fields, e.g.
|
||||
`[tag[todo]!tag[done]sort[modified]]` — a compact DSL that selects/orders tiddlers by field
|
||||
and tag. Lists, tables, and dynamic views are built from filters. This is a **native-query
|
||||
capability** (UC-52 tier) — less expressive than SPARQL/datalog but real, and computed over
|
||||
the tiddler store.
|
||||
|
||||
## 5. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | TiddlyWiki (single-file) | TiddlyWiki (Node `.tid`) |
|
||||
|--------------------------------|--------------------------|--------------------------|
|
||||
| Attachment mode | **file-store: one HTML file** | **file-store: dir of `.tid` files** |
|
||||
| Addressing granularity | tiddler (`title`) within the file | tiddler = one file |
|
||||
| Content identity | **`title`** (placement-bound) | title ↔ filename |
|
||||
| Structure | flat record store w/ arbitrary **fields** + tags | same |
|
||||
| History | none in-file (whole-file save) | **per-file git history** |
|
||||
| Merge model | whole-file replace (no merge) | git 3-way per tiddler |
|
||||
| Native query | **filter expressions** | filter expressions |
|
||||
| Translation | WikiText (or per-tiddler `type`: markdown/json/…) | same |
|
||||
| **Write granularity** | **whole file** (the anchor) | **file per tiddler** |
|
||||
| Operational envelope | trivial — a browser; no server | a Node server |
|
||||
| Access grant | file access = full access | server/file perms |
|
||||
| Content opacity | transparent (parse the HTML store) | transparent text |
|
||||
| Provenance | created/modified fields | git + fields |
|
||||
|
||||
## 6. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Graceful degradation**: a single-file TiddlyWiki is a *trivial* read-only / projection /
|
||||
backup shard — parse the tiddlers out of the HTML, project pages; no server needed. The
|
||||
limited-backend-still-usable principle at its simplest.
|
||||
- **Markdown-first but backend-neutral**: tiddlers carry a `type`, so Markdown tiddlers
|
||||
coexist with WikiText — the page model's content-type field maps directly.
|
||||
- **Typed fields** (UC-34): arbitrary tiddler fields are a flexible record model the page
|
||||
model already accommodates.
|
||||
- **Backend-swap under stable identity** (UC-43): single-file ↔ Node `.tid` is the same
|
||||
logical wiki on two substrates — the migration UC-43 anticipates, within one engine.
|
||||
|
||||
### Divergences (boundaries / notes)
|
||||
|
||||
- **Whole-file write granularity** is a real constraint: in single-file mode shard-wiki
|
||||
cannot write one page atomically — an overlay applied "to one page" still **rewrites the
|
||||
whole file** (T11). This is the coarsest write tier; model it explicitly so overlays/locks
|
||||
account for it (a write to any page conflicts with any concurrent write).
|
||||
- **Identity = title**, file-local; cross-shard identity (T16) layered above.
|
||||
- **The app is in the file**: when parsing a single-file TiddlyWiki, shard-wiki must extract
|
||||
the **tiddler store** and ignore the embedded engine — i.e. treat the HTML as a *container
|
||||
format*, not as page content (don't mistake the app for content).
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Single-file self-contained wiki as a first-class file-store shard** — container-format
|
||||
parse, whole-file write granularity (UC-78); the portability/granularity extreme.
|
||||
2. **Whole-file write granularity as a named tier** (T11) with overlay/lock implications.
|
||||
3. **Dual-substrate binding** (single-file vs `.tid` dir) as another instance of
|
||||
substrate-choice under one identity (UC-43/UC-62).
|
||||
|
||||
## 7. UC seed
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-78 | Attach a **single-file self-contained wiki** (TiddlyWiki HTML) as one shard — parse tiddlers out of the container, project pages; **write = rewrite the whole file** (whole-file write granularity, the coarsest tier) | **new** |
|
||||
| — | whole-file write granularity anchor + overlay/lock implications | enrich **UC-35** |
|
||||
| — | single HTML file as a file-store shard (container format) | enrich **UC-40** |
|
||||
| — | tiddler arbitrary fields = flexible record | enrich **UC-34** |
|
||||
| — | filter expressions as a native-query tier | enrich **UC-52** |
|
||||
| — | single-file ↔ Node `.tid` substrate swap | enrich **UC-43** |
|
||||
|
||||
## 8. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T11 (capability / write granularity):** confirm **whole-file** as the coarsest named
|
||||
write tier (anchored by single-file TiddlyWiki), with the implication that an overlay to
|
||||
*any* page conflicts with concurrent writes (no per-page atomicity). File-per-tiddler is
|
||||
the fine tier on the same engine.
|
||||
- **T14 (attach binding):** a single-file wiki binds as a **container-format file-store**
|
||||
(parse tiddler store, ignore embedded engine); a Node TiddlyWiki binds as a **dir of
|
||||
`.tid` files** (git-diffable). One engine, two bindings — parameterize like UC-43.
|
||||
- **Native query:** filter expressions are a low-mid native-query tier between "none" and
|
||||
datalog/SPARQL — delegate where present (UC-52).
|
||||
|
||||
## 9. Open questions
|
||||
|
||||
1. In single-file mode, how does shard-wiki represent **per-page overlays** when writes are
|
||||
whole-file — buffer overlays and re-serialize, or require the Node `.tid` substrate for
|
||||
write-through and treat single-file as read/projection/backup only?
|
||||
2. Is a single-file TiddlyWiki's **embedded plugins/config** ever relevant to the union, or
|
||||
strictly ignored as app-internals (parse only content tiddlers)?
|
||||
3. Does shard-wiki expose tiddler **filter expressions** as a delegated query, or only its
|
||||
own union query over projected tiddlers?
|
||||
|
||||
## 10. Sources
|
||||
|
||||
- TiddlyWiki.com — *Tiddlers*, *TiddlerFields*, *Filters*, *Saving*, *Node.js* docs
|
||||
- *TiddlyWiki5* GitHub (Jermolene/TiddlyWiki5) — `.tid` file format, store structure
|
||||
- prior: `research/260614-logseq-deep-dive/` (file/DB dual substrate, UC-62)
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
New UC **UC-78** carries the marker **⊡** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-35, UC-40, UC-34, UC-52, UC-43. Architecture
|
||||
cross-refs: SHARD-WP-0002 T11 (whole-file tier), T14 (dual binding).
|
||||
53
research/260614-trilium-deep-dive/README.md
Normal file
53
research/260614-trilium-deep-dive/README.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# 260614 — Trilium (TriliumNext) deep dive (note cloning, attribute inheritance, HTML-native)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Trilium / TriliumNext** — the open-source hierarchical personal
|
||||
knowledge base — read through shard-wiki's lens. It is another SQLite-local note app, but
|
||||
it brings a structural feature **none of the prior twelve systems had: note cloning** —
|
||||
a single note can sit in **multiple places in the tree at once**, so the hierarchy is a
|
||||
**DAG, not a tree**, with **note identity cleanly separated from placement** (a note has
|
||||
many "branches"). It also has an **attribute system with inheritance + templates**
|
||||
(computed metadata) and is one of the few **HTML-native** (not Markdown) tools.
|
||||
|
||||
Lineage: **TriliumNext** is the community fork of the original **Trilium** (zadam, now
|
||||
maintenance-only) — the same pattern as TWiki→Foswiki.
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — single **SQLite** file (`document.db`); desktop or self-host server;
|
||||
multi-instance **sync protocol w/ conflict resolution** + WebSocket; 12-char IDs
|
||||
(`noteId`/`branchId`/`attributeId`/`attachmentId`)
|
||||
- **Note cloning** — note vs **branch** (identity vs placement); DAG hierarchy; no single
|
||||
canonical path
|
||||
- **Attributes** — labels (`#tag`) + typed relations (`~relation`), **inheritable** +
|
||||
**templates** (`~template`) → effective vs own metadata
|
||||
- **Content** — **HTML** (CKEditor5), plus canvas/mindmap/spreadsheet/geo/render note
|
||||
types; export to MD/HTML (lossy)
|
||||
- **Extension** — **scripting** (frontend/backend code notes, Script API) + **ETAPI**
|
||||
(token REST); **per-note encryption** (protected notes)
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, note cloning/DAG, attributes+inheritance, HTML content, scripting/ETAPI, per-note encryption, capability profile, INTENT mapping, UC seeds, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-66
|
||||
attach a shard with a DAG hierarchy / note cloning, identity separated from placement;
|
||||
UC-67 preserve inherited/templated attributes — effective vs own metadata); UC-15/22/34/
|
||||
38/42/61 enriched. Logged for `SHARD-WP-0002` (T11/T12/T14/T15/T16): DAG namespace +
|
||||
identity/placement (note/branch) split, computed/inherited metadata in the page model,
|
||||
**per-item** content opacity (refining the 12th spectrum), HTML as a source content model,
|
||||
and scripting + ETAPI host surfaces.
|
||||
|
||||
**Key takeaways recorded:** Trilium's **note/branch split (identity ≠ placement)** is the
|
||||
clean model for pages that live in multiple locations/shards and the namespace-level form
|
||||
of the clone/reference primitive; metadata is **computed** (own + inherited + template),
|
||||
not a flat bag; and **content opacity is per-item** (per-note encryption), not only
|
||||
whole-shard. **Boundary:** one SQLite-local candidate shard (DAG hierarchy, HTML-native),
|
||||
best attached via ETAPI; not a substrate and not the federation layer.
|
||||
</content>
|
||||
274
research/260614-trilium-deep-dive/findings.md
Normal file
274
research/260614-trilium-deep-dive/findings.md
Normal file
@@ -0,0 +1,274 @@
|
||||
# Findings — Trilium (TriliumNext): note cloning, attribute inheritance, HTML-native
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — an open-source hierarchical PKB; a *candidate
|
||||
shard* whose distinctive traits are **note cloning (a DAG hierarchy)**, an
|
||||
**attribute/relation system with inheritance + templates**, and **HTML-native** content
|
||||
Lens: shard-wiki — the namespace model (identity vs placement), inherited/computed
|
||||
metadata, HTML translation, per-note encryption, and the scripting/ETAPI surfaces
|
||||
|
||||
> Why Trilium earns a dive. It is another SQLite-local note app, but it brings a
|
||||
> structural feature **none of the prior twelve systems had: note cloning** — a single
|
||||
> note can sit in **multiple places in the tree at once**, so the hierarchy is a **DAG,
|
||||
> not a tree**, and **note identity is cleanly separated from placement** (a note has
|
||||
> many "branches"). That directly challenges shard-wiki's namespace model (UC-22 assumes
|
||||
> a path) and is the clone/reference primitive made concrete at the *namespace* level.
|
||||
> Trilium also has an **attribute system with inheritance and templates** (effective
|
||||
> metadata is computed, not just stored), and it is one of the few **HTML-native** (not
|
||||
> Markdown) tools — a useful stress on "Markdown-first must degrade."
|
||||
|
||||
Lineage note (like TWiki→Foswiki): **TriliumNext** is the community fork of the original
|
||||
**Trilium** (zadam) after it went maintenance-only; the active project is TriliumNext.
|
||||
|
||||
Contrast set: Joplin (SQLite-local, files-on-sync, page-level), Logseq (block-graph on
|
||||
files), Notion (hosted DB, schema+relations). Trilium = **SQLite-local, DAG hierarchy,
|
||||
inherited attributes, HTML content, self-host sync + ETAPI**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture — one SQLite file, server-syncable
|
||||
|
||||
- **Storage:** a single **SQLite** file (`document.db`, via better-sqlite3) holding
|
||||
notes, attachments, history, and settings. Local DB store (like Joplin; not files).
|
||||
- **Clients/sync:** desktop (Electron) or a self-hosted **server**; multi-instance
|
||||
**sync protocol** with **conflict resolution + entity change tracking**, plus
|
||||
**WebSocket** realtime updates. Attach via the server, the ETAPI (§5), or the DB.
|
||||
- **IDs:** 12-char IDs — **`noteId`, `branchId`, `attributeId`, `attachmentId`**. The
|
||||
**note vs branch** split is the key idea (§2).
|
||||
- **Export:** to Markdown/HTML (lossy — content is HTML, §4).
|
||||
|
||||
---
|
||||
|
||||
## 2. Note cloning — a DAG hierarchy, identity separated from placement
|
||||
|
||||
Notes form an arbitrarily deep tree, **but a single note can be placed in multiple
|
||||
locations** — "cloning," implemented via the **branches** model:
|
||||
|
||||
- A **note** (`noteId`) is the content + identity. A **branch** (`branchId`) is *one
|
||||
placement* of that note under a parent (with an optional **branch prefix** giving
|
||||
per-location context). A note with several branches is **cloned** — it appears in
|
||||
several places at once; editing it anywhere edits the one note.
|
||||
- So the hierarchy is a **DAG**, not a tree, and **identity (note) is separated from
|
||||
location (branch)**. There is no single canonical path for a cloned note.
|
||||
|
||||
This is the most shard-wiki-relevant feature in the dive:
|
||||
|
||||
- It breaks the assumption behind **UC-22** (resolve a page by *a* path). shard-wiki's
|
||||
namespace model must allow **a page in multiple namespace locations** with no single
|
||||
canonical path (UC-66).
|
||||
- The **note/branch split** is a clean model shard-wiki should borrow: **page identity ≠
|
||||
page placement**. A page is one entity; its locations are separate placement records —
|
||||
exactly how shard-wiki should treat a page that appears under multiple paths/shards.
|
||||
- It is the **clone/reference-not-copy primitive** (Xanadu clone, ZigZag clone, UC-44/45,
|
||||
T16) realized at the *namespace* level: same note, many positions, one source of truth.
|
||||
|
||||
---
|
||||
|
||||
## 3. Attributes — labels + relations, with inheritance and templates
|
||||
|
||||
Trilium's metadata system is **attributes**:
|
||||
|
||||
- **Labels** (`#tag`, optionally `#key=value`) — typed metadata on a note.
|
||||
- **Relations** (`~relation`) — **typed links to other notes** (a knowledge graph).
|
||||
- **Inheritance** — attributes can be **inherited down the subtree** (inheritable
|
||||
attributes), and **templates** (a note whose attributes/structure are applied to
|
||||
instances via a `~template` relation) inject attribute sets. So a note's **effective
|
||||
metadata = its own attributes + inherited + templated** — computed, not just stored.
|
||||
- Promoted attributes give a form-like UI; attributes drive **search/queries** and
|
||||
**scripting**.
|
||||
|
||||
shard-wiki read: this is structured data (UC-34) and typed relations (UC-58), but with a
|
||||
**new wrinkle — inheritance/templates make metadata *computed***. Projecting such a shard
|
||||
must distinguish **effective vs own** attributes (and record their provenance: own /
|
||||
inherited-from / template), not flatten them (UC-67). Templates also reinforce blueprint
|
||||
pages (UC-15).
|
||||
|
||||
---
|
||||
|
||||
## 4. Content model — HTML-native, many note types
|
||||
|
||||
- **Text notes are HTML** (WYSIWYG via **CKEditor5**), **not Markdown**. Trilium is one
|
||||
of the few HTML-first tools studied. Markdown participation therefore needs **HTML↔
|
||||
Markdown translation** — more tractable than Notion's block model but still **lossy**
|
||||
for some constructs (CKEditor features, includes) → a fidelity-aware case bridging
|
||||
UC-42 (lossless ideal) and UC-59 (lossy-with-report).
|
||||
- **Note types:** text (HTML), code (CodeMirror), **canvas (Excalidraw)**, relation
|
||||
maps, mind maps (Mind Elixir), spreadsheets (Univer), geo maps (Leaflet), **render
|
||||
notes**, file/image, book — i.e. lots of **non-Markdown content** (UC-55) and
|
||||
**script/render-generated dynamic notes** (UC-54).
|
||||
|
||||
---
|
||||
|
||||
## 5. Extension surfaces — scripting (in-app) + ETAPI (external)
|
||||
|
||||
Two surfaces, like Joplin:
|
||||
|
||||
- **Scripting (in-engine host).** **Code notes** in JS run as **frontend** (UI widgets,
|
||||
buttons) or **backend** (server-side automation) scripts against a **Script API**
|
||||
(create/query notes, attributes, etc.). Executable, in-app — the adapter-host path
|
||||
(UC-38). "Render notes" + scripts produce **dynamic, generated content** (UC-54).
|
||||
- **ETAPI (external REST API).** Trilium's public REST API (since v0.50), **token auth**
|
||||
(`Bearer ETAPITOKEN`), with client libs (`trilium-py`): CRUD over notes/branches/
|
||||
attributes/attachments, search, import/export — an external attach surface (UC-57)
|
||||
that, unlike Joplin's localhost-only Data API, is the **designed external integration
|
||||
API** for a (often self-hosted) server.
|
||||
|
||||
---
|
||||
|
||||
## 6. Security — per-note (partial) encryption
|
||||
|
||||
Trilium offers **strong per-note encryption** ("protected notes") unlocked in a
|
||||
**protected session** (password; TOTP/OpenID for login). Crucially this is **per-note**:
|
||||
a shard can hold **some encrypted and some plaintext notes at once**. This refines the
|
||||
**content-opacity** dimension (UC-61, proposed twelfth spectrum from Joplin/Anytype):
|
||||
opacity is **per-item, not only whole-shard** — the adapter must handle a shard where
|
||||
*part* of the content is opaque without a key (protected notes projectable only as
|
||||
structure-shell; unprotected notes fully).
|
||||
|
||||
---
|
||||
|
||||
## 7. Trilium as a shard — capability profile
|
||||
|
||||
| Capability | Trilium | Notes for the adapter contract |
|
||||
|------------|---------|--------------------------------|
|
||||
| Read | yes (ETAPI / sync / DB) | ETAPI REST (server) is the clean surface |
|
||||
| Write | yes (ETAPI / script) | per-note; via ETAPI or backend code notes |
|
||||
| Write granularity | per-note (page) | — |
|
||||
| Identity / addressing | **`noteId` + `branchId`** (12-char) | **identity ≠ placement**; cloned notes have many branches (UC-51, UC-66) |
|
||||
| Hierarchy | **DAG (cloning)** | a note in multiple locations; no single canonical path (UC-66) |
|
||||
| Structure | **labels + relations, inherited + templated** | effective vs own metadata (UC-34, UC-58, UC-67) |
|
||||
| Content | **HTML (CKEditor)** + many types | HTML↔MD lossy translation (UC-42/59); non-MD types (UC-55) |
|
||||
| History | internal revisions in the DB | not portable git → supplement (UC-36) |
|
||||
| Native query | attribute search / script queries | delegate or build index (UC-52/63) |
|
||||
| Subscribe | WebSocket + sync protocol | push; conflict resolution built in (UC-31) |
|
||||
| Content opacity | **per-note encryption (partial)** | per-item opacity (UC-61 refined) |
|
||||
| Extension | **scripting (in-app) + ETAPI (external)** | dual surface (UC-38, UC-57) |
|
||||
| Templates | yes (`~template`) | blueprint pages (UC-15) |
|
||||
|
||||
Verdict: a capable **SQLite-local, server-syncable** shard best attached via **ETAPI**.
|
||||
Standout demands: the **DAG hierarchy / note-branch identity model** (UC-66) and
|
||||
**inherited/templated attributes** (UC-67); plus HTML-native translation, per-note
|
||||
opacity, and the scripting host.
|
||||
|
||||
---
|
||||
|
||||
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 8.1 Reinforcements
|
||||
|
||||
- **Identity ≠ placement** (note/branch) is a model shard-wiki should adopt for pages
|
||||
that appear under multiple paths or in multiple shards — provenance and union without
|
||||
erasure depend on separating "what a page is" from "where it sits."
|
||||
- **Typed relations + templates** validate the structured/relation demand (UC-58) and
|
||||
blueprints (UC-15) on a local-first, self-hostable backend.
|
||||
- **ETAPI** is a clean example of a **designed external REST surface** for a self-hosted
|
||||
server (cf. Notion external API, but self-hosted/trust-boundary friendly).
|
||||
|
||||
### 8.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **Don't force a single canonical path.** A cloned note has many placements; modeling
|
||||
it with one path loses information (UC-66). Use the note/branch separation.
|
||||
2. **Don't flatten computed metadata.** Effective attributes include inherited/templated
|
||||
values; record provenance (own vs inherited vs template), don't collapse (UC-67).
|
||||
3. **HTML-native, not Markdown.** Translate HTML↔Markdown with a fidelity report; degrade
|
||||
to read-only where lossy (UC-42/59/03). Markdown-first must degrade gracefully.
|
||||
4. **Per-note opacity.** Some notes are encrypted; never surface protected-note
|
||||
ciphertext; project them as structure-shell (UC-61).
|
||||
5. **One Trilium instance = one shard**, not the federation layer; its sync protocol is
|
||||
its own — attach via ETAPI/replica, don't re-drive it (not-a-sync-daemon).
|
||||
|
||||
### 8.3 What Trilium teaches that shard-wiki should keep
|
||||
|
||||
- **Separate page identity from placement** (note vs branch) — the cleanest model for
|
||||
multi-location / multi-shard pages and for the clone/reference primitive (T16).
|
||||
- **Model metadata as computed** (own + inherited + templated) with per-attribute
|
||||
provenance (UC-67) — not a flat key/value bag.
|
||||
- **Content opacity is per-item**, not only whole-shard (UC-61 refinement).
|
||||
|
||||
---
|
||||
|
||||
## 9. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-65**. New UCs **UC-66, UC-67** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach a shard with a DAG hierarchy / note cloning** — a page may occupy multiple namespace locations at once (note identity separated from placement via branches); no single canonical path | **UC-66 (new)** |
|
||||
| **Preserve inherited / templated attributes** — project a structured shard whose metadata is computed (own + inherited + template), distinguishing effective vs own with per-attribute provenance | **UC-67 (new)** |
|
||||
| DAG/multi-parent vs tree; note/branch (identity vs placement) | **enriches UC-22** |
|
||||
| Labels (`#tag`) + typed relations (`~relation`); HTML-native | **enriches UC-34** |
|
||||
| Templates (`~template`) inject attribute sets | **enriches UC-15** |
|
||||
| HTML (CKEditor) content → HTML↔Markdown lossy translation | **enriches UC-42** (links UC-59) |
|
||||
| Per-note encryption (protected notes) = **partial** content opacity | **enriches UC-61** |
|
||||
| Scripting (frontend/backend code notes, Script API) = in-app host | **enriches UC-38** |
|
||||
| ETAPI (token REST) = designed external surface for a self-host server | links UC-57 |
|
||||
| Render/script notes + attribute queries = dynamic content | links UC-54; typed relations link UC-58 |
|
||||
| `noteId`/`branchId` 12-char IDs | links UC-51 |
|
||||
|
||||
---
|
||||
|
||||
## 10. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Namespace model must support a DAG** and **separate page identity from placement**
|
||||
(note/branch). A page is one entity with N placements (paths/shards). Feeds the page/
|
||||
namespace model and the clone/reference primitive. (T12, T16.)
|
||||
- **Computed/inherited metadata** (UC-67): the page model's structured-data
|
||||
representation must carry **effective vs own** with per-attribute provenance
|
||||
(own/inherited/template), not a flat bag. (T12.)
|
||||
- **Content opacity is per-item** (UC-61 refinement): the content-opacity capability
|
||||
(proposed twelfth spectrum) should be **granular** (per-note), not only whole-shard.
|
||||
(T11.)
|
||||
- **HTML as a source content model** joins TML/Notion-blocks in the translation
|
||||
capability (HTML↔Markdown, lossy-aware). (T15.)
|
||||
- **Scripting as an in-engine host** + **ETAPI external REST** are two more adapter-host
|
||||
exemplars (with Roam/Obsidian/Joplin). (T14.)
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions (for spec / workplans)
|
||||
|
||||
1. How does shard-wiki represent a **cloned note** in the union — one page with multiple
|
||||
path placements, or a page transcluded into multiple locations? (UC-66 vs UC-44/45.)
|
||||
2. When projecting **inherited attributes** (UC-67), does shard-wiki materialize
|
||||
effective values (snapshot) or compute them live from the shard's tree/templates?
|
||||
3. Is **HTML↔Markdown** round-trip lossless enough for write-back overlays, or are
|
||||
Trilium overlays read-only/native-HTML (cf. UC-42 Q2)?
|
||||
4. For **per-note encryption** (UC-61), is a partially-opaque shard projected with
|
||||
protected notes as visible-but-opaque placeholders, or hidden entirely?
|
||||
|
||||
---
|
||||
|
||||
## 12. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| TriliumNext/Trilium — DeepWiki (https://deepwiki.com/TriliumNext/Trilium) | SQLite `document.db`/better-sqlite3; tree + cloning via branches; note types; sync |
|
||||
| Cloning notes — TriliumNext wiki (https://github.com/TriliumNext/Trilium/wiki/Cloning-notes) | Note in multiple locations; branches; branch prefixes |
|
||||
| TriliumNext — DeepWiki API & Synchronization (https://deepwiki.com/TriliumNext/Trilium/4.3-api-and-synchronization) | WebSocket + sync protocol w/ conflict resolution; 12-char IDs (note/branch/attribute/attachment) |
|
||||
| ETAPI (REST API) — Trilium docs (https://docs.triliumnotes.org/User%20Guide/User%20Guide/Advanced%20Usage/ETAPI%20(REST%20API)/) | Public REST API since v0.50; token/Bearer auth; trilium-py |
|
||||
| Script API — TriliumNext wiki (https://github.com/TriliumNext/Trilium/wiki/Script-API) | Frontend/backend code notes; Script API |
|
||||
| BrightCoding — TriliumNext overview (https://www.blog.brightcoding.dev/2025/09/20/triliumnext-notes...) | Attributes (labels/relations), inheritance, templates, per-note encryption, note types |
|
||||
|
||||
Cross-references: `research/260614-joplin-deep-dive/findings.md` (SQLite-local, dual
|
||||
surface, content opacity), `research/260614-notion-deep-dive/findings.md` (typed
|
||||
relations, external API), `research/260614-zigzag-deep-dive/findings.md` (clone /
|
||||
dimensions), `research/260614-shard-spectrum-synthesis/findings.md` (spectra this
|
||||
refines), `spec/UseCaseCatalog.md` (UC-15, UC-22, UC-34, UC-38, UC-42, UC-51, UC-54,
|
||||
UC-57, UC-58, UC-61), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T12,
|
||||
T14, T15, T16).
|
||||
|
||||
---
|
||||
|
||||
## 13. Traceability
|
||||
|
||||
- New UCs: **UC-66, UC-67** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-15, UC-22, UC-34, UC-38, UC-42, UC-61** (links UC-51, UC-54, UC-57,
|
||||
UC-58, UC-59).
|
||||
- Architecture (no UC): DAG namespace + identity/placement (note/branch) split;
|
||||
computed/inherited metadata; per-item content opacity; HTML source model; scripting +
|
||||
ETAPI host surfaces → `SHARD-WP-0002` (T11, T12, T14, T15, T16).
|
||||
- Boundary recorded: Trilium (TriliumNext) is **one SQLite-local candidate shard** with a
|
||||
DAG hierarchy and computed metadata, best attached via **ETAPI**; HTML-native (lossy to
|
||||
Markdown), per-note opacity; not a substrate, not the federation layer (INTENT
|
||||
graceful-degradation, no-silent-mutation, not-a-sync-daemon).
|
||||
</content>
|
||||
14
research/260614-usemodwiki-deep-dive/README.md
Normal file
14
research/260614-usemodwiki-deep-dive/README.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# 260614 — UseModWiki deep dive
|
||||
|
||||
Deep dive on **UseModWiki**: the **flat-file ancestor** — a Perl CGI wiki (Clifford Adams,
|
||||
2000) descended from AtisWiki/CvWiki, **CamelCase** linking, plain flat-file page + history
|
||||
storage, and the **engine Wikipedia originally ran on** (MediaWiki Phase I). Origins/lineage
|
||||
value: the minimal flat-file page+history model the whole field descends from.
|
||||
|
||||
- `findings.md` — the model, lineage, capability profile, INTENT mapping, enrichments (no new
|
||||
UC — reinforces the minimal flat-file baseline UC-82), architecture notes, sources,
|
||||
traceability.
|
||||
|
||||
Catalog yield: **enrichment-only** (reinforces UC-82) — enriched UC-01 (open wiki), UC-40
|
||||
(flat-file store), UC-25 (CamelCase naming), UC-36/41 (flat-file history). Lineage noted for
|
||||
the origins record. Feeds SHARD-WP-0002 T11 (minimal profile lineage).
|
||||
87
research/260614-usemodwiki-deep-dive/findings.md
Normal file
87
research/260614-usemodwiki-deep-dive/findings.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# UseModWiki — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T9 · **Subject:** UseModWiki, Clifford
|
||||
Adams's flat-file Perl wiki — the lineage root of much of the field.
|
||||
|
||||
## Why this dive
|
||||
|
||||
This is a **lineage** dive, not a new-capability one. UseModWiki is the **ancestor**: the
|
||||
minimal flat-file page+history wiki the c2-era engines and **MediaWiki Phase I** descend
|
||||
from (Wikipedia ran on UseModWiki, 2001–2002). It pairs with
|
||||
`research/260608-c2-wiki-origins/` to record *where the page+history model came from*. It
|
||||
adds no new shard capability beyond the minimal flat-file floor (UC-82, Oddmuse) — its value
|
||||
is **historical grounding** and confirming the floor is genuinely the field's common root.
|
||||
|
||||
## 1. The model
|
||||
|
||||
- **Perl CGI** (`wiki.pl`), single-script, descended from AtisWiki → CvWiki → UseModWiki
|
||||
(c. 2000), by Clifford Adams.
|
||||
- **Flat-file storage**: each page stored as a text file (under a `db/` data directory),
|
||||
with **page history** kept (older revisions retained as files / in the page record). No
|
||||
database.
|
||||
- **Linking**: **CamelCase** (`WikiWord`) auto-links by default; later added **free links**
|
||||
`[[Like This]]`. This is the canonical CamelCase-naming lineage (UC-25).
|
||||
- **Open editing**, recent-changes, simple markup — the c2 ethos in a portable script.
|
||||
|
||||
## 2. Lineage
|
||||
|
||||
- **MediaWiki Phase I** *was* UseModWiki — early Wikipedia ran on it before the PHP rewrite
|
||||
(Phase II/III). So UseModWiki is the **direct ancestor of MediaWiki/Wikibase** (T2) and a
|
||||
sibling-root of the c2 tradition.
|
||||
- The **flat-file page+history model** here is the shape Oddmuse (UC-82), TWiki's file store
|
||||
(UC-40), and others elaborate — confirming the **minimal file-store floor** is the field's
|
||||
common origin, not a modern simplification.
|
||||
|
||||
## 3. Capability profile
|
||||
|
||||
Essentially **identical to the minimal floor** (UC-82, Oddmuse):
|
||||
|
||||
| Dimension | UseModWiki |
|
||||
|-----------|------------|
|
||||
| Attachment mode | **file-store** (flat files under `db/`); CGI, no API |
|
||||
| Addressing | page = file; **CamelCase** name = identity |
|
||||
| Structure | flat page space; CamelCase link graph |
|
||||
| History | flat-file retained revisions (may be limited) |
|
||||
| Native query | none |
|
||||
| Translation | simple wiki markup → Markdown (lossy) |
|
||||
| Write granularity | page (file) |
|
||||
| Access | open editing (optional admin password) |
|
||||
| Provenance | timestamp, optional username |
|
||||
|
||||
## 4. INTENT mapping
|
||||
|
||||
- **Reinforces the minimal flat-file baseline** (UC-82): UseModWiki is the *historical*
|
||||
instance of the graceful-degradation floor — attach via its flat files; partial history
|
||||
surfaced honestly.
|
||||
- **CamelCase naming** (UC-25): the canonical source of WikiWord auto-linking — the page
|
||||
model's name/identity and link-resolution must accommodate CamelCase-derived identities.
|
||||
- **Open wiki** (UC-01): the c2 open-editing ethos at the root.
|
||||
- **Lineage grounding**: confirms shard-wiki's "Git-based Markdown" page model descends from
|
||||
(and must remain attach-compatible with) the flat-file ancestor.
|
||||
|
||||
**No new UC** — UseModWiki adds historical grounding, not a new orchestration scenario;
|
||||
its capabilities are subsumed by UC-82 (minimal flat-file baseline) plus UC-25 (CamelCase).
|
||||
|
||||
## 5. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T11:** UseModWiki = a second instance of the **minimal/floor profile** (with Oddmuse);
|
||||
confirms the floor is the field's common root, so the floor profile is the right baseline.
|
||||
- **Naming/identity:** CamelCase-derived page identities (UC-25) are part of the legacy
|
||||
identity surface the adapter must parse (and translate to the page model's identity).
|
||||
|
||||
## 6. Open questions
|
||||
|
||||
(None new — covered by UC-82 partial-history honesty, catalog open-Q30, and the c2 origins
|
||||
record. CamelCase resolution is UC-25.)
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- UseModWiki — usemod.com/cgi-bin/wiki.pl (the wiki about itself); Wikipedia: *UseModWiki*,
|
||||
*MediaWiki* (Phase I history)
|
||||
- prior: `research/260608-c2-wiki-origins/`; `research/260614-oddmuse-deep-dive/` (minimal
|
||||
floor, UC-82)
|
||||
|
||||
## 8. Traceability
|
||||
|
||||
**No new UC** (reinforces UC-82). Enriched: UC-01, UC-40, UC-25, UC-36, UC-41. Lineage noted
|
||||
for the origins record. Architecture cross-ref: SHARD-WP-0002 T11 (minimal-profile lineage).
|
||||
16
research/260614-wikibase-deep-dive/README.md
Normal file
16
research/260614-wikibase-deep-dive/README.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# 260614 — Wikibase / Wikidata deep dive
|
||||
|
||||
Deep dive on **Wikibase** (the MediaWiki extension behind **Wikidata**) as the
|
||||
**entity-statement (RDF) far end** of the structure and native-query spectra: items and
|
||||
properties, statements = claim + qualifiers + references + rank, stable opaque IDs, and
|
||||
**SPARQL** (Wikidata Query Service / Blazegraph) including **federated** queries.
|
||||
|
||||
- `findings.md` — data model, RDF/SPARQL surface, storage & history, capability profile,
|
||||
INTENT mapping, UC seeds (UC-73–UC-75), architecture notes for SHARD-WP-0002, open
|
||||
questions, sources, traceability.
|
||||
|
||||
Catalog yield: UC-73 (attach a typed entity-statement / RDF shard, lossy render to a page),
|
||||
UC-74 (graph query the union via SPARQL + federated `SERVICE` cross-endpoint query), UC-75
|
||||
(statement-level provenance — references + rank per assertion). Enriched UC-34/58/52/24.
|
||||
Feeds SHARD-WP-0002 T12 (structured/typed page model) and T16 (stable language-neutral
|
||||
identity ≠ label/placement).
|
||||
200
research/260614-wikibase-deep-dive/findings.md
Normal file
200
research/260614-wikibase-deep-dive/findings.md
Normal file
@@ -0,0 +1,200 @@
|
||||
# Wikibase / Wikidata — deep dive (findings)
|
||||
|
||||
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T2 · **Subject:** Wikibase (MediaWiki
|
||||
extension) and its flagship instance **Wikidata**, incl. the Wikidata Query Service (SPARQL).
|
||||
|
||||
## Why this dive
|
||||
|
||||
Every structured shard so far tops out at *typed records in a database*: Notion's
|
||||
database-pages, XWiki's XObjects/classes, Trilium's typed relations, Roam/Logseq's
|
||||
attribute blocks. Wikibase is a *different kind of structure altogether* — a **typed
|
||||
knowledge graph of entities and provenance-bearing statements**, queried with **SPARQL**
|
||||
over an **RDF** projection. It is the **far end of the structure spectrum** and of the
|
||||
**native-query spectrum**, and it pushes **provenance down to the individual assertion**.
|
||||
The question for shard-wiki: what does a shard look like when its "page" is *not prose but a
|
||||
set of statements*, and what does the page model / adapter contract owe such a shard?
|
||||
|
||||
## 1. The data model — entities, statements, snaks
|
||||
|
||||
**Entities** are the top-level objects, each on its own MediaWiki page with a **stable
|
||||
opaque ID**:
|
||||
|
||||
- **Item** — `Q42`. Has multilingual **labels / descriptions / aliases**, a set of
|
||||
**statements**, and **sitelinks** (links to wiki articles). The label is *annotation*,
|
||||
not identity — `Q42` is the identity, "Douglas Adams" is just its English label.
|
||||
- **Property** — `P31` ("instance of"). Also has labels/descriptions/aliases, plus a
|
||||
**fixed datatype** constraining its values (item-reference, string, time,
|
||||
globe-coordinate, quantity, monolingual-text, url, external-id, commons-media, …).
|
||||
|
||||
**Statement** = the unit of assertion on an item. Structure:
|
||||
|
||||
```
|
||||
statement = claim + references[] + rank
|
||||
claim = mainSnak + qualifiers[]
|
||||
snak = property + snaktype + (value) # snaktype ∈ value | somevalue | novalue
|
||||
```
|
||||
|
||||
- **Main snak** — the core property→value assertion (e.g. `P31` → `Q5` "human").
|
||||
- **Qualifiers** — snaks that *contextualize* the claim without being the subject (validity
|
||||
time, "as of", determination method, units). E.g. *population (P1082) = 8.4M, **point in
|
||||
time (P585) = 2020***.
|
||||
- **References** — lists of snaks citing **where the claim comes from** (a source item, a
|
||||
URL, a page number). **Provenance attached to the individual statement, not the page.**
|
||||
- **Rank** — `preferred` | `normal` | `deprecated`: relative importance among same-property
|
||||
statements (lets multiple, even contradictory, values coexist with a curation signal —
|
||||
the structured analogue of fedwiki's "chorus").
|
||||
- Each statement carries a **stable GUID** (`Q42$<uuid>`), so statements are individually
|
||||
addressable.
|
||||
|
||||
`somevalue` (known to exist, value unknown) and `novalue` (known *not* to have a value) are
|
||||
**first-class** — the model represents *known-unknowns* explicitly, which prose and most
|
||||
DBs cannot.
|
||||
|
||||
## 2. The RDF / SPARQL surface
|
||||
|
||||
Wikibase **projects entities to RDF**; the **Wikidata Query Service (WDQS)** is a
|
||||
**Blazegraph** triple store exposing a **SPARQL** endpoint. The projection is deliberately
|
||||
multi-layered:
|
||||
|
||||
- **Truthy** triples (`wdt:` prefix) — the simple "best" value, for easy queries:
|
||||
`wd:Q42 wdt:P31 wd:Q5`.
|
||||
- **Full** statements — reified so qualifiers/references/rank survive: `wd:Q42 p:P31
|
||||
?stmt . ?stmt ps:P31 wd:Q5 ; pq:P585 ?time ; prov:wasDerivedFrom ?ref`. (`p:`=statement
|
||||
node, `ps:`=statement value, `pq:`=qualifier, `pr:`/`prov:`=reference.)
|
||||
- **Federated SPARQL** — the `SERVICE <endpoint> { … }` keyword runs a sub-query against
|
||||
*another* SPARQL endpoint and joins the results. **Query-level federation is built into
|
||||
the query language** — a different federation primitive from fedwiki's fork/neighborhood.
|
||||
- **EntitySchemas / ShEx** — schemas (`E`-ids) that *validate* an entity's shape (Shape
|
||||
Expressions). Optional, declarative structure validation over the open graph.
|
||||
|
||||
## 3. Storage, identity, history
|
||||
|
||||
- **Storage:** each entity is a **JSON blob stored as a MediaWiki page** (`Item:` /
|
||||
`Property:` content model). The RDF/SPARQL store is a **derived index** rebuilt from these
|
||||
canonical JSON entities (an *update stream* feeds WDQS) — exactly shard-wiki's
|
||||
"derived query index over a canonical store" pattern (UC-63), at planet scale.
|
||||
- **Identity:** the **opaque Q/P/L IDs are the identity**, fully decoupled from
|
||||
human-readable labels and from language. This is the cleanest real-world instance of
|
||||
**stable, language-neutral identity ≠ label/placement** — a strong reinforcement of our
|
||||
identity model (T16).
|
||||
- **History:** because each entity is one MediaWiki page, history is **page-level MediaWiki
|
||||
revisions** — every edit is a full-entity JSON snapshot with author/timestamp/comment.
|
||||
*Coarse* history granularity (whole entity per revision), but the **edit API is
|
||||
fine-grained** (`wbsetclaim`, `wbeditentity` patch individual statements). So: **fine
|
||||
write API over a coarse history unit** — a distinct point on the write/history spectra.
|
||||
|
||||
## 4. Capability profile
|
||||
|
||||
| Dimension (synthesis spectrum) | Wikibase / Wikidata |
|
||||
|--------------------------------|---------------------|
|
||||
| Attachment mode | **external-API** (MediaWiki Action API + REST) **and** a derived **SPARQL endpoint**; self-hostable |
|
||||
| Addressing granularity | **statement** (each has a GUID) within an **entity** (Q/P id) |
|
||||
| Content identity | **stable opaque ID** (Q/P/L); labels are multilingual annotations |
|
||||
| Identity vs placement | **fully separated** — identity is language- and label-neutral |
|
||||
| Structure | **typed knowledge graph**: entities + statements (claim+qualifiers+refs+rank) |
|
||||
| History | **page-level revisions** (whole-entity JSON snapshots); fine-grained edit API |
|
||||
| Merge model | MediaWiki last-writer / edit-conflict; rank lets contradictory values coexist |
|
||||
| Native query | **SPARQL** (RDF) + **federated `SERVICE`** cross-endpoint join — the far end |
|
||||
| Translation | **not Markdown** — content *is* statements; render to prose is a lossy projection |
|
||||
| Attachment/write granularity | **statement-level writes** via API; coarse history unit |
|
||||
| Operational envelope | huge derived index (Blazegraph), rate-limited public endpoints |
|
||||
| Access grant | open read; MediaWiki user/permission model for write; self-host = own ACL |
|
||||
| Content opacity | transparent (public JSON + RDF); not encrypted |
|
||||
| Provenance | **statement-level** — references + rank per assertion (new far end) |
|
||||
|
||||
## 5. INTENT mapping
|
||||
|
||||
### Reinforcements
|
||||
|
||||
- **Stable identity ≠ placement** (T16): Q/P IDs decoupled from labels/language are the
|
||||
textbook case — adopt the principle that a page's *identity* is an opaque stable handle,
|
||||
display names are annotations.
|
||||
- **Derived index over canonical store** (UC-63): WDQS is exactly a SPARQL index rebuilt
|
||||
from canonical JSON entities via an update stream — validates the projection pattern.
|
||||
- **Union without erasure / chorus**: **rank** lets multiple (even contradictory) statements
|
||||
coexist with a curation signal rather than forcing one truth — the *structured* analogue
|
||||
of fedwiki's chorus (UC-72) and our "view multiple versions" (UC-27).
|
||||
- **Mechanism over policy**: references + rank are *mechanism* for representing disagreement
|
||||
and sourcing; which statement "wins" is left to the consumer/query.
|
||||
|
||||
### Divergences (boundaries / design notes)
|
||||
|
||||
- **Content is not Markdown.** A Wikibase "page" is a set of statements; there is no prose
|
||||
body. This is the **structure far-end**: shard-wiki must either (a) treat such a shard as
|
||||
a **structured/typed shard** projected to a *lossy* Markdown/table rendering (UC-55/UC-73),
|
||||
or (b) model a page whose payload is typed statements (T12). Forcing it into Markdown-first
|
||||
erases the graph — a design-bug if done silently; render-with-provenance instead.
|
||||
- **Provenance granularity is finer than ours.** Our provenance is per-page/per-shard;
|
||||
Wikibase is **per-statement** (references) and even per-value (rank). The page model and
|
||||
coordination journal should *allow* sub-page provenance (UC-75) even if MVP records it per
|
||||
page.
|
||||
- **Query is graph, not text/datalog.** SPARQL over RDF (with federated `SERVICE`) is a
|
||||
richer query far-end than Roam/Logseq datalog or Notion filters (UC-52) — and its
|
||||
`SERVICE` federation is a *query-time* cross-shard join, distinct from fedwiki structural
|
||||
federation. Note both as native-query tiers.
|
||||
|
||||
### What to keep
|
||||
|
||||
1. **Opaque stable identity, labels-as-annotations** as the identity model (T16).
|
||||
2. **Statement/assertion-level provenance** (references) and a **coexistence-with-rank**
|
||||
model as the structured form of union-without-erasure (UC-75).
|
||||
3. **Derived SPARQL/graph index over a canonical entity store** as a projection pattern
|
||||
(UC-63/UC-74), incl. **federated query** as a first-class federation mode.
|
||||
4. A **typed-graph page payload** option in the page model (T12), with **lossy
|
||||
render-to-Markdown** as the projection (never silent flattening).
|
||||
|
||||
## 6. UC seeds
|
||||
|
||||
| # | Seed | Disposition |
|
||||
|---|------|-------------|
|
||||
| UC-73 | Attach a **Wikibase** as a **typed entity-statement (RDF) shard** (items/properties/statements w/ qualifiers); project to a rendered page view, lossy to Markdown, preserving the graph | **new** |
|
||||
| UC-74 | **Graph-query the union** via **SPARQL** and **federate queries across endpoints** (`SERVICE`) — graph query as a native-query tier + query-time cross-shard join | **new** |
|
||||
| UC-75 | Preserve **statement-level provenance** — references + rank attached to each assertion (sub-page provenance granularity) | **new** |
|
||||
| — | typed records → typed *graph* entities | enrich **UC-34** |
|
||||
| — | inter-record relations → typed graph edges with qualifiers | enrich **UC-58** |
|
||||
| — | native query → SPARQL/RDF + federated SERVICE | enrich **UC-52** |
|
||||
| — | provenance → statement/assertion granularity | enrich **UC-24** |
|
||||
|
||||
## 7. Architecture notes for SHARD-WP-0002
|
||||
|
||||
- **T12 (structured/typed page model):** add a **typed-graph payload** tier above
|
||||
typed-records — a page whose content is **entities + statements (claim + qualifiers +
|
||||
references + rank)**, with `somevalue`/`novalue` known-unknowns. Render-to-Markdown is a
|
||||
**lossy projection**, not the canonical form.
|
||||
- **T16 (identity / addressing):** adopt **opaque stable identity with labels-as-annotation**
|
||||
(Q/P model); record **statement GUIDs** as an example of *sub-page addressable units*.
|
||||
- **Native-query tiering:** SPARQL/RDF + federated `SERVICE` is the **graph far-end** of the
|
||||
query spectrum (above datalog/filters); `SERVICE` is also a **query-time federation**
|
||||
mode to sit beside fedwiki's structural federation.
|
||||
- **Provenance model:** allow **per-statement references + rank** (sub-page provenance,
|
||||
coexistence-with-curation) in the union, even if MVP collapses to per-page.
|
||||
- **Derived index:** WDQS = canonical JSON entities → update stream → Blazegraph SPARQL
|
||||
index; the reference implementation of UC-63 at scale (per-shard or core-built index, Q16).
|
||||
|
||||
## 8. Open questions
|
||||
|
||||
1. Does shard-wiki model a **typed-graph page** natively (T12), or always treat Wikibase as
|
||||
a structured shard **projected to a Markdown/table rendering** (UC-55), or both
|
||||
(canonical graph + lossy view)?
|
||||
2. Is **SPARQL/graph query** exposed as a union-level capability (translate to a common
|
||||
query layer) or only as a **pass-through** to graph-capable shards? How does federated
|
||||
`SERVICE` relate to shard-wiki's own cross-shard query?
|
||||
3. At what granularity does the coordination journal record **provenance** — per page
|
||||
(MVP), per statement (Wikibase-native), or configurable?
|
||||
4. Is **rank** (coexisting contradictory values w/ curation) representable in the union as a
|
||||
first-class "chorus of statements," unifying with fedwiki's page-level chorus (UC-72/27)?
|
||||
|
||||
## 9. Sources
|
||||
|
||||
- Wikibase/DataModel and **DataModel/Primer** — mediawiki.org
|
||||
- Help:Qualifiers; Wikidata SPARQL query service + Query Help; SPARQL tutorial — wikidata.org
|
||||
- Wikidata Query Service / User Manual — mediawiki.org; Wikitech (Blazegraph, updater)
|
||||
- "The wikibase model" — Vanderbilt Libraries Digital Lab (heardlibrary.github.io)
|
||||
- RaiseWikibase — Wikibase Data Model functions (ub-mannheim.github.io)
|
||||
- WShEx / EntitySchemas (ShEx) — arxiv.org/abs/2208.02697, ceur-ws.org Vol-3262
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
New UCs **UC-73–UC-75** carry the marker **⬡** in the wikiengines column of
|
||||
`spec/UseCaseCatalog.md`. Enriched: UC-34, UC-58, UC-52, UC-24. Architecture cross-refs:
|
||||
SHARD-WP-0002 T12, T16, native-query tiering, provenance model, UC-63 derived index.
|
||||
49
research/260614-wikijs-deep-dive/README.md
Normal file
49
research/260614-wikijs-deep-dive/README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# 260614 — Wiki.js deep dive (the storage-module engine that already speaks Git + Markdown)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Wiki.js** — the modern Node.js server wiki — read through
|
||||
shard-wiki's lens. Wiki.js is the **closest existing engine to shard-wiki's own design**:
|
||||
it is DB-canonical (Postgres/MySQL/SQLite) **but abstracts persistence behind pluggable
|
||||
"storage modules"** — Git, local filesystem, S3, Azure Blob — each acting as **backup or
|
||||
source of truth**, with the **Git module bidirectionally syncing clean `.md` files (+ YAML
|
||||
frontmatter)** to a repo. That storage-module interface is, in effect, a shipped version
|
||||
of shard-wiki's adapter contract — the second concrete prior art after **Foswiki::Store**,
|
||||
and the closer one (medium = Markdown in Git).
|
||||
|
||||
Distinctive material:
|
||||
- **Architecture** — Node.js/Vue; **GraphQL API** over all resources; DB-canonical;
|
||||
Markdown (primary) + HTML + AsciiDoc; path-based hierarchy; modular
|
||||
(storage/auth/search/render/editor)
|
||||
- **Storage modules** — provider connector (Git/FS/S3/Azure) with properties + create/
|
||||
update/delete/sync methods, **backup-or-source-of-truth**, `injectMetadata()` YAML
|
||||
frontmatter; **Git module commits every change → version-controlled clean Markdown**,
|
||||
bidirectional
|
||||
- **GraphQL** — typed, **introspectable** (capability/schema discovery), **selective-field**
|
||||
projection
|
||||
- **Auth modules** (delegated authn) + **path-based rule ACL**
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | Architecture, storage modules (≈ adapter contract), Git mirror as attach+write surface, GraphQL, access control, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-68
|
||||
attach an engine-maintained bidirectional Git mirror of clean Markdown, incl. write-by-
|
||||
commit; UC-69 attach via a typed, introspectable API — GraphQL — for schema discovery +
|
||||
selective projection); UC-06/36/38/40/42/57 enriched. Logged for `SHARD-WP-0002`
|
||||
(T11/T14): the **storage-module abstraction as a second adapter-contract prior art** (with
|
||||
Foswiki::Store), the **engine-maintained Git mirror** as preferred attach+write surface,
|
||||
and **GraphQL introspection** for capability discovery + selective projection.
|
||||
|
||||
**Key takeaway recorded:** Wiki.js is the most shard-wiki-shaped engine yet — its
|
||||
storage-module interface is near-isomorphic to the adapter contract, and its
|
||||
Markdown-in-Git mirror is the ideal file-store attach (clean MD, git history,
|
||||
write-by-commit). **Boundary:** one server-engine candidate shard, DB-canonical with a
|
||||
maintained Git mirror; don't double-sync; honor its access rules; not the federation layer.
|
||||
</content>
|
||||
249
research/260614-wikijs-deep-dive/findings.md
Normal file
249
research/260614-wikijs-deep-dive/findings.md
Normal file
@@ -0,0 +1,249 @@
|
||||
# Findings — Wiki.js: the storage-module engine that already speaks Git + Markdown
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **modern shipped product** — a server wiki engine; a *candidate shard* and
|
||||
the closest existing engine to shard-wiki's own design (DB-canonical, but with a
|
||||
**pluggable storage-module abstraction** that bidirectionally syncs clean Markdown to
|
||||
**Git**/FS/S3/Azure)
|
||||
Lens: shard-wiki — storage-module-as-adapter-contract prior art, the engine-maintained
|
||||
Git mirror as an attach + write surface, GraphQL as a typed/introspectable API, and
|
||||
path-based access rules
|
||||
|
||||
> Why Wiki.js is special in this set. Every prior engine/tool stored content in *one*
|
||||
> substrate and exposed *one or two* surfaces. Wiki.js is DB-canonical (Postgres/MySQL/
|
||||
> SQLite) **but abstracts persistence behind pluggable "storage modules"** — Git, local
|
||||
> filesystem, S3, Azure Blob — each of which can act as **backup *or* source of truth**,
|
||||
> and the **Git module bidirectionally syncs clean `.md` files (with YAML frontmatter)**
|
||||
> to a repo. In other words, Wiki.js already does, internally, much of what shard-wiki's
|
||||
> adapter contract is meant to do: a **versioned, capability-bearing, multi-provider
|
||||
> persistence interface over Markdown**. It is the second concrete prior art for the
|
||||
> contract after **Foswiki::Store** — and arguably the closer one, because its content
|
||||
> is **Markdown in Git**, exactly shard-wiki's native shape.
|
||||
|
||||
Contrast set: the engine dives (XWiki DB/components, TWiki/Foswiki file+RCS) and the
|
||||
modern tools. Wiki.js = **modern server engine, DB-canonical, Git-storage-of-Markdown,
|
||||
GraphQL API, pluggable modules**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core architecture
|
||||
|
||||
- **Stack:** Node.js backend, Vue front end; **everything is accessible via a GraphQL
|
||||
API** (`/graphql`, with a Playground). The DB is **Postgres** (primary; MySQL/SQLite
|
||||
supported), read-replica-capable.
|
||||
- **Content:** **Markdown (primary)**, plus **HTML** and **AsciiDoc**, via pluggable
|
||||
**editors** (Markdown, WYSIWYG/visual, raw HTML). Pages live on **paths** (a path-based
|
||||
hierarchy) with tags.
|
||||
- **Modular architecture:** typed pluggable **modules** — **storage**, **auth**,
|
||||
**search**, **rendering**, **editor**. The contract-relevant ones are *storage* (§2)
|
||||
and *auth* (§4).
|
||||
- **3.0:** a large rewrite is in development; storage remains a headline subsystem.
|
||||
|
||||
---
|
||||
|
||||
## 2. Storage modules — the headline (≈ shard-wiki's adapter contract)
|
||||
|
||||
A **storage module** "connects Wiki.js with a local or remote storage provider, to act as
|
||||
**backup or source of truth** for content. It consists of **properties** (user-set) and
|
||||
**methods called on events** — create, update, delete content, and sync." Providers
|
||||
include **Git, local filesystem, S3, Azure Blob**.
|
||||
|
||||
The **Git module** is the most-used: **every page change commits to a Git repository**,
|
||||
turning the wiki into **version-controlled `.md` files**. Content is written as **clean
|
||||
Markdown**, and `injectMetadata()` prepends **YAML frontmatter** (for Markdown) or HTML
|
||||
comments (for HTML) carrying page metadata. The sync is **bidirectional** with the remote
|
||||
repo, and access to the synced files honors Wiki.js **access rules** (§4).
|
||||
|
||||
Why this matters to shard-wiki — three distinct payoffs:
|
||||
|
||||
1. **Prior art for the adapter contract.** Wiki.js's storage-module interface is a
|
||||
shipped instance of what `SHARD-WP-0002` T11 specifies: a *versioned, capability-
|
||||
bearing, multi-provider* persistence abstraction, with a backup-vs-source-of-truth
|
||||
choice and lifecycle methods (create/update/delete/sync). It joins **Foswiki::Store**
|
||||
as concrete prior art — and is closer, because its medium is **Markdown in Git**.
|
||||
2. **The ideal file-store attach.** A Wiki.js instance with the Git module **is a git
|
||||
repo of clean Markdown + YAML frontmatter**, maintained by the engine. shard-wiki can
|
||||
attach that repo **directly** (UC-40) — better than reading the DB — and get **git
|
||||
history for free** (UC-36, *adopt* not supplement, via the mirror).
|
||||
3. **A write path via Git, no API needed.** Because the Git sync is **bidirectional**,
|
||||
shard-wiki can **write-through by committing Markdown** to the repo; Wiki.js **ingests
|
||||
git→DB**. This is overlay/patch application as a *git commit*, not an API call (UC-68).
|
||||
Caution: Wiki.js owns the DB↔git sync — **don't double-sync**, and coordinate which
|
||||
side is **source of truth** (configurable).
|
||||
|
||||
---
|
||||
|
||||
## 3. GraphQL — a typed, introspectable external API
|
||||
|
||||
Unlike the REST/local-REST tools (Joplin, Trilium, Notion), Wiki.js exposes a **GraphQL
|
||||
API** over all resources. Two shard-wiki-relevant properties:
|
||||
|
||||
- **Introspection = capability/schema discovery.** A GraphQL schema is self-describing;
|
||||
an adapter can introspect the content/page schema rather than hard-coding it — feeding
|
||||
T11's *capability discovery*.
|
||||
- **Selective-field projection.** GraphQL fetches **exactly the fields a projection
|
||||
needs** (page body vs. metadata vs. tags), reducing over-fetch — relevant to projection
|
||||
efficiency and the operational envelope. (UC-69; an external-API sub-mode beside
|
||||
Notion's REST, UC-57.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Auth modules + path-based access rules
|
||||
|
||||
- **Auth modules** (local, LDAP, OAuth, SAML, …) — **authentication is delegated to
|
||||
pluggable providers**, exactly shard-wiki's **authn-delegated** stance
|
||||
([[shard-wiki-auth-in-core-decision]]).
|
||||
- **Access rules** are **path-based / rule-based ACL**: allow/deny access to all or
|
||||
specific sections of the wiki per group; the Git-synced files honor the same rules. For
|
||||
shard-wiki this is a richer ACL than TWiki's per-topic flags (UC-06) — **rule-based by
|
||||
path pattern** — and informs how a projection should **honor/surface a shard's
|
||||
restricted regions** (union without erasure: show that a region is restricted, don't
|
||||
silently drop or expose it).
|
||||
|
||||
---
|
||||
|
||||
## 5. Wiki.js as a shard — capability profile
|
||||
|
||||
| Capability | Wiki.js | Notes for the adapter contract |
|
||||
|------------|---------|--------------------------------|
|
||||
| Read | yes (Git mirror / GraphQL / DB) | **Git mirror = clean Markdown** is the best surface |
|
||||
| Write | yes (GraphQL **or git commit**) | bidirectional Git ingest = write-by-commit (UC-68) |
|
||||
| Write granularity | per-page | — |
|
||||
| Identity / addressing | path (+ page id) | path-based; YAML frontmatter carries metadata/id |
|
||||
| Structure | YAML frontmatter (in-file via Git module) | git-diffable in-text structure (good) |
|
||||
| History | **git-native via Git module** (else DB versions) | adopt the mirror's git history (UC-36) |
|
||||
| Native query | GraphQL; pluggable search (DB/Elastic/Algolia) | delegate search; GraphQL selective fetch (UC-69) |
|
||||
| Translation | Markdown primary; HTML/AsciiDoc too | mostly native; multi-format = light translation (UC-42) |
|
||||
| Attach modes | **file-store (engine-maintained Git mirror)** / external-API (GraphQL) | clean MD in git is the ideal file-store (UC-40, UC-68) |
|
||||
| Access control | **path-based rule ACL**; delegated auth modules | authn-delegated; honor restricted regions (UC-06) |
|
||||
| Storage abstraction | **pluggable storage modules (Git/FS/S3/Azure)** | **adapter-contract prior art** (T11) |
|
||||
| Content types | pages + media/assets | assets via storage modules |
|
||||
|
||||
Verdict: **the most shard-wiki-shaped engine yet.** Best attached via its **Git storage
|
||||
mirror** (clean Markdown + frontmatter + git history, with write-by-commit), or via
|
||||
**GraphQL** for live/selective access. Its standout gift is **architectural**: a shipped
|
||||
storage-module abstraction that is near-isomorphic to shard-wiki's adapter contract.
|
||||
|
||||
---
|
||||
|
||||
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 6.1 Reinforcements
|
||||
|
||||
- **Storage modules = the adapter contract, shipped.** A versioned, capability-bearing,
|
||||
multi-provider persistence interface with backup-vs-source-of-truth — exactly T11's
|
||||
shape, in production (with Foswiki::Store as the other prior art).
|
||||
- **Markdown-in-Git is the native shape.** Wiki.js validates that a serious engine can
|
||||
keep content as **clean `.md` + frontmatter in git** — shard-wiki's preferred medium —
|
||||
while still offering a rich app over a DB.
|
||||
- **authn-delegated** (auth modules) and **git coordination** (Git storage) are both core
|
||||
shard-wiki stances, here shipped.
|
||||
|
||||
### 6.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **DB is Wiki.js's canonical store; the Git repo is a maintained mirror.** Treat the
|
||||
mirror as the attach surface, but respect that **Wiki.js runs the DB↔git sync** —
|
||||
don't double-sync; coordinate source-of-truth (UC-68).
|
||||
2. **Bidirectional write needs care.** Committing Markdown is a real write path, but
|
||||
races with the engine's own sync; apply overlays as commits the engine can ingest
|
||||
cleanly, never out-of-band DB edits (overlay before mutation, no silent mutation).
|
||||
3. **One Wiki.js instance = one shard**, not the federation layer; its access rules are
|
||||
the shard's, to be honored/surfaced, not re-implemented.
|
||||
4. **HTML/AsciiDoc pages** need light translation; Markdown pages are native.
|
||||
|
||||
### 6.3 What Wiki.js teaches that shard-wiki should keep
|
||||
|
||||
- **Model the contract on a real storage-module interface** (Wiki.js + Foswiki::Store):
|
||||
properties + lifecycle methods (create/update/delete/sync) + a **backup-vs-source-of-
|
||||
truth** flag + a provider list. (T11.)
|
||||
- **Prefer the engine-maintained Git mirror** as the attach surface for DB-canonical
|
||||
engines that offer one — clean Markdown, git history, and write-by-commit (UC-68).
|
||||
- **Use GraphQL introspection for capability discovery and selective projection** where a
|
||||
shard offers it (UC-69).
|
||||
|
||||
---
|
||||
|
||||
## 7. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-67**. New UCs **UC-68, UC-69** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Attach an engine-maintained bidirectional Git mirror of clean Markdown** (Wiki.js Git storage module): attach the repo as a file-store shard (clean MD + YAML frontmatter + git history) and optionally **write-through by committing Markdown the engine ingests**; coordinate source-of-truth, don't double-sync | **UC-68 (new)** |
|
||||
| **Attach via a typed, introspectable API (GraphQL)**: discover the shard's schema by introspection (capability discovery) and fetch only the fields a projection needs (selective projection) | **UC-69 (new)** |
|
||||
| Pluggable **storage modules** (Git/FS/S3/Azure) = adapter-contract prior art; modular auth/search/render/editor | **enriches UC-38** |
|
||||
| Git storage = **clean Markdown in git**, the ideal engine-maintained file-store attach | **enriches UC-40** |
|
||||
| Git history natively via the storage module (**adopt** via mirror) | **enriches UC-36** |
|
||||
| GraphQL external-API variant (typed/introspectable/selective) | **enriches UC-57** |
|
||||
| Multi-format (Markdown primary; HTML/AsciiDoc) | **enriches UC-42** |
|
||||
| Path-based rule ACL; delegated auth modules | **enriches UC-06** (links authz decision) |
|
||||
| Engine-maintained clean git shard | links UC-02 |
|
||||
|
||||
---
|
||||
|
||||
## 8. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- **Second adapter-contract prior art (T11).** Wiki.js's **storage-module interface**
|
||||
(properties + create/update/delete/sync methods + backup-vs-source-of-truth + multi-
|
||||
provider Git/FS/S3/Azure) is, with **Foswiki::Store**, the model to base the contract
|
||||
on — and the closer one (medium = Markdown in Git). Cite both in T11.
|
||||
- **Engine-maintained Git mirror (T14).** Add to the attachment taxonomy: for a
|
||||
DB-canonical engine that offers a bidirectional Git-of-Markdown mirror, the **mirror is
|
||||
the preferred attach + write surface** (clean MD, git history, write-by-commit). A
|
||||
cleaner sibling of Joplin's proprietary interchange mirror (UC-60) and the file-store
|
||||
native-store kind (UC-40).
|
||||
- **GraphQL → capability discovery + selective projection (T11/T14).** Typed/
|
||||
introspectable APIs give schema discovery (capability negotiation) and reduce over-fetch
|
||||
— note alongside Notion's REST external-API mode.
|
||||
- **authn-delegated + path-based ACL** reinforce the settled authz decision and inform how
|
||||
projections honor/surface restricted regions (union without erasure).
|
||||
|
||||
---
|
||||
|
||||
## 9. Open questions (for spec / workplans)
|
||||
|
||||
1. For an engine with a bidirectional Git mirror (UC-68), is the **mirror or the DB the
|
||||
source of truth** from shard-wiki's view, and how do we avoid races with the engine's
|
||||
own sync (write-by-commit cadence, locking)?
|
||||
2. When both a **Git mirror and GraphQL** are available, which does a binding prefer —
|
||||
mirror for content/history, GraphQL for live/selective metadata? Can one binding use
|
||||
both?
|
||||
3. How does shard-wiki **honor/surface path-based access rules** (UC-06) in a projection
|
||||
without re-implementing the shard's ACL engine?
|
||||
4. Should the adapter contract's **storage-module shape** be standardized directly on
|
||||
Wiki.js + Foswiki::Store, or generalized further?
|
||||
|
||||
---
|
||||
|
||||
## 10. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Wiki.js — GraphQL API docs (https://docs.requarks.io/dev/api) | Everything via GraphQL; `/graphql` Playground; access/modify all resources |
|
||||
| Wiki.js — Git storage (https://docs.requarks.io/storage/git) | Per-change commit to git; clean `.md`; bidirectional sync with remote repo |
|
||||
| Wiki.js — Storage overview + dev/storage.md (https://docs.requarks.io/storage ; https://github.com/requarks/wiki-docs/blob/master/dev/storage.md) | Storage module = provider connector; backup-or-source-of-truth; properties + create/update/delete/sync methods; `injectMetadata()` YAML frontmatter; providers Git/FS/S3/Azure |
|
||||
| Wiki.js 3.0 storage preview (https://beta.js.wiki/blog/2021-wiki-js-3-feature-preview-storage) | Storage subsystem direction (3.0) |
|
||||
| Wikipedia — Wiki.js (https://en.wikipedia.org/wiki/Wiki.js) | Node.js/Vue, DB options, formats, modular architecture |
|
||||
| BrightCoding / Grokipedia overviews | Auth providers, access rules, editors, read replicas |
|
||||
|
||||
Cross-references: `research/260613-foswiki-deep-dive/findings.md` (Foswiki::Store, the
|
||||
other storage-interface prior art), `research/260614-joplin-deep-dive/findings.md`
|
||||
(interchange/sync-mirror contrast), `research/260614-notion-deep-dive/findings.md`
|
||||
(external-API/REST contrast), `research/260614-shard-spectrum-synthesis/findings.md`
|
||||
(attachment-mode spectrum), `spec/UseCaseCatalog.md` (UC-02, UC-06, UC-36, UC-38, UC-40,
|
||||
UC-42, UC-57), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
|
||||
|
||||
---
|
||||
|
||||
## 11. Traceability
|
||||
|
||||
- New UCs: **UC-68, UC-69** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-06, UC-36, UC-38, UC-40, UC-42, UC-57** (links UC-02, authz decision).
|
||||
- Architecture (no UC): **storage-module abstraction = 2nd adapter-contract prior art**
|
||||
(with Foswiki::Store); engine-maintained Git mirror as attach+write surface; GraphQL
|
||||
introspection → capability discovery + selective projection; path-based ACL + authn-
|
||||
delegated → `SHARD-WP-0002` (T11, T14).
|
||||
- Boundary recorded: Wiki.js is **one server-engine candidate shard**, DB-canonical with a
|
||||
**bidirectional Git-of-Markdown mirror** that is the preferred attach surface; not a
|
||||
substrate and not the federation layer (don't double-sync; honor its access rules).
|
||||
</content>
|
||||
47
research/260614-xanadu-deep-dive/README.md
Normal file
47
research/260614-xanadu-deep-dive/README.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# 260614 — Project Xanadu deep dive (the docuverse, the EDL, reference-not-copy)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Project Xanadu** (Ted Nelson) read through shard-wiki's lens.
|
||||
Unlike the engine dives (`xwiki`, `twiki`, `foswiki`), Xanadu is **not a candidate
|
||||
shard backend** — it never shipped at scale and there is nothing to attach. It is
|
||||
studied as the **deepest conceptual ancestor** of shard-wiki's own model: several
|
||||
shard-wiki primitives turn out to be Xanadu mechanisms under different names.
|
||||
|
||||
The distinctive material:
|
||||
- the **EDL / xanadoc** — a document that contains *no content*, only an ordered list
|
||||
of **spans** (content references) plus **xanalinks** (separate link tables); the
|
||||
client assembles the page by reference → this is shard-wiki **projection** + **union**
|
||||
- the storage substrate — **tumblers** (stable fine-grained addresses), **istream**
|
||||
(invariant content pool), **enfilades / spanfilade** (virtual↔content mapping,
|
||||
version comparison by span-set intersection)
|
||||
- **content-identity, bidirectional transclusion** — content "knowably in more than
|
||||
one place" that remembers all its appearances → stronger than UC-32
|
||||
- **transcopyright / micropayment** — a baked-in rights policy shard-wiki must keep
|
||||
*configurable*, not inherit
|
||||
|
||||
Goes deliberately **underneath** the surface pattern table in
|
||||
`research/260608-federation-concepts/findings.md` §3.3, which it extends rather than
|
||||
repeats.
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | EDL/xanadoc, addressing substrate, transclusion, versioning, rights, INTENT mapping (reinforcements + design-bug divergences), UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-44 compose-by-reference, UC-45 reverse transclusion, UC-46 content-identity
|
||||
equivalence); UC-24/27/29/32 enriched. Span-addressing as an adapter capability,
|
||||
content identity, composition manifests, and reuse-terms metadata logged as
|
||||
architecture for `workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
|
||||
Key boundary recorded: shard-wiki inherits Xanadu's **reference-not-copy** mechanisms
|
||||
but **rejects** the single-global-docuverse premise, the single-canonical-instance
|
||||
model, and the baked-in economic policy — those would violate shard sovereignty,
|
||||
parallel-version support (UC-27), and mechanism-over-policy.
|
||||
</content>
|
||||
322
research/260614-xanadu-deep-dive/findings.md
Normal file
322
research/260614-xanadu-deep-dive/findings.md
Normal file
@@ -0,0 +1,322 @@
|
||||
# Findings — Project Xanadu: the docuverse, the EDL, and reference-not-copy
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **conceptual / architectural prior art** (not a deployable engine, not a
|
||||
candidate shard backend)
|
||||
Lens: shard-wiki orchestration — projection, overlay, transclusion, provenance,
|
||||
addressing, coordination journal
|
||||
|
||||
> Reading guide. Every previous deep dive (`xwiki`, `twiki`, `foswiki`) studied a
|
||||
> *shippable engine* as a candidate **shard**. Xanadu is different: it is the
|
||||
> **deepest conceptual ancestor** of shard-wiki's own model and is *not* a backend
|
||||
> we would ever attach. It is studied here for its **mechanisms** —
|
||||
> reference-not-copy documents, separated link tables, content-identity transclusion,
|
||||
> stable fine-grained addressing — several of which shard-wiki already reinvents
|
||||
> under different names (projection, overlay, union BackLinks, coordination journal).
|
||||
> The job of this file is to (a) name what shard-wiki inherits, (b) flag what would
|
||||
> be a **design bug** to inherit, and (c) extract use cases that are genuinely new
|
||||
> versus the federation track already in the catalog.
|
||||
|
||||
Pairs with — and deliberately extends, does not repeat —
|
||||
`research/260608-federation-concepts/findings.md` §3.3, which treats Xanadu only as a
|
||||
six-row pattern table and (correctly) labels it "speculative design / pattern
|
||||
language, not deployable federation." This dive goes underneath those patterns into
|
||||
the actual data architecture (EDL, istream, enfilades, tumblers) because *that* is
|
||||
where the resonance with shard-wiki's storage-neutral page model lives.
|
||||
|
||||
---
|
||||
|
||||
## 1. Origin and status
|
||||
|
||||
Project Xanadu (Ted Nelson, 1960– ; the term *hypertext* and *transclusion* are his)
|
||||
is the original electronic-literature system: a global **docuverse** of permanent,
|
||||
addressable documents joined by visible, two-way links. It famously never shipped at
|
||||
scale; partial artifacts exist — **Udanax Green / Udanax Gold** (source released 1999
|
||||
to establish prior art against patents), **XanaduSpace**, and **OpenXanadu** (2014, a
|
||||
small working browser demo built on the EDL format). Treat all of these as
|
||||
*reference designs*, not running software to integrate.
|
||||
|
||||
Why it matters to shard-wiki: Nelson's critique of the Web — **"one-way,
|
||||
ever-breaking links and no management of version or contents"** — is precisely the
|
||||
gap shard-wiki addresses for wikis. shard-wiki is, in effect, a *pragmatic, federated,
|
||||
Git-backed, heterogeneous-backend* descendant of the Xanadu idea, scoped to
|
||||
Markdown wiki pages and explicitly **refusing** Xanadu's one-global-store premise.
|
||||
|
||||
---
|
||||
|
||||
## 2. The headline insight — a document is a manifest of spans, not content
|
||||
|
||||
The single most shard-wiki-relevant Xanadu artifact is the **EDL (Edit Decision
|
||||
List)**, also called a **xanadoc**:
|
||||
|
||||
- A xanadoc is a *connected document that contains no original content*. It is a file
|
||||
listing (a) **spans** — "portions of content to bring in" addressed in remote
|
||||
sources — and (b) **xanalinks** — tables saying what to connect to what.
|
||||
- The client reads the EDL, **fetches each span from its source**, and **assembles**
|
||||
the document locally per the xanalinks. Content is referenced, never embedded.
|
||||
- Structure is strictly separated: **spans first** (content references), **xanalinks
|
||||
after** (relationships). Content and link structure live apart.
|
||||
|
||||
This is *the same move* shard-wiki makes:
|
||||
|
||||
| Xanadu (xanadoc / EDL) | shard-wiki concept |
|
||||
|------------------------|--------------------|
|
||||
| Document = ordered list of span references, no embedded content | **Projection** — lazy, cache-like view assembled from shard storage, not a copy |
|
||||
| Spans pulled from multiple remote sources at view time | **Union of pages** across heterogeneous shards |
|
||||
| Xanalinks stored separately from content | **Overlay** — non-destructive edits/annotations as separate objects, "overlay before mutation" |
|
||||
| "All media should be permanized and addressable" | **Coordination journal** + Git content-addressable storage |
|
||||
|
||||
shard-wiki's INTENT lines — *"prefer lazy projection over eager copying"* and *"union
|
||||
without erasure"* — are the xanadoc principle restated for wikis.
|
||||
|
||||
---
|
||||
|
||||
## 3. Addressing and the storage substrate
|
||||
|
||||
Xanadu's machinery underneath the EDL:
|
||||
|
||||
- **Tumblers** (Miller / Gregory): a transfinite-number addressing scheme where one
|
||||
address simultaneously encodes **machine, author, document version, byte span, and
|
||||
links**. A single tumbler is a stable, fine-grained, hierarchical pointer into the
|
||||
whole docuverse.
|
||||
- **istream (invariant stream):** a growing pool of *shareable content pieces*.
|
||||
Documents do not store text; they reference istream content by virtual address. A
|
||||
family of **enfilades** (tree structures using *dsps* = relative displacements and
|
||||
*wids* = ranges) maps **virtual addresses ↔ istream addresses** bidirectionally.
|
||||
- **Spanfilade:** indexes which istream spans each document uses. Other filades:
|
||||
**granfilade** (storage across disks/network), **POOMfilade** (permutation-of-order
|
||||
matrix mapping document position → istream location).
|
||||
- Implicit substrate: content pieces are **invariant** (append-only, never rewritten);
|
||||
versions are new arrangements of shared, permanent pieces. Nothing is deleted, so
|
||||
references never break.
|
||||
|
||||
shard-wiki mapping:
|
||||
|
||||
- **istream + invariance ≈ Git's content-addressable, immutable blob store**, and the
|
||||
**coordination journal** as the append-only record. shard-wiki gets "references
|
||||
never break" *for Git-native shards for free*; for non-Git shards it must
|
||||
approximate via the journal/projection cache.
|
||||
- **Tumbler ≈ the still-open question of a portable, stable, fine-grained span
|
||||
address** that survives projection, overlay, and versioning across heterogeneous
|
||||
backends. shard-wiki has no such addressing scheme yet; this is the hardest part
|
||||
Xanadu solved on paper and the part that never shipped. (See §10, §11.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Links and transclusion — separate, two-way, content-identity
|
||||
|
||||
- **Links are first-class objects stored apart from content** (the xanalinks tables),
|
||||
and are **visible and followable from all endpoints** (rule 7) — i.e. inherently
|
||||
**bidirectional**. Backlinks are not derived after the fact; they are the same
|
||||
object seen from the other end.
|
||||
- **Transclusion** = Nelson's "the same content *knowably* in more than one place."
|
||||
The content piece **remembers its identity and can trace back to all its
|
||||
appearances**. This is the crucial delta from every weak modern form (server-side
|
||||
includes, HTML embeds, Roam block-refs): those *copy or cache*; Xanadu keeps one
|
||||
permanent instance that is **aware of where it is reused**.
|
||||
|
||||
shard-wiki mapping:
|
||||
|
||||
- **Separate link tables ≈ overlays and union BackLinks as first-class.** An overlay
|
||||
is shard-wiki's xanalink: a non-destructive object spanning content it does not own.
|
||||
- **Content-identity, bidirectional transclusion** is *stronger* than UC-32 as
|
||||
currently written (which is path/span fetch with freshness). Xanadu says the
|
||||
*content itself* knows its appearances → enables **reverse transclusion** ("where is
|
||||
this paragraph used across all shards?") and **content-identity equivalence** ("these
|
||||
two pages in different shards are versions of the same content"). These are new
|
||||
capabilities, surfaced as UC-45 and UC-46 below.
|
||||
|
||||
---
|
||||
|
||||
## 5. Versioning and comparison — by span-set intersection
|
||||
|
||||
Because a document is a set of spans over an invariant pool, **comparing two documents
|
||||
or two versions = intersecting their span-sets** (the spanfilade operation). Shared
|
||||
spans are literally shared subtrees; differences fall out of the set arithmetic. The
|
||||
same operation surfaces links between documents.
|
||||
|
||||
shard-wiki mapping: this is a **content-identity diff/merge** model. shard-wiki's
|
||||
diff/merge capability (an adapter capability in the contract) is today implicitly
|
||||
path/title- and text-based. Xanadu shows a path-independent alternative: detect that
|
||||
page A in shard X and page B in shard Y are *the same or derived content* by span
|
||||
overlap, **without relying on matching titles or paths**. This directly serves UC-27
|
||||
(view parallel versions of equivalent pages) by giving an *equivalence-detection
|
||||
mechanism* rather than assuming naming conventions align across sovereign shards.
|
||||
|
||||
---
|
||||
|
||||
## 6. Rights and economics — transcopyright, micropayment, implicit permission
|
||||
|
||||
Three of the 17 rules are an economic/rights layer:
|
||||
|
||||
- Rule 8: **publication grants implicit permission to link/transclude.**
|
||||
- Rule 9: **granular royalty / micropayment** on any accessed portion.
|
||||
- **Transcopyright:** pre-granted permission for virtual republication *by reference*,
|
||||
with the attribution chain to origin preserved automatically because the content is
|
||||
transcluded, not copied.
|
||||
|
||||
shard-wiki mapping — handle with the **mechanism-over-policy** rule:
|
||||
|
||||
- The *mechanism* shard-wiki should provide: a reference (overlay/transclusion) that
|
||||
**carries provenance and reuse terms with it**, so attribution and permission travel
|
||||
with the span. This strengthens UC-29 (remix with portable attribution).
|
||||
- The *policy* (whether reuse requires permission, whether anything is metered) must
|
||||
**not be hard-coded**. Xanadu baked one global economic policy into the substrate;
|
||||
shard-wiki keeps editorial/economic policy configurable. Baking in a payments or
|
||||
permission model would be a design-bug per INTENT.
|
||||
- This intersects the **settled authz-in-core / authn-delegated decision**
|
||||
([[shard-wiki-auth-in-core-decision]]): "publication grants implicit linking
|
||||
permission" is exactly a *policy* that an information space might or might not adopt
|
||||
on its L0→L4 ladder. shard-wiki must be able to *represent* the permission on a
|
||||
transclusion; it must not *assume* Xanadu's answer.
|
||||
|
||||
---
|
||||
|
||||
## 7. Xanadu through shard-wiki's lens — it is architecture, not a shard
|
||||
|
||||
Unlike the engine dives, there is **no capability profile to fill in** — Xanadu is not
|
||||
a backend you attach. The useful framing is the inverse: *which shard-wiki primitives
|
||||
are Xanadu mechanisms in disguise?*
|
||||
|
||||
| shard-wiki primitive | Xanadu mechanism | Inheritance verdict |
|
||||
|----------------------|------------------|---------------------|
|
||||
| Projection (lazy view, no copy) | xanadoc / EDL span assembly | **Inherit** — same idea, scope to wiki pages |
|
||||
| Overlay (non-destructive edit) | xanalinks as separate objects | **Inherit** — overlays are xanalinks |
|
||||
| Union BackLinks | two-way links visible from all endpoints | **Inherit** — generalize to sub-page spans |
|
||||
| Transclusion (UC-32) | content-identity, content-aware reuse | **Inherit, strengthen** — add reverse + equivalence |
|
||||
| Coordination journal | invariant istream / permanent storage | **Inherit** (Git gives this for Git-native shards) |
|
||||
| Stable span address | tumblers | **Aspire** — hard, partly unshippable; degrade gracefully |
|
||||
| Editorial/economic policy | transcopyright + micropayment | **Reject as substrate** — keep configurable |
|
||||
| One global docuverse | the docuverse premise | **Reject** — see §8.2 |
|
||||
|
||||
---
|
||||
|
||||
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 8.1 Reinforcements
|
||||
|
||||
- **Reference-not-copy** is the core of both. Xanadu validates shard-wiki's
|
||||
"lazy projection over eager copying" and "union without erasure" as a coherent,
|
||||
decades-deep design lineage, not an ad-hoc preference.
|
||||
- **Provenance is structural, not decorative.** In Xanadu, knowing where content came
|
||||
from and where it is reused is *built into the storage model*. shard-wiki's "never
|
||||
hide authorship, conflicts, freshness, backend limits" is the same commitment.
|
||||
- **Links/edits as separate objects** validates overlays-before-mutation and
|
||||
first-class BackLinks.
|
||||
|
||||
### 8.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **No single docuverse / no universal address space.** Xanadu requires one global
|
||||
permanent store, universal addressing, and its own published client–server
|
||||
protocol (rules 12, 17). shard-wiki's whole reason to exist is the opposite:
|
||||
**federate heterogeneous, sovereign backends** (Git repos, Gitea, Obsidian,
|
||||
WebDAV, Coulomb), each keeping its own storage, history, identity, and limits.
|
||||
Adopting Xanadu's universality would violate **shard sovereignty** and **graceful
|
||||
degradation**. shard-wiki's addressing must tolerate backends that *cannot* offer
|
||||
stable span addresses, treating fine-grained addressing as an **adapter
|
||||
capability**, not a precondition.
|
||||
2. **One canonical content instance vs. parallel divergent versions.** Xanadu
|
||||
transclusion centers a *single source of truth*. shard-wiki must support
|
||||
**equivalent-but-divergent** pages across shards with conflicts *visible* (UC-27),
|
||||
not collapsed into one instance. Content-identity is a *detection* tool here, not a
|
||||
mandate to unify.
|
||||
3. **Baked-in economic/permission policy.** See §6 — reject as substrate, keep as
|
||||
configurable policy.
|
||||
4. **Permanence as a hard requirement.** Xanadu forbids deletion globally. shard-wiki
|
||||
cannot impose that on sovereign shards; permanence holds for the **coordination
|
||||
journal** and Git-native shards, and is *approximated* (cache/projection/snapshot)
|
||||
for backends that delete. This is graceful degradation, not a weaker promise.
|
||||
|
||||
### 8.3 What Xanadu teaches that shard-wiki should not lose
|
||||
|
||||
- A page view can legitimately be a **composition manifest** (a list of references),
|
||||
not a file. Build the page model so a page *can* be authored as references into
|
||||
other shards (UC-44), even if the common case is a plain Markdown file.
|
||||
- **Content identity** (not path/title) is the durable basis for transclusion,
|
||||
equivalence, and reverse lookup across sovereign shards (UC-45, UC-46). Lean on
|
||||
Git blob hashes where available; define a content-fingerprint fallback elsewhere.
|
||||
- Getting **stable fine-grained addressing** right is the hard, valuable, historically
|
||||
*unshippable* part. Scope it as an adapter capability with explicit degradation
|
||||
rather than promising tumbler-grade universality.
|
||||
|
||||
---
|
||||
|
||||
## 9. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-43**. New UCs **UC-44–UC-46** added; several existing UCs
|
||||
enriched (no new scenario, stronger mechanism/notes).
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Compose-by-reference page** — author a page whose body is an ordered list of spans pulled from multiple shards, stored as a manifest (xanadoc/EDL), not a copy | **UC-44 (new)** |
|
||||
| **Reverse transclusion** — find every page/shard where a given span (paragraph/section) appears or is transcluded | **UC-45 (new)** |
|
||||
| **Content-identity equivalence** — detect that two pages in different shards are the same or derived content via span/content overlap, without matching titles or paths | **UC-46 (new)** |
|
||||
| Content-identity, content-aware (bidirectional) transclusion | **enriches UC-32** |
|
||||
| Equivalence detection mechanism for parallel versions | **enriches UC-27** |
|
||||
| Reference carries provenance + reuse terms (transcopyright as representable policy) | **enriches UC-29**; links [[shard-wiki-auth-in-core-decision]] |
|
||||
| Provenance: content remembers its appearances | **enriches UC-24** |
|
||||
|
||||
---
|
||||
|
||||
## 10. Adapter-contract / architecture notes for SHARD-WP-0002
|
||||
|
||||
Logged as architecture (no UC):
|
||||
|
||||
- **Fine-grained span addressing is an adapter capability**, not a core assumption.
|
||||
The shard adapter contract should model whether a shard can mint a *stable address
|
||||
for a sub-page span* that survives edits/versions (tumbler-grade), down through
|
||||
whole-page-only, down to path-only. Transclusion/overlay capabilities depend on it.
|
||||
- **Content identity** should be a contract-level concept: a shard advertises how it
|
||||
fingerprints content (Git blob hash, normalized-text hash, none). UC-45/UC-46 and
|
||||
cross-shard diff/merge consume it.
|
||||
- **Composition manifests** (UC-44) imply the wiki page model must permit a page whose
|
||||
canonical form is a reference list. This is an INTENT-level page-model decision —
|
||||
flag for the page-model spec, not just the adapter contract.
|
||||
- Reuse-terms metadata on a reference (UC-29 / §6) is **policy data the core carries
|
||||
but does not interpret** — consistent with mechanism-over-policy and the L0→L4
|
||||
authz ladder.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions (for spec / workplans)
|
||||
|
||||
1. What is shard-wiki's **portable span address**? Git blob+range works for Git-native
|
||||
shards; what is the fallback for Obsidian/WebDAV/Gitea-wiki, and how does it
|
||||
survive a shard's storage swap (cf. UC-43 Foswiki RCS↔PlainFile)?
|
||||
2. Is **compose-by-reference (UC-44)** core orchestrator, adapter-provided, or
|
||||
reference-UI — and is it MVP or deferred with UC-32?
|
||||
3. Does **content-identity equivalence (UC-46)** belong in core (cross-shard union
|
||||
logic) or as an adapter-provided index? How expensive is span-set intersection at
|
||||
wiki scale without enfilades?
|
||||
4. How far do we take **reverse transclusion (UC-45)** — exact span identity only, or
|
||||
fuzzy/derived-content tracking? The latter is research-grade.
|
||||
5. Where does **reuse-terms/transcopyright metadata** sit on the L0→L4 ladder, and
|
||||
does any tier ever *enforce* it, or is it always advisory provenance?
|
||||
|
||||
---
|
||||
|
||||
## 12. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Wikipedia — Project Xanadu (https://en.wikipedia.org/wiki/Project_Xanadu) | 17 rules, tumblers, history, Web critique |
|
||||
| xanadu.com — The Edit Decision List / Xanadoc File (https://xanadu.com/xuEDL.html) | EDL/xanadoc structure: spans + xanalinks, client assembly |
|
||||
| Wikipedia — Enfilade (Xanadu) (https://en.wikipedia.org/wiki/Enfilade_(Xanadu)) | istream, enfilades (dsp/wid), spanfilade, granfilade, POOMfilade, version comparison by span-set intersection |
|
||||
| Wikipedia — Transclusion (https://en.wikipedia.org/wiki/Transclusion) | "same content knowably in more than one place," content-aware reuse, transcopyright, micropayment vs. weak modern forms |
|
||||
| Maggie Appleton — Xanadu Patterns (https://maggieappleton.com/xanadu-patterns) | pattern naming: visible links, parallel documents, transpointing windows, modular blocks, stable addresses, annotation |
|
||||
|
||||
Cross-references: `research/260608-federation-concepts/findings.md` §3.3 (prior
|
||||
surface treatment), `spec/UseCaseCatalog.md` (UC-24, UC-27, UC-29, UC-32),
|
||||
`workplans/SHARD-WP-0002-federation-architecture.md` (adapter contract).
|
||||
|
||||
---
|
||||
|
||||
## 13. Traceability
|
||||
|
||||
- New UCs: **UC-44, UC-45, UC-46** → `spec/UseCaseCatalog.md` (section F, federation).
|
||||
- Enriched UCs: **UC-24, UC-27, UC-29, UC-32**.
|
||||
- Architecture (no UC): span-addressing capability, content-identity, composition
|
||||
manifest, reuse-terms metadata → `SHARD-WP-0002`.
|
||||
- Decision link: transcopyright-as-policy → [[shard-wiki-auth-in-core-decision]].
|
||||
</content>
|
||||
</invoke>
|
||||
43
research/260614-zigzag-deep-dive/README.md
Normal file
43
research/260614-zigzag-deep-dive/README.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# 260614 — ZigZag / zzstructure deep dive (an information space as orthogonal dimensions)
|
||||
|
||||
Date: 2026-06-14
|
||||
|
||||
## What this is
|
||||
|
||||
A focused study of **Ted Nelson's ZigZag** and its data model **zzstructure**, asked
|
||||
through a specific shard-wiki question: a wiki page is linked in many independent ways
|
||||
at once — a **namespace** tree, a **created-from / genealogy** lineage, **hyperlinks**,
|
||||
plus shard-wiki's own **shard / overlay / version / transclusion / equivalence**
|
||||
relations. Could the information space be modelled as a zzstructure, where each
|
||||
relationship is a first-class **dimension** and none is privileged?
|
||||
|
||||
Distinctive material:
|
||||
- the zzstructure model — **cells**, **dimensions** (edge colours), **ranks**,
|
||||
**posward/negward**, and the one rule that defines it: *at most one posward and one
|
||||
negward neighbour per dimension per cell* (a directed coloured multigraph)
|
||||
- **views** — I-view (one dimension), **H-view** (pivot two dimensions into a cross),
|
||||
raster-vs-view
|
||||
- trees/lists/tables as *spent dimensions*; many-to-many graphs as the hard case
|
||||
(handled by **clones**)
|
||||
- **clone ↔ transclusion** convergence with the Xanadu dive
|
||||
|
||||
## Contents
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `findings.md` | zzstructure model, views, fit analysis, a proposed dimension map for a shard-wiki space, recommendation (lens not store), INTENT mapping, UC seeds, architecture notes, sources |
|
||||
|
||||
## Status
|
||||
|
||||
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
|
||||
(UC-47 navigate-by-dimension, UC-48 H-view pivot/cross-tab, UC-49 genealogy
|
||||
dimension); UC-05/17–22/26/29 enriched. A **dimensional projection layer** is logged
|
||||
as architecture for `workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
|
||||
**Recommendation recorded:** adopt zzstructure as a **navigation / visualization /
|
||||
indexing lens** (a derived dimensional projection over the union), **not** as the
|
||||
storage substrate — Git and sovereign shards remain canonical (INTENT Stability Note).
|
||||
The many-to-many hyperlink graph does not fit zzstructure's rank constraint and stays a
|
||||
separate graph index. Pairs with `research/260614-xanadu-deep-dive/` (clone ↔
|
||||
transclusion).
|
||||
</content>
|
||||
294
research/260614-zigzag-deep-dive/findings.md
Normal file
294
research/260614-zigzag-deep-dive/findings.md
Normal file
@@ -0,0 +1,294 @@
|
||||
# Findings — ZigZag / zzstructure: an information space as orthogonal dimensions
|
||||
|
||||
Date: 2026-06-14
|
||||
Source kind: **conceptual / data-model prior art** (a data model + visualization
|
||||
paradigm, not a deployable wiki engine, not a candidate shard backend)
|
||||
Lens: shard-wiki — modelling a wiki information space where a page participates in
|
||||
*many* relationship structures at once (namespace, genealogy, links, versions, shards)
|
||||
|
||||
> Motivating question (from the user). A wiki page is linked in several independent
|
||||
> ways at once: a **primary namespace** association giving a tree-like hierarchy; a
|
||||
> **created-from / genealogy** association (fork/derivation lineage); **hyperlinks**
|
||||
> to other pages; and — in shard-wiki — *which shard* it lives in, its **overlays**,
|
||||
> its **versions**, its **transclusions**, and its **equivalents** elsewhere. Could
|
||||
> the information space be modelled as Ted Nelson's **ZigZag / zzstructure**, where
|
||||
> each kind of relationship is a separate, first-class **dimension** and no single
|
||||
> relationship is privileged as *the* structure?
|
||||
>
|
||||
> Short answer: **yes as a navigation / visualization / indexing lens; no as the
|
||||
> storage substrate.** zzstructure is an unusually good *conceptual* match for
|
||||
> shard-wiki's "union without erasure" thesis — it makes every relationship
|
||||
> co-equal and separately navigable — but its one-neighbour-per-direction
|
||||
> constraint makes it a poor *direct* encoding of the many-to-many hyperlink graph,
|
||||
> and Git/heterogeneous shards must remain the canonical store. Details below.
|
||||
|
||||
Companion to the Xanadu dive (`research/260614-xanadu-deep-dive/`): ZigZag is
|
||||
Nelson's *other* lifelong project, the structural/visual counterpart to Xanadu's
|
||||
literary/addressing one. Where Xanadu gives shard-wiki **reference-not-copy**, ZigZag
|
||||
gives it **many co-equal structures over the same cells**.
|
||||
|
||||
---
|
||||
|
||||
## 1. What ZigZag / zzstructure is
|
||||
|
||||
ZigZag (Ted Nelson, conceived 1981 at Datapoint; first Perl prototype by Andrew Pam,
|
||||
1997; the **GZigZag / Gzz** Java project 2000–2003; US patent 6,262,736, expired
|
||||
2019) is **"a data model that deconstructs the spreadsheet to allow irregular
|
||||
relations, generalizing the idea to multiple dimensions."** The generic, vendor-free
|
||||
name for the data model is **zzstructure**.
|
||||
|
||||
Core vocabulary:
|
||||
|
||||
- **Cell (zzcell)** — an atom of data (a value, a record, a page). Cells are the only
|
||||
content; everything else is connection.
|
||||
- **Dimension** — a *named kind of connection* (an edge colour). Dimensions are
|
||||
**first-class** — in ZigZag, dimensions are themselves cells.
|
||||
- **Link (zzlink)** — a connection between two cells *along one dimension*, with a
|
||||
**posward** (positive) and **negward** (negative) direction.
|
||||
- **Rank** — the maximal chain of cells along one dimension (a path, or a cycle). A
|
||||
rank is ZigZag's equivalent of "a row" / "a list" — but it exists only along one
|
||||
named dimension.
|
||||
- **Cursor** — the current focus cell from which views are drawn.
|
||||
- **Clone** — a copy of a cell that shares identity with its head cell (along a
|
||||
special clone dimension), used to let one logical item appear in several places /
|
||||
participate in many-to-one relations.
|
||||
|
||||
---
|
||||
|
||||
## 2. The one rule that defines everything
|
||||
|
||||
Formally (McGuffin & schraefel, HT2004; McGuffin's graph-theoretic intro), a
|
||||
zzstructure is:
|
||||
|
||||
> a **directed multigraph whose edges are coloured (typed)**, subject to the
|
||||
> restriction that **each node has at most one incoming edge of each colour and at
|
||||
> most one outgoing edge of each colour.**
|
||||
|
||||
Equivalently: **along any single dimension, a cell has at most one posward and one
|
||||
negward neighbour.** That single constraint is the whole design:
|
||||
|
||||
- Each dimension decomposes the cell set into disjoint **ranks** (paths and cycles) —
|
||||
never a branching tree, never a fan-out, *within one dimension*.
|
||||
- Many dimensions coexist over the **same** cells, each imposing its own independent
|
||||
rank structure. A cell sits at the intersection of as many dimensions as it
|
||||
participates in.
|
||||
- There is **no inherent hierarchy.** A tree is not the substrate; it is *one
|
||||
possible dimension* (encoded left-child/right-sibling — see §4).
|
||||
|
||||
This is the property that matters for shard-wiki: **the same set of pages can carry
|
||||
many orthogonal structures simultaneously, with none privileged.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Views — how multiple dimensions are seen
|
||||
|
||||
Because no screen can show all dimensions, ZigZag *rotates* two into view at a time:
|
||||
|
||||
- **I-view** — cells along **one** dimension from the cursor, drawn as a line.
|
||||
- **H-view** — a 2-D cross: **two** dimensions through the cursor, one drawn
|
||||
horizontally and one vertically; only cells actually connected to the cursor along
|
||||
those two dimensions appear (empty spreadsheet cells simply do not exist).
|
||||
- **I+/H+ (augmented) views** — fill remaining space with further neighbours.
|
||||
- **Raster vs. view (terminology)** — a *raster* selects which cells to take from the
|
||||
structure; a *view* places them on screen. "Pivoting" rotates a different dimension
|
||||
into the horizontal or vertical slot.
|
||||
|
||||
The H-view "pivot" is the genuinely interesting UX idea for a *federated* wiki: put
|
||||
**pages of a namespace** on one axis and **shards** (or **versions across shards**)
|
||||
on the other, then rotate to re-cross-tabulate the same union by a different pair of
|
||||
relationships.
|
||||
|
||||
---
|
||||
|
||||
## 4. Trees, lists, tables, graphs — what fits and what does not
|
||||
|
||||
zzstructure subsumes simpler structures by spending dimensions:
|
||||
|
||||
- **List** = one rank along one dimension.
|
||||
- **Table** = two dimensions (rows / columns), shown as an H-view.
|
||||
- **Tree** = encoded as a zzstructure "analogous to the left-child, right-sibling
|
||||
pointer implementation" — i.e. a *parent→firstchild* dimension plus a
|
||||
*sibling→sibling* dimension. So a hierarchy is **two dimensions**, not a primitive.
|
||||
- **Many-to-one / many-to-many** = NOT directly expressible (the one-edge-per-colour
|
||||
rule forbids fan-out). It is simulated with **clones**: the head cell links to many
|
||||
clone cells along the clone dimension, and each clone participates in a different
|
||||
rank.
|
||||
|
||||
**This is the critical caveat for wikis.** A namespace hierarchy (each page has *one*
|
||||
parent) and a created-from genealogy (each page has *one* origin) fit zzstructure
|
||||
*natively* and beautifully. But a wiki's **hyperlink graph is many-to-many** (a page
|
||||
links to many pages and is linked from many) — it is *not* a rank and cannot be one
|
||||
dimension. Representing links/backlinks in zzstructure requires cloning or treating
|
||||
the link set as data, which is awkward. zzstructure is excellent for the
|
||||
**functional/sequential/hierarchical** relationships and weak for the **arbitrary
|
||||
graph** ones.
|
||||
|
||||
---
|
||||
|
||||
## 5. Mapping a shard-wiki information space onto dimensions
|
||||
|
||||
Taking the user's framing literally — model the union of pages as cells, and each
|
||||
relationship as a dimension:
|
||||
|
||||
| Dimension (proposed) | Rank meaning | Fits zzstructure? | shard-wiki source |
|
||||
|----------------------|--------------|-------------------|-------------------|
|
||||
| `d.namespace-child` + `d.namespace-sibling` | namespace tree (left-child/right-sibling) | **Yes** (2 dims) | UC-22, yawex topics |
|
||||
| `d.genealogy` | created-from / fork lineage (each page one origin) | **Yes** (functional) | UC-26 fork, UC-29 remix |
|
||||
| `d.version` | revision order of a page | **Yes** (a rank = a history) | coordination journal |
|
||||
| `d.shard` | pages grouped by owning shard | **Yes** | shard sovereignty |
|
||||
| `d.overlay` | page → its overlays | **Yes** (or clones) | overlay model |
|
||||
| `d.equivalence` | parallel versions of the same topic across shards | **Yes** (a rank of equivalents) | UC-27, UC-46 |
|
||||
| `d.recent` | temporal order of changes | **Yes** | UC-17 RecentChanges |
|
||||
| `d.alphabetical` | AllPages ordering | **Yes** | UC-19 |
|
||||
| `d.links` / `d.backlinks` | hyperlink graph | **No** (many-to-many) — needs clones / stays a graph index | UC-05, UC-18 |
|
||||
| `d.transclusion` | content reused in many places | **No** as rank — but maps to **clones** (= "same content in many places") | UC-32, UC-44/45 |
|
||||
|
||||
Observation: shard-wiki's existing **derived views** (UC-05, UC-17–UC-20) are, in
|
||||
zzstructure terms, *dimensions + rasters*: RecentChanges is a raster along `d.recent`,
|
||||
AllPages along `d.alphabetical`, SiteMap is an H-view over the two namespace
|
||||
dimensions. zzstructure offers a **single unifying vocabulary** for what shard-wiki
|
||||
currently treats as a grab-bag of separate derived views.
|
||||
|
||||
Observation 2: ZigZag **clones** are conceptually the same move as Xanadu
|
||||
transclusion ("the same content knowably in more than one place") — the two Nelson
|
||||
projects converge here. A transcluded span / equivalent page is a clone that
|
||||
participates in multiple ranks.
|
||||
|
||||
---
|
||||
|
||||
## 6. Is zzstructure the right model for shard-wiki? (recommendation)
|
||||
|
||||
**Adopt as a lens, not a substrate.**
|
||||
|
||||
- **As a navigation & visualization model: promising.** Letting a reader pick which
|
||||
*dimension* to traverse (namespace vs. genealogy vs. version vs. shard) and pivot
|
||||
two dimensions into an H-view is a real, differentiated UX for a federated,
|
||||
provenance-preserving wiki. It operationalizes "union without erasure": every
|
||||
relationship is co-equal and visible, none hidden. (UC-47, UC-48.)
|
||||
- **As a conceptual model for the page graph: useful and clarifying.** Reframing the
|
||||
scattered derived views as *dimensions over one cell set* is a clean mental model
|
||||
and a candidate internal index/API shape. It also de-privileges the namespace tree
|
||||
(UC-49) — consistent with **mechanism over policy** (the hierarchy is one
|
||||
configurable dimension, not the canonical structure).
|
||||
- **As the storage substrate: reject.** Git and heterogeneous sovereign shards remain
|
||||
canonical (per INTENT Stability Note — Git's role is architectural). A zzstructure
|
||||
is a **derived, projected index** computed over the union, like BackLinks today —
|
||||
not a database that replaces the shards. The many-to-many hyperlink graph does not
|
||||
fit the rank constraint, and forcing the wiki into a zz-DB would violate shard
|
||||
sovereignty and the orchestrator-not-engine boundary.
|
||||
|
||||
Net: zzstructure is a strong answer to *"how do we present a page's many
|
||||
simultaneous relationships without picking a winner?"* — a **dimensional projection**
|
||||
layer over the existing union, not a new store.
|
||||
|
||||
---
|
||||
|
||||
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
|
||||
|
||||
### 7.1 Reinforcements
|
||||
|
||||
- **Union without erasure ↔ co-equal dimensions.** zzstructure's refusal to privilege
|
||||
any one structure is the data-model expression of shard-wiki's refusal to hide
|
||||
provenance, conflicts, or alternative arrangements.
|
||||
- **Mechanism over policy ↔ hierarchy-as-just-a-dimension.** The namespace tree being
|
||||
one dimension among many is exactly "do not hard-code one canonical structure."
|
||||
- **Provenance ↔ genealogy dimension.** Created-from lineage as a navigable rank makes
|
||||
provenance a first-class traversal, not metadata buried in a footer.
|
||||
|
||||
### 7.2 Deliberate divergences (design bugs if conflated)
|
||||
|
||||
1. **zzstructure as store.** Replacing Git/shards with a zz-database would break the
|
||||
Stability-Note boundary on Git's role and on orchestrator-vs-engine. zzstructure is
|
||||
a projection, full stop.
|
||||
2. **Forcing the hyperlink graph into ranks.** The link/backlink graph is many-to-many
|
||||
and must stay a graph index; do not contort it into a single dimension. Use clones
|
||||
or keep it as a separate graph projection.
|
||||
3. **One global zz-space.** Like Xanadu's docuverse, a single universal zz-space would
|
||||
fight shard sovereignty. Each information space gets its *own* dimensional
|
||||
projection over *its* attached shards.
|
||||
|
||||
### 7.3 What ZigZag teaches that shard-wiki should keep
|
||||
|
||||
- A page is best understood as **a cell at the intersection of many independent
|
||||
relationships**, not as a node in one tree. Build the page model and the navigation
|
||||
API so *adding a new relationship is adding a dimension*, not special-casing a view.
|
||||
- **Pivoting two relationships** (H-view) is a powerful, under-used exploration
|
||||
primitive for federation (namespace × shard, page × version, topic × equivalent).
|
||||
|
||||
---
|
||||
|
||||
## 8. Use-case seeds → catalog (promoted 2026-06-14)
|
||||
|
||||
Last existing UC is **UC-46**. New UCs **UC-47–UC-49** added; existing UCs enriched.
|
||||
|
||||
| Seed | Catalog action |
|
||||
|------|----------------|
|
||||
| **Navigate along a chosen relationship dimension** — pick namespace / genealogy / links / version / shard as the active traversal axis | **UC-47 (new)** |
|
||||
| **Pivot two dimensions as a cross-tab (H-view)** — e.g. namespace × shard, page × version-across-shards | **UC-48 (new)** |
|
||||
| **First-class created-from / fork genealogy dimension** — navigate derivation lineage as a rank | **UC-49 (new)** |
|
||||
| Namespace hierarchy is *one dimension among many*, not the privileged structure | **enriches UC-22** |
|
||||
| Derived views (RecentChanges, AllPages, SiteMap, BackLinks) reframed as dimensions + rasters | **enriches UC-05** (anchor for the UC-17–UC-20 view reframing) |
|
||||
| Fork/remix produce a genealogy edge consumed by UC-49 | **enriches UC-26, UC-29** |
|
||||
| Clone = "same content in many places" = transclusion convergence | links UC-32, UC-44/45 |
|
||||
|
||||
---
|
||||
|
||||
## 9. Architecture notes for SHARD-WP-0002 (no UC)
|
||||
|
||||
- A **dimensional projection layer** over the union is a candidate internal model:
|
||||
pages = cells; relationships (namespace, genealogy, version, shard, equivalence,
|
||||
overlay, recent, alphabetical) = dimensions; existing derived views = rasters over
|
||||
them. It is **derived/lazy**, never canonical store.
|
||||
- The **link/backlink graph stays a separate many-to-many index**; do not model it as
|
||||
a rank. Reconcile with the dimensional model via clones or by keeping graph and
|
||||
dimensional projections side by side.
|
||||
- **Genealogy** (created-from) should be recorded as an edge when fork/remix/import
|
||||
happens (UC-26, UC-29) so the `d.genealogy` rank can be reconstructed — ties to the
|
||||
coordination journal.
|
||||
- Whether the dimensional model is exposed in a **public navigation API** or stays an
|
||||
internal organizing concept is open (see §10).
|
||||
|
||||
---
|
||||
|
||||
## 10. Open questions (for spec / workplans)
|
||||
|
||||
1. Is the dimensional/zzstructure model a **public navigation API** and UI paradigm,
|
||||
or only an *internal* organizing concept for the existing derived views?
|
||||
2. How is the **many-to-many link graph** reconciled with the rank-based dimensional
|
||||
model — clones, a parallel graph index, or both?
|
||||
3. Which dimensions are **core** (namespace, version, shard, genealogy, recent) vs.
|
||||
**adapter-provided** vs. **computed on demand** (equivalence, backlinks)?
|
||||
4. Is **H-view pivoting** worth a reference-UI investment, or is it research-only?
|
||||
5. Does the **clone = transclusion** convergence justify a single internal primitive
|
||||
shared by UC-44/45 (transclusion) and UC-47/48 (dimensional navigation)?
|
||||
|
||||
---
|
||||
|
||||
## 11. Sources
|
||||
|
||||
| Source | Used for |
|
||||
|--------|----------|
|
||||
| Wikipedia — ZigZag (software) (https://en.wikipedia.org/wiki/ZigZag_(software)) | History, spreadsheet-deconstruction framing, one-connection-per-dimension rule, pivoting |
|
||||
| Wikipedia — Zzstructure (https://en.wikipedia.org/wiki/Zzstructure) | Directed-coloured-multigraph definition, dimensions-are-cells, raster vs. view |
|
||||
| McGuffin & schraefel, "A Comparison of Hyperstructures: Zzstructures, mSpaces, and Polyarchies," ACM Hypertext 2004 (https://www.dgp.toronto.edu/papers/mmcguffin_HT2004.pdf) | Graph-theoretic formal definition; taxonomy vs. mSpaces/polyarchies |
|
||||
| McGuffin, "A Graph-Theoretic Introduction to Ted Nelson's Zzstructures" (https://www.dgp.toronto.edu/~mjmcguff/research/zigzag/) | Ranks, I-view/H-view/augmented views, clones, tree-as-left-child/right-sibling, "multiple arrangements coexist via dimensions" |
|
||||
| Moore & Brailsford, "Unified Hyperstructures for Bioinformatics," JoDI (https://jodi-ojs-tdl.tdl.org/jodi/article/download/jodi-138/127) | Applied zzstructure as a unifying model over heterogeneous structures |
|
||||
| Nelson, "A Cosmology for a Different Computer Universe" (zzstructure context) | Nelson's own framing of cells/dimensions/visualization |
|
||||
| Gzz — "A Gentle Introduction to Ted Nelson's ZigZag Structure" (https://www.nongnu.org/gzz/gi/gi.html) | Practical cell/dimension/rank examples (rate-limited; corroborated by above) |
|
||||
|
||||
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (clone↔transclusion
|
||||
convergence), `spec/UseCaseCatalog.md` (UC-05, UC-17–UC-22, UC-26, UC-27, UC-29,
|
||||
UC-32, UC-44/45/46), `workplans/SHARD-WP-0002-federation-architecture.md`.
|
||||
|
||||
---
|
||||
|
||||
## 12. Traceability
|
||||
|
||||
- New UCs: **UC-47, UC-48, UC-49** → `spec/UseCaseCatalog.md`.
|
||||
- Enriched UCs: **UC-05** (anchors the UC-17–UC-20 derived-view reframing), **UC-22,
|
||||
UC-26, UC-29**.
|
||||
- Architecture (no UC): dimensional projection layer; link-graph-stays-graph;
|
||||
genealogy edges in the journal; clone↔transclusion primitive → `SHARD-WP-0002`.
|
||||
- Decision boundary recorded: zzstructure is a **derived lens, not the store**;
|
||||
Git/shards remain canonical (INTENT Stability Note).
|
||||
</content>
|
||||
@@ -13,4 +13,35 @@ when multiple files or sources are involved. Findings here inform `spec/` and
|
||||
| 2026-06-08 | `260608-yawex-prior-art/` | yawex 0.7.4 Perl wiki prior art; federation design seeds |
|
||||
| 2026-06-08 | `260608-c2-wiki-origins/` | Ward Cunningham & WikiWikiWeb origins; terms and use cases |
|
||||
| 2026-06-08 | `260608-federation-concepts/` | Federated Wiki, git/ActivityPub/Xanadu federation models |
|
||||
| 2026-06-08 | `260608-wikiengines-overview/` | Wiki engine landscape survey (Perplexity-assisted) |
|
||||
| 2026-06-08 | `260608-wikiengines-overview/` | Wiki engine landscape survey (Perplexity-assisted) |
|
||||
| 2026-06-13 | `260613-xwiki-deep-dive/` | XWiki impl, extension interfaces, ecosystem; UC-38/39 |
|
||||
| 2026-06-13 | `260613-twiki-deep-dive/` | TWiki impl, plugin API, ecosystem; UC-40/41 |
|
||||
| 2026-06-13 | `260613-foswiki-deep-dive/` | Foswiki store abstraction, extension API; UC-42/43 |
|
||||
| 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 |
|
||||
| 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 |
|
||||
| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 |
|
||||
| 2026-06-14 | `260614-obsidian-deep-dive/` | Obsidian — file-over-app vaults, plugin API, ecosystem-popularity → UC signal; UC-53/54/55/56 |
|
||||
| 2026-06-14 | `260614-notion-deep-dive/` | Notion — closed block-DB SaaS, external REST API only, database-as-pages; UC-57/58/59 |
|
||||
| 2026-06-14 | `260614-shard-spectrum-synthesis/` | Synthesis (v3) — shard family matrix + **fourteen** capability spectra across ~23 systems + **federation-model taxonomy** (fork+journal / VCS-replication+ping / query-join / engine-mirror); feeds SHARD-WP-0002 T1–T6 + T11–T16 |
|
||||
| 2026-06-14 | `260614-joplin-deep-dive/` | Joplin — SQLite-local/Markdown-on-sync, interchange-format attach, E2EE content opacity; UC-60/61 |
|
||||
| 2026-06-14 | `260614-logseq-deep-dive/` | Logseq — block-graph on plain Markdown files, in-file block IDs, derived Datalog index; UC-62/63 |
|
||||
| 2026-06-14 | `260614-localfirst-workspaces-deep-dive/` | Anytype · AFFiNE · AppFlowy — CRDT local-first workspaces (any-sync/Yjs/Yrs), native merge, P2P/E2EE; UC-64/65 |
|
||||
| 2026-06-14 | `260614-trilium-deep-dive/` | Trilium/TriliumNext — note cloning (DAG hierarchy), attribute inheritance/templates, HTML-native, scripting+ETAPI; UC-66/67 |
|
||||
| 2026-06-14 | `260614-wikijs-deep-dive/` | Wiki.js — storage-module engine (DB↔Git Markdown), GraphQL API, pluggable modules ≈ adapter-contract prior art; UC-68/69 |
|
||||
| 2026-06-14 | `260614-federated-wiki-deep-dive/` | Federated Wiki — fork-with-provenance, per-page semantic-action journal (story=replay), neighborhood/roster + chorus; prior art for our coordination journal / overlay / union pillars; UC-70/71/72 |
|
||||
| 2026-06-14 | `260614-wikibase-deep-dive/` | Wikibase/Wikidata — typed entity-statement knowledge graph (claim+qualifiers+refs+rank), SPARQL/RDF + federated SERVICE, opaque stable IDs, statement-level provenance; structure & query far-end; UC-73/74/75 |
|
||||
| 2026-06-14 | `260614-forge-wikis-deep-dive/` | Gitea · GitLab · GitHub wikis — a wiki is a separate `.wiki.git` of Markdown; git-clone universal, wiki API capability-varying (GitHub git-only); git IS the store (resolves UC-68 race); the home case; UC-76/77 |
|
||||
| 2026-06-14 | `260614-tiddlywiki-deep-dive/` | TiddlyWiki — entire wiki (content + app) in one self-contained HTML file = whole-file write-granularity extreme; Node `.tid` file-per-tiddler substrate (git-diffable); tiddler/field records, filter-expression query; UC-78 |
|
||||
| 2026-06-14 | `260614-ikiwiki-deep-dive/` | ikiwiki — wiki compiler: git-canonical Markdown source → static HTML (derived publish/projection); git-distributed clone federation + XML-RPC pinger (third federation flavor); UC-79 |
|
||||
| 2026-06-14 | `260614-quip-deep-dive/` | Salesforce Quip — closed-SaaS live docs + embedded spreadsheets/live apps; REST + lossy HTML import/export; Salesforce enterprise ACL; external-API payload-format facet + inline-object page model; UC-80 |
|
||||
| 2026-06-14 | `260614-mojomojo-deep-dive/` | MojoMojo — Perl Catalyst/DBIx::Class DB-backed wiki; pages + history in SQL tables, no file store/API → direct-DB-read binding; DB version rows as history source; UC-81 |
|
||||
| 2026-06-14 | `260614-oddmuse-deep-dive/` | Oddmuse — single Perl CGI, plain-text `page/` files + `keep/` revisions, no DB; the minimal file-store floor / graceful-degradation baseline; partial-history honesty; UC-82 |
|
||||
| 2026-06-14 | `260614-usemodwiki-deep-dive/` | UseModWiki — flat-file ancestor (Wikipedia's MediaWiki Phase I); CamelCase linking; lineage grounding for the minimal file-store floor; enrichment-only (reinforces UC-82, UC-25) |
|
||||
| 2026-06-14 | `260614-literate-programming-deep-dive/` | Literate programming (Knuth's WEB / weave / tangle) — one source → N co-equal derived projections (docs + code); named-chunk transclusion; splits replication- vs derivation-projection; SHARD-WP-0004 T1; UC-83 |
|
||||
| 2026-06-14 | `260614-jupyter-deep-dive/` | Jupyter Notebooks — `.ipynb` JSON cells + embedded computed outputs with fragile execution provenance; derived output stored *inside* the source; non-Markdown/lossy; kernel = capability; SHARD-WP-0004 T3; UC-84 |
|
||||
| 2026-06-14 | `260614-glamorous-toolkit-deep-dive/` | Glamorous Toolkit (moldable development on Pharo) — `gtView` open set of co-equal type-keyed computed views (none canonical) = moldable view registry; Lepiter live notebook over git files; SHARD-WP-0004 T7; enrichment-only (UC-47/48/54) |
|
||||
| 2026-06-14 | `260614-mathematica-deep-dive/` | Mathematica Notebooks — the original computational notebook (`.nb` = a Wolfram expression); nestable cell groups, structured re-evaluable outputs, `Manipulate` live widgets, CDF; confirms UC-84 notebook shape is a genus; SHARD-WP-0004 T2; enrichment-only (reinforces UC-84; UC-54/55) |
|
||||
| 2026-06-14 | `260614-squeak-pharo-deep-dive/` | Squeak & Pharo (image-based Smalltalk) — the live-object image (purest "live" end); image-is-not-a-store boundary (export→files only); Pharo Tonel/Iceberg externalizes code to git text; names the live↔snapshot projection axis; SHARD-WP-0004 **T6 + T8** (merged); boundary/enrichment-only, no new UC |
|
||||
| 2026-06-14 | `260614-processing-deep-dive/` | Processing / p5.js — program-as-page rendered live at view time (no cached output); adds materialization-timing + continuity facets to projection; execute-in-viewer = capability+trust; SHARD-WP-0004 T4; enrichment-only (UC-54/55) |
|
||||
| 2026-06-14 | `260614-strudel-deep-dive/` | Strudel.cc (TidalCycles JS) live-coding REPL — code as live time-based audio performance; the far live end (no faithful static form; static = source + marked recording); honesty test for graceful degradation; SHARD-WP-0004 T5; enrichment-only (UC-54/55) |
|
||||
| 2026-06-15 | `260614-computational-page-model-synthesis/` | **SHARD-WP-0004 synthesis** — *the computational page model*: source canonical / everything rendered is a projection; two axes (replication↔derivation, live↔snapshot); four computational page shapes; recommendation — executable content in scope as page-model+projection, out of scope as an execution platform (no INTENT amendment); feeds SHARD-WP-0002 T11–T16 |
|
||||
976
spec/CoreArchitectureBlueprint.md
Normal file
976
spec/CoreArchitectureBlueprint.md
Normal file
@@ -0,0 +1,976 @@
|
||||
# CoreArchitectureBlueprint — shard-wiki
|
||||
|
||||
Status: **draft for review** · Date: 2026-06-15 · Owner: tegwick
|
||||
|
||||
The whole-system architecture for shard-wiki, synthesised from `INTENT.md`, the 84-entry
|
||||
`UseCaseCatalog.md`, and the full research arc (`research/260608-*`, `research/260613-*`,
|
||||
`research/260614-*` — ~23 wiki/knowledge systems plus two cross-dive syntheses). This is the
|
||||
**core** blueprint: it defines the layers, the abstractions, and the load-bearing decisions
|
||||
that everything else implements.
|
||||
|
||||
Scope relationship to the other specs:
|
||||
|
||||
- **`ArchitectureBlueprint.md`** (existing) is the **authorization & history sub-blueprint**
|
||||
(the L0–L4 ladder). This document references it as the design of the cross-cutting
|
||||
authorization layer (§9) and does not restate it.
|
||||
- **`SHARD-WP-0002`** is the workplan that turns §6–§8 into
|
||||
`spec/FederationArchitecture.md` + the adapter-contract section of
|
||||
`spec/TechnicalSpecificationDocument.md`.
|
||||
- **`UseCaseCatalog.md`** is the demand this architecture must satisfy; UC references below
|
||||
are load tests, not decoration.
|
||||
- **`WikiEngineCoreArchitecture.md`** designs shard-wiki's native, headless, API-first wiki
|
||||
engine as a **canonical-mode shard backend** (one shard behind §6/§A — federation, union, and
|
||||
projection stay here in the orchestrator, not in the engine). Added per the 2026-06-15 INTENT
|
||||
amendment (decision `84ffdb48`, SHARD-WP-0013).
|
||||
|
||||
---
|
||||
|
||||
## 1. The thesis: *canonical vs derived* (three states)
|
||||
|
||||
Everything in shard-wiki follows from one organising decision — that state comes in exactly
|
||||
**three kinds**, and only one of them is disposable:
|
||||
|
||||
> **1. Sharded-canonical** — content owned by each shard (shard sovereignty).
|
||||
> **2. Coordination-canonical** — durable state *born inside shard-wiki* that encodes human
|
||||
> or cross-shard decisions and exists nowhere else: overlays (the local truth against a
|
||||
> read-only shard), curator equivalence bindings, alias tables, merge/reconciliation
|
||||
> decisions. It is recorded as an **append-only decision log in the Git coordination
|
||||
> journal** (event-sourced, §8.1); the *queryable current form* of that state (the effective
|
||||
> alias table, the equivalence set) is a **derived fold** of the log — i.e. tier 3, not tier 2.
|
||||
> What is canonical is the **log of decisions**, not any mutable snapshot of them.
|
||||
> **3. Derived-disposable** — everything shard-wiki *computes* from (1)+(2): the union graph,
|
||||
> equivalence index, query indexes, projections, views. It can be deleted and recomputed.
|
||||
>
|
||||
> **Canonical = sharded ∪ coordination. Derived = a pure function of canonical:**
|
||||
> `derived = f(sharded, coordination)`.
|
||||
|
||||
This is the architectural form of "orchestrator, not engine." shard-wiki never *becomes* the
|
||||
source of truth; it composes sources and records the decisions it makes about them. The
|
||||
research earned the *files-canonical* half empirically — every serious system externalises its
|
||||
durable truth to files+VCS and treats the rest as derived: Logseq (DataScript index over plain
|
||||
files), ikiwiki (static HTML compiled from a git repo), Glamorous Toolkit / Lepiter (live
|
||||
views over git-versioned JSON), Pharo (Tonel/Iceberg code as git text), Jupyter teams
|
||||
(nbstripout — outputs are derived noise). The one tradition that refuses this — the Smalltalk
|
||||
**image** — is exactly the one we record as a *boundary, not a backend*
|
||||
(`research/260614-squeak-pharo-deep-dive`).
|
||||
|
||||
The earlier draft of this blueprint used a two-bucket framing ("canonical at the edges,
|
||||
derived in the middle"). That was wrong by omission: it had no home for **coordination-
|
||||
canonical** state, and so contradicted itself by listing curator bindings and alias tables as
|
||||
"derived/rebuildable" when a human binding manifestly cannot be rebuilt. The three-state model
|
||||
fixes that crack (review finding A-1) and makes `derived = f(canonical)` *literally* true.
|
||||
|
||||
Three consequences fall straight out, and they are the spine of the rest of this document:
|
||||
|
||||
1. **Graceful degradation is free.** If the derived tier is always recomputable, a backend
|
||||
that can only be read is still a first-class participant — you just derive less from it.
|
||||
2. **Provenance is tractable.** Because shard-wiki never claims to *be* the source, every
|
||||
derived artifact can always point back to the canonical input it came from (union without
|
||||
erasure is a structural property, not a feature bolted on).
|
||||
3. **The derived tier is a pure function of canonical state.** `derived = f(sharded,
|
||||
coordination)`. Bugs in the derived tier are recoverable by recompute; only the two
|
||||
canonical tiers must be durably protected — sharded by each shard, coordination by the
|
||||
Git journal (history). *Recomputability is a correctness property of the derived tier, not
|
||||
a promise that a from-scratch rebuild is operationally cheap — see §8.4 and the
|
||||
operational-envelope axis.*
|
||||
|
||||
### The dual narrow waist
|
||||
|
||||
Heterogeneity is mediated at exactly two interfaces, and nowhere else:
|
||||
|
||||
- **Bottom waist — the Shard Adapter Contract (§6).** Every backend, however weird, enters
|
||||
through one versioned, capability-described interface.
|
||||
- **Top waist — the Wiki Page Model (§7).** Every consumer, however demanding, sees one
|
||||
backend-neutral, Markdown-first-but-stretchable page model.
|
||||
|
||||
Between the waists, core logic is written **once** against capabilities and the page model —
|
||||
never against a specific backend. Adding TiddlyWiki or Notion or a git forge is writing an
|
||||
adapter and declaring a capability profile, not editing core algorithms.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architectural invariants
|
||||
|
||||
These are non-negotiable. Violating one is a design bug, not a tradeoff. They are INTENT's
|
||||
principles fused with the research through-lines.
|
||||
|
||||
| # | Invariant | Source |
|
||||
|---|-----------|--------|
|
||||
| I-1 | **Orchestrator, not engine.** Core composes shards; it never replaces or homogenises them. | INTENT Stability Note |
|
||||
| I-2 | **Three states; derived = f(canonical).** State is sharded-canonical, coordination-canonical (journal), or derived-disposable. The derived tier (union/index/projection) is a pure, recomputable function of the two canonical tiers; only canonical state is durably protected. | §1; Logseq/ikiwiki/GT through-line |
|
||||
| I-3 | **Capability-awareness is data.** A binding's abilities are a *profile* (positions on spectra), read by generic core logic — not per-backend branches. | synthesis v3 §2; INTENT capability-aware adapters |
|
||||
| I-4 | **Union without erasure.** Every page/revision/projection/overlay/view carries its provenance, freshness, liveness, and divergence. | INTENT; provenance-granularity spectrum (Wikibase) |
|
||||
| I-5 | **Overlay before mutation.** Writes to anything below write-through land as drafts/patches/MRs first; no silent remote mutation. | INTENT |
|
||||
| I-6 | **Git-addressable coordination.** Every information space has a Git-backed journal even when its shards are not git-native. | INTENT |
|
||||
| I-7 | **Mechanism over policy.** Canonical-source, conflict, editorial, sync cadence are configurable presets, never hard-coded. | INTENT |
|
||||
| I-8 | **Graceful degradation.** A limited backend is still usable as read-only / cache / projection / backup / patch target. | INTENT |
|
||||
| I-9 | **Identity ≠ placement.** A page is an entity that may occupy N locations; address by identity, not by path. | Trilium note/branch; ZigZag |
|
||||
| I-10 | **History is the floor.** Every write is a recoverable commit; recoverability, not gatekeeping, is the baseline protection. | ArchitectureBlueprint §2 |
|
||||
| I-11 | **Authorization in core, authentication delegated.** Core decides who-may; an external provider says who-is. | INTENT; ArchitectureBlueprint |
|
||||
| I-12 | **Not a file-sync daemon; not an execution platform.** Sync is wiki-page-semantic; computation is recognised+projected, not hosted. | INTENT; computational-page-model synthesis |
|
||||
| I-13 | **Tenant-partitioned derived state.** Derived state is partitioned by tenant/root entity; no derived artifact spans tenants except via explicit, authorised cross-root federation. | §9.1; review B-3 |
|
||||
|
||||
---
|
||||
|
||||
## 3. The layered architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ L6 Consumers — Orchestrator API · CLI/agents · Web/Obsidian │
|
||||
├───────────────────────────────────────────────────────────────┤
|
||||
X-cut │ L5 Authorization (PEP/PDP, identity-provider iface) → │ X-cut
|
||||
Prove- │ see ArchitectureBlueprint.md (L0–L4 ladder) │ Capa-
|
||||
nance ├───────────────────────────────────────────────────────────────┤ bility
|
||||
▲ │ L4 Union & Projection (DERIVED · rebuild=fallback) │ ▲
|
||||
│ │ identity resolution · equivalence/chorus · union graph · │ │
|
||||
│ │ replication+derivation projections · moldable view registry│ │
|
||||
│ │ · derived query index │ │
|
||||
│ ├───────────────────────────────────────────────────────────────┤ │
|
||||
│ │ L3 Coordination (Git journal · overlay/patch engine · │ │
|
||||
│ │ federation-model strategies · reconciliation) │ │
|
||||
│ ├───────────────────────────────────────────────────────────────┤ │
|
||||
│ │ L2 Wiki Page Model ── TOP WAIST ── │ │
|
||||
│ │ backend-neutral pages · identity≠placement · span address ·│ │
|
||||
│ │ provenance envelope · the page shapes │ │
|
||||
│ ├───────────────────────────────────────────────────────────────┤ │
|
||||
│ │ L1 Shard Adapter Contract ── BOTTOM WAIST ── │ │
|
||||
│ │ versioned iface · capability profile (orthogonal) · │ │
|
||||
│ │ attachment-mode binding · operation verbs │ │
|
||||
└──── ├───────────────────────────────────────────────────────────────┤ ──┘
|
||||
│ L0 Backends (not ours): git repos, wiki/ subdirs, Gitea/ │
|
||||
│ GitLab/GitHub wikis, folders, Obsidian, WebDAV, Notion, │
|
||||
│ Coulomb spaces, notebooks, … │
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Provenance** and **Capability** are drawn as vertical rails because they are not layers —
|
||||
they are present at every layer. A page object at L2 carries provenance; a projection at L4
|
||||
carries provenance; an authz decision at L5 records the context under which content was read.
|
||||
Likewise a capability profile declared at L1 is consulted at L3 (can we write-through?), L4
|
||||
(can we delegate a query?), and L5 (can this principal even reach the op?).
|
||||
|
||||
The dependency rule is strict and downward, and it tracks the **three states (§1)**, not the
|
||||
layer numbers: **the derived-disposable tier (the whole of L4) may be deleted and recomputed
|
||||
from canonical state (sharded content at L1 + coordination-canonical state in the L3 journal).**
|
||||
Nothing canonical may depend on derived state. Note the journal at L3 is *canonical* (it holds
|
||||
overlays, bindings, aliases, merges); only L4 is disposable.
|
||||
|
||||
---
|
||||
|
||||
## 4. Core abstractions (the vocabulary code must use)
|
||||
|
||||
Straight from INTENT, sharpened by research. New code maps onto these; it does not invent
|
||||
parallel terms.
|
||||
|
||||
- **Shard** — an independently meaningful page store attached to a root entity, with
|
||||
*sovereignty*: its own backend, capability profile, history, identity model, limits.
|
||||
- **Root entity / information space** — the joined space shards attach to; the unit of
|
||||
Git coordination and of multi-tenancy (a tenant maps to a root entity, ArchitectureBlueprint).
|
||||
- **Shard adapter contract** — the versioned L1 interface; the bottom waist.
|
||||
- **Capability profile** — a shard binding's position on each of the 15 spectra (§6) plus its
|
||||
supported verbs. *The* data structure that drives degradation.
|
||||
- **Wiki page model** — the L2 backend-neutral page; the top waist.
|
||||
- **Page identity vs placement vs equivalence** — a page is an entity with a *stable handle*
|
||||
(identity); it may have N placements (paths/shards); **addressing and transclusion key on
|
||||
identity, but equivalence keys on content fingerprint *across* identities** (§7.2, I-9). The
|
||||
three are distinct mechanisms — never conflate identity with a fingerprint.
|
||||
- **Provenance envelope** — the metadata each artifact carries (source shard, freshness,
|
||||
liveness, authz context, overlay status, divergence, lineage), stored **layered**: a
|
||||
page-level envelope + span-level *deltas*, so per-span cost is near-zero when uniform (§7.3).
|
||||
- **Coordination journal** — the L3 Git-backed, **append-only decision log** for a space: the
|
||||
durable home of all **coordination-canonical** state (§1, §8.1) as *events* (overlay-created,
|
||||
binding-made, alias-set, merge-decided), plus the content change-flow record. It is event-
|
||||
sourced — committed, never overwritten; the queryable current coordination state is a derived
|
||||
fold of it (§8.1).
|
||||
- **Overlay** — a non-destructive local edit against a remote/read-only/limited shard,
|
||||
representable as draft/patch/commit/MR before destructive apply. Coordination-canonical: an
|
||||
unapplied overlay is the local truth and lives in the journal.
|
||||
- **Projection** — a derived view of shard content. The default is a **plain lazy
|
||||
replication-projection** (a freshness-stamped cache); only *source* content needing
|
||||
transform/evaluate uses the **derivation-projection** extension point with its two-axis
|
||||
typing (kind × liveness) and the moldable view registry (§8.4–§8.5).
|
||||
- **Federation model** — the selected coordination strategy for a space (§ taxonomy, T17).
|
||||
- **Shard mode** — read-only · write-through · mirrored · projected · cached · canonical
|
||||
(a *policy* selection constrained by the capability profile).
|
||||
|
||||
---
|
||||
|
||||
## 5. Why "layered" and not "pipeline" or "plugin-bus"
|
||||
|
||||
Two rejected alternatives, recorded so the choice is legible:
|
||||
|
||||
- **A sync pipeline** (source → transform → sink) was rejected: it implies a privileged
|
||||
direction and a canonical sink, which violates shard sovereignty (I-1) and union-without-
|
||||
erasure (I-4). shard-wiki is a *star* (many shards ↔ one space), not a pipe.
|
||||
- **A flat plugin bus** (every backend a peer plugin emitting events) was rejected as the
|
||||
*top-level* shape: it has no narrow waist, so heterogeneity leaks into every consumer.
|
||||
We keep the plugin idea but confine it to L1 (adapters) and L3 (federation strategies),
|
||||
behind the waists.
|
||||
|
||||
The layered-with-rails shape is what makes I-2/I-3/I-4 hold simultaneously.
|
||||
|
||||
---
|
||||
|
||||
## 6. Bottom waist — the Shard Adapter Contract (L1)
|
||||
|
||||
The single most important design decision in the project: **the adapter contract models
|
||||
positions on capability spectra, not a flat checklist of boolean verbs.** A backend is not
|
||||
"can/can't merge"; it sits *somewhere* on the merge spectrum, and federation operations
|
||||
degrade by position. This is the lesson of putting ~23 systems in one matrix
|
||||
(`research/260614-shard-spectrum-synthesis`, v3).
|
||||
|
||||
### 6.1 The fifteen capability spectra
|
||||
|
||||
Each binding declares a position on each axis. Core algorithms read these positions; there is
|
||||
no per-backend code in core (I-3).
|
||||
|
||||
1. **Addressing granularity** — none → path → page-level store-id → in-file span → in-file
|
||||
block id (Logseq `id::`) → store-UUID → portable tumbler (Xanadu, the unreached ideal)
|
||||
2. **Content identity** — none → path/title → fingerprint → span-set
|
||||
3. **Identity vs placement** — path=identity → identity separated from placement (Trilium
|
||||
note/branch = a DAG)
|
||||
4. **Structure** — flat MD → frontmatter/`key::` → `%META%` → typed objects → DB schema+
|
||||
relations → object-graph/ontology → computed (inherited+templated) → typed-graph statements
|
||||
5. **History** — none → internal-only / CRDT-log → open-file → git-native
|
||||
6. **Merge model** — none → git/text → conflict-notes/keep-both → native-CRDT → coexist-with-rank
|
||||
7. **Native query** — none → text → build-your-own derived index → datalog/graph → DB query → SPARQL
|
||||
8. **Translation** — native → lossless → lossy-with-fidelity-report (incl. HTML)
|
||||
9. **Attachment mode** — file-store (native | interchange-mirror) → git-IS-store → in-engine-host
|
||||
→ local-REST → external-API → direct-DB → CRDT-replica → P2P/no-central-endpoint
|
||||
10. **Operational envelope** — local/unbounded → realtime CRDT/WebSocket → rate-limited/
|
||||
eventually-consistent/paginated
|
||||
11. **Access grant** — open → token → OAuth scoped+revocable → P2P key/invite → enterprise ACL
|
||||
12. **Content opacity** — plaintext → structured re-evaluable value → encrypted whole-shard →
|
||||
per-item → proprietary-lossy-exportable
|
||||
13. **Write granularity** — whole-file (TiddlyWiki) → per-page → section/anchor → per-block → story-item
|
||||
14. **Provenance granularity** — per-shard → per-page → per-edit → per-statement/value (Wikibase rank+refs)
|
||||
15. **Computational / liveness** — static source → captured-output snapshot → live-over-files →
|
||||
view-time render → irreducibly-live/temporal
|
||||
|
||||
### 6.2 Operation verbs
|
||||
|
||||
`read, write, diff, merge, lock, version, publish, notify, transclude-source,
|
||||
translate-syntax, structured-payload, derive-projection, execute/evaluate`. The last two are
|
||||
**gated, off by default** (§8, computational content). Verb support is part of the profile and
|
||||
must reconcile with the federation-ops capability matrix (SHARD-WP-0002 T10).
|
||||
|
||||
### 6.3 Attachment-mode taxonomy (axis 9, expanded)
|
||||
|
||||
A backend may offer **several** modes; attach mode is a **per-binding, capability-gated
|
||||
choice**, with one declared authoritative. Modes: file-store (native vault/folder *or* an
|
||||
interchange/sync mirror), **git-IS-store** (the home case — forge wikis & ikiwiki: git is the
|
||||
store *and* the journal at once, resolving the engine-mirror write-race), in-engine hosted
|
||||
adapter (XWiki component, Obsidian/Logseq/Roam plugin, Trilium script), local-REST (Joplin
|
||||
Data API, Trilium ETAPI), external-API-only (Notion), direct-DB (MojoMojo schema→model),
|
||||
CRDT-replica (Anytype/AFFiNE/AppFlowy), P2P/no-central-endpoint. **Boundary:** a monolithic
|
||||
live-memory blob (Smalltalk image, a kernel) is **never** an attach target — it participates
|
||||
only via export→files (I-12).
|
||||
|
||||
### 6.4 Contract rules
|
||||
|
||||
- **Versioned interface** (Foswiki::Store + Foswiki::Meta is the proof that a stable
|
||||
store-interface-with-swappable-backends works). Capability discovery is a static profile
|
||||
with optional runtime negotiation.
|
||||
- **Backend-swap tolerance** — shard identity/provenance survives a substrate change
|
||||
(RCS↔PlainFile, folder→Git, Logseq file→SQLite): bind to *capabilities*, not to "it's files."
|
||||
- **Absence is first-class** — the profile must express *can't* cleanly (Oddmuse floor), so
|
||||
degradation paths are explicit, never guessed.
|
||||
|
||||
### 6.5 Orthogonal core, implied positions, and the interaction subset
|
||||
|
||||
Fifteen independent ordinal axes is *descriptively* right but would be *operationally* a mess
|
||||
if treated as fifteen free dimensions: the axes are **not orthogonal**, and a degradation
|
||||
function over all 15 jointly is the flat-checklist problem returning in higher dimensions
|
||||
(review D-1). Three rules tame it.
|
||||
|
||||
**(a) A small orthogonal core; the rest are implied.** Most axes are *correlated* and collapse
|
||||
to a few independent choices. The **core axes** an adapter must independently declare:
|
||||
|
||||
1. **Substrate** → drives attachment-mode, history, merge, and native-query positions together
|
||||
(git-IS-store ⟹ history=git-native ⟹ merge=git/text ⟹ query=build-your-own-index;
|
||||
relational-DB ⟹ direct-DB attach ⟹ DB-version-row history ⟹ DB query).
|
||||
2. **Write granularity** → drives addressing granularity and the overlay/patch shape.
|
||||
3. **Content opacity** → drives translation and (where encrypted) collapses native-query.
|
||||
4. **Operational envelope** → drives freshness mode (§8.8) and rebuild expectations (§8.7).
|
||||
5. **Access grant** → independent (authz, L5).
|
||||
6. **Computational/liveness** → independent (projection kind, §8.5).
|
||||
|
||||
The remaining axes are **implied/derived** from these via published implication rules; an
|
||||
adapter *may* override an implied position, but the default is computed, not hand-set. This
|
||||
turns ~15 free dimensions into ~6 independent ones plus derivations — fewer things to get
|
||||
wrong, and impossible combinations become unrepresentable.
|
||||
|
||||
**(b) Implication rules forbid impossible profiles.** E.g. `attachment=git-IS-store ⟹
|
||||
history≥git-native`; `opacity=encrypted-whole-shard ⟹ native-query=none ∧ translation≤opaque`;
|
||||
`merge=native-CRDT ⟹ history=CRDT-log ∧ envelope=realtime`. A profile that violates an
|
||||
implication is rejected at registration — capability-as-data (I-3) with integrity constraints.
|
||||
|
||||
**(c) The degradation function reads a *named, small* interaction subset — not all pairs.**
|
||||
"No per-backend code" is only credible if we say *which* axis interactions the generic logic
|
||||
actually consults. They are:
|
||||
|
||||
| Operation | Axes consulted (jointly) |
|
||||
|-----------|--------------------------|
|
||||
| **write / overlay-apply** | write-granularity × merge-model × history × access-grant |
|
||||
| **transclude / address a span** | addressing-granularity × write-granularity × identity-vs-placement |
|
||||
| **project / cache** | operational-envelope × computational-liveness × content-opacity |
|
||||
| **query** | native-query × content-opacity (encrypted ⇒ derive-index-or-none) |
|
||||
| **translate** | translation × content-opacity × structure |
|
||||
| **federate** | substrate × history × merge (per the §8.3 model) |
|
||||
|
||||
Everything else is a single-axis check. This table *is* the degradation contract: it is small,
|
||||
enumerated, and testable — the proof obligation behind "core logic written once."
|
||||
|
||||
### 6.6 Conformance — profiles are verified, never self-asserted
|
||||
|
||||
Capability-as-data (I-3) and the entire degradation contract (§6.5) rest on one assumption:
|
||||
**the profile tells the truth.** If an adapter declares `merge=git/text` but corrupts merges,
|
||||
or claims `notify` and never emits, it silently poisons every degradation decision in core —
|
||||
the failure is invisible because core *believed the data* (review B-2). So the profile is not
|
||||
taken on trust:
|
||||
|
||||
- **The contract ships a versioned conformance suite.** A published battery that, given a live
|
||||
binding, **exercises each declared verb and each declared spectrum position and checks that
|
||||
observed behaviour matches the claim** (a `write` round-trips; a `diff` is real; `notify`
|
||||
actually fires; an "encrypted/opaque" shard genuinely refuses plaintext query; an
|
||||
implication-rule position, §6.5(b), holds). The suite is versioned *with* the contract, so an
|
||||
adapter proves conformance against a known contract version.
|
||||
- **Passing conformance is an admissibility precondition.** A binding that fails (declares a
|
||||
capability it does not honour) is **rejected at registration**, not run in production with a
|
||||
lying profile. Capability discovery (§6.4) therefore yields a *verified* profile.
|
||||
- **Self-reported, then verified.** Adapters still *declare* their profile (discovery stays
|
||||
cheap); conformance *verifies* the declaration. The two together are what make I-3 and §6.5
|
||||
sound rather than aspirational — degradation logic acts on verified data.
|
||||
- **Mismatch is data, not a crash.** A conformance gap is reported as a precise
|
||||
capability-by-capability diff (what was claimed vs observed), so an adapter author fixes the
|
||||
profile or the code; degraded-but-honest registration (drop the unsupported claim) is allowed.
|
||||
|
||||
This is the same discipline a versioned store interface needs in general (the `Foswiki::Store`
|
||||
lineage that inspired the contract): a backend may only participate behind the interface if it
|
||||
*demonstrably* behaves as the interface says.
|
||||
|
||||
---
|
||||
|
||||
## 7. Top waist — the Wiki Page Model (L2)
|
||||
|
||||
Backend-neutral, **Markdown-first but stretchable many ways at once**. The page model is the
|
||||
lingua franca every consumer sees; an adapter's job is to project its backend into this model
|
||||
(read) and accept overlays back (write), within its capabilities.
|
||||
|
||||
### 7.1 Page shapes the model must carry
|
||||
|
||||
- **Prose Markdown** — the baseline.
|
||||
- **Typed / computed records** — frontmatter/`%META%`/XObjects/Notion DB rows; **computed
|
||||
metadata** (Trilium inherited+templated) represented as *effective-vs-own with per-attribute
|
||||
provenance*.
|
||||
- **Typed-graph statements** — Wikibase claim + qualifiers + references + rank (structure
|
||||
far-end).
|
||||
- **Inline-embedded objects** — Quip/Notion spreadsheets & live apps inside prose.
|
||||
- **Non-Markdown assets** — drawings, canvases, images: typed asset / opaque blob / pluggable
|
||||
content-type registry, never silent-flattened.
|
||||
- **The four computational shapes** (§8): one-source-many-projections, notebook (embedded
|
||||
computed output), program-as-page, live/temporal.
|
||||
|
||||
All shapes reduce to a common skeleton: **`(content | source, structure, provenance envelope,
|
||||
[derivation rule])`**. The page model stores the richest faithful form as canonical and treats
|
||||
any Markdown rendering of a non-Markdown shape as a *lossy projection* (I-4 + fidelity report).
|
||||
|
||||
### 7.2 Identity, placement, addressing — three distinct concepts
|
||||
|
||||
The earlier draft used "identity" for two different things and (worse) suggested deriving page
|
||||
identity from a content fingerprint — which would make *editing a page change its identity* and
|
||||
break every reference to it (review bug B-1). They are pulled apart here:
|
||||
|
||||
- **Page identity — a *stable handle*.** A shard-scoped, durable key that **survives edits**:
|
||||
the backend's native page/note id where one exists (Roam/Notion/Trilium uid, a git path
|
||||
treated as a name, a wiki page name), wrapped in a shard scope so it survives projection and
|
||||
never collides across shards. Identity is *assigned/minted, not computed from content*.
|
||||
References, placement, transclusion targets, and overlays all key on identity.
|
||||
- **Placement — *where* an identity sits.** One identity → N placements (paths/shards) = a DAG;
|
||||
no single canonical path (I-9). Placement can change without changing identity.
|
||||
- **Content equivalence — *detecting sameness*, never identity.** A **content fingerprint** (or
|
||||
span-set overlap) identifies a *version / a piece of content*, used to detect that two
|
||||
*distinct identities* hold the same or derived content (the equivalence/chorus mechanism,
|
||||
§8.4). A fingerprint is never a page's identity: same page, edited → new fingerprint, **same
|
||||
identity**; two pages, identical content → same fingerprint, **different identities**.
|
||||
- **Span addressing** — a sub-page address within an identity: adopt native span IDs where
|
||||
minted (Roam `:block/uid`, Logseq `id::`, Notion/CRDT UUID); else a *position* address
|
||||
(path+range) or a *content-fingerprint* address for equivalence/transclusion. The Xanadu
|
||||
tumbler is the portable ideal the scheme aims at without requiring.
|
||||
- **Provenance envelope** rides on pages and spans (see §7.3 for its layered, low-cost form).
|
||||
|
||||
So the chain is: **identity (stable) → placements (N, mutable) → equivalence (cross-identity
|
||||
sameness, fingerprint-based)** — three concepts, three mechanisms, never conflated.
|
||||
|
||||
### 7.3 Provenance is layered, not per-span-duplicated
|
||||
|
||||
A provenance envelope on *every span* (source shard, freshness, liveness, overlay status,
|
||||
authz context, divergence, lineage) would, at block granularity, mean ~10k near-identical
|
||||
envelopes for a 10k-block page — provenance dwarfing content (review D-2). The fix is the exact
|
||||
pattern the page model already uses for Trilium's computed metadata: **effective-vs-own**.
|
||||
|
||||
- **Page-level envelope** holds the values that are uniform across the page (almost always:
|
||||
source shard, observed-at, liveness, authz context).
|
||||
- **Span-level deltas** record *only where a span differs* from its page envelope — a
|
||||
transcluded span from another shard, an overlaid span, a span that diverges. A span with no
|
||||
delta inherits the page envelope at zero storage cost.
|
||||
- **Effective provenance** for any span = page envelope ⊕ span delta, computed on read.
|
||||
|
||||
Per-span cost is therefore **near-zero in the common (uniform) case** and pays only for genuine
|
||||
heterogeneity — the same "carry only the difference" principle, applied to shard-wiki's own
|
||||
metadata. Provenance remains complete (I-4); it is just not redundantly materialised.
|
||||
|
||||
---
|
||||
|
||||
## 8. Coordination, federation & projection
|
||||
|
||||
### 8.1 Coordination journal (L3) — Git as the spine
|
||||
|
||||
Every information space has a Git-backed coordination journal (I-6). It records cross-shard
|
||||
operations (fork, import, reconcile, overlay-apply, space-branch) and **is** the history floor
|
||||
(I-10). For git-IS-store shards the shard's own git log *is* this journal; for non-git shards
|
||||
the journal supplements (begins-now / mirrors-forward / snapshots-replica) or imports
|
||||
(backfill open file history). History portability is a spectrum, handled per profile (axis 5).
|
||||
|
||||
**The journal is an append-only decision log; current coordination state is a derived fold
|
||||
(review B-3).** The first draft said coordination-canonical state "lives in the journal"
|
||||
without saying how Git — excellent for history, poor for mutable structured state — represents
|
||||
an alias table or an equivalence graph. Resolution: **event sourcing.** The journal stores
|
||||
*decisions as events* (`overlay-created`, `binding-made`, `alias-set`, `merge-decided`,
|
||||
`page-forked`), append-only and git-addressable (so history/patch/review/backup over
|
||||
coordination state come for free — I-6 is *strengthened*, not bypassed). The **queryable
|
||||
current state** (the effective alias table, the live equivalence set) is a **derived fold** of
|
||||
the log — tier-3 disposable, indexed like any other derived structure (§8.7), rebuilt by
|
||||
replaying the log. So "all equivalences touching X" is an index lookup, not an O(scan) of Git.
|
||||
This is the clean form of the §1 three-state model: **the log is canonical; its folded current
|
||||
state is derived.**
|
||||
|
||||
**Concurrency: who may append (review B-1).** A multi-tenant L4 deployment runs several
|
||||
orchestrator instances, so "the journal is local Git, single writer" is not given. The model:
|
||||
|
||||
- **One *append authority* per information space.** Appends to a space's log are serialized
|
||||
through a single logical writer (a per-space lease/leader; instances without the lease forward
|
||||
their append intents to it). This makes the log a **totally-ordered event sequence** per space
|
||||
— the ordering authority §8.6 relies on — without a distributed transaction. Spaces are
|
||||
independent, so this scales horizontally *across* spaces (the unit of partition is the space /
|
||||
root entity, matching the tenant partition, I-13); it is a per-space serialization point, not
|
||||
a global one.
|
||||
- **Git is the durable, addressable form; appends are commits** (or fast objects batched into
|
||||
commits) under the lease — no concurrent-writer merge races because there is one writer at a
|
||||
time per space.
|
||||
- **Read-your-writes** holds within a space because every reader resolves current state from
|
||||
the same ordered log (or its fold); across spaces there is no shared state to be inconsistent.
|
||||
- **HA / failover:** the lease is time-bounded and re-grantable; a failed append-authority is
|
||||
replaced and resumes from the log's head (the log is the recovery point). A partition that
|
||||
splits the authority degrades that *space* to read-only until a single writer is re-elected —
|
||||
it never forks the log (availability yields to log integrity; an explicit, stated trade).
|
||||
- **Open residual (→ §12, O-3-adjacent):** whether very high append rates need per-space log
|
||||
*sharding* (sub-logs merged by a deterministic order) is an implementation spike, not an
|
||||
architectural change.
|
||||
|
||||
**History must stay recoverable *and* bounded (review C-3).** "Every write is a commit" + open
|
||||
L0 means an unbounded, bot-/vandalism-amplified journal that eventually degrades Git itself.
|
||||
Recoverability (I-10) is non-negotiable, so the answer is *compaction, not deletion*:
|
||||
|
||||
- **Routine git maintenance** — background `gc`/repack, commit-graph, and (for very large
|
||||
spaces) partial-clone / sparse strategies; operational, no semantic change.
|
||||
- **Squash-compaction of low-value churn (policy, §10)** — long runs of rapid same-author
|
||||
edits or revert-pairs can be folded into checkpoint commits *while preserving the recoverable
|
||||
endpoints*; what is squashed is configurable and always leaves the content recoverable (it
|
||||
compacts the *path*, not the *reachable states*).
|
||||
- **Per-shard history offload** — a git-IS-store shard keeps its own history in its own repo;
|
||||
the coordination journal references it rather than duplicating it (the journal records
|
||||
*coordination* events, not a second copy of every shard commit).
|
||||
- **Anti-abuse hooks (policy)** — rate-limiting / quarantine for anonymous L0 writers feed the
|
||||
authz/policy layer; they throttle *abuse*, never legitimate history. Recoverability is the
|
||||
floor; bounding is how it survives at scale.
|
||||
|
||||
### 8.2 Overlay / patch engine (L3)
|
||||
|
||||
The default write path for anything below write-through capability (I-5): an edit becomes a
|
||||
draft → patch/commit → MR, applied destructively only on explicit intent and only where the
|
||||
profile + policy both permit. This is what lets a read-only or rate-limited or lossy backend
|
||||
still be *edited* safely.
|
||||
|
||||
### 8.3 Federation is plural & composable (L3) — the model taxonomy
|
||||
|
||||
Federation is not one mechanism. shard-wiki selects a **federation model per space and
|
||||
composes per shard** (mechanism over policy, I-7):
|
||||
|
||||
| Model | Anchor | Coordination shape |
|
||||
|-------|--------|--------------------|
|
||||
| **Fork + journal** (default home case) | Federated Wiki | copy-with-provenance + per-page action journal (story = replay) |
|
||||
| **VCS-replication + ping** | ikiwiki | git clone/pull/push + change-ping |
|
||||
| **Query-time graph-join** | Wikibase SPARQL `SERVICE` | join remote graphs at query time, no copy |
|
||||
| **Feed aggregation** | RSS/Atom | inbound feed → pages |
|
||||
| **Activity streams** | ActivityPub | Create/Update events, notify or content-bearing |
|
||||
| **Engine-mirror** | Wiki.js DB↔Git | engine syncs its own store to a git mirror |
|
||||
|
||||
### 8.4 Union & projection (L4) — the derived cache
|
||||
|
||||
This whole layer is **derived-disposable**: recomputable from canonical state — sharded
|
||||
content + the **coordination-canonical** inputs in the journal (I-2). Crucially, the *automatic*
|
||||
equivalence results are derived, but the **human/curatorial inputs they consume — alias tables
|
||||
and curator equivalence bindings — are coordination-canonical (they live in the journal), not
|
||||
derived**; recompute reads them, never regenerates them. It comprises:
|
||||
|
||||
- **Identity resolution & equivalence** — detect "same topic / derived content" path-
|
||||
independently from *derived* signals (content fingerprint, span-set overlap) **plus** the
|
||||
*coordination-canonical* inputs (alias table, curator binding); present as
|
||||
**chorus-of-voices** or designated-canonical (a *policy* preset). (Scaling: §8.7.)
|
||||
- **Union graph** — the navigable join of pages, links, and dimensions (namespace, genealogy,
|
||||
version, shard, equivalence). A *derived lens over canonical files+journal, never a new
|
||||
store* (the ZigZag boundary).
|
||||
- **Transclusion** — one **reference-not-copy** primitive unifying Xanadu transclusion, ZigZag
|
||||
clone, Roam/Obsidian/Logseq embed, Notion synced block, Trilium note-cloning, and literate
|
||||
named-chunk assembly, over the addressable union.
|
||||
- **Projection — trivial by default, extensible for the tail.** The 95% case (Markdown in a
|
||||
shard) must cost nothing conceptually, so:
|
||||
- **Default = plain lazy replication-projection** — a freshness-stamped cache of remote
|
||||
content (§8.8). This is *the* projection for ordinary pages; it needs no taxonomy, no
|
||||
liveness reasoning, no registry. Most shards never touch anything below.
|
||||
- **Extension point — derivation-projection** — invoked *only* for content that is a
|
||||
*source* needing transform/compile/weave/evaluate (computational/typed content, §8.5). It
|
||||
adds the liveness axis (static → captured → live-over-files → view-time → irreducibly-live)
|
||||
and facets (materialization timing, multiplicity, continuity); the irreducibly-live far end
|
||||
has no faithful static form (source + a marked recording). A binding that never serves such
|
||||
content never instantiates any of this.
|
||||
- Both kinds stamp freshness + provenance; only derivation carries the liveness machinery.
|
||||
- **Moldable view registry — also an extension point, not a tax on every page.** Where a content
|
||||
type offers multiple co-equal views (typed/computed/dimensional content), they are registered
|
||||
as an **open, type-keyed set, none canonical-by-fact** (display-canonical is policy; GT prior
|
||||
art, answers the "pluggable content-type registry" question). An ordinary Markdown page has
|
||||
exactly one view and never consults the registry — the registry is queried only when a type
|
||||
declares >1 view.
|
||||
- **Derived query index** — delegate to a shard's native query engine where present
|
||||
(Roam/Logseq Datalog, Notion DB query, XWiki XWQL, Wikibase SPARQL); else build a derived
|
||||
index over the projection (the Logseq DataScript-over-files pattern). The index is
|
||||
disposable (I-2).
|
||||
|
||||
### 8.5 Computational / executable content — the scope decision
|
||||
|
||||
**In scope as a page-model + projection concern; out of scope as an execution platform.**
|
||||
shard-wiki *recognises* computational types, attaches the **canonical source**, and presents
|
||||
derived forms as **provenance- and liveness-marked projections**. Driving a derivation
|
||||
(tangle/weave, re-execute a notebook, render a sketch, evaluate a pattern) is a **gated
|
||||
capability, off by default, with a trust/sandbox concern, degrading to a captured snapshot**.
|
||||
One snapshot-provenance record (run id, source rev, timestamp, environment "unguaranteed")
|
||||
serves notebooks, renders, and recordings alike. **No INTENT amendment is required** — this
|
||||
lives inside the existing page model (L2) and projection model (L4).
|
||||
|
||||
### 8.6 Consistency, concurrency & conflict model
|
||||
|
||||
INTENT makes real-time cross-shard consistency a non-goal — but "no strong consistency" is not
|
||||
the same as "no defined consistency." This is the guarantee shard-wiki *does* offer, and the
|
||||
mechanism (not policy) that makes concurrent editing safe (review bug B-2).
|
||||
|
||||
**The consistency guarantee — causal, anchored on the journal:**
|
||||
|
||||
- **Read-your-writes for coordination-canonical state.** Once an overlay/binding/merge is
|
||||
appended to the space's decision log, every reader of that space sees it — because the log is
|
||||
a **single totally-ordered sequence per space** (one append authority, §8.1), and all readers
|
||||
resolve current state from that one order. The guarantee holds across orchestrator instances,
|
||||
not just within one process; it is cheap because ordering is per-space, never global.
|
||||
- **Causal consistency across the derived tier.** The union/index/projections reflect a causal
|
||||
cut of `(sharded inputs seen so far, journal)`. Effects never appear before their causes; a
|
||||
projection that has seen journal commit *C* has seen everything *C* depends on.
|
||||
- **Eventual convergence for sharded-canonical inputs.** Remote shard content is pulled
|
||||
asynchronously (lazily or by notify/poll, §8.7); the union converges to each shard's latest
|
||||
*as observed*, bounded by the shard's operational envelope. Freshness is always *shown*
|
||||
(provenance envelope), never faked — a stale projection is labelled stale, not wrong.
|
||||
|
||||
So: **strong + read-your-writes for what shard-wiki owns (the journal); causal for what it
|
||||
derives; eventual + freshness-labelled for what shards own.** No global clock, no distributed
|
||||
transaction, no two-phase commit across shards — none is needed, because shard-wiki coordinates
|
||||
rather than controls.
|
||||
|
||||
**Conflict detection & representation is core mechanism; only resolution is policy (I-7).**
|
||||
The split the earlier draft elided:
|
||||
|
||||
- **Detection (core).** Divergence is detected structurally: two identities resolve as
|
||||
equivalent (§8.4) but their content fingerprints differ, or an overlay's base revision no
|
||||
longer matches the shard's current revision. Detection is always on; it is never optional.
|
||||
- **Representation (core).** A detected conflict is **first-class data in the union**, not an
|
||||
error: equivalent-but-divergent pages are presented as a **coexisting set** (the
|
||||
chorus/keep-both representation), each fully attributed, with the divergence recorded in the
|
||||
provenance envelope (union without erasure — a conflict is information, not a failure).
|
||||
- **Resolution (policy).** *Which* version wins, or whether they stay coexisting, is a
|
||||
configurable preset (§10): chorus / designated-canonical / git-merge / vote-to-merge /
|
||||
overlay-only. Core never hard-codes one.
|
||||
|
||||
**Overlay-apply under source drift (the concurrent-write case).** An overlay carries the
|
||||
**base revision** of the shard content it was authored against. On apply, core compares base to
|
||||
the shard's current revision:
|
||||
|
||||
- *unchanged* → apply (fast-forward), commit to journal, propagate if the profile permits;
|
||||
- *changed, non-overlapping* → three-way merge where the merge capability allows (axis 6),
|
||||
else keep-both;
|
||||
- *changed, overlapping* → **refuse + re-present** as a conflict (above); never silently
|
||||
clobber (I-5, no silent remote mutation). The unapplied overlay remains coordination-
|
||||
canonical and valid against its base.
|
||||
|
||||
**Ordering.** The journal commit is the ordering authority for coordination-canonical effects;
|
||||
a shard-native write is only *acknowledged* in the journal after the adapter confirms it, so a
|
||||
crash between journal-intent and shard-write is recoverable (the intent is replayable, the
|
||||
write is idempotent-keyed on identity+base-rev). Cross-shard operations are ordered by their
|
||||
journal commits, giving the causal cut above.
|
||||
|
||||
**Residual open items** (tracked in *Known scaling risks & open problems*, §12, not pretended
|
||||
solved): the exact convergence bound for
|
||||
high-write CRDT shards under partition, and whether per-equivalence-set divergence needs a
|
||||
vector clock vs. a simple base-rev comparison, are deferred to implementation spikes.
|
||||
|
||||
### 8.7 Scaling the union — incremental-first, rebuild as fallback
|
||||
|
||||
The derived tier is *recomputable* (I-2) but recompute must never be the **operational**
|
||||
mechanism. A from-scratch rebuild reads every page of every shard — including rate-limited,
|
||||
paginated external APIs (Notion) and irreducibly-live sources — which can take hours to days
|
||||
and directly fights the operational-envelope axis (review C-2). So:
|
||||
|
||||
**Incremental, change-driven maintenance is the primary mechanism.** Each shard's `notify`
|
||||
capability (or a poll/ETag fallback where it has none, §8.8) emits **change events**; an event
|
||||
drives a **delta update** to exactly the affected union nodes, equivalence candidates, indexes,
|
||||
and projections. The derived tier is a continuously-maintained materialised view, not a
|
||||
periodically-recomputed one. Steady-state cost is O(changes), not O(corpus).
|
||||
|
||||
**Full rebuild is a rare, bounded fallback** — for cold start, schema/algorithm change, or
|
||||
suspected corruption — and it is **explicitly not required to be cheap**. It respects each
|
||||
shard's envelope (it may be slow, throttled, or resumable for a rate-limited shard) and runs
|
||||
*concurrently with serving the existing derived tier*; it swaps in atomically on completion.
|
||||
I-2 guarantees rebuild is *possible and correct*, not instant.
|
||||
|
||||
**Equivalence detection is indexed, not pairwise (review C-1).** Naive fingerprint/span-set
|
||||
comparison across all pages of all shards is O(N²) and is forbidden. Instead:
|
||||
|
||||
1. **Blocking / candidate generation** — cheap keys bucket pages that *could* be equivalent:
|
||||
normalised title, normalised path tail, explicit alias-table entries (coordination-
|
||||
canonical), and **MinHash/LSH bands over content shingles** for near-duplicate and
|
||||
derived-content detection. Only within-bucket pairs are considered — turning O(N²) into
|
||||
≈O(N) candidates.
|
||||
2. **Verification** — candidate pairs are confirmed by full fingerprint / span-set overlap and
|
||||
any curator binding. Confirmed equivalences become union edges.
|
||||
3. **Incremental maintenance — the delta is *not* additive (review B-4).** A changed page may
|
||||
*leave* buckets as well as *enter* them, and leaving a bucket can **break an existing
|
||||
equivalence edge** another page relied on. So a change is processed as: (i) recompute the
|
||||
page's bucket membership; (ii) for buckets it **left**, re-verify the pairs that depended on
|
||||
the shared bucket and **retract** edges no longer supported; (iii) for buckets it **entered**,
|
||||
verify the new candidate pairs and **add** edges; (iv) **propagate** to the equivalence
|
||||
neighbours of any retracted/added edge (equivalence is transitive-ish via chorus sets, so a
|
||||
retraction can split a set). Maintenance is per-change and bounded by the page's
|
||||
neighbourhood, but it covers retraction and propagation — not just additions.
|
||||
|
||||
**The index is itself derived** (disposable, recomputable) and per-tenant-partitioned (§9).
|
||||
Its parameters (LSH band/row counts, shingle size, precision/recall) are tunable; the accepted
|
||||
**false-negative rate of blocking** is a known, tracked limitation (§12) — blocking trades a
|
||||
small miss rate for tractability, and curator bindings are the escape hatch for misses.
|
||||
|
||||
**Verifying I-2 (`derived = f(canonical)`) — eventually, not on faith (review B-4).**
|
||||
Incremental maintenance can drift from a from-scratch fold over time (a missed retraction, a
|
||||
dropped event, a bug). I-2 is therefore an **eventually-verified** property, not a free one,
|
||||
and the architecture names the mechanism that verifies it:
|
||||
|
||||
- **A digest of the derived tier.** Each partition's derived tier carries a rolling content
|
||||
digest (a Merkle-style hash over union nodes/edges/index entries) maintained alongside the
|
||||
incremental updates.
|
||||
- **A background consistency-checker** periodically recomputes the digest over a *sampled* (or,
|
||||
on a slow cadence, full) fold of canonical state and compares. A mismatch localises the drift
|
||||
to a partition/region and triggers a **scoped recompute** of just that region — cheap relative
|
||||
to a global rebuild, and self-healing.
|
||||
- **So I-2 holds *eventually and verifiably*:** the incremental engine is the fast path, the
|
||||
checker is the guarantee, and divergence is detected and repaired rather than silently
|
||||
accumulating. The exact sampling rate / digest granularity is an implementation spike (§12).
|
||||
|
||||
### 8.8 Cache freshness & invalidation
|
||||
|
||||
Replication-projection caches remote shard content; cache invalidation is the actual hard part
|
||||
and was missing from the first draft (review C-2). The protocol is **per-binding, driven by the
|
||||
capability profile**, with one rule: **freshness is always represented, never assumed** — every
|
||||
cached page's provenance envelope carries `(observed-at, source-rev-if-known, staleness-state)`,
|
||||
so a consumer can always tell live from stale.
|
||||
|
||||
**Three invalidation modes, chosen by capability, not hard-coded:**
|
||||
|
||||
| Mode | When | Mechanism |
|
||||
|------|------|-----------|
|
||||
| **Event-driven (push)** | shard has `notify` | a change event invalidates exactly the affected entries and enqueues a delta refresh (§8.7); the preferred mode |
|
||||
| **Validator poll** | shard exposes ETag / Last-Modified / rev | conditional fetch (`If-None-Match`); cheap "still fresh?" checks without transferring bodies |
|
||||
| **TTL** | shard offers neither | time-bounded staleness; the floor mode (Oddmuse-class shards) |
|
||||
|
||||
Most real bindings are **hybrid**: event-driven for invalidation + a long TTL as a safety net
|
||||
for missed events + validator polls on read when an entry is past a soft age.
|
||||
|
||||
**Operational-envelope coupling.** The mode is constrained by axis-10: a **rate-limited** shard
|
||||
(Notion) *must* favour event-driven + long TTL and *must not* poll per-read — the freshness
|
||||
policy is capability-gated like everything else. A local file shard can watch the filesystem
|
||||
(near-instant invalidation, effectively event-driven for free).
|
||||
|
||||
**Thundering-herd / coalescing.** Concurrent reads of the same stale entry trigger a **single
|
||||
in-flight refresh** (single-flight); other readers await it or are served the stale-but-labelled
|
||||
value per policy. Bulk invalidations (a shard-wide event) are **batched and rate-shaped** to the
|
||||
shard's envelope rather than fired as N concurrent fetches.
|
||||
|
||||
**Staleness is a policy knob, not a correctness bug.** Whether a reader gets *stale-but-fast* or
|
||||
*blocks-for-fresh* is a §10 preset (per space or per request); either way the envelope tells the
|
||||
truth about what was served. This is union-without-erasure applied to time.
|
||||
|
||||
---
|
||||
|
||||
## 9. Cross-cut — Authorization (L5)
|
||||
|
||||
Fully specified in **`ArchitectureBlueprint.md`** (the access & history sub-blueprint);
|
||||
summarised here for completeness:
|
||||
|
||||
- **One core, a ladder of modes** L0 (open/c2, zero deps) → L1 (attributed) → L2
|
||||
(authenticated) → L3 (role/group) → L4 (multi-tenant enterprise). Climbing is configuration,
|
||||
not re-architecture.
|
||||
- **PEP** wraps every adapter op; **PDP** decides `(principal, action, target)` over actions
|
||||
`read/write/patch/merge/administer`, layered on the adapter's capability profile (a shard
|
||||
that can't write can't be written regardless of policy — L5 consults the L1 rail).
|
||||
- **Authentication delegated** to a pluggable IdentityProvider (null provider = L0 default);
|
||||
real identity from `user-engine` over `net-kingdom` IAM.
|
||||
- **Fail open only at L0, fail closed at L2+.** Authorization is pure/offline once a Principal
|
||||
is resolved. Provenance carries authz context so the union never leaks unreadable content
|
||||
(the L5↔provenance-rail interaction).
|
||||
|
||||
### 9.1 Tenant isolation of the derived tier (review B-3)
|
||||
|
||||
Read-time authz filtering is necessary but **not sufficient** when the derived tier is
|
||||
*persisted*: a single cross-tenant union/index cache guarded only by a filter on read is a
|
||||
standing leak surface (one filtering bug exposes another tenant's content). So isolation is
|
||||
**structural, not just procedural**:
|
||||
|
||||
- **The derived tier is partitioned per tenant / root entity.** A tenant maps to a root entity
|
||||
(§4); its union graph, equivalence index, projections, and caches live in a **separate
|
||||
partition** keyed by that tenant. There is no shared cross-tenant derived store to leak from.
|
||||
- **No cross-tenant equivalence by default.** Blocking/LSH (§8.7) operates *within* a partition;
|
||||
cross-tenant equivalence is an explicit, authorised, opt-in federation between roots, never an
|
||||
accident of a shared index.
|
||||
- **Read-time filtering remains, as defence-in-depth** — the provenance envelope's authz context
|
||||
is still checked, so even within a partition a principal sees only what it may; partitioning
|
||||
removes the *blast radius*, filtering removes the *fine-grained* leak.
|
||||
- **This reconciles I-2 with L5:** recomputability (a persisted-but-disposable derived tier) is
|
||||
preserved *per partition* — each tenant's derived tier is independently rebuildable from that
|
||||
tenant's canonical state — so isolation costs nothing in the rebuild model. At L0/L1 (single
|
||||
tenant) there is one partition and the machinery is invisible.
|
||||
|
||||
**Isolation invariant (add to §2 as I-13):** *derived state is partitioned by tenant; no
|
||||
derived artifact spans tenants except through an explicit, authorised cross-root federation.*
|
||||
|
||||
---
|
||||
|
||||
## 10. The policy surface (mechanism over policy, made concrete)
|
||||
|
||||
I-7 only means something if the policy knobs are enumerated and kept *out* of core algorithms.
|
||||
The configurable presets are:
|
||||
|
||||
- **Canonical-source policy** — chorus / designated-canonical / git-merge / overlay-only /
|
||||
vote-to-merge (per space or per equivalence set).
|
||||
- **Federation model** — the §8.3 taxonomy, per space, composable per shard.
|
||||
- **Shard mode** — read-only / write-through / mirrored / projected / cached / canonical
|
||||
(constrained by the capability profile).
|
||||
- **Reconciliation cadence & conflict exposure** — push/poll/manual; show-conflicts vs
|
||||
auto-merge-when-supported.
|
||||
- **Conflict-resolution preset** — chorus / designated-canonical / git-merge / vote-to-merge /
|
||||
overlay-only (the *resolution* policy over §8.6's core detection; per space or equivalence set).
|
||||
- **Freshness / invalidation mode** — event-driven / validator-poll / TTL / hybrid, and
|
||||
stale-but-fast vs block-for-fresh on read (§8.8; constrained by the operational envelope).
|
||||
- **History compaction** — squash policy for low-value churn, gc/repack cadence, per-shard
|
||||
offload (§8.1), always preserving recoverable endpoints.
|
||||
- **Tenant partition mapping** — tenant ↔ root-entity, and any explicit cross-root federation
|
||||
(§9.1, I-13).
|
||||
- **Execution policy** — derive/execute off (default) / sandboxed / per-shard-allowed.
|
||||
- **Authorization mode** — the L0–L4 ladder.
|
||||
- **Projection materialization** — lazy/eager; snapshot vs view-time; recording retention.
|
||||
|
||||
Core ships sane defaults (L0 open; fork+journal; lazy replication-projection; event-driven+TTL
|
||||
freshness; overlay-before-mutation; execution off; one tenant = one root) and never hard-codes
|
||||
any of the above. (**Preset bundles** that package coherent knob-sets per persona are tracked
|
||||
as O-8, §12 — flexibility without bundles is operator burden.)
|
||||
|
||||
---
|
||||
|
||||
## 11. Concrete module structure (bridge to implementation)
|
||||
|
||||
A proposed package layout for `src/shard_wiki/`, mapping 1:1 to the layers so the dependency
|
||||
rule (downward only; the derived tier is incrementally maintained, rebuild = fallback) is
|
||||
enforceable by import lint:
|
||||
|
||||
```
|
||||
src/shard_wiki/
|
||||
model/ # L2 top waist: Page, Identity, Placement, ProvenanceEnvelope,
|
||||
# Span, the page-shape types; capability-spectrum value types
|
||||
adapters/ # L1 bottom waist: AdapterContract (versioned iface), CapabilityProfile,
|
||||
# attachment-mode binding; concrete adapters:
|
||||
git/ folder/ gitea/ obsidian/ webdav/ notion/ … # each: profile + verbs
|
||||
coordination/ # L3: DecisionLog (append-only, git-backed, per-space append authority/
|
||||
# lease), OverlayEngine (draft→patch→MR), reconcile
|
||||
# (current coordination state = a derived fold → lives in union/)
|
||||
federation/ # L3: FederationModel strategies (fork_journal, vcs_ping,
|
||||
# graph_join, feed, activitypub, engine_mirror)
|
||||
union/ # L4 (derived): IdentityResolver, EquivalenceGraph, UnionGraph,
|
||||
# Transclusion (reference-not-copy)
|
||||
projection/ # L4 (derived): ReplicationProjection, DerivationProjection,
|
||||
# ViewRegistry (moldable), QueryIndex (delegate|derive)
|
||||
authz/ # L5 cross-cut: PDP, PEP, IdentityProvider iface, NullProvider
|
||||
provenance/ # cross-cut LEAF: ProvenanceEnvelope type + ⊕ (effective) only — pure data
|
||||
policy/ # cross-cut LEAF: the §10 policy surface (presets + a resolve() read by
|
||||
# coordination/federation/projection/authz); owns NO mechanism
|
||||
api/ # L6: orchestrator API (server-side union for agents/CLI)
|
||||
```
|
||||
|
||||
**The cross-cutting rails are leaves, not god-modules (review D-4).** `provenance/` and
|
||||
`policy/` are imported widely, so they are the highest coupling risk; the discipline that caps
|
||||
it is: **they may import *nothing* in the tree and contain *only* stable data types + pure
|
||||
functions** (the envelope and its `⊕`; the policy presets and a `resolve(question) → choice`).
|
||||
Mechanism never lives in a rail — `policy/` says *what* the preset is, `coordination/`/
|
||||
`projection/` decide *how* to honour it. A change to a rail is then a change to a small, stable,
|
||||
dependency-free leaf, not a ripple through every layer. Capability-spectrum value types live in
|
||||
`model/` (also leaf-like) for the same reason.
|
||||
|
||||
Hard import rules (enforced by import lint):
|
||||
- `union/` and `projection/` may import `model/`, `adapters/`, `coordination/`, `policy/`,
|
||||
`provenance/` — but **nothing may import them** (they are the disposable derived tier).
|
||||
- `model/`, `adapters/`, `provenance/`, `policy/` import nothing else in the tree (the waists
|
||||
and rails stay thin); `provenance/` and `policy/` import nothing at all.
|
||||
- `coordination/` and `federation/` may import the waists + rails, never the derived tier.
|
||||
|
||||
---
|
||||
|
||||
## 12. Known scaling risks & open problems
|
||||
|
||||
Tracked honestly rather than pretend-solved (review disposition F). Each has a **chosen
|
||||
direction** and a **revisit trigger** — the thing that, if observed, forces a redesign.
|
||||
|
||||
| # | Risk / open problem | Chosen direction | Revisit trigger |
|
||||
|---|---------------------|------------------|-----------------|
|
||||
| O-1 | **Equivalence blocking misses true matches** (LSH false negatives, §8.7) | accept a small miss rate; curator bindings are the escape hatch | measured recall below an agreed threshold on real corpora |
|
||||
| O-2 | **Convergence bound for high-write CRDT shards under partition** (§8.6) | causal via journal + CRDT-native merge at the shard; no global bound promised | user-visible divergence that outlives a partition |
|
||||
| O-3 | **Per-equivalence-set divergence tracking** (§8.6) | start with base-rev comparison; add vector clocks only if needed | 3-way concurrent divergence that base-rev mis-orders |
|
||||
| O-4 | **Persisted derived-tier cost ceiling** (§8.7/§9.1) | per-tenant partition, incremental-maintained, rebuild is fallback | a tenant whose incremental cost still exceeds budget |
|
||||
| O-5 | **Axis-interaction completeness** (§6.5) | the named interaction table is the contract; extend deliberately | a real adapter needing an interaction not in the table |
|
||||
| O-6 | **Span-address portability across projection** (§7.2) | shard-scoped native-id wrapping now; tumbler later | cross-shard transclusion that native ids can't satisfy |
|
||||
| O-7 | **Squash-compaction vs. perfect auditability** (§8.1) | compact the *path*, preserve reachable states; configurable | a compliance need for every intermediate keystroke |
|
||||
| O-8 | **Policy-knob proliferation → operator burden** (§10) | ship named **preset bundles** ("personal vault" / "team wiki" / "enterprise federation") over the policy surface | operators mis-configuring interacting knobs |
|
||||
| O-9 | **Shard sharing across roots vs tenant partition** (§9.1, I-13) | shard exclusive to one root by default; explicit shared-read binding otherwise (avoids double-caching a rate-limited shard) | a shard legitimately needed live in two tenants |
|
||||
| O-10 | **Span-level authz under transclusion** (aggregation/inference leak; ⊕ across boundaries, §7.3/§9) | a transcluded span inherits the **stricter** of source & host authz; provenance ⊕ composes the source-page envelope under the host | a real cross-authz transclusion |
|
||||
| O-11 | **Union under shard unavailability** (§8.8 covers stale, not down) | **partial union** + per-shard "unavailable" provenance + last-known-projection where policy allows | an SLA need on partial reads |
|
||||
| O-12 | **Per-space append-log throughput ceiling** (§8.1 append authority) | single writer per space scales across spaces; per-space log sharding if needed | a single space exceeding one writer's append rate |
|
||||
|
||||
These are the spec-writing inputs for `SHARD-WP-0002`; none blocks the architecture, each
|
||||
scopes an implementation spike.
|
||||
|
||||
---
|
||||
|
||||
## 13. Canonical data flows (the architecture exercised)
|
||||
|
||||
**A. Attach a shard.** Adapter binds (chosen attachment mode) → probes/declares a capability
|
||||
profile → core registers the shard under a root entity → if not git-native, the coordination
|
||||
journal is seeded (begin-now/mirror/import per axis 5). No union rebuild yet (lazy).
|
||||
|
||||
**B. Read a page through the union.** Consumer asks the union for an identity → Identity
|
||||
resolver maps it to placements across shards → equivalence yields chorus or canonical →
|
||||
replication-projection lazily fetches from each shard (cache + freshness) → page returned
|
||||
wrapped in its provenance envelope → L5 filters anything the principal can't see at source.
|
||||
|
||||
**C. Edit a read-only / limited shard.** Write request → L5 PDP allows → capability profile
|
||||
says < write-through → OverlayEngine records a draft → renders a patch/MR in the shard's native
|
||||
syntax (lossless) or Markdown (lossy-with-report) → on explicit apply, commit to the journal
|
||||
and (if the profile permits) propagate; otherwise the overlay stands as the local truth, fully
|
||||
attributed.
|
||||
|
||||
**D. Attach a computational notebook.** Adapter declares profile (attachment=file-store,
|
||||
opacity=mixed, computational=captured-output). Core attaches the `.ipynb` **source** as
|
||||
canonical; presents cells + embedded outputs as **derivation-projection snapshots** marked
|
||||
"run N, env unguaranteed"; offers a static render via the view registry; re-execution stays
|
||||
gated off. History uses paired-text/nbdime per axis 5.
|
||||
|
||||
---
|
||||
|
||||
## 14. Key tradeoffs & decisions
|
||||
|
||||
Decided:
|
||||
|
||||
- **Capability spectra over a verb checklist** — richer contract for precise, uniform
|
||||
degradation; tamed by an orthogonal core + implied positions + a named interaction table
|
||||
(§6.5). (Decided.)
|
||||
- **Three states; derived = f(canonical)** — sharded + coordination canonical, derived
|
||||
disposable (§1). (Decided; supersedes the earlier "edges vs middle" framing.)
|
||||
- **Event-sourced coordination, one append authority per space** — coordination-canonical state
|
||||
is an append-only **decision log** in the git journal; current state is a derived fold; a
|
||||
per-space append lease gives a totally-ordered log and read-your-writes across orchestrator
|
||||
instances (§8.1). (Decided — resolves the single-vs-multi-writer keystone.)
|
||||
- **Profiles are verified, not asserted** — a versioned **conformance suite** gates adapter
|
||||
admission; capability-as-data acts on verified data (§6.6). (Decided.)
|
||||
- **I-2 is eventually-verified** — incremental maintenance is the fast path; a digest +
|
||||
background consistency-checker detects and self-heals drift (§8.7). (Decided.)
|
||||
- **Incremental-first, rebuild-as-fallback** — the derived tier is continuously maintained from
|
||||
change events; full rebuild is rare and need not be cheap (§8.7). (Decided — resolves the
|
||||
earlier "union graph persistence" open item: **persisted, per-tenant, incrementally
|
||||
maintained, rebuildable**, §9.1.)
|
||||
- **Identity ≠ fingerprint** — page identity is a stable handle; fingerprints are for
|
||||
equivalence (§7.2). (Decided.)
|
||||
- **Consistency = read-your-writes (journal) + causal (derived) + eventual/freshness-labelled
|
||||
(shards)**; conflict detection/representation is core, resolution is policy (§8.6). (Decided.)
|
||||
- **Address scheme** — shard-scoped native-id wrapping now; portable tumbler later (§7.2, O-6).
|
||||
(Decided.)
|
||||
- **Default federation = fork+journal over Git**; other models opt-in (§8.3). (Decided.)
|
||||
- **Execution off by default** — recognise+project always; execute only when gated (§8.5). (Decided.)
|
||||
- **Derived tier is tenant-partitioned** (I-13, §9.1). (Decided.)
|
||||
|
||||
Still open (carried to §12 / policy):
|
||||
|
||||
1. **L1 "attributed-but-open" mode** — ship it or jump L0→L2? (Carried from ArchitectureBlueprint.)
|
||||
2. **Per-page ACL default** — off (per-shard/namespace) confirmed; revisit only if demand appears.
|
||||
3. The implementation spikes in **§12** (O-1…O-7).
|
||||
|
||||
---
|
||||
|
||||
## 15. What this architecture is *not*
|
||||
|
||||
- Not a wiki engine, UI, or rendering pipeline (those are consumers at L6).
|
||||
- Not a canonical-source-of-truth — shards keep sovereignty; the middle is derived.
|
||||
- Not a generic file-sync daemon — synchronisation is wiki-page-semantic.
|
||||
- Not an execution platform — computation is recognised and projected, not hosted.
|
||||
- Not a universal ontology — no single schema is imposed on all shards.
|
||||
- Not an authentication/identity store — that is delegated (authorization is owned).
|
||||
|
||||
---
|
||||
|
||||
## 16. Traceability
|
||||
|
||||
- **INTENT** — every invariant in §2 (I-1…I-13) cites an INTENT principle or boundary; no
|
||||
invariant contradicts the Stability Note.
|
||||
- **Review & hardening** — this revision folds in
|
||||
`history/260615-core-architecture-blueprint-review.md` via **`SHARD-WP-0005`**: A-1→§1/§3/§4
|
||||
(three-state re-frame), B-1→§7.2 (identity vs equivalence), B-2→§8.6 (consistency/conflict),
|
||||
B-3→§9.1+I-13 (tenant isolation), C-1/C-2→§8.7/§8.8 (incremental + indexed + invalidation),
|
||||
C-3→§8.1 (history scaling), D-1→§6.5 (orthogonal core), D-2→§7.3 (layered provenance),
|
||||
D-3→§8.4 (common-case projection), D-4→§11 (policy module + rail discipline); open items→§12.
|
||||
- **Round-2 review & hardening II** — folds in
|
||||
`history/260615-core-architecture-blueprint-review-2.md` via **`SHARD-WP-0006`**:
|
||||
A-1…A-4→§3/§4/§10/§11 (overview reconciled to the body), B-1+B-3→§8.1 (event-sourced
|
||||
coordination + per-space append authority), B-2→§6.6 (adapter conformance suite),
|
||||
B-4→§8.7 (incremental retraction/propagation + I-2 digest/checker); C-1…C-4→§12 (O-8…O-11).
|
||||
- **Research** — §6 (spectra) ← `260614-shard-spectrum-synthesis` v3; §8.3 (federation
|
||||
taxonomy) ← v3 §2.5; §8.4–§8.5 (two-axis projection, view registry, computational scope) ←
|
||||
`260614-computational-page-model-synthesis`; §7 page shapes ← the engine + modern-tool +
|
||||
computational dives; §1 thesis ← the files-canonical/index-derived through-line across
|
||||
Logseq/ikiwiki/GT/Pharo/Jupyter.
|
||||
- **Use cases** — the architecture is sized to UC-01–UC-84: federation/coordination (UC-01–07,
|
||||
26–33, 56, 71–72, 79) → §8; attachment/adapter (UC-34–43, 50, 53, 57, 60–62, 64–66, 68–70,
|
||||
76–82) → §6; page model & fidelity (UC-34, 39, 42, 55, 58–59, 67, 73, 80, 83–84) → §7/§8.5;
|
||||
addressing/identity/query (UC-32, 44–49, 51–52, 54, 63, 74) → §7.2/§8.4; provenance &
|
||||
metadata (UC-24–25, 75) → the provenance rail; collaboration & discovery (UC-08–23) → L6
|
||||
consumers over the union.
|
||||
- **Workplans** — §6–§8 are the design target of `SHARD-WP-0002` (T11–T18); §9 is owned by
|
||||
`ArchitectureBlueprint.md`; §1 (yawex-derived resolution/overlay) aligns with
|
||||
`SHARD-WP-0001`.
|
||||
|
||||
---
|
||||
|
||||
## 17. Stability note
|
||||
|
||||
This document defines shard-wiki's **internal** architecture; it may evolve as the spec
|
||||
workplans land. But the **thesis (§1)**, the **invariants (§2)**, and the **dual narrow waist
|
||||
(§1, §6, §7)** are load-bearing — changing any of them is an architectural change in the sense
|
||||
of INTENT's Stability Note and should be rare and deliberate.
|
||||
220
spec/FederationArchitecture.md
Normal file
220
spec/FederationArchitecture.md
Normal file
@@ -0,0 +1,220 @@
|
||||
# FederationArchitecture
|
||||
|
||||
Status: **draft for review** · Date: 2026-06-15 · Deliverable of **SHARD-WP-0002** (T1–T10, T17)
|
||||
|
||||
The federation **design decisions** for shard-wiki: *what the union does*. It records, per
|
||||
topic, a decision with rationale, tradeoffs, and a **Decided / Deferred / Open** footer. It
|
||||
**references** `spec/CoreArchitectureBlueprint.md` (the whole-system architecture) and
|
||||
`spec/FederationRequirements.md` (yawex-derived ADRs) rather than restating them; the adapter
|
||||
contract (*what a backend must expose*) is the companion deliverable in
|
||||
`spec/TechnicalSpecificationDocument.md` §A (T11–T16, T18). UC references → `UseCaseCatalog.md`.
|
||||
|
||||
Cross-cutting invariants assumed throughout (blueprint §2): orchestrator-not-engine (I-1),
|
||||
three-state canonical/derived (I-2), capability-as-data (I-3), union-without-erasure (I-4),
|
||||
overlay-before-mutation (I-5), git-addressable coordination (I-6), mechanism-over-policy (I-7),
|
||||
graceful degradation (I-8), identity≠placement (I-9), history-as-floor (I-10), authz-in-core
|
||||
(I-11), tenant-partitioned derived state (I-13).
|
||||
|
||||
---
|
||||
|
||||
## T1 — Orchestrator positioning & boundaries
|
||||
|
||||
**Decision.** shard-wiki is an **adapter-layer orchestrator** in a **star** shape (many
|
||||
sovereign shards ↔ one information space), **not** a homogeneous federation network. It
|
||||
**compares, does not equate** (Federated Wiki homogeneous JSON sites, ikiwiki homogeneous git
|
||||
wikis, ActivityPub activity streams are *models it can speak*, §T17 — not shapes it imposes).
|
||||
Core owns: the union, the coordination journal, projection, overlay, resolution, authorization.
|
||||
Adapters own backend specifics; UI and editorial policy live outside core. (Blueprint §1, §5;
|
||||
UC-02, UC-26.)
|
||||
|
||||
*Decided:* star orchestrator; compare-not-equate; core/adapter/UI/policy boundaries.
|
||||
*Deferred:* the reference UI (L6) is out of scope here. *Open:* none.
|
||||
|
||||
## T2 — Remix primitives: reference / projection / overlay / import-fork
|
||||
|
||||
**Decision.** Four primitives, escalating in commitment; **overlay-before-mutation is the
|
||||
default write path** (I-5), fork is *one federation model* (§T17) not the default for editing:
|
||||
|
||||
| Primitive | Trigger | Writes remote? | Coordination-canonical? |
|
||||
|-----------|---------|----------------|--------------------------|
|
||||
| **Reference** | link only | no | no (a link in content) |
|
||||
| **Projection** | read remote page | no (cache) | no (derived) |
|
||||
| **Overlay** | edit a sub-write-through shard | not until explicit apply | **yes** (decision log) |
|
||||
| **Import / fork** | copy into a writable shard / fork-with-provenance | source unchanged | **yes** (fork event) |
|
||||
|
||||
(Blueprint §8.2; FederationRequirements ADR-05; UC-04, UC-26, UC-29.)
|
||||
|
||||
*Decided:* the four primitives + overlay-default. *Deferred:* fork-attribution portability
|
||||
format. *Open:* whether "import" and "fork" are one primitive with a flag or two (impl spike).
|
||||
|
||||
## T3 — Equivalent-page identity & multi-version presentation
|
||||
|
||||
**Decision.** Separate **identity** (stable handle), **placement** (N paths/shards), and
|
||||
**equivalence** (content-fingerprint / span-set overlap *across* identities), per
|
||||
FederationRequirements ADR-01/ADR-02 (I-9). Equivalence signals: normalized title/path, alias
|
||||
table (coordination-canonical), link-graph overlap, fingerprint, **curator binding**. Default
|
||||
presentation = **chorus-of-voices** (all equivalent versions, attributed); **designated-
|
||||
canonical** is a policy preset (§T9). Divergence is data in the provenance envelope (I-4),
|
||||
detected as core mechanism (blueprint §8.6). (UC-07, UC-27; UC-46.)
|
||||
|
||||
*Decided:* identity/placement/equivalence split; chorus default. *Deferred:* fingerprint
|
||||
algorithm + blocking params (blueprint O-1). *Open:* curator-binding UX (L6).
|
||||
|
||||
## T4 — History, attribution & the coordination journal
|
||||
|
||||
**Decision.** Each information space has a **git-addressable, event-sourced coordination
|
||||
journal** (blueprint §8.1, I-6/I-10): coordination-canonical decisions (overlay/binding/alias/
|
||||
merge/fork) are append-only events; current state is a derived fold. **One append authority per
|
||||
space** gives a total order (read-your-writes across instances). Per-shard native history is
|
||||
**adopted** (git-native), **supplemented** (DB/CRDT internal → journal begins-now/mirrors), or
|
||||
**imported** (open file history backfilled) per the history-portability axis (TSD §A T13).
|
||||
Attribution is portable on the event (UC-29). (UC-29, UC-33.)
|
||||
|
||||
*Decided:* event-sourced journal + per-space append authority + adopt/supplement/import.
|
||||
*Deferred:* per-space log sharding for extreme write rates (blueprint O-12). *Open:* none.
|
||||
|
||||
## T5 — Union composition layer
|
||||
|
||||
**Decision.** The **server-side orchestrator union is primary** — the derived tier (union
|
||||
graph, indexes, projections) is composed in core for agents/CLI/non-browser consumers
|
||||
(blueprint §8.4), **incrementally maintained** (not recomputed, §8.7), **tenant-partitioned**
|
||||
(I-13). **Client-side composition** (Federated-Wiki-style browser pull) is a supported
|
||||
*consumer pattern over the same union*, not the only path. Freshness/caching per §8.8. (UC-05,
|
||||
UC-27, UC-03, UC-31.)
|
||||
|
||||
*Decided:* server-primary, client-optional; incremental, partitioned. *Deferred:* client
|
||||
SDK shape. *Open:* persisted-union digest granularity (blueprint O-4, B-4 checker).
|
||||
|
||||
## T6 — Change notification & subscription transports
|
||||
|
||||
**Decision.** Change-notification is an **optional adapter capability** (`notify`), and it is
|
||||
the **primary driver of incremental maintenance** (blueprint §8.7/§8.8). Transports, per
|
||||
profile: git hook / ikiwiki-style ping, ActivityPub Create/Update, webhook, WebDAV/HTTP ETag
|
||||
or `Last-Modified` poll, plain polling fallback. A notification → invalidate affected entries
|
||||
+ enqueue delta refresh + a RecentChanges union entry (FederationRequirements ADR-03; UC-31,
|
||||
UC-17). Rate-limited shards favour event-driven + long TTL (operational-envelope axis).
|
||||
|
||||
*Decided:* notify = optional capability, drives incremental + RecentChanges; transport per
|
||||
profile. *Deferred:* ActivityPub as content-bearing (vs notify-only) — start notify-only.
|
||||
*Open:* fediverse-dependency posture for v1.
|
||||
|
||||
## T7 — Information-space lifecycle
|
||||
|
||||
**Decision.** Root entities have lifecycle states: **active → read-only-archived → retired
|
||||
(detached)**, and **merged-into-successor**. **Carry-forward** is selective import from an
|
||||
archived shard (UC-28, UC-30). **Space-fork / branch** (UC-33) = branch the coordination
|
||||
journal + shard-config (a fork event, §T2/§T17 fork+journal model). Retirement **preserves
|
||||
read-only union entries** by default (history-as-floor, I-10), never hard-deletes projections.
|
||||
(UC-28, UC-30, UC-33.)
|
||||
|
||||
*Decided:* the lifecycle states + carry-forward + space-branch; retire-preserves. *Deferred:*
|
||||
ephemeral "happenings" as a first-class mode. *Open:* GC policy for retired-space derived tier.
|
||||
|
||||
## T8 — Transclusion & projection depth
|
||||
|
||||
**Decision.** Transclusion is **one reference-not-copy primitive** over the addressable union
|
||||
(blueprint §8.4), at three depths:
|
||||
|
||||
| Level | UC | Behaviour |
|
||||
|-------|-----|-----------|
|
||||
| Whole-page projection | UC-03 | lazy full page from a remote shard (replication-projection) |
|
||||
| Block/span transclusion | UC-32 | inline embed by span address, with origin + freshness |
|
||||
| Link reference | UC-08 | pointer only (ADR-06) |
|
||||
|
||||
Every transcluded artifact carries provenance + **staleness** (I-4, §8.8); live-transclusion
|
||||
fragility (link rot / remote down) degrades to last-known + an `unavailable`/`stale` marker
|
||||
(blueprint O-11). Span-level authz under transclusion = the stricter of source/host (O-10).
|
||||
Reference-not-copy unifies Xanadu transclusion, ZigZag clone, embeds, synced blocks, Trilium
|
||||
note-clone, literate named-chunk. (UC-03, UC-32.)
|
||||
|
||||
*Decided:* one primitive, three depths, provenance+staleness mandatory. *Deferred:* snapshot-
|
||||
on-import vs live default (a policy). *Open:* span-authz composition (O-10), addressing (O-6).
|
||||
|
||||
## T9 — Consensus & reconciliation policy catalog
|
||||
|
||||
**Decision.** **Detection is core, resolution is policy** (I-7, blueprint §8.6). Core always
|
||||
detects divergence and represents it as a coexisting (chorus) set; the **resolution preset** is
|
||||
configurable per space / per equivalence set:
|
||||
|
||||
- **chorus-spread** (versions coexist; default) · **designated-canonical** (explicit write
|
||||
target) · **git-merge** (where merge capability allows) · **vote-to-merge / editorial gate**
|
||||
(optional, L6-assisted) · **overlay-only** (no destructive merge on read-only sources).
|
||||
|
||||
(FederationRequirements ADR-03/ADR-05; UC-07, UC-27.)
|
||||
|
||||
*Decided:* the preset catalog; detection-core/resolution-policy. *Deferred:* vote/editorial
|
||||
gate mechanics (L6). *Open:* default preset per persona bundle (blueprint O-8).
|
||||
|
||||
## T10 — Federation-operations capability matrix
|
||||
|
||||
**Decision.** Each federation operation requires a minimum **verified capability profile**
|
||||
(TSD §A T11/§6.6); below it, the op **degrades** along the named interaction axes (blueprint
|
||||
§6.5) rather than failing. The matrix (consuming the T11 vocabulary):
|
||||
|
||||
| Federation op | Min capabilities | Degradation when absent |
|
||||
|---------------|------------------|--------------------------|
|
||||
| **read / project** | `read` | (floor — every shard) |
|
||||
| **transclude span** | `read` + addressing≥span | whole-page projection only |
|
||||
| **overlay** | `read` | always available (overlay is local) |
|
||||
| **write-through** | `write` (+ `merge` for concurrent) | → overlay/patch target |
|
||||
| **diff / 3-way merge** | `diff`,`merge` | → keep-both / chorus |
|
||||
| **notify-driven freshness** | `notify` | → validator-poll → TTL (§8.8) |
|
||||
| **native search delegate** | native-query≥index | → derived index over projection |
|
||||
| **publish / mirror** | `publish` | → read-only / static projection |
|
||||
| **fork+journal federation** | `read` + history (any) | → projection-only participant |
|
||||
|
||||
Every backend, down to the Oddmuse flat-file floor, is usable as **read-only / cache /
|
||||
projection / backup / patch target** (I-8). This matrix and the T11 spectra are kept in sync
|
||||
(it consumes that vocabulary). (UC-02–UC-07.)
|
||||
|
||||
*Decided:* the op→capability matrix + degradation paths. *Deferred:* exhaustive per-op test
|
||||
vectors → the conformance suite (§6.6). *Open:* axis-interaction completeness (blueprint O-5).
|
||||
|
||||
---
|
||||
|
||||
## T17 — Federation-model taxonomy (selectable / composable)
|
||||
|
||||
**Decision.** Federation is **plural and composable**, not one mechanism (blueprint §8.3). A
|
||||
space selects a model (composing per shard); the default is **fork+journal over Git** (the home
|
||||
case):
|
||||
|
||||
| Model | Anchor | Coordination shape | Capability floor | UCs |
|
||||
|-------|--------|--------------------|------------------|-----|
|
||||
| **Fork + journal** *(default)* | Federated Wiki | copy-with-provenance + per-page action journal (story = replay) | `read` + journal | UC-26, UC-71, UC-72 |
|
||||
| **VCS-replication + ping** | ikiwiki | git clone/pull/push + change-ping | git-IS-store | UC-31, UC-33, UC-79 |
|
||||
| **Query-time graph-join** | Wikibase `SERVICE` | join remote graphs at query, no copy | native-query≥graph | UC-74 |
|
||||
| **Feed aggregation** | RSS/Atom | inbound feed → pages | `read` | UC-03 |
|
||||
| **Activity streams** | ActivityPub | Create/Update events (notify or content) | `notify` | UC-31 |
|
||||
| **Engine-mirror** | Wiki.js DB↔Git | engine mirrors its store to git | `publish`/mirror | UC-68, UC-69 |
|
||||
|
||||
The coordination journal (T4) is the fork+journal model's home; **git IS the journal** in
|
||||
VCS-replication and engine-mirror (forge wikis make git the store *and* journal, resolving the
|
||||
engine-mirror write-race). Models compose: e.g. fork+journal locally, query-join for a
|
||||
typed-graph shard, feed-aggregation for an external source. (Mechanism over policy, I-7.)
|
||||
|
||||
*Decided:* the six models, selectable per space + composable per shard, default fork+journal.
|
||||
*Deferred:* activity-streams content-bearing mode. *Open:* multi-model interaction edge cases
|
||||
(when two models touch one shard).
|
||||
|
||||
---
|
||||
|
||||
## Consolidated decisions / deferred / open
|
||||
|
||||
- **Decided (architecture-settling):** star orchestrator (T1); overlay-default remix (T2);
|
||||
identity/placement/equivalence + chorus (T3); event-sourced journal + append authority (T4);
|
||||
server-primary incremental union (T5); notify-driven freshness (T6); space lifecycle (T7);
|
||||
reference-not-copy transclusion (T8); detection-core/resolution-policy preset catalog (T9);
|
||||
op→capability matrix (T10); selectable/composable federation models (T17).
|
||||
- **Deferred (to impl / L6 / persona bundles):** reference UI; fork-attribution format; client
|
||||
SDK; vote/editorial mechanics; ephemeral spaces; ActivityPub content-bearing; default-preset
|
||||
bundles (O-8).
|
||||
- **Open (tracked in blueprint §12):** O-1 equivalence blocking, O-4 union digest, O-5 axis
|
||||
interactions, O-6 addressing, O-8 presets, O-10 span-authz, O-11 unavailability, O-12 log
|
||||
throughput.
|
||||
|
||||
## Acceptance (SHARD-WP-0002 federation half)
|
||||
|
||||
T1–T10 + T17 each have a decision record with a Decided/Deferred/Open footer; all honour INTENT;
|
||||
UC-26–UC-33, UC-56, UC-71–UC-72, UC-74, UC-79 trace here; the adapter-contract companion is
|
||||
`spec/TechnicalSpecificationDocument.md` §A (T11–T16, T18). Conflicts with SHARD-WP-0001 are
|
||||
none (FederationRequirements ADRs are consumed, not duplicated).
|
||||
214
spec/FederationRequirements.md
Normal file
214
spec/FederationRequirements.md
Normal file
@@ -0,0 +1,214 @@
|
||||
# FederationRequirements — yawex-derived design notes
|
||||
|
||||
Status: **draft for review** · Date: 2026-06-15 · Deliverable of **SHARD-WP-0001**
|
||||
|
||||
Concrete, ADR-ready design decisions for the union/federation layer, distilled from the yawex
|
||||
prior-art review (`research/260608-yawex-prior-art/findings.md`) and made **consistent with**
|
||||
`spec/CoreArchitectureBlueprint.md` (the whole-system architecture) and `INTENT.md`. yawex is
|
||||
*inspiration and a case checklist*, never an interface to inherit (decision 2026-06-08).
|
||||
|
||||
These are **requirements** (what the union must do); `spec/FederationArchitecture.md`
|
||||
(SHARD-WP-0002) is the architecture that realises them. UC references are to
|
||||
`spec/UseCaseCatalog.md`.
|
||||
|
||||
**Resolved precondition.** The yawex "minimal access model in core" thread is **settled**:
|
||||
authorization-in-core / authentication-delegated, the L0–L4 ladder — INTENT amended,
|
||||
`ArchitectureBlueprint.md` ratified. Auth is therefore *not* re-litigated here; these ADRs
|
||||
assume it (the resolver and overlay paths run behind the L5 PEP/PDP).
|
||||
|
||||
---
|
||||
|
||||
## ADR-01 — Cross-shard page resolution
|
||||
|
||||
**Status:** accepted · **CoreArchitectureBlueprint:** §8.4 (union), §7.2 (identity)
|
||||
|
||||
**Context.** yawex's most-tested component was `PageLookUp`, a context-relative resolver with
|
||||
ordered states `LOCAL, CLIMB, DROP, GLOBAL, REMOTE, SWITCH, JUMP, VIRTUAL, FAILED`. Decision
|
||||
(2026-06-08): **inspiration only** — use the states as a *checklist of cases*, design the
|
||||
federation resolver fresh.
|
||||
|
||||
**Decision.** Resolution is a pure function over the derived union:
|
||||
`resolve(name, from_context, policy) → ResolutionResult`, evaluated in a defined order and
|
||||
keyed on **page identity** (the stable handle), not path (I-9):
|
||||
|
||||
1. **Same-namespace, same-shard** exact identity match — *(yawex LOCAL)*.
|
||||
2. **Namespace walk** — climb/descend ancestor namespaces of `from_context` — *(CLIMB/DROP)*.
|
||||
3. **Union lookup** — match identity / alias-table entry across all attached shards — *(GLOBAL)*.
|
||||
4. **Equivalence set** — if several shards hold an equivalent page, return the **chorus set**;
|
||||
the *canonical-source* policy preset (chorus / designated-canonical) decides presentation —
|
||||
never silently pick one (union without erasure, I-4). *(implicit in yawex; explicit here.)*
|
||||
5. **Projection / virtual** — a page whose content lives in a remote shard (lazy replication,
|
||||
*REMOTE*) or is computed/query-defined (derivation, *VIRTUAL*).
|
||||
6. **Explicit address** — a shard- or space-qualified reference resolves directly — *(SWITCH/JUMP)*.
|
||||
7. **Not found** → **red-link**: a resolvable *target* with no page yet, createable (UC-23) —
|
||||
*(FAILED)*.
|
||||
|
||||
`ResolutionResult` carries the resolved identity (or red-link), the placement(s), the
|
||||
provenance envelope (ADR-04), and — on ambiguity — the full chorus set. Resolution is a
|
||||
**derived-tier read** (union graph + equivalence index, §8.4); it is incremental-maintained and
|
||||
respects the L5 authz filter (a principal never resolves to content it can't see).
|
||||
|
||||
**Consequences.** Deterministic, policy-driven, no privileged shard; every yawex state has a
|
||||
home without inheriting its structure. **Open:** cross-space `JUMP` addressing syntax ties to
|
||||
the portable-address scheme (CoreArchitectureBlueprint O-6).
|
||||
|
||||
---
|
||||
|
||||
## ADR-02 — Namespace / path model & shard roles
|
||||
|
||||
**Status:** accepted · **CoreArchitectureBlueprint:** §7.2 (identity≠placement), §4 (shard mode)
|
||||
|
||||
**Context.** yawex modelled **topics as directories** (a topic's gateway page shares the dir
|
||||
name) with relative (`../`) / absolute (`/`) paths and normalization, and page classes
|
||||
`local / global / virtual`.
|
||||
|
||||
**Decision.**
|
||||
- The union exposes a **namespace tree**; a **path is a placement coordinate, not an identity**
|
||||
(I-9). One identity may have **N placements** (paths/shards) → a DAG, no single canonical path.
|
||||
- **Path grammar:** relative (`../`, `./`) resolved against `from_context`; absolute (`/`)
|
||||
against the space root; **normalization** (collapse `.`/`..`; case & separator handling) is a
|
||||
per-space **policy** knob (§10) — defaulting to case-preserving, `/`-separated.
|
||||
- **yawex page classes → shard roles / modes:** `local → canonical`, `global → cross-cutting`
|
||||
(a shard whose pages augment/overlay across namespaces), `virtual → projected/computed`. These
|
||||
are the INTENT shard modes (read-only · write-through · mirrored · projected · cached ·
|
||||
canonical), selected by policy, constrained by the capability profile.
|
||||
- Cross-shard name collisions are resolved by ADR-01's order + equivalence, not by a global
|
||||
flat namespace.
|
||||
|
||||
**Consequences.** The namespace is a navigable view over placements (a dimension of the union),
|
||||
not a storage layout. **Open:** per-space normalization policy defaults (case-sensitivity,
|
||||
unicode) — a policy preset, bundled per persona (O-8).
|
||||
|
||||
---
|
||||
|
||||
## ADR-03 — Union-level derived views: core vs adapter
|
||||
|
||||
**Status:** accepted · **CoreArchitectureBlueprint:** §8.4 (derived query index), §8.7
|
||||
(incremental) · **resolves findings open-Q #3**
|
||||
|
||||
**Context.** yawex shipped BackLinks, RecentChanges, AllPages, SiteMap, full-text Search.
|
||||
|
||||
**Decision.** Each is a **derived-tier** view (rebuildable, incrementally maintained); classify
|
||||
by where it is computed:
|
||||
|
||||
| View | Tier | Rationale |
|
||||
|------|------|-----------|
|
||||
| **BackLinks** | **core** | the link-graph over the union — squarely shard-wiki's concern (UC-18); the strongest core view |
|
||||
| **RecentChanges** | **core** | merge the coordination journal (§8.1) with shard change events (notify/poll, §8.8) across the union (UC-17) |
|
||||
| **AllPages / SiteMap** | **core** | an enumeration/projection of the union graph; cheap |
|
||||
| **Search (full-text)** | **hybrid** | **delegate** to a shard's native search where the native-query axis allows; **else** build a derived index over projections (the Logseq DataScript-over-files pattern, UC-19/UC-63) |
|
||||
|
||||
All are computed in the derived tier and carry provenance; **presentation is L6 (UI)**, never
|
||||
hard-coded in core (mechanism over policy, I-7).
|
||||
|
||||
**Consequences.** BackLinks/RecentChanges/AllPages/SiteMap are core deliverables; Search is a
|
||||
capability-gated delegate-or-derive. **Open:** cross-shard search ranking/relevance is a policy
|
||||
concern, not core mechanism.
|
||||
|
||||
---
|
||||
|
||||
## ADR-04 — Provenance & freshness model (concrete fields)
|
||||
|
||||
**Status:** accepted · **CoreArchitectureBlueprint:** §7.3 (layered provenance), §8.8 (freshness)
|
||||
|
||||
**Context.** yawex `Page::info` exposed modtime (and TODO'd last-editor / hits / edits). INTENT
|
||||
mandates explicit provenance & freshness (I-4).
|
||||
|
||||
**Decision.** Concretise the **provenance envelope** (layered: page-level + span-level deltas):
|
||||
|
||||
```
|
||||
ProvenanceEnvelope (page-level):
|
||||
source_shard # which shard this came from
|
||||
source_rev? # shard-native revision id, if the shard exposes one
|
||||
observed_at # when shard-wiki last read it
|
||||
liveness # static | captured-snapshot | live-over-files | view-time | irreducibly-live
|
||||
staleness_state # live | fresh | stale | unavailable
|
||||
authz_context # the L5 context under which it was read (no-leak, §9)
|
||||
overlay_state # none | draft | patch-pending | applied
|
||||
divergence[] # equivalence-set peers that differ (the chorus, ADR-01 step 4)
|
||||
derivation_lineage? # for derived/derivation-projection content (source → this view)
|
||||
Span-level: only the fields that differ from the page envelope (effective = page ⊕ delta).
|
||||
```
|
||||
|
||||
`freshness = (observed_at, source_rev?, staleness_state)`; `unavailable` is the dead-shard
|
||||
state (feeds ADR-03 RecentChanges and the union-under-unavailability open item O-11). This
|
||||
envelope is the data behind union-without-erasure and the input to conflict display.
|
||||
|
||||
**Consequences.** Provenance is queryable, layered (cheap per-span), and drives views/conflict
|
||||
UI. **Open / DROP:** hits/edits *analytics* are an L6/analytics concern, out of core (dropped
|
||||
from the yawex feature set).
|
||||
|
||||
---
|
||||
|
||||
## ADR-05 — Overlay / lightweight-patch model
|
||||
|
||||
**Status:** accepted · **CoreArchitectureBlueprint:** §8.2 (overlay engine), §8.1 (decision
|
||||
log), §8.6 (apply-under-drift)
|
||||
|
||||
**Context.** yawex had an append + threaded-comment workflow (edit a page without a full
|
||||
rewrite). INTENT mandates overlay-before-mutation (I-5).
|
||||
|
||||
**Decision.** The overlay lifecycle for any shard below write-through capability:
|
||||
|
||||
1. **Draft** — an edit becomes a draft recorded as a **coordination-canonical event** in the
|
||||
space's decision log (§8.1); the unapplied overlay is the **local truth**, fully attributed
|
||||
(`overlay_state`, ADR-04).
|
||||
2. **Patch / MR** — rendered in the shard's **native syntax (lossless)** or **Markdown
|
||||
(lossy-with-fidelity-report)** per the translation axis.
|
||||
3. **Apply** — only on explicit intent, only where profile + policy permit, with
|
||||
**apply-under-drift** semantics (§8.6: fast-forward / three-way / refuse-and-re-present).
|
||||
4. **Append / comment** = a **constrained overlay subtype** (purely additive, low-conflict) —
|
||||
the direct generalisation of yawex's workflow; safe even on the most limited shards.
|
||||
|
||||
This makes read-only, rate-limited, and lossy shards **editable** without silent remote
|
||||
mutation (graceful degradation, I-8; no silent mutation).
|
||||
|
||||
**Consequences.** One overlay mechanism spans drafts, patches, MRs, and comments; storage is the
|
||||
decision log (no separate store). **Open:** whether threaded-comment threading is a first-class
|
||||
overlay subtype or a generic structured-append — defer to implementation.
|
||||
|
||||
---
|
||||
|
||||
## ADR-06 — Markdown link semantics (wikilink + red-link)
|
||||
|
||||
**Status:** accepted · **CoreArchitectureBlueprint:** §7 (page model), ADR-01 (resolution) ·
|
||||
**resolves findings open-Q #2**
|
||||
|
||||
**Context.** yawex had CamelCase WikiLinks, `[[free links]]`, red-`?` links for nonexistent
|
||||
pages, `::` labels. INTENT mandates Markdown-first; TRANSFORM the *semantics*, drop the bespoke
|
||||
syntax.
|
||||
|
||||
**Decision.**
|
||||
- Adopt a **CommonMark wikilink extension**: `[[Target]]` / `[[Target|label]]`, resolved via
|
||||
ADR-01 (by identity/name across the union).
|
||||
- **Red-link** = a wikilink whose target resolves to FAILED (ADR-01 step 7): a valid,
|
||||
*createable* target with no page yet (the soft-create affordance, UC-23).
|
||||
- **CamelCase auto-linking is OFF by default** (a legacy affordance); opt-in **per space** for
|
||||
migrating CamelCase wikis (UseModWiki/c2 lineage, UC-25).
|
||||
- Links are stored **as text** (git-diffable; structure/links federate iff in-text).
|
||||
- **Boundary (resolves open-Q #2):** the link **model + resolution is core**; the **visual
|
||||
rendering** of wikilinks/red-links is a **reference-UI (L6)** concern. Core decides *what a
|
||||
link resolves to and whether it's a red-link*; the UI decides how it looks.
|
||||
|
||||
**Consequences.** Markdown-first, backend-neutral, resolution-unified with ADR-01. **Open:**
|
||||
cross-shard wikilink target disambiguation syntax ties to the portable-address scheme (O-6).
|
||||
|
||||
---
|
||||
|
||||
## Coverage & open questions
|
||||
|
||||
- **Findings open-Q #1** (per-page vs per-shard ACL) — answered by the settled access model:
|
||||
per-shard / per-namespace default, per-page ACL opt-in at L4 (`ArchitectureBlueprint.md`).
|
||||
- **Findings open-Q #2** (wikilink core vs UI) — ADR-06: model/resolution core, presentation UI.
|
||||
- **Findings open-Q #3** (derived views core vs adapter) — ADR-03.
|
||||
- Carried to CoreArchitectureBlueprint §12: cross-space addressing syntax (O-6), namespace
|
||||
normalization presets (O-8), cross-shard search ranking (policy), union-under-unavailability
|
||||
(O-11).
|
||||
|
||||
## Acceptance (SHARD-WP-0001)
|
||||
|
||||
Each task T1–T6 has an ADR (T1→ADR-01, T2→ADR-02, T3→ADR-03, T4→ADR-04, T5→ADR-05, T6→ADR-06);
|
||||
all honour INTENT (mechanism over policy, union without erasure, overlay before mutation,
|
||||
capability-aware adapters) and are consistent with `CoreArchitectureBlueprint.md`. The
|
||||
access-model thread is ratified (not deferred). The next implementation workplan (domain model /
|
||||
adapter contract) can proceed without unresolved yawex-derived design gaps.
|
||||
@@ -6,10 +6,15 @@ Background on document types: InfoTechPrimers on coulomb.social.
|
||||
|
||||
| File | Status | Role |
|
||||
|------|--------|------|
|
||||
| `CoreArchitectureBlueprint.md` | draft for review | **Whole-system architecture** — layers, abstractions, load-bearing decisions (synthesised from all research) |
|
||||
| `FederationArchitecture.md` | draft for review | federation design — *what the union does*: T1–T10 decision records + the federation-model taxonomy (SHARD-WP-0002) |
|
||||
| `WikiEngineCoreArchitecture.md` | draft for review | the native **headless, API-first wiki engine** — small page-store kernel + typed-extension framework, as a canonical-mode shard backend (SHARD-WP-0013) |
|
||||
| `adr/` | living | Architecture Decision Records (ADR-0001: engine activation via feature-control) |
|
||||
| `FederationRequirements.md` | draft for review | yawex-derived union/federation design notes — resolution, namespace, derived views, provenance, overlay, links (ADR-01…06; SHARD-WP-0001) |
|
||||
| `ProductRequirementsDocument.md` | draft scaffold | What the product must deliver |
|
||||
| `TechnicalSpecificationDocument.md` | draft scaffold | How the system is built |
|
||||
| `UseCaseCatalog.md` | draft | 25 use cases promoted from c2 + yawex research |
|
||||
| `ArchitectureBlueprint.md` | draft | Access, history, and identity architecture |
|
||||
| `TechnicalSpecificationDocument.md` | draft + §A | How the system is built; **§A = the normative shard adapter contract** (T11–T16, T18; SHARD-WP-0002) |
|
||||
| `UseCaseCatalog.md` | draft | 84 use cases promoted from c2 + yawex + ~23-system research |
|
||||
| `ArchitectureBlueprint.md` | draft | Access, history, and identity sub-blueprint (the L0–L4 authorization ladder; referenced by CoreArchitectureBlueprint §9) |
|
||||
|
||||
Promote material from `research/` and reviewed items from `demand/` into spec
|
||||
before treating it as implementation authority.
|
||||
@@ -33,7 +33,134 @@ per information space.
|
||||
|
||||
## 4. Architecture References
|
||||
|
||||
- `spec/ArchitectureBlueprint.md` — access, history, identity delegation
|
||||
- `spec/CoreArchitectureBlueprint.md` — whole-system architecture (layers, the dual narrow
|
||||
waist, the 15 capability spectra, projection, consistency); the normative source for §A.
|
||||
- `spec/FederationArchitecture.md` — federation design (*what the union does*); §A is its
|
||||
companion (*what a backend must expose*).
|
||||
- `spec/FederationRequirements.md` — yawex-derived ADRs (resolution, page model, overlay).
|
||||
- `spec/ArchitectureBlueprint.md` — access, history, identity delegation (L5).
|
||||
|
||||
---
|
||||
|
||||
# A. Shard Adapter Contract (normative)
|
||||
|
||||
Deliverable of **SHARD-WP-0002** (T11–T16, T18): the versioned interface a backend implements
|
||||
to participate as a shard, and the *normative* rules core relies on. It is the **bottom narrow
|
||||
waist** (blueprint §6); the page model is the top waist (§7). This section is normative where it
|
||||
says **MUST/SHOULD**; design rationale lives in the blueprint (cited, not restated).
|
||||
|
||||
## A.1 (T11) Capability model & versioned interface
|
||||
|
||||
- The contract is a **versioned interface** (`Foswiki::Store`/`Foswiki::Meta` is the proof a
|
||||
swappable-backend-behind-a-stable-interface works). A binding declares the contract version
|
||||
it implements; core checks compatibility at registration.
|
||||
- **Operation verbs:** `read, write, diff, merge, lock, version, publish, notify,
|
||||
transclude-source, translate-syntax, structured-payload, derive-projection, execute`. The
|
||||
last two are **gated, OFF by default** (A.6/T18). Verb support is part of the profile and MUST
|
||||
reconcile with the T10 federation-ops matrix.
|
||||
- **Capability profile = a position on each capability spectrum**, not a boolean checklist. The
|
||||
full set is the **fifteen spectra** (blueprint §6.1); operationally they reduce to a **small
|
||||
orthogonal core** (substrate, write-granularity, content-opacity, operational-envelope,
|
||||
access-grant, computational/liveness) with the rest **implied** via published rules that
|
||||
**forbid impossible profiles** (blueprint §6.5). Degradation reads only the **named
|
||||
axis-interaction subset** (blueprint §6.5 table) — that table *is* the degradation contract.
|
||||
- A profile MUST express **absence** cleanly (the Oddmuse floor); core never guesses a missing
|
||||
capability.
|
||||
|
||||
## A.2 (T11/§6.6) Conformance — profiles are verified, not asserted
|
||||
|
||||
- The contract ships a **versioned conformance suite**. A binding is **admissible only if it
|
||||
passes**: the suite exercises each declared verb and spectrum position and checks observed
|
||||
behaviour matches the claim (a `write` round-trips, `notify` actually fires, an opaque shard
|
||||
refuses plaintext query, implied-position rules hold). A lying/buggy profile is **rejected at
|
||||
registration**, not run in production (blueprint §6.6).
|
||||
- Conformance makes capability-as-data (I-3) and the §6.5 degradation contract **sound**.
|
||||
Mismatch is reported as a capability-by-capability diff; an adapter may register
|
||||
*degraded-but-honest* (drop the unsupported claim).
|
||||
|
||||
## A.3 (T14) Adapter binding — attachment-mode taxonomy
|
||||
|
||||
A backend MAY offer several modes; attach mode is a **per-binding, capability-gated choice**
|
||||
with one declared authoritative (blueprint §6.3):
|
||||
|
||||
- **file-store** (native vault/folder *or* an interchange/sync mirror) · **git-IS-store** (forge
|
||||
wikis & ikiwiki — git is store *and* journal; the home case) · **in-engine-hosted** (XWiki
|
||||
component, Obsidian/Logseq/Roam plugin, Trilium script) · **local-REST** (Joplin Data API,
|
||||
Trilium ETAPI) · **external-API-only** (Notion) · **direct-DB** (MojoMojo schema→model) ·
|
||||
**CRDT-replica** (Anytype/AFFiNE/AppFlowy) · **P2P / no-central-endpoint**.
|
||||
- **Backend-swap tolerance:** identity/provenance MUST survive a substrate change (bind to
|
||||
capabilities, not "it's files"). **Boundary:** an **image / live-memory blob is never an
|
||||
attach target** (I-12) — participation is export→files only.
|
||||
- (UC-38, UC-40, UC-43, UC-50, UC-53, UC-57, UC-60, UC-62, UC-64, UC-65, UC-76, UC-79, UC-81.)
|
||||
|
||||
## A.4 (T12) Page model — structured / computational payload
|
||||
|
||||
The backend-neutral, Markdown-first page model MUST carry, without lossy flattening, every
|
||||
shape in blueprint §7.1: prose; typed/computed records (incl. effective-vs-own computed
|
||||
metadata); typed-graph statements (Wikibase); inline-embedded objects (Quip/Notion);
|
||||
non-Markdown assets (typed asset / opaque blob / content-type registry); and the **four
|
||||
computational shapes** (one-source-many-projections UC-83; notebook-with-embedded-output UC-84;
|
||||
program-as-page; live/temporal). All reduce to `(content|source, structure, provenance
|
||||
envelope, [derivation rule])`. **Identity is a stable handle; placement is separate; equivalence
|
||||
is fingerprint-based** (blueprint §7.2, FederationRequirements ADR-01/02, I-9). The **provenance
|
||||
envelope is layered** (page + span deltas; effective = page ⊕ delta, §7.3). (UC-34, UC-39,
|
||||
UC-55, UC-58, UC-54, UC-44, UC-66, UC-67, UC-73, UC-83, UC-84.)
|
||||
|
||||
## A.5 (T13) History portability — adopt / supplement / import
|
||||
|
||||
Per the history axis: **adopt** git-native history as-is; **supplement** non-portable internal
|
||||
history (DB / Notion / Joplin / Trilium revisions, CRDT-log) — the journal begins-now / mirrors
|
||||
/ snapshots; **import** open file history (RCS, PlainFile, MojoMojo DB-version-rows) backfilled
|
||||
preserving author/timestamp. **Partial/truncated history MUST be reported honestly** ("history
|
||||
begins at", UC-24), never implied complete. Embedded-output documents (notebooks) use
|
||||
paired-text (Jupytext) / cell-aware merge (nbdime). The space's own coordination history is the
|
||||
event-sourced decision log (blueprint §8.1). (UC-36, UC-41, UC-24.)
|
||||
|
||||
## A.6 (T15) Syntax translation & content fidelity
|
||||
|
||||
Translation is a spectrum: **native → lossless → lossy-with-fidelity-report**. Read native
|
||||
markup (TML, XWiki syntax, HTML) into the page model; accept Markdown overlays back via
|
||||
**lossless bidirectional translation** where possible (Foswiki WysiwygPlugin is the proof). For
|
||||
non-round-tripping models (Notion blocks, Trilium HTML, CRDT, typed-graph, notebook JSON/MIME):
|
||||
translate lossily but **make the fidelity loss visible** — a per-shard/per-page report of what
|
||||
projects cleanly vs degrades, with non-mappable elements preserved as provenance/sidecar (I-4).
|
||||
Add a **structured-re-evaluable-value** point to the opacity spectrum (Wolfram expression).
|
||||
Where no acceptable translation exists, the shard is a **read-only/projection** participant
|
||||
(I-8), never silently corrupted. (UC-42, UC-59, UC-03, UC-73.)
|
||||
|
||||
## A.7 (T16) Addressing, identity & navigation
|
||||
|
||||
- **Span addressing:** adopt native span IDs where minted (Roam `:block/uid`, Logseq `id::`,
|
||||
Notion/CRDT UUID), shard-scoped so they survive projection and don't collide; else a
|
||||
position address (path+range) or content-fingerprint address. Portable tumbler is the ideal
|
||||
(blueprint O-6).
|
||||
- **Transclusion** = one reference-not-copy primitive over the addressable union (FederationArch
|
||||
T8). **Query/navigation:** delegate to a shard's native query where capable (Datalog, DB
|
||||
query, XWQL, SPARQL), **else build a derived index over the projection** (Logseq pattern);
|
||||
dimensional/query-defined views are derived-tier. (UC-44–48, UC-51, UC-52, UC-54, UC-63,
|
||||
UC-74.)
|
||||
|
||||
## A.8 (T18) Computational / executable content capability
|
||||
|
||||
**In scope as a page-model + projection concern; out of scope as an execution platform**
|
||||
(blueprint §8.5). Core **recognises** computational types, attaches the **canonical source**,
|
||||
and presents derived forms as **provenance- and liveness-marked projections**. `derive-
|
||||
projection`/`execute` are **gated capabilities, OFF by default**, carrying a **trust/sandbox**
|
||||
concern, **degrading to a captured snapshot / static render / recording**. One snapshot-
|
||||
provenance record (run id, source rev, timestamp, environment "unguaranteed") serves notebooks,
|
||||
renders, recordings. No INTENT amendment required. (UC-54, UC-55, UC-83, UC-84.)
|
||||
|
||||
## A.9 Conformance & module mapping
|
||||
|
||||
The contract maps to `src/shard_wiki/adapters/` (the bottom waist: `AdapterContract`,
|
||||
`CapabilityProfile`, attachment-mode binding, the conformance suite) consuming `model/`
|
||||
(page model + capability value types) and `provenance/` (blueprint §11). Each concrete adapter
|
||||
ships its declared profile + a conformance pass.
|
||||
|
||||
*Decided:* A.1–A.8 (versioned capability contract, verified conformance, binding taxonomy,
|
||||
page model, history portability, translation, addressing, gated computational content).
|
||||
*Deferred:* per-backend adapter specs (one per backend, later). *Open:* blueprint §12 O-items
|
||||
(addressing O-6, axis interactions O-5, span-authz O-10).
|
||||
|
||||
## 5. Integration Boundaries
|
||||
|
||||
@@ -47,10 +174,13 @@ Package scaffold only (`__version__`, smoke tests). Domain model not yet coded.
|
||||
|
||||
## 7. Use Cases
|
||||
|
||||
`spec/UseCaseCatalog.md` — 25 use cases (UC-01–UC-25) promoted from c2 wiki
|
||||
origins and yawex prior-art research.
|
||||
`spec/UseCaseCatalog.md` — 84 use cases (UC-01–UC-84) from c2/yawex origins,
|
||||
the wiki-engine + modern-tool + computational research, and the syntheses.
|
||||
|
||||
## 8. Next Specification Work
|
||||
|
||||
Outputs from `SHARD-WP-0001` tasks (page resolution, namespaces, derived views,
|
||||
provenance, overlays, link semantics) will be incorporated here as they complete.
|
||||
The design layer is complete: `SHARD-WP-0001` (→ `FederationRequirements.md`),
|
||||
`SHARD-WP-0002` (→ `FederationArchitecture.md` + §A above), and the hardened
|
||||
`CoreArchitectureBlueprint.md`. The next workplan is the **implementation** of the
|
||||
domain model + adapter contract (starting from §A and blueprint §11's module layout),
|
||||
likely with a first spike on the keystone (the event-sourced decision log + derived fold).
|
||||
File diff suppressed because it is too large
Load Diff
269
spec/WikiEngineCoreArchitecture.md
Normal file
269
spec/WikiEngineCoreArchitecture.md
Normal file
@@ -0,0 +1,269 @@
|
||||
# WikiEngineCoreArchitecture
|
||||
|
||||
Status: **draft for review** · Date: 2026-06-15 · Deliverable of **SHARD-WP-0013 T5**
|
||||
|
||||
The architecture of shard-wiki's **native reference wiki-engine**: a **headless, API-first**
|
||||
engine — a **small core** plus a **stringent typed-extension framework** — that addresses the
|
||||
whole use-case catalogue, mediates conflicting requirements into one integrated featureset, and
|
||||
lets each shard **activate only what it needs**. Authoritative as of the ratified INTENT
|
||||
amendment (2026-06-15, decision `84ffdb48`): the engine is **additive** and is shard-wiki's
|
||||
**reference first-party shard backend (a canonical-mode shard)** — not a replacement for other
|
||||
engines, not a UI.
|
||||
|
||||
Relation to other specs (referenced, not restated):
|
||||
- `CoreArchitectureBlueprint.md` — the orchestrator/whole-system architecture. **The engine is
|
||||
one shard behind §A; federation, union, projection, and cross-shard coordination are the
|
||||
orchestrator's job, not the engine's.** That is what keeps the engine small.
|
||||
- `TechnicalSpecificationDocument.md §A` — the shard adapter contract the engine implements.
|
||||
- `FederationRequirements.md` — page resolution, overlay, link semantics (ADRs the engine reuses).
|
||||
- `UseCaseCatalog.md` "Capability structure" layer (T2) — the core-vs-extension map + the
|
||||
conflict-mediation map this document realizes.
|
||||
- reuse surface (`capability.wiki.*`, plus consumed `feature-control` / `authorization`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Thesis: a small page-store kernel; everything else is a typed extension
|
||||
|
||||
> **The engine is a page-store kernel with a typed-extension runtime. Every capability beyond
|
||||
> the c2-minimum is a *typed extension* a shard activates only if it needs it — and a shard's
|
||||
> externally-visible capability profile is *computed from its active extension set*.**
|
||||
|
||||
That single chain — **configuration (which extensions) → capability (what the shard can do) →
|
||||
conformance (verified)** — is the whole design. It mirrors the orchestrator's discipline
|
||||
(`CoreArchitectureBlueprint` §6.5: capability-as-data, verified, no per-backend code) and turns
|
||||
"integrated whole, yet activate only what you need" from a slogan into a mechanism.
|
||||
|
||||
The engine stays small for a structural reason: it is **one shard**, not a federation layer.
|
||||
Union, projection, equivalence, cross-shard overlay-orchestration, and the federation models all
|
||||
live in shard-wiki's orchestrator (the blueprint). The engine implements `ShardAdapter` (§A) and
|
||||
nothing above it. So "wiki engine" here means *a really good single canonical shard with a
|
||||
typed-extension framework and a headless agent-first API* — not a re-implementation of shard-wiki.
|
||||
|
||||
---
|
||||
|
||||
## 2. Engine invariants
|
||||
|
||||
| # | Invariant | Why |
|
||||
|---|-----------|-----|
|
||||
| E-1 | **One shard, not a federation layer.** The engine implements `ShardAdapter` (§A); union/projection/federation are the orchestrator's. | Keeps the engine small; no duplication of the blueprint. |
|
||||
| E-2 | **Small kernel.** The kernel is only: page store + history, the page model (reused), the extension runtime, the API. | Common case (a plain wiki) is trivial. |
|
||||
| E-3 | **Everything else is a typed extension.** No feature beyond the c2-minimum is baked into the kernel. | Integrated-whole-yet-selective; testable boundary. |
|
||||
| E-4 | **Per-shard activation.** A shard runs an *activation profile* (a set of extensions + config); unused features cost nothing. | "Activate only what you need." |
|
||||
| E-5 | **Capability profile is derived from active extensions.** The §A profile the engine declares is computed from its activation profile, then conformance-verified. | One source of truth; honest, verified capabilities. |
|
||||
| E-6 | **Headless & API-first.** The API is the only interface; no bundled UI/rendering (consumer concern, L6). | INTENT amendment; clean orchestrator/consumer split. |
|
||||
| E-7 | **Agent-first ergonomics.** The API is typed, introspectable, batchable, low-round-trip. | INTENT: optimized for efficient agent/automation access. |
|
||||
| E-8 | **Reuse over reinvent.** Page model, history/journal, activation, and authz are *consumed* (existing capabilities), not rebuilt. | Smallness; reuse-surface alignment. |
|
||||
| E-9 | **Extensions are typed & verified.** An extension declares its types/hooks/deps; activation is rejected if types conflict or deps are unmet (impossible profiles forbidden). | Stringency; mirrors §6.5 + conformance. |
|
||||
|
||||
---
|
||||
|
||||
## 3. The kernel (four concepts)
|
||||
|
||||
The kernel is deliberately four things — nothing more is mandatory.
|
||||
|
||||
1. **Page** — the backend-neutral page model (`capability.wiki.page-model`, reused as-is):
|
||||
stable identity ≠ placement, layered provenance, page shapes. The kernel does **not** redefine
|
||||
it; extensions may *register additional shapes/types* (§4).
|
||||
2. **Store + history** — a git-backed page store (the engine is the *git-IS-store* case from the
|
||||
blueprint): a write is a commit; history is native and recoverable (E-3/I-10). Coordination
|
||||
decisions reuse the event-sourced journal (`capability.wiki.coordination-journal`).
|
||||
3. **Extension runtime** — the typed-extension registry, hook dispatcher, type checker, and
|
||||
activation engine (§4). *This is the core innovation; it is the only “framework” in the kernel.*
|
||||
4. **API** — the headless, typed, agent-first surface (§7). Kernel endpoints cover the c2-minimum
|
||||
(page CRUD-as-history, links, history); extensions extend the surface through typed routes.
|
||||
|
||||
The **c2-minimum** a kernel-only shard delivers (no extensions): write a page, link pages
|
||||
(`[[wikilink]]` + red-link), never lose an edit. That is a complete, useful headless wiki.
|
||||
|
||||
---
|
||||
|
||||
## 4. The typed-extension model (the framework)
|
||||
|
||||
An **Extension** is a typed unit declaring a contract the runtime enforces:
|
||||
|
||||
```
|
||||
Extension:
|
||||
id : reverse-domain id (e.g. ext.struct.typed-records)
|
||||
provides : capability ids it realizes (reuse-surface; e.g. capability.wiki.page-model[typed])
|
||||
types : page shapes / field schemas / content-types it introduces (typed, validated)
|
||||
hooks : kernel lifecycle bindings it implements (see below)
|
||||
api : typed routes it adds to the headless surface
|
||||
depends_on : other extensions / consumed capabilities required
|
||||
conflicts_with: extensions it cannot co-activate with
|
||||
config : declared, schema-checked activation parameters
|
||||
```
|
||||
|
||||
**Hooks (the kernel lifecycle the runtime dispatches):**
|
||||
`on_resolve` (name→page), `on_read`, `on_write` (validate/transform a draft), `on_link`
|
||||
(link/transclusion resolution), `on_history`, `on_query`, `on_render_request` (produce a derived
|
||||
representation for a consumer), `on_profile` (contribute capability-spectrum positions, E-5).
|
||||
Hooks are **typed** (typed inputs/outputs) and dispatched in a **declared, deterministic order**.
|
||||
|
||||
**Typing & composition (stringency):**
|
||||
- At activation, the runtime builds the **dependency closure**, checks **type consistency** (no
|
||||
two active extensions claim incompatible types for the same page shape/field; `conflicts_with`
|
||||
honoured), and rejects an **impossible profile** — exactly the §6.5 implication-rule discipline,
|
||||
applied to extensions. A rejected profile fails fast at boot, never silently.
|
||||
- Composition is **deterministic**: hook order is declared; conflicts are resolved by explicit
|
||||
precedence or rejection, never by accident.
|
||||
- Extensions ship a **conformance check** (mirrors §6.6): an activated extension is exercised
|
||||
against its declared types/hooks before the shard serves traffic — *typed contracts verified,
|
||||
not trusted*.
|
||||
|
||||
**Per-shard activation (reuse, not reinvent):**
|
||||
- A shard's **activation profile** = `{extension id → config}`. Activation/evaluation **reuses
|
||||
`capability.feature-control.evaluate`** (helix_forge/feature-control) — shard-wiki does not
|
||||
build a bespoke flagging system (T3 consumption).
|
||||
- **E-5 in action:** the engine's `on_profile` hooks fold the active extensions into the §A
|
||||
**capability profile** the shard advertises to the orchestrator (e.g. activate
|
||||
`ext.struct.typed-records` → the `structure` spectrum rises and `structured-payload` is
|
||||
declared). The profile is then conformance-verified (§A.2). *Configuration → capability →
|
||||
conformance is one chain.*
|
||||
|
||||
---
|
||||
|
||||
## 5. Featureset map: core vs extensions, and conflict mediation
|
||||
|
||||
The engine realizes the T2 "Capability structure" layer (`UseCaseCatalog.md`). Mapping (the
|
||||
*page/content-level* clusters; **X-FED and X-ATT are orchestrator concerns, not engine
|
||||
extensions** — E-1):
|
||||
|
||||
| Engine kernel (always on) | T2 | reuse-surface |
|
||||
|---------------------------|----|---------------|
|
||||
| Page lifecycle, identity/placement, history, links, store | EC-1…EC-5 | `capability.wiki.page-model`, `…coordination-journal`, `…adapter-contract` |
|
||||
|
||||
| Built-in typed extension | T2 cluster | provides / consumes | default |
|
||||
|--------------------------|-----------|---------------------|---------|
|
||||
| `ext.overlay` | X-OVERLAY | `capability.wiki.overlay` | on (no-op locally) |
|
||||
| `ext.authz` (L0→L4 tiers) | X-AUTHZ | consumes `capability.authorization.policy-evaluate` | L0 |
|
||||
| `ext.views` (BackLinks/RecentChanges/…) | X-VIEW | `capability.wiki.derived-views` | BackLinks/RecentChanges on |
|
||||
| `ext.struct` (typed/computed/graph) | X-STRUCT | `capability.wiki.page-model[typed]` | off |
|
||||
| `ext.addr` (span addr / transclusion / query) | X-ADDR | `capability.wiki.page-model`+query | transclusion on |
|
||||
| `ext.compute` (literate/notebook/program/live) | X-COMP | `capability.wiki.engine-typed-extensions` | off (gated, sandbox) |
|
||||
| `ext.prov` (rich provenance/metadata) | X-PROV | `capability.wiki.page-model[provenance]` | base on |
|
||||
| `ext.collab` (c2 social patterns) | X-COLLAB | (UI/convention; mostly consumer) | off |
|
||||
|
||||
**Conflict mediation (T2 map) realized by the framework** — every tension is a *mechanism*, not a
|
||||
baked-in choice, so one featureset serves all:
|
||||
|
||||
| Tension | Realized by |
|
||||
|---------|-------------|
|
||||
| open vs governed | `ext.authz` tiers (additive); kernel history is the floor at L0 |
|
||||
| lossless vs lossy | a `translate` hook + fidelity report (consumes the proposed `capability.content.translation-fidelity`, G2) |
|
||||
| live vs snapshot | `ext.compute`/`ext.addr` mark liveness; degrade to snapshot (never imply live) |
|
||||
| canonical vs chorus | detection in kernel; resolution is a policy preset (orchestrator) |
|
||||
| integrated-whole vs only-what-you-need | **the activation profile** (E-4) + typed composition (§4) — the headline mediation |
|
||||
| minimal vs feature-rich | small kernel (§3) + extensions; nothing beyond c2 is mandatory |
|
||||
|
||||
---
|
||||
|
||||
## 6. The engine as a canonical-mode shard
|
||||
|
||||
The engine exposes itself through an `EngineShardAdapter` implementing §A:
|
||||
|
||||
- **Substrate** git-IS-store; **history** git-native; **write** = commit; `current_rev` = sha
|
||||
(apply-under-drift works out of the box). It is the **most capable shard** shard-wiki can
|
||||
attach — it dogfoods the contract.
|
||||
- Its **capability profile is computed from active extensions** (E-5) and **conformance-verified**
|
||||
(§A.2) — so the orchestrator sees an honest profile, and federation ops degrade by the engine's
|
||||
*actually-activated* capabilities.
|
||||
- The orchestrator attaches it like any shard; **federation/union/projection are not in the
|
||||
engine** (E-1). A standalone deployment is "the engine as the sole canonical shard"; a
|
||||
federated deployment is "the engine as one shard among many." Same engine, no re-architecture.
|
||||
|
||||
This is the precise realization of the INTENT reconciliation: shard-wiki orchestrates; the engine
|
||||
is the first-party shard it can attach.
|
||||
|
||||
---
|
||||
|
||||
## 7. Headless API surface & agent ergonomics (E-6/E-7)
|
||||
|
||||
API-first means the typed API is the product; there is no UI. Agent-first means it is designed
|
||||
for cheap, deterministic machine consumption:
|
||||
|
||||
- **Typed resource API** over pages, links, history, spans — content-negotiated (raw Markdown,
|
||||
the structured page model, or an extension-rendered representation via `on_render_request`).
|
||||
- **Capability/extension introspection** — an endpoint returns the shard's **active extensions,
|
||||
their types, and the derived §A capability profile**, so an agent can discover *what this shard
|
||||
can do* before acting (no trial-and-error). This is the agent-facing twin of E-5.
|
||||
- **Batch & query** — multi-page reads, link-graph and RecentChanges queries (via `ext.views`),
|
||||
and `on_query` delegation — minimizing round-trips.
|
||||
- **Write via overlay** — edits go through the overlay path (FederationRequirements ADR-05), so
|
||||
agent writes are safe (draft → apply-under-drift) and attributable.
|
||||
- **Deterministic & provenance-carrying** — every response carries the provenance envelope;
|
||||
identical inputs yield identical outputs (no hidden state) — friendly to caching agents.
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation sketch (module layout)
|
||||
|
||||
The engine lives under the shard-wiki package as a backend (it sits at L0/L1 — a shard behind the
|
||||
adapter; nothing in the orchestrator depends *up* on it):
|
||||
|
||||
```
|
||||
src/shard_wiki/engine/
|
||||
kernel.py # page store + history (git-IS-store), lifecycle; reuses model/, provenance/, coordination/
|
||||
extension.py # Extension contract, registry, typed hook dispatcher, type checker
|
||||
activation.py # activation profile; reuses capability.feature-control.evaluate
|
||||
profile.py # derive the §A CapabilityProfile from active extensions (E-5) + conformance
|
||||
api.py # headless, typed, agent-first surface (+ extension introspection)
|
||||
adapter.py # EngineShardAdapter implements adapters/ ShardAdapter (canonical-mode shard)
|
||||
extensions/ # built-ins: overlay/ authz/ views/ struct/ addr/ compute/ prov/ collab/
|
||||
```
|
||||
|
||||
Dependency rule: `engine/` consumes `model/`, `provenance/`, `coordination/`, `adapters/`
|
||||
(contract), `policy/`; it is consumed *only* via its `EngineShardAdapter` (the orchestrator
|
||||
attaches it as a shard). No orchestrator-tier (`union/`, `projection/`) import.
|
||||
|
||||
---
|
||||
|
||||
## 9. Reuse (what the engine consumes vs registers)
|
||||
|
||||
- **Consumes:** `capability.feature-control.evaluate` (activation), `capability.authorization.
|
||||
policy-evaluate` (`ext.authz`), the proposed `capability.content.translation-fidelity` (G2,
|
||||
lossy translation), and shard-wiki's own `capability.wiki.{page-model, coordination-journal,
|
||||
adapter-contract, overlay, derived-views}`.
|
||||
- **Registers / realizes:** `capability.wiki.engine-typed-extensions` (this document is its
|
||||
Discovery evidence — D2→D3 on ratification). The cross-cutting **typed-extension framework**
|
||||
pattern is proposed back to the reuse surface as **G1** (`capability.platform.typed-extension-
|
||||
framework`); this engine is its first instance.
|
||||
|
||||
---
|
||||
|
||||
## 10. Traceability
|
||||
|
||||
- **INTENT** — realizes the 2026-06-15 amendment (decision `84ffdb48`): headless, API-first,
|
||||
additive native engine = canonical-mode shard backend; honours all engine invariants and the
|
||||
orchestrator boundary (E-1).
|
||||
- **Use cases** — the kernel/extension split *is* the T2 "Capability structure" layer
|
||||
(`UseCaseCatalog.md`); every UC is either kernel (EC-1…EC-5) or a named extension; conflicts
|
||||
use the T2 mediation map (§5). The engine must ultimately cover UC-01–UC-84 (per-shard subsets).
|
||||
- **Architecture** — consistent with `CoreArchitectureBlueprint` (engine = canonical-mode shard,
|
||||
§6 contract, §7 page model, §8.1 journal) and `TechnicalSpecificationDocument §A` (the contract
|
||||
it implements). `FederationRequirements` ADR-05/06 supply overlay + link semantics.
|
||||
- **Reuse surface** — §9; G1/G2 proposals from SHARD-WP-0013 T3.
|
||||
|
||||
## 11. Decisions / deferred / open
|
||||
|
||||
**Decided:** small page-store kernel + typed-extension runtime (E-2/E-3); engine is one shard,
|
||||
not a federation layer (E-1); capability profile derived from active extensions (E-5); headless,
|
||||
API-first, agent-first (E-6/E-7); activation reuses `feature-control` (E-8); extensions are
|
||||
typed + conformance-verified (E-9).
|
||||
|
||||
**Deferred:** the concrete extension SDK/ABI and hook signatures; the API protocol (REST/GraphQL/
|
||||
MCP) — agent-first introspection is required, the wire format is an implementation spike; the
|
||||
built-in extensions' internal designs (each is a later workplan).
|
||||
|
||||
**Open (tracked):** does `ext.compute` ever execute in-process or strictly delegate/snapshot
|
||||
(ties blueprint §8.5 + trust/sandbox); is the typed-extension framework promoted to the
|
||||
reuse-surface platform capability (G1) and then *consumed* here rather than engine-owned;
|
||||
introspection granularity vs. leaking internal structure to agents.
|
||||
|
||||
## 12. Stability note
|
||||
|
||||
The **thesis (§1)** and **invariants (§2)** — especially *engine-is-one-shard* (E-1),
|
||||
*small-kernel/everything-else-typed-extension* (E-2/E-3), and *capability-profile-derived-from-
|
||||
extensions* (E-5) — are load-bearing. Changing them (e.g. moving federation into the engine, or
|
||||
baking a feature into the kernel) is an architectural change in the sense of INTENT's Stability
|
||||
Note and should be rare and deliberate. The headless/API-first posture is fixed by the ratified
|
||||
INTENT amendment.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user