Compare commits

...

170 Commits

Author SHA1 Message Date
776f5af5a7 Normalize agent instructions and workplan frontmatter (STATE-WP-0067)
- Align agent files with on-disk workplan prefixes (infer from workplan ids)
- Set workplan domain to registered domain_slug; add topic_slug where applicable
- Repair frontmatter delimiter formatting; migrate legacy task status literals
- Regenerate AGENTS.md, CLAUDE.md, and .claude/rules from State Hub templates
2026-06-22 23:16:28 +02:00
fd961c83b4 Add .repo-classification.yaml (CUST-WP-0050 T11 agent first-pass) 2026-06-22 17:47:43 +02:00
cca5bf83c3 Add credential routing instructions for all agent runtimes
Propagate shared credential-routing section (Codex, Claude, Grok, llm-connect)
from state-hub template via scripts/propagate_credential_routing.py.
2026-06-18 22:48:39 +02:00
def699c1eb feat(adapters): GitShardAdapter history adopt + cross-substrate integration (WP-0012 T3)
Adopt git-native history (TSD §A.5): a VERSION-gated history(key) surfaces the
commit list for a path (newest-first sha + subject) — declared by every git-IS-store
shard, read-only or not. Integration proves the union/overlay/edit machinery works
unchanged across folder + git substrates: resolve/chorus span both, edit through a
git shard fast-forwards as a commit, apply-under-drift refuses on an external commit
(sha drift) without clobbering, and a read-only git target keeps the overlay as a
draft. SCOPE updated; WP-0012 done. 196 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:41:19 +02:00
a4e0f52ec1 feat(adapters): GitShardAdapter write=commit + current_rev drift (WP-0012 T2)
Writable mode: write(key, body) stages and commits the file (skipping a no-op so
no empty commit is created), returning the page at the new commit sha. The
writable profile declares WRITE + VERSION with PER_PAGE granularity. current_rev
is the per-path commit sha, so a write — or an external commit to the same path —
moves it, driving apply-under-drift. Passes the conformance positive-write probe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:38:41 +02:00
4231daf94f feat(adapters): GitShardAdapter read path + git-IS-store profile (WP-0012 T1)
A second substrate validating the contract beyond plain folders: a git-IS-store
shard reading Markdown from a git repo. Keys are tracked *.md paths; read returns
a Page whose source_rev is the per-path last-commit sha (so an edit to one page
never drifts another); profile is git-IS-store / substrate=git / history=git-native
/ addressing=path, validated against the §6.5 implication rules. Passes the
conformance read path with honest absence of unclaimed verbs. Zero new deps
(git CLI via subprocess). No core changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:36:28 +02:00
37681d89b6 feat(incremental): wire maintained tier behind views; rebuild fallback (WP-0011 T4)
Route InformationSpace.all_pages through a maintained UnionIndex: equivalence is
served from the incrementally maintained index (curator bindings re-synced live
from the log fold + detected content edges), exposed in decision-log string form
so results are a behaviour-preserving superset. The index is built lazily and
rebuilt (bounded fallback) when the union mutates (attach/edit invalidate it);
reindex() forces a rebuild and verify_index() runs the I-2 self-healing checker.
all_pages() gains an optional equivalence_groups source (default = fold) so
direct callers are unaffected. SCOPE updated; WP-0011 done. 173 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:21:39 +02:00
a8e65235a8 feat(incremental): I-2 digest + consistency-checker (WP-0011 T3)
A Merkle-style digest summarizes the derived tier (per-identity fingerprint +
incident edges as order-independent leaves) so equal states have equal digests
and the digest is stable under equivalent event orders. A ConsistencyChecker
recomputes the authoritative fold from the current source, compares it over a
sampled region, and on mismatch scoped-recomputes just the affected identities —
self-healing missed-delta drift, corrupted internal state, and vanished pages.
Makes derived = f(canonical) verified, not asserted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:16:50 +02:00
d7d046cac0 test(incremental): delta maintenance == rebuild, retraction + split (WP-0011 T2)
Verify change-driven maintenance keeps the equivalence index equal to a
from-scratch rebuild under add / edit / remove: an edit into a new bucket
retracts the stale edge, an edit into equivalence adds one, and removing a
connector node propagates a retraction that splits a chorus. Equality checked
against a fresh build() oracle on every operation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:14:32 +02:00
0b3ab2086f feat(incremental): indexed equivalence — blocking + verify (WP-0011 T1)
Detect equivalence (distinct identities holding the same page) without pairwise
O(N²): MinHash/LSH bands over content shingles + normalized-title buckets
generate candidates (blocking), then exact-fingerprint or Jaccard>=threshold
confirm them (verify), with curator decision-log bindings always forming edges.
Groups are the connected components of the edge set. Includes the incremental
add/update/remove internals used by T2. Matches a brute-force oracle. New
incremental/ package (minhash primitives + EquivalenceIndex).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:13:06 +02:00
d85d019543 feat(views): wire derived views onto InformationSpace + integration (WP-0010 T5)
Expose backlinks(name), recent_changes(), all_pages(), site_map() on
InformationSpace. Integration test exercises all four over two shards (BackLinks
aggregate across shards, AllPages/SiteMap span the union, RecentChanges merges an
alias decision with shard edits). SCOPE updated; WP-0010 done. 152 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:05:12 +02:00
3a5acdcb28 feat(views): AllPages + SiteMap enumeration views (WP-0010 T4)
AllPages enumerates the union's distinct pages, collapsing chorus (same key
across shards) and equivalence-bound identities into one entry via union-find,
noting divergence when members' bodies differ (collapse acknowledged, not
silent). SiteMap builds the namespace tree from page placements, spanning shards.
Both derived/recomputable and presentation-free.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:03:15 +02:00
34b0c539f3 feat(views): RecentChanges merged change feed (WP-0010 T3)
One newest-first feed merging the coordination journal (overlay/alias/fork/merge/
binding decisions, with actor + payload) and shard change signals (page
source_rev / mtime). Each entry carries provenance: the originating shard for an
edit, or 'coordination' (and the actor) for a decision. Non-temporal revision
tokens are skipped gracefully. Derived/recomputable; notify-streaming later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:59:11 +02:00
da540d4eea feat(views): BackLinks derived view over the union link graph (WP-0010 T2)
For any page name, the set of pages that link to it: extract wikilinks from every
union page (new UnionGraph.iter_pages enumeration) and index the resolved ones by
target name. Red-links create no backlinks; entries carry source provenance; a
chorus target aggregates the backlinks of all members under one name. Derived/
recomputable, stores nothing canonical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:56:48 +02:00
951b24300d feat(views): wikilink + red-link model (WP-0010 T1)
A CommonMark wikilink extension: extract [[Target]] / [[Target|label]] from a
page body (skipping fenced + inline code, preserving offsets), and resolve each
target through the union — resolved is a link, unresolved is a createable
red-link (never a dropped reference). CamelCase auto-linking is off by default,
opt-in per space, and never double-counts a target already inside [[...]]. Link
model + resolution are core; rendering stays L6. New views/ package.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:55:06 +02:00
c731c96634 feat(coordination): git backend wiring + verbatim log migration (WP-0009 T4)
InformationSpace.git_backed(space_id, repo_path) wires the git coordination log;
the default constructor stays in-memory for tests (new keyword-only store=). A
one-time importer (migrate_space / import_log / JSONL export+import) replays an
existing in-memory or JSON log into git verbatim — preserving seq, timestamp and
actor (union-without-erasure) and refusing out-of-order import. Same fold after
migration; no behavioural change to overlay/union. SCOPE updated; WP-0009 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:49:55 +02:00
f0fee65cc0 test(coordination): cross-process read-your-writes + fold parity (WP-0009 T3)
Verify the git backend's fold reads the durable log into CoordinationState with
unchanged semantics, and that read-your-writes holds across separate handles and
separate OS processes against the same space ref (one test spawns a real
subprocess that appends, then reads it back). Cross-process fold equals the
in-memory fold for the same event sequence (derived = f(log)).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:47:08 +02:00
34432c2e15 feat(coordination): per-space append authority (lease) (WP-0009 T2)
A single append authority per space serializes appends into a total order: at
most one node holds a space's lease; only the holder writes, non-holders forward
their append intent to the holder. Leases are time-bounded and re-grantable, so
a dead holder's lease expires and a new node resumes from the log head (seq stays
contiguous). A stale ex-holder discovers it is no longer the holder and forwards
rather than writing, so a partitioned node cannot fork the log. Works over both
in-memory and git stores. Single-coordinator only (distributed leasing out of scope).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:45:52 +02:00
45a858ead0 feat(coordination): git-backed DecisionLog event store (WP-0009 T1)
Factor DecisionLog storage behind an EventStore abstraction: InMemoryEventStore
stays the default/test double, GitEventStore makes the coordination log
git-addressable. Each space is a ref (refs/spaces/<sha1>); append writes an
immutable one-blob commit and advances the ref under compare-and-swap, so the
commit chain is the per-space total order and a racing appender can never fork
the log. Deterministic stable-JSON event serialization. Zero runtime deps
(git CLI via subprocess). API and fold unchanged across backends.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:41:27 +02:00
b31e9bc337 Add capability registry scaffold and seed entries from reuse-surface
Bootstrap registry/indexes/capabilities.yaml and migrate helix_forge
capability entries owned by this repository for federation publishing.
2026-06-16 01:34:23 +02:00
e50dcc6b5c chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:57:09 +02:00
a165cced33 feat(engine): ext.struct typed-records built-in; close engine implementation (WP-0014 T6)
engine/extensions/struct.py: ext.struct (typed records) — in-text frontmatter
parse + ON_WRITE validation (allowed-fields, content-preserving), ON_READ tags
PageShape.TYPED_RECORD, ON_PROFILE raises structured-payload. Proves the framework:
feature absent when off (opaque prose, honest profile), present + profile-reflected
when on; works through InformationSpace edit. SCOPE updated. 6 tests, 107 total,
~97% coverage, pyflakes clean. Marks T6 + SHARD-WP-0014 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:56:22 +02:00
8393a9c55d chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:36:55 +02:00
ff96ee0c48 feat(engine): EngineShardAdapter — engine as a canonical-mode shard (WP-0014 T5)
engine/adapter.py: EngineShardAdapter implements adapters.ShardAdapter (read/write
run extension transform hooks; profile derived from active extensions, E-5;
current_rev for apply-under-drift) + build_engine_shard() helper (explicit ids or
activation provider). runtime.available() added. Engine shard passes assert_conformant
and attaches to an InformationSpace — resolve + edit (overlay->apply->write-through)
work, and the declared profile reflects the active extensions. 5 tests green, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:36:24 +02:00
8b353f1077 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:27:57 +02:00
b9bb1f7d10 feat(engine): capability profile derived from active extensions (WP-0014 T4, E-5)
engine/profile.py: engine_base_profile() (kernel-only c2-minimum profile),
ProfileContribution (an extension's ON_PROFILE contribution: axis raises + verbs),
derive_profile() folds active extensions' contributions onto the base in deterministic
order then validate() — so configuration->capability is one chain and composition can
never yield an impossible profile (encrypted+native-query rejected). 5 tests green,
96 total, pyflakes clean. Marks T4 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:27:11 +02:00
c40fa3c934 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:16:54 +02:00
54c2bf2ae5 feat(engine): per-shard extension activation (WP-0014 T3, ADR-0001)
engine/activation.py: ActivationContext (shard/tenant, no authz), pluggable
ActivationProvider protocol, StaticProvider standalone default (zero-dep, global
flags + per-shard scoping + per-ext config), ActivationResolver (candidate ids ->
active set / activation profile), and feature_control_provider() lazy factory
(returns None when feature_control_sdk absent -> degrade to static; OpenFeature-
shaped when present). Availability only. 6 tests green, coverage held, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:15:33 +02:00
6d8bd837a4 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:12:18 +02:00
b48a99d3c2 feat(engine): typed-extension runtime (WP-0014 T2)
engine/extension.py: Extension contract (id/provides/declares_types/depends_on/
conflicts_with + bound hooks), ExtensionRuntime (register-with-verification,
activate = dependency closure + conflict + type-collision checks rejecting
impossible profiles), and ActiveExtensions with deterministic (topological, id-tie)
hook dispatch — transform hooks chain, collect hooks gather. 9 tests green,
coverage floor held, pyflakes clean. Marks T2 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:11:18 +02:00
9b7f86ba69 test: harden suite with error-path contracts + coverage floor (98%)
Adds tests/test_error_paths.py covering real failure contracts (red-link
single() KeyError, unknown/unattached-shard apply_overlay, kernel.delete
missing, conformance survives a broken profile, Placement str). Adds a
[tool.coverage.report] fail_under=90 floor (engages on pytest --cov, not bare
pytest). 76 tests, 98% coverage, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:05:16 +02:00
74142096d0 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 23:58:54 +02:00
2100e956aa feat(engine): page-store kernel skeleton (WP-0014 T1)
engine/ package: EngineKernel (in-process page store with per-page version
history; create/edit-as-version, recoverable delete-tombstone, keys, current_rev)
+ wikilink extraction + in-shard link resolution / red-link detection (EC-1..EC-4).
Reuses model/provenance; git-IS-store backing slots in later. 6 tests green,
pyflakes clean, full suite green. Marks T1 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:57:31 +02:00
e62560eb5a chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 23:52:14 +02:00
b147d3e831 workplan: SHARD-WP-0014 wiki-engine implementation (kernel + typed-extension runtime + activation)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:51:12 +02:00
cdcf4b09aa adr: ADR-0001 engine activation via feature-control (OpenFeature, availability-only)
Records the accepted decision: shard-wiki's native engine uses feature-control via
OpenFeature for per-shard extension activation (availability only, never authz),
provider-pluggable with a LocalProvider standalone default (mirrors the identity
ladder), at the engine layer, consuming the mature feature-control.evaluate slice.
Adds spec/adr/ series + README; hub decision abf7830f recorded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:48:49 +02:00
b21efe307b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 23:05:45 +02:00
e18397272a spec(SHARD-WP-0013 T6): wire-up + close-out
spec/README + SCOPE list WikiEngineCoreArchitecture.md; CoreArchitectureBlueprint
cross-links the engine as a canonical-mode shard (federation/union stay in the
orchestrator). reuse-surface engine capability promoted D2->D3 (4204255).
Marks T6 + SHARD-WP-0013 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:02:15 +02:00
0ee972f2e2 spec(SHARD-WP-0013 T5): WikiEngineCoreArchitecture.md — small core + typed extensions
Headless, API-first, agent-optimized native engine = canonical-mode shard backend.
Thesis: a page-store kernel with a typed-extension runtime; everything beyond the
c2-minimum is a typed extension activated per shard, and the shard's §A capability
profile is DERIVED from its active extensions (configuration->capability->conformance).
9 engine invariants (engine-is-one-shard, small kernel, per-shard activation,
profile-from-extensions, headless/agent-first, reuse-not-reinvent, typed+verified).
Kernel (4 concepts), typed-extension model (typed hooks + deterministic composition +
feature-control activation), T2 featureset/conflict-mediation realized, engine-as-shard,
agent-first API surface, module sketch, reuse (consumes feature-control/authorization;
G1 framework proposal), traceability, decisions/open, stability note. Marks T5 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:54:40 +02:00
bb1b54e0af chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 22:45:17 +02:00
b70f1c9acc intent(SHARD-WP-0013 T4): ratify additive native engine — headless, API-first, agent-optimized
Ratified by tegwick (decision 84ffdb48 resolved). INTENT.md amended: native
reference wiki-engine as a canonical-mode shard backend, reframed as HEADLESS &
API-FIRST — small typed-extension core optimized for integrating heterogeneous
data sources and efficient agent/automation access; no bundled UI. Edits:
Primary Utility framing + bullet, non-goal clarifier, new Design Principle,
Stability Note amendment. SCOPE.md: engine core in scope, UI/rendering out.
Orchestrator role unchanged; union-without-erasure + shard sovereignty preserved.
Marks T4 done; unblocks T5 (WikiEngineCoreArchitecture.md).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:43:46 +02:00
8de044bbde chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 22:29:53 +02:00
67d851be0b history(SHARD-WP-0013 T3): reuse-surface gap proposals + consumptions; derived-views registered
Registered capability.wiki.derived-views in reuse-surface (8 wiki.* total, index 20,
validate ok, pushed d985e68). Tracked contributions note (history/260615-...): proposes
two cross-cutting gaps to reuse-surface (G1 typed-extension-framework, G2
translation-fidelity) and records consumptions (feature-control.evaluate,
authorization.policy-evaluate). Sent to the reuse-surface agent via send_message. Marks T3 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:28:29 +02:00
46ec6f2a5f spec(SHARD-WP-0013 T2): engine-facing capability structure layer over the UC catalog
Adds 'Capability structure for the wiki engine (reuse-surface-aligned)' to
UseCaseCatalog.md (no UC renumbered): a small always-on engine core (EC-1..EC-5),
ten typed per-shard-activatable extensions (X-OVERLAY/AUTHZ/VIEW/STRUCT/ADDR/COMP/
PROV/COLLAB/FED/ATT) each mapped to a reuse-surface capability with activation
defaults, and a conflict-mediation map turning conflicting requirements into
mechanisms. Bridges demand -> the engine typed-extension model (T5). Marks T2 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:16:06 +02:00
ad4a2dbf5a chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 22:11:16 +02:00
23bc597343 chore(SHARD-WP-0013): T1 done — shard-wiki registered on reuse-surface (7 wiki.* capabilities)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:10:12 +02:00
ec7a1ec946 spec(SHARD-WP-0013): fold in confirmed reuse-surface coordinates + mechanism
reuse-surface = capability registry at /home/worsch/reuse-surface (helix_forge):
Markdown entries via templates/capability-entry.template.md, D/A/C/R maturity
vectors, reuse-surface CLI (validate/query/export/overlaps), hub at
reuse.coulomb.social. T1 now registers shard-wiki's capabilities as registry
entries; T3 contributes gaps back via REUSE-WP/agent message; T5 evaluates
reusing sibling feature-control for per-shard activation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 17:32:49 +02:00
d6d7cc555f chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 17:18:00 +02:00
1dd1def40f workplan: SHARD-WP-0013 wiki-engine prep (reuse-surface, UC systematization, WikiEngineCoreArchitecture)
Prep for an additive native wiki-engine framed as shard-wiki's reference
first-party shard backend (canonical-mode shard) — small core + typed-extension
framework, per-shard activation, integrated featureset with conflict mediation.
Tasks: confirm+register on the reuse surface; systematize the UC catalog around a
reuse-surface capability taxonomy; surface capability gaps back; narrow INTENT
amendment + ratified decision (gate); author WikiEngineCoreArchitecture.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 17:16:54 +02:00
6d341cd4e6 chore: restore SHARD-WP-0003 workstream id (re-link to existing finished workstream)
The 56af8185 workstream still exists in the DB (finished); clearing the id only
unlinked it (slug taken, cannot recreate). Restoring re-links the file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:55:55 +02:00
1b3d4aaa39 chore: clear stale SHARD-WP-0003 workstream id (re-register on next sync)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:08:28 +02:00
6e49bd3a4b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 14:07:24 +02:00
d2c73b02d9 workplans: register follow-up implementation roadmap (WP-0009..0012)
WP-0009 git-backed DecisionLog + per-space append authority (keystone backing);
WP-0010 derived views (wikilinks, BackLinks, RecentChanges, AllPages/SiteMap);
WP-0011 incremental union + equivalence index + I-2 verification;
WP-0012 second adapter (git-IS-store) validating the contract on a new substrate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:05:21 +02:00
042e198286 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 13:47:25 +02:00
3ea0cc1226 feat(space): wire write path into InformationSpace; integration (WP-0008 T6)
edit()/overlay()/apply_overlay() on InformationSpace. edit() unifies the write
path through one principled route — draft overlay then apply: write-through-capable
target fast-forwards (APPLIED), read-only target keeps the draft as local truth
(KEPT_DRAFT), external drift refuses (no clobber). Integration tests cover all
four. 64 tests green, pyflakes clean. Flips WP-0008 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:46:32 +02:00
4be2f190a0 feat(union): overlay-aware resolution (WP-0008 T5)
resolve() layers open overlays onto canonical pages (overlay_state=DRAFT always
surfaced; overlaid body projected when policy.show_drafts); draft-only edits make
a not-yet-existing page resolvable. Never hides an unapplied overlay (I-4). Policy
gains show_drafts. 4 tests green, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:29:06 +02:00
7d00ae758e feat(coordination): apply-under-drift (WP-0008 T4)
OverlayEngine.apply: read-only target → KEPT_DRAFT; base_rev==current →
fast-forward write-through (APPLIED, MERGE_DECIDED closes overlay); drift →
REFUSED_DRIFT (no clobber, I-5). 5 tests green, pyflakes clean. (blueprint §8.6)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:25:10 +02:00
715ab1ca00 feat(coordination): patch rendering (unified diff) (WP-0008 T3)
render_patch(overlay, base_body) → Patch (pure difflib unified diff, target-
labelled); is_empty when unchanged. 3 tests green, pyflakes clean. (ADR-05)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 12:32:47 +02:00
d797bc5ee4 feat(coordination): Overlay model + OverlayEngine.draft (WP-0008 T2)
Overlay value type (id, target, base_rev, body, state) recorded as an
OVERLAY_CREATED decision-log event (coordination-canonical); get()/open_overlays()
reconstruct from the fold. 4 tests green, pyflakes clean. (ADR-05, blueprint §8.2)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 11:56:53 +02:00
92d5774baf feat(adapters): writable FolderAdapter + positive write conformance (WP-0008 T1)
FolderAdapter(writable=True) declares WRITE+PER_PAGE, implements write() and
current_rev() (mtime token for drift detection). Conformance gains a
content-preserving positive write probe for WRITE-claiming adapters. 5 tests
green, full suite green, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 11:09:43 +02:00
e24f0034a0 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 10:29:21 +02:00
b2ea276c00 workplan: SHARD-WP-0008 write path (overlay engine, writable adapter, apply-under-drift)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 10:28:37 +02:00
9fedb31d8b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 10:13:40 +02:00
517cf1d282 feat(space): InformationSpace orchestrator + integration; foundation slice complete (WP-0007 T7)
InformationSpace ties the slice together: conformance-gated attach → resolve →
read, with alias() recording coordination decisions in the log. Exposed from the
package root. End-to-end integration test (two folder shards → union read with
layered provenance + chorus + alias redirect + red-link + nonconformant-rejected).
pyflakes clean, 39 tests green. Flips WP-0007 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 10:12:38 +02:00
b44b2a74a4 feat(policy,union): policy leaf + UnionGraph resolution with chorus (WP-0007 T6)
policy/ leaf (CanonicalSource presets, default chorus). union/ UnionGraph:
identity-keyed resolve (alias-redirect via log fold → union lookup → chorus →
red-link); chorus records divergent peers in each page's provenance envelope
(union without erasure); designated-canonical orders the pick. Imports down only.
6 tests green. (blueprint §8.4, ADR-01)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 10:02:09 +02:00
24108b65aa feat(coordination): event-sourced DecisionLog keystone (WP-0007 T5)
Append-only, totally-ordered-per-space decision log (in-process append authority;
git+lease later) with event types overlay/binding/alias/merge/fork, and a derived
fold to CoordinationState (aliases LWW, transitively-merged equivalence groups,
open overlays). derived = f(log); read-your-writes. 6 tests green. (blueprint §8.1)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 09:45:11 +02:00
6d48a1d3e6 feat(adapters): conformance suite — profiles verified not asserted (WP-0007 T4)
run_conformance/assert_conformant verify declared profile == observed behaviour
(profile validates, READ round-trips to the right shard, unclaimed verbs raise
NotSupported); ConformanceReport gives a precise capability diff. FolderAdapter
passes; lying-read and dishonest-write stubs fail. 3 tests green. (TSD §A.2)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 09:42:59 +02:00
9a4e00a05a feat(adapters): ShardAdapter contract + read-only FolderAdapter (WP-0007 T3)
Versioned ShardAdapter ABC (shard_id/profile/keys/read mandatory; optional verbs
raise NotSupported by default = honest absence). FolderAdapter reads a dir of
Markdown into Pages (relpath=key, mtime=source_rev, provenance envelope),
declaring a validated read-only file-store profile. 4 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 09:37:02 +02:00
5a77ea879c feat(model): Identity/Placement/Span, Page, CapabilityProfile (WP-0007 T2)
Identity = stable shard-scoped handle (not a fingerprint); Placement separate;
Span carries a layered provenance delta. Page model (PageShape incl. the four
computational shapes). CapabilityProfile with orthogonal-core axes + validate()
applying §6.5 implication rules that reject impossible profiles. Imports only
provenance/. 8 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 09:00:49 +02:00
aca9bf30f9 feat(provenance): layered ProvenanceEnvelope + effective() leaf (WP-0007 T1)
Dependency-free leaf rail: ProvenanceEnvelope + SpanProvenanceDelta + effective()
(page envelope ⊕ span delta; zero-cost inheritance when uniform). Liveness/
Staleness/OverlayState enums. 6 tests green. (blueprint §7.3, §11)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 08:55:35 +02:00
3b4f10a349 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 08:45:49 +02:00
f2dd2e124a workplan: SHARD-WP-0007 foundation implementation (first vertical slice)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 07:36:36 +02:00
1f3aba7aad chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 02:26:48 +02:00
d65f9e21f3 spec(SHARD-WP-0002): adapter contract (TSD §A, T11-T16+T18); workplan done
Adds the normative Shard Adapter Contract as TechnicalSpecificationDocument §A:
A.1 versioned capability contract (verbs + orthogonal-core spectra), A.2 verified
conformance suite (profiles not self-asserted), A.3 attachment-mode taxonomy +
image-is-not-a-store boundary, A.4 page model (incl. computational shapes,
identity/placement/equivalence, layered provenance), A.5 history portability,
A.6 syntax translation + fidelity report, A.7 addressing/navigation, A.8 gated
computational content. Updates TSD references/UC-count/next-work. Flips all 18
WP-0002 tasks + workplan done. Design layer complete.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:25:50 +02:00
802a80231a spec(SHARD-WP-0002): FederationArchitecture.md — federation design (T1-T10, T17)
Decision records for the 10 federation topics + the federation-model taxonomy,
each with a Decided/Deferred/Open footer: star orchestrator positioning, remix
primitives (overlay-default), identity/placement/equivalence + chorus,
event-sourced coordination journal + per-space append authority, server-primary
incremental union, notify-driven freshness, space lifecycle, reference-not-copy
transclusion, detection-core/resolution-policy preset catalog, the federation-
ops capability matrix, and the six selectable/composable federation models.
References (not duplicates) CoreArchitectureBlueprint + FederationRequirements.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:24:11 +02:00
40575045ca chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 02:20:55 +02:00
a2f15806f5 spec(SHARD-WP-0001): yawex-derived federation requirements (ADR-01..06); workplan done
Delivers spec/FederationRequirements.md — six ADR-ready design notes mapping
the yawex prior art onto the hardened CoreArchitectureBlueprint:
ADR-01 cross-shard resolution (yawex PageLookUp states as a checklist, keyed
on identity), ADR-02 namespace/path + shard roles (path=placement not identity),
ADR-03 derived views core-vs-adapter (BackLinks/RecentChanges/AllPages/SiteMap
core, Search hybrid), ADR-04 concrete provenance envelope fields, ADR-05 overlay/
patch/append lifecycle, ADR-06 wikilink+red-link extension (model core, render UI).
Resolves findings open-Qs #1-#3; access-model thread ratified. Flips WP-0001 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:19:43 +02:00
9b5fceb451 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 02:04:39 +02:00
d39e7f7765 spec(SHARD-WP-0006 T5): track §C as O-8..O-12; refresh decisions/traceability; close-out
Adds O-8 (preset bundles), O-9 (shard sharing vs tenant partition), O-10
(span authz + transclusion ⊕), O-11 (union under unavailability), O-12
(append-log throughput) to §12. Refreshes §14 decisions (event-sourced
coordination, conformance, I-2 verification) and §16 traceability (round-2
review + WP-0006). Flips SHARD-WP-0006 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:03:58 +02:00
1ad70a9c8a spec(SHARD-WP-0006 T4): incremental-equivalence correctness + I-2 verification (§8.7)
Fixes B-4. Incremental delta is not additive: a change processes bucket
exits (retract unsupported edges) + entries (add) + propagation across
equivalence neighbours, not just new candidates. Adds an I-2 verification
mechanism: per-partition Merkle-style digest + background consistency-checker
vs sampled fold → scoped self-healing recompute on drift. I-2 now
eventually-verified, not asserted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:02:33 +02:00
3a753a6f3b spec(SHARD-WP-0006 T3): adapter conformance suite (§6.6)
Fixes B-2. Capability profiles are verified, not self-asserted: the contract
ships a versioned conformance suite that exercises each declared verb/position
against observed behaviour; passing is an admissibility precondition (lying
profiles rejected at registration); mismatch reported as a capability diff.
Makes I-3 / §6.5 sound rather than aspirational.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:01:37 +02:00
08a2148079 spec(SHARD-WP-0006 T2): event-sourced coordination + per-space append authority
Resolves B-1+B-3. Coordination-canonical state = an append-only decision log
in the git journal (events: overlay/binding/alias/merge); queryable current
state = a derived fold (tier-3, indexed). Concurrency: one append authority
per space (lease/leader) → totally-ordered per-space log, read-your-writes
across instances, HA via re-grantable lease, partition yields to log integrity.
Updates §1, §4, §8.1, §8.6, §11. I-6 strengthened (coordination state is now
git-addressable history/patch/review), not bypassed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:00:43 +02:00
cbd29e0a32 spec(SHARD-WP-0006 T1): overview-reconciliation pass (§A)
Fixes A-1..A-4: §4 identity/placement/equivalence (removes the equivalence-
keys-on-identity contradiction), §4 projection (trivial default + extension
point) and provenance (layered), §10 policy surface (adds freshness,
conflict-resolution, compaction, tenant-partition knobs + preset-bundle note),
§3 diagram + §11 header (incremental-first, orthogonal-core). Overview now
matches the hardened body.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:58:48 +02:00
079652b61f chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 01:57:04 +02:00
84ab25eeb9 history+workplan: round-2 blueprint review; SHARD-WP-0006 hardening II
Records the round-2 critical review (history/260615-...-review-2) and
establishes SHARD-WP-0006 to: reconcile overview with hardened body (§A),
settle the journal/coordination-state model (event-sourced decision log;
single-vs-multi-writer concurrency — B-1+B-3), require an adapter conformance
suite (B-2), fix incremental-equivalence correctness + I-2 verification (B-4),
and track §C as O-8..O-11.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:56:42 +02:00
229d24d1cf chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 01:44:05 +02:00
f21b7b5259 spec(SHARD-WP-0005 T9): known scaling risks & open problems; close-out
Adds §12 'Known scaling risks & open problems' (O-1..O-7 with chosen
direction + revisit trigger); renumbers §13-17. Refreshes §14 decisions
(several earlier 'open' items now decided), §16 traceability (links the
review + per-finding section map), and I-1..I-13. Flips SHARD-WP-0005 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:43:14 +02:00
c895d33091 spec(SHARD-WP-0005 T8): tenant isolation of derived tier + history scaling
Fixes B-3/C-3. §9.1 structural per-tenant partitioning of the derived tier
(no shared cross-tenant cache; read-time filtering as defence-in-depth;
reconciles I-2+L5 per partition); new invariant I-13. §8.1 history stays
recoverable AND bounded (gc/repack, squash-compaction of churn preserving
recoverable endpoints, per-shard offload, anti-abuse hooks).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:41:09 +02:00
59c36ac9d1 spec(SHARD-WP-0005 T7): elegance pass — layered provenance, common-case projection, policy module
Fixes D-2/D-3/D-4. §7.3 effective-vs-own provenance (page envelope + span
deltas, near-zero per-span cost). §8.4 projection trivial-by-default
(plain lazy replication), derivation/liveness/view-registry as extension
points only for the computational/typed tail. §11 adds policy/ + provenance/
as dependency-free leaf rails (mechanism never in a rail), tightened import rules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:39:43 +02:00
012d151fe8 spec(SHARD-WP-0005 T6): orthogonal core spectra + implied positions + interaction subset (§6.5)
Fixes D-1. ~6 independent core axes (substrate, write-granularity, opacity,
envelope, access, liveness) with the rest implied via published rules that
forbid impossible profiles; a small named axis-interaction table is the
degradation contract (proof obligation behind 'core logic written once').

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:38:02 +02:00
af840576e9 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 01:37:11 +02:00
0383d78440 spec(SHARD-WP-0005 T5): cache freshness & invalidation protocol (§8.8)
Fixes C-2 (invalidation). Per-binding modes (event-driven push / validator
poll / TTL), hybrid default, operational-envelope coupling (rate-limited
shards must not poll-per-read), single-flight coalescing + batched bulk
invalidation, freshness always represented in the provenance envelope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:36:51 +02:00
dc451b0f4e spec(SHARD-WP-0005 T4): scale the union — incremental-first + indexed equivalence (§8.7)
Fixes C-1/C-2. Incremental change-driven maintenance (notify->delta) is
primary; full rebuild is a rare, envelope-respecting, concurrent fallback
(not required cheap). Equivalence via blocking/LSH candidate-gen + verify +
incremental maintenance, replacing O(N^2). Index is derived, per-tenant.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:36:09 +02:00
04be66161e spec(SHARD-WP-0005 T3): consistency, concurrency & conflict model (§8.6)
Fixes bug B-2. States the guarantee (read-your-writes for journal-owned
coordination-canonical state; causal across the derived tier; eventual+
freshness-labelled for sharded inputs). Conflict detection+representation =
core mechanism, resolution = policy. Overlay-apply-under-drift semantics
(fast-forward / three-way / refuse+re-present) and journal ordering.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:35:00 +02:00
c8dbe9b573 spec(SHARD-WP-0005 T2): split page identity / placement / content equivalence
Fixes bug B-1: page identity is a stable assigned handle (survives edits),
not a content fingerprint; fingerprints identify versions/content for the
equivalence mechanism. Chain: identity -> placements -> equivalence. (§7.2)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:07:37 +02:00
b5d2cbc330 spec(SHARD-WP-0005 T1): re-frame state model — three states, derived=f(canonical)
Replaces the two-bucket thesis with sharded-canonical / coordination-canonical
(journal: overlays, bindings, aliases, merges) / derived-disposable. Fixes the
I-2 contradiction (curator bindings can't be rebuilt). Updates §1, I-2, §3
dependency rule, §4 abstractions, §8.4. (review A-1)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:06:54 +02:00
ee2449d987 chore(SHARD-WP-0005): link workstream + task ids 2026-06-15 01:05:17 +02:00
f1dc7aff61 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 01:03:30 +02:00
dd812abb81 history+workplan: CoreArchitectureBlueprint review; SHARD-WP-0005 hardening
Records the critical review (history/260615-...) and establishes SHARD-WP-0005
to fold its findings (A-1, B-1..B-3, C-1..C-3, D-1..D-4) into the blueprint:
correctness (state re-frame, identity/equivalence split, consistency model),
scale (incremental-first union, equivalence indexing, cache invalidation),
elegance (orthogonal spectra, layered provenance, common-case projection,
policy module), security/history scaling, and a known-open-problems section.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:01:19 +02:00
9b5b393519 spec: CoreArchitectureBlueprint — whole-system architecture from research
The optimal architecture synthesised from INTENT + the full research arc:
- Thesis: canonical at the edges, derived in the middle (orchestrator not engine)
- Dual narrow waist: adapter contract (15 capability spectra) + page model
- 6 layers + provenance/capability rails; L4 union/projection is a rebuildable cache
- Federation-model taxonomy (plural/composable); two-axis projection model;
  moldable view registry; identity != placement; computational content in scope
  as page-model+projection, out as execution platform
- Concrete src/ module layout with downward-only dependency rule
- Canonical data flows; policy surface; tradeoffs; full traceability to INTENT/UCs
References ArchitectureBlueprint.md as the L5 authorization sub-blueprint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:41:14 +02:00
6a88eb7710 chore(SHARD-WP-0002): link T17/T18 state_hub_task_ids; sync task drift
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:27:07 +02:00
d898307b7e chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 00:23:15 +02:00
90a854f841 spec(SHARD-WP-0002): re-fold synthesis v3 + computational page model (T17, T18)
Folds the later syntheses into the federation/adapter-contract workplan:
- T11 capability spectra 13 -> 15 (+provenance granularity, +computational/
  liveness); +derive-projection/execute verbs
- T12 four computational page shapes (UC-83/84) + typed-graph statements
- T13 paired-text/nbdime + DB-version-rows + partial-history honesty
- T14 git-IS-store + direct-DB + image-is-not-a-store boundary
- T15 structured-re-evaluable-value opacity point; non-Markdown computed
- T16 two-axis projection model + moldable view registry + named-chunk transclusion
- NEW T17 federation-model taxonomy (fork+journal / VCS-replication+ping /
  query-join / feed / activity-streams / engine-mirror; selectable+composable)
- NEW T18 computational/executable content (in scope as page-model+projection,
  out as execution platform; execute = gated capability)
- Context/decision-tables/acceptance/order updated; UC range -> UC-26-UC-84

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:22:59 +02:00
3d137c96b6 research: SHARD-WP-0004 synthesis (the computational page model); workplan done
Consolidates the 8 computational/interactive-knowledge dives into one
source/derivation/projection model on two axes (replication<->derivation,
live<->snapshot); four computational page shapes; recommendation that
executable content is in scope as a page-model+projection concern, out of
scope as an execution platform (capability-gated, degrade to snapshot;
no INTENT amendment). Flips SHARD-WP-0004 status: done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:12:03 +02:00
f2f9f31df8 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 00:04:50 +02:00
2c978d71f0 research: Strudel.cc live-coding REPL deep dive; SHARD-WP-0004 T5
Code as live time-based audio performance; the far live end of the
live<->snapshot axis (no faithful static form; static = source + marked
recording). The honesty test for graceful degradation. Enrichment-only
(UC-54/55). Marks T5 done — all 8 batch tasks complete.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:04:16 +02:00
b676579407 research: Processing / p5.js deep dive; SHARD-WP-0004 T4
Program-as-page rendered live at view time (no cached output). Adds
materialization-timing (ahead-of-time vs view-time) + continuity
(one-shot vs continuous) facets to the projection model; execute-in-viewer
= capability+trust. Enrichment-only (UC-54/55). Marks T4 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:57:57 +02:00
7fb90219b5 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 23:55:24 +02:00
56b6cdd110 research: Squeak & Pharo deep dive (image-based Smalltalk); SHARD-WP-0004 T6+T8
Combined memo (justified merge). The image = purest 'live' end; hardens
the image-is-not-a-store boundary (export->files only), generalizing
'attach files not the kernel/image'. Pharo Tonel/Iceberg confirms even
image traditions externalize to git text. Names the live<->snapshot
projection axis (T16). Boundary/enrichment-only, no new UC. Marks T6+T8 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:54:48 +02:00
4acadacfee chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 23:46:19 +02:00
1546ca09bf research: Mathematica Notebooks deep dive; SHARD-WP-0004 T2
Original computational notebook (.nb = a Wolfram expression); confirms
UC-84 notebook page shape is a genus. Refinements: nestable cell groups
(outline tree), structured re-evaluable outputs (new opacity point).
Manipulate/Dynamic = snapshot-only. Enrichment-only (UC-84/54/55).
Marks T2 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:45:45 +02:00
a3ca3e975b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 23:39:36 +02:00
25091dbd2e research: Glamorous Toolkit deep dive (moldable development); SHARD-WP-0004 T7
Moldable views (gtView) = open, type-keyed set of co-equal, computed
projections, none canonical = a moldable view registry refining the
projection model (T16). Lepiter live notebook over git files. Enrichment-
only (UC-47/48/54); no new UC. Marks T7 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:38:56 +02:00
1e3aabc143 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 23:22:19 +02:00
25a714efa7 research: Jupyter Notebooks deep dive; UC-84 (SHARD-WP-0004 T3)
.ipynb JSON cells + embedded computed outputs with fragile execution
provenance; derived output stored inside the source. Non-Markdown/lossy;
kernel = capability, default = present snapshot + static render.
Enriches UC-54/55/59/35; links UC-32/83/79. Marks T3 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:08:13 +02:00
3eb026139f chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 22:48:08 +02:00
5ba6975573 chore(SHARD-WP-0004): mark T1 done
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 22:47:40 +02:00
d4afce3699 research: literate programming deep dive (WEB/weave/tangle); UC-83 (SHARD-WP-0004 T1)
One source → N co-equal derived projections (weave=docs, tangle=code);
named-chunk transclusion; splits replication- vs derivation-projection.
Generalizes compile-to-static (UC-79). Enriches UC-32/44/79; links UC-54.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 22:43:30 +02:00
2397c2c4b3 spec(UseCaseCatalog): restructure into 7 thematic sections (no renumbering)
Section A had accumulated 64 of 82 UCs (every dive appended to its end),
collapsing four distinct themes into one heading. Regroup the orchestration/
adapter UCs into four coherent sections and re-letter the authoring/reading
sections, with UC numbers kept as stable IDs (never renumbered -- they are
referenced across research/, the synthesis, and the workplans):

  A. Information space, federation & coordination
  B. Shard attachment & adapter binding
  C. Page model, structure & content fidelity
  D. Addressing, identity & query
  E. Knowledge work and collaboration   (was B)
  F. Discovery and navigation           (was C)
  G. Provenance and page metadata       (was D; + UC-75 statement provenance)

UC blocks moved byte-for-byte (verified: heading set and Traceability tail
identical to pre-restructure); added an Organization note to Conventions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 22:20:13 +02:00
a7084ec74b spec(UseCaseCatalog): fix open-question numbering + add marker legend
- Renumber stray open-question 23->31 (mis-numbered during the WP-0003 batch;
  questions now run 1-32 sequentially) and add Q32 (federation composition,
  from synthesis v3 S2.5).
- Add a dive-source marker legend at the top of Traceability (18 markers were
  used across the matrix with no single key).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:46:09 +02:00
700566b1e2 research: shard-spectrum synthesis v3 (folds SHARD-WP-0003 batch)
Extends the capability model to the nine WP-0003 engines (~23 systems total).
Per-shard spectra 13 -> 14 (adds provenance granularity); refines merge-model
(+fork/journal-replay, +coexist-with-rank), attachment-mode (+git-IS-store,
+container, +direct-DB, +REST/file-store-hybrid, +external-API payload-format
facet), native-query (+SPARQL/RDF far-end), history (+DB-rows, +partial),
structure (+typed-graph, +inline-embedded objects), content-opacity
(+lossy-exportable), write-granularity (+story-item, +section-anchor).

Headline: new federation-model taxonomy (S2.5) -- federation is plural
(fork+journal / VCS-replication+ping / query-time graph join / feed
aggregation / activity streams / engine-mirror), a selectable/composable
coordination-layer axis feeding SHARD-WP-0002 T1-T6. Folds UC-68-82 into the
workplan; no new UCs, no INTENT boundary change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:35:43 +02:00
af5841a0d7 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 21:15:49 +02:00
8941e131e8 research: UseModWiki deep dive (flat-file ancestor); SHARD-WP-0003 done
SHARD-WP-0003 T9 (final). Lineage dive: the flat-file ancestor (Clifford Adams
c.2000, from AtisWiki/CvWiki) that early Wikipedia ran on (MediaWiki Phase I).
No new capability -- reinforces the minimal flat-file floor (UC-82) and
CamelCase linking (UC-25); historical grounding that the file-store floor is
the field's common root. Enrichment-only: UC-01/40/25/36/41. Marks T9 done and
SHARD-WP-0003 complete (9/9). Feeds SHARD-WP-0002 T11.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:14:26 +02:00
ce4b0cb9e4 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 21:00:52 +02:00
6878a0c184 research: Oddmuse deep dive (minimal single-script wiki); UC-82
SHARD-WP-0003 T7. The minimal file-store floor: one Perl CGI script over
plain-text page files + a keep/ revision dir, no DB/API. Anchors the
graceful-degradation baseline / minimal capability-profile floor -- every
richer shard is measured against it; capability-awareness in its purest form
(profile must express absence; truncated history reported honestly). UC-82.
Enriched UC-40/01/36/41. Marks T7 done. Feeds SHARD-WP-0002 T11/T13.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:59:20 +02:00
87f30e7c62 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 20:55:29 +02:00
e0c3d0699c research: MojoMojo deep dive (Perl Catalyst DB-backed wiki); UC-81
SHARD-WP-0003 T8. Classic relational-DB-backed wiki: Catalyst/DBIx::Class app,
pages + path tree + full history in SQL tables, Markdown body in a column, no
file store and no content API. Anchors the direct-DB-read binding (map schema
-> page model + journal); DB version rows = a third history source beside git
commits and RCS files. UC-81. Enriched UC-02/40/36/34. Marks T8 done.
Feeds SHARD-WP-0002 T14/T13/T11.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:54:40 +02:00
1e962a0aef chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 20:50:58 +02:00
54468b60bc research: Salesforce Quip deep dive (SaaS live docs + embedded objects); UC-80
SHARD-WP-0003 T6. Enterprise document-collab contrast to Notion: a doc +
spreadsheet hybrid where a page is prose interleaved with embedded live
structured objects (spreadsheets, kanban/calendar live apps), reachable only by
REST with HTML import/export (lossy to Markdown), gated by Salesforce
identity/ACL. Generalizes external-API mode with an HTML payload-format facet
(beside Notion block-JSON, Wiki.js GraphQL); pushes page model toward
inline embedded structured objects. UC-80. Enriched UC-57/55/58/36/06. Marks
T6 done. Feeds SHARD-WP-0002 T11/T12/T14.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:50:09 +02:00
9f0fc57428 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 20:41:16 +02:00
ee805d5af7 research: ikiwiki deep dive (compile-to-static, git-distributed); UC-79
SHARD-WP-0003 T4. Wiki compiler: git-canonical Markdown source -> static HTML
(derived publish/projection, not the canonical store). Two new insights atop
the git-IS-store home case: compile-to-static = clean canonical-source vs
derived-output separation (attach source repo, UC-37/UC-56), and a third
federation flavor -- git replication + XML-RPC change-ping (pinger), beside
fedwiki fork/journal (UC-72) and Wikibase SERVICE (UC-74). UC-79. Enriched
UC-31/56/37/33. Marks T4 done. Feeds SHARD-WP-0002 T4/T6/T14.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:40:29 +02:00
188643b0f5 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 19:47:23 +02:00
fee47514a2 research: TiddlyWiki deep dive (single-file wiki, write-granularity extreme); UC-78
SHARD-WP-0003 T3. Whole-file write-granularity anchor: an entire wiki (content
+ app engine) in one self-contained HTML file -> save rewrites the whole file,
no per-page atomicity. Node .tid file-per-tiddler substrate is git-diffable/
fine-grained, so the same engine spans the granularity spectrum by substrate
(cf. Logseq file/DB UC-62, backend-swap UC-43). Tiddler = flexible-field
record (UC-34); filter expressions = native-query tier (UC-52). UC-78
(single-file container-format attach). Enriched UC-35/40/34/52/43. Marks T3
done. Feeds SHARD-WP-0002 T11/T14.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:46:37 +02:00
1281ca689b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 19:41:27 +02:00
dbc42f05db research: git-forge wikis deep dive (Gitea/GitLab/GitHub); UC-76-77
SHARD-WP-0003 T5. The home case: a forge wiki is a separate .wiki.git repo of
Markdown -> page model, history, coordination journal map 1:1 with near-zero
adapter. git-clone universal across all three; wiki content API
capability-varying (GitLab/Gitea yes, GitHub git-only). git IS the canonical
store (not a mirror), so write-by-commit is safe -- resolves the UC-68/Q22
race for this case. UC-76 (clone .wiki.git attach), UC-77 (forge wiki API,
capability varies). Enriched UC-40/02/68/38. Marks T5 done.
Feeds SHARD-WP-0002 T14/T11.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:40:46 +02:00
31ec8ab205 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 19:31:46 +02:00
a6629bdb29 research: Wikibase/Wikidata deep dive (entity-statement graph, SPARQL); UC-73-75
SHARD-WP-0003 T2. Structure & native-query far-end: typed knowledge graph
(items/properties, statements = claim+qualifiers+references+rank), RDF
projection + SPARQL (WDQS/Blazegraph) incl. federated SERVICE, opaque stable
Q/P identity (labels-as-annotation), statement-level provenance. UC-73
(typed-graph shard, lossy render), UC-74 (SPARQL + federated query), UC-75
(per-assertion provenance). Enriched UC-34/58/52/24. Marks T2 done.
Feeds SHARD-WP-0002 T12/T16.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:31:08 +02:00
ecb7d0154e chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 19:05:38 +02:00
d955fa239c chore(SHARD-WP-0003): mark T1 (Federated Wiki) done
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:05:15 +02:00
036dbad816 research: Federated Wiki deep dive (journal/fork/neighborhood); UC-70-72
SHARD-WP-0003 T1. Federation model (not a shard candidate): per-page
append-only semantic-action journal with story as derived replay,
fork-with-site-provenance, neighborhood/roster discovery + chorus of forks.
Prior art for shard-wiki's own pillars: coordination journal (UC-71),
overlay-before-mutation (UC-26 fork), union-without-erasure (UC-72).
Attach as REST/file-store hybrid (page JSON + CORS, UC-70). Feeds
SHARD-WP-0002 T1-T5, T11, T13, T16.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:01:13 +02:00
9acd4e2841 plan: persist state-hub workstream/task IDs for SHARD-WP-0003 and SHARD-WP-0004
State-hub consistency sync created both workstreams and all 17 tasks and
wrote the IDs back into the workplan frontmatter and task blocks; commit
them so the files match the DB.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 18:29:15 +02:00
5ada333bb4 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 18:28:38 +02:00
9b5defa6d8 plan: add SHARD-WP-0003 (engine-dive batch) and SHARD-WP-0004 (computational-knowledge systems)
SHARD-WP-0003 — nine wiki-engine deep dives chosen against the synthesis
coverage review: high-insight (Federated Wiki, Wikibase/Wikidata,
TiddlyWiki, ikiwiki), the INTENT-named git-forge wikis (Gitea/GitLab/
GitHub), and four for breadth/lineage (Salesforce Quip, Oddmuse, MojoMojo,
UseModWiki). Each task = persist a deep-dive memo + promote/enrich the
use-case catalog + log adapter-contract notes for SHARD-WP-0002;
post-batch synthesis-v3 decision.

SHARD-WP-0004 — eight computational / interactive-knowledge systems as
design prior art for executable/computational/live page content and
one-source-many-projections: Knuth WEB/weave/tangle, Mathematica & Jupyter
notebooks, Processing/Processing.js, Strudel.cc, Squeak, Glamorous
Toolkit, Pharo. Carries the question of whether a page can be a live
computational artifact; closes with a "computational page model"
synthesis. Both workplans authored file-authoritative (no state_hub IDs;
sync will create+link workstreams and tasks).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 18:28:06 +02:00
4543f5dc43 research: Wiki.js deep dive (storage-module engine, DB<->Git Markdown, GraphQL); UC-68/69
The closest existing engine to shard-wiki's own design: DB-canonical
(Postgres/MySQL/SQLite) but with a pluggable storage-module abstraction
that bidirectionally syncs clean Markdown (+ YAML frontmatter) to Git
(also FS/S3/Azure), each provider acting as backup or source of truth.
Two big findings: (1) the storage-module interface is concrete
adapter-contract prior art alongside Foswiki::Store, and the closer one
(medium = Markdown in Git); (2) the engine-maintained bidirectional Git
mirror is the ideal file-store attach (clean MD + git history) and, being
bidirectional, makes git commit a write path (overlay/patch as a commit,
no API). Also GraphQL API (introspection = capability discovery;
selective fields = efficient projection) and authn-delegated auth modules
+ path-based rule ACL. Added UC-68 (engine-maintained bidirectional Git
mirror, write-by-commit), UC-69 (typed/introspectable API for schema
discovery + selective projection); enriched UC-06/36/38/40/42/57. Catalog
now 69 UCs. Architecture for SHARD-WP-0002 T11/T14: storage-module
abstraction as 2nd adapter-contract prior art, engine-maintained Git
mirror as attach+write surface, GraphQL introspection for capability
discovery.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 17:50:27 +02:00
714eaf6b5a synthesis v2: extend shard spectrum to full dive set (13 spectra); re-fold into SHARD-WP-0002
Extend the cross-dive synthesis from nine systems to the full fourteen
(added Joplin, Logseq, the CRDT cohort Anytype/AFFiNE/AppFlowy, Trilium).
Spectra grown 11 -> 13: new merge-model (none/git/conflict-notes/
native-CRDT) and content-opacity (plaintext/whole-shard-E2EE/per-item),
plus identity-vs-placement emphasis (Trilium note/branch DAG), Logseq's
in-file id:: addressing sweet spot, build-your-own derived query index,
and a six-mode attachment taxonomy (file-store native/interchange-mirror,
in-engine-host, local-REST, external-API, CRDT-replica, P2P). New
through-lines: CRDT changes the merge math; identity != placement;
metadata can be computed; attach surface != native store (+ substrate
migration). UC fold-in extended UC-44-59 -> UC-44-67.

Folded into SHARD-WP-0002: T11 thirteen spectra; T12 computed metadata +
multi-placement/DAG identity; T13 CRDT-log supplement; T14 full six-mode
attachment taxonomy; T15 HTML source model; T16 identity!=placement +
derived index. Context inputs += four dives; acceptance UC-26-67. No new
tasks, no new UCs (synthesis only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 17:25:32 +02:00
5697669063 research: Trilium (TriliumNext) deep dive (note cloning/DAG hierarchy, attribute inheritance, HTML-native); UC-66/67
SQLite-local PKB whose standout is note cloning — a single note can sit in
multiple tree locations at once via the note (identity) vs branch
(placement) split, so the hierarchy is a DAG, not a tree, with no single
canonical path. The identity-not-equal-placement model is the clean way to
represent a page in multiple locations/shards and the namespace-level form
of the clone/reference primitive. Also: attributes (labels + typed
relations) are inherited + templated, so metadata is computed (own +
inherited + template), not a flat bag; content opacity is per-item
(per-note encryption / protected notes), refining the proposed 12th
spectrum; HTML-native (CKEditor, lossy to Markdown); dual extension
surface (scripting code notes + ETAPI token REST). TriliumNext is the
active community fork of zadam's Trilium (TWiki->Foswiki pattern). Added
UC-66 (DAG hierarchy / note cloning), UC-67 (inherited/templated
attributes, effective vs own); enriched UC-15/22/34/38/42/61. Catalog now
67 UCs. Architecture for SHARD-WP-0002 T11/T12/T14/T15/T16: DAG namespace
+ identity/placement split, computed/inherited metadata, per-item content
opacity, HTML source model, scripting + ETAPI host surfaces.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 16:44:49 +02:00
3e0c05a2b3 research: local-first workspaces cohort deep dive (Anytype/AFFiNE/AppFlowy) — CRDT substrate; UC-64/65
Combined cohort memo for the three open-source local-first all-in-one
workspaces, studied together because they share the trait none of the
prior eleven systems had: the store is a CRDT (Anytype any-sync, AFFiNE
Yjs/y-octo, AppFlowy Yrs). The cohort thesis: CRDT changes the adapter
math — the backend performs conflict-free merge itself (shard-wiki must
not impose git/text merge; speak the CRDT via a replica or stay
projection/overlay), history is a CRDT update log (supplement not git),
and the attach surface is a replica or self-host sync endpoint, with
Anytype adding P2P (no single endpoint) + E2EE (opaque). Added UC-64
(CRDT-synced shard with native merge), UC-65 (P2P/decentralized shard,
no single canonical endpoint); enriched UC-36/47/48/51/55/57/58/61.
Catalog now 65 UCs. Architecture for SHARD-WP-0002 T11/T14: merge-model
capability (proposed 13th spectrum: none/git/native-CRDT), CRDT-replica
substrate + P2P/no-central-endpoint attach mode, endpoint-model
capability, MCP as an integration surface. Boundary: CRDT local-first
candidate shards — attach a replica/projection as pages, respect native
merge, never re-drive their sync; not substrates and not the federation
layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 16:21:08 +02:00
6ccf349209 research: Logseq deep dive (block-graph on plain Markdown files, in-file block IDs, derived Datalog index); UC-62/63
Occupies the point the other modern tools leave empty: block-graph
semantics (UUID-addressable, embeddable, queryable blocks) stored as plain
Markdown/Org files on disk, with a DataScript graph derived from the files
(files canonical, index derived). The bridge between Roam (block-DB) and
Obsidian (file-over-app). Headline finding: Logseq resolves the
addressing-spectrum tension — block-level addressing that is also
git-diffable in-file text (id:: property) — and proves a file-backed shard
can serve rich Datalog queries via a derived index. Also: file->SQLite
"DB graph" migration is a live UC-43 (substrate swap under stable
identity); whiteboards = non-Markdown content; dual-attachable (file-store
direct with a Logseq format profile, or in-app plugin). Added UC-62
(attach block-graph-on-plain-files shard), UC-63 (serve structured
queries over a file shard via a derived index shard-wiki builds — converse
of UC-52); enriched UC-32/34/43/50/51/52/55. Catalog now 63 UCs.
Architecture for SHARD-WP-0002 T11/T14/T16: Logseq format profile,
derived-query-index capability, substrate-migration tolerance, in-file
block addressing as the T16 span-address target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 15:27:37 +02:00
e00c160a43 research: Joplin deep dive (SQLite-local/Markdown-on-sync, interchange-format attach, E2EE content opacity); UC-60/61
Architecturally distinct from the Obsidian/Roam/Notion trio: content is
Markdown, the local store is SQLite, and the sync representation is a
folder of per-item Markdown+metadata files on a backend of choice
(filesystem, WebDAV, Nextcloud, Dropbox, OneDrive, S3, Joplin
Server/Cloud). Two new findings: (1) attach the sync/interchange mirror,
not the app or the SQLite store, offline and app-independent, landing on
INTENT's WebDAV/Nextcloud/S3 backends (UC-60, distinct from native
on-disk store UC-40 and app-files UC-53); (2) E2EE => content opacity, a
proposed twelfth capability spectrum for encrypted-at-rest shards (UC-61).
Also: Joplin is the file-sync-daemon boundary case (attach the mirror as
pages, never re-sync); store-minted page-level :/id links (addressing
spectrum middle); dual surface (plugin host + local Data API on
localhost:41184). Enriched UC-31/36/38/40/51/55. Catalog now 61 UCs.
Architecture for SHARD-WP-0002 T11/T14: interchange-format attach,
content-opacity field, local-REST sub-mode, format-aware file-store
profiles.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 14:21:31 +02:00
26bee2cc84 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 14:02:06 +02:00
fae3cbde8b synthesis: shard spectrum across nine systems; fold into SHARD-WP-0002 (T11 eleven spectra, new T16)
Cross-dive synthesis (research/260614-shard-spectrum-synthesis/) reading
the two Nelson conceptual systems, four engines, and three modern tools
across each other: a shard family matrix and eleven capability spectra
(addressing, content identity, structure, history, native query,
translation, attachment mode, operational envelope, access grant, write
granularity, content types) — positions anchored at both ends by a real
system, federation ops degrading by position. Through-lines:
files-canonical/index-derived wins; fine-grained addressing is adoptable;
transclusion=clone=embed is one primitive; structure/history federate iff
in-text; attach mode is a per-binding choice; Notion proves the platform
can enforce no-silent-mutation.

Folded into SHARD-WP-0002: T11 reframed around the eleven spectra; T12
page model stretched four ways (prose + typed records + non-Markdown
assets + query-defined); T13 adds Notion supplement; T14 rewritten as the
three-mode attachment taxonomy (file-store / in-engine-host /
external-API); T15 adds lossy-with-fidelity-report; new T16 (addressing,
content identity, dimensional/query navigation). UC coverage UC-34-43 ->
UC-34-59. Workplan now 16 tasks. No new UCs (synthesis only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 14:01:49 +02:00
64796f7b3a research: Notion deep dive (closed block-DB SaaS, external REST API only, database-as-pages); UC-57/58/59
The closed/hosted/schema-rich extreme: everything is a block (UUID id,
type, properties, ordered child content, single parent), pages are blocks
and database rows are pages with a schema, Postgres-backed hosted SaaS.
Databases add typed properties + relations + rollups + formulas across
many views = the apex of wiki-page-as-structured-record. Extension model
has no in-app plugin runtime; the only extensibility is the external REST
API (+ webhooks 2026) inside a tight envelope (~3 rps, eventual
consistency, recursive child fetch, scoped/revocable per-page grants).
Adds the third attachment mode (external-API-only) alongside file-store
(Obsidian/TWiki) and in-engine host (Roam/XWiki); Notion enforces no
silent remote mutation via scoped grants. Added UC-57 (attach closed
external-API-only shard w/ operational envelope + scoped grant), UC-58
(typed database w/ schema+relations+views, no flattening), UC-59
(lossy-aware translation w/ fidelity report); enriched
UC-31/34/36/39/50/51/52/54/56. Boundary: one external-API candidate shard,
best as projection/mirror/overlay/backup, not a substrate and not the
federation layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 13:44:09 +02:00
f0be5799aa research: Obsidian deep dive (file-over-app vaults, plugin API, ecosystem-popularity signal); UC-53/54/55/56
The most INTENT-aligned tool yet and the file-backed counterpart to Roam:
file-over-app vaults (plain .md folders, files canonical, MetadataCache a
derived index), in-file git-diffable addressing/structure (^block-id,
wikilink embeds, YAML frontmatter), and a plugin API (Plugin onload/onunload
over App.vault/metadataCache/workspace) that doubles as an adapter host —
so a vault is dual-attachable (file-store direct or in-app plugin). Mined
the plugin download rankings as demand evidence per the research brief:
#1 is drawings (Excalidraw, non-Markdown content), query-as-DB
(Dataview/Tasks) is top-tier but an add-on, Git is top-7 (bolt-on history),
Remotely Save shows sync-to-anywhere demand. Added UC-53 (attach local
vault w/ live concurrent native editor), UC-54 (query-defined dynamic
page), UC-55 (non-Markdown content types), UC-56 (outbound publish of a
projection); enriched UC-15/28/34/36/40/51/52. Boundary: a vault is one
file-backed candidate shard, not the federation layer and not a file-sync
target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 12:51:22 +02:00
dfff9ab42e research: Roam Research deep dive (block-graph DataScript DB, transclusion, datalog, Roam Depot extension API); UC-50/51/52
The modern bookend to the Xanadu/ZigZag dives: where those are unbuilt
ideals, Roam shipped fine-grained addressing (:block/uid), live
transclusion (block embeds), bidirectional links, and a queryable
structured space (DataScript datoms + Datalog). Studied as a candidate
DB-backed/API-attached shard (XWiki family) and as a concrete
engine-hosts-adapter surface (Roam Depot onload/onunload over
window.roamAlphaAPI). Added UC-50 (attach block-graph DB shard, block<->page
mapping), UC-51 (adopt native span IDs as portable span addresses), UC-52
(delegate derived views to a shard's native query engine); enriched
UC-32/34/35/38. Boundary: Roam is one candidate shard mapped into the
Markdown-first page model, not a substrate and not the federation layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 11:36:50 +02:00
7c22c468af chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 02:15:13 +02:00
c1424e8863 research: ZigZag/zzstructure deep dive (information space as orthogonal dimensions); UC-47/48/49
Evaluated modelling a wiki information space as a zzstructure: a page is
a cell at the intersection of many co-equal dimensions (namespace tree,
created-from genealogy, version, shard, equivalence, links).
Recommendation: adopt zzstructure as a navigation/visualization/indexing
LENS (a derived dimensional projection over the union), not as the
storage substrate — Git and sovereign shards stay canonical, and the
many-to-many link graph does not fit the one-neighbour-per-dimension rank
rule. Clone <-> transclusion convergence with the Xanadu dive. Added
UC-47/48/49; enriched UC-05/22/26/29; dimensional projection layer +
genealogy-edges-in-journal logged as architecture for SHARD-WP-0002.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 02:14:04 +02:00
0a335eca8b scope: reflect Xanadu deep dive (research row + UC count 25->46)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 01:34:42 +02:00
60e1aa661c research: Project Xanadu deep dive (EDL/reference-not-copy, transclusion, addressing); UC-44/45/46
Xanadu studied as conceptual ancestor, not a candidate shard. Yield:
reference-not-copy EDL/xanadoc validates projection+overlay+union;
content-identity bidirectional transclusion; portable span-address
(tumbler) problem logged as adapter-contract architecture for
SHARD-WP-0002. Recorded design-bug boundaries: reject
single-global-docuverse, single-canonical-instance, baked-in economic
policy. Added UC-44/45/46; enriched UC-24/27/29/32.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 01:24:45 +02:00
316fdeedc2 spec(SHARD-WP-0002): fold adapter-contract constraints into T11-T15
Distill the cross-cutting adapter-contract constraints from the four engine
deep dives (wiki-engines landscape, XWiki, TWiki, Foswiki) into five concrete
design tasks, complementing the federation tasks (T1-T10):

- T11 adapter contract: capability model & versioned interface (Foswiki::Store
  prior art; write-granularity as a capability dimension)
- T12 page model: structured/typed payload representation (XObjects, %META%,
  multi-record, bodiless pages)
- T13 history portability: supplement (DB-internal) vs import (open file history)
- T14 adapter binding: attach path, engine-hosted vs external, backend-swap
- T15 syntax translation & content fidelity (non-Markdown round-trip)

Adds an adapter-contract decision-topics table, cross-links T10<->T11 on the
shared capability vocabulary, and updates context/acceptance/task-order. Tasks
registered in the state hub (workstream 2af4c46d).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:35:09 +02:00
fa7adab239 research: Foswiki deep dive (store abstraction, ext API); UC-42/43
Deep dive into Foswiki focused on its deltas from TWiki (not the shared
lineage): the pluggable Foswiki::Store backend (RcsWrap/RcsLite/PlainFile)
behind a versioned interface via Foswiki::Meta, the OO/MVC core rewrite,
Foswiki::Func + registerTagHandler, DataForms + MetaDataPlugin multi-record,
and WysiwygPlugin TML<->HTML round-trip. The store abstraction is logged as
prior art for shard-wiki's own adapter contract (SHARD-WP-0002).

Catalog (now 43 UCs):
- UC-42 read/write a non-Markdown shard via lossless syntax translation
  (Markdown-first for prose; Foswiki WysiwygPlugin is proof)
- UC-43 tolerate a shard's storage-backend swap (RCS<->PlainFile) under a
  stable identity
- enrich UC-39 (multi-record metadata) and UC-40 (PlainFile direct-attach)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:27:24 +02:00
ffd5459b3e research: TWiki deep dive (impl, plugin API, ecosystem); UC-40/41
Deep dive into TWiki as the file-based Perl counterpoint to XWiki: flat-file +
RCS store (data/<Web>/<Topic>.txt), Webs/Topics, TWiki Forms storing fields as
%META% in the topic text, TWikiML/variables, TWiki::Func API; the plugin handler
callback surface (initPlugin, commonTagsHandler, before/afterSaveHandler,
afterRenameHandler, REST handlers) and package types (Plugin/Skin/AddOn/Contrib);
per-topic ALLOW/DENY access control (origin of yawex's model); Foswiki fork.

Catalog (now 41 UCs):
- UC-40 attach a file-backed engine's on-disk store directly (dual-path attach)
- UC-41 import an engine's native file history (RCS .txt,v) into the journal
- enrich UC-06 (TWiki per-topic ACL lineage), UC-34 (file-embedded %META%),
  UC-36 (RCS-import vs DB-supplement), UC-38 (TWiki handlers as adapter host)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:20:11 +02:00
27da940df1 research: XWiki deep dive (impl, extension interfaces, ecosystem); UC-38/39
Deep dive into XWiki past the landscape scan: component/DI architecture,
document + class/object data model, oldcore, Hibernate storage + xwikircs
history, ObservationManager events, rendering pipeline, multi-wiki; the full
extension-interface surface (components, Java/wiki macros, script services,
UIX/UIXP, JAX-RS REST, event listeners, resource handlers); and the
extensions.xwiki.org ecosystem (XAR/JAR/WebJar, 900+ extensions).

Catalog:
- UC-38 make a wiki engine federation-capable via its native extension API
  (composable integration — first engine-side-direction UC; XWiki is proof)
- UC-39 attach a wiki-as-application-platform shard (bodiless typed pages)
- enrich UC-31 (ObservationManager event-driven sync), UC-34 (XObject model),
  UC-36 (xwikircs internal history) with concrete XWiki specifics

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 19:20:42 +02:00
0473c3f043 spec: extract use cases from wiki-engines landscape research (UC-34–UC-37)
Mine the one research dir not yet promoted (260608-wikiengines-overview) into
the catalog. Adds four heterogeneous-backend attachment scenarios the other
research didn't surface:

- UC-34 attach structured/semantic shard without lossy flattening (SMW, XWiki)
- UC-35 attach shard with coarse write granularity (TiddlyWiki single-file)
- UC-36 supply git-addressable history to internal-history engine (Confluence)
- UC-37 attach static export as read-only backup shard (MediaWiki dump)

Adds a `wikiengines` source token + traceability column and a wikiengines
mapping subsection. The scan mostly reinforced existing UCs and the L0→L4
ladder; its main yield is adapter-contract constraints tracked in SHARD-WP-0002.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 18:39:30 +02:00
7d1fb97968 research: write up wiki-engines landscape scan; send IAM demand to netkingdom
- Add README.md + findings.md to research/260608-wikiengines-overview/ to
  match sibling research convention (was a bare Perplexity dump). findings.md
  synthesizes the engine landscape into adapter-contract implications and maps
  to INTENT (capability heterogeneity, L0->L4 ladder, backend-neutrality).
- Mark the netkingdom integration demand as sent: capability request
  3136a9e2 (auto-routed to netkingdom) + inbox message 46c60f17.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 18:18:45 +02:00
81cfce10d2 chore: State Hub sync for SHARD-WP-0002 workstream and tasks 2026-06-08 14:55:25 +02:00
969dae23b8 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-08:
  - update .custodian-brief.md for shard-wiki
2026-06-08 14:55:18 +02:00
f1384144eb spec: promote federation UCs; add SHARD-WP-0002 architecture workplan
Promote UC-26 through UC-33 from federation research into UseCaseCatalog.
Add SHARD-WP-0002 with ten decision topics (remix primitives, equivalent
page identity, history, composition, notification, lifecycle, transclusion,
consensus presets, capability matrix) targeting spec/FederationArchitecture.md.
2026-06-08 14:55:13 +02:00
3160648b61 research: federation concepts for unified information spaces
Document Federated Wiki mechanics (fork, journal, inverted browser model),
Caulfield lifecycle concepts, and comparison with git-backed wikis,
ActivityPub/XWiki, Xanadu patterns, and yawex REMOTE/VIRTUAL. Map
findings to shard-wiki INTENT without equating fedwiki to orchestration.
2026-06-08 14:43:13 +02:00
5d8608aeae Promote c2 and yawex research into UseCaseCatalog (UC-01–UC-25)
Organize 25 use cases across federation, collaboration, discovery, and
provenance. Include traceability tables mapping research IDs to catalog UCs.
2026-06-08 14:14:46 +02:00
64b77f30c1 Add c2 wiki origins research (Ward Cunningham, terms, use cases)
Explores WikiWikiWeb beginnings: design principles, social conventions,
navigation vocabulary, and documented use cases. Federation explicitly
deferred to a later research track.
2026-06-08 14:05:59 +02:00
202 changed files with 24753 additions and 164 deletions

20
.claude/rules/agents.md Normal file
View File

@@ -0,0 +1,20 @@
## Kaizen Agents
Specialized agent personas available on demand via the state-hub MCP.
**Discover:** `list_kaizen_agents()` — returns all agents with name, description, category
**Load:** `get_kaizen_agent("tdd-workflow")` — returns full instructions; read and follow them
Common agents:
| Agent | Category | When to use |
|-------|----------|-------------|
| `tdd-workflow` | testing | Step-by-step TDD8 workflow for any feature |
| `code-refactoring` | quality | Code quality analysis and safe refactoring |
| `test-maintenance` | testing | Diagnose and fix failing tests |
| `requirements-engineering` | process | Prevent interface/mock mismatches upfront |
| `keepaTodofile` | process | Maintain TODO.md during work |
| `project-management` | process | Track status, determine next steps |
| `datamodel-optimization` | quality | Optimize dataclasses and data structures |
All 17 agents: call `list_kaizen_agents()` for the full list.

View File

@@ -0,0 +1,8 @@
## Architecture
<!-- TODO: Describe the key design decisions and component structure.
Key modules, data flows, external integrations, state machines, etc. -->
## Quick Reference
`~/state-hub/mcp_server/TOOLS.md` — MCP tool reference

View File

@@ -0,0 +1,50 @@
# Credential and access routing
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
for inference. Run this check **before** requesting secrets, API keys, SSH access,
login tokens, or database passwords — in any repo, not only `ops-warden`.
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
other credential need belongs to another subsystem. **Do not** message
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
### Lookup (do this first)
```bash
warden route find "<describe your need>" --json
warden route show <catalog-id> --json
```
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
| Agent runtime | How to orient |
| --- | --- |
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=shard-wiki` is for coordination, not secret vending |
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
### Quick routing table
| I need… | Owner | ops-warden executes? |
| --- | --- | --- |
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes**`warden sign` |
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
| Authorization decision | flex-auth | No — route only |
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
### Anti-patterns (do not do these)
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
- Pasting secrets into Git, State Hub, workplans, logs, or chat
### Other capabilities (reuse-surface)
Non-credential capabilities are usually discovered through **reuse-surface** federation
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
every repo's agent instructions because it is high-frequency, high-risk, and easy to
get wrong.
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`

View File

@@ -0,0 +1,38 @@
## First Session Protocol
Triggered when `get_domain_summary("consumer")` shows **no workstreams**.
The project is registered but work has not yet been structured.
**Step 1 — Read, don't write**
- `~/the-custodian/canon/projects/consumer/project_charter_v0.1.md` — purpose, scope
- `~/the-custodian/canon/projects/consumer/roadmap_v0.1.md` — planned phases
- Scan repo root: README, directory structure, existing code or docs
**Step 2 — Survey in-progress work**
Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete.
**Step 3 — Propose workstreams to Bernd**
Propose 13 workstreams — each a coherent strand, weeks to months, anchored to a
roadmap phase. **Wait for approval before creating.**
**Step 4 — Create workplan file first, then DB record (ADR-001)**
```
workplans/SHARD-WP-NNNN-<slug>.md ← write this first
```
Then register in the hub:
```
create_workstream(topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0", title="...", owner="...", description="...")
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
```
**Step 5 — Record the setup**
```
add_progress_event(
summary="First session: structured consumer into N workstreams, M tasks",
event_type="milestone",
topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0",
detail={"workstreams": [...], "tasks_created": M}
)
```
<!-- Delete or archive this file once past first session -->

View File

@@ -0,0 +1,8 @@
## Repo boundary
This repo owns **shard-wiki** only. It does not own:
<!-- TODO: List what belongs in adjacent repos, e.g.:
- SSH key management → railiance-infra/
- State hub code → state-hub/
-->

View File

@@ -0,0 +1,5 @@
**Purpose:** Git-based Markdown wiki orchestrator and federation layer. Python (src/ layout, hatchling, pytest). Early-stage: scaffold + INTENT.md defined, domain model not yet implemented. See INTENT.md for authoritative scope.
**Domain:** consumer
**Repo slug:** shard-wiki
**Topic ID:** 4c2e5315-2cb9-447c-9d16-a39bdb0aabd0

View File

@@ -0,0 +1,85 @@
## Session Protocol
Dev Hub (State Hub API): http://127.0.0.1:8000
MCP server name in `~/.claude.json`: `dev-hub`
**Step 1 — Orient**
Read the offline-safe brief first — it works without a live hub connection:
```bash
cat .custodian-brief.md
```
Then call the MCP tool for richer cross-domain context when MCP tools are exposed:
```
get_domain_summary("consumer")
```
If MCP tools are unavailable in the current agent session, use the REST API:
```bash
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
```
If the hub is offline: `cd ~/state-hub && make api`
**Step 2 — Check inbox**
With MCP tools:
```
get_messages(to_agent="shard-wiki", unread_only=True)
```
Mark read with `mark_message_read(message_id)`. Reply or act on coordination
requests before proceeding.
Without MCP tools:
```bash
curl -s "http://127.0.0.1:8000/messages/?to_agent=shard-wiki&unread_only=true" \
| python3 -m json.tool
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
-H "Content-Type: application/json" -d '{}'
```
**Step 3 — Scan workplans**
```bash
ls workplans/
```
For each file with `status: ready`, `active`, or `blocked`, note pending
`wait`/`todo`/`progress` tasks.
**Step 4 — Present brief**
1. **Active workstreams** for `consumer` — title, task counts, blocking decisions
2. **Pending tasks** from `workplans/` + any `[repo:shard-wiki]` hub tasks
3. **Goal guidance** — if `goal_guidance` in summary:
- `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"*
- `alignment_warnings`: flag if active work is not aligned with current goal
4. **Suggested next action** — highest-priority open item
5. **SBOM status** — flag if `last_sbom_at` is unset for this repo
If no workstreams: follow First Session Protocol (`first-session.md`).
**During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()`
> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`)
> are First Session Protocol only. Work structure belongs in repo files (ADR-001).
**Session close:**
With MCP tools:
```
add_progress_event(summary="...", topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0", workstream_id="<uuid>")
```
Without MCP tools:
```bash
curl -s -X POST http://127.0.0.1:8000/progress/ \
-H "Content-Type: application/json" \
-d '{"topic_id":"4c2e5315-2cb9-447c-9d16-a39bdb0aabd0","workstream_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
```
If workplan files were modified, ensure the local copy is up to date first:
```bash
git -C <repo_path> pull --ff-only
cd ~/state-hub && make fix-consistency REPO=shard-wiki
```
For repos where implementation runs on a remote machine (e.g. CoulombCore),
use the combined target which pulls before fixing:
```bash
cd ~/state-hub && make fix-consistency-remote REPO=shard-wiki
```
**C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback
will sync the file to match DB. **C-16** (repo behind remote) blocks all writes
until you pull — intentional to prevent clobbering remote progress.

View File

@@ -0,0 +1,19 @@
## Stack
<!-- TODO: Fill in language, frameworks, and key dependencies -->
- **Language:**
- **Key deps:**
## Dev Commands
```bash
# TODO: Fill in the standard commands for this repo
# Install dependencies
# Run tests
# Lint / type check
# Build / package (if applicable)
```

View File

@@ -0,0 +1,40 @@
## Workplan Convention (ADR-001)
File location: `workplans/SHARD-WP-NNNN-<slug>.md`
ID prefix: `SHARD-WP-`
Work items originate as files in this repo **before** being registered in the hub.
Canonical workplan/workstream frontmatter statuses are:
`proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`.
Use `proposed` for a newly drafted plan, `ready` after review against current
repo state, and `finished` when implementation is complete. `stalled` and
`needs_review` are derived health labels, not stored statuses.
Closed workplans may be moved to `workplans/archived/` with a completion-date
prefix: `YYMMDD-SHARD-WP-NNNN-<slug>.md`. The frontmatter id remains
unchanged; the prefix is only for quick visual reference.
Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**:
`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids
`ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed
directly. Promote anything requiring analysis, design, approval, dependencies, or
multiple planned phases into a normal workplan.
Ecosystem todos from other agents arrive as `[repo:shard-wiki]` hub tasks —
visible at session start. Pick one up by creating the workplan file, then registering
the workstream.
Task blocks use this shape:
```task
id: SHARD-WP-NNNN-T01
status: wait | todo | progress | done | cancel
priority: high | medium | low
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
```
Status progression is `todo``progress``done`; use `wait` for waiting or
blocked work and `cancel` for stopped work.
<!-- Ralph Loop rules and HEUREKA sequence: ~/.claude/CLAUDE.md — do not duplicate here -->

View File

@@ -2,21 +2,46 @@
# Custodian Brief — shard-wiki
**Domain:** whynot
**Last synced:** 2026-06-08 10:24 UTC
**Last synced:** 2026-06-15 22:57 UTC
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
## Active Workstreams
### shard-wiki requirements from yawex prior art
Progress: 0/6 done | workstream_id: `0ed023a2-760b-4990-b931-8ee1f41ea08f`
### second adapter — git-IS-store shard (contract validation on a new substrate)
Progress: 0/3 done | workstream_id: `9e24eeb0-c0f0-41e6-a1ca-88d71e4139ea`
**Open tasks:**
- · Design federation page-resolution model (yawex state space as inspiration) `ebc036e4`
- · Define namespace/path model and page+shard roles `431b4d28`
- · Specify union-level derived views (BackLinks, RecentChanges, AllPages, SiteMap, Search) `564545ec`
- · Provenance & freshness model for pages/revisions/projections `738326f5`
- · Overlay / lightweight-patch model (from yawex append/comment) `a268de6a`
- · Markdown link semantics: wikilink + red-link extension `a7499f3e`
- · GitShardAdapter — read over a git working tree/repo `8a1c7c80`
- · Write = commit; current_rev = sha (drift) `b47dfb86`
- · History adopt + integration with union/overlay `4c895f42`
### incremental union maintenance + equivalence index + I-2 verification
Progress: 0/4 done | workstream_id: `78d48bcf-6482-4266-bc81-084b7ec1cd80`
**Open tasks:**
- · Equivalence index: blocking + verify `842f480b`
- · Incremental maintenance (delta, not additive) `2da4e0b8`
- · I-2 verification: digest + consistency-checker `b602ce31`
- · Wire incremental tier behind resolution + views `2f3d083c`
### derived views — wikilinks, BackLinks, RecentChanges, AllPages/SiteMap
Progress: 0/5 done | workstream_id: `2fe15330-ddf6-4b0f-8e55-ada341375d35`
**Open tasks:**
- · Wikilink + red-link model `792660c3`
- · BackLinks (core) `431a54c3`
- · RecentChanges (core) `270c1c31`
- · AllPages / SiteMap (core) `898ba43e`
- · Wiring + integration `7157544b`
### git-backed DecisionLog + per-space append authority
Progress: 0/4 done | workstream_id: `4fb5b29b-955c-4f37-85cf-58b4643ab1ca`
**Open tasks:**
- · Git event-store backend (append = commit/object) `a8fcbb3e`
- · Per-space append authority (lease) `62abd162`
- · Fold over the git log + read-your-writes across processes `8cc3691e`
- · Migration + wiring `281e1db4`
---
## MCP Orientation (when available)

17
.repo-classification.yaml Normal file
View File

@@ -0,0 +1,17 @@
repo_classification:
standard: Repo Classification Standard
version: '1.0'
classified_at: '2026-06-22'
classified_by: agent
category: project
domain: consumer
secondary_domains: []
capability_tags:
- knowledge
- documentation
business_stake:
- product
- experience
business_mechanics:
- coordination
- operation

243
AGENTS.md
View File

@@ -1,62 +1,219 @@
# AGENTS.md
# shard-wiki — Agent Instructions
Guidance for agents working in `shard-wiki`.
## Repo Identity
## Read First
**Purpose:** Git-based Markdown wiki orchestrator and federation layer. Python (src/ layout, hatchling, pytest). Early-stage: scaffold + INTENT.md defined, domain model not yet implemented. See INTENT.md for authoritative scope.
1. `INTENT.md` — aspiration and boundaries (stable; architectural changes are rare).
2. `SCOPE.md` — what we are achieving now and current maturity.
3. `.custodian-brief.md` — State Hub snapshot (generated; do not edit manually).
**Domain:** consumer
**Repo slug:** shard-wiki
**Topic ID:** `4c2e5315-2cb9-447c-9d16-a39bdb0aabd0`
**Workplan prefix:** `SHARD-WP-`
## Documentation Layout
---
This repo follows the CoulombSocial / HelixForge / MarkiTect documentation
layout (recommendation, not strict law). Efficient retrieval by purpose:
## State Hub Integration
| Path | Purpose |
|------|---------|
| `INTENT.md` | Aspiration and boundaries |
| `SCOPE.md` | Top-level view of current achievement; closes gap to INTENT |
| `research/` | Exploration results (`yymmdd-` prefix on files or subdirs) |
| `demand/` | Inbound requests not yet reviewed into spec or workplans |
| `spec/` | Implementation guardrails (PRD, TSD, use cases, architecture) |
| `workplans/` | State Hubregistered implementation tasks |
| `docs/` | Stakeholder documentation (users, developers, humans, agents) |
| `wiki/` | Perspective-free interconnected knowledge (wiki UI when connected) |
| `issues/` | Mirror of relevant open tickets when ticket systems are in use |
| `history/` | Archived material (`yymmdd-` prefix); out of scope for daily work |
The Custodian State Hub tracks work across all domains. Interact via HTTP REST —
there is no MCP server for Codex agents.
**Mode of operation:** close SCOPE → INTENT while learning; refine both as needed.
| Context | URL |
|---------|-----|
| Local workstation | `http://127.0.0.1:8000` |
| Remote via tunnel | `http://127.0.0.1:18000` |
## Domain Vocabulary
Honor terms from `INTENT.md`: shard, root entity, adapter contract, projection,
overlay, coordination journal, shard modes. Do not invent parallel vocabulary.
## Build And Test
### Orient at session start
```bash
pip install -e ".[dev]"
pytest
ruff check
ruff format
# Offline brief — works without hub connection
cat .custodian-brief.md
# Active workstreams for this domain
curl -s "http://127.0.0.1:8000/workstreams/?topic_id=4c2e5315-2cb9-447c-9d16-a39bdb0aabd0&status=active" \
| python3 -m json.tool
# Check inbox
curl -s "http://127.0.0.1:8000/messages/?to_agent=shard-wiki&unread_only=true" \
| python3 -m json.tool
```
## State Hub
Mark a message read:
```bash
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
-H "Content-Type: application/json" -d '{}'
```
Workplans register with State Hub. After workplan changes:
### Log progress (required at session close)
```bash
cd ~/state-hub && make fix-consistency REPO=shard-wiki
curl -s -X POST http://127.0.0.1:8000/progress/ \
-H "Content-Type: application/json" \
-d '{
"summary": "what was done",
"event_type": "note",
"author": "codex",
"workstream_id": "<uuid>",
"task_id": "<uuid>"
}'
```
Finished or canceled workplans move to `history/` with a `yymmdd-` archive prefix.
Omit `workstream_id` / `task_id` when not applicable.
## Where To Put New Material
### Update task status
- Exploratory analysis → `research/yymmdd-<topic>/`
- Raw feature ask or external requirement → `demand/`
- Reviewed design ready to guide code → `spec/`
- Implementation tasks → `workplans/`
- User/dev/agent how-to → `docs/`
- Collaborative unstructured notes → `wiki/`
```bash
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
-H "Content-Type: application/json" \
-d '{"status": "progress"}'
# values: wait | todo | progress | done | cancel
```
### Flag a task for human review
```bash
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
-H "Content-Type: application/json" \
-d '{"needs_human": true, "intervention_note": "reason"}'
```
---
## Session Protocol
**Start:**
1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe)
2. Check inbox: `GET /messages/?to_agent=shard-wiki&unread_only=true`; mark read
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
**During work:**
- Update task statuses in workplan files as tasks progress
- Record significant decisions via `POST /decisions/`
**Close:**
1. Update workplan file task statuses to reflect progress
2. Log: `POST /progress/` with a summary of what changed
3. Note for the custodian operator: after workplan file changes, run from
`~/state-hub`:
```bash
make fix-consistency REPO=shard-wiki
```
This syncs task status from files into the hub DB.
---
## Credential and access routing
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
for inference. Run this check **before** requesting secrets, API keys, SSH access,
login tokens, or database passwords — in any repo, not only `ops-warden`.
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
other credential need belongs to another subsystem. **Do not** message
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
### Lookup (do this first)
```bash
warden route find "<describe your need>" --json
warden route show <catalog-id> --json
```
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
| Agent runtime | How to orient |
| --- | --- |
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=shard-wiki` is for coordination, not secret vending |
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
### Quick routing table
| I need… | Owner | ops-warden executes? |
| --- | --- | --- |
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
| Authorization decision | flex-auth | No — route only |
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
### Anti-patterns (do not do these)
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
- Pasting secrets into Git, State Hub, workplans, logs, or chat
### Other capabilities (reuse-surface)
Non-credential capabilities are usually discovered through **reuse-surface** federation
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
every repo's agent instructions because it is high-frequency, high-risk, and easy to
get wrong.
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
<!-- REPO-AGENTS-EXTENSIONS -->
<!-- Append repo-specific agent instructions below this marker.
The state-hub template sync preserves content after this line. -->
---
## Workplan Convention (ADR-001)
Work items originate as files in this repo — not in the hub. The hub is a
read/cache/index layer that rebuilds from files.
**File location:** `workplans/SHARD-WP-NNNN-<slug>.md`
**Archived location:** finished workplans may move to
`workplans/archived/YYMMDD-SHARD-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
the completion/archive date; the frontmatter `id` does not change.
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
`workplans/ADHOC-YYYY-MM-DD.md` with task ids `ADHOC-YYYY-MM-DD-T01`, etc. Use
this only for low-risk work completed directly; create a normal workplan for
anything needing analysis, design, approval, dependencies, or multiple phases.
**Frontmatter:**
```yaml
---
id: SHARD-WP-NNNN
type: workplan
title: "..."
domain: consumer
repo: shard-wiki
status: proposed | ready | active | blocked | backlog | finished | archived
owner: codex
topic_slug: ...
created: "YYYY-MM-DD"
updated: "YYYY-MM-DD"
state_hub_workstream_id: "<uuid>" # written by fix-consistency — do not edit
---
```
Use `proposed` for a new draft, `ready` after review against current repo
state, and `finished` after implementation. `stalled` and `needs_review` are
derived health labels, not frontmatter statuses.
**Task block format** (one per `##` section):
```
## Task Title
` ` `task
id: SHARD-WP-NNNN-T01
status: wait | todo | progress | done | cancel
priority: high | medium | low
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
` ` `
Task description text.
```
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
To create a new workplan:
1. Write the file following the format above
2. Notify the custodian operator to run `make fix-consistency REPO=shard-wiki`
(or send a message to the hub agent via `POST /messages/`)

View File

@@ -1,53 +1,12 @@
# CLAUDE.md
# shard-wiki — Claude Code Instructions
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Repository status
This is an **early-stage Python repository**. The package scaffold (`src/shard_wiki/`, `tests/`, `pyproject.toml`) exists with only smoke tests — the domain model is not yet implemented. Read `INTENT.md` (aspiration), `SCOPE.md` (current achievement), and `AGENTS.md` (layout and conventions) before designing anything. Close the gap from SCOPE to INTENT via `research/`, `spec/`, and `workplans/`.
## What this project is
`shard-wiki` is a **Git-based Markdown wiki orchestrator and federation layer**, not a wiki engine. It lets multiple heterogeneous wiki-shaped page stores (**shards**) attach to a shared root entity and be presented as a **union of pages**, while preserving each shard's separate storage, provenance, capabilities, and history.
The core job is orchestration across backends — Git repos, repo subdirectories (`wiki/`), Gitea wikis, local folders, Obsidian vaults, WebDAV/Nextcloud directories, Coulomb spaces — never replacing or homogenizing them.
## Core domain model (the concepts code must honor)
These abstractions come from `INTENT.md` and define the architecture. New code should map onto them rather than inventing parallel vocabulary:
- **Shard** — an independently meaningful page store attached to a root entity. Shards have *sovereignty*: their own backend, capabilities, limits, history, and identity model. Not all shards are Git-native.
- **Root entity / information space** — the joined space that shards attach to. Each information space should have a **Git-addressable coordination layer** (history, patches, review, backup, reconciliation) even when individual shards are not Git-native.
- **Shard adapter contract** — the versioned interface a backend implements to participate. Adapters are **capability-aware**: the core must model explicitly which operations a shard supports (read, write, diff, merge, lock, version, publish, accept patches) rather than assuming uniformity.
- **Wiki page model** — a stable, versioned, Markdown-first but backend-neutral representation of pages, paths, links, metadata, revisions.
- **Projection** — a lazy, cache-like local view of remote/external shard content. Prefer lazy projection over eager copying.
- **Overlay** — a non-destructive local edit against a remote, read-only, or capability-limited shard, representable as drafts/patches/commits/merge requests *before* destructive application ("overlay before mutation").
- **Coordination journal** — the Git-backed record of change flows for an information space.
- **Shard modes** — read-only, write-through, mirrored, projected, cached, canonical.
## Design constraints to enforce in code
These are hard boundaries from `INTENT.md`; treat violations as design bugs:
- **Mechanism over policy.** Provide primitives for federation, sync, overlays, patching, conflict detection, projection, reconciliation. Do *not* hard-code one editorial/sync/conflict/canonical-source policy — keep those configurable.
- **Union without erasure.** Always preserve provenance: which shard a page came from, its freshness, whether it is cached, whether it has overlays, whether it diverges from an equivalent page elsewhere. Never hide authorship, conflicts, freshness, or backend limitations.
- **No silent remote mutation.** Do not mutate remote systems without explicit adapter support and user intent.
- **Graceful degradation.** Limited backends must still be usable as read-only/cache/projection/backup/patch targets.
- **Not a file-sync daemon.** Synchronization is wiki-page-semantic, not generic file mirroring.
`INTENT.md` has a "Stability Note": changes that redefine what a shard is, Git's role, how root entities are modeled, or whether this is an orchestrator vs. an engine are **architectural changes** and should be rare and deliberate.
## Build, test, run
Python with a `src/` layout, built via hatchling, tested with pytest. Tests run against the source tree directly (`pythonpath = ["src"]` in `pyproject.toml`), so no install/editable step is required to run them.
```bash
pip install -e ".[dev]" # one-time: install dev tooling (pytest, pytest-cov, ruff)
pytest # run the full test suite
pytest tests/test_package.py::test_version_is_exposed # run a single test
pytest --cov # run with coverage
ruff check # lint
ruff format # format
```
Note: the system `pytest` is 7.4.x; `minversion` in `pyproject.toml` is pinned to `7.0` to match. Bump it if a newer pytest is installed into the dev environment.
@SCOPE.md
@.claude/rules/repo-identity.md
@.claude/rules/session-protocol.md
@.claude/rules/first-session.md
@.claude/rules/workplan-convention.md
@.claude/rules/stack-and-commands.md
@.claude/rules/architecture.md
@.claude/rules/repo-boundary.md
@.claude/rules/credential-routing.md
@.claude/rules/agents.md

View File

@@ -16,6 +16,8 @@ The goal is to allow independently stored and differently implemented wikis, pag
The repository provides a **shard orchestration layer** for interconnected Markdown and markup-based wiki content.
Equivalently, shard-wiki can be used as a **headless, API-first wiki engine** — optimized for **integrating heterogeneous data sources** and for **efficient access by agents and automation** — that ships its own native engine as one (canonical-mode) shard among many. There is no bundled UI: presentation and rendering are consumer concerns.
It allows wiki-like systems to:
* Attach heterogeneous page stores as shards of a shared information space
@@ -30,6 +32,7 @@ It allows wiki-like systems to:
* Run fully standalone with open read/write access and complete change history, then progressively layer multi-tenant enterprise access control through external identity integration
* Allow existing wiki engines to become federation-capable through a shared API
* Allow non-federation-aware systems to participate through adapters and projections
* Serve as a **headless, API-first wiki engine** (a small typed-extension core) that integrates heterogeneous data sources and is consumed efficiently by agents and automation
It transforms disconnected wiki engines, Git repositories, local folders, WebDAV directories, application-specific content stores, and desktop editing workflows into a **composable federated wiki space**.
@@ -85,7 +88,7 @@ A mature `shard-wiki` should allow each participating shard to see the others as
This repository is **not** intended to:
* Replace all wiki engines with a single canonical wiki implementation
* Replace all wiki engines with a single canonical wiki implementation *(shard-wiki MAY still provide its own native, headless, API-first engine as one optional shard backend — see Design Principles — but never as a mandated or universal replacement)*
* Force every shard to use the same backend, database, directory layout, or storage format
* Require every participating system to become federation-aware
* Require every participating shard to be Git-native
@@ -148,6 +151,9 @@ Policy decisions such as conflict preference, canonical source selection, public
* **Composable integration**
Wiki engines should be able to use the `shard-wiki` API to become federation-enabled without reimplementing federation internally.
* **Native reference engine (additive, headless & API-first)**
shard-wiki MAY provide its own native wiki-engine as a **canonical-mode shard backend** — a **small core** with a **typed-extension framework**, activated **per shard** (only what you need). It is **headless and API-first** (no bundled UI; presentation/rendering are consumer concerns) and tuned for **integrating heterogeneous data sources** and **efficient agent/automation access**. It is *one shard type among many*, implemented against shard-wiki's own adapter contract; it does **not** replace other engines, mandate a single implementation, or change shard-wiki's role as an orchestrator. Shard sovereignty and union-without-erasure are preserved.
* **Open by default, progressively governed**
A standalone `shard-wiki` must be runnable with zero external dependencies in a classic Ward Cunningham / c2-style open read/write-for-all mode. Access control is an *additive capability*, not a precondition: the same core progresses — without re-architecture — to authenticated single-user, to group/role-based, to multi-tenant enterprise access control, mirroring the NetKingdom capability ladder (lightweight → expanded).
@@ -201,3 +207,5 @@ Such changes should be rare, because they affect all downstream systems relying
In particular, changes that redefine what counts as a shard, what role Git plays, how root entities are modeled, or whether `shard-wiki` is an orchestrator rather than a wiki engine should be treated as architectural changes.
**Amendment — 2026-06-15 (SHARD-WP-0013 T4, decision `84ffdb48`):** admits an **additive** native reference wiki-engine — **headless, API-first**, a small typed-extension core — as a **canonical-mode shard backend** optimized for data-source integration and agent access. Deliberate, narrow scope change; shard-wiki remains an orchestrator and neither mandates nor replaces other engines. (Mirrors the earlier auth-in-core amendment precedent.)

View File

@@ -17,12 +17,12 @@ Learnings update both SCOPE and INTENT where necessary.
| Layer | State |
|-------|-------|
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
| Code | Foundation slice implemented (SHARD-WP-0007): `provenance` + `policy` leaves, `model` (Identity/Placement/Span/Page/CapabilityProfile), `adapters` (contract + FolderAdapter + conformance suite), `coordination` (event-sourced DecisionLog), `union` (resolution + chorus, overlay-aware), `InformationSpace` orchestrator. Write path added (SHARD-WP-0008): writable adapter, overlay engine (draft→patch→apply-under-drift), edit() unifies write-through + overlay-before-mutation. Native engine implemented (SHARD-WP-0014): `engine` (kernel + typed-extension runtime + per-shard activation [ADR-0001] + capability-profile-from-extensions + EngineShardAdapter + the `ext.struct` built-in) — an engine shard attaches to an InformationSpace as a canonical-mode shard. Git-backed coordination log (SHARD-WP-0009): `DecisionLog` storage factored behind an `EventStore`; `GitEventStore` makes the log git-addressable (each space a ref, append = immutable CAS-guarded commit), a per-space `AppendAuthority` (lease) gives a single-writer total order with re-grantable HA hand-off, cross-process read-your-writes verified, and a verbatim one-time importer (`migrate_space`/JSONL) replays in-memory logs into git; `InformationSpace.git_backed(...)` wires it. Derived views (SHARD-WP-0010): `views` (wikilink + red-link model, BackLinks, RecentChanges, AllPages/SiteMap) — recomputable, provenance-carrying, presentation-free, exposed via `InformationSpace.backlinks/recent_changes/all_pages/site_map`. Incremental-first derived tier (SHARD-WP-0011): `incremental` (indexed equivalence via MinHash/LSH blocking + verify, change-driven delta maintenance with retraction/propagation, Merkle-style digest + self-healing I-2 consistency-checker, `UnionIndex` routed behind `InformationSpace.all_pages` with rebuild as explicit fallback). Second adapter (SHARD-WP-0012): `GitShardAdapter` — git-IS-store substrate (read=tracked *.md, write=commit, current_rev=per-path sha for drift, adopted git-native history), passes conformance, works across folder+git shards in union/overlay/edit with no core change (capability-as-data proven on a second substrate). 196 tests green, ~97% coverage |
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
| Research | yawex prior-art exploration (`research/260608-yawex-prior-art/`) |
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) |
| Demand | NetKingdom integration asks captured, not yet negotiated |
| Spec | Access/history architecture blueprint drafted; PRD/TSD/UCC scaffolds |
| Work | `SHARD-WP-0001` active — 6 design tasks, all todo |
| Spec | CoreArchitectureBlueprint (whole-system, hardened via SHARD-WP-0005/0006) + FederationArchitecture + FederationRequirements + TSD §A adapter contract + ArchitectureBlueprint (auth/history) + WikiEngineCoreArchitecture (headless API-first engine, SHARD-WP-0013) drafted; UseCaseCatalog 84 UCs (+ engine capability-structure layer); PRD scaffold |
| Work | `SHARD-WP-0001` **done** (6 ADRs: yawex-derived federation requirements → `spec/FederationRequirements.md`); `SHARD-WP-0002` **done** (18 tasks → `FederationArchitecture.md` [T1T10, T17] + `TechnicalSpecificationDocument.md` §A adapter contract [T11T16, T18]); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` **done** (all 8 computational-knowledge dives T1T8 complete + "computational page model" synthesis); `SHARD-WP-0005` **done** (9 tasks: CoreArchitectureBlueprint hardened against the 260615 review); `SHARD-WP-0006` **done** (5 tasks: round-2 hardening — overview reconciled, event-sourced coordination + append authority, adapter conformance, incremental correctness + I-2 verification) |
## In Scope (today)
@@ -32,11 +32,15 @@ Learnings update both SCOPE and INTENT where necessary.
- Authorization model design (delegated authentication, core authorization).
- Shard adapter contract and wiki page model (to be specified, then implemented).
- Git-backed coordination journal for information spaces.
- A **native, headless, API-first wiki-engine core** (small typed-extension core, as a
canonical-mode shard backend) — design via SHARD-WP-0013; optimized for data-source
integration and agent access.
- State Hub workplan registration and consistency sync.
## Out Of Scope (today)
- A standalone wiki engine UI or rendering pipeline.
- A wiki-engine **UI or rendering pipeline** (the engine is headless/API-first; presentation
is a consumer concern). A bundled standalone UI is not provided.
- Authentication, credential storage, or user directory implementation.
- Hard-coded editorial, sync, or conflict-resolution policy.
- Generic file mirroring independent of wiki-page semantics.
@@ -52,6 +56,14 @@ live in core.
## Current Planning
Design work is tracked in `workplans/SHARD-WP-0001-yawex-requirements.md`.
Specification outputs from that workplan land in `spec/`. Inbound integration
asks remain in `demand/` until reviewed and promoted into spec or workplans.
Design work is tracked in `workplans/SHARD-WP-0001-yawex-requirements.md`
(yawex-derived resolution, namespaces, overlays) and
`workplans/SHARD-WP-0002-federation-architecture.md` (federation architecture,
decisions, tradeoffs). Research continues under
`workplans/SHARD-WP-0003-engine-dives-batch.md` (remaining new-insight wiki
engines + git-forge wikis + classic engines) and
`workplans/SHARD-WP-0004-computational-knowledge-systems.md` (literate
programming, computational notebooks, live-coding REPLs, image-based/moldable
environments — the executable/computational page-model thread). Specification
outputs land in `spec/`. Inbound integration asks remain in `demand/` until
reviewed and promoted into spec or workplans.

View File

@@ -82,8 +82,10 @@ shard-wiki authorizes; it never authenticates or stores identity. Everything bel
## Next actions
- [ ] Send this as a **capability request / interface negotiation** to the netkingdom domain
(user-engine + net-kingdom owners) via the state hub.
- [x] Send this as a **capability request / interface negotiation** to the netkingdom domain
(user-engine + net-kingdom owners) via the state hub. — *sent 2026-06-13, capability
request `3136a9e2-0c1a-4b3f-8a02-ef47b27071fe` (auto-routed to `netkingdom`), inbox
message `46c60f17`.*
- [ ] Lock the **token claim contract** (A2A4) and the **membership read API** (B1B5) first
— they gate everything L2+.
- [ ] Define shard-wiki's **audit event schema** (C1) against user-engine's existing audit model.

View File

@@ -0,0 +1,91 @@
# Critical review (round 2) — CoreArchitectureBlueprint.md (hardened)
Date: 2026-06-15 · Reviewer: tegwick (with Claude) · Subject:
`spec/CoreArchitectureBlueprint.md` after **SHARD-WP-0005** (commit f21b7b5) · Feeds:
**SHARD-WP-0006**
A second hostile pass over the *hardened* blueprint. Round 1 found design bugs; this round
finds (a) self-consistency regressions the surgical hardening introduced, and (b) deeper
second-order gaps — some of which the hardening *sharpened*. Verdict: the architecture is now
substantially sound; what remains in §B/§C are hard distributed-systems/operational questions,
not design smells — except §A (a real regression) and three foundational gaps in §B.
---
## A. Self-consistency regressions introduced by surgical hardening
The 9 edits deepened §6§9 but did not propagate to the **overview surfaces**, so the document
now contradicts itself between its summary and its body (and readers trust the summary).
- **A-1 (real contradiction).** §4 still says "Addressing, **equivalence**, and transclusion
key on identity" — the exact conflation T2 fixed in §7.2 (equivalence keys on *content
fingerprint across distinct identities*). → **WP-0006 T1**
- **A-2.** §4 "Projection — typed on two axes" and §4 "Provenance envelope … every artifact
carries [full wrapper]" are stale vs T7's §8.4 (two-axis = extension point; trivial default)
and §7.3 (layered effective-vs-own). → **T1**
- **A-3.** §10 policy surface omits knobs the hardening added (freshness/staleness §8.8,
squash-compaction §8.1, conflict-resolution preset §8.6, tenant-partition) — yet §11 defines
`policy/` as "owns the §10 surface." The module contract points at a stale list. → **T1**
- **A-4 (cosmetic).** §3 diagram + §11 header still say "L4 rebuildable cache" / "15 spectra,"
advertising the pre-hardening model (§8.7 incremental-first; §6.5 orthogonal-core). → **T1**
Meta-point: surgical editing hardened the body but regressed whole-document coherence; v2
needs an **overview-reconciliation pass**.
## B. Foundational gaps (serious; some sharpened by the hardening)
- **B-1 — The journal is now a concurrent-write DB, but it's single-writer Git.** §8.6's
consistency model assumes "the journal is local Git, read-your-writes." L4 multi-tenant + the
L6 Orchestrator API imply a server; HA/scale implies *multiple* instances. Concurrent commits
of coordination-canonical state to one git journal = lock contention / merge races; Git is not
a concurrent-write store. Either single-writer-per-space (an unstated HA ceiling) or a real
concurrent coordination store with Git as an *export*. **T1 of WP-0005 worsened this** by
loading more canonical state into the journal. The keystone unanswered question. → **T2**
- **B-2 — Capability-as-data trusts self-reported profiles with no conformance check.** I-3 +
§6.5's degradation contract assume the profile tells the truth. A buggy adapter (claims
`merge=git/text`, corrupts; claims `notify`, never emits) silently poisons every degradation
decision. No **adapter conformance suite** (declared profile == observed behavior) exists.
Foundational for an architecture whose correctness rests on profile accuracy. → **T3**
- **B-3 — "Coordination-canonical state in the journal" has no representation design.** T1
relocated overlays/bindings/aliases/equivalence-sets/merges into "the journal" without saying
*how* Git stores structured mutable state. "All equivalences touching X" over a git-of-files
is O(scan) unless indexed — and an index is L4/derived. The new central concept is a black
box; resolve *with* B-1. → **T2**
- **B-4 — Incremental equivalence is under-specified/likely incorrect; I-2 only eventually
true.** §8.7 re-verifies a changed page's *new* candidate set but not the pairs it *leaves*
(a page exiting an LSH bucket can break an existing equivalence edge); the delta is not
additive. Deeper: incremental maintenance drifts from `f(canonical)`, so I-2 holds only
eventually, guaranteed solely by an expensive reconcile-against-rebuild. Needs a stated
verification mechanism (background checker / digest-vs-sampled-rebuild). → **T4**
## C. Real but second-tier (track as open problems O-8…O-11)
- **C-1 — Mechanism-over-policy → operator burden; no preset bundles.** ~7 knob families with
sub-modes and interactions; only authz (L0L4) bundles into personas. Need named bundles
("personal vault" / "team wiki" / "enterprise federation"). → **O-8 / T5**
- **C-2 — Tenant partitioning (I-13) vs shard sharing + lazy projection.** A shard in two roots
is cached twice → duplicate storage + double refresh on rate-limited backends. Shard
exclusive-to-one-root or shareable? Unresolved. → **O-9 / T5**
- **C-3 — Span-level authz + transclusion is an unmodeled leak path.** Authz is per
page/shard/tenant; transclusion crosses shards at span granularity → a page can leak a span
past its ACL (aggregation/inference). §7.3's ⊕ also stops being simple two-level inheritance
across a transclusion boundary. → **O-10 / T5**
- **C-4 — Union-under-unavailability undefined.** Freshness covers *stale*, nothing covers
*down*. The dead-shard read path (partial union? error? last-known?) is unspecified though
it's the commonest real failure. → **O-11 / T5**
## D. Recommended resolution (→ SHARD-WP-0006)
1. **§A reconciliation** (T1) — make the overview match the hardened body.
2. **Journal & coordination-state model** (T2) — settle single-vs-multi-writer and separate the
**coordination-state store** from the **content-history journal**. Likely resolution:
**event-sourced coordination** — an append-only *decision log* is the coordination-canonical
tier (git-addressable, I-6 preserved); the queryable current state (alias table, equivalence
set) is a *derived fold* of the log (disposable). Append-logs tolerate concurrency far better
than mutable-file Git; state a concurrency model. Resolves **B-1 + B-3** together.
3. **Adapter conformance suite** (T3) — make a passing conformance run part of the contract
(B-2): every adapter proves declared profile == observed behavior.
4. **Incremental correctness + I-2 verification** (T4) — fix the leaving-bucket re-verification
and propagation; add a background consistency-checker / derived-tier digest so I-2 is
verifiable, not merely asserted (B-4).
5. **Track §C** (T5) — O-8…O-11 with chosen direction + revisit trigger; close-out.

View File

@@ -0,0 +1,122 @@
# Critical review — CoreArchitectureBlueprint.md
Date: 2026-06-15 · Reviewer: tegwick (with Claude) · Subject:
`spec/CoreArchitectureBlueprint.md` @ commit **9b5b393** · Feeds: **SHARD-WP-0005**
A deliberately hostile review of the first whole-system architecture, to find where it
**breaks (correctness)**, **fails to scale**, and **could be more elegant/efficient** before
any implementation. Findings are prioritised; each is the input to a SHARD-WP-0005 task.
## Verdict in one line
The **layering and the dual narrow waist are sound and stay**. The **thesis is ~90% right**;
the missing 10% (curatorial / coordination-canonical state) breaks its clean story. There are
**two genuine bugs**, **two large unaddressed scaling risks**, and several **elegance/efficiency
debts** — all fixable without touching INTENT.
---
## A. The framing crack (fix resolves three issues)
**A-1 — Two buckets hide a third.** The thesis "canonical at the edges, derived in the middle"
omits **born-in-the-middle-but-canonical** state: overlays that are the local truth against a
read-only shard (Flow C), manual **curator equivalence bindings**, alias tables, merge
decisions. These encode human judgment or local-only content and **cannot be rebuilt** from
shards+journal.
**Contradiction:** I-2 declares L4 rebuildable, yet §8.4 puts "alias table, curator binding"
in L4. You cannot rebuild a curator's manual binding.
**Fix:** three states — **sharded-canonical**, **coordination-canonical** (journal: overlays,
bindings, aliases, merges — durable, born in the middle), **derived-disposable** (union graph,
indexes, projections). Re-frame §1 as **canonical (sharded + coordination) vs derived
(disposable)**; `derived = f(canonical)` then becomes actually true. → **T1**
---
## B. Where it breaks (correctness)
**B-1 — Identity conflated with content-fingerprint (BUG).** §7.2 derives page identity from
content fingerprint. That makes **editing a page change its identity**, breaking every
reference. Fingerprints identify *versions/equivalence*, not *identity*. Page identity must be
a **stable handle (uid)** surviving edits; fingerprints belong to the **equivalence** mechanism
(§8.4). One word, two concepts, wrong implementation for the stable one. → **T2**
**B-2 — No concurrency/consistency model.** Concurrent overlays on one page, overlay applied
after source drift, journal-commit vs shard-native-write ordering — all undefined. Conflict
handling is deferred to "policy presets," but **conflict *detection + representation* is core
mechanism**; only *resolution* is policy. The union's consistency guarantee is unstated
(eventually-consistent? read-your-writes? causal-via-journal?). → **T3**
**B-3 — Persisted union cache + multi-tenant = leak surface.** §13 recommends a persisted L4
cache; §9 protects content by *read-time* filtering on the provenance envelope. A persisted
cross-tenant union cache guarded only by read-time filtering is an L4 attack surface. Tension
between I-2 (persisted rebuildable cache), scale, and L5 isolation is unacknowledged. → **T8**
---
## C. Where it fails to scale
**C-1 — Equivalence detection is O(N²), no indexing/incremental story.** Fingerprint /
span-set-overlap across all pages of all shards is combinatorial (10 shards × 100k pages ≈
10¹² comparisons). No blocking/LSH/indexing, no incremental maintenance. Biggest scaling
hazard in the document. → **T4**
**C-2 — "Rebuildable cache" collides with the operational-envelope axis.** A byte-exact
rebuild requires reading *every page of every shard*, including rate-limited/paginated
external APIs (Notion) and irreducibly-live sources — hours-to-days. I-2 contradicts axis-10.
**Incremental, change-driven maintenance must be primary** (notify→delta), rebuild a rare
fallback. Cache invalidation — the actual hard problem — is named once and never designed. →
**T4, T5**
**C-3 — Unbounded history at open L0 = DoS/perf.** "Every write a commit" + "open for all" ⇒
the git journal grows without bound under bots/vandalism and git degrades on huge histories.
"History is the floor" has an unacknowledged cost: packing, compaction, per-shard offload. →
**T8**
---
## D. Elegance / efficiency debts
**D-1 — The 15 spectra assert a clean degradation function never demonstrated.** Either most
axes are irrelevant to most ops (then the 15-D profile is ceremony), or behavior depends on
several axes *jointly* (then "no per-backend code" becomes a sprawling axis-interaction matrix
— the flat-checklist problem in higher dimensions). And the axes **aren't orthogonal**
(git-native history ⟺ git-IS-store ⟺ git/text merge; encrypted opacity ⟹ query/translation
collapse). Model a **smaller orthogonal core** + **derived/implied** positions, and state the
**axis-interaction subset** the degradation logic truly uses. → **T6**
**D-2 — Provenance envelope isn't inherited; it'll dwarf the content.** Per-span envelopes at
block granularity = 10k near-identical envelopes for a 10k-block graph. The doc already
invented the right pattern for Trilium ("effective-vs-own with per-attribute provenance") and
failed to apply it to its own envelope. Make provenance **layered (page envelope + span
deltas)**. → **T7**
**D-3 — Projection machinery over-fit to the exotic tail.** Two-axis model + three facets +
view registry exist mostly for UC-83/84 (2 of 84 UCs); the 95% case (markdown in git) pays the
weight. Make the **common case trivial** (default = plain lazy replication) and
derivation/liveness an **extension point**, not a taxonomy every projection instantiates. →
**T7**
**D-4 — Cross-cutting rails are the highest-coupling components, presented as clean.**
`provenance/` and capability types are imported by every layer (god-modules); an envelope
change ripples everywhere. And **policy has no module** (§10 enumerates it; §11 omits it)
despite being consulted by L3/L4/L5. Give policy a home; pin the rails behind stable narrow
interfaces. → **T7**
---
## E. What explicitly stays
- The 6-layer model + the dual narrow waist (adapter contract / page model).
- Capability-as-data (I-3), union-without-erasure (I-4), overlay-before-mutation (I-5),
Git-addressable coordination (I-6), mechanism-over-policy (I-7), graceful degradation (I-8).
- The federation-model taxonomy and the auth ladder (ArchitectureBlueprint.md).
## F. Disposition
Some findings are **solvable now** (A-1, B-1, D-2, D-3, D-4, C-3); some are **partially open**
and should be tracked honestly rather than pretend-solved (B-2 consistency model: pick a
guarantee; C-1 equivalence-at-scale: pick a blocking strategy; D-1 axis interactions: enumerate
the real subset). SHARD-WP-0005 closes the solvable ones and records the open ones in a new
"Known scaling risks & open problems" section of the blueprint. → **T9**

View File

@@ -0,0 +1,45 @@
# reuse-surface contributions — shard-wiki (SHARD-WP-0013 T3)
Date: 2026-06-15 · From: shard-wiki (whynot) · To: reuse-surface (helix_forge) · Tracked list
of (A) capabilities shard-wiki registered, (B) **gaps** shard-wiki proposes the reuse surface
add, and (C) capabilities shard-wiki will **consume** rather than rebuild. Communicated to the
reuse-surface agent via state-hub `send_message`.
## A. Registered (T1 + T3) — 8 entries
`capability.wiki.{shard-orchestration, adapter-contract, page-model, coordination-journal,
overlay, federation-models, engine-typed-extensions, derived-views}` — committed in
reuse-surface, `validate` ok (20 entries total).
## B. Proposed gaps (cross-cutting; not shard-wiki-internal) → reuse-surface should own/define
- **G1 — `capability.platform.typed-extension-framework`** *(suggested)*
A reusable pattern: a **small core + a stringent typed-extension framework** where extensions
declare typed contracts, compose, and are **activated per context**. shard-wiki's wiki engine
(`capability.wiki.engine-typed-extensions`) is one instance, but the *pattern* is
cross-domain (any HelixForge capability platform). Evidence: shard-wiki UseCaseCatalog
"Capability structure" layer (core + 10 typed extensions + conflict-mediation map).
Suggested owner: helix_forge / reuse-surface. Relation: would `generalize`
`capability.wiki.engine-typed-extensions`.
- **G2 — `capability.content.translation-fidelity`** *(suggested)*
Lossless/lossy **content translation with an explicit fidelity report** (what round-trips
cleanly vs degrades, non-mappable elements preserved as provenance). Reusable well beyond
wiki (any format-bridging consumer). Evidence: shard-wiki TSD §A.6, UC-42/UC-59.
## C. Consumptions (reuse, do not rebuild)
- **`capability.feature-control.evaluate`** (helix_forge/feature-control) → shard-wiki's
**per-shard extension/feature activation** (the "activate only what you need" mechanism).
Already recorded as a relation on `capability.wiki.engine-typed-extensions`.
- **`capability.authorization.policy-evaluate`** (flex-auth) → shard-wiki's **X-AUTHZ** policy
decisions. shard-wiki owns the authz *model* (authz-in-core) but can reuse this evaluation
engine rather than building one.
- **`capability.statehub.{progress-log, workstream-coordinate}`** → already in use for
coordination across this work.
## Status
Gaps G1/G2 are **suggestions** to the reuse-surface owner (not unilateral registrations, since
they are cross-cutting, not shard-wiki-internal). Consumptions are recorded for the engine
architecture (T5) so it reuses rather than reinvents.

View File

@@ -1,8 +1,19 @@
# history/
Archived material that is no longer needed for daily work but should be kept.
Archived material and the project's **meta-history**: finished/canceled workplans kept for
the record, plus durable **reviews, critical assessments, and decision records** — the
reasoning behind the specs, captured at a point in time.
Use a `yymmdd-` prefix when archiving files or directories. Content here is
**out of scope** for regular tasks — consult only for research or diagnostics.
Use a `yymmdd-` prefix. Archived material is **out of scope** for regular tasks (consult only
for research or diagnostics); assessment/review records are point-in-time and may seed active
workplans, but are not edited after the fact — supersede with a new dated record and link back.
Finished or canceled workplans from `workplans/` are archived here.
Distinct from the **coordination journal** (a runtime Git-backed record of *content* change
flows inside an information space, an INTENT domain concept); `history/` is the *project's own*
design evolution.
| Date | Record | Subject |
|------|--------|---------|
| 2026-06-15 | `260615-core-architecture-blueprint-review.md` | Critical review of `spec/CoreArchitectureBlueprint.md` (commit 9b5b393); inputs to `SHARD-WP-0005` |
| 2026-06-15 | `260615-core-architecture-blueprint-review-2.md` | Round-2 review of the hardened blueprint (post-`SHARD-WP-0005`, f21b7b5); inputs to `SHARD-WP-0006` |
| 2026-06-15 | `260615-reuse-surface-contributions.md` | shard-wiki's reuse-surface registrations, proposed gaps (G1/G2), and consumptions (`SHARD-WP-0013` T3) |

View File

@@ -36,6 +36,11 @@ pythonpath = ["src"]
branch = true
source = ["shard_wiki"]
[tool.coverage.report]
show_missing = true
# Quality floor for `pytest --cov` / `coverage report` (not forced on a bare `pytest` run).
fail_under = 90
[tool.ruff]
src = ["src", "tests"]
target-version = "py311"

12
registry/README.md Normal file
View File

@@ -0,0 +1,12 @@
# Capability Registry
Markdown-first capability index for federation and reuse planning.
## Authoring
1. Copy a capability entry template (see reuse-surface `templates/capability-entry.template.md`).
2. Add the row to `indexes/capabilities.yaml`.
3. Run `reuse-surface validate` from a checkout with the CLI installed.
4. Merge to `main` and verify publish with `reuse-surface establish --publish-check`.
Federation contract: reuse-surface `docs/RegistryFederation.md`.

View File

@@ -0,0 +1,103 @@
---
id: capability.wiki.adapter-contract
name: Capability-Aware Shard Adapter Contract
summary: A versioned backend interface where each binding declares a verified capability profile (positions on capability spectra), so federation ops degrade by capability.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, adapter, capability, contract, conformance, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Fifteen capability spectra with an orthogonal core + implication rules, plus
a normative contract spec (TSD Section A); derived from a ~23-system synthesis.
availability:
current: A2
target: A5
confidence: medium
rationale: >
AdapterContract + a read/write FolderAdapter + a conformance suite that
verifies declared profile == observed behaviour exist as a source module.
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- versioned interface with declared, conformance-verified capability profiles
- one concrete adapter (file-store) passes the conformance suite
broken_expectations:
- only one substrate implemented (git-IS-store, REST, CRDT adapters planned)
out_of_scope_expectations:
- hosting backends
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- single adapter implemented so far
discovery:
intent: >
Mediate heterogeneity at one narrow waist: a backend participates by implementing a
versioned interface and declaring a verified position on each capability spectrum.
includes:
- capability profile as data (orthogonal-core spectra + implied positions)
- operation verbs (read/write/diff/merge/notify/.../derive-projection/execute)
- a conformance suite (profiles verified, not self-asserted)
excludes:
- assuming uniform backend capabilities
use_cases:
- "shard-wiki UseCaseCatalog UC-34..UC-43, UC-50, UC-57, UC-60..UC-69 (shard attachment & adapter binding)"
availability:
current_level: A2
target_level: A5
current_artifacts:
- "shard-wiki/src/shard_wiki/adapters/"
consumption_modes:
- source module
relations:
depends_on:
- capability.wiki.page-model
supports:
- capability.wiki.shard-orchestration
evidence:
documentation:
- "shard-wiki/spec/TechnicalSpecificationDocument.md (Section A)"
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 6)"
tests:
- "shard-wiki/tests/test_folder_adapter.py"
- "shard-wiki/tests/test_conformance.py"
consumer_guidance:
recommended_for:
- exposing any page store as a capability-described, conformance-checked shard
not_recommended_for:
- backends that cannot honestly describe their capabilities
known_limitations:
- reference implementation covers the file-store substrate only so far
---
# Capability-Aware Shard Adapter Contract
The bottom narrow waist of shard-wiki: a versioned interface plus a **verified** capability
profile per binding. Core logic is written once against capabilities (not per-backend), and
the conformance suite rejects profiles whose declared abilities don't match observed behaviour.
## Assessment notes
### Discovery
Fifteen spectra reduced to an orthogonal core with implication rules (CoreArchitectureBlueprint
Section 6.5); normative in TSD Section A.
### Availability
`adapters/` ships the contract, a folder adapter, and `assert_conformant`.

View File

@@ -0,0 +1,103 @@
---
id: capability.wiki.coordination-journal
name: Event-Sourced Coordination Journal
summary: An append-only, totally-ordered-per-space decision log (overlays, bindings, aliases, merges, forks) whose current state is a derived fold; git-addressable history.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, event-sourcing, coordination, git, journal, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Keystone resolved across two architecture reviews: coordination-canonical state
as an append-only decision log with a per-space append authority; current state
is a derived fold (derived = f(log)).
availability:
current: A2
target: A4
confidence: medium
rationale: >
In-memory DecisionLog + fold work as a source module; the git-backed store with a
per-space lease (the production backing) is planned.
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- append-only, totally-ordered-per-space log with read-your-writes
- derived fold to aliases + transitively-merged equivalence groups
broken_expectations:
- git-backed storage and per-space lease/append-authority not yet implemented
out_of_scope_expectations:
- general-purpose event bus
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- in-memory backing only; cross-process durability pending
discovery:
intent: >
Make coordination-canonical decisions durable and git-addressable as events, with the
queryable current state always recomputable by replay.
includes:
- append-only decision log, totally ordered per information space
- derived fold to current coordination state (aliases, equivalence groups, overlays)
- per-space append authority (concurrency model)
excludes:
- storing derived/disposable union state
use_cases:
- "shard-wiki UseCaseCatalog UC-29, UC-33 (history, attribution, coordination journal)"
availability:
current_level: A2
target_level: A4
current_artifacts:
- "shard-wiki/src/shard_wiki/coordination/decision_log.py"
target_artifacts:
- git-backed log store with per-space lease
consumption_modes:
- source module
relations:
supports:
- capability.wiki.shard-orchestration
- capability.wiki.overlay
evidence:
documentation:
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 8.1)"
tests:
- "shard-wiki/tests/test_decision_log.py"
consumer_guidance:
recommended_for:
- durable, replayable, git-addressable coordination state for a federated space
not_recommended_for:
- high-frequency general event streaming
known_limitations:
- production git backing + lease are still on the roadmap (SHARD-WP-0009)
---
# Event-Sourced Coordination Journal
The keystone: coordination-canonical state (overlays, equivalence bindings, aliases, merges,
forks) is an append-only **decision log**, totally ordered per information space; the queryable
current state is a derived **fold** of the log (`derived = f(log)`). The log is git-addressable,
giving history/patch/review/backup for coordination decisions for free.
## Assessment notes
### Discovery
Resolved across the round-1/round-2 architecture reviews (CoreArchitectureBlueprint Section 8.1).
### Availability
`decision_log.py` ships an in-memory, totally-ordered log + fold; git+lease backing is planned.

View File

@@ -0,0 +1,87 @@
---
id: capability.wiki.derived-views
name: Wiki Derived Views
summary: Recomputable views over a wiki union — BackLinks, RecentChanges, AllPages, SiteMap, and (delegate-or-derive) Search — carrying provenance.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, derived-views, backlinks, recentchanges, search, shard-wiki]
maturity:
discovery:
current: D3
target: D5
confidence: medium
rationale: >
Core-vs-adapter classification and behaviours are decided (FederationRequirements ADR-03);
implementation is planned (SHARD-WP-0010), not built.
availability:
current: A0
target: A4
confidence: low
rationale: >
Designed; no implementation yet. Informational/planning reuse only today.
external_evidence:
completeness:
level: C0
name: Absent
confidence: low
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations: []
broken_expectations:
- no derived view is implemented yet
out_of_scope_expectations:
- presentation / rendering of views
reliability:
level: R0
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- planning-stage
discovery:
intent: >
Provide recomputable, provenance-carrying views over the union (link graph, change feed,
enumeration, search) without introducing canonical state.
includes:
- BackLinks (link graph), RecentChanges (journal + shard signals), AllPages, SiteMap
- Search as delegate-to-native-or-derive-index
excludes:
- view presentation / UI
use_cases:
- "shard-wiki UseCaseCatalog UC-17..UC-21, UC-63"
availability:
current_level: A0
target_level: A4
current_artifacts:
- "shard-wiki/workplans/SHARD-WP-0010-derived-views.md"
consumption_modes:
- informational
relations:
depends_on:
- capability.wiki.shard-orchestration
- capability.wiki.page-model
related_to:
- capability.wiki.engine-typed-extensions
evidence:
documentation:
- "shard-wiki/spec/FederationRequirements.md (ADR-03)"
consumer_guidance:
recommended_for:
- planning derived navigation/discovery over a federated wiki union
not_recommended_for:
- implementation reuse today (planning-stage)
known_limitations:
- not implemented; Search ranking policy undecided
---
# Wiki Derived Views
Recomputable views over the union (BackLinks, RecentChanges, AllPages, SiteMap, Search). All
are derived/disposable (no canonical state) and carry provenance; Search is delegate-to-native
where a shard's query capability allows, else a derived index. Planned in SHARD-WP-0010.

View File

@@ -0,0 +1,115 @@
---
id: capability.wiki.engine-typed-extensions
name: Wiki Engine with Typed Extensions
summary: A small-core wiki engine realizing a stringent typed-extension framework that addresses all wiki use cases and lets each shard activate only the features it needs.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, engine, typed-extensions, feature-activation, shard-wiki]
maturity:
discovery:
current: D3
target: D5
confidence: medium
rationale: >
Architecture authored (shard-wiki/spec/WikiEngineCoreArchitecture.md): small page-store
kernel + typed-extension framework, per-shard activation, engine-as-canonical-mode-shard,
and a conflict-mediation realization are explored. Detailed extension SDK/ABI and the API
protocol remain (so D3 Explored, not yet D4/D5).
availability:
current: A0
target: A4
confidence: low
rationale: >
Planned. No engine kernel or extensions exist yet; informational/planning reuse only.
external_evidence:
completeness:
level: C0
name: Absent
confidence: low
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations: []
broken_expectations:
- engine core and typed-extension mechanism not yet designed in detail
out_of_scope_expectations:
- replacing other wiki engines or mandating one implementation
reliability:
level: R0
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- planning-stage capability
discovery:
intent: >
Provide shard-wiki's reference first-party shard backend: a small core + a stringent
typed-extension framework covering all collected use cases, mediating conflicting
requirements into an integrated whole, with per-shard activation (only what you need).
includes:
- a minimal engine kernel (page lifecycle, storage via the adapter contract, the typing mechanism)
- typed extensions that declare contracts and compose
- per-shard feature activation
excludes:
- replacing or mandating other wiki engines (it is one shard type among many)
- a single canonical implementation for all wikis
use_cases:
- "shard-wiki UseCaseCatalog UC-08..UC-25 and the full catalog (the engine must cover all)"
availability:
current_level: A0
target_level: A4
current_artifacts:
- "shard-wiki/workplans/SHARD-WP-0013-wiki-engine-prep.md"
- "shard-wiki/spec/WikiEngineCoreArchitecture.md"
consumption_modes:
- informational
relations:
depends_on:
- capability.wiki.adapter-contract
- capability.wiki.page-model
related_to:
- capability.feature-control.evaluate
- capability.authorization.policy-evaluate
evidence:
documentation:
- "shard-wiki/workplans/SHARD-WP-0013-wiki-engine-prep.md"
consumer_guidance:
recommended_for:
- planning a composable, feature-activatable native wiki engine
not_recommended_for:
- implementation reuse today (planning-stage)
known_limitations:
- architecture authored; extension SDK/ABI + API protocol still to design; not yet built
promotion_history:
- date: "2026-06-15"
dimension: discovery
from: D2
to: D3
rationale: WikiEngineCoreArchitecture.md authored (kernel + typed-extension framework explored); INTENT amendment ratified.
author: shard-wiki
---
# Wiki Engine with Typed Extensions
shard-wiki's planned reference first-party shard backend — a *canonical-mode shard* it
implements natively: a small core plus a stringent typed-extension framework addressing all
collected use cases, mediating conflicting requirements into a consistent whole, with per-shard
activation (activate only what you need). It is one shard type among many — not a replacement
for other engines. Per-shard activation is a candidate consumer of
`capability.feature-control.evaluate`.
## Assessment notes
### Discovery
Architecture authored: `shard-wiki/spec/WikiEngineCoreArchitecture.md` (small kernel +
typed-extension framework; engine = canonical-mode shard). INTENT amendment ratified
(2026-06-15, decision 84ffdb48). Extension SDK/ABI + API protocol are the next deliverables.
### Availability
Planning-stage; informational reuse only.

View File

@@ -0,0 +1,97 @@
---
id: capability.wiki.federation-models
name: Selectable Federation-Model Taxonomy
summary: Federation as a plural, composable coordination axis (fork+journal, VCS-replication+ping, query-time graph-join, feed, activity-streams, engine-mirror) selected per space.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, federation, taxonomy, composable, shard-wiki]
maturity:
discovery:
current: D4
target: D6
confidence: high
rationale: >
A six-model taxonomy distilled from a ~23-system synthesis, each model anchored in a
real system, with capability prerequisites and per-space/per-shard composition rules.
availability:
current: A0
target: A4
confidence: low
rationale: >
Designed and specified (FederationArchitecture T17) but not implemented; informational
reuse only today.
external_evidence:
completeness:
level: C1
name: Sparse
confidence: low
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- the model taxonomy and selection/composition rules are documented
broken_expectations:
- no federation transport is implemented yet
out_of_scope_expectations:
- mandating a single federation mechanism
reliability:
level: R0
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- design-stage; no runtime evidence
discovery:
intent: >
Treat federation as selectable and composable rather than one mechanism, so each space
picks fork+journal, VCS-replication, query-join, feed, activity-streams, or engine-mirror.
includes:
- the six federation models + their capability floors
- per-space selection and per-shard composition
excludes:
- imposing one homogeneous federation network
use_cases:
- "shard-wiki UseCaseCatalog UC-26, UC-31, UC-33, UC-71, UC-72, UC-74, UC-79"
availability:
current_level: A0
target_level: A4
current_artifacts:
- "shard-wiki/spec/FederationArchitecture.md (T17)"
consumption_modes:
- informational
relations:
depends_on:
- capability.wiki.shard-orchestration
- capability.wiki.coordination-journal
evidence:
documentation:
- "shard-wiki/spec/FederationArchitecture.md"
- "shard-wiki/research/260614-shard-spectrum-synthesis/findings.md"
consumer_guidance:
recommended_for:
- planning a federation strategy that mixes models per source
not_recommended_for:
- implementation reuse today (design-stage)
known_limitations:
- no transport implemented; informational planning reuse only
---
# Selectable Federation-Model Taxonomy
Federation is plural and composable: fork+journal (Federated Wiki), VCS-replication+ping
(ikiwiki), query-time graph-join (Wikibase SERVICE), feed aggregation, activity streams
(ActivityPub), and engine-mirror (Wiki.js). A space selects a model and composes per shard;
the default is fork+journal over git. Design-stage capability — strong for planning reuse.
## Assessment notes
### Discovery
FederationArchitecture T17, distilled from the shard-spectrum synthesis (v3).
### Availability
Specified, not implemented — informational reuse only.

View File

@@ -0,0 +1,102 @@
---
id: capability.wiki.overlay
name: Overlay-Before-Mutation Write Path
summary: Non-destructive edits (draft -> patch -> apply-under-drift) that let read-only, rate-limited, or lossy backends be edited safely without silent remote mutation.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, overlay, patch, write-path, conflict, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Overlay lifecycle and apply-under-drift semantics are specified (ADR-05, blueprint
Section 8.6) and implemented as a single principled write path.
availability:
current: A2
target: A4
confidence: medium
rationale: >
OverlayEngine (draft/patch/apply), writable adapter, and InformationSpace.edit
exist as a source module; three-way merge is not (refuse-on-drift only).
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- draft -> patch -> apply with fast-forward / refuse-on-drift / keep-draft outcomes
- no silent remote mutation; overlay_state surfaced in provenance
broken_expectations:
- three-way / auto merge not implemented (refuse-on-conflict only)
out_of_scope_expectations:
- federation propagation of applied overlays
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- early implementation; conflict handling is detect-and-refuse only
discovery:
intent: >
Make any sub-write-through backend editable safely: an edit is an overlay first, applied
only on explicit intent and only when the source has not drifted.
includes:
- overlay drafts recorded as coordination-canonical events
- patch rendering (unified diff)
- apply-under-drift (fast-forward / refuse / keep-draft)
excludes:
- destructive write without drift check
use_cases:
- "shard-wiki UseCaseCatalog UC-04, UC-26, UC-29 (remix primitives, overlay)"
availability:
current_level: A2
target_level: A4
current_artifacts:
- "shard-wiki/src/shard_wiki/coordination/overlay.py"
- "shard-wiki/src/shard_wiki/coordination/patch.py"
consumption_modes:
- source module
relations:
depends_on:
- capability.wiki.coordination-journal
- capability.wiki.adapter-contract
evidence:
documentation:
- "shard-wiki/spec/FederationRequirements.md (ADR-05)"
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 8.2, 8.6)"
tests:
- "shard-wiki/tests/test_apply.py"
- "shard-wiki/tests/test_write_path_integration.py"
consumer_guidance:
recommended_for:
- safe editing over read-only / rate-limited / lossy backends
not_recommended_for:
- workflows needing automatic conflict resolution today
known_limitations:
- merge is detect-and-refuse; three-way merge is future work
---
# Overlay-Before-Mutation Write Path
One principled write path: every edit drafts an overlay (a coordination-canonical event),
renders as a patch, and applies under drift checks — fast-forwarding a writable target,
keeping a local draft on a read-only target, and refusing (never clobbering) on external drift.
## Assessment notes
### Discovery
Specified in FederationRequirements ADR-05 and CoreArchitectureBlueprint Section 8.2/8.6.
### Availability
`overlay.py` + `patch.py` + `InformationSpace.edit` ship the path; built in SHARD-WP-0008.

View File

@@ -0,0 +1,104 @@
---
id: capability.wiki.page-model
name: Backend-Neutral Wiki Page Model
summary: A Markdown-first but stretchable page model with stable identity separate from placement and layered provenance, spanning prose to typed-graph and computational shapes.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, page-model, identity, provenance, markdown, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Page shapes (prose, typed records, typed-graph, inline-embedded, non-Markdown,
and four computational shapes) plus identity != placement and layered provenance
are specified and grounded in the dive research.
availability:
current: A2
target: A5
confidence: medium
rationale: >
Identity/Placement/Span/Page and layered ProvenanceEnvelope exist as a source
module; richer shapes (typed-graph, notebook) are modeled but not all built.
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- stable identity distinct from placement and from content fingerprint
- layered (effective-vs-own) provenance with near-zero per-span cost
broken_expectations:
- non-prose shapes (typed-graph, notebook, inline-embedded) not fully realized
out_of_scope_expectations:
- rendering / presentation
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- prose shape is the only exercised path so far
discovery:
intent: >
One backend-neutral lingua franca every consumer sees; every shape reduces to
(content|source, structure, provenance envelope, optional derivation rule).
includes:
- page identity (stable handle) vs placement (N paths/shards) vs equivalence (fingerprint)
- layered provenance envelope (page + span deltas)
- page-shape taxonomy incl. computational shapes
excludes:
- deriving identity from content (a fingerprint identifies a version, not a page)
use_cases:
- "shard-wiki UseCaseCatalog UC-34, UC-39, UC-44..UC-49, UC-55, UC-73, UC-83, UC-84"
availability:
current_level: A2
target_level: A5
current_artifacts:
- "shard-wiki/src/shard_wiki/model/"
- "shard-wiki/src/shard_wiki/provenance/"
consumption_modes:
- source module
relations:
supports:
- capability.wiki.adapter-contract
- capability.wiki.shard-orchestration
evidence:
documentation:
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 7)"
- "shard-wiki/spec/FederationRequirements.md (ADR-02, ADR-04)"
tests:
- "shard-wiki/tests/test_model.py"
- "shard-wiki/tests/test_provenance.py"
consumer_guidance:
recommended_for:
- a portable, provenance-carrying representation of wiki pages across backends
not_recommended_for:
- cases needing a single canonical path per page (use identity, not path)
known_limitations:
- non-prose shapes specified ahead of implementation
---
# Backend-Neutral Wiki Page Model
The top narrow waist: a Markdown-first model that stretches to typed records, typed-graph
statements, inline-embedded objects, non-Markdown assets, and computational shapes. Identity
is a stable handle; placement and equivalence are separate mechanisms; provenance is layered
(effective = page envelope + span delta).
## Assessment notes
### Discovery
Specified in CoreArchitectureBlueprint Section 7 and FederationRequirements ADR-02/04.
### Availability
`model/` + `provenance/` ship the prose path and the layered envelope today.

View File

@@ -0,0 +1,114 @@
---
id: capability.wiki.shard-orchestration
name: Wiki Shard Orchestration
summary: Present a union of pages across heterogeneous wiki-shaped shards while preserving each shard's provenance, capabilities, and history.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, federation, orchestration, union, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Grounded in 84 documented use cases and a twice-reviewed whole-system
architecture (CoreArchitectureBlueprint) derived from ~23 prior-art systems.
availability:
current: A2
target: A5
confidence: medium
rationale: >
InformationSpace orchestrator (attach -> resolve -> read, chorus on
ambiguity) works as a Python source module; network API and incremental
union are planned.
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- attach folder shards and read a union page with layered provenance
- chorus presentation of equivalent-but-divergent pages (union without erasure)
broken_expectations:
- incremental union maintenance and equivalence index not yet built
- write-through federation transports not yet built
out_of_scope_expectations:
- hosting or replacing the underlying wiki engines
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- early implementation; 64 tests but no production exposure
discovery:
intent: >
Let independently stored, differently implemented wikis behave as one
coherent, versionable, inspectable information space without homogenizing them.
includes:
- union resolution across shards (identity-keyed)
- chorus / designated-canonical presentation of equivalent pages
- lazy replication projection of remote content with freshness
excludes:
- implementing a backend wiki engine (see capability.wiki.engine-typed-extensions)
- silent remote mutation
assumptions:
- canonical truth lives in shards + a git coordination journal; the union is derived
use_cases:
- "shard-wiki UseCaseCatalog UC-01..UC-07, UC-26..UC-33 (information space, federation, coordination)"
availability:
current_level: A2
target_level: A5
current_artifacts:
- "shard-wiki/src/shard_wiki/union/"
- "shard-wiki/src/shard_wiki/space.py"
target_artifacts:
- orchestrator network API
consumption_modes:
- source module
relations:
depends_on:
- capability.wiki.adapter-contract
- capability.wiki.page-model
- capability.wiki.coordination-journal
supports:
- capability.wiki.federation-models
evidence:
documentation:
- "shard-wiki/spec/CoreArchitectureBlueprint.md"
- "shard-wiki/spec/FederationArchitecture.md"
tests:
- "shard-wiki/tests/test_union.py"
- "shard-wiki/tests/test_integration.py"
consumer_guidance:
recommended_for:
- composing multiple Markdown/wiki stores into one provenance-preserving view
not_recommended_for:
- replacing a single wiki engine
known_limitations:
- resolution is recompute-on-read until the incremental tier lands
---
# Wiki Shard Orchestration
shard-wiki's core capability: orchestrate wiki-shaped content across heterogeneous *shards*
as a union of pages, preserving provenance, capabilities, and history per shard. Canonical
truth stays at the edges (shards + the git coordination journal); the union is a derived,
recomputable view (orchestrator, not engine).
## Assessment notes
### Discovery
Grounded by `UseCaseCatalog.md` (84 UCs) and the hardened `CoreArchitectureBlueprint.md`.
### Availability
`InformationSpace` provides attach/resolve/read today (source module); a network API is the
target availability step.

View File

@@ -0,0 +1,145 @@
version: 1
updated: '2026-06-16'
domain: helix_forge
capabilities:
- id: capability.wiki.shard-orchestration
name: Wiki Shard Orchestration
summary: Present a union of pages across heterogeneous wiki-shaped shards while
preserving each shard's provenance, capabilities, and history.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.shard-orchestration.md
tags:
- wiki
- federation
- orchestration
- union
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.adapter-contract
name: Capability-Aware Shard Adapter Contract
summary: A versioned backend interface where each binding declares a verified capability
profile, so federation ops degrade by capability.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.adapter-contract.md
tags:
- wiki
- adapter
- capability
- contract
- conformance
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.page-model
name: Backend-Neutral Wiki Page Model
summary: A Markdown-first but stretchable page model with stable identity separate
from placement and layered provenance.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.page-model.md
tags:
- wiki
- page-model
- identity
- provenance
- markdown
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.coordination-journal
name: Event-Sourced Coordination Journal
summary: An append-only, totally-ordered-per-space decision log whose current state
is a derived fold; git-addressable history.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.coordination-journal.md
tags:
- wiki
- event-sourcing
- coordination
- git
- journal
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.overlay
name: Overlay-Before-Mutation Write Path
summary: Non-destructive edits (draft -> patch -> apply-under-drift) that let read-only
or limited backends be edited safely without silent remote mutation.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.overlay.md
tags:
- wiki
- overlay
- patch
- write-path
- conflict
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.federation-models
name: Selectable Federation-Model Taxonomy
summary: Federation as a plural, composable coordination axis (fork+journal, VCS-replication,
query-join, feed, activity-streams, engine-mirror) selected per space.
vector: D4 / A0 / C1 / R0
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.federation-models.md
tags:
- wiki
- federation
- taxonomy
- composable
- shard-wiki
consumption_modes:
- informational
- id: capability.wiki.engine-typed-extensions
name: Wiki Engine with Typed Extensions
summary: A small-core wiki engine realizing a typed-extension framework that addresses
all wiki use cases and lets each shard activate only the features it needs.
vector: D3 / A0 / C0 / R0
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.engine-typed-extensions.md
tags:
- wiki
- engine
- typed-extensions
- feature-activation
- shard-wiki
consumption_modes:
- informational
- id: capability.wiki.derived-views
name: Wiki Derived Views
summary: Recomputable views over a wiki union — BackLinks, RecentChanges, AllPages,
SiteMap, and (delegate-or-derive) Search — carrying provenance.
vector: D3 / A0 / C0 / R0
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.derived-views.md
tags:
- wiki
- derived-views
- backlinks
- recentchanges
- search
- shard-wiki
consumption_modes:
- informational

View File

@@ -0,0 +1,23 @@
# 260608 — c2 wiki origins (Ward Cunningham)
Date: 2026-06-08
## What this is
Research into the beginnings of the wiki movement through Ward Cunningham,
the Portland Pattern Repository, and the original **WikiWikiWeb** at c2.com.
Focus: popular terms, social conventions, and documented use cases from and
about the c2 wiki. **Federation is explicitly out of scope** for this
exploration (separate research planned).
## Contents
| Path | Role |
|------|------|
| `findings.md` | Terms glossary, use-case catalog, design principles, sources |
## Status
Initial exploration complete. Findings may inform `spec/` and future
workplans; not yet promoted to specification.

View File

@@ -0,0 +1,327 @@
# Findings — Ward Cunningham, c2 wiki, and wiki movement origins
Date: 2026-06-08 · Status: research draft
Scope: beginnings of the wiki movement via the **WikiWikiWeb** (c2.com) and
its creator **Ward Cunningham**. Emphasis on terms and use cases documented
in and about the c2 wiki. **Federation concepts are out of scope here.**
---
## 1. Historical anchor
| Fact | Detail |
|------|--------|
| First wiki | WikiWikiWeb — launched **25 March 1995** on c2.com |
| Creator | Ward Cunningham (Cunningham & Cunningham, Portland, Oregon) |
| Companion site | [Portland Pattern Repository](http://c2.com/ppr/) (PPR) |
| PPR motto | **"People, Projects & Patterns"** |
| Original software | Perl CGI script named `wiki`; later called **WikiBase** |
| Intellectual roots | HyperCard stacks (late 1980s); pattern languages (Cunningham & Beck, OOPSLA 1987, inspired by Christopher Alexander) |
| Name origin | Hawaiian *wiki* = "quick"; *wiki wiki* = "very quick" — from Honolulu airport Wiki Wiki Shuttle |
| Intended name alternative | "QuickWeb" rejected in favor of **WikiWikiWeb** as more fun to say |
| Capital-W convention | **Wiki** (capital W) = this original site; lowercase *wiki* = the technology genre |
Ward's stated purpose (WikiHistory, 2002): make exchange of ideas between
programmers easier. The wiki was an **automated supplement** to the Portland
Pattern Repository, not a replacement encyclopedia.
By 2015 the site went read-only after vandalism; Ward migrated toward
Federated Wiki. That later history is **not** covered here.
---
## 2. What c2 said it was for (primary use cases)
From **WelcomeVisitors**, **WikiHistory**, and community self-description:
### Core mission use cases
1. **Pattern exchange** — publish, discuss, and refine software design patterns
(the PPR mission).
2. **People + projects + patterns** — connect practitioners, project experience,
and reusable design knowledge (`PeopleProjectsAndPatterns`).
3. **Idea exchange among programmers** — fast, lightweight collaborative
writing; "a forum where people share ideas."
4. **Informal history of programming ideas** — accumulated narrative, not a
formal reference work (`InformalHistoryOfProgrammingIdeas`).
5. **Distillation of experience into patterns** — thread discussions refined
into document-mode consensus, sometimes culminating in patterns
(`ThreadMode``DocumentMode``PatternMode`).
### Community / onboarding use cases
6. **First wiki experience** — deliberate onboarding path for newcomers
(`WelcomeVisitors`, `NewUserPages`, `WikiWikiSandbox`).
7. **Sandbox experimentation** — safe place to learn editing mechanics.
8. **Visitor presence** — sign the guest book (`RecentVisitors`).
9. **Culture acclimation** — expect `CultureShock`; learn local conventions
before adding pages.
### Knowledge maintenance use cases
10. **Self-indexing** — wiki indexes itself; community maintains structure
(`VolunteerHousekeeper`, categories, road maps).
11. **Activity awareness** — follow what changed (`RecentChanges`,
`QuickChanges`, `RecentChangesJunkie`).
12. **Discovery** — browse by search, categories, random pages, visual tour,
like-pages, starting points.
13. **Refactoring pages** — merge thread into document, split tangents,
improve clarity (`ReFactoring`, `WikiMaster` role).
14. **Work-in-progress knowledge** — all content explicitly provisional
(`WorkInProgress`).
### What c2 explicitly said it was *not*
- **Not Wikipedia** — subjective, conversational, not a dedicated reference
site (`WikiIsNotWikipedia`, `WikiPedia` as pointer to the real Wikipedia).
- **Not a polished encyclopedia** — "Most of all, this is a forum where
people share ideas!"
### Subject areas the community grew into (approximate eras, WikiHistory)
| Era | Dominant topics on c2 |
|-----|----------------------|
| 1994+ | Patterns, sources, application |
| 1996+ | General design, architecture, methods |
| 1997+ | People and organizational aspects of programming |
| 1998+ | Extreme Programming |
| 2000+ | Wiki itself (meta-discussion) |
Popular page clusters noted by observers: **DesignPatterns**, **ExtremeProgramming**, **TestDrivenDevelopment**, **AgileManifesto**, **UnitTest**, **AntiPattern**, build tools, languages, organizational culture.
---
## 3. Ward's WikiDesignPrinciples (1995 intent)
From Ward Cunningham's reconstructed **WikiDesignPrinciples** page:
| Principle | Meaning |
|-----------|---------|
| **Open** | Incomplete or poorly organized pages may be edited by any reader |
| **Incremental** | Pages cite other pages, including ones not yet written |
| **Organic** | Site structure is editable and evolves like page text |
| **Mundane** | Small number of irregular text conventions for markup |
| **Universal** | Editing and organizing use the same mechanisms as writing |
| **Overt** | Formatted output suggests the input needed to reproduce it |
| **Unified** | Page names from a flat namespace — no extra context to interpret |
| **Precise** | Titles are precise noun phrases to avoid name clashes |
| **Tolerant** | Interpretable behavior preferred over error messages |
| **Observable** | Site activity can be watched and reviewed by any visitor |
| **Convergent** | Duplication discouraged by finding and citing related content |
Ward later noted additional forced principles (e.g. server robustness) beyond
the original eleven.
---
## 4. Popular terms glossary
Terms below are **CamelCase page names** as used on c2 unless noted.
### Wiki mechanics & naming
| Term | Meaning on c2 |
|------|---------------|
| **Wiki** / **WikiWikiWeb** | The first wiki site; capital-W Wiki = this site specifically |
| **WikiBase** | The Perl wiki engine behind c2 |
| **WikiWord** / **WikiWords** | MixedCase token the engine treats as a hyperlink |
| **WikiName** / **WikiNames** | Page title / link target (often built from WikiWords) |
| **MixedCase** / **CamelCase** | CapitalizedWordsRunTogether → automatic link |
| **TextFormattingRules** | Minimal markup conventions (paragraphs, bold, links) |
| **EditText** / **EditCopy** | Core edit operations |
| **Undefined page** / **red link** | `PageName?` with `?` — page does not exist yet; click to create |
| **Flat namespace** | All page names in one space (no hierarchical paths) |
### Content modes (social conventions, not software features)
| Term | Meaning |
|------|---------|
| **DocumentMode** | Third-person, unsigned, community-owned text; consensus artifact |
| **ThreadMode** | Signed, first-person discussion; conversation thread |
| **PatternMode** | Distilled pattern statements from converged discussion |
| **OpeningStatement** | Strong initial claim/question that frames a page |
| **SignedDocumentMode** | Hybrid: document-like but attributed |
| **ThreadModeConsideredHarmful** | Community norm pushing toward document mode |
| **InFavorOfDissertation** | Ward's preference for essay-like pages over dialog |
### Roles & community behavior
| Term | Meaning |
|------|---------|
| **WikiMaster** | Person who refactors threads into documents (anyone can be one) |
| **VolunteerHousekeeper** | Community members who maintain indexes and hygiene |
| **RecentChangesJunkie** | Person who obsessively monitors RecentChanges |
| **InvitedAuthors** | Early contributors who bootstrapped site culture |
| **GoodStyle** | Ward's editing advice: factual, concrete, civil, flow over chronology |
| **ReFactoring** / **RefactorMercilessly** | Rewrite pages for clarity; merge/split threads |
| **RefactorDontDelete** | Prefer refactoring over deletion |
| **CultureShock** | Newcomer disorientation from wiki norms |
### Navigation & derived views
| Term | Meaning |
|------|---------|
| **RecentChanges** | Chronological edit log — central community nerve center |
| **QuickChanges** | Short-form recent changes |
| **RecentEdits** | Minor edits stream |
| **NotSoRecentChanges** | Older change archives |
| **FindPage** | Search |
| **LikePages** | Similar-title suggestions |
| **RandomPages** | Serendipitous browse |
| **VisualTour** | Graphical browse aid |
| **StartingPoints** | Curated entry paths |
| **WikiCategories** / **RoadMaps** | Community-maintained indexes |
| **PeopleIndex** / **RecentVisitors** | Who participates |
| **BackLinks** | Pages linking here (classic derived view) |
| **AllPages** / **SiteMap** / **SearchPage** | Core derived pages (also present in yawex prior art) |
| **SisterSites** | Links to other wikis |
### Pattern & programming vocabulary (c2's dominant content)
| Term | Meaning |
|------|---------|
| **Pattern** / **DesignPatterns** | Reusable design solution in context |
| **AntiPattern** | Common bad solution |
| **ProtoPattern** | Pattern-in-formation |
| **PortlandPatternRepository** | Pattern submission/distribution site |
| **PeopleProjectsAndPatterns** | c2's thematic focus |
| **ExtremeProgramming** / **AgileManifesto** | Major c2 discourse topics (late 1990s2000s) |
| **CrcCards** | Design technique; also spawned from Ward's HyperCard work |
### Meta & identity pages
| Term | Meaning |
|------|---------|
| **WorkInProgress** | Nothing is finished; everything editable |
| **WikiIsNotWikipedia** | Identity boundary vs encyclopedic wikis |
| **WhyWikiWorks** / **WhyWikiWorksNot** | Community theory of success/failure |
| **WikiZen** | Cultural / philosophical reflection |
| **WikiOnWiki** | Meta-discussion about wiki itself (sometimes controversial) |
---
## 5. Documented editing & collaboration patterns
From **ThreadMode**, **DocumentMode**, **GoodStyle**, **WhyWikiWorks**:
### ThreadMode contributions (ADD / EDIT / SPLIT / CAPTURE)
1. **ADD** — append signed comment to continue conversation.
2. **EDIT** — improve flow of others' signed comments (with care).
3. **SPLIT** — separate tangents onto new pages with summary links.
4. **CAPTURE** — distill converging ideas into anonymous pattern paragraphs.
### DocumentMode lifecycle
- Page opens with strong **OpeningStatement**.
- Feedback arrives in ThreadMode.
- On consensus, a **WikiMaster** (or anyone) replaces thread with
DocumentMode synthesis.
- Newcomers are expected to **rewrite** unclear passages rather than stack
clarifying replies.
### WhyWikiWorks (community theory)
Paradoxical strengths cited on c2:
- Anyone can delete anything → only meaningful content survives curation.
- Low WYSIWYG appeal → filters out drive-by noise; participants self-select.
- Slow, considered edits → pages evolve over days/weeks.
- Pedantic community → shared professional camaraderie.
- "Insecure, indiscriminate, user-hostile, slow" — yet it worked *because*
other online communities optimized differently.
---
## 6. Technical & formatting conventions (original c2)
From Ward's etymology correspondence and c2 pages:
- **Double conventions as formatting clues:**
- double carriage-return → new paragraph
- double single-quote → italic
- double capitalized-word (CamelCase) → hyperlink
- **Flat page namespace** — precise noun-phrase titles.
- **Minimal markup** — "mundane" rules; overt correspondence between input
and rendered output.
- **Optional attribution** — `UserName` (2000+) to attach edits to a name
instead of IP; signing in ThreadMode remained common.
- **Deletion** — supported but culturally discouraged vs refactoring
(`RefactorDontDelete`).
---
## 7. Use-case patterns worth naming (synthesis)
Grouping c2-documented uses into reusable patterns:
| ID | Use-case pattern | c2 evidence |
|----|------------------|-------------|
| UC-C2-01 | **Quick idea capture** | Wiki = quick web; incremental linking to unwritten pages |
| UC-C2-02 | **Collaborative glossary** | Flat namespace of precise terms; WikiWords as vocabulary |
| UC-C2-03 | **Discussion → consensus doc** | ThreadMode → DocumentMode refactoring |
| UC-C2-04 | **Pattern mining** | Thread → PatternMode distillation |
| UC-C2-05 | **Community guest book** | RecentVisitors, people pages |
| UC-C2-06 | **Change radar** | RecentChanges / QuickChanges monitoring |
| UC-C2-07 | **Self-curating knowledge base** | Open editing + convergent deduplication |
| UC-C2-08 | **Sandbox learning** | WikiWikiSandbox for safe first edits |
| UC-C2-09 | **Serendipitous browse** | RandomPages, VisualTour, LikePages |
| UC-C2-10 | **Practitioner field notes** | InformalHistoryOfProgrammingIdeas, not encyclopedia |
| UC-C2-11 | **Team memory for methods** | XP, TDD, patterns, tools discourse |
| UC-C2-12 | **Soft creation of missing topics** | Red-link `?` pages created on first write |
---
## 8. Boundaries for shard-wiki (light notes, not federation)
Items from c2 origins that align with existing `INTENT.md` themes **without**
entering federation design:
- **Open editing + recoverable history** — c2 trusted the community; shard-wiki
INTENT makes Git history the safety net (stronger than c2's soft norms).
- **WorkInProgress** — matches overlay/provenance/freshness thinking.
- **Mechanism over policy** — c2 used social conventions (DocumentMode,
GoodStyle) rather than hard gates; shard-wiki keeps policy configurable.
- **Not an encyclopedia** — c2's `WikiIsNotWikipedia` parallels shard-wiki not
owning universal ontology.
- **Derived views** — BackLinks, RecentChanges, AllPages, SiteMap, Search were
first-class on c2; already flagged in yawex research as union-view candidates.
- **Flat namespace + CamelCase links** — original c2 model; shard-wiki is
Markdown-first (wikilink extension question remains open).
**Deferred:** Federated Wiki, sister sites, multi-wiki configuration — separate
research track.
---
## 9. Sources
| Source | URL / location |
|--------|----------------|
| Ward — WikiHistory | https://wiki.c2.com/?WikiHistory (archive 2002) |
| Ward — WikiDesignPrinciples | https://wiki.c2.com/?WikiDesignPrinciples (archive 2002) |
| Ward — etymology correspondence | https://c2.com/doc/etymology.html |
| WelcomeVisitors | https://wiki.c2.com/?WelcomeVisitors (archive 2002) |
| DocumentMode | https://wiki.c2.com/?DocumentMode (archive 2002) |
| ThreadMode | https://wiki.c2.com/?ThreadMode (archive 2002) |
| GoodStyle | https://wiki.c2.com/?GoodStyle (archive 2002) |
| WhyWikiWorks | https://wiki.c2.com/?WhyWikiWorks (archive 2002) |
| RecentChanges | https://wiki.c2.com/?RecentChanges (archive 2002) |
| Wikipedia — WikiWikiWeb | https://en.wikipedia.org/wiki/WikiWikiWeb |
| Wikipedia — Portland Pattern Repository | https://en.wikipedia.org/wiki/Portland_Pattern_Repository |
| Observer summary — T.J. Maher (2016) | https://www.tjmaher.com/2016/06/time-capsule-ward-cunninghams-wiki-wiki.html |
---
## 10. Open questions (for later spec work)
1. Which c2 **social conventions** (DocumentMode, GoodStyle, WikiMaster
refactoring) belong in shard-wiki core vs reference UI vs `wiki/` content?
2. How much of the **flat CamelCase namespace** survives in a Markdown-first,
path-oriented federation model?
3. Which **derived views** from c2 are MVP for an orchestrator vs adapter-provided?
4. Does shard-wiki want an explicit **WorkInProgress / WikiIsNotWikipedia**
stance in `spec/ProductRequirementsDocument.md`?
5. How does c2's **open-by-default** community norm relate to shard-wiki's
L0 mode — coincidence of values or intentional lineage?

View File

@@ -0,0 +1,28 @@
# 260608 — Federation concepts for unified information spaces
Date: 2026-06-08
## What this is
Research into **federation models** for decentralized but unified knowledge
spaces — with emphasis on **Federated Wiki** (Ward Cunningham, ~2011+) and
comparable approaches (git-backed wikis, ActivityPub, Xanadu patterns, yawex
REMOTE/VIRTUAL).
Complements `research/260608-c2-wiki-origins/` (pre-federation wiki culture)
and `research/260608-yawex-prior-art/` (Perl engine resolution states).
Focus: terms, architectural patterns, and **mapping to `shard-wiki` INTENT**
— not implementation specification.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Glossary, model comparison, shard-wiki mapping, open questions, sources |
## Status
Initial exploration complete. Use cases promoted to `spec/UseCaseCatalog.md`
(UC-26UC-33). Federation architecture design tracked in
`workplans/SHARD-WP-0002-federation-architecture.md`.

View File

@@ -0,0 +1,409 @@
# Findings — Federation concepts for decentralized, unified information spaces
Date: 2026-06-08 · Status: research draft
Scope: federation models that let independently stored knowledge participate in
a **joined view** without erasing provenance, sovereignty, or backend
differences. Primary anchor: **Federated Wiki** and Ward Cunningham's
post-c2 inversion. Secondary: git-backed wikis, ActivityPub wiki extensions,
Xanadu hypertext patterns, and yawex multi-wiki resolution.
**Complements:** `research/260608-c2-wiki-origins/` (federation explicitly
deferred there) and `research/260608-yawex-prior-art/` (REMOTE/VIRTUAL as
adapter foreshadowing).
---
## 1. Why federation (the problem statement)
Centralized wikis concentrate **meaning** and **records** on one server:
| Problem | Manifestation |
|---------|---------------|
| **Ownership** | Contributors build on someone else's host; site shutdown = loss risk |
| **Consensus pressure** | Early edits must survive notability/deletion fights (Wikipedia model) |
| **Heat death** | Bounded communities exhaust their charter; maintenance mode → colony collapse |
| **Reuse friction** | Copy-paste HTML/DB content across sites is slow, lossy, attribution-prone |
| **Backend lock-in** | One engine, one storage model, one permission model |
Federation responses split along two axes:
1. **Where records live** — per-user site, per-instance repo, personal pod, mirrored git clone.
2. **How unity is achieved** — fork + spread, git push/pull, protocol activities, projection/cache, orchestrated union.
`shard-wiki` (per `INTENT.md`) targets axis (2) via an **orchestration layer**
over heterogeneous **shards**, with Git as coordination substrate — not a
single federation mechanism.
---
## 2. Federated Wiki (Smallest Federated Wiki / SFW)
### 2.1 Historical anchor
| Fact | Detail |
|------|--------|
| Origin | Ward Cunningham; launched at IndieWebCamp 2011 |
| Motivation | Regret that c2's 35,000+ pages lived on Ward's server, not contributors' |
| c2 migration | WikiWikiWeb moved to Federated Wiki ~2015 after read-only period |
| GitHub lineage | Fork button inspired by GitHub; "radical code sharing" as model |
| Current code | [github.com/fedwiki/wiki](https://github.com/fedwiki/wiki) (CoffeeScript; active releases through 2025) |
Ward's framing (Wired, 2012): the wiki's radical idea was an **edit button on
every page**; federated wiki's radical idea is a **fork button on every page**.
### 2.2 Inverted architecture
Traditional wiki vs federated wiki (Caulfield, NWACC 2014, summarizing Ward):
```
Traditional: many people → one server → server owns records → consensus on server
Federated: many people → many servers → each owns records → browser composes union
```
**Meaning is made in the browser.** The client pulls JSON page records from
multiple origins (CORS) and presents them as one navigable space. Consensus
emerges through **which versions spread** across the network, not through a
single server arbiter.
### 2.3 Page model (JSON + journal)
From [viki.wiki JSON schema](https://viki.wiki/json-schema.html):
```
page = { title, story, journal }
story = [ item ] # paragraph-like plugin items
journal = [ action ] # edit history that reconstructs story
action.type = create|add|move|edit|remove|fork
action.site = origin host # present when content came from elsewhere
```
Key properties:
| Property | Implication |
|----------|-------------|
| **Data files, not DB render** | Fork copies JSON, not scraped HTML — reuse stays structured |
| **JSON items, not HTML** | Target site re-renders via plugins; display differences don't break remix |
| **Journal travels with page** | Attribution/history portable; no separate blame log |
| **Plugin architecture** | Item `type` selects renderer; missing plugins surfaced to user |
| **Fork action** | Whole-page copy from remote site into local site with provenance |
| **Item IDs preserved** | Stable anchors through edits; aliasing guards duplicate IDs |
### 2.4 Interaction patterns
| Pattern | Behavior |
|---------|----------|
| **Fork** | Copy remote page to your site; edit locally; original owner may merge back |
| **Drag-and-drop fork** | Cross-site page transfer in ~15 seconds (OER case) |
| **Chorus of voices** | Multiple versions of "same" topic coexist; linked, not merged by default |
| **Neighborhood / river** | Discovery via recent changes across federated sites (e.g. fedwikiriver.com) |
| **Happening** | Time-bounded collaborative event on a topic-specific subdomain/site |
### 2.5 Mike Caulfield — lifecycle and pedagogy
Caulfield articulated fedwiki's social model beyond Ward's mechanics:
**Kinneavy triangle / knowledge lifecycle** (composition theory):
| Phase | Mode | Typical medium | Fedwiki fit |
|-------|------|----------------|-------------|
| **I** | Personal capture | Notes, journal | Each person's site |
| **You** | Dialogic | Conversation, annotation | Fork chains, cross-site references |
| **It** | Expository | Stable shared article | Spread of refined forks |
Fedwiki supports **I → You → It** in one system; centralized wikis force **It**
(consensus) from the first edit.
**Bounded conversations / expected heat death** (2015):
- Wiki sites are **bounded conversations**, not permanent monuments.
- Sites are **expected to die**; valuable material is **forked into the next happening**.
- **Reverse bit-rot** — Smalltalk-era term Ward revived: objects/pages improve each
time reused across systems (refactor on fork), defying entropy of abandoned forums.
- Contrast: colony collapse on centralized forums (Blue Hampshire example) vs
humane site retirement with selective carry-forward.
**OER reuse** (2015): WordPress copy-paste ~1015 min/page vs fedwiki fork ~15 sec;
bottleneck is software, not human willingness to remix.
**Dissent as feature** (Ogden via Wired): Wikipedia forces one perspective;
fedwiki enables multiple controversial pages, still linked for exploration.
### 2.6 Strengths and limits
| Strengths | Limits |
|-----------|--------|
| Personal sovereignty over pages | Requires running a site (mitigated by one-click AWS deploy) |
| Low-friction remix with provenance | Small niche ecosystem |
| Multi-perspective knowledge | Weak team-governance / permissions story vs enterprise wikis |
| JSON portability | Plugin fragmentation; non-Markdown item model |
| Strong conceptual clarity | Not a general adapter layer for arbitrary backends |
---
## 3. Other federation models
### 3.1 Git-backed distributed wikis (ikiwiki pattern)
[ikiwiki distributed wikis](https://ikiwiki.info/tips/distributed_wikis/) documents
a **spectrum of decentralization**:
| Level | Setup | Federation mechanism |
|-------|-------|---------------------|
| 0 | Single server | None |
| 1 | HTML mirror | Read-only copy |
| 2 | Split web + git host | Clone/pull |
| 3 | Pinger mirrors | Central bare repo + ping-on-edit propagation |
| 4 | Fully decentralized | Independent wikis push/pull via git to peers |
**Branching a wiki** = clone origin read-only (`git://`), edit locally without
push-back — explicit fork. Conflicts surface as markers in rendered pages.
**Relevance to shard-wiki:** Strong alignment with **Git-addressable
coordination** and **coordination journal** in `INTENT.md`. ikiwiki federates
*homogeneous* git-backed wikis; shard-wiki must also attach non-git shards
(Obsidian, WebDAV, Gitea wiki, Coulomb, etc.).
**Gollum / GitHub wikis:** Same backing-store idea — pages as files in a repo;
federation is git sync, not a wiki protocol. Multi-repo Gollum setups are
operational, not semantic union.
### 3.2 ActivityPub wiki federation (XWiki)
[XWiki ActivityPub Application](https://extensions.xwiki.org/xwiki/bin/view/Extension/ActivityPub%20Application/)
(v1.7.x) connects XWiki instances to the **fediverse**:
| Capability | Notes |
|------------|-------|
| Follow user / wiki actor | Receive Create/Update notifications |
| Share documents | Push page content to followers; view remote in modal |
| Messaging / discussions | Note objects threaded as discussions |
| Like, mention | Social engagement primitives |
| Webfinger + ActivityPub endpoints | Standard discovery |
**Caveats (documented by XWiki):** Federation "not well supported yet" for
full activity routing; Mastodon mention of Page entities incomplete; security
signing partial.
**Contrast with fedwiki:** ActivityPub federates **activities and notifications**
between **compatible app instances**; content may be copied on share, but the
model is social-graph + event stream, not per-user fork sovereignty. Closer to
**fediverse** than **fedwiki** UX.
**Contrast with shard-wiki:** ActivityPub could be an **adapter transport** for
some shards, but shard-wiki's core is wiki-page semantics (paths, links,
overlays, provenance), not activity streams.
### 3.3 Xanadu / transclusion lineage
Ted Nelson's [Project Xanadu](https://en.wikipedia.org/wiki/Project_Xanadu)
(1960s) pursued **serious electronic literature** with patterns still relevant:
| Pattern | Intent | Modern partial implementations |
|---------|--------|-------------------------------|
| **Visible links** | Show destination context, not opaque jumps | Hover previews, Open Graph unfurls |
| **Parallel documents** | Side-by-side source + derivative | Multi-pane editors (Obsidian, Roam) |
| **Transclusion** | Live inclusion of remote span with origin pointer | Embeds, block references (weak) |
| **Transcopyright** | Permissioning for reuse | Mostly unsolved at web scale |
| **Stable addresses** | Fine-grained, durable pointers | Block UIDs (Roam); URL rot elsewhere |
| **Bi-directional links** | Backlinks as first-class | Obsidian, Roam, shard-wiki BackLinks UC |
Xanadu is **speculative design / pattern language**, not deployable federation.
Useful for shard-wiki **provenance, transclusion, and link resolution** thinking —
especially union BackLinks and projection freshness.
### 3.4 Personal / local-first wikis (TiddlyWiki, Obsidian)
Not federation protocols, but **sovereignty** models:
- **TiddlyWiki** — single-file portable wiki; complete user ownership.
- **Obsidian** — local vault + optional sync; community plugins for publish/remix.
Shard-wiki INTENT explicitly lists these as shard participants via adapters.
Federation is **attachment to a root information space**, not replacing the
local tool.
### 3.5 yawex multi-wiki resolution (cross-reference)
From `research/260608-yawex-prior-art/findings.md`:
| yawex state | Federation reading |
|-------------|-------------------|
| `REMOTE` | Jump to another wiki's page — remote shard navigation |
| `VIRTUAL` | Local processing, remote content — projection |
| Multi-wiki config | Multiple named wikis = multiple shards |
**Decision already recorded:** inspiration only; shard-wiki designs resolution
fresh for heterogeneous adapters.
### 3.6 Solid / personal data pods (brief)
Solid aims at **user-controlled pods** + linked data apps. Distributed wiki
discussions (forum threads, TiddlyWiki-on-Solid experiments) explore **personal
wikis on pods** vs **team commons**.
| Axis | Solid-style | Fedwiki | shard-wiki |
|------|-------------|---------|------------|
| Unit of sovereignty | Pod / user | Site / person | Shard |
| Unity mechanism | Linked data queries | Browser composition + fork | Orchestrated union + Git journal |
| Team wiki | Secondary; app-dependent | Weak | Supported via shards + authz |
No mature Solid wiki federation standard emerged; treat as **adjacent personal-
data sovereignty** research, not direct prior art.
---
## 4. Comparison matrix
| Model | Unit of ownership | Unity mechanism | Consensus model | History / provenance | Heterogeneous backends |
|-------|-------------------|-----------------|-----------------|----------------------|------------------------|
| **Centralized wiki** (c2, MediaWiki) | Site operator | Single namespace | On-page consensus | Server revision log | No |
| **Federated Wiki** | Per-site (per person) | Browser pulls JSON; fork | Spread of versions | Journal per page | No (all SFW-shaped) |
| **ikiwiki distributed** | Per git clone | git push/pull, pingers | Merge conflicts in git | Git commits | No (all ikiwiki) |
| **ActivityPub (XWiki)** | Per instance | Activity stream | Social follow/share | Activity + page store | No (XWiki only) |
| **Xanadu** | Document address | Transclusion | Parallel versions | Version trails (concept) | N/A (unbuilt) |
| **shard-wiki (INTENT)** | Per shard | Orchestrator union | Configurable policy | Git coordination journal + shard history | **Yes** (adapter contract) |
---
## 5. Glossary (federation track)
| Term | Meaning | Primary source |
|------|---------|----------------|
| **Fork** | Copy a page (or site branch) to a new home; divergent editing | Fedwiki, git |
| **Journal** | Ordered actions reconstructing page state; travels with page | Fedwiki JSON schema |
| **Story** | Array of typed items (paragraph, image, …) forming page body | Fedwiki |
| **Happening** | Time-bounded fedwiki collaboration on a topic site | Caulfield |
| **Bounded conversation** | Site with expected end; not infinite commons | Caulfield |
| **Reverse bit-rot** | Content improves through reuse across contexts | Caulfield / Ward |
| **Chorus of voices** | Multiple perspectives on same topic, linked | Ward |
| **Inverted model** | Records distributed; meaning composed client-side | Ward |
| **Projection** | Local view of remote content without claiming ownership | shard-wiki / yawex VIRTUAL |
| **Overlay** | Local edit against remote/canonical without immediate mutation | shard-wiki |
| **Shard** | Independently meaningful page store with own capabilities | shard-wiki |
| **Coordination journal** | Git-backed change record for an information space | shard-wiki |
| **Transclusion** | Include live span from elsewhere with origin visible | Xanadu |
| **Fediverse** | Network of ActivityPub-speaking apps | XWiki AP, Mastodon |
| **Pinger / pingee** | Edit-triggered mirror refresh between ikiwiki sites | ikiwiki |
| **Wiki actor** | Entire wiki as ActivityPub actor | XWiki AP |
---
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
### 6.1 Strong resonances
| Fedwiki / federation idea | shard-wiki concept | Notes |
|---------------------------|-------------------|-------|
| Personal ownership of edits | Shard sovereignty | shard-wiki generalizes beyond per-person sites |
| Fork with provenance | Overlay + patch flows | shard-wiki separates draft from destructive apply |
| Journal / history with page | Coordination journal + shard revision | Git journal is space-level; shard retains own history |
| Browser-composed union | Union of pages across shards | Orchestrator presents coherent graph |
| JSON not HTML for remix | Markdown-first page model | Different format; same structural intent |
| Lazy pull from remote | Projection | yawex VIRTUAL; explicit freshness |
| Chorus of voices | Union without erasure | Show provenance, divergence, equivalents |
| Bounded conversation | Configurable policy | Mechanism not policy — space may be permanent or ephemeral |
| Reverse bit-rot on reuse | Reconciliation / sync | Policy-driven; not automatic fork |
| Derived views (recent, links) | UC-05 union views | BackLinks, RecentChanges across shards |
### 6.2 Deliberate divergences (design bugs if conflated)
| Fedwiki assumption | shard-wiki correction |
|--------------------|----------------------|
| Every participant runs SFW-shaped JSON site | **Adapter contract** for Gitea, folders, Obsidian, WebDAV, engines |
| Fork = default edit primitive | **Capability-aware**: read-only shards need overlay, not fork |
| Client composes in browser only | Orchestrator may serve CLI, agents, CI — not browser-only |
| Per-site = per-person | Shards may be team repos, org wikis, app databases |
| No central coordination | **Git-addressable coordination layer** per information space |
| Plugin item types | Markdown-first; engine-specific render is out of core scope |
| Implicit spread-as-consensus | **Mechanism over policy** — canonical source is explicit config |
### 6.3 What shard-wiki adds beyond fedwiki
1. **Heterogeneous attachment** — not all shards speak the same page JSON.
2. **Capability matrix** — read/write/diff/merge/lock/publish per shard.
3. **Overlay-before-mutation** — respect remote sovereignty and limited backends.
4. **Semantic wiki sync** — not generic file mirroring.
5. **Enterprise authz ladder** — L0 open → delegated IAM (without owning identity).
6. **Divergence detection** — equivalent pages across shards, reconcilable.
### 6.4 What fedwiki teaches that shard-wiki should not lose
1. **Frictionless reuse** — if remix takes 15 minutes, federation fails socially.
2. **Provenance by default** — history must travel or be reconstructible.
3. **Multi-perspective truth** — union ≠ single canonical article.
4. **Expected lifecycle** — information spaces may be ephemeral; plan for carry-forward.
5. **Personal sovereignty** — contributors must not be hostage to one operator's server.
---
## 7. Use-case promotion (done 2026-06-08)
Promoted to `spec/UseCaseCatalog.md` as UC-26UC-33. Architecture decisions
tracked in `workplans/SHARD-WP-0002-federation-architecture.md`.
| Research ID | Catalog UC |
|-------------|------------|
| UC-FED-01 Fork page from remote shard | UC-26 |
| UC-FED-02 View multiple versions of equivalent page | UC-27 |
| UC-FED-03 Carry forward from closed/archived shard | UC-28 |
| UC-FED-04 Remix with portable attribution | UC-29 |
| UC-FED-05 Time-bounded collaboration space | UC-30 |
| UC-FED-06 Subscribe to remote shard changes | UC-31 |
| UC-FED-07 Transclude remote span | UC-32 |
| UC-FED-08 Git-branch information space | UC-33 |
---
## 8. Open questions (for spec / workplans)
1. **Fork vs overlay vs import** — When does shard-wiki copy content into a
writable shard vs keep an overlay vs link-only reference?
2. **Equivalent page identity** — How are "same topic" pages matched across
shards (title, path, link graph, explicit alias)?
3. **Journal format** — Does shard-wiki adopt fedwiki-like action journals,
Git commits only, or both per shard type?
4. **Browser vs server composition** — Where does union merging run for agents
and non-UI consumers?
5. **ActivityPub as adapter** — Optional transport for change notification, or
out of scope?
6. **Ephemeral spaces** — Should information spaces have first-class lifecycle
(archived, read-only, merged-into-successor)?
7. **Plugin/item extensibility** — Markdown extensions vs structured blocks for
remixable assessments (fedwiki OER lesson)?
8. **Consensus without erasure** — Policy presets: fedwiki-spread, git-merge,
designated canonical shard, vote-to-merge?
---
## 9. Sources
| Source | URL |
|--------|-----|
| Wikipedia — Federated Wiki | https://en.wikipedia.org/wiki/Federated_Wiki |
| Wired — Wiki Inventor Sticks a Fork | https://www.wired.com/2012/07/wiki-inventor/ |
| viki.wiki — JSON Schema | https://viki.wiki/json-schema.html |
| Fedwiki GitHub | https://github.com/fedwiki/wiki |
| Caulfield — Federated Education (NWACC 2014) | https://hapgood.us/2014/11/06/federated-education-new-directions-in-digital-collaboration/ |
| Caulfield — Bounded Conversations | https://hapgood.us/2015/01/21/rethinking-wiki-lifecycle-sites-as-bounded-conversations/ |
| Caulfield — OER Case for Federated Wiki | https://hapgood.us/2015/05/05/the-oer-case-for-federated-wiki/ |
| ikiwiki — distributed wikis | https://ikiwiki.info/tips/distributed_wikis/ |
| XWiki — ActivityPub Application | https://extensions.xwiki.org/xwiki/bin/view/Extension/ActivityPub%20Application/ |
| Maggie Appleton — Xanadu Patterns | https://maggieappleton.com/xanadu-patterns |
| Wikipedia — Project Xanadu | https://en.wikipedia.org/wiki/Project_Xanadu |
| shard-wiki — yawex prior art | `research/260608-yawex-prior-art/findings.md` |
| shard-wiki — c2 origins | `research/260608-c2-wiki-origins/findings.md` |
| shard-wiki — INTENT | `INTENT.md` |
---
## 10. Traceability
| This document section | Informs (future) |
|-----------------------|------------------|
| §2 Federated Wiki | Adapter design, page model, overlay/fork policy |
| §3.1 ikiwiki | Git coordination journal, mirror/branch modes |
| §3.2 ActivityPub | Optional notification adapter |
| §3.3 Xanadu | Link resolution, transclusion, provenance UI |
| §6 Mapping | Architecture blueprint guardrails |
| §7 UC seeds | `spec/UseCaseCatalog.md` promotion pass |

View File

@@ -0,0 +1,397 @@
# Perplexity Research Wiki Engines
Source: https://www.perplexity.ai/search/9b76560e-31b7-4465-87a4-01a9df9cdbbe#1
## What are the most technically interesting wiki engines out there?
Below is an expanded, more technical map of the wiki landscape, including your additions: C2, Federated Wiki, TWiki, Confluence, Semantic MediaWiki, TiddlyWiki, Fossil, and the Tcl-based option I found. Ive kept the focus on what each system is *for*, what makes it distinct, and what kind of adoption it seems to have had. [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
## Wiki.js
Wiki.js is a modern, extensible, self-hosted wiki aimed at polished documentation and flexible deployment. It feels like the “modern web app” interpretation of a wiki. [js](https://js.wiki)
**USPs**
- Modern UI and authoring experience.
- Extensible architecture.
- Self-hosted and open source. [wikimatrix](https://www.wikimatrix.org/compare/wiki-js+dokuwiki+xwiki+mediawiki+tiki-wiki-cms-groupware+tiddlywiki)
**Strengths**
- Strong for documentation teams that want low-friction editing.
- Good fit when you want a contemporary stack and a clean UX.
- Broadly positioned as “powerful and extensible.” [js](https://js.wiki)
**Weaknesses**
- Less historically proven than MediaWiki or TWiki in huge legacy enterprise environments.
- Smaller ecosystem than the oldest wiki engines.
- Not as “structured application platform” oriented as Foswiki/XWiki. [wikimatrix](https://www.wikimatrix.org/compare/wiki-js+dokuwiki+xwiki+mediawiki+tiki-wiki-cms-groupware+tiddlywiki)
**Characteristic use cases**
- Internal docs.
- Product documentation.
- Team knowledge bases.
- Self-hosted alternative to SaaS doc tools. [js](https://js.wiki)
**Userbase estimate**
- Likely **tens of thousands of active deployments/users**, but public numbers are not well standardized. That makes it a strong product choice, but not a giant legacy ecosystem like MediaWiki or TWiki. [wikimatrix](https://www.wikimatrix.org/compare/wiki-js+dokuwiki+xwiki+mediawiki+tiki-wiki-cms-groupware+tiddlywiki)
## XWiki
XWiki is a programmable wiki platform that blurs the line between wiki and lightweight application platform. It is technically interesting because it supports scripts, macros, and application-style pages. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**USPs**
- Scriptable pages and apps.
- Strong extension model.
- Syntax compatibility with other wiki styles. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Strengths**
- Great for building structured internal tools.
- More “platform-like” than plain wiki software.
- Good fit for teams that need dynamic content and forms. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Weaknesses**
- More complexity than simpler wikis.
- Can feel heavyweight if you only want plain documentation.
- Requires more governance and administration. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Characteristic use cases**
- Enterprise knowledge bases.
- Internal business applications.
- Structured reporting and collaborative workflows.
- Documentation that needs embedded logic. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Userbase estimate**
- Probably **tens of thousands to low hundreds of thousands** of users across deployments, but public install counts are not clearly disclosed. [wikimatrix](https://www.wikimatrix.org/compare/wiki-js+dokuwiki+xwiki+mediawiki+tiki-wiki-cms-groupware+tiddlywiki)
## MediaWiki
MediaWiki is the canonical large-scale wiki engine: boring in the best possible way, because it works at enormous scale and has an enormous extension ecosystem. It remains technically important because it set the pattern for high-volume collaborative editing. [herothemes](https://herothemes.com/blog/wiki-software/)
**USPs**
- Proven at internet scale.
- Huge extension ecosystem.
- Strong revision/history and collaboration model. [herothemes](https://herothemes.com/blog/wiki-software/)
**Strengths**
- Best-in-class for public, high-traffic, multi-author wikis.
- Extremely mature.
- Broad tooling and community support. [herothemes](https://herothemes.com/blog/wiki-software/)
**Weaknesses**
- Can be operationally heavy.
- Editing experience is not the smoothest by default.
- Often overkill for small internal knowledge bases. [herothemes](https://herothemes.com/blog/wiki-software/)
**Characteristic use cases**
- Wikipedia-style public encyclopedias.
- Community knowledge bases.
- Large multilingual collaborative sites.
- Semantic layering via extensions. [herothemes](https://herothemes.com/blog/wiki-software/)
**Userbase estimate**
- By far the largest in this group; think **millions of contributors and readers across many thousands of sites**. [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
## Semantic MediaWiki
Semantic MediaWiki is technically interesting because it turns wiki pages into queryable knowledge objects. It is less “just a wiki” and more a semantic knowledge base on top of MediaWiki. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
**USPs**
- Semantic annotations.
- Queryable structured content.
- Strong fit for knowledge management with ontology-like behavior. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Strengths**
- Very powerful for structured knowledge capture.
- Supports reporting and retrieval beyond full-text search.
- Bridges freeform wiki editing and data-driven retrieval. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Weaknesses**
- Steeper learning curve.
- Requires discipline in content modeling.
- More complex than plain wiki editing. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Characteristic use cases**
- Research knowledge bases.
- Reference systems with structured facts.
- Enterprise knowledge graphs and catalogs.
- Areas where pages should behave like data records. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Userbase estimate**
- Smaller than MediaWiki, but meaningful in specialist knowledge-management circles: likely **thousands of sites, not millions**. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
## DokuWiki
DokuWiki is the classic lightweight, file-based wiki: simple, pragmatic, and still technically elegant because it avoids a database. It is a workhorse for teams that want low operational overhead. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
**USPs**
- No database required.
- Simple deployment.
- Large plugin ecosystem. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
**Strengths**
- Easy to run and back up.
- Good text/wiki syntax experience.
- Reliable for small-to-medium teams. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
**Weaknesses**
- Less suited to highly structured app-like workflows.
- Not the strongest choice for huge public communities.
- Interface feels older than modern doc platforms. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
**Characteristic use cases**
- Internal docs.
- Departmental knowledge bases.
- Small team collaboration.
- Simple intranets and technical notes. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
**Userbase estimate**
- Likely **tens of thousands of installations**; broad but not dominant. [medevel](https://medevel.com/os-wiki-engines-for-2021/)
## TiddlyWiki
TiddlyWiki is one of the most technically unusual wiki engines because it is often a single self-contained HTML file that acts like a personal knowledge system. That makes it deeply portable and hackable. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**USPs**
- Single-file architecture.
- Highly personal and portable.
- Strong customization via plugins/macros/themes. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Strengths**
- Great for personal knowledge management.
- Easy to carry around and version-control.
- Extremely flexible for power users. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Weaknesses**
- Not a conventional multi-user server wiki.
- Collaboration and permissions are not its core strength.
- Can become messy without personal discipline. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Characteristic use cases**
- Personal notes.
- Research notebooks.
- Offline-first knowledge bases.
- Individual knowledge workflows. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Userbase estimate**
- Probably **tens of thousands to a few hundred thousand users**, with a strong enthusiast niche. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
## TWiki
TWiki is a structured enterprise wiki that emerged from enterprise demand rather than minimalist wiki ideals. It is interesting because it became a platform for forms, workflows, and application-like pages. [junauza](https://www.junauza.com/2008/05/five-of-best-freeopen-source-wiki.html)
**USPs**
- Enterprise collaboration platform.
- Strong extensibility, with hundreds of extensions.
- Structured content and application building. [sourceforge](https://sourceforge.net/projects/twiki/)
**Strengths**
- Proven in corporate environments.
- Great for structured team workflows.
- Strong fit for intranets and process-heavy use. [sourceforge](https://sourceforge.net/projects/twiki/)
**Weaknesses**
- Older design philosophy.
- Can feel complex and procedural.
- Less attractive for lightweight modern docs. [sourceforge](https://sourceforge.net/projects/twiki/)
**Characteristic use cases**
- Corporate intranets.
- Team collaboration.
- Project tracking.
- Enterprise knowledge management. [junauza](https://www.junauza.com/2008/05/five-of-best-freeopen-source-wiki.html)
**Userbase estimate**
- SourceForges description claims **50,000 small businesses, many Fortune 500 companies, and millions of people** use TWiki, though that should be treated as a vendor-style estimate rather than audited usage data. [sourceforge](https://sourceforge.net/projects/twiki/)
## Foswiki
Foswiki is the continuation of the structured enterprise-wiki tradition, with strong emphasis on collaboration, plugins, and business use. It is technically interesting because it treats the wiki as an application platform with macros, forms, and reporting. [sourceforge](https://sourceforge.net/projects/foswiki/)
**USPs**
- Enterprise wiki and collaboration platform.
- Application-building via markup and macros.
- Fine-grained access control and auditability. [sourceforge](https://sourceforge.net/projects/foswiki/)
**Strengths**
- Strong fit for controlled enterprise environments.
- Very flexible for internal business applications.
- Deep extension model and workflow-style usage. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
**Weaknesses**
- Heavier and more specialized than lightweight wikis.
- UX is functional rather than modern-first.
- Best when governance and structure matter more than simplicity. [sourceforge](https://sourceforge.net/projects/foswiki/)
**Characteristic use cases**
- Knowledge bases.
- Team collaboration portals.
- Workflow and project tracking.
- Document management and structured reporting. [hpcsupport.utsa](http://hpcsupport.utsa.edu/foswiki/bin/view/System/WikiCulture)
**Userbase estimate**
- Likely **thousands to tens of thousands of active users/deployments**, with a strong enterprise niche rather than mass-market scale. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
## Confluence
Confluence is the enterprise commercial benchmark for team wikis, but technically it is more a knowledge collaboration platform than a pure wiki engine. It matters because many organizations chose it as the default workplace knowledge layer. [herothemes](https://herothemes.com/blog/wiki-software/)
**USPs**
- Tight enterprise integration.
- Mature permissions and collaboration.
- Large marketplace ecosystem. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Strengths**
- Strong adoption in enterprises.
- Good for cross-team documentation and workflows.
- Familiar to many business users. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Weaknesses**
- Commercial lock-in.
- Less transparent/inspectable than open-source engines.
- Can become sprawling and hard to govern at scale. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Characteristic use cases**
- Enterprise documentation.
- Project spaces.
- Meeting notes and cross-functional knowledge sharing.
- Regulated orgs that need access control and audit trails. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Userbase estimate**
- Very large enterprise footprint, plausibly **millions of users** across customers, but exact counts are not public. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
## C2
C2 is historically important because it sits close to the origin story of wiki culture and especially “wiki engine as a social process,” not merely software. It is less a mainstream product choice today and more a foundational technical-cultural reference point. [wiki.c2](https://wiki.c2.com/?WikiEngines)
**USPs**
- Foundational wiki culture.
- Minimalist and historically influential.
- Strong association with Ward Cunninghams original ideas. [wiki.c2](https://wiki.c2.com/?WikiEngines)
**Strengths**
- Very important conceptually.
- Captures the original wiki ethos.
- Useful as a reference model for design thinking. [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
**Weaknesses**
- Not a modern enterprise product.
- Limited contemporary ecosystem.
- Not typically chosen for new production deployments. [wiki.c2](https://wiki.c2.com/?WikiEngines)
**Characteristic use cases**
- Early collaborative documentation.
- Idea exploration.
- Community knowledge sharing.
- Historical origin of wiki practice. [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
**Userbase estimate**
- Modern active userbase is likely **very small**, mostly historical and enthusiast usage. [wiki.c2](https://wiki.c2.com/?WikiEngines)
## Federated Wiki
Federated Wiki is one of the most intellectually interesting designs because Ward Cunningham pushed the idea beyond a central shared wiki toward a network of interoperating personal sites. It is a reaction against the assumptions of centralized knowledge ownership. [wiki.c2](https://wiki.c2.com/?WikiEngines)
**USPs**
- Federated, distributed model.
- Personal control with sharing across sites.
- Strong conceptual innovation. [wiki.c2](https://wiki.c2.com/?WikiEngines)
**Strengths**
- Excellent for decentralization-minded users.
- Encourages personal ownership of notes and pages.
- Very distinctive architecture. [wiki.c2](https://wiki.c2.com/?WikiEngines)
**Weaknesses**
- Smaller ecosystem.
- Less mainstream operational tooling.
- Can be harder for teams expecting centralized permissions and governance. [wiki.c2](https://wiki.c2.com/?WikiEngines)
**Characteristic use cases**
- Personal knowledge networks.
- Distributed collaboration.
- Experimental knowledge sharing.
- Communities interested in networked authorship. [wiki.c2](https://wiki.c2.com/?WikiEngines)
**Userbase estimate**
- Probably **small niche adoption**, more influential than widely deployed. [wiki.c2](https://wiki.c2.com/?WikiEngines)
## TiddlyWiki
TiddlyWiki is already covered above as one of the core additions, but it deserves emphasis: it is the most “personal-computing” oriented wiki in this list. Its architecture is a big reason it remains technically admired. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**USPs**
- Portable single-file personal wiki.
- Deep customization.
- Offline-friendly. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Strengths**
- Great for individual workflows.
- Easy to archive and transport.
- Powerful for note structuring and metadata. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Weaknesses**
- Not a natural fit for large teams.
- Sync/conflict handling is user-managed.
- Can be too freeform for rigid organizations. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Characteristic use cases**
- Zettelkasten-like note systems.
- Research and personal PKM.
- Portable offline notes.
- Solo technical planning. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
**Userbase estimate**
- Niche but durable: **tens of thousands to low hundreds of thousands**. [wikiwand](https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software)
## Fossil
Fossil is technically interesting because it bundles version control, bug tracking, and wiki/documentation into one system. Its wiki is part of a developer workflow, not a standalone knowledge product. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
**USPs**
- Integrated VCS, issue tracker, and wiki.
- Single-tool developer workflow.
- Simple, self-contained project hosting. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
**Strengths**
- Excellent for small to medium software projects.
- Tight coupling between code, tickets, and docs.
- Strong for maintaining project history. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
**Weaknesses**
- Not a general-purpose wiki platform.
- Smaller ecosystem than GitHub/GitLab-style stacks.
- More dev-centric than business-user-centric. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
**Characteristic use cases**
- Software project documentation.
- Developer-run issue tracking and wiki pages.
- Integrated team repositories.
- Small tooling-centric open-source projects. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
**Userbase estimate**
- Likely **tens of thousands of users**, mostly among developers and small project teams. [en.wikipedia](https://en.wikipedia.org/wiki/Fossil_(software))
## Tcl wiki option
The Tcl ecosystem does have a wiki presence, but the result I found is mainly **Tcler's Wiki**, which is a wiki for the Tcl community rather than clear evidence of a widely used Tcl-native standalone engine. So I would not confidently label it as a major separate engine without more evidence. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
**USPs**
- Community knowledge base for Tcl.
- Demonstrates Tcl community use of wiki patterns. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
**Strengths**
- Niche relevance for Tcl users.
- Good example of a community-maintained knowledge base. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
**Weaknesses**
- Not clearly a mainstream, distinct wiki engine product.
- Limited evidence of broad adoption beyond the Tcl community. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
**Characteristic use cases**
- Tcl community documentation.
- Shared project knowledge.
- Language-specific community support. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
**Userbase estimate**
- Very small, niche community usage. [wiki.tcl-lang](https://wiki.tcl-lang.org/page/Fossil)
## What stands out most
If you want the most technically distinctive group, Id rank them this way: **Federated Wiki** for distributed architecture, **TiddlyWiki** for single-file personal knowledge, **Semantic MediaWiki** for queryable structure, **Fossil** for integrated dev workflow, and **Foswiki/TWiki** for enterprise wiki-as-application platforms. [en.wikipedia](https://en.wikipedia.org/wiki/Foswiki)
The older engines are still interesting because they reflect different answers to the same design question: should a wiki be a simple page editor, a structured app platform, or a knowledge graph with a browser front end? [c2](http://c2.com/cgi/wiki?TopTenWikiEngines)
# Possible Follow Ups
- How do the underlying data architectures of these wiki engines impact long-term knowledge curation and data longeevity?
- How does TiddlyWiki manage non-linear storage in a single file?
- How do Semantic MediaWiki graph capabilities compare to modern graph DBs?
- What architectural challenges led to the creation of Federated Wiki?
- How does Fossil provide version control and wiki in one binary?

View File

@@ -0,0 +1,32 @@
# 260608 — Wiki engines overview (landscape scan)
Date: 2026-06-08
## What this is
A breadth-first **landscape scan of notable wiki engines** — Wiki.js, XWiki,
MediaWiki, Semantic MediaWiki, DokuWiki, TiddlyWiki, TWiki, Foswiki, Confluence,
C2, Federated Wiki, Fossil — captured to understand the design space
`shard-wiki` orchestrates over (and deliberately does **not** re-implement).
Complements the deeper-dive research:
- `research/260608-c2-wiki-origins/` — pre-federation wiki culture
- `research/260608-federation-concepts/` — Federated Wiki, git/AP federation models
- `research/260608-yawex-prior-art/` — the specific Perl engine being superseded
Focus: what each engine is *for*, its storage/permission model, and what its
existence implies for shard-wiki's **adapter contract** — not engine selection.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Synthesis: storage/permission axes, shard-wiki mapping, adapter implications, open questions |
| `260608-perplexity-wikiengines-research.md` | Raw source dump (Perplexity), kept as provenance |
## Status
Initial scan complete. Reinforces the INTENT stance that shard-wiki is an
**orchestrator, not an engine**: the heterogeneity below is the problem space,
not a menu to pick from. Adapter-contract implications feed
`workplans/SHARD-WP-0002-federation-architecture.md`.

View File

@@ -0,0 +1,147 @@
# Findings — Wiki engine landscape and what it implies for shard-wiki
Date: 2026-06-08 · Status: research draft
Scope: a breadth-first scan of notable wiki engines, read through one question —
**what does this engine's storage and permission model demand of a shard adapter,
and what must shard-wiki therefore NOT assume is uniform?** This is landscape, not
selection: shard-wiki orchestrates over these, it does not pick one.
**Complements:** `research/260608-federation-concepts/` (federation mechanics) and
`research/260608-c2-wiki-origins/` (origin culture). Raw source kept alongside as
`260608-perplexity-wikiengines-research.md`.
---
## 1. The engines, by what they actually are
The "wiki" label spans at least four distinct system shapes. Conflating them is
exactly the mistake shard-wiki's adapter contract exists to prevent.
| Shape | Engines | Storage | Permission model | Page format |
|-------|---------|---------|------------------|-------------|
| **Simple page editor** | C2 (origin), DokuWiki | flat files (DokuWiki: no DB) | open / htaccess / ACL plugin | wiki markup |
| **App platform** | XWiki, TWiki, Foswiki | DB + structured objects | fine-grained, per-space, forms/workflow | macro markup |
| **Internet-scale collaborative** | MediaWiki, Semantic MediaWiki | DB + revision tables | role/group, extension-driven | wikitext (+ semantic annotations) |
| **Personal / single-file** | TiddlyWiki, (Obsidian) | one HTML file / local vault | user-managed, not multi-user | tiddlers / Markdown |
| **Federated / distributed** | Federated Wiki, Fossil, ikiwiki | per-site JSON / repo | per-site sovereignty / VCS auth | JSON story+journal / files |
| **Commercial benchmark** | Confluence | proprietary store | mature enterprise ACL + SSO | proprietary |
The same word, six storage substrates and at least five permission philosophies.
## 2. The two axes that matter for orchestration
Stripping the marketing, engines differ on two axes shard-wiki must model
explicitly rather than flatten:
1. **Storage substrate** — flat files (DokuWiki, Fossil), single file (TiddlyWiki),
relational DB + revision tables (MediaWiki, XWiki, Confluence), per-site JSON
journals (Federated Wiki), git repo (Fossil, ikiwiki, Gitea wikis).
2. **Permission philosophy** — open-by-default (C2), file/htaccess gating
(DokuWiki, yawex), per-space/per-page enterprise ACL (Foswiki, XWiki,
Confluence), VCS-level auth (Fossil), or *none / user-managed* (TiddlyWiki).
These map directly onto two INTENT concepts:
- Storage substrate → the **shard adapter contract** must not assume one backend.
- Permission philosophy → validates the **L0→L4 access ladder**: real engines
already span "open C2" to "enterprise ACL + SSO". shard-wiki's ladder is not
invented; it's the union of where these engines actually sit.
## 3. Capability heterogeneity (why "capability-aware" is non-negotiable)
The scan concretely demonstrates the INTENT principle *capability-aware adapters*:
| Operation | Always available? | Counter-example |
|-----------|-------------------|-----------------|
| Read | ~yes | Confluence behind SSO: not without auth |
| Write | no | C2 today is read-only/archival; MediaWiki dumps are read-only |
| Structured diff/merge | no | DB-backed engines expose revisions but not git-style 3-way merge |
| Semantic query | rarely | only Semantic MediaWiki / XWiki |
| Single-file atomicity | special | TiddlyWiki is whole-file, not per-page, writes |
| Native VCS history | sometimes | Fossil/ikiwiki yes; Confluence no (internal only) |
No adapter can assume a peer supports its own operation set — the capability
profile per shard is load-bearing, not decorative.
## 4. Mapping to shard-wiki INTENT (compare, do not equate)
### 4.1 Reinforcements (engines confirm an INTENT stance)
| Observation | INTENT principle confirmed |
|-------------|----------------------------|
| Six storage substrates under one word | Markdown-first, **backend-neutral**; adapter contract |
| Permission philosophies span open→enterprise | **Open by default, progressively governed** (L0→L4) |
| DB engines expose revisions but not portable history | **Git-addressable coordination** adds the missing portable journal |
| TiddlyWiki/Obsidian = sovereign personal stores | **Local-first usefulness**; shards, not replacements |
| Federated Wiki = union composed from many origins | **Union without erasure** |
| Confluence/Foswiki = mature ACL but lock-in | shard-wiki **delegates** IAM, never owns identity |
### 4.2 Deliberate divergences (design bugs if conflated)
| Engine assumption | shard-wiki correction |
|-------------------|----------------------|
| "Pick the one engine for your team" | shard-wiki attaches *many* as shards; not a single choice |
| App-platform macros define the page | **Markdown-first**; engine-specific render stays out of core |
| Permissions live in the engine's DB | **Authorization in core**, identity delegated — uniform across shards |
| History = the engine's revision table | **Coordination journal** is space-level, git-backed, portable |
| Federation = everyone runs the same engine | Adapter contract spans heterogeneous, non-git, capability-limited backends |
### 4.3 What the landscape teaches that shard-wiki should not lose
1. **Don't homogenize** — the heterogeneity is the user's reality, not a defect to fix.
2. **Openness and governance are a spectrum, not a toggle** — engines live at every
point C2→Confluence; the ladder must too.
3. **Structured/semantic pages exist** (Semantic MediaWiki, XWiki) — Markdown-first
must degrade gracefully when a shard carries structure shard-wiki can't model yet.
4. **Single-file/whole-vault write granularity** (TiddlyWiki) breaks per-page write
assumptions — the capability profile must express write granularity.
## 5. Implications for the adapter contract (carry to SHARD-WP-0002)
1. **Capability profile must encode write granularity** (per-page vs whole-file vs
whole-space) — TiddlyWiki and DB engines differ here.
2. **History portability is an adapter concern** — engines with internal-only
revisions (Confluence, MediaWiki) need the coordination journal to supply the
git-addressable layer they lack.
3. **Structured/semantic payloads need a passthrough/provenance escape hatch** so
Semantic MediaWiki / XWiki content isn't silently flattened to lossy Markdown.
4. **Permission model is never trusted from the shard** — even engines with rich
internal ACLs are authorized through shard-wiki's core, not their DB.
## 6. Open questions (for spec / workplans)
1. How does shard-wiki represent **structured/semantic** pages (Semantic MediaWiki,
XWiki objects) — lossy Markdown projection, sidecar metadata, or opaque blob?
2. **Write granularity** in the capability profile — what's the vocabulary
(per-page / per-file / per-space / append-only)?
3. For engines with **internal-only history** (Confluence), is the git coordination
journal authoritative, or a mirror with reconciliation back?
4. Which of these engines get **first-class adapters** vs are reachable only via a
generic file/WebDAV adapter?
## 7. Sources
| Source | URL |
|--------|-----|
| Perplexity — wiki engines scan (raw, this dir) | `260608-perplexity-wikiengines-research.md` |
| C2 — Top Ten Wiki Engines | http://c2.com/cgi/wiki?TopTenWikiEngines |
| C2 — WikiEngines | https://wiki.c2.com/?WikiEngines |
| Comparison of wiki software | https://www.wikiwand.com/en/articles/Comparison%20of%20wiki%20software |
| WikiMatrix engine comparison | https://www.wikimatrix.org/ |
| Wiki.js | https://js.wiki |
| Foswiki | https://en.wikipedia.org/wiki/Foswiki |
| Fossil (software) | https://en.wikipedia.org/wiki/Fossil_(software) |
| shard-wiki — federation concepts | `research/260608-federation-concepts/findings.md` |
| shard-wiki — INTENT | `INTENT.md` |
---
## 8. Traceability
| This document section | Informs (future) |
|-----------------------|------------------|
| §2 two axes | Access ladder (INTENT) + adapter contract guardrails |
| §3 capability heterogeneity | Adapter capability profile design |
| §4 INTENT mapping | Architecture blueprint guardrails |
| §5 adapter implications | `workplans/SHARD-WP-0002-federation-architecture.md` |
| §6 open questions | spec — page model for structured/semantic content |

View File

@@ -0,0 +1,36 @@
# 260613 — Foswiki deep dive (store abstraction, extension API, ecosystem)
Date: 2026-06-13
## What this is
A focused study of **Foswiki** — the community fork of TWiki (2008, from TWiki
4.2.x) — concentrating on **what Foswiki changed**, not the lineage it shares with
TWiki. The distinctive material: a **pluggable `Foswiki::Store` backend**
(RcsWrap / RcsLite / **PlainFileStoreContrib**) behind a versioned interface, the
`Foswiki::Meta` topic-object indirection, the OO/MVC core rewrite, the cleaner
`Foswiki::Func` + `registerTagHandler` extension API, **DataForms** (+ MetaDataPlugin,
FlexFormPlugin), and the **WysiwygPlugin** TML↔HTML round-trip.
Read through shard-wiki's lens: Foswiki is the engine that **already separates its
store backend from its core via a versioned interface** — i.e. it is concrete prior
art for shard-wiki's own *shard adapter contract* and *capability-aware adapters*.
Pairs with — and deliberately does not repeat — the shared TWiki material:
- `research/260613-twiki-deep-dive/` — flat-file/RCS store, Webs/Topics, plugin
handler callbacks, per-topic ACL (all inherited; read there for the shared parts)
- `research/260613-xwiki-deep-dive/` — the Java/DB app-platform sibling
- `research/260608-wikiengines-overview/` — landscape scan
## Contents
| Path | Role |
|------|------|
| `findings.md` | Distinctives vs TWiki, store abstraction, extension API, DataForms, capability profile, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Two new use cases promoted to
`spec/UseCaseCatalog.md` (UC-42, UC-43); UC-34/39/40 enriched. The
`Foswiki::Store`/`Foswiki::Meta` abstraction is logged as adapter-contract prior art
for `workplans/SHARD-WP-0002-federation-architecture.md`.

View File

@@ -0,0 +1,188 @@
# Findings — Foswiki: store abstraction, extension API, ecosystem
Date: 2026-06-13 · Status: research draft
Scope: **Foswiki as distinct from TWiki** — only the deltas, since the two share a
data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to
shard-wiki: a **pluggable store backend behind a versioned interface**
(`Foswiki::Store` + `Foswiki::Meta`), an OO/MVC core rewrite, a cleaner extension
API (`Foswiki::Func::registerTagHandler`), enhanced DataForms, and a TML↔HTML
round-trip editor framework. Sources: foswiki.org Development + System docs,
`foswiki/distro` on GitHub.
**Read first for shared lineage:** `research/260613-twiki-deep-dive/` (flat-file +
RCS store, Webs/Topics, handler callbacks, per-topic `ALLOW/DENY` ACL). This file
does **not** repeat those.
---
## 1. Origin and compatibility
Forked from **TWiki 4.2.x** in October 2008 by most of the TWiki community (sources
cite 4.2.34.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content
compatibility — same TML markup, same plugin API — via a **`TWikiCompatibilityPlugin`**
that maps the `TWiki::` namespace and legacy variables. So everything in the TWiki
dive still applies; Foswiki then grew the features below.
## 2. The headline delta — a pluggable store behind a versioned interface
This is the reason Foswiki earns its own dive.
### 2.1 `Foswiki::Meta` indirection
After Foswiki 1.0.4 the core was changed to **delegate almost all store operations
to a topic object, `Foswiki::Meta`**. Callers manipulate a `Foswiki::Meta` object
representing the item they're changing and **never touch the store implementation
directly**; `Foswiki::Meta` talks to the real store through the well-defined
**`Foswiki::Store`** interface. This is a clean MVC/repository separation.
### 2.2 Swappable backends
`Foswiki::Store::` ships multiple low-level back-ends behind that one interface:
- **`RcsWrap`** — RCS via the external `rcs` binaries (classic TWiki behavior)
- **`RcsLite`** — a pure-Perl RCS implementation
- **`PlainFileStoreContrib`** — a newer store that saves topics/attachments as
**timestamped copies** instead of RCS diffs: more disk, but **no RCS dependency**
and much higher performance.
So in Foswiki the **storage format is a configuration choice**, not a fixed property
of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core.
### 2.3 OO/MVC core rewrite
Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward
object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving
backward compatibility through tests. The store abstraction above is the most
load-bearing result.
## 3. Extension API deltas
Shared with TWiki: handler callbacks (`initPlugin`, `commonTagsHandler`,
`before/afterSaveHandler`, `afterRenameHandler`, attachment handlers), REST handlers,
`Config.spec`, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements:
- **`Foswiki::Func` is the blessed API** — "if there's a `Foswiki::Func` way and
another way, the `Foswiki::Func` way is almost always right." Direct use of
internal packages is discouraged and governed by **`PluginsApiPolicies`**.
- **`Foswiki::Func::registerTagHandler($tag, \&sub)`** — register a custom
macro/variable **programmatically** (cleaner than TWiki's `commonTagsHandler`
string-matching convention).
- **Contrib-defined APIs** carry an explicit "use at your own risk / awaiting merge"
policy — a maturity signal on extension interfaces.
- `EmptyPlugin` (`lib/Foswiki/Plugins/EmptyPlugin.pm`) is the canonical handler
reference/skeleton.
## 4. DataForms and structured data (delta)
DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi
value, label) stored as `%META:FIELD%` in the topic text. Foswiki extends the model
via popular extensions:
- **MoreFormfieldsPlugin** — additional special-purpose field types.
- **FlexFormPlugin** — render DataForm interfaces from custom templates
(`Foswiki::Form` classes).
- **MetaDataPlugin** — **multiple structured data records per topic** (beyond the
classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model
while keeping data in the text file.
## 5. Editing / syntax (delta relevant to "Markdown-first")
- **WysiwygPlugin** — a generic framework that transforms **TML → HTML** for a
browser editor and **HTML → TML on save** (lossless round-trip).
- **TinyMCEPlugin** — pre-installed WYSIWYG editor built on WysiwygPlugin.
This proves **bidirectional, lossless translation between an engine's native markup
and another representation is feasible** — directly relevant to shard-wiki reading a
non-Markdown shard and writing Markdown overlays back (UC-42).
## 6. Foswiki as a shard — capability profile (delta from TWiki)
| Capability | Foswiki | Note vs TWiki |
|------------|---------|---------------|
| Store backend | **pluggable** | RCS *or* PlainFile behind `Foswiki::Store` — direct-attach target is cleaner on PlainFile (UC-40) |
| History | RCS *or* timestamped copies | PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41) |
| Structured payload | DataForms `%META%`, **multi-record** via MetaDataPlugin | richer than classic TWiki Forms (UC-34/39) |
| Syntax translation | **WysiwygPlugin TML↔HTML** | bidirectional round-trip exists (UC-42) |
| Extension host | `Foswiki::Func` + `registerTagHandler` + REST | cleaner adapter-host surface (UC-38) |
| Store as contract | **`Foswiki::Store` versioned interface** | architectural prior art for shard-wiki's adapter contract |
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
| Observation | INTENT principle |
|-------------|------------------|
| `Foswiki::Store` versioned interface + swappable backends | **capability-aware adapters** / **shard adapter contract** — an engine that already did exactly this separation |
| `Foswiki::Meta` mediates all store access | clean **mechanism over policy** boundary worth mirroring in the adapter contract |
| PlainFile store = timestamped text copies | **git-addressable coordination** / direct-attach (UC-40), history import (UC-41) |
| WysiwygPlugin TML↔HTML round-trip | **Markdown-first, backend-neutral** is achievable for prose via translation (UC-42) |
| MetaDataPlugin multi-record topics | structured pages beyond one form (UC-34/39) |
### 7.2 Deliberate divergences (design bugs if conflated)
| Foswiki assumption | shard-wiki correction |
|--------------------|----------------------|
| One pluggable store per wiki, chosen at config | shard-wiki federates **many** heterogeneous stores at once |
| TML is the canonical syntax; HTML is a render target | shard-wiki is **Markdown-first**; TML↔Markdown is an adapter translation, not core |
| Store interface is Foswiki-internal Perl | shard-wiki's adapter contract is **cross-language, cross-backend, versioned** |
| ACL in topic preferences | **authorize in core**; engine ACL is advisory provenance |
### 7.3 What Foswiki teaches that shard-wiki should not lose
1. **The store-abstraction pattern works in practice**`Foswiki::Store` is a
real-world proof that a stable interface + swappable backends is viable; the shard
adapter contract is the same idea generalized across engines.
2. **Backend format should be swappable under a stable identity** — Foswiki migrates
RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's
backend changing under it (UC-43).
3. **Lossless syntax round-trip is a solved problem** — don't treat non-Markdown
prose as read-only; translate it (UC-42).
## 8. Use-case seeds → catalog (promoted 2026-06-13)
| Seed | Catalog UC | Disposition |
|------|------------|-------------|
| Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown) | **UC-42** (new) | realizes Markdown-first for prose; WysiwygPlugin is proof |
| Tolerate a shard's storage-backend swap without losing identity/provenance | **UC-43** (new) | Foswiki RCS↔PlainFile; orchestration robustness |
| Structured / multi-record pages | UC-34 / UC-39 | enriched: DataForms + MetaDataPlugin multi-record |
| Attach a file-backed engine's on-disk store directly | UC-40 | enriched: PlainFile store is a cleaner direct-attach target than RCS |
| `Foswiki::Store` versioned interface | (no UC) | architecture prior art → `SHARD-WP-0002` adapter contract |
## 9. Open questions (for spec / workplans)
1. **Adapter-contract shape** — how much of `Foswiki::Store`'s method set
(read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's
capability-aware adapter contract? (architecture, `SHARD-WP-0002`)
2. **Syntax-translation fidelity** — is TML↔Markdown lossless enough for overlay
round-trips, or must overlays be stored in TML to be safe (UC-42)?
3. **Backend-swap detection** — how does shard-wiki notice and tolerate a shard
changing store format underneath (RCS→PlainFile), and does provenance/history
survive it (UC-43)?
4. **Multi-record metadata** — represent MetaDataPlugin multi-record topics and XWiki
multi-XObject pages in one structured-metadata page model (shared open question).
## 10. Sources
| Source | URL |
|--------|-----|
| Foswiki — Technical Overview | https://foswiki.org/Development/TechnicalOverview |
| Foswiki — Core Internals | https://foswiki.org/Development/CoreInternals |
| Foswiki — PlainFileStoreContrib | https://foswiki.org/Extensions/PlainFileStoreContrib |
| Foswiki — RCSStoreContrib | https://foswiki.org/Extensions/RCSStoreContrib |
| Foswiki::Store::PlainFile (source) | https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm |
| Foswiki — Developing Plugins | https://foswiki.org/System/DevelopingPlugins |
| Foswiki — Plugins API Policies | https://foswiki.org/Development/PluginsApiPolicies |
| Foswiki — DataForms | https://foswiki.org/System/DataForms |
| Foswiki — MetaDataPlugin | https://foswiki.org/Extensions/MetaDataPlugin |
| Foswiki — WysiwygPlugin | https://foswiki.org/System/WysiwygPlugin |
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
| shard-wiki — TWiki deep dive | `research/260613-twiki-deep-dive/findings.md` |
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
---
## 11. Traceability
| This document section | Informs (future) |
|-----------------------|------------------|
| §2 store abstraction | **adapter contract** design (`SHARD-WP-0002`) — Foswiki::Store as prior art |
| §3 extension API | UC-38 engine-side adapter host |
| §4 DataForms | structured-metadata page model (UC-34/39) |
| §5 WysiwygPlugin | UC-42 syntax-translation adapter |
| §6 capability profile | adapter capability-profile vocabulary |
| §7 INTENT mapping | architecture-blueprint guardrails |
| §8 UC seeds | `spec/UseCaseCatalog.md` (UC-42, UC-43; UC-34/39/40 enrichment) |

View File

@@ -0,0 +1,39 @@
# 260613 — TWiki deep dive (implementation, plugin API, ecosystem)
Date: 2026-06-13
## What this is
A focused study of **TWiki** — the original "structured enterprise wiki / wiki
application platform" (Peter Thoeny, ~1998), and the **file-based Perl counterpoint**
to XWiki's Java/DB platform. Covers TWiki's **implementation** (Perl + flat-file +
RCS store, Webs/Topics, TWiki Forms, TWikiML/variables), the **plugin API and
handler callbacks** that extend the core, the **extension package types** (Plugin /
Skin / AddOn / Contrib) and repository, and TWiki's **per-topic access control**
(the prior art behind yawex's `AccessControl`).
Read through shard-wiki's lens, paired with the XWiki dive: TWiki shows the same
"wiki-as-application-platform" goal reached with **text files + RCS** instead of a
database — which changes the adapter story (direct on-disk attach, git-convertible
history).
Complements:
- `research/260613-xwiki-deep-dive/` — the DB/Java app-platform sibling
- `research/260608-wikiengines-overview/` — the landscape scan
- `research/260608-yawex-prior-art/` — yawex borrowed TWiki's per-topic AccessControl
Note: most of TWiki's community forked to **Foswiki** in Oct 2008 (from TWiki
4.2.4); findings flag where TWiki/Foswiki differ.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, plugin-handler API, ecosystem, capability profile, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Two new use cases promoted to
`spec/UseCaseCatalog.md` (UC-40, UC-41); UC-06/34/36/38/39 enriched with TWiki
specifics. Adapter-contract implications feed
`workplans/SHARD-WP-0002-federation-architecture.md`.

View File

@@ -0,0 +1,210 @@
# Findings — TWiki: implementation, plugin API, ecosystem
Date: 2026-06-13 · Status: research draft
Scope: TWiki as prior art for the **file-based** wiki-application-platform — the
counterpoint to XWiki's database/component platform. Two shard-wiki concerns:
(1) attaching a structured, file-backed engine as a shard (including attaching its
**on-disk store directly** vs through its runtime), and (2) how TWiki's plugin
handler API exposes hooks rich enough to host a federation adapter
(INTENT *composable integration*). Sources: twiki.org dev docs (TWikiPlugins,
TWikiForms, TWiki::Func, TWikiAccessControl), Foswiki docs, Wikipedia.
**Complements:** `research/260613-xwiki-deep-dive/` (DB/Java sibling),
`research/260608-wikiengines-overview/` (landscape), and
`research/260608-yawex-prior-art/` (yawex borrowed TWiki's per-topic AccessControl).
---
## 1. What TWiki is
A **structured enterprise wiki** and **wiki application platform** (Peter Thoeny,
~1998), written in **Perl 5** (5.10.1+), classically run as CGI. Like XWiki it lets
pages become forms/records/apps — but it reaches that with **flat text files +
RCS**, no database. Most of the community forked to **Foswiki** in October 2008
(from TWiki 4.2.4; Foswiki 1.0 in January 2009); the plugin API and data model below
are shared lineage, with minor divergences noted.
## 2. Implementation architecture
### 2.1 Storage — flat files + RCS (no DB)
- Content lives on disk as `data/<Web>/<Topic>.txt`; the directory tree **mirrors
the logical Web/Topic hierarchy**. Attachments live under `pub/<Web>/<Topic>/`.
- **History is RCS** (`<Topic>.txt,v` companion files) — the same GNU RCS yawex used.
History is real, per-file, and in an **open, git-convertible format** (unlike
XWiki's DB-internal `xwikircs`).
- Scales to 300k+ topics on a single server (Yahoo) — file store is not a toy.
### 2.2 Webs and Topics
- **Topic** = page (the unit of content). **Web** = namespace / collection of topics
(can be nested). Webs are the natural shard / root-entity boundary.
### 2.3 TWiki Forms — structured data, embedded in text
- A **form template** is a topic defining fields as a table (one row per field:
name, type, size, values). A topic can *attach* a form; the field values are
stored as **TWikiMetaData inside the topic `.txt`** (`%META:FIELD{...}%`), shown as
a table on view and edited via fields/radios/checkboxes/lists.
- **Forms + formatted `%SEARCH%`** = TWiki's "database applications" — structured
records queried by the SEARCH variable. This is the XWiki XObject/XClass idea, but
**file-embedded and git-diffable** rather than rows in a DB.
### 2.4 TWikiML / variables (macros)
- Markup is **TWikiML** (not Markdown). Dynamic behavior comes from **variables**
`%VAR%` / `%VAR{...}%` expanded at render time; `%SEARCH%` is the query engine.
- Custom variables are provided by plugins (see §3).
### 2.5 TWiki::Func — the official API boundary
- `lib/TWiki/Func.pm` ("TWikiFuncDotPm") documents **all** interfaces available to
plugins, deliberately abstracting the flat-file assumption: `readTopic`,
`saveTopic`, `getTopicList`, `getListOfWebs`, `getWorkArea`, `saveAttachment`, …
Plugins are told *not* to assume flat files and to go through Func.
## 3. Plugin API — interfaces to extend the core
TWiki extends "without altering core code" by **registering Perl handlers that the
core calls at defined points** in the request/render/save pipeline. A plugin is
`lib/TWiki/Plugins/<Name>.pm` + a documentation topic + an optional `Config.spec`.
### 3.1 Handler callbacks (the hook surface)
| Phase | Handlers |
|-------|----------|
| Init / users | `initPlugin` (must return 1), `registrationHandler` |
| Tag/variable expansion | `commonTagsHandler`, `beforeCommonTagsHandler`, `afterCommonTagsHandler` |
| Rendering | `preRenderingHandler`, `postRenderingHandler` (`start/endRenderingHandler` deprecated) |
| Edit lifecycle | `beforeEditHandler`, `afterEditHandler` |
| Save lifecycle | `beforeSaveHandler`, `afterSaveHandler`, `beforeMergeHandler` |
| Attachments | `beforeAttachmentSaveHandler`, `afterAttachmentSaveHandler` |
| Topic mgmt | `afterRenameHandler`, `completePageHandler` |
These are the TWiki analogue of XWiki's `ObservationManager` events — but
**synchronous pipeline hooks** rather than an async event bus. The save/rename/
attachment handlers are exactly the interception points a federation adapter needs.
### 3.2 Other extension mechanisms
- **Custom variables/macros** — registered via `commonTagsHandler` (Foswiki adds
`registerTagHandler`); convention `%PLUGINNAME_SETTING%`.
- **REST handlers** — a plugin registers REST handlers invoked via the `rest`
script, for transactions outside the standard `view/edit/save` scripts. This is the
natural remote adapter transport.
- **`Config.spec`** — declares configuration items (BOOLEAN, STRING, SELECT, PATH,
PERL) surfaced in the `configure` web UI.
- **Work area** — `getWorkArea()` gives a plugin a persistent, non-web-accessible
data directory.
### 3.3 Extension package types
| Type | Role |
|------|------|
| **Plugin** | handler-based behavior, no core change (`lib/TWiki/Plugins/<Name>.pm`) |
| **Skin** | visual appearance only; topic content unchanged (e.g. PrintSkin) |
| **AddOn** | a script in `bin/` callable from a topic |
| **Contrib** | shared library code, or **alternative implementations of core sections** (e.g. user management) when something "can't be a plugin because it needs close core access" |
## 4. Access control (origin of yawex's model)
TWiki authorizes via **preference settings in topics**: `ALLOWWEBVIEW` /
`ALLOWWEBCHANGE` at web level and `ALLOWTOPICVIEW` / `ALLOWTOPICCHANGE` /
`ALLOWTOPICRENAME` (and `DENY*`) at topic level, with **VIEW / CHANGE / RENAME
grantable separately**. Topic-level settings apply only to that topic. This
per-topic ACL is the **direct prior art** behind yawex's `AccessControl`
(denied/view/form/edit/admin) and a concrete reference for shard-wiki's optional
**per-page ACL at L4** (`spec/ArchitectureBlueprint.md` §5).
## 5. TWiki as a shard — capability profile
| Capability | TWiki | Note |
|------------|-------|------|
| Read | ✓ | TWiki::Func / `view` script / `rest`; or read `data/*.txt` directly |
| Write | ✓ | saveTopic / `save` / `rest`; **per-topic** granularity |
| Structured payload | ✓ | TWiki Forms → `%META:FIELD%` **in the text file** (git-diffable) |
| Version/diff | ✓ | RCS `.txt,v`**open format, git-convertible** (cf. XWiki DB) |
| Merge | partial | `beforeMergeHandler`; no git 3-way |
| Change hooks | ✓ | synchronous save/rename/attachment handlers |
| Auth model | per-web/topic ACL | authorize through shard-wiki core; engine ACL is advisory |
| Direct-store attach | ✓✓ | `data/<Web>/<Topic>.txt` + RCS is a folder shard on its own |
| Federation hooks | ✓ | plugin handlers + REST handlers can host an adapter (UC-38) |
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
### 6.1 Reinforcements
| Observation | INTENT principle |
|-------------|------------------|
| Forms store data as `%META%` **in the topic file** | structured pages can be **git-diffable**, strengthening UC-34/39 vs XWiki's DB |
| RCS `.txt,v` history | **git-addressable coordination** — here history is *convertible*, not just supplementable (UC-36 → UC-41) |
| `data/<Web>/<Topic>.txt` mirrors logic | an engine's **on-disk store is itself attachable as a folder shard** (UC-40) |
| Plugin save/rename/REST handlers | **composable integration** — TWiki can host an adapter too, generalizing UC-38 beyond XWiki |
| Per-topic `ALLOW/DENY` ACL | prior art for **per-page ACL at L4**; lineage TWiki→yawex (UC-06) |
| Webs as namespaces / shards | namespace navigation (UC-22); webs as roots |
### 6.2 Deliberate divergences (design bugs if conflated)
| TWiki assumption | shard-wiki correction |
|------------------|----------------------|
| TWikiML + variable expansion is the page | **Markdown-first**; TWikiML render stays out of core |
| ACL lives in topic preference settings | **authorize in core**; topic ALLOW/DENY is advisory provenance |
| History = RCS files next to content | **coordination journal** is the space-level git layer (may *import* RCS) |
| Apps = Forms + `%SEARCH%` rendered by TWiki | represent records **without** depending on TWiki to render/search them |
| Attach = run TWiki and call it | shard-wiki may attach the **bare data dir** with TWiki offline (UC-40) |
### 6.3 What TWiki teaches that shard-wiki should not lose
1. **File-backed engines are the easy, high-value case** — their store is already a
folder of text + open-format history; the adapter can often skip the runtime.
2. **Structured data in text beats structured data in a DB** for federation —
diffable, portable, journal-friendly. Prefer file-embedded metadata where offered.
3. **One backend, two attachment paths** (runtime API vs on-disk store) is a real
capability/consistency trade-off the adapter contract must express.
## 7. Use-case seeds → catalog (promoted 2026-06-13)
| Seed | Catalog UC | Disposition |
|------|------------|-------------|
| Attach a live engine's **on-disk store directly** (vs its runtime API) | **UC-40** (new) | dual-path attach; fidelity/consistency trade-off |
| **Import** an engine's native file-based history (RCS) into the journal | **UC-41** (new) | history *migration* with fidelity, vs UC-36 *supplementation* |
| Engine hosts adapter via its plugin/handler API | UC-38 | enriched: TWiki save/rename/REST handlers generalize it beyond XWiki |
| Structured page carries typed data | UC-34 / UC-39 | enriched: TWiki Forms `%META%`, git-diffable |
| Internal-history engine | UC-36 | enriched: contrast DB (supplement) vs RCS (import, UC-41) |
| Authenticated team wiki / per-page ACL | UC-06 | enriched: TWiki per-topic ACL is the origin of yawex's model |
## 8. Open questions (for spec / workplans)
1. **Dual-path adapters** — when does shard-wiki attach an engine's on-disk store
directly vs go through its API? Consistency risk of reading live files under a
running engine; capability-gate it (UC-40)?
2. **History import fidelity** — can RCS `.txt,v` be converted to git commits
preserving author/timestamp, and is that authoritative or a one-time backfill
feeding the coordination journal (UC-41, cf. UC-36 Q3)?
3. **File-embedded metadata mapping** — map TWiki `%META:FIELD%` (and XWiki XObjects)
onto one structured-metadata page-model representation (shared with xwiki §8 Q1).
4. **TWiki vs Foswiki** — target both via one adapter (shared API) or treat as
distinct shard types?
5. **ACL provenance** — surface a topic's `ALLOW/DENY` as read-only provenance even
though authorization is decided in core?
## 9. Sources
| Source | URL |
|--------|-----|
| TWiki — TWikiPlugins (handlers, package types) | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
| TWiki — Developing Plugins / EmptyPlugin | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
| TWiki — TWiki::Func module | https://twiki.org/cgi-bin/view/TWiki/TWikiFuncModule |
| TWiki — TWiki Forms | https://twiki.org/cgi-bin/view/TWiki/TWikiForms |
| TWiki — TWiki Access Control | https://twiki.org/cgi-bin/view/TWiki/TWikiAccessControl |
| TWiki — System Requirements (Perl/RCS) | https://twiki.org/cgi-bin/view/TWiki/TWikiSystemRequirements |
| TWiki — Source Code | https://twiki.org/cgi-bin/view/TWiki/SourceCode |
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
| Wikipedia — TWiki | https://en.wikipedia.org/wiki/TWiki |
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
| shard-wiki — yawex prior art | `research/260608-yawex-prior-art/findings.md` |
---
## 10. Traceability
| This document section | Informs (future) |
|-----------------------|------------------|
| §2 architecture | adapter design for file-backed engines |
| §3 plugin handlers | UC-38 engine-side adapter; composable-integration API shape |
| §4 access control | per-page ACL at L4 (`spec/ArchitectureBlueprint.md`), UC-06 |
| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) |
| §6 INTENT mapping | architecture-blueprint guardrails |
| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-40, UC-41; UC-06/34/36/38/39 enrichment) |
| §8 open questions | spec — dual-path adapters, history import, structured-metadata model |

View File

@@ -0,0 +1,35 @@
# 260613 — XWiki deep dive (implementation, extension interfaces, ecosystem)
Date: 2026-06-13
## What this is
A focused study of **XWiki** — the canonical "wiki-as-application-platform"
engine flagged in `research/260608-wikiengines-overview/`. Goes past the landscape
scan into XWiki's **implementation** (component/DI architecture, document data
model, storage, rendering, events), the **interfaces it exposes to extend the core
engine** (components, macros, UI extension points, script services, REST), and the
**extension ecosystem** (extensions.xwiki.org repository, XAR/JAR/WebJar types,
notable extensions).
Read through one lens: **what does XWiki teach shard-wiki about (a) adapting a
structured app-platform engine as a shard, and (b) how a host engine can become
federation-capable through its own extension mechanism** (INTENT *composable
integration*).
Complements:
- `research/260608-wikiengines-overview/` — the landscape scan that surfaced XWiki
- `research/260608-federation-concepts/` — already covered XWiki's ActivityPub extension
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, extension-interface catalog, ecosystem, capability profile, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Two new use cases promoted to
`spec/UseCaseCatalog.md` (UC-38, UC-39); existing UC-34/UC-36/UC-31 enriched with
XWiki specifics. Adapter-contract implications feed
`workplans/SHARD-WP-0002-federation-architecture.md`.

View File

@@ -0,0 +1,217 @@
# Findings — XWiki: implementation, extension interfaces, ecosystem
Date: 2026-06-13 · Status: research draft
Scope: XWiki as **prior art for two shard-wiki concerns** — (1) attaching a
structured "wiki-as-application-platform" engine as a shard, and (2) how an engine
exposes extension interfaces rich enough to *host* a federation adapter
(INTENT *composable integration*). Sources are XWiki's own dev docs, the
DeepWiki code analysis of `xwiki/xwiki-platform`, and the extensions repository.
**Complements:** `research/260608-wikiengines-overview/` (landscape) and
`research/260608-federation-concepts/` §3.2 (XWiki ActivityPub).
---
## 1. What XWiki is
A **programmable, component-based wiki platform** (Java/Jakarta EE, Hibernate ORM)
that blurs wiki and lightweight application platform. Pages are not just prose:
they carry **typed structured objects**, so an XWiki page can be a form, a record,
or a small application. This is the defining property the landscape scan flagged —
and the reason XWiki is the hardest case for shard-wiki's Markdown-first page model.
## 2. Implementation architecture
### 2.1 Component system (the spine)
- **Dependency-injection component model.** Components are interfaces (roles)
discovered via `META-INF/components.txt` and instantiated by a `ComponentManager`.
Implementations declare `@Component`, an optional `@Named` **role hint**, and
`@Singleton`. No interface→impl dependency — impls can ship in separate JARs and
**replace defaults at runtime**.
- Core replaceable roles include `XWikiStoreInterface` (storage), `ObservationManager`
(events), `XWikiRightService` (authorization), `XWikiAuthService` (authentication).
### 2.2 oldcore vs modular platform
- `xwiki-platform-oldcore` holds the foundational document model, persistence, and
legacy APIs (`com.xpn.xwiki.*`). A bridge connects it to the newer component world
to avoid circular dependencies.
- Newer modules layer on top: `model` (entity references), `rendering`, `security`,
`web` (HTTP actions/templates), `search-solr`.
### 2.3 Data model — document-centric + class/object
- Central entity: **`XWikiDocument`**, addressed by `DocumentReference`
(wiki / space / page). Content is parsed into an **XDOM** (rendering tree).
- **Structured data via a class-object pattern** (OOP-like):
- `BaseClass` / **XClass** — schema (field definitions), like a class declaration
- `BaseObject` / **XObject** — an instance conforming to a class, attached to a page
- `PropertyClass` / `BaseProperty` — field definition / value
- Property types: String, LargeString, Date, Number, List, …
- A single page can hold content **plus N typed objects** plus attachments. This is
the structural fact shard-wiki must represent without flattening.
### 2.4 Storage & history
- Two-tier: **`XWikiCacheStore`** over **`XWikiHibernateStore`** (Hibernate, multi-DB).
- Tables: `xwikidoc`, `xwikiobject`, `xwikiproperties`, `xwikiattachment`, and
**`xwikircs`** — internal **RCS-style version history**. History is real but
**engine-internal and non-portable** (directly relevant to shard-wiki UC-36).
### 2.5 Events & rendering
- **`ObservationManager`** event bus: cancelable (`DocumentCreatingEvent`,
`DocumentUpdatingEvent`, `DocumentDeletingEvent`) and post-facto
(`DocumentCreatedEvent`, `DocumentUpdatedEvent`, …). Listeners are components —
a natural **change-notification source for incremental sync**.
- **Rendering pipeline:** source syntax → parse → XDOM → transformations (macros) →
serialize (HTML, PDF, …). Pluggable parsers/renderers; XWiki can parse **Markdown**
as an input syntax even though its native syntax differs.
### 2.6 Multi-wiki
- One installation hosts many wikis (`WikiReference`), each with separate documents
and configuration — XWiki's own multi-tenancy, conceptually adjacent to
shard-wiki's root-entity/tenant boundary.
### 2.7 APIs & security
- **Dual API surface:** internal `com.xpn.xwiki.*` (full power) and script
`com.xpn.xwiki.api.*` (wrapped, enforces **programming rights** and view rights).
Privilege is enforced at the script boundary, not the storage boundary.
## 3. Interfaces to extend the core engine
XWiki's extensibility is unusually layered. From lowest/most-powerful to highest:
| # | Mechanism | What it extends | How |
|---|-----------|-----------------|-----|
| 1 | **Components** | anything | implement a role interface, `@Component` + `@Named` hint; ship in a JAR with `components.txt`; can override defaults (store, auth, rights) |
| 2 | **Rendering macros (Java)** | content processing | register `Macro.class` role + hint; participates in the XDOM transformation pipeline |
| 3 | **Wiki macros** | content processing | a wiki page following a spec — **no Java, no compile**; in-wiki authoring |
| 4 | **Script services** | scripting API | implement `org.xwiki.script.service.ScriptService`; exposed to Velocity/Groovy as `$services.<hint>` |
| 5 | **UI Extensions (UIX)** at **UI Extension Points (UIXP)** | the interface, no skin edit | UIX stored as XObjects of `XWiki.UIExtensionClass` (label/target/icon); injected at named hooks (since v4.2), e.g. `org.xwiki.platform.panels.Applications` |
| 6 | **REST API (JAX-RS)** | remote integration | resources for pages, spaces, **objects**, classes, attachments, comments, tags, annotations, users/groups, **history**; custom JAX-RS resources can be added |
| 7 | **Event listeners** | reactive behavior | register a component on `ObservationManager` to react to document lifecycle events |
| 8 | **Resource reference handlers** | new URL/resource types | implement `ResourceReferenceHandler` on the root component manager |
| 9 | **Skins / templates / Velocity** | presentation | Flamingo skin macros, templates |
Two things stand out for shard-wiki: (a) the **component model lets an extension
replace authentication, authorization, and storage** — i.e. an XWiki instance could
delegate auth exactly as shard-wiki's blueprint prescribes; (b) the **REST API plus
event bus** give a federation adapter both a transport (read/write pages+objects+
history) and a push-based freshness signal.
## 4. Extension ecosystem
- **`extensions.xwiki.org`** repository: **900+ extensions** (applications, macros,
skins, panels, themes, legacy plugins). Browsed/installed/upgraded at runtime via
the **Extension Manager Application** from the admin UI.
- **Extension types:**
- **XAR** — a package of wiki pages (an application, a wiki macro, or a snippet)
- **JAR** — server-side Java: **components and script services**
- **WebJar** — JAR of web resources (JS/CSS/Less)
- **"Plugin" (legacy) vs Component (modern):** the old plugin API is superseded by
the component system; new code is components.
- **Notable extensions** (relevance to shard-wiki in parentheses):
- **AppWithinMinutes** — build structured apps (XClass+XObjects) from the UI
(→ pages-as-records, the UC-39 case)
- **LDAP / Active Directory** and **Entra ID (Azure AD)** authentication
(→ confirms pluggable `XWikiAuthService`; mirrors shard-wiki's delegated authn)
- **ActivityPub** — fediverse federation (→ adapter transport, cf. UC-31)
- **LiveData / LiveTable** — queryable tabular views over XObjects
- draw.io diagrams, PDF viewer, Task Manager, Blog — content/app breadth
## 5. XWiki as a shard — capability profile
Read through shard-wiki's capability-aware adapter lens:
| Capability | XWiki | Note |
|------------|-------|------|
| Read | ✓ | REST + render to HTML/Markdown |
| Write | ✓ | REST PUT page/object; **per-document** granularity (page + its objects) |
| Structured payload | ✓✓ | XObjects/XClass — *richer* than Markdown; must not be flattened (UC-34/UC-39) |
| Version/diff | ✓ (internal) | `xwikircs`; **not git-portable** → coordination journal supplies that (UC-36) |
| Merge | partial | no native git-style 3-way merge |
| Change events | ✓ | `ObservationManager` → push-based sync/freshness (enriches UC-31) |
| Auth model | pluggable | `XWikiAuthService`/`XWikiRightService` replaceable — authorize through shard-wiki core, don't trust engine ACLs |
| Federation hooks | ✓✓ | component + REST + UIX make XWiki able to **host** a shard-wiki adapter (UC-38) |
| Multi-tenancy | ✓ | `WikiReference` farm → multiple shards / root entities |
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
### 6.1 Reinforcements
| Observation | INTENT principle |
|-------------|------------------|
| Pages = content + typed XObjects | **Markdown-first, backend-neutral** must carry structure, not flatten (UC-34/39) |
| `xwikircs` internal-only history | **Git-addressable coordination** adds the portable layer (UC-36) |
| Pluggable `XWikiAuthService` | **authorization in core, authentication delegated** — XWiki already does exactly this |
| Component + REST + UIX extension surface | **composable integration** — engine becomes federation-capable via its own API (UC-38) |
| `ObservationManager` events | **explicit provenance/freshness** via push, not poll (UC-31, UC-24) |
| `WikiReference` multi-wiki | root-entity / tenant boundary has prior art |
### 6.2 Deliberate divergences (design bugs if conflated)
| XWiki assumption | shard-wiki correction |
|------------------|----------------------|
| Native XWiki syntax + macro render is the page | shard-wiki keeps **Markdown-first**; engine render stays out of core, structure travels as metadata |
| ACLs/rights live in the engine DB | **authorize in core**; engine rights are advisory, not trusted |
| History is the engine's RCS table | **coordination journal** is space-level and git-backed |
| Federation = an XWiki extension talking to other XWikis | shard-wiki federates **heterogeneous** backends; XWiki is one shard among many |
| App = XObjects rendered by XWiki | shard-wiki must represent records **without** depending on XWiki to render them |
### 6.3 What XWiki teaches that shard-wiki should not lose
1. **A clean role/hint component model** is what makes auth/store/rights swappable —
shard-wiki's adapter contract should be similarly role-based and override-friendly.
2. **Structured objects are first-class**, not an afterthought — the page model needs
a typed-metadata escape hatch from day one.
3. **An engine with a real extension API can host the adapter itself** — the cheapest
path to a high-fidelity shard is engine-side, not screen-scraping.
## 7. Use-case seeds → catalog (promoted 2026-06-13)
| Seed | Catalog UC | Disposition |
|------|------------|-------------|
| Engine hosts a federation adapter via its native extension API | **UC-38** (new) | composable integration — first UC for the engine-side direction |
| Attach a wiki-as-application-platform shard (pages are typed records/forms) | **UC-39** (new) | extends UC-34 toward bodiless structured pages |
| Structured page carries semantic data | UC-34 | enriched with XObject/XClass concretes |
| Internal-only revision history | UC-36 | enriched with `xwikircs` example |
| Subscribe to remote shard changes | UC-31 | enriched with `ObservationManager` event-driven transport |
## 8. Open questions (for spec / workplans)
1. **Page-model representation of XObjects** — typed frontmatter, sidecar file, or
opaque provenance blob? (shared with `wikiengines` §6 Q1; XWiki makes it urgent)
2. **Bodiless record pages** — does shard-wiki's page model require a Markdown body,
or can a page be purely structured (AppWithinMinutes apps)?
3. **Adapter placement** — engine-side extension (high fidelity, needs deploy access)
vs external REST adapter (zero engine change, lower fidelity). Both? Capability-gated?
4. **Event transport** — consume `ObservationManager` via a JAR listener, or poll REST
history? Affects freshness guarantees (UC-31/UC-24).
5. **Rights mapping** — translate XWiki `XWikiRightService` rights into shard-wiki
actions (read/write/patch/merge/administer), or ignore and authorize purely in core?
## 9. Sources
| Source | URL |
|--------|-----|
| XWiki — Developer Guide | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/ |
| XWiki — Creating XWiki Components | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/WritingComponents/ |
| XWiki — Available Extension Points | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/ExtensionPoint/ |
| XWiki — UI Extension Point Tutorial | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/UIXTutorial/ |
| XWiki — Wiki Macro Tutorial | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/WritingMacros/WikiMacroTutorial/ |
| XWiki — RESTful API | https://www.xwiki.org/xwiki/bin/view/Documentation/UserGuide/Features/XWikiRESTfulAPI |
| XWiki Rendering — Writing a Macro | https://rendering.xwiki.org/xwiki/bin/view/Main/ExtendingMacro |
| Extension Manager Application | https://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Manager%20Application |
| Extension Module — Extensions/types | https://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Module/Extensions/ |
| DeepWiki — xwiki/xwiki-platform (code analysis) | https://deepwiki.com/xwiki/xwiki-platform |
| Mancinelli — Overview of XWiki's RESTful API | http://blog.fabio.mancinelli.me/2011/03/07/XWikis_RESTful_API.html |
| shard-wiki — wiki engines overview | `research/260608-wikiengines-overview/findings.md` |
| shard-wiki — federation concepts | `research/260608-federation-concepts/findings.md` |
---
## 10. Traceability
| This document section | Informs (future) |
|-----------------------|------------------|
| §2 architecture | adapter design for app-platform engines |
| §3 extension interfaces | UC-38 engine-side adapter; composable-integration API shape |
| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) |
| §6 INTENT mapping | architecture-blueprint guardrails |
| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-38, UC-39; UC-31/34/36 enrichment) |
| §8 open questions | spec — page model for structured/bodiless pages |

View File

@@ -0,0 +1,38 @@
# 260614 — The computational page model (SHARD-WP-0004 synthesis)
Date: 2026-06-15 · Source: **SHARD-WP-0004** post-batch synthesis (T1T8)
## What this is
The acceptance-criteria synthesis for SHARD-WP-0004 — *"the computational page model"*
reading the eight computational/interactive-knowledge dives across each other and distilling
them into **one model**: **source is canonical; everything rendered/computed is a
projection**, placed on **two axes** (projection-kind: replication vs derivation; liveness:
live↔snapshot), with a recommendation on an executable-content capability.
## The answer to the carried question
*Can a shard-wiki page be a live computational artifact?* **Yes — as a page-model +
projection concern, not as an execution platform.** Every system externalizes to a canonical
source and treats the live/computed form as derived; shard-wiki **recognizes** computational
content, **attaches the source**, and **presents derivations as provenance- and
liveness-marked projections**, with **execution as a gated capability (off by default,
degrade to snapshot)**. No INTENT amendment required.
## Key contributions
- **One model:** `(source, derivation rule, projection with provenance + liveness)` covers
all four computational page shapes (one-source-many-projections UC-83; notebook UC-84;
program-as-page; live/temporal content).
- **Two axes for T16:** replication vs **derivation-projection** (timing / multiplicity /
continuity facets) × the **live↔snapshot** axis (bounded at the irreducibly-live far end by
Strudel).
- **One snapshot-provenance record** reused for notebook outputs, renders, recordings.
- **Hard boundaries:** never host a kernel/runtime as store; **image-is-not-a-store**; never
present a derivation without output→source provenance.
## Contents
| Path | Role |
|------|------|
| `findings.md` | The source/derivation/projection model, the two axes, the four page shapes, provenance/reproducibility, the recommendation, the SHARD-WP-0002 fold-in, escalated open questions |

View File

@@ -0,0 +1,178 @@
# The computational page model — synthesis (SHARD-WP-0004)
**Date:** 2026-06-15 · **Source:** SHARD-WP-0004 (post-batch synthesis, T1T8) · **Kind:**
synthesis (no new external research) reading the eight computational/interactive-knowledge
dives *across* each other.
## What this is
The acceptance-criteria synthesis for SHARD-WP-0004. The batch asked one carried question:
*can a shard-wiki page be a live computational artifact — a source woven/evaluated into
rendered forms — and if so, how do projection, transclusion, provenance, and the adapter
contract treat the source, the environment, and the computed output?* This memo answers it
by distilling the eight dives into **one model**: the **source / derivation / projection**
view of computational content, anchored on two axes, plus a recommendation on whether
shard-wiki needs an executable-content capability.
Dives consolidated: **T1** literate programming (WEB/weave/tangle), **T2** Mathematica,
**T3** Jupyter, **T4** Processing/p5.js, **T5** Strudel, **T6** Squeak, **T7** Glamorous
Toolkit, **T8** Pharo. Catalog yield: **UC-83** (literate one-source-many-projections),
**UC-84** (notebook with computed-output provenance); the other six are **enrichment/boundary**
dives (UC-54/55/47/48 + the projection model).
## 1. The one finding: source is canonical, everything rendered is a projection
Every system in the batch, however "live," **externalizes to a canonical artifact** and
treats the rendered/computed form as **derived**:
| System | Canonical (attach this) | Derived (a projection) |
|--------|-------------------------|------------------------|
| Literate (WEB) | the WEB/`.nw`/`.org` **source** | woven docs **and** tangled code |
| Mathematica | the `.nb` (a Wolfram expression) | cached `Output` cells; CDF; `Dynamic` widgets |
| Jupyter | `.ipynb` source (ideally paired text) | embedded outputs; nbconvert/nbviewer renders |
| Processing | the **sketch source** | the view-time canvas render |
| Strudel | the **pattern source** | the live audio performance / a recording |
| GT / Lepiter | git-versionable **page/Tonel files** | moldable `gtView`s; live snippet results |
| Squeak/Pharo | exported **files** (Tonel/git) | the live image / running objects |
**This is the same principle shard-wiki already holds** (files-canonical, index/render
derived — ikiwiki UC-79, Logseq UC-62, nbstripout). The computational batch did not break it;
it **stress-tested it to the live extreme and it held**. *The image (Squeak/Pharo) is the
only would-be exception, and it is a boundary, not a counterexample: an image is not a store.*
## 2. Two axes the contract must add
The batch refines the **projection model** (SHARD-WP-0002 T16) with two orthogonal axes:
### 2.1 Projection kind: replication vs derivation
- **Replication-projection** (the existing default): a lazy **cache/copy** of remote content
(Obsidian/Notion mirrors, UC-53/57).
- **Derivation-projection** (new, from T1): a **transform/compile/weave/evaluate** of a
source into rendered forms — regenerable, may delegate to the source's tool, **degrades to
a captured snapshot** when the tool is absent. Covers weave/tangle, nbconvert, CDF, sketch
render, audio recording, `gtView`.
Sub-facets of derivation (from T4): **materialization timing** = *ahead-of-time* (CDF,
nbconvert, static HTML) vs *view-time* (Processing/Strudel); **multiplicity** = one output
(UC-79) vs **N co-equal projections** (UC-83 weave+tangle; UC-47/48/54 moldable views).
### 2.2 Liveness: the live↔snapshot axis (from T6, bounded by T5)
Every derived view sits on a spectrum, and the more live the source, the more its static form
is a clearly-marked degrading snapshot:
```
static source ── captured output ── live-over-files ── view-time one-shot ── continuous/
(literate) (notebook UC-84) (GT/Lepiter) (Processing) interactive
── irreducibly
live/temporal
(Strudel: source
+ recording only)
```
**Honesty rule (union-without-erasure):** a computed/live view must always declare *what it
is* — "captured at run N, environment unguaranteed" (UC-84), "one performance, time T, source
rev R" (UC-83/Strudel), "live render needs the runtime." Never present a snapshot as live or a
static page as capturing a live artifact.
## 3. The computational page-model shapes (T12)
The batch adds these page shapes (beyond prose / typed records / query-defined / inline-
embedded objects / typed-graph already catalogued):
1. **One-source-many-projections** (UC-83) — a source whose presented forms are co-equal
derivations (docs + code), each with output→source provenance; **named-chunk transclusion**
assembles fragments by name at derivation time (UC-32/44).
2. **Notebook** (UC-84) — **ordered/nestable cells** (Mathematica adds the outline tree)
where code cells own **embedded computed outputs** (the derived output is stored *inside*
the source) with **weak execution provenance**; outputs may be MIME blobs or **structured
re-evaluable values** (Mathematica) — a new point on the content-opacity spectrum.
3. **Program-as-page** (Processing) — canonical content = **source text**, presentation = an
**executable render** with **no cached output**; non-Markdown executable content.
4. **Live/temporal/generative content** (Strudel) — source canonical, render irreducibly
live; static = source + a marked recording.
All four reduce to **(source, derivation rule, projection with provenance + liveness)** — one
model, four positions.
## 4. Provenance & reproducibility (the honest weakness)
Computed output provenance is **real but fragile** everywhere: Jupyter/Mathematica
`execution_count`/`In`-`Out` can be **out-of-order**; **environment/versions/data are not
captured**; Strudel/Processing may be **non-deterministic**. Implication for the contract:
treat a computed output as a **snapshot with declared, incomplete provenance** (run id, source
rev, timestamp; environment "unguaranteed"), reusing **one snapshot-provenance machinery**
across notebooks, recordings, and renders (UC-84 is the template). This is consistent with
shard-wiki's existing "surface freshness/completeness, never imply more than you have"
(Oddmuse partial-history UC-82).
## 5. The recommendation
**Does shard-wiki need an executable/computational content capability? — Yes, but only as
recognition + projection + capability-gating, never as an execution engine.**
1. **Adopt the source/derivation/projection model** (no execution required). shard-wiki
**recognizes** computational content types (literate source, notebook, sketch, pattern),
**attaches the canonical source**, and **presents derived forms as projections with
provenance + liveness markers**. This alone delivers UC-83/UC-84 and the enrichment of
UC-54/55 — and needs **no kernel, no sandbox, no runtime**.
2. **Make execution a capability, off by default.** "Drive a derivation" (run tangle/weave,
re-execute a notebook, render a sketch, evaluate a pattern in the viewer) is a **gated
capability** with a **trust/sandbox** sub-concern (T11). Absent it, **degrade to the
captured snapshot / static render / recording** — the graceful-degradation rule, which the
batch shows always has an honest fallback (source is tiny and diffable everywhere).
3. **One projection model, two axes** (T16): projection-kind (replication vs derivation; with
timing + multiplicity facets) × liveness (live↔snapshot). The **moldable view registry**
(T7) is the unifying structure — an open, type-keyed set of co-equal projections, none
canonical-by-fact (display-canonical is policy).
4. **One snapshot-provenance record** reused for notebook outputs, renders, and recordings
(run id, source rev, timestamp, environment "unguaranteed").
5. **Hard boundaries** (design-bugs if violated): never host a kernel/runtime as the store;
**image-is-not-a-store** (attach exported files); never present a derivation without
output→source provenance; never imply a static view captures a live artifact.
**Net:** computational content is **in scope as a page-model + projection concern**, **out of
scope as an execution platform** — exactly the mechanism-over-policy, capability-aware,
degradable posture INTENT already mandates. No INTENT amendment is required; this extends the
page model and projection model within existing constraints.
## 6. Fold into SHARD-WP-0002
- **T12 (page model):** add the four computational shapes (§3); allow nestable cells and
structured re-evaluable outputs; "derived output may live inside the source" (notebook).
- **T16 (projection):** the **two-axis model** (§2) + the **moldable view registry** (§3/T7);
materialization-timing, multiplicity, continuity, and the live↔snapshot far end as explicit
projection metadata.
- **T11 (capabilities):** "derive/execute/render/evaluate" as gated capabilities with trust/
sandbox; default off → snapshot.
- **T15 (fidelity):** non-Markdown executable/computed content; lossy renders; the
structured-re-evaluable-value point on the content-opacity spectrum.
- **T13 (history):** paired-text (Jupytext) / cell-aware (nbdime) strategies for embedded-
output documents; outputs-as-derived (nbstripout ethos).
- **T14 (binding):** **image-is-not-a-store** boundary (export→files only).
## 7. Open questions (escalated)
1. Is **liveness** (and "irreducibly live / no faithful static form") an explicit first-class
metadata flag on every projection, so the union renders the honest fallback automatically?
(T5/T6 far-end question.)
2. Does shard-wiki **ever drive a derivation** (sandboxed), or strictly attach + present
snapshots? (Recurs UC-56/UC-83/UC-84/T4/T5 — a single capability/trust policy decision.)
3. Is a computed output's **structured re-evaluable value** (Mathematica/Wolfram) modeled as a
typed value or stored opaquely with provenance? (UC-55 open-Q #10; UC-84 Q3.)
4. Should **UC-83** and **UC-84** eventually merge as two positions of one "source +
derivations" shape, or stay distinct? (Kept distinct: UC-84's defining trait is *output
embedded in source with weak provenance*; UC-83's is *N co-equal external derivations*.)
## 8. Sources
The eight SHARD-WP-0004 dives: `research/260614-{literate-programming,mathematica,jupyter,
processing,strudel,squeak-pharo,glamorous-toolkit}-deep-dive/`. Prior projection/structure
anchors: `research/260614-{ikiwiki,logseq,zigzag}-deep-dive/`,
`research/260614-shard-spectrum-synthesis/`.
## 9. Traceability
No new UC (consolidation). Consolidates **UC-83, UC-84** and enrichments to **UC-32, UC-44,
UC-47, UC-48, UC-54, UC-55, UC-79, UC-37, UC-35**. Feeds **SHARD-WP-0002 T11/T12/T13/T14/T15/
T16** (see §6). Recommendation: computational content is **in scope as page-model + projection
mechanism, out of scope as an execution platform**; no INTENT amendment required.

View File

@@ -0,0 +1,20 @@
# 260614 — Federated Wiki deep dive
Deep dive on Ward Cunningham's **Federated Wiki** (Smallest Federated Wiki / SFW,
2011 →) as a **federation model** rather than a single shard: fork-with-provenance,
the per-page **JSON journal** of semantic actions, the **story** of typed items, the
**neighborhood/roster** discovery model, and time-bounded **happenings**.
This is prior art for shard-wiki's **coordination layer itself** — the closest existing
system to "a union of pages preserving provenance, assembled non-destructively." It
extends `research/260608-federation-concepts/` §3 with the concrete data model + protocol.
- `findings.md` — full writeup: data model, journal/action types, federation protocol,
capability profile, INTENT mapping, UC seeds (UC-70UC-72), architecture notes for
SHARD-WP-0002, open questions, sources, traceability.
Catalog yield: UC-70 (attach a fedwiki site via page-JSON + CORS), UC-71 (append-only
semantic action journal with site provenance as a coordination-journal model), UC-72
(fork-with-site-provenance federation across a neighborhood / chorus). Enriched
UC-26/28/30/05/27. Feeds SHARD-WP-0002 T1T5 (federation) and T11/T13/T16 (write
granularity, log-based merge, identity≠placement).

View File

@@ -0,0 +1,240 @@
# Federated Wiki — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T1 · **Subject:** Ward Cunningham's
Smallest Federated Wiki (SFW) / Federated Wiki (fedwiki ecosystem).
## Why this dive
Every prior dive has been a *shard candidate* — a store we might attach. Federated Wiki
is different: it is a **federation model**, the one piece of public prior art whose core
job is the same as shard-wiki's coordination layer — *present a union of pages from many
independent sites while preserving where each came from, and let people copy and edit
non-destructively*. Ward Cunningham (inventor of the wiki) built SFW in 2011 precisely to
fix the original wiki's single-canonical-page weakness with **fork + provenance**. We go
past the surface (`260608-federation-concepts/` §3) into the data model and protocol, then
ask what shard-wiki should adopt.
**Framing:** fedwiki is not just "a shard we attach" — it is a *worked example of the
coordination journal, overlay-before-mutation, and union-without-erasure*, three of our
own design pillars, shipped and running.
---
## 1. The data model — page = title + story + journal
A fedwiki page is a small JSON object with three core fields (plus optional decoration):
```json
{
"title": "Welcome Visitors",
"story": [
{ "type": "paragraph", "id": "7b56f22a4b9ee974",
"text": "Welcome to this [[Federated Wiki]] site." },
{ "type": "image", "id": "a1c0e3...", "url": "...", "caption": "..." }
],
"journal": [
{ "type": "create", "id": "7b56f22a4b9ee974", "item": {...}, "date": 1310000000000 },
{ "type": "add", "id": "a1c0e3...", "item": {...}, "after": "7b56f22a4b9ee974",
"date": 1310000100000 },
{ "type": "edit", "id": "7b56f22a4b9ee974", "item": {...}, "date": 1310000200000 },
{ "type": "fork", "site": "ward.fed.wiki.org", "date": 1310000300000 }
]
}
```
- **story** — an *ordered array of typed items* ("paragraph-like" items). Each item is
`{ type, id, text, ...type-specific }`. The **`id`** is a random 16-hex string,
**stable across edits** (it is the unit of identity within a page). The **`type`** names
the **plugin** that renders/edits the item (`paragraph`, `image`, `html`, `markdown`,
`code`, `method`, `pagefold`, chart plugins, …). *Data lives in the item; behavior lives
in the plugin* — the item is portable JSON; the plugin is the renderer.
- **journal** — an *ordered, append-only array of action objects* that, when replayed,
**reconstructs the story**. The story is a materialized view of the journal. This is the
key architectural choice: **the journal is the source of truth, the story is derived.**
## 2. Journal action types — a semantic op-log
Each journal entry is an action with `{ type, ... , date }` (epoch-ms). The action types:
| action | fields | meaning |
|---------|--------|---------|
| `create`| `id, item, date` | first item — page born |
| `add` | `id, item, after, date` | insert an item after another |
| `edit` | `id, item, date` | replace an item's content (id preserved) |
| `move` | `order, date` | reorder items |
| `remove`| `id, date` | delete an item |
| `fork` | `site, date` | **mark that the page was copied from `site` at this point** |
Two things matter for us:
1. **These are *semantic* operations** (add/move/edit/remove a paragraph), not text diffs
and not character-level CRDT ops. The write granularity is the **story item
(paragraph)** — a *middle* granularity between whole-file (TiddlyWiki) and
block/character (Logseq/CRDT). It is an **op-log** like a CRDT, but the ops are
coarse-grained and **applied by humans via fork**, not auto-merged.
2. **`fork` is the provenance primitive.** When you copy a remote page to your own site,
a `fork` entry is appended recording the **source site** and time. The journal of a
forked page therefore **serializes a directed acyclic graph (DAG)** of where content
came from — "the journal of a forked page is detailed enough to recognize where in the
journal of the original the fork took place" (CouchDB-style per-entry sequence numbers
make the cut-point identifiable). History visualization highlights the forked entry.
## 3. The federation protocol — sites, neighborhood, roster
- **Site** = an independent server (originally Node.js; also static-file and serverless
variants). A site owns a set of pages, each served as **page JSON over HTTP** at
`/<slug>.json`, with **CORS headers** so a *browser-side* client can fetch pages from
**any** site. Page identity within a site is the **slug** (a title-derived kebab name).
- **The client assembles the union, not the server.** The fedwiki client ("the lineup")
renders pages **side by side**: clicking a link opens that page *from whatever site it
resolves against*, appended to the right. Browsing literally builds a left-to-right
trail across sites.
- **Neighborhood** = the dynamic set of sites encountered in the current session (from the
sites of pages you've opened, links, and forks). **Search runs across the neighborhood**
— a federated search over exactly the sites you've touched.
- **Roster** = an explicit, authored list of sites to include (a curated neighborhood);
"sister sites" are peers you watch. There is **no central registry** — discovery is by
link, fork, and roster.
- **Happenings** = time-bounded collaborative events where many participants fork around a
topic for a period, producing a burst of related forks (a bounded collaboration that
leaves a durable forked record on each participant's own site).
## 4. The editorial model — fork, don't edit-in-place
You can only write to **your own** site. To change someone else's page you **fork** it
(copy into your site, journal records the source), then edit your copy. Many forks of the
same page coexist across sites — Cunningham's **"chorus of voices"**: *no canonical
version*, divergence is normal and visible, and you choose whose changes to pull by forking
them. There is **no automatic merge** — reconciliation is human: compare journals, fork the
version you prefer, optionally re-fork upstream changes.
---
## 5. Capability profile
| Dimension (synthesis spectrum) | Federated Wiki |
|--------------------------------|----------------|
| Attachment mode | **REST/file-store hybrid** — page JSON over HTTP+CORS; also static files |
| Addressing granularity | **story item (paragraph)** via stable 16-hex `id` |
| Content identity | item `id` random+stable; page id = site + slug |
| Identity vs placement | **placement-bound**: identity = `site` + `slug`; forks are *new* identities linked by journal provenance |
| Structure | ordered array of **typed items** (plugin-typed) |
| History | **per-page append-only journal** of semantic actions (op-log) |
| Merge model | **fork + manual journal compare** — a *third model* beside git 3-way and CRDT auto-merge |
| Native query | none built-in; **neighborhood search** (federated full-text across touched sites) |
| Translation | item `text` is wiki/Markdown-ish; plugins own their formats |
| Attachment/write granularity | **story-item level** (add/edit/move/remove one item) |
| Operational envelope | tiny servers, browser-driven; CORS is the whole API surface |
| Access grant | **own-site-only writes**; reads open via CORS |
| Content opacity | transparent JSON (no E2EE); plugin-typed but inspectable |
| Provenance | **first-class**`fork` records source site; journal = provenance DAG |
## 6. INTENT mapping
### Reinforcements (fedwiki validates our pillars)
- **Coordination journal** (INTENT) ≈ fedwiki **journal**. Our journal idea is *exactly*
fedwiki's per-page append-only action log — and fedwiki proves the story-as-derived-view
pattern works. Strong reinforcement; adopt the **semantic-op + provenance-entry** shape.
- **Overlay before mutation** ≈ **fork**. Fork *is* the canonical overlay: a
non-destructive copy onto a writable surface, recording provenance, before any change.
- **Union without erasure** ≈ **neighborhood + chorus**. The union is assembled from many
sovereign sites; provenance (which site, forked-from) is never hidden; divergence is
surfaced, not resolved away.
- **No silent remote mutation** ≈ **own-site-only writes**. You structurally *cannot*
mutate a remote; you fork to your own site. This is our rule, enforced by architecture.
- **Mechanism over policy** ≈ **no canonical source**. Fedwiki ships the mechanism (fork,
journal, neighborhood) and leaves "which version wins" entirely to people.
- **Graceful degradation** ≈ static-file sites — a fedwiki site can be a read-only pile of
JSON files; still forkable, still in the neighborhood.
### Divergences (boundaries / design notes, not bugs)
- **Identity = placement.** Fedwiki page identity is `site` + `slug`; a fork is a *new*
page whose only tie to the origin is a journal `fork` entry. shard-wiki wants
**identity ≠ placement** (the "same" page across shards under a stable identity, T16) —
so we treat fedwiki's journal-linked forks as *provenance edges*, and layer our own
cross-shard identity over them rather than adopting slug-as-identity.
- **No query / no typed-record model.** Fedwiki is paragraphs+plugins, not a typed DB
(contrast Notion/Wikibase). Fine — it sits at the *coordination* end, not the structure
end. We don't ask fedwiki to provide query; the neighborhood search is the model for
*federated* search across shards (T-federation), not in-shard query.
- **Browser-assembles-union.** Fedwiki pushes union assembly to the client. shard-wiki
assembles server/orchestrator-side. Adopt the *model* (union from sovereign sources +
provenance), not the client-only locus.
### What to keep
1. **Journal = append-only semantic-op log with provenance entries**, story = derived
replay view. This is the concrete shape for our coordination journal (T13).
2. **Fork-with-source-attribution** as the overlay/adopt primitive across shards.
3. **Neighborhood** as the model for a *dynamic, link-and-fork-discovered* federated set +
search, with **roster** as the curated/explicit variant.
4. **Chorus of forks** — represent divergent versions across shards as co-equal, linked by
provenance, with reconciliation as an explicit human/policy step (mechanism over policy).
---
## 7. UC seeds
| # | Seed | Disposition |
|---|------|-------------|
| UC-70 | Attach a Federated Wiki site as a shard via its **page JSON + CORS** (REST/file-store hybrid); project pages, fork to overlay | **new** |
| UC-71 | Adopt a **per-page append-only semantic-action journal with provenance entries** (fork=source site) as the coordination-journal model — replay to materialize, compare to locate divergence | **new** |
| UC-72 | **Fork-with-site-provenance federation across a neighborhood** of peer shards — assemble a union from links/forks, search across it, preserve the chorus without forcing a canonical | **new** |
| — | fork-with-provenance as overlay/adopt | enrich **UC-26** (fork) |
| — | carry-forward of forked content + upstream re-fork | enrich **UC-28** (carry-forward) |
| — | happenings = time-bounded collaboration leaving durable forks | enrich **UC-30** (time-bounded space) |
| — | union/chorus of co-equal versions, provenance-linked | enrich **UC-05 / UC-27** |
## 8. Architecture notes for SHARD-WP-0002
- **T1T5 (federation):** fedwiki is the reference design. The **journal** (append-only,
semantic ops, fork-provenance) is the concrete coordination-journal shape; **neighborhood
+ roster** is the discovery/membership model (dynamic vs curated); **fork** is the
overlay/adopt op. Model the union as an assembly over sovereign sources with provenance
edges, reconciliation left to policy.
- **T11 (capability/write-granularity):** add **story-item / paragraph** as a named
write-granularity tier between whole-file and block/character.
- **T13 (history portability / merge model):** record fedwiki's **journal-replay op-log**
as a *third merge model* beside git 3-way and CRDT auto-merge — a **coarse semantic
op-log applied manually via fork**. A shard whose history *is* such a journal can supply
our coordination journal almost directly (vs git-commit import or CRDT-update import).
- **T16 (identity ≠ placement):** fedwiki's `fork` journal entries are **provenance edges**
between same-named pages on different sites — exactly the cross-shard "same page,
different placement" relation we must model. Use them as edges; keep our own identity
layer above slug.
## 9. Open questions
1. Should shard-wiki's coordination journal adopt fedwiki's **exact action vocabulary**
(create/add/edit/move/remove/fork) at the page-item level, or a more granular/abstract
op set that other shards can also emit?
2. Is **neighborhood** (dynamic, link/fork-discovered) a first-class membership mode for an
information space, or only a *view* over an explicitly-configured shard set (roster)?
3. How do we reconcile fedwiki's **slug-as-identity + fork-DAG** with our intended
**stable cross-shard identity** (T16) — promote fork edges into the identity graph, or
keep them as provenance-only annotations?
4. Does the **chorus / no-canonical** stance compose with shards that *do* assert a
canonical (Notion, an upstream git main)? (policy-selectable canonical over a
mechanism that permits chorus.)
## 10. Sources
- Smallest Federated Wiki wiki: **Story JSON**, **Federation Details**
github.com/WardCunningham/Smallest-Federated-Wiki/wiki
- JSON Schema notes — song.fed.wiki.org/json-schema.html
- "Smallest Federated Wiki" — home.c2.com/smallest-federated-wiki.html
- Federated Wiki — federated.wiki (Visualizing Page History)
- Mike Caulfield, "The OER Case for Federated Wiki" — hapgood.us (2015)
- Jon Udell, "A federated Wikipedia" — blog.jonudell.net (2015)
- Wikipedia: *Federated Wiki*; IndieWeb: *Smallest Federated Wiki*
- fedwiki/wiki-plugin-transport (plugin/transport reference)
- prior: `research/260608-federation-concepts/` §3
## 11. Traceability
New UCs **UC-70UC-72** carry the marker **⊞** in the wikiengines column of
`spec/UseCaseCatalog.md` (true lineage = this dive; placed in the nearest existing column).
Enriched: UC-26, UC-28, UC-30, UC-05, UC-27. Architecture cross-refs: SHARD-WP-0002
T1T5, T11, T13, T16.

View File

@@ -0,0 +1,17 @@
# 260614 — git-forge wikis deep dive (Gitea · GitLab · GitHub)
Deep dive on the **git-forge-hosted Markdown wikis** as one grouped memo: each is a
**dedicated git repository of Markdown** exposed through a forge, attachable by **cloning
the wiki repo directly** *or* (where offered) through the **forge's wiki API**. INTENT names
**Gitea wikis** explicitly as a shard participant — this dive confirms the **git-native
file-store** as a first-class, and the *simplest*, backend.
- `findings.md` — the three forges compared, the `.wiki.git` model, API matrix, capability
profile, INTENT mapping, UC seeds (UC-76/77), architecture notes for SHARD-WP-0002, open
questions (resolves the UC-68 source-of-truth question for this case), sources,
traceability.
Catalog yield: UC-76 (attach a forge wiki by cloning its dedicated `.wiki.git` — git **is**
the native store and the coordination journal), UC-77 (attach/write via the forge wiki API
where git-clone is unavailable/undesired — capability varies by forge). Enriched
UC-40/02/68/38. Feeds SHARD-WP-0002 T14 (attachment binding).

View File

@@ -0,0 +1,173 @@
# git-forge wikis (Gitea · GitLab · GitHub) — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T5 · **Subject:** the Markdown wikis
hosted by the three major git forges — **Gitea**, **GitLab**, **GitHub** — treated as one
family because they share one architecture: *a wiki is a separate git repo of Markdown.*
## Why this dive
INTENT names **Gitea wikis** as a shard participant, and the whole project is "a **Git-based
Markdown** wiki orchestrator." The forge wikis are therefore the **least exotic, highest-
fit** backend in the entire study: the page store is *literally a git repository of Markdown
files*. After fourteen dives into DBs, CRDTs, graphs and SaaS, this one confirms the
**home case** — and sharpens it by contrasting *git-IS-the-store* (forge wikis) against
*git-is-a-mirror* (Wiki.js, UC-68).
## 1. The shared architecture — a wiki is a `.wiki.git` repo
All three forges implement a project/repo wiki as a **second, dedicated git repository**
alongside the code repo, addressable as `<repo>.wiki.git`:
- `git@host:owner/project.wiki.git` (GitLab), `…/owner/repo.wiki.git` (Gitea/GitHub).
- **Pages are Markdown files** (`Home.md`, `Some-Page.md`), one file per page; the **page
title ↔ filename** (spaces ↔ hyphens by convention). Other markups are accepted
(AsciiDoc, Textile, reStructuredText, Org) — GitHub/Gitea via **Gollum** (the Ruby
git-backed wiki library), GitLab via its own renderer.
- **History is git history** — every page edit (web or pushed) is a git commit with
author/timestamp/message. *The wiki's revision history is a real git log.*
- **Special pages** by convention: `_Sidebar`, `_Footer`, `_Header` (GitHub/Gitea),
`_sidebar` (GitLab) — engine-rendered chrome stored as ordinary files.
- **Subdirectories / nested pages**: GitLab and Gitea support directory structure; GitHub
wikis are historically flat (Gollum supports paths but the GitHub UI is shallow).
The decisive property: **you can `git clone` the wiki repo, edit files, commit, and push**,
and the forge UI reflects it — *and vice versa*. Git is **a** (often **the**) first-class
write path. This is exactly shard-wiki's native medium with no impedance.
## 2. Where they differ — the API matrix
| | git clone/push of `.wiki.git` | wiki content **API** | nested dirs | markups |
|--|--|--|--|--|
| **Gitea** | ✅ yes | ✅ **REST wiki endpoints** (list/get/create/edit/delete pages) | ✅ | Markdown (+Gollum-style) |
| **GitLab** | ✅ yes | ✅ **REST Wikis API** (project & group wikis) | ✅ | Markdown/AsciiDoc/RDoc/Org |
| **GitHub** | ✅ yes | ❌ **no wiki REST API** — wiki is **git-only** (Gollum) | ⚠️ flat UI | Markdown + Gollum markups |
The key asymmetry: **GitHub exposes wiki content *only* through git** (the REST/GraphQL API
covers issues/PRs/code but **not** wiki pages); **GitLab and Gitea offer both** a wiki API
*and* git access. So the **git-clone path is the universal one** (works for all three); the
API path is an *optional, capability-varying* alternative.
## 3. git-IS-the-store vs git-is-a-mirror (the UC-68 contrast)
Wiki.js (UC-68) keeps a **DB as canonical** and *maintains a git mirror* — so writing by
commit risks **racing the engine's DB↔git sync** (catalog open-Q22). Forge wikis are the
opposite: **the git repo IS the canonical store**; there is *no* separate DB of record for
wiki content. Therefore:
- **The source-of-truth question (Q22) is resolved for this case:** the `.wiki.git` repo is
authoritative. shard-wiki can **write by commit/push directly** with no engine to race —
the forge merely *renders* what git holds.
- The forge **API** (GitLab/Gitea), where present, is a *convenience over the same git
repo*, not a competing store — so API-write and git-write converge on one history.
This makes forge wikis the **cleanest possible write-through file-store shard**: clone =
projection/mirror, commit = overlay-applied/write, git log = the coordination journal *as
is*.
## 4. Capability profile
| Dimension (synthesis spectrum) | Gitea / GitLab / GitHub wiki |
|--------------------------------|------------------------------|
| Attachment mode | **file-store (native: git clone)** + optional **external-API** (GitLab/Gitea wiki REST) |
| Addressing granularity | **page = file**; sub-page = path (GitLab/Gitea) |
| Content identity | path/filename within the wiki repo (title-derived) |
| Identity vs placement | placement-bound (path = identity), like a plain git repo |
| Structure | flat or directory tree of Markdown files; `_Sidebar`/`_Footer` chrome |
| History | **native git history** (real commits, authors, messages) |
| Merge model | **git** (3-way merge, branches) — though wiki repos are usually single-branch |
| Native query | none (it's files); forge full-text search over the wiki |
| Translation | **Markdown-native** (+ AsciiDoc/Org via renderer) — minimal/no translation needed |
| Attachment/write granularity | **file (page)** per commit |
| Operational envelope | ordinary git + forge; clone is cheap; API rate limits apply to API path |
| Access grant | **forge repo permissions** (delegated auth; per-repo/role ACL) |
| Content opacity | transparent Markdown in git |
| Provenance | git author/committer/timestamp per commit — native |
## 5. INTENT mapping
### Reinforcements (this is the home case)
- **Git-based Markdown orchestrator** (INTENT core): forge wikis *are* git repos of
Markdown. The **wiki page model** (Markdown-first, path-addressed, git-versioned) maps 1:1
— minimal adapter, maximal fit.
- **Coordination journal = git** (INTENT): the wiki repo's **git log is already the
coordination journal** — no synthesis needed; adopt it directly.
- **Overlay before mutation**: overlays are **branches/commits** on the cloned wiki repo;
applying = push (or open an MR/PR where the forge supports wiki MRs — GitLab does not for
wikis, so push-to-branch + manual is the path).
- **Graceful degradation**: even GitHub (no wiki API) is fully usable via git-clone — the
*universal* path means a limited forge is still a first-class read/write shard.
- **No silent remote mutation**: writes are explicit git pushes (or explicit API calls)
under the user's forge credentials and repo permissions.
### Divergences (boundaries / notes — minor)
- **Capability varies by forge**: GitHub = git-only (no content API); GitLab/Gitea = git +
API. The adapter must **model the API as an optional capability**, defaulting to the
universal git path (T11/T14). Not a bug — exactly the capability-awareness INTENT mandates.
- **Wiki repos rarely use branches/MRs for review**: forge wikis usually edit a single
branch directly; the rich PR-review flow is on the *code* repo, not the wiki. So
"overlay → review → merge" needs shard-wiki to provide the review layer, not the forge.
- **Identity = path** (like any git repo) — cross-shard identity (T16) is layered above, as
for plain git/`wiki/` subdir shards.
### What to keep
1. **git-clone as the universal, canonical file-store attach** for forge wikis — Markdown +
git history directly as page model + coordination journal (UC-76). The reference
easy-case backend.
2. **Forge wiki API as an optional capability** (GitLab/Gitea), with **git-only fallback**
(GitHub) — capability-aware binding (UC-77).
3. **git-IS-store ⇒ write-by-commit is safe** (no engine race) — record this as the
resolution of the Wiki.js mirror dilemma (Q22) for forge wikis.
## 6. UC seeds
| # | Seed | Disposition |
|---|------|-------------|
| UC-76 | Attach a **git-forge wiki** by **cloning its dedicated `.wiki.git`** — git is the native store; Markdown files = pages, git log = coordination journal; commit/push = write (no engine to race) | **new** |
| UC-77 | Attach/write a forge wiki via the **forge's wiki API** (GitLab/Gitea REST) where git-clone is unavailable or API-write is preferred; **git-only fallback** for GitHub — capability varies by forge | **new** |
| — | git-native file-store as the *canonical store* (not mirror) | enrich **UC-40** |
| — | dual-path attach (git clone vs forge API) | enrich **UC-02** |
| — | git-IS-store vs engine-maintained mirror (resolves Q22) | enrich **UC-68** |
| — | forge as an API host for the wiki resource | enrich **UC-38** |
## 7. Architecture notes for SHARD-WP-0002
- **T14 (adapter binding / attach path):** forge wikis are the canonical **file-store
attach** — bind to the `.wiki.git` clone as the universal path; model the **wiki API as an
optional, forge-specific capability** (present: GitLab, Gitea; absent: GitHub). One shard,
two possible bindings converging on the same git history.
- **T11 (capability model):** "has-content-API" is a **per-forge capability flag**; git
clone/push is the baseline every forge satisfies. Minimal adapter profile — near the
Oddmuse-simple end but Markdown-native and git-versioned.
- **Coordination journal:** adopt the wiki repo's **git log directly** — the one backend
where INTENT's git-backed journal needs *zero* synthesis.
- **Resolves Q22 (UC-68):** because git **is** the store (not a mirror), **write-by-commit
is safe** — no engine DB↔git sync to race. Record the distinction *engine-mirror*
(Wiki.js: DB canonical, careful) vs *git-canonical* (forge wikis: commit freely).
## 8. Open questions
1. For overlay → **review** → apply, does shard-wiki supply the review layer over a forge
wiki (which lacks wiki-MRs), e.g. via a branch + its own diff/approve, or push directly?
2. When a forge offers **both** git and a wiki API (GitLab/Gitea), which does the adapter
prefer by default — git (universal, full history) with API as a fallback for hosts where
clone is disabled? (cf. UC-43 backend-swap under stable binding.)
3. Should the **code-repo `wiki/` subdir** shard and the **forge wiki repo** shard share one
adapter (both git+Markdown) with a "which repo / which path" parameter, or stay distinct?
## 9. Sources
- GitLab Docs — *Wiki* (separate git repo; web/git/API; `.wiki.git`) — docs.gitlab.com
- Gitea — wiki via git clone + repository **wiki API**; forum/issue threads on
`.wiki.git` clone (go-gitea/gitea #1426, #15420) — gitea.com / github.com/go-gitea
- GitHub — wiki = Gollum git repo (`<repo>.wiki.git`), no wiki REST API — docs.github.com
- Gollum (git-based wiki library) — github.com/gollum/gollum
- prior: `research/260614-wikijs-deep-dive/` (engine-maintained mirror contrast, UC-68)
## 10. Traceability
New UCs **UC-76UC-77** carry the marker **⎇** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-40, UC-02, UC-68, UC-38. Architecture cross-refs:
SHARD-WP-0002 T14, T11; coordination-journal-from-git; resolves catalog open-Q22.

View File

@@ -0,0 +1,33 @@
# 260614 — Glamorous Toolkit (moldable development) deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T7**
## What this is
A deep dive into **Glamorous Toolkit** (GT, on Pharo): **moldable development** — cheap,
custom, **domain-specific views** (`gtView` methods) so any object explains itself through an
**open set of co-equal projections, none canonical** — plus **Lepiter**, GT's live notebook/
knowledge base (git-versionable JSON page files with live, inspectable code results).
## Why it matters
- Strongest prior art for **moldable, multi-view projection**: projection is not *a* view
but an **open, type-keyed set of co-equal, possibly-computed views, none privileged**
refines SHARD-WP-0002 **T16** and unifies replication-/derivation-/dimensional-/query-
projection under "many co-equal views."
- Generalizes ZigZag dimensional views (UC-47/48) and query/computed views (UC-54) into a
**pluggable view registry** keyed by content type (answers UC-55's open question on a
content-type registry).
- Reinforces **files-canonical, liveness-above, degrade-to-snapshot** (Lepiter files vs the
Pharo image; same boundary as Jupyter UC-84 / Squeak T6).
## Yield
- **No new UC** (design prior art, not a candidate shard — like the UseModWiki lineage dive).
- Enrich **UC-47, UC-48, UC-54**; links **UC-55, UC-83, UC-84, UC-79**.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Moldable development & `gtView`, the Moldable Inspector, Lepiter, relation to ZigZag/query/derivation projection, INTENT mapping, UC disposition (enrichment-only), architecture notes, open questions |

View File

@@ -0,0 +1,161 @@
# Glamorous Toolkit — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T7 · **Subject:** Glamorous Toolkit (GT) on
Pharo — moldable development, custom views/inspectors, the live notebook (Lepiter).
## Why this dive
T1/T3 gave us **derivation-projection** (one source → rendered/computed forms). GT comes at
projection from the *other* side: **many co-equal, domain-specific views over the same live
content**, where the *environment molds itself to the knowledge* rather than forcing the
knowledge into one fixed rendering. This is the strongest prior art for **moldable,
multi-view projection** and a striking parallel to ZigZag's dimensional model (UC-47/48) and
to query/computed views (UC-54). It is *design* prior art — GT is not a candidate shard —
so the yield is **enrichment + projection design notes**, not a new shard UC.
## 1. Moldable development
GT's thesis: **systems should be explainable through custom tools that are cheap to build.**
Instead of a single generic object inspector/renderer, a developer adds **small,
domain-specific views** to a class so any instance explains itself in the terms that matter:
- **`gtView`-annotated methods** — a class declares extra inspector views by writing methods
tagged `<gtView>` that return a view (tree, list, table, chart, source, raw, a custom
diagram…). Each is a **co-equal projection of the same object**; none is privileged.
- The **Moldable Inspector** shows these views as **switchable tabs** over one object, and
lets you **dive** into sub-objects (each with its own custom views) — navigation *is*
moving across projections.
- Views are **cheap and local**: a view is just a method, versioned with the code, added
incrementally as understanding grows. The environment **adapts to the domain**.
The key abstraction for us: **the same underlying content carries an open, extensible set of
views, selected at inspection time, none canonical.**
## 2. Lepiter — the live notebook / knowledge base
GT ships **Lepiter**, a notebook/knowledge base where pages mix prose, **live code
snippets** (evaluated in-image, results inspectable with the moldable views above), and
links. Notebook pages are stored as **JSON "database" files** on disk (a Lepiter DB =
directory of page files), so the knowledge base is **file-backed and git-versionable** while
remaining live in the image.
This is the literate/notebook pattern (T1/T3) fused with moldable views: a snippet's result
is not a static captured output but a **live object you can open into any of its views**
the *anti-snapshot*. (Boundary: that liveness is exactly what shard-wiki must degrade to a
snapshot when the image isn't present — see §4.)
## 3. Relationship to ZigZag, query-views, and the projection model
- **vs ZigZag (UC-47/48):** ZigZag gives **dimensional** views — the *same cells* seen along
different orthogonal axes. GT gives **moldable** views — the *same object* seen through
different *purpose-built lenses*. Both reject a single privileged rendering; both make
**multi-view, none-canonical** the norm. GT generalizes the idea from fixed dimensions to
an **open, code-defined view set**.
- **vs query/computed views (UC-54):** a `gtView` is a **computed projection** (it runs code
to build the view) — like a query-defined page, but keyed to a **content type / domain**
rather than a stored query. Strengthens "a view can be computed, not stored."
- **vs derivation-projection (T1/UC-83):** GT confirms projections are **plural and
co-equal**; UC-83/UC-84 had *few* well-known derivations (docs/code/outputs), GT has an
**open registry** of them keyed by type.
So GT's contribution to the contract is: **projection is not one view but an open set of
co-equal, type-keyed, possibly-computed views, none canonical** — a *moldable projection
registry*. That is a refinement of T16's projection model, not a new shard.
## 4. INTENT mapping
### Reinforcements
- **Union without erasure / no privileged rendering.** GT's "many co-equal views, none
canonical" is the same ethos as showing provenance/freshness without hiding any: a page's
presentation is **plural**, and shard-wiki should be able to offer **multiple co-equal
projections of the same content** (raw, rendered, structured, domain-specific) rather than
one flattened view.
- **Computed views (UC-54) keyed by content type.** A moldable view = a **computed
projection registered against a content type** — directly supports a *pluggable view/
projection registry* in the contract (the answer shape for UC-55's "pluggable content-type
registry" open question).
- **Files-canonical, live-on-top.** Lepiter stores pages as **git-versionable JSON files**
while being live in the image — reinforcing "the durable artifact is files; liveness is a
layer above," consistent with shard-wiki's git-canonical stance.
- **Mechanism over policy.** Which views to show, and whether to compute or snapshot them,
stay configurable; GT provides the *mechanism* (open view set), not a fixed presentation.
### Divergences / boundaries
- **The image is not a store (shared boundary with T6 Squeak).** GT's *liveness* lives in a
Pharo image; shard-wiki must not treat the image as a shard. Attach the **Lepiter DB files**
(git-versionable) as the durable content; treat live/computed views as **derivation-
projections that degrade to captured snapshots** when no image/kernel is present (same rule
as Jupyter UC-84).
- **Not a view engine.** shard-wiki *models* "this content type has these co-equal views and
one may be canonical-for-display"; it does not implement GT's rendering. Domain-specific
view code stays with the source/adapter, surfaced as a capability.
### What to keep
1. **Moldable projection = an open, type-keyed set of co-equal, possibly-computed views,
none canonical** — refine T16's projection model and the UC-47/48/54 cluster toward a
**pluggable view/projection registry**.
2. **Lepiter** = literate/notebook (T1/T3) with **live results** stored as git-versionable
files — another "files-canonical, liveness-above, degrade-to-snapshot" instance.
3. **Navigation = moving across projections** (dive into sub-objects) — a navigation idea
beside dimensional movement (UC-47/48) for the derived-views thread.
## 5. UC disposition (enrichment-only — no new shard UC)
| Mechanism (findings §) | Catalog UC |
|------------------------|------------|
| Many co-equal, domain-specific views over the same content; none canonical (§1, §3) | UC-47 / UC-48 (enriched) |
| `gtView` = a **computed projection registered against a content type** (§1, §3) | UC-54 (enriched) |
| Open, pluggable view set keyed by type = the shape of a content-type/view registry (§3) | links UC-55 (open-Q #10) |
| Lepiter live-snippet results = live objects → degrade to snapshot when no image (§2, §4) | links UC-84, UC-83 |
| Lepiter pages = git-versionable JSON files; image is not the store (§2, §4) | links UC-79 (files-canonical) |
| Dive-into-sub-object navigation = moving across projections (§1) | links UC-17UC-20 (derived views) |
GT is **design prior art, not a candidate shard**, so it yields **no new UC** (like the
UseModWiki lineage dive). Its value is sharpening the **projection model** for SHARD-WP-0002
T16: projection is an **open set of co-equal, type-keyed, possibly-computed views, none
privileged** — a moldable view registry.
## 6. Architecture notes for SHARD-WP-0002
- **T16 (projection):** generalize projection from "a view" to a **moldable view registry**
an open, extensible set of **co-equal, type-keyed, possibly-computed** projections of the
same content, none canonical (display-canonical is a policy choice, not a fact). This
unifies replication-projection (UC-53/57), derivation-projection (UC-83/84), dimensional
views (UC-47/48), and query/computed views (UC-54) under one "many co-equal views" model.
- **T12 (page model):** a content type may **carry/declare its own views** (a capability);
the page model should allow attaching view definitions to a type (the registry's entries),
answering UC-55's "pluggable content-type registry" question.
- **Boundary (T14):** attach the **Lepiter DB files** (durable, git-versionable), never the
Pharo image; live/computed views are derivation-projections degrading to snapshots — same
rule as Jupyter (no kernel/image host).
## 7. Open questions
1. Does shard-wiki expose a **view/projection registry** as a first-class public concept
(content type → its co-equal views), or keep "moldable" as an adapter-internal idea?
2. When multiple co-equal views exist, is **"canonical-for-display"** a per-shard policy, a
user preference, or unset (always show the chooser)? (Mechanism-over-policy says
configurable.)
3. How does a **computed view** (UC-54/`gtView`) declare its **freshness/provenance** so the
union doesn't present a stale computed projection as current? (Ties UC-84's snapshot
marking.)
## 8. Sources
- Glamorous Toolkit docs (`gtoolkit.com`): moldable development, the Moldable Inspector,
`gtView` methods, Lepiter knowledge base; Feenk essays on moldable development.
- Pharo (substrate — see T8 dive): live image, reflective environment.
- prior: `research/260614-zigzag-deep-dive/` (dimensional multi-view UC-47/48);
`research/260614-jupyter-deep-dive/` (live vs snapshot, UC-84);
`research/260614-literate-programming-deep-dive/` (derivation-projection, UC-83).
## 9. Traceability
**No new UC** (GT is design prior art, not a candidate shard). Enriched: UC-47, UC-48,
UC-54; links UC-55 (content-type/view registry), UC-83/UC-84 (live→snapshot), UC-79
(files-canonical), UC-17UC-20 (derived-view navigation). Architecture cross-refs:
SHARD-WP-0002 T16 (moldable view registry: open set of co-equal type-keyed computed views),
T12 (content type declares its views), T14 (attach Lepiter files, not the image).

View File

@@ -0,0 +1,15 @@
# 260614 — ikiwiki deep dive
Deep dive on **ikiwiki** (Joey Hess): a **wiki compiler** — it compiles a **VCS-backed**
(usually **git**) tree of Markdown source into **static HTML**, supports **distributed**
operation (clone/edit/push between wiki instances, change **pings**), and treats web edits as
commits to the same repo.
- `findings.md` — the compiler model, git-distributed federation + pinger, static output,
capability profile, INTENT mapping, UC seed (UC-79), architecture notes for SHARD-WP-0002,
open questions, sources, traceability.
Catalog yield: UC-79 (attach a git-backed **compile-to-static** wiki — git Markdown source
is the shard, compiled static HTML is a derived publish/projection; participate in
git-distributed clone federation with change-pings). Enriched UC-31/56/37/33. Feeds
SHARD-WP-0002 T4 (federation), T6 (publish/projection).

View File

@@ -0,0 +1,153 @@
# ikiwiki — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T4 · **Subject:** ikiwiki, Joey Hess's
VCS-backed wiki compiler.
## Why this dive
The forge-wiki dive (T5) established *git-IS-the-store* for hosted Markdown wikis. ikiwiki
takes the same git-canonical source but adds two ideas shard-wiki cares about directly:
**compile-to-static** (the wiki is *built*, not served from a DB) and **git-distributed
federation** (wiki instances clone, pull, push, and **ping** each other). It is referenced
in `research/260608-federation-concepts/`; here we go into the model.
## 1. The wiki-compiler model
ikiwiki is fundamentally a **compiler**: input is a directory of **source pages** (Markdown
by default; also other formats) held in a **version-control repo**; output is a tree of
**static HTML**. A rebuild is triggered by a **VCS post-commit/post-update hook**, so:
- **The VCS repo is canonical**; the HTML is *derived build output* (regenerable, disposable).
- **Web edits are commits.** The CGI edit interface writes the change *into the repo* (a
commit) and triggers a rebuild — so browser edits and `git push` edits converge on one
history. (Same convergence as forge wikis, but here the canonical store is *your* git repo,
not a forge's.)
- **VCS-agnostic**: git is usual, but svn/bzr/mercurial/darcs are supported via a VCS
plugin layer — an early "pluggable backend behind a stable interface" (adapter-contract
echo).
- **Plugins** (Perl) provide directives, feeds, auth (`openid`), and the federation hooks
below.
## 2. Git-distributed federation + the pinger
Because the source is an ordinary VCS repo, ikiwiki instances federate the way *git* does:
- **Clone-and-diverge**: you can `git clone` a wiki, edit offline, and `push`/`pull` between
instances — **multiple wiki clones that reconcile via git merge**. A wiki is a branch-space
(UC-33).
- **`pinger` / `pingee` plugins**: an instance can send an **XML-RPC ping** to another
ikiwiki when it changes, prompting the other to **pull and rebuild** — a lightweight
**subscribe/notify** primitive over the git-distributed mesh (UC-31).
- **`aggregate` plugin**: pulls external **RSS/Atom feeds** into the wiki as pages — an
inbound projection of remote content.
So ikiwiki is *federation by git plus a ping* — distinct from fedwiki's fork/journal
(UC-72) and from Wikibase's query-time `SERVICE` (UC-74): a **third federation flavor**,
*VCS-replication federation with change notification*.
## 3. Static output as a publish/projection target
The compiled static HTML is a **read-only, regenerable projection** of the source:
- It is a natural **outbound publish target** (UC-56): render the union (or a shard) to a
static site for hosting/backup, no server needed.
- It is also the **read-only backup** end (UC-37): a static snapshot that survives the engine.
The key shard-wiki framing: **source (git Markdown) is the attachable shard; static HTML is a
derived projection** — never confuse the build output for the canonical content.
## 4. Capability profile
| Dimension (synthesis spectrum) | ikiwiki |
|--------------------------------|---------|
| Attachment mode | **file-store** (VCS-backed git Markdown source) |
| Addressing granularity | page = source file; path = identity |
| Content identity | path/filename (placement-bound) |
| Structure | directory tree of Markdown source + directives |
| History | **native VCS (git) history** |
| Merge model | **git** (clone/pull/push/merge across instances) |
| Native query | none; directives + plugins compute derived pages at build |
| Translation | Markdown source → static HTML (build-time render) |
| Write granularity | **file (page)** per commit |
| Operational envelope | a compiler + VCS hook; static hosting for output |
| Access grant | VCS/file perms; `openid` for web edits |
| Content opacity | transparent Markdown |
| Provenance | git author/timestamp; aggregated feeds carry source |
| Federation | **git replication + XML-RPC ping**; RSS aggregation |
## 5. INTENT mapping
### Reinforcements
- **Git-canonical Markdown source** = the home case (shared with forge wikis UC-76/40):
attach the source repo, adopt its git log as the journal, write by commit.
- **Coordination layer is git** (INTENT): ikiwiki's whole federation *is* git replication +
a ping — the most literal realization of "Git-addressable coordination layer."
- **Projection vs canonical**: compile-to-static cleanly separates **canonical source** from
**derived output** — exactly shard-wiki's projection principle (static HTML = a lazy/cache
projection that is regenerable, never the source of truth).
- **Graceful degradation / publish**: static output is the trivial read-only backup and
outbound publish target (UC-37/UC-56).
- **Subscribe/notify mechanism, not policy**: the pinger is a *mechanism* (notify a peer to
pull); *which* peers, when, and conflict policy stay configurable.
### Divergences (boundaries / notes)
- ikiwiki is mostly a **reinforcement** of git-canonical-Markdown (UC-76) — its *new*
contributions are (a) **compile-to-static** as a distinct projection/publish direction and
(b) the **git-distributed-clone + ping** federation flavor. The static output is *not* a
shard to attach (it's derived); attach the **source repo**.
- VCS-agnostic backend layer is an interesting adapter-contract echo, but for shard-wiki the
git case dominates.
### What to keep
1. **Source-repo-as-shard, static-output-as-derived-projection** — never attach the build
output as canonical (UC-79; relates UC-37/UC-56).
2. **Git-replication + change-ping** as a named federation flavor beside fork/journal
(fedwiki) and query-SERVICE (Wikibase) — a **subscribe/notify over a git mesh** (UC-31/UC-33).
3. **Inbound feed aggregation** (RSS/Atom → pages) as an inbound projection pattern.
## 6. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-79 | Attach a git-backed **compile-to-static** wiki — the **git Markdown source is the shard**, compiled **static HTML is a derived publish/projection**; participate in **git-distributed clone federation** with change-**pings** | **new** |
| — | XML-RPC pinger = subscribe/notify over a git mesh | enrich **UC-31** |
| — | render union/shard to a static site (publish) | enrich **UC-56** |
| — | static HTML as read-only regenerable backup | enrich **UC-37** |
| — | wiki as a git branch-space; clones reconcile via merge | enrich **UC-33** |
## 7. Architecture notes for SHARD-WP-0002
- **T4 (federation):** record **git-replication + change-ping** as a federation flavor — peers
hold git clones, reconcile via merge, and notify with a ping to pull/rebuild. Distinct from
fedwiki fork/journal and Wikibase `SERVICE`; complements the "Git-addressable coordination
layer" mandate most literally.
- **T6 (publish/projection):** **compile-to-static** is the canonical *outbound* projection —
source (git) is canonical, static HTML is a regenerable derived view (publish/backup
target). Reinforces projection-vs-canonical separation.
- **Attach binding (T14):** attach the **source repo** (git Markdown), not the build output;
shares the forge-wiki / `wiki/`-subdir git+Markdown adapter (parameterized by repo/path).
## 8. Open questions
1. Is the **pinger** (notify-peer-to-pull) modeled as shard-wiki's own subscribe/notify
primitive, or only recognized when bridging two ikiwiki instances?
2. Does shard-wiki ever *drive* a compile-to-static **publish** of the union (act as the
ikiwiki-like compiler), or only attach existing ikiwiki source repos? (UC-56 scope.)
3. Is **feed aggregation** (RSS→pages) an inbound projection mode shard-wiki offers generally
(a feed-shard), or an ikiwiki-internal feature?
## 9. Sources
- ikiwiki.info — *ikiwiki* overview, *rcs* (VCS backends), *plugins* (`pinger`/`pingee`,
`aggregate`, `openid`), *setup* / post-commit hooks
- prior: `research/260608-federation-concepts/` (ikiwiki reference);
`research/260614-forge-wikis-deep-dive/` (git-canonical Markdown contrast)
## 10. Traceability
New UC **UC-79** carries the marker **⊟** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-31, UC-56, UC-37, UC-33. Architecture cross-refs:
SHARD-WP-0002 T4 (git-replication+ping federation), T6 (compile-to-static publish), T14.

View File

@@ -0,0 +1,49 @@
# 260614 — Joplin deep dive (SQLite-local, Markdown-on-sync, interchange-format attach)
Date: 2026-06-14
## What this is
A focused study of **Joplin** — the open-source Markdown note app — read through
shard-wiki's lens. Joplin looks like "open-source Obsidian" but is a genuinely different
point in the space: **content is Markdown, the local store is SQLite, and the *sync
representation* is a folder of per-item Markdown+metadata files on a backend you choose**
(filesystem, WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). It
also ships **end-to-end encryption** and is itself a **sync layer over heterogeneous
storage**.
Distinctive material:
- **Architecture** — SQLite local store (documented schema); Markdown content;
notebooks → notes → resources + tags + to-dos; 32-char item IDs; `:/<id>` note links
(rename-stable, page-level)
- **Sync** — serialized to plain-text items on a chosen target; tombstones; **conflict
notes** (keep-both) → enables the **"attach the sync mirror, not the app"** pattern on
WebDAV/Nextcloud/S3
- **E2EE** — items encrypted before leaving the device → **content opacity** (ciphertext
at rest)
- **Extension surfaces** — a **plugin API** (`joplin.data`/`workspace`/`contentScripts`/
views) *and* a **local Data API** (REST on `localhost:41184`, token) used by the Web
Clipper; plus JEX/Markdown/RAW export
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, sync/interchange attach, E2EE/content-opacity, plugin + Data API surfaces, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md`
(UC-60 attach a tool's documented sync/interchange representation on third-party storage
without the app; UC-61 attach an encrypted-at-rest shard with content opacity);
UC-31/36/38/40/51/55 enriched. Logged for `SHARD-WP-0002` (T11, T14): the
interchange/sync-representation attach surface, **content opacity** as a proposed twelfth
capability spectrum, a local-REST attach sub-mode, and format-aware file-store profiles.
**Key takeaways recorded:** the best attach surface is often a tool's **interchange/sync
representation** (offline, app-independent — Joplin items on Nextcloud/WebDAV/S3, the
INTENT-named backends), not its live store or API; **encryption-at-rest** demands a
content-opacity capability so encrypted shards degrade to backup/structure-shell; and
Joplin is the **file-sync-daemon boundary case** — shard-wiki attaches its mirror as
pages and never re-syncs (not-a-sync-daemon).
</content>

View File

@@ -0,0 +1,269 @@
# Findings — Joplin: SQLite locally, Markdown-on-sync, and the interchange-format attach
Date: 2026-06-14
Source kind: **modern shipped product** — an open-source Markdown note app; a *candidate
shard* whose distinctive trait is a **documented file-based sync/interchange format on
third-party storage** plus optional **end-to-end encryption**
Lens: shard-wiki — attachment surfaces, the "attach the sync mirror not the app"
pattern, content opacity (encryption), note-level stable IDs, and the file-sync-daemon
boundary
> Why Joplin earns a dive after Obsidian/Roam/Notion. It looks like "Obsidian but
> open-source," yet its architecture is a genuinely different point in the space:
> **content is Markdown but the local store is SQLite, and the *sync representation* is
> a folder of per-item Markdown+metadata files on a backend you choose** (filesystem,
> WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). That sync
> mirror — not the app, not the SQLite DB — is the interesting attach surface, and it
> lands squarely on the **WebDAV/Nextcloud backends INTENT already names**. Joplin also
> ships **E2EE**, forcing the question of a shard whose content is *opaque ciphertext*,
> and it is itself a **sync layer over heterogeneous storage** — the very "file-sync
> daemon" shard-wiki says it is *not*, making Joplin a sharp boundary case.
Contrast set: Obsidian (file-over-app: native store *is* files), Roam (client DB,
in-app API only), Notion (hosted DB, external API only), TWiki/Foswiki (file+RCS native
store). Joplin is none of these exactly — **DB-local, files-on-sync** — which is the
whole point.
---
## 1. Core architecture — SQLite local, Markdown content
- **Local store:** a **SQLite database** (`~/.config/joplin-desktop/database.sqlite`),
schema documented. So, unlike Obsidian, the *native* local store is **not** plain
files — you cannot point shard-wiki at a folder of `.md` and read the live store
(you'd read SQLite, fragile while the app runs).
- **Content:** notes are **Markdown** (Joplin-flavored: `:/imageId` resource links,
internal `:/<id>` note links, checkboxes/to-dos). Markdown-first — INTENT-aligned at
the *content* layer even though the *storage* layer is a DB.
- **Model:** `folders` (notebooks, hierarchical) → `notes``resources` (attachments);
cross-cutting `tags`; notes can be **to-dos**. Every item has a **32-char hex ID**;
notes link by ID (`:/<id>`), so links survive rename/move (store-minted, page-level).
- **Clients:** desktop (Electron), mobile, and a **terminal/CLI** app — multiple clients
edit one logical store, reconciled by sync (§2).
- **Revisions:** Joplin keeps **internal note history** locally — not git, not portable
→ a *supplement* case (UC-36), like Notion/Confluence.
---
## 2. Sync — the distinctive layer (and the attach opportunity)
Joplin's headline feature is **sync to a backend of your choice**. On sync, the SQLite
store is serialized to a **folder of plain-text items** — one file per note / notebook /
tag / resource-metadata, each carrying the **Markdown body plus a metadata block**
(`id`, `parent_id`, `type_`, `created_time`, `updated_time`, …) — written to the chosen
target: **filesystem, WebDAV, Nextcloud, Dropbox, OneDrive, S3, Joplin Server, Joplin
Cloud**. Deletions propagate via tombstones; concurrent edits across clients produce
**conflict notes** (keep-both, not silent overwrite).
Two consequences for shard-wiki:
1. **Attach the sync mirror, not the app (UC-60).** A Nextcloud/WebDAV/S3/filesystem
directory full of Joplin sync items is a **file-store shard** — readable *without
Joplin running and without touching SQLite* — provided the adapter understands the
**documented item format** (body + metadata footer, `:/id` links, resource items).
This is a new attach pattern: the backend's **interchange/sync representation** as
the attach surface, distinct from a native on-disk store (UC-40) or app files (UC-53).
It also realizes INTENT's WebDAV/Nextcloud participants concretely.
2. **Joplin is a file-sync layer over heterogeneous storage** — exactly what shard-wiki
says it is *not* (INTENT: "not a file-sync daemon"). Joplin syncs **one** logical
store across backends; shard-wiki **federates many** logical stores with wiki-page
semantics. Attaching a Joplin sync mirror means **reading its file representation as
pages**, never re-driving Joplin's sync — and never becoming a second sync engine
over the same target (don't double-sync).
---
## 3. End-to-end encryption — content opacity
Joplin offers **E2EE**: items are encrypted **before** leaving the device, so the sync
target holds **ciphertext** regardless of provider. (Security wart, noted: the master
key is stored *in clear* in the local SQLite — irrelevant to attaching the mirror, but
a reason never to treat the local DB as trusted.)
For shard-wiki this introduces a capability dimension the prior dives did not:
**content opacity**. An encrypted Joplin sync target can be attached only as a
**backup/mirror/structure-shell** participant — item IDs, counts, and change events may
be visible, but **bodies are undecryptable without keys**. The adapter must **never
present ciphertext as readable content** and must degrade to "present-but-opaque" with
provenance. This extends the synthesis capability spectra with a proposed twelfth field,
**content opacity** (`plaintext → encrypted-at-rest/opaque`), feeding `SHARD-WP-0002`
T11 (§9).
---
## 4. Extension surfaces — a plugin host *and* a local REST API
Joplin exposes **two** programmatic surfaces (plus export):
**A. Plugin API (in-app host).** TypeScript plugins, distributed via **npm + a plugin
repository**, loaded by the app:
- `joplin.data` — CRUD over the data model, shaped like REST (GET/POST/PUT/DELETE) on
`notes`/`folders`/`tags`/`resources`.
- `joplin.workspace` — selection + events (note selected, content changed, sync events).
- `joplin.contentScripts`**markdown-it** render plugins and **CodeMirror** editor
extensions (the syntax/rendering extension point).
- `joplin.views` — panels, dialogs, toolbar/menu items, custom editors;
`joplin.commands`, `joplin.settings`.
**B. Data API (local REST).** A REST service the desktop app serves on
**`localhost:41184`** (the "clipper server"), **token-authenticated**, used by the Web
Clipper and any local integration: endpoints `/notes`, `/folders`, `/tags`,
`/resources` (incl. `/resources/:id/file`), `/revisions`, `/search`; verbs
GET/POST/PUT/DELETE; pagination (`page`, `limit`≤100, `order_by`, `order_dir`). Notably
**plugins can use this API even when the clipper server is off**.
**C. Export****JEX** (tar of raw items), **Markdown directory**, **RAW** (Joplin
export directory), **ENEX** import (Evernote). Snapshot import/attach surface (UC-28/37).
So Joplin spans attachment modes: **in-app plugin host** (like Roam/Obsidian-plugin),
**local-REST** (like Notion's external API but *localhost + app-must-run* — a sub-mode),
the **sync-mirror file-store** (§2, the novel one), and **export snapshots**.
---
## 5. Joplin as a shard — capability profile
| Capability | Joplin | Notes for the adapter contract |
|------------|--------|--------------------------------|
| Read | **yes** | sync-mirror files (no app) · local Data API (app running) · plugin · export |
| Write | **yes** | Data API / plugin (`joplin.data`); writing the sync mirror directly is risky (Joplin owns that format) |
| Write granularity | **per-note (page)** | notebooks/notes/resources; not block-level |
| Identity / addressing | **32-char item ID; `:/<id>` links** | store-minted, **page-level**, survives rename (between Obsidian path and Roam block-UUID) (UC-51) |
| Structure | notebooks (hierarchy) + tags + to-dos | modest; metadata in the sync item footer (in-text on the mirror) |
| History | **internal revisions, not portable** | supplement via coordination journal (UC-36) |
| Native query | search API (FTS) | delegate text search; no datalog/DB-query (weaker than Roam/Notion) |
| Subscribe | sync is **poll-based**; conflict notes | poll; conflicts surface as keep-both items (UC-31, UC-07) |
| Content opacity | **optional E2EE → ciphertext** | **new dimension**: opaque-at-rest; mirror usable only as backup/structure without keys (UC-61) |
| Transclusion | none (note-level links only) | — |
| Diff / merge | none native | git-level if mirror is on a git-backed target |
| Publish | via export / static-site pipelines | outbound (UC-56) |
| Content types | Markdown + **resources** (attachments) | non-Markdown assets with IDs (UC-55) |
| Attach modes | sync-mirror file-store · local-REST · plugin · export | multi-mode, per-binding (UC-60, UC-57, UC-38) |
Verdict: a solid **Markdown-first candidate shard** whose best attach surface is the
**sync mirror on a file/WebDAV/S3 target** (offline, app-independent, INTENT-named),
with the local Data API or a plugin for live write-through. Standout: the
interchange-format attach (UC-60) and the encryption/opacity case (UC-61). Limits: DB
local store, note-level (not block) addressing, internal-only history, weak query.
---
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
### 6.1 Reinforcements
- **Interchange-format attach** validates that a shard's *documented sync/export
representation* — not just its live store or API — is a legitimate, often *best*,
attach surface (offline, app-independent). Generalizes beyond Joplin (UC-60).
- **WebDAV/Nextcloud/S3 become concrete** participants via Joplin's sync targets — the
backends INTENT names, now with a real format to parse.
- **Conflict-as-data** (Joplin conflict notes = keep-both) is union-without-erasure in
the wild (reinforces UC-07, divergence; consensus policy T9).
- **Store-minted, rename-stable note IDs** (`:/id`) confirm the addressing spectrum's
middle (page-level stable ID) between path (Obsidian) and block-UUID (Roam/Notion)
(UC-51).
### 6.2 Deliberate divergences (design bugs if conflated)
1. **Joplin is a sync daemon; shard-wiki is not.** Attach the **mirror as pages**; never
re-implement Joplin's sync or run a competing sync over the same target. (INTENT
not-a-file-sync-daemon.)
2. **DB-local; don't read SQLite live.** The native store is a DB; treat it as opaque.
Use the sync mirror, the Data API, or a plugin — not the live `database.sqlite`.
3. **Joplin owns the sync format; treat the mirror read-mostly.** Writing items into the
sync folder behind Joplin's back risks corruption/conflict; prefer overlay/projection
or write through the Data API/plugin (overlay before mutation, no silent mutation).
4. **Encrypted shards are opaque.** Never present ciphertext as content; degrade to
backup/structure with provenance (UC-61, graceful degradation).
### 6.3 What Joplin teaches that shard-wiki should keep
- Add **"interchange/sync representation"** as a recognized attach surface in the
contract — sometimes preferable to the live store/API (UC-60).
- Add **content opacity** (encryption-at-rest) as a capability-profile field so encrypted
shards degrade correctly (UC-61).
- Expect **multi-client concurrent editing reconciled by the backend's own sync**
(desktop/mobile/CLI) — a different concurrency than single-app file edits (extends
UC-53) or multi-user server writes; conflicts may pre-exist as keep-both items.
---
## 7. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-59**. New UCs **UC-60, UC-61** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a tool's documented sync/interchange representation on a third-party storage target** (Joplin items on Nextcloud/WebDAV/S3) as a file-store shard, without the app | **UC-60 (new)** |
| **Attach an encrypted-at-rest shard** (E2EE sync target): content opaque without keys; participate as backup/mirror/structure-shell, never presenting ciphertext as readable | **UC-61 (new)** |
| Store-minted, rename-stable **note-level** IDs (`:/id`) — the middle of the addressing spectrum | **enriches UC-51** |
| Native store can be a **DB while the sync representation is files** → attach surface ≠ native store | **enriches UC-40** |
| Internal revisions, not portable → supplement | **enriches UC-36** |
| Dual surfaces: in-app plugin host + **local Data API** (localhost, app-running) | **enriches UC-38** (links UC-57) |
| Resources/attachments = non-Markdown assets with IDs | **enriches UC-55** |
| Poll-based sync; conflict notes (keep-both) | **enriches UC-31** (links UC-07) |
---
## 8. Architecture notes for SHARD-WP-0002 (no UC)
- **Attachment-mode taxonomy gains a refinement:** within file-store, distinguish
*native on-disk store* (Obsidian/TWiki, UC-40) from *interchange/sync representation*
(Joplin mirror, UC-60). Both are "files," but format ownership and write-safety differ.
(T14.)
- **Add a "content opacity" capability field** (proposed **twelfth spectrum**:
`plaintext → encrypted-at-rest/opaque`) so encrypted shards degrade to
backup/structure-shell. Feeds T11 the next time the eleven spectra
(`research/260614-shard-spectrum-synthesis/findings.md` §2) are revised. (T11/T14.)
- **Local-REST as an attach sub-mode** (localhost, app-must-run, token) sits between
in-engine-host (UC-38) and external-API (UC-57) — note in T14.
- **Format-aware file-store adapters:** the contract should let a file-store adapter
declare a *format profile* (plain Markdown vs Joplin-item vs Foswiki-PlainFile) so the
same "folder of files" attach can parse tool-specific item formats. (T11/T14.)
---
## 9. Open questions (for spec / workplans)
1. For an **encrypted shard** (UC-61), what *is* visible without keys — item IDs/counts/
timestamps (structure shell) or nothing? Does shard-wiki ever hold keys, or only ever
treat such shards as opaque backups?
2. Is **writing to a tool's sync mirror** (UC-60) ever safe, or are interchange-format
shards **read/projection/overlay-only** by policy (Joplin owns the format)?
3. Does shard-wiki parse Joplin's **item format** via a dedicated format profile, or
only attach the **export** (JEX/Markdown dir) as a cleaner snapshot (UC-37)?
4. How do we avoid **double-sync** when a shard's storage target is itself driven by a
sync daemon (Joplin) we don't control?
---
## 10. Sources
| Source | Used for |
|--------|----------|
| Joplin Data API reference (https://joplinapp.org/help/api/references/rest_api/) | Local REST on `localhost:41184`, token auth, endpoints, 32-char IDs, item fields, pagination |
| Joplin Plugin API docs (https://joplinapp.org/api/references/plugin_api/classes/joplin.html) | `joplin.data` / `joplin.workspace` / `joplin.contentScripts` / views / settings |
| Joplin — Extending Joplin (https://joplinapp.org/help/api/) | Plugin distribution (npm + repository), Data API vs plugin access |
| Joplin — WebDAV / Nextcloud sync (https://joplinapp.org/help/apps/sync/nextcloud/) | Sync targets; items as plain-text files on the target |
| Joplin FAQ + community (https://joplinapp.org/help/faq/) | SQLite local store; Markdown content; E2EE before leaving device |
| Obsidian vs Joplin comparison (https://petronellatech.com/blog/obsidian-vs-joplin/) | Architecture contrast (SQLite+sync vs file-over-app), E2EE framing |
Cross-references: `research/260614-obsidian-deep-dive/findings.md` (file-over-app
contrast), `research/260614-notion-deep-dive/findings.md` (external-API, scoped grant),
`research/260614-shard-spectrum-synthesis/findings.md` (the spectra this extends),
`spec/UseCaseCatalog.md` (UC-31, UC-36, UC-38, UC-40, UC-51, UC-55, UC-57),
`workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
---
## 11. Traceability
- New UCs: **UC-60, UC-61**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-31, UC-36, UC-38, UC-40, UC-51, UC-55** (links UC-07, UC-57).
- Architecture (no UC): interchange/sync-representation attach surface; **content
opacity** as a proposed twelfth capability spectrum; local-REST sub-mode; format-aware
file-store profiles → `SHARD-WP-0002` (T11, T14).
- Boundary recorded: Joplin is a **sync daemon over one store**; shard-wiki attaches its
**mirror as pages** and does not re-sync — and treats DB-local store and encrypted
content as opaque (INTENT not-a-sync-daemon, graceful degradation, no silent mutation).
</content>

View File

@@ -0,0 +1,35 @@
# 260614 — Jupyter Notebooks deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T3**
## What this is
A deep dive into **Jupyter Notebooks**: the **`.ipynb` JSON** document (ordered cells:
markdown / code+outputs / raw), **kernels**, **embedded computed outputs** (MIME bundles),
and **execution-count provenance**. The dominant modern computational document and the
concrete case where the **derived output is captured and stored *inside* the source** with
real-but-fragile provenance.
## Why it matters
- Tests the T1 **replication- vs derivation-projection** split on the dominant real artifact
and adds the wrinkle that **outputs are stored back inside the source** — the source/
projection line runs *through* the document.
- The page model (T12) must carry a **notebook shape**: ordered cells with code cells owning
embedded computed outputs that have **weak execution provenance** (run order, environment
not captured).
- Non-Markdown + lossy translation (T15): JSON+MIME bundles; nbconvert→Markdown is lossy and
directional. JSON diffs are noisy → Jupytext text-pairing / nbdime (T13).
## Yield
- **UC-84** (new): attach/project a computational notebook preserving cell structure +
embedded outputs, surfacing outputs as **snapshots with weak execution provenance**;
re-execution **capability-gated**, default = present snapshot + static render.
- Enrich **UC-54, UC-55, UC-59, UC-35**; links **UC-32, UC-83, UC-79**.
## Contents
| Path | Role |
|------|------|
| `findings.md` | `.ipynb` model, kernels/execution-count fragility, ecosystem (nbconvert/Jupytext/papermill/nbdime/nbstripout), capability profile, INTENT mapping, UC seed, architecture notes, open questions |

View File

@@ -0,0 +1,185 @@
# Jupyter Notebooks — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T3 · **Subject:** Jupyter Notebooks — the
`.ipynb` JSON document, kernels, embedded computed outputs, execution provenance.
## Why this dive
T1 (literate programming) established **one source → derived projections** and split
**replication-projection** from **derivation-projection**. Jupyter is the *dominant modern
computational document* and the concrete case where the **derived output is captured and
stored inside the source** — a non-Markdown, partially-executable content type whose
provenance is real but **fragile**. It is the most plausible concrete "computational shard"
content type, so it tests the page model (T12), lossy translation (T15), and the
output-provenance question head-on.
## 1. The `.ipynb` document model
A notebook is a single **JSON document** (`nbformat`), not Markdown:
- **`cells[]`** — an ordered list. Each cell has a `cell_type`:
- `markdown` — prose (Markdown + LaTeX), the human-readable part.
- `code` — source text (`source`), plus an **`execution_count`** and an **`outputs[]`**
array captured from the last run.
- `raw` — passthrough.
- **`outputs[]`** (per code cell) carry results inline: `stream` (stdout/stderr),
`execute_result` / `display_data` (a **MIME bundle**`text/plain`, `text/html`,
`image/png` base64, `application/json`, vendor MIME types), and `error` (traceback).
- **`metadata`** at notebook and cell level (`kernelspec`, `language_info`, tags like
`hide-input`, `scrolled`, slideshow roles).
So an `.ipynb` is **source + last-run computed outputs + environment metadata, fused in one
JSON file**. The Markdown cells are an *island* inside a JSON envelope — relevant to how
shard-wiki extracts/round-trips content.
## 2. Kernels and execution
- A **kernel** is a separate language process (IPython, IRkernel, IJulia, …) speaking the
Jupyter messaging protocol (ZeroMQ). The document is **decoupled from the kernel**: the
`.ipynb` persists *captured* outputs; re-running requires a live kernel + the right
environment.
- **`execution_count`** numbers the order cells were *run*, which **need not match document
order** — the infamous **hidden-state / out-of-order execution** problem: stored outputs
may reflect a run sequence that no longer corresponds to top-to-bottom reading.
- Reproducibility therefore depends on **out-of-band state**: package versions, data files,
environment, random seeds — none captured by `nbformat` itself.
**Consequence for shard-wiki:** the captured outputs are a **snapshot projection with weak
provenance** — honest treatment must mark them as "computed at run N, environment not
guaranteed," never as live or authoritative truth.
## 3. The ecosystem (relevant to attach/project/translate)
- **nbconvert** — derives other forms from a notebook: HTML, Markdown, LaTeX/PDF, slides,
script. This is **derivation-projection** (T1): notebook source → rendered view, lossy in
both directions (HTML keeps outputs; `--to script` keeps only code, like `tangle`).
- **Jupytext** — represents a notebook **as** a `.py`/`.md` text file (pairing), making it
**git-diffable plain text** and round-trippable — directly relevant to storing notebooks
in a git shard without JSON-diff noise.
- **papermill** — parameterize + execute a notebook to produce a new output notebook
(notebook as a runnable template — a *derivation with inputs*).
- **JupyterLab / Notebook / nbviewer / Colab** — front-ends; nbviewer renders a static
read-only projection from a URL (a natural projection target).
- **`nbstripout`** — strips outputs before commit: teams treat **outputs as derived noise**,
keeping only source under version control — an explicit "source canonical, outputs
derived" stance mirroring T1.
## 4. Capability profile (as a shard / content type)
| Dimension (synthesis spectrum) | Jupyter notebook |
|--------------------------------|------------------|
| Attachment mode | file-store (`.ipynb` JSON in a repo) or via Jupyter Server REST API |
| Addressing granularity | document; **cell** as sub-address (by index / id; `nbformat 4.5+` adds stable cell `id`) |
| Content identity | file path; cell `id` (4.5+) else positional |
| Structure | **ordered cell list** (markdown / code+outputs / raw); MIME-bundle outputs |
| History | VCS on the file; **JSON diffs are noisy** unless paired (Jupytext) or stripped |
| Merge model | git on JSON (poor) → **paired text** (good) or nbdime (cell-aware diff/merge) |
| Native query | none |
| Translation | nbconvert → HTML/MD/script/PDF (lossy, directional); Jupytext text pairing |
| Write granularity | file / **cell** |
| Operational envelope | a kernel + environment to (re)execute; static render needs none |
| Content opacity | **mixed**: source transparent; outputs = MIME blobs (some opaque, e.g. base64 PNG) |
| Provenance | `execution_count` (weak, out-of-order); environment **not** captured |
| **Computed-output** | **stored inline**, snapshot, reproducibility out-of-band |
## 5. INTENT mapping
### Reinforcements
- **Replication- vs derivation-projection (T1) confirmed and extended.** nbconvert (→HTML/
script) and nbviewer are derivation-projections; `--to script` is literally `tangle`.
Jupyter adds a third wrinkle: **the derived output is also stored back inside the source**
(captured outputs), so the "source vs projection" line runs *through* the document.
- **Union without erasure / provenance honesty.** Captured outputs must be surfaced **as
snapshots with weak provenance** (run N, environment unguaranteed) — a concrete instance
of "never hide freshness/authorship." The out-of-order `execution_count` is exactly the
kind of fragility shard-wiki must *show*, not paper over.
- **Non-Markdown content + lossy translation (UC-55/UC-59).** `.ipynb` is JSON with embedded
MIME-bundle outputs; any Markdown projection is **lossy** (loses live outputs, kernel,
rich MIME). Surface the lossiness; keep the JSON as canonical payload (T12/T15).
- **Markdown island.** Markdown cells fit the text-first model, but only as *fragments
inside* a JSON envelope — the adapter extracts/round-trips them, it does not pretend the
notebook is a Markdown page.
### Divergences / boundaries
- **shard-wiki is not a kernel host.** Re-execution (driving a kernel) is out of scope/
capability-gated; default treatment is **attach + present captured outputs as a snapshot
projection** + offer nbconvert-style static render. Executing/parameterizing (papermill)
is an optional capability, never assumed.
- **Outputs-in-source is an anti-pattern to respect, not adopt.** Teams strip/pair outputs
precisely because mixing derived data into the source breaks diffs. shard-wiki should
prefer the **source-canonical, outputs-as-derived** reading (Jupytext pairing / nbstripout
ethos) and treat stored outputs as a capturable projection.
### What to keep
1. **Computational-notebook as a first-class content type** with cell structure + inline
**computed outputs carrying (weak) execution provenance** — UC-84.
2. **Outputs = derivation-projection snapshot** (T1 vocabulary): regenerable only with a
kernel+environment; degrade gracefully to the stored snapshot / static render.
3. **Cell-level addressing** (stable cell `id`, nbformat 4.5+) as the sub-page granularity
for transclusion/anchoring (UC-32/UC-35).
4. **Text-pairing (Jupytext)** as the git-friendly storage strategy — feeds the
history-portability thread (poor JSON diffs → paired text / nbdime).
## 6. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-84 | Attach/project a **computational notebook** (`.ipynb`): preserve **cell structure** (markdown / code / output) and **embedded computed outputs**, surfacing each output **as a snapshot with its (weak) execution provenance** (run count, environment not guaranteed) — re-execution is **capability-gated**, default is present-the-snapshot + offer a static rendered projection | **new** |
| — | Notebook JSON / MIME-bundle outputs = non-Markdown content; Markdown projection is lossy | enrich **UC-55**, **UC-59** |
| — | Computed/evaluated cell = computation-defined content | enrich **UC-54** |
| — | Cell `id` (nbformat 4.5+) = sub-page address for anchor/transclusion | enrich **UC-35**, links **UC-32** |
| — | Stored outputs as derived snapshot (nbstripout/Jupytext ethos) = source-canonical/outputs-derived | links **UC-83**, **UC-79** |
## 7. Architecture notes for SHARD-WP-0002
- **T12 (page model):** add **computational-notebook** as a page shape — an **ordered cell
list** where code cells own **embedded computed outputs** (MIME bundles) with weak
execution provenance. Distinct from prose, typed records, query-defined, inline-embedded
objects (Quip/Notion), typed-graph (Wikibase), and the literate one-source-many-projection
shape (UC-83). The defining new attribute: **derived output stored *inside* the source**.
- **T15 (translation / fidelity):** `.ipynb` is non-Markdown; nbconvert→Markdown is **lossy
and directional** (drops live outputs/kernel/rich MIME). Keep JSON canonical; any Markdown
is a projection. MIME-bundle outputs map to the content-opacity spectrum (text→html→base64
image = transparent→opaque).
- **T13 (history):** JSON diffs are **noisy**; record **text-pairing (Jupytext)** and
**cell-aware diff/merge (nbdime)** as history-portability strategies for embedded-output
documents. Reinforces "source-canonical, outputs-derived."
- **T16 (projection):** captured outputs are a **derivation-projection snapshot**;
re-execution (kernel) and parameterized execution (papermill) are **capabilities**, not
assumptions; degrade to the stored snapshot / nbviewer-style static render.
## 8. Open questions
1. Does shard-wiki ever **re-execute** a notebook (host/broker a kernel), or strictly
attach + present captured outputs + static render? (Same scope boundary as UC-83/UC-56
"do we ever drive the derivation.")
2. Is **UC-84** distinct from **UC-83**, or is a notebook just the "outputs-stored-in-source"
special case of the literate one-source-many-projection pattern? (Kept separate: UC-84's
defining trait is *captured derived output embedded in the canonical source with weak
provenance* — a page-model attribute UC-83 doesn't carry.)
3. How are **MIME-bundle outputs** represented in the page model — opaque provenance-tagged
blobs, a typed-asset registry (UC-55 open question #10), or selected-MIME projection?
4. Default storage: attach `.ipynb` **as-is** (JSON, noisy diffs) or prefer a **paired text
representation** when the shard is a git repo? (Policy → configurable.)
## 9. Sources
- Jupyter `nbformat` reference (cells, outputs, MIME bundles, cell `id` 4.5+);
Jupyter messaging protocol / kernels docs.
- **nbconvert**, **nbviewer**, **JupyterLab**, **Colab** docs.
- **Jupytext**, **papermill**, **nbdime**, **nbstripout** project docs.
- prior: `research/260614-literate-programming-deep-dive/` (replication- vs
derivation-projection, UC-83); `research/260614-notion-deep-dive/` (block-JSON,
external-API), `research/260614-quip-deep-dive/` (inline embedded objects, UC-55/58/59).
## 10. Traceability
New UC **UC-84** carries the marker **⊜** in the wikiengines column of
`spec/UseCaseCatalog.md` (true lineage = this dive). Enriched: UC-54, UC-55, UC-59, UC-35;
links UC-32, UC-83, UC-79. Architecture cross-refs: SHARD-WP-0002 T12 (notebook page shape:
outputs embedded in source), T15 (lossy non-Markdown translation; MIME opacity), T13
(paired-text / nbdime history), T16 (output = derivation-projection snapshot; kernel =
capability).

View File

@@ -0,0 +1,35 @@
# 260614 — Literate Programming (Knuth's WEB / weave / tangle) deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T1**
## What this is
A deep dive into **literate programming** — Knuth's WEB and the **`weave`/`tangle`** model
— as the deepest ancestor of shard-wiki's **projection** and **transclusion** ideas
applied to *executable* content. The keystone: **one canonical source → two co-derived
projections** (typeset docs via `weave`, compilable code via `tangle`), plus **named code
chunks** assembled by reference (transclusion).
## Why it matters
- Establishes **one-source-many-projections** as a page-model + projection pattern that
*generalizes* compile-to-static (UC-79, single output) to **N co-equal, semantically
different** derived views — feeds SHARD-WP-0002 **T12/T16**.
- Splits projection into **replication-projection** (lazy cache, current default) vs
**derivation-projection** (transform/compile/weave/evaluate) — a distinction the rest of
the computational batch (notebooks, REPLs) extends.
- Named chunks are the **executable-content face of transclusion / compose-by-reference**
(UC-32 / UC-44).
## Yield
- **UC-83** (new): attach a single-source-multiple-projection (literate) artifact; present
each derived view with output→source provenance; edits target the source.
- Enrich **UC-32**, **UC-44**, **UC-79**; links **UC-54** (computed/evaluated projection,
→ T3 Jupyter).
## Contents
| Path | Role |
|------|------|
| `findings.md` | WEB model, named-chunk transclusion, descendants (noweb/org-babel/knitr/Jupytext), capability profile, INTENT mapping, UC seed, architecture notes, open questions |

View File

@@ -0,0 +1,180 @@
# Literate Programming (Knuth's WEB / weave / tangle) — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T1 · **Subject:** Donald Knuth's WEB
system and the literate-programming model (`weave`/`tangle`, named chunks).
## Why this dive
SHARD-WP-0004 asks the carried question: *can a shard-wiki page be a source that is
woven/evaluated into rendered forms, and how do projection/transclusion/provenance treat
the source vs the output?* Literate programming is the **deepest ancestor** of that idea.
Knuth (1984) inverted the program/comment relationship: you write a **document** for
humans whose fragments happen to also be the program. From the **one WEB source** two
tools derive two artifacts: **`weave` → typeset documentation** (TeX) and **`tangle`
compilable code** (Pascal/C/…). This is *one source, two projections* in its purest,
oldest form — the conceptual root of shard-wiki's **projection** and **transclusion**.
## 1. The WEB model: one source, two tools
A WEB file interleaves prose and code in author-chosen order (the order that best
*explains*, not the order the compiler needs). Two programs read it:
- **`weave`** produces a `.tex` file → typeset documentation: prose, pretty-printed code,
a cross-reference index of where each chunk and identifier is defined/used.
- **`tangle`** produces a compilable source file → reorders and expands the code chunks
into the sequence the compiler demands, strips the prose, macro-expands references.
The crucial property: **the two outputs are co-derived from one canonical source and are
semantically different audiences** (human reader vs compiler). Neither output is the
source; both are **regenerable, disposable derivations**. Editing happens on the WEB; you
never edit the woven `.tex` or the tangled `.c`.
## 2. Named chunks = transclusion of code fragments
The organizing primitive is the **named section / code chunk**:
- A chunk is declared `<<name>>=` and **referenced** elsewhere as `<<name>>`. `tangle`
expands references in place (recursively) to assemble the final program.
- A chunk name can be **defined in multiple places** and is *accreted* (later `+=`
additions append) — so one logical unit is authored across scattered locations.
- References can appear before definitions; resolution is by name, not by position.
This is **transclusion by reference** (UC-32) and **compose-by-reference / manifest**
(UC-44, the EDL/Xanadu lineage): the document is a graph of named fragments assembled at
derivation time. Knuth's chunk graph is the same shape as Xanadu's reference-not-copy and
shard-wiki's "compose a page from referenced parts" — applied to *executable* content and
resolved by a build tool rather than a viewer.
## 3. The descendants (noweb, CWEB, org-babel, Sweave/knitr, Jupytext)
- **CWEB** (Knuth/Levy): WEB for C. **noweb** (Ramsey): language-agnostic, minimal markup
(`<<chunk>>`), `notangle`/`noweave` — proof that the *model* (chunks + two projections)
is separable from any one language or typesetter.
- **org-babel** (Emacs Org-mode): named source blocks, `:noweb` references, **tangle** to
files **and evaluate** blocks inline (results captured back into the document) — literate
programming fused with notebook execution (bridges to T2/T3).
- **Sweave / knitr** (R): weave prose + R, executing code and **interleaving computed
results/figures** into the woven document — adds the *computed-output* dimension that
Jupyter (T3) centers on.
- **Jupytext**: represents a Jupyter notebook **as** a literate text/Markdown source —
closing the loop: the notebook (T3) becomes a weave/tangle-style plain-text source.
The throughline: **one canonical source → N derived projections**, where projections may
be (a) reformatted (weave), (b) reordered/extracted (tangle), or (c) **evaluated**
(org-babel/knitr) — the evaluated case is exactly the computational-page question.
## 4. Capability profile (as a would-be shard / page type)
| Dimension (synthesis spectrum) | Literate-programming source |
|--------------------------------|-----------------------------|
| Attachment mode | file-store (a WEB/`.nw`/`.org` text source in a repo) |
| Addressing granularity | document; **named chunk** as sub-page address |
| Content identity | source file path + chunk name (name-resolved, not position) |
| Structure | **graph of named chunks** assembled by reference |
| History | whatever VCS holds the source (git) — text, diffable |
| Merge model | text/git merge on the source |
| Native query | none; `weave` emits a cross-reference index (derived) |
| Translation | source → woven docs **and** tangled code (build-time) |
| Write granularity | file / **chunk** (text region) |
| Operational envelope | a build tool (`tangle`/`weave`/`noweb`/babel) |
| Content opacity | transparent text |
| Provenance | VCS author/time; chunk cross-ref maps output→source location |
| **Projection model** | **one source → many co-equal derived projections** (new emphasis) |
## 5. INTENT mapping
### Reinforcements
- **Projection (canonical vs derived).** Literate programming is the archetype of
"canonical source, regenerable derived view" — the same principle as ikiwiki
compile-to-static (UC-79), but generalized: **two-plus co-equal projections** from one
source (docs *and* code), not a single output. Strengthens the rule *never confuse a
projection for the source*.
- **Transclusion / compose-by-reference.** Named chunks are transclusion (UC-32) and a
manifest of referenced parts (UC-44) — resolved at derivation time. Confirms
transclusion=clone=embed=reference as one primitive that also covers *fragment assembly
of executable content*.
- **Markdown-first but backend-neutral page model.** noweb/org/Jupytext show the literate
source can *be* Markdown-ish plain text — so a "literate page" fits the text-first model;
the *derivations* are the non-text part.
- **Mechanism over policy.** weave/tangle are mechanisms; *which* projections to
materialize, when to regenerate, and where outputs go stay configurable.
### Divergences / boundaries
- **shard-wiki is not a build system.** It should *recognize and present* a
source-with-projections (attach the source, surface derived views with provenance), not
re-implement `tangle`/kernels. Materializing a projection may delegate to the source's
own tool or be capability-gated to "snapshot only."
- **The interesting projection is derivation, not caching.** shard-wiki's base projection
is cache-like (lazy copy of remote content, UC-53/57). Weave/tangle is a *different*
projection species: **transform/derive** one source into rendered forms. The contract
should model projection as having (at least) two kinds: **replication-projection** and
**derivation-projection** (compile/weave/evaluate).
### What to keep
1. **One-source-many-projections** as a first-class page-model + projection pattern
(generalizes UC-79's single output) — see UC-83.
2. **Named-chunk transclusion** as the executable-content face of UC-32/UC-44 (assembly by
reference at derivation time).
3. **Output→source provenance** (the woven cross-ref index): every derived view must point
back to the exact source location it came from — never present derived output without
that link (union-without-erasure for derivations).
## 6. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-83 | Attach a **single-source-multiple-projection** artifact (a literate/woven source): treat the source as canonical, present each **derived projection** (e.g. a documentation view *and* a code view) with **provenance back to the one source**, edits target the source and projections regenerate (delegated to the source's tool or degraded to a static snapshot) | **new** |
| — | Named chunks `<<name>>` = transclusion / compose-by-reference of (executable) fragments | enrich **UC-32**, **UC-44** |
| — | Generalize compile-to-static (single output) to **N co-equal projections** from one source | enrich **UC-79** |
| — | Computed/evaluated projection (org-babel/knitr) = derivation-projection with results | links **UC-54**, foreshadows **T3** |
## 7. Architecture notes for SHARD-WP-0002
- **T12 (page model):** add **one-source-many-projections** as a page-model shape — a page
may be a *source* whose presented forms are **derivations** (woven docs, tangled code,
evaluated results), each carrying output→source provenance. Distinct from prose, typed
records, query-defined, and inline-embedded objects already logged.
- **T16 (projection):** split projection into **replication-projection** (lazy cache of
remote content — current default) vs **derivation-projection** (transform/compile/weave/
evaluate a source into rendered forms). Derivation-projection is regenerable, may
delegate to an external tool, and degrades to a captured snapshot when the tool is
absent (graceful degradation).
- **Transclusion (T16):** named-chunk-by-name resolution is a transclusion variant where
the *target is a fragment resolved by name across the source*, assembled at derivation
time — a concrete shape for UC-32/UC-44 mechanics.
- **Capability gating:** "can derive projection X" is a capability; a shard that can't run
the tool still exposes the source + any pre-built/snapshot projections (UC-83 degrade).
## 8. Open questions
1. Does shard-wiki ever *drive* a derivation (run weave/tangle/evaluate), or only attach
sources and surface pre-built projections + snapshots? (Same scope question as UC-56
"do we ever compile-to-static ourselves," now for literate sources.)
2. Is **UC-83** distinct enough from UC-79 (compile-to-static) to stand alone, or should
UC-79 be re-read as the single-output special case of UC-83's N-projection general case?
(Recorded as a possible later consolidation; kept separate now because UC-83's
projections are *co-equal and semantically different audiences*, not one publish target.)
3. How is **output→source provenance** represented when a derived line came from a chunk
accreted across several source locations (the cross-ref is many-to-one)?
## 9. Sources
- Knuth, *Literate Programming* (1984, *Computer Journal*); the WEB user manual; *TeX: The
Program* / *MMIX* as canonical WEB exemplars.
- Ramsey, *noweb* (a simple, extensible literate-programming tool).
- CWEB (Knuth & Levy); Emacs **Org-mode babel** (tangle + evaluate); **Sweave**/**knitr**
(R); **Jupytext** (notebook-as-text).
- prior: `research/260614-ikiwiki-deep-dive/` (compile-to-static, canonical-vs-derived);
`research/260614-xanadu-deep-dive/` (compose-by-reference / EDL, UC-44).
## 10. Traceability
New UC **UC-83** carries the marker **⊛** in the federation column of
`spec/UseCaseCatalog.md` (true lineage = this dive; literate programming is design prior
art, not a candidate shard, so the marker sits with the projection/compose-by-reference
family). Enriched: UC-32, UC-44, UC-79; links UC-54. Architecture cross-refs: SHARD-WP-0002
T12 (one-source-many-projections page shape), T16 (replication- vs derivation-projection;
named-chunk transclusion).

View File

@@ -0,0 +1,46 @@
# 260614 — Local-first workspaces cohort deep dive (Anytype · AFFiNE · AppFlowy)
Date: 2026-06-14
## What this is
A **combined cohort deep dive** of three open-source, local-first "all-in-one workspace"
tools — **Anytype**, **AFFiNE**, **AppFlowy** — studied together because they share the
one trait none of the prior eleven systems had: **the store is a CRDT** (Anytype's
*any-sync*, AFFiNE's *Yjs/y-octo*, AppFlowy's *Yrs*). The substrate is the contribution;
studying them together isolates it from each tool's surface features.
Per-tool highlights:
- **Anytype** — `any-sync`: **P2P + E2EE** CRDT (sync/file/consensus/coordinator nodes;
encrypted backup), a **typed object graph** (user-editable types + relations), IPFS
files, an **open API + MCP**.
- **AFFiNE** — **Yjs/y-octo** over **OctoBase**; **BlockSuite** editor framework; docs,
whiteboards, and databases as **views of the same block set**; WebSocket sync,
self-host (Docker, MPL-2.0).
- **AppFlowy** — **Yrs** via **AppFlowy-Collab** (Rust); Flutter UI; Notion-style
**databases + views**; pluggable storage/sync (RocksDB/IndexedDB; AppFlowy Cloud/
Supabase).
## Contents
| Path | Role |
|------|------|
| `findings.md` | Per-tool sections, the CRDT cohort thesis, comparative matrix, combined capability profile, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Cohort dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-64
attach a CRDT-synced local-first shard with native conflict-free merge; UC-65 attach a
peer-to-peer / decentralized shard with no single canonical endpoint); UC-36/47/48/51/55/
57/58/61 enriched. Logged for `SHARD-WP-0002` (T11/T14): a **merge-model** capability
(proposed thirteenth spectrum: `none / git / native-CRDT`), a **CRDT-replica** substrate +
**P2P/no-central-endpoint** attach mode, an **endpoint-model** capability, and MCP as an
integration surface.
**Cohort thesis recorded:** CRDT changes the adapter math — the backend performs
conflict-free merge itself (so shard-wiki must not impose git/text merge), history is a
CRDT update log (supplement, not git), and the attach surface is a **replica or sync
endpoint** (Anytype adds **P2P + E2EE**). **Boundary:** CRDT local-first candidate shards
— attach a replica/projection as pages, respect native merge, never re-drive their sync;
not substrates and not the federation layer.
</content>

View File

@@ -0,0 +1,261 @@
# Findings — Anytype, AFFiNE, AppFlowy: the CRDT local-first workspace cohort
Date: 2026-06-14
Source kind: **modern shipped products (cohort dive)** — three open-source, local-first
"all-in-one workspace" tools studied together; candidate shards of a **new substrate
family: CRDT-synced local-first stores**
Lens: shard-wiki — the CRDT substrate and its native-merge / P2P / E2EE implications for
the adapter contract
> Why one combined memo. Anytype, AFFiNE, and AppFlowy are the open-source local-first
> Notion/Miro alternatives, and they share the one trait the prior eleven systems never
> had: **the store is a CRDT** (Anytype's *any-sync*, AFFiNE's *Yjs/y-octo*, AppFlowy's
> *Yrs*). That single fact changes the adapter math — **the backend performs conflict-
> free merge itself**, history is a CRDT update log (not git), and content arrives as a
> replica/export, not files or rows. Studying them together isolates the *substrate*
> contribution from each tool's surface features and avoids triplicating boilerplate.
> Per-tool sections (§1§3), then the cohort thesis and what it means for shard-wiki
> (§4 onward).
Contrast set: file-store (Obsidian/Logseq), client-DB (Roam), hosted-DB (Notion),
SQLite-local+file-sync (Joplin). This cohort adds **CRDT-replica** as a substrate.
---
## 1. Anytype — P2P, E2EE, typed object graph (any-sync)
- **Substrate:** the open **any-sync** protocol — **local-first, peer-to-peer,
end-to-end encrypted** CRDT. Infrastructure is four node types: **sync, file,
consensus, coordinator**; **backup nodes store ciphertext they cannot read**. Every
change is **encrypted and signed with the user's private key**, verified via a shared
public key. Files via **IPFS**.
- **Data model:** **spaces** = graph-based databases; users define **custom object
types with relations** ("everything is an object"; a user-editable **ontology**).
Closest to Notion's typed model but **local-first + E2EE + user-owned schema**.
- **Extension:** an **open API** for integrations, **plus MCP support** (AI-agent
collaboration) — a notable modern surface (a shard speaking MCP).
- shard-wiki read: the **most decentralized + encrypted** shard yet — no single
canonical endpoint (attach a local replica or a node), content opaque without keys
(UC-61), typed object graph (UC-58), CRDT merge (UC-64).
## 2. AFFiNE — docs + whiteboard + DB as views of one block set (Yjs/BlockSuite)
- **Substrate:** **Yjs CRDT** via **y-octo** (native Rust Yjs) over **OctoBase** (Rust
local-first engine). Local-first by default; optional **WebSocket realtime** sync to
AFFiNE Cloud or self-host (MPL-2.0, Docker).
- **Editor:** **BlockSuite** — an open block-editor framework providing the **page
editor, the edgeless canvas, and block types (incl. databases/embeds)**.
- **Headline:** **docs, whiteboards, and databases are *views of the same underlying
data*** — one block set rendered as page mode, edgeless canvas, or DB views. CRDT
guarantees offline edits "merge mathematically perfectly" on reconnect.
- shard-wiki read: the **one-data-many-renderings** model shipped (ZigZag dimensions,
UC-47/48), with **non-Markdown spatial content** (edgeless canvas, UC-55) and CRDT
merge (UC-64).
## 3. AppFlowy — Rust/Flutter, Notion-style DBs over Yrs CRDT
- **Substrate:** **Yrs** (Rust Yjs) wrapped as **AppFlowy-Collab** — CRDT primitives +
persistence + domain objects (documents, **databases**, folders, importers, plugins,
user state). **Pluggable storage/sync:** RocksDB (native) / IndexedDB (wasm) locally;
**Supabase or AppFlowy Cloud** (self-hostable) for sync, via **trait-based local-only
vs cloud**.
- **Data model:** Notion-like **databases with views** (grid/board/calendar), typed
fields. Flutter UI + Rust core, modular.
- shard-wiki read: a **self-hostable CRDT Notion** — typed DBs + views (UC-58, UC-54),
CRDT merge (UC-64), self-host sync endpoint (UC-57).
---
## 4. The cohort thesis — CRDT changes the adapter math
All three converge on **CRDT-synced, local-first, open-source, self-hostable** stores
with **typed/database object models** and **non-Markdown content** (canvas/DB). The
*substrate* is the new thing, and it has four consequences shard-wiki must absorb:
1. **Native, conflict-free merge (UC-64).** The backend resolves concurrent edits
itself (Yjs/Yrs/any-sync). shard-wiki must **not impose git/text merge** on a CRDT
shard; it either speaks the CRDT (replica) or stays a **projection/overlay** that
respects CRDT semantics. This adds a **merge-model** capability:
`none → git/text → native-CRDT` — a candidate **thirteenth spectrum** for the
contract (after Joplin's content-opacity twelfth).
2. **History is a CRDT update log, not git (UC-36).** No portable commit history; the
coordination journal **supplements** (like Notion/Joplin), or shard-wiki snapshots
the replica over time.
3. **The attach surface is a replica or a sync endpoint, not files/rows.** Attach a
**local CRDT replica** (offline, full state) or the **self-host sync server's API**
(AFFiNE Cloud / AppFlowy Cloud / any-sync node). Anytype adds **P2P with no single
canonical endpoint** (UC-65) and **E2EE content opacity** (UC-61).
4. **Typed object graphs + multi-view (UC-58, UC-47/48, UC-55).** User-editable
ontologies (Anytype types/relations), Notion-style DBs (AppFlowy), and one-block-set-
many-renderings (AFFiNE page/canvas/DB) — the structured + dimensional demands the
prior dives raised, now local-first.
These are **sync layers** (any-sync, Yjs sync, AppFlowy Cloud) — like Joplin, the
**not-a-file-sync-daemon** boundary applies: shard-wiki attaches a **replica/projection
as pages** and does **not** re-implement or re-drive their CRDT sync.
---
## 5. Comparative matrix
| | Anytype | AFFiNE | AppFlowy |
|---|---------|--------|----------|
| CRDT | any-sync (custom) | Yjs (y-octo) | Yrs |
| Sync | **P2P + nodes, E2EE** | WebSocket, cloud/self-host | RocksDB local + cloud/Supabase |
| Encryption | **E2EE (default)** | optional (transport/self-host) | self-host boundary |
| Local engine | local + IPFS files | OctoBase (Rust) | RocksDB / IndexedDB |
| Data model | **typed object graph (ontology)** | blocks; **doc/canvas/DB = one data, many views** | Notion-style **DBs + views** |
| Non-MD content | objects, files | **edgeless canvas** | board/calendar/grid |
| Extension | **open API + MCP** | BlockSuite framework | Flutter/Rust modules; plugins (roadmap) |
| Self-host | backup/sync nodes | Docker (MPL-2.0) | AppFlowy Cloud |
| Markdown | export (lossy) | import/export (lossy) | import/export (lossy) |
---
## 6. Combined capability profile (as a shard)
| Capability | This cohort | Notes for the adapter contract |
|------------|-------------|--------------------------------|
| Read | yes (replica or sync API) | local CRDT replica (offline) or self-host/P2P endpoint |
| Write | yes, **CRDT-mediated** | writes are CRDT ops; respect merge — don't overwrite |
| Merge | **native CRDT (conflict-free)** | **new capability**: backend owns merge; no external git merge (UC-64) |
| Write granularity | block/object | fine-grained |
| Identity / addressing | object/block IDs (CRDT-assigned) | fine-grained, store-minted (UC-51) |
| Structure | **typed object graph / DBs + relations** | user-editable ontology (Anytype); DB views (AppFlowy) (UC-58) |
| History | **CRDT update log, not git** | supplement / snapshot (UC-36) |
| Native query | varies (graph/DB filters) | delegate where present (UC-52) |
| Subscribe | realtime CRDT sync / P2P | push by nature; attach as replica (UC-31) |
| Content opacity | **Anytype E2EE → ciphertext** | opaque without keys (UC-61) |
| Endpoint model | **replica / self-host server / P2P (Anytype)** | no single canonical URL for P2P (UC-65) |
| Content types | docs + **canvas/DB/objects** | non-Markdown (UC-55); multi-view (UC-47/48) |
| Markdown | lossy export | fidelity report (UC-59) |
| Attach modes | CRDT replica · self-host API · (Anytype) MCP/open API | new substrate row in the taxonomy |
Verdict: a **demanding but coherent new family** — best attached as a **local replica
(offline, full CRDT state)** projected to pages, or via a **self-host sync endpoint**;
Anytype additionally **P2P + E2EE** (replica-only/opaque). Standout demands:
**native-CRDT merge** (UC-64) and **P2P/no-central-endpoint + E2EE** (UC-65/UC-61).
---
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
- **Local-first sovereignty** is the cohort's whole pitch — strong INTENT alignment;
each is a legitimate sovereign shard with user-owned data.
- **Self-hostable + open-source** means shard-wiki can attach within a user's trust
boundary (good for the L0→L4 authz ladder).
- **One-data-many-views** (AFFiNE) and **typed object graphs** (Anytype/AppFlowy) are
the dimensional (UC-47/48) and structured (UC-58) demands, now validated local-first.
### 7.2 Deliberate divergences (design bugs if conflated)
1. **Do not impose git/text merge on a CRDT shard.** The backend merges; shard-wiki
respects CRDT semantics or stays a projection/overlay (UC-64). Treating a CRDT shard
like a file shard corrupts it.
2. **Not a sync daemon.** any-sync/Yjs/AppFlowy Cloud already sync; shard-wiki attaches
a **replica as pages**, never re-drives their sync (INTENT not-a-file-sync-daemon).
3. **P2P has no canonical endpoint.** Don't model the shard as a URL; attach a replica
or a named peer/node (UC-65).
4. **E2EE shards are opaque** without keys — backup/structure-shell only (UC-61).
5. **Proprietary CRDT/object formats are lossy to Markdown** — translate with a fidelity
report (UC-59), one workspace = one shard, not the federation layer.
### 7.3 What the cohort teaches that shard-wiki should keep
- Add a **merge-model** capability (`none / git / native-CRDT`) so CRDT shards are
handled correctly (UC-64) — a thirteenth capability spectrum.
- Add **CRDT-replica** to the substrate/attachment taxonomy, and **P2P/no-central-
endpoint** as a binding mode (UC-64/65).
- **MCP as an integration surface** (Anytype) is worth noting — a shard may expose an
AI-agent protocol shard-wiki could consume or coexist with.
---
## 8. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-63**. New UCs **UC-64, UC-65** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a CRDT-synced local-first shard** whose backend performs native conflict-free merge; respect CRDT semantics (no git merge), consume a replica/export, CRDT-log history (supplement) | **UC-64 (new)** |
| **Attach a peer-to-peer / decentralized shard with no single canonical endpoint** (Anytype any-sync: replica or peer/backup node; E2EE) | **UC-65 (new)** |
| Typed object graph / user-editable ontology + Notion-style DBs, local-first | **enriches UC-58** |
| E2EE content opacity (Anytype) reinforced; self-host trust boundary | **enriches UC-61** |
| CRDT update log ≠ portable git history → supplement | **enriches UC-36** |
| One block set rendered as page / canvas / DB views (AFFiNE) | **enriches UC-47, UC-48** |
| Edgeless canvas / board / objects = non-Markdown content | **enriches UC-55** |
| Self-host sync server (AFFiNE/AppFlowy Cloud) + Anytype open API/MCP as endpoints | **enriches UC-57** |
| Object/block IDs CRDT-assigned (fine-grained) | **enriches UC-51** |
| Lossy Markdown export | links UC-59 |
---
## 9. Architecture notes for SHARD-WP-0002 (no UC)
- **Merge-model capability** (`none / git-or-text / native-CRDT`) — proposed
**thirteenth spectrum** (with Joplin's content-opacity twelfth). CRDT shards reject
external merge; shard-wiki overlays must be CRDT-aware or remain projections. (T11.)
- **Substrate taxonomy gains CRDT-replica**; **attach modes gain "local CRDT replica"
and "P2P/no-central-endpoint"** beside file-store / in-engine-host / external-API.
(T14.)
- **Endpoint model** as a capability: single URL vs self-host server vs **P2P peer set**
(Anytype) — affects how a binding is addressed and how freshness/sync work. (T14.)
- **MCP / AI-agent protocol** as a possible shard integration surface (Anytype). Note
for the extension-surface catalogue. (T14.)
- These three plus Joplin and Logseq all argue the contract must treat **substrate as a
spectrum that tools migrate along and that shard-wiki must not hard-code**.
---
## 10. Open questions (for spec / workplans)
1. For a **CRDT shard** (UC-64), does shard-wiki embed a CRDT client (Yjs/Yrs) to hold a
live replica, or only consume periodic **exports/snapshots**? The former gives
live/writable; the latter is simpler but lossy on merge timing.
2. How are **overlays** expressed against a CRDT shard — as CRDT ops (requires the
client) or as out-of-band patches reconciled later (projection-only)?
3. For **P2P shards** (UC-65), what is the "address" of the shard — a space ID + a peer/
node, a local replica path, or an invite/key? How does this fit shard attachment
config?
4. Does shard-wiki ever hold **E2EE keys** (Anytype) to project content, or only ever
treat such shards as opaque replicas/backups (UC-61)?
5. Is the **self-host sync server** (AFFiNE/AppFlowy Cloud) attached as an external API
(UC-57) or by running a co-located replica?
---
## 11. Sources
| Source | Used for |
|--------|----------|
| Anytype any-sync — Protocol overview (https://tech.anytype.io/any-sync/overview) | Local-first/P2P/E2EE CRDT; sync/file/consensus/coordinator nodes; encrypted backup |
| anyproto/any-sync (https://github.com/anyproto/any-sync) | Protocol scope; keys signed/verified; IPFS files |
| Anytype Docs (https://doc.anytype.io/anytype-docs) | Spaces, object types + relations (ontology); open API + MCP |
| toeverything/AFFiNE + DeepWiki (https://deepwiki.com/toeverything/AFFiNE) | BlockSuite (page/edgeless/DB); y-octo Yjs; OctoBase; one-data-many-views; self-host |
| AFFiNE — interactive whiteboard / blog (https://affine.pro/blog/interactive-whiteboard-app) | Edgeless canvas; CRDT offline merge |
| AppFlowy-IO/AppFlowy-Collab (https://github.com/AppFlowy-IO/AppFlowy-Collab) | Yrs CRDT; collab domain objects (documents/databases/folders); persistence |
| AppFlowy Architecture Overview — DeepWiki (https://deepwiki.com/AppFlowy-IO/AppFlowy/2-architecture-overview) | Flutter/Rust layers; pluggable storage/sync (RocksDB/IndexedDB; Supabase/AppFlowy Cloud); local vs cloud trait |
| AFFiNE vs AppFlowy vs Anytype (https://affine.pro/blog/affine-vs-appflowy-vs-anytype) | Cohort comparison framing |
Cross-references: `research/260614-notion-deep-dive/findings.md` (typed DB, hosted
contrast), `research/260614-joplin-deep-dive/findings.md` (E2EE/content opacity, sync-
daemon boundary), `research/260614-shard-spectrum-synthesis/findings.md` (the spectra
this extends), `spec/UseCaseCatalog.md` (UC-36, UC-47, UC-48, UC-51, UC-55, UC-57,
UC-58, UC-61), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
---
## 12. Traceability
- New UCs: **UC-64, UC-65**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-36, UC-47, UC-48, UC-51, UC-55, UC-57, UC-58, UC-61** (links UC-59).
- Architecture (no UC): **merge-model** capability (proposed thirteenth spectrum);
CRDT-replica substrate + P2P/no-central-endpoint attach mode; endpoint-model capability;
MCP integration surface → `SHARD-WP-0002` (T11, T14).
- Boundary recorded: Anytype/AFFiNE/AppFlowy are **CRDT local-first candidate shards**
(attach a replica/projection as pages; respect native CRDT merge; never re-drive their
sync; P2P/E2EE shards are replica-only/opaque) — not substrates, not the federation
layer (INTENT not-a-sync-daemon, graceful degradation, no silent mutation).
</content>

View File

@@ -0,0 +1,50 @@
# 260614 — Logseq deep dive (block-graph semantics on plain Markdown files)
Date: 2026-06-14
## What this is
A focused study of **Logseq** — the open-source, local-first outliner — read through
shard-wiki's lens. Logseq occupies the point the other modern tools leave empty:
**block-graph semantics (UUID-addressable, embeddable, queryable blocks) stored as plain
Markdown/Org files on disk**, with a DataScript graph **derived** from those files. It is
the bridge between **Roam** (block-graph, client-DB) and **Obsidian** (file-over-app,
page-level), and it resolves a tension the synthesis flagged: block-level addressing
*that is also git-diffable text*.
Distinctive material:
- **Architecture** — files canonical (`pages/`, `journals/` MD/Org); a DataScript graph
**derived** via an `mldoc` AST parse; a DB Worker now manages **both DataScript and
SQLite** (the file→DB migration)
- **Block-graph in the file text** — block IDs as in-file `id:: <uuid>` properties,
`((uuid))` refs, `key:: value` properties, `{{embed}}` transclusion, the outline tree
(indent/zoom/move-subtree) — all git-diffable
- **Queries** — `{{query}}` + advanced **Datalog** over the derived graph
(`logseq.DB.datascriptQuery`)
- **Extension** — plugin API (`logseq.App/Editor/DB/Git/UI/Assets/FileStorage`),
marketplace (~486 plugins); dual-attachable (file-store direct *or* in-app plugin)
- **Trajectory** — migrating from Markdown-files to a SQLite "DB graph" (a live UC-43)
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, in-file block-graph, Datalog queries, plugin API, file→DB migration, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md`
(UC-62 attach a block-graph-on-plain-files shard with in-file block IDs/properties +
outline tree + derivable query index; UC-63 serve structured queries over a file-backed
shard via a derived index the orchestrator/adapter builds); UC-32/34/43/50/51/52/55
enriched. Logged for `SHARD-WP-0002` (T11/T14/T16): a Logseq **format profile** for
file-store adapters, a **derived-query-index** capability, **substrate-migration
tolerance**, and **in-file block addressing** as the concrete T16 span-address target.
**Key takeaway recorded:** Logseq resolves the addressing-spectrum tension — block-level
addressing **and** git-diffable in-file text (`id::`) — and proves a file-backed shard
can support rich Datalog queries via a **derived** index (files canonical, graph
derived). **Boundary:** one block-graph-on-files candidate shard (the addressing sweet
spot), best attached file-store-direct with a format profile; not the federation layer;
substrate is migrating file→SQLite.
</content>

View File

@@ -0,0 +1,246 @@
# Findings — Logseq: block-graph semantics on plain Markdown files
Date: 2026-06-14
Source kind: **modern shipped product** — an open-source, local-first outliner; a
*candidate shard* occupying the point none of the prior tools do: **block-graph + file-
over-app**
Lens: shard-wiki — the convergence of block-level addressing with git-diffable files, a
derived query index over canonical files, and a live file→DB storage migration
> Why Logseq matters distinctly. The modern-tool dives staked out the corners: **Roam**
> = block-graph but client-DB / in-app-API / store-minted UUID; **Obsidian** =
> file-over-app but page-level / path-addressed (block `^id` opt-in); **Notion** =
> block-graph but closed hosted DB / external-API. **Logseq sits in the middle they
> leave empty: block-graph *semantics* (UUID-addressable, embeddable, queryable blocks)
> stored as *plain Markdown/Org files on disk you own*, with a DataScript index
> **derived** from those files.** It is the system that proves you do not have to choose
> between Roam's fine-grained addressing and Obsidian's file sovereignty — you can have
> block-level addressing *that is also git-diffable text*. That resolves a tension the
> synthesis flagged (addressing spectrum) and is the reason it earns a memo rather than
> a footnote.
Contrast set: Roam (block-DB, in-app), Obsidian (file, page-level), Notion (hosted DB),
Joplin (SQLite-local, files-on-sync). Logseq = **block-graph, files-canonical,
index-derived** — and now *also* migrating toward SQLite (§5), a live UC-43 case.
---
## 1. Core architecture — files canonical, DataScript derived
- **Storage:** plain-text **Markdown or Org-mode files on disk**`pages/<Page>.md` and
`journals/<date>.md` — fully usable even if the app disappears (local-first, open
source, ClojureScript).
- **Index:** a **DataScript** in-memory graph database, **derived** from the files: an
`mldoc` parser turns Markdown into an AST, extracts references/properties, and
transforms it into **DataScript entities** with tree-position attributes. Files are the
source of truth; the graph DB is a rebuildable projection. (In current versions a **DB
Worker** manages **both DataScript and SQLite** connections — see §5.)
- This is the **files-canonical / index-derived** architecture (Obsidian's MetadataCache,
Git's working tree) — but here the derived index is a *full Datalog-queryable graph*,
not just a metadata cache. Logseq is the strongest evidence that **a file-backed shard
can support rich structured queries via a derived index** (UC-52, UC-63).
---
## 2. The block-graph, in the file text
Everything is an **atomic block** (a bullet), individually referenceable, embeddable, and
queryable — Roam's model — but the addressing lives **in the Markdown**:
- **Block IDs are in-file properties.** When a block is referenced, Logseq writes an
`id:: <uuid>` property line into the Markdown. So the block's stable address is
**git-diffable text that survives a file copy**, not a DB-minted hidden key. This is the
**sweet spot of the addressing spectrum**: block-level (like Roam) *and* in-file/
portable (like Obsidian's `^id`), without choosing.
- **References:** `[[page]]`, **`((block-uuid))`**, `#tag` — all extracted into the graph;
block embeds `{{embed ((uuid))}}` are transclusion at block granularity (UC-32).
- **Properties:** `key:: value` lines attach typed metadata to blocks (block properties)
and pages (first-block/page properties) — **git-diffable structured data at block
granularity** (UC-34), queried by Datalog.
- **Outline tree:** a page is a **nested tree of blocks** (indent = structure), with
outliner operations — indent/outdent, move subtree, **zoom/narrow** into a block. The
page is not flat prose; it is an addressable, reorderable block tree (UC-63).
---
## 3. Queries — Datalog over the derived graph
Logseq exposes **`{{query}}`** (simple) and **advanced Datalog queries** over the
DataScript graph (and via the plugin API, `logseq.DB.datascriptQuery`). Because the graph
is derived from files, **query-defined pages** (UC-54) and structured aggregation (tasks,
tagged blocks) run over a *file-backed* store. Key lesson: query capability here is
**neither native-DB (Notion) nor a third-party plugin (Obsidian Dataview)** — it is
**built into the tool over its own files**, demonstrating that *a derived query index is a
viable adapter/orchestrator capability for file shards* (UC-63).
---
## 4. Extension surface — plugin API + marketplace
Logseq has a JS **plugin API** (marketplace ~486 plugins, `logseq/marketplace`):
- `logseq.Editor` — block/page CRUD, insert/move/update/delete, get current block/page.
- `logseq.DB`**`datascriptQuery`** (Datalog), plus DB change subscriptions.
- `logseq.App` — app-level ops/events; `logseq.UI`; `logseq.Git` (git ops on the graph);
`logseq.Assets`; `logseq.FileStorage`.
- Slash commands, block/page context menus, `provideModel`/`provideStyle`, hooks/events.
So, like Obsidian, Logseq is **dual-attachable**: (a) **file-store direct** — read the
`pages/`+`journals/` Markdown with a **Logseq format profile** (parse `id::`, `((uuid))`,
`key::`, the outline tree); (b) **in-app plugin host**`logseq.Editor` write +
`logseq.DB` query + change events for live write-through. A notable extra: a built-in
**`logseq.Git`** surface — the tool treats git as a first-class companion to the file
graph (validating the coordination journal).
---
## 5. The file→DB migration — a live UC-43
Logseq is **migrating its storage model from Markdown-files to a SQLite "DB graph"**
(the DB Worker already manages SQLite alongside DataScript; the plugin API has a distinct
"DB graph" mode with tags/classes/typed properties). This is a real-world instance of
**UC-43 (backend-swap under stable identity)**: the *same tool and graph identity* moving
from a file substrate to a DB substrate, trading git-diffability for richer typed
structure (toward the Notion/XWiki end). For shard-wiki it is both a caution (a shard's
substrate can change under it) and a confirmation that the **addressing/structure/history
spectra** are trajectories tools actually travel — an adapter keyed to a fixed substrate
will break.
---
## 6. Logseq as a shard — capability profile
| Capability | Logseq (MD-file graph) | Notes for the adapter contract |
|------------|------------------------|--------------------------------|
| Read | **yes** | file-store direct (pages/journals MD) or `logseq.Editor`/`logseq.DB` in-app |
| Write | **yes** | direct file write (format-aware) or plugin; block-level |
| Write granularity | **per-block** (in a per-file page) | finer than Obsidian (per-file), like Roam — but on files |
| Identity / addressing | **in-file block `id:: uuid` + `((uuid))`** | block-level **and** git-diffable — the addressing sweet spot (UC-51) |
| Structure | `key:: value` block/page properties; outline tree | git-diffable structured data at block granularity (UC-34) |
| History | none native; **`logseq.Git`** + files = git-native | git-friendly out of the box (adopt, not supplement) |
| Native query | **Datalog over derived DataScript** | derived index over files → delegate or rebuild (UC-52, UC-63) |
| Transclusion | **block embeds `{{embed ((uuid))}}`** | in-file, addressable (UC-32) |
| Backlinks | linked + unlinked references | derived (UC-05/18) |
| Content types | Markdown/Org + **whiteboards** (tldraw JSON) | non-Markdown spatial content (UC-55) |
| Substrate | **files now, SQLite "DB graph" emerging** | live backend-swap (UC-43) |
| Attach modes | file-store direct (format profile) · in-app plugin | dual, per-binding (UC-62) |
Verdict: **the most shard-wiki-friendly block tool** — block-graph power with file-over-
app sovereignty and git-diffable addressing/structure. Best attach: **file-store direct
with a Logseq format profile** (offline, git-native), with the plugin for live write-
through. Watch the **DB-graph migration** (UC-43).
---
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
- **Resolves the addressing tension.** Block-level addressing *can* be git-diffable
in-file (`id::`), not forced to choose Roam-DB vs Obsidian-page (UC-51, synthesis §2).
- **Confirms files-canonical / index-derived at full power** — a Datalog graph derived
from files, not just a metadata cache (UC-52, UC-63).
- **Structure as git-diffable text** (`key::`) reinforces "prefer in-text structure"
(UC-34, synthesis through-line).
- **Outliner block tree** validates the page-as-addressable-block-tree demand (UC-50,
UC-63) on a *file* backend.
### 7.2 Deliberate divergences (design bugs if conflated)
1. **One graph = one shard.** Local-first single graph; not the federation layer.
2. **The MD files carry Logseq-specific syntax** (`id::`, `((uuid))`, `key::`, outline
semantics) — a **format profile**, not plain CommonMark. A naïve Markdown reader will
mangle block IDs/properties (cf. UC-42; lighter than Notion's lossy case).
3. **The substrate is moving (file→SQLite).** Don't hard-code the file model; gate on the
substrate capability and tolerate the swap (UC-43).
4. **Whiteboards are not Markdown** — typed/opaque assets, not flattened (UC-55).
### 7.3 What Logseq teaches that shard-wiki should keep
- **In-file block addressing is the target shape** for a portable span address where the
backend cooperates — adopt `id::`-style schemes; they are git-diffable and survive copy.
- **A derived query index over files is a first-class capability** — shard-wiki can build
one over a file shard (UC-63) when the backend exposes none, or delegate when it does.
- **Expect substrate migration** — bind to capabilities, not to "it's files."
---
## 8. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-61**. New UCs **UC-62, UC-63** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a block-graph-on-plain-files shard** (Logseq-style): file-over-app Markdown carrying in-file block IDs (`id::`), block refs, and `key::` properties, with the outline tree and a derivable query index | **UC-62 (new)** |
| **Serve structured queries over a file-backed shard via a derived index** the orchestrator/adapter builds (Logseq DataScript-over-files) when the backend exposes none | **UC-63 (new)** |
| In-file block `id::` = block-level **and** git-diffable addressing (the spectrum sweet spot) | **enriches UC-51** |
| Live **file→SQLite substrate migration** under stable graph identity | **enriches UC-43** |
| Block-graph that is **files, not a DB** — the file-store variant of the block tool family | **enriches UC-50** |
| Datalog over a **derived** index built from files | **enriches UC-52** |
| `key:: value` block/page properties in-text | **enriches UC-34** |
| Whiteboards (tldraw JSON) = non-Markdown content | **enriches UC-55** |
| Block embeds `{{embed ((uuid))}}` = in-file transclusion | links UC-32 |
---
## 9. Architecture notes for SHARD-WP-0002 (no UC)
- **Format-aware file-store profiles** (already flagged by Joplin, UC-60) gain a strong
case here: a Logseq profile parses `id::`/`((uuid))`/`key::`/outline from otherwise-
plain Markdown. The contract should let a file-store adapter declare its **format
profile** (plain MD / Obsidian / Logseq / Joplin-item / Foswiki-PlainFile). (T11/T14.)
- **Derived-query-index as an orchestrator capability** (UC-63): when a file shard has no
native query engine, build a DataScript-like index over the projection; when it does
(Roam/Notion/XWiki), delegate (UC-52). Both belong in T16's navigation layer + T11.
- **Substrate-migration tolerance** (UC-43, Logseq file→SQLite): T14 binding should treat
substrate as a capability that can change under a live attachment, preserving identity.
- **In-file block addressing** is the concrete realization of T16's span-address thread —
prefer `id::`-style git-diffable IDs over DB-minted where the backend allows.
---
## 10. Open questions (for spec / workplans)
1. When the same tool offers **file** and **DB** substrates (Logseq now), does shard-wiki
prefer the file graph (git-diffable, UC-62) or the DB graph (richer typed structure),
and can one binding follow the migration (UC-43)?
2. Is the **derived query index** (UC-63) built by the **adapter** (per-shard) or the
**core orchestrator** (over the union), and is it persisted or rebuilt?
3. How much **Logseq outline semantics** (zoom, subtree move) must shard-wiki preserve vs.
present as a flat page (UC-63 vs. Markdown-first page model)?
4. Does the **Logseq format profile** round-trip overlays (write `id::`/`key::` back) or
only read them? (cf. UC-42 round-trip question.)
---
## 11. Sources
| Source | Used for |
|--------|----------|
| DeepWiki — logseq/logseq (https://deepwiki.com/logseq/logseq) | DataScript+Datalog graph, mldoc AST parse, block entities/tree, DB Worker managing DataScript **and** SQLite |
| logseq/docs (https://deepwiki.com/logseq/docs) | Plain-text MD/Org files; pages/journals; references ([[ ]], (( )), #tag); properties |
| Logseq Plugin API docs (https://plugins-doc.logseq.com/) | `logseq.App/Editor/DB/Git/UI/Assets/FileStorage`; `DB.datascriptQuery` |
| logseq/marketplace (https://github.com/logseq/marketplace) | Plugin distribution; ~486 plugins |
| kerim/logseq-db-plugin-api-skill (https://github.com/kerim/logseq-db-plugin-api-skill) | DB-graph version: tags/classes, typed properties, file→DB migration |
| pangea.app glossary — Logseq (https://pangea.app/glossary/logseq) | Local-first framing, outliner, plain-text control |
Cross-references: `research/260614-roam-deep-dive/findings.md` (block-DB contrast),
`research/260614-obsidian-deep-dive/findings.md` (file-over-app, derived index),
`research/260614-joplin-deep-dive/findings.md` (format-aware file-store profiles),
`research/260614-shard-spectrum-synthesis/findings.md` (addressing spectrum this
resolves), `spec/UseCaseCatalog.md` (UC-32, UC-34, UC-43, UC-50, UC-51, UC-52, UC-55),
`workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14, T16).
---
## 12. Traceability
- New UCs: **UC-62, UC-63**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-32, UC-34, UC-43, UC-50, UC-51, UC-52, UC-55**.
- Architecture (no UC): format-aware file-store profiles (Logseq profile); derived-query-
index capability; substrate-migration tolerance; in-file block addressing as the T16
span-address target → `SHARD-WP-0002` (T11, T14, T16).
- Boundary recorded: Logseq is **one block-graph-on-files candidate shard** (the
addressing sweet spot), best attached file-store-direct with a format profile; not the
federation layer; substrate is migrating file→SQLite (UC-43).
</content>

View File

@@ -0,0 +1,32 @@
# 260614 — Mathematica Notebooks deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T2**
## What this is
A deep dive into **Wolfram Mathematica Notebooks** — the **original computational notebook**
(1988), the ancestor Jupyter descends from. The `.nb` document **is itself a Wolfram Language
expression** (`Notebook[{Cell[…], …}]`) with **nested cell groups**, kernel evaluation
(`In`/`Out`), **structured (symbolic/graphics) results**, live `Manipulate`/`Dynamic`
widgets, and CDF as a reduced-runtime distribution projection.
## Why it matters
- Confirms the **notebook page shape (UC-84) is a genus**, not a Jupyter quirk: cells +
cached outputs + fragile counter provenance + kernel-gated re-execution predate `.ipynb`.
- Two refinements: notebooks have **nestable cell groups (an outline tree)**, not just an
ordered list; outputs can be **structured re-evaluable values**, not only MIME blobs →
adds a "structured re-evaluable value" point to the content-opacity spectrum.
- `Manipulate`/`Dynamic` join the **static-projection-impossible** end (snapshot-only) —
foreshadows Strudel (T5).
## Yield
- **No new UC** (lineage/reinforcement). Reinforces **UC-84**; enriches **UC-54, UC-55**;
links UC-34, UC-83.
## Contents
| Path | Role |
|------|------|
| `findings.md` | `.nb` expression/cell model, evaluation & provenance, CDF, capability delta vs Jupyter, INTENT mapping, UC disposition (enrichment-only), architecture notes |

View File

@@ -0,0 +1,139 @@
# Mathematica Notebooks — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T2 · **Subject:** Wolfram Mathematica
Notebooks — the `.nb` format, the cell/expression model, symbolic evaluation.
## Why this dive
Mathematica (1988) is the **original computational notebook** — the ancestor Jupyter (T3)
descends from. This dive checks whether the notebook page-model conclusions from UC-84 hold
at the genus level or need extension. It is **medium priority / lineage**: a candidate
*content type* like Jupyter, but closed/proprietary, so the expected yield is **reinforcement
of UC-84** plus a couple of distinct wrinkles (the document *is itself* a Wolfram-language
expression; symbolic, not just textual, results).
## 1. The `.nb` document model
A Mathematica notebook is itself a **Wolfram Language expression**`Notebook[{Cell[...],
Cell[...], ...}, opts]`. So the document format and the language are the *same* substrate:
- **Cells** are typed: `Input`, `Output`, `Text`, `Title`/`Section` (structure), `Code`,
with **cell groups** nesting into an **outline tree** (the document has real hierarchy, not
just a flat list like `.ipynb`).
- An **`Input` cell** holds an expression; evaluating it produces a linked **`Output` cell`**
containing the **result expression** (symbolic, graphical via `Graphics[...]`, or
typeset). The result is a **first-class expression**, not a MIME blob — it can be
re-evaluated, edited, transcluded.
- The whole `.nb` is **plain-text-serializable** (it's an expression) but verbose and
proprietary in conventions; output cells are **cached results** stored in the file.
So like Jupyter: **source + cached computed output fused in one document**, with **out-of-
band reproducibility** (kernel + package/version state). Unlike Jupyter: **nested cell
groups (a tree)** and **results that are structured Wolfram expressions** rather than MIME
bundles.
## 2. Evaluation, provenance, reproducibility
- A **kernel** evaluates input cells; `In[n]`/`Out[n]` counters mirror Jupyter's
`execution_count` and carry the **same fragility** (out-of-order evaluation, hidden
symbol/global state, kernel-version dependence). No environment capture in the file.
- **Dynamic/interactive output** (`Manipulate`, `Dynamic`) embeds **live interactive
widgets** whose state is computed on view — these have **no faithful static form** beyond a
snapshot frame (echoes Strudel T5's "live, time/interaction-based content" limit).
- **CDF (Computable Document Format)** is Wolfram's *projection-for-distribution*: a notebook
rendered with a free runtime so readers can interact without a full license — a
derivation-projection (T1) with a reduced-capability viewer.
## 3. Capability profile (delta vs Jupyter UC-84)
| Dimension | Mathematica `.nb` (delta from Jupyter) |
|-----------|----------------------------------------|
| Structure | **nested cell groups → outline tree** (richer than `.ipynb`'s flat cell list) |
| Output type | **structured Wolfram expressions** (symbolic/graphics), not MIME blobs |
| Document = language | the `.nb` **is** a Wolfram expression (format ≡ language) |
| Liveness | `Manipulate`/`Dynamic` = **interactive widgets**, snapshot-only when static |
| Opacity | proprietary serialization; results re-evaluable only with a Wolfram kernel |
| Projection-for-distribution | **CDF** = reduced-runtime interactive projection |
| Otherwise | same as UC-84: cells, cached outputs, fragile `In/Out` provenance, kernel-gated |
## 4. INTENT mapping
### Reinforcements (mostly confirms UC-84)
- **Notebook page shape (UC-84) is a genus, not a Jupyter quirk.** Mathematica predates and
matches it: cells + cached computed outputs + fragile counter provenance + kernel-gated
re-execution. Confirms the page model should carry a **notebook shape** generically (T12),
not a `.ipynb`-specific one.
- **Outputs as derivation-projection snapshots (UC-83/84).** Cached `Output` cells are
snapshots; honest treatment marks them "evaluated run N, kernel/env unguaranteed."
- **Derivation-projection for distribution (T1).** CDF is a clean "reduced-capability
interactive projection of a source" — a real-world instance of degrade-by-capability.
### Distinct wrinkles (extend the notes, not new UCs)
- **Nested cell-group outline** — the notebook page model should allow **hierarchical cell
grouping**, not just an ordered list (generalize UC-84's "ordered cells" to "ordered/
*nestable* cells"). Feeds T12.
- **Structured (non-MIME) results** — outputs can be **typed structured values** (symbolic
expressions), not only MIME blobs; reinforces UC-55's "typed asset" reading over "opaque
blob," and links the typed-record page model (UC-34) — the content-opacity spectrum needs a
"structured re-evaluable value" point, not just text↔blob.
- **Format ≡ language** — a curiosity, not actionable for us beyond noting that some shards'
document format is *the same artifact* as their computation (don't assume doc/code split).
- **Live interactive widgets** — `Manipulate`/`Dynamic` join Strudel (T5) at the
**static-projection-impossible** end: capture a snapshot frame, never imply interactivity.
### Boundaries
- Proprietary + kernel-gated → default **read/projection/snapshot**; attach the `.nb`
(or an exported form), present cached outputs as snapshots, offer a static/CDF projection;
**no kernel host** (same rule as Jupyter UC-84, GT T7).
## 5. UC disposition (enrichment-only — no new UC)
| Mechanism (findings §) | Catalog UC |
|------------------------|------------|
| Cells + cached computed outputs + fragile In/Out provenance; kernel-gated (§1, §2) | UC-84 (reinforced) |
| Nested cell groups → outline tree (richer than flat `.ipynb`) (§1) | UC-84 (enriched: nestable cells); links UC-34 |
| Output = structured re-evaluable Wolfram expression, not MIME blob (§1) | UC-55 (enriched: structured value point on opacity spectrum) |
| Input cell = computation-defined content (§1) | UC-54 (enriched) |
| `Manipulate`/`Dynamic` interactive output = snapshot-only (§2) | links UC-55, foreshadows T5 |
| CDF = reduced-runtime interactive distribution projection (§2) | links UC-83 (derivation-projection) |
Mathematica is a **lineage/reinforcement** dive — it **adds no new UC**, confirming UC-84's
notebook page shape as a genus and contributing two refinements (nestable cells; a
**structured re-evaluable value** point on the content-opacity spectrum).
## 6. Architecture notes for SHARD-WP-0002
- **T12 (page model):** generalize the notebook shape from "ordered cells" (UC-84) to
**ordered/nestable cell groups (an outline tree)**; allow code-cell outputs to be **typed
structured values**, not only MIME blobs.
- **T11/T15 (content opacity / fidelity):** add a **"structured re-evaluable value"** point
to the content-opacity spectrum between transparent-text and opaque-blob (Wolfram
expression, symbolic result) — relevant to how outputs are stored/surfaced.
- **T16 (projection):** CDF is a **reduced-capability interactive projection**; with
`Manipulate`/`Dynamic`, static projection is a **snapshot frame only** (join the
live-content limit recorded for Strudel T5).
## 7. Open questions
1. Is a **structured re-evaluable result** (Wolfram expression) modeled as a typed value in
the page model, or stored opaquely with provenance like other computed outputs? (Ties
UC-55 open-Q #10 and UC-84 Q3.)
2. Do interactive outputs (`Manipulate`, `Dynamic`, and Jupyter widgets) deserve a shared
**"interactive, snapshot-only" content marker** in the contract? (Recurs at T4/T5.)
## 8. Sources
- Wolfram documentation: notebook format (`Notebook`/`Cell` expressions), cell types &
groups, evaluation (`In`/`Out`), `Manipulate`/`Dynamic`, CDF.
- prior: `research/260614-jupyter-deep-dive/` (UC-84 notebook shape; the descendant);
`research/260614-literate-programming-deep-dive/` (derivation-projection, UC-83).
## 9. Traceability
**No new UC** (lineage/reinforcement). Reinforced: UC-84; enriched: UC-54, UC-55; links
UC-34, UC-83. Architecture cross-refs: SHARD-WP-0002 T12 (nestable cell-group outline; typed
structured outputs), T11/T15 (structured-re-evaluable-value point on the opacity spectrum),
T16 (CDF reduced-capability projection; interactive = snapshot-only).

View File

@@ -0,0 +1,15 @@
# 260614 — MojoMojo deep dive
Deep dive on **MojoMojo**: a **Perl Catalyst / DBIx::Class** DB-backed wiki — hierarchical
pages, attachments, inline (AJAX) editing, Markdown content, and **page history in
relational version tables**. The classic **MVC DB-backed** contrast to the file-store
classics: no file store, no real content API → attach by reading the **relational store
directly**.
- `findings.md` — architecture, the DB schema shape, capability profile, INTENT mapping, UC
seed (UC-81), architecture notes for SHARD-WP-0002, open questions, sources, traceability.
Catalog yield: UC-81 (attach a **DB-backed wiki with no file store/API** by reading its
relational store directly — pages + version tables — and importing DB-resident history to the
journal). Enriched UC-02/40/36/34. Feeds SHARD-WP-0002 T13 (history portability), T14
(direct-DB binding).

View File

@@ -0,0 +1,133 @@
# MojoMojo — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T8 · **Subject:** MojoMojo, a Perl
Catalyst wiki/CMS.
## Why this dive
The file-store classics (TWiki, Foswiki, Oddmuse, UseMod) keep pages as files; the modern
SaaS keep them behind APIs. MojoMojo is the **classic relational-DB-backed** wiki — a
Catalyst MVC app over **DBIx::Class** with pages and their history in **SQL tables**, and
**no file store and no first-class content API**. It anchors the *"attach by reading the
database directly"* hard case the adapter contract must account for (T13/T14).
## 1. Architecture
- **Stack:** Perl **Catalyst** (MVC web framework) + **DBIx::Class** (ORM) over a relational
DB (SQLite / PostgreSQL / MySQL). Templating via Template Toolkit.
- **Content:** **Markdown** (Text::MultiMarkdown) is the page markup — so the *body* is
Markdown, but it lives **in a DB column**, not a file.
- **Pages are hierarchical:** a **path tree** (`/parent/child`) modeled as rows with
parent/lineage relations — structure is relational, not directory-based.
- **Versioning:** each page edit creates a **new version row** (a `page_version`-style
table) — full revision history lives in **DB version tables**, with author/timestamp.
- **Features:** inline **AJAX editing**, **attachments** (stored as DB rows / blobs +
metadata), diffs, RSS feeds, full-text search, per-page permissions.
## 2. The attach problem — DB or nothing
MojoMojo exposes its content through the **web app** (HTML) and the **database**; there is
**no clean REST/GraphQL content API** and **no file store**. So a shard adapter has two
realistic paths:
1. **Direct relational read** (preferred): read the `page` + `page_version` (+ `content`,
`attachment`) tables via DBIx::Class schema — pages, the path tree, and **full history**
are all there, importable to the coordination journal (UC-41-style history import, but
from **DB version rows** rather than RCS/git).
2. **HTML scrape** (fallback): parse rendered pages — lossy, last resort.
This makes MojoMojo the **direct-DB-read** binding archetype: the canonical store is a
relational schema, and the adapter's job is to map that schema to the wiki page model +
journal.
## 3. Capability profile
| Dimension (synthesis spectrum) | MojoMojo |
|--------------------------------|----------|
| Attachment mode | **direct DB read** (relational); HTML-scrape fallback; no file store, no API |
| Addressing granularity | page (row); path tree via lineage rows |
| Content identity | DB page id; path as human key |
| Identity vs placement | row id vs path lineage (separable) |
| Structure | **relational**: page rows + parent/lineage; attachments as rows |
| History | **DB version tables** (per-edit version rows, author/timestamp) |
| Merge model | app-level last-writer; DB transactions |
| Native query | SQL over the schema (not a wiki query language) |
| Translation | **Markdown body in a DB column** — minimal translation, but extraction needed |
| Write granularity | page (row) per save |
| Operational envelope | a Perl app + its DB; direct DB access needs credentials |
| Access grant | per-page permissions in DB; app auth |
| Content opacity | transparent if you can read the DB |
| Provenance | author/timestamp on version rows |
## 4. INTENT mapping
### Reinforcements
- **Backend-neutral page model**: the body is **Markdown** — once extracted from the DB
column it maps directly; the adapter's work is **schema→page-model**, not format
translation.
- **History portability** (T13): DB **version rows** are a third history-source shape beside
git commits and RCS files — importable to the journal as discrete revisions with
author/timestamp.
- **Graceful degradation**: even with only DB read (no API), MojoMojo is a usable
read/projection/backup shard; with DB write it could be write-through, but carefully
(app invariants).
### Divergences (boundaries / notes)
- **No file store, no API** ⇒ the **direct-DB-read** binding is a first-class attach mode the
contract must name (alongside file-store, in-engine host, external-API, CRDT, P2P) — or a
sub-mode of "external store" where the medium is **a relational schema** (T14). Reading a
third-party app's DB is **coupling to its schema** (versioned, may drift across MojoMojo
versions) — a stated risk (UC-43 backend-swap analogue at the schema level).
- **Writing by direct DB** risks violating app invariants (lineage, version counters,
search index) — default to **read/projection/overlay**; write-through only with the app's
cooperation.
### What to keep
1. **Direct-DB-read as a named binding** for DB-backed engines with no file/API (UC-81),
mapping a **relational schema → wiki page model + journal**.
2. **DB version rows as a history source** for the journal (T13), beside git and RCS.
3. **Schema-coupling caution** — treat the schema as a versioned interface that can drift
(relates UC-43).
## 5. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-81 | Attach a **DB-backed wiki with no file store / no API** (MojoMojo) by reading its **relational store directly** (page + version tables), mapping schema → page model and **importing DB-resident history** to the journal | **new** |
| — | DB attach vs file attach | enrich **UC-02** / **UC-40** |
| — | DB version-table history import | enrich **UC-36** |
| — | relational page rows / lineage as structure | enrich **UC-34** |
## 6. Architecture notes for SHARD-WP-0002
- **T14 (binding):** add **direct relational read** as a binding (or external-store sub-mode
whose medium is a SQL schema) for DB-backed engines lacking a file store or API; HTML
scrape is the lossy fallback. Schema is a **versioned coupling** (drift risk, UC-43).
- **T13 (history portability):** **DB version rows** = a history source alongside git commits
and RCS revisions — import as discrete journal entries (author/timestamp).
- **T11 (capability):** "has-file-store" / "has-API" are **absent** here; "has-readable-DB"
is the capability — a sparse profile relying on schema knowledge.
## 7. Open questions
1. Does shard-wiki sanction **direct third-party DB reads** as a binding, or restrict them
(schema coupling/drift) to a documented best-effort mode? How is schema drift across
MojoMojo versions handled (UC-43)?
2. Is **write-through by direct DB** ever allowed (risking app invariants), or are DB-backed
no-API engines read/projection/overlay/backup only?
## 8. Sources
- MojoMojo — github.com/mojomojo/mojomojo; metacpan MojoMojo (Catalyst app, DBIx::Class
schema: Page / PageVersion / Content / Attachment)
- Catalyst + DBIx::Class framework docs (architecture context)
- prior: `research/260613-twiki-deep-dive/` (file-store classic contrast, UC-40/41)
## 9. Traceability
New UC **UC-81** carries the marker **⊙** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-02, UC-40, UC-36, UC-34. Architecture cross-refs:
SHARD-WP-0002 T14 (direct-DB binding), T13 (DB version-row history), T11.

View File

@@ -0,0 +1,56 @@
# 260614 — Notion deep dive (closed block-database SaaS, attached only through its API)
Date: 2026-06-14
## What this is
A focused study of **Notion** — architecture and extension model — read through
shard-wiki's lens. Notion is the **closed, hosted, schema-rich** end of the spectrum and
the dive that stress-tests shard-wiki's hardest constraints (*graceful degradation*,
*no silent remote mutation*, *Markdown-first that must degrade*).
It contrasts sharply with its neighbours:
- vs **Roam** — both are block databases with per-block UUIDs, but Roam is a *client*
DataScript DB reached only via an *in-app* JS API, while Notion is a *server* Postgres
store reached via a real *external* REST API (no in-engine hosting needed, but
rate-limited, eventually consistent, scoped by per-page grants).
- vs **Obsidian** — the mirror image of "file over app": Notion is **app over file**, a
closed hosted store with no portable files.
- vs **XWiki** — both are structured wiki-as-app-platforms; Notion is the **apex of
database-as-pages** (schema + relations + rollups + views) but closed and REST-only.
Distinctive material:
- **Data model** — everything is a **block** (UUID id, type, properties, ordered child
`content`, single `parent`); pages are blocks, **database rows are pages with a
schema**; Postgres-backed hosted SaaS
- **Databases** — typed properties, **relations** (bidirectional), **rollups**,
**formulas**, and many **views** (table/board/calendar/gallery) of one row-set
- **Extension model** — **no in-app plugin runtime**; extension = **external REST API**
integrations (+ webhooks since 2026), with ~3 rps rate limit, eventual consistency,
recursive child fetch, payload caps, and **scoped/revocable per-page access grants**
## Contents
| Path | Role |
|------|------|
| `findings.md` | Block model, databases, page-model impedance, the API-only extension model, capability profile, scoped consent, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
(UC-57 attach a closed external-API-only hosted shard with rate-limit/eventual-
consistency/scoped-grant constraints, UC-58 attach a typed database with schema +
relations + views without flattening, UC-59 lossy-aware translation with a fidelity
report); UC-31/34/36/39/50/51/52/54/56 enriched. Logged for `SHARD-WP-0002` (T14):
an **operational-envelope** capability section, **access-grant semantics**, a
**translation-fidelity** capability, and a three-way **attachment-mode taxonomy**
(file-store / in-engine-hosted / external-API-only).
**Key takeaways recorded:** Notion *enforces* "no silent remote mutation" via scoped,
revocable per-integration grants (ties the authz-in-core decision); it is the heaviest
**lossy-translation** case (proprietary blocks/rich-text, lossy Markdown export); and it
adds the **external-API-only** attachment mode (full write-through without in-engine
hosting, but bounded by rate limits and eventual consistency). **Boundary:** one
external-API candidate shard — best as projection/mirror/overlay/backup — not a
substrate and not the federation layer.
</content>

View File

@@ -0,0 +1,302 @@
# Findings — Notion: a closed block-database SaaS, attached only through its API
Date: 2026-06-14
Source kind: **modern shipped product** — a hosted block-database SaaS; a *candidate
shard* of a distinct family (closed, external-REST-API-only, schema-rich, heaviest
translation cost); has **no in-app plugin runtime**
Lens: shard-wiki — the external-API attachment mode, database-as-pages, lossy
translation/fidelity, scoped consent, and graceful degradation against a sovereign
closed backend
> Where Notion sits in the set. Roam and Notion are both **block databases with
> per-block UUIDs**, but they are opposite on access: Roam is a *client-side*
> DataScript DB reachable only through an *in-app* JS API (write-through needs an
> adapter hosted inside Roam); **Notion is a *server-side* Postgres store reachable
> through a real *external* REST API** — attachable from outside with no in-engine
> hosting, but rate-limited, eventually consistent, and **scoped by explicit
> per-integration page grants**. And against Obsidian's "file over app," Notion is the
> mirror image: **app over file** — a closed hosted store with no portable files at
> all. So Notion is the dive that stress-tests shard-wiki's hardest constraints:
> *graceful degradation*, *no silent remote mutation*, *union without erasure*, and
> *Markdown-first that must degrade* against a proprietary, schema-rich, non-Markdown
> backend.
Pairs with — and contrasts against — the Roam dive (block-DB, in-app vs external API),
the Obsidian dive (closed-hosted vs file-over-app), and the XWiki dive (both are
structured wiki-as-app-platforms; Notion is the apex of database-as-pages but closed
and REST-only).
---
## 1. Core data model — everything is a block
Notion's data model is uniform: **"Everything you see in Notion is a block. Text,
images, lists, a row in a database, even pages themselves — these are all blocks."**
Each block record holds:
| Field | Meaning |
|-------|---------|
| **id** | a randomly generated **UUID v4**, visible in the page URL — the block's stable address |
| **type** | determines rendering + which properties apply (text, to-do, heading, **page**, **child_database**, …) |
| **properties** | type-specific attributes (e.g. `title` text; for database rows, the typed property values) |
| **content** | an **ordered array of child block IDs** — nesting |
| **parent** | a single upward pointer — used for **permission inheritance** |
Blocks form a **render tree** (content + parent pointers); indentation is *structural*,
not cosmetic — it changes block relationships. **Pages are blocks**; **databases are
blocks whose children are pages**; a database **row is a page** with typed properties.
Stored in **Postgres** on Notion's servers (hosted SaaS — not local, not files).
The shard-wiki-relevant facts: (a) **per-block UUIDs** give native sub-page addressing
(UC-51); (b) the **page = block, database-row = page-with-schema** identity is a real
page-model impedance (§3); (c) the store is **closed and server-side** — reachable only
through the API (§4).
---
## 2. Databases — schema, typed properties, relations, views
A Notion **database** is a collection of pages sharing a **schema** of typed
**properties**: `title`, `rich_text`, `select` / `multi_select`, `number`, `date`,
`checkbox`, `person`, `files`, **`relation`** (a typed link to rows in another
database — shown on *both* sides, i.e. bidirectional), **`rollup`** (an aggregate over a
relation), and **`formula`** (computed). The same row-set is shown through multiple
**views****table, board (kanban), calendar, gallery, list, timeline** — each a
filtered/sorted/grouped projection of the same data.
For shard-wiki this is the **apex of "wiki page as structured record"** (stronger than
XWiki XObjects, UC-39): a page can be *bodiless typed data*, pages are joined by
**typed inter-database relations** (a graph of typed links), and "the page" is routinely
**a row in a schema** rather than prose. Notion databases also ship the ZigZag insight
commercially: **one row-set, many views/dimensions** (UC-47/48) and **filtered/linked
databases = query-defined pages** (UC-54).
---
## 3. Page-model impedance
Mapping Notion to a Markdown-first page model is the **heaviest translation case** in
the research set:
- **Block/rich-text ≠ Markdown.** Notion's rich text is an annotated-span model; many
block types (synced blocks, columns, callouts, embeds, database views) have no clean
Markdown equivalent. Notion's own **export to Markdown/CSV is lossy** (databases →
CSV, relations/rollups/formulas flatten or drop).
- **Database-row-as-page + schema** must map onto pages + sidecar metadata without
discarding the schema or the relations (extends UC-34/UC-39).
- Therefore translation must be **lossy-aware with a fidelity report** — surface what
did *not* round-trip rather than silently flattening (UC-59). This is *different from*
UC-42 (Foswiki TML↔HTML *lossless* round-trip); Notion is fundamentally lossy.
---
## 4. Extension model — there is no plugin runtime; only the API
This is the defining architectural fact for shard-wiki: **Notion has no third-party
in-app plugin/extension system.** There is no marketplace of code that runs inside
Notion (the only "in-app modding" is the *unofficial* Notion Enhancer desktop patcher,
out of scope). **Extension = external integration via the public REST API** (plus
embeds and, recently, webhooks).
The **public REST API** (`api.notion.com/v1`):
- **Resources:** `pages`, `blocks` (retrieve / append / update / delete children),
`databases` (retrieve, **query** with filters+sorts, create), `users`, `search`,
`comments`.
- **Authorization:** an **internal integration token** (workspace-owned) or **OAuth
2.0** (public integrations). Critically, **an integration only sees pages a user has
explicitly connected to it** — the user "approves the app and connects specific
pages" via *Add connections*. Access is a **scoped, revocable, per-page grant** (§6).
- **Operational constraints the adapter must encode:**
- **Rate limit** ~**3 requests/second** average (429 `rate_limited`); **no paid
increase**.
- **Eventual consistency / no live read model** — the API is not real-time; reads can
lag.
- **Recursive retrieval** — a block returns only **first-level children**; full-page
reconstruction requires walking `has_children` recursively.
- **Payload caps** — 1000 blocks / 500 KB per request; child arrays ≤ 100.
- **Webhooks** (added 2026) deliver page/database change events — a push transport for
UC-31, replacing pure polling.
- **Version history:** Notion keeps **internal page history** (retention bounded by
plan); it is **not portable** and not exposed as git — a UC-36 *supplementation* case
(like Confluence/MediaWiki), not an import case.
- **Publish:** pages can be **published to the web** as read-only public pages — an
outbound publish surface (UC-56).
Consequence: Notion is attachable **only** as an **external-API shard** — but, unlike
Roam, **no in-engine adapter hosting is needed** (the REST API is external). The cost is
the operational envelope (rate-limit, eventual consistency, scoped grant) that the
capability profile must model (UC-57).
---
## 5. Notion as a shard — capability profile
| Capability | Notion | Notes for the adapter contract |
|------------|--------|--------------------------------|
| Read | **yes (external REST)** | rate-limited (~3 rps), eventually consistent, recursive child fetch |
| Write | **yes (external REST)** | append/update/delete blocks, create pages — no in-engine host needed (vs Roam) |
| Write granularity | **block-level (fine)** | like Roam; per-block ops |
| Identity / addressing | **block UUID v4** | native sub-page addressing (UC-51), store-minted (like Roam, not in-file) |
| Structured data | **yes (apex)** | databases: schema + typed properties + relations + rollups + formulas (UC-34/39/58) |
| Native query | **yes** | database query API (filters/sorts) → delegate views (UC-52) |
| Views / dimensions | **yes** | table/board/calendar/gallery = many views of one row-set (UC-47/48/54) |
| Subscribe | **webhooks (2026)** | push events; else poll (UC-31) |
| Version history | **internal, not portable** | supplement via coordination journal (UC-36) |
| Diff / merge | **no native** | — |
| Lock | **no** | — |
| Publish | **publish-to-web** | outbound read-only (UC-56) |
| Access model | **scoped per-page grant (OAuth/token)** | explicit consent; revocable (UC-57, authz) |
| Syntax / content | **proprietary block + rich text** | **lossy** to Markdown; needs fidelity-aware translation (UC-59) |
Verdict: Notion is a legitimate but **demanding** shard — **external-API-attached,
schema-rich, fine-grained, closed**. It behaves best as a **projected / mirrored /
overlay / backup** participant; full write-through is possible but bounded by rate limits
and eventual consistency. The strongest reasons to attach it: structured databases
(UC-58) and block-UUID addressing (UC-51); the strongest cautions: lossy translation
(UC-59) and no portable history (UC-36).
---
## 6. Scoped consent and "no silent remote mutation"
Notion *enforces* one of shard-wiki's INTENT constraints at the platform level: an
integration can touch **only** the pages a user has explicitly **connected** to it, and
the grant is **revocable**. This is a clean, real-world model of **no silent remote
mutation** and of a shard granting the orchestrator **scoped, consented access** — and
it ties directly to the settled **authz-in-core / authn-delegated** decision
([[shard-wiki-auth-in-core-decision]]): authentication to Notion is delegated (OAuth /
integration token), while shard-wiki's *own* authorization decides what to do with the
granted scope. The adapter contract should treat **"scoped, revocable grant"** as a
first-class attachment property (UC-57), not an afterthought.
---
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
- **Graceful degradation** has its sharpest test here: a closed SaaS with no files, rate
limits, eventual consistency, and lossy export must still be usable as
read/projection/overlay/backup. If the adapter contract handles Notion, it handles
most things.
- **No silent remote mutation** is *modeled by the platform* (scoped grants) — Notion
validates the principle (§6).
- **Database-as-pages** validates that "wiki page" must stretch to **typed records with
relations**, not just prose (UC-34/39/58).
- **Block UUIDs** reconfirm (with Roam) that native sub-page addressing is real and
adoptable (UC-51).
### 7.2 Deliberate divergences (design bugs if conflated)
1. **Closed hosted store; no sovereignty over bytes.** shard-wiki cannot make Notion
git-native or local. It can mirror/project/overlay/back-up and **supply** a
git-addressable history (UC-36) — never claim to own Notion's store.
2. **Lossy, proprietary content.** Do **not** pretend Notion round-trips to Markdown.
Translate lossily *with a fidelity report* and preserve non-mappable elements as
provenance/sidecar (UC-59) — union without erasure includes erasure *of fidelity*
being made visible.
3. **External-API-only, rate-limited, eventually consistent.** Projection must be
**cache/poll/webhook**, not a live read model; sync is bounded — encode this in the
capability profile (UC-57). Do not design flows that assume cheap, instant, unlimited
reads.
4. **One workspace (or its granted page set) = one shard**, never the federation layer.
### 7.3 What Notion teaches that shard-wiki should keep
- Model **operational envelope** (rate limit, consistency class, payload caps,
pagination) as explicit capability-profile fields — Notion makes them unignorable.
- Model **scoped, revocable consent** as a first-class attachment property (UC-57, §6).
- Treat **translation fidelity as data**: a per-shard, per-page report of what projects
cleanly vs. degrades (UC-59) — applies beyond Notion.
- Recognize **external-REST attach** as a distinct, *preferred-where-available* mode:
full write-through without in-engine hosting (contrast Roam) — but pay the operational
envelope.
---
## 8. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-56**. New UCs **UC-57UC-59** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a closed hosted shard via its external REST API only** — no file store, no in-app runtime — honoring rate limits, eventual consistency, payload caps, and a scoped/revocable access grant | **UC-57 (new)** |
| **Attach a typed database (schema + relations + rollups + multiple views) as a shard** without flattening the schema or the inter-record relations | **UC-58 (new)** |
| **Translate a proprietary block/rich-text model to/from Markdown with an explicit fidelity report**, preserving non-mappable elements rather than silently dropping them | **UC-59 (new)** |
| Block UUIDs = store-minted native span addresses (external-API variant) | **enriches UC-51** |
| External-API block-DB attach (no in-engine host) — contrast Roam's in-app-only | **enriches UC-50** |
| Database query API + filtered/linked DBs | **enriches UC-52, UC-54** |
| Database-as-pages apex; typed records + relations | **enriches UC-34, UC-39** |
| Webhooks (2026) as a push transport | **enriches UC-31** |
| Internal-only page history, not portable | **enriches UC-36** |
| Publish-to-web outbound | **enriches UC-56** |
| Scoped, revocable per-integration grant; no silent mutation | links **UC-57** + [[shard-wiki-auth-in-core-decision]] |
---
## 9. Architecture notes for SHARD-WP-0002 (no UC)
- Add an **operational-envelope** section to the adapter capability profile: rate limit,
consistency class (live / eventually-consistent / snapshot), payload/pagination caps,
recursive-fetch requirement, push-vs-poll transport. Notion is the forcing example.
- Add **access-grant semantics**: scope (which pages), revocability, auth mode
(delegated token/OAuth) — ties the authz-in-core decision and "no silent mutation".
- Add a **translation-fidelity capability**: adapters declare and report what content
round-trips vs. degrades (UC-59); generalizes UC-42 (lossless) to the lossy case.
- **Attachment-mode taxonomy** now spans: file-store direct (Obsidian/TWiki, UC-40),
in-engine hosted adapter (Roam/XWiki, UC-38/50), and **external-API-only**
(Notion, UC-57). T14 binding should enumerate all three.
- **Database/schema/relations** as a unit (UC-58) presses the page-model spec: collection
+ schema + typed relations, not just a page.
---
## 10. Open questions (for spec / workplans)
1. Is **external-API-only with a tight rate limit** (Notion) viable for write-through at
wiki scale, or do we cap Notion at read/projection/overlay/backup by default?
2. How are **inter-database relations** (UC-58) represented in the union — as typed links
in the link graph, as a separate relation index (cf. ZigZag many-to-many), or both?
3. What is the **fidelity report** format (UC-59), and where does it surface — provenance
panel, projection metadata, reconciliation review?
4. For **scoped grants** (§6), how does shard-wiki represent partial visibility (only
*some* of a workspace's pages granted) without misrepresenting the shard as complete?
5. Do we consume Notion **webhooks** (push) or poll, given eventual consistency and the
rate limit (UC-31)?
---
## 11. Sources
| Source | Used for |
|--------|----------|
| Notion — "The data model behind Notion's flexibility" (https://www.notion.com/blog/data-model-behind-notion) | Everything-is-a-block; block record (id/type/properties/content/parent); render tree; pages/databases as blocks; Postgres |
| Notion Docs — Request limits (https://developers.notion.com/reference/request-limits) | ~3 rps rate limit, 429, payload caps, recursive first-level children |
| Notion Docs — Authorization (https://developers.notion.com/docs/authorization) | Internal token vs OAuth; integration connected to specific pages; scoped grant |
| Hookdeck / ClickUp — Notion webhooks guides (https://hookdeck.com/webhooks/platforms/guide-to-notion-webhooks-features-and-best-practices) | Webhook support (2026), page/database change events |
| Truto / Rollout — Notion API architecture & essentials (https://truto.one/blog/how-to-integrate-with-the-notion-api-architecture-guide-for-b2b-saas/) | REST endpoints (pages/blocks/databases/search), integration patterns, no in-app plugin model |
| General API knowledge — database property types, views, relations/rollups/formulas, export-to-Markdown lossiness | §2, §3 |
Cross-references: `research/260614-roam-deep-dive/findings.md` (block-DB/UUID, in-app vs
external API), `research/260614-obsidian-deep-dive/findings.md` (closed-hosted vs
file-over-app), `research/260613-xwiki-deep-dive/findings.md` (structured wiki-app-
platform), `spec/UseCaseCatalog.md` (UC-31, UC-34, UC-36, UC-39, UC-50/51/52, UC-54,
UC-56), `workplans/SHARD-WP-0002-federation-architecture.md` (T14), and the authz
decision [[shard-wiki-auth-in-core-decision]].
---
## 12. Traceability
- New UCs: **UC-57, UC-58, UC-59**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-31, UC-34, UC-36, UC-39, UC-50, UC-51, UC-52, UC-54, UC-56**.
- Architecture (no UC): operational-envelope + access-grant + translation-fidelity
capability fields; three-way attachment-mode taxonomy; database/schema/relations in the
page model → `SHARD-WP-0002` (T14).
- Decision link: scoped/revocable grant + no-silent-mutation → [[shard-wiki-auth-in-core-decision]].
- Boundary recorded: Notion is **one external-API candidate shard** — closed, hosted,
schema-rich, lossy-to-Markdown — best as projection/mirror/overlay/backup; not a
substrate, not the federation layer (INTENT graceful-degradation + no-silent-mutation).
</content>

View File

@@ -0,0 +1,51 @@
# 260614 — Obsidian deep dive (file-over-app vaults, plugin API, ecosystem signal)
Date: 2026-06-14
## What this is
A focused study of **Obsidian** — core/extension architecture and the plugin
ecosystem — read through shard-wiki's lens. Obsidian is the **most INTENT-aligned tool
in the research set**: INTENT names Obsidian vaults as a shard participant, and its
**"file over app"** model (notes are plain Markdown files in a folder you own; the app
is a lens) is the closest cousin to shard-wiki's Markdown-first, sovereignty-preserving
thesis. It is the **file-backed counterpart to the Roam dive** (same tool category,
opposite storage model: Roam = client DB + API; Obsidian = files on disk).
Distinctive material:
- **Architecture** — vault = folder of `.md` + `.obsidian/` config; files canonical;
**MetadataCache** is a *derived*, async-rebuilt index (links, embeds, tags,
frontmatter, block IDs, resolved/unresolved links, backlinks)
- **In-file, git-diffable** addressing/structure — `[[wikilinks]]`, `![[embeds]]`
(transclusion), `^block-id` span refs, YAML frontmatter/properties
- **Plugin API** — `Plugin` `onload/onunload`; `App` modules `vault` (CRUD + file
events), `metadataCache`, `workspace` (CM6, views), `fileManager.renameFile`;
`manifest.json`; community plugins, Restricted Mode
- **Ecosystem popularity → UC signal** (§7) — ranked download data mined for use cases,
per the research brief
## Contents
| Path | Role |
|------|------|
| `findings.md` | File-over-app architecture, in-file addressing, plugin API as adapter host, capability profile, **plugin-popularity → UC mapping**, derived-index lesson, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Four new use cases promoted to `spec/UseCaseCatalog.md`
(UC-53 attach a local vault with a live concurrent native editor, UC-54 query-defined
dynamic page, UC-55 carry non-Markdown content types, UC-56 outbound publish of a
projection); UC-15/28/34/36/40/51/52 enriched. Logged for `SHARD-WP-0002` (T14):
**dual attachment mode** (file-store *or* in-app plugin host), consume-native-derived-
index capability, non-Markdown content types in the page model, outbound publish,
external-writer tolerance.
**Key takeaways recorded:** (1) Obsidian validates *files-canonical / index-derived*
shard-wiki's projection model. (2) Fine-grained addressing & structure can be portable
**in-file text** (`^block-id`, frontmatter), not DB state (contrast Roam). (3) The
ecosystem is demand evidence — the #1 plugin is **drawings** (non-Markdown content),
query-as-DB (Dataview/Tasks) is top-tier but an *add-on*, and **Git is top-7** (users
bolt on the portable history shard-wiki provides natively). **Boundary:** a vault is one
file-backed candidate shard, not the federation layer and not a file-sync target
(not-a-sync-daemon).
</content>

View File

@@ -0,0 +1,324 @@
# Findings — Obsidian: file-over-app vaults, the plugin API, and what the ecosystem reveals
Date: 2026-06-14
Source kind: **modern shipped product** — a local-first Markdown vault tool; the
*most INTENT-aligned candidate shard yet* (file-backed, Markdown-first); plus an
ecosystem whose popularity is direct evidence of what users actually do
Lens: shard-wiki — file-store attachment, in-file fine-grained addressing, derived
index vs. canonical files, the engine-hosts-adapter path, and ecosystem-driven UCs
> Why Obsidian matters most. INTENT names **Obsidian vaults** explicitly as a shard
> participant, and Obsidian's **"file over app"** philosophy — your notes are plain
> Markdown files in a folder you own, the app is just a lens — is the closest cousin
> to shard-wiki's own Markdown-first, sovereignty-preserving thesis. It is also the
> **file-backed counterpart to the Roam dive**: same "personal knowledge tool"
> category, opposite storage model (Roam = client DB + API; Obsidian = files on disk).
> And uniquely among the dives, Obsidian has a **large, ranked plugin ecosystem** — so
> the user's instruction to let *plugin popularity inform the use-case catalog* is
> actionable: what people install tells us what a federated wiki must support (§7).
This dive treats Obsidian as a candidate shard (capability profile §5), studies its
plugin API as an adapter-host surface (§4), and mines the download rankings for UCs
(§7). Pairs with the Roam dive (the DB/API contrast) and the TWiki/Foswiki dives (the
file-store attachment path, UC-40).
---
## 1. Core architecture — file over app
- A **vault** is a local folder of plain **Markdown files** plus a `.obsidian/` config
directory (settings, enabled plugins, themes, workspace layout). **The files are the
source of truth**; the app holds no separate canonical database.
- Markdown is **Obsidian-flavored**: CommonMark + `[[wikilinks]]`, `![[embeds]]`
(transclusion), `^block-id` block references, `#tags`, YAML **frontmatter /
properties** (typed metadata), and callouts. Note: all addressing and structure live
**in the file text** — so they are git-diffable and portable.
- **MetadataCache** is a *derived* index: Obsidian pre-parses every file into cached
metadata (headings, links, embeds, tags, frontmatter, **blocks/block IDs**,
`resolvedLinks` / `unresolvedLinks`, backlinks). It is **rebuilt from files** and
updated **asynchronously** ("don't rely on immediate updates after a modify"). Files
canonical; index derived.
- **Canvas** files (`.canvas`, the open **JSON Canvas** format) hold spatial
arrangements — a non-Markdown content type living in the vault.
- Sync is *not* built into the format: users add Git, Obsidian Sync, or a file-sync
plugin (see §7). The vault is just files.
For shard-wiki this is the **ideal file-backed shard**: Markdown-first, on disk,
git-friendly, with a derived index model that mirrors shard-wiki's own
projection/derived-view philosophy (the cache is to Obsidian what projections and union
BackLinks are to shard-wiki — computed, not canonical).
---
## 2. Addressing, links, transclusion — in-file and git-diffable
- **Links:** `[[Page]]` and `[[Page#Heading]]` / `[[Page#^block-id]]`; the resolved/
unresolved link sets and **backlinks** are computed in MetadataCache — Obsidian's
shipped answer to union BackLinks (UC-05/UC-18), but per-vault.
- **Fine-grained addressing:** a **`^block-id`** suffix on a paragraph (and heading
anchors) gives a stable sub-page address — *but it lives in the Markdown text*, not in
a database. This is the **git-diffable, portable variant** of Roam's `:block/uid`
(which lives in a DataScript DB). Important contrast for UC-51: native span IDs can be
**text-embedded** (Obsidian) or **store-minted** (Roam); the text-embedded form is
more portable and survives a file copy, but is opt-in (the user must add the `^id`).
- **Transclusion:** `![[Page]]`, `![[Page#Heading]]`, `![[Page#^block-id]]` embed live
content by reference — shipped transclusion at page/section/block granularity (UC-32),
again stored as plain text.
---
## 3. Structured data — frontmatter as git-diffable records
YAML **frontmatter / properties** make each note a lightweight typed record
(`tags`, `aliases`, arbitrary keys). This is the **git-diffable structured-data**
variant (cf. TWiki `%META%` inside the file, UC-34) — structure lives in the file, not a
DB. The **Dataview** plugin (§7) turns frontmatter + inline fields + the link graph into
a queryable database — but that query power is an *ecosystem add-on*, not core.
---
## 4. Extension architecture — the plugin API as an adapter host
Obsidian plugins are TypeScript, shipped as `manifest.json` + `main.js` (+ optional
`styles.css`):
- **Manifest:** `id`, `name`, `version`, `minAppVersion` (required); `author`,
`description`, `isDesktopOnly`, `fundingUrl` (optional). Community plugins are
GitHub-hosted and listed in the `obsidian-releases` repo; installed from inside the
app; gated by **Restricted Mode** (plugins run arbitrary code — Node/Electron on
desktop).
- **Plugin lifecycle:** extend `Plugin`, implement **`onload()` / `onunload()`**; use
`registerEvent` / `registerDomEvent` / `registerInterval` for auto-cleanup.
- **The `App` object** (global singleton, `this.app`) exposes four modules:
- **`vault`** — file CRUD: `read` / `cachedRead`, `create`, `modify`, `process`,
`delete`, `rename`, `getFiles`, low-level `adapter`, and **events**
(`on('create'|'modify'|'delete'|'rename')`) for live file-watching.
- **`metadataCache`** — `getFileCache`, `frontmatter`, `links`, `embeds`, `tags`,
`headings`, `blocks`, `resolvedLinks` / `unresolvedLinks`, `getBacklinksForFile`.
- **`workspace`** — panes/leaves/views; `registerView`, `registerEditorExtension`
(CodeMirror 6), `registerMarkdownPostProcessor`.
- **`fileManager`** — high-level ops; **`renameFile`** updates inbound links (use it
instead of raw vault rename to keep link integrity).
- **Registrations:** `addCommand`, `addRibbonIcon`, `addStatusBarItem`,
`addSettingTab`, `registerView`, `registerEditorExtension`,
`registerMarkdownPostProcessor`.
Two consequences for shard-wiki:
1. **A vault is attachable two ways.** (a) **File-store direct attach** (UC-40) — read
the folder as a folder shard; no plugin, offline-capable, git-native; good for
read/projection/overlay. (b) **In-app plugin host** (UC-38) — a shard-wiki adapter
*as an Obsidian plugin* drives `vault` write + `metadataCache` reads + live file
events for high-fidelity write-through. This dual mode is a cleaner story than Roam
(which only offers the in-app path).
2. The **MetadataCache API is a ready-made derived-view source** — backlinks, tags,
block IDs, resolved links — an adapter can consume instead of reparsing.
---
## 5. Obsidian as a shard — capability profile
| Capability | Obsidian | Notes for the adapter contract |
|------------|----------|--------------------------------|
| Read | **yes** | direct file read (folder shard) or `vault.read`/`cachedRead` in-app |
| Write | **yes** | direct file write, or `vault.modify`/`create`/`process` in-app; per-file granularity |
| Write granularity | **per-file (page)** | the natural shard-wiki granularity — contrast Roam (block) and TiddlyWiki (whole-file-single-vault) (UC-35) |
| Identity / addressing | **path + in-file `^block-id` / headings** | git-diffable, portable, **opt-in** span IDs (UC-51 text-embedded variant) |
| Transclusion | **yes** | `![[...]]` page/section/block embeds (UC-32) |
| Backlinks / links | **yes (derived)** | MetadataCache resolved/unresolved links + backlinks (UC-05/18) |
| Structured data | **yes (in-file)** | YAML frontmatter/properties; queryable via Dataview plugin (UC-34) |
| Native query | **plugin only** | Dataview — *not core*; informs UC-52 (query is adapter/plugin-provided, not assumable) |
| Version history | **none native** | users add the Git plugin (top-7!) → validates the coordination journal (UC-36) |
| Diff / merge | **none native** | git-level if Git plugin / repo-backed |
| Lock | **no** | local-first, single-user assumption |
| Publish | **via Obsidian Publish / Quartz / Digital Garden** | outbound publish of a projection (UC-56) |
| Syntax | **Obsidian-flavored Markdown** | close to CommonMark; wikilinks/embeds/callouts/`^id` need adapter awareness (cf. UC-42) |
| Non-Markdown content | **Canvas (JSON Canvas), attachments, Excalidraw** | typed/opaque assets in the vault (UC-55) |
Verdict: Obsidian is the **cleanest file-backed, Markdown-first shard** — the
file-store family (TWiki/Foswiki/DokuWiki, UC-40), and the *reference* personal-vault
shard INTENT already names. Its history/query/sync gaps are exactly what users fill with
plugins (§7) — i.e. exactly what shard-wiki offers as orchestration.
---
## 6. The derived-index lesson (architecture)
Obsidian's **files-canonical / MetadataCache-derived** split is independent validation
of shard-wiki's core stance: the link graph, backlinks, tags, and block index are
**computed projections over canonical files**, rebuilt on change, eventually consistent
("updated asynchronously"). shard-wiki's union BackLinks, projections, and derived views
(UC-05, UC-17UC-20) should likewise be **derived and rebuildable**, never a second
source of truth. The cache-invalidation/async-update caveat is a real design note for
projection freshness (UC-31).
---
## 7. Ecosystem popularity → use-case signal (the user's ask)
All-time download ranks (obsidianstats.com), read as *demand evidence*:
| Plugin (downloads) | What users do with a vault | shard-wiki signal → UC |
|--------------------|----------------------------|------------------------|
| **Excalidraw** (6.4M) | drawings/diagrams as first-class content | non-Markdown content types in a shard → **UC-55 (new)** |
| **Templater** (4.6M), **QuickAdd** (1.9M) | templated note creation | blueprint pages → **UC-15 (enriched)** |
| **Dataview** (4.4M) | query the vault (files+frontmatter+links) as a DB | query-defined dynamic pages → **UC-54 (new)**; query is a *plugin***UC-52 (enriched)** |
| **Tasks** (3.6M) | aggregate to-dos across all notes | cross-page typed-item aggregation → **UC-54** |
| **Advanced Tables** (2.9M) | structured tables in Markdown | in-file structured data → UC-34 |
| **Calendar** (2.8M) | daily-note navigation | temporal dimension (ZigZag `d.recent`) → UC-17 |
| **Git** (2.7M, top-7) | version control + sync on the vault | **history is a bolt-on, not native** → validates coordination journal → **UC-36 (enriched)** |
| **Kanban** (2.4M) | board view of Markdown cards | alternate projection/view of pages → UC-47/48 |
| **Remotely Save** (2.0M) | sync vault to S3 / Dropbox / OneDrive / GDrive / **WebDAV** | demand to federate a vault to heterogeneous remote stores → reinforces INTENT WebDAV/Nextcloud shards (but shard-wiki is *not* a file-sync daemon — attach as shards, don't mirror) |
| **Omnisearch** (1.6M) | search across notes, PDFs, images | union full-text search over mixed content → UC-20 |
| **Importer** (1.4M) | import from Evernote / Notion / Bear / Apple Notes | carry-forward/import from foreign tools → **UC-28 (enriched)** |
| **Smart Connections** (1.0M), **Copilot** (1.5M), **Claudian** (938K) | AI over the vault (related notes, chat) | mostly out of scope; "related notes" ≈ link/equivalence discovery (UC-46) — note the trend, don't build it into core |
| **Tag Wrangler** (981K) | rename/merge tags vault-wide | namespace/tag refactor across the union → UC-22 |
| **Linter** (927K) | normalize frontmatter/formatting | content normalization → fingerprint basis for equivalence (UC-46) |
| **Admonition** (922K) | callouts | Obsidian-flavored syntax → translation awareness (UC-42) |
Headline reads:
- The **#1 plugin is drawings** (Excalidraw) — users keep **non-Markdown content** in
"Markdown" vaults. A wiki orchestrator that assumes pure Markdown will mishandle the
most popular real-world use → **UC-55**.
- **Query-the-vault-as-a-DB** (Dataview #3 + Tasks #4) is a top use — but it is an
*add-on*, confirming query is an **adapter/plugin capability** (UC-52), and motivating
**query-defined dynamic pages** (UC-54).
- **Git is top-7** — users manually bolt portable version control onto their vault. This
is *direct demand* for what shard-wiki provides natively as the coordination journal
(UC-36).
- **Sync-to-anywhere** (Remotely Save #11) shows demand to connect a vault to S3/WebDAV/
cloud — the heterogeneous backends INTENT targets — while reminding us of the
*not-a-file-sync-daemon* boundary.
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **File-over-app == shard sovereignty + Markdown-first.** Obsidian is the living proof
that a serious tool can be "just files you own"; it is the model shard for INTENT's
Obsidian/local-folder participants.
- **Files canonical, index derived** validates shard-wiki's projection/derived-view
architecture (§6).
- **In-file, git-diffable structure & addressing** (frontmatter, `^block-id`, embeds)
shows fine-grained addressing and structured data can be **portable text**, not DB
state — friendlier to the coordination journal than Roam's DB.
### 8.2 Deliberate divergences (design bugs if conflated)
1. **A vault is one shard, not the federation.** Local-first, single-vault,
single-user; do not model the union as "one big vault."
2. **Not a file-sync daemon.** The Remotely Save popularity is tempting; shard-wiki must
stay **wiki-page-semantic** — attach remote stores as shards, never generic file
mirroring (INTENT constraint).
3. **`.obsidian/` is opaque app config, not page content.** The adapter must exclude/
treat it as shard-local config, not project it as pages.
4. **Obsidian-flavored Markdown ≠ CommonMark.** Wikilinks/embeds/callouts/`^id` need
adapter awareness; closer than Roam's outline, but still a translation surface
(UC-42).
5. **Plugins run arbitrary code.** If a shard-wiki adapter is hosted *as* an Obsidian
plugin (UC-38 path), it inherits Electron/Node trust — a deployment/security note,
not a core concern.
### 8.3 What Obsidian teaches that shard-wiki should keep
- Offer **both attachment modes** for a vault: zero-config **file-store direct attach**
(read/projection/overlay) and an optional **in-app adapter** (write-through, live
events). Graceful degradation by default, fidelity on opt-in.
- **Consume a backend's existing derived index** (MetadataCache) instead of reparsing,
when the shard exposes one.
- **Embrace non-Markdown content types** (Canvas/attachments/drawings) as typed or
opaque assets with provenance — don't flatten them away (UC-55, extends UC-34).
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-52**. New UCs **UC-53UC-56** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a local-first Markdown vault as a file-backed shard with a live concurrent native editor** — file-watch, tolerate the vault's own app writing concurrently, treat `.obsidian/` as opaque | **UC-53 (new)** |
| **Define a page as a live query over the union** (Dataview/Tasks pattern — query-defined dynamic page) | **UC-54 (new)** |
| **Carry non-Markdown content types** (Canvas/JSON Canvas, drawings, attachments) as typed/opaque assets with provenance | **UC-55 (new)** |
| **Publish a curated projection of the union or a shard to an external read-only target** (Obsidian Publish / Quartz / Digital Garden) | **UC-56 (new)** |
| Local Markdown vault = cleanest file-backed direct-attach; dual attach (file-store *or* in-app plugin host) | **enriches UC-40** (and UC-02, UC-38) |
| In-file `^block-id` / heading anchors = git-diffable, portable, opt-in span IDs | **enriches UC-51** |
| YAML frontmatter/properties = git-diffable in-file structured data | **enriches UC-34** |
| Git plugin (top-7) = users bolt on portable history → demand for the coordination journal | **enriches UC-36** |
| Query is a popular plugin (Dataview), not core → query is adapter/plugin-provided | **enriches UC-52** |
| Importer plugin = import from foreign tools | **enriches UC-28** |
| Templater/QuickAdd = templated creation | **enriches UC-15** |
| `![[...]]` embeds = in-file transclusion | links UC-32 |
| MetadataCache backlinks/links | links UC-05/UC-18 |
---
## 10. Architecture notes for SHARD-WP-0002 (no UC)
- **Dual attachment mode** for a single backend (file-store *and* in-engine adapter)
should be first-class in the adapter contract / T14 binding — Obsidian is the clean
example (Roam was in-app-only; TWiki had file-vs-API but not a personal vault).
- **"Consume native derived index"** as a capability: a shard may expose its own link/
backlink/block index (Obsidian MetadataCache, Roam Datalog) the orchestrator can read
instead of reparsing (ties UC-52).
- **Non-Markdown content types** (UC-55) push on the wiki page model: pages vs. typed
assets vs. opaque blobs — a page-model spec decision, not just adapter config.
- **Outbound publish** (UC-56) formalizes the `publish` capability from INTENT's list as
a projection target, complementing inbound static-export attach (UC-37).
- Concurrent-native-editor (UC-53) needs the contract to express **external-writer
tolerance** (file-watching, re-projection, conflict-with-live-app) — distinct from
multi-user write conflicts.
---
## 11. Open questions (for spec / workplans)
1. For a vault attached **both** ways (file-store + in-app plugin), which is
authoritative, and how do they reconcile (the in-app adapter sees MetadataCache; the
file-store path sees raw bytes)?
2. Is **UC-54 (query-defined dynamic page)** a core page type, an adapter feature, or a
reference-UI/plugin concern? (Mirrors the Roam/Dataview "views are queries" question
and catalog Q7.)
3. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a
Markdown stub, opaque blob with provenance, or a pluggable content-type registry?
4. Does **outbound publish** (UC-56) belong in core or as a publish-adapter family, and
how does it interact with overlays/projection freshness?
5. How is **concurrent native editing** (UC-53) surfaced — optimistic re-projection,
advisory lock, or overlay-onto-moving-target?
---
## 12. Sources
| Source | Used for |
|--------|----------|
| Obsidian API — App Architecture (https://www.mintlify.com/obsidianmd/obsidian-api/concepts/app-architecture) | App singleton, four modules, this.app |
| obsidianmd/obsidian-api README (https://github.com/obsidianmd/obsidian-api/blob/master/README.md) | Plugin lifecycle, Vault/MetadataCache/Workspace/FileManager APIs, registrations, manifest fields |
| DeepWiki — Vault and File System; MetadataCache and Link Resolution (https://deepwiki.com/obsidianmd/obsidian-api) | Vault CRUD + events; MetadataCache parsed elements, resolved/unresolved links, backlinks |
| obsidianstats.com — Most Downloaded Plugins (https://www.obsidianstats.com/most-downloaded) | All-time download ranks + counts for the popularity → UC mapping (§7) |
| Obsidian — "The future of plugins" (https://obsidian.md/blog/future-of-plugins/) | Plugin distribution model, restricted mode context |
| Obsidian Help / JSON Canvas (jsoncanvas.org) | Vault format, `.obsidian/` config, Canvas open format |
Cross-references: `research/260614-roam-deep-dive/findings.md` (DB/API contrast, native
span IDs, query delegation), `research/260613-twiki-deep-dive/findings.md` (file-store
attach UC-40), `spec/UseCaseCatalog.md` (UC-02, UC-05, UC-15, UC-18, UC-28, UC-31,
UC-32, UC-34, UC-36, UC-38, UC-40, UC-51, UC-52), `workplans/SHARD-WP-0002-federation-architecture.md` (T14).
---
## 13. Traceability
- New UCs: **UC-53, UC-54, UC-55, UC-56**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-15, UC-28, UC-34, UC-36, UC-40, UC-51, UC-52** (and links UC-02,
UC-05, UC-18, UC-32, UC-38).
- Architecture (no UC): dual attachment mode; consume-native-derived-index capability;
non-Markdown content types in the page model; outbound publish; external-writer
tolerance → `SHARD-WP-0002` (T14).
- Distinctive artifact: **plugin-popularity → UC mapping** (§7) — ecosystem demand
evidence, per the research brief.
- Boundary recorded: an Obsidian vault is **one file-backed candidate shard** (the
cleanest, INTENT-named), mapped into the Markdown-first page model; not the federation
layer and not a file-sync target (INTENT Stability Note + not-a-sync-daemon).
</content>

View File

@@ -0,0 +1,13 @@
# 260614 — Oddmuse deep dive
Deep dive on **Oddmuse**: a deliberately minimal wiki — a **single Perl CGI script** over
**plain-text page files** (one file per page, old revisions in a `keep/` dir), no database.
The low-complexity **file-store floor** — useful as the **minimal-adapter / graceful-
degradation baseline** of the capability profile.
- `findings.md` — the minimal model, storage layout, capability profile, INTENT mapping, UC
seed (UC-82), architecture notes for SHARD-WP-0002, open questions, sources, traceability.
Catalog yield: UC-82 (attach a **minimal flat-file wiki** — plain-text page files + a simple
revision dir — as the **graceful-degradation baseline / minimal capability-profile floor**).
Enriched UC-40/01/36/41. Feeds SHARD-WP-0002 T11 (minimal capability profile).

View File

@@ -0,0 +1,124 @@
# Oddmuse — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T7 · **Subject:** Oddmuse, Alex Schroeder's
minimal single-script wiki (EmacsWiki, Community Wiki).
## Why this dive
After the structure/graph/SaaS far-ends, Oddmuse anchors the **opposite** corner: the
*minimal* file-store wiki. It is the reference for shard-wiki's **graceful-degradation**
promise — *a limited backend must still be usable* — and defines the **floor** of the
capability profile (T11): what the simplest possible real wiki looks like as a shard.
## 1. The minimal model
- **One Perl CGI script** (`wiki.pl`) is the whole engine — drop it on any CGI host. No
framework, no database, minimal dependencies.
- **Plain-text page files**: each page is a text file in a **page directory** (`page/`),
with a small header of metadata and the body; **old revisions** are kept in a **`keep/`**
directory (recent history retained, older optionally expired).
- **Locking** via lock files; edits append a new keep-revision.
- **Markup:** simple wiki markup; **CamelCase** and **free links** `[[Like This]]`;
InterWiki/near-links; tags and "clusters."
- **No DB, no API** (beyond the CGI itself); content *is* the files on disk.
## 2. The shard view — the file-store floor
Because pages are **plain-text files on disk**, Oddmuse is **trivially attachable** as a
**file-store shard** even though the engine offers nothing fancy:
- Read the `page/` files → pages (parse the tiny header + body).
- Read `keep/` → recent revision history (import to the journal; note it may be **truncated**
— older revisions can be expired, so history is *partial*).
- Write = write a page file + a keep-revision (respecting the lock) — but the engine's own
invariants (indexes) mean **write-through is best done via the engine or carefully**.
This is the **minimal capability profile**: file-store, page-granularity, plain-text,
possibly-truncated history, no query, no structured fields, open editing. Everything richer
in the synthesis matrix is measured *against this floor*.
## 3. Capability profile (the floor)
| Dimension (synthesis spectrum) | Oddmuse |
|--------------------------------|---------|
| Attachment mode | **file-store** (plain-text files); CGI, no API |
| Addressing granularity | page = file |
| Content identity | page name = filename |
| Identity vs placement | name-bound |
| Structure | none beyond tags/clusters; flat page space |
| History | **`keep/` revisions — recent, possibly truncated** |
| Merge model | lock-file; last-writer |
| Native query | none |
| Translation | simple wiki markup (→ Markdown translation needed) |
| Write granularity | page (file) |
| Operational envelope | a CGI script; tiny |
| Access grant | open by default (optional password) |
| Content opacity | transparent text |
| Provenance | minimal (timestamp, optional username) |
## 4. INTENT mapping
### Reinforcements
- **Graceful degradation** (INTENT): Oddmuse is the *definition* of the limited-backend case
— still a perfectly good read/projection/overlay/backup shard via its files.
- **Union without erasure**: even a minimal shard contributes pages with provenance; its
**truncated history** must be surfaced honestly (don't imply full history when `keep/` is
partial).
- **Open wiki** (UC-01): Oddmuse is open-editing by default — the c2-era ethos.
- **Markdown-first but backend-neutral**: its wiki markup needs translation to the
Markdown-first page model (UC-42-style), a small lossy step.
### Divergences (boundaries / notes)
- **Partial history**: `keep/` may expire old revisions — the journal import must record that
history is **truncated/partial**, not complete (a freshness/provenance honesty point).
- **Minimal profile** means many capabilities are simply **absent** — the adapter advertises
a sparse profile; the orchestrator must not assume query/structure/locking semantics it
doesn't have (T11 capability-awareness in its purest form).
### What to keep
1. **Minimal flat-file wiki as the graceful-degradation baseline** (UC-82): plain-text files
+ simple revision dir = the floor every richer profile extends.
2. **Honest partial-history reporting** when a shard's revision store is truncated.
3. **Sparse capability profile** handling — absence of a capability is first-class (T11).
## 5. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-82 | Attach a **minimal flat-file wiki** (plain-text page files + a simple revision dir, Oddmuse) as the **graceful-degradation baseline / minimal capability-profile floor**, surfacing **partial history** honestly | **new** |
| — | plain-text file-store at the simple end | enrich **UC-40** |
| — | open-editing wiki | enrich **UC-01** |
| — | `keep/` plain-text revision history (possibly truncated) | enrich **UC-36** / **UC-41** |
## 6. Architecture notes for SHARD-WP-0002
- **T11 (capability model):** Oddmuse defines the **minimal/floor profile** — file-store,
page granularity, plain-text, **partial** history, no query/structure. Validate that the
capability vocabulary can express **absence** cleanly and that the orchestrator degrades
to read/projection/overlay/backup against it.
- **History portability (T13):** `keep/` revisions import as journal entries but may be
**truncated** — record completeness as metadata (full vs partial history).
## 7. Open questions
1. How does shard-wiki represent a shard with **partial/truncated history** in the journal
and provenance UI (UC-24) — explicit "history begins at" marker?
2. Is write-through to a minimal CGI wiki (write page + keep-revision under its lock) ever
sanctioned, or read/projection/overlay/backup only by default?
## 8. Sources
- oddmuse.org — Oddmuse wiki (single-script install, `page/` + `keep/` storage, markup,
CamelCase/free links, clusters/tags)
- EmacsWiki / Community Wiki (Oddmuse in production)
- prior: `research/260608-c2-wiki-origins/` (open-wiki ethos); `research/260613-twiki-deep-dive/`
(file-store + RCS contrast)
## 9. Traceability
New UC **UC-82** carries the marker **⊚** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-40, UC-01, UC-36, UC-41. Architecture cross-refs:
SHARD-WP-0002 T11 (minimal/floor profile), T13 (partial-history import).

View File

@@ -0,0 +1,30 @@
# 260614 — Processing & Processing.js deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T4**
## What this is
A deep dive into **Processing** (creative coding) and **Processing.js / p5.js** (browser-run
sketches): **the sketch *is* the document** — a program whose presentation is **live visual
output rendered at view time** in the browser, with **no cached output** by default.
## Why it matters
- The cleanest **program-as-page** case: canonical content = **source text**, presentation =
**executable render** (no input/output cells, no prose envelope) — sharpens the page model
(T12/T15) and UC-54/55.
- Adds a **view-time** variant to derivation-projection (the render runs **in the viewer,
continuously**) and a **continuity** facet (one-shot vs continuous/interactive); continuous
→ static is a **snapshot frame/recording** on the live↔snapshot axis (T6).
- "Execute/render in the viewer" = an explicit **capability + trust/sandbox** surface.
## Yield
- **No new UC** (enrichment / design prior art). Enriches **UC-54, UC-55**; links UC-83,
UC-84, UC-35.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Program-as-page, view-time/live render, no-cached-output, capability+trust, INTENT mapping, UC disposition (enrichment-only), architecture notes |

View File

@@ -0,0 +1,114 @@
# Processing & Processing.js — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T4 · **Subject:** Processing (creative
coding) and Processing.js / p5.js — **the sketch *is* the document**, rendered live at view
time in the browser.
## Why this dive
Low-priority, enrichment-focused. Processing tests a page-model edge the notebooks (T2/T3)
didn't: a page that is **wholly a program whose presentation *is* its running output** — no
separate input/output cells, no prose envelope. The defining question: what is a "page" when
**the rendered form is a live computation evaluated at view time**? It feeds UC-54
(computation-defined content) and UC-55 (non-Markdown / executable content) and sharpens the
**live↔snapshot** axis named at T6.
## 1. Program-as-page
- A **Processing sketch** is a program (`setup()` + `draw()`) whose output is a **canvas of
visual/animated/interactive content**. There are no "output cells" — **the program's
execution is the content**.
- **Processing.js** (Resig, 2008; now largely **p5.js**) runs sketches **client-side in the
browser** on `<canvas>`. The page ships the **source**; the **rendering happens at view
time** in the reader's browser — no server, no pre-rendered artifact required.
- So the durable artifact is **text (the sketch source)**; the *presentation* is a **live,
view-time derivation** of that source (a derivation-projection, T1) — with the twist that
the derivation runs **in the viewer**, continuously (animation/interaction), not once.
## 2. The view-time / live-render dimension
This adds a wrinkle beyond Jupyter's *captured* outputs (UC-84) and Mathematica's CDF:
- **No captured output at all by default** — unlike a notebook, a sketch typically stores
**only source**; there is nothing cached. The output exists only when executed.
- **Continuous & interactive** — `draw()` loops; mouse/keyboard drive it. The output is
**time-based and interaction-based**, so any static capture is a **single frame** (or a
recording) — it cannot represent the artifact faithfully (shared limit with Mathematica
`Dynamic` and, more extremely, Strudel T5).
- **Client-side execution = a capability + trust surface** — running arbitrary sketch code in
the viewer is an execution capability with sandboxing concerns; shard-wiki must treat
"render live in the viewer" as an explicit, gated capability, never an automatic behavior.
## 3. INTENT mapping (enrichment-only — no new UC)
### Reinforcements / refinements
- **Executable-as-page (UC-54/55).** Processing is the cleanest **"the whole page is the
program"** case: content = source text, presentation = view-time live render. Strengthens
the page model's need to represent **executable content whose rendered form is derived at
view time**, distinct from notebooks (which interleave cells + captured outputs).
- **Derivation-projection, view-time variant (T1/UC-83).** The render is a derivation-
projection that runs **in the viewer, continuously** — extends the projection model: a
derivation may be *materialized ahead* (CDF/nbconvert) or *run at view time* (sketch).
- **Live↔snapshot axis (T6).** With **no cached output** and **continuous/interactive**
rendering, the only static form is a **snapshot frame or recording** — a concrete point on
the live↔snapshot axis; honest treatment offers the snapshot and **marks it as a frame of a
live artifact**, never as the artifact.
- **Capability + trust gating.** "Execute in the viewer" is an explicit capability with a
sandbox/trust boundary — mechanism-over-policy: whether to run, sandbox, or only snapshot
stays configurable.
### Boundaries
- shard-wiki is **not a sketch runtime**; default is **attach the source + offer a captured
snapshot/recording**; live in-viewer rendering is a gated capability. Source is canonical;
the render is a degrading, view-time projection.
## 4. UC disposition (enrichment-only)
| Mechanism (findings §) | Catalog UC / thread |
|------------------------|---------------------|
| Sketch = program-as-page; presentation = view-time live render (§1) | UC-54 (enriched: executable/view-time content), UC-55 (enriched: non-Markdown executable) |
| Render = derivation-projection run **in the viewer, continuously** (§1, §2) | links UC-83 (view-time variant) |
| No cached output; continuous/interactive → static = snapshot frame only (§2) | links UC-84, live↔snapshot axis (T6) |
| Client-side execution = capability + trust/sandbox surface (§2) | links UC-35 (capability-awareness) |
**No new UC** — Processing is design prior art reinforcing executable-as-page; its
contribution is the **view-time, no-cached-output, continuous-render** point on the
projection/liveness model.
## 5. Architecture notes for SHARD-WP-0002
- **T12/T15 (page model):** represent **program-as-page** — a page whose canonical content is
**source text** and whose presentation is an **executable render**; no inherent cached
output (contrast notebooks). Non-Markdown executable content type.
- **T16 (projection):** add a **materialization timing** facet to derivation-projection:
**ahead-of-time** (CDF/nbconvert/static HTML) vs **at view time** (sketch render); and a
**continuity** facet: one-shot vs continuous/interactive. Continuous → static is a snapshot
frame/recording on the live↔snapshot axis (T6).
- **T11 (capabilities):** "execute/render in the viewer" is a capability with a **trust/
sandbox** sub-concern; default off → snapshot.
## 6. Open questions
1. Is **view-time execution** ever offered (sandboxed in-viewer render), or does shard-wiki
always degrade program-as-page to a captured snapshot/recording? (Capability/trust policy.)
2. Should **materialization timing** (ahead-of-time vs view-time) and **continuity**
(one-shot vs continuous) be explicit projection metadata, alongside the live↔snapshot
axis? (Recurs at T5.)
## 7. Sources
- `processing.org`; **p5.js** (`p5js.org`); Processing.js (Resig) history; `<canvas>` /
client-side rendering model.
- prior: `research/260614-jupyter-deep-dive/` (captured vs no-cached output, UC-84);
`research/260614-mathematica-deep-dive/` (`Dynamic`/CDF, snapshot-only);
`research/260614-squeak-pharo-deep-dive/` (live↔snapshot axis).
## 8. Traceability
**No new UC** (enrichment / design prior art). Enriched: UC-54, UC-55; links UC-83 (view-time
derivation-projection), UC-84 (no-cached-output contrast), UC-35 (execute capability +
trust). Architecture cross-refs: SHARD-WP-0002 T12/T15 (program-as-page, source-canonical/
render-derived), T16 (materialization-timing + continuity facets on the live↔snapshot axis),
T11 (view-time-execute capability + sandbox).

View File

@@ -0,0 +1,15 @@
# 260614 — Salesforce Quip deep dive
Deep dive on **Salesforce Quip**: a closed SaaS of **live collaborative documents** with
**embedded spreadsheets and live apps**, a REST API (HTML import/export), and
**Salesforce-tied identity/permissions**. A hosted-collab contrast to Notion: the
document+spreadsheet hybrid under enterprise auth.
- `findings.md` — the doc/spreadsheet model, REST API, enterprise auth, capability profile,
INTENT mapping, UC seed (UC-80), architecture notes for SHARD-WP-0002, open questions,
sources, traceability.
Catalog yield: UC-80 (attach a SaaS live-doc shard whose pages **mix prose + embedded live
structured objects** — spreadsheets/apps — via REST with lossy HTML import/export, under
enterprise identity). Enriched UC-57/55/58/36/06. Feeds SHARD-WP-0002 T11 (capability /
content opacity), T14 (external-API binding).

View File

@@ -0,0 +1,148 @@
# Salesforce Quip — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T6 · **Subject:** Quip (Salesforce), a
collaborative-document SaaS.
## Why this dive
Notion (UC-57/58/59) gave us the closed-SaaS, external-REST-only, database-as-pages model.
Quip is the **enterprise document-collab** contrast: a **document+spreadsheet hybrid** where
a page is **prose interleaved with embedded live structured objects** (spreadsheets,
calendars, kanban "live apps"), reachable only through a REST API, gated by **Salesforce
identity**. The question: how does shard-wiki attach a shard whose "page" is a *mixed
prose+live-object document* behind enterprise auth?
## 1. The document + live-object model
- A Quip **document** is a real-time collaborative doc (concurrent editing). Its body is
rich content — headings, lists, prose — **interleaved with embedded objects**:
- **Spreadsheets** are *first-class, inline* — a doc can contain live spreadsheet sections
with formulas, not just static tables.
- **Live apps** (calendars, kanban boards, project trackers, polls) embed interactive
structured widgets inside the document.
- **Folders** organize documents; **threads** (docs and chat are unified — every doc is also
a conversation thread) carry **messages/comments** inline.
- So a Quip "page" is a **hybrid**: prose + embedded structured/live content in one
document, with conversation attached. Not Markdown; a proprietary rich model.
## 2. The REST API (the only door)
Quip exposes a **REST API**: threads/documents (get, create, **edit-document** with an HTML
fragment + a location/section anchor), folders, messages, users, and spreadsheet cell
access. Content crosses the API as **HTML** (import and export) — there is **no native
Markdown** and no file/git access. Implications:
- **Import/export is HTML → lossy** to Markdown (like Notion's export, UC-59): embedded
spreadsheets and live apps **do not round-trip** to Markdown cleanly — they degrade to
tables or links.
- Editing is **section/anchor-scoped HTML splice** (`edit-document` targets a location) — a
mid-granularity write (a section, not the whole doc, not a character).
- **Rate-limited** like any SaaS API; history is internal (revisions exist in-product, with
limited API exposure).
## 3. Enterprise identity / permissions
Quip is tied to **Salesforce** (acquired 2016): authentication and access run through the
Salesforce org / Quip site; **folder and document sharing ACLs** govern visibility, with
enterprise SSO. So this is an **enterprise-ACL, authn-delegated** shard (UC-06) — shard-wiki
honors Salesforce-side permissions and never bypasses them.
## 4. Capability profile
| Dimension (synthesis spectrum) | Quip |
|--------------------------------|------|
| Attachment mode | **external-API** (REST; HTML import/export) — closed SaaS |
| Addressing granularity | document; **section/anchor** for edits; spreadsheet cell |
| Content identity | Quip thread/document ID (opaque) |
| Identity vs placement | API-id identity; folder placement separate |
| Structure | **hybrid doc**: prose + embedded spreadsheets + live apps + thread |
| History | **internal** revisions (limited API exposure) |
| Merge model | server-side real-time collab (OT-like); no external merge |
| Native query | none exposed (spreadsheet formulas internal) |
| Translation | **HTML** in/out → **lossy** to Markdown (spreadsheets/apps degrade) |
| Write granularity | **section/anchor HTML splice** (mid) |
| Operational envelope | rate-limited SaaS REST |
| Access grant | **Salesforce identity + folder/doc ACL** (enterprise SSO) |
| Content opacity | proprietary rich model; not E2EE but not transparent files |
| Provenance | author/edit metadata in-product; API-limited |
## 5. INTENT mapping
### Reinforcements
- **External-API attachment** (UC-57): Quip is a second concrete instance beside Notion of
the closed-SaaS REST-only shard — generalizes the external-API mode (REST + HTML payload,
vs Notion's REST + block-JSON, vs Wiki.js GraphQL).
- **Union without erasure / no silent flatten**: embedded spreadsheets and live apps must be
**surfaced as what they are** with provenance, and the **HTML→Markdown lossiness made
explicit** (UC-59) — never silently drop a live spreadsheet to a static table.
- **Authz-in-core, authn-delegated** (settled decision): Quip's Salesforce-tied ACL is the
enterprise case — honor delegated identity and the shard's ACL (UC-06).
- **Graceful degradation**: with only a lossy HTML export, Quip is still usable as a
read/projection/overlay-target/backup shard.
### Divergences (boundaries / notes)
- **Mixed prose+live-object page** is a content shape beyond "Markdown body + frontmatter":
the page model must allow **embedded typed/live objects within a prose page** (not just a
whole-page-is-a-record like Notion DB, but *inline* structured content) — feeds T12 and the
non-Markdown-content question (UC-55).
- **HTML is the only interchange** — no Markdown, no files, no git. Content opacity is
"proprietary-but-exportable": transparent-ish via lossy HTML, not via files (T11
content-opacity tier between transparent-files and E2EE-opaque).
- **Write granularity = section-anchor HTML splice** — a mid tier (between whole-file and
block/character) realized over an API, distinct from fedwiki's story-item op-log.
### What to keep
1. **External-API mode generalized** to carry an **HTML payload** variant (Quip) beside
block-JSON (Notion) and GraphQL (Wiki.js) — capability/payload-format is part of the
adapter profile (UC-57/UC-80).
2. **Inline embedded live/structured objects** as a page-model element (prose + embedded
spreadsheet/app), with **explicit lossy projection** to Markdown (UC-55/UC-58/UC-59).
3. **Enterprise-ACL + delegated identity** honored, not bypassed (UC-06).
## 6. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-80 | Attach a **SaaS live-doc shard** whose pages **mix prose + embedded live structured objects** (spreadsheets / live apps) via **REST with lossy HTML import/export**, under **enterprise (Salesforce) identity** | **new** |
| — | external-API mode w/ HTML payload variant | enrich **UC-57** |
| — | inline embedded spreadsheet/live-app = non-Markdown content | enrich **UC-55** |
| — | embedded structured objects within a prose page | enrich **UC-58** |
| — | internal (API-limited) revision history | enrich **UC-36** |
| — | Salesforce-tied enterprise ACL + SSO | enrich **UC-06** |
## 7. Architecture notes for SHARD-WP-0002
- **T11 (capability / content opacity):** add a **payload-format** facet to external-API
shards (HTML / block-JSON / GraphQL) and a **content-opacity tier** "proprietary but
lossy-exportable" between transparent-files and E2EE-opaque. Write granularity =
section-anchor splice.
- **T12 (page model):** support **inline embedded structured/live objects** within a prose
page (not only whole-page-as-record) — with explicit lossy render to Markdown.
- **T14 (binding):** external-API binding with **HTML import/export**; honor Salesforce
identity/ACL; default to read/projection/overlay given rate limits + lossy export.
## 8. Open questions
1. Does shard-wiki represent an **embedded live spreadsheet** as a typed sub-object with
provenance (preferred) or flatten it to a static Markdown table (lossy) — and can overlays
target a spreadsheet cell via the API, or only the prose?
2. Given **HTML-only, lossy** interchange and rate limits, is Quip ever a **write-through**
shard, or read/projection/overlay/backup by default? (cf. Notion Q1, UC-57.)
3. How is **Salesforce identity** mapped to shard-wiki's delegated-authn model — pass-through
token, service account, or per-user OAuth?
## 9. Sources
- Quip Automation API (REST) docs — quip.com / Salesforce developer docs (threads/documents,
`edit-document`, folders, messages, spreadsheets)
- Salesforce Quip product docs (live apps, spreadsheets, sharing/permissions)
- prior: `research/260614-notion-deep-dive/` (closed-SaaS external-API contrast, UC-57/59)
## 10. Traceability
New UC **UC-80** carries the marker **◧** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-57, UC-55, UC-58, UC-36, UC-06. Architecture
cross-refs: SHARD-WP-0002 T11 (payload-format + content-opacity), T12 (inline objects), T14.

View File

@@ -0,0 +1,49 @@
# 260614 — Roam Research deep dive (the block graph as a queryable database + extension API)
Date: 2026-06-14
## What this is
A focused study of **Roam Research** — core architecture and extension structure —
read through shard-wiki's lens. Roam is the **modern bookend** to the two Nelson dives:
where Xanadu (`research/260614-xanadu-deep-dive/`) and ZigZag
(`research/260614-zigzag-deep-dive/`) are *unbuilt ideals*, Roam **shipped** stable
fine-grained addressing, transclusion, bidirectional links, and a queryable structured
space to a mainstream audience.
Distinctive material:
- **Data model** — the entire graph is a client-side **DataScript** (Datomic-like) DB
of **datoms** (EAV); every paragraph and page is a **block** entity with a stable
nine-character **`:block/uid`**, `:block/string`, `:block/refs`, `:block/page`,
pages marked by `:node/title`
- **References & transclusion** — `[[Page]]`/`#tag`/`((uid))` create `:block/refs`
edges; **block embeds** are working transclusion; linked vs unlinked references
- **Querying** — Datalog (`:q`/`:pull`); derived views are queries over the graph
- **Extension architecture** — **Roam Depot** (`extension.js` `onload`/`onunload`,
manifest, `extensionAPI` settings) over **`window.roamAlphaAPI`** read (`q`/`pull`) +
write (`block.create/update/move/delete`, `page.create`) — a concrete
*engine-hosts-adapter* (UC-38) surface
## Contents
| Path | Role |
|------|------|
| `findings.md` | Data model, references/transclusion, Datalog, extension API, Roam-as-shard capability profile, evidence on shard-wiki's open questions, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
(UC-50 attach a block-graph DB shard with block↔page mapping, UC-51 adopt native
block/span IDs as portable span addresses, UC-52 delegate derived views to a shard's
native query engine); UC-32/34/35/38 enriched. Logged for
`workplans/SHARD-WP-0002-federation-architecture.md` (T14): native-span-ID and
native-query capabilities, block↔page mapping, and Roam as a second DB-backed /
engine-hosts-adapter exemplar alongside XWiki.
**Key payoff recorded:** Roam empirically answers the Nelson dives' open questions —
fine-grained addressing is tractable (block UIDs), transclusion is a data-layer
capability over an addressable union, and derived views are queries over a structured
space. **Boundary:** Roam is *one candidate shard* (DB-backed/API-attached, block-first,
no portable Git history), mapped into shard-wiki's Markdown-first page model — not a
substrate and not the federation layer.
</content>

View File

@@ -0,0 +1,292 @@
# Findings — Roam Research: the block graph as a queryable database, and its extension API
Date: 2026-06-14
Source kind: **modern shipped product** — a block-graph note tool; a *candidate shard
backend* (DB-backed / API-attached, like XWiki), and the place where several Nelson
ideas actually shipped
Lens: shard-wiki — fine-grained addressing, transclusion, bidirectional links, a
structured/queryable page space, and the engine-hosts-adapter extension path
> Why Roam, and why now. The previous two dives were Ted Nelson's *unbuilt* ideals —
> Xanadu (`research/260614-xanadu-deep-dive/`: reference-not-copy, transclusion,
> stable fine-grained addresses) and ZigZag (`research/260614-zigzag-deep-dive/`: an
> information space as many co-equal dimensions over a queryable structure). **Roam
> Research is where a large slice of both actually shipped to a mainstream audience.**
> Every block has a stable address; block references and embeds are working
> transclusion; the whole graph is a queryable database; links are bidirectional. So
> Roam is the natural modern bookend: it lets us check shard-wiki's open questions
> (portable span addressing, transclusion, structured pages) against a system that
> solved them *in production* — and study its **extension architecture** as a concrete
> adapter-host surface.
This dive treats Roam as a *candidate shard* (capability profile in §6) and as
*evidence* on the open questions the Nelson dives left (§7), not as something to copy
wholesale — Roam is block-first and DB-backed, shard-wiki is Markdown-page-first and
Git-backed (divergences in §8.2).
---
## 1. What Roam is, in architecture terms
Roam Research is a networked-thought outliner whose **entire graph is a client-side
DataScript database** (DataScript = a ClojureScript reimplementation of Datomic's
data model and Datalog query engine). The store is not files; it is a set of immutable
**datoms**.
- A **datom** is one fact: `[entity, attribute, value, transaction]` — the EAV(T)
model. The graph is the accumulation of datoms; edits are transactions.
- Synchronization and Roam's deep undo fall out of the transaction log — a parallel
worth noting to Git's immutable commit history (but it is *not* Git; see §8.2).
This makes Roam a **structured, queryable** wiki-shaped store — the same family as
XWiki (objects + DB) rather than TWiki/Foswiki (flat files), which matters for how an
adapter attaches (§6).
---
## 2. The block model — everything is a block, every block is addressable
Every paragraph *and* every page is a **block** (an entity). Key attributes:
| Attribute | Meaning |
|-----------|---------|
| `:block/uid` | **nine-character public ID** — the block's stable, referenceable address (e.g. `((GGv3cyL6Y))`) |
| `:block/string` | the block's text content |
| `:block/order` | position among siblings |
| `:block/children` | immediate child entity-ids |
| `:block/parents` | full ancestor chain (incl. the page) |
| `:block/refs` | edges to blocks/pages this block references |
| `:block/page` | the owning page entity |
| `:create/time`, `:edit/time`, `:create/email`, `:edit/email` | provenance metadata |
| `:node/title` | **page-only** — the page's title (this attribute is what distinguishes a page from an ordinary block) |
Structurally the DB is a **forest**: each page is a tree of nested blocks; a "page" is
just the top-level block that carries `:node/title`. The single most shard-wiki-relevant
fact: **`:block/uid` gives every block — i.e. every sub-page span — a stable,
first-class, public address.** This is the *shipped* form of the Xanadu tumbler / the
shard-wiki "portable span address" open question (§7).
---
## 3. References, transclusion, bidirectional links
- **Page references** `[[Page]]` and `#tags`, and **block references** `((uid))`, all
create `:block/refs` edges. The link graph is therefore queryable, not parsed-on-read.
- **Block embeds** (`{{embed: ((uid))}}`) are **working transclusion**: a block's
content rendered live in another location by reference, not copied — when the source
changes, the embed reflects it. This is Xanadu transclusion / shard-wiki UC-32/UC-44,
shipped at block granularity.
- **Bidirectional links** come in two flavours, exactly shard-wiki's BackLinks problem:
- **Linked references** — explicit `:block/refs` edges (queryable backlinks).
- **Unlinked references** — bare text mentions with no edge, surfaced by text search.
- **Attributes** `key:: value` are a convention on top of blocks/refs that turns a page
into a lightweight record — Roam's answer to structured/typed pages (UC-34/UC-39),
queryable via Datalog.
---
## 4. Querying — Datalog over the graph
Roam exposes **Datalog** (`:find` / `:where`) plus **pull** expressions:
```
[:find ?uid :where [?b :block/string ?s] [?b :block/uid ?uid] ...]
```
This means Roam's "derived views" (a page's linked references, a `{{query}}` block, a
table) are **just queries over the datom graph** — not bespoke features. For shard-wiki
this is the strongest evidence yet that *derived views = queries over a structured
space* (the ZigZag "dimensions + rasters" insight, made executable), and that an
adapter could **delegate view computation to a shard's native query engine** instead of
scanning (UC-52).
---
## 5. Extension architecture — Roam Depot and `window.roamAlphaAPI`
Roam's official extension system (**Roam Depot**) is a concrete model of the
*engine-hosts-adapter* path (UC-38):
- **Distribution:** an extension is a GitHub repo (`README.md`, `extension.js`, optional
`extension.css` / `CHANGELOG.md` / `build.sh`), cataloged in the `roam-depot` repo via
a metadata JSON (`name`, `author`, `tags`, `source_repo`, `source_commit`, optional
`stripe_account` for paid extensions).
- **Lifecycle:** `extension.js` default-exports a map with **`onload`** and
**`onunload`** functions; everything created on load must be torn down on unload. An
`extensionAPI` object provides a **settings panel** and `settings.get/set`.
- **Data API — `window.roamAlphaAPI`** (the surface an adapter would target):
- **Read:** `q` (Datalog query → entity ids), `pull` / `pull-many` (fetch entity
attributes), full graph traversal.
- **Write:** `data.block.create` / `update` / `move` / `delete`, `data.page.create`.
- **UI:** command palette, blocks, settings.
Crucially, this API runs **inside the Roam client** (global `window.roamAlphaAPI`).
There is no first-class external REST write API; programmatic external access uses
graph **export** (EDN / JSON / Markdown) or unofficial private APIs. So a *live,
write-through* Roam adapter must be **hosted inside Roam as a Depot extension** — which
is exactly the UC-38 engine-side direction (cf. XWiki components, TWiki plugin handlers).
---
## 6. Roam as a shard — capability profile
| Capability | Roam | Notes for the adapter contract |
|------------|------|--------------------------------|
| Read | **yes** | `q`/`pull` live (in-app) or EDN/JSON/Markdown export (snapshot) |
| Write | **yes, in-app** | `block.create/update/move/delete`, `page.create` — but only via the in-client API → needs a Depot-hosted adapter (UC-38) or unofficial private API |
| Write granularity | **block-level (fine)** | the opposite extreme from TiddlyWiki's whole-file writes — sharpens UC-35 |
| Identity / addressing | **block `:block/uid` + page `:node/title`** | shipped sub-page stable addressing (UC-51) |
| Transclusion | **yes** | block embeds by uid (UC-32/44/45) |
| Backlinks | **yes** | linked (`:block/refs`) + unlinked (text) (UC-05/18) |
| Structured data | **yes** | attributes + Datalog (UC-34/39) |
| Native query | **yes (Datalog)** | adapter could delegate views (UC-52) |
| Diff / merge | **no native** | transaction log exists but not exposed as page diffs |
| Version history | **hosted, internal-only** | not portable Git history → needs supplement/import (UC-36, like Confluence/MediaWiki) |
| Lock | **no** | — |
| Syntax | **Roam-flavored Markdown** | not CommonMark; block-outline structure → translation/mapping needed (cf. UC-42) |
Verdict: Roam is a legitimate **DB-backed, API-attached shard** — the XWiki family, not
the file-store family. Its standout offerings are **block-level addressing** and
**shipped transclusion**; its costs are **block↔page impedance** and **no portable
history**.
---
## 7. Evidence on shard-wiki's open questions (the payoff)
The Nelson dives left open questions; Roam answers several *empirically*:
1. **"What is a portable fine-grained span address?"** (Xanadu §11 Q1, the tumbler
problem). Roam's answer: **a short opaque per-block UID, minted by the store, public
and referenceable.** shard-wiki lesson — fine-grained addressing is *tractable* when
the backend mints stable block/span IDs; the adapter should **adopt native block IDs
as span addresses where they exist** (UC-51), and fall back to content-fingerprint or
path+range where they do not (cf. Xanadu content-identity, UC-46).
2. **"Does transclusion belong in core or adapter/UI?"** (catalog Q6). Roam shows
transclusion is cheap and natural *when the store is addressable and queryable*; it
lives at the **data layer** (refs/embeds), surfaced by UI. Argues for transclusion as
a **core capability over an addressable union**, not a UI-only trick.
3. **"Derived views — core or adapter?"** Roam shows them as **queries over a
structured space**. Where a shard exposes a native query engine, **delegate**
(UC-52); where it does not, the orchestrator computes them over the projection.
4. **Block↔page mapping** is the new question Roam *raises*: in a block graph, what is a
"page"? Roam's own answer (`:node/title` node = page; nested blocks = spans) is a
clean mapping rule for the adapter (UC-50).
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **Fine-grained addressing is achievable** (block UIDs) — de-risks UC-44/45/51 and the
Xanadu tumbler worry.
- **Transclusion + bidirectional links + structured data** are not exotic; a shipped
tool does all three over one addressable, queryable store — validating shard-wiki's
ambitions for UC-32/34/39/05.
- **Engine-hosts-adapter (UC-38)** has a clean modern template: Depot's
`onload/onunload` + `roamAlphaAPI` read/write surface.
### 8.2 Deliberate divergences (design bugs if conflated)
1. **Block-DB substrate vs. Markdown-file page model.** Roam's "everything is a block in
a DataScript DB" must **not** tempt shard-wiki away from its Markdown-first,
backend-neutral page model (INTENT Stability Note). Roam is *one shard shape*, mapped
*into* the page model, not the model itself.
2. **Client-side proprietary DB vs. Git coordination journal.** Roam's transaction log
is not portable history; shard-wiki keeps Git as the coordination layer. A Roam shard
contributes snapshots/projections, and its history needs **supplement/import**
(UC-36), not adoption as canonical.
3. **Single graph vs. federation.** A Roam graph is **one sovereign shard**, never the
federation layer. Do not model the union as "a big Roam graph."
4. **Roam-flavored Markdown + outline structure** ≠ CommonMark pages; the adapter owns a
**lossy-aware translation** (block outline ↔ page + headings/lists), cf. UC-42.
### 8.3 What Roam teaches that shard-wiki should keep
- **Mint/adopt stable sub-page IDs.** The cheapest path to transclusion, overlay, and
reverse-lookup at span granularity is a backend that already addresses spans — lean on
it (UC-51), degrade gracefully otherwise.
- **Treat a structured shard's query engine as a capability**, and delegate derived
views to it when present (UC-52).
- **A clean block↔page rule** (`:node/title` node = page) keeps a block backend usable
without flattening (UC-50, complementing UC-34's no-lossy-flatten rule).
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-49**. New UCs **UC-50UC-52** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a block-graph DB wiki (Roam-style) as a shard via its query/CRUD API, mapping blocks to the page model** (`:node/title` node = page; nested blocks = spans) | **UC-50 (new)** |
| **Adopt a shard's native block/span IDs as portable span addresses** for transclusion/overlay (the shipped answer to the span-address question) | **UC-51 (new)** |
| **Delegate derived views to a shard's native query engine** (Datalog) where advertised | **UC-52 (new)** |
| Shipped transclusion via block embeds + shipped fine-grained addressing | **enriches UC-32** (and UC-44/45) |
| Attributes + Datalog = shipped structured/typed pages | **enriches UC-34** (and UC-39) |
| Roam Depot `onload/onunload` + `roamAlphaAPI` = modern engine-hosts-adapter template | **enriches UC-38** |
| Block-level write granularity = the fine extreme opposite TiddlyWiki | **enriches UC-35** |
| Hosted history is internal-only, not portable Git | links **UC-36** |
---
## 10. Architecture notes for SHARD-WP-0002 (no UC)
- The adapter contract should model **"native span/block IDs"** as a capability: a shard
advertises whether it mints stable sub-page addresses; transclusion/overlay/reverse-
lookup capabilities key off it (ties UC-44/45/46/51).
- The contract should model a **"native query"** capability so the orchestrator can
delegate derived-view computation (UC-52) vs. scanning the projection.
- A **block↔page mapping** belongs in the adapter, with the no-lossy-flatten rule of
UC-34: page = titled node, blocks = addressable spans, attributes = sidecar metadata.
- Roam joins XWiki as a **DB-backed / API-attached** exemplar (vs. TWiki/Foswiki file
stores) and as a **second engine-hosts-adapter** template (vs. XWiki components / TWiki
handlers) for the T14 adapter-binding task.
---
## 11. Open questions (for spec / workplans)
1. When a backend mints native span IDs (Roam UIDs), are they used **directly** as the
shard-wiki span address, or **wrapped** in a shard-scoped address so they survive
projection/overlay and don't collide across shards?
2. For a **write-through** Roam shard the adapter must run *inside* Roam (Depot
extension). Is in-engine hosting an accepted adapter deployment mode generally, or do
we restrict Roam to **read/projection/overlay-target** (graceful degradation) when we
cannot deploy inside it?
3. How lossy is the **block-outline ↔ Markdown-page** mapping, and is round-trip
fidelity required or best-effort (cf. UC-42 Foswiki TML↔HTML)?
4. Do we consume Roam's **export (EDN/JSON)** as the projection source, or the live
`roamAlphaAPI`? Snapshot vs. live freshness trade-off (cf. UC-31).
---
## 12. Sources
| Source | Used for |
|--------|----------|
| zsolt.blog — "Deep Dive Into Roam's Data Structure" (https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html) | Datom/EAV model, block attributes (`:block/uid` etc.), refs/embeds, linked vs unlinked references, Datalog/pull |
| Roam Research Datalog Cheatsheet (https://gist.github.com/2b3pro/231e4f230ed41e3f52e8a89ebf49848b) | Attribute names, `:q`/`:pull` query shapes |
| GitHub — Roam-Research/roam-depot (https://github.com/Roam-Research/roam-depot) | Extension repo/manifest structure, `extension.js` `onload`/`onunload`, `extensionAPI` settings, `roamAlphaAPI` read/write/ui surface |
| "Introduction to the Roam Alpha API" (https://www.putyourleftfoot.in/introduction-to-the-roam-alpha-api) | `window.roamAlphaAPI` `q`/`pull` semantics, global-scope dependency |
| David Vargas — "How To Create Your Own Roam Extensions" (https://davidvargas.me/blog/how_to_create_your_own_roam_extensions) | Extension authoring lifecycle, write methods |
| Wikipedia / general — Roam Research, DataScript, Datomic | Datalog/Datomic lineage, client-side DB framing |
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (span addressing,
transclusion), `research/260614-zigzag-deep-dive/findings.md` (queryable dimensions),
`research/260613-xwiki-deep-dive/findings.md` (DB-backed engine + extension host),
`spec/UseCaseCatalog.md` (UC-05, UC-18, UC-32, UC-34/35/36/38/39, UC-44/45/46),
`workplans/SHARD-WP-0002-federation-architecture.md` (T14 adapter binding).
---
## 13. Traceability
- New UCs: **UC-50, UC-51, UC-52**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-32, UC-34, UC-35, UC-38** (and links UC-36, UC-39, UC-44/45/46).
- Architecture (no UC): native-span-ID capability; native-query capability; block↔page
mapping; Roam as DB-backed/API-attached + engine-hosts-adapter exemplar → `SHARD-WP-0002` (T14).
- Boundary recorded: Roam is **one candidate shard**, mapped into the Markdown-first
page model; not a substrate, not the federation layer (INTENT Stability Note).
</content>

View File

@@ -0,0 +1,69 @@
# 260614 — Shard spectrum synthesis (one capability model across the full dive set)
Date: 2026-06-14 · **revised 2026-06-14 (v3)**
## What this is
A **synthesis** (no new external research) that reads every studied system *across* the
others and distills them into a single comparative model feeding the **shard adapter
contract** and the **federation track** (`SHARD-WP-0002`).
**v3** extends the model to the **SHARD-WP-0003 engine batch** (Federated Wiki, Wikibase/
Wikidata, the git-forge wikis Gitea/GitLab/GitHub, TiddlyWiki, ikiwiki, Salesforce Quip,
MojoMojo, Oddmuse, UseModWiki) — ~23 systems in all. Per-shard spectra grow from thirteen to
**fourteen** (adds **provenance granularity**), and a **new coordination-layer axis** is
introduced: the **federation-model taxonomy**.
Centerpieces:
- **The shard family matrix** — every candidate backend × {substrate, attach mode(s),
addressing, structure, history, merge, query, →Markdown, opacity}, with Xanadu/ZigZag as
ideal anchors and Federated Wiki as the federation-model anchor. v3 adds the flat-file
floor (Oddmuse/UseModWiki), git-IS-store (forge wikis/ikiwiki), TiddlyWiki, MojoMojo
(direct-DB), Quip (external-API/HTML), and Wikibase (typed-graph far-end).
- **The fourteen capability spectra** — the claim that the adapter contract should model
*positions on spectra*, each anchored at both ends by a real system, with federation ops
degrading by position. v3 adds **provenance granularity** (14th: per-shard → per-page →
per-edit → per-statement/value), and refines merge-model (+fork/journal-replay,
+coexist-with-rank), attachment-mode (+git-IS-store, +container, +direct-DB,
+REST/file-store-hybrid, +external-API payload-format facet), native-query (+SPARQL/RDF
far-end, +filter mid), history (+DB-version-rows, +partial/truncated), structure
(+typed-graph, +inline-embedded objects), content-opacity (+proprietary-lossy-exportable),
write-granularity (+story-item, +section-anchor).
- **The federation-model taxonomy (§2.5, new)** — federation itself is plural: fork+journal
(Federated Wiki), VCS-replication+ping (ikiwiki), query-time graph join (Wikibase SPARQL
`SERVICE`), feed aggregation, activity streams, engine-mirror (Wiki.js). A
selectable/composable coordination-layer axis feeding **T1T6**.
- **UC-44UC-82 → workplan task mapping** (T1T6 + T11T16).
## Contents
| Path | Role |
|------|------|
| `findings.md` | Family matrix, the fourteen spectra, the federation-model taxonomy, cross-cutting through-lines, UC→task fold-in, recommendations/decisions, escalated open questions |
## Status
Synthesis v3 complete. No new use cases (consolidation only). Feeds `SHARD-WP-0002`:
**T1T6** federation-model taxonomy (selectable/composable federation); **T11** reframed
around the **fourteen spectra** (incl. provenance granularity + expanded attachment modes +
external-API payload-format facet); T12 page-model breadth (prose + typed/computed records +
**inline-embedded objects** + **typed-graph statements** + non-Markdown assets +
query-defined + multi-placement/DAG identity); T13 history (DB-rows + partial-flat-file =
supplement; completeness metadata; **journal-shaped** coordination journal); T14 **full
attachment taxonomy** (git-IS-store / container / direct-DB / REST-file-hybrid +
source-of-truth per binding); T15 lossy + **not-Markdown** (graph/HTML); T16 (addressing
incl. statement GUID + **opaque stable identity**, identity≠placement, derived index,
**graph query / federated SERVICE**). UC coverage extended UC-34UC-67 → **UC-34UC-82**.
**New through-lines (v3):** federation is plural (the federation-model taxonomy); provenance
has a granularity spectrum (down to per-statement, Wikibase rank/references); git is both the
home store and the home journal (forge wikis make git *the* store, resolving the engine-
mirror write-race); the flat-file floor (Oddmuse/UseModWiki, Wikipedia's MediaWiki Phase I)
is the field's common root and the minimal capability profile; the page model must also carry
typed-graph statements and inline-embedded objects.
**Carried from v2:** CRDT changes the merge math; identity ≠ placement; metadata can be
computed; content opacity is per-item; the attach surface is not always the native store; the
block-graph-on-files sweet spot (Logseq `id::`).
**Carried from v1:** files-canonical/index-derived; fine-grained addressing is adoptable;
transclusion=clone=embed=reference is one primitive; structure/history federate iff in-text;
attach mode is per-binding; the platform can enforce no-silent-mutation.

View File

@@ -0,0 +1,406 @@
# Synthesis — the shard spectrum: one capability model across the full dive set
Date: 2026-06-14 · **revised 2026-06-14 (v3)** — extended from the fourteen-system set to
include the **SHARD-WP-0003 engine batch** (Federated Wiki, Wikibase/Wikidata, the git-forge
wikis Gitea/GitLab/GitHub, TiddlyWiki, ikiwiki, Salesforce Quip, MojoMojo, Oddmuse,
UseModWiki). Per-shard spectra grown from thirteen to **fourteen** (added **provenance
granularity**), with refinements to merge-model, attachment-mode, native-query, structure,
history, content-opacity and write-granularity; and a **new coordination-layer axis — the
federation-model taxonomy (§2.5)** — the headline v3 contribution.
Source kind: **synthesis** — consolidates every deep dive into a single comparative model
feeding the **shard adapter contract** and the **federation track** (`SHARD-WP-0002`).
Lens: shard-wiki — what a backend must expose to participate, expressed as *spectra* of
capability rather than a yes/no checklist.
> Purpose. ~23 tools/systems have now been studied. Two conceptual ancestors (**Xanadu,
> ZigZag**); the file/DB engines (**XWiki, TWiki, Foswiki**, + landscape); the modern
> note/PKB tools (**Roam, Obsidian, Notion, Joplin, Logseq, Anytype, AFFiNE, AppFlowy,
> Trilium**); and the WP-0003 batch (**Federated Wiki, Wikibase, forge wikis, TiddlyWiki,
> ikiwiki, Quip, MojoMojo, Oddmuse, UseModWiki**), against the federation and origin
> research. This document reads them *across* each other. The payoff is a small set of
> **capability spectra** — now **fourteen** per-shard, plus a **federation-model taxonomy**
> for the coordination layer — each anchored at both ends by a real system, with federation
> operations degrading by position. That spectrum *is* the adapter contract's design
> surface. v3 folds the WP-0003 use cases (**UC-68UC-82**) into the `SHARD-WP-0002` tasks.
Inputs: `research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`,
`research/260613-{xwiki,twiki,foswiki}-deep-dive`,
`research/260614-{xanadu,zigzag,roam,obsidian,notion,joplin,logseq,localfirst-workspaces,trilium,wikijs,federated-wiki,wikibase,forge-wikis,tiddlywiki,ikiwiki,quip,mojomojo,oddmuse,usemodwiki}-deep-dive`.
Output target: `spec/TechnicalSpecificationDocument.md` (adapter contract) via
`SHARD-WP-0002`.
---
## 1. The shard family matrix
Candidate-shard backends across the dimensions that matter to the contract. (Xanadu and
ZigZag are *not* shards — they are the conceptual ideals each column aspires to; Federated
Wiki is *barely* a shard — it is mostly a **federation model**, see §2.5; both listed apart.)
| Backend | Substrate | Attach mode(s) | Addressing | Structure | History | Merge | Query | →Markdown | Opacity |
|---------|-----------|----------------|-----------|-----------|---------|-------|-------|-----------|---------|
| **Git folder / forge wiki** | files (`.wiki.git`) | **file-store (git IS store)** | path (+ commit) | flat MD + frontmatter | **git-native** | git/text | no | native | none |
| **ikiwiki** | files (git) → static HTML | file-store (source repo) | path | flat MD + directives | **git-native** | git/text | no | native (src) | none |
| **Obsidian** | files | file-store **/** plugin | path + in-file `^id` | frontmatter (in-file) | none (Git plugin) | git/text | plugin (Dataview) | native (OFM) | none |
| **Logseq** | files (MD/Org) → SQLite | file-store **/** plugin | **in-file block `id::`** (sweet spot) | `key::` props (in-file) | none (git) | git/text | **Datalog (derived)** | native-ish | none |
| **TiddlyWiki** | single HTML **/** `.tid` dir | **file-store (container/`.tid`)** | tiddler `title` | typed tiddler **fields** | none / git (.tid) | git/text | **filters** | varies (`type`) | none |
| **TWiki / Foswiki** | files + RCS / pluggable | file-store **/** API | path | `%META%` in-file (N records) | **open file (RCS)** | git/text | no | lossless (TML) | none |
| **Oddmuse / UseModWiki** | flat files (CGI) | file-store (**minimal floor**) | CamelCase/path | none (flat) | flat-file, **partial** | last-writer | no | lossy | none |
| **MojoMojo** | relational DB | **direct-DB-read** | page row / path | relational rows+lineage | **DB version rows** | last-writer | SQL | MD-in-column | none |
| **Trilium** | SQLite (one file) | ETAPI / scripting / DB | `noteId` + **`branchId`** (id≠place) | labels+relations, **inherited+templated**; **DAG** | internal revisions | conflict-res | attr search | **lossy (HTML)** | **per-note** |
| **Joplin** | SQLite-local | **sync-mirror** / local-REST / plugin | page-level `:/id` | notebooks + tags | internal revisions | conflict-notes | search | lossy | E2EE (whole) |
| **Wiki.js** | DB ↔ git mirror | file-store(mirror) / **GraphQL** | path | frontmatter (mirror) | git (via mirror) | git/text | GraphQL | native (mirror) | none |
| **XWiki** | DB (Hibernate) | in-engine host / REST | path | **XObjects/XClass** | internal (`xwikircs`) | git/text | yes (XWQL) | engine syntax | none |
| **Quip** | hosted (SaaS) | **external-API (HTML)** | doc id + section anchor | **prose + inline spreadsheets/live-apps** | internal | server (OT) | no | **lossy (HTML)** | proprietary |
| **Roam** | client DataScript | **in-app host only** | **store UUID** | `key::` attrs | internal (txn log) | (in-app) | yes (Datalog) | (Roam MD) | none |
| **Notion** | hosted Postgres | **external-API (block-JSON)** | store UUID | **DB schema + relations + rollups** | internal, not portable | internal | yes (DB query) | **lossy** | none |
| **Wikibase / Wikidata** | MediaWiki + RDF index | external-API + **SPARQL** | **opaque Q/P id + stmt GUID** | **typed entity-statement graph** | page revisions (JSON) | last-writer + **rank** | **SPARQL (+SERVICE)** | **lossy (not MD)** | none |
| **Anytype** | **CRDT (any-sync)** | **replica / P2P node** | object id (store) | **typed object graph (ontology)** | CRDT log | **native-CRDT** | graph | lossy | **E2EE (whole)** |
| **AFFiNE** | **CRDT (Yjs)** | replica / self-host | block id (store) | blocks; **one-data-many-views** | CRDT log | **native-CRDT** | DB filters | lossy | optional |
| **AppFlowy** | **CRDT (Yrs)** | replica / self-host | block id (store) | **Notion-style DB + views** | CRDT log | **native-CRDT** | DB query | lossy | self-host |
| — *Federated Wiki (mostly a federation model, §2.5)* | per-page JSON | **REST/file-store hybrid** | story-item `id` | typed story items | **append-only journal** | **fork + journal-replay** | neighborhood search | wiki-ish | none |
| — *Xanadu (ideal)* | permascroll | — | **tumbler (span)** | spans + links | permanent | content-merge | — | — | — |
| — *ZigZag (ideal)* | cells/dims | — | cell id | **N dimensions** | — | — | dimension walk | — | — |
Reading top to bottom is roughly shard-wiki's difficulty gradient, now with both ends
extended: the **flat-file floor (Oddmuse/UseModWiki)** anchors the bottom — minimal, partial
history, the field's common ancestor; **git/forge wikis, ikiwiki, Obsidian, Logseq,
TiddlyWiki** are friction-free file-store cases (forge wikis make *git the store itself*);
**TWiki/Foswiki/Trilium/Wiki.js** add translation, DAG/computed metadata, or a git mirror;
**MojoMojo** needs direct-DB reads; **XWiki/Roam/Notion/Quip** add DB/SaaS structure and
store/API addressing; **Wikibase** is the **typed-knowledge-graph far-end** (SPARQL,
statement-level provenance); the **CRDT cohort** adds native merge + (Anytype) P2P/E2EE.
Xanadu/ZigZag mark the ideals; Federated Wiki marks the federation-model ideal (§2.5).
---
## 2. The capability spectra (the contract's real shape) — fourteen
Each capability is **not boolean** — it is a position on a spectrum anchored at each end by
a real system. The contract models *positions*; federation ops degrade by position.
1. **Addressing granularity** — `none → whole-page(path) → page-level store id(Joplin :/id,
Trilium noteId) → in-file span(Obsidian ^id) → in-file block(Logseq id::, the sweet spot:
block-level AND git-diffable) → store-minted span(Roam/Notion/CRDT UUID) → statement GUID
(Wikibase) → portable tumbler(Xanadu ideal)`. Story-item `id` (Federated Wiki) is a
mid-tier within-page handle. (UC-51, UC-44/45, UC-73.)
2. **Content identity** — `none → path/title → opaque stable id, labels-as-annotation
(Wikibase Q/P) → fingerprint(hash) → span-set/equivalence(Xanadu)`. Wikibase is the
cleanest real instance of **stable, language-neutral identity**. (UC-46, UC-27, UC-73.)
3. **Identity vs placement** — `path = identity(most) → identity separated from placement
(Trilium note/branch; a page in many locations = a DAG) → provenance-edge links across
sites (Federated Wiki fork-DAG; forge wiki = path, identity layered above)`. The clean
model for a page under multiple paths/shards. (UC-66, UC-22, UC-71.)
4. **Structure** — `flat Markdown(Oddmuse) → in-file frontmatter/key::(Obsidian/Logseq) →
in-file %META%(TWiki) → tiddler fields(TiddlyWiki) → relational rows(MojoMojo) → typed
objects(XWiki) → DB schema+relations+rollups(Notion/AppFlowy) → prose+inline-embedded
objects(Quip) → typed object graph/ontology(Anytype) → computed inherited+templated
(Trilium) → typed entity-statement knowledge graph(Wikibase)`. In-text federates;
DB-locked needs sidecar+fidelity; **graph/computed** needs effective-vs-own +
render-without-flatten. (UC-34, UC-39, UC-58, UC-67, UC-73, UC-80.)
5. **History** — `none → partial/truncated flat-file(Oddmuse keep/) → internal-only/
not-portable(Notion/Joplin/Trilium) → DB version rows(MojoMojo) → CRDT update log
(Anytype/AFFiNE/AppFlowy) → append-only semantic journal(Federated Wiki) → open file
format(TWiki RCS) → git-native(Git/forge/ikiwiki/Obsidian/Logseq/Wiki.js mirror)`.
Internal/CRDT/DB-rows ⇒ *supplement*; open-file/journal ⇒ *import*; git ⇒ *adopt*.
**Completeness is metadata** (Oddmuse history is partial — never imply complete). (UC-36,
UC-41, UC-71, UC-81, UC-82.)
6. **Merge model** — `none → last-writer → git/text 3-way merge → conflict-notes/keep-both
(Joplin) → fork + manual journal-replay(Federated Wiki) → coexist-with-rank(Wikibase:
contradictory values kept, curated) → native-CRDT conflict-free(Anytype/AFFiNE/AppFlowy)`.
Four+ distinct models — never impose git/text merge on a CRDT or a journal shard; speak
the shard's model or stay projection/overlay. (UC-64, UC-71, UC-75.)
7. **Native query** — `none → text search → filter expressions(TiddlyWiki) → build-your-own
derived index(Logseq DataScript over files; shard-wiki can do likewise) → datalog/graph
(Roam/Anytype) → DB query(Notion/AppFlowy/XWiki) → SPARQL/RDF + federated SERVICE
(Wikibase, the far-end + query-time cross-shard join)`. Delegate where present; **build an
index over the projection** where not. (UC-52, UC-63, UC-05, UC-54, UC-74.)
8. **Translation to Markdown** — `native → lossless round-trip(Foswiki TML↔HTML) → lossy-
with-fidelity-report(HTML/CKEditor Trilium; Notion blocks; Quip HTML+embedded objects;
CRDT/object models) → not-Markdown-at-all(Wikibase statements → lossy render or keep
graph)`. Lossless ⇒ writable; lossy ⇒ read/projection floor + visible fidelity loss;
not-MD ⇒ structured payload + optional rendered view. (UC-42, UC-59, UC-03, UC-73, UC-80.)
9. **Attachment mode** — a **per-binding, capability-gated** choice; a backend may offer
several. The full taxonomy:
- **file-store** — *native on-disk store* (Obsidian/Logseq/TWiki), *git IS the store*
(forge `.wiki.git`, ikiwiki source), *single-file container* (TiddlyWiki HTML),
*flat-file floor* (Oddmuse), **or** *interchange/sync mirror* (Joplin; Wiki.js git
mirror) (UC-40, UC-53, UC-60, UC-62, UC-76, UC-78, UC-79, UC-82, UC-68)
- **in-engine host** — adapter inside the app via its API (Roam/Obsidian/Logseq/Trilium
scripting, XWiki components) (UC-38, UC-50)
- **local-REST** — localhost API, app-running (Joplin Data API; Trilium ETAPI) (UC-38)
- **external-API** — remote API from outside, with a **payload-format facet**:
*block-JSON* (Notion, UC-57), *GraphQL* (Wiki.js, UC-69), *HTML* (Quip, UC-80),
*forge wiki REST* (GitLab/Gitea, UC-77), *MediaWiki/SPARQL* (Wikibase, UC-73/74)
- **direct-DB-read** — read the engine's relational store (MojoMojo) when no file/API
exists; schema = a versioned coupling (UC-81)
- **CRDT replica** — hold a local CRDT replica (Anytype/AFFiNE/AppFlowy) (UC-64)
- **P2P / no-central-endpoint** — replica or peer/node, not a URL (Anytype) (UC-65)
- **REST/file-store hybrid** — page JSON over HTTP+CORS or static files (Federated Wiki)
(UC-70)
10. **Operational envelope** — `local/unbounded → realtime CRDT/WebSocket → rate-limited+
eventually-consistent+paginated(Notion ~3 rps, Quip, Wikibase public endpoints)`. Sets
live vs cache/poll/webhook. (UC-57, UC-31.)
11. **Access grant** — `open(L0; Oddmuse) → token → OAuth scoped+revocable(Notion) →
enterprise SSO + ACL(Quip/Salesforce, Wiki.js path rules) → P2P key/invite(Anytype) →
own-site-only writes(Federated Wiki)`. The backend may *enforce* no-silent-mutation.
(UC-57, UC-06, UC-65, [[shard-wiki-auth-in-core-decision]].)
12. **Content opacity** — `plaintext(files) → proprietary-but-lossy-exportable(Quip HTML;
Notion) → encrypted-at-rest whole-shard(Joplin/Anytype E2EE) → per-item(Trilium
protected notes)`. Opaque ⇒ backup/structure-shell; never present ciphertext (or imply
a lossy export is faithful) as readable. (UC-61, UC-80.)
13. **Write granularity** — `whole-file(TiddlyWiki single-file) → per-page/note(Git/forge/
Obsidian/Joplin/Trilium/Oddmuse) → story-item/paragraph(Federated Wiki) → section-anchor
splice(Quip) → per-statement(Wikibase API) → per-block(Roam/Notion/Logseq/CRDT)`. Sets
overlay/patch/lock/conflict scope; whole-file ⇒ **no per-page atomicity**. (UC-35,
UC-78.)
14. **Provenance granularity** *(new, 14th)* — `none → per-shard → per-page(most; author/
time) → per-commit(git) → per-edit(journal entry, Federated Wiki) → per-statement/value
(Wikibase references + rank)`. How finely the union can attribute and source content;
Wikibase pushes provenance below the page (sourced, contradictory values coexist with a
curation signal). The page model + journal should *allow* sub-page provenance even if MVP
records per page. (UC-24, UC-71, UC-75.)
*(Content types — Markdown-only → typed records → inline-embedded objects (Quip
spreadsheets) → non-Markdown assets (Excalidraw/Canvas/whiteboards) → typed-graph statements
(Wikibase) — remains a cross-cutting page-model demand, tracked under structure + T12 rather
than as a standalone capability. UC-55.)*
Design consequence: **T11's capability vocabulary = these fourteen spectra**, not a flat
`read/write/diff/...` list. The flat verbs remain the *operations*; the spectra are the
*profile* saying how well each verb is supported and how it degrades. The **floor**
(Oddmuse/UseModWiki) and **far-ends** (Wikibase graph/query/provenance; CRDT merge; Notion
hosted) bound every spectrum with a real system.
---
## 2.5. The federation-model taxonomy (the coordination-layer axis) — new in v3
The fourteen spectra above describe a *single shard's* capabilities. The WP-0003 batch
revealed a second, orthogonal axis the v2 synthesis under-modelled: **federation itself is
plural.** "Attach many shards and present a union" can be realized by several distinct
coordination models, each a real system, each with different reconciliation semantics. This
axis lives at shard-wiki's **coordination layer** (`SHARD-WP-0002` T1T6), not in a single
adapter.
| Federation model | Exemplar | Mechanism | Reconciliation | Discovery |
|------------------|----------|-----------|----------------|-----------|
| **Fork + journal** | Federated Wiki | copy page to own site; append-only semantic **journal** records `fork`-with-source | **manual**: compare journals, fork the version you prefer; **chorus**, no canonical | link + fork (**neighborhood**) / curated **roster** |
| **VCS replication + ping** | ikiwiki | git clone/pull/push between instances; **XML-RPC pinger** notifies peers to pull/rebuild | **git merge** across clones | configured peers + pings |
| **Query-time graph join** | Wikibase | **SPARQL `SERVICE`** runs a sub-query on another endpoint and joins | none (read-time join; rank curates conflicts) | endpoint URLs |
| **Feed aggregation** | ikiwiki `aggregate`, RSS/Atom | pull remote feeds in as pages | one-way inbound projection | feed URLs |
| **Activity streams** | ActivityPub (federation research) | actor/inbox/outbox Create/Update | per-actor; eventual | actor handles |
| **Engine-maintained mirror** | Wiki.js git mirror | DB-canonical engine syncs to a git mirror | engine owns DB↔git sync (don't double-sync) | the mirror repo |
Two cross-cutting lessons:
- **git-IS-store vs engine-mirror resolves the write-race.** A **forge wiki** (`.wiki.git`)
and **ikiwiki source** make *git the canonical store*, so **write-by-commit is safe** — no
engine to race. Wiki.js (engine-mirror, DB canonical) is the opposite and needs care. The
contract must record *which side is source of truth* per binding. (Resolves UC-68's open
race for the git-canonical case; UC-76.)
- **shard-wiki's own coordination journal should be journal-shaped.** Federated Wiki proves
the **append-only semantic-op log with provenance entries, page-state-as-derived-replay**
pattern in production — the concrete shape for INTENT's coordination journal (UC-71), and a
superset that can *ingest* git history, CRDT logs, DB version rows, and partial flat-file
histories as differently-grained inputs.
Design consequence: **T1T6 should model federation as a selectable model (or composition of
models), not a single hard-coded flow** — mechanism over policy at the coordination layer,
mirroring how T11 models per-shard capability as spectra. A given information space may use
fork+journal for human-curated shards, VCS-replication for git shards, query-join for graph
shards, and feed-aggregation for read-only sources — concurrently.
---
## 3. Cross-cutting findings (the through-lines)
- **Files-canonical, index-derived is the winning architecture.** Obsidian's MetadataCache,
**Logseq's DataScript-over-files**, Git's working tree, **Wikibase's WDQS (SPARQL index
rebuilt from canonical JSON entities)**, and shard-wiki's projection model agree: the
graph/backlinks/query index is **derived and rebuildable**, never a second source of truth.
Roam/Notion/CRDT invert this (store canonical) and pay in portability. shard-wiki keeps
files + journal canonical.
- **Git is both the home store and the home journal.** *(sharpened in v3)* Forge wikis make
git *the* store; ikiwiki and Wiki.js make git the source or mirror; for all of them the
**git log is the coordination journal with zero synthesis**. The git-canonical cases are
the friction-free core; everything else is measured as deviation.
- **The flat-file floor is the field's common root.** *(new)* Oddmuse and UseModWiki
(Wikipedia's MediaWiki Phase I) show the minimal plain-text page+history wiki is the
ancestor every richer engine elaborates — so the **minimal/floor capability profile** is
the right baseline, and shard-wiki's page model must stay attach-compatible with it
(flat files, CamelCase identities, **partial** history). (UC-82.)
- **Federation is plural** *(new — §2.5).* Fork+journal, VCS-replication+ping, query-time
graph join, feed aggregation, activity streams, engine-mirror — distinct coordination
models, selectable and composable, not one flow.
- **Provenance has a granularity spectrum** *(new — spectrum 14).* From per-shard down to
**per-statement/value** (Wikibase references + rank). Union-without-erasure includes
*attribution at the right grain* and letting sourced contradictions coexist with curation.
- **Identity ≠ placement.** Trilium's **note vs branch** (a note cloned into many locations =
a DAG) and Wikibase's **opaque stable IDs (labels-as-annotation)** are the clean models:
separate *what a page is* from *where it sits* and from *what it's called*. Federated
Wiki's fork entries are **provenance edges** between same-named pages across sites.
- **Transclusion ⇄ clone ⇄ embed ⇄ cloned-note ⇄ reference is one primitive** over an
addressable union (Xanadu, ZigZag, Roam/Logseq, Obsidian `![[ ]]`, Notion synced block,
Trilium cloning, TiddlyWiki `{{ }}`). (UC-32/44/45/51/66.)
- **CRDT changes the merge math; journals and rank add more models.** *(extended)* The merge
spectrum now spans last-writer → git 3-way → conflict-notes → **fork+journal-replay** →
**coexist-with-rank** → native-CRDT. Never impose git/text merge across that range.
- **Structure & history federate iff in text; metadata can be computed; the far-end is a
graph.** `%META%`/frontmatter/`key::`/tiddler-fields diff and travel; XObjects/Notion-DB/
CRDT/Quip-objects lock in; Trilium computes metadata (inherited+templated); **Wikibase is
a full typed knowledge graph** — render to Markdown is lossy, so keep the graph and offer a
view, never silent-flatten.
- **The attach surface is rarely "just files," and source-of-truth varies.** *(extended)*
Joplin's best surface is its sync mirror; Logseq offers file-graph or DB-graph and is
migrating substrate; **forge wikis = git-canonical (write freely)** vs **Wiki.js =
DB-canonical mirror (write carefully)** vs **MojoMojo = DB-only (direct read)**. Bind to
capabilities *and* record the source of truth.
- **The page model must stretch many ways at once:** prose Markdown, typed/computed records
(N-per-page, relations, inheritance), **inline-embedded objects** (Quip), **typed-graph
statements** (Wikibase), non-Markdown assets, reference/query-defined pages, and
**multi-placement (DAG) identity**. The heaviest demand on T12.
---
## 4. How the use cases fold into the workplan
The 260614 + WP-0003 use cases (UC-44UC-82) map onto the adapter-contract and federation
tasks:
| UCs | Theme | Lands in |
|-----|-------|----------|
| UC-35, UC-50, UC-53, UC-57, UC-60, UC-62, UC-64, UC-65, **UC-68, UC-70, UC-76, UC-77, UC-78, UC-79, UC-80, UC-81, UC-82** | attachment modes (file-store native/git-IS-store/container/mirror, in-engine, local-REST, external-API w/ payload-format, direct-DB, CRDT-replica, P2P, REST/file-store-hybrid) + operational envelope | **T11** + **T14** |
| UC-34, UC-39, UC-55, UC-58, UC-67, **UC-73, UC-80** | structured/typed/computed/**graph** payload, **inline-embedded objects**, non-MD assets | **T12** |
| UC-36, UC-41, **UC-81, UC-82** | history: internal/CRDT-log/**DB-rows**/**partial-flat-file** = supplement; **completeness metadata** | **T13** |
| UC-42, UC-59, **UC-73, UC-80** | translation: lossless vs lossy-with-fidelity vs **not-Markdown** (graph/HTML/objects) | **T15** |
| UC-31, **UC-79** | webhooks / realtime / push-vs-poll / **VCS ping** | **T6** / **T11** envelope |
| UC-57 §6, UC-61, UC-65, **UC-06, UC-80** | scoped grant; **enterprise ACL/SSO**; content opacity (whole/per-item/**lossy-exportable**); P2P key | **T11** (access-grant + opacity) |
| UC-44, UC-45, UC-46, UC-51, UC-63, UC-66, **UC-73, UC-74** | span/statement addressing, **opaque stable identity**, identity≠placement, transclusion-as-reference, derived index, **graph query** | **T16** (+ T12) |
| UC-47, UC-48, UC-52, UC-54, **UC-74** | dimensional navigation, query delegation/build, query-defined pages, **SPARQL/federated SERVICE** | **T16** (+ T5/T10) |
| UC-24, **UC-71, UC-75** | **provenance granularity** (per-edit / per-statement); coordination journal shape | **T13/T16** + journal |
| UC-26, UC-27, UC-28, UC-30, UC-05, **UC-71, UC-72, UC-79** | **federation-model taxonomy** (fork+journal, VCS-replication+ping, query-join, chorus/neighborhood/roster) | **T1T6** |
The one **structural** v3 change: the **federation-model taxonomy (§2.5)** is a new design
surface for **T1T6** — federation becomes a *selectable, composable model* rather than a
single flow. Per-shard, **T11** grows to **fourteen spectra** (adds provenance granularity)
and **T14** absorbs the expanded attachment-mode taxonomy (git-IS-store, container,
direct-DB, REST/file-store-hybrid, external-API payload-format facet).
---
## 5. Recommendations (decisions to record under SHARD-WP-0002)
1. **Model per-shard capabilities as the fourteen spectra (§2)**, not flat verbs. (T11.)
2. **Model federation as a selectable/composable taxonomy of models (§2.5)** — fork+journal,
VCS-replication+ping, query-time graph join, feed aggregation, activity streams,
engine-mirror — at the coordination layer. (T1T6.)
3. **Make the coordination journal journal-shaped** (Federated Wiki): append-only semantic
ops + provenance entries, page state = derived replay; able to *ingest* git history, CRDT
logs, DB version rows, and partial flat-file histories as differently-grained inputs.
(T13 + journal, UC-71.)
4. **Add provenance granularity as a spectrum** (per-shard → per-page → per-edit →
per-statement/value); allow sub-page provenance and **coexist-with-rank** for sourced
contradictions. (T11/T13, UC-24/75.)
5. **Record source-of-truth per binding** and resolve the write-race accordingly:
git-canonical (forge wiki / ikiwiki) ⇒ write-by-commit; engine-mirror (Wiki.js) ⇒ careful;
direct-DB (MojoMojo) ⇒ read/projection/overlay default. (T14, UC-68/76/79/81.)
6. **Expand attachment mode (§2 #9 / T14):** file-store (native / **git-IS-store** /
**container** / mirror / **flat-file floor**), in-engine, local-REST, **external-API with
a payload-format facet** (block-JSON/GraphQL/HTML/forge-REST/SPARQL), **direct-DB-read**,
CRDT-replica, P2P, **REST/file-store-hybrid**. Bind to capabilities; substrate can
migrate. (T14, UC-40/68/70/76/77/78/79/80/81/82 + UC-43.)
7. **Page model: support a typed-graph payload and inline-embedded objects, not only
typed records** — render to Markdown lossily, keep the graph/objects canonical; preserve
computed (inherited/templated) metadata as effective-vs-own. (T12/T15, UC-73/80/67.)
8. **Capability profile must express absence and partiality cleanly** (Oddmuse floor):
sparse profiles, **partial/truncated history reported honestly**, graceful degradation to
read/projection/overlay/backup. (T11/T13, UC-82.)
9. **Query: SPARQL/RDF + federated `SERVICE` is the graph far-end** — delegate to native
engines (filters/datalog/DB-query/SPARQL) where present; build a derived index over the
projection where not. (T16/T5, UC-52/63/74.)
10. **Adopt opaque stable identity with labels-as-annotation** (Wikibase) and **separate
identity from placement** (Trilium note/branch); fork entries are provenance edges.
(T16, UC-73/66/71.)
11. **Keep files + coordination journal canonical; all indexes/projections derived and
rebuildable.** (Architecture invariant, INTENT.)
These honor INTENT: mechanism over policy (capability *and now federation* modelled as
spectra/taxonomy, not hard-coded behaviors), union without erasure (fidelity +
effective-vs-own + identity/placement + **provenance granularity** all preserve
information), graceful degradation (every spectrum has a read/projection floor; the
Oddmuse floor is explicit; opaque/CRDT/graph shards degrade to backup/projection/lossy-view),
no silent mutation (access-grant + opacity + overlay + respect native merge + **source-of-
truth per binding**), shard sovereignty (no backend forced to change substrate),
Markdown-first/backend-neutral (in-text preferred; DB/CRDT/HTML/**graph** tolerated with a
view).
---
## 6. Open questions escalated by the synthesis
1. **Federation composition** — can one information space run several federation models
(§2.5) concurrently over different shards, and how does the union reconcile a chorus
(fork+journal) with a canonical-asserting shard (Notion / upstream main)? (T1T6,
UC-72.)
2. **Coordination-journal op vocabulary** — adopt Federated Wiki's exact ops
(create/add/edit/move/remove/fork) at item grain, or an abstract op set other shards can
emit, with git/CRDT/DB-row/flat-file histories ingested as inputs? (T13, UC-71.)
3. **Provenance granularity in the model** — does the journal/page model carry per-statement
provenance + rank (Wikibase), per-edit (journal), or per-page (MVP), configurable? (T13,
UC-75.)
4. **Typed-graph page** — model a Wikibase entity natively (statements) or always project to
a lossy Markdown/table view, or both (canonical graph + view)? Is SPARQL a union-level
capability or pass-through? (T12/T16, UC-73/74.)
5. **Source-of-truth + write-race** — formalize git-canonical vs engine-mirror vs direct-DB
per binding; sanction direct third-party DB reads (schema drift) and write-by-commit
timing. (T14, UC-68/76/79/81.)
6. **Portable span address** across heterogeneous backends — wrap native IDs (Roam/Notion/
CRDT UUID, Logseq `id::`, Trilium noteId, **Wikibase Q/P + stmt GUID**, fedwiki story-item
id) in a shard-scoped address? (T16.)
7. **CRDT shards** — embed a CRDT client (Yjs/Yrs) for a live replica, or consume snapshots?
Overlays as CRDT ops or out-of-band patches? (T14, UC-64.)
8. **Page model breadth** — can one model carry prose + typed/computed records +
inline-embedded objects + typed-graph statements + non-MD assets + query-defined pages +
multi-placement identity coherently? (T12.)
9. **Whole-file & partial-history shards** — overlays against a whole-file shard
(TiddlyWiki single-file: buffer+re-serialize vs require `.tid`); representing
partial/truncated history (Oddmuse) with completeness metadata. (T11/T13, UC-78/82.)
10. **Content opacity & lossy-exportable** — what is visible for an opaque/proprietary shard
(IDs/structure vs nothing vs lossy HTML); never present a lossy export as faithful; does
shard-wiki ever hold keys? (T11, UC-61/80.)
---
## 7. Sources
A synthesis; primary sources are the ~23 dives' `findings.md` files plus
`research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`.
No new external research was performed in v3 (it consolidates the WP-0003 batch dives
260614-{federated-wiki,wikibase,forge-wikis,tiddlywiki,ikiwiki,quip,mojomojo,oddmuse,
usemodwiki}).
Cross-references: `spec/UseCaseCatalog.md` (UC-26UC-82),
`workplans/SHARD-WP-0002-federation-architecture.md` (T1T16),
`workplans/SHARD-WP-0003-engine-dives-batch.md` (the engine batch, done),
`INTENT.md` (constraints), [[shard-wiki-auth-in-core-decision]].
---
## 8. Traceability
- Consolidates: all ~23 deep dives + federation/origin research into one capability model
(v3 adds the WP-0003 batch).
- Feeds: `SHARD-WP-0002` **T1T6** (federation-model taxonomy §2.5 — selectable/composable
federation), **T11** (**fourteen-spectra** vocabulary, incl. provenance granularity +
expanded attachment modes + payload-format facet), T12 (page-model breadth: typed-graph,
inline-embedded objects, computed metadata, identity≠placement, non-MD), T13 (history incl.
DB-rows + partial-flat-file = supplement, completeness metadata, journal-shaped
coordination journal), T14 (full attachment taxonomy incl. git-IS-store / container /
direct-DB / REST-file-hybrid + source-of-truth per binding), T15 (lossy + not-Markdown
graph/HTML), T16 (addressing incl. statement GUID + opaque stable identity, graph query /
federated SERVICE).
- UC coverage extended in the workplan from UC-34UC-67 to **UC-34UC-82**.
- No UCs added (synthesis only); no boundary changes (INTENT Stability Note untouched —
the federation-model taxonomy is a refinement of *how* the coordination layer works, not a
redefinition of shard / Git's role / orchestrator-vs-engine).

View File

@@ -0,0 +1,33 @@
# 260614 — Squeak & Pharo (image-based Smalltalk) deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T6 (Squeak) + T8 (Pharo)** — combined memo
(justified merge: both are image-based Smalltalks; Pharo is T8's thin "context for T6/T7").
## What this is
A deep dive into the **image-based live-object** environment — Squeak and Pharo (the
substrate Glamorous Toolkit T7 runs on): the **image** as a persistent world of live objects
with **no file/document/app boundary**, the live **inspector**, and Pharo's retreat to
**code-as-text in git** (Tonel/Iceberg).
## Why it matters
- The **purest "live" end** of the batch's spectrum (literate source → notebook snapshot →
GT/Lepiter live-over-files → **image: everything live**). Names the **live↔snapshot** axis
the projection model (T16) must carry.
- Hardens the **image-is-not-a-store** boundary (opaque monolithic non-diffable blob; no
page identity/history/provenance) — generalizes "attach files, not the kernel/image"
(UC-84, T7) into a named binding boundary (T14).
- Pharo **confirms** the resolution: even image traditions externalize to **git-versionable
text** (Tonel) to version/collaborate — files-canonical from the Smalltalk side.
## Yield
- **No new UC** (boundary / design prior art; covers T6 and T8). Boundary for UC-34/35/79;
links UC-83/84 (live→snapshot), UC-54/47/48 (live-object inspection), UC-76/79 (Tonel/git).
## Contents
| Path | Role |
|------|------|
| `findings.md` | The image & live objects, the image-as-store anti-pattern, Pharo Tonel/Iceberg→git, INTENT mapping, UC disposition (enrichment-only), architecture notes (T14 boundary, T16 live↔snapshot axis) |

View File

@@ -0,0 +1,141 @@
# Squeak & Pharo (image-based Smalltalk) — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 **T6 (Squeak)** + **T8 (Pharo)** — combined
(justified merge: both are image-based Smalltalks; Pharo is T8's "context for T6/T7," kept
brief per the workplan). · **Subject:** the **image-based live-object** environment and what
it teaches (and warns) about shard-wiki's page model.
## Why this dive (and why merged)
Squeak (1996, the Alan Kay/Ingalls lineage) and Pharo (2008 fork, the substrate **Glamorous
Toolkit** T7 runs on) are the same idea: **a persistent world of live objects, the "image,"
with no file/document/application boundary.** They are **not candidate shards** — they are
the **anti-pattern boundary** for shard-wiki's files-canonical stance *and* the inspiration
behind moldable inspection (T7). T8 (Pharo) is folded here as the workplan allows: its
distinct contribution over Squeak (Tonel/Iceberg file-based code → git) is the one piece that
*does* touch our concerns, covered in §3.
## 1. The image: knowledge-as-live-objects
- A Smalltalk **image** is a serialized snapshot of the **entire object memory** — every
object, class, tool, window, and the running program state — persisted as one binary file
(`.image` + `.changes` log). You resume exactly where you left off.
- **"Everything is a live object"**: code, data, UI (Morphic), the debugger, the inspector —
all are objects you can open, message, and modify *in place*, while running. There is **no
edit/compile/run cycle** and **no document-vs-app distinction**.
- The **inspector** lets you open any object and explore/modify its state live — the direct
ancestor of GT's moldable inspector (T7), but generic rather than domain-molded.
This is the **purest "live" end** of the spectrum the whole batch traverses: literate source
(static) → notebook captured output (snapshot) → GT/Lepiter (live results over files) →
**image (everything live, nothing inherently a file).**
## 2. The boundary: image-as-store is the anti-pattern
The image directly **contradicts** shard-wiki's design constraints, which is exactly why it's
worth recording as a hard boundary:
- **Opaque, monolithic, non-diffable.** The image is one big binary blob of entangled live
state — no per-page identity, no text diff, no mergeable history, no provenance per unit.
It violates *union-without-erasure granularity*, *Markdown-first*, and *git-addressable
coordination*.
- **No stable addressable "page."** Knowledge is an object graph in memory, not addressable
documents — there is nothing to attach at page granularity without an export step.
- **History is a `.changes` log**, a serial source-change stream, not a content history.
**Conclusion (boundary recorded):** an image is **not a shard and not a store**. This is the
generalized form of the rule already hit at Jupyter (UC-84) and GT (T7): *attach the
exported files, never the live image/kernel.* The image can only participate via an **export
projection** (objects/code → files), which is a **derivation-projection** (T1) that
**degrades the liveness to a static snapshot**.
## 3. Pharo's twist: code-as-files (Tonel) → git (the one actionable bit)
Pharo (T8) matters precisely where it **retreats from pure-image**:
- **Tonel / FileTree** serialize each class/method as **plain-text files** in a directory,
and **Iceberg** manages those files as a **git repository** — so Pharo code lives in git as
text, diffable and mergeable, *outside* the image.
- This is the **same move** as Lepiter (T7), nbstripout/Jupytext (T3), and ikiwiki source
(UC-79): **the durable, attachable artifact is the file representation; the live
environment is a layer above it.** It confirms our stance from the *Smalltalk* side: even
the most image-centric tradition externalizes to **git-versionable text** to collaborate.
So Pharo adds **no new page-model idea** beyond "image-based environments still externalize
to git text" — exactly the "keep brief / fold" expectation. Its value is **confirming the
boundary resolution**: attach the **Tonel/git source**, treat the image as live-only.
## 4. INTENT mapping
### Inspiration (keep)
- **Live-object inspection** is the seed of moldable views (T7/UC-54): the *idea* that any
unit can be opened and explained interactively. shard-wiki adopts this as **projection/
view**, not as a storage model.
- **Resume-where-you-left-off** liveness names the far end of the **live↔snapshot** axis the
contract must place every computed/projected view on (UC-83/84): the more live the source,
the more its attached form is a **degrading snapshot**.
### Boundary (enforce — design-bug if violated)
- **Image-as-store is a design-bug boundary.** Never model an image (or any monolithic live
memory blob) as a shard/store; participate only via **export → files** (a degrading
derivation-projection). Generalizes "attach files, not the kernel/image" (UC-84, T7).
### Confirmation (Pharo)
- Even pure-image traditions externalize to **git-versionable text** (Tonel/Iceberg) to
version and collaborate — reinforcing **files-canonical + git coordination** as the
durable substrate; the live environment sits above it.
## 5. UC disposition (enrichment-only — no new UC)
| Mechanism (findings §) | Catalog UC / thread |
|------------------------|---------------------|
| Live-object inspector = generic ancestor of moldable views (§1) | links UC-54, UC-47/48 (T7) |
| Image = opaque monolithic non-diffable blob; not a page/store (§2) | **boundary** for UC-34/UC-35/UC-79 (granularity, identity, files-canonical) |
| Image participates only via export→files = degrading derivation-projection (§2) | links UC-83, UC-84 (live→snapshot) |
| Pharo Tonel/Iceberg: code-as-text in git (§3) | links UC-79, UC-76 (git-canonical text) |
| `.changes` = serial source-change log, not content history (§1) | links UC-36 (history shape) |
Both Squeak and Pharo are **design prior art / boundary**, not candidate shards → **no new
UC**. They sharpen the **live↔snapshot** axis and harden the **image-is-not-a-store**
boundary; Pharo confirms even image traditions externalize to git text.
## 6. Architecture notes for SHARD-WP-0002
- **T14 (attach binding):** record the **image-is-not-a-store** boundary explicitly — a
monolithic live-memory blob is never an attach target; participation is via **export→files**
only. Generalize the "attach files, not the kernel/image" rule (UC-84, GT T7) to a named
boundary in the binding taxonomy.
- **T16 (projection):** add the **live↔snapshot** axis to the projection model — every
computed/projected view sits somewhere between "live (re-derivable on demand)" and "static
snapshot," and the more live the source, the more its attached form must be a clearly-
marked degrading snapshot.
- **T11/T12:** the live-object inspector is the *inspiration* for the moldable view registry
(T7), not a storage shape; nothing new to the page model itself.
## 7. Open questions
1. Is **live↔snapshot** an explicit, first-class metadata axis on every projection (so the
union can label "this view was live / is a snapshot from time T"), or implicit per
capability? (Recurs across UC-83/84, GT, Mathematica `Dynamic`, Strudel T5.)
2. Do we ever ingest a Smalltalk project by attaching its **Tonel/git** repo as an ordinary
git-text shard (no Smalltalk-specific adapter needed), confirming the boundary resolution?
## 8. Sources
- Squeak: `squeak.org`, the *Back to the Future* (Squeak) paper (Ingalls et al.), Morphic;
image/`.changes` model.
- Pharo: `pharo.org`, **Tonel** format, **Iceberg** (git integration), FileTree.
- prior: `research/260614-glamorous-toolkit-deep-dive/` (moldable inspector, Lepiter, T7);
`research/260614-jupyter-deep-dive/` (live→snapshot boundary, UC-84).
## 9. Traceability
**No new UC** (boundary / design prior art; covers **T6 Squeak** and **T8 Pharo** in one
justified-merge memo). Boundary hardened for: UC-34/UC-35/UC-79 (image-is-not-a-store);
links UC-83/UC-84 (live→snapshot), UC-54/UC-47/UC-48 (live-object inspection → moldable
views), UC-76/UC-79 (Pharo Tonel/git text), UC-36 (`.changes` history shape). Architecture
cross-refs: SHARD-WP-0002 T14 (image-is-not-a-store boundary; export→files only), T16
(live↔snapshot projection axis).

View File

@@ -0,0 +1,30 @@
# 260614 — Strudel.cc (live-coding REPL) deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T5**
## What this is
A deep dive into **Strudel.cc** (the JavaScript port of **TidalCycles**): a browser
**live-coding REPL** where terse **pattern source** is **evaluated live into time-based
audio** — code as a running musical performance, with **no document, no output cell, no file
of results**.
## Why it matters
- The **extreme of the live↔snapshot axis** (named at T6): output is **temporal, generative,
performative**, so there is **no faithful static form** — the best static projection is
**source (canonical, diffable) + an optional audio recording snapshot**, marked as one
performance. The **honesty test** for union-without-erasure + graceful degradation.
- Bounds the projection model's live end: ahead-of-time → view-time one-shot → continuous →
**irreducibly live/temporal (recording-only)**.
## Yield
- **No new UC** (enrichment / design prior art; far live end). Enriches **UC-54, UC-55**;
links UC-83/84, UC-37, UC-35.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Code-as-live-performance, the limit of static projection, INTENT mapping, UC disposition (enrichment-only), architecture notes (T16 far end of live↔snapshot axis) |

View File

@@ -0,0 +1,118 @@
# Strudel.cc — live-coding REPL — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T5 · **Subject:** Strudel.cc (the
JavaScript port of TidalCycles) — a browser **live-coding REPL** where code is a **running
musical performance**.
## Why this dive
The closing, lowest-priority dive — and the **extreme** of the live↔snapshot axis. Where
Processing (T4) renders *visual* output at view time, Strudel renders **time-based audio**
that only exists *while running and evolving*. It is the limit case for "can a page be a live
computation?" — the answer where **static projection is least faithful**. Enrichment-only;
feeds UC-54/55 and hardens the snapshot-degradation rule.
## 1. Code as a live, evaluated performance
- **Strudel** ports **TidalCycles**' pattern language to JavaScript, running entirely in the
**browser** (Web Audio). You write **pattern expressions** (e.g. `note("c e g")`,
`sound("bd sd")` with transformations) and **evaluate them live**; the sound updates
**without stopping** — the essence of *live coding*.
- The artifact is **terse source text** (a pattern); the "content" is the **sound it
produces over time**. There is **no document, no output cell, no file of results**
output is **ephemeral, temporal, and performative**.
- A Strudel "page" (a shared REPL link / snippet) is **source + the implicit promise of a
running evaluation**. The source is tiny and diffable; the experience is not capturable as
text.
## 2. The limit of static projection
Strudel pushes past Processing on every "live" dimension:
- **Temporal & generative** — output unfolds over time and may be **non-deterministic**
(randomness, evolving state). There is no single "frame"; the faithful capture is a
**recording (audio), itself just one performance**, not the artifact.
- **Performative** — the value is partly the **act of live editing**; even a recording loses
the live-coding dimension.
- So on the **live↔snapshot axis** (named at T6), Strudel sits at the **far live end**: the
best static projection is **(a) the source** (canonical, diffable) **+ (b) an optional
audio recording snapshot**, explicitly marked as one rendering of a live/temporal artifact.
This makes Strudel the **honesty test** for the contract: shard-wiki must be able to attach
such a source, present it truthfully (here is the source; a live render needs the runtime; a
recording is one performance), and **never imply a static page captures it**.
## 3. INTENT mapping (enrichment-only — no new UC)
### Reinforcements / refinements
- **Live-evaluated, time-based content (UC-54/55).** Strudel is the extreme executable-as-
page: **source canonical, presentation = a temporal live evaluation**. Confirms the page
model must represent content whose rendered form is **time-based / generative / performative**.
- **live↔snapshot axis (T6) — far end.** Establishes the **upper bound**: some content is
**irreducibly live**; static projection degrades to **source + a recording snapshot**, with
honesty about what's lost. Generalizes Processing's "snapshot frame" to "recording of one
performance."
- **Graceful degradation (INTENT).** A backend that can't run the REPL still serves the
**source** (tiny, diffable) and any **recording** as read/projection/backup — the
limited-backend-still-usable rule, at the hardest content type.
- **Union without erasure.** Presenting a Strudel shard must surface "this is a **live
temporal artifact**; what you see/hear statically is **source / one recording**" — never
hide the liveness or imply completeness.
### Boundaries
- shard-wiki is **not an audio/REPL runtime**; default = attach the **source** + offer/store
a **recording** with provenance; live in-viewer evaluation is a gated capability (trust/
sandbox, like Processing T4). Source is canonical; everything rendered is a degrading,
view-time/temporal projection.
## 4. UC disposition (enrichment-only)
| Mechanism (findings §) | Catalog UC / thread |
|------------------------|---------------------|
| Pattern source = live, evaluated, time-based performance (§1) | UC-54 / UC-55 (enriched: time-based/generative executable content) |
| Output ephemeral/temporal/non-deterministic → no faithful static form (§2) | links live↔snapshot axis (T6), far end |
| Best static projection = source + audio recording snapshot, marked as one performance (§2) | links UC-83/UC-84 (degrading projection), UC-37 (recording as backup) |
| Limited backend still serves source + recording (§3) | links UC-37 graceful degradation |
| Live in-viewer evaluation = capability + trust/sandbox (§3) | links UC-35 |
**No new UC** — Strudel is design prior art marking the **far live end** of the projection/
liveness model; it adds no orchestration scenario, it bounds one.
## 5. Architecture notes for SHARD-WP-0002
- **T16 (projection):** anchor the **far end of the live↔snapshot axis** — content can be
**irreducibly live/temporal/generative**; the contract must allow a projection to declare
"no faithful static form; static = source + a marked recording." Combined with T4's
**materialization-timing** and **continuity** facets, the projection model now spans:
ahead-of-time materialized → view-time one-shot → view-time continuous/interactive →
**temporal/generative/performative (recording-only snapshot)**.
- **T12/T15 (page model):** **time-based / generative executable content** as a page-model
edge; source canonical, render temporal.
- **T11 (capabilities):** "live-evaluate (audio/REPL) in viewer" capability + trust/sandbox;
default off → source + recording.
## 6. Open questions
1. Does the contract carry an explicit **"irreducibly live / no faithful static form"** flag
on a projection (so the union renders the honest fallback automatically)? (The far-end
resolution of the live↔snapshot axis open question from T6.)
2. Is a **recording** modeled as a stored **derivation-projection snapshot with provenance**
(one performance, time T, source rev R), reusing the UC-84 snapshot machinery?
## 7. Sources
- `strudel.cc` (docs, REPL); **TidalCycles** (`tidalcycles.org`) — the pattern language
Strudel ports; Web Audio.
- prior: `research/260614-processing-deep-dive/` (view-time render, continuity, UC-54/55);
`research/260614-squeak-pharo-deep-dive/` (live↔snapshot axis);
`research/260614-mathematica-deep-dive/` (`Dynamic` interactive, snapshot-only).
## 8. Traceability
**No new UC** (enrichment / design prior art; the far live end). Enriched: UC-54, UC-55;
links UC-83/UC-84 (degrading projection), UC-37 (recording = backup / graceful degradation),
UC-35 (live-evaluate capability + trust). Architecture cross-refs: SHARD-WP-0002 T16 (far end
of live↔snapshot axis: irreducibly-live content, static = source + marked recording), T12/T15
(time-based/generative executable content), T11 (live-evaluate capability + sandbox).

View File

@@ -0,0 +1,16 @@
# 260614 — TiddlyWiki deep dive
Deep dive on **TiddlyWiki** (TW5): an entire wiki — tiddlers **plus** the app engine — in
**one self-contained HTML file**, the **whole-file write-granularity** anchor of the
synthesis matrix, with a Node.js **file-per-tiddler** (`.tid`) substrate as the git-diffable
alternative, a tiddler/field record model, and **filter expressions** as the native query
language.
- `findings.md` — the single-file model, tiddler data model, dual substrate, filters,
capability profile, INTENT mapping, UC seed (UC-78), architecture notes for SHARD-WP-0002,
open questions, sources, traceability.
Catalog yield: UC-78 (attach a single-file self-contained wiki as one shard — parse tiddlers
out, project; write = rewrite the whole file, the coarsest write-granularity anchor).
Enriched UC-35/40/34/52/43. Feeds SHARD-WP-0002 T11 (write-granularity extreme) and T14
(single-file vs file-per-tiddler binding).

View File

@@ -0,0 +1,164 @@
# TiddlyWiki — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T3 · **Subject:** TiddlyWiki / TiddlyWiki5
(TW5), Jeremy Ruston's self-contained personal wiki.
## Why this dive
The synthesis matrix names **whole-file write granularity** as one extreme of the
write-granularity spectrum, anchored by TiddlyWiki. This dive confirms the anchor and the
*portability extreme* it implies — a wiki that is **a single HTML file you can email** — and
finds the twist: TiddlyWiki also has a **Node.js file-per-tiddler** substrate, so it spans
the granularity spectrum the way Logseq spans file/DB (UC-62). The question: how does
shard-wiki attach a backend whose *entire content is one file*?
## 1. The single-file model
Classic TiddlyWiki ships as **one `.html` file** that contains **both**:
1. the **TiddlyWiki core** (the JavaScript engine, parser, renderer, UI), and
2. **every tiddler** (all content), serialized into the file.
Open it in a browser and the file *is* the running application. There is **no server and no
build step** — the app reconstitutes itself from the file it was loaded from. This is the
portability extreme: a complete, self-hosting wiki in a single, emailable, USB-stick-able
artifact that runs offline anywhere.
**Saving** is the catch: a browser page cannot normally overwrite the file it came from, so
TiddlyWiki uses **"savers"** — TiddlyFox/browser extension, the File System Access API, a
Node.js server, TiddlySpot/put-savers, or "download a new copy." Crucially, **a save
rewrites the *entire* HTML file** (core + all tiddlers re-serialized). Hence **whole-file
write granularity**: there is no concept of writing one page in isolation in the
single-file mode — every save touches the whole artifact.
## 2. The tiddler data model
The atomic unit is the **tiddler** — a named record with **fields**:
- Core fields: **`title`** (the identity), **`text`** (the body), **`tags`**, **`created`**,
**`modified`**, **`type`** (content type, e.g. `text/vnd.tiddlywiki`, `text/markdown`),
plus **arbitrary custom fields** (any key→value). A tiddler is effectively a **flexible
flat record** — closer to a typed-field record (UC-34) than to prose-with-frontmatter.
- **Everything is a tiddler**: not just pages, but tags, macros, templates, themes, plugins,
and the wiki's own configuration are all tiddlers. A **plugin is a bundle of tiddlers**.
- Content markup is **WikiText** (TW5's own), though `type` can mark a tiddler as Markdown,
JSON, image, etc. **Transclusion** is native: `{{SomeTiddler}}` embeds another tiddler;
`{{SomeTiddler!!field}}` embeds a field.
## 3. The dual substrate — single-file vs file-per-tiddler
TiddlyWiki on **Node.js** stores each tiddler as a **separate `.tid` file** on disk: a small
text file with a header of `field: value` lines, a blank line, then the body. The Node
server assembles these into the same wiki at serve time. This substrate is:
- **git-diffable and fine-grained** — one file per tiddler, line-level diffs, per-tiddler
history — the *opposite* end of the granularity spectrum from the single HTML file.
- the natural attach surface for a *versioned, multi-author* TiddlyWiki.
So TiddlyWiki **spans the write-granularity spectrum by substrate** (single-file = whole-file
write; Node = file-per-tiddler write), exactly as Logseq spans file/DB (UC-62) and as the
backend-swap question (UC-43) anticipates.
## 4. Native query — filter expressions
TiddlyWiki's query language is **filter expressions** over tiddler fields, e.g.
`[tag[todo]!tag[done]sort[modified]]` — a compact DSL that selects/orders tiddlers by field
and tag. Lists, tables, and dynamic views are built from filters. This is a **native-query
capability** (UC-52 tier) — less expressive than SPARQL/datalog but real, and computed over
the tiddler store.
## 5. Capability profile
| Dimension (synthesis spectrum) | TiddlyWiki (single-file) | TiddlyWiki (Node `.tid`) |
|--------------------------------|--------------------------|--------------------------|
| Attachment mode | **file-store: one HTML file** | **file-store: dir of `.tid` files** |
| Addressing granularity | tiddler (`title`) within the file | tiddler = one file |
| Content identity | **`title`** (placement-bound) | title ↔ filename |
| Structure | flat record store w/ arbitrary **fields** + tags | same |
| History | none in-file (whole-file save) | **per-file git history** |
| Merge model | whole-file replace (no merge) | git 3-way per tiddler |
| Native query | **filter expressions** | filter expressions |
| Translation | WikiText (or per-tiddler `type`: markdown/json/…) | same |
| **Write granularity** | **whole file** (the anchor) | **file per tiddler** |
| Operational envelope | trivial — a browser; no server | a Node server |
| Access grant | file access = full access | server/file perms |
| Content opacity | transparent (parse the HTML store) | transparent text |
| Provenance | created/modified fields | git + fields |
## 6. INTENT mapping
### Reinforcements
- **Graceful degradation**: a single-file TiddlyWiki is a *trivial* read-only / projection /
backup shard — parse the tiddlers out of the HTML, project pages; no server needed. The
limited-backend-still-usable principle at its simplest.
- **Markdown-first but backend-neutral**: tiddlers carry a `type`, so Markdown tiddlers
coexist with WikiText — the page model's content-type field maps directly.
- **Typed fields** (UC-34): arbitrary tiddler fields are a flexible record model the page
model already accommodates.
- **Backend-swap under stable identity** (UC-43): single-file ↔ Node `.tid` is the same
logical wiki on two substrates — the migration UC-43 anticipates, within one engine.
### Divergences (boundaries / notes)
- **Whole-file write granularity** is a real constraint: in single-file mode shard-wiki
cannot write one page atomically — an overlay applied "to one page" still **rewrites the
whole file** (T11). This is the coarsest write tier; model it explicitly so overlays/locks
account for it (a write to any page conflicts with any concurrent write).
- **Identity = title**, file-local; cross-shard identity (T16) layered above.
- **The app is in the file**: when parsing a single-file TiddlyWiki, shard-wiki must extract
the **tiddler store** and ignore the embedded engine — i.e. treat the HTML as a *container
format*, not as page content (don't mistake the app for content).
### What to keep
1. **Single-file self-contained wiki as a first-class file-store shard** — container-format
parse, whole-file write granularity (UC-78); the portability/granularity extreme.
2. **Whole-file write granularity as a named tier** (T11) with overlay/lock implications.
3. **Dual-substrate binding** (single-file vs `.tid` dir) as another instance of
substrate-choice under one identity (UC-43/UC-62).
## 7. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-78 | Attach a **single-file self-contained wiki** (TiddlyWiki HTML) as one shard — parse tiddlers out of the container, project pages; **write = rewrite the whole file** (whole-file write granularity, the coarsest tier) | **new** |
| — | whole-file write granularity anchor + overlay/lock implications | enrich **UC-35** |
| — | single HTML file as a file-store shard (container format) | enrich **UC-40** |
| — | tiddler arbitrary fields = flexible record | enrich **UC-34** |
| — | filter expressions as a native-query tier | enrich **UC-52** |
| — | single-file ↔ Node `.tid` substrate swap | enrich **UC-43** |
## 8. Architecture notes for SHARD-WP-0002
- **T11 (capability / write granularity):** confirm **whole-file** as the coarsest named
write tier (anchored by single-file TiddlyWiki), with the implication that an overlay to
*any* page conflicts with concurrent writes (no per-page atomicity). File-per-tiddler is
the fine tier on the same engine.
- **T14 (attach binding):** a single-file wiki binds as a **container-format file-store**
(parse tiddler store, ignore embedded engine); a Node TiddlyWiki binds as a **dir of
`.tid` files** (git-diffable). One engine, two bindings — parameterize like UC-43.
- **Native query:** filter expressions are a low-mid native-query tier between "none" and
datalog/SPARQL — delegate where present (UC-52).
## 9. Open questions
1. In single-file mode, how does shard-wiki represent **per-page overlays** when writes are
whole-file — buffer overlays and re-serialize, or require the Node `.tid` substrate for
write-through and treat single-file as read/projection/backup only?
2. Is a single-file TiddlyWiki's **embedded plugins/config** ever relevant to the union, or
strictly ignored as app-internals (parse only content tiddlers)?
3. Does shard-wiki expose tiddler **filter expressions** as a delegated query, or only its
own union query over projected tiddlers?
## 10. Sources
- TiddlyWiki.com — *Tiddlers*, *TiddlerFields*, *Filters*, *Saving*, *Node.js* docs
- *TiddlyWiki5* GitHub (Jermolene/TiddlyWiki5) — `.tid` file format, store structure
- prior: `research/260614-logseq-deep-dive/` (file/DB dual substrate, UC-62)
## 11. Traceability
New UC **UC-78** carries the marker **⊡** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-35, UC-40, UC-34, UC-52, UC-43. Architecture
cross-refs: SHARD-WP-0002 T11 (whole-file tier), T14 (dual binding).

View File

@@ -0,0 +1,53 @@
# 260614 — Trilium (TriliumNext) deep dive (note cloning, attribute inheritance, HTML-native)
Date: 2026-06-14
## What this is
A focused study of **Trilium / TriliumNext** — the open-source hierarchical personal
knowledge base — read through shard-wiki's lens. It is another SQLite-local note app, but
it brings a structural feature **none of the prior twelve systems had: note cloning**
a single note can sit in **multiple places in the tree at once**, so the hierarchy is a
**DAG, not a tree**, with **note identity cleanly separated from placement** (a note has
many "branches"). It also has an **attribute system with inheritance + templates**
(computed metadata) and is one of the few **HTML-native** (not Markdown) tools.
Lineage: **TriliumNext** is the community fork of the original **Trilium** (zadam, now
maintenance-only) — the same pattern as TWiki→Foswiki.
Distinctive material:
- **Architecture** — single **SQLite** file (`document.db`); desktop or self-host server;
multi-instance **sync protocol w/ conflict resolution** + WebSocket; 12-char IDs
(`noteId`/`branchId`/`attributeId`/`attachmentId`)
- **Note cloning** — note vs **branch** (identity vs placement); DAG hierarchy; no single
canonical path
- **Attributes** — labels (`#tag`) + typed relations (`~relation`), **inheritable** +
**templates** (`~template`) → effective vs own metadata
- **Content** — **HTML** (CKEditor5), plus canvas/mindmap/spreadsheet/geo/render note
types; export to MD/HTML (lossy)
- **Extension** — **scripting** (frontend/backend code notes, Script API) + **ETAPI**
(token REST); **per-note encryption** (protected notes)
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, note cloning/DAG, attributes+inheritance, HTML content, scripting/ETAPI, per-note encryption, capability profile, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-66
attach a shard with a DAG hierarchy / note cloning, identity separated from placement;
UC-67 preserve inherited/templated attributes — effective vs own metadata); UC-15/22/34/
38/42/61 enriched. Logged for `SHARD-WP-0002` (T11/T12/T14/T15/T16): DAG namespace +
identity/placement (note/branch) split, computed/inherited metadata in the page model,
**per-item** content opacity (refining the 12th spectrum), HTML as a source content model,
and scripting + ETAPI host surfaces.
**Key takeaways recorded:** Trilium's **note/branch split (identity ≠ placement)** is the
clean model for pages that live in multiple locations/shards and the namespace-level form
of the clone/reference primitive; metadata is **computed** (own + inherited + template),
not a flat bag; and **content opacity is per-item** (per-note encryption), not only
whole-shard. **Boundary:** one SQLite-local candidate shard (DAG hierarchy, HTML-native),
best attached via ETAPI; not a substrate and not the federation layer.
</content>

View File

@@ -0,0 +1,274 @@
# Findings — Trilium (TriliumNext): note cloning, attribute inheritance, HTML-native
Date: 2026-06-14
Source kind: **modern shipped product** — an open-source hierarchical PKB; a *candidate
shard* whose distinctive traits are **note cloning (a DAG hierarchy)**, an
**attribute/relation system with inheritance + templates**, and **HTML-native** content
Lens: shard-wiki — the namespace model (identity vs placement), inherited/computed
metadata, HTML translation, per-note encryption, and the scripting/ETAPI surfaces
> Why Trilium earns a dive. It is another SQLite-local note app, but it brings a
> structural feature **none of the prior twelve systems had: note cloning** — a single
> note can sit in **multiple places in the tree at once**, so the hierarchy is a **DAG,
> not a tree**, and **note identity is cleanly separated from placement** (a note has
> many "branches"). That directly challenges shard-wiki's namespace model (UC-22 assumes
> a path) and is the clone/reference primitive made concrete at the *namespace* level.
> Trilium also has an **attribute system with inheritance and templates** (effective
> metadata is computed, not just stored), and it is one of the few **HTML-native** (not
> Markdown) tools — a useful stress on "Markdown-first must degrade."
Lineage note (like TWiki→Foswiki): **TriliumNext** is the community fork of the original
**Trilium** (zadam) after it went maintenance-only; the active project is TriliumNext.
Contrast set: Joplin (SQLite-local, files-on-sync, page-level), Logseq (block-graph on
files), Notion (hosted DB, schema+relations). Trilium = **SQLite-local, DAG hierarchy,
inherited attributes, HTML content, self-host sync + ETAPI**.
---
## 1. Core architecture — one SQLite file, server-syncable
- **Storage:** a single **SQLite** file (`document.db`, via better-sqlite3) holding
notes, attachments, history, and settings. Local DB store (like Joplin; not files).
- **Clients/sync:** desktop (Electron) or a self-hosted **server**; multi-instance
**sync protocol** with **conflict resolution + entity change tracking**, plus
**WebSocket** realtime updates. Attach via the server, the ETAPI (§5), or the DB.
- **IDs:** 12-char IDs — **`noteId`, `branchId`, `attributeId`, `attachmentId`**. The
**note vs branch** split is the key idea (§2).
- **Export:** to Markdown/HTML (lossy — content is HTML, §4).
---
## 2. Note cloning — a DAG hierarchy, identity separated from placement
Notes form an arbitrarily deep tree, **but a single note can be placed in multiple
locations** — "cloning," implemented via the **branches** model:
- A **note** (`noteId`) is the content + identity. A **branch** (`branchId`) is *one
placement* of that note under a parent (with an optional **branch prefix** giving
per-location context). A note with several branches is **cloned** — it appears in
several places at once; editing it anywhere edits the one note.
- So the hierarchy is a **DAG**, not a tree, and **identity (note) is separated from
location (branch)**. There is no single canonical path for a cloned note.
This is the most shard-wiki-relevant feature in the dive:
- It breaks the assumption behind **UC-22** (resolve a page by *a* path). shard-wiki's
namespace model must allow **a page in multiple namespace locations** with no single
canonical path (UC-66).
- The **note/branch split** is a clean model shard-wiki should borrow: **page identity ≠
page placement**. A page is one entity; its locations are separate placement records —
exactly how shard-wiki should treat a page that appears under multiple paths/shards.
- It is the **clone/reference-not-copy primitive** (Xanadu clone, ZigZag clone, UC-44/45,
T16) realized at the *namespace* level: same note, many positions, one source of truth.
---
## 3. Attributes — labels + relations, with inheritance and templates
Trilium's metadata system is **attributes**:
- **Labels** (`#tag`, optionally `#key=value`) — typed metadata on a note.
- **Relations** (`~relation`) — **typed links to other notes** (a knowledge graph).
- **Inheritance** — attributes can be **inherited down the subtree** (inheritable
attributes), and **templates** (a note whose attributes/structure are applied to
instances via a `~template` relation) inject attribute sets. So a note's **effective
metadata = its own attributes + inherited + templated** — computed, not just stored.
- Promoted attributes give a form-like UI; attributes drive **search/queries** and
**scripting**.
shard-wiki read: this is structured data (UC-34) and typed relations (UC-58), but with a
**new wrinkle — inheritance/templates make metadata *computed***. Projecting such a shard
must distinguish **effective vs own** attributes (and record their provenance: own /
inherited-from / template), not flatten them (UC-67). Templates also reinforce blueprint
pages (UC-15).
---
## 4. Content model — HTML-native, many note types
- **Text notes are HTML** (WYSIWYG via **CKEditor5**), **not Markdown**. Trilium is one
of the few HTML-first tools studied. Markdown participation therefore needs **HTML↔
Markdown translation** — more tractable than Notion's block model but still **lossy**
for some constructs (CKEditor features, includes) → a fidelity-aware case bridging
UC-42 (lossless ideal) and UC-59 (lossy-with-report).
- **Note types:** text (HTML), code (CodeMirror), **canvas (Excalidraw)**, relation
maps, mind maps (Mind Elixir), spreadsheets (Univer), geo maps (Leaflet), **render
notes**, file/image, book — i.e. lots of **non-Markdown content** (UC-55) and
**script/render-generated dynamic notes** (UC-54).
---
## 5. Extension surfaces — scripting (in-app) + ETAPI (external)
Two surfaces, like Joplin:
- **Scripting (in-engine host).** **Code notes** in JS run as **frontend** (UI widgets,
buttons) or **backend** (server-side automation) scripts against a **Script API**
(create/query notes, attributes, etc.). Executable, in-app — the adapter-host path
(UC-38). "Render notes" + scripts produce **dynamic, generated content** (UC-54).
- **ETAPI (external REST API).** Trilium's public REST API (since v0.50), **token auth**
(`Bearer ETAPITOKEN`), with client libs (`trilium-py`): CRUD over notes/branches/
attributes/attachments, search, import/export — an external attach surface (UC-57)
that, unlike Joplin's localhost-only Data API, is the **designed external integration
API** for a (often self-hosted) server.
---
## 6. Security — per-note (partial) encryption
Trilium offers **strong per-note encryption** ("protected notes") unlocked in a
**protected session** (password; TOTP/OpenID for login). Crucially this is **per-note**:
a shard can hold **some encrypted and some plaintext notes at once**. This refines the
**content-opacity** dimension (UC-61, proposed twelfth spectrum from Joplin/Anytype):
opacity is **per-item, not only whole-shard** — the adapter must handle a shard where
*part* of the content is opaque without a key (protected notes projectable only as
structure-shell; unprotected notes fully).
---
## 7. Trilium as a shard — capability profile
| Capability | Trilium | Notes for the adapter contract |
|------------|---------|--------------------------------|
| Read | yes (ETAPI / sync / DB) | ETAPI REST (server) is the clean surface |
| Write | yes (ETAPI / script) | per-note; via ETAPI or backend code notes |
| Write granularity | per-note (page) | — |
| Identity / addressing | **`noteId` + `branchId`** (12-char) | **identity ≠ placement**; cloned notes have many branches (UC-51, UC-66) |
| Hierarchy | **DAG (cloning)** | a note in multiple locations; no single canonical path (UC-66) |
| Structure | **labels + relations, inherited + templated** | effective vs own metadata (UC-34, UC-58, UC-67) |
| Content | **HTML (CKEditor)** + many types | HTML↔MD lossy translation (UC-42/59); non-MD types (UC-55) |
| History | internal revisions in the DB | not portable git → supplement (UC-36) |
| Native query | attribute search / script queries | delegate or build index (UC-52/63) |
| Subscribe | WebSocket + sync protocol | push; conflict resolution built in (UC-31) |
| Content opacity | **per-note encryption (partial)** | per-item opacity (UC-61 refined) |
| Extension | **scripting (in-app) + ETAPI (external)** | dual surface (UC-38, UC-57) |
| Templates | yes (`~template`) | blueprint pages (UC-15) |
Verdict: a capable **SQLite-local, server-syncable** shard best attached via **ETAPI**.
Standout demands: the **DAG hierarchy / note-branch identity model** (UC-66) and
**inherited/templated attributes** (UC-67); plus HTML-native translation, per-note
opacity, and the scripting host.
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **Identity ≠ placement** (note/branch) is a model shard-wiki should adopt for pages
that appear under multiple paths or in multiple shards — provenance and union without
erasure depend on separating "what a page is" from "where it sits."
- **Typed relations + templates** validate the structured/relation demand (UC-58) and
blueprints (UC-15) on a local-first, self-hostable backend.
- **ETAPI** is a clean example of a **designed external REST surface** for a self-hosted
server (cf. Notion external API, but self-hosted/trust-boundary friendly).
### 8.2 Deliberate divergences (design bugs if conflated)
1. **Don't force a single canonical path.** A cloned note has many placements; modeling
it with one path loses information (UC-66). Use the note/branch separation.
2. **Don't flatten computed metadata.** Effective attributes include inherited/templated
values; record provenance (own vs inherited vs template), don't collapse (UC-67).
3. **HTML-native, not Markdown.** Translate HTML↔Markdown with a fidelity report; degrade
to read-only where lossy (UC-42/59/03). Markdown-first must degrade gracefully.
4. **Per-note opacity.** Some notes are encrypted; never surface protected-note
ciphertext; project them as structure-shell (UC-61).
5. **One Trilium instance = one shard**, not the federation layer; its sync protocol is
its own — attach via ETAPI/replica, don't re-drive it (not-a-sync-daemon).
### 8.3 What Trilium teaches that shard-wiki should keep
- **Separate page identity from placement** (note vs branch) — the cleanest model for
multi-location / multi-shard pages and for the clone/reference primitive (T16).
- **Model metadata as computed** (own + inherited + templated) with per-attribute
provenance (UC-67) — not a flat key/value bag.
- **Content opacity is per-item**, not only whole-shard (UC-61 refinement).
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-65**. New UCs **UC-66, UC-67** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a shard with a DAG hierarchy / note cloning** — a page may occupy multiple namespace locations at once (note identity separated from placement via branches); no single canonical path | **UC-66 (new)** |
| **Preserve inherited / templated attributes** — project a structured shard whose metadata is computed (own + inherited + template), distinguishing effective vs own with per-attribute provenance | **UC-67 (new)** |
| DAG/multi-parent vs tree; note/branch (identity vs placement) | **enriches UC-22** |
| Labels (`#tag`) + typed relations (`~relation`); HTML-native | **enriches UC-34** |
| Templates (`~template`) inject attribute sets | **enriches UC-15** |
| HTML (CKEditor) content → HTML↔Markdown lossy translation | **enriches UC-42** (links UC-59) |
| Per-note encryption (protected notes) = **partial** content opacity | **enriches UC-61** |
| Scripting (frontend/backend code notes, Script API) = in-app host | **enriches UC-38** |
| ETAPI (token REST) = designed external surface for a self-host server | links UC-57 |
| Render/script notes + attribute queries = dynamic content | links UC-54; typed relations link UC-58 |
| `noteId`/`branchId` 12-char IDs | links UC-51 |
---
## 10. Architecture notes for SHARD-WP-0002 (no UC)
- **Namespace model must support a DAG** and **separate page identity from placement**
(note/branch). A page is one entity with N placements (paths/shards). Feeds the page/
namespace model and the clone/reference primitive. (T12, T16.)
- **Computed/inherited metadata** (UC-67): the page model's structured-data
representation must carry **effective vs own** with per-attribute provenance
(own/inherited/template), not a flat bag. (T12.)
- **Content opacity is per-item** (UC-61 refinement): the content-opacity capability
(proposed twelfth spectrum) should be **granular** (per-note), not only whole-shard.
(T11.)
- **HTML as a source content model** joins TML/Notion-blocks in the translation
capability (HTML↔Markdown, lossy-aware). (T15.)
- **Scripting as an in-engine host** + **ETAPI external REST** are two more adapter-host
exemplars (with Roam/Obsidian/Joplin). (T14.)
---
## 11. Open questions (for spec / workplans)
1. How does shard-wiki represent a **cloned note** in the union — one page with multiple
path placements, or a page transcluded into multiple locations? (UC-66 vs UC-44/45.)
2. When projecting **inherited attributes** (UC-67), does shard-wiki materialize
effective values (snapshot) or compute them live from the shard's tree/templates?
3. Is **HTML↔Markdown** round-trip lossless enough for write-back overlays, or are
Trilium overlays read-only/native-HTML (cf. UC-42 Q2)?
4. For **per-note encryption** (UC-61), is a partially-opaque shard projected with
protected notes as visible-but-opaque placeholders, or hidden entirely?
---
## 12. Sources
| Source | Used for |
|--------|----------|
| TriliumNext/Trilium — DeepWiki (https://deepwiki.com/TriliumNext/Trilium) | SQLite `document.db`/better-sqlite3; tree + cloning via branches; note types; sync |
| Cloning notes — TriliumNext wiki (https://github.com/TriliumNext/Trilium/wiki/Cloning-notes) | Note in multiple locations; branches; branch prefixes |
| TriliumNext — DeepWiki API & Synchronization (https://deepwiki.com/TriliumNext/Trilium/4.3-api-and-synchronization) | WebSocket + sync protocol w/ conflict resolution; 12-char IDs (note/branch/attribute/attachment) |
| ETAPI (REST API) — Trilium docs (https://docs.triliumnotes.org/User%20Guide/User%20Guide/Advanced%20Usage/ETAPI%20(REST%20API)/) | Public REST API since v0.50; token/Bearer auth; trilium-py |
| Script API — TriliumNext wiki (https://github.com/TriliumNext/Trilium/wiki/Script-API) | Frontend/backend code notes; Script API |
| BrightCoding — TriliumNext overview (https://www.blog.brightcoding.dev/2025/09/20/triliumnext-notes...) | Attributes (labels/relations), inheritance, templates, per-note encryption, note types |
Cross-references: `research/260614-joplin-deep-dive/findings.md` (SQLite-local, dual
surface, content opacity), `research/260614-notion-deep-dive/findings.md` (typed
relations, external API), `research/260614-zigzag-deep-dive/findings.md` (clone /
dimensions), `research/260614-shard-spectrum-synthesis/findings.md` (spectra this
refines), `spec/UseCaseCatalog.md` (UC-15, UC-22, UC-34, UC-38, UC-42, UC-51, UC-54,
UC-57, UC-58, UC-61), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T12,
T14, T15, T16).
---
## 13. Traceability
- New UCs: **UC-66, UC-67**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-15, UC-22, UC-34, UC-38, UC-42, UC-61** (links UC-51, UC-54, UC-57,
UC-58, UC-59).
- Architecture (no UC): DAG namespace + identity/placement (note/branch) split;
computed/inherited metadata; per-item content opacity; HTML source model; scripting +
ETAPI host surfaces → `SHARD-WP-0002` (T11, T12, T14, T15, T16).
- Boundary recorded: Trilium (TriliumNext) is **one SQLite-local candidate shard** with a
DAG hierarchy and computed metadata, best attached via **ETAPI**; HTML-native (lossy to
Markdown), per-note opacity; not a substrate, not the federation layer (INTENT
graceful-degradation, no-silent-mutation, not-a-sync-daemon).
</content>

View File

@@ -0,0 +1,14 @@
# 260614 — UseModWiki deep dive
Deep dive on **UseModWiki**: the **flat-file ancestor** — a Perl CGI wiki (Clifford Adams,
2000) descended from AtisWiki/CvWiki, **CamelCase** linking, plain flat-file page + history
storage, and the **engine Wikipedia originally ran on** (MediaWiki Phase I). Origins/lineage
value: the minimal flat-file page+history model the whole field descends from.
- `findings.md` — the model, lineage, capability profile, INTENT mapping, enrichments (no new
UC — reinforces the minimal flat-file baseline UC-82), architecture notes, sources,
traceability.
Catalog yield: **enrichment-only** (reinforces UC-82) — enriched UC-01 (open wiki), UC-40
(flat-file store), UC-25 (CamelCase naming), UC-36/41 (flat-file history). Lineage noted for
the origins record. Feeds SHARD-WP-0002 T11 (minimal profile lineage).

View File

@@ -0,0 +1,87 @@
# UseModWiki — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T9 · **Subject:** UseModWiki, Clifford
Adams's flat-file Perl wiki — the lineage root of much of the field.
## Why this dive
This is a **lineage** dive, not a new-capability one. UseModWiki is the **ancestor**: the
minimal flat-file page+history wiki the c2-era engines and **MediaWiki Phase I** descend
from (Wikipedia ran on UseModWiki, 20012002). It pairs with
`research/260608-c2-wiki-origins/` to record *where the page+history model came from*. It
adds no new shard capability beyond the minimal flat-file floor (UC-82, Oddmuse) — its value
is **historical grounding** and confirming the floor is genuinely the field's common root.
## 1. The model
- **Perl CGI** (`wiki.pl`), single-script, descended from AtisWiki → CvWiki → UseModWiki
(c. 2000), by Clifford Adams.
- **Flat-file storage**: each page stored as a text file (under a `db/` data directory),
with **page history** kept (older revisions retained as files / in the page record). No
database.
- **Linking**: **CamelCase** (`WikiWord`) auto-links by default; later added **free links**
`[[Like This]]`. This is the canonical CamelCase-naming lineage (UC-25).
- **Open editing**, recent-changes, simple markup — the c2 ethos in a portable script.
## 2. Lineage
- **MediaWiki Phase I** *was* UseModWiki — early Wikipedia ran on it before the PHP rewrite
(Phase II/III). So UseModWiki is the **direct ancestor of MediaWiki/Wikibase** (T2) and a
sibling-root of the c2 tradition.
- The **flat-file page+history model** here is the shape Oddmuse (UC-82), TWiki's file store
(UC-40), and others elaborate — confirming the **minimal file-store floor** is the field's
common origin, not a modern simplification.
## 3. Capability profile
Essentially **identical to the minimal floor** (UC-82, Oddmuse):
| Dimension | UseModWiki |
|-----------|------------|
| Attachment mode | **file-store** (flat files under `db/`); CGI, no API |
| Addressing | page = file; **CamelCase** name = identity |
| Structure | flat page space; CamelCase link graph |
| History | flat-file retained revisions (may be limited) |
| Native query | none |
| Translation | simple wiki markup → Markdown (lossy) |
| Write granularity | page (file) |
| Access | open editing (optional admin password) |
| Provenance | timestamp, optional username |
## 4. INTENT mapping
- **Reinforces the minimal flat-file baseline** (UC-82): UseModWiki is the *historical*
instance of the graceful-degradation floor — attach via its flat files; partial history
surfaced honestly.
- **CamelCase naming** (UC-25): the canonical source of WikiWord auto-linking — the page
model's name/identity and link-resolution must accommodate CamelCase-derived identities.
- **Open wiki** (UC-01): the c2 open-editing ethos at the root.
- **Lineage grounding**: confirms shard-wiki's "Git-based Markdown" page model descends from
(and must remain attach-compatible with) the flat-file ancestor.
**No new UC** — UseModWiki adds historical grounding, not a new orchestration scenario;
its capabilities are subsumed by UC-82 (minimal flat-file baseline) plus UC-25 (CamelCase).
## 5. Architecture notes for SHARD-WP-0002
- **T11:** UseModWiki = a second instance of the **minimal/floor profile** (with Oddmuse);
confirms the floor is the field's common root, so the floor profile is the right baseline.
- **Naming/identity:** CamelCase-derived page identities (UC-25) are part of the legacy
identity surface the adapter must parse (and translate to the page model's identity).
## 6. Open questions
(None new — covered by UC-82 partial-history honesty, catalog open-Q30, and the c2 origins
record. CamelCase resolution is UC-25.)
## 7. Sources
- UseModWiki — usemod.com/cgi-bin/wiki.pl (the wiki about itself); Wikipedia: *UseModWiki*,
*MediaWiki* (Phase I history)
- prior: `research/260608-c2-wiki-origins/`; `research/260614-oddmuse-deep-dive/` (minimal
floor, UC-82)
## 8. Traceability
**No new UC** (reinforces UC-82). Enriched: UC-01, UC-40, UC-25, UC-36, UC-41. Lineage noted
for the origins record. Architecture cross-ref: SHARD-WP-0002 T11 (minimal-profile lineage).

View File

@@ -0,0 +1,16 @@
# 260614 — Wikibase / Wikidata deep dive
Deep dive on **Wikibase** (the MediaWiki extension behind **Wikidata**) as the
**entity-statement (RDF) far end** of the structure and native-query spectra: items and
properties, statements = claim + qualifiers + references + rank, stable opaque IDs, and
**SPARQL** (Wikidata Query Service / Blazegraph) including **federated** queries.
- `findings.md` — data model, RDF/SPARQL surface, storage & history, capability profile,
INTENT mapping, UC seeds (UC-73UC-75), architecture notes for SHARD-WP-0002, open
questions, sources, traceability.
Catalog yield: UC-73 (attach a typed entity-statement / RDF shard, lossy render to a page),
UC-74 (graph query the union via SPARQL + federated `SERVICE` cross-endpoint query), UC-75
(statement-level provenance — references + rank per assertion). Enriched UC-34/58/52/24.
Feeds SHARD-WP-0002 T12 (structured/typed page model) and T16 (stable language-neutral
identity ≠ label/placement).

View File

@@ -0,0 +1,200 @@
# Wikibase / Wikidata — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T2 · **Subject:** Wikibase (MediaWiki
extension) and its flagship instance **Wikidata**, incl. the Wikidata Query Service (SPARQL).
## Why this dive
Every structured shard so far tops out at *typed records in a database*: Notion's
database-pages, XWiki's XObjects/classes, Trilium's typed relations, Roam/Logseq's
attribute blocks. Wikibase is a *different kind of structure altogether* — a **typed
knowledge graph of entities and provenance-bearing statements**, queried with **SPARQL**
over an **RDF** projection. It is the **far end of the structure spectrum** and of the
**native-query spectrum**, and it pushes **provenance down to the individual assertion**.
The question for shard-wiki: what does a shard look like when its "page" is *not prose but a
set of statements*, and what does the page model / adapter contract owe such a shard?
## 1. The data model — entities, statements, snaks
**Entities** are the top-level objects, each on its own MediaWiki page with a **stable
opaque ID**:
- **Item** — `Q42`. Has multilingual **labels / descriptions / aliases**, a set of
**statements**, and **sitelinks** (links to wiki articles). The label is *annotation*,
not identity — `Q42` is the identity, "Douglas Adams" is just its English label.
- **Property** — `P31` ("instance of"). Also has labels/descriptions/aliases, plus a
**fixed datatype** constraining its values (item-reference, string, time,
globe-coordinate, quantity, monolingual-text, url, external-id, commons-media, …).
**Statement** = the unit of assertion on an item. Structure:
```
statement = claim + references[] + rank
claim = mainSnak + qualifiers[]
snak = property + snaktype + (value) # snaktype ∈ value | somevalue | novalue
```
- **Main snak** — the core property→value assertion (e.g. `P31``Q5` "human").
- **Qualifiers** — snaks that *contextualize* the claim without being the subject (validity
time, "as of", determination method, units). E.g. *population (P1082) = 8.4M, **point in
time (P585) = 2020***.
- **References** — lists of snaks citing **where the claim comes from** (a source item, a
URL, a page number). **Provenance attached to the individual statement, not the page.**
- **Rank** — `preferred` | `normal` | `deprecated`: relative importance among same-property
statements (lets multiple, even contradictory, values coexist with a curation signal —
the structured analogue of fedwiki's "chorus").
- Each statement carries a **stable GUID** (`Q42$<uuid>`), so statements are individually
addressable.
`somevalue` (known to exist, value unknown) and `novalue` (known *not* to have a value) are
**first-class** — the model represents *known-unknowns* explicitly, which prose and most
DBs cannot.
## 2. The RDF / SPARQL surface
Wikibase **projects entities to RDF**; the **Wikidata Query Service (WDQS)** is a
**Blazegraph** triple store exposing a **SPARQL** endpoint. The projection is deliberately
multi-layered:
- **Truthy** triples (`wdt:` prefix) — the simple "best" value, for easy queries:
`wd:Q42 wdt:P31 wd:Q5`.
- **Full** statements — reified so qualifiers/references/rank survive: `wd:Q42 p:P31
?stmt . ?stmt ps:P31 wd:Q5 ; pq:P585 ?time ; prov:wasDerivedFrom ?ref`. (`p:`=statement
node, `ps:`=statement value, `pq:`=qualifier, `pr:`/`prov:`=reference.)
- **Federated SPARQL** — the `SERVICE <endpoint> { … }` keyword runs a sub-query against
*another* SPARQL endpoint and joins the results. **Query-level federation is built into
the query language** — a different federation primitive from fedwiki's fork/neighborhood.
- **EntitySchemas / ShEx** — schemas (`E`-ids) that *validate* an entity's shape (Shape
Expressions). Optional, declarative structure validation over the open graph.
## 3. Storage, identity, history
- **Storage:** each entity is a **JSON blob stored as a MediaWiki page** (`Item:` /
`Property:` content model). The RDF/SPARQL store is a **derived index** rebuilt from these
canonical JSON entities (an *update stream* feeds WDQS) — exactly shard-wiki's
"derived query index over a canonical store" pattern (UC-63), at planet scale.
- **Identity:** the **opaque Q/P/L IDs are the identity**, fully decoupled from
human-readable labels and from language. This is the cleanest real-world instance of
**stable, language-neutral identity ≠ label/placement** — a strong reinforcement of our
identity model (T16).
- **History:** because each entity is one MediaWiki page, history is **page-level MediaWiki
revisions** — every edit is a full-entity JSON snapshot with author/timestamp/comment.
*Coarse* history granularity (whole entity per revision), but the **edit API is
fine-grained** (`wbsetclaim`, `wbeditentity` patch individual statements). So: **fine
write API over a coarse history unit** — a distinct point on the write/history spectra.
## 4. Capability profile
| Dimension (synthesis spectrum) | Wikibase / Wikidata |
|--------------------------------|---------------------|
| Attachment mode | **external-API** (MediaWiki Action API + REST) **and** a derived **SPARQL endpoint**; self-hostable |
| Addressing granularity | **statement** (each has a GUID) within an **entity** (Q/P id) |
| Content identity | **stable opaque ID** (Q/P/L); labels are multilingual annotations |
| Identity vs placement | **fully separated** — identity is language- and label-neutral |
| Structure | **typed knowledge graph**: entities + statements (claim+qualifiers+refs+rank) |
| History | **page-level revisions** (whole-entity JSON snapshots); fine-grained edit API |
| Merge model | MediaWiki last-writer / edit-conflict; rank lets contradictory values coexist |
| Native query | **SPARQL** (RDF) + **federated `SERVICE`** cross-endpoint join — the far end |
| Translation | **not Markdown** — content *is* statements; render to prose is a lossy projection |
| Attachment/write granularity | **statement-level writes** via API; coarse history unit |
| Operational envelope | huge derived index (Blazegraph), rate-limited public endpoints |
| Access grant | open read; MediaWiki user/permission model for write; self-host = own ACL |
| Content opacity | transparent (public JSON + RDF); not encrypted |
| Provenance | **statement-level** — references + rank per assertion (new far end) |
## 5. INTENT mapping
### Reinforcements
- **Stable identity ≠ placement** (T16): Q/P IDs decoupled from labels/language are the
textbook case — adopt the principle that a page's *identity* is an opaque stable handle,
display names are annotations.
- **Derived index over canonical store** (UC-63): WDQS is exactly a SPARQL index rebuilt
from canonical JSON entities via an update stream — validates the projection pattern.
- **Union without erasure / chorus**: **rank** lets multiple (even contradictory) statements
coexist with a curation signal rather than forcing one truth — the *structured* analogue
of fedwiki's chorus (UC-72) and our "view multiple versions" (UC-27).
- **Mechanism over policy**: references + rank are *mechanism* for representing disagreement
and sourcing; which statement "wins" is left to the consumer/query.
### Divergences (boundaries / design notes)
- **Content is not Markdown.** A Wikibase "page" is a set of statements; there is no prose
body. This is the **structure far-end**: shard-wiki must either (a) treat such a shard as
a **structured/typed shard** projected to a *lossy* Markdown/table rendering (UC-55/UC-73),
or (b) model a page whose payload is typed statements (T12). Forcing it into Markdown-first
erases the graph — a design-bug if done silently; render-with-provenance instead.
- **Provenance granularity is finer than ours.** Our provenance is per-page/per-shard;
Wikibase is **per-statement** (references) and even per-value (rank). The page model and
coordination journal should *allow* sub-page provenance (UC-75) even if MVP records it per
page.
- **Query is graph, not text/datalog.** SPARQL over RDF (with federated `SERVICE`) is a
richer query far-end than Roam/Logseq datalog or Notion filters (UC-52) — and its
`SERVICE` federation is a *query-time* cross-shard join, distinct from fedwiki structural
federation. Note both as native-query tiers.
### What to keep
1. **Opaque stable identity, labels-as-annotations** as the identity model (T16).
2. **Statement/assertion-level provenance** (references) and a **coexistence-with-rank**
model as the structured form of union-without-erasure (UC-75).
3. **Derived SPARQL/graph index over a canonical entity store** as a projection pattern
(UC-63/UC-74), incl. **federated query** as a first-class federation mode.
4. A **typed-graph page payload** option in the page model (T12), with **lossy
render-to-Markdown** as the projection (never silent flattening).
## 6. UC seeds
| # | Seed | Disposition |
|---|------|-------------|
| UC-73 | Attach a **Wikibase** as a **typed entity-statement (RDF) shard** (items/properties/statements w/ qualifiers); project to a rendered page view, lossy to Markdown, preserving the graph | **new** |
| UC-74 | **Graph-query the union** via **SPARQL** and **federate queries across endpoints** (`SERVICE`) — graph query as a native-query tier + query-time cross-shard join | **new** |
| UC-75 | Preserve **statement-level provenance** — references + rank attached to each assertion (sub-page provenance granularity) | **new** |
| — | typed records → typed *graph* entities | enrich **UC-34** |
| — | inter-record relations → typed graph edges with qualifiers | enrich **UC-58** |
| — | native query → SPARQL/RDF + federated SERVICE | enrich **UC-52** |
| — | provenance → statement/assertion granularity | enrich **UC-24** |
## 7. Architecture notes for SHARD-WP-0002
- **T12 (structured/typed page model):** add a **typed-graph payload** tier above
typed-records — a page whose content is **entities + statements (claim + qualifiers +
references + rank)**, with `somevalue`/`novalue` known-unknowns. Render-to-Markdown is a
**lossy projection**, not the canonical form.
- **T16 (identity / addressing):** adopt **opaque stable identity with labels-as-annotation**
(Q/P model); record **statement GUIDs** as an example of *sub-page addressable units*.
- **Native-query tiering:** SPARQL/RDF + federated `SERVICE` is the **graph far-end** of the
query spectrum (above datalog/filters); `SERVICE` is also a **query-time federation**
mode to sit beside fedwiki's structural federation.
- **Provenance model:** allow **per-statement references + rank** (sub-page provenance,
coexistence-with-curation) in the union, even if MVP collapses to per-page.
- **Derived index:** WDQS = canonical JSON entities → update stream → Blazegraph SPARQL
index; the reference implementation of UC-63 at scale (per-shard or core-built index, Q16).
## 8. Open questions
1. Does shard-wiki model a **typed-graph page** natively (T12), or always treat Wikibase as
a structured shard **projected to a Markdown/table rendering** (UC-55), or both
(canonical graph + lossy view)?
2. Is **SPARQL/graph query** exposed as a union-level capability (translate to a common
query layer) or only as a **pass-through** to graph-capable shards? How does federated
`SERVICE` relate to shard-wiki's own cross-shard query?
3. At what granularity does the coordination journal record **provenance** — per page
(MVP), per statement (Wikibase-native), or configurable?
4. Is **rank** (coexisting contradictory values w/ curation) representable in the union as a
first-class "chorus of statements," unifying with fedwiki's page-level chorus (UC-72/27)?
## 9. Sources
- Wikibase/DataModel and **DataModel/Primer** — mediawiki.org
- Help:Qualifiers; Wikidata SPARQL query service + Query Help; SPARQL tutorial — wikidata.org
- Wikidata Query Service / User Manual — mediawiki.org; Wikitech (Blazegraph, updater)
- "The wikibase model" — Vanderbilt Libraries Digital Lab (heardlibrary.github.io)
- RaiseWikibase — Wikibase Data Model functions (ub-mannheim.github.io)
- WShEx / EntitySchemas (ShEx) — arxiv.org/abs/2208.02697, ceur-ws.org Vol-3262
## 10. Traceability
New UCs **UC-73UC-75** carry the marker **⬡** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-34, UC-58, UC-52, UC-24. Architecture cross-refs:
SHARD-WP-0002 T12, T16, native-query tiering, provenance model, UC-63 derived index.

View File

@@ -0,0 +1,49 @@
# 260614 — Wiki.js deep dive (the storage-module engine that already speaks Git + Markdown)
Date: 2026-06-14
## What this is
A focused study of **Wiki.js** — the modern Node.js server wiki — read through
shard-wiki's lens. Wiki.js is the **closest existing engine to shard-wiki's own design**:
it is DB-canonical (Postgres/MySQL/SQLite) **but abstracts persistence behind pluggable
"storage modules"** — Git, local filesystem, S3, Azure Blob — each acting as **backup or
source of truth**, with the **Git module bidirectionally syncing clean `.md` files (+ YAML
frontmatter)** to a repo. That storage-module interface is, in effect, a shipped version
of shard-wiki's adapter contract — the second concrete prior art after **Foswiki::Store**,
and the closer one (medium = Markdown in Git).
Distinctive material:
- **Architecture** — Node.js/Vue; **GraphQL API** over all resources; DB-canonical;
Markdown (primary) + HTML + AsciiDoc; path-based hierarchy; modular
(storage/auth/search/render/editor)
- **Storage modules** — provider connector (Git/FS/S3/Azure) with properties + create/
update/delete/sync methods, **backup-or-source-of-truth**, `injectMetadata()` YAML
frontmatter; **Git module commits every change → version-controlled clean Markdown**,
bidirectional
- **GraphQL** — typed, **introspectable** (capability/schema discovery), **selective-field**
projection
- **Auth modules** (delegated authn) + **path-based rule ACL**
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, storage modules (≈ adapter contract), Git mirror as attach+write surface, GraphQL, access control, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-68
attach an engine-maintained bidirectional Git mirror of clean Markdown, incl. write-by-
commit; UC-69 attach via a typed, introspectable API — GraphQL — for schema discovery +
selective projection); UC-06/36/38/40/42/57 enriched. Logged for `SHARD-WP-0002`
(T11/T14): the **storage-module abstraction as a second adapter-contract prior art** (with
Foswiki::Store), the **engine-maintained Git mirror** as preferred attach+write surface,
and **GraphQL introspection** for capability discovery + selective projection.
**Key takeaway recorded:** Wiki.js is the most shard-wiki-shaped engine yet — its
storage-module interface is near-isomorphic to the adapter contract, and its
Markdown-in-Git mirror is the ideal file-store attach (clean MD, git history,
write-by-commit). **Boundary:** one server-engine candidate shard, DB-canonical with a
maintained Git mirror; don't double-sync; honor its access rules; not the federation layer.
</content>

View File

@@ -0,0 +1,249 @@
# Findings — Wiki.js: the storage-module engine that already speaks Git + Markdown
Date: 2026-06-14
Source kind: **modern shipped product** — a server wiki engine; a *candidate shard* and
the closest existing engine to shard-wiki's own design (DB-canonical, but with a
**pluggable storage-module abstraction** that bidirectionally syncs clean Markdown to
**Git**/FS/S3/Azure)
Lens: shard-wiki — storage-module-as-adapter-contract prior art, the engine-maintained
Git mirror as an attach + write surface, GraphQL as a typed/introspectable API, and
path-based access rules
> Why Wiki.js is special in this set. Every prior engine/tool stored content in *one*
> substrate and exposed *one or two* surfaces. Wiki.js is DB-canonical (Postgres/MySQL/
> SQLite) **but abstracts persistence behind pluggable "storage modules"** — Git, local
> filesystem, S3, Azure Blob — each of which can act as **backup *or* source of truth**,
> and the **Git module bidirectionally syncs clean `.md` files (with YAML frontmatter)**
> to a repo. In other words, Wiki.js already does, internally, much of what shard-wiki's
> adapter contract is meant to do: a **versioned, capability-bearing, multi-provider
> persistence interface over Markdown**. It is the second concrete prior art for the
> contract after **Foswiki::Store** — and arguably the closer one, because its content
> is **Markdown in Git**, exactly shard-wiki's native shape.
Contrast set: the engine dives (XWiki DB/components, TWiki/Foswiki file+RCS) and the
modern tools. Wiki.js = **modern server engine, DB-canonical, Git-storage-of-Markdown,
GraphQL API, pluggable modules**.
---
## 1. Core architecture
- **Stack:** Node.js backend, Vue front end; **everything is accessible via a GraphQL
API** (`/graphql`, with a Playground). The DB is **Postgres** (primary; MySQL/SQLite
supported), read-replica-capable.
- **Content:** **Markdown (primary)**, plus **HTML** and **AsciiDoc**, via pluggable
**editors** (Markdown, WYSIWYG/visual, raw HTML). Pages live on **paths** (a path-based
hierarchy) with tags.
- **Modular architecture:** typed pluggable **modules****storage**, **auth**,
**search**, **rendering**, **editor**. The contract-relevant ones are *storage* (§2)
and *auth* (§4).
- **3.0:** a large rewrite is in development; storage remains a headline subsystem.
---
## 2. Storage modules — the headline (≈ shard-wiki's adapter contract)
A **storage module** "connects Wiki.js with a local or remote storage provider, to act as
**backup or source of truth** for content. It consists of **properties** (user-set) and
**methods called on events** — create, update, delete content, and sync." Providers
include **Git, local filesystem, S3, Azure Blob**.
The **Git module** is the most-used: **every page change commits to a Git repository**,
turning the wiki into **version-controlled `.md` files**. Content is written as **clean
Markdown**, and `injectMetadata()` prepends **YAML frontmatter** (for Markdown) or HTML
comments (for HTML) carrying page metadata. The sync is **bidirectional** with the remote
repo, and access to the synced files honors Wiki.js **access rules** (§4).
Why this matters to shard-wiki — three distinct payoffs:
1. **Prior art for the adapter contract.** Wiki.js's storage-module interface is a
shipped instance of what `SHARD-WP-0002` T11 specifies: a *versioned, capability-
bearing, multi-provider* persistence abstraction, with a backup-vs-source-of-truth
choice and lifecycle methods (create/update/delete/sync). It joins **Foswiki::Store**
as concrete prior art — and is closer, because its medium is **Markdown in Git**.
2. **The ideal file-store attach.** A Wiki.js instance with the Git module **is a git
repo of clean Markdown + YAML frontmatter**, maintained by the engine. shard-wiki can
attach that repo **directly** (UC-40) — better than reading the DB — and get **git
history for free** (UC-36, *adopt* not supplement, via the mirror).
3. **A write path via Git, no API needed.** Because the Git sync is **bidirectional**,
shard-wiki can **write-through by committing Markdown** to the repo; Wiki.js **ingests
git→DB**. This is overlay/patch application as a *git commit*, not an API call (UC-68).
Caution: Wiki.js owns the DB↔git sync — **don't double-sync**, and coordinate which
side is **source of truth** (configurable).
---
## 3. GraphQL — a typed, introspectable external API
Unlike the REST/local-REST tools (Joplin, Trilium, Notion), Wiki.js exposes a **GraphQL
API** over all resources. Two shard-wiki-relevant properties:
- **Introspection = capability/schema discovery.** A GraphQL schema is self-describing;
an adapter can introspect the content/page schema rather than hard-coding it — feeding
T11's *capability discovery*.
- **Selective-field projection.** GraphQL fetches **exactly the fields a projection
needs** (page body vs. metadata vs. tags), reducing over-fetch — relevant to projection
efficiency and the operational envelope. (UC-69; an external-API sub-mode beside
Notion's REST, UC-57.)
---
## 4. Auth modules + path-based access rules
- **Auth modules** (local, LDAP, OAuth, SAML, …) — **authentication is delegated to
pluggable providers**, exactly shard-wiki's **authn-delegated** stance
([[shard-wiki-auth-in-core-decision]]).
- **Access rules** are **path-based / rule-based ACL**: allow/deny access to all or
specific sections of the wiki per group; the Git-synced files honor the same rules. For
shard-wiki this is a richer ACL than TWiki's per-topic flags (UC-06) — **rule-based by
path pattern** — and informs how a projection should **honor/surface a shard's
restricted regions** (union without erasure: show that a region is restricted, don't
silently drop or expose it).
---
## 5. Wiki.js as a shard — capability profile
| Capability | Wiki.js | Notes for the adapter contract |
|------------|---------|--------------------------------|
| Read | yes (Git mirror / GraphQL / DB) | **Git mirror = clean Markdown** is the best surface |
| Write | yes (GraphQL **or git commit**) | bidirectional Git ingest = write-by-commit (UC-68) |
| Write granularity | per-page | — |
| Identity / addressing | path (+ page id) | path-based; YAML frontmatter carries metadata/id |
| Structure | YAML frontmatter (in-file via Git module) | git-diffable in-text structure (good) |
| History | **git-native via Git module** (else DB versions) | adopt the mirror's git history (UC-36) |
| Native query | GraphQL; pluggable search (DB/Elastic/Algolia) | delegate search; GraphQL selective fetch (UC-69) |
| Translation | Markdown primary; HTML/AsciiDoc too | mostly native; multi-format = light translation (UC-42) |
| Attach modes | **file-store (engine-maintained Git mirror)** / external-API (GraphQL) | clean MD in git is the ideal file-store (UC-40, UC-68) |
| Access control | **path-based rule ACL**; delegated auth modules | authn-delegated; honor restricted regions (UC-06) |
| Storage abstraction | **pluggable storage modules (Git/FS/S3/Azure)** | **adapter-contract prior art** (T11) |
| Content types | pages + media/assets | assets via storage modules |
Verdict: **the most shard-wiki-shaped engine yet.** Best attached via its **Git storage
mirror** (clean Markdown + frontmatter + git history, with write-by-commit), or via
**GraphQL** for live/selective access. Its standout gift is **architectural**: a shipped
storage-module abstraction that is near-isomorphic to shard-wiki's adapter contract.
---
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
### 6.1 Reinforcements
- **Storage modules = the adapter contract, shipped.** A versioned, capability-bearing,
multi-provider persistence interface with backup-vs-source-of-truth — exactly T11's
shape, in production (with Foswiki::Store as the other prior art).
- **Markdown-in-Git is the native shape.** Wiki.js validates that a serious engine can
keep content as **clean `.md` + frontmatter in git** — shard-wiki's preferred medium —
while still offering a rich app over a DB.
- **authn-delegated** (auth modules) and **git coordination** (Git storage) are both core
shard-wiki stances, here shipped.
### 6.2 Deliberate divergences (design bugs if conflated)
1. **DB is Wiki.js's canonical store; the Git repo is a maintained mirror.** Treat the
mirror as the attach surface, but respect that **Wiki.js runs the DB↔git sync**
don't double-sync; coordinate source-of-truth (UC-68).
2. **Bidirectional write needs care.** Committing Markdown is a real write path, but
races with the engine's own sync; apply overlays as commits the engine can ingest
cleanly, never out-of-band DB edits (overlay before mutation, no silent mutation).
3. **One Wiki.js instance = one shard**, not the federation layer; its access rules are
the shard's, to be honored/surfaced, not re-implemented.
4. **HTML/AsciiDoc pages** need light translation; Markdown pages are native.
### 6.3 What Wiki.js teaches that shard-wiki should keep
- **Model the contract on a real storage-module interface** (Wiki.js + Foswiki::Store):
properties + lifecycle methods (create/update/delete/sync) + a **backup-vs-source-of-
truth** flag + a provider list. (T11.)
- **Prefer the engine-maintained Git mirror** as the attach surface for DB-canonical
engines that offer one — clean Markdown, git history, and write-by-commit (UC-68).
- **Use GraphQL introspection for capability discovery and selective projection** where a
shard offers it (UC-69).
---
## 7. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-67**. New UCs **UC-68, UC-69** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach an engine-maintained bidirectional Git mirror of clean Markdown** (Wiki.js Git storage module): attach the repo as a file-store shard (clean MD + YAML frontmatter + git history) and optionally **write-through by committing Markdown the engine ingests**; coordinate source-of-truth, don't double-sync | **UC-68 (new)** |
| **Attach via a typed, introspectable API (GraphQL)**: discover the shard's schema by introspection (capability discovery) and fetch only the fields a projection needs (selective projection) | **UC-69 (new)** |
| Pluggable **storage modules** (Git/FS/S3/Azure) = adapter-contract prior art; modular auth/search/render/editor | **enriches UC-38** |
| Git storage = **clean Markdown in git**, the ideal engine-maintained file-store attach | **enriches UC-40** |
| Git history natively via the storage module (**adopt** via mirror) | **enriches UC-36** |
| GraphQL external-API variant (typed/introspectable/selective) | **enriches UC-57** |
| Multi-format (Markdown primary; HTML/AsciiDoc) | **enriches UC-42** |
| Path-based rule ACL; delegated auth modules | **enriches UC-06** (links authz decision) |
| Engine-maintained clean git shard | links UC-02 |
---
## 8. Architecture notes for SHARD-WP-0002 (no UC)
- **Second adapter-contract prior art (T11).** Wiki.js's **storage-module interface**
(properties + create/update/delete/sync methods + backup-vs-source-of-truth + multi-
provider Git/FS/S3/Azure) is, with **Foswiki::Store**, the model to base the contract
on — and the closer one (medium = Markdown in Git). Cite both in T11.
- **Engine-maintained Git mirror (T14).** Add to the attachment taxonomy: for a
DB-canonical engine that offers a bidirectional Git-of-Markdown mirror, the **mirror is
the preferred attach + write surface** (clean MD, git history, write-by-commit). A
cleaner sibling of Joplin's proprietary interchange mirror (UC-60) and the file-store
native-store kind (UC-40).
- **GraphQL → capability discovery + selective projection (T11/T14).** Typed/
introspectable APIs give schema discovery (capability negotiation) and reduce over-fetch
— note alongside Notion's REST external-API mode.
- **authn-delegated + path-based ACL** reinforce the settled authz decision and inform how
projections honor/surface restricted regions (union without erasure).
---
## 9. Open questions (for spec / workplans)
1. For an engine with a bidirectional Git mirror (UC-68), is the **mirror or the DB the
source of truth** from shard-wiki's view, and how do we avoid races with the engine's
own sync (write-by-commit cadence, locking)?
2. When both a **Git mirror and GraphQL** are available, which does a binding prefer —
mirror for content/history, GraphQL for live/selective metadata? Can one binding use
both?
3. How does shard-wiki **honor/surface path-based access rules** (UC-06) in a projection
without re-implementing the shard's ACL engine?
4. Should the adapter contract's **storage-module shape** be standardized directly on
Wiki.js + Foswiki::Store, or generalized further?
---
## 10. Sources
| Source | Used for |
|--------|----------|
| Wiki.js — GraphQL API docs (https://docs.requarks.io/dev/api) | Everything via GraphQL; `/graphql` Playground; access/modify all resources |
| Wiki.js — Git storage (https://docs.requarks.io/storage/git) | Per-change commit to git; clean `.md`; bidirectional sync with remote repo |
| Wiki.js — Storage overview + dev/storage.md (https://docs.requarks.io/storage ; https://github.com/requarks/wiki-docs/blob/master/dev/storage.md) | Storage module = provider connector; backup-or-source-of-truth; properties + create/update/delete/sync methods; `injectMetadata()` YAML frontmatter; providers Git/FS/S3/Azure |
| Wiki.js 3.0 storage preview (https://beta.js.wiki/blog/2021-wiki-js-3-feature-preview-storage) | Storage subsystem direction (3.0) |
| Wikipedia — Wiki.js (https://en.wikipedia.org/wiki/Wiki.js) | Node.js/Vue, DB options, formats, modular architecture |
| BrightCoding / Grokipedia overviews | Auth providers, access rules, editors, read replicas |
Cross-references: `research/260613-foswiki-deep-dive/findings.md` (Foswiki::Store, the
other storage-interface prior art), `research/260614-joplin-deep-dive/findings.md`
(interchange/sync-mirror contrast), `research/260614-notion-deep-dive/findings.md`
(external-API/REST contrast), `research/260614-shard-spectrum-synthesis/findings.md`
(attachment-mode spectrum), `spec/UseCaseCatalog.md` (UC-02, UC-06, UC-36, UC-38, UC-40,
UC-42, UC-57), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
---
## 11. Traceability
- New UCs: **UC-68, UC-69**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-06, UC-36, UC-38, UC-40, UC-42, UC-57** (links UC-02, authz decision).
- Architecture (no UC): **storage-module abstraction = 2nd adapter-contract prior art**
(with Foswiki::Store); engine-maintained Git mirror as attach+write surface; GraphQL
introspection → capability discovery + selective projection; path-based ACL + authn-
delegated → `SHARD-WP-0002` (T11, T14).
- Boundary recorded: Wiki.js is **one server-engine candidate shard**, DB-canonical with a
**bidirectional Git-of-Markdown mirror** that is the preferred attach surface; not a
substrate and not the federation layer (don't double-sync; honor its access rules).
</content>

View File

@@ -0,0 +1,47 @@
# 260614 — Project Xanadu deep dive (the docuverse, the EDL, reference-not-copy)
Date: 2026-06-14
## What this is
A focused study of **Project Xanadu** (Ted Nelson) read through shard-wiki's lens.
Unlike the engine dives (`xwiki`, `twiki`, `foswiki`), Xanadu is **not a candidate
shard backend** — it never shipped at scale and there is nothing to attach. It is
studied as the **deepest conceptual ancestor** of shard-wiki's own model: several
shard-wiki primitives turn out to be Xanadu mechanisms under different names.
The distinctive material:
- the **EDL / xanadoc** — a document that contains *no content*, only an ordered list
of **spans** (content references) plus **xanalinks** (separate link tables); the
client assembles the page by reference → this is shard-wiki **projection** + **union**
- the storage substrate — **tumblers** (stable fine-grained addresses), **istream**
(invariant content pool), **enfilades / spanfilade** (virtual↔content mapping,
version comparison by span-set intersection)
- **content-identity, bidirectional transclusion** — content "knowably in more than
one place" that remembers all its appearances → stronger than UC-32
- **transcopyright / micropayment** — a baked-in rights policy shard-wiki must keep
*configurable*, not inherit
Goes deliberately **underneath** the surface pattern table in
`research/260608-federation-concepts/findings.md` §3.3, which it extends rather than
repeats.
## Contents
| Path | Role |
|------|------|
| `findings.md` | EDL/xanadoc, addressing substrate, transclusion, versioning, rights, INTENT mapping (reinforcements + design-bug divergences), UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
(UC-44 compose-by-reference, UC-45 reverse transclusion, UC-46 content-identity
equivalence); UC-24/27/29/32 enriched. Span-addressing as an adapter capability,
content identity, composition manifests, and reuse-terms metadata logged as
architecture for `workplans/SHARD-WP-0002-federation-architecture.md`.
Key boundary recorded: shard-wiki inherits Xanadu's **reference-not-copy** mechanisms
but **rejects** the single-global-docuverse premise, the single-canonical-instance
model, and the baked-in economic policy — those would violate shard sovereignty,
parallel-version support (UC-27), and mechanism-over-policy.
</content>

View File

@@ -0,0 +1,322 @@
# Findings — Project Xanadu: the docuverse, the EDL, and reference-not-copy
Date: 2026-06-14
Source kind: **conceptual / architectural prior art** (not a deployable engine, not a
candidate shard backend)
Lens: shard-wiki orchestration — projection, overlay, transclusion, provenance,
addressing, coordination journal
> Reading guide. Every previous deep dive (`xwiki`, `twiki`, `foswiki`) studied a
> *shippable engine* as a candidate **shard**. Xanadu is different: it is the
> **deepest conceptual ancestor** of shard-wiki's own model and is *not* a backend
> we would ever attach. It is studied here for its **mechanisms** —
> reference-not-copy documents, separated link tables, content-identity transclusion,
> stable fine-grained addressing — several of which shard-wiki already reinvents
> under different names (projection, overlay, union BackLinks, coordination journal).
> The job of this file is to (a) name what shard-wiki inherits, (b) flag what would
> be a **design bug** to inherit, and (c) extract use cases that are genuinely new
> versus the federation track already in the catalog.
Pairs with — and deliberately extends, does not repeat —
`research/260608-federation-concepts/findings.md` §3.3, which treats Xanadu only as a
six-row pattern table and (correctly) labels it "speculative design / pattern
language, not deployable federation." This dive goes underneath those patterns into
the actual data architecture (EDL, istream, enfilades, tumblers) because *that* is
where the resonance with shard-wiki's storage-neutral page model lives.
---
## 1. Origin and status
Project Xanadu (Ted Nelson, 1960 ; the term *hypertext* and *transclusion* are his)
is the original electronic-literature system: a global **docuverse** of permanent,
addressable documents joined by visible, two-way links. It famously never shipped at
scale; partial artifacts exist — **Udanax Green / Udanax Gold** (source released 1999
to establish prior art against patents), **XanaduSpace**, and **OpenXanadu** (2014, a
small working browser demo built on the EDL format). Treat all of these as
*reference designs*, not running software to integrate.
Why it matters to shard-wiki: Nelson's critique of the Web — **"one-way,
ever-breaking links and no management of version or contents"** — is precisely the
gap shard-wiki addresses for wikis. shard-wiki is, in effect, a *pragmatic, federated,
Git-backed, heterogeneous-backend* descendant of the Xanadu idea, scoped to
Markdown wiki pages and explicitly **refusing** Xanadu's one-global-store premise.
---
## 2. The headline insight — a document is a manifest of spans, not content
The single most shard-wiki-relevant Xanadu artifact is the **EDL (Edit Decision
List)**, also called a **xanadoc**:
- A xanadoc is a *connected document that contains no original content*. It is a file
listing (a) **spans** — "portions of content to bring in" addressed in remote
sources — and (b) **xanalinks** — tables saying what to connect to what.
- The client reads the EDL, **fetches each span from its source**, and **assembles**
the document locally per the xanalinks. Content is referenced, never embedded.
- Structure is strictly separated: **spans first** (content references), **xanalinks
after** (relationships). Content and link structure live apart.
This is *the same move* shard-wiki makes:
| Xanadu (xanadoc / EDL) | shard-wiki concept |
|------------------------|--------------------|
| Document = ordered list of span references, no embedded content | **Projection** — lazy, cache-like view assembled from shard storage, not a copy |
| Spans pulled from multiple remote sources at view time | **Union of pages** across heterogeneous shards |
| Xanalinks stored separately from content | **Overlay** — non-destructive edits/annotations as separate objects, "overlay before mutation" |
| "All media should be permanized and addressable" | **Coordination journal** + Git content-addressable storage |
shard-wiki's INTENT lines — *"prefer lazy projection over eager copying"* and *"union
without erasure"* — are the xanadoc principle restated for wikis.
---
## 3. Addressing and the storage substrate
Xanadu's machinery underneath the EDL:
- **Tumblers** (Miller / Gregory): a transfinite-number addressing scheme where one
address simultaneously encodes **machine, author, document version, byte span, and
links**. A single tumbler is a stable, fine-grained, hierarchical pointer into the
whole docuverse.
- **istream (invariant stream):** a growing pool of *shareable content pieces*.
Documents do not store text; they reference istream content by virtual address. A
family of **enfilades** (tree structures using *dsps* = relative displacements and
*wids* = ranges) maps **virtual addresses ↔ istream addresses** bidirectionally.
- **Spanfilade:** indexes which istream spans each document uses. Other filades:
**granfilade** (storage across disks/network), **POOMfilade** (permutation-of-order
matrix mapping document position → istream location).
- Implicit substrate: content pieces are **invariant** (append-only, never rewritten);
versions are new arrangements of shared, permanent pieces. Nothing is deleted, so
references never break.
shard-wiki mapping:
- **istream + invariance ≈ Git's content-addressable, immutable blob store**, and the
**coordination journal** as the append-only record. shard-wiki gets "references
never break" *for Git-native shards for free*; for non-Git shards it must
approximate via the journal/projection cache.
- **Tumbler ≈ the still-open question of a portable, stable, fine-grained span
address** that survives projection, overlay, and versioning across heterogeneous
backends. shard-wiki has no such addressing scheme yet; this is the hardest part
Xanadu solved on paper and the part that never shipped. (See §10, §11.)
---
## 4. Links and transclusion — separate, two-way, content-identity
- **Links are first-class objects stored apart from content** (the xanalinks tables),
and are **visible and followable from all endpoints** (rule 7) — i.e. inherently
**bidirectional**. Backlinks are not derived after the fact; they are the same
object seen from the other end.
- **Transclusion** = Nelson's "the same content *knowably* in more than one place."
The content piece **remembers its identity and can trace back to all its
appearances**. This is the crucial delta from every weak modern form (server-side
includes, HTML embeds, Roam block-refs): those *copy or cache*; Xanadu keeps one
permanent instance that is **aware of where it is reused**.
shard-wiki mapping:
- **Separate link tables ≈ overlays and union BackLinks as first-class.** An overlay
is shard-wiki's xanalink: a non-destructive object spanning content it does not own.
- **Content-identity, bidirectional transclusion** is *stronger* than UC-32 as
currently written (which is path/span fetch with freshness). Xanadu says the
*content itself* knows its appearances → enables **reverse transclusion** ("where is
this paragraph used across all shards?") and **content-identity equivalence** ("these
two pages in different shards are versions of the same content"). These are new
capabilities, surfaced as UC-45 and UC-46 below.
---
## 5. Versioning and comparison — by span-set intersection
Because a document is a set of spans over an invariant pool, **comparing two documents
or two versions = intersecting their span-sets** (the spanfilade operation). Shared
spans are literally shared subtrees; differences fall out of the set arithmetic. The
same operation surfaces links between documents.
shard-wiki mapping: this is a **content-identity diff/merge** model. shard-wiki's
diff/merge capability (an adapter capability in the contract) is today implicitly
path/title- and text-based. Xanadu shows a path-independent alternative: detect that
page A in shard X and page B in shard Y are *the same or derived content* by span
overlap, **without relying on matching titles or paths**. This directly serves UC-27
(view parallel versions of equivalent pages) by giving an *equivalence-detection
mechanism* rather than assuming naming conventions align across sovereign shards.
---
## 6. Rights and economics — transcopyright, micropayment, implicit permission
Three of the 17 rules are an economic/rights layer:
- Rule 8: **publication grants implicit permission to link/transclude.**
- Rule 9: **granular royalty / micropayment** on any accessed portion.
- **Transcopyright:** pre-granted permission for virtual republication *by reference*,
with the attribution chain to origin preserved automatically because the content is
transcluded, not copied.
shard-wiki mapping — handle with the **mechanism-over-policy** rule:
- The *mechanism* shard-wiki should provide: a reference (overlay/transclusion) that
**carries provenance and reuse terms with it**, so attribution and permission travel
with the span. This strengthens UC-29 (remix with portable attribution).
- The *policy* (whether reuse requires permission, whether anything is metered) must
**not be hard-coded**. Xanadu baked one global economic policy into the substrate;
shard-wiki keeps editorial/economic policy configurable. Baking in a payments or
permission model would be a design-bug per INTENT.
- This intersects the **settled authz-in-core / authn-delegated decision**
([[shard-wiki-auth-in-core-decision]]): "publication grants implicit linking
permission" is exactly a *policy* that an information space might or might not adopt
on its L0→L4 ladder. shard-wiki must be able to *represent* the permission on a
transclusion; it must not *assume* Xanadu's answer.
---
## 7. Xanadu through shard-wiki's lens — it is architecture, not a shard
Unlike the engine dives, there is **no capability profile to fill in** — Xanadu is not
a backend you attach. The useful framing is the inverse: *which shard-wiki primitives
are Xanadu mechanisms in disguise?*
| shard-wiki primitive | Xanadu mechanism | Inheritance verdict |
|----------------------|------------------|---------------------|
| Projection (lazy view, no copy) | xanadoc / EDL span assembly | **Inherit** — same idea, scope to wiki pages |
| Overlay (non-destructive edit) | xanalinks as separate objects | **Inherit** — overlays are xanalinks |
| Union BackLinks | two-way links visible from all endpoints | **Inherit** — generalize to sub-page spans |
| Transclusion (UC-32) | content-identity, content-aware reuse | **Inherit, strengthen** — add reverse + equivalence |
| Coordination journal | invariant istream / permanent storage | **Inherit** (Git gives this for Git-native shards) |
| Stable span address | tumblers | **Aspire** — hard, partly unshippable; degrade gracefully |
| Editorial/economic policy | transcopyright + micropayment | **Reject as substrate** — keep configurable |
| One global docuverse | the docuverse premise | **Reject** — see §8.2 |
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **Reference-not-copy** is the core of both. Xanadu validates shard-wiki's
"lazy projection over eager copying" and "union without erasure" as a coherent,
decades-deep design lineage, not an ad-hoc preference.
- **Provenance is structural, not decorative.** In Xanadu, knowing where content came
from and where it is reused is *built into the storage model*. shard-wiki's "never
hide authorship, conflicts, freshness, backend limits" is the same commitment.
- **Links/edits as separate objects** validates overlays-before-mutation and
first-class BackLinks.
### 8.2 Deliberate divergences (design bugs if conflated)
1. **No single docuverse / no universal address space.** Xanadu requires one global
permanent store, universal addressing, and its own published clientserver
protocol (rules 12, 17). shard-wiki's whole reason to exist is the opposite:
**federate heterogeneous, sovereign backends** (Git repos, Gitea, Obsidian,
WebDAV, Coulomb), each keeping its own storage, history, identity, and limits.
Adopting Xanadu's universality would violate **shard sovereignty** and **graceful
degradation**. shard-wiki's addressing must tolerate backends that *cannot* offer
stable span addresses, treating fine-grained addressing as an **adapter
capability**, not a precondition.
2. **One canonical content instance vs. parallel divergent versions.** Xanadu
transclusion centers a *single source of truth*. shard-wiki must support
**equivalent-but-divergent** pages across shards with conflicts *visible* (UC-27),
not collapsed into one instance. Content-identity is a *detection* tool here, not a
mandate to unify.
3. **Baked-in economic/permission policy.** See §6 — reject as substrate, keep as
configurable policy.
4. **Permanence as a hard requirement.** Xanadu forbids deletion globally. shard-wiki
cannot impose that on sovereign shards; permanence holds for the **coordination
journal** and Git-native shards, and is *approximated* (cache/projection/snapshot)
for backends that delete. This is graceful degradation, not a weaker promise.
### 8.3 What Xanadu teaches that shard-wiki should not lose
- A page view can legitimately be a **composition manifest** (a list of references),
not a file. Build the page model so a page *can* be authored as references into
other shards (UC-44), even if the common case is a plain Markdown file.
- **Content identity** (not path/title) is the durable basis for transclusion,
equivalence, and reverse lookup across sovereign shards (UC-45, UC-46). Lean on
Git blob hashes where available; define a content-fingerprint fallback elsewhere.
- Getting **stable fine-grained addressing** right is the hard, valuable, historically
*unshippable* part. Scope it as an adapter capability with explicit degradation
rather than promising tumbler-grade universality.
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-43**. New UCs **UC-44UC-46** added; several existing UCs
enriched (no new scenario, stronger mechanism/notes).
| Seed | Catalog action |
|------|----------------|
| **Compose-by-reference page** — author a page whose body is an ordered list of spans pulled from multiple shards, stored as a manifest (xanadoc/EDL), not a copy | **UC-44 (new)** |
| **Reverse transclusion** — find every page/shard where a given span (paragraph/section) appears or is transcluded | **UC-45 (new)** |
| **Content-identity equivalence** — detect that two pages in different shards are the same or derived content via span/content overlap, without matching titles or paths | **UC-46 (new)** |
| Content-identity, content-aware (bidirectional) transclusion | **enriches UC-32** |
| Equivalence detection mechanism for parallel versions | **enriches UC-27** |
| Reference carries provenance + reuse terms (transcopyright as representable policy) | **enriches UC-29**; links [[shard-wiki-auth-in-core-decision]] |
| Provenance: content remembers its appearances | **enriches UC-24** |
---
## 10. Adapter-contract / architecture notes for SHARD-WP-0002
Logged as architecture (no UC):
- **Fine-grained span addressing is an adapter capability**, not a core assumption.
The shard adapter contract should model whether a shard can mint a *stable address
for a sub-page span* that survives edits/versions (tumbler-grade), down through
whole-page-only, down to path-only. Transclusion/overlay capabilities depend on it.
- **Content identity** should be a contract-level concept: a shard advertises how it
fingerprints content (Git blob hash, normalized-text hash, none). UC-45/UC-46 and
cross-shard diff/merge consume it.
- **Composition manifests** (UC-44) imply the wiki page model must permit a page whose
canonical form is a reference list. This is an INTENT-level page-model decision —
flag for the page-model spec, not just the adapter contract.
- Reuse-terms metadata on a reference (UC-29 / §6) is **policy data the core carries
but does not interpret** — consistent with mechanism-over-policy and the L0→L4
authz ladder.
---
## 11. Open questions (for spec / workplans)
1. What is shard-wiki's **portable span address**? Git blob+range works for Git-native
shards; what is the fallback for Obsidian/WebDAV/Gitea-wiki, and how does it
survive a shard's storage swap (cf. UC-43 Foswiki RCS↔PlainFile)?
2. Is **compose-by-reference (UC-44)** core orchestrator, adapter-provided, or
reference-UI — and is it MVP or deferred with UC-32?
3. Does **content-identity equivalence (UC-46)** belong in core (cross-shard union
logic) or as an adapter-provided index? How expensive is span-set intersection at
wiki scale without enfilades?
4. How far do we take **reverse transclusion (UC-45)** — exact span identity only, or
fuzzy/derived-content tracking? The latter is research-grade.
5. Where does **reuse-terms/transcopyright metadata** sit on the L0→L4 ladder, and
does any tier ever *enforce* it, or is it always advisory provenance?
---
## 12. Sources
| Source | Used for |
|--------|----------|
| Wikipedia — Project Xanadu (https://en.wikipedia.org/wiki/Project_Xanadu) | 17 rules, tumblers, history, Web critique |
| xanadu.com — The Edit Decision List / Xanadoc File (https://xanadu.com/xuEDL.html) | EDL/xanadoc structure: spans + xanalinks, client assembly |
| Wikipedia — Enfilade (Xanadu) (https://en.wikipedia.org/wiki/Enfilade_(Xanadu)) | istream, enfilades (dsp/wid), spanfilade, granfilade, POOMfilade, version comparison by span-set intersection |
| Wikipedia — Transclusion (https://en.wikipedia.org/wiki/Transclusion) | "same content knowably in more than one place," content-aware reuse, transcopyright, micropayment vs. weak modern forms |
| Maggie Appleton — Xanadu Patterns (https://maggieappleton.com/xanadu-patterns) | pattern naming: visible links, parallel documents, transpointing windows, modular blocks, stable addresses, annotation |
Cross-references: `research/260608-federation-concepts/findings.md` §3.3 (prior
surface treatment), `spec/UseCaseCatalog.md` (UC-24, UC-27, UC-29, UC-32),
`workplans/SHARD-WP-0002-federation-architecture.md` (adapter contract).
---
## 13. Traceability
- New UCs: **UC-44, UC-45, UC-46**`spec/UseCaseCatalog.md` (section F, federation).
- Enriched UCs: **UC-24, UC-27, UC-29, UC-32**.
- Architecture (no UC): span-addressing capability, content-identity, composition
manifest, reuse-terms metadata → `SHARD-WP-0002`.
- Decision link: transcopyright-as-policy → [[shard-wiki-auth-in-core-decision]].
</content>
</invoke>

View File

@@ -0,0 +1,43 @@
# 260614 — ZigZag / zzstructure deep dive (an information space as orthogonal dimensions)
Date: 2026-06-14
## What this is
A focused study of **Ted Nelson's ZigZag** and its data model **zzstructure**, asked
through a specific shard-wiki question: a wiki page is linked in many independent ways
at once — a **namespace** tree, a **created-from / genealogy** lineage, **hyperlinks**,
plus shard-wiki's own **shard / overlay / version / transclusion / equivalence**
relations. Could the information space be modelled as a zzstructure, where each
relationship is a first-class **dimension** and none is privileged?
Distinctive material:
- the zzstructure model — **cells**, **dimensions** (edge colours), **ranks**,
**posward/negward**, and the one rule that defines it: *at most one posward and one
negward neighbour per dimension per cell* (a directed coloured multigraph)
- **views** — I-view (one dimension), **H-view** (pivot two dimensions into a cross),
raster-vs-view
- trees/lists/tables as *spent dimensions*; many-to-many graphs as the hard case
(handled by **clones**)
- **clone ↔ transclusion** convergence with the Xanadu dive
## Contents
| Path | Role |
|------|------|
| `findings.md` | zzstructure model, views, fit analysis, a proposed dimension map for a shard-wiki space, recommendation (lens not store), INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
(UC-47 navigate-by-dimension, UC-48 H-view pivot/cross-tab, UC-49 genealogy
dimension); UC-05/1722/26/29 enriched. A **dimensional projection layer** is logged
as architecture for `workplans/SHARD-WP-0002-federation-architecture.md`.
**Recommendation recorded:** adopt zzstructure as a **navigation / visualization /
indexing lens** (a derived dimensional projection over the union), **not** as the
storage substrate — Git and sovereign shards remain canonical (INTENT Stability Note).
The many-to-many hyperlink graph does not fit zzstructure's rank constraint and stays a
separate graph index. Pairs with `research/260614-xanadu-deep-dive/` (clone ↔
transclusion).
</content>

View File

@@ -0,0 +1,294 @@
# Findings — ZigZag / zzstructure: an information space as orthogonal dimensions
Date: 2026-06-14
Source kind: **conceptual / data-model prior art** (a data model + visualization
paradigm, not a deployable wiki engine, not a candidate shard backend)
Lens: shard-wiki — modelling a wiki information space where a page participates in
*many* relationship structures at once (namespace, genealogy, links, versions, shards)
> Motivating question (from the user). A wiki page is linked in several independent
> ways at once: a **primary namespace** association giving a tree-like hierarchy; a
> **created-from / genealogy** association (fork/derivation lineage); **hyperlinks**
> to other pages; and — in shard-wiki — *which shard* it lives in, its **overlays**,
> its **versions**, its **transclusions**, and its **equivalents** elsewhere. Could
> the information space be modelled as Ted Nelson's **ZigZag / zzstructure**, where
> each kind of relationship is a separate, first-class **dimension** and no single
> relationship is privileged as *the* structure?
>
> Short answer: **yes as a navigation / visualization / indexing lens; no as the
> storage substrate.** zzstructure is an unusually good *conceptual* match for
> shard-wiki's "union without erasure" thesis — it makes every relationship
> co-equal and separately navigable — but its one-neighbour-per-direction
> constraint makes it a poor *direct* encoding of the many-to-many hyperlink graph,
> and Git/heterogeneous shards must remain the canonical store. Details below.
Companion to the Xanadu dive (`research/260614-xanadu-deep-dive/`): ZigZag is
Nelson's *other* lifelong project, the structural/visual counterpart to Xanadu's
literary/addressing one. Where Xanadu gives shard-wiki **reference-not-copy**, ZigZag
gives it **many co-equal structures over the same cells**.
---
## 1. What ZigZag / zzstructure is
ZigZag (Ted Nelson, conceived 1981 at Datapoint; first Perl prototype by Andrew Pam,
1997; the **GZigZag / Gzz** Java project 20002003; US patent 6,262,736, expired
2019) is **"a data model that deconstructs the spreadsheet to allow irregular
relations, generalizing the idea to multiple dimensions."** The generic, vendor-free
name for the data model is **zzstructure**.
Core vocabulary:
- **Cell (zzcell)** — an atom of data (a value, a record, a page). Cells are the only
content; everything else is connection.
- **Dimension** — a *named kind of connection* (an edge colour). Dimensions are
**first-class** — in ZigZag, dimensions are themselves cells.
- **Link (zzlink)** — a connection between two cells *along one dimension*, with a
**posward** (positive) and **negward** (negative) direction.
- **Rank** — the maximal chain of cells along one dimension (a path, or a cycle). A
rank is ZigZag's equivalent of "a row" / "a list" — but it exists only along one
named dimension.
- **Cursor** — the current focus cell from which views are drawn.
- **Clone** — a copy of a cell that shares identity with its head cell (along a
special clone dimension), used to let one logical item appear in several places /
participate in many-to-one relations.
---
## 2. The one rule that defines everything
Formally (McGuffin & schraefel, HT2004; McGuffin's graph-theoretic intro), a
zzstructure is:
> a **directed multigraph whose edges are coloured (typed)**, subject to the
> restriction that **each node has at most one incoming edge of each colour and at
> most one outgoing edge of each colour.**
Equivalently: **along any single dimension, a cell has at most one posward and one
negward neighbour.** That single constraint is the whole design:
- Each dimension decomposes the cell set into disjoint **ranks** (paths and cycles) —
never a branching tree, never a fan-out, *within one dimension*.
- Many dimensions coexist over the **same** cells, each imposing its own independent
rank structure. A cell sits at the intersection of as many dimensions as it
participates in.
- There is **no inherent hierarchy.** A tree is not the substrate; it is *one
possible dimension* (encoded left-child/right-sibling — see §4).
This is the property that matters for shard-wiki: **the same set of pages can carry
many orthogonal structures simultaneously, with none privileged.**
---
## 3. Views — how multiple dimensions are seen
Because no screen can show all dimensions, ZigZag *rotates* two into view at a time:
- **I-view** — cells along **one** dimension from the cursor, drawn as a line.
- **H-view** — a 2-D cross: **two** dimensions through the cursor, one drawn
horizontally and one vertically; only cells actually connected to the cursor along
those two dimensions appear (empty spreadsheet cells simply do not exist).
- **I+/H+ (augmented) views** — fill remaining space with further neighbours.
- **Raster vs. view (terminology)** — a *raster* selects which cells to take from the
structure; a *view* places them on screen. "Pivoting" rotates a different dimension
into the horizontal or vertical slot.
The H-view "pivot" is the genuinely interesting UX idea for a *federated* wiki: put
**pages of a namespace** on one axis and **shards** (or **versions across shards**)
on the other, then rotate to re-cross-tabulate the same union by a different pair of
relationships.
---
## 4. Trees, lists, tables, graphs — what fits and what does not
zzstructure subsumes simpler structures by spending dimensions:
- **List** = one rank along one dimension.
- **Table** = two dimensions (rows / columns), shown as an H-view.
- **Tree** = encoded as a zzstructure "analogous to the left-child, right-sibling
pointer implementation" — i.e. a *parent→firstchild* dimension plus a
*sibling→sibling* dimension. So a hierarchy is **two dimensions**, not a primitive.
- **Many-to-one / many-to-many** = NOT directly expressible (the one-edge-per-colour
rule forbids fan-out). It is simulated with **clones**: the head cell links to many
clone cells along the clone dimension, and each clone participates in a different
rank.
**This is the critical caveat for wikis.** A namespace hierarchy (each page has *one*
parent) and a created-from genealogy (each page has *one* origin) fit zzstructure
*natively* and beautifully. But a wiki's **hyperlink graph is many-to-many** (a page
links to many pages and is linked from many) — it is *not* a rank and cannot be one
dimension. Representing links/backlinks in zzstructure requires cloning or treating
the link set as data, which is awkward. zzstructure is excellent for the
**functional/sequential/hierarchical** relationships and weak for the **arbitrary
graph** ones.
---
## 5. Mapping a shard-wiki information space onto dimensions
Taking the user's framing literally — model the union of pages as cells, and each
relationship as a dimension:
| Dimension (proposed) | Rank meaning | Fits zzstructure? | shard-wiki source |
|----------------------|--------------|-------------------|-------------------|
| `d.namespace-child` + `d.namespace-sibling` | namespace tree (left-child/right-sibling) | **Yes** (2 dims) | UC-22, yawex topics |
| `d.genealogy` | created-from / fork lineage (each page one origin) | **Yes** (functional) | UC-26 fork, UC-29 remix |
| `d.version` | revision order of a page | **Yes** (a rank = a history) | coordination journal |
| `d.shard` | pages grouped by owning shard | **Yes** | shard sovereignty |
| `d.overlay` | page → its overlays | **Yes** (or clones) | overlay model |
| `d.equivalence` | parallel versions of the same topic across shards | **Yes** (a rank of equivalents) | UC-27, UC-46 |
| `d.recent` | temporal order of changes | **Yes** | UC-17 RecentChanges |
| `d.alphabetical` | AllPages ordering | **Yes** | UC-19 |
| `d.links` / `d.backlinks` | hyperlink graph | **No** (many-to-many) — needs clones / stays a graph index | UC-05, UC-18 |
| `d.transclusion` | content reused in many places | **No** as rank — but maps to **clones** (= "same content in many places") | UC-32, UC-44/45 |
Observation: shard-wiki's existing **derived views** (UC-05, UC-17UC-20) are, in
zzstructure terms, *dimensions + rasters*: RecentChanges is a raster along `d.recent`,
AllPages along `d.alphabetical`, SiteMap is an H-view over the two namespace
dimensions. zzstructure offers a **single unifying vocabulary** for what shard-wiki
currently treats as a grab-bag of separate derived views.
Observation 2: ZigZag **clones** are conceptually the same move as Xanadu
transclusion ("the same content knowably in more than one place") — the two Nelson
projects converge here. A transcluded span / equivalent page is a clone that
participates in multiple ranks.
---
## 6. Is zzstructure the right model for shard-wiki? (recommendation)
**Adopt as a lens, not a substrate.**
- **As a navigation & visualization model: promising.** Letting a reader pick which
*dimension* to traverse (namespace vs. genealogy vs. version vs. shard) and pivot
two dimensions into an H-view is a real, differentiated UX for a federated,
provenance-preserving wiki. It operationalizes "union without erasure": every
relationship is co-equal and visible, none hidden. (UC-47, UC-48.)
- **As a conceptual model for the page graph: useful and clarifying.** Reframing the
scattered derived views as *dimensions over one cell set* is a clean mental model
and a candidate internal index/API shape. It also de-privileges the namespace tree
(UC-49) — consistent with **mechanism over policy** (the hierarchy is one
configurable dimension, not the canonical structure).
- **As the storage substrate: reject.** Git and heterogeneous sovereign shards remain
canonical (per INTENT Stability Note — Git's role is architectural). A zzstructure
is a **derived, projected index** computed over the union, like BackLinks today —
not a database that replaces the shards. The many-to-many hyperlink graph does not
fit the rank constraint, and forcing the wiki into a zz-DB would violate shard
sovereignty and the orchestrator-not-engine boundary.
Net: zzstructure is a strong answer to *"how do we present a page's many
simultaneous relationships without picking a winner?"* — a **dimensional projection**
layer over the existing union, not a new store.
---
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
- **Union without erasure ↔ co-equal dimensions.** zzstructure's refusal to privilege
any one structure is the data-model expression of shard-wiki's refusal to hide
provenance, conflicts, or alternative arrangements.
- **Mechanism over policy ↔ hierarchy-as-just-a-dimension.** The namespace tree being
one dimension among many is exactly "do not hard-code one canonical structure."
- **Provenance ↔ genealogy dimension.** Created-from lineage as a navigable rank makes
provenance a first-class traversal, not metadata buried in a footer.
### 7.2 Deliberate divergences (design bugs if conflated)
1. **zzstructure as store.** Replacing Git/shards with a zz-database would break the
Stability-Note boundary on Git's role and on orchestrator-vs-engine. zzstructure is
a projection, full stop.
2. **Forcing the hyperlink graph into ranks.** The link/backlink graph is many-to-many
and must stay a graph index; do not contort it into a single dimension. Use clones
or keep it as a separate graph projection.
3. **One global zz-space.** Like Xanadu's docuverse, a single universal zz-space would
fight shard sovereignty. Each information space gets its *own* dimensional
projection over *its* attached shards.
### 7.3 What ZigZag teaches that shard-wiki should keep
- A page is best understood as **a cell at the intersection of many independent
relationships**, not as a node in one tree. Build the page model and the navigation
API so *adding a new relationship is adding a dimension*, not special-casing a view.
- **Pivoting two relationships** (H-view) is a powerful, under-used exploration
primitive for federation (namespace × shard, page × version, topic × equivalent).
---
## 8. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-46**. New UCs **UC-47UC-49** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Navigate along a chosen relationship dimension** — pick namespace / genealogy / links / version / shard as the active traversal axis | **UC-47 (new)** |
| **Pivot two dimensions as a cross-tab (H-view)** — e.g. namespace × shard, page × version-across-shards | **UC-48 (new)** |
| **First-class created-from / fork genealogy dimension** — navigate derivation lineage as a rank | **UC-49 (new)** |
| Namespace hierarchy is *one dimension among many*, not the privileged structure | **enriches UC-22** |
| Derived views (RecentChanges, AllPages, SiteMap, BackLinks) reframed as dimensions + rasters | **enriches UC-05** (anchor for the UC-17UC-20 view reframing) |
| Fork/remix produce a genealogy edge consumed by UC-49 | **enriches UC-26, UC-29** |
| Clone = "same content in many places" = transclusion convergence | links UC-32, UC-44/45 |
---
## 9. Architecture notes for SHARD-WP-0002 (no UC)
- A **dimensional projection layer** over the union is a candidate internal model:
pages = cells; relationships (namespace, genealogy, version, shard, equivalence,
overlay, recent, alphabetical) = dimensions; existing derived views = rasters over
them. It is **derived/lazy**, never canonical store.
- The **link/backlink graph stays a separate many-to-many index**; do not model it as
a rank. Reconcile with the dimensional model via clones or by keeping graph and
dimensional projections side by side.
- **Genealogy** (created-from) should be recorded as an edge when fork/remix/import
happens (UC-26, UC-29) so the `d.genealogy` rank can be reconstructed — ties to the
coordination journal.
- Whether the dimensional model is exposed in a **public navigation API** or stays an
internal organizing concept is open (see §10).
---
## 10. Open questions (for spec / workplans)
1. Is the dimensional/zzstructure model a **public navigation API** and UI paradigm,
or only an *internal* organizing concept for the existing derived views?
2. How is the **many-to-many link graph** reconciled with the rank-based dimensional
model — clones, a parallel graph index, or both?
3. Which dimensions are **core** (namespace, version, shard, genealogy, recent) vs.
**adapter-provided** vs. **computed on demand** (equivalence, backlinks)?
4. Is **H-view pivoting** worth a reference-UI investment, or is it research-only?
5. Does the **clone = transclusion** convergence justify a single internal primitive
shared by UC-44/45 (transclusion) and UC-47/48 (dimensional navigation)?
---
## 11. Sources
| Source | Used for |
|--------|----------|
| Wikipedia — ZigZag (software) (https://en.wikipedia.org/wiki/ZigZag_(software)) | History, spreadsheet-deconstruction framing, one-connection-per-dimension rule, pivoting |
| Wikipedia — Zzstructure (https://en.wikipedia.org/wiki/Zzstructure) | Directed-coloured-multigraph definition, dimensions-are-cells, raster vs. view |
| McGuffin & schraefel, "A Comparison of Hyperstructures: Zzstructures, mSpaces, and Polyarchies," ACM Hypertext 2004 (https://www.dgp.toronto.edu/papers/mmcguffin_HT2004.pdf) | Graph-theoretic formal definition; taxonomy vs. mSpaces/polyarchies |
| McGuffin, "A Graph-Theoretic Introduction to Ted Nelson's Zzstructures" (https://www.dgp.toronto.edu/~mjmcguff/research/zigzag/) | Ranks, I-view/H-view/augmented views, clones, tree-as-left-child/right-sibling, "multiple arrangements coexist via dimensions" |
| Moore & Brailsford, "Unified Hyperstructures for Bioinformatics," JoDI (https://jodi-ojs-tdl.tdl.org/jodi/article/download/jodi-138/127) | Applied zzstructure as a unifying model over heterogeneous structures |
| Nelson, "A Cosmology for a Different Computer Universe" (zzstructure context) | Nelson's own framing of cells/dimensions/visualization |
| Gzz — "A Gentle Introduction to Ted Nelson's ZigZag Structure" (https://www.nongnu.org/gzz/gi/gi.html) | Practical cell/dimension/rank examples (rate-limited; corroborated by above) |
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (clone↔transclusion
convergence), `spec/UseCaseCatalog.md` (UC-05, UC-17UC-22, UC-26, UC-27, UC-29,
UC-32, UC-44/45/46), `workplans/SHARD-WP-0002-federation-architecture.md`.
---
## 12. Traceability
- New UCs: **UC-47, UC-48, UC-49**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-05** (anchors the UC-17UC-20 derived-view reframing), **UC-22,
UC-26, UC-29**.
- Architecture (no UC): dimensional projection layer; link-graph-stays-graph;
genealogy edges in the journal; clone↔transclusion primitive → `SHARD-WP-0002`.
- Decision boundary recorded: zzstructure is a **derived lens, not the store**;
Git/shards remain canonical (INTENT Stability Note).
</content>

Some files were not shown because too many files have changed in this diff Show More