Compare commits

...

163 Commits

Author SHA1 Message Date
776f5af5a7 Normalize agent instructions and workplan frontmatter (STATE-WP-0067)
- Align agent files with on-disk workplan prefixes (infer from workplan ids)
- Set workplan domain to registered domain_slug; add topic_slug where applicable
- Repair frontmatter delimiter formatting; migrate legacy task status literals
- Regenerate AGENTS.md, CLAUDE.md, and .claude/rules from State Hub templates
2026-06-22 23:16:28 +02:00
fd961c83b4 Add .repo-classification.yaml (CUST-WP-0050 T11 agent first-pass) 2026-06-22 17:47:43 +02:00
cca5bf83c3 Add credential routing instructions for all agent runtimes
Propagate shared credential-routing section (Codex, Claude, Grok, llm-connect)
from state-hub template via scripts/propagate_credential_routing.py.
2026-06-18 22:48:39 +02:00
def699c1eb feat(adapters): GitShardAdapter history adopt + cross-substrate integration (WP-0012 T3)
Adopt git-native history (TSD §A.5): a VERSION-gated history(key) surfaces the
commit list for a path (newest-first sha + subject) — declared by every git-IS-store
shard, read-only or not. Integration proves the union/overlay/edit machinery works
unchanged across folder + git substrates: resolve/chorus span both, edit through a
git shard fast-forwards as a commit, apply-under-drift refuses on an external commit
(sha drift) without clobbering, and a read-only git target keeps the overlay as a
draft. SCOPE updated; WP-0012 done. 196 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:41:19 +02:00
a4e0f52ec1 feat(adapters): GitShardAdapter write=commit + current_rev drift (WP-0012 T2)
Writable mode: write(key, body) stages and commits the file (skipping a no-op so
no empty commit is created), returning the page at the new commit sha. The
writable profile declares WRITE + VERSION with PER_PAGE granularity. current_rev
is the per-path commit sha, so a write — or an external commit to the same path —
moves it, driving apply-under-drift. Passes the conformance positive-write probe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:38:41 +02:00
4231daf94f feat(adapters): GitShardAdapter read path + git-IS-store profile (WP-0012 T1)
A second substrate validating the contract beyond plain folders: a git-IS-store
shard reading Markdown from a git repo. Keys are tracked *.md paths; read returns
a Page whose source_rev is the per-path last-commit sha (so an edit to one page
never drifts another); profile is git-IS-store / substrate=git / history=git-native
/ addressing=path, validated against the §6.5 implication rules. Passes the
conformance read path with honest absence of unclaimed verbs. Zero new deps
(git CLI via subprocess). No core changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:36:28 +02:00
37681d89b6 feat(incremental): wire maintained tier behind views; rebuild fallback (WP-0011 T4)
Route InformationSpace.all_pages through a maintained UnionIndex: equivalence is
served from the incrementally maintained index (curator bindings re-synced live
from the log fold + detected content edges), exposed in decision-log string form
so results are a behaviour-preserving superset. The index is built lazily and
rebuilt (bounded fallback) when the union mutates (attach/edit invalidate it);
reindex() forces a rebuild and verify_index() runs the I-2 self-healing checker.
all_pages() gains an optional equivalence_groups source (default = fold) so
direct callers are unaffected. SCOPE updated; WP-0011 done. 173 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:21:39 +02:00
a8e65235a8 feat(incremental): I-2 digest + consistency-checker (WP-0011 T3)
A Merkle-style digest summarizes the derived tier (per-identity fingerprint +
incident edges as order-independent leaves) so equal states have equal digests
and the digest is stable under equivalent event orders. A ConsistencyChecker
recomputes the authoritative fold from the current source, compares it over a
sampled region, and on mismatch scoped-recomputes just the affected identities —
self-healing missed-delta drift, corrupted internal state, and vanished pages.
Makes derived = f(canonical) verified, not asserted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:16:50 +02:00
d7d046cac0 test(incremental): delta maintenance == rebuild, retraction + split (WP-0011 T2)
Verify change-driven maintenance keeps the equivalence index equal to a
from-scratch rebuild under add / edit / remove: an edit into a new bucket
retracts the stale edge, an edit into equivalence adds one, and removing a
connector node propagates a retraction that splits a chorus. Equality checked
against a fresh build() oracle on every operation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:14:32 +02:00
0b3ab2086f feat(incremental): indexed equivalence — blocking + verify (WP-0011 T1)
Detect equivalence (distinct identities holding the same page) without pairwise
O(N²): MinHash/LSH bands over content shingles + normalized-title buckets
generate candidates (blocking), then exact-fingerprint or Jaccard>=threshold
confirm them (verify), with curator decision-log bindings always forming edges.
Groups are the connected components of the edge set. Includes the incremental
add/update/remove internals used by T2. Matches a brute-force oracle. New
incremental/ package (minhash primitives + EquivalenceIndex).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:13:06 +02:00
d85d019543 feat(views): wire derived views onto InformationSpace + integration (WP-0010 T5)
Expose backlinks(name), recent_changes(), all_pages(), site_map() on
InformationSpace. Integration test exercises all four over two shards (BackLinks
aggregate across shards, AllPages/SiteMap span the union, RecentChanges merges an
alias decision with shard edits). SCOPE updated; WP-0010 done. 152 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:05:12 +02:00
3a5acdcb28 feat(views): AllPages + SiteMap enumeration views (WP-0010 T4)
AllPages enumerates the union's distinct pages, collapsing chorus (same key
across shards) and equivalence-bound identities into one entry via union-find,
noting divergence when members' bodies differ (collapse acknowledged, not
silent). SiteMap builds the namespace tree from page placements, spanning shards.
Both derived/recomputable and presentation-free.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 02:03:15 +02:00
34b0c539f3 feat(views): RecentChanges merged change feed (WP-0010 T3)
One newest-first feed merging the coordination journal (overlay/alias/fork/merge/
binding decisions, with actor + payload) and shard change signals (page
source_rev / mtime). Each entry carries provenance: the originating shard for an
edit, or 'coordination' (and the actor) for a decision. Non-temporal revision
tokens are skipped gracefully. Derived/recomputable; notify-streaming later.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:59:11 +02:00
da540d4eea feat(views): BackLinks derived view over the union link graph (WP-0010 T2)
For any page name, the set of pages that link to it: extract wikilinks from every
union page (new UnionGraph.iter_pages enumeration) and index the resolved ones by
target name. Red-links create no backlinks; entries carry source provenance; a
chorus target aggregates the backlinks of all members under one name. Derived/
recomputable, stores nothing canonical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:56:48 +02:00
951b24300d feat(views): wikilink + red-link model (WP-0010 T1)
A CommonMark wikilink extension: extract [[Target]] / [[Target|label]] from a
page body (skipping fenced + inline code, preserving offsets), and resolve each
target through the union — resolved is a link, unresolved is a createable
red-link (never a dropped reference). CamelCase auto-linking is off by default,
opt-in per space, and never double-counts a target already inside [[...]]. Link
model + resolution are core; rendering stays L6. New views/ package.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:55:06 +02:00
c731c96634 feat(coordination): git backend wiring + verbatim log migration (WP-0009 T4)
InformationSpace.git_backed(space_id, repo_path) wires the git coordination log;
the default constructor stays in-memory for tests (new keyword-only store=). A
one-time importer (migrate_space / import_log / JSONL export+import) replays an
existing in-memory or JSON log into git verbatim — preserving seq, timestamp and
actor (union-without-erasure) and refusing out-of-order import. Same fold after
migration; no behavioural change to overlay/union. SCOPE updated; WP-0009 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:49:55 +02:00
f0fee65cc0 test(coordination): cross-process read-your-writes + fold parity (WP-0009 T3)
Verify the git backend's fold reads the durable log into CoordinationState with
unchanged semantics, and that read-your-writes holds across separate handles and
separate OS processes against the same space ref (one test spawns a real
subprocess that appends, then reads it back). Cross-process fold equals the
in-memory fold for the same event sequence (derived = f(log)).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:47:08 +02:00
34432c2e15 feat(coordination): per-space append authority (lease) (WP-0009 T2)
A single append authority per space serializes appends into a total order: at
most one node holds a space's lease; only the holder writes, non-holders forward
their append intent to the holder. Leases are time-bounded and re-grantable, so
a dead holder's lease expires and a new node resumes from the log head (seq stays
contiguous). A stale ex-holder discovers it is no longer the holder and forwards
rather than writing, so a partitioned node cannot fork the log. Works over both
in-memory and git stores. Single-coordinator only (distributed leasing out of scope).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:45:52 +02:00
45a858ead0 feat(coordination): git-backed DecisionLog event store (WP-0009 T1)
Factor DecisionLog storage behind an EventStore abstraction: InMemoryEventStore
stays the default/test double, GitEventStore makes the coordination log
git-addressable. Each space is a ref (refs/spaces/<sha1>); append writes an
immutable one-blob commit and advances the ref under compare-and-swap, so the
commit chain is the per-space total order and a racing appender can never fork
the log. Deterministic stable-JSON event serialization. Zero runtime deps
(git CLI via subprocess). API and fold unchanged across backends.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 01:41:27 +02:00
b31e9bc337 Add capability registry scaffold and seed entries from reuse-surface
Bootstrap registry/indexes/capabilities.yaml and migrate helix_forge
capability entries owned by this repository for federation publishing.
2026-06-16 01:34:23 +02:00
e50dcc6b5c chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:57:09 +02:00
a165cced33 feat(engine): ext.struct typed-records built-in; close engine implementation (WP-0014 T6)
engine/extensions/struct.py: ext.struct (typed records) — in-text frontmatter
parse + ON_WRITE validation (allowed-fields, content-preserving), ON_READ tags
PageShape.TYPED_RECORD, ON_PROFILE raises structured-payload. Proves the framework:
feature absent when off (opaque prose, honest profile), present + profile-reflected
when on; works through InformationSpace edit. SCOPE updated. 6 tests, 107 total,
~97% coverage, pyflakes clean. Marks T6 + SHARD-WP-0014 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:56:22 +02:00
8393a9c55d chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:36:55 +02:00
ff96ee0c48 feat(engine): EngineShardAdapter — engine as a canonical-mode shard (WP-0014 T5)
engine/adapter.py: EngineShardAdapter implements adapters.ShardAdapter (read/write
run extension transform hooks; profile derived from active extensions, E-5;
current_rev for apply-under-drift) + build_engine_shard() helper (explicit ids or
activation provider). runtime.available() added. Engine shard passes assert_conformant
and attaches to an InformationSpace — resolve + edit (overlay->apply->write-through)
work, and the declared profile reflects the active extensions. 5 tests green, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:36:24 +02:00
8b353f1077 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:27:57 +02:00
b9bb1f7d10 feat(engine): capability profile derived from active extensions (WP-0014 T4, E-5)
engine/profile.py: engine_base_profile() (kernel-only c2-minimum profile),
ProfileContribution (an extension's ON_PROFILE contribution: axis raises + verbs),
derive_profile() folds active extensions' contributions onto the base in deterministic
order then validate() — so configuration->capability is one chain and composition can
never yield an impossible profile (encrypted+native-query rejected). 5 tests green,
96 total, pyflakes clean. Marks T4 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:27:11 +02:00
c40fa3c934 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:16:54 +02:00
54c2bf2ae5 feat(engine): per-shard extension activation (WP-0014 T3, ADR-0001)
engine/activation.py: ActivationContext (shard/tenant, no authz), pluggable
ActivationProvider protocol, StaticProvider standalone default (zero-dep, global
flags + per-shard scoping + per-ext config), ActivationResolver (candidate ids ->
active set / activation profile), and feature_control_provider() lazy factory
(returns None when feature_control_sdk absent -> degrade to static; OpenFeature-
shaped when present). Availability only. 6 tests green, coverage held, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:15:33 +02:00
6d8bd837a4 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-16:
  - update .custodian-brief.md for shard-wiki
2026-06-16 00:12:18 +02:00
b48a99d3c2 feat(engine): typed-extension runtime (WP-0014 T2)
engine/extension.py: Extension contract (id/provides/declares_types/depends_on/
conflicts_with + bound hooks), ExtensionRuntime (register-with-verification,
activate = dependency closure + conflict + type-collision checks rejecting
impossible profiles), and ActiveExtensions with deterministic (topological, id-tie)
hook dispatch — transform hooks chain, collect hooks gather. 9 tests green,
coverage floor held, pyflakes clean. Marks T2 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:11:18 +02:00
9b7f86ba69 test: harden suite with error-path contracts + coverage floor (98%)
Adds tests/test_error_paths.py covering real failure contracts (red-link
single() KeyError, unknown/unattached-shard apply_overlay, kernel.delete
missing, conformance survives a broken profile, Placement str). Adds a
[tool.coverage.report] fail_under=90 floor (engages on pytest --cov, not bare
pytest). 76 tests, 98% coverage, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-16 00:05:16 +02:00
74142096d0 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 23:58:54 +02:00
2100e956aa feat(engine): page-store kernel skeleton (WP-0014 T1)
engine/ package: EngineKernel (in-process page store with per-page version
history; create/edit-as-version, recoverable delete-tombstone, keys, current_rev)
+ wikilink extraction + in-shard link resolution / red-link detection (EC-1..EC-4).
Reuses model/provenance; git-IS-store backing slots in later. 6 tests green,
pyflakes clean, full suite green. Marks T1 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:57:31 +02:00
e62560eb5a chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 23:52:14 +02:00
b147d3e831 workplan: SHARD-WP-0014 wiki-engine implementation (kernel + typed-extension runtime + activation)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:51:12 +02:00
cdcf4b09aa adr: ADR-0001 engine activation via feature-control (OpenFeature, availability-only)
Records the accepted decision: shard-wiki's native engine uses feature-control via
OpenFeature for per-shard extension activation (availability only, never authz),
provider-pluggable with a LocalProvider standalone default (mirrors the identity
ladder), at the engine layer, consuming the mature feature-control.evaluate slice.
Adds spec/adr/ series + README; hub decision abf7830f recorded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:48:49 +02:00
b21efe307b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 23:05:45 +02:00
e18397272a spec(SHARD-WP-0013 T6): wire-up + close-out
spec/README + SCOPE list WikiEngineCoreArchitecture.md; CoreArchitectureBlueprint
cross-links the engine as a canonical-mode shard (federation/union stay in the
orchestrator). reuse-surface engine capability promoted D2->D3 (4204255).
Marks T6 + SHARD-WP-0013 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 23:02:15 +02:00
0ee972f2e2 spec(SHARD-WP-0013 T5): WikiEngineCoreArchitecture.md — small core + typed extensions
Headless, API-first, agent-optimized native engine = canonical-mode shard backend.
Thesis: a page-store kernel with a typed-extension runtime; everything beyond the
c2-minimum is a typed extension activated per shard, and the shard's §A capability
profile is DERIVED from its active extensions (configuration->capability->conformance).
9 engine invariants (engine-is-one-shard, small kernel, per-shard activation,
profile-from-extensions, headless/agent-first, reuse-not-reinvent, typed+verified).
Kernel (4 concepts), typed-extension model (typed hooks + deterministic composition +
feature-control activation), T2 featureset/conflict-mediation realized, engine-as-shard,
agent-first API surface, module sketch, reuse (consumes feature-control/authorization;
G1 framework proposal), traceability, decisions/open, stability note. Marks T5 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:54:40 +02:00
bb1b54e0af chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 22:45:17 +02:00
b70f1c9acc intent(SHARD-WP-0013 T4): ratify additive native engine — headless, API-first, agent-optimized
Ratified by tegwick (decision 84ffdb48 resolved). INTENT.md amended: native
reference wiki-engine as a canonical-mode shard backend, reframed as HEADLESS &
API-FIRST — small typed-extension core optimized for integrating heterogeneous
data sources and efficient agent/automation access; no bundled UI. Edits:
Primary Utility framing + bullet, non-goal clarifier, new Design Principle,
Stability Note amendment. SCOPE.md: engine core in scope, UI/rendering out.
Orchestrator role unchanged; union-without-erasure + shard sovereignty preserved.
Marks T4 done; unblocks T5 (WikiEngineCoreArchitecture.md).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:43:46 +02:00
8de044bbde chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 22:29:53 +02:00
67d851be0b history(SHARD-WP-0013 T3): reuse-surface gap proposals + consumptions; derived-views registered
Registered capability.wiki.derived-views in reuse-surface (8 wiki.* total, index 20,
validate ok, pushed d985e68). Tracked contributions note (history/260615-...): proposes
two cross-cutting gaps to reuse-surface (G1 typed-extension-framework, G2
translation-fidelity) and records consumptions (feature-control.evaluate,
authorization.policy-evaluate). Sent to the reuse-surface agent via send_message. Marks T3 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:28:29 +02:00
46ec6f2a5f spec(SHARD-WP-0013 T2): engine-facing capability structure layer over the UC catalog
Adds 'Capability structure for the wiki engine (reuse-surface-aligned)' to
UseCaseCatalog.md (no UC renumbered): a small always-on engine core (EC-1..EC-5),
ten typed per-shard-activatable extensions (X-OVERLAY/AUTHZ/VIEW/STRUCT/ADDR/COMP/
PROV/COLLAB/FED/ATT) each mapped to a reuse-surface capability with activation
defaults, and a conflict-mediation map turning conflicting requirements into
mechanisms. Bridges demand -> the engine typed-extension model (T5). Marks T2 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:16:06 +02:00
ad4a2dbf5a chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 22:11:16 +02:00
23bc597343 chore(SHARD-WP-0013): T1 done — shard-wiki registered on reuse-surface (7 wiki.* capabilities)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 22:10:12 +02:00
ec7a1ec946 spec(SHARD-WP-0013): fold in confirmed reuse-surface coordinates + mechanism
reuse-surface = capability registry at /home/worsch/reuse-surface (helix_forge):
Markdown entries via templates/capability-entry.template.md, D/A/C/R maturity
vectors, reuse-surface CLI (validate/query/export/overlaps), hub at
reuse.coulomb.social. T1 now registers shard-wiki's capabilities as registry
entries; T3 contributes gaps back via REUSE-WP/agent message; T5 evaluates
reusing sibling feature-control for per-shard activation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 17:32:49 +02:00
d6d7cc555f chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 17:18:00 +02:00
1dd1def40f workplan: SHARD-WP-0013 wiki-engine prep (reuse-surface, UC systematization, WikiEngineCoreArchitecture)
Prep for an additive native wiki-engine framed as shard-wiki's reference
first-party shard backend (canonical-mode shard) — small core + typed-extension
framework, per-shard activation, integrated featureset with conflict mediation.
Tasks: confirm+register on the reuse surface; systematize the UC catalog around a
reuse-surface capability taxonomy; surface capability gaps back; narrow INTENT
amendment + ratified decision (gate); author WikiEngineCoreArchitecture.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 17:16:54 +02:00
6d341cd4e6 chore: restore SHARD-WP-0003 workstream id (re-link to existing finished workstream)
The 56af8185 workstream still exists in the DB (finished); clearing the id only
unlinked it (slug taken, cannot recreate). Restoring re-links the file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 15:55:55 +02:00
1b3d4aaa39 chore: clear stale SHARD-WP-0003 workstream id (re-register on next sync)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:08:28 +02:00
6e49bd3a4b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 14:07:24 +02:00
d2c73b02d9 workplans: register follow-up implementation roadmap (WP-0009..0012)
WP-0009 git-backed DecisionLog + per-space append authority (keystone backing);
WP-0010 derived views (wikilinks, BackLinks, RecentChanges, AllPages/SiteMap);
WP-0011 incremental union + equivalence index + I-2 verification;
WP-0012 second adapter (git-IS-store) validating the contract on a new substrate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 14:05:21 +02:00
042e198286 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 13:47:25 +02:00
3ea0cc1226 feat(space): wire write path into InformationSpace; integration (WP-0008 T6)
edit()/overlay()/apply_overlay() on InformationSpace. edit() unifies the write
path through one principled route — draft overlay then apply: write-through-capable
target fast-forwards (APPLIED), read-only target keeps the draft as local truth
(KEPT_DRAFT), external drift refuses (no clobber). Integration tests cover all
four. 64 tests green, pyflakes clean. Flips WP-0008 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:46:32 +02:00
4be2f190a0 feat(union): overlay-aware resolution (WP-0008 T5)
resolve() layers open overlays onto canonical pages (overlay_state=DRAFT always
surfaced; overlaid body projected when policy.show_drafts); draft-only edits make
a not-yet-existing page resolvable. Never hides an unapplied overlay (I-4). Policy
gains show_drafts. 4 tests green, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:29:06 +02:00
7d00ae758e feat(coordination): apply-under-drift (WP-0008 T4)
OverlayEngine.apply: read-only target → KEPT_DRAFT; base_rev==current →
fast-forward write-through (APPLIED, MERGE_DECIDED closes overlay); drift →
REFUSED_DRIFT (no clobber, I-5). 5 tests green, pyflakes clean. (blueprint §8.6)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 13:25:10 +02:00
715ab1ca00 feat(coordination): patch rendering (unified diff) (WP-0008 T3)
render_patch(overlay, base_body) → Patch (pure difflib unified diff, target-
labelled); is_empty when unchanged. 3 tests green, pyflakes clean. (ADR-05)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 12:32:47 +02:00
d797bc5ee4 feat(coordination): Overlay model + OverlayEngine.draft (WP-0008 T2)
Overlay value type (id, target, base_rev, body, state) recorded as an
OVERLAY_CREATED decision-log event (coordination-canonical); get()/open_overlays()
reconstruct from the fold. 4 tests green, pyflakes clean. (ADR-05, blueprint §8.2)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 11:56:53 +02:00
92d5774baf feat(adapters): writable FolderAdapter + positive write conformance (WP-0008 T1)
FolderAdapter(writable=True) declares WRITE+PER_PAGE, implements write() and
current_rev() (mtime token for drift detection). Conformance gains a
content-preserving positive write probe for WRITE-claiming adapters. 5 tests
green, full suite green, pyflakes clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 11:09:43 +02:00
e24f0034a0 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 10:29:21 +02:00
b2ea276c00 workplan: SHARD-WP-0008 write path (overlay engine, writable adapter, apply-under-drift)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 10:28:37 +02:00
9fedb31d8b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 10:13:40 +02:00
517cf1d282 feat(space): InformationSpace orchestrator + integration; foundation slice complete (WP-0007 T7)
InformationSpace ties the slice together: conformance-gated attach → resolve →
read, with alias() recording coordination decisions in the log. Exposed from the
package root. End-to-end integration test (two folder shards → union read with
layered provenance + chorus + alias redirect + red-link + nonconformant-rejected).
pyflakes clean, 39 tests green. Flips WP-0007 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 10:12:38 +02:00
b44b2a74a4 feat(policy,union): policy leaf + UnionGraph resolution with chorus (WP-0007 T6)
policy/ leaf (CanonicalSource presets, default chorus). union/ UnionGraph:
identity-keyed resolve (alias-redirect via log fold → union lookup → chorus →
red-link); chorus records divergent peers in each page's provenance envelope
(union without erasure); designated-canonical orders the pick. Imports down only.
6 tests green. (blueprint §8.4, ADR-01)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 10:02:09 +02:00
24108b65aa feat(coordination): event-sourced DecisionLog keystone (WP-0007 T5)
Append-only, totally-ordered-per-space decision log (in-process append authority;
git+lease later) with event types overlay/binding/alias/merge/fork, and a derived
fold to CoordinationState (aliases LWW, transitively-merged equivalence groups,
open overlays). derived = f(log); read-your-writes. 6 tests green. (blueprint §8.1)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 09:45:11 +02:00
6d48a1d3e6 feat(adapters): conformance suite — profiles verified not asserted (WP-0007 T4)
run_conformance/assert_conformant verify declared profile == observed behaviour
(profile validates, READ round-trips to the right shard, unclaimed verbs raise
NotSupported); ConformanceReport gives a precise capability diff. FolderAdapter
passes; lying-read and dishonest-write stubs fail. 3 tests green. (TSD §A.2)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 09:42:59 +02:00
9a4e00a05a feat(adapters): ShardAdapter contract + read-only FolderAdapter (WP-0007 T3)
Versioned ShardAdapter ABC (shard_id/profile/keys/read mandatory; optional verbs
raise NotSupported by default = honest absence). FolderAdapter reads a dir of
Markdown into Pages (relpath=key, mtime=source_rev, provenance envelope),
declaring a validated read-only file-store profile. 4 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 09:37:02 +02:00
5a77ea879c feat(model): Identity/Placement/Span, Page, CapabilityProfile (WP-0007 T2)
Identity = stable shard-scoped handle (not a fingerprint); Placement separate;
Span carries a layered provenance delta. Page model (PageShape incl. the four
computational shapes). CapabilityProfile with orthogonal-core axes + validate()
applying §6.5 implication rules that reject impossible profiles. Imports only
provenance/. 8 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 09:00:49 +02:00
aca9bf30f9 feat(provenance): layered ProvenanceEnvelope + effective() leaf (WP-0007 T1)
Dependency-free leaf rail: ProvenanceEnvelope + SpanProvenanceDelta + effective()
(page envelope ⊕ span delta; zero-cost inheritance when uniform). Liveness/
Staleness/OverlayState enums. 6 tests green. (blueprint §7.3, §11)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 08:55:35 +02:00
3b4f10a349 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 08:45:49 +02:00
f2dd2e124a workplan: SHARD-WP-0007 foundation implementation (first vertical slice)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 07:36:36 +02:00
1f3aba7aad chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 02:26:48 +02:00
d65f9e21f3 spec(SHARD-WP-0002): adapter contract (TSD §A, T11-T16+T18); workplan done
Adds the normative Shard Adapter Contract as TechnicalSpecificationDocument §A:
A.1 versioned capability contract (verbs + orthogonal-core spectra), A.2 verified
conformance suite (profiles not self-asserted), A.3 attachment-mode taxonomy +
image-is-not-a-store boundary, A.4 page model (incl. computational shapes,
identity/placement/equivalence, layered provenance), A.5 history portability,
A.6 syntax translation + fidelity report, A.7 addressing/navigation, A.8 gated
computational content. Updates TSD references/UC-count/next-work. Flips all 18
WP-0002 tasks + workplan done. Design layer complete.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:25:50 +02:00
802a80231a spec(SHARD-WP-0002): FederationArchitecture.md — federation design (T1-T10, T17)
Decision records for the 10 federation topics + the federation-model taxonomy,
each with a Decided/Deferred/Open footer: star orchestrator positioning, remix
primitives (overlay-default), identity/placement/equivalence + chorus,
event-sourced coordination journal + per-space append authority, server-primary
incremental union, notify-driven freshness, space lifecycle, reference-not-copy
transclusion, detection-core/resolution-policy preset catalog, the federation-
ops capability matrix, and the six selectable/composable federation models.
References (not duplicates) CoreArchitectureBlueprint + FederationRequirements.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:24:11 +02:00
40575045ca chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 02:20:55 +02:00
a2f15806f5 spec(SHARD-WP-0001): yawex-derived federation requirements (ADR-01..06); workplan done
Delivers spec/FederationRequirements.md — six ADR-ready design notes mapping
the yawex prior art onto the hardened CoreArchitectureBlueprint:
ADR-01 cross-shard resolution (yawex PageLookUp states as a checklist, keyed
on identity), ADR-02 namespace/path + shard roles (path=placement not identity),
ADR-03 derived views core-vs-adapter (BackLinks/RecentChanges/AllPages/SiteMap
core, Search hybrid), ADR-04 concrete provenance envelope fields, ADR-05 overlay/
patch/append lifecycle, ADR-06 wikilink+red-link extension (model core, render UI).
Resolves findings open-Qs #1-#3; access-model thread ratified. Flips WP-0001 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:19:43 +02:00
9b5fceb451 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 02:04:39 +02:00
d39e7f7765 spec(SHARD-WP-0006 T5): track §C as O-8..O-12; refresh decisions/traceability; close-out
Adds O-8 (preset bundles), O-9 (shard sharing vs tenant partition), O-10
(span authz + transclusion ⊕), O-11 (union under unavailability), O-12
(append-log throughput) to §12. Refreshes §14 decisions (event-sourced
coordination, conformance, I-2 verification) and §16 traceability (round-2
review + WP-0006). Flips SHARD-WP-0006 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:03:58 +02:00
1ad70a9c8a spec(SHARD-WP-0006 T4): incremental-equivalence correctness + I-2 verification (§8.7)
Fixes B-4. Incremental delta is not additive: a change processes bucket
exits (retract unsupported edges) + entries (add) + propagation across
equivalence neighbours, not just new candidates. Adds an I-2 verification
mechanism: per-partition Merkle-style digest + background consistency-checker
vs sampled fold → scoped self-healing recompute on drift. I-2 now
eventually-verified, not asserted.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:02:33 +02:00
3a753a6f3b spec(SHARD-WP-0006 T3): adapter conformance suite (§6.6)
Fixes B-2. Capability profiles are verified, not self-asserted: the contract
ships a versioned conformance suite that exercises each declared verb/position
against observed behaviour; passing is an admissibility precondition (lying
profiles rejected at registration); mismatch reported as a capability diff.
Makes I-3 / §6.5 sound rather than aspirational.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:01:37 +02:00
08a2148079 spec(SHARD-WP-0006 T2): event-sourced coordination + per-space append authority
Resolves B-1+B-3. Coordination-canonical state = an append-only decision log
in the git journal (events: overlay/binding/alias/merge); queryable current
state = a derived fold (tier-3, indexed). Concurrency: one append authority
per space (lease/leader) → totally-ordered per-space log, read-your-writes
across instances, HA via re-grantable lease, partition yields to log integrity.
Updates §1, §4, §8.1, §8.6, §11. I-6 strengthened (coordination state is now
git-addressable history/patch/review), not bypassed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 02:00:43 +02:00
cbd29e0a32 spec(SHARD-WP-0006 T1): overview-reconciliation pass (§A)
Fixes A-1..A-4: §4 identity/placement/equivalence (removes the equivalence-
keys-on-identity contradiction), §4 projection (trivial default + extension
point) and provenance (layered), §10 policy surface (adds freshness,
conflict-resolution, compaction, tenant-partition knobs + preset-bundle note),
§3 diagram + §11 header (incremental-first, orthogonal-core). Overview now
matches the hardened body.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:58:48 +02:00
079652b61f chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 01:57:04 +02:00
84ab25eeb9 history+workplan: round-2 blueprint review; SHARD-WP-0006 hardening II
Records the round-2 critical review (history/260615-...-review-2) and
establishes SHARD-WP-0006 to: reconcile overview with hardened body (§A),
settle the journal/coordination-state model (event-sourced decision log;
single-vs-multi-writer concurrency — B-1+B-3), require an adapter conformance
suite (B-2), fix incremental-equivalence correctness + I-2 verification (B-4),
and track §C as O-8..O-11.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:56:42 +02:00
229d24d1cf chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 01:44:05 +02:00
f21b7b5259 spec(SHARD-WP-0005 T9): known scaling risks & open problems; close-out
Adds §12 'Known scaling risks & open problems' (O-1..O-7 with chosen
direction + revisit trigger); renumbers §13-17. Refreshes §14 decisions
(several earlier 'open' items now decided), §16 traceability (links the
review + per-finding section map), and I-1..I-13. Flips SHARD-WP-0005 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:43:14 +02:00
c895d33091 spec(SHARD-WP-0005 T8): tenant isolation of derived tier + history scaling
Fixes B-3/C-3. §9.1 structural per-tenant partitioning of the derived tier
(no shared cross-tenant cache; read-time filtering as defence-in-depth;
reconciles I-2+L5 per partition); new invariant I-13. §8.1 history stays
recoverable AND bounded (gc/repack, squash-compaction of churn preserving
recoverable endpoints, per-shard offload, anti-abuse hooks).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:41:09 +02:00
59c36ac9d1 spec(SHARD-WP-0005 T7): elegance pass — layered provenance, common-case projection, policy module
Fixes D-2/D-3/D-4. §7.3 effective-vs-own provenance (page envelope + span
deltas, near-zero per-span cost). §8.4 projection trivial-by-default
(plain lazy replication), derivation/liveness/view-registry as extension
points only for the computational/typed tail. §11 adds policy/ + provenance/
as dependency-free leaf rails (mechanism never in a rail), tightened import rules.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:39:43 +02:00
012d151fe8 spec(SHARD-WP-0005 T6): orthogonal core spectra + implied positions + interaction subset (§6.5)
Fixes D-1. ~6 independent core axes (substrate, write-granularity, opacity,
envelope, access, liveness) with the rest implied via published rules that
forbid impossible profiles; a small named axis-interaction table is the
degradation contract (proof obligation behind 'core logic written once').

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:38:02 +02:00
af840576e9 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 01:37:11 +02:00
0383d78440 spec(SHARD-WP-0005 T5): cache freshness & invalidation protocol (§8.8)
Fixes C-2 (invalidation). Per-binding modes (event-driven push / validator
poll / TTL), hybrid default, operational-envelope coupling (rate-limited
shards must not poll-per-read), single-flight coalescing + batched bulk
invalidation, freshness always represented in the provenance envelope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:36:51 +02:00
dc451b0f4e spec(SHARD-WP-0005 T4): scale the union — incremental-first + indexed equivalence (§8.7)
Fixes C-1/C-2. Incremental change-driven maintenance (notify->delta) is
primary; full rebuild is a rare, envelope-respecting, concurrent fallback
(not required cheap). Equivalence via blocking/LSH candidate-gen + verify +
incremental maintenance, replacing O(N^2). Index is derived, per-tenant.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:36:09 +02:00
04be66161e spec(SHARD-WP-0005 T3): consistency, concurrency & conflict model (§8.6)
Fixes bug B-2. States the guarantee (read-your-writes for journal-owned
coordination-canonical state; causal across the derived tier; eventual+
freshness-labelled for sharded inputs). Conflict detection+representation =
core mechanism, resolution = policy. Overlay-apply-under-drift semantics
(fast-forward / three-way / refuse+re-present) and journal ordering.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:35:00 +02:00
c8dbe9b573 spec(SHARD-WP-0005 T2): split page identity / placement / content equivalence
Fixes bug B-1: page identity is a stable assigned handle (survives edits),
not a content fingerprint; fingerprints identify versions/content for the
equivalence mechanism. Chain: identity -> placements -> equivalence. (§7.2)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:07:37 +02:00
b5d2cbc330 spec(SHARD-WP-0005 T1): re-frame state model — three states, derived=f(canonical)
Replaces the two-bucket thesis with sharded-canonical / coordination-canonical
(journal: overlays, bindings, aliases, merges) / derived-disposable. Fixes the
I-2 contradiction (curator bindings can't be rebuilt). Updates §1, I-2, §3
dependency rule, §4 abstractions, §8.4. (review A-1)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:06:54 +02:00
ee2449d987 chore(SHARD-WP-0005): link workstream + task ids 2026-06-15 01:05:17 +02:00
f1dc7aff61 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 01:03:30 +02:00
dd812abb81 history+workplan: CoreArchitectureBlueprint review; SHARD-WP-0005 hardening
Records the critical review (history/260615-...) and establishes SHARD-WP-0005
to fold its findings (A-1, B-1..B-3, C-1..C-3, D-1..D-4) into the blueprint:
correctness (state re-frame, identity/equivalence split, consistency model),
scale (incremental-first union, equivalence indexing, cache invalidation),
elegance (orthogonal spectra, layered provenance, common-case projection,
policy module), security/history scaling, and a known-open-problems section.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:01:19 +02:00
9b5b393519 spec: CoreArchitectureBlueprint — whole-system architecture from research
The optimal architecture synthesised from INTENT + the full research arc:
- Thesis: canonical at the edges, derived in the middle (orchestrator not engine)
- Dual narrow waist: adapter contract (15 capability spectra) + page model
- 6 layers + provenance/capability rails; L4 union/projection is a rebuildable cache
- Federation-model taxonomy (plural/composable); two-axis projection model;
  moldable view registry; identity != placement; computational content in scope
  as page-model+projection, out as execution platform
- Concrete src/ module layout with downward-only dependency rule
- Canonical data flows; policy surface; tradeoffs; full traceability to INTENT/UCs
References ArchitectureBlueprint.md as the L5 authorization sub-blueprint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:41:14 +02:00
6a88eb7710 chore(SHARD-WP-0002): link T17/T18 state_hub_task_ids; sync task drift
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:27:07 +02:00
d898307b7e chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 00:23:15 +02:00
90a854f841 spec(SHARD-WP-0002): re-fold synthesis v3 + computational page model (T17, T18)
Folds the later syntheses into the federation/adapter-contract workplan:
- T11 capability spectra 13 -> 15 (+provenance granularity, +computational/
  liveness); +derive-projection/execute verbs
- T12 four computational page shapes (UC-83/84) + typed-graph statements
- T13 paired-text/nbdime + DB-version-rows + partial-history honesty
- T14 git-IS-store + direct-DB + image-is-not-a-store boundary
- T15 structured-re-evaluable-value opacity point; non-Markdown computed
- T16 two-axis projection model + moldable view registry + named-chunk transclusion
- NEW T17 federation-model taxonomy (fork+journal / VCS-replication+ping /
  query-join / feed / activity-streams / engine-mirror; selectable+composable)
- NEW T18 computational/executable content (in scope as page-model+projection,
  out as execution platform; execute = gated capability)
- Context/decision-tables/acceptance/order updated; UC range -> UC-26-UC-84

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:22:59 +02:00
3d137c96b6 research: SHARD-WP-0004 synthesis (the computational page model); workplan done
Consolidates the 8 computational/interactive-knowledge dives into one
source/derivation/projection model on two axes (replication<->derivation,
live<->snapshot); four computational page shapes; recommendation that
executable content is in scope as a page-model+projection concern, out of
scope as an execution platform (capability-gated, degrade to snapshot;
no INTENT amendment). Flips SHARD-WP-0004 status: done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:12:03 +02:00
f2f9f31df8 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-15:
  - update .custodian-brief.md for shard-wiki
2026-06-15 00:04:50 +02:00
2c978d71f0 research: Strudel.cc live-coding REPL deep dive; SHARD-WP-0004 T5
Code as live time-based audio performance; the far live end of the
live<->snapshot axis (no faithful static form; static = source + marked
recording). The honesty test for graceful degradation. Enrichment-only
(UC-54/55). Marks T5 done — all 8 batch tasks complete.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 00:04:16 +02:00
b676579407 research: Processing / p5.js deep dive; SHARD-WP-0004 T4
Program-as-page rendered live at view time (no cached output). Adds
materialization-timing (ahead-of-time vs view-time) + continuity
(one-shot vs continuous) facets to the projection model; execute-in-viewer
= capability+trust. Enrichment-only (UC-54/55). Marks T4 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:57:57 +02:00
7fb90219b5 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 23:55:24 +02:00
56b6cdd110 research: Squeak & Pharo deep dive (image-based Smalltalk); SHARD-WP-0004 T6+T8
Combined memo (justified merge). The image = purest 'live' end; hardens
the image-is-not-a-store boundary (export->files only), generalizing
'attach files not the kernel/image'. Pharo Tonel/Iceberg confirms even
image traditions externalize to git text. Names the live<->snapshot
projection axis (T16). Boundary/enrichment-only, no new UC. Marks T6+T8 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:54:48 +02:00
4acadacfee chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 23:46:19 +02:00
1546ca09bf research: Mathematica Notebooks deep dive; SHARD-WP-0004 T2
Original computational notebook (.nb = a Wolfram expression); confirms
UC-84 notebook page shape is a genus. Refinements: nestable cell groups
(outline tree), structured re-evaluable outputs (new opacity point).
Manipulate/Dynamic = snapshot-only. Enrichment-only (UC-84/54/55).
Marks T2 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:45:45 +02:00
a3ca3e975b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 23:39:36 +02:00
25091dbd2e research: Glamorous Toolkit deep dive (moldable development); SHARD-WP-0004 T7
Moldable views (gtView) = open, type-keyed set of co-equal, computed
projections, none canonical = a moldable view registry refining the
projection model (T16). Lepiter live notebook over git files. Enrichment-
only (UC-47/48/54); no new UC. Marks T7 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:38:56 +02:00
1e3aabc143 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 23:22:19 +02:00
25a714efa7 research: Jupyter Notebooks deep dive; UC-84 (SHARD-WP-0004 T3)
.ipynb JSON cells + embedded computed outputs with fragile execution
provenance; derived output stored inside the source. Non-Markdown/lossy;
kernel = capability, default = present snapshot + static render.
Enriches UC-54/55/59/35; links UC-32/83/79. Marks T3 done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 23:08:13 +02:00
3eb026139f chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 22:48:08 +02:00
5ba6975573 chore(SHARD-WP-0004): mark T1 done
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 22:47:40 +02:00
d4afce3699 research: literate programming deep dive (WEB/weave/tangle); UC-83 (SHARD-WP-0004 T1)
One source → N co-equal derived projections (weave=docs, tangle=code);
named-chunk transclusion; splits replication- vs derivation-projection.
Generalizes compile-to-static (UC-79). Enriches UC-32/44/79; links UC-54.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 22:43:30 +02:00
2397c2c4b3 spec(UseCaseCatalog): restructure into 7 thematic sections (no renumbering)
Section A had accumulated 64 of 82 UCs (every dive appended to its end),
collapsing four distinct themes into one heading. Regroup the orchestration/
adapter UCs into four coherent sections and re-letter the authoring/reading
sections, with UC numbers kept as stable IDs (never renumbered -- they are
referenced across research/, the synthesis, and the workplans):

  A. Information space, federation & coordination
  B. Shard attachment & adapter binding
  C. Page model, structure & content fidelity
  D. Addressing, identity & query
  E. Knowledge work and collaboration   (was B)
  F. Discovery and navigation           (was C)
  G. Provenance and page metadata       (was D; + UC-75 statement provenance)

UC blocks moved byte-for-byte (verified: heading set and Traceability tail
identical to pre-restructure); added an Organization note to Conventions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 22:20:13 +02:00
a7084ec74b spec(UseCaseCatalog): fix open-question numbering + add marker legend
- Renumber stray open-question 23->31 (mis-numbered during the WP-0003 batch;
  questions now run 1-32 sequentially) and add Q32 (federation composition,
  from synthesis v3 S2.5).
- Add a dive-source marker legend at the top of Traceability (18 markers were
  used across the matrix with no single key).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:46:09 +02:00
700566b1e2 research: shard-spectrum synthesis v3 (folds SHARD-WP-0003 batch)
Extends the capability model to the nine WP-0003 engines (~23 systems total).
Per-shard spectra 13 -> 14 (adds provenance granularity); refines merge-model
(+fork/journal-replay, +coexist-with-rank), attachment-mode (+git-IS-store,
+container, +direct-DB, +REST/file-store-hybrid, +external-API payload-format
facet), native-query (+SPARQL/RDF far-end), history (+DB-rows, +partial),
structure (+typed-graph, +inline-embedded objects), content-opacity
(+lossy-exportable), write-granularity (+story-item, +section-anchor).

Headline: new federation-model taxonomy (S2.5) -- federation is plural
(fork+journal / VCS-replication+ping / query-time graph join / feed
aggregation / activity streams / engine-mirror), a selectable/composable
coordination-layer axis feeding SHARD-WP-0002 T1-T6. Folds UC-68-82 into the
workplan; no new UCs, no INTENT boundary change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:35:43 +02:00
af5841a0d7 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 21:15:49 +02:00
8941e131e8 research: UseModWiki deep dive (flat-file ancestor); SHARD-WP-0003 done
SHARD-WP-0003 T9 (final). Lineage dive: the flat-file ancestor (Clifford Adams
c.2000, from AtisWiki/CvWiki) that early Wikipedia ran on (MediaWiki Phase I).
No new capability -- reinforces the minimal flat-file floor (UC-82) and
CamelCase linking (UC-25); historical grounding that the file-store floor is
the field's common root. Enrichment-only: UC-01/40/25/36/41. Marks T9 done and
SHARD-WP-0003 complete (9/9). Feeds SHARD-WP-0002 T11.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 21:14:26 +02:00
ce4b0cb9e4 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 21:00:52 +02:00
6878a0c184 research: Oddmuse deep dive (minimal single-script wiki); UC-82
SHARD-WP-0003 T7. The minimal file-store floor: one Perl CGI script over
plain-text page files + a keep/ revision dir, no DB/API. Anchors the
graceful-degradation baseline / minimal capability-profile floor -- every
richer shard is measured against it; capability-awareness in its purest form
(profile must express absence; truncated history reported honestly). UC-82.
Enriched UC-40/01/36/41. Marks T7 done. Feeds SHARD-WP-0002 T11/T13.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:59:20 +02:00
87f30e7c62 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 20:55:29 +02:00
e0c3d0699c research: MojoMojo deep dive (Perl Catalyst DB-backed wiki); UC-81
SHARD-WP-0003 T8. Classic relational-DB-backed wiki: Catalyst/DBIx::Class app,
pages + path tree + full history in SQL tables, Markdown body in a column, no
file store and no content API. Anchors the direct-DB-read binding (map schema
-> page model + journal); DB version rows = a third history source beside git
commits and RCS files. UC-81. Enriched UC-02/40/36/34. Marks T8 done.
Feeds SHARD-WP-0002 T14/T13/T11.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:54:40 +02:00
1e962a0aef chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 20:50:58 +02:00
54468b60bc research: Salesforce Quip deep dive (SaaS live docs + embedded objects); UC-80
SHARD-WP-0003 T6. Enterprise document-collab contrast to Notion: a doc +
spreadsheet hybrid where a page is prose interleaved with embedded live
structured objects (spreadsheets, kanban/calendar live apps), reachable only by
REST with HTML import/export (lossy to Markdown), gated by Salesforce
identity/ACL. Generalizes external-API mode with an HTML payload-format facet
(beside Notion block-JSON, Wiki.js GraphQL); pushes page model toward
inline embedded structured objects. UC-80. Enriched UC-57/55/58/36/06. Marks
T6 done. Feeds SHARD-WP-0002 T11/T12/T14.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:50:09 +02:00
9f0fc57428 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 20:41:16 +02:00
ee805d5af7 research: ikiwiki deep dive (compile-to-static, git-distributed); UC-79
SHARD-WP-0003 T4. Wiki compiler: git-canonical Markdown source -> static HTML
(derived publish/projection, not the canonical store). Two new insights atop
the git-IS-store home case: compile-to-static = clean canonical-source vs
derived-output separation (attach source repo, UC-37/UC-56), and a third
federation flavor -- git replication + XML-RPC change-ping (pinger), beside
fedwiki fork/journal (UC-72) and Wikibase SERVICE (UC-74). UC-79. Enriched
UC-31/56/37/33. Marks T4 done. Feeds SHARD-WP-0002 T4/T6/T14.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 20:40:29 +02:00
188643b0f5 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 19:47:23 +02:00
fee47514a2 research: TiddlyWiki deep dive (single-file wiki, write-granularity extreme); UC-78
SHARD-WP-0003 T3. Whole-file write-granularity anchor: an entire wiki (content
+ app engine) in one self-contained HTML file -> save rewrites the whole file,
no per-page atomicity. Node .tid file-per-tiddler substrate is git-diffable/
fine-grained, so the same engine spans the granularity spectrum by substrate
(cf. Logseq file/DB UC-62, backend-swap UC-43). Tiddler = flexible-field
record (UC-34); filter expressions = native-query tier (UC-52). UC-78
(single-file container-format attach). Enriched UC-35/40/34/52/43. Marks T3
done. Feeds SHARD-WP-0002 T11/T14.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:46:37 +02:00
1281ca689b chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 19:41:27 +02:00
dbc42f05db research: git-forge wikis deep dive (Gitea/GitLab/GitHub); UC-76-77
SHARD-WP-0003 T5. The home case: a forge wiki is a separate .wiki.git repo of
Markdown -> page model, history, coordination journal map 1:1 with near-zero
adapter. git-clone universal across all three; wiki content API
capability-varying (GitLab/Gitea yes, GitHub git-only). git IS the canonical
store (not a mirror), so write-by-commit is safe -- resolves the UC-68/Q22
race for this case. UC-76 (clone .wiki.git attach), UC-77 (forge wiki API,
capability varies). Enriched UC-40/02/68/38. Marks T5 done.
Feeds SHARD-WP-0002 T14/T11.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:40:46 +02:00
31ec8ab205 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 19:31:46 +02:00
a6629bdb29 research: Wikibase/Wikidata deep dive (entity-statement graph, SPARQL); UC-73-75
SHARD-WP-0003 T2. Structure & native-query far-end: typed knowledge graph
(items/properties, statements = claim+qualifiers+references+rank), RDF
projection + SPARQL (WDQS/Blazegraph) incl. federated SERVICE, opaque stable
Q/P identity (labels-as-annotation), statement-level provenance. UC-73
(typed-graph shard, lossy render), UC-74 (SPARQL + federated query), UC-75
(per-assertion provenance). Enriched UC-34/58/52/24. Marks T2 done.
Feeds SHARD-WP-0002 T12/T16.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:31:08 +02:00
ecb7d0154e chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 19:05:38 +02:00
d955fa239c chore(SHARD-WP-0003): mark T1 (Federated Wiki) done
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:05:15 +02:00
036dbad816 research: Federated Wiki deep dive (journal/fork/neighborhood); UC-70-72
SHARD-WP-0003 T1. Federation model (not a shard candidate): per-page
append-only semantic-action journal with story as derived replay,
fork-with-site-provenance, neighborhood/roster discovery + chorus of forks.
Prior art for shard-wiki's own pillars: coordination journal (UC-71),
overlay-before-mutation (UC-26 fork), union-without-erasure (UC-72).
Attach as REST/file-store hybrid (page JSON + CORS, UC-70). Feeds
SHARD-WP-0002 T1-T5, T11, T13, T16.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 19:01:13 +02:00
9acd4e2841 plan: persist state-hub workstream/task IDs for SHARD-WP-0003 and SHARD-WP-0004
State-hub consistency sync created both workstreams and all 17 tasks and
wrote the IDs back into the workplan frontmatter and task blocks; commit
them so the files match the DB.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 18:29:15 +02:00
5ada333bb4 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 18:28:38 +02:00
9b5defa6d8 plan: add SHARD-WP-0003 (engine-dive batch) and SHARD-WP-0004 (computational-knowledge systems)
SHARD-WP-0003 — nine wiki-engine deep dives chosen against the synthesis
coverage review: high-insight (Federated Wiki, Wikibase/Wikidata,
TiddlyWiki, ikiwiki), the INTENT-named git-forge wikis (Gitea/GitLab/
GitHub), and four for breadth/lineage (Salesforce Quip, Oddmuse, MojoMojo,
UseModWiki). Each task = persist a deep-dive memo + promote/enrich the
use-case catalog + log adapter-contract notes for SHARD-WP-0002;
post-batch synthesis-v3 decision.

SHARD-WP-0004 — eight computational / interactive-knowledge systems as
design prior art for executable/computational/live page content and
one-source-many-projections: Knuth WEB/weave/tangle, Mathematica & Jupyter
notebooks, Processing/Processing.js, Strudel.cc, Squeak, Glamorous
Toolkit, Pharo. Carries the question of whether a page can be a live
computational artifact; closes with a "computational page model"
synthesis. Both workplans authored file-authoritative (no state_hub IDs;
sync will create+link workstreams and tasks).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 18:28:06 +02:00
4543f5dc43 research: Wiki.js deep dive (storage-module engine, DB<->Git Markdown, GraphQL); UC-68/69
The closest existing engine to shard-wiki's own design: DB-canonical
(Postgres/MySQL/SQLite) but with a pluggable storage-module abstraction
that bidirectionally syncs clean Markdown (+ YAML frontmatter) to Git
(also FS/S3/Azure), each provider acting as backup or source of truth.
Two big findings: (1) the storage-module interface is concrete
adapter-contract prior art alongside Foswiki::Store, and the closer one
(medium = Markdown in Git); (2) the engine-maintained bidirectional Git
mirror is the ideal file-store attach (clean MD + git history) and, being
bidirectional, makes git commit a write path (overlay/patch as a commit,
no API). Also GraphQL API (introspection = capability discovery;
selective fields = efficient projection) and authn-delegated auth modules
+ path-based rule ACL. Added UC-68 (engine-maintained bidirectional Git
mirror, write-by-commit), UC-69 (typed/introspectable API for schema
discovery + selective projection); enriched UC-06/36/38/40/42/57. Catalog
now 69 UCs. Architecture for SHARD-WP-0002 T11/T14: storage-module
abstraction as 2nd adapter-contract prior art, engine-maintained Git
mirror as attach+write surface, GraphQL introspection for capability
discovery.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 17:50:27 +02:00
714eaf6b5a synthesis v2: extend shard spectrum to full dive set (13 spectra); re-fold into SHARD-WP-0002
Extend the cross-dive synthesis from nine systems to the full fourteen
(added Joplin, Logseq, the CRDT cohort Anytype/AFFiNE/AppFlowy, Trilium).
Spectra grown 11 -> 13: new merge-model (none/git/conflict-notes/
native-CRDT) and content-opacity (plaintext/whole-shard-E2EE/per-item),
plus identity-vs-placement emphasis (Trilium note/branch DAG), Logseq's
in-file id:: addressing sweet spot, build-your-own derived query index,
and a six-mode attachment taxonomy (file-store native/interchange-mirror,
in-engine-host, local-REST, external-API, CRDT-replica, P2P). New
through-lines: CRDT changes the merge math; identity != placement;
metadata can be computed; attach surface != native store (+ substrate
migration). UC fold-in extended UC-44-59 -> UC-44-67.

Folded into SHARD-WP-0002: T11 thirteen spectra; T12 computed metadata +
multi-placement/DAG identity; T13 CRDT-log supplement; T14 full six-mode
attachment taxonomy; T15 HTML source model; T16 identity!=placement +
derived index. Context inputs += four dives; acceptance UC-26-67. No new
tasks, no new UCs (synthesis only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 17:25:32 +02:00
5697669063 research: Trilium (TriliumNext) deep dive (note cloning/DAG hierarchy, attribute inheritance, HTML-native); UC-66/67
SQLite-local PKB whose standout is note cloning — a single note can sit in
multiple tree locations at once via the note (identity) vs branch
(placement) split, so the hierarchy is a DAG, not a tree, with no single
canonical path. The identity-not-equal-placement model is the clean way to
represent a page in multiple locations/shards and the namespace-level form
of the clone/reference primitive. Also: attributes (labels + typed
relations) are inherited + templated, so metadata is computed (own +
inherited + template), not a flat bag; content opacity is per-item
(per-note encryption / protected notes), refining the proposed 12th
spectrum; HTML-native (CKEditor, lossy to Markdown); dual extension
surface (scripting code notes + ETAPI token REST). TriliumNext is the
active community fork of zadam's Trilium (TWiki->Foswiki pattern). Added
UC-66 (DAG hierarchy / note cloning), UC-67 (inherited/templated
attributes, effective vs own); enriched UC-15/22/34/38/42/61. Catalog now
67 UCs. Architecture for SHARD-WP-0002 T11/T12/T14/T15/T16: DAG namespace
+ identity/placement split, computed/inherited metadata, per-item content
opacity, HTML source model, scripting + ETAPI host surfaces.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 16:44:49 +02:00
3e0c05a2b3 research: local-first workspaces cohort deep dive (Anytype/AFFiNE/AppFlowy) — CRDT substrate; UC-64/65
Combined cohort memo for the three open-source local-first all-in-one
workspaces, studied together because they share the trait none of the
prior eleven systems had: the store is a CRDT (Anytype any-sync, AFFiNE
Yjs/y-octo, AppFlowy Yrs). The cohort thesis: CRDT changes the adapter
math — the backend performs conflict-free merge itself (shard-wiki must
not impose git/text merge; speak the CRDT via a replica or stay
projection/overlay), history is a CRDT update log (supplement not git),
and the attach surface is a replica or self-host sync endpoint, with
Anytype adding P2P (no single endpoint) + E2EE (opaque). Added UC-64
(CRDT-synced shard with native merge), UC-65 (P2P/decentralized shard,
no single canonical endpoint); enriched UC-36/47/48/51/55/57/58/61.
Catalog now 65 UCs. Architecture for SHARD-WP-0002 T11/T14: merge-model
capability (proposed 13th spectrum: none/git/native-CRDT), CRDT-replica
substrate + P2P/no-central-endpoint attach mode, endpoint-model
capability, MCP as an integration surface. Boundary: CRDT local-first
candidate shards — attach a replica/projection as pages, respect native
merge, never re-drive their sync; not substrates and not the federation
layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 16:21:08 +02:00
6ccf349209 research: Logseq deep dive (block-graph on plain Markdown files, in-file block IDs, derived Datalog index); UC-62/63
Occupies the point the other modern tools leave empty: block-graph
semantics (UUID-addressable, embeddable, queryable blocks) stored as plain
Markdown/Org files on disk, with a DataScript graph derived from the files
(files canonical, index derived). The bridge between Roam (block-DB) and
Obsidian (file-over-app). Headline finding: Logseq resolves the
addressing-spectrum tension — block-level addressing that is also
git-diffable in-file text (id:: property) — and proves a file-backed shard
can serve rich Datalog queries via a derived index. Also: file->SQLite
"DB graph" migration is a live UC-43 (substrate swap under stable
identity); whiteboards = non-Markdown content; dual-attachable (file-store
direct with a Logseq format profile, or in-app plugin). Added UC-62
(attach block-graph-on-plain-files shard), UC-63 (serve structured
queries over a file shard via a derived index shard-wiki builds — converse
of UC-52); enriched UC-32/34/43/50/51/52/55. Catalog now 63 UCs.
Architecture for SHARD-WP-0002 T11/T14/T16: Logseq format profile,
derived-query-index capability, substrate-migration tolerance, in-file
block addressing as the T16 span-address target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 15:27:37 +02:00
e00c160a43 research: Joplin deep dive (SQLite-local/Markdown-on-sync, interchange-format attach, E2EE content opacity); UC-60/61
Architecturally distinct from the Obsidian/Roam/Notion trio: content is
Markdown, the local store is SQLite, and the sync representation is a
folder of per-item Markdown+metadata files on a backend of choice
(filesystem, WebDAV, Nextcloud, Dropbox, OneDrive, S3, Joplin
Server/Cloud). Two new findings: (1) attach the sync/interchange mirror,
not the app or the SQLite store, offline and app-independent, landing on
INTENT's WebDAV/Nextcloud/S3 backends (UC-60, distinct from native
on-disk store UC-40 and app-files UC-53); (2) E2EE => content opacity, a
proposed twelfth capability spectrum for encrypted-at-rest shards (UC-61).
Also: Joplin is the file-sync-daemon boundary case (attach the mirror as
pages, never re-sync); store-minted page-level :/id links (addressing
spectrum middle); dual surface (plugin host + local Data API on
localhost:41184). Enriched UC-31/36/38/40/51/55. Catalog now 61 UCs.
Architecture for SHARD-WP-0002 T11/T14: interchange-format attach,
content-opacity field, local-REST sub-mode, format-aware file-store
profiles.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 14:21:31 +02:00
26bee2cc84 chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 14:02:06 +02:00
fae3cbde8b synthesis: shard spectrum across nine systems; fold into SHARD-WP-0002 (T11 eleven spectra, new T16)
Cross-dive synthesis (research/260614-shard-spectrum-synthesis/) reading
the two Nelson conceptual systems, four engines, and three modern tools
across each other: a shard family matrix and eleven capability spectra
(addressing, content identity, structure, history, native query,
translation, attachment mode, operational envelope, access grant, write
granularity, content types) — positions anchored at both ends by a real
system, federation ops degrading by position. Through-lines:
files-canonical/index-derived wins; fine-grained addressing is adoptable;
transclusion=clone=embed is one primitive; structure/history federate iff
in-text; attach mode is a per-binding choice; Notion proves the platform
can enforce no-silent-mutation.

Folded into SHARD-WP-0002: T11 reframed around the eleven spectra; T12
page model stretched four ways (prose + typed records + non-Markdown
assets + query-defined); T13 adds Notion supplement; T14 rewritten as the
three-mode attachment taxonomy (file-store / in-engine-host /
external-API); T15 adds lossy-with-fidelity-report; new T16 (addressing,
content identity, dimensional/query navigation). UC coverage UC-34-43 ->
UC-34-59. Workplan now 16 tasks. No new UCs (synthesis only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 14:01:49 +02:00
64796f7b3a research: Notion deep dive (closed block-DB SaaS, external REST API only, database-as-pages); UC-57/58/59
The closed/hosted/schema-rich extreme: everything is a block (UUID id,
type, properties, ordered child content, single parent), pages are blocks
and database rows are pages with a schema, Postgres-backed hosted SaaS.
Databases add typed properties + relations + rollups + formulas across
many views = the apex of wiki-page-as-structured-record. Extension model
has no in-app plugin runtime; the only extensibility is the external REST
API (+ webhooks 2026) inside a tight envelope (~3 rps, eventual
consistency, recursive child fetch, scoped/revocable per-page grants).
Adds the third attachment mode (external-API-only) alongside file-store
(Obsidian/TWiki) and in-engine host (Roam/XWiki); Notion enforces no
silent remote mutation via scoped grants. Added UC-57 (attach closed
external-API-only shard w/ operational envelope + scoped grant), UC-58
(typed database w/ schema+relations+views, no flattening), UC-59
(lossy-aware translation w/ fidelity report); enriched
UC-31/34/36/39/50/51/52/54/56. Boundary: one external-API candidate shard,
best as projection/mirror/overlay/backup, not a substrate and not the
federation layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 13:44:09 +02:00
f0be5799aa research: Obsidian deep dive (file-over-app vaults, plugin API, ecosystem-popularity signal); UC-53/54/55/56
The most INTENT-aligned tool yet and the file-backed counterpart to Roam:
file-over-app vaults (plain .md folders, files canonical, MetadataCache a
derived index), in-file git-diffable addressing/structure (^block-id,
wikilink embeds, YAML frontmatter), and a plugin API (Plugin onload/onunload
over App.vault/metadataCache/workspace) that doubles as an adapter host —
so a vault is dual-attachable (file-store direct or in-app plugin). Mined
the plugin download rankings as demand evidence per the research brief:
#1 is drawings (Excalidraw, non-Markdown content), query-as-DB
(Dataview/Tasks) is top-tier but an add-on, Git is top-7 (bolt-on history),
Remotely Save shows sync-to-anywhere demand. Added UC-53 (attach local
vault w/ live concurrent native editor), UC-54 (query-defined dynamic
page), UC-55 (non-Markdown content types), UC-56 (outbound publish of a
projection); enriched UC-15/28/34/36/40/51/52. Boundary: a vault is one
file-backed candidate shard, not the federation layer and not a file-sync
target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 12:51:22 +02:00
dfff9ab42e research: Roam Research deep dive (block-graph DataScript DB, transclusion, datalog, Roam Depot extension API); UC-50/51/52
The modern bookend to the Xanadu/ZigZag dives: where those are unbuilt
ideals, Roam shipped fine-grained addressing (:block/uid), live
transclusion (block embeds), bidirectional links, and a queryable
structured space (DataScript datoms + Datalog). Studied as a candidate
DB-backed/API-attached shard (XWiki family) and as a concrete
engine-hosts-adapter surface (Roam Depot onload/onunload over
window.roamAlphaAPI). Added UC-50 (attach block-graph DB shard, block<->page
mapping), UC-51 (adopt native span IDs as portable span addresses), UC-52
(delegate derived views to a shard's native query engine); enriched
UC-32/34/35/38. Boundary: Roam is one candidate shard mapped into the
Markdown-first page model, not a substrate and not the federation layer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 11:36:50 +02:00
7c22c468af chore(consistency): sync task status from DB [auto]
Updated by fix-consistency on 2026-06-14:
  - update .custodian-brief.md for shard-wiki
2026-06-14 02:15:13 +02:00
c1424e8863 research: ZigZag/zzstructure deep dive (information space as orthogonal dimensions); UC-47/48/49
Evaluated modelling a wiki information space as a zzstructure: a page is
a cell at the intersection of many co-equal dimensions (namespace tree,
created-from genealogy, version, shard, equivalence, links).
Recommendation: adopt zzstructure as a navigation/visualization/indexing
LENS (a derived dimensional projection over the union), not as the
storage substrate — Git and sovereign shards stay canonical, and the
many-to-many link graph does not fit the one-neighbour-per-dimension rank
rule. Clone <-> transclusion convergence with the Xanadu dive. Added
UC-47/48/49; enriched UC-05/22/26/29; dimensional projection layer +
genealogy-edges-in-journal logged as architecture for SHARD-WP-0002.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 02:14:04 +02:00
0a335eca8b scope: reflect Xanadu deep dive (research row + UC count 25->46)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 01:34:42 +02:00
60e1aa661c research: Project Xanadu deep dive (EDL/reference-not-copy, transclusion, addressing); UC-44/45/46
Xanadu studied as conceptual ancestor, not a candidate shard. Yield:
reference-not-copy EDL/xanadoc validates projection+overlay+union;
content-identity bidirectional transclusion; portable span-address
(tumbler) problem logged as adapter-contract architecture for
SHARD-WP-0002. Recorded design-bug boundaries: reject
single-global-docuverse, single-canonical-instance, baked-in economic
policy. Added UC-44/45/46; enriched UC-24/27/29/32.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 01:24:45 +02:00
316fdeedc2 spec(SHARD-WP-0002): fold adapter-contract constraints into T11-T15
Distill the cross-cutting adapter-contract constraints from the four engine
deep dives (wiki-engines landscape, XWiki, TWiki, Foswiki) into five concrete
design tasks, complementing the federation tasks (T1-T10):

- T11 adapter contract: capability model & versioned interface (Foswiki::Store
  prior art; write-granularity as a capability dimension)
- T12 page model: structured/typed payload representation (XObjects, %META%,
  multi-record, bodiless pages)
- T13 history portability: supplement (DB-internal) vs import (open file history)
- T14 adapter binding: attach path, engine-hosted vs external, backend-swap
- T15 syntax translation & content fidelity (non-Markdown round-trip)

Adds an adapter-contract decision-topics table, cross-links T10<->T11 on the
shared capability vocabulary, and updates context/acceptance/task-order. Tasks
registered in the state hub (workstream 2af4c46d).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:35:09 +02:00
fa7adab239 research: Foswiki deep dive (store abstraction, ext API); UC-42/43
Deep dive into Foswiki focused on its deltas from TWiki (not the shared
lineage): the pluggable Foswiki::Store backend (RcsWrap/RcsLite/PlainFile)
behind a versioned interface via Foswiki::Meta, the OO/MVC core rewrite,
Foswiki::Func + registerTagHandler, DataForms + MetaDataPlugin multi-record,
and WysiwygPlugin TML<->HTML round-trip. The store abstraction is logged as
prior art for shard-wiki's own adapter contract (SHARD-WP-0002).

Catalog (now 43 UCs):
- UC-42 read/write a non-Markdown shard via lossless syntax translation
  (Markdown-first for prose; Foswiki WysiwygPlugin is proof)
- UC-43 tolerate a shard's storage-backend swap (RCS<->PlainFile) under a
  stable identity
- enrich UC-39 (multi-record metadata) and UC-40 (PlainFile direct-attach)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:27:24 +02:00
ffd5459b3e research: TWiki deep dive (impl, plugin API, ecosystem); UC-40/41
Deep dive into TWiki as the file-based Perl counterpoint to XWiki: flat-file +
RCS store (data/<Web>/<Topic>.txt), Webs/Topics, TWiki Forms storing fields as
%META% in the topic text, TWikiML/variables, TWiki::Func API; the plugin handler
callback surface (initPlugin, commonTagsHandler, before/afterSaveHandler,
afterRenameHandler, REST handlers) and package types (Plugin/Skin/AddOn/Contrib);
per-topic ALLOW/DENY access control (origin of yawex's model); Foswiki fork.

Catalog (now 41 UCs):
- UC-40 attach a file-backed engine's on-disk store directly (dual-path attach)
- UC-41 import an engine's native file history (RCS .txt,v) into the journal
- enrich UC-06 (TWiki per-topic ACL lineage), UC-34 (file-embedded %META%),
  UC-36 (RCS-import vs DB-supplement), UC-38 (TWiki handlers as adapter host)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 23:20:11 +02:00
27da940df1 research: XWiki deep dive (impl, extension interfaces, ecosystem); UC-38/39
Deep dive into XWiki past the landscape scan: component/DI architecture,
document + class/object data model, oldcore, Hibernate storage + xwikircs
history, ObservationManager events, rendering pipeline, multi-wiki; the full
extension-interface surface (components, Java/wiki macros, script services,
UIX/UIXP, JAX-RS REST, event listeners, resource handlers); and the
extensions.xwiki.org ecosystem (XAR/JAR/WebJar, 900+ extensions).

Catalog:
- UC-38 make a wiki engine federation-capable via its native extension API
  (composable integration — first engine-side-direction UC; XWiki is proof)
- UC-39 attach a wiki-as-application-platform shard (bodiless typed pages)
- enrich UC-31 (ObservationManager event-driven sync), UC-34 (XObject model),
  UC-36 (xwikircs internal history) with concrete XWiki specifics

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 19:20:42 +02:00
0473c3f043 spec: extract use cases from wiki-engines landscape research (UC-34–UC-37)
Mine the one research dir not yet promoted (260608-wikiengines-overview) into
the catalog. Adds four heterogeneous-backend attachment scenarios the other
research didn't surface:

- UC-34 attach structured/semantic shard without lossy flattening (SMW, XWiki)
- UC-35 attach shard with coarse write granularity (TiddlyWiki single-file)
- UC-36 supply git-addressable history to internal-history engine (Confluence)
- UC-37 attach static export as read-only backup shard (MediaWiki dump)

Adds a `wikiengines` source token + traceability column and a wikiengines
mapping subsection. The scan mostly reinforced existing UCs and the L0→L4
ladder; its main yield is adapter-contract constraints tracked in SHARD-WP-0002.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 18:39:30 +02:00
193 changed files with 22699 additions and 226 deletions

20
.claude/rules/agents.md Normal file
View File

@@ -0,0 +1,20 @@
## Kaizen Agents
Specialized agent personas available on demand via the state-hub MCP.
**Discover:** `list_kaizen_agents()` — returns all agents with name, description, category
**Load:** `get_kaizen_agent("tdd-workflow")` — returns full instructions; read and follow them
Common agents:
| Agent | Category | When to use |
|-------|----------|-------------|
| `tdd-workflow` | testing | Step-by-step TDD8 workflow for any feature |
| `code-refactoring` | quality | Code quality analysis and safe refactoring |
| `test-maintenance` | testing | Diagnose and fix failing tests |
| `requirements-engineering` | process | Prevent interface/mock mismatches upfront |
| `keepaTodofile` | process | Maintain TODO.md during work |
| `project-management` | process | Track status, determine next steps |
| `datamodel-optimization` | quality | Optimize dataclasses and data structures |
All 17 agents: call `list_kaizen_agents()` for the full list.

View File

@@ -0,0 +1,8 @@
## Architecture
<!-- TODO: Describe the key design decisions and component structure.
Key modules, data flows, external integrations, state machines, etc. -->
## Quick Reference
`~/state-hub/mcp_server/TOOLS.md` — MCP tool reference

View File

@@ -0,0 +1,50 @@
# Credential and access routing
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
for inference. Run this check **before** requesting secrets, API keys, SSH access,
login tokens, or database passwords — in any repo, not only `ops-warden`.
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
other credential need belongs to another subsystem. **Do not** message
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
### Lookup (do this first)
```bash
warden route find "<describe your need>" --json
warden route show <catalog-id> --json
```
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
| Agent runtime | How to orient |
| --- | --- |
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=shard-wiki` is for coordination, not secret vending |
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
### Quick routing table
| I need… | Owner | ops-warden executes? |
| --- | --- | --- |
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes**`warden sign` |
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
| Authorization decision | flex-auth | No — route only |
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
### Anti-patterns (do not do these)
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
- Pasting secrets into Git, State Hub, workplans, logs, or chat
### Other capabilities (reuse-surface)
Non-credential capabilities are usually discovered through **reuse-surface** federation
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
every repo's agent instructions because it is high-frequency, high-risk, and easy to
get wrong.
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`

View File

@@ -0,0 +1,38 @@
## First Session Protocol
Triggered when `get_domain_summary("consumer")` shows **no workstreams**.
The project is registered but work has not yet been structured.
**Step 1 — Read, don't write**
- `~/the-custodian/canon/projects/consumer/project_charter_v0.1.md` — purpose, scope
- `~/the-custodian/canon/projects/consumer/roadmap_v0.1.md` — planned phases
- Scan repo root: README, directory structure, existing code or docs
**Step 2 — Survey in-progress work**
Look for TODOs, open branches, half-finished files. Note done vs. started but incomplete.
**Step 3 — Propose workstreams to Bernd**
Propose 13 workstreams — each a coherent strand, weeks to months, anchored to a
roadmap phase. **Wait for approval before creating.**
**Step 4 — Create workplan file first, then DB record (ADR-001)**
```
workplans/SHARD-WP-NNNN-<slug>.md ← write this first
```
Then register in the hub:
```
create_workstream(topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0", title="...", owner="...", description="...")
create_task(workstream_id="<id>", title="...", priority="high|medium|low")
```
**Step 5 — Record the setup**
```
add_progress_event(
summary="First session: structured consumer into N workstreams, M tasks",
event_type="milestone",
topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0",
detail={"workstreams": [...], "tasks_created": M}
)
```
<!-- Delete or archive this file once past first session -->

View File

@@ -0,0 +1,8 @@
## Repo boundary
This repo owns **shard-wiki** only. It does not own:
<!-- TODO: List what belongs in adjacent repos, e.g.:
- SSH key management → railiance-infra/
- State hub code → state-hub/
-->

View File

@@ -0,0 +1,5 @@
**Purpose:** Git-based Markdown wiki orchestrator and federation layer. Python (src/ layout, hatchling, pytest). Early-stage: scaffold + INTENT.md defined, domain model not yet implemented. See INTENT.md for authoritative scope.
**Domain:** consumer
**Repo slug:** shard-wiki
**Topic ID:** 4c2e5315-2cb9-447c-9d16-a39bdb0aabd0

View File

@@ -0,0 +1,85 @@
## Session Protocol
Dev Hub (State Hub API): http://127.0.0.1:8000
MCP server name in `~/.claude.json`: `dev-hub`
**Step 1 — Orient**
Read the offline-safe brief first — it works without a live hub connection:
```bash
cat .custodian-brief.md
```
Then call the MCP tool for richer cross-domain context when MCP tools are exposed:
```
get_domain_summary("consumer")
```
If MCP tools are unavailable in the current agent session, use the REST API:
```bash
curl -s "http://127.0.0.1:8000/state/summary" | python3 -m json.tool
```
If the hub is offline: `cd ~/state-hub && make api`
**Step 2 — Check inbox**
With MCP tools:
```
get_messages(to_agent="shard-wiki", unread_only=True)
```
Mark read with `mark_message_read(message_id)`. Reply or act on coordination
requests before proceeding.
Without MCP tools:
```bash
curl -s "http://127.0.0.1:8000/messages/?to_agent=shard-wiki&unread_only=true" \
| python3 -m json.tool
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
-H "Content-Type: application/json" -d '{}'
```
**Step 3 — Scan workplans**
```bash
ls workplans/
```
For each file with `status: ready`, `active`, or `blocked`, note pending
`wait`/`todo`/`progress` tasks.
**Step 4 — Present brief**
1. **Active workstreams** for `consumer` — title, task counts, blocking decisions
2. **Pending tasks** from `workplans/` + any `[repo:shard-wiki]` hub tasks
3. **Goal guidance** — if `goal_guidance` in summary:
- `needs_workplan`: surface as top action — *"Repo goal '{title}' has no workplan yet"*
- `alignment_warnings`: flag if active work is not aligned with current goal
4. **Suggested next action** — highest-priority open item
5. **SBOM status** — flag if `last_sbom_at` is unset for this repo
If no workstreams: follow First Session Protocol (`first-session.md`).
**During work:** `record_decision()` · `add_progress_event()` · `resolve_decision()`
> State Hub is a *read model*. Bootstrap tools (`create_workstream`, `create_task`)
> are First Session Protocol only. Work structure belongs in repo files (ADR-001).
**Session close:**
With MCP tools:
```
add_progress_event(summary="...", topic_id="4c2e5315-2cb9-447c-9d16-a39bdb0aabd0", workstream_id="<uuid>")
```
Without MCP tools:
```bash
curl -s -X POST http://127.0.0.1:8000/progress/ \
-H "Content-Type: application/json" \
-d '{"topic_id":"4c2e5315-2cb9-447c-9d16-a39bdb0aabd0","workstream_id":"<uuid>","event_type":"note","summary":"what changed","author":"codex"}'
```
If workplan files were modified, ensure the local copy is up to date first:
```bash
git -C <repo_path> pull --ff-only
cd ~/state-hub && make fix-consistency REPO=shard-wiki
```
For repos where implementation runs on a remote machine (e.g. CoulombCore),
use the combined target which pulls before fixing:
```bash
cd ~/state-hub && make fix-consistency-remote REPO=shard-wiki
```
**C-15** (DB task ahead of file) is normal in multi-machine workflows — writeback
will sync the file to match DB. **C-16** (repo behind remote) blocks all writes
until you pull — intentional to prevent clobbering remote progress.

View File

@@ -0,0 +1,19 @@
## Stack
<!-- TODO: Fill in language, frameworks, and key dependencies -->
- **Language:**
- **Key deps:**
## Dev Commands
```bash
# TODO: Fill in the standard commands for this repo
# Install dependencies
# Run tests
# Lint / type check
# Build / package (if applicable)
```

View File

@@ -0,0 +1,40 @@
## Workplan Convention (ADR-001)
File location: `workplans/SHARD-WP-NNNN-<slug>.md`
ID prefix: `SHARD-WP-`
Work items originate as files in this repo **before** being registered in the hub.
Canonical workplan/workstream frontmatter statuses are:
`proposed`, `ready`, `active`, `blocked`, `backlog`, `finished`, `archived`.
Use `proposed` for a newly drafted plan, `ready` after review against current
repo state, and `finished` when implementation is complete. `stalled` and
`needs_review` are derived health labels, not stored statuses.
Closed workplans may be moved to `workplans/archived/` with a completion-date
prefix: `YYMMDD-SHARD-WP-NNNN-<slug>.md`. The frontmatter id remains
unchanged; the prefix is only for quick visual reference.
Small opportunistic tasks discovered during another session use **Ad Hoc Tasks**:
`workplans/ADHOC-YYYY-MM-DD.md`, workstream slug `adhoc-YYYY-MM-DD`, and task ids
`ADHOC-YYYY-MM-DD-T01`, `T02`, etc. Use adhocs only for low-risk work completed
directly. Promote anything requiring analysis, design, approval, dependencies, or
multiple planned phases into a normal workplan.
Ecosystem todos from other agents arrive as `[repo:shard-wiki]` hub tasks —
visible at session start. Pick one up by creating the workplan file, then registering
the workstream.
Task blocks use this shape:
```task
id: SHARD-WP-NNNN-T01
status: wait | todo | progress | done | cancel
priority: high | medium | low
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
```
Status progression is `todo``progress``done`; use `wait` for waiting or
blocked work and `cancel` for stopped work.
<!-- Ralph Loop rules and HEUREKA sequence: ~/.claude/CLAUDE.md — do not duplicate here -->

View File

@@ -2,34 +2,46 @@
# Custodian Brief — shard-wiki
**Domain:** whynot
**Last synced:** 2026-06-08 12:55 UTC
**Last synced:** 2026-06-15 22:57 UTC
**State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)*
## Active Workstreams
### federation architecture design
Progress: 0/10 done | workstream_id: `2af4c46d-cbfd-40ea-a94b-d9e60b0f9945`
### second adapter — git-IS-store shard (contract validation on a new substrate)
Progress: 0/3 done | workstream_id: `9e24eeb0-c0f0-41e6-a1ca-88d71e4139ea`
**Open tasks:**
- · Architecture positioning and boundaries `ea8fdb22`
- · Remix primitives: fork, overlay, import, reference `fb7d4bce`
- · Equivalent page identity and multi-version presentation `8f2a333d`
- · History, attribution, and coordination journal `5f39f48d`
- · Union composition layer `3ff71e11`
- · Change notification and subscription transports `9596e5e8`
- · Information space lifecycle `38134064`
- … and 3 more open tasks
- · GitShardAdapter — read over a git working tree/repo `8a1c7c80`
- · Write = commit; current_rev = sha (drift) `b47dfb86`
- · History adopt + integration with union/overlay `4c895f42`
### shard-wiki requirements from yawex prior art
Progress: 0/6 done | workstream_id: `0ed023a2-760b-4990-b931-8ee1f41ea08f`
### incremental union maintenance + equivalence index + I-2 verification
Progress: 0/4 done | workstream_id: `78d48bcf-6482-4266-bc81-084b7ec1cd80`
**Open tasks:**
- · Design federation page-resolution model (yawex state space as inspiration) `ebc036e4`
- · Define namespace/path model and page+shard roles `431b4d28`
- · Specify union-level derived views (BackLinks, RecentChanges, AllPages, SiteMap, Search) `564545ec`
- · Provenance & freshness model for pages/revisions/projections `738326f5`
- · Overlay / lightweight-patch model (from yawex append/comment) `a268de6a`
- · Markdown link semantics: wikilink + red-link extension `a7499f3e`
- · Equivalence index: blocking + verify `842f480b`
- · Incremental maintenance (delta, not additive) `2da4e0b8`
- · I-2 verification: digest + consistency-checker `b602ce31`
- · Wire incremental tier behind resolution + views `2f3d083c`
### derived views — wikilinks, BackLinks, RecentChanges, AllPages/SiteMap
Progress: 0/5 done | workstream_id: `2fe15330-ddf6-4b0f-8e55-ada341375d35`
**Open tasks:**
- · Wikilink + red-link model `792660c3`
- · BackLinks (core) `431a54c3`
- · RecentChanges (core) `270c1c31`
- · AllPages / SiteMap (core) `898ba43e`
- · Wiring + integration `7157544b`
### git-backed DecisionLog + per-space append authority
Progress: 0/4 done | workstream_id: `4fb5b29b-955c-4f37-85cf-58b4643ab1ca`
**Open tasks:**
- · Git event-store backend (append = commit/object) `a8fcbb3e`
- · Per-space append authority (lease) `62abd162`
- · Fold over the git log + read-your-writes across processes `8cc3691e`
- · Migration + wiring `281e1db4`
---
## MCP Orientation (when available)

17
.repo-classification.yaml Normal file
View File

@@ -0,0 +1,17 @@
repo_classification:
standard: Repo Classification Standard
version: '1.0'
classified_at: '2026-06-22'
classified_by: agent
category: project
domain: consumer
secondary_domains: []
capability_tags:
- knowledge
- documentation
business_stake:
- product
- experience
business_mechanics:
- coordination
- operation

243
AGENTS.md
View File

@@ -1,62 +1,219 @@
# AGENTS.md
# shard-wiki — Agent Instructions
Guidance for agents working in `shard-wiki`.
## Repo Identity
## Read First
**Purpose:** Git-based Markdown wiki orchestrator and federation layer. Python (src/ layout, hatchling, pytest). Early-stage: scaffold + INTENT.md defined, domain model not yet implemented. See INTENT.md for authoritative scope.
1. `INTENT.md` — aspiration and boundaries (stable; architectural changes are rare).
2. `SCOPE.md` — what we are achieving now and current maturity.
3. `.custodian-brief.md` — State Hub snapshot (generated; do not edit manually).
**Domain:** consumer
**Repo slug:** shard-wiki
**Topic ID:** `4c2e5315-2cb9-447c-9d16-a39bdb0aabd0`
**Workplan prefix:** `SHARD-WP-`
## Documentation Layout
---
This repo follows the CoulombSocial / HelixForge / MarkiTect documentation
layout (recommendation, not strict law). Efficient retrieval by purpose:
## State Hub Integration
| Path | Purpose |
|------|---------|
| `INTENT.md` | Aspiration and boundaries |
| `SCOPE.md` | Top-level view of current achievement; closes gap to INTENT |
| `research/` | Exploration results (`yymmdd-` prefix on files or subdirs) |
| `demand/` | Inbound requests not yet reviewed into spec or workplans |
| `spec/` | Implementation guardrails (PRD, TSD, use cases, architecture) |
| `workplans/` | State Hubregistered implementation tasks |
| `docs/` | Stakeholder documentation (users, developers, humans, agents) |
| `wiki/` | Perspective-free interconnected knowledge (wiki UI when connected) |
| `issues/` | Mirror of relevant open tickets when ticket systems are in use |
| `history/` | Archived material (`yymmdd-` prefix); out of scope for daily work |
The Custodian State Hub tracks work across all domains. Interact via HTTP REST —
there is no MCP server for Codex agents.
**Mode of operation:** close SCOPE → INTENT while learning; refine both as needed.
| Context | URL |
|---------|-----|
| Local workstation | `http://127.0.0.1:8000` |
| Remote via tunnel | `http://127.0.0.1:18000` |
## Domain Vocabulary
Honor terms from `INTENT.md`: shard, root entity, adapter contract, projection,
overlay, coordination journal, shard modes. Do not invent parallel vocabulary.
## Build And Test
### Orient at session start
```bash
pip install -e ".[dev]"
pytest
ruff check
ruff format
# Offline brief — works without hub connection
cat .custodian-brief.md
# Active workstreams for this domain
curl -s "http://127.0.0.1:8000/workstreams/?topic_id=4c2e5315-2cb9-447c-9d16-a39bdb0aabd0&status=active" \
| python3 -m json.tool
# Check inbox
curl -s "http://127.0.0.1:8000/messages/?to_agent=shard-wiki&unread_only=true" \
| python3 -m json.tool
```
## State Hub
Mark a message read:
```bash
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
-H "Content-Type: application/json" -d '{}'
```
Workplans register with State Hub. After workplan changes:
### Log progress (required at session close)
```bash
cd ~/state-hub && make fix-consistency REPO=shard-wiki
curl -s -X POST http://127.0.0.1:8000/progress/ \
-H "Content-Type: application/json" \
-d '{
"summary": "what was done",
"event_type": "note",
"author": "codex",
"workstream_id": "<uuid>",
"task_id": "<uuid>"
}'
```
Finished or canceled workplans move to `history/` with a `yymmdd-` archive prefix.
Omit `workstream_id` / `task_id` when not applicable.
## Where To Put New Material
### Update task status
- Exploratory analysis → `research/yymmdd-<topic>/`
- Raw feature ask or external requirement → `demand/`
- Reviewed design ready to guide code → `spec/`
- Implementation tasks → `workplans/`
- User/dev/agent how-to → `docs/`
- Collaborative unstructured notes → `wiki/`
```bash
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
-H "Content-Type: application/json" \
-d '{"status": "progress"}'
# values: wait | todo | progress | done | cancel
```
### Flag a task for human review
```bash
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
-H "Content-Type: application/json" \
-d '{"needs_human": true, "intervention_note": "reason"}'
```
---
## Session Protocol
**Start:**
1. `cat .custodian-brief.md` — domain goal and open workstreams (offline-safe)
2. Check inbox: `GET /messages/?to_agent=shard-wiki&unread_only=true`; mark read
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
**During work:**
- Update task statuses in workplan files as tasks progress
- Record significant decisions via `POST /decisions/`
**Close:**
1. Update workplan file task statuses to reflect progress
2. Log: `POST /progress/` with a summary of what changed
3. Note for the custodian operator: after workplan file changes, run from
`~/state-hub`:
```bash
make fix-consistency REPO=shard-wiki
```
This syncs task status from files into the hub DB.
---
## Credential and access routing
**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect**
for inference. Run this check **before** requesting secrets, API keys, SSH access,
login tokens, or database passwords — in any repo, not only `ops-warden`.
ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every
other credential need belongs to another subsystem. **Do not** message
`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key.
### Lookup (do this first)
```bash
warden route find "<describe your need>" --json
warden route show <catalog-id> --json
```
Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`).
| Agent runtime | How to orient |
| --- | --- |
| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=shard-wiki` is for coordination, not secret vending |
| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workstreams; **still** use `warden route` for credential ownership |
| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` |
### Quick routing table
| I need… | Owner | ops-warden executes? |
| --- | --- | --- |
| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` |
| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only |
| Login / OIDC / MFA | key-cape / Keycloak | No — route only |
| Authorization decision | flex-auth | No — route only |
| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` |
| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only |
### Anti-patterns (do not do these)
- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc.
- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist
- Pasting secrets into Git, State Hub, workplans, logs, or chat
### Other capabilities (reuse-surface)
Non-credential capabilities are usually discovered through **reuse-surface** federation
(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in
every repo's agent instructions because it is high-frequency, high-risk, and easy to
get wrong.
**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml`
<!-- REPO-AGENTS-EXTENSIONS -->
<!-- Append repo-specific agent instructions below this marker.
The state-hub template sync preserves content after this line. -->
---
## Workplan Convention (ADR-001)
Work items originate as files in this repo — not in the hub. The hub is a
read/cache/index layer that rebuilds from files.
**File location:** `workplans/SHARD-WP-NNNN-<slug>.md`
**Archived location:** finished workplans may move to
`workplans/archived/YYMMDD-SHARD-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
the completion/archive date; the frontmatter `id` does not change.
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
`workplans/ADHOC-YYYY-MM-DD.md` with task ids `ADHOC-YYYY-MM-DD-T01`, etc. Use
this only for low-risk work completed directly; create a normal workplan for
anything needing analysis, design, approval, dependencies, or multiple phases.
**Frontmatter:**
```yaml
---
id: SHARD-WP-NNNN
type: workplan
title: "..."
domain: consumer
repo: shard-wiki
status: proposed | ready | active | blocked | backlog | finished | archived
owner: codex
topic_slug: ...
created: "YYYY-MM-DD"
updated: "YYYY-MM-DD"
state_hub_workstream_id: "<uuid>" # written by fix-consistency — do not edit
---
```
Use `proposed` for a new draft, `ready` after review against current repo
state, and `finished` after implementation. `stalled` and `needs_review` are
derived health labels, not frontmatter statuses.
**Task block format** (one per `##` section):
```
## Task Title
` ` `task
id: SHARD-WP-NNNN-T01
status: wait | todo | progress | done | cancel
priority: high | medium | low
state_hub_task_id: "<uuid>" # written by fix-consistency — do not edit
` ` `
Task description text.
```
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
To create a new workplan:
1. Write the file following the format above
2. Notify the custodian operator to run `make fix-consistency REPO=shard-wiki`
(or send a message to the hub agent via `POST /messages/`)

View File

@@ -1,53 +1,12 @@
# CLAUDE.md
# shard-wiki — Claude Code Instructions
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Repository status
This is an **early-stage Python repository**. The package scaffold (`src/shard_wiki/`, `tests/`, `pyproject.toml`) exists with only smoke tests — the domain model is not yet implemented. Read `INTENT.md` (aspiration), `SCOPE.md` (current achievement), and `AGENTS.md` (layout and conventions) before designing anything. Close the gap from SCOPE to INTENT via `research/`, `spec/`, and `workplans/`.
## What this project is
`shard-wiki` is a **Git-based Markdown wiki orchestrator and federation layer**, not a wiki engine. It lets multiple heterogeneous wiki-shaped page stores (**shards**) attach to a shared root entity and be presented as a **union of pages**, while preserving each shard's separate storage, provenance, capabilities, and history.
The core job is orchestration across backends — Git repos, repo subdirectories (`wiki/`), Gitea wikis, local folders, Obsidian vaults, WebDAV/Nextcloud directories, Coulomb spaces — never replacing or homogenizing them.
## Core domain model (the concepts code must honor)
These abstractions come from `INTENT.md` and define the architecture. New code should map onto them rather than inventing parallel vocabulary:
- **Shard** — an independently meaningful page store attached to a root entity. Shards have *sovereignty*: their own backend, capabilities, limits, history, and identity model. Not all shards are Git-native.
- **Root entity / information space** — the joined space that shards attach to. Each information space should have a **Git-addressable coordination layer** (history, patches, review, backup, reconciliation) even when individual shards are not Git-native.
- **Shard adapter contract** — the versioned interface a backend implements to participate. Adapters are **capability-aware**: the core must model explicitly which operations a shard supports (read, write, diff, merge, lock, version, publish, accept patches) rather than assuming uniformity.
- **Wiki page model** — a stable, versioned, Markdown-first but backend-neutral representation of pages, paths, links, metadata, revisions.
- **Projection** — a lazy, cache-like local view of remote/external shard content. Prefer lazy projection over eager copying.
- **Overlay** — a non-destructive local edit against a remote, read-only, or capability-limited shard, representable as drafts/patches/commits/merge requests *before* destructive application ("overlay before mutation").
- **Coordination journal** — the Git-backed record of change flows for an information space.
- **Shard modes** — read-only, write-through, mirrored, projected, cached, canonical.
## Design constraints to enforce in code
These are hard boundaries from `INTENT.md`; treat violations as design bugs:
- **Mechanism over policy.** Provide primitives for federation, sync, overlays, patching, conflict detection, projection, reconciliation. Do *not* hard-code one editorial/sync/conflict/canonical-source policy — keep those configurable.
- **Union without erasure.** Always preserve provenance: which shard a page came from, its freshness, whether it is cached, whether it has overlays, whether it diverges from an equivalent page elsewhere. Never hide authorship, conflicts, freshness, or backend limitations.
- **No silent remote mutation.** Do not mutate remote systems without explicit adapter support and user intent.
- **Graceful degradation.** Limited backends must still be usable as read-only/cache/projection/backup/patch targets.
- **Not a file-sync daemon.** Synchronization is wiki-page-semantic, not generic file mirroring.
`INTENT.md` has a "Stability Note": changes that redefine what a shard is, Git's role, how root entities are modeled, or whether this is an orchestrator vs. an engine are **architectural changes** and should be rare and deliberate.
## Build, test, run
Python with a `src/` layout, built via hatchling, tested with pytest. Tests run against the source tree directly (`pythonpath = ["src"]` in `pyproject.toml`), so no install/editable step is required to run them.
```bash
pip install -e ".[dev]" # one-time: install dev tooling (pytest, pytest-cov, ruff)
pytest # run the full test suite
pytest tests/test_package.py::test_version_is_exposed # run a single test
pytest --cov # run with coverage
ruff check # lint
ruff format # format
```
Note: the system `pytest` is 7.4.x; `minversion` in `pyproject.toml` is pinned to `7.0` to match. Bump it if a newer pytest is installed into the dev environment.
@SCOPE.md
@.claude/rules/repo-identity.md
@.claude/rules/session-protocol.md
@.claude/rules/first-session.md
@.claude/rules/workplan-convention.md
@.claude/rules/stack-and-commands.md
@.claude/rules/architecture.md
@.claude/rules/repo-boundary.md
@.claude/rules/credential-routing.md
@.claude/rules/agents.md

View File

@@ -16,6 +16,8 @@ The goal is to allow independently stored and differently implemented wikis, pag
The repository provides a **shard orchestration layer** for interconnected Markdown and markup-based wiki content.
Equivalently, shard-wiki can be used as a **headless, API-first wiki engine** — optimized for **integrating heterogeneous data sources** and for **efficient access by agents and automation** — that ships its own native engine as one (canonical-mode) shard among many. There is no bundled UI: presentation and rendering are consumer concerns.
It allows wiki-like systems to:
* Attach heterogeneous page stores as shards of a shared information space
@@ -30,6 +32,7 @@ It allows wiki-like systems to:
* Run fully standalone with open read/write access and complete change history, then progressively layer multi-tenant enterprise access control through external identity integration
* Allow existing wiki engines to become federation-capable through a shared API
* Allow non-federation-aware systems to participate through adapters and projections
* Serve as a **headless, API-first wiki engine** (a small typed-extension core) that integrates heterogeneous data sources and is consumed efficiently by agents and automation
It transforms disconnected wiki engines, Git repositories, local folders, WebDAV directories, application-specific content stores, and desktop editing workflows into a **composable federated wiki space**.
@@ -85,7 +88,7 @@ A mature `shard-wiki` should allow each participating shard to see the others as
This repository is **not** intended to:
* Replace all wiki engines with a single canonical wiki implementation
* Replace all wiki engines with a single canonical wiki implementation *(shard-wiki MAY still provide its own native, headless, API-first engine as one optional shard backend — see Design Principles — but never as a mandated or universal replacement)*
* Force every shard to use the same backend, database, directory layout, or storage format
* Require every participating system to become federation-aware
* Require every participating shard to be Git-native
@@ -148,6 +151,9 @@ Policy decisions such as conflict preference, canonical source selection, public
* **Composable integration**
Wiki engines should be able to use the `shard-wiki` API to become federation-enabled without reimplementing federation internally.
* **Native reference engine (additive, headless & API-first)**
shard-wiki MAY provide its own native wiki-engine as a **canonical-mode shard backend** — a **small core** with a **typed-extension framework**, activated **per shard** (only what you need). It is **headless and API-first** (no bundled UI; presentation/rendering are consumer concerns) and tuned for **integrating heterogeneous data sources** and **efficient agent/automation access**. It is *one shard type among many*, implemented against shard-wiki's own adapter contract; it does **not** replace other engines, mandate a single implementation, or change shard-wiki's role as an orchestrator. Shard sovereignty and union-without-erasure are preserved.
* **Open by default, progressively governed**
A standalone `shard-wiki` must be runnable with zero external dependencies in a classic Ward Cunningham / c2-style open read/write-for-all mode. Access control is an *additive capability*, not a precondition: the same core progresses — without re-architecture — to authenticated single-user, to group/role-based, to multi-tenant enterprise access control, mirroring the NetKingdom capability ladder (lightweight → expanded).
@@ -201,3 +207,5 @@ Such changes should be rare, because they affect all downstream systems relying
In particular, changes that redefine what counts as a shard, what role Git plays, how root entities are modeled, or whether `shard-wiki` is an orchestrator rather than a wiki engine should be treated as architectural changes.
**Amendment — 2026-06-15 (SHARD-WP-0013 T4, decision `84ffdb48`):** admits an **additive** native reference wiki-engine — **headless, API-first**, a small typed-extension core — as a **canonical-mode shard backend** optimized for data-source integration and agent access. Deliberate, narrow scope change; shard-wiki remains an orchestrator and neither mandates nor replaces other engines. (Mirrors the earlier auth-in-core amendment precedent.)

View File

@@ -17,12 +17,12 @@ Learnings update both SCOPE and INTENT where necessary.
| Layer | State |
|-------|-------|
| Code | Python package scaffold (`src/shard_wiki/`, smoke tests only) |
| Code | Foundation slice implemented (SHARD-WP-0007): `provenance` + `policy` leaves, `model` (Identity/Placement/Span/Page/CapabilityProfile), `adapters` (contract + FolderAdapter + conformance suite), `coordination` (event-sourced DecisionLog), `union` (resolution + chorus, overlay-aware), `InformationSpace` orchestrator. Write path added (SHARD-WP-0008): writable adapter, overlay engine (draft→patch→apply-under-drift), edit() unifies write-through + overlay-before-mutation. Native engine implemented (SHARD-WP-0014): `engine` (kernel + typed-extension runtime + per-shard activation [ADR-0001] + capability-profile-from-extensions + EngineShardAdapter + the `ext.struct` built-in) — an engine shard attaches to an InformationSpace as a canonical-mode shard. Git-backed coordination log (SHARD-WP-0009): `DecisionLog` storage factored behind an `EventStore`; `GitEventStore` makes the log git-addressable (each space a ref, append = immutable CAS-guarded commit), a per-space `AppendAuthority` (lease) gives a single-writer total order with re-grantable HA hand-off, cross-process read-your-writes verified, and a verbatim one-time importer (`migrate_space`/JSONL) replays in-memory logs into git; `InformationSpace.git_backed(...)` wires it. Derived views (SHARD-WP-0010): `views` (wikilink + red-link model, BackLinks, RecentChanges, AllPages/SiteMap) — recomputable, provenance-carrying, presentation-free, exposed via `InformationSpace.backlinks/recent_changes/all_pages/site_map`. Incremental-first derived tier (SHARD-WP-0011): `incremental` (indexed equivalence via MinHash/LSH blocking + verify, change-driven delta maintenance with retraction/propagation, Merkle-style digest + self-healing I-2 consistency-checker, `UnionIndex` routed behind `InformationSpace.all_pages` with rebuild as explicit fallback). Second adapter (SHARD-WP-0012): `GitShardAdapter` — git-IS-store substrate (read=tracked *.md, write=commit, current_rev=per-path sha for drift, adopted git-native history), passes conformance, works across folder+git shards in union/overlay/edit with no core change (capability-as-data proven on a second substrate). 196 tests green, ~97% coverage |
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`) |
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) |
| Demand | NetKingdom integration asks captured, not yet negotiated |
| Spec | Architecture blueprint drafted; UseCaseCatalog 25 UCs from research; PRD/TSD scaffolds |
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (10 tasks) |
| Spec | CoreArchitectureBlueprint (whole-system, hardened via SHARD-WP-0005/0006) + FederationArchitecture + FederationRequirements + TSD §A adapter contract + ArchitectureBlueprint (auth/history) + WikiEngineCoreArchitecture (headless API-first engine, SHARD-WP-0013) drafted; UseCaseCatalog 84 UCs (+ engine capability-structure layer); PRD scaffold |
| Work | `SHARD-WP-0001` **done** (6 ADRs: yawex-derived federation requirements → `spec/FederationRequirements.md`); `SHARD-WP-0002` **done** (18 tasks → `FederationArchitecture.md` [T1T10, T17] + `TechnicalSpecificationDocument.md` §A adapter contract [T11T16, T18]); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` **done** (all 8 computational-knowledge dives T1T8 complete + "computational page model" synthesis); `SHARD-WP-0005` **done** (9 tasks: CoreArchitectureBlueprint hardened against the 260615 review); `SHARD-WP-0006` **done** (5 tasks: round-2 hardening — overview reconciled, event-sourced coordination + append authority, adapter conformance, incremental correctness + I-2 verification) |
## In Scope (today)
@@ -32,11 +32,15 @@ Learnings update both SCOPE and INTENT where necessary.
- Authorization model design (delegated authentication, core authorization).
- Shard adapter contract and wiki page model (to be specified, then implemented).
- Git-backed coordination journal for information spaces.
- A **native, headless, API-first wiki-engine core** (small typed-extension core, as a
canonical-mode shard backend) — design via SHARD-WP-0013; optimized for data-source
integration and agent access.
- State Hub workplan registration and consistency sync.
## Out Of Scope (today)
- A standalone wiki engine UI or rendering pipeline.
- A wiki-engine **UI or rendering pipeline** (the engine is headless/API-first; presentation
is a consumer concern). A bundled standalone UI is not provided.
- Authentication, credential storage, or user directory implementation.
- Hard-coded editorial, sync, or conflict-resolution policy.
- Generic file mirroring independent of wiki-page semantics.
@@ -55,6 +59,11 @@ live in core.
Design work is tracked in `workplans/SHARD-WP-0001-yawex-requirements.md`
(yawex-derived resolution, namespaces, overlays) and
`workplans/SHARD-WP-0002-federation-architecture.md` (federation architecture,
decisions, tradeoffs). Specification outputs land in `spec/`. Inbound
integration asks remain in `demand/` until reviewed and promoted into spec or
workplans.
decisions, tradeoffs). Research continues under
`workplans/SHARD-WP-0003-engine-dives-batch.md` (remaining new-insight wiki
engines + git-forge wikis + classic engines) and
`workplans/SHARD-WP-0004-computational-knowledge-systems.md` (literate
programming, computational notebooks, live-coding REPLs, image-based/moldable
environments — the executable/computational page-model thread). Specification
outputs land in `spec/`. Inbound integration asks remain in `demand/` until
reviewed and promoted into spec or workplans.

View File

@@ -0,0 +1,91 @@
# Critical review (round 2) — CoreArchitectureBlueprint.md (hardened)
Date: 2026-06-15 · Reviewer: tegwick (with Claude) · Subject:
`spec/CoreArchitectureBlueprint.md` after **SHARD-WP-0005** (commit f21b7b5) · Feeds:
**SHARD-WP-0006**
A second hostile pass over the *hardened* blueprint. Round 1 found design bugs; this round
finds (a) self-consistency regressions the surgical hardening introduced, and (b) deeper
second-order gaps — some of which the hardening *sharpened*. Verdict: the architecture is now
substantially sound; what remains in §B/§C are hard distributed-systems/operational questions,
not design smells — except §A (a real regression) and three foundational gaps in §B.
---
## A. Self-consistency regressions introduced by surgical hardening
The 9 edits deepened §6§9 but did not propagate to the **overview surfaces**, so the document
now contradicts itself between its summary and its body (and readers trust the summary).
- **A-1 (real contradiction).** §4 still says "Addressing, **equivalence**, and transclusion
key on identity" — the exact conflation T2 fixed in §7.2 (equivalence keys on *content
fingerprint across distinct identities*). → **WP-0006 T1**
- **A-2.** §4 "Projection — typed on two axes" and §4 "Provenance envelope … every artifact
carries [full wrapper]" are stale vs T7's §8.4 (two-axis = extension point; trivial default)
and §7.3 (layered effective-vs-own). → **T1**
- **A-3.** §10 policy surface omits knobs the hardening added (freshness/staleness §8.8,
squash-compaction §8.1, conflict-resolution preset §8.6, tenant-partition) — yet §11 defines
`policy/` as "owns the §10 surface." The module contract points at a stale list. → **T1**
- **A-4 (cosmetic).** §3 diagram + §11 header still say "L4 rebuildable cache" / "15 spectra,"
advertising the pre-hardening model (§8.7 incremental-first; §6.5 orthogonal-core). → **T1**
Meta-point: surgical editing hardened the body but regressed whole-document coherence; v2
needs an **overview-reconciliation pass**.
## B. Foundational gaps (serious; some sharpened by the hardening)
- **B-1 — The journal is now a concurrent-write DB, but it's single-writer Git.** §8.6's
consistency model assumes "the journal is local Git, read-your-writes." L4 multi-tenant + the
L6 Orchestrator API imply a server; HA/scale implies *multiple* instances. Concurrent commits
of coordination-canonical state to one git journal = lock contention / merge races; Git is not
a concurrent-write store. Either single-writer-per-space (an unstated HA ceiling) or a real
concurrent coordination store with Git as an *export*. **T1 of WP-0005 worsened this** by
loading more canonical state into the journal. The keystone unanswered question. → **T2**
- **B-2 — Capability-as-data trusts self-reported profiles with no conformance check.** I-3 +
§6.5's degradation contract assume the profile tells the truth. A buggy adapter (claims
`merge=git/text`, corrupts; claims `notify`, never emits) silently poisons every degradation
decision. No **adapter conformance suite** (declared profile == observed behavior) exists.
Foundational for an architecture whose correctness rests on profile accuracy. → **T3**
- **B-3 — "Coordination-canonical state in the journal" has no representation design.** T1
relocated overlays/bindings/aliases/equivalence-sets/merges into "the journal" without saying
*how* Git stores structured mutable state. "All equivalences touching X" over a git-of-files
is O(scan) unless indexed — and an index is L4/derived. The new central concept is a black
box; resolve *with* B-1. → **T2**
- **B-4 — Incremental equivalence is under-specified/likely incorrect; I-2 only eventually
true.** §8.7 re-verifies a changed page's *new* candidate set but not the pairs it *leaves*
(a page exiting an LSH bucket can break an existing equivalence edge); the delta is not
additive. Deeper: incremental maintenance drifts from `f(canonical)`, so I-2 holds only
eventually, guaranteed solely by an expensive reconcile-against-rebuild. Needs a stated
verification mechanism (background checker / digest-vs-sampled-rebuild). → **T4**
## C. Real but second-tier (track as open problems O-8…O-11)
- **C-1 — Mechanism-over-policy → operator burden; no preset bundles.** ~7 knob families with
sub-modes and interactions; only authz (L0L4) bundles into personas. Need named bundles
("personal vault" / "team wiki" / "enterprise federation"). → **O-8 / T5**
- **C-2 — Tenant partitioning (I-13) vs shard sharing + lazy projection.** A shard in two roots
is cached twice → duplicate storage + double refresh on rate-limited backends. Shard
exclusive-to-one-root or shareable? Unresolved. → **O-9 / T5**
- **C-3 — Span-level authz + transclusion is an unmodeled leak path.** Authz is per
page/shard/tenant; transclusion crosses shards at span granularity → a page can leak a span
past its ACL (aggregation/inference). §7.3's ⊕ also stops being simple two-level inheritance
across a transclusion boundary. → **O-10 / T5**
- **C-4 — Union-under-unavailability undefined.** Freshness covers *stale*, nothing covers
*down*. The dead-shard read path (partial union? error? last-known?) is unspecified though
it's the commonest real failure. → **O-11 / T5**
## D. Recommended resolution (→ SHARD-WP-0006)
1. **§A reconciliation** (T1) — make the overview match the hardened body.
2. **Journal & coordination-state model** (T2) — settle single-vs-multi-writer and separate the
**coordination-state store** from the **content-history journal**. Likely resolution:
**event-sourced coordination** — an append-only *decision log* is the coordination-canonical
tier (git-addressable, I-6 preserved); the queryable current state (alias table, equivalence
set) is a *derived fold* of the log (disposable). Append-logs tolerate concurrency far better
than mutable-file Git; state a concurrency model. Resolves **B-1 + B-3** together.
3. **Adapter conformance suite** (T3) — make a passing conformance run part of the contract
(B-2): every adapter proves declared profile == observed behavior.
4. **Incremental correctness + I-2 verification** (T4) — fix the leaving-bucket re-verification
and propagation; add a background consistency-checker / derived-tier digest so I-2 is
verifiable, not merely asserted (B-4).
5. **Track §C** (T5) — O-8…O-11 with chosen direction + revisit trigger; close-out.

View File

@@ -0,0 +1,122 @@
# Critical review — CoreArchitectureBlueprint.md
Date: 2026-06-15 · Reviewer: tegwick (with Claude) · Subject:
`spec/CoreArchitectureBlueprint.md` @ commit **9b5b393** · Feeds: **SHARD-WP-0005**
A deliberately hostile review of the first whole-system architecture, to find where it
**breaks (correctness)**, **fails to scale**, and **could be more elegant/efficient** before
any implementation. Findings are prioritised; each is the input to a SHARD-WP-0005 task.
## Verdict in one line
The **layering and the dual narrow waist are sound and stay**. The **thesis is ~90% right**;
the missing 10% (curatorial / coordination-canonical state) breaks its clean story. There are
**two genuine bugs**, **two large unaddressed scaling risks**, and several **elegance/efficiency
debts** — all fixable without touching INTENT.
---
## A. The framing crack (fix resolves three issues)
**A-1 — Two buckets hide a third.** The thesis "canonical at the edges, derived in the middle"
omits **born-in-the-middle-but-canonical** state: overlays that are the local truth against a
read-only shard (Flow C), manual **curator equivalence bindings**, alias tables, merge
decisions. These encode human judgment or local-only content and **cannot be rebuilt** from
shards+journal.
**Contradiction:** I-2 declares L4 rebuildable, yet §8.4 puts "alias table, curator binding"
in L4. You cannot rebuild a curator's manual binding.
**Fix:** three states — **sharded-canonical**, **coordination-canonical** (journal: overlays,
bindings, aliases, merges — durable, born in the middle), **derived-disposable** (union graph,
indexes, projections). Re-frame §1 as **canonical (sharded + coordination) vs derived
(disposable)**; `derived = f(canonical)` then becomes actually true. → **T1**
---
## B. Where it breaks (correctness)
**B-1 — Identity conflated with content-fingerprint (BUG).** §7.2 derives page identity from
content fingerprint. That makes **editing a page change its identity**, breaking every
reference. Fingerprints identify *versions/equivalence*, not *identity*. Page identity must be
a **stable handle (uid)** surviving edits; fingerprints belong to the **equivalence** mechanism
(§8.4). One word, two concepts, wrong implementation for the stable one. → **T2**
**B-2 — No concurrency/consistency model.** Concurrent overlays on one page, overlay applied
after source drift, journal-commit vs shard-native-write ordering — all undefined. Conflict
handling is deferred to "policy presets," but **conflict *detection + representation* is core
mechanism**; only *resolution* is policy. The union's consistency guarantee is unstated
(eventually-consistent? read-your-writes? causal-via-journal?). → **T3**
**B-3 — Persisted union cache + multi-tenant = leak surface.** §13 recommends a persisted L4
cache; §9 protects content by *read-time* filtering on the provenance envelope. A persisted
cross-tenant union cache guarded only by read-time filtering is an L4 attack surface. Tension
between I-2 (persisted rebuildable cache), scale, and L5 isolation is unacknowledged. → **T8**
---
## C. Where it fails to scale
**C-1 — Equivalence detection is O(N²), no indexing/incremental story.** Fingerprint /
span-set-overlap across all pages of all shards is combinatorial (10 shards × 100k pages ≈
10¹² comparisons). No blocking/LSH/indexing, no incremental maintenance. Biggest scaling
hazard in the document. → **T4**
**C-2 — "Rebuildable cache" collides with the operational-envelope axis.** A byte-exact
rebuild requires reading *every page of every shard*, including rate-limited/paginated
external APIs (Notion) and irreducibly-live sources — hours-to-days. I-2 contradicts axis-10.
**Incremental, change-driven maintenance must be primary** (notify→delta), rebuild a rare
fallback. Cache invalidation — the actual hard problem — is named once and never designed. →
**T4, T5**
**C-3 — Unbounded history at open L0 = DoS/perf.** "Every write a commit" + "open for all" ⇒
the git journal grows without bound under bots/vandalism and git degrades on huge histories.
"History is the floor" has an unacknowledged cost: packing, compaction, per-shard offload. →
**T8**
---
## D. Elegance / efficiency debts
**D-1 — The 15 spectra assert a clean degradation function never demonstrated.** Either most
axes are irrelevant to most ops (then the 15-D profile is ceremony), or behavior depends on
several axes *jointly* (then "no per-backend code" becomes a sprawling axis-interaction matrix
— the flat-checklist problem in higher dimensions). And the axes **aren't orthogonal**
(git-native history ⟺ git-IS-store ⟺ git/text merge; encrypted opacity ⟹ query/translation
collapse). Model a **smaller orthogonal core** + **derived/implied** positions, and state the
**axis-interaction subset** the degradation logic truly uses. → **T6**
**D-2 — Provenance envelope isn't inherited; it'll dwarf the content.** Per-span envelopes at
block granularity = 10k near-identical envelopes for a 10k-block graph. The doc already
invented the right pattern for Trilium ("effective-vs-own with per-attribute provenance") and
failed to apply it to its own envelope. Make provenance **layered (page envelope + span
deltas)**. → **T7**
**D-3 — Projection machinery over-fit to the exotic tail.** Two-axis model + three facets +
view registry exist mostly for UC-83/84 (2 of 84 UCs); the 95% case (markdown in git) pays the
weight. Make the **common case trivial** (default = plain lazy replication) and
derivation/liveness an **extension point**, not a taxonomy every projection instantiates. →
**T7**
**D-4 — Cross-cutting rails are the highest-coupling components, presented as clean.**
`provenance/` and capability types are imported by every layer (god-modules); an envelope
change ripples everywhere. And **policy has no module** (§10 enumerates it; §11 omits it)
despite being consulted by L3/L4/L5. Give policy a home; pin the rails behind stable narrow
interfaces. → **T7**
---
## E. What explicitly stays
- The 6-layer model + the dual narrow waist (adapter contract / page model).
- Capability-as-data (I-3), union-without-erasure (I-4), overlay-before-mutation (I-5),
Git-addressable coordination (I-6), mechanism-over-policy (I-7), graceful degradation (I-8).
- The federation-model taxonomy and the auth ladder (ArchitectureBlueprint.md).
## F. Disposition
Some findings are **solvable now** (A-1, B-1, D-2, D-3, D-4, C-3); some are **partially open**
and should be tracked honestly rather than pretend-solved (B-2 consistency model: pick a
guarantee; C-1 equivalence-at-scale: pick a blocking strategy; D-1 axis interactions: enumerate
the real subset). SHARD-WP-0005 closes the solvable ones and records the open ones in a new
"Known scaling risks & open problems" section of the blueprint. → **T9**

View File

@@ -0,0 +1,45 @@
# reuse-surface contributions — shard-wiki (SHARD-WP-0013 T3)
Date: 2026-06-15 · From: shard-wiki (whynot) · To: reuse-surface (helix_forge) · Tracked list
of (A) capabilities shard-wiki registered, (B) **gaps** shard-wiki proposes the reuse surface
add, and (C) capabilities shard-wiki will **consume** rather than rebuild. Communicated to the
reuse-surface agent via state-hub `send_message`.
## A. Registered (T1 + T3) — 8 entries
`capability.wiki.{shard-orchestration, adapter-contract, page-model, coordination-journal,
overlay, federation-models, engine-typed-extensions, derived-views}` — committed in
reuse-surface, `validate` ok (20 entries total).
## B. Proposed gaps (cross-cutting; not shard-wiki-internal) → reuse-surface should own/define
- **G1 — `capability.platform.typed-extension-framework`** *(suggested)*
A reusable pattern: a **small core + a stringent typed-extension framework** where extensions
declare typed contracts, compose, and are **activated per context**. shard-wiki's wiki engine
(`capability.wiki.engine-typed-extensions`) is one instance, but the *pattern* is
cross-domain (any HelixForge capability platform). Evidence: shard-wiki UseCaseCatalog
"Capability structure" layer (core + 10 typed extensions + conflict-mediation map).
Suggested owner: helix_forge / reuse-surface. Relation: would `generalize`
`capability.wiki.engine-typed-extensions`.
- **G2 — `capability.content.translation-fidelity`** *(suggested)*
Lossless/lossy **content translation with an explicit fidelity report** (what round-trips
cleanly vs degrades, non-mappable elements preserved as provenance). Reusable well beyond
wiki (any format-bridging consumer). Evidence: shard-wiki TSD §A.6, UC-42/UC-59.
## C. Consumptions (reuse, do not rebuild)
- **`capability.feature-control.evaluate`** (helix_forge/feature-control) → shard-wiki's
**per-shard extension/feature activation** (the "activate only what you need" mechanism).
Already recorded as a relation on `capability.wiki.engine-typed-extensions`.
- **`capability.authorization.policy-evaluate`** (flex-auth) → shard-wiki's **X-AUTHZ** policy
decisions. shard-wiki owns the authz *model* (authz-in-core) but can reuse this evaluation
engine rather than building one.
- **`capability.statehub.{progress-log, workstream-coordinate}`** → already in use for
coordination across this work.
## Status
Gaps G1/G2 are **suggestions** to the reuse-surface owner (not unilateral registrations, since
they are cross-cutting, not shard-wiki-internal). Consumptions are recorded for the engine
architecture (T5) so it reuses rather than reinvents.

View File

@@ -1,8 +1,19 @@
# history/
Archived material that is no longer needed for daily work but should be kept.
Archived material and the project's **meta-history**: finished/canceled workplans kept for
the record, plus durable **reviews, critical assessments, and decision records** — the
reasoning behind the specs, captured at a point in time.
Use a `yymmdd-` prefix when archiving files or directories. Content here is
**out of scope** for regular tasks — consult only for research or diagnostics.
Use a `yymmdd-` prefix. Archived material is **out of scope** for regular tasks (consult only
for research or diagnostics); assessment/review records are point-in-time and may seed active
workplans, but are not edited after the fact — supersede with a new dated record and link back.
Finished or canceled workplans from `workplans/` are archived here.
Distinct from the **coordination journal** (a runtime Git-backed record of *content* change
flows inside an information space, an INTENT domain concept); `history/` is the *project's own*
design evolution.
| Date | Record | Subject |
|------|--------|---------|
| 2026-06-15 | `260615-core-architecture-blueprint-review.md` | Critical review of `spec/CoreArchitectureBlueprint.md` (commit 9b5b393); inputs to `SHARD-WP-0005` |
| 2026-06-15 | `260615-core-architecture-blueprint-review-2.md` | Round-2 review of the hardened blueprint (post-`SHARD-WP-0005`, f21b7b5); inputs to `SHARD-WP-0006` |
| 2026-06-15 | `260615-reuse-surface-contributions.md` | shard-wiki's reuse-surface registrations, proposed gaps (G1/G2), and consumptions (`SHARD-WP-0013` T3) |

View File

@@ -36,6 +36,11 @@ pythonpath = ["src"]
branch = true
source = ["shard_wiki"]
[tool.coverage.report]
show_missing = true
# Quality floor for `pytest --cov` / `coverage report` (not forced on a bare `pytest` run).
fail_under = 90
[tool.ruff]
src = ["src", "tests"]
target-version = "py311"

12
registry/README.md Normal file
View File

@@ -0,0 +1,12 @@
# Capability Registry
Markdown-first capability index for federation and reuse planning.
## Authoring
1. Copy a capability entry template (see reuse-surface `templates/capability-entry.template.md`).
2. Add the row to `indexes/capabilities.yaml`.
3. Run `reuse-surface validate` from a checkout with the CLI installed.
4. Merge to `main` and verify publish with `reuse-surface establish --publish-check`.
Federation contract: reuse-surface `docs/RegistryFederation.md`.

View File

@@ -0,0 +1,103 @@
---
id: capability.wiki.adapter-contract
name: Capability-Aware Shard Adapter Contract
summary: A versioned backend interface where each binding declares a verified capability profile (positions on capability spectra), so federation ops degrade by capability.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, adapter, capability, contract, conformance, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Fifteen capability spectra with an orthogonal core + implication rules, plus
a normative contract spec (TSD Section A); derived from a ~23-system synthesis.
availability:
current: A2
target: A5
confidence: medium
rationale: >
AdapterContract + a read/write FolderAdapter + a conformance suite that
verifies declared profile == observed behaviour exist as a source module.
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- versioned interface with declared, conformance-verified capability profiles
- one concrete adapter (file-store) passes the conformance suite
broken_expectations:
- only one substrate implemented (git-IS-store, REST, CRDT adapters planned)
out_of_scope_expectations:
- hosting backends
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- single adapter implemented so far
discovery:
intent: >
Mediate heterogeneity at one narrow waist: a backend participates by implementing a
versioned interface and declaring a verified position on each capability spectrum.
includes:
- capability profile as data (orthogonal-core spectra + implied positions)
- operation verbs (read/write/diff/merge/notify/.../derive-projection/execute)
- a conformance suite (profiles verified, not self-asserted)
excludes:
- assuming uniform backend capabilities
use_cases:
- "shard-wiki UseCaseCatalog UC-34..UC-43, UC-50, UC-57, UC-60..UC-69 (shard attachment & adapter binding)"
availability:
current_level: A2
target_level: A5
current_artifacts:
- "shard-wiki/src/shard_wiki/adapters/"
consumption_modes:
- source module
relations:
depends_on:
- capability.wiki.page-model
supports:
- capability.wiki.shard-orchestration
evidence:
documentation:
- "shard-wiki/spec/TechnicalSpecificationDocument.md (Section A)"
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 6)"
tests:
- "shard-wiki/tests/test_folder_adapter.py"
- "shard-wiki/tests/test_conformance.py"
consumer_guidance:
recommended_for:
- exposing any page store as a capability-described, conformance-checked shard
not_recommended_for:
- backends that cannot honestly describe their capabilities
known_limitations:
- reference implementation covers the file-store substrate only so far
---
# Capability-Aware Shard Adapter Contract
The bottom narrow waist of shard-wiki: a versioned interface plus a **verified** capability
profile per binding. Core logic is written once against capabilities (not per-backend), and
the conformance suite rejects profiles whose declared abilities don't match observed behaviour.
## Assessment notes
### Discovery
Fifteen spectra reduced to an orthogonal core with implication rules (CoreArchitectureBlueprint
Section 6.5); normative in TSD Section A.
### Availability
`adapters/` ships the contract, a folder adapter, and `assert_conformant`.

View File

@@ -0,0 +1,103 @@
---
id: capability.wiki.coordination-journal
name: Event-Sourced Coordination Journal
summary: An append-only, totally-ordered-per-space decision log (overlays, bindings, aliases, merges, forks) whose current state is a derived fold; git-addressable history.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, event-sourcing, coordination, git, journal, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Keystone resolved across two architecture reviews: coordination-canonical state
as an append-only decision log with a per-space append authority; current state
is a derived fold (derived = f(log)).
availability:
current: A2
target: A4
confidence: medium
rationale: >
In-memory DecisionLog + fold work as a source module; the git-backed store with a
per-space lease (the production backing) is planned.
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- append-only, totally-ordered-per-space log with read-your-writes
- derived fold to aliases + transitively-merged equivalence groups
broken_expectations:
- git-backed storage and per-space lease/append-authority not yet implemented
out_of_scope_expectations:
- general-purpose event bus
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- in-memory backing only; cross-process durability pending
discovery:
intent: >
Make coordination-canonical decisions durable and git-addressable as events, with the
queryable current state always recomputable by replay.
includes:
- append-only decision log, totally ordered per information space
- derived fold to current coordination state (aliases, equivalence groups, overlays)
- per-space append authority (concurrency model)
excludes:
- storing derived/disposable union state
use_cases:
- "shard-wiki UseCaseCatalog UC-29, UC-33 (history, attribution, coordination journal)"
availability:
current_level: A2
target_level: A4
current_artifacts:
- "shard-wiki/src/shard_wiki/coordination/decision_log.py"
target_artifacts:
- git-backed log store with per-space lease
consumption_modes:
- source module
relations:
supports:
- capability.wiki.shard-orchestration
- capability.wiki.overlay
evidence:
documentation:
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 8.1)"
tests:
- "shard-wiki/tests/test_decision_log.py"
consumer_guidance:
recommended_for:
- durable, replayable, git-addressable coordination state for a federated space
not_recommended_for:
- high-frequency general event streaming
known_limitations:
- production git backing + lease are still on the roadmap (SHARD-WP-0009)
---
# Event-Sourced Coordination Journal
The keystone: coordination-canonical state (overlays, equivalence bindings, aliases, merges,
forks) is an append-only **decision log**, totally ordered per information space; the queryable
current state is a derived **fold** of the log (`derived = f(log)`). The log is git-addressable,
giving history/patch/review/backup for coordination decisions for free.
## Assessment notes
### Discovery
Resolved across the round-1/round-2 architecture reviews (CoreArchitectureBlueprint Section 8.1).
### Availability
`decision_log.py` ships an in-memory, totally-ordered log + fold; git+lease backing is planned.

View File

@@ -0,0 +1,87 @@
---
id: capability.wiki.derived-views
name: Wiki Derived Views
summary: Recomputable views over a wiki union — BackLinks, RecentChanges, AllPages, SiteMap, and (delegate-or-derive) Search — carrying provenance.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, derived-views, backlinks, recentchanges, search, shard-wiki]
maturity:
discovery:
current: D3
target: D5
confidence: medium
rationale: >
Core-vs-adapter classification and behaviours are decided (FederationRequirements ADR-03);
implementation is planned (SHARD-WP-0010), not built.
availability:
current: A0
target: A4
confidence: low
rationale: >
Designed; no implementation yet. Informational/planning reuse only today.
external_evidence:
completeness:
level: C0
name: Absent
confidence: low
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations: []
broken_expectations:
- no derived view is implemented yet
out_of_scope_expectations:
- presentation / rendering of views
reliability:
level: R0
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- planning-stage
discovery:
intent: >
Provide recomputable, provenance-carrying views over the union (link graph, change feed,
enumeration, search) without introducing canonical state.
includes:
- BackLinks (link graph), RecentChanges (journal + shard signals), AllPages, SiteMap
- Search as delegate-to-native-or-derive-index
excludes:
- view presentation / UI
use_cases:
- "shard-wiki UseCaseCatalog UC-17..UC-21, UC-63"
availability:
current_level: A0
target_level: A4
current_artifacts:
- "shard-wiki/workplans/SHARD-WP-0010-derived-views.md"
consumption_modes:
- informational
relations:
depends_on:
- capability.wiki.shard-orchestration
- capability.wiki.page-model
related_to:
- capability.wiki.engine-typed-extensions
evidence:
documentation:
- "shard-wiki/spec/FederationRequirements.md (ADR-03)"
consumer_guidance:
recommended_for:
- planning derived navigation/discovery over a federated wiki union
not_recommended_for:
- implementation reuse today (planning-stage)
known_limitations:
- not implemented; Search ranking policy undecided
---
# Wiki Derived Views
Recomputable views over the union (BackLinks, RecentChanges, AllPages, SiteMap, Search). All
are derived/disposable (no canonical state) and carry provenance; Search is delegate-to-native
where a shard's query capability allows, else a derived index. Planned in SHARD-WP-0010.

View File

@@ -0,0 +1,115 @@
---
id: capability.wiki.engine-typed-extensions
name: Wiki Engine with Typed Extensions
summary: A small-core wiki engine realizing a stringent typed-extension framework that addresses all wiki use cases and lets each shard activate only the features it needs.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, engine, typed-extensions, feature-activation, shard-wiki]
maturity:
discovery:
current: D3
target: D5
confidence: medium
rationale: >
Architecture authored (shard-wiki/spec/WikiEngineCoreArchitecture.md): small page-store
kernel + typed-extension framework, per-shard activation, engine-as-canonical-mode-shard,
and a conflict-mediation realization are explored. Detailed extension SDK/ABI and the API
protocol remain (so D3 Explored, not yet D4/D5).
availability:
current: A0
target: A4
confidence: low
rationale: >
Planned. No engine kernel or extensions exist yet; informational/planning reuse only.
external_evidence:
completeness:
level: C0
name: Absent
confidence: low
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations: []
broken_expectations:
- engine core and typed-extension mechanism not yet designed in detail
out_of_scope_expectations:
- replacing other wiki engines or mandating one implementation
reliability:
level: R0
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- planning-stage capability
discovery:
intent: >
Provide shard-wiki's reference first-party shard backend: a small core + a stringent
typed-extension framework covering all collected use cases, mediating conflicting
requirements into an integrated whole, with per-shard activation (only what you need).
includes:
- a minimal engine kernel (page lifecycle, storage via the adapter contract, the typing mechanism)
- typed extensions that declare contracts and compose
- per-shard feature activation
excludes:
- replacing or mandating other wiki engines (it is one shard type among many)
- a single canonical implementation for all wikis
use_cases:
- "shard-wiki UseCaseCatalog UC-08..UC-25 and the full catalog (the engine must cover all)"
availability:
current_level: A0
target_level: A4
current_artifacts:
- "shard-wiki/workplans/SHARD-WP-0013-wiki-engine-prep.md"
- "shard-wiki/spec/WikiEngineCoreArchitecture.md"
consumption_modes:
- informational
relations:
depends_on:
- capability.wiki.adapter-contract
- capability.wiki.page-model
related_to:
- capability.feature-control.evaluate
- capability.authorization.policy-evaluate
evidence:
documentation:
- "shard-wiki/workplans/SHARD-WP-0013-wiki-engine-prep.md"
consumer_guidance:
recommended_for:
- planning a composable, feature-activatable native wiki engine
not_recommended_for:
- implementation reuse today (planning-stage)
known_limitations:
- architecture authored; extension SDK/ABI + API protocol still to design; not yet built
promotion_history:
- date: "2026-06-15"
dimension: discovery
from: D2
to: D3
rationale: WikiEngineCoreArchitecture.md authored (kernel + typed-extension framework explored); INTENT amendment ratified.
author: shard-wiki
---
# Wiki Engine with Typed Extensions
shard-wiki's planned reference first-party shard backend — a *canonical-mode shard* it
implements natively: a small core plus a stringent typed-extension framework addressing all
collected use cases, mediating conflicting requirements into a consistent whole, with per-shard
activation (activate only what you need). It is one shard type among many — not a replacement
for other engines. Per-shard activation is a candidate consumer of
`capability.feature-control.evaluate`.
## Assessment notes
### Discovery
Architecture authored: `shard-wiki/spec/WikiEngineCoreArchitecture.md` (small kernel +
typed-extension framework; engine = canonical-mode shard). INTENT amendment ratified
(2026-06-15, decision 84ffdb48). Extension SDK/ABI + API protocol are the next deliverables.
### Availability
Planning-stage; informational reuse only.

View File

@@ -0,0 +1,97 @@
---
id: capability.wiki.federation-models
name: Selectable Federation-Model Taxonomy
summary: Federation as a plural, composable coordination axis (fork+journal, VCS-replication+ping, query-time graph-join, feed, activity-streams, engine-mirror) selected per space.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, federation, taxonomy, composable, shard-wiki]
maturity:
discovery:
current: D4
target: D6
confidence: high
rationale: >
A six-model taxonomy distilled from a ~23-system synthesis, each model anchored in a
real system, with capability prerequisites and per-space/per-shard composition rules.
availability:
current: A0
target: A4
confidence: low
rationale: >
Designed and specified (FederationArchitecture T17) but not implemented; informational
reuse only today.
external_evidence:
completeness:
level: C1
name: Sparse
confidence: low
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- the model taxonomy and selection/composition rules are documented
broken_expectations:
- no federation transport is implemented yet
out_of_scope_expectations:
- mandating a single federation mechanism
reliability:
level: R0
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- design-stage; no runtime evidence
discovery:
intent: >
Treat federation as selectable and composable rather than one mechanism, so each space
picks fork+journal, VCS-replication, query-join, feed, activity-streams, or engine-mirror.
includes:
- the six federation models + their capability floors
- per-space selection and per-shard composition
excludes:
- imposing one homogeneous federation network
use_cases:
- "shard-wiki UseCaseCatalog UC-26, UC-31, UC-33, UC-71, UC-72, UC-74, UC-79"
availability:
current_level: A0
target_level: A4
current_artifacts:
- "shard-wiki/spec/FederationArchitecture.md (T17)"
consumption_modes:
- informational
relations:
depends_on:
- capability.wiki.shard-orchestration
- capability.wiki.coordination-journal
evidence:
documentation:
- "shard-wiki/spec/FederationArchitecture.md"
- "shard-wiki/research/260614-shard-spectrum-synthesis/findings.md"
consumer_guidance:
recommended_for:
- planning a federation strategy that mixes models per source
not_recommended_for:
- implementation reuse today (design-stage)
known_limitations:
- no transport implemented; informational planning reuse only
---
# Selectable Federation-Model Taxonomy
Federation is plural and composable: fork+journal (Federated Wiki), VCS-replication+ping
(ikiwiki), query-time graph-join (Wikibase SERVICE), feed aggregation, activity streams
(ActivityPub), and engine-mirror (Wiki.js). A space selects a model and composes per shard;
the default is fork+journal over git. Design-stage capability — strong for planning reuse.
## Assessment notes
### Discovery
FederationArchitecture T17, distilled from the shard-spectrum synthesis (v3).
### Availability
Specified, not implemented — informational reuse only.

View File

@@ -0,0 +1,102 @@
---
id: capability.wiki.overlay
name: Overlay-Before-Mutation Write Path
summary: Non-destructive edits (draft -> patch -> apply-under-drift) that let read-only, rate-limited, or lossy backends be edited safely without silent remote mutation.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, overlay, patch, write-path, conflict, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Overlay lifecycle and apply-under-drift semantics are specified (ADR-05, blueprint
Section 8.6) and implemented as a single principled write path.
availability:
current: A2
target: A4
confidence: medium
rationale: >
OverlayEngine (draft/patch/apply), writable adapter, and InformationSpace.edit
exist as a source module; three-way merge is not (refuse-on-drift only).
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- draft -> patch -> apply with fast-forward / refuse-on-drift / keep-draft outcomes
- no silent remote mutation; overlay_state surfaced in provenance
broken_expectations:
- three-way / auto merge not implemented (refuse-on-conflict only)
out_of_scope_expectations:
- federation propagation of applied overlays
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- early implementation; conflict handling is detect-and-refuse only
discovery:
intent: >
Make any sub-write-through backend editable safely: an edit is an overlay first, applied
only on explicit intent and only when the source has not drifted.
includes:
- overlay drafts recorded as coordination-canonical events
- patch rendering (unified diff)
- apply-under-drift (fast-forward / refuse / keep-draft)
excludes:
- destructive write without drift check
use_cases:
- "shard-wiki UseCaseCatalog UC-04, UC-26, UC-29 (remix primitives, overlay)"
availability:
current_level: A2
target_level: A4
current_artifacts:
- "shard-wiki/src/shard_wiki/coordination/overlay.py"
- "shard-wiki/src/shard_wiki/coordination/patch.py"
consumption_modes:
- source module
relations:
depends_on:
- capability.wiki.coordination-journal
- capability.wiki.adapter-contract
evidence:
documentation:
- "shard-wiki/spec/FederationRequirements.md (ADR-05)"
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 8.2, 8.6)"
tests:
- "shard-wiki/tests/test_apply.py"
- "shard-wiki/tests/test_write_path_integration.py"
consumer_guidance:
recommended_for:
- safe editing over read-only / rate-limited / lossy backends
not_recommended_for:
- workflows needing automatic conflict resolution today
known_limitations:
- merge is detect-and-refuse; three-way merge is future work
---
# Overlay-Before-Mutation Write Path
One principled write path: every edit drafts an overlay (a coordination-canonical event),
renders as a patch, and applies under drift checks — fast-forwarding a writable target,
keeping a local draft on a read-only target, and refusing (never clobbering) on external drift.
## Assessment notes
### Discovery
Specified in FederationRequirements ADR-05 and CoreArchitectureBlueprint Section 8.2/8.6.
### Availability
`overlay.py` + `patch.py` + `InformationSpace.edit` ship the path; built in SHARD-WP-0008.

View File

@@ -0,0 +1,104 @@
---
id: capability.wiki.page-model
name: Backend-Neutral Wiki Page Model
summary: A Markdown-first but stretchable page model with stable identity separate from placement and layered provenance, spanning prose to typed-graph and computational shapes.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, page-model, identity, provenance, markdown, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Page shapes (prose, typed records, typed-graph, inline-embedded, non-Markdown,
and four computational shapes) plus identity != placement and layered provenance
are specified and grounded in the dive research.
availability:
current: A2
target: A5
confidence: medium
rationale: >
Identity/Placement/Span/Page and layered ProvenanceEnvelope exist as a source
module; richer shapes (typed-graph, notebook) are modeled but not all built.
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- stable identity distinct from placement and from content fingerprint
- layered (effective-vs-own) provenance with near-zero per-span cost
broken_expectations:
- non-prose shapes (typed-graph, notebook, inline-embedded) not fully realized
out_of_scope_expectations:
- rendering / presentation
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- prose shape is the only exercised path so far
discovery:
intent: >
One backend-neutral lingua franca every consumer sees; every shape reduces to
(content|source, structure, provenance envelope, optional derivation rule).
includes:
- page identity (stable handle) vs placement (N paths/shards) vs equivalence (fingerprint)
- layered provenance envelope (page + span deltas)
- page-shape taxonomy incl. computational shapes
excludes:
- deriving identity from content (a fingerprint identifies a version, not a page)
use_cases:
- "shard-wiki UseCaseCatalog UC-34, UC-39, UC-44..UC-49, UC-55, UC-73, UC-83, UC-84"
availability:
current_level: A2
target_level: A5
current_artifacts:
- "shard-wiki/src/shard_wiki/model/"
- "shard-wiki/src/shard_wiki/provenance/"
consumption_modes:
- source module
relations:
supports:
- capability.wiki.adapter-contract
- capability.wiki.shard-orchestration
evidence:
documentation:
- "shard-wiki/spec/CoreArchitectureBlueprint.md (Section 7)"
- "shard-wiki/spec/FederationRequirements.md (ADR-02, ADR-04)"
tests:
- "shard-wiki/tests/test_model.py"
- "shard-wiki/tests/test_provenance.py"
consumer_guidance:
recommended_for:
- a portable, provenance-carrying representation of wiki pages across backends
not_recommended_for:
- cases needing a single canonical path per page (use identity, not path)
known_limitations:
- non-prose shapes specified ahead of implementation
---
# Backend-Neutral Wiki Page Model
The top narrow waist: a Markdown-first model that stretches to typed records, typed-graph
statements, inline-embedded objects, non-Markdown assets, and computational shapes. Identity
is a stable handle; placement and equivalence are separate mechanisms; provenance is layered
(effective = page envelope + span delta).
## Assessment notes
### Discovery
Specified in CoreArchitectureBlueprint Section 7 and FederationRequirements ADR-02/04.
### Availability
`model/` + `provenance/` ship the prose path and the layered envelope today.

View File

@@ -0,0 +1,114 @@
---
id: capability.wiki.shard-orchestration
name: Wiki Shard Orchestration
summary: Present a union of pages across heterogeneous wiki-shaped shards while preserving each shard's provenance, capabilities, and history.
owner: shard-wiki
status: draft
domain: helix_forge
tags: [wiki, federation, orchestration, union, shard-wiki]
maturity:
discovery:
current: D5
target: D6
confidence: high
rationale: >
Grounded in 84 documented use cases and a twice-reviewed whole-system
architecture (CoreArchitectureBlueprint) derived from ~23 prior-art systems.
availability:
current: A2
target: A5
confidence: medium
rationale: >
InformationSpace orchestrator (attach -> resolve -> read, chorus on
ambiguity) works as a Python source module; network API and incremental
union are planned.
external_evidence:
completeness:
level: C2
name: Partial
confidence: medium
basis: scope_vs_intent_and_consumer_expectations
satisfied_expectations:
- attach folder shards and read a union page with layered provenance
- chorus presentation of equivalent-but-divergent pages (union without erasure)
broken_expectations:
- incremental union maintenance and equivalence index not yet built
- write-through federation transports not yet built
out_of_scope_expectations:
- hosting or replacing the underlying wiki engines
reliability:
level: R1
confidence: low
basis: consumer_quality_signals
known_reliability_risks:
- early implementation; 64 tests but no production exposure
discovery:
intent: >
Let independently stored, differently implemented wikis behave as one
coherent, versionable, inspectable information space without homogenizing them.
includes:
- union resolution across shards (identity-keyed)
- chorus / designated-canonical presentation of equivalent pages
- lazy replication projection of remote content with freshness
excludes:
- implementing a backend wiki engine (see capability.wiki.engine-typed-extensions)
- silent remote mutation
assumptions:
- canonical truth lives in shards + a git coordination journal; the union is derived
use_cases:
- "shard-wiki UseCaseCatalog UC-01..UC-07, UC-26..UC-33 (information space, federation, coordination)"
availability:
current_level: A2
target_level: A5
current_artifacts:
- "shard-wiki/src/shard_wiki/union/"
- "shard-wiki/src/shard_wiki/space.py"
target_artifacts:
- orchestrator network API
consumption_modes:
- source module
relations:
depends_on:
- capability.wiki.adapter-contract
- capability.wiki.page-model
- capability.wiki.coordination-journal
supports:
- capability.wiki.federation-models
evidence:
documentation:
- "shard-wiki/spec/CoreArchitectureBlueprint.md"
- "shard-wiki/spec/FederationArchitecture.md"
tests:
- "shard-wiki/tests/test_union.py"
- "shard-wiki/tests/test_integration.py"
consumer_guidance:
recommended_for:
- composing multiple Markdown/wiki stores into one provenance-preserving view
not_recommended_for:
- replacing a single wiki engine
known_limitations:
- resolution is recompute-on-read until the incremental tier lands
---
# Wiki Shard Orchestration
shard-wiki's core capability: orchestrate wiki-shaped content across heterogeneous *shards*
as a union of pages, preserving provenance, capabilities, and history per shard. Canonical
truth stays at the edges (shards + the git coordination journal); the union is a derived,
recomputable view (orchestrator, not engine).
## Assessment notes
### Discovery
Grounded by `UseCaseCatalog.md` (84 UCs) and the hardened `CoreArchitectureBlueprint.md`.
### Availability
`InformationSpace` provides attach/resolve/read today (source module); a network API is the
target availability step.

View File

@@ -0,0 +1,145 @@
version: 1
updated: '2026-06-16'
domain: helix_forge
capabilities:
- id: capability.wiki.shard-orchestration
name: Wiki Shard Orchestration
summary: Present a union of pages across heterogeneous wiki-shaped shards while
preserving each shard's provenance, capabilities, and history.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.shard-orchestration.md
tags:
- wiki
- federation
- orchestration
- union
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.adapter-contract
name: Capability-Aware Shard Adapter Contract
summary: A versioned backend interface where each binding declares a verified capability
profile, so federation ops degrade by capability.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.adapter-contract.md
tags:
- wiki
- adapter
- capability
- contract
- conformance
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.page-model
name: Backend-Neutral Wiki Page Model
summary: A Markdown-first but stretchable page model with stable identity separate
from placement and layered provenance.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.page-model.md
tags:
- wiki
- page-model
- identity
- provenance
- markdown
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.coordination-journal
name: Event-Sourced Coordination Journal
summary: An append-only, totally-ordered-per-space decision log whose current state
is a derived fold; git-addressable history.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.coordination-journal.md
tags:
- wiki
- event-sourcing
- coordination
- git
- journal
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.overlay
name: Overlay-Before-Mutation Write Path
summary: Non-destructive edits (draft -> patch -> apply-under-drift) that let read-only
or limited backends be edited safely without silent remote mutation.
vector: D5 / A2 / C2 / R1
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.overlay.md
tags:
- wiki
- overlay
- patch
- write-path
- conflict
- shard-wiki
consumption_modes:
- source module
- id: capability.wiki.federation-models
name: Selectable Federation-Model Taxonomy
summary: Federation as a plural, composable coordination axis (fork+journal, VCS-replication,
query-join, feed, activity-streams, engine-mirror) selected per space.
vector: D4 / A0 / C1 / R0
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.federation-models.md
tags:
- wiki
- federation
- taxonomy
- composable
- shard-wiki
consumption_modes:
- informational
- id: capability.wiki.engine-typed-extensions
name: Wiki Engine with Typed Extensions
summary: A small-core wiki engine realizing a typed-extension framework that addresses
all wiki use cases and lets each shard activate only the features it needs.
vector: D3 / A0 / C0 / R0
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.engine-typed-extensions.md
tags:
- wiki
- engine
- typed-extensions
- feature-activation
- shard-wiki
consumption_modes:
- informational
- id: capability.wiki.derived-views
name: Wiki Derived Views
summary: Recomputable views over a wiki union — BackLinks, RecentChanges, AllPages,
SiteMap, and (delegate-or-derive) Search — carrying provenance.
vector: D3 / A0 / C0 / R0
domain: helix_forge
status: draft
owner: shard-wiki
path: registry/capabilities/capability.wiki.derived-views.md
tags:
- wiki
- derived-views
- backlinks
- recentchanges
- search
- shard-wiki
consumption_modes:
- informational

View File

@@ -0,0 +1,36 @@
# 260613 — Foswiki deep dive (store abstraction, extension API, ecosystem)
Date: 2026-06-13
## What this is
A focused study of **Foswiki** — the community fork of TWiki (2008, from TWiki
4.2.x) — concentrating on **what Foswiki changed**, not the lineage it shares with
TWiki. The distinctive material: a **pluggable `Foswiki::Store` backend**
(RcsWrap / RcsLite / **PlainFileStoreContrib**) behind a versioned interface, the
`Foswiki::Meta` topic-object indirection, the OO/MVC core rewrite, the cleaner
`Foswiki::Func` + `registerTagHandler` extension API, **DataForms** (+ MetaDataPlugin,
FlexFormPlugin), and the **WysiwygPlugin** TML↔HTML round-trip.
Read through shard-wiki's lens: Foswiki is the engine that **already separates its
store backend from its core via a versioned interface** — i.e. it is concrete prior
art for shard-wiki's own *shard adapter contract* and *capability-aware adapters*.
Pairs with — and deliberately does not repeat — the shared TWiki material:
- `research/260613-twiki-deep-dive/` — flat-file/RCS store, Webs/Topics, plugin
handler callbacks, per-topic ACL (all inherited; read there for the shared parts)
- `research/260613-xwiki-deep-dive/` — the Java/DB app-platform sibling
- `research/260608-wikiengines-overview/` — landscape scan
## Contents
| Path | Role |
|------|------|
| `findings.md` | Distinctives vs TWiki, store abstraction, extension API, DataForms, capability profile, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Two new use cases promoted to
`spec/UseCaseCatalog.md` (UC-42, UC-43); UC-34/39/40 enriched. The
`Foswiki::Store`/`Foswiki::Meta` abstraction is logged as adapter-contract prior art
for `workplans/SHARD-WP-0002-federation-architecture.md`.

View File

@@ -0,0 +1,188 @@
# Findings — Foswiki: store abstraction, extension API, ecosystem
Date: 2026-06-13 · Status: research draft
Scope: **Foswiki as distinct from TWiki** — only the deltas, since the two share a
data model, markup, plugin-handler API, and per-topic ACL. The deltas that matter to
shard-wiki: a **pluggable store backend behind a versioned interface**
(`Foswiki::Store` + `Foswiki::Meta`), an OO/MVC core rewrite, a cleaner extension
API (`Foswiki::Func::registerTagHandler`), enhanced DataForms, and a TML↔HTML
round-trip editor framework. Sources: foswiki.org Development + System docs,
`foswiki/distro` on GitHub.
**Read first for shared lineage:** `research/260613-twiki-deep-dive/` (flat-file +
RCS store, Webs/Topics, handler callbacks, per-topic `ALLOW/DENY` ACL). This file
does **not** repeat those.
---
## 1. Origin and compatibility
Forked from **TWiki 4.2.x** in October 2008 by most of the TWiki community (sources
cite 4.2.34.2.4; the TWiki dive recorded 4.2.4). Design goal: ~100% content
compatibility — same TML markup, same plugin API — via a **`TWikiCompatibilityPlugin`**
that maps the `TWiki::` namespace and legacy variables. So everything in the TWiki
dive still applies; Foswiki then grew the features below.
## 2. The headline delta — a pluggable store behind a versioned interface
This is the reason Foswiki earns its own dive.
### 2.1 `Foswiki::Meta` indirection
After Foswiki 1.0.4 the core was changed to **delegate almost all store operations
to a topic object, `Foswiki::Meta`**. Callers manipulate a `Foswiki::Meta` object
representing the item they're changing and **never touch the store implementation
directly**; `Foswiki::Meta` talks to the real store through the well-defined
**`Foswiki::Store`** interface. This is a clean MVC/repository separation.
### 2.2 Swappable backends
`Foswiki::Store::` ships multiple low-level back-ends behind that one interface:
- **`RcsWrap`** — RCS via the external `rcs` binaries (classic TWiki behavior)
- **`RcsLite`** — a pure-Perl RCS implementation
- **`PlainFileStoreContrib`** — a newer store that saves topics/attachments as
**timestamped copies** instead of RCS diffs: more disk, but **no RCS dependency**
and much higher performance.
So in Foswiki the **storage format is a configuration choice**, not a fixed property
of the engine — the same wiki can run on RCS or PlainFile behind an unchanged core.
### 2.3 OO/MVC core rewrite
Foswiki has steadily rearchitected from "disconnected Perl CGI scripts" toward
object-oriented, unit-tested, MVC Perl ("solidify the sand piles"), preserving
backward compatibility through tests. The store abstraction above is the most
load-bearing result.
## 3. Extension API deltas
Shared with TWiki: handler callbacks (`initPlugin`, `commonTagsHandler`,
`before/afterSaveHandler`, `afterRenameHandler`, attachment handlers), REST handlers,
`Config.spec`, package types (Plugin / Skin / AddOn / Contrib). Foswiki refinements:
- **`Foswiki::Func` is the blessed API** — "if there's a `Foswiki::Func` way and
another way, the `Foswiki::Func` way is almost always right." Direct use of
internal packages is discouraged and governed by **`PluginsApiPolicies`**.
- **`Foswiki::Func::registerTagHandler($tag, \&sub)`** — register a custom
macro/variable **programmatically** (cleaner than TWiki's `commonTagsHandler`
string-matching convention).
- **Contrib-defined APIs** carry an explicit "use at your own risk / awaiting merge"
policy — a maturity signal on extension interfaces.
- `EmptyPlugin` (`lib/Foswiki/Plugins/EmptyPlugin.pm`) is the canonical handler
reference/skeleton.
## 4. DataForms and structured data (delta)
DataForms = Foswiki's name for TWiki Forms: typed fields (text, date, single/multi
value, label) stored as `%META:FIELD%` in the topic text. Foswiki extends the model
via popular extensions:
- **MoreFormfieldsPlugin** — additional special-purpose field types.
- **FlexFormPlugin** — render DataForm interfaces from custom templates
(`Foswiki::Form` classes).
- **MetaDataPlugin** — **multiple structured data records per topic** (beyond the
classic one-form-per-topic), closing the gap toward XWiki's multi-XObject model
while keeping data in the text file.
## 5. Editing / syntax (delta relevant to "Markdown-first")
- **WysiwygPlugin** — a generic framework that transforms **TML → HTML** for a
browser editor and **HTML → TML on save** (lossless round-trip).
- **TinyMCEPlugin** — pre-installed WYSIWYG editor built on WysiwygPlugin.
This proves **bidirectional, lossless translation between an engine's native markup
and another representation is feasible** — directly relevant to shard-wiki reading a
non-Markdown shard and writing Markdown overlays back (UC-42).
## 6. Foswiki as a shard — capability profile (delta from TWiki)
| Capability | Foswiki | Note vs TWiki |
|------------|---------|---------------|
| Store backend | **pluggable** | RCS *or* PlainFile behind `Foswiki::Store` — direct-attach target is cleaner on PlainFile (UC-40) |
| History | RCS *or* timestamped copies | PlainFile = whole-version snapshots, not diffs; both file-format, git-importable (UC-41) |
| Structured payload | DataForms `%META%`, **multi-record** via MetaDataPlugin | richer than classic TWiki Forms (UC-34/39) |
| Syntax translation | **WysiwygPlugin TML↔HTML** | bidirectional round-trip exists (UC-42) |
| Extension host | `Foswiki::Func` + `registerTagHandler` + REST | cleaner adapter-host surface (UC-38) |
| Store as contract | **`Foswiki::Store` versioned interface** | architectural prior art for shard-wiki's adapter contract |
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
| Observation | INTENT principle |
|-------------|------------------|
| `Foswiki::Store` versioned interface + swappable backends | **capability-aware adapters** / **shard adapter contract** — an engine that already did exactly this separation |
| `Foswiki::Meta` mediates all store access | clean **mechanism over policy** boundary worth mirroring in the adapter contract |
| PlainFile store = timestamped text copies | **git-addressable coordination** / direct-attach (UC-40), history import (UC-41) |
| WysiwygPlugin TML↔HTML round-trip | **Markdown-first, backend-neutral** is achievable for prose via translation (UC-42) |
| MetaDataPlugin multi-record topics | structured pages beyond one form (UC-34/39) |
### 7.2 Deliberate divergences (design bugs if conflated)
| Foswiki assumption | shard-wiki correction |
|--------------------|----------------------|
| One pluggable store per wiki, chosen at config | shard-wiki federates **many** heterogeneous stores at once |
| TML is the canonical syntax; HTML is a render target | shard-wiki is **Markdown-first**; TML↔Markdown is an adapter translation, not core |
| Store interface is Foswiki-internal Perl | shard-wiki's adapter contract is **cross-language, cross-backend, versioned** |
| ACL in topic preferences | **authorize in core**; engine ACL is advisory provenance |
### 7.3 What Foswiki teaches that shard-wiki should not lose
1. **The store-abstraction pattern works in practice**`Foswiki::Store` is a
real-world proof that a stable interface + swappable backends is viable; the shard
adapter contract is the same idea generalized across engines.
2. **Backend format should be swappable under a stable identity** — Foswiki migrates
RCS↔PlainFile without changing the wiki; shard-wiki should tolerate a shard's
backend changing under it (UC-43).
3. **Lossless syntax round-trip is a solved problem** — don't treat non-Markdown
prose as read-only; translate it (UC-42).
## 8. Use-case seeds → catalog (promoted 2026-06-13)
| Seed | Catalog UC | Disposition |
|------|------------|-------------|
| Read/write a non-Markdown shard via lossless syntax translation (TML↔Markdown) | **UC-42** (new) | realizes Markdown-first for prose; WysiwygPlugin is proof |
| Tolerate a shard's storage-backend swap without losing identity/provenance | **UC-43** (new) | Foswiki RCS↔PlainFile; orchestration robustness |
| Structured / multi-record pages | UC-34 / UC-39 | enriched: DataForms + MetaDataPlugin multi-record |
| Attach a file-backed engine's on-disk store directly | UC-40 | enriched: PlainFile store is a cleaner direct-attach target than RCS |
| `Foswiki::Store` versioned interface | (no UC) | architecture prior art → `SHARD-WP-0002` adapter contract |
## 9. Open questions (for spec / workplans)
1. **Adapter-contract shape** — how much of `Foswiki::Store`'s method set
(read/save/move/getRevisionInfo/getRevisionHistory/lock) maps onto shard-wiki's
capability-aware adapter contract? (architecture, `SHARD-WP-0002`)
2. **Syntax-translation fidelity** — is TML↔Markdown lossless enough for overlay
round-trips, or must overlays be stored in TML to be safe (UC-42)?
3. **Backend-swap detection** — how does shard-wiki notice and tolerate a shard
changing store format underneath (RCS→PlainFile), and does provenance/history
survive it (UC-43)?
4. **Multi-record metadata** — represent MetaDataPlugin multi-record topics and XWiki
multi-XObject pages in one structured-metadata page model (shared open question).
## 10. Sources
| Source | URL |
|--------|-----|
| Foswiki — Technical Overview | https://foswiki.org/Development/TechnicalOverview |
| Foswiki — Core Internals | https://foswiki.org/Development/CoreInternals |
| Foswiki — PlainFileStoreContrib | https://foswiki.org/Extensions/PlainFileStoreContrib |
| Foswiki — RCSStoreContrib | https://foswiki.org/Extensions/RCSStoreContrib |
| Foswiki::Store::PlainFile (source) | https://github.com/foswiki/distro/blob/master/PlainFileStoreContrib/lib/Foswiki/Store/PlainFile.pm |
| Foswiki — Developing Plugins | https://foswiki.org/System/DevelopingPlugins |
| Foswiki — Plugins API Policies | https://foswiki.org/Development/PluginsApiPolicies |
| Foswiki — DataForms | https://foswiki.org/System/DataForms |
| Foswiki — MetaDataPlugin | https://foswiki.org/Extensions/MetaDataPlugin |
| Foswiki — WysiwygPlugin | https://foswiki.org/System/WysiwygPlugin |
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
| shard-wiki — TWiki deep dive | `research/260613-twiki-deep-dive/findings.md` |
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
---
## 11. Traceability
| This document section | Informs (future) |
|-----------------------|------------------|
| §2 store abstraction | **adapter contract** design (`SHARD-WP-0002`) — Foswiki::Store as prior art |
| §3 extension API | UC-38 engine-side adapter host |
| §4 DataForms | structured-metadata page model (UC-34/39) |
| §5 WysiwygPlugin | UC-42 syntax-translation adapter |
| §6 capability profile | adapter capability-profile vocabulary |
| §7 INTENT mapping | architecture-blueprint guardrails |
| §8 UC seeds | `spec/UseCaseCatalog.md` (UC-42, UC-43; UC-34/39/40 enrichment) |

View File

@@ -0,0 +1,39 @@
# 260613 — TWiki deep dive (implementation, plugin API, ecosystem)
Date: 2026-06-13
## What this is
A focused study of **TWiki** — the original "structured enterprise wiki / wiki
application platform" (Peter Thoeny, ~1998), and the **file-based Perl counterpoint**
to XWiki's Java/DB platform. Covers TWiki's **implementation** (Perl + flat-file +
RCS store, Webs/Topics, TWiki Forms, TWikiML/variables), the **plugin API and
handler callbacks** that extend the core, the **extension package types** (Plugin /
Skin / AddOn / Contrib) and repository, and TWiki's **per-topic access control**
(the prior art behind yawex's `AccessControl`).
Read through shard-wiki's lens, paired with the XWiki dive: TWiki shows the same
"wiki-as-application-platform" goal reached with **text files + RCS** instead of a
database — which changes the adapter story (direct on-disk attach, git-convertible
history).
Complements:
- `research/260613-xwiki-deep-dive/` — the DB/Java app-platform sibling
- `research/260608-wikiengines-overview/` — the landscape scan
- `research/260608-yawex-prior-art/` — yawex borrowed TWiki's per-topic AccessControl
Note: most of TWiki's community forked to **Foswiki** in Oct 2008 (from TWiki
4.2.4); findings flag where TWiki/Foswiki differ.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, plugin-handler API, ecosystem, capability profile, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Two new use cases promoted to
`spec/UseCaseCatalog.md` (UC-40, UC-41); UC-06/34/36/38/39 enriched with TWiki
specifics. Adapter-contract implications feed
`workplans/SHARD-WP-0002-federation-architecture.md`.

View File

@@ -0,0 +1,210 @@
# Findings — TWiki: implementation, plugin API, ecosystem
Date: 2026-06-13 · Status: research draft
Scope: TWiki as prior art for the **file-based** wiki-application-platform — the
counterpoint to XWiki's database/component platform. Two shard-wiki concerns:
(1) attaching a structured, file-backed engine as a shard (including attaching its
**on-disk store directly** vs through its runtime), and (2) how TWiki's plugin
handler API exposes hooks rich enough to host a federation adapter
(INTENT *composable integration*). Sources: twiki.org dev docs (TWikiPlugins,
TWikiForms, TWiki::Func, TWikiAccessControl), Foswiki docs, Wikipedia.
**Complements:** `research/260613-xwiki-deep-dive/` (DB/Java sibling),
`research/260608-wikiengines-overview/` (landscape), and
`research/260608-yawex-prior-art/` (yawex borrowed TWiki's per-topic AccessControl).
---
## 1. What TWiki is
A **structured enterprise wiki** and **wiki application platform** (Peter Thoeny,
~1998), written in **Perl 5** (5.10.1+), classically run as CGI. Like XWiki it lets
pages become forms/records/apps — but it reaches that with **flat text files +
RCS**, no database. Most of the community forked to **Foswiki** in October 2008
(from TWiki 4.2.4; Foswiki 1.0 in January 2009); the plugin API and data model below
are shared lineage, with minor divergences noted.
## 2. Implementation architecture
### 2.1 Storage — flat files + RCS (no DB)
- Content lives on disk as `data/<Web>/<Topic>.txt`; the directory tree **mirrors
the logical Web/Topic hierarchy**. Attachments live under `pub/<Web>/<Topic>/`.
- **History is RCS** (`<Topic>.txt,v` companion files) — the same GNU RCS yawex used.
History is real, per-file, and in an **open, git-convertible format** (unlike
XWiki's DB-internal `xwikircs`).
- Scales to 300k+ topics on a single server (Yahoo) — file store is not a toy.
### 2.2 Webs and Topics
- **Topic** = page (the unit of content). **Web** = namespace / collection of topics
(can be nested). Webs are the natural shard / root-entity boundary.
### 2.3 TWiki Forms — structured data, embedded in text
- A **form template** is a topic defining fields as a table (one row per field:
name, type, size, values). A topic can *attach* a form; the field values are
stored as **TWikiMetaData inside the topic `.txt`** (`%META:FIELD{...}%`), shown as
a table on view and edited via fields/radios/checkboxes/lists.
- **Forms + formatted `%SEARCH%`** = TWiki's "database applications" — structured
records queried by the SEARCH variable. This is the XWiki XObject/XClass idea, but
**file-embedded and git-diffable** rather than rows in a DB.
### 2.4 TWikiML / variables (macros)
- Markup is **TWikiML** (not Markdown). Dynamic behavior comes from **variables**
`%VAR%` / `%VAR{...}%` expanded at render time; `%SEARCH%` is the query engine.
- Custom variables are provided by plugins (see §3).
### 2.5 TWiki::Func — the official API boundary
- `lib/TWiki/Func.pm` ("TWikiFuncDotPm") documents **all** interfaces available to
plugins, deliberately abstracting the flat-file assumption: `readTopic`,
`saveTopic`, `getTopicList`, `getListOfWebs`, `getWorkArea`, `saveAttachment`, …
Plugins are told *not* to assume flat files and to go through Func.
## 3. Plugin API — interfaces to extend the core
TWiki extends "without altering core code" by **registering Perl handlers that the
core calls at defined points** in the request/render/save pipeline. A plugin is
`lib/TWiki/Plugins/<Name>.pm` + a documentation topic + an optional `Config.spec`.
### 3.1 Handler callbacks (the hook surface)
| Phase | Handlers |
|-------|----------|
| Init / users | `initPlugin` (must return 1), `registrationHandler` |
| Tag/variable expansion | `commonTagsHandler`, `beforeCommonTagsHandler`, `afterCommonTagsHandler` |
| Rendering | `preRenderingHandler`, `postRenderingHandler` (`start/endRenderingHandler` deprecated) |
| Edit lifecycle | `beforeEditHandler`, `afterEditHandler` |
| Save lifecycle | `beforeSaveHandler`, `afterSaveHandler`, `beforeMergeHandler` |
| Attachments | `beforeAttachmentSaveHandler`, `afterAttachmentSaveHandler` |
| Topic mgmt | `afterRenameHandler`, `completePageHandler` |
These are the TWiki analogue of XWiki's `ObservationManager` events — but
**synchronous pipeline hooks** rather than an async event bus. The save/rename/
attachment handlers are exactly the interception points a federation adapter needs.
### 3.2 Other extension mechanisms
- **Custom variables/macros** — registered via `commonTagsHandler` (Foswiki adds
`registerTagHandler`); convention `%PLUGINNAME_SETTING%`.
- **REST handlers** — a plugin registers REST handlers invoked via the `rest`
script, for transactions outside the standard `view/edit/save` scripts. This is the
natural remote adapter transport.
- **`Config.spec`** — declares configuration items (BOOLEAN, STRING, SELECT, PATH,
PERL) surfaced in the `configure` web UI.
- **Work area** — `getWorkArea()` gives a plugin a persistent, non-web-accessible
data directory.
### 3.3 Extension package types
| Type | Role |
|------|------|
| **Plugin** | handler-based behavior, no core change (`lib/TWiki/Plugins/<Name>.pm`) |
| **Skin** | visual appearance only; topic content unchanged (e.g. PrintSkin) |
| **AddOn** | a script in `bin/` callable from a topic |
| **Contrib** | shared library code, or **alternative implementations of core sections** (e.g. user management) when something "can't be a plugin because it needs close core access" |
## 4. Access control (origin of yawex's model)
TWiki authorizes via **preference settings in topics**: `ALLOWWEBVIEW` /
`ALLOWWEBCHANGE` at web level and `ALLOWTOPICVIEW` / `ALLOWTOPICCHANGE` /
`ALLOWTOPICRENAME` (and `DENY*`) at topic level, with **VIEW / CHANGE / RENAME
grantable separately**. Topic-level settings apply only to that topic. This
per-topic ACL is the **direct prior art** behind yawex's `AccessControl`
(denied/view/form/edit/admin) and a concrete reference for shard-wiki's optional
**per-page ACL at L4** (`spec/ArchitectureBlueprint.md` §5).
## 5. TWiki as a shard — capability profile
| Capability | TWiki | Note |
|------------|-------|------|
| Read | ✓ | TWiki::Func / `view` script / `rest`; or read `data/*.txt` directly |
| Write | ✓ | saveTopic / `save` / `rest`; **per-topic** granularity |
| Structured payload | ✓ | TWiki Forms → `%META:FIELD%` **in the text file** (git-diffable) |
| Version/diff | ✓ | RCS `.txt,v`**open format, git-convertible** (cf. XWiki DB) |
| Merge | partial | `beforeMergeHandler`; no git 3-way |
| Change hooks | ✓ | synchronous save/rename/attachment handlers |
| Auth model | per-web/topic ACL | authorize through shard-wiki core; engine ACL is advisory |
| Direct-store attach | ✓✓ | `data/<Web>/<Topic>.txt` + RCS is a folder shard on its own |
| Federation hooks | ✓ | plugin handlers + REST handlers can host an adapter (UC-38) |
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
### 6.1 Reinforcements
| Observation | INTENT principle |
|-------------|------------------|
| Forms store data as `%META%` **in the topic file** | structured pages can be **git-diffable**, strengthening UC-34/39 vs XWiki's DB |
| RCS `.txt,v` history | **git-addressable coordination** — here history is *convertible*, not just supplementable (UC-36 → UC-41) |
| `data/<Web>/<Topic>.txt` mirrors logic | an engine's **on-disk store is itself attachable as a folder shard** (UC-40) |
| Plugin save/rename/REST handlers | **composable integration** — TWiki can host an adapter too, generalizing UC-38 beyond XWiki |
| Per-topic `ALLOW/DENY` ACL | prior art for **per-page ACL at L4**; lineage TWiki→yawex (UC-06) |
| Webs as namespaces / shards | namespace navigation (UC-22); webs as roots |
### 6.2 Deliberate divergences (design bugs if conflated)
| TWiki assumption | shard-wiki correction |
|------------------|----------------------|
| TWikiML + variable expansion is the page | **Markdown-first**; TWikiML render stays out of core |
| ACL lives in topic preference settings | **authorize in core**; topic ALLOW/DENY is advisory provenance |
| History = RCS files next to content | **coordination journal** is the space-level git layer (may *import* RCS) |
| Apps = Forms + `%SEARCH%` rendered by TWiki | represent records **without** depending on TWiki to render/search them |
| Attach = run TWiki and call it | shard-wiki may attach the **bare data dir** with TWiki offline (UC-40) |
### 6.3 What TWiki teaches that shard-wiki should not lose
1. **File-backed engines are the easy, high-value case** — their store is already a
folder of text + open-format history; the adapter can often skip the runtime.
2. **Structured data in text beats structured data in a DB** for federation —
diffable, portable, journal-friendly. Prefer file-embedded metadata where offered.
3. **One backend, two attachment paths** (runtime API vs on-disk store) is a real
capability/consistency trade-off the adapter contract must express.
## 7. Use-case seeds → catalog (promoted 2026-06-13)
| Seed | Catalog UC | Disposition |
|------|------------|-------------|
| Attach a live engine's **on-disk store directly** (vs its runtime API) | **UC-40** (new) | dual-path attach; fidelity/consistency trade-off |
| **Import** an engine's native file-based history (RCS) into the journal | **UC-41** (new) | history *migration* with fidelity, vs UC-36 *supplementation* |
| Engine hosts adapter via its plugin/handler API | UC-38 | enriched: TWiki save/rename/REST handlers generalize it beyond XWiki |
| Structured page carries typed data | UC-34 / UC-39 | enriched: TWiki Forms `%META%`, git-diffable |
| Internal-history engine | UC-36 | enriched: contrast DB (supplement) vs RCS (import, UC-41) |
| Authenticated team wiki / per-page ACL | UC-06 | enriched: TWiki per-topic ACL is the origin of yawex's model |
## 8. Open questions (for spec / workplans)
1. **Dual-path adapters** — when does shard-wiki attach an engine's on-disk store
directly vs go through its API? Consistency risk of reading live files under a
running engine; capability-gate it (UC-40)?
2. **History import fidelity** — can RCS `.txt,v` be converted to git commits
preserving author/timestamp, and is that authoritative or a one-time backfill
feeding the coordination journal (UC-41, cf. UC-36 Q3)?
3. **File-embedded metadata mapping** — map TWiki `%META:FIELD%` (and XWiki XObjects)
onto one structured-metadata page-model representation (shared with xwiki §8 Q1).
4. **TWiki vs Foswiki** — target both via one adapter (shared API) or treat as
distinct shard types?
5. **ACL provenance** — surface a topic's `ALLOW/DENY` as read-only provenance even
though authorization is decided in core?
## 9. Sources
| Source | URL |
|--------|-----|
| TWiki — TWikiPlugins (handlers, package types) | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
| TWiki — Developing Plugins / EmptyPlugin | https://twiki.org/cgi-bin/view/TWiki/TWikiPlugins |
| TWiki — TWiki::Func module | https://twiki.org/cgi-bin/view/TWiki/TWikiFuncModule |
| TWiki — TWiki Forms | https://twiki.org/cgi-bin/view/TWiki/TWikiForms |
| TWiki — TWiki Access Control | https://twiki.org/cgi-bin/view/TWiki/TWikiAccessControl |
| TWiki — System Requirements (Perl/RCS) | https://twiki.org/cgi-bin/view/TWiki/TWikiSystemRequirements |
| TWiki — Source Code | https://twiki.org/cgi-bin/view/TWiki/SourceCode |
| Foswiki — Why this fork | https://foswiki.org/Home/WhyThisFork |
| Wikipedia — TWiki | https://en.wikipedia.org/wiki/TWiki |
| Wikipedia — Foswiki | https://en.wikipedia.org/wiki/Foswiki |
| shard-wiki — XWiki deep dive | `research/260613-xwiki-deep-dive/findings.md` |
| shard-wiki — yawex prior art | `research/260608-yawex-prior-art/findings.md` |
---
## 10. Traceability
| This document section | Informs (future) |
|-----------------------|------------------|
| §2 architecture | adapter design for file-backed engines |
| §3 plugin handlers | UC-38 engine-side adapter; composable-integration API shape |
| §4 access control | per-page ACL at L4 (`spec/ArchitectureBlueprint.md`), UC-06 |
| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) |
| §6 INTENT mapping | architecture-blueprint guardrails |
| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-40, UC-41; UC-06/34/36/38/39 enrichment) |
| §8 open questions | spec — dual-path adapters, history import, structured-metadata model |

View File

@@ -0,0 +1,35 @@
# 260613 — XWiki deep dive (implementation, extension interfaces, ecosystem)
Date: 2026-06-13
## What this is
A focused study of **XWiki** — the canonical "wiki-as-application-platform"
engine flagged in `research/260608-wikiengines-overview/`. Goes past the landscape
scan into XWiki's **implementation** (component/DI architecture, document data
model, storage, rendering, events), the **interfaces it exposes to extend the core
engine** (components, macros, UI extension points, script services, REST), and the
**extension ecosystem** (extensions.xwiki.org repository, XAR/JAR/WebJar types,
notable extensions).
Read through one lens: **what does XWiki teach shard-wiki about (a) adapting a
structured app-platform engine as a shard, and (b) how a host engine can become
federation-capable through its own extension mechanism** (INTENT *composable
integration*).
Complements:
- `research/260608-wikiengines-overview/` — the landscape scan that surfaced XWiki
- `research/260608-federation-concepts/` — already covered XWiki's ActivityPub extension
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, extension-interface catalog, ecosystem, capability profile, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Two new use cases promoted to
`spec/UseCaseCatalog.md` (UC-38, UC-39); existing UC-34/UC-36/UC-31 enriched with
XWiki specifics. Adapter-contract implications feed
`workplans/SHARD-WP-0002-federation-architecture.md`.

View File

@@ -0,0 +1,217 @@
# Findings — XWiki: implementation, extension interfaces, ecosystem
Date: 2026-06-13 · Status: research draft
Scope: XWiki as **prior art for two shard-wiki concerns** — (1) attaching a
structured "wiki-as-application-platform" engine as a shard, and (2) how an engine
exposes extension interfaces rich enough to *host* a federation adapter
(INTENT *composable integration*). Sources are XWiki's own dev docs, the
DeepWiki code analysis of `xwiki/xwiki-platform`, and the extensions repository.
**Complements:** `research/260608-wikiengines-overview/` (landscape) and
`research/260608-federation-concepts/` §3.2 (XWiki ActivityPub).
---
## 1. What XWiki is
A **programmable, component-based wiki platform** (Java/Jakarta EE, Hibernate ORM)
that blurs wiki and lightweight application platform. Pages are not just prose:
they carry **typed structured objects**, so an XWiki page can be a form, a record,
or a small application. This is the defining property the landscape scan flagged —
and the reason XWiki is the hardest case for shard-wiki's Markdown-first page model.
## 2. Implementation architecture
### 2.1 Component system (the spine)
- **Dependency-injection component model.** Components are interfaces (roles)
discovered via `META-INF/components.txt` and instantiated by a `ComponentManager`.
Implementations declare `@Component`, an optional `@Named` **role hint**, and
`@Singleton`. No interface→impl dependency — impls can ship in separate JARs and
**replace defaults at runtime**.
- Core replaceable roles include `XWikiStoreInterface` (storage), `ObservationManager`
(events), `XWikiRightService` (authorization), `XWikiAuthService` (authentication).
### 2.2 oldcore vs modular platform
- `xwiki-platform-oldcore` holds the foundational document model, persistence, and
legacy APIs (`com.xpn.xwiki.*`). A bridge connects it to the newer component world
to avoid circular dependencies.
- Newer modules layer on top: `model` (entity references), `rendering`, `security`,
`web` (HTTP actions/templates), `search-solr`.
### 2.3 Data model — document-centric + class/object
- Central entity: **`XWikiDocument`**, addressed by `DocumentReference`
(wiki / space / page). Content is parsed into an **XDOM** (rendering tree).
- **Structured data via a class-object pattern** (OOP-like):
- `BaseClass` / **XClass** — schema (field definitions), like a class declaration
- `BaseObject` / **XObject** — an instance conforming to a class, attached to a page
- `PropertyClass` / `BaseProperty` — field definition / value
- Property types: String, LargeString, Date, Number, List, …
- A single page can hold content **plus N typed objects** plus attachments. This is
the structural fact shard-wiki must represent without flattening.
### 2.4 Storage & history
- Two-tier: **`XWikiCacheStore`** over **`XWikiHibernateStore`** (Hibernate, multi-DB).
- Tables: `xwikidoc`, `xwikiobject`, `xwikiproperties`, `xwikiattachment`, and
**`xwikircs`** — internal **RCS-style version history**. History is real but
**engine-internal and non-portable** (directly relevant to shard-wiki UC-36).
### 2.5 Events & rendering
- **`ObservationManager`** event bus: cancelable (`DocumentCreatingEvent`,
`DocumentUpdatingEvent`, `DocumentDeletingEvent`) and post-facto
(`DocumentCreatedEvent`, `DocumentUpdatedEvent`, …). Listeners are components —
a natural **change-notification source for incremental sync**.
- **Rendering pipeline:** source syntax → parse → XDOM → transformations (macros) →
serialize (HTML, PDF, …). Pluggable parsers/renderers; XWiki can parse **Markdown**
as an input syntax even though its native syntax differs.
### 2.6 Multi-wiki
- One installation hosts many wikis (`WikiReference`), each with separate documents
and configuration — XWiki's own multi-tenancy, conceptually adjacent to
shard-wiki's root-entity/tenant boundary.
### 2.7 APIs & security
- **Dual API surface:** internal `com.xpn.xwiki.*` (full power) and script
`com.xpn.xwiki.api.*` (wrapped, enforces **programming rights** and view rights).
Privilege is enforced at the script boundary, not the storage boundary.
## 3. Interfaces to extend the core engine
XWiki's extensibility is unusually layered. From lowest/most-powerful to highest:
| # | Mechanism | What it extends | How |
|---|-----------|-----------------|-----|
| 1 | **Components** | anything | implement a role interface, `@Component` + `@Named` hint; ship in a JAR with `components.txt`; can override defaults (store, auth, rights) |
| 2 | **Rendering macros (Java)** | content processing | register `Macro.class` role + hint; participates in the XDOM transformation pipeline |
| 3 | **Wiki macros** | content processing | a wiki page following a spec — **no Java, no compile**; in-wiki authoring |
| 4 | **Script services** | scripting API | implement `org.xwiki.script.service.ScriptService`; exposed to Velocity/Groovy as `$services.<hint>` |
| 5 | **UI Extensions (UIX)** at **UI Extension Points (UIXP)** | the interface, no skin edit | UIX stored as XObjects of `XWiki.UIExtensionClass` (label/target/icon); injected at named hooks (since v4.2), e.g. `org.xwiki.platform.panels.Applications` |
| 6 | **REST API (JAX-RS)** | remote integration | resources for pages, spaces, **objects**, classes, attachments, comments, tags, annotations, users/groups, **history**; custom JAX-RS resources can be added |
| 7 | **Event listeners** | reactive behavior | register a component on `ObservationManager` to react to document lifecycle events |
| 8 | **Resource reference handlers** | new URL/resource types | implement `ResourceReferenceHandler` on the root component manager |
| 9 | **Skins / templates / Velocity** | presentation | Flamingo skin macros, templates |
Two things stand out for shard-wiki: (a) the **component model lets an extension
replace authentication, authorization, and storage** — i.e. an XWiki instance could
delegate auth exactly as shard-wiki's blueprint prescribes; (b) the **REST API plus
event bus** give a federation adapter both a transport (read/write pages+objects+
history) and a push-based freshness signal.
## 4. Extension ecosystem
- **`extensions.xwiki.org`** repository: **900+ extensions** (applications, macros,
skins, panels, themes, legacy plugins). Browsed/installed/upgraded at runtime via
the **Extension Manager Application** from the admin UI.
- **Extension types:**
- **XAR** — a package of wiki pages (an application, a wiki macro, or a snippet)
- **JAR** — server-side Java: **components and script services**
- **WebJar** — JAR of web resources (JS/CSS/Less)
- **"Plugin" (legacy) vs Component (modern):** the old plugin API is superseded by
the component system; new code is components.
- **Notable extensions** (relevance to shard-wiki in parentheses):
- **AppWithinMinutes** — build structured apps (XClass+XObjects) from the UI
(→ pages-as-records, the UC-39 case)
- **LDAP / Active Directory** and **Entra ID (Azure AD)** authentication
(→ confirms pluggable `XWikiAuthService`; mirrors shard-wiki's delegated authn)
- **ActivityPub** — fediverse federation (→ adapter transport, cf. UC-31)
- **LiveData / LiveTable** — queryable tabular views over XObjects
- draw.io diagrams, PDF viewer, Task Manager, Blog — content/app breadth
## 5. XWiki as a shard — capability profile
Read through shard-wiki's capability-aware adapter lens:
| Capability | XWiki | Note |
|------------|-------|------|
| Read | ✓ | REST + render to HTML/Markdown |
| Write | ✓ | REST PUT page/object; **per-document** granularity (page + its objects) |
| Structured payload | ✓✓ | XObjects/XClass — *richer* than Markdown; must not be flattened (UC-34/UC-39) |
| Version/diff | ✓ (internal) | `xwikircs`; **not git-portable** → coordination journal supplies that (UC-36) |
| Merge | partial | no native git-style 3-way merge |
| Change events | ✓ | `ObservationManager` → push-based sync/freshness (enriches UC-31) |
| Auth model | pluggable | `XWikiAuthService`/`XWikiRightService` replaceable — authorize through shard-wiki core, don't trust engine ACLs |
| Federation hooks | ✓✓ | component + REST + UIX make XWiki able to **host** a shard-wiki adapter (UC-38) |
| Multi-tenancy | ✓ | `WikiReference` farm → multiple shards / root entities |
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
### 6.1 Reinforcements
| Observation | INTENT principle |
|-------------|------------------|
| Pages = content + typed XObjects | **Markdown-first, backend-neutral** must carry structure, not flatten (UC-34/39) |
| `xwikircs` internal-only history | **Git-addressable coordination** adds the portable layer (UC-36) |
| Pluggable `XWikiAuthService` | **authorization in core, authentication delegated** — XWiki already does exactly this |
| Component + REST + UIX extension surface | **composable integration** — engine becomes federation-capable via its own API (UC-38) |
| `ObservationManager` events | **explicit provenance/freshness** via push, not poll (UC-31, UC-24) |
| `WikiReference` multi-wiki | root-entity / tenant boundary has prior art |
### 6.2 Deliberate divergences (design bugs if conflated)
| XWiki assumption | shard-wiki correction |
|------------------|----------------------|
| Native XWiki syntax + macro render is the page | shard-wiki keeps **Markdown-first**; engine render stays out of core, structure travels as metadata |
| ACLs/rights live in the engine DB | **authorize in core**; engine rights are advisory, not trusted |
| History is the engine's RCS table | **coordination journal** is space-level and git-backed |
| Federation = an XWiki extension talking to other XWikis | shard-wiki federates **heterogeneous** backends; XWiki is one shard among many |
| App = XObjects rendered by XWiki | shard-wiki must represent records **without** depending on XWiki to render them |
### 6.3 What XWiki teaches that shard-wiki should not lose
1. **A clean role/hint component model** is what makes auth/store/rights swappable —
shard-wiki's adapter contract should be similarly role-based and override-friendly.
2. **Structured objects are first-class**, not an afterthought — the page model needs
a typed-metadata escape hatch from day one.
3. **An engine with a real extension API can host the adapter itself** — the cheapest
path to a high-fidelity shard is engine-side, not screen-scraping.
## 7. Use-case seeds → catalog (promoted 2026-06-13)
| Seed | Catalog UC | Disposition |
|------|------------|-------------|
| Engine hosts a federation adapter via its native extension API | **UC-38** (new) | composable integration — first UC for the engine-side direction |
| Attach a wiki-as-application-platform shard (pages are typed records/forms) | **UC-39** (new) | extends UC-34 toward bodiless structured pages |
| Structured page carries semantic data | UC-34 | enriched with XObject/XClass concretes |
| Internal-only revision history | UC-36 | enriched with `xwikircs` example |
| Subscribe to remote shard changes | UC-31 | enriched with `ObservationManager` event-driven transport |
## 8. Open questions (for spec / workplans)
1. **Page-model representation of XObjects** — typed frontmatter, sidecar file, or
opaque provenance blob? (shared with `wikiengines` §6 Q1; XWiki makes it urgent)
2. **Bodiless record pages** — does shard-wiki's page model require a Markdown body,
or can a page be purely structured (AppWithinMinutes apps)?
3. **Adapter placement** — engine-side extension (high fidelity, needs deploy access)
vs external REST adapter (zero engine change, lower fidelity). Both? Capability-gated?
4. **Event transport** — consume `ObservationManager` via a JAR listener, or poll REST
history? Affects freshness guarantees (UC-31/UC-24).
5. **Rights mapping** — translate XWiki `XWikiRightService` rights into shard-wiki
actions (read/write/patch/merge/administer), or ignore and authorize purely in core?
## 9. Sources
| Source | URL |
|--------|-----|
| XWiki — Developer Guide | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/ |
| XWiki — Creating XWiki Components | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/WritingComponents/ |
| XWiki — Available Extension Points | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/ExtensionPoint/ |
| XWiki — UI Extension Point Tutorial | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/UIXTutorial/ |
| XWiki — Wiki Macro Tutorial | https://www.xwiki.org/xwiki/bin/view/Documentation/DevGuide/Tutorials/WritingMacros/WikiMacroTutorial/ |
| XWiki — RESTful API | https://www.xwiki.org/xwiki/bin/view/Documentation/UserGuide/Features/XWikiRESTfulAPI |
| XWiki Rendering — Writing a Macro | https://rendering.xwiki.org/xwiki/bin/view/Main/ExtendingMacro |
| Extension Manager Application | https://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Manager%20Application |
| Extension Module — Extensions/types | https://extensions.xwiki.org/xwiki/bin/view/Extension/Extension%20Module/Extensions/ |
| DeepWiki — xwiki/xwiki-platform (code analysis) | https://deepwiki.com/xwiki/xwiki-platform |
| Mancinelli — Overview of XWiki's RESTful API | http://blog.fabio.mancinelli.me/2011/03/07/XWikis_RESTful_API.html |
| shard-wiki — wiki engines overview | `research/260608-wikiengines-overview/findings.md` |
| shard-wiki — federation concepts | `research/260608-federation-concepts/findings.md` |
---
## 10. Traceability
| This document section | Informs (future) |
|-----------------------|------------------|
| §2 architecture | adapter design for app-platform engines |
| §3 extension interfaces | UC-38 engine-side adapter; composable-integration API shape |
| §5 capability profile | adapter capability-profile vocabulary (`SHARD-WP-0002`) |
| §6 INTENT mapping | architecture-blueprint guardrails |
| §7 UC seeds | `spec/UseCaseCatalog.md` (UC-38, UC-39; UC-31/34/36 enrichment) |
| §8 open questions | spec — page model for structured/bodiless pages |

View File

@@ -0,0 +1,38 @@
# 260614 — The computational page model (SHARD-WP-0004 synthesis)
Date: 2026-06-15 · Source: **SHARD-WP-0004** post-batch synthesis (T1T8)
## What this is
The acceptance-criteria synthesis for SHARD-WP-0004 — *"the computational page model"*
reading the eight computational/interactive-knowledge dives across each other and distilling
them into **one model**: **source is canonical; everything rendered/computed is a
projection**, placed on **two axes** (projection-kind: replication vs derivation; liveness:
live↔snapshot), with a recommendation on an executable-content capability.
## The answer to the carried question
*Can a shard-wiki page be a live computational artifact?* **Yes — as a page-model +
projection concern, not as an execution platform.** Every system externalizes to a canonical
source and treats the live/computed form as derived; shard-wiki **recognizes** computational
content, **attaches the source**, and **presents derivations as provenance- and
liveness-marked projections**, with **execution as a gated capability (off by default,
degrade to snapshot)**. No INTENT amendment required.
## Key contributions
- **One model:** `(source, derivation rule, projection with provenance + liveness)` covers
all four computational page shapes (one-source-many-projections UC-83; notebook UC-84;
program-as-page; live/temporal content).
- **Two axes for T16:** replication vs **derivation-projection** (timing / multiplicity /
continuity facets) × the **live↔snapshot** axis (bounded at the irreducibly-live far end by
Strudel).
- **One snapshot-provenance record** reused for notebook outputs, renders, recordings.
- **Hard boundaries:** never host a kernel/runtime as store; **image-is-not-a-store**; never
present a derivation without output→source provenance.
## Contents
| Path | Role |
|------|------|
| `findings.md` | The source/derivation/projection model, the two axes, the four page shapes, provenance/reproducibility, the recommendation, the SHARD-WP-0002 fold-in, escalated open questions |

View File

@@ -0,0 +1,178 @@
# The computational page model — synthesis (SHARD-WP-0004)
**Date:** 2026-06-15 · **Source:** SHARD-WP-0004 (post-batch synthesis, T1T8) · **Kind:**
synthesis (no new external research) reading the eight computational/interactive-knowledge
dives *across* each other.
## What this is
The acceptance-criteria synthesis for SHARD-WP-0004. The batch asked one carried question:
*can a shard-wiki page be a live computational artifact — a source woven/evaluated into
rendered forms — and if so, how do projection, transclusion, provenance, and the adapter
contract treat the source, the environment, and the computed output?* This memo answers it
by distilling the eight dives into **one model**: the **source / derivation / projection**
view of computational content, anchored on two axes, plus a recommendation on whether
shard-wiki needs an executable-content capability.
Dives consolidated: **T1** literate programming (WEB/weave/tangle), **T2** Mathematica,
**T3** Jupyter, **T4** Processing/p5.js, **T5** Strudel, **T6** Squeak, **T7** Glamorous
Toolkit, **T8** Pharo. Catalog yield: **UC-83** (literate one-source-many-projections),
**UC-84** (notebook with computed-output provenance); the other six are **enrichment/boundary**
dives (UC-54/55/47/48 + the projection model).
## 1. The one finding: source is canonical, everything rendered is a projection
Every system in the batch, however "live," **externalizes to a canonical artifact** and
treats the rendered/computed form as **derived**:
| System | Canonical (attach this) | Derived (a projection) |
|--------|-------------------------|------------------------|
| Literate (WEB) | the WEB/`.nw`/`.org` **source** | woven docs **and** tangled code |
| Mathematica | the `.nb` (a Wolfram expression) | cached `Output` cells; CDF; `Dynamic` widgets |
| Jupyter | `.ipynb` source (ideally paired text) | embedded outputs; nbconvert/nbviewer renders |
| Processing | the **sketch source** | the view-time canvas render |
| Strudel | the **pattern source** | the live audio performance / a recording |
| GT / Lepiter | git-versionable **page/Tonel files** | moldable `gtView`s; live snippet results |
| Squeak/Pharo | exported **files** (Tonel/git) | the live image / running objects |
**This is the same principle shard-wiki already holds** (files-canonical, index/render
derived — ikiwiki UC-79, Logseq UC-62, nbstripout). The computational batch did not break it;
it **stress-tested it to the live extreme and it held**. *The image (Squeak/Pharo) is the
only would-be exception, and it is a boundary, not a counterexample: an image is not a store.*
## 2. Two axes the contract must add
The batch refines the **projection model** (SHARD-WP-0002 T16) with two orthogonal axes:
### 2.1 Projection kind: replication vs derivation
- **Replication-projection** (the existing default): a lazy **cache/copy** of remote content
(Obsidian/Notion mirrors, UC-53/57).
- **Derivation-projection** (new, from T1): a **transform/compile/weave/evaluate** of a
source into rendered forms — regenerable, may delegate to the source's tool, **degrades to
a captured snapshot** when the tool is absent. Covers weave/tangle, nbconvert, CDF, sketch
render, audio recording, `gtView`.
Sub-facets of derivation (from T4): **materialization timing** = *ahead-of-time* (CDF,
nbconvert, static HTML) vs *view-time* (Processing/Strudel); **multiplicity** = one output
(UC-79) vs **N co-equal projections** (UC-83 weave+tangle; UC-47/48/54 moldable views).
### 2.2 Liveness: the live↔snapshot axis (from T6, bounded by T5)
Every derived view sits on a spectrum, and the more live the source, the more its static form
is a clearly-marked degrading snapshot:
```
static source ── captured output ── live-over-files ── view-time one-shot ── continuous/
(literate) (notebook UC-84) (GT/Lepiter) (Processing) interactive
── irreducibly
live/temporal
(Strudel: source
+ recording only)
```
**Honesty rule (union-without-erasure):** a computed/live view must always declare *what it
is* — "captured at run N, environment unguaranteed" (UC-84), "one performance, time T, source
rev R" (UC-83/Strudel), "live render needs the runtime." Never present a snapshot as live or a
static page as capturing a live artifact.
## 3. The computational page-model shapes (T12)
The batch adds these page shapes (beyond prose / typed records / query-defined / inline-
embedded objects / typed-graph already catalogued):
1. **One-source-many-projections** (UC-83) — a source whose presented forms are co-equal
derivations (docs + code), each with output→source provenance; **named-chunk transclusion**
assembles fragments by name at derivation time (UC-32/44).
2. **Notebook** (UC-84) — **ordered/nestable cells** (Mathematica adds the outline tree)
where code cells own **embedded computed outputs** (the derived output is stored *inside*
the source) with **weak execution provenance**; outputs may be MIME blobs or **structured
re-evaluable values** (Mathematica) — a new point on the content-opacity spectrum.
3. **Program-as-page** (Processing) — canonical content = **source text**, presentation = an
**executable render** with **no cached output**; non-Markdown executable content.
4. **Live/temporal/generative content** (Strudel) — source canonical, render irreducibly
live; static = source + a marked recording.
All four reduce to **(source, derivation rule, projection with provenance + liveness)** — one
model, four positions.
## 4. Provenance & reproducibility (the honest weakness)
Computed output provenance is **real but fragile** everywhere: Jupyter/Mathematica
`execution_count`/`In`-`Out` can be **out-of-order**; **environment/versions/data are not
captured**; Strudel/Processing may be **non-deterministic**. Implication for the contract:
treat a computed output as a **snapshot with declared, incomplete provenance** (run id, source
rev, timestamp; environment "unguaranteed"), reusing **one snapshot-provenance machinery**
across notebooks, recordings, and renders (UC-84 is the template). This is consistent with
shard-wiki's existing "surface freshness/completeness, never imply more than you have"
(Oddmuse partial-history UC-82).
## 5. The recommendation
**Does shard-wiki need an executable/computational content capability? — Yes, but only as
recognition + projection + capability-gating, never as an execution engine.**
1. **Adopt the source/derivation/projection model** (no execution required). shard-wiki
**recognizes** computational content types (literate source, notebook, sketch, pattern),
**attaches the canonical source**, and **presents derived forms as projections with
provenance + liveness markers**. This alone delivers UC-83/UC-84 and the enrichment of
UC-54/55 — and needs **no kernel, no sandbox, no runtime**.
2. **Make execution a capability, off by default.** "Drive a derivation" (run tangle/weave,
re-execute a notebook, render a sketch, evaluate a pattern in the viewer) is a **gated
capability** with a **trust/sandbox** sub-concern (T11). Absent it, **degrade to the
captured snapshot / static render / recording** — the graceful-degradation rule, which the
batch shows always has an honest fallback (source is tiny and diffable everywhere).
3. **One projection model, two axes** (T16): projection-kind (replication vs derivation; with
timing + multiplicity facets) × liveness (live↔snapshot). The **moldable view registry**
(T7) is the unifying structure — an open, type-keyed set of co-equal projections, none
canonical-by-fact (display-canonical is policy).
4. **One snapshot-provenance record** reused for notebook outputs, renders, and recordings
(run id, source rev, timestamp, environment "unguaranteed").
5. **Hard boundaries** (design-bugs if violated): never host a kernel/runtime as the store;
**image-is-not-a-store** (attach exported files); never present a derivation without
output→source provenance; never imply a static view captures a live artifact.
**Net:** computational content is **in scope as a page-model + projection concern**, **out of
scope as an execution platform** — exactly the mechanism-over-policy, capability-aware,
degradable posture INTENT already mandates. No INTENT amendment is required; this extends the
page model and projection model within existing constraints.
## 6. Fold into SHARD-WP-0002
- **T12 (page model):** add the four computational shapes (§3); allow nestable cells and
structured re-evaluable outputs; "derived output may live inside the source" (notebook).
- **T16 (projection):** the **two-axis model** (§2) + the **moldable view registry** (§3/T7);
materialization-timing, multiplicity, continuity, and the live↔snapshot far end as explicit
projection metadata.
- **T11 (capabilities):** "derive/execute/render/evaluate" as gated capabilities with trust/
sandbox; default off → snapshot.
- **T15 (fidelity):** non-Markdown executable/computed content; lossy renders; the
structured-re-evaluable-value point on the content-opacity spectrum.
- **T13 (history):** paired-text (Jupytext) / cell-aware (nbdime) strategies for embedded-
output documents; outputs-as-derived (nbstripout ethos).
- **T14 (binding):** **image-is-not-a-store** boundary (export→files only).
## 7. Open questions (escalated)
1. Is **liveness** (and "irreducibly live / no faithful static form") an explicit first-class
metadata flag on every projection, so the union renders the honest fallback automatically?
(T5/T6 far-end question.)
2. Does shard-wiki **ever drive a derivation** (sandboxed), or strictly attach + present
snapshots? (Recurs UC-56/UC-83/UC-84/T4/T5 — a single capability/trust policy decision.)
3. Is a computed output's **structured re-evaluable value** (Mathematica/Wolfram) modeled as a
typed value or stored opaquely with provenance? (UC-55 open-Q #10; UC-84 Q3.)
4. Should **UC-83** and **UC-84** eventually merge as two positions of one "source +
derivations" shape, or stay distinct? (Kept distinct: UC-84's defining trait is *output
embedded in source with weak provenance*; UC-83's is *N co-equal external derivations*.)
## 8. Sources
The eight SHARD-WP-0004 dives: `research/260614-{literate-programming,mathematica,jupyter,
processing,strudel,squeak-pharo,glamorous-toolkit}-deep-dive/`. Prior projection/structure
anchors: `research/260614-{ikiwiki,logseq,zigzag}-deep-dive/`,
`research/260614-shard-spectrum-synthesis/`.
## 9. Traceability
No new UC (consolidation). Consolidates **UC-83, UC-84** and enrichments to **UC-32, UC-44,
UC-47, UC-48, UC-54, UC-55, UC-79, UC-37, UC-35**. Feeds **SHARD-WP-0002 T11/T12/T13/T14/T15/
T16** (see §6). Recommendation: computational content is **in scope as page-model + projection
mechanism, out of scope as an execution platform**; no INTENT amendment required.

View File

@@ -0,0 +1,20 @@
# 260614 — Federated Wiki deep dive
Deep dive on Ward Cunningham's **Federated Wiki** (Smallest Federated Wiki / SFW,
2011 →) as a **federation model** rather than a single shard: fork-with-provenance,
the per-page **JSON journal** of semantic actions, the **story** of typed items, the
**neighborhood/roster** discovery model, and time-bounded **happenings**.
This is prior art for shard-wiki's **coordination layer itself** — the closest existing
system to "a union of pages preserving provenance, assembled non-destructively." It
extends `research/260608-federation-concepts/` §3 with the concrete data model + protocol.
- `findings.md` — full writeup: data model, journal/action types, federation protocol,
capability profile, INTENT mapping, UC seeds (UC-70UC-72), architecture notes for
SHARD-WP-0002, open questions, sources, traceability.
Catalog yield: UC-70 (attach a fedwiki site via page-JSON + CORS), UC-71 (append-only
semantic action journal with site provenance as a coordination-journal model), UC-72
(fork-with-site-provenance federation across a neighborhood / chorus). Enriched
UC-26/28/30/05/27. Feeds SHARD-WP-0002 T1T5 (federation) and T11/T13/T16 (write
granularity, log-based merge, identity≠placement).

View File

@@ -0,0 +1,240 @@
# Federated Wiki — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T1 · **Subject:** Ward Cunningham's
Smallest Federated Wiki (SFW) / Federated Wiki (fedwiki ecosystem).
## Why this dive
Every prior dive has been a *shard candidate* — a store we might attach. Federated Wiki
is different: it is a **federation model**, the one piece of public prior art whose core
job is the same as shard-wiki's coordination layer — *present a union of pages from many
independent sites while preserving where each came from, and let people copy and edit
non-destructively*. Ward Cunningham (inventor of the wiki) built SFW in 2011 precisely to
fix the original wiki's single-canonical-page weakness with **fork + provenance**. We go
past the surface (`260608-federation-concepts/` §3) into the data model and protocol, then
ask what shard-wiki should adopt.
**Framing:** fedwiki is not just "a shard we attach" — it is a *worked example of the
coordination journal, overlay-before-mutation, and union-without-erasure*, three of our
own design pillars, shipped and running.
---
## 1. The data model — page = title + story + journal
A fedwiki page is a small JSON object with three core fields (plus optional decoration):
```json
{
"title": "Welcome Visitors",
"story": [
{ "type": "paragraph", "id": "7b56f22a4b9ee974",
"text": "Welcome to this [[Federated Wiki]] site." },
{ "type": "image", "id": "a1c0e3...", "url": "...", "caption": "..." }
],
"journal": [
{ "type": "create", "id": "7b56f22a4b9ee974", "item": {...}, "date": 1310000000000 },
{ "type": "add", "id": "a1c0e3...", "item": {...}, "after": "7b56f22a4b9ee974",
"date": 1310000100000 },
{ "type": "edit", "id": "7b56f22a4b9ee974", "item": {...}, "date": 1310000200000 },
{ "type": "fork", "site": "ward.fed.wiki.org", "date": 1310000300000 }
]
}
```
- **story** — an *ordered array of typed items* ("paragraph-like" items). Each item is
`{ type, id, text, ...type-specific }`. The **`id`** is a random 16-hex string,
**stable across edits** (it is the unit of identity within a page). The **`type`** names
the **plugin** that renders/edits the item (`paragraph`, `image`, `html`, `markdown`,
`code`, `method`, `pagefold`, chart plugins, …). *Data lives in the item; behavior lives
in the plugin* — the item is portable JSON; the plugin is the renderer.
- **journal** — an *ordered, append-only array of action objects* that, when replayed,
**reconstructs the story**. The story is a materialized view of the journal. This is the
key architectural choice: **the journal is the source of truth, the story is derived.**
## 2. Journal action types — a semantic op-log
Each journal entry is an action with `{ type, ... , date }` (epoch-ms). The action types:
| action | fields | meaning |
|---------|--------|---------|
| `create`| `id, item, date` | first item — page born |
| `add` | `id, item, after, date` | insert an item after another |
| `edit` | `id, item, date` | replace an item's content (id preserved) |
| `move` | `order, date` | reorder items |
| `remove`| `id, date` | delete an item |
| `fork` | `site, date` | **mark that the page was copied from `site` at this point** |
Two things matter for us:
1. **These are *semantic* operations** (add/move/edit/remove a paragraph), not text diffs
and not character-level CRDT ops. The write granularity is the **story item
(paragraph)** — a *middle* granularity between whole-file (TiddlyWiki) and
block/character (Logseq/CRDT). It is an **op-log** like a CRDT, but the ops are
coarse-grained and **applied by humans via fork**, not auto-merged.
2. **`fork` is the provenance primitive.** When you copy a remote page to your own site,
a `fork` entry is appended recording the **source site** and time. The journal of a
forked page therefore **serializes a directed acyclic graph (DAG)** of where content
came from — "the journal of a forked page is detailed enough to recognize where in the
journal of the original the fork took place" (CouchDB-style per-entry sequence numbers
make the cut-point identifiable). History visualization highlights the forked entry.
## 3. The federation protocol — sites, neighborhood, roster
- **Site** = an independent server (originally Node.js; also static-file and serverless
variants). A site owns a set of pages, each served as **page JSON over HTTP** at
`/<slug>.json`, with **CORS headers** so a *browser-side* client can fetch pages from
**any** site. Page identity within a site is the **slug** (a title-derived kebab name).
- **The client assembles the union, not the server.** The fedwiki client ("the lineup")
renders pages **side by side**: clicking a link opens that page *from whatever site it
resolves against*, appended to the right. Browsing literally builds a left-to-right
trail across sites.
- **Neighborhood** = the dynamic set of sites encountered in the current session (from the
sites of pages you've opened, links, and forks). **Search runs across the neighborhood**
— a federated search over exactly the sites you've touched.
- **Roster** = an explicit, authored list of sites to include (a curated neighborhood);
"sister sites" are peers you watch. There is **no central registry** — discovery is by
link, fork, and roster.
- **Happenings** = time-bounded collaborative events where many participants fork around a
topic for a period, producing a burst of related forks (a bounded collaboration that
leaves a durable forked record on each participant's own site).
## 4. The editorial model — fork, don't edit-in-place
You can only write to **your own** site. To change someone else's page you **fork** it
(copy into your site, journal records the source), then edit your copy. Many forks of the
same page coexist across sites — Cunningham's **"chorus of voices"**: *no canonical
version*, divergence is normal and visible, and you choose whose changes to pull by forking
them. There is **no automatic merge** — reconciliation is human: compare journals, fork the
version you prefer, optionally re-fork upstream changes.
---
## 5. Capability profile
| Dimension (synthesis spectrum) | Federated Wiki |
|--------------------------------|----------------|
| Attachment mode | **REST/file-store hybrid** — page JSON over HTTP+CORS; also static files |
| Addressing granularity | **story item (paragraph)** via stable 16-hex `id` |
| Content identity | item `id` random+stable; page id = site + slug |
| Identity vs placement | **placement-bound**: identity = `site` + `slug`; forks are *new* identities linked by journal provenance |
| Structure | ordered array of **typed items** (plugin-typed) |
| History | **per-page append-only journal** of semantic actions (op-log) |
| Merge model | **fork + manual journal compare** — a *third model* beside git 3-way and CRDT auto-merge |
| Native query | none built-in; **neighborhood search** (federated full-text across touched sites) |
| Translation | item `text` is wiki/Markdown-ish; plugins own their formats |
| Attachment/write granularity | **story-item level** (add/edit/move/remove one item) |
| Operational envelope | tiny servers, browser-driven; CORS is the whole API surface |
| Access grant | **own-site-only writes**; reads open via CORS |
| Content opacity | transparent JSON (no E2EE); plugin-typed but inspectable |
| Provenance | **first-class**`fork` records source site; journal = provenance DAG |
## 6. INTENT mapping
### Reinforcements (fedwiki validates our pillars)
- **Coordination journal** (INTENT) ≈ fedwiki **journal**. Our journal idea is *exactly*
fedwiki's per-page append-only action log — and fedwiki proves the story-as-derived-view
pattern works. Strong reinforcement; adopt the **semantic-op + provenance-entry** shape.
- **Overlay before mutation** ≈ **fork**. Fork *is* the canonical overlay: a
non-destructive copy onto a writable surface, recording provenance, before any change.
- **Union without erasure** ≈ **neighborhood + chorus**. The union is assembled from many
sovereign sites; provenance (which site, forked-from) is never hidden; divergence is
surfaced, not resolved away.
- **No silent remote mutation** ≈ **own-site-only writes**. You structurally *cannot*
mutate a remote; you fork to your own site. This is our rule, enforced by architecture.
- **Mechanism over policy** ≈ **no canonical source**. Fedwiki ships the mechanism (fork,
journal, neighborhood) and leaves "which version wins" entirely to people.
- **Graceful degradation** ≈ static-file sites — a fedwiki site can be a read-only pile of
JSON files; still forkable, still in the neighborhood.
### Divergences (boundaries / design notes, not bugs)
- **Identity = placement.** Fedwiki page identity is `site` + `slug`; a fork is a *new*
page whose only tie to the origin is a journal `fork` entry. shard-wiki wants
**identity ≠ placement** (the "same" page across shards under a stable identity, T16) —
so we treat fedwiki's journal-linked forks as *provenance edges*, and layer our own
cross-shard identity over them rather than adopting slug-as-identity.
- **No query / no typed-record model.** Fedwiki is paragraphs+plugins, not a typed DB
(contrast Notion/Wikibase). Fine — it sits at the *coordination* end, not the structure
end. We don't ask fedwiki to provide query; the neighborhood search is the model for
*federated* search across shards (T-federation), not in-shard query.
- **Browser-assembles-union.** Fedwiki pushes union assembly to the client. shard-wiki
assembles server/orchestrator-side. Adopt the *model* (union from sovereign sources +
provenance), not the client-only locus.
### What to keep
1. **Journal = append-only semantic-op log with provenance entries**, story = derived
replay view. This is the concrete shape for our coordination journal (T13).
2. **Fork-with-source-attribution** as the overlay/adopt primitive across shards.
3. **Neighborhood** as the model for a *dynamic, link-and-fork-discovered* federated set +
search, with **roster** as the curated/explicit variant.
4. **Chorus of forks** — represent divergent versions across shards as co-equal, linked by
provenance, with reconciliation as an explicit human/policy step (mechanism over policy).
---
## 7. UC seeds
| # | Seed | Disposition |
|---|------|-------------|
| UC-70 | Attach a Federated Wiki site as a shard via its **page JSON + CORS** (REST/file-store hybrid); project pages, fork to overlay | **new** |
| UC-71 | Adopt a **per-page append-only semantic-action journal with provenance entries** (fork=source site) as the coordination-journal model — replay to materialize, compare to locate divergence | **new** |
| UC-72 | **Fork-with-site-provenance federation across a neighborhood** of peer shards — assemble a union from links/forks, search across it, preserve the chorus without forcing a canonical | **new** |
| — | fork-with-provenance as overlay/adopt | enrich **UC-26** (fork) |
| — | carry-forward of forked content + upstream re-fork | enrich **UC-28** (carry-forward) |
| — | happenings = time-bounded collaboration leaving durable forks | enrich **UC-30** (time-bounded space) |
| — | union/chorus of co-equal versions, provenance-linked | enrich **UC-05 / UC-27** |
## 8. Architecture notes for SHARD-WP-0002
- **T1T5 (federation):** fedwiki is the reference design. The **journal** (append-only,
semantic ops, fork-provenance) is the concrete coordination-journal shape; **neighborhood
+ roster** is the discovery/membership model (dynamic vs curated); **fork** is the
overlay/adopt op. Model the union as an assembly over sovereign sources with provenance
edges, reconciliation left to policy.
- **T11 (capability/write-granularity):** add **story-item / paragraph** as a named
write-granularity tier between whole-file and block/character.
- **T13 (history portability / merge model):** record fedwiki's **journal-replay op-log**
as a *third merge model* beside git 3-way and CRDT auto-merge — a **coarse semantic
op-log applied manually via fork**. A shard whose history *is* such a journal can supply
our coordination journal almost directly (vs git-commit import or CRDT-update import).
- **T16 (identity ≠ placement):** fedwiki's `fork` journal entries are **provenance edges**
between same-named pages on different sites — exactly the cross-shard "same page,
different placement" relation we must model. Use them as edges; keep our own identity
layer above slug.
## 9. Open questions
1. Should shard-wiki's coordination journal adopt fedwiki's **exact action vocabulary**
(create/add/edit/move/remove/fork) at the page-item level, or a more granular/abstract
op set that other shards can also emit?
2. Is **neighborhood** (dynamic, link/fork-discovered) a first-class membership mode for an
information space, or only a *view* over an explicitly-configured shard set (roster)?
3. How do we reconcile fedwiki's **slug-as-identity + fork-DAG** with our intended
**stable cross-shard identity** (T16) — promote fork edges into the identity graph, or
keep them as provenance-only annotations?
4. Does the **chorus / no-canonical** stance compose with shards that *do* assert a
canonical (Notion, an upstream git main)? (policy-selectable canonical over a
mechanism that permits chorus.)
## 10. Sources
- Smallest Federated Wiki wiki: **Story JSON**, **Federation Details**
github.com/WardCunningham/Smallest-Federated-Wiki/wiki
- JSON Schema notes — song.fed.wiki.org/json-schema.html
- "Smallest Federated Wiki" — home.c2.com/smallest-federated-wiki.html
- Federated Wiki — federated.wiki (Visualizing Page History)
- Mike Caulfield, "The OER Case for Federated Wiki" — hapgood.us (2015)
- Jon Udell, "A federated Wikipedia" — blog.jonudell.net (2015)
- Wikipedia: *Federated Wiki*; IndieWeb: *Smallest Federated Wiki*
- fedwiki/wiki-plugin-transport (plugin/transport reference)
- prior: `research/260608-federation-concepts/` §3
## 11. Traceability
New UCs **UC-70UC-72** carry the marker **⊞** in the wikiengines column of
`spec/UseCaseCatalog.md` (true lineage = this dive; placed in the nearest existing column).
Enriched: UC-26, UC-28, UC-30, UC-05, UC-27. Architecture cross-refs: SHARD-WP-0002
T1T5, T11, T13, T16.

View File

@@ -0,0 +1,17 @@
# 260614 — git-forge wikis deep dive (Gitea · GitLab · GitHub)
Deep dive on the **git-forge-hosted Markdown wikis** as one grouped memo: each is a
**dedicated git repository of Markdown** exposed through a forge, attachable by **cloning
the wiki repo directly** *or* (where offered) through the **forge's wiki API**. INTENT names
**Gitea wikis** explicitly as a shard participant — this dive confirms the **git-native
file-store** as a first-class, and the *simplest*, backend.
- `findings.md` — the three forges compared, the `.wiki.git` model, API matrix, capability
profile, INTENT mapping, UC seeds (UC-76/77), architecture notes for SHARD-WP-0002, open
questions (resolves the UC-68 source-of-truth question for this case), sources,
traceability.
Catalog yield: UC-76 (attach a forge wiki by cloning its dedicated `.wiki.git` — git **is**
the native store and the coordination journal), UC-77 (attach/write via the forge wiki API
where git-clone is unavailable/undesired — capability varies by forge). Enriched
UC-40/02/68/38. Feeds SHARD-WP-0002 T14 (attachment binding).

View File

@@ -0,0 +1,173 @@
# git-forge wikis (Gitea · GitLab · GitHub) — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T5 · **Subject:** the Markdown wikis
hosted by the three major git forges — **Gitea**, **GitLab**, **GitHub** — treated as one
family because they share one architecture: *a wiki is a separate git repo of Markdown.*
## Why this dive
INTENT names **Gitea wikis** as a shard participant, and the whole project is "a **Git-based
Markdown** wiki orchestrator." The forge wikis are therefore the **least exotic, highest-
fit** backend in the entire study: the page store is *literally a git repository of Markdown
files*. After fourteen dives into DBs, CRDTs, graphs and SaaS, this one confirms the
**home case** — and sharpens it by contrasting *git-IS-the-store* (forge wikis) against
*git-is-a-mirror* (Wiki.js, UC-68).
## 1. The shared architecture — a wiki is a `.wiki.git` repo
All three forges implement a project/repo wiki as a **second, dedicated git repository**
alongside the code repo, addressable as `<repo>.wiki.git`:
- `git@host:owner/project.wiki.git` (GitLab), `…/owner/repo.wiki.git` (Gitea/GitHub).
- **Pages are Markdown files** (`Home.md`, `Some-Page.md`), one file per page; the **page
title ↔ filename** (spaces ↔ hyphens by convention). Other markups are accepted
(AsciiDoc, Textile, reStructuredText, Org) — GitHub/Gitea via **Gollum** (the Ruby
git-backed wiki library), GitLab via its own renderer.
- **History is git history** — every page edit (web or pushed) is a git commit with
author/timestamp/message. *The wiki's revision history is a real git log.*
- **Special pages** by convention: `_Sidebar`, `_Footer`, `_Header` (GitHub/Gitea),
`_sidebar` (GitLab) — engine-rendered chrome stored as ordinary files.
- **Subdirectories / nested pages**: GitLab and Gitea support directory structure; GitHub
wikis are historically flat (Gollum supports paths but the GitHub UI is shallow).
The decisive property: **you can `git clone` the wiki repo, edit files, commit, and push**,
and the forge UI reflects it — *and vice versa*. Git is **a** (often **the**) first-class
write path. This is exactly shard-wiki's native medium with no impedance.
## 2. Where they differ — the API matrix
| | git clone/push of `.wiki.git` | wiki content **API** | nested dirs | markups |
|--|--|--|--|--|
| **Gitea** | ✅ yes | ✅ **REST wiki endpoints** (list/get/create/edit/delete pages) | ✅ | Markdown (+Gollum-style) |
| **GitLab** | ✅ yes | ✅ **REST Wikis API** (project & group wikis) | ✅ | Markdown/AsciiDoc/RDoc/Org |
| **GitHub** | ✅ yes | ❌ **no wiki REST API** — wiki is **git-only** (Gollum) | ⚠️ flat UI | Markdown + Gollum markups |
The key asymmetry: **GitHub exposes wiki content *only* through git** (the REST/GraphQL API
covers issues/PRs/code but **not** wiki pages); **GitLab and Gitea offer both** a wiki API
*and* git access. So the **git-clone path is the universal one** (works for all three); the
API path is an *optional, capability-varying* alternative.
## 3. git-IS-the-store vs git-is-a-mirror (the UC-68 contrast)
Wiki.js (UC-68) keeps a **DB as canonical** and *maintains a git mirror* — so writing by
commit risks **racing the engine's DB↔git sync** (catalog open-Q22). Forge wikis are the
opposite: **the git repo IS the canonical store**; there is *no* separate DB of record for
wiki content. Therefore:
- **The source-of-truth question (Q22) is resolved for this case:** the `.wiki.git` repo is
authoritative. shard-wiki can **write by commit/push directly** with no engine to race —
the forge merely *renders* what git holds.
- The forge **API** (GitLab/Gitea), where present, is a *convenience over the same git
repo*, not a competing store — so API-write and git-write converge on one history.
This makes forge wikis the **cleanest possible write-through file-store shard**: clone =
projection/mirror, commit = overlay-applied/write, git log = the coordination journal *as
is*.
## 4. Capability profile
| Dimension (synthesis spectrum) | Gitea / GitLab / GitHub wiki |
|--------------------------------|------------------------------|
| Attachment mode | **file-store (native: git clone)** + optional **external-API** (GitLab/Gitea wiki REST) |
| Addressing granularity | **page = file**; sub-page = path (GitLab/Gitea) |
| Content identity | path/filename within the wiki repo (title-derived) |
| Identity vs placement | placement-bound (path = identity), like a plain git repo |
| Structure | flat or directory tree of Markdown files; `_Sidebar`/`_Footer` chrome |
| History | **native git history** (real commits, authors, messages) |
| Merge model | **git** (3-way merge, branches) — though wiki repos are usually single-branch |
| Native query | none (it's files); forge full-text search over the wiki |
| Translation | **Markdown-native** (+ AsciiDoc/Org via renderer) — minimal/no translation needed |
| Attachment/write granularity | **file (page)** per commit |
| Operational envelope | ordinary git + forge; clone is cheap; API rate limits apply to API path |
| Access grant | **forge repo permissions** (delegated auth; per-repo/role ACL) |
| Content opacity | transparent Markdown in git |
| Provenance | git author/committer/timestamp per commit — native |
## 5. INTENT mapping
### Reinforcements (this is the home case)
- **Git-based Markdown orchestrator** (INTENT core): forge wikis *are* git repos of
Markdown. The **wiki page model** (Markdown-first, path-addressed, git-versioned) maps 1:1
— minimal adapter, maximal fit.
- **Coordination journal = git** (INTENT): the wiki repo's **git log is already the
coordination journal** — no synthesis needed; adopt it directly.
- **Overlay before mutation**: overlays are **branches/commits** on the cloned wiki repo;
applying = push (or open an MR/PR where the forge supports wiki MRs — GitLab does not for
wikis, so push-to-branch + manual is the path).
- **Graceful degradation**: even GitHub (no wiki API) is fully usable via git-clone — the
*universal* path means a limited forge is still a first-class read/write shard.
- **No silent remote mutation**: writes are explicit git pushes (or explicit API calls)
under the user's forge credentials and repo permissions.
### Divergences (boundaries / notes — minor)
- **Capability varies by forge**: GitHub = git-only (no content API); GitLab/Gitea = git +
API. The adapter must **model the API as an optional capability**, defaulting to the
universal git path (T11/T14). Not a bug — exactly the capability-awareness INTENT mandates.
- **Wiki repos rarely use branches/MRs for review**: forge wikis usually edit a single
branch directly; the rich PR-review flow is on the *code* repo, not the wiki. So
"overlay → review → merge" needs shard-wiki to provide the review layer, not the forge.
- **Identity = path** (like any git repo) — cross-shard identity (T16) is layered above, as
for plain git/`wiki/` subdir shards.
### What to keep
1. **git-clone as the universal, canonical file-store attach** for forge wikis — Markdown +
git history directly as page model + coordination journal (UC-76). The reference
easy-case backend.
2. **Forge wiki API as an optional capability** (GitLab/Gitea), with **git-only fallback**
(GitHub) — capability-aware binding (UC-77).
3. **git-IS-store ⇒ write-by-commit is safe** (no engine race) — record this as the
resolution of the Wiki.js mirror dilemma (Q22) for forge wikis.
## 6. UC seeds
| # | Seed | Disposition |
|---|------|-------------|
| UC-76 | Attach a **git-forge wiki** by **cloning its dedicated `.wiki.git`** — git is the native store; Markdown files = pages, git log = coordination journal; commit/push = write (no engine to race) | **new** |
| UC-77 | Attach/write a forge wiki via the **forge's wiki API** (GitLab/Gitea REST) where git-clone is unavailable or API-write is preferred; **git-only fallback** for GitHub — capability varies by forge | **new** |
| — | git-native file-store as the *canonical store* (not mirror) | enrich **UC-40** |
| — | dual-path attach (git clone vs forge API) | enrich **UC-02** |
| — | git-IS-store vs engine-maintained mirror (resolves Q22) | enrich **UC-68** |
| — | forge as an API host for the wiki resource | enrich **UC-38** |
## 7. Architecture notes for SHARD-WP-0002
- **T14 (adapter binding / attach path):** forge wikis are the canonical **file-store
attach** — bind to the `.wiki.git` clone as the universal path; model the **wiki API as an
optional, forge-specific capability** (present: GitLab, Gitea; absent: GitHub). One shard,
two possible bindings converging on the same git history.
- **T11 (capability model):** "has-content-API" is a **per-forge capability flag**; git
clone/push is the baseline every forge satisfies. Minimal adapter profile — near the
Oddmuse-simple end but Markdown-native and git-versioned.
- **Coordination journal:** adopt the wiki repo's **git log directly** — the one backend
where INTENT's git-backed journal needs *zero* synthesis.
- **Resolves Q22 (UC-68):** because git **is** the store (not a mirror), **write-by-commit
is safe** — no engine DB↔git sync to race. Record the distinction *engine-mirror*
(Wiki.js: DB canonical, careful) vs *git-canonical* (forge wikis: commit freely).
## 8. Open questions
1. For overlay → **review** → apply, does shard-wiki supply the review layer over a forge
wiki (which lacks wiki-MRs), e.g. via a branch + its own diff/approve, or push directly?
2. When a forge offers **both** git and a wiki API (GitLab/Gitea), which does the adapter
prefer by default — git (universal, full history) with API as a fallback for hosts where
clone is disabled? (cf. UC-43 backend-swap under stable binding.)
3. Should the **code-repo `wiki/` subdir** shard and the **forge wiki repo** shard share one
adapter (both git+Markdown) with a "which repo / which path" parameter, or stay distinct?
## 9. Sources
- GitLab Docs — *Wiki* (separate git repo; web/git/API; `.wiki.git`) — docs.gitlab.com
- Gitea — wiki via git clone + repository **wiki API**; forum/issue threads on
`.wiki.git` clone (go-gitea/gitea #1426, #15420) — gitea.com / github.com/go-gitea
- GitHub — wiki = Gollum git repo (`<repo>.wiki.git`), no wiki REST API — docs.github.com
- Gollum (git-based wiki library) — github.com/gollum/gollum
- prior: `research/260614-wikijs-deep-dive/` (engine-maintained mirror contrast, UC-68)
## 10. Traceability
New UCs **UC-76UC-77** carry the marker **⎇** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-40, UC-02, UC-68, UC-38. Architecture cross-refs:
SHARD-WP-0002 T14, T11; coordination-journal-from-git; resolves catalog open-Q22.

View File

@@ -0,0 +1,33 @@
# 260614 — Glamorous Toolkit (moldable development) deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T7**
## What this is
A deep dive into **Glamorous Toolkit** (GT, on Pharo): **moldable development** — cheap,
custom, **domain-specific views** (`gtView` methods) so any object explains itself through an
**open set of co-equal projections, none canonical** — plus **Lepiter**, GT's live notebook/
knowledge base (git-versionable JSON page files with live, inspectable code results).
## Why it matters
- Strongest prior art for **moldable, multi-view projection**: projection is not *a* view
but an **open, type-keyed set of co-equal, possibly-computed views, none privileged**
refines SHARD-WP-0002 **T16** and unifies replication-/derivation-/dimensional-/query-
projection under "many co-equal views."
- Generalizes ZigZag dimensional views (UC-47/48) and query/computed views (UC-54) into a
**pluggable view registry** keyed by content type (answers UC-55's open question on a
content-type registry).
- Reinforces **files-canonical, liveness-above, degrade-to-snapshot** (Lepiter files vs the
Pharo image; same boundary as Jupyter UC-84 / Squeak T6).
## Yield
- **No new UC** (design prior art, not a candidate shard — like the UseModWiki lineage dive).
- Enrich **UC-47, UC-48, UC-54**; links **UC-55, UC-83, UC-84, UC-79**.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Moldable development & `gtView`, the Moldable Inspector, Lepiter, relation to ZigZag/query/derivation projection, INTENT mapping, UC disposition (enrichment-only), architecture notes, open questions |

View File

@@ -0,0 +1,161 @@
# Glamorous Toolkit — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T7 · **Subject:** Glamorous Toolkit (GT) on
Pharo — moldable development, custom views/inspectors, the live notebook (Lepiter).
## Why this dive
T1/T3 gave us **derivation-projection** (one source → rendered/computed forms). GT comes at
projection from the *other* side: **many co-equal, domain-specific views over the same live
content**, where the *environment molds itself to the knowledge* rather than forcing the
knowledge into one fixed rendering. This is the strongest prior art for **moldable,
multi-view projection** and a striking parallel to ZigZag's dimensional model (UC-47/48) and
to query/computed views (UC-54). It is *design* prior art — GT is not a candidate shard —
so the yield is **enrichment + projection design notes**, not a new shard UC.
## 1. Moldable development
GT's thesis: **systems should be explainable through custom tools that are cheap to build.**
Instead of a single generic object inspector/renderer, a developer adds **small,
domain-specific views** to a class so any instance explains itself in the terms that matter:
- **`gtView`-annotated methods** — a class declares extra inspector views by writing methods
tagged `<gtView>` that return a view (tree, list, table, chart, source, raw, a custom
diagram…). Each is a **co-equal projection of the same object**; none is privileged.
- The **Moldable Inspector** shows these views as **switchable tabs** over one object, and
lets you **dive** into sub-objects (each with its own custom views) — navigation *is*
moving across projections.
- Views are **cheap and local**: a view is just a method, versioned with the code, added
incrementally as understanding grows. The environment **adapts to the domain**.
The key abstraction for us: **the same underlying content carries an open, extensible set of
views, selected at inspection time, none canonical.**
## 2. Lepiter — the live notebook / knowledge base
GT ships **Lepiter**, a notebook/knowledge base where pages mix prose, **live code
snippets** (evaluated in-image, results inspectable with the moldable views above), and
links. Notebook pages are stored as **JSON "database" files** on disk (a Lepiter DB =
directory of page files), so the knowledge base is **file-backed and git-versionable** while
remaining live in the image.
This is the literate/notebook pattern (T1/T3) fused with moldable views: a snippet's result
is not a static captured output but a **live object you can open into any of its views**
the *anti-snapshot*. (Boundary: that liveness is exactly what shard-wiki must degrade to a
snapshot when the image isn't present — see §4.)
## 3. Relationship to ZigZag, query-views, and the projection model
- **vs ZigZag (UC-47/48):** ZigZag gives **dimensional** views — the *same cells* seen along
different orthogonal axes. GT gives **moldable** views — the *same object* seen through
different *purpose-built lenses*. Both reject a single privileged rendering; both make
**multi-view, none-canonical** the norm. GT generalizes the idea from fixed dimensions to
an **open, code-defined view set**.
- **vs query/computed views (UC-54):** a `gtView` is a **computed projection** (it runs code
to build the view) — like a query-defined page, but keyed to a **content type / domain**
rather than a stored query. Strengthens "a view can be computed, not stored."
- **vs derivation-projection (T1/UC-83):** GT confirms projections are **plural and
co-equal**; UC-83/UC-84 had *few* well-known derivations (docs/code/outputs), GT has an
**open registry** of them keyed by type.
So GT's contribution to the contract is: **projection is not one view but an open set of
co-equal, type-keyed, possibly-computed views, none canonical** — a *moldable projection
registry*. That is a refinement of T16's projection model, not a new shard.
## 4. INTENT mapping
### Reinforcements
- **Union without erasure / no privileged rendering.** GT's "many co-equal views, none
canonical" is the same ethos as showing provenance/freshness without hiding any: a page's
presentation is **plural**, and shard-wiki should be able to offer **multiple co-equal
projections of the same content** (raw, rendered, structured, domain-specific) rather than
one flattened view.
- **Computed views (UC-54) keyed by content type.** A moldable view = a **computed
projection registered against a content type** — directly supports a *pluggable view/
projection registry* in the contract (the answer shape for UC-55's "pluggable content-type
registry" open question).
- **Files-canonical, live-on-top.** Lepiter stores pages as **git-versionable JSON files**
while being live in the image — reinforcing "the durable artifact is files; liveness is a
layer above," consistent with shard-wiki's git-canonical stance.
- **Mechanism over policy.** Which views to show, and whether to compute or snapshot them,
stay configurable; GT provides the *mechanism* (open view set), not a fixed presentation.
### Divergences / boundaries
- **The image is not a store (shared boundary with T6 Squeak).** GT's *liveness* lives in a
Pharo image; shard-wiki must not treat the image as a shard. Attach the **Lepiter DB files**
(git-versionable) as the durable content; treat live/computed views as **derivation-
projections that degrade to captured snapshots** when no image/kernel is present (same rule
as Jupyter UC-84).
- **Not a view engine.** shard-wiki *models* "this content type has these co-equal views and
one may be canonical-for-display"; it does not implement GT's rendering. Domain-specific
view code stays with the source/adapter, surfaced as a capability.
### What to keep
1. **Moldable projection = an open, type-keyed set of co-equal, possibly-computed views,
none canonical** — refine T16's projection model and the UC-47/48/54 cluster toward a
**pluggable view/projection registry**.
2. **Lepiter** = literate/notebook (T1/T3) with **live results** stored as git-versionable
files — another "files-canonical, liveness-above, degrade-to-snapshot" instance.
3. **Navigation = moving across projections** (dive into sub-objects) — a navigation idea
beside dimensional movement (UC-47/48) for the derived-views thread.
## 5. UC disposition (enrichment-only — no new shard UC)
| Mechanism (findings §) | Catalog UC |
|------------------------|------------|
| Many co-equal, domain-specific views over the same content; none canonical (§1, §3) | UC-47 / UC-48 (enriched) |
| `gtView` = a **computed projection registered against a content type** (§1, §3) | UC-54 (enriched) |
| Open, pluggable view set keyed by type = the shape of a content-type/view registry (§3) | links UC-55 (open-Q #10) |
| Lepiter live-snippet results = live objects → degrade to snapshot when no image (§2, §4) | links UC-84, UC-83 |
| Lepiter pages = git-versionable JSON files; image is not the store (§2, §4) | links UC-79 (files-canonical) |
| Dive-into-sub-object navigation = moving across projections (§1) | links UC-17UC-20 (derived views) |
GT is **design prior art, not a candidate shard**, so it yields **no new UC** (like the
UseModWiki lineage dive). Its value is sharpening the **projection model** for SHARD-WP-0002
T16: projection is an **open set of co-equal, type-keyed, possibly-computed views, none
privileged** — a moldable view registry.
## 6. Architecture notes for SHARD-WP-0002
- **T16 (projection):** generalize projection from "a view" to a **moldable view registry**
an open, extensible set of **co-equal, type-keyed, possibly-computed** projections of the
same content, none canonical (display-canonical is a policy choice, not a fact). This
unifies replication-projection (UC-53/57), derivation-projection (UC-83/84), dimensional
views (UC-47/48), and query/computed views (UC-54) under one "many co-equal views" model.
- **T12 (page model):** a content type may **carry/declare its own views** (a capability);
the page model should allow attaching view definitions to a type (the registry's entries),
answering UC-55's "pluggable content-type registry" question.
- **Boundary (T14):** attach the **Lepiter DB files** (durable, git-versionable), never the
Pharo image; live/computed views are derivation-projections degrading to snapshots — same
rule as Jupyter (no kernel/image host).
## 7. Open questions
1. Does shard-wiki expose a **view/projection registry** as a first-class public concept
(content type → its co-equal views), or keep "moldable" as an adapter-internal idea?
2. When multiple co-equal views exist, is **"canonical-for-display"** a per-shard policy, a
user preference, or unset (always show the chooser)? (Mechanism-over-policy says
configurable.)
3. How does a **computed view** (UC-54/`gtView`) declare its **freshness/provenance** so the
union doesn't present a stale computed projection as current? (Ties UC-84's snapshot
marking.)
## 8. Sources
- Glamorous Toolkit docs (`gtoolkit.com`): moldable development, the Moldable Inspector,
`gtView` methods, Lepiter knowledge base; Feenk essays on moldable development.
- Pharo (substrate — see T8 dive): live image, reflective environment.
- prior: `research/260614-zigzag-deep-dive/` (dimensional multi-view UC-47/48);
`research/260614-jupyter-deep-dive/` (live vs snapshot, UC-84);
`research/260614-literate-programming-deep-dive/` (derivation-projection, UC-83).
## 9. Traceability
**No new UC** (GT is design prior art, not a candidate shard). Enriched: UC-47, UC-48,
UC-54; links UC-55 (content-type/view registry), UC-83/UC-84 (live→snapshot), UC-79
(files-canonical), UC-17UC-20 (derived-view navigation). Architecture cross-refs:
SHARD-WP-0002 T16 (moldable view registry: open set of co-equal type-keyed computed views),
T12 (content type declares its views), T14 (attach Lepiter files, not the image).

View File

@@ -0,0 +1,15 @@
# 260614 — ikiwiki deep dive
Deep dive on **ikiwiki** (Joey Hess): a **wiki compiler** — it compiles a **VCS-backed**
(usually **git**) tree of Markdown source into **static HTML**, supports **distributed**
operation (clone/edit/push between wiki instances, change **pings**), and treats web edits as
commits to the same repo.
- `findings.md` — the compiler model, git-distributed federation + pinger, static output,
capability profile, INTENT mapping, UC seed (UC-79), architecture notes for SHARD-WP-0002,
open questions, sources, traceability.
Catalog yield: UC-79 (attach a git-backed **compile-to-static** wiki — git Markdown source
is the shard, compiled static HTML is a derived publish/projection; participate in
git-distributed clone federation with change-pings). Enriched UC-31/56/37/33. Feeds
SHARD-WP-0002 T4 (federation), T6 (publish/projection).

View File

@@ -0,0 +1,153 @@
# ikiwiki — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T4 · **Subject:** ikiwiki, Joey Hess's
VCS-backed wiki compiler.
## Why this dive
The forge-wiki dive (T5) established *git-IS-the-store* for hosted Markdown wikis. ikiwiki
takes the same git-canonical source but adds two ideas shard-wiki cares about directly:
**compile-to-static** (the wiki is *built*, not served from a DB) and **git-distributed
federation** (wiki instances clone, pull, push, and **ping** each other). It is referenced
in `research/260608-federation-concepts/`; here we go into the model.
## 1. The wiki-compiler model
ikiwiki is fundamentally a **compiler**: input is a directory of **source pages** (Markdown
by default; also other formats) held in a **version-control repo**; output is a tree of
**static HTML**. A rebuild is triggered by a **VCS post-commit/post-update hook**, so:
- **The VCS repo is canonical**; the HTML is *derived build output* (regenerable, disposable).
- **Web edits are commits.** The CGI edit interface writes the change *into the repo* (a
commit) and triggers a rebuild — so browser edits and `git push` edits converge on one
history. (Same convergence as forge wikis, but here the canonical store is *your* git repo,
not a forge's.)
- **VCS-agnostic**: git is usual, but svn/bzr/mercurial/darcs are supported via a VCS
plugin layer — an early "pluggable backend behind a stable interface" (adapter-contract
echo).
- **Plugins** (Perl) provide directives, feeds, auth (`openid`), and the federation hooks
below.
## 2. Git-distributed federation + the pinger
Because the source is an ordinary VCS repo, ikiwiki instances federate the way *git* does:
- **Clone-and-diverge**: you can `git clone` a wiki, edit offline, and `push`/`pull` between
instances — **multiple wiki clones that reconcile via git merge**. A wiki is a branch-space
(UC-33).
- **`pinger` / `pingee` plugins**: an instance can send an **XML-RPC ping** to another
ikiwiki when it changes, prompting the other to **pull and rebuild** — a lightweight
**subscribe/notify** primitive over the git-distributed mesh (UC-31).
- **`aggregate` plugin**: pulls external **RSS/Atom feeds** into the wiki as pages — an
inbound projection of remote content.
So ikiwiki is *federation by git plus a ping* — distinct from fedwiki's fork/journal
(UC-72) and from Wikibase's query-time `SERVICE` (UC-74): a **third federation flavor**,
*VCS-replication federation with change notification*.
## 3. Static output as a publish/projection target
The compiled static HTML is a **read-only, regenerable projection** of the source:
- It is a natural **outbound publish target** (UC-56): render the union (or a shard) to a
static site for hosting/backup, no server needed.
- It is also the **read-only backup** end (UC-37): a static snapshot that survives the engine.
The key shard-wiki framing: **source (git Markdown) is the attachable shard; static HTML is a
derived projection** — never confuse the build output for the canonical content.
## 4. Capability profile
| Dimension (synthesis spectrum) | ikiwiki |
|--------------------------------|---------|
| Attachment mode | **file-store** (VCS-backed git Markdown source) |
| Addressing granularity | page = source file; path = identity |
| Content identity | path/filename (placement-bound) |
| Structure | directory tree of Markdown source + directives |
| History | **native VCS (git) history** |
| Merge model | **git** (clone/pull/push/merge across instances) |
| Native query | none; directives + plugins compute derived pages at build |
| Translation | Markdown source → static HTML (build-time render) |
| Write granularity | **file (page)** per commit |
| Operational envelope | a compiler + VCS hook; static hosting for output |
| Access grant | VCS/file perms; `openid` for web edits |
| Content opacity | transparent Markdown |
| Provenance | git author/timestamp; aggregated feeds carry source |
| Federation | **git replication + XML-RPC ping**; RSS aggregation |
## 5. INTENT mapping
### Reinforcements
- **Git-canonical Markdown source** = the home case (shared with forge wikis UC-76/40):
attach the source repo, adopt its git log as the journal, write by commit.
- **Coordination layer is git** (INTENT): ikiwiki's whole federation *is* git replication +
a ping — the most literal realization of "Git-addressable coordination layer."
- **Projection vs canonical**: compile-to-static cleanly separates **canonical source** from
**derived output** — exactly shard-wiki's projection principle (static HTML = a lazy/cache
projection that is regenerable, never the source of truth).
- **Graceful degradation / publish**: static output is the trivial read-only backup and
outbound publish target (UC-37/UC-56).
- **Subscribe/notify mechanism, not policy**: the pinger is a *mechanism* (notify a peer to
pull); *which* peers, when, and conflict policy stay configurable.
### Divergences (boundaries / notes)
- ikiwiki is mostly a **reinforcement** of git-canonical-Markdown (UC-76) — its *new*
contributions are (a) **compile-to-static** as a distinct projection/publish direction and
(b) the **git-distributed-clone + ping** federation flavor. The static output is *not* a
shard to attach (it's derived); attach the **source repo**.
- VCS-agnostic backend layer is an interesting adapter-contract echo, but for shard-wiki the
git case dominates.
### What to keep
1. **Source-repo-as-shard, static-output-as-derived-projection** — never attach the build
output as canonical (UC-79; relates UC-37/UC-56).
2. **Git-replication + change-ping** as a named federation flavor beside fork/journal
(fedwiki) and query-SERVICE (Wikibase) — a **subscribe/notify over a git mesh** (UC-31/UC-33).
3. **Inbound feed aggregation** (RSS/Atom → pages) as an inbound projection pattern.
## 6. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-79 | Attach a git-backed **compile-to-static** wiki — the **git Markdown source is the shard**, compiled **static HTML is a derived publish/projection**; participate in **git-distributed clone federation** with change-**pings** | **new** |
| — | XML-RPC pinger = subscribe/notify over a git mesh | enrich **UC-31** |
| — | render union/shard to a static site (publish) | enrich **UC-56** |
| — | static HTML as read-only regenerable backup | enrich **UC-37** |
| — | wiki as a git branch-space; clones reconcile via merge | enrich **UC-33** |
## 7. Architecture notes for SHARD-WP-0002
- **T4 (federation):** record **git-replication + change-ping** as a federation flavor — peers
hold git clones, reconcile via merge, and notify with a ping to pull/rebuild. Distinct from
fedwiki fork/journal and Wikibase `SERVICE`; complements the "Git-addressable coordination
layer" mandate most literally.
- **T6 (publish/projection):** **compile-to-static** is the canonical *outbound* projection —
source (git) is canonical, static HTML is a regenerable derived view (publish/backup
target). Reinforces projection-vs-canonical separation.
- **Attach binding (T14):** attach the **source repo** (git Markdown), not the build output;
shares the forge-wiki / `wiki/`-subdir git+Markdown adapter (parameterized by repo/path).
## 8. Open questions
1. Is the **pinger** (notify-peer-to-pull) modeled as shard-wiki's own subscribe/notify
primitive, or only recognized when bridging two ikiwiki instances?
2. Does shard-wiki ever *drive* a compile-to-static **publish** of the union (act as the
ikiwiki-like compiler), or only attach existing ikiwiki source repos? (UC-56 scope.)
3. Is **feed aggregation** (RSS→pages) an inbound projection mode shard-wiki offers generally
(a feed-shard), or an ikiwiki-internal feature?
## 9. Sources
- ikiwiki.info — *ikiwiki* overview, *rcs* (VCS backends), *plugins* (`pinger`/`pingee`,
`aggregate`, `openid`), *setup* / post-commit hooks
- prior: `research/260608-federation-concepts/` (ikiwiki reference);
`research/260614-forge-wikis-deep-dive/` (git-canonical Markdown contrast)
## 10. Traceability
New UC **UC-79** carries the marker **⊟** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-31, UC-56, UC-37, UC-33. Architecture cross-refs:
SHARD-WP-0002 T4 (git-replication+ping federation), T6 (compile-to-static publish), T14.

View File

@@ -0,0 +1,49 @@
# 260614 — Joplin deep dive (SQLite-local, Markdown-on-sync, interchange-format attach)
Date: 2026-06-14
## What this is
A focused study of **Joplin** — the open-source Markdown note app — read through
shard-wiki's lens. Joplin looks like "open-source Obsidian" but is a genuinely different
point in the space: **content is Markdown, the local store is SQLite, and the *sync
representation* is a folder of per-item Markdown+metadata files on a backend you choose**
(filesystem, WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). It
also ships **end-to-end encryption** and is itself a **sync layer over heterogeneous
storage**.
Distinctive material:
- **Architecture** — SQLite local store (documented schema); Markdown content;
notebooks → notes → resources + tags + to-dos; 32-char item IDs; `:/<id>` note links
(rename-stable, page-level)
- **Sync** — serialized to plain-text items on a chosen target; tombstones; **conflict
notes** (keep-both) → enables the **"attach the sync mirror, not the app"** pattern on
WebDAV/Nextcloud/S3
- **E2EE** — items encrypted before leaving the device → **content opacity** (ciphertext
at rest)
- **Extension surfaces** — a **plugin API** (`joplin.data`/`workspace`/`contentScripts`/
views) *and* a **local Data API** (REST on `localhost:41184`, token) used by the Web
Clipper; plus JEX/Markdown/RAW export
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, sync/interchange attach, E2EE/content-opacity, plugin + Data API surfaces, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md`
(UC-60 attach a tool's documented sync/interchange representation on third-party storage
without the app; UC-61 attach an encrypted-at-rest shard with content opacity);
UC-31/36/38/40/51/55 enriched. Logged for `SHARD-WP-0002` (T11, T14): the
interchange/sync-representation attach surface, **content opacity** as a proposed twelfth
capability spectrum, a local-REST attach sub-mode, and format-aware file-store profiles.
**Key takeaways recorded:** the best attach surface is often a tool's **interchange/sync
representation** (offline, app-independent — Joplin items on Nextcloud/WebDAV/S3, the
INTENT-named backends), not its live store or API; **encryption-at-rest** demands a
content-opacity capability so encrypted shards degrade to backup/structure-shell; and
Joplin is the **file-sync-daemon boundary case** — shard-wiki attaches its mirror as
pages and never re-syncs (not-a-sync-daemon).
</content>

View File

@@ -0,0 +1,269 @@
# Findings — Joplin: SQLite locally, Markdown-on-sync, and the interchange-format attach
Date: 2026-06-14
Source kind: **modern shipped product** — an open-source Markdown note app; a *candidate
shard* whose distinctive trait is a **documented file-based sync/interchange format on
third-party storage** plus optional **end-to-end encryption**
Lens: shard-wiki — attachment surfaces, the "attach the sync mirror not the app"
pattern, content opacity (encryption), note-level stable IDs, and the file-sync-daemon
boundary
> Why Joplin earns a dive after Obsidian/Roam/Notion. It looks like "Obsidian but
> open-source," yet its architecture is a genuinely different point in the space:
> **content is Markdown but the local store is SQLite, and the *sync representation* is
> a folder of per-item Markdown+metadata files on a backend you choose** (filesystem,
> WebDAV, **Nextcloud**, Dropbox, OneDrive, **S3**, Joplin Server/Cloud). That sync
> mirror — not the app, not the SQLite DB — is the interesting attach surface, and it
> lands squarely on the **WebDAV/Nextcloud backends INTENT already names**. Joplin also
> ships **E2EE**, forcing the question of a shard whose content is *opaque ciphertext*,
> and it is itself a **sync layer over heterogeneous storage** — the very "file-sync
> daemon" shard-wiki says it is *not*, making Joplin a sharp boundary case.
Contrast set: Obsidian (file-over-app: native store *is* files), Roam (client DB,
in-app API only), Notion (hosted DB, external API only), TWiki/Foswiki (file+RCS native
store). Joplin is none of these exactly — **DB-local, files-on-sync** — which is the
whole point.
---
## 1. Core architecture — SQLite local, Markdown content
- **Local store:** a **SQLite database** (`~/.config/joplin-desktop/database.sqlite`),
schema documented. So, unlike Obsidian, the *native* local store is **not** plain
files — you cannot point shard-wiki at a folder of `.md` and read the live store
(you'd read SQLite, fragile while the app runs).
- **Content:** notes are **Markdown** (Joplin-flavored: `:/imageId` resource links,
internal `:/<id>` note links, checkboxes/to-dos). Markdown-first — INTENT-aligned at
the *content* layer even though the *storage* layer is a DB.
- **Model:** `folders` (notebooks, hierarchical) → `notes``resources` (attachments);
cross-cutting `tags`; notes can be **to-dos**. Every item has a **32-char hex ID**;
notes link by ID (`:/<id>`), so links survive rename/move (store-minted, page-level).
- **Clients:** desktop (Electron), mobile, and a **terminal/CLI** app — multiple clients
edit one logical store, reconciled by sync (§2).
- **Revisions:** Joplin keeps **internal note history** locally — not git, not portable
→ a *supplement* case (UC-36), like Notion/Confluence.
---
## 2. Sync — the distinctive layer (and the attach opportunity)
Joplin's headline feature is **sync to a backend of your choice**. On sync, the SQLite
store is serialized to a **folder of plain-text items** — one file per note / notebook /
tag / resource-metadata, each carrying the **Markdown body plus a metadata block**
(`id`, `parent_id`, `type_`, `created_time`, `updated_time`, …) — written to the chosen
target: **filesystem, WebDAV, Nextcloud, Dropbox, OneDrive, S3, Joplin Server, Joplin
Cloud**. Deletions propagate via tombstones; concurrent edits across clients produce
**conflict notes** (keep-both, not silent overwrite).
Two consequences for shard-wiki:
1. **Attach the sync mirror, not the app (UC-60).** A Nextcloud/WebDAV/S3/filesystem
directory full of Joplin sync items is a **file-store shard** — readable *without
Joplin running and without touching SQLite* — provided the adapter understands the
**documented item format** (body + metadata footer, `:/id` links, resource items).
This is a new attach pattern: the backend's **interchange/sync representation** as
the attach surface, distinct from a native on-disk store (UC-40) or app files (UC-53).
It also realizes INTENT's WebDAV/Nextcloud participants concretely.
2. **Joplin is a file-sync layer over heterogeneous storage** — exactly what shard-wiki
says it is *not* (INTENT: "not a file-sync daemon"). Joplin syncs **one** logical
store across backends; shard-wiki **federates many** logical stores with wiki-page
semantics. Attaching a Joplin sync mirror means **reading its file representation as
pages**, never re-driving Joplin's sync — and never becoming a second sync engine
over the same target (don't double-sync).
---
## 3. End-to-end encryption — content opacity
Joplin offers **E2EE**: items are encrypted **before** leaving the device, so the sync
target holds **ciphertext** regardless of provider. (Security wart, noted: the master
key is stored *in clear* in the local SQLite — irrelevant to attaching the mirror, but
a reason never to treat the local DB as trusted.)
For shard-wiki this introduces a capability dimension the prior dives did not:
**content opacity**. An encrypted Joplin sync target can be attached only as a
**backup/mirror/structure-shell** participant — item IDs, counts, and change events may
be visible, but **bodies are undecryptable without keys**. The adapter must **never
present ciphertext as readable content** and must degrade to "present-but-opaque" with
provenance. This extends the synthesis capability spectra with a proposed twelfth field,
**content opacity** (`plaintext → encrypted-at-rest/opaque`), feeding `SHARD-WP-0002`
T11 (§9).
---
## 4. Extension surfaces — a plugin host *and* a local REST API
Joplin exposes **two** programmatic surfaces (plus export):
**A. Plugin API (in-app host).** TypeScript plugins, distributed via **npm + a plugin
repository**, loaded by the app:
- `joplin.data` — CRUD over the data model, shaped like REST (GET/POST/PUT/DELETE) on
`notes`/`folders`/`tags`/`resources`.
- `joplin.workspace` — selection + events (note selected, content changed, sync events).
- `joplin.contentScripts`**markdown-it** render plugins and **CodeMirror** editor
extensions (the syntax/rendering extension point).
- `joplin.views` — panels, dialogs, toolbar/menu items, custom editors;
`joplin.commands`, `joplin.settings`.
**B. Data API (local REST).** A REST service the desktop app serves on
**`localhost:41184`** (the "clipper server"), **token-authenticated**, used by the Web
Clipper and any local integration: endpoints `/notes`, `/folders`, `/tags`,
`/resources` (incl. `/resources/:id/file`), `/revisions`, `/search`; verbs
GET/POST/PUT/DELETE; pagination (`page`, `limit`≤100, `order_by`, `order_dir`). Notably
**plugins can use this API even when the clipper server is off**.
**C. Export****JEX** (tar of raw items), **Markdown directory**, **RAW** (Joplin
export directory), **ENEX** import (Evernote). Snapshot import/attach surface (UC-28/37).
So Joplin spans attachment modes: **in-app plugin host** (like Roam/Obsidian-plugin),
**local-REST** (like Notion's external API but *localhost + app-must-run* — a sub-mode),
the **sync-mirror file-store** (§2, the novel one), and **export snapshots**.
---
## 5. Joplin as a shard — capability profile
| Capability | Joplin | Notes for the adapter contract |
|------------|--------|--------------------------------|
| Read | **yes** | sync-mirror files (no app) · local Data API (app running) · plugin · export |
| Write | **yes** | Data API / plugin (`joplin.data`); writing the sync mirror directly is risky (Joplin owns that format) |
| Write granularity | **per-note (page)** | notebooks/notes/resources; not block-level |
| Identity / addressing | **32-char item ID; `:/<id>` links** | store-minted, **page-level**, survives rename (between Obsidian path and Roam block-UUID) (UC-51) |
| Structure | notebooks (hierarchy) + tags + to-dos | modest; metadata in the sync item footer (in-text on the mirror) |
| History | **internal revisions, not portable** | supplement via coordination journal (UC-36) |
| Native query | search API (FTS) | delegate text search; no datalog/DB-query (weaker than Roam/Notion) |
| Subscribe | sync is **poll-based**; conflict notes | poll; conflicts surface as keep-both items (UC-31, UC-07) |
| Content opacity | **optional E2EE → ciphertext** | **new dimension**: opaque-at-rest; mirror usable only as backup/structure without keys (UC-61) |
| Transclusion | none (note-level links only) | — |
| Diff / merge | none native | git-level if mirror is on a git-backed target |
| Publish | via export / static-site pipelines | outbound (UC-56) |
| Content types | Markdown + **resources** (attachments) | non-Markdown assets with IDs (UC-55) |
| Attach modes | sync-mirror file-store · local-REST · plugin · export | multi-mode, per-binding (UC-60, UC-57, UC-38) |
Verdict: a solid **Markdown-first candidate shard** whose best attach surface is the
**sync mirror on a file/WebDAV/S3 target** (offline, app-independent, INTENT-named),
with the local Data API or a plugin for live write-through. Standout: the
interchange-format attach (UC-60) and the encryption/opacity case (UC-61). Limits: DB
local store, note-level (not block) addressing, internal-only history, weak query.
---
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
### 6.1 Reinforcements
- **Interchange-format attach** validates that a shard's *documented sync/export
representation* — not just its live store or API — is a legitimate, often *best*,
attach surface (offline, app-independent). Generalizes beyond Joplin (UC-60).
- **WebDAV/Nextcloud/S3 become concrete** participants via Joplin's sync targets — the
backends INTENT names, now with a real format to parse.
- **Conflict-as-data** (Joplin conflict notes = keep-both) is union-without-erasure in
the wild (reinforces UC-07, divergence; consensus policy T9).
- **Store-minted, rename-stable note IDs** (`:/id`) confirm the addressing spectrum's
middle (page-level stable ID) between path (Obsidian) and block-UUID (Roam/Notion)
(UC-51).
### 6.2 Deliberate divergences (design bugs if conflated)
1. **Joplin is a sync daemon; shard-wiki is not.** Attach the **mirror as pages**; never
re-implement Joplin's sync or run a competing sync over the same target. (INTENT
not-a-file-sync-daemon.)
2. **DB-local; don't read SQLite live.** The native store is a DB; treat it as opaque.
Use the sync mirror, the Data API, or a plugin — not the live `database.sqlite`.
3. **Joplin owns the sync format; treat the mirror read-mostly.** Writing items into the
sync folder behind Joplin's back risks corruption/conflict; prefer overlay/projection
or write through the Data API/plugin (overlay before mutation, no silent mutation).
4. **Encrypted shards are opaque.** Never present ciphertext as content; degrade to
backup/structure with provenance (UC-61, graceful degradation).
### 6.3 What Joplin teaches that shard-wiki should keep
- Add **"interchange/sync representation"** as a recognized attach surface in the
contract — sometimes preferable to the live store/API (UC-60).
- Add **content opacity** (encryption-at-rest) as a capability-profile field so encrypted
shards degrade correctly (UC-61).
- Expect **multi-client concurrent editing reconciled by the backend's own sync**
(desktop/mobile/CLI) — a different concurrency than single-app file edits (extends
UC-53) or multi-user server writes; conflicts may pre-exist as keep-both items.
---
## 7. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-59**. New UCs **UC-60, UC-61** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a tool's documented sync/interchange representation on a third-party storage target** (Joplin items on Nextcloud/WebDAV/S3) as a file-store shard, without the app | **UC-60 (new)** |
| **Attach an encrypted-at-rest shard** (E2EE sync target): content opaque without keys; participate as backup/mirror/structure-shell, never presenting ciphertext as readable | **UC-61 (new)** |
| Store-minted, rename-stable **note-level** IDs (`:/id`) — the middle of the addressing spectrum | **enriches UC-51** |
| Native store can be a **DB while the sync representation is files** → attach surface ≠ native store | **enriches UC-40** |
| Internal revisions, not portable → supplement | **enriches UC-36** |
| Dual surfaces: in-app plugin host + **local Data API** (localhost, app-running) | **enriches UC-38** (links UC-57) |
| Resources/attachments = non-Markdown assets with IDs | **enriches UC-55** |
| Poll-based sync; conflict notes (keep-both) | **enriches UC-31** (links UC-07) |
---
## 8. Architecture notes for SHARD-WP-0002 (no UC)
- **Attachment-mode taxonomy gains a refinement:** within file-store, distinguish
*native on-disk store* (Obsidian/TWiki, UC-40) from *interchange/sync representation*
(Joplin mirror, UC-60). Both are "files," but format ownership and write-safety differ.
(T14.)
- **Add a "content opacity" capability field** (proposed **twelfth spectrum**:
`plaintext → encrypted-at-rest/opaque`) so encrypted shards degrade to
backup/structure-shell. Feeds T11 the next time the eleven spectra
(`research/260614-shard-spectrum-synthesis/findings.md` §2) are revised. (T11/T14.)
- **Local-REST as an attach sub-mode** (localhost, app-must-run, token) sits between
in-engine-host (UC-38) and external-API (UC-57) — note in T14.
- **Format-aware file-store adapters:** the contract should let a file-store adapter
declare a *format profile* (plain Markdown vs Joplin-item vs Foswiki-PlainFile) so the
same "folder of files" attach can parse tool-specific item formats. (T11/T14.)
---
## 9. Open questions (for spec / workplans)
1. For an **encrypted shard** (UC-61), what *is* visible without keys — item IDs/counts/
timestamps (structure shell) or nothing? Does shard-wiki ever hold keys, or only ever
treat such shards as opaque backups?
2. Is **writing to a tool's sync mirror** (UC-60) ever safe, or are interchange-format
shards **read/projection/overlay-only** by policy (Joplin owns the format)?
3. Does shard-wiki parse Joplin's **item format** via a dedicated format profile, or
only attach the **export** (JEX/Markdown dir) as a cleaner snapshot (UC-37)?
4. How do we avoid **double-sync** when a shard's storage target is itself driven by a
sync daemon (Joplin) we don't control?
---
## 10. Sources
| Source | Used for |
|--------|----------|
| Joplin Data API reference (https://joplinapp.org/help/api/references/rest_api/) | Local REST on `localhost:41184`, token auth, endpoints, 32-char IDs, item fields, pagination |
| Joplin Plugin API docs (https://joplinapp.org/api/references/plugin_api/classes/joplin.html) | `joplin.data` / `joplin.workspace` / `joplin.contentScripts` / views / settings |
| Joplin — Extending Joplin (https://joplinapp.org/help/api/) | Plugin distribution (npm + repository), Data API vs plugin access |
| Joplin — WebDAV / Nextcloud sync (https://joplinapp.org/help/apps/sync/nextcloud/) | Sync targets; items as plain-text files on the target |
| Joplin FAQ + community (https://joplinapp.org/help/faq/) | SQLite local store; Markdown content; E2EE before leaving device |
| Obsidian vs Joplin comparison (https://petronellatech.com/blog/obsidian-vs-joplin/) | Architecture contrast (SQLite+sync vs file-over-app), E2EE framing |
Cross-references: `research/260614-obsidian-deep-dive/findings.md` (file-over-app
contrast), `research/260614-notion-deep-dive/findings.md` (external-API, scoped grant),
`research/260614-shard-spectrum-synthesis/findings.md` (the spectra this extends),
`spec/UseCaseCatalog.md` (UC-31, UC-36, UC-38, UC-40, UC-51, UC-55, UC-57),
`workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
---
## 11. Traceability
- New UCs: **UC-60, UC-61**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-31, UC-36, UC-38, UC-40, UC-51, UC-55** (links UC-07, UC-57).
- Architecture (no UC): interchange/sync-representation attach surface; **content
opacity** as a proposed twelfth capability spectrum; local-REST sub-mode; format-aware
file-store profiles → `SHARD-WP-0002` (T11, T14).
- Boundary recorded: Joplin is a **sync daemon over one store**; shard-wiki attaches its
**mirror as pages** and does not re-sync — and treats DB-local store and encrypted
content as opaque (INTENT not-a-sync-daemon, graceful degradation, no silent mutation).
</content>

View File

@@ -0,0 +1,35 @@
# 260614 — Jupyter Notebooks deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T3**
## What this is
A deep dive into **Jupyter Notebooks**: the **`.ipynb` JSON** document (ordered cells:
markdown / code+outputs / raw), **kernels**, **embedded computed outputs** (MIME bundles),
and **execution-count provenance**. The dominant modern computational document and the
concrete case where the **derived output is captured and stored *inside* the source** with
real-but-fragile provenance.
## Why it matters
- Tests the T1 **replication- vs derivation-projection** split on the dominant real artifact
and adds the wrinkle that **outputs are stored back inside the source** — the source/
projection line runs *through* the document.
- The page model (T12) must carry a **notebook shape**: ordered cells with code cells owning
embedded computed outputs that have **weak execution provenance** (run order, environment
not captured).
- Non-Markdown + lossy translation (T15): JSON+MIME bundles; nbconvert→Markdown is lossy and
directional. JSON diffs are noisy → Jupytext text-pairing / nbdime (T13).
## Yield
- **UC-84** (new): attach/project a computational notebook preserving cell structure +
embedded outputs, surfacing outputs as **snapshots with weak execution provenance**;
re-execution **capability-gated**, default = present snapshot + static render.
- Enrich **UC-54, UC-55, UC-59, UC-35**; links **UC-32, UC-83, UC-79**.
## Contents
| Path | Role |
|------|------|
| `findings.md` | `.ipynb` model, kernels/execution-count fragility, ecosystem (nbconvert/Jupytext/papermill/nbdime/nbstripout), capability profile, INTENT mapping, UC seed, architecture notes, open questions |

View File

@@ -0,0 +1,185 @@
# Jupyter Notebooks — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T3 · **Subject:** Jupyter Notebooks — the
`.ipynb` JSON document, kernels, embedded computed outputs, execution provenance.
## Why this dive
T1 (literate programming) established **one source → derived projections** and split
**replication-projection** from **derivation-projection**. Jupyter is the *dominant modern
computational document* and the concrete case where the **derived output is captured and
stored inside the source** — a non-Markdown, partially-executable content type whose
provenance is real but **fragile**. It is the most plausible concrete "computational shard"
content type, so it tests the page model (T12), lossy translation (T15), and the
output-provenance question head-on.
## 1. The `.ipynb` document model
A notebook is a single **JSON document** (`nbformat`), not Markdown:
- **`cells[]`** — an ordered list. Each cell has a `cell_type`:
- `markdown` — prose (Markdown + LaTeX), the human-readable part.
- `code` — source text (`source`), plus an **`execution_count`** and an **`outputs[]`**
array captured from the last run.
- `raw` — passthrough.
- **`outputs[]`** (per code cell) carry results inline: `stream` (stdout/stderr),
`execute_result` / `display_data` (a **MIME bundle**`text/plain`, `text/html`,
`image/png` base64, `application/json`, vendor MIME types), and `error` (traceback).
- **`metadata`** at notebook and cell level (`kernelspec`, `language_info`, tags like
`hide-input`, `scrolled`, slideshow roles).
So an `.ipynb` is **source + last-run computed outputs + environment metadata, fused in one
JSON file**. The Markdown cells are an *island* inside a JSON envelope — relevant to how
shard-wiki extracts/round-trips content.
## 2. Kernels and execution
- A **kernel** is a separate language process (IPython, IRkernel, IJulia, …) speaking the
Jupyter messaging protocol (ZeroMQ). The document is **decoupled from the kernel**: the
`.ipynb` persists *captured* outputs; re-running requires a live kernel + the right
environment.
- **`execution_count`** numbers the order cells were *run*, which **need not match document
order** — the infamous **hidden-state / out-of-order execution** problem: stored outputs
may reflect a run sequence that no longer corresponds to top-to-bottom reading.
- Reproducibility therefore depends on **out-of-band state**: package versions, data files,
environment, random seeds — none captured by `nbformat` itself.
**Consequence for shard-wiki:** the captured outputs are a **snapshot projection with weak
provenance** — honest treatment must mark them as "computed at run N, environment not
guaranteed," never as live or authoritative truth.
## 3. The ecosystem (relevant to attach/project/translate)
- **nbconvert** — derives other forms from a notebook: HTML, Markdown, LaTeX/PDF, slides,
script. This is **derivation-projection** (T1): notebook source → rendered view, lossy in
both directions (HTML keeps outputs; `--to script` keeps only code, like `tangle`).
- **Jupytext** — represents a notebook **as** a `.py`/`.md` text file (pairing), making it
**git-diffable plain text** and round-trippable — directly relevant to storing notebooks
in a git shard without JSON-diff noise.
- **papermill** — parameterize + execute a notebook to produce a new output notebook
(notebook as a runnable template — a *derivation with inputs*).
- **JupyterLab / Notebook / nbviewer / Colab** — front-ends; nbviewer renders a static
read-only projection from a URL (a natural projection target).
- **`nbstripout`** — strips outputs before commit: teams treat **outputs as derived noise**,
keeping only source under version control — an explicit "source canonical, outputs
derived" stance mirroring T1.
## 4. Capability profile (as a shard / content type)
| Dimension (synthesis spectrum) | Jupyter notebook |
|--------------------------------|------------------|
| Attachment mode | file-store (`.ipynb` JSON in a repo) or via Jupyter Server REST API |
| Addressing granularity | document; **cell** as sub-address (by index / id; `nbformat 4.5+` adds stable cell `id`) |
| Content identity | file path; cell `id` (4.5+) else positional |
| Structure | **ordered cell list** (markdown / code+outputs / raw); MIME-bundle outputs |
| History | VCS on the file; **JSON diffs are noisy** unless paired (Jupytext) or stripped |
| Merge model | git on JSON (poor) → **paired text** (good) or nbdime (cell-aware diff/merge) |
| Native query | none |
| Translation | nbconvert → HTML/MD/script/PDF (lossy, directional); Jupytext text pairing |
| Write granularity | file / **cell** |
| Operational envelope | a kernel + environment to (re)execute; static render needs none |
| Content opacity | **mixed**: source transparent; outputs = MIME blobs (some opaque, e.g. base64 PNG) |
| Provenance | `execution_count` (weak, out-of-order); environment **not** captured |
| **Computed-output** | **stored inline**, snapshot, reproducibility out-of-band |
## 5. INTENT mapping
### Reinforcements
- **Replication- vs derivation-projection (T1) confirmed and extended.** nbconvert (→HTML/
script) and nbviewer are derivation-projections; `--to script` is literally `tangle`.
Jupyter adds a third wrinkle: **the derived output is also stored back inside the source**
(captured outputs), so the "source vs projection" line runs *through* the document.
- **Union without erasure / provenance honesty.** Captured outputs must be surfaced **as
snapshots with weak provenance** (run N, environment unguaranteed) — a concrete instance
of "never hide freshness/authorship." The out-of-order `execution_count` is exactly the
kind of fragility shard-wiki must *show*, not paper over.
- **Non-Markdown content + lossy translation (UC-55/UC-59).** `.ipynb` is JSON with embedded
MIME-bundle outputs; any Markdown projection is **lossy** (loses live outputs, kernel,
rich MIME). Surface the lossiness; keep the JSON as canonical payload (T12/T15).
- **Markdown island.** Markdown cells fit the text-first model, but only as *fragments
inside* a JSON envelope — the adapter extracts/round-trips them, it does not pretend the
notebook is a Markdown page.
### Divergences / boundaries
- **shard-wiki is not a kernel host.** Re-execution (driving a kernel) is out of scope/
capability-gated; default treatment is **attach + present captured outputs as a snapshot
projection** + offer nbconvert-style static render. Executing/parameterizing (papermill)
is an optional capability, never assumed.
- **Outputs-in-source is an anti-pattern to respect, not adopt.** Teams strip/pair outputs
precisely because mixing derived data into the source breaks diffs. shard-wiki should
prefer the **source-canonical, outputs-as-derived** reading (Jupytext pairing / nbstripout
ethos) and treat stored outputs as a capturable projection.
### What to keep
1. **Computational-notebook as a first-class content type** with cell structure + inline
**computed outputs carrying (weak) execution provenance** — UC-84.
2. **Outputs = derivation-projection snapshot** (T1 vocabulary): regenerable only with a
kernel+environment; degrade gracefully to the stored snapshot / static render.
3. **Cell-level addressing** (stable cell `id`, nbformat 4.5+) as the sub-page granularity
for transclusion/anchoring (UC-32/UC-35).
4. **Text-pairing (Jupytext)** as the git-friendly storage strategy — feeds the
history-portability thread (poor JSON diffs → paired text / nbdime).
## 6. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-84 | Attach/project a **computational notebook** (`.ipynb`): preserve **cell structure** (markdown / code / output) and **embedded computed outputs**, surfacing each output **as a snapshot with its (weak) execution provenance** (run count, environment not guaranteed) — re-execution is **capability-gated**, default is present-the-snapshot + offer a static rendered projection | **new** |
| — | Notebook JSON / MIME-bundle outputs = non-Markdown content; Markdown projection is lossy | enrich **UC-55**, **UC-59** |
| — | Computed/evaluated cell = computation-defined content | enrich **UC-54** |
| — | Cell `id` (nbformat 4.5+) = sub-page address for anchor/transclusion | enrich **UC-35**, links **UC-32** |
| — | Stored outputs as derived snapshot (nbstripout/Jupytext ethos) = source-canonical/outputs-derived | links **UC-83**, **UC-79** |
## 7. Architecture notes for SHARD-WP-0002
- **T12 (page model):** add **computational-notebook** as a page shape — an **ordered cell
list** where code cells own **embedded computed outputs** (MIME bundles) with weak
execution provenance. Distinct from prose, typed records, query-defined, inline-embedded
objects (Quip/Notion), typed-graph (Wikibase), and the literate one-source-many-projection
shape (UC-83). The defining new attribute: **derived output stored *inside* the source**.
- **T15 (translation / fidelity):** `.ipynb` is non-Markdown; nbconvert→Markdown is **lossy
and directional** (drops live outputs/kernel/rich MIME). Keep JSON canonical; any Markdown
is a projection. MIME-bundle outputs map to the content-opacity spectrum (text→html→base64
image = transparent→opaque).
- **T13 (history):** JSON diffs are **noisy**; record **text-pairing (Jupytext)** and
**cell-aware diff/merge (nbdime)** as history-portability strategies for embedded-output
documents. Reinforces "source-canonical, outputs-derived."
- **T16 (projection):** captured outputs are a **derivation-projection snapshot**;
re-execution (kernel) and parameterized execution (papermill) are **capabilities**, not
assumptions; degrade to the stored snapshot / nbviewer-style static render.
## 8. Open questions
1. Does shard-wiki ever **re-execute** a notebook (host/broker a kernel), or strictly
attach + present captured outputs + static render? (Same scope boundary as UC-83/UC-56
"do we ever drive the derivation.")
2. Is **UC-84** distinct from **UC-83**, or is a notebook just the "outputs-stored-in-source"
special case of the literate one-source-many-projection pattern? (Kept separate: UC-84's
defining trait is *captured derived output embedded in the canonical source with weak
provenance* — a page-model attribute UC-83 doesn't carry.)
3. How are **MIME-bundle outputs** represented in the page model — opaque provenance-tagged
blobs, a typed-asset registry (UC-55 open question #10), or selected-MIME projection?
4. Default storage: attach `.ipynb` **as-is** (JSON, noisy diffs) or prefer a **paired text
representation** when the shard is a git repo? (Policy → configurable.)
## 9. Sources
- Jupyter `nbformat` reference (cells, outputs, MIME bundles, cell `id` 4.5+);
Jupyter messaging protocol / kernels docs.
- **nbconvert**, **nbviewer**, **JupyterLab**, **Colab** docs.
- **Jupytext**, **papermill**, **nbdime**, **nbstripout** project docs.
- prior: `research/260614-literate-programming-deep-dive/` (replication- vs
derivation-projection, UC-83); `research/260614-notion-deep-dive/` (block-JSON,
external-API), `research/260614-quip-deep-dive/` (inline embedded objects, UC-55/58/59).
## 10. Traceability
New UC **UC-84** carries the marker **⊜** in the wikiengines column of
`spec/UseCaseCatalog.md` (true lineage = this dive). Enriched: UC-54, UC-55, UC-59, UC-35;
links UC-32, UC-83, UC-79. Architecture cross-refs: SHARD-WP-0002 T12 (notebook page shape:
outputs embedded in source), T15 (lossy non-Markdown translation; MIME opacity), T13
(paired-text / nbdime history), T16 (output = derivation-projection snapshot; kernel =
capability).

View File

@@ -0,0 +1,35 @@
# 260614 — Literate Programming (Knuth's WEB / weave / tangle) deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T1**
## What this is
A deep dive into **literate programming** — Knuth's WEB and the **`weave`/`tangle`** model
— as the deepest ancestor of shard-wiki's **projection** and **transclusion** ideas
applied to *executable* content. The keystone: **one canonical source → two co-derived
projections** (typeset docs via `weave`, compilable code via `tangle`), plus **named code
chunks** assembled by reference (transclusion).
## Why it matters
- Establishes **one-source-many-projections** as a page-model + projection pattern that
*generalizes* compile-to-static (UC-79, single output) to **N co-equal, semantically
different** derived views — feeds SHARD-WP-0002 **T12/T16**.
- Splits projection into **replication-projection** (lazy cache, current default) vs
**derivation-projection** (transform/compile/weave/evaluate) — a distinction the rest of
the computational batch (notebooks, REPLs) extends.
- Named chunks are the **executable-content face of transclusion / compose-by-reference**
(UC-32 / UC-44).
## Yield
- **UC-83** (new): attach a single-source-multiple-projection (literate) artifact; present
each derived view with output→source provenance; edits target the source.
- Enrich **UC-32**, **UC-44**, **UC-79**; links **UC-54** (computed/evaluated projection,
→ T3 Jupyter).
## Contents
| Path | Role |
|------|------|
| `findings.md` | WEB model, named-chunk transclusion, descendants (noweb/org-babel/knitr/Jupytext), capability profile, INTENT mapping, UC seed, architecture notes, open questions |

View File

@@ -0,0 +1,180 @@
# Literate Programming (Knuth's WEB / weave / tangle) — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T1 · **Subject:** Donald Knuth's WEB
system and the literate-programming model (`weave`/`tangle`, named chunks).
## Why this dive
SHARD-WP-0004 asks the carried question: *can a shard-wiki page be a source that is
woven/evaluated into rendered forms, and how do projection/transclusion/provenance treat
the source vs the output?* Literate programming is the **deepest ancestor** of that idea.
Knuth (1984) inverted the program/comment relationship: you write a **document** for
humans whose fragments happen to also be the program. From the **one WEB source** two
tools derive two artifacts: **`weave` → typeset documentation** (TeX) and **`tangle`
compilable code** (Pascal/C/…). This is *one source, two projections* in its purest,
oldest form — the conceptual root of shard-wiki's **projection** and **transclusion**.
## 1. The WEB model: one source, two tools
A WEB file interleaves prose and code in author-chosen order (the order that best
*explains*, not the order the compiler needs). Two programs read it:
- **`weave`** produces a `.tex` file → typeset documentation: prose, pretty-printed code,
a cross-reference index of where each chunk and identifier is defined/used.
- **`tangle`** produces a compilable source file → reorders and expands the code chunks
into the sequence the compiler demands, strips the prose, macro-expands references.
The crucial property: **the two outputs are co-derived from one canonical source and are
semantically different audiences** (human reader vs compiler). Neither output is the
source; both are **regenerable, disposable derivations**. Editing happens on the WEB; you
never edit the woven `.tex` or the tangled `.c`.
## 2. Named chunks = transclusion of code fragments
The organizing primitive is the **named section / code chunk**:
- A chunk is declared `<<name>>=` and **referenced** elsewhere as `<<name>>`. `tangle`
expands references in place (recursively) to assemble the final program.
- A chunk name can be **defined in multiple places** and is *accreted* (later `+=`
additions append) — so one logical unit is authored across scattered locations.
- References can appear before definitions; resolution is by name, not by position.
This is **transclusion by reference** (UC-32) and **compose-by-reference / manifest**
(UC-44, the EDL/Xanadu lineage): the document is a graph of named fragments assembled at
derivation time. Knuth's chunk graph is the same shape as Xanadu's reference-not-copy and
shard-wiki's "compose a page from referenced parts" — applied to *executable* content and
resolved by a build tool rather than a viewer.
## 3. The descendants (noweb, CWEB, org-babel, Sweave/knitr, Jupytext)
- **CWEB** (Knuth/Levy): WEB for C. **noweb** (Ramsey): language-agnostic, minimal markup
(`<<chunk>>`), `notangle`/`noweave` — proof that the *model* (chunks + two projections)
is separable from any one language or typesetter.
- **org-babel** (Emacs Org-mode): named source blocks, `:noweb` references, **tangle** to
files **and evaluate** blocks inline (results captured back into the document) — literate
programming fused with notebook execution (bridges to T2/T3).
- **Sweave / knitr** (R): weave prose + R, executing code and **interleaving computed
results/figures** into the woven document — adds the *computed-output* dimension that
Jupyter (T3) centers on.
- **Jupytext**: represents a Jupyter notebook **as** a literate text/Markdown source —
closing the loop: the notebook (T3) becomes a weave/tangle-style plain-text source.
The throughline: **one canonical source → N derived projections**, where projections may
be (a) reformatted (weave), (b) reordered/extracted (tangle), or (c) **evaluated**
(org-babel/knitr) — the evaluated case is exactly the computational-page question.
## 4. Capability profile (as a would-be shard / page type)
| Dimension (synthesis spectrum) | Literate-programming source |
|--------------------------------|-----------------------------|
| Attachment mode | file-store (a WEB/`.nw`/`.org` text source in a repo) |
| Addressing granularity | document; **named chunk** as sub-page address |
| Content identity | source file path + chunk name (name-resolved, not position) |
| Structure | **graph of named chunks** assembled by reference |
| History | whatever VCS holds the source (git) — text, diffable |
| Merge model | text/git merge on the source |
| Native query | none; `weave` emits a cross-reference index (derived) |
| Translation | source → woven docs **and** tangled code (build-time) |
| Write granularity | file / **chunk** (text region) |
| Operational envelope | a build tool (`tangle`/`weave`/`noweb`/babel) |
| Content opacity | transparent text |
| Provenance | VCS author/time; chunk cross-ref maps output→source location |
| **Projection model** | **one source → many co-equal derived projections** (new emphasis) |
## 5. INTENT mapping
### Reinforcements
- **Projection (canonical vs derived).** Literate programming is the archetype of
"canonical source, regenerable derived view" — the same principle as ikiwiki
compile-to-static (UC-79), but generalized: **two-plus co-equal projections** from one
source (docs *and* code), not a single output. Strengthens the rule *never confuse a
projection for the source*.
- **Transclusion / compose-by-reference.** Named chunks are transclusion (UC-32) and a
manifest of referenced parts (UC-44) — resolved at derivation time. Confirms
transclusion=clone=embed=reference as one primitive that also covers *fragment assembly
of executable content*.
- **Markdown-first but backend-neutral page model.** noweb/org/Jupytext show the literate
source can *be* Markdown-ish plain text — so a "literate page" fits the text-first model;
the *derivations* are the non-text part.
- **Mechanism over policy.** weave/tangle are mechanisms; *which* projections to
materialize, when to regenerate, and where outputs go stay configurable.
### Divergences / boundaries
- **shard-wiki is not a build system.** It should *recognize and present* a
source-with-projections (attach the source, surface derived views with provenance), not
re-implement `tangle`/kernels. Materializing a projection may delegate to the source's
own tool or be capability-gated to "snapshot only."
- **The interesting projection is derivation, not caching.** shard-wiki's base projection
is cache-like (lazy copy of remote content, UC-53/57). Weave/tangle is a *different*
projection species: **transform/derive** one source into rendered forms. The contract
should model projection as having (at least) two kinds: **replication-projection** and
**derivation-projection** (compile/weave/evaluate).
### What to keep
1. **One-source-many-projections** as a first-class page-model + projection pattern
(generalizes UC-79's single output) — see UC-83.
2. **Named-chunk transclusion** as the executable-content face of UC-32/UC-44 (assembly by
reference at derivation time).
3. **Output→source provenance** (the woven cross-ref index): every derived view must point
back to the exact source location it came from — never present derived output without
that link (union-without-erasure for derivations).
## 6. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-83 | Attach a **single-source-multiple-projection** artifact (a literate/woven source): treat the source as canonical, present each **derived projection** (e.g. a documentation view *and* a code view) with **provenance back to the one source**, edits target the source and projections regenerate (delegated to the source's tool or degraded to a static snapshot) | **new** |
| — | Named chunks `<<name>>` = transclusion / compose-by-reference of (executable) fragments | enrich **UC-32**, **UC-44** |
| — | Generalize compile-to-static (single output) to **N co-equal projections** from one source | enrich **UC-79** |
| — | Computed/evaluated projection (org-babel/knitr) = derivation-projection with results | links **UC-54**, foreshadows **T3** |
## 7. Architecture notes for SHARD-WP-0002
- **T12 (page model):** add **one-source-many-projections** as a page-model shape — a page
may be a *source* whose presented forms are **derivations** (woven docs, tangled code,
evaluated results), each carrying output→source provenance. Distinct from prose, typed
records, query-defined, and inline-embedded objects already logged.
- **T16 (projection):** split projection into **replication-projection** (lazy cache of
remote content — current default) vs **derivation-projection** (transform/compile/weave/
evaluate a source into rendered forms). Derivation-projection is regenerable, may
delegate to an external tool, and degrades to a captured snapshot when the tool is
absent (graceful degradation).
- **Transclusion (T16):** named-chunk-by-name resolution is a transclusion variant where
the *target is a fragment resolved by name across the source*, assembled at derivation
time — a concrete shape for UC-32/UC-44 mechanics.
- **Capability gating:** "can derive projection X" is a capability; a shard that can't run
the tool still exposes the source + any pre-built/snapshot projections (UC-83 degrade).
## 8. Open questions
1. Does shard-wiki ever *drive* a derivation (run weave/tangle/evaluate), or only attach
sources and surface pre-built projections + snapshots? (Same scope question as UC-56
"do we ever compile-to-static ourselves," now for literate sources.)
2. Is **UC-83** distinct enough from UC-79 (compile-to-static) to stand alone, or should
UC-79 be re-read as the single-output special case of UC-83's N-projection general case?
(Recorded as a possible later consolidation; kept separate now because UC-83's
projections are *co-equal and semantically different audiences*, not one publish target.)
3. How is **output→source provenance** represented when a derived line came from a chunk
accreted across several source locations (the cross-ref is many-to-one)?
## 9. Sources
- Knuth, *Literate Programming* (1984, *Computer Journal*); the WEB user manual; *TeX: The
Program* / *MMIX* as canonical WEB exemplars.
- Ramsey, *noweb* (a simple, extensible literate-programming tool).
- CWEB (Knuth & Levy); Emacs **Org-mode babel** (tangle + evaluate); **Sweave**/**knitr**
(R); **Jupytext** (notebook-as-text).
- prior: `research/260614-ikiwiki-deep-dive/` (compile-to-static, canonical-vs-derived);
`research/260614-xanadu-deep-dive/` (compose-by-reference / EDL, UC-44).
## 10. Traceability
New UC **UC-83** carries the marker **⊛** in the federation column of
`spec/UseCaseCatalog.md` (true lineage = this dive; literate programming is design prior
art, not a candidate shard, so the marker sits with the projection/compose-by-reference
family). Enriched: UC-32, UC-44, UC-79; links UC-54. Architecture cross-refs: SHARD-WP-0002
T12 (one-source-many-projections page shape), T16 (replication- vs derivation-projection;
named-chunk transclusion).

View File

@@ -0,0 +1,46 @@
# 260614 — Local-first workspaces cohort deep dive (Anytype · AFFiNE · AppFlowy)
Date: 2026-06-14
## What this is
A **combined cohort deep dive** of three open-source, local-first "all-in-one workspace"
tools — **Anytype**, **AFFiNE**, **AppFlowy** — studied together because they share the
one trait none of the prior eleven systems had: **the store is a CRDT** (Anytype's
*any-sync*, AFFiNE's *Yjs/y-octo*, AppFlowy's *Yrs*). The substrate is the contribution;
studying them together isolates it from each tool's surface features.
Per-tool highlights:
- **Anytype** — `any-sync`: **P2P + E2EE** CRDT (sync/file/consensus/coordinator nodes;
encrypted backup), a **typed object graph** (user-editable types + relations), IPFS
files, an **open API + MCP**.
- **AFFiNE** — **Yjs/y-octo** over **OctoBase**; **BlockSuite** editor framework; docs,
whiteboards, and databases as **views of the same block set**; WebSocket sync,
self-host (Docker, MPL-2.0).
- **AppFlowy** — **Yrs** via **AppFlowy-Collab** (Rust); Flutter UI; Notion-style
**databases + views**; pluggable storage/sync (RocksDB/IndexedDB; AppFlowy Cloud/
Supabase).
## Contents
| Path | Role |
|------|------|
| `findings.md` | Per-tool sections, the CRDT cohort thesis, comparative matrix, combined capability profile, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Cohort dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-64
attach a CRDT-synced local-first shard with native conflict-free merge; UC-65 attach a
peer-to-peer / decentralized shard with no single canonical endpoint); UC-36/47/48/51/55/
57/58/61 enriched. Logged for `SHARD-WP-0002` (T11/T14): a **merge-model** capability
(proposed thirteenth spectrum: `none / git / native-CRDT`), a **CRDT-replica** substrate +
**P2P/no-central-endpoint** attach mode, an **endpoint-model** capability, and MCP as an
integration surface.
**Cohort thesis recorded:** CRDT changes the adapter math — the backend performs
conflict-free merge itself (so shard-wiki must not impose git/text merge), history is a
CRDT update log (supplement, not git), and the attach surface is a **replica or sync
endpoint** (Anytype adds **P2P + E2EE**). **Boundary:** CRDT local-first candidate shards
— attach a replica/projection as pages, respect native merge, never re-drive their sync;
not substrates and not the federation layer.
</content>

View File

@@ -0,0 +1,261 @@
# Findings — Anytype, AFFiNE, AppFlowy: the CRDT local-first workspace cohort
Date: 2026-06-14
Source kind: **modern shipped products (cohort dive)** — three open-source, local-first
"all-in-one workspace" tools studied together; candidate shards of a **new substrate
family: CRDT-synced local-first stores**
Lens: shard-wiki — the CRDT substrate and its native-merge / P2P / E2EE implications for
the adapter contract
> Why one combined memo. Anytype, AFFiNE, and AppFlowy are the open-source local-first
> Notion/Miro alternatives, and they share the one trait the prior eleven systems never
> had: **the store is a CRDT** (Anytype's *any-sync*, AFFiNE's *Yjs/y-octo*, AppFlowy's
> *Yrs*). That single fact changes the adapter math — **the backend performs conflict-
> free merge itself**, history is a CRDT update log (not git), and content arrives as a
> replica/export, not files or rows. Studying them together isolates the *substrate*
> contribution from each tool's surface features and avoids triplicating boilerplate.
> Per-tool sections (§1§3), then the cohort thesis and what it means for shard-wiki
> (§4 onward).
Contrast set: file-store (Obsidian/Logseq), client-DB (Roam), hosted-DB (Notion),
SQLite-local+file-sync (Joplin). This cohort adds **CRDT-replica** as a substrate.
---
## 1. Anytype — P2P, E2EE, typed object graph (any-sync)
- **Substrate:** the open **any-sync** protocol — **local-first, peer-to-peer,
end-to-end encrypted** CRDT. Infrastructure is four node types: **sync, file,
consensus, coordinator**; **backup nodes store ciphertext they cannot read**. Every
change is **encrypted and signed with the user's private key**, verified via a shared
public key. Files via **IPFS**.
- **Data model:** **spaces** = graph-based databases; users define **custom object
types with relations** ("everything is an object"; a user-editable **ontology**).
Closest to Notion's typed model but **local-first + E2EE + user-owned schema**.
- **Extension:** an **open API** for integrations, **plus MCP support** (AI-agent
collaboration) — a notable modern surface (a shard speaking MCP).
- shard-wiki read: the **most decentralized + encrypted** shard yet — no single
canonical endpoint (attach a local replica or a node), content opaque without keys
(UC-61), typed object graph (UC-58), CRDT merge (UC-64).
## 2. AFFiNE — docs + whiteboard + DB as views of one block set (Yjs/BlockSuite)
- **Substrate:** **Yjs CRDT** via **y-octo** (native Rust Yjs) over **OctoBase** (Rust
local-first engine). Local-first by default; optional **WebSocket realtime** sync to
AFFiNE Cloud or self-host (MPL-2.0, Docker).
- **Editor:** **BlockSuite** — an open block-editor framework providing the **page
editor, the edgeless canvas, and block types (incl. databases/embeds)**.
- **Headline:** **docs, whiteboards, and databases are *views of the same underlying
data*** — one block set rendered as page mode, edgeless canvas, or DB views. CRDT
guarantees offline edits "merge mathematically perfectly" on reconnect.
- shard-wiki read: the **one-data-many-renderings** model shipped (ZigZag dimensions,
UC-47/48), with **non-Markdown spatial content** (edgeless canvas, UC-55) and CRDT
merge (UC-64).
## 3. AppFlowy — Rust/Flutter, Notion-style DBs over Yrs CRDT
- **Substrate:** **Yrs** (Rust Yjs) wrapped as **AppFlowy-Collab** — CRDT primitives +
persistence + domain objects (documents, **databases**, folders, importers, plugins,
user state). **Pluggable storage/sync:** RocksDB (native) / IndexedDB (wasm) locally;
**Supabase or AppFlowy Cloud** (self-hostable) for sync, via **trait-based local-only
vs cloud**.
- **Data model:** Notion-like **databases with views** (grid/board/calendar), typed
fields. Flutter UI + Rust core, modular.
- shard-wiki read: a **self-hostable CRDT Notion** — typed DBs + views (UC-58, UC-54),
CRDT merge (UC-64), self-host sync endpoint (UC-57).
---
## 4. The cohort thesis — CRDT changes the adapter math
All three converge on **CRDT-synced, local-first, open-source, self-hostable** stores
with **typed/database object models** and **non-Markdown content** (canvas/DB). The
*substrate* is the new thing, and it has four consequences shard-wiki must absorb:
1. **Native, conflict-free merge (UC-64).** The backend resolves concurrent edits
itself (Yjs/Yrs/any-sync). shard-wiki must **not impose git/text merge** on a CRDT
shard; it either speaks the CRDT (replica) or stays a **projection/overlay** that
respects CRDT semantics. This adds a **merge-model** capability:
`none → git/text → native-CRDT` — a candidate **thirteenth spectrum** for the
contract (after Joplin's content-opacity twelfth).
2. **History is a CRDT update log, not git (UC-36).** No portable commit history; the
coordination journal **supplements** (like Notion/Joplin), or shard-wiki snapshots
the replica over time.
3. **The attach surface is a replica or a sync endpoint, not files/rows.** Attach a
**local CRDT replica** (offline, full state) or the **self-host sync server's API**
(AFFiNE Cloud / AppFlowy Cloud / any-sync node). Anytype adds **P2P with no single
canonical endpoint** (UC-65) and **E2EE content opacity** (UC-61).
4. **Typed object graphs + multi-view (UC-58, UC-47/48, UC-55).** User-editable
ontologies (Anytype types/relations), Notion-style DBs (AppFlowy), and one-block-set-
many-renderings (AFFiNE page/canvas/DB) — the structured + dimensional demands the
prior dives raised, now local-first.
These are **sync layers** (any-sync, Yjs sync, AppFlowy Cloud) — like Joplin, the
**not-a-file-sync-daemon** boundary applies: shard-wiki attaches a **replica/projection
as pages** and does **not** re-implement or re-drive their CRDT sync.
---
## 5. Comparative matrix
| | Anytype | AFFiNE | AppFlowy |
|---|---------|--------|----------|
| CRDT | any-sync (custom) | Yjs (y-octo) | Yrs |
| Sync | **P2P + nodes, E2EE** | WebSocket, cloud/self-host | RocksDB local + cloud/Supabase |
| Encryption | **E2EE (default)** | optional (transport/self-host) | self-host boundary |
| Local engine | local + IPFS files | OctoBase (Rust) | RocksDB / IndexedDB |
| Data model | **typed object graph (ontology)** | blocks; **doc/canvas/DB = one data, many views** | Notion-style **DBs + views** |
| Non-MD content | objects, files | **edgeless canvas** | board/calendar/grid |
| Extension | **open API + MCP** | BlockSuite framework | Flutter/Rust modules; plugins (roadmap) |
| Self-host | backup/sync nodes | Docker (MPL-2.0) | AppFlowy Cloud |
| Markdown | export (lossy) | import/export (lossy) | import/export (lossy) |
---
## 6. Combined capability profile (as a shard)
| Capability | This cohort | Notes for the adapter contract |
|------------|-------------|--------------------------------|
| Read | yes (replica or sync API) | local CRDT replica (offline) or self-host/P2P endpoint |
| Write | yes, **CRDT-mediated** | writes are CRDT ops; respect merge — don't overwrite |
| Merge | **native CRDT (conflict-free)** | **new capability**: backend owns merge; no external git merge (UC-64) |
| Write granularity | block/object | fine-grained |
| Identity / addressing | object/block IDs (CRDT-assigned) | fine-grained, store-minted (UC-51) |
| Structure | **typed object graph / DBs + relations** | user-editable ontology (Anytype); DB views (AppFlowy) (UC-58) |
| History | **CRDT update log, not git** | supplement / snapshot (UC-36) |
| Native query | varies (graph/DB filters) | delegate where present (UC-52) |
| Subscribe | realtime CRDT sync / P2P | push by nature; attach as replica (UC-31) |
| Content opacity | **Anytype E2EE → ciphertext** | opaque without keys (UC-61) |
| Endpoint model | **replica / self-host server / P2P (Anytype)** | no single canonical URL for P2P (UC-65) |
| Content types | docs + **canvas/DB/objects** | non-Markdown (UC-55); multi-view (UC-47/48) |
| Markdown | lossy export | fidelity report (UC-59) |
| Attach modes | CRDT replica · self-host API · (Anytype) MCP/open API | new substrate row in the taxonomy |
Verdict: a **demanding but coherent new family** — best attached as a **local replica
(offline, full CRDT state)** projected to pages, or via a **self-host sync endpoint**;
Anytype additionally **P2P + E2EE** (replica-only/opaque). Standout demands:
**native-CRDT merge** (UC-64) and **P2P/no-central-endpoint + E2EE** (UC-65/UC-61).
---
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
- **Local-first sovereignty** is the cohort's whole pitch — strong INTENT alignment;
each is a legitimate sovereign shard with user-owned data.
- **Self-hostable + open-source** means shard-wiki can attach within a user's trust
boundary (good for the L0→L4 authz ladder).
- **One-data-many-views** (AFFiNE) and **typed object graphs** (Anytype/AppFlowy) are
the dimensional (UC-47/48) and structured (UC-58) demands, now validated local-first.
### 7.2 Deliberate divergences (design bugs if conflated)
1. **Do not impose git/text merge on a CRDT shard.** The backend merges; shard-wiki
respects CRDT semantics or stays a projection/overlay (UC-64). Treating a CRDT shard
like a file shard corrupts it.
2. **Not a sync daemon.** any-sync/Yjs/AppFlowy Cloud already sync; shard-wiki attaches
a **replica as pages**, never re-drives their sync (INTENT not-a-file-sync-daemon).
3. **P2P has no canonical endpoint.** Don't model the shard as a URL; attach a replica
or a named peer/node (UC-65).
4. **E2EE shards are opaque** without keys — backup/structure-shell only (UC-61).
5. **Proprietary CRDT/object formats are lossy to Markdown** — translate with a fidelity
report (UC-59), one workspace = one shard, not the federation layer.
### 7.3 What the cohort teaches that shard-wiki should keep
- Add a **merge-model** capability (`none / git / native-CRDT`) so CRDT shards are
handled correctly (UC-64) — a thirteenth capability spectrum.
- Add **CRDT-replica** to the substrate/attachment taxonomy, and **P2P/no-central-
endpoint** as a binding mode (UC-64/65).
- **MCP as an integration surface** (Anytype) is worth noting — a shard may expose an
AI-agent protocol shard-wiki could consume or coexist with.
---
## 8. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-63**. New UCs **UC-64, UC-65** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a CRDT-synced local-first shard** whose backend performs native conflict-free merge; respect CRDT semantics (no git merge), consume a replica/export, CRDT-log history (supplement) | **UC-64 (new)** |
| **Attach a peer-to-peer / decentralized shard with no single canonical endpoint** (Anytype any-sync: replica or peer/backup node; E2EE) | **UC-65 (new)** |
| Typed object graph / user-editable ontology + Notion-style DBs, local-first | **enriches UC-58** |
| E2EE content opacity (Anytype) reinforced; self-host trust boundary | **enriches UC-61** |
| CRDT update log ≠ portable git history → supplement | **enriches UC-36** |
| One block set rendered as page / canvas / DB views (AFFiNE) | **enriches UC-47, UC-48** |
| Edgeless canvas / board / objects = non-Markdown content | **enriches UC-55** |
| Self-host sync server (AFFiNE/AppFlowy Cloud) + Anytype open API/MCP as endpoints | **enriches UC-57** |
| Object/block IDs CRDT-assigned (fine-grained) | **enriches UC-51** |
| Lossy Markdown export | links UC-59 |
---
## 9. Architecture notes for SHARD-WP-0002 (no UC)
- **Merge-model capability** (`none / git-or-text / native-CRDT`) — proposed
**thirteenth spectrum** (with Joplin's content-opacity twelfth). CRDT shards reject
external merge; shard-wiki overlays must be CRDT-aware or remain projections. (T11.)
- **Substrate taxonomy gains CRDT-replica**; **attach modes gain "local CRDT replica"
and "P2P/no-central-endpoint"** beside file-store / in-engine-host / external-API.
(T14.)
- **Endpoint model** as a capability: single URL vs self-host server vs **P2P peer set**
(Anytype) — affects how a binding is addressed and how freshness/sync work. (T14.)
- **MCP / AI-agent protocol** as a possible shard integration surface (Anytype). Note
for the extension-surface catalogue. (T14.)
- These three plus Joplin and Logseq all argue the contract must treat **substrate as a
spectrum that tools migrate along and that shard-wiki must not hard-code**.
---
## 10. Open questions (for spec / workplans)
1. For a **CRDT shard** (UC-64), does shard-wiki embed a CRDT client (Yjs/Yrs) to hold a
live replica, or only consume periodic **exports/snapshots**? The former gives
live/writable; the latter is simpler but lossy on merge timing.
2. How are **overlays** expressed against a CRDT shard — as CRDT ops (requires the
client) or as out-of-band patches reconciled later (projection-only)?
3. For **P2P shards** (UC-65), what is the "address" of the shard — a space ID + a peer/
node, a local replica path, or an invite/key? How does this fit shard attachment
config?
4. Does shard-wiki ever hold **E2EE keys** (Anytype) to project content, or only ever
treat such shards as opaque replicas/backups (UC-61)?
5. Is the **self-host sync server** (AFFiNE/AppFlowy Cloud) attached as an external API
(UC-57) or by running a co-located replica?
---
## 11. Sources
| Source | Used for |
|--------|----------|
| Anytype any-sync — Protocol overview (https://tech.anytype.io/any-sync/overview) | Local-first/P2P/E2EE CRDT; sync/file/consensus/coordinator nodes; encrypted backup |
| anyproto/any-sync (https://github.com/anyproto/any-sync) | Protocol scope; keys signed/verified; IPFS files |
| Anytype Docs (https://doc.anytype.io/anytype-docs) | Spaces, object types + relations (ontology); open API + MCP |
| toeverything/AFFiNE + DeepWiki (https://deepwiki.com/toeverything/AFFiNE) | BlockSuite (page/edgeless/DB); y-octo Yjs; OctoBase; one-data-many-views; self-host |
| AFFiNE — interactive whiteboard / blog (https://affine.pro/blog/interactive-whiteboard-app) | Edgeless canvas; CRDT offline merge |
| AppFlowy-IO/AppFlowy-Collab (https://github.com/AppFlowy-IO/AppFlowy-Collab) | Yrs CRDT; collab domain objects (documents/databases/folders); persistence |
| AppFlowy Architecture Overview — DeepWiki (https://deepwiki.com/AppFlowy-IO/AppFlowy/2-architecture-overview) | Flutter/Rust layers; pluggable storage/sync (RocksDB/IndexedDB; Supabase/AppFlowy Cloud); local vs cloud trait |
| AFFiNE vs AppFlowy vs Anytype (https://affine.pro/blog/affine-vs-appflowy-vs-anytype) | Cohort comparison framing |
Cross-references: `research/260614-notion-deep-dive/findings.md` (typed DB, hosted
contrast), `research/260614-joplin-deep-dive/findings.md` (E2EE/content opacity, sync-
daemon boundary), `research/260614-shard-spectrum-synthesis/findings.md` (the spectra
this extends), `spec/UseCaseCatalog.md` (UC-36, UC-47, UC-48, UC-51, UC-55, UC-57,
UC-58, UC-61), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
---
## 12. Traceability
- New UCs: **UC-64, UC-65**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-36, UC-47, UC-48, UC-51, UC-55, UC-57, UC-58, UC-61** (links UC-59).
- Architecture (no UC): **merge-model** capability (proposed thirteenth spectrum);
CRDT-replica substrate + P2P/no-central-endpoint attach mode; endpoint-model capability;
MCP integration surface → `SHARD-WP-0002` (T11, T14).
- Boundary recorded: Anytype/AFFiNE/AppFlowy are **CRDT local-first candidate shards**
(attach a replica/projection as pages; respect native CRDT merge; never re-drive their
sync; P2P/E2EE shards are replica-only/opaque) — not substrates, not the federation
layer (INTENT not-a-sync-daemon, graceful degradation, no silent mutation).
</content>

View File

@@ -0,0 +1,50 @@
# 260614 — Logseq deep dive (block-graph semantics on plain Markdown files)
Date: 2026-06-14
## What this is
A focused study of **Logseq** — the open-source, local-first outliner — read through
shard-wiki's lens. Logseq occupies the point the other modern tools leave empty:
**block-graph semantics (UUID-addressable, embeddable, queryable blocks) stored as plain
Markdown/Org files on disk**, with a DataScript graph **derived** from those files. It is
the bridge between **Roam** (block-graph, client-DB) and **Obsidian** (file-over-app,
page-level), and it resolves a tension the synthesis flagged: block-level addressing
*that is also git-diffable text*.
Distinctive material:
- **Architecture** — files canonical (`pages/`, `journals/` MD/Org); a DataScript graph
**derived** via an `mldoc` AST parse; a DB Worker now manages **both DataScript and
SQLite** (the file→DB migration)
- **Block-graph in the file text** — block IDs as in-file `id:: <uuid>` properties,
`((uuid))` refs, `key:: value` properties, `{{embed}}` transclusion, the outline tree
(indent/zoom/move-subtree) — all git-diffable
- **Queries** — `{{query}}` + advanced **Datalog** over the derived graph
(`logseq.DB.datascriptQuery`)
- **Extension** — plugin API (`logseq.App/Editor/DB/Git/UI/Assets/FileStorage`),
marketplace (~486 plugins); dual-attachable (file-store direct *or* in-app plugin)
- **Trajectory** — migrating from Markdown-files to a SQLite "DB graph" (a live UC-43)
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, in-file block-graph, Datalog queries, plugin API, file→DB migration, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md`
(UC-62 attach a block-graph-on-plain-files shard with in-file block IDs/properties +
outline tree + derivable query index; UC-63 serve structured queries over a file-backed
shard via a derived index the orchestrator/adapter builds); UC-32/34/43/50/51/52/55
enriched. Logged for `SHARD-WP-0002` (T11/T14/T16): a Logseq **format profile** for
file-store adapters, a **derived-query-index** capability, **substrate-migration
tolerance**, and **in-file block addressing** as the concrete T16 span-address target.
**Key takeaway recorded:** Logseq resolves the addressing-spectrum tension — block-level
addressing **and** git-diffable in-file text (`id::`) — and proves a file-backed shard
can support rich Datalog queries via a **derived** index (files canonical, graph
derived). **Boundary:** one block-graph-on-files candidate shard (the addressing sweet
spot), best attached file-store-direct with a format profile; not the federation layer;
substrate is migrating file→SQLite.
</content>

View File

@@ -0,0 +1,246 @@
# Findings — Logseq: block-graph semantics on plain Markdown files
Date: 2026-06-14
Source kind: **modern shipped product** — an open-source, local-first outliner; a
*candidate shard* occupying the point none of the prior tools do: **block-graph + file-
over-app**
Lens: shard-wiki — the convergence of block-level addressing with git-diffable files, a
derived query index over canonical files, and a live file→DB storage migration
> Why Logseq matters distinctly. The modern-tool dives staked out the corners: **Roam**
> = block-graph but client-DB / in-app-API / store-minted UUID; **Obsidian** =
> file-over-app but page-level / path-addressed (block `^id` opt-in); **Notion** =
> block-graph but closed hosted DB / external-API. **Logseq sits in the middle they
> leave empty: block-graph *semantics* (UUID-addressable, embeddable, queryable blocks)
> stored as *plain Markdown/Org files on disk you own*, with a DataScript index
> **derived** from those files.** It is the system that proves you do not have to choose
> between Roam's fine-grained addressing and Obsidian's file sovereignty — you can have
> block-level addressing *that is also git-diffable text*. That resolves a tension the
> synthesis flagged (addressing spectrum) and is the reason it earns a memo rather than
> a footnote.
Contrast set: Roam (block-DB, in-app), Obsidian (file, page-level), Notion (hosted DB),
Joplin (SQLite-local, files-on-sync). Logseq = **block-graph, files-canonical,
index-derived** — and now *also* migrating toward SQLite (§5), a live UC-43 case.
---
## 1. Core architecture — files canonical, DataScript derived
- **Storage:** plain-text **Markdown or Org-mode files on disk**`pages/<Page>.md` and
`journals/<date>.md` — fully usable even if the app disappears (local-first, open
source, ClojureScript).
- **Index:** a **DataScript** in-memory graph database, **derived** from the files: an
`mldoc` parser turns Markdown into an AST, extracts references/properties, and
transforms it into **DataScript entities** with tree-position attributes. Files are the
source of truth; the graph DB is a rebuildable projection. (In current versions a **DB
Worker** manages **both DataScript and SQLite** connections — see §5.)
- This is the **files-canonical / index-derived** architecture (Obsidian's MetadataCache,
Git's working tree) — but here the derived index is a *full Datalog-queryable graph*,
not just a metadata cache. Logseq is the strongest evidence that **a file-backed shard
can support rich structured queries via a derived index** (UC-52, UC-63).
---
## 2. The block-graph, in the file text
Everything is an **atomic block** (a bullet), individually referenceable, embeddable, and
queryable — Roam's model — but the addressing lives **in the Markdown**:
- **Block IDs are in-file properties.** When a block is referenced, Logseq writes an
`id:: <uuid>` property line into the Markdown. So the block's stable address is
**git-diffable text that survives a file copy**, not a DB-minted hidden key. This is the
**sweet spot of the addressing spectrum**: block-level (like Roam) *and* in-file/
portable (like Obsidian's `^id`), without choosing.
- **References:** `[[page]]`, **`((block-uuid))`**, `#tag` — all extracted into the graph;
block embeds `{{embed ((uuid))}}` are transclusion at block granularity (UC-32).
- **Properties:** `key:: value` lines attach typed metadata to blocks (block properties)
and pages (first-block/page properties) — **git-diffable structured data at block
granularity** (UC-34), queried by Datalog.
- **Outline tree:** a page is a **nested tree of blocks** (indent = structure), with
outliner operations — indent/outdent, move subtree, **zoom/narrow** into a block. The
page is not flat prose; it is an addressable, reorderable block tree (UC-63).
---
## 3. Queries — Datalog over the derived graph
Logseq exposes **`{{query}}`** (simple) and **advanced Datalog queries** over the
DataScript graph (and via the plugin API, `logseq.DB.datascriptQuery`). Because the graph
is derived from files, **query-defined pages** (UC-54) and structured aggregation (tasks,
tagged blocks) run over a *file-backed* store. Key lesson: query capability here is
**neither native-DB (Notion) nor a third-party plugin (Obsidian Dataview)** — it is
**built into the tool over its own files**, demonstrating that *a derived query index is a
viable adapter/orchestrator capability for file shards* (UC-63).
---
## 4. Extension surface — plugin API + marketplace
Logseq has a JS **plugin API** (marketplace ~486 plugins, `logseq/marketplace`):
- `logseq.Editor` — block/page CRUD, insert/move/update/delete, get current block/page.
- `logseq.DB`**`datascriptQuery`** (Datalog), plus DB change subscriptions.
- `logseq.App` — app-level ops/events; `logseq.UI`; `logseq.Git` (git ops on the graph);
`logseq.Assets`; `logseq.FileStorage`.
- Slash commands, block/page context menus, `provideModel`/`provideStyle`, hooks/events.
So, like Obsidian, Logseq is **dual-attachable**: (a) **file-store direct** — read the
`pages/`+`journals/` Markdown with a **Logseq format profile** (parse `id::`, `((uuid))`,
`key::`, the outline tree); (b) **in-app plugin host**`logseq.Editor` write +
`logseq.DB` query + change events for live write-through. A notable extra: a built-in
**`logseq.Git`** surface — the tool treats git as a first-class companion to the file
graph (validating the coordination journal).
---
## 5. The file→DB migration — a live UC-43
Logseq is **migrating its storage model from Markdown-files to a SQLite "DB graph"**
(the DB Worker already manages SQLite alongside DataScript; the plugin API has a distinct
"DB graph" mode with tags/classes/typed properties). This is a real-world instance of
**UC-43 (backend-swap under stable identity)**: the *same tool and graph identity* moving
from a file substrate to a DB substrate, trading git-diffability for richer typed
structure (toward the Notion/XWiki end). For shard-wiki it is both a caution (a shard's
substrate can change under it) and a confirmation that the **addressing/structure/history
spectra** are trajectories tools actually travel — an adapter keyed to a fixed substrate
will break.
---
## 6. Logseq as a shard — capability profile
| Capability | Logseq (MD-file graph) | Notes for the adapter contract |
|------------|------------------------|--------------------------------|
| Read | **yes** | file-store direct (pages/journals MD) or `logseq.Editor`/`logseq.DB` in-app |
| Write | **yes** | direct file write (format-aware) or plugin; block-level |
| Write granularity | **per-block** (in a per-file page) | finer than Obsidian (per-file), like Roam — but on files |
| Identity / addressing | **in-file block `id:: uuid` + `((uuid))`** | block-level **and** git-diffable — the addressing sweet spot (UC-51) |
| Structure | `key:: value` block/page properties; outline tree | git-diffable structured data at block granularity (UC-34) |
| History | none native; **`logseq.Git`** + files = git-native | git-friendly out of the box (adopt, not supplement) |
| Native query | **Datalog over derived DataScript** | derived index over files → delegate or rebuild (UC-52, UC-63) |
| Transclusion | **block embeds `{{embed ((uuid))}}`** | in-file, addressable (UC-32) |
| Backlinks | linked + unlinked references | derived (UC-05/18) |
| Content types | Markdown/Org + **whiteboards** (tldraw JSON) | non-Markdown spatial content (UC-55) |
| Substrate | **files now, SQLite "DB graph" emerging** | live backend-swap (UC-43) |
| Attach modes | file-store direct (format profile) · in-app plugin | dual, per-binding (UC-62) |
Verdict: **the most shard-wiki-friendly block tool** — block-graph power with file-over-
app sovereignty and git-diffable addressing/structure. Best attach: **file-store direct
with a Logseq format profile** (offline, git-native), with the plugin for live write-
through. Watch the **DB-graph migration** (UC-43).
---
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
- **Resolves the addressing tension.** Block-level addressing *can* be git-diffable
in-file (`id::`), not forced to choose Roam-DB vs Obsidian-page (UC-51, synthesis §2).
- **Confirms files-canonical / index-derived at full power** — a Datalog graph derived
from files, not just a metadata cache (UC-52, UC-63).
- **Structure as git-diffable text** (`key::`) reinforces "prefer in-text structure"
(UC-34, synthesis through-line).
- **Outliner block tree** validates the page-as-addressable-block-tree demand (UC-50,
UC-63) on a *file* backend.
### 7.2 Deliberate divergences (design bugs if conflated)
1. **One graph = one shard.** Local-first single graph; not the federation layer.
2. **The MD files carry Logseq-specific syntax** (`id::`, `((uuid))`, `key::`, outline
semantics) — a **format profile**, not plain CommonMark. A naïve Markdown reader will
mangle block IDs/properties (cf. UC-42; lighter than Notion's lossy case).
3. **The substrate is moving (file→SQLite).** Don't hard-code the file model; gate on the
substrate capability and tolerate the swap (UC-43).
4. **Whiteboards are not Markdown** — typed/opaque assets, not flattened (UC-55).
### 7.3 What Logseq teaches that shard-wiki should keep
- **In-file block addressing is the target shape** for a portable span address where the
backend cooperates — adopt `id::`-style schemes; they are git-diffable and survive copy.
- **A derived query index over files is a first-class capability** — shard-wiki can build
one over a file shard (UC-63) when the backend exposes none, or delegate when it does.
- **Expect substrate migration** — bind to capabilities, not to "it's files."
---
## 8. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-61**. New UCs **UC-62, UC-63** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a block-graph-on-plain-files shard** (Logseq-style): file-over-app Markdown carrying in-file block IDs (`id::`), block refs, and `key::` properties, with the outline tree and a derivable query index | **UC-62 (new)** |
| **Serve structured queries over a file-backed shard via a derived index** the orchestrator/adapter builds (Logseq DataScript-over-files) when the backend exposes none | **UC-63 (new)** |
| In-file block `id::` = block-level **and** git-diffable addressing (the spectrum sweet spot) | **enriches UC-51** |
| Live **file→SQLite substrate migration** under stable graph identity | **enriches UC-43** |
| Block-graph that is **files, not a DB** — the file-store variant of the block tool family | **enriches UC-50** |
| Datalog over a **derived** index built from files | **enriches UC-52** |
| `key:: value` block/page properties in-text | **enriches UC-34** |
| Whiteboards (tldraw JSON) = non-Markdown content | **enriches UC-55** |
| Block embeds `{{embed ((uuid))}}` = in-file transclusion | links UC-32 |
---
## 9. Architecture notes for SHARD-WP-0002 (no UC)
- **Format-aware file-store profiles** (already flagged by Joplin, UC-60) gain a strong
case here: a Logseq profile parses `id::`/`((uuid))`/`key::`/outline from otherwise-
plain Markdown. The contract should let a file-store adapter declare its **format
profile** (plain MD / Obsidian / Logseq / Joplin-item / Foswiki-PlainFile). (T11/T14.)
- **Derived-query-index as an orchestrator capability** (UC-63): when a file shard has no
native query engine, build a DataScript-like index over the projection; when it does
(Roam/Notion/XWiki), delegate (UC-52). Both belong in T16's navigation layer + T11.
- **Substrate-migration tolerance** (UC-43, Logseq file→SQLite): T14 binding should treat
substrate as a capability that can change under a live attachment, preserving identity.
- **In-file block addressing** is the concrete realization of T16's span-address thread —
prefer `id::`-style git-diffable IDs over DB-minted where the backend allows.
---
## 10. Open questions (for spec / workplans)
1. When the same tool offers **file** and **DB** substrates (Logseq now), does shard-wiki
prefer the file graph (git-diffable, UC-62) or the DB graph (richer typed structure),
and can one binding follow the migration (UC-43)?
2. Is the **derived query index** (UC-63) built by the **adapter** (per-shard) or the
**core orchestrator** (over the union), and is it persisted or rebuilt?
3. How much **Logseq outline semantics** (zoom, subtree move) must shard-wiki preserve vs.
present as a flat page (UC-63 vs. Markdown-first page model)?
4. Does the **Logseq format profile** round-trip overlays (write `id::`/`key::` back) or
only read them? (cf. UC-42 round-trip question.)
---
## 11. Sources
| Source | Used for |
|--------|----------|
| DeepWiki — logseq/logseq (https://deepwiki.com/logseq/logseq) | DataScript+Datalog graph, mldoc AST parse, block entities/tree, DB Worker managing DataScript **and** SQLite |
| logseq/docs (https://deepwiki.com/logseq/docs) | Plain-text MD/Org files; pages/journals; references ([[ ]], (( )), #tag); properties |
| Logseq Plugin API docs (https://plugins-doc.logseq.com/) | `logseq.App/Editor/DB/Git/UI/Assets/FileStorage`; `DB.datascriptQuery` |
| logseq/marketplace (https://github.com/logseq/marketplace) | Plugin distribution; ~486 plugins |
| kerim/logseq-db-plugin-api-skill (https://github.com/kerim/logseq-db-plugin-api-skill) | DB-graph version: tags/classes, typed properties, file→DB migration |
| pangea.app glossary — Logseq (https://pangea.app/glossary/logseq) | Local-first framing, outliner, plain-text control |
Cross-references: `research/260614-roam-deep-dive/findings.md` (block-DB contrast),
`research/260614-obsidian-deep-dive/findings.md` (file-over-app, derived index),
`research/260614-joplin-deep-dive/findings.md` (format-aware file-store profiles),
`research/260614-shard-spectrum-synthesis/findings.md` (addressing spectrum this
resolves), `spec/UseCaseCatalog.md` (UC-32, UC-34, UC-43, UC-50, UC-51, UC-52, UC-55),
`workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14, T16).
---
## 12. Traceability
- New UCs: **UC-62, UC-63**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-32, UC-34, UC-43, UC-50, UC-51, UC-52, UC-55**.
- Architecture (no UC): format-aware file-store profiles (Logseq profile); derived-query-
index capability; substrate-migration tolerance; in-file block addressing as the T16
span-address target → `SHARD-WP-0002` (T11, T14, T16).
- Boundary recorded: Logseq is **one block-graph-on-files candidate shard** (the
addressing sweet spot), best attached file-store-direct with a format profile; not the
federation layer; substrate is migrating file→SQLite (UC-43).
</content>

View File

@@ -0,0 +1,32 @@
# 260614 — Mathematica Notebooks deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T2**
## What this is
A deep dive into **Wolfram Mathematica Notebooks** — the **original computational notebook**
(1988), the ancestor Jupyter descends from. The `.nb` document **is itself a Wolfram Language
expression** (`Notebook[{Cell[…], …}]`) with **nested cell groups**, kernel evaluation
(`In`/`Out`), **structured (symbolic/graphics) results**, live `Manipulate`/`Dynamic`
widgets, and CDF as a reduced-runtime distribution projection.
## Why it matters
- Confirms the **notebook page shape (UC-84) is a genus**, not a Jupyter quirk: cells +
cached outputs + fragile counter provenance + kernel-gated re-execution predate `.ipynb`.
- Two refinements: notebooks have **nestable cell groups (an outline tree)**, not just an
ordered list; outputs can be **structured re-evaluable values**, not only MIME blobs →
adds a "structured re-evaluable value" point to the content-opacity spectrum.
- `Manipulate`/`Dynamic` join the **static-projection-impossible** end (snapshot-only) —
foreshadows Strudel (T5).
## Yield
- **No new UC** (lineage/reinforcement). Reinforces **UC-84**; enriches **UC-54, UC-55**;
links UC-34, UC-83.
## Contents
| Path | Role |
|------|------|
| `findings.md` | `.nb` expression/cell model, evaluation & provenance, CDF, capability delta vs Jupyter, INTENT mapping, UC disposition (enrichment-only), architecture notes |

View File

@@ -0,0 +1,139 @@
# Mathematica Notebooks — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T2 · **Subject:** Wolfram Mathematica
Notebooks — the `.nb` format, the cell/expression model, symbolic evaluation.
## Why this dive
Mathematica (1988) is the **original computational notebook** — the ancestor Jupyter (T3)
descends from. This dive checks whether the notebook page-model conclusions from UC-84 hold
at the genus level or need extension. It is **medium priority / lineage**: a candidate
*content type* like Jupyter, but closed/proprietary, so the expected yield is **reinforcement
of UC-84** plus a couple of distinct wrinkles (the document *is itself* a Wolfram-language
expression; symbolic, not just textual, results).
## 1. The `.nb` document model
A Mathematica notebook is itself a **Wolfram Language expression**`Notebook[{Cell[...],
Cell[...], ...}, opts]`. So the document format and the language are the *same* substrate:
- **Cells** are typed: `Input`, `Output`, `Text`, `Title`/`Section` (structure), `Code`,
with **cell groups** nesting into an **outline tree** (the document has real hierarchy, not
just a flat list like `.ipynb`).
- An **`Input` cell** holds an expression; evaluating it produces a linked **`Output` cell`**
containing the **result expression** (symbolic, graphical via `Graphics[...]`, or
typeset). The result is a **first-class expression**, not a MIME blob — it can be
re-evaluated, edited, transcluded.
- The whole `.nb` is **plain-text-serializable** (it's an expression) but verbose and
proprietary in conventions; output cells are **cached results** stored in the file.
So like Jupyter: **source + cached computed output fused in one document**, with **out-of-
band reproducibility** (kernel + package/version state). Unlike Jupyter: **nested cell
groups (a tree)** and **results that are structured Wolfram expressions** rather than MIME
bundles.
## 2. Evaluation, provenance, reproducibility
- A **kernel** evaluates input cells; `In[n]`/`Out[n]` counters mirror Jupyter's
`execution_count` and carry the **same fragility** (out-of-order evaluation, hidden
symbol/global state, kernel-version dependence). No environment capture in the file.
- **Dynamic/interactive output** (`Manipulate`, `Dynamic`) embeds **live interactive
widgets** whose state is computed on view — these have **no faithful static form** beyond a
snapshot frame (echoes Strudel T5's "live, time/interaction-based content" limit).
- **CDF (Computable Document Format)** is Wolfram's *projection-for-distribution*: a notebook
rendered with a free runtime so readers can interact without a full license — a
derivation-projection (T1) with a reduced-capability viewer.
## 3. Capability profile (delta vs Jupyter UC-84)
| Dimension | Mathematica `.nb` (delta from Jupyter) |
|-----------|----------------------------------------|
| Structure | **nested cell groups → outline tree** (richer than `.ipynb`'s flat cell list) |
| Output type | **structured Wolfram expressions** (symbolic/graphics), not MIME blobs |
| Document = language | the `.nb` **is** a Wolfram expression (format ≡ language) |
| Liveness | `Manipulate`/`Dynamic` = **interactive widgets**, snapshot-only when static |
| Opacity | proprietary serialization; results re-evaluable only with a Wolfram kernel |
| Projection-for-distribution | **CDF** = reduced-runtime interactive projection |
| Otherwise | same as UC-84: cells, cached outputs, fragile `In/Out` provenance, kernel-gated |
## 4. INTENT mapping
### Reinforcements (mostly confirms UC-84)
- **Notebook page shape (UC-84) is a genus, not a Jupyter quirk.** Mathematica predates and
matches it: cells + cached computed outputs + fragile counter provenance + kernel-gated
re-execution. Confirms the page model should carry a **notebook shape** generically (T12),
not a `.ipynb`-specific one.
- **Outputs as derivation-projection snapshots (UC-83/84).** Cached `Output` cells are
snapshots; honest treatment marks them "evaluated run N, kernel/env unguaranteed."
- **Derivation-projection for distribution (T1).** CDF is a clean "reduced-capability
interactive projection of a source" — a real-world instance of degrade-by-capability.
### Distinct wrinkles (extend the notes, not new UCs)
- **Nested cell-group outline** — the notebook page model should allow **hierarchical cell
grouping**, not just an ordered list (generalize UC-84's "ordered cells" to "ordered/
*nestable* cells"). Feeds T12.
- **Structured (non-MIME) results** — outputs can be **typed structured values** (symbolic
expressions), not only MIME blobs; reinforces UC-55's "typed asset" reading over "opaque
blob," and links the typed-record page model (UC-34) — the content-opacity spectrum needs a
"structured re-evaluable value" point, not just text↔blob.
- **Format ≡ language** — a curiosity, not actionable for us beyond noting that some shards'
document format is *the same artifact* as their computation (don't assume doc/code split).
- **Live interactive widgets** — `Manipulate`/`Dynamic` join Strudel (T5) at the
**static-projection-impossible** end: capture a snapshot frame, never imply interactivity.
### Boundaries
- Proprietary + kernel-gated → default **read/projection/snapshot**; attach the `.nb`
(or an exported form), present cached outputs as snapshots, offer a static/CDF projection;
**no kernel host** (same rule as Jupyter UC-84, GT T7).
## 5. UC disposition (enrichment-only — no new UC)
| Mechanism (findings §) | Catalog UC |
|------------------------|------------|
| Cells + cached computed outputs + fragile In/Out provenance; kernel-gated (§1, §2) | UC-84 (reinforced) |
| Nested cell groups → outline tree (richer than flat `.ipynb`) (§1) | UC-84 (enriched: nestable cells); links UC-34 |
| Output = structured re-evaluable Wolfram expression, not MIME blob (§1) | UC-55 (enriched: structured value point on opacity spectrum) |
| Input cell = computation-defined content (§1) | UC-54 (enriched) |
| `Manipulate`/`Dynamic` interactive output = snapshot-only (§2) | links UC-55, foreshadows T5 |
| CDF = reduced-runtime interactive distribution projection (§2) | links UC-83 (derivation-projection) |
Mathematica is a **lineage/reinforcement** dive — it **adds no new UC**, confirming UC-84's
notebook page shape as a genus and contributing two refinements (nestable cells; a
**structured re-evaluable value** point on the content-opacity spectrum).
## 6. Architecture notes for SHARD-WP-0002
- **T12 (page model):** generalize the notebook shape from "ordered cells" (UC-84) to
**ordered/nestable cell groups (an outline tree)**; allow code-cell outputs to be **typed
structured values**, not only MIME blobs.
- **T11/T15 (content opacity / fidelity):** add a **"structured re-evaluable value"** point
to the content-opacity spectrum between transparent-text and opaque-blob (Wolfram
expression, symbolic result) — relevant to how outputs are stored/surfaced.
- **T16 (projection):** CDF is a **reduced-capability interactive projection**; with
`Manipulate`/`Dynamic`, static projection is a **snapshot frame only** (join the
live-content limit recorded for Strudel T5).
## 7. Open questions
1. Is a **structured re-evaluable result** (Wolfram expression) modeled as a typed value in
the page model, or stored opaquely with provenance like other computed outputs? (Ties
UC-55 open-Q #10 and UC-84 Q3.)
2. Do interactive outputs (`Manipulate`, `Dynamic`, and Jupyter widgets) deserve a shared
**"interactive, snapshot-only" content marker** in the contract? (Recurs at T4/T5.)
## 8. Sources
- Wolfram documentation: notebook format (`Notebook`/`Cell` expressions), cell types &
groups, evaluation (`In`/`Out`), `Manipulate`/`Dynamic`, CDF.
- prior: `research/260614-jupyter-deep-dive/` (UC-84 notebook shape; the descendant);
`research/260614-literate-programming-deep-dive/` (derivation-projection, UC-83).
## 9. Traceability
**No new UC** (lineage/reinforcement). Reinforced: UC-84; enriched: UC-54, UC-55; links
UC-34, UC-83. Architecture cross-refs: SHARD-WP-0002 T12 (nestable cell-group outline; typed
structured outputs), T11/T15 (structured-re-evaluable-value point on the opacity spectrum),
T16 (CDF reduced-capability projection; interactive = snapshot-only).

View File

@@ -0,0 +1,15 @@
# 260614 — MojoMojo deep dive
Deep dive on **MojoMojo**: a **Perl Catalyst / DBIx::Class** DB-backed wiki — hierarchical
pages, attachments, inline (AJAX) editing, Markdown content, and **page history in
relational version tables**. The classic **MVC DB-backed** contrast to the file-store
classics: no file store, no real content API → attach by reading the **relational store
directly**.
- `findings.md` — architecture, the DB schema shape, capability profile, INTENT mapping, UC
seed (UC-81), architecture notes for SHARD-WP-0002, open questions, sources, traceability.
Catalog yield: UC-81 (attach a **DB-backed wiki with no file store/API** by reading its
relational store directly — pages + version tables — and importing DB-resident history to the
journal). Enriched UC-02/40/36/34. Feeds SHARD-WP-0002 T13 (history portability), T14
(direct-DB binding).

View File

@@ -0,0 +1,133 @@
# MojoMojo — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T8 · **Subject:** MojoMojo, a Perl
Catalyst wiki/CMS.
## Why this dive
The file-store classics (TWiki, Foswiki, Oddmuse, UseMod) keep pages as files; the modern
SaaS keep them behind APIs. MojoMojo is the **classic relational-DB-backed** wiki — a
Catalyst MVC app over **DBIx::Class** with pages and their history in **SQL tables**, and
**no file store and no first-class content API**. It anchors the *"attach by reading the
database directly"* hard case the adapter contract must account for (T13/T14).
## 1. Architecture
- **Stack:** Perl **Catalyst** (MVC web framework) + **DBIx::Class** (ORM) over a relational
DB (SQLite / PostgreSQL / MySQL). Templating via Template Toolkit.
- **Content:** **Markdown** (Text::MultiMarkdown) is the page markup — so the *body* is
Markdown, but it lives **in a DB column**, not a file.
- **Pages are hierarchical:** a **path tree** (`/parent/child`) modeled as rows with
parent/lineage relations — structure is relational, not directory-based.
- **Versioning:** each page edit creates a **new version row** (a `page_version`-style
table) — full revision history lives in **DB version tables**, with author/timestamp.
- **Features:** inline **AJAX editing**, **attachments** (stored as DB rows / blobs +
metadata), diffs, RSS feeds, full-text search, per-page permissions.
## 2. The attach problem — DB or nothing
MojoMojo exposes its content through the **web app** (HTML) and the **database**; there is
**no clean REST/GraphQL content API** and **no file store**. So a shard adapter has two
realistic paths:
1. **Direct relational read** (preferred): read the `page` + `page_version` (+ `content`,
`attachment`) tables via DBIx::Class schema — pages, the path tree, and **full history**
are all there, importable to the coordination journal (UC-41-style history import, but
from **DB version rows** rather than RCS/git).
2. **HTML scrape** (fallback): parse rendered pages — lossy, last resort.
This makes MojoMojo the **direct-DB-read** binding archetype: the canonical store is a
relational schema, and the adapter's job is to map that schema to the wiki page model +
journal.
## 3. Capability profile
| Dimension (synthesis spectrum) | MojoMojo |
|--------------------------------|----------|
| Attachment mode | **direct DB read** (relational); HTML-scrape fallback; no file store, no API |
| Addressing granularity | page (row); path tree via lineage rows |
| Content identity | DB page id; path as human key |
| Identity vs placement | row id vs path lineage (separable) |
| Structure | **relational**: page rows + parent/lineage; attachments as rows |
| History | **DB version tables** (per-edit version rows, author/timestamp) |
| Merge model | app-level last-writer; DB transactions |
| Native query | SQL over the schema (not a wiki query language) |
| Translation | **Markdown body in a DB column** — minimal translation, but extraction needed |
| Write granularity | page (row) per save |
| Operational envelope | a Perl app + its DB; direct DB access needs credentials |
| Access grant | per-page permissions in DB; app auth |
| Content opacity | transparent if you can read the DB |
| Provenance | author/timestamp on version rows |
## 4. INTENT mapping
### Reinforcements
- **Backend-neutral page model**: the body is **Markdown** — once extracted from the DB
column it maps directly; the adapter's work is **schema→page-model**, not format
translation.
- **History portability** (T13): DB **version rows** are a third history-source shape beside
git commits and RCS files — importable to the journal as discrete revisions with
author/timestamp.
- **Graceful degradation**: even with only DB read (no API), MojoMojo is a usable
read/projection/backup shard; with DB write it could be write-through, but carefully
(app invariants).
### Divergences (boundaries / notes)
- **No file store, no API** ⇒ the **direct-DB-read** binding is a first-class attach mode the
contract must name (alongside file-store, in-engine host, external-API, CRDT, P2P) — or a
sub-mode of "external store" where the medium is **a relational schema** (T14). Reading a
third-party app's DB is **coupling to its schema** (versioned, may drift across MojoMojo
versions) — a stated risk (UC-43 backend-swap analogue at the schema level).
- **Writing by direct DB** risks violating app invariants (lineage, version counters,
search index) — default to **read/projection/overlay**; write-through only with the app's
cooperation.
### What to keep
1. **Direct-DB-read as a named binding** for DB-backed engines with no file/API (UC-81),
mapping a **relational schema → wiki page model + journal**.
2. **DB version rows as a history source** for the journal (T13), beside git and RCS.
3. **Schema-coupling caution** — treat the schema as a versioned interface that can drift
(relates UC-43).
## 5. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-81 | Attach a **DB-backed wiki with no file store / no API** (MojoMojo) by reading its **relational store directly** (page + version tables), mapping schema → page model and **importing DB-resident history** to the journal | **new** |
| — | DB attach vs file attach | enrich **UC-02** / **UC-40** |
| — | DB version-table history import | enrich **UC-36** |
| — | relational page rows / lineage as structure | enrich **UC-34** |
## 6. Architecture notes for SHARD-WP-0002
- **T14 (binding):** add **direct relational read** as a binding (or external-store sub-mode
whose medium is a SQL schema) for DB-backed engines lacking a file store or API; HTML
scrape is the lossy fallback. Schema is a **versioned coupling** (drift risk, UC-43).
- **T13 (history portability):** **DB version rows** = a history source alongside git commits
and RCS revisions — import as discrete journal entries (author/timestamp).
- **T11 (capability):** "has-file-store" / "has-API" are **absent** here; "has-readable-DB"
is the capability — a sparse profile relying on schema knowledge.
## 7. Open questions
1. Does shard-wiki sanction **direct third-party DB reads** as a binding, or restrict them
(schema coupling/drift) to a documented best-effort mode? How is schema drift across
MojoMojo versions handled (UC-43)?
2. Is **write-through by direct DB** ever allowed (risking app invariants), or are DB-backed
no-API engines read/projection/overlay/backup only?
## 8. Sources
- MojoMojo — github.com/mojomojo/mojomojo; metacpan MojoMojo (Catalyst app, DBIx::Class
schema: Page / PageVersion / Content / Attachment)
- Catalyst + DBIx::Class framework docs (architecture context)
- prior: `research/260613-twiki-deep-dive/` (file-store classic contrast, UC-40/41)
## 9. Traceability
New UC **UC-81** carries the marker **⊙** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-02, UC-40, UC-36, UC-34. Architecture cross-refs:
SHARD-WP-0002 T14 (direct-DB binding), T13 (DB version-row history), T11.

View File

@@ -0,0 +1,56 @@
# 260614 — Notion deep dive (closed block-database SaaS, attached only through its API)
Date: 2026-06-14
## What this is
A focused study of **Notion** — architecture and extension model — read through
shard-wiki's lens. Notion is the **closed, hosted, schema-rich** end of the spectrum and
the dive that stress-tests shard-wiki's hardest constraints (*graceful degradation*,
*no silent remote mutation*, *Markdown-first that must degrade*).
It contrasts sharply with its neighbours:
- vs **Roam** — both are block databases with per-block UUIDs, but Roam is a *client*
DataScript DB reached only via an *in-app* JS API, while Notion is a *server* Postgres
store reached via a real *external* REST API (no in-engine hosting needed, but
rate-limited, eventually consistent, scoped by per-page grants).
- vs **Obsidian** — the mirror image of "file over app": Notion is **app over file**, a
closed hosted store with no portable files.
- vs **XWiki** — both are structured wiki-as-app-platforms; Notion is the **apex of
database-as-pages** (schema + relations + rollups + views) but closed and REST-only.
Distinctive material:
- **Data model** — everything is a **block** (UUID id, type, properties, ordered child
`content`, single `parent`); pages are blocks, **database rows are pages with a
schema**; Postgres-backed hosted SaaS
- **Databases** — typed properties, **relations** (bidirectional), **rollups**,
**formulas**, and many **views** (table/board/calendar/gallery) of one row-set
- **Extension model** — **no in-app plugin runtime**; extension = **external REST API**
integrations (+ webhooks since 2026), with ~3 rps rate limit, eventual consistency,
recursive child fetch, payload caps, and **scoped/revocable per-page access grants**
## Contents
| Path | Role |
|------|------|
| `findings.md` | Block model, databases, page-model impedance, the API-only extension model, capability profile, scoped consent, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
(UC-57 attach a closed external-API-only hosted shard with rate-limit/eventual-
consistency/scoped-grant constraints, UC-58 attach a typed database with schema +
relations + views without flattening, UC-59 lossy-aware translation with a fidelity
report); UC-31/34/36/39/50/51/52/54/56 enriched. Logged for `SHARD-WP-0002` (T14):
an **operational-envelope** capability section, **access-grant semantics**, a
**translation-fidelity** capability, and a three-way **attachment-mode taxonomy**
(file-store / in-engine-hosted / external-API-only).
**Key takeaways recorded:** Notion *enforces* "no silent remote mutation" via scoped,
revocable per-integration grants (ties the authz-in-core decision); it is the heaviest
**lossy-translation** case (proprietary blocks/rich-text, lossy Markdown export); and it
adds the **external-API-only** attachment mode (full write-through without in-engine
hosting, but bounded by rate limits and eventual consistency). **Boundary:** one
external-API candidate shard — best as projection/mirror/overlay/backup — not a
substrate and not the federation layer.
</content>

View File

@@ -0,0 +1,302 @@
# Findings — Notion: a closed block-database SaaS, attached only through its API
Date: 2026-06-14
Source kind: **modern shipped product** — a hosted block-database SaaS; a *candidate
shard* of a distinct family (closed, external-REST-API-only, schema-rich, heaviest
translation cost); has **no in-app plugin runtime**
Lens: shard-wiki — the external-API attachment mode, database-as-pages, lossy
translation/fidelity, scoped consent, and graceful degradation against a sovereign
closed backend
> Where Notion sits in the set. Roam and Notion are both **block databases with
> per-block UUIDs**, but they are opposite on access: Roam is a *client-side*
> DataScript DB reachable only through an *in-app* JS API (write-through needs an
> adapter hosted inside Roam); **Notion is a *server-side* Postgres store reachable
> through a real *external* REST API** — attachable from outside with no in-engine
> hosting, but rate-limited, eventually consistent, and **scoped by explicit
> per-integration page grants**. And against Obsidian's "file over app," Notion is the
> mirror image: **app over file** — a closed hosted store with no portable files at
> all. So Notion is the dive that stress-tests shard-wiki's hardest constraints:
> *graceful degradation*, *no silent remote mutation*, *union without erasure*, and
> *Markdown-first that must degrade* against a proprietary, schema-rich, non-Markdown
> backend.
Pairs with — and contrasts against — the Roam dive (block-DB, in-app vs external API),
the Obsidian dive (closed-hosted vs file-over-app), and the XWiki dive (both are
structured wiki-as-app-platforms; Notion is the apex of database-as-pages but closed
and REST-only).
---
## 1. Core data model — everything is a block
Notion's data model is uniform: **"Everything you see in Notion is a block. Text,
images, lists, a row in a database, even pages themselves — these are all blocks."**
Each block record holds:
| Field | Meaning |
|-------|---------|
| **id** | a randomly generated **UUID v4**, visible in the page URL — the block's stable address |
| **type** | determines rendering + which properties apply (text, to-do, heading, **page**, **child_database**, …) |
| **properties** | type-specific attributes (e.g. `title` text; for database rows, the typed property values) |
| **content** | an **ordered array of child block IDs** — nesting |
| **parent** | a single upward pointer — used for **permission inheritance** |
Blocks form a **render tree** (content + parent pointers); indentation is *structural*,
not cosmetic — it changes block relationships. **Pages are blocks**; **databases are
blocks whose children are pages**; a database **row is a page** with typed properties.
Stored in **Postgres** on Notion's servers (hosted SaaS — not local, not files).
The shard-wiki-relevant facts: (a) **per-block UUIDs** give native sub-page addressing
(UC-51); (b) the **page = block, database-row = page-with-schema** identity is a real
page-model impedance (§3); (c) the store is **closed and server-side** — reachable only
through the API (§4).
---
## 2. Databases — schema, typed properties, relations, views
A Notion **database** is a collection of pages sharing a **schema** of typed
**properties**: `title`, `rich_text`, `select` / `multi_select`, `number`, `date`,
`checkbox`, `person`, `files`, **`relation`** (a typed link to rows in another
database — shown on *both* sides, i.e. bidirectional), **`rollup`** (an aggregate over a
relation), and **`formula`** (computed). The same row-set is shown through multiple
**views****table, board (kanban), calendar, gallery, list, timeline** — each a
filtered/sorted/grouped projection of the same data.
For shard-wiki this is the **apex of "wiki page as structured record"** (stronger than
XWiki XObjects, UC-39): a page can be *bodiless typed data*, pages are joined by
**typed inter-database relations** (a graph of typed links), and "the page" is routinely
**a row in a schema** rather than prose. Notion databases also ship the ZigZag insight
commercially: **one row-set, many views/dimensions** (UC-47/48) and **filtered/linked
databases = query-defined pages** (UC-54).
---
## 3. Page-model impedance
Mapping Notion to a Markdown-first page model is the **heaviest translation case** in
the research set:
- **Block/rich-text ≠ Markdown.** Notion's rich text is an annotated-span model; many
block types (synced blocks, columns, callouts, embeds, database views) have no clean
Markdown equivalent. Notion's own **export to Markdown/CSV is lossy** (databases →
CSV, relations/rollups/formulas flatten or drop).
- **Database-row-as-page + schema** must map onto pages + sidecar metadata without
discarding the schema or the relations (extends UC-34/UC-39).
- Therefore translation must be **lossy-aware with a fidelity report** — surface what
did *not* round-trip rather than silently flattening (UC-59). This is *different from*
UC-42 (Foswiki TML↔HTML *lossless* round-trip); Notion is fundamentally lossy.
---
## 4. Extension model — there is no plugin runtime; only the API
This is the defining architectural fact for shard-wiki: **Notion has no third-party
in-app plugin/extension system.** There is no marketplace of code that runs inside
Notion (the only "in-app modding" is the *unofficial* Notion Enhancer desktop patcher,
out of scope). **Extension = external integration via the public REST API** (plus
embeds and, recently, webhooks).
The **public REST API** (`api.notion.com/v1`):
- **Resources:** `pages`, `blocks` (retrieve / append / update / delete children),
`databases` (retrieve, **query** with filters+sorts, create), `users`, `search`,
`comments`.
- **Authorization:** an **internal integration token** (workspace-owned) or **OAuth
2.0** (public integrations). Critically, **an integration only sees pages a user has
explicitly connected to it** — the user "approves the app and connects specific
pages" via *Add connections*. Access is a **scoped, revocable, per-page grant** (§6).
- **Operational constraints the adapter must encode:**
- **Rate limit** ~**3 requests/second** average (429 `rate_limited`); **no paid
increase**.
- **Eventual consistency / no live read model** — the API is not real-time; reads can
lag.
- **Recursive retrieval** — a block returns only **first-level children**; full-page
reconstruction requires walking `has_children` recursively.
- **Payload caps** — 1000 blocks / 500 KB per request; child arrays ≤ 100.
- **Webhooks** (added 2026) deliver page/database change events — a push transport for
UC-31, replacing pure polling.
- **Version history:** Notion keeps **internal page history** (retention bounded by
plan); it is **not portable** and not exposed as git — a UC-36 *supplementation* case
(like Confluence/MediaWiki), not an import case.
- **Publish:** pages can be **published to the web** as read-only public pages — an
outbound publish surface (UC-56).
Consequence: Notion is attachable **only** as an **external-API shard** — but, unlike
Roam, **no in-engine adapter hosting is needed** (the REST API is external). The cost is
the operational envelope (rate-limit, eventual consistency, scoped grant) that the
capability profile must model (UC-57).
---
## 5. Notion as a shard — capability profile
| Capability | Notion | Notes for the adapter contract |
|------------|--------|--------------------------------|
| Read | **yes (external REST)** | rate-limited (~3 rps), eventually consistent, recursive child fetch |
| Write | **yes (external REST)** | append/update/delete blocks, create pages — no in-engine host needed (vs Roam) |
| Write granularity | **block-level (fine)** | like Roam; per-block ops |
| Identity / addressing | **block UUID v4** | native sub-page addressing (UC-51), store-minted (like Roam, not in-file) |
| Structured data | **yes (apex)** | databases: schema + typed properties + relations + rollups + formulas (UC-34/39/58) |
| Native query | **yes** | database query API (filters/sorts) → delegate views (UC-52) |
| Views / dimensions | **yes** | table/board/calendar/gallery = many views of one row-set (UC-47/48/54) |
| Subscribe | **webhooks (2026)** | push events; else poll (UC-31) |
| Version history | **internal, not portable** | supplement via coordination journal (UC-36) |
| Diff / merge | **no native** | — |
| Lock | **no** | — |
| Publish | **publish-to-web** | outbound read-only (UC-56) |
| Access model | **scoped per-page grant (OAuth/token)** | explicit consent; revocable (UC-57, authz) |
| Syntax / content | **proprietary block + rich text** | **lossy** to Markdown; needs fidelity-aware translation (UC-59) |
Verdict: Notion is a legitimate but **demanding** shard — **external-API-attached,
schema-rich, fine-grained, closed**. It behaves best as a **projected / mirrored /
overlay / backup** participant; full write-through is possible but bounded by rate limits
and eventual consistency. The strongest reasons to attach it: structured databases
(UC-58) and block-UUID addressing (UC-51); the strongest cautions: lossy translation
(UC-59) and no portable history (UC-36).
---
## 6. Scoped consent and "no silent remote mutation"
Notion *enforces* one of shard-wiki's INTENT constraints at the platform level: an
integration can touch **only** the pages a user has explicitly **connected** to it, and
the grant is **revocable**. This is a clean, real-world model of **no silent remote
mutation** and of a shard granting the orchestrator **scoped, consented access** — and
it ties directly to the settled **authz-in-core / authn-delegated** decision
([[shard-wiki-auth-in-core-decision]]): authentication to Notion is delegated (OAuth /
integration token), while shard-wiki's *own* authorization decides what to do with the
granted scope. The adapter contract should treat **"scoped, revocable grant"** as a
first-class attachment property (UC-57), not an afterthought.
---
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
- **Graceful degradation** has its sharpest test here: a closed SaaS with no files, rate
limits, eventual consistency, and lossy export must still be usable as
read/projection/overlay/backup. If the adapter contract handles Notion, it handles
most things.
- **No silent remote mutation** is *modeled by the platform* (scoped grants) — Notion
validates the principle (§6).
- **Database-as-pages** validates that "wiki page" must stretch to **typed records with
relations**, not just prose (UC-34/39/58).
- **Block UUIDs** reconfirm (with Roam) that native sub-page addressing is real and
adoptable (UC-51).
### 7.2 Deliberate divergences (design bugs if conflated)
1. **Closed hosted store; no sovereignty over bytes.** shard-wiki cannot make Notion
git-native or local. It can mirror/project/overlay/back-up and **supply** a
git-addressable history (UC-36) — never claim to own Notion's store.
2. **Lossy, proprietary content.** Do **not** pretend Notion round-trips to Markdown.
Translate lossily *with a fidelity report* and preserve non-mappable elements as
provenance/sidecar (UC-59) — union without erasure includes erasure *of fidelity*
being made visible.
3. **External-API-only, rate-limited, eventually consistent.** Projection must be
**cache/poll/webhook**, not a live read model; sync is bounded — encode this in the
capability profile (UC-57). Do not design flows that assume cheap, instant, unlimited
reads.
4. **One workspace (or its granted page set) = one shard**, never the federation layer.
### 7.3 What Notion teaches that shard-wiki should keep
- Model **operational envelope** (rate limit, consistency class, payload caps,
pagination) as explicit capability-profile fields — Notion makes them unignorable.
- Model **scoped, revocable consent** as a first-class attachment property (UC-57, §6).
- Treat **translation fidelity as data**: a per-shard, per-page report of what projects
cleanly vs. degrades (UC-59) — applies beyond Notion.
- Recognize **external-REST attach** as a distinct, *preferred-where-available* mode:
full write-through without in-engine hosting (contrast Roam) — but pay the operational
envelope.
---
## 8. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-56**. New UCs **UC-57UC-59** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a closed hosted shard via its external REST API only** — no file store, no in-app runtime — honoring rate limits, eventual consistency, payload caps, and a scoped/revocable access grant | **UC-57 (new)** |
| **Attach a typed database (schema + relations + rollups + multiple views) as a shard** without flattening the schema or the inter-record relations | **UC-58 (new)** |
| **Translate a proprietary block/rich-text model to/from Markdown with an explicit fidelity report**, preserving non-mappable elements rather than silently dropping them | **UC-59 (new)** |
| Block UUIDs = store-minted native span addresses (external-API variant) | **enriches UC-51** |
| External-API block-DB attach (no in-engine host) — contrast Roam's in-app-only | **enriches UC-50** |
| Database query API + filtered/linked DBs | **enriches UC-52, UC-54** |
| Database-as-pages apex; typed records + relations | **enriches UC-34, UC-39** |
| Webhooks (2026) as a push transport | **enriches UC-31** |
| Internal-only page history, not portable | **enriches UC-36** |
| Publish-to-web outbound | **enriches UC-56** |
| Scoped, revocable per-integration grant; no silent mutation | links **UC-57** + [[shard-wiki-auth-in-core-decision]] |
---
## 9. Architecture notes for SHARD-WP-0002 (no UC)
- Add an **operational-envelope** section to the adapter capability profile: rate limit,
consistency class (live / eventually-consistent / snapshot), payload/pagination caps,
recursive-fetch requirement, push-vs-poll transport. Notion is the forcing example.
- Add **access-grant semantics**: scope (which pages), revocability, auth mode
(delegated token/OAuth) — ties the authz-in-core decision and "no silent mutation".
- Add a **translation-fidelity capability**: adapters declare and report what content
round-trips vs. degrades (UC-59); generalizes UC-42 (lossless) to the lossy case.
- **Attachment-mode taxonomy** now spans: file-store direct (Obsidian/TWiki, UC-40),
in-engine hosted adapter (Roam/XWiki, UC-38/50), and **external-API-only**
(Notion, UC-57). T14 binding should enumerate all three.
- **Database/schema/relations** as a unit (UC-58) presses the page-model spec: collection
+ schema + typed relations, not just a page.
---
## 10. Open questions (for spec / workplans)
1. Is **external-API-only with a tight rate limit** (Notion) viable for write-through at
wiki scale, or do we cap Notion at read/projection/overlay/backup by default?
2. How are **inter-database relations** (UC-58) represented in the union — as typed links
in the link graph, as a separate relation index (cf. ZigZag many-to-many), or both?
3. What is the **fidelity report** format (UC-59), and where does it surface — provenance
panel, projection metadata, reconciliation review?
4. For **scoped grants** (§6), how does shard-wiki represent partial visibility (only
*some* of a workspace's pages granted) without misrepresenting the shard as complete?
5. Do we consume Notion **webhooks** (push) or poll, given eventual consistency and the
rate limit (UC-31)?
---
## 11. Sources
| Source | Used for |
|--------|----------|
| Notion — "The data model behind Notion's flexibility" (https://www.notion.com/blog/data-model-behind-notion) | Everything-is-a-block; block record (id/type/properties/content/parent); render tree; pages/databases as blocks; Postgres |
| Notion Docs — Request limits (https://developers.notion.com/reference/request-limits) | ~3 rps rate limit, 429, payload caps, recursive first-level children |
| Notion Docs — Authorization (https://developers.notion.com/docs/authorization) | Internal token vs OAuth; integration connected to specific pages; scoped grant |
| Hookdeck / ClickUp — Notion webhooks guides (https://hookdeck.com/webhooks/platforms/guide-to-notion-webhooks-features-and-best-practices) | Webhook support (2026), page/database change events |
| Truto / Rollout — Notion API architecture & essentials (https://truto.one/blog/how-to-integrate-with-the-notion-api-architecture-guide-for-b2b-saas/) | REST endpoints (pages/blocks/databases/search), integration patterns, no in-app plugin model |
| General API knowledge — database property types, views, relations/rollups/formulas, export-to-Markdown lossiness | §2, §3 |
Cross-references: `research/260614-roam-deep-dive/findings.md` (block-DB/UUID, in-app vs
external API), `research/260614-obsidian-deep-dive/findings.md` (closed-hosted vs
file-over-app), `research/260613-xwiki-deep-dive/findings.md` (structured wiki-app-
platform), `spec/UseCaseCatalog.md` (UC-31, UC-34, UC-36, UC-39, UC-50/51/52, UC-54,
UC-56), `workplans/SHARD-WP-0002-federation-architecture.md` (T14), and the authz
decision [[shard-wiki-auth-in-core-decision]].
---
## 12. Traceability
- New UCs: **UC-57, UC-58, UC-59**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-31, UC-34, UC-36, UC-39, UC-50, UC-51, UC-52, UC-54, UC-56**.
- Architecture (no UC): operational-envelope + access-grant + translation-fidelity
capability fields; three-way attachment-mode taxonomy; database/schema/relations in the
page model → `SHARD-WP-0002` (T14).
- Decision link: scoped/revocable grant + no-silent-mutation → [[shard-wiki-auth-in-core-decision]].
- Boundary recorded: Notion is **one external-API candidate shard** — closed, hosted,
schema-rich, lossy-to-Markdown — best as projection/mirror/overlay/backup; not a
substrate, not the federation layer (INTENT graceful-degradation + no-silent-mutation).
</content>

View File

@@ -0,0 +1,51 @@
# 260614 — Obsidian deep dive (file-over-app vaults, plugin API, ecosystem signal)
Date: 2026-06-14
## What this is
A focused study of **Obsidian** — core/extension architecture and the plugin
ecosystem — read through shard-wiki's lens. Obsidian is the **most INTENT-aligned tool
in the research set**: INTENT names Obsidian vaults as a shard participant, and its
**"file over app"** model (notes are plain Markdown files in a folder you own; the app
is a lens) is the closest cousin to shard-wiki's Markdown-first, sovereignty-preserving
thesis. It is the **file-backed counterpart to the Roam dive** (same tool category,
opposite storage model: Roam = client DB + API; Obsidian = files on disk).
Distinctive material:
- **Architecture** — vault = folder of `.md` + `.obsidian/` config; files canonical;
**MetadataCache** is a *derived*, async-rebuilt index (links, embeds, tags,
frontmatter, block IDs, resolved/unresolved links, backlinks)
- **In-file, git-diffable** addressing/structure — `[[wikilinks]]`, `![[embeds]]`
(transclusion), `^block-id` span refs, YAML frontmatter/properties
- **Plugin API** — `Plugin` `onload/onunload`; `App` modules `vault` (CRUD + file
events), `metadataCache`, `workspace` (CM6, views), `fileManager.renameFile`;
`manifest.json`; community plugins, Restricted Mode
- **Ecosystem popularity → UC signal** (§7) — ranked download data mined for use cases,
per the research brief
## Contents
| Path | Role |
|------|------|
| `findings.md` | File-over-app architecture, in-file addressing, plugin API as adapter host, capability profile, **plugin-popularity → UC mapping**, derived-index lesson, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Four new use cases promoted to `spec/UseCaseCatalog.md`
(UC-53 attach a local vault with a live concurrent native editor, UC-54 query-defined
dynamic page, UC-55 carry non-Markdown content types, UC-56 outbound publish of a
projection); UC-15/28/34/36/40/51/52 enriched. Logged for `SHARD-WP-0002` (T14):
**dual attachment mode** (file-store *or* in-app plugin host), consume-native-derived-
index capability, non-Markdown content types in the page model, outbound publish,
external-writer tolerance.
**Key takeaways recorded:** (1) Obsidian validates *files-canonical / index-derived*
shard-wiki's projection model. (2) Fine-grained addressing & structure can be portable
**in-file text** (`^block-id`, frontmatter), not DB state (contrast Roam). (3) The
ecosystem is demand evidence — the #1 plugin is **drawings** (non-Markdown content),
query-as-DB (Dataview/Tasks) is top-tier but an *add-on*, and **Git is top-7** (users
bolt on the portable history shard-wiki provides natively). **Boundary:** a vault is one
file-backed candidate shard, not the federation layer and not a file-sync target
(not-a-sync-daemon).
</content>

View File

@@ -0,0 +1,324 @@
# Findings — Obsidian: file-over-app vaults, the plugin API, and what the ecosystem reveals
Date: 2026-06-14
Source kind: **modern shipped product** — a local-first Markdown vault tool; the
*most INTENT-aligned candidate shard yet* (file-backed, Markdown-first); plus an
ecosystem whose popularity is direct evidence of what users actually do
Lens: shard-wiki — file-store attachment, in-file fine-grained addressing, derived
index vs. canonical files, the engine-hosts-adapter path, and ecosystem-driven UCs
> Why Obsidian matters most. INTENT names **Obsidian vaults** explicitly as a shard
> participant, and Obsidian's **"file over app"** philosophy — your notes are plain
> Markdown files in a folder you own, the app is just a lens — is the closest cousin
> to shard-wiki's own Markdown-first, sovereignty-preserving thesis. It is also the
> **file-backed counterpart to the Roam dive**: same "personal knowledge tool"
> category, opposite storage model (Roam = client DB + API; Obsidian = files on disk).
> And uniquely among the dives, Obsidian has a **large, ranked plugin ecosystem** — so
> the user's instruction to let *plugin popularity inform the use-case catalog* is
> actionable: what people install tells us what a federated wiki must support (§7).
This dive treats Obsidian as a candidate shard (capability profile §5), studies its
plugin API as an adapter-host surface (§4), and mines the download rankings for UCs
(§7). Pairs with the Roam dive (the DB/API contrast) and the TWiki/Foswiki dives (the
file-store attachment path, UC-40).
---
## 1. Core architecture — file over app
- A **vault** is a local folder of plain **Markdown files** plus a `.obsidian/` config
directory (settings, enabled plugins, themes, workspace layout). **The files are the
source of truth**; the app holds no separate canonical database.
- Markdown is **Obsidian-flavored**: CommonMark + `[[wikilinks]]`, `![[embeds]]`
(transclusion), `^block-id` block references, `#tags`, YAML **frontmatter /
properties** (typed metadata), and callouts. Note: all addressing and structure live
**in the file text** — so they are git-diffable and portable.
- **MetadataCache** is a *derived* index: Obsidian pre-parses every file into cached
metadata (headings, links, embeds, tags, frontmatter, **blocks/block IDs**,
`resolvedLinks` / `unresolvedLinks`, backlinks). It is **rebuilt from files** and
updated **asynchronously** ("don't rely on immediate updates after a modify"). Files
canonical; index derived.
- **Canvas** files (`.canvas`, the open **JSON Canvas** format) hold spatial
arrangements — a non-Markdown content type living in the vault.
- Sync is *not* built into the format: users add Git, Obsidian Sync, or a file-sync
plugin (see §7). The vault is just files.
For shard-wiki this is the **ideal file-backed shard**: Markdown-first, on disk,
git-friendly, with a derived index model that mirrors shard-wiki's own
projection/derived-view philosophy (the cache is to Obsidian what projections and union
BackLinks are to shard-wiki — computed, not canonical).
---
## 2. Addressing, links, transclusion — in-file and git-diffable
- **Links:** `[[Page]]` and `[[Page#Heading]]` / `[[Page#^block-id]]`; the resolved/
unresolved link sets and **backlinks** are computed in MetadataCache — Obsidian's
shipped answer to union BackLinks (UC-05/UC-18), but per-vault.
- **Fine-grained addressing:** a **`^block-id`** suffix on a paragraph (and heading
anchors) gives a stable sub-page address — *but it lives in the Markdown text*, not in
a database. This is the **git-diffable, portable variant** of Roam's `:block/uid`
(which lives in a DataScript DB). Important contrast for UC-51: native span IDs can be
**text-embedded** (Obsidian) or **store-minted** (Roam); the text-embedded form is
more portable and survives a file copy, but is opt-in (the user must add the `^id`).
- **Transclusion:** `![[Page]]`, `![[Page#Heading]]`, `![[Page#^block-id]]` embed live
content by reference — shipped transclusion at page/section/block granularity (UC-32),
again stored as plain text.
---
## 3. Structured data — frontmatter as git-diffable records
YAML **frontmatter / properties** make each note a lightweight typed record
(`tags`, `aliases`, arbitrary keys). This is the **git-diffable structured-data**
variant (cf. TWiki `%META%` inside the file, UC-34) — structure lives in the file, not a
DB. The **Dataview** plugin (§7) turns frontmatter + inline fields + the link graph into
a queryable database — but that query power is an *ecosystem add-on*, not core.
---
## 4. Extension architecture — the plugin API as an adapter host
Obsidian plugins are TypeScript, shipped as `manifest.json` + `main.js` (+ optional
`styles.css`):
- **Manifest:** `id`, `name`, `version`, `minAppVersion` (required); `author`,
`description`, `isDesktopOnly`, `fundingUrl` (optional). Community plugins are
GitHub-hosted and listed in the `obsidian-releases` repo; installed from inside the
app; gated by **Restricted Mode** (plugins run arbitrary code — Node/Electron on
desktop).
- **Plugin lifecycle:** extend `Plugin`, implement **`onload()` / `onunload()`**; use
`registerEvent` / `registerDomEvent` / `registerInterval` for auto-cleanup.
- **The `App` object** (global singleton, `this.app`) exposes four modules:
- **`vault`** — file CRUD: `read` / `cachedRead`, `create`, `modify`, `process`,
`delete`, `rename`, `getFiles`, low-level `adapter`, and **events**
(`on('create'|'modify'|'delete'|'rename')`) for live file-watching.
- **`metadataCache`** — `getFileCache`, `frontmatter`, `links`, `embeds`, `tags`,
`headings`, `blocks`, `resolvedLinks` / `unresolvedLinks`, `getBacklinksForFile`.
- **`workspace`** — panes/leaves/views; `registerView`, `registerEditorExtension`
(CodeMirror 6), `registerMarkdownPostProcessor`.
- **`fileManager`** — high-level ops; **`renameFile`** updates inbound links (use it
instead of raw vault rename to keep link integrity).
- **Registrations:** `addCommand`, `addRibbonIcon`, `addStatusBarItem`,
`addSettingTab`, `registerView`, `registerEditorExtension`,
`registerMarkdownPostProcessor`.
Two consequences for shard-wiki:
1. **A vault is attachable two ways.** (a) **File-store direct attach** (UC-40) — read
the folder as a folder shard; no plugin, offline-capable, git-native; good for
read/projection/overlay. (b) **In-app plugin host** (UC-38) — a shard-wiki adapter
*as an Obsidian plugin* drives `vault` write + `metadataCache` reads + live file
events for high-fidelity write-through. This dual mode is a cleaner story than Roam
(which only offers the in-app path).
2. The **MetadataCache API is a ready-made derived-view source** — backlinks, tags,
block IDs, resolved links — an adapter can consume instead of reparsing.
---
## 5. Obsidian as a shard — capability profile
| Capability | Obsidian | Notes for the adapter contract |
|------------|----------|--------------------------------|
| Read | **yes** | direct file read (folder shard) or `vault.read`/`cachedRead` in-app |
| Write | **yes** | direct file write, or `vault.modify`/`create`/`process` in-app; per-file granularity |
| Write granularity | **per-file (page)** | the natural shard-wiki granularity — contrast Roam (block) and TiddlyWiki (whole-file-single-vault) (UC-35) |
| Identity / addressing | **path + in-file `^block-id` / headings** | git-diffable, portable, **opt-in** span IDs (UC-51 text-embedded variant) |
| Transclusion | **yes** | `![[...]]` page/section/block embeds (UC-32) |
| Backlinks / links | **yes (derived)** | MetadataCache resolved/unresolved links + backlinks (UC-05/18) |
| Structured data | **yes (in-file)** | YAML frontmatter/properties; queryable via Dataview plugin (UC-34) |
| Native query | **plugin only** | Dataview — *not core*; informs UC-52 (query is adapter/plugin-provided, not assumable) |
| Version history | **none native** | users add the Git plugin (top-7!) → validates the coordination journal (UC-36) |
| Diff / merge | **none native** | git-level if Git plugin / repo-backed |
| Lock | **no** | local-first, single-user assumption |
| Publish | **via Obsidian Publish / Quartz / Digital Garden** | outbound publish of a projection (UC-56) |
| Syntax | **Obsidian-flavored Markdown** | close to CommonMark; wikilinks/embeds/callouts/`^id` need adapter awareness (cf. UC-42) |
| Non-Markdown content | **Canvas (JSON Canvas), attachments, Excalidraw** | typed/opaque assets in the vault (UC-55) |
Verdict: Obsidian is the **cleanest file-backed, Markdown-first shard** — the
file-store family (TWiki/Foswiki/DokuWiki, UC-40), and the *reference* personal-vault
shard INTENT already names. Its history/query/sync gaps are exactly what users fill with
plugins (§7) — i.e. exactly what shard-wiki offers as orchestration.
---
## 6. The derived-index lesson (architecture)
Obsidian's **files-canonical / MetadataCache-derived** split is independent validation
of shard-wiki's core stance: the link graph, backlinks, tags, and block index are
**computed projections over canonical files**, rebuilt on change, eventually consistent
("updated asynchronously"). shard-wiki's union BackLinks, projections, and derived views
(UC-05, UC-17UC-20) should likewise be **derived and rebuildable**, never a second
source of truth. The cache-invalidation/async-update caveat is a real design note for
projection freshness (UC-31).
---
## 7. Ecosystem popularity → use-case signal (the user's ask)
All-time download ranks (obsidianstats.com), read as *demand evidence*:
| Plugin (downloads) | What users do with a vault | shard-wiki signal → UC |
|--------------------|----------------------------|------------------------|
| **Excalidraw** (6.4M) | drawings/diagrams as first-class content | non-Markdown content types in a shard → **UC-55 (new)** |
| **Templater** (4.6M), **QuickAdd** (1.9M) | templated note creation | blueprint pages → **UC-15 (enriched)** |
| **Dataview** (4.4M) | query the vault (files+frontmatter+links) as a DB | query-defined dynamic pages → **UC-54 (new)**; query is a *plugin***UC-52 (enriched)** |
| **Tasks** (3.6M) | aggregate to-dos across all notes | cross-page typed-item aggregation → **UC-54** |
| **Advanced Tables** (2.9M) | structured tables in Markdown | in-file structured data → UC-34 |
| **Calendar** (2.8M) | daily-note navigation | temporal dimension (ZigZag `d.recent`) → UC-17 |
| **Git** (2.7M, top-7) | version control + sync on the vault | **history is a bolt-on, not native** → validates coordination journal → **UC-36 (enriched)** |
| **Kanban** (2.4M) | board view of Markdown cards | alternate projection/view of pages → UC-47/48 |
| **Remotely Save** (2.0M) | sync vault to S3 / Dropbox / OneDrive / GDrive / **WebDAV** | demand to federate a vault to heterogeneous remote stores → reinforces INTENT WebDAV/Nextcloud shards (but shard-wiki is *not* a file-sync daemon — attach as shards, don't mirror) |
| **Omnisearch** (1.6M) | search across notes, PDFs, images | union full-text search over mixed content → UC-20 |
| **Importer** (1.4M) | import from Evernote / Notion / Bear / Apple Notes | carry-forward/import from foreign tools → **UC-28 (enriched)** |
| **Smart Connections** (1.0M), **Copilot** (1.5M), **Claudian** (938K) | AI over the vault (related notes, chat) | mostly out of scope; "related notes" ≈ link/equivalence discovery (UC-46) — note the trend, don't build it into core |
| **Tag Wrangler** (981K) | rename/merge tags vault-wide | namespace/tag refactor across the union → UC-22 |
| **Linter** (927K) | normalize frontmatter/formatting | content normalization → fingerprint basis for equivalence (UC-46) |
| **Admonition** (922K) | callouts | Obsidian-flavored syntax → translation awareness (UC-42) |
Headline reads:
- The **#1 plugin is drawings** (Excalidraw) — users keep **non-Markdown content** in
"Markdown" vaults. A wiki orchestrator that assumes pure Markdown will mishandle the
most popular real-world use → **UC-55**.
- **Query-the-vault-as-a-DB** (Dataview #3 + Tasks #4) is a top use — but it is an
*add-on*, confirming query is an **adapter/plugin capability** (UC-52), and motivating
**query-defined dynamic pages** (UC-54).
- **Git is top-7** — users manually bolt portable version control onto their vault. This
is *direct demand* for what shard-wiki provides natively as the coordination journal
(UC-36).
- **Sync-to-anywhere** (Remotely Save #11) shows demand to connect a vault to S3/WebDAV/
cloud — the heterogeneous backends INTENT targets — while reminding us of the
*not-a-file-sync-daemon* boundary.
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **File-over-app == shard sovereignty + Markdown-first.** Obsidian is the living proof
that a serious tool can be "just files you own"; it is the model shard for INTENT's
Obsidian/local-folder participants.
- **Files canonical, index derived** validates shard-wiki's projection/derived-view
architecture (§6).
- **In-file, git-diffable structure & addressing** (frontmatter, `^block-id`, embeds)
shows fine-grained addressing and structured data can be **portable text**, not DB
state — friendlier to the coordination journal than Roam's DB.
### 8.2 Deliberate divergences (design bugs if conflated)
1. **A vault is one shard, not the federation.** Local-first, single-vault,
single-user; do not model the union as "one big vault."
2. **Not a file-sync daemon.** The Remotely Save popularity is tempting; shard-wiki must
stay **wiki-page-semantic** — attach remote stores as shards, never generic file
mirroring (INTENT constraint).
3. **`.obsidian/` is opaque app config, not page content.** The adapter must exclude/
treat it as shard-local config, not project it as pages.
4. **Obsidian-flavored Markdown ≠ CommonMark.** Wikilinks/embeds/callouts/`^id` need
adapter awareness; closer than Roam's outline, but still a translation surface
(UC-42).
5. **Plugins run arbitrary code.** If a shard-wiki adapter is hosted *as* an Obsidian
plugin (UC-38 path), it inherits Electron/Node trust — a deployment/security note,
not a core concern.
### 8.3 What Obsidian teaches that shard-wiki should keep
- Offer **both attachment modes** for a vault: zero-config **file-store direct attach**
(read/projection/overlay) and an optional **in-app adapter** (write-through, live
events). Graceful degradation by default, fidelity on opt-in.
- **Consume a backend's existing derived index** (MetadataCache) instead of reparsing,
when the shard exposes one.
- **Embrace non-Markdown content types** (Canvas/attachments/drawings) as typed or
opaque assets with provenance — don't flatten them away (UC-55, extends UC-34).
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-52**. New UCs **UC-53UC-56** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a local-first Markdown vault as a file-backed shard with a live concurrent native editor** — file-watch, tolerate the vault's own app writing concurrently, treat `.obsidian/` as opaque | **UC-53 (new)** |
| **Define a page as a live query over the union** (Dataview/Tasks pattern — query-defined dynamic page) | **UC-54 (new)** |
| **Carry non-Markdown content types** (Canvas/JSON Canvas, drawings, attachments) as typed/opaque assets with provenance | **UC-55 (new)** |
| **Publish a curated projection of the union or a shard to an external read-only target** (Obsidian Publish / Quartz / Digital Garden) | **UC-56 (new)** |
| Local Markdown vault = cleanest file-backed direct-attach; dual attach (file-store *or* in-app plugin host) | **enriches UC-40** (and UC-02, UC-38) |
| In-file `^block-id` / heading anchors = git-diffable, portable, opt-in span IDs | **enriches UC-51** |
| YAML frontmatter/properties = git-diffable in-file structured data | **enriches UC-34** |
| Git plugin (top-7) = users bolt on portable history → demand for the coordination journal | **enriches UC-36** |
| Query is a popular plugin (Dataview), not core → query is adapter/plugin-provided | **enriches UC-52** |
| Importer plugin = import from foreign tools | **enriches UC-28** |
| Templater/QuickAdd = templated creation | **enriches UC-15** |
| `![[...]]` embeds = in-file transclusion | links UC-32 |
| MetadataCache backlinks/links | links UC-05/UC-18 |
---
## 10. Architecture notes for SHARD-WP-0002 (no UC)
- **Dual attachment mode** for a single backend (file-store *and* in-engine adapter)
should be first-class in the adapter contract / T14 binding — Obsidian is the clean
example (Roam was in-app-only; TWiki had file-vs-API but not a personal vault).
- **"Consume native derived index"** as a capability: a shard may expose its own link/
backlink/block index (Obsidian MetadataCache, Roam Datalog) the orchestrator can read
instead of reparsing (ties UC-52).
- **Non-Markdown content types** (UC-55) push on the wiki page model: pages vs. typed
assets vs. opaque blobs — a page-model spec decision, not just adapter config.
- **Outbound publish** (UC-56) formalizes the `publish` capability from INTENT's list as
a projection target, complementing inbound static-export attach (UC-37).
- Concurrent-native-editor (UC-53) needs the contract to express **external-writer
tolerance** (file-watching, re-projection, conflict-with-live-app) — distinct from
multi-user write conflicts.
---
## 11. Open questions (for spec / workplans)
1. For a vault attached **both** ways (file-store + in-app plugin), which is
authoritative, and how do they reconcile (the in-app adapter sees MetadataCache; the
file-store path sees raw bytes)?
2. Is **UC-54 (query-defined dynamic page)** a core page type, an adapter feature, or a
reference-UI/plugin concern? (Mirrors the Roam/Dataview "views are queries" question
and catalog Q7.)
3. How does shard-wiki represent **non-Markdown content** (UC-55) — typed asset with a
Markdown stub, opaque blob with provenance, or a pluggable content-type registry?
4. Does **outbound publish** (UC-56) belong in core or as a publish-adapter family, and
how does it interact with overlays/projection freshness?
5. How is **concurrent native editing** (UC-53) surfaced — optimistic re-projection,
advisory lock, or overlay-onto-moving-target?
---
## 12. Sources
| Source | Used for |
|--------|----------|
| Obsidian API — App Architecture (https://www.mintlify.com/obsidianmd/obsidian-api/concepts/app-architecture) | App singleton, four modules, this.app |
| obsidianmd/obsidian-api README (https://github.com/obsidianmd/obsidian-api/blob/master/README.md) | Plugin lifecycle, Vault/MetadataCache/Workspace/FileManager APIs, registrations, manifest fields |
| DeepWiki — Vault and File System; MetadataCache and Link Resolution (https://deepwiki.com/obsidianmd/obsidian-api) | Vault CRUD + events; MetadataCache parsed elements, resolved/unresolved links, backlinks |
| obsidianstats.com — Most Downloaded Plugins (https://www.obsidianstats.com/most-downloaded) | All-time download ranks + counts for the popularity → UC mapping (§7) |
| Obsidian — "The future of plugins" (https://obsidian.md/blog/future-of-plugins/) | Plugin distribution model, restricted mode context |
| Obsidian Help / JSON Canvas (jsoncanvas.org) | Vault format, `.obsidian/` config, Canvas open format |
Cross-references: `research/260614-roam-deep-dive/findings.md` (DB/API contrast, native
span IDs, query delegation), `research/260613-twiki-deep-dive/findings.md` (file-store
attach UC-40), `spec/UseCaseCatalog.md` (UC-02, UC-05, UC-15, UC-18, UC-28, UC-31,
UC-32, UC-34, UC-36, UC-38, UC-40, UC-51, UC-52), `workplans/SHARD-WP-0002-federation-architecture.md` (T14).
---
## 13. Traceability
- New UCs: **UC-53, UC-54, UC-55, UC-56**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-15, UC-28, UC-34, UC-36, UC-40, UC-51, UC-52** (and links UC-02,
UC-05, UC-18, UC-32, UC-38).
- Architecture (no UC): dual attachment mode; consume-native-derived-index capability;
non-Markdown content types in the page model; outbound publish; external-writer
tolerance → `SHARD-WP-0002` (T14).
- Distinctive artifact: **plugin-popularity → UC mapping** (§7) — ecosystem demand
evidence, per the research brief.
- Boundary recorded: an Obsidian vault is **one file-backed candidate shard** (the
cleanest, INTENT-named), mapped into the Markdown-first page model; not the federation
layer and not a file-sync target (INTENT Stability Note + not-a-sync-daemon).
</content>

View File

@@ -0,0 +1,13 @@
# 260614 — Oddmuse deep dive
Deep dive on **Oddmuse**: a deliberately minimal wiki — a **single Perl CGI script** over
**plain-text page files** (one file per page, old revisions in a `keep/` dir), no database.
The low-complexity **file-store floor** — useful as the **minimal-adapter / graceful-
degradation baseline** of the capability profile.
- `findings.md` — the minimal model, storage layout, capability profile, INTENT mapping, UC
seed (UC-82), architecture notes for SHARD-WP-0002, open questions, sources, traceability.
Catalog yield: UC-82 (attach a **minimal flat-file wiki** — plain-text page files + a simple
revision dir — as the **graceful-degradation baseline / minimal capability-profile floor**).
Enriched UC-40/01/36/41. Feeds SHARD-WP-0002 T11 (minimal capability profile).

View File

@@ -0,0 +1,124 @@
# Oddmuse — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T7 · **Subject:** Oddmuse, Alex Schroeder's
minimal single-script wiki (EmacsWiki, Community Wiki).
## Why this dive
After the structure/graph/SaaS far-ends, Oddmuse anchors the **opposite** corner: the
*minimal* file-store wiki. It is the reference for shard-wiki's **graceful-degradation**
promise — *a limited backend must still be usable* — and defines the **floor** of the
capability profile (T11): what the simplest possible real wiki looks like as a shard.
## 1. The minimal model
- **One Perl CGI script** (`wiki.pl`) is the whole engine — drop it on any CGI host. No
framework, no database, minimal dependencies.
- **Plain-text page files**: each page is a text file in a **page directory** (`page/`),
with a small header of metadata and the body; **old revisions** are kept in a **`keep/`**
directory (recent history retained, older optionally expired).
- **Locking** via lock files; edits append a new keep-revision.
- **Markup:** simple wiki markup; **CamelCase** and **free links** `[[Like This]]`;
InterWiki/near-links; tags and "clusters."
- **No DB, no API** (beyond the CGI itself); content *is* the files on disk.
## 2. The shard view — the file-store floor
Because pages are **plain-text files on disk**, Oddmuse is **trivially attachable** as a
**file-store shard** even though the engine offers nothing fancy:
- Read the `page/` files → pages (parse the tiny header + body).
- Read `keep/` → recent revision history (import to the journal; note it may be **truncated**
— older revisions can be expired, so history is *partial*).
- Write = write a page file + a keep-revision (respecting the lock) — but the engine's own
invariants (indexes) mean **write-through is best done via the engine or carefully**.
This is the **minimal capability profile**: file-store, page-granularity, plain-text,
possibly-truncated history, no query, no structured fields, open editing. Everything richer
in the synthesis matrix is measured *against this floor*.
## 3. Capability profile (the floor)
| Dimension (synthesis spectrum) | Oddmuse |
|--------------------------------|---------|
| Attachment mode | **file-store** (plain-text files); CGI, no API |
| Addressing granularity | page = file |
| Content identity | page name = filename |
| Identity vs placement | name-bound |
| Structure | none beyond tags/clusters; flat page space |
| History | **`keep/` revisions — recent, possibly truncated** |
| Merge model | lock-file; last-writer |
| Native query | none |
| Translation | simple wiki markup (→ Markdown translation needed) |
| Write granularity | page (file) |
| Operational envelope | a CGI script; tiny |
| Access grant | open by default (optional password) |
| Content opacity | transparent text |
| Provenance | minimal (timestamp, optional username) |
## 4. INTENT mapping
### Reinforcements
- **Graceful degradation** (INTENT): Oddmuse is the *definition* of the limited-backend case
— still a perfectly good read/projection/overlay/backup shard via its files.
- **Union without erasure**: even a minimal shard contributes pages with provenance; its
**truncated history** must be surfaced honestly (don't imply full history when `keep/` is
partial).
- **Open wiki** (UC-01): Oddmuse is open-editing by default — the c2-era ethos.
- **Markdown-first but backend-neutral**: its wiki markup needs translation to the
Markdown-first page model (UC-42-style), a small lossy step.
### Divergences (boundaries / notes)
- **Partial history**: `keep/` may expire old revisions — the journal import must record that
history is **truncated/partial**, not complete (a freshness/provenance honesty point).
- **Minimal profile** means many capabilities are simply **absent** — the adapter advertises
a sparse profile; the orchestrator must not assume query/structure/locking semantics it
doesn't have (T11 capability-awareness in its purest form).
### What to keep
1. **Minimal flat-file wiki as the graceful-degradation baseline** (UC-82): plain-text files
+ simple revision dir = the floor every richer profile extends.
2. **Honest partial-history reporting** when a shard's revision store is truncated.
3. **Sparse capability profile** handling — absence of a capability is first-class (T11).
## 5. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-82 | Attach a **minimal flat-file wiki** (plain-text page files + a simple revision dir, Oddmuse) as the **graceful-degradation baseline / minimal capability-profile floor**, surfacing **partial history** honestly | **new** |
| — | plain-text file-store at the simple end | enrich **UC-40** |
| — | open-editing wiki | enrich **UC-01** |
| — | `keep/` plain-text revision history (possibly truncated) | enrich **UC-36** / **UC-41** |
## 6. Architecture notes for SHARD-WP-0002
- **T11 (capability model):** Oddmuse defines the **minimal/floor profile** — file-store,
page granularity, plain-text, **partial** history, no query/structure. Validate that the
capability vocabulary can express **absence** cleanly and that the orchestrator degrades
to read/projection/overlay/backup against it.
- **History portability (T13):** `keep/` revisions import as journal entries but may be
**truncated** — record completeness as metadata (full vs partial history).
## 7. Open questions
1. How does shard-wiki represent a shard with **partial/truncated history** in the journal
and provenance UI (UC-24) — explicit "history begins at" marker?
2. Is write-through to a minimal CGI wiki (write page + keep-revision under its lock) ever
sanctioned, or read/projection/overlay/backup only by default?
## 8. Sources
- oddmuse.org — Oddmuse wiki (single-script install, `page/` + `keep/` storage, markup,
CamelCase/free links, clusters/tags)
- EmacsWiki / Community Wiki (Oddmuse in production)
- prior: `research/260608-c2-wiki-origins/` (open-wiki ethos); `research/260613-twiki-deep-dive/`
(file-store + RCS contrast)
## 9. Traceability
New UC **UC-82** carries the marker **⊚** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-40, UC-01, UC-36, UC-41. Architecture cross-refs:
SHARD-WP-0002 T11 (minimal/floor profile), T13 (partial-history import).

View File

@@ -0,0 +1,30 @@
# 260614 — Processing & Processing.js deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T4**
## What this is
A deep dive into **Processing** (creative coding) and **Processing.js / p5.js** (browser-run
sketches): **the sketch *is* the document** — a program whose presentation is **live visual
output rendered at view time** in the browser, with **no cached output** by default.
## Why it matters
- The cleanest **program-as-page** case: canonical content = **source text**, presentation =
**executable render** (no input/output cells, no prose envelope) — sharpens the page model
(T12/T15) and UC-54/55.
- Adds a **view-time** variant to derivation-projection (the render runs **in the viewer,
continuously**) and a **continuity** facet (one-shot vs continuous/interactive); continuous
→ static is a **snapshot frame/recording** on the live↔snapshot axis (T6).
- "Execute/render in the viewer" = an explicit **capability + trust/sandbox** surface.
## Yield
- **No new UC** (enrichment / design prior art). Enriches **UC-54, UC-55**; links UC-83,
UC-84, UC-35.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Program-as-page, view-time/live render, no-cached-output, capability+trust, INTENT mapping, UC disposition (enrichment-only), architecture notes |

View File

@@ -0,0 +1,114 @@
# Processing & Processing.js — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T4 · **Subject:** Processing (creative
coding) and Processing.js / p5.js — **the sketch *is* the document**, rendered live at view
time in the browser.
## Why this dive
Low-priority, enrichment-focused. Processing tests a page-model edge the notebooks (T2/T3)
didn't: a page that is **wholly a program whose presentation *is* its running output** — no
separate input/output cells, no prose envelope. The defining question: what is a "page" when
**the rendered form is a live computation evaluated at view time**? It feeds UC-54
(computation-defined content) and UC-55 (non-Markdown / executable content) and sharpens the
**live↔snapshot** axis named at T6.
## 1. Program-as-page
- A **Processing sketch** is a program (`setup()` + `draw()`) whose output is a **canvas of
visual/animated/interactive content**. There are no "output cells" — **the program's
execution is the content**.
- **Processing.js** (Resig, 2008; now largely **p5.js**) runs sketches **client-side in the
browser** on `<canvas>`. The page ships the **source**; the **rendering happens at view
time** in the reader's browser — no server, no pre-rendered artifact required.
- So the durable artifact is **text (the sketch source)**; the *presentation* is a **live,
view-time derivation** of that source (a derivation-projection, T1) — with the twist that
the derivation runs **in the viewer**, continuously (animation/interaction), not once.
## 2. The view-time / live-render dimension
This adds a wrinkle beyond Jupyter's *captured* outputs (UC-84) and Mathematica's CDF:
- **No captured output at all by default** — unlike a notebook, a sketch typically stores
**only source**; there is nothing cached. The output exists only when executed.
- **Continuous & interactive** — `draw()` loops; mouse/keyboard drive it. The output is
**time-based and interaction-based**, so any static capture is a **single frame** (or a
recording) — it cannot represent the artifact faithfully (shared limit with Mathematica
`Dynamic` and, more extremely, Strudel T5).
- **Client-side execution = a capability + trust surface** — running arbitrary sketch code in
the viewer is an execution capability with sandboxing concerns; shard-wiki must treat
"render live in the viewer" as an explicit, gated capability, never an automatic behavior.
## 3. INTENT mapping (enrichment-only — no new UC)
### Reinforcements / refinements
- **Executable-as-page (UC-54/55).** Processing is the cleanest **"the whole page is the
program"** case: content = source text, presentation = view-time live render. Strengthens
the page model's need to represent **executable content whose rendered form is derived at
view time**, distinct from notebooks (which interleave cells + captured outputs).
- **Derivation-projection, view-time variant (T1/UC-83).** The render is a derivation-
projection that runs **in the viewer, continuously** — extends the projection model: a
derivation may be *materialized ahead* (CDF/nbconvert) or *run at view time* (sketch).
- **Live↔snapshot axis (T6).** With **no cached output** and **continuous/interactive**
rendering, the only static form is a **snapshot frame or recording** — a concrete point on
the live↔snapshot axis; honest treatment offers the snapshot and **marks it as a frame of a
live artifact**, never as the artifact.
- **Capability + trust gating.** "Execute in the viewer" is an explicit capability with a
sandbox/trust boundary — mechanism-over-policy: whether to run, sandbox, or only snapshot
stays configurable.
### Boundaries
- shard-wiki is **not a sketch runtime**; default is **attach the source + offer a captured
snapshot/recording**; live in-viewer rendering is a gated capability. Source is canonical;
the render is a degrading, view-time projection.
## 4. UC disposition (enrichment-only)
| Mechanism (findings §) | Catalog UC / thread |
|------------------------|---------------------|
| Sketch = program-as-page; presentation = view-time live render (§1) | UC-54 (enriched: executable/view-time content), UC-55 (enriched: non-Markdown executable) |
| Render = derivation-projection run **in the viewer, continuously** (§1, §2) | links UC-83 (view-time variant) |
| No cached output; continuous/interactive → static = snapshot frame only (§2) | links UC-84, live↔snapshot axis (T6) |
| Client-side execution = capability + trust/sandbox surface (§2) | links UC-35 (capability-awareness) |
**No new UC** — Processing is design prior art reinforcing executable-as-page; its
contribution is the **view-time, no-cached-output, continuous-render** point on the
projection/liveness model.
## 5. Architecture notes for SHARD-WP-0002
- **T12/T15 (page model):** represent **program-as-page** — a page whose canonical content is
**source text** and whose presentation is an **executable render**; no inherent cached
output (contrast notebooks). Non-Markdown executable content type.
- **T16 (projection):** add a **materialization timing** facet to derivation-projection:
**ahead-of-time** (CDF/nbconvert/static HTML) vs **at view time** (sketch render); and a
**continuity** facet: one-shot vs continuous/interactive. Continuous → static is a snapshot
frame/recording on the live↔snapshot axis (T6).
- **T11 (capabilities):** "execute/render in the viewer" is a capability with a **trust/
sandbox** sub-concern; default off → snapshot.
## 6. Open questions
1. Is **view-time execution** ever offered (sandboxed in-viewer render), or does shard-wiki
always degrade program-as-page to a captured snapshot/recording? (Capability/trust policy.)
2. Should **materialization timing** (ahead-of-time vs view-time) and **continuity**
(one-shot vs continuous) be explicit projection metadata, alongside the live↔snapshot
axis? (Recurs at T5.)
## 7. Sources
- `processing.org`; **p5.js** (`p5js.org`); Processing.js (Resig) history; `<canvas>` /
client-side rendering model.
- prior: `research/260614-jupyter-deep-dive/` (captured vs no-cached output, UC-84);
`research/260614-mathematica-deep-dive/` (`Dynamic`/CDF, snapshot-only);
`research/260614-squeak-pharo-deep-dive/` (live↔snapshot axis).
## 8. Traceability
**No new UC** (enrichment / design prior art). Enriched: UC-54, UC-55; links UC-83 (view-time
derivation-projection), UC-84 (no-cached-output contrast), UC-35 (execute capability +
trust). Architecture cross-refs: SHARD-WP-0002 T12/T15 (program-as-page, source-canonical/
render-derived), T16 (materialization-timing + continuity facets on the live↔snapshot axis),
T11 (view-time-execute capability + sandbox).

View File

@@ -0,0 +1,15 @@
# 260614 — Salesforce Quip deep dive
Deep dive on **Salesforce Quip**: a closed SaaS of **live collaborative documents** with
**embedded spreadsheets and live apps**, a REST API (HTML import/export), and
**Salesforce-tied identity/permissions**. A hosted-collab contrast to Notion: the
document+spreadsheet hybrid under enterprise auth.
- `findings.md` — the doc/spreadsheet model, REST API, enterprise auth, capability profile,
INTENT mapping, UC seed (UC-80), architecture notes for SHARD-WP-0002, open questions,
sources, traceability.
Catalog yield: UC-80 (attach a SaaS live-doc shard whose pages **mix prose + embedded live
structured objects** — spreadsheets/apps — via REST with lossy HTML import/export, under
enterprise identity). Enriched UC-57/55/58/36/06. Feeds SHARD-WP-0002 T11 (capability /
content opacity), T14 (external-API binding).

View File

@@ -0,0 +1,148 @@
# Salesforce Quip — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T6 · **Subject:** Quip (Salesforce), a
collaborative-document SaaS.
## Why this dive
Notion (UC-57/58/59) gave us the closed-SaaS, external-REST-only, database-as-pages model.
Quip is the **enterprise document-collab** contrast: a **document+spreadsheet hybrid** where
a page is **prose interleaved with embedded live structured objects** (spreadsheets,
calendars, kanban "live apps"), reachable only through a REST API, gated by **Salesforce
identity**. The question: how does shard-wiki attach a shard whose "page" is a *mixed
prose+live-object document* behind enterprise auth?
## 1. The document + live-object model
- A Quip **document** is a real-time collaborative doc (concurrent editing). Its body is
rich content — headings, lists, prose — **interleaved with embedded objects**:
- **Spreadsheets** are *first-class, inline* — a doc can contain live spreadsheet sections
with formulas, not just static tables.
- **Live apps** (calendars, kanban boards, project trackers, polls) embed interactive
structured widgets inside the document.
- **Folders** organize documents; **threads** (docs and chat are unified — every doc is also
a conversation thread) carry **messages/comments** inline.
- So a Quip "page" is a **hybrid**: prose + embedded structured/live content in one
document, with conversation attached. Not Markdown; a proprietary rich model.
## 2. The REST API (the only door)
Quip exposes a **REST API**: threads/documents (get, create, **edit-document** with an HTML
fragment + a location/section anchor), folders, messages, users, and spreadsheet cell
access. Content crosses the API as **HTML** (import and export) — there is **no native
Markdown** and no file/git access. Implications:
- **Import/export is HTML → lossy** to Markdown (like Notion's export, UC-59): embedded
spreadsheets and live apps **do not round-trip** to Markdown cleanly — they degrade to
tables or links.
- Editing is **section/anchor-scoped HTML splice** (`edit-document` targets a location) — a
mid-granularity write (a section, not the whole doc, not a character).
- **Rate-limited** like any SaaS API; history is internal (revisions exist in-product, with
limited API exposure).
## 3. Enterprise identity / permissions
Quip is tied to **Salesforce** (acquired 2016): authentication and access run through the
Salesforce org / Quip site; **folder and document sharing ACLs** govern visibility, with
enterprise SSO. So this is an **enterprise-ACL, authn-delegated** shard (UC-06) — shard-wiki
honors Salesforce-side permissions and never bypasses them.
## 4. Capability profile
| Dimension (synthesis spectrum) | Quip |
|--------------------------------|------|
| Attachment mode | **external-API** (REST; HTML import/export) — closed SaaS |
| Addressing granularity | document; **section/anchor** for edits; spreadsheet cell |
| Content identity | Quip thread/document ID (opaque) |
| Identity vs placement | API-id identity; folder placement separate |
| Structure | **hybrid doc**: prose + embedded spreadsheets + live apps + thread |
| History | **internal** revisions (limited API exposure) |
| Merge model | server-side real-time collab (OT-like); no external merge |
| Native query | none exposed (spreadsheet formulas internal) |
| Translation | **HTML** in/out → **lossy** to Markdown (spreadsheets/apps degrade) |
| Write granularity | **section/anchor HTML splice** (mid) |
| Operational envelope | rate-limited SaaS REST |
| Access grant | **Salesforce identity + folder/doc ACL** (enterprise SSO) |
| Content opacity | proprietary rich model; not E2EE but not transparent files |
| Provenance | author/edit metadata in-product; API-limited |
## 5. INTENT mapping
### Reinforcements
- **External-API attachment** (UC-57): Quip is a second concrete instance beside Notion of
the closed-SaaS REST-only shard — generalizes the external-API mode (REST + HTML payload,
vs Notion's REST + block-JSON, vs Wiki.js GraphQL).
- **Union without erasure / no silent flatten**: embedded spreadsheets and live apps must be
**surfaced as what they are** with provenance, and the **HTML→Markdown lossiness made
explicit** (UC-59) — never silently drop a live spreadsheet to a static table.
- **Authz-in-core, authn-delegated** (settled decision): Quip's Salesforce-tied ACL is the
enterprise case — honor delegated identity and the shard's ACL (UC-06).
- **Graceful degradation**: with only a lossy HTML export, Quip is still usable as a
read/projection/overlay-target/backup shard.
### Divergences (boundaries / notes)
- **Mixed prose+live-object page** is a content shape beyond "Markdown body + frontmatter":
the page model must allow **embedded typed/live objects within a prose page** (not just a
whole-page-is-a-record like Notion DB, but *inline* structured content) — feeds T12 and the
non-Markdown-content question (UC-55).
- **HTML is the only interchange** — no Markdown, no files, no git. Content opacity is
"proprietary-but-exportable": transparent-ish via lossy HTML, not via files (T11
content-opacity tier between transparent-files and E2EE-opaque).
- **Write granularity = section-anchor HTML splice** — a mid tier (between whole-file and
block/character) realized over an API, distinct from fedwiki's story-item op-log.
### What to keep
1. **External-API mode generalized** to carry an **HTML payload** variant (Quip) beside
block-JSON (Notion) and GraphQL (Wiki.js) — capability/payload-format is part of the
adapter profile (UC-57/UC-80).
2. **Inline embedded live/structured objects** as a page-model element (prose + embedded
spreadsheet/app), with **explicit lossy projection** to Markdown (UC-55/UC-58/UC-59).
3. **Enterprise-ACL + delegated identity** honored, not bypassed (UC-06).
## 6. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-80 | Attach a **SaaS live-doc shard** whose pages **mix prose + embedded live structured objects** (spreadsheets / live apps) via **REST with lossy HTML import/export**, under **enterprise (Salesforce) identity** | **new** |
| — | external-API mode w/ HTML payload variant | enrich **UC-57** |
| — | inline embedded spreadsheet/live-app = non-Markdown content | enrich **UC-55** |
| — | embedded structured objects within a prose page | enrich **UC-58** |
| — | internal (API-limited) revision history | enrich **UC-36** |
| — | Salesforce-tied enterprise ACL + SSO | enrich **UC-06** |
## 7. Architecture notes for SHARD-WP-0002
- **T11 (capability / content opacity):** add a **payload-format** facet to external-API
shards (HTML / block-JSON / GraphQL) and a **content-opacity tier** "proprietary but
lossy-exportable" between transparent-files and E2EE-opaque. Write granularity =
section-anchor splice.
- **T12 (page model):** support **inline embedded structured/live objects** within a prose
page (not only whole-page-as-record) — with explicit lossy render to Markdown.
- **T14 (binding):** external-API binding with **HTML import/export**; honor Salesforce
identity/ACL; default to read/projection/overlay given rate limits + lossy export.
## 8. Open questions
1. Does shard-wiki represent an **embedded live spreadsheet** as a typed sub-object with
provenance (preferred) or flatten it to a static Markdown table (lossy) — and can overlays
target a spreadsheet cell via the API, or only the prose?
2. Given **HTML-only, lossy** interchange and rate limits, is Quip ever a **write-through**
shard, or read/projection/overlay/backup by default? (cf. Notion Q1, UC-57.)
3. How is **Salesforce identity** mapped to shard-wiki's delegated-authn model — pass-through
token, service account, or per-user OAuth?
## 9. Sources
- Quip Automation API (REST) docs — quip.com / Salesforce developer docs (threads/documents,
`edit-document`, folders, messages, spreadsheets)
- Salesforce Quip product docs (live apps, spreadsheets, sharing/permissions)
- prior: `research/260614-notion-deep-dive/` (closed-SaaS external-API contrast, UC-57/59)
## 10. Traceability
New UC **UC-80** carries the marker **◧** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-57, UC-55, UC-58, UC-36, UC-06. Architecture
cross-refs: SHARD-WP-0002 T11 (payload-format + content-opacity), T12 (inline objects), T14.

View File

@@ -0,0 +1,49 @@
# 260614 — Roam Research deep dive (the block graph as a queryable database + extension API)
Date: 2026-06-14
## What this is
A focused study of **Roam Research** — core architecture and extension structure —
read through shard-wiki's lens. Roam is the **modern bookend** to the two Nelson dives:
where Xanadu (`research/260614-xanadu-deep-dive/`) and ZigZag
(`research/260614-zigzag-deep-dive/`) are *unbuilt ideals*, Roam **shipped** stable
fine-grained addressing, transclusion, bidirectional links, and a queryable structured
space to a mainstream audience.
Distinctive material:
- **Data model** — the entire graph is a client-side **DataScript** (Datomic-like) DB
of **datoms** (EAV); every paragraph and page is a **block** entity with a stable
nine-character **`:block/uid`**, `:block/string`, `:block/refs`, `:block/page`,
pages marked by `:node/title`
- **References & transclusion** — `[[Page]]`/`#tag`/`((uid))` create `:block/refs`
edges; **block embeds** are working transclusion; linked vs unlinked references
- **Querying** — Datalog (`:q`/`:pull`); derived views are queries over the graph
- **Extension architecture** — **Roam Depot** (`extension.js` `onload`/`onunload`,
manifest, `extensionAPI` settings) over **`window.roamAlphaAPI`** read (`q`/`pull`) +
write (`block.create/update/move/delete`, `page.create`) — a concrete
*engine-hosts-adapter* (UC-38) surface
## Contents
| Path | Role |
|------|------|
| `findings.md` | Data model, references/transclusion, Datalog, extension API, Roam-as-shard capability profile, evidence on shard-wiki's open questions, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
(UC-50 attach a block-graph DB shard with block↔page mapping, UC-51 adopt native
block/span IDs as portable span addresses, UC-52 delegate derived views to a shard's
native query engine); UC-32/34/35/38 enriched. Logged for
`workplans/SHARD-WP-0002-federation-architecture.md` (T14): native-span-ID and
native-query capabilities, block↔page mapping, and Roam as a second DB-backed /
engine-hosts-adapter exemplar alongside XWiki.
**Key payoff recorded:** Roam empirically answers the Nelson dives' open questions —
fine-grained addressing is tractable (block UIDs), transclusion is a data-layer
capability over an addressable union, and derived views are queries over a structured
space. **Boundary:** Roam is *one candidate shard* (DB-backed/API-attached, block-first,
no portable Git history), mapped into shard-wiki's Markdown-first page model — not a
substrate and not the federation layer.
</content>

View File

@@ -0,0 +1,292 @@
# Findings — Roam Research: the block graph as a queryable database, and its extension API
Date: 2026-06-14
Source kind: **modern shipped product** — a block-graph note tool; a *candidate shard
backend* (DB-backed / API-attached, like XWiki), and the place where several Nelson
ideas actually shipped
Lens: shard-wiki — fine-grained addressing, transclusion, bidirectional links, a
structured/queryable page space, and the engine-hosts-adapter extension path
> Why Roam, and why now. The previous two dives were Ted Nelson's *unbuilt* ideals —
> Xanadu (`research/260614-xanadu-deep-dive/`: reference-not-copy, transclusion,
> stable fine-grained addresses) and ZigZag (`research/260614-zigzag-deep-dive/`: an
> information space as many co-equal dimensions over a queryable structure). **Roam
> Research is where a large slice of both actually shipped to a mainstream audience.**
> Every block has a stable address; block references and embeds are working
> transclusion; the whole graph is a queryable database; links are bidirectional. So
> Roam is the natural modern bookend: it lets us check shard-wiki's open questions
> (portable span addressing, transclusion, structured pages) against a system that
> solved them *in production* — and study its **extension architecture** as a concrete
> adapter-host surface.
This dive treats Roam as a *candidate shard* (capability profile in §6) and as
*evidence* on the open questions the Nelson dives left (§7), not as something to copy
wholesale — Roam is block-first and DB-backed, shard-wiki is Markdown-page-first and
Git-backed (divergences in §8.2).
---
## 1. What Roam is, in architecture terms
Roam Research is a networked-thought outliner whose **entire graph is a client-side
DataScript database** (DataScript = a ClojureScript reimplementation of Datomic's
data model and Datalog query engine). The store is not files; it is a set of immutable
**datoms**.
- A **datom** is one fact: `[entity, attribute, value, transaction]` — the EAV(T)
model. The graph is the accumulation of datoms; edits are transactions.
- Synchronization and Roam's deep undo fall out of the transaction log — a parallel
worth noting to Git's immutable commit history (but it is *not* Git; see §8.2).
This makes Roam a **structured, queryable** wiki-shaped store — the same family as
XWiki (objects + DB) rather than TWiki/Foswiki (flat files), which matters for how an
adapter attaches (§6).
---
## 2. The block model — everything is a block, every block is addressable
Every paragraph *and* every page is a **block** (an entity). Key attributes:
| Attribute | Meaning |
|-----------|---------|
| `:block/uid` | **nine-character public ID** — the block's stable, referenceable address (e.g. `((GGv3cyL6Y))`) |
| `:block/string` | the block's text content |
| `:block/order` | position among siblings |
| `:block/children` | immediate child entity-ids |
| `:block/parents` | full ancestor chain (incl. the page) |
| `:block/refs` | edges to blocks/pages this block references |
| `:block/page` | the owning page entity |
| `:create/time`, `:edit/time`, `:create/email`, `:edit/email` | provenance metadata |
| `:node/title` | **page-only** — the page's title (this attribute is what distinguishes a page from an ordinary block) |
Structurally the DB is a **forest**: each page is a tree of nested blocks; a "page" is
just the top-level block that carries `:node/title`. The single most shard-wiki-relevant
fact: **`:block/uid` gives every block — i.e. every sub-page span — a stable,
first-class, public address.** This is the *shipped* form of the Xanadu tumbler / the
shard-wiki "portable span address" open question (§7).
---
## 3. References, transclusion, bidirectional links
- **Page references** `[[Page]]` and `#tags`, and **block references** `((uid))`, all
create `:block/refs` edges. The link graph is therefore queryable, not parsed-on-read.
- **Block embeds** (`{{embed: ((uid))}}`) are **working transclusion**: a block's
content rendered live in another location by reference, not copied — when the source
changes, the embed reflects it. This is Xanadu transclusion / shard-wiki UC-32/UC-44,
shipped at block granularity.
- **Bidirectional links** come in two flavours, exactly shard-wiki's BackLinks problem:
- **Linked references** — explicit `:block/refs` edges (queryable backlinks).
- **Unlinked references** — bare text mentions with no edge, surfaced by text search.
- **Attributes** `key:: value` are a convention on top of blocks/refs that turns a page
into a lightweight record — Roam's answer to structured/typed pages (UC-34/UC-39),
queryable via Datalog.
---
## 4. Querying — Datalog over the graph
Roam exposes **Datalog** (`:find` / `:where`) plus **pull** expressions:
```
[:find ?uid :where [?b :block/string ?s] [?b :block/uid ?uid] ...]
```
This means Roam's "derived views" (a page's linked references, a `{{query}}` block, a
table) are **just queries over the datom graph** — not bespoke features. For shard-wiki
this is the strongest evidence yet that *derived views = queries over a structured
space* (the ZigZag "dimensions + rasters" insight, made executable), and that an
adapter could **delegate view computation to a shard's native query engine** instead of
scanning (UC-52).
---
## 5. Extension architecture — Roam Depot and `window.roamAlphaAPI`
Roam's official extension system (**Roam Depot**) is a concrete model of the
*engine-hosts-adapter* path (UC-38):
- **Distribution:** an extension is a GitHub repo (`README.md`, `extension.js`, optional
`extension.css` / `CHANGELOG.md` / `build.sh`), cataloged in the `roam-depot` repo via
a metadata JSON (`name`, `author`, `tags`, `source_repo`, `source_commit`, optional
`stripe_account` for paid extensions).
- **Lifecycle:** `extension.js` default-exports a map with **`onload`** and
**`onunload`** functions; everything created on load must be torn down on unload. An
`extensionAPI` object provides a **settings panel** and `settings.get/set`.
- **Data API — `window.roamAlphaAPI`** (the surface an adapter would target):
- **Read:** `q` (Datalog query → entity ids), `pull` / `pull-many` (fetch entity
attributes), full graph traversal.
- **Write:** `data.block.create` / `update` / `move` / `delete`, `data.page.create`.
- **UI:** command palette, blocks, settings.
Crucially, this API runs **inside the Roam client** (global `window.roamAlphaAPI`).
There is no first-class external REST write API; programmatic external access uses
graph **export** (EDN / JSON / Markdown) or unofficial private APIs. So a *live,
write-through* Roam adapter must be **hosted inside Roam as a Depot extension** — which
is exactly the UC-38 engine-side direction (cf. XWiki components, TWiki plugin handlers).
---
## 6. Roam as a shard — capability profile
| Capability | Roam | Notes for the adapter contract |
|------------|------|--------------------------------|
| Read | **yes** | `q`/`pull` live (in-app) or EDN/JSON/Markdown export (snapshot) |
| Write | **yes, in-app** | `block.create/update/move/delete`, `page.create` — but only via the in-client API → needs a Depot-hosted adapter (UC-38) or unofficial private API |
| Write granularity | **block-level (fine)** | the opposite extreme from TiddlyWiki's whole-file writes — sharpens UC-35 |
| Identity / addressing | **block `:block/uid` + page `:node/title`** | shipped sub-page stable addressing (UC-51) |
| Transclusion | **yes** | block embeds by uid (UC-32/44/45) |
| Backlinks | **yes** | linked (`:block/refs`) + unlinked (text) (UC-05/18) |
| Structured data | **yes** | attributes + Datalog (UC-34/39) |
| Native query | **yes (Datalog)** | adapter could delegate views (UC-52) |
| Diff / merge | **no native** | transaction log exists but not exposed as page diffs |
| Version history | **hosted, internal-only** | not portable Git history → needs supplement/import (UC-36, like Confluence/MediaWiki) |
| Lock | **no** | — |
| Syntax | **Roam-flavored Markdown** | not CommonMark; block-outline structure → translation/mapping needed (cf. UC-42) |
Verdict: Roam is a legitimate **DB-backed, API-attached shard** — the XWiki family, not
the file-store family. Its standout offerings are **block-level addressing** and
**shipped transclusion**; its costs are **block↔page impedance** and **no portable
history**.
---
## 7. Evidence on shard-wiki's open questions (the payoff)
The Nelson dives left open questions; Roam answers several *empirically*:
1. **"What is a portable fine-grained span address?"** (Xanadu §11 Q1, the tumbler
problem). Roam's answer: **a short opaque per-block UID, minted by the store, public
and referenceable.** shard-wiki lesson — fine-grained addressing is *tractable* when
the backend mints stable block/span IDs; the adapter should **adopt native block IDs
as span addresses where they exist** (UC-51), and fall back to content-fingerprint or
path+range where they do not (cf. Xanadu content-identity, UC-46).
2. **"Does transclusion belong in core or adapter/UI?"** (catalog Q6). Roam shows
transclusion is cheap and natural *when the store is addressable and queryable*; it
lives at the **data layer** (refs/embeds), surfaced by UI. Argues for transclusion as
a **core capability over an addressable union**, not a UI-only trick.
3. **"Derived views — core or adapter?"** Roam shows them as **queries over a
structured space**. Where a shard exposes a native query engine, **delegate**
(UC-52); where it does not, the orchestrator computes them over the projection.
4. **Block↔page mapping** is the new question Roam *raises*: in a block graph, what is a
"page"? Roam's own answer (`:node/title` node = page; nested blocks = spans) is a
clean mapping rule for the adapter (UC-50).
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **Fine-grained addressing is achievable** (block UIDs) — de-risks UC-44/45/51 and the
Xanadu tumbler worry.
- **Transclusion + bidirectional links + structured data** are not exotic; a shipped
tool does all three over one addressable, queryable store — validating shard-wiki's
ambitions for UC-32/34/39/05.
- **Engine-hosts-adapter (UC-38)** has a clean modern template: Depot's
`onload/onunload` + `roamAlphaAPI` read/write surface.
### 8.2 Deliberate divergences (design bugs if conflated)
1. **Block-DB substrate vs. Markdown-file page model.** Roam's "everything is a block in
a DataScript DB" must **not** tempt shard-wiki away from its Markdown-first,
backend-neutral page model (INTENT Stability Note). Roam is *one shard shape*, mapped
*into* the page model, not the model itself.
2. **Client-side proprietary DB vs. Git coordination journal.** Roam's transaction log
is not portable history; shard-wiki keeps Git as the coordination layer. A Roam shard
contributes snapshots/projections, and its history needs **supplement/import**
(UC-36), not adoption as canonical.
3. **Single graph vs. federation.** A Roam graph is **one sovereign shard**, never the
federation layer. Do not model the union as "a big Roam graph."
4. **Roam-flavored Markdown + outline structure** ≠ CommonMark pages; the adapter owns a
**lossy-aware translation** (block outline ↔ page + headings/lists), cf. UC-42.
### 8.3 What Roam teaches that shard-wiki should keep
- **Mint/adopt stable sub-page IDs.** The cheapest path to transclusion, overlay, and
reverse-lookup at span granularity is a backend that already addresses spans — lean on
it (UC-51), degrade gracefully otherwise.
- **Treat a structured shard's query engine as a capability**, and delegate derived
views to it when present (UC-52).
- **A clean block↔page rule** (`:node/title` node = page) keeps a block backend usable
without flattening (UC-50, complementing UC-34's no-lossy-flatten rule).
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-49**. New UCs **UC-50UC-52** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a block-graph DB wiki (Roam-style) as a shard via its query/CRUD API, mapping blocks to the page model** (`:node/title` node = page; nested blocks = spans) | **UC-50 (new)** |
| **Adopt a shard's native block/span IDs as portable span addresses** for transclusion/overlay (the shipped answer to the span-address question) | **UC-51 (new)** |
| **Delegate derived views to a shard's native query engine** (Datalog) where advertised | **UC-52 (new)** |
| Shipped transclusion via block embeds + shipped fine-grained addressing | **enriches UC-32** (and UC-44/45) |
| Attributes + Datalog = shipped structured/typed pages | **enriches UC-34** (and UC-39) |
| Roam Depot `onload/onunload` + `roamAlphaAPI` = modern engine-hosts-adapter template | **enriches UC-38** |
| Block-level write granularity = the fine extreme opposite TiddlyWiki | **enriches UC-35** |
| Hosted history is internal-only, not portable Git | links **UC-36** |
---
## 10. Architecture notes for SHARD-WP-0002 (no UC)
- The adapter contract should model **"native span/block IDs"** as a capability: a shard
advertises whether it mints stable sub-page addresses; transclusion/overlay/reverse-
lookup capabilities key off it (ties UC-44/45/46/51).
- The contract should model a **"native query"** capability so the orchestrator can
delegate derived-view computation (UC-52) vs. scanning the projection.
- A **block↔page mapping** belongs in the adapter, with the no-lossy-flatten rule of
UC-34: page = titled node, blocks = addressable spans, attributes = sidecar metadata.
- Roam joins XWiki as a **DB-backed / API-attached** exemplar (vs. TWiki/Foswiki file
stores) and as a **second engine-hosts-adapter** template (vs. XWiki components / TWiki
handlers) for the T14 adapter-binding task.
---
## 11. Open questions (for spec / workplans)
1. When a backend mints native span IDs (Roam UIDs), are they used **directly** as the
shard-wiki span address, or **wrapped** in a shard-scoped address so they survive
projection/overlay and don't collide across shards?
2. For a **write-through** Roam shard the adapter must run *inside* Roam (Depot
extension). Is in-engine hosting an accepted adapter deployment mode generally, or do
we restrict Roam to **read/projection/overlay-target** (graceful degradation) when we
cannot deploy inside it?
3. How lossy is the **block-outline ↔ Markdown-page** mapping, and is round-trip
fidelity required or best-effort (cf. UC-42 Foswiki TML↔HTML)?
4. Do we consume Roam's **export (EDN/JSON)** as the projection source, or the live
`roamAlphaAPI`? Snapshot vs. live freshness trade-off (cf. UC-31).
---
## 12. Sources
| Source | Used for |
|--------|----------|
| zsolt.blog — "Deep Dive Into Roam's Data Structure" (https://www.zsolt.blog/2021/01/Roam-Data-Structure-Query.html) | Datom/EAV model, block attributes (`:block/uid` etc.), refs/embeds, linked vs unlinked references, Datalog/pull |
| Roam Research Datalog Cheatsheet (https://gist.github.com/2b3pro/231e4f230ed41e3f52e8a89ebf49848b) | Attribute names, `:q`/`:pull` query shapes |
| GitHub — Roam-Research/roam-depot (https://github.com/Roam-Research/roam-depot) | Extension repo/manifest structure, `extension.js` `onload`/`onunload`, `extensionAPI` settings, `roamAlphaAPI` read/write/ui surface |
| "Introduction to the Roam Alpha API" (https://www.putyourleftfoot.in/introduction-to-the-roam-alpha-api) | `window.roamAlphaAPI` `q`/`pull` semantics, global-scope dependency |
| David Vargas — "How To Create Your Own Roam Extensions" (https://davidvargas.me/blog/how_to_create_your_own_roam_extensions) | Extension authoring lifecycle, write methods |
| Wikipedia / general — Roam Research, DataScript, Datomic | Datalog/Datomic lineage, client-side DB framing |
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (span addressing,
transclusion), `research/260614-zigzag-deep-dive/findings.md` (queryable dimensions),
`research/260613-xwiki-deep-dive/findings.md` (DB-backed engine + extension host),
`spec/UseCaseCatalog.md` (UC-05, UC-18, UC-32, UC-34/35/36/38/39, UC-44/45/46),
`workplans/SHARD-WP-0002-federation-architecture.md` (T14 adapter binding).
---
## 13. Traceability
- New UCs: **UC-50, UC-51, UC-52**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-32, UC-34, UC-35, UC-38** (and links UC-36, UC-39, UC-44/45/46).
- Architecture (no UC): native-span-ID capability; native-query capability; block↔page
mapping; Roam as DB-backed/API-attached + engine-hosts-adapter exemplar → `SHARD-WP-0002` (T14).
- Boundary recorded: Roam is **one candidate shard**, mapped into the Markdown-first
page model; not a substrate, not the federation layer (INTENT Stability Note).
</content>

View File

@@ -0,0 +1,69 @@
# 260614 — Shard spectrum synthesis (one capability model across the full dive set)
Date: 2026-06-14 · **revised 2026-06-14 (v3)**
## What this is
A **synthesis** (no new external research) that reads every studied system *across* the
others and distills them into a single comparative model feeding the **shard adapter
contract** and the **federation track** (`SHARD-WP-0002`).
**v3** extends the model to the **SHARD-WP-0003 engine batch** (Federated Wiki, Wikibase/
Wikidata, the git-forge wikis Gitea/GitLab/GitHub, TiddlyWiki, ikiwiki, Salesforce Quip,
MojoMojo, Oddmuse, UseModWiki) — ~23 systems in all. Per-shard spectra grow from thirteen to
**fourteen** (adds **provenance granularity**), and a **new coordination-layer axis** is
introduced: the **federation-model taxonomy**.
Centerpieces:
- **The shard family matrix** — every candidate backend × {substrate, attach mode(s),
addressing, structure, history, merge, query, →Markdown, opacity}, with Xanadu/ZigZag as
ideal anchors and Federated Wiki as the federation-model anchor. v3 adds the flat-file
floor (Oddmuse/UseModWiki), git-IS-store (forge wikis/ikiwiki), TiddlyWiki, MojoMojo
(direct-DB), Quip (external-API/HTML), and Wikibase (typed-graph far-end).
- **The fourteen capability spectra** — the claim that the adapter contract should model
*positions on spectra*, each anchored at both ends by a real system, with federation ops
degrading by position. v3 adds **provenance granularity** (14th: per-shard → per-page →
per-edit → per-statement/value), and refines merge-model (+fork/journal-replay,
+coexist-with-rank), attachment-mode (+git-IS-store, +container, +direct-DB,
+REST/file-store-hybrid, +external-API payload-format facet), native-query (+SPARQL/RDF
far-end, +filter mid), history (+DB-version-rows, +partial/truncated), structure
(+typed-graph, +inline-embedded objects), content-opacity (+proprietary-lossy-exportable),
write-granularity (+story-item, +section-anchor).
- **The federation-model taxonomy (§2.5, new)** — federation itself is plural: fork+journal
(Federated Wiki), VCS-replication+ping (ikiwiki), query-time graph join (Wikibase SPARQL
`SERVICE`), feed aggregation, activity streams, engine-mirror (Wiki.js). A
selectable/composable coordination-layer axis feeding **T1T6**.
- **UC-44UC-82 → workplan task mapping** (T1T6 + T11T16).
## Contents
| Path | Role |
|------|------|
| `findings.md` | Family matrix, the fourteen spectra, the federation-model taxonomy, cross-cutting through-lines, UC→task fold-in, recommendations/decisions, escalated open questions |
## Status
Synthesis v3 complete. No new use cases (consolidation only). Feeds `SHARD-WP-0002`:
**T1T6** federation-model taxonomy (selectable/composable federation); **T11** reframed
around the **fourteen spectra** (incl. provenance granularity + expanded attachment modes +
external-API payload-format facet); T12 page-model breadth (prose + typed/computed records +
**inline-embedded objects** + **typed-graph statements** + non-Markdown assets +
query-defined + multi-placement/DAG identity); T13 history (DB-rows + partial-flat-file =
supplement; completeness metadata; **journal-shaped** coordination journal); T14 **full
attachment taxonomy** (git-IS-store / container / direct-DB / REST-file-hybrid +
source-of-truth per binding); T15 lossy + **not-Markdown** (graph/HTML); T16 (addressing
incl. statement GUID + **opaque stable identity**, identity≠placement, derived index,
**graph query / federated SERVICE**). UC coverage extended UC-34UC-67 → **UC-34UC-82**.
**New through-lines (v3):** federation is plural (the federation-model taxonomy); provenance
has a granularity spectrum (down to per-statement, Wikibase rank/references); git is both the
home store and the home journal (forge wikis make git *the* store, resolving the engine-
mirror write-race); the flat-file floor (Oddmuse/UseModWiki, Wikipedia's MediaWiki Phase I)
is the field's common root and the minimal capability profile; the page model must also carry
typed-graph statements and inline-embedded objects.
**Carried from v2:** CRDT changes the merge math; identity ≠ placement; metadata can be
computed; content opacity is per-item; the attach surface is not always the native store; the
block-graph-on-files sweet spot (Logseq `id::`).
**Carried from v1:** files-canonical/index-derived; fine-grained addressing is adoptable;
transclusion=clone=embed=reference is one primitive; structure/history federate iff in-text;
attach mode is per-binding; the platform can enforce no-silent-mutation.

View File

@@ -0,0 +1,406 @@
# Synthesis — the shard spectrum: one capability model across the full dive set
Date: 2026-06-14 · **revised 2026-06-14 (v3)** — extended from the fourteen-system set to
include the **SHARD-WP-0003 engine batch** (Federated Wiki, Wikibase/Wikidata, the git-forge
wikis Gitea/GitLab/GitHub, TiddlyWiki, ikiwiki, Salesforce Quip, MojoMojo, Oddmuse,
UseModWiki). Per-shard spectra grown from thirteen to **fourteen** (added **provenance
granularity**), with refinements to merge-model, attachment-mode, native-query, structure,
history, content-opacity and write-granularity; and a **new coordination-layer axis — the
federation-model taxonomy (§2.5)** — the headline v3 contribution.
Source kind: **synthesis** — consolidates every deep dive into a single comparative model
feeding the **shard adapter contract** and the **federation track** (`SHARD-WP-0002`).
Lens: shard-wiki — what a backend must expose to participate, expressed as *spectra* of
capability rather than a yes/no checklist.
> Purpose. ~23 tools/systems have now been studied. Two conceptual ancestors (**Xanadu,
> ZigZag**); the file/DB engines (**XWiki, TWiki, Foswiki**, + landscape); the modern
> note/PKB tools (**Roam, Obsidian, Notion, Joplin, Logseq, Anytype, AFFiNE, AppFlowy,
> Trilium**); and the WP-0003 batch (**Federated Wiki, Wikibase, forge wikis, TiddlyWiki,
> ikiwiki, Quip, MojoMojo, Oddmuse, UseModWiki**), against the federation and origin
> research. This document reads them *across* each other. The payoff is a small set of
> **capability spectra** — now **fourteen** per-shard, plus a **federation-model taxonomy**
> for the coordination layer — each anchored at both ends by a real system, with federation
> operations degrading by position. That spectrum *is* the adapter contract's design
> surface. v3 folds the WP-0003 use cases (**UC-68UC-82**) into the `SHARD-WP-0002` tasks.
Inputs: `research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`,
`research/260613-{xwiki,twiki,foswiki}-deep-dive`,
`research/260614-{xanadu,zigzag,roam,obsidian,notion,joplin,logseq,localfirst-workspaces,trilium,wikijs,federated-wiki,wikibase,forge-wikis,tiddlywiki,ikiwiki,quip,mojomojo,oddmuse,usemodwiki}-deep-dive`.
Output target: `spec/TechnicalSpecificationDocument.md` (adapter contract) via
`SHARD-WP-0002`.
---
## 1. The shard family matrix
Candidate-shard backends across the dimensions that matter to the contract. (Xanadu and
ZigZag are *not* shards — they are the conceptual ideals each column aspires to; Federated
Wiki is *barely* a shard — it is mostly a **federation model**, see §2.5; both listed apart.)
| Backend | Substrate | Attach mode(s) | Addressing | Structure | History | Merge | Query | →Markdown | Opacity |
|---------|-----------|----------------|-----------|-----------|---------|-------|-------|-----------|---------|
| **Git folder / forge wiki** | files (`.wiki.git`) | **file-store (git IS store)** | path (+ commit) | flat MD + frontmatter | **git-native** | git/text | no | native | none |
| **ikiwiki** | files (git) → static HTML | file-store (source repo) | path | flat MD + directives | **git-native** | git/text | no | native (src) | none |
| **Obsidian** | files | file-store **/** plugin | path + in-file `^id` | frontmatter (in-file) | none (Git plugin) | git/text | plugin (Dataview) | native (OFM) | none |
| **Logseq** | files (MD/Org) → SQLite | file-store **/** plugin | **in-file block `id::`** (sweet spot) | `key::` props (in-file) | none (git) | git/text | **Datalog (derived)** | native-ish | none |
| **TiddlyWiki** | single HTML **/** `.tid` dir | **file-store (container/`.tid`)** | tiddler `title` | typed tiddler **fields** | none / git (.tid) | git/text | **filters** | varies (`type`) | none |
| **TWiki / Foswiki** | files + RCS / pluggable | file-store **/** API | path | `%META%` in-file (N records) | **open file (RCS)** | git/text | no | lossless (TML) | none |
| **Oddmuse / UseModWiki** | flat files (CGI) | file-store (**minimal floor**) | CamelCase/path | none (flat) | flat-file, **partial** | last-writer | no | lossy | none |
| **MojoMojo** | relational DB | **direct-DB-read** | page row / path | relational rows+lineage | **DB version rows** | last-writer | SQL | MD-in-column | none |
| **Trilium** | SQLite (one file) | ETAPI / scripting / DB | `noteId` + **`branchId`** (id≠place) | labels+relations, **inherited+templated**; **DAG** | internal revisions | conflict-res | attr search | **lossy (HTML)** | **per-note** |
| **Joplin** | SQLite-local | **sync-mirror** / local-REST / plugin | page-level `:/id` | notebooks + tags | internal revisions | conflict-notes | search | lossy | E2EE (whole) |
| **Wiki.js** | DB ↔ git mirror | file-store(mirror) / **GraphQL** | path | frontmatter (mirror) | git (via mirror) | git/text | GraphQL | native (mirror) | none |
| **XWiki** | DB (Hibernate) | in-engine host / REST | path | **XObjects/XClass** | internal (`xwikircs`) | git/text | yes (XWQL) | engine syntax | none |
| **Quip** | hosted (SaaS) | **external-API (HTML)** | doc id + section anchor | **prose + inline spreadsheets/live-apps** | internal | server (OT) | no | **lossy (HTML)** | proprietary |
| **Roam** | client DataScript | **in-app host only** | **store UUID** | `key::` attrs | internal (txn log) | (in-app) | yes (Datalog) | (Roam MD) | none |
| **Notion** | hosted Postgres | **external-API (block-JSON)** | store UUID | **DB schema + relations + rollups** | internal, not portable | internal | yes (DB query) | **lossy** | none |
| **Wikibase / Wikidata** | MediaWiki + RDF index | external-API + **SPARQL** | **opaque Q/P id + stmt GUID** | **typed entity-statement graph** | page revisions (JSON) | last-writer + **rank** | **SPARQL (+SERVICE)** | **lossy (not MD)** | none |
| **Anytype** | **CRDT (any-sync)** | **replica / P2P node** | object id (store) | **typed object graph (ontology)** | CRDT log | **native-CRDT** | graph | lossy | **E2EE (whole)** |
| **AFFiNE** | **CRDT (Yjs)** | replica / self-host | block id (store) | blocks; **one-data-many-views** | CRDT log | **native-CRDT** | DB filters | lossy | optional |
| **AppFlowy** | **CRDT (Yrs)** | replica / self-host | block id (store) | **Notion-style DB + views** | CRDT log | **native-CRDT** | DB query | lossy | self-host |
| — *Federated Wiki (mostly a federation model, §2.5)* | per-page JSON | **REST/file-store hybrid** | story-item `id` | typed story items | **append-only journal** | **fork + journal-replay** | neighborhood search | wiki-ish | none |
| — *Xanadu (ideal)* | permascroll | — | **tumbler (span)** | spans + links | permanent | content-merge | — | — | — |
| — *ZigZag (ideal)* | cells/dims | — | cell id | **N dimensions** | — | — | dimension walk | — | — |
Reading top to bottom is roughly shard-wiki's difficulty gradient, now with both ends
extended: the **flat-file floor (Oddmuse/UseModWiki)** anchors the bottom — minimal, partial
history, the field's common ancestor; **git/forge wikis, ikiwiki, Obsidian, Logseq,
TiddlyWiki** are friction-free file-store cases (forge wikis make *git the store itself*);
**TWiki/Foswiki/Trilium/Wiki.js** add translation, DAG/computed metadata, or a git mirror;
**MojoMojo** needs direct-DB reads; **XWiki/Roam/Notion/Quip** add DB/SaaS structure and
store/API addressing; **Wikibase** is the **typed-knowledge-graph far-end** (SPARQL,
statement-level provenance); the **CRDT cohort** adds native merge + (Anytype) P2P/E2EE.
Xanadu/ZigZag mark the ideals; Federated Wiki marks the federation-model ideal (§2.5).
---
## 2. The capability spectra (the contract's real shape) — fourteen
Each capability is **not boolean** — it is a position on a spectrum anchored at each end by
a real system. The contract models *positions*; federation ops degrade by position.
1. **Addressing granularity** — `none → whole-page(path) → page-level store id(Joplin :/id,
Trilium noteId) → in-file span(Obsidian ^id) → in-file block(Logseq id::, the sweet spot:
block-level AND git-diffable) → store-minted span(Roam/Notion/CRDT UUID) → statement GUID
(Wikibase) → portable tumbler(Xanadu ideal)`. Story-item `id` (Federated Wiki) is a
mid-tier within-page handle. (UC-51, UC-44/45, UC-73.)
2. **Content identity** — `none → path/title → opaque stable id, labels-as-annotation
(Wikibase Q/P) → fingerprint(hash) → span-set/equivalence(Xanadu)`. Wikibase is the
cleanest real instance of **stable, language-neutral identity**. (UC-46, UC-27, UC-73.)
3. **Identity vs placement** — `path = identity(most) → identity separated from placement
(Trilium note/branch; a page in many locations = a DAG) → provenance-edge links across
sites (Federated Wiki fork-DAG; forge wiki = path, identity layered above)`. The clean
model for a page under multiple paths/shards. (UC-66, UC-22, UC-71.)
4. **Structure** — `flat Markdown(Oddmuse) → in-file frontmatter/key::(Obsidian/Logseq) →
in-file %META%(TWiki) → tiddler fields(TiddlyWiki) → relational rows(MojoMojo) → typed
objects(XWiki) → DB schema+relations+rollups(Notion/AppFlowy) → prose+inline-embedded
objects(Quip) → typed object graph/ontology(Anytype) → computed inherited+templated
(Trilium) → typed entity-statement knowledge graph(Wikibase)`. In-text federates;
DB-locked needs sidecar+fidelity; **graph/computed** needs effective-vs-own +
render-without-flatten. (UC-34, UC-39, UC-58, UC-67, UC-73, UC-80.)
5. **History** — `none → partial/truncated flat-file(Oddmuse keep/) → internal-only/
not-portable(Notion/Joplin/Trilium) → DB version rows(MojoMojo) → CRDT update log
(Anytype/AFFiNE/AppFlowy) → append-only semantic journal(Federated Wiki) → open file
format(TWiki RCS) → git-native(Git/forge/ikiwiki/Obsidian/Logseq/Wiki.js mirror)`.
Internal/CRDT/DB-rows ⇒ *supplement*; open-file/journal ⇒ *import*; git ⇒ *adopt*.
**Completeness is metadata** (Oddmuse history is partial — never imply complete). (UC-36,
UC-41, UC-71, UC-81, UC-82.)
6. **Merge model** — `none → last-writer → git/text 3-way merge → conflict-notes/keep-both
(Joplin) → fork + manual journal-replay(Federated Wiki) → coexist-with-rank(Wikibase:
contradictory values kept, curated) → native-CRDT conflict-free(Anytype/AFFiNE/AppFlowy)`.
Four+ distinct models — never impose git/text merge on a CRDT or a journal shard; speak
the shard's model or stay projection/overlay. (UC-64, UC-71, UC-75.)
7. **Native query** — `none → text search → filter expressions(TiddlyWiki) → build-your-own
derived index(Logseq DataScript over files; shard-wiki can do likewise) → datalog/graph
(Roam/Anytype) → DB query(Notion/AppFlowy/XWiki) → SPARQL/RDF + federated SERVICE
(Wikibase, the far-end + query-time cross-shard join)`. Delegate where present; **build an
index over the projection** where not. (UC-52, UC-63, UC-05, UC-54, UC-74.)
8. **Translation to Markdown** — `native → lossless round-trip(Foswiki TML↔HTML) → lossy-
with-fidelity-report(HTML/CKEditor Trilium; Notion blocks; Quip HTML+embedded objects;
CRDT/object models) → not-Markdown-at-all(Wikibase statements → lossy render or keep
graph)`. Lossless ⇒ writable; lossy ⇒ read/projection floor + visible fidelity loss;
not-MD ⇒ structured payload + optional rendered view. (UC-42, UC-59, UC-03, UC-73, UC-80.)
9. **Attachment mode** — a **per-binding, capability-gated** choice; a backend may offer
several. The full taxonomy:
- **file-store** — *native on-disk store* (Obsidian/Logseq/TWiki), *git IS the store*
(forge `.wiki.git`, ikiwiki source), *single-file container* (TiddlyWiki HTML),
*flat-file floor* (Oddmuse), **or** *interchange/sync mirror* (Joplin; Wiki.js git
mirror) (UC-40, UC-53, UC-60, UC-62, UC-76, UC-78, UC-79, UC-82, UC-68)
- **in-engine host** — adapter inside the app via its API (Roam/Obsidian/Logseq/Trilium
scripting, XWiki components) (UC-38, UC-50)
- **local-REST** — localhost API, app-running (Joplin Data API; Trilium ETAPI) (UC-38)
- **external-API** — remote API from outside, with a **payload-format facet**:
*block-JSON* (Notion, UC-57), *GraphQL* (Wiki.js, UC-69), *HTML* (Quip, UC-80),
*forge wiki REST* (GitLab/Gitea, UC-77), *MediaWiki/SPARQL* (Wikibase, UC-73/74)
- **direct-DB-read** — read the engine's relational store (MojoMojo) when no file/API
exists; schema = a versioned coupling (UC-81)
- **CRDT replica** — hold a local CRDT replica (Anytype/AFFiNE/AppFlowy) (UC-64)
- **P2P / no-central-endpoint** — replica or peer/node, not a URL (Anytype) (UC-65)
- **REST/file-store hybrid** — page JSON over HTTP+CORS or static files (Federated Wiki)
(UC-70)
10. **Operational envelope** — `local/unbounded → realtime CRDT/WebSocket → rate-limited+
eventually-consistent+paginated(Notion ~3 rps, Quip, Wikibase public endpoints)`. Sets
live vs cache/poll/webhook. (UC-57, UC-31.)
11. **Access grant** — `open(L0; Oddmuse) → token → OAuth scoped+revocable(Notion) →
enterprise SSO + ACL(Quip/Salesforce, Wiki.js path rules) → P2P key/invite(Anytype) →
own-site-only writes(Federated Wiki)`. The backend may *enforce* no-silent-mutation.
(UC-57, UC-06, UC-65, [[shard-wiki-auth-in-core-decision]].)
12. **Content opacity** — `plaintext(files) → proprietary-but-lossy-exportable(Quip HTML;
Notion) → encrypted-at-rest whole-shard(Joplin/Anytype E2EE) → per-item(Trilium
protected notes)`. Opaque ⇒ backup/structure-shell; never present ciphertext (or imply
a lossy export is faithful) as readable. (UC-61, UC-80.)
13. **Write granularity** — `whole-file(TiddlyWiki single-file) → per-page/note(Git/forge/
Obsidian/Joplin/Trilium/Oddmuse) → story-item/paragraph(Federated Wiki) → section-anchor
splice(Quip) → per-statement(Wikibase API) → per-block(Roam/Notion/Logseq/CRDT)`. Sets
overlay/patch/lock/conflict scope; whole-file ⇒ **no per-page atomicity**. (UC-35,
UC-78.)
14. **Provenance granularity** *(new, 14th)* — `none → per-shard → per-page(most; author/
time) → per-commit(git) → per-edit(journal entry, Federated Wiki) → per-statement/value
(Wikibase references + rank)`. How finely the union can attribute and source content;
Wikibase pushes provenance below the page (sourced, contradictory values coexist with a
curation signal). The page model + journal should *allow* sub-page provenance even if MVP
records per page. (UC-24, UC-71, UC-75.)
*(Content types — Markdown-only → typed records → inline-embedded objects (Quip
spreadsheets) → non-Markdown assets (Excalidraw/Canvas/whiteboards) → typed-graph statements
(Wikibase) — remains a cross-cutting page-model demand, tracked under structure + T12 rather
than as a standalone capability. UC-55.)*
Design consequence: **T11's capability vocabulary = these fourteen spectra**, not a flat
`read/write/diff/...` list. The flat verbs remain the *operations*; the spectra are the
*profile* saying how well each verb is supported and how it degrades. The **floor**
(Oddmuse/UseModWiki) and **far-ends** (Wikibase graph/query/provenance; CRDT merge; Notion
hosted) bound every spectrum with a real system.
---
## 2.5. The federation-model taxonomy (the coordination-layer axis) — new in v3
The fourteen spectra above describe a *single shard's* capabilities. The WP-0003 batch
revealed a second, orthogonal axis the v2 synthesis under-modelled: **federation itself is
plural.** "Attach many shards and present a union" can be realized by several distinct
coordination models, each a real system, each with different reconciliation semantics. This
axis lives at shard-wiki's **coordination layer** (`SHARD-WP-0002` T1T6), not in a single
adapter.
| Federation model | Exemplar | Mechanism | Reconciliation | Discovery |
|------------------|----------|-----------|----------------|-----------|
| **Fork + journal** | Federated Wiki | copy page to own site; append-only semantic **journal** records `fork`-with-source | **manual**: compare journals, fork the version you prefer; **chorus**, no canonical | link + fork (**neighborhood**) / curated **roster** |
| **VCS replication + ping** | ikiwiki | git clone/pull/push between instances; **XML-RPC pinger** notifies peers to pull/rebuild | **git merge** across clones | configured peers + pings |
| **Query-time graph join** | Wikibase | **SPARQL `SERVICE`** runs a sub-query on another endpoint and joins | none (read-time join; rank curates conflicts) | endpoint URLs |
| **Feed aggregation** | ikiwiki `aggregate`, RSS/Atom | pull remote feeds in as pages | one-way inbound projection | feed URLs |
| **Activity streams** | ActivityPub (federation research) | actor/inbox/outbox Create/Update | per-actor; eventual | actor handles |
| **Engine-maintained mirror** | Wiki.js git mirror | DB-canonical engine syncs to a git mirror | engine owns DB↔git sync (don't double-sync) | the mirror repo |
Two cross-cutting lessons:
- **git-IS-store vs engine-mirror resolves the write-race.** A **forge wiki** (`.wiki.git`)
and **ikiwiki source** make *git the canonical store*, so **write-by-commit is safe** — no
engine to race. Wiki.js (engine-mirror, DB canonical) is the opposite and needs care. The
contract must record *which side is source of truth* per binding. (Resolves UC-68's open
race for the git-canonical case; UC-76.)
- **shard-wiki's own coordination journal should be journal-shaped.** Federated Wiki proves
the **append-only semantic-op log with provenance entries, page-state-as-derived-replay**
pattern in production — the concrete shape for INTENT's coordination journal (UC-71), and a
superset that can *ingest* git history, CRDT logs, DB version rows, and partial flat-file
histories as differently-grained inputs.
Design consequence: **T1T6 should model federation as a selectable model (or composition of
models), not a single hard-coded flow** — mechanism over policy at the coordination layer,
mirroring how T11 models per-shard capability as spectra. A given information space may use
fork+journal for human-curated shards, VCS-replication for git shards, query-join for graph
shards, and feed-aggregation for read-only sources — concurrently.
---
## 3. Cross-cutting findings (the through-lines)
- **Files-canonical, index-derived is the winning architecture.** Obsidian's MetadataCache,
**Logseq's DataScript-over-files**, Git's working tree, **Wikibase's WDQS (SPARQL index
rebuilt from canonical JSON entities)**, and shard-wiki's projection model agree: the
graph/backlinks/query index is **derived and rebuildable**, never a second source of truth.
Roam/Notion/CRDT invert this (store canonical) and pay in portability. shard-wiki keeps
files + journal canonical.
- **Git is both the home store and the home journal.** *(sharpened in v3)* Forge wikis make
git *the* store; ikiwiki and Wiki.js make git the source or mirror; for all of them the
**git log is the coordination journal with zero synthesis**. The git-canonical cases are
the friction-free core; everything else is measured as deviation.
- **The flat-file floor is the field's common root.** *(new)* Oddmuse and UseModWiki
(Wikipedia's MediaWiki Phase I) show the minimal plain-text page+history wiki is the
ancestor every richer engine elaborates — so the **minimal/floor capability profile** is
the right baseline, and shard-wiki's page model must stay attach-compatible with it
(flat files, CamelCase identities, **partial** history). (UC-82.)
- **Federation is plural** *(new — §2.5).* Fork+journal, VCS-replication+ping, query-time
graph join, feed aggregation, activity streams, engine-mirror — distinct coordination
models, selectable and composable, not one flow.
- **Provenance has a granularity spectrum** *(new — spectrum 14).* From per-shard down to
**per-statement/value** (Wikibase references + rank). Union-without-erasure includes
*attribution at the right grain* and letting sourced contradictions coexist with curation.
- **Identity ≠ placement.** Trilium's **note vs branch** (a note cloned into many locations =
a DAG) and Wikibase's **opaque stable IDs (labels-as-annotation)** are the clean models:
separate *what a page is* from *where it sits* and from *what it's called*. Federated
Wiki's fork entries are **provenance edges** between same-named pages across sites.
- **Transclusion ⇄ clone ⇄ embed ⇄ cloned-note ⇄ reference is one primitive** over an
addressable union (Xanadu, ZigZag, Roam/Logseq, Obsidian `![[ ]]`, Notion synced block,
Trilium cloning, TiddlyWiki `{{ }}`). (UC-32/44/45/51/66.)
- **CRDT changes the merge math; journals and rank add more models.** *(extended)* The merge
spectrum now spans last-writer → git 3-way → conflict-notes → **fork+journal-replay** →
**coexist-with-rank** → native-CRDT. Never impose git/text merge across that range.
- **Structure & history federate iff in text; metadata can be computed; the far-end is a
graph.** `%META%`/frontmatter/`key::`/tiddler-fields diff and travel; XObjects/Notion-DB/
CRDT/Quip-objects lock in; Trilium computes metadata (inherited+templated); **Wikibase is
a full typed knowledge graph** — render to Markdown is lossy, so keep the graph and offer a
view, never silent-flatten.
- **The attach surface is rarely "just files," and source-of-truth varies.** *(extended)*
Joplin's best surface is its sync mirror; Logseq offers file-graph or DB-graph and is
migrating substrate; **forge wikis = git-canonical (write freely)** vs **Wiki.js =
DB-canonical mirror (write carefully)** vs **MojoMojo = DB-only (direct read)**. Bind to
capabilities *and* record the source of truth.
- **The page model must stretch many ways at once:** prose Markdown, typed/computed records
(N-per-page, relations, inheritance), **inline-embedded objects** (Quip), **typed-graph
statements** (Wikibase), non-Markdown assets, reference/query-defined pages, and
**multi-placement (DAG) identity**. The heaviest demand on T12.
---
## 4. How the use cases fold into the workplan
The 260614 + WP-0003 use cases (UC-44UC-82) map onto the adapter-contract and federation
tasks:
| UCs | Theme | Lands in |
|-----|-------|----------|
| UC-35, UC-50, UC-53, UC-57, UC-60, UC-62, UC-64, UC-65, **UC-68, UC-70, UC-76, UC-77, UC-78, UC-79, UC-80, UC-81, UC-82** | attachment modes (file-store native/git-IS-store/container/mirror, in-engine, local-REST, external-API w/ payload-format, direct-DB, CRDT-replica, P2P, REST/file-store-hybrid) + operational envelope | **T11** + **T14** |
| UC-34, UC-39, UC-55, UC-58, UC-67, **UC-73, UC-80** | structured/typed/computed/**graph** payload, **inline-embedded objects**, non-MD assets | **T12** |
| UC-36, UC-41, **UC-81, UC-82** | history: internal/CRDT-log/**DB-rows**/**partial-flat-file** = supplement; **completeness metadata** | **T13** |
| UC-42, UC-59, **UC-73, UC-80** | translation: lossless vs lossy-with-fidelity vs **not-Markdown** (graph/HTML/objects) | **T15** |
| UC-31, **UC-79** | webhooks / realtime / push-vs-poll / **VCS ping** | **T6** / **T11** envelope |
| UC-57 §6, UC-61, UC-65, **UC-06, UC-80** | scoped grant; **enterprise ACL/SSO**; content opacity (whole/per-item/**lossy-exportable**); P2P key | **T11** (access-grant + opacity) |
| UC-44, UC-45, UC-46, UC-51, UC-63, UC-66, **UC-73, UC-74** | span/statement addressing, **opaque stable identity**, identity≠placement, transclusion-as-reference, derived index, **graph query** | **T16** (+ T12) |
| UC-47, UC-48, UC-52, UC-54, **UC-74** | dimensional navigation, query delegation/build, query-defined pages, **SPARQL/federated SERVICE** | **T16** (+ T5/T10) |
| UC-24, **UC-71, UC-75** | **provenance granularity** (per-edit / per-statement); coordination journal shape | **T13/T16** + journal |
| UC-26, UC-27, UC-28, UC-30, UC-05, **UC-71, UC-72, UC-79** | **federation-model taxonomy** (fork+journal, VCS-replication+ping, query-join, chorus/neighborhood/roster) | **T1T6** |
The one **structural** v3 change: the **federation-model taxonomy (§2.5)** is a new design
surface for **T1T6** — federation becomes a *selectable, composable model* rather than a
single flow. Per-shard, **T11** grows to **fourteen spectra** (adds provenance granularity)
and **T14** absorbs the expanded attachment-mode taxonomy (git-IS-store, container,
direct-DB, REST/file-store-hybrid, external-API payload-format facet).
---
## 5. Recommendations (decisions to record under SHARD-WP-0002)
1. **Model per-shard capabilities as the fourteen spectra (§2)**, not flat verbs. (T11.)
2. **Model federation as a selectable/composable taxonomy of models (§2.5)** — fork+journal,
VCS-replication+ping, query-time graph join, feed aggregation, activity streams,
engine-mirror — at the coordination layer. (T1T6.)
3. **Make the coordination journal journal-shaped** (Federated Wiki): append-only semantic
ops + provenance entries, page state = derived replay; able to *ingest* git history, CRDT
logs, DB version rows, and partial flat-file histories as differently-grained inputs.
(T13 + journal, UC-71.)
4. **Add provenance granularity as a spectrum** (per-shard → per-page → per-edit →
per-statement/value); allow sub-page provenance and **coexist-with-rank** for sourced
contradictions. (T11/T13, UC-24/75.)
5. **Record source-of-truth per binding** and resolve the write-race accordingly:
git-canonical (forge wiki / ikiwiki) ⇒ write-by-commit; engine-mirror (Wiki.js) ⇒ careful;
direct-DB (MojoMojo) ⇒ read/projection/overlay default. (T14, UC-68/76/79/81.)
6. **Expand attachment mode (§2 #9 / T14):** file-store (native / **git-IS-store** /
**container** / mirror / **flat-file floor**), in-engine, local-REST, **external-API with
a payload-format facet** (block-JSON/GraphQL/HTML/forge-REST/SPARQL), **direct-DB-read**,
CRDT-replica, P2P, **REST/file-store-hybrid**. Bind to capabilities; substrate can
migrate. (T14, UC-40/68/70/76/77/78/79/80/81/82 + UC-43.)
7. **Page model: support a typed-graph payload and inline-embedded objects, not only
typed records** — render to Markdown lossily, keep the graph/objects canonical; preserve
computed (inherited/templated) metadata as effective-vs-own. (T12/T15, UC-73/80/67.)
8. **Capability profile must express absence and partiality cleanly** (Oddmuse floor):
sparse profiles, **partial/truncated history reported honestly**, graceful degradation to
read/projection/overlay/backup. (T11/T13, UC-82.)
9. **Query: SPARQL/RDF + federated `SERVICE` is the graph far-end** — delegate to native
engines (filters/datalog/DB-query/SPARQL) where present; build a derived index over the
projection where not. (T16/T5, UC-52/63/74.)
10. **Adopt opaque stable identity with labels-as-annotation** (Wikibase) and **separate
identity from placement** (Trilium note/branch); fork entries are provenance edges.
(T16, UC-73/66/71.)
11. **Keep files + coordination journal canonical; all indexes/projections derived and
rebuildable.** (Architecture invariant, INTENT.)
These honor INTENT: mechanism over policy (capability *and now federation* modelled as
spectra/taxonomy, not hard-coded behaviors), union without erasure (fidelity +
effective-vs-own + identity/placement + **provenance granularity** all preserve
information), graceful degradation (every spectrum has a read/projection floor; the
Oddmuse floor is explicit; opaque/CRDT/graph shards degrade to backup/projection/lossy-view),
no silent mutation (access-grant + opacity + overlay + respect native merge + **source-of-
truth per binding**), shard sovereignty (no backend forced to change substrate),
Markdown-first/backend-neutral (in-text preferred; DB/CRDT/HTML/**graph** tolerated with a
view).
---
## 6. Open questions escalated by the synthesis
1. **Federation composition** — can one information space run several federation models
(§2.5) concurrently over different shards, and how does the union reconcile a chorus
(fork+journal) with a canonical-asserting shard (Notion / upstream main)? (T1T6,
UC-72.)
2. **Coordination-journal op vocabulary** — adopt Federated Wiki's exact ops
(create/add/edit/move/remove/fork) at item grain, or an abstract op set other shards can
emit, with git/CRDT/DB-row/flat-file histories ingested as inputs? (T13, UC-71.)
3. **Provenance granularity in the model** — does the journal/page model carry per-statement
provenance + rank (Wikibase), per-edit (journal), or per-page (MVP), configurable? (T13,
UC-75.)
4. **Typed-graph page** — model a Wikibase entity natively (statements) or always project to
a lossy Markdown/table view, or both (canonical graph + view)? Is SPARQL a union-level
capability or pass-through? (T12/T16, UC-73/74.)
5. **Source-of-truth + write-race** — formalize git-canonical vs engine-mirror vs direct-DB
per binding; sanction direct third-party DB reads (schema drift) and write-by-commit
timing. (T14, UC-68/76/79/81.)
6. **Portable span address** across heterogeneous backends — wrap native IDs (Roam/Notion/
CRDT UUID, Logseq `id::`, Trilium noteId, **Wikibase Q/P + stmt GUID**, fedwiki story-item
id) in a shard-scoped address? (T16.)
7. **CRDT shards** — embed a CRDT client (Yjs/Yrs) for a live replica, or consume snapshots?
Overlays as CRDT ops or out-of-band patches? (T14, UC-64.)
8. **Page model breadth** — can one model carry prose + typed/computed records +
inline-embedded objects + typed-graph statements + non-MD assets + query-defined pages +
multi-placement identity coherently? (T12.)
9. **Whole-file & partial-history shards** — overlays against a whole-file shard
(TiddlyWiki single-file: buffer+re-serialize vs require `.tid`); representing
partial/truncated history (Oddmuse) with completeness metadata. (T11/T13, UC-78/82.)
10. **Content opacity & lossy-exportable** — what is visible for an opaque/proprietary shard
(IDs/structure vs nothing vs lossy HTML); never present a lossy export as faithful; does
shard-wiki ever hold keys? (T11, UC-61/80.)
---
## 7. Sources
A synthesis; primary sources are the ~23 dives' `findings.md` files plus
`research/260608-{federation-concepts,wikiengines-overview,c2-wiki-origins,yawex-prior-art}`.
No new external research was performed in v3 (it consolidates the WP-0003 batch dives
260614-{federated-wiki,wikibase,forge-wikis,tiddlywiki,ikiwiki,quip,mojomojo,oddmuse,
usemodwiki}).
Cross-references: `spec/UseCaseCatalog.md` (UC-26UC-82),
`workplans/SHARD-WP-0002-federation-architecture.md` (T1T16),
`workplans/SHARD-WP-0003-engine-dives-batch.md` (the engine batch, done),
`INTENT.md` (constraints), [[shard-wiki-auth-in-core-decision]].
---
## 8. Traceability
- Consolidates: all ~23 deep dives + federation/origin research into one capability model
(v3 adds the WP-0003 batch).
- Feeds: `SHARD-WP-0002` **T1T6** (federation-model taxonomy §2.5 — selectable/composable
federation), **T11** (**fourteen-spectra** vocabulary, incl. provenance granularity +
expanded attachment modes + payload-format facet), T12 (page-model breadth: typed-graph,
inline-embedded objects, computed metadata, identity≠placement, non-MD), T13 (history incl.
DB-rows + partial-flat-file = supplement, completeness metadata, journal-shaped
coordination journal), T14 (full attachment taxonomy incl. git-IS-store / container /
direct-DB / REST-file-hybrid + source-of-truth per binding), T15 (lossy + not-Markdown
graph/HTML), T16 (addressing incl. statement GUID + opaque stable identity, graph query /
federated SERVICE).
- UC coverage extended in the workplan from UC-34UC-67 to **UC-34UC-82**.
- No UCs added (synthesis only); no boundary changes (INTENT Stability Note untouched —
the federation-model taxonomy is a refinement of *how* the coordination layer works, not a
redefinition of shard / Git's role / orchestrator-vs-engine).

View File

@@ -0,0 +1,33 @@
# 260614 — Squeak & Pharo (image-based Smalltalk) deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T6 (Squeak) + T8 (Pharo)** — combined memo
(justified merge: both are image-based Smalltalks; Pharo is T8's thin "context for T6/T7").
## What this is
A deep dive into the **image-based live-object** environment — Squeak and Pharo (the
substrate Glamorous Toolkit T7 runs on): the **image** as a persistent world of live objects
with **no file/document/app boundary**, the live **inspector**, and Pharo's retreat to
**code-as-text in git** (Tonel/Iceberg).
## Why it matters
- The **purest "live" end** of the batch's spectrum (literate source → notebook snapshot →
GT/Lepiter live-over-files → **image: everything live**). Names the **live↔snapshot** axis
the projection model (T16) must carry.
- Hardens the **image-is-not-a-store** boundary (opaque monolithic non-diffable blob; no
page identity/history/provenance) — generalizes "attach files, not the kernel/image"
(UC-84, T7) into a named binding boundary (T14).
- Pharo **confirms** the resolution: even image traditions externalize to **git-versionable
text** (Tonel) to version/collaborate — files-canonical from the Smalltalk side.
## Yield
- **No new UC** (boundary / design prior art; covers T6 and T8). Boundary for UC-34/35/79;
links UC-83/84 (live→snapshot), UC-54/47/48 (live-object inspection), UC-76/79 (Tonel/git).
## Contents
| Path | Role |
|------|------|
| `findings.md` | The image & live objects, the image-as-store anti-pattern, Pharo Tonel/Iceberg→git, INTENT mapping, UC disposition (enrichment-only), architecture notes (T14 boundary, T16 live↔snapshot axis) |

View File

@@ -0,0 +1,141 @@
# Squeak & Pharo (image-based Smalltalk) — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 **T6 (Squeak)** + **T8 (Pharo)** — combined
(justified merge: both are image-based Smalltalks; Pharo is T8's "context for T6/T7," kept
brief per the workplan). · **Subject:** the **image-based live-object** environment and what
it teaches (and warns) about shard-wiki's page model.
## Why this dive (and why merged)
Squeak (1996, the Alan Kay/Ingalls lineage) and Pharo (2008 fork, the substrate **Glamorous
Toolkit** T7 runs on) are the same idea: **a persistent world of live objects, the "image,"
with no file/document/application boundary.** They are **not candidate shards** — they are
the **anti-pattern boundary** for shard-wiki's files-canonical stance *and* the inspiration
behind moldable inspection (T7). T8 (Pharo) is folded here as the workplan allows: its
distinct contribution over Squeak (Tonel/Iceberg file-based code → git) is the one piece that
*does* touch our concerns, covered in §3.
## 1. The image: knowledge-as-live-objects
- A Smalltalk **image** is a serialized snapshot of the **entire object memory** — every
object, class, tool, window, and the running program state — persisted as one binary file
(`.image` + `.changes` log). You resume exactly where you left off.
- **"Everything is a live object"**: code, data, UI (Morphic), the debugger, the inspector —
all are objects you can open, message, and modify *in place*, while running. There is **no
edit/compile/run cycle** and **no document-vs-app distinction**.
- The **inspector** lets you open any object and explore/modify its state live — the direct
ancestor of GT's moldable inspector (T7), but generic rather than domain-molded.
This is the **purest "live" end** of the spectrum the whole batch traverses: literate source
(static) → notebook captured output (snapshot) → GT/Lepiter (live results over files) →
**image (everything live, nothing inherently a file).**
## 2. The boundary: image-as-store is the anti-pattern
The image directly **contradicts** shard-wiki's design constraints, which is exactly why it's
worth recording as a hard boundary:
- **Opaque, monolithic, non-diffable.** The image is one big binary blob of entangled live
state — no per-page identity, no text diff, no mergeable history, no provenance per unit.
It violates *union-without-erasure granularity*, *Markdown-first*, and *git-addressable
coordination*.
- **No stable addressable "page."** Knowledge is an object graph in memory, not addressable
documents — there is nothing to attach at page granularity without an export step.
- **History is a `.changes` log**, a serial source-change stream, not a content history.
**Conclusion (boundary recorded):** an image is **not a shard and not a store**. This is the
generalized form of the rule already hit at Jupyter (UC-84) and GT (T7): *attach the
exported files, never the live image/kernel.* The image can only participate via an **export
projection** (objects/code → files), which is a **derivation-projection** (T1) that
**degrades the liveness to a static snapshot**.
## 3. Pharo's twist: code-as-files (Tonel) → git (the one actionable bit)
Pharo (T8) matters precisely where it **retreats from pure-image**:
- **Tonel / FileTree** serialize each class/method as **plain-text files** in a directory,
and **Iceberg** manages those files as a **git repository** — so Pharo code lives in git as
text, diffable and mergeable, *outside* the image.
- This is the **same move** as Lepiter (T7), nbstripout/Jupytext (T3), and ikiwiki source
(UC-79): **the durable, attachable artifact is the file representation; the live
environment is a layer above it.** It confirms our stance from the *Smalltalk* side: even
the most image-centric tradition externalizes to **git-versionable text** to collaborate.
So Pharo adds **no new page-model idea** beyond "image-based environments still externalize
to git text" — exactly the "keep brief / fold" expectation. Its value is **confirming the
boundary resolution**: attach the **Tonel/git source**, treat the image as live-only.
## 4. INTENT mapping
### Inspiration (keep)
- **Live-object inspection** is the seed of moldable views (T7/UC-54): the *idea* that any
unit can be opened and explained interactively. shard-wiki adopts this as **projection/
view**, not as a storage model.
- **Resume-where-you-left-off** liveness names the far end of the **live↔snapshot** axis the
contract must place every computed/projected view on (UC-83/84): the more live the source,
the more its attached form is a **degrading snapshot**.
### Boundary (enforce — design-bug if violated)
- **Image-as-store is a design-bug boundary.** Never model an image (or any monolithic live
memory blob) as a shard/store; participate only via **export → files** (a degrading
derivation-projection). Generalizes "attach files, not the kernel/image" (UC-84, T7).
### Confirmation (Pharo)
- Even pure-image traditions externalize to **git-versionable text** (Tonel/Iceberg) to
version and collaborate — reinforcing **files-canonical + git coordination** as the
durable substrate; the live environment sits above it.
## 5. UC disposition (enrichment-only — no new UC)
| Mechanism (findings §) | Catalog UC / thread |
|------------------------|---------------------|
| Live-object inspector = generic ancestor of moldable views (§1) | links UC-54, UC-47/48 (T7) |
| Image = opaque monolithic non-diffable blob; not a page/store (§2) | **boundary** for UC-34/UC-35/UC-79 (granularity, identity, files-canonical) |
| Image participates only via export→files = degrading derivation-projection (§2) | links UC-83, UC-84 (live→snapshot) |
| Pharo Tonel/Iceberg: code-as-text in git (§3) | links UC-79, UC-76 (git-canonical text) |
| `.changes` = serial source-change log, not content history (§1) | links UC-36 (history shape) |
Both Squeak and Pharo are **design prior art / boundary**, not candidate shards → **no new
UC**. They sharpen the **live↔snapshot** axis and harden the **image-is-not-a-store**
boundary; Pharo confirms even image traditions externalize to git text.
## 6. Architecture notes for SHARD-WP-0002
- **T14 (attach binding):** record the **image-is-not-a-store** boundary explicitly — a
monolithic live-memory blob is never an attach target; participation is via **export→files**
only. Generalize the "attach files, not the kernel/image" rule (UC-84, GT T7) to a named
boundary in the binding taxonomy.
- **T16 (projection):** add the **live↔snapshot** axis to the projection model — every
computed/projected view sits somewhere between "live (re-derivable on demand)" and "static
snapshot," and the more live the source, the more its attached form must be a clearly-
marked degrading snapshot.
- **T11/T12:** the live-object inspector is the *inspiration* for the moldable view registry
(T7), not a storage shape; nothing new to the page model itself.
## 7. Open questions
1. Is **live↔snapshot** an explicit, first-class metadata axis on every projection (so the
union can label "this view was live / is a snapshot from time T"), or implicit per
capability? (Recurs across UC-83/84, GT, Mathematica `Dynamic`, Strudel T5.)
2. Do we ever ingest a Smalltalk project by attaching its **Tonel/git** repo as an ordinary
git-text shard (no Smalltalk-specific adapter needed), confirming the boundary resolution?
## 8. Sources
- Squeak: `squeak.org`, the *Back to the Future* (Squeak) paper (Ingalls et al.), Morphic;
image/`.changes` model.
- Pharo: `pharo.org`, **Tonel** format, **Iceberg** (git integration), FileTree.
- prior: `research/260614-glamorous-toolkit-deep-dive/` (moldable inspector, Lepiter, T7);
`research/260614-jupyter-deep-dive/` (live→snapshot boundary, UC-84).
## 9. Traceability
**No new UC** (boundary / design prior art; covers **T6 Squeak** and **T8 Pharo** in one
justified-merge memo). Boundary hardened for: UC-34/UC-35/UC-79 (image-is-not-a-store);
links UC-83/UC-84 (live→snapshot), UC-54/UC-47/UC-48 (live-object inspection → moldable
views), UC-76/UC-79 (Pharo Tonel/git text), UC-36 (`.changes` history shape). Architecture
cross-refs: SHARD-WP-0002 T14 (image-is-not-a-store boundary; export→files only), T16
(live↔snapshot projection axis).

View File

@@ -0,0 +1,30 @@
# 260614 — Strudel.cc (live-coding REPL) deep dive
Date: 2026-06-14 · Source: **SHARD-WP-0004 T5**
## What this is
A deep dive into **Strudel.cc** (the JavaScript port of **TidalCycles**): a browser
**live-coding REPL** where terse **pattern source** is **evaluated live into time-based
audio** — code as a running musical performance, with **no document, no output cell, no file
of results**.
## Why it matters
- The **extreme of the live↔snapshot axis** (named at T6): output is **temporal, generative,
performative**, so there is **no faithful static form** — the best static projection is
**source (canonical, diffable) + an optional audio recording snapshot**, marked as one
performance. The **honesty test** for union-without-erasure + graceful degradation.
- Bounds the projection model's live end: ahead-of-time → view-time one-shot → continuous →
**irreducibly live/temporal (recording-only)**.
## Yield
- **No new UC** (enrichment / design prior art; far live end). Enriches **UC-54, UC-55**;
links UC-83/84, UC-37, UC-35.
## Contents
| Path | Role |
|------|------|
| `findings.md` | Code-as-live-performance, the limit of static projection, INTENT mapping, UC disposition (enrichment-only), architecture notes (T16 far end of live↔snapshot axis) |

View File

@@ -0,0 +1,118 @@
# Strudel.cc — live-coding REPL — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0004 T5 · **Subject:** Strudel.cc (the
JavaScript port of TidalCycles) — a browser **live-coding REPL** where code is a **running
musical performance**.
## Why this dive
The closing, lowest-priority dive — and the **extreme** of the live↔snapshot axis. Where
Processing (T4) renders *visual* output at view time, Strudel renders **time-based audio**
that only exists *while running and evolving*. It is the limit case for "can a page be a live
computation?" — the answer where **static projection is least faithful**. Enrichment-only;
feeds UC-54/55 and hardens the snapshot-degradation rule.
## 1. Code as a live, evaluated performance
- **Strudel** ports **TidalCycles**' pattern language to JavaScript, running entirely in the
**browser** (Web Audio). You write **pattern expressions** (e.g. `note("c e g")`,
`sound("bd sd")` with transformations) and **evaluate them live**; the sound updates
**without stopping** — the essence of *live coding*.
- The artifact is **terse source text** (a pattern); the "content" is the **sound it
produces over time**. There is **no document, no output cell, no file of results**
output is **ephemeral, temporal, and performative**.
- A Strudel "page" (a shared REPL link / snippet) is **source + the implicit promise of a
running evaluation**. The source is tiny and diffable; the experience is not capturable as
text.
## 2. The limit of static projection
Strudel pushes past Processing on every "live" dimension:
- **Temporal & generative** — output unfolds over time and may be **non-deterministic**
(randomness, evolving state). There is no single "frame"; the faithful capture is a
**recording (audio), itself just one performance**, not the artifact.
- **Performative** — the value is partly the **act of live editing**; even a recording loses
the live-coding dimension.
- So on the **live↔snapshot axis** (named at T6), Strudel sits at the **far live end**: the
best static projection is **(a) the source** (canonical, diffable) **+ (b) an optional
audio recording snapshot**, explicitly marked as one rendering of a live/temporal artifact.
This makes Strudel the **honesty test** for the contract: shard-wiki must be able to attach
such a source, present it truthfully (here is the source; a live render needs the runtime; a
recording is one performance), and **never imply a static page captures it**.
## 3. INTENT mapping (enrichment-only — no new UC)
### Reinforcements / refinements
- **Live-evaluated, time-based content (UC-54/55).** Strudel is the extreme executable-as-
page: **source canonical, presentation = a temporal live evaluation**. Confirms the page
model must represent content whose rendered form is **time-based / generative / performative**.
- **live↔snapshot axis (T6) — far end.** Establishes the **upper bound**: some content is
**irreducibly live**; static projection degrades to **source + a recording snapshot**, with
honesty about what's lost. Generalizes Processing's "snapshot frame" to "recording of one
performance."
- **Graceful degradation (INTENT).** A backend that can't run the REPL still serves the
**source** (tiny, diffable) and any **recording** as read/projection/backup — the
limited-backend-still-usable rule, at the hardest content type.
- **Union without erasure.** Presenting a Strudel shard must surface "this is a **live
temporal artifact**; what you see/hear statically is **source / one recording**" — never
hide the liveness or imply completeness.
### Boundaries
- shard-wiki is **not an audio/REPL runtime**; default = attach the **source** + offer/store
a **recording** with provenance; live in-viewer evaluation is a gated capability (trust/
sandbox, like Processing T4). Source is canonical; everything rendered is a degrading,
view-time/temporal projection.
## 4. UC disposition (enrichment-only)
| Mechanism (findings §) | Catalog UC / thread |
|------------------------|---------------------|
| Pattern source = live, evaluated, time-based performance (§1) | UC-54 / UC-55 (enriched: time-based/generative executable content) |
| Output ephemeral/temporal/non-deterministic → no faithful static form (§2) | links live↔snapshot axis (T6), far end |
| Best static projection = source + audio recording snapshot, marked as one performance (§2) | links UC-83/UC-84 (degrading projection), UC-37 (recording as backup) |
| Limited backend still serves source + recording (§3) | links UC-37 graceful degradation |
| Live in-viewer evaluation = capability + trust/sandbox (§3) | links UC-35 |
**No new UC** — Strudel is design prior art marking the **far live end** of the projection/
liveness model; it adds no orchestration scenario, it bounds one.
## 5. Architecture notes for SHARD-WP-0002
- **T16 (projection):** anchor the **far end of the live↔snapshot axis** — content can be
**irreducibly live/temporal/generative**; the contract must allow a projection to declare
"no faithful static form; static = source + a marked recording." Combined with T4's
**materialization-timing** and **continuity** facets, the projection model now spans:
ahead-of-time materialized → view-time one-shot → view-time continuous/interactive →
**temporal/generative/performative (recording-only snapshot)**.
- **T12/T15 (page model):** **time-based / generative executable content** as a page-model
edge; source canonical, render temporal.
- **T11 (capabilities):** "live-evaluate (audio/REPL) in viewer" capability + trust/sandbox;
default off → source + recording.
## 6. Open questions
1. Does the contract carry an explicit **"irreducibly live / no faithful static form"** flag
on a projection (so the union renders the honest fallback automatically)? (The far-end
resolution of the live↔snapshot axis open question from T6.)
2. Is a **recording** modeled as a stored **derivation-projection snapshot with provenance**
(one performance, time T, source rev R), reusing the UC-84 snapshot machinery?
## 7. Sources
- `strudel.cc` (docs, REPL); **TidalCycles** (`tidalcycles.org`) — the pattern language
Strudel ports; Web Audio.
- prior: `research/260614-processing-deep-dive/` (view-time render, continuity, UC-54/55);
`research/260614-squeak-pharo-deep-dive/` (live↔snapshot axis);
`research/260614-mathematica-deep-dive/` (`Dynamic` interactive, snapshot-only).
## 8. Traceability
**No new UC** (enrichment / design prior art; the far live end). Enriched: UC-54, UC-55;
links UC-83/UC-84 (degrading projection), UC-37 (recording = backup / graceful degradation),
UC-35 (live-evaluate capability + trust). Architecture cross-refs: SHARD-WP-0002 T16 (far end
of live↔snapshot axis: irreducibly-live content, static = source + marked recording), T12/T15
(time-based/generative executable content), T11 (live-evaluate capability + sandbox).

View File

@@ -0,0 +1,16 @@
# 260614 — TiddlyWiki deep dive
Deep dive on **TiddlyWiki** (TW5): an entire wiki — tiddlers **plus** the app engine — in
**one self-contained HTML file**, the **whole-file write-granularity** anchor of the
synthesis matrix, with a Node.js **file-per-tiddler** (`.tid`) substrate as the git-diffable
alternative, a tiddler/field record model, and **filter expressions** as the native query
language.
- `findings.md` — the single-file model, tiddler data model, dual substrate, filters,
capability profile, INTENT mapping, UC seed (UC-78), architecture notes for SHARD-WP-0002,
open questions, sources, traceability.
Catalog yield: UC-78 (attach a single-file self-contained wiki as one shard — parse tiddlers
out, project; write = rewrite the whole file, the coarsest write-granularity anchor).
Enriched UC-35/40/34/52/43. Feeds SHARD-WP-0002 T11 (write-granularity extreme) and T14
(single-file vs file-per-tiddler binding).

View File

@@ -0,0 +1,164 @@
# TiddlyWiki — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T3 · **Subject:** TiddlyWiki / TiddlyWiki5
(TW5), Jeremy Ruston's self-contained personal wiki.
## Why this dive
The synthesis matrix names **whole-file write granularity** as one extreme of the
write-granularity spectrum, anchored by TiddlyWiki. This dive confirms the anchor and the
*portability extreme* it implies — a wiki that is **a single HTML file you can email** — and
finds the twist: TiddlyWiki also has a **Node.js file-per-tiddler** substrate, so it spans
the granularity spectrum the way Logseq spans file/DB (UC-62). The question: how does
shard-wiki attach a backend whose *entire content is one file*?
## 1. The single-file model
Classic TiddlyWiki ships as **one `.html` file** that contains **both**:
1. the **TiddlyWiki core** (the JavaScript engine, parser, renderer, UI), and
2. **every tiddler** (all content), serialized into the file.
Open it in a browser and the file *is* the running application. There is **no server and no
build step** — the app reconstitutes itself from the file it was loaded from. This is the
portability extreme: a complete, self-hosting wiki in a single, emailable, USB-stick-able
artifact that runs offline anywhere.
**Saving** is the catch: a browser page cannot normally overwrite the file it came from, so
TiddlyWiki uses **"savers"** — TiddlyFox/browser extension, the File System Access API, a
Node.js server, TiddlySpot/put-savers, or "download a new copy." Crucially, **a save
rewrites the *entire* HTML file** (core + all tiddlers re-serialized). Hence **whole-file
write granularity**: there is no concept of writing one page in isolation in the
single-file mode — every save touches the whole artifact.
## 2. The tiddler data model
The atomic unit is the **tiddler** — a named record with **fields**:
- Core fields: **`title`** (the identity), **`text`** (the body), **`tags`**, **`created`**,
**`modified`**, **`type`** (content type, e.g. `text/vnd.tiddlywiki`, `text/markdown`),
plus **arbitrary custom fields** (any key→value). A tiddler is effectively a **flexible
flat record** — closer to a typed-field record (UC-34) than to prose-with-frontmatter.
- **Everything is a tiddler**: not just pages, but tags, macros, templates, themes, plugins,
and the wiki's own configuration are all tiddlers. A **plugin is a bundle of tiddlers**.
- Content markup is **WikiText** (TW5's own), though `type` can mark a tiddler as Markdown,
JSON, image, etc. **Transclusion** is native: `{{SomeTiddler}}` embeds another tiddler;
`{{SomeTiddler!!field}}` embeds a field.
## 3. The dual substrate — single-file vs file-per-tiddler
TiddlyWiki on **Node.js** stores each tiddler as a **separate `.tid` file** on disk: a small
text file with a header of `field: value` lines, a blank line, then the body. The Node
server assembles these into the same wiki at serve time. This substrate is:
- **git-diffable and fine-grained** — one file per tiddler, line-level diffs, per-tiddler
history — the *opposite* end of the granularity spectrum from the single HTML file.
- the natural attach surface for a *versioned, multi-author* TiddlyWiki.
So TiddlyWiki **spans the write-granularity spectrum by substrate** (single-file = whole-file
write; Node = file-per-tiddler write), exactly as Logseq spans file/DB (UC-62) and as the
backend-swap question (UC-43) anticipates.
## 4. Native query — filter expressions
TiddlyWiki's query language is **filter expressions** over tiddler fields, e.g.
`[tag[todo]!tag[done]sort[modified]]` — a compact DSL that selects/orders tiddlers by field
and tag. Lists, tables, and dynamic views are built from filters. This is a **native-query
capability** (UC-52 tier) — less expressive than SPARQL/datalog but real, and computed over
the tiddler store.
## 5. Capability profile
| Dimension (synthesis spectrum) | TiddlyWiki (single-file) | TiddlyWiki (Node `.tid`) |
|--------------------------------|--------------------------|--------------------------|
| Attachment mode | **file-store: one HTML file** | **file-store: dir of `.tid` files** |
| Addressing granularity | tiddler (`title`) within the file | tiddler = one file |
| Content identity | **`title`** (placement-bound) | title ↔ filename |
| Structure | flat record store w/ arbitrary **fields** + tags | same |
| History | none in-file (whole-file save) | **per-file git history** |
| Merge model | whole-file replace (no merge) | git 3-way per tiddler |
| Native query | **filter expressions** | filter expressions |
| Translation | WikiText (or per-tiddler `type`: markdown/json/…) | same |
| **Write granularity** | **whole file** (the anchor) | **file per tiddler** |
| Operational envelope | trivial — a browser; no server | a Node server |
| Access grant | file access = full access | server/file perms |
| Content opacity | transparent (parse the HTML store) | transparent text |
| Provenance | created/modified fields | git + fields |
## 6. INTENT mapping
### Reinforcements
- **Graceful degradation**: a single-file TiddlyWiki is a *trivial* read-only / projection /
backup shard — parse the tiddlers out of the HTML, project pages; no server needed. The
limited-backend-still-usable principle at its simplest.
- **Markdown-first but backend-neutral**: tiddlers carry a `type`, so Markdown tiddlers
coexist with WikiText — the page model's content-type field maps directly.
- **Typed fields** (UC-34): arbitrary tiddler fields are a flexible record model the page
model already accommodates.
- **Backend-swap under stable identity** (UC-43): single-file ↔ Node `.tid` is the same
logical wiki on two substrates — the migration UC-43 anticipates, within one engine.
### Divergences (boundaries / notes)
- **Whole-file write granularity** is a real constraint: in single-file mode shard-wiki
cannot write one page atomically — an overlay applied "to one page" still **rewrites the
whole file** (T11). This is the coarsest write tier; model it explicitly so overlays/locks
account for it (a write to any page conflicts with any concurrent write).
- **Identity = title**, file-local; cross-shard identity (T16) layered above.
- **The app is in the file**: when parsing a single-file TiddlyWiki, shard-wiki must extract
the **tiddler store** and ignore the embedded engine — i.e. treat the HTML as a *container
format*, not as page content (don't mistake the app for content).
### What to keep
1. **Single-file self-contained wiki as a first-class file-store shard** — container-format
parse, whole-file write granularity (UC-78); the portability/granularity extreme.
2. **Whole-file write granularity as a named tier** (T11) with overlay/lock implications.
3. **Dual-substrate binding** (single-file vs `.tid` dir) as another instance of
substrate-choice under one identity (UC-43/UC-62).
## 7. UC seed
| # | Seed | Disposition |
|---|------|-------------|
| UC-78 | Attach a **single-file self-contained wiki** (TiddlyWiki HTML) as one shard — parse tiddlers out of the container, project pages; **write = rewrite the whole file** (whole-file write granularity, the coarsest tier) | **new** |
| — | whole-file write granularity anchor + overlay/lock implications | enrich **UC-35** |
| — | single HTML file as a file-store shard (container format) | enrich **UC-40** |
| — | tiddler arbitrary fields = flexible record | enrich **UC-34** |
| — | filter expressions as a native-query tier | enrich **UC-52** |
| — | single-file ↔ Node `.tid` substrate swap | enrich **UC-43** |
## 8. Architecture notes for SHARD-WP-0002
- **T11 (capability / write granularity):** confirm **whole-file** as the coarsest named
write tier (anchored by single-file TiddlyWiki), with the implication that an overlay to
*any* page conflicts with concurrent writes (no per-page atomicity). File-per-tiddler is
the fine tier on the same engine.
- **T14 (attach binding):** a single-file wiki binds as a **container-format file-store**
(parse tiddler store, ignore embedded engine); a Node TiddlyWiki binds as a **dir of
`.tid` files** (git-diffable). One engine, two bindings — parameterize like UC-43.
- **Native query:** filter expressions are a low-mid native-query tier between "none" and
datalog/SPARQL — delegate where present (UC-52).
## 9. Open questions
1. In single-file mode, how does shard-wiki represent **per-page overlays** when writes are
whole-file — buffer overlays and re-serialize, or require the Node `.tid` substrate for
write-through and treat single-file as read/projection/backup only?
2. Is a single-file TiddlyWiki's **embedded plugins/config** ever relevant to the union, or
strictly ignored as app-internals (parse only content tiddlers)?
3. Does shard-wiki expose tiddler **filter expressions** as a delegated query, or only its
own union query over projected tiddlers?
## 10. Sources
- TiddlyWiki.com — *Tiddlers*, *TiddlerFields*, *Filters*, *Saving*, *Node.js* docs
- *TiddlyWiki5* GitHub (Jermolene/TiddlyWiki5) — `.tid` file format, store structure
- prior: `research/260614-logseq-deep-dive/` (file/DB dual substrate, UC-62)
## 11. Traceability
New UC **UC-78** carries the marker **⊡** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-35, UC-40, UC-34, UC-52, UC-43. Architecture
cross-refs: SHARD-WP-0002 T11 (whole-file tier), T14 (dual binding).

View File

@@ -0,0 +1,53 @@
# 260614 — Trilium (TriliumNext) deep dive (note cloning, attribute inheritance, HTML-native)
Date: 2026-06-14
## What this is
A focused study of **Trilium / TriliumNext** — the open-source hierarchical personal
knowledge base — read through shard-wiki's lens. It is another SQLite-local note app, but
it brings a structural feature **none of the prior twelve systems had: note cloning**
a single note can sit in **multiple places in the tree at once**, so the hierarchy is a
**DAG, not a tree**, with **note identity cleanly separated from placement** (a note has
many "branches"). It also has an **attribute system with inheritance + templates**
(computed metadata) and is one of the few **HTML-native** (not Markdown) tools.
Lineage: **TriliumNext** is the community fork of the original **Trilium** (zadam, now
maintenance-only) — the same pattern as TWiki→Foswiki.
Distinctive material:
- **Architecture** — single **SQLite** file (`document.db`); desktop or self-host server;
multi-instance **sync protocol w/ conflict resolution** + WebSocket; 12-char IDs
(`noteId`/`branchId`/`attributeId`/`attachmentId`)
- **Note cloning** — note vs **branch** (identity vs placement); DAG hierarchy; no single
canonical path
- **Attributes** — labels (`#tag`) + typed relations (`~relation`), **inheritable** +
**templates** (`~template`) → effective vs own metadata
- **Content** — **HTML** (CKEditor5), plus canvas/mindmap/spreadsheet/geo/render note
types; export to MD/HTML (lossy)
- **Extension** — **scripting** (frontend/backend code notes, Script API) + **ETAPI**
(token REST); **per-note encryption** (protected notes)
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, note cloning/DAG, attributes+inheritance, HTML content, scripting/ETAPI, per-note encryption, capability profile, INTENT mapping, UC seeds, sources |
## Status
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-66
attach a shard with a DAG hierarchy / note cloning, identity separated from placement;
UC-67 preserve inherited/templated attributes — effective vs own metadata); UC-15/22/34/
38/42/61 enriched. Logged for `SHARD-WP-0002` (T11/T12/T14/T15/T16): DAG namespace +
identity/placement (note/branch) split, computed/inherited metadata in the page model,
**per-item** content opacity (refining the 12th spectrum), HTML as a source content model,
and scripting + ETAPI host surfaces.
**Key takeaways recorded:** Trilium's **note/branch split (identity ≠ placement)** is the
clean model for pages that live in multiple locations/shards and the namespace-level form
of the clone/reference primitive; metadata is **computed** (own + inherited + template),
not a flat bag; and **content opacity is per-item** (per-note encryption), not only
whole-shard. **Boundary:** one SQLite-local candidate shard (DAG hierarchy, HTML-native),
best attached via ETAPI; not a substrate and not the federation layer.
</content>

View File

@@ -0,0 +1,274 @@
# Findings — Trilium (TriliumNext): note cloning, attribute inheritance, HTML-native
Date: 2026-06-14
Source kind: **modern shipped product** — an open-source hierarchical PKB; a *candidate
shard* whose distinctive traits are **note cloning (a DAG hierarchy)**, an
**attribute/relation system with inheritance + templates**, and **HTML-native** content
Lens: shard-wiki — the namespace model (identity vs placement), inherited/computed
metadata, HTML translation, per-note encryption, and the scripting/ETAPI surfaces
> Why Trilium earns a dive. It is another SQLite-local note app, but it brings a
> structural feature **none of the prior twelve systems had: note cloning** — a single
> note can sit in **multiple places in the tree at once**, so the hierarchy is a **DAG,
> not a tree**, and **note identity is cleanly separated from placement** (a note has
> many "branches"). That directly challenges shard-wiki's namespace model (UC-22 assumes
> a path) and is the clone/reference primitive made concrete at the *namespace* level.
> Trilium also has an **attribute system with inheritance and templates** (effective
> metadata is computed, not just stored), and it is one of the few **HTML-native** (not
> Markdown) tools — a useful stress on "Markdown-first must degrade."
Lineage note (like TWiki→Foswiki): **TriliumNext** is the community fork of the original
**Trilium** (zadam) after it went maintenance-only; the active project is TriliumNext.
Contrast set: Joplin (SQLite-local, files-on-sync, page-level), Logseq (block-graph on
files), Notion (hosted DB, schema+relations). Trilium = **SQLite-local, DAG hierarchy,
inherited attributes, HTML content, self-host sync + ETAPI**.
---
## 1. Core architecture — one SQLite file, server-syncable
- **Storage:** a single **SQLite** file (`document.db`, via better-sqlite3) holding
notes, attachments, history, and settings. Local DB store (like Joplin; not files).
- **Clients/sync:** desktop (Electron) or a self-hosted **server**; multi-instance
**sync protocol** with **conflict resolution + entity change tracking**, plus
**WebSocket** realtime updates. Attach via the server, the ETAPI (§5), or the DB.
- **IDs:** 12-char IDs — **`noteId`, `branchId`, `attributeId`, `attachmentId`**. The
**note vs branch** split is the key idea (§2).
- **Export:** to Markdown/HTML (lossy — content is HTML, §4).
---
## 2. Note cloning — a DAG hierarchy, identity separated from placement
Notes form an arbitrarily deep tree, **but a single note can be placed in multiple
locations** — "cloning," implemented via the **branches** model:
- A **note** (`noteId`) is the content + identity. A **branch** (`branchId`) is *one
placement* of that note under a parent (with an optional **branch prefix** giving
per-location context). A note with several branches is **cloned** — it appears in
several places at once; editing it anywhere edits the one note.
- So the hierarchy is a **DAG**, not a tree, and **identity (note) is separated from
location (branch)**. There is no single canonical path for a cloned note.
This is the most shard-wiki-relevant feature in the dive:
- It breaks the assumption behind **UC-22** (resolve a page by *a* path). shard-wiki's
namespace model must allow **a page in multiple namespace locations** with no single
canonical path (UC-66).
- The **note/branch split** is a clean model shard-wiki should borrow: **page identity ≠
page placement**. A page is one entity; its locations are separate placement records —
exactly how shard-wiki should treat a page that appears under multiple paths/shards.
- It is the **clone/reference-not-copy primitive** (Xanadu clone, ZigZag clone, UC-44/45,
T16) realized at the *namespace* level: same note, many positions, one source of truth.
---
## 3. Attributes — labels + relations, with inheritance and templates
Trilium's metadata system is **attributes**:
- **Labels** (`#tag`, optionally `#key=value`) — typed metadata on a note.
- **Relations** (`~relation`) — **typed links to other notes** (a knowledge graph).
- **Inheritance** — attributes can be **inherited down the subtree** (inheritable
attributes), and **templates** (a note whose attributes/structure are applied to
instances via a `~template` relation) inject attribute sets. So a note's **effective
metadata = its own attributes + inherited + templated** — computed, not just stored.
- Promoted attributes give a form-like UI; attributes drive **search/queries** and
**scripting**.
shard-wiki read: this is structured data (UC-34) and typed relations (UC-58), but with a
**new wrinkle — inheritance/templates make metadata *computed***. Projecting such a shard
must distinguish **effective vs own** attributes (and record their provenance: own /
inherited-from / template), not flatten them (UC-67). Templates also reinforce blueprint
pages (UC-15).
---
## 4. Content model — HTML-native, many note types
- **Text notes are HTML** (WYSIWYG via **CKEditor5**), **not Markdown**. Trilium is one
of the few HTML-first tools studied. Markdown participation therefore needs **HTML↔
Markdown translation** — more tractable than Notion's block model but still **lossy**
for some constructs (CKEditor features, includes) → a fidelity-aware case bridging
UC-42 (lossless ideal) and UC-59 (lossy-with-report).
- **Note types:** text (HTML), code (CodeMirror), **canvas (Excalidraw)**, relation
maps, mind maps (Mind Elixir), spreadsheets (Univer), geo maps (Leaflet), **render
notes**, file/image, book — i.e. lots of **non-Markdown content** (UC-55) and
**script/render-generated dynamic notes** (UC-54).
---
## 5. Extension surfaces — scripting (in-app) + ETAPI (external)
Two surfaces, like Joplin:
- **Scripting (in-engine host).** **Code notes** in JS run as **frontend** (UI widgets,
buttons) or **backend** (server-side automation) scripts against a **Script API**
(create/query notes, attributes, etc.). Executable, in-app — the adapter-host path
(UC-38). "Render notes" + scripts produce **dynamic, generated content** (UC-54).
- **ETAPI (external REST API).** Trilium's public REST API (since v0.50), **token auth**
(`Bearer ETAPITOKEN`), with client libs (`trilium-py`): CRUD over notes/branches/
attributes/attachments, search, import/export — an external attach surface (UC-57)
that, unlike Joplin's localhost-only Data API, is the **designed external integration
API** for a (often self-hosted) server.
---
## 6. Security — per-note (partial) encryption
Trilium offers **strong per-note encryption** ("protected notes") unlocked in a
**protected session** (password; TOTP/OpenID for login). Crucially this is **per-note**:
a shard can hold **some encrypted and some plaintext notes at once**. This refines the
**content-opacity** dimension (UC-61, proposed twelfth spectrum from Joplin/Anytype):
opacity is **per-item, not only whole-shard** — the adapter must handle a shard where
*part* of the content is opaque without a key (protected notes projectable only as
structure-shell; unprotected notes fully).
---
## 7. Trilium as a shard — capability profile
| Capability | Trilium | Notes for the adapter contract |
|------------|---------|--------------------------------|
| Read | yes (ETAPI / sync / DB) | ETAPI REST (server) is the clean surface |
| Write | yes (ETAPI / script) | per-note; via ETAPI or backend code notes |
| Write granularity | per-note (page) | — |
| Identity / addressing | **`noteId` + `branchId`** (12-char) | **identity ≠ placement**; cloned notes have many branches (UC-51, UC-66) |
| Hierarchy | **DAG (cloning)** | a note in multiple locations; no single canonical path (UC-66) |
| Structure | **labels + relations, inherited + templated** | effective vs own metadata (UC-34, UC-58, UC-67) |
| Content | **HTML (CKEditor)** + many types | HTML↔MD lossy translation (UC-42/59); non-MD types (UC-55) |
| History | internal revisions in the DB | not portable git → supplement (UC-36) |
| Native query | attribute search / script queries | delegate or build index (UC-52/63) |
| Subscribe | WebSocket + sync protocol | push; conflict resolution built in (UC-31) |
| Content opacity | **per-note encryption (partial)** | per-item opacity (UC-61 refined) |
| Extension | **scripting (in-app) + ETAPI (external)** | dual surface (UC-38, UC-57) |
| Templates | yes (`~template`) | blueprint pages (UC-15) |
Verdict: a capable **SQLite-local, server-syncable** shard best attached via **ETAPI**.
Standout demands: the **DAG hierarchy / note-branch identity model** (UC-66) and
**inherited/templated attributes** (UC-67); plus HTML-native translation, per-note
opacity, and the scripting host.
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **Identity ≠ placement** (note/branch) is a model shard-wiki should adopt for pages
that appear under multiple paths or in multiple shards — provenance and union without
erasure depend on separating "what a page is" from "where it sits."
- **Typed relations + templates** validate the structured/relation demand (UC-58) and
blueprints (UC-15) on a local-first, self-hostable backend.
- **ETAPI** is a clean example of a **designed external REST surface** for a self-hosted
server (cf. Notion external API, but self-hosted/trust-boundary friendly).
### 8.2 Deliberate divergences (design bugs if conflated)
1. **Don't force a single canonical path.** A cloned note has many placements; modeling
it with one path loses information (UC-66). Use the note/branch separation.
2. **Don't flatten computed metadata.** Effective attributes include inherited/templated
values; record provenance (own vs inherited vs template), don't collapse (UC-67).
3. **HTML-native, not Markdown.** Translate HTML↔Markdown with a fidelity report; degrade
to read-only where lossy (UC-42/59/03). Markdown-first must degrade gracefully.
4. **Per-note opacity.** Some notes are encrypted; never surface protected-note
ciphertext; project them as structure-shell (UC-61).
5. **One Trilium instance = one shard**, not the federation layer; its sync protocol is
its own — attach via ETAPI/replica, don't re-drive it (not-a-sync-daemon).
### 8.3 What Trilium teaches that shard-wiki should keep
- **Separate page identity from placement** (note vs branch) — the cleanest model for
multi-location / multi-shard pages and for the clone/reference primitive (T16).
- **Model metadata as computed** (own + inherited + templated) with per-attribute
provenance (UC-67) — not a flat key/value bag.
- **Content opacity is per-item**, not only whole-shard (UC-61 refinement).
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-65**. New UCs **UC-66, UC-67** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach a shard with a DAG hierarchy / note cloning** — a page may occupy multiple namespace locations at once (note identity separated from placement via branches); no single canonical path | **UC-66 (new)** |
| **Preserve inherited / templated attributes** — project a structured shard whose metadata is computed (own + inherited + template), distinguishing effective vs own with per-attribute provenance | **UC-67 (new)** |
| DAG/multi-parent vs tree; note/branch (identity vs placement) | **enriches UC-22** |
| Labels (`#tag`) + typed relations (`~relation`); HTML-native | **enriches UC-34** |
| Templates (`~template`) inject attribute sets | **enriches UC-15** |
| HTML (CKEditor) content → HTML↔Markdown lossy translation | **enriches UC-42** (links UC-59) |
| Per-note encryption (protected notes) = **partial** content opacity | **enriches UC-61** |
| Scripting (frontend/backend code notes, Script API) = in-app host | **enriches UC-38** |
| ETAPI (token REST) = designed external surface for a self-host server | links UC-57 |
| Render/script notes + attribute queries = dynamic content | links UC-54; typed relations link UC-58 |
| `noteId`/`branchId` 12-char IDs | links UC-51 |
---
## 10. Architecture notes for SHARD-WP-0002 (no UC)
- **Namespace model must support a DAG** and **separate page identity from placement**
(note/branch). A page is one entity with N placements (paths/shards). Feeds the page/
namespace model and the clone/reference primitive. (T12, T16.)
- **Computed/inherited metadata** (UC-67): the page model's structured-data
representation must carry **effective vs own** with per-attribute provenance
(own/inherited/template), not a flat bag. (T12.)
- **Content opacity is per-item** (UC-61 refinement): the content-opacity capability
(proposed twelfth spectrum) should be **granular** (per-note), not only whole-shard.
(T11.)
- **HTML as a source content model** joins TML/Notion-blocks in the translation
capability (HTML↔Markdown, lossy-aware). (T15.)
- **Scripting as an in-engine host** + **ETAPI external REST** are two more adapter-host
exemplars (with Roam/Obsidian/Joplin). (T14.)
---
## 11. Open questions (for spec / workplans)
1. How does shard-wiki represent a **cloned note** in the union — one page with multiple
path placements, or a page transcluded into multiple locations? (UC-66 vs UC-44/45.)
2. When projecting **inherited attributes** (UC-67), does shard-wiki materialize
effective values (snapshot) or compute them live from the shard's tree/templates?
3. Is **HTML↔Markdown** round-trip lossless enough for write-back overlays, or are
Trilium overlays read-only/native-HTML (cf. UC-42 Q2)?
4. For **per-note encryption** (UC-61), is a partially-opaque shard projected with
protected notes as visible-but-opaque placeholders, or hidden entirely?
---
## 12. Sources
| Source | Used for |
|--------|----------|
| TriliumNext/Trilium — DeepWiki (https://deepwiki.com/TriliumNext/Trilium) | SQLite `document.db`/better-sqlite3; tree + cloning via branches; note types; sync |
| Cloning notes — TriliumNext wiki (https://github.com/TriliumNext/Trilium/wiki/Cloning-notes) | Note in multiple locations; branches; branch prefixes |
| TriliumNext — DeepWiki API & Synchronization (https://deepwiki.com/TriliumNext/Trilium/4.3-api-and-synchronization) | WebSocket + sync protocol w/ conflict resolution; 12-char IDs (note/branch/attribute/attachment) |
| ETAPI (REST API) — Trilium docs (https://docs.triliumnotes.org/User%20Guide/User%20Guide/Advanced%20Usage/ETAPI%20(REST%20API)/) | Public REST API since v0.50; token/Bearer auth; trilium-py |
| Script API — TriliumNext wiki (https://github.com/TriliumNext/Trilium/wiki/Script-API) | Frontend/backend code notes; Script API |
| BrightCoding — TriliumNext overview (https://www.blog.brightcoding.dev/2025/09/20/triliumnext-notes...) | Attributes (labels/relations), inheritance, templates, per-note encryption, note types |
Cross-references: `research/260614-joplin-deep-dive/findings.md` (SQLite-local, dual
surface, content opacity), `research/260614-notion-deep-dive/findings.md` (typed
relations, external API), `research/260614-zigzag-deep-dive/findings.md` (clone /
dimensions), `research/260614-shard-spectrum-synthesis/findings.md` (spectra this
refines), `spec/UseCaseCatalog.md` (UC-15, UC-22, UC-34, UC-38, UC-42, UC-51, UC-54,
UC-57, UC-58, UC-61), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T12,
T14, T15, T16).
---
## 13. Traceability
- New UCs: **UC-66, UC-67**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-15, UC-22, UC-34, UC-38, UC-42, UC-61** (links UC-51, UC-54, UC-57,
UC-58, UC-59).
- Architecture (no UC): DAG namespace + identity/placement (note/branch) split;
computed/inherited metadata; per-item content opacity; HTML source model; scripting +
ETAPI host surfaces → `SHARD-WP-0002` (T11, T12, T14, T15, T16).
- Boundary recorded: Trilium (TriliumNext) is **one SQLite-local candidate shard** with a
DAG hierarchy and computed metadata, best attached via **ETAPI**; HTML-native (lossy to
Markdown), per-note opacity; not a substrate, not the federation layer (INTENT
graceful-degradation, no-silent-mutation, not-a-sync-daemon).
</content>

View File

@@ -0,0 +1,14 @@
# 260614 — UseModWiki deep dive
Deep dive on **UseModWiki**: the **flat-file ancestor** — a Perl CGI wiki (Clifford Adams,
2000) descended from AtisWiki/CvWiki, **CamelCase** linking, plain flat-file page + history
storage, and the **engine Wikipedia originally ran on** (MediaWiki Phase I). Origins/lineage
value: the minimal flat-file page+history model the whole field descends from.
- `findings.md` — the model, lineage, capability profile, INTENT mapping, enrichments (no new
UC — reinforces the minimal flat-file baseline UC-82), architecture notes, sources,
traceability.
Catalog yield: **enrichment-only** (reinforces UC-82) — enriched UC-01 (open wiki), UC-40
(flat-file store), UC-25 (CamelCase naming), UC-36/41 (flat-file history). Lineage noted for
the origins record. Feeds SHARD-WP-0002 T11 (minimal profile lineage).

View File

@@ -0,0 +1,87 @@
# UseModWiki — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T9 · **Subject:** UseModWiki, Clifford
Adams's flat-file Perl wiki — the lineage root of much of the field.
## Why this dive
This is a **lineage** dive, not a new-capability one. UseModWiki is the **ancestor**: the
minimal flat-file page+history wiki the c2-era engines and **MediaWiki Phase I** descend
from (Wikipedia ran on UseModWiki, 20012002). It pairs with
`research/260608-c2-wiki-origins/` to record *where the page+history model came from*. It
adds no new shard capability beyond the minimal flat-file floor (UC-82, Oddmuse) — its value
is **historical grounding** and confirming the floor is genuinely the field's common root.
## 1. The model
- **Perl CGI** (`wiki.pl`), single-script, descended from AtisWiki → CvWiki → UseModWiki
(c. 2000), by Clifford Adams.
- **Flat-file storage**: each page stored as a text file (under a `db/` data directory),
with **page history** kept (older revisions retained as files / in the page record). No
database.
- **Linking**: **CamelCase** (`WikiWord`) auto-links by default; later added **free links**
`[[Like This]]`. This is the canonical CamelCase-naming lineage (UC-25).
- **Open editing**, recent-changes, simple markup — the c2 ethos in a portable script.
## 2. Lineage
- **MediaWiki Phase I** *was* UseModWiki — early Wikipedia ran on it before the PHP rewrite
(Phase II/III). So UseModWiki is the **direct ancestor of MediaWiki/Wikibase** (T2) and a
sibling-root of the c2 tradition.
- The **flat-file page+history model** here is the shape Oddmuse (UC-82), TWiki's file store
(UC-40), and others elaborate — confirming the **minimal file-store floor** is the field's
common origin, not a modern simplification.
## 3. Capability profile
Essentially **identical to the minimal floor** (UC-82, Oddmuse):
| Dimension | UseModWiki |
|-----------|------------|
| Attachment mode | **file-store** (flat files under `db/`); CGI, no API |
| Addressing | page = file; **CamelCase** name = identity |
| Structure | flat page space; CamelCase link graph |
| History | flat-file retained revisions (may be limited) |
| Native query | none |
| Translation | simple wiki markup → Markdown (lossy) |
| Write granularity | page (file) |
| Access | open editing (optional admin password) |
| Provenance | timestamp, optional username |
## 4. INTENT mapping
- **Reinforces the minimal flat-file baseline** (UC-82): UseModWiki is the *historical*
instance of the graceful-degradation floor — attach via its flat files; partial history
surfaced honestly.
- **CamelCase naming** (UC-25): the canonical source of WikiWord auto-linking — the page
model's name/identity and link-resolution must accommodate CamelCase-derived identities.
- **Open wiki** (UC-01): the c2 open-editing ethos at the root.
- **Lineage grounding**: confirms shard-wiki's "Git-based Markdown" page model descends from
(and must remain attach-compatible with) the flat-file ancestor.
**No new UC** — UseModWiki adds historical grounding, not a new orchestration scenario;
its capabilities are subsumed by UC-82 (minimal flat-file baseline) plus UC-25 (CamelCase).
## 5. Architecture notes for SHARD-WP-0002
- **T11:** UseModWiki = a second instance of the **minimal/floor profile** (with Oddmuse);
confirms the floor is the field's common root, so the floor profile is the right baseline.
- **Naming/identity:** CamelCase-derived page identities (UC-25) are part of the legacy
identity surface the adapter must parse (and translate to the page model's identity).
## 6. Open questions
(None new — covered by UC-82 partial-history honesty, catalog open-Q30, and the c2 origins
record. CamelCase resolution is UC-25.)
## 7. Sources
- UseModWiki — usemod.com/cgi-bin/wiki.pl (the wiki about itself); Wikipedia: *UseModWiki*,
*MediaWiki* (Phase I history)
- prior: `research/260608-c2-wiki-origins/`; `research/260614-oddmuse-deep-dive/` (minimal
floor, UC-82)
## 8. Traceability
**No new UC** (reinforces UC-82). Enriched: UC-01, UC-40, UC-25, UC-36, UC-41. Lineage noted
for the origins record. Architecture cross-ref: SHARD-WP-0002 T11 (minimal-profile lineage).

View File

@@ -0,0 +1,16 @@
# 260614 — Wikibase / Wikidata deep dive
Deep dive on **Wikibase** (the MediaWiki extension behind **Wikidata**) as the
**entity-statement (RDF) far end** of the structure and native-query spectra: items and
properties, statements = claim + qualifiers + references + rank, stable opaque IDs, and
**SPARQL** (Wikidata Query Service / Blazegraph) including **federated** queries.
- `findings.md` — data model, RDF/SPARQL surface, storage & history, capability profile,
INTENT mapping, UC seeds (UC-73UC-75), architecture notes for SHARD-WP-0002, open
questions, sources, traceability.
Catalog yield: UC-73 (attach a typed entity-statement / RDF shard, lossy render to a page),
UC-74 (graph query the union via SPARQL + federated `SERVICE` cross-endpoint query), UC-75
(statement-level provenance — references + rank per assertion). Enriched UC-34/58/52/24.
Feeds SHARD-WP-0002 T12 (structured/typed page model) and T16 (stable language-neutral
identity ≠ label/placement).

View File

@@ -0,0 +1,200 @@
# Wikibase / Wikidata — deep dive (findings)
**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T2 · **Subject:** Wikibase (MediaWiki
extension) and its flagship instance **Wikidata**, incl. the Wikidata Query Service (SPARQL).
## Why this dive
Every structured shard so far tops out at *typed records in a database*: Notion's
database-pages, XWiki's XObjects/classes, Trilium's typed relations, Roam/Logseq's
attribute blocks. Wikibase is a *different kind of structure altogether* — a **typed
knowledge graph of entities and provenance-bearing statements**, queried with **SPARQL**
over an **RDF** projection. It is the **far end of the structure spectrum** and of the
**native-query spectrum**, and it pushes **provenance down to the individual assertion**.
The question for shard-wiki: what does a shard look like when its "page" is *not prose but a
set of statements*, and what does the page model / adapter contract owe such a shard?
## 1. The data model — entities, statements, snaks
**Entities** are the top-level objects, each on its own MediaWiki page with a **stable
opaque ID**:
- **Item** — `Q42`. Has multilingual **labels / descriptions / aliases**, a set of
**statements**, and **sitelinks** (links to wiki articles). The label is *annotation*,
not identity — `Q42` is the identity, "Douglas Adams" is just its English label.
- **Property** — `P31` ("instance of"). Also has labels/descriptions/aliases, plus a
**fixed datatype** constraining its values (item-reference, string, time,
globe-coordinate, quantity, monolingual-text, url, external-id, commons-media, …).
**Statement** = the unit of assertion on an item. Structure:
```
statement = claim + references[] + rank
claim = mainSnak + qualifiers[]
snak = property + snaktype + (value) # snaktype ∈ value | somevalue | novalue
```
- **Main snak** — the core property→value assertion (e.g. `P31``Q5` "human").
- **Qualifiers** — snaks that *contextualize* the claim without being the subject (validity
time, "as of", determination method, units). E.g. *population (P1082) = 8.4M, **point in
time (P585) = 2020***.
- **References** — lists of snaks citing **where the claim comes from** (a source item, a
URL, a page number). **Provenance attached to the individual statement, not the page.**
- **Rank** — `preferred` | `normal` | `deprecated`: relative importance among same-property
statements (lets multiple, even contradictory, values coexist with a curation signal —
the structured analogue of fedwiki's "chorus").
- Each statement carries a **stable GUID** (`Q42$<uuid>`), so statements are individually
addressable.
`somevalue` (known to exist, value unknown) and `novalue` (known *not* to have a value) are
**first-class** — the model represents *known-unknowns* explicitly, which prose and most
DBs cannot.
## 2. The RDF / SPARQL surface
Wikibase **projects entities to RDF**; the **Wikidata Query Service (WDQS)** is a
**Blazegraph** triple store exposing a **SPARQL** endpoint. The projection is deliberately
multi-layered:
- **Truthy** triples (`wdt:` prefix) — the simple "best" value, for easy queries:
`wd:Q42 wdt:P31 wd:Q5`.
- **Full** statements — reified so qualifiers/references/rank survive: `wd:Q42 p:P31
?stmt . ?stmt ps:P31 wd:Q5 ; pq:P585 ?time ; prov:wasDerivedFrom ?ref`. (`p:`=statement
node, `ps:`=statement value, `pq:`=qualifier, `pr:`/`prov:`=reference.)
- **Federated SPARQL** — the `SERVICE <endpoint> { … }` keyword runs a sub-query against
*another* SPARQL endpoint and joins the results. **Query-level federation is built into
the query language** — a different federation primitive from fedwiki's fork/neighborhood.
- **EntitySchemas / ShEx** — schemas (`E`-ids) that *validate* an entity's shape (Shape
Expressions). Optional, declarative structure validation over the open graph.
## 3. Storage, identity, history
- **Storage:** each entity is a **JSON blob stored as a MediaWiki page** (`Item:` /
`Property:` content model). The RDF/SPARQL store is a **derived index** rebuilt from these
canonical JSON entities (an *update stream* feeds WDQS) — exactly shard-wiki's
"derived query index over a canonical store" pattern (UC-63), at planet scale.
- **Identity:** the **opaque Q/P/L IDs are the identity**, fully decoupled from
human-readable labels and from language. This is the cleanest real-world instance of
**stable, language-neutral identity ≠ label/placement** — a strong reinforcement of our
identity model (T16).
- **History:** because each entity is one MediaWiki page, history is **page-level MediaWiki
revisions** — every edit is a full-entity JSON snapshot with author/timestamp/comment.
*Coarse* history granularity (whole entity per revision), but the **edit API is
fine-grained** (`wbsetclaim`, `wbeditentity` patch individual statements). So: **fine
write API over a coarse history unit** — a distinct point on the write/history spectra.
## 4. Capability profile
| Dimension (synthesis spectrum) | Wikibase / Wikidata |
|--------------------------------|---------------------|
| Attachment mode | **external-API** (MediaWiki Action API + REST) **and** a derived **SPARQL endpoint**; self-hostable |
| Addressing granularity | **statement** (each has a GUID) within an **entity** (Q/P id) |
| Content identity | **stable opaque ID** (Q/P/L); labels are multilingual annotations |
| Identity vs placement | **fully separated** — identity is language- and label-neutral |
| Structure | **typed knowledge graph**: entities + statements (claim+qualifiers+refs+rank) |
| History | **page-level revisions** (whole-entity JSON snapshots); fine-grained edit API |
| Merge model | MediaWiki last-writer / edit-conflict; rank lets contradictory values coexist |
| Native query | **SPARQL** (RDF) + **federated `SERVICE`** cross-endpoint join — the far end |
| Translation | **not Markdown** — content *is* statements; render to prose is a lossy projection |
| Attachment/write granularity | **statement-level writes** via API; coarse history unit |
| Operational envelope | huge derived index (Blazegraph), rate-limited public endpoints |
| Access grant | open read; MediaWiki user/permission model for write; self-host = own ACL |
| Content opacity | transparent (public JSON + RDF); not encrypted |
| Provenance | **statement-level** — references + rank per assertion (new far end) |
## 5. INTENT mapping
### Reinforcements
- **Stable identity ≠ placement** (T16): Q/P IDs decoupled from labels/language are the
textbook case — adopt the principle that a page's *identity* is an opaque stable handle,
display names are annotations.
- **Derived index over canonical store** (UC-63): WDQS is exactly a SPARQL index rebuilt
from canonical JSON entities via an update stream — validates the projection pattern.
- **Union without erasure / chorus**: **rank** lets multiple (even contradictory) statements
coexist with a curation signal rather than forcing one truth — the *structured* analogue
of fedwiki's chorus (UC-72) and our "view multiple versions" (UC-27).
- **Mechanism over policy**: references + rank are *mechanism* for representing disagreement
and sourcing; which statement "wins" is left to the consumer/query.
### Divergences (boundaries / design notes)
- **Content is not Markdown.** A Wikibase "page" is a set of statements; there is no prose
body. This is the **structure far-end**: shard-wiki must either (a) treat such a shard as
a **structured/typed shard** projected to a *lossy* Markdown/table rendering (UC-55/UC-73),
or (b) model a page whose payload is typed statements (T12). Forcing it into Markdown-first
erases the graph — a design-bug if done silently; render-with-provenance instead.
- **Provenance granularity is finer than ours.** Our provenance is per-page/per-shard;
Wikibase is **per-statement** (references) and even per-value (rank). The page model and
coordination journal should *allow* sub-page provenance (UC-75) even if MVP records it per
page.
- **Query is graph, not text/datalog.** SPARQL over RDF (with federated `SERVICE`) is a
richer query far-end than Roam/Logseq datalog or Notion filters (UC-52) — and its
`SERVICE` federation is a *query-time* cross-shard join, distinct from fedwiki structural
federation. Note both as native-query tiers.
### What to keep
1. **Opaque stable identity, labels-as-annotations** as the identity model (T16).
2. **Statement/assertion-level provenance** (references) and a **coexistence-with-rank**
model as the structured form of union-without-erasure (UC-75).
3. **Derived SPARQL/graph index over a canonical entity store** as a projection pattern
(UC-63/UC-74), incl. **federated query** as a first-class federation mode.
4. A **typed-graph page payload** option in the page model (T12), with **lossy
render-to-Markdown** as the projection (never silent flattening).
## 6. UC seeds
| # | Seed | Disposition |
|---|------|-------------|
| UC-73 | Attach a **Wikibase** as a **typed entity-statement (RDF) shard** (items/properties/statements w/ qualifiers); project to a rendered page view, lossy to Markdown, preserving the graph | **new** |
| UC-74 | **Graph-query the union** via **SPARQL** and **federate queries across endpoints** (`SERVICE`) — graph query as a native-query tier + query-time cross-shard join | **new** |
| UC-75 | Preserve **statement-level provenance** — references + rank attached to each assertion (sub-page provenance granularity) | **new** |
| — | typed records → typed *graph* entities | enrich **UC-34** |
| — | inter-record relations → typed graph edges with qualifiers | enrich **UC-58** |
| — | native query → SPARQL/RDF + federated SERVICE | enrich **UC-52** |
| — | provenance → statement/assertion granularity | enrich **UC-24** |
## 7. Architecture notes for SHARD-WP-0002
- **T12 (structured/typed page model):** add a **typed-graph payload** tier above
typed-records — a page whose content is **entities + statements (claim + qualifiers +
references + rank)**, with `somevalue`/`novalue` known-unknowns. Render-to-Markdown is a
**lossy projection**, not the canonical form.
- **T16 (identity / addressing):** adopt **opaque stable identity with labels-as-annotation**
(Q/P model); record **statement GUIDs** as an example of *sub-page addressable units*.
- **Native-query tiering:** SPARQL/RDF + federated `SERVICE` is the **graph far-end** of the
query spectrum (above datalog/filters); `SERVICE` is also a **query-time federation**
mode to sit beside fedwiki's structural federation.
- **Provenance model:** allow **per-statement references + rank** (sub-page provenance,
coexistence-with-curation) in the union, even if MVP collapses to per-page.
- **Derived index:** WDQS = canonical JSON entities → update stream → Blazegraph SPARQL
index; the reference implementation of UC-63 at scale (per-shard or core-built index, Q16).
## 8. Open questions
1. Does shard-wiki model a **typed-graph page** natively (T12), or always treat Wikibase as
a structured shard **projected to a Markdown/table rendering** (UC-55), or both
(canonical graph + lossy view)?
2. Is **SPARQL/graph query** exposed as a union-level capability (translate to a common
query layer) or only as a **pass-through** to graph-capable shards? How does federated
`SERVICE` relate to shard-wiki's own cross-shard query?
3. At what granularity does the coordination journal record **provenance** — per page
(MVP), per statement (Wikibase-native), or configurable?
4. Is **rank** (coexisting contradictory values w/ curation) representable in the union as a
first-class "chorus of statements," unifying with fedwiki's page-level chorus (UC-72/27)?
## 9. Sources
- Wikibase/DataModel and **DataModel/Primer** — mediawiki.org
- Help:Qualifiers; Wikidata SPARQL query service + Query Help; SPARQL tutorial — wikidata.org
- Wikidata Query Service / User Manual — mediawiki.org; Wikitech (Blazegraph, updater)
- "The wikibase model" — Vanderbilt Libraries Digital Lab (heardlibrary.github.io)
- RaiseWikibase — Wikibase Data Model functions (ub-mannheim.github.io)
- WShEx / EntitySchemas (ShEx) — arxiv.org/abs/2208.02697, ceur-ws.org Vol-3262
## 10. Traceability
New UCs **UC-73UC-75** carry the marker **⬡** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-34, UC-58, UC-52, UC-24. Architecture cross-refs:
SHARD-WP-0002 T12, T16, native-query tiering, provenance model, UC-63 derived index.

View File

@@ -0,0 +1,49 @@
# 260614 — Wiki.js deep dive (the storage-module engine that already speaks Git + Markdown)
Date: 2026-06-14
## What this is
A focused study of **Wiki.js** — the modern Node.js server wiki — read through
shard-wiki's lens. Wiki.js is the **closest existing engine to shard-wiki's own design**:
it is DB-canonical (Postgres/MySQL/SQLite) **but abstracts persistence behind pluggable
"storage modules"** — Git, local filesystem, S3, Azure Blob — each acting as **backup or
source of truth**, with the **Git module bidirectionally syncing clean `.md` files (+ YAML
frontmatter)** to a repo. That storage-module interface is, in effect, a shipped version
of shard-wiki's adapter contract — the second concrete prior art after **Foswiki::Store**,
and the closer one (medium = Markdown in Git).
Distinctive material:
- **Architecture** — Node.js/Vue; **GraphQL API** over all resources; DB-canonical;
Markdown (primary) + HTML + AsciiDoc; path-based hierarchy; modular
(storage/auth/search/render/editor)
- **Storage modules** — provider connector (Git/FS/S3/Azure) with properties + create/
update/delete/sync methods, **backup-or-source-of-truth**, `injectMetadata()` YAML
frontmatter; **Git module commits every change → version-controlled clean Markdown**,
bidirectional
- **GraphQL** — typed, **introspectable** (capability/schema discovery), **selective-field**
projection
- **Auth modules** (delegated authn) + **path-based rule ACL**
## Contents
| Path | Role |
|------|------|
| `findings.md` | Architecture, storage modules (≈ adapter contract), Git mirror as attach+write surface, GraphQL, access control, capability profile, INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Two new use cases promoted to `spec/UseCaseCatalog.md` (UC-68
attach an engine-maintained bidirectional Git mirror of clean Markdown, incl. write-by-
commit; UC-69 attach via a typed, introspectable API — GraphQL — for schema discovery +
selective projection); UC-06/36/38/40/42/57 enriched. Logged for `SHARD-WP-0002`
(T11/T14): the **storage-module abstraction as a second adapter-contract prior art** (with
Foswiki::Store), the **engine-maintained Git mirror** as preferred attach+write surface,
and **GraphQL introspection** for capability discovery + selective projection.
**Key takeaway recorded:** Wiki.js is the most shard-wiki-shaped engine yet — its
storage-module interface is near-isomorphic to the adapter contract, and its
Markdown-in-Git mirror is the ideal file-store attach (clean MD, git history,
write-by-commit). **Boundary:** one server-engine candidate shard, DB-canonical with a
maintained Git mirror; don't double-sync; honor its access rules; not the federation layer.
</content>

View File

@@ -0,0 +1,249 @@
# Findings — Wiki.js: the storage-module engine that already speaks Git + Markdown
Date: 2026-06-14
Source kind: **modern shipped product** — a server wiki engine; a *candidate shard* and
the closest existing engine to shard-wiki's own design (DB-canonical, but with a
**pluggable storage-module abstraction** that bidirectionally syncs clean Markdown to
**Git**/FS/S3/Azure)
Lens: shard-wiki — storage-module-as-adapter-contract prior art, the engine-maintained
Git mirror as an attach + write surface, GraphQL as a typed/introspectable API, and
path-based access rules
> Why Wiki.js is special in this set. Every prior engine/tool stored content in *one*
> substrate and exposed *one or two* surfaces. Wiki.js is DB-canonical (Postgres/MySQL/
> SQLite) **but abstracts persistence behind pluggable "storage modules"** — Git, local
> filesystem, S3, Azure Blob — each of which can act as **backup *or* source of truth**,
> and the **Git module bidirectionally syncs clean `.md` files (with YAML frontmatter)**
> to a repo. In other words, Wiki.js already does, internally, much of what shard-wiki's
> adapter contract is meant to do: a **versioned, capability-bearing, multi-provider
> persistence interface over Markdown**. It is the second concrete prior art for the
> contract after **Foswiki::Store** — and arguably the closer one, because its content
> is **Markdown in Git**, exactly shard-wiki's native shape.
Contrast set: the engine dives (XWiki DB/components, TWiki/Foswiki file+RCS) and the
modern tools. Wiki.js = **modern server engine, DB-canonical, Git-storage-of-Markdown,
GraphQL API, pluggable modules**.
---
## 1. Core architecture
- **Stack:** Node.js backend, Vue front end; **everything is accessible via a GraphQL
API** (`/graphql`, with a Playground). The DB is **Postgres** (primary; MySQL/SQLite
supported), read-replica-capable.
- **Content:** **Markdown (primary)**, plus **HTML** and **AsciiDoc**, via pluggable
**editors** (Markdown, WYSIWYG/visual, raw HTML). Pages live on **paths** (a path-based
hierarchy) with tags.
- **Modular architecture:** typed pluggable **modules****storage**, **auth**,
**search**, **rendering**, **editor**. The contract-relevant ones are *storage* (§2)
and *auth* (§4).
- **3.0:** a large rewrite is in development; storage remains a headline subsystem.
---
## 2. Storage modules — the headline (≈ shard-wiki's adapter contract)
A **storage module** "connects Wiki.js with a local or remote storage provider, to act as
**backup or source of truth** for content. It consists of **properties** (user-set) and
**methods called on events** — create, update, delete content, and sync." Providers
include **Git, local filesystem, S3, Azure Blob**.
The **Git module** is the most-used: **every page change commits to a Git repository**,
turning the wiki into **version-controlled `.md` files**. Content is written as **clean
Markdown**, and `injectMetadata()` prepends **YAML frontmatter** (for Markdown) or HTML
comments (for HTML) carrying page metadata. The sync is **bidirectional** with the remote
repo, and access to the synced files honors Wiki.js **access rules** (§4).
Why this matters to shard-wiki — three distinct payoffs:
1. **Prior art for the adapter contract.** Wiki.js's storage-module interface is a
shipped instance of what `SHARD-WP-0002` T11 specifies: a *versioned, capability-
bearing, multi-provider* persistence abstraction, with a backup-vs-source-of-truth
choice and lifecycle methods (create/update/delete/sync). It joins **Foswiki::Store**
as concrete prior art — and is closer, because its medium is **Markdown in Git**.
2. **The ideal file-store attach.** A Wiki.js instance with the Git module **is a git
repo of clean Markdown + YAML frontmatter**, maintained by the engine. shard-wiki can
attach that repo **directly** (UC-40) — better than reading the DB — and get **git
history for free** (UC-36, *adopt* not supplement, via the mirror).
3. **A write path via Git, no API needed.** Because the Git sync is **bidirectional**,
shard-wiki can **write-through by committing Markdown** to the repo; Wiki.js **ingests
git→DB**. This is overlay/patch application as a *git commit*, not an API call (UC-68).
Caution: Wiki.js owns the DB↔git sync — **don't double-sync**, and coordinate which
side is **source of truth** (configurable).
---
## 3. GraphQL — a typed, introspectable external API
Unlike the REST/local-REST tools (Joplin, Trilium, Notion), Wiki.js exposes a **GraphQL
API** over all resources. Two shard-wiki-relevant properties:
- **Introspection = capability/schema discovery.** A GraphQL schema is self-describing;
an adapter can introspect the content/page schema rather than hard-coding it — feeding
T11's *capability discovery*.
- **Selective-field projection.** GraphQL fetches **exactly the fields a projection
needs** (page body vs. metadata vs. tags), reducing over-fetch — relevant to projection
efficiency and the operational envelope. (UC-69; an external-API sub-mode beside
Notion's REST, UC-57.)
---
## 4. Auth modules + path-based access rules
- **Auth modules** (local, LDAP, OAuth, SAML, …) — **authentication is delegated to
pluggable providers**, exactly shard-wiki's **authn-delegated** stance
([[shard-wiki-auth-in-core-decision]]).
- **Access rules** are **path-based / rule-based ACL**: allow/deny access to all or
specific sections of the wiki per group; the Git-synced files honor the same rules. For
shard-wiki this is a richer ACL than TWiki's per-topic flags (UC-06) — **rule-based by
path pattern** — and informs how a projection should **honor/surface a shard's
restricted regions** (union without erasure: show that a region is restricted, don't
silently drop or expose it).
---
## 5. Wiki.js as a shard — capability profile
| Capability | Wiki.js | Notes for the adapter contract |
|------------|---------|--------------------------------|
| Read | yes (Git mirror / GraphQL / DB) | **Git mirror = clean Markdown** is the best surface |
| Write | yes (GraphQL **or git commit**) | bidirectional Git ingest = write-by-commit (UC-68) |
| Write granularity | per-page | — |
| Identity / addressing | path (+ page id) | path-based; YAML frontmatter carries metadata/id |
| Structure | YAML frontmatter (in-file via Git module) | git-diffable in-text structure (good) |
| History | **git-native via Git module** (else DB versions) | adopt the mirror's git history (UC-36) |
| Native query | GraphQL; pluggable search (DB/Elastic/Algolia) | delegate search; GraphQL selective fetch (UC-69) |
| Translation | Markdown primary; HTML/AsciiDoc too | mostly native; multi-format = light translation (UC-42) |
| Attach modes | **file-store (engine-maintained Git mirror)** / external-API (GraphQL) | clean MD in git is the ideal file-store (UC-40, UC-68) |
| Access control | **path-based rule ACL**; delegated auth modules | authn-delegated; honor restricted regions (UC-06) |
| Storage abstraction | **pluggable storage modules (Git/FS/S3/Azure)** | **adapter-contract prior art** (T11) |
| Content types | pages + media/assets | assets via storage modules |
Verdict: **the most shard-wiki-shaped engine yet.** Best attached via its **Git storage
mirror** (clean Markdown + frontmatter + git history, with write-by-commit), or via
**GraphQL** for live/selective access. Its standout gift is **architectural**: a shipped
storage-module abstraction that is near-isomorphic to shard-wiki's adapter contract.
---
## 6. Mapping to shard-wiki INTENT (compare, do not equate)
### 6.1 Reinforcements
- **Storage modules = the adapter contract, shipped.** A versioned, capability-bearing,
multi-provider persistence interface with backup-vs-source-of-truth — exactly T11's
shape, in production (with Foswiki::Store as the other prior art).
- **Markdown-in-Git is the native shape.** Wiki.js validates that a serious engine can
keep content as **clean `.md` + frontmatter in git** — shard-wiki's preferred medium —
while still offering a rich app over a DB.
- **authn-delegated** (auth modules) and **git coordination** (Git storage) are both core
shard-wiki stances, here shipped.
### 6.2 Deliberate divergences (design bugs if conflated)
1. **DB is Wiki.js's canonical store; the Git repo is a maintained mirror.** Treat the
mirror as the attach surface, but respect that **Wiki.js runs the DB↔git sync**
don't double-sync; coordinate source-of-truth (UC-68).
2. **Bidirectional write needs care.** Committing Markdown is a real write path, but
races with the engine's own sync; apply overlays as commits the engine can ingest
cleanly, never out-of-band DB edits (overlay before mutation, no silent mutation).
3. **One Wiki.js instance = one shard**, not the federation layer; its access rules are
the shard's, to be honored/surfaced, not re-implemented.
4. **HTML/AsciiDoc pages** need light translation; Markdown pages are native.
### 6.3 What Wiki.js teaches that shard-wiki should keep
- **Model the contract on a real storage-module interface** (Wiki.js + Foswiki::Store):
properties + lifecycle methods (create/update/delete/sync) + a **backup-vs-source-of-
truth** flag + a provider list. (T11.)
- **Prefer the engine-maintained Git mirror** as the attach surface for DB-canonical
engines that offer one — clean Markdown, git history, and write-by-commit (UC-68).
- **Use GraphQL introspection for capability discovery and selective projection** where a
shard offers it (UC-69).
---
## 7. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-67**. New UCs **UC-68, UC-69** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Attach an engine-maintained bidirectional Git mirror of clean Markdown** (Wiki.js Git storage module): attach the repo as a file-store shard (clean MD + YAML frontmatter + git history) and optionally **write-through by committing Markdown the engine ingests**; coordinate source-of-truth, don't double-sync | **UC-68 (new)** |
| **Attach via a typed, introspectable API (GraphQL)**: discover the shard's schema by introspection (capability discovery) and fetch only the fields a projection needs (selective projection) | **UC-69 (new)** |
| Pluggable **storage modules** (Git/FS/S3/Azure) = adapter-contract prior art; modular auth/search/render/editor | **enriches UC-38** |
| Git storage = **clean Markdown in git**, the ideal engine-maintained file-store attach | **enriches UC-40** |
| Git history natively via the storage module (**adopt** via mirror) | **enriches UC-36** |
| GraphQL external-API variant (typed/introspectable/selective) | **enriches UC-57** |
| Multi-format (Markdown primary; HTML/AsciiDoc) | **enriches UC-42** |
| Path-based rule ACL; delegated auth modules | **enriches UC-06** (links authz decision) |
| Engine-maintained clean git shard | links UC-02 |
---
## 8. Architecture notes for SHARD-WP-0002 (no UC)
- **Second adapter-contract prior art (T11).** Wiki.js's **storage-module interface**
(properties + create/update/delete/sync methods + backup-vs-source-of-truth + multi-
provider Git/FS/S3/Azure) is, with **Foswiki::Store**, the model to base the contract
on — and the closer one (medium = Markdown in Git). Cite both in T11.
- **Engine-maintained Git mirror (T14).** Add to the attachment taxonomy: for a
DB-canonical engine that offers a bidirectional Git-of-Markdown mirror, the **mirror is
the preferred attach + write surface** (clean MD, git history, write-by-commit). A
cleaner sibling of Joplin's proprietary interchange mirror (UC-60) and the file-store
native-store kind (UC-40).
- **GraphQL → capability discovery + selective projection (T11/T14).** Typed/
introspectable APIs give schema discovery (capability negotiation) and reduce over-fetch
— note alongside Notion's REST external-API mode.
- **authn-delegated + path-based ACL** reinforce the settled authz decision and inform how
projections honor/surface restricted regions (union without erasure).
---
## 9. Open questions (for spec / workplans)
1. For an engine with a bidirectional Git mirror (UC-68), is the **mirror or the DB the
source of truth** from shard-wiki's view, and how do we avoid races with the engine's
own sync (write-by-commit cadence, locking)?
2. When both a **Git mirror and GraphQL** are available, which does a binding prefer —
mirror for content/history, GraphQL for live/selective metadata? Can one binding use
both?
3. How does shard-wiki **honor/surface path-based access rules** (UC-06) in a projection
without re-implementing the shard's ACL engine?
4. Should the adapter contract's **storage-module shape** be standardized directly on
Wiki.js + Foswiki::Store, or generalized further?
---
## 10. Sources
| Source | Used for |
|--------|----------|
| Wiki.js — GraphQL API docs (https://docs.requarks.io/dev/api) | Everything via GraphQL; `/graphql` Playground; access/modify all resources |
| Wiki.js — Git storage (https://docs.requarks.io/storage/git) | Per-change commit to git; clean `.md`; bidirectional sync with remote repo |
| Wiki.js — Storage overview + dev/storage.md (https://docs.requarks.io/storage ; https://github.com/requarks/wiki-docs/blob/master/dev/storage.md) | Storage module = provider connector; backup-or-source-of-truth; properties + create/update/delete/sync methods; `injectMetadata()` YAML frontmatter; providers Git/FS/S3/Azure |
| Wiki.js 3.0 storage preview (https://beta.js.wiki/blog/2021-wiki-js-3-feature-preview-storage) | Storage subsystem direction (3.0) |
| Wikipedia — Wiki.js (https://en.wikipedia.org/wiki/Wiki.js) | Node.js/Vue, DB options, formats, modular architecture |
| BrightCoding / Grokipedia overviews | Auth providers, access rules, editors, read replicas |
Cross-references: `research/260613-foswiki-deep-dive/findings.md` (Foswiki::Store, the
other storage-interface prior art), `research/260614-joplin-deep-dive/findings.md`
(interchange/sync-mirror contrast), `research/260614-notion-deep-dive/findings.md`
(external-API/REST contrast), `research/260614-shard-spectrum-synthesis/findings.md`
(attachment-mode spectrum), `spec/UseCaseCatalog.md` (UC-02, UC-06, UC-36, UC-38, UC-40,
UC-42, UC-57), `workplans/SHARD-WP-0002-federation-architecture.md` (T11, T14).
---
## 11. Traceability
- New UCs: **UC-68, UC-69**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-06, UC-36, UC-38, UC-40, UC-42, UC-57** (links UC-02, authz decision).
- Architecture (no UC): **storage-module abstraction = 2nd adapter-contract prior art**
(with Foswiki::Store); engine-maintained Git mirror as attach+write surface; GraphQL
introspection → capability discovery + selective projection; path-based ACL + authn-
delegated → `SHARD-WP-0002` (T11, T14).
- Boundary recorded: Wiki.js is **one server-engine candidate shard**, DB-canonical with a
**bidirectional Git-of-Markdown mirror** that is the preferred attach surface; not a
substrate and not the federation layer (don't double-sync; honor its access rules).
</content>

View File

@@ -0,0 +1,47 @@
# 260614 — Project Xanadu deep dive (the docuverse, the EDL, reference-not-copy)
Date: 2026-06-14
## What this is
A focused study of **Project Xanadu** (Ted Nelson) read through shard-wiki's lens.
Unlike the engine dives (`xwiki`, `twiki`, `foswiki`), Xanadu is **not a candidate
shard backend** — it never shipped at scale and there is nothing to attach. It is
studied as the **deepest conceptual ancestor** of shard-wiki's own model: several
shard-wiki primitives turn out to be Xanadu mechanisms under different names.
The distinctive material:
- the **EDL / xanadoc** — a document that contains *no content*, only an ordered list
of **spans** (content references) plus **xanalinks** (separate link tables); the
client assembles the page by reference → this is shard-wiki **projection** + **union**
- the storage substrate — **tumblers** (stable fine-grained addresses), **istream**
(invariant content pool), **enfilades / spanfilade** (virtual↔content mapping,
version comparison by span-set intersection)
- **content-identity, bidirectional transclusion** — content "knowably in more than
one place" that remembers all its appearances → stronger than UC-32
- **transcopyright / micropayment** — a baked-in rights policy shard-wiki must keep
*configurable*, not inherit
Goes deliberately **underneath** the surface pattern table in
`research/260608-federation-concepts/findings.md` §3.3, which it extends rather than
repeats.
## Contents
| Path | Role |
|------|------|
| `findings.md` | EDL/xanadoc, addressing substrate, transclusion, versioning, rights, INTENT mapping (reinforcements + design-bug divergences), UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
(UC-44 compose-by-reference, UC-45 reverse transclusion, UC-46 content-identity
equivalence); UC-24/27/29/32 enriched. Span-addressing as an adapter capability,
content identity, composition manifests, and reuse-terms metadata logged as
architecture for `workplans/SHARD-WP-0002-federation-architecture.md`.
Key boundary recorded: shard-wiki inherits Xanadu's **reference-not-copy** mechanisms
but **rejects** the single-global-docuverse premise, the single-canonical-instance
model, and the baked-in economic policy — those would violate shard sovereignty,
parallel-version support (UC-27), and mechanism-over-policy.
</content>

View File

@@ -0,0 +1,322 @@
# Findings — Project Xanadu: the docuverse, the EDL, and reference-not-copy
Date: 2026-06-14
Source kind: **conceptual / architectural prior art** (not a deployable engine, not a
candidate shard backend)
Lens: shard-wiki orchestration — projection, overlay, transclusion, provenance,
addressing, coordination journal
> Reading guide. Every previous deep dive (`xwiki`, `twiki`, `foswiki`) studied a
> *shippable engine* as a candidate **shard**. Xanadu is different: it is the
> **deepest conceptual ancestor** of shard-wiki's own model and is *not* a backend
> we would ever attach. It is studied here for its **mechanisms** —
> reference-not-copy documents, separated link tables, content-identity transclusion,
> stable fine-grained addressing — several of which shard-wiki already reinvents
> under different names (projection, overlay, union BackLinks, coordination journal).
> The job of this file is to (a) name what shard-wiki inherits, (b) flag what would
> be a **design bug** to inherit, and (c) extract use cases that are genuinely new
> versus the federation track already in the catalog.
Pairs with — and deliberately extends, does not repeat —
`research/260608-federation-concepts/findings.md` §3.3, which treats Xanadu only as a
six-row pattern table and (correctly) labels it "speculative design / pattern
language, not deployable federation." This dive goes underneath those patterns into
the actual data architecture (EDL, istream, enfilades, tumblers) because *that* is
where the resonance with shard-wiki's storage-neutral page model lives.
---
## 1. Origin and status
Project Xanadu (Ted Nelson, 1960 ; the term *hypertext* and *transclusion* are his)
is the original electronic-literature system: a global **docuverse** of permanent,
addressable documents joined by visible, two-way links. It famously never shipped at
scale; partial artifacts exist — **Udanax Green / Udanax Gold** (source released 1999
to establish prior art against patents), **XanaduSpace**, and **OpenXanadu** (2014, a
small working browser demo built on the EDL format). Treat all of these as
*reference designs*, not running software to integrate.
Why it matters to shard-wiki: Nelson's critique of the Web — **"one-way,
ever-breaking links and no management of version or contents"** — is precisely the
gap shard-wiki addresses for wikis. shard-wiki is, in effect, a *pragmatic, federated,
Git-backed, heterogeneous-backend* descendant of the Xanadu idea, scoped to
Markdown wiki pages and explicitly **refusing** Xanadu's one-global-store premise.
---
## 2. The headline insight — a document is a manifest of spans, not content
The single most shard-wiki-relevant Xanadu artifact is the **EDL (Edit Decision
List)**, also called a **xanadoc**:
- A xanadoc is a *connected document that contains no original content*. It is a file
listing (a) **spans** — "portions of content to bring in" addressed in remote
sources — and (b) **xanalinks** — tables saying what to connect to what.
- The client reads the EDL, **fetches each span from its source**, and **assembles**
the document locally per the xanalinks. Content is referenced, never embedded.
- Structure is strictly separated: **spans first** (content references), **xanalinks
after** (relationships). Content and link structure live apart.
This is *the same move* shard-wiki makes:
| Xanadu (xanadoc / EDL) | shard-wiki concept |
|------------------------|--------------------|
| Document = ordered list of span references, no embedded content | **Projection** — lazy, cache-like view assembled from shard storage, not a copy |
| Spans pulled from multiple remote sources at view time | **Union of pages** across heterogeneous shards |
| Xanalinks stored separately from content | **Overlay** — non-destructive edits/annotations as separate objects, "overlay before mutation" |
| "All media should be permanized and addressable" | **Coordination journal** + Git content-addressable storage |
shard-wiki's INTENT lines — *"prefer lazy projection over eager copying"* and *"union
without erasure"* — are the xanadoc principle restated for wikis.
---
## 3. Addressing and the storage substrate
Xanadu's machinery underneath the EDL:
- **Tumblers** (Miller / Gregory): a transfinite-number addressing scheme where one
address simultaneously encodes **machine, author, document version, byte span, and
links**. A single tumbler is a stable, fine-grained, hierarchical pointer into the
whole docuverse.
- **istream (invariant stream):** a growing pool of *shareable content pieces*.
Documents do not store text; they reference istream content by virtual address. A
family of **enfilades** (tree structures using *dsps* = relative displacements and
*wids* = ranges) maps **virtual addresses ↔ istream addresses** bidirectionally.
- **Spanfilade:** indexes which istream spans each document uses. Other filades:
**granfilade** (storage across disks/network), **POOMfilade** (permutation-of-order
matrix mapping document position → istream location).
- Implicit substrate: content pieces are **invariant** (append-only, never rewritten);
versions are new arrangements of shared, permanent pieces. Nothing is deleted, so
references never break.
shard-wiki mapping:
- **istream + invariance ≈ Git's content-addressable, immutable blob store**, and the
**coordination journal** as the append-only record. shard-wiki gets "references
never break" *for Git-native shards for free*; for non-Git shards it must
approximate via the journal/projection cache.
- **Tumbler ≈ the still-open question of a portable, stable, fine-grained span
address** that survives projection, overlay, and versioning across heterogeneous
backends. shard-wiki has no such addressing scheme yet; this is the hardest part
Xanadu solved on paper and the part that never shipped. (See §10, §11.)
---
## 4. Links and transclusion — separate, two-way, content-identity
- **Links are first-class objects stored apart from content** (the xanalinks tables),
and are **visible and followable from all endpoints** (rule 7) — i.e. inherently
**bidirectional**. Backlinks are not derived after the fact; they are the same
object seen from the other end.
- **Transclusion** = Nelson's "the same content *knowably* in more than one place."
The content piece **remembers its identity and can trace back to all its
appearances**. This is the crucial delta from every weak modern form (server-side
includes, HTML embeds, Roam block-refs): those *copy or cache*; Xanadu keeps one
permanent instance that is **aware of where it is reused**.
shard-wiki mapping:
- **Separate link tables ≈ overlays and union BackLinks as first-class.** An overlay
is shard-wiki's xanalink: a non-destructive object spanning content it does not own.
- **Content-identity, bidirectional transclusion** is *stronger* than UC-32 as
currently written (which is path/span fetch with freshness). Xanadu says the
*content itself* knows its appearances → enables **reverse transclusion** ("where is
this paragraph used across all shards?") and **content-identity equivalence** ("these
two pages in different shards are versions of the same content"). These are new
capabilities, surfaced as UC-45 and UC-46 below.
---
## 5. Versioning and comparison — by span-set intersection
Because a document is a set of spans over an invariant pool, **comparing two documents
or two versions = intersecting their span-sets** (the spanfilade operation). Shared
spans are literally shared subtrees; differences fall out of the set arithmetic. The
same operation surfaces links between documents.
shard-wiki mapping: this is a **content-identity diff/merge** model. shard-wiki's
diff/merge capability (an adapter capability in the contract) is today implicitly
path/title- and text-based. Xanadu shows a path-independent alternative: detect that
page A in shard X and page B in shard Y are *the same or derived content* by span
overlap, **without relying on matching titles or paths**. This directly serves UC-27
(view parallel versions of equivalent pages) by giving an *equivalence-detection
mechanism* rather than assuming naming conventions align across sovereign shards.
---
## 6. Rights and economics — transcopyright, micropayment, implicit permission
Three of the 17 rules are an economic/rights layer:
- Rule 8: **publication grants implicit permission to link/transclude.**
- Rule 9: **granular royalty / micropayment** on any accessed portion.
- **Transcopyright:** pre-granted permission for virtual republication *by reference*,
with the attribution chain to origin preserved automatically because the content is
transcluded, not copied.
shard-wiki mapping — handle with the **mechanism-over-policy** rule:
- The *mechanism* shard-wiki should provide: a reference (overlay/transclusion) that
**carries provenance and reuse terms with it**, so attribution and permission travel
with the span. This strengthens UC-29 (remix with portable attribution).
- The *policy* (whether reuse requires permission, whether anything is metered) must
**not be hard-coded**. Xanadu baked one global economic policy into the substrate;
shard-wiki keeps editorial/economic policy configurable. Baking in a payments or
permission model would be a design-bug per INTENT.
- This intersects the **settled authz-in-core / authn-delegated decision**
([[shard-wiki-auth-in-core-decision]]): "publication grants implicit linking
permission" is exactly a *policy* that an information space might or might not adopt
on its L0→L4 ladder. shard-wiki must be able to *represent* the permission on a
transclusion; it must not *assume* Xanadu's answer.
---
## 7. Xanadu through shard-wiki's lens — it is architecture, not a shard
Unlike the engine dives, there is **no capability profile to fill in** — Xanadu is not
a backend you attach. The useful framing is the inverse: *which shard-wiki primitives
are Xanadu mechanisms in disguise?*
| shard-wiki primitive | Xanadu mechanism | Inheritance verdict |
|----------------------|------------------|---------------------|
| Projection (lazy view, no copy) | xanadoc / EDL span assembly | **Inherit** — same idea, scope to wiki pages |
| Overlay (non-destructive edit) | xanalinks as separate objects | **Inherit** — overlays are xanalinks |
| Union BackLinks | two-way links visible from all endpoints | **Inherit** — generalize to sub-page spans |
| Transclusion (UC-32) | content-identity, content-aware reuse | **Inherit, strengthen** — add reverse + equivalence |
| Coordination journal | invariant istream / permanent storage | **Inherit** (Git gives this for Git-native shards) |
| Stable span address | tumblers | **Aspire** — hard, partly unshippable; degrade gracefully |
| Editorial/economic policy | transcopyright + micropayment | **Reject as substrate** — keep configurable |
| One global docuverse | the docuverse premise | **Reject** — see §8.2 |
---
## 8. Mapping to shard-wiki INTENT (compare, do not equate)
### 8.1 Reinforcements
- **Reference-not-copy** is the core of both. Xanadu validates shard-wiki's
"lazy projection over eager copying" and "union without erasure" as a coherent,
decades-deep design lineage, not an ad-hoc preference.
- **Provenance is structural, not decorative.** In Xanadu, knowing where content came
from and where it is reused is *built into the storage model*. shard-wiki's "never
hide authorship, conflicts, freshness, backend limits" is the same commitment.
- **Links/edits as separate objects** validates overlays-before-mutation and
first-class BackLinks.
### 8.2 Deliberate divergences (design bugs if conflated)
1. **No single docuverse / no universal address space.** Xanadu requires one global
permanent store, universal addressing, and its own published clientserver
protocol (rules 12, 17). shard-wiki's whole reason to exist is the opposite:
**federate heterogeneous, sovereign backends** (Git repos, Gitea, Obsidian,
WebDAV, Coulomb), each keeping its own storage, history, identity, and limits.
Adopting Xanadu's universality would violate **shard sovereignty** and **graceful
degradation**. shard-wiki's addressing must tolerate backends that *cannot* offer
stable span addresses, treating fine-grained addressing as an **adapter
capability**, not a precondition.
2. **One canonical content instance vs. parallel divergent versions.** Xanadu
transclusion centers a *single source of truth*. shard-wiki must support
**equivalent-but-divergent** pages across shards with conflicts *visible* (UC-27),
not collapsed into one instance. Content-identity is a *detection* tool here, not a
mandate to unify.
3. **Baked-in economic/permission policy.** See §6 — reject as substrate, keep as
configurable policy.
4. **Permanence as a hard requirement.** Xanadu forbids deletion globally. shard-wiki
cannot impose that on sovereign shards; permanence holds for the **coordination
journal** and Git-native shards, and is *approximated* (cache/projection/snapshot)
for backends that delete. This is graceful degradation, not a weaker promise.
### 8.3 What Xanadu teaches that shard-wiki should not lose
- A page view can legitimately be a **composition manifest** (a list of references),
not a file. Build the page model so a page *can* be authored as references into
other shards (UC-44), even if the common case is a plain Markdown file.
- **Content identity** (not path/title) is the durable basis for transclusion,
equivalence, and reverse lookup across sovereign shards (UC-45, UC-46). Lean on
Git blob hashes where available; define a content-fingerprint fallback elsewhere.
- Getting **stable fine-grained addressing** right is the hard, valuable, historically
*unshippable* part. Scope it as an adapter capability with explicit degradation
rather than promising tumbler-grade universality.
---
## 9. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-43**. New UCs **UC-44UC-46** added; several existing UCs
enriched (no new scenario, stronger mechanism/notes).
| Seed | Catalog action |
|------|----------------|
| **Compose-by-reference page** — author a page whose body is an ordered list of spans pulled from multiple shards, stored as a manifest (xanadoc/EDL), not a copy | **UC-44 (new)** |
| **Reverse transclusion** — find every page/shard where a given span (paragraph/section) appears or is transcluded | **UC-45 (new)** |
| **Content-identity equivalence** — detect that two pages in different shards are the same or derived content via span/content overlap, without matching titles or paths | **UC-46 (new)** |
| Content-identity, content-aware (bidirectional) transclusion | **enriches UC-32** |
| Equivalence detection mechanism for parallel versions | **enriches UC-27** |
| Reference carries provenance + reuse terms (transcopyright as representable policy) | **enriches UC-29**; links [[shard-wiki-auth-in-core-decision]] |
| Provenance: content remembers its appearances | **enriches UC-24** |
---
## 10. Adapter-contract / architecture notes for SHARD-WP-0002
Logged as architecture (no UC):
- **Fine-grained span addressing is an adapter capability**, not a core assumption.
The shard adapter contract should model whether a shard can mint a *stable address
for a sub-page span* that survives edits/versions (tumbler-grade), down through
whole-page-only, down to path-only. Transclusion/overlay capabilities depend on it.
- **Content identity** should be a contract-level concept: a shard advertises how it
fingerprints content (Git blob hash, normalized-text hash, none). UC-45/UC-46 and
cross-shard diff/merge consume it.
- **Composition manifests** (UC-44) imply the wiki page model must permit a page whose
canonical form is a reference list. This is an INTENT-level page-model decision —
flag for the page-model spec, not just the adapter contract.
- Reuse-terms metadata on a reference (UC-29 / §6) is **policy data the core carries
but does not interpret** — consistent with mechanism-over-policy and the L0→L4
authz ladder.
---
## 11. Open questions (for spec / workplans)
1. What is shard-wiki's **portable span address**? Git blob+range works for Git-native
shards; what is the fallback for Obsidian/WebDAV/Gitea-wiki, and how does it
survive a shard's storage swap (cf. UC-43 Foswiki RCS↔PlainFile)?
2. Is **compose-by-reference (UC-44)** core orchestrator, adapter-provided, or
reference-UI — and is it MVP or deferred with UC-32?
3. Does **content-identity equivalence (UC-46)** belong in core (cross-shard union
logic) or as an adapter-provided index? How expensive is span-set intersection at
wiki scale without enfilades?
4. How far do we take **reverse transclusion (UC-45)** — exact span identity only, or
fuzzy/derived-content tracking? The latter is research-grade.
5. Where does **reuse-terms/transcopyright metadata** sit on the L0→L4 ladder, and
does any tier ever *enforce* it, or is it always advisory provenance?
---
## 12. Sources
| Source | Used for |
|--------|----------|
| Wikipedia — Project Xanadu (https://en.wikipedia.org/wiki/Project_Xanadu) | 17 rules, tumblers, history, Web critique |
| xanadu.com — The Edit Decision List / Xanadoc File (https://xanadu.com/xuEDL.html) | EDL/xanadoc structure: spans + xanalinks, client assembly |
| Wikipedia — Enfilade (Xanadu) (https://en.wikipedia.org/wiki/Enfilade_(Xanadu)) | istream, enfilades (dsp/wid), spanfilade, granfilade, POOMfilade, version comparison by span-set intersection |
| Wikipedia — Transclusion (https://en.wikipedia.org/wiki/Transclusion) | "same content knowably in more than one place," content-aware reuse, transcopyright, micropayment vs. weak modern forms |
| Maggie Appleton — Xanadu Patterns (https://maggieappleton.com/xanadu-patterns) | pattern naming: visible links, parallel documents, transpointing windows, modular blocks, stable addresses, annotation |
Cross-references: `research/260608-federation-concepts/findings.md` §3.3 (prior
surface treatment), `spec/UseCaseCatalog.md` (UC-24, UC-27, UC-29, UC-32),
`workplans/SHARD-WP-0002-federation-architecture.md` (adapter contract).
---
## 13. Traceability
- New UCs: **UC-44, UC-45, UC-46**`spec/UseCaseCatalog.md` (section F, federation).
- Enriched UCs: **UC-24, UC-27, UC-29, UC-32**.
- Architecture (no UC): span-addressing capability, content-identity, composition
manifest, reuse-terms metadata → `SHARD-WP-0002`.
- Decision link: transcopyright-as-policy → [[shard-wiki-auth-in-core-decision]].
</content>
</invoke>

View File

@@ -0,0 +1,43 @@
# 260614 — ZigZag / zzstructure deep dive (an information space as orthogonal dimensions)
Date: 2026-06-14
## What this is
A focused study of **Ted Nelson's ZigZag** and its data model **zzstructure**, asked
through a specific shard-wiki question: a wiki page is linked in many independent ways
at once — a **namespace** tree, a **created-from / genealogy** lineage, **hyperlinks**,
plus shard-wiki's own **shard / overlay / version / transclusion / equivalence**
relations. Could the information space be modelled as a zzstructure, where each
relationship is a first-class **dimension** and none is privileged?
Distinctive material:
- the zzstructure model — **cells**, **dimensions** (edge colours), **ranks**,
**posward/negward**, and the one rule that defines it: *at most one posward and one
negward neighbour per dimension per cell* (a directed coloured multigraph)
- **views** — I-view (one dimension), **H-view** (pivot two dimensions into a cross),
raster-vs-view
- trees/lists/tables as *spent dimensions*; many-to-many graphs as the hard case
(handled by **clones**)
- **clone ↔ transclusion** convergence with the Xanadu dive
## Contents
| Path | Role |
|------|------|
| `findings.md` | zzstructure model, views, fit analysis, a proposed dimension map for a shard-wiki space, recommendation (lens not store), INTENT mapping, UC seeds, architecture notes, sources |
## Status
Initial deep dive complete. Three new use cases promoted to `spec/UseCaseCatalog.md`
(UC-47 navigate-by-dimension, UC-48 H-view pivot/cross-tab, UC-49 genealogy
dimension); UC-05/1722/26/29 enriched. A **dimensional projection layer** is logged
as architecture for `workplans/SHARD-WP-0002-federation-architecture.md`.
**Recommendation recorded:** adopt zzstructure as a **navigation / visualization /
indexing lens** (a derived dimensional projection over the union), **not** as the
storage substrate — Git and sovereign shards remain canonical (INTENT Stability Note).
The many-to-many hyperlink graph does not fit zzstructure's rank constraint and stays a
separate graph index. Pairs with `research/260614-xanadu-deep-dive/` (clone ↔
transclusion).
</content>

View File

@@ -0,0 +1,294 @@
# Findings — ZigZag / zzstructure: an information space as orthogonal dimensions
Date: 2026-06-14
Source kind: **conceptual / data-model prior art** (a data model + visualization
paradigm, not a deployable wiki engine, not a candidate shard backend)
Lens: shard-wiki — modelling a wiki information space where a page participates in
*many* relationship structures at once (namespace, genealogy, links, versions, shards)
> Motivating question (from the user). A wiki page is linked in several independent
> ways at once: a **primary namespace** association giving a tree-like hierarchy; a
> **created-from / genealogy** association (fork/derivation lineage); **hyperlinks**
> to other pages; and — in shard-wiki — *which shard* it lives in, its **overlays**,
> its **versions**, its **transclusions**, and its **equivalents** elsewhere. Could
> the information space be modelled as Ted Nelson's **ZigZag / zzstructure**, where
> each kind of relationship is a separate, first-class **dimension** and no single
> relationship is privileged as *the* structure?
>
> Short answer: **yes as a navigation / visualization / indexing lens; no as the
> storage substrate.** zzstructure is an unusually good *conceptual* match for
> shard-wiki's "union without erasure" thesis — it makes every relationship
> co-equal and separately navigable — but its one-neighbour-per-direction
> constraint makes it a poor *direct* encoding of the many-to-many hyperlink graph,
> and Git/heterogeneous shards must remain the canonical store. Details below.
Companion to the Xanadu dive (`research/260614-xanadu-deep-dive/`): ZigZag is
Nelson's *other* lifelong project, the structural/visual counterpart to Xanadu's
literary/addressing one. Where Xanadu gives shard-wiki **reference-not-copy**, ZigZag
gives it **many co-equal structures over the same cells**.
---
## 1. What ZigZag / zzstructure is
ZigZag (Ted Nelson, conceived 1981 at Datapoint; first Perl prototype by Andrew Pam,
1997; the **GZigZag / Gzz** Java project 20002003; US patent 6,262,736, expired
2019) is **"a data model that deconstructs the spreadsheet to allow irregular
relations, generalizing the idea to multiple dimensions."** The generic, vendor-free
name for the data model is **zzstructure**.
Core vocabulary:
- **Cell (zzcell)** — an atom of data (a value, a record, a page). Cells are the only
content; everything else is connection.
- **Dimension** — a *named kind of connection* (an edge colour). Dimensions are
**first-class** — in ZigZag, dimensions are themselves cells.
- **Link (zzlink)** — a connection between two cells *along one dimension*, with a
**posward** (positive) and **negward** (negative) direction.
- **Rank** — the maximal chain of cells along one dimension (a path, or a cycle). A
rank is ZigZag's equivalent of "a row" / "a list" — but it exists only along one
named dimension.
- **Cursor** — the current focus cell from which views are drawn.
- **Clone** — a copy of a cell that shares identity with its head cell (along a
special clone dimension), used to let one logical item appear in several places /
participate in many-to-one relations.
---
## 2. The one rule that defines everything
Formally (McGuffin & schraefel, HT2004; McGuffin's graph-theoretic intro), a
zzstructure is:
> a **directed multigraph whose edges are coloured (typed)**, subject to the
> restriction that **each node has at most one incoming edge of each colour and at
> most one outgoing edge of each colour.**
Equivalently: **along any single dimension, a cell has at most one posward and one
negward neighbour.** That single constraint is the whole design:
- Each dimension decomposes the cell set into disjoint **ranks** (paths and cycles) —
never a branching tree, never a fan-out, *within one dimension*.
- Many dimensions coexist over the **same** cells, each imposing its own independent
rank structure. A cell sits at the intersection of as many dimensions as it
participates in.
- There is **no inherent hierarchy.** A tree is not the substrate; it is *one
possible dimension* (encoded left-child/right-sibling — see §4).
This is the property that matters for shard-wiki: **the same set of pages can carry
many orthogonal structures simultaneously, with none privileged.**
---
## 3. Views — how multiple dimensions are seen
Because no screen can show all dimensions, ZigZag *rotates* two into view at a time:
- **I-view** — cells along **one** dimension from the cursor, drawn as a line.
- **H-view** — a 2-D cross: **two** dimensions through the cursor, one drawn
horizontally and one vertically; only cells actually connected to the cursor along
those two dimensions appear (empty spreadsheet cells simply do not exist).
- **I+/H+ (augmented) views** — fill remaining space with further neighbours.
- **Raster vs. view (terminology)** — a *raster* selects which cells to take from the
structure; a *view* places them on screen. "Pivoting" rotates a different dimension
into the horizontal or vertical slot.
The H-view "pivot" is the genuinely interesting UX idea for a *federated* wiki: put
**pages of a namespace** on one axis and **shards** (or **versions across shards**)
on the other, then rotate to re-cross-tabulate the same union by a different pair of
relationships.
---
## 4. Trees, lists, tables, graphs — what fits and what does not
zzstructure subsumes simpler structures by spending dimensions:
- **List** = one rank along one dimension.
- **Table** = two dimensions (rows / columns), shown as an H-view.
- **Tree** = encoded as a zzstructure "analogous to the left-child, right-sibling
pointer implementation" — i.e. a *parent→firstchild* dimension plus a
*sibling→sibling* dimension. So a hierarchy is **two dimensions**, not a primitive.
- **Many-to-one / many-to-many** = NOT directly expressible (the one-edge-per-colour
rule forbids fan-out). It is simulated with **clones**: the head cell links to many
clone cells along the clone dimension, and each clone participates in a different
rank.
**This is the critical caveat for wikis.** A namespace hierarchy (each page has *one*
parent) and a created-from genealogy (each page has *one* origin) fit zzstructure
*natively* and beautifully. But a wiki's **hyperlink graph is many-to-many** (a page
links to many pages and is linked from many) — it is *not* a rank and cannot be one
dimension. Representing links/backlinks in zzstructure requires cloning or treating
the link set as data, which is awkward. zzstructure is excellent for the
**functional/sequential/hierarchical** relationships and weak for the **arbitrary
graph** ones.
---
## 5. Mapping a shard-wiki information space onto dimensions
Taking the user's framing literally — model the union of pages as cells, and each
relationship as a dimension:
| Dimension (proposed) | Rank meaning | Fits zzstructure? | shard-wiki source |
|----------------------|--------------|-------------------|-------------------|
| `d.namespace-child` + `d.namespace-sibling` | namespace tree (left-child/right-sibling) | **Yes** (2 dims) | UC-22, yawex topics |
| `d.genealogy` | created-from / fork lineage (each page one origin) | **Yes** (functional) | UC-26 fork, UC-29 remix |
| `d.version` | revision order of a page | **Yes** (a rank = a history) | coordination journal |
| `d.shard` | pages grouped by owning shard | **Yes** | shard sovereignty |
| `d.overlay` | page → its overlays | **Yes** (or clones) | overlay model |
| `d.equivalence` | parallel versions of the same topic across shards | **Yes** (a rank of equivalents) | UC-27, UC-46 |
| `d.recent` | temporal order of changes | **Yes** | UC-17 RecentChanges |
| `d.alphabetical` | AllPages ordering | **Yes** | UC-19 |
| `d.links` / `d.backlinks` | hyperlink graph | **No** (many-to-many) — needs clones / stays a graph index | UC-05, UC-18 |
| `d.transclusion` | content reused in many places | **No** as rank — but maps to **clones** (= "same content in many places") | UC-32, UC-44/45 |
Observation: shard-wiki's existing **derived views** (UC-05, UC-17UC-20) are, in
zzstructure terms, *dimensions + rasters*: RecentChanges is a raster along `d.recent`,
AllPages along `d.alphabetical`, SiteMap is an H-view over the two namespace
dimensions. zzstructure offers a **single unifying vocabulary** for what shard-wiki
currently treats as a grab-bag of separate derived views.
Observation 2: ZigZag **clones** are conceptually the same move as Xanadu
transclusion ("the same content knowably in more than one place") — the two Nelson
projects converge here. A transcluded span / equivalent page is a clone that
participates in multiple ranks.
---
## 6. Is zzstructure the right model for shard-wiki? (recommendation)
**Adopt as a lens, not a substrate.**
- **As a navigation & visualization model: promising.** Letting a reader pick which
*dimension* to traverse (namespace vs. genealogy vs. version vs. shard) and pivot
two dimensions into an H-view is a real, differentiated UX for a federated,
provenance-preserving wiki. It operationalizes "union without erasure": every
relationship is co-equal and visible, none hidden. (UC-47, UC-48.)
- **As a conceptual model for the page graph: useful and clarifying.** Reframing the
scattered derived views as *dimensions over one cell set* is a clean mental model
and a candidate internal index/API shape. It also de-privileges the namespace tree
(UC-49) — consistent with **mechanism over policy** (the hierarchy is one
configurable dimension, not the canonical structure).
- **As the storage substrate: reject.** Git and heterogeneous sovereign shards remain
canonical (per INTENT Stability Note — Git's role is architectural). A zzstructure
is a **derived, projected index** computed over the union, like BackLinks today —
not a database that replaces the shards. The many-to-many hyperlink graph does not
fit the rank constraint, and forcing the wiki into a zz-DB would violate shard
sovereignty and the orchestrator-not-engine boundary.
Net: zzstructure is a strong answer to *"how do we present a page's many
simultaneous relationships without picking a winner?"* — a **dimensional projection**
layer over the existing union, not a new store.
---
## 7. Mapping to shard-wiki INTENT (compare, do not equate)
### 7.1 Reinforcements
- **Union without erasure ↔ co-equal dimensions.** zzstructure's refusal to privilege
any one structure is the data-model expression of shard-wiki's refusal to hide
provenance, conflicts, or alternative arrangements.
- **Mechanism over policy ↔ hierarchy-as-just-a-dimension.** The namespace tree being
one dimension among many is exactly "do not hard-code one canonical structure."
- **Provenance ↔ genealogy dimension.** Created-from lineage as a navigable rank makes
provenance a first-class traversal, not metadata buried in a footer.
### 7.2 Deliberate divergences (design bugs if conflated)
1. **zzstructure as store.** Replacing Git/shards with a zz-database would break the
Stability-Note boundary on Git's role and on orchestrator-vs-engine. zzstructure is
a projection, full stop.
2. **Forcing the hyperlink graph into ranks.** The link/backlink graph is many-to-many
and must stay a graph index; do not contort it into a single dimension. Use clones
or keep it as a separate graph projection.
3. **One global zz-space.** Like Xanadu's docuverse, a single universal zz-space would
fight shard sovereignty. Each information space gets its *own* dimensional
projection over *its* attached shards.
### 7.3 What ZigZag teaches that shard-wiki should keep
- A page is best understood as **a cell at the intersection of many independent
relationships**, not as a node in one tree. Build the page model and the navigation
API so *adding a new relationship is adding a dimension*, not special-casing a view.
- **Pivoting two relationships** (H-view) is a powerful, under-used exploration
primitive for federation (namespace × shard, page × version, topic × equivalent).
---
## 8. Use-case seeds → catalog (promoted 2026-06-14)
Last existing UC is **UC-46**. New UCs **UC-47UC-49** added; existing UCs enriched.
| Seed | Catalog action |
|------|----------------|
| **Navigate along a chosen relationship dimension** — pick namespace / genealogy / links / version / shard as the active traversal axis | **UC-47 (new)** |
| **Pivot two dimensions as a cross-tab (H-view)** — e.g. namespace × shard, page × version-across-shards | **UC-48 (new)** |
| **First-class created-from / fork genealogy dimension** — navigate derivation lineage as a rank | **UC-49 (new)** |
| Namespace hierarchy is *one dimension among many*, not the privileged structure | **enriches UC-22** |
| Derived views (RecentChanges, AllPages, SiteMap, BackLinks) reframed as dimensions + rasters | **enriches UC-05** (anchor for the UC-17UC-20 view reframing) |
| Fork/remix produce a genealogy edge consumed by UC-49 | **enriches UC-26, UC-29** |
| Clone = "same content in many places" = transclusion convergence | links UC-32, UC-44/45 |
---
## 9. Architecture notes for SHARD-WP-0002 (no UC)
- A **dimensional projection layer** over the union is a candidate internal model:
pages = cells; relationships (namespace, genealogy, version, shard, equivalence,
overlay, recent, alphabetical) = dimensions; existing derived views = rasters over
them. It is **derived/lazy**, never canonical store.
- The **link/backlink graph stays a separate many-to-many index**; do not model it as
a rank. Reconcile with the dimensional model via clones or by keeping graph and
dimensional projections side by side.
- **Genealogy** (created-from) should be recorded as an edge when fork/remix/import
happens (UC-26, UC-29) so the `d.genealogy` rank can be reconstructed — ties to the
coordination journal.
- Whether the dimensional model is exposed in a **public navigation API** or stays an
internal organizing concept is open (see §10).
---
## 10. Open questions (for spec / workplans)
1. Is the dimensional/zzstructure model a **public navigation API** and UI paradigm,
or only an *internal* organizing concept for the existing derived views?
2. How is the **many-to-many link graph** reconciled with the rank-based dimensional
model — clones, a parallel graph index, or both?
3. Which dimensions are **core** (namespace, version, shard, genealogy, recent) vs.
**adapter-provided** vs. **computed on demand** (equivalence, backlinks)?
4. Is **H-view pivoting** worth a reference-UI investment, or is it research-only?
5. Does the **clone = transclusion** convergence justify a single internal primitive
shared by UC-44/45 (transclusion) and UC-47/48 (dimensional navigation)?
---
## 11. Sources
| Source | Used for |
|--------|----------|
| Wikipedia — ZigZag (software) (https://en.wikipedia.org/wiki/ZigZag_(software)) | History, spreadsheet-deconstruction framing, one-connection-per-dimension rule, pivoting |
| Wikipedia — Zzstructure (https://en.wikipedia.org/wiki/Zzstructure) | Directed-coloured-multigraph definition, dimensions-are-cells, raster vs. view |
| McGuffin & schraefel, "A Comparison of Hyperstructures: Zzstructures, mSpaces, and Polyarchies," ACM Hypertext 2004 (https://www.dgp.toronto.edu/papers/mmcguffin_HT2004.pdf) | Graph-theoretic formal definition; taxonomy vs. mSpaces/polyarchies |
| McGuffin, "A Graph-Theoretic Introduction to Ted Nelson's Zzstructures" (https://www.dgp.toronto.edu/~mjmcguff/research/zigzag/) | Ranks, I-view/H-view/augmented views, clones, tree-as-left-child/right-sibling, "multiple arrangements coexist via dimensions" |
| Moore & Brailsford, "Unified Hyperstructures for Bioinformatics," JoDI (https://jodi-ojs-tdl.tdl.org/jodi/article/download/jodi-138/127) | Applied zzstructure as a unifying model over heterogeneous structures |
| Nelson, "A Cosmology for a Different Computer Universe" (zzstructure context) | Nelson's own framing of cells/dimensions/visualization |
| Gzz — "A Gentle Introduction to Ted Nelson's ZigZag Structure" (https://www.nongnu.org/gzz/gi/gi.html) | Practical cell/dimension/rank examples (rate-limited; corroborated by above) |
Cross-references: `research/260614-xanadu-deep-dive/findings.md` (clone↔transclusion
convergence), `spec/UseCaseCatalog.md` (UC-05, UC-17UC-22, UC-26, UC-27, UC-29,
UC-32, UC-44/45/46), `workplans/SHARD-WP-0002-federation-architecture.md`.
---
## 12. Traceability
- New UCs: **UC-47, UC-48, UC-49**`spec/UseCaseCatalog.md`.
- Enriched UCs: **UC-05** (anchors the UC-17UC-20 derived-view reframing), **UC-22,
UC-26, UC-29**.
- Architecture (no UC): dimensional projection layer; link-graph-stays-graph;
genealogy edges in the journal; clone↔transclusion primitive → `SHARD-WP-0002`.
- Decision boundary recorded: zzstructure is a **derived lens, not the store**;
Git/shards remain canonical (INTENT Stability Note).
</content>

View File

@@ -13,4 +13,35 @@ when multiple files or sources are involved. Findings here inform `spec/` and
| 2026-06-08 | `260608-yawex-prior-art/` | yawex 0.7.4 Perl wiki prior art; federation design seeds |
| 2026-06-08 | `260608-c2-wiki-origins/` | Ward Cunningham & WikiWikiWeb origins; terms and use cases |
| 2026-06-08 | `260608-federation-concepts/` | Federated Wiki, git/ActivityPub/Xanadu federation models |
| 2026-06-08 | `260608-wikiengines-overview/` | Wiki engine landscape survey (Perplexity-assisted) |
| 2026-06-08 | `260608-wikiengines-overview/` | Wiki engine landscape survey (Perplexity-assisted) |
| 2026-06-13 | `260613-xwiki-deep-dive/` | XWiki impl, extension interfaces, ecosystem; UC-38/39 |
| 2026-06-13 | `260613-twiki-deep-dive/` | TWiki impl, plugin API, ecosystem; UC-40/41 |
| 2026-06-13 | `260613-foswiki-deep-dive/` | Foswiki store abstraction, extension API; UC-42/43 |
| 2026-06-14 | `260614-xanadu-deep-dive/` | Project Xanadu — EDL/reference-not-copy, transclusion, addressing; UC-44/45/46 |
| 2026-06-14 | `260614-zigzag-deep-dive/` | ZigZag/zzstructure — information space as orthogonal dimensions; UC-47/48/49 |
| 2026-06-14 | `260614-roam-deep-dive/` | Roam Research — block-graph DataScript DB, transclusion, datalog, Roam Depot extension API; UC-50/51/52 |
| 2026-06-14 | `260614-obsidian-deep-dive/` | Obsidian — file-over-app vaults, plugin API, ecosystem-popularity → UC signal; UC-53/54/55/56 |
| 2026-06-14 | `260614-notion-deep-dive/` | Notion — closed block-DB SaaS, external REST API only, database-as-pages; UC-57/58/59 |
| 2026-06-14 | `260614-shard-spectrum-synthesis/` | Synthesis (v3) — shard family matrix + **fourteen** capability spectra across ~23 systems + **federation-model taxonomy** (fork+journal / VCS-replication+ping / query-join / engine-mirror); feeds SHARD-WP-0002 T1T6 + T11T16 |
| 2026-06-14 | `260614-joplin-deep-dive/` | Joplin — SQLite-local/Markdown-on-sync, interchange-format attach, E2EE content opacity; UC-60/61 |
| 2026-06-14 | `260614-logseq-deep-dive/` | Logseq — block-graph on plain Markdown files, in-file block IDs, derived Datalog index; UC-62/63 |
| 2026-06-14 | `260614-localfirst-workspaces-deep-dive/` | Anytype · AFFiNE · AppFlowy — CRDT local-first workspaces (any-sync/Yjs/Yrs), native merge, P2P/E2EE; UC-64/65 |
| 2026-06-14 | `260614-trilium-deep-dive/` | Trilium/TriliumNext — note cloning (DAG hierarchy), attribute inheritance/templates, HTML-native, scripting+ETAPI; UC-66/67 |
| 2026-06-14 | `260614-wikijs-deep-dive/` | Wiki.js — storage-module engine (DB↔Git Markdown), GraphQL API, pluggable modules ≈ adapter-contract prior art; UC-68/69 |
| 2026-06-14 | `260614-federated-wiki-deep-dive/` | Federated Wiki — fork-with-provenance, per-page semantic-action journal (story=replay), neighborhood/roster + chorus; prior art for our coordination journal / overlay / union pillars; UC-70/71/72 |
| 2026-06-14 | `260614-wikibase-deep-dive/` | Wikibase/Wikidata — typed entity-statement knowledge graph (claim+qualifiers+refs+rank), SPARQL/RDF + federated SERVICE, opaque stable IDs, statement-level provenance; structure & query far-end; UC-73/74/75 |
| 2026-06-14 | `260614-forge-wikis-deep-dive/` | Gitea · GitLab · GitHub wikis — a wiki is a separate `.wiki.git` of Markdown; git-clone universal, wiki API capability-varying (GitHub git-only); git IS the store (resolves UC-68 race); the home case; UC-76/77 |
| 2026-06-14 | `260614-tiddlywiki-deep-dive/` | TiddlyWiki — entire wiki (content + app) in one self-contained HTML file = whole-file write-granularity extreme; Node `.tid` file-per-tiddler substrate (git-diffable); tiddler/field records, filter-expression query; UC-78 |
| 2026-06-14 | `260614-ikiwiki-deep-dive/` | ikiwiki — wiki compiler: git-canonical Markdown source → static HTML (derived publish/projection); git-distributed clone federation + XML-RPC pinger (third federation flavor); UC-79 |
| 2026-06-14 | `260614-quip-deep-dive/` | Salesforce Quip — closed-SaaS live docs + embedded spreadsheets/live apps; REST + lossy HTML import/export; Salesforce enterprise ACL; external-API payload-format facet + inline-object page model; UC-80 |
| 2026-06-14 | `260614-mojomojo-deep-dive/` | MojoMojo — Perl Catalyst/DBIx::Class DB-backed wiki; pages + history in SQL tables, no file store/API → direct-DB-read binding; DB version rows as history source; UC-81 |
| 2026-06-14 | `260614-oddmuse-deep-dive/` | Oddmuse — single Perl CGI, plain-text `page/` files + `keep/` revisions, no DB; the minimal file-store floor / graceful-degradation baseline; partial-history honesty; UC-82 |
| 2026-06-14 | `260614-usemodwiki-deep-dive/` | UseModWiki — flat-file ancestor (Wikipedia's MediaWiki Phase I); CamelCase linking; lineage grounding for the minimal file-store floor; enrichment-only (reinforces UC-82, UC-25) |
| 2026-06-14 | `260614-literate-programming-deep-dive/` | Literate programming (Knuth's WEB / weave / tangle) — one source → N co-equal derived projections (docs + code); named-chunk transclusion; splits replication- vs derivation-projection; SHARD-WP-0004 T1; UC-83 |
| 2026-06-14 | `260614-jupyter-deep-dive/` | Jupyter Notebooks — `.ipynb` JSON cells + embedded computed outputs with fragile execution provenance; derived output stored *inside* the source; non-Markdown/lossy; kernel = capability; SHARD-WP-0004 T3; UC-84 |
| 2026-06-14 | `260614-glamorous-toolkit-deep-dive/` | Glamorous Toolkit (moldable development on Pharo) — `gtView` open set of co-equal type-keyed computed views (none canonical) = moldable view registry; Lepiter live notebook over git files; SHARD-WP-0004 T7; enrichment-only (UC-47/48/54) |
| 2026-06-14 | `260614-mathematica-deep-dive/` | Mathematica Notebooks — the original computational notebook (`.nb` = a Wolfram expression); nestable cell groups, structured re-evaluable outputs, `Manipulate` live widgets, CDF; confirms UC-84 notebook shape is a genus; SHARD-WP-0004 T2; enrichment-only (reinforces UC-84; UC-54/55) |
| 2026-06-14 | `260614-squeak-pharo-deep-dive/` | Squeak & Pharo (image-based Smalltalk) — the live-object image (purest "live" end); image-is-not-a-store boundary (export→files only); Pharo Tonel/Iceberg externalizes code to git text; names the live↔snapshot projection axis; SHARD-WP-0004 **T6 + T8** (merged); boundary/enrichment-only, no new UC |
| 2026-06-14 | `260614-processing-deep-dive/` | Processing / p5.js — program-as-page rendered live at view time (no cached output); adds materialization-timing + continuity facets to projection; execute-in-viewer = capability+trust; SHARD-WP-0004 T4; enrichment-only (UC-54/55) |
| 2026-06-14 | `260614-strudel-deep-dive/` | Strudel.cc (TidalCycles JS) live-coding REPL — code as live time-based audio performance; the far live end (no faithful static form; static = source + marked recording); honesty test for graceful degradation; SHARD-WP-0004 T5; enrichment-only (UC-54/55) |
| 2026-06-15 | `260614-computational-page-model-synthesis/` | **SHARD-WP-0004 synthesis***the computational page model*: source canonical / everything rendered is a projection; two axes (replication↔derivation, live↔snapshot); four computational page shapes; recommendation — executable content in scope as page-model+projection, out of scope as an execution platform (no INTENT amendment); feeds SHARD-WP-0002 T11T16 |

View File

@@ -0,0 +1,976 @@
# CoreArchitectureBlueprint — shard-wiki
Status: **draft for review** · Date: 2026-06-15 · Owner: tegwick
The whole-system architecture for shard-wiki, synthesised from `INTENT.md`, the 84-entry
`UseCaseCatalog.md`, and the full research arc (`research/260608-*`, `research/260613-*`,
`research/260614-*` — ~23 wiki/knowledge systems plus two cross-dive syntheses). This is the
**core** blueprint: it defines the layers, the abstractions, and the load-bearing decisions
that everything else implements.
Scope relationship to the other specs:
- **`ArchitectureBlueprint.md`** (existing) is the **authorization & history sub-blueprint**
(the L0L4 ladder). This document references it as the design of the cross-cutting
authorization layer (§9) and does not restate it.
- **`SHARD-WP-0002`** is the workplan that turns §6§8 into
`spec/FederationArchitecture.md` + the adapter-contract section of
`spec/TechnicalSpecificationDocument.md`.
- **`UseCaseCatalog.md`** is the demand this architecture must satisfy; UC references below
are load tests, not decoration.
- **`WikiEngineCoreArchitecture.md`** designs shard-wiki's native, headless, API-first wiki
engine as a **canonical-mode shard backend** (one shard behind §6/§A — federation, union, and
projection stay here in the orchestrator, not in the engine). Added per the 2026-06-15 INTENT
amendment (decision `84ffdb48`, SHARD-WP-0013).
---
## 1. The thesis: *canonical vs derived* (three states)
Everything in shard-wiki follows from one organising decision — that state comes in exactly
**three kinds**, and only one of them is disposable:
> **1. Sharded-canonical** — content owned by each shard (shard sovereignty).
> **2. Coordination-canonical** — durable state *born inside shard-wiki* that encodes human
> or cross-shard decisions and exists nowhere else: overlays (the local truth against a
> read-only shard), curator equivalence bindings, alias tables, merge/reconciliation
> decisions. It is recorded as an **append-only decision log in the Git coordination
> journal** (event-sourced, §8.1); the *queryable current form* of that state (the effective
> alias table, the equivalence set) is a **derived fold** of the log — i.e. tier 3, not tier 2.
> What is canonical is the **log of decisions**, not any mutable snapshot of them.
> **3. Derived-disposable** — everything shard-wiki *computes* from (1)+(2): the union graph,
> equivalence index, query indexes, projections, views. It can be deleted and recomputed.
>
> **Canonical = sharded coordination. Derived = a pure function of canonical:**
> `derived = f(sharded, coordination)`.
This is the architectural form of "orchestrator, not engine." shard-wiki never *becomes* the
source of truth; it composes sources and records the decisions it makes about them. The
research earned the *files-canonical* half empirically — every serious system externalises its
durable truth to files+VCS and treats the rest as derived: Logseq (DataScript index over plain
files), ikiwiki (static HTML compiled from a git repo), Glamorous Toolkit / Lepiter (live
views over git-versioned JSON), Pharo (Tonel/Iceberg code as git text), Jupyter teams
(nbstripout — outputs are derived noise). The one tradition that refuses this — the Smalltalk
**image** — is exactly the one we record as a *boundary, not a backend*
(`research/260614-squeak-pharo-deep-dive`).
The earlier draft of this blueprint used a two-bucket framing ("canonical at the edges,
derived in the middle"). That was wrong by omission: it had no home for **coordination-
canonical** state, and so contradicted itself by listing curator bindings and alias tables as
"derived/rebuildable" when a human binding manifestly cannot be rebuilt. The three-state model
fixes that crack (review finding A-1) and makes `derived = f(canonical)` *literally* true.
Three consequences fall straight out, and they are the spine of the rest of this document:
1. **Graceful degradation is free.** If the derived tier is always recomputable, a backend
that can only be read is still a first-class participant — you just derive less from it.
2. **Provenance is tractable.** Because shard-wiki never claims to *be* the source, every
derived artifact can always point back to the canonical input it came from (union without
erasure is a structural property, not a feature bolted on).
3. **The derived tier is a pure function of canonical state.** `derived = f(sharded,
coordination)`. Bugs in the derived tier are recoverable by recompute; only the two
canonical tiers must be durably protected — sharded by each shard, coordination by the
Git journal (history). *Recomputability is a correctness property of the derived tier, not
a promise that a from-scratch rebuild is operationally cheap — see §8.4 and the
operational-envelope axis.*
### The dual narrow waist
Heterogeneity is mediated at exactly two interfaces, and nowhere else:
- **Bottom waist — the Shard Adapter Contract (§6).** Every backend, however weird, enters
through one versioned, capability-described interface.
- **Top waist — the Wiki Page Model (§7).** Every consumer, however demanding, sees one
backend-neutral, Markdown-first-but-stretchable page model.
Between the waists, core logic is written **once** against capabilities and the page model —
never against a specific backend. Adding TiddlyWiki or Notion or a git forge is writing an
adapter and declaring a capability profile, not editing core algorithms.
---
## 2. Architectural invariants
These are non-negotiable. Violating one is a design bug, not a tradeoff. They are INTENT's
principles fused with the research through-lines.
| # | Invariant | Source |
|---|-----------|--------|
| I-1 | **Orchestrator, not engine.** Core composes shards; it never replaces or homogenises them. | INTENT Stability Note |
| I-2 | **Three states; derived = f(canonical).** State is sharded-canonical, coordination-canonical (journal), or derived-disposable. The derived tier (union/index/projection) is a pure, recomputable function of the two canonical tiers; only canonical state is durably protected. | §1; Logseq/ikiwiki/GT through-line |
| I-3 | **Capability-awareness is data.** A binding's abilities are a *profile* (positions on spectra), read by generic core logic — not per-backend branches. | synthesis v3 §2; INTENT capability-aware adapters |
| I-4 | **Union without erasure.** Every page/revision/projection/overlay/view carries its provenance, freshness, liveness, and divergence. | INTENT; provenance-granularity spectrum (Wikibase) |
| I-5 | **Overlay before mutation.** Writes to anything below write-through land as drafts/patches/MRs first; no silent remote mutation. | INTENT |
| I-6 | **Git-addressable coordination.** Every information space has a Git-backed journal even when its shards are not git-native. | INTENT |
| I-7 | **Mechanism over policy.** Canonical-source, conflict, editorial, sync cadence are configurable presets, never hard-coded. | INTENT |
| I-8 | **Graceful degradation.** A limited backend is still usable as read-only / cache / projection / backup / patch target. | INTENT |
| I-9 | **Identity ≠ placement.** A page is an entity that may occupy N locations; address by identity, not by path. | Trilium note/branch; ZigZag |
| I-10 | **History is the floor.** Every write is a recoverable commit; recoverability, not gatekeeping, is the baseline protection. | ArchitectureBlueprint §2 |
| I-11 | **Authorization in core, authentication delegated.** Core decides who-may; an external provider says who-is. | INTENT; ArchitectureBlueprint |
| I-12 | **Not a file-sync daemon; not an execution platform.** Sync is wiki-page-semantic; computation is recognised+projected, not hosted. | INTENT; computational-page-model synthesis |
| I-13 | **Tenant-partitioned derived state.** Derived state is partitioned by tenant/root entity; no derived artifact spans tenants except via explicit, authorised cross-root federation. | §9.1; review B-3 |
---
## 3. The layered architecture
```
┌───────────────────────────────────────────────────────────────┐
│ L6 Consumers — Orchestrator API · CLI/agents · Web/Obsidian │
├───────────────────────────────────────────────────────────────┤
X-cut │ L5 Authorization (PEP/PDP, identity-provider iface) → │ X-cut
Prove- │ see ArchitectureBlueprint.md (L0L4 ladder) │ Capa-
nance ├───────────────────────────────────────────────────────────────┤ bility
▲ │ L4 Union & Projection (DERIVED · rebuild=fallback) │ ▲
│ │ identity resolution · equivalence/chorus · union graph · │ │
│ │ replication+derivation projections · moldable view registry│ │
│ │ · derived query index │ │
│ ├───────────────────────────────────────────────────────────────┤ │
│ │ L3 Coordination (Git journal · overlay/patch engine · │ │
│ │ federation-model strategies · reconciliation) │ │
│ ├───────────────────────────────────────────────────────────────┤ │
│ │ L2 Wiki Page Model ── TOP WAIST ── │ │
│ │ backend-neutral pages · identity≠placement · span address ·│ │
│ │ provenance envelope · the page shapes │ │
│ ├───────────────────────────────────────────────────────────────┤ │
│ │ L1 Shard Adapter Contract ── BOTTOM WAIST ── │ │
│ │ versioned iface · capability profile (orthogonal) · │ │
│ │ attachment-mode binding · operation verbs │ │
└──── ├───────────────────────────────────────────────────────────────┤ ──┘
│ L0 Backends (not ours): git repos, wiki/ subdirs, Gitea/ │
│ GitLab/GitHub wikis, folders, Obsidian, WebDAV, Notion, │
│ Coulomb spaces, notebooks, … │
└───────────────────────────────────────────────────────────────┘
```
**Provenance** and **Capability** are drawn as vertical rails because they are not layers —
they are present at every layer. A page object at L2 carries provenance; a projection at L4
carries provenance; an authz decision at L5 records the context under which content was read.
Likewise a capability profile declared at L1 is consulted at L3 (can we write-through?), L4
(can we delegate a query?), and L5 (can this principal even reach the op?).
The dependency rule is strict and downward, and it tracks the **three states (§1)**, not the
layer numbers: **the derived-disposable tier (the whole of L4) may be deleted and recomputed
from canonical state (sharded content at L1 + coordination-canonical state in the L3 journal).**
Nothing canonical may depend on derived state. Note the journal at L3 is *canonical* (it holds
overlays, bindings, aliases, merges); only L4 is disposable.
---
## 4. Core abstractions (the vocabulary code must use)
Straight from INTENT, sharpened by research. New code maps onto these; it does not invent
parallel terms.
- **Shard** — an independently meaningful page store attached to a root entity, with
*sovereignty*: its own backend, capability profile, history, identity model, limits.
- **Root entity / information space** — the joined space shards attach to; the unit of
Git coordination and of multi-tenancy (a tenant maps to a root entity, ArchitectureBlueprint).
- **Shard adapter contract** — the versioned L1 interface; the bottom waist.
- **Capability profile** — a shard binding's position on each of the 15 spectra (§6) plus its
supported verbs. *The* data structure that drives degradation.
- **Wiki page model** — the L2 backend-neutral page; the top waist.
- **Page identity vs placement vs equivalence** — a page is an entity with a *stable handle*
(identity); it may have N placements (paths/shards); **addressing and transclusion key on
identity, but equivalence keys on content fingerprint *across* identities** (§7.2, I-9). The
three are distinct mechanisms — never conflate identity with a fingerprint.
- **Provenance envelope** — the metadata each artifact carries (source shard, freshness,
liveness, authz context, overlay status, divergence, lineage), stored **layered**: a
page-level envelope + span-level *deltas*, so per-span cost is near-zero when uniform (§7.3).
- **Coordination journal** — the L3 Git-backed, **append-only decision log** for a space: the
durable home of all **coordination-canonical** state (§1, §8.1) as *events* (overlay-created,
binding-made, alias-set, merge-decided), plus the content change-flow record. It is event-
sourced — committed, never overwritten; the queryable current coordination state is a derived
fold of it (§8.1).
- **Overlay** — a non-destructive local edit against a remote/read-only/limited shard,
representable as draft/patch/commit/MR before destructive apply. Coordination-canonical: an
unapplied overlay is the local truth and lives in the journal.
- **Projection** — a derived view of shard content. The default is a **plain lazy
replication-projection** (a freshness-stamped cache); only *source* content needing
transform/evaluate uses the **derivation-projection** extension point with its two-axis
typing (kind × liveness) and the moldable view registry (§8.4§8.5).
- **Federation model** — the selected coordination strategy for a space (§ taxonomy, T17).
- **Shard mode** — read-only · write-through · mirrored · projected · cached · canonical
(a *policy* selection constrained by the capability profile).
---
## 5. Why "layered" and not "pipeline" or "plugin-bus"
Two rejected alternatives, recorded so the choice is legible:
- **A sync pipeline** (source → transform → sink) was rejected: it implies a privileged
direction and a canonical sink, which violates shard sovereignty (I-1) and union-without-
erasure (I-4). shard-wiki is a *star* (many shards ↔ one space), not a pipe.
- **A flat plugin bus** (every backend a peer plugin emitting events) was rejected as the
*top-level* shape: it has no narrow waist, so heterogeneity leaks into every consumer.
We keep the plugin idea but confine it to L1 (adapters) and L3 (federation strategies),
behind the waists.
The layered-with-rails shape is what makes I-2/I-3/I-4 hold simultaneously.
---
## 6. Bottom waist — the Shard Adapter Contract (L1)
The single most important design decision in the project: **the adapter contract models
positions on capability spectra, not a flat checklist of boolean verbs.** A backend is not
"can/can't merge"; it sits *somewhere* on the merge spectrum, and federation operations
degrade by position. This is the lesson of putting ~23 systems in one matrix
(`research/260614-shard-spectrum-synthesis`, v3).
### 6.1 The fifteen capability spectra
Each binding declares a position on each axis. Core algorithms read these positions; there is
no per-backend code in core (I-3).
1. **Addressing granularity** — none → path → page-level store-id → in-file span → in-file
block id (Logseq `id::`) → store-UUID → portable tumbler (Xanadu, the unreached ideal)
2. **Content identity** — none → path/title → fingerprint → span-set
3. **Identity vs placement** — path=identity → identity separated from placement (Trilium
note/branch = a DAG)
4. **Structure** — flat MD → frontmatter/`key::` → `%META%` → typed objects → DB schema+
relations → object-graph/ontology → computed (inherited+templated) → typed-graph statements
5. **History** — none → internal-only / CRDT-log → open-file → git-native
6. **Merge model** — none → git/text → conflict-notes/keep-both → native-CRDT → coexist-with-rank
7. **Native query** — none → text → build-your-own derived index → datalog/graph → DB query → SPARQL
8. **Translation** — native → lossless → lossy-with-fidelity-report (incl. HTML)
9. **Attachment mode** — file-store (native | interchange-mirror) → git-IS-store → in-engine-host
→ local-REST → external-API → direct-DB → CRDT-replica → P2P/no-central-endpoint
10. **Operational envelope** — local/unbounded → realtime CRDT/WebSocket → rate-limited/
eventually-consistent/paginated
11. **Access grant** — open → token → OAuth scoped+revocable → P2P key/invite → enterprise ACL
12. **Content opacity** — plaintext → structured re-evaluable value → encrypted whole-shard →
per-item → proprietary-lossy-exportable
13. **Write granularity** — whole-file (TiddlyWiki) → per-page → section/anchor → per-block → story-item
14. **Provenance granularity** — per-shard → per-page → per-edit → per-statement/value (Wikibase rank+refs)
15. **Computational / liveness** — static source → captured-output snapshot → live-over-files →
view-time render → irreducibly-live/temporal
### 6.2 Operation verbs
`read, write, diff, merge, lock, version, publish, notify, transclude-source,
translate-syntax, structured-payload, derive-projection, execute/evaluate`. The last two are
**gated, off by default** (§8, computational content). Verb support is part of the profile and
must reconcile with the federation-ops capability matrix (SHARD-WP-0002 T10).
### 6.3 Attachment-mode taxonomy (axis 9, expanded)
A backend may offer **several** modes; attach mode is a **per-binding, capability-gated
choice**, with one declared authoritative. Modes: file-store (native vault/folder *or* an
interchange/sync mirror), **git-IS-store** (the home case — forge wikis & ikiwiki: git is the
store *and* the journal at once, resolving the engine-mirror write-race), in-engine hosted
adapter (XWiki component, Obsidian/Logseq/Roam plugin, Trilium script), local-REST (Joplin
Data API, Trilium ETAPI), external-API-only (Notion), direct-DB (MojoMojo schema→model),
CRDT-replica (Anytype/AFFiNE/AppFlowy), P2P/no-central-endpoint. **Boundary:** a monolithic
live-memory blob (Smalltalk image, a kernel) is **never** an attach target — it participates
only via export→files (I-12).
### 6.4 Contract rules
- **Versioned interface** (Foswiki::Store + Foswiki::Meta is the proof that a stable
store-interface-with-swappable-backends works). Capability discovery is a static profile
with optional runtime negotiation.
- **Backend-swap tolerance** — shard identity/provenance survives a substrate change
(RCS↔PlainFile, folder→Git, Logseq file→SQLite): bind to *capabilities*, not to "it's files."
- **Absence is first-class** — the profile must express *can't* cleanly (Oddmuse floor), so
degradation paths are explicit, never guessed.
### 6.5 Orthogonal core, implied positions, and the interaction subset
Fifteen independent ordinal axes is *descriptively* right but would be *operationally* a mess
if treated as fifteen free dimensions: the axes are **not orthogonal**, and a degradation
function over all 15 jointly is the flat-checklist problem returning in higher dimensions
(review D-1). Three rules tame it.
**(a) A small orthogonal core; the rest are implied.** Most axes are *correlated* and collapse
to a few independent choices. The **core axes** an adapter must independently declare:
1. **Substrate** → drives attachment-mode, history, merge, and native-query positions together
(git-IS-store ⟹ history=git-native ⟹ merge=git/text ⟹ query=build-your-own-index;
relational-DB ⟹ direct-DB attach ⟹ DB-version-row history ⟹ DB query).
2. **Write granularity** → drives addressing granularity and the overlay/patch shape.
3. **Content opacity** → drives translation and (where encrypted) collapses native-query.
4. **Operational envelope** → drives freshness mode (§8.8) and rebuild expectations (§8.7).
5. **Access grant** → independent (authz, L5).
6. **Computational/liveness** → independent (projection kind, §8.5).
The remaining axes are **implied/derived** from these via published implication rules; an
adapter *may* override an implied position, but the default is computed, not hand-set. This
turns ~15 free dimensions into ~6 independent ones plus derivations — fewer things to get
wrong, and impossible combinations become unrepresentable.
**(b) Implication rules forbid impossible profiles.** E.g. `attachment=git-IS-store ⟹
history≥git-native`; `opacity=encrypted-whole-shard ⟹ native-query=none ∧ translation≤opaque`;
`merge=native-CRDT ⟹ history=CRDT-log ∧ envelope=realtime`. A profile that violates an
implication is rejected at registration — capability-as-data (I-3) with integrity constraints.
**(c) The degradation function reads a *named, small* interaction subset — not all pairs.**
"No per-backend code" is only credible if we say *which* axis interactions the generic logic
actually consults. They are:
| Operation | Axes consulted (jointly) |
|-----------|--------------------------|
| **write / overlay-apply** | write-granularity × merge-model × history × access-grant |
| **transclude / address a span** | addressing-granularity × write-granularity × identity-vs-placement |
| **project / cache** | operational-envelope × computational-liveness × content-opacity |
| **query** | native-query × content-opacity (encrypted ⇒ derive-index-or-none) |
| **translate** | translation × content-opacity × structure |
| **federate** | substrate × history × merge (per the §8.3 model) |
Everything else is a single-axis check. This table *is* the degradation contract: it is small,
enumerated, and testable — the proof obligation behind "core logic written once."
### 6.6 Conformance — profiles are verified, never self-asserted
Capability-as-data (I-3) and the entire degradation contract (§6.5) rest on one assumption:
**the profile tells the truth.** If an adapter declares `merge=git/text` but corrupts merges,
or claims `notify` and never emits, it silently poisons every degradation decision in core —
the failure is invisible because core *believed the data* (review B-2). So the profile is not
taken on trust:
- **The contract ships a versioned conformance suite.** A published battery that, given a live
binding, **exercises each declared verb and each declared spectrum position and checks that
observed behaviour matches the claim** (a `write` round-trips; a `diff` is real; `notify`
actually fires; an "encrypted/opaque" shard genuinely refuses plaintext query; an
implication-rule position, §6.5(b), holds). The suite is versioned *with* the contract, so an
adapter proves conformance against a known contract version.
- **Passing conformance is an admissibility precondition.** A binding that fails (declares a
capability it does not honour) is **rejected at registration**, not run in production with a
lying profile. Capability discovery (§6.4) therefore yields a *verified* profile.
- **Self-reported, then verified.** Adapters still *declare* their profile (discovery stays
cheap); conformance *verifies* the declaration. The two together are what make I-3 and §6.5
sound rather than aspirational — degradation logic acts on verified data.
- **Mismatch is data, not a crash.** A conformance gap is reported as a precise
capability-by-capability diff (what was claimed vs observed), so an adapter author fixes the
profile or the code; degraded-but-honest registration (drop the unsupported claim) is allowed.
This is the same discipline a versioned store interface needs in general (the `Foswiki::Store`
lineage that inspired the contract): a backend may only participate behind the interface if it
*demonstrably* behaves as the interface says.
---
## 7. Top waist — the Wiki Page Model (L2)
Backend-neutral, **Markdown-first but stretchable many ways at once**. The page model is the
lingua franca every consumer sees; an adapter's job is to project its backend into this model
(read) and accept overlays back (write), within its capabilities.
### 7.1 Page shapes the model must carry
- **Prose Markdown** — the baseline.
- **Typed / computed records** — frontmatter/`%META%`/XObjects/Notion DB rows; **computed
metadata** (Trilium inherited+templated) represented as *effective-vs-own with per-attribute
provenance*.
- **Typed-graph statements** — Wikibase claim + qualifiers + references + rank (structure
far-end).
- **Inline-embedded objects** — Quip/Notion spreadsheets & live apps inside prose.
- **Non-Markdown assets** — drawings, canvases, images: typed asset / opaque blob / pluggable
content-type registry, never silent-flattened.
- **The four computational shapes** (§8): one-source-many-projections, notebook (embedded
computed output), program-as-page, live/temporal.
All shapes reduce to a common skeleton: **`(content | source, structure, provenance envelope,
[derivation rule])`**. The page model stores the richest faithful form as canonical and treats
any Markdown rendering of a non-Markdown shape as a *lossy projection* (I-4 + fidelity report).
### 7.2 Identity, placement, addressing — three distinct concepts
The earlier draft used "identity" for two different things and (worse) suggested deriving page
identity from a content fingerprint — which would make *editing a page change its identity* and
break every reference to it (review bug B-1). They are pulled apart here:
- **Page identity — a *stable handle*.** A shard-scoped, durable key that **survives edits**:
the backend's native page/note id where one exists (Roam/Notion/Trilium uid, a git path
treated as a name, a wiki page name), wrapped in a shard scope so it survives projection and
never collides across shards. Identity is *assigned/minted, not computed from content*.
References, placement, transclusion targets, and overlays all key on identity.
- **Placement — *where* an identity sits.** One identity → N placements (paths/shards) = a DAG;
no single canonical path (I-9). Placement can change without changing identity.
- **Content equivalence — *detecting sameness*, never identity.** A **content fingerprint** (or
span-set overlap) identifies a *version / a piece of content*, used to detect that two
*distinct identities* hold the same or derived content (the equivalence/chorus mechanism,
§8.4). A fingerprint is never a page's identity: same page, edited → new fingerprint, **same
identity**; two pages, identical content → same fingerprint, **different identities**.
- **Span addressing** — a sub-page address within an identity: adopt native span IDs where
minted (Roam `:block/uid`, Logseq `id::`, Notion/CRDT UUID); else a *position* address
(path+range) or a *content-fingerprint* address for equivalence/transclusion. The Xanadu
tumbler is the portable ideal the scheme aims at without requiring.
- **Provenance envelope** rides on pages and spans (see §7.3 for its layered, low-cost form).
So the chain is: **identity (stable) → placements (N, mutable) → equivalence (cross-identity
sameness, fingerprint-based)** — three concepts, three mechanisms, never conflated.
### 7.3 Provenance is layered, not per-span-duplicated
A provenance envelope on *every span* (source shard, freshness, liveness, overlay status,
authz context, divergence, lineage) would, at block granularity, mean ~10k near-identical
envelopes for a 10k-block page — provenance dwarfing content (review D-2). The fix is the exact
pattern the page model already uses for Trilium's computed metadata: **effective-vs-own**.
- **Page-level envelope** holds the values that are uniform across the page (almost always:
source shard, observed-at, liveness, authz context).
- **Span-level deltas** record *only where a span differs* from its page envelope — a
transcluded span from another shard, an overlaid span, a span that diverges. A span with no
delta inherits the page envelope at zero storage cost.
- **Effective provenance** for any span = page envelope ⊕ span delta, computed on read.
Per-span cost is therefore **near-zero in the common (uniform) case** and pays only for genuine
heterogeneity — the same "carry only the difference" principle, applied to shard-wiki's own
metadata. Provenance remains complete (I-4); it is just not redundantly materialised.
---
## 8. Coordination, federation & projection
### 8.1 Coordination journal (L3) — Git as the spine
Every information space has a Git-backed coordination journal (I-6). It records cross-shard
operations (fork, import, reconcile, overlay-apply, space-branch) and **is** the history floor
(I-10). For git-IS-store shards the shard's own git log *is* this journal; for non-git shards
the journal supplements (begins-now / mirrors-forward / snapshots-replica) or imports
(backfill open file history). History portability is a spectrum, handled per profile (axis 5).
**The journal is an append-only decision log; current coordination state is a derived fold
(review B-3).** The first draft said coordination-canonical state "lives in the journal"
without saying how Git — excellent for history, poor for mutable structured state — represents
an alias table or an equivalence graph. Resolution: **event sourcing.** The journal stores
*decisions as events* (`overlay-created`, `binding-made`, `alias-set`, `merge-decided`,
`page-forked`), append-only and git-addressable (so history/patch/review/backup over
coordination state come for free — I-6 is *strengthened*, not bypassed). The **queryable
current state** (the effective alias table, the live equivalence set) is a **derived fold** of
the log — tier-3 disposable, indexed like any other derived structure (§8.7), rebuilt by
replaying the log. So "all equivalences touching X" is an index lookup, not an O(scan) of Git.
This is the clean form of the §1 three-state model: **the log is canonical; its folded current
state is derived.**
**Concurrency: who may append (review B-1).** A multi-tenant L4 deployment runs several
orchestrator instances, so "the journal is local Git, single writer" is not given. The model:
- **One *append authority* per information space.** Appends to a space's log are serialized
through a single logical writer (a per-space lease/leader; instances without the lease forward
their append intents to it). This makes the log a **totally-ordered event sequence** per space
— the ordering authority §8.6 relies on — without a distributed transaction. Spaces are
independent, so this scales horizontally *across* spaces (the unit of partition is the space /
root entity, matching the tenant partition, I-13); it is a per-space serialization point, not
a global one.
- **Git is the durable, addressable form; appends are commits** (or fast objects batched into
commits) under the lease — no concurrent-writer merge races because there is one writer at a
time per space.
- **Read-your-writes** holds within a space because every reader resolves current state from
the same ordered log (or its fold); across spaces there is no shared state to be inconsistent.
- **HA / failover:** the lease is time-bounded and re-grantable; a failed append-authority is
replaced and resumes from the log's head (the log is the recovery point). A partition that
splits the authority degrades that *space* to read-only until a single writer is re-elected —
it never forks the log (availability yields to log integrity; an explicit, stated trade).
- **Open residual (→ §12, O-3-adjacent):** whether very high append rates need per-space log
*sharding* (sub-logs merged by a deterministic order) is an implementation spike, not an
architectural change.
**History must stay recoverable *and* bounded (review C-3).** "Every write is a commit" + open
L0 means an unbounded, bot-/vandalism-amplified journal that eventually degrades Git itself.
Recoverability (I-10) is non-negotiable, so the answer is *compaction, not deletion*:
- **Routine git maintenance** — background `gc`/repack, commit-graph, and (for very large
spaces) partial-clone / sparse strategies; operational, no semantic change.
- **Squash-compaction of low-value churn (policy, §10)** — long runs of rapid same-author
edits or revert-pairs can be folded into checkpoint commits *while preserving the recoverable
endpoints*; what is squashed is configurable and always leaves the content recoverable (it
compacts the *path*, not the *reachable states*).
- **Per-shard history offload** — a git-IS-store shard keeps its own history in its own repo;
the coordination journal references it rather than duplicating it (the journal records
*coordination* events, not a second copy of every shard commit).
- **Anti-abuse hooks (policy)** — rate-limiting / quarantine for anonymous L0 writers feed the
authz/policy layer; they throttle *abuse*, never legitimate history. Recoverability is the
floor; bounding is how it survives at scale.
### 8.2 Overlay / patch engine (L3)
The default write path for anything below write-through capability (I-5): an edit becomes a
draft → patch/commit → MR, applied destructively only on explicit intent and only where the
profile + policy both permit. This is what lets a read-only or rate-limited or lossy backend
still be *edited* safely.
### 8.3 Federation is plural & composable (L3) — the model taxonomy
Federation is not one mechanism. shard-wiki selects a **federation model per space and
composes per shard** (mechanism over policy, I-7):
| Model | Anchor | Coordination shape |
|-------|--------|--------------------|
| **Fork + journal** (default home case) | Federated Wiki | copy-with-provenance + per-page action journal (story = replay) |
| **VCS-replication + ping** | ikiwiki | git clone/pull/push + change-ping |
| **Query-time graph-join** | Wikibase SPARQL `SERVICE` | join remote graphs at query time, no copy |
| **Feed aggregation** | RSS/Atom | inbound feed → pages |
| **Activity streams** | ActivityPub | Create/Update events, notify or content-bearing |
| **Engine-mirror** | Wiki.js DB↔Git | engine syncs its own store to a git mirror |
### 8.4 Union & projection (L4) — the derived cache
This whole layer is **derived-disposable**: recomputable from canonical state — sharded
content + the **coordination-canonical** inputs in the journal (I-2). Crucially, the *automatic*
equivalence results are derived, but the **human/curatorial inputs they consume — alias tables
and curator equivalence bindings — are coordination-canonical (they live in the journal), not
derived**; recompute reads them, never regenerates them. It comprises:
- **Identity resolution & equivalence** — detect "same topic / derived content" path-
independently from *derived* signals (content fingerprint, span-set overlap) **plus** the
*coordination-canonical* inputs (alias table, curator binding); present as
**chorus-of-voices** or designated-canonical (a *policy* preset). (Scaling: §8.7.)
- **Union graph** — the navigable join of pages, links, and dimensions (namespace, genealogy,
version, shard, equivalence). A *derived lens over canonical files+journal, never a new
store* (the ZigZag boundary).
- **Transclusion** — one **reference-not-copy** primitive unifying Xanadu transclusion, ZigZag
clone, Roam/Obsidian/Logseq embed, Notion synced block, Trilium note-cloning, and literate
named-chunk assembly, over the addressable union.
- **Projection — trivial by default, extensible for the tail.** The 95% case (Markdown in a
shard) must cost nothing conceptually, so:
- **Default = plain lazy replication-projection** — a freshness-stamped cache of remote
content (§8.8). This is *the* projection for ordinary pages; it needs no taxonomy, no
liveness reasoning, no registry. Most shards never touch anything below.
- **Extension point — derivation-projection** — invoked *only* for content that is a
*source* needing transform/compile/weave/evaluate (computational/typed content, §8.5). It
adds the liveness axis (static → captured → live-over-files → view-time → irreducibly-live)
and facets (materialization timing, multiplicity, continuity); the irreducibly-live far end
has no faithful static form (source + a marked recording). A binding that never serves such
content never instantiates any of this.
- Both kinds stamp freshness + provenance; only derivation carries the liveness machinery.
- **Moldable view registry — also an extension point, not a tax on every page.** Where a content
type offers multiple co-equal views (typed/computed/dimensional content), they are registered
as an **open, type-keyed set, none canonical-by-fact** (display-canonical is policy; GT prior
art, answers the "pluggable content-type registry" question). An ordinary Markdown page has
exactly one view and never consults the registry — the registry is queried only when a type
declares >1 view.
- **Derived query index** — delegate to a shard's native query engine where present
(Roam/Logseq Datalog, Notion DB query, XWiki XWQL, Wikibase SPARQL); else build a derived
index over the projection (the Logseq DataScript-over-files pattern). The index is
disposable (I-2).
### 8.5 Computational / executable content — the scope decision
**In scope as a page-model + projection concern; out of scope as an execution platform.**
shard-wiki *recognises* computational types, attaches the **canonical source**, and presents
derived forms as **provenance- and liveness-marked projections**. Driving a derivation
(tangle/weave, re-execute a notebook, render a sketch, evaluate a pattern) is a **gated
capability, off by default, with a trust/sandbox concern, degrading to a captured snapshot**.
One snapshot-provenance record (run id, source rev, timestamp, environment "unguaranteed")
serves notebooks, renders, and recordings alike. **No INTENT amendment is required** — this
lives inside the existing page model (L2) and projection model (L4).
### 8.6 Consistency, concurrency & conflict model
INTENT makes real-time cross-shard consistency a non-goal — but "no strong consistency" is not
the same as "no defined consistency." This is the guarantee shard-wiki *does* offer, and the
mechanism (not policy) that makes concurrent editing safe (review bug B-2).
**The consistency guarantee — causal, anchored on the journal:**
- **Read-your-writes for coordination-canonical state.** Once an overlay/binding/merge is
appended to the space's decision log, every reader of that space sees it — because the log is
a **single totally-ordered sequence per space** (one append authority, §8.1), and all readers
resolve current state from that one order. The guarantee holds across orchestrator instances,
not just within one process; it is cheap because ordering is per-space, never global.
- **Causal consistency across the derived tier.** The union/index/projections reflect a causal
cut of `(sharded inputs seen so far, journal)`. Effects never appear before their causes; a
projection that has seen journal commit *C* has seen everything *C* depends on.
- **Eventual convergence for sharded-canonical inputs.** Remote shard content is pulled
asynchronously (lazily or by notify/poll, §8.7); the union converges to each shard's latest
*as observed*, bounded by the shard's operational envelope. Freshness is always *shown*
(provenance envelope), never faked — a stale projection is labelled stale, not wrong.
So: **strong + read-your-writes for what shard-wiki owns (the journal); causal for what it
derives; eventual + freshness-labelled for what shards own.** No global clock, no distributed
transaction, no two-phase commit across shards — none is needed, because shard-wiki coordinates
rather than controls.
**Conflict detection & representation is core mechanism; only resolution is policy (I-7).**
The split the earlier draft elided:
- **Detection (core).** Divergence is detected structurally: two identities resolve as
equivalent (§8.4) but their content fingerprints differ, or an overlay's base revision no
longer matches the shard's current revision. Detection is always on; it is never optional.
- **Representation (core).** A detected conflict is **first-class data in the union**, not an
error: equivalent-but-divergent pages are presented as a **coexisting set** (the
chorus/keep-both representation), each fully attributed, with the divergence recorded in the
provenance envelope (union without erasure — a conflict is information, not a failure).
- **Resolution (policy).** *Which* version wins, or whether they stay coexisting, is a
configurable preset (§10): chorus / designated-canonical / git-merge / vote-to-merge /
overlay-only. Core never hard-codes one.
**Overlay-apply under source drift (the concurrent-write case).** An overlay carries the
**base revision** of the shard content it was authored against. On apply, core compares base to
the shard's current revision:
- *unchanged* → apply (fast-forward), commit to journal, propagate if the profile permits;
- *changed, non-overlapping* → three-way merge where the merge capability allows (axis 6),
else keep-both;
- *changed, overlapping* → **refuse + re-present** as a conflict (above); never silently
clobber (I-5, no silent remote mutation). The unapplied overlay remains coordination-
canonical and valid against its base.
**Ordering.** The journal commit is the ordering authority for coordination-canonical effects;
a shard-native write is only *acknowledged* in the journal after the adapter confirms it, so a
crash between journal-intent and shard-write is recoverable (the intent is replayable, the
write is idempotent-keyed on identity+base-rev). Cross-shard operations are ordered by their
journal commits, giving the causal cut above.
**Residual open items** (tracked in *Known scaling risks & open problems*, §12, not pretended
solved): the exact convergence bound for
high-write CRDT shards under partition, and whether per-equivalence-set divergence needs a
vector clock vs. a simple base-rev comparison, are deferred to implementation spikes.
### 8.7 Scaling the union — incremental-first, rebuild as fallback
The derived tier is *recomputable* (I-2) but recompute must never be the **operational**
mechanism. A from-scratch rebuild reads every page of every shard — including rate-limited,
paginated external APIs (Notion) and irreducibly-live sources — which can take hours to days
and directly fights the operational-envelope axis (review C-2). So:
**Incremental, change-driven maintenance is the primary mechanism.** Each shard's `notify`
capability (or a poll/ETag fallback where it has none, §8.8) emits **change events**; an event
drives a **delta update** to exactly the affected union nodes, equivalence candidates, indexes,
and projections. The derived tier is a continuously-maintained materialised view, not a
periodically-recomputed one. Steady-state cost is O(changes), not O(corpus).
**Full rebuild is a rare, bounded fallback** — for cold start, schema/algorithm change, or
suspected corruption — and it is **explicitly not required to be cheap**. It respects each
shard's envelope (it may be slow, throttled, or resumable for a rate-limited shard) and runs
*concurrently with serving the existing derived tier*; it swaps in atomically on completion.
I-2 guarantees rebuild is *possible and correct*, not instant.
**Equivalence detection is indexed, not pairwise (review C-1).** Naive fingerprint/span-set
comparison across all pages of all shards is O(N²) and is forbidden. Instead:
1. **Blocking / candidate generation** — cheap keys bucket pages that *could* be equivalent:
normalised title, normalised path tail, explicit alias-table entries (coordination-
canonical), and **MinHash/LSH bands over content shingles** for near-duplicate and
derived-content detection. Only within-bucket pairs are considered — turning O(N²) into
≈O(N) candidates.
2. **Verification** — candidate pairs are confirmed by full fingerprint / span-set overlap and
any curator binding. Confirmed equivalences become union edges.
3. **Incremental maintenance — the delta is *not* additive (review B-4).** A changed page may
*leave* buckets as well as *enter* them, and leaving a bucket can **break an existing
equivalence edge** another page relied on. So a change is processed as: (i) recompute the
page's bucket membership; (ii) for buckets it **left**, re-verify the pairs that depended on
the shared bucket and **retract** edges no longer supported; (iii) for buckets it **entered**,
verify the new candidate pairs and **add** edges; (iv) **propagate** to the equivalence
neighbours of any retracted/added edge (equivalence is transitive-ish via chorus sets, so a
retraction can split a set). Maintenance is per-change and bounded by the page's
neighbourhood, but it covers retraction and propagation — not just additions.
**The index is itself derived** (disposable, recomputable) and per-tenant-partitioned (§9).
Its parameters (LSH band/row counts, shingle size, precision/recall) are tunable; the accepted
**false-negative rate of blocking** is a known, tracked limitation (§12) — blocking trades a
small miss rate for tractability, and curator bindings are the escape hatch for misses.
**Verifying I-2 (`derived = f(canonical)`) — eventually, not on faith (review B-4).**
Incremental maintenance can drift from a from-scratch fold over time (a missed retraction, a
dropped event, a bug). I-2 is therefore an **eventually-verified** property, not a free one,
and the architecture names the mechanism that verifies it:
- **A digest of the derived tier.** Each partition's derived tier carries a rolling content
digest (a Merkle-style hash over union nodes/edges/index entries) maintained alongside the
incremental updates.
- **A background consistency-checker** periodically recomputes the digest over a *sampled* (or,
on a slow cadence, full) fold of canonical state and compares. A mismatch localises the drift
to a partition/region and triggers a **scoped recompute** of just that region — cheap relative
to a global rebuild, and self-healing.
- **So I-2 holds *eventually and verifiably*:** the incremental engine is the fast path, the
checker is the guarantee, and divergence is detected and repaired rather than silently
accumulating. The exact sampling rate / digest granularity is an implementation spike (§12).
### 8.8 Cache freshness & invalidation
Replication-projection caches remote shard content; cache invalidation is the actual hard part
and was missing from the first draft (review C-2). The protocol is **per-binding, driven by the
capability profile**, with one rule: **freshness is always represented, never assumed** — every
cached page's provenance envelope carries `(observed-at, source-rev-if-known, staleness-state)`,
so a consumer can always tell live from stale.
**Three invalidation modes, chosen by capability, not hard-coded:**
| Mode | When | Mechanism |
|------|------|-----------|
| **Event-driven (push)** | shard has `notify` | a change event invalidates exactly the affected entries and enqueues a delta refresh (§8.7); the preferred mode |
| **Validator poll** | shard exposes ETag / Last-Modified / rev | conditional fetch (`If-None-Match`); cheap "still fresh?" checks without transferring bodies |
| **TTL** | shard offers neither | time-bounded staleness; the floor mode (Oddmuse-class shards) |
Most real bindings are **hybrid**: event-driven for invalidation + a long TTL as a safety net
for missed events + validator polls on read when an entry is past a soft age.
**Operational-envelope coupling.** The mode is constrained by axis-10: a **rate-limited** shard
(Notion) *must* favour event-driven + long TTL and *must not* poll per-read — the freshness
policy is capability-gated like everything else. A local file shard can watch the filesystem
(near-instant invalidation, effectively event-driven for free).
**Thundering-herd / coalescing.** Concurrent reads of the same stale entry trigger a **single
in-flight refresh** (single-flight); other readers await it or are served the stale-but-labelled
value per policy. Bulk invalidations (a shard-wide event) are **batched and rate-shaped** to the
shard's envelope rather than fired as N concurrent fetches.
**Staleness is a policy knob, not a correctness bug.** Whether a reader gets *stale-but-fast* or
*blocks-for-fresh* is a §10 preset (per space or per request); either way the envelope tells the
truth about what was served. This is union-without-erasure applied to time.
---
## 9. Cross-cut — Authorization (L5)
Fully specified in **`ArchitectureBlueprint.md`** (the access & history sub-blueprint);
summarised here for completeness:
- **One core, a ladder of modes** L0 (open/c2, zero deps) → L1 (attributed) → L2
(authenticated) → L3 (role/group) → L4 (multi-tenant enterprise). Climbing is configuration,
not re-architecture.
- **PEP** wraps every adapter op; **PDP** decides `(principal, action, target)` over actions
`read/write/patch/merge/administer`, layered on the adapter's capability profile (a shard
that can't write can't be written regardless of policy — L5 consults the L1 rail).
- **Authentication delegated** to a pluggable IdentityProvider (null provider = L0 default);
real identity from `user-engine` over `net-kingdom` IAM.
- **Fail open only at L0, fail closed at L2+.** Authorization is pure/offline once a Principal
is resolved. Provenance carries authz context so the union never leaks unreadable content
(the L5↔provenance-rail interaction).
### 9.1 Tenant isolation of the derived tier (review B-3)
Read-time authz filtering is necessary but **not sufficient** when the derived tier is
*persisted*: a single cross-tenant union/index cache guarded only by a filter on read is a
standing leak surface (one filtering bug exposes another tenant's content). So isolation is
**structural, not just procedural**:
- **The derived tier is partitioned per tenant / root entity.** A tenant maps to a root entity
(§4); its union graph, equivalence index, projections, and caches live in a **separate
partition** keyed by that tenant. There is no shared cross-tenant derived store to leak from.
- **No cross-tenant equivalence by default.** Blocking/LSH (§8.7) operates *within* a partition;
cross-tenant equivalence is an explicit, authorised, opt-in federation between roots, never an
accident of a shared index.
- **Read-time filtering remains, as defence-in-depth** — the provenance envelope's authz context
is still checked, so even within a partition a principal sees only what it may; partitioning
removes the *blast radius*, filtering removes the *fine-grained* leak.
- **This reconciles I-2 with L5:** recomputability (a persisted-but-disposable derived tier) is
preserved *per partition* — each tenant's derived tier is independently rebuildable from that
tenant's canonical state — so isolation costs nothing in the rebuild model. At L0/L1 (single
tenant) there is one partition and the machinery is invisible.
**Isolation invariant (add to §2 as I-13):** *derived state is partitioned by tenant; no
derived artifact spans tenants except through an explicit, authorised cross-root federation.*
---
## 10. The policy surface (mechanism over policy, made concrete)
I-7 only means something if the policy knobs are enumerated and kept *out* of core algorithms.
The configurable presets are:
- **Canonical-source policy** — chorus / designated-canonical / git-merge / overlay-only /
vote-to-merge (per space or per equivalence set).
- **Federation model** — the §8.3 taxonomy, per space, composable per shard.
- **Shard mode** — read-only / write-through / mirrored / projected / cached / canonical
(constrained by the capability profile).
- **Reconciliation cadence & conflict exposure** — push/poll/manual; show-conflicts vs
auto-merge-when-supported.
- **Conflict-resolution preset** — chorus / designated-canonical / git-merge / vote-to-merge /
overlay-only (the *resolution* policy over §8.6's core detection; per space or equivalence set).
- **Freshness / invalidation mode** — event-driven / validator-poll / TTL / hybrid, and
stale-but-fast vs block-for-fresh on read (§8.8; constrained by the operational envelope).
- **History compaction** — squash policy for low-value churn, gc/repack cadence, per-shard
offload (§8.1), always preserving recoverable endpoints.
- **Tenant partition mapping** — tenant ↔ root-entity, and any explicit cross-root federation
(§9.1, I-13).
- **Execution policy** — derive/execute off (default) / sandboxed / per-shard-allowed.
- **Authorization mode** — the L0L4 ladder.
- **Projection materialization** — lazy/eager; snapshot vs view-time; recording retention.
Core ships sane defaults (L0 open; fork+journal; lazy replication-projection; event-driven+TTL
freshness; overlay-before-mutation; execution off; one tenant = one root) and never hard-codes
any of the above. (**Preset bundles** that package coherent knob-sets per persona are tracked
as O-8, §12 — flexibility without bundles is operator burden.)
---
## 11. Concrete module structure (bridge to implementation)
A proposed package layout for `src/shard_wiki/`, mapping 1:1 to the layers so the dependency
rule (downward only; the derived tier is incrementally maintained, rebuild = fallback) is
enforceable by import lint:
```
src/shard_wiki/
model/ # L2 top waist: Page, Identity, Placement, ProvenanceEnvelope,
# Span, the page-shape types; capability-spectrum value types
adapters/ # L1 bottom waist: AdapterContract (versioned iface), CapabilityProfile,
# attachment-mode binding; concrete adapters:
git/ folder/ gitea/ obsidian/ webdav/ notion/ … # each: profile + verbs
coordination/ # L3: DecisionLog (append-only, git-backed, per-space append authority/
# lease), OverlayEngine (draft→patch→MR), reconcile
# (current coordination state = a derived fold → lives in union/)
federation/ # L3: FederationModel strategies (fork_journal, vcs_ping,
# graph_join, feed, activitypub, engine_mirror)
union/ # L4 (derived): IdentityResolver, EquivalenceGraph, UnionGraph,
# Transclusion (reference-not-copy)
projection/ # L4 (derived): ReplicationProjection, DerivationProjection,
# ViewRegistry (moldable), QueryIndex (delegate|derive)
authz/ # L5 cross-cut: PDP, PEP, IdentityProvider iface, NullProvider
provenance/ # cross-cut LEAF: ProvenanceEnvelope type + ⊕ (effective) only — pure data
policy/ # cross-cut LEAF: the §10 policy surface (presets + a resolve() read by
# coordination/federation/projection/authz); owns NO mechanism
api/ # L6: orchestrator API (server-side union for agents/CLI)
```
**The cross-cutting rails are leaves, not god-modules (review D-4).** `provenance/` and
`policy/` are imported widely, so they are the highest coupling risk; the discipline that caps
it is: **they may import *nothing* in the tree and contain *only* stable data types + pure
functions** (the envelope and its ``; the policy presets and a `resolve(question) → choice`).
Mechanism never lives in a rail — `policy/` says *what* the preset is, `coordination/`/
`projection/` decide *how* to honour it. A change to a rail is then a change to a small, stable,
dependency-free leaf, not a ripple through every layer. Capability-spectrum value types live in
`model/` (also leaf-like) for the same reason.
Hard import rules (enforced by import lint):
- `union/` and `projection/` may import `model/`, `adapters/`, `coordination/`, `policy/`,
`provenance/` — but **nothing may import them** (they are the disposable derived tier).
- `model/`, `adapters/`, `provenance/`, `policy/` import nothing else in the tree (the waists
and rails stay thin); `provenance/` and `policy/` import nothing at all.
- `coordination/` and `federation/` may import the waists + rails, never the derived tier.
---
## 12. Known scaling risks & open problems
Tracked honestly rather than pretend-solved (review disposition F). Each has a **chosen
direction** and a **revisit trigger** — the thing that, if observed, forces a redesign.
| # | Risk / open problem | Chosen direction | Revisit trigger |
|---|---------------------|------------------|-----------------|
| O-1 | **Equivalence blocking misses true matches** (LSH false negatives, §8.7) | accept a small miss rate; curator bindings are the escape hatch | measured recall below an agreed threshold on real corpora |
| O-2 | **Convergence bound for high-write CRDT shards under partition** (§8.6) | causal via journal + CRDT-native merge at the shard; no global bound promised | user-visible divergence that outlives a partition |
| O-3 | **Per-equivalence-set divergence tracking** (§8.6) | start with base-rev comparison; add vector clocks only if needed | 3-way concurrent divergence that base-rev mis-orders |
| O-4 | **Persisted derived-tier cost ceiling** (§8.7/§9.1) | per-tenant partition, incremental-maintained, rebuild is fallback | a tenant whose incremental cost still exceeds budget |
| O-5 | **Axis-interaction completeness** (§6.5) | the named interaction table is the contract; extend deliberately | a real adapter needing an interaction not in the table |
| O-6 | **Span-address portability across projection** (§7.2) | shard-scoped native-id wrapping now; tumbler later | cross-shard transclusion that native ids can't satisfy |
| O-7 | **Squash-compaction vs. perfect auditability** (§8.1) | compact the *path*, preserve reachable states; configurable | a compliance need for every intermediate keystroke |
| O-8 | **Policy-knob proliferation → operator burden** (§10) | ship named **preset bundles** ("personal vault" / "team wiki" / "enterprise federation") over the policy surface | operators mis-configuring interacting knobs |
| O-9 | **Shard sharing across roots vs tenant partition** (§9.1, I-13) | shard exclusive to one root by default; explicit shared-read binding otherwise (avoids double-caching a rate-limited shard) | a shard legitimately needed live in two tenants |
| O-10 | **Span-level authz under transclusion** (aggregation/inference leak; ⊕ across boundaries, §7.3/§9) | a transcluded span inherits the **stricter** of source & host authz; provenance ⊕ composes the source-page envelope under the host | a real cross-authz transclusion |
| O-11 | **Union under shard unavailability** (§8.8 covers stale, not down) | **partial union** + per-shard "unavailable" provenance + last-known-projection where policy allows | an SLA need on partial reads |
| O-12 | **Per-space append-log throughput ceiling** (§8.1 append authority) | single writer per space scales across spaces; per-space log sharding if needed | a single space exceeding one writer's append rate |
These are the spec-writing inputs for `SHARD-WP-0002`; none blocks the architecture, each
scopes an implementation spike.
---
## 13. Canonical data flows (the architecture exercised)
**A. Attach a shard.** Adapter binds (chosen attachment mode) → probes/declares a capability
profile → core registers the shard under a root entity → if not git-native, the coordination
journal is seeded (begin-now/mirror/import per axis 5). No union rebuild yet (lazy).
**B. Read a page through the union.** Consumer asks the union for an identity → Identity
resolver maps it to placements across shards → equivalence yields chorus or canonical →
replication-projection lazily fetches from each shard (cache + freshness) → page returned
wrapped in its provenance envelope → L5 filters anything the principal can't see at source.
**C. Edit a read-only / limited shard.** Write request → L5 PDP allows → capability profile
says < write-through → OverlayEngine records a draft → renders a patch/MR in the shard's native
syntax (lossless) or Markdown (lossy-with-report) → on explicit apply, commit to the journal
and (if the profile permits) propagate; otherwise the overlay stands as the local truth, fully
attributed.
**D. Attach a computational notebook.** Adapter declares profile (attachment=file-store,
opacity=mixed, computational=captured-output). Core attaches the `.ipynb` **source** as
canonical; presents cells + embedded outputs as **derivation-projection snapshots** marked
"run N, env unguaranteed"; offers a static render via the view registry; re-execution stays
gated off. History uses paired-text/nbdime per axis 5.
---
## 14. Key tradeoffs & decisions
Decided:
- **Capability spectra over a verb checklist** — richer contract for precise, uniform
degradation; tamed by an orthogonal core + implied positions + a named interaction table
(§6.5). (Decided.)
- **Three states; derived = f(canonical)** — sharded + coordination canonical, derived
disposable (§1). (Decided; supersedes the earlier "edges vs middle" framing.)
- **Event-sourced coordination, one append authority per space** — coordination-canonical state
is an append-only **decision log** in the git journal; current state is a derived fold; a
per-space append lease gives a totally-ordered log and read-your-writes across orchestrator
instances (§8.1). (Decided — resolves the single-vs-multi-writer keystone.)
- **Profiles are verified, not asserted** — a versioned **conformance suite** gates adapter
admission; capability-as-data acts on verified data (§6.6). (Decided.)
- **I-2 is eventually-verified** — incremental maintenance is the fast path; a digest +
background consistency-checker detects and self-heals drift (§8.7). (Decided.)
- **Incremental-first, rebuild-as-fallback** — the derived tier is continuously maintained from
change events; full rebuild is rare and need not be cheap (§8.7). (Decided — resolves the
earlier "union graph persistence" open item: **persisted, per-tenant, incrementally
maintained, rebuildable**, §9.1.)
- **Identity ≠ fingerprint** — page identity is a stable handle; fingerprints are for
equivalence (§7.2). (Decided.)
- **Consistency = read-your-writes (journal) + causal (derived) + eventual/freshness-labelled
(shards)**; conflict detection/representation is core, resolution is policy (§8.6). (Decided.)
- **Address scheme** — shard-scoped native-id wrapping now; portable tumbler later (§7.2, O-6).
(Decided.)
- **Default federation = fork+journal over Git**; other models opt-in (§8.3). (Decided.)
- **Execution off by default** — recognise+project always; execute only when gated (§8.5). (Decided.)
- **Derived tier is tenant-partitioned** (I-13, §9.1). (Decided.)
Still open (carried to §12 / policy):
1. **L1 "attributed-but-open" mode** — ship it or jump L0→L2? (Carried from ArchitectureBlueprint.)
2. **Per-page ACL default** — off (per-shard/namespace) confirmed; revisit only if demand appears.
3. The implementation spikes in **§12** (O-1…O-7).
---
## 15. What this architecture is *not*
- Not a wiki engine, UI, or rendering pipeline (those are consumers at L6).
- Not a canonical-source-of-truth — shards keep sovereignty; the middle is derived.
- Not a generic file-sync daemon — synchronisation is wiki-page-semantic.
- Not an execution platform — computation is recognised and projected, not hosted.
- Not a universal ontology — no single schema is imposed on all shards.
- Not an authentication/identity store — that is delegated (authorization is owned).
---
## 16. Traceability
- **INTENT** — every invariant in §2 (I-1…I-13) cites an INTENT principle or boundary; no
invariant contradicts the Stability Note.
- **Review & hardening** — this revision folds in
`history/260615-core-architecture-blueprint-review.md` via **`SHARD-WP-0005`**: A-1→§1/§3/§4
(three-state re-frame), B-1→§7.2 (identity vs equivalence), B-2→§8.6 (consistency/conflict),
B-3→§9.1+I-13 (tenant isolation), C-1/C-2→§8.7/§8.8 (incremental + indexed + invalidation),
C-3→§8.1 (history scaling), D-1→§6.5 (orthogonal core), D-2→§7.3 (layered provenance),
D-3→§8.4 (common-case projection), D-4→§11 (policy module + rail discipline); open items→§12.
- **Round-2 review & hardening II** — folds in
`history/260615-core-architecture-blueprint-review-2.md` via **`SHARD-WP-0006`**:
A-1…A-4→§3/§4/§10/§11 (overview reconciled to the body), B-1+B-3→§8.1 (event-sourced
coordination + per-space append authority), B-2→§6.6 (adapter conformance suite),
B-4→§8.7 (incremental retraction/propagation + I-2 digest/checker); C-1…C-4→§12 (O-8…O-11).
- **Research** — §6 (spectra) ← `260614-shard-spectrum-synthesis` v3; §8.3 (federation
taxonomy) ← v3 §2.5; §8.4§8.5 (two-axis projection, view registry, computational scope) ←
`260614-computational-page-model-synthesis`; §7 page shapes ← the engine + modern-tool +
computational dives; §1 thesis ← the files-canonical/index-derived through-line across
Logseq/ikiwiki/GT/Pharo/Jupyter.
- **Use cases** — the architecture is sized to UC-01UC-84: federation/coordination (UC-0107,
2633, 56, 7172, 79) → §8; attachment/adapter (UC-3443, 50, 53, 57, 6062, 6466, 6870,
7682) → §6; page model & fidelity (UC-34, 39, 42, 55, 5859, 67, 73, 80, 8384) → §7/§8.5;
addressing/identity/query (UC-32, 4449, 5152, 54, 63, 74) → §7.2/§8.4; provenance &
metadata (UC-2425, 75) → the provenance rail; collaboration & discovery (UC-0823) → L6
consumers over the union.
- **Workplans** — §6§8 are the design target of `SHARD-WP-0002` (T11T18); §9 is owned by
`ArchitectureBlueprint.md`; §1 (yawex-derived resolution/overlay) aligns with
`SHARD-WP-0001`.
---
## 17. Stability note
This document defines shard-wiki's **internal** architecture; it may evolve as the spec
workplans land. But the **thesis (§1)**, the **invariants (§2)**, and the **dual narrow waist
(§1, §6, §7)** are load-bearing — changing any of them is an architectural change in the sense
of INTENT's Stability Note and should be rare and deliberate.

View File

@@ -0,0 +1,220 @@
# FederationArchitecture
Status: **draft for review** · Date: 2026-06-15 · Deliverable of **SHARD-WP-0002** (T1T10, T17)
The federation **design decisions** for shard-wiki: *what the union does*. It records, per
topic, a decision with rationale, tradeoffs, and a **Decided / Deferred / Open** footer. It
**references** `spec/CoreArchitectureBlueprint.md` (the whole-system architecture) and
`spec/FederationRequirements.md` (yawex-derived ADRs) rather than restating them; the adapter
contract (*what a backend must expose*) is the companion deliverable in
`spec/TechnicalSpecificationDocument.md` §A (T11T16, T18). UC references → `UseCaseCatalog.md`.
Cross-cutting invariants assumed throughout (blueprint §2): orchestrator-not-engine (I-1),
three-state canonical/derived (I-2), capability-as-data (I-3), union-without-erasure (I-4),
overlay-before-mutation (I-5), git-addressable coordination (I-6), mechanism-over-policy (I-7),
graceful degradation (I-8), identity≠placement (I-9), history-as-floor (I-10), authz-in-core
(I-11), tenant-partitioned derived state (I-13).
---
## T1 — Orchestrator positioning & boundaries
**Decision.** shard-wiki is an **adapter-layer orchestrator** in a **star** shape (many
sovereign shards ↔ one information space), **not** a homogeneous federation network. It
**compares, does not equate** (Federated Wiki homogeneous JSON sites, ikiwiki homogeneous git
wikis, ActivityPub activity streams are *models it can speak*, §T17 — not shapes it imposes).
Core owns: the union, the coordination journal, projection, overlay, resolution, authorization.
Adapters own backend specifics; UI and editorial policy live outside core. (Blueprint §1, §5;
UC-02, UC-26.)
*Decided:* star orchestrator; compare-not-equate; core/adapter/UI/policy boundaries.
*Deferred:* the reference UI (L6) is out of scope here. *Open:* none.
## T2 — Remix primitives: reference / projection / overlay / import-fork
**Decision.** Four primitives, escalating in commitment; **overlay-before-mutation is the
default write path** (I-5), fork is *one federation model* (§T17) not the default for editing:
| Primitive | Trigger | Writes remote? | Coordination-canonical? |
|-----------|---------|----------------|--------------------------|
| **Reference** | link only | no | no (a link in content) |
| **Projection** | read remote page | no (cache) | no (derived) |
| **Overlay** | edit a sub-write-through shard | not until explicit apply | **yes** (decision log) |
| **Import / fork** | copy into a writable shard / fork-with-provenance | source unchanged | **yes** (fork event) |
(Blueprint §8.2; FederationRequirements ADR-05; UC-04, UC-26, UC-29.)
*Decided:* the four primitives + overlay-default. *Deferred:* fork-attribution portability
format. *Open:* whether "import" and "fork" are one primitive with a flag or two (impl spike).
## T3 — Equivalent-page identity & multi-version presentation
**Decision.** Separate **identity** (stable handle), **placement** (N paths/shards), and
**equivalence** (content-fingerprint / span-set overlap *across* identities), per
FederationRequirements ADR-01/ADR-02 (I-9). Equivalence signals: normalized title/path, alias
table (coordination-canonical), link-graph overlap, fingerprint, **curator binding**. Default
presentation = **chorus-of-voices** (all equivalent versions, attributed); **designated-
canonical** is a policy preset (§T9). Divergence is data in the provenance envelope (I-4),
detected as core mechanism (blueprint §8.6). (UC-07, UC-27; UC-46.)
*Decided:* identity/placement/equivalence split; chorus default. *Deferred:* fingerprint
algorithm + blocking params (blueprint O-1). *Open:* curator-binding UX (L6).
## T4 — History, attribution & the coordination journal
**Decision.** Each information space has a **git-addressable, event-sourced coordination
journal** (blueprint §8.1, I-6/I-10): coordination-canonical decisions (overlay/binding/alias/
merge/fork) are append-only events; current state is a derived fold. **One append authority per
space** gives a total order (read-your-writes across instances). Per-shard native history is
**adopted** (git-native), **supplemented** (DB/CRDT internal → journal begins-now/mirrors), or
**imported** (open file history backfilled) per the history-portability axis (TSD §A T13).
Attribution is portable on the event (UC-29). (UC-29, UC-33.)
*Decided:* event-sourced journal + per-space append authority + adopt/supplement/import.
*Deferred:* per-space log sharding for extreme write rates (blueprint O-12). *Open:* none.
## T5 — Union composition layer
**Decision.** The **server-side orchestrator union is primary** — the derived tier (union
graph, indexes, projections) is composed in core for agents/CLI/non-browser consumers
(blueprint §8.4), **incrementally maintained** (not recomputed, §8.7), **tenant-partitioned**
(I-13). **Client-side composition** (Federated-Wiki-style browser pull) is a supported
*consumer pattern over the same union*, not the only path. Freshness/caching per §8.8. (UC-05,
UC-27, UC-03, UC-31.)
*Decided:* server-primary, client-optional; incremental, partitioned. *Deferred:* client
SDK shape. *Open:* persisted-union digest granularity (blueprint O-4, B-4 checker).
## T6 — Change notification & subscription transports
**Decision.** Change-notification is an **optional adapter capability** (`notify`), and it is
the **primary driver of incremental maintenance** (blueprint §8.7/§8.8). Transports, per
profile: git hook / ikiwiki-style ping, ActivityPub Create/Update, webhook, WebDAV/HTTP ETag
or `Last-Modified` poll, plain polling fallback. A notification → invalidate affected entries
+ enqueue delta refresh + a RecentChanges union entry (FederationRequirements ADR-03; UC-31,
UC-17). Rate-limited shards favour event-driven + long TTL (operational-envelope axis).
*Decided:* notify = optional capability, drives incremental + RecentChanges; transport per
profile. *Deferred:* ActivityPub as content-bearing (vs notify-only) — start notify-only.
*Open:* fediverse-dependency posture for v1.
## T7 — Information-space lifecycle
**Decision.** Root entities have lifecycle states: **active → read-only-archived → retired
(detached)**, and **merged-into-successor**. **Carry-forward** is selective import from an
archived shard (UC-28, UC-30). **Space-fork / branch** (UC-33) = branch the coordination
journal + shard-config (a fork event, §T2/§T17 fork+journal model). Retirement **preserves
read-only union entries** by default (history-as-floor, I-10), never hard-deletes projections.
(UC-28, UC-30, UC-33.)
*Decided:* the lifecycle states + carry-forward + space-branch; retire-preserves. *Deferred:*
ephemeral "happenings" as a first-class mode. *Open:* GC policy for retired-space derived tier.
## T8 — Transclusion & projection depth
**Decision.** Transclusion is **one reference-not-copy primitive** over the addressable union
(blueprint §8.4), at three depths:
| Level | UC | Behaviour |
|-------|-----|-----------|
| Whole-page projection | UC-03 | lazy full page from a remote shard (replication-projection) |
| Block/span transclusion | UC-32 | inline embed by span address, with origin + freshness |
| Link reference | UC-08 | pointer only (ADR-06) |
Every transcluded artifact carries provenance + **staleness** (I-4, §8.8); live-transclusion
fragility (link rot / remote down) degrades to last-known + an `unavailable`/`stale` marker
(blueprint O-11). Span-level authz under transclusion = the stricter of source/host (O-10).
Reference-not-copy unifies Xanadu transclusion, ZigZag clone, embeds, synced blocks, Trilium
note-clone, literate named-chunk. (UC-03, UC-32.)
*Decided:* one primitive, three depths, provenance+staleness mandatory. *Deferred:* snapshot-
on-import vs live default (a policy). *Open:* span-authz composition (O-10), addressing (O-6).
## T9 — Consensus & reconciliation policy catalog
**Decision.** **Detection is core, resolution is policy** (I-7, blueprint §8.6). Core always
detects divergence and represents it as a coexisting (chorus) set; the **resolution preset** is
configurable per space / per equivalence set:
- **chorus-spread** (versions coexist; default) · **designated-canonical** (explicit write
target) · **git-merge** (where merge capability allows) · **vote-to-merge / editorial gate**
(optional, L6-assisted) · **overlay-only** (no destructive merge on read-only sources).
(FederationRequirements ADR-03/ADR-05; UC-07, UC-27.)
*Decided:* the preset catalog; detection-core/resolution-policy. *Deferred:* vote/editorial
gate mechanics (L6). *Open:* default preset per persona bundle (blueprint O-8).
## T10 — Federation-operations capability matrix
**Decision.** Each federation operation requires a minimum **verified capability profile**
(TSD §A T11/§6.6); below it, the op **degrades** along the named interaction axes (blueprint
§6.5) rather than failing. The matrix (consuming the T11 vocabulary):
| Federation op | Min capabilities | Degradation when absent |
|---------------|------------------|--------------------------|
| **read / project** | `read` | (floor — every shard) |
| **transclude span** | `read` + addressing≥span | whole-page projection only |
| **overlay** | `read` | always available (overlay is local) |
| **write-through** | `write` (+ `merge` for concurrent) | → overlay/patch target |
| **diff / 3-way merge** | `diff`,`merge` | → keep-both / chorus |
| **notify-driven freshness** | `notify` | → validator-poll → TTL (§8.8) |
| **native search delegate** | native-query≥index | → derived index over projection |
| **publish / mirror** | `publish` | → read-only / static projection |
| **fork+journal federation** | `read` + history (any) | → projection-only participant |
Every backend, down to the Oddmuse flat-file floor, is usable as **read-only / cache /
projection / backup / patch target** (I-8). This matrix and the T11 spectra are kept in sync
(it consumes that vocabulary). (UC-02UC-07.)
*Decided:* the op→capability matrix + degradation paths. *Deferred:* exhaustive per-op test
vectors → the conformance suite (§6.6). *Open:* axis-interaction completeness (blueprint O-5).
---
## T17 — Federation-model taxonomy (selectable / composable)
**Decision.** Federation is **plural and composable**, not one mechanism (blueprint §8.3). A
space selects a model (composing per shard); the default is **fork+journal over Git** (the home
case):
| Model | Anchor | Coordination shape | Capability floor | UCs |
|-------|--------|--------------------|------------------|-----|
| **Fork + journal** *(default)* | Federated Wiki | copy-with-provenance + per-page action journal (story = replay) | `read` + journal | UC-26, UC-71, UC-72 |
| **VCS-replication + ping** | ikiwiki | git clone/pull/push + change-ping | git-IS-store | UC-31, UC-33, UC-79 |
| **Query-time graph-join** | Wikibase `SERVICE` | join remote graphs at query, no copy | native-query≥graph | UC-74 |
| **Feed aggregation** | RSS/Atom | inbound feed → pages | `read` | UC-03 |
| **Activity streams** | ActivityPub | Create/Update events (notify or content) | `notify` | UC-31 |
| **Engine-mirror** | Wiki.js DB↔Git | engine mirrors its store to git | `publish`/mirror | UC-68, UC-69 |
The coordination journal (T4) is the fork+journal model's home; **git IS the journal** in
VCS-replication and engine-mirror (forge wikis make git the store *and* journal, resolving the
engine-mirror write-race). Models compose: e.g. fork+journal locally, query-join for a
typed-graph shard, feed-aggregation for an external source. (Mechanism over policy, I-7.)
*Decided:* the six models, selectable per space + composable per shard, default fork+journal.
*Deferred:* activity-streams content-bearing mode. *Open:* multi-model interaction edge cases
(when two models touch one shard).
---
## Consolidated decisions / deferred / open
- **Decided (architecture-settling):** star orchestrator (T1); overlay-default remix (T2);
identity/placement/equivalence + chorus (T3); event-sourced journal + append authority (T4);
server-primary incremental union (T5); notify-driven freshness (T6); space lifecycle (T7);
reference-not-copy transclusion (T8); detection-core/resolution-policy preset catalog (T9);
op→capability matrix (T10); selectable/composable federation models (T17).
- **Deferred (to impl / L6 / persona bundles):** reference UI; fork-attribution format; client
SDK; vote/editorial mechanics; ephemeral spaces; ActivityPub content-bearing; default-preset
bundles (O-8).
- **Open (tracked in blueprint §12):** O-1 equivalence blocking, O-4 union digest, O-5 axis
interactions, O-6 addressing, O-8 presets, O-10 span-authz, O-11 unavailability, O-12 log
throughput.
## Acceptance (SHARD-WP-0002 federation half)
T1T10 + T17 each have a decision record with a Decided/Deferred/Open footer; all honour INTENT;
UC-26UC-33, UC-56, UC-71UC-72, UC-74, UC-79 trace here; the adapter-contract companion is
`spec/TechnicalSpecificationDocument.md` §A (T11T16, T18). Conflicts with SHARD-WP-0001 are
none (FederationRequirements ADRs are consumed, not duplicated).

View File

@@ -0,0 +1,214 @@
# FederationRequirements — yawex-derived design notes
Status: **draft for review** · Date: 2026-06-15 · Deliverable of **SHARD-WP-0001**
Concrete, ADR-ready design decisions for the union/federation layer, distilled from the yawex
prior-art review (`research/260608-yawex-prior-art/findings.md`) and made **consistent with**
`spec/CoreArchitectureBlueprint.md` (the whole-system architecture) and `INTENT.md`. yawex is
*inspiration and a case checklist*, never an interface to inherit (decision 2026-06-08).
These are **requirements** (what the union must do); `spec/FederationArchitecture.md`
(SHARD-WP-0002) is the architecture that realises them. UC references are to
`spec/UseCaseCatalog.md`.
**Resolved precondition.** The yawex "minimal access model in core" thread is **settled**:
authorization-in-core / authentication-delegated, the L0L4 ladder — INTENT amended,
`ArchitectureBlueprint.md` ratified. Auth is therefore *not* re-litigated here; these ADRs
assume it (the resolver and overlay paths run behind the L5 PEP/PDP).
---
## ADR-01 — Cross-shard page resolution
**Status:** accepted · **CoreArchitectureBlueprint:** §8.4 (union), §7.2 (identity)
**Context.** yawex's most-tested component was `PageLookUp`, a context-relative resolver with
ordered states `LOCAL, CLIMB, DROP, GLOBAL, REMOTE, SWITCH, JUMP, VIRTUAL, FAILED`. Decision
(2026-06-08): **inspiration only** — use the states as a *checklist of cases*, design the
federation resolver fresh.
**Decision.** Resolution is a pure function over the derived union:
`resolve(name, from_context, policy) → ResolutionResult`, evaluated in a defined order and
keyed on **page identity** (the stable handle), not path (I-9):
1. **Same-namespace, same-shard** exact identity match — *(yawex LOCAL)*.
2. **Namespace walk** — climb/descend ancestor namespaces of `from_context`*(CLIMB/DROP)*.
3. **Union lookup** — match identity / alias-table entry across all attached shards — *(GLOBAL)*.
4. **Equivalence set** — if several shards hold an equivalent page, return the **chorus set**;
the *canonical-source* policy preset (chorus / designated-canonical) decides presentation —
never silently pick one (union without erasure, I-4). *(implicit in yawex; explicit here.)*
5. **Projection / virtual** — a page whose content lives in a remote shard (lazy replication,
*REMOTE*) or is computed/query-defined (derivation, *VIRTUAL*).
6. **Explicit address** — a shard- or space-qualified reference resolves directly — *(SWITCH/JUMP)*.
7. **Not found****red-link**: a resolvable *target* with no page yet, createable (UC-23) —
*(FAILED)*.
`ResolutionResult` carries the resolved identity (or red-link), the placement(s), the
provenance envelope (ADR-04), and — on ambiguity — the full chorus set. Resolution is a
**derived-tier read** (union graph + equivalence index, §8.4); it is incremental-maintained and
respects the L5 authz filter (a principal never resolves to content it can't see).
**Consequences.** Deterministic, policy-driven, no privileged shard; every yawex state has a
home without inheriting its structure. **Open:** cross-space `JUMP` addressing syntax ties to
the portable-address scheme (CoreArchitectureBlueprint O-6).
---
## ADR-02 — Namespace / path model & shard roles
**Status:** accepted · **CoreArchitectureBlueprint:** §7.2 (identity≠placement), §4 (shard mode)
**Context.** yawex modelled **topics as directories** (a topic's gateway page shares the dir
name) with relative (`../`) / absolute (`/`) paths and normalization, and page classes
`local / global / virtual`.
**Decision.**
- The union exposes a **namespace tree**; a **path is a placement coordinate, not an identity**
(I-9). One identity may have **N placements** (paths/shards) → a DAG, no single canonical path.
- **Path grammar:** relative (`../`, `./`) resolved against `from_context`; absolute (`/`)
against the space root; **normalization** (collapse `.`/`..`; case & separator handling) is a
per-space **policy** knob (§10) — defaulting to case-preserving, `/`-separated.
- **yawex page classes → shard roles / modes:** `local → canonical`, `global → cross-cutting`
(a shard whose pages augment/overlay across namespaces), `virtual → projected/computed`. These
are the INTENT shard modes (read-only · write-through · mirrored · projected · cached ·
canonical), selected by policy, constrained by the capability profile.
- Cross-shard name collisions are resolved by ADR-01's order + equivalence, not by a global
flat namespace.
**Consequences.** The namespace is a navigable view over placements (a dimension of the union),
not a storage layout. **Open:** per-space normalization policy defaults (case-sensitivity,
unicode) — a policy preset, bundled per persona (O-8).
---
## ADR-03 — Union-level derived views: core vs adapter
**Status:** accepted · **CoreArchitectureBlueprint:** §8.4 (derived query index), §8.7
(incremental) · **resolves findings open-Q #3**
**Context.** yawex shipped BackLinks, RecentChanges, AllPages, SiteMap, full-text Search.
**Decision.** Each is a **derived-tier** view (rebuildable, incrementally maintained); classify
by where it is computed:
| View | Tier | Rationale |
|------|------|-----------|
| **BackLinks** | **core** | the link-graph over the union — squarely shard-wiki's concern (UC-18); the strongest core view |
| **RecentChanges** | **core** | merge the coordination journal (§8.1) with shard change events (notify/poll, §8.8) across the union (UC-17) |
| **AllPages / SiteMap** | **core** | an enumeration/projection of the union graph; cheap |
| **Search (full-text)** | **hybrid** | **delegate** to a shard's native search where the native-query axis allows; **else** build a derived index over projections (the Logseq DataScript-over-files pattern, UC-19/UC-63) |
All are computed in the derived tier and carry provenance; **presentation is L6 (UI)**, never
hard-coded in core (mechanism over policy, I-7).
**Consequences.** BackLinks/RecentChanges/AllPages/SiteMap are core deliverables; Search is a
capability-gated delegate-or-derive. **Open:** cross-shard search ranking/relevance is a policy
concern, not core mechanism.
---
## ADR-04 — Provenance & freshness model (concrete fields)
**Status:** accepted · **CoreArchitectureBlueprint:** §7.3 (layered provenance), §8.8 (freshness)
**Context.** yawex `Page::info` exposed modtime (and TODO'd last-editor / hits / edits). INTENT
mandates explicit provenance & freshness (I-4).
**Decision.** Concretise the **provenance envelope** (layered: page-level + span-level deltas):
```
ProvenanceEnvelope (page-level):
source_shard # which shard this came from
source_rev? # shard-native revision id, if the shard exposes one
observed_at # when shard-wiki last read it
liveness # static | captured-snapshot | live-over-files | view-time | irreducibly-live
staleness_state # live | fresh | stale | unavailable
authz_context # the L5 context under which it was read (no-leak, §9)
overlay_state # none | draft | patch-pending | applied
divergence[] # equivalence-set peers that differ (the chorus, ADR-01 step 4)
derivation_lineage? # for derived/derivation-projection content (source → this view)
Span-level: only the fields that differ from the page envelope (effective = page ⊕ delta).
```
`freshness = (observed_at, source_rev?, staleness_state)`; `unavailable` is the dead-shard
state (feeds ADR-03 RecentChanges and the union-under-unavailability open item O-11). This
envelope is the data behind union-without-erasure and the input to conflict display.
**Consequences.** Provenance is queryable, layered (cheap per-span), and drives views/conflict
UI. **Open / DROP:** hits/edits *analytics* are an L6/analytics concern, out of core (dropped
from the yawex feature set).
---
## ADR-05 — Overlay / lightweight-patch model
**Status:** accepted · **CoreArchitectureBlueprint:** §8.2 (overlay engine), §8.1 (decision
log), §8.6 (apply-under-drift)
**Context.** yawex had an append + threaded-comment workflow (edit a page without a full
rewrite). INTENT mandates overlay-before-mutation (I-5).
**Decision.** The overlay lifecycle for any shard below write-through capability:
1. **Draft** — an edit becomes a draft recorded as a **coordination-canonical event** in the
space's decision log (§8.1); the unapplied overlay is the **local truth**, fully attributed
(`overlay_state`, ADR-04).
2. **Patch / MR** — rendered in the shard's **native syntax (lossless)** or **Markdown
(lossy-with-fidelity-report)** per the translation axis.
3. **Apply** — only on explicit intent, only where profile + policy permit, with
**apply-under-drift** semantics (§8.6: fast-forward / three-way / refuse-and-re-present).
4. **Append / comment** = a **constrained overlay subtype** (purely additive, low-conflict) —
the direct generalisation of yawex's workflow; safe even on the most limited shards.
This makes read-only, rate-limited, and lossy shards **editable** without silent remote
mutation (graceful degradation, I-8; no silent mutation).
**Consequences.** One overlay mechanism spans drafts, patches, MRs, and comments; storage is the
decision log (no separate store). **Open:** whether threaded-comment threading is a first-class
overlay subtype or a generic structured-append — defer to implementation.
---
## ADR-06 — Markdown link semantics (wikilink + red-link)
**Status:** accepted · **CoreArchitectureBlueprint:** §7 (page model), ADR-01 (resolution) ·
**resolves findings open-Q #2**
**Context.** yawex had CamelCase WikiLinks, `[[free links]]`, red-`?` links for nonexistent
pages, `::` labels. INTENT mandates Markdown-first; TRANSFORM the *semantics*, drop the bespoke
syntax.
**Decision.**
- Adopt a **CommonMark wikilink extension**: `[[Target]]` / `[[Target|label]]`, resolved via
ADR-01 (by identity/name across the union).
- **Red-link** = a wikilink whose target resolves to FAILED (ADR-01 step 7): a valid,
*createable* target with no page yet (the soft-create affordance, UC-23).
- **CamelCase auto-linking is OFF by default** (a legacy affordance); opt-in **per space** for
migrating CamelCase wikis (UseModWiki/c2 lineage, UC-25).
- Links are stored **as text** (git-diffable; structure/links federate iff in-text).
- **Boundary (resolves open-Q #2):** the link **model + resolution is core**; the **visual
rendering** of wikilinks/red-links is a **reference-UI (L6)** concern. Core decides *what a
link resolves to and whether it's a red-link*; the UI decides how it looks.
**Consequences.** Markdown-first, backend-neutral, resolution-unified with ADR-01. **Open:**
cross-shard wikilink target disambiguation syntax ties to the portable-address scheme (O-6).
---
## Coverage & open questions
- **Findings open-Q #1** (per-page vs per-shard ACL) — answered by the settled access model:
per-shard / per-namespace default, per-page ACL opt-in at L4 (`ArchitectureBlueprint.md`).
- **Findings open-Q #2** (wikilink core vs UI) — ADR-06: model/resolution core, presentation UI.
- **Findings open-Q #3** (derived views core vs adapter) — ADR-03.
- Carried to CoreArchitectureBlueprint §12: cross-space addressing syntax (O-6), namespace
normalization presets (O-8), cross-shard search ranking (policy), union-under-unavailability
(O-11).
## Acceptance (SHARD-WP-0001)
Each task T1T6 has an ADR (T1→ADR-01, T2→ADR-02, T3→ADR-03, T4→ADR-04, T5→ADR-05, T6→ADR-06);
all honour INTENT (mechanism over policy, union without erasure, overlay before mutation,
capability-aware adapters) and are consistent with `CoreArchitectureBlueprint.md`. The
access-model thread is ratified (not deferred). The next implementation workplan (domain model /
adapter contract) can proceed without unresolved yawex-derived design gaps.

View File

@@ -6,10 +6,15 @@ Background on document types: InfoTechPrimers on coulomb.social.
| File | Status | Role |
|------|--------|------|
| `CoreArchitectureBlueprint.md` | draft for review | **Whole-system architecture** — layers, abstractions, load-bearing decisions (synthesised from all research) |
| `FederationArchitecture.md` | draft for review | federation design — *what the union does*: T1T10 decision records + the federation-model taxonomy (SHARD-WP-0002) |
| `WikiEngineCoreArchitecture.md` | draft for review | the native **headless, API-first wiki engine** — small page-store kernel + typed-extension framework, as a canonical-mode shard backend (SHARD-WP-0013) |
| `adr/` | living | Architecture Decision Records (ADR-0001: engine activation via feature-control) |
| `FederationRequirements.md` | draft for review | yawex-derived union/federation design notes — resolution, namespace, derived views, provenance, overlay, links (ADR-01…06; SHARD-WP-0001) |
| `ProductRequirementsDocument.md` | draft scaffold | What the product must deliver |
| `TechnicalSpecificationDocument.md` | draft scaffold | How the system is built |
| `UseCaseCatalog.md` | draft | 25 use cases promoted from c2 + yawex research |
| `ArchitectureBlueprint.md` | draft | Access, history, and identity architecture |
| `TechnicalSpecificationDocument.md` | draft + §A | How the system is built; **§A = the normative shard adapter contract** (T11T16, T18; SHARD-WP-0002) |
| `UseCaseCatalog.md` | draft | 84 use cases promoted from c2 + yawex + ~23-system research |
| `ArchitectureBlueprint.md` | draft | Access, history, and identity sub-blueprint (the L0L4 authorization ladder; referenced by CoreArchitectureBlueprint §9) |
Promote material from `research/` and reviewed items from `demand/` into spec
before treating it as implementation authority.

View File

@@ -33,7 +33,134 @@ per information space.
## 4. Architecture References
- `spec/ArchitectureBlueprint.md`access, history, identity delegation
- `spec/CoreArchitectureBlueprint.md`whole-system architecture (layers, the dual narrow
waist, the 15 capability spectra, projection, consistency); the normative source for §A.
- `spec/FederationArchitecture.md` — federation design (*what the union does*); §A is its
companion (*what a backend must expose*).
- `spec/FederationRequirements.md` — yawex-derived ADRs (resolution, page model, overlay).
- `spec/ArchitectureBlueprint.md` — access, history, identity delegation (L5).
---
# A. Shard Adapter Contract (normative)
Deliverable of **SHARD-WP-0002** (T11T16, T18): the versioned interface a backend implements
to participate as a shard, and the *normative* rules core relies on. It is the **bottom narrow
waist** (blueprint §6); the page model is the top waist (§7). This section is normative where it
says **MUST/SHOULD**; design rationale lives in the blueprint (cited, not restated).
## A.1 (T11) Capability model & versioned interface
- The contract is a **versioned interface** (`Foswiki::Store`/`Foswiki::Meta` is the proof a
swappable-backend-behind-a-stable-interface works). A binding declares the contract version
it implements; core checks compatibility at registration.
- **Operation verbs:** `read, write, diff, merge, lock, version, publish, notify,
transclude-source, translate-syntax, structured-payload, derive-projection, execute`. The
last two are **gated, OFF by default** (A.6/T18). Verb support is part of the profile and MUST
reconcile with the T10 federation-ops matrix.
- **Capability profile = a position on each capability spectrum**, not a boolean checklist. The
full set is the **fifteen spectra** (blueprint §6.1); operationally they reduce to a **small
orthogonal core** (substrate, write-granularity, content-opacity, operational-envelope,
access-grant, computational/liveness) with the rest **implied** via published rules that
**forbid impossible profiles** (blueprint §6.5). Degradation reads only the **named
axis-interaction subset** (blueprint §6.5 table) — that table *is* the degradation contract.
- A profile MUST express **absence** cleanly (the Oddmuse floor); core never guesses a missing
capability.
## A.2 (T11/§6.6) Conformance — profiles are verified, not asserted
- The contract ships a **versioned conformance suite**. A binding is **admissible only if it
passes**: the suite exercises each declared verb and spectrum position and checks observed
behaviour matches the claim (a `write` round-trips, `notify` actually fires, an opaque shard
refuses plaintext query, implied-position rules hold). A lying/buggy profile is **rejected at
registration**, not run in production (blueprint §6.6).
- Conformance makes capability-as-data (I-3) and the §6.5 degradation contract **sound**.
Mismatch is reported as a capability-by-capability diff; an adapter may register
*degraded-but-honest* (drop the unsupported claim).
## A.3 (T14) Adapter binding — attachment-mode taxonomy
A backend MAY offer several modes; attach mode is a **per-binding, capability-gated choice**
with one declared authoritative (blueprint §6.3):
- **file-store** (native vault/folder *or* an interchange/sync mirror) · **git-IS-store** (forge
wikis & ikiwiki — git is store *and* journal; the home case) · **in-engine-hosted** (XWiki
component, Obsidian/Logseq/Roam plugin, Trilium script) · **local-REST** (Joplin Data API,
Trilium ETAPI) · **external-API-only** (Notion) · **direct-DB** (MojoMojo schema→model) ·
**CRDT-replica** (Anytype/AFFiNE/AppFlowy) · **P2P / no-central-endpoint**.
- **Backend-swap tolerance:** identity/provenance MUST survive a substrate change (bind to
capabilities, not "it's files"). **Boundary:** an **image / live-memory blob is never an
attach target** (I-12) — participation is export→files only.
- (UC-38, UC-40, UC-43, UC-50, UC-53, UC-57, UC-60, UC-62, UC-64, UC-65, UC-76, UC-79, UC-81.)
## A.4 (T12) Page model — structured / computational payload
The backend-neutral, Markdown-first page model MUST carry, without lossy flattening, every
shape in blueprint §7.1: prose; typed/computed records (incl. effective-vs-own computed
metadata); typed-graph statements (Wikibase); inline-embedded objects (Quip/Notion);
non-Markdown assets (typed asset / opaque blob / content-type registry); and the **four
computational shapes** (one-source-many-projections UC-83; notebook-with-embedded-output UC-84;
program-as-page; live/temporal). All reduce to `(content|source, structure, provenance
envelope, [derivation rule])`. **Identity is a stable handle; placement is separate; equivalence
is fingerprint-based** (blueprint §7.2, FederationRequirements ADR-01/02, I-9). The **provenance
envelope is layered** (page + span deltas; effective = page ⊕ delta, §7.3). (UC-34, UC-39,
UC-55, UC-58, UC-54, UC-44, UC-66, UC-67, UC-73, UC-83, UC-84.)
## A.5 (T13) History portability — adopt / supplement / import
Per the history axis: **adopt** git-native history as-is; **supplement** non-portable internal
history (DB / Notion / Joplin / Trilium revisions, CRDT-log) — the journal begins-now / mirrors
/ snapshots; **import** open file history (RCS, PlainFile, MojoMojo DB-version-rows) backfilled
preserving author/timestamp. **Partial/truncated history MUST be reported honestly** ("history
begins at", UC-24), never implied complete. Embedded-output documents (notebooks) use
paired-text (Jupytext) / cell-aware merge (nbdime). The space's own coordination history is the
event-sourced decision log (blueprint §8.1). (UC-36, UC-41, UC-24.)
## A.6 (T15) Syntax translation & content fidelity
Translation is a spectrum: **native → lossless → lossy-with-fidelity-report**. Read native
markup (TML, XWiki syntax, HTML) into the page model; accept Markdown overlays back via
**lossless bidirectional translation** where possible (Foswiki WysiwygPlugin is the proof). For
non-round-tripping models (Notion blocks, Trilium HTML, CRDT, typed-graph, notebook JSON/MIME):
translate lossily but **make the fidelity loss visible** — a per-shard/per-page report of what
projects cleanly vs degrades, with non-mappable elements preserved as provenance/sidecar (I-4).
Add a **structured-re-evaluable-value** point to the opacity spectrum (Wolfram expression).
Where no acceptable translation exists, the shard is a **read-only/projection** participant
(I-8), never silently corrupted. (UC-42, UC-59, UC-03, UC-73.)
## A.7 (T16) Addressing, identity & navigation
- **Span addressing:** adopt native span IDs where minted (Roam `:block/uid`, Logseq `id::`,
Notion/CRDT UUID), shard-scoped so they survive projection and don't collide; else a
position address (path+range) or content-fingerprint address. Portable tumbler is the ideal
(blueprint O-6).
- **Transclusion** = one reference-not-copy primitive over the addressable union (FederationArch
T8). **Query/navigation:** delegate to a shard's native query where capable (Datalog, DB
query, XWQL, SPARQL), **else build a derived index over the projection** (Logseq pattern);
dimensional/query-defined views are derived-tier. (UC-4448, UC-51, UC-52, UC-54, UC-63,
UC-74.)
## A.8 (T18) Computational / executable content capability
**In scope as a page-model + projection concern; out of scope as an execution platform**
(blueprint §8.5). Core **recognises** computational types, attaches the **canonical source**,
and presents derived forms as **provenance- and liveness-marked projections**. `derive-
projection`/`execute` are **gated capabilities, OFF by default**, carrying a **trust/sandbox**
concern, **degrading to a captured snapshot / static render / recording**. One snapshot-
provenance record (run id, source rev, timestamp, environment "unguaranteed") serves notebooks,
renders, recordings. No INTENT amendment required. (UC-54, UC-55, UC-83, UC-84.)
## A.9 Conformance & module mapping
The contract maps to `src/shard_wiki/adapters/` (the bottom waist: `AdapterContract`,
`CapabilityProfile`, attachment-mode binding, the conformance suite) consuming `model/`
(page model + capability value types) and `provenance/` (blueprint §11). Each concrete adapter
ships its declared profile + a conformance pass.
*Decided:* A.1A.8 (versioned capability contract, verified conformance, binding taxonomy,
page model, history portability, translation, addressing, gated computational content).
*Deferred:* per-backend adapter specs (one per backend, later). *Open:* blueprint §12 O-items
(addressing O-6, axis interactions O-5, span-authz O-10).
## 5. Integration Boundaries
@@ -47,10 +174,13 @@ Package scaffold only (`__version__`, smoke tests). Domain model not yet coded.
## 7. Use Cases
`spec/UseCaseCatalog.md`25 use cases (UC-01UC-25) promoted from c2 wiki
origins and yawex prior-art research.
`spec/UseCaseCatalog.md` — 84 use cases (UC-01UC-84) from c2/yawex origins,
the wiki-engine + modern-tool + computational research, and the syntheses.
## 8. Next Specification Work
Outputs from `SHARD-WP-0001` tasks (page resolution, namespaces, derived views,
provenance, overlays, link semantics) will be incorporated here as they complete.
The design layer is complete: `SHARD-WP-0001` (→ `FederationRequirements.md`),
`SHARD-WP-0002` (→ `FederationArchitecture.md` + §A above), and the hardened
`CoreArchitectureBlueprint.md`. The next workplan is the **implementation** of the
domain model + adapter contract (starting from §A and blueprint §11's module layout),
likely with a first spike on the keystone (the event-sourced decision log + derived fold).

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,269 @@
# WikiEngineCoreArchitecture
Status: **draft for review** · Date: 2026-06-15 · Deliverable of **SHARD-WP-0013 T5**
The architecture of shard-wiki's **native reference wiki-engine**: a **headless, API-first**
engine — a **small core** plus a **stringent typed-extension framework** — that addresses the
whole use-case catalogue, mediates conflicting requirements into one integrated featureset, and
lets each shard **activate only what it needs**. Authoritative as of the ratified INTENT
amendment (2026-06-15, decision `84ffdb48`): the engine is **additive** and is shard-wiki's
**reference first-party shard backend (a canonical-mode shard)** — not a replacement for other
engines, not a UI.
Relation to other specs (referenced, not restated):
- `CoreArchitectureBlueprint.md` — the orchestrator/whole-system architecture. **The engine is
one shard behind §A; federation, union, projection, and cross-shard coordination are the
orchestrator's job, not the engine's.** That is what keeps the engine small.
- `TechnicalSpecificationDocument.md §A` — the shard adapter contract the engine implements.
- `FederationRequirements.md` — page resolution, overlay, link semantics (ADRs the engine reuses).
- `UseCaseCatalog.md` "Capability structure" layer (T2) — the core-vs-extension map + the
conflict-mediation map this document realizes.
- reuse surface (`capability.wiki.*`, plus consumed `feature-control` / `authorization`).
---
## 1. Thesis: a small page-store kernel; everything else is a typed extension
> **The engine is a page-store kernel with a typed-extension runtime. Every capability beyond
> the c2-minimum is a *typed extension* a shard activates only if it needs it — and a shard's
> externally-visible capability profile is *computed from its active extension set*.**
That single chain — **configuration (which extensions) → capability (what the shard can do) →
conformance (verified)** — is the whole design. It mirrors the orchestrator's discipline
(`CoreArchitectureBlueprint` §6.5: capability-as-data, verified, no per-backend code) and turns
"integrated whole, yet activate only what you need" from a slogan into a mechanism.
The engine stays small for a structural reason: it is **one shard**, not a federation layer.
Union, projection, equivalence, cross-shard overlay-orchestration, and the federation models all
live in shard-wiki's orchestrator (the blueprint). The engine implements `ShardAdapter` (§A) and
nothing above it. So "wiki engine" here means *a really good single canonical shard with a
typed-extension framework and a headless agent-first API* — not a re-implementation of shard-wiki.
---
## 2. Engine invariants
| # | Invariant | Why |
|---|-----------|-----|
| E-1 | **One shard, not a federation layer.** The engine implements `ShardAdapter` (§A); union/projection/federation are the orchestrator's. | Keeps the engine small; no duplication of the blueprint. |
| E-2 | **Small kernel.** The kernel is only: page store + history, the page model (reused), the extension runtime, the API. | Common case (a plain wiki) is trivial. |
| E-3 | **Everything else is a typed extension.** No feature beyond the c2-minimum is baked into the kernel. | Integrated-whole-yet-selective; testable boundary. |
| E-4 | **Per-shard activation.** A shard runs an *activation profile* (a set of extensions + config); unused features cost nothing. | "Activate only what you need." |
| E-5 | **Capability profile is derived from active extensions.** The §A profile the engine declares is computed from its activation profile, then conformance-verified. | One source of truth; honest, verified capabilities. |
| E-6 | **Headless & API-first.** The API is the only interface; no bundled UI/rendering (consumer concern, L6). | INTENT amendment; clean orchestrator/consumer split. |
| E-7 | **Agent-first ergonomics.** The API is typed, introspectable, batchable, low-round-trip. | INTENT: optimized for efficient agent/automation access. |
| E-8 | **Reuse over reinvent.** Page model, history/journal, activation, and authz are *consumed* (existing capabilities), not rebuilt. | Smallness; reuse-surface alignment. |
| E-9 | **Extensions are typed & verified.** An extension declares its types/hooks/deps; activation is rejected if types conflict or deps are unmet (impossible profiles forbidden). | Stringency; mirrors §6.5 + conformance. |
---
## 3. The kernel (four concepts)
The kernel is deliberately four things — nothing more is mandatory.
1. **Page** — the backend-neutral page model (`capability.wiki.page-model`, reused as-is):
stable identity ≠ placement, layered provenance, page shapes. The kernel does **not** redefine
it; extensions may *register additional shapes/types* (§4).
2. **Store + history** — a git-backed page store (the engine is the *git-IS-store* case from the
blueprint): a write is a commit; history is native and recoverable (E-3/I-10). Coordination
decisions reuse the event-sourced journal (`capability.wiki.coordination-journal`).
3. **Extension runtime** — the typed-extension registry, hook dispatcher, type checker, and
activation engine (§4). *This is the core innovation; it is the only “framework” in the kernel.*
4. **API** — the headless, typed, agent-first surface (§7). Kernel endpoints cover the c2-minimum
(page CRUD-as-history, links, history); extensions extend the surface through typed routes.
The **c2-minimum** a kernel-only shard delivers (no extensions): write a page, link pages
(`[[wikilink]]` + red-link), never lose an edit. That is a complete, useful headless wiki.
---
## 4. The typed-extension model (the framework)
An **Extension** is a typed unit declaring a contract the runtime enforces:
```
Extension:
id : reverse-domain id (e.g. ext.struct.typed-records)
provides : capability ids it realizes (reuse-surface; e.g. capability.wiki.page-model[typed])
types : page shapes / field schemas / content-types it introduces (typed, validated)
hooks : kernel lifecycle bindings it implements (see below)
api : typed routes it adds to the headless surface
depends_on : other extensions / consumed capabilities required
conflicts_with: extensions it cannot co-activate with
config : declared, schema-checked activation parameters
```
**Hooks (the kernel lifecycle the runtime dispatches):**
`on_resolve` (name→page), `on_read`, `on_write` (validate/transform a draft), `on_link`
(link/transclusion resolution), `on_history`, `on_query`, `on_render_request` (produce a derived
representation for a consumer), `on_profile` (contribute capability-spectrum positions, E-5).
Hooks are **typed** (typed inputs/outputs) and dispatched in a **declared, deterministic order**.
**Typing & composition (stringency):**
- At activation, the runtime builds the **dependency closure**, checks **type consistency** (no
two active extensions claim incompatible types for the same page shape/field; `conflicts_with`
honoured), and rejects an **impossible profile** — exactly the §6.5 implication-rule discipline,
applied to extensions. A rejected profile fails fast at boot, never silently.
- Composition is **deterministic**: hook order is declared; conflicts are resolved by explicit
precedence or rejection, never by accident.
- Extensions ship a **conformance check** (mirrors §6.6): an activated extension is exercised
against its declared types/hooks before the shard serves traffic — *typed contracts verified,
not trusted*.
**Per-shard activation (reuse, not reinvent):**
- A shard's **activation profile** = `{extension id → config}`. Activation/evaluation **reuses
`capability.feature-control.evaluate`** (helix_forge/feature-control) — shard-wiki does not
build a bespoke flagging system (T3 consumption).
- **E-5 in action:** the engine's `on_profile` hooks fold the active extensions into the §A
**capability profile** the shard advertises to the orchestrator (e.g. activate
`ext.struct.typed-records` → the `structure` spectrum rises and `structured-payload` is
declared). The profile is then conformance-verified (§A.2). *Configuration → capability →
conformance is one chain.*
---
## 5. Featureset map: core vs extensions, and conflict mediation
The engine realizes the T2 "Capability structure" layer (`UseCaseCatalog.md`). Mapping (the
*page/content-level* clusters; **X-FED and X-ATT are orchestrator concerns, not engine
extensions** — E-1):
| Engine kernel (always on) | T2 | reuse-surface |
|---------------------------|----|---------------|
| Page lifecycle, identity/placement, history, links, store | EC-1…EC-5 | `capability.wiki.page-model`, `…coordination-journal`, `…adapter-contract` |
| Built-in typed extension | T2 cluster | provides / consumes | default |
|--------------------------|-----------|---------------------|---------|
| `ext.overlay` | X-OVERLAY | `capability.wiki.overlay` | on (no-op locally) |
| `ext.authz` (L0→L4 tiers) | X-AUTHZ | consumes `capability.authorization.policy-evaluate` | L0 |
| `ext.views` (BackLinks/RecentChanges/…) | X-VIEW | `capability.wiki.derived-views` | BackLinks/RecentChanges on |
| `ext.struct` (typed/computed/graph) | X-STRUCT | `capability.wiki.page-model[typed]` | off |
| `ext.addr` (span addr / transclusion / query) | X-ADDR | `capability.wiki.page-model`+query | transclusion on |
| `ext.compute` (literate/notebook/program/live) | X-COMP | `capability.wiki.engine-typed-extensions` | off (gated, sandbox) |
| `ext.prov` (rich provenance/metadata) | X-PROV | `capability.wiki.page-model[provenance]` | base on |
| `ext.collab` (c2 social patterns) | X-COLLAB | (UI/convention; mostly consumer) | off |
**Conflict mediation (T2 map) realized by the framework** — every tension is a *mechanism*, not a
baked-in choice, so one featureset serves all:
| Tension | Realized by |
|---------|-------------|
| open vs governed | `ext.authz` tiers (additive); kernel history is the floor at L0 |
| lossless vs lossy | a `translate` hook + fidelity report (consumes the proposed `capability.content.translation-fidelity`, G2) |
| live vs snapshot | `ext.compute`/`ext.addr` mark liveness; degrade to snapshot (never imply live) |
| canonical vs chorus | detection in kernel; resolution is a policy preset (orchestrator) |
| integrated-whole vs only-what-you-need | **the activation profile** (E-4) + typed composition (§4) — the headline mediation |
| minimal vs feature-rich | small kernel (§3) + extensions; nothing beyond c2 is mandatory |
---
## 6. The engine as a canonical-mode shard
The engine exposes itself through an `EngineShardAdapter` implementing §A:
- **Substrate** git-IS-store; **history** git-native; **write** = commit; `current_rev` = sha
(apply-under-drift works out of the box). It is the **most capable shard** shard-wiki can
attach — it dogfoods the contract.
- Its **capability profile is computed from active extensions** (E-5) and **conformance-verified**
(§A.2) — so the orchestrator sees an honest profile, and federation ops degrade by the engine's
*actually-activated* capabilities.
- The orchestrator attaches it like any shard; **federation/union/projection are not in the
engine** (E-1). A standalone deployment is "the engine as the sole canonical shard"; a
federated deployment is "the engine as one shard among many." Same engine, no re-architecture.
This is the precise realization of the INTENT reconciliation: shard-wiki orchestrates; the engine
is the first-party shard it can attach.
---
## 7. Headless API surface & agent ergonomics (E-6/E-7)
API-first means the typed API is the product; there is no UI. Agent-first means it is designed
for cheap, deterministic machine consumption:
- **Typed resource API** over pages, links, history, spans — content-negotiated (raw Markdown,
the structured page model, or an extension-rendered representation via `on_render_request`).
- **Capability/extension introspection** — an endpoint returns the shard's **active extensions,
their types, and the derived §A capability profile**, so an agent can discover *what this shard
can do* before acting (no trial-and-error). This is the agent-facing twin of E-5.
- **Batch & query** — multi-page reads, link-graph and RecentChanges queries (via `ext.views`),
and `on_query` delegation — minimizing round-trips.
- **Write via overlay** — edits go through the overlay path (FederationRequirements ADR-05), so
agent writes are safe (draft → apply-under-drift) and attributable.
- **Deterministic & provenance-carrying** — every response carries the provenance envelope;
identical inputs yield identical outputs (no hidden state) — friendly to caching agents.
---
## 8. Implementation sketch (module layout)
The engine lives under the shard-wiki package as a backend (it sits at L0/L1 — a shard behind the
adapter; nothing in the orchestrator depends *up* on it):
```
src/shard_wiki/engine/
kernel.py # page store + history (git-IS-store), lifecycle; reuses model/, provenance/, coordination/
extension.py # Extension contract, registry, typed hook dispatcher, type checker
activation.py # activation profile; reuses capability.feature-control.evaluate
profile.py # derive the §A CapabilityProfile from active extensions (E-5) + conformance
api.py # headless, typed, agent-first surface (+ extension introspection)
adapter.py # EngineShardAdapter implements adapters/ ShardAdapter (canonical-mode shard)
extensions/ # built-ins: overlay/ authz/ views/ struct/ addr/ compute/ prov/ collab/
```
Dependency rule: `engine/` consumes `model/`, `provenance/`, `coordination/`, `adapters/`
(contract), `policy/`; it is consumed *only* via its `EngineShardAdapter` (the orchestrator
attaches it as a shard). No orchestrator-tier (`union/`, `projection/`) import.
---
## 9. Reuse (what the engine consumes vs registers)
- **Consumes:** `capability.feature-control.evaluate` (activation), `capability.authorization.
policy-evaluate` (`ext.authz`), the proposed `capability.content.translation-fidelity` (G2,
lossy translation), and shard-wiki's own `capability.wiki.{page-model, coordination-journal,
adapter-contract, overlay, derived-views}`.
- **Registers / realizes:** `capability.wiki.engine-typed-extensions` (this document is its
Discovery evidence — D2→D3 on ratification). The cross-cutting **typed-extension framework**
pattern is proposed back to the reuse surface as **G1** (`capability.platform.typed-extension-
framework`); this engine is its first instance.
---
## 10. Traceability
- **INTENT** — realizes the 2026-06-15 amendment (decision `84ffdb48`): headless, API-first,
additive native engine = canonical-mode shard backend; honours all engine invariants and the
orchestrator boundary (E-1).
- **Use cases** — the kernel/extension split *is* the T2 "Capability structure" layer
(`UseCaseCatalog.md`); every UC is either kernel (EC-1…EC-5) or a named extension; conflicts
use the T2 mediation map (§5). The engine must ultimately cover UC-01UC-84 (per-shard subsets).
- **Architecture** — consistent with `CoreArchitectureBlueprint` (engine = canonical-mode shard,
§6 contract, §7 page model, §8.1 journal) and `TechnicalSpecificationDocument §A` (the contract
it implements). `FederationRequirements` ADR-05/06 supply overlay + link semantics.
- **Reuse surface** — §9; G1/G2 proposals from SHARD-WP-0013 T3.
## 11. Decisions / deferred / open
**Decided:** small page-store kernel + typed-extension runtime (E-2/E-3); engine is one shard,
not a federation layer (E-1); capability profile derived from active extensions (E-5); headless,
API-first, agent-first (E-6/E-7); activation reuses `feature-control` (E-8); extensions are
typed + conformance-verified (E-9).
**Deferred:** the concrete extension SDK/ABI and hook signatures; the API protocol (REST/GraphQL/
MCP) — agent-first introspection is required, the wire format is an implementation spike; the
built-in extensions' internal designs (each is a later workplan).
**Open (tracked):** does `ext.compute` ever execute in-process or strictly delegate/snapshot
(ties blueprint §8.5 + trust/sandbox); is the typed-extension framework promoted to the
reuse-surface platform capability (G1) and then *consumed* here rather than engine-owned;
introspection granularity vs. leaking internal structure to agents.
## 12. Stability note
The **thesis (§1)** and **invariants (§2)** — especially *engine-is-one-shard* (E-1),
*small-kernel/everything-else-typed-extension* (E-2/E-3), and *capability-profile-derived-from-
extensions* (E-5) — are load-bearing. Changing them (e.g. moving federation into the engine, or
baking a feature into the kernel) is an architectural change in the sense of INTENT's Stability
Note and should be rare and deliberate. The headless/API-first posture is fixed by the ratified
INTENT amendment.

Some files were not shown because too many files have changed in this diff Show More