generated from coulomb/repo-seed
spec: CoreArchitectureBlueprint — whole-system architecture from research
The optimal architecture synthesised from INTENT + the full research arc: - Thesis: canonical at the edges, derived in the middle (orchestrator not engine) - Dual narrow waist: adapter contract (15 capability spectra) + page model - 6 layers + provenance/capability rails; L4 union/projection is a rebuildable cache - Federation-model taxonomy (plural/composable); two-axis projection model; moldable view registry; identity != placement; computational content in scope as page-model+projection, out as execution platform - Concrete src/ module layout with downward-only dependency rule - Canonical data flows; policy surface; tradeoffs; full traceability to INTENT/UCs References ArchitectureBlueprint.md as the L5 authorization sub-blueprint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2
SCOPE.md
2
SCOPE.md
@@ -21,7 +21,7 @@ Learnings update both SCOPE and INTENT where necessary.
|
|||||||
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
|
| Intent | `INTENT.md` established; authorization-in-core amendments drafted |
|
||||||
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) |
|
| Research | yawex prior art; c2 origins; federation concepts; wikiengines overview (`research/260608-*/`); XWiki/TWiki/Foswiki deep dives (`research/260613-*/`); Xanadu + ZigZag + Roam + Obsidian + Notion + Joplin + Logseq + local-first workspaces (Anytype/AFFiNE/AppFlowy) + Trilium + Wiki.js + Federated Wiki + Wikibase + git-forge wikis + TiddlyWiki + ikiwiki + Quip + MojoMojo + Oddmuse + UseModWiki deep dives & shard-spectrum synthesis (`research/260614-*/`) |
|
||||||
| Demand | NetKingdom integration asks captured, not yet negotiated |
|
| Demand | NetKingdom integration asks captured, not yet negotiated |
|
||||||
| Spec | Architecture blueprint drafted; UseCaseCatalog 84 UCs from research; PRD/TSD scaffolds |
|
| Spec | CoreArchitectureBlueprint (whole-system architecture) + ArchitectureBlueprint (auth/history) drafted; UseCaseCatalog 84 UCs from research; PRD/TSD scaffolds |
|
||||||
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (18 tasks: T1–T10 federation + T11–T16 adapter contract + T17 federation-model taxonomy + T18 computational content, re-folded from synthesis v3 + the computational page model); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` **done** (all 8 computational-knowledge dives T1–T8 complete + "computational page model" synthesis) |
|
| Work | `SHARD-WP-0001` active (6 tasks); `SHARD-WP-0002` active (18 tasks: T1–T10 federation + T11–T16 adapter contract + T17 federation-model taxonomy + T18 computational content, re-folded from synthesis v3 + the computational page model); `SHARD-WP-0003` **done** (9 engine dives complete); `SHARD-WP-0004` **done** (all 8 computational-knowledge dives T1–T8 complete + "computational page model" synthesis) |
|
||||||
|
|
||||||
## In Scope (today)
|
## In Scope (today)
|
||||||
|
|||||||
510
spec/CoreArchitectureBlueprint.md
Normal file
510
spec/CoreArchitectureBlueprint.md
Normal file
@@ -0,0 +1,510 @@
|
|||||||
|
# CoreArchitectureBlueprint — shard-wiki
|
||||||
|
|
||||||
|
Status: **draft for review** · Date: 2026-06-15 · Owner: tegwick
|
||||||
|
|
||||||
|
The whole-system architecture for shard-wiki, synthesised from `INTENT.md`, the 84-entry
|
||||||
|
`UseCaseCatalog.md`, and the full research arc (`research/260608-*`, `research/260613-*`,
|
||||||
|
`research/260614-*` — ~23 wiki/knowledge systems plus two cross-dive syntheses). This is the
|
||||||
|
**core** blueprint: it defines the layers, the abstractions, and the load-bearing decisions
|
||||||
|
that everything else implements.
|
||||||
|
|
||||||
|
Scope relationship to the other specs:
|
||||||
|
|
||||||
|
- **`ArchitectureBlueprint.md`** (existing) is the **authorization & history sub-blueprint**
|
||||||
|
(the L0–L4 ladder). This document references it as the design of the cross-cutting
|
||||||
|
authorization layer (§9) and does not restate it.
|
||||||
|
- **`SHARD-WP-0002`** is the workplan that turns §6–§8 into
|
||||||
|
`spec/FederationArchitecture.md` + the adapter-contract section of
|
||||||
|
`spec/TechnicalSpecificationDocument.md`.
|
||||||
|
- **`UseCaseCatalog.md`** is the demand this architecture must satisfy; UC references below
|
||||||
|
are load tests, not decoration.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The thesis: *canonical at the edges, derived in the middle*
|
||||||
|
|
||||||
|
Everything in shard-wiki follows from one organising decision:
|
||||||
|
|
||||||
|
> **The canonical truth lives at the edges — in each shard (content) and in the Git
|
||||||
|
> coordination journal (history). Everything shard-wiki computes in between — the union, the
|
||||||
|
> projections, the views, the query indexes — is _derived state that can be deleted and
|
||||||
|
> rebuilt_ from those edges.**
|
||||||
|
|
||||||
|
This is the architectural form of "orchestrator, not engine." shard-wiki never *becomes* the
|
||||||
|
source of truth; it composes sources. The research earned this thesis empirically — every
|
||||||
|
serious system externalises its durable truth to files+VCS and treats the rest as derived:
|
||||||
|
Logseq (DataScript index over plain files), ikiwiki (static HTML compiled from a git repo),
|
||||||
|
Glamorous Toolkit / Lepiter (live views over git-versioned JSON), Pharo (Tonel/Iceberg code
|
||||||
|
as git text), Jupyter teams (nbstripout — outputs are derived noise). The one tradition that
|
||||||
|
refuses this — the Smalltalk **image** — is exactly the one we record as a *boundary, not a
|
||||||
|
backend* (`research/260614-squeak-pharo-deep-dive`).
|
||||||
|
|
||||||
|
Three consequences fall straight out, and they are the spine of the rest of this document:
|
||||||
|
|
||||||
|
1. **Graceful degradation is free.** If the derived middle is always rebuildable, a backend
|
||||||
|
that can only be read is still a first-class participant — you just derive less from it.
|
||||||
|
2. **Provenance is tractable.** Because shard-wiki never claims to *be* the source, every
|
||||||
|
derived artifact can always point back to the canonical edge it came from (union without
|
||||||
|
erasure is a structural property, not a feature bolted on).
|
||||||
|
3. **The system is a pure function of its inputs.** `union/index/projection = f(shards,
|
||||||
|
journal)`. Bugs in the middle are recoverable by rebuild; the edges are the only thing
|
||||||
|
that must be protected (and history protects them).
|
||||||
|
|
||||||
|
### The dual narrow waist
|
||||||
|
|
||||||
|
Heterogeneity is mediated at exactly two interfaces, and nowhere else:
|
||||||
|
|
||||||
|
- **Bottom waist — the Shard Adapter Contract (§6).** Every backend, however weird, enters
|
||||||
|
through one versioned, capability-described interface.
|
||||||
|
- **Top waist — the Wiki Page Model (§7).** Every consumer, however demanding, sees one
|
||||||
|
backend-neutral, Markdown-first-but-stretchable page model.
|
||||||
|
|
||||||
|
Between the waists, core logic is written **once** against capabilities and the page model —
|
||||||
|
never against a specific backend. Adding TiddlyWiki or Notion or a git forge is writing an
|
||||||
|
adapter and declaring a capability profile, not editing core algorithms.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Architectural invariants
|
||||||
|
|
||||||
|
These are non-negotiable. Violating one is a design bug, not a tradeoff. They are INTENT's
|
||||||
|
principles fused with the research through-lines.
|
||||||
|
|
||||||
|
| # | Invariant | Source |
|
||||||
|
|---|-----------|--------|
|
||||||
|
| I-1 | **Orchestrator, not engine.** Core composes shards; it never replaces or homogenises them. | INTENT Stability Note |
|
||||||
|
| I-2 | **Canonical at the edges, derived in the middle.** Union/index/projection are rebuildable from shards + journal. | §1; Logseq/ikiwiki/GT through-line |
|
||||||
|
| I-3 | **Capability-awareness is data.** A binding's abilities are a *profile* (positions on spectra), read by generic core logic — not per-backend branches. | synthesis v3 §2; INTENT capability-aware adapters |
|
||||||
|
| I-4 | **Union without erasure.** Every page/revision/projection/overlay/view carries its provenance, freshness, liveness, and divergence. | INTENT; provenance-granularity spectrum (Wikibase) |
|
||||||
|
| I-5 | **Overlay before mutation.** Writes to anything below write-through land as drafts/patches/MRs first; no silent remote mutation. | INTENT |
|
||||||
|
| I-6 | **Git-addressable coordination.** Every information space has a Git-backed journal even when its shards are not git-native. | INTENT |
|
||||||
|
| I-7 | **Mechanism over policy.** Canonical-source, conflict, editorial, sync cadence are configurable presets, never hard-coded. | INTENT |
|
||||||
|
| I-8 | **Graceful degradation.** A limited backend is still usable as read-only / cache / projection / backup / patch target. | INTENT |
|
||||||
|
| I-9 | **Identity ≠ placement.** A page is an entity that may occupy N locations; address by identity, not by path. | Trilium note/branch; ZigZag |
|
||||||
|
| I-10 | **History is the floor.** Every write is a recoverable commit; recoverability, not gatekeeping, is the baseline protection. | ArchitectureBlueprint §2 |
|
||||||
|
| I-11 | **Authorization in core, authentication delegated.** Core decides who-may; an external provider says who-is. | INTENT; ArchitectureBlueprint |
|
||||||
|
| I-12 | **Not a file-sync daemon; not an execution platform.** Sync is wiki-page-semantic; computation is recognised+projected, not hosted. | INTENT; computational-page-model synthesis |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. The layered architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌───────────────────────────────────────────────────────────────┐
|
||||||
|
│ L6 Consumers — Orchestrator API · CLI/agents · Web/Obsidian │
|
||||||
|
├───────────────────────────────────────────────────────────────┤
|
||||||
|
X-cut │ L5 Authorization (PEP/PDP, identity-provider iface) → │ X-cut
|
||||||
|
Prove- │ see ArchitectureBlueprint.md (L0–L4 ladder) │ Capa-
|
||||||
|
nance ├───────────────────────────────────────────────────────────────┤ bility
|
||||||
|
▲ │ L4 Union & Projection (DERIVED, rebuildable cache) │ ▲
|
||||||
|
│ │ identity resolution · equivalence/chorus · union graph · │ │
|
||||||
|
│ │ replication+derivation projections · moldable view registry│ │
|
||||||
|
│ │ · derived query index │ │
|
||||||
|
│ ├───────────────────────────────────────────────────────────────┤ │
|
||||||
|
│ │ L3 Coordination (Git journal · overlay/patch engine · │ │
|
||||||
|
│ │ federation-model strategies · reconciliation) │ │
|
||||||
|
│ ├───────────────────────────────────────────────────────────────┤ │
|
||||||
|
│ │ L2 Wiki Page Model ── TOP WAIST ── │ │
|
||||||
|
│ │ backend-neutral pages · identity≠placement · span address ·│ │
|
||||||
|
│ │ provenance envelope · the page shapes │ │
|
||||||
|
│ ├───────────────────────────────────────────────────────────────┤ │
|
||||||
|
│ │ L1 Shard Adapter Contract ── BOTTOM WAIST ── │ │
|
||||||
|
│ │ versioned iface · capability profile (15 spectra) · │ │
|
||||||
|
│ │ attachment-mode binding · operation verbs │ │
|
||||||
|
└──── ├───────────────────────────────────────────────────────────────┤ ──┘
|
||||||
|
│ L0 Backends (not ours): git repos, wiki/ subdirs, Gitea/ │
|
||||||
|
│ GitLab/GitHub wikis, folders, Obsidian, WebDAV, Notion, │
|
||||||
|
│ Coulomb spaces, notebooks, … │
|
||||||
|
└───────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Provenance** and **Capability** are drawn as vertical rails because they are not layers —
|
||||||
|
they are present at every layer. A page object at L2 carries provenance; a projection at L4
|
||||||
|
carries provenance; an authz decision at L5 records the context under which content was read.
|
||||||
|
Likewise a capability profile declared at L1 is consulted at L3 (can we write-through?), L4
|
||||||
|
(can we delegate a query?), and L5 (can this principal even reach the op?).
|
||||||
|
|
||||||
|
The dependency rule is strict and downward: **L4 may be deleted and recomputed from L1–L3.**
|
||||||
|
Nothing at L1–L3 may depend on L4 state.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Core abstractions (the vocabulary code must use)
|
||||||
|
|
||||||
|
Straight from INTENT, sharpened by research. New code maps onto these; it does not invent
|
||||||
|
parallel terms.
|
||||||
|
|
||||||
|
- **Shard** — an independently meaningful page store attached to a root entity, with
|
||||||
|
*sovereignty*: its own backend, capability profile, history, identity model, limits.
|
||||||
|
- **Root entity / information space** — the joined space shards attach to; the unit of
|
||||||
|
Git coordination and of multi-tenancy (a tenant maps to a root entity, ArchitectureBlueprint).
|
||||||
|
- **Shard adapter contract** — the versioned L1 interface; the bottom waist.
|
||||||
|
- **Capability profile** — a shard binding's position on each of the 15 spectra (§6) plus its
|
||||||
|
supported verbs. *The* data structure that drives degradation.
|
||||||
|
- **Wiki page model** — the L2 backend-neutral page; the top waist.
|
||||||
|
- **Page identity vs placement** — a page is an entity (identity); it may have N placements
|
||||||
|
(paths/shards). Addressing, equivalence, and transclusion key on identity (I-9).
|
||||||
|
- **Provenance envelope** — the metadata wrapper every artifact carries: source shard,
|
||||||
|
freshness, liveness, authorization context, overlay status, divergence, derivation lineage.
|
||||||
|
- **Coordination journal** — the L3 Git-backed record of change flows for a space.
|
||||||
|
- **Overlay** — a non-destructive local edit against a remote/read-only/limited shard,
|
||||||
|
representable as draft/patch/commit/MR before destructive apply.
|
||||||
|
- **Projection** — a derived view of shard content, typed on two axes (§8): *kind*
|
||||||
|
(replication | derivation) × *liveness* (static … irreducibly-live).
|
||||||
|
- **Federation model** — the selected coordination strategy for a space (§ taxonomy, T17).
|
||||||
|
- **Shard mode** — read-only · write-through · mirrored · projected · cached · canonical
|
||||||
|
(a *policy* selection constrained by the capability profile).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Why "layered" and not "pipeline" or "plugin-bus"
|
||||||
|
|
||||||
|
Two rejected alternatives, recorded so the choice is legible:
|
||||||
|
|
||||||
|
- **A sync pipeline** (source → transform → sink) was rejected: it implies a privileged
|
||||||
|
direction and a canonical sink, which violates shard sovereignty (I-1) and union-without-
|
||||||
|
erasure (I-4). shard-wiki is a *star* (many shards ↔ one space), not a pipe.
|
||||||
|
- **A flat plugin bus** (every backend a peer plugin emitting events) was rejected as the
|
||||||
|
*top-level* shape: it has no narrow waist, so heterogeneity leaks into every consumer.
|
||||||
|
We keep the plugin idea but confine it to L1 (adapters) and L3 (federation strategies),
|
||||||
|
behind the waists.
|
||||||
|
|
||||||
|
The layered-with-rails shape is what makes I-2/I-3/I-4 hold simultaneously.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Bottom waist — the Shard Adapter Contract (L1)
|
||||||
|
|
||||||
|
The single most important design decision in the project: **the adapter contract models
|
||||||
|
positions on capability spectra, not a flat checklist of boolean verbs.** A backend is not
|
||||||
|
"can/can't merge"; it sits *somewhere* on the merge spectrum, and federation operations
|
||||||
|
degrade by position. This is the lesson of putting ~23 systems in one matrix
|
||||||
|
(`research/260614-shard-spectrum-synthesis`, v3).
|
||||||
|
|
||||||
|
### 6.1 The fifteen capability spectra
|
||||||
|
|
||||||
|
Each binding declares a position on each axis. Core algorithms read these positions; there is
|
||||||
|
no per-backend code in core (I-3).
|
||||||
|
|
||||||
|
1. **Addressing granularity** — none → path → page-level store-id → in-file span → in-file
|
||||||
|
block id (Logseq `id::`) → store-UUID → portable tumbler (Xanadu, the unreached ideal)
|
||||||
|
2. **Content identity** — none → path/title → fingerprint → span-set
|
||||||
|
3. **Identity vs placement** — path=identity → identity separated from placement (Trilium
|
||||||
|
note/branch = a DAG)
|
||||||
|
4. **Structure** — flat MD → frontmatter/`key::` → `%META%` → typed objects → DB schema+
|
||||||
|
relations → object-graph/ontology → computed (inherited+templated) → typed-graph statements
|
||||||
|
5. **History** — none → internal-only / CRDT-log → open-file → git-native
|
||||||
|
6. **Merge model** — none → git/text → conflict-notes/keep-both → native-CRDT → coexist-with-rank
|
||||||
|
7. **Native query** — none → text → build-your-own derived index → datalog/graph → DB query → SPARQL
|
||||||
|
8. **Translation** — native → lossless → lossy-with-fidelity-report (incl. HTML)
|
||||||
|
9. **Attachment mode** — file-store (native | interchange-mirror) → git-IS-store → in-engine-host
|
||||||
|
→ local-REST → external-API → direct-DB → CRDT-replica → P2P/no-central-endpoint
|
||||||
|
10. **Operational envelope** — local/unbounded → realtime CRDT/WebSocket → rate-limited/
|
||||||
|
eventually-consistent/paginated
|
||||||
|
11. **Access grant** — open → token → OAuth scoped+revocable → P2P key/invite → enterprise ACL
|
||||||
|
12. **Content opacity** — plaintext → structured re-evaluable value → encrypted whole-shard →
|
||||||
|
per-item → proprietary-lossy-exportable
|
||||||
|
13. **Write granularity** — whole-file (TiddlyWiki) → per-page → section/anchor → per-block → story-item
|
||||||
|
14. **Provenance granularity** — per-shard → per-page → per-edit → per-statement/value (Wikibase rank+refs)
|
||||||
|
15. **Computational / liveness** — static source → captured-output snapshot → live-over-files →
|
||||||
|
view-time render → irreducibly-live/temporal
|
||||||
|
|
||||||
|
### 6.2 Operation verbs
|
||||||
|
|
||||||
|
`read, write, diff, merge, lock, version, publish, notify, transclude-source,
|
||||||
|
translate-syntax, structured-payload, derive-projection, execute/evaluate`. The last two are
|
||||||
|
**gated, off by default** (§8, computational content). Verb support is part of the profile and
|
||||||
|
must reconcile with the federation-ops capability matrix (SHARD-WP-0002 T10).
|
||||||
|
|
||||||
|
### 6.3 Attachment-mode taxonomy (axis 9, expanded)
|
||||||
|
|
||||||
|
A backend may offer **several** modes; attach mode is a **per-binding, capability-gated
|
||||||
|
choice**, with one declared authoritative. Modes: file-store (native vault/folder *or* an
|
||||||
|
interchange/sync mirror), **git-IS-store** (the home case — forge wikis & ikiwiki: git is the
|
||||||
|
store *and* the journal at once, resolving the engine-mirror write-race), in-engine hosted
|
||||||
|
adapter (XWiki component, Obsidian/Logseq/Roam plugin, Trilium script), local-REST (Joplin
|
||||||
|
Data API, Trilium ETAPI), external-API-only (Notion), direct-DB (MojoMojo schema→model),
|
||||||
|
CRDT-replica (Anytype/AFFiNE/AppFlowy), P2P/no-central-endpoint. **Boundary:** a monolithic
|
||||||
|
live-memory blob (Smalltalk image, a kernel) is **never** an attach target — it participates
|
||||||
|
only via export→files (I-12).
|
||||||
|
|
||||||
|
### 6.4 Contract rules
|
||||||
|
|
||||||
|
- **Versioned interface** (Foswiki::Store + Foswiki::Meta is the proof that a stable
|
||||||
|
store-interface-with-swappable-backends works). Capability discovery is a static profile
|
||||||
|
with optional runtime negotiation.
|
||||||
|
- **Backend-swap tolerance** — shard identity/provenance survives a substrate change
|
||||||
|
(RCS↔PlainFile, folder→Git, Logseq file→SQLite): bind to *capabilities*, not to "it's files."
|
||||||
|
- **Absence is first-class** — the profile must express *can't* cleanly (Oddmuse floor), so
|
||||||
|
degradation paths are explicit, never guessed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Top waist — the Wiki Page Model (L2)
|
||||||
|
|
||||||
|
Backend-neutral, **Markdown-first but stretchable many ways at once**. The page model is the
|
||||||
|
lingua franca every consumer sees; an adapter's job is to project its backend into this model
|
||||||
|
(read) and accept overlays back (write), within its capabilities.
|
||||||
|
|
||||||
|
### 7.1 Page shapes the model must carry
|
||||||
|
|
||||||
|
- **Prose Markdown** — the baseline.
|
||||||
|
- **Typed / computed records** — frontmatter/`%META%`/XObjects/Notion DB rows; **computed
|
||||||
|
metadata** (Trilium inherited+templated) represented as *effective-vs-own with per-attribute
|
||||||
|
provenance*.
|
||||||
|
- **Typed-graph statements** — Wikibase claim + qualifiers + references + rank (structure
|
||||||
|
far-end).
|
||||||
|
- **Inline-embedded objects** — Quip/Notion spreadsheets & live apps inside prose.
|
||||||
|
- **Non-Markdown assets** — drawings, canvases, images: typed asset / opaque blob / pluggable
|
||||||
|
content-type registry, never silent-flattened.
|
||||||
|
- **The four computational shapes** (§8): one-source-many-projections, notebook (embedded
|
||||||
|
computed output), program-as-page, live/temporal.
|
||||||
|
|
||||||
|
All shapes reduce to a common skeleton: **`(content | source, structure, provenance envelope,
|
||||||
|
[derivation rule])`**. The page model stores the richest faithful form as canonical and treats
|
||||||
|
any Markdown rendering of a non-Markdown shape as a *lossy projection* (I-4 + fidelity report).
|
||||||
|
|
||||||
|
### 7.2 Identity, placement, addressing
|
||||||
|
|
||||||
|
- **Identity** is a content-derived or native-id-wrapped key, *shard-scoped* so native UUIDs
|
||||||
|
survive projection and don't collide.
|
||||||
|
- **Placement** is separate: one identity → N placements (paths/shards) = a DAG. No single
|
||||||
|
canonical path (I-9).
|
||||||
|
- **Span addressing** — adopt native span IDs where minted (Roam `:block/uid`, Logseq `id::`,
|
||||||
|
Notion/CRDT UUID); else content fingerprint or path+range. The Xanadu tumbler is the ideal
|
||||||
|
the scheme aims at without requiring.
|
||||||
|
- **Provenance envelope** rides on every page and span (source shard, freshness, liveness,
|
||||||
|
overlay status, authz context, divergence, derivation lineage).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Coordination, federation & projection
|
||||||
|
|
||||||
|
### 8.1 Coordination journal (L3) — Git as the spine
|
||||||
|
|
||||||
|
Every information space has a Git-backed coordination journal (I-6). It records cross-shard
|
||||||
|
operations (fork, import, reconcile, overlay-apply, space-branch) and **is** the history floor
|
||||||
|
(I-10). For git-IS-store shards the shard's own git log *is* this journal; for non-git shards
|
||||||
|
the journal supplements (begins-now / mirrors-forward / snapshots-replica) or imports
|
||||||
|
(backfill open file history). History portability is a spectrum, handled per profile (axis 5).
|
||||||
|
|
||||||
|
### 8.2 Overlay / patch engine (L3)
|
||||||
|
|
||||||
|
The default write path for anything below write-through capability (I-5): an edit becomes a
|
||||||
|
draft → patch/commit → MR, applied destructively only on explicit intent and only where the
|
||||||
|
profile + policy both permit. This is what lets a read-only or rate-limited or lossy backend
|
||||||
|
still be *edited* safely.
|
||||||
|
|
||||||
|
### 8.3 Federation is plural & composable (L3) — the model taxonomy
|
||||||
|
|
||||||
|
Federation is not one mechanism. shard-wiki selects a **federation model per space and
|
||||||
|
composes per shard** (mechanism over policy, I-7):
|
||||||
|
|
||||||
|
| Model | Anchor | Coordination shape |
|
||||||
|
|-------|--------|--------------------|
|
||||||
|
| **Fork + journal** (default home case) | Federated Wiki | copy-with-provenance + per-page action journal (story = replay) |
|
||||||
|
| **VCS-replication + ping** | ikiwiki | git clone/pull/push + change-ping |
|
||||||
|
| **Query-time graph-join** | Wikibase SPARQL `SERVICE` | join remote graphs at query time, no copy |
|
||||||
|
| **Feed aggregation** | RSS/Atom | inbound feed → pages |
|
||||||
|
| **Activity streams** | ActivityPub | Create/Update events, notify or content-bearing |
|
||||||
|
| **Engine-mirror** | Wiki.js DB↔Git | engine syncs its own store to a git mirror |
|
||||||
|
|
||||||
|
### 8.4 Union & projection (L4) — the derived cache
|
||||||
|
|
||||||
|
This whole layer is rebuildable from L1–L3 (I-2). It comprises:
|
||||||
|
|
||||||
|
- **Identity resolution & equivalence** — detect "same topic / derived content" path-
|
||||||
|
independently (fingerprint, span-set overlap, alias table, curator binding); present as
|
||||||
|
**chorus-of-voices** or designated-canonical (a *policy* preset).
|
||||||
|
- **Union graph** — the navigable join of pages, links, and dimensions (namespace, genealogy,
|
||||||
|
version, shard, equivalence). A *derived lens over canonical files+journal, never a new
|
||||||
|
store* (the ZigZag boundary).
|
||||||
|
- **Transclusion** — one **reference-not-copy** primitive unifying Xanadu transclusion, ZigZag
|
||||||
|
clone, Roam/Obsidian/Logseq embed, Notion synced block, Trilium note-cloning, and literate
|
||||||
|
named-chunk assembly, over the addressable union.
|
||||||
|
- **Projection — the two-axis model:**
|
||||||
|
- *Kind:* **replication-projection** (lazy cache of remote content — the default) vs
|
||||||
|
**derivation-projection** (transform/compile/weave/evaluate a source).
|
||||||
|
- *Liveness:* static → captured snapshot → live-over-files → view-time → irreducibly-live.
|
||||||
|
- Derivation facets: materialization timing (ahead-of-time vs view-time), multiplicity (one
|
||||||
|
output vs N co-equal), continuity (one-shot vs continuous). Every projection declares its
|
||||||
|
liveness + freshness + provenance; the irreducibly-live far end has no faithful static
|
||||||
|
form (source + a marked recording).
|
||||||
|
- **Moldable view registry** — projection generalises to an **open, type-keyed set of
|
||||||
|
co-equal, possibly-computed views, none canonical-by-fact** (display-canonical is policy).
|
||||||
|
This unifies replication/derivation/dimensional/query projection and answers the "pluggable
|
||||||
|
content-type registry" question (GT prior art).
|
||||||
|
- **Derived query index** — delegate to a shard's native query engine where present
|
||||||
|
(Roam/Logseq Datalog, Notion DB query, XWiki XWQL, Wikibase SPARQL); else build a derived
|
||||||
|
index over the projection (the Logseq DataScript-over-files pattern). The index is
|
||||||
|
disposable (I-2).
|
||||||
|
|
||||||
|
### 8.5 Computational / executable content — the scope decision
|
||||||
|
|
||||||
|
**In scope as a page-model + projection concern; out of scope as an execution platform.**
|
||||||
|
shard-wiki *recognises* computational types, attaches the **canonical source**, and presents
|
||||||
|
derived forms as **provenance- and liveness-marked projections**. Driving a derivation
|
||||||
|
(tangle/weave, re-execute a notebook, render a sketch, evaluate a pattern) is a **gated
|
||||||
|
capability, off by default, with a trust/sandbox concern, degrading to a captured snapshot**.
|
||||||
|
One snapshot-provenance record (run id, source rev, timestamp, environment "unguaranteed")
|
||||||
|
serves notebooks, renders, and recordings alike. **No INTENT amendment is required** — this
|
||||||
|
lives inside the existing page model (L2) and projection model (L4).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Cross-cut — Authorization (L5)
|
||||||
|
|
||||||
|
Fully specified in **`ArchitectureBlueprint.md`** (the access & history sub-blueprint);
|
||||||
|
summarised here for completeness:
|
||||||
|
|
||||||
|
- **One core, a ladder of modes** L0 (open/c2, zero deps) → L1 (attributed) → L2
|
||||||
|
(authenticated) → L3 (role/group) → L4 (multi-tenant enterprise). Climbing is configuration,
|
||||||
|
not re-architecture.
|
||||||
|
- **PEP** wraps every adapter op; **PDP** decides `(principal, action, target)` over actions
|
||||||
|
`read/write/patch/merge/administer`, layered on the adapter's capability profile (a shard
|
||||||
|
that can't write can't be written regardless of policy — L5 consults the L1 rail).
|
||||||
|
- **Authentication delegated** to a pluggable IdentityProvider (null provider = L0 default);
|
||||||
|
real identity from `user-engine` over `net-kingdom` IAM.
|
||||||
|
- **Fail open only at L0, fail closed at L2+.** Authorization is pure/offline once a Principal
|
||||||
|
is resolved. Provenance carries authz context so the union never leaks unreadable content
|
||||||
|
(the L5↔provenance-rail interaction).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. The policy surface (mechanism over policy, made concrete)
|
||||||
|
|
||||||
|
I-7 only means something if the policy knobs are enumerated and kept *out* of core algorithms.
|
||||||
|
The configurable presets are:
|
||||||
|
|
||||||
|
- **Canonical-source policy** — chorus / designated-canonical / git-merge / overlay-only /
|
||||||
|
vote-to-merge (per space or per equivalence set).
|
||||||
|
- **Federation model** — the §8.3 taxonomy, per space, composable per shard.
|
||||||
|
- **Shard mode** — read-only / write-through / mirrored / projected / cached / canonical
|
||||||
|
(constrained by the capability profile).
|
||||||
|
- **Reconciliation cadence & conflict exposure** — push/poll/manual; show-conflicts vs
|
||||||
|
auto-merge-when-supported.
|
||||||
|
- **Execution policy** — derive/execute off (default) / sandboxed / per-shard-allowed.
|
||||||
|
- **Authorization mode** — the L0–L4 ladder.
|
||||||
|
- **Projection materialization** — lazy/eager; snapshot vs view-time; recording retention.
|
||||||
|
|
||||||
|
Core ships sane defaults (L0 open; fork+journal; lazy replication-projection; overlay-before-
|
||||||
|
mutation; execution off) and never hard-codes any of the above.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Concrete module structure (bridge to implementation)
|
||||||
|
|
||||||
|
A proposed package layout for `src/shard_wiki/`, mapping 1:1 to the layers so the dependency
|
||||||
|
rule (downward only; L4 rebuildable) is enforceable by import lint:
|
||||||
|
|
||||||
|
```
|
||||||
|
src/shard_wiki/
|
||||||
|
model/ # L2 top waist: Page, Identity, Placement, ProvenanceEnvelope,
|
||||||
|
# Span, the page-shape types; capability-spectrum value types
|
||||||
|
adapters/ # L1 bottom waist: AdapterContract (versioned iface), CapabilityProfile,
|
||||||
|
# attachment-mode binding; concrete adapters:
|
||||||
|
git/ folder/ gitea/ obsidian/ webdav/ notion/ … # each: profile + verbs
|
||||||
|
coordination/ # L3: GitJournal, OverlayEngine (draft→patch→MR), reconcile
|
||||||
|
federation/ # L3: FederationModel strategies (fork_journal, vcs_ping,
|
||||||
|
# graph_join, feed, activitypub, engine_mirror)
|
||||||
|
union/ # L4 (derived): IdentityResolver, EquivalenceGraph, UnionGraph,
|
||||||
|
# Transclusion (reference-not-copy)
|
||||||
|
projection/ # L4 (derived): ReplicationProjection, DerivationProjection,
|
||||||
|
# ViewRegistry (moldable), QueryIndex (delegate|derive)
|
||||||
|
authz/ # L5 cross-cut: PDP, PEP, IdentityProvider iface, NullProvider
|
||||||
|
provenance/ # cross-cut: the envelope plumbing used by every layer
|
||||||
|
api/ # L6: orchestrator API (server-side union for agents/CLI)
|
||||||
|
```
|
||||||
|
|
||||||
|
Hard import rules: `union/` and `projection/` may import `model/`, `adapters/`,
|
||||||
|
`coordination/` but **nothing may import them** (they are the disposable middle). `model/` and
|
||||||
|
`adapters/` import nothing else in the tree except `provenance/` (the waists stay thin).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Canonical data flows (the architecture exercised)
|
||||||
|
|
||||||
|
**A. Attach a shard.** Adapter binds (chosen attachment mode) → probes/declares a capability
|
||||||
|
profile → core registers the shard under a root entity → if not git-native, the coordination
|
||||||
|
journal is seeded (begin-now/mirror/import per axis 5). No union rebuild yet (lazy).
|
||||||
|
|
||||||
|
**B. Read a page through the union.** Consumer asks the union for an identity → Identity
|
||||||
|
resolver maps it to placements across shards → equivalence yields chorus or canonical →
|
||||||
|
replication-projection lazily fetches from each shard (cache + freshness) → page returned
|
||||||
|
wrapped in its provenance envelope → L5 filters anything the principal can't see at source.
|
||||||
|
|
||||||
|
**C. Edit a read-only / limited shard.** Write request → L5 PDP allows → capability profile
|
||||||
|
says < write-through → OverlayEngine records a draft → renders a patch/MR in the shard's native
|
||||||
|
syntax (lossless) or Markdown (lossy-with-report) → on explicit apply, commit to the journal
|
||||||
|
and (if the profile permits) propagate; otherwise the overlay stands as the local truth, fully
|
||||||
|
attributed.
|
||||||
|
|
||||||
|
**D. Attach a computational notebook.** Adapter declares profile (attachment=file-store,
|
||||||
|
opacity=mixed, computational=captured-output). Core attaches the `.ipynb` **source** as
|
||||||
|
canonical; presents cells + embedded outputs as **derivation-projection snapshots** marked
|
||||||
|
"run N, env unguaranteed"; offers a static render via the view registry; re-execution stays
|
||||||
|
gated off. History uses paired-text/nbdime per axis 5.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Key tradeoffs & decisions to confirm
|
||||||
|
|
||||||
|
Resolved here:
|
||||||
|
|
||||||
|
- **Capability spectra over a verb checklist** — accept richer contract complexity for precise,
|
||||||
|
uniform degradation. (Decided: spectra.)
|
||||||
|
- **Derived middle is a cache, not a store** — accept recompute cost for rebuildability,
|
||||||
|
provenance, and graceful degradation. (Decided: cache.)
|
||||||
|
- **Default federation = fork+journal over Git** — the home case; other models opt-in. (Decided.)
|
||||||
|
- **Execution off by default** — recognise+project always; execute only when gated on. (Decided.)
|
||||||
|
|
||||||
|
Open — to confirm before SHARD-WP-0002 spec-writing finalises:
|
||||||
|
|
||||||
|
1. **Union graph persistence.** Pure-recompute (simplest, honours I-2 hardest) vs a persisted-
|
||||||
|
but-disposable cache (faster, must guarantee rebuild equivalence). *Recommendation:
|
||||||
|
persisted-but-disposable with a `rebuild` that must reproduce it byte-for-byte.*
|
||||||
|
2. **Address scheme.** Ship shard-scoped native-id wrapping now and treat a portable tumbler as
|
||||||
|
a later capability, or design the tumbler up front? *Recommendation: wrap native ids now.*
|
||||||
|
3. **L1 "attributed-but-open" mode** — ship it or jump L0→L2? (Carried from ArchitectureBlueprint.)
|
||||||
|
4. **Per-page ACL default** — off (per-shard/namespace) confirmed; revisit only if demand appears.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. What this architecture is *not*
|
||||||
|
|
||||||
|
- Not a wiki engine, UI, or rendering pipeline (those are consumers at L6).
|
||||||
|
- Not a canonical-source-of-truth — shards keep sovereignty; the middle is derived.
|
||||||
|
- Not a generic file-sync daemon — synchronisation is wiki-page-semantic.
|
||||||
|
- Not an execution platform — computation is recognised and projected, not hosted.
|
||||||
|
- Not a universal ontology — no single schema is imposed on all shards.
|
||||||
|
- Not an authentication/identity store — that is delegated (authorization is owned).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 15. Traceability
|
||||||
|
|
||||||
|
- **INTENT** — every invariant in §2 cites an INTENT principle or boundary; no invariant
|
||||||
|
contradicts the Stability Note.
|
||||||
|
- **Research** — §6 (spectra) ← `260614-shard-spectrum-synthesis` v3; §8.3 (federation
|
||||||
|
taxonomy) ← v3 §2.5; §8.4–§8.5 (two-axis projection, view registry, computational scope) ←
|
||||||
|
`260614-computational-page-model-synthesis`; §7 page shapes ← the engine + modern-tool +
|
||||||
|
computational dives; §1 thesis ← the files-canonical/index-derived through-line across
|
||||||
|
Logseq/ikiwiki/GT/Pharo/Jupyter.
|
||||||
|
- **Use cases** — the architecture is sized to UC-01–UC-84: federation/coordination (UC-01–07,
|
||||||
|
26–33, 56, 71–72, 79) → §8; attachment/adapter (UC-34–43, 50, 53, 57, 60–62, 64–66, 68–70,
|
||||||
|
76–82) → §6; page model & fidelity (UC-34, 39, 42, 55, 58–59, 67, 73, 80, 83–84) → §7/§8.5;
|
||||||
|
addressing/identity/query (UC-32, 44–49, 51–52, 54, 63, 74) → §7.2/§8.4; provenance &
|
||||||
|
metadata (UC-24–25, 75) → the provenance rail; collaboration & discovery (UC-08–23) → L6
|
||||||
|
consumers over the union.
|
||||||
|
- **Workplans** — §6–§8 are the design target of `SHARD-WP-0002` (T11–T18); §9 is owned by
|
||||||
|
`ArchitectureBlueprint.md`; §1 (yawex-derived resolution/overlay) aligns with
|
||||||
|
`SHARD-WP-0001`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 16. Stability note
|
||||||
|
|
||||||
|
This document defines shard-wiki's **internal** architecture; it may evolve as the spec
|
||||||
|
workplans land. But the **thesis (§1)**, the **invariants (§2)**, and the **dual narrow waist
|
||||||
|
(§1, §6, §7)** are load-bearing — changing any of them is an architectural change in the sense
|
||||||
|
of INTENT's Stability Note and should be rare and deliberate.
|
||||||
@@ -6,10 +6,11 @@ Background on document types: InfoTechPrimers on coulomb.social.
|
|||||||
|
|
||||||
| File | Status | Role |
|
| File | Status | Role |
|
||||||
|------|--------|------|
|
|------|--------|------|
|
||||||
|
| `CoreArchitectureBlueprint.md` | draft for review | **Whole-system architecture** — layers, abstractions, load-bearing decisions (synthesised from all research) |
|
||||||
| `ProductRequirementsDocument.md` | draft scaffold | What the product must deliver |
|
| `ProductRequirementsDocument.md` | draft scaffold | What the product must deliver |
|
||||||
| `TechnicalSpecificationDocument.md` | draft scaffold | How the system is built |
|
| `TechnicalSpecificationDocument.md` | draft scaffold | How the system is built |
|
||||||
| `UseCaseCatalog.md` | draft | 25 use cases promoted from c2 + yawex research |
|
| `UseCaseCatalog.md` | draft | 84 use cases promoted from c2 + yawex + ~23-system research |
|
||||||
| `ArchitectureBlueprint.md` | draft | Access, history, and identity architecture |
|
| `ArchitectureBlueprint.md` | draft | Access, history, and identity sub-blueprint (the L0–L4 authorization ladder; referenced by CoreArchitectureBlueprint §9) |
|
||||||
|
|
||||||
Promote material from `research/` and reviewed items from `demand/` into spec
|
Promote material from `research/` and reviewed items from `demand/` into spec
|
||||||
before treating it as implementation authority.
|
before treating it as implementation authority.
|
||||||
Reference in New Issue
Block a user