spec(SHARD-WP-0005 T5): cache freshness & invalidation protocol (§8.8)

Fixes C-2 (invalidation). Per-binding modes (event-driven push / validator
poll / TTL), hybrid default, operational-envelope coupling (rate-limited
shards must not poll-per-read), single-flight coalescing + batched bulk
invalidation, freshness always represented in the provenance envelope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

spec/CoreArchitectureBlueprint.md

@@ -21,34 +21,51 @@ Scope relationship to the other specs:
 
 ---
 
-## 1. The thesis: *canonical at the edges, derived in the middle*
+## 1. The thesis: *canonical vs derived* (three states)
 
-Everything in shard-wiki follows from one organising decision:
+Everything in shard-wiki follows from one organising decision — that state comes in exactly
+**three kinds**, and only one of them is disposable:
 
-> **The canonical truth lives at the edges — in each shard (content) and in the Git
-> coordination journal (history). Everything shard-wiki computes in between — the union, the
-> projections, the views, the query indexes — is _derived state that can be deleted and
-> rebuilt_ from those edges.**
+> **1. Sharded-canonical** — content owned by each shard (shard sovereignty).
+> **2. Coordination-canonical** — durable state *born inside shard-wiki* that encodes human
+> or cross-shard decisions and exists nowhere else: overlays (the local truth against a
+> read-only shard), curator equivalence bindings, alias tables, merge/reconciliation
+> decisions. It lives in the **Git coordination journal**.
+> **3. Derived-disposable** — everything shard-wiki *computes* from (1)+(2): the union graph,
+> equivalence index, query indexes, projections, views. It can be deleted and recomputed.
+>
+> **Canonical = sharded ∪ coordination. Derived = a pure function of canonical:**
+> `derived = f(sharded, coordination)`.
 
 This is the architectural form of "orchestrator, not engine." shard-wiki never *becomes* the
-source of truth; it composes sources. The research earned this thesis empirically — every
-serious system externalises its durable truth to files+VCS and treats the rest as derived:
-Logseq (DataScript index over plain files), ikiwiki (static HTML compiled from a git repo),
-Glamorous Toolkit / Lepiter (live views over git-versioned JSON), Pharo (Tonel/Iceberg code
-as git text), Jupyter teams (nbstripout — outputs are derived noise). The one tradition that
-refuses this — the Smalltalk **image** — is exactly the one we record as a *boundary, not a
-backend* (`research/260614-squeak-pharo-deep-dive`).
+source of truth; it composes sources and records the decisions it makes about them. The
+research earned the *files-canonical* half empirically — every serious system externalises its
+durable truth to files+VCS and treats the rest as derived: Logseq (DataScript index over plain
+files), ikiwiki (static HTML compiled from a git repo), Glamorous Toolkit / Lepiter (live
+views over git-versioned JSON), Pharo (Tonel/Iceberg code as git text), Jupyter teams
+(nbstripout — outputs are derived noise). The one tradition that refuses this — the Smalltalk
+**image** — is exactly the one we record as a *boundary, not a backend*
+(`research/260614-squeak-pharo-deep-dive`).
+
+The earlier draft of this blueprint used a two-bucket framing ("canonical at the edges,
+derived in the middle"). That was wrong by omission: it had no home for **coordination-
+canonical** state, and so contradicted itself by listing curator bindings and alias tables as
+"derived/rebuildable" when a human binding manifestly cannot be rebuilt. The three-state model
+fixes that crack (review finding A-1) and makes `derived = f(canonical)` *literally* true.
 
 Three consequences fall straight out, and they are the spine of the rest of this document:
 
-1. **Graceful degradation is free.** If the derived middle is always rebuildable, a backend
+1. **Graceful degradation is free.** If the derived tier is always recomputable, a backend
    that can only be read is still a first-class participant — you just derive less from it.
 2. **Provenance is tractable.** Because shard-wiki never claims to *be* the source, every
-   derived artifact can always point back to the canonical edge it came from (union without
+   derived artifact can always point back to the canonical input it came from (union without
    erasure is a structural property, not a feature bolted on).
-3. **The system is a pure function of its inputs.** `union/index/projection = f(shards,
-   journal)`. Bugs in the middle are recoverable by rebuild; the edges are the only thing
-   that must be protected (and history protects them).
+3. **The derived tier is a pure function of canonical state.** `derived = f(sharded,
+   coordination)`. Bugs in the derived tier are recoverable by recompute; only the two
+   canonical tiers must be durably protected — sharded by each shard, coordination by the
+   Git journal (history). *Recomputability is a correctness property of the derived tier, not
+   a promise that a from-scratch rebuild is operationally cheap — see §8.4 and the
+   operational-envelope axis.*
 
 ### The dual narrow waist
 
@@ -73,7 +90,7 @@ principles fused with the research through-lines.
 | # | Invariant | Source |
 |---|-----------|--------|
 | I-1 | **Orchestrator, not engine.** Core composes shards; it never replaces or homogenises them. | INTENT Stability Note |
-| I-2 | **Canonical at the edges, derived in the middle.** Union/index/projection are rebuildable from shards + journal. | §1; Logseq/ikiwiki/GT through-line |
+| I-2 | **Three states; derived = f(canonical).** State is sharded-canonical, coordination-canonical (journal), or derived-disposable. The derived tier (union/index/projection) is a pure, recomputable function of the two canonical tiers; only canonical state is durably protected. | §1; Logseq/ikiwiki/GT through-line |
 | I-3 | **Capability-awareness is data.** A binding's abilities are a *profile* (positions on spectra), read by generic core logic — not per-backend branches. | synthesis v3 §2; INTENT capability-aware adapters |
 | I-4 | **Union without erasure.** Every page/revision/projection/overlay/view carries its provenance, freshness, liveness, and divergence. | INTENT; provenance-granularity spectrum (Wikibase) |
 | I-5 | **Overlay before mutation.** Writes to anything below write-through land as drafts/patches/MRs first; no silent remote mutation. | INTENT |
@@ -124,8 +141,11 @@ carries provenance; an authz decision at L5 records the context under which cont
 Likewise a capability profile declared at L1 is consulted at L3 (can we write-through?), L4
 (can we delegate a query?), and L5 (can this principal even reach the op?).
 
-The dependency rule is strict and downward: **L4 may be deleted and recomputed from L1–L3.**
-Nothing at L1–L3 may depend on L4 state.
+The dependency rule is strict and downward, and it tracks the **three states (§1)**, not the
+layer numbers: **the derived-disposable tier (the whole of L4) may be deleted and recomputed
+from canonical state (sharded content at L1 + coordination-canonical state in the L3 journal).**
+Nothing canonical may depend on derived state. Note the journal at L3 is *canonical* (it holds
+overlays, bindings, aliases, merges); only L4 is disposable.
 
 ---
 
@@ -146,9 +166,13 @@ parallel terms.
   (paths/shards). Addressing, equivalence, and transclusion key on identity (I-9).
 - **Provenance envelope** — the metadata wrapper every artifact carries: source shard,
   freshness, liveness, authorization context, overlay status, divergence, derivation lineage.
-- **Coordination journal** — the L3 Git-backed record of change flows for a space.
+- **Coordination journal** — the L3 Git-backed record of change flows for a space, and the
+  durable home of all **coordination-canonical** state (§1): overlays, curator equivalence
+  bindings, alias tables, merge/reconciliation decisions. This state is born inside shard-wiki,
+  exists nowhere else, and is *not* derived — it must be committed, never recomputed.
 - **Overlay** — a non-destructive local edit against a remote/read-only/limited shard,
-  representable as draft/patch/commit/MR before destructive apply.
+  representable as draft/patch/commit/MR before destructive apply. Coordination-canonical: an
+  unapplied overlay is the local truth and lives in the journal.
 - **Projection** — a derived view of shard content, typed on two axes (§8): *kind*
   (replication | derivation) × *liveness* (static … irreducibly-live).
 - **Federation model** — the selected coordination strategy for a space (§ taxonomy, T17).
@@ -264,17 +288,32 @@ All shapes reduce to a common skeleton: **`(content | source, structure, provena
 [derivation rule])`**. The page model stores the richest faithful form as canonical and treats
 any Markdown rendering of a non-Markdown shape as a *lossy projection* (I-4 + fidelity report).
 
-### 7.2 Identity, placement, addressing
+### 7.2 Identity, placement, addressing — three distinct concepts
 
-- **Identity** is a content-derived or native-id-wrapped key, *shard-scoped* so native UUIDs
-  survive projection and don't collide.
-- **Placement** is separate: one identity → N placements (paths/shards) = a DAG. No single
-  canonical path (I-9).
-- **Span addressing** — adopt native span IDs where minted (Roam `:block/uid`, Logseq `id::`,
-  Notion/CRDT UUID); else content fingerprint or path+range. The Xanadu tumbler is the ideal
-  the scheme aims at without requiring.
-- **Provenance envelope** rides on every page and span (source shard, freshness, liveness,
-  overlay status, authz context, divergence, derivation lineage).
+The earlier draft used "identity" for two different things and (worse) suggested deriving page
+identity from a content fingerprint — which would make *editing a page change its identity* and
+break every reference to it (review bug B-1). They are pulled apart here:
+
+- **Page identity — a *stable handle*.** A shard-scoped, durable key that **survives edits**:
+  the backend's native page/note id where one exists (Roam/Notion/Trilium uid, a git path
+  treated as a name, a wiki page name), wrapped in a shard scope so it survives projection and
+  never collides across shards. Identity is *assigned/minted, not computed from content*.
+  References, placement, transclusion targets, and overlays all key on identity.
+- **Placement — *where* an identity sits.** One identity → N placements (paths/shards) = a DAG;
+  no single canonical path (I-9). Placement can change without changing identity.
+- **Content equivalence — *detecting sameness*, never identity.** A **content fingerprint** (or
+  span-set overlap) identifies a *version / a piece of content*, used to detect that two
+  *distinct identities* hold the same or derived content (the equivalence/chorus mechanism,
+  §8.4). A fingerprint is never a page's identity: same page, edited → new fingerprint, **same
+  identity**; two pages, identical content → same fingerprint, **different identities**.
+- **Span addressing** — a sub-page address within an identity: adopt native span IDs where
+  minted (Roam `:block/uid`, Logseq `id::`, Notion/CRDT UUID); else a *position* address
+  (path+range) or a *content-fingerprint* address for equivalence/transclusion. The Xanadu
+  tumbler is the portable ideal the scheme aims at without requiring.
+- **Provenance envelope** rides on pages and spans (see §7.3 for its layered, low-cost form).
+
+So the chain is: **identity (stable) → placements (N, mutable) → equivalence (cross-identity
+sameness, fingerprint-based)** — three concepts, three mechanisms, never conflated.
 
 ---
 
@@ -311,11 +350,16 @@ composes per shard** (mechanism over policy, I-7):
 
 ### 8.4 Union & projection (L4) — the derived cache
 
-This whole layer is rebuildable from L1–L3 (I-2). It comprises:
+This whole layer is **derived-disposable**: recomputable from canonical state — sharded
+content + the **coordination-canonical** inputs in the journal (I-2). Crucially, the *automatic*
+equivalence results are derived, but the **human/curatorial inputs they consume — alias tables
+and curator equivalence bindings — are coordination-canonical (they live in the journal), not
+derived**; recompute reads them, never regenerates them. It comprises:
 
 - **Identity resolution & equivalence** — detect "same topic / derived content" path-
-  independently (fingerprint, span-set overlap, alias table, curator binding); present as
-  **chorus-of-voices** or designated-canonical (a *policy* preset).
+  independently from *derived* signals (content fingerprint, span-set overlap) **plus** the
+  *coordination-canonical* inputs (alias table, curator binding); present as
+  **chorus-of-voices** or designated-canonical (a *policy* preset). (Scaling: §8.7.)
 - **Union graph** — the navigable join of pages, links, and dimensions (namespace, genealogy,
   version, shard, equivalence). A *derived lens over canonical files+journal, never a new
   store* (the ZigZag boundary).
@@ -350,6 +394,136 @@ One snapshot-provenance record (run id, source rev, timestamp, environment "ungu
 serves notebooks, renders, and recordings alike. **No INTENT amendment is required** — this
 lives inside the existing page model (L2) and projection model (L4).
 
+### 8.6 Consistency, concurrency & conflict model
+
+INTENT makes real-time cross-shard consistency a non-goal — but "no strong consistency" is not
+the same as "no defined consistency." This is the guarantee shard-wiki *does* offer, and the
+mechanism (not policy) that makes concurrent editing safe (review bug B-2).
+
+**The consistency guarantee — causal, anchored on the journal:**
+
+- **Read-your-writes for coordination-canonical state.** Once an overlay/binding/merge is
+  committed to the journal, this client always sees it (the journal is the client's own causal
+  spine). This is a *strong* local guarantee, cheap because the journal is local Git.
+- **Causal consistency across the derived tier.** The union/index/projections reflect a causal
+  cut of `(sharded inputs seen so far, journal)`. Effects never appear before their causes; a
+  projection that has seen journal commit *C* has seen everything *C* depends on.
+- **Eventual convergence for sharded-canonical inputs.** Remote shard content is pulled
+  asynchronously (lazily or by notify/poll, §8.7); the union converges to each shard's latest
+  *as observed*, bounded by the shard's operational envelope. Freshness is always *shown*
+  (provenance envelope), never faked — a stale projection is labelled stale, not wrong.
+
+So: **strong + read-your-writes for what shard-wiki owns (the journal); causal for what it
+derives; eventual + freshness-labelled for what shards own.** No global clock, no distributed
+transaction, no two-phase commit across shards — none is needed, because shard-wiki coordinates
+rather than controls.
+
+**Conflict detection & representation is core mechanism; only resolution is policy (I-7).**
+The split the earlier draft elided:
+
+- **Detection (core).** Divergence is detected structurally: two identities resolve as
+  equivalent (§8.4) but their content fingerprints differ, or an overlay's base revision no
+  longer matches the shard's current revision. Detection is always on; it is never optional.
+- **Representation (core).** A detected conflict is **first-class data in the union**, not an
+  error: equivalent-but-divergent pages are presented as a **coexisting set** (the
+  chorus/keep-both representation), each fully attributed, with the divergence recorded in the
+  provenance envelope (union without erasure — a conflict is information, not a failure).
+- **Resolution (policy).** *Which* version wins, or whether they stay coexisting, is a
+  configurable preset (§10): chorus / designated-canonical / git-merge / vote-to-merge /
+  overlay-only. Core never hard-codes one.
+
+**Overlay-apply under source drift (the concurrent-write case).** An overlay carries the
+**base revision** of the shard content it was authored against. On apply, core compares base to
+the shard's current revision:
+
+- *unchanged* → apply (fast-forward), commit to journal, propagate if the profile permits;
+- *changed, non-overlapping* → three-way merge where the merge capability allows (axis 6),
+  else keep-both;
+- *changed, overlapping* → **refuse + re-present** as a conflict (above); never silently
+  clobber (I-5, no silent remote mutation). The unapplied overlay remains coordination-
+  canonical and valid against its base.
+
+**Ordering.** The journal commit is the ordering authority for coordination-canonical effects;
+a shard-native write is only *acknowledged* in the journal after the adapter confirms it, so a
+crash between journal-intent and shard-write is recoverable (the intent is replayable, the
+write is idempotent-keyed on identity+base-rev). Cross-shard operations are ordered by their
+journal commits, giving the causal cut above.
+
+**Residual open items** (tracked in *Known scaling risks & open problems*, §12, not pretended
+solved): the exact convergence bound for
+high-write CRDT shards under partition, and whether per-equivalence-set divergence needs a
+vector clock vs. a simple base-rev comparison, are deferred to implementation spikes.
+
+### 8.7 Scaling the union — incremental-first, rebuild as fallback
+
+The derived tier is *recomputable* (I-2) but recompute must never be the **operational**
+mechanism. A from-scratch rebuild reads every page of every shard — including rate-limited,
+paginated external APIs (Notion) and irreducibly-live sources — which can take hours to days
+and directly fights the operational-envelope axis (review C-2). So:
+
+**Incremental, change-driven maintenance is the primary mechanism.** Each shard's `notify`
+capability (or a poll/ETag fallback where it has none, §8.8) emits **change events**; an event
+drives a **delta update** to exactly the affected union nodes, equivalence candidates, indexes,
+and projections. The derived tier is a continuously-maintained materialised view, not a
+periodically-recomputed one. Steady-state cost is O(changes), not O(corpus).
+
+**Full rebuild is a rare, bounded fallback** — for cold start, schema/algorithm change, or
+suspected corruption — and it is **explicitly not required to be cheap**. It respects each
+shard's envelope (it may be slow, throttled, or resumable for a rate-limited shard) and runs
+*concurrently with serving the existing derived tier*; it swaps in atomically on completion.
+I-2 guarantees rebuild is *possible and correct*, not instant.
+
+**Equivalence detection is indexed, not pairwise (review C-1).** Naive fingerprint/span-set
+comparison across all pages of all shards is O(N²) and is forbidden. Instead:
+
+1. **Blocking / candidate generation** — cheap keys bucket pages that *could* be equivalent:
+   normalised title, normalised path tail, explicit alias-table entries (coordination-
+   canonical), and **MinHash/LSH bands over content shingles** for near-duplicate and
+   derived-content detection. Only within-bucket pairs are considered — turning O(N²) into
+   ≈O(N) candidates.
+2. **Verification** — candidate pairs are confirmed by full fingerprint / span-set overlap and
+   any curator binding. Confirmed equivalences become union edges.
+3. **Incremental maintenance** — a changed page is re-bucketed and only its *new* candidate set
+   is re-verified; equivalence is maintained per-change, never recomputed globally.
+
+**The index is itself derived** (disposable, recomputable) and per-tenant-partitioned (§9).
+Its parameters (LSH band/row counts, shingle size, precision/recall) are tunable; the accepted
+**false-negative rate of blocking** is a known, tracked limitation (§12) — blocking trades a
+small miss rate for tractability, and curator bindings are the escape hatch for misses.
+
+### 8.8 Cache freshness & invalidation
+
+Replication-projection caches remote shard content; cache invalidation is the actual hard part
+and was missing from the first draft (review C-2). The protocol is **per-binding, driven by the
+capability profile**, with one rule: **freshness is always represented, never assumed** — every
+cached page's provenance envelope carries `(observed-at, source-rev-if-known, staleness-state)`,
+so a consumer can always tell live from stale.
+
+**Three invalidation modes, chosen by capability, not hard-coded:**
+
+| Mode | When | Mechanism |
+|------|------|-----------|
+| **Event-driven (push)** | shard has `notify` | a change event invalidates exactly the affected entries and enqueues a delta refresh (§8.7); the preferred mode |
+| **Validator poll** | shard exposes ETag / Last-Modified / rev | conditional fetch (`If-None-Match`); cheap "still fresh?" checks without transferring bodies |
+| **TTL** | shard offers neither | time-bounded staleness; the floor mode (Oddmuse-class shards) |
+
+Most real bindings are **hybrid**: event-driven for invalidation + a long TTL as a safety net
+for missed events + validator polls on read when an entry is past a soft age.
+
+**Operational-envelope coupling.** The mode is constrained by axis-10: a **rate-limited** shard
+(Notion) *must* favour event-driven + long TTL and *must not* poll per-read — the freshness
+policy is capability-gated like everything else. A local file shard can watch the filesystem
+(near-instant invalidation, effectively event-driven for free).
+
+**Thundering-herd / coalescing.** Concurrent reads of the same stale entry trigger a **single
+in-flight refresh** (single-flight); other readers await it or are served the stale-but-labelled
+value per policy. Bulk invalidations (a shard-wide event) are **batched and rate-shaped** to the
+shard's envelope rather than fired as N concurrent fetches.
+
+**Staleness is a policy knob, not a correctness bug.** Whether a reader gets *stale-but-fast* or
+*blocks-for-fresh* is a §10 preset (per space or per request); either way the envelope tells the
+truth about what was served. This is union-without-erasure applied to time.
+
 ---
 
 ## 9. Cross-cut — Authorization (L5)

workplans/SHARD-WP-0005-architecture-hardening.md

@@ -11,6 +11,7 @@ created: "2026-06-15"
 updated: "2026-06-15"
 depends_on:
   - SHARD-WP-0002
+state_hub_workstream_id: "7d277f39-f78f-4998-8cab-875f2bf69763"
 ---
 
 # SHARD-WP-0005 — Core architecture hardening
@@ -51,8 +52,9 @@ folded in) plus a new **"Known scaling risks & open problems"** section.
 
 ```task
 id: SHARD-WP-0005-T1
-status: todo
+status: done
 priority: high
+state_hub_task_id: "0bb17223-b458-41a4-b86c-c36b2689cf81"
 ```
 
 Fix finding **A-1** (and its I-2 contradiction). Replace the two-bucket thesis with **three
@@ -70,8 +72,9 @@ out of L4-rebuildable into the coordination tier.
 
 ```task
 id: SHARD-WP-0005-T2
-status: todo
+status: done
 priority: high
+state_hub_task_id: "d76a6b07-5157-426c-9343-a3e7efedf4a9"
 ```
 
 Fix bug **B-1**. Separate two concepts §7.2/§8.4 conflate: **page identity** = a *stable
@@ -87,8 +90,9 @@ sameness).
 
 ```task
 id: SHARD-WP-0005-T3
-status: todo
+status: done
 priority: high
+state_hub_task_id: "fb91f43f-3bf0-41f5-ad0a-bfd15a7fad17"
 ```
 
 Fix bug **B-2**. Add a new section stating shard-wiki's **consistency guarantee** (choose and
@@ -105,8 +109,9 @@ residual as open (→ T9).
 
 ```task
 id: SHARD-WP-0005-T4
-status: todo
+status: done
 priority: high
+state_hub_task_id: "eb885f03-2569-4ade-b3aa-174bc6062672"
 ```
 
 Fix scaling findings **C-1, C-2**. Make **incremental, change-driven maintenance the primary
@@ -126,6 +131,7 @@ operational expectation).
 id: SHARD-WP-0005-T5
 status: todo
 priority: medium
+state_hub_task_id: "b031715c-00d0-4a0a-99f1-4e74a236d04b"
 ```
 
 Fix finding **C-2 (invalidation)**. Design the replication-projection **freshness/invalidation
@@ -142,6 +148,7 @@ operational-envelope axis (rate-limited shards favour event-driven + long TTL).
 id: SHARD-WP-0005-T6
 status: todo
 priority: medium
+state_hub_task_id: "f04ce101-0d95-4e1a-ab8b-80dfff9d2dda"
 ```
 
 Fix elegance finding **D-1**. Identify a **smaller orthogonal core** of capability axes and
@@ -159,6 +166,7 @@ implied-position rules. Update §6.
 id: SHARD-WP-0005-T7
 status: todo
 priority: medium
+state_hub_task_id: "b84b790f-d208-4a76-af9f-1402a6a87ac1"
 ```
 
 Fix findings **D-2, D-3, D-4** together (the structural elegance/efficiency cluster):
@@ -181,6 +189,7 @@ Fix findings **D-2, D-3, D-4** together (the structural elegance/efficiency clus
 id: SHARD-WP-0005-T8
 status: todo
 priority: medium
+state_hub_task_id: "6ce3584f-ffa1-4568-8175-460adc8e2083"
 ```
 
 Fix findings **B-3, C-3**:
@@ -200,6 +209,7 @@ Fix findings **B-3, C-3**:
 id: SHARD-WP-0005-T9
 status: todo
 priority: medium
+state_hub_task_id: "c876a63c-4cfd-4794-be29-1d8226643c82"
 ```
 
 Close out (finding **F**). Add a new **"Known scaling risks & open problems"** section listing