diff --git a/history/260615-core-architecture-blueprint-review-2.md b/history/260615-core-architecture-blueprint-review-2.md new file mode 100644 index 0000000..100d9e5 --- /dev/null +++ b/history/260615-core-architecture-blueprint-review-2.md @@ -0,0 +1,91 @@ +# Critical review (round 2) — CoreArchitectureBlueprint.md (hardened) + +Date: 2026-06-15 · Reviewer: tegwick (with Claude) · Subject: +`spec/CoreArchitectureBlueprint.md` after **SHARD-WP-0005** (commit f21b7b5) · Feeds: +**SHARD-WP-0006** + +A second hostile pass over the *hardened* blueprint. Round 1 found design bugs; this round +finds (a) self-consistency regressions the surgical hardening introduced, and (b) deeper +second-order gaps — some of which the hardening *sharpened*. Verdict: the architecture is now +substantially sound; what remains in §B/§C are hard distributed-systems/operational questions, +not design smells — except §A (a real regression) and three foundational gaps in §B. + +--- + +## A. Self-consistency regressions introduced by surgical hardening + +The 9 edits deepened §6–§9 but did not propagate to the **overview surfaces**, so the document +now contradicts itself between its summary and its body (and readers trust the summary). + +- **A-1 (real contradiction).** §4 still says "Addressing, **equivalence**, and transclusion + key on identity" — the exact conflation T2 fixed in §7.2 (equivalence keys on *content + fingerprint across distinct identities*). → **WP-0006 T1** +- **A-2.** §4 "Projection — typed on two axes" and §4 "Provenance envelope … every artifact + carries [full wrapper]" are stale vs T7's §8.4 (two-axis = extension point; trivial default) + and §7.3 (layered effective-vs-own). → **T1** +- **A-3.** §10 policy surface omits knobs the hardening added (freshness/staleness §8.8, + squash-compaction §8.1, conflict-resolution preset §8.6, tenant-partition) — yet §11 defines + `policy/` as "owns the §10 surface." The module contract points at a stale list. → **T1** +- **A-4 (cosmetic).** §3 diagram + §11 header still say "L4 rebuildable cache" / "15 spectra," + advertising the pre-hardening model (§8.7 incremental-first; §6.5 orthogonal-core). → **T1** + +Meta-point: surgical editing hardened the body but regressed whole-document coherence; v2 +needs an **overview-reconciliation pass**. + +## B. Foundational gaps (serious; some sharpened by the hardening) + +- **B-1 — The journal is now a concurrent-write DB, but it's single-writer Git.** §8.6's + consistency model assumes "the journal is local Git, read-your-writes." L4 multi-tenant + the + L6 Orchestrator API imply a server; HA/scale implies *multiple* instances. Concurrent commits + of coordination-canonical state to one git journal = lock contention / merge races; Git is not + a concurrent-write store. Either single-writer-per-space (an unstated HA ceiling) or a real + concurrent coordination store with Git as an *export*. **T1 of WP-0005 worsened this** by + loading more canonical state into the journal. The keystone unanswered question. → **T2** +- **B-2 — Capability-as-data trusts self-reported profiles with no conformance check.** I-3 + + §6.5's degradation contract assume the profile tells the truth. A buggy adapter (claims + `merge=git/text`, corrupts; claims `notify`, never emits) silently poisons every degradation + decision. No **adapter conformance suite** (declared profile == observed behavior) exists. + Foundational for an architecture whose correctness rests on profile accuracy. → **T3** +- **B-3 — "Coordination-canonical state in the journal" has no representation design.** T1 + relocated overlays/bindings/aliases/equivalence-sets/merges into "the journal" without saying + *how* Git stores structured mutable state. "All equivalences touching X" over a git-of-files + is O(scan) unless indexed — and an index is L4/derived. The new central concept is a black + box; resolve *with* B-1. → **T2** +- **B-4 — Incremental equivalence is under-specified/likely incorrect; I-2 only eventually + true.** §8.7 re-verifies a changed page's *new* candidate set but not the pairs it *leaves* + (a page exiting an LSH bucket can break an existing equivalence edge); the delta is not + additive. Deeper: incremental maintenance drifts from `f(canonical)`, so I-2 holds only + eventually, guaranteed solely by an expensive reconcile-against-rebuild. Needs a stated + verification mechanism (background checker / digest-vs-sampled-rebuild). → **T4** + +## C. Real but second-tier (track as open problems O-8…O-11) + +- **C-1 — Mechanism-over-policy → operator burden; no preset bundles.** ~7 knob families with + sub-modes and interactions; only authz (L0–L4) bundles into personas. Need named bundles + ("personal vault" / "team wiki" / "enterprise federation"). → **O-8 / T5** +- **C-2 — Tenant partitioning (I-13) vs shard sharing + lazy projection.** A shard in two roots + is cached twice → duplicate storage + double refresh on rate-limited backends. Shard + exclusive-to-one-root or shareable? Unresolved. → **O-9 / T5** +- **C-3 — Span-level authz + transclusion is an unmodeled leak path.** Authz is per + page/shard/tenant; transclusion crosses shards at span granularity → a page can leak a span + past its ACL (aggregation/inference). §7.3's ⊕ also stops being simple two-level inheritance + across a transclusion boundary. → **O-10 / T5** +- **C-4 — Union-under-unavailability undefined.** Freshness covers *stale*, nothing covers + *down*. The dead-shard read path (partial union? error? last-known?) is unspecified though + it's the commonest real failure. → **O-11 / T5** + +## D. Recommended resolution (→ SHARD-WP-0006) + +1. **§A reconciliation** (T1) — make the overview match the hardened body. +2. **Journal & coordination-state model** (T2) — settle single-vs-multi-writer and separate the + **coordination-state store** from the **content-history journal**. Likely resolution: + **event-sourced coordination** — an append-only *decision log* is the coordination-canonical + tier (git-addressable, I-6 preserved); the queryable current state (alias table, equivalence + set) is a *derived fold* of the log (disposable). Append-logs tolerate concurrency far better + than mutable-file Git; state a concurrency model. Resolves **B-1 + B-3** together. +3. **Adapter conformance suite** (T3) — make a passing conformance run part of the contract + (B-2): every adapter proves declared profile == observed behavior. +4. **Incremental correctness + I-2 verification** (T4) — fix the leaving-bucket re-verification + and propagation; add a background consistency-checker / derived-tier digest so I-2 is + verifiable, not merely asserted (B-4). +5. **Track §C** (T5) — O-8…O-11 with chosen direction + revisit trigger; close-out. diff --git a/history/README.md b/history/README.md index 7ab1b17..f1b9f39 100644 --- a/history/README.md +++ b/history/README.md @@ -14,4 +14,5 @@ design evolution. | Date | Record | Subject | |------|--------|---------| -| 2026-06-15 | `260615-core-architecture-blueprint-review.md` | Critical review of `spec/CoreArchitectureBlueprint.md` (commit 9b5b393); inputs to `SHARD-WP-0005` | \ No newline at end of file +| 2026-06-15 | `260615-core-architecture-blueprint-review.md` | Critical review of `spec/CoreArchitectureBlueprint.md` (commit 9b5b393); inputs to `SHARD-WP-0005` | +| 2026-06-15 | `260615-core-architecture-blueprint-review-2.md` | Round-2 review of the hardened blueprint (post-`SHARD-WP-0005`, f21b7b5); inputs to `SHARD-WP-0006` | \ No newline at end of file diff --git a/workplans/SHARD-WP-0006-architecture-hardening-2.md b/workplans/SHARD-WP-0006-architecture-hardening-2.md new file mode 100644 index 0000000..4b7123e --- /dev/null +++ b/workplans/SHARD-WP-0006-architecture-hardening-2.md @@ -0,0 +1,168 @@ +--- +id: SHARD-WP-0006 +type: workplan +title: "core architecture hardening II (round-2 review fixes)" +domain: whynot +repo: shard-wiki +status: active +owner: tegwick +topic_slug: whynot +created: "2026-06-15" +updated: "2026-06-15" +depends_on: + - SHARD-WP-0005 +--- + +# SHARD-WP-0006 — Core architecture hardening II + +## Goal + +Resolve the round-2 review (`history/260615-core-architecture-blueprint-review-2.md`): +reconcile the document's overview with its hardened body (§A), settle the three foundational +gaps (§B: journal concurrency + coordination-state representation; adapter conformance; +incremental-maintenance correctness + I-2 verification), and track the second-tier items +(§C) as open problems. Deliverable: a revised `spec/CoreArchitectureBlueprint.md` that is +**self-consistent** and answers the keystone question — *single- vs multi-writer journal, and +where mutable coordination state actually lives*. + +## Context + +- Round-2 review: `history/260615-core-architecture-blueprint-review-2.md` (A-1…A-4, B-1…B-4, + C-1…C-4; resolution §D). +- Round-1 review + first hardening: `history/260615-core-architecture-blueprint-review.md`, + `SHARD-WP-0005`. +- Constraint to honour: INTENT I-6 / "Git-addressable coordination layer" — any coordination + store must remain git-addressable (history/patches/review/backup), not abandon Git. + +**Non-goal:** Implement anything. Architecture spec revision only. + +--- + +## §A overview-reconciliation pass + +```task +id: SHARD-WP-0006-T1 +status: todo +priority: high +``` + +Make the overview match the hardened body (review A-1…A-4): + +- **A-1 (bug):** §4 "Page identity vs placement" — remove "equivalence … key on identity"; + state equivalence keys on content fingerprint across identities (align with §7.2). +- **A-2:** §4 "Projection" → trivial-default + extension-point framing (align §8.4); §4 + "Provenance envelope" → layered effective-vs-own (align §7.3). +- **A-3:** §10 policy surface — add the missing knobs (freshness/staleness mode, history + squash-compaction, conflict-resolution preset, tenant-partition); keep §11's `policy/` + pointer accurate. +- **A-4:** §3 diagram + §11 header — "incremental-maintained derived tier (rebuild = fallback)" + and "orthogonal-core capability profile," not "rebuildable cache / 15 spectra." + +--- + +## Journal & coordination-state model (single vs multi-writer) + +```task +id: SHARD-WP-0006-T2 +status: todo +priority: high +``` + +Settle the keystone (review B-1 + B-3 together). Decide and document: + +- **Separate the content-history journal from the live coordination state.** Proposed + resolution: **event-sourced coordination** — an append-only **decision log** (overlays, + bindings, aliases, merges as events) is the **coordination-canonical** tier and is + **git-addressable** (preserves I-6); the queryable *current* coordination state (effective + alias table, equivalence set) is a **derived fold** of the log (disposable, indexable, L4). + This resolves B-3 (representation = event log; queries = derived index) and refines the §1 + three-state framing (coordination-canonical = the decision *log*, not a mutable file). +- **State the concurrency model (B-1).** How multiple orchestrator instances append safely: + serialized append per space (with the HA story made explicit) and/or + conflict-free ordering of independent appends; what "read-your-writes" means across + instances; where Git is authoritative vs a materialised export. +- Update §1 (refine coordination-canonical = decision log), §4 (Coordination journal / + Overlay), §8.1/§8.2, §8.6 (consistency now grounded on the log + a stated concurrency model), + §11 (a `coordination/` event-log store + derived-fold index). + +--- + +## Adapter conformance suite (profiles must be verified, not asserted) + +```task +id: SHARD-WP-0006-T3 +status: todo +priority: high +``` + +Fix B-2. Make capability-as-data trustworthy: add to the adapter contract (§6) a +**conformance-suite requirement** — a published battery that, given a binding, verifies its +**declared capability profile matches observed behaviour** (each declared verb/position is +exercised; mismatches fail registration). State that an adapter is only admissible if it +passes; that the suite is versioned with the contract; and that profile claims are +**verified, never self-asserted** (this is what makes I-3 / §6.5 sound). Update §6.4/§6.5 and +add the conformance obligation to the acceptance posture. + +--- + +## Incremental-maintenance correctness & I-2 verification + +```task +id: SHARD-WP-0006-T4 +status: todo +priority: medium +``` + +Fix B-4. In §8.7: correct the incremental equivalence algorithm — a changed page must +**re-verify the candidate pairs it leaves** (bucket exits) and **propagate** to pages whose +equivalence depended on it, not only its new candidates; the delta is not additive. Add an +**I-2 verification mechanism**: a background consistency-checker comparing a **digest of the +derived tier** against a sampled/periodic recompute, so `derived = f(canonical)` is *verified*, +not merely asserted; on mismatch, recompute the affected partition. State that I-2 is an +*eventually-verified* property with a named checker, and note the residual in §12. + +--- + +## Track second-tier items; close-out + +```task +id: SHARD-WP-0006-T5 +status: todo +priority: medium +``` + +Add review §C as open problems in §12 (each with chosen direction + revisit trigger): + +- **O-8** preset bundles ("personal vault" / "team wiki" / "enterprise federation") over the + §10 policy surface — direction: ship a small set of named personas; trigger: operators + mis-configuring interacting knobs. +- **O-9** shard sharing across roots vs tenant partition (I-13) — direction: shard exclusive to + one root by default, explicit shared-read binding otherwise; trigger: a shard legitimately + needed live in two tenants. +- **O-10** span-level authz under transclusion (aggregation/inference leak) + ⊕ composition + across transclusion boundaries — direction: a transcluded span inherits the *stricter* of + source and host authz; provenance ⊕ composes source-page-envelope under the host; trigger: a + cross-authz transclusion in practice. +- **O-11** union-under-shard-unavailability — direction: partial union + per-shard + "unavailable" provenance + last-known-projection where policy allows; trigger: SLA needs on + partial reads. + +Refresh §14 decisions and §16 traceability (link this review + SHARD-WP-0006); refresh +invariants if any added. Final `check_repo_consistency` pass. + +--- + +## Acceptance criteria + +- Every round-2 finding A-1…A-4, B-1…B-4 is **resolved in the blueprint**; C-1…C-4 are tracked + as O-8…O-11 with a chosen direction (none silently dropped). +- The blueprint is **self-consistent**: overview (§1–§5, §10) agrees with body (§6–§9). +- The **journal/coordination-state model** answers single-vs-multi-writer and preserves I-6 + (git-addressable coordination). +- The **adapter conformance** obligation makes capability-as-data (I-3) verified, not asserted. +- Each task committed; SCOPE/spec-README updated where status changes; state-hub synced. + +## Suggested task order + +T1 (reconcile, fast) → T2 (keystone journal/coordination model) → T3 (conformance) → +T4 (incremental correctness) → T5 (track §C + close-out).