diff --git a/workplans/SHARD-WP-0009-git-decision-log.md b/workplans/SHARD-WP-0009-git-decision-log.md new file mode 100644 index 0000000..4c1f8d3 --- /dev/null +++ b/workplans/SHARD-WP-0009-git-decision-log.md @@ -0,0 +1,101 @@ +--- +id: SHARD-WP-0009 +type: workplan +title: "git-backed DecisionLog + per-space append authority" +domain: whynot +repo: shard-wiki +status: active +owner: tegwick +topic_slug: whynot +created: "2026-06-15" +updated: "2026-06-15" +depends_on: + - SHARD-WP-0007 +--- + +# SHARD-WP-0009 — git-backed DecisionLog + per-space append authority + +## Goal + +Back the keystone with real storage: swap the in-memory `DecisionLog` (SHARD-WP-0007 T5) for a +**git-addressable, append-only** log with a **per-space append authority (lease)**, exactly as +`CoreArchitectureBlueprint` §8.1 specifies and `SHARD-WP-0006` T2 decided (resolves review +B-1/B-3). The log API and the derived fold are unchanged; only storage + the concurrency model +become real. This makes coordination-canonical state git-addressable (history/patch/review/ +backup — I-6) and read-your-writes correct across processes, not just in-process. + +**Non-goal:** distributed multi-node leasing beyond a single coordinator; per-space log +sharding (blueprint O-12). Single append authority per space is the target. + +## Context + +- Keystone spec: `CoreArchitectureBlueprint.md` §8.1; decision `SHARD-WP-0006` T2. +- Current impl: `src/shard_wiki/coordination/decision_log.py` (in-memory; same API to preserve). + +--- + +## Git event-store backend (append = commit/object) + +```task +id: SHARD-WP-0009-T1 +status: todo +priority: high +``` + +A git-backed `DecisionLog` storage backend: each space is a ref/branch (or an objects stream); +`append` writes an immutable event object and advances the ref; `events(space)` reads the +ordered log; events serialize deterministically (stable JSON). Preserve the in-memory backend +as the test/default double behind the same interface. Tests: append→git→read round-trips; +ordering preserved; deterministic serialization. + +## Per-space append authority (lease) + +```task +id: SHARD-WP-0009-T2 +status: todo +priority: high +``` + +A **single append authority per space**: a lease/leader abstraction that serializes appends so +the log is a total order (the ordering authority §8.6 relies on). Non-holders forward append +intents to the holder; the lease is time-bounded and re-grantable (HA: a failed holder is +replaced and resumes from the log head). Tests: concurrent append attempts serialize; lease +hand-off resumes from head; a partitioned non-holder cannot fork the log. + +## Fold over the git log + read-your-writes across processes + +```task +id: SHARD-WP-0009-T3 +status: todo +priority: high +``` + +`fold` reads the git log into `CoordinationState` (unchanged semantics); verify **read-your- +writes across separate processes/handles** against the same space ref. Tests: process A appends, +process B (new handle) sees it; fold equals the in-memory fold for the same event sequence. + +## Migration + wiring + +```task +id: SHARD-WP-0009-T4 +status: todo +priority: medium +``` + +Make `InformationSpace` use the git backend (configurable; in-memory remains for tests). A +one-time importer replays an in-memory/JSON log into git. Update SCOPE; `pytest` + pyflakes +green; no behavioural change to overlay/union (same fold). + +--- + +## Acceptance criteria + +- `DecisionLog` git backend passes the same tests as the in-memory one (API/fold parity). +- Appends are totally ordered per space via a single append authority; read-your-writes holds + across processes; a non-holder never forks the log. +- Coordination-canonical state is git-addressable (history/patch/review/backup). +- `pytest` + pyflakes green; each task committed; state-hub synced. + +## Suggested task order + +T1 git store → T2 append authority → T3 cross-process fold → T4 migration + wiring. diff --git a/workplans/SHARD-WP-0010-derived-views.md b/workplans/SHARD-WP-0010-derived-views.md new file mode 100644 index 0000000..809e412 --- /dev/null +++ b/workplans/SHARD-WP-0010-derived-views.md @@ -0,0 +1,109 @@ +--- +id: SHARD-WP-0010 +type: workplan +title: "derived views — wikilinks, BackLinks, RecentChanges, AllPages/SiteMap" +domain: whynot +repo: shard-wiki +status: active +owner: tegwick +topic_slug: whynot +created: "2026-06-15" +updated: "2026-06-15" +depends_on: + - SHARD-WP-0007 +--- + +# SHARD-WP-0010 — Derived views + +## Goal + +Implement the **core derived views** over the union (`FederationRequirements` ADR-03): the +wikilink/red-link model (ADR-06), **BackLinks** (the strongest core view), **RecentChanges**, +and **AllPages / SiteMap**. All are derived-tier (recomputable; incrementally maintainable +later by SHARD-WP-0011) and carry provenance. Presentation stays out of core (L6). + +**Non-goal:** full-text Search (hybrid delegate-or-derive — a later workplan); ranking; a UI. + +## Context + +- Views decision: `FederationRequirements.md` ADR-03; links: ADR-06. Blueprint §8.4. +- Builds on the union (`src/shard_wiki/union/`) + coordination journal. + +--- + +## Wikilink + red-link model + +```task +id: SHARD-WP-0010-T1 +status: todo +priority: high +``` + +A CommonMark **wikilink extension** parser (`[[Target]]` / `[[Target|label]]`) that extracts +link targets from a page body and resolves each via the union (ADR-01): resolved → link; +unresolved → **red-link** (createable, UC-23). CamelCase auto-link off by default (opt-in per +space). The link **model + resolution is core**; rendering is L6. Tests: extraction, resolve vs +red-link, CamelCase opt-in. + +## BackLinks (core) + +```task +id: SHARD-WP-0010-T2 +status: todo +priority: high +``` + +A derived **BackLinks** index over the union link graph: for any identity, the set of pages +linking to it (UC-18). Built from T1 link extraction across attached shards; entries carry +source provenance. Tests: A→B link yields B.backlinks ⊇ {A}; red-links don't create backlinks; +chorus pages aggregate. + +## RecentChanges (core) + +```task +id: SHARD-WP-0010-T3 +status: todo +priority: medium +``` + +A **RecentChanges** view merging the coordination journal (overlay/apply/alias/fork events) +with shard change signals (mtime/`current_rev`; `notify` later), ordered, across the union +(UC-17). Each entry carries provenance (which shard / which decision). Tests: an edit and an +alias both appear, newest-first; per-shard attribution present. + +## AllPages / SiteMap (core) + +```task +id: SHARD-WP-0010-T4 +status: todo +priority: low +``` + +**AllPages** (enumerate union identities, de-duplicated by equivalence/chorus) and a simple +**SiteMap** (namespace tree from placements). Tests: union enumeration spans shards; chorus +collapses to one entry with divergence noted; sitemap reflects paths. + +## Wiring + integration + +```task +id: SHARD-WP-0010-T5 +status: todo +priority: medium +``` + +Expose `InformationSpace.backlinks(name)`, `recent_changes()`, `all_pages()`, `site_map()`. +Integration test over two shards. Update SCOPE; `pytest` + pyflakes green. + +--- + +## Acceptance criteria + +- Wikilink resolution + red-link detection work via the union (ADR-01/ADR-06). +- BackLinks, RecentChanges, AllPages, SiteMap computed over the union, provenance-carrying, + presentation-free (core only). +- Views are derived/recomputable; no canonical state introduced. +- `pytest` + pyflakes green; each task committed; state-hub synced. + +## Suggested task order + +T1 wikilinks → T2 BackLinks → T3 RecentChanges → T4 AllPages/SiteMap → T5 wiring. diff --git a/workplans/SHARD-WP-0011-incremental-union.md b/workplans/SHARD-WP-0011-incremental-union.md new file mode 100644 index 0000000..ffa73ef --- /dev/null +++ b/workplans/SHARD-WP-0011-incremental-union.md @@ -0,0 +1,104 @@ +--- +id: SHARD-WP-0011 +type: workplan +title: "incremental union maintenance + equivalence index + I-2 verification" +domain: whynot +repo: shard-wiki +status: active +owner: tegwick +topic_slug: whynot +created: "2026-06-15" +updated: "2026-06-15" +depends_on: + - SHARD-WP-0007 + - SHARD-WP-0010 +--- + +# SHARD-WP-0011 — Incremental union + equivalence index + +## Goal + +Replace direct, recompute-on-read resolution with the **incremental-first** derived tier from +`CoreArchitectureBlueprint` §8.7: change-driven delta maintenance of the union/indexes, an +**indexed equivalence** path (blocking/LSH, not O(N²)) with correct retraction/propagation +(review B-4), and the **I-2 verification** mechanism (digest + background consistency-checker). +Rebuild becomes a bounded fallback, not the operational path. + +**Non-goal:** distributed maintenance; persisted on-disk index store (in-memory derived tier is +fine for this slice). Per-tenant partitioning (I-13) is honoured structurally but multi-tenant +deployment is later. + +## Context + +- Spec: blueprint §8.7 (incremental, blocking/LSH, rebuild-as-fallback), §8.4; review B-4 + + open items O-1/O-4. +- Builds on union resolution (SHARD-WP-0007) and the views (SHARD-WP-0010) it accelerates. + +--- + +## Equivalence index: blocking + verify + +```task +id: SHARD-WP-0011-T1 +status: todo +priority: high +``` + +A candidate-generation (**blocking**) layer — normalized title/path buckets + MinHash/LSH bands +over content shingles — then **verify** (fingerprint / span-set overlap + curator bindings) to +produce equivalence edges. Replaces pairwise O(N²). Tests: near-duplicates bucket together; +unrelated pages don't; verified edges match a brute-force oracle on a small corpus. + +## Incremental maintenance (delta, not additive) + +```task +id: SHARD-WP-0011-T2 +status: todo +priority: high +``` + +Change-driven delta updates: a changed/added/removed page **re-buckets**, then (per B-4) +**retracts** edges it leaves, **adds** edges it enters, and **propagates** to equivalence +neighbours (a retraction can split a chorus set). Drives union/BackLinks/RecentChanges deltas. +Tests: add/edit/remove keep the index equal to a from-scratch rebuild; a bucket-exit retracts a +stale edge. + +## I-2 verification: digest + consistency-checker + +```task +id: SHARD-WP-0011-T3 +status: todo +priority: high +``` + +A per-partition **Merkle-style digest** of the derived tier, maintained alongside deltas, and a +**background consistency-checker** that recomputes a sampled fold and compares; mismatch → +**scoped recompute** of the affected region (self-healing). Makes `derived = f(canonical)` +*verified*, not asserted. Tests: induced drift is detected and repaired; digest stable under +equivalent event orders. + +## Wire incremental tier behind resolution + views + +```task +id: SHARD-WP-0011-T4 +status: todo +priority: medium +``` + +Route `UnionGraph.resolve` and the SHARD-WP-0010 views through the maintained index (rebuild = +explicit fallback). Behaviour is unchanged from the consumer's view; only freshness/cost change. +Update SCOPE; `pytest` + pyflakes green. + +--- + +## Acceptance criteria + +- Equivalence is indexed (blocking/LSH + verify), not pairwise; matches a brute-force oracle. +- Incremental maintenance (with retraction + propagation) keeps the derived tier equal to a + from-scratch rebuild; rebuild is a bounded fallback. +- I-2 is verified by a digest + consistency-checker that detects and self-heals drift. +- Consumer-visible resolution/views behaviour unchanged; `pytest` + pyflakes green; synced. + +## Suggested task order + +T1 equivalence index → T2 incremental maintenance → T3 I-2 verification → T4 wiring. diff --git a/workplans/SHARD-WP-0012-git-shard-adapter.md b/workplans/SHARD-WP-0012-git-shard-adapter.md new file mode 100644 index 0000000..1b2fdb5 --- /dev/null +++ b/workplans/SHARD-WP-0012-git-shard-adapter.md @@ -0,0 +1,96 @@ +--- +id: SHARD-WP-0012 +type: workplan +title: "second adapter — git-IS-store shard (contract validation on a new substrate)" +domain: whynot +repo: shard-wiki +status: active +owner: tegwick +topic_slug: whynot +created: "2026-06-15" +updated: "2026-06-15" +depends_on: + - SHARD-WP-0007 + - SHARD-WP-0008 +--- + +# SHARD-WP-0012 — git-IS-store shard adapter + +## Goal + +Add a **second adapter on a different substrate** to validate the contract beyond the plain +folder: a **git-IS-store** shard (`FederationArchitecture` T17 / TSD §A.3 — the home case where +git is the store *and* the journal). It exercises git-native history (TSD §A.5 *adopt*), +commit-as-write, `current_rev`=commit-sha drift detection, and the implication rules +(attachment=git-IS-store ⟹ substrate=git ∧ history=git-native). Proves "writing an adapter + +declaring a verified profile" is all it takes — no core changes. + +**Non-goal:** remote push/pull federation (VCS-replication+ping is a federation workplan); +merge beyond fast-forward (apply-under-drift refuse is enough, as in SHARD-WP-0008). + +## Context + +- Contract: `TechnicalSpecificationDocument.md` §A; binding taxonomy §A.3; history §A.5. +- Reuses the existing contract/conformance/overlay machinery unchanged. + +--- + +## GitShardAdapter — read over a git working tree/repo + +```task +id: SHARD-WP-0012-T1 +status: todo +priority: high +``` + +A `GitShardAdapter` reading Markdown from a git repo: keys = tracked `*.md` paths; `read` +returns a `Page` whose `source_rev` is the **commit sha** (HEAD or blob commit); profile = +git-IS-store / substrate=git / history=git-native / addressing=path, validated against the +implication rules. Tests: read tracked files; profile validates; conformance read path passes. + +## Write = commit; current_rev = sha (drift) + +```task +id: SHARD-WP-0012-T2 +status: todo +priority: high +``` + +`write(key, body)` stages + **commits** (author/message), returning the updated `Page` at the +new sha; `current_rev(key)` returns the current commit sha for drift detection. Declare `WRITE` ++ `version`. Tests: write commits; conformance positive-write probe passes; `current_rev` +changes after an external commit. + +## History adopt + integration with union/overlay + +```task +id: SHARD-WP-0012-T3 +status: todo +priority: medium +``` + +Adopt git-native history (TSD §A.5 *adopt*): expose a minimal page history (commit list for a +path) behind a `version`-gated method. Integration: attach a `GitShardAdapter` to an +`InformationSpace` alongside a `FolderAdapter` → resolve/chorus/edit/overlay/apply all work +across substrates (apply-under-drift uses sha rev). Update SCOPE; `pytest` + pyflakes green. + +--- + +## Acceptance criteria + +- `GitShardAdapter` passes the conformance suite (read + positive write) with a profile whose + implication rules hold (git-IS-store ⟹ git substrate + git-native history). +- Write = commit; `current_rev` = sha drives apply-under-drift correctly. +- The union/overlay/edit machinery works unchanged across folder + git shards (no core change — + capability-as-data proven on a second substrate). +- `pytest` + pyflakes green; each task committed; state-hub synced. + +## Notes + +If `git` plumbing via subprocess proves heavy for tests, a thin in-repo git helper (or +`tmp_path` repos created in tests) is acceptable; no new runtime dependency — use stdlib + +`git` CLI already present. + +## Suggested task order + +T1 read adapter + profile → T2 write=commit + current_rev → T3 history adopt + integration.