feat(gems): three-pass schema migration aligning state-hub with GEMS
Implements CUST-WP-0007. Resolves inconsistencies I-1, I-2, I-5, I-6
identified in the GEMS audit (GenericEntityModellingSystem.md).
Pass 1 (e1f2a3b4c5d6): domain_id FK on extension_points and
technical_debt (replaces raw string column); repo_id FK on contributions.
Fixes domain-filtering bugs in EP/TD dashboard pages.
Pass 2 (f2a3b4c5d6e7): repo_id nullable FK on workstreams, aligning
the GEMS primary attachment with ADR-001 (repo > topic). Dashboard
pages updated to prefer repo->domain over topic->domain.
Pass 3 (a3b4c5d6e7f8): SBOMSnapshot container entity (GEMS Complex
between Repository and SBOMEntry). Ingest is now additive — each call
creates a new snapshot; history is retained. List/report endpoints
filter to latest snapshot per repo via _latest_snapshot_ids_subquery().
New endpoints: GET /sbom/snapshots/, GET /sbom/snapshots/{id}/.
Dashboard gains a Snapshot History section.
Also adds GEMS analysis artefacts: wiki/GEMS-StateHub-TypeRegistry.md,
wiki/GEMS-StateHub-SWOT.md, workplans/CUST-WP-0006 (analysis),
workplans/CUST-WP-0007 (migration, now completed).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
190
workplans/CUST-WP-0006-gems-state-hub.md
Normal file
190
workplans/CUST-WP-0006-gems-state-hub.md
Normal file
@@ -0,0 +1,190 @@
|
||||
---
|
||||
id: CUST-WP-0006
|
||||
type: workplan
|
||||
title: GEMS Analysis & State-Hub Migration Planning
|
||||
domain: custodian
|
||||
status: active
|
||||
owner: custodian
|
||||
topic_slug: the-custodian
|
||||
created: 2026-03-02
|
||||
updated: 2026-03-02
|
||||
state_hub_workstream_id: "7ce13282-d534-492a-8d42-b3a134028823"
|
||||
---
|
||||
|
||||
# CUST-WP-0006 — GEMS Analysis & State-Hub Migration Planning
|
||||
|
||||
## Purpose
|
||||
|
||||
Apply the Generic Entity Modelling System (GEMS) — documented in
|
||||
`wiki/GenericEntityModellingSystem.md` — to the state-hub data store.
|
||||
|
||||
This workplan covers:
|
||||
1. A domain-specific GEMS implementation skeleton (type registry + hierarchy)
|
||||
2. An audit of current model inconsistencies against GEMS principles
|
||||
3. Escalation of all non-trivial structural decisions
|
||||
4. A SWOT analysis of migrating the state-hub data store to the GEMS model
|
||||
5. (Deferred) A refined migration workplan, once key decisions are resolved
|
||||
|
||||
Companion documents:
|
||||
- `wiki/GEMS-StateHub-TypeRegistry.md` — canonical type registry
|
||||
- `wiki/GEMS-StateHub-SWOT.md` — SWOT analysis
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — GEMS Skeleton Definition
|
||||
|
||||
### Task T01: Map current state-hub entities to GEMS types
|
||||
|
||||
```task
|
||||
id: T01
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
**Deliverable:** `wiki/GEMS-StateHub-TypeRegistry.md`
|
||||
|
||||
See companion document. Summary:
|
||||
|
||||
| Current Table | GEMS Kind | GEMS Primary | Notes |
|
||||
|---|---|---|---|
|
||||
| `Domain` | Complex | Ecosystem (implicit root) | 6 canonical domains |
|
||||
| `Topic` | Complex | Domain | Focus area / active project |
|
||||
| `ManagedRepo` | Complex | Domain | Managed git repo |
|
||||
| `Workstream` | Complex | **Repository** (currently Topic) | Work package — ADR-001 mismatch |
|
||||
| `SBOMSnapshot` | Complex | Repository | Does not yet exist as an entity |
|
||||
| `Task` | Atom | Workstream | ✓ correct |
|
||||
| `Decision` | Atom | Repository (currently Topic or Workstream) | Dual-attach ambiguity |
|
||||
| `TechnicalDebt` | Atom | Repository (currently domain: str) | String FK inconsistency |
|
||||
| `ExtensionPoint` | Atom | Repository (currently domain: str) | String FK inconsistency |
|
||||
| `Contribution` | Atom | Repository (no domain FK) | No domain affiliation |
|
||||
| `ProgressEvent` | Atom | Workstream (or Topic) | Multi-attach ambiguity |
|
||||
| `SBOMEntry` | Atom | SBOMSnapshot (currently ManagedRepo) | No container |
|
||||
| `WorkstreamDependency` | Relation | Domain | Flat junction table |
|
||||
|
||||
---
|
||||
|
||||
### Task T02: Inconsistency audit — current model vs GEMS
|
||||
|
||||
```task
|
||||
id: T02
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
**Identified inconsistencies:**
|
||||
|
||||
**I-1 — String domain field (high severity)**
|
||||
`ExtensionPoint.domain` and `TechnicalDebt.domain` are `String(50)` columns, not FKs
|
||||
to `domains.id`. The rename_domain API patches these manually via string updates —
|
||||
there is no referential integrity. Dashboard filtering silently returns empty results
|
||||
when slugs drift.
|
||||
|
||||
**I-2 — Workstream primary container is Topic, not Repository (critical severity)**
|
||||
GEMS §7 places `Workstream.primary = Repository`. ADR-001 states that workplans
|
||||
(the file backing a workstream) must originate in a repository. However, the current
|
||||
schema has `Workstream.topic_id NOT NULL` — Topic is the enforced primary container.
|
||||
This is an ADR-001 violation embedded in the schema itself. There is currently no
|
||||
`repo_id` on Workstream.
|
||||
|
||||
**I-3 — Decision dual attachment without clear hierarchy (medium severity)**
|
||||
`Decision` has both `topic_id` and `workstream_id` FKs, with a CHECK constraint
|
||||
requiring at least one. GEMS requires exactly one primary attachment. The current model
|
||||
allows ambiguous "where does this Decision live" answers.
|
||||
|
||||
**I-4 — ManagedRepo has a nullable `topic_id` FK (low-medium severity)**
|
||||
`ManagedRepo.topic_id` is a nullable FK to `topics`. In GEMS, Repository is a Complex
|
||||
whose primary is Domain, not Topic. The topic_id on Repo suggests a second, conflicting
|
||||
hierarchy path.
|
||||
|
||||
**I-5 — No SBOMSnapshot container entity (medium severity)**
|
||||
SBOM entries are flat rows tagged with `repo_id` and `snapshot_at`. GEMS §7 defines
|
||||
`SBOM (Complex, primary=Repository)` as an organizer. Without this container, it is
|
||||
impossible to query "all packages in snapshot X" as a first-class concept, or to model
|
||||
snapshot-to-snapshot diffs.
|
||||
|
||||
**I-6 — Contribution has no domain or repository FK (medium severity)**
|
||||
`Contribution` has only optional `related_topic_id` and `related_workstream_id`. There
|
||||
is no direct link to Domain or Repository, making domain-scoped contribution queries
|
||||
fragile.
|
||||
|
||||
**I-7 — No Ecosystem root entity (low severity)**
|
||||
GEMS §3.2 requires at least one root Complex. The current model has no explicit root —
|
||||
Domain is the de-facto root but is not declared as such. This matters when you want
|
||||
to express cross-domain relations or system-level policies.
|
||||
|
||||
**I-8 — WorkstreamDependency as flat junction table (low severity)**
|
||||
GEMS §4 defines Relations as first-class entities whose primary is a Complex. The
|
||||
current `WorkstreamDependency` is a flat table. For the current usage this works, but
|
||||
it makes contextual queries (e.g. "all dependencies within domain X") less uniform.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Decision Escalation
|
||||
|
||||
### Task T03: Register non-trivial decisions in state-hub
|
||||
|
||||
```task
|
||||
id: T03
|
||||
status: done
|
||||
priority: critical
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
Six decisions were escalated (see state-hub records):
|
||||
- DEC-GEMS-001: GEMS implementation architecture (typed tables vs. generic entity model)
|
||||
- DEC-GEMS-002: Workstream primary container — Topic vs. Repository
|
||||
- DEC-GEMS-003: Domain string → FK migration for ExtensionPoint and TechnicalDebt
|
||||
- DEC-GEMS-004: SBOMSnapshot container entity
|
||||
- DEC-GEMS-005: Ecosystem root entity
|
||||
- DEC-GEMS-006: WorkstreamDependency as first-class Relation entity
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — SWOT Analysis
|
||||
|
||||
### Task T04: Produce SWOT analysis document
|
||||
|
||||
```task
|
||||
id: T04
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
**Deliverable:** `wiki/GEMS-StateHub-SWOT.md`
|
||||
|
||||
See companion document for the full analysis. Summary verdict:
|
||||
|
||||
The migration is **worth pursuing incrementally** (Pattern C from GEMS §9). The most
|
||||
impactful and least risky first move is:
|
||||
1. Fix I-1: migrate domain string → FK on EP/TD (low risk, high consistency gain)
|
||||
2. Fix I-2: add `repo_id` to Workstream (medium risk, fixes ADR-001 alignment)
|
||||
3. Add SBOMSnapshot container (medium risk, enables snapshot diffing)
|
||||
|
||||
Full generic entity table architecture (Option A) is deferred until after the typed-table
|
||||
alignment is stable and validated.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Migration Workplan Refinement (deferred)
|
||||
|
||||
### Task T05: Write detailed migration workplan (CUST-WP-0007)
|
||||
|
||||
```task
|
||||
id: T05
|
||||
status: blocked
|
||||
priority: high
|
||||
assignee: custodian
|
||||
blocking_reason: "Blocked on decisions DEC-GEMS-001 through DEC-GEMS-006"
|
||||
```
|
||||
|
||||
Once the six decisions are resolved, produce `workplans/CUST-WP-0007-gems-migration.md`
|
||||
covering:
|
||||
- Schema migrations (Alembic versions)
|
||||
- Data backfill scripts
|
||||
- API router changes
|
||||
- MCP tool changes
|
||||
- Dashboard updates
|
||||
- ADR-001 workplan file format updates
|
||||
270
workplans/CUST-WP-0007-gems-migration.md
Normal file
270
workplans/CUST-WP-0007-gems-migration.md
Normal file
@@ -0,0 +1,270 @@
|
||||
---
|
||||
id: CUST-WP-0007
|
||||
type: workplan
|
||||
title: GEMS Migration — Three-Pass State-Hub Alignment
|
||||
domain: custodian
|
||||
status: completed
|
||||
owner: custodian
|
||||
topic_slug: the-custodian
|
||||
repo_slug: the-custodian
|
||||
created: 2026-03-02
|
||||
updated: 2026-03-02
|
||||
state_hub_workstream_id: "22e18151-fc83-438c-b732-10e056e64a20"
|
||||
---
|
||||
|
||||
# CUST-WP-0007 — GEMS Migration: Three-Pass State-Hub Alignment
|
||||
|
||||
Implements the migration decided in CUST-WP-0006. Fixes all structural
|
||||
inconsistencies identified in the GEMS audit (I-1 through I-6) in three
|
||||
independently releasable passes.
|
||||
|
||||
Decisions resolved: DEC-GEMS-001 (Option C), DEC-GEMS-002, DEC-GEMS-003,
|
||||
DEC-GEMS-004. DEC-GEMS-005 and DEC-GEMS-006 deferred.
|
||||
|
||||
---
|
||||
|
||||
## Pass 1 — Fix Domain FK Inconsistencies
|
||||
|
||||
**Scope:** Resolve I-1 and I-6. No API breaking changes. Fixes observable
|
||||
dashboard domain-filtering bugs on EP/TD pages.
|
||||
|
||||
**Alembic migration ID:** `e1f2a3b4c5d6` (down_revision: `d3e4f5a6b7c8`)
|
||||
|
||||
### Task T01: Alembic migration — Pass 1
|
||||
|
||||
```task
|
||||
id: T01
|
||||
status: done
|
||||
priority: critical
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
Operations:
|
||||
1. Add `domain_id` UUID FK (nullable) to `extension_points`
|
||||
2. Add `domain_id` UUID FK (nullable) to `technical_debt`
|
||||
3. Add `repo_id` UUID FK (nullable) to `contributions`
|
||||
4. Backfill `extension_points.domain_id` from `domains.slug` match
|
||||
5. Backfill `technical_debt.domain_id` from `domains.slug` match
|
||||
6. Make `domain_id` NOT NULL on both tables (all rows must be backfilled first)
|
||||
7. Drop `domain` String column from `extension_points`
|
||||
8. Drop `domain` String column from `technical_debt`
|
||||
|
||||
### Task T02: Update ExtensionPoint and TechnicalDebt models
|
||||
|
||||
```task
|
||||
id: T02
|
||||
status: done
|
||||
priority: critical
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
Replace `domain: Mapped[str]` with `domain_id: Mapped[uuid.UUID]` FK +
|
||||
`domain: Mapped["Domain"]` relationship. Add domain_slug property.
|
||||
|
||||
### Task T03: Update Contribution model
|
||||
|
||||
```task
|
||||
id: T03
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
Add `repo_id: Mapped[uuid.UUID | None]` nullable FK to `managed_repos`.
|
||||
|
||||
### Task T04: Update EP and TD routers
|
||||
|
||||
```task
|
||||
id: T04
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
- Filter by `domain_id` FK instead of domain string
|
||||
- Accept `domain` slug in create/filter params, resolve to `domain_id`
|
||||
|
||||
### Task T05: Update MCP EP/TD tools
|
||||
|
||||
```task
|
||||
id: T05
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
`register_extension_point` and `register_technical_debt` still accept
|
||||
`domain` as a slug string. Router resolves to `domain_id` FK.
|
||||
|
||||
### Task T06: Update EP/TD dashboard pages
|
||||
|
||||
```task
|
||||
id: T06
|
||||
status: done
|
||||
priority: medium
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
`extensions.md` and `techdept.md` load domain list from `/domains/` API and
|
||||
use `domain_id` FK for filtering. Remove reliance on string comparison.
|
||||
|
||||
---
|
||||
|
||||
## Pass 2 — Align Workstream with ADR-001
|
||||
|
||||
**Scope:** Resolve I-2. Breaking change to workstream schema. `topic_id`
|
||||
becomes a nullable secondary annotation; `repo_id` becomes the primary FK.
|
||||
Workplan frontmatter format gains `repo_slug` field.
|
||||
|
||||
**Alembic migration ID:** `f2a3b4c5d6e7` (down_revision: `e1f2a3b4c5d6`)
|
||||
|
||||
### Task T07: Alembic migration — Pass 2
|
||||
|
||||
```task
|
||||
id: T07
|
||||
status: done
|
||||
priority: critical
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
Operations:
|
||||
1. Add `repo_id` UUID FK (nullable) to `workstreams`
|
||||
2. Backfill `repo_id` using heuristic: workstream → topic → domain → first
|
||||
repo for that domain (adequate for current data; all custodian workstreams
|
||||
map to the-custodian repo)
|
||||
3. For topics without a repo: leave nullable (MCP tooling handles this)
|
||||
|
||||
### Task T08: Update Workstream model
|
||||
|
||||
```task
|
||||
id: T08
|
||||
status: done
|
||||
priority: critical
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
Add `repo_id: Mapped[uuid.UUID | None]` nullable FK to `managed_repos`.
|
||||
Keep `topic_id` as nullable secondary. Add `repo` relationship.
|
||||
|
||||
### Task T09: Update workstream router and MCP tools
|
||||
|
||||
```task
|
||||
id: T09
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
- `create_workstream` MCP tool: add optional `repo_id` / `repo_slug` param
|
||||
- Workstream read schema: expose `repo_id` and `repo_slug`
|
||||
- Dependency resolution in `state/summary` uses `repo.domain` when available
|
||||
|
||||
### Task T10: Update workplan frontmatter format
|
||||
|
||||
```task
|
||||
id: T10
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
Add `repo_slug` field to ADR-001 workplan frontmatter spec. Update existing
|
||||
workplan files (CUST-WP-0001 through CUST-WP-0006) to include `repo_slug`.
|
||||
|
||||
### Task T11: Update Dependencies dashboard domain resolution
|
||||
|
||||
```task
|
||||
id: T11
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
`dependencies.md` currently resolves domain via `topicMap[w.topic_id]?.domain_slug`.
|
||||
Change to prefer `wsMap[w.id]?.repo?.domain_slug` when available.
|
||||
|
||||
---
|
||||
|
||||
## Pass 3 — SBOMSnapshot Container
|
||||
|
||||
**Scope:** Resolve I-5. Adds `sbom_snapshots` as a container entity between
|
||||
Repository and SBOMEntry. Enables snapshot history and diff queries.
|
||||
|
||||
**Alembic migration ID:** `a3b4c5d6e7f8` (down_revision: `f2a3b4c5d6e7`)
|
||||
|
||||
### Task T12: Alembic migration — Pass 3
|
||||
|
||||
```task
|
||||
id: T12
|
||||
status: done
|
||||
priority: critical
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
Operations:
|
||||
1. Create `sbom_snapshots` table (id, repo_id FK, snapshot_at, source, created_at)
|
||||
2. Add `snapshot_id` UUID FK (nullable) to `sbom_entries`
|
||||
3. Backfill: for each (repo_id, snapshot_at) group in sbom_entries, create one
|
||||
sbom_snapshots row; set snapshot_id on all matching entries
|
||||
4. Make `snapshot_id` NOT NULL on `sbom_entries`
|
||||
5. Consider: drop `repo_id` from `sbom_entries` (reachable via snapshot)
|
||||
|
||||
### Task T13: Add SBOMSnapshot model
|
||||
|
||||
```task
|
||||
id: T13
|
||||
status: done
|
||||
priority: critical
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
New model `api/models/sbom_snapshot.py` with FK to managed_repos.
|
||||
|
||||
### Task T14: Update SBOMEntry model
|
||||
|
||||
```task
|
||||
id: T14
|
||||
status: done
|
||||
priority: critical
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
Add `snapshot_id` FK to `sbom_snapshots`. Update `repo` relationship to go
|
||||
via snapshot.
|
||||
|
||||
### Task T15: Update SBOM router and ingest API
|
||||
|
||||
```task
|
||||
id: T15
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
- Ingest creates/finds a snapshot record, then creates entries under it
|
||||
- New endpoints: `GET /sbom/snapshots/`, `GET /sbom/snapshots/{id}/`
|
||||
- Existing `GET /sbom/` still returns flat entries for backward compatibility
|
||||
|
||||
### Task T16: Update MCP ingest tool and SBOM resources
|
||||
|
||||
```task
|
||||
id: T16
|
||||
status: done
|
||||
priority: high
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
`ingest_sbom_tool` returns `snapshot_id` in result. New MCP resource:
|
||||
`state://sbom/snapshots/{repo_slug}`.
|
||||
|
||||
### Task T17: Update SBOM dashboard
|
||||
|
||||
```task
|
||||
id: T17
|
||||
status: done
|
||||
priority: medium
|
||||
assignee: custodian
|
||||
```
|
||||
|
||||
`sbom.md` "By Repo" section adds a snapshot history row showing ingest dates
|
||||
with package count delta from previous snapshot.
|
||||
Reference in New Issue
Block a user