coulomb/artifact-store
Archived
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs+plans: reconcile blueprint with ambition, add ADRs, sequence workplans

Browse Source 
Aligns the v1 architecture with the longer-horizon platform thesis so we can
start implementation without the schema-level inconsistencies the prior
review surfaced.

ADRs (docs/adr/0001..0006): content-addressed dual-digest storage, append-only
event log as source of truth, canonical CBOR manifests, control/data-plane
contract, v1 tech stack (Python 3.12 / uv / FastAPI / SQLAlchemy Core +
asyncpg / Alembic / cbor2 / blake3 / ruff / mypy / pytest / typer), OCI
compatibility kept reachable.

Architecture blueprint rewritten to v2: library-first (ffmpeg-shaped) module
layout, materialised-view data model over the event log, upload-session and
event-stream endpoints pinned, retrieval tiering promoted into the schema.

Roadmap added (docs/ROADMAP.md) with three phases. WP-0001 rewritten as the
Foundation plan (scaffold + kernels + local FS + minimal app). WP-0002..0005
created carrying the existing state_hub_task_ids forward semantically:
ingestion API (T004), retention lifecycle (T005), S3-compatible backend
(T006), guide-board pilot (T007). T001/T002/T003/T008 remain in WP-0001
with refined acceptance.

README and AGENTS.md refreshed to reflect the new repo shape.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-05-15 21:16:17 +02:00
parent 403d903585
commit 747afc27a6
16 changed files with 1761 additions and 404 deletions
Download Patch File Download Diff File Expand all files Collapse all files

342
workplans/ARTIFACT-STORE-WP-0001-service-baseline.md
View File

@@ -1,7 +1,7 @@
---
id: ARTIFACT-STORE-WP-0001
type: workplan
title: "Artifact Store Service Baseline"
title: "Foundation: Scaffold, Core Kernels, Local FS Backend"
repo: artifact-store
domain: stack
status: active
@@ -14,51 +14,53 @@ updated: "2026-05-15"
state_hub_workstream_id: "aebf996c-8721-4e8c-9e56-61d5e4bf8dcb"
---
# ARTIFACT-STORE-WP-0001: Artifact Store Service Baseline
# ARTIFACT-STORE-WP-0001: Foundation — Scaffold, Core Kernels, Local FS Backend
## Purpose
Implement the first usable artifact registry and storage gateway. The service
should preserve artifact packages, index their metadata, delegate bytes to a
configured storage backend, apply default retention rules, and expose stable
package identifiers that Statehub and producer repositories can link to.
Stand up the smallest credible `artifact-store` core. By the end of
this workplan, the library can ingest a directory of files into a
package, compute dual digests, write canonical-CBOR manifests, persist
state through the append-only event log, store bytes on local
filesystem, and replay materialised views from the event log. No HTTP
API yet (that lands in WP-0002); a `/health` endpoint exists so that
the dev loop has something to hit.
The first producer target is a guide-board assessment run, including OpenCMIS TCK
reports and raw assessment artifacts.
The shape is **library-first** (ffmpeg-style). HTTP server and CLI are
explicitly thin consumers of `artifactstore.registry`.
## Background
## Constraints (must satisfy)
Guide-board can already produce self-contained run directories with retention
summaries, assessment packages, raw artifacts, scorecards, and log reviews. Those
directories should not live only in `/tmp`, and committing raw evidence into
producer repositories is the wrong long-term shape.
`artifact-store` becomes the shared preservation layer:
- producers generate files,
- artifact-store registers and stores them,
- Statehub records the work outcome and links to the registry package,
- storage backends handle durable bytes.
Ceph is the likely self-hosted production backend through its S3-compatible RGW
interface, but the service must keep the backend interface generic.
## Target Architecture
```text
producer package
-> registry API
-> metadata database
-> retention policy engine
-> storage adapter
-> local filesystem or S3-compatible object storage
```
- ADR-0001 — content-addressed storage with dual digest.
- ADR-0002 — append-only event log as source of truth.
- ADR-0003 — manifest canonicalisation = canonical CBOR.
- ADR-0004 — control plane / data plane SPI named.
- ADR-0005 — v1 technology stack pinned (Python 3.12, uv, FastAPI,
SQLAlchemy Core, asyncpg, alembic, cbor2, blake3, ruff, mypy, pytest).
- ADR-0006 — OCI compatibility kept reachable.
- `docs/ARCHITECTURE-BLUEPRINT.md` data model and module layout.
## Boundary
This workplan owns the first service implementation and API contract. It does
not need to build a UI, implement cold-storage restore tiers, replace Statehub,
or provide formal records-management certification.
This workplan builds the library and a minimal `/health` endpoint. It
does NOT implement: package CRUD HTTP API (WP-0002), retention rules
beyond the seed (WP-0003), S3-compatible backend (WP-0004), guide-board
producer wiring (WP-0005), GC of unreferenced bytes (WP-0006).
## Target architecture (this workplan)
```text
artifactstore (library)
identity ──┐
manifest ──┼──> registry (orchestrator) ──> events (WAL + views)
events ───┘ │
retention (seed only) └──> dataplane.spi ──> dataplane.inproc ──> storage.spi ──> storage.backends.local
audit (view) └──> filesystem
storage.spi
dataplane.spi + inproc
api.http (just /health)
cli (just `artifactstore version`, `artifactstore migrate`, `artifactstore replay`)
```
## D1.1 - Service Scaffold And Repository Identity
@@ -71,14 +73,71 @@ state_hub_task_id: "84209430-ec3b-4c5e-924e-019c25434230"
Acceptance:
- Replace the seed README with artifact-store service instructions.
- Add a Python service scaffold with a clear package/module layout.
- Provide a local development command.
- Provide a test command.
- Keep generated artifact bytes and local databases ignored by git.
- Document required environment variables.
- `pyproject.toml` with `hatchling` build backend, pinned dependencies
per ADR-0005.
- `uv.lock` committed.
- `Makefile` exposes: `make dev`, `make test`, `make lint`, `make
type`, `make migrate`. Each target is a thin shim, no logic inline.
- `src/artifactstore/` package skeleton matches ADR-0005's layout
(empty `__init__.py` and one placeholder module per top-level
concern: `identity`, `manifest`, `events`, `retention`, `audit`,
`storage`, `dataplane`, `registry`, `api/http`, `cli`, `config`).
- `tests/{unit,integration}/conftest.py` in place.
- `.env.example` documents required environment variables:
`ARTIFACTSTORE_DATABASE_URL`, `ARTIFACTSTORE_STORAGE_LOCAL_ROOT`,
`ARTIFACTSTORE_LOG_LEVEL`.
- CI-equivalent local commands: `make lint && make type && make test`
pass on a clean checkout.
- `README.md` replaces the seed README: install with `uv sync`, run
with `make dev`, test with `make test`, links to ADRs and blueprint.
## D1.2 - Registry Data Model
## D1.2 - Digest Abstraction And Content Address
```task
id: ARTIFACT-STORE-WP-0001-T009
status: todo
priority: high
```
Acceptance:
- `identity.Digest` value type with `algorithm: str` and `hex: str`,
immutable, hashable.
- `identity.ContentAddress` — string-form `<algorithm>:<hex>` with
validating parser and emitter.
- `identity.digest_stream(reader) -> {primary: Digest, sha256: Digest}` —
single-pass dual-hash over an `AsyncIterator[bytes]`. Default primary
algorithm: `blake3`.
- Algorithm registry with `blake3` and `sha256` registered at import.
- Property test: digest over random byte sequences round-trips through
serialisation; `sha256` matches `hashlib.sha256(...).hexdigest()`;
`blake3` matches `blake3.blake3(...).hexdigest()`.
## D1.3 - Manifest Codec (Canonical CBOR + JCS Projection)
```task
id: ARTIFACT-STORE-WP-0001-T010
status: todo
priority: high
```
Acceptance:
- `manifest.Manifest` dataclass with the v1 fields enumerated in the
blueprint (`manifest_version=1`, package, files, storage_receipts,
retention_summary, provenance).
- `manifest.codec.encode(m) -> bytes` produces canonical CBOR
(RFC 8949 §4.2.2): definite-length, shortest-form integers,
sorted map keys.
- `manifest.codec.decode(b) -> Manifest`.
- `manifest.projection.jcs(m) -> bytes` produces RFC 8785 canonical
JSON.
- Property test: `decode(encode(m)) == m` for randomly-generated
manifests; `encode(decode(jcs_to_cbor(jcs(m)))) == encode(m)`.
- Manifest digest helper: `manifest_digest(m) -> ContentAddress` using
BLAKE3 over the canonical CBOR bytes.
## D1.4 - Registry Data Model And Migrations
```task
id: ARTIFACT-STORE-WP-0001-T002
@@ -89,16 +148,44 @@ state_hub_task_id: "e5249a39-46a2-4b56-813e-0339c52cd14e"
Acceptance:
- Define persistent models for artifact packages, files, storage locations,
retention rules, retention events, and audit events.
- Store package metadata as structured JSON while keeping core query fields
explicit.
- Record package lifecycle status: created, uploading, finalized, deleted, and
failed.
- Record file `sha256`, size, media type, and logical relative path.
- Add migrations or a reproducible schema initialization path.
- Alembic configured with `migrations/` directory; `alembic upgrade
head` works against both SQLite (dev) and PostgreSQL (prod).
- `events`, `artifact_packages`, `artifact_files`, `storage_locations`,
`retention_classes`, `retention_state`, `metadata_schemas` tables
match the blueprint schema.
- Seed migration populates `retention_classes` with the five v1 entries.
- A `make migrate` and `make migrate-fresh` target work end-to-end on
a clean DB.
- All schema columns required by ADR-0001 (`digest_algorithm`,
`digest_primary`, `digest_sha256`, `content_address`), ADR-0002
(full `events` table), and the blueprint's `retrieval_tier` and
`restore_status` are present.
## D1.3 - Local Filesystem Storage Backend
## D1.5 - Event Log Persistence And Replay
```task
id: ARTIFACT-STORE-WP-0001-T011
status: todo
priority: high
```
Acceptance:
- `events.write(transaction, Event)` writes one row in the given DB
transaction. Sequence numbers are assigned by the DB
(`BIGSERIAL`) and are guaranteed monotonic and gapless within a
registry instance.
- `events.tail(since_sequence) -> AsyncIterator[Event]` long-polls
the table (notify-style on PostgreSQL via `LISTEN/NOTIFY`,
poll-style on SQLite).
- `events.replay(into=ViewWriter)` rebuilds all materialised view
tables from `events` deterministically.
- Test: ingesting a fixed sequence of events, then rebuilding the
views from scratch, yields byte-identical materialised state.
- Event payloads use canonical CBOR (`manifest.codec`) so the same
bytes flow through registry → DB → tail consumer without re-encoding.
## D1.6 - Storage Adapter SPI And Local Filesystem Backend
```task
id: ARTIFACT-STORE-WP-0001-T003
@@ -109,90 +196,81 @@ state_hub_task_id: "68f9a752-0012-4cc1-8768-ec3f75295e7a"
Acceptance:
- Implement a storage adapter interface.
- Implement a local filesystem backend for development and tests.
- Store objects under deterministic package/file keys.
- Prevent path traversal and accidental writes outside the configured storage
root.
- Add backend health reporting.
- Add tests for put, get, head, and delete operations.
- `storage.spi.StorageBackend` Protocol matches the blueprint.
- `storage.backends.local.LocalBackend` implements the SPI:
- Object key layout `<root>/<algo>/<hex[0:2]>/<hex[2:4]>/<hex>`.
- Atomic write via `fsync(tmpfile) + rename`.
- Path traversal rejected at the SPI boundary.
- `health()` returns disk usage and root accessibility.
- Backend registry resolves by `backend_id` string (per ADR-0004).
- Unit tests cover: put, get, head, delete, double-put idempotency,
delete-of-missing, range read.
## D1.4 - Package Ingestion API
## D1.7 - Data Plane SPI And In-Process Implementation
```task
id: ARTIFACT-STORE-WP-0001-T004
id: ARTIFACT-STORE-WP-0001-T012
status: todo
priority: high
state_hub_task_id: "e3879111-4be9-4731-8aea-15abb874f960"
```
Acceptance:
- Add endpoints to create a package, upload files, finalize a package, retrieve
package metadata, list packages, and download files.
- Compute file hashes server-side during ingestion.
- Reject duplicate logical paths within one package unless explicitly replacing
a non-finalized file.
- Produce a package manifest after finalization.
- Add API tests covering successful ingestion and validation failures.
- `dataplane.spi.DataPlane` Protocol matches ADR-0004.
- `dataplane.inproc.InProcessDataPlane` implements all five operations
on top of a configured `StorageBackend`.
- `ingest_stream` computes both digests in a single pass, writes to
the backend keyed by the primary content address, and returns an
`IngestResult` containing both digests, size, and the
`StorageReceipt`.
- `serve_object` and `verify_object` re-read bytes through the
backend; `verify_object` re-digests and returns mismatches if any.
- Lint rule (or test): no code outside `dataplane.*` imports
`storage.backends.*` directly.
## D1.5 - Retention Baseline
## D1.8 - Registry Orchestrator (Library Surface)
```task
id: ARTIFACT-STORE-WP-0001-T005
id: ARTIFACT-STORE-WP-0001-T013
status: todo
priority: high
state_hub_task_id: "2d6cbd83-c348-45ad-a223-7870a3412225"
```
Acceptance:
- Seed default retention classes for transient, raw-evidence, summary-evidence,
release-evidence, and permanent-record.
- Apply a default `expires_at` when a package is created or finalized.
- Add endpoints to extend retention and apply or release holds.
- Record retention changes as retention events and audit events.
- Expose deletion eligibility without deleting bytes automatically in the first
implementation.
- `registry.Registry` exposes: `create_package`, `ingest_file`,
`finalize_package`, `get_manifest_bytes` (CBOR + JCS), `get_file`,
`tail_events`. Plus stubs for the retention operations that lighten
WP-0003.
- Each mutating operation is one DB transaction that writes events
AND updates materialised views.
- Finalisation writes one `v1.package.finalized` event whose payload
*is* the canonical CBOR manifest, and stamps `manifest_digest` on
`artifact_packages`.
- Duplicate `relative_path` within one not-yet-finalised package is
rejected unless an explicit replace is requested.
- Integration test: end-to-end ingest of a 3-file package against
local backend → finalize → read manifest → verify digests
→ tail events → replay rebuilds identical state.
## D1.6 - S3-Compatible Backend Design Hook
## D1.9 - Minimal HTTP App And CLI
```task
id: ARTIFACT-STORE-WP-0001-T006
id: ARTIFACT-STORE-WP-0001-T014
status: todo
priority: medium
state_hub_task_id: "7b980a55-2364-48c3-98ac-081629a8d2b7"
```
Acceptance:
- Define configuration fields for an S3-compatible backend.
- Keep the adapter contract compatible with Ceph RGW.
- Add an implementation stub or feature-flagged backend if dependencies are not
ready.
- Document expected Ceph/S3 configuration without requiring a live Ceph service
for baseline tests.
- `api.http.app` is a FastAPI app with one route: `GET /health`
reporting registry liveness, DB connectivity, and backend health.
- `cli` exposes `artifactstore version`, `artifactstore migrate`,
`artifactstore replay`, `artifactstore health`.
- `make dev` starts the API on `127.0.0.1:8000` with SQLite +
local FS backend by default.
## D1.7 - Guide-Board Pilot Ingestion
```task
id: ARTIFACT-STORE-WP-0001-T007
status: todo
priority: high
state_hub_task_id: "eb822821-353c-4cd2-95bf-acb2f084b7ea"
```
Acceptance:
- Provide a CLI helper or documented curl flow to register a guide-board run
directory as one package.
- Preserve guide-board run metadata: run id, target profile, assessment profile,
evidence result counts, finding counts, source commits, and report paths.
- Ingest the CMIS pilot run shape, including scorecard and log-review reports.
- Return a package id suitable for recording in Statehub.
- Add a fixture-based test that does not require the real OpenCMIS TCK.
## D1.8 - Operator Documentation And Handoff
## D1.10 - Operator Documentation And ADR Cross-Linking
```task
id: ARTIFACT-STORE-WP-0001-T008
@@ -203,27 +281,33 @@ state_hub_task_id: "9b60036c-61f2-4c22-ad31-7213473d42d0"
Acceptance:
- Document local run, test, and package ingestion commands.
- Document retention behavior and extension flow.
- Document the boundary between artifact-store and Statehub.
- Include a dev-agent handoff section listing the first implementation order.
- Keep architecture docs aligned with the implemented API.
- `README.md` updated with current run / test / migrate commands.
- `AGENTS.md` "Current Repo Shape" section reflects the scaffold.
- An `docs/OPERATOR.md` page documents environment variables, local
vs PostgreSQL setup, replay command, and a smoke-test recipe.
- Every ADR is cross-linked from at least one of: blueprint, this
workplan, or `OPERATOR.md`.
## Suggested Implementation Order
## Suggested implementation order
1. Service scaffold, test harness, and README.
2. Metadata models and local database setup.
3. Local filesystem storage adapter.
4. Package create/upload/finalize/download API.
5. Retention defaults, extension, hold, and audit events.
6. Guide-board run ingestion helper.
7. S3-compatible backend configuration and Ceph notes.
1. T001 — scaffold and tooling (no other task can start without this).
2. T009 — digest abstraction (unblocks T010, T012).
3. T010 — manifest codec (unblocks T013).
4. T002 — schema and migrations (unblocks T011, T013).
5. T011 — event log + replay.
6. T003 — storage SPI + local backend.
7. T012 — data plane SPI + in-process impl.
8. T013 — registry orchestrator.
9. T014 — minimal HTTP app and CLI.
10. T008 — docs.
## First Pilot Success Criteria
## Success criteria
- A completed guide-board CMIS run can be ingested from a local directory.
- The package manifest lists every stored file with SHA-256 and size.
- The registry returns a stable package id.
- Files can be downloaded through the service.
- Default retention is visible and can be extended.
- Statehub can record the package id and summary without storing artifact bytes.
- `make dev && make test` round-trips on a clean checkout.
- A scripted integration test ingests a directory of fixture files,
finalises the package, reads the manifest, downloads each file, and
verifies digests end-to-end against the local backend.
- Replaying events from sequence 1 reproduces the materialised view
state byte-for-byte.
- The library can be imported and exercised without an HTTP server
running (embedding test).