coulomb/artifact-store
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs+plans: reconcile blueprint with ambition, add ADRs, sequence workplans

Browse Source 
Aligns the v1 architecture with the longer-horizon platform thesis so we can
start implementation without the schema-level inconsistencies the prior
review surfaced.

ADRs (docs/adr/0001..0006): content-addressed dual-digest storage, append-only
event log as source of truth, canonical CBOR manifests, control/data-plane
contract, v1 tech stack (Python 3.12 / uv / FastAPI / SQLAlchemy Core +
asyncpg / Alembic / cbor2 / blake3 / ruff / mypy / pytest / typer), OCI
compatibility kept reachable.

Architecture blueprint rewritten to v2: library-first (ffmpeg-shaped) module
layout, materialised-view data model over the event log, upload-session and
event-stream endpoints pinned, retrieval tiering promoted into the schema.

Roadmap added (docs/ROADMAP.md) with three phases. WP-0001 rewritten as the
Foundation plan (scaffold + kernels + local FS + minimal app). WP-0002..0005
created carrying the existing state_hub_task_ids forward semantically:
ingestion API (T004), retention lifecycle (T005), S3-compatible backend
(T006), guide-board pilot (T007). T001/T002/T003/T008 remain in WP-0001
with refined acceptance.

README and AGENTS.md refreshed to reflect the new repo shape.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-05-15 21:16:17 +02:00
parent 403d903585
commit 747afc27a6
16 changed files with 1761 additions and 404 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
docs/ROADMAP.md Normal file
View File

@@ -0,0 +1,93 @@
# Roadmap
Status: living document
Updated: 2026-05-15
The roadmap sequences `artifact-store` from "no code" to a credible
production v1 to the longer-horizon platform shape recorded in
`docs/PLATFORM-AMBITION.md`. Each row is a self-contained workplan with
its own acceptance criteria; nothing here is a binding milestone.
The sequencing principle is **library-first** (ffmpeg-shaped):
foundational kernels and contracts before any consumer code. The HTTP
server and CLI exist only after the core library can be exercised
end-to-end against a local filesystem backend.
## Phase 0 — Cleanup (done 2026-05-15)
- ADR-0001 through ADR-0006 accepted.
- Architecture blueprint rewritten to v2.
- Platform ambition and assembly experiment documented.
- Workplans re-sequenced.
## Phase 1 — Foundation and pilot (v0.1)
Goal: ingest a real guide-board run end-to-end, against a local
filesystem backend, with retention applied and events logged.
| ID | Title | Carries existing task IDs | Notes |
|---|---|---|---|
| WP-0001 | Foundation: scaffold, core kernels, local FS backend | T001, T002, T003, T008 | All of the library-shaped modules; no HTTP API yet beyond `/health`. |
| WP-0002 | Ingestion API + manifest surface | T004 | The HTTP API. Builds on WP-0001's library. |
| WP-0003 | Retention lifecycle | T005 | Retention engine, extensions, holds, deletion eligibility. |
| WP-0004 | S3-compatible backend (Ceph RGW target) | T006 | Second concrete adapter. |
| WP-0005 | Guide-board pilot ingestion | T007 | First real producer wired up. |
Exit criteria for v0.1: WP-0001 through WP-0005 done; a guide-board
CMIS run round-trips through artifact-store with manifest, retention,
and Statehub linkage; backend swappable between local FS and an
S3-compatible store.
## Phase 2 — Production hardening (v0.2 – v0.3)
| ID | Title | Notes |
|---|---|---|
| WP-0006 | Garbage collection + reference counting | Required by ADR-0001 global dedup. Mark-eligible already lands in WP-0003; this workplan does the byte-deletion pass. |
| WP-0007 | Resumable / chunked upload implementation | The wire shape lands in WP-0002; this workplan makes the implementation actually streaming. |
| WP-0008 | Auth, multi-tenancy, quota | OIDC integration; tenant namespacing; per-tenant rate limit and storage quota. |
| WP-0009 | Observability: metrics, tracing, structured logs | OpenTelemetry SDK; latency / throughput SLOs published. |
| WP-0010 | Event stream out (CDC) | NATS or Kafka topic of registry events; long-poll `/events` becomes a fallback. |
| WP-0011 | Signed manifests | Sigstore / cosign integration; signature recorded alongside manifest digest. |
Exit criteria for v0.3: a deployment is operatable by humans without
internal knowledge; SLOs are measurable; access is authenticated;
artifacts can be signed and verified.
## Phase 3 — Platform features (v0.4 – v1.0)
| ID | Title | Notes |
|---|---|---|
| WP-0012 | OCI artifact `/v2/` endpoint | Implements OCI Distribution Spec on top of the same storage (ADR-0006). |
| WP-0013 | Content-defined chunking + global dedup at chunk level | FastCDC; chunked storage. Builds toward `docs/ASSEMBLY-EXPERIMENT.md`. |
| WP-0014 | Rust data plane extraction | Move `dataplane.inproc` to `dataplane.remote` (ADR-0004). |
| WP-0015 | WASM plugin host | Extension surface for indexers, redactors, scorecard generators. |
| WP-0016 | Cold-tier adapters | Glacier / Tape / IA classes; restore flow. |
| WP-0017 | Federation and replication | Signed manifest exchange between artifact-store instances. |
Exit criteria for v1.0: artifact-store is embeddable as a library, runs
as a single-binary CLI, runs as a server, speaks OCI, federates between
instances, and is fast enough to be a credible commercial substrate.
## What this roadmap deliberately does NOT promise
- Specific calendar dates. Cadence is set by sessions, not quarters.
- A UI. UIs are out-of-tree (see `docs/PLATFORM-AMBITION.md`).
- ML-specific or container-specific features. Use OCI compatibility.
- A storage backend for every cloud. Adapters are community surface.
## How to add a workplan
1. Pick the next free `ARTIFACT-STORE-WP-NNNN` number.
2. Create `workplans/ARTIFACT-STORE-WP-NNNN-<slug>.md` with the
frontmatter and task block format in `AGENTS.md`.
3. Cite the ADRs the workplan depends on in its `## Constraints`
section.
4. Append a row to the appropriate phase table in this file.
5. Notify the custodian operator to run
`make fix-consistency REPO=artifact-store`.
## How to retire a workplan
1. Set `status: done` in the frontmatter when all tasks are `done`.
2. Move the file to `workplans/archived/YYMMDD-ARTIFACT-STORE-WP-NNNN-<slug>.md`.
3. Update this roadmap to reflect the new state.