generated from coulomb/repo-seed
147 lines
4.3 KiB
Markdown
147 lines
4.3 KiB
Markdown
---
|
|
id: ARTIFACT-STORE-WP-0004
|
|
type: workplan
|
|
title: "S3-Compatible Backend (Ceph RGW Target)"
|
|
repo: artifact-store
|
|
domain: stack
|
|
status: done
|
|
owner: codex
|
|
topic_slug: stack
|
|
planning_priority: medium
|
|
planning_order: 4
|
|
created: "2026-05-15"
|
|
updated: "2026-05-17"
|
|
state_hub_workstream_id: "d0526cfc-e532-431f-970d-f3e548d27a80"
|
|
---
|
|
|
|
# ARTIFACT-STORE-WP-0004: S3-Compatible Backend
|
|
|
|
## Purpose
|
|
|
|
Add a second concrete storage backend that speaks the S3 protocol.
|
|
Validated targets: Ceph RGW (primary self-hosted production target),
|
|
MinIO (dev / CI), AWS S3 (interop check). The backend must satisfy
|
|
the storage SPI without any leaks of S3-specific concepts into the
|
|
registry.
|
|
|
|
## Constraints
|
|
|
|
- `storage.spi.StorageBackend` Protocol from WP-0001 is the contract.
|
|
- No S3 vocabulary leaks into `registry.*` or `api.*`.
|
|
- `docs/ARCHITECTURE-BLUEPRINT.md` storage-backend section.
|
|
|
|
## Prerequisites
|
|
|
|
- WP-0001 done (SPI exists, local backend exists as a reference).
|
|
|
|
## D4.1 - Configuration Surface
|
|
|
|
```task
|
|
id: ARTIFACT-STORE-WP-0004-T001
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "1db0d548-cdac-4b07-962b-bcafa3aae30e"
|
|
```
|
|
|
|
Acceptance:
|
|
|
|
- `s3` backend configuration accepts: `endpoint_url`, `region`,
|
|
`bucket`, `key_prefix`, `access_key_ref`, `secret_key_ref`,
|
|
`storage_class`, `sse` (optional), `multipart_threshold_bytes`,
|
|
`multipart_chunk_bytes`.
|
|
- Credential references resolve from env vars or mounted files; never
|
|
from request bodies.
|
|
- Documented Ceph RGW configuration example checked in under
|
|
`docs/OPERATOR.md`.
|
|
|
|
## D4.2 - S3 Backend Implementation
|
|
|
|
```task
|
|
id: ARTIFACT-STORE-WP-0004-T002
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "14b50595-5820-4369-b037-b015fcbddcc4"
|
|
```
|
|
|
|
Acceptance:
|
|
|
|
- `storage.backends.s3.S3Backend` implements the SPI using `aioboto3`
|
|
or `aiobotocore` (decision recorded in the workplan; whichever is
|
|
better-maintained at implementation time).
|
|
- Object key layout
|
|
`<key_prefix>/<digest_algorithm>/<hex[0:2]>/<hex[2:4]>/<hex>`.
|
|
- `put` uses multipart for objects above the configured threshold.
|
|
- `get` supports `Range`.
|
|
- `head`, `delete`, `health` implemented.
|
|
- `delete` is idempotent (delete-of-missing returns success).
|
|
|
|
Decision: use `aioboto3` as the optional S3 client dependency. The
|
|
backend imports it lazily so local-only deployments do not need S3
|
|
dependencies installed.
|
|
|
|
## D4.3 - Backend Selection And Routing
|
|
|
|
```task
|
|
id: ARTIFACT-STORE-WP-0004-T003
|
|
status: done
|
|
priority: medium
|
|
state_hub_task_id: "725dafd6-3337-4f81-b221-bb9f3a564d7e"
|
|
```
|
|
|
|
Acceptance:
|
|
|
|
- A registry can have multiple backends configured; package creation
|
|
records which backend a file is stored in.
|
|
- Per-package backend selection rule: configurable function of
|
|
`retention_class` + producer; default routes everything to a single
|
|
backend.
|
|
- `storage_locations.backend_id` reflects the actual storage.
|
|
|
|
## D4.4 - Test Strategy: MinIO In CI, RGW As Documented Manual Smoke
|
|
|
|
```task
|
|
id: ARTIFACT-STORE-WP-0004-T004
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "4fd7b73b-7058-4edd-b5e3-edca396760d4"
|
|
```
|
|
|
|
Acceptance:
|
|
|
|
- Integration tests run against MinIO via `testcontainers-python`
|
|
(or a docker-compose fixture if testcontainers fights the WSL2
|
|
environment).
|
|
- A documented manual procedure tests against a real Ceph RGW
|
|
endpoint; results recorded in `docs/OPERATOR.md`.
|
|
- No CI dependency on a live Ceph or AWS account.
|
|
|
|
Closure note: the S3 backend implementation and local verification
|
|
for artifact-store are complete. MinIO-specific compatibility,
|
|
testcontainers/bootstrap, and community-fork assessment have been moved
|
|
to ARTIFACT-STORE-WP-0007 so this backend workstream can close without
|
|
hiding the remaining external-platform work.
|
|
|
|
## D4.5 - Verification Pass
|
|
|
|
```task
|
|
id: ARTIFACT-STORE-WP-0004-T005
|
|
status: done
|
|
priority: medium
|
|
state_hub_task_id: "5a55546f-288f-4da0-a646-3d9319908279"
|
|
```
|
|
|
|
Acceptance:
|
|
|
|
- `artifactstore storage verify --backend s3` re-reads every object in
|
|
the backend, recomputes its primary digest, and emits
|
|
`v1.storage.location_verified` events.
|
|
- Mismatches are reported as `failed` locations and surfaced via the
|
|
health endpoint.
|
|
|
|
## Success criteria
|
|
|
|
- The same package ingestion flow that worked against `local` in
|
|
WP-0001 works unchanged against `s3`.
|
|
- Switching backend by config — without code changes in the registry
|
|
or API layers — is the smoke test.
|