artifact-store/workplans/ARTIFACT-STORE-WP-0004-s3-compatible-backend.md

id, type, title, repo, domain, status, owner, topic_slug, planning_priority, planning_order, created, updated, state_hub_workstream_id

id	type	title	repo	domain	status	owner	topic_slug	planning_priority	planning_order	created	updated	state_hub_workstream_id
ARTIFACT-STORE-WP-0004	workplan	S3-Compatible Backend (Ceph RGW Target)	artifact-store	stack	active	codex	stack	medium	4	2026-05-15	2026-05-16	d0526cfc-e532-431f-970d-f3e548d27a80

ARTIFACT-STORE-WP-0004: S3-Compatible Backend

Purpose

Add a second concrete storage backend that speaks the S3 protocol. Validated targets: Ceph RGW (primary self-hosted production target), MinIO (dev / CI), AWS S3 (interop check). The backend must satisfy the storage SPI without any leaks of S3-specific concepts into the registry.

Constraints

• storage.spi.StorageBackend Protocol from WP-0001 is the contract.
• No S3 vocabulary leaks into registry.* or api.*.
• docs/ARCHITECTURE-BLUEPRINT.md storage-backend section.

Prerequisites

• WP-0001 done (SPI exists, local backend exists as a reference).

D4.1 - Configuration Surface

id: ARTIFACT-STORE-WP-0004-T001
status: done
priority: high
state_hub_task_id: "1db0d548-cdac-4b07-962b-bcafa3aae30e"

Acceptance:

• s3 backend configuration accepts: endpoint_url, region, bucket, key_prefix, access_key_ref, secret_key_ref, storage_class, sse (optional), multipart_threshold_bytes, multipart_chunk_bytes.
• Credential references resolve from env vars or mounted files; never from request bodies.
• Documented Ceph RGW configuration example checked in under docs/OPERATOR.md.

D4.2 - S3 Backend Implementation

id: ARTIFACT-STORE-WP-0004-T002
status: done
priority: high
state_hub_task_id: "14b50595-5820-4369-b037-b015fcbddcc4"

Acceptance:

• storage.backends.s3.S3Backend implements the SPI using aioboto3 or aiobotocore (decision recorded in the workplan; whichever is better-maintained at implementation time).
• Object key layout <key_prefix>/<digest_algorithm>/<hex[0:2]>/<hex[2:4]>/<hex>.
• put uses multipart for objects above the configured threshold.
• get supports Range.
• head, delete, health implemented.
• delete is idempotent (delete-of-missing returns success).

Decision: use aioboto3 as the optional S3 client dependency. The backend imports it lazily so local-only deployments do not need S3 dependencies installed.

D4.3 - Backend Selection And Routing

id: ARTIFACT-STORE-WP-0004-T003
status: done
priority: medium
state_hub_task_id: "725dafd6-3337-4f81-b221-bb9f3a564d7e"

Acceptance:

• A registry can have multiple backends configured; package creation records which backend a file is stored in.
• Per-package backend selection rule: configurable function of retention_class + producer; default routes everything to a single backend.
• storage_locations.backend_id reflects the actual storage.

D4.4 - Test Strategy: MinIO In CI, RGW As Documented Manual Smoke

id: ARTIFACT-STORE-WP-0004-T004
status: blocked
priority: high
state_hub_task_id: "4fd7b73b-7058-4edd-b5e3-edca396760d4"

Acceptance:

• Integration tests run against MinIO via testcontainers-python (or a docker-compose fixture if testcontainers fights the WSL2 environment).
• A documented manual procedure tests against a real Ceph RGW endpoint; results recorded in docs/OPERATOR.md.
• No CI dependency on a live Ceph or AWS account.

Blocked note: Docker is available, but this environment does not have aioboto3, boto3, testcontainers, uv, or pip; MinIO container tests need dependency/bootstrap support before they can be run honestly.

D4.5 - Verification Pass

id: ARTIFACT-STORE-WP-0004-T005
status: done
priority: medium
state_hub_task_id: "5a55546f-288f-4da0-a646-3d9319908279"

Acceptance:

• artifactstore storage verify --backend s3 re-reads every object in the backend, recomputes its primary digest, and emits v1.storage.location_verified events.
• Mismatches are reported as failed locations and surfaced via the health endpoint.

Success criteria

• The same package ingestion flow that worked against local in WP-0001 works unchanged against s3.
• Switching backend by config — without code changes in the registry or API layers — is the smoke test.