Add S3 backend and storage verification

docs/OPERATOR.md

@@ -54,6 +54,19 @@ All settings are prefixed with ``ARTIFACTSTORE_`` and read by
 | `ARTIFACTSTORE_API_TOKEN`         | empty                                         | Default bearer token used by HTTP-backed CLI commands. |
 | `ARTIFACTSTORE_RETENTION_CONFIG_PATH` | empty                                     | Optional TOML file overriding retention-class default durations. |
 | `ARTIFACTSTORE_RETENTION_SWEEP_INTERVAL_SECONDS` | `3600`                         | Default interval for external schedulers that invoke the retention sweeper. |
+| `ARTIFACTSTORE_STORAGE_BACKENDS`  | `local`                                      | Comma-separated backend IDs to configure (`local`, `s3`). |
+| `ARTIFACTSTORE_STORAGE_DEFAULT_BACKEND` | `local`                                | Backend used when no routing rule matches. |
+| `ARTIFACTSTORE_STORAGE_BACKEND_ROUTES` | empty                                  | Comma-separated `producer:retention_class=backend_id` rules; `*` is a wildcard. |
+| `ARTIFACTSTORE_S3_ENDPOINT_URL`   | empty                                        | S3-compatible endpoint URL for Ceph RGW / MinIO / AWS S3. |
+| `ARTIFACTSTORE_S3_REGION`         | `us-east-1`                                  | S3 signing region. |
+| `ARTIFACTSTORE_S3_BUCKET`         | empty                                        | Bucket/container for artifact objects. |
+| `ARTIFACTSTORE_S3_KEY_PREFIX`     | empty                                        | Optional object-key prefix before `<algorithm>/<hex...>`. |
+| `ARTIFACTSTORE_S3_ACCESS_KEY_REF` | empty                                        | Access key reference, `env:NAME` or `file:/mounted/path`. |
+| `ARTIFACTSTORE_S3_SECRET_KEY_REF` | empty                                        | Secret key reference, `env:NAME` or `file:/mounted/path`. |
+| `ARTIFACTSTORE_S3_STORAGE_CLASS`  | empty                                        | Optional storage class sent on writes. |
+| `ARTIFACTSTORE_S3_SSE`            | empty                                        | Optional server-side encryption value, e.g. `AES256`. |
+| `ARTIFACTSTORE_S3_MULTIPART_THRESHOLD_BYTES` | `67108864`                    | Multipart threshold for the S3 backend. |
+| `ARTIFACTSTORE_S3_MULTIPART_CHUNK_BYTES` | `8388608`                       | Multipart part size for the S3 backend. |
 
 See [`.env.example`](../.env.example) for the canonical template.
 
@@ -131,8 +144,48 @@ Objects are addressed by content (`blake3:<hex>`) and laid out as
 <root>/<algorithm>/<hex[0:2]>/<hex[2:4]>/<hex>
 ```
 
-with atomic writes (tmpfile + fsync + rename). The S3-compatible backend
-lands in WP-0004.
+with atomic writes (tmpfile + fsync + rename).
+
+### S3-compatible backend
+
+The `s3` backend targets Ceph RGW first, with MinIO as the development
+stand-in and AWS S3 as an interoperability check. Install the optional S3
+dependency before enabling it:
+
+```sh
+uv sync --all-extras --extra s3
+```
+
+Ceph RGW example:
+
+```sh
+export ARTIFACTSTORE_STORAGE_BACKENDS=local,s3
+export ARTIFACTSTORE_STORAGE_DEFAULT_BACKEND=s3
+export ARTIFACTSTORE_STORAGE_BACKEND_ROUTES='guide-board:release-evidence=s3,*:*=local'
+export ARTIFACTSTORE_S3_ENDPOINT_URL=https://rgw.example.internal
+export ARTIFACTSTORE_S3_REGION=us-east-1
+export ARTIFACTSTORE_S3_BUCKET=artifact-store
+export ARTIFACTSTORE_S3_KEY_PREFIX=prod/artifact-store
+export ARTIFACTSTORE_S3_ACCESS_KEY_REF=env:ARTIFACTSTORE_RGW_ACCESS_KEY
+export ARTIFACTSTORE_S3_SECRET_KEY_REF=file:/run/secrets/artifactstore-rgw-secret
+export ARTIFACTSTORE_S3_STORAGE_CLASS=STANDARD
+export ARTIFACTSTORE_S3_SSE=AES256
+```
+
+Manual smoke against Ceph RGW:
+
+```sh
+artifactstore health
+artifactstore push ./fixtures/smoke \
+  --producer guide-board \
+  --subject rgw-smoke \
+  --retention-class release-evidence
+artifactstore storage verify --backend s3
+```
+
+The verification command re-reads stored objects, recomputes the primary
+digest, emits `v1.storage.location_verified`, and marks failed locations as
+`failed`. A nonzero failed-location count degrades `/health`.
 
 ## CLI reference
 
@@ -147,6 +200,7 @@ lands in WP-0004.
 | `artifactstore push <dir>` | Push a directory through the HTTP API and finalize the package. |
 | `artifactstore manifest <package_id>` | Fetch the JSON manifest projection through the HTTP API. |
 | `artifactstore retention sweep` | Run one deletion-eligibility sweep against the configured DB. |
+| `artifactstore storage verify --backend <id>` | Re-read stored objects for a backend and record verification events. |
 
 The CLI is a thin client over `artifactstore.registry.Registry`
 (see [ADR-0005](adr/0005-v1-tech-stack.md)).