WP-0001-T008: operator docs + ADR cross-linking; mark WP-0001 done

docs/OPERATOR.md (new): runbook with prerequisites, quick start,
environment variables, SQLite + PostgreSQL setup, storage layout,
CLI reference, HTTP /health reference, an end-to-end Python smoke
test (create_package -> ingest_file -> finalize -> manifest), the
replay / disaster-recovery procedure, common failure modes, and a
References section that links every ADR (0001..0006), the
blueprint, platform ambition, roadmap, and assembly experiment.

README.md: refreshed to v0.1 baseline status. Quick-start uses the
real flow (uv sync, migrate-fresh, dev, /health, artifactstore health).
Make targets and CLI commands tabulated. Links docs/OPERATOR.md.

AGENTS.md: Current Repo Shape now reflects the landed scaffold +
library + CLI + HTTP app rather than "no runnable scaffold yet";
links OPERATOR.md and lists the canonical local commands.

workplans/ARTIFACT-STORE-WP-0001-service-baseline.md:
- T008 marked done.
- frontmatter status: active -> done; updated: 2026-05-16.

All ten WP-0001 tasks are now done (T001/T002/T003/T008/T009/T010/
T011/T012/T013/T014). Foundation workplan retires.

Gates: ruff clean, mypy --strict clean, 83 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-16 09:02:36 +02:00
parent fe47058e1f
commit f90c761ef6
4 changed files with 288 additions and 29 deletions

View File

@@ -162,22 +162,33 @@ To create a new workplan:
## Current Repo Shape ## Current Repo Shape
This repository is in service-baseline planning. The current sources of truth are: v0.1 baseline (WP-0001) is live: library, CLI, minimal HTTP app, local FS
backend, end-to-end ingest + finalize + replay. The pinned tech stack is in
[ADR-0005](docs/adr/0005-v1-tech-stack.md).
Sources of truth:
- `INTENT.md` — purpose, product thesis, scope, service boundary. - `INTENT.md` — purpose, product thesis, scope, service boundary.
- `SCOPE.md` — lightweight orientation. - `SCOPE.md` — lightweight orientation.
- `docs/OPERATOR.md` — runbook: env vars, DB backends, CLI / HTTP reference, smoke test, replay procedure.
- `docs/ARCHITECTURE-BLUEPRINT.md` — architecture v2: modules, data model, API shape. - `docs/ARCHITECTURE-BLUEPRINT.md` — architecture v2: modules, data model, API shape.
- `docs/PLATFORM-AMBITION.md` — longer-horizon thesis and the v1 schema commitments (A1A9). - `docs/PLATFORM-AMBITION.md` — longer-horizon thesis and the v1 schema commitments (A1A9).
- `docs/adr/` — architecture decision records ADR-0001 … ADR-0006 (content-addressed storage, event log as source of truth, canonical CBOR manifests, control/data plane contract, v1 tech stack, OCI reachability). - `docs/adr/` — architecture decision records ADR-0001 … ADR-0006 (content-addressed storage, event log as source of truth, canonical CBOR manifests, control/data plane contract, v1 tech stack, OCI reachability).
- `docs/ROADMAP.md` — workplan sequencing across phases. - `docs/ROADMAP.md` — workplan sequencing across phases.
- `docs/ASSEMBLY-EXPERIMENT.md` — opt-in research line on hand-tuned asm for hot kernels. - `docs/ASSEMBLY-EXPERIMENT.md` — opt-in research line on hand-tuned asm for hot kernels.
- `workplans/ARTIFACT-STORE-WP-0001-service-baseline.md` — Foundation workplan; first to start. - `workplans/ARTIFACT-STORE-WP-0001-service-baseline.md` — Foundation workplan (done).
- `workplans/ARTIFACT-STORE-WP-{0002..0005}-*.md` — planned next workplans. - `workplans/ARTIFACT-STORE-WP-{0002..0005}-*.md` — planned next workplans.
No runnable service scaffold exists yet. The pinned tech stack is in ADR-0005 Local commands:
(Python 3.12, uv, FastAPI, SQLAlchemy Core + asyncpg/aiosqlite, Alembic,
cbor2, blake3, ruff, mypy, pytest, typer). Add install, dev-server, and test ```sh
commands here when `ARTIFACT-STORE-WP-0001-T001` lands. make install # uv sync --all-extras
make migrate-fresh # drop + re-create the dev SQLite DB
make dev # uvicorn on 127.0.0.1:8000
make test # pytest
make lint type # ruff + mypy --strict
artifactstore health
```
## Repo Boundary ## Repo Boundary

View File

@@ -6,53 +6,78 @@ artifacts.
The registry owns artifact identity, metadata, provenance, retention The registry owns artifact identity, metadata, provenance, retention
policy, and retrieval records. Bytes are delegated to configured storage policy, and retrieval records. Bytes are delegated to configured storage
backends (local filesystem in v1, S3-compatible / Ceph RGW next). backends (local filesystem in v1, S3-compatible / Ceph RGW in WP-0004).
The shape is library-first (`artifactstore` Python package); the HTTP The shape is library-first (`artifactstore` Python package); the HTTP
server and the CLI are thin consumers. Content is addressed by digest; server and the CLI are thin consumers. Content is addressed by digest;
state is authoritative in an append-only event log; materialised views state is authoritative in an append-only event log
are rebuildable. ([ADR-0002](docs/adr/0002-event-log-source-of-truth.md)); materialised
views are rebuildable.
## Status ## Status
Scaffold landed. The core kernels and local FS backend follow in the **v0.1 (WP-0001 baseline)** — library, CLI, minimal HTTP app, local FS
remaining tasks of [WP-0001](workplans/ARTIFACT-STORE-WP-0001-service-baseline.md). backend, end-to-end ingest + finalize + replay all working. The full
package CRUD HTTP API lands in WP-0002.
## Develop ## Quick start
Requires Python ≥ 3.12 and [`uv`](https://docs.astral.sh/uv/) on the path. Requires Python ≥ 3.12 and [`uv`](https://docs.astral.sh/uv/).
```sh ```sh
make install # uv sync --all-extras uv sync --all-extras # install dependencies (creates .venv + uv.lock)
cp .env.example .env cp .env.example .env # optional; defaults work out of the box
make dev # uvicorn artifactstore.api.http:app --reload make migrate-fresh # initialise ./var/artifactstore.db
make test # pytest make dev # uvicorn on 127.0.0.1:8000
make lint # ruff check + ruff format --check
make type # mypy --strict # in another terminal
make migrate # alembic upgrade head (configured in WP-0001-T002) curl -s http://127.0.0.1:8000/health | python3 -m json.tool
artifactstore health
``` ```
The dev server listens on `127.0.0.1:8000`. The scaffold root route ## Make targets
returns `{"service": "artifact-store", "status": "scaffold"}`; the real
`/health` endpoint lands in `WP-0001-T014`. | Target | Purpose |
|----------------------|---------|
| `make install` | `uv sync --all-extras` |
| `make dev` | run the FastAPI app with reload |
| `make test` | run the pytest suite |
| `make lint` | ruff check + ruff format --check |
| `make format` | apply ruff format + ruff check --fix |
| `make type` | mypy --strict |
| `make migrate` | `alembic upgrade head` |
| `make migrate-fresh` | drop the local SQLite DB and re-run migrations |
| `make clean` | remove caches and build artefacts |
## CLI
```sh
artifactstore version # print the package version
artifactstore migrate # alembic upgrade head
artifactstore replay # rebuild materialised views from the event log
artifactstore health # JSON liveness summary (same payload as /health)
```
## Documentation ## Documentation
- [INTENT.md](INTENT.md) — purpose, product thesis, scope, boundary. - [INTENT.md](INTENT.md) — purpose, product thesis, scope, boundary.
- [SCOPE.md](SCOPE.md) — lightweight orientation. - [SCOPE.md](SCOPE.md) — lightweight orientation.
- [docs/OPERATOR.md](docs/OPERATOR.md) — runbook: env vars, DB backends,
CLI / HTTP reference, end-to-end smoke test, replay procedure.
- [docs/ARCHITECTURE-BLUEPRINT.md](docs/ARCHITECTURE-BLUEPRINT.md) — v2 - [docs/ARCHITECTURE-BLUEPRINT.md](docs/ARCHITECTURE-BLUEPRINT.md) — v2
architecture: modules, data model, API shape. architecture: modules, data model, API shape.
- [docs/PLATFORM-AMBITION.md](docs/PLATFORM-AMBITION.md) — longer-horizon - [docs/PLATFORM-AMBITION.md](docs/PLATFORM-AMBITION.md) — longer-horizon
thesis and v1 schema commitments. thesis and v1 schema commitments.
- [docs/ROADMAP.md](docs/ROADMAP.md) — workplan sequencing across phases. - [docs/ROADMAP.md](docs/ROADMAP.md) — workplan sequencing across phases.
- [docs/adr/](docs/adr/) — architecture decision records. - [docs/adr/](docs/adr/) — architecture decision records (ADR-0001 …
ADR-0006).
- [docs/ASSEMBLY-EXPERIMENT.md](docs/ASSEMBLY-EXPERIMENT.md) — opt-in - [docs/ASSEMBLY-EXPERIMENT.md](docs/ASSEMBLY-EXPERIMENT.md) — opt-in
research line on hand-tuned asm for hot kernels. research line on hand-tuned asm for hot kernels.
## Active workplans ## Workplans
- [WP-0001 — Foundation: scaffold, core kernels, local FS backend](workplans/ARTIFACT-STORE-WP-0001-service-baseline.md) - [WP-0001 — Foundation: scaffold, core kernels, local FS backend](workplans/ARTIFACT-STORE-WP-0001-service-baseline.md)**done**
- [WP-0002 — Ingestion API and manifest surface](workplans/ARTIFACT-STORE-WP-0002-ingestion-api.md) (planned) - [WP-0002 — Ingestion API and manifest surface](workplans/ARTIFACT-STORE-WP-0002-ingestion-api.md) (planned)
- [WP-0003 — Retention lifecycle](workplans/ARTIFACT-STORE-WP-0003-retention-lifecycle.md) (planned) - [WP-0003 — Retention lifecycle](workplans/ARTIFACT-STORE-WP-0003-retention-lifecycle.md) (planned)
- [WP-0004 — S3-compatible backend](workplans/ARTIFACT-STORE-WP-0004-s3-compatible-backend.md) (planned) - [WP-0004 — S3-compatible backend](workplans/ARTIFACT-STORE-WP-0004-s3-compatible-backend.md) (planned)

223
docs/OPERATOR.md Normal file
View File

@@ -0,0 +1,223 @@
# Operator Guide
Status: v0.1 (WP-0001 baseline)
Updated: 2026-05-16
This guide is the user manual for running `artifact-store` v0.1 — the
library + CLI + minimal HTTP app that landed in WP-0001. Ingest, finalize,
and retrieve workflows go through the Python library today; the HTTP
upload API arrives in WP-0002.
For architectural background see
[ARCHITECTURE-BLUEPRINT.md](ARCHITECTURE-BLUEPRINT.md), the ADRs under
[adr/](adr/), and the [ROADMAP](ROADMAP.md).
## Prerequisites
- Python 3.12 or 3.13
- [`uv`](https://docs.astral.sh/uv/) on the PATH (one static binary)
- A POSIX-ish shell (Linux, macOS, WSL2)
The pinned tech stack is documented in
[ADR-0005](adr/0005-v1-tech-stack.md).
## Quick start
```sh
uv sync --all-extras # install deps; produces .venv/ and uv.lock
cp .env.example .env # optional — the defaults work out of the box
make migrate-fresh # creates ./var/artifactstore.db and applies migrations
make dev # uvicorn on 127.0.0.1:8000
```
In another terminal:
```sh
curl -s http://127.0.0.1:8000/health | python3 -m json.tool
artifactstore health
```
Both should report ``status: ok``.
## Environment variables
All settings are prefixed with ``ARTIFACTSTORE_`` and read by
`pydantic-settings` from the environment and (optionally) `./.env`.
| Variable | Default | Purpose |
|-----------------------------------|-----------------------------------------------|---------|
| `ARTIFACTSTORE_DATABASE_URL` | `sqlite+aiosqlite:///./var/artifactstore.db` | SQLAlchemy async URL. Alembic translates `+aiosqlite` and `+asyncpg` to their sync drivers at migrate-time. |
| `ARTIFACTSTORE_STORAGE_LOCAL_ROOT`| `./var/storage` | Root directory for the local filesystem storage backend. Created on first use. |
| `ARTIFACTSTORE_LOG_LEVEL` | `INFO` | Python logging level (`DEBUG` / `INFO` / `WARNING` / `ERROR`). |
See [`.env.example`](../.env.example) for the canonical template.
## Database backends
### SQLite (development default)
Zero-config. The database file lives at `./var/artifactstore.db` by default
and is gitignored.
```sh
make migrate-fresh # drop and re-create
make migrate # idempotent: apply pending migrations
```
### PostgreSQL 16+ (shared deployments)
Install the optional `postgres` extra (pulls in `psycopg[binary]` for
Alembic's sync driver):
```sh
uv sync --all-extras --extra postgres
```
Set the URL with the async driver; Alembic switches to `+psycopg` for
migrations automatically:
```sh
export ARTIFACTSTORE_DATABASE_URL=postgresql+asyncpg://artifactstore:secret@db.internal:5432/artifactstore
make migrate
```
The schema is identical to SQLite (per
[ADR-0002](adr/0002-event-log-source-of-truth.md) the events table drives
all materialised views).
## Storage backends
The storage adapter SPI is documented in
[ADR-0001](adr/0001-content-addressed-storage.md) and
[ADR-0004](adr/0004-control-plane-data-plane-contract.md).
### Local filesystem (default)
Objects are addressed by content (`blake3:<hex>`) and laid out as
```
<root>/<algorithm>/<hex[0:2]>/<hex[2:4]>/<hex>
```
with atomic writes (tmpfile + fsync + rename). The S3-compatible backend
lands in WP-0004.
## CLI reference
`artifactstore --help` lists every subcommand. The v0.1 set:
| Command | Purpose |
|--------------------------|---------|
| `artifactstore version` | Print the package version and exit. |
| `artifactstore migrate` | Run `alembic upgrade head` against the configured database. |
| `artifactstore replay` | Truncate every materialised view and rebuild it from the event log; prints the highest sequence applied. |
| `artifactstore health` | JSON liveness summary (db, backend, status). Same payload as the HTTP `/health` endpoint. |
The CLI is a thin client over `artifactstore.registry.Registry`
(see [ADR-0005](adr/0005-v1-tech-stack.md)).
## HTTP reference (v0.1)
| Route | Purpose |
|----------------|---------|
| `GET /` | Service banner (scaffold marker). |
| `GET /health` | Liveness summary. Returns ``{status, db, backend, version}``. `status` is `ok` only when both the DB probe (`SELECT 1`) and the backend `health()` succeed. |
| `GET /docs` | FastAPI's interactive OpenAPI docs (`/openapi.json` underneath). |
Package CRUD, file upload/download, manifest retrieval, retention controls,
and the event stream all land in WP-0002WP-0003. Today they are reachable
via the Python library.
## End-to-end smoke test (Python library)
This exercises every layer (identity, manifest, events, dataplane, storage,
registry, replay) end-to-end against the default SQLite + local FS configuration.
```python
import asyncio
from collections.abc import AsyncIterator
from artifactstore.app import build_registry
from artifactstore.manifest import decode as manifest_decode
async def chunks(data: bytes) -> AsyncIterator[bytes]:
yield data
async def main() -> None:
registry = build_registry()
try:
pkg = await registry.create_package(
name="smoke-test",
producer="ops",
subject="example.org",
retention_class="raw-evidence",
actor="ops",
metadata={"smoke": True},
)
await registry.ingest_file(
pkg, relative_path="hello.txt", media_type="text/plain",
stream=chunks(b"hello world"), actor="ops",
)
manifest_addr = await registry.finalize_package(pkg, actor="ops")
cbor = await registry.get_manifest_bytes(pkg, format="cbor")
manifest = manifest_decode(cbor)
print("package:", pkg)
print("manifest digest:", manifest_addr)
print("files in manifest:", [f.relative_path for f in manifest.files])
finally:
await registry.dispose()
asyncio.run(main())
```
Prerequisites: `make migrate-fresh` has been run so the schema and the
retention class seeds exist.
## Replay / disaster recovery
Every state-changing operation writes one row to `events` and updates the
materialised views in the same transaction
([ADR-0002](adr/0002-event-log-source-of-truth.md)). If the materialised
views are lost or corrupted, rebuild them from the event log:
```sh
artifactstore replay
```
The command drops every row from `artifact_packages`, `artifact_files`,
`storage_locations`, and `retention_state`, then replays the events in
sequence order through the canonical view writer. The result is
**byte-identical** to the materialised state before the replay
(verified by the WP-0001-T013 integration test).
## Failure modes operators should expect
| Symptom | Likely cause | Fix |
|--------------------------------------------------|----------------------------------------------|-----|
| `/health` returns `status: degraded`, `db.healthy: false` | DB unreachable or migrations not applied | Check `ARTIFACTSTORE_DATABASE_URL`; run `make migrate`. |
| `/health` returns `status: degraded`, `backend.healthy: false` | Storage root missing or unreadable | Recreate `ARTIFACTSTORE_STORAGE_LOCAL_ROOT` or fix permissions. |
| `ObjectNotFoundError` from `get_file` | Underlying bytes deleted but the file row remains | Investigate; v1 does not garbage-collect orphaned rows (WP-0006). |
| `DuplicateRelativePathError` from `ingest_file` | Same package + path ingested twice | Use a distinct `relative_path` per file within one package. |
## References
- [INTENT.md](../INTENT.md) — purpose and scope.
- [SCOPE.md](../SCOPE.md) — what this repo does and does not own.
- [ARCHITECTURE-BLUEPRINT.md](ARCHITECTURE-BLUEPRINT.md) — module layout,
data model, API shape.
- [PLATFORM-AMBITION.md](PLATFORM-AMBITION.md) — longer-horizon thesis
and the v1 schema commitments.
- [ROADMAP.md](ROADMAP.md) — workplan sequencing.
- [ASSEMBLY-EXPERIMENT.md](ASSEMBLY-EXPERIMENT.md) — opt-in asm research line.
### Architecture Decision Records
- [ADR-0001 — Content-Addressed Storage with Dual Digest](adr/0001-content-addressed-storage.md)
- [ADR-0002 — Append-Only Event Log as Source of Truth](adr/0002-event-log-source-of-truth.md)
- [ADR-0003 — Manifest Canonicalisation = Canonical CBOR](adr/0003-manifest-canonical-cbor.md)
- [ADR-0004 — Control Plane / Data Plane Contract](adr/0004-control-plane-data-plane-contract.md)
- [ADR-0005 — V1 Technology Stack](adr/0005-v1-tech-stack.md)
- [ADR-0006 — OCI Artifact Compatibility Kept Reachable](adr/0006-oci-compatibility-reachable.md)

View File

@@ -4,13 +4,13 @@ type: workplan
title: "Foundation: Scaffold, Core Kernels, Local FS Backend" title: "Foundation: Scaffold, Core Kernels, Local FS Backend"
repo: artifact-store repo: artifact-store
domain: stack domain: stack
status: active status: done
owner: codex owner: codex
topic_slug: stack topic_slug: stack
planning_priority: high planning_priority: high
planning_order: 1 planning_order: 1
created: "2026-05-15" created: "2026-05-15"
updated: "2026-05-15" updated: "2026-05-16"
state_hub_workstream_id: "aebf996c-8721-4e8c-9e56-61d5e4bf8dcb" state_hub_workstream_id: "aebf996c-8721-4e8c-9e56-61d5e4bf8dcb"
--- ---
@@ -280,7 +280,7 @@ Acceptance:
```task ```task
id: ARTIFACT-STORE-WP-0001-T008 id: ARTIFACT-STORE-WP-0001-T008
status: todo status: done
priority: medium priority: medium
state_hub_task_id: "9b60036c-61f2-4c22-ad31-7213473d42d0" state_hub_task_id: "9b60036c-61f2-4c22-ad31-7213473d42d0"
``` ```