From f90c761ef6195ea2d5cf7da687d7dd1d8c309e0b Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 16 May 2026 09:02:36 +0200 Subject: [PATCH] WP-0001-T008: operator docs + ADR cross-linking; mark WP-0001 done docs/OPERATOR.md (new): runbook with prerequisites, quick start, environment variables, SQLite + PostgreSQL setup, storage layout, CLI reference, HTTP /health reference, an end-to-end Python smoke test (create_package -> ingest_file -> finalize -> manifest), the replay / disaster-recovery procedure, common failure modes, and a References section that links every ADR (0001..0006), the blueprint, platform ambition, roadmap, and assembly experiment. README.md: refreshed to v0.1 baseline status. Quick-start uses the real flow (uv sync, migrate-fresh, dev, /health, artifactstore health). Make targets and CLI commands tabulated. Links docs/OPERATOR.md. AGENTS.md: Current Repo Shape now reflects the landed scaffold + library + CLI + HTTP app rather than "no runnable scaffold yet"; links OPERATOR.md and lists the canonical local commands. workplans/ARTIFACT-STORE-WP-0001-service-baseline.md: - T008 marked done. - frontmatter status: active -> done; updated: 2026-05-16. All ten WP-0001 tasks are now done (T001/T002/T003/T008/T009/T010/ T011/T012/T013/T014). Foundation workplan retires. Gates: ruff clean, mypy --strict clean, 83 tests pass. Co-Authored-By: Claude Opus 4.7 --- AGENTS.md | 23 +- README.md | 65 +++-- docs/OPERATOR.md | 223 ++++++++++++++++++ ...ARTIFACT-STORE-WP-0001-service-baseline.md | 6 +- 4 files changed, 288 insertions(+), 29 deletions(-) create mode 100644 docs/OPERATOR.md diff --git a/AGENTS.md b/AGENTS.md index 77a277d..d1310ba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -162,22 +162,33 @@ To create a new workplan: ## Current Repo Shape -This repository is in service-baseline planning. The current sources of truth are: +v0.1 baseline (WP-0001) is live: library, CLI, minimal HTTP app, local FS +backend, end-to-end ingest + finalize + replay. The pinned tech stack is in +[ADR-0005](docs/adr/0005-v1-tech-stack.md). + +Sources of truth: - `INTENT.md` — purpose, product thesis, scope, service boundary. - `SCOPE.md` — lightweight orientation. +- `docs/OPERATOR.md` — runbook: env vars, DB backends, CLI / HTTP reference, smoke test, replay procedure. - `docs/ARCHITECTURE-BLUEPRINT.md` — architecture v2: modules, data model, API shape. - `docs/PLATFORM-AMBITION.md` — longer-horizon thesis and the v1 schema commitments (A1–A9). - `docs/adr/` — architecture decision records ADR-0001 … ADR-0006 (content-addressed storage, event log as source of truth, canonical CBOR manifests, control/data plane contract, v1 tech stack, OCI reachability). - `docs/ROADMAP.md` — workplan sequencing across phases. - `docs/ASSEMBLY-EXPERIMENT.md` — opt-in research line on hand-tuned asm for hot kernels. -- `workplans/ARTIFACT-STORE-WP-0001-service-baseline.md` — Foundation workplan; first to start. +- `workplans/ARTIFACT-STORE-WP-0001-service-baseline.md` — Foundation workplan (done). - `workplans/ARTIFACT-STORE-WP-{0002..0005}-*.md` — planned next workplans. -No runnable service scaffold exists yet. The pinned tech stack is in ADR-0005 -(Python 3.12, uv, FastAPI, SQLAlchemy Core + asyncpg/aiosqlite, Alembic, -cbor2, blake3, ruff, mypy, pytest, typer). Add install, dev-server, and test -commands here when `ARTIFACT-STORE-WP-0001-T001` lands. +Local commands: + +```sh +make install # uv sync --all-extras +make migrate-fresh # drop + re-create the dev SQLite DB +make dev # uvicorn on 127.0.0.1:8000 +make test # pytest +make lint type # ruff + mypy --strict +artifactstore health +``` ## Repo Boundary diff --git a/README.md b/README.md index 211f53a..ee4c2df 100644 --- a/README.md +++ b/README.md @@ -6,53 +6,78 @@ artifacts. The registry owns artifact identity, metadata, provenance, retention policy, and retrieval records. Bytes are delegated to configured storage -backends (local filesystem in v1, S3-compatible / Ceph RGW next). +backends (local filesystem in v1, S3-compatible / Ceph RGW in WP-0004). The shape is library-first (`artifactstore` Python package); the HTTP server and the CLI are thin consumers. Content is addressed by digest; -state is authoritative in an append-only event log; materialised views -are rebuildable. +state is authoritative in an append-only event log +([ADR-0002](docs/adr/0002-event-log-source-of-truth.md)); materialised +views are rebuildable. ## Status -Scaffold landed. The core kernels and local FS backend follow in the -remaining tasks of [WP-0001](workplans/ARTIFACT-STORE-WP-0001-service-baseline.md). +**v0.1 (WP-0001 baseline)** — library, CLI, minimal HTTP app, local FS +backend, end-to-end ingest + finalize + replay all working. The full +package CRUD HTTP API lands in WP-0002. -## Develop +## Quick start -Requires Python ≥ 3.12 and [`uv`](https://docs.astral.sh/uv/) on the path. +Requires Python ≥ 3.12 and [`uv`](https://docs.astral.sh/uv/). ```sh -make install # uv sync --all-extras -cp .env.example .env +uv sync --all-extras # install dependencies (creates .venv + uv.lock) +cp .env.example .env # optional; defaults work out of the box -make dev # uvicorn artifactstore.api.http:app --reload -make test # pytest -make lint # ruff check + ruff format --check -make type # mypy --strict -make migrate # alembic upgrade head (configured in WP-0001-T002) +make migrate-fresh # initialise ./var/artifactstore.db +make dev # uvicorn on 127.0.0.1:8000 + +# in another terminal +curl -s http://127.0.0.1:8000/health | python3 -m json.tool +artifactstore health ``` -The dev server listens on `127.0.0.1:8000`. The scaffold root route -returns `{"service": "artifact-store", "status": "scaffold"}`; the real -`/health` endpoint lands in `WP-0001-T014`. +## Make targets + +| Target | Purpose | +|----------------------|---------| +| `make install` | `uv sync --all-extras` | +| `make dev` | run the FastAPI app with reload | +| `make test` | run the pytest suite | +| `make lint` | ruff check + ruff format --check | +| `make format` | apply ruff format + ruff check --fix | +| `make type` | mypy --strict | +| `make migrate` | `alembic upgrade head` | +| `make migrate-fresh` | drop the local SQLite DB and re-run migrations | +| `make clean` | remove caches and build artefacts | + +## CLI + +```sh +artifactstore version # print the package version +artifactstore migrate # alembic upgrade head +artifactstore replay # rebuild materialised views from the event log +artifactstore health # JSON liveness summary (same payload as /health) +``` ## Documentation - [INTENT.md](INTENT.md) — purpose, product thesis, scope, boundary. - [SCOPE.md](SCOPE.md) — lightweight orientation. +- [docs/OPERATOR.md](docs/OPERATOR.md) — runbook: env vars, DB backends, + CLI / HTTP reference, end-to-end smoke test, replay procedure. - [docs/ARCHITECTURE-BLUEPRINT.md](docs/ARCHITECTURE-BLUEPRINT.md) — v2 architecture: modules, data model, API shape. - [docs/PLATFORM-AMBITION.md](docs/PLATFORM-AMBITION.md) — longer-horizon thesis and v1 schema commitments. - [docs/ROADMAP.md](docs/ROADMAP.md) — workplan sequencing across phases. -- [docs/adr/](docs/adr/) — architecture decision records. +- [docs/adr/](docs/adr/) — architecture decision records (ADR-0001 … + ADR-0006). - [docs/ASSEMBLY-EXPERIMENT.md](docs/ASSEMBLY-EXPERIMENT.md) — opt-in research line on hand-tuned asm for hot kernels. -## Active workplans +## Workplans -- [WP-0001 — Foundation: scaffold, core kernels, local FS backend](workplans/ARTIFACT-STORE-WP-0001-service-baseline.md) +- [WP-0001 — Foundation: scaffold, core kernels, local FS backend](workplans/ARTIFACT-STORE-WP-0001-service-baseline.md) — **done** - [WP-0002 — Ingestion API and manifest surface](workplans/ARTIFACT-STORE-WP-0002-ingestion-api.md) (planned) - [WP-0003 — Retention lifecycle](workplans/ARTIFACT-STORE-WP-0003-retention-lifecycle.md) (planned) - [WP-0004 — S3-compatible backend](workplans/ARTIFACT-STORE-WP-0004-s3-compatible-backend.md) (planned) diff --git a/docs/OPERATOR.md b/docs/OPERATOR.md new file mode 100644 index 0000000..c58282d --- /dev/null +++ b/docs/OPERATOR.md @@ -0,0 +1,223 @@ +# Operator Guide + +Status: v0.1 (WP-0001 baseline) +Updated: 2026-05-16 + +This guide is the user manual for running `artifact-store` v0.1 — the +library + CLI + minimal HTTP app that landed in WP-0001. Ingest, finalize, +and retrieve workflows go through the Python library today; the HTTP +upload API arrives in WP-0002. + +For architectural background see +[ARCHITECTURE-BLUEPRINT.md](ARCHITECTURE-BLUEPRINT.md), the ADRs under +[adr/](adr/), and the [ROADMAP](ROADMAP.md). + +## Prerequisites + +- Python 3.12 or 3.13 +- [`uv`](https://docs.astral.sh/uv/) on the PATH (one static binary) +- A POSIX-ish shell (Linux, macOS, WSL2) + +The pinned tech stack is documented in +[ADR-0005](adr/0005-v1-tech-stack.md). + +## Quick start + +```sh +uv sync --all-extras # install deps; produces .venv/ and uv.lock +cp .env.example .env # optional — the defaults work out of the box + +make migrate-fresh # creates ./var/artifactstore.db and applies migrations +make dev # uvicorn on 127.0.0.1:8000 +``` + +In another terminal: + +```sh +curl -s http://127.0.0.1:8000/health | python3 -m json.tool +artifactstore health +``` + +Both should report ``status: ok``. + +## Environment variables + +All settings are prefixed with ``ARTIFACTSTORE_`` and read by +`pydantic-settings` from the environment and (optionally) `./.env`. + +| Variable | Default | Purpose | +|-----------------------------------|-----------------------------------------------|---------| +| `ARTIFACTSTORE_DATABASE_URL` | `sqlite+aiosqlite:///./var/artifactstore.db` | SQLAlchemy async URL. Alembic translates `+aiosqlite` and `+asyncpg` to their sync drivers at migrate-time. | +| `ARTIFACTSTORE_STORAGE_LOCAL_ROOT`| `./var/storage` | Root directory for the local filesystem storage backend. Created on first use. | +| `ARTIFACTSTORE_LOG_LEVEL` | `INFO` | Python logging level (`DEBUG` / `INFO` / `WARNING` / `ERROR`). | + +See [`.env.example`](../.env.example) for the canonical template. + +## Database backends + +### SQLite (development default) + +Zero-config. The database file lives at `./var/artifactstore.db` by default +and is gitignored. + +```sh +make migrate-fresh # drop and re-create +make migrate # idempotent: apply pending migrations +``` + +### PostgreSQL 16+ (shared deployments) + +Install the optional `postgres` extra (pulls in `psycopg[binary]` for +Alembic's sync driver): + +```sh +uv sync --all-extras --extra postgres +``` + +Set the URL with the async driver; Alembic switches to `+psycopg` for +migrations automatically: + +```sh +export ARTIFACTSTORE_DATABASE_URL=postgresql+asyncpg://artifactstore:secret@db.internal:5432/artifactstore +make migrate +``` + +The schema is identical to SQLite (per +[ADR-0002](adr/0002-event-log-source-of-truth.md) the events table drives +all materialised views). + +## Storage backends + +The storage adapter SPI is documented in +[ADR-0001](adr/0001-content-addressed-storage.md) and +[ADR-0004](adr/0004-control-plane-data-plane-contract.md). + +### Local filesystem (default) + +Objects are addressed by content (`blake3:`) and laid out as + +``` +//// +``` + +with atomic writes (tmpfile + fsync + rename). The S3-compatible backend +lands in WP-0004. + +## CLI reference + +`artifactstore --help` lists every subcommand. The v0.1 set: + +| Command | Purpose | +|--------------------------|---------| +| `artifactstore version` | Print the package version and exit. | +| `artifactstore migrate` | Run `alembic upgrade head` against the configured database. | +| `artifactstore replay` | Truncate every materialised view and rebuild it from the event log; prints the highest sequence applied. | +| `artifactstore health` | JSON liveness summary (db, backend, status). Same payload as the HTTP `/health` endpoint. | + +The CLI is a thin client over `artifactstore.registry.Registry` +(see [ADR-0005](adr/0005-v1-tech-stack.md)). + +## HTTP reference (v0.1) + +| Route | Purpose | +|----------------|---------| +| `GET /` | Service banner (scaffold marker). | +| `GET /health` | Liveness summary. Returns ``{status, db, backend, version}``. `status` is `ok` only when both the DB probe (`SELECT 1`) and the backend `health()` succeed. | +| `GET /docs` | FastAPI's interactive OpenAPI docs (`/openapi.json` underneath). | + +Package CRUD, file upload/download, manifest retrieval, retention controls, +and the event stream all land in WP-0002–WP-0003. Today they are reachable +via the Python library. + +## End-to-end smoke test (Python library) + +This exercises every layer (identity, manifest, events, dataplane, storage, +registry, replay) end-to-end against the default SQLite + local FS configuration. + +```python +import asyncio +from collections.abc import AsyncIterator +from artifactstore.app import build_registry +from artifactstore.manifest import decode as manifest_decode + + +async def chunks(data: bytes) -> AsyncIterator[bytes]: + yield data + + +async def main() -> None: + registry = build_registry() + try: + pkg = await registry.create_package( + name="smoke-test", + producer="ops", + subject="example.org", + retention_class="raw-evidence", + actor="ops", + metadata={"smoke": True}, + ) + await registry.ingest_file( + pkg, relative_path="hello.txt", media_type="text/plain", + stream=chunks(b"hello world"), actor="ops", + ) + manifest_addr = await registry.finalize_package(pkg, actor="ops") + cbor = await registry.get_manifest_bytes(pkg, format="cbor") + manifest = manifest_decode(cbor) + print("package:", pkg) + print("manifest digest:", manifest_addr) + print("files in manifest:", [f.relative_path for f in manifest.files]) + finally: + await registry.dispose() + + +asyncio.run(main()) +``` + +Prerequisites: `make migrate-fresh` has been run so the schema and the +retention class seeds exist. + +## Replay / disaster recovery + +Every state-changing operation writes one row to `events` and updates the +materialised views in the same transaction +([ADR-0002](adr/0002-event-log-source-of-truth.md)). If the materialised +views are lost or corrupted, rebuild them from the event log: + +```sh +artifactstore replay +``` + +The command drops every row from `artifact_packages`, `artifact_files`, +`storage_locations`, and `retention_state`, then replays the events in +sequence order through the canonical view writer. The result is +**byte-identical** to the materialised state before the replay +(verified by the WP-0001-T013 integration test). + +## Failure modes operators should expect + +| Symptom | Likely cause | Fix | +|--------------------------------------------------|----------------------------------------------|-----| +| `/health` returns `status: degraded`, `db.healthy: false` | DB unreachable or migrations not applied | Check `ARTIFACTSTORE_DATABASE_URL`; run `make migrate`. | +| `/health` returns `status: degraded`, `backend.healthy: false` | Storage root missing or unreadable | Recreate `ARTIFACTSTORE_STORAGE_LOCAL_ROOT` or fix permissions. | +| `ObjectNotFoundError` from `get_file` | Underlying bytes deleted but the file row remains | Investigate; v1 does not garbage-collect orphaned rows (WP-0006). | +| `DuplicateRelativePathError` from `ingest_file` | Same package + path ingested twice | Use a distinct `relative_path` per file within one package. | + +## References + +- [INTENT.md](../INTENT.md) — purpose and scope. +- [SCOPE.md](../SCOPE.md) — what this repo does and does not own. +- [ARCHITECTURE-BLUEPRINT.md](ARCHITECTURE-BLUEPRINT.md) — module layout, + data model, API shape. +- [PLATFORM-AMBITION.md](PLATFORM-AMBITION.md) — longer-horizon thesis + and the v1 schema commitments. +- [ROADMAP.md](ROADMAP.md) — workplan sequencing. +- [ASSEMBLY-EXPERIMENT.md](ASSEMBLY-EXPERIMENT.md) — opt-in asm research line. + +### Architecture Decision Records + +- [ADR-0001 — Content-Addressed Storage with Dual Digest](adr/0001-content-addressed-storage.md) +- [ADR-0002 — Append-Only Event Log as Source of Truth](adr/0002-event-log-source-of-truth.md) +- [ADR-0003 — Manifest Canonicalisation = Canonical CBOR](adr/0003-manifest-canonical-cbor.md) +- [ADR-0004 — Control Plane / Data Plane Contract](adr/0004-control-plane-data-plane-contract.md) +- [ADR-0005 — V1 Technology Stack](adr/0005-v1-tech-stack.md) +- [ADR-0006 — OCI Artifact Compatibility Kept Reachable](adr/0006-oci-compatibility-reachable.md) diff --git a/workplans/ARTIFACT-STORE-WP-0001-service-baseline.md b/workplans/ARTIFACT-STORE-WP-0001-service-baseline.md index bee694c..fe169f3 100644 --- a/workplans/ARTIFACT-STORE-WP-0001-service-baseline.md +++ b/workplans/ARTIFACT-STORE-WP-0001-service-baseline.md @@ -4,13 +4,13 @@ type: workplan title: "Foundation: Scaffold, Core Kernels, Local FS Backend" repo: artifact-store domain: stack -status: active +status: done owner: codex topic_slug: stack planning_priority: high planning_order: 1 created: "2026-05-15" -updated: "2026-05-15" +updated: "2026-05-16" state_hub_workstream_id: "aebf996c-8721-4e8c-9e56-61d5e4bf8dcb" --- @@ -280,7 +280,7 @@ Acceptance: ```task id: ARTIFACT-STORE-WP-0001-T008 -status: todo +status: done priority: medium state_hub_task_id: "9b60036c-61f2-4c22-ad31-7213473d42d0" ```