coulomb/artifact-store
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

WP-0001-T008: operator docs + ADR cross-linking; mark WP-0001 done

Browse Source 
docs/OPERATOR.md (new): runbook with prerequisites, quick start,
environment variables, SQLite + PostgreSQL setup, storage layout,
CLI reference, HTTP /health reference, an end-to-end Python smoke
test (create_package -> ingest_file -> finalize -> manifest), the
replay / disaster-recovery procedure, common failure modes, and a
References section that links every ADR (0001..0006), the
blueprint, platform ambition, roadmap, and assembly experiment.

README.md: refreshed to v0.1 baseline status. Quick-start uses the
real flow (uv sync, migrate-fresh, dev, /health, artifactstore health).
Make targets and CLI commands tabulated. Links docs/OPERATOR.md.

AGENTS.md: Current Repo Shape now reflects the landed scaffold +
library + CLI + HTTP app rather than "no runnable scaffold yet";
links OPERATOR.md and lists the canonical local commands.

workplans/ARTIFACT-STORE-WP-0001-service-baseline.md:
- T008 marked done.
- frontmatter status: active -> done; updated: 2026-05-16.

All ten WP-0001 tasks are now done (T001/T002/T003/T008/T009/T010/
T011/T012/T013/T014). Foundation workplan retires.

Gates: ruff clean, mypy --strict clean, 83 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-05-16 09:02:36 +02:00
parent fe47058e1f
commit f90c761ef6
4 changed files with 288 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
AGENTS.md
View File

@@ -162,22 +162,33 @@ To create a new workplan:
## Current Repo Shape
This repository is in service-baseline planning. The current sources of truth are:
v0.1 baseline (WP-0001) is live: library, CLI, minimal HTTP app, local FS
backend, end-to-end ingest + finalize + replay. The pinned tech stack is in
[ADR-0005](docs/adr/0005-v1-tech-stack.md).
Sources of truth:
- `INTENT.md` — purpose, product thesis, scope, service boundary.
- `SCOPE.md` — lightweight orientation.
- `docs/OPERATOR.md` — runbook: env vars, DB backends, CLI / HTTP reference, smoke test, replay procedure.
- `docs/ARCHITECTURE-BLUEPRINT.md` — architecture v2: modules, data model, API shape.
- `docs/PLATFORM-AMBITION.md` — longer-horizon thesis and the v1 schema commitments (A1–A9).
- `docs/adr/` — architecture decision records ADR-0001 … ADR-0006 (content-addressed storage, event log as source of truth, canonical CBOR manifests, control/data plane contract, v1 tech stack, OCI reachability).
- `docs/ROADMAP.md` — workplan sequencing across phases.
- `docs/ASSEMBLY-EXPERIMENT.md` — opt-in research line on hand-tuned asm for hot kernels.
- `workplans/ARTIFACT-STORE-WP-0001-service-baseline.md` — Foundation workplan; first to start.
- `workplans/ARTIFACT-STORE-WP-0001-service-baseline.md` — Foundation workplan (done).
- `workplans/ARTIFACT-STORE-WP-{0002..0005}-*.md` — planned next workplans.
No runnable service scaffold exists yet. The pinned tech stack is in ADR-0005
(Python 3.12, uv, FastAPI, SQLAlchemy Core + asyncpg/aiosqlite, Alembic,
cbor2, blake3, ruff, mypy, pytest, typer). Add install, dev-server, and test
commands here when `ARTIFACT-STORE-WP-0001-T001` lands.
Local commands:
```sh
make install # uv sync --all-extras
make migrate-fresh # drop + re-create the dev SQLite DB
make dev # uvicorn on 127.0.0.1:8000
make test # pytest
make lint type # ruff + mypy --strict
artifactstore health
```
## Repo Boundary

65
README.md
View File

@@ -6,53 +6,78 @@ artifacts.
The registry owns artifact identity, metadata, provenance, retention
policy, and retrieval records. Bytes are delegated to configured storage
backends (local filesystem in v1, S3-compatible / Ceph RGW next).
backends (local filesystem in v1, S3-compatible / Ceph RGW in WP-0004).
The shape is library-first (`artifactstore` Python package); the HTTP
server and the CLI are thin consumers. Content is addressed by digest;
state is authoritative in an append-only event log; materialised views
are rebuildable.
state is authoritative in an append-only event log
([ADR-0002](docs/adr/0002-event-log-source-of-truth.md)); materialised
views are rebuildable.
## Status
Scaffold landed. The core kernels and local FS backend follow in the
remaining tasks of [WP-0001](workplans/ARTIFACT-STORE-WP-0001-service-baseline.md).
**v0.1 (WP-0001 baseline)** — library, CLI, minimal HTTP app, local FS
backend, end-to-end ingest + finalize + replay all working. The full
package CRUD HTTP API lands in WP-0002.
## Develop
## Quick start
Requires Python ≥ 3.12 and [`uv`](https://docs.astral.sh/uv/) on the path.
Requires Python ≥ 3.12 and [`uv`](https://docs.astral.sh/uv/).
```sh
make install # uv sync --all-extras
cp .env.example .env
uv sync --all-extras # install dependencies (creates .venv + uv.lock)
cp .env.example .env # optional; defaults work out of the box
make dev # uvicorn artifactstore.api.http:app --reload
make test # pytest
make lint # ruff check + ruff format --check
make type # mypy --strict
make migrate # alembic upgrade head (configured in WP-0001-T002)
make migrate-fresh # initialise ./var/artifactstore.db
make dev # uvicorn on 127.0.0.1:8000
# in another terminal
curl -s http://127.0.0.1:8000/health | python3 -m json.tool
artifactstore health
```
The dev server listens on `127.0.0.1:8000`. The scaffold root route
returns `{"service": "artifact-store", "status": "scaffold"}`; the real
`/health` endpoint lands in `WP-0001-T014`.
## Make targets
| Target | Purpose |
|----------------------|---------|
| `make install` | `uv sync --all-extras` |
| `make dev` | run the FastAPI app with reload |
| `make test` | run the pytest suite |
| `make lint` | ruff check + ruff format --check |
| `make format` | apply ruff format + ruff check --fix |
| `make type` | mypy --strict |
| `make migrate` | `alembic upgrade head` |
| `make migrate-fresh` | drop the local SQLite DB and re-run migrations |
| `make clean` | remove caches and build artefacts |
## CLI
```sh
artifactstore version # print the package version
artifactstore migrate # alembic upgrade head
artifactstore replay # rebuild materialised views from the event log
artifactstore health # JSON liveness summary (same payload as /health)
```
## Documentation
- [INTENT.md](INTENT.md) — purpose, product thesis, scope, boundary.
- [SCOPE.md](SCOPE.md) — lightweight orientation.
- [docs/OPERATOR.md](docs/OPERATOR.md) — runbook: env vars, DB backends,
CLI / HTTP reference, end-to-end smoke test, replay procedure.
- [docs/ARCHITECTURE-BLUEPRINT.md](docs/ARCHITECTURE-BLUEPRINT.md) — v2
architecture: modules, data model, API shape.
- [docs/PLATFORM-AMBITION.md](docs/PLATFORM-AMBITION.md) — longer-horizon
thesis and v1 schema commitments.
- [docs/ROADMAP.md](docs/ROADMAP.md) — workplan sequencing across phases.
- [docs/adr/](docs/adr/) — architecture decision records.
- [docs/adr/](docs/adr/) — architecture decision records (ADR-0001 …
ADR-0006).
- [docs/ASSEMBLY-EXPERIMENT.md](docs/ASSEMBLY-EXPERIMENT.md) — opt-in
research line on hand-tuned asm for hot kernels.
## Active workplans
## Workplans
- [WP-0001 — Foundation: scaffold, core kernels, local FS backend](workplans/ARTIFACT-STORE-WP-0001-service-baseline.md)
- [WP-0001 — Foundation: scaffold, core kernels, local FS backend](workplans/ARTIFACT-STORE-WP-0001-service-baseline.md) — **done**
- [WP-0002 — Ingestion API and manifest surface](workplans/ARTIFACT-STORE-WP-0002-ingestion-api.md) (planned)
- [WP-0003 — Retention lifecycle](workplans/ARTIFACT-STORE-WP-0003-retention-lifecycle.md) (planned)
- [WP-0004 — S3-compatible backend](workplans/ARTIFACT-STORE-WP-0004-s3-compatible-backend.md) (planned)

223
docs/OPERATOR.md Normal file
View File

@@ -0,0 +1,223 @@
# Operator Guide
Status: v0.1 (WP-0001 baseline)
Updated: 2026-05-16
This guide is the user manual for running `artifact-store` v0.1 — the
library + CLI + minimal HTTP app that landed in WP-0001. Ingest, finalize,
and retrieve workflows go through the Python library today; the HTTP
upload API arrives in WP-0002.
For architectural background see
[ARCHITECTURE-BLUEPRINT.md](ARCHITECTURE-BLUEPRINT.md), the ADRs under
[adr/](adr/), and the [ROADMAP](ROADMAP.md).
## Prerequisites
- Python 3.12 or 3.13
- [`uv`](https://docs.astral.sh/uv/) on the PATH (one static binary)
- A POSIX-ish shell (Linux, macOS, WSL2)
The pinned tech stack is documented in
[ADR-0005](adr/0005-v1-tech-stack.md).
## Quick start
```sh
uv sync --all-extras # install deps; produces .venv/ and uv.lock
cp .env.example .env # optional — the defaults work out of the box
make migrate-fresh # creates ./var/artifactstore.db and applies migrations
make dev # uvicorn on 127.0.0.1:8000
```
In another terminal:
```sh
curl -s http://127.0.0.1:8000/health | python3 -m json.tool
artifactstore health
```
Both should report ``status: ok``.
## Environment variables
All settings are prefixed with ``ARTIFACTSTORE_`` and read by
`pydantic-settings` from the environment and (optionally) `./.env`.
| Variable | Default | Purpose |
|-----------------------------------|-----------------------------------------------|---------|
| `ARTIFACTSTORE_DATABASE_URL` | `sqlite+aiosqlite:///./var/artifactstore.db` | SQLAlchemy async URL. Alembic translates `+aiosqlite` and `+asyncpg` to their sync drivers at migrate-time. |
| `ARTIFACTSTORE_STORAGE_LOCAL_ROOT`| `./var/storage` | Root directory for the local filesystem storage backend. Created on first use. |
| `ARTIFACTSTORE_LOG_LEVEL` | `INFO` | Python logging level (`DEBUG` / `INFO` / `WARNING` / `ERROR`). |
See [`.env.example`](../.env.example) for the canonical template.
## Database backends
### SQLite (development default)
Zero-config. The database file lives at `./var/artifactstore.db` by default
and is gitignored.
```sh
make migrate-fresh # drop and re-create
make migrate # idempotent: apply pending migrations
```
### PostgreSQL 16+ (shared deployments)
Install the optional `postgres` extra (pulls in `psycopg[binary]` for
Alembic's sync driver):
```sh
uv sync --all-extras --extra postgres
```
Set the URL with the async driver; Alembic switches to `+psycopg` for
migrations automatically:
```sh
export ARTIFACTSTORE_DATABASE_URL=postgresql+asyncpg://artifactstore:secret@db.internal:5432/artifactstore
make migrate
```
The schema is identical to SQLite (per
[ADR-0002](adr/0002-event-log-source-of-truth.md) the events table drives
all materialised views).
## Storage backends
The storage adapter SPI is documented in
[ADR-0001](adr/0001-content-addressed-storage.md) and
[ADR-0004](adr/0004-control-plane-data-plane-contract.md).
### Local filesystem (default)
Objects are addressed by content (`blake3:<hex>`) and laid out as
```
<root>/<algorithm>/<hex[0:2]>/<hex[2:4]>/<hex>
```
with atomic writes (tmpfile + fsync + rename). The S3-compatible backend
lands in WP-0004.
## CLI reference
`artifactstore --help` lists every subcommand. The v0.1 set:
| Command | Purpose |
|--------------------------|---------|
| `artifactstore version` | Print the package version and exit. |
| `artifactstore migrate` | Run `alembic upgrade head` against the configured database. |
| `artifactstore replay` | Truncate every materialised view and rebuild it from the event log; prints the highest sequence applied. |
| `artifactstore health` | JSON liveness summary (db, backend, status). Same payload as the HTTP `/health` endpoint. |
The CLI is a thin client over `artifactstore.registry.Registry`
(see [ADR-0005](adr/0005-v1-tech-stack.md)).
## HTTP reference (v0.1)
| Route | Purpose |
|----------------|---------|
| `GET /` | Service banner (scaffold marker). |
| `GET /health` | Liveness summary. Returns ``{status, db, backend, version}``. `status` is `ok` only when both the DB probe (`SELECT 1`) and the backend `health()` succeed. |
| `GET /docs` | FastAPI's interactive OpenAPI docs (`/openapi.json` underneath). |
Package CRUD, file upload/download, manifest retrieval, retention controls,
and the event stream all land in WP-0002–WP-0003. Today they are reachable
via the Python library.
## End-to-end smoke test (Python library)
This exercises every layer (identity, manifest, events, dataplane, storage,
registry, replay) end-to-end against the default SQLite + local FS configuration.
```python
import asyncio
from collections.abc import AsyncIterator
from artifactstore.app import build_registry
from artifactstore.manifest import decode as manifest_decode
async def chunks(data: bytes) -> AsyncIterator[bytes]:
yield data
async def main() -> None:
registry = build_registry()
try:
pkg = await registry.create_package(
name="smoke-test",
producer="ops",
subject="example.org",
retention_class="raw-evidence",
actor="ops",
metadata={"smoke": True},
)
await registry.ingest_file(
pkg, relative_path="hello.txt", media_type="text/plain",
stream=chunks(b"hello world"), actor="ops",
)
manifest_addr = await registry.finalize_package(pkg, actor="ops")
cbor = await registry.get_manifest_bytes(pkg, format="cbor")
manifest = manifest_decode(cbor)
print("package:", pkg)
print("manifest digest:", manifest_addr)
print("files in manifest:", [f.relative_path for f in manifest.files])
finally:
await registry.dispose()
asyncio.run(main())
```
Prerequisites: `make migrate-fresh` has been run so the schema and the
retention class seeds exist.
## Replay / disaster recovery
Every state-changing operation writes one row to `events` and updates the
materialised views in the same transaction
([ADR-0002](adr/0002-event-log-source-of-truth.md)). If the materialised
views are lost or corrupted, rebuild them from the event log:
```sh
artifactstore replay
```
The command drops every row from `artifact_packages`, `artifact_files`,
`storage_locations`, and `retention_state`, then replays the events in
sequence order through the canonical view writer. The result is
**byte-identical** to the materialised state before the replay
(verified by the WP-0001-T013 integration test).
## Failure modes operators should expect
| Symptom | Likely cause | Fix |
|--------------------------------------------------|----------------------------------------------|-----|
| `/health` returns `status: degraded`, `db.healthy: false` | DB unreachable or migrations not applied | Check `ARTIFACTSTORE_DATABASE_URL`; run `make migrate`. |
| `/health` returns `status: degraded`, `backend.healthy: false` | Storage root missing or unreadable | Recreate `ARTIFACTSTORE_STORAGE_LOCAL_ROOT` or fix permissions. |
| `ObjectNotFoundError` from `get_file` | Underlying bytes deleted but the file row remains | Investigate; v1 does not garbage-collect orphaned rows (WP-0006). |
| `DuplicateRelativePathError` from `ingest_file` | Same package + path ingested twice | Use a distinct `relative_path` per file within one package. |
## References
- [INTENT.md](../INTENT.md) — purpose and scope.
- [SCOPE.md](../SCOPE.md) — what this repo does and does not own.
- [ARCHITECTURE-BLUEPRINT.md](ARCHITECTURE-BLUEPRINT.md) — module layout,
data model, API shape.
- [PLATFORM-AMBITION.md](PLATFORM-AMBITION.md) — longer-horizon thesis
and the v1 schema commitments.
- [ROADMAP.md](ROADMAP.md) — workplan sequencing.
- [ASSEMBLY-EXPERIMENT.md](ASSEMBLY-EXPERIMENT.md) — opt-in asm research line.
### Architecture Decision Records
- [ADR-0001 — Content-Addressed Storage with Dual Digest](adr/0001-content-addressed-storage.md)
- [ADR-0002 — Append-Only Event Log as Source of Truth](adr/0002-event-log-source-of-truth.md)
- [ADR-0003 — Manifest Canonicalisation = Canonical CBOR](adr/0003-manifest-canonical-cbor.md)
- [ADR-0004 — Control Plane / Data Plane Contract](adr/0004-control-plane-data-plane-contract.md)
- [ADR-0005 — V1 Technology Stack](adr/0005-v1-tech-stack.md)
- [ADR-0006 — OCI Artifact Compatibility Kept Reachable](adr/0006-oci-compatibility-reachable.md)

6
workplans/ARTIFACT-STORE-WP-0001-service-baseline.md
View File

@@ -4,13 +4,13 @@ type: workplan
title: "Foundation: Scaffold, Core Kernels, Local FS Backend"
repo: artifact-store
domain: stack
status: active
status: done
owner: codex
topic_slug: stack
planning_priority: high
planning_order: 1
created: "2026-05-15"
updated: "2026-05-15"
updated: "2026-05-16"
state_hub_workstream_id: "aebf996c-8721-4e8c-9e56-61d5e4bf8dcb"
---
@@ -280,7 +280,7 @@ Acceptance:
```task
id: ARTIFACT-STORE-WP-0001-T008
status: todo
status: done
priority: medium
state_hub_task_id: "9b60036c-61f2-4c22-ad31-7213473d42d0"
```