artifact-store/docs/PLATFORM-AMBITION.md

# Platform Ambition

Status: draft
Created: 2026-05-15

This document records the longer-horizon thesis behind `artifact-store` and
captures which decisions are taken now to keep that horizon reachable without
expanding the v1 workplan. It sits beside, not above, `INTENT.md` and
`SCOPE.md`. INTENT defines what we build first; this document defines what
the v1 must not foreclose.

## Thesis

Generated artifacts — evidence packages, build outputs, ML models, logs,
snapshots, reports, scorecards, exports — are first-class durable objects in
modern software work. They sit somewhere between source code (well-served by
Git) and binary releases (well-served by OCI registries). The space between
is currently filled by a fragmented mix of bespoke directories, ad-hoc S3
buckets, vendor registries (Artifactory, Nexus), and document-management
systems that were not built for machine producers.

`artifact-store` aims to occupy that gap with one substrate: a generic,
content-addressed, signed, deduplicated, retention-aware artifact registry
and storage gateway that other tools embed or speak to.

The reference points are deliberate. **VLC** and **ffmpeg** lead their domain
not by being the prettiest applications but by being correct, fast, embeddable,
portable, and indispensable infrastructure for everyone else. The same
strategy applies here: build a kernel that is so good at the bytes-and-
identity layer that every artifact-producing tool would rather speak its
protocol than reinvent it.

## Commercial horizon

The longer-horizon commercial target is **a sovereign artifact-storage
product line for European cloud providers** — Stack IT (Schwarz Group) is
the concrete example. The thesis is:

- Hyperscalers (AWS S3, GCS, Azure Blob) sell raw object storage. They do
  not sell *artifact identity, retention, attestation, federation, evidence
  preservation*. Customers either build it themselves or buy proprietary
  registries on top.
- A European hyperscaler that ships a turnkey, sovereign, GDPR-aligned
  artifact substrate on top of its own object storage has a defensible
  differentiation against AWS — not in raw price-per-GB, which is a losing
  race, but in regulated workloads (evidence retention, audit, signed
  attestations, legal-hold, sovereign jurisdiction guarantees).
- Open source is the wedge. A widely-adopted upstream that the provider
  ships, supports, and extends is far stronger than a proprietary stack.

This is a multi-year horizon, not a v1 deliverable. It is recorded here so
schema and protocol decisions made now keep that path open.

## Reference points

| System          | What we learn from it                                        |
|-----------------|--------------------------------------------------------------|
| ffmpeg          | Embeddable core, hand-tuned hot paths, runtime CPU dispatch  |
| VLC             | Plugin architecture, portability, ubiquity through being a library too |
| Git             | Content-addressed storage, Merkle DAG, pack files, integrity |
| restic          | Single static binary, CDC + dedup, encryption by default     |
| IPFS            | Content-addressing, federation, partial replication          |
| OCI Registry    | Standardised manifest + blob model with broad ecosystem      |
| Sigstore / cosign | Signed attestations as a first-class artifact property     |
| MinIO           | Operator ergonomics, S3 wire compat as adoption vector       |
| SeaweedFS / Ceph | Separation of metadata plane from data plane                |
| RocksDB / LMDB  | Embeddable storage engines with predictable performance      |
| Kafka           | Log-as-source-of-truth, materialised views                   |
| BLAKE3          | Modern hash primitive: parallel, Merkle-tree-native, asm-tuned |

We are not trying to reproduce any of these. We are trying to occupy a
specific gap between them with the best ideas from each.

## Non-goals (still)

The platform ambition does not change the v1 boundary in `INTENT.md` or
`SCOPE.md`. In particular it does not:

- replace StateHub as the work / decision system of record;
- encode producer-specific assessment semantics in the registry core;
- require any of the optimisations listed in the "near-horizon" section
  below to land in v1;
- commit the project to writing assembly. The assembly-experiment line
  (`docs/ASSEMBLY-EXPERIMENT.md`) is opt-in research, not roadmap-critical.

## Architectural commitments — preserved by v1

The following decisions are taken now because reversing them later is
expensive. Each lands in v1 as a schema or contract decision; full
exploitation is later work.

### A1. Content as the primary address

Internal canonical key for stored bytes is `<algo>:<digest>`, not a logical
path. Files within a package keep a `relative_path` as logical metadata,
but the storage backend sees and addresses content hashes.

- Enables: global dedup, Merkle integrity proofs, partial mirrors,
  federation, OCI compatibility.
- v1 cost: one schema column (`content_address`) and a deterministic key
  derivation; no behaviour change.

### A2. BLAKE3 as native digest, SHA-256 retained for interop

`digest_algorithm` is a column on `artifact_files`. v1 default may remain
`sha256` to ship the pilot quickly; the column exists so `blake3` can ship
without migration.

- Enables: faster hashing, free Merkle root over a package, alignment with
  modern signing tooling.
- v1 cost: column + adapter table mapping algo → hashing impl.

### A3. Append-only event log as source of truth

An `events` table with a monotonic sequence number is the authoritative
record of registry mutations. The current metadata tables are a
materialised view rebuildable from the log.

- Enables: CDC feeds, audit, replication, point-in-time recovery, signed
  event streams.
- v1 cost: one extra table written on the same transaction as today's
  mutations.

### A4. Signed manifests, canonicalisation pinned

Manifest serialisation uses a canonical form (recommendation: canonical
CBOR; JCS as alternative) so byte-identical signing is possible across
languages and time. v1 may not actually sign — the pin guarantees that
when signing lands, every prior manifest is re-signable byte-for-byte.

- Enables: cosign / Sigstore, in-toto, SLSA attestations, OCI-style
  manifest digests.
- v1 cost: pick one canonicalisation library and use it for manifest
  writes. Zero runtime cost.

### A5. Control plane / data plane separation at the contract

Even if v1 implements both in one Python process, the boundary between
"registry / API / retention" (control plane) and "hash / chunk / store /
serve" (data plane) is a named contract. When the data plane is later
extracted into a Rust binary, the API does not change.

- Enables: native-speed ingestion, language flexibility on the hot path,
  independent scaling.
- v1 cost: discipline (separate Python module with no API leakage), not
  code.

### A6. Resumable upload wire shape

API exposes upload sessions: `POST /uploads`, `PATCH /uploads/{id}` with
range, `POST /uploads/{id}/complete`. v1 implementation may still be
single-shot multipart under the hood, but the resource shape exists so
chunked / resumable upload is additive.

- Enables: streaming, retry-safe ingestion, very-large-package support.
- v1 cost: route definitions only; underlying logic can remain simple.

### A7. Tiering as a property of storage locations

`storage_location` carries `retrieval_tier` (`hot|warm|cold|archive`) and
`restore_status` columns, nullable, default `hot`. The API can already
return "not immediately available" without changing artifact identity.

- Enables: future cold storage, Glacier-style restore flows.
- v1 cost: two nullable columns.

### A8. Schema-typed metadata with open escape hatch

Producers register a metadata schema (JSON Schema) per variant
(e.g. `guide-board.run.v1`). Stored as open JSON, validated against the
registered schema at ingest time. Queries can use typed views.

- Enables: tooling, search, GraphQL views, typed clients without losing
  flexibility.
- v1 cost: a `metadata_schemas` table; v1 validation can be a no-op.

### A9. OCI compatibility kept reachable

We do not promise OCI compatibility in v1, but we do not adopt any
data model that prevents it. Concretely: keep content addresses as
`<algo>:<hex>`, keep manifest structure compatible with an OCI image
manifest (config + layers + annotations), and avoid invariants that the
OCI spec forbids.

- Enables: future `oras push` / `cosign sign` / Helm ecosystem entry.
- v1 cost: one design review per schema change against the OCI spec.

## Near-horizon technical roadmap (post-baseline)

Roughly ordered. Not commitments; planning hooks.

1. **Rust data-plane binary.** Receives chunked uploads, runs BLAKE3 + CDC +
   optional Zstd + optional AES-GCM, writes to storage adapter. Speaks a
   minimal gRPC or framed-bincode protocol to the Python control plane over
   a Unix socket.
2. **Content-defined chunking (FastCDC).** Stored chunks become the dedup
   unit. Package manifest references chunk digests; package digest is the
   Merkle root.
3. **Cosign-compatible signing pipeline.** Every finalised manifest can be
   signed; signatures stored alongside the manifest.
4. **Event stream out.** NATS or Kafka topic of registry events for
   downstream consumers.
5. **OCI artifact endpoint.** A `/v2/` namespace that speaks the OCI
   distribution spec on top of the same storage.
6. **WASM plugin host.** Producers and operators can ship signed `.wasm`
   modules for content extraction, redaction, scorecard generation,
   custom hashing, indexing. This is the "ffmpeg moment" — open extension
   surface that does not require forking the core.
7. **Federation.** Signed manifest exchange between artifact-store
   instances. Gossip or explicit peering.
8. **Cold tier adapters.** S3 Glacier, Tape, IA classes.

## How this document is used

- Every schema change in WP-0001 (or successors) is checked against
  commitments A1–A9. A change that violates one is either rejected or
  documented as a deliberate revision of this document.
- Every "we could do this faster in native code" idea is filed against
  `docs/ASSEMBLY-EXPERIMENT.md`, not bolted onto a workplan.
- Every new producer integration is checked against the commercial
  horizon: does it generalise, or does it bake in producer-specific
  assumptions?

This document is allowed to be wrong. It is not allowed to be silent.
Update it when the thesis changes; do not let v1 quietly close doors that
the v3 needs open.