shard-wiki/research/260614-wikibase-deep-dive/findings.md

# Wikibase / Wikidata — deep dive (findings)

**Date:** 2026-06-14 · **Source:** SHARD-WP-0003 T2 · **Subject:** Wikibase (MediaWiki
extension) and its flagship instance **Wikidata**, incl. the Wikidata Query Service (SPARQL).

## Why this dive

Every structured shard so far tops out at *typed records in a database*: Notion's
database-pages, XWiki's XObjects/classes, Trilium's typed relations, Roam/Logseq's
attribute blocks. Wikibase is a *different kind of structure altogether* — a **typed
knowledge graph of entities and provenance-bearing statements**, queried with **SPARQL**
over an **RDF** projection. It is the **far end of the structure spectrum** and of the
**native-query spectrum**, and it pushes **provenance down to the individual assertion**.
The question for shard-wiki: what does a shard look like when its "page" is *not prose but a
set of statements*, and what does the page model / adapter contract owe such a shard?

## 1. The data model — entities, statements, snaks

**Entities** are the top-level objects, each on its own MediaWiki page with a **stable
opaque ID**:

- **Item** — `Q42`. Has multilingual **labels / descriptions / aliases**, a set of
  **statements**, and **sitelinks** (links to wiki articles). The label is *annotation*,
  not identity — `Q42` is the identity, "Douglas Adams" is just its English label.
- **Property** — `P31` ("instance of"). Also has labels/descriptions/aliases, plus a
  **fixed datatype** constraining its values (item-reference, string, time,
  globe-coordinate, quantity, monolingual-text, url, external-id, commons-media, …).

**Statement** = the unit of assertion on an item. Structure:

```
statement = claim + references[] + rank
claim     = mainSnak + qualifiers[]
snak      = property + snaktype + (value)        # snaktype ∈ value | somevalue | novalue
```

- **Main snak** — the core property→value assertion (e.g. `P31` → `Q5` "human").
- **Qualifiers** — snaks that *contextualize* the claim without being the subject (validity
  time, "as of", determination method, units). E.g. *population (P1082) = 8.4M, **point in
  time (P585) = 2020***.
- **References** — lists of snaks citing **where the claim comes from** (a source item, a
  URL, a page number). **Provenance attached to the individual statement, not the page.**
- **Rank** — `preferred` | `normal` | `deprecated`: relative importance among same-property
  statements (lets multiple, even contradictory, values coexist with a curation signal —
  the structured analogue of fedwiki's "chorus").
- Each statement carries a **stable GUID** (`Q42$<uuid>`), so statements are individually
  addressable.

`somevalue` (known to exist, value unknown) and `novalue` (known *not* to have a value) are
**first-class** — the model represents *known-unknowns* explicitly, which prose and most
DBs cannot.

## 2. The RDF / SPARQL surface

Wikibase **projects entities to RDF**; the **Wikidata Query Service (WDQS)** is a
**Blazegraph** triple store exposing a **SPARQL** endpoint. The projection is deliberately
multi-layered:

- **Truthy** triples (`wdt:` prefix) — the simple "best" value, for easy queries:
  `wd:Q42 wdt:P31 wd:Q5`.
- **Full** statements — reified so qualifiers/references/rank survive: `wd:Q42 p:P31
  ?stmt . ?stmt ps:P31 wd:Q5 ; pq:P585 ?time ; prov:wasDerivedFrom ?ref`. (`p:`=statement
  node, `ps:`=statement value, `pq:`=qualifier, `pr:`/`prov:`=reference.)
- **Federated SPARQL** — the `SERVICE <endpoint> { … }` keyword runs a sub-query against
  *another* SPARQL endpoint and joins the results. **Query-level federation is built into
  the query language** — a different federation primitive from fedwiki's fork/neighborhood.
- **EntitySchemas / ShEx** — schemas (`E`-ids) that *validate* an entity's shape (Shape
  Expressions). Optional, declarative structure validation over the open graph.

## 3. Storage, identity, history

- **Storage:** each entity is a **JSON blob stored as a MediaWiki page** (`Item:` /
  `Property:` content model). The RDF/SPARQL store is a **derived index** rebuilt from these
  canonical JSON entities (an *update stream* feeds WDQS) — exactly shard-wiki's
  "derived query index over a canonical store" pattern (UC-63), at planet scale.
- **Identity:** the **opaque Q/P/L IDs are the identity**, fully decoupled from
  human-readable labels and from language. This is the cleanest real-world instance of
  **stable, language-neutral identity ≠ label/placement** — a strong reinforcement of our
  identity model (T16).
- **History:** because each entity is one MediaWiki page, history is **page-level MediaWiki
  revisions** — every edit is a full-entity JSON snapshot with author/timestamp/comment.
  *Coarse* history granularity (whole entity per revision), but the **edit API is
  fine-grained** (`wbsetclaim`, `wbeditentity` patch individual statements). So: **fine
  write API over a coarse history unit** — a distinct point on the write/history spectra.

## 4. Capability profile

| Dimension (synthesis spectrum) | Wikibase / Wikidata |
|--------------------------------|---------------------|
| Attachment mode | **external-API** (MediaWiki Action API + REST) **and** a derived **SPARQL endpoint**; self-hostable |
| Addressing granularity | **statement** (each has a GUID) within an **entity** (Q/P id) |
| Content identity | **stable opaque ID** (Q/P/L); labels are multilingual annotations |
| Identity vs placement | **fully separated** — identity is language- and label-neutral |
| Structure | **typed knowledge graph**: entities + statements (claim+qualifiers+refs+rank) |
| History | **page-level revisions** (whole-entity JSON snapshots); fine-grained edit API |
| Merge model | MediaWiki last-writer / edit-conflict; rank lets contradictory values coexist |
| Native query | **SPARQL** (RDF) + **federated `SERVICE`** cross-endpoint join — the far end |
| Translation | **not Markdown** — content *is* statements; render to prose is a lossy projection |
| Attachment/write granularity | **statement-level writes** via API; coarse history unit |
| Operational envelope | huge derived index (Blazegraph), rate-limited public endpoints |
| Access grant | open read; MediaWiki user/permission model for write; self-host = own ACL |
| Content opacity | transparent (public JSON + RDF); not encrypted |
| Provenance | **statement-level** — references + rank per assertion (new far end) |

## 5. INTENT mapping

### Reinforcements

- **Stable identity ≠ placement** (T16): Q/P IDs decoupled from labels/language are the
  textbook case — adopt the principle that a page's *identity* is an opaque stable handle,
  display names are annotations.
- **Derived index over canonical store** (UC-63): WDQS is exactly a SPARQL index rebuilt
  from canonical JSON entities via an update stream — validates the projection pattern.
- **Union without erasure / chorus**: **rank** lets multiple (even contradictory) statements
  coexist with a curation signal rather than forcing one truth — the *structured* analogue
  of fedwiki's chorus (UC-72) and our "view multiple versions" (UC-27).
- **Mechanism over policy**: references + rank are *mechanism* for representing disagreement
  and sourcing; which statement "wins" is left to the consumer/query.

### Divergences (boundaries / design notes)

- **Content is not Markdown.** A Wikibase "page" is a set of statements; there is no prose
  body. This is the **structure far-end**: shard-wiki must either (a) treat such a shard as
  a **structured/typed shard** projected to a *lossy* Markdown/table rendering (UC-55/UC-73),
  or (b) model a page whose payload is typed statements (T12). Forcing it into Markdown-first
  erases the graph — a design-bug if done silently; render-with-provenance instead.
- **Provenance granularity is finer than ours.** Our provenance is per-page/per-shard;
  Wikibase is **per-statement** (references) and even per-value (rank). The page model and
  coordination journal should *allow* sub-page provenance (UC-75) even if MVP records it per
  page.
- **Query is graph, not text/datalog.** SPARQL over RDF (with federated `SERVICE`) is a
  richer query far-end than Roam/Logseq datalog or Notion filters (UC-52) — and its
  `SERVICE` federation is a *query-time* cross-shard join, distinct from fedwiki structural
  federation. Note both as native-query tiers.

### What to keep

1. **Opaque stable identity, labels-as-annotations** as the identity model (T16).
2. **Statement/assertion-level provenance** (references) and a **coexistence-with-rank**
   model as the structured form of union-without-erasure (UC-75).
3. **Derived SPARQL/graph index over a canonical entity store** as a projection pattern
   (UC-63/UC-74), incl. **federated query** as a first-class federation mode.
4. A **typed-graph page payload** option in the page model (T12), with **lossy
   render-to-Markdown** as the projection (never silent flattening).

## 6. UC seeds

| # | Seed | Disposition |
|---|------|-------------|
| UC-73 | Attach a **Wikibase** as a **typed entity-statement (RDF) shard** (items/properties/statements w/ qualifiers); project to a rendered page view, lossy to Markdown, preserving the graph | **new** |
| UC-74 | **Graph-query the union** via **SPARQL** and **federate queries across endpoints** (`SERVICE`) — graph query as a native-query tier + query-time cross-shard join | **new** |
| UC-75 | Preserve **statement-level provenance** — references + rank attached to each assertion (sub-page provenance granularity) | **new** |
| — | typed records → typed *graph* entities | enrich **UC-34** |
| — | inter-record relations → typed graph edges with qualifiers | enrich **UC-58** |
| — | native query → SPARQL/RDF + federated SERVICE | enrich **UC-52** |
| — | provenance → statement/assertion granularity | enrich **UC-24** |

## 7. Architecture notes for SHARD-WP-0002

- **T12 (structured/typed page model):** add a **typed-graph payload** tier above
  typed-records — a page whose content is **entities + statements (claim + qualifiers +
  references + rank)**, with `somevalue`/`novalue` known-unknowns. Render-to-Markdown is a
  **lossy projection**, not the canonical form.
- **T16 (identity / addressing):** adopt **opaque stable identity with labels-as-annotation**
  (Q/P model); record **statement GUIDs** as an example of *sub-page addressable units*.
- **Native-query tiering:** SPARQL/RDF + federated `SERVICE` is the **graph far-end** of the
  query spectrum (above datalog/filters); `SERVICE` is also a **query-time federation**
  mode to sit beside fedwiki's structural federation.
- **Provenance model:** allow **per-statement references + rank** (sub-page provenance,
  coexistence-with-curation) in the union, even if MVP collapses to per-page.
- **Derived index:** WDQS = canonical JSON entities → update stream → Blazegraph SPARQL
  index; the reference implementation of UC-63 at scale (per-shard or core-built index, Q16).

## 8. Open questions

1. Does shard-wiki model a **typed-graph page** natively (T12), or always treat Wikibase as
   a structured shard **projected to a Markdown/table rendering** (UC-55), or both
   (canonical graph + lossy view)?
2. Is **SPARQL/graph query** exposed as a union-level capability (translate to a common
   query layer) or only as a **pass-through** to graph-capable shards? How does federated
   `SERVICE` relate to shard-wiki's own cross-shard query?
3. At what granularity does the coordination journal record **provenance** — per page
   (MVP), per statement (Wikibase-native), or configurable?
4. Is **rank** (coexisting contradictory values w/ curation) representable in the union as a
   first-class "chorus of statements," unifying with fedwiki's page-level chorus (UC-72/27)?

## 9. Sources

- Wikibase/DataModel and **DataModel/Primer** — mediawiki.org
- Help:Qualifiers; Wikidata SPARQL query service + Query Help; SPARQL tutorial — wikidata.org
- Wikidata Query Service / User Manual — mediawiki.org; Wikitech (Blazegraph, updater)
- "The wikibase model" — Vanderbilt Libraries Digital Lab (heardlibrary.github.io)
- RaiseWikibase — Wikibase Data Model functions (ub-mannheim.github.io)
- WShEx / EntitySchemas (ShEx) — arxiv.org/abs/2208.02697, ceur-ws.org Vol-3262

## 10. Traceability

New UCs **UC-73–UC-75** carry the marker **⬡** in the wikiengines column of
`spec/UseCaseCatalog.md`. Enriched: UC-34, UC-58, UC-52, UC-24. Architecture cross-refs:
SHARD-WP-0002 T12, T16, native-query tiering, provenance model, UC-63 derived index.