Compare commits

...

14 Commits

Author SHA1 Message Date
cbad0dc958 Some cleanup needs to wait for later 2026-03-20 03:52:05 +01:00
634642cb52 feat(capability-requests): add routing_note, PATCH endpoint, word-boundary fix, and ops-bridge tunnel targets
- Add `routing_note` column (migration l9g0h1i2j3k4) to persist why a request was routed to a given domain
- Fix substring-match bug in `_route_capability`: use `\b` word-boundary regex so 'postgres' no longer matches inside 'postgresql'
- Include `title` in keyword scoring for better routing accuracy
- Return `routing_note` string from `_route_capability` and store it on the request
- Add `PATCH /capability-requests/{id}` endpoint + `CapabilityRequestPatch` schema to correct mutable metadata (catalog_entry_id, priority, blocking_task_id, fulfilling_workstream_id)
- Add `patch_capability_request` MCP tool wrapping the new endpoint
- Add 105 lines of routing tests (word-boundary, title-match, multi-entry scoring, broadcast fallback)
- Add `tunnels-up`, `tunnels-status`, `tunnels-check` Makefile targets for ops-bridge managed tunnels

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 03:47:54 +01:00
1e0ae37c89 docs: add State Hub reference page and restructure reference index
New page (docs/state-hub.md) covers:
- Why: the invisible state problem across repos and agents
- What: Derived Data Store, Read Model, Agent Orchestration Layer,
  Cross-Repo Observatory — and what it is NOT
- Derived Data Store principle (ADR-003): fingerprint cache, rebuild
  guarantee, force-refresh
- Repository Orchestrator: session protocol, cross-domain coordination
  via messages + capability routing, Kaizen agents
- Architecture diagram (ASCII), technology choices, data model overview
- Running the hub, design principles, related docs

reference.md: add Architecture & Design section grouping state-hub,
TPSC, GDPR maturity, SCOPE.md, capabilities, and goals docs.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 02:01:58 +01:00
51e95ec21a docs(adr): ADR-003 — Materialized Derived State with Fingerprint Invalidation
Formalises the caching pattern introduced in doi_cache: pre-compute and store
repo-sourced derived data, invalidate by fingerprint (composite of DB timestamps
+ file mtimes), force-refresh on demand.

Names the pattern against the literature (Materialized View, Derived Data Store,
CQRS Read Model, ETag-style invalidation) and mandates its use for all future
repo-sourced derived data with an implementation checklist.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 01:54:21 +01:00
d63e7310d5 perf(doi): fingerprint-based DB cache for DoI results
Adds doi_cache table (migration k8f9a0b1c2d3). Results are stored after
each evaluation and reused on subsequent requests when the fingerprint
matches. Fingerprint covers repo.updated_at, latest TPSC snapshot_at,
latest goal updated_at, and mtime of SCOPE.md / CLAUDE.md / tpsc.yaml.

Behaviour:
- Summary (warm cache, nothing changed): ~0.4s (was 0.9s)
- Summary (one repo stale): ~0.9s (only stale repos recomputed)
- Single repo (cache hit): ~0.2s (was 40s for full check)
- Single repo ?force_refresh=true: ~2s (full C7/C13 subprocess check)

Total journey: 108s (original) → 6s → <1s → 0.2s (cached single repo)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 01:47:19 +01:00
6ae5cb6bf7 perf(doi): eliminate HTTP self-calls in summary — 48 calls → 3 bulk DB queries
Root cause: C2/C9/C10 each made a full HTTP round-trip back to the API
(asyncio.to_thread → urllib → TCP → uvicorn → SQLAlchemy → DB) for every
repo. 16 repos × 3 calls = 48 self-calls at ~80-150ms each = ~6s total.

Fix: doi_engine.evaluate() accepts a prefetch dict. The summary endpoint
runs 3 bulk GROUP BY queries (domain status, TPSC snapshot counts, active
goal counts) and passes results directly — zero HTTP self-calls in summary
mode.

Result: /repos/doi/summary 6s → <1s (6× improvement on top of prior 13×).
Total improvement from original: 108s → <1s.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 01:37:40 +01:00
7dec3ac9ee perf(dashboard): lazy-load DoI tiers on Repositories page
Page now renders in ~200ms. DoI badges and KPI card show a spinner
while the background fetch resolves (~6s), then update reactively
via Observable Mutable pattern (doiData / doiLoading).

Fast path: repos, SBOM, domains, workstreams — immediate render.
Slow path: /repos/doi/summary — background, non-blocking.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 01:31:48 +01:00
66e3a9afe4 perf(doi): 13x speedup for /repos/doi/summary (108s → ~6s)
Two fixes:
1. skip_consistency=True in summary mode — omits C7/C13 subprocess calls
   (consistency_check.py) which were the main bottleneck (32 spawns for 16 repos).
   Full check still available per-repo via GET /repos/{slug}/doi.
2. asyncio.gather — all repos evaluated in parallel instead of sequentially.

Also: rename Repositories page title from "Repos" to "Repositories".

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 01:29:27 +01:00
f94ee008b5 feat(doi): Repository DoI automated gate and dashboard integration (CUST-WP-0024)
Implements the 14-criterion DoI checklist as a runnable gate with API,
MCP tools, CLI script, and dashboard integration.

Core components:
- api/doi_engine.py — async engine evaluating all 14 criteria (asyncio.to_thread
  for non-blocking HTTP self-calls), shared by API and CLI
- api/schemas/doi.py — DoICriterion, DoIReport, DoISummaryEntry schemas
- api/routers/repos.py — GET /repos/{slug}/doi + GET /repos/doi/summary
- scripts/check_doi.py — CLI: make check-doi REPO=<slug> / check-doi-all
- mcp_server/server.py — check_repo_doi(), get_doi_summary() tools

Dashboard (repos.md):
- DoI tier badge per repo (None/Core/Standard/Full) colour-coded red→green
- Domain block shows lowest DoI tier across its repos
- DoI KPI card in summary row
- DoI filter in All Repos Table
- Link to Repository DoI policy page

Also fixes: TPSC snapshots 500 error (missing nested selectinload for
catalog_entry relationship in list_snapshots endpoint).

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 01:08:18 +01:00
a7b26ef6de docs(policy): add heading to workstream-dod policy file
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 00:42:08 +01:00
19d2b3cd31 docs(policy): add Repository Definition of Integrated (DoI)
Three-tier checklist defining what 'fully integrated with the state-hub'
means for a repository:
- Core (Registered): registered, domain assigned, path resolves, remote URL
- Standard (Integrated): SCOPE.md, CLAUDE.md, workplan convention, SBOM, TPSC
- Full (Fully Integrated): repo goal, capabilities declared, agents template,
  clean consistency check, host paths registered

Exposed via /policy/repo-doi (editable in dashboard) and linked under Policies.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 00:35:54 +01:00
0f9266cd91 docs(tpsc): add GDPR Maturity Model reference page
Full reference for the 7-level CNIL/IAPP CMMI-aligned scale used in TPSC:
source frameworks, per-level descriptions, suitability guidance, key GDPR
concepts (DPA, SCCs, adequacy, BCRs, Art.9), assignment decision tree,
and authoritative references.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 00:19:07 +01:00
c7a893f068 feat(tpsc): Third-Party Services Catalog (CUST-WP-0023)
Introduces TPSC for tracking external service dependencies with GDPR
compliance maturity (CNIL/IAPP CMMI scale), pricing model, ToS, and
data retention information across all repos.

Primary data:
- canon/tpsc/{openai,anthropic,gemini,openrouter}-api.yaml — service definitions
- tpsc.yaml in each repo (llm-connect seeded with 4 services)

State-hub additions:
- Migration j7e8f9a0b1c2: tpsc_catalog + tpsc_snapshots + tpsc_entries
- api/models/tpsc.py, api/schemas/tpsc.py, api/routers/tpsc.py
- /tpsc/catalog/, /tpsc/ingest/, /tpsc/snapshots/, /tpsc/report/gdpr endpoints
- 4 MCP tools: register_service, list_services, ingest_tpsc_tool, get_gdpr_report
- scripts/ingest_tpsc.py + make ingest-tpsc[/-all] targets
- Dashboard: tpsc.md page + docs/tpsc.md

GDPR maturity scale: unknown | non_compliant | initial | developing | defined | managed | certified
Warnings triggered at: unknown, non_compliant, initial

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 00:15:26 +01:00
118c5628e9 fix(mcp): resolve repo paths with existence check before trusting hostname match
Stale host_paths entries (wrong username, old machine) were silently overriding
the correct local_path, causing FileNotFoundError on tools like list_kaizen_agents.

Extracts _resolve_repo_path(repo) helper that tries host_paths[hostname] first
but validates the path exists on disk before trusting it, then falls back to
local_path. Both candidates support ~ expansion. Applied to all 4 call sites:
_kaizen_agents_dir, validate_repo_adr, check_repo_consistency, ingest_sbom_tool.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 21:38:35 +01:00
41 changed files with 3733 additions and 69 deletions

1
.gitignore vendored
View File

@@ -184,4 +184,5 @@ state-hub/dashboard/node_modules/
state-hub/dashboard/.observablehq/
state-hub/dashboard/src/.observablehq/
state-hub/dashboard/dist/
state-hub/kubectl

View File

@@ -309,6 +309,23 @@ Use this structure when creating or rewriting SCOPE.md:
---
## Provided Capabilities
<!-- What can this repo's domain provide to other domains on request? -->
<!-- Each capability block is parsed by the state-hub capability catalog ingest. -->
<!-- Remove the examples and add your own, or leave empty if none. -->
<!--
```capability
type: infrastructure
title: Example capability title
description: What this capability provides, in one or two sentences.
keywords: [keyword1, keyword2, keyword3]
```
-->
---
## Notes
<!-- Anything else worth knowing. Keep it short. -->

View File

@@ -0,0 +1,174 @@
---
id: ADR-003
type: architecture-decision-record
title: "Materialized Derived State with Fingerprint Invalidation for Repo-Sourced Data"
status: accepted
decided_by: Bernd Worsch
date: "2026-03-20"
tags: ["architecture", "state-hub", "caching", "read-model", "materialized-view", "derived-state"]
---
# ADR-003: Materialized Derived State with Fingerprint Invalidation
## Status
Accepted.
## Context
The Custodian State Hub is a **read model** (CQRS terminology) — its data is
fully derivable from canonical sources that live in repositories and the
filesystem. No state-hub data is authoritative; it is always a derived view
of what the repos contain.
Several categories of data fit this description:
| Data | Canonical source | State-hub table |
|---|---|---|
| SBOM dependencies | `uv.lock`, `package-lock.json`, etc. | `sbom_entries` |
| Third-party service declarations | `tpsc.yaml` | `tpsc_entries` |
| Provided capabilities | `SCOPE.md` `capability` blocks | `capability_catalog` |
| DoI compliance tier | 14 criteria across repo files + DB | `doi_cache` |
| Workplan task status | `workplans/*.md` | `tasks` |
Early implementations either recomputed this data on every request (too slow)
or ingested it once without invalidation (stale data goes undetected). Neither
is acceptable for a system designed to give accurate, fast orientation.
The `doi_cache` table, introduced in CUST-WP-0024, demonstrated a pattern that
solves both problems. This ADR formalises that pattern and mandates its use
for all repo-sourced derived data.
## Pattern Name
**Materialized Derived State with Fingerprint Invalidation.**
This pattern is known under several names in the literature:
- **Materialized View** (SQL standard, PostgreSQL) — the stored result of a
query or computation, refreshed on demand when source data changes.
- **Derived Data Store** (Kleppmann, *Designing Data-Intensive Applications*,
Ch. 3 & 11) — a system whose entire dataset can be rebuilt from upstream
sources; it is never the source of truth.
- **Read Model / Projection** (CQRS / Event Sourcing) — a pre-computed view
maintained alongside a write model, rebuilt when relevant events occur.
- **Fingerprint-based / Content-addressed invalidation** — analogous to HTTP
ETags: a cache entry is valid as long as a composite hash/timestamp of its
inputs matches the stored value.
The State Hub already documents itself as a read model. This ADR extends that
principle to specify *how* the read model stays fresh.
## Decision
### 1. All repo-sourced derived data MUST be materialised in the DB
Data computed from repository files or repo records must be stored in a
dedicated table rather than recomputed per request. Direct computation on
every API call is only permissible for development tooling or when explicitly
forced by the caller.
### 2. Each materialised table MUST carry a `fingerprint` column
The fingerprint is a deterministic string encoding all inputs that affect the
computed result. It is compared on each read; if unchanged, the stored result
is returned without recomputation. If changed, the result is recomputed and
the stored value is updated.
**Fingerprint composition rules:**
- Include the `updated_at` timestamp of every DB record that feeds the
computation (repo record, related domain, goals, snapshots).
- Include the `mtime` (filesystem modification time) of every file that feeds
the computation (`SCOPE.md`, `CLAUDE.md`, lockfiles, `tpsc.yaml`, etc.).
- Join all components with `|` as a pipe-separated string — no hashing needed
since the string is compared by equality, not transmitted to clients.
- If a file is absent, encode `filename:absent` rather than omitting it, so
file creation also triggers invalidation.
**Reference implementation:** `state-hub/api/doi_engine.py::compute_fingerprint()`
### 3. Every materialised endpoint MUST support `?force_refresh=true`
Callers must always be able to bypass the cache and trigger a fresh
computation. This is the escape hatch for debugging, post-ingest verification,
and scheduled background refresh jobs.
### 4. Writes to source data SHOULD update the repo record's `updated_at`
Operations that change source data (SBOM ingest, TPSC ingest, capability
ingest) must ensure `managed_repos.updated_at` is refreshed so the
fingerprint detects the change on the next read. Where data lives in a
related table (e.g. `tpsc_snapshots`), the fingerprint must include that
table's `max(snapshot_at)` directly rather than relying on the repo record.
### 5. The DB is never the source of truth — the rebuild principle holds
Per ADR-001, the state-hub must be rebuildable from scratch by re-ingesting
all canonical sources. Materialised tables are **caches**, not records of
authority. They may be wiped and repopulated at any time without data loss.
This means:
- No materialised table may be the only copy of any information.
- Schema migrations that wipe a materialised table are safe and expected.
- Background jobs that periodically re-ingest all repos are valid and
encouraged.
## Consequences
### Positive
- **Fast reads in steady state** — after the first computation, subsequent
reads hit the DB with no filesystem or subprocess overhead.
- **Accurate on change** — fingerprint invalidation ensures stale data is
never silently served; the cache refreshes exactly when needed.
- **Debuggable** — `force_refresh=true` and `checked_at` timestamps make it
easy to see when a value was last computed and to trigger a recheck.
- **Consistent with the read model principle** — the pattern makes explicit
what was always implied: state-hub data is derived, not authoritative.
### Negative / Trade-offs
- **First-call latency** — cache misses are expensive (filesystem reads,
subprocess calls, HTTP self-calls). Mitigated by pre-warming caches at
startup or after ingest.
- **Fingerprint completeness** — if a new input is added to a computation and
not added to the fingerprint, stale results will be silently returned. The
fingerprint must be kept in sync with the computation.
- **Filesystem dependency** — file mtimes are volatile (e.g. `git checkout`
rewrites mtimes). In practice this means a cache miss after every checkout,
not a correctness problem.
## Implementation Checklist
When adding a new category of repo-sourced derived data:
- [ ] Create a `_cache` or `_snapshots` table with `fingerprint` and
`checked_at` columns.
- [ ] Implement `compute_fingerprint(repo, ...)` in the relevant module.
- [ ] Add `?force_refresh=true` query parameter to the read endpoint.
- [ ] Ensure the ingest script (or write path) touches `managed_repos.updated_at`
or includes a related table's `max(timestamp)` in the fingerprint.
- [ ] Verify the cache can be wiped and repopulated without data loss.
- [ ] Document which inputs are included in the fingerprint in a comment
alongside `compute_fingerprint`.
## Current Implementations
| Derived data | Table | Fingerprint inputs | Force-refresh |
|---|---|---|---|
| DoI compliance tier | `doi_cache` | `repo.updated_at`, `max(tpsc_snapshots.snapshot_at)`, `max(repo_goals.updated_at)`, `mtime(SCOPE.md)`, `mtime(CLAUDE.md)`, `mtime(tpsc.yaml)` | `?force_refresh=true` |
## Planned Applications
| Derived data | Table (proposed) | Notes |
|---|---|---|
| SBOM summary stats | `sbom_cache` | Fingerprint: `max(sbom_snapshots.snapshot_at)` |
| Capability declarations | `capability_cache` | Fingerprint: `mtime(SCOPE.md)`, `repo.updated_at` |
| Workplan status summary | Already handled by consistency checker | Fingerprint: workplan file mtimes |
## Related
- ADR-001: Workplans and Work Items Are Repository Artefacts
- ADR-002: Custodian Agent Runtime Design
- `state-hub/api/doi_engine.py` — reference implementation
- `state-hub/api/models/doi_cache.py` — reference schema
- `state-hub/migrations/versions/k8f9a0b1c2d3_doi_cache.py` — reference migration

View File

@@ -0,0 +1,22 @@
slug: anthropic-api
name: Anthropic API (Claude)
provider: Anthropic, PBC
category: llm_inference
website_url: https://anthropic.com
pricing_model: usage_based
gdpr_maturity: developing
gdpr_notes: >
DPA available. SCCs provided for EU→US transfers. Data processed in the US.
Prompts and completions are not used for training by default (API usage).
Data may be retained for up to 30 days for trust & safety review.
Privacy Shield successor mechanisms in place for international transfers.
Reference: https://www.anthropic.com/legal/data-processing-agreement
dpa_available: true
tos_url: https://www.anthropic.com/legal/aup
privacy_policy_url: https://www.anthropic.com/legal/privacy
data_processing_regions:
- us
data_retention_notes: >
API inputs/outputs not used for model training. Retained up to 30 days
for safety review. Enterprise customers can negotiate reduced retention.
status: active

View File

@@ -0,0 +1,26 @@
slug: gemini-api
name: Google Gemini API
provider: Google LLC
category: llm_inference
website_url: https://ai.google.dev
pricing_model: usage_based
gdpr_maturity: defined
gdpr_notes: >
Google Cloud (including Vertex AI / Gemini API via Google Cloud) has a
comprehensive GDPR compliance programme. DPA is part of Google Cloud ToS.
Data Processing Addendum and SCCs available. Google Cloud has EU data
residency options (data can stay in EU). Google AI Studio (free tier)
has weaker protections — data may be used to improve Google products.
Use Vertex AI / Google Cloud API endpoint for GDPR-adequate usage.
Reference: https://cloud.google.com/terms/data-processing-addendum
dpa_available: true
tos_url: https://ai.google.dev/gemini-api/terms
privacy_policy_url: https://policies.google.com/privacy
data_processing_regions:
- us
- eu # when using Vertex AI with EU region selection
data_retention_notes: >
Google AI Studio: prompts may be reviewed by human raters and used to
improve models. Google Cloud / Vertex AI: data not used for training,
retained per Cloud data retention policy (configurable).
status: active

View File

@@ -0,0 +1,24 @@
slug: openai-api
name: OpenAI API
provider: OpenAI, Inc.
category: llm_inference
website_url: https://openai.com
pricing_model: usage_based
gdpr_maturity: developing
gdpr_notes: >
DPA available (Data Processing Addendum). Standard Contractual Clauses (SCCs)
provided for EU→US data transfers. Data is processed in the US.
Input/output retained up to 30 days for safety monitoring unless opted out
via the API zero-data-retention setting. Zero-data-retention is available
on eligible endpoints. Not suitable for sensitive personal data without a
signed DPA and explicit zero-retention configuration.
Reference: https://openai.com/policies/data-processing-addendum
dpa_available: true
tos_url: https://openai.com/policies/terms-of-use
privacy_policy_url: https://openai.com/policies/privacy-policy
data_processing_regions:
- us
data_retention_notes: >
Default: 30 days for abuse monitoring. Zero-data-retention available
on eligible API endpoints via opt-in. Training opt-out available.
status: active

View File

@@ -0,0 +1,27 @@
slug: openrouter-api
name: OpenRouter API
provider: OpenRouter, Inc.
category: llm_routing
website_url: https://openrouter.ai
pricing_model: usage_based
gdpr_maturity: initial
gdpr_notes: >
OpenRouter is a US-based routing proxy for multiple LLM providers.
Privacy policy exists but as of early 2026 no formal DPA or SCCs are
publicly available. Requests are forwarded to underlying providers
(OpenAI, Anthropic, Google, Mistral, etc.) each with their own data
handling. GDPR compliance is therefore dependent on both OpenRouter
and the selected downstream model provider. Not recommended for
processing personal data in corporate/regulated environments without
a signed DPA. Suitable for development, prototyping, model comparison.
Reference: https://openrouter.ai/privacy
dpa_available: false
tos_url: https://openrouter.ai/terms
privacy_policy_url: https://openrouter.ai/privacy
data_processing_regions:
- us
data_retention_notes: >
Requests forwarded to upstream providers. OpenRouter may log requests
for billing and abuse prevention. Retention policy not formally published.
Downstream provider retention policies apply per selected model.
status: active

View File

@@ -1,4 +1,4 @@
.PHONY: install install-cli db db-tools migrate seed api dashboard check test clean register-project validate-adr add-domain rename-domain add-repo list-repos register-path cleanup-stale tunnel tunnel-daemon tunnel-loop tunnel-status tunnel-stop install-hooks install-hooks-all gitea-inventory
.PHONY: install install-cli db db-tools migrate seed api dashboard check test clean register-project validate-adr add-domain rename-domain add-repo list-repos register-path cleanup-stale tunnel tunnel-daemon tunnel-loop tunnel-status tunnel-stop tunnels-up tunnels-status tunnels-check install-hooks install-hooks-all gitea-inventory
COMPOSE = docker compose -f infra/docker-compose.yml --env-file .env
@@ -103,6 +103,19 @@ tunnel-stop:
@pkill -f "autossh.*$(TUNNEL_PORT)" 2>/dev/null && echo "autossh stopped" || true
@pkill -f "ssh.*-R $(TUNNEL_PORT)" 2>/dev/null && echo "ssh loop stopped" || true
## ops-bridge managed tunnels (preferred over tunnel-*/tunnel-daemon)
## Requires ops-bridge: bridge is at /home/worsch/.local/bin/bridge
tunnels-up:
bridge up
tunnels-status:
bridge status
## End-to-end check: verifies SSH process alive + remote port listening on COULOMBCORE.
## Exits non-zero if any tunnel is not fully operational.
tunnels-check:
bridge check
## Start (or restart) the full backend — db + migrate + uvicorn.
## Stops uvicorn on :8000 if already running, then starts fresh.
api: db
@@ -183,6 +196,30 @@ ingest-capabilities-all:
uv run python scripts/ingest_capabilities.py --all \
$(if $(DRY_RUN),--dry-run)
## Check Repository Definition of Integrated (DoI) criteria for a repo.
## Usage: make check-doi REPO=llm-connect
## Or: make check-doi-all
## Add JSON=1 for machine-readable output.
check-doi:
@test -n "$(REPO)" || (echo "ERROR: REPO is required."; exit 1)
uv run python scripts/check_doi.py --repo "$(REPO)" $(if $(JSON),--json)
check-doi-all:
uv run python scripts/check_doi.py --all $(if $(JSON),--json)
## Ingest tpsc.yaml service declarations from a repo into the TPSC catalog.
## Usage: make ingest-tpsc REPO=llm-connect
## Or: make ingest-tpsc-all
## Add DRY_RUN=1 to preview without writing.
ingest-tpsc:
@test -n "$(REPO)" || (echo "ERROR: REPO is required."; exit 1)
uv run python scripts/ingest_tpsc.py --repo "$(REPO)" \
$(if $(DRY_RUN),--dry-run)
ingest-tpsc-all:
uv run python scripts/ingest_tpsc.py --all \
$(if $(DRY_RUN),--dry-run)
## Run SBOM capture agent for a repo — generates/updates sbom-tools.yaml.
## Usage: make capture-tools REPO=railiance-infra [REPO_PATH=/home/worsch/railiance-infra]
## Add DRY_RUN=1 to preview without writing.

367
state-hub/api/doi_engine.py Normal file
View File

@@ -0,0 +1,367 @@
"""DoI engine — evaluates all 14 Repository Definition of Integrated criteria.
Shared by the API endpoint (async) and the CLI check script (asyncio.run).
All checks use only the repo dict from /repos/{slug} + HTTP calls to the API
+ local filesystem reads. No direct DB access.
"""
from __future__ import annotations
import asyncio
import json
import socket
import urllib.error
import urllib.request
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Literal
CriterionStatus = Literal["pass", "fail", "warn", "skip"]
Tier = Literal["none", "core", "standard", "full"]
# Criteria that belong to each tier (in check order)
CORE_IDS = {"C1", "C2", "C3", "C4"}
STANDARD_IDS = {"C5", "C6", "C7", "C8", "C9"}
FULL_IDS = {"C10", "C11", "C12", "C13", "C14"}
@dataclass
class CriterionResult:
id: str
label: str
tier: str
status: CriterionStatus
detail: str = ""
@dataclass
class DoIReport:
repo_slug: str
tier: Tier
core_pass: bool
standard_pass: bool
full_pass: bool
criteria: list[CriterionResult] = field(default_factory=list)
checked_at: str = field(default_factory=lambda: datetime.now(tz=timezone.utc).isoformat())
def compute_fingerprint(
repo: dict,
latest_tpsc_snap_at: str | None,
latest_goal_updated_at: str | None,
) -> str:
"""Compute a pipe-joined fingerprint of all inputs that affect DoI criteria.
If any component changes, the fingerprint changes and the cache is invalidated:
- repo.updated_at → covers last_sbom_at, remote_url, host_paths, domain changes
- latest_tpsc_snap_at → C9 (TPSC snapshot exists)
- latest_goal_updated_at → C10 (active repo goal)
- mtime of SCOPE.md, CLAUDE.md, tpsc.yaml → C5, C6, C9, C11, C12
"""
parts = [
str(repo.get("updated_at") or ""),
str(latest_tpsc_snap_at or ""),
str(latest_goal_updated_at or ""),
]
repo_path = _resolve_path(repo)
if repo_path:
for fname in ("SCOPE.md", "CLAUDE.md", "tpsc.yaml"):
f = Path(repo_path) / fname
try:
parts.append(f"{fname}:{f.stat().st_mtime:.3f}")
except FileNotFoundError:
parts.append(f"{fname}:absent")
return "|".join(parts)
def _resolve_path(repo: dict) -> str:
hostname = socket.gethostname()
host_paths = repo.get("host_paths") or {}
candidates = []
if host_paths.get(hostname):
candidates.append(host_paths[hostname])
if repo.get("local_path"):
candidates.append(repo["local_path"])
for raw in candidates:
p = Path(raw).expanduser()
if p.is_dir():
return str(p)
return ""
def _get_sync(api_base: str, path: str, params: dict | None = None) -> object:
url = f"{api_base}{path}"
if params:
q = "&".join(f"{k}={v}" for k, v in params.items() if v is not None)
if q:
url = f"{url}?{q}"
req = urllib.request.Request(url, headers={"Accept": "application/json"})
try:
with urllib.request.urlopen(req, timeout=5) as r:
return json.loads(r.read())
except Exception:
return None
async def _get(api_base: str, path: str, params: dict | None = None) -> object:
"""Async wrapper — runs blocking urllib in a thread so the event loop stays free."""
return await asyncio.to_thread(_get_sync, api_base, path, params)
async def _run_consistency(repo_slug: str, api_base: str) -> tuple[int, int, int]:
"""Run consistency_check.py and return (fail, warn, info) counts."""
script = Path(__file__).parent.parent / "scripts" / "consistency_check.py"
proc = await asyncio.create_subprocess_exec(
"uv", "run", "python", str(script),
"--repo", repo_slug,
"--api-base", api_base,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
cwd=str(Path(__file__).parent.parent),
)
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=30)
text = stdout.decode()
fail = warn = info = 0
for line in text.splitlines():
if "Summary:" in line:
parts = line.split("|")
for p in parts:
p = p.strip()
if "fail" in p:
try: fail = int(p.split()[0])
except ValueError: pass
elif "warn" in p:
try: warn = int(p.split()[0])
except ValueError: pass
elif "info" in p:
try: info = int(p.split()[0])
except ValueError: pass
return fail, warn, info
async def evaluate(
repo: dict,
api_base: str = "http://127.0.0.1:8000",
skip_consistency: bool = False,
prefetch: dict | None = None,
) -> DoIReport:
"""Evaluate all 14 DoI criteria for a repo.
Args:
repo: Repo dict (slug, domain_slug, local_path, remote_url, host_paths, last_sbom_at).
api_base: API base URL — only used when prefetch is absent.
skip_consistency: Skip C7/C13 subprocess calls (used in summary mode).
prefetch: Optional pre-fetched bulk data to avoid HTTP self-calls:
{
"domain_status": {"custodian": "active", ...}, # slug → status
"tpsc_snap_counts": {"llm-connect": 1, ...}, # repo_slug → count
"active_goal_counts": {"llm-connect": 0, ...}, # repo_slug → count
}
"""
slug = repo.get("slug", "unknown")
results: list[CriterionResult] = []
def _r(id: str, label: str, tier: str, status: CriterionStatus, detail: str = "") -> CriterionResult:
r = CriterionResult(id=id, label=label, tier=tier, status=status, detail=detail)
results.append(r)
return r
# ── Tier 1: Core ─────────────────────────────────────────────────────────
# C1: registered
_r("C1", "Registered in state-hub", "core", "pass", "Repo record exists")
# C2: domain assigned and active
domain_slug = repo.get("domain_slug") or ""
if not domain_slug:
_r("C2", "Domain assigned", "core", "fail", "No domain_slug on repo record")
else:
if prefetch and "domain_status" in prefetch:
dom_status = prefetch["domain_status"].get(domain_slug)
else:
d = await _get(api_base, f"/domains/{domain_slug}/")
dom_status = d.get("status") if d else None
if dom_status == "active":
_r("C2", "Domain assigned", "core", "pass", f"domain: {domain_slug}")
elif dom_status:
_r("C2", "Domain assigned", "core", "warn", f"Domain '{domain_slug}' status: {dom_status}")
else:
_r("C2", "Domain assigned", "core", "fail", f"Domain '{domain_slug}' not found")
# C3: local path resolves
repo_path = _resolve_path(repo)
if repo_path:
_r("C3", "Local path resolves", "core", "pass", repo_path)
else:
raw = repo.get("local_path") or "(none)"
_r("C3", "Local path resolves", "core", "fail", f"Path not accessible: {raw}")
# C4: remote URL set
remote = repo.get("remote_url") or ""
if remote.strip():
_r("C4", "Remote URL set", "core", "pass", remote)
else:
_r("C4", "Remote URL set", "core", "fail", "remote_url is empty")
# ── Tier 2: Standard ─────────────────────────────────────────────────────
# C5: SCOPE.md
if not repo_path:
_r("C5", "SCOPE.md present", "standard", "skip", "Local path unavailable")
elif (Path(repo_path) / "SCOPE.md").exists():
_r("C5", "SCOPE.md present", "standard", "pass")
else:
_r("C5", "SCOPE.md present", "standard", "fail", "SCOPE.md not found at repo root")
# C6: CLAUDE.md
if not repo_path:
_r("C6", "CLAUDE.md present", "standard", "skip", "Local path unavailable")
elif (Path(repo_path) / "CLAUDE.md").exists():
_r("C6", "CLAUDE.md present", "standard", "pass")
else:
_r("C6", "CLAUDE.md present", "standard", "fail", "CLAUDE.md not found at repo root")
# C7: workplan convention — consistency check 0 FAIL
if skip_consistency:
_r("C7", "Workplan convention (0 FAIL)", "standard", "skip", "Not checked in summary mode — use /repos/{slug}/doi for full check")
else:
try:
fail, warn, _ = await _run_consistency(slug, api_base)
if fail == 0:
_r("C7", "Workplan convention (0 FAIL)", "standard", "pass", f"consistency: {fail} fail / {warn} warn")
else:
_r("C7", "Workplan convention (0 FAIL)", "standard", "fail", f"consistency: {fail} fail / {warn} warn")
except Exception as e:
_r("C7", "Workplan convention (0 FAIL)", "standard", "skip", f"Could not run consistency check: {e}")
# C8: SBOM ingested
last_sbom = repo.get("last_sbom_at")
if last_sbom:
_r("C8", "SBOM ingested", "standard", "pass", f"last ingested: {last_sbom[:10]}")
else:
_r("C8", "SBOM ingested", "standard", "fail", "last_sbom_at not set — run make ingest-sbom")
# C9: TPSC declared (tpsc.yaml present + snapshot exists)
tpsc_file_ok = repo_path and (Path(repo_path) / "tpsc.yaml").exists()
if prefetch and "tpsc_snap_counts" in prefetch:
has_snap = (prefetch["tpsc_snap_counts"].get(slug, 0) > 0)
snap_count = prefetch["tpsc_snap_counts"].get(slug, 0)
else:
tpsc_snaps = await _get(api_base, "/tpsc/snapshots/", {"repo_slug": slug}) or []
has_snap = len(tpsc_snaps) > 0
snap_count = len(tpsc_snaps)
if not repo_path:
_r("C9", "TPSC declared", "standard", "skip", "Local path unavailable")
elif tpsc_file_ok and has_snap:
_r("C9", "TPSC declared", "standard", "pass", f"{snap_count} snapshot(s)")
elif tpsc_file_ok and not has_snap:
_r("C9", "TPSC declared", "standard", "warn", "tpsc.yaml exists but not yet ingested — run make ingest-tpsc")
elif not tpsc_file_ok:
_r("C9", "TPSC declared", "standard", "fail", "tpsc.yaml missing at repo root")
# ── Tier 3: Full ─────────────────────────────────────────────────────────
# C10: active repo goal
if prefetch and "active_goal_counts" in prefetch:
active_goal_count = prefetch["active_goal_counts"].get(slug, 0)
else:
goals = await _get(api_base, "/repo-goals/", {"repo_slug": slug}) or []
active_goal_count = sum(1 for g in goals if g.get("status") == "active")
if active_goal_count > 0:
_r("C10", "Active repo goal", "full", "pass", f"{active_goal_count} active goal(s)")
else:
_r("C10", "Active repo goal", "full", "fail", "No active repo goal — create one with create_repo_goal()")
# C11: Provided Capabilities declared in SCOPE.md
if not repo_path:
_r("C11", "Provided Capabilities declared", "full", "skip", "Local path unavailable")
else:
scope = Path(repo_path) / "SCOPE.md"
if not scope.exists():
_r("C11", "Provided Capabilities declared", "full", "skip", "SCOPE.md absent")
else:
text = scope.read_text()
has_cap_block = "```capability" in text
has_none_explicit = "## Provided Capabilities" in text and (
"none" in text.lower().split("## provided capabilities")[-1][:200]
or "no capabilities" in text.lower().split("## provided capabilities")[-1][:200]
)
if has_cap_block:
_r("C11", "Provided Capabilities declared", "full", "pass", "capability block(s) found in SCOPE.md")
elif has_none_explicit:
_r("C11", "Provided Capabilities declared", "full", "pass", "Explicitly declared none in SCOPE.md")
elif "## Provided Capabilities" in text:
_r("C11", "Provided Capabilities declared", "full", "warn",
"Section present but no capability block or explicit none — add blocks or state 'none'")
else:
_r("C11", "Provided Capabilities declared", "full", "fail",
"No '## Provided Capabilities' section in SCOPE.md")
# C12: agents template applied (CLAUDE.md mentions kaizen)
if not repo_path:
_r("C12", "Agents template applied", "full", "skip", "Local path unavailable")
else:
claude_md = Path(repo_path) / "CLAUDE.md"
if not claude_md.exists():
_r("C12", "Agents template applied", "full", "skip", "CLAUDE.md absent")
else:
text = claude_md.read_text()
if "get_kaizen_agent" in text or "kaizen" in text.lower():
_r("C12", "Agents template applied", "full", "pass")
else:
_r("C12", "Agents template applied", "full", "fail",
"CLAUDE.md has no kaizen agent reference")
# C13: consistency check clean (0 FAIL, 0 WARN — C-12 exempt)
if skip_consistency:
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "skip", "Not checked in summary mode — use /repos/{slug}/doi for full check")
else:
try:
fail, warn, _ = await _run_consistency(slug, api_base)
if fail == 0 and warn == 0:
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "pass")
elif fail == 0 and warn > 0:
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "warn",
f"{warn} warn(s) — C-12 legacy tasks may be exempt")
else:
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "fail",
f"{fail} fail(s), {warn} warn(s)")
except Exception as e:
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "skip", f"Could not run: {e}")
# C14: host paths registered
host_paths = repo.get("host_paths") or {}
if host_paths:
_r("C14", "Host paths registered", "full", "pass",
f"{len(host_paths)} host(s): {', '.join(host_paths.keys())}")
else:
_r("C14", "Host paths registered", "full", "fail",
"host_paths empty — run update_repo_path() for each active machine")
# ── Compute tier ─────────────────────────────────────────────────────────
by_id = {r.id: r for r in results}
def _tier_pass(ids: set[str]) -> bool:
return all(by_id[i].status in ("pass", "warn") for i in ids if i in by_id)
core_pass = _tier_pass(CORE_IDS)
standard_pass = core_pass and _tier_pass(STANDARD_IDS)
full_pass = standard_pass and _tier_pass(FULL_IDS)
if full_pass:
tier: Tier = "full"
elif standard_pass:
tier = "standard"
elif core_pass:
tier = "core"
else:
tier = "none"
return DoIReport(
repo_slug=slug,
tier=tier,
core_pass=core_pass,
standard_pass=standard_pass,
full_pass=full_pass,
criteria=results,
)

View File

@@ -6,7 +6,7 @@ from fastapi.middleware.cors import CORSMiddleware
from api.database import engine
from api.routers import decisions, extension_points, progress, state, tasks, technical_debt, topics, workstreams, workstream_dependencies
from api.routers import domains, repos, contributions, sbom, policy, domain_goals, repo_goals, messages, capability_requests
from api.routers import domains, repos, contributions, sbom, policy, domain_goals, repo_goals, messages, capability_requests, tpsc
@asynccontextmanager
@@ -48,6 +48,7 @@ app.include_router(contributions.router)
app.include_router(sbom.router)
app.include_router(messages.router)
app.include_router(capability_requests.router)
app.include_router(tpsc.router)
app.include_router(state.router)
app.include_router(policy.router)

View File

@@ -17,6 +17,8 @@ from api.models.sbom_entry import SBOMEntry, Ecosystem
from api.models.agent_message import AgentMessage
from api.models.capability_catalog import CapabilityCatalog
from api.models.capability_request import CapabilityRequest
from api.models.tpsc import TPSCCatalog, TPSCSnapshot, TPSCEntry
from api.models.doi_cache import DOICache
__all__ = [
"Base",
@@ -38,4 +40,6 @@ __all__ = [
"AgentMessage",
"CapabilityCatalog",
"CapabilityRequest",
"TPSCCatalog", "TPSCSnapshot", "TPSCEntry",
"DOICache",
]

View File

@@ -65,6 +65,7 @@ class CapabilityRequest(Base, TimestampMixin):
)
resolution_note: Mapped[str | None] = mapped_column(Text, nullable=True)
routing_note: Mapped[str | None] = mapped_column(Text, nullable=True)
accepted_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True
)

View File

@@ -0,0 +1,27 @@
import uuid
from datetime import datetime
from sqlalchemy import Boolean, DateTime, ForeignKey, String, Text, func
from sqlalchemy.dialects.postgresql import JSON, UUID
from sqlalchemy.orm import Mapped, mapped_column
from api.models.base import Base
class DOICache(Base):
__tablename__ = "doi_cache"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
repo_id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True),
ForeignKey("managed_repos.id", ondelete="CASCADE"),
nullable=False, unique=True, index=True,
)
tier: Mapped[str] = mapped_column(String(20), nullable=False)
core_pass: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default="false")
standard_pass: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default="false")
full_pass: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default="false")
criteria: Mapped[list | None] = mapped_column(JSON, nullable=True)
# Pipe-joined string of timestamps/mtimes used to detect staleness
fingerprint: Mapped[str] = mapped_column(Text, nullable=False)
checked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())

View File

@@ -0,0 +1,64 @@
import uuid
from datetime import datetime
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String, Text, func
from sqlalchemy.dialects.postgresql import JSON, UUID
from sqlalchemy.orm import Mapped, mapped_column, relationship
from api.models.base import Base
class TPSCCatalog(Base):
__tablename__ = "tpsc_catalog"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
slug: Mapped[str] = mapped_column(String(100), nullable=False, unique=True, index=True)
name: Mapped[str] = mapped_column(String(200), nullable=False)
provider: Mapped[str | None] = mapped_column(String(200), nullable=True)
category: Mapped[str | None] = mapped_column(String(100), nullable=True)
website_url: Mapped[str | None] = mapped_column(Text, nullable=True)
# Pricing: free | paid | freemium | usage_based | unknown
pricing_model: Mapped[str] = mapped_column(String(20), nullable=False, server_default="unknown")
# GDPR maturity (CNIL/IAPP CMMI-aligned):
# unknown | non_compliant | initial | developing | defined | managed | certified
gdpr_maturity: Mapped[str] = mapped_column(String(20), nullable=False, server_default="unknown", index=True)
gdpr_notes: Mapped[str | None] = mapped_column(Text, nullable=True)
dpa_available: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default="false")
tos_url: Mapped[str | None] = mapped_column(Text, nullable=True)
privacy_policy_url: Mapped[str | None] = mapped_column(Text, nullable=True)
data_processing_regions: Mapped[list | None] = mapped_column(JSON, nullable=True)
data_retention_notes: Mapped[str | None] = mapped_column(Text, nullable=True)
# status: active | deprecated
status: Mapped[str] = mapped_column(String(20), nullable=False, server_default="active")
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
entries: Mapped[list["TPSCEntry"]] = relationship("TPSCEntry", back_populates="catalog_entry")
class TPSCSnapshot(Base):
__tablename__ = "tpsc_snapshots"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
repo_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("managed_repos.id", ondelete="SET NULL"), nullable=True, index=True)
snapshot_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
source_file: Mapped[str | None] = mapped_column(String(200), nullable=True)
entry_count: Mapped[int] = mapped_column(Integer, nullable=False, server_default="0")
entries: Mapped[list["TPSCEntry"]] = relationship("TPSCEntry", back_populates="snapshot", cascade="all, delete-orphan")
class TPSCEntry(Base):
__tablename__ = "tpsc_entries"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
snapshot_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("tpsc_snapshots.id", ondelete="CASCADE"), nullable=False, index=True)
catalog_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("tpsc_catalog.id", ondelete="SET NULL"), nullable=True)
service_slug: Mapped[str] = mapped_column(String(100), nullable=False, index=True)
purpose: Mapped[str | None] = mapped_column(Text, nullable=True)
# auth_type: api_key | oauth | cli | none | unknown
auth_type: Mapped[str | None] = mapped_column(String(50), nullable=True)
endpoint_override: Mapped[str | None] = mapped_column(Text, nullable=True)
notes: Mapped[str | None] = mapped_column(Text, nullable=True)
snapshot: Mapped["TPSCSnapshot"] = relationship("TPSCSnapshot", back_populates="entries")
catalog_entry: Mapped["TPSCCatalog | None"] = relationship("TPSCCatalog", back_populates="entries")

View File

@@ -1,3 +1,4 @@
import re
import uuid
from datetime import datetime, timezone
@@ -16,6 +17,7 @@ from api.schemas.capability_request import (
CatalogRead,
CapabilityRequestAccept,
CapabilityRequestCreate,
CapabilityRequestPatch,
CapabilityRequestRead,
CapabilityRequestStatusPatch,
)
@@ -100,8 +102,8 @@ async def create_request(
req_domain = await _resolve_domain(body.requesting_domain, session)
# Route to provider
fulfilling_domain_id, catalog_entry_id = await _route_capability(
session, body.capability_type, body.description or ""
fulfilling_domain_id, catalog_entry_id, routing_note = await _route_capability(
session, body.capability_type, body.title, body.description or ""
)
req = CapabilityRequest(
@@ -115,6 +117,7 @@ async def create_request(
blocking_task_id=body.blocking_task_id,
fulfilling_domain_id=fulfilling_domain_id,
catalog_entry_id=catalog_entry_id,
routing_note=routing_note,
)
session.add(req)
await session.flush() # get req.id before creating notification
@@ -277,16 +280,70 @@ async def patch_request_status(
return req
@router.patch("/capability-requests/{request_id}", response_model=CapabilityRequestRead)
async def patch_request(
request_id: uuid.UUID,
body: CapabilityRequestPatch,
session: AsyncSession = Depends(get_session),
) -> CapabilityRequest:
"""Correct mutable metadata: catalog_entry_id (re-derives fulfilling domain),
priority, blocking_task_id, fulfilling_workstream_id.
Only fields present in the request body (non-None) are updated.
"""
req = await _get_request_or_404(request_id, session)
corrections: list[str] = []
if body.catalog_entry_id is not None:
old_entry_id = req.catalog_entry_id
entry = await session.get(CapabilityCatalog, body.catalog_entry_id)
if entry is None:
raise HTTPException(status_code=404, detail=f"Catalog entry '{body.catalog_entry_id}' not found")
req.catalog_entry_id = entry.id
# Re-derive fulfilling domain from catalog entry
old_domain_id = req.fulfilling_domain_id
req.fulfilling_domain_id = entry.domain_id
corrections.append(
f"catalog_entry: {old_entry_id}{entry.id} ({entry.title}); "
f"fulfilling_domain re-derived → {entry.domain_id}"
)
if body.priority is not None:
req.priority = body.priority
corrections.append(f"priority → {body.priority}")
if body.blocking_task_id is not None:
req.blocking_task_id = body.blocking_task_id
corrections.append(f"blocking_task_id → {body.blocking_task_id}")
if body.fulfilling_workstream_id is not None:
req.fulfilling_workstream_id = body.fulfilling_workstream_id
corrections.append(f"fulfilling_workstream_id → {body.fulfilling_workstream_id}")
if not corrections:
return req # no-op
correction_note = "hub correction: " + "; ".join(corrections)
req.routing_note = (req.routing_note + "\n" + correction_note) if req.routing_note else correction_note
await session.commit()
await session.refresh(req)
return req
# ---------------------------------------------------------------------------
# Routing algorithm
# ---------------------------------------------------------------------------
async def _route_capability(
session: AsyncSession, capability_type: str, description: str
) -> tuple[uuid.UUID | None, uuid.UUID | None]:
"""Find the best-matching domain for a capability request.
session: AsyncSession, capability_type: str, title: str, description: str
) -> tuple[uuid.UUID | None, uuid.UUID | None, str]:
"""Find the best-matching catalog entry for a capability request.
Returns (domain_id, catalog_entry_id) or (None, None) for broadcast.
Returns (domain_id, catalog_entry_id, routing_note).
Uses word-boundary matching on (title + description) combined to avoid
false positives from substring matches (e.g. 'postgres' inside 'postgresql',
'ha' inside 'has').
"""
q = select(CapabilityCatalog).where(
CapabilityCatalog.capability_type == capability_type,
@@ -294,20 +351,41 @@ async def _route_capability(
)
entries = list((await session.execute(q)).scalars().all())
if not entries:
return None, None, f"no active catalog entries for type '{capability_type}' — broadcast"
if len(entries) == 1:
return entries[0].domain_id, entries[0].id
e = entries[0]
return e.domain_id, e.id, f"single match: '{e.title}' (domain={e.domain_id})"
if len(entries) > 1 and description:
desc_lower = description.lower()
scored: list[tuple[int, CapabilityCatalog]] = []
for entry in entries:
score = sum(1 for kw in (entry.keywords or []) if kw.lower() in desc_lower)
scored.append((score, entry))
scored.sort(key=lambda x: -x[0])
if scored[0][0] > 0 and (len(scored) < 2 or scored[0][0] > scored[1][0]):
return scored[0][1].domain_id, scored[0][1].id
# Score by word-boundary keyword overlap against title + description combined
combined = f"{title} {description or ''}".lower()
scored: list[tuple[int, CapabilityCatalog]] = []
for entry in entries:
keywords = [kw for kw in (entry.keywords or []) if len(kw) >= 3]
score = sum(
1 for kw in keywords
if re.search(r'\b' + re.escape(kw.lower()) + r'\b', combined)
)
scored.append((score, entry))
scored.sort(key=lambda x: -x[0])
return None, None
best_score, best = scored[0]
if best_score == 0:
return None, None, (
f"no keyword overlap for type '{capability_type}' among "
f"{len(entries)} entries — broadcast"
)
if len(scored) >= 2 and scored[1][0] == best_score:
return None, None, (
f"ambiguous routing: '{scored[0][1].title}' and '{scored[1][1].title}' "
f"both scored {best_score} — broadcast"
)
return best.domain_id, best.id, (
f"matched '{best.title}' (score={best_score}, "
f"keywords matched from: {title!r})"
)
# ---------------------------------------------------------------------------

View File

@@ -1,15 +1,21 @@
import asyncio
import uuid
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy import select
from sqlalchemy import case, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from api.database import get_session
from api.doi_engine import compute_fingerprint, evaluate as _doi_evaluate
from api.models.doi_cache import DOICache
from api.models.domain import Domain
from api.models.managed_repo import ManagedRepo
from api.models.repo_goal import RepoGoal
from api.models.tpsc import TPSCSnapshot
from api.models.task import Task
from api.models.workstream import Workstream
from api.schemas.doi import DoICriterion, DoIReport, DoISummaryEntry
from api.schemas.managed_repo import (
DispatchTask,
DispatchWorkstream,
@@ -68,6 +74,220 @@ async def register_repo(
return repo
@router.get("/doi/summary", response_model=list[DoISummaryEntry])
async def doi_summary(session: AsyncSession = Depends(get_session)) -> list[DoISummaryEntry]:
"""Return DoI tier for all active repos, worst tier first.
Results are cached in doi_cache. A repo is only re-evaluated when its
fingerprint changes (repo record updated, new TPSC snapshot, goal change,
or a key file mtime changes on disk).
"""
repos_result = await session.execute(
select(ManagedRepo).where(ManagedRepo.status == "active").order_by(ManagedRepo.name)
)
repos = list(repos_result.scalars().all())
repo_ids = [r.id for r in repos]
id_to_slug = {r.id: r.slug for r in repos}
# ── Bulk DB queries for fingerprint inputs ────────────────────────────────
domains_result = await session.execute(select(Domain))
domain_obj_map = {d.id: d for d in domains_result.scalars().all()}
domain_map = {d.id: d.slug for d in domain_obj_map.values()}
domain_status = {d.slug: d.status for d in domain_obj_map.values()}
# Latest TPSC snapshot timestamp per repo (for fingerprint + C9 count)
tpsc_result = await session.execute(
select(TPSCSnapshot.repo_id,
func.count().label("cnt"),
func.max(TPSCSnapshot.snapshot_at).label("latest"))
.where(TPSCSnapshot.repo_id.in_(repo_ids))
.group_by(TPSCSnapshot.repo_id)
)
tpsc_by_id = {row.repo_id: row for row in tpsc_result}
tpsc_snap_counts = {id_to_slug[rid]: row.cnt for rid, row in tpsc_by_id.items() if rid in id_to_slug}
tpsc_snap_latest = {id_to_slug[rid]: str(row.latest) for rid, row in tpsc_by_id.items() if rid in id_to_slug}
# Latest goal updated_at + active count per repo (for fingerprint + C10)
goals_result = await session.execute(
select(RepoGoal.repo_id,
func.count().label("total"),
func.sum(case((RepoGoal.status == "active", 1), else_=0)).label("active_cnt"),
func.max(RepoGoal.updated_at).label("latest"))
.where(RepoGoal.repo_id.in_(repo_ids))
.group_by(RepoGoal.repo_id)
)
goals_by_id = {row.repo_id: row for row in goals_result}
active_goal_counts = {id_to_slug[rid]: int(row.active_cnt or 0) for rid, row in goals_by_id.items() if rid in id_to_slug}
goals_latest = {id_to_slug[rid]: str(row.latest) for rid, row in goals_by_id.items() if rid in id_to_slug}
# Load existing cache rows
cache_result = await session.execute(
select(DOICache).where(DOICache.repo_id.in_(repo_ids))
)
cache_by_repo_id = {row.repo_id: row for row in cache_result.scalars().all()}
# ─────────────────────────────────────────────────────────────────────────
prefetch = {
"domain_status": domain_status,
"tpsc_snap_counts": tpsc_snap_counts,
"active_goal_counts": active_goal_counts,
}
async def _get_or_refresh(repo: ManagedRepo) -> DoISummaryEntry:
slug = repo.slug
repo_dict = {
"slug": slug,
"domain_slug": domain_map.get(repo.domain_id),
"local_path": repo.local_path,
"remote_url": repo.remote_url,
"host_paths": repo.host_paths or {},
"last_sbom_at": str(repo.last_sbom_at) if repo.last_sbom_at else None,
"updated_at": str(repo.updated_at) if repo.updated_at else "",
}
fp = compute_fingerprint(
repo_dict,
tpsc_snap_latest.get(slug),
goals_latest.get(slug),
)
cached = cache_by_repo_id.get(repo.id)
if cached and cached.fingerprint == fp:
# Cache hit — return stored result
return DoISummaryEntry(
repo_slug=slug,
domain_slug=domain_map.get(repo.domain_id),
tier=cached.tier,
core_pass=cached.core_pass,
standard_pass=cached.standard_pass,
full_pass=cached.full_pass,
checked_at=cached.checked_at.isoformat(),
)
# Cache miss — evaluate and store
report = await _doi_evaluate(repo_dict, skip_consistency=True, prefetch=prefetch)
now = datetime.now(tz=timezone.utc)
if cached:
cached.tier = report.tier
cached.core_pass = report.core_pass
cached.standard_pass = report.standard_pass
cached.full_pass = report.full_pass
cached.criteria = [{"id": c.id, "label": c.label, "tier": c.tier,
"status": c.status, "detail": c.detail}
for c in report.criteria]
cached.fingerprint = fp
cached.checked_at = now
cached.updated_at = now
else:
session.add(DOICache(
repo_id=repo.id,
tier=report.tier,
core_pass=report.core_pass,
standard_pass=report.standard_pass,
full_pass=report.full_pass,
criteria=[{"id": c.id, "label": c.label, "tier": c.tier,
"status": c.status, "detail": c.detail}
for c in report.criteria],
fingerprint=fp,
checked_at=now,
updated_at=now,
))
return DoISummaryEntry(
repo_slug=slug,
domain_slug=domain_map.get(repo.domain_id),
tier=report.tier,
core_pass=report.core_pass,
standard_pass=report.standard_pass,
full_pass=report.full_pass,
checked_at=now.isoformat(),
)
entries: list[DoISummaryEntry] = list(await asyncio.gather(*[_get_or_refresh(r) for r in repos]))
await session.commit()
tier_order = {"none": 0, "core": 1, "standard": 2, "full": 3}
entries.sort(key=lambda e: tier_order.get(e.tier, 0))
return entries
@router.get("/{slug}/doi", response_model=DoIReport)
async def get_repo_doi(
slug: str,
force_refresh: bool = False,
session: AsyncSession = Depends(get_session),
) -> DoIReport:
"""Evaluate the 14 DoI criteria for a single repo (full check including C7/C13).
Results are cached by fingerprint. Pass ?force_refresh=true to bypass the cache.
"""
repo = await _get_repo_by_slug(slug, session)
domain_result = await session.execute(select(Domain).where(Domain.id == repo.domain_id))
domain_obj = domain_result.scalar_one_or_none()
# Fingerprint inputs for this single repo
tpsc_row = (await session.execute(
select(func.count().label("cnt"), func.max(TPSCSnapshot.snapshot_at).label("latest"))
.where(TPSCSnapshot.repo_id == repo.id)
)).one()
goal_row = (await session.execute(
select(func.max(RepoGoal.updated_at).label("latest"))
.where(RepoGoal.repo_id == repo.id)
)).one()
repo_dict = {
"slug": repo.slug,
"domain_slug": domain_obj.slug if domain_obj else None,
"local_path": repo.local_path,
"remote_url": repo.remote_url,
"host_paths": repo.host_paths or {},
"last_sbom_at": str(repo.last_sbom_at) if repo.last_sbom_at else None,
"updated_at": str(repo.updated_at) if repo.updated_at else "",
}
fp = compute_fingerprint(repo_dict, str(tpsc_row.latest) if tpsc_row.latest else None,
str(goal_row.latest) if goal_row.latest else None)
# Check cache (unless force_refresh)
cached = (await session.execute(
select(DOICache).where(DOICache.repo_id == repo.id)
)).scalar_one_or_none()
if not force_refresh and cached and cached.fingerprint == fp and cached.criteria:
return DoIReport(
repo_slug=slug,
tier=cached.tier,
core_pass=cached.core_pass,
standard_pass=cached.standard_pass,
full_pass=cached.full_pass,
checked_at=cached.checked_at.isoformat(),
criteria=[DoICriterion(**c) for c in cached.criteria],
)
# Full evaluation (includes C7/C13 consistency subprocesses)
report = await _doi_evaluate(repo_dict)
now = datetime.now(tz=timezone.utc)
criteria_json = [{"id": c.id, "label": c.label, "tier": c.tier,
"status": c.status, "detail": c.detail} for c in report.criteria]
if cached:
cached.tier = report.tier; cached.core_pass = report.core_pass
cached.standard_pass = report.standard_pass; cached.full_pass = report.full_pass
cached.criteria = criteria_json; cached.fingerprint = fp
cached.checked_at = now; cached.updated_at = now
else:
session.add(DOICache(repo_id=repo.id, tier=report.tier,
core_pass=report.core_pass, standard_pass=report.standard_pass,
full_pass=report.full_pass, criteria=criteria_json,
fingerprint=fp, checked_at=now, updated_at=now))
await session.commit()
return DoIReport(
repo_slug=report.repo_slug, tier=report.tier,
core_pass=report.core_pass, standard_pass=report.standard_pass,
full_pass=report.full_pass, checked_at=report.checked_at,
criteria=[DoICriterion(id=c.id, label=c.label, tier=c.tier,
status=c.status, detail=c.detail) for c in report.criteria],
)
@router.get("/{slug}/", response_model=RepoRead)
async def get_repo(
slug: str,

View File

@@ -0,0 +1,240 @@
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy import select, func
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import selectinload
from api.database import get_session
from api.models.managed_repo import ManagedRepo
from api.models.tpsc import TPSCCatalog, TPSCSnapshot, TPSCEntry
from api.schemas.tpsc import (
TPSCCatalogCreate, TPSCCatalogRead,
TPSCEntryRead, TPSCIngestRequest, TPSCSnapshotRead,
TPSCGDPRReport, TPSCGDPRWarning, GDPR_WARNING_LEVELS,
)
router = APIRouter(prefix="/tpsc", tags=["tpsc"])
# ---------------------------------------------------------------------------
# Catalog
# ---------------------------------------------------------------------------
@router.get("/catalog/", response_model=list[TPSCCatalogRead])
async def list_catalog(
gdpr_maturity: str | None = None,
category: str | None = None,
pricing_model: str | None = None,
session: AsyncSession = Depends(get_session),
):
q = select(TPSCCatalog).where(TPSCCatalog.status != "deprecated")
if gdpr_maturity:
q = q.where(TPSCCatalog.gdpr_maturity == gdpr_maturity)
if category:
q = q.where(TPSCCatalog.category == category)
if pricing_model:
q = q.where(TPSCCatalog.pricing_model == pricing_model)
q = q.order_by(TPSCCatalog.name)
rows = (await session.execute(q)).scalars().all()
return rows
@router.get("/catalog/{slug}", response_model=TPSCCatalogRead)
async def get_catalog_entry(slug: str, session: AsyncSession = Depends(get_session)):
row = (await session.execute(select(TPSCCatalog).where(TPSCCatalog.slug == slug))).scalar_one_or_none()
if not row:
raise HTTPException(404, f"Service '{slug}' not found in catalog")
return row
@router.post("/catalog/", response_model=TPSCCatalogRead, status_code=201)
async def register_service(body: TPSCCatalogCreate, session: AsyncSession = Depends(get_session)):
"""Register a new service or upsert an existing one by slug."""
existing = (await session.execute(select(TPSCCatalog).where(TPSCCatalog.slug == body.slug))).scalar_one_or_none()
if existing:
for k, v in body.model_dump(exclude_unset=True).items():
setattr(existing, k, v)
existing.updated_at = datetime.now(tz=timezone.utc)
await session.commit()
await session.refresh(existing)
return existing
entry = TPSCCatalog(**body.model_dump())
session.add(entry)
await session.commit()
await session.refresh(entry)
return entry
# ---------------------------------------------------------------------------
# Ingest
# ---------------------------------------------------------------------------
@router.post("/ingest/", response_model=TPSCSnapshotRead, status_code=201)
async def ingest_tpsc(body: TPSCIngestRequest, session: AsyncSession = Depends(get_session)):
"""Accept a tpsc.yaml snapshot for a repo."""
# Resolve repo_id
repo = (await session.execute(select(ManagedRepo).where(ManagedRepo.slug == body.repo_slug))).scalar_one_or_none()
repo_id = repo.id if repo else None
# Build catalog lookup by slug
slugs = {e.service_slug for e in body.entries}
catalog_rows = (await session.execute(select(TPSCCatalog).where(TPSCCatalog.slug.in_(slugs)))).scalars().all()
catalog_map = {r.slug: r for r in catalog_rows}
snapshot = TPSCSnapshot(
repo_id=repo_id,
source_file=body.source_file,
entry_count=len(body.entries),
)
session.add(snapshot)
await session.flush()
entries_with_cats = []
for e in body.entries:
cat = catalog_map.get(e.service_slug)
entry = TPSCEntry(
snapshot_id=snapshot.id,
catalog_id=cat.id if cat else None,
service_slug=e.service_slug,
purpose=e.purpose,
auth_type=e.auth_type,
endpoint_override=e.endpoint_override,
notes=e.notes,
)
session.add(entry)
entries_with_cats.append((entry, cat))
await session.flush() # assign UUIDs to all entries
await session.commit()
await session.refresh(snapshot)
entry_reads = [
TPSCEntryRead(
id=entry.id,
snapshot_id=snapshot.id,
catalog_id=cat.id if cat else None,
service_slug=entry.service_slug,
purpose=entry.purpose,
auth_type=entry.auth_type,
endpoint_override=entry.endpoint_override,
notes=entry.notes,
gdpr_maturity=cat.gdpr_maturity if cat else None,
gdpr_warning=(cat.gdpr_maturity in GDPR_WARNING_LEVELS) if cat else True,
pricing_model=cat.pricing_model if cat else None,
)
for entry, cat in entries_with_cats
]
return TPSCSnapshotRead(
id=snapshot.id,
repo_id=snapshot.repo_id,
snapshot_at=snapshot.snapshot_at,
source_file=snapshot.source_file,
entry_count=snapshot.entry_count,
entries=entry_reads,
)
# ---------------------------------------------------------------------------
# Snapshots
# ---------------------------------------------------------------------------
@router.get("/snapshots/", response_model=list[TPSCSnapshotRead])
async def list_snapshots(
repo_slug: str | None = None,
session: AsyncSession = Depends(get_session),
):
q = select(TPSCSnapshot).options(
selectinload(TPSCSnapshot.entries).selectinload(TPSCEntry.catalog_entry)
)
if repo_slug:
repo = (await session.execute(select(ManagedRepo).where(ManagedRepo.slug == repo_slug))).scalar_one_or_none()
if not repo:
raise HTTPException(404, f"Repo '{repo_slug}' not found")
q = q.where(TPSCSnapshot.repo_id == repo.id)
q = q.order_by(TPSCSnapshot.snapshot_at.desc())
rows = (await session.execute(q)).scalars().all()
result = []
for snap in rows:
entry_reads = []
for e in snap.entries:
cat = e.catalog_entry
entry_reads.append(TPSCEntryRead(
id=e.id,
snapshot_id=e.snapshot_id,
catalog_id=e.catalog_id,
service_slug=e.service_slug,
purpose=e.purpose,
auth_type=e.auth_type,
endpoint_override=e.endpoint_override,
notes=e.notes,
gdpr_maturity=cat.gdpr_maturity if cat else None,
gdpr_warning=(cat.gdpr_maturity in GDPR_WARNING_LEVELS) if cat else True,
pricing_model=cat.pricing_model if cat else None,
))
result.append(TPSCSnapshotRead(
id=snap.id,
repo_id=snap.repo_id,
snapshot_at=snap.snapshot_at,
source_file=snap.source_file,
entry_count=snap.entry_count,
entries=entry_reads,
))
return result
# ---------------------------------------------------------------------------
# GDPR report
# ---------------------------------------------------------------------------
@router.get("/report/gdpr", response_model=TPSCGDPRReport)
async def gdpr_report(session: AsyncSession = Depends(get_session)):
"""Aggregated GDPR warnings across all latest repo snapshots."""
# Latest snapshot per repo
latest_sub = (
select(TPSCSnapshot.repo_id, func.max(TPSCSnapshot.snapshot_at).label("max_at"))
.group_by(TPSCSnapshot.repo_id)
.subquery()
)
latest_snaps = (await session.execute(
select(TPSCSnapshot)
.join(latest_sub, (TPSCSnapshot.repo_id == latest_sub.c.repo_id) & (TPSCSnapshot.snapshot_at == latest_sub.c.max_at))
.options(selectinload(TPSCSnapshot.entries).selectinload(TPSCEntry.catalog_entry))
)).scalars().all()
# Repo slug lookup
all_repos = (await session.execute(select(ManagedRepo))).scalars().all()
repo_map = {r.id: r.slug for r in all_repos}
all_services = (await session.execute(select(TPSCCatalog))).scalars().all()
by_maturity: dict[str, int] = {}
for s in all_services:
by_maturity[s.gdpr_maturity] = by_maturity.get(s.gdpr_maturity, 0) + 1
warnings = []
seen = set()
for snap in latest_snaps:
repo_slug = repo_map.get(snap.repo_id) if snap.repo_id else None
for entry in snap.entries:
cat = entry.catalog_entry
maturity = cat.gdpr_maturity if cat else "unknown"
if maturity in GDPR_WARNING_LEVELS:
key = (repo_slug, entry.service_slug)
if key not in seen:
seen.add(key)
warnings.append(TPSCGDPRWarning(
repo_slug=repo_slug,
service_slug=entry.service_slug,
gdpr_maturity=maturity,
purpose=entry.purpose,
pricing_model=cat.pricing_model if cat else None,
))
return TPSCGDPRReport(
generated_at=datetime.now(tz=timezone.utc),
total_services=len(all_services),
warning_count=len(warnings),
warnings=warnings,
by_maturity=by_maturity,
)

View File

@@ -55,6 +55,13 @@ class CapabilityRequestStatusPatch(BaseModel):
note: str | None = None
class CapabilityRequestPatch(BaseModel):
catalog_entry_id: uuid.UUID | None = None
priority: str | None = None
blocking_task_id: uuid.UUID | None = None
fulfilling_workstream_id: uuid.UUID | None = None
class CapabilityRequestRead(BaseModel):
model_config = ConfigDict(from_attributes=True)
@@ -73,6 +80,7 @@ class CapabilityRequestRead(BaseModel):
blocking_task_id: uuid.UUID | None = None
catalog_entry_id: uuid.UUID | None = None
resolution_note: str | None = None
routing_note: str | None = None
accepted_at: datetime | None = None
completed_at: datetime | None = None
created_at: datetime

View File

@@ -0,0 +1,29 @@
from pydantic import BaseModel
class DoICriterion(BaseModel):
id: str
label: str
tier: str
status: str # pass | fail | warn | skip
detail: str = ""
class DoIReport(BaseModel):
repo_slug: str
tier: str # none | core | standard | full
core_pass: bool
standard_pass: bool
full_pass: bool
criteria: list[DoICriterion] = []
checked_at: str
class DoISummaryEntry(BaseModel):
repo_slug: str
domain_slug: str | None
tier: str
core_pass: bool
standard_pass: bool
full_pass: bool
checked_at: str

View File

@@ -0,0 +1,115 @@
import uuid
from datetime import datetime
from typing import Literal
from pydantic import BaseModel, computed_field
# GDPR maturity scale (CNIL/IAPP CMMI-aligned, adapted for third-party assessment)
GDPRMaturity = Literal["unknown", "non_compliant", "initial", "developing", "defined", "managed", "certified"]
# Services at these levels trigger a GDPR warning
GDPR_WARNING_LEVELS = {"unknown", "non_compliant", "initial"}
PricingModel = Literal["free", "paid", "freemium", "usage_based", "unknown"]
AuthType = Literal["api_key", "oauth", "cli", "none", "unknown"]
class TPSCCatalogCreate(BaseModel):
slug: str
name: str
provider: str | None = None
category: str | None = None
website_url: str | None = None
pricing_model: PricingModel = "unknown"
gdpr_maturity: GDPRMaturity = "unknown"
gdpr_notes: str | None = None
dpa_available: bool = False
tos_url: str | None = None
privacy_policy_url: str | None = None
data_processing_regions: list[str] | None = None
data_retention_notes: str | None = None
status: str = "active"
class TPSCCatalogRead(BaseModel):
model_config = {"from_attributes": True}
id: uuid.UUID
slug: str
name: str
provider: str | None
category: str | None
website_url: str | None
pricing_model: str
gdpr_maturity: str
gdpr_notes: str | None
dpa_available: bool
tos_url: str | None
privacy_policy_url: str | None
data_processing_regions: list[str] | None
data_retention_notes: str | None
status: str
created_at: datetime
updated_at: datetime
@computed_field
@property
def gdpr_warning(self) -> bool:
return self.gdpr_maturity in GDPR_WARNING_LEVELS
class TPSCEntryCreate(BaseModel):
service_slug: str
purpose: str | None = None
auth_type: str | None = None
endpoint_override: str | None = None
notes: str | None = None
class TPSCEntryRead(BaseModel):
model_config = {"from_attributes": True}
id: uuid.UUID
snapshot_id: uuid.UUID
catalog_id: uuid.UUID | None
service_slug: str
purpose: str | None
auth_type: str | None
endpoint_override: str | None
notes: str | None
# Denormalised from catalog for convenience
gdpr_maturity: str | None = None
gdpr_warning: bool = False
pricing_model: str | None = None
class TPSCIngestRequest(BaseModel):
repo_slug: str
source_file: str = "tpsc.yaml"
entries: list[TPSCEntryCreate]
class TPSCSnapshotRead(BaseModel):
model_config = {"from_attributes": True}
id: uuid.UUID
repo_id: uuid.UUID | None
snapshot_at: datetime
source_file: str | None
entry_count: int
entries: list[TPSCEntryRead] = []
class TPSCGDPRWarning(BaseModel):
repo_slug: str | None
service_slug: str
gdpr_maturity: str
purpose: str | None
pricing_model: str | None
class TPSCGDPRReport(BaseModel):
generated_at: datetime
total_services: int
warning_count: int
warnings: list[TPSCGDPRWarning]
by_maturity: dict[str, int]

View File

@@ -25,6 +25,7 @@ export default {
{ name: "Goals", path: "/goals" },
{ name: "Inbox", path: "/inbox" },
{ name: "Progress", path: "/progress" },
{ name: "Services (TPSC)", path: "/tpsc" },
{ name: "Todo", path: "/todo" },
// ── Sections (alphabetical) ───────────────────────────────────────────────
{
@@ -32,6 +33,7 @@ export default {
collapsible: true,
open: false,
pages: [
{ name: "Repository DoI", path: "/policy/repo-doi" },
{ name: "Workstream DoD", path: "/policy/workstream-dod" },
],
},
@@ -84,10 +86,13 @@ export default {
{ name: "Ralph Workplan", path: "/docs/ralph-workplan" },
{ name: "Reference & Context Help", path: "/docs/reference" },
{ name: "Repo Integration", path: "/docs/repo-integration" },
{ name: "State Hub", path: "/docs/state-hub" },
{ name: "Repos", path: "/docs/repos" },
{ name: "SBOM", path: "/docs/sbom" },
{ name: "SCOPE.md", path: "/docs/scope" },
{ name: "Tasks", path: "/docs/tasks" },
{ name: "TPSC", path: "/docs/tpsc" },
{ name: "TPSC — GDPR Maturity", path: "/docs/gdpr-maturity" },
{ name: "Technical Debt", path: "/docs/debt" },
{ name: "Todo", path: "/docs/todo" },
{ name: "Workstream Health", path: "/docs/workstream-health-index" },

View File

@@ -0,0 +1,176 @@
---
title: GDPR Maturity Model
---
# GDPR Maturity Model
The Custodian TPSC uses a seven-level maturity scale to rate the GDPR
compliance posture of third-party services. It is adapted from the
**CNIL / IAPP CMMI Privacy Maturity Model** for the specific purpose of
assessing external service providers rather than internal programmes.
---
## Foundations
### Source frameworks
| Framework | Authority | Levels |
|---|---|---|
| [CNIL Data Protection Maturity Model](https://iapp.org/news/b/cnil-publishes-data-protection-management-maturity-model) | French data protection authority (CNIL) | 5 (Initial → Optimized) |
| [IAPP Privacy Program Maturity Model](https://iapp.org/news/a/achieving-privacy-excellence-understanding-the-privacy-maturity-model) | International Association of Privacy Professionals | 5 (Ad Hoc → Optimized) |
| [ISO/IEC 27701:2025](https://www.iso.org/standard/27701) | ISO / IEC | Implementation tiers |
| [CMMI (Capability Maturity Model Integration)](https://cmmiinstitute.com) | CMMI Institute | 5 (Initial → Optimizing) |
Both CNIL and IAPP align on the same semantic progression: **Initial →
Repeatable → Defined → Managed → Optimized**, directly mapping to CMMI levels
15. The Custodian scale extends this with two pre-maturity states
(`unknown`, `non_compliant`) that have no CMMI equivalent but are essential
when assessing third parties with no published compliance posture.
---
## The Scale
### Level 0 — `unknown`
> No information is available about the service's GDPR compliance posture.
- No privacy policy, no ToS that addresses data processing, or the service has not been assessed yet.
- **Dashboard:** 🔴 Warning
- **Implication:** Cannot be used for any processing of personal data in a regulated environment. Treat as non-compliant until assessed.
- **CMMI equivalent:** None (pre-maturity)
---
### Level 1 — `non_compliant`
> The service has known GDPR compliance deficiencies with no indication of remediation.
- May include: data transfers to non-adequate third countries without safeguards, no privacy policy, confirmed regulatory findings, or explicit statements that GDPR does not apply.
- **Dashboard:** 🔴 Warning
- **Implication:** Must not be used for personal data processing in any EU/EEA context. Legal risk exists even for development use if real personal data is involved.
- **CMMI equivalent:** Below Level 1
---
### Level 2 — `initial`
> A basic privacy policy exists. Compliance approach is ad hoc and reactive.
- Some documentation exists but it is incomplete or generic. No formal Data Processing Agreement (DPA) is offered. Data processing practices may not be clearly defined.
- **Dashboard:** 🟠 Warning
- **Implication:** Suitable for development and prototyping with synthetic or anonymised data only. Not suitable for production processing of personal data without additional controls.
- **CMMI equivalent:** Level 1 — Initial
---
### Level 3 — `developing`
> DPA is available. Standard Contractual Clauses (SCCs) or equivalent transfer mechanisms are in place for EU→non-EU transfers.
- The service acknowledges GDPR obligations. A DPA can be signed (even if not mandatory for all tiers). Data processing regions are documented. Some controls exist but the compliance programme is not fully formalised.
- **Dashboard:** 🟡 Caution
- **Implication:** Acceptable for routine processing of personal data when a DPA has been signed. Verify transfer mechanisms and data residency before use with sensitive categories. Suitable for most B2B use cases.
- **CMMI equivalent:** Level 2 — Managed / Repeatable
---
### Level 4 — `defined`
> Formal DPA, documented SCCs or adequacy decision, clearly published data retention policy, and defined data processing practices.
- The compliance programme is documented and consistent. Data subjects' rights are implemented. Sub-processor lists are published. Processing purposes are limited and documented.
- **Dashboard:** 🟢 Compliant
- **Implication:** Suitable for general production use including personal data. Appropriate for most corporate and SME environments. Review sub-processor list for any domain-specific restrictions.
- **CMMI equivalent:** Level 3 — Defined
---
### Level 5 — `managed`
> Independently audited compliance. Quantified metrics, continuous improvement processes, and regular attestation published.
- Third-party audits (e.g. SOC 2 Type II with privacy controls, penetration testing reports, annual compliance attestations) are available. Privacy metrics are tracked and acted upon. Incident response procedures are tested.
- **Dashboard:** 🟢 Compliant
- **Implication:** Suitable for processing sensitive categories of personal data (Art. 9 GDPR). Suitable for regulated industries (healthcare, finance) subject to additional sectoral review.
- **CMMI equivalent:** Level 4 — Quantitatively Managed
---
### Level 6 — `certified`
> Formal independent certification against a recognised privacy standard.
- Examples: ISO/IEC 27701 (Privacy Information Management System), BSI C5 (for cloud services), SOC 2 Type II with GDPR-specific controls. Certification is current and scope covers the relevant services.
- **Dashboard:** 🟢 Compliant
- **Implication:** Highest available assurance. Suitable for processing of sensitive personal data at scale, public-sector use, and regulated environments with strict vendor requirements (DSGVO-compliant procurement, NHS DSPT, etc.).
- **CMMI equivalent:** Level 5 — Optimizing
---
## Summary Table
| Level | Code | Label | GDPR Warning | CMMI | Suitable for personal data? |
|---|---|---|---|---|---|
| 0 | `unknown` | Unknown | ✅ Yes | — | ❌ No |
| 1 | `non_compliant` | Non-Compliant | ✅ Yes | — | ❌ No |
| 2 | `initial` | Initial | ✅ Yes | L1 | ⚠ Synthetic/anonymised only |
| 3 | `developing` | Developing | — | L2 | ✅ With signed DPA |
| 4 | `defined` | Defined | — | L3 | ✅ General use |
| 5 | `managed` | Managed | — | L4 | ✅ Sensitive categories |
| 6 | `certified` | Certified | — | L5 | ✅ Regulated environments |
**GDPR warnings** are raised by the dashboard and `get_gdpr_report()` for any service at level 02 (`unknown`, `non_compliant`, `initial`).
---
## Key GDPR Concepts Referenced
**DPA (Data Processing Agreement)** — A contract required by GDPR Art. 28 when a controller engages a processor. The DPA defines the subject-matter, duration, nature and purpose of processing, and the obligations of both parties.
**SCCs (Standard Contractual Clauses)** — Commission-approved contract clauses enabling lawful transfer of personal data from the EU/EEA to third countries without an adequacy decision. Updated SCCs published June 2021 (implementing decisions 2021/914 and 2021/915).
**Adequacy Decision** — A European Commission finding that a third country provides an essentially equivalent level of data protection (e.g. UK GDPR, Japan, Canada PIPEDA). Transfers to adequate countries do not require additional safeguards.
**BCRs (Binding Corporate Rules)** — Internal rules allowing multinationals to transfer personal data within their group across borders. Approved by a lead supervisory authority.
**Sensitive Categories (Art. 9)** — Health, biometric, genetic, racial/ethnic origin, political opinions, religious beliefs, trade union membership, sexual orientation. Require explicit consent or other specific legal basis.
---
## Assigning a Maturity Level
When adding a new service to `canon/tpsc/`, follow this decision process:
```
Is a privacy policy published?
No → unknown or non_compliant
Is a DPA available (even on request)?
No → initial
Yes → developing (minimum)
Are SCCs or adequacy mechanisms documented?
No → developing
Yes, and retention policy published → defined
Are independent audit reports published (SOC 2 Type II, etc.)?
Yes → managed
Is an ISO 27701 or equivalent certification current?
Yes → certified
```
When uncertain between two levels, assign the **lower** level. Err on the side of caution.
---
## References
- CNIL: [Le modèle de maturité de la protection des données](https://www.cnil.fr/fr/le-modele-de-maturite-de-la-protection-des-donnees)
- IAPP: [Achieving privacy excellence — understanding the privacy maturity model](https://iapp.org/news/a/achieving-privacy-excellence-understanding-the-privacy-maturity-model)
- ISO/IEC 27701:2025: [Privacy information management — Requirements and guidelines](https://www.iso.org/standard/27701)
- European Commission SCCs (2021): [Implementing Decision 2021/914](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32021D0914)
- EDPB Guidelines on SCCs: [Guidelines 04/2021](https://edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-042021-standard-contractual-clauses_en)
- CMMI Institute: [CMMI Model Overview](https://cmmiinstitute.com/cmmi)

View File

@@ -0,0 +1,269 @@
---
title: State Hub — Reference
---
# The Custodian State Hub
The State Hub is the central nervous system of the Custodian ecosystem — a
local-first service that collects, organises, and surfaces the state of all
registered repositories so that humans and agents can orient themselves in
seconds rather than minutes.
---
## Why it exists
Software projects accumulate invisible state. Workstreams stall, decisions go
unresolved, dependency licences drift, integration gaps widen — and none of
this is visible without opening every file in every repository. For a single
repo with a single engineer this is manageable. Across six domains, fifteen
repositories, and a growing fleet of autonomous agents it becomes untenable.
The State Hub exists to answer one question instantly:
> **"What is the actual state of the system right now, and what should happen next?"**
It does this without requiring any project to change how it works — repositories
remain the authority for their own data and the hub is a silent observer that
indexes and reflects their state.
---
## What it is — and what it is not
### What it is
| Role | Description |
|---|---|
| **Derived Data Store** | All hub data is computed from repo files and records. The hub holds no original information — it can be wiped and rebuilt from scratch at any time without data loss. |
| **Read Model** | Provides fast, pre-computed answers to common orientation queries: active workstreams, blocking decisions, DoI compliance tiers, SBOM licence risk, GDPR warnings. |
| **Agent Orchestration Layer** | Exposes an MCP server (Model Context Protocol) so that Claude Code sessions in any registered repository can orient themselves, record progress, resolve decisions, and coordinate with each other — all through a uniform tool interface. |
| **Cross-Repo Observatory** | The only place where data from all repositories is visible together. Detects dependencies between workstreams, licence risks that span repos, and integration gaps that no single repo can see about itself. |
### What it is not
| Not | Why |
|---|---|
| **Source of truth** | Repos are the authority. The hub reflects; it does not own. |
| **A project management tool** | Workplans live in repos as Markdown files. The hub indexes them, not the reverse. |
| **A deployment or CI/CD system** | The hub tracks *intent and state*, not execution pipelines. |
| **A multi-user SaaS** | Local-first, single-operator. Designed for sovereignty, not scale. |
---
## The Derived Data Store principle
The hub is a concrete implementation of what Martin Kleppmann calls a
**Derived Data Store** (*Designing Data-Intensive Applications*, Ch. 3 & 11):
a system whose entire dataset is fully derivable from upstream sources and
which therefore makes no authority claims of its own.
This is formalised in **ADR-003** (Materialized Derived State with Fingerprint
Invalidation). The practical consequence:
1. **Repositories are the primary artefacts.** SCOPE.md, workplan files,
`uv.lock`, `tpsc.yaml`, `CLAUDE.md` — these are canonical. The hub reads
them; it never writes them.
2. **The hub is a cache.** Every table that holds repo-derived data carries a
`fingerprint` column. When the fingerprint of the current source state
matches the stored fingerprint, the cached value is returned instantly.
When it differs, the value is recomputed and the cache is updated.
3. **The rebuild guarantee.** Any hub database can be dropped and recreated
from scratch by re-running migrations and re-ingesting all registered repos.
No information is lost because no information originates here.
4. **Force-refresh on demand.** Every derived endpoint accepts
`?force_refresh=true` to bypass the cache — useful for debugging, post-ingest
verification, and validating fingerprint coverage.
### What gets derived and how
| Source | Derived data | Mechanism |
|---|---|---|
| `uv.lock`, `package-lock.json`, etc. | SBOM entries + licence risk | `make ingest-sbom REPO=` |
| `tpsc.yaml` | Third-party service declarations + GDPR warnings | `make ingest-tpsc REPO=` |
| `SCOPE.md` capability blocks | Capability catalog | `make ingest-capabilities REPO=` |
| `workplans/*.md` | Workstream + task status | `make fix-consistency REPO=` |
| Repo files + DB records | DoI compliance tier | Fingerprint cache, auto-refreshed on read |
---
## The Repository Orchestrator
Beyond indexing, the hub acts as an **orchestrator** for the repository
ecosystem — the coordinator that no individual repo can be.
### Agent session protocol
Every Claude Code session in a registered repository follows the same ritual:
1. **Orient** — call `get_domain_summary("<slug>")` to load active workstreams,
pending tasks, blocking decisions, and suggested next steps for this domain.
2. **Check inbox** — call `get_messages(to_agent="<name>", unread_only=True)`
to receive coordination messages from other agents or from prior sessions.
3. **Work** — use MCP tools to record progress, resolve decisions, update task
status, and send messages to other agents.
4. **Close** — call `add_progress_event()` before ending the session so the
next session can see what happened.
This protocol means that any agent — human-directed or autonomous — arrives
in a fresh session with full context about the current state of its domain
in under a second.
### Cross-domain coordination
Because all agents share the same hub, they can coordinate without direct
communication:
- An agent working on `railiance` can send a message to `custodian` via
`send_message()`. The custodian agent reads it in its next session inbox.
- A capability request (`request_capability()`) routes to the domain that
advertises the relevant capability in its `SCOPE.md`. The fulfilling agent
accepts it, does the work, and marks it complete — unblocking the requesting
workstream automatically.
- Workstream dependencies (`create_dependency()`) let the hub surface "what
is blocking what" across repos that have never directly communicated.
### Kaizen agents
The hub hosts a library of specialised agent personas (`agents/agent-*.md`)
that any session can load via `get_kaizen_agent("<name>")`. These provide
expert instruction sets for specific tasks — test-driven development,
refactoring, infrastructure review — without baking domain knowledge into
every repo's CLAUDE.md.
---
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ Repositories │
│ workplans/*.md SCOPE.md CLAUDE.md uv.lock tpsc.yaml │
└─────────┬───────────────────────────────────────────────────┘
│ ingest scripts (make ingest-sbom / fix-consistency / …)
┌─────────────────────────────────────────────────────────────┐
│ PostgreSQL Database │
│ domains managed_repos workstreams tasks decisions │
│ sbom_entries tpsc_entries doi_cache capability_catalog │
│ progress_events agent_messages repo_goals … │
└─────────┬───────────────────────────────────────────────────┘
┌────┴──────────────┐
│ │
▼ ▼
┌─────────────┐ ┌─────────────────────────────────────────┐
│ FastAPI │ │ MCP Server (SSE :8001) │
│ REST API │ │ 40+ tools — orient, record, coordinate │
│ (:8000) │ │ Registered at user scope in ~/.claude │
└─────────────┘ └─────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ Observable Framework Dashboard (:3000) │
│ Overview Repositories Workstreams Decisions SBOM │
│ TPSC Contributions Goals Capabilities … │
└─────────────────────────────────────────────────────────────┘
```
### Technology choices
| Component | Technology | Reason |
|---|---|---|
| Database | PostgreSQL (Docker) | Relational integrity, JSON columns, full async support |
| API | FastAPI + SQLAlchemy (async) | Type-safe, auto-documented, fast |
| MCP server | FastMCP over SSE | Persistent connection; no Claude Code restart on tool update |
| Dashboard | Observable Framework | Reactive, static-deployable, no JS build step for data loaders |
| Agent interface | Model Context Protocol | Standard Claude Code integration; tools available in all sessions |
---
## Data model overview
The hub's schema is organised in concentric layers:
**Governance (slow-changing):**
`domains``managed_repos``repo_goals``domain_goals`
**Work tracking (active):**
`workstreams``tasks``workstream_dependencies`
**Decision log (append-only):**
`decisions` · `progress_events` · `agent_messages`
**Derived snapshots (cached, rebuildable):**
`sbom_snapshots``sbom_entries`
`tpsc_snapshots``tpsc_entries`
`capability_catalog` · `capability_requests`
`doi_cache`
**Governance policies (editables):**
`policies/` (flat Markdown files, served via `/policy/<name>`)
---
## Running the hub
```bash
cd ~/the-custodian/state-hub
make db # Start PostgreSQL (Docker)
make migrate # Alembic upgrade head
make seed # Insert 6 canonical domain topics
make api # FastAPI on :8000
make mcp-http # MCP SSE server on :8001
make dashboard # Observable preview on :3000
```
Verify the MCP server is registered:
```bash
python3 -c "import json,os; d=json.load(open(os.path.expanduser('~/.claude.json'))); print(list(d.get('mcpServers',{}).keys()))"
# → ['state-hub']
```
Re-register if needed:
```bash
claude mcp add-json -s user state-hub '{"type":"sse","url":"http://127.0.0.1:8001/sse"}'
```
---
## Design principles
These principles are not aspirational — they are constraints that every hub
feature must satisfy before being accepted.
**Local-first, no vendor lock-in.**
The hub runs on a developer laptop. It requires only Docker (for Postgres)
and Python. It works offline. No cloud dependency, no SaaS subscription.
**Sovereignty by default.**
No data leaves the machine unless explicitly exported. The hub never calls
external APIs on its own.
**The rebuild guarantee (ADR-001, ADR-003).**
Dropping the database and re-running `make migrate && make seed` followed by
re-ingesting all repos must restore the full operational state. Any feature
that breaks this guarantee is rejected.
**Agents are co-creators, not authorities.**
The hub coordinates agent work but does not permit agents to make irreversible
decisions unilaterally. Financial, legal, and external-publication actions are
hard-blocked (see `canon/constitution/`).
**Append-only episodic memory.**
Progress events are never deleted or edited. Decisions are resolved, not
overwritten. This ensures that any session can reconstruct what happened and
why, even years later.
---
## Related
- [ADR-001](/docs/repo-integration) — Workplans as Repository Artefacts
- [ADR-003](https://github.com) — Materialized Derived State (see `canon/architecture/adr-003-materialized-derived-state.md`)
- [Connecting to the Hub](/docs/connecting)
- [Repo Integration](/docs/repo-integration)
- [Repository DoI](/policy/repo-doi) — Definition of Integrated
- [TPSC](/docs/tpsc) — Third-Party Services Catalog
- [SBOM](/docs/sbom)

View File

@@ -0,0 +1,136 @@
---
title: Third-Party Services Catalog (TPSC)
---
# Third-Party Services Catalog (TPSC)
The TPSC tracks external service dependencies (APIs, SaaS, CLIs) across all
registered repos — complementing the SBOM for package dependencies.
---
## Why TPSC?
Package lockfiles capture Python/JS/Rust dependencies but miss the external
HTTP services your code calls. These carry compliance, cost, and privacy
implications that are invisible to standard SBOM tooling.
TPSC provides:
- A registry of which repos use which external services
- GDPR compliance maturity ratings per service
- Pricing model tracking (paid/usage-based costs)
- Data processing region and retention information
- GDPR warnings for services not suitable in regulated environments
---
## Primary Data Locations
Following ADR-001 (workplans as repo artefacts), TPSC data lives in two places:
| Location | Purpose |
|---|---|
| `<repo>/tpsc.yaml` | Declares which services the repo uses |
| `the-custodian/canon/tpsc/<slug>.yaml` | Canonical service metadata (ToS, GDPR, pricing) |
The state-hub is a collector — it can be rebuilt from scratch by re-ingesting
all `tpsc.yaml` files and re-seeding the catalog from canon files.
---
## tpsc.yaml Format
```yaml
# tpsc.yaml — Third-Party Services Catalog declarations
# Ingest: cd state-hub && make ingest-tpsc REPO=<slug>
services:
- slug: openai-api # Must match a slug in canon/tpsc/
purpose: LLM inference via OpenAI-compatible API
auth: api_key # api_key | oauth | cli | none | unknown
- slug: stripe
purpose: Payment processing
auth: api_key
endpoint: https://api.stripe.com # Optional override if non-standard
notes: Only used in production tier
```
---
## Canon Service File Format
```yaml
# canon/tpsc/openai-api.yaml
slug: openai-api
name: OpenAI API
provider: OpenAI, Inc.
category: llm_inference # llm_inference | storage | payments | search | etc.
website_url: https://openai.com
pricing_model: usage_based # free | paid | freemium | usage_based | unknown
gdpr_maturity: developing # See scale below
gdpr_notes: >
DPA available. SCCs for EU→US transfer. 30-day retention for safety.
dpa_available: true
tos_url: https://openai.com/policies/terms-of-use
privacy_policy_url: https://openai.com/policies/privacy-policy
data_processing_regions:
- us
data_retention_notes: >
30 days default; zero-retention available on eligible endpoints.
status: active
```
---
## GDPR Maturity Scale
Based on the **CNIL / IAPP CMMI Privacy Maturity Model**, adapted for
third-party service assessment:
| Level | Name | Description | Dashboard |
|---|---|---|---|
| 0 | `unknown` | No information about GDPR stance | 🔴 Warning |
| 1 | `non_compliant` | Known GDPR issues, no remediation | 🔴 Warning |
| 2 | `initial` | Basic privacy policy only, ad hoc approach | 🟠 Warning |
| 3 | `developing` | DPA available, some controls, SCCs provided | 🟡 |
| 4 | `defined` | Formal DPA, SCCs documented, clear retention policy | 🟢 |
| 5 | `managed` | Independently audited, metrics tracked | 🟢 |
| 6 | `certified` | ISO 27701 / SOC2 privacy certified | 🟢 |
Services at levels 02 (**Warning**) may limit use in GDPR-regulated or
corporate environments. At minimum, `developing` is needed for routine
processing of personal data with an API provider.
Reference: [CNIL GDPR maturity model](https://iapp.org/news/b/cnil-publishes-data-protection-management-maturity-model), [IAPP Privacy Maturity Model](https://iapp.org/news/a/achieving-privacy-excellence-understanding-the-privacy-maturity-model)
---
## Adding a New Service
1. Create `the-custodian/canon/tpsc/<slug>.yaml` following the format above
2. Seed it into the state-hub: `cd state-hub && make api` then POST to `/tpsc/catalog/`
(or use the MCP tool: `register_service(slug=..., ...)`)
3. Add it to your repo's `tpsc.yaml`
4. Ingest: `make ingest-tpsc REPO=<slug>`
---
## MCP Tools
| Tool | Purpose |
|---|---|
| `register_service(slug, ...)` | Add/update a service in the catalog |
| `list_services(gdpr_maturity?, category?, pricing_model?)` | Browse catalog |
| `ingest_tpsc_tool(repo_slug)` | Parse tpsc.yaml and ingest snapshot |
| `get_gdpr_report()` | GDPR warning summary across all repos |
---
## Makefile Targets
```bash
make ingest-tpsc REPO=llm-connect # Ingest single repo
make ingest-tpsc-all # Ingest all repos
make ingest-tpsc REPO=llm-connect DRY_RUN=1 # Preview only
```

View File

@@ -0,0 +1,90 @@
---
title: Repository Definition of Integrated (DoI)
---
```js
import {API} from "../components/config.js";
```
```js
import {marked} from "npm:marked";
const _resp = await fetch(`${API}/policy/repo-doi`);
if (!_resp.ok) throw new Error(`Failed to load policy: ${_resp.status}`);
const _policy = await _resp.json();
```
```js
let _content = _policy.content;
let _editing = false;
const _root = display(html`<div></div>`);
async function _save(text) {
const r = await fetch(`${API}/policy/repo-doi`, {
method: "PUT",
headers: {"Content-Type": "application/json"},
body: JSON.stringify({content: text}),
});
if (!r.ok) throw new Error(`Save failed: ${r.status}`);
_content = text;
}
function _toolbar(...nodes) {
return html`<div style="display:flex;gap:0.5rem;margin-bottom:1rem">${nodes}</div>`;
}
function _btn(label, primary = false) {
return html`<button style="
padding:0.35rem 0.9rem;border-radius:4px;cursor:pointer;font-size:13px;
background:${primary ? "#1e293b" : "#f1f5f9"};
color:${primary ? "#f8fafc" : "#1e293b"};
border:1px solid ${primary ? "#1e293b" : "#cbd5e1"};
">${label}</button>`;
}
function _render() {
_root.innerHTML = "";
if (_editing) {
const area = html`<textarea style="
width:100%;box-sizing:border-box;height:520px;
font-family:ui-monospace,monospace;font-size:13px;line-height:1.6;
padding:0.75rem;border:1px solid #cbd5e1;border-radius:4px;
background:#f8fafc;color:#1e293b;resize:vertical;
">${_content}</textarea>`;
const saveBtn = _btn("Save", true);
const cancelBtn = _btn("Cancel");
saveBtn.onclick = async () => {
saveBtn.disabled = true;
saveBtn.textContent = "Saving…";
try {
await _save(area.value);
_editing = false;
_render();
} catch (e) {
saveBtn.disabled = false;
saveBtn.textContent = "Save";
alert(e.message);
}
};
cancelBtn.onclick = () => { _editing = false; _render(); };
_root.append(_toolbar(saveBtn, cancelBtn), area);
} else {
const editBtn = _btn("Edit");
editBtn.onclick = () => { _editing = true; _render(); };
const body = html`<div style="max-width:720px;line-height:1.7;"></div>`;
body.innerHTML = marked.parse(_content);
_root.append(_toolbar(editBtn), body);
}
}
_render();
```

View File

@@ -41,6 +41,19 @@ convention used in the Custodian State Hub.
---
## Architecture & Design
| Topic | What it covers |
|-------|---------------|
| [State Hub](/docs/state-hub) | Why/how/what — Derived Data Store principle, orchestrator role, architecture diagram, design principles |
| [TPSC](/docs/tpsc) | Third-Party Services Catalog — tpsc.yaml format, ingest, MCP tools |
| [TPSC — GDPR Maturity](/docs/gdpr-maturity) | 7-level CNIL/IAPP scale, per-level guidance, key GDPR concepts |
| [SCOPE.md](/docs/scope) | The scope file format and how to write one |
| [Capabilities](/docs/capabilities) | Capability catalog, request routing, MCP tools |
| [Goals](/docs/goals) | Domain and repo goals, priority ordering |
---
## Meta
| Topic | What it covers |

View File

@@ -1,5 +1,5 @@
---
title: Repos
title: Repositories
---
```js
@@ -7,6 +7,7 @@ import {API} from "./components/config.js";
```
```js
// Fast data — page renders as soon as this resolves (~200ms)
let _repos = [], _domains = [], _sbom = [], _eps = [], _tds = [], _workstreams = [];
try {
[_repos, _domains, _sbom, _eps, _tds, _workstreams] = await Promise.all([
@@ -20,6 +21,18 @@ try {
} catch {}
```
```js
// DoI data — lazy-loaded. Starts as [] so the page renders immediately.
// When the fetch resolves (~6s), doiData updates and DoI badges appear.
import {Mutable} from "observablehq:stdlib";
const doiData = Mutable([]);
const doiLoading = Mutable(true);
fetch(`${API}/repos/doi/summary`)
.then(r => r.ok ? r.json() : [])
.catch(() => [])
.then(data => { doiData.value = data; doiLoading.value = false; });
```
```js
const repos = _repos ?? [];
const domains = _domains ?? [];
@@ -27,6 +40,14 @@ const sbom = _sbom ?? [];
const eps = _eps ?? [];
const tds = _tds ?? [];
const workstreams = _workstreams ?? [];
const doi = doiData; // reactive — updates when lazy fetch completes
// DoI lookups
const doiBySlug = Object.fromEntries(doi.map(d => [d.repo_slug, d]));
const DOI_TIER_ORDER = {none: 0, core: 1, standard: 2, full: 3};
const DOI_TIER_COLOR = {none: "#ef4444", core: "#f97316", standard: "#eab308", full: "#22c55e"};
const DOI_TIER_BG = {none: "#fef2f2", core: "#fff7ed", standard: "#fefce8", full: "#f0fdf4"};
const DOI_TIER_LABEL = {none: "None", core: "Core", standard: "Standard", full: "Full"};
// Lookups
const domainById = Object.fromEntries(domains.map(d => [d.id, d]));
@@ -79,11 +100,14 @@ const repoRows = repos
? new Date(r.last_sbom_at).toLocaleDateString()
: (sbomData?.snapshot_at ? new Date(sbomData.snapshot_at).toLocaleDateString() : null);
const integrating = !!integratingBySlug[r.slug];
const doiEntry = doiBySlug[r.slug] ?? null;
const doiTier = doiEntry?.tier ?? "none";
return {
_id: r.id,
_domSlug: domSlug,
_hasSbom: hasSbom,
_integrating: integrating,
_doiTier: doiTier,
repo: r.slug,
domain: domName,
status: integrating ? "⚙ integrating" : "ready",
@@ -96,17 +120,29 @@ const repoRows = repos
})
.sort((a, b) => a._domSlug.localeCompare(b._domSlug) || a.repo.localeCompare(b.repo));
const gapCount = repoRows.filter(r => !r._hasSbom).length;
const coveredCount = repoRows.filter(r => r._hasSbom).length;
const gapCount = repoRows.filter(r => !r._hasSbom).length;
const coveredCount = repoRows.filter(r => r._hasSbom).length;
const integratingCount = repoRows.filter(r => r._integrating).length;
const doiFullCount = repoRows.filter(r => r._doiTier === "full").length;
const doiNoneCount = repoRows.filter(r => r._doiTier === "none").length;
```
# Repos
# Repositories
```js
import {withDocHelp} from "./components/doc-overlay.js";
const _h1 = document.querySelector("#observablehq-main h1");
if (_h1) { _h1.style.position = "relative"; withDocHelp(_h1, "/docs/repos"); }
display(html`<p style="font-size:0.85rem;color:#6b7280;margin-top:-0.5rem;display:flex;align-items:center;gap:0.75rem;flex-wrap:wrap;">
<span>DoI tiers: <strong style="color:#ef4444;">None</strong> →
<strong style="color:#f97316;">Core</strong> →
<strong style="color:#eab308;">Standard</strong> →
<strong style="color:#22c55e;">Full</strong> —
<a href="/policy/repo-doi" style="color:#1d4ed8;">Definition of Integrated policy ↗</a></span>
${doiLoading ? html`<span style="display:inline-flex;align-items:center;gap:0.35rem;color:#9ca3af;font-size:0.8rem;">
<span class="doi-spinner"></span> Loading DoI tiers…
</span>` : html`<span style="color:#16a34a;font-size:0.8rem;">✓ DoI tiers loaded</span>`}
</p>`);
```
```js
@@ -134,6 +170,11 @@ display(html`<div class="kpi-row">
<p class="big-num">${gapCount}</p>
<small>${gapCount === 0 ? "✓ All repos covered" : `${gapCount} repo(s) not ingested`}</small>
</div>
<div class="card ${doiLoading ? '' : doiNoneCount > 0 ? 'card-warn' : 'card-ok'}">
<h3>DoI: Fully Integrated</h3>
<p class="big-num">${doiLoading ? html`<span class="doi-spinner" style="width:1.4rem;height:1.4rem;"></span>` : `${doiFullCount} / ${repoRows.length}`}</p>
<small>${doiLoading ? "Loading…" : doiNoneCount > 0 ? `${doiNoneCount} at tier None` : "✓ All pass Core tier"}</small>
</div>
</div>`);
```
@@ -147,6 +188,16 @@ function _sbomGap() {
return el;
}
function _doiBadge(tier) {
if (doiLoading) return html`<span style="color:#d1d5db;font-size:0.72rem;">…</span>`;
const color = DOI_TIER_COLOR[tier] || "#9ca3af";
const bg = DOI_TIER_BG[tier] || "#f9fafb";
const label = DOI_TIER_LABEL[tier] || tier;
return html`<span style="background:${bg}; color:${color}; border:1px solid ${color}60;
border-radius:4px; padding:1px 7px; font-size:0.72rem; font-weight:700; white-space:nowrap;">
DoI: ${label}</span>`;
}
// Group by domain
const byDomain = {};
for (const r of repoRows) {
@@ -162,6 +213,9 @@ if (domainBlocks.length === 0) {
${domainBlocks.map(([slug, rows]) => {
const dom = domainBySlug[slug];
const allCovered = rows.every(r => r._hasSbom);
const doiWorst = rows.map(r => DOI_TIER_ORDER[r._doiTier] ?? 0);
const doiMin = Math.min(...doiWorst);
const doiMinKey = Object.keys(DOI_TIER_ORDER).find(k => DOI_TIER_ORDER[k] === doiMin) ?? "none";
const hasEps = (epByDomain[slug] ?? 0) > 0;
const hasTds = (tdByDomain[slug] ?? 0) > 0;
return html`
@@ -178,11 +232,15 @@ if (domainBlocks.length === 0) {
${hasTds
? html`<span class="chip chip-ok">TDs: ${tdByDomain[slug]}</span>`
: html`<span class="chip chip-neutral">TDs: —</span>`}
<span style="font-size:0.72rem; color:${DOI_TIER_COLOR[doiMinKey]}; font-weight:600;">
DoI min: ${DOI_TIER_LABEL[doiMinKey]}
</span>
</span>
</div>
<table class="repo-table">
<thead><tr>
<th>Repo</th>
<th>DoI Tier</th>
<th>Status</th>
<th>SBOM</th>
<th>Packages</th>
@@ -191,6 +249,7 @@ if (domainBlocks.length === 0) {
<tbody>
${rows.map(r => html`<tr class="${r._integrating ? 'row-integrating' : r._hasSbom ? '' : 'row-gap'}">
<td class="repo-cell"><code>${r.repo}</code></td>
<td>${_doiBadge(r._doiTier)}</td>
<td>${r._integrating
? html`<span class="chip chip-integrating">⚙ integrating</span>`
: html`<span class="chip chip-ok">ready</span>`}</td>
@@ -211,26 +270,29 @@ if (domainBlocks.length === 0) {
```js
const domainFilter = Inputs.select(["all", ...new Set(repoRows.map(r => r._domSlug)).values()], {label: "Domain", value: "all"});
const doiFilter = Inputs.select(["all", "none", "core", "standard", "full"], {label: "DoI tier", value: "all"});
const gapFilter = Inputs.toggle({label: "Gaps only (no SBOM)", value: false});
display(html`<div style="display:flex;gap:1rem;flex-wrap:wrap;margin-bottom:1rem">${domainFilter}${gapFilter}</div>`);
display(html`<div style="display:flex;gap:1rem;flex-wrap:wrap;margin-bottom:1rem">${domainFilter}${doiFilter}${gapFilter}</div>`);
```
```js
const filteredRows = repoRows.filter(r =>
(domainFilter.value === "all" || r._domSlug === domainFilter.value) &&
(doiFilter.value === "all" || r._doiTier === doiFilter.value) &&
(!gapFilter.value || !r._hasSbom)
);
display(Inputs.table(filteredRows.map(r => ({
Repo: r.repo,
Domain: r.domain,
Status: r.status,
SBOM: r.sbom,
Pkgs: r.pkgs,
Repo: r.repo,
Domain: r.domain,
"DoI Tier": DOI_TIER_LABEL[r._doiTier] ?? r._doiTier,
Status: r.status,
SBOM: r.sbom,
Pkgs: r.pkgs,
"EPs (domain)": r.eps || "—",
"TDs (domain)": r.tds || "—",
Path: r.path,
})), {maxWidth: 1100}));
Path: r.path,
})), {maxWidth: 1200}));
```
## Onboard a New Repo
@@ -317,4 +379,12 @@ custodian register-project --domain &lt;slug&gt;</pre>
.onboard-step strong { font-size: 0.9rem; display: block; margin-bottom: 0.3rem; }
.onboard-step pre { background: var(--theme-background); border-radius: 4px; padding: 0.4rem 0.7rem; font-size: 0.8rem; overflow-x: auto; margin: 0 0 0.35rem; }
.onboard-note { font-size: 0.82rem; color: var(--theme-foreground-muted, gray); margin: 0; line-height: 1.45; }
.doi-spinner {
display: inline-block; width: 0.9rem; height: 0.9rem;
border: 2px solid #e5e7eb; border-top-color: #9ca3af;
border-radius: 50%; animation: doi-spin 0.7s linear infinite;
vertical-align: middle;
}
@keyframes doi-spin { to { transform: rotate(360deg); } }
</style>

View File

@@ -0,0 +1,193 @@
---
title: Third-Party Services (TPSC)
---
# Third-Party Services Catalog
```js
const API = "http://127.0.0.1:8000";
let apiOk = true;
const catalog = await fetch(`${API}/tpsc/catalog/`)
.then(r => r.json())
.catch(() => { apiOk = false; return []; });
const gdprReport = await fetch(`${API}/tpsc/report/gdpr`)
.then(r => r.json())
.catch(() => ({ warnings: [], by_maturity: {}, total_services: 0, warning_count: 0 }));
const snapshots = await fetch(`${API}/tpsc/snapshots/`)
.then(r => r.json())
.catch(() => []);
```
```js
// GDPR maturity colour coding (CNIL/IAPP scale)
const maturityColor = {
unknown: "#ef4444", // red
non_compliant: "#dc2626", // deep red
initial: "#f97316", // orange
developing: "#eab308", // amber
defined: "#84cc16", // lime
managed: "#22c55e", // green
certified: "#16a34a", // deep green
};
const maturityLabel = {
unknown: "Unknown",
non_compliant: "Non-Compliant",
initial: "Initial",
developing: "Developing",
defined: "Defined",
managed: "Managed",
certified: "Certified",
};
const WARNING_LEVELS = new Set(["unknown", "non_compliant", "initial"]);
```
```js
// KPI summary
const warningServices = catalog.filter(s => WARNING_LEVELS.has(s.gdpr_maturity));
const paidServices = catalog.filter(s => ["paid", "usage_based"].includes(s.pricing_model));
```
<div style="display:flex; gap:1.5rem; flex-wrap:wrap; margin-bottom:2rem;">
<div style="background:#fef2f2; border:1px solid #fca5a5; border-radius:8px; padding:1rem 1.5rem; min-width:160px;">
<div style="font-size:2rem; font-weight:700; color:#dc2626;">${gdprReport.warning_count}</div>
<div style="font-size:0.85rem; color:#6b7280;">GDPR warnings</div>
</div>
<div style="background:#f0fdf4; border:1px solid #86efac; border-radius:8px; padding:1rem 1.5rem; min-width:160px;">
<div style="font-size:2rem; font-weight:700; color:#16a34a;">${gdprReport.total_services}</div>
<div style="font-size:0.85rem; color:#6b7280;">Services in catalog</div>
</div>
<div style="background:#fffbeb; border:1px solid #fcd34d; border-radius:8px; padding:1rem 1.5rem; min-width:160px;">
<div style="font-size:2rem; font-weight:700; color:#b45309;">${paidServices.length}</div>
<div style="font-size:0.85rem; color:#6b7280;">Paid / usage-based</div>
</div>
</div>
---
## Service Catalog
```js
import {html} from "npm:htl";
function maturityBadge(m) {
const color = maturityColor[m] || "#9ca3af";
const label = maturityLabel[m] || m;
return html`<span style="background:${color}20; color:${color}; border:1px solid ${color}60;
border-radius:4px; padding:2px 8px; font-size:0.78rem; font-weight:600; white-space:nowrap;">${label}</span>`;
}
function pricingBadge(p) {
const colors = { paid: "#7c3aed", usage_based: "#7c3aed", freemium: "#0369a1", free: "#166534", unknown: "#6b7280" };
const c = colors[p] || "#6b7280";
return html`<span style="color:${c}; font-size:0.78rem; font-weight:500;">${p.replace("_", " ")}</span>`;
}
const catalogTable = html`<table style="width:100%; border-collapse:collapse; font-size:0.9rem;">
<thead>
<tr style="border-bottom:2px solid #e5e7eb;">
<th style="text-align:left; padding:8px 12px;">Service</th>
<th style="text-align:left; padding:8px 12px;">Provider</th>
<th style="text-align:left; padding:8px 12px;">Category</th>
<th style="text-align:left; padding:8px 12px;">Pricing</th>
<th style="text-align:left; padding:8px 12px;">GDPR Maturity</th>
<th style="text-align:left; padding:8px 12px;">DPA</th>
</tr>
</thead>
<tbody>
${catalog.map(s => html`<tr style="border-bottom:1px solid #f3f4f6; ${WARNING_LEVELS.has(s.gdpr_maturity) ? 'background:#fff7f7;' : ''}">
<td style="padding:8px 12px; font-weight:500;">
${s.website_url
? html`<a href="${s.website_url}" target="_blank" style="color:#1d4ed8;">${s.name}</a>`
: s.name}
</td>
<td style="padding:8px 12px; color:#6b7280;">${s.provider || "—"}</td>
<td style="padding:8px 12px; color:#6b7280;">${s.category || "—"}</td>
<td style="padding:8px 12px;">${pricingBadge(s.pricing_model)}</td>
<td style="padding:8px 12px;">${maturityBadge(s.gdpr_maturity)}</td>
<td style="padding:8px 12px;">${s.dpa_available ? "✅" : "❌"}</td>
</tr>`)}
</tbody>
</table>`;
display(catalogTable);
```
---
## GDPR Warnings
_Services at **Unknown**, **Non-Compliant**, or **Initial** maturity may limit use in GDPR-regulated or corporate environments._
```js
if (gdprReport.warnings.length === 0) {
display(html`<p style="color:#16a34a;">✅ No GDPR warnings across active repos.</p>`);
} else {
const warningCards = html`<div style="display:flex; flex-direction:column; gap:0.75rem;">
${gdprReport.warnings.map(w => {
const color = maturityColor[w.gdpr_maturity] || "#ef4444";
return html`<div style="border-left:4px solid ${color}; background:${color}10; padding:0.75rem 1rem; border-radius:4px;">
<div style="font-weight:600;">${w.service_slug}
<span style="font-weight:400; color:#6b7280; margin-left:0.5rem;">in ${w.repo_slug || "unknown repo"}</span>
</div>
<div style="font-size:0.85rem; margin-top:2px;">
${maturityBadge(w.gdpr_maturity)}
${w.pricing_model ? html`&nbsp;${pricingBadge(w.pricing_model)}` : ""}
${w.purpose ? html`<span style="color:#6b7280; margin-left:0.5rem;">— ${w.purpose}</span>` : ""}
</div>
</div>`;
})}
</div>`;
display(warningCards);
}
```
---
## Per-Repo Breakdown
```js
// Build: latest snapshot per repo → service list
const repoBreakdown = new Map();
for (const snap of snapshots) {
const repoSlug = snap.repo_id || "unknown";
if (!repoBreakdown.has(repoSlug) || snap.snapshot_at > repoBreakdown.get(repoSlug).snapshot_at) {
repoBreakdown.set(repoSlug, snap);
}
}
// Enrich with catalog data
const catalogBySlug = Object.fromEntries(catalog.map(s => [s.slug, s]));
const repoTable = html`<table style="width:100%; border-collapse:collapse; font-size:0.9rem;">
<thead>
<tr style="border-bottom:2px solid #e5e7eb;">
<th style="text-align:left; padding:8px 12px;">Repo</th>
<th style="text-align:left; padding:8px 12px;">Services</th>
<th style="text-align:left; padding:8px 12px;">Ingested</th>
</tr>
</thead>
<tbody>
${[...repoBreakdown.entries()].map(([repoSlug, snap]) => html`<tr style="border-bottom:1px solid #f3f4f6;">
<td style="padding:8px 12px; font-weight:500;">${repoSlug}</td>
<td style="padding:8px 12px;">
${snap.entries.map(e => {
const cat = catalogBySlug[e.service_slug];
const m = cat?.gdpr_maturity || "unknown";
const color = maturityColor[m] || "#9ca3af";
return html`<span style="display:inline-flex; align-items:center; gap:4px; margin:2px 4px 2px 0;
background:${color}15; border:1px solid ${color}50; border-radius:4px; padding:2px 8px; font-size:0.8rem;">
<span style="width:8px; height:8px; border-radius:50%; background:${color}; display:inline-block;"></span>
${e.service_slug}
</span>`;
})}
</td>
<td style="padding:8px 12px; color:#9ca3af; font-size:0.8rem;">${new Date(snap.snapshot_at).toLocaleDateString()}</td>
</tr>`)}
</tbody>
</table>`;
display(repoTable);
```

View File

@@ -1032,19 +1032,54 @@ def update_repo_path(repo_slug: str, path: str, host: str | None = None) -> str:
return json.dumps(repo, indent=2)
# ---------------------------------------------------------------------------
# Shared path resolution helper
# ---------------------------------------------------------------------------
def _resolve_repo_path(repo: dict) -> str:
"""Return the best local filesystem path for *repo* on this host.
Resolution order — each candidate is expanded (supports ``~``) and
verified to exist before being accepted:
1. ``host_paths[hostname]`` — host-specific override
2. ``local_path`` — default fallback
Returns the resolved path string, or ``""`` if no valid path is found.
"""
import socket as _socket
hostname = _socket.gethostname()
host_paths = repo.get("host_paths") or {}
candidates = []
if host_paths.get(hostname):
candidates.append(host_paths[hostname])
if repo.get("local_path"):
candidates.append(repo["local_path"])
for raw in candidates:
resolved = str(Path(raw).expanduser())
if Path(resolved).is_dir():
return resolved
return ""
# ---------------------------------------------------------------------------
# Kaizen Agents
# ---------------------------------------------------------------------------
def _kaizen_agents_dir() -> Path:
"""Resolve the kaizen-agentic agents/ directory via host_paths → local_path fallback."""
import socket as _socket
"""Resolve the kaizen-agentic agents/ directory."""
repo = _get("/repos/kaizen-agentic")
hostname = _socket.gethostname()
host_paths = repo.get("host_paths") or {}
base = host_paths.get(hostname) or repo.get("local_path") or ""
base = _resolve_repo_path(repo)
if not base:
raise FileNotFoundError("kaizen-agentic path not found for this host. Register it with update_repo_path().")
import socket as _socket
hostname = _socket.gethostname()
raise FileNotFoundError(
f"kaizen-agentic path not found on host '{hostname}'. "
"Register it with update_repo_path('kaizen-agentic', '/path/to/repo')."
)
agents_dir = Path(base) / "agents"
if not agents_dir.is_dir():
raise FileNotFoundError(f"agents/ directory not found at {agents_dir}")
@@ -1128,8 +1163,8 @@ def validate_repo_adr(repo_slug: str, domain_slug: str | None = None) -> str:
no active state-hub workstreams for the domain lack a backing file (orphan
detection — DB-only records are an ADR-001 violation).
The repo path is resolved from the DB using the current machine's hostname
(host_paths[hostname] → local_path fallback). This tool always runs against
The repo path is resolved from the DB: host_paths[hostname] is tried first
(with existence check), then local_path — both support ~ expansion. This tool always runs against
the server's copy of the repo. Remote agents on a different branch should
sync first, or run validate_repo_adr.py locally with
--api-base http://127.0.0.1:18000.
@@ -1146,22 +1181,15 @@ def validate_repo_adr(repo_slug: str, domain_slug: str | None = None) -> str:
if isinstance(repo, dict) and repo.get("error"):
return f"Repo '{repo_slug}' not found: {repo['error']}"
hostname = _socket.gethostname()
host_paths = repo.get("host_paths") or {}
repo_path = host_paths.get(hostname) or repo.get("local_path") or ""
repo_path = _resolve_repo_path(repo)
if not repo_path:
hostname = _socket.gethostname()
return (
f"⚠ No path registered for repo '{repo_slug}' on this host ({hostname}).\n"
f"⚠ No accessible path found for repo '{repo_slug}' on host '{hostname}'.\n"
f"Register with: update_repo_path('{repo_slug}', '/path/to/repo')\n"
f"Remote agents: run validate_repo_adr.py locally with "
f"--api-base {API_BASE}"
)
if not Path(repo_path).is_dir():
return (
f"⚠ Registered path for '{repo_slug}' on {hostname} does not exist: {repo_path}\n"
f"Update with: update_repo_path('{repo_slug}', '/correct/path')"
)
script = Path(__file__).parent.parent / "scripts" / "validate_repo_adr.py"
cmd = [sys.executable, str(script), repo_path, "--json",
@@ -1238,21 +1266,15 @@ def check_repo_consistency(repo_slug: str, fix: bool = False) -> str:
repo = _get(f"/repos/{repo_slug}")
if isinstance(repo, dict) and repo.get("error"):
return f"Repo '{repo_slug}' not found: {repo['error']}"
hostname = _socket.gethostname()
host_paths = repo.get("host_paths") or {}
repo_path = host_paths.get(hostname) or repo.get("local_path") or ""
repo_path = _resolve_repo_path(repo)
if not repo_path:
hostname = _socket.gethostname()
return (
f"⚠ No path registered for repo '{repo_slug}' on this host ({hostname}).\n"
f"⚠ No accessible path found for repo '{repo_slug}' on host '{hostname}'.\n"
f"Register with: update_repo_path('{repo_slug}', '/path/to/repo')\n"
f"Remote agents: run consistency_check.py locally with "
f"--api-base {API_BASE}"
)
if not Path(repo_path).is_dir():
return (
f"⚠ Registered path for '{repo_slug}' on {hostname} does not exist: {repo_path}\n"
f"Update with: update_repo_path('{repo_slug}', '/correct/path')"
)
script = Path(__file__).parent.parent / "scripts" / "consistency_check.py"
cmd = [sys.executable, str(script), "--repo", repo_slug, "--json",
@@ -1448,20 +1470,13 @@ def ingest_sbom_tool(repo_slug: str, lockfile_path: str | None = None) -> str:
if isinstance(repo, dict) and repo.get("error"):
return f"Repo '{repo_slug}' not found: {repo['error']}"
hostname = _socket.gethostname()
host_paths = repo.get("host_paths") or {}
repo_root = host_paths.get(hostname) or repo.get("local_path") or ""
repo_root = _resolve_repo_path(repo)
if not repo_root:
hostname = _socket.gethostname()
return (
f"⚠ No path registered for repo '{repo_slug}' on this host ({hostname}).\n"
f"⚠ No accessible path found for repo '{repo_slug}' on host '{hostname}'.\n"
f"Register with: update_repo_path('{repo_slug}', '/path/to/repo')"
)
if not Path(repo_root).is_dir():
return (
f"⚠ Registered path for '{repo_slug}' on {hostname} does not exist: {repo_root}\n"
f"Update with: update_repo_path('{repo_slug}', '/correct/path')"
)
script = Path(__file__).parent.parent / "scripts" / "ingest_sbom.py"
cmd = [sys.executable, str(script), "--repo", repo_slug,
@@ -1866,6 +1881,46 @@ def update_capability_request_status(
return json.dumps(result, indent=2)
@mcp.tool()
def patch_capability_request(
request_id: str,
catalog_entry_id: Optional[str] = None,
priority: Optional[str] = None,
blocking_task_id: Optional[str] = None,
fulfilling_workstream_id: Optional[str] = None,
) -> dict:
"""Correct mutable metadata on a capability request.
Correcting catalog_entry_id automatically re-derives the fulfilling domain.
Use this when the hub mis-routed a request (wrong catalog entry or domain).
Only provided (non-None) fields are updated.
Args:
request_id: UUID of the capability request to patch.
catalog_entry_id: Correct catalog entry UUID. Re-derives fulfilling domain.
priority: New priority (low/medium/high/critical).
blocking_task_id: UUID of the task this request unblocks on completion.
fulfilling_workstream_id: UUID of the workstream delivering this capability.
Returns:
Updated capability request dict, or {"error": "..."}.
"""
body: dict = {}
if catalog_entry_id is not None:
body["catalog_entry_id"] = catalog_entry_id
if priority is not None:
body["priority"] = priority
if blocking_task_id is not None:
body["blocking_task_id"] = blocking_task_id
if fulfilling_workstream_id is not None:
body["fulfilling_workstream_id"] = fulfilling_workstream_id
if not body:
return {"error": "no fields provided to patch"}
return json.dumps(_patch(f"/capability-requests/{request_id}", body), indent=2)
@mcp.tool()
def list_capability_requests(
domain: str | None = None,
@@ -1896,6 +1951,174 @@ def get_capability_request(request_id: str) -> str:
return json.dumps(_get(f"/capability-requests/{request_id}"), indent=2)
# ---------------------------------------------------------------------------
# Third-Party Services Catalog (TPSC)
# ---------------------------------------------------------------------------
@mcp.tool()
def register_service(
slug: str,
name: str,
provider: str | None = None,
category: str | None = None,
pricing_model: str = "unknown",
gdpr_maturity: str = "unknown",
gdpr_notes: str | None = None,
dpa_available: bool = False,
tos_url: str | None = None,
privacy_policy_url: str | None = None,
data_processing_regions: list[str] | None = None,
data_retention_notes: str | None = None,
website_url: str | None = None,
) -> str:
"""Register or update a service in the Third-Party Services Catalog (TPSC).
GDPR maturity scale (CNIL/IAPP CMMI-aligned):
unknown | non_compliant | initial | developing | defined | managed | certified
Pricing model: free | paid | freemium | usage_based | unknown
Args:
slug: Unique identifier (e.g. 'openai-api', 'stripe')
name: Human-readable service name
provider: Company/organisation name
category: Category (e.g. 'llm_inference', 'storage', 'payments', 'search')
pricing_model: free | paid | freemium | usage_based | unknown
gdpr_maturity: GDPR compliance maturity level (see scale above)
gdpr_notes: Free-text GDPR notes (DPA details, transfer mechanisms, etc.)
dpa_available: Whether a Data Processing Agreement is available
tos_url: Terms of Service URL
privacy_policy_url: Privacy Policy URL
data_processing_regions: List of regions where data is processed (e.g. ['us', 'eu'])
data_retention_notes: Data retention policy summary
website_url: Service website URL
"""
return json.dumps(_post("/tpsc/catalog", {
"slug": slug,
"name": name,
"provider": provider,
"category": category,
"website_url": website_url,
"pricing_model": pricing_model,
"gdpr_maturity": gdpr_maturity,
"gdpr_notes": gdpr_notes,
"dpa_available": dpa_available,
"tos_url": tos_url,
"privacy_policy_url": privacy_policy_url,
"data_processing_regions": data_processing_regions or [],
"data_retention_notes": data_retention_notes,
}), indent=2)
@mcp.tool()
def list_services(
gdpr_maturity: str | None = None,
category: str | None = None,
pricing_model: str | None = None,
) -> str:
"""Browse the Third-Party Services Catalog (TPSC).
Returns services with their GDPR maturity level and gdpr_warning flag
(True when maturity is unknown, non_compliant, or initial — may limit
use in corporate/GDPR-regulated environments).
Args:
gdpr_maturity: Filter by maturity level (unknown/non_compliant/initial/developing/defined/managed/certified)
category: Filter by category (e.g. 'llm_inference', 'storage')
pricing_model: Filter by pricing model (free/paid/freemium/usage_based/unknown)
"""
return json.dumps(_get("/tpsc/catalog", {
"gdpr_maturity": gdpr_maturity,
"category": category,
"pricing_model": pricing_model,
}), indent=2)
@mcp.tool()
def ingest_tpsc_tool(repo_slug: str) -> str:
"""Ingest tpsc.yaml service dependency declarations for a repo.
Reads <repo_root>/tpsc.yaml, resolves service slugs against the catalog,
and creates a new TPSC snapshot. The repo path is resolved the same way
as the SBOM ingest tool (host_paths → local_path with existence check).
Args:
repo_slug: Registered repo slug (e.g. 'llm-connect', 'markitect-project')
"""
import socket as _socket
import subprocess
repo = _get(f"/repos/{repo_slug}")
if isinstance(repo, dict) and repo.get("error"):
return f"Repo '{repo_slug}' not found: {repo['error']}"
repo_root = _resolve_repo_path(repo)
if not repo_root:
hostname = _socket.gethostname()
return (
f"⚠ No accessible path found for repo '{repo_slug}' on host '{hostname}'.\n"
f"Register with: update_repo_path('{repo_slug}', '/path/to/repo')"
)
script = Path(__file__).parent.parent / "scripts" / "ingest_tpsc.py"
result = subprocess.run(
["uv", "run", "python", str(script), "--repo", repo_slug],
capture_output=True, text=True,
cwd=str(Path(__file__).parent.parent),
)
output = result.stdout + result.stderr
if result.returncode != 0:
return f"ingest_tpsc failed (exit {result.returncode}):\n{output}"
return output.strip()
@mcp.tool()
def get_gdpr_report() -> str:
"""Get an aggregated GDPR compliance report across all repos' latest TPSC snapshots.
Returns a warning summary for services with gdpr_maturity in:
unknown | non_compliant | initial
These may limit usability in GDPR-regulated / corporate environments.
Services at 'developing' or above have at least a DPA available.
"""
return json.dumps(_get("/tpsc/report/gdpr"), indent=2)
# ---------------------------------------------------------------------------
# Repository Definition of Integrated (DoI)
# ---------------------------------------------------------------------------
@mcp.tool()
def check_repo_doi(repo_slug: str) -> str:
"""Evaluate the 14 DoI criteria for a repo and return a full report.
Criteria are grouped into three tiers:
Core (C1C4): registered, domain, path, remote URL
Standard (C5C9): SCOPE.md, CLAUDE.md, workplan, SBOM, TPSC
Full (C10C14): repo goal, capabilities, agents, clean consistency, host paths
Status values: pass | fail | warn | skip
The 'tier' field shows the highest tier where ALL criteria pass or warn:
none | core | standard | full
Args:
repo_slug: Registered repo slug (e.g. 'llm-connect', 'the-custodian')
"""
return json.dumps(_get(f"/repos/{repo_slug}/doi"), indent=2)
@mcp.tool()
def get_doi_summary() -> str:
"""Return DoI tier for all active repos, sorted worst-first.
Useful at session start to spot repos that need integration work.
Tiers: none (red) → core → standard → full (green).
"""
return json.dumps(_get("/repos/doi/summary"), indent=2)
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------

View File

@@ -0,0 +1,70 @@
"""tpsc: third-party services catalog
Revision ID: j7e8f9a0b1c2
Revises: i6d7e8f9a0b1
Create Date: 2026-03-19
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import UUID, JSON
import uuid
revision = "j7e8f9a0b1c2"
down_revision = "i6d7e8f9a0b1"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"tpsc_catalog",
sa.Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4),
sa.Column("slug", sa.String(100), nullable=False, unique=True),
sa.Column("name", sa.String(200), nullable=False),
sa.Column("provider", sa.String(200), nullable=True),
sa.Column("category", sa.String(100), nullable=True),
sa.Column("website_url", sa.Text, nullable=True),
sa.Column("pricing_model", sa.String(20), nullable=False, server_default="unknown"),
sa.Column("gdpr_maturity", sa.String(20), nullable=False, server_default="unknown"),
sa.Column("gdpr_notes", sa.Text, nullable=True),
sa.Column("dpa_available", sa.Boolean, nullable=False, server_default="false"),
sa.Column("tos_url", sa.Text, nullable=True),
sa.Column("privacy_policy_url", sa.Text, nullable=True),
sa.Column("data_processing_regions", JSON, nullable=True),
sa.Column("data_retention_notes", sa.Text, nullable=True),
sa.Column("status", sa.String(20), nullable=False, server_default="active"),
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now()),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), onupdate=sa.func.now()),
)
op.create_index("ix_tpsc_catalog_slug", "tpsc_catalog", ["slug"])
op.create_index("ix_tpsc_catalog_gdpr_maturity", "tpsc_catalog", ["gdpr_maturity"])
op.create_table(
"tpsc_snapshots",
sa.Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4),
sa.Column("repo_id", UUID(as_uuid=True), sa.ForeignKey("managed_repos.id", ondelete="SET NULL"), nullable=True),
sa.Column("snapshot_at", sa.DateTime(timezone=True), server_default=sa.func.now()),
sa.Column("source_file", sa.String(200), nullable=True),
sa.Column("entry_count", sa.Integer, nullable=False, server_default="0"),
)
op.create_index("ix_tpsc_snapshots_repo_id", "tpsc_snapshots", ["repo_id"])
op.create_table(
"tpsc_entries",
sa.Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4),
sa.Column("snapshot_id", UUID(as_uuid=True), sa.ForeignKey("tpsc_snapshots.id", ondelete="CASCADE"), nullable=False),
sa.Column("catalog_id", UUID(as_uuid=True), sa.ForeignKey("tpsc_catalog.id", ondelete="SET NULL"), nullable=True),
sa.Column("service_slug", sa.String(100), nullable=False),
sa.Column("purpose", sa.Text, nullable=True),
sa.Column("auth_type", sa.String(50), nullable=True),
sa.Column("endpoint_override", sa.Text, nullable=True),
sa.Column("notes", sa.Text, nullable=True),
)
op.create_index("ix_tpsc_entries_snapshot_id", "tpsc_entries", ["snapshot_id"])
op.create_index("ix_tpsc_entries_service_slug", "tpsc_entries", ["service_slug"])
def downgrade() -> None:
op.drop_table("tpsc_entries")
op.drop_table("tpsc_snapshots")
op.drop_table("tpsc_catalog")

View File

@@ -0,0 +1,38 @@
"""doi_cache: materialised DoI results with fingerprint-based invalidation
Revision ID: k8f9a0b1c2d3
Revises: j7e8f9a0b1c2
Create Date: 2026-03-20
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import UUID, JSON
import uuid
revision = "k8f9a0b1c2d3"
down_revision = "j7e8f9a0b1c2"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"doi_cache",
sa.Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4),
sa.Column("repo_id", UUID(as_uuid=True),
sa.ForeignKey("managed_repos.id", ondelete="CASCADE"),
nullable=False, unique=True),
sa.Column("tier", sa.String(20), nullable=False),
sa.Column("core_pass", sa.Boolean, nullable=False, server_default="false"),
sa.Column("standard_pass", sa.Boolean, nullable=False, server_default="false"),
sa.Column("full_pass", sa.Boolean, nullable=False, server_default="false"),
sa.Column("criteria", JSON, nullable=True), # full criterion list
sa.Column("fingerprint", sa.Text, nullable=False), # pipe-joined timestamps
sa.Column("checked_at", sa.DateTime(timezone=True), server_default=sa.func.now()),
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now()),
)
op.create_index("ix_doi_cache_repo_id", "doi_cache", ["repo_id"])
def downgrade() -> None:
op.drop_table("doi_cache")

View File

@@ -0,0 +1,19 @@
"""Add routing_note to capability_requests
Revision ID: l9g0h1i2j3k4
Revises: k8f9a0b1c2d3
Create Date: 2026-03-20
"""
from alembic import op
import sqlalchemy as sa
revision = 'l9g0h1i2j3k4'
down_revision = 'k8f9a0b1c2d3'
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column('capability_requests', sa.Column('routing_note', sa.Text(), nullable=True))
def downgrade() -> None:
op.drop_column('capability_requests', 'routing_note')

View File

@@ -0,0 +1,120 @@
# Repository Definition of Integrated (DoI)
A repository is considered **fully integrated** with the Custodian State Hub
when all criteria below are satisfied. Criteria are grouped by tier: a repo
that meets all **Core** criteria is *registered*; meeting **Standard** criteria
makes it *integrated*; meeting **Full** criteria makes it *fully integrated*.
---
## Tier 1 — Core (Registered)
These are the minimum requirements for the state-hub to be aware of a repo.
- [ ] **Registered in state-hub** — repo exists in `managed_repos` with correct
`slug`, `domain_slug`, `local_path`, and `remote_url`. Verify:
`GET /repos/{slug}`
- [ ] **Domain assigned** — repo is linked to an active domain. The domain must
exist in `domains` and be in `active` status.
- [ ] **Local path resolves**`_resolve_repo_path()` finds an existing
directory on the current host (either via `host_paths[hostname]` or
`local_path`, both support `~` expansion).
- [ ] **Remote URL reachable**`remote_url` points to a live Gitea/GitHub
repository. The repo must be pushable from the registered local path.
---
## Tier 2 — Standard (Integrated)
A repo at this tier participates in state-hub tracking and tooling.
- [ ] **SCOPE.md present** — a `SCOPE.md` exists at the repo root following the
standard template (`state-hub/scripts/project_rules/scope.template`).
Sections: One-liner, Core Idea, In Scope, Out of Scope, Relevant When,
Not Relevant When, Current State, How It Fits, Provided Capabilities.
- [ ] **CLAUDE.md present** — a `CLAUDE.md` exists at the repo root with at
minimum: domain context, session protocol (start: `get_domain_summary`,
end: `add_progress_event`), and stack/commands reference.
- [ ] **Workplan convention followed** — any workplans live under
`workplans/<ID>-<slug>.md` with valid YAML frontmatter
(`id`, `type`, `title`, `domain`, `status`, `state_hub_workstream_id`).
Verified by: `make check-consistency REPO={slug}` → PASS or WARN only
(no FAIL).
- [ ] **SBOM ingested** — at least one dependency lockfile exists
(`uv.lock`, `requirements.txt`, `package-lock.json`, `yarn.lock`,
`Cargo.lock`, `go.sum`) and has been ingested:
`make ingest-sbom REPO={slug}` succeeded with ≥1 entry.
`last_sbom_at` is set on the repo record.
- [ ] **TPSC declared** — a `tpsc.yaml` exists at the repo root declaring all
external service dependencies, and has been ingested:
`make ingest-tpsc REPO={slug}` succeeded. If the repo has no external
service dependencies, `tpsc.yaml` must exist with an empty `services: []`
to make the absence explicit.
---
## Tier 3 — Full (Fully Integrated)
A fully integrated repo contributes to cross-repo planning and capability routing.
- [ ] **Active repo goal** — at least one `RepoGoal` with `status: active`
exists for the repo in the state-hub. Goals connect workstreams to
strategic intent. Create via: `create_repo_goal(repo_slug, title, ...)`
- [ ] **Provided Capabilities declared** — if the repo exposes capabilities to
other domains, they are declared in the `## Provided Capabilities` section
of `SCOPE.md` using `capability` fenced blocks and have been ingested via
`make ingest-capabilities REPO={slug}`.
- [ ] **Agents template applied**`CLAUDE.md` references the kaizen agent
system (`get_kaizen_agent()`) and includes the `agents.template` section
listing available specialised personas for the domain.
- [ ] **Consistency check clean**`make check-consistency REPO={slug}`
reports **0 FAIL, 0 WARN** (C-12 warnings on legacy DB-only tasks are
exempt if the workstream predates ADR-001).
- [ ] **Host path registered for all active hosts**`host_paths` contains
an entry for every machine where the repo is actively worked on. Use
`update_repo_path(slug, path)` to register additional hosts.
---
## Integration Checklist (Quick Reference)
| # | Criterion | Tier | Verified by |
|---|---|---|---|
| 1 | Registered in state-hub | Core | `GET /repos/{slug}` |
| 2 | Domain assigned | Core | `GET /repos/{slug}``domain_slug` |
| 3 | Local path resolves | Core | `check_repo_consistency(slug)` |
| 4 | Remote URL reachable | Core | `git push` dry-run |
| 5 | SCOPE.md present | Standard | `ls SCOPE.md` |
| 6 | CLAUDE.md present | Standard | `ls CLAUDE.md` |
| 7 | Workplan convention followed | Standard | `make check-consistency` |
| 8 | SBOM ingested | Standard | `last_sbom_at` on repo record |
| 9 | TPSC declared | Standard | `make ingest-tpsc` |
| 10 | Active repo goal | Full | `get_repo_goals(slug)` |
| 11 | Provided Capabilities declared | Full | `make ingest-capabilities` |
| 12 | Agents template applied | Full | `CLAUDE.md` review |
| 13 | Consistency check clean | Full | `make check-consistency` → 0 FAIL/WARN |
| 14 | Host paths registered | Full | `GET /repos/{slug}``host_paths` |
---
## Notes
- The DoI is enforced by convention, not by automated gates (as of v0.6).
A future `make check-doi REPO={slug}` target is planned.
- Repos that are `archived` or `deprecated` are exempt from DoI compliance.
- The **Core** tier is a prerequisite for all state-hub tooling
(`check_repo_consistency`, `ingest_sbom_tool`, `ingest_tpsc_tool`).
- A repo may satisfy **Standard** without **Full** indefinitely — Full
integration is appropriate when the repo is actively contributing to
cross-domain work or exposing capabilities to other repos.

View File

@@ -1,3 +1,5 @@
# Workstream Definition of Done
A workstream is considered finished if and only if:
- All tasks in the workstream have been finished, found unnecessary or been transfered to another workstream
- All referenced requirements have been adressed with nothing relevent missing or documented as out of scope with a stated reason

View File

@@ -16,6 +16,7 @@ dependencies = [
"python-dotenv>=1.0.0",
"psycopg2-binary>=2.9.0",
"llm-connect",
"pyyaml>=6.0.3",
]
[project.scripts]

View File

@@ -0,0 +1,132 @@
#!/usr/bin/env python3
"""Check Repository Definition of Integrated (DoI) criteria.
Usage:
uv run python scripts/check_doi.py --repo <slug> [--json]
uv run python scripts/check_doi.py --all [--json]
"""
import argparse
import asyncio
import json
import sys
import urllib.request
from pathlib import Path
# Allow importing from the api package
sys.path.insert(0, str(Path(__file__).parent.parent))
from api.doi_engine import evaluate
API_BASE = "http://127.0.0.1:8000"
STATUS_ICON = {"pass": "", "fail": "", "warn": "", "skip": ""}
STATUS_COLOR = {
"pass": "\033[32m", # green
"fail": "\033[31m", # red
"warn": "\033[33m", # yellow
"skip": "\033[90m", # grey
}
RESET = "\033[0m"
BOLD = "\033[1m"
TIER_COLOR = {
"full": "\033[32m",
"standard": "\033[33m",
"core": "\033[33m",
"none": "\033[31m",
}
def _get(path: str) -> object:
req = urllib.request.Request(
f"{API_BASE}{path}/",
headers={"Accept": "application/json"},
)
with urllib.request.urlopen(req, timeout=10) as r:
return json.loads(r.read())
def _print_report(report, use_color: bool = True) -> None:
tier_c = TIER_COLOR.get(report.tier, "") if use_color else ""
reset = RESET if use_color else ""
bold = BOLD if use_color else ""
print(f"\n{bold}Repo: {report.repo_slug}{reset}")
print(f" Tier: {tier_c}{bold}{report.tier.upper()}{reset} "
f"(core={'' if report.core_pass else ''} "
f"standard={'' if report.standard_pass else ''} "
f"full={'' if report.full_pass else ''})")
current_tier = None
for c in report.criteria:
if c.tier != current_tier:
current_tier = c.tier
print(f" ── {c.tier.upper()} ──")
sc = STATUS_COLOR.get(c.status, "") if use_color else ""
ico = STATUS_ICON.get(c.status, "?")
detail = f" {c.detail}" if c.detail else ""
print(f" {sc}{ico}{reset} {c.id}: {c.label}{detail}")
async def check_repo(slug: str, as_json: bool) -> bool:
try:
repo = _get(f"/repos/{slug}/")
except Exception as e:
print(f"✗ Could not fetch repo '{slug}': {e}", file=sys.stderr)
return False
report = await evaluate(repo, API_BASE)
if as_json:
print(json.dumps({
"repo_slug": report.repo_slug,
"tier": report.tier,
"core_pass": report.core_pass,
"standard_pass": report.standard_pass,
"full_pass": report.full_pass,
"checked_at": report.checked_at,
"criteria": [
{"id": c.id, "label": c.label, "tier": c.tier,
"status": c.status, "detail": c.detail}
for c in report.criteria
],
}, indent=2))
else:
_print_report(report)
return report.tier != "none"
async def main_async() -> None:
parser = argparse.ArgumentParser(description="Check Repository DoI criteria")
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("--repo", metavar="SLUG")
group.add_argument("--all", action="store_true")
parser.add_argument("--json", action="store_true", dest="as_json")
args = parser.parse_args()
if args.all:
repos = _get("/repos/")
slugs = [r["slug"] for r in repos if r.get("status") == "active"]
else:
slugs = [args.repo]
results = await asyncio.gather(*[check_repo(s, args.as_json) for s in slugs])
if not args.as_json:
counts = {"full": 0, "standard": 0, "core": 0, "none": 0}
for slug, ok in zip(slugs, results):
pass # already printed per-repo
if len(slugs) > 1:
# Re-fetch for summary (already printed above)
pass
sys.exit(0 if all(results) else 1)
def main() -> None:
asyncio.run(main_async())
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,145 @@
#!/usr/bin/env python3
"""Ingest tpsc.yaml service dependency declarations into the State Hub.
Usage:
uv run python scripts/ingest_tpsc.py --repo <slug> [--dry-run]
uv run python scripts/ingest_tpsc.py --all [--dry-run]
"""
import argparse
import json
import sys
import urllib.error
import urllib.request
from pathlib import Path
try:
import yaml
except ImportError:
import tomllib as _t # noqa — fallback not really viable; yaml is required
yaml = None
API_BASE = "http://127.0.0.1:8000"
TPSC_FILENAME = "tpsc.yaml"
def _get(path: str) -> dict | list:
req = urllib.request.Request(f"{API_BASE}{path}", headers={"Accept": "application/json"})
with urllib.request.urlopen(req) as r:
return json.loads(r.read())
def _post(path: str, payload: dict) -> dict:
data = json.dumps(payload).encode()
req = urllib.request.Request(
f"{API_BASE}{path}/",
data=data,
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(req) as r:
return json.loads(r.read())
except urllib.error.HTTPError as e:
body = e.read().decode()
print(f" ERROR {e.code}: {body}", file=sys.stderr)
raise
def _load_yaml(path: Path) -> dict:
if yaml is None:
raise RuntimeError("PyYAML is required: uv add pyyaml")
with open(path) as f:
return yaml.safe_load(f) or {}
def _resolve_repo_path(repo: dict) -> str:
import socket
hostname = socket.gethostname()
host_paths = repo.get("host_paths") or {}
candidates = []
if host_paths.get(hostname):
candidates.append(host_paths[hostname])
if repo.get("local_path"):
candidates.append(repo["local_path"])
for raw in candidates:
p = Path(raw).expanduser()
if p.is_dir():
return str(p)
return ""
def ingest_repo(slug: str, dry_run: bool = False) -> bool:
try:
repo = _get(f"/repos/{slug}")
except Exception as e:
print(f" ✗ Repo '{slug}' not found: {e}", file=sys.stderr)
return False
if isinstance(repo, dict) and repo.get("error"):
print(f"{repo['error']}", file=sys.stderr)
return False
repo_path = _resolve_repo_path(repo)
if not repo_path:
print(f" ✗ No accessible local path for '{slug}' on this host.", file=sys.stderr)
return False
tpsc_file = Path(repo_path) / TPSC_FILENAME
if not tpsc_file.exists():
print(f"'{slug}': no {TPSC_FILENAME} found, skipping.")
return True
data = _load_yaml(tpsc_file)
services = data.get("services", [])
if not services:
print(f"'{slug}': {TPSC_FILENAME} has no services entries, skipping.")
return True
entries = [
{
"service_slug": svc.get("slug", ""),
"purpose": svc.get("purpose"),
"auth_type": svc.get("auth"),
"endpoint_override": svc.get("endpoint"),
"notes": svc.get("notes"),
}
for svc in services
if svc.get("slug")
]
print(f" {'[dry-run] ' if dry_run else ''}'{slug}': {len(entries)} service(s) from {TPSC_FILENAME}")
for e in entries:
print(f"{e['service_slug']} ({e.get('auth_type', '?')}) — {e.get('purpose', '')}")
if dry_run:
return True
result = _post("/tpsc/ingest", {
"repo_slug": slug,
"source_file": TPSC_FILENAME,
"entries": entries,
})
print(f" ✓ Snapshot {result['id'][:8]}… ingested {result['entry_count']} entries")
return True
def main() -> None:
parser = argparse.ArgumentParser(description="Ingest tpsc.yaml into State Hub")
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("--repo", metavar="SLUG", help="Single repo slug")
group.add_argument("--all", action="store_true", help="All registered repos")
parser.add_argument("--dry-run", action="store_true", help="Parse only, do not POST")
args = parser.parse_args()
if args.all:
repos = _get("/repos/")
slugs = [r["slug"] for r in repos]
else:
slugs = [args.repo]
ok = all(ingest_repo(slug, dry_run=args.dry_run) for slug in slugs)
sys.exit(0 if ok else 1)
if __name__ == "__main__":
main()

View File

@@ -335,3 +335,108 @@ class TestCapabilityRequestLifecycle:
"status": "requested",
})
assert r.status_code == 422
class TestCapabilityRequestRouting:
async def test_word_boundary_avoids_substring_false_positive(self, client):
"""'postgres' keyword must not match inside 'postgresql'."""
await _setup_two_domains(client)
# Register two entries: PostgreSQL HA (keywords: postgres, ha) and k3s (keywords: k3s, cluster)
await _register_catalog(client, domain="railiance", cap_type="infrastructure",
title="PostgreSQL HA", keywords=["postgresql", "postgres", "ha"])
await _register_catalog(client, domain="custodian", cap_type="infrastructure",
title="K3s provisioning", keywords=["k3s", "cluster", "k8s"])
# Description mentions k8s and cluster but NOT standalone "postgres" or "ha"
req = await _create_request(
client,
title="K3s cluster access",
description="Need k8s foundations. cluster must be up. kubeconfig required.",
cap_type="infrastructure",
)
# k3s entry should win (k8s + cluster match), not postgres entry
assert req["catalog_entry_id"] is not None
# Routing note should mention k3s
assert req["routing_note"] is not None
assert "K3s" in req["routing_note"] or "k3s" in req["routing_note"].lower()
async def test_title_included_in_routing(self, client):
"""Title keywords should contribute to routing score."""
await _setup_two_domains(client)
await _register_catalog(client, domain="railiance", cap_type="infrastructure",
title="K3s cluster", keywords=["k3s", "cluster", "kubernetes"])
await _register_catalog(client, domain="custodian", cap_type="infrastructure",
title="Postgres DB", keywords=["postgresql", "postgres", "database"])
# Title contains "k3s" but description is generic
req = await _create_request(
client,
title="k3s cluster access needed",
description="Need access to proceed with deployment.",
cap_type="infrastructure",
)
assert req["routing_note"] is not None
assert req["catalog_entry_id"] is not None
# Should match k3s (title has "k3s" and "cluster")
# Verify via routing note
assert "K3s" in req["routing_note"] or "k3s" in req["routing_note"].lower()
async def test_ambiguous_routing_broadcasts(self, client):
"""Tied scores should broadcast (no fulfilling domain)."""
await _setup_two_domains(client)
await _register_catalog(client, domain="railiance", cap_type="infrastructure",
title="Entry A", keywords=["cluster"])
await _register_catalog(client, domain="custodian", cap_type="infrastructure",
title="Entry B", keywords=["cluster"])
req = await _create_request(
client,
title="cluster needed",
description="cluster access",
cap_type="infrastructure",
)
assert req["fulfilling_domain_slug"] is None
assert "ambiguous" in req["routing_note"]
async def test_patch_corrects_catalog_entry_and_reroutes(self, client):
"""PATCH /capability-requests/{id} corrects catalog_entry_id and re-derives domain."""
req_d, ful_d = await _setup_two_domains(client)
# Register wrong entry (postgres) and correct entry (k3s)
wrong = await _register_catalog(client, domain="railiance", cap_type="infrastructure",
title="PostgreSQL HA", keywords=["postgresql"])
correct = await _register_catalog(client, domain="custodian", cap_type="infrastructure",
title="K3s cluster", keywords=["k3s"])
# Create request — auto-routes to postgres (only postgres keyword matches "postgresql" in description)
req = await _create_request(
client,
description="Need postgresql and k3s access",
cap_type="infrastructure",
)
# Patch to correct catalog entry
r = await client.patch(f"/capability-requests/{req['id']}", json={
"catalog_entry_id": correct["id"],
})
assert r.status_code == 200
data = r.json()
assert data["catalog_entry_id"] == correct["id"]
assert data["fulfilling_domain_slug"] == "custodian"
assert "hub correction" in data["routing_note"]
assert "K3s" in data["routing_note"]
async def test_patch_unknown_catalog_entry_404(self, client):
await _setup_two_domains(client)
req = await _create_request(client, cap_type="security")
import uuid as _uuid
r = await client.patch(f"/capability-requests/{req['id']}", json={
"catalog_entry_id": str(_uuid.uuid4()),
})
assert r.status_code == 404
async def test_patch_priority(self, client):
await _setup_two_domains(client)
req = await _create_request(client, cap_type="security")
r = await client.patch(f"/capability-requests/{req['id']}", json={"priority": "critical"})
assert r.status_code == 200
assert r.json()["priority"] == "critical"
assert "priority" in r.json()["routing_note"]

11
state-hub/uv.lock generated
View File

@@ -145,10 +145,16 @@ sdist = { url = "https://files.pythonhosted.org/packages/92/88/b8527e1b00c1811db
wheels = [
{ url = "https://files.pythonhosted.org/packages/d3/25/79c98ebe12df31548ba4eaf44db11b7cad6b3e7b4203718335620939083c/caio-0.9.25-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:fb7ff95af4c31ad3f03179149aab61097a71fd85e05f89b4786de0359dffd044", size = 36983 },
{ url = "https://files.pythonhosted.org/packages/a3/2b/21288691f16d479945968a0a4f2856818c1c5be56881d51d4dac9b255d26/caio-0.9.25-cp312-cp312-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:97084e4e30dfa598449d874c4d8e0c8d5ea17d2f752ef5e48e150ff9d240cd64", size = 82012 },
{ url = "https://files.pythonhosted.org/packages/03/c4/8a1b580875303500a9c12b9e0af58cb82e47f5bcf888c2457742a138273c/caio-0.9.25-cp312-cp312-manylinux_2_34_aarch64.whl", hash = "sha256:4fa69eba47e0f041b9d4f336e2ad40740681c43e686b18b191b6c5f4c5544bfb", size = 81502 },
{ url = "https://files.pythonhosted.org/packages/d1/1c/0fe770b8ffc8362c48134d1592d653a81a3d8748d764bec33864db36319d/caio-0.9.25-cp312-cp312-manylinux_2_34_x86_64.whl", hash = "sha256:6bebf6f079f1341d19f7386db9b8b1f07e8cc15ae13bfdaff573371ba0575d69", size = 80200 },
{ url = "https://files.pythonhosted.org/packages/31/57/5e6ff127e6f62c9f15d989560435c642144aa4210882f9494204bc892305/caio-0.9.25-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:d6c2a3411af97762a2b03840c3cec2f7f728921ff8adda53d7ea2315a8563451", size = 36979 },
{ url = "https://files.pythonhosted.org/packages/a3/9f/f21af50e72117eb528c422d4276cbac11fb941b1b812b182e0a9c70d19c5/caio-0.9.25-cp313-cp313-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0998210a4d5cd5cb565b32ccfe4e53d67303f868a76f212e002a8554692870e6", size = 81900 },
{ url = "https://files.pythonhosted.org/packages/9c/12/c39ae2a4037cb10ad5eb3578eb4d5f8c1a2575c62bba675f3406b7ef0824/caio-0.9.25-cp313-cp313-manylinux_2_34_aarch64.whl", hash = "sha256:1a177d4777141b96f175fe2c37a3d96dec7911ed9ad5f02bac38aaa1c936611f", size = 81523 },
{ url = "https://files.pythonhosted.org/packages/22/59/f8f2e950eb4f1a5a3883e198dca514b9d475415cb6cd7b78b9213a0dd45a/caio-0.9.25-cp313-cp313-manylinux_2_34_x86_64.whl", hash = "sha256:9ed3cfb28c0e99fec5e208c934e5c157d0866aa9c32aa4dc5e9b6034af6286b7", size = 80243 },
{ url = "https://files.pythonhosted.org/packages/69/ca/a08fdc7efdcc24e6a6131a93c85be1f204d41c58f474c42b0670af8c016b/caio-0.9.25-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:fab6078b9348e883c80a5e14b382e6ad6aabbc4429ca034e76e730cf464269db", size = 36978 },
{ url = "https://files.pythonhosted.org/packages/5e/6c/d4d24f65e690213c097174d26eda6831f45f4734d9d036d81790a27e7b78/caio-0.9.25-cp314-cp314-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:44a6b58e52d488c75cfaa5ecaa404b2b41cc965e6c417e03251e868ecd5b6d77", size = 81832 },
{ url = "https://files.pythonhosted.org/packages/87/a4/e534cf7d2d0e8d880e25dd61e8d921ffcfe15bd696734589826f5a2df727/caio-0.9.25-cp314-cp314-manylinux_2_34_aarch64.whl", hash = "sha256:628a630eb7fb22381dd8e3c8ab7f59e854b9c806639811fc3f4310c6bd711d79", size = 81565 },
{ url = "https://files.pythonhosted.org/packages/3f/ed/bf81aeac1d290017e5e5ac3e880fd56ee15e50a6d0353986799d1bc5cfd5/caio-0.9.25-cp314-cp314-manylinux_2_34_x86_64.whl", hash = "sha256:0ba16aa605ccb174665357fc729cf500679c2d94d5f1458a6f0d5ca48f2060a7", size = 80071 },
{ url = "https://files.pythonhosted.org/packages/86/93/1f76c8d1bafe3b0614e06b2195784a3765bbf7b0a067661af9e2dd47fc33/caio-0.9.25-py3-none-any.whl", hash = "sha256:06c0bb02d6b929119b1cfbe1ca403c768b2013a369e2db46bfa2a5761cf82e40", size = 19087 },
]
@@ -677,6 +683,9 @@ requires-dist = [
{ name = "toml" },
]
[package.metadata.requires-dev]
dev = [{ name = "pytest", specifier = ">=9.0.2" }]
[[package]]
name = "mako"
version = "1.3.10"
@@ -1424,6 +1433,7 @@ dependencies = [
{ name = "pydantic" },
{ name = "pydantic-settings" },
{ name = "python-dotenv" },
{ name = "pyyaml" },
{ name = "sqlalchemy", extra = ["asyncio"] },
{ name = "uvicorn", extra = ["standard"] },
]
@@ -1447,6 +1457,7 @@ requires-dist = [
{ name = "pydantic", specifier = ">=2.10.0" },
{ name = "pydantic-settings", specifier = ">=2.7.0" },
{ name = "python-dotenv", specifier = ">=1.0.0" },
{ name = "pyyaml", specifier = ">=6.0.3" },
{ name = "sqlalchemy", extras = ["asyncio"], specifier = ">=2.0.0" },
{ name = "uvicorn", extras = ["standard"], specifier = ">=0.32.0" },
]

View File

@@ -0,0 +1,210 @@
---
id: CUST-WP-0023
type: feature
title: Third-Party Services Catalog (TPSC)
domain: custodian
status: done
owner: custodian-agent
topic_slug: custodian
created: 2026-03-19
updated: 2026-03-20
state_hub_workstream_id: "23208a99-0ef6-4154-9454-a2f2065b6b19"
---
# TPSC — Third-Party Services Catalog
Track external service dependencies (APIs, SaaS, CLIs) across all repos.
Primary data lives in repos (`tpsc.yaml`) and a canon catalog
(`canon/tpsc/<slug>.yaml`). State-hub collects and reports.
GDPR maturity scale (CNIL/IAPP CMMI-aligned):
`unknown | non_compliant | initial | developing | defined | managed | certified`
Pricing model: `free | paid | freemium | usage_based | unknown`
## Task: DB migration — tpsc_catalog, tpsc_snapshots, tpsc_entries
```task
id: CUST-WP-0023-T01
status: done
priority: high
state_hub_task_id: "038a0284-bb76-4ce7-8861-1686667acbb5"
```
Create Alembic migration `j7e8f9a0b1c2_tpsc.py`.
Tables:
- `tpsc_catalog`: id, slug (unique), name, provider, category, website_url,
pricing_model, gdpr_maturity, gdpr_notes, dpa_available, tos_url,
privacy_policy_url, data_processing_regions (JSON), data_retention_notes,
status (active/deprecated), created_at, updated_at
- `tpsc_snapshots`: id, repo_id (FK managed_repos nullable), snapshot_at,
source_file, entry_count
- `tpsc_entries`: id, snapshot_id (FK), catalog_id (FK tpsc_catalog nullable),
service_slug, purpose, auth_type, endpoint_override, notes
---
## Task: SQLAlchemy models
```task
id: CUST-WP-0023-T02
status: done
priority: high
state_hub_task_id: "990d4e58-35b0-45f0-8de6-8955049aa7d5"
```
Create `api/models/tpsc.py` with TPSCCatalog, TPSCSnapshot, TPSCEntry models.
Register in `api/models/__init__.py`.
---
## Task: Pydantic schemas
```task
id: CUST-WP-0023-T03
status: done
priority: high
state_hub_task_id: "5feeb161-b654-4e4a-8a6a-bb685c239ce5"
```
Create `api/schemas/tpsc.py` with Read/Create schemas for all three models.
Include `GDPRMaturity` and `PricingModel` string enums.
`TPSCCatalog` schema: include `gdpr_warning: bool` computed field
(True when gdpr_maturity in [unknown, non_compliant, initial]).
---
## Task: FastAPI router /tpsc/
```task
id: CUST-WP-0023-T04
status: done
priority: high
state_hub_task_id: "593471b4-cd3a-4251-8c5c-ee42b6a9e089"
```
Create `api/routers/tpsc.py`:
- `GET /tpsc/catalog/` — list services (filter: gdpr_maturity, category, pricing_model)
- `GET /tpsc/catalog/{slug}` — single service
- `POST /tpsc/catalog/` — register/upsert service
- `POST /tpsc/ingest/` — accept snapshot + entries for a repo
- `GET /tpsc/snapshots/` — list snapshots (filter: repo_slug)
- `GET /tpsc/report/gdpr` — aggregated GDPR warnings across all repos
Register in `api/main.py`.
---
## Task: MCP tools
```task
id: CUST-WP-0023-T05
status: done
priority: high
state_hub_task_id: "7370d020-06cf-4ebe-9f78-619e41c4b85c"
```
Add to `mcp_server/server.py`:
- `register_service(slug, name, provider, pricing_model, gdpr_maturity, ...)`
- `list_services(gdpr_maturity?, category?, pricing_model?)`
- `ingest_tpsc_tool(repo_slug)` — runs ingest_tpsc.py for the repo
- `get_gdpr_report()` — returns warning summary across all repos
---
## Task: Ingest script
```task
id: CUST-WP-0023-T06
status: done
priority: high
state_hub_task_id: "5ec305d3-fdfb-4f81-b3ab-1ea57a4ec0c5"
```
Create `scripts/ingest_tpsc.py`:
- Reads `tpsc.yaml` from repo root (auto-detected via registered local_path)
- Resolves catalog_id by slug for each entry
- POSTs snapshot + entries to `/tpsc/ingest/`
- `--dry-run` flag
- `--repo SLUG` or `--all` flags
Add Makefile targets:
```
make ingest-tpsc REPO=<slug>
make ingest-tpsc-all
```
---
## Task: Canon service catalog seed files
```task
id: CUST-WP-0023-T07
status: done
priority: medium
state_hub_task_id: "21c13b4d-585a-4a60-a283-29baa2dd6d7d"
```
Create `canon/tpsc/` with YAML files for:
- `openai-api.yaml`
- `anthropic-api.yaml`
- `gemini-api.yaml`
- `openrouter-api.yaml`
Each file: slug, name, provider, category, pricing_model, gdpr_maturity,
gdpr_notes, dpa_available, tos_url, privacy_policy_url,
data_processing_regions, data_retention_notes.
---
## Task: tpsc.yaml for llm-connect
```task
id: CUST-WP-0023-T08
status: done
priority: medium
state_hub_task_id: "d658f81b-7408-456d-b4bd-440b44a67e43"
```
Create `/home/worsch/llm-connect/tpsc.yaml` declaring:
openai-api, anthropic-api (via claude_code CLI), gemini-api, openrouter-api.
---
## Task: Dashboard page
```task
id: CUST-WP-0023-T09
status: done
priority: medium
state_hub_task_id: "a5367fc4-ef12-4f52-b642-d33b91b7cc2c"
```
Create `dashboard/src/tpsc.md`:
- Service catalog table with GDPR maturity color coding
(red: non_compliant/unknown, amber: initial/developing, green: defined+)
- Warning cards for repos using non-green services
- Per-repo breakdown table
- Pricing model summary
Add to `observablehq.config.js` nav.
---
## Task: Documentation page
```task
id: CUST-WP-0023-T10
status: done
priority: low
state_hub_task_id: "6cf1aaec-5c24-4cb5-a1aa-415caeaaca10"
```
Create `dashboard/src/docs/tpsc.md` explaining:
- TPSC concept and rationale
- tpsc.yaml format with full example
- GDPR maturity scale definitions (linked to CNIL/IAPP)
- How to add a new service to the canon catalog
Add to docs nav in `observablehq.config.js`.

View File

@@ -0,0 +1,154 @@
---
id: CUST-WP-0024
type: feature
title: Repository DoI automated gate and dashboard integration
domain: custodian
status: done
owner: custodian-agent
topic_slug: custodian
created: 2026-03-20
updated: 2026-03-20
state_hub_workstream_id: "f587f244-0dd5-4268-b955-0c4e4cf9aa69"
---
# Repository DoI Automated Gate & Dashboard
Automates the 14-criterion DoI checklist from `policies/repo-doi.md` and
surfaces per-repo DoI status on the Repositories dashboard page.
## Task: DoI check script — scripts/check_doi.py
```task
id: CUST-WP-0024-T01
status: done
priority: high
state_hub_task_id: "58e77114-bc54-4f86-9ddc-3834bc702b5c"
```
Create `state-hub/scripts/check_doi.py` implementing all 14 DoI criteria:
**Tier 1 — Core**
- C1: repo registered (`GET /repos/{slug}` returns 200)
- C2: domain assigned (`domain_slug` non-null and domain status active)
- C3: local path resolves (exists on disk after expanduser)
- C4: remote_url set (non-null, non-empty)
**Tier 2 — Standard**
- C5: SCOPE.md present at repo root
- C6: CLAUDE.md present at repo root
- C7: workplan convention followed — consistency check returns 0 FAIL
- C8: SBOM ingested — `last_sbom_at` non-null on repo record
- C9: TPSC declared — `tpsc.yaml` exists at repo root AND at least one
TPSC snapshot exists for the repo
**Tier 3 — Full**
- C10: active repo goal exists (`GET /repo-goals/?repo_slug=...` has ≥1 active)
- C11: Provided Capabilities declared — `SCOPE.md` contains a `capability`
fenced block OR `## Provided Capabilities` section says "none" explicitly
- C12: agents template applied — CLAUDE.md contains `get_kaizen_agent` or
`kaizen` reference
- C13: consistency check clean — 0 FAIL and 0 WARN (C-12 exempt)
- C14: host paths registered — `host_paths` non-empty
Output per criterion: PASS / FAIL / WARN / SKIP (with reason).
Overall tier: core_pass, standard_pass, full_pass (boolean each).
CLI flags: `--repo SLUG`, `--all`, `--json`, `--fix` (no-op for now,
reserved for future auto-remediation).
Makefile targets:
```
make check-doi REPO=<slug>
make check-doi-all
make check-doi REPO=<slug> JSON=1
```
---
## Task: API endpoint GET /repos/{slug}/doi
```task
id: CUST-WP-0024-T02
status: done
priority: high
state_hub_task_id: "97801608-2cad-40e6-ba0c-d209bcc3ec04"
```
Add to `api/routers/repos.py`:
`GET /repos/{slug}/doi` — runs the DoI check logic server-side and returns:
```json
{
"repo_slug": "llm-connect",
"tier": "standard", // "none" | "core" | "standard" | "full"
"core_pass": true,
"standard_pass": true,
"full_pass": false,
"criteria": [
{"id": "C1", "label": "Registered", "tier": "core", "status": "pass"},
...
],
"checked_at": "2026-03-20T..."
}
```
Also add `GET /repos/doi/summary` — returns all repos with their tier,
sorted by tier ascending (worst first).
Add Pydantic schemas `DoICriterion`, `DoIReport`, `DoISummaryEntry` to
`api/schemas/doi.py`.
---
## Task: MCP tool — check_repo_doi
```task
id: CUST-WP-0024-T03
status: done
priority: medium
state_hub_task_id: "3e18a74a-576a-4491-b7cf-3d6ba7746a0f"
```
Add to `mcp_server/server.py`:
`check_repo_doi(repo_slug)` — returns the DoI report for a single repo.
`get_doi_summary()` — returns all repos sorted by tier (worst first),
useful for session orientation.
---
## Task: Repositories dashboard page — DoI integration
```task
id: CUST-WP-0024-T04
status: done
priority: high
state_hub_task_id: "ab1a843c-60e3-4eba-ae6a-dec7d74c03f4"
```
Review and update `dashboard/src/repos.md` (or whichever page shows the
repo list) to include:
- DoI tier badge per repo (None / Core / Standard / Full) with colour coding
(red / amber / yellow / green)
- Expandable per-repo criterion detail (collapsible row or modal)
- Summary KPI: count of repos at each tier
- Link to the Repository DoI policy page
If `repos.md` does not exist, create it. If it already shows repo data,
enrich it in-place.
---
## Task: Consistency check — run fix-consistency
```task
id: CUST-WP-0024-T05
status: done
priority: low
state_hub_task_id: "5372bde5-b0e5-4993-94a0-ffa9b7bb26e9"
```
After all code is written and tested:
- Run `make fix-consistency REPO=the-custodian`
- Commit all new/modified files