Compare commits
14 Commits
f85c5e4d49
...
cbad0dc958
| Author | SHA1 | Date | |
|---|---|---|---|
| cbad0dc958 | |||
| 634642cb52 | |||
| 1e0ae37c89 | |||
| 51e95ec21a | |||
| d63e7310d5 | |||
| 6ae5cb6bf7 | |||
| 7dec3ac9ee | |||
| 66e3a9afe4 | |||
| f94ee008b5 | |||
| a7b26ef6de | |||
| 19d2b3cd31 | |||
| 0f9266cd91 | |||
| c7a893f068 | |||
| 118c5628e9 |
1
.gitignore
vendored
1
.gitignore
vendored
@@ -184,4 +184,5 @@ state-hub/dashboard/node_modules/
|
||||
state-hub/dashboard/.observablehq/
|
||||
state-hub/dashboard/src/.observablehq/
|
||||
state-hub/dashboard/dist/
|
||||
state-hub/kubectl
|
||||
|
||||
|
||||
@@ -309,6 +309,23 @@ Use this structure when creating or rewriting SCOPE.md:
|
||||
|
||||
---
|
||||
|
||||
## Provided Capabilities
|
||||
|
||||
<!-- What can this repo's domain provide to other domains on request? -->
|
||||
<!-- Each capability block is parsed by the state-hub capability catalog ingest. -->
|
||||
<!-- Remove the examples and add your own, or leave empty if none. -->
|
||||
|
||||
<!--
|
||||
```capability
|
||||
type: infrastructure
|
||||
title: Example capability title
|
||||
description: What this capability provides, in one or two sentences.
|
||||
keywords: [keyword1, keyword2, keyword3]
|
||||
```
|
||||
-->
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
<!-- Anything else worth knowing. Keep it short. -->
|
||||
|
||||
174
canon/architecture/adr-003-materialized-derived-state.md
Normal file
174
canon/architecture/adr-003-materialized-derived-state.md
Normal file
@@ -0,0 +1,174 @@
|
||||
---
|
||||
id: ADR-003
|
||||
type: architecture-decision-record
|
||||
title: "Materialized Derived State with Fingerprint Invalidation for Repo-Sourced Data"
|
||||
status: accepted
|
||||
decided_by: Bernd Worsch
|
||||
date: "2026-03-20"
|
||||
tags: ["architecture", "state-hub", "caching", "read-model", "materialized-view", "derived-state"]
|
||||
---
|
||||
|
||||
# ADR-003: Materialized Derived State with Fingerprint Invalidation
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
## Context
|
||||
|
||||
The Custodian State Hub is a **read model** (CQRS terminology) — its data is
|
||||
fully derivable from canonical sources that live in repositories and the
|
||||
filesystem. No state-hub data is authoritative; it is always a derived view
|
||||
of what the repos contain.
|
||||
|
||||
Several categories of data fit this description:
|
||||
|
||||
| Data | Canonical source | State-hub table |
|
||||
|---|---|---|
|
||||
| SBOM dependencies | `uv.lock`, `package-lock.json`, etc. | `sbom_entries` |
|
||||
| Third-party service declarations | `tpsc.yaml` | `tpsc_entries` |
|
||||
| Provided capabilities | `SCOPE.md` `capability` blocks | `capability_catalog` |
|
||||
| DoI compliance tier | 14 criteria across repo files + DB | `doi_cache` |
|
||||
| Workplan task status | `workplans/*.md` | `tasks` |
|
||||
|
||||
Early implementations either recomputed this data on every request (too slow)
|
||||
or ingested it once without invalidation (stale data goes undetected). Neither
|
||||
is acceptable for a system designed to give accurate, fast orientation.
|
||||
|
||||
The `doi_cache` table, introduced in CUST-WP-0024, demonstrated a pattern that
|
||||
solves both problems. This ADR formalises that pattern and mandates its use
|
||||
for all repo-sourced derived data.
|
||||
|
||||
## Pattern Name
|
||||
|
||||
**Materialized Derived State with Fingerprint Invalidation.**
|
||||
|
||||
This pattern is known under several names in the literature:
|
||||
|
||||
- **Materialized View** (SQL standard, PostgreSQL) — the stored result of a
|
||||
query or computation, refreshed on demand when source data changes.
|
||||
- **Derived Data Store** (Kleppmann, *Designing Data-Intensive Applications*,
|
||||
Ch. 3 & 11) — a system whose entire dataset can be rebuilt from upstream
|
||||
sources; it is never the source of truth.
|
||||
- **Read Model / Projection** (CQRS / Event Sourcing) — a pre-computed view
|
||||
maintained alongside a write model, rebuilt when relevant events occur.
|
||||
- **Fingerprint-based / Content-addressed invalidation** — analogous to HTTP
|
||||
ETags: a cache entry is valid as long as a composite hash/timestamp of its
|
||||
inputs matches the stored value.
|
||||
|
||||
The State Hub already documents itself as a read model. This ADR extends that
|
||||
principle to specify *how* the read model stays fresh.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. All repo-sourced derived data MUST be materialised in the DB
|
||||
|
||||
Data computed from repository files or repo records must be stored in a
|
||||
dedicated table rather than recomputed per request. Direct computation on
|
||||
every API call is only permissible for development tooling or when explicitly
|
||||
forced by the caller.
|
||||
|
||||
### 2. Each materialised table MUST carry a `fingerprint` column
|
||||
|
||||
The fingerprint is a deterministic string encoding all inputs that affect the
|
||||
computed result. It is compared on each read; if unchanged, the stored result
|
||||
is returned without recomputation. If changed, the result is recomputed and
|
||||
the stored value is updated.
|
||||
|
||||
**Fingerprint composition rules:**
|
||||
- Include the `updated_at` timestamp of every DB record that feeds the
|
||||
computation (repo record, related domain, goals, snapshots).
|
||||
- Include the `mtime` (filesystem modification time) of every file that feeds
|
||||
the computation (`SCOPE.md`, `CLAUDE.md`, lockfiles, `tpsc.yaml`, etc.).
|
||||
- Join all components with `|` as a pipe-separated string — no hashing needed
|
||||
since the string is compared by equality, not transmitted to clients.
|
||||
- If a file is absent, encode `filename:absent` rather than omitting it, so
|
||||
file creation also triggers invalidation.
|
||||
|
||||
**Reference implementation:** `state-hub/api/doi_engine.py::compute_fingerprint()`
|
||||
|
||||
### 3. Every materialised endpoint MUST support `?force_refresh=true`
|
||||
|
||||
Callers must always be able to bypass the cache and trigger a fresh
|
||||
computation. This is the escape hatch for debugging, post-ingest verification,
|
||||
and scheduled background refresh jobs.
|
||||
|
||||
### 4. Writes to source data SHOULD update the repo record's `updated_at`
|
||||
|
||||
Operations that change source data (SBOM ingest, TPSC ingest, capability
|
||||
ingest) must ensure `managed_repos.updated_at` is refreshed so the
|
||||
fingerprint detects the change on the next read. Where data lives in a
|
||||
related table (e.g. `tpsc_snapshots`), the fingerprint must include that
|
||||
table's `max(snapshot_at)` directly rather than relying on the repo record.
|
||||
|
||||
### 5. The DB is never the source of truth — the rebuild principle holds
|
||||
|
||||
Per ADR-001, the state-hub must be rebuildable from scratch by re-ingesting
|
||||
all canonical sources. Materialised tables are **caches**, not records of
|
||||
authority. They may be wiped and repopulated at any time without data loss.
|
||||
This means:
|
||||
- No materialised table may be the only copy of any information.
|
||||
- Schema migrations that wipe a materialised table are safe and expected.
|
||||
- Background jobs that periodically re-ingest all repos are valid and
|
||||
encouraged.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- **Fast reads in steady state** — after the first computation, subsequent
|
||||
reads hit the DB with no filesystem or subprocess overhead.
|
||||
- **Accurate on change** — fingerprint invalidation ensures stale data is
|
||||
never silently served; the cache refreshes exactly when needed.
|
||||
- **Debuggable** — `force_refresh=true` and `checked_at` timestamps make it
|
||||
easy to see when a value was last computed and to trigger a recheck.
|
||||
- **Consistent with the read model principle** — the pattern makes explicit
|
||||
what was always implied: state-hub data is derived, not authoritative.
|
||||
|
||||
### Negative / Trade-offs
|
||||
|
||||
- **First-call latency** — cache misses are expensive (filesystem reads,
|
||||
subprocess calls, HTTP self-calls). Mitigated by pre-warming caches at
|
||||
startup or after ingest.
|
||||
- **Fingerprint completeness** — if a new input is added to a computation and
|
||||
not added to the fingerprint, stale results will be silently returned. The
|
||||
fingerprint must be kept in sync with the computation.
|
||||
- **Filesystem dependency** — file mtimes are volatile (e.g. `git checkout`
|
||||
rewrites mtimes). In practice this means a cache miss after every checkout,
|
||||
not a correctness problem.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
When adding a new category of repo-sourced derived data:
|
||||
|
||||
- [ ] Create a `_cache` or `_snapshots` table with `fingerprint` and
|
||||
`checked_at` columns.
|
||||
- [ ] Implement `compute_fingerprint(repo, ...)` in the relevant module.
|
||||
- [ ] Add `?force_refresh=true` query parameter to the read endpoint.
|
||||
- [ ] Ensure the ingest script (or write path) touches `managed_repos.updated_at`
|
||||
or includes a related table's `max(timestamp)` in the fingerprint.
|
||||
- [ ] Verify the cache can be wiped and repopulated without data loss.
|
||||
- [ ] Document which inputs are included in the fingerprint in a comment
|
||||
alongside `compute_fingerprint`.
|
||||
|
||||
## Current Implementations
|
||||
|
||||
| Derived data | Table | Fingerprint inputs | Force-refresh |
|
||||
|---|---|---|---|
|
||||
| DoI compliance tier | `doi_cache` | `repo.updated_at`, `max(tpsc_snapshots.snapshot_at)`, `max(repo_goals.updated_at)`, `mtime(SCOPE.md)`, `mtime(CLAUDE.md)`, `mtime(tpsc.yaml)` | `?force_refresh=true` |
|
||||
|
||||
## Planned Applications
|
||||
|
||||
| Derived data | Table (proposed) | Notes |
|
||||
|---|---|---|
|
||||
| SBOM summary stats | `sbom_cache` | Fingerprint: `max(sbom_snapshots.snapshot_at)` |
|
||||
| Capability declarations | `capability_cache` | Fingerprint: `mtime(SCOPE.md)`, `repo.updated_at` |
|
||||
| Workplan status summary | Already handled by consistency checker | Fingerprint: workplan file mtimes |
|
||||
|
||||
## Related
|
||||
|
||||
- ADR-001: Workplans and Work Items Are Repository Artefacts
|
||||
- ADR-002: Custodian Agent Runtime Design
|
||||
- `state-hub/api/doi_engine.py` — reference implementation
|
||||
- `state-hub/api/models/doi_cache.py` — reference schema
|
||||
- `state-hub/migrations/versions/k8f9a0b1c2d3_doi_cache.py` — reference migration
|
||||
22
canon/tpsc/anthropic-api.yaml
Normal file
22
canon/tpsc/anthropic-api.yaml
Normal file
@@ -0,0 +1,22 @@
|
||||
slug: anthropic-api
|
||||
name: Anthropic API (Claude)
|
||||
provider: Anthropic, PBC
|
||||
category: llm_inference
|
||||
website_url: https://anthropic.com
|
||||
pricing_model: usage_based
|
||||
gdpr_maturity: developing
|
||||
gdpr_notes: >
|
||||
DPA available. SCCs provided for EU→US transfers. Data processed in the US.
|
||||
Prompts and completions are not used for training by default (API usage).
|
||||
Data may be retained for up to 30 days for trust & safety review.
|
||||
Privacy Shield successor mechanisms in place for international transfers.
|
||||
Reference: https://www.anthropic.com/legal/data-processing-agreement
|
||||
dpa_available: true
|
||||
tos_url: https://www.anthropic.com/legal/aup
|
||||
privacy_policy_url: https://www.anthropic.com/legal/privacy
|
||||
data_processing_regions:
|
||||
- us
|
||||
data_retention_notes: >
|
||||
API inputs/outputs not used for model training. Retained up to 30 days
|
||||
for safety review. Enterprise customers can negotiate reduced retention.
|
||||
status: active
|
||||
26
canon/tpsc/gemini-api.yaml
Normal file
26
canon/tpsc/gemini-api.yaml
Normal file
@@ -0,0 +1,26 @@
|
||||
slug: gemini-api
|
||||
name: Google Gemini API
|
||||
provider: Google LLC
|
||||
category: llm_inference
|
||||
website_url: https://ai.google.dev
|
||||
pricing_model: usage_based
|
||||
gdpr_maturity: defined
|
||||
gdpr_notes: >
|
||||
Google Cloud (including Vertex AI / Gemini API via Google Cloud) has a
|
||||
comprehensive GDPR compliance programme. DPA is part of Google Cloud ToS.
|
||||
Data Processing Addendum and SCCs available. Google Cloud has EU data
|
||||
residency options (data can stay in EU). Google AI Studio (free tier)
|
||||
has weaker protections — data may be used to improve Google products.
|
||||
Use Vertex AI / Google Cloud API endpoint for GDPR-adequate usage.
|
||||
Reference: https://cloud.google.com/terms/data-processing-addendum
|
||||
dpa_available: true
|
||||
tos_url: https://ai.google.dev/gemini-api/terms
|
||||
privacy_policy_url: https://policies.google.com/privacy
|
||||
data_processing_regions:
|
||||
- us
|
||||
- eu # when using Vertex AI with EU region selection
|
||||
data_retention_notes: >
|
||||
Google AI Studio: prompts may be reviewed by human raters and used to
|
||||
improve models. Google Cloud / Vertex AI: data not used for training,
|
||||
retained per Cloud data retention policy (configurable).
|
||||
status: active
|
||||
24
canon/tpsc/openai-api.yaml
Normal file
24
canon/tpsc/openai-api.yaml
Normal file
@@ -0,0 +1,24 @@
|
||||
slug: openai-api
|
||||
name: OpenAI API
|
||||
provider: OpenAI, Inc.
|
||||
category: llm_inference
|
||||
website_url: https://openai.com
|
||||
pricing_model: usage_based
|
||||
gdpr_maturity: developing
|
||||
gdpr_notes: >
|
||||
DPA available (Data Processing Addendum). Standard Contractual Clauses (SCCs)
|
||||
provided for EU→US data transfers. Data is processed in the US.
|
||||
Input/output retained up to 30 days for safety monitoring unless opted out
|
||||
via the API zero-data-retention setting. Zero-data-retention is available
|
||||
on eligible endpoints. Not suitable for sensitive personal data without a
|
||||
signed DPA and explicit zero-retention configuration.
|
||||
Reference: https://openai.com/policies/data-processing-addendum
|
||||
dpa_available: true
|
||||
tos_url: https://openai.com/policies/terms-of-use
|
||||
privacy_policy_url: https://openai.com/policies/privacy-policy
|
||||
data_processing_regions:
|
||||
- us
|
||||
data_retention_notes: >
|
||||
Default: 30 days for abuse monitoring. Zero-data-retention available
|
||||
on eligible API endpoints via opt-in. Training opt-out available.
|
||||
status: active
|
||||
27
canon/tpsc/openrouter-api.yaml
Normal file
27
canon/tpsc/openrouter-api.yaml
Normal file
@@ -0,0 +1,27 @@
|
||||
slug: openrouter-api
|
||||
name: OpenRouter API
|
||||
provider: OpenRouter, Inc.
|
||||
category: llm_routing
|
||||
website_url: https://openrouter.ai
|
||||
pricing_model: usage_based
|
||||
gdpr_maturity: initial
|
||||
gdpr_notes: >
|
||||
OpenRouter is a US-based routing proxy for multiple LLM providers.
|
||||
Privacy policy exists but as of early 2026 no formal DPA or SCCs are
|
||||
publicly available. Requests are forwarded to underlying providers
|
||||
(OpenAI, Anthropic, Google, Mistral, etc.) each with their own data
|
||||
handling. GDPR compliance is therefore dependent on both OpenRouter
|
||||
and the selected downstream model provider. Not recommended for
|
||||
processing personal data in corporate/regulated environments without
|
||||
a signed DPA. Suitable for development, prototyping, model comparison.
|
||||
Reference: https://openrouter.ai/privacy
|
||||
dpa_available: false
|
||||
tos_url: https://openrouter.ai/terms
|
||||
privacy_policy_url: https://openrouter.ai/privacy
|
||||
data_processing_regions:
|
||||
- us
|
||||
data_retention_notes: >
|
||||
Requests forwarded to upstream providers. OpenRouter may log requests
|
||||
for billing and abuse prevention. Retention policy not formally published.
|
||||
Downstream provider retention policies apply per selected model.
|
||||
status: active
|
||||
@@ -1,4 +1,4 @@
|
||||
.PHONY: install install-cli db db-tools migrate seed api dashboard check test clean register-project validate-adr add-domain rename-domain add-repo list-repos register-path cleanup-stale tunnel tunnel-daemon tunnel-loop tunnel-status tunnel-stop install-hooks install-hooks-all gitea-inventory
|
||||
.PHONY: install install-cli db db-tools migrate seed api dashboard check test clean register-project validate-adr add-domain rename-domain add-repo list-repos register-path cleanup-stale tunnel tunnel-daemon tunnel-loop tunnel-status tunnel-stop tunnels-up tunnels-status tunnels-check install-hooks install-hooks-all gitea-inventory
|
||||
|
||||
COMPOSE = docker compose -f infra/docker-compose.yml --env-file .env
|
||||
|
||||
@@ -103,6 +103,19 @@ tunnel-stop:
|
||||
@pkill -f "autossh.*$(TUNNEL_PORT)" 2>/dev/null && echo "autossh stopped" || true
|
||||
@pkill -f "ssh.*-R $(TUNNEL_PORT)" 2>/dev/null && echo "ssh loop stopped" || true
|
||||
|
||||
## ops-bridge managed tunnels (preferred over tunnel-*/tunnel-daemon)
|
||||
## Requires ops-bridge: bridge is at /home/worsch/.local/bin/bridge
|
||||
tunnels-up:
|
||||
bridge up
|
||||
|
||||
tunnels-status:
|
||||
bridge status
|
||||
|
||||
## End-to-end check: verifies SSH process alive + remote port listening on COULOMBCORE.
|
||||
## Exits non-zero if any tunnel is not fully operational.
|
||||
tunnels-check:
|
||||
bridge check
|
||||
|
||||
## Start (or restart) the full backend — db + migrate + uvicorn.
|
||||
## Stops uvicorn on :8000 if already running, then starts fresh.
|
||||
api: db
|
||||
@@ -183,6 +196,30 @@ ingest-capabilities-all:
|
||||
uv run python scripts/ingest_capabilities.py --all \
|
||||
$(if $(DRY_RUN),--dry-run)
|
||||
|
||||
## Check Repository Definition of Integrated (DoI) criteria for a repo.
|
||||
## Usage: make check-doi REPO=llm-connect
|
||||
## Or: make check-doi-all
|
||||
## Add JSON=1 for machine-readable output.
|
||||
check-doi:
|
||||
@test -n "$(REPO)" || (echo "ERROR: REPO is required."; exit 1)
|
||||
uv run python scripts/check_doi.py --repo "$(REPO)" $(if $(JSON),--json)
|
||||
|
||||
check-doi-all:
|
||||
uv run python scripts/check_doi.py --all $(if $(JSON),--json)
|
||||
|
||||
## Ingest tpsc.yaml service declarations from a repo into the TPSC catalog.
|
||||
## Usage: make ingest-tpsc REPO=llm-connect
|
||||
## Or: make ingest-tpsc-all
|
||||
## Add DRY_RUN=1 to preview without writing.
|
||||
ingest-tpsc:
|
||||
@test -n "$(REPO)" || (echo "ERROR: REPO is required."; exit 1)
|
||||
uv run python scripts/ingest_tpsc.py --repo "$(REPO)" \
|
||||
$(if $(DRY_RUN),--dry-run)
|
||||
|
||||
ingest-tpsc-all:
|
||||
uv run python scripts/ingest_tpsc.py --all \
|
||||
$(if $(DRY_RUN),--dry-run)
|
||||
|
||||
## Run SBOM capture agent for a repo — generates/updates sbom-tools.yaml.
|
||||
## Usage: make capture-tools REPO=railiance-infra [REPO_PATH=/home/worsch/railiance-infra]
|
||||
## Add DRY_RUN=1 to preview without writing.
|
||||
|
||||
367
state-hub/api/doi_engine.py
Normal file
367
state-hub/api/doi_engine.py
Normal file
@@ -0,0 +1,367 @@
|
||||
"""DoI engine — evaluates all 14 Repository Definition of Integrated criteria.
|
||||
|
||||
Shared by the API endpoint (async) and the CLI check script (asyncio.run).
|
||||
All checks use only the repo dict from /repos/{slug} + HTTP calls to the API
|
||||
+ local filesystem reads. No direct DB access.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import socket
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
CriterionStatus = Literal["pass", "fail", "warn", "skip"]
|
||||
Tier = Literal["none", "core", "standard", "full"]
|
||||
|
||||
# Criteria that belong to each tier (in check order)
|
||||
CORE_IDS = {"C1", "C2", "C3", "C4"}
|
||||
STANDARD_IDS = {"C5", "C6", "C7", "C8", "C9"}
|
||||
FULL_IDS = {"C10", "C11", "C12", "C13", "C14"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class CriterionResult:
|
||||
id: str
|
||||
label: str
|
||||
tier: str
|
||||
status: CriterionStatus
|
||||
detail: str = ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class DoIReport:
|
||||
repo_slug: str
|
||||
tier: Tier
|
||||
core_pass: bool
|
||||
standard_pass: bool
|
||||
full_pass: bool
|
||||
criteria: list[CriterionResult] = field(default_factory=list)
|
||||
checked_at: str = field(default_factory=lambda: datetime.now(tz=timezone.utc).isoformat())
|
||||
|
||||
|
||||
def compute_fingerprint(
|
||||
repo: dict,
|
||||
latest_tpsc_snap_at: str | None,
|
||||
latest_goal_updated_at: str | None,
|
||||
) -> str:
|
||||
"""Compute a pipe-joined fingerprint of all inputs that affect DoI criteria.
|
||||
|
||||
If any component changes, the fingerprint changes and the cache is invalidated:
|
||||
- repo.updated_at → covers last_sbom_at, remote_url, host_paths, domain changes
|
||||
- latest_tpsc_snap_at → C9 (TPSC snapshot exists)
|
||||
- latest_goal_updated_at → C10 (active repo goal)
|
||||
- mtime of SCOPE.md, CLAUDE.md, tpsc.yaml → C5, C6, C9, C11, C12
|
||||
"""
|
||||
parts = [
|
||||
str(repo.get("updated_at") or ""),
|
||||
str(latest_tpsc_snap_at or ""),
|
||||
str(latest_goal_updated_at or ""),
|
||||
]
|
||||
repo_path = _resolve_path(repo)
|
||||
if repo_path:
|
||||
for fname in ("SCOPE.md", "CLAUDE.md", "tpsc.yaml"):
|
||||
f = Path(repo_path) / fname
|
||||
try:
|
||||
parts.append(f"{fname}:{f.stat().st_mtime:.3f}")
|
||||
except FileNotFoundError:
|
||||
parts.append(f"{fname}:absent")
|
||||
return "|".join(parts)
|
||||
|
||||
|
||||
def _resolve_path(repo: dict) -> str:
|
||||
hostname = socket.gethostname()
|
||||
host_paths = repo.get("host_paths") or {}
|
||||
candidates = []
|
||||
if host_paths.get(hostname):
|
||||
candidates.append(host_paths[hostname])
|
||||
if repo.get("local_path"):
|
||||
candidates.append(repo["local_path"])
|
||||
for raw in candidates:
|
||||
p = Path(raw).expanduser()
|
||||
if p.is_dir():
|
||||
return str(p)
|
||||
return ""
|
||||
|
||||
|
||||
def _get_sync(api_base: str, path: str, params: dict | None = None) -> object:
|
||||
url = f"{api_base}{path}"
|
||||
if params:
|
||||
q = "&".join(f"{k}={v}" for k, v in params.items() if v is not None)
|
||||
if q:
|
||||
url = f"{url}?{q}"
|
||||
req = urllib.request.Request(url, headers={"Accept": "application/json"})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=5) as r:
|
||||
return json.loads(r.read())
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
async def _get(api_base: str, path: str, params: dict | None = None) -> object:
|
||||
"""Async wrapper — runs blocking urllib in a thread so the event loop stays free."""
|
||||
return await asyncio.to_thread(_get_sync, api_base, path, params)
|
||||
|
||||
|
||||
async def _run_consistency(repo_slug: str, api_base: str) -> tuple[int, int, int]:
|
||||
"""Run consistency_check.py and return (fail, warn, info) counts."""
|
||||
script = Path(__file__).parent.parent / "scripts" / "consistency_check.py"
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"uv", "run", "python", str(script),
|
||||
"--repo", repo_slug,
|
||||
"--api-base", api_base,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
cwd=str(Path(__file__).parent.parent),
|
||||
)
|
||||
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=30)
|
||||
text = stdout.decode()
|
||||
fail = warn = info = 0
|
||||
for line in text.splitlines():
|
||||
if "Summary:" in line:
|
||||
parts = line.split("|")
|
||||
for p in parts:
|
||||
p = p.strip()
|
||||
if "fail" in p:
|
||||
try: fail = int(p.split()[0])
|
||||
except ValueError: pass
|
||||
elif "warn" in p:
|
||||
try: warn = int(p.split()[0])
|
||||
except ValueError: pass
|
||||
elif "info" in p:
|
||||
try: info = int(p.split()[0])
|
||||
except ValueError: pass
|
||||
return fail, warn, info
|
||||
|
||||
|
||||
async def evaluate(
|
||||
repo: dict,
|
||||
api_base: str = "http://127.0.0.1:8000",
|
||||
skip_consistency: bool = False,
|
||||
prefetch: dict | None = None,
|
||||
) -> DoIReport:
|
||||
"""Evaluate all 14 DoI criteria for a repo.
|
||||
|
||||
Args:
|
||||
repo: Repo dict (slug, domain_slug, local_path, remote_url, host_paths, last_sbom_at).
|
||||
api_base: API base URL — only used when prefetch is absent.
|
||||
skip_consistency: Skip C7/C13 subprocess calls (used in summary mode).
|
||||
prefetch: Optional pre-fetched bulk data to avoid HTTP self-calls:
|
||||
{
|
||||
"domain_status": {"custodian": "active", ...}, # slug → status
|
||||
"tpsc_snap_counts": {"llm-connect": 1, ...}, # repo_slug → count
|
||||
"active_goal_counts": {"llm-connect": 0, ...}, # repo_slug → count
|
||||
}
|
||||
"""
|
||||
slug = repo.get("slug", "unknown")
|
||||
results: list[CriterionResult] = []
|
||||
|
||||
def _r(id: str, label: str, tier: str, status: CriterionStatus, detail: str = "") -> CriterionResult:
|
||||
r = CriterionResult(id=id, label=label, tier=tier, status=status, detail=detail)
|
||||
results.append(r)
|
||||
return r
|
||||
|
||||
# ── Tier 1: Core ─────────────────────────────────────────────────────────
|
||||
|
||||
# C1: registered
|
||||
_r("C1", "Registered in state-hub", "core", "pass", "Repo record exists")
|
||||
|
||||
# C2: domain assigned and active
|
||||
domain_slug = repo.get("domain_slug") or ""
|
||||
if not domain_slug:
|
||||
_r("C2", "Domain assigned", "core", "fail", "No domain_slug on repo record")
|
||||
else:
|
||||
if prefetch and "domain_status" in prefetch:
|
||||
dom_status = prefetch["domain_status"].get(domain_slug)
|
||||
else:
|
||||
d = await _get(api_base, f"/domains/{domain_slug}/")
|
||||
dom_status = d.get("status") if d else None
|
||||
if dom_status == "active":
|
||||
_r("C2", "Domain assigned", "core", "pass", f"domain: {domain_slug}")
|
||||
elif dom_status:
|
||||
_r("C2", "Domain assigned", "core", "warn", f"Domain '{domain_slug}' status: {dom_status}")
|
||||
else:
|
||||
_r("C2", "Domain assigned", "core", "fail", f"Domain '{domain_slug}' not found")
|
||||
|
||||
# C3: local path resolves
|
||||
repo_path = _resolve_path(repo)
|
||||
if repo_path:
|
||||
_r("C3", "Local path resolves", "core", "pass", repo_path)
|
||||
else:
|
||||
raw = repo.get("local_path") or "(none)"
|
||||
_r("C3", "Local path resolves", "core", "fail", f"Path not accessible: {raw}")
|
||||
|
||||
# C4: remote URL set
|
||||
remote = repo.get("remote_url") or ""
|
||||
if remote.strip():
|
||||
_r("C4", "Remote URL set", "core", "pass", remote)
|
||||
else:
|
||||
_r("C4", "Remote URL set", "core", "fail", "remote_url is empty")
|
||||
|
||||
# ── Tier 2: Standard ─────────────────────────────────────────────────────
|
||||
|
||||
# C5: SCOPE.md
|
||||
if not repo_path:
|
||||
_r("C5", "SCOPE.md present", "standard", "skip", "Local path unavailable")
|
||||
elif (Path(repo_path) / "SCOPE.md").exists():
|
||||
_r("C5", "SCOPE.md present", "standard", "pass")
|
||||
else:
|
||||
_r("C5", "SCOPE.md present", "standard", "fail", "SCOPE.md not found at repo root")
|
||||
|
||||
# C6: CLAUDE.md
|
||||
if not repo_path:
|
||||
_r("C6", "CLAUDE.md present", "standard", "skip", "Local path unavailable")
|
||||
elif (Path(repo_path) / "CLAUDE.md").exists():
|
||||
_r("C6", "CLAUDE.md present", "standard", "pass")
|
||||
else:
|
||||
_r("C6", "CLAUDE.md present", "standard", "fail", "CLAUDE.md not found at repo root")
|
||||
|
||||
# C7: workplan convention — consistency check 0 FAIL
|
||||
if skip_consistency:
|
||||
_r("C7", "Workplan convention (0 FAIL)", "standard", "skip", "Not checked in summary mode — use /repos/{slug}/doi for full check")
|
||||
else:
|
||||
try:
|
||||
fail, warn, _ = await _run_consistency(slug, api_base)
|
||||
if fail == 0:
|
||||
_r("C7", "Workplan convention (0 FAIL)", "standard", "pass", f"consistency: {fail} fail / {warn} warn")
|
||||
else:
|
||||
_r("C7", "Workplan convention (0 FAIL)", "standard", "fail", f"consistency: {fail} fail / {warn} warn")
|
||||
except Exception as e:
|
||||
_r("C7", "Workplan convention (0 FAIL)", "standard", "skip", f"Could not run consistency check: {e}")
|
||||
|
||||
# C8: SBOM ingested
|
||||
last_sbom = repo.get("last_sbom_at")
|
||||
if last_sbom:
|
||||
_r("C8", "SBOM ingested", "standard", "pass", f"last ingested: {last_sbom[:10]}")
|
||||
else:
|
||||
_r("C8", "SBOM ingested", "standard", "fail", "last_sbom_at not set — run make ingest-sbom")
|
||||
|
||||
# C9: TPSC declared (tpsc.yaml present + snapshot exists)
|
||||
tpsc_file_ok = repo_path and (Path(repo_path) / "tpsc.yaml").exists()
|
||||
if prefetch and "tpsc_snap_counts" in prefetch:
|
||||
has_snap = (prefetch["tpsc_snap_counts"].get(slug, 0) > 0)
|
||||
snap_count = prefetch["tpsc_snap_counts"].get(slug, 0)
|
||||
else:
|
||||
tpsc_snaps = await _get(api_base, "/tpsc/snapshots/", {"repo_slug": slug}) or []
|
||||
has_snap = len(tpsc_snaps) > 0
|
||||
snap_count = len(tpsc_snaps)
|
||||
if not repo_path:
|
||||
_r("C9", "TPSC declared", "standard", "skip", "Local path unavailable")
|
||||
elif tpsc_file_ok and has_snap:
|
||||
_r("C9", "TPSC declared", "standard", "pass", f"{snap_count} snapshot(s)")
|
||||
elif tpsc_file_ok and not has_snap:
|
||||
_r("C9", "TPSC declared", "standard", "warn", "tpsc.yaml exists but not yet ingested — run make ingest-tpsc")
|
||||
elif not tpsc_file_ok:
|
||||
_r("C9", "TPSC declared", "standard", "fail", "tpsc.yaml missing at repo root")
|
||||
|
||||
# ── Tier 3: Full ─────────────────────────────────────────────────────────
|
||||
|
||||
# C10: active repo goal
|
||||
if prefetch and "active_goal_counts" in prefetch:
|
||||
active_goal_count = prefetch["active_goal_counts"].get(slug, 0)
|
||||
else:
|
||||
goals = await _get(api_base, "/repo-goals/", {"repo_slug": slug}) or []
|
||||
active_goal_count = sum(1 for g in goals if g.get("status") == "active")
|
||||
if active_goal_count > 0:
|
||||
_r("C10", "Active repo goal", "full", "pass", f"{active_goal_count} active goal(s)")
|
||||
else:
|
||||
_r("C10", "Active repo goal", "full", "fail", "No active repo goal — create one with create_repo_goal()")
|
||||
|
||||
# C11: Provided Capabilities declared in SCOPE.md
|
||||
if not repo_path:
|
||||
_r("C11", "Provided Capabilities declared", "full", "skip", "Local path unavailable")
|
||||
else:
|
||||
scope = Path(repo_path) / "SCOPE.md"
|
||||
if not scope.exists():
|
||||
_r("C11", "Provided Capabilities declared", "full", "skip", "SCOPE.md absent")
|
||||
else:
|
||||
text = scope.read_text()
|
||||
has_cap_block = "```capability" in text
|
||||
has_none_explicit = "## Provided Capabilities" in text and (
|
||||
"none" in text.lower().split("## provided capabilities")[-1][:200]
|
||||
or "no capabilities" in text.lower().split("## provided capabilities")[-1][:200]
|
||||
)
|
||||
if has_cap_block:
|
||||
_r("C11", "Provided Capabilities declared", "full", "pass", "capability block(s) found in SCOPE.md")
|
||||
elif has_none_explicit:
|
||||
_r("C11", "Provided Capabilities declared", "full", "pass", "Explicitly declared none in SCOPE.md")
|
||||
elif "## Provided Capabilities" in text:
|
||||
_r("C11", "Provided Capabilities declared", "full", "warn",
|
||||
"Section present but no capability block or explicit none — add blocks or state 'none'")
|
||||
else:
|
||||
_r("C11", "Provided Capabilities declared", "full", "fail",
|
||||
"No '## Provided Capabilities' section in SCOPE.md")
|
||||
|
||||
# C12: agents template applied (CLAUDE.md mentions kaizen)
|
||||
if not repo_path:
|
||||
_r("C12", "Agents template applied", "full", "skip", "Local path unavailable")
|
||||
else:
|
||||
claude_md = Path(repo_path) / "CLAUDE.md"
|
||||
if not claude_md.exists():
|
||||
_r("C12", "Agents template applied", "full", "skip", "CLAUDE.md absent")
|
||||
else:
|
||||
text = claude_md.read_text()
|
||||
if "get_kaizen_agent" in text or "kaizen" in text.lower():
|
||||
_r("C12", "Agents template applied", "full", "pass")
|
||||
else:
|
||||
_r("C12", "Agents template applied", "full", "fail",
|
||||
"CLAUDE.md has no kaizen agent reference")
|
||||
|
||||
# C13: consistency check clean (0 FAIL, 0 WARN — C-12 exempt)
|
||||
if skip_consistency:
|
||||
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "skip", "Not checked in summary mode — use /repos/{slug}/doi for full check")
|
||||
else:
|
||||
try:
|
||||
fail, warn, _ = await _run_consistency(slug, api_base)
|
||||
if fail == 0 and warn == 0:
|
||||
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "pass")
|
||||
elif fail == 0 and warn > 0:
|
||||
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "warn",
|
||||
f"{warn} warn(s) — C-12 legacy tasks may be exempt")
|
||||
else:
|
||||
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "fail",
|
||||
f"{fail} fail(s), {warn} warn(s)")
|
||||
except Exception as e:
|
||||
_r("C13", "Consistency check clean (0 FAIL/WARN)", "full", "skip", f"Could not run: {e}")
|
||||
|
||||
# C14: host paths registered
|
||||
host_paths = repo.get("host_paths") or {}
|
||||
if host_paths:
|
||||
_r("C14", "Host paths registered", "full", "pass",
|
||||
f"{len(host_paths)} host(s): {', '.join(host_paths.keys())}")
|
||||
else:
|
||||
_r("C14", "Host paths registered", "full", "fail",
|
||||
"host_paths empty — run update_repo_path() for each active machine")
|
||||
|
||||
# ── Compute tier ─────────────────────────────────────────────────────────
|
||||
by_id = {r.id: r for r in results}
|
||||
|
||||
def _tier_pass(ids: set[str]) -> bool:
|
||||
return all(by_id[i].status in ("pass", "warn") for i in ids if i in by_id)
|
||||
|
||||
core_pass = _tier_pass(CORE_IDS)
|
||||
standard_pass = core_pass and _tier_pass(STANDARD_IDS)
|
||||
full_pass = standard_pass and _tier_pass(FULL_IDS)
|
||||
|
||||
if full_pass:
|
||||
tier: Tier = "full"
|
||||
elif standard_pass:
|
||||
tier = "standard"
|
||||
elif core_pass:
|
||||
tier = "core"
|
||||
else:
|
||||
tier = "none"
|
||||
|
||||
return DoIReport(
|
||||
repo_slug=slug,
|
||||
tier=tier,
|
||||
core_pass=core_pass,
|
||||
standard_pass=standard_pass,
|
||||
full_pass=full_pass,
|
||||
criteria=results,
|
||||
)
|
||||
@@ -6,7 +6,7 @@ from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from api.database import engine
|
||||
from api.routers import decisions, extension_points, progress, state, tasks, technical_debt, topics, workstreams, workstream_dependencies
|
||||
from api.routers import domains, repos, contributions, sbom, policy, domain_goals, repo_goals, messages, capability_requests
|
||||
from api.routers import domains, repos, contributions, sbom, policy, domain_goals, repo_goals, messages, capability_requests, tpsc
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
@@ -48,6 +48,7 @@ app.include_router(contributions.router)
|
||||
app.include_router(sbom.router)
|
||||
app.include_router(messages.router)
|
||||
app.include_router(capability_requests.router)
|
||||
app.include_router(tpsc.router)
|
||||
app.include_router(state.router)
|
||||
app.include_router(policy.router)
|
||||
|
||||
|
||||
@@ -17,6 +17,8 @@ from api.models.sbom_entry import SBOMEntry, Ecosystem
|
||||
from api.models.agent_message import AgentMessage
|
||||
from api.models.capability_catalog import CapabilityCatalog
|
||||
from api.models.capability_request import CapabilityRequest
|
||||
from api.models.tpsc import TPSCCatalog, TPSCSnapshot, TPSCEntry
|
||||
from api.models.doi_cache import DOICache
|
||||
|
||||
__all__ = [
|
||||
"Base",
|
||||
@@ -38,4 +40,6 @@ __all__ = [
|
||||
"AgentMessage",
|
||||
"CapabilityCatalog",
|
||||
"CapabilityRequest",
|
||||
"TPSCCatalog", "TPSCSnapshot", "TPSCEntry",
|
||||
"DOICache",
|
||||
]
|
||||
|
||||
@@ -65,6 +65,7 @@ class CapabilityRequest(Base, TimestampMixin):
|
||||
)
|
||||
|
||||
resolution_note: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
routing_note: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
accepted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
|
||||
27
state-hub/api/models/doi_cache.py
Normal file
27
state-hub/api/models/doi_cache.py
Normal file
@@ -0,0 +1,27 @@
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, String, Text, func
|
||||
from sqlalchemy.dialects.postgresql import JSON, UUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from api.models.base import Base
|
||||
|
||||
|
||||
class DOICache(Base):
|
||||
__tablename__ = "doi_cache"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
repo_id: Mapped[uuid.UUID] = mapped_column(
|
||||
UUID(as_uuid=True),
|
||||
ForeignKey("managed_repos.id", ondelete="CASCADE"),
|
||||
nullable=False, unique=True, index=True,
|
||||
)
|
||||
tier: Mapped[str] = mapped_column(String(20), nullable=False)
|
||||
core_pass: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default="false")
|
||||
standard_pass: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default="false")
|
||||
full_pass: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default="false")
|
||||
criteria: Mapped[list | None] = mapped_column(JSON, nullable=True)
|
||||
# Pipe-joined string of timestamps/mtimes used to detect staleness
|
||||
fingerprint: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
checked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
64
state-hub/api/models/tpsc.py
Normal file
64
state-hub/api/models/tpsc.py
Normal file
@@ -0,0 +1,64 @@
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, String, Text, func
|
||||
from sqlalchemy.dialects.postgresql import JSON, UUID
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
|
||||
from api.models.base import Base
|
||||
|
||||
|
||||
class TPSCCatalog(Base):
|
||||
__tablename__ = "tpsc_catalog"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
slug: Mapped[str] = mapped_column(String(100), nullable=False, unique=True, index=True)
|
||||
name: Mapped[str] = mapped_column(String(200), nullable=False)
|
||||
provider: Mapped[str | None] = mapped_column(String(200), nullable=True)
|
||||
category: Mapped[str | None] = mapped_column(String(100), nullable=True)
|
||||
website_url: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
# Pricing: free | paid | freemium | usage_based | unknown
|
||||
pricing_model: Mapped[str] = mapped_column(String(20), nullable=False, server_default="unknown")
|
||||
# GDPR maturity (CNIL/IAPP CMMI-aligned):
|
||||
# unknown | non_compliant | initial | developing | defined | managed | certified
|
||||
gdpr_maturity: Mapped[str] = mapped_column(String(20), nullable=False, server_default="unknown", index=True)
|
||||
gdpr_notes: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
dpa_available: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default="false")
|
||||
tos_url: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
privacy_policy_url: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
data_processing_regions: Mapped[list | None] = mapped_column(JSON, nullable=True)
|
||||
data_retention_notes: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
# status: active | deprecated
|
||||
status: Mapped[str] = mapped_column(String(20), nullable=False, server_default="active")
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
|
||||
|
||||
entries: Mapped[list["TPSCEntry"]] = relationship("TPSCEntry", back_populates="catalog_entry")
|
||||
|
||||
|
||||
class TPSCSnapshot(Base):
|
||||
__tablename__ = "tpsc_snapshots"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
repo_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("managed_repos.id", ondelete="SET NULL"), nullable=True, index=True)
|
||||
snapshot_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
source_file: Mapped[str | None] = mapped_column(String(200), nullable=True)
|
||||
entry_count: Mapped[int] = mapped_column(Integer, nullable=False, server_default="0")
|
||||
|
||||
entries: Mapped[list["TPSCEntry"]] = relationship("TPSCEntry", back_populates="snapshot", cascade="all, delete-orphan")
|
||||
|
||||
|
||||
class TPSCEntry(Base):
|
||||
__tablename__ = "tpsc_entries"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
snapshot_id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), ForeignKey("tpsc_snapshots.id", ondelete="CASCADE"), nullable=False, index=True)
|
||||
catalog_id: Mapped[uuid.UUID | None] = mapped_column(UUID(as_uuid=True), ForeignKey("tpsc_catalog.id", ondelete="SET NULL"), nullable=True)
|
||||
service_slug: Mapped[str] = mapped_column(String(100), nullable=False, index=True)
|
||||
purpose: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
# auth_type: api_key | oauth | cli | none | unknown
|
||||
auth_type: Mapped[str | None] = mapped_column(String(50), nullable=True)
|
||||
endpoint_override: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
notes: Mapped[str | None] = mapped_column(Text, nullable=True)
|
||||
|
||||
snapshot: Mapped["TPSCSnapshot"] = relationship("TPSCSnapshot", back_populates="entries")
|
||||
catalog_entry: Mapped["TPSCCatalog | None"] = relationship("TPSCCatalog", back_populates="entries")
|
||||
@@ -1,3 +1,4 @@
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
@@ -16,6 +17,7 @@ from api.schemas.capability_request import (
|
||||
CatalogRead,
|
||||
CapabilityRequestAccept,
|
||||
CapabilityRequestCreate,
|
||||
CapabilityRequestPatch,
|
||||
CapabilityRequestRead,
|
||||
CapabilityRequestStatusPatch,
|
||||
)
|
||||
@@ -100,8 +102,8 @@ async def create_request(
|
||||
req_domain = await _resolve_domain(body.requesting_domain, session)
|
||||
|
||||
# Route to provider
|
||||
fulfilling_domain_id, catalog_entry_id = await _route_capability(
|
||||
session, body.capability_type, body.description or ""
|
||||
fulfilling_domain_id, catalog_entry_id, routing_note = await _route_capability(
|
||||
session, body.capability_type, body.title, body.description or ""
|
||||
)
|
||||
|
||||
req = CapabilityRequest(
|
||||
@@ -115,6 +117,7 @@ async def create_request(
|
||||
blocking_task_id=body.blocking_task_id,
|
||||
fulfilling_domain_id=fulfilling_domain_id,
|
||||
catalog_entry_id=catalog_entry_id,
|
||||
routing_note=routing_note,
|
||||
)
|
||||
session.add(req)
|
||||
await session.flush() # get req.id before creating notification
|
||||
@@ -277,16 +280,70 @@ async def patch_request_status(
|
||||
return req
|
||||
|
||||
|
||||
@router.patch("/capability-requests/{request_id}", response_model=CapabilityRequestRead)
|
||||
async def patch_request(
|
||||
request_id: uuid.UUID,
|
||||
body: CapabilityRequestPatch,
|
||||
session: AsyncSession = Depends(get_session),
|
||||
) -> CapabilityRequest:
|
||||
"""Correct mutable metadata: catalog_entry_id (re-derives fulfilling domain),
|
||||
priority, blocking_task_id, fulfilling_workstream_id.
|
||||
Only fields present in the request body (non-None) are updated.
|
||||
"""
|
||||
req = await _get_request_or_404(request_id, session)
|
||||
|
||||
corrections: list[str] = []
|
||||
|
||||
if body.catalog_entry_id is not None:
|
||||
old_entry_id = req.catalog_entry_id
|
||||
entry = await session.get(CapabilityCatalog, body.catalog_entry_id)
|
||||
if entry is None:
|
||||
raise HTTPException(status_code=404, detail=f"Catalog entry '{body.catalog_entry_id}' not found")
|
||||
req.catalog_entry_id = entry.id
|
||||
# Re-derive fulfilling domain from catalog entry
|
||||
old_domain_id = req.fulfilling_domain_id
|
||||
req.fulfilling_domain_id = entry.domain_id
|
||||
corrections.append(
|
||||
f"catalog_entry: {old_entry_id} → {entry.id} ({entry.title}); "
|
||||
f"fulfilling_domain re-derived → {entry.domain_id}"
|
||||
)
|
||||
|
||||
if body.priority is not None:
|
||||
req.priority = body.priority
|
||||
corrections.append(f"priority → {body.priority}")
|
||||
|
||||
if body.blocking_task_id is not None:
|
||||
req.blocking_task_id = body.blocking_task_id
|
||||
corrections.append(f"blocking_task_id → {body.blocking_task_id}")
|
||||
|
||||
if body.fulfilling_workstream_id is not None:
|
||||
req.fulfilling_workstream_id = body.fulfilling_workstream_id
|
||||
corrections.append(f"fulfilling_workstream_id → {body.fulfilling_workstream_id}")
|
||||
|
||||
if not corrections:
|
||||
return req # no-op
|
||||
|
||||
correction_note = "hub correction: " + "; ".join(corrections)
|
||||
req.routing_note = (req.routing_note + "\n" + correction_note) if req.routing_note else correction_note
|
||||
|
||||
await session.commit()
|
||||
await session.refresh(req)
|
||||
return req
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Routing algorithm
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _route_capability(
|
||||
session: AsyncSession, capability_type: str, description: str
|
||||
) -> tuple[uuid.UUID | None, uuid.UUID | None]:
|
||||
"""Find the best-matching domain for a capability request.
|
||||
session: AsyncSession, capability_type: str, title: str, description: str
|
||||
) -> tuple[uuid.UUID | None, uuid.UUID | None, str]:
|
||||
"""Find the best-matching catalog entry for a capability request.
|
||||
|
||||
Returns (domain_id, catalog_entry_id) or (None, None) for broadcast.
|
||||
Returns (domain_id, catalog_entry_id, routing_note).
|
||||
Uses word-boundary matching on (title + description) combined to avoid
|
||||
false positives from substring matches (e.g. 'postgres' inside 'postgresql',
|
||||
'ha' inside 'has').
|
||||
"""
|
||||
q = select(CapabilityCatalog).where(
|
||||
CapabilityCatalog.capability_type == capability_type,
|
||||
@@ -294,20 +351,41 @@ async def _route_capability(
|
||||
)
|
||||
entries = list((await session.execute(q)).scalars().all())
|
||||
|
||||
if not entries:
|
||||
return None, None, f"no active catalog entries for type '{capability_type}' — broadcast"
|
||||
|
||||
if len(entries) == 1:
|
||||
return entries[0].domain_id, entries[0].id
|
||||
e = entries[0]
|
||||
return e.domain_id, e.id, f"single match: '{e.title}' (domain={e.domain_id})"
|
||||
|
||||
if len(entries) > 1 and description:
|
||||
desc_lower = description.lower()
|
||||
scored: list[tuple[int, CapabilityCatalog]] = []
|
||||
for entry in entries:
|
||||
score = sum(1 for kw in (entry.keywords or []) if kw.lower() in desc_lower)
|
||||
scored.append((score, entry))
|
||||
scored.sort(key=lambda x: -x[0])
|
||||
if scored[0][0] > 0 and (len(scored) < 2 or scored[0][0] > scored[1][0]):
|
||||
return scored[0][1].domain_id, scored[0][1].id
|
||||
# Score by word-boundary keyword overlap against title + description combined
|
||||
combined = f"{title} {description or ''}".lower()
|
||||
scored: list[tuple[int, CapabilityCatalog]] = []
|
||||
for entry in entries:
|
||||
keywords = [kw for kw in (entry.keywords or []) if len(kw) >= 3]
|
||||
score = sum(
|
||||
1 for kw in keywords
|
||||
if re.search(r'\b' + re.escape(kw.lower()) + r'\b', combined)
|
||||
)
|
||||
scored.append((score, entry))
|
||||
scored.sort(key=lambda x: -x[0])
|
||||
|
||||
return None, None
|
||||
best_score, best = scored[0]
|
||||
if best_score == 0:
|
||||
return None, None, (
|
||||
f"no keyword overlap for type '{capability_type}' among "
|
||||
f"{len(entries)} entries — broadcast"
|
||||
)
|
||||
if len(scored) >= 2 and scored[1][0] == best_score:
|
||||
return None, None, (
|
||||
f"ambiguous routing: '{scored[0][1].title}' and '{scored[1][1].title}' "
|
||||
f"both scored {best_score} — broadcast"
|
||||
)
|
||||
|
||||
return best.domain_id, best.id, (
|
||||
f"matched '{best.title}' (score={best_score}, "
|
||||
f"keywords matched from: {title!r})"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1,15 +1,21 @@
|
||||
import asyncio
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy import case, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from api.database import get_session
|
||||
from api.doi_engine import compute_fingerprint, evaluate as _doi_evaluate
|
||||
from api.models.doi_cache import DOICache
|
||||
from api.models.domain import Domain
|
||||
from api.models.managed_repo import ManagedRepo
|
||||
from api.models.repo_goal import RepoGoal
|
||||
from api.models.tpsc import TPSCSnapshot
|
||||
from api.models.task import Task
|
||||
from api.models.workstream import Workstream
|
||||
from api.schemas.doi import DoICriterion, DoIReport, DoISummaryEntry
|
||||
from api.schemas.managed_repo import (
|
||||
DispatchTask,
|
||||
DispatchWorkstream,
|
||||
@@ -68,6 +74,220 @@ async def register_repo(
|
||||
return repo
|
||||
|
||||
|
||||
@router.get("/doi/summary", response_model=list[DoISummaryEntry])
|
||||
async def doi_summary(session: AsyncSession = Depends(get_session)) -> list[DoISummaryEntry]:
|
||||
"""Return DoI tier for all active repos, worst tier first.
|
||||
|
||||
Results are cached in doi_cache. A repo is only re-evaluated when its
|
||||
fingerprint changes (repo record updated, new TPSC snapshot, goal change,
|
||||
or a key file mtime changes on disk).
|
||||
"""
|
||||
repos_result = await session.execute(
|
||||
select(ManagedRepo).where(ManagedRepo.status == "active").order_by(ManagedRepo.name)
|
||||
)
|
||||
repos = list(repos_result.scalars().all())
|
||||
repo_ids = [r.id for r in repos]
|
||||
id_to_slug = {r.id: r.slug for r in repos}
|
||||
|
||||
# ── Bulk DB queries for fingerprint inputs ────────────────────────────────
|
||||
domains_result = await session.execute(select(Domain))
|
||||
domain_obj_map = {d.id: d for d in domains_result.scalars().all()}
|
||||
domain_map = {d.id: d.slug for d in domain_obj_map.values()}
|
||||
domain_status = {d.slug: d.status for d in domain_obj_map.values()}
|
||||
|
||||
# Latest TPSC snapshot timestamp per repo (for fingerprint + C9 count)
|
||||
tpsc_result = await session.execute(
|
||||
select(TPSCSnapshot.repo_id,
|
||||
func.count().label("cnt"),
|
||||
func.max(TPSCSnapshot.snapshot_at).label("latest"))
|
||||
.where(TPSCSnapshot.repo_id.in_(repo_ids))
|
||||
.group_by(TPSCSnapshot.repo_id)
|
||||
)
|
||||
tpsc_by_id = {row.repo_id: row for row in tpsc_result}
|
||||
tpsc_snap_counts = {id_to_slug[rid]: row.cnt for rid, row in tpsc_by_id.items() if rid in id_to_slug}
|
||||
tpsc_snap_latest = {id_to_slug[rid]: str(row.latest) for rid, row in tpsc_by_id.items() if rid in id_to_slug}
|
||||
|
||||
# Latest goal updated_at + active count per repo (for fingerprint + C10)
|
||||
goals_result = await session.execute(
|
||||
select(RepoGoal.repo_id,
|
||||
func.count().label("total"),
|
||||
func.sum(case((RepoGoal.status == "active", 1), else_=0)).label("active_cnt"),
|
||||
func.max(RepoGoal.updated_at).label("latest"))
|
||||
.where(RepoGoal.repo_id.in_(repo_ids))
|
||||
.group_by(RepoGoal.repo_id)
|
||||
)
|
||||
goals_by_id = {row.repo_id: row for row in goals_result}
|
||||
active_goal_counts = {id_to_slug[rid]: int(row.active_cnt or 0) for rid, row in goals_by_id.items() if rid in id_to_slug}
|
||||
goals_latest = {id_to_slug[rid]: str(row.latest) for rid, row in goals_by_id.items() if rid in id_to_slug}
|
||||
|
||||
# Load existing cache rows
|
||||
cache_result = await session.execute(
|
||||
select(DOICache).where(DOICache.repo_id.in_(repo_ids))
|
||||
)
|
||||
cache_by_repo_id = {row.repo_id: row for row in cache_result.scalars().all()}
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
prefetch = {
|
||||
"domain_status": domain_status,
|
||||
"tpsc_snap_counts": tpsc_snap_counts,
|
||||
"active_goal_counts": active_goal_counts,
|
||||
}
|
||||
|
||||
async def _get_or_refresh(repo: ManagedRepo) -> DoISummaryEntry:
|
||||
slug = repo.slug
|
||||
repo_dict = {
|
||||
"slug": slug,
|
||||
"domain_slug": domain_map.get(repo.domain_id),
|
||||
"local_path": repo.local_path,
|
||||
"remote_url": repo.remote_url,
|
||||
"host_paths": repo.host_paths or {},
|
||||
"last_sbom_at": str(repo.last_sbom_at) if repo.last_sbom_at else None,
|
||||
"updated_at": str(repo.updated_at) if repo.updated_at else "",
|
||||
}
|
||||
fp = compute_fingerprint(
|
||||
repo_dict,
|
||||
tpsc_snap_latest.get(slug),
|
||||
goals_latest.get(slug),
|
||||
)
|
||||
|
||||
cached = cache_by_repo_id.get(repo.id)
|
||||
if cached and cached.fingerprint == fp:
|
||||
# Cache hit — return stored result
|
||||
return DoISummaryEntry(
|
||||
repo_slug=slug,
|
||||
domain_slug=domain_map.get(repo.domain_id),
|
||||
tier=cached.tier,
|
||||
core_pass=cached.core_pass,
|
||||
standard_pass=cached.standard_pass,
|
||||
full_pass=cached.full_pass,
|
||||
checked_at=cached.checked_at.isoformat(),
|
||||
)
|
||||
|
||||
# Cache miss — evaluate and store
|
||||
report = await _doi_evaluate(repo_dict, skip_consistency=True, prefetch=prefetch)
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
if cached:
|
||||
cached.tier = report.tier
|
||||
cached.core_pass = report.core_pass
|
||||
cached.standard_pass = report.standard_pass
|
||||
cached.full_pass = report.full_pass
|
||||
cached.criteria = [{"id": c.id, "label": c.label, "tier": c.tier,
|
||||
"status": c.status, "detail": c.detail}
|
||||
for c in report.criteria]
|
||||
cached.fingerprint = fp
|
||||
cached.checked_at = now
|
||||
cached.updated_at = now
|
||||
else:
|
||||
session.add(DOICache(
|
||||
repo_id=repo.id,
|
||||
tier=report.tier,
|
||||
core_pass=report.core_pass,
|
||||
standard_pass=report.standard_pass,
|
||||
full_pass=report.full_pass,
|
||||
criteria=[{"id": c.id, "label": c.label, "tier": c.tier,
|
||||
"status": c.status, "detail": c.detail}
|
||||
for c in report.criteria],
|
||||
fingerprint=fp,
|
||||
checked_at=now,
|
||||
updated_at=now,
|
||||
))
|
||||
|
||||
return DoISummaryEntry(
|
||||
repo_slug=slug,
|
||||
domain_slug=domain_map.get(repo.domain_id),
|
||||
tier=report.tier,
|
||||
core_pass=report.core_pass,
|
||||
standard_pass=report.standard_pass,
|
||||
full_pass=report.full_pass,
|
||||
checked_at=now.isoformat(),
|
||||
)
|
||||
|
||||
entries: list[DoISummaryEntry] = list(await asyncio.gather(*[_get_or_refresh(r) for r in repos]))
|
||||
await session.commit()
|
||||
|
||||
tier_order = {"none": 0, "core": 1, "standard": 2, "full": 3}
|
||||
entries.sort(key=lambda e: tier_order.get(e.tier, 0))
|
||||
return entries
|
||||
|
||||
|
||||
@router.get("/{slug}/doi", response_model=DoIReport)
|
||||
async def get_repo_doi(
|
||||
slug: str,
|
||||
force_refresh: bool = False,
|
||||
session: AsyncSession = Depends(get_session),
|
||||
) -> DoIReport:
|
||||
"""Evaluate the 14 DoI criteria for a single repo (full check including C7/C13).
|
||||
|
||||
Results are cached by fingerprint. Pass ?force_refresh=true to bypass the cache.
|
||||
"""
|
||||
repo = await _get_repo_by_slug(slug, session)
|
||||
domain_result = await session.execute(select(Domain).where(Domain.id == repo.domain_id))
|
||||
domain_obj = domain_result.scalar_one_or_none()
|
||||
|
||||
# Fingerprint inputs for this single repo
|
||||
tpsc_row = (await session.execute(
|
||||
select(func.count().label("cnt"), func.max(TPSCSnapshot.snapshot_at).label("latest"))
|
||||
.where(TPSCSnapshot.repo_id == repo.id)
|
||||
)).one()
|
||||
goal_row = (await session.execute(
|
||||
select(func.max(RepoGoal.updated_at).label("latest"))
|
||||
.where(RepoGoal.repo_id == repo.id)
|
||||
)).one()
|
||||
|
||||
repo_dict = {
|
||||
"slug": repo.slug,
|
||||
"domain_slug": domain_obj.slug if domain_obj else None,
|
||||
"local_path": repo.local_path,
|
||||
"remote_url": repo.remote_url,
|
||||
"host_paths": repo.host_paths or {},
|
||||
"last_sbom_at": str(repo.last_sbom_at) if repo.last_sbom_at else None,
|
||||
"updated_at": str(repo.updated_at) if repo.updated_at else "",
|
||||
}
|
||||
fp = compute_fingerprint(repo_dict, str(tpsc_row.latest) if tpsc_row.latest else None,
|
||||
str(goal_row.latest) if goal_row.latest else None)
|
||||
|
||||
# Check cache (unless force_refresh)
|
||||
cached = (await session.execute(
|
||||
select(DOICache).where(DOICache.repo_id == repo.id)
|
||||
)).scalar_one_or_none()
|
||||
|
||||
if not force_refresh and cached and cached.fingerprint == fp and cached.criteria:
|
||||
return DoIReport(
|
||||
repo_slug=slug,
|
||||
tier=cached.tier,
|
||||
core_pass=cached.core_pass,
|
||||
standard_pass=cached.standard_pass,
|
||||
full_pass=cached.full_pass,
|
||||
checked_at=cached.checked_at.isoformat(),
|
||||
criteria=[DoICriterion(**c) for c in cached.criteria],
|
||||
)
|
||||
|
||||
# Full evaluation (includes C7/C13 consistency subprocesses)
|
||||
report = await _doi_evaluate(repo_dict)
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
criteria_json = [{"id": c.id, "label": c.label, "tier": c.tier,
|
||||
"status": c.status, "detail": c.detail} for c in report.criteria]
|
||||
if cached:
|
||||
cached.tier = report.tier; cached.core_pass = report.core_pass
|
||||
cached.standard_pass = report.standard_pass; cached.full_pass = report.full_pass
|
||||
cached.criteria = criteria_json; cached.fingerprint = fp
|
||||
cached.checked_at = now; cached.updated_at = now
|
||||
else:
|
||||
session.add(DOICache(repo_id=repo.id, tier=report.tier,
|
||||
core_pass=report.core_pass, standard_pass=report.standard_pass,
|
||||
full_pass=report.full_pass, criteria=criteria_json,
|
||||
fingerprint=fp, checked_at=now, updated_at=now))
|
||||
await session.commit()
|
||||
|
||||
return DoIReport(
|
||||
repo_slug=report.repo_slug, tier=report.tier,
|
||||
core_pass=report.core_pass, standard_pass=report.standard_pass,
|
||||
full_pass=report.full_pass, checked_at=report.checked_at,
|
||||
criteria=[DoICriterion(id=c.id, label=c.label, tier=c.tier,
|
||||
status=c.status, detail=c.detail) for c in report.criteria],
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{slug}/", response_model=RepoRead)
|
||||
async def get_repo(
|
||||
slug: str,
|
||||
|
||||
240
state-hub/api/routers/tpsc.py
Normal file
240
state-hub/api/routers/tpsc.py
Normal file
@@ -0,0 +1,240 @@
|
||||
from datetime import datetime, timezone
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy import select, func
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from api.database import get_session
|
||||
from api.models.managed_repo import ManagedRepo
|
||||
from api.models.tpsc import TPSCCatalog, TPSCSnapshot, TPSCEntry
|
||||
from api.schemas.tpsc import (
|
||||
TPSCCatalogCreate, TPSCCatalogRead,
|
||||
TPSCEntryRead, TPSCIngestRequest, TPSCSnapshotRead,
|
||||
TPSCGDPRReport, TPSCGDPRWarning, GDPR_WARNING_LEVELS,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/tpsc", tags=["tpsc"])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Catalog
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@router.get("/catalog/", response_model=list[TPSCCatalogRead])
|
||||
async def list_catalog(
|
||||
gdpr_maturity: str | None = None,
|
||||
category: str | None = None,
|
||||
pricing_model: str | None = None,
|
||||
session: AsyncSession = Depends(get_session),
|
||||
):
|
||||
q = select(TPSCCatalog).where(TPSCCatalog.status != "deprecated")
|
||||
if gdpr_maturity:
|
||||
q = q.where(TPSCCatalog.gdpr_maturity == gdpr_maturity)
|
||||
if category:
|
||||
q = q.where(TPSCCatalog.category == category)
|
||||
if pricing_model:
|
||||
q = q.where(TPSCCatalog.pricing_model == pricing_model)
|
||||
q = q.order_by(TPSCCatalog.name)
|
||||
rows = (await session.execute(q)).scalars().all()
|
||||
return rows
|
||||
|
||||
|
||||
@router.get("/catalog/{slug}", response_model=TPSCCatalogRead)
|
||||
async def get_catalog_entry(slug: str, session: AsyncSession = Depends(get_session)):
|
||||
row = (await session.execute(select(TPSCCatalog).where(TPSCCatalog.slug == slug))).scalar_one_or_none()
|
||||
if not row:
|
||||
raise HTTPException(404, f"Service '{slug}' not found in catalog")
|
||||
return row
|
||||
|
||||
|
||||
@router.post("/catalog/", response_model=TPSCCatalogRead, status_code=201)
|
||||
async def register_service(body: TPSCCatalogCreate, session: AsyncSession = Depends(get_session)):
|
||||
"""Register a new service or upsert an existing one by slug."""
|
||||
existing = (await session.execute(select(TPSCCatalog).where(TPSCCatalog.slug == body.slug))).scalar_one_or_none()
|
||||
if existing:
|
||||
for k, v in body.model_dump(exclude_unset=True).items():
|
||||
setattr(existing, k, v)
|
||||
existing.updated_at = datetime.now(tz=timezone.utc)
|
||||
await session.commit()
|
||||
await session.refresh(existing)
|
||||
return existing
|
||||
entry = TPSCCatalog(**body.model_dump())
|
||||
session.add(entry)
|
||||
await session.commit()
|
||||
await session.refresh(entry)
|
||||
return entry
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ingest
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@router.post("/ingest/", response_model=TPSCSnapshotRead, status_code=201)
|
||||
async def ingest_tpsc(body: TPSCIngestRequest, session: AsyncSession = Depends(get_session)):
|
||||
"""Accept a tpsc.yaml snapshot for a repo."""
|
||||
# Resolve repo_id
|
||||
repo = (await session.execute(select(ManagedRepo).where(ManagedRepo.slug == body.repo_slug))).scalar_one_or_none()
|
||||
repo_id = repo.id if repo else None
|
||||
|
||||
# Build catalog lookup by slug
|
||||
slugs = {e.service_slug for e in body.entries}
|
||||
catalog_rows = (await session.execute(select(TPSCCatalog).where(TPSCCatalog.slug.in_(slugs)))).scalars().all()
|
||||
catalog_map = {r.slug: r for r in catalog_rows}
|
||||
|
||||
snapshot = TPSCSnapshot(
|
||||
repo_id=repo_id,
|
||||
source_file=body.source_file,
|
||||
entry_count=len(body.entries),
|
||||
)
|
||||
session.add(snapshot)
|
||||
await session.flush()
|
||||
|
||||
entries_with_cats = []
|
||||
for e in body.entries:
|
||||
cat = catalog_map.get(e.service_slug)
|
||||
entry = TPSCEntry(
|
||||
snapshot_id=snapshot.id,
|
||||
catalog_id=cat.id if cat else None,
|
||||
service_slug=e.service_slug,
|
||||
purpose=e.purpose,
|
||||
auth_type=e.auth_type,
|
||||
endpoint_override=e.endpoint_override,
|
||||
notes=e.notes,
|
||||
)
|
||||
session.add(entry)
|
||||
entries_with_cats.append((entry, cat))
|
||||
|
||||
await session.flush() # assign UUIDs to all entries
|
||||
await session.commit()
|
||||
await session.refresh(snapshot)
|
||||
|
||||
entry_reads = [
|
||||
TPSCEntryRead(
|
||||
id=entry.id,
|
||||
snapshot_id=snapshot.id,
|
||||
catalog_id=cat.id if cat else None,
|
||||
service_slug=entry.service_slug,
|
||||
purpose=entry.purpose,
|
||||
auth_type=entry.auth_type,
|
||||
endpoint_override=entry.endpoint_override,
|
||||
notes=entry.notes,
|
||||
gdpr_maturity=cat.gdpr_maturity if cat else None,
|
||||
gdpr_warning=(cat.gdpr_maturity in GDPR_WARNING_LEVELS) if cat else True,
|
||||
pricing_model=cat.pricing_model if cat else None,
|
||||
)
|
||||
for entry, cat in entries_with_cats
|
||||
]
|
||||
|
||||
return TPSCSnapshotRead(
|
||||
id=snapshot.id,
|
||||
repo_id=snapshot.repo_id,
|
||||
snapshot_at=snapshot.snapshot_at,
|
||||
source_file=snapshot.source_file,
|
||||
entry_count=snapshot.entry_count,
|
||||
entries=entry_reads,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Snapshots
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@router.get("/snapshots/", response_model=list[TPSCSnapshotRead])
|
||||
async def list_snapshots(
|
||||
repo_slug: str | None = None,
|
||||
session: AsyncSession = Depends(get_session),
|
||||
):
|
||||
q = select(TPSCSnapshot).options(
|
||||
selectinload(TPSCSnapshot.entries).selectinload(TPSCEntry.catalog_entry)
|
||||
)
|
||||
if repo_slug:
|
||||
repo = (await session.execute(select(ManagedRepo).where(ManagedRepo.slug == repo_slug))).scalar_one_or_none()
|
||||
if not repo:
|
||||
raise HTTPException(404, f"Repo '{repo_slug}' not found")
|
||||
q = q.where(TPSCSnapshot.repo_id == repo.id)
|
||||
q = q.order_by(TPSCSnapshot.snapshot_at.desc())
|
||||
rows = (await session.execute(q)).scalars().all()
|
||||
|
||||
result = []
|
||||
for snap in rows:
|
||||
entry_reads = []
|
||||
for e in snap.entries:
|
||||
cat = e.catalog_entry
|
||||
entry_reads.append(TPSCEntryRead(
|
||||
id=e.id,
|
||||
snapshot_id=e.snapshot_id,
|
||||
catalog_id=e.catalog_id,
|
||||
service_slug=e.service_slug,
|
||||
purpose=e.purpose,
|
||||
auth_type=e.auth_type,
|
||||
endpoint_override=e.endpoint_override,
|
||||
notes=e.notes,
|
||||
gdpr_maturity=cat.gdpr_maturity if cat else None,
|
||||
gdpr_warning=(cat.gdpr_maturity in GDPR_WARNING_LEVELS) if cat else True,
|
||||
pricing_model=cat.pricing_model if cat else None,
|
||||
))
|
||||
result.append(TPSCSnapshotRead(
|
||||
id=snap.id,
|
||||
repo_id=snap.repo_id,
|
||||
snapshot_at=snap.snapshot_at,
|
||||
source_file=snap.source_file,
|
||||
entry_count=snap.entry_count,
|
||||
entries=entry_reads,
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GDPR report
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@router.get("/report/gdpr", response_model=TPSCGDPRReport)
|
||||
async def gdpr_report(session: AsyncSession = Depends(get_session)):
|
||||
"""Aggregated GDPR warnings across all latest repo snapshots."""
|
||||
# Latest snapshot per repo
|
||||
latest_sub = (
|
||||
select(TPSCSnapshot.repo_id, func.max(TPSCSnapshot.snapshot_at).label("max_at"))
|
||||
.group_by(TPSCSnapshot.repo_id)
|
||||
.subquery()
|
||||
)
|
||||
latest_snaps = (await session.execute(
|
||||
select(TPSCSnapshot)
|
||||
.join(latest_sub, (TPSCSnapshot.repo_id == latest_sub.c.repo_id) & (TPSCSnapshot.snapshot_at == latest_sub.c.max_at))
|
||||
.options(selectinload(TPSCSnapshot.entries).selectinload(TPSCEntry.catalog_entry))
|
||||
)).scalars().all()
|
||||
|
||||
# Repo slug lookup
|
||||
all_repos = (await session.execute(select(ManagedRepo))).scalars().all()
|
||||
repo_map = {r.id: r.slug for r in all_repos}
|
||||
|
||||
all_services = (await session.execute(select(TPSCCatalog))).scalars().all()
|
||||
by_maturity: dict[str, int] = {}
|
||||
for s in all_services:
|
||||
by_maturity[s.gdpr_maturity] = by_maturity.get(s.gdpr_maturity, 0) + 1
|
||||
|
||||
warnings = []
|
||||
seen = set()
|
||||
for snap in latest_snaps:
|
||||
repo_slug = repo_map.get(snap.repo_id) if snap.repo_id else None
|
||||
for entry in snap.entries:
|
||||
cat = entry.catalog_entry
|
||||
maturity = cat.gdpr_maturity if cat else "unknown"
|
||||
if maturity in GDPR_WARNING_LEVELS:
|
||||
key = (repo_slug, entry.service_slug)
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
warnings.append(TPSCGDPRWarning(
|
||||
repo_slug=repo_slug,
|
||||
service_slug=entry.service_slug,
|
||||
gdpr_maturity=maturity,
|
||||
purpose=entry.purpose,
|
||||
pricing_model=cat.pricing_model if cat else None,
|
||||
))
|
||||
|
||||
return TPSCGDPRReport(
|
||||
generated_at=datetime.now(tz=timezone.utc),
|
||||
total_services=len(all_services),
|
||||
warning_count=len(warnings),
|
||||
warnings=warnings,
|
||||
by_maturity=by_maturity,
|
||||
)
|
||||
@@ -55,6 +55,13 @@ class CapabilityRequestStatusPatch(BaseModel):
|
||||
note: str | None = None
|
||||
|
||||
|
||||
class CapabilityRequestPatch(BaseModel):
|
||||
catalog_entry_id: uuid.UUID | None = None
|
||||
priority: str | None = None
|
||||
blocking_task_id: uuid.UUID | None = None
|
||||
fulfilling_workstream_id: uuid.UUID | None = None
|
||||
|
||||
|
||||
class CapabilityRequestRead(BaseModel):
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
@@ -73,6 +80,7 @@ class CapabilityRequestRead(BaseModel):
|
||||
blocking_task_id: uuid.UUID | None = None
|
||||
catalog_entry_id: uuid.UUID | None = None
|
||||
resolution_note: str | None = None
|
||||
routing_note: str | None = None
|
||||
accepted_at: datetime | None = None
|
||||
completed_at: datetime | None = None
|
||||
created_at: datetime
|
||||
|
||||
29
state-hub/api/schemas/doi.py
Normal file
29
state-hub/api/schemas/doi.py
Normal file
@@ -0,0 +1,29 @@
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class DoICriterion(BaseModel):
|
||||
id: str
|
||||
label: str
|
||||
tier: str
|
||||
status: str # pass | fail | warn | skip
|
||||
detail: str = ""
|
||||
|
||||
|
||||
class DoIReport(BaseModel):
|
||||
repo_slug: str
|
||||
tier: str # none | core | standard | full
|
||||
core_pass: bool
|
||||
standard_pass: bool
|
||||
full_pass: bool
|
||||
criteria: list[DoICriterion] = []
|
||||
checked_at: str
|
||||
|
||||
|
||||
class DoISummaryEntry(BaseModel):
|
||||
repo_slug: str
|
||||
domain_slug: str | None
|
||||
tier: str
|
||||
core_pass: bool
|
||||
standard_pass: bool
|
||||
full_pass: bool
|
||||
checked_at: str
|
||||
115
state-hub/api/schemas/tpsc.py
Normal file
115
state-hub/api/schemas/tpsc.py
Normal file
@@ -0,0 +1,115 @@
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from typing import Literal
|
||||
from pydantic import BaseModel, computed_field
|
||||
|
||||
# GDPR maturity scale (CNIL/IAPP CMMI-aligned, adapted for third-party assessment)
|
||||
GDPRMaturity = Literal["unknown", "non_compliant", "initial", "developing", "defined", "managed", "certified"]
|
||||
|
||||
# Services at these levels trigger a GDPR warning
|
||||
GDPR_WARNING_LEVELS = {"unknown", "non_compliant", "initial"}
|
||||
|
||||
PricingModel = Literal["free", "paid", "freemium", "usage_based", "unknown"]
|
||||
AuthType = Literal["api_key", "oauth", "cli", "none", "unknown"]
|
||||
|
||||
|
||||
class TPSCCatalogCreate(BaseModel):
|
||||
slug: str
|
||||
name: str
|
||||
provider: str | None = None
|
||||
category: str | None = None
|
||||
website_url: str | None = None
|
||||
pricing_model: PricingModel = "unknown"
|
||||
gdpr_maturity: GDPRMaturity = "unknown"
|
||||
gdpr_notes: str | None = None
|
||||
dpa_available: bool = False
|
||||
tos_url: str | None = None
|
||||
privacy_policy_url: str | None = None
|
||||
data_processing_regions: list[str] | None = None
|
||||
data_retention_notes: str | None = None
|
||||
status: str = "active"
|
||||
|
||||
|
||||
class TPSCCatalogRead(BaseModel):
|
||||
model_config = {"from_attributes": True}
|
||||
|
||||
id: uuid.UUID
|
||||
slug: str
|
||||
name: str
|
||||
provider: str | None
|
||||
category: str | None
|
||||
website_url: str | None
|
||||
pricing_model: str
|
||||
gdpr_maturity: str
|
||||
gdpr_notes: str | None
|
||||
dpa_available: bool
|
||||
tos_url: str | None
|
||||
privacy_policy_url: str | None
|
||||
data_processing_regions: list[str] | None
|
||||
data_retention_notes: str | None
|
||||
status: str
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
@computed_field
|
||||
@property
|
||||
def gdpr_warning(self) -> bool:
|
||||
return self.gdpr_maturity in GDPR_WARNING_LEVELS
|
||||
|
||||
|
||||
class TPSCEntryCreate(BaseModel):
|
||||
service_slug: str
|
||||
purpose: str | None = None
|
||||
auth_type: str | None = None
|
||||
endpoint_override: str | None = None
|
||||
notes: str | None = None
|
||||
|
||||
|
||||
class TPSCEntryRead(BaseModel):
|
||||
model_config = {"from_attributes": True}
|
||||
|
||||
id: uuid.UUID
|
||||
snapshot_id: uuid.UUID
|
||||
catalog_id: uuid.UUID | None
|
||||
service_slug: str
|
||||
purpose: str | None
|
||||
auth_type: str | None
|
||||
endpoint_override: str | None
|
||||
notes: str | None
|
||||
# Denormalised from catalog for convenience
|
||||
gdpr_maturity: str | None = None
|
||||
gdpr_warning: bool = False
|
||||
pricing_model: str | None = None
|
||||
|
||||
|
||||
class TPSCIngestRequest(BaseModel):
|
||||
repo_slug: str
|
||||
source_file: str = "tpsc.yaml"
|
||||
entries: list[TPSCEntryCreate]
|
||||
|
||||
|
||||
class TPSCSnapshotRead(BaseModel):
|
||||
model_config = {"from_attributes": True}
|
||||
|
||||
id: uuid.UUID
|
||||
repo_id: uuid.UUID | None
|
||||
snapshot_at: datetime
|
||||
source_file: str | None
|
||||
entry_count: int
|
||||
entries: list[TPSCEntryRead] = []
|
||||
|
||||
|
||||
class TPSCGDPRWarning(BaseModel):
|
||||
repo_slug: str | None
|
||||
service_slug: str
|
||||
gdpr_maturity: str
|
||||
purpose: str | None
|
||||
pricing_model: str | None
|
||||
|
||||
|
||||
class TPSCGDPRReport(BaseModel):
|
||||
generated_at: datetime
|
||||
total_services: int
|
||||
warning_count: int
|
||||
warnings: list[TPSCGDPRWarning]
|
||||
by_maturity: dict[str, int]
|
||||
@@ -25,6 +25,7 @@ export default {
|
||||
{ name: "Goals", path: "/goals" },
|
||||
{ name: "Inbox", path: "/inbox" },
|
||||
{ name: "Progress", path: "/progress" },
|
||||
{ name: "Services (TPSC)", path: "/tpsc" },
|
||||
{ name: "Todo", path: "/todo" },
|
||||
// ── Sections (alphabetical) ───────────────────────────────────────────────
|
||||
{
|
||||
@@ -32,6 +33,7 @@ export default {
|
||||
collapsible: true,
|
||||
open: false,
|
||||
pages: [
|
||||
{ name: "Repository DoI", path: "/policy/repo-doi" },
|
||||
{ name: "Workstream DoD", path: "/policy/workstream-dod" },
|
||||
],
|
||||
},
|
||||
@@ -84,10 +86,13 @@ export default {
|
||||
{ name: "Ralph Workplan", path: "/docs/ralph-workplan" },
|
||||
{ name: "Reference & Context Help", path: "/docs/reference" },
|
||||
{ name: "Repo Integration", path: "/docs/repo-integration" },
|
||||
{ name: "State Hub", path: "/docs/state-hub" },
|
||||
{ name: "Repos", path: "/docs/repos" },
|
||||
{ name: "SBOM", path: "/docs/sbom" },
|
||||
{ name: "SCOPE.md", path: "/docs/scope" },
|
||||
{ name: "Tasks", path: "/docs/tasks" },
|
||||
{ name: "TPSC", path: "/docs/tpsc" },
|
||||
{ name: "TPSC — GDPR Maturity", path: "/docs/gdpr-maturity" },
|
||||
{ name: "Technical Debt", path: "/docs/debt" },
|
||||
{ name: "Todo", path: "/docs/todo" },
|
||||
{ name: "Workstream Health", path: "/docs/workstream-health-index" },
|
||||
|
||||
176
state-hub/dashboard/src/docs/gdpr-maturity.md
Normal file
176
state-hub/dashboard/src/docs/gdpr-maturity.md
Normal file
@@ -0,0 +1,176 @@
|
||||
---
|
||||
title: GDPR Maturity Model
|
||||
---
|
||||
|
||||
# GDPR Maturity Model
|
||||
|
||||
The Custodian TPSC uses a seven-level maturity scale to rate the GDPR
|
||||
compliance posture of third-party services. It is adapted from the
|
||||
**CNIL / IAPP CMMI Privacy Maturity Model** for the specific purpose of
|
||||
assessing external service providers rather than internal programmes.
|
||||
|
||||
---
|
||||
|
||||
## Foundations
|
||||
|
||||
### Source frameworks
|
||||
|
||||
| Framework | Authority | Levels |
|
||||
|---|---|---|
|
||||
| [CNIL Data Protection Maturity Model](https://iapp.org/news/b/cnil-publishes-data-protection-management-maturity-model) | French data protection authority (CNIL) | 5 (Initial → Optimized) |
|
||||
| [IAPP Privacy Program Maturity Model](https://iapp.org/news/a/achieving-privacy-excellence-understanding-the-privacy-maturity-model) | International Association of Privacy Professionals | 5 (Ad Hoc → Optimized) |
|
||||
| [ISO/IEC 27701:2025](https://www.iso.org/standard/27701) | ISO / IEC | Implementation tiers |
|
||||
| [CMMI (Capability Maturity Model Integration)](https://cmmiinstitute.com) | CMMI Institute | 5 (Initial → Optimizing) |
|
||||
|
||||
Both CNIL and IAPP align on the same semantic progression: **Initial →
|
||||
Repeatable → Defined → Managed → Optimized**, directly mapping to CMMI levels
|
||||
1–5. The Custodian scale extends this with two pre-maturity states
|
||||
(`unknown`, `non_compliant`) that have no CMMI equivalent but are essential
|
||||
when assessing third parties with no published compliance posture.
|
||||
|
||||
---
|
||||
|
||||
## The Scale
|
||||
|
||||
### Level 0 — `unknown`
|
||||
|
||||
> No information is available about the service's GDPR compliance posture.
|
||||
|
||||
- No privacy policy, no ToS that addresses data processing, or the service has not been assessed yet.
|
||||
- **Dashboard:** 🔴 Warning
|
||||
- **Implication:** Cannot be used for any processing of personal data in a regulated environment. Treat as non-compliant until assessed.
|
||||
- **CMMI equivalent:** None (pre-maturity)
|
||||
|
||||
---
|
||||
|
||||
### Level 1 — `non_compliant`
|
||||
|
||||
> The service has known GDPR compliance deficiencies with no indication of remediation.
|
||||
|
||||
- May include: data transfers to non-adequate third countries without safeguards, no privacy policy, confirmed regulatory findings, or explicit statements that GDPR does not apply.
|
||||
- **Dashboard:** 🔴 Warning
|
||||
- **Implication:** Must not be used for personal data processing in any EU/EEA context. Legal risk exists even for development use if real personal data is involved.
|
||||
- **CMMI equivalent:** Below Level 1
|
||||
|
||||
---
|
||||
|
||||
### Level 2 — `initial`
|
||||
|
||||
> A basic privacy policy exists. Compliance approach is ad hoc and reactive.
|
||||
|
||||
- Some documentation exists but it is incomplete or generic. No formal Data Processing Agreement (DPA) is offered. Data processing practices may not be clearly defined.
|
||||
- **Dashboard:** 🟠 Warning
|
||||
- **Implication:** Suitable for development and prototyping with synthetic or anonymised data only. Not suitable for production processing of personal data without additional controls.
|
||||
- **CMMI equivalent:** Level 1 — Initial
|
||||
|
||||
---
|
||||
|
||||
### Level 3 — `developing`
|
||||
|
||||
> DPA is available. Standard Contractual Clauses (SCCs) or equivalent transfer mechanisms are in place for EU→non-EU transfers.
|
||||
|
||||
- The service acknowledges GDPR obligations. A DPA can be signed (even if not mandatory for all tiers). Data processing regions are documented. Some controls exist but the compliance programme is not fully formalised.
|
||||
- **Dashboard:** 🟡 Caution
|
||||
- **Implication:** Acceptable for routine processing of personal data when a DPA has been signed. Verify transfer mechanisms and data residency before use with sensitive categories. Suitable for most B2B use cases.
|
||||
- **CMMI equivalent:** Level 2 — Managed / Repeatable
|
||||
|
||||
---
|
||||
|
||||
### Level 4 — `defined`
|
||||
|
||||
> Formal DPA, documented SCCs or adequacy decision, clearly published data retention policy, and defined data processing practices.
|
||||
|
||||
- The compliance programme is documented and consistent. Data subjects' rights are implemented. Sub-processor lists are published. Processing purposes are limited and documented.
|
||||
- **Dashboard:** 🟢 Compliant
|
||||
- **Implication:** Suitable for general production use including personal data. Appropriate for most corporate and SME environments. Review sub-processor list for any domain-specific restrictions.
|
||||
- **CMMI equivalent:** Level 3 — Defined
|
||||
|
||||
---
|
||||
|
||||
### Level 5 — `managed`
|
||||
|
||||
> Independently audited compliance. Quantified metrics, continuous improvement processes, and regular attestation published.
|
||||
|
||||
- Third-party audits (e.g. SOC 2 Type II with privacy controls, penetration testing reports, annual compliance attestations) are available. Privacy metrics are tracked and acted upon. Incident response procedures are tested.
|
||||
- **Dashboard:** 🟢 Compliant
|
||||
- **Implication:** Suitable for processing sensitive categories of personal data (Art. 9 GDPR). Suitable for regulated industries (healthcare, finance) subject to additional sectoral review.
|
||||
- **CMMI equivalent:** Level 4 — Quantitatively Managed
|
||||
|
||||
---
|
||||
|
||||
### Level 6 — `certified`
|
||||
|
||||
> Formal independent certification against a recognised privacy standard.
|
||||
|
||||
- Examples: ISO/IEC 27701 (Privacy Information Management System), BSI C5 (for cloud services), SOC 2 Type II with GDPR-specific controls. Certification is current and scope covers the relevant services.
|
||||
- **Dashboard:** 🟢 Compliant
|
||||
- **Implication:** Highest available assurance. Suitable for processing of sensitive personal data at scale, public-sector use, and regulated environments with strict vendor requirements (DSGVO-compliant procurement, NHS DSPT, etc.).
|
||||
- **CMMI equivalent:** Level 5 — Optimizing
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Level | Code | Label | GDPR Warning | CMMI | Suitable for personal data? |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `unknown` | Unknown | ✅ Yes | — | ❌ No |
|
||||
| 1 | `non_compliant` | Non-Compliant | ✅ Yes | — | ❌ No |
|
||||
| 2 | `initial` | Initial | ✅ Yes | L1 | ⚠ Synthetic/anonymised only |
|
||||
| 3 | `developing` | Developing | — | L2 | ✅ With signed DPA |
|
||||
| 4 | `defined` | Defined | — | L3 | ✅ General use |
|
||||
| 5 | `managed` | Managed | — | L4 | ✅ Sensitive categories |
|
||||
| 6 | `certified` | Certified | — | L5 | ✅ Regulated environments |
|
||||
|
||||
**GDPR warnings** are raised by the dashboard and `get_gdpr_report()` for any service at level 0–2 (`unknown`, `non_compliant`, `initial`).
|
||||
|
||||
---
|
||||
|
||||
## Key GDPR Concepts Referenced
|
||||
|
||||
**DPA (Data Processing Agreement)** — A contract required by GDPR Art. 28 when a controller engages a processor. The DPA defines the subject-matter, duration, nature and purpose of processing, and the obligations of both parties.
|
||||
|
||||
**SCCs (Standard Contractual Clauses)** — Commission-approved contract clauses enabling lawful transfer of personal data from the EU/EEA to third countries without an adequacy decision. Updated SCCs published June 2021 (implementing decisions 2021/914 and 2021/915).
|
||||
|
||||
**Adequacy Decision** — A European Commission finding that a third country provides an essentially equivalent level of data protection (e.g. UK GDPR, Japan, Canada PIPEDA). Transfers to adequate countries do not require additional safeguards.
|
||||
|
||||
**BCRs (Binding Corporate Rules)** — Internal rules allowing multinationals to transfer personal data within their group across borders. Approved by a lead supervisory authority.
|
||||
|
||||
**Sensitive Categories (Art. 9)** — Health, biometric, genetic, racial/ethnic origin, political opinions, religious beliefs, trade union membership, sexual orientation. Require explicit consent or other specific legal basis.
|
||||
|
||||
---
|
||||
|
||||
## Assigning a Maturity Level
|
||||
|
||||
When adding a new service to `canon/tpsc/`, follow this decision process:
|
||||
|
||||
```
|
||||
Is a privacy policy published?
|
||||
No → unknown or non_compliant
|
||||
|
||||
Is a DPA available (even on request)?
|
||||
No → initial
|
||||
Yes → developing (minimum)
|
||||
|
||||
Are SCCs or adequacy mechanisms documented?
|
||||
No → developing
|
||||
Yes, and retention policy published → defined
|
||||
|
||||
Are independent audit reports published (SOC 2 Type II, etc.)?
|
||||
Yes → managed
|
||||
|
||||
Is an ISO 27701 or equivalent certification current?
|
||||
Yes → certified
|
||||
```
|
||||
|
||||
When uncertain between two levels, assign the **lower** level. Err on the side of caution.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- CNIL: [Le modèle de maturité de la protection des données](https://www.cnil.fr/fr/le-modele-de-maturite-de-la-protection-des-donnees)
|
||||
- IAPP: [Achieving privacy excellence — understanding the privacy maturity model](https://iapp.org/news/a/achieving-privacy-excellence-understanding-the-privacy-maturity-model)
|
||||
- ISO/IEC 27701:2025: [Privacy information management — Requirements and guidelines](https://www.iso.org/standard/27701)
|
||||
- European Commission SCCs (2021): [Implementing Decision 2021/914](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32021D0914)
|
||||
- EDPB Guidelines on SCCs: [Guidelines 04/2021](https://edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-042021-standard-contractual-clauses_en)
|
||||
- CMMI Institute: [CMMI Model Overview](https://cmmiinstitute.com/cmmi)
|
||||
269
state-hub/dashboard/src/docs/state-hub.md
Normal file
269
state-hub/dashboard/src/docs/state-hub.md
Normal file
@@ -0,0 +1,269 @@
|
||||
---
|
||||
title: State Hub — Reference
|
||||
---
|
||||
|
||||
# The Custodian State Hub
|
||||
|
||||
The State Hub is the central nervous system of the Custodian ecosystem — a
|
||||
local-first service that collects, organises, and surfaces the state of all
|
||||
registered repositories so that humans and agents can orient themselves in
|
||||
seconds rather than minutes.
|
||||
|
||||
---
|
||||
|
||||
## Why it exists
|
||||
|
||||
Software projects accumulate invisible state. Workstreams stall, decisions go
|
||||
unresolved, dependency licences drift, integration gaps widen — and none of
|
||||
this is visible without opening every file in every repository. For a single
|
||||
repo with a single engineer this is manageable. Across six domains, fifteen
|
||||
repositories, and a growing fleet of autonomous agents it becomes untenable.
|
||||
|
||||
The State Hub exists to answer one question instantly:
|
||||
|
||||
> **"What is the actual state of the system right now, and what should happen next?"**
|
||||
|
||||
It does this without requiring any project to change how it works — repositories
|
||||
remain the authority for their own data and the hub is a silent observer that
|
||||
indexes and reflects their state.
|
||||
|
||||
---
|
||||
|
||||
## What it is — and what it is not
|
||||
|
||||
### What it is
|
||||
|
||||
| Role | Description |
|
||||
|---|---|
|
||||
| **Derived Data Store** | All hub data is computed from repo files and records. The hub holds no original information — it can be wiped and rebuilt from scratch at any time without data loss. |
|
||||
| **Read Model** | Provides fast, pre-computed answers to common orientation queries: active workstreams, blocking decisions, DoI compliance tiers, SBOM licence risk, GDPR warnings. |
|
||||
| **Agent Orchestration Layer** | Exposes an MCP server (Model Context Protocol) so that Claude Code sessions in any registered repository can orient themselves, record progress, resolve decisions, and coordinate with each other — all through a uniform tool interface. |
|
||||
| **Cross-Repo Observatory** | The only place where data from all repositories is visible together. Detects dependencies between workstreams, licence risks that span repos, and integration gaps that no single repo can see about itself. |
|
||||
|
||||
### What it is not
|
||||
|
||||
| Not | Why |
|
||||
|---|---|
|
||||
| **Source of truth** | Repos are the authority. The hub reflects; it does not own. |
|
||||
| **A project management tool** | Workplans live in repos as Markdown files. The hub indexes them, not the reverse. |
|
||||
| **A deployment or CI/CD system** | The hub tracks *intent and state*, not execution pipelines. |
|
||||
| **A multi-user SaaS** | Local-first, single-operator. Designed for sovereignty, not scale. |
|
||||
|
||||
---
|
||||
|
||||
## The Derived Data Store principle
|
||||
|
||||
The hub is a concrete implementation of what Martin Kleppmann calls a
|
||||
**Derived Data Store** (*Designing Data-Intensive Applications*, Ch. 3 & 11):
|
||||
a system whose entire dataset is fully derivable from upstream sources and
|
||||
which therefore makes no authority claims of its own.
|
||||
|
||||
This is formalised in **ADR-003** (Materialized Derived State with Fingerprint
|
||||
Invalidation). The practical consequence:
|
||||
|
||||
1. **Repositories are the primary artefacts.** SCOPE.md, workplan files,
|
||||
`uv.lock`, `tpsc.yaml`, `CLAUDE.md` — these are canonical. The hub reads
|
||||
them; it never writes them.
|
||||
2. **The hub is a cache.** Every table that holds repo-derived data carries a
|
||||
`fingerprint` column. When the fingerprint of the current source state
|
||||
matches the stored fingerprint, the cached value is returned instantly.
|
||||
When it differs, the value is recomputed and the cache is updated.
|
||||
3. **The rebuild guarantee.** Any hub database can be dropped and recreated
|
||||
from scratch by re-running migrations and re-ingesting all registered repos.
|
||||
No information is lost because no information originates here.
|
||||
4. **Force-refresh on demand.** Every derived endpoint accepts
|
||||
`?force_refresh=true` to bypass the cache — useful for debugging, post-ingest
|
||||
verification, and validating fingerprint coverage.
|
||||
|
||||
### What gets derived and how
|
||||
|
||||
| Source | Derived data | Mechanism |
|
||||
|---|---|---|
|
||||
| `uv.lock`, `package-lock.json`, etc. | SBOM entries + licence risk | `make ingest-sbom REPO=` |
|
||||
| `tpsc.yaml` | Third-party service declarations + GDPR warnings | `make ingest-tpsc REPO=` |
|
||||
| `SCOPE.md` capability blocks | Capability catalog | `make ingest-capabilities REPO=` |
|
||||
| `workplans/*.md` | Workstream + task status | `make fix-consistency REPO=` |
|
||||
| Repo files + DB records | DoI compliance tier | Fingerprint cache, auto-refreshed on read |
|
||||
|
||||
---
|
||||
|
||||
## The Repository Orchestrator
|
||||
|
||||
Beyond indexing, the hub acts as an **orchestrator** for the repository
|
||||
ecosystem — the coordinator that no individual repo can be.
|
||||
|
||||
### Agent session protocol
|
||||
|
||||
Every Claude Code session in a registered repository follows the same ritual:
|
||||
|
||||
1. **Orient** — call `get_domain_summary("<slug>")` to load active workstreams,
|
||||
pending tasks, blocking decisions, and suggested next steps for this domain.
|
||||
2. **Check inbox** — call `get_messages(to_agent="<name>", unread_only=True)`
|
||||
to receive coordination messages from other agents or from prior sessions.
|
||||
3. **Work** — use MCP tools to record progress, resolve decisions, update task
|
||||
status, and send messages to other agents.
|
||||
4. **Close** — call `add_progress_event()` before ending the session so the
|
||||
next session can see what happened.
|
||||
|
||||
This protocol means that any agent — human-directed or autonomous — arrives
|
||||
in a fresh session with full context about the current state of its domain
|
||||
in under a second.
|
||||
|
||||
### Cross-domain coordination
|
||||
|
||||
Because all agents share the same hub, they can coordinate without direct
|
||||
communication:
|
||||
|
||||
- An agent working on `railiance` can send a message to `custodian` via
|
||||
`send_message()`. The custodian agent reads it in its next session inbox.
|
||||
- A capability request (`request_capability()`) routes to the domain that
|
||||
advertises the relevant capability in its `SCOPE.md`. The fulfilling agent
|
||||
accepts it, does the work, and marks it complete — unblocking the requesting
|
||||
workstream automatically.
|
||||
- Workstream dependencies (`create_dependency()`) let the hub surface "what
|
||||
is blocking what" across repos that have never directly communicated.
|
||||
|
||||
### Kaizen agents
|
||||
|
||||
The hub hosts a library of specialised agent personas (`agents/agent-*.md`)
|
||||
that any session can load via `get_kaizen_agent("<name>")`. These provide
|
||||
expert instruction sets for specific tasks — test-driven development,
|
||||
refactoring, infrastructure review — without baking domain knowledge into
|
||||
every repo's CLAUDE.md.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Repositories │
|
||||
│ workplans/*.md SCOPE.md CLAUDE.md uv.lock tpsc.yaml │
|
||||
└─────────┬───────────────────────────────────────────────────┘
|
||||
│ ingest scripts (make ingest-sbom / fix-consistency / …)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ PostgreSQL Database │
|
||||
│ domains managed_repos workstreams tasks decisions │
|
||||
│ sbom_entries tpsc_entries doi_cache capability_catalog │
|
||||
│ progress_events agent_messages repo_goals … │
|
||||
└─────────┬───────────────────────────────────────────────────┘
|
||||
│
|
||||
┌────┴──────────────┐
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────┐ ┌─────────────────────────────────────────┐
|
||||
│ FastAPI │ │ MCP Server (SSE :8001) │
|
||||
│ REST API │ │ 40+ tools — orient, record, coordinate │
|
||||
│ (:8000) │ │ Registered at user scope in ~/.claude │
|
||||
└─────────────┘ └─────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Observable Framework Dashboard (:3000) │
|
||||
│ Overview Repositories Workstreams Decisions SBOM │
|
||||
│ TPSC Contributions Goals Capabilities … │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Technology choices
|
||||
|
||||
| Component | Technology | Reason |
|
||||
|---|---|---|
|
||||
| Database | PostgreSQL (Docker) | Relational integrity, JSON columns, full async support |
|
||||
| API | FastAPI + SQLAlchemy (async) | Type-safe, auto-documented, fast |
|
||||
| MCP server | FastMCP over SSE | Persistent connection; no Claude Code restart on tool update |
|
||||
| Dashboard | Observable Framework | Reactive, static-deployable, no JS build step for data loaders |
|
||||
| Agent interface | Model Context Protocol | Standard Claude Code integration; tools available in all sessions |
|
||||
|
||||
---
|
||||
|
||||
## Data model overview
|
||||
|
||||
The hub's schema is organised in concentric layers:
|
||||
|
||||
**Governance (slow-changing):**
|
||||
`domains` → `managed_repos` → `repo_goals` → `domain_goals`
|
||||
|
||||
**Work tracking (active):**
|
||||
`workstreams` → `tasks` → `workstream_dependencies`
|
||||
|
||||
**Decision log (append-only):**
|
||||
`decisions` · `progress_events` · `agent_messages`
|
||||
|
||||
**Derived snapshots (cached, rebuildable):**
|
||||
`sbom_snapshots` → `sbom_entries`
|
||||
`tpsc_snapshots` → `tpsc_entries`
|
||||
`capability_catalog` · `capability_requests`
|
||||
`doi_cache`
|
||||
|
||||
**Governance policies (editables):**
|
||||
`policies/` (flat Markdown files, served via `/policy/<name>`)
|
||||
|
||||
---
|
||||
|
||||
## Running the hub
|
||||
|
||||
```bash
|
||||
cd ~/the-custodian/state-hub
|
||||
|
||||
make db # Start PostgreSQL (Docker)
|
||||
make migrate # Alembic upgrade head
|
||||
make seed # Insert 6 canonical domain topics
|
||||
make api # FastAPI on :8000
|
||||
make mcp-http # MCP SSE server on :8001
|
||||
make dashboard # Observable preview on :3000
|
||||
```
|
||||
|
||||
Verify the MCP server is registered:
|
||||
```bash
|
||||
python3 -c "import json,os; d=json.load(open(os.path.expanduser('~/.claude.json'))); print(list(d.get('mcpServers',{}).keys()))"
|
||||
# → ['state-hub']
|
||||
```
|
||||
|
||||
Re-register if needed:
|
||||
```bash
|
||||
claude mcp add-json -s user state-hub '{"type":"sse","url":"http://127.0.0.1:8001/sse"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Design principles
|
||||
|
||||
These principles are not aspirational — they are constraints that every hub
|
||||
feature must satisfy before being accepted.
|
||||
|
||||
**Local-first, no vendor lock-in.**
|
||||
The hub runs on a developer laptop. It requires only Docker (for Postgres)
|
||||
and Python. It works offline. No cloud dependency, no SaaS subscription.
|
||||
|
||||
**Sovereignty by default.**
|
||||
No data leaves the machine unless explicitly exported. The hub never calls
|
||||
external APIs on its own.
|
||||
|
||||
**The rebuild guarantee (ADR-001, ADR-003).**
|
||||
Dropping the database and re-running `make migrate && make seed` followed by
|
||||
re-ingesting all repos must restore the full operational state. Any feature
|
||||
that breaks this guarantee is rejected.
|
||||
|
||||
**Agents are co-creators, not authorities.**
|
||||
The hub coordinates agent work but does not permit agents to make irreversible
|
||||
decisions unilaterally. Financial, legal, and external-publication actions are
|
||||
hard-blocked (see `canon/constitution/`).
|
||||
|
||||
**Append-only episodic memory.**
|
||||
Progress events are never deleted or edited. Decisions are resolved, not
|
||||
overwritten. This ensures that any session can reconstruct what happened and
|
||||
why, even years later.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- [ADR-001](/docs/repo-integration) — Workplans as Repository Artefacts
|
||||
- [ADR-003](https://github.com) — Materialized Derived State (see `canon/architecture/adr-003-materialized-derived-state.md`)
|
||||
- [Connecting to the Hub](/docs/connecting)
|
||||
- [Repo Integration](/docs/repo-integration)
|
||||
- [Repository DoI](/policy/repo-doi) — Definition of Integrated
|
||||
- [TPSC](/docs/tpsc) — Third-Party Services Catalog
|
||||
- [SBOM](/docs/sbom)
|
||||
136
state-hub/dashboard/src/docs/tpsc.md
Normal file
136
state-hub/dashboard/src/docs/tpsc.md
Normal file
@@ -0,0 +1,136 @@
|
||||
---
|
||||
title: Third-Party Services Catalog (TPSC)
|
||||
---
|
||||
|
||||
# Third-Party Services Catalog (TPSC)
|
||||
|
||||
The TPSC tracks external service dependencies (APIs, SaaS, CLIs) across all
|
||||
registered repos — complementing the SBOM for package dependencies.
|
||||
|
||||
---
|
||||
|
||||
## Why TPSC?
|
||||
|
||||
Package lockfiles capture Python/JS/Rust dependencies but miss the external
|
||||
HTTP services your code calls. These carry compliance, cost, and privacy
|
||||
implications that are invisible to standard SBOM tooling.
|
||||
|
||||
TPSC provides:
|
||||
- A registry of which repos use which external services
|
||||
- GDPR compliance maturity ratings per service
|
||||
- Pricing model tracking (paid/usage-based costs)
|
||||
- Data processing region and retention information
|
||||
- GDPR warnings for services not suitable in regulated environments
|
||||
|
||||
---
|
||||
|
||||
## Primary Data Locations
|
||||
|
||||
Following ADR-001 (workplans as repo artefacts), TPSC data lives in two places:
|
||||
|
||||
| Location | Purpose |
|
||||
|---|---|
|
||||
| `<repo>/tpsc.yaml` | Declares which services the repo uses |
|
||||
| `the-custodian/canon/tpsc/<slug>.yaml` | Canonical service metadata (ToS, GDPR, pricing) |
|
||||
|
||||
The state-hub is a collector — it can be rebuilt from scratch by re-ingesting
|
||||
all `tpsc.yaml` files and re-seeding the catalog from canon files.
|
||||
|
||||
---
|
||||
|
||||
## tpsc.yaml Format
|
||||
|
||||
```yaml
|
||||
# tpsc.yaml — Third-Party Services Catalog declarations
|
||||
# Ingest: cd state-hub && make ingest-tpsc REPO=<slug>
|
||||
|
||||
services:
|
||||
- slug: openai-api # Must match a slug in canon/tpsc/
|
||||
purpose: LLM inference via OpenAI-compatible API
|
||||
auth: api_key # api_key | oauth | cli | none | unknown
|
||||
|
||||
- slug: stripe
|
||||
purpose: Payment processing
|
||||
auth: api_key
|
||||
endpoint: https://api.stripe.com # Optional override if non-standard
|
||||
notes: Only used in production tier
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Canon Service File Format
|
||||
|
||||
```yaml
|
||||
# canon/tpsc/openai-api.yaml
|
||||
slug: openai-api
|
||||
name: OpenAI API
|
||||
provider: OpenAI, Inc.
|
||||
category: llm_inference # llm_inference | storage | payments | search | etc.
|
||||
website_url: https://openai.com
|
||||
pricing_model: usage_based # free | paid | freemium | usage_based | unknown
|
||||
gdpr_maturity: developing # See scale below
|
||||
gdpr_notes: >
|
||||
DPA available. SCCs for EU→US transfer. 30-day retention for safety.
|
||||
dpa_available: true
|
||||
tos_url: https://openai.com/policies/terms-of-use
|
||||
privacy_policy_url: https://openai.com/policies/privacy-policy
|
||||
data_processing_regions:
|
||||
- us
|
||||
data_retention_notes: >
|
||||
30 days default; zero-retention available on eligible endpoints.
|
||||
status: active
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GDPR Maturity Scale
|
||||
|
||||
Based on the **CNIL / IAPP CMMI Privacy Maturity Model**, adapted for
|
||||
third-party service assessment:
|
||||
|
||||
| Level | Name | Description | Dashboard |
|
||||
|---|---|---|---|
|
||||
| 0 | `unknown` | No information about GDPR stance | 🔴 Warning |
|
||||
| 1 | `non_compliant` | Known GDPR issues, no remediation | 🔴 Warning |
|
||||
| 2 | `initial` | Basic privacy policy only, ad hoc approach | 🟠 Warning |
|
||||
| 3 | `developing` | DPA available, some controls, SCCs provided | 🟡 |
|
||||
| 4 | `defined` | Formal DPA, SCCs documented, clear retention policy | 🟢 |
|
||||
| 5 | `managed` | Independently audited, metrics tracked | 🟢 |
|
||||
| 6 | `certified` | ISO 27701 / SOC2 privacy certified | 🟢 |
|
||||
|
||||
Services at levels 0–2 (**Warning**) may limit use in GDPR-regulated or
|
||||
corporate environments. At minimum, `developing` is needed for routine
|
||||
processing of personal data with an API provider.
|
||||
|
||||
Reference: [CNIL GDPR maturity model](https://iapp.org/news/b/cnil-publishes-data-protection-management-maturity-model), [IAPP Privacy Maturity Model](https://iapp.org/news/a/achieving-privacy-excellence-understanding-the-privacy-maturity-model)
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Service
|
||||
|
||||
1. Create `the-custodian/canon/tpsc/<slug>.yaml` following the format above
|
||||
2. Seed it into the state-hub: `cd state-hub && make api` then POST to `/tpsc/catalog/`
|
||||
(or use the MCP tool: `register_service(slug=..., ...)`)
|
||||
3. Add it to your repo's `tpsc.yaml`
|
||||
4. Ingest: `make ingest-tpsc REPO=<slug>`
|
||||
|
||||
---
|
||||
|
||||
## MCP Tools
|
||||
|
||||
| Tool | Purpose |
|
||||
|---|---|
|
||||
| `register_service(slug, ...)` | Add/update a service in the catalog |
|
||||
| `list_services(gdpr_maturity?, category?, pricing_model?)` | Browse catalog |
|
||||
| `ingest_tpsc_tool(repo_slug)` | Parse tpsc.yaml and ingest snapshot |
|
||||
| `get_gdpr_report()` | GDPR warning summary across all repos |
|
||||
|
||||
---
|
||||
|
||||
## Makefile Targets
|
||||
|
||||
```bash
|
||||
make ingest-tpsc REPO=llm-connect # Ingest single repo
|
||||
make ingest-tpsc-all # Ingest all repos
|
||||
make ingest-tpsc REPO=llm-connect DRY_RUN=1 # Preview only
|
||||
```
|
||||
90
state-hub/dashboard/src/policy/repo-doi.md
Normal file
90
state-hub/dashboard/src/policy/repo-doi.md
Normal file
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: Repository Definition of Integrated (DoI)
|
||||
---
|
||||
|
||||
```js
|
||||
import {API} from "../components/config.js";
|
||||
```
|
||||
|
||||
```js
|
||||
import {marked} from "npm:marked";
|
||||
|
||||
const _resp = await fetch(`${API}/policy/repo-doi`);
|
||||
if (!_resp.ok) throw new Error(`Failed to load policy: ${_resp.status}`);
|
||||
const _policy = await _resp.json();
|
||||
```
|
||||
|
||||
```js
|
||||
let _content = _policy.content;
|
||||
let _editing = false;
|
||||
|
||||
const _root = display(html`<div></div>`);
|
||||
|
||||
async function _save(text) {
|
||||
const r = await fetch(`${API}/policy/repo-doi`, {
|
||||
method: "PUT",
|
||||
headers: {"Content-Type": "application/json"},
|
||||
body: JSON.stringify({content: text}),
|
||||
});
|
||||
if (!r.ok) throw new Error(`Save failed: ${r.status}`);
|
||||
_content = text;
|
||||
}
|
||||
|
||||
function _toolbar(...nodes) {
|
||||
return html`<div style="display:flex;gap:0.5rem;margin-bottom:1rem">${nodes}</div>`;
|
||||
}
|
||||
|
||||
function _btn(label, primary = false) {
|
||||
return html`<button style="
|
||||
padding:0.35rem 0.9rem;border-radius:4px;cursor:pointer;font-size:13px;
|
||||
background:${primary ? "#1e293b" : "#f1f5f9"};
|
||||
color:${primary ? "#f8fafc" : "#1e293b"};
|
||||
border:1px solid ${primary ? "#1e293b" : "#cbd5e1"};
|
||||
">${label}</button>`;
|
||||
}
|
||||
|
||||
function _render() {
|
||||
_root.innerHTML = "";
|
||||
|
||||
if (_editing) {
|
||||
const area = html`<textarea style="
|
||||
width:100%;box-sizing:border-box;height:520px;
|
||||
font-family:ui-monospace,monospace;font-size:13px;line-height:1.6;
|
||||
padding:0.75rem;border:1px solid #cbd5e1;border-radius:4px;
|
||||
background:#f8fafc;color:#1e293b;resize:vertical;
|
||||
">${_content}</textarea>`;
|
||||
|
||||
const saveBtn = _btn("Save", true);
|
||||
const cancelBtn = _btn("Cancel");
|
||||
|
||||
saveBtn.onclick = async () => {
|
||||
saveBtn.disabled = true;
|
||||
saveBtn.textContent = "Saving…";
|
||||
try {
|
||||
await _save(area.value);
|
||||
_editing = false;
|
||||
_render();
|
||||
} catch (e) {
|
||||
saveBtn.disabled = false;
|
||||
saveBtn.textContent = "Save";
|
||||
alert(e.message);
|
||||
}
|
||||
};
|
||||
|
||||
cancelBtn.onclick = () => { _editing = false; _render(); };
|
||||
|
||||
_root.append(_toolbar(saveBtn, cancelBtn), area);
|
||||
|
||||
} else {
|
||||
const editBtn = _btn("Edit");
|
||||
editBtn.onclick = () => { _editing = true; _render(); };
|
||||
|
||||
const body = html`<div style="max-width:720px;line-height:1.7;"></div>`;
|
||||
body.innerHTML = marked.parse(_content);
|
||||
|
||||
_root.append(_toolbar(editBtn), body);
|
||||
}
|
||||
}
|
||||
|
||||
_render();
|
||||
```
|
||||
@@ -41,6 +41,19 @@ convention used in the Custodian State Hub.
|
||||
|
||||
---
|
||||
|
||||
## Architecture & Design
|
||||
|
||||
| Topic | What it covers |
|
||||
|-------|---------------|
|
||||
| [State Hub](/docs/state-hub) | Why/how/what — Derived Data Store principle, orchestrator role, architecture diagram, design principles |
|
||||
| [TPSC](/docs/tpsc) | Third-Party Services Catalog — tpsc.yaml format, ingest, MCP tools |
|
||||
| [TPSC — GDPR Maturity](/docs/gdpr-maturity) | 7-level CNIL/IAPP scale, per-level guidance, key GDPR concepts |
|
||||
| [SCOPE.md](/docs/scope) | The scope file format and how to write one |
|
||||
| [Capabilities](/docs/capabilities) | Capability catalog, request routing, MCP tools |
|
||||
| [Goals](/docs/goals) | Domain and repo goals, priority ordering |
|
||||
|
||||
---
|
||||
|
||||
## Meta
|
||||
|
||||
| Topic | What it covers |
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
title: Repos
|
||||
title: Repositories
|
||||
---
|
||||
|
||||
```js
|
||||
@@ -7,6 +7,7 @@ import {API} from "./components/config.js";
|
||||
```
|
||||
|
||||
```js
|
||||
// Fast data — page renders as soon as this resolves (~200ms)
|
||||
let _repos = [], _domains = [], _sbom = [], _eps = [], _tds = [], _workstreams = [];
|
||||
try {
|
||||
[_repos, _domains, _sbom, _eps, _tds, _workstreams] = await Promise.all([
|
||||
@@ -20,6 +21,18 @@ try {
|
||||
} catch {}
|
||||
```
|
||||
|
||||
```js
|
||||
// DoI data — lazy-loaded. Starts as [] so the page renders immediately.
|
||||
// When the fetch resolves (~6s), doiData updates and DoI badges appear.
|
||||
import {Mutable} from "observablehq:stdlib";
|
||||
const doiData = Mutable([]);
|
||||
const doiLoading = Mutable(true);
|
||||
fetch(`${API}/repos/doi/summary`)
|
||||
.then(r => r.ok ? r.json() : [])
|
||||
.catch(() => [])
|
||||
.then(data => { doiData.value = data; doiLoading.value = false; });
|
||||
```
|
||||
|
||||
```js
|
||||
const repos = _repos ?? [];
|
||||
const domains = _domains ?? [];
|
||||
@@ -27,6 +40,14 @@ const sbom = _sbom ?? [];
|
||||
const eps = _eps ?? [];
|
||||
const tds = _tds ?? [];
|
||||
const workstreams = _workstreams ?? [];
|
||||
const doi = doiData; // reactive — updates when lazy fetch completes
|
||||
|
||||
// DoI lookups
|
||||
const doiBySlug = Object.fromEntries(doi.map(d => [d.repo_slug, d]));
|
||||
const DOI_TIER_ORDER = {none: 0, core: 1, standard: 2, full: 3};
|
||||
const DOI_TIER_COLOR = {none: "#ef4444", core: "#f97316", standard: "#eab308", full: "#22c55e"};
|
||||
const DOI_TIER_BG = {none: "#fef2f2", core: "#fff7ed", standard: "#fefce8", full: "#f0fdf4"};
|
||||
const DOI_TIER_LABEL = {none: "None", core: "Core", standard: "Standard", full: "Full"};
|
||||
|
||||
// Lookups
|
||||
const domainById = Object.fromEntries(domains.map(d => [d.id, d]));
|
||||
@@ -79,11 +100,14 @@ const repoRows = repos
|
||||
? new Date(r.last_sbom_at).toLocaleDateString()
|
||||
: (sbomData?.snapshot_at ? new Date(sbomData.snapshot_at).toLocaleDateString() : null);
|
||||
const integrating = !!integratingBySlug[r.slug];
|
||||
const doiEntry = doiBySlug[r.slug] ?? null;
|
||||
const doiTier = doiEntry?.tier ?? "none";
|
||||
return {
|
||||
_id: r.id,
|
||||
_domSlug: domSlug,
|
||||
_hasSbom: hasSbom,
|
||||
_integrating: integrating,
|
||||
_doiTier: doiTier,
|
||||
repo: r.slug,
|
||||
domain: domName,
|
||||
status: integrating ? "⚙ integrating" : "ready",
|
||||
@@ -96,17 +120,29 @@ const repoRows = repos
|
||||
})
|
||||
.sort((a, b) => a._domSlug.localeCompare(b._domSlug) || a.repo.localeCompare(b.repo));
|
||||
|
||||
const gapCount = repoRows.filter(r => !r._hasSbom).length;
|
||||
const coveredCount = repoRows.filter(r => r._hasSbom).length;
|
||||
const gapCount = repoRows.filter(r => !r._hasSbom).length;
|
||||
const coveredCount = repoRows.filter(r => r._hasSbom).length;
|
||||
const integratingCount = repoRows.filter(r => r._integrating).length;
|
||||
const doiFullCount = repoRows.filter(r => r._doiTier === "full").length;
|
||||
const doiNoneCount = repoRows.filter(r => r._doiTier === "none").length;
|
||||
```
|
||||
|
||||
# Repos
|
||||
# Repositories
|
||||
|
||||
```js
|
||||
import {withDocHelp} from "./components/doc-overlay.js";
|
||||
const _h1 = document.querySelector("#observablehq-main h1");
|
||||
if (_h1) { _h1.style.position = "relative"; withDocHelp(_h1, "/docs/repos"); }
|
||||
display(html`<p style="font-size:0.85rem;color:#6b7280;margin-top:-0.5rem;display:flex;align-items:center;gap:0.75rem;flex-wrap:wrap;">
|
||||
<span>DoI tiers: <strong style="color:#ef4444;">None</strong> →
|
||||
<strong style="color:#f97316;">Core</strong> →
|
||||
<strong style="color:#eab308;">Standard</strong> →
|
||||
<strong style="color:#22c55e;">Full</strong> —
|
||||
<a href="/policy/repo-doi" style="color:#1d4ed8;">Definition of Integrated policy ↗</a></span>
|
||||
${doiLoading ? html`<span style="display:inline-flex;align-items:center;gap:0.35rem;color:#9ca3af;font-size:0.8rem;">
|
||||
<span class="doi-spinner"></span> Loading DoI tiers…
|
||||
</span>` : html`<span style="color:#16a34a;font-size:0.8rem;">✓ DoI tiers loaded</span>`}
|
||||
</p>`);
|
||||
```
|
||||
|
||||
```js
|
||||
@@ -134,6 +170,11 @@ display(html`<div class="kpi-row">
|
||||
<p class="big-num">${gapCount}</p>
|
||||
<small>${gapCount === 0 ? "✓ All repos covered" : `⚠ ${gapCount} repo(s) not ingested`}</small>
|
||||
</div>
|
||||
<div class="card ${doiLoading ? '' : doiNoneCount > 0 ? 'card-warn' : 'card-ok'}">
|
||||
<h3>DoI: Fully Integrated</h3>
|
||||
<p class="big-num">${doiLoading ? html`<span class="doi-spinner" style="width:1.4rem;height:1.4rem;"></span>` : `${doiFullCount} / ${repoRows.length}`}</p>
|
||||
<small>${doiLoading ? "Loading…" : doiNoneCount > 0 ? `⚠ ${doiNoneCount} at tier None` : "✓ All pass Core tier"}</small>
|
||||
</div>
|
||||
</div>`);
|
||||
```
|
||||
|
||||
@@ -147,6 +188,16 @@ function _sbomGap() {
|
||||
return el;
|
||||
}
|
||||
|
||||
function _doiBadge(tier) {
|
||||
if (doiLoading) return html`<span style="color:#d1d5db;font-size:0.72rem;">…</span>`;
|
||||
const color = DOI_TIER_COLOR[tier] || "#9ca3af";
|
||||
const bg = DOI_TIER_BG[tier] || "#f9fafb";
|
||||
const label = DOI_TIER_LABEL[tier] || tier;
|
||||
return html`<span style="background:${bg}; color:${color}; border:1px solid ${color}60;
|
||||
border-radius:4px; padding:1px 7px; font-size:0.72rem; font-weight:700; white-space:nowrap;">
|
||||
DoI: ${label}</span>`;
|
||||
}
|
||||
|
||||
// Group by domain
|
||||
const byDomain = {};
|
||||
for (const r of repoRows) {
|
||||
@@ -162,6 +213,9 @@ if (domainBlocks.length === 0) {
|
||||
${domainBlocks.map(([slug, rows]) => {
|
||||
const dom = domainBySlug[slug];
|
||||
const allCovered = rows.every(r => r._hasSbom);
|
||||
const doiWorst = rows.map(r => DOI_TIER_ORDER[r._doiTier] ?? 0);
|
||||
const doiMin = Math.min(...doiWorst);
|
||||
const doiMinKey = Object.keys(DOI_TIER_ORDER).find(k => DOI_TIER_ORDER[k] === doiMin) ?? "none";
|
||||
const hasEps = (epByDomain[slug] ?? 0) > 0;
|
||||
const hasTds = (tdByDomain[slug] ?? 0) > 0;
|
||||
return html`
|
||||
@@ -178,11 +232,15 @@ if (domainBlocks.length === 0) {
|
||||
${hasTds
|
||||
? html`<span class="chip chip-ok">TDs: ${tdByDomain[slug]}</span>`
|
||||
: html`<span class="chip chip-neutral">TDs: —</span>`}
|
||||
<span style="font-size:0.72rem; color:${DOI_TIER_COLOR[doiMinKey]}; font-weight:600;">
|
||||
DoI min: ${DOI_TIER_LABEL[doiMinKey]}
|
||||
</span>
|
||||
</span>
|
||||
</div>
|
||||
<table class="repo-table">
|
||||
<thead><tr>
|
||||
<th>Repo</th>
|
||||
<th>DoI Tier</th>
|
||||
<th>Status</th>
|
||||
<th>SBOM</th>
|
||||
<th>Packages</th>
|
||||
@@ -191,6 +249,7 @@ if (domainBlocks.length === 0) {
|
||||
<tbody>
|
||||
${rows.map(r => html`<tr class="${r._integrating ? 'row-integrating' : r._hasSbom ? '' : 'row-gap'}">
|
||||
<td class="repo-cell"><code>${r.repo}</code></td>
|
||||
<td>${_doiBadge(r._doiTier)}</td>
|
||||
<td>${r._integrating
|
||||
? html`<span class="chip chip-integrating">⚙ integrating</span>`
|
||||
: html`<span class="chip chip-ok">ready</span>`}</td>
|
||||
@@ -211,26 +270,29 @@ if (domainBlocks.length === 0) {
|
||||
|
||||
```js
|
||||
const domainFilter = Inputs.select(["all", ...new Set(repoRows.map(r => r._domSlug)).values()], {label: "Domain", value: "all"});
|
||||
const doiFilter = Inputs.select(["all", "none", "core", "standard", "full"], {label: "DoI tier", value: "all"});
|
||||
const gapFilter = Inputs.toggle({label: "Gaps only (no SBOM)", value: false});
|
||||
display(html`<div style="display:flex;gap:1rem;flex-wrap:wrap;margin-bottom:1rem">${domainFilter}${gapFilter}</div>`);
|
||||
display(html`<div style="display:flex;gap:1rem;flex-wrap:wrap;margin-bottom:1rem">${domainFilter}${doiFilter}${gapFilter}</div>`);
|
||||
```
|
||||
|
||||
```js
|
||||
const filteredRows = repoRows.filter(r =>
|
||||
(domainFilter.value === "all" || r._domSlug === domainFilter.value) &&
|
||||
(doiFilter.value === "all" || r._doiTier === doiFilter.value) &&
|
||||
(!gapFilter.value || !r._hasSbom)
|
||||
);
|
||||
|
||||
display(Inputs.table(filteredRows.map(r => ({
|
||||
Repo: r.repo,
|
||||
Domain: r.domain,
|
||||
Status: r.status,
|
||||
SBOM: r.sbom,
|
||||
Pkgs: r.pkgs,
|
||||
Repo: r.repo,
|
||||
Domain: r.domain,
|
||||
"DoI Tier": DOI_TIER_LABEL[r._doiTier] ?? r._doiTier,
|
||||
Status: r.status,
|
||||
SBOM: r.sbom,
|
||||
Pkgs: r.pkgs,
|
||||
"EPs (domain)": r.eps || "—",
|
||||
"TDs (domain)": r.tds || "—",
|
||||
Path: r.path,
|
||||
})), {maxWidth: 1100}));
|
||||
Path: r.path,
|
||||
})), {maxWidth: 1200}));
|
||||
```
|
||||
|
||||
## Onboard a New Repo
|
||||
@@ -317,4 +379,12 @@ custodian register-project --domain <slug></pre>
|
||||
.onboard-step strong { font-size: 0.9rem; display: block; margin-bottom: 0.3rem; }
|
||||
.onboard-step pre { background: var(--theme-background); border-radius: 4px; padding: 0.4rem 0.7rem; font-size: 0.8rem; overflow-x: auto; margin: 0 0 0.35rem; }
|
||||
.onboard-note { font-size: 0.82rem; color: var(--theme-foreground-muted, gray); margin: 0; line-height: 1.45; }
|
||||
|
||||
.doi-spinner {
|
||||
display: inline-block; width: 0.9rem; height: 0.9rem;
|
||||
border: 2px solid #e5e7eb; border-top-color: #9ca3af;
|
||||
border-radius: 50%; animation: doi-spin 0.7s linear infinite;
|
||||
vertical-align: middle;
|
||||
}
|
||||
@keyframes doi-spin { to { transform: rotate(360deg); } }
|
||||
</style>
|
||||
|
||||
193
state-hub/dashboard/src/tpsc.md
Normal file
193
state-hub/dashboard/src/tpsc.md
Normal file
@@ -0,0 +1,193 @@
|
||||
---
|
||||
title: Third-Party Services (TPSC)
|
||||
---
|
||||
|
||||
# Third-Party Services Catalog
|
||||
|
||||
```js
|
||||
const API = "http://127.0.0.1:8000";
|
||||
let apiOk = true;
|
||||
|
||||
const catalog = await fetch(`${API}/tpsc/catalog/`)
|
||||
.then(r => r.json())
|
||||
.catch(() => { apiOk = false; return []; });
|
||||
|
||||
const gdprReport = await fetch(`${API}/tpsc/report/gdpr`)
|
||||
.then(r => r.json())
|
||||
.catch(() => ({ warnings: [], by_maturity: {}, total_services: 0, warning_count: 0 }));
|
||||
|
||||
const snapshots = await fetch(`${API}/tpsc/snapshots/`)
|
||||
.then(r => r.json())
|
||||
.catch(() => []);
|
||||
```
|
||||
|
||||
```js
|
||||
// GDPR maturity colour coding (CNIL/IAPP scale)
|
||||
const maturityColor = {
|
||||
unknown: "#ef4444", // red
|
||||
non_compliant: "#dc2626", // deep red
|
||||
initial: "#f97316", // orange
|
||||
developing: "#eab308", // amber
|
||||
defined: "#84cc16", // lime
|
||||
managed: "#22c55e", // green
|
||||
certified: "#16a34a", // deep green
|
||||
};
|
||||
|
||||
const maturityLabel = {
|
||||
unknown: "Unknown",
|
||||
non_compliant: "Non-Compliant",
|
||||
initial: "Initial",
|
||||
developing: "Developing",
|
||||
defined: "Defined",
|
||||
managed: "Managed",
|
||||
certified: "Certified",
|
||||
};
|
||||
|
||||
const WARNING_LEVELS = new Set(["unknown", "non_compliant", "initial"]);
|
||||
```
|
||||
|
||||
```js
|
||||
// KPI summary
|
||||
const warningServices = catalog.filter(s => WARNING_LEVELS.has(s.gdpr_maturity));
|
||||
const paidServices = catalog.filter(s => ["paid", "usage_based"].includes(s.pricing_model));
|
||||
```
|
||||
|
||||
<div style="display:flex; gap:1.5rem; flex-wrap:wrap; margin-bottom:2rem;">
|
||||
<div style="background:#fef2f2; border:1px solid #fca5a5; border-radius:8px; padding:1rem 1.5rem; min-width:160px;">
|
||||
<div style="font-size:2rem; font-weight:700; color:#dc2626;">${gdprReport.warning_count}</div>
|
||||
<div style="font-size:0.85rem; color:#6b7280;">GDPR warnings</div>
|
||||
</div>
|
||||
<div style="background:#f0fdf4; border:1px solid #86efac; border-radius:8px; padding:1rem 1.5rem; min-width:160px;">
|
||||
<div style="font-size:2rem; font-weight:700; color:#16a34a;">${gdprReport.total_services}</div>
|
||||
<div style="font-size:0.85rem; color:#6b7280;">Services in catalog</div>
|
||||
</div>
|
||||
<div style="background:#fffbeb; border:1px solid #fcd34d; border-radius:8px; padding:1rem 1.5rem; min-width:160px;">
|
||||
<div style="font-size:2rem; font-weight:700; color:#b45309;">${paidServices.length}</div>
|
||||
<div style="font-size:0.85rem; color:#6b7280;">Paid / usage-based</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Service Catalog
|
||||
|
||||
```js
|
||||
import {html} from "npm:htl";
|
||||
|
||||
function maturityBadge(m) {
|
||||
const color = maturityColor[m] || "#9ca3af";
|
||||
const label = maturityLabel[m] || m;
|
||||
return html`<span style="background:${color}20; color:${color}; border:1px solid ${color}60;
|
||||
border-radius:4px; padding:2px 8px; font-size:0.78rem; font-weight:600; white-space:nowrap;">${label}</span>`;
|
||||
}
|
||||
|
||||
function pricingBadge(p) {
|
||||
const colors = { paid: "#7c3aed", usage_based: "#7c3aed", freemium: "#0369a1", free: "#166534", unknown: "#6b7280" };
|
||||
const c = colors[p] || "#6b7280";
|
||||
return html`<span style="color:${c}; font-size:0.78rem; font-weight:500;">${p.replace("_", " ")}</span>`;
|
||||
}
|
||||
|
||||
const catalogTable = html`<table style="width:100%; border-collapse:collapse; font-size:0.9rem;">
|
||||
<thead>
|
||||
<tr style="border-bottom:2px solid #e5e7eb;">
|
||||
<th style="text-align:left; padding:8px 12px;">Service</th>
|
||||
<th style="text-align:left; padding:8px 12px;">Provider</th>
|
||||
<th style="text-align:left; padding:8px 12px;">Category</th>
|
||||
<th style="text-align:left; padding:8px 12px;">Pricing</th>
|
||||
<th style="text-align:left; padding:8px 12px;">GDPR Maturity</th>
|
||||
<th style="text-align:left; padding:8px 12px;">DPA</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
${catalog.map(s => html`<tr style="border-bottom:1px solid #f3f4f6; ${WARNING_LEVELS.has(s.gdpr_maturity) ? 'background:#fff7f7;' : ''}">
|
||||
<td style="padding:8px 12px; font-weight:500;">
|
||||
${s.website_url
|
||||
? html`<a href="${s.website_url}" target="_blank" style="color:#1d4ed8;">${s.name}</a>`
|
||||
: s.name}
|
||||
</td>
|
||||
<td style="padding:8px 12px; color:#6b7280;">${s.provider || "—"}</td>
|
||||
<td style="padding:8px 12px; color:#6b7280;">${s.category || "—"}</td>
|
||||
<td style="padding:8px 12px;">${pricingBadge(s.pricing_model)}</td>
|
||||
<td style="padding:8px 12px;">${maturityBadge(s.gdpr_maturity)}</td>
|
||||
<td style="padding:8px 12px;">${s.dpa_available ? "✅" : "❌"}</td>
|
||||
</tr>`)}
|
||||
</tbody>
|
||||
</table>`;
|
||||
display(catalogTable);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## GDPR Warnings
|
||||
|
||||
_Services at **Unknown**, **Non-Compliant**, or **Initial** maturity may limit use in GDPR-regulated or corporate environments._
|
||||
|
||||
```js
|
||||
if (gdprReport.warnings.length === 0) {
|
||||
display(html`<p style="color:#16a34a;">✅ No GDPR warnings across active repos.</p>`);
|
||||
} else {
|
||||
const warningCards = html`<div style="display:flex; flex-direction:column; gap:0.75rem;">
|
||||
${gdprReport.warnings.map(w => {
|
||||
const color = maturityColor[w.gdpr_maturity] || "#ef4444";
|
||||
return html`<div style="border-left:4px solid ${color}; background:${color}10; padding:0.75rem 1rem; border-radius:4px;">
|
||||
<div style="font-weight:600;">${w.service_slug}
|
||||
<span style="font-weight:400; color:#6b7280; margin-left:0.5rem;">in ${w.repo_slug || "unknown repo"}</span>
|
||||
</div>
|
||||
<div style="font-size:0.85rem; margin-top:2px;">
|
||||
${maturityBadge(w.gdpr_maturity)}
|
||||
${w.pricing_model ? html` ${pricingBadge(w.pricing_model)}` : ""}
|
||||
${w.purpose ? html`<span style="color:#6b7280; margin-left:0.5rem;">— ${w.purpose}</span>` : ""}
|
||||
</div>
|
||||
</div>`;
|
||||
})}
|
||||
</div>`;
|
||||
display(warningCards);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Per-Repo Breakdown
|
||||
|
||||
```js
|
||||
// Build: latest snapshot per repo → service list
|
||||
const repoBreakdown = new Map();
|
||||
for (const snap of snapshots) {
|
||||
const repoSlug = snap.repo_id || "unknown";
|
||||
if (!repoBreakdown.has(repoSlug) || snap.snapshot_at > repoBreakdown.get(repoSlug).snapshot_at) {
|
||||
repoBreakdown.set(repoSlug, snap);
|
||||
}
|
||||
}
|
||||
|
||||
// Enrich with catalog data
|
||||
const catalogBySlug = Object.fromEntries(catalog.map(s => [s.slug, s]));
|
||||
|
||||
const repoTable = html`<table style="width:100%; border-collapse:collapse; font-size:0.9rem;">
|
||||
<thead>
|
||||
<tr style="border-bottom:2px solid #e5e7eb;">
|
||||
<th style="text-align:left; padding:8px 12px;">Repo</th>
|
||||
<th style="text-align:left; padding:8px 12px;">Services</th>
|
||||
<th style="text-align:left; padding:8px 12px;">Ingested</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
${[...repoBreakdown.entries()].map(([repoSlug, snap]) => html`<tr style="border-bottom:1px solid #f3f4f6;">
|
||||
<td style="padding:8px 12px; font-weight:500;">${repoSlug}</td>
|
||||
<td style="padding:8px 12px;">
|
||||
${snap.entries.map(e => {
|
||||
const cat = catalogBySlug[e.service_slug];
|
||||
const m = cat?.gdpr_maturity || "unknown";
|
||||
const color = maturityColor[m] || "#9ca3af";
|
||||
return html`<span style="display:inline-flex; align-items:center; gap:4px; margin:2px 4px 2px 0;
|
||||
background:${color}15; border:1px solid ${color}50; border-radius:4px; padding:2px 8px; font-size:0.8rem;">
|
||||
<span style="width:8px; height:8px; border-radius:50%; background:${color}; display:inline-block;"></span>
|
||||
${e.service_slug}
|
||||
</span>`;
|
||||
})}
|
||||
</td>
|
||||
<td style="padding:8px 12px; color:#9ca3af; font-size:0.8rem;">${new Date(snap.snapshot_at).toLocaleDateString()}</td>
|
||||
</tr>`)}
|
||||
</tbody>
|
||||
</table>`;
|
||||
display(repoTable);
|
||||
```
|
||||
@@ -1032,19 +1032,54 @@ def update_repo_path(repo_slug: str, path: str, host: str | None = None) -> str:
|
||||
return json.dumps(repo, indent=2)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared path resolution helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _resolve_repo_path(repo: dict) -> str:
|
||||
"""Return the best local filesystem path for *repo* on this host.
|
||||
|
||||
Resolution order — each candidate is expanded (supports ``~``) and
|
||||
verified to exist before being accepted:
|
||||
|
||||
1. ``host_paths[hostname]`` — host-specific override
|
||||
2. ``local_path`` — default fallback
|
||||
|
||||
Returns the resolved path string, or ``""`` if no valid path is found.
|
||||
"""
|
||||
import socket as _socket
|
||||
hostname = _socket.gethostname()
|
||||
host_paths = repo.get("host_paths") or {}
|
||||
|
||||
candidates = []
|
||||
if host_paths.get(hostname):
|
||||
candidates.append(host_paths[hostname])
|
||||
if repo.get("local_path"):
|
||||
candidates.append(repo["local_path"])
|
||||
|
||||
for raw in candidates:
|
||||
resolved = str(Path(raw).expanduser())
|
||||
if Path(resolved).is_dir():
|
||||
return resolved
|
||||
|
||||
return ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Kaizen Agents
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _kaizen_agents_dir() -> Path:
|
||||
"""Resolve the kaizen-agentic agents/ directory via host_paths → local_path fallback."""
|
||||
import socket as _socket
|
||||
"""Resolve the kaizen-agentic agents/ directory."""
|
||||
repo = _get("/repos/kaizen-agentic")
|
||||
hostname = _socket.gethostname()
|
||||
host_paths = repo.get("host_paths") or {}
|
||||
base = host_paths.get(hostname) or repo.get("local_path") or ""
|
||||
base = _resolve_repo_path(repo)
|
||||
if not base:
|
||||
raise FileNotFoundError("kaizen-agentic path not found for this host. Register it with update_repo_path().")
|
||||
import socket as _socket
|
||||
hostname = _socket.gethostname()
|
||||
raise FileNotFoundError(
|
||||
f"kaizen-agentic path not found on host '{hostname}'. "
|
||||
"Register it with update_repo_path('kaizen-agentic', '/path/to/repo')."
|
||||
)
|
||||
agents_dir = Path(base) / "agents"
|
||||
if not agents_dir.is_dir():
|
||||
raise FileNotFoundError(f"agents/ directory not found at {agents_dir}")
|
||||
@@ -1128,8 +1163,8 @@ def validate_repo_adr(repo_slug: str, domain_slug: str | None = None) -> str:
|
||||
no active state-hub workstreams for the domain lack a backing file (orphan
|
||||
detection — DB-only records are an ADR-001 violation).
|
||||
|
||||
The repo path is resolved from the DB using the current machine's hostname
|
||||
(host_paths[hostname] → local_path fallback). This tool always runs against
|
||||
The repo path is resolved from the DB: host_paths[hostname] is tried first
|
||||
(with existence check), then local_path — both support ~ expansion. This tool always runs against
|
||||
the server's copy of the repo. Remote agents on a different branch should
|
||||
sync first, or run validate_repo_adr.py locally with
|
||||
--api-base http://127.0.0.1:18000.
|
||||
@@ -1146,22 +1181,15 @@ def validate_repo_adr(repo_slug: str, domain_slug: str | None = None) -> str:
|
||||
if isinstance(repo, dict) and repo.get("error"):
|
||||
return f"Repo '{repo_slug}' not found: {repo['error']}"
|
||||
|
||||
hostname = _socket.gethostname()
|
||||
host_paths = repo.get("host_paths") or {}
|
||||
repo_path = host_paths.get(hostname) or repo.get("local_path") or ""
|
||||
|
||||
repo_path = _resolve_repo_path(repo)
|
||||
if not repo_path:
|
||||
hostname = _socket.gethostname()
|
||||
return (
|
||||
f"⚠ No path registered for repo '{repo_slug}' on this host ({hostname}).\n"
|
||||
f"⚠ No accessible path found for repo '{repo_slug}' on host '{hostname}'.\n"
|
||||
f"Register with: update_repo_path('{repo_slug}', '/path/to/repo')\n"
|
||||
f"Remote agents: run validate_repo_adr.py locally with "
|
||||
f"--api-base {API_BASE}"
|
||||
)
|
||||
if not Path(repo_path).is_dir():
|
||||
return (
|
||||
f"⚠ Registered path for '{repo_slug}' on {hostname} does not exist: {repo_path}\n"
|
||||
f"Update with: update_repo_path('{repo_slug}', '/correct/path')"
|
||||
)
|
||||
|
||||
script = Path(__file__).parent.parent / "scripts" / "validate_repo_adr.py"
|
||||
cmd = [sys.executable, str(script), repo_path, "--json",
|
||||
@@ -1238,21 +1266,15 @@ def check_repo_consistency(repo_slug: str, fix: bool = False) -> str:
|
||||
repo = _get(f"/repos/{repo_slug}")
|
||||
if isinstance(repo, dict) and repo.get("error"):
|
||||
return f"Repo '{repo_slug}' not found: {repo['error']}"
|
||||
hostname = _socket.gethostname()
|
||||
host_paths = repo.get("host_paths") or {}
|
||||
repo_path = host_paths.get(hostname) or repo.get("local_path") or ""
|
||||
repo_path = _resolve_repo_path(repo)
|
||||
if not repo_path:
|
||||
hostname = _socket.gethostname()
|
||||
return (
|
||||
f"⚠ No path registered for repo '{repo_slug}' on this host ({hostname}).\n"
|
||||
f"⚠ No accessible path found for repo '{repo_slug}' on host '{hostname}'.\n"
|
||||
f"Register with: update_repo_path('{repo_slug}', '/path/to/repo')\n"
|
||||
f"Remote agents: run consistency_check.py locally with "
|
||||
f"--api-base {API_BASE}"
|
||||
)
|
||||
if not Path(repo_path).is_dir():
|
||||
return (
|
||||
f"⚠ Registered path for '{repo_slug}' on {hostname} does not exist: {repo_path}\n"
|
||||
f"Update with: update_repo_path('{repo_slug}', '/correct/path')"
|
||||
)
|
||||
|
||||
script = Path(__file__).parent.parent / "scripts" / "consistency_check.py"
|
||||
cmd = [sys.executable, str(script), "--repo", repo_slug, "--json",
|
||||
@@ -1448,20 +1470,13 @@ def ingest_sbom_tool(repo_slug: str, lockfile_path: str | None = None) -> str:
|
||||
if isinstance(repo, dict) and repo.get("error"):
|
||||
return f"Repo '{repo_slug}' not found: {repo['error']}"
|
||||
|
||||
hostname = _socket.gethostname()
|
||||
host_paths = repo.get("host_paths") or {}
|
||||
repo_root = host_paths.get(hostname) or repo.get("local_path") or ""
|
||||
|
||||
repo_root = _resolve_repo_path(repo)
|
||||
if not repo_root:
|
||||
hostname = _socket.gethostname()
|
||||
return (
|
||||
f"⚠ No path registered for repo '{repo_slug}' on this host ({hostname}).\n"
|
||||
f"⚠ No accessible path found for repo '{repo_slug}' on host '{hostname}'.\n"
|
||||
f"Register with: update_repo_path('{repo_slug}', '/path/to/repo')"
|
||||
)
|
||||
if not Path(repo_root).is_dir():
|
||||
return (
|
||||
f"⚠ Registered path for '{repo_slug}' on {hostname} does not exist: {repo_root}\n"
|
||||
f"Update with: update_repo_path('{repo_slug}', '/correct/path')"
|
||||
)
|
||||
|
||||
script = Path(__file__).parent.parent / "scripts" / "ingest_sbom.py"
|
||||
cmd = [sys.executable, str(script), "--repo", repo_slug,
|
||||
@@ -1866,6 +1881,46 @@ def update_capability_request_status(
|
||||
return json.dumps(result, indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def patch_capability_request(
|
||||
request_id: str,
|
||||
catalog_entry_id: Optional[str] = None,
|
||||
priority: Optional[str] = None,
|
||||
blocking_task_id: Optional[str] = None,
|
||||
fulfilling_workstream_id: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""Correct mutable metadata on a capability request.
|
||||
|
||||
Correcting catalog_entry_id automatically re-derives the fulfilling domain.
|
||||
Use this when the hub mis-routed a request (wrong catalog entry or domain).
|
||||
Only provided (non-None) fields are updated.
|
||||
|
||||
Args:
|
||||
request_id: UUID of the capability request to patch.
|
||||
catalog_entry_id: Correct catalog entry UUID. Re-derives fulfilling domain.
|
||||
priority: New priority (low/medium/high/critical).
|
||||
blocking_task_id: UUID of the task this request unblocks on completion.
|
||||
fulfilling_workstream_id: UUID of the workstream delivering this capability.
|
||||
|
||||
Returns:
|
||||
Updated capability request dict, or {"error": "..."}.
|
||||
"""
|
||||
body: dict = {}
|
||||
if catalog_entry_id is not None:
|
||||
body["catalog_entry_id"] = catalog_entry_id
|
||||
if priority is not None:
|
||||
body["priority"] = priority
|
||||
if blocking_task_id is not None:
|
||||
body["blocking_task_id"] = blocking_task_id
|
||||
if fulfilling_workstream_id is not None:
|
||||
body["fulfilling_workstream_id"] = fulfilling_workstream_id
|
||||
|
||||
if not body:
|
||||
return {"error": "no fields provided to patch"}
|
||||
|
||||
return json.dumps(_patch(f"/capability-requests/{request_id}", body), indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def list_capability_requests(
|
||||
domain: str | None = None,
|
||||
@@ -1896,6 +1951,174 @@ def get_capability_request(request_id: str) -> str:
|
||||
return json.dumps(_get(f"/capability-requests/{request_id}"), indent=2)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Third-Party Services Catalog (TPSC)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@mcp.tool()
|
||||
def register_service(
|
||||
slug: str,
|
||||
name: str,
|
||||
provider: str | None = None,
|
||||
category: str | None = None,
|
||||
pricing_model: str = "unknown",
|
||||
gdpr_maturity: str = "unknown",
|
||||
gdpr_notes: str | None = None,
|
||||
dpa_available: bool = False,
|
||||
tos_url: str | None = None,
|
||||
privacy_policy_url: str | None = None,
|
||||
data_processing_regions: list[str] | None = None,
|
||||
data_retention_notes: str | None = None,
|
||||
website_url: str | None = None,
|
||||
) -> str:
|
||||
"""Register or update a service in the Third-Party Services Catalog (TPSC).
|
||||
|
||||
GDPR maturity scale (CNIL/IAPP CMMI-aligned):
|
||||
unknown | non_compliant | initial | developing | defined | managed | certified
|
||||
|
||||
Pricing model: free | paid | freemium | usage_based | unknown
|
||||
|
||||
Args:
|
||||
slug: Unique identifier (e.g. 'openai-api', 'stripe')
|
||||
name: Human-readable service name
|
||||
provider: Company/organisation name
|
||||
category: Category (e.g. 'llm_inference', 'storage', 'payments', 'search')
|
||||
pricing_model: free | paid | freemium | usage_based | unknown
|
||||
gdpr_maturity: GDPR compliance maturity level (see scale above)
|
||||
gdpr_notes: Free-text GDPR notes (DPA details, transfer mechanisms, etc.)
|
||||
dpa_available: Whether a Data Processing Agreement is available
|
||||
tos_url: Terms of Service URL
|
||||
privacy_policy_url: Privacy Policy URL
|
||||
data_processing_regions: List of regions where data is processed (e.g. ['us', 'eu'])
|
||||
data_retention_notes: Data retention policy summary
|
||||
website_url: Service website URL
|
||||
"""
|
||||
return json.dumps(_post("/tpsc/catalog", {
|
||||
"slug": slug,
|
||||
"name": name,
|
||||
"provider": provider,
|
||||
"category": category,
|
||||
"website_url": website_url,
|
||||
"pricing_model": pricing_model,
|
||||
"gdpr_maturity": gdpr_maturity,
|
||||
"gdpr_notes": gdpr_notes,
|
||||
"dpa_available": dpa_available,
|
||||
"tos_url": tos_url,
|
||||
"privacy_policy_url": privacy_policy_url,
|
||||
"data_processing_regions": data_processing_regions or [],
|
||||
"data_retention_notes": data_retention_notes,
|
||||
}), indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def list_services(
|
||||
gdpr_maturity: str | None = None,
|
||||
category: str | None = None,
|
||||
pricing_model: str | None = None,
|
||||
) -> str:
|
||||
"""Browse the Third-Party Services Catalog (TPSC).
|
||||
|
||||
Returns services with their GDPR maturity level and gdpr_warning flag
|
||||
(True when maturity is unknown, non_compliant, or initial — may limit
|
||||
use in corporate/GDPR-regulated environments).
|
||||
|
||||
Args:
|
||||
gdpr_maturity: Filter by maturity level (unknown/non_compliant/initial/developing/defined/managed/certified)
|
||||
category: Filter by category (e.g. 'llm_inference', 'storage')
|
||||
pricing_model: Filter by pricing model (free/paid/freemium/usage_based/unknown)
|
||||
"""
|
||||
return json.dumps(_get("/tpsc/catalog", {
|
||||
"gdpr_maturity": gdpr_maturity,
|
||||
"category": category,
|
||||
"pricing_model": pricing_model,
|
||||
}), indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def ingest_tpsc_tool(repo_slug: str) -> str:
|
||||
"""Ingest tpsc.yaml service dependency declarations for a repo.
|
||||
|
||||
Reads <repo_root>/tpsc.yaml, resolves service slugs against the catalog,
|
||||
and creates a new TPSC snapshot. The repo path is resolved the same way
|
||||
as the SBOM ingest tool (host_paths → local_path with existence check).
|
||||
|
||||
Args:
|
||||
repo_slug: Registered repo slug (e.g. 'llm-connect', 'markitect-project')
|
||||
"""
|
||||
import socket as _socket
|
||||
import subprocess
|
||||
|
||||
repo = _get(f"/repos/{repo_slug}")
|
||||
if isinstance(repo, dict) and repo.get("error"):
|
||||
return f"Repo '{repo_slug}' not found: {repo['error']}"
|
||||
|
||||
repo_root = _resolve_repo_path(repo)
|
||||
if not repo_root:
|
||||
hostname = _socket.gethostname()
|
||||
return (
|
||||
f"⚠ No accessible path found for repo '{repo_slug}' on host '{hostname}'.\n"
|
||||
f"Register with: update_repo_path('{repo_slug}', '/path/to/repo')"
|
||||
)
|
||||
|
||||
script = Path(__file__).parent.parent / "scripts" / "ingest_tpsc.py"
|
||||
result = subprocess.run(
|
||||
["uv", "run", "python", str(script), "--repo", repo_slug],
|
||||
capture_output=True, text=True,
|
||||
cwd=str(Path(__file__).parent.parent),
|
||||
)
|
||||
output = result.stdout + result.stderr
|
||||
if result.returncode != 0:
|
||||
return f"ingest_tpsc failed (exit {result.returncode}):\n{output}"
|
||||
return output.strip()
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def get_gdpr_report() -> str:
|
||||
"""Get an aggregated GDPR compliance report across all repos' latest TPSC snapshots.
|
||||
|
||||
Returns a warning summary for services with gdpr_maturity in:
|
||||
unknown | non_compliant | initial
|
||||
|
||||
These may limit usability in GDPR-regulated / corporate environments.
|
||||
Services at 'developing' or above have at least a DPA available.
|
||||
"""
|
||||
return json.dumps(_get("/tpsc/report/gdpr"), indent=2)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Repository Definition of Integrated (DoI)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@mcp.tool()
|
||||
def check_repo_doi(repo_slug: str) -> str:
|
||||
"""Evaluate the 14 DoI criteria for a repo and return a full report.
|
||||
|
||||
Criteria are grouped into three tiers:
|
||||
Core (C1–C4): registered, domain, path, remote URL
|
||||
Standard (C5–C9): SCOPE.md, CLAUDE.md, workplan, SBOM, TPSC
|
||||
Full (C10–C14): repo goal, capabilities, agents, clean consistency, host paths
|
||||
|
||||
Status values: pass | fail | warn | skip
|
||||
|
||||
The 'tier' field shows the highest tier where ALL criteria pass or warn:
|
||||
none | core | standard | full
|
||||
|
||||
Args:
|
||||
repo_slug: Registered repo slug (e.g. 'llm-connect', 'the-custodian')
|
||||
"""
|
||||
return json.dumps(_get(f"/repos/{repo_slug}/doi"), indent=2)
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
def get_doi_summary() -> str:
|
||||
"""Return DoI tier for all active repos, sorted worst-first.
|
||||
|
||||
Useful at session start to spot repos that need integration work.
|
||||
Tiers: none (red) → core → standard → full (green).
|
||||
"""
|
||||
return json.dumps(_get("/repos/doi/summary"), indent=2)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
70
state-hub/migrations/versions/j7e8f9a0b1c2_tpsc.py
Normal file
70
state-hub/migrations/versions/j7e8f9a0b1c2_tpsc.py
Normal file
@@ -0,0 +1,70 @@
|
||||
"""tpsc: third-party services catalog
|
||||
|
||||
Revision ID: j7e8f9a0b1c2
|
||||
Revises: i6d7e8f9a0b1
|
||||
Create Date: 2026-03-19
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects.postgresql import UUID, JSON
|
||||
import uuid
|
||||
|
||||
revision = "j7e8f9a0b1c2"
|
||||
down_revision = "i6d7e8f9a0b1"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"tpsc_catalog",
|
||||
sa.Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4),
|
||||
sa.Column("slug", sa.String(100), nullable=False, unique=True),
|
||||
sa.Column("name", sa.String(200), nullable=False),
|
||||
sa.Column("provider", sa.String(200), nullable=True),
|
||||
sa.Column("category", sa.String(100), nullable=True),
|
||||
sa.Column("website_url", sa.Text, nullable=True),
|
||||
sa.Column("pricing_model", sa.String(20), nullable=False, server_default="unknown"),
|
||||
sa.Column("gdpr_maturity", sa.String(20), nullable=False, server_default="unknown"),
|
||||
sa.Column("gdpr_notes", sa.Text, nullable=True),
|
||||
sa.Column("dpa_available", sa.Boolean, nullable=False, server_default="false"),
|
||||
sa.Column("tos_url", sa.Text, nullable=True),
|
||||
sa.Column("privacy_policy_url", sa.Text, nullable=True),
|
||||
sa.Column("data_processing_regions", JSON, nullable=True),
|
||||
sa.Column("data_retention_notes", sa.Text, nullable=True),
|
||||
sa.Column("status", sa.String(20), nullable=False, server_default="active"),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), onupdate=sa.func.now()),
|
||||
)
|
||||
op.create_index("ix_tpsc_catalog_slug", "tpsc_catalog", ["slug"])
|
||||
op.create_index("ix_tpsc_catalog_gdpr_maturity", "tpsc_catalog", ["gdpr_maturity"])
|
||||
|
||||
op.create_table(
|
||||
"tpsc_snapshots",
|
||||
sa.Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4),
|
||||
sa.Column("repo_id", UUID(as_uuid=True), sa.ForeignKey("managed_repos.id", ondelete="SET NULL"), nullable=True),
|
||||
sa.Column("snapshot_at", sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
sa.Column("source_file", sa.String(200), nullable=True),
|
||||
sa.Column("entry_count", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
op.create_index("ix_tpsc_snapshots_repo_id", "tpsc_snapshots", ["repo_id"])
|
||||
|
||||
op.create_table(
|
||||
"tpsc_entries",
|
||||
sa.Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4),
|
||||
sa.Column("snapshot_id", UUID(as_uuid=True), sa.ForeignKey("tpsc_snapshots.id", ondelete="CASCADE"), nullable=False),
|
||||
sa.Column("catalog_id", UUID(as_uuid=True), sa.ForeignKey("tpsc_catalog.id", ondelete="SET NULL"), nullable=True),
|
||||
sa.Column("service_slug", sa.String(100), nullable=False),
|
||||
sa.Column("purpose", sa.Text, nullable=True),
|
||||
sa.Column("auth_type", sa.String(50), nullable=True),
|
||||
sa.Column("endpoint_override", sa.Text, nullable=True),
|
||||
sa.Column("notes", sa.Text, nullable=True),
|
||||
)
|
||||
op.create_index("ix_tpsc_entries_snapshot_id", "tpsc_entries", ["snapshot_id"])
|
||||
op.create_index("ix_tpsc_entries_service_slug", "tpsc_entries", ["service_slug"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("tpsc_entries")
|
||||
op.drop_table("tpsc_snapshots")
|
||||
op.drop_table("tpsc_catalog")
|
||||
38
state-hub/migrations/versions/k8f9a0b1c2d3_doi_cache.py
Normal file
38
state-hub/migrations/versions/k8f9a0b1c2d3_doi_cache.py
Normal file
@@ -0,0 +1,38 @@
|
||||
"""doi_cache: materialised DoI results with fingerprint-based invalidation
|
||||
|
||||
Revision ID: k8f9a0b1c2d3
|
||||
Revises: j7e8f9a0b1c2
|
||||
Create Date: 2026-03-20
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects.postgresql import UUID, JSON
|
||||
import uuid
|
||||
|
||||
revision = "k8f9a0b1c2d3"
|
||||
down_revision = "j7e8f9a0b1c2"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"doi_cache",
|
||||
sa.Column("id", UUID(as_uuid=True), primary_key=True, default=uuid.uuid4),
|
||||
sa.Column("repo_id", UUID(as_uuid=True),
|
||||
sa.ForeignKey("managed_repos.id", ondelete="CASCADE"),
|
||||
nullable=False, unique=True),
|
||||
sa.Column("tier", sa.String(20), nullable=False),
|
||||
sa.Column("core_pass", sa.Boolean, nullable=False, server_default="false"),
|
||||
sa.Column("standard_pass", sa.Boolean, nullable=False, server_default="false"),
|
||||
sa.Column("full_pass", sa.Boolean, nullable=False, server_default="false"),
|
||||
sa.Column("criteria", JSON, nullable=True), # full criterion list
|
||||
sa.Column("fingerprint", sa.Text, nullable=False), # pipe-joined timestamps
|
||||
sa.Column("checked_at", sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
)
|
||||
op.create_index("ix_doi_cache_repo_id", "doi_cache", ["repo_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("doi_cache")
|
||||
@@ -0,0 +1,19 @@
|
||||
"""Add routing_note to capability_requests
|
||||
|
||||
Revision ID: l9g0h1i2j3k4
|
||||
Revises: k8f9a0b1c2d3
|
||||
Create Date: 2026-03-20
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = 'l9g0h1i2j3k4'
|
||||
down_revision = 'k8f9a0b1c2d3'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column('capability_requests', sa.Column('routing_note', sa.Text(), nullable=True))
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column('capability_requests', 'routing_note')
|
||||
120
state-hub/policies/repo-doi.md
Normal file
120
state-hub/policies/repo-doi.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Repository Definition of Integrated (DoI)
|
||||
|
||||
A repository is considered **fully integrated** with the Custodian State Hub
|
||||
when all criteria below are satisfied. Criteria are grouped by tier: a repo
|
||||
that meets all **Core** criteria is *registered*; meeting **Standard** criteria
|
||||
makes it *integrated*; meeting **Full** criteria makes it *fully integrated*.
|
||||
|
||||
---
|
||||
|
||||
## Tier 1 — Core (Registered)
|
||||
|
||||
These are the minimum requirements for the state-hub to be aware of a repo.
|
||||
|
||||
- [ ] **Registered in state-hub** — repo exists in `managed_repos` with correct
|
||||
`slug`, `domain_slug`, `local_path`, and `remote_url`. Verify:
|
||||
`GET /repos/{slug}`
|
||||
|
||||
- [ ] **Domain assigned** — repo is linked to an active domain. The domain must
|
||||
exist in `domains` and be in `active` status.
|
||||
|
||||
- [ ] **Local path resolves** — `_resolve_repo_path()` finds an existing
|
||||
directory on the current host (either via `host_paths[hostname]` or
|
||||
`local_path`, both support `~` expansion).
|
||||
|
||||
- [ ] **Remote URL reachable** — `remote_url` points to a live Gitea/GitHub
|
||||
repository. The repo must be pushable from the registered local path.
|
||||
|
||||
---
|
||||
|
||||
## Tier 2 — Standard (Integrated)
|
||||
|
||||
A repo at this tier participates in state-hub tracking and tooling.
|
||||
|
||||
- [ ] **SCOPE.md present** — a `SCOPE.md` exists at the repo root following the
|
||||
standard template (`state-hub/scripts/project_rules/scope.template`).
|
||||
Sections: One-liner, Core Idea, In Scope, Out of Scope, Relevant When,
|
||||
Not Relevant When, Current State, How It Fits, Provided Capabilities.
|
||||
|
||||
- [ ] **CLAUDE.md present** — a `CLAUDE.md` exists at the repo root with at
|
||||
minimum: domain context, session protocol (start: `get_domain_summary`,
|
||||
end: `add_progress_event`), and stack/commands reference.
|
||||
|
||||
- [ ] **Workplan convention followed** — any workplans live under
|
||||
`workplans/<ID>-<slug>.md` with valid YAML frontmatter
|
||||
(`id`, `type`, `title`, `domain`, `status`, `state_hub_workstream_id`).
|
||||
Verified by: `make check-consistency REPO={slug}` → PASS or WARN only
|
||||
(no FAIL).
|
||||
|
||||
- [ ] **SBOM ingested** — at least one dependency lockfile exists
|
||||
(`uv.lock`, `requirements.txt`, `package-lock.json`, `yarn.lock`,
|
||||
`Cargo.lock`, `go.sum`) and has been ingested:
|
||||
`make ingest-sbom REPO={slug}` succeeded with ≥1 entry.
|
||||
`last_sbom_at` is set on the repo record.
|
||||
|
||||
- [ ] **TPSC declared** — a `tpsc.yaml` exists at the repo root declaring all
|
||||
external service dependencies, and has been ingested:
|
||||
`make ingest-tpsc REPO={slug}` succeeded. If the repo has no external
|
||||
service dependencies, `tpsc.yaml` must exist with an empty `services: []`
|
||||
to make the absence explicit.
|
||||
|
||||
---
|
||||
|
||||
## Tier 3 — Full (Fully Integrated)
|
||||
|
||||
A fully integrated repo contributes to cross-repo planning and capability routing.
|
||||
|
||||
- [ ] **Active repo goal** — at least one `RepoGoal` with `status: active`
|
||||
exists for the repo in the state-hub. Goals connect workstreams to
|
||||
strategic intent. Create via: `create_repo_goal(repo_slug, title, ...)`
|
||||
|
||||
- [ ] **Provided Capabilities declared** — if the repo exposes capabilities to
|
||||
other domains, they are declared in the `## Provided Capabilities` section
|
||||
of `SCOPE.md` using `capability` fenced blocks and have been ingested via
|
||||
`make ingest-capabilities REPO={slug}`.
|
||||
|
||||
- [ ] **Agents template applied** — `CLAUDE.md` references the kaizen agent
|
||||
system (`get_kaizen_agent()`) and includes the `agents.template` section
|
||||
listing available specialised personas for the domain.
|
||||
|
||||
- [ ] **Consistency check clean** — `make check-consistency REPO={slug}`
|
||||
reports **0 FAIL, 0 WARN** (C-12 warnings on legacy DB-only tasks are
|
||||
exempt if the workstream predates ADR-001).
|
||||
|
||||
- [ ] **Host path registered for all active hosts** — `host_paths` contains
|
||||
an entry for every machine where the repo is actively worked on. Use
|
||||
`update_repo_path(slug, path)` to register additional hosts.
|
||||
|
||||
---
|
||||
|
||||
## Integration Checklist (Quick Reference)
|
||||
|
||||
| # | Criterion | Tier | Verified by |
|
||||
|---|---|---|---|
|
||||
| 1 | Registered in state-hub | Core | `GET /repos/{slug}` |
|
||||
| 2 | Domain assigned | Core | `GET /repos/{slug}` → `domain_slug` |
|
||||
| 3 | Local path resolves | Core | `check_repo_consistency(slug)` |
|
||||
| 4 | Remote URL reachable | Core | `git push` dry-run |
|
||||
| 5 | SCOPE.md present | Standard | `ls SCOPE.md` |
|
||||
| 6 | CLAUDE.md present | Standard | `ls CLAUDE.md` |
|
||||
| 7 | Workplan convention followed | Standard | `make check-consistency` |
|
||||
| 8 | SBOM ingested | Standard | `last_sbom_at` on repo record |
|
||||
| 9 | TPSC declared | Standard | `make ingest-tpsc` |
|
||||
| 10 | Active repo goal | Full | `get_repo_goals(slug)` |
|
||||
| 11 | Provided Capabilities declared | Full | `make ingest-capabilities` |
|
||||
| 12 | Agents template applied | Full | `CLAUDE.md` review |
|
||||
| 13 | Consistency check clean | Full | `make check-consistency` → 0 FAIL/WARN |
|
||||
| 14 | Host paths registered | Full | `GET /repos/{slug}` → `host_paths` |
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- The DoI is enforced by convention, not by automated gates (as of v0.6).
|
||||
A future `make check-doi REPO={slug}` target is planned.
|
||||
- Repos that are `archived` or `deprecated` are exempt from DoI compliance.
|
||||
- The **Core** tier is a prerequisite for all state-hub tooling
|
||||
(`check_repo_consistency`, `ingest_sbom_tool`, `ingest_tpsc_tool`).
|
||||
- A repo may satisfy **Standard** without **Full** indefinitely — Full
|
||||
integration is appropriate when the repo is actively contributing to
|
||||
cross-domain work or exposing capabilities to other repos.
|
||||
@@ -1,3 +1,5 @@
|
||||
# Workstream Definition of Done
|
||||
|
||||
A workstream is considered finished if and only if:
|
||||
- All tasks in the workstream have been finished, found unnecessary or been transfered to another workstream
|
||||
- All referenced requirements have been adressed with nothing relevent missing or documented as out of scope with a stated reason
|
||||
|
||||
@@ -16,6 +16,7 @@ dependencies = [
|
||||
"python-dotenv>=1.0.0",
|
||||
"psycopg2-binary>=2.9.0",
|
||||
"llm-connect",
|
||||
"pyyaml>=6.0.3",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
|
||||
132
state-hub/scripts/check_doi.py
Normal file
132
state-hub/scripts/check_doi.py
Normal file
@@ -0,0 +1,132 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check Repository Definition of Integrated (DoI) criteria.
|
||||
|
||||
Usage:
|
||||
uv run python scripts/check_doi.py --repo <slug> [--json]
|
||||
uv run python scripts/check_doi.py --all [--json]
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
# Allow importing from the api package
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
from api.doi_engine import evaluate
|
||||
|
||||
API_BASE = "http://127.0.0.1:8000"
|
||||
|
||||
STATUS_ICON = {"pass": "✓", "fail": "✗", "warn": "⚠", "skip": "—"}
|
||||
STATUS_COLOR = {
|
||||
"pass": "\033[32m", # green
|
||||
"fail": "\033[31m", # red
|
||||
"warn": "\033[33m", # yellow
|
||||
"skip": "\033[90m", # grey
|
||||
}
|
||||
RESET = "\033[0m"
|
||||
BOLD = "\033[1m"
|
||||
|
||||
TIER_COLOR = {
|
||||
"full": "\033[32m",
|
||||
"standard": "\033[33m",
|
||||
"core": "\033[33m",
|
||||
"none": "\033[31m",
|
||||
}
|
||||
|
||||
|
||||
def _get(path: str) -> object:
|
||||
req = urllib.request.Request(
|
||||
f"{API_BASE}{path}/",
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=10) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
def _print_report(report, use_color: bool = True) -> None:
|
||||
tier_c = TIER_COLOR.get(report.tier, "") if use_color else ""
|
||||
reset = RESET if use_color else ""
|
||||
bold = BOLD if use_color else ""
|
||||
|
||||
print(f"\n{bold}Repo: {report.repo_slug}{reset}")
|
||||
print(f" Tier: {tier_c}{bold}{report.tier.upper()}{reset} "
|
||||
f"(core={'✓' if report.core_pass else '✗'} "
|
||||
f"standard={'✓' if report.standard_pass else '✗'} "
|
||||
f"full={'✓' if report.full_pass else '✗'})")
|
||||
|
||||
current_tier = None
|
||||
for c in report.criteria:
|
||||
if c.tier != current_tier:
|
||||
current_tier = c.tier
|
||||
print(f" ── {c.tier.upper()} ──")
|
||||
sc = STATUS_COLOR.get(c.status, "") if use_color else ""
|
||||
ico = STATUS_ICON.get(c.status, "?")
|
||||
detail = f" {c.detail}" if c.detail else ""
|
||||
print(f" {sc}{ico}{reset} {c.id}: {c.label}{detail}")
|
||||
|
||||
|
||||
async def check_repo(slug: str, as_json: bool) -> bool:
|
||||
try:
|
||||
repo = _get(f"/repos/{slug}/")
|
||||
except Exception as e:
|
||||
print(f"✗ Could not fetch repo '{slug}': {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
report = await evaluate(repo, API_BASE)
|
||||
|
||||
if as_json:
|
||||
print(json.dumps({
|
||||
"repo_slug": report.repo_slug,
|
||||
"tier": report.tier,
|
||||
"core_pass": report.core_pass,
|
||||
"standard_pass": report.standard_pass,
|
||||
"full_pass": report.full_pass,
|
||||
"checked_at": report.checked_at,
|
||||
"criteria": [
|
||||
{"id": c.id, "label": c.label, "tier": c.tier,
|
||||
"status": c.status, "detail": c.detail}
|
||||
for c in report.criteria
|
||||
],
|
||||
}, indent=2))
|
||||
else:
|
||||
_print_report(report)
|
||||
|
||||
return report.tier != "none"
|
||||
|
||||
|
||||
async def main_async() -> None:
|
||||
parser = argparse.ArgumentParser(description="Check Repository DoI criteria")
|
||||
group = parser.add_mutually_exclusive_group(required=True)
|
||||
group.add_argument("--repo", metavar="SLUG")
|
||||
group.add_argument("--all", action="store_true")
|
||||
parser.add_argument("--json", action="store_true", dest="as_json")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.all:
|
||||
repos = _get("/repos/")
|
||||
slugs = [r["slug"] for r in repos if r.get("status") == "active"]
|
||||
else:
|
||||
slugs = [args.repo]
|
||||
|
||||
results = await asyncio.gather(*[check_repo(s, args.as_json) for s in slugs])
|
||||
|
||||
if not args.as_json:
|
||||
counts = {"full": 0, "standard": 0, "core": 0, "none": 0}
|
||||
for slug, ok in zip(slugs, results):
|
||||
pass # already printed per-repo
|
||||
|
||||
if len(slugs) > 1:
|
||||
# Re-fetch for summary (already printed above)
|
||||
pass
|
||||
|
||||
sys.exit(0 if all(results) else 1)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
asyncio.run(main_async())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
145
state-hub/scripts/ingest_tpsc.py
Normal file
145
state-hub/scripts/ingest_tpsc.py
Normal file
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Ingest tpsc.yaml service dependency declarations into the State Hub.
|
||||
|
||||
Usage:
|
||||
uv run python scripts/ingest_tpsc.py --repo <slug> [--dry-run]
|
||||
uv run python scripts/ingest_tpsc.py --all [--dry-run]
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
import tomllib as _t # noqa — fallback not really viable; yaml is required
|
||||
yaml = None
|
||||
|
||||
API_BASE = "http://127.0.0.1:8000"
|
||||
TPSC_FILENAME = "tpsc.yaml"
|
||||
|
||||
|
||||
def _get(path: str) -> dict | list:
|
||||
req = urllib.request.Request(f"{API_BASE}{path}", headers={"Accept": "application/json"})
|
||||
with urllib.request.urlopen(req) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
def _post(path: str, payload: dict) -> dict:
|
||||
data = json.dumps(payload).encode()
|
||||
req = urllib.request.Request(
|
||||
f"{API_BASE}{path}/",
|
||||
data=data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req) as r:
|
||||
return json.loads(r.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode()
|
||||
print(f" ERROR {e.code}: {body}", file=sys.stderr)
|
||||
raise
|
||||
|
||||
|
||||
def _load_yaml(path: Path) -> dict:
|
||||
if yaml is None:
|
||||
raise RuntimeError("PyYAML is required: uv add pyyaml")
|
||||
with open(path) as f:
|
||||
return yaml.safe_load(f) or {}
|
||||
|
||||
|
||||
def _resolve_repo_path(repo: dict) -> str:
|
||||
import socket
|
||||
hostname = socket.gethostname()
|
||||
host_paths = repo.get("host_paths") or {}
|
||||
candidates = []
|
||||
if host_paths.get(hostname):
|
||||
candidates.append(host_paths[hostname])
|
||||
if repo.get("local_path"):
|
||||
candidates.append(repo["local_path"])
|
||||
for raw in candidates:
|
||||
p = Path(raw).expanduser()
|
||||
if p.is_dir():
|
||||
return str(p)
|
||||
return ""
|
||||
|
||||
|
||||
def ingest_repo(slug: str, dry_run: bool = False) -> bool:
|
||||
try:
|
||||
repo = _get(f"/repos/{slug}")
|
||||
except Exception as e:
|
||||
print(f" ✗ Repo '{slug}' not found: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
if isinstance(repo, dict) and repo.get("error"):
|
||||
print(f" ✗ {repo['error']}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
repo_path = _resolve_repo_path(repo)
|
||||
if not repo_path:
|
||||
print(f" ✗ No accessible local path for '{slug}' on this host.", file=sys.stderr)
|
||||
return False
|
||||
|
||||
tpsc_file = Path(repo_path) / TPSC_FILENAME
|
||||
if not tpsc_file.exists():
|
||||
print(f" — '{slug}': no {TPSC_FILENAME} found, skipping.")
|
||||
return True
|
||||
|
||||
data = _load_yaml(tpsc_file)
|
||||
services = data.get("services", [])
|
||||
if not services:
|
||||
print(f" — '{slug}': {TPSC_FILENAME} has no services entries, skipping.")
|
||||
return True
|
||||
|
||||
entries = [
|
||||
{
|
||||
"service_slug": svc.get("slug", ""),
|
||||
"purpose": svc.get("purpose"),
|
||||
"auth_type": svc.get("auth"),
|
||||
"endpoint_override": svc.get("endpoint"),
|
||||
"notes": svc.get("notes"),
|
||||
}
|
||||
for svc in services
|
||||
if svc.get("slug")
|
||||
]
|
||||
|
||||
print(f" {'[dry-run] ' if dry_run else ''}'{slug}': {len(entries)} service(s) from {TPSC_FILENAME}")
|
||||
for e in entries:
|
||||
print(f" • {e['service_slug']} ({e.get('auth_type', '?')}) — {e.get('purpose', '')}")
|
||||
|
||||
if dry_run:
|
||||
return True
|
||||
|
||||
result = _post("/tpsc/ingest", {
|
||||
"repo_slug": slug,
|
||||
"source_file": TPSC_FILENAME,
|
||||
"entries": entries,
|
||||
})
|
||||
print(f" ✓ Snapshot {result['id'][:8]}… ingested {result['entry_count']} entries")
|
||||
return True
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Ingest tpsc.yaml into State Hub")
|
||||
group = parser.add_mutually_exclusive_group(required=True)
|
||||
group.add_argument("--repo", metavar="SLUG", help="Single repo slug")
|
||||
group.add_argument("--all", action="store_true", help="All registered repos")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Parse only, do not POST")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.all:
|
||||
repos = _get("/repos/")
|
||||
slugs = [r["slug"] for r in repos]
|
||||
else:
|
||||
slugs = [args.repo]
|
||||
|
||||
ok = all(ingest_repo(slug, dry_run=args.dry_run) for slug in slugs)
|
||||
sys.exit(0 if ok else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -335,3 +335,108 @@ class TestCapabilityRequestLifecycle:
|
||||
"status": "requested",
|
||||
})
|
||||
assert r.status_code == 422
|
||||
|
||||
|
||||
class TestCapabilityRequestRouting:
|
||||
async def test_word_boundary_avoids_substring_false_positive(self, client):
|
||||
"""'postgres' keyword must not match inside 'postgresql'."""
|
||||
await _setup_two_domains(client)
|
||||
# Register two entries: PostgreSQL HA (keywords: postgres, ha) and k3s (keywords: k3s, cluster)
|
||||
await _register_catalog(client, domain="railiance", cap_type="infrastructure",
|
||||
title="PostgreSQL HA", keywords=["postgresql", "postgres", "ha"])
|
||||
await _register_catalog(client, domain="custodian", cap_type="infrastructure",
|
||||
title="K3s provisioning", keywords=["k3s", "cluster", "k8s"])
|
||||
|
||||
# Description mentions k8s and cluster but NOT standalone "postgres" or "ha"
|
||||
req = await _create_request(
|
||||
client,
|
||||
title="K3s cluster access",
|
||||
description="Need k8s foundations. cluster must be up. kubeconfig required.",
|
||||
cap_type="infrastructure",
|
||||
)
|
||||
# k3s entry should win (k8s + cluster match), not postgres entry
|
||||
assert req["catalog_entry_id"] is not None
|
||||
# Routing note should mention k3s
|
||||
assert req["routing_note"] is not None
|
||||
assert "K3s" in req["routing_note"] or "k3s" in req["routing_note"].lower()
|
||||
|
||||
async def test_title_included_in_routing(self, client):
|
||||
"""Title keywords should contribute to routing score."""
|
||||
await _setup_two_domains(client)
|
||||
await _register_catalog(client, domain="railiance", cap_type="infrastructure",
|
||||
title="K3s cluster", keywords=["k3s", "cluster", "kubernetes"])
|
||||
await _register_catalog(client, domain="custodian", cap_type="infrastructure",
|
||||
title="Postgres DB", keywords=["postgresql", "postgres", "database"])
|
||||
|
||||
# Title contains "k3s" but description is generic
|
||||
req = await _create_request(
|
||||
client,
|
||||
title="k3s cluster access needed",
|
||||
description="Need access to proceed with deployment.",
|
||||
cap_type="infrastructure",
|
||||
)
|
||||
assert req["routing_note"] is not None
|
||||
assert req["catalog_entry_id"] is not None
|
||||
# Should match k3s (title has "k3s" and "cluster")
|
||||
# Verify via routing note
|
||||
assert "K3s" in req["routing_note"] or "k3s" in req["routing_note"].lower()
|
||||
|
||||
async def test_ambiguous_routing_broadcasts(self, client):
|
||||
"""Tied scores should broadcast (no fulfilling domain)."""
|
||||
await _setup_two_domains(client)
|
||||
await _register_catalog(client, domain="railiance", cap_type="infrastructure",
|
||||
title="Entry A", keywords=["cluster"])
|
||||
await _register_catalog(client, domain="custodian", cap_type="infrastructure",
|
||||
title="Entry B", keywords=["cluster"])
|
||||
req = await _create_request(
|
||||
client,
|
||||
title="cluster needed",
|
||||
description="cluster access",
|
||||
cap_type="infrastructure",
|
||||
)
|
||||
assert req["fulfilling_domain_slug"] is None
|
||||
assert "ambiguous" in req["routing_note"]
|
||||
|
||||
async def test_patch_corrects_catalog_entry_and_reroutes(self, client):
|
||||
"""PATCH /capability-requests/{id} corrects catalog_entry_id and re-derives domain."""
|
||||
req_d, ful_d = await _setup_two_domains(client)
|
||||
# Register wrong entry (postgres) and correct entry (k3s)
|
||||
wrong = await _register_catalog(client, domain="railiance", cap_type="infrastructure",
|
||||
title="PostgreSQL HA", keywords=["postgresql"])
|
||||
correct = await _register_catalog(client, domain="custodian", cap_type="infrastructure",
|
||||
title="K3s cluster", keywords=["k3s"])
|
||||
|
||||
# Create request — auto-routes to postgres (only postgres keyword matches "postgresql" in description)
|
||||
req = await _create_request(
|
||||
client,
|
||||
description="Need postgresql and k3s access",
|
||||
cap_type="infrastructure",
|
||||
)
|
||||
|
||||
# Patch to correct catalog entry
|
||||
r = await client.patch(f"/capability-requests/{req['id']}", json={
|
||||
"catalog_entry_id": correct["id"],
|
||||
})
|
||||
assert r.status_code == 200
|
||||
data = r.json()
|
||||
assert data["catalog_entry_id"] == correct["id"]
|
||||
assert data["fulfilling_domain_slug"] == "custodian"
|
||||
assert "hub correction" in data["routing_note"]
|
||||
assert "K3s" in data["routing_note"]
|
||||
|
||||
async def test_patch_unknown_catalog_entry_404(self, client):
|
||||
await _setup_two_domains(client)
|
||||
req = await _create_request(client, cap_type="security")
|
||||
import uuid as _uuid
|
||||
r = await client.patch(f"/capability-requests/{req['id']}", json={
|
||||
"catalog_entry_id": str(_uuid.uuid4()),
|
||||
})
|
||||
assert r.status_code == 404
|
||||
|
||||
async def test_patch_priority(self, client):
|
||||
await _setup_two_domains(client)
|
||||
req = await _create_request(client, cap_type="security")
|
||||
r = await client.patch(f"/capability-requests/{req['id']}", json={"priority": "critical"})
|
||||
assert r.status_code == 200
|
||||
assert r.json()["priority"] == "critical"
|
||||
assert "priority" in r.json()["routing_note"]
|
||||
|
||||
11
state-hub/uv.lock
generated
11
state-hub/uv.lock
generated
@@ -145,10 +145,16 @@ sdist = { url = "https://files.pythonhosted.org/packages/92/88/b8527e1b00c1811db
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d3/25/79c98ebe12df31548ba4eaf44db11b7cad6b3e7b4203718335620939083c/caio-0.9.25-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:fb7ff95af4c31ad3f03179149aab61097a71fd85e05f89b4786de0359dffd044", size = 36983 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a3/2b/21288691f16d479945968a0a4f2856818c1c5be56881d51d4dac9b255d26/caio-0.9.25-cp312-cp312-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:97084e4e30dfa598449d874c4d8e0c8d5ea17d2f752ef5e48e150ff9d240cd64", size = 82012 },
|
||||
{ url = "https://files.pythonhosted.org/packages/03/c4/8a1b580875303500a9c12b9e0af58cb82e47f5bcf888c2457742a138273c/caio-0.9.25-cp312-cp312-manylinux_2_34_aarch64.whl", hash = "sha256:4fa69eba47e0f041b9d4f336e2ad40740681c43e686b18b191b6c5f4c5544bfb", size = 81502 },
|
||||
{ url = "https://files.pythonhosted.org/packages/d1/1c/0fe770b8ffc8362c48134d1592d653a81a3d8748d764bec33864db36319d/caio-0.9.25-cp312-cp312-manylinux_2_34_x86_64.whl", hash = "sha256:6bebf6f079f1341d19f7386db9b8b1f07e8cc15ae13bfdaff573371ba0575d69", size = 80200 },
|
||||
{ url = "https://files.pythonhosted.org/packages/31/57/5e6ff127e6f62c9f15d989560435c642144aa4210882f9494204bc892305/caio-0.9.25-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:d6c2a3411af97762a2b03840c3cec2f7f728921ff8adda53d7ea2315a8563451", size = 36979 },
|
||||
{ url = "https://files.pythonhosted.org/packages/a3/9f/f21af50e72117eb528c422d4276cbac11fb941b1b812b182e0a9c70d19c5/caio-0.9.25-cp313-cp313-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0998210a4d5cd5cb565b32ccfe4e53d67303f868a76f212e002a8554692870e6", size = 81900 },
|
||||
{ url = "https://files.pythonhosted.org/packages/9c/12/c39ae2a4037cb10ad5eb3578eb4d5f8c1a2575c62bba675f3406b7ef0824/caio-0.9.25-cp313-cp313-manylinux_2_34_aarch64.whl", hash = "sha256:1a177d4777141b96f175fe2c37a3d96dec7911ed9ad5f02bac38aaa1c936611f", size = 81523 },
|
||||
{ url = "https://files.pythonhosted.org/packages/22/59/f8f2e950eb4f1a5a3883e198dca514b9d475415cb6cd7b78b9213a0dd45a/caio-0.9.25-cp313-cp313-manylinux_2_34_x86_64.whl", hash = "sha256:9ed3cfb28c0e99fec5e208c934e5c157d0866aa9c32aa4dc5e9b6034af6286b7", size = 80243 },
|
||||
{ url = "https://files.pythonhosted.org/packages/69/ca/a08fdc7efdcc24e6a6131a93c85be1f204d41c58f474c42b0670af8c016b/caio-0.9.25-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:fab6078b9348e883c80a5e14b382e6ad6aabbc4429ca034e76e730cf464269db", size = 36978 },
|
||||
{ url = "https://files.pythonhosted.org/packages/5e/6c/d4d24f65e690213c097174d26eda6831f45f4734d9d036d81790a27e7b78/caio-0.9.25-cp314-cp314-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:44a6b58e52d488c75cfaa5ecaa404b2b41cc965e6c417e03251e868ecd5b6d77", size = 81832 },
|
||||
{ url = "https://files.pythonhosted.org/packages/87/a4/e534cf7d2d0e8d880e25dd61e8d921ffcfe15bd696734589826f5a2df727/caio-0.9.25-cp314-cp314-manylinux_2_34_aarch64.whl", hash = "sha256:628a630eb7fb22381dd8e3c8ab7f59e854b9c806639811fc3f4310c6bd711d79", size = 81565 },
|
||||
{ url = "https://files.pythonhosted.org/packages/3f/ed/bf81aeac1d290017e5e5ac3e880fd56ee15e50a6d0353986799d1bc5cfd5/caio-0.9.25-cp314-cp314-manylinux_2_34_x86_64.whl", hash = "sha256:0ba16aa605ccb174665357fc729cf500679c2d94d5f1458a6f0d5ca48f2060a7", size = 80071 },
|
||||
{ url = "https://files.pythonhosted.org/packages/86/93/1f76c8d1bafe3b0614e06b2195784a3765bbf7b0a067661af9e2dd47fc33/caio-0.9.25-py3-none-any.whl", hash = "sha256:06c0bb02d6b929119b1cfbe1ca403c768b2013a369e2db46bfa2a5761cf82e40", size = 19087 },
|
||||
]
|
||||
|
||||
@@ -677,6 +683,9 @@ requires-dist = [
|
||||
{ name = "toml" },
|
||||
]
|
||||
|
||||
[package.metadata.requires-dev]
|
||||
dev = [{ name = "pytest", specifier = ">=9.0.2" }]
|
||||
|
||||
[[package]]
|
||||
name = "mako"
|
||||
version = "1.3.10"
|
||||
@@ -1424,6 +1433,7 @@ dependencies = [
|
||||
{ name = "pydantic" },
|
||||
{ name = "pydantic-settings" },
|
||||
{ name = "python-dotenv" },
|
||||
{ name = "pyyaml" },
|
||||
{ name = "sqlalchemy", extra = ["asyncio"] },
|
||||
{ name = "uvicorn", extra = ["standard"] },
|
||||
]
|
||||
@@ -1447,6 +1457,7 @@ requires-dist = [
|
||||
{ name = "pydantic", specifier = ">=2.10.0" },
|
||||
{ name = "pydantic-settings", specifier = ">=2.7.0" },
|
||||
{ name = "python-dotenv", specifier = ">=1.0.0" },
|
||||
{ name = "pyyaml", specifier = ">=6.0.3" },
|
||||
{ name = "sqlalchemy", extras = ["asyncio"], specifier = ">=2.0.0" },
|
||||
{ name = "uvicorn", extras = ["standard"], specifier = ">=0.32.0" },
|
||||
]
|
||||
|
||||
210
workplans/CUST-WP-0023-tpsc.md
Normal file
210
workplans/CUST-WP-0023-tpsc.md
Normal file
@@ -0,0 +1,210 @@
|
||||
---
|
||||
id: CUST-WP-0023
|
||||
type: feature
|
||||
title: Third-Party Services Catalog (TPSC)
|
||||
domain: custodian
|
||||
status: done
|
||||
owner: custodian-agent
|
||||
topic_slug: custodian
|
||||
created: 2026-03-19
|
||||
updated: 2026-03-20
|
||||
state_hub_workstream_id: "23208a99-0ef6-4154-9454-a2f2065b6b19"
|
||||
---
|
||||
|
||||
# TPSC — Third-Party Services Catalog
|
||||
|
||||
Track external service dependencies (APIs, SaaS, CLIs) across all repos.
|
||||
Primary data lives in repos (`tpsc.yaml`) and a canon catalog
|
||||
(`canon/tpsc/<slug>.yaml`). State-hub collects and reports.
|
||||
|
||||
GDPR maturity scale (CNIL/IAPP CMMI-aligned):
|
||||
`unknown | non_compliant | initial | developing | defined | managed | certified`
|
||||
|
||||
Pricing model: `free | paid | freemium | usage_based | unknown`
|
||||
|
||||
## Task: DB migration — tpsc_catalog, tpsc_snapshots, tpsc_entries
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T01
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "038a0284-bb76-4ce7-8861-1686667acbb5"
|
||||
```
|
||||
|
||||
Create Alembic migration `j7e8f9a0b1c2_tpsc.py`.
|
||||
|
||||
Tables:
|
||||
- `tpsc_catalog`: id, slug (unique), name, provider, category, website_url,
|
||||
pricing_model, gdpr_maturity, gdpr_notes, dpa_available, tos_url,
|
||||
privacy_policy_url, data_processing_regions (JSON), data_retention_notes,
|
||||
status (active/deprecated), created_at, updated_at
|
||||
- `tpsc_snapshots`: id, repo_id (FK managed_repos nullable), snapshot_at,
|
||||
source_file, entry_count
|
||||
- `tpsc_entries`: id, snapshot_id (FK), catalog_id (FK tpsc_catalog nullable),
|
||||
service_slug, purpose, auth_type, endpoint_override, notes
|
||||
|
||||
---
|
||||
|
||||
## Task: SQLAlchemy models
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T02
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "990d4e58-35b0-45f0-8de6-8955049aa7d5"
|
||||
```
|
||||
|
||||
Create `api/models/tpsc.py` with TPSCCatalog, TPSCSnapshot, TPSCEntry models.
|
||||
Register in `api/models/__init__.py`.
|
||||
|
||||
---
|
||||
|
||||
## Task: Pydantic schemas
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T03
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "5feeb161-b654-4e4a-8a6a-bb685c239ce5"
|
||||
```
|
||||
|
||||
Create `api/schemas/tpsc.py` with Read/Create schemas for all three models.
|
||||
Include `GDPRMaturity` and `PricingModel` string enums.
|
||||
`TPSCCatalog` schema: include `gdpr_warning: bool` computed field
|
||||
(True when gdpr_maturity in [unknown, non_compliant, initial]).
|
||||
|
||||
---
|
||||
|
||||
## Task: FastAPI router /tpsc/
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T04
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "593471b4-cd3a-4251-8c5c-ee42b6a9e089"
|
||||
```
|
||||
|
||||
Create `api/routers/tpsc.py`:
|
||||
- `GET /tpsc/catalog/` — list services (filter: gdpr_maturity, category, pricing_model)
|
||||
- `GET /tpsc/catalog/{slug}` — single service
|
||||
- `POST /tpsc/catalog/` — register/upsert service
|
||||
- `POST /tpsc/ingest/` — accept snapshot + entries for a repo
|
||||
- `GET /tpsc/snapshots/` — list snapshots (filter: repo_slug)
|
||||
- `GET /tpsc/report/gdpr` — aggregated GDPR warnings across all repos
|
||||
|
||||
Register in `api/main.py`.
|
||||
|
||||
---
|
||||
|
||||
## Task: MCP tools
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T05
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "7370d020-06cf-4ebe-9f78-619e41c4b85c"
|
||||
```
|
||||
|
||||
Add to `mcp_server/server.py`:
|
||||
- `register_service(slug, name, provider, pricing_model, gdpr_maturity, ...)`
|
||||
- `list_services(gdpr_maturity?, category?, pricing_model?)`
|
||||
- `ingest_tpsc_tool(repo_slug)` — runs ingest_tpsc.py for the repo
|
||||
- `get_gdpr_report()` — returns warning summary across all repos
|
||||
|
||||
---
|
||||
|
||||
## Task: Ingest script
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T06
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "5ec305d3-fdfb-4f81-b3ab-1ea57a4ec0c5"
|
||||
```
|
||||
|
||||
Create `scripts/ingest_tpsc.py`:
|
||||
- Reads `tpsc.yaml` from repo root (auto-detected via registered local_path)
|
||||
- Resolves catalog_id by slug for each entry
|
||||
- POSTs snapshot + entries to `/tpsc/ingest/`
|
||||
- `--dry-run` flag
|
||||
- `--repo SLUG` or `--all` flags
|
||||
|
||||
Add Makefile targets:
|
||||
```
|
||||
make ingest-tpsc REPO=<slug>
|
||||
make ingest-tpsc-all
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task: Canon service catalog seed files
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T07
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "21c13b4d-585a-4a60-a283-29baa2dd6d7d"
|
||||
```
|
||||
|
||||
Create `canon/tpsc/` with YAML files for:
|
||||
- `openai-api.yaml`
|
||||
- `anthropic-api.yaml`
|
||||
- `gemini-api.yaml`
|
||||
- `openrouter-api.yaml`
|
||||
|
||||
Each file: slug, name, provider, category, pricing_model, gdpr_maturity,
|
||||
gdpr_notes, dpa_available, tos_url, privacy_policy_url,
|
||||
data_processing_regions, data_retention_notes.
|
||||
|
||||
---
|
||||
|
||||
## Task: tpsc.yaml for llm-connect
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T08
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "d658f81b-7408-456d-b4bd-440b44a67e43"
|
||||
```
|
||||
|
||||
Create `/home/worsch/llm-connect/tpsc.yaml` declaring:
|
||||
openai-api, anthropic-api (via claude_code CLI), gemini-api, openrouter-api.
|
||||
|
||||
---
|
||||
|
||||
## Task: Dashboard page
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T09
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "a5367fc4-ef12-4f52-b642-d33b91b7cc2c"
|
||||
```
|
||||
|
||||
Create `dashboard/src/tpsc.md`:
|
||||
- Service catalog table with GDPR maturity color coding
|
||||
(red: non_compliant/unknown, amber: initial/developing, green: defined+)
|
||||
- Warning cards for repos using non-green services
|
||||
- Per-repo breakdown table
|
||||
- Pricing model summary
|
||||
|
||||
Add to `observablehq.config.js` nav.
|
||||
|
||||
---
|
||||
|
||||
## Task: Documentation page
|
||||
|
||||
```task
|
||||
id: CUST-WP-0023-T10
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "6cf1aaec-5c24-4cb5-a1aa-415caeaaca10"
|
||||
```
|
||||
|
||||
Create `dashboard/src/docs/tpsc.md` explaining:
|
||||
- TPSC concept and rationale
|
||||
- tpsc.yaml format with full example
|
||||
- GDPR maturity scale definitions (linked to CNIL/IAPP)
|
||||
- How to add a new service to the canon catalog
|
||||
|
||||
Add to docs nav in `observablehq.config.js`.
|
||||
154
workplans/CUST-WP-0024-repo-doi-gate.md
Normal file
154
workplans/CUST-WP-0024-repo-doi-gate.md
Normal file
@@ -0,0 +1,154 @@
|
||||
---
|
||||
id: CUST-WP-0024
|
||||
type: feature
|
||||
title: Repository DoI automated gate and dashboard integration
|
||||
domain: custodian
|
||||
status: done
|
||||
owner: custodian-agent
|
||||
topic_slug: custodian
|
||||
created: 2026-03-20
|
||||
updated: 2026-03-20
|
||||
state_hub_workstream_id: "f587f244-0dd5-4268-b955-0c4e4cf9aa69"
|
||||
---
|
||||
|
||||
# Repository DoI Automated Gate & Dashboard
|
||||
|
||||
Automates the 14-criterion DoI checklist from `policies/repo-doi.md` and
|
||||
surfaces per-repo DoI status on the Repositories dashboard page.
|
||||
|
||||
## Task: DoI check script — scripts/check_doi.py
|
||||
|
||||
```task
|
||||
id: CUST-WP-0024-T01
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "58e77114-bc54-4f86-9ddc-3834bc702b5c"
|
||||
```
|
||||
|
||||
Create `state-hub/scripts/check_doi.py` implementing all 14 DoI criteria:
|
||||
|
||||
**Tier 1 — Core**
|
||||
- C1: repo registered (`GET /repos/{slug}` returns 200)
|
||||
- C2: domain assigned (`domain_slug` non-null and domain status active)
|
||||
- C3: local path resolves (exists on disk after expanduser)
|
||||
- C4: remote_url set (non-null, non-empty)
|
||||
|
||||
**Tier 2 — Standard**
|
||||
- C5: SCOPE.md present at repo root
|
||||
- C6: CLAUDE.md present at repo root
|
||||
- C7: workplan convention followed — consistency check returns 0 FAIL
|
||||
- C8: SBOM ingested — `last_sbom_at` non-null on repo record
|
||||
- C9: TPSC declared — `tpsc.yaml` exists at repo root AND at least one
|
||||
TPSC snapshot exists for the repo
|
||||
|
||||
**Tier 3 — Full**
|
||||
- C10: active repo goal exists (`GET /repo-goals/?repo_slug=...` has ≥1 active)
|
||||
- C11: Provided Capabilities declared — `SCOPE.md` contains a `capability`
|
||||
fenced block OR `## Provided Capabilities` section says "none" explicitly
|
||||
- C12: agents template applied — CLAUDE.md contains `get_kaizen_agent` or
|
||||
`kaizen` reference
|
||||
- C13: consistency check clean — 0 FAIL and 0 WARN (C-12 exempt)
|
||||
- C14: host paths registered — `host_paths` non-empty
|
||||
|
||||
Output per criterion: PASS / FAIL / WARN / SKIP (with reason).
|
||||
Overall tier: core_pass, standard_pass, full_pass (boolean each).
|
||||
|
||||
CLI flags: `--repo SLUG`, `--all`, `--json`, `--fix` (no-op for now,
|
||||
reserved for future auto-remediation).
|
||||
|
||||
Makefile targets:
|
||||
```
|
||||
make check-doi REPO=<slug>
|
||||
make check-doi-all
|
||||
make check-doi REPO=<slug> JSON=1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task: API endpoint GET /repos/{slug}/doi
|
||||
|
||||
```task
|
||||
id: CUST-WP-0024-T02
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "97801608-2cad-40e6-ba0c-d209bcc3ec04"
|
||||
```
|
||||
|
||||
Add to `api/routers/repos.py`:
|
||||
|
||||
`GET /repos/{slug}/doi` — runs the DoI check logic server-side and returns:
|
||||
```json
|
||||
{
|
||||
"repo_slug": "llm-connect",
|
||||
"tier": "standard", // "none" | "core" | "standard" | "full"
|
||||
"core_pass": true,
|
||||
"standard_pass": true,
|
||||
"full_pass": false,
|
||||
"criteria": [
|
||||
{"id": "C1", "label": "Registered", "tier": "core", "status": "pass"},
|
||||
...
|
||||
],
|
||||
"checked_at": "2026-03-20T..."
|
||||
}
|
||||
```
|
||||
|
||||
Also add `GET /repos/doi/summary` — returns all repos with their tier,
|
||||
sorted by tier ascending (worst first).
|
||||
|
||||
Add Pydantic schemas `DoICriterion`, `DoIReport`, `DoISummaryEntry` to
|
||||
`api/schemas/doi.py`.
|
||||
|
||||
---
|
||||
|
||||
## Task: MCP tool — check_repo_doi
|
||||
|
||||
```task
|
||||
id: CUST-WP-0024-T03
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "3e18a74a-576a-4491-b7cf-3d6ba7746a0f"
|
||||
```
|
||||
|
||||
Add to `mcp_server/server.py`:
|
||||
|
||||
`check_repo_doi(repo_slug)` — returns the DoI report for a single repo.
|
||||
`get_doi_summary()` — returns all repos sorted by tier (worst first),
|
||||
useful for session orientation.
|
||||
|
||||
---
|
||||
|
||||
## Task: Repositories dashboard page — DoI integration
|
||||
|
||||
```task
|
||||
id: CUST-WP-0024-T04
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "ab1a843c-60e3-4eba-ae6a-dec7d74c03f4"
|
||||
```
|
||||
|
||||
Review and update `dashboard/src/repos.md` (or whichever page shows the
|
||||
repo list) to include:
|
||||
|
||||
- DoI tier badge per repo (None / Core / Standard / Full) with colour coding
|
||||
(red / amber / yellow / green)
|
||||
- Expandable per-repo criterion detail (collapsible row or modal)
|
||||
- Summary KPI: count of repos at each tier
|
||||
- Link to the Repository DoI policy page
|
||||
|
||||
If `repos.md` does not exist, create it. If it already shows repo data,
|
||||
enrich it in-place.
|
||||
|
||||
---
|
||||
|
||||
## Task: Consistency check — run fix-consistency
|
||||
|
||||
```task
|
||||
id: CUST-WP-0024-T05
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "5372bde5-b0e5-4993-94a0-ffa9b7bb26e9"
|
||||
```
|
||||
|
||||
After all code is written and tested:
|
||||
- Run `make fix-consistency REPO=the-custodian`
|
||||
- Commit all new/modified files
|
||||
Reference in New Issue
Block a user