feat(tpsc): Third-Party Services Catalog (CUST-WP-0023)

Introduces TPSC for tracking external service dependencies with GDPR compliance maturity (CNIL/IAPP CMMI scale), pricing model, ToS, and data retention information across all repos. Primary data: - canon/tpsc/{openai,anthropic,gemini,openrouter}-api.yaml — service definitions - tpsc.yaml in each repo (llm-connect seeded with 4 services) State-hub additions: - Migration j7e8f9a0b1c2: tpsc_catalog + tpsc_snapshots + tpsc_entries - api/models/tpsc.py, api/schemas/tpsc.py, api/routers/tpsc.py - /tpsc/catalog/, /tpsc/ingest/, /tpsc/snapshots/, /tpsc/report/gdpr endpoints - 4 MCP tools: register_service, list_services, ingest_tpsc_tool, get_gdpr_report - scripts/ingest_tpsc.py + make ingest-tpsc[/-all] targets - Dashboard: tpsc.md page + docs/tpsc.md GDPR maturity scale: unknown | non_compliant | initial | developing | defined | managed | certified Warnings triggered at: unknown, non_compliant, initial Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 00:15:26 +01:00
parent 4e28cab297
commit 60beb1ff35
14 changed files with 1126 additions and 1 deletions
--- a/dashboard/src/docs/tpsc.md
+++ b/dashboard/src/docs/tpsc.md
@@ -0,0 +1,136 @@
+---
+title: Third-Party Services Catalog (TPSC)
+---
+
+# Third-Party Services Catalog (TPSC)
+
+The TPSC tracks external service dependencies (APIs, SaaS, CLIs) across all
+registered repos — complementing the SBOM for package dependencies.
+
+---
+
+## Why TPSC?
+
+Package lockfiles capture Python/JS/Rust dependencies but miss the external
+HTTP services your code calls. These carry compliance, cost, and privacy
+implications that are invisible to standard SBOM tooling.
+
+TPSC provides:
+- A registry of which repos use which external services
+- GDPR compliance maturity ratings per service
+- Pricing model tracking (paid/usage-based costs)
+- Data processing region and retention information
+- GDPR warnings for services not suitable in regulated environments
+
+---
+
+## Primary Data Locations
+
+Following ADR-001 (workplans as repo artefacts), TPSC data lives in two places:
+
+| Location | Purpose |
+|---|---|
+| `<repo>/tpsc.yaml` | Declares which services the repo uses |
+| `the-custodian/canon/tpsc/<slug>.yaml` | Canonical service metadata (ToS, GDPR, pricing) |
+
+The state-hub is a collector — it can be rebuilt from scratch by re-ingesting
+all `tpsc.yaml` files and re-seeding the catalog from canon files.
+
+---
+
+## tpsc.yaml Format
+
+```yaml
+# tpsc.yaml — Third-Party Services Catalog declarations
+# Ingest: cd state-hub && make ingest-tpsc REPO=<slug>
+
+services:
+  - slug: openai-api          # Must match a slug in canon/tpsc/
+    purpose: LLM inference via OpenAI-compatible API
+    auth: api_key             # api_key | oauth | cli | none | unknown
+
+  - slug: stripe
+    purpose: Payment processing
+    auth: api_key
+    endpoint: https://api.stripe.com  # Optional override if non-standard
+    notes: Only used in production tier
+```
+
+---
+
+## Canon Service File Format
+
+```yaml
+# canon/tpsc/openai-api.yaml
+slug: openai-api
+name: OpenAI API
+provider: OpenAI, Inc.
+category: llm_inference       # llm_inference | storage | payments | search | etc.
+website_url: https://openai.com
+pricing_model: usage_based    # free | paid | freemium | usage_based | unknown
+gdpr_maturity: developing     # See scale below
+gdpr_notes: >
+  DPA available. SCCs for EU→US transfer. 30-day retention for safety.
+dpa_available: true
+tos_url: https://openai.com/policies/terms-of-use
+privacy_policy_url: https://openai.com/policies/privacy-policy
+data_processing_regions:
+  - us
+data_retention_notes: >
+  30 days default; zero-retention available on eligible endpoints.
+status: active
+```
+
+---
+
+## GDPR Maturity Scale
+
+Based on the **CNIL / IAPP CMMI Privacy Maturity Model**, adapted for
+third-party service assessment:
+
+| Level | Name | Description | Dashboard |
+|---|---|---|---|
+| 0 | `unknown` | No information about GDPR stance | 🔴 Warning |
+| 1 | `non_compliant` | Known GDPR issues, no remediation | 🔴 Warning |
+| 2 | `initial` | Basic privacy policy only, ad hoc approach | 🟠 Warning |
+| 3 | `developing` | DPA available, some controls, SCCs provided | 🟡 |
+| 4 | `defined` | Formal DPA, SCCs documented, clear retention policy | 🟢 |
+| 5 | `managed` | Independently audited, metrics tracked | 🟢 |
+| 6 | `certified` | ISO 27701 / SOC2 privacy certified | 🟢 |
+
+Services at levels 0–2 (**Warning**) may limit use in GDPR-regulated or
+corporate environments. At minimum, `developing` is needed for routine
+processing of personal data with an API provider.
+
+Reference: [CNIL GDPR maturity model](https://iapp.org/news/b/cnil-publishes-data-protection-management-maturity-model), [IAPP Privacy Maturity Model](https://iapp.org/news/a/achieving-privacy-excellence-understanding-the-privacy-maturity-model)
+
+---
+
+## Adding a New Service
+
+1. Create `the-custodian/canon/tpsc/<slug>.yaml` following the format above
+2. Seed it into the state-hub: `cd state-hub && make api` then POST to `/tpsc/catalog/`
+   (or use the MCP tool: `register_service(slug=..., ...)`)
+3. Add it to your repo's `tpsc.yaml`
+4. Ingest: `make ingest-tpsc REPO=<slug>`
+
+---
+
+## MCP Tools
+
+| Tool | Purpose |
+|---|---|
+| `register_service(slug, ...)` | Add/update a service in the catalog |
+| `list_services(gdpr_maturity?, category?, pricing_model?)` | Browse catalog |
+| `ingest_tpsc_tool(repo_slug)` | Parse tpsc.yaml and ingest snapshot |
+| `get_gdpr_report()` | GDPR warning summary across all repos |
+
+---
+
+## Makefile Targets
+
+```bash
+make ingest-tpsc REPO=llm-connect      # Ingest single repo
+make ingest-tpsc-all                    # Ingest all repos
+make ingest-tpsc REPO=llm-connect DRY_RUN=1  # Preview only
+```
--- a/dashboard/src/tpsc.md
+++ b/dashboard/src/tpsc.md
@@ -0,0 +1,193 @@
+---
+title: Third-Party Services (TPSC)
+---
+
+# Third-Party Services Catalog
+
+```js
+const API = "http://127.0.0.1:8000";
+let apiOk = true;
+
+const catalog = await fetch(`${API}/tpsc/catalog/`)
+  .then(r => r.json())
+  .catch(() => { apiOk = false; return []; });
+
+const gdprReport = await fetch(`${API}/tpsc/report/gdpr`)
+  .then(r => r.json())
+  .catch(() => ({ warnings: [], by_maturity: {}, total_services: 0, warning_count: 0 }));
+
+const snapshots = await fetch(`${API}/tpsc/snapshots/`)
+  .then(r => r.json())
+  .catch(() => []);
+```
+
+```js
+// GDPR maturity colour coding (CNIL/IAPP scale)
+const maturityColor = {
+  unknown:       "#ef4444",  // red
+  non_compliant: "#dc2626",  // deep red
+  initial:       "#f97316",  // orange
+  developing:    "#eab308",  // amber
+  defined:       "#84cc16",  // lime
+  managed:       "#22c55e",  // green
+  certified:     "#16a34a",  // deep green
+};
+
+const maturityLabel = {
+  unknown:       "Unknown",
+  non_compliant: "Non-Compliant",
+  initial:       "Initial",
+  developing:    "Developing",
+  defined:       "Defined",
+  managed:       "Managed",
+  certified:     "Certified",
+};
+
+const WARNING_LEVELS = new Set(["unknown", "non_compliant", "initial"]);
+```
+
+```js
+// KPI summary
+const warningServices = catalog.filter(s => WARNING_LEVELS.has(s.gdpr_maturity));
+const paidServices = catalog.filter(s => ["paid", "usage_based"].includes(s.pricing_model));
+```
+
+<div style="display:flex; gap:1.5rem; flex-wrap:wrap; margin-bottom:2rem;">
+  <div style="background:#fef2f2; border:1px solid #fca5a5; border-radius:8px; padding:1rem 1.5rem; min-width:160px;">
+    <div style="font-size:2rem; font-weight:700; color:#dc2626;">${gdprReport.warning_count}</div>
+    <div style="font-size:0.85rem; color:#6b7280;">GDPR warnings</div>
+  </div>
+  <div style="background:#f0fdf4; border:1px solid #86efac; border-radius:8px; padding:1rem 1.5rem; min-width:160px;">
+    <div style="font-size:2rem; font-weight:700; color:#16a34a;">${gdprReport.total_services}</div>
+    <div style="font-size:0.85rem; color:#6b7280;">Services in catalog</div>
+  </div>
+  <div style="background:#fffbeb; border:1px solid #fcd34d; border-radius:8px; padding:1rem 1.5rem; min-width:160px;">
+    <div style="font-size:2rem; font-weight:700; color:#b45309;">${paidServices.length}</div>
+    <div style="font-size:0.85rem; color:#6b7280;">Paid / usage-based</div>
+  </div>
+</div>
+
+---
+
+## Service Catalog
+
+```js
+import {html} from "npm:htl";
+
+function maturityBadge(m) {
+  const color = maturityColor[m] || "#9ca3af";
+  const label = maturityLabel[m] || m;
+  return html`<span style="background:${color}20; color:${color}; border:1px solid ${color}60;
+    border-radius:4px; padding:2px 8px; font-size:0.78rem; font-weight:600; white-space:nowrap;">${label}</span>`;
+}
+
+function pricingBadge(p) {
+  const colors = { paid: "#7c3aed", usage_based: "#7c3aed", freemium: "#0369a1", free: "#166534", unknown: "#6b7280" };
+  const c = colors[p] || "#6b7280";
+  return html`<span style="color:${c}; font-size:0.78rem; font-weight:500;">${p.replace("_", " ")}</span>`;
+}
+
+const catalogTable = html`<table style="width:100%; border-collapse:collapse; font-size:0.9rem;">
+  <thead>
+    <tr style="border-bottom:2px solid #e5e7eb;">
+      <th style="text-align:left; padding:8px 12px;">Service</th>
+      <th style="text-align:left; padding:8px 12px;">Provider</th>
+      <th style="text-align:left; padding:8px 12px;">Category</th>
+      <th style="text-align:left; padding:8px 12px;">Pricing</th>
+      <th style="text-align:left; padding:8px 12px;">GDPR Maturity</th>
+      <th style="text-align:left; padding:8px 12px;">DPA</th>
+    </tr>
+  </thead>
+  <tbody>
+    ${catalog.map(s => html`<tr style="border-bottom:1px solid #f3f4f6; ${WARNING_LEVELS.has(s.gdpr_maturity) ? 'background:#fff7f7;' : ''}">
+      <td style="padding:8px 12px; font-weight:500;">
+        ${s.website_url
+          ? html`<a href="${s.website_url}" target="_blank" style="color:#1d4ed8;">${s.name}</a>`
+          : s.name}
+      </td>
+      <td style="padding:8px 12px; color:#6b7280;">${s.provider || "—"}</td>
+      <td style="padding:8px 12px; color:#6b7280;">${s.category || "—"}</td>
+      <td style="padding:8px 12px;">${pricingBadge(s.pricing_model)}</td>
+      <td style="padding:8px 12px;">${maturityBadge(s.gdpr_maturity)}</td>
+      <td style="padding:8px 12px;">${s.dpa_available ? "✅" : "❌"}</td>
+    </tr>`)}
+  </tbody>
+</table>`;
+display(catalogTable);
+```
+
+---
+
+## GDPR Warnings
+
+_Services at **Unknown**, **Non-Compliant**, or **Initial** maturity may limit use in GDPR-regulated or corporate environments._
+
+```js
+if (gdprReport.warnings.length === 0) {
+  display(html`<p style="color:#16a34a;">✅ No GDPR warnings across active repos.</p>`);
+} else {
+  const warningCards = html`<div style="display:flex; flex-direction:column; gap:0.75rem;">
+    ${gdprReport.warnings.map(w => {
+      const color = maturityColor[w.gdpr_maturity] || "#ef4444";
+      return html`<div style="border-left:4px solid ${color}; background:${color}10; padding:0.75rem 1rem; border-radius:4px;">
+        <div style="font-weight:600;">${w.service_slug}
+          <span style="font-weight:400; color:#6b7280; margin-left:0.5rem;">in ${w.repo_slug || "unknown repo"}</span>
+        </div>
+        <div style="font-size:0.85rem; margin-top:2px;">
+          ${maturityBadge(w.gdpr_maturity)}
+          ${w.pricing_model ? html`&nbsp;${pricingBadge(w.pricing_model)}` : ""}
+          ${w.purpose ? html`<span style="color:#6b7280; margin-left:0.5rem;">— ${w.purpose}</span>` : ""}
+        </div>
+      </div>`;
+    })}
+  </div>`;
+  display(warningCards);
+}
+```
+
+---
+
+## Per-Repo Breakdown
+
+```js
+// Build: latest snapshot per repo → service list
+const repoBreakdown = new Map();
+for (const snap of snapshots) {
+  const repoSlug = snap.repo_id || "unknown";
+  if (!repoBreakdown.has(repoSlug) || snap.snapshot_at > repoBreakdown.get(repoSlug).snapshot_at) {
+    repoBreakdown.set(repoSlug, snap);
+  }
+}
+
+// Enrich with catalog data
+const catalogBySlug = Object.fromEntries(catalog.map(s => [s.slug, s]));
+
+const repoTable = html`<table style="width:100%; border-collapse:collapse; font-size:0.9rem;">
+  <thead>
+    <tr style="border-bottom:2px solid #e5e7eb;">
+      <th style="text-align:left; padding:8px 12px;">Repo</th>
+      <th style="text-align:left; padding:8px 12px;">Services</th>
+      <th style="text-align:left; padding:8px 12px;">Ingested</th>
+    </tr>
+  </thead>
+  <tbody>
+    ${[...repoBreakdown.entries()].map(([repoSlug, snap]) => html`<tr style="border-bottom:1px solid #f3f4f6;">
+      <td style="padding:8px 12px; font-weight:500;">${repoSlug}</td>
+      <td style="padding:8px 12px;">
+        ${snap.entries.map(e => {
+          const cat = catalogBySlug[e.service_slug];
+          const m = cat?.gdpr_maturity || "unknown";
+          const color = maturityColor[m] || "#9ca3af";
+          return html`<span style="display:inline-flex; align-items:center; gap:4px; margin:2px 4px 2px 0;
+            background:${color}15; border:1px solid ${color}50; border-radius:4px; padding:2px 8px; font-size:0.8rem;">
+            <span style="width:8px; height:8px; border-radius:50%; background:${color}; display:inline-block;"></span>
+            ${e.service_slug}
+          </span>`;
+        })}
+      </td>
+      <td style="padding:8px 12px; color:#9ca3af; font-size:0.8rem;">${new Date(snap.snapshot_at).toLocaleDateString()}</td>
+    </tr>`)}
+  </tbody>
+</table>`;
+display(repoTable);
+```