Complete WARDEN-WP-0012 routing scenario playbooks

Add platform-secret playbooks for issue-core ingestion, OpenRouter llm-connect, object-storage STS, and database dynamic credentials. Extend the routing catalog with draft entries and implement `warden route list --stale` for quarterly drift review. Document the review cadence in AccessRouting and mark the workplan finished.
2026-06-25 10:27:23 +02:00
parent 318f2558f5
commit 1237cc767b
12 changed files with 720 additions and 30 deletions
--- a/wiki/AccessRouting.md
+++ b/wiki/AccessRouting.md
@@ -65,9 +65,10 @@ OpenBao, flex-auth, key-cape, or any other subsystem, and never returns secret
 material.

 ```bash
-warden route list [--json] [--all] [--tag <keyword>]   # active-only unless --all
-warden route show <id> [--json]                         # owner + pointers; SSH adds steps
-warden route find "<free text need>" [--json] [--all]   # rank by keyword overlap
+warden route list [--json] [--all] [--tag <keyword>]              # active-only unless --all
+warden route list --stale [--stale-days 90] [--all] [--json]    # past review cadence
+warden route show <id> [--json]                                 # owner + pointers; SSH adds steps
+warden route find "<free text need>" [--json] [--all]             # rank by keyword overlap
 ```

 Agent-oriented examples:
@@ -113,6 +114,46 @@ Report drift via a custodian workplan or a State Hub message to `ops-warden`.

 ---

+## Drift review cadence
+
+Every catalog entry carries a `reviewed:` date (`YYYY-MM-DD`) — the last time an
+ops-warden steward confirmed the pointer still matches net-kingdom canon and the
+owner repo's shipped path.
+
+| Cadence | Action |
+| --- | --- |
+| **Quarterly** (default 90 days) | Run `warden route list --stale` — reconcile every listed entry against canon |
+| **On canon change** | When net-kingdom security docs change, review affected `canon_ref` entries immediately |
+| **On owner ship** | When an owning repo merges a new OpenBao path or playbook, promote `draft` → `active` and bump `reviewed` |
+| **On agent confusion** | If `warden route find` misses a common query, add `need_keywords` or a playbook — do not restate owner procedure in the catalog |
+
+### Stale check (operators and agents)
+
+```bash
+# Entries not reviewed in the last 90 days (default threshold)
+warden route list --stale
+
+# Include draft scenarios in the stale report
+warden route list --stale --all
+
+# Custom threshold (e.g. monthly review)
+warden route list --stale --stale-days 30 --json
+```
+
+For each stale entry:
+
+1. Open `canon_ref` in net-kingdom — confirm ownership and vocabulary unchanged.
+2. Open `wiki_ref` in this repo — update the playbook section if canon moved.
+3. Confirm the owner path still exists (anti-stale rule: unshipped paths stay `draft`).
+4. Bump `reviewed:` in `registry/routing/catalog.yaml` to today's date.
+5. Run `uv run pytest tests/test_routing.py` — anchor resolution must still pass.
+
+CI enforces structural drift (every `wiki_ref` anchor resolves; no-double-source
+rule). The quarterly cadence catches **semantic** drift CI cannot detect — canon
+moved but anchors still resolve.
+
+---
+
 ## See also

 - `CredentialRouting.md` — worker decision tree and routing table
--- a/wiki/CredentialRouting.md
+++ b/wiki/CredentialRouting.md
@@ -87,6 +87,16 @@ executes.
 | `ops-bridge-tunnel` | "ops-bridge owns transport — supply a `cert_command`" | Open the tunnel with ops-bridge |
 | `railiance-infra-principals` | "railiance-infra deploys host principals" | Run the infra Ansible |
 | `activity-core-issue-sink` | "activity-core + issue-core own emission — pair `ISSUE_CORE_*` env vars" | See `wiki/playbooks/activity-core-issue-sink.md` |
+| `inter-hub-bootstrap-ssh` | "Inter-Hub bootstrap SSH envelope — attended vs unattended branches" | See `wiki/InterHubBootstrapAccessLane.md` |
+
+**Draft** (hidden from default lookup until owner path ships — `warden route list --all`):
+
+| Catalog `id` | Routing focus | Playbook |
+| --- | --- | --- |
+| `issue-core-ingestion-api-key` | OpenBao KV + ESO for `ISSUE_CORE_API_KEY` | `wiki/playbooks/issue-core-ingestion-api-key.md` |
+| `openrouter-llm-connect` | OpenRouter key → `llm-connect` in activity-core | `wiki/playbooks/openrouter-llm-connect.md` |
+| `object-storage-sts` | NK-WP-0007 STS vending path | `wiki/playbooks/object-storage-sts.md` |
+| `database-dynamic-credentials` | OpenBao database secrets engine | `wiki/playbooks/database-dynamic-credentials.md` |

 ops-warden answers *where + who*; the worker acts on the owning system. ops-warden
 never performs the non-SSH step on the worker's behalf.
--- a/wiki/playbooks/database-dynamic-credentials.md
+++ b/wiki/playbooks/database-dynamic-credentials.md
@@ -0,0 +1,102 @@
+# Database Dynamic Credentials — OpenBao
+
+Date: 2026-06-24  
+Workplan: WARDEN-WP-0012 T4  
+Catalog: `database-dynamic-credentials` (draft until engine ships)
+
+Pointer playbook for short-lived database passwords issued by OpenBao dynamic
+secret engines (e.g. CNPG-managed PostgreSQL). ops-warden does not issue DB
+credentials — custody and engine configuration belong to `railiance-platform`;
+consumers request credentials through approved paths after flex-auth policy where
+required.
+
+---
+
+## Owners
+
+| Concern | Owner repo | Authoritative doc |
+| --- | --- | --- |
+| OpenBao database engine, paths, policies | `railiance-platform` | `docs/openbao.md`, `workplans/RAIL-PL-WP-0002-openbao-platform-secrets-service.md` |
+| Authorization before sensitive reads | `flex-auth` | `INTENT.md` |
+| Application connection and lease handling | Owning app repo | App-specific deployment docs |
+
+---
+
+## Do not ask ops-warden
+
+```bash
+warden route show openbao-api-key --json
+warden route show database-dynamic-credentials --json   # after promotion
+```
+
+Never paste DB passwords, connection strings with credentials, or root DB admin
+tokens in Git, State Hub, logs, or agent chat.
+
+---
+
+## Platform path convention
+
+From `railiance-platform/docs/openbao.md`:
+
+```text
+platform/databases/<consumer>
+```
+
+Dynamic credentials are issued via OpenBao database secrets engine roles — not
+static KV copies. Coordinate the exact mount and role name with platform before
+wiring workloads.
+
+**Promotion gate:** catalog entry stays `status: draft` until the database
+secrets engine and consumer role exist in the live cluster.
+
+---
+
+## Worker checklist
+
+### 1. Confirm need type
+
+- [ ] Short-lived DB password (dynamic) vs long-lived KV secret — prefer dynamic
+- [ ] Target database identified (CNPG cluster, service name, database name)
+- [ ] flex-auth policy requires approval for this read (if tenant policy says so)
+
+### 2. Platform provisioning (operator)
+
+- [ ] Database secrets engine configured with least-privilege creation statements
+- [ ] Role TTL aligned to workload session (minutes–hours, not days)
+- [ ] Path registered under `platform/databases/<consumer>`
+- [ ] Audit logging enabled on secret access
+
+### 3. Workload consumption
+
+- [ ] App uses ESO or CSI to materialize username/password into K8s Secret
+- [ ] Connection pool handles credential rotation before lease expiry
+- [ ] No hard-coded passwords in Helm values or ConfigMaps
+
+### 4. Verify
+
+- [ ] App connects with issued credentials
+- [ ] Lease renewal or re-read succeeds before expiry
+- [ ] Revocation on pod teardown (if policy requires)
+
+### 5. Rotation / revocation
+
+- [ ] OpenBao revokes lease on role change
+- [ ] Platform operator documents break-glass DB admin path separately (not via warden)
+
+---
+
+## Owner-repo next actions
+
+| Repo | Action |
+| --- | --- |
+| `railiance-platform` | Configure database secrets engine, roles, and policies |
+| Owning application | Wire ESO/CSI and connection handling for lease TTL |
+| `flex-auth` | Policy for database credential requests (if gated) |
+
+---
+
+## See also
+
+- `railiance-platform/docs/openbao.md`
+- `railiance-platform/workplans/RAIL-PL-WP-0002-openbao-platform-secrets-service.md`
+- `wiki/CredentialRouting.md#routing-table`
--- a/wiki/playbooks/issue-core-ingestion-api-key.md
+++ b/wiki/playbooks/issue-core-ingestion-api-key.md
@@ -0,0 +1,122 @@
+# issue-core Ingestion API Key — OpenBao Custody
+
+Date: 2026-06-24  
+Workplan: WARDEN-WP-0012 T1  
+Catalog: `issue-core-ingestion-api-key` (draft until path ships)
+
+Pointer playbook for agents and operators wiring the **shared ingestion key**
+between `activity-core` IssueSink emission and `issue-core` REST ingestion.
+ops-warden does not vend this key — custody belongs to `railiance-platform`
+(OpenBao) and the consuming workloads.
+
+---
+
+## Owners
+
+| Concern | Owner repo | Authoritative doc |
+| --- | --- | --- |
+| OpenBao path, ESO delivery, rotation ceremony | `railiance-platform` | `docs/argocd-gitops.md` — OpenBao path convention |
+| Ingestion server (`POST /issues/`) | `issue-core` | `README.md` — REST Ingestion Server |
+| IssueSink consumer | `activity-core` | `docs/issue-core-emission-boundary.md` |
+| Emission pairing checklist | `ops-warden` | `wiki/playbooks/activity-core-issue-sink.md` |
+
+---
+
+## Do not ask ops-warden
+
+`ISSUE_CORE_API_KEY` is not an SSH certificate. Generic API-key routing:
+
+```bash
+warden route show openbao-api-key --json
+warden route show activity-core-issue-sink --json
+```
+
+Never paste key values into Git, State Hub, workplans, logs, or agent chat.
+
+---
+
+## Canonical OpenBao path (expected)
+
+Coordinate with `railiance-platform` before writing secrets. Documented custody
+shape:
+
+```text
+platform/workloads/issue-core/issue-core/issue-core-runtime
+```
+
+Expected properties (names only — no values):
+
+```text
+ISSUE_CORE_API_KEY
+GITEA_BACKEND_TOKEN
+```
+
+The ExternalSecret manifest belongs in `issue-core` workload manifests (tenant
+repo owns runtime deployment). Platform owns mount policy and path provisioning.
+
+**Promotion gate:** catalog entry stays `status: draft` until this path exists
+in the live OpenBao cluster and an owner-repo ExternalSecret is merged.
+
+---
+
+## Worker checklist
+
+### 1. Confirm path with platform owner
+
+- [ ] Path exists: `platform/workloads/issue-core/issue-core/issue-core-runtime`
+- [ ] KV policy allows `issue-core` service account read (workload-kv-read template)
+- [ ] `railiance-platform` workplan records the canonical path (no forked conventions)
+
+### 2. External Secrets Operator pattern
+
+Prefer ESO for values that become Kubernetes Secrets consumed by Helm charts
+(`railiance-platform/docs/openbao.md`, `docs/argocd-gitops.md`):
+
+- [ ] `ExternalSecret` in `issue-core` namespace targets the path above
+- [ ] Secret keys map to `ISSUE_CORE_API_KEY` (and `GITEA_BACKEND_TOKEN` if used)
+- [ ] `activity-core` deployment receives the **same** key value via its own
+      ExternalSecret (paired env vars — see activity-core-issue-sink playbook)
+- [ ] Do not use the OpenBao injector in the current deployment
+
+### 3. Local dev (no OpenBao)
+
+Generate once and export on both processes — not for production:
+
+```bash
+export ISSUE_CORE_API_KEY="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
+```
+
+See `wiki/playbooks/activity-core-issue-sink.md#worker-checklist` for pairing steps.
+
+### 4. Rotation
+
+- [ ] Generate new key in OpenBao (platform operator ceremony)
+- [ ] Update both `issue-core` and `activity-core` Secrets before revoking old value
+- [ ] Verify one live POST returns `201` with `issue_id`
+- [ ] Record rotation in platform audit log — not in git
+
+### 5. Privileged read policy
+
+Break-glass and operator reads follow `railiance-platform/docs/openbao.md` —
+scoped tokens only, never root token for routine workload secret inspection.
+
+---
+
+## Owner-repo next actions
+
+| Repo | Action |
+| --- | --- |
+| `railiance-platform` | Provision KV path, policy, and document in OpenBao runbook |
+| `issue-core` | Merge ExternalSecret + Deployment env from synced Secret |
+| `activity-core` | Mirror `ISSUE_CORE_API_KEY` injection for REST sink mode |
+
+When the path ships, ops-warden promotes `issue-core-ingestion-api-key` to
+`status: active` with this `wiki_ref`.
+
+---
+
+## See also
+
+- `wiki/playbooks/activity-core-issue-sink.md`
+- `railiance-platform/docs/argocd-gitops.md`
+- `warden route show issue-core-ingestion-api-key --all --json`
--- a/wiki/playbooks/object-storage-sts.md
+++ b/wiki/playbooks/object-storage-sts.md
@@ -0,0 +1,123 @@
+# Object-Storage STS Credential Vending
+
+Date: 2026-06-24  
+Workplan: WARDEN-WP-0012 T4  
+Catalog: `object-storage-sts` (draft until vending path ships)
+
+Pointer playbook for short-lived S3-compatible credentials. NetKingdom canon
+defines the pattern; `flex-auth` decides, OpenBao brokers, `railiance-platform`
+configures backends, and consumers (e.g. `artifact-store`) refresh credentials.
+
+ops-warden does not vend object-storage credentials.
+
+---
+
+## Owners
+
+| Concern | Owner repo | Authoritative doc |
+| --- | --- | --- |
+| Architecture and trust boundaries | `net-kingdom` | `docs/object-storage-sts-credential-vending.md` |
+| Policy decision (may this principal access bucket/prefix?) | `flex-auth` | `INTENT.md` |
+| OpenBao broker config, audit, bootstrap parent creds | `railiance-platform` | `docs/openbao.md` — Artifact-Store handoff |
+| S3 client refresh and package behavior | `artifact-store` | `ARTIFACT-STORE-WP-0007` |
+
+---
+
+## Do not ask ops-warden
+
+```bash
+warden route show openbao-api-key --json
+warden route show object-storage-sts --json   # after promotion
+```
+
+Never paste access keys, session tokens, or parent credentials in Git, State Hub,
+logs, or agent chat.
+
+---
+
+## Core flow (pointer only)
+
+Full procedure is in net-kingdom canon. Summary for routing:
+
+```text
+Principal (human/service/agent)
+  → IAM Profile token (key-cape / Keycloak)
+  → credential-vending service
+  → flex-auth decision (tenant, bucket, prefix, actions, TTL)
+  → backend exchange (STS / OpenBao-assisted broker)
+  → temporary S3 credentials → consumer
+```
+
+OpenBao is runtime secret infrastructure — not the canonical authorization engine.
+
+---
+
+## Platform path conventions
+
+From `railiance-platform/docs/openbao.md`:
+
+```text
+platform/object-storage/<consumer>
+```
+
+Example bootstrap bridge (static key, pre-STS):
+
+```text
+platform/object-storage/artifact-store
+```
+
+STS vending remains governed by NK-WP-0007 / `ARTIFACT-STORE-WP-0007`. Promote
+catalog entry to `active` only when the approved vending path for your consumer
+exists in live OpenBao policy and canon.
+
+---
+
+## Worker checklist
+
+### 1. Confirm consumer and canon
+
+- [ ] Read `net-kingdom/docs/object-storage-sts-credential-vending.md`
+- [ ] Identify `protected_system_id` (e.g. `object-storage:artifact-store-prod`)
+- [ ] Confirm flex-auth policy package for your tenant/resource
+
+### 2. Authorization before secret read
+
+- [ ] Obtain IAM Profile token with required claims
+- [ ] flex-auth returns allow + obligations (TTL, prefix scope, actions)
+- [ ] Do not skip flex-auth and read parent credentials from OpenBao directly
+
+### 3. Credential delivery
+
+- [ ] Platform provisions broker config under `platform/object-storage/...`
+- [ ] Consumer receives credentials via approved delivery (ESO, CSI, sidecar)
+- [ ] For `artifact-store`: configure `ARTIFACTSTORE_S3_*_REF` file/env refs
+
+### 4. Verify
+
+```bash
+artifactstore storage verify --backend s3
+```
+
+### 5. Rotation / expiry
+
+- [ ] Prefer lease expiry and dynamic regeneration over long-lived keys
+- [ ] Consumer must support session-token refresh or sidecar refresh (see canon gap notes)
+
+---
+
+## Owner-repo next actions
+
+| Repo | Action |
+| --- | --- |
+| `net-kingdom` | Maintain STS vending canon; NK-WP-0007 decisions |
+| `flex-auth` | Policy packages for object-storage resources |
+| `railiance-platform` | Backend parent creds, OpenBao mounts, audit |
+| `artifact-store` | S3 backend refresh behavior and verify smoke |
+
+---
+
+## See also
+
+- `net-kingdom/docs/object-storage-sts-credential-vending.md`
+- `railiance-platform/docs/openbao.md#artifact-store-object-storage-handoff`
+- `wiki/CredentialRouting.md#quick-decision-tree`
--- a/wiki/playbooks/openrouter-llm-connect.md
+++ b/wiki/playbooks/openrouter-llm-connect.md
@@ -0,0 +1,104 @@
+# OpenRouter API Key — llm-connect in activity-core
+
+Date: 2026-06-24  
+Workplan: WARDEN-WP-0012 T4  
+Catalog: `openrouter-llm-connect` (draft until OpenBao path ships)
+
+Pointer playbook for LLM provider credentials consumed by `llm-connect` in the
+`activity-core` namespace. ops-warden issues SSH certs only — API keys are an
+OpenBao → Kubernetes Secret action owned by `railiance-platform` and
+`activity-core` deployment repos.
+
+---
+
+## Owners
+
+| Concern | Owner repo | Authoritative doc |
+| --- | --- | --- |
+| OpenBao path and ESO delivery | `railiance-platform` | `docs/openbao.md` — path convention |
+| llm-connect K8s overlay and smoke | `llm-connect` | `deploy/k8s/activity-core-llm-connect/README.md` |
+| activity-core runtime config (`LLM_CONNECT_URL`) | `activity-core` | `llm-connect/docs/activity-core-llm-endpoint.md` |
+
+---
+
+## Do not ask ops-warden
+
+```bash
+warden route show openbao-api-key --json
+warden route show openrouter-llm-connect --json   # after promotion
+```
+
+`OPENROUTER_API_KEY` must not appear in Git, State Hub, workplans, logs, or chat.
+
+---
+
+## Expected custody shape
+
+Documented platform path convention (coordinate before writing secrets):
+
+```text
+platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets
+```
+
+Property name: `OPENROUTER_API_KEY`
+
+Until the OpenBao path is provisioned, operators may create the K8s Secret
+directly for pilot smoke (`llm-connect` README) — that is a bootstrap bridge,
+not the long-term custody model.
+
+**Promotion gate:** catalog entry stays `status: draft` until the OpenBao path
+exists and ESO (or approved equivalent) delivers the Secret in cluster.
+
+---
+
+## Worker checklist
+
+### 1. Confirm need
+
+- [ ] Consumer is `llm-connect` in `activity-core` namespace (not a generic OpenRouter client)
+- [ ] Default profile uses `provider=openrouter` (`llm-connect/docs/activity-core-llm-endpoint.md`)
+- [ ] flex-auth policy applies if your tenant requires pre-approval for secret reads
+
+### 2. Platform path (production)
+
+- [ ] Path provisioned under `platform/workloads/activity-core/...`
+- [ ] Workload KV read policy scoped to `llm-connect` service account
+- [ ] ExternalSecret syncs to Secret `llm-connect-provider-secrets`
+
+### 3. Deployment wiring
+
+- [ ] `kubectl apply -k deploy/k8s/activity-core-llm-connect` (llm-connect repo)
+- [ ] Deployment mounts provider Secret; env provides `OPENROUTER_API_KEY`
+- [ ] activity-core sets `LLM_CONNECT_URL` to in-cluster service URL
+
+### 4. Smoke
+
+```bash
+# From llm-connect repo — cluster smoke after apply
+kubectl -n activity-core rollout status deployment/llm-connect
+# See deploy/k8s/activity-core-llm-connect/README.md for endpoint smoke script
+```
+
+### 5. Rotation
+
+- [ ] Update OpenBao KV value
+- [ ] ESO refresh or rollout restart llm-connect Deployment
+- [ ] Run cluster smoke; confirm activity-core triage profile still reaches provider
+
+---
+
+## Owner-repo next actions
+
+| Repo | Action |
+| --- | --- |
+| `railiance-platform` | Provision OpenBao path + policy for activity-core llm-connect |
+| `llm-connect` | Maintain K8s overlay and document Secret key names |
+| `activity-core` | Set `LLM_CONNECT_URL` and triage profile after llm-connect is live |
+
+---
+
+## See also
+
+- `llm-connect/docs/activity-core-llm-endpoint.md`
+- `wiki/CredentialRouting.md#examples-do-not-ask-ops-warden`
+- `net-kingdom/docs/platform-identity-security-architecture.md`