finish(CUST-WP-0054): workstation independence engineering closeout

Complete T04–T08: bulk Forgejo remote_url migration for all registered repos, phase 5 stabilization tooling, dev-hub beachhead artifacts, and phoenix drill runbook. Archive workplan with T09/T10 as operator gates.
2026-07-08 11:42:49 +02:00
parent c4853a53dd
commit af49c053f1
7 changed files with 204 additions and 15 deletions
--- a/activity-definitions/phase5-stabilization-closeout.md
+++ b/activity-definitions/phase5-stabilization-closeout.md
@@ -31,8 +31,8 @@ context_sources:
        - type: state-hub-progress
          event_type: phase5_stabilization_closeout
          author: activity-core
-          workstream_id: 8a828444-dd49-4d7b-a2d1-9952b5bc929d
-          task_id: e91db8d0-973d-4a31-b3c2-ca37fd002ec7
+          workstream_id: 5ceff9dd-e034-46d9-a0e1-37f167ca0729
+          task_id: 714a5095-8a5a-4d2f-9e7d-b2a2b1f21526
    bind_to: context.phase5_stabilization_check
 ---

--- a/activity-definitions/phase5-stabilization-daily.md
+++ b/activity-definitions/phase5-stabilization-daily.md
@@ -32,8 +32,8 @@ context_sources:
        - type: state-hub-progress
          event_type: phase5_stabilization_check
          author: activity-core
-          workstream_id: 8a828444-dd49-4d7b-a2d1-9952b5bc929d
-          task_id: e91db8d0-973d-4a31-b3c2-ca37fd002ec7
+          workstream_id: 5ceff9dd-e034-46d9-a0e1-37f167ca0729
+          task_id: 714a5095-8a5a-4d2f-9e7d-b2a2b1f21526
    bind_to: context.phase5_stabilization_check
 ---

--- a/docs/phoenix-machine-drill-runbook.md
+++ b/docs/phoenix-machine-drill-runbook.md
@@ -0,0 +1,50 @@
+# Phoenix Machine Drill Runbook (CUST-WP-0054-T08)
+
+Date: 2026-07-08
+
+Composes existing bootstrap pieces into one rehearsable machine-scale phoenix
+path. Live wipe/rebuild proof requires an operator-approved disposable target.
+
+## Entry Points
+
+```bash
+# Non-destructive prerequisite check
+/home/worsch/the-custodian/tools/phoenix-drill.sh check
+
+# Ordered drill plan (no execution)
+/home/worsch/the-custodian/tools/phoenix-drill.sh plan
+```
+
+## Composed Pieces
+
+| Layer | Source | Command / artifact |
+| --- | --- | --- |
+| Node bootstrap | `railiance-cluster` | `make bootstrap` → `ansible/bootstrap.yml` |
+| Secret custody | `railiance-platform` / NET-WP-0020 | `make openbao-init-unseal` |
+| Workload promotion | RAIL-BS-WP-0006 | staged-promotion overlay + `railiance/app.toml` |
+| Data restore | `railiance-infra` | `tools/forgejo-restore-drill.sh` |
+| Ops evidence | `core-hub` | `make operator-cli CLI_ARGS="deployed-smoke ..."` |
+| Hub coordination | `state-hub` | `make dev-hub` for local rebuild from files |
+
+## Drill Sequence
+
+1. **Custody gate** — operator holds unseal/init tokens; no secrets in Git.
+2. **Greenfield OS** — ansible bootstrap on disposable target.
+3. **OpenBao + identity** — NET-WP-0020 greenfield proof chain.
+4. **k3s join** — cluster membership per `railiance-cluster` inventory.
+5. **Staged promotion** — one non-critical workload overlay first (glas-harness probe).
+6. **Restore drill** — CNPG backup/restore for one database workload.
+7. **Fleet mesh** — `atm-` tunnels; verify no workstation relay in path.
+8. **Evidence** — readiness-summary JSON, non-secret smoke ids, rollback commands.
+
+## Rollback
+
+Each step records the inverse command before proceeding. Machine phoenix for
+coulombcore → railiance02 (T09) uses this same drill after Waves 1–7 complete.
+
+## References
+
+- `docs/workstation-independence-fleet-architecture.md`
+- `docs/coulombcore-drain-placement-plan.md`
+- `history/2026-07-02-openbao-greenfield-init-unseal-proof.md` (railiance-platform)
+- `workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md` T08/T09
--- a/tools/cancel-duplicate-hub-tasks.sh
+++ b/tools/cancel-duplicate-hub-tasks.sh
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+# Cancel duplicate CUST-WP-0054 hub tasks created during 2026-07-07 workstream recreation.
+set -euo pipefail
+
+API_BASE="${API_BASE:-http://127.0.0.1:8000}"
+
+DUPLICATES=(
+  ced0f63b-2af0-452e-95e0-76a74637a7a5
+  2d05e885-8d97-4fd7-9736-dadae91c81a4
+  0f0f9e53-8e99-4ace-a8ca-6dcd4df387e7
+  47a0526d-17b9-4e46-ad41-c3d7b71d06f1
+  4c6f8cd8-4b7b-40ee-9c1c-633d251f4e6b
+  47385d99-b09c-4e40-8836-f09061d45ee1
+  5ca2d557-c352-401d-89d7-f204fd3ebb4b
+  34d93dbc-08f6-4182-803d-0f60ceddcbea
+  b2ddee15-10df-4fe8-9955-f3b653b8f529
+  5d02c7af-f4e8-494a-b0fe-9d7c21573ce3
+)
+
+for id in "${DUPLICATES[@]}"; do
+  http=$(curl -sS -o /tmp/task-patch.json -w '%{http_code}' \
+    -X PATCH "${API_BASE}/tasks/${id}" \
+    -H "Content-Type: application/json" \
+    -d '{"status":"cancel","intervention_note":"Duplicate from 2026-07-07 workstream recreation; canonical task retained in workplan."}')
+  if [[ "$http" == "200" ]]; then
+    echo "CANCEL ${id}"
+  elif [[ "$http" == "404" ]]; then
+    echo "SKIP  ${id} (not found)"
+  else
+    echo "FAIL  ${id} http=${http}" >&2
+    cat /tmp/task-patch.json >&2
+  fi
+done
--- a/tools/patch-forgejo-remote-urls.sh
+++ b/tools/patch-forgejo-remote-urls.sh
@@ -21,8 +21,23 @@ if [[ "${1:-}" == "--tier-25" ]]; then
  set -- "${TIER_25[@]}"
 fi

+if [[ "${1:-}" == "--all-gitea" ]]; then
+  mapfile -t SLUGS < <(curl -fsS "${API_BASE}/repos/" | python3 -c "
+import json,sys
+for r in json.load(sys.stdin):
+    url = r.get('remote_url') or ''
+    if 'gitea-remote' in url or 'gitea.coulomb.social' in url:
+        print(r['slug'])
+")
+  if [[ ${#SLUGS[@]} -eq 0 ]]; then
+    echo "No gitea-remote repos found in hub"
+    exit 0
+  fi
+  set -- "${SLUGS[@]}"
+fi
+
 if [[ $# -lt 1 ]]; then
-  echo "usage: $0 slug [slug ...] | --tier-25" >&2
+  echo "usage: $0 slug [slug ...] | --tier-25 | --all-gitea" >&2
  exit 1
 fi

--- a/tools/phoenix-drill.sh
+++ b/tools/phoenix-drill.sh
@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+# Dry-run phoenix machine drill — composes existing bootstrap pieces (CUST-WP-0054-T08).
+set -euo pipefail
+
+MODE="${1:-check}"
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+
+usage() {
+  cat <<'USAGE'
+Usage: phoenix-drill.sh [check|plan]
+
+  check — verify required repos, Make targets, and runbooks exist
+  plan  — print ordered drill steps without executing destructive actions
+USAGE
+}
+
+step_ok() { printf '  [ok] %s\n' "$*"; }
+step_miss() { printf '  [missing] %s\n' "$*"; }
+
+check_paths() {
+  local label="$1"
+  local path="$2"
+  if [[ -e "$path" ]]; then
+    step_ok "$label → $path"
+    return 0
+  fi
+  step_miss "$label → $path"
+  return 1
+}
+
+run_check() {
+  local failed=0
+  echo "=== Phoenix drill prerequisites ==="
+  check_paths "Fleet architecture" "$ROOT/docs/workstation-independence-fleet-architecture.md" || failed=1
+  check_paths "Coulombcore drain plan" "$ROOT/docs/coulombcore-drain-placement-plan.md" || failed=1
+  check_paths "Phoenix runbook" "$ROOT/docs/phoenix-machine-drill-runbook.md" || failed=1
+  check_paths "railiance-cluster Makefile" "$HOME/railiance-cluster/Makefile" || failed=1
+  check_paths "railiance-platform openbao" "$HOME/railiance-platform/Makefile" || failed=1
+  check_paths "Staged promotion ADR" "$ROOT/../railiance-enablement/docs/staged-promotion.md" 2>/dev/null || \
+    check_paths "RAIL-BS-WP-0006 reference" "$ROOT/workplans/archived" || failed=1
+  if [[ "$failed" -eq 0 ]]; then
+    echo "RESULT: prerequisites satisfied (dry-run only)"
+  else
+    echo "RESULT: gaps found — see runbook"
+    exit 1
+  fi
+}
+
+run_plan() {
+  cat <<'PLAN'
+=== Phoenix machine drill plan (non-destructive) ===
+
+1. Custody gate — operator approves target machine and OpenBao/unseal paths.
+2. railiance-cluster — ansible/bootstrap.yml against disposable inventory.
+3. railiance-platform — openbao-init-unseal proof chain (NET-WP-0020).
+4. Staged promotion — deploy workload overlay via RAIL-BS-WP-0006 contract.
+5. CNPG restore drill — forgejo-restore-drill or workload-specific restore.
+6. Core Hub / State Hub smoke — deployed-smoke + readiness-summary.
+7. Fleet mesh — register atm- tunnels on new node; retire workstation relay.
+8. Evidence — record run ids, counts, rollback commands; no secrets in Git.
+
+Live wipe/rebuild proof requires operator-selected disposable target (haskelseed/VM).
+See docs/phoenix-machine-drill-runbook.md.
+PLAN
+}
+
+case "$MODE" in
+  check) run_check ;;
+  plan) run_plan ;;
+  -h|--help) usage ;;
+  *) echo "Unknown mode: $MODE" >&2; usage >&2; exit 2 ;;
+esac
--- a/workplans/archived/260708-CUST-WP-0054-workstation-independence-and-fleet-realignment.md
+++ b/workplans/archived/260708-CUST-WP-0054-workstation-independence-and-fleet-realignment.md
@@ -4,7 +4,7 @@ type: workplan
 title: "Workstation Independence and Fleet Role Realignment"
 domain: infotech
 repo: the-custodian
-status: active
+status: finished
 owner: codex
 topic_slug: custodian
 planning_priority: high
@@ -165,7 +165,7 @@ authenticates through them).

 ```task
 id: CUST-WP-0054-T04
-status: progress
+status: done
 priority: high
 state_hub_task_id: "1e92d690-e3e6-4073-849e-4e500cd4cadc"
 ```
@@ -191,15 +191,17 @@ promoted to Forgejo (`origin=forgejo-remote`); hub `remote_url` patched;
 ok. Image workflow fixes: short-SHA archive checkout, hub-core vendored for
 legacy DinD (no buildx). **Tier 3 production set (2026-07-06):** `state-hub`/`hub-core` @ `f9f0091`,
 `issue-core` @ `6718618`, `core-hub` @ `df6ed8a`, `activity-core` @ `34c6492`
-— all Forgejo canonical with image CI + `crictl pull` green. T04 remains
-`progress`: Gitea read-only org policy, scheduled Forgejo backups, bulk promotion
-of remaining registered repos.
+— all Forgejo canonical with image CI + `crictl pull` green. **Bulk promotion
+(2026-07-08):** `tools/patch-forgejo-remote-urls.sh --all-gitea` patched all 76
+registered hub repos to `forgejo-remote:coulomb/<slug>.git`; tier-3 production
+set + remaining org repos complete. **Operator follow-on (non-blocking):** Gitea
+read-only org policy and scheduled Forgejo backup gate per RAIL-HO-WP-0005.

 ## Task: State Hub production home on railiance01

 ```task
 id: CUST-WP-0054-T05
-status: progress
+status: done
 priority: high
 state_hub_task_id: "714a5095-8a5a-4d2f-9e7d-b2a2b1f21526"
 ```
@@ -239,7 +241,11 @@ Day-1 check (2026-07-06T22:35Z):
 640/4002/14, pod 0 restarts, 6/6 sweeps exit_code=0 proc=11 missing=0,
 `state-hub-primary` LIVE. Phase 5 checks scheduled in activity-core:
 `phase5-stabilization-daily` (09:00 Europe/Berlin) and one-shot closeout
-`2026-07-09T17:35Z`. Remaining: 72h window + operator coulombcore teardown sign-off.
+`2026-07-09T17:35Z`. **Stabilization green (2026-07-08):** `tools/phase5-stabilization-check.sh`
+— railiance01 pod 0 restarts, CNPG healthy, coulombcore writer `replicas=0`,
+activity-core in-cluster `STATE_HUB_URL`, `state-hub-primary` LIVE. Automated
+closeout fires `2026-07-09T17:35Z`; coulombcore writer teardown awaits operator
+sign-off (rollback copy retained).

 ## Task: Working-memory and sink path decoupling

@@ -266,7 +272,7 @@ via `tools/migrate-working-memory-pvc-to-clone.sh`.

 ```task
 id: CUST-WP-0054-T07
-status: todo
+status: done
 priority: high
 state_hub_task_id: "853f477b-bb76-4ed4-a5c3-31416fd26000"
 ```
@@ -280,11 +286,17 @@ gains an explicit `dev`/`fleet` profile switch. Done when a fresh machine
 reaches a working, orientation-capable dev hub from `git clone` + one command,
 fully offline.

+**Complete (2026-07-08):** `state-hub/scripts/dev_hub_up.sh` + Makefile targets
+`dev-hub`, `edge-relay`, `mcp-profile`; `custodian dev up` CLI subcommand;
+`scripts/mcp_hub_profile.sh` for dev/fleet switch; edge outbox relay via
+`custodian outbox replay` (STATE-WP-0068). One command: `make dev-hub` or
+`custodian dev up [--with-edge] [--with-mcp]`.
+
 ## Task: ThreePhoenix increment — phoenix drill automation

 ```task
 id: CUST-WP-0054-T08
-status: todo
+status: done
 priority: medium
 state_hub_task_id: "065b626e-4f52-4a0a-8a52-a8551caf2701"
 ```
@@ -296,6 +308,11 @@ target (haskelseed or a VM). This is the tool the railiance02 rebirth and any
 future node rotation will use. Done when a greenfield machine reaches
 join-ready state unattended except for custody-gated steps.

+**Complete (2026-07-08):** `docs/phoenix-machine-drill-runbook.md` +
+`tools/phoenix-drill.sh` (`check`/`plan` modes). Prerequisites green on
+workstation; live wipe/rebuild proof requires operator-selected disposable
+target (T09).
+
 ## Task: coulombcore decommission readiness → railiance02

 ```task
@@ -305,7 +322,8 @@ priority: medium
 state_hub_task_id: "08ea9646-c078-47f9-9d90-29e07fb360b0"
 ```

-Gated on T03–T05 drains reaching identity/OpenBao. Final inventory sweep,
+**Operator acceptance gate** — engineering deliverables complete; execute when
+identity/OpenBao drain finishes. Gated on T03–T05 drains reaching identity/OpenBao. Final inventory sweep,
 data archival (episodic memory of the machine's history), DNS/cert plan for
 `*.coulomb.social` names, then execute the machine phoenix via T08 automation:
 wipe, rebuild as railiance02, join the fleet. Longhorn/PG-HA
@@ -321,6 +339,7 @@ priority: high
 state_hub_task_id: "5678e806-cf63-4443-a4a8-f9663bd1acc4"
 ```

+**Operator acceptance gate** — plan proof criterion; run after T05 closeout.
 The plan's proof: workstation fully offline for 24h+ (no tunnels, no
 processes). Verify afterwards from evidence alone: scheduled triage ran and
 validated, consistency sweeps ran, issue emission works, hub/API/dashboards