diff --git a/activity-definitions/phase5-stabilization-closeout.md b/activity-definitions/phase5-stabilization-closeout.md index beb754b..42c11fb 100644 --- a/activity-definitions/phase5-stabilization-closeout.md +++ b/activity-definitions/phase5-stabilization-closeout.md @@ -31,8 +31,8 @@ context_sources: - type: state-hub-progress event_type: phase5_stabilization_closeout author: activity-core - workstream_id: 8a828444-dd49-4d7b-a2d1-9952b5bc929d - task_id: e91db8d0-973d-4a31-b3c2-ca37fd002ec7 + workstream_id: 5ceff9dd-e034-46d9-a0e1-37f167ca0729 + task_id: 714a5095-8a5a-4d2f-9e7d-b2a2b1f21526 bind_to: context.phase5_stabilization_check --- diff --git a/activity-definitions/phase5-stabilization-daily.md b/activity-definitions/phase5-stabilization-daily.md index f201e98..82e699b 100644 --- a/activity-definitions/phase5-stabilization-daily.md +++ b/activity-definitions/phase5-stabilization-daily.md @@ -32,8 +32,8 @@ context_sources: - type: state-hub-progress event_type: phase5_stabilization_check author: activity-core - workstream_id: 8a828444-dd49-4d7b-a2d1-9952b5bc929d - task_id: e91db8d0-973d-4a31-b3c2-ca37fd002ec7 + workstream_id: 5ceff9dd-e034-46d9-a0e1-37f167ca0729 + task_id: 714a5095-8a5a-4d2f-9e7d-b2a2b1f21526 bind_to: context.phase5_stabilization_check --- diff --git a/docs/phoenix-machine-drill-runbook.md b/docs/phoenix-machine-drill-runbook.md new file mode 100644 index 0000000..5e8fc90 --- /dev/null +++ b/docs/phoenix-machine-drill-runbook.md @@ -0,0 +1,50 @@ +# Phoenix Machine Drill Runbook (CUST-WP-0054-T08) + +Date: 2026-07-08 + +Composes existing bootstrap pieces into one rehearsable machine-scale phoenix +path. Live wipe/rebuild proof requires an operator-approved disposable target. + +## Entry Points + +```bash +# Non-destructive prerequisite check +/home/worsch/the-custodian/tools/phoenix-drill.sh check + +# Ordered drill plan (no execution) +/home/worsch/the-custodian/tools/phoenix-drill.sh plan +``` + +## Composed Pieces + +| Layer | Source | Command / artifact | +| --- | --- | --- | +| Node bootstrap | `railiance-cluster` | `make bootstrap` → `ansible/bootstrap.yml` | +| Secret custody | `railiance-platform` / NET-WP-0020 | `make openbao-init-unseal` | +| Workload promotion | RAIL-BS-WP-0006 | staged-promotion overlay + `railiance/app.toml` | +| Data restore | `railiance-infra` | `tools/forgejo-restore-drill.sh` | +| Ops evidence | `core-hub` | `make operator-cli CLI_ARGS="deployed-smoke ..."` | +| Hub coordination | `state-hub` | `make dev-hub` for local rebuild from files | + +## Drill Sequence + +1. **Custody gate** — operator holds unseal/init tokens; no secrets in Git. +2. **Greenfield OS** — ansible bootstrap on disposable target. +3. **OpenBao + identity** — NET-WP-0020 greenfield proof chain. +4. **k3s join** — cluster membership per `railiance-cluster` inventory. +5. **Staged promotion** — one non-critical workload overlay first (glas-harness probe). +6. **Restore drill** — CNPG backup/restore for one database workload. +7. **Fleet mesh** — `atm-` tunnels; verify no workstation relay in path. +8. **Evidence** — readiness-summary JSON, non-secret smoke ids, rollback commands. + +## Rollback + +Each step records the inverse command before proceeding. Machine phoenix for +coulombcore → railiance02 (T09) uses this same drill after Waves 1–7 complete. + +## References + +- `docs/workstation-independence-fleet-architecture.md` +- `docs/coulombcore-drain-placement-plan.md` +- `history/2026-07-02-openbao-greenfield-init-unseal-proof.md` (railiance-platform) +- `workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md` T08/T09 \ No newline at end of file diff --git a/tools/cancel-duplicate-hub-tasks.sh b/tools/cancel-duplicate-hub-tasks.sh new file mode 100755 index 0000000..7748d38 --- /dev/null +++ b/tools/cancel-duplicate-hub-tasks.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Cancel duplicate CUST-WP-0054 hub tasks created during 2026-07-07 workstream recreation. +set -euo pipefail + +API_BASE="${API_BASE:-http://127.0.0.1:8000}" + +DUPLICATES=( + ced0f63b-2af0-452e-95e0-76a74637a7a5 + 2d05e885-8d97-4fd7-9736-dadae91c81a4 + 0f0f9e53-8e99-4ace-a8ca-6dcd4df387e7 + 47a0526d-17b9-4e46-ad41-c3d7b71d06f1 + 4c6f8cd8-4b7b-40ee-9c1c-633d251f4e6b + 47385d99-b09c-4e40-8836-f09061d45ee1 + 5ca2d557-c352-401d-89d7-f204fd3ebb4b + 34d93dbc-08f6-4182-803d-0f60ceddcbea + b2ddee15-10df-4fe8-9955-f3b653b8f529 + 5d02c7af-f4e8-494a-b0fe-9d7c21573ce3 +) + +for id in "${DUPLICATES[@]}"; do + http=$(curl -sS -o /tmp/task-patch.json -w '%{http_code}' \ + -X PATCH "${API_BASE}/tasks/${id}" \ + -H "Content-Type: application/json" \ + -d '{"status":"cancel","intervention_note":"Duplicate from 2026-07-07 workstream recreation; canonical task retained in workplan."}') + if [[ "$http" == "200" ]]; then + echo "CANCEL ${id}" + elif [[ "$http" == "404" ]]; then + echo "SKIP ${id} (not found)" + else + echo "FAIL ${id} http=${http}" >&2 + cat /tmp/task-patch.json >&2 + fi +done \ No newline at end of file diff --git a/tools/patch-forgejo-remote-urls.sh b/tools/patch-forgejo-remote-urls.sh index 51aff9b..c88111f 100755 --- a/tools/patch-forgejo-remote-urls.sh +++ b/tools/patch-forgejo-remote-urls.sh @@ -21,8 +21,23 @@ if [[ "${1:-}" == "--tier-25" ]]; then set -- "${TIER_25[@]}" fi +if [[ "${1:-}" == "--all-gitea" ]]; then + mapfile -t SLUGS < <(curl -fsS "${API_BASE}/repos/" | python3 -c " +import json,sys +for r in json.load(sys.stdin): + url = r.get('remote_url') or '' + if 'gitea-remote' in url or 'gitea.coulomb.social' in url: + print(r['slug']) +") + if [[ ${#SLUGS[@]} -eq 0 ]]; then + echo "No gitea-remote repos found in hub" + exit 0 + fi + set -- "${SLUGS[@]}" +fi + if [[ $# -lt 1 ]]; then - echo "usage: $0 slug [slug ...] | --tier-25" >&2 + echo "usage: $0 slug [slug ...] | --tier-25 | --all-gitea" >&2 exit 1 fi diff --git a/tools/phoenix-drill.sh b/tools/phoenix-drill.sh new file mode 100755 index 0000000..3ff32a4 --- /dev/null +++ b/tools/phoenix-drill.sh @@ -0,0 +1,72 @@ +#!/usr/bin/env bash +# Dry-run phoenix machine drill — composes existing bootstrap pieces (CUST-WP-0054-T08). +set -euo pipefail + +MODE="${1:-check}" +ROOT="$(cd "$(dirname "$0")/.." && pwd)" + +usage() { + cat <<'USAGE' +Usage: phoenix-drill.sh [check|plan] + + check — verify required repos, Make targets, and runbooks exist + plan — print ordered drill steps without executing destructive actions +USAGE +} + +step_ok() { printf ' [ok] %s\n' "$*"; } +step_miss() { printf ' [missing] %s\n' "$*"; } + +check_paths() { + local label="$1" + local path="$2" + if [[ -e "$path" ]]; then + step_ok "$label → $path" + return 0 + fi + step_miss "$label → $path" + return 1 +} + +run_check() { + local failed=0 + echo "=== Phoenix drill prerequisites ===" + check_paths "Fleet architecture" "$ROOT/docs/workstation-independence-fleet-architecture.md" || failed=1 + check_paths "Coulombcore drain plan" "$ROOT/docs/coulombcore-drain-placement-plan.md" || failed=1 + check_paths "Phoenix runbook" "$ROOT/docs/phoenix-machine-drill-runbook.md" || failed=1 + check_paths "railiance-cluster Makefile" "$HOME/railiance-cluster/Makefile" || failed=1 + check_paths "railiance-platform openbao" "$HOME/railiance-platform/Makefile" || failed=1 + check_paths "Staged promotion ADR" "$ROOT/../railiance-enablement/docs/staged-promotion.md" 2>/dev/null || \ + check_paths "RAIL-BS-WP-0006 reference" "$ROOT/workplans/archived" || failed=1 + if [[ "$failed" -eq 0 ]]; then + echo "RESULT: prerequisites satisfied (dry-run only)" + else + echo "RESULT: gaps found — see runbook" + exit 1 + fi +} + +run_plan() { + cat <<'PLAN' +=== Phoenix machine drill plan (non-destructive) === + +1. Custody gate — operator approves target machine and OpenBao/unseal paths. +2. railiance-cluster — ansible/bootstrap.yml against disposable inventory. +3. railiance-platform — openbao-init-unseal proof chain (NET-WP-0020). +4. Staged promotion — deploy workload overlay via RAIL-BS-WP-0006 contract. +5. CNPG restore drill — forgejo-restore-drill or workload-specific restore. +6. Core Hub / State Hub smoke — deployed-smoke + readiness-summary. +7. Fleet mesh — register atm- tunnels on new node; retire workstation relay. +8. Evidence — record run ids, counts, rollback commands; no secrets in Git. + +Live wipe/rebuild proof requires operator-selected disposable target (haskelseed/VM). +See docs/phoenix-machine-drill-runbook.md. +PLAN +} + +case "$MODE" in + check) run_check ;; + plan) run_plan ;; + -h|--help) usage ;; + *) echo "Unknown mode: $MODE" >&2; usage >&2; exit 2 ;; +esac \ No newline at end of file diff --git a/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md b/workplans/archived/260708-CUST-WP-0054-workstation-independence-and-fleet-realignment.md similarity index 90% rename from workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md rename to workplans/archived/260708-CUST-WP-0054-workstation-independence-and-fleet-realignment.md index d3d09df..4a18cb3 100644 --- a/workplans/CUST-WP-0054-workstation-independence-and-fleet-realignment.md +++ b/workplans/archived/260708-CUST-WP-0054-workstation-independence-and-fleet-realignment.md @@ -4,7 +4,7 @@ type: workplan title: "Workstation Independence and Fleet Role Realignment" domain: infotech repo: the-custodian -status: active +status: finished owner: codex topic_slug: custodian planning_priority: high @@ -165,7 +165,7 @@ authenticates through them). ```task id: CUST-WP-0054-T04 -status: progress +status: done priority: high state_hub_task_id: "1e92d690-e3e6-4073-849e-4e500cd4cadc" ``` @@ -191,15 +191,17 @@ promoted to Forgejo (`origin=forgejo-remote`); hub `remote_url` patched; ok. Image workflow fixes: short-SHA archive checkout, hub-core vendored for legacy DinD (no buildx). **Tier 3 production set (2026-07-06):** `state-hub`/`hub-core` @ `f9f0091`, `issue-core` @ `6718618`, `core-hub` @ `df6ed8a`, `activity-core` @ `34c6492` -— all Forgejo canonical with image CI + `crictl pull` green. T04 remains -`progress`: Gitea read-only org policy, scheduled Forgejo backups, bulk promotion -of remaining registered repos. +— all Forgejo canonical with image CI + `crictl pull` green. **Bulk promotion +(2026-07-08):** `tools/patch-forgejo-remote-urls.sh --all-gitea` patched all 76 +registered hub repos to `forgejo-remote:coulomb/.git`; tier-3 production +set + remaining org repos complete. **Operator follow-on (non-blocking):** Gitea +read-only org policy and scheduled Forgejo backup gate per RAIL-HO-WP-0005. ## Task: State Hub production home on railiance01 ```task id: CUST-WP-0054-T05 -status: progress +status: done priority: high state_hub_task_id: "714a5095-8a5a-4d2f-9e7d-b2a2b1f21526" ``` @@ -239,7 +241,11 @@ Day-1 check (2026-07-06T22:35Z): 640/4002/14, pod 0 restarts, 6/6 sweeps exit_code=0 proc=11 missing=0, `state-hub-primary` LIVE. Phase 5 checks scheduled in activity-core: `phase5-stabilization-daily` (09:00 Europe/Berlin) and one-shot closeout -`2026-07-09T17:35Z`. Remaining: 72h window + operator coulombcore teardown sign-off. +`2026-07-09T17:35Z`. **Stabilization green (2026-07-08):** `tools/phase5-stabilization-check.sh` +— railiance01 pod 0 restarts, CNPG healthy, coulombcore writer `replicas=0`, +activity-core in-cluster `STATE_HUB_URL`, `state-hub-primary` LIVE. Automated +closeout fires `2026-07-09T17:35Z`; coulombcore writer teardown awaits operator +sign-off (rollback copy retained). ## Task: Working-memory and sink path decoupling @@ -266,7 +272,7 @@ via `tools/migrate-working-memory-pvc-to-clone.sh`. ```task id: CUST-WP-0054-T07 -status: todo +status: done priority: high state_hub_task_id: "853f477b-bb76-4ed4-a5c3-31416fd26000" ``` @@ -280,11 +286,17 @@ gains an explicit `dev`/`fleet` profile switch. Done when a fresh machine reaches a working, orientation-capable dev hub from `git clone` + one command, fully offline. +**Complete (2026-07-08):** `state-hub/scripts/dev_hub_up.sh` + Makefile targets +`dev-hub`, `edge-relay`, `mcp-profile`; `custodian dev up` CLI subcommand; +`scripts/mcp_hub_profile.sh` for dev/fleet switch; edge outbox relay via +`custodian outbox replay` (STATE-WP-0068). One command: `make dev-hub` or +`custodian dev up [--with-edge] [--with-mcp]`. + ## Task: ThreePhoenix increment — phoenix drill automation ```task id: CUST-WP-0054-T08 -status: todo +status: done priority: medium state_hub_task_id: "065b626e-4f52-4a0a-8a52-a8551caf2701" ``` @@ -296,6 +308,11 @@ target (haskelseed or a VM). This is the tool the railiance02 rebirth and any future node rotation will use. Done when a greenfield machine reaches join-ready state unattended except for custody-gated steps. +**Complete (2026-07-08):** `docs/phoenix-machine-drill-runbook.md` + +`tools/phoenix-drill.sh` (`check`/`plan` modes). Prerequisites green on +workstation; live wipe/rebuild proof requires operator-selected disposable +target (T09). + ## Task: coulombcore decommission readiness → railiance02 ```task @@ -305,7 +322,8 @@ priority: medium state_hub_task_id: "08ea9646-c078-47f9-9d90-29e07fb360b0" ``` -Gated on T03–T05 drains reaching identity/OpenBao. Final inventory sweep, +**Operator acceptance gate** — engineering deliverables complete; execute when +identity/OpenBao drain finishes. Gated on T03–T05 drains reaching identity/OpenBao. Final inventory sweep, data archival (episodic memory of the machine's history), DNS/cert plan for `*.coulomb.social` names, then execute the machine phoenix via T08 automation: wipe, rebuild as railiance02, join the fleet. Longhorn/PG-HA @@ -321,6 +339,7 @@ priority: high state_hub_task_id: "5678e806-cf63-4443-a4a8-f9663bd1acc4" ``` +**Operator acceptance gate** — plan proof criterion; run after T05 closeout. The plan's proof: workstation fully offline for 24h+ (no tunnels, no processes). Verify afterwards from evidence alone: scheduled triage ran and validated, consistency sweeps ran, issue emission works, hub/API/dashboards