generated from coulomb/repo-seed
feat(STATE-WP-0064): add consistency sweep remote-all API endpoint
Expose POST /consistency/sweep/remote-all so activity-core can trigger the workstation ADR-001 remote-all sweep via the bridge tunnel pattern. Records consistency_sweep_remote_all progress events and documents the cutover runbook while the local custodian-sync timer remains interim.
This commit is contained in:
105
docs/consistency-sweep-runbook.md
Normal file
105
docs/consistency-sweep-runbook.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# State Hub Consistency Sweep Runbook
|
||||
|
||||
## Purpose
|
||||
|
||||
This runbook answers whether the 15-minute State Hub consistency sync ran
|
||||
without relying on the local `custodian-sync.timer`.
|
||||
|
||||
The intended steady state after `STATE-WP-0064` cutover is:
|
||||
|
||||
- activity-core on Railiance01 owns the `*/15 * * * *` UTC schedule and
|
||||
ActivityRun audit trail.
|
||||
- State Hub on the workstation owns `scripts/consistency_check.py`, lock
|
||||
semantics, reconciliation, and the `consistency_sweep_remote_all`
|
||||
progress event.
|
||||
- The local systemd timer is disabled after the parallel week passes.
|
||||
|
||||
## API Surface
|
||||
|
||||
Manual or cluster-triggered invocation:
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/consistency/sweep/remote-all \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"max_seconds": 300}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
From Railiance01 through the bridge tunnel, use the `STATE_HUB_URL`
|
||||
configured for activity-core (for example the `actcore-state-hub-bridge`
|
||||
service target).
|
||||
|
||||
## Schedule Check
|
||||
|
||||
From the activity-core host, confirm the definition is synced and the
|
||||
Temporal schedule exists:
|
||||
|
||||
```bash
|
||||
cd ~/activity-core
|
||||
ACTIVITY_DEFINITION_DIRS=/home/worsch/the-custodian make sync-activity-definitions
|
||||
make sync-schedules
|
||||
```
|
||||
|
||||
Expected definition:
|
||||
|
||||
- name: `State Hub Consistency Sweep`
|
||||
- trigger: `*/15 * * * *`
|
||||
- timezone: `UTC`
|
||||
- misfire policy: `skip`
|
||||
- enabled: `false` until manual canary passes, then `true` after cutover
|
||||
|
||||
## Progress Event Check
|
||||
|
||||
Query State Hub for the latest sweep event:
|
||||
|
||||
```bash
|
||||
curl -s "http://127.0.0.1:8000/progress/?event_type=consistency_sweep_remote_all&limit=5" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
Healthy evidence includes:
|
||||
|
||||
- `lock_skipped: false` on normal runs
|
||||
- `repos_processed` entries only for repos that needed action
|
||||
- `skipped_clean`, `skipped_missing`, and `skipped_budget` metadata when
|
||||
applicable
|
||||
- `exit_code: 0` for warn-only remote-all sweeps
|
||||
|
||||
A `lock_skipped: true` response is normal when the local timer and the
|
||||
cluster schedule overlap during the parallel week.
|
||||
|
||||
## ActivityRun Check
|
||||
|
||||
Query the activity-core database for the most recent run of the sweep
|
||||
definition:
|
||||
|
||||
```sql
|
||||
select
|
||||
run_id,
|
||||
fired_at,
|
||||
scheduled_for,
|
||||
context_snapshot->'consistency_sweep_remote_all' as sweep_result
|
||||
from activity_runs
|
||||
where definition_id = '7c4e9a12-8f3b-4d5e-9c6a-1b2d3e4f5a6b'
|
||||
order by fired_at desc
|
||||
limit 5;
|
||||
```
|
||||
|
||||
## Manual Canary
|
||||
|
||||
Before enabling the cluster schedule:
|
||||
|
||||
1. Confirm `state-hub-railiance01` tunnel health from ops-bridge.
|
||||
2. Trigger one manual ActivityRun or POST the API through the bridge URL.
|
||||
3. Verify the progress event and ActivityRun context snapshot.
|
||||
4. Confirm idempotence when the local timer also fires (lock skip is OK).
|
||||
|
||||
## Cutover
|
||||
|
||||
After one parallel week (`STATE-WP-0064-T03`):
|
||||
|
||||
```bash
|
||||
systemctl --user disable --now custodian-sync.timer
|
||||
```
|
||||
|
||||
Then enable the activity-core definition and treat the cluster schedule
|
||||
as the sole primary runner.
|
||||
Reference in New Issue
Block a user