coulomb/state-hub
Archived
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

finish(STATE-WP-0064): cut over scheduler and split sweep errors from failures

Browse Source 
STATE-WP-0064 cutover (state-hub only):
- Retire local custodian-sync.timer; archive units under infra/systemd/archived/
- Mark workplan finished; update infra/README, cron-migration, runbook, AGENTS.md
- Point activity-core-delegation at the consistency-sweep runbook

Consistency engine — automation error vs assessment failure:
- C-00 is an automation error; C-01..C-23 assessment failures are recorded
  for follow-up but no longer fail --remote --all scheduled sweeps (exit 0)
- Skip workplans/README.md in the workplan glob (human index, not a workplan)
- Progress events and compare script expose automation_error and
  assessment_failures separately from exit_code
This commit is contained in:
Bernd Worsch
2026-06-22 01:20:59 +02:00
parent 270033a50d
commit 39ed5459b9
14 changed files with 221 additions and 180 deletions
Download Patch File Download Diff File Expand all files Collapse all files

101
infra/README.md
View File

@@ -15,89 +15,38 @@ The compose file is `infra/docker-compose.yml`. Copy `.env.example` to `.env` an
---
## Periodic Repo Sync — systemd user timer
## Periodic Repo Sync — activity-core (Railiance01)
The **State Hub consistency sync** timer (legacy unit name `custodian-sync`)
runs `consistency_check.py --remote --all` every 15 minutes, keeping workplan
file state in sync with the state-hub DB automatically (belt-and-suspenders
alongside the per-repo git post-commit hooks).
The **State Hub consistency sync** runs every 15 minutes (`*/15 * * * *` UTC)
on activity-core (Railiance01). The cluster schedule triggers
`POST /consistency/sweep/remote-all` on the workstation State Hub via the
`actcore-state-hub-bridge` tunnel.
> **Interim local runner (STATE-WP-0063):** units must target the standalone
> repo at `/home/worsch/state-hub` and invoke consistency via
> `/home/worsch/.local/bin/uv run python …`. The pre-extraction path
> `/home/worsch/the-custodian/state-hub` is obsolete.
>
> **Cluster runner (STATE-WP-0064):** activity-core on Railiance01 runs the
> same sweep on `*/15 * * * *` UTC (parallel week started 2026-06-21). Both
> runners use `POST /consistency/sweep/remote-all` with `detail.source`
> tagging (`local-timer` vs `activity-core`). Disable this local timer after
> T04 cutover per [`docs/consistency-sweep-runbook.md`](../docs/consistency-sweep-runbook.md).
Operator runbook: [`docs/consistency-sweep-runbook.md`](../docs/consistency-sweep-runbook.md).
The all-repo remote sweep has two built-in load guards:
**Prerequisites for cluster-triggered sweeps:**
- Workstation State Hub API running (`make api` or equivalent)
- `state-hub-railiance01` ops-bridge tunnel `connected`
- Workstation awake (execution still runs locally; only scheduling moved)
Per-repo git post-commit hooks remain the immediate consistency path after
each commit. The 15-minute sweep is belt-and-suspenders across all registered
repos.
The all-repo remote sweep has built-in load guards:
- A nonblocking process lock at `/tmp/custodian-consistency-remote-all.lock`;
if a prior sweep is still active, the next timer run exits cleanly.
overlapping triggers exit cleanly with `lock_skipped: true`.
- A wall-clock budget, defaulting to 300 seconds. Remaining repos are skipped
once the budget is exhausted. Override with `--max-seconds N` or set
`CONSISTENCY_REMOTE_ALL_MAX_SECONDS`.
- Warn-only sweeps exit 0 in `--remote --all` mode so the systemd unit only
goes failed for hard consistency failures.
once the budget is exhausted.
### Unit files
### Retired local timer
| File | Repo template | Installed copy |
|------|---------------|----------------|
| `custodian-sync.service` | `infra/systemd/custodian-sync.service` | `~/.config/systemd/user/custodian-sync.service` |
| `custodian-sync.timer` | `infra/systemd/custodian-sync.timer` | `~/.config/systemd/user/custodian-sync.timer` |
Install or refresh from the repo templates:
```bash
mkdir -p ~/.config/systemd/user
cp ~/state-hub/infra/systemd/custodian-sync.service ~/.config/systemd/user/
cp ~/state-hub/infra/systemd/custodian-sync.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now custodian-sync.timer
```
### Management commands
```bash
# Check status
systemctl --user status custodian-sync.timer
systemctl --user list-timers custodian-sync.timer
# View recent logs
journalctl --user -u custodian-sync.service -n 50
# Trigger immediately (for testing)
systemctl --user start custodian-sync.service
# Disable
systemctl --user disable --now custodian-sync.timer
# Re-enable
systemctl --user enable --now custodian-sync.timer
```
### Guard condition
The service uses `ExecStartPre` to check the API is reachable before running:
```
ExecStartPre=/usr/bin/curl -sf http://127.0.0.1:8000/state/health
```
If the API is offline, the service exits cleanly without error (the timer will retry
in 15 minutes).
### WSL2 note
systemd user mode works in WSL2 when `systemd=true` is set in `/etc/wsl.conf`.
If systemd is not available, fall back to crontab:
```bash
# Crontab fallback (run crontab -e and add):
*/15 * * * * curl -sf http://127.0.0.1:8000/state/health && cd ~/state-hub && /home/worsch/.local/bin/uv run python scripts/consistency_check.py --remote --all >> /tmp/custodian-sync.log 2>&1
```
The legacy `custodian-sync.{service,timer}` systemd units were disabled
2026-06-21 (`STATE-WP-0064`). Archived templates live in
[`infra/systemd/archived/`](systemd/archived/). Do not re-enable unless
debugging a cluster scheduling outage.
---
@@ -118,4 +67,4 @@ make remove-hooks REPO=marki-docx
```
The hook is idempotent (guarded by `# custodian-sync-hook` marker) and runs
in the background so it does not block the commit.
in the background so it does not block the commit.