state-hub/docs/cron-migration.md

State Hub Cron → activity-core ActivityDefinition Migration

CUST-WP-0040 T04. Partially implemented as of STATE-WP-0064. The consistency sweep API surface and ActivityDefinition are landed; cluster cutover still requires manual canary, parallel week, and local timer retirement.

The state hub currently runs two recurring maintenance jobs and one per-repo event hook. Once activity-core is ready, each becomes an ActivityDefinition file checked into the appropriate repo. The state hub keeps the underlying scripts; only the scheduling moves.

---

1. Inventory of current maintenance automations

#	Source	Trigger today	Script invoked	What it does
1	systemd user timer	every 15 min	scripts/consistency_check.py --remote --all	Pull every registered repo, reconcile workplan files ↔ DB, run C-15 writeback + C-16 pull gate
2	manual / daily cron	make cleanup-stale (suggested 0 3 * * *)	scripts/cleanup_stale_tasks.py	Cancel tasks still open in finished/archived workstreams; emits org.statehub.task.stale
3	git post-commit	every commit in a registered repo	make fix-consistency REPO=<slug>	Per-repo workplan ↔ DB sync immediately after a commit

Honourable mentions (not currently scheduled, on-demand only — listed for completeness so they don't get mistakenly picked up):

• scripts/ingest_sbom.py — invoked via make ingest-sbom REPO=<slug>.
• scripts/ingest_capabilities.py — invoked via make ingest-capabilities[-all].
• scripts/check_doi.py — invoked via make check-doi[-all].
• scripts/validate_repo_adr.py — invoked manually for canon promotion.
• scripts/ingest_tpsc.py — invoked via make ingest-tpsc[-all].

These are not in scope for cron migration — they remain on-demand operator/CI commands. They become candidates only if we later decide to run them on a schedule.

---

2. Target ActivityDefinitions

A. state-hub-consistency-sweep (implemented)

Landed in the-custodian/activity-definitions/state-hub-consistency-sweep.md with enabled: false until canary and cutover.

Invocation path (matches the hourly RecentlyOnScope pattern):

• activity-core context query: consistency_sweep_remote_all
• State Hub endpoint: POST /consistency/sweep/remote-all
• payload: {"max_seconds": 300}
• progress event: consistency_sweep_remote_all

State Hub runs scripts/consistency_check.py --remote --all --json on the workstation host. activity-core does not shell into the laptop repo checkout from the cluster.

Operator runbook: docs/consistency-sweep-runbook.md.

Notes:

• Replaces the custodian-sync.service + custodian-sync.timer pair after parallel week and cutover.
• Lock semantics (/tmp/custodian-consistency-remote-all.lock) stay in the script — activity-core just sets the cadence.
• Local timer retirement is tracked in STATE-WP-0064-T04.

B. state-hub-stale-task-cleanup

# activity-definitions/state-hub-stale-task-cleanup.yaml
id: the-custodian.state-hub-stale-task-cleanup
description: |
  Daily sweep that cancels tasks still `wait|todo|progress` inside
  finished or archived workstreams. Each cancellation also emits
  org.statehub.task.stale on NATS for downstream reaction.
trigger:
  trigger_type: cron
  cron_expression: "0 3 * * *"
  timezone: UTC
instruction:
  kind: shell
  cmd: >-
    cd /home/worsch/state-hub &&
    .venv/bin/python scripts/cleanup_stale_tasks.py

Notes:

• Replaces the documented (Cron example: 0 3 * * * …) daily run.
• The script already emits NATS events (see CUST-WP-0040 T03), so downstream ActivityDefinitions can react per-task without a second pass.

C. Per-commit consistency sync (currently a git hook)

The git post-commit hook installed by state-hub/scripts/install_hooks.sh is event-driven, not cron-based. Migrating it to activity-core would require a repo.commit.pushed event channel that doesn't exist yet.

Recommendation: keep the git hook as-is for now. Revisit once an event source (e.g. Gitea webhook fed into NATS) is available, at which point an event-triggered ActivityDefinition can replace it cleanly:

trigger:
  trigger_type: event
  event_type: org.repo.commit.pushed
  filters:
    repo_slug: "*"

---

3. Required context queries

Implemented for A:

• consistency_sweep_remote_all — POST /consistency/sweep/remote-all with a 330s resolver timeout (sweep budget default 300s).

Still optional for B and future splits:

• state-hub.health — GET /state/health → {status, db, ...}
• (optional) state-hub.repos — GET /repos/?status=active for per-repo ActivityDefinitions if the monolithic sweep is split later.

---

4. Blockers / sequencing

Blocker	Owner	Where it lands
activity-core ActivityDefinition file ingestion + cron executor (WP-0003)	activity-core	activity-core/src/...
activity-core shell instruction kind with on_failure semantics	activity-core	activity-core/src/...
state-hub adapter exposing state-hub.health as a context source	activity-core	activity-core/adapters/

Until B lands and A is cut over, the state hub continues to schedule the consistency sweep via the local systemd timer.

---

5. Cutover plan (when ready)

1. Land ActivityDefinitions A + B in activity-core.
2. Enable them in staging; verify they fire on schedule and produce the same DB / NATS effects as the current cron entries.
3. Run both in parallel for one week (cron + ActivityDefinition). The scripts are idempotent — duplicate runs are no-ops on a clean state.
4. Disable the systemd timer: systemctl --user disable --now custodian-sync.timer
5. Remove the cleanup-stale cron entry from crontab -e.
6. Update infra/README.md to point at the ActivityDefinitions and archive the systemd unit files.
7. Per-commit hook stays until a repo.commit.pushed event exists.

---

6. Open questions

• Locking: should activity-core wrap shell instructions with a process lock (today the script self-locks via /tmp/...)? If yes, the state-hub script's lock can be removed.
• Failure surfacing: today systemd journals capture stderr. Where does an ActivityDefinition's shell stderr go? (logs ? activity history ?) — needs activity-core docs before cutover.
• Per-repo split: do we split A into one ActivityDefinition per registered repo (so failures don't poison the sweep), or keep the monolithic --all mode? The latter is simpler and matches today's behaviour; the former gives better observability.