coulomb/state-hub
Archived
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(state-hub): STATE-WP-0063 T03 done — tunnel cleanup restored activity-core

Browse Source 
Document stale remote sshd forward on Railiance01 :18000 as root cause of
reconnect loop; T03 verified after bridge maintenance cleanup and manual
canaries for hourly RecentlyOnScope and daily WSJF triage.
This commit is contained in:
Bernd Worsch
2026-06-21 19:47:56 +02:00
parent dff8cfe128
commit 323599f2fc
1 changed files with 15 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
workplans/STATE-WP-0063-weekend-automation-repair.md
View File

@@ -115,12 +115,15 @@ Result 2026-06-21:
**workstation availability + ops-bridge tunnel reachability** to the local **workstation availability + ops-bridge tunnel reachability** to the local
State Hub API. Cluster-side `actcore-state-hub-bridge` proxies to node-local State Hub API. Cluster-side `actcore-state-hub-bridge` proxies to node-local
`127.0.0.1:18000` as designed. `127.0.0.1:18000` as designed.
- **Retry fix (2026-06-21):** stale orphan `sshd` remote forward on Railiance01
port 18000 blocked new tunnel binds (`ExitOnForwardFailure` → exit 255).
`bridge maintenance cleanup state-hub-railiance01 --restart` cleared it.
## T3 — Restore daily WSJF triage evidence ## T3 — Restore daily WSJF triage evidence
```task ```task
id: STATE-WP-0063-T03 id: STATE-WP-0063-T03
status: progress status: done
priority: medium priority: medium
state_hub_task_id: "4b68b207-a80e-4c71-a246-10035ef69625" state_hub_task_id: "4b68b207-a80e-4c71-a246-10035ef69625"
``` ```
@@ -134,12 +137,17 @@ Railiance01 and confirm:
If the schedule was paused, unpause and verify the next 07:20 Europe/Berlin If the schedule was paused, unpause and verify the next 07:20 Europe/Berlin
fire is armed. fire is armed.
Progress 2026-06-21: Manual trigger via actcore-api returned workflow Progress 2026-06-21 (first attempt): sink failed while tunnel reconnecting.
`activity-6fca51fa…:manual-ca469cb5…`. Schedule is armed (next 07:20 Europe/Berlin).
Report sink `state-hub-progress` is still failing with `[Errno 111] Connection Result 2026-06-21 (retry): `bridge maintenance cleanup state-hub-railiance01
refused` while `state-hub-railiance01` tunnel is reconnecting. Re-verify after --restart` cleared stale remote `sshd` on port 18000 (orphan forward with many
tunnel stabilises (`bridge check state-hub-railiance01` + new `daily_triage` CLOSE_WAIT). Tunnel now **connected**; node `curl 127.0.0.1:18000/state/health`
progress event). and worker `actcore-state-hub-bridge:8000/state/health` both return ok. Manual
canaries succeeded:
- hourly RecentlyOnScope → `recently_on_scope_hourly` at 2026-06-21T17:45:29Z
- daily WSJF triage → `daily_triage` at 2026-06-21T17:45:46Z + working-memory
report under `the-custodian/memory/working/`
## T4 — Repair ancillary local crons ## T4 — Repair ancillary local crons