Add a test driving the actual captured 2026-06-26 failure payload
(tests/fixtures/wp0016/...partial.json): it now recovers 6+ valid recommendations
and quarantines the truncated tail, where before WP-0016 it discarded the whole run.
Add an over-depth guardrail test. Together with T03/T04 the regression set now covers
truncation, one-bad-item, oversized-string, over-depth, allow-list/injection-shaped,
and happy-path count cap.
In-repo portion of T05 complete; the live railiance01 graceful-degradation smoke is
operator-owned cluster work (deploy-coupled with the T02 bundle changes) and remains
outstanding. Hand-back notes posted to WP-0006-T03 and WP-0010-T04. Full suite: 220
passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add ADR-004 documenting the producer trust boundary: untrusted producers (LLM,
agent, human; erroneous and malicious), the trust-but-handle vs verify-and-mitigate
postures, error-locality and quarantine-with-provenance principles, and the concrete
activity-core mechanisms.
Implement producer-agnostic guardrails in executor.py, applied uniformly on the
happy path and the recovery path via _partition_items: structural-type -> schema ->
structural caps (_MAX_DEPTH, _MAX_STRING_LEN) -> reference allow-list -> count cap.
Each quarantine carries a reason. Closes the happy-path maxItems count cap deferred
from T03 (valid 9-item report keeps 7, quarantines 2). Reference allow-list reads
context["known_candidates"] via _allow_list_from_context; inert until a resolver
populates it. SCOPE.md updated (executor bullet + ADR list); no INTENT drift.
New tests: happy-path count cap, oversized-string guardrail, allow-list rejection.
Full suite: 218 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the whole-document parse + one retry still fail, report instructions now run
_resilient_report before the total-loss path. A brace/quote-aware scanner
(_extract_object_spans) recovers each recommendation object whether pretty-printed
across many lines or NDJSON one-per-line; a truncated tail gets a best-effort
_try_repair; _partition_items validates each recovered object against the T02 item
schema. Valid items survive (output_validated=True, partial=True), malformed/
over-maxItems items are quarantined with provenance (index, error, raw, reason),
capped at 20. Error locality now matches the unit of work: one bad item costs one
item, not the whole report.
Verified against the real 06-26 shape: 7 valid recommendations + a truncated tail
now recovers all 7 and quarantines the broken tail (previously the whole run was
discarded). Happy-path maxItems top-N enforcement is deferred to T04 (count caps).
Full suite: 215 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the accept-anything recommendations.items ({type: object}) with a strict
per-item contract (required [rank, candidate, action, why] + typed wsjf) and a
maxItems:7 hint. Strict item structure is what lets the T03 boundary parser
validate each recommendation independently and quarantine only malformed ones.
maxItems is a producer hint (prompt + llm-connect json_schema + T03 mitigation),
NOT a hard reject — a hard maxItems reject would discard a whole 16-item report,
the blast-radius bug WP-0016 removes. DEPLOY COUPLING: the strict schema is also
consumed by the current whole-doc validator, so it must ship with T03's per-item
quarantine parser; until then it increases whole-doc hard-fails. Prompt + max_tokens
headroom + NDJSON framing are documented as a runtime-bundle handoff.
Updated four tests to the strict contract; the forwarded-schema test now reads the
live schema file instead of hard-coding it. Full suite: 213 passed, 1 skipped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Local analysis of the 2026-06-26 daily-triage validation failure: the unbounded
~1-recommendation-per-workstream list (16 active workstreams; JSON break at char
5268, ~rank 8-9) is the structural cause; both the first attempt and the retry
failed. The exact offending token and finish_reason are unrecoverable from
activity-core data — complete() drops finish_reason/usage, the report sink caps
raw output at 4000 chars (< 5268), and the log preview at 2000. Confirming the
exact token needs llm-connect producer-side logs on railiance01 (operator-owned);
mitigation (T02/T03) is identical regardless. Partial fixture captured.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add WP-0016 to make the instruction-executor output contract robust after the
2026-06-26 daily-triage validation failure (one malformed delimiter discarded a
whole report). Per-item framing for error locality, verify-and-mitigate boundary
parsing with a quarantine lane, producer-trust-boundary guardrails (ADR-004), and
regression/calibration tests. Unblocks WP-0006-T03 / WP-0010-T04.
Also record the 06-26 recheck outcome (streak reset at two) in WP-0006-T03.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
T04 done: built+deployed the WP-0014 image to railiance01, applied catchup_latest
to daily-statehub-wsjf-triage, /admin/sync clean (6 defs, 4 schedules, 0 errors).
Live schedule verified OverlapPolicy=BufferOne, CatchupWindow=1d; pods healthy.
All tasks T01-T05 complete; beachhead-endpoint adoption tracked in WP-0015.
Workplan status -> finished.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Idempotent-writes half of T05 is done in-repo; the externally-blocked endpoint
adoption + actcore-state-hub-bridge proxy retirement move to ACTIVITY-WP-0015
(blocked on the state-hub beachhead) so WP-0014 can close on completed work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add activity_core/state_hub_write: every State Hub write (report-sink,
ops-evidence, schedule-miss) now sends a stable Idempotency-Key header derived
from run_id:instruction_id:event_type. Makes writes safe to buffer/replay under
the future state-hub beachhead without duplicate progress/triage events. The
read-based _progress_exists dedup is now best-effort (returns False on connection
error instead of hard-failing), so the guarantee lives on the keyed write rather
than a live read. Tests + runbook note. Endpoint adoption / proxy retirement stays
blocked on the state-hub beachhead capability.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resilience (queue/cache) is handed to custodian/state-hub as a per-machine
beachhead; activity-core keeps only idempotent writes + adopt-beachhead-endpoint
and retires its bespoke actcore-state-hub-bridge proxy. Proposal sent to state-hub.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live inspection of railiance01 (ssh + in-node kubectl/temporal) overturns the
catchup_window hypothesis: the daily-triage schedule is healthy (CatchupWindow
365d default, 0 MissedCatchupWindow). The 2026-06-23T05:20Z fire ran but Failed
at the report sink with '[Errno 111] Connection refused' posting to State Hub.
railiance01 reaches State Hub via a reverse tunnel back to the workstation, which
is unreachable at 07:20 Europe/Berlin (102 resolver timeouts in 24h). Mark T01
done; add T05 for resilient sinks/resolvers as the real incident fix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add activity_core/schedule_health: a pure evaluate_schedule_health() verdict
(built on Temporal's num_actions_missed_catchup_window plus a staleness check),
an async check_schedule_health() reader, and post_missed_fire_alert() that emits
a schedule_miss State Hub progress event. Makes a missed fire visible even under
misfire_policy=skip, where Temporal drops it by design. Unit tests for the
verdict logic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set Temporal catchup_window on cron schedules so a fire missed during a
worker/Temporal outage is no longer silently dropped. Redefine misfire_policy
into three explicit modes — skip, catchup_all, catchup_latest — mapping to
(catchup_window, overlap) pairs; legacy catchup/compress aliased. Add
catchup_window_seconds override. Remove the ad-hoc upsert-time 1h backfill in
favour of native catchup. Apply catchup_latest to daily-statehub-wsjf-triage in
the Railiance runtime manifest and document run-miss policies in the runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cron fires are silently dropped: _build_schedule() sets SchedulePolicy(overlap=)
but never catchup_window, so a brief worker/Temporal outage at trigger time drops
the fire with no recovery and no signal (root cause of missing 06-22/06-23 daily
triage runs). Define three explicit run-miss policies: skip, catchup_all,
catchup_latest, plus missed-fire detection.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The actcore-state-hub-bridge readiness probe hit /state/summary through
the tunnel proxy chain. Cold-cache summary requests and intermittent
tunnel stalls routinely exceeded the 5s probe timeout (1584 failures
over 17h), leaving the pod 0/1 Ready and breaking hourly/triage sinks.
Use /state/health instead — same signal the ops inventory already
expects, and completes in ~30ms through the bridge.
Apply the new 'tooling' category (reusable internal tooling/infrastructure)
from the Repo Classification Standard. First-pass agent classification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First-pass agent classification per the Repo Classification Standard v1.0
(canon-repo-classification); pending human review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Note the 2026-06-19 live reconciliation on railiance01: llm-connect
deployed, worker restarted with LLM_CONNECT_URL, fixture smoke passed.
Manual daily triage still blocked on actcore-state-hub-bridge reachability.
