generated from coulomb/repo-seed
feat(ACTIVITY-WP-0016): register LLM output robustness & producer trust boundary workplan
Add WP-0016 to make the instruction-executor output contract robust after the 2026-06-26 daily-triage validation failure (one malformed delimiter discarded a whole report). Per-item framing for error locality, verify-and-mitigate boundary parsing with a quarantine lane, producer-trust-boundary guardrails (ADR-004), and regression/calibration tests. Unblocks WP-0006-T03 / WP-0010-T04. Also record the 06-26 recheck outcome (streak reset at two) in WP-0006-T03. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -174,6 +174,27 @@ the worker consumes the configured URL, then produce schema-valid daily triage
|
||||
evidence and three clean scheduled runs. This narrower path is tracked in
|
||||
`ACTIVITY-WP-0010`.
|
||||
|
||||
2026-06-25: Consecutive-run streak resumed. State Hub `daily_triage` progress
|
||||
events from author `activity-core` fired on time on **2026-06-24 05:20:56Z** and
|
||||
**2026-06-25 05:20:47Z** (07:20 Berlin), both delivered, no misfires. That is two
|
||||
clean consecutive scheduled runs. **RECHECK 2026-06-26 (after 05:20Z):** confirm
|
||||
the 06-26 scheduled `daily_triage` event delivered. If clean, that completes three
|
||||
clean consecutive scheduled runs (06-24 / 06-25 / 06-26) — record the calibration
|
||||
result in State Hub and close T03. If the 06-26 run misfires or is missing, the
|
||||
streak resets and T03 stays `wait`. Flag deliberately kept in-repo (agent-agnostic)
|
||||
rather than tied to any single coding agent's scheduler.
|
||||
|
||||
2026-06-26 recheck outcome: **streak reset at two.** The 06-26 scheduled run fired
|
||||
on time (`daily_triage` event 05:20:57Z) — scheduling layer healthy, no misfire —
|
||||
but the `daily-triage-report` instruction output **failed schema validation**:
|
||||
`Expecting ',' delimiter: line 136 column 22 (char 5268)`. The model produced a
|
||||
long ranked WSJF recommendation list (reached rank 7+ with nested `wsjf` objects)
|
||||
whose JSON broke ~char 5268; only a bounded 4000-char preview is preserved in the
|
||||
State Hub event, so the exact offending token needs the runtime llm-connect log.
|
||||
This is an LLM-output-quality failure (tracked by `ACTIVITY-WP-0010`), not a
|
||||
runtime/projection failure. T03 stays `wait`; three clean consecutive scheduled
|
||||
runs not yet achieved (06-24 ✅, 06-25 ✅, 06-26 ✗-validation).
|
||||
|
||||
## Rule Action Contract Documentation
|
||||
|
||||
```task
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
id: ACTIVITY-WP-0016
|
||||
type: workplan
|
||||
title: "LLM Output Robustness & The Producer Trust Boundary"
|
||||
domain: custodian
|
||||
repo: activity-core
|
||||
status: proposed
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-06-26"
|
||||
updated: "2026-06-26"
|
||||
state_hub_workstream_id: "4ef0d53b-1777-41ae-80c6-1b69fdb34726"
|
||||
---
|
||||
|
||||
# ACTIVITY-WP-0016 — LLM Output Robustness & The Producer Trust Boundary
|
||||
|
||||
## Context
|
||||
|
||||
On 2026-06-26 the scheduled `daily-statehub-wsjf-triage` instruction fired on
|
||||
time (`daily_triage` event 05:20:57Z) but its output **failed schema
|
||||
validation**: `Expecting ',' delimiter: line 136 column 22 (char 5268)`. The
|
||||
model emitted a long ranked WSJF recommendation list (reached rank 7+ with
|
||||
nested `wsjf` objects) and the JSON broke deep in that list. Because the report
|
||||
is a single monolithic JSON document, one malformed delimiter discarded the
|
||||
**entire** run. This reset the three-clean-consecutive-scheduled-runs streak in
|
||||
`ACTIVITY-WP-0006-T03` (06-24 ✅, 06-25 ✅, 06-26 ✗-validation) and is the
|
||||
LLM-output-quality surface deferred from `ACTIVITY-WP-0010`.
|
||||
|
||||
The scheduling/runtime layer is healthy — this is purely an output-robustness
|
||||
and boundary-design problem. Today's code (`src/activity_core/rules/executor.py`)
|
||||
already: passes the output schema to llm-connect as a `json_schema` model param
|
||||
(`_llm_run_config`), retries once, runs a fenced/`raw_decode` tolerant parser
|
||||
(`_parse_json_output`), and preserves a bounded 4000-char preview on hard
|
||||
failure (`_invalid_output_report`). None of that helps when error locality is
|
||||
zero: the failure unit is the whole document, not the offending item.
|
||||
|
||||
## Design Frame — The Producer Trust Boundary
|
||||
|
||||
This workplan is anchored to a deliberate architectural stance, not just a bug
|
||||
fix. Capture it in an ADR (T04) so future work inherits it.
|
||||
|
||||
**Premise.** activity-core has a *trust boundary* where free-form producer
|
||||
output meets strict deterministic consumers (JSON Schema validators, the task
|
||||
emitter, classic compute pipelines). The producers are **LLMs and humans (and
|
||||
agents acting for either)**. Both are *untrusted producers*: their output may be
|
||||
|
||||
- **erroneous** — hallucination, truncation (token-limit cutoff), drift,
|
||||
type slips, typos; or
|
||||
- **malicious** — prompt injection, crafted payloads, oversized/deeply-nested
|
||||
structures aimed at exhausting or confusing the consumer.
|
||||
|
||||
The architecture should treat the boundary as an adversarial frontier and place
|
||||
**guardrails + error-correction tooling there**, rather than letting raw
|
||||
producer output flow into deterministic consumers and fail (or worse, partially
|
||||
succeed) downstream.
|
||||
|
||||
**Two non-fail-fast postures.** When we do *not* want to hard-fail on a problem,
|
||||
there are two sensible strategies — and they compose:
|
||||
|
||||
- **A) Trust but handle exceptions** (optimistic / reactive). Consume the output
|
||||
as-is; on exception, catch → repair → retry → or quarantine. Cheap on the
|
||||
happy path. Blast radius depends entirely on how granular the catch is. Good
|
||||
when failures are rare and locally recoverable. Risk: failures surface late,
|
||||
possibly after partial side effects.
|
||||
- **B) Verify and mitigate** (defensive / proactive). Validate, sanitize, clamp,
|
||||
and normalize the output to a known-good shape *before* it enters the pipeline
|
||||
— drop bad items, coerce types, bound sizes/depth, allow-list references — so
|
||||
the consumer only ever sees clean input. Higher upfront cost, smaller blast
|
||||
radius, no partial side effects. Good when failures are common or
|
||||
consequences are high.
|
||||
|
||||
**Governing principles for this repo:**
|
||||
|
||||
1. **Push verification to the boundary; keep the interior strict.** Apply
|
||||
posture **B** at the producer→consumer boundary (verify+mitigate structure);
|
||||
keep posture **A** for residual exceptions inside the verified core. Never
|
||||
relax the interior schema to absorb producer sloppiness.
|
||||
2. **Make error locality match the unit of work.** One bad recommendation must
|
||||
cost one recommendation, not the whole report. Framing the payload so each
|
||||
item is independently parseable is the single highest-leverage change.
|
||||
3. **Quarantine, never silently drop.** Invalid units are preserved as bounded,
|
||||
provenance-tagged artifacts (index, error, raw snippet) so they can be
|
||||
debugged or replayed — degraded-but-usable is distinct from total loss.
|
||||
4. **Both human and agent input get the same rigor.** Guardrails are
|
||||
producer-agnostic: the same size/depth/count caps, reference allow-lists, and
|
||||
truncation detection apply whether the producer is an LLM, an agent, or a
|
||||
human form submission.
|
||||
|
||||
## Reproduce & Root-Cause The Failure
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0016-T01
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: "74fd16a5-4ea5-4dfe-8526-dfa27cf76138"
|
||||
```
|
||||
|
||||
Recover the **full** raw llm-connect response for the 06-26 failure (the State
|
||||
Hub event keeps only a 4000-char preview; the break is at char 5268) and
|
||||
establish the precise cause.
|
||||
|
||||
Done when:
|
||||
|
||||
- the full raw response is pulled from the runtime llm-connect log / response
|
||||
store and the exact offending token at char 5268 is identified;
|
||||
- `finish_reason` is captured to confirm or rule out token-limit **truncation**
|
||||
vs a structural mid-stream glitch;
|
||||
- it is confirmed whether llm-connect actually **enforced** the `json_schema`
|
||||
constrained-decoding hint or merely accepted it as advisory (this determines
|
||||
whether the schema param is load-bearing);
|
||||
- the failing payload is captured as a regression fixture under `tests/`.
|
||||
|
||||
## Schema + Prompt Redesign For Error Locality
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0016-T02
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: "ae67ca8c-ee01-4a8d-9e8a-a0a36c999758"
|
||||
```
|
||||
|
||||
Redesign the daily-triage report contract so a single malformed item can no
|
||||
longer discard the whole report (principle #2).
|
||||
|
||||
Done when:
|
||||
|
||||
- the recommendation list is **bounded** (configurable top-N, default 5–7) in
|
||||
both the prompt and the output schema — long lists are where the model drifts;
|
||||
- the report uses a **per-item-framed** shape (JSON Lines / NDJSON — one
|
||||
recommendation object per line — or an equivalent delimited per-item form)
|
||||
behind a minimal stable envelope (`summary` + framed items), so each item is
|
||||
an independent parse unit;
|
||||
- the prompt explicitly states the contract, the per-item framing, the cap, and
|
||||
a "if uncertain, emit fewer well-formed items rather than more" instruction;
|
||||
- `max_tokens` is set with headroom for the bounded list so truncation cannot
|
||||
occur at the expected size;
|
||||
- the output schema file (`_load_output_schema` target) is updated to match.
|
||||
|
||||
## Boundary Parser — Verify & Mitigate (Posture B)
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0016-T03
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: "d65a6281-f1f9-4a9b-a835-da065411b709"
|
||||
```
|
||||
|
||||
Implement item-granular parsing with a quarantine lane in
|
||||
`src/activity_core/rules/executor.py`, applying posture **B** at the boundary
|
||||
(principles #1–#3).
|
||||
|
||||
Done when:
|
||||
|
||||
- the parser splits the envelope from the framed items, then parses **each item
|
||||
independently**; a malformed item is routed to a bounded `quarantined_items`
|
||||
artifact (index + validation error + raw snippet), not raised;
|
||||
- a run with some valid and some invalid items emits a report over the surviving
|
||||
valid items with `output_validated=true`, plus `partial=true` and
|
||||
`quarantined_count` / `quarantined_items` markers — degraded-but-usable is
|
||||
reported distinctly from total loss;
|
||||
- a best-effort **repair** pass (close unterminated brackets/quotes, recover the
|
||||
valid prefix) is attempted per item before quarantining it;
|
||||
- truncation detected in T01 is handled as its own signal (recover whole items
|
||||
emitted before the cutoff rather than failing the document);
|
||||
- the existing monolithic-document path remains as the fallback when framing is
|
||||
absent (backward compatible with task-only instructions).
|
||||
|
||||
## Producer Guardrails + ADR-004
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0016-T04
|
||||
status: todo
|
||||
priority: medium
|
||||
state_hub_task_id: "f5c3af5b-9e28-42b0-9af5-4c99284e99b9"
|
||||
```
|
||||
|
||||
Write the architecture decision record and add the producer-agnostic guardrails
|
||||
(principle #4).
|
||||
|
||||
Done when:
|
||||
|
||||
- `docs/adr/adr-004-producer-trust-boundary.md` documents the trust boundary,
|
||||
the untrusted-producer premise (erroneous **and** malicious; human and agent),
|
||||
the A vs B taxonomy and where each applies, the error-locality principle, and
|
||||
the quarantine-with-provenance rule;
|
||||
- boundary guardrails are enforced at the consumer edge: max item **count**, max
|
||||
string length, max nesting **depth**, and a **reference allow-list** (e.g. a
|
||||
recommendation `candidate` / a task `target_repo` must resolve to a known
|
||||
workstream/repo before it is acted on);
|
||||
- guardrail rejections are quarantined with provenance, consistent with T03;
|
||||
- SCOPE.md / INTENT.md are checked for drift and updated if the boundary stance
|
||||
changes the documented contract.
|
||||
|
||||
## Tests + Calibration Re-Entry
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0016-T05
|
||||
status: todo
|
||||
priority: high
|
||||
state_hub_task_id: "c881500b-5459-4620-81c0-b176971e989f"
|
||||
```
|
||||
|
||||
Prove the new posture and hand back to the calibration gates.
|
||||
|
||||
Done when:
|
||||
|
||||
- regression tests cover: the captured 06-26 payload, a truncated-mid-list
|
||||
payload, a one-bad-item-among-good payload (asserts quarantine + partial), an
|
||||
oversized/over-deep payload (asserts guardrail rejection), and an
|
||||
injection-shaped reference (asserts allow-list rejection);
|
||||
- the full suite passes and the result is recorded here with the count;
|
||||
- a daily-triage smoke against the live runtime shows a previously-failing
|
||||
payload now **degrades gracefully** (valid items delivered, bad items
|
||||
quarantined) instead of discarding the run;
|
||||
- a progress note hands back to `ACTIVITY-WP-0010-T04` and `ACTIVITY-WP-0006-T03`
|
||||
that the output-robustness blocker is cleared so the three-clean-run gate can
|
||||
resume on its own.
|
||||
|
||||
## Relationships
|
||||
|
||||
- **Blocks / feeds:** `ACTIVITY-WP-0006-T03` (three clean scheduled runs) and
|
||||
`ACTIVITY-WP-0010-T04` (collect three clean scheduled runs) — both stalled on
|
||||
the same output-quality failure this workplan removes.
|
||||
- **References:** `ACTIVITY-WP-0009` (scheduled-run trust gap).
|
||||
- **Boundary discipline:** keeps activity-core inside its SCOPE — this hardens
|
||||
the instruction-executor output contract; it does not move provider
|
||||
credentials, cluster reconciliation, or task lifecycle into this repo.
|
||||
Reference in New Issue
Block a user