Log raw LLM output preview on instruction validation failure

The CUST-WP-0045 canary failed validation twice without leaving any
record of what the model actually returned. The warning logged only the
error message ($: missing required property 'summary'), not the JSON
shape that triggered it — so diagnosing required modifying code and
re-running. Log a 2KB preview of the offending raw output alongside the
error so the next failure of this shape is one grep away from diagnosis.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-06-02 13:06:43 +02:00
parent c79d0980a9
commit 4b4e162c44

View File

@@ -156,9 +156,13 @@ def _execute(
raw_output = llm_client.complete(retry_prompt, model=instr.model, config=llm_config)
task_specs, report, error = _validate_output(raw_output, instr)
if error:
# Truncate to keep log volume bounded but long enough to see the
# actual JSON shape mismatch (typical reports are <2KB).
preview = (raw_output or "")[:2000]
logger.warning(
"instruction_output_error: instruction=%r, prompt_hash=%s, error=%s",
instr.id, prompt_hash, error,
"instruction_output_error: instruction=%r, prompt_hash=%s, "
"error=%s, raw_output_preview=%r",
instr.id, prompt_hash, error, preview,
)
return _empty_result(instr, prompt_hash=prompt_hash)