generated from coulomb/repo-seed
Log raw LLM output preview on instruction validation failure
The CUST-WP-0045 canary failed validation twice without leaving any record of what the model actually returned. The warning logged only the error message ($: missing required property 'summary'), not the JSON shape that triggered it — so diagnosing required modifying code and re-running. Log a 2KB preview of the offending raw output alongside the error so the next failure of this shape is one grep away from diagnosis. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -156,9 +156,13 @@ def _execute(
|
||||
raw_output = llm_client.complete(retry_prompt, model=instr.model, config=llm_config)
|
||||
task_specs, report, error = _validate_output(raw_output, instr)
|
||||
if error:
|
||||
# Truncate to keep log volume bounded but long enough to see the
|
||||
# actual JSON shape mismatch (typical reports are <2KB).
|
||||
preview = (raw_output or "")[:2000]
|
||||
logger.warning(
|
||||
"instruction_output_error: instruction=%r, prompt_hash=%s, error=%s",
|
||||
instr.id, prompt_hash, error,
|
||||
"instruction_output_error: instruction=%r, prompt_hash=%s, "
|
||||
"error=%s, raw_output_preview=%r",
|
||||
instr.id, prompt_hash, error, preview,
|
||||
)
|
||||
return _empty_result(instr, prompt_hash=prompt_hash)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user