coulomb/activity-core
2
0
Fork 0
generated from coulomb/repo-seed
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Log raw LLM output preview on instruction validation failure

Browse Source 
The CUST-WP-0045 canary failed validation twice without leaving any
record of what the model actually returned. The warning logged only the
error message ($: missing required property 'summary'), not the JSON
shape that triggered it — so diagnosing required modifying code and
re-running. Log a 2KB preview of the offending raw output alongside the
error so the next failure of this shape is one grep away from diagnosis.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Bernd Worsch
2026-06-02 13:06:43 +02:00
parent c79d0980a9
commit 4b4e162c44
1 changed files with 6 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
src/activity_core/rules/executor.py
View File

@@ -156,9 +156,13 @@ def _execute(
raw_output = llm_client.complete(retry_prompt, model=instr.model, config=llm_config) raw_output = llm_client.complete(retry_prompt, model=instr.model, config=llm_config)
task_specs, report, error = _validate_output(raw_output, instr) task_specs, report, error = _validate_output(raw_output, instr)
if error: if error:
# Truncate to keep log volume bounded but long enough to see the
# actual JSON shape mismatch (typical reports are <2KB).
preview = (raw_output or "")[:2000]
logger.warning( logger.warning(
"instruction_output_error: instruction=%r, prompt_hash=%s, error=%s", "instruction_output_error: instruction=%r, prompt_hash=%s, "
instr.id, prompt_hash, error, "error=%s, raw_output_preview=%r",
instr.id, prompt_hash, error, preview,
) )
return _empty_result(instr, prompt_hash=prompt_hash) return _empty_result(instr, prompt_hash=prompt_hash)