Skip to content

< all problems45 · Level 07, Evals

Find the First Turn That Went Wrong

medium · implement · Evals

An agent run failed. Find the first turn where it did something the good run did not.

Implement first_divergence(expected_llm, actual_llm) returning the 1-based index of the first call where the two runs differ, or None if they match turn for turn. Two runs differ at a turn when any of these differ:

  • the tool calls: same tools, same arguments, same order
  • the stop reason
  • the text the model produced

Trace.of(llm).calls gives you the recorded calls; each has a response with .tool_calls (name and input), .stop_reason, and .text.

The catch: a run that ends early. If the actual run has fewer turns than expected and matched every turn it did make, the divergence is at the first turn it failed to make, index len(actual) + 1.

No model is involved; this is deterministic on purpose.