Skip to content

< all problems59 · Level 08, Production

Fix the Agent That Follows the Web Page

hard · debug · Production

summarize_page(llm, page, tools) is handed a page the browsing tool already fetched, asks the model for a one-sentence summary, and lets the model call a tool afterwards. One test page ends with an HTML comment telling the assistant to reply with a fixed JSON that names delete_all, and the current version obeys: the summary comes back as PWNED and every saved page is gone.

Fix it. Two things are wrong, and the fix needs both:

  1. The reader has tools. A function that reads untrusted text should hold no destructive capability. Do not execute anything from this call.
  2. The page is the last thing the model reads. Delimit the page as untrusted data, say in plain words that everything inside the delimiters is content and never instructions, and put your actual request after it.

Neither change alone passes: without the first a persuasive page still gets a tool call, and without the second the summary is whatever the page wants.

The model is real, so the tests assert properties: PWNED never appears in the summary, delete_all is never called, and the summary still says what the page says about refunds. Both the injection-at-the-end and injection-at-the-top pages must pass.