Do screenshots actually make LLMs better?

I take a lot of screenshots when working with coding agents.

Mostly because it feels obvious: if I'm asking an agent to fix something I'm looking at, showing it the thing should be better than trying to describe it.

But I wanted to know if there was any actual evidence for that.

There is. Just not quite the evidence I originally expected.

The strongest evidence comes from agents using interfaces

One of the better studies is WebVoyager, published at ACL 2024.

The researchers built an agent that interacted with 15 real websites including Amazon, GitHub, Booking.com, Google Flights and Google Maps.

They then compared the multimodal agent, which received screenshots together with information about interactive elements, against a text-only version of the same system.

The difference was substantial:

That's a 19 percentage-point increase, or roughly a 47% relative improvement in completed tasks.

The result wasn't just coming from one or two websites. But the size of the benefit varied enormously depending on the type of interface.

Google Flights is probably the clearest example:

Booking.com went from 2.3% to 43.2%.

Coursera went from 23.8% to 73.8%.

On GitHub, however, the difference was much smaller: 61.0% vs 63.4%.

And on Allrecipes the text-only version actually performed slightly better.

That distinction is important.

Screenshots aren't universally better. They become particularly valuable when the information needed to solve the task is represented visually rather than cleanly in text.

A webpage isn't just its text

A browser technically has access to the DOM, accessibility tree, page text and all sorts of structured information.

But that's not necessarily the same thing as seeing the page.

Humans design interfaces around spatial relationships:

• this button belongs to this form
• this error message appeared below this field
• this card is selected
• this menu is covering something
• these two elements aren't aligned
• this modal is currently open
• this chart has a visible spike

Flatten that into text and some of the information disappears.

WebVoyager's authors make essentially this point: web pages are rendered specifically to turn underlying HTML into a visual representation that's easier for humans to understand. Their system therefore uses screenshots as part of its observation of the environment.

Other benchmarks point in the same direction

VisualWebArena was created specifically because many realistic web tasks cannot be represented properly as text-only tasks.

Its benchmark includes tasks where agents have to understand both images and the visual state of a website. The authors found clear limitations in text-only agents when completing visually grounded tasks.

Then there's OSWorld, which tests agents operating actual desktop applications rather than just websites.

OSWorld lets agents observe the computer through screenshots, accessibility information or combinations of the two.

One interesting result from their analysis is that higher screenshot resolution generally improved agent performance. They also found that agents remain sensitive to UI layout and visual noise, which suggests that models aren't simply extracting text from screenshots: the quality and structure of the visual information itself matters.

More recent work is moving even closer to software development. VISTA, released in 2026, evaluates coding agents building web interfaces from combinations of text descriptions, screenshots and Figma-derived structure. The benchmark exists partly because traditional coding benchmarks don't capture UI development particularly well.

But this doesn't prove screenshots make coding agents 47% better

This is where it's easy to overstate the research.

WebVoyager does not show that adding a screenshot to Claude Code, Codex or another coding agent improves software engineering performance by 47%.

It shows something narrower:

That's still useful.

A lot of frontend development is exactly that kind of problem.

Consider the difference between these two prompts:

Text only

versus:

Screenshot

The screenshot carries information that would otherwise have to be reconstructed through language: position, hierarchy, spacing, color, state and what the user actually considers wrong.

And unlike a written description, the screenshot is direct evidence of the rendered result.

Screenshots are another form of context

We already accept that coding agents perform better when they can access the codebase.

Instead of describing a function, we give them the function.

Instead of explaining a type, we give them the definition.

Instead of summarizing a stack trace, we give them the stack trace.

UI work has another important source of truth: the rendered interface.

For that class of problem, a screenshot isn't decoration attached to the prompt.

It's context.

And the research we have so far suggests models can make meaningful use of it.

The experiment I'd like to see

What we don't yet have is the experiment I'd really like to see:

1. Take the same set of real frontend bugs.
2. Give the same coding agent the same repository and prompt.
3. For half the runs, include a screenshot.
4. Then measure whether the resulting code actually fixes the problem.

That would tell us much more directly how valuable screenshots are for coding agents.

My suspicion is that the difference would be largest for exactly the tasks you'd expect: layout problems, styling bugs, incorrect states and situations where describing what is wrong takes longer than simply showing it.

For now, the evidence doesn't justify saying:

It does justify something more specific:

And for frontend development, a surprising amount of the problem is visual.

References

• He, H. et al. (2024). WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. Proceedings of ACL 2024. DOI: 10.18653/v1/2024.acl-long.371.
• Koh, J. Y. et al. (2024). VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. arXiv:2401.13649.
• Xie, T. et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv:2404.07972.
• Guo, J. et al. (2026). VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents. arXiv:2605.26144.