---
title: "Do screenshots actually make LLMs better?"
description: "What the research says about giving AI agents screenshots. In WebVoyager, screenshots raised task success from 40% to 59%, with very large gains on visual sites and none on others. What that does and doesn't tell us about coding agents."
url: https://www.bigshot.pics/blog/do-screenshots-make-llms-better
---
# Do screenshots actually make LLMs better?

*2026-10-29*


I take a lot of screenshots when working with coding agents.

Mostly because it feels obvious: if I'm asking an agent to fix something I'm
looking at, showing it the thing should be better than trying to describe it.

But I wanted to know if there was any actual evidence for that.

There is. Just not quite the evidence I originally expected.

## The strongest evidence comes from agents using interfaces

One of the better studies is
[WebVoyager](https://aclanthology.org/2024.acl-long.371/), published at ACL
2024.

The researchers built an agent that interacted with 15 real websites
including Amazon, GitHub, Booking.com, Google Flights and Google Maps.

They then compared the multimodal agent, which received screenshots together
with information about interactive elements, against a text-only version of
the same system.

The difference was substantial:

| Agent | Task success |
|---|---|
| Text-only WebVoyager | 40.1% |
| Multimodal WebVoyager | 59.1% |

That's a 19 percentage-point increase, or roughly a 47% relative improvement
in completed tasks.

The result wasn't just coming from one or two websites. But the size of the
benefit varied enormously depending on the type of interface.

Google Flights is probably the clearest example:

| Input | Success |
|---|---|
| Text only | 7.1% |
| Screenshot + text | 59.5% |

Booking.com went from 2.3% to 43.2%.

Coursera went from 23.8% to 73.8%.

On GitHub, however, the difference was much smaller: 61.0% vs 63.4%.

And on Allrecipes the text-only version actually performed slightly better.

![WebVoyager task success with text only and with screenshots, for all 15 websites and five of them.](https://www.bigshot.pics/blog/webvoyager-screenshots.svg)

That distinction is important.

Screenshots aren't universally better. They become particularly valuable
when the information needed to solve the task is represented visually rather
than cleanly in text.

## A webpage isn't just its text

A browser technically has access to the DOM, accessibility tree, page text
and all sorts of structured information.

But that's not necessarily the same thing as seeing the page.

Humans design interfaces around spatial relationships:

- this button belongs to this form
- this error message appeared below this field
- this card is selected
- this menu is covering something
- these two elements aren't aligned
- this modal is currently open
- this chart has a visible spike

Flatten that into text and some of the information disappears.

WebVoyager's authors make essentially this point: web pages are rendered
specifically to turn underlying HTML into a visual representation that's
easier for humans to understand. Their system therefore uses screenshots as
part of its observation of the environment.

## Other benchmarks point in the same direction

[VisualWebArena](https://arxiv.org/abs/2401.13649) was created specifically
because many realistic web tasks cannot be represented properly as text-only
tasks.

Its benchmark includes tasks where agents have to understand both images and
the visual state of a website. The authors found clear limitations in
text-only agents when completing visually grounded tasks.

Then there's [OSWorld](https://os-world.github.io/), which tests agents
operating actual desktop applications rather than just websites.

OSWorld lets agents observe the computer through screenshots, accessibility
information or combinations of the two.

One interesting result from their analysis is that higher screenshot
resolution generally improved agent performance. They also found that agents
remain sensitive to UI layout and visual noise, which suggests that models
aren't simply extracting text from screenshots: the quality and structure of
the visual information itself matters.

More recent work is moving even closer to software development.
[VISTA](https://arxiv.org/abs/2605.26144), released in 2026, evaluates coding
agents building web interfaces from combinations of text descriptions,
screenshots and Figma-derived structure. The benchmark exists partly because
traditional coding benchmarks don't capture UI development particularly
well.

## But this doesn't prove screenshots make coding agents 47% better

This is where it's easy to overstate the research.

WebVoyager does not show that adding a screenshot to Claude Code, Codex or
another coding agent improves software engineering performance by 47%.

It shows something narrower:

> When an agent has to reason about a visual interface, giving it access to
> that visual interface can substantially improve its ability to complete the
> task.

That's still useful.

A lot of frontend development is exactly that kind of problem.

Consider the difference between these two prompts:

**Text only**

> The padding around the button looks wrong and the card underneath isn't
> aligned properly.

versus:

**Screenshot**

> Fix this.

The screenshot carries information that would otherwise have to be
reconstructed through language: position, hierarchy, spacing, color, state
and what the user actually considers wrong.

And unlike a written description, the screenshot is direct evidence of the
rendered result.

## Screenshots are another form of context

We already accept that coding agents perform better when they can access the
codebase.

Instead of describing a function, we give them the function.

Instead of explaining a type, we give them the definition.

Instead of summarizing a stack trace, we give them the stack trace.

UI work has another important source of truth: the rendered interface.

For that class of problem, a screenshot isn't decoration attached to the
prompt.

It's context.

And the research we have so far suggests models can make meaningful use of
it.

## The experiment I'd like to see

What we don't yet have is the experiment I'd really like to see:

1. Take the same set of real frontend bugs.
2. Give the same coding agent the same repository and prompt.
3. For half the runs, include a screenshot.
4. Then measure whether the resulting code actually fixes the problem.

That would tell us much more directly how valuable screenshots are for
coding agents.

My suspicion is that the difference would be largest for exactly the tasks
you'd expect: layout problems, styling bugs, incorrect states and situations
where describing what is wrong takes longer than simply showing it.

For now, the evidence doesn't justify saying:

> "Screenshots make LLMs 47% better."

It does justify something more specific:

> When the problem is visual, giving the model visual context can make a
> very large difference.

And for frontend development, a surprising amount of the problem is visual.

## References

- He, H. et al. (2024).
  [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://aclanthology.org/2024.acl-long.371/).
  Proceedings of ACL 2024. DOI: 10.18653/v1/2024.acl-long.371.
- Koh, J. Y. et al. (2024).
  [VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks](https://arxiv.org/abs/2401.13649).
  arXiv:2401.13649.
- Xie, T. et al. (2024).
  [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://arxiv.org/abs/2404.07972).
  arXiv:2404.07972.
- Guo, J. et al. (2026).
  [VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents](https://arxiv.org/abs/2605.26144).
  arXiv:2605.26144.