Definition
Plain language
An AI that uses a website the way a person does — looking at screenshots and clicking — instead of reading the underlying code.
As stated in the literature
A vision-language model wired to a real browser that perceives pages as screenshots and acts via mouse and keyboard operations, rather than through structured DOM or API access.
Also called: visual web agents
Why it matters: It matters because perceiving sites visually lets an agent use the vast web that offers no clean code or API access, the way real people do.
For example, to book a flight, a visual web agent looks at a screenshot of the airline's site and clicks the date and seat just as a person would, rather than reading the page's code.
Heard on the show
“A visual web agent — let's ground what that even is.”Episode 111 — How a 4B Web Agent Beat Models 60x Its Size on 500 Demonstrations