L4 Acting in the world
Perception and grounding
Lets a browser or screen agent act on named elements, look again before every move, report missed actions plainly, and keep screenshots from flooding the context.
Why it matters
It is what separates a working browser or desktop agent from a demo. Without it the agent clicks guessed positions on an old view, believes failed clicks worked, and fills its context with stale screenshots.
What to build
Act only on snapshot element ids
The default the research recommends. The reasoning and the code behind it are in the app.
What it takes, by kind of agent
- Conversational assistantNot applicable
- CodingNot applicable
- Deep researchNot applicable
- Browser/GUIMust-have
How well it is studied
- Harnesses checked
- 24
- Checked facts behind it
- 142
See the 142 checked facts and how each harness does it
The facts, the code lines they come from and the rules for this topic are in the app. Sign in with Google; new accounts are approved by hand.
Also in L4 Acting in the world
- Tool dispatchSettled practice
Defines each tool once, checks every call before it runs, and hands every failure back to the model as a result it can fix.
- Assistant Must-have
- Coding Must-have
- Research Must-have
- Browser Must-have
- Extension hooksEmerging
Fixed points in the loop where plugins can watch, change or veto what the agent does.
- Assistant Recommended
- Coding Recommended
- Research Optional
- Browser Optional