AI Explained
Computer-use agents: AI that uses your screen
What if AI could use a computer like you do? Computer-use agents look at the screen, then click and type. How they work, where they fail, and how to run them safely.
On this page
Most AI tools work through APIs. A computer-use agent works through the screen: it looks at what's displayed, then moves the mouse and types on the keyboard, just like you do. Anthropic and OpenAI both offer models that can do this, through a tool you call from your own code.
Why it matters
A lot of software has no API. Old admin panels, desktop apps, internal tools and forms built only for people. Until now, automating them meant brittle scripts. A computer-use agent can operate them the way a person would.
How it works: the agent loop
- The agent takes a screenshot.
- The model looks at it and picks the next step: click here, scroll, type this.
- Your app performs that action.
- It takes another screenshot to see what changed.
- Repeat until the task is done.
That repetition is the agent loop. Importantly, the model never touches your computer directly. As Anthropic's docs explain, your application translates each request into a real action and sends back the result. To move faster, models can also batch a few actions (click, type, then screenshot) in one step.
An example
Ask it to file an expense report. It opens the expense portal, finds the right form, fills in each field, attaches the receipt and, before pressing Submit, stops and asks you. A good agent pauses at the step with real consequences.
How good are they?
Getting better fast, but not human level yet. When OpenAI introduced its Computer-Using Agent in January 2025, it scored 38.1% on the OSWorld benchmark of real computer tasks. People scored 72.4%. Newer models score higher, but the gap is still there.
Known weak spots:
- They're slower than a person.
- Picking the exact spot to click is hard, so misclicks happen.
- They get lost in unfamiliar or niche apps.
The big risk: everything on screen is input
A malicious web page can hide instructions that hijack the agent, a form of prompt injection. So follow the safety basics both providers recommend:
- Run the agent in a sandboxed virtual machine or container with minimal privileges.
- Keep passwords and login details out of its reach.
- Allow only the sites it needs.
- Have a human confirm anything with real-world consequences.
Where it shines
Testing web apps end to end, moving data between old tools that don't talk to each other, and boring, repetitive forms.
No API? Use computer use. Got an API? Use the API.