Computer-Use Agent
Also called Computer use, Computer-using agent, CUA
An AI agent that operates a computer the way a person does: it looks at screenshots, then moves the mouse, clicks and types.
What is a computer-use agent?
A computer-use agent is an AI agent that controls a computer through its screen, keyboard and mouse, rather than through an API. You give it a task, such as "download last month's invoices and put the totals in a spreadsheet", and it works through the apps on screen to do it.
How does it work?
The agent runs in a loop:
- It takes a screenshot and a multimodal model works out what's on screen.
- The model decides the next action: click here, type this, scroll, press a key.
- The software carries out that action through tool calling, takes a new screenshot, and repeats until the task is done or it gets stuck.
Agents like this usually run in a virtual machine or sandbox, so mistakes can't damage the real system.
What is it commonly confused with?
- A browser agent. A browser agent works only inside a web browser; a computer-use agent can operate any desktop app.
- Traditional automation. Scripts and robotic process automation follow fixed steps and break when a screen changes. A computer-use agent reads the screen each time, so it can cope with changes, but it's slower and less predictable.
Why does it matter?
Computer use lets AI work with software that has no API, which describes a great deal of real business software. It's still slower and more error-prone than API integrations, and it carries real risks: on-screen text can contain prompt injection, and an agent with your sign-ins can take actions you didn't intend. Keep a person in the loop for anything irreversible, such as payments, sending messages or deleting data (human in the loop).
