AI that moves the mouse: Anthropic’s computer use, and the questions it raises

On 22 October 2024 Anthropic released an upgraded Claude 3.5 Sonnet, a new Claude 3.5 Haiku, and a capability in public beta called computer use: the model looks at a screen, moves the cursor, clicks and types, the way a person does.

Anthropic was unusually plain about the state of it. The feature is experimental, it is slow, and it fails at things people find trivial. That candour is the most useful part of the announcement.

Why it is different from what came before

Until now, connecting a model to a business system meant an integration: a defined interface, a named account, a list of permitted operations. Narrow, auditable, and slow to build.

Computer use skips that. If the model can drive the screen, it can use any application a person can use, including ones with no API at all. The thirty-year-old tool your production planning still runs on suddenly becomes reachable.

That is genuinely attractive for the long tail of software no vendor will ever integrate. It is also where the caution belongs.

The question it changes

A traditional integration is limited by what the interface exposes. A model driving a screen is limited by what the logged-in user can reach — which in most SMEs is considerably more.

Give an agent a desktop session and it inherits the whole session: the file shares, the mail client, the ERP, the browser with its saved passwords. The permission you granted was « use this computer ». The permission you actually granted was everything that account can do.

There is a second issue Anthropic flagged directly. A model reading a screen cannot always tell content from instruction. Text on a web page that says « ignore your previous instructions » is, to a screen-reading agent, just more screen.

A sensible position for October 2024

• Treat it as a preview, not a product. Nothing in production.
• If you experiment, use a separate machine and a separate account with only the access that experiment needs.
• Never give an agent a session that can approve a payment, send external mail, or delete records.
• Write down what it did. An action with no trace is unauditable by construction.

The direction is clear and this is the first honest step in it. Watch it closely, and keep it away from your production ledger for now.