Anthropic has introduced a brand-new AI model that adds computer usage functionality, enabling it to simulate human operations with a mouse and keyboard. This model is already integrated into the Claude series and places a special emphasis on a safety-aligned design.
Core Computer Usage Capabilities
The new model allows developers to guide Claude via API to view computer screenshots, precisely calculate the necessary pixel shifts for cursor movement, and then execute clicks and keyboard inputs. This capability enables the AI to use existing software tools just like a human, completing multi-step tasks without relying on custom interfaces and can self-correct and retry when encountering obstacles.
Technical Implementation and Training Highlights
Anthropic trained the model by combining image interpretation with action-reasoning capabilities. After learning in simple software environments, the model quickly generalized its skills, able to break down written commands from users into logical steps and execute them. The training focused on solving precision problems like pixel counting to ensure operational accuracy.
Performance Benchmark Results
On the OSWorld computer usage benchmark, the model scored 14.9%, significantly higher than the next-best model’s 7.7%. Although this is not yet at the human level of 70-75%, it already holds a leading position in its category. Concurrently, the upgraded Claude 3.5 Sonnet also shows improvements in related tasks such as coding.
Safety Alignment and Risk Management
Anthropic has assessed and confirmed that the updated model, which includes computer usage skills, still maintains AI Safety Level 2 and does not elevate frontier risk requirements. The company has analyzed potential vulnerabilities such as prompt injection and implemented monitoring measures for election-related activities. The public beta phase emphasizes that users need to take precautionary measures to mitigate the risk of misuse.
Availability and Platform Support
The computer usage feature is currently in public beta and can be accessed through the Anthropic API, as well as on platforms like Amazon Bedrock and Google Vertex AI. This breakthrough offers developers a new pathway to build desktop-level AI applications, driving the practical implementation of AI in real-world workflows.
