Anthropic Unveils ‘Computer Use’ Feature, Enabling Desktop-Level Interaction for its AI Coding Assistant
AI company Anthropic recently released a research preview feature named ‘Computer Use’ for its AI coding assistant, Claude Code. This feature grants the Claude model the ability to directly interact with a user’s desktop environment, allowing it to observe the screen, control the mouse, and use the keyboard to complete complex software development and testing tasks. The aim is to extend the AI’s capabilities from mere code generation to the actual validation and debugging of code, building a more complete, closed-loop automated development cycle.
How AI ‘Sees’ and ‘Operates’ a Computer
The core of the ‘Computer Use’ feature is equipping Claude with visual perception and the ability to operate a Graphical User Interface (GUI). After a user gives a command in natural language, Claude can perform a series of actions just like a human. Its main use cases include:
- Native App Building and Validation: A developer can ask Claude to write a macOS application. The AI will automatically write the code (e.g., in Swift), use the Xcode command-line tool
xcodebuild to compile it, and then launch the app and click through the UI elements to validate its functionality, finally reporting the results to the user with screenshots.
- End-to-End UI Automation Testing: Traditional UI testing relies on frameworks like Playwright and tedious script writing. Now, a user only needs to describe the test flow in natural language (e.g., ‘test the user registration flow’), and Claude can independently open the application, fill out forms, click buttons, and take screenshots of key steps, all without writing any test code.
- Visual Bug Debugging: For visual bugs like ‘the content of a pop-up is obscured when the window is resized,’ Claude can understand the problem description, proactively resize the window to reproduce the issue, locate the relevant CSS or layout code, make changes, and then re-verify the fix.
- Automation for Tools Without APIs: For legacy GUI software that doesn’t offer a Command-Line Interface (CLI) or Application Programming Interface (API), such as some design tools or hardware control panels, this feature opens up new possibilities for automating their operation.
Technical Implementation, Safety Mechanisms, and Activation
Anthropic has designed a thorough technical implementation and robust safety controls for this feature to ensure a secure and positive user experience.
On the technical implementation side, the model has intelligent tool selection capabilities. It will prioritize more precise and efficient methods to perform tasks, such as using an internal server (MCP Server), the command line (Bash), or browser plugins. Only when these methods are insufficient will it resort to desktop control as a ‘last resort’.
Regarding safety mechanisms, the feature follows a multi-layered authorization model and the principle of least privilege:
- Per-Application Authorization: Before interacting with a new application, Claude will prompt the user for confirmation, preventing unauthorized access. For high-privilege applications like Terminal or Finder, the system will display additional risk warnings.
- Focused Workspace: While performing an action, the system automatically hides irrelevant application windows, keeping only the target application and the terminal visible. The original layout is restored upon completion, ensuring the AI does not ‘see’ or interact with unauthorized content.
- Instant Interruption: Users can press the
Esc key at any time to immediately terminate all of Claude’s actions and regain control.
The activation process is relatively simple. In an interactive session with Claude Code, the user needs to enable the computer-use service with a specific command and, upon first use, grant ‘Accessibility’ and ‘Screen Recording’ permissions in macOS System Settings.
Current Limitations and Impact on the Development Field
Currently, the ‘Computer Use’ feature is still in the ‘research preview’ stage and has specific scope and limitations:
- Platform: Only supports macOS.
- Subscription: Only available to Claude Pro and Max subscribers.
- Version: Requires Claude Code v2.1.85 or higher.
- Environment: Can only be used in interactive sessions on the official claude.ai website. It is not supported via API calls or on third-party cloud platforms (like Amazon Bedrock, Vertex AI).
- Concurrency: Only one session can use the feature on a single device at any given time.
Despite these limitations, the feature already shows immense potential to reshape parts of the software development and testing workflow. For indie developers and startup teams, it provides a tireless ‘virtual tester’ that can significantly reduce the human cost of regression testing and basic feature validation. For professional QA engineers, it acts as a powerful efficiency multiplier, freeing them from repetitive, mechanical manual testing to focus on higher-value work like complex test strategy design, performance analysis, and exploratory testing.
From a broader perspective, Anthropic is driving the evolution of AI from the role of a ‘coder’ to that of a ‘software engineer.’ An AI that can write, compile, run, test, and fix its own code heralds the arrival of a new era of truly end-to-end automated software development.
