OpenAI Unveils Operator AI Agent: Executes Complex Computer Tasks via Image Understanding
OpenAI recently announced Operator, a new AI agent that can complete complex tasks by observing screen images and directly interacting with the computer interface. The agent operates autonomously in a real browser environment, supporting actions like online shopping, form filling, and data entry. Operator represents a significant advancement in AI for visual task processing, with plans to open more features to developers in the future.
Core Technical Capabilities of Operator
Operator is built upon OpenAI’s previously released computer usage model, leveraging the GPT-4o vision model and reinforcement learning to achieve image understanding and control of the mouse and keyboard. The agent observes interface elements via screenshots, makes autonomous decisions, and executes actions like clicking and typing, rather than relying on predefined scripts. In the Mind2Web benchmark, Operator achieved a success rate of 38.1%, significantly higher than the 9.7% of previous models. Additionally, its performance in the WebArena test also surpassed that of existing open-source agents.
Practical Task Execution Demo
Demonstration videos show Operator handling e-commerce shopping flows, including searching for products, adding items to the cart, and checking out. It can also complete recruitment forms by extracting information from a resume and entering it into the appropriate fields. The agent supports multi-step task planning and performs troubleshooting when it encounters obstacles, such as identifying CAPTCHAs or adjusting browser zoom. OpenAI emphasizes that Operator is currently in a research preview stage, accessible only to Pro users via the Playground, with an expected release to ChatGPT Pro users soon.
Comparison with Existing Models and Limitations
Compared to traditional text-prompt-based AI tools, Operator’s image-driven approach more closely mimics how humans use computers, avoiding reliance on webpage structure. Currently, Operator has weaker support for desktop applications, focusing mainly on web-based tasks, and its processing time is lengthy—simple tasks take several minutes, while complex ones can exceed 10 minutes. OpenAI researchers note that future improvements in speed and accuracy will come from more training data and optimization.
Future Development and Opening-Up Plan
OpenAI plans to extend Operator-related technologies to third-party developers, offering an API for integrating custom tasks. The release of this agent continues OpenAI’s strategy in the AI agent space, following the computer usage capabilities introduced with the o1 model. Reports from multiple media outlets, such as TechCrunch and The Verge, have confirmed Operator’s capabilities through public demonstrations, pushing AI’s evolution toward general-purpose agents. (Approx. 580 words)
