New Benchmark Reveals Limitations of AI Coding Capabilities
For a long time, SWE-bench has been considered the core standard for evaluating the capabilities of Code Agents. As frontier models like Gemini 3 Pro and GPT-5.2 achieved solve rates exceeding 80% on this benchmark, the industry widely believed that AI was rapidly approaching the level of human programmers. However, SWE-bench primarily assesses the ability to perform small-scale fixes within a single code repository, with each modification averaging only 1.3 files and 11.6 lines of code, a far cry from the complexity of real-world software engineering. Given that it can no longer effectively differentiate between models, institutions including OpenAI have begun seeking more challenging evaluation methods.
In response to this situation, the research team at the Gaoling School of Artificial Intelligence, Renmin University of China, released a new benchmark named BeyondSWE on March 20, 2026. By expanding two dimensions, “Resolution Scope” and “Knowledge Scope,” it systematically evaluates AI performance in scenarios that more closely resemble real-world work.
BeyondSWE: Four Tasks Simulating Real-World Software Engineering Challenges
BeyondSWE contains 500 high-quality instances from 246 GitHub repositories. On average, each task involves modifying 5.6 files and 209.9 lines of code, making it far more complex than previous benchmarks. The benchmark is composed of four core task modules, each corresponding to different quadrants of engineering capabilities.
1. CrossRepo: Cross-Repository Knowledge Retrieval and Application
This task includes 200 instances and requires the AI to fix an issue in the current repository, but the key information for the solution is hidden in external code repositories, technical forums, or Issue discussions. It aims to test the AI’s understanding of the open-source ecosystem and its ability to retrieve and integrate information across projects, rather than simple multi-repository code collaboration.
2. DomainFix: Deep Integration of Specialized Domain Knowledge
Containing 72 instances, this task was constructed in collaboration with experts from 11 disciplines, including quantum physics, bioinformatics, and convex optimization. Fixing these issues requires not only correct code syntax but also an understanding of the underlying specialized knowledge, such as physical formulas or biological principles, posing a severe test to the AI’s domain knowledge reserves and application.
3. DepMigrate: Large-Scale, Systematic Dependency Migration
This task includes 178 instances, simulating scenarios of major dependency library version upgrades, such as NumPy 1.x to 2.0 or Pydantic v1 to v2. This type of task requires the AI to perform consistent and correct global API replacements and refactoring across an entire codebase (potentially involving dozens of files), testing its systematic modification capabilities.
4. Doc2Repo: System Construction from Scratch
Containing 50 instances, this task requires the AI to build a complete software project from the ground up, based solely on a functional design document. The names and structures of the original repositories were intentionally obscured during evaluation to prevent the model from “memorizing” existing implementations. This task aims to evaluate the AI’s architectural design, module partitioning, and end-to-end implementation abilities.
Experimental Results: Top Models’ Pass Rates Plummet
The research team, using the OpenHands framework, tested several frontier models, including Gemini 3 Pro, GPT-5.2, and DeepSeek-V3.2. The results showed that no single model’s overall pass rate exceeded 45%, a significant drop compared to their performance on SWE-bench.
Specifically, the models’ failure patterns varied:
- Significant Skill Specialization: Different models performed differently on various tasks. For example, Seed-Coder led in the CrossRepo task (44.72%), while DeepSeek-V3.2 performed best in the Doc2Repo task (54.99%), and Gemini 3 Pro had the greatest advantage in the DepMigrate task (41.81%). This indicates that a universally competent AI coding model does not yet exist.
- Domain Knowledge as the Biggest Barrier: On the DomainFix task, almost all models had a pass rate below 36%, proving that specialized domain knowledge is a shortcoming that current AI struggles to overcome through large-scale code training alone.
- Insufficient System Construction Ability: Although the pass rate on the Doc2Repo task appeared to be between 45%-55%, instances where a complete, coherent system was built that passed 100% of the tests were extremely rare. This reveals that AI is adept at implementing isolated functional points but lacks the engineering mindset to build self-consistent systems.
SearchSWE: Is Web Search the Solution?
To explore whether external information retrieval could enhance AI performance, the team concurrently proposed the SearchSWE framework. This framework introduces a search engine and browser tools for the AI agent, allowing it to autonomously query information during the coding process. The experimental design included strict anti-cheating mechanisms to prevent the AI from directly searching for the answers.
The experimental results show that introducing search capabilities is not a silver bullet:
- Effectiveness Varies by Model: Among the 9 models tested, 6 showed performance improvements after integrating search, while 3 actually regressed. Search was most helpful for knowledge-intensive tasks like DomainFix, where Gemini 3 Pro’s performance improved by 7.5%.
- Quality, Not Frequency, of Search is Key: Gemini 3 Pro invoked search only 0.8-1.1 times per task on average yet achieved the highest overall gain (+2.0%); whereas DeepSeek-V3.2 searched 4.2-5.4 times on average, and its overall performance slightly decreased. This suggests that accurately judging when to search and how to utilize the results is more important than frequent searching.
The research further analyzed three root causes for the failure of search and coding integration:
- Information Landscape Mismatch: Search engines are adept at retrieving high-level documentation designed for humans, while code tasks often require source code details, commit diffs, and other “technical artifacts” that are poorly indexed.
- Version-Time Discrepancy: Search results and the model’s internal knowledge tend to favor the latest software versions, whereas task environments are often locked to older versions, leading the AI to use incompatible APIs for fixes.
- Semantic Drift and Noise Contamination: The ambiguity of technical terms can cause search to return high-authority results from irrelevant domains. An AI lacking discernment is easily misled.
Conclusion: Moving Towards “Deep Research for Coding”
The experiments with BeyondSWE and SearchSWE collectively point to one core conclusion: simply improving code generation or information retrieval as standalone capabilities is no longer sufficient. The deep integration of the two, termed “Deep Research for Coding,” is key to advancing AI coding capabilities to the next stage. BeyondSWE provides a more comprehensive evaluation framework to measure and guide research in this direction, pushing AI to evolve from a “test-taker” to an “intelligent agent” truly equipped to solve open-world engineering problems.
