AISHPerf Benchmark Released: The First AIOps Evaluation System to Test LLMs’ Real-World Troubleshooting Skills
Industry Dilemma: Lack of a Unified Standard for Evaluating AIOps Agents
As the capabilities of Large Language Models (LLMs) expand, AI agents are transitioning from auxiliary tasks like content generation and information retrieval to core production roles such as software development, network management, and AI Infrastructure (AI Infra) operations. In the operations domain, using AI agents to analyze alerts, troubleshoot faults, and manage resources to improve efficiency and reduce labor costs has become an industry trend.
However, the operational environment of AI computing clusters is extremely complex, spanning multiple technical layers including GPU hardware, RDMA networks, distributed storage, container orchestration (like Kubernetes), and deep learning frameworks. Fault symptoms are often vague, incomplete, or even contradictory, requiring agents to possess comprehensive abilities to actively explore, validate hypotheses, and independently define problem boundaries and solution paths in an open environment. Most current industry benchmarks for models focus on question-answering and cannot effectively measure an agent’s real-world performance in multi-step reasoning, tool usage, and open-ended decision-making. This has left the development of AIOps agents in a state of “lacking a standard,” with no objective basis for progress evaluation and directional iteration.
Data-Driven: From Billions of Real Data Points to 103 High-Fidelity Test Cases
To address this industry pain point, the China Academy of Information and Communications Technology (CAICT), in collaboration with AI-native infrastructure company Infinigence, has developed and open-sourced the world’s first agent evaluation benchmark for AI Infra operations scenarios—AISHPerf.
The benchmark’s construction is rooted in real production environments. The technical team mined over ten billion real-world operational data points accumulated by Infinigence since its inception, with data sources from early 2024 to 2026 including user tickets, instant messaging records, operational knowledge bases, and monitoring alert logs from online clusters. After multiple rounds of rigorous data cleaning, anonymization, and deduplication, and removing non-generalizable cases tightly bound to specific business logic, researchers further refined and abstracted 100,000 valid data points into 103 high-fidelity evaluation cases.
These 103 cases feature:
- Broad Coverage: Problem types are categorized into five major classes: host machines, high-performance devices (GPU/NPU), container platforms, training/inference scripts, and security/service providers. They cover 44 specific problem phenomena and 22 sub-fault domains, comprehensively spanning common and difficult scenarios in AI infrastructure operations.
- Difficulty Grading: All cases are classified into three levels—easy, medium, and hard—based on their complexity. The average manual resolution time for these cases is 1.5 hours, ensuring the benchmark’s challenge.
- Domestic Hardware Compatibility: AISHPerf is the first benchmark to include domestic computing platforms in its evaluation scope, covering specific compatibility and operational issues encountered with mainstream domestic chips such as Iluvatar CoreX, Biren Technology, MetaX, Moore Threads, and Ascend, enhancing the benchmark’s practical relevance.
Paradigm Shift: From “Question-Answering” to “Hands-On Troubleshooting”
AISHPerf’s greatest innovation lies in its evaluation paradigm. It moves away from the traditional “written test” model of benchmarks, where a model generates an answer directly from a problem description. Instead, it introduces a “hands-on” model that emphasizes the agent’s ability to interact with a real environment.
At the start of an evaluation, the system provides the agent with only a limited user problem description, without revealing any clues about the root cause. The agent must operate within a real, pre-configured faulty environment, independently using tools like the Shell to view logs, execute diagnostic commands, and analyze system status. This series of actions allows it to gradually reproduce the problem, locate the cause, and finally implement a fix. This open-ended evaluation closely simulates the workflow of a real-world operations engineer.
To support this evaluation model, the project has released two core tools:
- AIops-Chaos: A fault injection engine for GPU clusters. It accurately simulates hardware and network anomalies specific to AI computing clusters at the software level—such as GPU device loss, VRAM errors, NVLink communication failures, and network partitioning. This allows for the creation of high-fidelity, reproducible test environments without damaging physical equipment, significantly reducing validation costs and risks.
- AIops-Eval: An end-to-end evaluation toolchain. Through its five core modules—User, Agent, Env, Evaluator, and Tracing—this toolchain can record and track every decision and action taken by the agent. This enables evaluation to go beyond just the final success or failure, allowing for in-depth analysis of whether the problem-solving path was efficient and logical. This prevents agents from getting high scores by simply “guessing” correctly, ensuring the rigor of the assessment.
Initial Tests Reveal: Top LLMs Still Struggle with Complex Operations
The research team used AISHPerf to conduct preliminary tests on several leading domestic and international LLMs, including Claude Sonnet. To ensure fairness, the test environment only provided a basic Shell tool and prohibited models from accessing the internet, forcing them to rely entirely on their own knowledge and reasoning abilities.
The results showed that while all participating models were significantly faster than human engineers, their overall scores were all below 50. This indicates that even the most advanced LLMs are far from being mature and reliable enough for independently handling complex AI infrastructure operations tasks.
Further analysis revealed the following trends:
- Difficulty Challenge: As the problem difficulty increased from medium to hard, the success rate of all models dropped sharply, with most falling below 50%. Although models attempted to gather more information by increasing tool calls, the efficiency of information utilization was low and did not translate into higher success rates.
- Capability Bias: Models were generally better at handling software-level issues like code logic and software configuration. However, they performed poorly when faced with infrastructure-level problems such as GPU hardware failures and network device anomalies, often consuming more resources without being able to locate the root cause.
- Typical Failure Modes: By analyzing numerous failed cases, the team summarized three typical problem categories: Stability Issues (e.g., incorrect tool call format, abnormal process termination), Low-Quality Reasoning Chains (e.g., solving superficial problems without addressing the root cause, reasoning without evidence), and Security Concerns (e.g., executing high-risk commands like deleting data).
These findings not only demonstrate that the AISHPerf benchmark has good discriminative power to effectively reveal the shortcomings of different models but also point to specific directions for optimizing LLMs for professional domains. The release of AISHPerf establishes a crucial, unified standard for measuring and improving the practical capabilities of AIOps agents, poised to drive the entire “AI for Infra” field toward more pragmatic and in-depth development.
