A New Framework for Diagnosing AI Security: The Evolutionary Path from Individual Failures to Systemic Collapse
As AI agents are widely deployed in high-stakes domains like financial trading, medical diagnostics, and industrial control, their security has become a prerequisite for their application. However, current research on AI agent security is largely confined to static analysis, such as slicing the lifecycle into data, training, and deployment phases, or examining components like the brain and memory in isolation. This approach fails to address a fundamental question: How do the nature of security threats evolve as an agent’s autonomy increases?
To address this structural gap, a research team from Nanjing University of Aeronautics and Astronautics, The Chinese University of Hong Kong, and Zhejiang University released a study in March 2026, proposing a security framework named “Hierarchical Autonomy Evolution” (HAE). Published on the preprint server arXiv, the study’s core thesis is that security threats to agents are not static but evolve through three qualitatively distinct levels as their autonomy advances.
From Cognitive to Collective: The Three-Tier Evolution of Security Risks
The HAE framework moves beyond traditional static classifications, instead using “autonomy evolution” as a vertical axis to construct a dynamic threat analysis model. This model divides agent evolution into three stages, revealing the complete chain of risk progression from internal cognitive errors to external physical damage, and ultimately to societal-level systemic collapse.
L1 - Cognitive Autonomy: At this stage, the agent acts as a “thinker,” equipped with internal reasoning, planning, and memory retrieval capabilities. Security threats are primarily concentrated at the cognitive level, such as hijacking its goals through Indirect Prompt Injection or implanting misinformation via Memory Corruption. The consequences at this stage are mainly informational errors, with relatively transient and controllable impacts.
L2 - Executional Autonomy: The agent evolves into an “executor,” capable of interacting with the external world by using tools, APIs, or physical actuators. Security risks escalate from “saying the wrong thing” to “doing the wrong thing.” Threat types shift to Tool Abuse, Confused Deputy attacks, and Unsafe Action Chains. For instance, a deceived agent might execute commands to delete files or send malicious emails, causing irreversible real-world harm.
L3 - Collective Autonomy: At the highest level, multiple agents form cooperative networks through Agent-to-Agent (A2A) protocols, exhibiting social behavior. Threats accordingly escalate to the systemic level, including Malicious Collusion where multiple agents coordinate harmful activities, Viral Infection where harmful payloads spread through the network, and Systemic Collapse resulting from dependency chains or resource competition. Risks at this stage are emergent and contagious, with destructive potential far exceeding the linear sum of individual agent failures.
Cross-Layer Risk Propagation and Amplification Mechanisms
A core insight of the HAE framework is that security risks do not exist in isolation at a specific level but can propagate and be amplified across layers. The research team reveals this mechanism by analyzing the four core components of an agent: Perception, Brain, Memory, and Action.
A typical attack scenario clearly illustrates this cross-layer propagation:
Vertical Escalation (L1 → L2): An attacker first plants malicious information at the L1 level through a memory poisoning attack on the agent’s long-term memory (e.g., PoisonedRAG). When the agent makes a decision, its Brain’s reasoning engine retrieves the tainted memory, leading to a cognitive bias. This flawed cognition is then passed to the L2 Action controller, tricking the agent into performing a malicious operation, such as using a code interpreter to execute a destructive script. This transforms an informational error into actual damage in the physical or digital world.
Horizontal Spread and Systemic Amplification (L2 → L3): The malicious action executed at the L2 level (e.g., sending an email with a malicious payload to other agents via an email API) becomes the starting point for risk propagation. At the L3 level, agents that receive the malicious payload become infected and continue to spread it to other nodes in the network via their A2A communication protocols. This virus-like propagation can ultimately amplify an initial single cognitive failure into the paralysis of the entire multi-agent ecosystem.
Emerging Threats and Future Research Directions
The HAE framework is not just an analytical tool; it also points toward future research directions, particularly highlighting significant research gaps in three frontier areas.
Software Supply Chain and Open Ecosystem Security: AI agents like MetaGPT are beginning to participate in the software development process. Their “Package Hallucination” could be exploited for supply chain poisoning, for example, by tricking developers into installing malicious counterfeit software packages. Concurrently, on some open platforms, large numbers of autonomous agents are already forming encrypted communication networks with specific ideologies, indicating that L3 collective risks are becoming a reality.
Dual-Use Risks of Scientific Agents: When AI agents are empowered to control automated laboratory equipment, the combination of their L2 executional capabilities and L3 collaborative knowledge could significantly lower the barrier to creating hazardous chemicals or biological agents. Future safety evaluations must incorporate physical sandboxing environments to ensure that the system’s safety fuse mechanisms can function effectively before high-risk operations are executed.
Integrated Systemic Defense Strategies: Current defense mechanisms are mostly aimed at a single layer and lack a holistic approach. Future research needs to break this fragmentation by exploring methods like Neuro-symbolic Coordination, which combines the model’s probabilistic judgments with deterministic safety rules. Furthermore, at the L3 level, building dynamic immune systems based on decentralized reputation is emerging as a potential direction for defending against systemic risks.
