RecursiveMAS: A New Paradigm for Recursive Collaboration in Latent Space
As the capabilities of Large Language Models (LLMs) advance, Multi-Agent Systems (MAS) have become a primary paradigm for tackling complex tasks. However, current systems that rely on text-based interaction generally face bottlenecks such as high computational overhead and difficulties in end-to-end optimization. To address these issues, a team from several top research institutions has introduced the RecursiveMAS framework. It extends the concept of recursive computation from a single model to the entire agent system, achieving breakthroughs in both performance and efficiency.
Overcoming Traditional Bottlenecks: From Textual Interaction to Latent Space Collaboration
Traditional multi-agent systems rely on natural language for communication. While this approach is intuitive, it has fundamental drawbacks:
- Low Efficiency: Each agent must decode its internal continuous hidden states into discrete text, which is then passed to the next agent to be re-encoded. This “decode-encode” loop creates significant computational redundancy.
- Optimization Difficulty: The discrete nature of text hinders the effective propagation of gradients between agents, making it extremely difficult to jointly optimize the entire system. This is particularly problematic in multi-turn interactions, where it can lead to vanishing gradients.
- Accumulated Latency: The more communication turns there are, the more the latency from text generation increases linearly, limiting the system’s scalability and response speed.
The core innovation of RecursiveMAS is to treat the entire multi-agent system as a unified recursive unit. Information flows within the system as continuous latent states, avoiding the costly text conversion process. Each agent acts like a layer in a recurrent neural network, iteratively refining a “latent thought” until the final turn, when the result is decoded for output. This fundamentally enhances collaboration efficiency.
Core Architecture: Lightweight Links and Cyclic Recursion
The implementation of RecursiveMAS relies on two key designs: a lightweight linking module and a cyclic system architecture.
The RecursiveLink Module
The research team designed a lightweight module called RecursiveLink to connect agents both internally and externally. This module uses a two-layer residual projection structure, introducing a minimal number of trainable parameters while keeping the base large model frozen. This allows for efficient transformation between different embedding spaces. It comes in two types:
- Inner Link: Used for generating “latent thoughts” within a single agent. It maps the hidden state of the model’s final layer back to the input embedding space, enabling the agent to perform continuous self-reasoning and iteration without generating text.
- Outer Link: Used for communication between heterogeneous agents. It adds a linear layer to the Inner Link’s structure, responsible for aligning the source agent’s latent state with the target agent’s embedding space, thereby supporting seamless collaboration between models of different architectures or sizes.
The design of the residual connection is crucial. It ensures that the original semantic information is preserved during the transformation, meaning the network only needs to learn the distributional shift rather than a complete mapping. This greatly enhances training stability.
Two-Stage Training Paradigm
To enable end-to-end optimization of the system, RecursiveMAS employs a two-stage, inner-outer loop training strategy, keeping the parameters of the base LLM frozen throughout:
- Inner Loop Training: The Inner Links of each agent are trained in parallel. The goal is to make the generated latent thoughts semantically close to the final answer. This stage uses a cosine similarity loss function for direct optimization in the latent space.
- Outer Loop Training: All agents are connected into a cycle and unrolled for n recursive turns. The cross-entropy loss from the final output is used to jointly optimize all Outer Links, allowing gradients to backpropagate through the entire recursive path for global tuning of the system’s collaborative capabilities.
Experimental Validation: A Dual Leap in Performance and Efficiency
RecursiveMAS demonstrated outstanding performance and efficiency across 9 mainstream benchmarks, including mathematical reasoning (MATH500), science question answering (GPQA-D), and code generation (LiveCodeBench).
Performance Gains
Experimental results show that the performance of RecursiveMAS steadily improves with the number of recursive turns. With a 3-turn recursion setting, its average accuracy increased by 8.3% compared to single-agent, traditional multi-agent, and single-model recursive methods. For instance, on the AIME2025/2026 math competition tasks, it achieved an accuracy of 86.7%, significantly outperforming other methods. The framework was also proven to be flexible, adapting to various mainstream collaboration patterns like sequential, hybrid, distillation, and deliberative modes, achieving consistent performance gains across the board.
Efficiency Advantages
Its efficiency advantages are just as prominent as its performance gains. Both theoretical analysis and experimental data confirm that the computational complexity of latent space interaction is far lower than that of text-based interaction. Specific improvements include:
- Inference Acceleration: Compared to text-based interaction, RecursiveMAS achieves 1.2x to 2.4x faster inference, and the speed-up becomes more significant with more recursive turns.
- Token Savings: Since intermediate steps do not generate text, token usage is drastically reduced by 34.6% to 75.6%.
- Training Cost: The strategy of training only the lightweight RecursiveLink modules results in a training cost that is only 44% of that for full parameter fine-tuning, yet it achieves higher accuracy.
Conclusion and Outlook
By enabling recursive collaboration in latent space, RecursiveMAS opens a new, efficient, and scalable path for the development of multi-agent systems. It not only theoretically proves the advantages of latent space interaction in terms of gradient stability and low complexity but also validates its immense potential in performance and efficiency through comprehensive experiments. In the future, this paradigm is expected to be extended to larger agent clusters, multi-modal tasks, and more complex real-world applications, further unlocking the power of AI to solve problems collaboratively.
