The Efficiency Dilemma in Multi-Agent Collaboration: From Exhaustive Inference to Intelligent Routing
As large language model technology shifts from single-model capabilities to multi-model collaboration, an architecture known as “Mixture-of-Agents” (MoA) has emerged. Its core idea is to leverage the specialized expertise of different models—such as in math, programming, or medical knowledge—to solve complex problems together. However, the performance benefits of the standard MoA framework come with high computational costs and significant latency. The main bottleneck is that to select the best answer, the system must first invoke all models in the pool for full inference. This “infer-then-select” model quickly becomes unsustainable as the number of models increases.
Even subsequent optimizations like Sparse MoA (SMoA), which introduce a reviewer model to reduce the burden of later fusion, do not escape the essence of exhaustive inference. They still require all candidate models to generate an answer before the review stage. This raises a core question: when deciding which model is best suited for a specific task, must we really pay the price of full-scale inference first?
RouteMoA: A New Paradigm of Dynamic Routing Without Pre-Inference
Addressing this challenge, a paper accepted by the ACL 2026 conference, titled “RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents,” proposes a novel solution. The research was led by the IWIN Center team from the Department of Automation, Shanghai Jiao Tong University. The first author is PhD student Jize Wang, supervised by Professor Xinping Guan, Professor Cailian Chen, and Professor Xinyi Le, in collaboration with Professor Dacheng Tao from Nanyang Technological University, Tencent, Shanghai AI Laboratory, and The Chinese University of Hong Kong.
The core innovation of RouteMoA is shifting the model selection process from post-inference (a posteriori judgment) to pre-inference (a priori prediction). It achieves efficient dynamic routing through a three-step process:
A Priori Filtering: A lightweight “scorer” is introduced, which predicts the potential performance of each model on a given task based solely on the features of the user query. This process involves no LLM inference calls, using rapid, coarse-grained scoring to filter the large model pool down to a manageable “candidate subset,” thus reducing invalid computations at the source.
A Posteriori Correction: To calibrate potential biases from the a priori filtering, RouteMoA designs a low-cost “mixture-of-judges” mechanism. This mechanism utilizes models that have already generated answers to perform self-assessment or leverages high-quality models for cross-assessment of others. All evaluations are based on existing outputs and do not trigger new inference calls, enabling lightweight adjustments to the decision.
Comprehensive Ranking and Decision-Making: Finally, when selecting the ultimate model to use, the system no longer solely pursues the theoretically optimal answer. Instead, it performs multi-objective optimization, comprehensively evaluating the model’s output quality, token cost, and inference latency to make a decision that balances performance and efficiency, offering greater practical engineering value.
Experimental Validation: 89.8% Cost Reduction and Superior Performance
The effectiveness of RouteMoA was validated in a large-scale experiment involving a pool of 15 models with diverse capabilities. The results showed that compared to traditional MoA and SMoA methods, RouteMoA achieved significant advantages in both efficiency and performance:
- 89.8% Cost Reduction: By avoiding invalid inference on non-essential models, computational resource consumption was drastically cut.
- 63.6% Latency Reduction: Fewer inference calls directly translate to faster system response times.
- Improved Accuracy: Surprisingly, while significantly reducing costs, RouteMoA’s overall accuracy actually increased. This suggests that by intelligently routing computational resources to the most suitable models, the system can amplify the correct signals and avoid interference from irrelevant information.
This outcome reveals that in a multi-model system, reducing invalid computations does not harm performance; on the contrary, it allows the system to focus on the “right” models, thereby achieving better results.
Core Insight: The Sparsity of Multi-Model Systems and a New Bottleneck
Behind RouteMoA’s success lies a deep insight into the nature of multi-model collaboration. The research found that for the vast majority of user queries, only a few key models can provide high-quality answers, meaning the system is inherently “sparse.” Experimental data shows that RouteMoA’s a priori scorer achieves a 98% probability of hitting the correct model within its top-3 choices. This indicates that as long as the initial routing stage can accurately identify and retain this critical minority, the subsequent collaboration mechanism is sufficient to ensure final success.
Furthermore, through a failure case analysis, the study uncovered an interesting phenomenon: in over 50% of the cases where the system erred, the mistake originated from the final answer fusion stage—a phenomenon termed “aggregation drift”—rather than from an initial model selection error. This suggests that as the routing and selection problems are solved, the bottleneck in multi-model systems is shifting from “who to select for the answer” to “how to better integrate multiple answers.”
In summary, RouteMoA not only provides a more efficient MoA variant but, more importantly, pioneers a new system design paradigm: shifting from a default where all models participate to first determining who is worthy of participating, and then using a collaborative mechanism to amplify their strengths. This foretells that in the era of multi-model collaboration, system-level scheduling, routing, and coordination strategies are becoming just as crucial as the capabilities of any single model.
