Background: Challenges of Applying Large Models to End-to-End Driving
The introduction of large vision-language models (VLMs) into autonomous driving has significantly enhanced the system’s ability to comprehend complex traffic scenarios, such as identifying the intentions of other vehicles, pedestrian dynamics, and traffic rules. However, effectively translating this high-level understanding into precise, real-time driving actions remains a core challenge for end-to-end autonomous driving.
Previous explorations have primarily fallen into three categories:
- Decoupled Architectures: Using the VLM as a scene encoder, with its output passed to a separate planner. This approach has clear responsibilities, but the understanding and planning processes are disconnected, leading to information loss during transmission.
- Auxiliary Architectures: The VLM outputs text prompts, such as risk assessments or driving intentions, to assist a traditional planner’s decision-making. This method requires minimal changes to existing systems but struggles to fully convert the VLM’s complex reasoning into fine-grained driving actions.
- Synchronous Unified Architectures: Integrating reasoning and action within a single Vision-Language-Action (VLA) model. This unifies understanding and action, but the VLM’s inference latency severely constrains the high-frequency update requirements for trajectory planning, limiting the system’s reaction speed.
The AutoMoT Architecture: Asynchronous Mixture-of-Transformers and Deep Fusion
To address these challenges, a research team from Nanyang Technological University’s AutoMan Lab, Harvard University, and Xiaomi Auto jointly proposed AutoMoT (A Unified Vision-Language-Action Model with Asynchronous Mixture-of-Transformers for End-to-End Autonomous Driving), which has been accepted by the ICML 2026 conference.
AutoMoT’s core design includes two key innovations:
Expert Specialization and Deep Attention Fusion
AutoMoT consists of two expert modules: an Understanding Expert (UE) for high-level comprehension and an Action Expert (AE) for concrete actions.
- UE Module: Based on a 4B-parameter Qwen3-VL backbone, it processes multi-frame images and navigation information to perform high-level semantic modeling of the scene.
- AE Module: With approximately 1.6 billion parameters, it receives current sensor data (RGB images, LiDAR BEV) and task instructions to generate decisions and plan trajectories.
The key to the model is that the UE and AE are not simply connected in series. AutoMoT introduces a Layer-wise Shared Attention mechanism, allowing the AE to directly access the UE’s intermediate representations at each layer of the network during action generation. Through a cross-task causal mask design, the information flow is explicitly defined as “understanding → decision-making → planning.” This ensures that trajectory generation is not merely geometric fitting but a reasoning process driven by both scene semantics and driving intent.
Asynchronous Inference and KV Cache Reuse
To solve the real-time problem, AutoMoT adopts an asynchronous inference strategy of “low-frequency understanding, high-frequency action.” High-level scene understanding (e.g., a construction zone ahead) persists over time and does not need to be re-evaluated in every control cycle. Therefore, the UE runs at a lower frequency. After completing one forward pass, its generated Key-Value Cache (KV Cache) is saved.
In subsequent control cycles, the higher-frequency AE can directly reuse this cached scene understanding state to quickly generate decisions and trajectories. This design decouples the time-consuming inference process of the large model from high-frequency action control, ensuring the system’s real-time responsiveness without sacrificing the model’s reasoning depth.
Experimental Performance: Achieving SOTA on Two Major Benchmarks
AutoMoT has demonstrated outstanding performance in both major open-loop and closed-loop autonomous driving benchmarks.
Bench2Drive Closed-Loop Evaluation: AutoMoT achieved a Driving Score (DS) of 87.34 and a Task Success Rate (SR) of 70.00%, surpassing the previous SOTA model, SimLingo (85.07 DS / 67.27% SR). After adding an Action Refiner, the performance of AutoMoT+ further improved to 89.42 DS and 74.09% SR.
nuScenes Open-Loop Planning Evaluation: The model excelled in safety metrics, with an average collision rate of just 0.07%, significantly outperforming similar models. Its average L2 trajectory error was 0.32, demonstrating the precision and safety of its planned trajectories.
Reflection: Balancing the General Capabilities of VLMs with the Specialization of Driving Tasks
The experiments with AutoMoT also sparked deeper reflection on adaptation strategies for large models: should the entire pre-trained model be fully fine-tuned for autonomous driving?
Through ablation studies, the research team found that while fully fine-tuning the entire model yields significant improvements on action-related tasks (like OmniDrive), the gains on high-level scene understanding tasks (like LingoQA) are limited. More importantly, full fine-tuning leads to a substantial performance drop on general-purpose visual question answering tasks (like TallyQA and InfographicVQA), sacrificing its valuable general reasoning capabilities.
AutoMoT’s design philosophy is to create a more effective division of labor: preserving the powerful pre-trained capabilities of the UE as a general scene understanding expert, while delegating the domain-specific adaptation for driving actions primarily to the AE module. This suggests that the most critical part requiring domain adaptation in an autonomous driving system is how to translate high-level understanding into concrete actions, rather than the foundational vision-language understanding itself. This design not only ensures high performance but also provides a new path for building more scalable and deployable VLA systems.
