#
On March 8, 2026, a joint research team from the Gaoling School of Artificial Intelligence’s GeWu-Lab at Renmin University of China and Beihang University published a study that systematically addresses a key challenge in robotic multimodal learning. The research reveals that in policies fusing vision and proprioception (e.g., joint angles, velocities), robots often experience performance degradation at critical moments of action transition. The root cause is that proprioceptive signals suppress the learning of visual information during training. In response, the team proposed a new algorithm named “Gradient Adjustment with Phase-guidance” (GAP). This work has been accepted by the International Conference on Learning Representations (ICLR 2026).
Background: The “Generalization Paradox” of Vision-Proprioception Fusion Policies
In the field of embodied intelligence, combining vision (seeing) with proprioception (feeling) is widely considered an effective way to enhance a robot’s manipulation capabilities. Theoretically, this multimodal fusion should enable the robot to perceive itself and its environment more comprehensively, thereby exhibiting stronger generalization in dynamic tasks. However, practice has shown a contradictory phenomenon: some studies indicate that policies incorporating proprioception perform worse than pure vision-based policies. This “generalization paradox” has puzzled the industry, raising questions about when vision-proprioception policies fail and what the underlying mechanisms are. Understanding this issue is crucial for designing truly robust robotic systems.
Root Cause: “Visual Failure” During Motion Transition Phases
To investigate the core of the problem, the research team designed a series of meticulous comparative experiments. They discovered that the performance decline in vision-proprioception policies does not occur throughout the entire process but is concentrated in the “motion transition phases” of a task, such as the moment of switching from “moving forward” to “precise alignment.” In these phases, the robot needs to rely on subtle visual cues to perform precise operations. Analysis of the training process shows that because proprioceptive signals (like joint positions) are more direct numerically and easier for the model to fit to reduce the loss function, the optimization process becomes overly reliant on proprioception. This “path dependence” in turn suppresses the effective learning of visual features. As a result, the visual modality “fails” at critical moments when it is most needed, leaving the robot unable to cope with unseen subtle changes and ultimately leading to task failure.
Solution: The GAP Gradient Adjustment Algorithm
Based on these findings, the team proposed the GAP algorithm. Its core idea is to intelligently identify motion transition phases during training and “make way” for the visual modality’s learning during these periods. The algorithm mainly consists of two steps:
Motion Transition Phase Identification: The algorithm first uses change-point detection techniques to analyze time-series data such as the robot’s end-effector position, orientation, and gripper status. This segments a continuous action trajectory into multiple “kinematically consistent” sub-phases. Subsequently, a temporal network model uses the changes in proprioceptive signals to predict in real-time the probability that the current moment belongs to a motion transition phase.
Dynamic Gradient Adjustment: When the model updates its parameters via backpropagation, the GAP algorithm utilizes the transition probability calculated in the previous step. When this probability is high, indicating that the robot is in a critical action switching period, the algorithm correspondingly reduces the magnitude of the gradient updates for the neural network modules through which proprioceptive information flows. This dynamic suppression creates more learning space for the visual modality, forcing the model to learn and rely on the visual cues that are crucial during these transition phases.
Experimental Validation and Performance Highlights
The research team conducted extensive validation of the GAP algorithm on both simulation and real-world robot platforms, covering a variety of complex manipulation tasks such as single-arm and dual-arm operations. The results show that after applying the GAP algorithm, the performance of the vision-proprioception policy was significantly improved. It not only surpassed the standard vision-proprioception baseline but also outperformed the pure vision policy, truly achieving a synergistic effect where 1+1 > 2. For instance, in an “object handover” task, a policy without GAP would continue to execute an incorrect action after a grasp failure due to the “inertia” of over-relying on proprioception. In contrast, the GAP-enhanced policy could use visual feedback to correct the error in time and ultimately complete the task. Furthermore, the algorithm was also successfully applied to Vision-Language-Action (VLA) models, resolving the issue where models like Octo experienced performance degradation after incorporating proprioceptive inputs. This demonstrates GAP’s excellent versatility and compatibility, allowing it to adapt to various mainstream multimodal fusion architectures.
