SWIFT: A Breakthrough Lightweight Reward Model Achieving Performance and Speed Gains with 0.005% Parameters
In the optimization process of Large Language Models (LLMs), reward models play a crucial role in evaluating and selecting the quality of model outputs, especially in the “Best-of-N” strategy. However, the large size and high inference costs of traditional reward models are becoming bottlenecks for their large-scale deployment. On January 23, 2026, a research team from Shanghai Jiao Tong University, the National University of Singapore, Tongji University, and the University of Illinois Chicago, in their paper “SWIFT: A Lightweight Reward Model with Less Than 0.005% Parameters,” proposed a lightweight reward model called SWIFT (Simple Weighted Intrinsic Feedback Technique) to address this challenge.
A New Path: Extracting Reward Signals from Model Hidden States
Traditional reward models typically evaluate by analyzing the complete text generated by a model, a process almost equivalent to running another large model. SWIFT’s core innovation is that it no longer “reads” the text output. Instead, it directly utilizes the internal hidden states generated by the LLM during the answer generation process. The researchers found that these hidden states themselves contain signals about the model’s judgment of the correctness and confidence of its reasoning path. SWIFT uses an extremely lightweight linear mapping structure to extract a reward score from these internal signals, thereby bypassing the complex text encoding process.
Order-of-Magnitude Efficiency Gains and Performance Advantages
SWIFT’s model structure is extremely concise, with its parameter count only in the order of 105, which is less than 0.005% of traditional reward models with tens of billions of parameters. This design brings significant efficiency improvements. Experimental data shows that after replacing traditional reward models with SWIFT, the end-to-end inference process is accelerated by 1.7 to 6.7 times, while the computational load (FLOPs) is also reduced to a tiny fraction of the original method. More importantly, in several standard benchmark tests such as MATH, GSM8K, and HellaSwag, SWIFT, under the Best-of-N setting, demonstrated superior or more stable accuracy compared to mainstream open-source reward models.
Generalization and Engineering Potential
SWIFT’s capabilities are not limited to improving the accuracy of inference tasks. The research team also verified its versatility in alignment evaluation tasks like PKU-SafeRLHF. The results show that SWIFT also outperforms many large-scale text-based reward models in evaluating the Helpfulness and Harmlessness of generated content. Furthermore, SWIFT possesses high engineering flexibility: it supports training using only the last few layers of states near the model’s output, and even in closed-source model scenarios where hidden states are inaccessible, it can be effectively trained relying solely on the output probability distribution (logits). These features make it not only a highly efficient alternative but also a lightweight module that can work synergistically with existing reward model systems to enhance overall system performance with almost no added cost.
