University of Toronto Introduces ETR Algorithm: Enhancing LLM Inference Efficiency and Accuracy by Optimizing Entropy Trends
Researchers from the University of Toronto and McMaster University have proposed a novel optimization algorithm called Entropy Trend Reward (ETR) in a preprint paper titled “ETR: Entropy Trend Reward for Efficient Chain-of-Thought Reasoning”. This algorithm aims to solve the “overthinking” problem—long, inefficient reasoning steps—common in Large Language Models (LLMs) during Chain-of-Thought (CoT) reasoning. Experimental results show that ETR can reduce the average inference length by 67% while improving accuracy on complex reasoning tasks by up to 9.9%.
1. The Efficiency Bottleneck in Chain-of-Thought Reasoning
Chain-of-Thought (CoT) prompting significantly enhances a model’s ability to handle complex tasks like math and logic by guiding it to generate step-by-step reasoning processes. However, this method introduces new challenges: the generated reasoning chains are often overly long, containing repetitive or contradictory content—a phenomenon known as “overthinking.” This not only drastically increases inference latency but also drives up the computational costs of cloud services.
Previous optimization methods have their own limitations:
- Length Penalty: Directly penalizing output length can lead to the loss of crucial logic during compression, resulting in decreased model accuracy.
- Global Entropy Suppression: Forcing the model to maintain low uncertainty (low entropy) throughout the entire reasoning process contradicts the natural human pattern of divergent thinking followed by convergence.
- Training-Free Truncation: Truncating reasoning without additional training generally leads to poor generalization, especially for smaller models.
ETR was developed to effectively address reasoning redundancy without sacrificing—and even while improving—accuracy.
2. Core Insight: From Absolute Entropy to Entropy Trend
The theoretical foundation of the ETR algorithm is a key discovery: it’s not the instantaneous absolute value of entropy, but its dynamic trend throughout the reasoning process that determines the efficiency of a CoT.
In information theory, entropy measures uncertainty. In the context of LLM text generation, high entropy means the model is “hesitant” when choosing the next token, facing multiple possibilities. Low entropy means the model is very certain about the next step. The researchers found that an efficient reasoning process exhibits a “continually decreasing” entropy trend, moving from initial exploration and uncertainty toward logical convergence and certainty. Conversely, a process with consistently rising or fluctuating entropy often corresponds to the model’s “self-reflection,” hesitation, and inefficient reasoning.
This finding shifts the optimization goal from “enforcing low entropy throughout” to “guiding a downward entropy trend,” allowing the model necessary exploration in the early stages of reasoning while ensuring it ultimately converges efficiently to the correct answer.
3. The ETR Algorithm: Momentum-Based Trajectory-Level Optimization
Based on this insight, the research team designed the Entropy Trend Reward (ETR) algorithm and combined it with the GRPO reinforcement learning framework for end-to-end optimization of the model’s reasoning trajectory. The core process involves three steps:
Step-wise Entropy Quantization: First, the model’s CoT output is divided into steps (e.g., by newlines). Then, the average prediction entropy Hₜ for all tokens in each step is calculated, along with the change in entropy between adjacent steps, Δₜ = Hₜ₋₁ - Hₜ. A positive Δₜ indicates a decrease in uncertainty, which is desired for effective reasoning.
Momentum Accumulation to Capture Trends: To smoothly measure the global trend and avoid interference from single-step entropy fluctuations, the algorithm introduces a momentum coefficient γ. The entropy change is accumulated using the formula Sₜ = γ·Sₜ₋₁ + Δₜ, creating a variable Sₜ that reflects the overall downward trend. This design gives higher weight to earlier entropy drops, encouraging the model to enter a convergence state sooner.
Dual-Constraint Reward Mechanism: To balance accuracy and efficiency, ETR employs a strict reward mechanism.
- Hard Accuracy Constraint: If the model’s final answer is incorrect, it receives a direct negative reward of -1. This ensures that the primary goal is always to maintain correctness.
- Conditional Efficiency Optimization: A positive reward is only given if the answer is correct. This reward consists of a base score (1) and an additional bonus
λ×Sₜ that is positively correlated with the entropy trend Sₜ. This design ensures that efficiency optimization does not come at the cost of accuracy.
4. Experimental Results: Dual Improvements in Efficiency and Accuracy
The researchers validated ETR’s effectiveness on several mainstream models, including DeepSeek-R1-7B and Qwen3-4B/8B, across multiple authoritative math reasoning benchmarks like AMC23, AIME24, MATH500, and GPQA Diamond.
- Core Performance: On the DeepSeek-R1-Distill-7B model, applying ETR optimization resulted in a 9.9% increase in accuracy while shortening the average length of the reasoning chain by 67%. Across other models and datasets, ETR also surpassed all baseline methods in AES (Accuracy-Efficiency Score).
- Generalization Ability: To test its versatility, an ETR-trained model, using only math data, was directly applied to the HumanEval code generation benchmark. On the Qwen3-4B model, ETR boosted code generation accuracy by 12.2% while reducing inference length by 46%. This demonstrates that ETR optimizes general reasoning logic rather than domain-specific knowledge.
- Mechanism Validation: Ablation studies showed that removing either the momentum accumulation or the correctness constraint led to performance degradation or failure to effectively shorten length, confirming the integrity and necessity of ETR’s design. After training, the correlation coefficient between entropy and step count in the model’s reasoning process shifted from positive to negative, visually demonstrating that the model learned an efficient “diverge-to-converge” reasoning pattern.
5. Industry Value and Future Outlook
The contribution of the ETR algorithm lies in providing a new, more cognitively aligned perspective for optimizing large model inference. By focusing on the dynamic trend of entropy rather than its static value, ETR successfully guides models to “think smarter, not less.”
The industrial value of this technology is primarily reflected in:
- Cost Reduction: In large-scale cloud deployment scenarios, reducing inference token consumption by more than half can directly translate into significant computational cost savings.
- Improved User Experience: Shorter reasoning paths mean faster response times, which can effectively enhance the user experience in AI applications.
- Facilitating On-Device Deployment: Improving the inference efficiency of smaller models creates opportunities for deploying powerful AI applications on edge devices like smartphones and vehicles.
The research team states that future work will include extending ETR to larger models (14B+) and multi-modal reasoning scenarios, further advancing the practical application of efficient inference technologies.
References
