As model capabilities advance, million-token long-context windows are transitioning from concept to reality. However, having large models directly process vast amounts of raw text often leads to low efficiency, the ‘lost-in-the-middle’ problem where key information is overlooked, and increased security risks. This raises a core question: must knowledge acquisition and logical reasoning be performed by the same model?
Addressing this challenge, a research team from the Shanghai AI Laboratory and Fudan University (main authors including Wenxuan Xie, Xin Tan, Chaochao Lu, and Xia Hu, with Xuhong Wang as the corresponding author) has proposed an innovative framework called DRIFT (Decoupled Reading then Reasoning over long-form text). Their findings are published in the paper ‘DRIFT: A Framework for Decoupled Reading then Reasoning over Long Documents’ (arXiv: 2402.10021).
The Dilemma of Long-Context Processing and Existing Solutions
Current mainstream methods for long-context processing primarily focus on “how to make models read more efficiently” and can be broadly categorized into three types:
- Input Compression: This involves reducing input length through hard compression (deleting less important tokens) or soft compression (mapping text to latent representations). However, these methods often rely on static or local estimations, risking the removal of key information or the retention of content irrelevant to the current task.
- Retrieval-Augmented Generation (RAG): This retrieves relevant snippets from an external knowledge base to serve as context. The effectiveness of this approach heavily depends on the retriever’s performance and strategy, and it may fall short on complex tasks requiring a comprehensive understanding of the entire document.
- Parametric Memory: This uses the model’s internal parameters or dedicated memory modules to store knowledge. While efficient for inference, this approach typically requires pre-training and struggles to process and absorb knowledge from new, ultra-long documents in real-time.
While these methods alleviate the pressure of long contexts to some extent, they do not fundamentally change the paradigm that “the reasoning model must directly read the raw or compressed text.” DRIFT, however, proposes a new structural approach.
The DRIFT Framework: Explicitly Decoupling Knowledge Acquisition and Reasoning
The core idea of DRIFT is to redefine how knowledge enters the reasoning model, explicitly separating knowledge acquisition and logical reasoning into two specialized models.
Its dual-model architecture includes:
- Knowledge Model: A lightweight model (e.g., 3B parameters) responsible for “reading”. It takes an ultra-long original document and, based on the user’s query, processes text chunks in parallel to extract task-relevant key information. It then compresses this information into a high-density latent knowledge representation, termed “Implicit Fact Tokens”.
- Reasoning Model: A more powerful large model (e.g., 7B parameters) responsible for “reasoning”. It no longer processes the lengthy raw text. Instead, it directly receives the compact Implicit Fact Tokens generated by the Knowledge Model and performs complex logical reasoning based on this high-density information to produce the final answer.
These “Implicit Fact Tokens” are not simple sentence summaries or retrieved chunks but a novel input modality with high information density, specifically designed for the reasoning task. This design fundamentally changes the workflow of the Reasoning Model, allowing it to focus on its core reasoning capabilities.
To achieve this, DRIFT employs a three-stage training strategy that respectively teaches the Knowledge Model how to perform information compression (LFRP stage), how to dynamically compress based on a query (QAFT-DC stage), and teaches the Reasoning Model how to understand and utilize this new knowledge modality for question answering (QAFT-QA stage).
Experimental Validation: Dual Gains in Performance and Efficiency at High Compression Ratios
The team conducted comprehensive tests on DRIFT using several authoritative long-context reasoning benchmarks, including LongBench-v2, LoCoMo, BAMBOO, and L-Eval. The experimental results show that the framework excels in both efficiency and performance:
- Performance Retention and Improvement: With a 3B Knowledge Model and a 7B Reasoning Model, even at a high compression ratio of 32x, DRIFT’s overall performance approached or even surpassed that of a full-context baseline model. At extreme compression ratios of 64x to 128x, its performance was consistently superior to other compression methods like ICAE and COCOM.
- Significant Latency Reduction: Because the Reasoning Model processes a drastically reduced number of tokens, DRIFT achieves the lowest or near-lowest inference latency across various context lengths, demonstrating extremely high computational efficiency.
- General-Purpose Abilities Preserved: A key concern was whether detachment from the raw text would weaken the Reasoning Model’s general abilities. Experiments proved that after training with the DRIFT framework, the Reasoning Model’s original capabilities in tasks like complex reasoning, knowledge QA, code generation, and general instruction following were not compromised.
Structural Advantages: Unexpected Security Benefits and Application Generalization
Beyond efficiency and performance, DRIFT’s decoupled architecture brings two significant additional advantages.
First is enhanced security. In tests on multiple mainstream safety benchmarks like Flames, SaladBench, and AutoDAN, DRIFT demonstrated greater robustness against jailbreak attacks than the original models. Notably, this improvement was achieved without any specialized safety training. The researchers believe this is because the Reasoning Model is “insulated”; it no longer directly interacts with the raw input, which may contain malicious prompts. Instead, it operates on an intermediate knowledge representation that has been “filtered” and “purified”, naturally reducing its vulnerability to attacks.
Second is the generalizability of the concept. This decoupled idea of “letting a specialist model read and a generalist model reason” has broad application potential. The research team also applied a similar principle in another project called “BioBridge” for protein understanding: a specialized Protein Language Model (PLM) is used to “read” and analyze protein sequences, generating an intermediate representation, which is then passed to a Large Language Model (LLM) for reasoning about biological functions. This model allows the system to combine the domain expertise of a specialized model with the reasoning power of a general-purpose model.
In summary, from DRIFT to BioBridge, the research shows that structurally decoupling knowledge acquisition from reasoning is a more efficient and scalable technical route than infinitely expanding the context window. It not only enhances the efficiency and effectiveness of models in processing long texts but also opens up new possibilities in areas like security and multi-modal fusion.
