Introducing the Monet Model: Internalizing Visual Thinking via Latent Space Reasoning
A new study named Monet introduces an innovative method that enables Multimodal Large Language Models (MLLMs) to perform visual thinking “in their minds,” much like humans. The model reasons by generating “latent visual embeddings” in a continuous latent space, freeing itself from reliance on external tools and truly internalizing its visual thinking capabilities. This research, a collaboration between Peking University, the Amazon AGI SF Lab, and other institutions, has been accepted by the CVPR 2026 conference.
Latent Visual Reasoning: A New Paradigm Beyond Tool Dependency
Currently, the dominant paradigm for enhancing the visual reasoning capabilities of Multimodal Large Language Models (MLLMs) is to “Think with images.” This involves invoking external tools or generating code to introduce auxiliary images (e.g., cropping, annotating, drawing guidelines) during the reasoning process. While this method has achieved some success, its dependence on external tools presents three major limitations: first, the model must learn additional tool interfaces, increasing training complexity and inference latency; second, the model’s capabilities are limited by the predefined set of tools, making it difficult to generalize to more complex tasks; and third, adding each new tool requires re-annotating data and designing new interfaces, making the model more of a “tool coordination hub” than a system that truly understands and imagines.
To address these issues, Qixun Wang and Yang Shi, doctoral students at Peking University, along with Yifei Wang from the Amazon AGI SF Lab and others, jointly proposed the Monet model. Instead of relying on external tools, this model simulates the human thought process of “sketching and drafting” within its own latent space by generating continuous “latent visual embeddings.” This approach internalizes visual thinking as an inherent capability of the model, achieving more flexible and general-purpose visual reasoning. The research was guided by Yuanxing Zhang from the Kuaishou Keling team, and Professors Xianghua Ying and Yisen Wang from Peking University.
SFT and VLPO: Monet’s Two-Stage Training Framework
Monet is built upon an innovative “Supervised Fine-Tuning (SFT) + Reinforcement Learning (RL)” training framework, using Qwen2.5-VL-7B as the base model. The core challenge lies in effectively supervising and optimizing the unobservable “latent embeddings.”
The Supervised Fine-Tuning (SFT) phase is divided into three steps, designed to progressively teach the model latent thinking:
- Warm-up: Initial fine-tuning on a specially crafted dataset of interleaved image-text Chain-of-Thought data, allowing the model to adapt and learn to use image information from intermediate reasoning steps.
- Acquiring High-Quality Latent Embeddings: This stage employs a sophisticated mechanism to indirectly guide the generation of latent embeddings. It aligns the model’s intermediate representations of key text under two conditions: one with a given auxiliary image and one with a given latent embedding. Special attention masks ensure that the auxiliary image’s features can only be “seen” by the latent embedding, forcing the information flow through the path “auxiliary image → latent embedding → subsequent text.” This compels the model to autonomously learn to encode valuable visual information into the latent embedding.
- Learning to Think “from Scratch”: The high-quality latent embeddings generated in the second stage are used as “teacher” signals to train the model to generate similarly effective embeddings without directly seeing the auxiliary image, thus achieving true independent latent thinking.
The Reinforcement Learning (RL) phase introduces a brand-new algorithm specifically designed for latent thinking: VLPO (Visual-latent Policy Optimization). Traditional RL algorithms like GRPO struggle to apply reward signals directly to the “thinking” process because they cannot compute the generation probability of latent embeddings. VLPO overcomes this by estimating the generation probability of these embeddings through a mathematical assumption, thereby incorporating the optimization of latent embeddings into the loss function. The intuitive effect is that if a sampled latent embedding leads to a correct answer, VLPO pulls the policy model’s subsequent embeddings closer to this “high-quality” one, directly optimizing the model’s latent thinking ability.
Monet-SFT-125K: Building a High-Quality Training Foundation
To train Monet effectively, the research team constructed a high-quality dataset of 125,000 samples, named Monet-SFT-125K. The team found that existing interleaved image-text Chain-of-Thought datasets suffered from issues like unnecessary or inaccurate auxiliary images and a lack of annotation for crucial information.
Therefore, they designed a multi-stage data correction pipeline:
- Necessity Filtering: First, they filter for samples that the base model (Qwen2.5-VL-7B) answers incorrectly without an auxiliary image, ensuring the image is necessary for solving the problem.
- Accuracy Filtering: Next, they use a more powerful model (Qwen2.5-VL-72B) to verify that the problem can be answered correctly with only the auxiliary image, ensuring the accuracy and sufficiency of the image’s information.
- Key Information Annotation: Finally, they use closed-source large models to annotate the text descriptions corresponding to the key visual information in the auxiliary images, providing valuable supervisory signals for the alignment learning in the second SFT stage.
The resulting Monet-SFT-125K dataset covers various task types, including real-world scenarios, charts, and OCR, as well as multiple visual operations like cropping, calibration, and drawing auxiliary lines, laying a solid foundation for the model to learn complex visual reasoning skills.
Experimental Validation: Significant Performance Gains on In-Distribution and Out-of-Distribution Tasks
Experimental results show that the Monet-7B model achieves excellent performance on multiple visual reasoning tasks. On in-distribution tasks similar to the training data (real-world, charts, and OCR), Monet demonstrates a performance improvement of 3% to 9.75% over the base model, outperforming baselines that use only SFT, SFT+GRPO, or other existing “think with images” methods.
More importantly, Monet also shows a 2.31% performance gain on out-of-distribution abstract visual reasoning tasks, proving the good generalization ability of its internalized visual thinking. Comprehensive ablation studies also confirmed the necessity of each stage of SFT design and the VLPO algorithm. It is particularly noteworthy that applying the traditional GRPO algorithm on top of the SFT model did not yield stable improvements, which highlights the superiority of the VLPO algorithm.
The research team also discovered that the Monet model exhibits a “test-time scaling law.” Especially after VLPO training, increasing the number of latent embeddings (i.e., “thinking steps”) at test time leads to improved performance, even when the number of steps exceeds what was used during training. This suggests that Monet not only learns how to think but can also solve more complex problems through deeper thinking.
