Mistral AI Launches Its First Multimodal Model, Pixtral 12B
French AI company Mistral AI has released its first large multimodal model, Pixtral 12B. This model can process interleaved image and text data, demonstrating excellent performance in visual document understanding and multimodal reasoning tasks. Pixtral 12B provides open-source weights under the Apache 2.0 license, bringing a high-performance vision-language processing solution to the open-source AI community.
Model Architecture and Core Specifications
Pixtral 12B uses a 12B-parameter multimodal decoder based on Mistral Nemo, equipped with a new 400M-parameter vision encoder trained from scratch. The model is trained on interleaved image and text data, enabling it to understand both natural images and document content. It supports variable image sizes and aspect ratios. A long context window of 128k tokens allows for the processing of any number of images.
Benchmark Performance Results
Pixtral 12B achieves leading results on several multimodal benchmarks, including 52.5% on the MMMU reasoning benchmark, 58.0% on Mathvista, 81.8% on ChartQA, 90.7% on DocVQA, and 78.6% on VQAv2. The model significantly outperforms other open-source models like Qwen2-VL 7B, LLaVa-OneVision 7B, and Phi-3.5 Vision in multimodal instruction following. It also maintains its leading performance on text benchmarks, making no compromises for its multimodal capabilities.
Key Capabilities and Use Cases
Pixtral 12B excels at tasks such as chart and diagram comprehension and document question-answering. Its key advantage is providing top-tier multimodal reasoning capabilities without sacrificing core text abilities like instruction-following, coding, and math. As a direct replacement for Mistral Nemo 12B, the model is well-suited for real-world scenarios requiring joint processing of images and text.
Open-Source Availability and Deployment
The model weights for Pixtral 12B are available for download from the Hugging Face repository mistralai/Pixtral-12B-2409. Developers can run it locally using the mistral-inference tool or achieve high-throughput serving with the vLLM library. Additionally, users can experience the model directly on Mistral AI’s La Plateforme and Le Chat platforms.
