AudioX-Turbo: A Unified Multimodal Audio Generation Framework
With the rapid development of AI video generation, the field of AI audio generation is also seeing new breakthroughs. Recently, AudioX-Turbo, a model jointly launched by the Hong Kong University of Science and Technology (HKUST), Tsinghua University, and Noiz AI, aims to solve the two core pain points of audio generation—inference speed and control precision—through a unified “Anything-to-Audio” framework.
The model is highly flexible, capable of handling six different generation tasks. It integrates functionalities such as Text-to-Audio, Text-to-Music, Video-to-Audio, Video-to-Music, and joint text-video generation into a single model, significantly enhancing the efficiency and possibilities of audio content creation.
Core Technology: 4-Step Inference with Student-Teacher Distillation
Traditional audio diffusion models often require dozens or even hundreds of sampling steps to generate audio, resulting in high latency that fails to meet the demands of real-time interactive applications. The core innovation of AudioX-Turbo is its ultra-fast inference capability, producing high-quality results in just 4 sampling steps and compressing generation latency from minutes to seconds.
This performance boost is achieved through a technique called student-teacher distillation. The team first trained a full multi-step diffusion model, AudioX-Base, as the “teacher model.” Then, using a method called Distribution Matching Distillation (DMD) combined with a diffusion discriminator, the teacher model’s knowledge was efficiently compressed into the “student model,” AudioX-Turbo. This architectural design provides a technically feasible path for developing real-time AI audio tools.
The Data Foundation: The 9.2 Million Sample IF-caps-Pro Dataset
High-quality, large-scale training data is fundamental to building high-performance generative models. To this end, the AudioX-Turbo team built a proprietary dataset named IF-caps-Pro, with a scale of approximately 9.2 million samples. The dataset was constructed through a two-stage data collection and fine-grained annotation process, ensuring data diversity and quality.
In comparison, common open-source audio models in the community often rely on public datasets like AudioCaps (approx. 50,000 samples) or MusicCaps (approx. 5,000 samples). IF-caps-Pro surpasses them by orders of magnitude in data volume, providing a solid foundation for the model to learn complex acoustic patterns and multimodal correlations, forming one of the project’s core technical moats.
Open Source and Technical Implementation
The AudioX-Turbo project is fully open-sourced on GitHub, with technical details described in an arXiv paper (ID: 2406.12555). The model weights have also been uploaded to the Hugging Face platform (HKUSTAudio/AudioX-Turbo) for community use.
The official recommendation is to run the model on an NVIDIA A100 or H800 GPU and requires a CUDA 12.1 environment. Users wishing to replicate the training process will also need to install the full CUDA toolkit and use the DeepSpeed framework.
Developers can perform inference in two ways:
- Gradio Web UI: Run a simple command-line script
python run_gradio.py to deploy an interactive demo interface locally.
- Python API: The project’s code repository provides detailed API call examples. Core functions include
load_audiox_turbo_model for loading the model and generate_diffusion_cond_dmd for performing the 4-step generation. The code also covers practical functionalities like video feature extraction (using Synchformer), audio post-processing, and audio-video merging, making it convenient for developers to integrate and build upon.
