MIT CSAIL Introduces CompreSSM: Dynamically Compressing AI State-Space Models During Training
Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), in collaboration with multiple institutions, have developed an innovative AI model compression technique called CompreSSM. This technique can compress the state space of a model during the training process, significantly reducing computational resource requirements while maintaining high performance.
How It Works
CompreSSM is designed for state-space models (SSMs), a family of AI architectures commonly used in fields like language processing, audio generation, and robotics control. The method draws from control theory, using Hankel singular values to measure the contribution of each internal state to the model’s overall behavior. About 10% into the training, it ranks the dimensions based on these values and removes less important components. The remaining 90% of training then continues on the smaller, compressed model. The research proves the stability of state importance changes using Weyl’s theorem and includes a fail-safe rollback mechanism to ensure reliability.
Benchmark Results
In image classification benchmarks, the training speed of the compressed model can be increased by 1.5x, with accuracy close to that of the full model. On the CIFAR-10 dataset, a model with its state dimension compressed to one-fourth of the original achieved 85.7% accuracy, showing a clear advantage over the 81.8% accuracy of a model of the same size trained from scratch.
Comparison with Existing Methods
Compared to traditional methods, CompreSSM’s advantages are significant. Traditional approaches either fully train a large model and then prune it, or start directly with a small model. The former is computationally expensive, while the latter often suffers from insufficient performance. This technique makes its decision mid-training, avoiding these drawbacks. Compared to knowledge distillation, CompreSSM performs better in heavily compressed scenarios. Compared to Hankel nuclear norm regularization, it is over 40 times faster and achieves higher accuracy.
Research Team and Significance
The research is led by first author Makram Chahine, a PhD student in the Department of Electrical Engineering and Computer Science at MIT CSAIL, with Professor Daniela Rus, the director of CSAIL, as the senior author. Collaborating institutions include the Max Planck Institute for Intelligent Systems, the European Laboratory for Learning and Intelligent Systems (ELLIS), ETH Zurich, and Liquid AI. The paper has been accepted to ICLR 2026, offering a new path for developing efficient AI systems, particularly for multi-input multi-output models.
