Challenging the “Standard” in Transformers
In mainstream Transformer architectures, Layer Normalization (LayerNorm) is an almost indispensable component used to stabilize the training process. However, it also introduces significant computational and memory access overhead, especially during the inference phase of large models. Consequently, “normalization-free” Transformers have become a long-term research direction, but previous methods have generally faced two major challenges: training instability and performance inferior to standard models.
Derf: A Simple and Effective Dynamic Error Function
To address these challenges, the research team proposed a novel activation layer called Derf (Dynamic erf). Derf’s structure is extremely simple; it is essentially a shifted and scaled Gauss error function (erf) with a small number of learnable parameters. As a point-wise layer, it does not rely on the statistical information of the input data and can directly replace the existing LayerNorm or RMSNorm layers in a model. This method is a further development of their earlier work, Dynamic Tanh (DyT).
Design Principles and Generalization Advantage
The research team systematically analyzed the functional properties affecting the performance of normalization-free models and summarized four key design principles: zero-centered, bounded, central sensitivity, and monotonicity. The Derf function satisfies all four conditions. A surprising finding is that Derf’s performance advantage does not stem from a stronger fitting capability. Experimental data shows that while normalization-based models achieve lower loss on the training set, Derf performs better on the test set. This indicates that Derf primarily gains its performance benefits by enhancing the model’s generalization ability.
Multi-Domain Experiments Validate Its Effectiveness
Derf’s effectiveness has been systematically validated across multiple domains and models, where its performance is consistently superior to LayerNorm and DyT. Specific results are as follows:
Vision and Generation
In the ImageNet-1K image classification task, ViT models using Derf achieved higher Top-1 accuracy. In the DiT diffusion model, Derf also obtained a lower FID score, indicating higher image generation quality.
Language, Speech, and DNA
For GPT-2 models pre-trained on the OpenWebText dataset, Derf’s performance was on par with LayerNorm and significantly better than DyT. In LibriSpeech pre-training and GenomicBenchmarks classification tasks, Derf similarly achieved superior results on models like wav2vec 2.0 and HyenaDNA. These experiments prove that a well-designed point-wise function can not only replace normalization layers but can even make Transformers stronger.
