AI Designing AI: The Evolutionary Methodology of AlphaEvolve
For a long time, the design of neural network activation functions has primarily relied on human intuition and experience. The evolution from ReLU to GELU has driven improvements in model performance. In their paper “Finding Generalizable Activation Functions,” published on February 7, 2026, Google DeepMind proposes a transformative automated discovery method—AlphaEvolve.
AlphaEvolve is an evolutionary coding system driven by a Large Language Model (LLM). It abandons the traditional Neural Architecture Search (NAS) paradigm of searching within a predefined set of operations, expanding the search space to the nearly infinite realm of Python functions. Its core workflow is as follows:
- Initialization: Starts with a basic function (e.g., ReLU).
- LLM Proposal: Uses a cutting-edge LLM like Gemini as a “mutation operator” to directly write new Python function code and generate comments explaining the design rationale.
- Micro-evaluation: Evaluates the new function’s performance in a small Multi-Layer Perceptron (MLP) on a specially designed synthetic dataset.
- Fitness Calculation: Crucially, fitness is determined by the model’s loss on out-of-distribution (OOD) test data, forcing the function to possess generalization rather than memorization capabilities.
- Iterative Evolution: The best-performing functions are kept and serve as the basis for the next round of mutations.
The Micro-Lab: Forging Generalization in Synthetic Data
To avoid the high computational costs of searching on large datasets like ImageNet, DeepMind constructed a “micro-lab.” This strategy uses a series of simple synthetic regression tasks, such as random polynomials and spherical harmonics, to test a model’s ability to capture underlying data structures and extrapolate. By creating a clear distributional shift between the training and testing sets (e.g., training on the interval (0, 0.5) and testing on (0.5, 1)), this lab can efficiently screen for activation functions with stronger inductive biases.
Periodic Perturbations: Discoveries from GELUSine to Sinc
The search results from AlphaEvolve revealed an interesting pattern: the best-performing activation functions often follow the structure of standard activation + periodic perturbation term. Two representative discoveries are:
- GELUSine:
GELU(x) + sin(x). The introduced sine term is thought to help the optimization process explore the loss landscape and avoid getting trapped in local minima.
- GELU-Sinc-Perturbation:
GELU(x) * (1 + 0.5 * sinc(x)). This function introduces controlled nonlinearity near the origin via the Sinc function, enhancing the model’s expressive power.
The research process also produced an important “failure” case—the “Turbulent Activation.” This function performed perfectly on synthetic data by overfitting to the batch statistics of the training data, but its performance dropped dramatically in real-world vision tasks. This inversely demonstrated the robustness and universality of point-wise activation functions.
Practical Validation: A Dual Victory in Algorithmic and Vision Tasks
The newly discovered activation functions proved their effectiveness across multiple benchmarks. On the CLRS-30 algorithmic reasoning benchmark, which emphasizes generalization, GELU-Sinc-Perturbation’s score (0.887) was significantly higher than ReLU’s (0.862) and GELU’s (0.874). This validates the core hypothesis of the “micro-lab” strategy: functions optimized on synthetic OOD data can effectively transfer to real tasks requiring strong generalization.
On the ImageNet image classification task, the Top-1 accuracy of new functions like GELUSine was on par with GELU (approx. 74.5%) and far surpassed ReLU (73.5%), indicating their strong competitiveness in standard vision tasks as well. The researchers believe that the periodic structures allow the model to “store” and “retrieve” frequency information, enabling more effective reasoning outside the training domain. This opens up a new way of thinking for activation function design.
