Background: The Cognitive Gap in AI Code Summarization
While current Large Language Models (LLMs) can generate fluent code summaries, their attention mechanisms often diverge from those of human developers, leading to summaries that miss the code’s core functionality. Models are frequently misled by superficial lexical features, whereas humans concentrate on crucial semantic regions such as method declarations and core variables.
Core Design: How EyeLayer Simulates Human Attention
To address this issue, the team at Vanderbilt University developed the lightweight module EyeLayer. Its core innovations include:
- Multimodal Gaussian Mixture Model: Through mathematical modeling, it precisely replicates the multiple focal points of attention and their scopes that occur when humans read code.
- Causality-Aware Re-distribution Mechanism: Without disrupting the model’s original causal structure, it injects the learned human attention priors into Transformer layers as residual perturbations, ensuring perfect compatibility with auto-regressive models such as LLaMA and Qwen.
Experimental Validation: Significant Performance Gains Across Architectures
The research team conducted tests on five mainstream models, including Llama3.2, Qwen3, and CodeBERT. The results indicate that EyeLayer delivered across-the-board performance enhancements, with especially notable gains for smaller models. For example, after integrating EyeLayer, Qwen3-1.7B’s ROUGE-L score improved by 5.28, and CodeBERT’s METEOR score increased by 1.83. Ablation studies proved that optimal performance is achieved when the module is placed in the later Transformer layers (such as the 13th layer in Llama3.2-1B) and employs the multimodal design.
Significance: Towards Human-Aligned AI for Code Comprehension
The value of EyeLayer extends beyond mere metric improvements; it opens up a new avenue of ‘human cognitive prior enhancement’ in the field of AI for Software Engineering (AI4SE). This framework is promising for broader software engineering tasks like bug localization and code review. By aligning AI’s attention with human focus, it aims to boost model interpretability and trustworthiness, advancing the evolution of human-AI collaborative development.
