New Benchmark: Simulating a Git Implementation to Measure AI Coding Efficiency
A new benchmark, released on April 12, 2026, was designed and executed by Ruby committer Yusuke Endoh to quantitatively evaluate the efficiency of large language models in generating usable code across different programming languages. The experiment used Anthropic’s Claude Code (based on the Opus 4.6 model) and tasked it with implementing a simplified version of the Git version control system from scratch in 13 different languages.
The test consisted of two stages, starting with basic init, add, and commit commands and moving on to more complex status and diff functionalities. To ensure fairness and isolate the impact of language ecosystems, the experiment used a custom hash algorithm, avoiding discrepancies from external library dependencies. Each language was run 20 times, for a total of over 600 test runs, with all code and results open-sourced on GitHub.
Cost and Speed Comparison: Dynamic Languages Come Out Ahead
The test data shows that dynamically-typed languages have a clear cost and speed advantage in AI code generation tasks.
The Leaders: Ruby, Python, and JavaScript took the top three spots. Ruby averaged a cost of $0.36 and a time of 73.1 seconds per run; Python cost $0.38 and took 74.6 seconds; and JavaScript cost $0.39 and took 81.1 seconds. These three languages were not only close in cost and time but also had extremely low variance, with all runs passing the tests successfully.
The Overhead of Static Languages: In contrast, generating code for statically-typed languages was 1.4 to 2.6 times slower and more expensive. The average cost for Go rose to $0.50 with a time of 101.6 seconds, and its time standard deviation was a high 37 seconds, indicating instability. Rust had an average cost of $0.54, the widest time range, and was one of only two languages to experience test failures. Due to generating 517 lines of code (compared to just 219 for Ruby), C became the most expensive mainstream language at $0.74.
Type Systems: A Key Factor in AI Generation Efficiency
One of the study’s most practical findings is how type systems directly impact the overhead of AI code generation. When strict type checking was introduced for dynamic languages, their generation efficiency dropped significantly.
- Introducing
mypy for strict type checking in the Python project slowed down generation speed by a factor of 1.6 to 1.7.
- Adding
Steep type checking to the Ruby project resulted in an even more severe performance hit, making it 2.0 to 3.2 times slower than pure Ruby.
- The comparison between TypeScript and JavaScript also confirmed this point. Although both produced a similar number of lines of code, TypeScript’s average cost ($0.62) was significantly higher than JavaScript’s ($0.39).
The researcher theorizes that this extra overhead comes not just from the tokens needed for type annotations, but more likely from the model consuming more “thought tokens” during its reasoning process to satisfy complex type constraints. This increases the model’s internal computational load and ultimate latency.
Experiment Limitations and In-depth Community Discussions
Endoh frankly pointed out several limitations of the experiment, including his potential bias as a Ruby contributor and the small scale of the test project (around 200 lines of code), meaning its conclusions may not generalize to the development of large, complex software. He also acknowledged that the advantages of statically-typed languages are likely more prominent in the maintenance of large codebases.
In tech communities like Lobsters and DEV Community, developers engaged in discussion on the topic. Some comments noted that conclusions based on small prototypes might be unreliable and that the experiment overlooked the significant advantages of mature package ecosystems in real-world development. Furthermore, some argued that the test failures in Rust and Haskell should not be simply seen as bugs, as they demonstrate the ability of their strong type systems to catch potential errors early in development—an advantage in software engineering.
Endoh responded that while a larger-scale benchmark would be more valuable, designing a complex test that is fair to multiple languages is extremely challenging. He also emphasized that in iterative, AI-assisted development, the difference between waiting 30 seconds versus 60 seconds has a significant impact on a developer’s “flow state.”
Out of over 600 total runs, only 3 failures occurred (two for Rust, one for Haskell), an extremely low failure rate. One of the Rust failures was attributed by the author to a model “hallucination,” where the model incorrectly concluded that the test case itself was flawed. The experiment was supported by Anthropic’s Claude Open Source Initiative, which granted the researcher free access to the model.
