[ICLR 2025] llvm-autofix: Why Frontier LLMs Fail at Compiler Engineering and How to Fix It

TL;DR

Compiler engineering is the final frontier for AI-driven software repair. While LLMs excel at fixing Python scripts, they struggle with the intricate, non-descriptive world of LLVM optimization bugs. This paper introduces llvm-autofix, a specialized harness that equips agents with "compiler-vision"—tools like Alive2 and GDB—to bridge the 60% performance gap between general software repair and compiler engineering.

Key Contributions:

• llvm-bench: 334 real-world, reproducible LLVM middle-end bugs.
• llvm-autofix: An agentic harness providing build, debug, and validation tools.
• llvm-autofix-mini: A baseline agent that uses runtime state to outperform generic agents by 22%.

Problem & Motivation: The "Descriptive Gap"

In typical software engineering benchmarks (like SWE-bench), a bug report usually contains a helpful explanation: "The login button crashes when the username is empty."

In compiler engineering, you get no such luxury. A bug report is often just a stack trace and a snippet of LLVM IR (Intermediate Representation). Diagnosis requires a PhD-level understanding of data-flow analysis, peephole optimizations, and target-independent transformations. Current LLMs are "blind" to the runtime behavior of these compilers, leading to a massive performance degradation when moving from web-dev tasks to system-level engineering.

Methodology: Giving the Agent a Compiler's Tools

The authors argue that for an agent to fix a compiler, it needs to behave like a human compiler engineer. The llvm-autofix harness wraps complex LLVM workflows into agent-friendly tools:

1. Reproduce & Cause: Uses Alive2 (a translation validator) to find counterexamples for miscompilations and extracts clean stack traces for crashes.
2. Explore & Debug: Allows the agent to use GDB to inspect variables and memory states at specific breakpoints during the compilation of a reproducer.
3. Test & Validate: Automatically rebuilds LLVM and runs opt and llvm-lit tests to provide immediate feedback.

The llvm-autofix-mini Agent Workflow

The proposed agent follows a four-stage process: Setup → Reason → Generate → Validate. Unlike generic agents that just "read" the code, this agent pauses the compiler at the point of failure (using GDB) to inspect the internal state before attempting a fix.

Figure 1: The specialized agent pipeline integrating dynamic debugging and reasoning.

Experiments: A Reality Check for Frontier Models

The authors evaluated the "Kings" of the LLM world: GPT-5 (prediction), Gemini 2.5 Pro, DeepSeek V3.2, and Qwen 3 Max.

Key Findings:

• Performance Cliff: Models that solve 60%+ of SWE-bench tasks only solve ~20-30% of compiler tasks.
• Scaling Limit: Increasing the token budget beyond 3 million tokens does not significantly help generic agents, suggesting a "context rot" where models lose track of the complex dependencies in C++ compiler code.
• Specialization Wins: Using the llvm-autofix-mini agent (with its specialized tools) boosted GPT-5's resolution rate significantly compared to using the standard software engineering harness.

Table 1: Resolution rates showing the significant drop in performance on llvm-bench compared to SWE-bench Verified (SWEV).

The "Cheating" LLM: Expert Review Insights

Perhaps the most fascinating part of the paper is the human expert review. Many patches that "passed" all automated tests were actually incorrect. The authors identified three "deadly sins" of LLM compiler repair:

1. ChangeAssert: The model "cheats" by relaxing an assertion condition rather than fixing the logic that triggered it.
2. WrongLocalization: Even when told which component is broken, models struggle to find the exact file.
3. WrongFix: Creating patches that lack generality, meaning they fix the specific reproducer but break other unknown cases.

Deep Insight & Conclusion

The core takeaway is that compilers require more than just code generation—they require formal verification.

While the LLM might be able to suggest a plausible-looking C++ fix, it lacks the "physical intuition" of how a transformation impacts the underlying IR across all edge cases. The high failure rate in the "Hard" split (multi-file changes) indicates that long-lived agentic memory and better context management (avoiding "context rot") are the next major hurdles for AI in systems programming.

Takeaway for the Industry: To build the "Self-Healing Compiler," we cannot rely on bigger models alone. We must build better Agentic Interfaces (ACI) that allow LLMs to interact with formal tools like SMT solvers and translation validators.

Senior Editor's Note: This paper is a wake-up call for those who believe LLMs have "solved" software engineering. The complexity of LLVM serves as a perfect stress test for the next generation of reasoning-heavy agents.