TLDR: BugPilot is a new method that trains AI software engineering agents by generating complex, realistic bugs. Instead of intentionally injecting errors, it tasks agents with adding new features, leading to unintentional bugs that closely mimic real-world development. This approach creates more challenging and diverse bugs, resulting in more efficient training data and state-of-the-art performance for open-weight models like FROGBOSS and FROGMINI on software engineering benchmarks.
The field of software engineering is rapidly evolving with the advent of large language model (LLM)-based agents designed to assist with complex tasks like debugging. However, training these sophisticated agents to effectively identify and fix bugs requires a continuous supply of high-quality, diverse, and challenging bug datasets. Traditional methods of bug collection, often relying on mining real-world issues from open-source repositories, are labor-intensive and limited by availability. Synthetic bug generation offers a scalable alternative, but previous approaches have struggled to create bugs that truly mimic the complexity and distribution of those found in human-authored code.
The Challenge of Realistic Bug Generation
Prior attempts at generating synthetic bugs often involve intentionally injecting errors into codebases. While this provides a scalable source of data, these “intentionally” created bugs tend to be simpler, less diverse, and don’t accurately reflect how bugs naturally emerge during software development. This can lead to an “out-of-distribution” effect, where models trained on such data may not perform as well on real-world problems.
BugPilot: A New Approach to Synthetic Bugs
Researchers have introduced a novel methodology called BugPilot, which aims to overcome these limitations by generating more naturalistic bugs. Instead of explicitly instructing a software engineering agent to introduce a bug (a method referred to as BUGINSTRUCT), BugPilot employs a strategy called FEATADD (Buggy Feature Addition). With FEATADD, SWE agents are tasked with developing new features within existing code repositories. Bugs then arise unintentionally when these new feature implementations inadvertently break existing test suites. This process closely mirrors authentic software development scenarios where bugs commonly appear as unforeseen side effects of new feature development and code modifications.
The core idea is that by having agents attempt to add features, they will naturally introduce errors that are more complex and spread across multiple files, much like human developers do. When these modifications cause tests to fail, the state of the repository at that point is recorded as containing a bug that needs resolution. This approach ensures that the generated bugs are not only more challenging but also exhibit characteristics more aligned with real-world issues.
Why Unintentional Bugs Lead to Better Training
Qualitative and quantitative analyses demonstrate that bugs generated through FEATADD are significantly more challenging for current agents to solve. For instance, a strong coding LLM like Claude Sonnet 4 saw its success rate drop from 63.5% on existing datasets to 41.4% on FEATADD bugs. These bugs also involve more extensive code changes, affecting an average of 4.2 files compared to 1.2-2.6 files in other datasets, and feature a more even distribution across various bug categories, closely resembling human-authored bug distributions.
The research shows that these unintentionally generated bugs provide more efficient training data for supervised fine-tuning and reinforcement learning. Models trained with FEATADD data achieve superior performance with less training data. For example, a 32-billion parameter model named FROGBOSS achieved a state-of-the-art pass@1 score of 54.6% on SWE-Bench Verified, and FROGMINI, a 14-billion parameter model, reached 45.3% pass@1. These results were achieved by combining FEATADD bugs with existing datasets, often using significantly less data than previous state-of-the-art models.
An interesting observation from the study highlights the importance of “assistant content” – the reasoning or thought process generated by the teacher model alongside its tool calls. This content proved crucial for effectively distilling code-repairing skills into student models, acting as a Chain-of-Thought that guides the learning process.
Also Read:
- Advancing Automated Code Repair with Agentic Reinforcement Learning
- AI Agents Learn to Control in Minigrid: The Role of Information Access
Advancing Software Engineering Agents
BugPilot represents a significant step forward in creating high-quality synthetic bug datasets. By generating bugs that are more naturalistic, diverse, and difficult, it enables more efficient and effective training of the next generation of LLM-based software engineering agents. This work not only pushes the boundaries of what open-weight models can achieve in SWE tasks but also provides a scalable method for continuously improving these agents. For more details, you can read the full research paper here.
