Yangqing Jia· @jiayq · X·· 15 天前AI 评分38
AI 导读
我们从真实项目中收集了约 10,000 个修复 bug 的 pull request，这些项目中引入 bug 的 commit 是已知的。经过逐一审核，我们在 57 个代码库、七种语言中保留了 284 个任务。 每个任务都模拟一次真实的代码审查。智能体能看到引入 bug 的 commit 以及完整的代码库上下文，但永远看不到修复方案。评判智能体只有在它于正确位置指出正确根因时，才算作成功发现。
正文
We started with about 10,000 bug-fix pull requests from real-world projects where the commit that introduced the bug was known. After auditing each one, we kept 284 tasks across 57 repositories and seven languages.
Each task mirrors a real code review. The agent sees the commit that introduced the bug, with full repository context, but never the fix. A judge agent counts it as caught only if it names the right root cause in the right place.
来源：Yangqing Jia · x.com