Research · University of Edinburgh collaboration · 2026 · ongoing
Do Coding Agents Write Complete Patches?
A benchmark that checks coding-agent patches against the fixes developers shipped later.
- Evaluation
- Benchmarks
- Coding agents
Overview
When a security fix is incomplete, developers usually learn it from a second advisory months later. I built a benchmark from those vulnerability chains (PatchChain): 149 tasks from 106 projects in 9 languages. Each task starts from the code just before the first fix, gives the agent only the original advisory, and grades the patch in a network-isolated verifier that replays proof-of-vulnerability tests for the reported bug and for every gap a later fix closed. The work continued remotely with Dr. Marc Juarez after my Edinburgh appointment.
Contributions and results
- Evaluated four frontier models in one fixed coding-agent harness over 1,788 runs: complete-patch rates of 32.7–45.4%.
- A grader that tests only the reported bug would score the same patches about 2× higher: 51% of patches that fix the reported bug without breaking tests leave a gap developers later fixed under a new CVE.
- Which task an agent gets explains far more of the outcome than which model runs it.
- Analysed every failed run with instructed model coders and derived six checkable design rules for patching agents — proposals, not yet tested as interventions.