Starter
Try AI PR review.
- 2 reviewers
- Code Review
- Security Review
- 3 seats included
- 15,000 credits to start, then 5,000/mo
- About 70 reviews to start, then about 25 a month
- Unlimited repos
Qodo v2 has the highest F1 of the nine reviewers in our October benchmark, 58.5%. Yet on 18 of the 50 pull requests, at least one other reviewer found more of the known issues than it did.[1] On 29 of them, none of the nine found every known issue.[2]
What lands on your pull request · Code Review
billing/checkout.ts:42 · A timed-out response can charge the order twice- await payments.charge(order);+ await payments.charge(order, {+ idempotencyKey: order.id,+ });
So the best reviewer depends on whose pull requests you ask about. An average ranks reviewers across 50 pull requests from five projects. Your team sends one project's pull requests, and the reviewer that leads changes from one project to the next. The chart counts, for each reviewer, the pull requests where nobody found more.
| Project | Pull requests | Known issues | Found the most |
|---|---|---|---|
| Cal.com | 10 | 37 | CodeRabbit, 29 |
| Discourse | 10 | 38 | Qodo v2 and CodeRabbit, 22 |
| Grafana | 10 | 25 | Qodo v2 and Greptile v4.1, 16 |
| Keycloak | 10 | 25 | CodeRabbit, 15 |
| Sentry | 10 | 33 | Qodo v2, 20 |
Qodo v2 was beaten least often, on 18 pull requests. CodeRabbit was beaten on 20, Greptile v4.1 on 27 and Cursor Bugbot on 31. GPT 5.6 Luna Pro, which posts the fewest findings, was beaten on 42.[2]
CodeRabbit found the most on Cal.com, 29 of 37 known issues, and on Keycloak, 15 of 25. Qodo v2 led on Sentry with 20 of 33. It tied with CodeRabbit on Discourse and with Greptile v4.1 on Grafana. The table gives each count.[2]
On just 1 pull request did every reviewer find every known issue. On 2, none of the nine found anything at all.[2]
IonWarp's default model, DeepSeek V4.1 Flash, was beaten on 29 of the 50, and its retry model, Muse Spark 1.3, on 30.[2] Our own reviewer is not the answer to this question either.
The Core set of Martian's open code review benchmark: 50 pull requests from Cal.com, Discourse, Grafana, Keycloak and Sentry, with 158 known issues.[3] Public code only, with no customer data.
IonWarp ran it, one trial per reviewer, with one Claude Opus 4.5 judge for all of them. Martian has not certified these results. The apps ran from Jan 23 to Apr 7, 2026, and the models from Sep 8 to Oct 6, 2026.[1]
Run a reviewer twice on the same pull request and it can find a different number of issues, so some of these swings are chance. One trial cannot tell chance from skill, and most of these pull requests hold only 2 to 4 known issues.
Every count on this page is in the caught column of the public cases file. Group the rows by pull request and compare them.[2]
One workspace price with seats included. Credits pool across your team. Unlimited repos on every plan.
Try AI PR review.
Get AI PR review right from day 1.
Scale AI PR review.
Custom reviewers + credits · GitHub Enterprise · SSO/SAML + audit logs · Custom security review + SLA
On average, Qodo v2 had the highest F1 in our October release, 58.5%. Pull request by pull request, each of the nine was beaten by another on at least 18 of the 50. Try two on your own pull requests before you choose.
Not in this set. On 29 of the 50 pull requests none of the nine found every known issue, and on 2 none of them found any.
Each project has its own language, size and kind of bug, and each project here has 25 to 38 known issues, so a few issues move the lead. Part of the swing is also the noise of a single trial.
Shortlist on a public benchmark, then run two or three on the same recent pull requests from your own repository and count what each found and what it got wrong.
Yes. The per pull request counts are in the public benchmark repository on GitHub, in the cases file for code review.