Skip to content
IonWarpTry for free
Best AI code reviewer

The best AI code reviewer by F1 found fewer issues than another on 18 of 50 pull requests

Qodo v2 has the highest F1 of the nine reviewers in our October benchmark, 58.5%. Yet on 18 of the 50 pull requests, at least one other reviewer found more of the known issues than it did.[1] On 29 of them, none of the nine found every known issue.[2]

Example pull request#310

What lands on your pull request · Code Review

P1Retry charges the same order twice
billing/checkout.ts:42 · A timed-out response can charge the order twice
- await payments.charge(order);+ await payments.charge(order, {+   idempotencyKey: order.id,+ });
When payment succeeds but the response times out, the caller retries the same order. Both calls charge it. Pass the stable order ID as the payment idempotency key.

So the best reviewer depends on whose pull requests you ask about. An average ranks reviewers across 50 pull requests from five projects. Your team sends one project's pull requests, and the reviewer that leads changes from one project to the next. The chart counts, for each reviewer, the pull requests where nobody found more.

Who found the most, one pull request at a time

  • No reviewer found more
  • Another reviewer found more
  • None of the nine found any
  1. Qodo v2Review app3018
  2. Muse Spark 1.3IonWarp Code Review1830
  3. DeepSeek V4 ProIonWarp Code Review1533
  4. Cursor BugbotReview app1731
  5. Greptile v4.1Review app2127
  6. DeepSeek V4.1 FlashIonWarp Code Review1929
  7. GLM 5.3 FlashIonWarp Code Review1335
  8. CodeRabbitReview app2820
  9. GPT 5.6 Luna ProIonWarp Code Review642
The release of Oct 6, 2026 has 50 pull requests and 158 known issues. When reviewers tie for the most on a pull request, it counts for each of them. Rows are in F1 order, highest first.
The most known issues found, by project
ProjectPull requestsKnown issuesFound the most
Cal.com1037CodeRabbit, 29
Discourse1038Qodo v2 and CodeRabbit, 22
Grafana1025Qodo v2 and Greptile v4.1, 16
Keycloak1025CodeRabbit, 15
Sentry1033Qodo v2, 20

What the per pull request results show

  • Every reviewer was beaten somewhere

    Qodo v2 was beaten least often, on 18 pull requests. CodeRabbit was beaten on 20, Greptile v4.1 on 27 and Cursor Bugbot on 31. GPT 5.6 Luna Pro, which posts the fewest findings, was beaten on 42.[2]

  • The leader changes by project

    CodeRabbit found the most on Cal.com, 29 of 37 known issues, and on Keycloak, 15 of 25. Qodo v2 led on Sentry with 20 of 33. It tied with CodeRabbit on Discourse and with Greptile v4.1 on Grafana. The table gives each count.[2]

  • Some issues got past all nine

    On just 1 pull request did every reviewer find every known issue. On 2, none of the nine found anything at all.[2]

  • Where IonWarp lands

    IonWarp's default model, DeepSeek V4.1 Flash, was beaten on 29 of the 50, and its retry model, Muse Spark 1.3, on 30.[2] Our own reviewer is not the answer to this question either.

Method and limits

  1. The pull requests

    The Core set of Martian's open code review benchmark: 50 pull requests from Cal.com, Discourse, Grafana, Keycloak and Sentry, with 158 known issues.[3] Public code only, with no customer data.

  2. The run

    IonWarp ran it, one trial per reviewer, with one Claude Opus 4.5 judge for all of them. Martian has not certified these results. The apps ran from Jan 23 to Apr 7, 2026, and the models from Sep 8 to Oct 6, 2026.[1]

  3. One trial is noisy

    Run a reviewer twice on the same pull request and it can find a different number of issues, so some of these swings are chance. One trial cannot tell chance from skill, and most of these pull requests hold only 2 to 4 known issues.

  4. Check it yourself

    Every count on this page is in the caught column of the public cases file. Group the rows by pull request and compare them.[2]

Pricing

Simple pricing for your whole team.

One workspace price with seats included. Credits pool across your team. Unlimited repos on every plan.

Starter

$0 /mo

Try AI PR review.

  • 2 reviewers
    • Code Review
    • Security Review
  • 3 seats included
  • 15,000 credits to start, then 5,000/mo
  • About 70 reviews to start, then about 25 a month
  • Unlimited repos
Get Started

Max

$149 /mo

Scale AI PR review.

  • +9 reviewers
    • UX Review
    • API Performance
    • Analytics Review
    • SEO Review
    • Changelog
    • CI/CD Performance
    • Plan Review
    • Repeat-Failure Learnings
    • SOC 2 Review
  • 10 seats included
  • +75,000 credits/mo
  • About 360 reviews a month
  • +$29/mo per added seat
  • Unlimited repos
  • Priority support
Choose Max
Enterprise

Custom reviewers + credits · GitHub Enterprise · SSO/SAML + audit logs · Custom security review + SLA

Contact us
FAQ

Frequently asked questions

What is the best AI code reviewer?

On average, Qodo v2 had the highest F1 in our October release, 58.5%. Pull request by pull request, each of the nine was beaten by another on at least 18 of the 50. Try two on your own pull requests before you choose.

Does any AI code reviewer find every bug?

Not in this set. On 29 of the 50 pull requests none of the nine found every known issue, and on 2 none of them found any.

Why does the leader change between projects?

Each project has its own language, size and kind of bug, and each project here has 25 to 38 known issues, so a few issues move the lead. Part of the swing is also the noise of a single trial.

How should I compare AI code reviewers?

Shortlist on a public benchmark, then run two or three on the same recent pull requests from your own repository and count what each found and what it got wrong.

Can I check these numbers?

Yes. The per pull request counts are in the public benchmark repository on GitHub, in the cases file for code review.

Sources

  1. The code review benchmark: 50 pull requests, 158 known issues (release of 2026-10-06)IonWarp benchmark · read
  2. ai-code-review-benchmark: code-review.cases.csv, one row per reviewer and pull requestIonWarp on GitHub · read
  3. withmartian/code-review-benchmarkMartian on GitHub · read