Skip to content
IonWarpTry for free
AI code review F1 score

AI code review F1 score: 0.6 points apart, 4.6 times the findings

By Ilya Volodarsky · Updated

CodeRabbit and GPT 5.6 Luna Pro have an AI code review F1 score 0.6 points apart, 42.2% and 41.6%, on the same 50 pull requests. CodeRabbit posted 288 findings and GPT 5.6 Luna Pro posted 63: 4.6 times as many, for nearly the same score.[1]

Try for free

F1 combines precision and recall into one number, so it cannot say which of the two a reviewer gave up. The chart shows both for the nine reviewers in our October benchmark, and the table gives the counts behind each score.

Precision and recall, one dot per reviewer

  • IonWarp Code Review, one model
  • Review app
  • Same F1, 42%
30%40%50%60%70%30%40%50%60%70%80%Precision: share of its findings that matched a known issueRecall: share of known issues foundQodo v2: precision 55.4%, recall 62.0%, F1 58.5%, 177 findingsQodo v2Muse Spark 1.3: precision 65.3%, recall 50.0%, F1 56.6%, 121 findingsMuse Spark 1.3DeepSeek V4 Pro: precision 70.3%, recall 40.5%, F1 51.4%, 91 findingsDeepSeek V4 ProCursor Bugbot: precision 56.9%, recall 46.8%, F1 51.4%, 130 findingsCursor BugbotGreptile v4.1: precision 50.9%, recall 51.3%, F1 51.1%, 159 findingsGreptile v4.1DeepSeek V4.1 Flash: precision 49.7%, recall 48.7%, F1 49.2%, 155 findingsDeepSeek V4.1 FlashGLM 5.3 Flash: precision 50.0%, recall 37.3%, F1 42.8%, 118 findingsGLM 5.3 FlashCodeRabbit: precision 32.6%, recall 59.5%, F1 42.2%, 288 findingsCodeRabbitGPT 5.6 Luna Pro: precision 73.0%, recall 29.1%, F1 41.6%, 63 findingsGPT 5.6 Luna Pro
50 pull requests and 158 known issues, release of Oct 6, 2026. CodeRabbit, GLM 5.3 Flash and GPT 5.6 Luna Pro sit on the dashed curve.
ReviewerF1PrecisionRecallFoundMatched noneFindings per PR
Qodo v2Review app58.5%55.4%62.0%98 of 158793.54
Muse Spark 1.3IonWarp Code Review56.6%65.3%50.0%79 of 158422.42
DeepSeek V4 ProIonWarp Code Review51.4%70.3%40.5%64 of 158271.82
Cursor BugbotReview app51.4%56.9%46.8%74 of 158562.60
Greptile v4.1Review app51.1%50.9%51.3%81 of 158783.18
DeepSeek V4.1 FlashIonWarp Code Review49.2%49.7%48.7%77 of 158783.10
GLM 5.3 FlashIonWarp Code Review42.8%50.0%37.3%59 of 158592.36
CodeRabbitReview app42.2%32.6%59.5%94 of 1581945.76
GPT 5.6 Luna ProIonWarp Code Review41.6%73.0%29.1%46 of 158171.26

What the score hides

  • CodeRabbit found the most known issues of the three

    94 of 158 known issues, 59.5% recall, and 194 findings that matched none. That is 5.76 findings per pull request, 1.88 of them matching a known issue. Only Qodo v2 found more.[1]

  • GPT 5.6 Luna Pro found half as many and posted far less

    46 of 158, 29.1% recall, and 17 findings that matched none: 1.26 findings per pull request, 0.92 of them matching.[1]

  • GLM 5.3 Flash sits between them

    59 of 158, 37.3% recall, and 59 findings that matched none: 2.36 findings per pull request, 1.18 of them matching. Its F1 is 42.8%.[1]

  • Where IonWarp lands

    Qodo v2 leads the nine at 58.5%. IonWarp's Code Review reaches 56.6% on Muse Spark 1.3 and 49.2% on its default model, DeepSeek V4.1 Flash, sixth of the nine.[1]

Method and limits

  1. The pull requests

    The Core set of Martian's open code review benchmark: 50 pull requests and 158 known issues.[2] Public code only, with no customer data.

  2. The run

    IonWarp ran it, one trial per reviewer, with one Claude Opus 4.5 judge for all of them. Martian has not certified these results, and the release is marked provisional.[1]

  3. The dates

    The apps ran from Jan 23 to Apr 7, 2026, and the models from Sep 8 to Oct 6, 2026. Every app has shipped changes since, and so have we.[1]

  4. Unmatched does not mean wrong

    A finding counts against precision when it matches none of the 158 known issues. It may still be a real problem the set does not list.

  5. Apps and models are not the same thing

    Each model row is IonWarp's Code Review reviewer running that model. Each app row is the vendor's own product. Read the shape of each reviewer here, not a ranking of products.

Pricing

Simple pricing for your whole team.

One workspace price with seats included. Credits pool across your team. Unlimited repos on every plan.

Starter

$0 /mo

Try AI PR review.

  • 2 reviewers
    • Code Review
    • Security Review
  • 3 seats included
  • 15,000 credits to start, then 5,000/mo
  • Unlimited repos
Get Started

Pro

Most popular
$49 /mo

Get AI PR review right from day 1.

  • +5 reviewers
    • DevOps Review
    • Docs Review
    • AI Architecture
    • Log & Errors
    • Cost Review
  • 5 seats included
  • +25,000 credits/mo
  • +$29/mo per added seat
  • Unlimited repos
Choose Pro

Max

$149 /mo

Scale AI PR review.

  • +9 reviewers
    • UX Review
    • API Performance
    • Analytics Review
    • SEO Review
    • Changelog
    • CI/CD Performance
    • Plan Review
    • Repeat-Failure Learnings
    • SOC 2 Review
  • 10 seats included
  • +75,000 credits/mo
  • +$29/mo per added seat
  • Unlimited repos
  • Priority support
Choose Max
Enterprise

Custom reviewers + credits · GitHub Enterprise · SSO/SAML + audit logs · Custom security review + SLA

Contact us
FAQ

Frequently asked questions

What is an F1 score in AI code review?

The harmonic mean of precision and recall. Precision is the share of a reviewer's findings that match a known issue, and recall is the share of known issues it found.

Why can two reviewers have nearly the same F1?

F1 trades precision against recall, so a reviewer that finds more and posts more can score about the same as one that finds less and posts less.

Does an unmatched finding mean the reviewer was wrong?

Not always. It means the finding matched none of the 158 known issues in the set. Some unmatched findings are real problems the set does not list.

Which AI code review tool has the highest F1?

In the release of Oct 6, 2026, Qodo v2 at 58.5%, then IonWarp's Code Review on Muse Spark 1.3 at 56.6%. IonWarp's default model, DeepSeek V4.1 Flash, scored 49.2%, sixth of nine.

Should I choose an AI code reviewer by F1?

Use it to shortlist. Then compare found and unmatched, and try the reviewer on your own pull requests, since a benchmark set is not your codebase.

Sources

  1. The code review benchmark: 50 pull requests, 158 known issues (release of 2026-10-06)IonWarp benchmark · read
  2. withmartian/code-review-benchmarkMartian on GitHub · read