Skip to content
IonWarpTry for free
CodeRabbit review

CodeRabbit review: what it caught and missed on 50 pull requests

By Ilya Volodarsky · Updated

CodeRabbit finds a lot and says a lot. On the 50 open-source pull requests of Martian's code review benchmark, which hold 158 known issues, it found 94, second only to Qodo's 98. It also posted 194 findings that matched no known issue, more than twice any other reviewer we ran, for an F1 of 42.2%, the lowest of the four apps.[1]

Try for free

We sell a competing reviewer, so check the numbers: every grade is on the code review benchmark, and CodeRabbit's own reviews of these pull requests are public.[2] IonWarp's Code Review scored 49.2% on the same table. CodeRabbit's reviews date from January to April 2026, so today's version may do better.

What we found

  • It catches bugs other apps miss

    Among the four apps, 15 known issues were CodeRabbit's alone, against 11 for Qodo, 5 for Bugbot and 3 for Greptile.[1] Of all nine reviewers, only CodeRabbit flagged a Discourse job that opens a URL from site settings, a server-side request forgery,[3] and a Grafana middleware that panics on a nil request.[4]

  • It is the noisiest reviewer we measured

    It posted 5.8 findings a pull request, 3.9 of them matching nothing. Cursor Bugbot posted 2.6 and 1.1, Greptile 3.2 and 1.6, Qodo 3.5 and 1.6.[1] On one Discourse pull request it left 25 findings: 6 matched and 19 did not.[2] Cursor Bugbot vs CodeRabbit sets the two side by side.

  • Unmatched does not mean wrong

    The answer key lists known issues, not every issue, so some of those 194 will be fair comments. Each one still takes someone's time to read and rule out.

  • It varies by codebase

    With 10 pull requests from each repository, it found 29 of 37 known issues on Cal.com (TypeScript) and 15 of 33 on Sentry (Python).[1]

  • Where it led

    AI architecture review: an F1 of 72.7%, the top of every reviewer, with 1 unmatched finding, on a small set of 11 pull requests and 6 known issues.[5]

How to decide if CodeRabbit fits your team

  1. Trial it on a busy repository

    New accounts get a 14-day trial.[6] For those two weeks, count the comments your team acts on and the ones it dismisses. That ratio tells you more than any benchmark, ours included.

  2. Teach it what to skip

    It learns from your feedback, and path instructions narrow what it checks.[7] If your team still dismisses most comments after that, move on.

  3. Check your git platform

    CodeRabbit reviews GitHub, GitLab, Azure DevOps and Bitbucket.[8] IonWarp reviews GitHub only, so off GitHub it is not an option.

  4. Price it for your team

    Essentials is $24 per developer a month billed annually, so 5 developers pay $120.[6] Every CodeRabbit plan and what a team pays. IonWarp Pro is $49/mo for 5 seats.

CodeRabbit on every review type we ran

Each cell: F1, then known issues found and findings that matched none. Only the code review row is a ranked result. The others use provisional answer keys on small sets, and score each app's general review against a narrower key, so read them as a direction. The app reviews date from January to April 2026. IonWarp's column is the model each reviewer runs by default (for docs, its retry model, Gemini 3.8 Flash), on the full set where the apps ran on fewer pull requests.

CriterionIonWarpCodeRabbitQodoCursor BugbotGreptile
Code review (50 PRs)49.2%: 77 of 158, 78 unmatched[1]42.2%: 94 of 158, 194 unmatched[1]58.5%: 98 of 158, 79 unmatched[1]51.4%: 74 of 158, 56 unmatched[1]51.1%: 81 of 158, 78 unmatched[1]
Security review (13 PRs)62.5%: 10 of 18, 4 unmatched[9]54.1%: 10 of 18, 9 unmatched[9]73.3%: 11 of 18, 1 unmatched[9]50.0%: 6 of 18, 0 unmatched[9]53.8%: 7 of 18, 1 unmatched[9]
AI architecture review (11 PRs)47.1%: 4 of 6, 7 unmatched[5]72.7%: 4 of 6, 1 unmatched[5]25.0%: 1 of 6, 1 unmatched[5]50.0%: 4 of 6, 6 unmatched[5]60.0%: 3 of 6, 1 unmatched[5]
Log and error review (17 PRs)59.0%: 18 of 31, 12 unmatched[10]59.2%: 21 of 31, 19 unmatched[10]72.7%: 24 of 31, 11 unmatched[10]65.6%: 21 of 31, 12 unmatched[10]67.8%: 20 of 31, 8 unmatched[10]
Docs review (11 PRs)68.6%: 12 of 17, 6 unmatched, on 19 PRs[11]55.6%: 5 of 9, 4 unmatched[11]26.7%: 2 of 9, 4 unmatched[11]36.4%: 2 of 9, 0 unmatched[11]28.6%: 2 of 9, 3 unmatched[11]
API performance review (12 PRs)53.3%: 8 of 13, 9 unmatched, on 18 PRs[12]44.4%: 2 of 7, 0 unmatched[12]66.7%: 6 of 7, 5 unmatched[12]71.4%: 5 of 7, 2 unmatched[12]76.9%: 5 of 7, 1 unmatched[12]
Pricing

Simple pricing for your whole team.

One workspace price with seats included. Credits pool across your team. Unlimited repos on every plan.

Starter

$0 /mo

Try AI PR review.

  • 2 reviewers
    • Code Review
    • Security Review
  • 3 seats included
  • 15,000 credits to start, then 5,000/mo
  • Unlimited repos
Get Started

Pro

Most popular
$49 /mo

Get AI PR review right from day 1.

  • +5 reviewers
    • DevOps Review
    • Docs Review
    • AI Architecture
    • Log & Errors
    • Cost Review
  • 5 seats included
  • +25,000 credits/mo
  • +$29/mo per added seat
  • Unlimited repos
Choose Pro

Max

$149 /mo

Scale AI PR review.

  • +9 reviewers
    • UX Review
    • API Performance
    • Analytics Review
    • SEO Review
    • Changelog
    • CI/CD Performance
    • Plan Review
    • Repeat-Failure Learnings
    • SOC 2 Review
  • 10 seats included
  • +75,000 credits/mo
  • +$29/mo per added seat
  • Unlimited repos
  • Priority support
Choose Max
Enterprise

Custom reviewers + credits · GitHub Enterprise · SSO/SAML + audit logs · Custom security review + SLA

Contact us
FAQ

Frequently asked questions

Is CodeRabbit good?

It is good at finding issues and poor at staying quiet. On 50 pull requests it found 94 of 158 known issues, second of the four apps, and posted 194 findings that matched none, the most of any reviewer. If your team can triage that volume, it catches bugs the other apps miss.

Is CodeRabbit worth it?

At $24 per developer a month billed annually (CodeRabbit pricing page, Nov 2026), it is worth a trial if a missed bug costs you more than extra comments do. If noise is your team's complaint, Cursor Bugbot and Greptile posted far fewer unmatched findings on the same pull requests.

Is CodeRabbit free?

In part. The free plan summarizes pull requests on any repository, and open-source projects get free reviews on public repositories. Pull request reviews on a private repository need a paid plan, and new accounts start with a 14-day trial.

How does CodeRabbit compare with Qodo, Bugbot and Greptile?

By F1 on the same 50 pull requests: Qodo 58.5%, Cursor Bugbot 51.4%, Greptile 51.1%, CodeRabbit 42.2%. CodeRabbit found more issues than Bugbot and Greptile, and posted more than twice as many unmatched findings as any of them.

How did you test CodeRabbit?

With Martian's open benchmark: 50 real pull requests from Cal.com, Discourse, Grafana, Keycloak and Sentry, each with its known issues. CodeRabbit's reviews are public on GitHub, and Claude Opus 4.5 graded whether each finding matched a known issue. One review of each pull request, so read small gaps as ties.

Sources

  1. The code review benchmark: 50 pull requests, 158 known issues (release of 2026-10-06)IonWarp benchmark · read
  2. CodeRabbit's review of Discourse pull request 4, in the benchmark's public forkCodeRabbit on GitHub · read
  3. discourse #4 (embed URL handling), graded code review findingsIonWarp benchmark · read
  4. grafana/grafana #76186 (contextual logger middleware), graded code review findingsIonWarp benchmark · read
  5. The AI architecture review benchmark: 11 pull requests, 6 known issues (release of 2026-09-24)IonWarp benchmark · read
  6. PricingCodeRabbit · read
  7. Pull request reviewCodeRabbit docs · read
  8. Platforms overviewCodeRabbit docs · read
  9. The security review benchmark: 13 pull requests, 18 known issues (release of 2026-09-24)IonWarp benchmark · read
  10. The log and error review benchmark: 17 pull requests, 31 known issues (release of 2026-09-24)IonWarp benchmark · read
  11. The docs review benchmark: 19 pull requests, 17 known issues (release of 2026-09-22)IonWarp benchmark · read
  12. The API performance review benchmark: 18 pull requests, 13 known issues (release of 2026-09-24)IonWarp benchmark · read