Methodology
How we measure AI code reviewers
Every number on the benchmark pages comes from the method below. The data behind it is public, so you can rerun any of it.
The pull requests
Code review uses the 50 pull requests of the Core set of Martian’s code-review-bench, pinned at revision e616e849 (MIT). They come from cal.com, Discourse, Grafana, Keycloak and Sentry, and the corpus’s 158 expected issues are the answer key.
Specialist lanes (Security, AI Architecture, Log & Errors, Docs, API Performance, SOC 2, CI/CD and the historical Cost lane) use 11 to 20 public pull requests each, with expected issues scoped to the lane’s kind of problem. Some reuse code review pull requests with scoped labels; others were found by searching for the lane’s own defects, labelled by one rater and admitted by a second. Only public repositories are used, so every case can be opened.
The reviewers
- Review apps (Qodo, Greptile, Cursor Bugbot, CodeRabbit): their own reviews of the same pull request heads, January to April 2026, kept as public mirror pull requests. Time runs from the review trigger to the app’s last review comment.
- Models: IonWarp’s reviewer prompt for each lane, run on each model, September and October 2026, one trial per model and pull request. Time runs from the reviewer’s start to its final output, tool calls and provider waits included.
Grading and F1
A judge model (Claude Opus 4.5) matches each finding to the expected issues. Code review uses Martian’s matching; specialist lanes use a version scoped to the lane.
- Caught is expected issues found, missed is expected issues not found, unmatched is findings that match no expected issue.
- Precision = caught ÷ (caught + unmatched). An unmatched finding is not necessarily wrong: nobody adjudicated it.
- Recall = expected issues caught ÷ expected issues.
- F1 is the harmonic mean of the two, 2 × P × R ÷ (P + R). It is the accuracy figure we quote everywhere.
The Reviewer Score
Score = (3 × F1 + time + cost) ÷ 5Each part is the row’s place between the worst (0) and the best (100) row in the same lane that was run on every pull request. Higher F1, a shorter median time and a lower cost score higher. Scores compare rows within one lane at one release, and move when a row is added. The leader of a lane is the highest F1; a tie goes to the cheaper reviewer, then the faster.
Time and cost
- Time is the median (p50) review time over the lane’s pull requests; the detail table also shows p90.
- Model rows cost the settled provider charges for one review, retries included. That is the inference cost, not a plan price.
- App rows cost the app’s list price ÷ reviews a month for 5 seats and 100 reviews, billed monthly, as checked on 2026-09-08. They are marked with ~.
Ranked and provisional
Only the code review lane is ranked: its pull requests and labels come from a public corpus. A specialist lane is ranked once it has at least 10 pull requests, each with at least two in-scope issues, and labels from two raters. Until then each lane names what still blocks it.
Limits
- One trial per model and pull request: an exploratory comparison, not a significance test.
- Specialist lanes have 6 to 31 expected issues, so one pull request can move F1 by several points.
- App reviews are from early 2026; model rows are newer runs.
- Matching rewards finding the labelled issues. A real issue nobody labelled counts as unmatched.
Check it yourself
The dataset is on GitHub under CC BY 4.0: every row, every pull request, and what each reviewer said. The same data is served as JSON at /benchmarks/v1. Each agent in IonWarp runs on the model named on its lane, and the app shows it on every review.