Ranked · release 2026-10-12
Code Review benchmark
50 PRs, 158 known issues. Qodo v2 leads at F1 58.5%. Code Review on DeepSeek V4.1 Flash: F1 49.2% at $0.102 a review.
9:41

Benchmark · Ranked
Code Review
50 real pull requests, 158 known issues
ReviewerScoreF1
- 1Muse Spark 1.3Model71.256.6%
- 2Qodo v2Review app64.758.5%
- 3DeepSeek V4 ProModel48.751.4%
- 4DeepSeek V4.1 FlashCode Review runs this41.049.2%
- 5GLM 5.3 FlashModel40.842.8%
- 6Greptile v4.1Review app39.951.1%
- 7Cursor BugbotReview app36.251.4%
- 8GPT 5.6 Luna ProModel31.841.6%
- 9CodeRabbitReview app3.142.1%
- Pinned Martian Core reproduction.
- Judge: Martian Core, Claude Opus 4.5 judge.
| Model / App | Reviewer Score | F1 | Latency | $/run |
|---|---|---|---|---|
| IonWarp · Muse Spark 1.3 | 71.2 | 56.6% | 3m 18s | $0.424 |
| Qodo v2 | 64.7 | 58.5% | 5m 40s | ~$1.67 |
| IonWarp · DeepSeek V4 Pro | 48.7 | 51.4% | 5m 59s | $0.165 |
| IonWarp · DeepSeek V4.1 FlashRuns on your PRs | 41.0 | 49.2% | 6m 56s | $0.102 |
| IonWarp · GLM 5.3 Flash | 40.8 | 42.8% | 2m 15s | $0.018 |
| Greptile v4.1 | 39.9 | 51.1% | 4m 58s | ~$1.75 |
| Cursor Bugbot | 36.2 | 51.4% | 7m 52s | ~$1.25 |
| IonWarp · GPT 5.6 Luna Pro | 31.8 | 41.6% | 1m 47s | $0.116 |
| CodeRabbit | 3.1 | 42.1% | 7m 34s | ~$1.50 |
Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Pinned Martian Core reproduction. Release 2026-10-12, CC BY 4.0.
Code Review
IonWarp inferenceApp price estimatesF1 on the same 50 open-source pull requests for every reviewer
| Model / app | Effort | Precision | Recall | Caught | Unmatched | Missed | PRs | Time p90 | Measured |
|---|---|---|---|---|---|---|---|---|---|
| IonWarp · Muse Spark 1.3 | medium | 65.3% | 50.0% | 79 | 42 | 79 | 50 / 50 | 7m 31s | 2026-09-08 – 2026-09-11 |
| Qodo v2 | — | 55.4% | 62.0% | 98 | 79 | 60 | 50 / 50 | 8m 10s | 2026-02-27 – 2026-02-27 |
| IonWarp · DeepSeek V4 Pro | medium | 70.3% | 40.5% | 64 | 27 | 94 | 50 / 50 | 13m 19s | 2026-09-11 – 2026-09-13 |
| IonWarp · DeepSeek V4.1 Flash | medium | 49.7% | 48.7% | 77 | 78 | 81 | 50 / 50 | 8m 15s | 2026-10-05 – 2026-10-06 |
| IonWarp · GLM 5.3 Flash | low | 50.0% | 37.3% | 59 | 59 | 99 | 50 / 50 | 9m 52s | 2026-09-08 – 2026-09-11 |
| Greptile v4.1 | — | 50.9% | 51.3% | 81 | 78 | 77 | 50 / 50 | 7m 09s | 2026-04-07 – 2026-04-07 |
| Cursor Bugbot | — | 56.9% | 46.8% | 74 | 56 | 84 | 50 / 50 | 11m 41s | 2026-01-28 – 2026-01-28 |
| IonWarp · GPT 5.6 Luna Pro | medium | 73.0% | 29.1% | 46 | 17 | 112 | 50 / 50 | 2m 44s | 2026-09-11 – 2026-09-13 |
| CodeRabbit | — | 32.6% | 59.5% | 94 | 194 | 64 | 50 / 50 | 12m 10s | 2026-01-23 – 2026-04-05 |
A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 9 reviewers; a wider screen shows them all.
Other lanes
The rest of the benchmark

Security Review
Qodo v2 leads, F1 73.3%. Security Review on DeepSeek V4.1 Flash: 62.5%.

AI Architecture
CodeRabbit leads, F1 72.7%. AI Architecture on DeepSeek V4.1 Flash: 47.1%.

Log & Errors
Qodo v2 leads, F1 72.7%. Log & Errors on DeepSeek V4.1 Flash: 59.0%.

Docs Review
IonWarp · Gemini 3.8 Flash leads, F1 68.6%. DeepSeek V4.1 Flash not measured yet.

API Performance
IonWarp · Muse Spark 1.3 leads, F1 56.0%. API Performance on DeepSeek V4.1 Flash: 53.3%.

SOC 2 Review
IonWarp · GLM 5.3 Flash leads, F1 66.7%. SOC 2 Review on DeepSeek V4.1 Flash: 66.7%.

Put Code Review on your next PR.
Try for free14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.