Provisional · release 2026-10-12
Docs Review benchmark
19 PRs, 17 known issues. Gemini 3.8 Flash leads at F1 68.6%. DeepSeek V4.1 Flash, which Docs Review runs, is not measured here yet.
9:41

Benchmark · Provisional
Docs Review
19 real pull requests, 17 known issues
ReviewerScoreF1
- 1Gemini 3.8 FlashModel79.268.6%
- 2DeepSeek V4 ProModel57.965.2%
- 3Muse Spark 1.3Model20.059.5%
- 4CodeRabbitReview app0.055.6%
- 5Cursor BugbotReview app0.036.4%
- 6Greptile v4.1Review app0.028.6%
- 7Qodo v2Review app0.026.7%
- Provisional scoped-label matching.
- Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
- Some pull requests and labels were added by us, not taken from a public corpus.
- Review apps were not run as specialists: their general reviews were rescored for this lane.
| Model / App | Reviewer Score | F1 | Latency | $/run |
|---|---|---|---|---|
| IonWarp · Gemini 3.8 Flash | 79.2 | 68.6% | 59s | $0.084 |
| IonWarp · DeepSeek V4 Pro | 57.9 | 65.2% | 3m 00s | $0.064 |
| IonWarp · Muse Spark 1.3 | 20.0 | 59.5% | 48s | $0.087 |
| CodeRabbit11 of 19 PRs | — | 55.6% | 7m 43s | ~$1.50 |
| Cursor Bugbot11 of 19 PRs | — | 36.4% | 9m 53s | ~$1.25 |
| Greptile v4.111 of 19 PRs | — | 28.6% | 5m 23s | ~$1.75 |
| Qodo v211 of 19 PRs | — | 26.7% | 6m 29s | ~$1.67 |
Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Rows measured on part of the PRs get no Score. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.
Docs Review
IonWarp inferenceApp price estimatesF1 on the same 19 open-source pull requests for every reviewer
| Model / app | Effort | Precision | Recall | Caught | Unmatched | Missed | PRs | Time p90 | Measured |
|---|---|---|---|---|---|---|---|---|---|
| IonWarp · Gemini 3.8 Flash | medium | 66.7% | 70.6% | 12 | 6 | 5 | 19 / 19 | 1m 33s | 2026-09-21 – 2026-09-22 |
| IonWarp · DeepSeek V4 Pro | medium | 51.7% | 88.2% | 15 | 14 | 2 | 19 / 19 | 7m 44s | 2026-09-21 – 2026-09-23 |
| IonWarp · Muse Spark 1.3 | medium | 55.0% | 64.7% | 11 | 9 | 6 | 19 / 19 | 1m 50s | 2026-09-21 – 2026-09-22 |
| CodeRabbit | — | 55.6% | 55.6% | 5 | 4 | 4 | 11 / 19 | 10m 10s | 2026-01-23 – 2026-04-05 |
| Cursor Bugbot | — | 100.0% | 22.2% | 2 | 0 | 7 | 11 / 19 | 12m 09s | 2026-01-28 – 2026-01-28 |
| Greptile v4.1 | — | 40.0% | 22.2% | 2 | 3 | 7 | 11 / 19 | 5m 51s | 2026-04-07 – 2026-04-07 |
| Qodo v2 | — | 33.3% | 22.2% | 2 | 4 | 7 | 11 / 19 | 9m 33s | 2026-02-27 – 2026-02-27 |
| Pull request | Issues | Gemini 3.8 Flash | DeepSeek V4 Pro | Muse Spark 1.3 | CodeRabbit | Cursor Bugbot | Greptile v4.1 | Qodo v2 |
|---|---|---|---|---|---|---|---|---|
| cal.com#10600 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| cal.com#22532 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| discourse-graphite#10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| grafana#79265 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 1 |
| grafana#97529 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| grafana#103633 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 1 |
| keycloak#36882 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| keycloak#37429 | 2 | 1 | 2 | 2 | 2 | 2 | 0 | 0 |
| keycloak#37634 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
| pandas#43951 | 1 | 1 | 1 | 1 | – | – | – | – |
| pandas#45411 | 1 | 1 | 1 | 0 | – | – | – | – |
| pandas#47633 | 1 | 1 | 1 | 1 | – | – | – | – |
| pandas#57347 | 1 | 0 | 1 | 0 | – | – | – | – |
| pandas#61032 | 1 | 1 | 1 | 1 | – | – | – | – |
| pandas#61264 | 1 | 0 | 1 | 1 | – | – | – | – |
| pandas#61467 | 1 | 1 | 1 | 0 | – | – | – | – |
| pandas#63384 | 1 | 1 | 1 | 1 | – | – | – | – |
| sentry#80168 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 0 |
| sentry#95633 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 7 reviewers; a wider screen shows them all.
Other lanes
The rest of the benchmark

Code Review
Qodo v2 leads, F1 58.5%. Code Review on DeepSeek V4.1 Flash: 49.2%.

Security Review
Qodo v2 leads, F1 73.3%. Security Review on DeepSeek V4.1 Flash: 62.5%.

AI Architecture
CodeRabbit leads, F1 72.7%. AI Architecture on DeepSeek V4.1 Flash: 47.1%.

Log & Errors
Qodo v2 leads, F1 72.7%. Log & Errors on DeepSeek V4.1 Flash: 59.0%.

API Performance
IonWarp · Muse Spark 1.3 leads, F1 56.0%. API Performance on DeepSeek V4.1 Flash: 53.3%.

SOC 2 Review
IonWarp · GLM 5.3 Flash leads, F1 66.7%. SOC 2 Review on DeepSeek V4.1 Flash: 66.7%.

Put Docs Review on your next PR.
Try for free14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.