Provisional · release 2026-10-12
API Performance benchmark
18 PRs, 13 known issues. Muse Spark 1.3 leads at F1 56.0%. API Performance on DeepSeek V4.1 Flash: F1 53.3% at $0.043 a review.
9:41

Benchmark · Provisional
API Performance
18 real pull requests, 13 known issues
ReviewerScoreF1
- 1Muse Spark 1.3Model77.856.0%
- 2GLM 5.3 FlashModel63.448.3%
- 3Greptile v4.1Review app61.976.9%
- 4GPT 5.6 Luna ProModel61.350.0%
- 5Qodo v2Review app60.966.7%
- 6Cursor BugbotReview app60.071.4%
- 7DeepSeek V4 ProModel59.752.2%
- 8DeepSeek V4.1 FlashAPI Performance runs this52.653.3%
- 9Gemini 3.8 FlashModel15.941.7%
- 10CodeRabbitReview app12.244.4%
- Provisional scoped-label matching.
- Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
- Some pull requests and labels were added by us, not taken from a public corpus.
- Review apps were not run as specialists: their general reviews were rescored for this lane.
| Model / App | Reviewer Score | F1 | Latency | $/run |
|---|---|---|---|---|
| IonWarp · Muse Spark 1.3 | 77.8 | 56.0% | 42s | $0.063 |
| IonWarp · GLM 5.3 Flash | 63.4 | 48.3% | 49s | $0.0049 |
| Greptile v4.112 of 18 PRs | — | 76.9% | 4m 59s | ~$1.75 |
| IonWarp · GPT 5.6 Luna Pro | 61.3 | 50.0% | 28s | $0.030 |
| Qodo v212 of 18 PRs | — | 66.7% | 5m 43s | ~$1.67 |
| Cursor Bugbot12 of 18 PRs | — | 71.4% | 7m 57s | ~$1.25 |
| IonWarp · DeepSeek V4 Pro | 59.7 | 52.2% | 1m 21s | $0.042 |
| IonWarp · DeepSeek V4.1 FlashRuns on your PRs | 52.6 | 53.3% | 6m 27s | $0.043 |
| IonWarp · Gemini 3.8 Flash | 15.9 | 41.7% | 47s | $0.071 |
| CodeRabbit12 of 18 PRs | — | 44.4% | 6m 00s | ~$1.50 |
Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Rows measured on part of the PRs get no Score. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.
API Performance
IonWarp inferenceApp price estimatesF1 on the same 18 open-source pull requests for every reviewer
| Model / app | Effort | Precision | Recall | Caught | Unmatched | Missed | PRs | Time p90 | Measured |
|---|---|---|---|---|---|---|---|---|---|
| IonWarp · Muse Spark 1.3 | medium | 58.3% | 53.8% | 7 | 5 | 6 | 18 / 18 | 1m 31s | 2026-09-19 – 2026-09-21 |
| IonWarp · GLM 5.3 Flash | low | 43.8% | 53.8% | 7 | 9 | 6 | 18 / 18 | 2m 32s | 2026-09-19 – 2026-09-21 |
| Greptile v4.1 | — | 83.3% | 71.4% | 5 | 1 | 2 | 12 / 18 | 5m 45s | 2026-04-07 – 2026-04-07 |
| IonWarp · GPT 5.6 Luna Pro | medium | 54.5% | 46.2% | 6 | 5 | 7 | 18 / 18 | 44s | 2026-09-19 – 2026-09-22 |
| Qodo v2 | — | 54.5% | 85.7% | 6 | 5 | 1 | 12 / 18 | 7m 29s | 2026-02-27 – 2026-02-27 |
| Cursor Bugbot | — | 71.4% | 71.4% | 5 | 2 | 2 | 12 / 18 | 11m 41s | 2026-01-28 – 2026-01-28 |
| IonWarp · DeepSeek V4 Pro | medium | 60.0% | 46.2% | 6 | 4 | 7 | 18 / 18 | 7m 35s | 2026-09-19 – 2026-09-21 |
| IonWarp · DeepSeek V4.1 Flash | medium | 47.1% | 61.5% | 8 | 9 | 5 | 18 / 18 | 16m 38s | 2026-09-24 – 2026-09-24 |
| IonWarp · Gemini 3.8 Flash | medium | 45.5% | 38.5% | 5 | 6 | 8 | 18 / 18 | 2m 15s | 2026-09-21 – 2026-09-22 |
| CodeRabbit | — | 100.0% | 28.6% | 2 | 0 | 5 | 12 / 18 | 10m 10s | 2026-01-23 – 2026-04-05 |
| Pull request | Issues | Muse Spark 1.3 | GLM 5.3 Flash | Greptile v4.1 | GPT 5.6 Luna Pro | Qodo v2 | Cursor Bugbot | DeepSeek V4 Pro | DeepSeek V4.1 Flash | Gemini 3.8 Flash | CodeRabbit |
|---|---|---|---|---|---|---|---|---|---|---|---|
| cal.com#10600 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
| discourse-graphite#9 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 |
| discourse-graphite#10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| grafana#80329 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 0 |
| grafana#90939 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 |
| grafana#97529 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 |
| keycloak#33832 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| keycloak#36882 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| keycloak#37634 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| netbox#15141 | 1 | 0 | 0 | – | 0 | – | – | 0 | 0 | 0 | – |
| pandas#50331 | 1 | 1 | 1 | – | 1 | – | – | 1 | 1 | 1 | – |
| pandas#65056 | 1 | 1 | 1 | – | 0 | – | – | 0 | 1 | 0 | – |
| sentry#11908 | 1 | 1 | 1 | – | 1 | – | – | 1 | 1 | 1 | – |
| sentry#40636 | 1 | 1 | 1 | – | 1 | – | – | 1 | 1 | 1 | – |
| sentry#45792 | 1 | 1 | 1 | – | 1 | – | – | 1 | 1 | 1 | – |
| sentry#80168 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| sentry#93824 | 2 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 |
| sentry#95633 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 10 reviewers; a wider screen shows them all.
Other lanes
The rest of the benchmark

Code Review
Qodo v2 leads, F1 58.5%. Code Review on DeepSeek V4.1 Flash: 49.2%.

Security Review
Qodo v2 leads, F1 73.3%. Security Review on DeepSeek V4.1 Flash: 62.5%.

AI Architecture
CodeRabbit leads, F1 72.7%. AI Architecture on DeepSeek V4.1 Flash: 47.1%.

Log & Errors
Qodo v2 leads, F1 72.7%. Log & Errors on DeepSeek V4.1 Flash: 59.0%.

Docs Review
IonWarp · Gemini 3.8 Flash leads, F1 68.6%. DeepSeek V4.1 Flash not measured yet.

SOC 2 Review
IonWarp · GLM 5.3 Flash leads, F1 66.7%. SOC 2 Review on DeepSeek V4.1 Flash: 66.7%.
Read next
Read next

Put API Performance on your next PR.
Try for free14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.