Try for free

Provisional · release 2026-10-12

API Performance benchmark

18 PRs, 13 known issues. Muse Spark 1.3 leads at F1 56.0%. API Performance on DeepSeek V4.1 Flash: F1 53.3% at $0.043 a review.

About this lane
  • Provisional scoped-label matching.
  • Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
  • Some pull requests and labels were added by us, not taken from a public corpus.
  • Review apps were not run as specialists: their general reviews were rescored for this lane.
API Performance: how each reviewer scoresProvisional
API Performance: how each reviewer scores
Model / AppReviewer ScoreF1Latency$/run
IonWarp · Muse Spark 1.377.856.0%42s$0.063
IonWarp · GLM 5.3 Flash63.448.3%49s$0.0049
Greptile v4.112 of 18 PRs—76.9%4m 59s~$1.75
IonWarp · GPT 5.6 Luna Pro61.350.0%28s$0.030
Qodo v212 of 18 PRs—66.7%5m 43s~$1.67
Cursor Bugbot12 of 18 PRs—71.4%7m 57s~$1.25
IonWarp · DeepSeek V4 Pro59.752.2%1m 21s$0.042
IonWarp · DeepSeek V4.1 FlashRuns on your PRs52.653.3%6m 27s$0.043
IonWarp · Gemini 3.8 Flash15.941.7%47s$0.071
CodeRabbit12 of 18 PRs—44.4%6m 00s~$1.50

Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Rows measured on part of the PRs get no Score. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.

API Performance

IonWarp inferenceApp price estimatesF1 on the same 18 open-source pull requests for every reviewer

40%50%60%$0.01$0.03$0.10Cost per review · USD · log scaleMuse Spark 1.3DeepSeek V4.1 FlashDeepSeek V4 ProGPT 5.6 Luna ProGLM 5.3 FlashGemini 3.8 Flash
Precision, recall and the counts behind F1
API Performance: precision, recall and the counts behind F1
Model / appEffortPrecisionRecallCaughtUnmatchedMissedPRsTime p90Measured
IonWarp · Muse Spark 1.3medium58.3%53.8%75618 / 181m 31s2026-09-19 – 2026-09-21
IonWarp · GLM 5.3 Flashlow43.8%53.8%79618 / 182m 32s2026-09-19 – 2026-09-21
Greptile v4.1—83.3%71.4%51212 / 185m 45s2026-04-07 – 2026-04-07
IonWarp · GPT 5.6 Luna Promedium54.5%46.2%65718 / 1844s2026-09-19 – 2026-09-22
Qodo v2—54.5%85.7%65112 / 187m 29s2026-02-27 – 2026-02-27
Cursor Bugbot—71.4%71.4%52212 / 1811m 41s2026-01-28 – 2026-01-28
IonWarp · DeepSeek V4 Promedium60.0%46.2%64718 / 187m 35s2026-09-19 – 2026-09-21
IonWarp · DeepSeek V4.1 Flashmedium47.1%61.5%89518 / 1816m 38s2026-09-24 – 2026-09-24
IonWarp · Gemini 3.8 Flashmedium45.5%38.5%56818 / 182m 15s2026-09-21 – 2026-09-22
CodeRabbit—100.0%28.6%20512 / 1810m 10s2026-01-23 – 2026-04-05
Every pull request: known issues caught by each reviewer18 PRs
Known issues caught by each reviewer, per pull request
Pull requestIssuesMuse Spark 1.3GLM 5.3 FlashGreptile v4.1GPT 5.6 Luna ProQodo v2Cursor BugbotDeepSeek V4 ProDeepSeek V4.1 FlashGemini 3.8 FlashCodeRabbit
cal.com#1060010010100000
discourse-graphite#910000110000
discourse-graphite#1000000000000
grafana#8032911111110110
grafana#9093911011111101
grafana#9752910110111101
keycloak#3383200000000000
keycloak#3688200000000000
keycloak#3763400000000000
netbox#15141100–0––000–
pandas#50331111–1––111–
pandas#65056111–0––010–
sentry#11908111–1––111–
sentry#40636111–1––111–
sentry#45792111–1––111–
sentry#8016800000000000
sentry#9382420010110000
sentry#9563300000000000

A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 10 reviewers; a wider screen shows them all.

Put API Performance on your next PR.

Try for free

14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.