Try for free

Provisional · release 2026-10-12

AI Architecture benchmark

11 PRs, 6 known issues. CodeRabbit leads at F1 72.7%. AI Architecture on DeepSeek V4.1 Flash: F1 47.1% at $0.050 a review.

About this lane
  • Provisional scoped-label matching.
  • Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
  • The lane’s labels were scoped from the code review labels and not yet checked by a second rater.
  • Review apps were not run as specialists: their general reviews were rescored for this lane.
AI Architecture: how each reviewer scoresProvisional
AI Architecture: how each reviewer scores
Model / AppReviewer ScoreF1Latency$/run
CodeRabbit60.672.7%7m 43s~$1.50
IonWarp · GLM 5.359.347.1%1m 30s$0.075
IonWarp · GPT 5.6 Luna Pro53.835.3%51s$0.046
IonWarp · DeepSeek V4 Pro53.444.4%1m 52s$0.102
Greptile v4.150.560.0%5m 06s~$1.75
IonWarp · DeepSeek V4.1 FlashRuns on your PRs47.847.1%6m 14s$0.050
IonWarp · GLM 5.3 Flash38.315.4%1m 02s$0.0064
Cursor Bugbot37.450.0%7m 46s~$1.25
IonWarp · Grok 4.634.034.8%3m 29s$0.284
IonWarp · Muse Spark 1.333.628.6%2m 04s$0.193
IonWarp · Gemini 3.8 Flash32.333.3%3m 50s$0.238
Qodo v212.825.0%5m 52s~$1.67

Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.

AI Architecture

IonWarp inferenceApp price estimatesF1 on the same 11 open-source pull requests for every reviewer

10%20%30%40%50%60%70%80%$0.01$0.03$0.10$0.30$1.00Cost per review · USD · log scaleCodeRabbitGreptile v4.1Cursor BugbotGLM 5.3DeepSeek V4.1 FlashDeepSeek V4 ProGPT 5.6 Luna ProGrok 4.6Gemini 3.8 FlashMuse Spark 1.3Qodo v2GLM 5.3 Flash
Precision, recall and the counts behind F1
AI Architecture: precision, recall and the counts behind F1
Model / appEffortPrecisionRecallCaughtUnmatchedMissedPRsTime p90Measured
CodeRabbit—80.0%66.7%41211 / 1112m 10s2026-01-23 – 2026-04-05
IonWarp · GLM 5.3medium36.4%66.7%47211 / 112m 06s2026-09-17 – 2026-09-22
IonWarp · GPT 5.6 Luna Promedium27.3%50.0%38311 / 111m 37s2026-09-10 – 2026-09-17
IonWarp · DeepSeek V4 Promedium33.3%66.7%48211 / 113m 08s2026-09-08 – 2026-09-17
Greptile v4.1—75.0%50.0%31311 / 116m 42s2026-04-07 – 2026-04-07
IonWarp · DeepSeek V4.1 Flashmedium36.4%66.7%47211 / 1110m 39s2026-09-24 – 2026-09-24
IonWarp · GLM 5.3 Flashlow14.3%16.7%16511 / 113m 44s2026-09-10 – 2026-09-17
Cursor Bugbot—40.0%66.7%46211 / 1112m 09s2026-01-28 – 2026-01-28
IonWarp · Grok 4.6medium23.5%66.7%413211 / 114m 12s2026-09-08 – 2026-09-17
IonWarp · Muse Spark 1.3medium20.0%50.0%312311 / 112m 58s2026-09-08 – 2026-09-17
IonWarp · Gemini 3.8 Flashmedium25.0%50.0%39311 / 115m 14s2026-09-08 – 2026-09-22
Qodo v2—50.0%16.7%11511 / 117m 29s2026-02-27 – 2026-02-27
Every pull request: known issues caught by each reviewer11 PRs
Known issues caught by each reviewer, per pull request
Pull requestIssuesCodeRabbitGLM 5.3GPT 5.6 Luna ProDeepSeek V4 ProGreptile v4.1DeepSeek V4.1 FlashGLM 5.3 FlashCursor BugbotGrok 4.6Muse Spark 1.3Gemini 3.8 FlashQodo v2
cal.com#109671100000000000
cal.com#223451010101011000
cal.com#225320000000000000
discourse-graphite#11111111011111
discourse-graphite#100000000000000
grafana#975290000000000000
keycloak#338321111111011110
keycloak#368801111111111110
keycloak#384460000000000000
sentry#805281000000000000
sentry#956330000000000000

A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 12 reviewers; a wider screen shows them all.

Put AI Architecture on your next PR.

Try for free

14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.