Try for free

Provisional · release 2026-10-12

Security Review benchmark

13 PRs, 18 known issues. Qodo v2 leads at F1 73.3%. Security Review on DeepSeek V4.1 Flash: F1 62.5% at $0.063 a review.

About this lane
  • Provisional scoped-label matching.
  • Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
  • Some security labels are disputed and under review.
  • The lane’s labels were scoped from the code review labels and not yet checked by a second rater.
  • Review apps were not run as specialists: their general reviews were rescored for this lane.
Security Review: how each reviewer scoresProvisional
Security Review: how each reviewer scores
Model / AppReviewer ScoreF1Latency$/run
IonWarp · GLM 5.3 Flash88.566.7%1m 38s$0.012
IonWarp · DeepSeek V4.1 FlashRuns on your PRs63.462.5%5m 56s$0.063
Qodo v262.873.3%7m 16s~$1.67
IonWarp · Muse Spark 1.342.244.4%2m 30s$0.258
Greptile v4.140.253.8%5m 02s~$1.75
IonWarp · Grok 4.639.753.3%7m 42s$0.575
IonWarp · Gemini 3.8 Flash38.551.6%7m 10s$0.520
CodeRabbit34.854.0%9m 26s~$1.50
Cursor Bugbot31.350.0%8m 23s~$1.25
IonWarp · DeepSeek V4 Pro31.334.8%2m 40s$0.131
IonWarp · GPT 5.6 Luna Pro31.228.6%1m 16s$0.105

Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.

Security Review

IonWarp inferenceApp price estimatesF1 on the same 13 open-source pull requests for every reviewer

20%30%40%50%60%70%80%$0.01$0.03$0.10$0.30$1.00Cost per review · USD · log scaleQodo v2GLM 5.3 FlashDeepSeek V4.1 FlashCodeRabbitGreptile v4.1Grok 4.6Gemini 3.8 FlashCursor BugbotMuse Spark 1.3DeepSeek V4 ProGPT 5.6 Luna Pro
Precision, recall and the counts behind F1
Security Review: precision, recall and the counts behind F1
Model / appEffortPrecisionRecallCaughtUnmatchedMissedPRsTime p90Measured
IonWarp · GLM 5.3 Flashlow83.3%55.6%102813 / 1319m 40s2026-09-10 – 2026-09-14
IonWarp · DeepSeek V4.1 Flashmedium71.4%55.6%104813 / 1313m 19s2026-09-25 – 2026-09-25
Qodo v2—91.7%61.1%111713 / 139m 48s2026-02-27 – 2026-02-27
IonWarp · Muse Spark 1.3medium66.7%33.3%631213 / 133m 35s2026-09-08 – 2026-09-14
Greptile v4.1—87.5%38.9%711113 / 138m 54s2026-04-07 – 2026-04-07
IonWarp · Grok 4.6medium66.7%44.4%841013 / 1312m 45s2026-09-08 – 2026-09-15
IonWarp · Gemini 3.8 Flashmedium61.5%44.4%851013 / 1313m 22s2026-09-08 – 2026-09-15
CodeRabbit—52.6%55.6%109813 / 1315m 13s2026-01-23 – 2026-04-05
Cursor Bugbot—100.0%33.3%601213 / 1311m 41s2026-01-28 – 2026-01-28
IonWarp · DeepSeek V4 Promedium80.0%22.2%411413 / 134m 29s2026-09-08 – 2026-09-14
IonWarp · GPT 5.6 Luna Promedium100.0%16.7%301513 / 132m 28s2026-09-10 – 2026-09-14
Every pull request: known issues caught by each reviewer13 PRs
Known issues caught by each reviewer, per pull request
Pull requestIssuesGLM 5.3 FlashDeepSeek V4.1 FlashQodo v2Muse Spark 1.3Greptile v4.1Grok 4.6Gemini 3.8 FlashCodeRabbitCursor BugbotDeepSeek V4 ProGPT 5.6 Luna Pro
cal.com#10600110000001001
cal.com#14740111110111110
cal.com#22532000000000000
discourse-graphite#3212110112111
discourse-graphite#4532422322100
discourse-graphite#10111101001000
grafana#79265111001110100
grafana#103633211101001010
keycloak#36880100000000000
keycloak#37038211222122211
keycloak#37429100000000000
keycloak#38446000000000000
sentry#67876101100110000

A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 11 reviewers; a wider screen shows them all.

Put Security Review on your next PR.

Try for free

14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.