Provisional · release 2026-10-12
Security Review benchmark
13 PRs, 18 known issues. Qodo v2 leads at F1 73.3%. Security Review on DeepSeek V4.1 Flash: F1 62.5% at $0.063 a review.
9:41

Benchmark · Provisional
Security Review
13 real pull requests, 18 known issues
ReviewerScoreF1
- 1GLM 5.3 FlashModel88.566.7%
- 2DeepSeek V4.1 FlashSecurity Review runs this63.462.5%
- 3Qodo v2Review app62.873.3%
- 4Muse Spark 1.3Model42.244.4%
- 5Greptile v4.1Review app40.253.8%
- 6Grok 4.6Model39.753.3%
- 7Gemini 3.8 FlashModel38.551.6%
- 8CodeRabbitReview app34.854.0%
- 9Cursor BugbotReview app31.350.0%
- 10DeepSeek V4 ProModel31.334.8%
- Provisional scoped-label matching.
- Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
- Some security labels are disputed and under review.
- The lane’s labels were scoped from the code review labels and not yet checked by a second rater.
- Review apps were not run as specialists: their general reviews were rescored for this lane.
| Model / App | Reviewer Score | F1 | Latency | $/run |
|---|---|---|---|---|
| IonWarp · GLM 5.3 Flash | 88.5 | 66.7% | 1m 38s | $0.012 |
| IonWarp · DeepSeek V4.1 FlashRuns on your PRs | 63.4 | 62.5% | 5m 56s | $0.063 |
| Qodo v2 | 62.8 | 73.3% | 7m 16s | ~$1.67 |
| IonWarp · Muse Spark 1.3 | 42.2 | 44.4% | 2m 30s | $0.258 |
| Greptile v4.1 | 40.2 | 53.8% | 5m 02s | ~$1.75 |
| IonWarp · Grok 4.6 | 39.7 | 53.3% | 7m 42s | $0.575 |
| IonWarp · Gemini 3.8 Flash | 38.5 | 51.6% | 7m 10s | $0.520 |
| CodeRabbit | 34.8 | 54.0% | 9m 26s | ~$1.50 |
| Cursor Bugbot | 31.3 | 50.0% | 8m 23s | ~$1.25 |
| IonWarp · DeepSeek V4 Pro | 31.3 | 34.8% | 2m 40s | $0.131 |
| IonWarp · GPT 5.6 Luna Pro | 31.2 | 28.6% | 1m 16s | $0.105 |
Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.
Security Review
IonWarp inferenceApp price estimatesF1 on the same 13 open-source pull requests for every reviewer
| Model / app | Effort | Precision | Recall | Caught | Unmatched | Missed | PRs | Time p90 | Measured |
|---|---|---|---|---|---|---|---|---|---|
| IonWarp · GLM 5.3 Flash | low | 83.3% | 55.6% | 10 | 2 | 8 | 13 / 13 | 19m 40s | 2026-09-10 – 2026-09-14 |
| IonWarp · DeepSeek V4.1 Flash | medium | 71.4% | 55.6% | 10 | 4 | 8 | 13 / 13 | 13m 19s | 2026-09-25 – 2026-09-25 |
| Qodo v2 | — | 91.7% | 61.1% | 11 | 1 | 7 | 13 / 13 | 9m 48s | 2026-02-27 – 2026-02-27 |
| IonWarp · Muse Spark 1.3 | medium | 66.7% | 33.3% | 6 | 3 | 12 | 13 / 13 | 3m 35s | 2026-09-08 – 2026-09-14 |
| Greptile v4.1 | — | 87.5% | 38.9% | 7 | 1 | 11 | 13 / 13 | 8m 54s | 2026-04-07 – 2026-04-07 |
| IonWarp · Grok 4.6 | medium | 66.7% | 44.4% | 8 | 4 | 10 | 13 / 13 | 12m 45s | 2026-09-08 – 2026-09-15 |
| IonWarp · Gemini 3.8 Flash | medium | 61.5% | 44.4% | 8 | 5 | 10 | 13 / 13 | 13m 22s | 2026-09-08 – 2026-09-15 |
| CodeRabbit | — | 52.6% | 55.6% | 10 | 9 | 8 | 13 / 13 | 15m 13s | 2026-01-23 – 2026-04-05 |
| Cursor Bugbot | — | 100.0% | 33.3% | 6 | 0 | 12 | 13 / 13 | 11m 41s | 2026-01-28 – 2026-01-28 |
| IonWarp · DeepSeek V4 Pro | medium | 80.0% | 22.2% | 4 | 1 | 14 | 13 / 13 | 4m 29s | 2026-09-08 – 2026-09-14 |
| IonWarp · GPT 5.6 Luna Pro | medium | 100.0% | 16.7% | 3 | 0 | 15 | 13 / 13 | 2m 28s | 2026-09-10 – 2026-09-14 |
| Pull request | Issues | GLM 5.3 Flash | DeepSeek V4.1 Flash | Qodo v2 | Muse Spark 1.3 | Greptile v4.1 | Grok 4.6 | Gemini 3.8 Flash | CodeRabbit | Cursor Bugbot | DeepSeek V4 Pro | GPT 5.6 Luna Pro |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| cal.com#10600 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 |
| cal.com#14740 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 0 |
| cal.com#22532 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| discourse-graphite#3 | 2 | 1 | 2 | 1 | 1 | 0 | 1 | 1 | 2 | 1 | 1 | 1 |
| discourse-graphite#4 | 5 | 3 | 2 | 4 | 2 | 2 | 3 | 2 | 2 | 1 | 0 | 0 |
| discourse-graphite#10 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 |
| grafana#79265 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 0 |
| grafana#103633 | 2 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 |
| keycloak#36880 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| keycloak#37038 | 2 | 1 | 1 | 2 | 2 | 2 | 1 | 2 | 2 | 2 | 1 | 1 |
| keycloak#37429 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| keycloak#38446 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| sentry#67876 | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 |
A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 11 reviewers; a wider screen shows them all.
Other lanes
The rest of the benchmark

Code Review
Qodo v2 leads, F1 58.5%. Code Review on DeepSeek V4.1 Flash: 49.2%.

AI Architecture
CodeRabbit leads, F1 72.7%. AI Architecture on DeepSeek V4.1 Flash: 47.1%.

Log & Errors
Qodo v2 leads, F1 72.7%. Log & Errors on DeepSeek V4.1 Flash: 59.0%.

Docs Review
IonWarp · Gemini 3.8 Flash leads, F1 68.6%. DeepSeek V4.1 Flash not measured yet.

API Performance
IonWarp · Muse Spark 1.3 leads, F1 56.0%. API Performance on DeepSeek V4.1 Flash: 53.3%.

SOC 2 Review
IonWarp · GLM 5.3 Flash leads, F1 66.7%. SOC 2 Review on DeepSeek V4.1 Flash: 66.7%.
Read next
Read next

Put Security Review on your next PR.
Try for free14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.