Provisional · release 2026-10-12
AI Architecture benchmark
11 PRs, 6 known issues. CodeRabbit leads at F1 72.7%. AI Architecture on DeepSeek V4.1 Flash: F1 47.1% at $0.050 a review.
9:41

Benchmark · Provisional
AI Architecture
11 real pull requests, 6 known issues
ReviewerScoreF1
- 1CodeRabbitReview app60.672.7%
- 2GLM 5.3Model59.347.1%
- 3GPT 5.6 Luna ProModel53.835.3%
- 4DeepSeek V4 ProModel53.444.4%
- 5Greptile v4.1Review app50.560.0%
- 6DeepSeek V4.1 FlashAI Architecture runs this47.847.1%
- 7GLM 5.3 FlashModel38.315.4%
- 8Cursor BugbotReview app37.450.0%
- 9Grok 4.6Model34.034.8%
- 10Muse Spark 1.3Model33.628.6%
- Provisional scoped-label matching.
- Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
- The lane’s labels were scoped from the code review labels and not yet checked by a second rater.
- Review apps were not run as specialists: their general reviews were rescored for this lane.
| Model / App | Reviewer Score | F1 | Latency | $/run |
|---|---|---|---|---|
| CodeRabbit | 60.6 | 72.7% | 7m 43s | ~$1.50 |
| IonWarp · GLM 5.3 | 59.3 | 47.1% | 1m 30s | $0.075 |
| IonWarp · GPT 5.6 Luna Pro | 53.8 | 35.3% | 51s | $0.046 |
| IonWarp · DeepSeek V4 Pro | 53.4 | 44.4% | 1m 52s | $0.102 |
| Greptile v4.1 | 50.5 | 60.0% | 5m 06s | ~$1.75 |
| IonWarp · DeepSeek V4.1 FlashRuns on your PRs | 47.8 | 47.1% | 6m 14s | $0.050 |
| IonWarp · GLM 5.3 Flash | 38.3 | 15.4% | 1m 02s | $0.0064 |
| Cursor Bugbot | 37.4 | 50.0% | 7m 46s | ~$1.25 |
| IonWarp · Grok 4.6 | 34.0 | 34.8% | 3m 29s | $0.284 |
| IonWarp · Muse Spark 1.3 | 33.6 | 28.6% | 2m 04s | $0.193 |
| IonWarp · Gemini 3.8 Flash | 32.3 | 33.3% | 3m 50s | $0.238 |
| Qodo v2 | 12.8 | 25.0% | 5m 52s | ~$1.67 |
Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.
AI Architecture
IonWarp inferenceApp price estimatesF1 on the same 11 open-source pull requests for every reviewer
| Model / app | Effort | Precision | Recall | Caught | Unmatched | Missed | PRs | Time p90 | Measured |
|---|---|---|---|---|---|---|---|---|---|
| CodeRabbit | — | 80.0% | 66.7% | 4 | 1 | 2 | 11 / 11 | 12m 10s | 2026-01-23 – 2026-04-05 |
| IonWarp · GLM 5.3 | medium | 36.4% | 66.7% | 4 | 7 | 2 | 11 / 11 | 2m 06s | 2026-09-17 – 2026-09-22 |
| IonWarp · GPT 5.6 Luna Pro | medium | 27.3% | 50.0% | 3 | 8 | 3 | 11 / 11 | 1m 37s | 2026-09-10 – 2026-09-17 |
| IonWarp · DeepSeek V4 Pro | medium | 33.3% | 66.7% | 4 | 8 | 2 | 11 / 11 | 3m 08s | 2026-09-08 – 2026-09-17 |
| Greptile v4.1 | — | 75.0% | 50.0% | 3 | 1 | 3 | 11 / 11 | 6m 42s | 2026-04-07 – 2026-04-07 |
| IonWarp · DeepSeek V4.1 Flash | medium | 36.4% | 66.7% | 4 | 7 | 2 | 11 / 11 | 10m 39s | 2026-09-24 – 2026-09-24 |
| IonWarp · GLM 5.3 Flash | low | 14.3% | 16.7% | 1 | 6 | 5 | 11 / 11 | 3m 44s | 2026-09-10 – 2026-09-17 |
| Cursor Bugbot | — | 40.0% | 66.7% | 4 | 6 | 2 | 11 / 11 | 12m 09s | 2026-01-28 – 2026-01-28 |
| IonWarp · Grok 4.6 | medium | 23.5% | 66.7% | 4 | 13 | 2 | 11 / 11 | 4m 12s | 2026-09-08 – 2026-09-17 |
| IonWarp · Muse Spark 1.3 | medium | 20.0% | 50.0% | 3 | 12 | 3 | 11 / 11 | 2m 58s | 2026-09-08 – 2026-09-17 |
| IonWarp · Gemini 3.8 Flash | medium | 25.0% | 50.0% | 3 | 9 | 3 | 11 / 11 | 5m 14s | 2026-09-08 – 2026-09-22 |
| Qodo v2 | — | 50.0% | 16.7% | 1 | 1 | 5 | 11 / 11 | 7m 29s | 2026-02-27 – 2026-02-27 |
| Pull request | Issues | CodeRabbit | GLM 5.3 | GPT 5.6 Luna Pro | DeepSeek V4 Pro | Greptile v4.1 | DeepSeek V4.1 Flash | GLM 5.3 Flash | Cursor Bugbot | Grok 4.6 | Muse Spark 1.3 | Gemini 3.8 Flash | Qodo v2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| cal.com#10967 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| cal.com#22345 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 |
| cal.com#22532 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| discourse-graphite#1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
| discourse-graphite#10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| grafana#97529 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| keycloak#33832 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 |
| keycloak#36880 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 |
| keycloak#38446 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| sentry#80528 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| sentry#95633 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 12 reviewers; a wider screen shows them all.
Other lanes
The rest of the benchmark

Code Review
Qodo v2 leads, F1 58.5%. Code Review on DeepSeek V4.1 Flash: 49.2%.

Security Review
Qodo v2 leads, F1 73.3%. Security Review on DeepSeek V4.1 Flash: 62.5%.

Log & Errors
Qodo v2 leads, F1 72.7%. Log & Errors on DeepSeek V4.1 Flash: 59.0%.

Docs Review
IonWarp · Gemini 3.8 Flash leads, F1 68.6%. DeepSeek V4.1 Flash not measured yet.

API Performance
IonWarp · Muse Spark 1.3 leads, F1 56.0%. API Performance on DeepSeek V4.1 Flash: 53.3%.

SOC 2 Review
IonWarp · GLM 5.3 Flash leads, F1 66.7%. SOC 2 Review on DeepSeek V4.1 Flash: 66.7%.

Put AI Architecture on your next PR.
Try for free14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.