Provisional · release 2026-10-12
CI/CD Performance benchmark
18 PRs, 13 known issues. DeepSeek V4 Pro leads at F1 72.7%. CI/CD Performance on DeepSeek V4.1 Flash: F1 59.5% at $0.0095 a review.
9:41

Benchmark · Provisional
CI/CD Performance
18 real pull requests, 13 known issues
ReviewerScoreF1
- 1DeepSeek V4 ProModel60.572.7%
- 2GLM 5.3 FlashModel53.555.0%
- 3DeepSeek V4.1 FlashCI/CD Performance runs this51.259.5%
- 4GPT 5.6 Luna ProModel48.554.5%
- 5Muse Spark 1.3Model11.644.4%
- Provisional scoped-label matching.
- Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
- Some pull requests and labels were added by us, not taken from a public corpus.
- Not run on this lane: CodeRabbit, Cursor Bugbot, Greptile v4.1 and Qodo v2.
| Model / App | Reviewer Score | F1 | Latency | $/run |
|---|---|---|---|---|
| IonWarp · DeepSeek V4 Pro | 60.5 | 72.7% | 1m 22s | $0.032 |
| IonWarp · GLM 5.3 Flash | 53.5 | 55.0% | 50s | $0.0027 |
| IonWarp · DeepSeek V4.1 FlashRuns on your PRs | 51.2 | 59.5% | 54s | $0.0095 |
| IonWarp · GPT 5.6 Luna Pro | 48.5 | 54.5% | 33s | $0.014 |
| IonWarp · Muse Spark 1.3 | 11.6 | 44.4% | 49s | $0.034 |
Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.
CI/CD Performance
IonWarp inferenceApp price estimatesF1 on the same 18 open-source pull requests for every reviewer
| Model / app | Effort | Precision | Recall | Caught | Unmatched | Missed | PRs | Time p90 | Measured |
|---|---|---|---|---|---|---|---|---|---|
| IonWarp · DeepSeek V4 Pro | medium | 60.0% | 92.3% | 12 | 8 | 1 | 18 / 18 | 6m 44s | 2026-09-21 – 2026-09-21 |
| IonWarp · GLM 5.3 Flash | low | 40.7% | 84.6% | 11 | 16 | 2 | 18 / 18 | 3m 35s | 2026-09-21 – 2026-09-21 |
| IonWarp · DeepSeek V4.1 Flash | medium | 45.8% | 84.6% | 11 | 13 | 2 | 18 / 18 | 11m 01s | 2026-09-21 – 2026-09-21 |
| IonWarp · GPT 5.6 Luna Pro | medium | 45.0% | 69.2% | 9 | 11 | 4 | 18 / 18 | 53s | 2026-09-21 – 2026-09-21 |
| IonWarp · Muse Spark 1.3 | medium | 42.9% | 46.2% | 6 | 8 | 7 | 18 / 18 | 1m 42s | 2026-09-21 – 2026-09-21 |
| Pull request | Issues | DeepSeek V4 Pro | GLM 5.3 Flash | DeepSeek V4.1 Flash | GPT 5.6 Luna Pro | Muse Spark 1.3 |
|---|---|---|---|---|---|---|
| aspire.dev#964 | 0 | 0 | 0 | 0 | 0 | 0 |
| django#21457 | 1 | 1 | 1 | 1 | 1 | 1 |
| esphome#9396 | 1 | 1 | 1 | 1 | 1 | 1 |
| gcp-ingestion#2901 | 1 | 1 | 1 | 1 | 1 | 0 |
| herdr#4023 | 1 | 1 | 0 | 1 | 0 | 0 |
| hyperswitch#14081 | 1 | 1 | 1 | 1 | 0 | 0 |
| iOS#5002 | 1 | 0 | 0 | 0 | 0 | 0 |
| kiali#10167 | 1 | 1 | 1 | 1 | 1 | 1 |
| KMMBridge#266 | 1 | 1 | 1 | 1 | 0 | 1 |
| marimo#9312 | 0 | 0 | 0 | 0 | 0 | 0 |
| mountpoint-s3-csi-driver#461 | 1 | 1 | 1 | 1 | 1 | 0 |
| opentelemetry-collector-contrib#37106 | 1 | 1 | 1 | 1 | 1 | 1 |
| outline#8534 | 1 | 1 | 1 | 1 | 1 | 1 |
| Plonky3#1969 | 1 | 1 | 1 | 1 | 1 | 0 |
| react-native-reanimated#8929 | 1 | 1 | 1 | 0 | 1 | 0 |
| ruff#28610 | 0 | 0 | 0 | 0 | 0 | 0 |
| supervisor#6807 | 0 | 0 | 0 | 0 | 0 | 0 |
| uiv#903 | 0 | 0 | 0 | 0 | 0 | 0 |
A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 5 reviewers; a wider screen shows them all.
Other lanes
The rest of the benchmark

Code Review
Qodo v2 leads, F1 58.5%. Code Review on DeepSeek V4.1 Flash: 49.2%.

Security Review
Qodo v2 leads, F1 73.3%. Security Review on DeepSeek V4.1 Flash: 62.5%.

AI Architecture
CodeRabbit leads, F1 72.7%. AI Architecture on DeepSeek V4.1 Flash: 47.1%.

Log & Errors
Qodo v2 leads, F1 72.7%. Log & Errors on DeepSeek V4.1 Flash: 59.0%.

Docs Review
IonWarp · Gemini 3.8 Flash leads, F1 68.6%. DeepSeek V4.1 Flash not measured yet.

API Performance
IonWarp · Muse Spark 1.3 leads, F1 56.0%. API Performance on DeepSeek V4.1 Flash: 53.3%.
Read next
Read next

Put CI/CD Performance on your next PR.
Try for free14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.