Historical · release 2026-10-12
Cost Review benchmark
20 PRs, 14 known issues. DeepSeek V4.1 Flash leads at F1 57.1%.
9:41

Benchmark · Historical
Cost Review
20 real pull requests, 14 known issues
ReviewerScoreF1
- 1GLM 5.3 FlashModel72.955.2%
- 2DeepSeek V4.1 FlashModel60.057.1%
- 3GPT-6 Luna ProModel36.250.0%
- Provisional scoped-label matching.
- Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
- Some pull requests and labels were added by us, not taken from a public corpus.
- Cost Review was a legacy agent; it is not one of the eight. Kept here as history.
- Not run on this lane: CodeRabbit, Cursor Bugbot, Greptile v4.1 and Qodo v2.
| Model / App | Reviewer Score | F1 | Latency | $/run |
|---|---|---|---|---|
| IonWarp · GLM 5.3 Flash | 72.9 | 55.2% | 2m 23s | $0.014 |
| IonWarp · DeepSeek V4.1 Flash | 60.0 | 57.1% | 10m 39s | $0.073 |
| IonWarp · GPT-6 Luna Pro | 36.2 | 50.0% | 27s | $0.019 |
Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.
Cost Review
IonWarp inferenceApp price estimatesF1 on the same 20 open-source pull requests for every reviewer
| Model / app | Effort | Precision | Recall | Caught | Unmatched | Missed | PRs | Time p90 | Measured |
|---|---|---|---|---|---|---|---|---|---|
| IonWarp · GLM 5.3 Flash | low | 53.3% | 57.1% | 8 | 7 | 6 | 20 / 20 | 6m 23s | 2026-09-22 – 2026-09-22 |
| IonWarp · DeepSeek V4.1 Flash | medium | 47.6% | 71.4% | 10 | 11 | 4 | 20 / 20 | 16m 17s | 2026-09-24 – 2026-09-24 |
| IonWarp · GPT-6 Luna Pro | medium | 60.0% | 42.9% | 6 | 4 | 8 | 20 / 20 | 5m 29s | 2026-09-22 – 2026-09-22 |
| Pull request | Issues | GLM 5.3 Flash | DeepSeek V4.1 Flash | GPT-6 Luna Pro |
|---|---|---|---|---|
| ai#17923 | 0 | 0 | 0 | 0 |
| cline#7795 | 1 | 1 | 1 | 1 |
| cline#9576 | 1 | 1 | 1 | 1 |
| dify#4140 | 0 | 0 | 0 | 0 |
| gateway#1373 | 0 | 0 | 0 | 0 |
| langchainjs#6922 | 1 | 1 | 1 | 1 |
| LibreChat#13014 | 1 | 0 | 1 | 0 |
| LibreChat#13683 | 0 | 0 | 0 | 0 |
| llama_index#20206 | 1 | 1 | 1 | 0 |
| mastra#12465 | 1 | 0 | 1 | 0 |
| onyx#10478 | 1 | 1 | 0 | 0 |
| open-webui#15122 | 0 | 0 | 0 | 0 |
| posthog#60403 | 1 | 0 | 0 | 0 |
| posthog#71860 | 1 | 0 | 0 | 0 |
| posthog#104266 | 0 | 0 | 0 | 0 |
| posthog-python#346 | 1 | 1 | 0 | 0 |
| pydantic-ai#2771 | 1 | 1 | 1 | 1 |
| Roo-Code#6864 | 1 | 1 | 1 | 1 |
| sentry-python#4612 | 1 | 0 | 1 | 1 |
| sentry-python#5319 | 1 | 0 | 1 | 0 |
A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR.
Other lanes
The rest of the benchmark

Code Review
Qodo v2 leads, F1 58.5%. Code Review on DeepSeek V4.1 Flash: 49.2%.

Security Review
Qodo v2 leads, F1 73.3%. Security Review on DeepSeek V4.1 Flash: 62.5%.

AI Architecture
CodeRabbit leads, F1 72.7%. AI Architecture on DeepSeek V4.1 Flash: 47.1%.

Log & Errors
Qodo v2 leads, F1 72.7%. Log & Errors on DeepSeek V4.1 Flash: 59.0%.

Docs Review
IonWarp · Gemini 3.8 Flash leads, F1 68.6%. DeepSeek V4.1 Flash not measured yet.

API Performance
IonWarp · Muse Spark 1.3 leads, F1 56.0%. API Performance on DeepSeek V4.1 Flash: 53.3%.
Put the fleet on your next PR.
Try for free14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.