Try for free

Historical · release 2026-10-12

Cost Review benchmark

20 PRs, 14 known issues. DeepSeek V4.1 Flash leads at F1 57.1%.

About this lane
  • Provisional scoped-label matching.
  • Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
  • Some pull requests and labels were added by us, not taken from a public corpus.
  • Cost Review was a legacy agent; it is not one of the eight. Kept here as history.
  • Not run on this lane: CodeRabbit, Cursor Bugbot, Greptile v4.1 and Qodo v2.
Cost Review: how each reviewer scoresHistorical
Cost Review: how each reviewer scores
Model / AppReviewer ScoreF1Latency$/run
IonWarp · GLM 5.3 Flash72.955.2%2m 23s$0.014
IonWarp · DeepSeek V4.1 Flash60.057.1%10m 39s$0.073
IonWarp · GPT-6 Luna Pro36.250.0%27s$0.019

Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.

Cost Review

IonWarp inferenceApp price estimatesF1 on the same 20 open-source pull requests for every reviewer

50%60%$0.01$0.03$0.10Cost per review · USD · log scaleDeepSeek V4.1 FlashGLM 5.3 FlashGPT-6 Luna Pro
Precision, recall and the counts behind F1
Cost Review: precision, recall and the counts behind F1
Model / appEffortPrecisionRecallCaughtUnmatchedMissedPRsTime p90Measured
IonWarp · GLM 5.3 Flashlow53.3%57.1%87620 / 206m 23s2026-09-22 – 2026-09-22
IonWarp · DeepSeek V4.1 Flashmedium47.6%71.4%1011420 / 2016m 17s2026-09-24 – 2026-09-24
IonWarp · GPT-6 Luna Promedium60.0%42.9%64820 / 205m 29s2026-09-22 – 2026-09-22
Every pull request: known issues caught by each reviewer20 PRs
Known issues caught by each reviewer, per pull request
Pull requestIssuesGLM 5.3 FlashDeepSeek V4.1 FlashGPT-6 Luna Pro
ai#179230000
cline#77951111
cline#95761111
dify#41400000
gateway#13730000
langchainjs#69221111
LibreChat#130141010
LibreChat#136830000
llama_index#202061110
mastra#124651010
onyx#104781100
open-webui#151220000
posthog#604031000
posthog#718601000
posthog#1042660000
posthog-python#3461100
pydantic-ai#27711111
Roo-Code#68641111
sentry-python#46121011
sentry-python#53191010

A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR.

Put the fleet on your next PR.

Try for free

14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.