Try for free

Provisional · release 2026-10-12

CI/CD Performance benchmark

18 PRs, 13 known issues. DeepSeek V4 Pro leads at F1 72.7%. CI/CD Performance on DeepSeek V4.1 Flash: F1 59.5% at $0.0095 a review.

About this lane
  • Provisional scoped-label matching.
  • Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
  • Some pull requests and labels were added by us, not taken from a public corpus.
  • Not run on this lane: CodeRabbit, Cursor Bugbot, Greptile v4.1 and Qodo v2.
CI/CD Performance: how each reviewer scoresProvisional
CI/CD Performance: how each reviewer scores
Model / AppReviewer ScoreF1Latency$/run
IonWarp · DeepSeek V4 Pro60.572.7%1m 22s$0.032
IonWarp · GLM 5.3 Flash53.555.0%50s$0.0027
IonWarp · DeepSeek V4.1 FlashRuns on your PRs51.259.5%54s$0.0095
IonWarp · GPT 5.6 Luna Pro48.554.5%33s$0.014
IonWarp · Muse Spark 1.311.644.4%49s$0.034

Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.

CI/CD Performance

IonWarp inferenceApp price estimatesF1 on the same 18 open-source pull requests for every reviewer

40%50%60%70%80%$0.01$0.03Cost per review · USD · log scaleDeepSeek V4 ProDeepSeek V4.1 FlashGLM 5.3 FlashGPT 5.6 Luna ProMuse Spark 1.3
Precision, recall and the counts behind F1
CI/CD Performance: precision, recall and the counts behind F1
Model / appEffortPrecisionRecallCaughtUnmatchedMissedPRsTime p90Measured
IonWarp · DeepSeek V4 Promedium60.0%92.3%128118 / 186m 44s2026-09-21 – 2026-09-21
IonWarp · GLM 5.3 Flashlow40.7%84.6%1116218 / 183m 35s2026-09-21 – 2026-09-21
IonWarp · DeepSeek V4.1 Flashmedium45.8%84.6%1113218 / 1811m 01s2026-09-21 – 2026-09-21
IonWarp · GPT 5.6 Luna Promedium45.0%69.2%911418 / 1853s2026-09-21 – 2026-09-21
IonWarp · Muse Spark 1.3medium42.9%46.2%68718 / 181m 42s2026-09-21 – 2026-09-21
Every pull request: known issues caught by each reviewer18 PRs
Known issues caught by each reviewer, per pull request
Pull requestIssuesDeepSeek V4 ProGLM 5.3 FlashDeepSeek V4.1 FlashGPT 5.6 Luna ProMuse Spark 1.3
aspire.dev#964000000
django#21457111111
esphome#9396111111
gcp-ingestion#2901111110
herdr#4023110100
hyperswitch#14081111100
iOS#5002100000
kiali#10167111111
KMMBridge#266111101
marimo#9312000000
mountpoint-s3-csi-driver#461111110
opentelemetry-collector-contrib#37106111111
outline#8534111111
Plonky3#1969111110
react-native-reanimated#8929111010
ruff#28610000000
supervisor#6807000000
uiv#903000000

A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 5 reviewers; a wider screen shows them all.

Put CI/CD Performance on your next PR.

Try for free

14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.