Try for free

Provisional · release 2026-10-12

Log & Errors benchmark

17 PRs, 31 known issues. Qodo v2 leads at F1 72.7%. Log & Errors on DeepSeek V4.1 Flash: F1 59.0% at $0.039 a review.

About this lane
  • Provisional scoped-label matching.
  • Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
  • The lane’s labels were scoped from the code review labels and not yet checked by a second rater.
  • Review apps were not run as specialists: their general reviews were rescored for this lane.
Log & Errors: how each reviewer scoresProvisional
Log & Errors: how each reviewer scores
Model / AppReviewer ScoreF1Latency$/run
IonWarp · Gemini 3.8 Flash76.267.7%1m 29s$0.100
Qodo v263.472.7%5m 52s~$1.67
IonWarp · DeepSeek V4.1 FlashRuns on your PRs57.059.0%4m 08s$0.039
Greptile v4.156.467.8%5m 02s~$1.75
Cursor Bugbot50.865.6%7m 34s~$1.25
IonWarp · Muse Spark 1.341.243.6%56s$0.085
CodeRabbit38.459.2%8m 47s~$1.50
IonWarp · GLM 5.3 Flash35.936.1%1m 14s$0.0058
IonWarp · GPT 5.6 Luna Pro32.736.0%44s$0.046

Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.

Log & Errors

IonWarp inferenceApp price estimatesF1 on the same 17 open-source pull requests for every reviewer

30%40%50%60%70%80%$0.01$0.03$0.10$0.30$1.00Cost per review · USD · log scaleQodo v2Greptile v4.1Gemini 3.8 FlashCursor BugbotCodeRabbitDeepSeek V4.1 FlashMuse Spark 1.3GLM 5.3 FlashGPT 5.6 Luna Pro
Precision, recall and the counts behind F1
Log & Errors: precision, recall and the counts behind F1
Model / appEffortPrecisionRecallCaughtUnmatchedMissedPRsTime p90Measured
IonWarp · Gemini 3.8 Flashmedium67.7%67.7%21101017 / 172m 44s2026-09-14 – 2026-09-17
Qodo v2—68.6%77.4%2411717 / 177m 29s2026-02-27 – 2026-02-27
IonWarp · DeepSeek V4.1 Flashmedium60.0%58.1%18121317 / 179m 02s2026-09-25 – 2026-09-25
Greptile v4.1—71.4%64.5%2081117 / 176m 49s2026-04-07 – 2026-04-07
Cursor Bugbot—63.6%67.7%21121017 / 1712m 09s2026-01-28 – 2026-01-28
IonWarp · Muse Spark 1.3medium50.0%38.7%12121917 / 171m 37s2026-09-14 – 2026-09-17
CodeRabbit—52.5%67.7%21191017 / 1712m 10s2026-01-23 – 2026-04-05
IonWarp · GLM 5.3 Flashlow36.7%35.5%11192017 / 174m 02s2026-09-13 – 2026-09-17
IonWarp · GPT 5.6 Luna Promedium47.4%29.0%9102217 / 1759s2026-09-14 – 2026-09-17
Every pull request: known issues caught by each reviewer17 PRs
Known issues caught by each reviewer, per pull request
Pull requestIssuesGemini 3.8 FlashQodo v2DeepSeek V4.1 FlashGreptile v4.1Cursor BugbotMuse Spark 1.3CodeRabbitGLM 5.3 FlashGPT 5.6 Luna Pro
cal.com#80871111111111
cal.com#109672121111210
cal.com#110595242442411
cal.com#223450000000000
cal.com#225321111100010
discourse-graphite#43231221311
discourse-graphite#103322130300
grafana#761863221111111
grafana#900453323333312
grafana#975291010100001
keycloak#338320000000000
sentry#678762111111100
sentry#805280000000000
sentry#938241111001011
sentry#956330000000000
sentry-greptile#14322331321
sentry-greptile#22122120010

A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 9 reviewers; a wider screen shows them all.

Put Log & Errors on your next PR.

Try for free

14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.