Try for free

Provisional · release 2026-10-12

Docs Review benchmark

19 PRs, 17 known issues. Gemini 3.8 Flash leads at F1 68.6%. DeepSeek V4.1 Flash, which Docs Review runs, is not measured here yet.

About this lane
  • Provisional scoped-label matching.
  • Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
  • Some pull requests and labels were added by us, not taken from a public corpus.
  • Review apps were not run as specialists: their general reviews were rescored for this lane.
Docs Review: how each reviewer scoresProvisional
Docs Review: how each reviewer scores
Model / AppReviewer ScoreF1Latency$/run
IonWarp · Gemini 3.8 Flash79.268.6%59s$0.084
IonWarp · DeepSeek V4 Pro57.965.2%3m 00s$0.064
IonWarp · Muse Spark 1.320.059.5%48s$0.087
CodeRabbit11 of 19 PRs—55.6%7m 43s~$1.50
Cursor Bugbot11 of 19 PRs—36.4%9m 53s~$1.25
Greptile v4.111 of 19 PRs—28.6%5m 23s~$1.75
Qodo v211 of 19 PRs—26.7%6m 29s~$1.67

Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Rows measured on part of the PRs get no Score. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.

Docs Review

IonWarp inferenceApp price estimatesF1 on the same 19 open-source pull requests for every reviewer

50%60%70%$0.10Cost per review · USD · log scaleGemini 3.8 FlashDeepSeek V4 ProMuse Spark 1.3
Precision, recall and the counts behind F1
Docs Review: precision, recall and the counts behind F1
Model / appEffortPrecisionRecallCaughtUnmatchedMissedPRsTime p90Measured
IonWarp · Gemini 3.8 Flashmedium66.7%70.6%126519 / 191m 33s2026-09-21 – 2026-09-22
IonWarp · DeepSeek V4 Promedium51.7%88.2%1514219 / 197m 44s2026-09-21 – 2026-09-23
IonWarp · Muse Spark 1.3medium55.0%64.7%119619 / 191m 50s2026-09-21 – 2026-09-22
CodeRabbit—55.6%55.6%54411 / 1910m 10s2026-01-23 – 2026-04-05
Cursor Bugbot—100.0%22.2%20711 / 1912m 09s2026-01-28 – 2026-01-28
Greptile v4.1—40.0%22.2%23711 / 195m 51s2026-04-07 – 2026-04-07
Qodo v2—33.3%22.2%24711 / 199m 33s2026-02-27 – 2026-02-27
Every pull request: known issues caught by each reviewer19 PRs
Known issues caught by each reviewer, per pull request
Pull requestIssuesGemini 3.8 FlashDeepSeek V4 ProMuse Spark 1.3CodeRabbitCursor BugbotGreptile v4.1Qodo v2
cal.com#1060011100000
cal.com#2253200000000
discourse-graphite#1000000000
grafana#7926510111011
grafana#9752900000000
grafana#10363311010001
keycloak#3688210000000
keycloak#3742921222200
keycloak#3763411111000
pandas#439511111––––
pandas#454111110––––
pandas#476331111––––
pandas#573471010––––
pandas#610321111––––
pandas#612641011––––
pandas#614671110––––
pandas#633841111––––
sentry#8016811111010
sentry#9563311100000

A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 7 reviewers; a wider screen shows them all.

Put Docs Review on your next PR.

Try for free

14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.