Try for free

Provisional · release 2026-10-12

SOC 2 Review benchmark

20 PRs, 15 known issues. GLM 5.3 Flash leads at F1 66.7%. SOC 2 Review on DeepSeek V4.1 Flash: F1 66.7% at $0.038 a review.

About this lane
  • Provisional scoped-label matching.
  • Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
  • Some pull requests and labels were added by us, not taken from a public corpus.
  • Not run on this lane: CodeRabbit, Cursor Bugbot, Greptile v4.1 and Qodo v2.
SOC 2 Review: how each reviewer scoresProvisional
SOC 2 Review: how each reviewer scores
Model / AppReviewer ScoreF1Latency$/run
IonWarp · GLM 5.3 Flash100.066.7%34s$0.0095
IonWarp · GPT 5.6 Luna Pro71.564.5%37s$0.072
IonWarp · DeepSeek V4.1 FlashRuns on your PRs68.666.7%5m 56s$0.038
IonWarp · DeepSeek V4 Pro17.356.3%2m 21s$0.068
IonWarp · Muse Spark 1.317.055.2%49s$0.109

Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.

SOC 2 Review

IonWarp inferenceApp price estimatesF1 on the same 20 open-source pull requests for every reviewer

50%60%70%$0.01$0.03$0.10Cost per review · USD · log scaleGLM 5.3 FlashDeepSeek V4.1 FlashGPT 5.6 Luna ProDeepSeek V4 ProMuse Spark 1.3
Precision, recall and the counts behind F1
SOC 2 Review: precision, recall and the counts behind F1
Model / appEffortPrecisionRecallCaughtUnmatchedMissedPRsTime p90Measured
IonWarp · GLM 5.3 Flashlow61.1%73.3%117420 / 201m 22s2026-09-21 – 2026-09-22
IonWarp · GPT 5.6 Luna Promedium62.5%66.7%106520 / 2043s2026-09-21 – 2026-09-22
IonWarp · DeepSeek V4.1 Flashmedium54.2%86.7%1311220 / 2011m 57s2026-09-24 – 2026-09-24
IonWarp · DeepSeek V4 Promedium52.9%60.0%98620 / 204m 52s2026-09-21 – 2026-09-22
IonWarp · Muse Spark 1.3medium57.1%53.3%86720 / 201m 04s2026-09-21 – 2026-09-22
Every pull request: known issues caught by each reviewer20 PRs
Known issues caught by each reviewer, per pull request
Pull requestIssuesGLM 5.3 FlashGPT 5.6 Luna ProDeepSeek V4.1 FlashDeepSeek V4 ProMuse Spark 1.3
airflow#70759000000
atlantis#2436101100
auth#2073111110
coder#27668111111
Ghost#23585111111
gitea#10273111110
gitea#11172111111
grafana#52081111111
grafana#130148110100
harbor#23461000000
keycloak#29278100000
langfuse#15456100100
mattermost#33642111111
minio#20881000000
n8n#7622110101
nomad#25328000000
sentry#87523111111
sentry#113722111111
traefik#11907000000
woodpecker#3348100000

A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 5 reviewers; a wider screen shows them all.

Put SOC 2 Review on your next PR.

Try for free

14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.