Provisional · release 2026-10-12
SOC 2 Review benchmark
20 PRs, 15 known issues. GLM 5.3 Flash leads at F1 66.7%. SOC 2 Review on DeepSeek V4.1 Flash: F1 66.7% at $0.038 a review.
9:41

Benchmark · Provisional
SOC 2 Review
20 real pull requests, 15 known issues
ReviewerScoreF1
- 1GLM 5.3 FlashModel100.066.7%
- 2GPT 5.6 Luna ProModel71.564.5%
- 3DeepSeek V4.1 FlashSOC 2 Review runs this68.666.7%
- 4DeepSeek V4 ProModel17.356.3%
- 5Muse Spark 1.3Model17.055.2%
- Provisional scoped-label matching.
- Judge: IonWarp adapted category-scoped judge, Claude Opus 4.5.
- Some pull requests and labels were added by us, not taken from a public corpus.
- Not run on this lane: CodeRabbit, Cursor Bugbot, Greptile v4.1 and Qodo v2.
| Model / App | Reviewer Score | F1 | Latency | $/run |
|---|---|---|---|---|
| IonWarp · GLM 5.3 Flash | 100.0 | 66.7% | 34s | $0.0095 |
| IonWarp · GPT 5.6 Luna Pro | 71.5 | 64.5% | 37s | $0.072 |
| IonWarp · DeepSeek V4.1 FlashRuns on your PRs | 68.6 | 66.7% | 5m 56s | $0.038 |
| IonWarp · DeepSeek V4 Pro | 17.3 | 56.3% | 2m 21s | $0.068 |
| IonWarp · Muse Spark 1.3 | 17.0 | 55.2% | 49s | $0.109 |
Reviewer Score = (3 × F1 + time + $/run) ÷ 5, each part 0 to 100 within this table. IonWarp rows cost the model’s inference; ~$ is each app’s list price for 5 seats at 100 reviews a month. Provisional scoped-label matching. Release 2026-10-12, CC BY 4.0.
SOC 2 Review
IonWarp inferenceApp price estimatesF1 on the same 20 open-source pull requests for every reviewer
| Model / app | Effort | Precision | Recall | Caught | Unmatched | Missed | PRs | Time p90 | Measured |
|---|---|---|---|---|---|---|---|---|---|
| IonWarp · GLM 5.3 Flash | low | 61.1% | 73.3% | 11 | 7 | 4 | 20 / 20 | 1m 22s | 2026-09-21 – 2026-09-22 |
| IonWarp · GPT 5.6 Luna Pro | medium | 62.5% | 66.7% | 10 | 6 | 5 | 20 / 20 | 43s | 2026-09-21 – 2026-09-22 |
| IonWarp · DeepSeek V4.1 Flash | medium | 54.2% | 86.7% | 13 | 11 | 2 | 20 / 20 | 11m 57s | 2026-09-24 – 2026-09-24 |
| IonWarp · DeepSeek V4 Pro | medium | 52.9% | 60.0% | 9 | 8 | 6 | 20 / 20 | 4m 52s | 2026-09-21 – 2026-09-22 |
| IonWarp · Muse Spark 1.3 | medium | 57.1% | 53.3% | 8 | 6 | 7 | 20 / 20 | 1m 04s | 2026-09-21 – 2026-09-22 |
| Pull request | Issues | GLM 5.3 Flash | GPT 5.6 Luna Pro | DeepSeek V4.1 Flash | DeepSeek V4 Pro | Muse Spark 1.3 |
|---|---|---|---|---|---|---|
| airflow#70759 | 0 | 0 | 0 | 0 | 0 | 0 |
| atlantis#2436 | 1 | 0 | 1 | 1 | 0 | 0 |
| auth#2073 | 1 | 1 | 1 | 1 | 1 | 0 |
| coder#27668 | 1 | 1 | 1 | 1 | 1 | 1 |
| Ghost#23585 | 1 | 1 | 1 | 1 | 1 | 1 |
| gitea#10273 | 1 | 1 | 1 | 1 | 1 | 0 |
| gitea#11172 | 1 | 1 | 1 | 1 | 1 | 1 |
| grafana#52081 | 1 | 1 | 1 | 1 | 1 | 1 |
| grafana#130148 | 1 | 1 | 0 | 1 | 0 | 0 |
| harbor#23461 | 0 | 0 | 0 | 0 | 0 | 0 |
| keycloak#29278 | 1 | 0 | 0 | 0 | 0 | 0 |
| langfuse#15456 | 1 | 0 | 0 | 1 | 0 | 0 |
| mattermost#33642 | 1 | 1 | 1 | 1 | 1 | 1 |
| minio#20881 | 0 | 0 | 0 | 0 | 0 | 0 |
| n8n#7622 | 1 | 1 | 0 | 1 | 0 | 1 |
| nomad#25328 | 0 | 0 | 0 | 0 | 0 | 0 |
| sentry#87523 | 1 | 1 | 1 | 1 | 1 | 1 |
| sentry#113722 | 1 | 1 | 1 | 1 | 1 | 1 |
| traefik#11907 | 0 | 0 | 0 | 0 | 0 | 0 |
| woodpecker#3348 | 1 | 0 | 0 | 0 | 0 | 0 |
A cell is how many of the PR’s known issues that reviewer caught; links open its review. – means the reviewer was not run on that PR. On a phone this shows the first 3 of 5 reviewers; a wider screen shows them all.
Other lanes
The rest of the benchmark

Code Review
Qodo v2 leads, F1 58.5%. Code Review on DeepSeek V4.1 Flash: 49.2%.

Security Review
Qodo v2 leads, F1 73.3%. Security Review on DeepSeek V4.1 Flash: 62.5%.

AI Architecture
CodeRabbit leads, F1 72.7%. AI Architecture on DeepSeek V4.1 Flash: 47.1%.

Log & Errors
Qodo v2 leads, F1 72.7%. Log & Errors on DeepSeek V4.1 Flash: 59.0%.

Docs Review
IonWarp · Gemini 3.8 Flash leads, F1 68.6%. DeepSeek V4.1 Flash not measured yet.

API Performance
IonWarp · Muse Spark 1.3 leads, F1 56.0%. API Performance on DeepSeek V4.1 Flash: 53.3%.
Read next
Read next

Put SOC 2 Review on your next PR.
Try for free14 days free, no card. Then Pro is $49 a month per workspace with 5 seats, or Max $149 with 10.