IonWarp reviews every pull request with a swarm of reviewer lenses — code
review, security, test holes, performance — and each lens is pinned to a
specific model. Those pins are supposed to evolve on evidence: when a better
model appears, we trial it, grade it against the incumbent, and repin if it
wins.
This is the story of a repin that almost happened for the wrong reason, and the
measurement that stopped it.
The setup
A small probe — roughly a 500-token payload — suggested that
poolside/laguna-s-2.1 was substantive: thoughtful output, good structure,
plausible findings. On the strength of that probe, a consolidation was drafted
that would have moved ten reviewer lenses onto laguna, keeping
x-ai/grok-4.5 only on the deep code-review-full lens.
Grok, meanwhile, carried baggage: an all-time timeout rate of 28%. On
reputation alone, laguna looked like the safer bet.
The probe document itself carried a caveat — a 500-token probe is nothing like
a production payload, which for our reviews runs 24,000–37,000 tokens. That
caveat turned out to be the whole story.
What 30 hours of production traffic showed
Before trusting the repin, we measured a 30-hour window of real reviews:
102 reviews, 400 reviewer executions, across three projects (ionwarp,
zest, and north). Four of those PRs also got a code-verified audit — a human
confirming, finding by finding, whether each reported issue was real.
| Model | Executions | Failure rate | Median latency | p90 latency | Median cost | Confirmed findings / run |
|---|---|---|---|---|---|---|
| x-ai/grok-4.5 | 108 | 1.9% | 161s | 390s | $0.051 | 2.27 |
| poolside/laguna-s-2.1 | 46 | 85% | — | — | — | 0.14 |
(Laguna's latency and cost columns are blank because most of its executions
produced nothing to time or price.)
The detail behind those two rows:
- laguna-s-2.1 failed 85% of its production executions, ranging from 63%
to 100% depending on the lens. The failure signature was consistent:
review_json_absent, text_bytes=0 — the model returned no usable review
at all on production-sized payloads. Its confirmed-findings rate was 0.14
per run, with zero A-grade reviews.
- grok-4.5 failed 1.9% of 108 executions. In the code-verified audit of 4
PRs, all top-5 verified findings came from grok — about 14 real findings in
total, with a signal ratio of roughly 0.75 (three out of four things it
flagged were real).
The probe lied by being too small
Nothing about the probe was fabricated. Laguna genuinely produced substantive
output on a 500-token input. But our production payloads are 24–37k tokens —
about 50–70× larger — and at that size laguna's output collapsed to empty.
A capability that only exists at toy scale is not a capability you can pin a
production lens to. The probe's own caveat said as much; the production data
confirmed it.
The timeout reputation was stale
The other half of the correction cut the opposite way. Grok's 28% all-time
timeout rate was real — but it was history, not present tense. The timeout
wave traced back to an infrastructure problem in mid-July, not to the model.
Post-fix, grok's failure rate in this window was 1.9%.
Judging a model by its all-time aggregate is the same mistake as judging it by
a tiny probe: both substitute a convenient number for the number that
describes production today.
What we repinned, and why
- code-review-full stays on `x-ai/grok-4.5`. 2.27 confirmed findings per
run at a median $0.051 and 161s, with every top verified finding in the
audit. Nothing else came close.
- The laguna consolidation was abandoned. The branch that moved ten lenses
onto laguna was never merged.
Two methodology caveats we carry forward honestly: the per-lens grades are
volume-weighted (they measure how much a model delivered, not independently
verified quality), and some lenses have small samples — a lens repinned on
n=4 gets a week of daily grading telemetry before we call it settled.
The rule we now follow
Before any repin: measure signal-to-noise on production traffic, at
production payload sizes, against the incumbent. A probe can nominate a
candidate; only production evidence can promote one. Concretely:
- Every candidate gets at least one production-path round before we trust it.
- The first post-repin run per lens must produce real output
(output_tokens > 200), or the lens is treated as suspect.
- Ties keep the incumbent. A repin needs a margin.
- Failure modes count as results: a $0, no-output, or timeout run is a FAILED
trial, not a discard.
The frontier moves daily, and we want to keep trying new models. The arena is
how a new model earns a pin — and how an incumbent keeps one.