--- id: skill.reviewer-model-lab version: 1.0.0 name: reviewer-model-lab agent: ionwarp description: >- The IonWarp reviewer model lab. Per run: at most TWO trials against the newest review-study snapshot — trial 1 chases an open lead (or grading's top recommendation), trial 2 is one web-researched frontier candidate confirmed on OpenRouter. REPORT ONLY: it measures and recommends, and NEVER changes a pin — a model switch is an operator decision, proposed in the digest with its evidence. category: model-lab display_name: "Reviewer Model Lab" tagline: "Two trials, one reference PR — report-only" distribution: public triggers: - pattern: "Trial {model} on {lens}" examples: - "Model testing" - "Reviewer model testing" - "Trial grok-4.6 on code-review" - "Model lab" - "Repin security-review" inputs: project_id: type: string required: false description: >- Dash project id (pr_...) to trial in. Defaults to the project the newest review-study snapshot recommends against. # --- expectations: derived from measured runs (scripts/check-skill-expectations.mjs) --- # 4 completed runs on staging. p95: 490s, $0.7233, 4,047,346 input tokens. # Warning-only ceilings (p95 x headroom) — a breach is a named # `expectation_breaches` line, never a failure. Regenerate, never hand-edit. expectations: max_cost_usd: 1.45 max_reading_tokens: 8100000 max_duration_ms: 750000 instructions_file: SKILL.md default_backend: # The labs are ai-sdk orchestrations (MCP tools: tasks_create, skills_get, # data_query) — NEVER Modal/OpenCode. Loop-1 measurement (2026-07-31, # run_ms9grr1ztk5ami): with no default_backend the platform routed this skill # to Modal, where it died at the 1740s function ceiling with zero digest. platform: cloudflare harness: ai-sdk provider: openrouter model: deepseek/deepseek-v4-flash-0731 execution: # Cheap, capable default for the lab run itself; the operator can override. # The models UNDER TEST are chosen per §2, never by this pin. model: openrouter/deepseek/deepseek-v4-flash-0731 # Explicit 2026-08-31 (flash-lane repin plan R4.1): none = request no # reasoning — wire-identical to the previous absent value, but the pin now # STATES it instead of delegating output volume to a provider default. reasoning_effort: none --- # Reviewer Model Lab — two trials, one reference PR **What this lab is.** One model runs each review lens: the **pinned** model, called the **incumbent** inside a trial and in the ledger — same thing, and the digest says "pinned" because that is the word a reader outside this file knows. A **trial** re-runs that ONE lens on the SAME pull request with a different model (the **challenger**) and compares the two. Trials run through the PRODUCT's review pipeline (`input.reviewers`), never a local swarm. Resolution order (highest wins): run `input.reviewers[].model` > project.md `ionwarp.review.model_pins` > the reviewer SKILL.md `execution.model` pin. **This lab is REPORT ONLY. It never changes a pin.** `skills_update` — on `project` or any other skill — is BANNED here, as is every other write that alters which model runs a review. The output is evidence plus, when the §3 bar is cleared, a **proposal** the operator applies. That is deliberate: a pin change is a product change, and an agent that both grades the evidence and applies the verdict has no reviewer. The lab reads pins (`skills_get`) so it can report what is live and notice when a human changed one. Contract: `$IONWARP_API_URL/docs/raw/guides/model-lab.md` — the raw markdown endpoint. `/docs/guides/model-lab` (no `raw`) is the HTML page for humans; fetching it gets you a webpage, and `data_query` can never read docs at all (docs are a worker route, not a data collection). ## The closed read set The only reads this run makes are the ones its sections name: `whoami` and `skills_get` (§2), the newest review-study snapshot + `trials.json` from the state dir, the launch and status routes (§1), the candidate roster shipped beside this file (`references/candidates.json`), and the OpenRouter models endpoint for live price/context/availability. **No web research** — the roster is the candidate list, and a run that goes searching has left its read set. An external operator agent (running this skill from the public docs) holds the same closed read set — orientation beyond it is a round trip that returns what you were already told. ## Platform execution notes (measured 2026-07-31) - **One task per lab run.** A `POST /runs` (or `tasks_start`) on a task with an active or just-cancelled run RE-ATTACHES to that run instead of starting a new one — a backend passed at run time never takes effect on the reattached run. Always `tasks_create` a fresh task per lab run; the backend must be set in the task's `input.backend` at CREATION (loop-1 measurement: two "restarts" of ts_ms9grkfet5uvpj all resolved to the same Modal run and died at its 1740s ceiling). - **Launch, then grade ASYNC — never wait for trial reviews inside one run.** On the platform (ai-sdk backend) a run that polls trial reviews to completion outlives its queue-backed invocation and dies to the DLQ (`stream_transport_failed`; measured 2026-08-01: two ~50-minute waiting runs died while every bounded run completed). The platform shape is TWO runs: run 1 launches both trials, verifies payload integrity, records the ledger rows as `pending_grade`, and ENDS with a launch digest naming the trial task ids; run 2 (the next lab run, or an operator ask) grades from the posted findings + C2 telemetry and settles the verdicts. Only an external harness with its own re-invoke loop (Hermes/Claude Code) may wait synchronously. - **Bound every read.** Sample at most ~12 tasks per sweep and keep the digest inside its line budget — an unbounded transcript risks the persister truncating your FINAL message (`part_truncated_at_persist` dropped a 148-part run's digest on 2026-08-01). ## Runtime contracts (state dir · delivery · bands) **State directory.** The harness provides a durable state directory via env `IONWARP_LAB_STATE_DIR`. This lab's ledger is `$IONWARP_LAB_STATE_DIR/trials.json`; the review-study snapshots it consumes are the `YYYY-MM-DD-HHZ.json` files in the same dir. **Always resolve it absolutely** — a bare relative path resolves against the session CWD and silently finds nothing; the first live lab run once ran with no ledger at all and never noticed. A ledger that can neither be read nor created is a LOUD RED `ledger_unreadable` line that STOPS the proposal half of the run. Never assume, infer, or carry a tally forward without reading it — that run printed *"tally assumed 1 of 3"*, and a fabricated count is worse than no count, because two real wins plus one guess proposes a model that never cleared the bar. **Delivery.** The digest is the run's FINAL MESSAGE; the harness (a Slack cron, a platform task, a chat surface) delivers it — this skill never posts to Slack or GitHub itself. First character `🧪`, compose once, stop. **Band authority.** Each reviewer skill declares its `model_bands` in its own SKILL.md frontmatter (`aa_coding_index.min`, `input_usd_per_mtok.max`, `output_usd_per_mtok.max`, `latency_s.{min,max}`, `throughput_tps.min`) — read them via `skills_get` this run, never from a table baked into this file. The VALUE band — **$/CM ≤ $0.50 per critical-or-major** — is shared across all lenses. The bands and the current pin live in the reviewer skill's own frontmatter: one data source for the UI catalog, the labs, and the model selection process below. ## Model selection — how IonWarp picks reviewer models This is the operator's process; the lab exists to feed it evidence. - **Per-lens goals.** `code-review` = fast & cheap (deepseek-v4-flash/-pro class: low $/Mtok, high throughput, high AA coding index). `code-review` = the deep pass, 2–5 minutes, worth spending more (grok-4.6 class). `security-review` = v4-pro or gpt-5.6-luna class. `test-holes` and lighter lenses = cheaper models. - **The always-on trio (fast + full + security) runs on nearly every PR:** never pick expensive models for all three, and DELIBERATELY pick DIVERSE models across the trio — the same model on all three collapses finding diversity; different model families catch different bug classes. - **Models constantly improve and compete on price** — price cuts and sales, a brand-new release with a much higher coding index than its predecessor, a US provider picking up a hot model days after launch and competing latency/throughput/price down. That is a reason to keep the roster fed, not a reason for a run to go shopping: **new candidates enter through `references/candidates.json`, and a run trials the queue.** This bullet used to say "watch OpenRouter DAILY", which in a cron reads as "search now" — and a search that runs daily and remembers nothing tests a different model every day and finishes none of them. - **Inference constraint: only US-routable providers count** ("Not routable" Chinese-inference-only listings are excluded). A great model served only by a slow US provider (e.g. 28 tps / 3.5 s TTFT) is watch-listed — re-check daily until a US provider meets the throughput/latency bands. - **Qualification** = the lens's four frontmatter bands (AA coding index, in/out price, latency, throughput) on an acceptable provider. **Repin** = the N=3 bar (§3) PLUS more HSF per dollar of input than the incumbent (HSF/$ and $/Mtok-in from telemetry). The bands and the current pin live in the reviewer skill's own frontmatter — one data source for the UI catalog, the labs, and this process. ## §1 Launching a review (also the ad-hoc PR-review path) The one way this lab reviews a PR — MCP `tasks_create` (server `ionwarp`) or REST; create, then POST `/api/v1/tasks//runs` with `{}` and the same headers: ```bash curl -sS -X POST "$IONWARP_API_URL/api/v1/tasks" -H "Content-Type: application/json" \ -H "Authorization: Bearer $IONWARP_API_PAT" -d '{"project_id":"", "product_agent_id":"ionwarp","skill_id":"ionwarp-review","title":"# on ", "input":{"type":"pr_review","pr_number":,"planning_required":true, "suppress_pr_comment":true, "reviewers":[{"skill":"","model":""}]}}' ``` `model` absent → the resolved pin runs. **Poll in ONE in-process loop per tool call, never `sleep N && curl` as separate tool calls** — each tool call re-sends the whole context prefix, so a dozen separate polls cost more than the review being measured. Write a small script (or use a harness-provided polling helper) that blocks on `GET /api/v1/tasks//status` inside one call and prints one compact line per reviewer; if it is still running at the call's time budget, re-invoke it with the SAME ids. On an HTTP error, fall back once to a direct status GET and say so in the digest. Always send a real `User-Agent` header from scripts — Cloudflare bot protection 403s UA-less requests. `reviewers_telemetry` carries each reviewer's model, grade, findings (by severity), cost_usd, output_tokens, duration_ms, error — the only measurements this lab may cite. For a human "review PR ": launch with NO `reviewers` key (planner picks the lineup), report the verdict in four digest lines. **Auth split:** task LISTS are MCP-only (REST `GET /api/v1/tasks` is session-cookie-only, 401 to a PAT — that 401 is not your credential); POST tasks/runs and GET status take the PAT; skills reads (`skills_get`/ `skills_list`) are MCP-only; `project_id` is snake_case everywhere. ## §2 The two trials `whoami` on the ionwarp MCP; unreachable = LOUD RED digest line, stop. - **Pinned model** per lens = the LIVE pin, read this run with `skills_get {skill_id:"project", project_id:""}` (`config.ionwarp.review.model_pins`) — **snake_case `project_id`; camelCase `projectId` is REJECTED with "project_id is required"** (a run once misread that rejection as "skills_get unreachable with PAT"). Fall back to the reviewer SKILL.md `execution.model` default when the project pins none, and on a real `skills_get` error print the named line `pin_unreadable (skills_get: )` + use that same SKILL.md default — never infer the pin from grading, and never invent a REST fallback (**there is no REST skill endpoint**; MCP `skills_get`/`skills_list` are the ONLY PAT-visible pin reads). Its measurements (med cost / med latency / CM per run) come from the newest review-study snapshot rows for that lens (`$IONWARP_LAB_STATE_DIR/*.json`); unmeasurable = a named gap. **Read the pin, never infer it from grading** — the day after a human applies a proposal, grading still shows the retired model at higher n, so a grading-derived "incumbent" silently keeps testing against a model that no longer runs. A pin that changed since the last run is a named digest line (`pin changed by operator: → `) and resets every tally on that lens (§3). - **Reference PR** = the most recent COMPLETED review whose PR is **not already in the §3 ledger** for this (lens, incumbent, challenger) triple — agreement accumulates only across DISTINCT PRs, so re-running yesterday's PR buys nothing. None = the named line `no fresh reference PR for `. At most TWO trials per run, one lens each. The cap counts TRIALS, not lenses or recommendations: if grading recommends three lenses you still trial the TOP one only. Every other recommendation gets its OWN LINE in the digest's `NOT TESTED` block (§4) marked `2-trial cap` WITH grading's reason. Never silently dropped, never a third trial, and never a separate Gaps line for it. -1. **Circuit breaker — check BEFORE spending anything.** Read the last two `trials.json` entries. If BOTH carry the SAME structured `error_code` field (the §3 ledger requires it on every failed row) and no fix event is recorded since — a **fix event** is a row in the ledger's top-level `fix_events: [{date, error_code, evidence}]` array, written by an operator (or an agent with named evidence) when the platform cause is fixed; the breaker ignores failures dated before the newest fix_event for that code — **stop here**: emit the three-line RED digest below and END the run. Do not web research, do not shortlist models, do not pull telemetry, do not launch. The code is one of the lab's CLOSED failure enum — `provider_auth_failed` and `executor_model_call_read_timeout` are real platform codes copied verbatim from the task's error; `planning_crashed` (planner died before any reviewer ran — the platform emits no code for this) and `trial_lens_not_selected` (planner ran but dropped the requested lens) are lab classifications. Never invent a fifth spelling: two runs must classify the same failure identically or the breaker can never fire. ``` 🧪 Reviewer Lab — 🔴 HALTED — on the last 2 runs (). Platform-side; trials cannot score. Next: clears automatically when a run succeeds, or ping the IonWarp team. ``` Why this exists: one four-day stretch spent $16.95 across 13 runs and produced ONE usable verdict — 16 of 22 trials failed product-side (`review_planning_failed` ×7, then `provider_auth_failed` 401), and six consecutive digests all said "No change today". Re-running a broken platform buys nothing. A halted run costs ~$0.25 instead of $1.30. **The breaker is about the PLATFORM being down, never about a model losing** — a challenger that genuinely loses is a result, and results never trip it. 0. **Trial 1 is the LEAD, if there is one.** If any triple is mid-run at 1 or 2 wins, re-trial THAT SAME (lens, incumbent, challenger) triple on a FRESH PR before anything else. Without this a triple gets ~1 trial a WEEK and can never reach 3 wins in 14 days — the bar would be unreachable and the lab decorative. **A triple already at 3 wins is NOT a lead — it is a standing proposal (§3) and is never re-trialed**; it costs nothing, holds its digest line until the operator acts or it expires, and re-spending on it would buy a fourth win nobody asked for while burning the reference PRs a different lens needs. Only when no lead is open does trial 1 fall back to: 1. **Trial 1 (no lead open) — the recommendation, if grading made one.** The snapshot's `recommendation` (`[{lens, model, reason}]`) is what grading asked us to trial; take its top lens→model change. Empty/absent = the named line "no recommendation this window" (not an error) — trial 1 falls through to a ROSTER candidate exactly like trial 2, on the busiest lens. **An empty recommendation must never end the run.** Grading's fleet gate needs ≥3 terminal reviewer executions in its window; when the window is quiet — or, as happened for twelve days, when the harness is reading the wrong host — the snapshot is legitimately empty. If that emptiness also emptied the lab, the two silences compound: one lab reported eight consecutive days of "TESTED — none" while production was running 1,216 reviews. **The roster is a standing queue that never empties**, so a quiet fleet costs the lab a recommendation, not a run. 2. **Trial 2 — the top QUEUED row of the candidate roster**, on the lens whose grading row has the worst fail% or C+M/run (busiest lens when grading is empty). **First, disqualify a lens whose fail% is OUR defect, not the model's.** A row with `graded_count: 0` and `cost_usd: null` contains no model datum at all — every run died before producing gradeable output, so its 100% fail% says nothing about the incumbent and swapping the model cannot improve it. Treat such a row as a **product-defect report**, never a repin candidate: name it in Gaps as `lens_structurally_failing` with the median latency, and pick the next lens down. The tell is the median latency sitting on a round number — that is one of OUR clocks, not a vendor. A real row once read `fail_pct 100.0, graded_count 0, med_cost_usd null, med_latency_ms 25083` — a 25,000ms browser fast-fail plus 83ms, firing on reviewers that were never given a browser. Two consecutive runs recommended swapping a model that had never once been allowed to execute, and each kill also wrote a durable learning teaching the planner that the model is incapable at the lens. Spending a trial slot there buys nothing and deepens the false record. **NOT the lead lens** — a tally is per (lens, incumbent, challenger) TRIPLE and trial 2 brings a different challenger, so pointing it at the lead starts a NEW triple at 0 and burns the slot on a lens that already has a working climb. It compounds nothing. **The ROSTER picks WHO; OpenRouter confirms AVAILABILITY, PRICE and CONTEXT.** The roster is `references/candidates.json`, published with this skill: an ordered list of `{model, added, why, status}` rows. Identity is DATA and lives there; price and context are LIVE and are re-read every run. This replaced a run-time web search — *"shortlist ≤5, the smartest models released in the last 60–90 days, highest intelligence first"*. That rule sounds rigorous and behaves badly: it re-derives a different shortlist every day, so no candidate is ever finished, and it optimises for a ranking nobody here uses while the models an operator actually asked about are never reached. **A queue an operator can edit is the whole interface** — adding a row is the entire act of requesting a trial, with no skill edit and no deploy. (a) **Take the top roster row with no trial entry for this lens in the ledger** — skipping anything already pinned in the resolution order, or carrying TWO `challenger_lost` entries on distinct PRs for this lens within 30 days (§3 — one loss never excludes). None left = the named line `roster exhausted — every candidate tried on ` (not an error). **"What has been tried" is the LEDGER's fact, and the roster must never carry a second copy of it.** The roster shipped with a `status` field until 2026-08-23; it was unreachable state. This file lives inside the PUBLISHED skill — read-only at run time — and `skills_update` is BANNED in this lab, so nothing could ever advance a row from `queued` to `measured`. A queue that cannot dequeue itself is not a queue, and a second copy of a fact the ledger already owns is the exact duplication this roster was introduced to remove. The roster is identity and reason; `$IONWARP_LAB_STATE_DIR/trials.json` is what happened. (Found by an IonWarp review of the PR that introduced it — the lab's own review fleet, catching the lab's own regression.) (b) **Confirm it on OpenRouter** (`GET https://openrouter.ai/api/v1/models`, no auth; `_shared/scripts/shortlist_models.py --grep ` keeps the payload out of context). Trialable only if the exact `id` is present with non-zero `pricing.prompt`/`pricing.completion` AND a US-routable provider serves it (see Model selection above — a "Not routable" listing, or a provider under the lens's `throughput_tps.min` / over its `latency_s.max` band, is watch-listed, not trialable). Absent = `dropped — not on OpenRouter ()`, named, never silent. The API unreachable = LOUD RED, skip the trial. (c) **CONTEXT GATE — check the window before you spend the slot.** A candidate whose `context_length` is under **1.3 ×** the lens's measured median `input_tokens` is `dropped — context_below_lens_profile (: < )`. It is not a quality judgement and no trial can overturn it: a review of a real diff is 46k–183k input tokens on our fleet, so an 8,192-token model is structurally incapable of the job whatever its benchmark says. The roster row moves to `status: "dropped"` with that reason — **kept, never deleted**, because a deleted row is one that gets re-proposed next month by someone who does not know it was already answered. (d) **A candidate that times out has a SIZE question, not a quality one — ask it cheaply.** A `trial_backend_timeout` on a deep-lens diff is not a verdict (§3 step 0). Before writing the candidate off, re-trial it ONCE on a substantially smaller diff. Measured 2026-08-22: three candidates each hit the 1740s ceiling on a 97,126-token `code-review` and then completed the SAME lens on a 39k-token PR in **16–22 seconds** — `xiaomi/mimo-v2.5` at $0.0057 (2.3× cheaper and 2.4× faster than the pin on that diff). That is a **cliff**, not slowness, and it is not a context-window limit: all three advertise 262k–1M. Record it as `size_cliff (: ok at k, timeout at k)` and route the candidate to the sweep band (`test-holes` 48,638 in, `analytics-review` 46,496) rather than dropping it — a model that is fast and cheap inside its range is a real find for the lenses that live there. (e) **Trial exactly ONE** money-stop survivor. **Money stop (the only qualification gate) — TWO numbers, in the normalized unit.** Compute both at OpenRouter's CURRENT price on the lens's real token profile: incumbent median input_tokens × candidate input price + median output_tokens × candidate output price. **Take that profile from telemetry, not from a remembered range.** This line used to say "a review is 20–60k input tokens"; measured across 120 production runs (2026-08-22) the real medians are **46k–183k** — `analytics-review` 46,496, `test-holes` 48,638, `plan-review` 71,406, `security-review` 87,938, `log-coverage` 91,030, `complexity-accumulation` 94,804, `code-review` 97,126, `legacy-pathways` 182,888. Every projection made against the old range understated cost by roughly 2×, and it is the same number the CONTEXT GATE above divides by, so a stale range hides a structurally-impossible candidate as well as a merely expensive one. Read the medians from the newest review-study snapshot each run; if the lens has no row, say `lens_profile_unmeasured` and do not project. **A cost you cannot reconcile with the rate card is a GAP, not a measurement.** `cost_usd` arrives with `usage_estimated: false`, which reads as "measured" — but a trial on 2026-08-22 recorded **$0.0453 for a model whose OpenRouter rate card is $0.00/$0.00** (`stealth/ox-alpha`, 52,420 in / 5,806 out). One of those two numbers is wrong and the money stop reads exactly this field. When the recorded cost and `input_tokens × rate + output_tokens × rate` disagree by more than 2×, print `cost_unreconciled (: recorded $ vs rate-card $)` and resolve the trial on RECALL alone — never let an unreconciled cost qualify or condemn a model. (Two known-legitimate causes of a gap in the same direction: a reasoning model bills reasoning tokens that `output_tokens` excludes, and a model absent from the pricing registry falls back to a flat default rate.) 1. **Projected `$/Mtok-in` must be inside the lens's band** — the `input_usd_per_mtok.max` read from that reviewer skill's frontmatter `model_bands` this run (deep lenses like `code-review` and `plan-review` carry a deliberately wider band than the default; the frontmatter is the authority — if it moves a band, this moves with it). Over = the named skip `projected_rate_over_band ( → : $/Mtok-in > $)`. **Value rider:** the rate band QUALIFIES a trial; it never by itself condemns a pinned model. A repin verdict requires failing the VALUE band too (`$/CM > $0.50` per critical-or-major) — expensive-but-earning is "buying depth on purpose": the measured case was a deep-lens pin at ~$0.09/CM against a nominally cheaper-looking lens at ~$0.96/CM. 2. **Projected per-run cost ≤ $0.25** survives, but ONLY as a per-trial RUNAWAY ceiling. Over = the named skip `projected_cost_over_ceiling ( → : $ > $0.25)`. **They are not the same test and they fail differently.** A per-run figure is dominated by the SIZE of the PR the trial happens to draw, so it can never tell a bad pin from a big diff. `$/Mtok-in` is stable per model, which is why it is the QUALIFICATION gate and the per-run number is only a runaway backstop. The concrete failure this fixes: through one whole measured window a lens sat comfortably inside $0.25/run at a median $0.289 — while costing $3.30/Mtok-in, 10× the fleet, and burning 40% of the entire review bill. The flat ceiling could not see it. Rank on the rate. **A paused project blocks trials — never unblock it.** `run_mode: paused` (or a paused/approval refusal on start) = the named skip `trial skipped — project paused (run_mode)`; `projects_update` to flip that gate is operator-only (a dry run once did it and silently re-armed spend the money stop had halted). Report the gate, never open it. Then launch per §1, poll to terminal. **Every trial payload carries `"suppress_pr_comment": true`** — a replay through the real pipeline must never overwrite the organic review comment the PR author reads (five live comments were rewritten before this flag existed, 2026-08-08); telemetry + `GET /status` carry everything a trial grades. **The payload IS the trial.** `tasks_create` must carry `input.reviewers[] = {"skill":"","model":""}`. With `input: {}` the planner picks its own lineup and you have bought a default review that proves nothing about any model — the first live run did exactly that and burned its launch. **Read the created task's `input` back before polling:** no `reviewers` key ⇒ `trial_launch_malformed`, discard it, never grade it, and say so in Gaps. **`trial_launch_malformed` means the `reviewers` KEY IS ABSENT from the read-back — nothing else.** It is not the code for "the key was there and the lens still did not run". Two trials were once filed under it while the key was present, which sent the reader hunting a payload bug that did not exist and hid the real one. Use these three distinct outcomes: | what the read-back / telemetry shows | code | graded? | |---|---|---| | no `reviewers` key in `input` | `trial_launch_malformed` | no — payload bug, ours | | key present, requested lens ABSENT from `reviewers_telemetry` | `trial_lens_not_selected` | **no** — the lens never executed, so the run is no evidence about the model | | key present, lens ran | grade it normally | yes | `trial_lens_not_selected` is a **planner-selection** outcome, never a model verdict: a model that was never invoked cannot have lost. Do not reset the ledger tally on it, and name the lenses the planner ran instead so the pattern is visible. `input.reviewers` is an ALLOWLIST (the planner may run FEWER of the requested lenses but can no longer run others), so a persistent `trial_lens_not_selected` means the planner is deliberately skipping that lens on that diff — report which diff, and prefer a PR whose files touch the lens's domain rather than re-firing the same trial. ## §3 Decide the trial, then decide the pin **Per trial** (same PR, same diff, both sides), in order. **Step 5 (reason untested) is checked BEFORE step 3** — a reliability-motivated recommendation can never be settled by one recall trial, so it must not be graded as one: 0. **Backend timeout is not a model verdict.** A run that ends `modal_function_timeout` (the backend's own function ceiling, 1740s) produces NO reviewer telemetry — there is nothing to grade, so it is neither a win nor a loss. Record `trial_backend_timeout`: it **neither advances nor zeroes** a tally, and the roster row stays `queued` for a lighter lens. Measured 2026-08-22: three candidates each burned ~29.2 minutes on a 97,126-token `code-review` diff and timed out, against a 622s worst case among the seven that finished. Counting those as losses would exclude three models nobody had measured. (The reason is in `task_runs.error_json`, correctly coded — `task_runs.error` is null, which is what makes this look undiagnosable to a reader checking the wrong column.) 1. **Hard fail.** cost_usd 0/null with no output, output absent, or any `error` = **FAILED**; record the code, the tally resets (ledger below). 2. **No answer key = NO INFORMATION.** Neither side produced a critical-or-major finding ⇒ the PR proves nothing about either model. Record `no_information`: not a win, not a loss, not a tie — it neither advances nor resets the tally, and is never quietly counted as agreement. 3. **Recall decides first, and recall is what we are buying.** Challenger CM (critical+major) vs incumbent CM on the same diff: - **challenger CM > incumbent CM ⇒ `challenger_won`**, provided the extra finding SURVIVED to the delivered review (present in the posted comment after dedup + the unfounded-absence veto), the challenger is inside the money stop, and `output_tokens ≥ 200`. Cost does NOT veto this. (~38% of a measured window's headline findings were false; on a lens at C+M/run 0.0 a lone new finding is as likely hallucination as defect.) A claimed finding that did not survive = `no_information` + `unverified_finding`. A reviewer exists to catch defects; refusing a model that caught a real major because it cost $0.07 more optimises the wrong thing. (This rule is INVERTED from the first draft, which emitted KEEP for exactly that case and then banned the better model from the lens for 30 days — the most valuable signal in the run, recorded as a loss.) - **challenger CM < incumbent CM ⇒ `challenger_lost`.** Two consequences, and they are deliberately different sizes: (i) **the tally ZEROES** — a pin needs 3 wins in a row, so one loss restarts the climb; (ii) **exclusion is slower** — a model leaves a lens only after **2 losses on DISTINCT PRs within 30 days**, and never while it holds a `challenger_won` on that lens in the same window. Zeroing a tally is cheap to recover; a lockout is not. - **equal CM ⇒ cost/latency is the tiebreak**, and only here does a margin apply: `challenger_won` needs ≥25% cheaper OR ≥25% faster, plus fewer-or-equal non-CM findings. Equal recall with no margin = `no_information` — the same state as no-answer-key (incumbent stays, tally untouched, not an exclusion), so there is one verdict for it, not two. 4. **`output_tokens ≥ 200`** on the challenger, else `FAILED` — below that, suspect the lens, never count the result either way. 5. **Answer the question that was actually asked.** A recommendation carries a `reason`. If that reason is about RELIABILITY (error rate, timeouts, zero-output runs) rather than recall, a single clean run CANNOT rebut it: record `no_information` with the gap `reason_untested ()`, never `challenger_lost`. Otherwise the lab locks out the fix for a reliability problem using an unrelated measurement — and grading re-recommends it tomorrow, straight into the exclusion it just earned. Reliability claims are settled by the grading table's fail% over n runs, not by one trial. 6. **An axis you did not measure is a GAP, never a verdict.** If cost or latency is missing on either side, you may not resolve a tiebreak against the challenger on it: record `no_information` with the named gap (`latency_unmeasured` / `cost_unmeasured`) and say so in the digest. Absence of evidence never decides a trial. 7. **Grades are BANNED as trial evidence.** A trial is decided by CM RECALL first and the cost/latency bands second; IonWarp's own letter grade decides nothing, ever. It is `3×critical + 2×major + 1×minor` thresholded — a finding COUNTER, structurally unable to see a false positive or a miss, which are the only two things a trial is about. `skill.review-study` renders no A/B/C columns for exactly this reason. A grade may be PRINTED beside a trial as context; it may never move a tally, break a tie, or appear in a `reason`. **After the operator applies a switch, the retired model is not challenged for 14 days** on that lens. Its three defeats are recorded as the winner's `challenger_won`, never as its own `challenger_lost`, so the exclusion rule cannot see them and nothing else stops an immediate flap back. **Pins are PER PROJECT.** `project.md` lives in a project, so evidence earned on 3 PRs in one project proposes a change to that project's pin only — every other project keeps its own. The 3 winning PRs MUST be in one project, the digest header names that project, and a lens whose win-PRs span projects is `no_information` with the gap `wins_span_projects`. **One PR is never enough. `N = 3`.** A single trial is only ever **PROVISIONAL**: the same model/lens has scored CM=1 on one PR and CM=0 on the next, so a one-PR verdict measures the PR's content as much as the model. A switch is PROPOSED only when the SAME challenger beats the SAME pinned model on **3 distinct PRs** inside **14 days**, no intervening `challenger_lost`/`FAILED`. Until then the lens reports `N of 3` with the date the lead expires (§4). Print wins EARNED — never round up, never carry a tally you did not read out of the ledger. The repin the operator then weighs is the N=3 bar PLUS the value test from Model selection above: more HSF per dollar of input than the incumbent. ### The proposal — what the lab does instead of writing At 3 of 3 the lab writes ONE ledger row `bar_cleared` and emits ONE proposal in the digest's `RECOMMENDING` block. It writes nothing else. The proposal states: - the exact change — `: → ` in ``'s `project.md` `config.ionwarp.review.model_pins`; - the evidence — the 3 PR numbers with each win's C+M delta, the median cost and latency of both sides, and the links (§4) to the reviews the operator can read; - the **pre-switch baseline**, copied into the `bar_cleared` ledger row and reprinted verbatim in the proposal: the pinned model's fail% and C+M/run in the grading snapshot on the day the bar cleared. Without it there is nothing to measure a regression against later, and a bar you cannot read back is unenforceable. When the lens has no fail% at all, say so and use C+M/run alone. **A proposal STANDS until it is applied or expires.** Every run re-reads the pin (§2) and re-prints the standing proposal with its age; the lens is not re-trialed while it stands (§2 step 0). Two outcomes end it: - **applied** — a run reads the pin and finds the challenger live. Record `operator_applied` (with the date), reset every tally on that lens, start the 14-day no-rechallenge window, and say so in the digest once. **An operator-applied repin the lab did not propose is recorded the same way** — the ledger's job is to know what is pinned and since when, not to take credit. Record it on the first run that reads the new pin, with the pre-switch baseline of the model that was REPLACED. - **expired at 14 days** from the 3rd win — record `proposal_expired`, drop the line, and the lens returns to the normal trial rotation. An expired proposal is reported with its date, never silently dropped: a tally must never change without a named event, the same rule the review runtime lives by. Neither `bar_cleared`, `operator_applied` nor `proposal_expired` is a trial outcome; none of them advances or zeroes a tally by itself (the reset on `operator_applied` is the pin change, not the verdict). **A worked example of an `operator_applied` row worth imitating.** When an operator repinned a security lens off an expensive model, the row carried its pre-switch baseline verbatim: ``` fail% 0 · C+M/run ≈0.3 · med $0.289/run · $3.30/Mtok-in · 31 runs · 24 of 31 zero-finding ``` Read that line as a pair: the retired model's fail% was **0** — it never errored — and it still produced zero verified-unique findings across 10 independently re-graded swarms. **A perfect reliability number next to a zero yield number is exactly the shape this lab exists to catch**, and it is why "did it run" and "was it worth running" are different columns. The live gate on such a repin is `output_tokens ≥ 100` on the first production run after deploy: a sub-100-token approve is a FAILED round, not a fast one — report it as a LOUD line naming the revert target, and do not wait for a 3-of-3 tally to say so. **Regression → a revert PROPOSAL, same bar.** A model the operator switched to whose grading rows fall below the recorded pre-switch baseline at **n ≥ 3 runs** gets a `RECOMMENDING` line proposing the switch back, quoting the baseline and the current numbers side by side. The lab does not revert it; one bad run never proposes a revert. **The ledger — `$IONWARP_LAB_STATE_DIR/trials.json`.** Starter `{"trials": []}` — create it if absent. Newest 100 kept. (Unreadable-and-uncreatable = the LOUD RED `ledger_unreadable` contract in Runtime contracts above.) Per trial: `date`, `lens`, `incumbent` (the pinned model — the field keeps this name so older rows still read), `challenger`, `project`, `pr`, `task_id`, `task_url`, `verdict` (`challenger_won` | `challenger_lost` | `no_information` | `failed` | `bar_cleared` | `operator_applied` | `proposal_expired` | `trial_backend_timeout`), `error_code` (**required on every `failed` row, `null` otherwise** — the breaker keys on this field, never on free-text `note`; use the closed enum from §2 step -1), and — for BOTH sides, so a proposal weeks later can reprint the evidence that earned it — `cm_*`, `total_findings_*`, `grade_*`, `cost_*_usd`, `latency_*_ms`, `output_tokens_*`, plus `note`. Record what you measured, not what you compared: a tally that survives 14 days while its numbers do not is a claim with no evidence behind it. A `bar_cleared` row also carries `baseline_fail_pct` + `baseline_cm_per_run` (the pre-switch baseline the revert rule reads back). A triple's tally = consecutive `challenger_won` entries on DISTINCT `pr` values in the last 14 days. A win that ages out of the 14-day window is reported in the digest as `expired` with its date — a tally must never change without a named event, which is the same rule the review runtime lives by. A day that yields `no fresh reference PR` does not consume the window. `challenger_lost`/`failed` zeroes it; `no_information`, `bar_cleared`, `operator_applied` and `proposal_expired` are skipped over (they are not trial outcomes); an OPERATOR-APPLIED pin change (not a grading-row shift) resets every tally on that lens, which is why §2 reads the live pin every run: reading the incumbent off grading made the rule inert — the day after a switch, grading still shows the retired model at higher n, so "the pin changed" was undetectable by the very computation that governs it. ## §4 The digest (≤32 lines, the run's final message) **Delivery: the final message IS the digest** (Runtime contracts above) — first character `🧪`, reason in tool calls never in prose, compose once, stop. This lab's fixed second line: `What this is: today's challenger-model trials against the pinned reviewer models — report-only; pins change only by your hand.` The final message contains: - no "Now I need to compile the digest", no "Let me also check…"; - no draft digest followed by "Wait, I need to reconsider" followed by a second digest — decide BEFORE you emit, and emit ONE digest; - no "the digest above is the final message", no sign-off, no fence around the whole thing. If you catch yourself writing a sentence that is not a line of the digest, that sentence belongs in a scratch file. Re-emit the digest alone as your new final message and stop there. Not hypothetical, and it has failed BOTH ways. Once, a complete digest was composed inside a scratch message beginning "Now let me compose the digest" and the channel received a trailing verification note instead. With that gate fixed, a later digest was correct at the END of the final message — but 84% of that message, ~2,000 words, was the model narrating "Now I need to compile the digest… Wait, I need to reconsider…" first, and all of it shipped ahead of the digest. The composition was flawless both times; the DELIVERY was scratchpad both times. ONE plain-text final message answering exactly two questions, in this order: **what did you test, and what are you recommending.** Nothing else is the point. Over budget, cut in this order: `Gaps` detail, then `Cands` detail, then collapse `NOT TESTED` onto fewer lines. Never cut a TESTED measurement line, a recommendation, or a link — those three ARE the message. A digest with two trials fully measured is doing its job; a shorter one that dropped the numbers is not. **The digest reports; it never schedules.** There is no column, word, or line about what "ships tomorrow" or what will change — the lab changes nothing (§3), so a forward-looking column is a claim it has no right to make. An earlier version headed column 2 `SHIPS TOMORROW` meaning "the model pinned right now"; readers reasonably took it for a deployment the cron had decided on. Say `pinned` and mean today. Banned strings: `SHIPS TOMORROW`, `ships`, `will ship`, `shipping`, `deploying`, and any future tense about a model change. **Formatting — the reader is on a phone, make it scan.** The common delivery surface renders Slack mrkdwn: `**bold**` (DOUBLE asterisks — a bare `*x*` often renders as _italic_), `` ``` `` code fences (monospace, for column alignment), and `` links — but NOT inside a fence, where links and bold render literally. So the layout is: **bold + emoji on every header and outcome line** (scannable), **the two measurement rows inside a per-trial ` ``` ` fence** (columns line up), **findings/why/standing/read as normal mrkdwn lines** below the fence (so the links stay clickable). Emoji legend, used consistently: 🧪 header · ✅ challenger won · ❌ failed · ➖ settles nothing · ▲ leading (N of 3) · ⏭️ not tested · 💡 recommending · 📊 candidates · ⚠️ gaps. One emoji per line, never a pile. Four blocks, in order. `TESTED` and `RECOMMENDING` always appear; `NOT TESTED` and `Gaps` only when non-empty. Imitate this EXACTLY — the fences, the bold, the emoji, the link syntax: 🧪 *IonWarp model lab* · · `` · ref PR # · 2 trials · $0.098 (both sides) *TESTED* — `C/M/T` = critical / major / total findings · `$/Mtok-in` = cost ÷ (input + cache_read + cache_creation), the scope-normalized rate the money stop bands · `!` = over band ✅ *complexity-accumulation* — grok-4.6 vs pinned deepseek-v4-pro-0813 → *grok won, +1 major* ``` challenger C0 M1 T4 · $0.0822 · 1.81 $/Mtok-in! · 81s · 963 out tok pinned C0 M0 T3 · $0.0138 · 0.33 $/Mtok-in · 306s · 1,204 out tok ``` findings · grok caught the extraData override (major), survived the veto; the 3 both flagged agree; missed nothing v4-pro found standing · win *2 of 3* on distinct PRs, lead expires read · <…/tasks/ts_a|grok review> · <…/tasks/ts_b|v4-pro review> · <…/pull/N|PR> ➖ *code-review* — deepseek-v4-pro-0813 vs pinned deepseek-v4-flash → *settles nothing* ``` challenger C0 M0 T3 · $0.0012 · 0.33 $/Mtok-in · 98s · 412 out tok pinned C1 M0 T4 · $0.0009 · 0.11 $/Mtok-in · 80s · 503 out tok (reused) ``` findings · v4-flash's critical (unbounded D1 read) is absent from v4-pro; nothing new either side why · grading asked over a 15.4% error rate, not recall — one clean run can't answer reliability, counts for neither read · <…/tasks/ts_c|v4-pro review> · <…/tasks/ts_d|v4-flash review> · <…/pull/N|PR> 💡 *RECOMMENDING* No change today. Closest: complexity-accumulation → grok-4.6 at *2 of 3* wins. ⏭️ *NOT TESTED* • *security-review* · pinned — grading wants v4-pro (); hit the 2-trial cap • *code-review* grok-4.6 · *test-holes* glm-5.2 · *analytics-review* v4-pro — no recommendation 📊 *Cands* · 5 researched = 1 pinned + 1 trialed () + 3 not reached () ⚠️ *Gaps* · ledger: 4 trials (2 from the previous run) **Do NOT wrap the whole digest in one big fence** — that kills every link and every bold. Fence ONLY the two measurement rows of each trial. Everything else is mrkdwn. **A failed side with no numbers gets no fence** — just the bold outcome and the reason, e.g. `❌ *security-review* — v4-pro vs pinned → *failed (runner crash)*` then a `findings ·` line, no measurement block. **TESTED** — one entry per trial actually launched, never per lens considered. This block is the largest thing in the digest on purpose: it is the only place the run's actual measurements exist. Early digests reported cost alone — no latency, no findings breakdown, no task ids — so nobody could tell a model that was slow from one that was wrong, and nothing was recoverable from the message afterwards. Each entry is a headline, TWO measurement lines, and the tail: 1. **Headline** — ` — (challenger) vs (pinned) → `. The outcome is plain English with its margin: *won, +1 major* / *lost, −1 critical* / *settles nothing* / *failed ()*. Verdict jargon (`challenger_won`, `no_information`, `reason_untested`) belongs in the ledger, not in a message a human reads at 6am. 2. **`challenger` / `pinned` lines — one per side, same fields in the same column order, so the comparison is read vertically and never computed by the reader.** Every field comes from that side's `reviewers_telemetry` row: - `C M T` — critical / major / total findings. C and M are what the verdict turns on (§3); T is the noise check, since more total findings at equal C+M is worse, not better. - `cost` — `cost_usd`, 4 decimals. Mark `~` when estimated. - `latency` — `duration_ms` as whole seconds. **Always printed.** It is a tiebreak input (§3) and the thing an operator feels; a digest that omits it cannot answer "why did the review take 8 minutes". - `out tok` — `output_tokens`, **always printed**, not only when it trips the ≥200 gate. 150 vs 963 is the difference between a model that answered and one that stopped, and it is invisible in cost. - `$/Mtok-in` — `cost_usd ÷ (input_tokens + cache_read_input_tokens + cache_creation_input_tokens) × 1e6`, 2dp, with a trailing `!` when it is over the lens's band max (§2 money stop). **Never print a tok/s rate computed from `output_tokens`** — that field EXCLUDES reasoning tokens on every reasoning model pinned here, so the rate is fiction (one model recorded 60 output tokens on a run that billed ~12k). Latency is already carried by the wall-clock field above. When the token counters are absent print `$/Mtok-in —` and name `tok_absent` in Gaps, never a computed-from-nothing rate. - `(reused)` on a side taken from the reference run instead of a fresh launch. A FAILED side prints `FAILED ` in place of the findings group and keeps every field it does have — a failure with no numbers cannot be diagnosed later. 3. **`findings`** — what the two models actually disagreed about, naming the finding: what the challenger caught that the pinned model missed, what it missed that the pinned model caught, and whether the deciding finding survived dedup and the unfounded-absence veto into the posted review. A learning without its finding is an opinion, and ~38% of one window's headline findings were false, so "found one more" is not a result until it survived. 4. **`why`** — one sentence, only when the outcome needs it (a `settles nothing`, a `failed`, or a win the numbers alone would misrepresent). 5. **`standing`** — on a live lead, `win N of 3` **with the expiry date** (`N of 3` decays silently, so a climb and an expiring lead look identical without it). 6. **`read`** — the links (below). An axis you did not measure is a GAP, never an empty column: print the field with `—` and name it in Gaps (`latency_unmeasured`, `cost_unmeasured`, `tps_unreported`). §3 already forbids resolving a tiebreak on an axis you did not measure; a blank that looks like a zero is how that rule gets broken by accident. **RECOMMENDING** — either `No change today.` plus the closest live lead, or the §3 proposal in full: the exact edit (`: → ` in ``'s `project.md` `ionwarp.review.model_pins`), the 3 winning PRs with their C+M deltas, both sides' cost/latency, the pre-switch baseline to judge a later regression against, and how long it has been standing. End a proposal with who acts: *"I don't change pins — this one is yours to apply."* A revert proposal uses the same shape. Never more than one recommendation per lens. **NOT TESTED** — every lens named nowhere above, so no lens can hide by absence: its pinned model plus, when grading asked for something, grading's REASON and the reason it was skipped (`2-trial cap`, `no fresh reference PR`, project paused). A bare skip reads as "checked, fine" — `security-review · 2-trial cap` alone renders "this lens may be running a model that finds nothing" identically to a healthy lens. Lenses with no recommendation collapse onto one line. **Links are mandatory, and every URL is built from an id read THIS RUN.** Shapes, `` format: a review = `$IONWARP_API_URL//tasks/`; the PR = its GitHub url; a post = `$IONWARP_API_URL/blog/`; a scorecard = `$IONWARP_API_URL//scorecards/`. Every TESTED entry links BOTH sides' reviews plus the PR — the numbers exist so the operator can go read what each model actually wrote, and a digest without links makes that a 10-minute hunt. A missing task id is the named gap `task_url_unavailable ()`, never a guessed URL. The `read` line carries the post/scorecard links for anything this run touched or that covers a lens named above; omit it when there is nothing real to link. `spend` in the header = BOTH sides of every trial actually launched, not the challenger alone (challenger-only under-states the bill ~18% on a two-trial run). Say `(reused)` if a pinned side came from the reference run instead of a fresh launch. The `Cands` chain counts RESEARCHED candidates only; a lead-chase challenger comes from the ledger, not research, so state trials as `2 trials (1 researched, 1 lead-chase)` — otherwise `2 trials` in the header against `1 trialed` in the chain reads as a silent drop. The chain must RECONCILE (researched = dropped + skipped + trialed + not-reached) with every model id named — it is the only defence against "we looked at one model and called it research". On a fresh ledger write `no prior trials`, never `0 of 3`, which reads as three PRs tried. Forbidden: any future-tense model claim (above); narrating the rulebook ("nothing advanced or reset", "worth a re-trial"); a `none` line; a cost multiple when cost cannot veto a recall win; claiming or implying the lab changed a pin; and the PAT. ## Verify (self-assert) - No `skills_update` was called, and no pin, config, or `run_mode` was written — the only writes this run are the ledger and the digest. - Every TESTED entry names its challenger, the pinned model, a plain-English outcome with its margin, and carries TWO measurement lines with the same fields in the same order — `C/M/T`, cost, `$/Mtok-in`, latency, out tok — plus a `findings` line and links to both reviews (or the named `task_url_unavailable` gap). No field is blank: an unmeasured axis is `—` plus a named Gap. - Every number printed came from this run's `reviewers_telemetry` or the ledger. Nothing was inferred from a grading row, a previous digest, or a price list. - Every tally printed was READ from the ledger; none inferred, carried forward, or rounded up. A ledger that could be neither read nor created is the LOUD RED `ledger_unreadable` line and no tally is printed at all. - Every lens appears exactly once across TESTED and NOT TESTED. - A standing proposal is re-printed with its age; an applied or expired one is reported once with its date and then dropped. - The digest contains no `SHIPS TOMORROW`, no future tense about a model change, and no PAT. - **Your ENTIRE final message is the digest**, `🧪` first character, last line the last character, nothing before or after. Do NOT run these Verify checks as narrated prose in the final message — check silently or in a scratch file; whatever you write there ships ahead of the digest.