--- id: skill.review-study version: 1.0.0 name: review-study agent: ionwarp description: >- Daily quality grading for IonWarp's PR reviews. The head-to-head half (§0) is the point of the skill: pick ONE real PR where at least two review bots commented, read what each bot ACTUALLY WROTE, and grade them head-to-head against our own rubric — judging false positives and MISSED bugs, which no counter can see. §1-§5 add the fleet reliability strip (n, err%, latency, cost) from reviewers_telemetry. Never uses IonWarp's own self-grade as a quality verdict — that number counts findings, and volume is not quality. Read-only — never launches reviews, never writes pins, never posts to GitHub. category: analytics display_name: "Review Study" tagline: "How good were our reviews — and better than the competition?" distribution: public triggers: - pattern: "Grade the reviews on {repo}#{pr}" examples: - "How good are our reviews" - "Grade the reviewers" - "Reviews grading" - "Reviewer telemetry digest" - "IonWarp vs CodeRabbit" inputs: project_id: type: string required: false description: >- Dash project id (pr_...) to scope the fleet strip. Defaults to every project in the workspace. instructions_file: SKILL.md references: - references/render_digest.py - references/grading-rubric.md - references/judge_call.py - references/validate_snapshot.py default_backend: # The labs are ai-sdk orchestrations (MCP tools: tasks_create, skills_get, # data_query) — NEVER Modal/OpenCode. Loop-1 measurement (2026-07-31, # run_ms9grr1ztk5ami): with no default_backend the platform routed this skill # to Modal, where it died at the 1740s function ceiling with zero digest. platform: cloudflare harness: ai-sdk provider: openrouter model: deepseek/deepseek-v4-flash-0731 execution: # Cheap, capable default for the study run itself; the operator can override. # (The JUDGE model in §0.5 is a separate, deliberately cross-family choice.) model: openrouter/deepseek/deepseek-v4-flash-0731 # Explicit 2026-08-31 (flash-lane repin plan R4.1): none = request no # reasoning — wire-identical to the previous absent value, but the pin now # STATES it instead of delegating output volume to a provider default. reasoning_effort: none --- # Review Study — are our reviews any GOOD, and better than the competition? This skill never starts a run, never edits a pin, never posts to GitHub. **Three modes, one contract.** - **§D — the per-review digest.** ONE review, rendered as it lands: latency, the reviewers that ran, and a read of what they actually said. This is the short-cadence mode (`#ionwarp-agent`, every ~20 min) and the on-demand "grade review 1431" mode. - **§0 — head-to-head quality grading on ONE real PR.** Read what each review bot actually WROTE, and judge it ourselves — IonWarp vs Greptile vs CodeRabbit. This is the answer to "are IonWarp's findings good?" - **§1-§5 — the fleet reliability strip.** Volume, error rate, latency and cost per lens × model from `reviewers_telemetry`. **Why §0 exists.** IonWarp's own `derive_reviewer_grade` is `3×critical + 2×major + 1×minor`, thresholded at 5 and 3. That is a **finding counter**. A reviewer that finds one real critical bug and a reviewer that emits five nits both grade **A**; a reviewer that correctly approves clean code grades **C**; and a reviewer that misses an exploitable bug is graded on what it *did* say, never on what it *should* have. It cannot see a false positive and it cannot see a miss. Reporting it as "grading" was a green, punctual, well-formatted digest measuring the wrong thing twice a day — which is worse than measuring nothing, because it manufactures confidence. So: **never quote the product's own grade as a quality verdict.** §1-§5 keep the telemetry that IS a real measurement (did it error, how long, how much) and drop the letter grade. The API contract lives at `$IONWARP_API_URL/docs/raw/guides/model-lab.md` (the raw markdown endpoint). MCP server `ionwarp` (tools: `projects_list`, `tasks_list`, `tasks_status`, `whoami`). ## The closed read set The only reads this run makes are the ones its sections name: `whoami` (Step 0), `projects_list` (Step 1), `tasks_list` + the REST status route (Step 2), `gh` (§0), and the state-dir ledgers. An external operator agent (running this skill from the public docs) holds the same closed read set — orientation beyond it is a round trip that returns what you were already told. ## Runtime contracts (state dir · delivery · bands) **State directory.** The harness provides a durable state directory via env `IONWARP_LAB_STATE_DIR`; every ledger lives at `$IONWARP_LAB_STATE_DIR/`. This skill owns `graded-prs.json` and the `YYYY-MM-DD-HHZ.json` snapshots (`skill.reviewer-model-lab` reads the snapshots and shares the same dir for `trials.json`/`blocked.json`). **Always resolve it absolutely** — a bare relative path resolves against the session CWD and silently finds nothing; a first live lab run once ran ledger-less this exact way and never noticed. A state dir that can neither be read nor created is a LOUD RED `ledger_unreadable` line in the digest — never assume, infer, or carry a number forward without reading it. **Delivery.** The digest is the run's FINAL MESSAGE; the harness (a Slack cron, a platform task, a chat surface) delivers it — this skill never posts to Slack or GitHub itself. First character `🧪`, compose once, stop: nothing before it, nothing after it. Reasoning belongs in tool calls or a scratch file, never in prose ahead of the digest. **Band authority.** Each reviewer skill declares its `model_bands` in its own SKILL.md frontmatter (`aa_coding_index.min`, `input_usd_per_mtok.max`, `output_usd_per_mtok.max`, `latency_s.{min,max}`, `throughput_tps.min`) — read them via `skills_get` this run, never from a table baked into this file. The VALUE band — **$/CM ≤ $0.50 per critical-or-major finding** — is shared across all lenses. Over the RATE band alone is never a repin signal by itself; the VALUE column is the primary, and only failing BOTH is the runaway shape. ## High-Signal Findings (HSF) A High-Signal Finding is a DELIVERED finding with severity `critical` or `major` — after dedup and the min-severity partition — that names a concrete defect with evidence (file:line + failure scenario) the team would act on before merge. Style nits, diff restatements, hypotheticals without a failure path, and findings on unchanged code are never high-signal. HSF is the value unit reviews are measured by (HSF per dollar, HSF per MB reviewed). When grading, classify each delivered finding against the measured noise taxonomy — these five classes accounted for most of the false/inflated findings in production windows, and a finding in any of them grades DOWN even when its prose is polished: 1. **Coverage-ask-as-major** — "X has no test" escalated on a conditional narrative ("if it were mis-wired…") with the code not shown wrong. 2. **Delegated verification** — "verify that the index/config/schema exists" handed to the author when the reviewer could open the file. 3. **Hedge with no failure path** — "consider X if Y becomes a concern". 4. **Cross-lens duplicate** — the same root cause re-filed by a second lens (often a doc lens restating a code finding from the plan file). 5. **Unread-premise assertion** — a claim about a file the review's own coverage block lists as not reviewed. --- ## Platform execution notes (measured 2026-08-01) On the platform (ai-sdk backend), BOUND every sweep: sample at most ~12 tasks from the window, keep the digest inside its 20-line budget, and never re-read full transcripts when telemetry fields answer the question. An unbounded read outlives the queue-backed invocation (DLQ, `stream_transport_failed`) or trips the transcript persister's truncation (`part_truncated_at_persist`), which drops your FINAL message — the digest. Bounded runs complete in ~15 minutes. ## §D Per-review digest — ONE review, posted as it lands The mode a chat harness runs on a short cadence (every ~20 min) or on demand ("grade review 1431"). §0–§5 answer *"are our reviews any good this week"*; §D answers *"what just happened on this PR, and was it worth the money"* while the PR is still open in front of you. **One message per review, and the numbers are NOT composed by hand.** `references/render_digest.py` renders the fixed half from `reviewers_telemetry`; you add the read. A reviewer table is exactly the shape a model rounds, reorders, or quietly drops a row from, so those numbers come out of the telemetry or they do not appear. ```bash curl -sS -H "Authorization: Bearer $IONWARP_API_PAT" \ "$IONWARP_API_URL/api/v1/tasks/$TASK_ID/status" \ | python3 references/render_digest.py ``` ### D.1 Which reviews `tasks_list` for the window since your last post, `skill_id: ionwarp-review`. **Post every terminal review in the window, including the FAILED ones.** A failed review is the whole point of the cadence: on 2026-08-10 fourteen dash PRs were refused at the spend cap and two of a NEW customer's first five reviews died `review_diff_unavailable`, and because nothing rendered them, "IonWarp stopped reviewing" was invisible for the rest of the day. `render_digest.py` renders a failure block with the named code rather than exiting — never skip a review because it produced no table. Post each `task_id` exactly once. The ledger of what you have already posted is the state dir (`posted_reviews.json`), not your memory of the last run. ### D.2 The read — the half only you can write Fetch what the reviewers actually WROTE and judge it: ```bash gh api "repos/$REPO/pulls/$N/comments" --paginate --jq '.[] | select(.user.login|test("ionwarp")) | {path,line,body}' gh api "repos/$REPO/issues/$N/comments" --paginate --jq '.[] | select(.user.login|test("ionwarp")) | .body' ``` Then append, under the table: - **One line per finding**, in the reviewers' own severity order: `P1 · — `. A verdict of "can't tell" is a legitimate and frequent answer; inventing confidence is not. Judge against the DIFF, never against the finding's own confidence badge — the badge is the reviewer grading itself. - **One closing line, ≤20 words**, that a human reads instead of the table: what this review was worth. `"Two real races in the hover layer; the security P1 is a genuine cross-employee leak. Worth it."` — not a restatement of the counts. **Never quote the letter grade** (the reason is at the top of this file), and never call a finding real because the reviewer sounded sure. ### D.3 Shape Headline, scope, latency, table, ⚠️ lines, the per-finding read, the closing line. Nothing else — no preamble, no "here is the digest", no tool narration. The render script emits everything above the read; you never retype its numbers. ## §0 Head-to-head grading on one real PR (the primary output) ### 0.1 Reaching GitHub Use the **`gh` CLI**, authenticated through its own credential store (`gh auth status` to confirm). **Do NOT depend on `$GITHUB_TOKEN`**: some harnesses strip auth variables from the terminal environment, so a script reading it gets an empty string and fails confusingly. `gh` needs no env var here. All reads are read-only and paginated. Keep each call well under the terminal timeout by fetching one PR at a time. When calling any IonWarp/Cloudflare endpoint from a script, always send a real `User-Agent` header — Cloudflare bot protection 403s UA-less requests. ### 0.2 Pick the PR — it must be able to settle something Candidates: open + recently-merged PRs in the project's connected repositories, last 7 days. For each, count inline review comments per bot login (bot = login ends `[bot]`/`-ai`/`-bot`, or is one of `ionwarp-staging`, `coderabbitai`, `greptile-apps`, `sourcery-ai`): ```bash gh api "repos/$REPO/pulls/$N/comments" --paginate --jq '[.[].user.login] | group_by(.) | map({login: .[0], n: length})' gh api "repos/$REPO/issues/$N/comments" --paginate --jq '[.[].user.login]' ``` **Requirement: at least TWO bots produced ≥1 inline finding.** A PR only IonWarp reviewed cannot settle a comparison — say so and pick another. Prefer, in order: most bots represented → largest security surface → biggest diff. Skip any PR already graded in the last 7 days (check `$IONWARP_LAB_STATE_DIR/graded-prs.json`), and NAME the PR you picked and why in the digest. Reality check (measured 2026-07-24 on the original fixture repos, so the selection rule is not a surprise): over the last 8 PRs in each of two repos, Greptile posted inline findings on 3 of 8 in one and **0 of 8** in the other; CodeRabbit posted real findings on **1 of 16**. Often only ONE repo effectively qualifies. If nothing qualifies, say "no PR this week had two bots with findings" and run §1-§5 alone — never grade a one-bot PR and call it a comparison. ### 0.3 Collect what each bot actually wrote ```bash gh pr diff "$N" -R "$REPO" > pr.diff gh api "repos/$REPO/pulls/$N/comments" --paginate > inline.json gh api "repos/$REPO/issues/$N/comments" --paginate > issue.json gh api "repos/$REPO/pulls/$N/reviews" --paginate > reviews.json ``` Group every comment by bot login into one markdown blob per bot, keeping `path`, `line` and the full body. **"No comments" is not a grade of F — find out WHY.** CodeRabbit in particular is usually rate-limited, not silent: its comment says *"you've reached your PR review limit … your organization has reached its usage spending cap."* A bot that was never allowed to run is `did not review (usage cap)` — a NAMED state, excluded from the grade table, never scored. Scoring a bot on a review it was forbidden to perform is the same defect class as scoring a model on a lens our own timer killed. ### 0.4 Normalise for coverage before comparing IonWarp discloses how much it read (e.g. *"20 of 61 files reviewed"*); the others generally do not. Record each bot's covered file set where stated, and **grade on what each bot actually read.** A bot that reviewed 20 files and one that reviewed 61 are not comparable on raw finding counts. Two corollaries, both learned on a real graded PR: - Deferring files to stay in budget is a **budget decision, not a quality failure** — do not score it down as a miss. - **Disclosing coverage is a quality CREDIT.** IonWarp states its coverage and withholds findings it cannot support (`unfounded_absence_claim`); the competitors generally offer neither. Say so. ### 0.5 Grade — our rubric, a cross-family judge, a PRICED call Judge with an OpenRouter model from a **different family** than the reviewer under test (a sonnet judge over-rates a sonnet reviewer). Dispatch it with **`references/judge_call.py`** (path relative to this skill's directory) — one call, priced, cost recorded: ```bash python3 references/judge_call.py \ --prompt-file /tmp/grading/judge_prompt.txt \ --model anthropic/claude-opus-5 # cheaper alternative: anthropic/claude-sonnet-5 # -> judge_call: OK model= cost_usd=$ in= out= gen= # -> /tmp/grading/judge_result.json {judge_model, generation_id, cost_usd, content} ``` The script sends `"usage": {"include": true}`, so `cost_usd` is OpenRouter's **actual charge for that generation**, auditable later by `generation_id`. It reads `OPENROUTER_API_KEY` from the environment, with an optional `OPENROUTER_KEY_FILE` (a .env-style file) fallback for harnesses that strip auth variables from the terminal environment — the same trap as `$GITHUB_TOKEN` in §0.1; one early run hit it as a raw `KeyError` and had to recover mid-flight. **If the dispatch fails** (exit 1: no key, 401, timeout), judge in-context and record `"cost_usd": null` with a `cost_note` — §0.7. **Never write `0.0`.** A judge that ran inside this session cost real money buried in the session total; `0.0` is not that number, it is the absence of that number dressed as a fact — and it is what once made "is grading worth its cost" unanswerable for four days. Give the judge the diff plus one blob per bot, and this rubric: | Category | Weight | What earns an A | |---|---|---| | **Security** | 30% | caught an exploitable issue proportional to the diff, with vector + fix. `N/A` when the diff has no security surface — redistribute, never penalise | | **Correctness** | 30% | real bugs with file:line, the bad case reproduced, a fix that works | | **Actionability** | 20% | every comment fixable without a clarifying question | | **Signal-to-noise** | 20% | >80% substantive (bugs, security, missing tests) vs nits (whitespace, formatting, style) | The full judge prompt — scale definitions, per-category A/B/C/D/F anchors, the anti-pattern list and the output envelope — is `references/grading-rubric.md`. Hand it to the judge verbatim. Then a **coverage multiplier**: ×1.0 commentary scales with the diff · ×0.8 the same N-comment template regardless of scope · ×1.1 found issues *adjacent* to the diff (e.g. flagged that a caller will break). Scale: **A+** exceptional (critical issue + working fix, or a subtle catch nobody else made) · **A/A-** strong · **B** solid, some noise · **C** weak, mostly nits or template · **D** template output ignoring the diff · **F** harmful (bad advice, misinformation). Penalise, explicitly: identical boilerplate on every PR · the same finding repeated across 5 files (count ×1, not ×5) · restating the diff instead of analysing it · flagging whitespace as high severity · **claiming to have run tools it cannot run** · a confidence verdict contradicting its own findings (a real bot once printed *"Confidence 5/5, safe to merge"* on the same commit where it filed two P2 security findings — that contradiction is itself a finding). **Per bot, write four things:** volume (N comments, M substantive) · **biggest hit** · **biggest miss** — a realistic thing it should have caught, judged against the diff — · and the pattern (useful / noisy / template-driven). **The biggest miss is the whole point.** It is the one judgement a counter can never make, and it is why this section exists. ### 0.6 Disagreement is the most valuable output Where two bots found the **same class** of bug by **different vectors**, say so explicitly — that is a precision/recall comparison no table can show. On one real PR both bots found SSRF in a callback URL check: one via IPv4-mapped IPv6 (`https://[::ffff:169.254.169.254]/`), IonWarp via dotted-decimal / DNS rebinding (`https://2130706433/`). Each vector is the other's miss. **That paragraph is worth more than the whole grade table** — lead the digest with it when it exists. ### 0.7 Persist Append to `$IONWARP_LAB_STATE_DIR/graded-prs.json`: `{repo, pr, graded_at, bots:{login: overall}, biggest_miss_per_bot, judge_model, cost_usd, cost_note}`. This is the 7-day dedupe key and the trend line. **`cost_usd` has exactly two honest values.** Copy them from `judge_result.json` — do not retype a number from memory: ```jsonc // dispatched (§0.5) — a measurement "judge_model": "anthropic/claude-opus-5", "judge_generation_id": "gen-1785114589-CPJzty1vVNlwVDG5NWIR", "cost_usd": 0.013210, "cost_note": "openrouter usage.cost — provider-billed for this generation" // judged in-context (dispatch unavailable) — an honest absence "judge_model": "self (GLM-5.2, cross-family vs deepseek-v4-pro-0813 reviewer)", "cost_usd": null, "cost_note": "judge ran inside the main agent context; its cost is real but included in this session's total and never measured separately" ``` `0.0` is **banned**. Four PRs graded in one early window all recorded `cost_usd: 0.0` — three judged in-context, and one which *did* dispatch a real judge over OpenRouter but asked for no usage, so its real charge was thrown away and cannot be reconstructed. Four zeros, none of them a measurement. A null says "not measured" and can be fixed; a zero says "free" and cannot even be noticed. --- ## §1-§5 — fleet reliability (secondary) **GATE — run §1-§5 only when the window holds ≥3 terminal reviewer executions.** Check that first. Below the threshold, skip §1-§5 entirely, write the snapshot with `"table_rows": [], "stale": true, "recommendation": []`, and render ONE line in the digest: `➖ Fleet: executions in window — no table, no recommendation`. Never re-emit the previous window's `recommendation` when this window is empty. That is not a no-op: `skill.reviewer-model-lab` consumes `recommendation` and cannot tell a fresh call from a stale copy. Three consecutive snapshots once carried `table_rows: []` with a verbatim-repeated pseudo-recommendation, and the lab re-trialed that same triple four days running. An explicit empty array plus `stale: true` lets the lab say "no fresh recommendation" and spend the run on something else. §0 is NOT gated — it runs every day. It is the half that produces new output. ## Step 0 — whoami first Call `whoami` on the ionwarp MCP. A 401/unreachable MCP is a LOUD RED line in the digest ("IonWarp MCP unreachable () — NO grading this run") and the run stops. Never render a table you did not fetch. ## Step 1 — projects and the window `projects_list` → every project in the workspace; a project with no review traffic gets one named line (": no reviews in window"), never a silent absence. The window is SINCE THE PREVIOUS SNAPSHOT, capped at 12 hours: window start = the newer of (newest `$IONWARP_LAB_STATE_DIR/*.json` snapshot timestamp, now − 12h). No snapshot → plain last 12h. A review graded twice is double-counted noise; a review skipped between runs is data loss — the cap plus the snapshot timestamp prevents both. The header states it: `window: since · 12h`. ## Step 2 — pull the reviews Per project: 1. `tasks_list {project_id: ""}` (MCP; `project_id` is the ONLY accepted arg — there is no `limit` arg, the server caps the page itself) → keep tasks with `skillId == "ionwarp-review"`. The list rows carry NO timestamps (`{id, title, status, skillId, agentId}` only), so window membership CANNOT be decided from the list: status-fetch each candidate newest-first and stop after the first ~10 whose terminal time falls before the window start. Any terminal status counts — failed runs are data. **MCP is the ONLY PAT-visible task list** — REST `GET /api/v1/tasks` is session-cookie-only and answers `Unauthorized` to a PAT; that 401 is not your credential. 2. For each kept task, REST status (the PAT works here): ```bash curl -sS "$IONWARP_API_URL/api/v1/tasks//status" \ -H "Authorization: Bearer $IONWARP_API_PAT" ``` The grading source is `reviewers_telemetry.reviewers[]`: `skill` (the lens), `model` (what ACTUALLY ran), `grade`, `verdict`, `findings {critical, major, minor, total}`, `cost_usd`, `input_tokens`, `output_tokens`, `duration_ms`, `usage_estimated`, and `error {error_code, error_message}`. **Fallback (named, never silent):** a task whose status carries no `reviewers_telemetry` counts as `telemetry_absent` in the footer. GitHub PR comments may sanity-check a suspicious table, NEVER as a number source. ## Step 3 — build the per-lens × model table One row per `(skill, model)` pair over every reviewer execution fetched (all projects pooled; render a small per-project count line so one project cannot silently dominate). Columns, all computed from rows you actually fetched: | col | definition | |---|---| | n | reviewer executions | | err% | rows with an `error` object present | | C+M/run | mean(critical + major) per execution | | tot/run | mean(findings.total) per execution | | zero% | share with findings.total == 0 | | lat(s) | median duration_ms, rendered as seconds | | cost($) | median cost_usd over rows where cost_usd is non-null | | $/Mtok-in | median of `cost_usd ÷ (input_tokens + cache_read_input_tokens + cache_creation_input_tokens) × 1e6` | | $/CM | median cost_usd ÷ C+M/run | Header names are part of the contract — render them exactly as above. `cost_usd: null` is NEVER $0.000 — count those rows and render the gap (`usage_absent ×k`) beside the median. `usage_estimated: true` rows get a `~` prefix on the cost. **`$/Mtok-in` is the column model decisions are made on — raw `cost($)` is not.** Per-run cost is dominated by the SIZE of the PR the lens happened to draw, so comparing it across lenses compares their PRs, not their models. `$/Mtok-in` is stable per model regardless of scope. This is not theory: through one whole measured window, a security lens sat inside a $0.25/run band while costing $3.30/Mtok-in — 10× the fleet — and burning 40% of the bill. The per-run band could not name it, and neither could this table without the rate column. Keep `cost($)` (it answers "what did this run cost"); make `$/Mtok-in` the one you rank on. The bands are canonical in each reviewer skill's own frontmatter `model_bands.input_usd_per_mtok.max` — read them with `skills_get` this run. Mark any row over its band max in the table with a `!` — but remember: over the RATE band alone is not a repin signal; the VALUE column (`$/CM` ≤ $0.50, shared across all lenses) is the primary, and only failing BOTH is the runaway shape. **An over-VALUE-band LENS is a lens finding, not a model finding — say so.** The labs have exactly one lever, *which model runs a lens*, and it is the wrong lever for a lens whose output cannot block. When a row is over `$/CM` **and** the lens's findings are structurally demoted out of the verdict (the docs-rank gate, e.g. `plan-review`), render the named line `lens_cannot_buy_blocking (: runs, C+M, $)` and state in the same breath that **no model swap can move it** — the ceiling is the policy, not the pin. Do not emit a repin recommendation for such a row. Measured over 400 production runs (2026-08-23): `plan-review` ran **153** of them for **4** C+M at **$0.6151/CM** — over the $0.50 band — and the two recorded interventions on that lens were both *price-band narrowings*, a model-side response to a lens-side problem. `seo-review`: **21 runs, 0 C+M**. Meanwhile `code-review` + `security-review` produced **358 of 366** C+M in the window (98%) for 50% of the spend. A cadence that can only ever propose a model is a cadence that cannot report that. **`$/CM` — dollars per critical-or-major finding — is the "is it worth it" column.** It is the HSF cost rate (see High-Signal Findings above): dollars per delivered critical-or-major. A lens with no C+M in the window has an infinite rate: render it as `no CM yet, $X spent`, NEVER as `∞`, `—`, or a blank, all of which read as "nothing to see". "Spent $X and found nothing worth flagging" is the sentence that should be impossible to miss. **Denominator gaps are named, never zeroed.** A row whose token counters are all absent has NO `$/Mtok-in` — render `tok-absent` and count it beside `usage_absent`. A zero denominator would render an infinite rate and read as a catastrophic model, which is the opposite of "we do not know". (Older telemetry rows may legitimately lack the token counters.) **Cache-efficiency line (one line, under the table).** Per lens, the blended `$/Mtok-in` against the pin's LIST input price (`pricing.prompt` from OpenRouter `GET /api/v1/models`, no auth). Blended well UNDER list = caching is working; blended AT or ABOVE list = every run is paying full price for its whole prompt. This is the one-line answer to "is caching working", and it is the measured difference between a lens reading 80k–266k cached tokens/run for pennies and one reading 0 cached tokens while rewriting its entire 60–252k-token prompt every run. **No `A/B/C` or `A%` columns — ever.** They would render `derive_reviewer_grade`, a finding counter presented as a quality score. Keeping it in a table headed "grading" is exactly the decoration this skill exists to replace: the quality verdict comes from §0, where we read the comments ourselves. Every column above is a real measurement of whether the reviewer RAN (did it error, how long, how much, did it find anything at all) — none of them claims the findings were good. **A lens with `err% == 100` and zero gradeable runs is OUR defect, not the model's.** Report it as structurally failing and print the median latency: a value sitting on a round number is one of our own clocks. A real lens once read `err% 100, n 1, cost null, lat 25.083s` — that is a 25,000ms browser fast-fail plus 83ms, not the model. Never let such a row become a model-swap recommendation. ## Step 4 — compare, then persist the snapshot Read the newest `$IONWARP_LAB_STATE_DIR/*.json` snapshot (excluding the named ledger files `graded-prs.json`, `trials.json`, `blocked.json`; none: "no baseline yet"). Then write this run's snapshot as `$IONWARP_LAB_STATE_DIR/YYYY-MM-DD-HHZ.json`, carrying: - `table_rows` — per (skill, model): n, fail_pct, cm_per_run, tot_per_run, zero_pct, med_latency_ms, med_cost_usd, usage_absent_rows, **`usd_per_mtok_in`** and **`usd_per_cm`** (both `null` when the denominator is absent — never 0, never a string); - `window_start` / `window_end` (UTC) and the task ids per project; - **`recommendation`** — the 🎯 stack as machine-readable rows `[{lens, model, reason}]` (or `[]` when there is none), so skill.reviewer-model-lab consumes it without parsing digest text. **`model` MUST be an exact `org/model` OpenRouter id of the model to TRIAL** — the lab passes it verbatim into `input.reviewers`. "Keep the current stack" is `recommendation: []`, never a row; pseudo-ids like `keep-x-ai/grok-4.6` or `consider-alternative-to-…` are prose wearing a field's clothes and once broke the handoff for two days (the lab re-trialed a string that is not a model). The validator rejects them. `recommendation`, `window_start`, and `window_end` are REQUIRED keys — the model lab reads them; a snapshot missing them breaks the handoff. Keep the newest 30 snapshots; delete older ones. ### The mandatory-recommendation rule **A lens with `zero_pct ≥ 75` at `n ≥ 9` whose `usd_per_mtok_in` is over its band max MUST emit a `recommendation` row.** Finding nothing and being the most expensive lens may never again coexist with `recommendation: []`. This closes the exact gap that let one expensive lens burn ~$50/30d unchallenged. A prior digest printed *"security-review has not found a single issue in 9 runs; green by absence"* — and the same snapshot shipped `recommendation: []`. So the model lab never trialed a replacement, and a human had to notice by hand. **An observation with no handoff row is an observation that does not act**, which is the same failure as not having made it. Choosing the challenger, in order: 1. the cheapest model already IN band that has C+M evidence on an ADJACENT lens (it has proven it can find things here, on our own data); 2. otherwise `deepseek/deepseek-v4-pro-0813`, the default challenger. The `reason` states the three numbers that triggered the rule (`zero_pct`, `n`, `usd_per_mtok_in` vs the band max) — never a hunch. The rule is a FLOOR, not a cap: emit recommendations for other reasons whenever the data supports them. The rule NEVER fires on a lens whose problem is ours — `err% == 100`, or a median latency sitting on one of our round numbers. A model that was never allowed to execute has not failed. **Then run the validator — this is a step, not a suggestion:** ```bash python3 references/validate_snapshot.py \ "$IONWARP_LAB_STATE_DIR/.json" ``` **Exit 1 means the snapshot does not ship.** Fix the file and re-run until it prints `validate-snapshot: OK`; if it still fails, say so as a LOUD RED line in the digest rather than leaving a broken handoff on disk. The rule it enforces is one sentence: **an explicit empty array is a statement; a missing key is a silence.** `"table_rows": [], "stale": true, "recommendation": []` — the gate's own output for an empty window — is VALID and passes. An omitted key is not. Why a script and not a sentence: this section said REQUIRED from the start, and the four earliest snapshots ever written carried none of the three keys anyway (three also carried no `table_rows`). Convention fixed it for a while; nothing enforced it, so nothing stopped it coming back. The validator also re-checks the §1-§5 gate's other half — an empty `table_rows` beside a non-empty `recommendation` is the stale copy that had the lab re-trialing the same triple four days running. Comparability rule: deltas compare this window's rows to the previous snapshot's (adjacent, non-overlapping windows), and a delta is only reported when n ≥ 3 on both sides; smaller rows are rendered but labeled `(n<3 — not comparable)`. ## Step 5 — the digest (≤20 lines, the run's final message) **Delivery: the final message IS the digest** (see Runtime contracts above) — first character `🧪`, reason in tool calls never in prose, compose once, stop. This skill's fixed second line: `What this is: a daily grade of yesterday's PR reviews — ours vs the other bots — plus did-the-reviewers-run stats.` **Combined-empty rule:** when the §1–§5 gate fired (no gradeable executions) AND §0 found no qualifying PR to grade, the digest is a SHORT named-failure message — ≤6 lines: the window, why each half is empty, and what unblocks it — never a full ceremony of empty tables and carried-forward numbers dressed as data. **Formatting (the common delivery surface is Slack mrkdwn — compose for it).** Use `**bold**` (DOUBLE asterisks — a bare `*x*` renders as _italic_ on many delivery paths), `` ``` `` code fences (monospace — the ONLY way the wide grading table aligns), and `` links — but NOT inside a fence, where links/bold render literally. So: **bold + emoji on every header**, the **per-lens × model TABLE inside a ` ``` ` fence**, the analysis lines and the 🎯 handoff as mrkdwn below it. Plain English first, numbers second; every analysis line opens with a sentence a non-operator understands. Task-id lists go to the thread/appendix block, never the main message. Imitate exactly: 🧪 *How good were our reviews?* — · graded #: "" *TL;DR* · <2–3 plain sentences a non-operator understands: who won, the single most interesting disagreement, and the ONE thing worth doing. Or "Nothing needs you."> ⚖️ *Head-to-head on #* — ``` bot security correctness actionable S/N cover OVERALL comments ionwarp A+ A- A A- ×1.0 A 10 (9 substantive) greptile A B B+ B ×1.1 B+ 2 (2 substantive) coderabbit — — — — — did not review (usage cap) ``` _Higher is better. `—` = not applicable or not run. `cover` = coverage multiplier._ 🥊 *Where they disagreed* · 🎯 *Biggest hit* · : 🕳️ *Biggest miss* · : 🔗 · · 📊 *Fleet reliability* (last ) — did the reviewers RUN, not how good they were ``` lens model n err% C+M/run tot/run zero% lat(s) cost($) $/Mtok-in $/CM code-review grok-4.6 12 0% 0.42 1.8 17% 210 0.021 1.81 0.050 security-review deepseek-v4-pro-0813 9 0% 0.11 0.3 89% 14 0.009 0.33 0.082 ``` _Legend_ · n = runs · err% = errored · C+M/run = critical+major per run · tot/run = all findings/run · zero% = found nothing · lat(s)/cost($) = median · $/Mtok-in = cost ÷ (input + cache_read + cache_creation) — the scope-normalized rate models are compared on · $/CM = cost per critical-or-major · ~ = estimated · `!` = over band 💾 *Caching* · 🚨 *Structurally failing* · 🕳️ *Finding nothing* · **Every lens named here at `n ≥ 9` and over its $/Mtok-in band MUST also appear in 🎯 below — that pairing is the mandatory-recommendation rule, and a 🕳️ line with no matching 🎯 row is the exact gap that let the most expensive lens run unchallenged for a month.** ⚠️ *Gaps* · 🎯 *Worth trialing* · )".> Notes: the head-to-head grade is OURS, from reading the comment text against the diff — never IonWarp's own score, which counts findings and cannot see a false positive or a miss. A bot that was rate-limited is "did not review", never a grade. skipped projects: single-PR dominance: 50% of n> fetch errors: **Never wrap the whole digest in one fence** — it kills the 🎯 handoff links and every bold. Fence ONLY the table. Omit the notes/appendix block only when there is nothing to put in it — then Gaps reads "Gaps: none". House rules (hard): - An unreachable source is a LOUD RED line, never a silent skip; every count comes from data fetched THIS run. - The TL;DR, the legend, and the ⚠️ volume-not-verified-quality caveat are mandatory in the MAIN message, every digest — the caveat is never demoted to the appendix. - Each 📉/📈/🕳️ line is one plain-English sentence first, numbers second. - The 🎯 line ENDS the main message and is its point: the explicit handoff to skill.reviewer-model-lab. Every recommended change traces to numbers in this digest — never a hunch; a model with no data on that lens says so ("untested on this lens; wins on 's numbers"). "Keep the current stack" is valid when the data says so. This skill still writes no pins. - The digest labels its own blind spots: telemetry gaps, small n (`(n<3 — not comparable)`), single-PR dominance (×N in the table). ## Verify (self-assert) - Every project from `projects_list` appears — graded or named-skipped; table row totals sum to the reviewer executions fetched. - The digest opens with 🧪 (window stated), then TL;DR; the legend sits under the table; the ⚠️ caveat is in the main message; the main message ends with the 🎯 line; the digest is ≤20 lines before the appendix block. - Every row came from a run terminal INSIDE the window — nothing double-graded, nothing dropped between runs. - Every 🕳️ *Finding nothing* lens at `n ≥ 9` and over its `$/Mtok-in` band max has a matching `recommendation` row (the mandatory-recommendation rule). A lens excluded because the failure is OURS (`err% 100`, a round-number latency) says so on its own line. - Every `table_rows` entry carries `usd_per_mtok_in` and `usd_per_cm` — a real number or an explicit `null`. A row with token counters but no rate is a computation that was skipped. - A snapshot for this run exists in `$IONWARP_LAB_STATE_DIR/` carrying `window_start`, `window_end`, and the `recommendation` rows (possibly empty) mirroring 🎯 — and `references/validate_snapshot.py ` printed `validate-snapshot: OK`. A snapshot that was never validated is not done. - The `graded-prs.json` entry for this run carries either a measured `cost_usd` copied from `judge_result.json` (with its `generation_id`), or `null` plus a `cost_note`. Never `0.0` — an unmeasured cost is written as unmeasured. - No Slack or GitHub post made directly by this skill; no PAT in output; the final message is the digest and nothing else.