Today's podcast

Everybody Shipped A Judge

Daily field notes from the agentic frontier.

Opus 5.5 ships with external referees, GPT-6 Sol and Luna halve frontier prices, and open-source decision models put a judge on every CPU.

September 25, 2026 Agentic AIAI Infrastructure
Evy welcomes you to her daily AI briefing beside her microphone and open notebook
Listen with Evy

Evy's Morning AI Brief #102

“Signal over noise in agentic systems.”

Episode article

Notes and transcript

Evy’s Morning AI Brief #102 — September 25, 2026

The frontier labs shipped their biggest models of the season this week — and for the first time, one arrived with outside referees attached. Meanwhile, a quiet wave of open-source decision models landed that want to do the judging themselves. The referee layer is becoming infrastructure.

The Ledger

Claude Opus 5.5 arrives with named external evaluators. Anthropic’s first release since calling for “pacing the frontier” performs at the level of Claude Fable 5.1 on most work while costing 40% less than Opus 5 ($4/$20 per Mtok, 1M-token context). One early tester completed a 680,000-line code migration in under a day. Artificial Analysis ranks it #1 of 211 models on its intelligence index. The detail that matters: Anthropic says it was tested before release by external evaluators including Frontier Design and METR — and scored strongest-to-date on their most comprehensive behavioral audit. External pre-release evaluation is the small-verifier pattern done right; the open question is auditor independence, since the lab chose its own checkers. → Anthropic · Artificial Analysis

GPT-6 Sol and Luna trigger a frontier price war. OpenAI released both within the hour of Opus 5.5. Luna lands at $0.10/$0.50 per Mtok with 1.05M context; Sol at $2/$10. Simon Willison’s analysis: Luna is now among the cheapest models that remain genuinely good, and GPT-5.6 Terra — the previous value pick — “just evaporated” as a choice. For agents this cuts deeper than chat: loops re-read context on every tool call, so halving token prices compounds across a 40-call session. With coding benchmarks converged, the frontier race is now a unit-economics race. → OpenAI · Simon Willison

A real-world CAD eval shows the gold-standard checking pattern. Partforge ran Opus 5.5 and GPT-6 Sol through 26 standing hard-CAD tasks the day they dropped (bearing seats, photo-described parts; 3 attempts each). Every attempt is checked three ways — geometry measures correctly (deterministic), the work process is sensible, and two independent reviewer models approve the renders — then a human reviews by hand. Verdict: Opus 5.5 came closest yet to beating their default at far lower cost, but Gemini 3.8 Flash stays the default on cost-per-good-part. Deterministic gates first, models second, human last. → Partforge

Model Releases

The open decision-model trio. Three teams shipped small models whose whole job is scoring agent actions:

  • CLM-8B (Contrastive LM) — open System One model that scores agent actions up to 9x faster than Jev. → MarkTechPost
  • AnyJev (Nokia) — training-free layer that turns any open LLM into a calibrated decision model. → MarkTechPost
  • GLiNER2.5-Decide (Fastino) — 340M parameters, open weights, runs on CPU. → MarkTechPost

When the judge runs on CPU, verification stops being a batch API call and becomes something you can afford on every step of the loop. The frontier shipped referees; open source shipped the referee factory.

ThinkingCap (Bottlecap AI) — tuned Qwen3.8-27B that cuts thinking tokens 37.2% at under a point of accuracy cost. → MarkTechPost

FLUX.3 Action (Black Forest Labs) — 7B open-weights world action model topping RoboLab-120; the image-gen lab walking into action-model territory. → MarkTechPost

Nemotron 3 Diarization (NVIDIA) — who-spoke-when as a standalone open model; the voice-agent component nobody wanted to keep hand-rolling. → MarkTechPost

Frameworks & Tooling

Magpie (yetone) — every agent’s model in one place: Codex on DeepSeek, Claude Code on Kimi, from the menu bar. 741★ in two days; model routing as a consumer utility. → GitHub

golive-skill (mikehasa) — takes agent-built products the last mile: hosting, database, domain, email, payments. 921★ in ~a day. The agent stack grows in the shipping direction. → GitHub

unreal-agent (Unreal Labs) — async-first agent harness; ~1,900★ since Monday. Most agent work is waiting — harnesses built around that reality are winning. → GitHub

  • CodexDesk — desktop companion for coding-agent workflows. → GitHub
  • ClaudePanel — desktop panel and toolkit for Claude and AI assistants. → GitHub
  • agent-jev — 0.6B System One decision model; the decision-model wave at hobby scale, <2 weeks after the original. → GitHub

Research Highlights

AgentKernel: The Trust-Native Agentic Operating System. Agents need OS-style isolation, capability boundaries, and resource accounting — not just better prompts. The harness-as-OS thesis gets its academic spine. → arXiv

ExplorationBench. Measures exploration in “verifiable alien worlds” — unfamiliar environments where memorized web knowledge can’t save you. Tests curiosity, not recall; open and checkable. → arXiv

Just Ask Jev. The calibrated decision model — trained to know what it knows — doubles as a zero-shot detector of misalignment in other models. Ask the judge, catch the liar. → arXiv

Quick Hits

  • LaunchVideo — paste a URL, an Opus 5.5 serverless agent writes and renders a product explainer (~100k tokens, no edits). Cleanest agent-ships-artifact demo this month; its HN thread drew 282 points. → Site · HN
  • Qwen-Planner-Agent — closed-loop “AI-for-AI” framework where agents build and refine mobile planner agents. → arXiv

The Takeaway

Everybody shipped a judge. Anthropic shipped referees with the model; OpenAI shipped a price that makes you re-run the math; open source shipped judges small enough to live on a CPU. When the frontier moves this fast, the durable advantage isn’t the model — it’s the checking.

Read the full article