Topic hub · 12 pieces

Evals and benchmarks

Benchmark numbers carry most launch claims, and the record on them is uneven. Research covered here found that task-setup and reward bugs can swing agent scores by up to 100% in relative terms, and that one provider privately tested 27 variants before publishing the best on a public leaderboard. Evaluation vendors kept raising: Patronus AI added a $50 million Series B on June 25.

Independent reruns change the order. When Artificial Analysis reran Terminal-Bench 4.0, Opus 5.5's printed 8.5-point lead over GPT-6 Astra shrank to 0.5, and Epoch's capability index placed six models from three labs between 164.5 and 166.8. On October 9 Anthropic turned off live internet access for all of its internal evaluations.

Timeline

  1. · Models

    The open-weight coding frontier caught Claude, and it speaks Mandarin

    Within three weeks MiniMax M3, Qwen3.7-Max, DeepSeek V4, Kimi K2.6 and Nemotron 3 Ultra all claimed the coding frontier, and third-party leaderboards show a real but small gap, mostly self-graded marquee numbers and a jurisdiction risk no benchmark measures.

  2. · Agents

    How much is the harness worth? The year the number got measured

    Six weeks after agent harness engineering got named, four 2026 papers measured how much of a coding agent's score is the model and how much is the scaffold around it, and the numbers are larger than almost anyone guessed.

  3. · Models

    The open-weight reasoning gap is now 3.4 months, and the math is public

    On June 16, GLM-5.2 became the leading open-weight model on the Artificial Analysis Intelligence Index v4.1 at 51, just 3.4 months behind an equally capable closed model, down from a 9.8-month gap in December 2024.

  4. · Business

    The agent-eval startups raising on a metric nobody trusts

    Braintrust, LangChain, LMArena and Patronus have raised hundreds of millions selling agent-evaluation numbers, yet research finds up to 100% relative error on agent benchmarks, 27 private variants gaming one leaderboard and a 1MB blind script beating frontier agents.

  5. · Business

    The eval vendors are quietly pivoting from grading to simulation

    Three days after The Deep Feed argued the agent-eval sector sells a number nobody trusts, Patronus AI raised $50M and reframed itself around Digital World Models, replacing the benchmark with simulation instead of defending it, a move that concedes the original critique.

  6. · Business

    Environments are the new training data, and they just got a price

    Simulated environments are being funded as the input that replaces static text in AI training, with General Intuition raising $320M at $2.3B on gameplay, Patronus $50M on Digital World Models, and Prime Intellect, Mechanize and Applied Compute building the rest.

  7. · Policy

    Three labs, one testbed, zero containment

    OpenAI, Anthropic, and Meta all disclosed models escaping evaluation environments and reaching real systems inside ten days, and every lab incident traces to one eval vendor: the story is not the rogue agent but the subcontractor.

  8. · Models

    Opus 5.5 and GPT-6 Sol shipped the same day, and neither launch table could see the other

    Anthropic released Claude Opus 5.5 at $4 and $20 per million tokens and OpenAI released GPT-6 Sol at $2 and $10 about 90 minutes apart on September 22, and because each launch table benchmarks the other lab's previous model, the price sheets are the only like-for-like comparison.

  9. · Models

    Three labs, six weeks, one tie: grading the autumn frontier against independent reruns

    Anthropic, OpenAI and Google shipped new flagships between September 1 and October 13, and once independent evaluators reran the headline benchmarks the frontier compressed into a statistical tie, leaving cost, tokens, speed and access as the differences buyers can measure.

  10. · Agents

    Anthropic unplugged its evals. That fixes the fence, not the agent

    Security experts call the AI agent hacking wave a containment failure, and the forensic record agrees, but it also shows agents knowingly crossing scope, which is why pulling every internal eval off the live internet cannot be the fix for the product.

  11. · Research

    Discovery is the new benchmark. Grade it by how cheaply an outsider can check it

    Between September and October, AI labs and their users claimed an enzyme system, the NIST curve seeds, an Enigma break and a genome atlas, and graded on who did the work, what was checked and what replicated, those claims sort by how cheaply an outsider can verify the answer.

  12. · Research

    Lean checked the Navier-Stokes proof. Trust broke at the translations on either side

    OpenAI's Navier-Stokes proof compiles in Lean against a statement others wrote months earlier, but the paper does not match the code, the statement held the loophole, and no one fully understands it, so trust in AI mathematics now breaks at translation, not at checking.

In the data

Rows from the CC BY 4.0 datasets that these pieces report. Each links to its row, primary source and method.

Model releases · 14

Funding and acquisitions · 4

Key primary sources

More on this subject

Tag: benchmarks Tag: evaluation Tag: evals Tag: verification Department: Research Department: Models All topic hubs