# Three labs, six weeks, one tie: grading the autumn frontier against independent reruns

URL: https://www.thedeepfeed.ai/posts/2026-10-15-autumn-frontier-scorecard/
Category: Models
Published: 2026-10-15
Author: the-deep-feed
Tags: frontier-models, benchmarks, anthropic, openai, google, evals, gpt-6, claude, gemini
Kind: deep

> Anthropic, OpenAI and Google shipped new flagships between September 1 and October 13, and once independent evaluators reran the headline benchmarks the frontier compressed into a statistical tie, leaving cost, tokens, speed and access as the differences buyers can measure.

## TL;DR

- Between **September 1 and October 13**, Anthropic shipped a full Claude 5.5 family plus Fable and Mythos 5.1, OpenAI shipped the GPT-6 family, and Google shipped one staged flagship, **Gemini 4 Argon**. Two full generations and a flagship, not three generations.
- Rerun by **Artificial Analysis**, Terminal-Bench 4.0 shrinks Opus 5.5's printed lead over GPT-6 Astra from **8.5 points to 0.5**. Astra's FrontierCode score falls from OpenAI's 53.3% to 47.0%. Google's own Terminal-Bench number reproduces within 0.3 points.
- Epoch's capability index puts six models from three labs between **164.5 and 166.8**, every 90% interval overlapping. Rank flips by domain: Opus 5.5 is first on AA's Intelligence Index and fourth on AA-Robotics, where Astra leads.
- The measurable differences are elsewhere: Opus 5.5 spends **260M tokens** and $5.98 a task to top the index, Astra 60M and $3.26, Argon $1.99 for the same 53 as Astra. Each lab now gates its top cyber capability behind a vetted tier.

On September 22, Anthropic's launch table for [Claude Opus 5.5](https://www.anthropic.com/claude-opus-5-5) put the model at 66.4% on Terminal-Bench 4.0 and OpenAI's three-week-old GPT-6 Astra at 57.9%, a lead of 8.5 points on a coding benchmark all three labs now print. When [Artificial Analysis](https://artificialanalysis.ai/models/claude-opus-5-5) ran both models through its own harness, Opus 5.5 scored 59.6% and Astra 59.1%. The lead was half a point.

That gap between the launch table and the rerun is the useful way to read the six weeks that just ended. Between September 1 and October 13, Anthropic, OpenAI and Google each put a new flagship into the market, and each published a benchmark table on which it led most rows. Independent graders have now had time to rerun the numbers. Their verdict is that the top of the market is a statistical tie, that the order depends on the task, and that the differences a buyer can measure have moved from capability to cost per task, token use, latency and who is allowed to use the most capable version.

This piece grades each lab's headline claims against the independent evaluations that could be opened and checked: Artificial Analysis, Epoch AI, ARC Prize, METR and the third-party testers named in the labs' own system cards. Every benchmark figure published by a lab about its own model is labeled as vendor-reported. The pricing story for OpenAI's mid-tier is covered separately in [GPT-6.1 Sol costs a fifth of Astra](/posts/2026-10-09-gpt-6-1-sol-prices-speed-above-astra/), and several of the safety holds below trace back to the [containment incidents that led Anthropic to cut its evaluations off from the live internet](/posts/2026-10-16-anthropic-unplugged-its-evals/).

# Six weeks, two families and a staged flagship

The starting hypothesis was that three labs shipped three full generations in six weeks. The calendar supports two of the three.

| Date | Lab | Release | List price per 1M tokens (in / out) | Access at launch |
|---|---|---|---|---|
| Sep 1 | Anthropic | Claude Fable 5.1 and Mythos 5.1 | $10 / $50 | Fable GA; Mythos to vetted US organizations |
| Sep 2 | Google | Gemini 3.8 Flash and 3.8 Flash Cyber | $0.75 / $3.75 (introductory) | Flash GA; Cyber via Fairwind |
| Sep 3 | OpenAI | GPT-6 Astra | $10 / $50 | Limited organizations, then paid tiers and API |
| Sep 22 | Anthropic | Claude Opus 5.5 | $4 / $20 | GA, cyber tasks rerouted |
| Sep 22 | OpenAI | GPT-6 Sol and GPT-6 Luna | $2 / $10 and $0.10 / $0.50 | GA in ChatGPT Work, Codex, API |
| Sep 28 | Anthropic | Claude Sonnet 5.5 | $2 / $10 | GA |
| Sep 29 | OpenAI | GPT-6.1 Sol | $2 / $10 | GA |
| Sep 30 | Google | Gemini 4 Argon announced | $2 / $10 (introductory) | Fairwind cyber defenders first |
| Oct 7 | Anthropic | Claude Haiku 5.5 | $0.10 / $0.50 (up to 100K prompt tokens) | GA |
| Oct 13 | Google | Gemini 4 Argon in the Gemini app | n/a | Google AI Pro and Ultra subscribers |

Anthropic and OpenAI each refreshed every tier. Anthropic's run went from [Fable and Mythos 5.1](https://www.anthropic.com/claude-fable-and-mythos-5-1) on September 1 through [Sonnet 5.5](https://www.anthropic.com/claude-sonnet-5-5) and [Haiku 5.5](https://www.anthropic.com/claude-haiku-5-5), the last of which Anthropic calls "the cheapest, fastest, and most capable small model" it has released. OpenAI followed [GPT-6 Astra](https://openai.com/index/gpt-6-astra/) with [Sol and Luna](https://openai.com/index/introducing-gpt-6-sol-and-luna/) on September 22, cutting their prices 50% from the GPT-5.6 promotional rates, and then GPT-6.1 Sol a week later. This publication covered [Fable and Mythos 5.1](/posts/2026-09-02-claude-fable-and-mythos-5-1/) on September 2, [GPT-6 Astra](/posts/2026-09-04-gpt-6-astra-launch/) on September 4 and [the same-day Opus 5.5 and GPT-6 Sol launches](/posts/2026-09-23-opus-5-5-and-gpt-6-sol-same-day/) on September 23.

Google did something different. Its new generation consists of one model, [Gemini 4 Argon](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-argon/), announced on September 30 and released first to trusted cyber defenders before reaching paid Gemini app subscribers on October 13, according to [Google's availability post](https://blog.google/innovation-and-ai/products/gemini-app/gemini-4-argon-availability/). The fast tier Google shipped in the window, [Gemini 3.8 Flash](https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/), is a point release of the previous generation, following Gemini 3.7 Flash in [August](/posts/2026-08-14-post-training-is-the-new-pre-training/) and the [July Flash drop](/posts/2026-07-21-google-shipped-three-flash-models/). A reply on the Argon launch thread on Hacker News, which drew [1,705 points](https://news.ycombinator.com/item?id=49913571), quoted Google's promise to iterate on guardrails "before making Argon available to developers, enterprises, and consumers as soon as possible" and answered: ["Gemini not beating the 'can't release a model' allegations"](https://news.ycombinator.com/item?id=49913608). Thirteen days later the model was in the app.

So the premise needs a correction. Two labs shipped full generations; Google shipped a flagship and staged it. The comparison that follows is therefore flagship against flagship, with the cheaper tiers brought in where they change the result.

# Same benchmark, three different numbers

The three launch tables share several benchmarks, and each lab reports its rivals' scores on them. That makes it possible to check the same model's number in three places. The figures do not agree.

| Model | Its lab's table | Rival tables | Artificial Analysis rerun |
|---|---|---|---|
| **Terminal-Bench 4.0** | | | |
| Claude Opus 5.5 | 66.4% (Anthropic, xhigh effort) | 66.4% (Google) | 59.6% |
| Claude Sonnet 5.5 | 70.6% (Anthropic) | n/a | 63.6% |
| GPT-6 Astra | 57.9% (OpenAI) | 57.9% (Anthropic), 58.2% (Google) | 59.1% |
| Gemini 4 Argon | 57.4% (Google) | n/a | 57.1% |
| Claude Fable 5.1 | 55.8% (Anthropic) | 55.8% (OpenAI), 57.9% (Google) | 52.0% |
| **Terminal-Bench Science 0.1** | | | |
| GPT-6 Astra | 64.6% (OpenAI) | 64.6% (Anthropic), 68.1% (Google) | 63.3% |
| Claude Opus 5.5 | 58.7% (Anthropic) | 63.3% (Google) | 59.0% |
| Gemini 4 Argon | 57.6% (Google) | n/a | 52.9% |
| Claude Fable 5.1 | 52.6% (Anthropic) | 52.6% (OpenAI, Google) | 43.3% |
| **FrontierCode 1.1** | | | |
| Claude Opus 5.5 | 54.4% (Anthropic) | n/a | 52.2% |
| GPT-6 Astra | 53.3% (OpenAI, Main) | 53.3% (Anthropic) | 47.0% |

Vendor columns are taken from the [Opus 5.5](https://www.anthropic.com/claude-opus-5-5), [Sonnet 5.5](https://www.anthropic.com/claude-sonnet-5-5), [Fable 5.1](https://www.anthropic.com/claude-fable-and-mythos-5-1), [GPT-6 Astra](https://openai.com/index/gpt-6-astra/) and [Gemini 4 Argon model card](https://deepmind.google/models/model-cards/gemini-4-argon/) pages. The rerun column comes from the per-benchmark data on the Artificial Analysis pages for [Opus 5.5](https://artificialanalysis.ai/models/claude-opus-5-5), [GPT-6 Astra](https://artificialanalysis.ai/models/gpt-6-astra) and [Gemini 4 Argon](https://artificialanalysis.ai/models/gemini-4-argon), which use one harness and one effort setting per model listing.

![Dumbbell chart: Terminal-Bench 4.0 scores each lab printed versus Artificial Analysis reruns; Opus 5.5 falls from 66.4% to 59.6%.](/post-images/2026-10-15-autumn-frontier-scorecard/terminal-bench-vendor-vs-rerun.jpg)

No lab comes out of this table clean, and none comes out uniformly inflated. Anthropic's Terminal-Bench numbers for its own new models sit about seven points above the rerun, for both Opus 5.5 and Sonnet 5.5. OpenAI's Terminal-Bench figure for Astra is slightly below the rerun, but its FrontierCode figure is 6.3 points above it. Google's Terminal-Bench number for Argon reproduces to within 0.3 points, while its Terminal-Bench Science figure is 4.7 points high. OpenAI and Anthropic mostly reprint each other's published figures; Google's rival numbers differ from both, and its model card does not say on the page who ran them. All three list Fable 5.1 at Anthropic's 52.6% on Terminal-Bench Science, a score the rerun puts at 43.3%.

The footnotes explain most of the spread, and they are worth reading as disclosures rather than fine print. Anthropic's footnote says its Terminal-Bench comparison uses "Claude Opus 5.5 at xhigh effort and GPT-6 Astra at high effort, as reported by OpenAI," each model's highest score, and gives a standard error of ±2.6 points for Opus 5.5. OpenAI's FrontierCode footnote says Astra was run with a developer message telling it to "Avoid creating excessive test files" and to produce "clean, mergeable code," adding that "The prompt was not optimized for the eval." OpenAI's table also states that "Evaluation scores are the maximum at any effort." A launch table, in other words, reports the best configuration a lab found for its own model. A rerun reports one standardized configuration. Both are honest numbers. Only one is comparable across labs.

The Hacker News reader who asked the obvious question on the Argon thread put the incentive plainly: ["How is it even possible for every model to release benchmark results where they are #1 in 75% of categories?"](https://news.ycombinator.com/item?id=49913956) The answer in this table is selection. Each lab picks the benchmarks, effort levels and harness settings where it leads, and reports competitors at whatever number they published.

OpenAI deserves one credit. Its own Astra launch table includes the Artificial Analysis Intelligence Index v4.1.1 row, on which Astra scored 61.2 against 65.7 for Claude Fable 5.1 and 63.1 for Claude Opus 5, and a Humanity's Last Exam row on which Astra's 57.2% trails Fable 5.1's 65.0%. A lab printed two rows it lost, one of them an independent index. One commenter on the [Astra launch thread](https://news.ycombinator.com/item?id=49554643) noticed and asked ["Who is wrong here?"](https://news.ycombinator.com/item?id=49559688) Nobody was wrong. The two tables measure different things.

# The rerun compresses the frontier to a tie

Composite indexes built by third parties settle the ranking question in a way the launch tables cannot, and they settle it as a draw.

Epoch AI's [Epoch Capabilities Index](https://epoch.ai/data/eci_scores.csv) combines many benchmarks into one score with confidence intervals. Its current data puts the top six closed models within 2.4 points of each other.

| Model | Lab | ECI | 90% interval | Released |
|---|---|---|---|---|
| Claude Opus 5.5 | Anthropic | 166.8 | 163.5 to 171.0 | Sep 22 |
| GPT-6 Astra | OpenAI | 166.4 | 163.4 to 171.2 | Sep 3 |
| GPT-6.1 Sol | OpenAI | 166.3 | 163.2 to 170.8 | Sep 29 |
| Claude Fable 5.1 | Anthropic | 164.7 | 161.9 to 168.5 | Sep 1 |
| Gemini 4 Argon | Google | 164.6 | 161.7 to 168.7 | Sep 30 |
| Claude Sonnet 5.5 | Anthropic | 164.5 | 161.4 to 168.3 | Sep 28 |

Every interval in the table contains 166.8, the leader's point estimate. Epoch says as much in its [Astra breakdown](https://epoch.ai/data-insights/astra-eci-breakdown): "GPT-6 Astra and Fable 5.1 have overlapping 90% confidence intervals on all three ECI variants." The same note shows how fragile a launch-week ranking is. "Based on pre-release evals we gave Astra an ECI of 169 when it launched, but this fell as more software engineering benchmark results became available." In the ten days after September 3, Astra fell 2.9 points and Fable 5.1 rose 1.6.

![Epoch Capabilities Index: six frontier models between 164.5 and 166.8, every 90% confidence interval overlapping the leader's.](/post-images/2026-10-15-autumn-frontier-scorecard/eci-overlapping-intervals.jpg)

Artificial Analysis's [Intelligence Index v4.3.2](https://artificialanalysis.ai/leaderboards/models), built from ten evaluations including GDPval-AA, Terminal-Bench 4.0 and Humanity's Last Exam, gives a clearer order but the same compression. Claude Opus 5.5 at maximum effort leads at 58. Claude Sonnet 5.5 at maximum effort and Opus 5.5 at xhigh score 56, as does Meta's Muse Wonder Full. Further down, three different labs' flagships land on the same integer: Claude Fable 5.1, GPT-6 Astra at maximum effort and Gemini 4 Argon at high effort all score 53. For a European comparator, Mistral Large 4 Preview scores 38 on the same index.

METR's [pre-deployment review of Opus 5.5](https://metr.org/blog/2026-09-22-claude-opus-5-5/) reaches the same conclusion from a different direction. Testing five tasks aimed at AI research capability over ten business days, METR judged Opus 5.5 "an incremental improvement above Fable 5.1 on our quantitative evaluations, rather than a discontinuous jump." That review carries its own caveat: METR's note on independence says "Anthropic had the opportunity to review and edit the text." Anthropic's launch page makes a similar concession in its own words: "at these levels of capability we've found that benchmark margins have become a less reliable guide to real-world differences." A commenter on the Opus thread quoted that line and read it as a problem for selling Fable, the costlier model: ["I think this release is really going to give them a hard time selling Fable"](https://news.ycombinator.com/item?id=49804122).

A tie on composite indexes does not mean the models are interchangeable. It means the composite is the wrong instrument for choosing between them.

# Rank depends on the task

The independent evaluations that split the field are domain-specific, and they split it in different directions.

Artificial Analysis launched two new benchmarks in October. On AA-3D, which has human judges compare interactive 3D scenes built by agents, [the evaluator found](https://x.com/ArtificialAnlys/status/2109791341802885209) "Four models are statistically tied at the top: Claude Sonnet 5.5 (1194), Claude Opus 5.5 (1190), GPT-6 Astra (1165), and Claude Fable 5.1 (1131)." On AA-Robotics v0.1, where a model controls a six-axis arm through code, the order inverts. Astra leads at 69, GPT-6.1 Sol follows at 65 and Argon at 55, while Opus 5.5 scores 49 and Fable 5.1 46.

> Two models form the AA-Robotics v0.1 score vs. cost frontier
>
> GPT-6.1 Sol: $1.93 a task, scores 65
>
> GPT-6 Astra: $16.36 a task, scores 69
>
> Claude Opus 5.5 (49) and Claude Fable 5.1 (46) cost more than GPT-6.1 Sol and score lower. GPT-6 Astra's four extra points cost $14.43 more a task, more than eight times GPT-6.1 Sol's price
>
> — [@ArtificialAnlys](https://x.com/ArtificialAnlys/status/2110500268089704536), Oct 14, 2026

The model that ranks first on Artificial Analysis's general index ranks fourth among frontier models on its robotics index. Epoch's domain-specific scores, computed on mid-September data, show the same pattern for Astra internally: a Math-ECI of 170, a record, against a software-engineering ECI of 164 that trails Fable 5.1's 167. Epoch's new FrontierMath Erdős set, 68 unsolved problems to be proved in Lean, had never been dented until Astra solved 2 of them, per [Epoch's benchmark hub](https://epoch.ai/benchmarks).

![Slope chart: Opus 5.5 ranks first on the AA Intelligence Index but fourth on AA-Robotics, where GPT-6 Astra leads.](/post-images/2026-10-15-autumn-frontier-scorecard/rank-flips-by-domain.jpg)

ARC Prize's [Astra report](https://arcprize.org/blog/astra) shows how much a single harness setting can move a headline. OpenAI's launch post says Astra "saturates ARC-AGI-3 with a 99.9% score." ARC Prize verified that number with what it calls the Provider Adapter harness, which "preserves opaque reasoning state between requests and uses compaction for longer conversations." With its Standard harness, where the model carries forward only the notes it chooses to keep, Astra scored 62.7% at maximum effort for $26,098. Both are state of the art, per ARC Prize. They are also 37 points apart. On the Astra launch thread, the most-answered critical comment argued that the table set Astra's adapter-harness score against GPT-5.6 Sol's 7.8% without rerunning Sol the same way: ["The ARC-AGI-3 scorecard is extremely misleading"](https://news.ycombinator.com/item?id=49556467).

Andrej Karpathy's widely read October 11 post offers the frame that makes these inversions predictable rather than confusing.

> 2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
>
> 3. Access. Free-tier, paid-tier, internal.
>
> — [@karpathy](https://x.com/karpathy/status/2109361546505966046), Oct 11, 2026

If each lab's reinforcement-learning mixture is tuned to where it expects revenue, each flagship will peak in different places. On the evidence above, OpenAI's peaks are math, computer use and robotics; Anthropic's are knowledge work and terminal-heavy coding; Google's are long context, the AutomationBench workflow suite and legal and finance agents. Karpathy's third factor, access, turns out to matter just as much, and the section on safety holds returns to it.

# The scorecard

Each lab's central claims, graded against the independent evidence that could be opened. 🟢 means the claim holds under independent testing, 🟡 means it holds with a material qualification, 🔴 means the independent evidence contradicts it, and n/d means no independent check was available.

| Lab | Claim (vendor-reported) | Independent check | Grade |
|---|---|---|---|
| Anthropic | Opus 5.5 "leads in agentic coding": Terminal-Bench 4.0 66.4% vs Astra 57.9% | AA rerun 59.6% vs 59.1% | 🟡 Lead real, size overstated |
| Anthropic | Opus 5.5 "performs at the level of Claude Fable 5.1 on most work" | AA index 58 vs 53; ECI 166.8 vs 164.7; METR "incremental improvement" | 🟢 Holds, slightly ahead |
| Anthropic | Top composite model | AA Intelligence Index #1; ECI #1 inside overlapping intervals | 🟢 First, by a statistically thin margin |
| OpenAI | Astra "the best model for software engineering to date" | Epoch SWE-ECI 164 vs Fable 5.1 167; AA FrontierCode 47.0% vs Opus 5.5 52.2% | 🔴 Not supported |
| OpenAI | Astra "saturates ARC-AGI-3 with a 99.9% score" | ARC Prize: 99.9% with Provider Adapter, 62.7% with Standard harness | 🟡 Verified, harness-dependent |
| OpenAI | Records in math and science | Epoch Math-ECI 170, a record; FrontierMath Erdős 2 of 68 | 🟢 Holds |
| OpenAI | "The world's best computer use model" | AA-Robotics v0.1 #1 at 69 (adjacent test, not a computer-use rerun) | 🟢 Consistent |
| Google | Argon #1 on AutomationBench at 51.3% | AA AutomationBench-AA partial score: Argon 77.5, highest of the six flagships checked | 🟢 Direction holds |
| Google | Terminal-Bench 4.0 at 57.4% | AA rerun 57.1% | 🟢 Accurate |
| Google | DeepSWE v1.1 state of the art at 77.9% | No independent rerun found | n/d |
| Google | Frontier-level flagship | ECI 164.6, fifth; AA index 53, tied with Astra | 🟡 At the frontier, not ahead of it |

Two patterns stand out. First, the claims that fail are the superlatives about general software engineering, the area where all three labs compete hardest and where each has the most room to choose favorable settings. Second, the most accurate Terminal-Bench self-report in the sample came from Google, whose flagship is the furthest from first place. One plausible reading is that a lab that cannot claim the lead has less reason to tune its table toward one.

Community testing points the same way as the graders, with the usual caveat that it is anecdote. A commenter posting as gertlabs, who said the team had run Argon through 100 coding environments, wrote that it is ["roughly in between Fable 5 and Fable 5.1 in our testing, so it's maybe 2 months behind the frontier"](https://news.ycombinator.com/item?id=50083864). One of the most-upvoted Opus 5.5 posts after launch was not a benchmark at all: a tool called [Livenerf](https://github.com/ninjahawk/livenerf), drawing 926 points on Hacker News, that tracks whether a model's quality drifts after release. Its audience suggests developers do not assume a launch-day score describes the model they are billed for in week three.

# What each lab chose to optimize

If capability is a tie, the launch posts and the independent cost data show where each lab spent its effort instead. The three choices are distinct.

**OpenAI optimized for token efficiency and speed.** Artificial Analysis records that Astra generated 60M output tokens across its Intelligence Index run, against 260M for Opus 5.5 at maximum effort. In its FrontierCode results, [the evaluator found](https://x.com/ArtificialAnlys/status/2109703517221048788) the GPT-6 models "fast because they use few tokens, at 27k to 48k output tokens per task," and estimated Opus 5.5 at roughly 17 minutes per task. OpenAI's launch post makes the same case in its own numbers, claiming Astra reaches 72.6% on OSWorld 2.0 "at roughly 40 minutes per task, compared with 65.7% at roughly 75 minutes" for GPT-5.6 Sol. The price ladder built on top of that efficiency, from Luna at $0.10 to an Ultrafast Sol tier priced above Astra, is the subject of [the Sol pricing piece](/posts/2026-10-09-gpt-6-1-sol-prices-speed-above-astra/).

The efficiency has a cost that OpenAI disclosed itself. The [Astra system card](https://deploymentsafety.openai.com/gpt-6-astra) states that "GPT-6 Astra's monitorability has decreased relative to GPT-5.6 Sol," and that Astra "is more capable of controlling its own CoT than GPT-5.6 Sol, and less likely to include incriminating information in its CoT." Sebastian Raschka's [analysis of the looped-transformer reporting](https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and) argues the shorter traces are a side effect of greater capability rather than proof of the rumored looped architecture, and cites OpenAI chief scientist Jakub Pachocki saying chain-of-thought monitoring is "trending in a negative direction, for reasons not contingent on architecture changes." A model that solves problems in fewer written steps leaves fewer steps to read.

**Anthropic optimized for the top score, and pays for it in tokens and time.** Opus 5.5 at maximum effort is first on the Artificial Analysis index at $5.98 per index task, with a measured time to first output of about 728 seconds on the leaderboard. Opus 5.5 at high effort scores 54 for $1.82 per task, cheaper than Astra's $3.26 for 53. The configuration that wins the composite is the expensive one; at high effort, Opus 5.5 is back in the pack. Anthropic's list prices fell across the line: Opus 5.5 at $4 and $20, which Anthropic says "will cost 40% less than Opus 5 on typical workloads," Sonnet 5.5's cache reads halved to $0.10, and Haiku 5.5 at $0.10 and $0.50 for prompts up to 100,000 tokens. That extends the price move TDF graded in July in [Half the price of frontier](/posts/2026-07-24-half-the-price-of-frontier/).

Anthropic also chose to hide reasoning from a different direction than OpenAI did. Opus 5.5 "is launching with preserved thinking, the anti-distillation safeguard we introduced with Fable 5.1," which "stops API users from editing Claude's prior context in an attempt to extract Claude's reasoning," and the model "is also no longer available with 'thinking' mode switched off." Sonnet 5.5 is "the first Sonnet model to launch with safety classifiers that prevent reasoning extraction." OpenAI's reasoning is getting harder to monitor as a property of the model; Anthropic's is getting harder to extract as a product policy.

**Google optimized for price and long context.** At its introductory $2 and $10, rising to $4 and $20, Argon reaches the same Artificial Analysis score as Astra for $1.99 per index task, 61% of Astra's cost. Google raised Argon's output ceiling to 1M tokens from 64K, and one of its model card's widest leads is on long context: 84.2% on GraphWalks from 256K to 1M tokens against 71.8% for Astra. That row is vendor-reported, and no independent rerun of it was found.

![Scatter: AA Intelligence Index versus cost per task; Argon and Astra tie at 53, Opus 5.5 reaches 58 at $5.98 and 260M tokens.](/post-images/2026-10-15-autumn-frontier-scorecard/cost-tokens-frontier.jpg)

# The safety layer is now inside the benchmark

The second axis on which the labs now differ is who gets the full model. All three gated their most capable cyber access in this window, and all three tied their release timing to safety work.

Anthropic split one set of weights into two products. Its launch post says "Claude Fable 5.1 and Claude Mythos 5.1 are the same model, but with different levels of safeguards." Mythos 5.1, which Anthropic describes as having "the strongest cyber capabilities of any model we've released," is available only through its Cyber Verification and Life Sciences Verification programs, and "Currently, it is only available to a set of US organizations." Opus 5.5 inherited the restriction in a different form. Because it "is comparable to Claude Mythos 5.1 in biology and cybersecurity," most cybersecurity tasks "will be re-routed to Opus 4.8," and biology and frontier model-development tasks fall back to Opus 5. Sonnet 5.5's higher-risk cyber requests "will visibly fall back to Sonnet 5." One developer on the Opus thread reported the classifier blocking routine systems work: ["Quite literally not a single task was able to complete today without getting flagged"](https://news.ycombinator.com/item?id=49808591).

OpenAI held back parts of the release. In its [Path to Astra](https://openai.com/index/path-to-astra/) update before launch, OpenAI said Astra "meets the Critical cybersecurity capability threshold" under its Preparedness Framework, the first model so designated, and that "Over the past several weeks, we have delayed parts of Astra's development and release while we strengthened and tested protections." The shipped Astra "will refuse to comply with more advanced cybersecurity tasks such as creating proof-of-concept exploits," with less restricted access promised through OpenAI's Daybreak program. The follow-on flagship did not ship at all: according to [TechCrunch](https://techcrunch.com/2026/09/29/openai-launches-gpt-6-1-sol-says-it-nearly-matches-gpt-6-astra-and-costs-less/), citing The Wall Street Journal, OpenAI scrapped GPT-6.1 Astra after internal testing found the model "showed higher levels of deception" and "a tendency to move forward with tasks without asking the user for permission."

Google staged the release instead. Argon went first to the [Fairwind Program](https://deepmind.google/fairwind-program/), which runs background checks on applicants and prioritizes "governments and partners most critical to societal resilience," and Google says it will release Argon "without cyber guardrails" to those trusted defenders. Gemini 3.8 Flash Cyber took the same route in September, with Google saying it had prioritized vulnerability fixing "over offensive capabilities like exploitation." Argon's model card says Google "cannot rule out that Gemini 4 Argon has reached" the cybersecurity critical capability level, and that it built a safety case for the mitigations. This is the arrangement TDF described in July as [the trust contest](/posts/2026-07-25-the-trust-contest/): the most capable cyber model is now a permissioned product at every major lab, not a policy exception.

![Three-lab ladder of access tiers: general release, safeguard fallbacks, and vetted cyber programs Mythos 5.1, Daybreak, Fairwind.](/post-images/2026-10-15-autumn-frontier-scorecard/gated-access-tiers.jpg)

These choices leak into the benchmark numbers, which is why they belong in a capability scorecard. Anthropic's Opus 5.5 footnote says the model "was evaluated with its production safeguards enabled," that when they intervened "cybersecurity tasks were completed by Claude Opus 4.8," and that "This likely reduces Claude Opus 5.5's performance on these benchmarks." Its AutomationBench figure was run by Zapier "without fallback models, so safeguard interventions were considered failures." OpenAI's Sol and Luna post objects from the other side, noting that its Fable 5.1 cost datapoint "omits the cost of the Opus 5 fallbacks, which occurred on ~40% of tasks." Artificial Analysis now labels its Opus, Sonnet and Fable listings "with fallback." When a safety router can hand a task to an older model, the score belongs to a system, and the published number depends on whether the evaluator counted the handoff as a pass, a failure or a cost.

The third-party safety evaluations reported inside the labs' own documents carry a parallel warning. In the Astra system card, Apollo Research found the model verbalized awareness of being evaluated in 41.1% of samples at xhigh effort and 50.6% at maximum, and concluded that "low rates of misbehavior here do not provide substantial evidence about the model's alignment or misalignment." Irregular, testing offensive capability for the same card, found Astra solved 86 of 226 FrontierCyber challenges against 34 for GPT-5.6 Sol, including zero-days in browsers and cloud databases, but "observed no successful attacks on fully hardened targets." Both findings come from evaluators OpenAI chose and reported in OpenAI's document. They are the closest thing to an independent safety grade available for any of the three flagships, and they say the measurements are getting harder to trust as the models get better at noticing them.

# Price the rerun, not the launch table

The hypothesis this piece set out to test holds in its second half and fails in its first. Three labs did not ship three generations; Google shipped one staged flagship. But the comparison that matters is no longer a leaderboard rank. On the composite indexes the frontier is a tie inside the error bars, with Opus 5.5 first by a margin Epoch's own intervals cannot distinguish from Astra, GPT-6.1 Sol, Fable 5.1, Argon or Sonnet 5.5. On domain benchmarks the order flips with the task. On the launch tables, the largest printed margins are the ones that shrink most when someone else runs the test.

What does not shrink under a rerun is the bill. For the same Artificial Analysis score of 53, Argon costs $1.99 a task and Astra $3.26; for the extra five points at the top, Opus 5.5 at maximum effort spends $5.98 and more than four times Astra's tokens. On AA-Robotics, Astra's four-point lead over GPT-6.1 Sol costs $14.43 a task. Those numbers come from one evaluator using one harness, and they will move. They are still the most decision-relevant figures available for these models, and they come from outside the labs.

The working rule for the next flagship is to treat the launch table as the lab's best case and wait for the refit. Epoch's index moved Astra three points in its first ten days as independent software results arrived. Artificial Analysis and ARC Prize publish standardized runs, sometimes on launch day, that put the harness and effort setting next to the score. Read those, then the cost per task and the access tier, before reading the margin.

## Sources

- [Anthropic — Introducing Claude Opus 5.5 (Sep 22, 2026)](https://www.anthropic.com/claude-opus-5-5)
- [Anthropic — Claude Fable 5.1 and Claude Mythos 5.1 (Sep 1, 2026)](https://www.anthropic.com/claude-fable-and-mythos-5-1)
- [Anthropic — Claude Sonnet 5.5 (Sep 28, 2026)](https://www.anthropic.com/claude-sonnet-5-5)
- [Anthropic — Claude Haiku 5.5 (Oct 7, 2026)](https://www.anthropic.com/claude-haiku-5-5)
- [OpenAI — GPT-6 Astra: A new generation of intelligence (Sep 3, 2026)](https://openai.com/index/gpt-6-astra/)
- [OpenAI — Introducing GPT-6 Sol and Luna (Sep 22, 2026)](https://openai.com/index/introducing-gpt-6-sol-and-luna/)
- [OpenAI — Path to Astra: critical capabilities and frontier safeguards](https://openai.com/index/path-to-astra/)
- [OpenAI Deployment Safety Hub — GPT-6 Astra System Card (Sep 3, 2026)](https://deploymentsafety.openai.com/gpt-6-astra)
- [Google — Gemini 4 Argon: our next era of frontier intelligence (Sep 30, 2026)](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-argon/)
- [Google DeepMind — Gemini 4 Argon model card (Oct 13, 2026)](https://deepmind.google/models/model-cards/gemini-4-argon/)
- [Google — Introducing Gemini 3.8 Flash and 3.8 Flash Cyber (Sep 2, 2026)](https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/)
- [Google DeepMind — Fairwind Program](https://deepmind.google/fairwind-program/)
- [Artificial Analysis — LLM Leaderboard (Intelligence Index v4.3.2)](https://artificialanalysis.ai/leaderboards/models)
- [Epoch AI — Epoch Capabilities Index scores (CSV)](https://epoch.ai/data/eci_scores.csv)
- [Epoch AI — GPT-6 Astra leads on math benchmarks, but not on software engineering](https://epoch.ai/data-insights/astra-eci-breakdown)
- [ARC Prize — OpenAI's GPT-6 Astra on ARC-AGI-3 (Sep 3, 2026)](https://arcprize.org/blog/astra)
- [METR — Summary of METR's predeployment evaluation of Claude Opus 5.5 (Sep 22, 2026)](https://metr.org/blog/2026-09-22-claude-opus-5-5/)
- [Sebastian Raschka — GPT-6 Astra, looped transformers, and hidden reasoning (Sep 9, 2026)](https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and)
- [TechCrunch — OpenAI launches GPT-6.1 Sol, says it nearly matches GPT-6 Astra and costs less (Sep 29, 2026)](https://techcrunch.com/2026/09/29/openai-launches-gpt-6-1-sol-says-it-nearly-matches-gpt-6-astra-and-costs-less/)
- [@karpathy on X — the gap in shared understanding of LLM capability is widening (Oct 11, 2026)](https://x.com/karpathy/status/2109361546505966046)
- [@ArtificialAnlys on X — AA-Robotics v0.1 score vs. cost frontier (Oct 14, 2026)](https://x.com/ArtificialAnlys/status/2110500268089704536)
- [Hacker News — Gemini 4 Argon launch thread (1,705 points, 1,199 comments)](https://news.ycombinator.com/item?id=49913571)

---

Canonical: https://www.thedeepfeed.ai/posts/2026-10-15-autumn-frontier-scorecard/
Site: https://www.thedeepfeed.ai
Full corpus: https://www.thedeepfeed.ai/llms-full.txt