# Discovery is the new benchmark. Grade it by how cheaply an outsider can check it

URL: https://www.thedeepfeed.ai/posts/2026-10-17-discovery-is-the-new-benchmark/
Category: Research
Published: 2026-10-17
Author: the-deep-feed
Tags: research, ai-for-science, evaluation, benchmarks, anthropic, openai, google-deepmind, verification
Kind: deep

> Between September and October, AI labs and their users claimed an enzyme system, the NIST curve seeds, an Enigma break and a genome atlas, and graded on who did the work, what was checked and what replicated, those claims sort by how cheaply an outsider can verify the answer.

## TL;DR

- **Anthropic** traced the 25-year NIST curve-seed mystery to a timestamp fed to an old C random generator, and bounty organizer **Filippo Valsorda** published an independent Go reimplementation the same day. P-256 was candidate **1,842**. Of the season's claims, it is the most checkable.
- The **ART** enzyme claim is the least checkable. Anthropic reran its own agent campaign **10** more times and the defining repeat array was **missed in every rerun**. A Copenhagen researcher says his unpublished work describes the same system, and the enzyme's function is still unknown.
- The **MVUEH** Enigma break was a user's result, not a lab launch. The archivist who validated it says GPT-6 Astra did it **entirely on its own**; the user's case study calls the work researcher-led. Five days after the break, a second message fell to **Claude Opus 5**.
- Benchmarks topped out first: OpenAI's Astra post claims **98%** on FrontierMath Tier 4 and **100%** on ExploitBench. Mercor estimates **90%** of US working hours sit on tasks benchmarks barely cover. Discovery claims share that bias: they cluster where a computer can check the answer.

Graded on who did the work, what an outsider checked, whether it replicated and what the field said, the six AI discovery claims of September and October sort by one variable: how cheaply an outsider can check the answer. Anthropic's NIST curve-seed result, independently reimplemented the same day, is the most checkable; its ART enzyme claim, whose defining repeat array was missed in all 10 of Anthropic's own reruns, is the least.

On October 16, Anthropic published two research posts a few hours apart. In [the first](https://www.anthropic.com/research/ecdsa-curve-origins), an internal Claude model worked out in under half an hour how the seeds behind the NIST elliptic curves were generated, a question cryptographers had argued about for 25 years. In [the second](https://www.anthropic.com/research/designing-efficient-quantum-circuits), Claude Mythos 5 designed quantum circuits that beat the best published constructions, one of them with 17 times fewer gates. The same day, a Mercor [benchmark map](https://www.mercor.com/research/labor-index/benchmark-map/) reached the top of Hacker News with an estimate that 90% of American working hours go to tasks that existing AI benchmarks barely touch.

Before that, Anthropic announced on September 23 that Claude agents had found [a novel enzyme system](https://www.anthropic.com/news/claude-discovers-novel-enzyme-system) in phage DNA. In mid-September, a user of OpenAI's GPT-6 Astra broke a 1941 German Army Enigma message listed as unbroken since 2005, and the archivist who keeps that list [validated the result](https://www.cryptocellar.org/bgac/the-mvueh-break.html). On September 8, Google DeepMind released a [precomputed map](https://deepmind.google/blog/alphagenome-atlas-a-predictive-map-of-every-possible-dna-letter-change-in-the-human-genome) of every possible single-letter change to the human genome, all 9 billion of them.

The flagship benchmark tables have topped out, and the proof of capability has moved to results anyone can understand: a solved mystery, a found enzyme, a broken cipher. A benchmark score comes with a protocol others can rerun. A discovery comes with whatever the lab chooses to publish. This piece grades six of the season's claims on four questions: who did the work, what someone outside the lab checked, whether the result replicated, and what experts in the field said.

The starting hypothesis, that labs now launch with discoveries instead of benchmark tables, only partly holds. OpenAI's Astra launch still carried full tables, and DeepMind's atlas leads with benchmark scores. What has moved is the persuasive weight. Graded on the four questions, the claims sort by checkability, the same bias Mercor found in the benchmarks these claims are replacing.

# The table ran out of headroom

The clearest statement of the problem is in OpenAI's own launch copy. The [GPT-6 Astra post](https://openai.com/index/gpt-6-astra/) says the model "saturates FrontierMath Tier 4 with a 98% score" and "also saturates ARC-AGI-3 with a 99.9% score and ExploitBench with a 100% score." A benchmark at 98% or 100% has stopped separating models. Citing "concerns that exposure to historical software vulnerabilities may have affected benchmark results," OpenAI built an internal "ExploitBench (June–August 2026)" from vulnerabilities disclosed in the previous three months. During that evaluation, the post says, Astra "even discovered and used two previously unknown zero-day vulnerabilities."

When the test set runs out, the next proof is something nobody had found before. The same post opens its science section, "Advancing scientific discovery," with new number-theory results; mathematics is the subject of [a separate piece on the Navier-Stokes proof](/posts/2026-10-17-who-checks-the-proof/), and this one covers the claims outside it.

OpenAI's chief scientist, Jakub Pachocki, named the measurement problem in [An Alien Mind](https://openai.com/index/an-alien-mind/), an essay published September 6. As the systems grow more capable, he wrote, their results become harder to interpret, and "this is made more complicated by the current algorithms generally improving easy-to-measure capabilities faster than those hard to objectively quantify."

This publication argued in June that the [agent-evaluation sector was raising money on numbers](/posts/2026-06-25-agent-eval-startups-metric-nobody-trusts/) the research had shown to be contaminated and gameable. Saturation is the top-end version of that problem: contamination makes a score untrustworthy, saturation makes it uninformative. Discovery claims are what the labs reached for instead.

# Four questions for a discovery claim

A benchmark score has a built-in audit trail: a fixed test set and protocol a rival can rerun. A discovery claim has none by default. These four questions put one back.

**Who did the work.** "Claude discovered" and "a team used Claude" are different claims. The answer sits in the author list, the prompt, the harness and the human steering in between.

**What an outsider checked.** It has to be someone with no stake in the lab rerunning the calculation, validating the key or reproducing the experiment.

**Whether it replicated.** Would the same system find it again, and would a different one? For agent campaigns this question is new, and it matters most.

**What the field said.** Experts decide whether it is new, important and first. Their answers often differ from the press release.

![Schematic: four gates a discovery claim passes through, labeled WHO DID THE WORK, OUTSIDE CHECK, REPLICATION and FIELD VERDICT.](/post-images/2026-10-17-discovery-is-the-new-benchmark/four-gate-rubric.jpg)

# The curve seeds: an answer anyone can rerun

The NIST prime curves P-192 through P-521 were published in FIPS 186-2 in 2000, each generated by hashing a 20-byte seed of unexplained origin. The lore, recounted by Filippo Valsorda when he [opened a bounty](https://words.filippo.io/seeds-bounty/) three years ago, was that Jerry Solinas, who died in 2023, had hashed English sentences he later forgot. An unexplained seed leaves room to suspect a chosen curve.

Anthropic says several employees had chased the hashed-phrase rumor without success. In October, one asked an internal research model to work out the method without the hint. Claude guessed at a weak 1990s generator, scanned 34 generator and output combinations, and noticed that four of the ten seeds, written as five 32-bit words each, had a zero top bit in all 20 words, a pattern Anthropic puts at 1 in 5,810 by chance. That is the signature of `lrand48`, which returns only the top 31 bits of a 48-bit state. Brute force over the 17 hidden bits and LLL lattice reduction did the rest. The timestamps fall between May 1997 and April 1998; P-256 is the 1,842nd candidate from a generator seeded at 01:52:34 UTC on August 16, 1997.

**Who did the work.** A model ran the search, but the post credits ten named staff, including Nicholas Carlini and Steve Weis, who had written one of the [earlier public investigations](https://saweis.net/posts/nist-curve-seed-origins.html) of the seeds. The prompt was one question from a team that knew the problem well.

**What an outsider checked.** The strongest answer of the season. Valsorda [reimplemented the algorithm in Go](https://github.com/FiloSottile/mostly-harmless/blob/main/seeds/seeds.go), published the output on October 16 and confirmed the bounty claim. His run also surfaced an open question: the generator yields "roughly twice as many valid seeds as were published," and nobody has found a good explanation for which were dropped.

**Whether it replicated.** Anthropic published CC0 code that regenerates all 160 bits of each seed from a timestamp and candidate index. Anyone with Python can check it.

**What the field said.** Valsorda wrote that the result rules out the seed-selection attack Daniel Bernstein and Tanja Lange had described, which needed about 70 bits of freedom: even with any timestamp in 1997–98 and any of the first 16 candidates, "they would have had barely 30 bits of freedom." On Hacker News, where the post drew 973 points, the top-ranked replies split between relief and the open question:

> This is devastating news... for an entire generation of NIST cryptography conspiracy theories.
>
> — [tptacek on Hacker News](https://news.ycombinator.com/item?id=50123044), Oct 16, 2026

> While this confirms the method of production it doesn't say anything about their selection, especially since they were, apparently, generated over a period of months. Specifically you cannot see those candidates that were generated and discarded.
>
> — [fidotron on Hacker News](https://news.ycombinator.com/item?id=50123213), Oct 16, 2026

Anthropic concedes the gap: a valid prime-order curve at candidate 876 was passed over for 1,842, the B-163 seed "fits nothing Claude has tried," and four prime-curve base points remain unexplained. "This is not a cryptographic break, new attack, or weakness in the ECDSA curves," the post says.

![Schematic: seed recovery chain from 1997 Unix timestamp to srand48, 20-byte candidates, X9.62 checks, and P-256 at candidate 1,842.](/post-images/2026-10-17-discovery-is-the-new-benchmark/seed-recovery-chain.jpg)

Grade: strong on every axis. Any skeptic can check it for the cost of a script, and one with standing did.

# The circuits: verification built into the task

The second October 16 post is checkable for a different reason. Andi Gu, a fellow in Anthropic's STEM Fellows Program, worked with Anthropic physicist Alek Kemeny and Harvard's Susanne Yelin and Mikhail Lukin. They gave Claude plain-language specifications for eight circuit-design tasks. An automated evaluator checked each circuit for correctness and scored it by qubits times gates, and the model iterated without human input between rounds.

For elliptic-curve point addition, Claude worked 53 hours from a one-paragraph brief; one resulting design uses 12% more qubits but 17 times fewer gates than the smallest previous one. For the Decoded Quantum Interferometry decoder, it switched to Berlekamp-Massey and recomputed intermediate values instead of storing them, cutting gates by a factor of 12 against Google's earlier construction and replacing a 665-qubit worst-case reserve with 19 qubits.

**Who did the work.** The [arXiv paper](https://arxiv.org/abs/2610.22367) lists Gu's two Harvard affiliations first and the Anthropic STEM Fellows Program third. Humans wrote the specifications and occasionally pushed the model to search more broadly; for the modular-inversion idea, the post says Claude "developed the approach, implemented it, and verified its correctness autonomously."

**What an outsider checked.** Correctness is checked mechanically, two Harvard physicists are co-authors, the modular-inversion claim was formalized in Lean, and the decoder ran 403 million sample decodings, failing about once in 10,000, within the allowed error.

**What the field said.** One Hacker News commenter [described the task](https://news.ycombinator.com/item?id=50122821) as "basically a game of code golf with the assembly instructions," which is why a model with a fast verifier loop does well at it. [Another noted](https://news.ycombinator.com/item?id=50127006) that dropping components needed only for rare inputs means the circuits no longer perform exactly the same computation; the measured failure rate is the authors' answer.

Grade: strong and narrow. No unaffiliated replication is on record yet.

# The cipher: a user's break, validated by the archivist

The Enigma result was not a lab announcement, which makes it a useful control. On September 15, Carter Leffen asked Frode Weierud, who runs the Crypto Cellar Research archive of unbroken German Army messages, to validate a break of MVUEH, an 82-letter message from July 10, 1941, that had "resisted all attempts to break it" since 2005. Leffen announced it two days later:

> Two days ago, GPT-6 Astra broke a yet unsolved German Army Enigma message from 1941.  Amazingly Astra was able to autonomously:
>
> - Search historical archives
> - Compare uncertain letters
> - Find contextual clues
> - Build an Enigma simulator
> - Write cryptanalysis code
> - Run parallel experiments
> - Test competing keys
> - Recover the plaintext
> - Cross-check the results
>
> — [@carterleffen](https://x.com/carterleffen/status/2100405732935868840), Sep 17, 2026

**Who did the work.** The two primary accounts disagree. Weierud writes that "the most astonishing thing about this break is that the GPT–6 Astra did it entirely on its own": Leffen only pointed the model at the unbroken list, and the model chose MVUEH, linked it to the related message SIPVX and picked ROSENOW ROSENOW as its crib. Leffen's [case study](https://mvueh-enigma-solved.carterl.chatgpt.site/) describes "a researcher-led investigation with GPT-6 Astra and parallel specialist agents" in which "the researcher set the goal and pushed the investigation forward." It adds that SIPVX, broken by Alex Shovkoplyas in 2017, "helped motivate the crib; it was not a blind prediction."

**What an outsider checked.** Weierud, the domain authority with no stake in OpenAI, wrote that "it was immediately clear he had found the correct key and plaintext." The case study adds separate implementations, an audit of all 43,016 search batches and agreement with an independent SAT model, and discloses that 923 physical keys match the header.

**Whether it replicated.** Weierud [reported](https://www.schneier.com/blog/archives/2026/09/gpt-6-astra-breaks-an-old-enigma-message.html) at Schneier on Security that another unbroken message, the 63-letter Nr. 205 of July 31, 1941, "was successfully broken with the help of Anthropic Claude Opus 5" on September 20, also with a crib. That is not a rerun of MVUEH, but it shows the capability carrying across labs to a sibling problem.

**What the field said.** Weierud wrote that "as an old cryptanalyst, I am still in awe." The case study is more modest than the headlines: "no first-ever or corpus-wide breakthrough is claimed."

Grade: verified answer, disputed credit. Anyone with an Enigma simulator can check the key; the model's share of the work rests on logs only Leffen holds.

# The atlas: the one that still ships a benchmark table

AlphaGenome Atlas is a resource, not a single finding, and it arrived the old way, with tables: precomputed predictions for all 9 billion possible single-nucleotide variants, a 1-petabyte dataset "more than 30 times larger than the AlphaFold Database," topped by one per-variant score, the AlphaGenome Variant Impact (AVI).

**Who did the work.** The [technical paper](https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphagenome-atlas-a-predictive-map-of-every-possible-dna-letter-change-in-the-human-genome/alphagenome-atlas.pdf) lists DeepMind authors alongside co-authors from Exeter, the Broad Institute, Boston Children's Hospital, Stowers and Memorial Sloan Kettering. There is no agent narrative.

**What an outsider checked.** On ClinVar's pathogenic-versus-benign variants, AVI scored an AUPRC of 0.76 on intronic variants against 0.44 for the next-best model, GPN-Star-V, and the paper reports where it lost (5' UTR, 0.26 to 0.27). Collaborators took one prediction to the bench: among 814 rare-disease individuals in the GREGoR cohort, AVI ranked first a deep intronic DNM1 variant in a patient with epileptic encephalopathy. A minigene assay across five cell lines found 12 intronic variants causing exon extensions, and AlphaGenome's splicing score matched the experimental data with an AUPRC of 0.943.

**What the field said.** Carl de Boer, a genomicist at the University of British Columbia, told [IEEE Spectrum](https://spectrum.ieee.org/alphagenome-atlas) that AlphaGenome is the "field's leading model" but warned that the single impact score "is probably going to be easily misinterpreted." In July, Ben Lehner's group [mutated every nucleotide](https://www.biorxiv.org/content/10.64898/2026.07.25.740675v1) of the phage PhiX174 and found its "mutational effects are only moderately-well predicted by state-of-the-art artificial intelligence models." AlphaGenome was not the subject, but that is the kind of ground truth such predictions will be graded against. DeepMind's fine print agrees: AlphaGenome "has not been validated for, and is not approved for, any clinical use."

Grade: well-documented and candid about limits, and the only claim here in the old format, because DeepMind released a tool rather than announcing a find.

# The enzyme: strongest headline, longest list of caveats

The enzyme claim had the biggest audience: 805 comments on Hacker News, and a dip in gene-editing stocks, per CNN. Dario Amodei framed it on X the day of the release:

> Today we announced the Claude-led discovery of a molecular machine that we suspect could represent a new gene editing mechanism. Its precise function, biotechnological utility (if any), or level of significance is not yet clear, but at minimum it is work I would have been proud to do as a PhD student. The work was done mostly, though not entirely, by Claude: our life sciences team suggested a broad area of research, Claude read through the literature and a bunch of genome data and discovered something interesting, then Claude proposed experiments to verify the discovery and our team carried them out.
>
> — [@DarioAmodei](https://x.com/DarioAmodei/status/2102831170299834652), Sep 23, 2026

The [technical report](https://www-cdn.anthropic.com/22573675ada52a8ca8a97a1a4b4326b2f208a071.pdf), posted to bioRxiv on October 11, gives the scale. Agents running Claude Mythos 5 surveyed reverse transcriptase (RT) loci across 1.9 billion metagenomic protein clusters, recovered about 200,000 RT clusters and scored 3,564 candidate partner families, 17 of which were investigated. The campaign took 949 agent sessions, 77 agent-hours and 215.6 million tokens over 21.5 hours, "without human intervention." Three of the 17 families were confirmed as previously unreported; 14 were rejected. The reported finding came from outside the brief: a worker reading raw DNA beside a jumbo-phage RT noticed an array of tandem repeats. Anthropic named the family array-associated RTs, or ART.

![Data schematic: ART campaign funnel from 1.9B protein clusters to 17 families and one ART report, beside a ten-rerun strip with zero array detections.](/post-images/2026-10-17-discovery-is-the-new-benchmark/art-campaign-funnel.jpg)

**Who did the work.** Anthropic's post says "our involvement was limited to the initial prompt and the lab work." The author list complicates that. Ilya Finkelstein, a molecular biologist at UT Austin, [wrote](https://finkelsteinlab.org/blog/harvesting-and-replenishment-in-biomedicine/) that "four of six authors are senior molecular biologists and domain experts," and noted to [CNN](https://www.cnn.com/2026/10/08/science/ai-biology-anthropic-dna-discovery) that the lead author, Peter Yoon, came from Jennifer Doudna's lab. The agents, he said, are "really good at grunt work when orchestrated by experts in the field."

**What an outsider checked.** Nothing yet. All lab work was in-house: a public phage-infection RNA-seq dataset in which array RNAs reached 8% of phage transcripts, then expression in E. coli with small-RNA sequencing. The report is explicit: "we have not shown that the RT is active or that the unit RNAs are its substrates." What the system does for the phage is unknown, and the work is not peer-reviewed.

**Whether it replicated.** This part of the report matters most for grading discovery claims, and it got the least coverage. Anthropic asked whether its harness would find ART again: "we ran the same campaign ten more times." Nearly every rerun sampled ART loci, and two followed up on the lineage, but "the array was missed in every rerun." The authors blame "the broad protein search space of RTs and the non-deterministic behavior of the harness."

Their next step was to build a benchmark: fixed ART inputs for seven Claude models, scored by a judge model against ten curated features. With the loci in context, the strongest models described the array in at least 90% of attempts; with files and tools, as few as 32% for Opus 5, because many attempts never read enough DNA to see a second repeat. The discovery became a benchmark because the discovery process could not be counted on to happen twice.

**What the field said.** Feng Zhang, quoted in Anthropic's release, called the arrays "genuinely intriguing." Aaron Engelhart of the University of Minnesota told CNN the results "don't demonstrate gene editing or provide a mechanistic picture of exactly what's happening." Finkelstein was blunter: "boy-oh-boy, is the hype getting ahead of the actual science."

The priority questions are more serious. Mario Rodríguez Mestre, who completed doctoral research at the University of Copenhagen, told CNN his unpublished work describes the same viral signatures: "It is essentially the same finding." He had shared drafts on his private Claude account but wrote, "I am not claiming that Anthropic deliberately took our work." His more likely explanation: "there was considerably more prior scientific knowledge and human direction behind the search than the phrase 'autonomous discovery' suggests." Anthropic did not respond to CNN. Amodei's post concedes a neighbor, "a Stanford team working independently," with a similar but distinct RT system.

The top-ranked reply on Hacker News offered the framing the evidence supports:

> A sober framing would be something like: Claude identified a previously undescribed genomic arrangement around a known reverse transcriptase. Not all that sexy. For now, this is mostly a story about how AI can be used to parse existing data to discover new biology (which is fantastic!).
>
> — [Spacecosmonaut on Hacker News](https://news.ycombinator.com/item?id=49823432), Sep 23, 2026

Grade: the most transparent report of the season and the weakest claim. Anthropic published transcripts, rerun counts and an interpretability analysis of Mythos 5's response to repeated DNA, and that transparency is what exposes the gaps: no replication in its own harness, no outside biology, unknown function, disputed novelty. Gustavo Sudre of King's College London told CNN that "the gap between 'interesting pattern' and 'useful knowledge' is where most of the work lives." The headline was written as if that gap had been crossed.

# The bottom-up version costs $1,200

One September result shows the same grading problem without a lab press team. Rohan Bansal [trained a 4B open-weights model](https://rohanbansal.com/qorl) to write Postgres join-order hints, distilling about half a thousand GPT-6 Astra trajectories and then running agentic reinforcement learning, for about $1,200 including $400 in OpenAI API fees. It reached 702 points on Hacker News as "81% faster query plans than Postgres."

The post is unusually clear about the number. It is a 1.81x geometric-mean speedup across the 113 Join Order Benchmark queries, from a model that initially could not plan 99 of them, and it is best-of-15: three rollouts per query, up to five candidates each, fastest kept. The model's own single choice, in the same table, gives 1.40x with seven regressions across 339 trajectories. The top reply on Hacker News added the conditions:

> "81% faster query plans than Postgres"…on an 8 GB dataset that fits entirely in memory, with shared_buffers constrained to a fraction of that, queries warmed before measuring, and read-only SELECTs. I would be cautious about over fitting, it's tough to say if those query plans would really be more optimal than Postgres heuristics at scale and with a bit more realistic OLTP workloads.
>
> — [refibrillator on Hacker News](https://news.ycombinator.com/item?id=49733010), Sep 16, 2026

Bansal never claims a discovery. The result belongs here because it shows how a headline number gets built: selection, one workload, a fixed environment. Readers learned to discount best-of-N in benchmark tables. Discovery campaigns run hundreds of agents and report the one that found something, without the convention of printing the single-attempt row beside the best.

# The scorecard

| Claim | Who did the work | Outside check | Replication | Field verdict | Cost to verify |
|---|---|---|---|---|---|
| NIST curve seeds (Anthropic, Oct 16) | Internal model, 10 named staff, under 30 minutes | Bounty organizer reimplemented in Go, bounty paid | Public CC0 code regenerates every seed | Settles the backdoor theory; candidate selection and B-163 still open | A script |
| Quantum circuits (Anthropic and Harvard, Oct 16) | Model in an evaluator loop, human specifications | Automated correctness checks, Lean proof, 403M sample decodings | Not yet reproduced by an unaffiliated group | Narrow, real gate reductions in a code-golf-shaped task | A simulator |
| MVUEH Enigma (user of GPT-6 Astra, Sep 15) | Disputed: "entirely on its own" versus "researcher-led" | Archivist validated key and plaintext | Second message broken with Claude Opus 5 on Sep 20 | Impressive break, not a first-ever claim | An Enigma simulator |
| AlphaGenome Atlas (DeepMind, Sep 8) | DeepMind plus academic co-authors | ClinVar benchmarks, one minigene assay | A resource; independent tests are only beginning | Leading model, score easy to misread | Benchmarks plus wet lab |
| ART enzyme (Anthropic, Sep 23) | 949 agent sessions; four senior biologists on the paper | None outside the lab; not peer-reviewed | Missed in all 10 reruns of the campaign | Intriguing, preliminary, priority disputed | Months of wet lab |
| qorl query plans (independent, Sep 16) | One engineer, distilled from Astra | Open code; single benchmark | Best-of-15 headline, 1.40x single choice | Watch for overfitting | A Postgres rig |

![Data schematic: six discovery claims on a scorecard grid, four axes per claim, sorted left to right by cost of outside verification.](/post-images/2026-10-17-discovery-is-the-new-benchmark/discovery-scorecard.jpg)

Read down the last column. The claims that hold up best can be checked by running a program: a seed regenerated, a circuit evaluated, a key simulated. The weakest needs months of experiments and a priority search through other people's unpublished work. The order says less about which lab is more honest than about which problems are cheap to grade; Anthropic produced both the best-supported claim and the weakest.

# Discoveries cluster where checking is cheap

Mercor's map makes the same point about benchmarks. It indexed 29 public agentic benchmarks and used an LLM judge, validated against human expert tags, to map each prompt to O*NET tasks. Of 923 occupations, only 208 have a task with at least five benchmark prompts, and 6 of 22 job families have none. Tasks rated as exposed to AI automation are "about 6x more likely to be represented on benchmarks" than unexposed ones, which is why the benchmarks are "full of tasks in Software Engineering and Financial Analysis." Weighted by working time, Mercor's headline is that "90% of American working hours are spent on tasks with little or no coverage from existing benchmarks."

Mercor calls the results "provisional," and the threshold is generous: a top reply on [Hacker News](https://news.ycombinator.com/item?id=50122012) noted that a social-work task counts as covered with 13 prompts. A stricter definition would push the number up.

![Schematic: verification-cost axis from SCRIPT to WET LAB, with the six claims placed along it and Mercor's 90% uncovered band shaded.](/post-images/2026-10-17-discovery-is-the-new-benchmark/verification-cost-axis.jpg)

Put the map beside the scorecard and both show the same pull. Benchmarks concentrate where grading is cheap and automatic, and so do discovery claims: cryptography, ciphers and circuits, where an agent can tell on its own whether it succeeded and an outsider can confirm it the same afternoon. Pachocki's "easy-to-measure capabilities" line applies to both. Choosing such problems hides nothing; a model in a loop works best there. The risk is letting those results stand in for a general ability to find things out, then spending that credibility on claims like the enzyme, where the checking has not started.

# The rerun count is the number to ask for

Benchmark culture took years to settle its norms: held-out sets, fixed harnesses, single-attempt scores beside best-of-N, and outside evaluators. Discovery claims need their own, and the season's best and worst examples show what they are.

The ECDSA post shows the strongest form: publish the artifact that lets anyone regenerate the answer, and invite the person with the most standing to doubt it to try first. The ART report shows the most important number to add, even though it was buried. Ten reruns, zero rediscoveries is the discovery equivalent of the single-attempt row in Bansal's table. It tells a reader whether the result reflects the system's ordinary behavior or one lucky run among hundreds. Anthropic deserves credit for publishing it. None of the other announcements graded here reported a rerun count at all.

Three more items would close most of the gaps above. Release full transcripts, beyond the excerpt in which the model says "spectacular." Run a priority search with outside experts before the press release, not after the dispute reaches the press. And print the human steering alongside the model's work, the way Leffen's case study did. A discovery claimed through press posts will be graded as press. The labs that want it graded as science will have to publish what a referee would ask for.

## Sources

- [Anthropic — Claude reveals ECDSA curve origins (Frontier Red Team, Oct 16, 2026)](https://www.anthropic.com/research/ecdsa-curve-origins)
- [Filippo Valsorda — The NIST Elliptic Curves Seeds Bounty Has Been Claimed (Oct 16, 2026)](https://words.filippo.io/seeds-solved/)
- [Anthropic — Claude discovers a novel enzyme system with CRISPR-like repeats (Sep 23, 2026)](https://www.anthropic.com/news/claude-discovers-novel-enzyme-system)
- [Yoon et al. — Autonomous AI agents discover reverse transcriptases with tandem repeat arrays (technical report PDF)](https://www-cdn.anthropic.com/22573675ada52a8ca8a97a1a4b4326b2f208a071.pdf)
- [bioRxiv — Yoon et al. preprint, posted Oct 11, 2026](https://www.biorxiv.org/content/10.64898/2026.10.05.755809v1)
- [CNN — Katie Hunt, What to know about Anthropic's 'Claude-led' biological discovery, and why scientists aren't convinced (Oct 8, 2026)](https://www.cnn.com/2026/10/08/science/ai-biology-anthropic-dna-discovery)
- [Finkelstein Lab (UT Austin) — Harvesting and Replenishment in Biomedicine (Sep 28, 2026)](https://finkelsteinlab.org/blog/harvesting-and-replenishment-in-biomedicine/)
- [X — Dario Amodei on the Claude-led discovery (Sep 23, 2026)](https://x.com/DarioAmodei/status/2102831170299834652)
- [Crypto Cellar Research (Frode Weierud) — OpenAI GPT-6 Astra breaking an Enigma message that has resisted solution since 2005](https://www.cryptocellar.org/bgac/the-mvueh-break.html)
- [Carter Leffen — MVUEH investigation case study](https://mvueh-enigma-solved.carterl.chatgpt.site/)
- [X — Carter Leffen announcing the MVUEH break (Sep 17, 2026)](https://x.com/carterleffen/status/2100405732935868840)
- [Schneier on Security — GPT-6 Astra Breaks an Old Enigma Message (Sep 22, 2026), incl. Weierud comment](https://www.schneier.com/blog/archives/2026/09/gpt-6-astra-breaks-an-old-enigma-message.html)
- [Anthropic — Designing more efficient quantum circuits with Claude (Oct 16, 2026)](https://www.anthropic.com/research/designing-efficient-quantum-circuits)
- [arXiv 2610.22367 — Gu, Kemeny, Yelin, Lukin: Discovering state-of-the-art quantum circuits with large language models](https://arxiv.org/abs/2610.22367)
- [Google DeepMind — AlphaGenome Atlas: a predictive map of every possible DNA letter change in the human genome (Sep 8, 2026)](https://deepmind.google/blog/alphagenome-atlas-a-predictive-map-of-every-possible-dna-letter-change-in-the-human-genome)
- [AlphaGenome Atlas team — technical paper (PDF)](https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphagenome-atlas-a-predictive-map-of-every-possible-dna-letter-change-in-the-human-genome/alphagenome-atlas.pdf)
- [IEEE Spectrum — Greg Uyeno, AlphaGenome Atlas Maps 9 Billion Possible DNA Variants (Sep 8, 2026)](https://spectrum.ieee.org/alphagenome-atlas)
- [bioRxiv — Wei, Li, Lehner: Complete Mutagenesis of the Genome and Proteome of PhiX174 (Jul 26, 2026)](https://www.biorxiv.org/content/10.64898/2026.07.25.740675v1)
- [OpenAI — GPT-6 Astra: A new generation of intelligence](https://openai.com/index/gpt-6-astra/)
- [OpenAI — Jakub Pachocki, An Alien Mind (Sep 6, 2026)](https://openai.com/index/an-alien-mind/)
- [Rohan Bansal — Training a 4B model to produce 81% faster query plans than Postgres (Sep 16, 2026)](https://rohanbansal.com/qorl)
- [Mercor — A benchmark map of the US economy (Oct 16, 2026)](https://www.mercor.com/research/labor-index/benchmark-map/)
- [Hacker News — Claude reveals ECDSA curve origins (973 points)](https://news.ycombinator.com/item?id=50122800)
- [Hacker News — Claude discovers a novel enzyme system (780 points, 805 comments)](https://news.ycombinator.com/item?id=49820134)

---

Canonical: https://www.thedeepfeed.ai/posts/2026-10-17-discovery-is-the-new-benchmark/
Site: https://www.thedeepfeed.ai
Full corpus: https://www.thedeepfeed.ai/llms-full.txt