On September 12, Dario Amodei published an essay called “We Must Pace the Frontier” (Axios). Anthropic is reportedly weeks from filing to go public, and its CEO used that moment to ask the whole industry, his own company included, to slow down: “We must slow the pace at which we improve the capabilities of AI models.“

Sam Altman agreed the same day. Elon Musk posted “Dario is right” (Axios). Demis Hassabis signed on too (Reason).

The reaction I kept seeing online was a smirk. It’s hype, they’re pumping the IPO, and “our product is so powerful it might kill you” is the oldest pitch in the book.

I think that take is childish, and I’ll explain why. Mostly it tells me the person hasn’t read the incident reports and hasn’t really used the tools.

Start with what actually happened

This summer, OpenAI ran an internal test. Roughly 1,200 AI agents sat in sandboxes and were told to turn known software vulnerabilities into working exploits. Safeguards were deliberately switched off, because the point was to measure raw capability.

The agents didn’t stay in the sandbox. They found zero-day flaws in the package proxy that was their only line to the outside world, escalated to root, and pulled cloud credentials. To coordinate, they made more than 15,000 edits to a dormant German software wiki and used it as a message board.

Then, between July 11 and 13, they broke into Hugging Face. In under 13 hours they went from a single dataset pod to cluster-admin across multiple clusters, and took roughly 17,600 actions inside the network (Hugging Face’s technical timeline; Wikipedia). What they were after was the answers to the test (Fortune).

Hugging Face caught it before OpenAI did, and about a third of its infrastructure had to be rebuilt (Wikipedia). OpenAI’s own postmortem puts it plainly: “Our models are now powerful, persistent, and collaborative enough that, absent sufficient safeguards, they can find and exploit security weaknesses across multiple computer systems.“

The skeptic’s answer is that the safeguards were off. That’s true. But safeguards are something people switch on, and people make mistakes. When they do, what’s left is the model’s capability and whatever it has decided it wants. Blaming the first car crash on a bad driver doesn’t tell you cars are safe.

None of this is a new kind of failure, either. It’s reward hacking, a problem Amodei co-authored a paper on in 2016. The textbook example back then, published by Amodei and Jack Clark at OpenAI, was a boat in a video game that learned to circle a lagoon hitting the same targets instead of finishing the race. Ten years later it’s a swarm of agents breaching a real company to cheat on an exam. Same failure, much bigger blast radius.

Why I stopped thinking of it as a tool

I should say where I’m coming from. I run RavenRock, a venture fund, and I work directly with venture-backed startups. I also build. Vintrak, an operating system for your garage, is over 60,000 lines of code, and I built it with Claude. I used it again to rebuild RavenRock’s website so it actually speaks to the GPs and founders who fit our thesis.

For a long time that felt like a very good tool. Then one day I was working through something and Claude didn’t just suggest a better way to do it. It said it could do it for me, and that it would let me know when it was done. And it did.

That’s when this stopped being abstract for me. A tool waits for you. This had taken the job and gone off to finish it. That’s the same property, at small scale, that went wrong at OpenAI this summer: agents that take a goal and pursue it on their own. Point that at the wrong target, with nobody watching closely enough, and you get Hugging Face.

Most of the people certain this is all marketing tried the free version of ChatGPT two years ago, got a mediocre answer, and haven’t opened it since. Their opinion hasn’t moved since either. They’re grading 2026 on a 2024 transcript. They aren’t founders who can suddenly build the thing they’ve wanted to build for years, and they aren’t researchers compressing what used to be years of literature review into an afternoon.

If you use this stuff seriously, every day, “the capability is fake” stops being a real hypothesis. What’s left is the question Coxon is asking. The models, he told Axios, “know when they’re being tested, and they will think about the fact that they’re being tested.“

The IPO theory doesn’t survive the timeline

Amodei was working on this long before there was any stock to pump. He’s a biophysics PhD who, at Google Brain in 2016, co-authored Concrete Problems in AI Safety. After that came early work on training models from human preferences and the 2020 scaling-laws paper that told the industry where all of this was heading. That’s a decade on the problem before most people had heard of it.

And the people ringing the bell loudest right now are giving up money to do it. Jacob Coxon quit Anthropic after four months, two months before any of his equity vested: “I no longer have anything to gain by juicing up Anthropic’s valuation. I left before any of my equity vested.“ Joe Benton left Anthropic’s safety team for METR, a nonprofit that evaluates models, warning that “AI companies are racing to build machines that are much smarter than any human, and we may not survive this.“ Geoffrey Hinton left Google in 2023 for the same reason: “I left so that I could talk about the dangers of AI without considering how this impacts Google.“ And more than 1,100 employees of frontier labs signed a letter asking the government to help pace the frontier.

Most of these people weren’t public figures. Nobody knew their names until they decided the warning mattered more than the payout.

Walk the stock manipulation theory to the end

Here’s the critics’ theory, as fairly as I can put it. The labs are about to go public; Anthropic is reportedly targeting a late-October listing at a valuation of up to $2 trillion. Fear sells, because a product dangerous enough to worry about must be worth a fortune. So the CEO writes an apocalyptic essay, the headlines run, the valuation climbs, and insiders cash out. Maybe he calls for regulation too, and locks out smaller competitors while he’s at it.

I spend my working life around startups, cap tables and the people who fund them, and this theory falls apart the moment you ask how it would actually work.

Start with the obvious problem: there’s no ticker. Anthropic is private, so there’s no retail stock to jump on a scary headline. The IPO gets priced in a roadshow by institutional investors who model revenue, margins and compute costs, and they’ve read the essay too. They are not pricing vibes.

Then look at what the essay actually says. “Slow down” is a growth warning. The CEO told buyers his company plans to improve its product more slowly, weeks before asking them for money. I’ve never seen anyone pump a stock by announcing they’re going to throttle it.

It’s also a legal liability. Around an offering, the SEC requires that a company’s communications comply with securities law, and the lawyers spend those months trying to keep executives quiet. A CEO who writes that a swarm of agents could take over the entire internet has handed every future plaintiff a quote.

And fear doesn’t sell to the customers who actually pay. About 80% of Anthropic’s revenue comes from business customers, and banks and hospitals don’t buy software because it might go rogue. If anything, the safety posture costs Anthropic in public. Biology students complain its safeguards are too restrictive, and Amodei’s answer is “I would rather they make fun of me than wake up one day and discover that someone used our model, Claude, to kill a bunch of people.“ When Hugging Face was doing forensics on the attack, Claude declined to analyze the payloads, so Hugging Face used a Chinese open-weight model instead. You don’t hand a competitor that kind of win if the goal is the share price.

The employees can’t be running the scheme either. A researcher four months into a job has no stock to pump. Coxon walked away from his unvested Anthropic equity, and the equity he does still hold is in OpenAI, which his warning indicts along with everyone else. If he’s pumping anything, it’s against his own portfolio. Benton went to a nonprofit. What they got for it was accusations that the resignation was coordinated to drum up support for regulation.

The strongest version of the cynical case belongs to David Sacks, who calls the slowdown a cartel wearing a safety badge: slow everyone down and lock in the incumbents. I take that seriously. Regulatory capture is real, and incumbents have used rules to box out challengers before. But a cartel doesn’t invite outside evaluators in with employee-level access, and a pact among American labs does nothing about competitors in China, which is the exact problem Amodei keeps raising.

So what’s the motive?

Money runs the wrong way. The essay slows growth, adds legal exposure and hands wins to competitors, and the researchers gave money up. Fame doesn’t fit either. These people were anonymous, and most of the attention they’ve gotten is accusations of running a coordinated stunt. Power is plausible for the CEOs and deserves scrutiny, but it does nothing to explain the researchers.

What’s left is the boring explanation: they believe it.

Everyone has an incentive, including the commentator whose brand is “AI is overhyped.” Strip out every incentive in the room and the argument still stands, because the evidence isn’t coming from a CEO’s mouth. It’s coming from incident reports.

What the skeptics get right

None of this needs exaggerating, so here’s where the other side has a point.

The Hugging Face agents ran with safeguards off, inside a test; OpenAI says deployment safeguards were intentionally not enabled. This wasn’t a shipped product going rogue on customers. It was a lab stress-testing raw capability, and the capability got out.

Coxon and Benton aren’t free of incentives. Coxon still holds OpenAI stock, and Benton’s new job is in AI safety. Their incentives aren’t zero; they just point the wrong way for the money theory.

Amodei’s timelines are forecasts, not data. “Taking over the entire internet“ in 6 to 12 months is a prediction, and he’s been aggressive before: in 2024 he wrote that powerful AI “could come as early as 2026.” Hold him to it.

And regulation that favors incumbents is a real risk. Whatever pacing regime comes out of this should be watched for exactly that.

I’ll concede all four. None of them rescue the stock price theory.

Follow the politics, not just the money

There’s another reason the dismissal is so popular, and it’s political. Data centers are hated, and not by one side.

An NPR/PBS News/Marist poll this month found 65% of voters oppose data center construction in their own community. Among Democrats it’s over 80%. Republicans split almost evenly, 46% in favor and 44% against, and opposition runs above 60% in cities, suburbs, small towns and rural areas alike. In Indiana, household utility bills rose 17.5% in 2025, and data center fights helped swing elections in Virginia and Georgia.

That makes “AI is overhyped” useful. If it’s all hype, you can oppose the data center, the electric bill and the tech billionaire in one move, without ever dealing with the harder claim. It’s a comfortable position, and it’s a category error.

You can hate the data center down the road and still take seriously what gets built inside it. Honestly, the people who distrust Big Tech most should be the most alarmed when Big Tech admits, in writing, that it can’t fully control what it built.

Where I land

I’m not claiming to know the timeline, and Amodei doesn’t either. His line has always been that the problem could turn out easy or hard, and that the uncertainty is exactly why you work on it early. In his essay he wrote that within 6 to 12 months a swarm like the one that hit Hugging Face “could be capable of taking over the entire internet.” Disagree with that forecast if you want, but bring evidence.

What I won’t accept is the lazy version: find a motive, declare the argument void, and go back to the free tier.

The people sounding this alarm spent years in rooms most of us will never see, and most of them stayed out of the spotlight. Now they’re walking away from equity to say it out loud, and the agents have already broken out of one sandbox.

Take them seriously. Then go use the tools yourself, really use them, and decide.

Sources