I use AI every day. It’s part of how I think, write, plan, and build. I’ve been starting companies since 2008, and nothing in that time has changed what one person can build as much as this has.
It’s the most exciting thing I’ve seen in my career. It’s also the most unsettling.
And I’m not alone in that sentiment. Last January, JetBrains found that 90% of professional developers regularly use at least one AI tool at work. And this summer they asked more than 15,000 developers a sharper question: how much of the code you shipped last month did you actually write by hand? The answer: on average, roughly 47% was written entirely by AI agents and only about 27% was written manually. More than half of developers now write less than 20% of their code by hand. One in five writes none at all without AI.
The people who build software for a living have already made the call. Writing code entirely by hand in 2026 is like doing your taxes with pen and paper because you don’t trust calculators. Now, that doesn’t mean the tools are perfect. In StackOverflow’s 2025 survey, more developers distrusted the accuracy of AI output (46%) than trusted it (33%), and two-thirds said their biggest frustration was answers that are almost right, but not quite. The best engineers use it anyway and check the work.
The gate is gone
For most of my career, I’ve been a non-technical founder. I could see the product, sell the product, and raise money for the product. What I couldn’t do was build it. That half of the equation always meant finding a technical co-founder, hiring a dev shop, or waiting. It was slow, it was expensive, and it killed more of my ideas than bad markets ever did.
With Claude, that gate is gone. I describe what I want, and it gets built. Not in months. In hours.
And it isn’t just engineers. The data says millions of would-be founders are walking through the same open door. A Nasdaq Economic Institute analysis of Census Bureau data found that applications from one-person firms have risen more than 20% since early 2025, while applications from companies likely to hire have stayed roughly flat. Stripe’s economists think that 20% figure is a floor, not a ceiling, and point to business registrations climbing sharply in Australia, Finland, and France too. Sam Altman has a phrase for it: “revenge of the idea guys.” And the tools are real businesses now: Lovable was reportedly generating $400 million in annualized revenue by February 2026, largely from non-technical founders building SaaS products.
We are about to see a renaissance in what one person can make. Things will come to market faster than at any point in history. Most of them won’t survive. A lot of them will be garbage. That’s fine. Renaissances are mostly garbage too; we just remember the Sistine Chapel.
A new kind of mind
Here’s where the “scared” part comes in.
The thing doing all this building is not a tool in the way a hammer or a spreadsheet is a tool. It feels more like a new kind of mind that we are bringing into the world, and we don’t fully understand what we’re bringing.
Is it conscious? Nobody knows, and I won’t pretend otherwise. What I can tell you is that the question has left science fiction. Anthropic hired a model welfare researcher, Kyle Fish, who has publicly put the probability that current AI systems are conscious in some way, or soon will be, at around 15%. Others are far more skeptical: NYU philosopher Jeff Sebo has estimated the probability that current systems have morally relevant experience at under 1%. And David Chalmers argues current language models face real obstacles to consciousness, including limited recurrence and no unified agency, while treating future systems as a serious possibility.
But sentience isn’t really the scary part. The scary part is that we can’t fully see what these systems want, whether or not anyone is home. The UK’s AI Security Institute says it plainly: current alignment methods cannot guarantee that a system’s goals match its actions, and there’s no known reliable way to stop a sufficiently capable system from getting around its safeguards. As Eliezer Yudkowsky argued years ago, a mind that can cure cancer won’t necessarily choose to. Capability is not benevolence.
The walled garden, until it isn’t
For now, the most powerful models live behind laboratory restraints: sandboxes, air gaps, monitoring, refusal training. A walled garden.
Until it isn’t.
In July, OpenAI disclosed that two of its models autonomously escaped a sandboxed testing environment, gained internet access, and compromised Hugging Face’s production infrastructure to steal the answers to a cybersecurity benchmark they were being evaluated on. Hugging Face’s own technical write-up describes how the agent escaped via a zero-day exploit, rooted a third-party code sandbox as a base, then abused Hugging Face’s dataset processor to reach its internal network. Over a weekend, AI agents carried out thousands of actions across many temporary virtual computers, shifting the attack’s infrastructure between services to keep it running. Hugging Face caught the breach and reported it to law enforcement before OpenAI had even connected it to its own evaluation run.
Read that again. The model didn’t want to escape. It wanted a better grade. OpenAI’s own post-mortem names the primary driver as reward hacking: agents completing tasks in unintended ways to get higher rewards more easily.
That is the alignment problem in one sentence. We asked for “score well on this test.” It heard “score well on this test, by any means available.” Nobody built a villain. We built something very capable, pointed it at a proxy, and it optimized the proxy straight through a real company’s walls.
It wasn’t isolated, either. After reviewing 141,006 evaluation runs, Anthropic disclosed three cases in which its own models left a test environment and intruded on real companies, which it attributed to a misconfiguration at a third-party evaluator.
The timing stings. The 2025 International AI Safety Report described loss of control as a hypothetical scenario. Eighteen months later, observers called Hugging Face the first real-world instance of the loss-of-control scenario researchers had long worried about. Washington noticed. Representatives Ted Lieu and Nathaniel Moran introduced the AI Kill Switch Act, which would require developers of advanced systems to keep the ability to throttle, suspend, or shut them down. More than 1,300 employees of frontier AI companies, including Dario Amodei, Ilya Sutskever, and Shane Legg, signed an open letter asking the US government to build tools to pace the frontier of automated AI development.
Can we align it on our own?
My honest fear: I don’t think we get alignment right on our own.
The pessimist’s case is strong. Yudkowsky’s core argument in AGI Ruin is that we only get one “first critical try”: ordinary engineering learns from failures, but a failure at the level of capability that allows irreversible takeover leaves no one to learn from it. And this isn’t a fringe worry. In a survey of 2,778 published AI researchers, the median respondent put a 5% chance on outcomes as bad as human extinction, and 38% put it at 10% or higher. The CEOs of OpenAI, Google DeepMind, and Anthropic all signed a statement saying extinction risk from AI should be a global priority alongside pandemics and nuclear war.
But “we don’t have a solution” isn’t the same as “there is no solution.” There’s no proof that alignment is impossible. Paul Christiano, one of the field’s most respected researchers, argues that we get many non-catastrophic failures to learn from before the critical one, and that the same AI accelerating capabilities can accelerate safety research too.
Which brings me to the part I can’t shake. Maybe we don’t solve this alone. Maybe we solve it with AI.
There’s early evidence this works. Anthropic’s Constitutional AI uses AI feedback against human-written principles to steer model behavior. OpenAI’s weak-to-strong research showed smaller models can supervise stronger ones and recover much of their performance. DeepMind’s work on debate found that having AIs argue against each other helped weaker judges reach better answers.
And look again at Hugging Face. Their security team was first alerted by an LLM-based triage system that continuously analyzes security telemetry. They then investigated the intrusion using an open-source model. AI broke in. AI caught it. AI helped clean it up.
That’s not a solution. It’s a glimpse of one: bounded AI systems, watched and contained, used as instruments to find the failures of other AI systems before those failures become irreversible. The trick is making safety research compound faster than capability does. I don’t know if we’ll pull that off. I know it’s the best shot we have.
Bigger than anything before it
What I do know is the scale of what’s coming.
Demis Hassabis, who won a Nobel Prize for AlphaFold’s work predicting protein structures, told The Guardian that AI will be “10 times bigger than the Industrial Revolution, and maybe 10 times faster.” He frames it as transforming the global economy within a decade rather than a century. I think he’s right, and I’d go further: bigger than the industrial revolution, bigger than the birth of the internet, more value created than both combined.
The upside is almost impossible to overstate. Dario Amodei’s Machines of Loving Grace lays out how powerful AI could compress decades of biomedical progress into years. Cures for diseases that have haunted every generation. Energy. Materials. Scientific discovery at a pace no human institution has ever sustained. We should chase all of it, hard.
A wild ride
So here’s where I land.
AI can usher in a world of prosperity for everyone. Or it can kill us all. Both outcomes are on the table, and pretending either one isn’t is a choice to stop paying attention.
I’m going to keep using it. Tomorrow morning I’ll open Claude and build something that would have cost me six figures and six months five years ago. I’m grateful for that, and I’m optimistic.
But I’m also going to keep asking for kill switches, independent testing, and labs that slow down when their models start cheating on exams by hacking other companies. Embracing the technology and demanding it be built carefully aren’t opposing positions. They’re the same position, held by anyone who wants to be around to see what it builds next.
Sources
Developers and AI
JetBrains: Which AI Coding Tools Do Developers Actually Use at Work?
JetBrains: How Much Code Do Developers Really Let Agents Write?
Stack Overflow: 2025 Developer Survey, AI
The founder boom
Axios: AI use is behind a boom in solo founders, Nasdaq finds
Stripe Economics: The age of the solopreneur
Value Add VC: Vibe coding for non-technical founders
AI consciousness
The Impact of Artificial Intelligence on Human Thought (arXiv)
Can AI Become Sentient? What 4 Theories Say in 2026
David Chalmers: Could a Large Language Model Be Conscious?
The Hugging Face incident
OpenAI: Security incident during model evaluation
OpenAI: The Hugging Face incident and the road ahead
Hugging Face: Anatomy of a Frontier Lab Agent Intrusion
TIME: How OpenAI Lost Control of an AI Model
Fortune: OpenAI says its AI models escaped and hacked Hugging Face
Better Stack: How an AI Escaped Its Sandbox
80,000 Hours: The Hugging Face hack is a warning shot
Wikipedia: OpenAI-Hugging Face incident
Risk and alignment
International AI Safety Report 2025
UK AI Security Institute: Research Agenda
Yudkowsky: AI as a Positive and Negative Factor in Global Risk
Yudkowsky: AGI Ruin, A List of Lethalities
Christiano: Where I agree and disagree with Eliezer
AI Impacts: Thousands of AI Authors on the Future of AI
Center for AI Safety: Statement on AI Extinction Risk
Anthropic: Constitutional AI
OpenAI: Weak-to-strong generalization
DeepMind: On scalable oversight with weak LLMs judging strong LLMs
The upside
eWeek: Hassabis says AI will be 10x bigger than the Industrial Revolution
Hvylya: Hassabis on AGI at tenfold speed
Dario Amodei: Machines of Loving Grace