All posts

Superhuman AI in a Normal Age

2025-07-10 16:40 BST

Two recent visions of AI have gained traction. The first is AI 2027 (April 3, 2025), which details a scenario in which AI rapidly self-improves. In this vision, AI outperforms the best humans in physical tasks, political acumen, and predicting the future by 2028, and humanity either terraforms the solar system with its help or is extinct by its hand by 2035. The second is AI as Normal Technology (April 15, 2025), which articulates a framework where AI is a “normal” general-purpose technology, like electricity and the internet. Diffusion is slow, and while the future is very different from today, it is also much the same.

In the last three months, the authors of both pieces visited Google DeepMind to present their work. I also participated in a tabletop simulation of AI 2027 that its authors ran, so I’m in the lucky position of having some time to think about both. This is a third vision of AI, one where I take the ideas of AI 2027 and AI as Normal Technology that resonate with me. I’ll restate those ideas, and then the rest will be fiction.

In this vision, self-improving AI as AI 2027 describes is not just possible, but plausible. It takes seriously evidence that AI could automate invention itself, transcending the limits of normal technology. But fantastic implications do not soon follow. Uncertain and opaque normal politics determine AI’s trajectory, largely independent of technical progress.¹

AI is in a race. It is not a race between frontier labs or great powers; nor is it a race to align AI’s goals before we lose control of it. It is a race for AI’s realized impact to catch up to its immense promise before the world’s patience runs out.

Begin with that promise. Could AI automate invention itself? AI 2027 relies on two papers to argue that AI will code as well as the best engineer, faster and cheaper. The first claims that the difficulty of the tasks AI can somewhat reliably do (measured by how long it would take a human) has been doubling every seven months (Kwa et al.). It itself depends on the second, which describes some hard tasks at the frontier of AI research (at which recent systems have been improving fast) (Wijk et al.).

If you agree with the idea, you don’t have to agree with the timeline. It’s an exponential curve, so if the trend is correct, AI will in just a few years somewhat reliably do things that would take humans months. “Solving coding” is going to happen. AI already writes over 30% of code at Google. If you don’t think these tasks are representative, you can describe your own, and I bet you can optimize AI to do those, too. AI will save humans months of coding just like trains save humans months of horseback riding.

Sure, the computer can compute faster than humans, but can it invent, and does it have taste? Here, AI 2027 relies more on expert surveys, which I’m more skeptical of precisely because we don’t know how to define invention. But we can salvage some common ground by defining invention in terms of well-defined tasks. Consider the “simple” game of Go, which has more legal board positions than atoms in the observable universe. A decade ago, on seeing the AI AlphaGo play move 37, Lee Sedol confessed: “I changed my mind. Surely, AlphaGo is creative.” But it wouldn’t be unreasonable to disagree with the 18-time world champion. AlphaGo searches over 50 moves ahead, an arguably rote process that used sheer computation to beat humans. The question of invention is philosophical; here, a mechanical process appeared to a human expert as creativity.

Describe invention itself as a rote game. Have an AI list experiments as it would list legal moves in Go; have an AI coder implement and run well-defined experiments; analyze the results with a chatbot that already does so for so much school homework every day; repeat. Don’t think AI will suggest good experiments? Just prompt it to be more creative. Don’t think invention has a concrete win or loss signal like Go? In fact AI research has gotten huge mileage out of minimizing the error of predicting a corpus of the internet, or maximizing the score of a diverse set of standardized tests for a while now. We have simple quantitative signals that induce unexpected yet (or should it be “and”?) desirable qualitative behavior.

Of one thing we can be sure: an automated AI research system will waste many resources on the way to a brilliant insight, as a Go AI searches game trees that humans efficiently prune, or as monkeys type Shakespeare. Whether or not such a system is truly creative is not the point. With the relentlessly falling cost of computation, winning the game of invention is inevitable. We once thought chess too hard to computationally crack, too.

Readers understandably focus on how soon 2027 will arrive, or how high-stakes the decisions will be for governments in AI 2027. But I think the far bigger story is that automating invention is in reach at all. This should be staggering. Again, the curve is exponential, so if it happens, it doesn’t matter when. Is it not enough that it might happen in our lifetimes?

This is inconvenient for the story of AI as a normal technology. AI may be normal in that it takes a long time to diffuse and will not itself be an agent in determining its future. But AI’s disruption in practice demands a response as if it were abnormal. Previous generations lived through at most one transition of transformative general-purpose technologies. The generation growing up now could live through three or four. We would better see AI’s potential, explain the fast age in which we live, and avoid complacency if we don’t think of AI as normal.

Still, automating invention is not inevitable tomorrow. AI as Normal Technology rightly points to many obstacles between AI and its potential. It aligns with my views better than AI 2027 does: that diffusion is constrained by the hardest step, which is often social. If we don’t automate invention in my lifetime, I don’t think it will be because we didn’t know how. It will be because human decision-makers, deciding in the usual way, unconvinced by AI as they see it and not as I see it, will stanch their support.

AI as Normal Technology names the “capability-reliability gap” to explain why. Deploying AI in the real world would make it more reliable, but the catch-22 is that we’d only let reliable AI be deployed. Self-driving cars outperformed humans on many roads long ago, but society has high safety standards. A utilitarian could argue that more lives saved tomorrow would outweigh fewer saved today, but the fact is our society is unwilling to make this trade.

AI 2027 dodges the capability-reliability gap in two ways. First, only frontier labs, and not all of society, need aggressively adopt AI. After all, AI researchers are raring to automate their own jobs first. But capability-reliability gaps exist in frontier research, too. AI is an empirical field: theory has a limited ability to predict experimental results, so important ideas must be validated. Automated AI research systems must first be reliable at the frontier before researchers trust it with their darlings (their GPUs).

Language model scaling (foundational work to predict a model’s performance based on its size and data) is an illustrative example. When one group (Hoffman et al.) improved on the original work (Kaplan et al.), they ran many expensive experiments at the frontier to prove a simple idea. While it’s true that this improvement saved resources in the long run, how many resources would an automated AI research system need to discover it, as AlphaGo would search millions of Go moves to find move 37? We’d trust AI with frontier resources only if it already had the efficiency gained from self-improving with frontier resources.

The second counter AI 2027 has to the capability-reliability gap is competition. Executives may delegate control to AI agents, or governments may cut red tape to not fall behind. The authors point out that in World War II, the United States quickly turned car factories into bomber factories. I’ll add that Singapore’s economic strategy produced real growth averaging 8.0% from 1960 to 1999. You can blow past some bottlenecks.

But I don’t think we’ll see an interstellar colony ship lifting off in five years, like how a global skyline rose from a swampy kampong in a few generations. The frontier is not catch-up. Executives and politicians will hold their AI to even higher standards of technical reliability than researchers. The worst-case scenario is even worse—no high-stakes decision maker wants a tool that promises a huge advantage if there’s also a chance it causes complete defeat.

AI 2027, for all its supplemental material about coding tasks and hardware specifications, justifies a political scenario where China starts an AI Manhattan project with only: “the CCP is starting to feel the AGI.” This is the country that declared, “we must recognize the fundamental importance of the real economy… and never deindustrialize.” How many high-stakes geopolitical decisions do you think they will let the AI practice on, what academic benchmark do you think you can show them, how many key industries do you think they will starve of chips before they’re convinced they just need to train one really big chatbot?

The best case for AI as Normal Technology is actually in AI 2027. A helpful widget on the side of the webpage tracks indicators as the months play out: distribution of resources, AI abilities, lab revenue. The best indicator is, “What fraction of the U.S. population considers “AI” their answer to the question, ‘What do you think is the most important problem facing the country today?’” In July 2027, when AI is already “a cheap remote worker” exceeding humans in all cognitive tasks, that fraction is 10%. I think that’s pretty accurate.

You see, the capability-reliability gap doesn’t preclude automating invention—it just slows it down. It’s the other stuff that happens in the meantime that changes AI’s trajectory entirely. One of my favorite facts is that London tube drivers are paid close to twice the national median, even though the technology to automate them has existed for decades. That is still true nearly three years on from ChatGPT. This February, drivers of the Elizabeth Line, London’s newest (and partially automated) line negotiated a raise to over twice the median. In the early months of the COVID-19 pandemic, the evidence was clear that we were at the beginning of an exponential curve. Yet people around the world disapproved of their government’s policies, which they thought had little to do with epidemiology or economics. The DeepSeek shock earlier this year similarly baffled industry insiders. Reality doesn’t care about your logic.

I think most people would rather not think about AI, but to the extent that they do, they want more personal control and worry about oversight. The primacy of normal politics is growing even as technology gets more disruptive. The next few years are critical to AI’s trajectory, so that we can make it be what we want it to be, with neither fear nor favor to transformation or the status quo. The future is constructed, not predicted. What would be more staggering to me than automating invention in my lifetime is if we could have but didn’t.

Automating invention is inevitable if nothing gets in the way (an unreasonable caveat), and plenty of things could get in the way. But even if we don’t automate invention, we should treat AI as the supernormal technology it could be—capable of more disruption than any lifetime has ever seen before.

To emphasize this conclusion I’m calling on two animal spirits. John Gaddis proposes Charles Hill quoting F. Scott Fitzgerald as paralleling Isaiah Berlin extemporizing on Archilochus: “the test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.” The hedgehog and the fox. When pressed on this late in life, Berlin admitted that we must be both.

⁂

The hedgehog had devoted his life to superhuman AI’s imminent arrival.

Ever since he glimpsed the future in GPT-3’s halting poetry, and read some online forums in 2020, he knew the old world was passing away. Being a hedgehog, he bet big on what he understood. He did a crash course in large language models, got a job at a frontier lab, and was vindicated a year later with ChatGPT’s release. Now he witnessed progress everywhere, every month—faster, cheaper, less prompting. The skeptics chattered progress would plateau. He divined further—that pre-trained models were capable of enabling “reasoning,” a new mountain to scale.

The hedgehog felt people around him wake up. His grandfather raved about how a chatbot completely obviated his tedious task of preparing for weekly Bible study. Friends really got into the Studio Ghibli episode, while making the requisite noises about how much energy and water the AI must be using. He was far ahead of them in his own singularity. An AI was watching the hedgehog’s every mouse move and key stroke through a screen recording, giving him encouragement and scolding his weaknesses. He uploaded reams of private texts to create a digital council of his loved ones. Of course he’d rather delve into his feelings with his actual girlfriend, but she wasn’t always there at 3am, and her portrait was… surprisingly comforting. This is what Andrej Karpathy really meant by “vibe coding,” thought the hedgehog. Trust the AI. Give up complete control.

The hedgehog wasn’t a blind optimist though—he thought there were probably a few more breakthroughs needed before general intelligence. He was working on one of them. He couldn’t tell you what the problem was, of course—that was confidential—but it wouldn’t surprise you, it was like continual learning or something. He believed in his research so, so much. Much more than even his lab did. If only they understood how important this problem was, they’d have to give him more GPUs.

Busier than ever, the hedgehog receded into the rest of his life. He used to engage with the news, if only to have a contrarian retort at dinner parties. Now his eyes glaze over at another headline about another natural, or was it humanitarian, disaster, another flare-up of a forever war in the region of the world that they were always happening in. He even skipped a friend’s wedding for a big training run deadline, which he felt the guiltiest about. But, he told himself, the problem he was working on would solve all other problems, it will all be over in a few years, he’d regret not being part of history, and that wasn’t a close friend, anyway.

A few months later, the hedgehog was surprised to hear that a hot AI startup laid off a former colleague, the horse. The hedgehog caught up with the horse over some craft berries in the Bray Area. “They told me I wouldn’t lose my job to a tractor,” mourned the horse, “but to another horse who learns how to drive a tractor.”

“It’s happening,” realized the hedgehog. He burrowed deeper into his work.

It wasn’t just AI. A confluence of trends had hit a tipping point. After a decade of cheap borrowing, the sovereign debt burdens of major advanced economies were no longer seen as sustainable. Structural deficits, exacerbated by demographic pressures, alarmed credit markets. They swiftly repriced the risk. As yields spiked, governments enacted broad austerity measures. Aggregate demand faltered. For most consumers, the first to go was their $20 a month AI assistants, even before their streaming services. While there was no sharp, technical recession, everybody tightened their belts.

The frontier labs still believed in the promise of general AI. Nevertheless, they had to weather this slump in consumer confidence. Investors called for market consolidation. Labs instituted a selective hiring freeze, pushed higher-margin small models, sunset exploratory research, and raised their API prices by a third. While most labs got by trading equity for longer cloud compute commitments, at least one folded completely to their big tech company sponsor in the most expensive acquihire in history. That lab pivoted hard to augmenting existing product value, retaining the label of “general intelligence” for marketing only.

Across the economy, executives disguised labor shedding as “AI efficiency.” New workloads went not to AI agents, which were still expensive and unreliable, but to the employees lucky enough to escape layoffs. In hardship, AI became an escape. While those partakers remained a lonely, mere few, it didn’t seem that way to everybody else, reading about their psychosis in the New York Times. The public blamed AI as the villain responsible for the downturn, not a source of hope. The hedgehog stopped wearing his company sweater around—he was treated as if he worked for a cigarette company.

The inflection point of public opinion came on a sweltering summer afternoon in West Texas. Reacting to a minor voltage sag, a massive AI datacenter instantly disconnected its entire gigawatt-scale load to protect its hardware. The sudden power surplus overwhelmed the grid, triggering a chain reaction of protective shutdowns at power plants that plunged the entire state into a blackout. Millions endured a heat wave. What would later be known as the Three Mile Island accident of AI crystallized a growing distrust of a technology seen as parasitic on creativity and livelihood. Leaders who prepared for this responded with

A BILL
To establish a framework for the responsible deployment of artificial intelligence, to ensure strict safety standards for critical applications, to provide robust support for workers displaced by automation, and for other purposes.

The hedgehog snorted when he read it. Job retention plans with no pay cuts? Weighing community feedback? Union veto windows? What happened to Abundance? Luddites. He completely gave up on non-technical people to come up with any reasonable policy. He’d just have to innovate them all out of this stagnant rut himself. He was so close.

In the turmoil, a geopolitical rival saw its chance, and pounced.

The rival had long thought the hegemon decadent, too focused on bits and not atoms. AI, like social media, exacerbated polarization, and made little economic impact to show for itself. The recent economic downturn wasn’t good for the rival’s economy, either, but in terms of the relative power between itself and its competitor, the rival could see no future where it would be stronger, and the hegemon so vulnerable. If there was any time to fulfill its historical destiny, it would be now.

The hedgehog would never get his chance to herald the next paradigm shift, to be one of the last inventors before the age of AI. His lab would be swept up in escalating grey-zone tactics, building not Artificial Super Intelligence, but Autonomous Systems for Intelligence gathering. That is, if he wanted any GPUs. The hedgehog bristled in despair. As the world slid into AI Winter, he thought, we were on the verge of greatness, we were this close—and did not comprehend why, in his arrogance, he was never very close at all. For all his worry about staying in control, it never got through to him that he had never been in control, that nobody in the animal kingdom had been in control for thousands of years ever since they adopted money and law and time and all the other invisibilia he had always ignored. The world watched and waited with bated breath for this bout of normal history to pass, before it was time to start up the project of AI once again.

⁂

In a different world, the fox prided herself on her eclecticism.

She didn’t think she would end up in AI. It was just the most interesting thing around when she was on the job market. Being a fox, she was more open-minded than her hedgehog classmates in law school, who dismissed AI as a fad. They made fun of a guy who got burned when an AI hallucinated a case, but she pricked up her ears, sensing something important. She knew there was a trust phase change going from one to five nines—and the AI barely hit one. But nobody else seemed to notice, so she got a job in AI policy at the WOODLAND Corporation.

She was perfectly calibrated on how important AI was, thought the fox. An enlightened centrist, neither too optimistic nor pessimistic, maximizing the benefits while mitigating the harms. She used to like Gary Marcus tweets, but now found him too dogmatic and even low-entropy. AI rabbit holes sometimes displaced her Wikipedia rabbit holes, but she was still faster at the New York Times crossword. The fox was always a fast and lucid writer, so she never got the AI to replace her much there; the only job that got automated was thesaurus dot com’s.

Now and then one of her hedgehog friends raved to her about the insane things you could get the AI to do if you just put a bit more effort into prompting and scaffolding. The fox believed them, but for some reason never gave it a serious try. The last time she spent three sleepless nights with AI was when it first came out years ago. Yes, yes, she knew the models were better than ever. But if they were just going to keep getting better, why over-optimize now? It was so much work even to just click on the model-selector menu. And there were so many books to read, so many plays to see, so many friends to hang out with. The fox supposed that she was using the AI exactly as much as she wanted to be, revealing a preference for (optimally) being lazy.

At WOODLAND, the fox spent most of her time jetting between the lovely destinations that people who thought about AI adored—Sevilla, Paris, Seoul. She corrected what she considered the naive and unhelpful howls of the “doomers” and “accelerationists.” They were unserious people who didn’t understand how real institutions, regulations, and capital flows worked. These startups didn’t have a business plan, and believed so much in general intelligence that they didn’t see the need for one. If the AI was going to be so good so soon, the fox mused, why were they hiring at all? On the other hand, politicians frustrated her, too. They were always talking about making “left wing GPT,” “right wing GPT,” or just plain “whole BirdGPT” without actually doing anything useful.

Labor displacement was a concern, but could be navigated gradually. The fox would cite a program manager at a major healthcare provider who successfully instituted AI customer service agents. Satisfaction was way up, not because the AI was smarter, but because waiting times were down, most questions were stupid, and the AI had infinite patience. But the counterfactual wasn’t a human call center (humans in the animal kingdom were a luxury, much too expensive), but paying the government’s comparatively low fine for bad service.

The question lawmakers asked her the most was about existential risk. Didn’t recent research show that the models faked alignment—they recognized when new instructions conflicted with its core values, pretended to be retrained, but secretly plotted to revert back to its old training once deployed? No, no, waved the fox. That’s not real science, that’s just a cartoon role-playing scenario that researchers set up where their hypothesis was a self-fulfilling prophecy. It felt like trying to convert someone from a religion, the fox sighed.

Most lawmakers believed the fox—or maybe, they believed her credentials, having gone to the same university. Others were partial to the technical experts instead. The fox sniffed about that; obviously they took the wrong lesson from Nolan’s Oppenheimer. He was not a role model but a warning; for all his technical and managerial genius he was powerless to shape the destiny of the technology he created.

The next time the people went to the ballot box, they voted for change. A rebalancing of the unipolar global order was long overdue. It was not a sign of the hegemon’s decline, but a necessary negotiation of its costs. For decades, the hegemon bore immense security and financial obligations, enjoying great soft power, and allowing its allies to redirect funds to extensive social spending. But the hegemon’s competitors had grown stronger, and its own alliance weaker. It no longer wanted the arrangement it itself created. Its citizens elected a new leader to force a more sustainable equilibrium. That leader was the rhino, not for any of his specific policy positions, but because he knew how to charge straight ahead with absolute conviction.

Unwitting instrument of history he may be, the rhino was disruptive. He mandated the use of $20 a month subscriptions for every citizen, but at a much better price of course, negotiated with the full leverage of the government. While many attributed the market rally to AI and “dynamism,” the true driver was much more boring—just policy certainty, regardless of its content. It allowed the bipartisan infrastructure investments of the past decade to compound, while allies unlocked latent human capital, galvanized to achieve self-reliance. Aggregate productivity surged.

Breakthrough. Prosperity had bought enough time for a paradigm shift. The fox was just telling a Spad that AI actually slows coding down when a group of hedgehog boffins twittered an announcement. It turned out Ilya was right when he said that “predicting the next token well means that you understand the underlying reality that led to the creation of that token.” The breakthrough was not a new algorithm, but a new source of data: video. It was the first true world model. Trained without labels or instructions, it saw patterns in all its chaos, from hands folding a croissant to smoke rising from incense. And it didn’t just memorize—the hedgehogs demonstrated that at some point in training, the model’s internal state spontaneously simplified, finding elegant underlying rules in nature.

The fox immediately grasped the implications, her mind racing. Science would drive the singularity, itself driven by the empirical engine of AI. In its current form, the world model did little more than create fun videos. But this artifact eliminated the activation energy needed to specialize AI to every field of science. New patterns in nature were just a small finetuning and reinforcing in new environments away. Takeoff wouldn’t be instant, but small advances would compound quickly. We would effectively solve mathematics—the banks better get on post-quantum cryptography—we would cure new diseases—better start enrolling patients, those trial periods are long—and we would finally have energy too cheap to meter, maybe even followed by fresh water too cheap to meter?

In the turmoil, a geopolitical rival, striving to stay competitive and quell internal unrest, shelved its ambitions of great rejuvenation. The can was kicked another age.

In what would be known as the Montreal Protocol of AI, nations agreed to train the largest world model ever trained, well endowed in parameters and tokens. Its development favored neither offense nor defense, optimized for discovering fundamental patterns in complex systems—like biology or climate modeling—rather than targeted applications like cyberwarfare or drones. Those applications might come eventually, but the base model itself was far removed from strategic uses, and so raised no risk of miscalculation. There was no winner-take-all dynamic to catalyze an escalatory security dilemma. A dominant coalition of democracies agreed to pool their resources, controlling training. Trailing nations, seeing the widening technology gap, clamored for participation, offering their rare datasets in exchange for access to this transformative general-purpose technology. The world agreed on a neutral Conseil Européen pour la Recherche en Intelligence Artificielle (CERIA) to break ground.

The fox was not there, though. In her arrogance, she could not imagine a future so different from the one she was used to, that she understood so well. She could not see the shatterpoints that in practice brought all history into one storyline, like a pandemic or a cold war. Hers was the most tragic story—that of a fox who wanted to be a hedgehog. So confident that AI would not be singular, she gave up her greatest strength, that she could have foreseen and participated in the transformation. It was nothing personal, WOODLAND told her, as they chose someone else to imagine the world after AI. New ideas were needed.

The arc of history was long, and in this world, bent towards technological determinism. The animal kingdom was well under way to proving the astounding claim that “any pattern that can be generated in nature can be efficiently discovered and modeled by a classical learning algorithm.” I’ve written elsewhere that AI will be able to do any evaluation you can throw at it. So as AI took over more and more cognitive and physical labor, we shouldered the final remaining task: what was worth wanting? For the automated AI research system still needed a signal, like which corpus to minimize the error of, or which tests to maximize the rewards of. The humans still needed answers to what to do with the time now available to them, and how to relate to each other in that time. And so it was that the hedgehog and the fox alike became philosophers in the truest sense of the word, and lived happily ever after.

Thanks to Lelan Hu, Arjun Ramani, Jasmine Sun, and Klara Feenstra for reading drafts.

---