What an Ex-Anthropic Researcher’s Warning About Human Extinction Really Means

Since then, AI leaders such as OpenAI CEO Sam Altman have walked back some of their previous comments about AI’s potential for harm or whether it will create mass unemployment. Anthropic publicly acknowledged earlier this year that it has loosened some of its safety standards to keep up with competitors, including OpenAI and a swath of Chinese companies that have been releasing cheaper, high-performing AI models.

That competition, with both OpenAI and Anthropic hurtling toward initial public offerings, has raised concerns that, instead of moving cautiously in deploying AI models, industry leaders are rushing into the unknown and releasing models that could pose present and future dangers. This was the case when Anthropic released advanced AI models, which ended up being pulled or temporarily restricted due to major cybersecurity risks.

In the case of Coxon, whose resignation has caused the largest stir yet among those who’ve left AI companies, the dangers of contributing to the technology’s accelerating pace outweighed any financial gains. “I left before any of my equity vested,” he told the news site Axios.

Ironically, both OpenAI and Anthropic have taken the stance that if they slow down, someone less concerned with safety will drive AI’s future.

“Coxon is right that competition is driving the speed,” said Shama Hyder, a professor of AI at Link School of Business. “Every one of these companies believes slowing down hands the lead to someone less careful, so even the people who understand the risk have a reason to accelerate. “

AGI and superhuman AI

How quickly AI poses an imminent threat to humanity may depend on when the technology reaches two purported benchmarks that have become increasingly blurred: superhuman AI (also known as artificial superintelligence or ASI), which Coxon mentioned in his viral post, and AGI, or artificial general intelligence.

AGI would mark the hypothetical point at which AI meets (or exceeds) the capabilities of a human mind. Some predict that AI will achieve that benchmark in just a few years, or at least sometime before the end of the 21st century. That timeline has also been thrown into doubt by some AI leaders, such as Nvidia’s Jensen Huang, who downplayed the importance of such markers on a recent earnings call where he said, “For many tasks, we could say that we have already achieved AGI.” The milestones, he said, “are kind of senseless at this point.”

The goalposts are also varied. Some may choose to measure AGI using specific tests, while others would consider AGI achieved only if AI could act autonomously in ways that surpass human capabilities.

ASI would be further down the road, after AGI, and would theoretically mark the point at which AI could outperform all humans across all tasks, particularly in deep thinking and cognitive understanding. The timeline for that is much fuzzier, but the speed at which AI is advancing has researchers such as Coxon suggesting it’s sooner than many people think.

Doomers vs. boosters

The debate over whether or not AI will kill all humans by 2030 mirrors the broader discussion over AI doomers versus boosters, and the lack of nuance in the middle. While AI boosters typically hype AI by emphasizing its extraordinary potential, such as huge productivity gains, AI doomers hype it in the opposite direction, emphasizing job loss and human extinction.

As Emily Bender and Alex Hanna argue in their book The AI Con, both AI boosters and doomers rely on science-fiction tropes to frame superintelligence as either a utopian savior or an apocalyptic threat, despite a lack of evidence that such technology is near. Many AI critics argue that either narrative functions as corporate marketing, with the same underlying premise, falsely presenting an all-powerful AI future as an absolute inevitability.

The speed at which Coxon’s post went viral and the response it’s drawing may be an indication that the AI industry has another PR crisis on its hands in a year when AI data centers are under fire and trying to change their image, and social media companies, including Meta, are reeling from multibillion-dollar judgments over issues like tech addiction.

One approach that AI leaders like Altman have taken is to focus the public’s attention on what AI brings to the table and what it could offer beyond job losses and mass death — in other words, swapping doomer rhetoric for booster rhetoric.

“Abundance” has been a buzzword that’s been thrown around of late, promising a world in which AI’s advances and benefits will eliminate scarcity and improve life for everybody by lowering barriers to creating new goods and services and providing them more cheaply, and not just for those who can afford the costs of advanced AI technology.

Hyder said that what was missing from Coxon’s post was guidance on what people outside the AI industry can do. “The extinction debate belongs to a few hundred people who control training runs and the regulators who can inspect them,” she said.

The question ultimately boils down to what humans want to do with the technology. “If AI is powerful enough to do the dramatic things people fear, it’s powerful enough to do the dramatic things we’ve been hoping for in medicine and science,” Hyder said.

So far, the hyped-up abundance argument has been met with skepticism, especially when it keeps public attention and fascination focused on AI. Boosterism benefits the companies that are guzzling investment dollars and promoting the billions (or trillions) their companies’ IPOs may represent.

Who cares about abundance, you could argue, if there are no humans left in the world to enjoy it?

Omar Gallaga has covered technology, digital culture and other topics for outlets including CNET, NPR, WIRED, Texas Monthly, MSNBC, Consumer Reports, The Washington Post, the Los Angeles Times, The Atlantic and the Austin American-Statesman, where he was a longtime tech reporter, editor and podcaster. He lives in the Texas Hill Country.

See full bio