Apocalyptic AI fantasies are a stupid, annoying distraction from its real problems

I think, for example, that Jacob Coxon, the 27-year-old former employee of Anthropic who kicked off the latest cycle, left the company because he was truly worried about the direction it was taking. I think Anthropic engineer Evan Hubinger, who followed up on Coxon’s tweet with the aforementioned risk estimate of AI-induced human extinction, was also acting in good faith.

What disturbs me is that the media is so ill-equipped to handle such claims. When Coxon was asked by CNN to explain the scenario that could lead to our extinction, he responded by asking us to contemplate the recent “hacking” by an AI model developed at OpenAI, in defiance of instructions from the engineers who made it. This should have been followed with: so that’s it? That’s all you’ve got? I could have extemporised that story myself – there was no “inside knowledge” involved. And no one seems curious how Hubinger arrived at his number. It seems plucked out of thin air, which is not how probabilities work.

Suggested Reading

What is the answer to life, the universe and everything? We asked AI…

Philip Ball

It’s the same with the pundits, like the hapless Bernie Sanders, who told Newsnight that when the experts themselves are saying this, you’d have to be a “moron” not to listen. After all, said Sanders, it’s not just junior software developers; computer scientist Geoffrey Hinton, who previously worked at Google, has also warned of a real danger of human extinction. And look, Hinton has a Nobel prize for his work in the field! What we don’t hear is that he also makes claims about AI and consciousness that leave cognitive scientists with their heads in their hands.

The problem cuts both ways. A poor discourse around AI leaves us vulnerable to hype too. When Demis Hassabis, co-founder of Google DeepMind and also a Nobel laureate, forecasts that AI could eliminate all disease in a decade or so, actual drug developers are reduced to thousand-yard stares. But Keir Starmer’s government seemed totally in thrall to hype about how AI will transform the economy. (Burnham’s position on AI seems currently to be more nebulous.)

Dangers from AI are real and present, and require urgent legislation. But not because of apocalyptic fantasies. What about fraud, child porn generation, violation of artistic copyright, deepfake subversion of democracy? As science policy expert Jack Stilgoe of University College London has pointed out, claims about existential threats may, whether intentionally or not, create a convenient smokescreen for the real dangers. Flashy alarmism might even become a kind of salesmanship: how powerful this technology must be if it can really wipe out all of civilization!

The narratives typically used by AI doomers also misdirect responsibility. The OpenAI hacking episode involved a model that was supposed to be tested in a sealed information environment but found a way to access the data platform Hugging Face. It was widely reported as having “gone rogue”, like some silicon serial killer. In a recent statement on AI risks, Anthropic co-founder Dario Amodei spoke about the “agents” in the model as though they were some natural phenomenon, like a virus. 

OpenAI commissioned a report on the event by an independent agency, which framed its description in corporate-speak with talk of the AI “agents” executing “workstreams” and “R&D projects”, with “coordinators”, “recruiters” and so on. As linguist Brigitte Nerlich of Nottingham University says, “There is no framing of the corporations themselves as, say, ‘principal investigators’ with a set of norms, conventions of behaviour and ethics.”

These are choices. As computer scientist Melanie Mitchell has pointed out, the breach of confinement was possible because safeguards that normally prevent “hacking” were turned off, and the engineers inadvertently left the AI model with a route to the internet. The AI was not out of control – it could have been switched off any time by the human operators, had they been paying attention. In short, says Mitchell, “OpenAI did not have proper security measures in place.”

Suggested Reading

AI is designing viruses – what could possibly go wrong?

Philip Ball

Coxon’s and Hubinger’s tweets “have gained huge coverage because these kinds of things always do”, says Kate Devlin, professor of AI and society at King’s College London. “It taps into our very primal fears – and our decades of watching dystopian sci-fi.” But Devlin adds, what’s “more mundane, but potentially very destructive, are AI agents let loose due to human carelessness or incompetence, hacking or taking control of insecure systems.” AI could do catastrophic damage if entrusted with things like nuclear arsenals or bioweapon design. But that doesn’t require superintelligence, just human stupidity.

If Coxon’s and Hubinger’s tweets, and Amodei’s somewhat self-serving proposal that AI development be more carefully “paced”, lead to more regulation, maybe that’s a good result. But there is no reason to suppose that AI developers and their bosses are best placed to comment on either their dangers or their potential value. Indeed, quite the opposite: there’s reason to think that industry involvement erodes one’s judgement, not just ethically but technically. 

We’d do better to take advice on the real benefits and risks of AI not from those who are making or selling it but from thoughtful commentators like Mitchell, Devlin, computational linguist Emily Bender (of “stochastic parrots” fame), and two astute AI ethicists formerly at Google, Shannon Vallor and Timnit Gebru. (If you notice something striking about that latter list, I don’t think it’s a coincidence either.)

More generally, to have a better standard of discourse both in the media and in policy we should recognise that, despite the prevailing attitude of many politicians towards academia, the humanities are an essential element in the discussions. Since its earliest days, those making AI have proved consistently bad forecasters of its prospects and capabilities. We must stop treating them all like gurus.