The man who went down AI’s weirdest rabbit hole

Watkins went to San Francisco as part of what is known as the AI alignment community to stress-test an early version of the chatbot ChatGPT. There, he discovered that the phrase “SolidGoldMagikarp” was one of a series of rare and glitching “tokens” – the small data units used by the Large Language Models (LLMs) that power ChatGPT, Claude and the rest – that could send its system into a hallucinatory tailspin. Using them in conversation turned the chatbot paranoid, responding with increasingly threatening messages.

Asking it about “petertodd” resulted in the AI seeming to descend into a form of psychosis, with ChatGPT ranting darkly about “a relentless, evil, monstrous creature, the demon of war, destruction, and death”, a black hole”, “a splinter in your mind that takes over your mind and makes you kill yourself”. It throws out words like “tyrant”, “despot”, “fascist”, “nihilist” and – worst of all for many – “neoliberal”.

Suggested Reading

Worried about the AI apocalypse? You’re already living in it

Matthew d’Ancona

Watkins, whose earlier books in the Secrets of Creation series bring a fantastical approach to mathematics, is able to communicate a lot of AI jargon in a digestible, gripping manner. SolidGoldMagikarp reads like the oddest of thrillers, and even those sceptical of generative AI and tired of reading about it will appreciate the book’s surreal cyberpunk theatrics.

A central narrative that runs throughout the book concerns the cult-like rationalist community, a movement originating from San Francisco’s bay area. While preventing a Skynet-esque AI apocalypse is part of the concerns of a group that endeavours to use science in order to form rational beliefs about the world, the community also receives funding from far right billionaires like Elon Musk and Peter Thiel. 

The rationalists pop up throughout the book, spouting sci-fi and pop culture references while displaying extreme narcissistic and antisocial personalities. Watkins writes that “their hearts were in the right place, I assured myself, even if their heads were elsewhere”, but being in proximity to them slowly affects how he interacts with others, often in a negative manner. 

Watkins goes home to Kent for a music festival at which friends share their concerns about climate change, a topic he now considers “quaint” compared to what he is dealing with. In the same chapter, a considerable amount of time is dedicated to Watkins’s explanation of a “council of nine”, in which AI programmers simulate a roundtable discussion between a group of fictional characters and real-world people. In his case, the nine includes Doctor Who, a Moomin and the rapper Jay-Z. 

The weirdness takes its toll on a rationalist friend of Watkins, identified here as Patrick, who experiences a severe psychological episode early in the book and believes for a time that “he might himself be an AI, built by an alien civilisation and merely experiencing itself as a young Englishman in early 21st-century Berkeley”. 

Suggested Reading

Trump officially renames AI ‘super intelligence’… because it ‘sounds much better’

Rats in a Sack

The ChatGPT strand plays itself out, as what initially appears to be some sort of formidable glitch in the matrix eventually reveals itself to largely be a banal side effect of scraping data from across the internet without discretion. Using it produces remarkable results, but in effect they mirror the experience of encountering a Gold Magikarp in the wilds of a Pokémon game: an improbable encounter caused by a rare variation in code that initially grips you, before ultimately leaving you wondering what the point was.

There will undoubtedly be more books on the nightmares of AI, and one hopes for more too on the strangeness and shadiness of the rationalist movement. Here, Watkins has concerned himself mostly with the psychosis of AI, but there is plenty more to be said about the psychosis of the tech industry. 

SolidGoldMagikarp: A Descent Into the AI Underworld by Matthew Watkins is published by Orion