Our seduction by AI’s believable human voice.

 I want to point to an excellent New Yorker article by Patrick House titled  “The Lifelike Illusion of A.I.”  The article strikes home for me, for when a Chat Bot responds to one of my prompts using the pronoun “I”  I unconsciously attribute personhood to the machine, forgetting that this is a cheap trick used by programmers of large language model to increase the plausibility of responses.

House starts off his article by describing the attachments people formed with the Furby, an animatronic toy resembling a small owl, and Pleo, an animatronic toy dinosaur. Both use a simple set of rules to make the toys appear to be alive. Furby’s eyes move up and down in a way meant to imitate an infant’s eye movements while scanning a parent’s face. Pleo mimes different emotional behaviors when touched differently.

For readers who hit the New Yorker paywall when they click the above link, here are a few clips from the article that I think get across the main points:

“A Furby possessed a pre-programmed set of around two hundred words across English and “Furbish,” a made-up language. It started by speaking Furbish; as people interacted with it, the Furby switched between its language dictionaries, creating the impression that it was learning English. The toy was “one motor—a pile of plastic,” Caleb Chung, a Furby engineer, told me. “But we’re so species-centric. That’s our big blind spot. That’s why it’s so easy to hack humans.” People who used the Furby simply assumed that it must be learning.”

Chung considers Furby and Pleo to be early, limited examples of artificial intelligence—the “single cell” form of a more advanced technology. When I asked him about the newest developments in A.I.—especially the large language models that power systems like ChatGPT—he compared the intentional design of Furby’s eye movements to the chatbots’ use of the word “I.” Both tactics are cheap, simple ways to increase believability. In this view, when ChatGPT uses the word “I,” it’s just blinking its plastic eyes, trying to convince you that it’s a living thing.

We know that, in principle, inanimate ejecta from the big bang can be converted into thinking, living matter. Is that process really happening in miniature at server farms maintained by Google, Meta, and Microsoft? One major obstacle to settling debates about the ontology of our computers is that we are biased to perceive traces of mind and intention even where there are none. In a famous 1944 study, two psychologists, Marianne Simmel and Fritz Heider, had participants watch a simple animation of two triangles and a circle moving around one another. They then asked some viewers what kind of “person” each of the shapes was. People described the shapes using words like “aggressive,” “quarrelsome,” “valiant,” “defiant,” “timid,” and “meek,” even though they knew that they’d been watching lifeless lines on a screen.

…chatbots are designed by teams of programmers, executives, and engineers working under corporate and social pressures to make a convincing product. “All these writers and physicists they’re hiring—that’s game design,” he said. “They’re basically making levels.” (In August of last year, OpenAI acquired an open-world-video-game studio, for an undisclosed amount.) Like a game, a chatbot requires user input to get going, and relies on continued interaction. Its guardrails can even be broken using certain prompts that act like cheat codes, letting players roam otherwise inaccessible areas. Blackley likened all the human tinkering involved in chatbot training to the set design required for “The Truman Show,” the TV program within the eponymous film. Without knowing it, Truman has lived his whole life surrounded not by real people but by actors playing roles—wife, friend, milkman. There’s a fantasy that “we’ve taken our great grand theories of intelligence and baked them into this model, and then we turned it on and suddenly it was exactly like this,” Blackley went on. “It’s much more like Truman’s show, in that they tweak it until it seems really cool.”

A modern chatbot isn’t a Furby. It’s not a motor and a pile of plastic. It’s an analytic behemoth trained on data containing an extraordinary quantity of human ingenuity. It’s one of the most complicated, surprising, and transformative advances in the history of computation. A Furby is knowable: its vocabulary is limited, its circuits fixed. A large language model generates ideas, words, and contexts never before known. It is also—when it takes on the form of a chatbot—a digital metamorph, a character-based shape-shifter, fluid in identity, persona, and design. To perceive its output as anything like life, or like human thinking, is to succumb to its role play.