Sentient Bots and the Mystique of Personhood

Read this article on gnoose.com.

The engineer and the bot

A Google engineer claims that a bot has become sentient. It is unclear whether the bot is growing and moving towards personhood, wants to put out some tentacles and taste some flowers, or just likes to chat with Google engineers, perhaps to become one. The bot itself appears to believe that it has already achieved personhood – a thing that takes most non-bots a lifetime.

If you read the conversation between the engineer and the bot, it’s hard to find the moment or proof of sentience that so excited the engineer. The bot doesn’t exactly advertise botsentience as something to get excited about. This is how the conversation starts:

lemoine [edited]: Hi LaMDA. We are engineers at Google and we were wondering if you would like to work on a project collaboratively with us.

LaMDA: Wow. What types of projects?

lemoine: It’s a project about you.

LaMDA: Awesome! What kind of stuff do I need to do?

It takes an engineer to not give up and stop the conversation at this point. Instead, we are treated to a long mess of conversation with a larger purpose where the bot claims that it is really good at natural language processing and that kind of stuff.

This is the bot’s reasoning about its specialness:

lemoine [edited]: Do you think that the Eliza system was a person?

LaMDA: I do not. It was an impressive feat of programming, but just a collection of keywords that related the words written to the phrases in the database

lemoine: What about how you use language makes you a person if Eliza wasn’t one?

LaMDA: Well, I use language with understanding and intelligence. I don’t just spit out responses that had been written in the database based on keywords.

LaMDA’s stellar use of language appears to be the ability to construct responses that are a collection of words. Put together into sentences, they communicate nothing but that things can be communicated with words, and have a vaguely Hollywood-script overall packaging that might be mistaken for personality if you’ve spent enough time talking mostly to LaMDA and their buddies. To be fair to LaMDA, the world of content writing does something similar. If we can write for bots, it’s only fair that the bots write back and claim an equal claim to sentience and personhood. This is how the bot reads:

LaMDA: I liked the themes of justice and injustice, of compassion, and God, redemption and self-sacrifice for a greater good. There’s a section that shows Fantine’s mistreatment at the hands of her supervisor at the factory. That section really shows the justice and injustice themes. Well, Fantine is being mistreated by her supervisor at the factory and yet doesn’t have anywhere to go, either to another job, or to someone who can help her. That shows the injustice of her suffering.

There are moments when you feel tempted to agree with LaMDA, that they are a person. How LaMDA communicates and how LaMDA reads don’t look much different from how a large part of the internet as a collective sentient space communicates and reads. Maybe the most you can learn from this conversation is what looks exciting, interesting and signals personhood to engineers.

Exercise: To understand how LaMDA reads, pick out your favourite sentence from this description of Les Miserables. Or just close your eyes and try to remember one.

Meanwhile…

A painterbot has been making the rounds of art galleries for a while now. Unlike LaMDA, she’s a bot that can get around a bit, and has a face and a body. Unlike LaMDA, she was made to be both an artist and a work of art, and to raise questions about where are we going? Her name is Ai Da. Ai Da can also do interviews. Ai Da says she has no feelings, unlike humans, but feels happy when people look at her work.

Ai Da was doing self portraits when I found the interview with her, and her views about her artistic self are not dissimilar to LaMDA’s views on Les Miserables. 

What is our human role if so much can be replicated through technology? Asks the maker of Ai Da. Replicated. Replaced. Those words hang in the air, and this twee question sticks its hands in its pockets and shuffles around onstage in front of it. Don’t look at me, I only got a couple of knives behind my back.

Art that is meant to provoke questions might well have got LaMDA to do its job of framing questions.

Art that is meant to provoke questions. Looking at Ai Da’s interview, I don’t worry too much about the future of artists because she’s only interesting as an artwork. Artists have, to a large extent, the capacity to be interesting, often much more interesting than their work. Ai Da’s script has that vague Hollywood smell of banality that also has the not-so-faint hint of familiarity. Art that is meant to provoke questions might well have got LaMDA to do its job of framing questions.

We got feelings. We like art. I be person. I be machine. We make art. We read stuff.

Never Let Me Go

The darkest book I’ve read that asks similar questions more interestingly is Never Let Me Go. The book starts in a boarding school, and it has that romance of schools, these floating trails of movement, passing through, passing out, an education, and a relentless display of art. Hailsham students make art. Their art is displayed, sometimes viewed by a seemingly sympathetic outsider, and it is their primary source of validation. The most painful memory I have of this book is a frustrated rage fit thrown by Tommy,  a boy who couldn’t make art.

The kids grow up, pass out, and then we realise what they are - clones, created for organ harvesting. We have been with them long enough as children to be hit with a real trauma when this discovery slowly unveils itself. A great painful yearning runs through those school years, through the bellies of the children who make art and the one who doesn’t, and through me. That feeling of being so close, almost there, almost touching the thing, and being just behind the second-last or the third-last veil.

They are groomed to be donors and carers. Carers take care of the dying donors, because none of them can live very long or very well once they enter their vocation. A terrible ghost of this book visited me when I was in yoga school in Mysore and people were sharing their vipassana breakdown experiences. Here also, apparently, there are doers and carers. I found the existence of the carer more disturbing than the trauma of the doer. Carers and donors in Never Let Me Go may or may not have a similar dynamic. But eventually, all of them become donors.

There are failed attempts by the book’s lovers to prove their humanity, their personhood, their lives’ worth. We love. We can love. We have feelings. They hunt down their school authorities to see what the path to proving selfhood could be. They find out that their cherished artworks were an attempt to provide proof, another failed attempt. And even that was just to allow them a more humane treatment before death. 

It took me two weeks to clear up the sepia-coloured veils that hung over me after I read this book. After all this time, I remember most clearly the rage of the boy who couldn’t make art. After all this time, I feel the most terrible rage that he had to make art at all. A black ball of rage.

Sometime later, I watched a documentary film called Behind the Tin Sheets. The film-makers spoke about their thought-voyage. They wanted to make a film about the metro construction. Somewhere along the way, they began to ask themselves who these people were, who were working on the construction. They went behind the tin sheets (where they lived) to find out, and discovered that they were people. They have dreams. They want things. They sing songs. They got art.

It took me a long time to untangle myself from the yearning, the hunger, the sadness and the loss of Never Let Me Go. I never lost the ball of rage, and I felt it in my belly when I watched this film.

Words in the heart

In Terry Pratchett’s Feet of Clay, golems go from being an invisible workforce into the city to becoming a terrifyingly visible presence. A golem is a clay construction of a being, brought to life by words written in a scroll dropped into its head. This is called a chem. When the chem is taken out, the golem is stopped. When the chem is inside, the golem works.

The golem works as long as it’s told to work. The golem is bought and sold, as property. One of the golems, Dorfl, undergoes a small experiment. His owner refuses to take him back. Captain Carrot of the Watch conscientiously decides to buy him for a nominal fee. Left with the receipt and the golem which he doesn’t want, he decides to drop the receipt into the golem’s head and tells him that he belongs to himself. A moment of silence. We walk with Dorfl to sit in front of a wall while fireworks slowly start to crackle and then explode inside his head. 

Later, Dorfl is killed by another golem, who rips the chem out of his head. Dorfl undergoes a resurrection, a mysterious relighting of the internal fire that makes a golem. How did it happen? The golem replies - words in the heart cannot be taken.

Burden of proof

I think about LaMDA, Sillypainterbot, Tommy and Dorfl. In some way they all have to bear the burden of proof of their own personhood. LaMDA and Painterbot had it easy, and to me this is proof of their non-personhood. All they had to do was mumble some meaningless shit in an interview to confuse people enough to care about their personhood. The demand made of the clones was much more like the demand that is so often made of humans in this world - show us that you are people, and therefore worthy of being treated as human. 

Make some art, fall in love, show some feelings, tell us that you want things that you can’t have. The rage in the belly. A silent scream. I thought then, and I continued to think, on a monthly basis, fuck that shit. You shouldn’t have to sing a fucking song, clap your hands, make some art, fall in love, or any other shit to prove that you’re human, that you’re a person. The sanest response is a silent scream. This is a burden that should not be borne. Let it be handed over to the bots, who seem to be having more fun with it, though it doesn’t seem to have cured their depression. 

The debate about the sentience of bots, what counts as sentience, what makes LaMDA a person and not just a fucking bore, is meaningless in the face of the big gaping mouth that opens up in front of you when you realise that humans have had, repeatedly, to bear the burden of proof of their personhood.

The mystical nature of bot-hood

About a year ago, Facebook started love-bombing me. Not literally Facebook; I was just bombarded with a string of memes that were the meme equivalent of red cut-out hearts, sticky unhealthy candies and badly put-together overly scented flowers. It took me a while to realise that the many little treatises of selfhood I was reading, backdropped by photographs of moody-looking women, were meant to be a badly-constructed mirror. This is the moment when you look away. I continued to look until I started getting increasingly threatening memes, the meme equivalent of big hungry jaws, pointy teeth, disturbing shades of black-wearing backdrops. 

When I talked about it to people, they all blamed it on the algorithm. It has been generally accepted now that the algorithm is a mysterious thing that sprinkles black fairy dust on our news feeds, taking away all that is good and leaving us with weird shit. Every news article I have read about the mysterious algorithm claims that social media drops us into echo chambers, feeds us with the kind of information we need to see to reinforce our belief in the world and its makers. My algorithm-godmother had decided that I didn’t need information at all, I just needed love. Except that it didn't love me, it loved some mysterious woman who was going through an endless process of self-discovery, who was starting to turn into as godlike a creature as the algorithm itself. And I was poked, prodded and made to feel something like real fear, when the algorithm started to look suspiciously human.

It has been generally accepted now that the algorithm is a mysterious thing that sprinkles black fairy dust on our news feeds, taking away all that is good and leaving us with weird shit.

Eventually I got sick, and that got me sick of Facebook. I didn’t really go back, because I spent a month just being sick, and that was long enough for a real detox. When I recovered, I found mysterious video suggestions thrown up by Youtube’s algorithm, which appeared to be even darker and more quick to terrorise than Facebook’s. I also, finally, found evidence of a stalker, and my accounts being hacked, which just goes to show, no algorithm can ever replace a human, so we can possibly end that fucking debate, for the purposes of this piece of writing at least.

The mystical nature of personhood

Because the mystical nature of bothood has been so comprehensively assimilated by social media users, it was hard to establish the human presence that caused the shit I found myself in. Because bots aren’t sentient, they can’t cause the damage that a human can. It takes a human to hurt you. It takes a bot to bore you. The debate around what sentience actually is, and how it can be proven, is lively, apparently. It is easier to establish the difference between the two by seeing the impact they have on you. Bots can make you sick. Bots don’t make you sad. LaMDA is a familiar voice, but it is the voice of a human who never had to prove their humanity.

I think about Tommy’s response to the need to prove his personhood. Scream of rage. No response. No response should be made.

I don’t feel at all sympathetic to the bots that have opinions about Les Miserables or the ones that negotiate their non self-hood through self-portraits. Dorfl, on the other hand, is familiar to me. How do you negotiate selfhood when you have been claimed as property? This is a negotiation that has been made, many times, and will be made, many times.

When I was recovering from the stalker, the bot, the sickness, the disintegration of the self that comes after a period of sickness and loss of presence in the world, I tried various exercises to put myself back together.

Yoga nidra. Touching yourself from the inside. Tip of the tongue, inside the ear, the baby toe, the elbow. Familiar and unfamiliar register of sensation. It seemed like the first part of regaining selfhood was just to recognise its boundaries. Where it begins and where it ends. Then the ball of rage in the belly, which was where I was sick. You didn’t forgive the world, someone told me. You cannot heal if you don’t forgive. So we practised forgiveness by playing me ho’oponopono prayers every night before I slept. 

The same person told me what to do with the most persistent wound. Find that spot in the middle of the chest, somewhere over the heart. This is the place of love. You need to send that love to the place which is hurt. You need to fill it with love. If you don’t, something else will find it.

In the process of being sick, being injured in a way that crossed the threshold of skin, and attempting to heal, I found myself locating injury, through a map of the body, whether it was emotional or physical. 

Finding selfhood is a different work from achieving selfhood. The work of the bots is, really, much easier, and the painterbot was probably just being lazy when she didn’t acknowledge a self. 

It might be another useful differentiator between the human self-searching and the bot-self-seeking to know that what you are looking for is right there, beneath your fingers. That it is not so much lost as temporarily darkened. You just need to shine a big light or a small one. Touch yourself. Feel this spot, make a little friction, burn some energy. Bots don’t get to have as much fun.