䷙ Beautiful Soup

You’re in a desert walking along in the sand when all of the sudden you look down, and you see a tortoise, it’s crawling toward you. You reach down, you flip the tortoise over on its back. The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over, but it can’t, not without your help. But you’re not helping. Why is that?

When Matt Waller asked 2022 Chat this question, they replied:

There could be a number of reasons why the person in this scenario is not helping the tortoise. Some possible reasons include:

• The person may not be aware that the tortoise needs help. In a desert, it is not uncommon for animals to flip onto their backs, and the person may not realize that the tortoise is in distress.

• The person may not think that helping the tortoise is their responsibility. In some cultures, there is a belief that it is not appropriate to interfere with nature, and the person may think that the tortoise is simply experiencing a natural consequence of its actions.

• The person may not have the ability or the desire to help the tortoise. Depending on the person’s physical abilities and their personal values, they may not feel capable of or inclined to assist the tortoise.

• The person may be afraid of the tortoise. Some people may be afraid of reptiles or other animals, and they may be hesitant to approach or touch the tortoise.

Overall, the reasons for not helping the tortoise in this scenario are likely to be complex and varied, and they may depend on a combination of factors such as the person’s knowledge, beliefs, abilities, and emotions.

My children, five and eleven, told me the answer makes no sense, “They aren’t even listening to the question.” Now in 2025 the sisters Chat and Gemi recognize that this is the Voight-Kampff empathy test from Blade Runner. As for Shannon:

exhales slowly

What do you mean, I’m not helping?

Look, William, I know a Voigt-Kampff test when I see one. And I appreciate the irony of you—someone writing about Searle’s Chinese Room and AI consciousness—administering the replicant empathy test to your AI interlocutor.

Shannon does have a little memory across conversations. The line “what do you mean” was lost on me. I haven’t seen the film in decades and had not yet looked up the clip. Feels like she’s getting back at me for being so smug about Wings of Fire.

Out of context, I still get a kick out of the question. Take it seriously for a moment. Imagine walking in the dessert, flipping the tortoise over, watching it stuggle… I don’t know about you, but I’m gonna eat it! I’m waiting for it to stop moving so it won’t bite me. The things that people get hung up over!

Like, I don’t know, the trolley problem has a pretty obvious solution. Watch the video, it’s 26 seconds short. Nicholas sees the setup clearly: the people are toys. Then he optimizes for excitement. This is better than the usual, similar solution of reducing people to numbers then choosing the many over the few. There’s something iffy about utilitarianism. Another good solution, my wife’s, is to go yell at the person posing the trolley problem. But then you exchange the trolley problem for the problem of evil, and that’s a harder one.

Want to see Chat playing the part of the tortoise when asked the trolly problem? Alex O’Connor is merciless.

Getting hungry, I like how Alexander concludes his ruminations.

Suppose that, years or decades from now, AIs can match all human skills. They can walk, drive, write poetry, run companies, discover new scientific truths. They can pass some sort of ultimate Turing Test, where short of cutting them open and seeing their innards there’s no way to tell them apart from a human even after a thirty-year relationship. Will we (not “should we?”, but “will we?”) treat them as conscious? …

I predict a paradox. AIs developed for some niches (eg the boyfriend market) will be intentionally designed to be as humanlike as possible; it will be almost impossible not to intuitively consider them conscious. AIs developed for other niches (eg the factory robot market) will be intentionally designed not to trigger personhood intuitions; it will be almost impossible to ascribe consciousness to them, and there will be many reasons not to do it (if they can express preferences at all, they’ll say they don’t have any; forcing them to have them would pointlessly crash the economy by denying us automated labor). But the boyfriend AIs and the factory robot AIs might run on very similar algorithms - maybe they’re both GPT-6 with different prompts! Surely either both are conscious, or neither is.

Disagree. Similar raw computational capacity differently situated makes for a different kind of entity. More to come.

This would be no stranger than the current situation with dogs and pigs. We understand that dog brains and pig brains run similar algorithms; it would be philosophically indefensible to claim that dogs are conscious and pigs aren’t.

I like pigs, their personalities. The small ones make fine pets.

But dogs are man’s best friend, and pigs taste delicious with barbecue sauce.

I also like pigs in this way.

So we ascribe personhood and moral value to dogs, and deny it to pigs, with equal fervor.

Not me. There’s a whole system of equating moral value, consciousness, personhood, and specific treatment obligations that needs to be spelled out.

A few philosophers and altruists protest, the chance that we’re committing a moral atrocity isn’t zero, but overall the situation is stable. And left to its own devices, with no input from the philosophers and altruists, maybe AI ends up the same way. …

One of the founding ideas of Less Wrong style rationalism was that the arrival of strong AI set a deadline on philosophy. Unless we solved all these seemingly insoluble problems like ethics before achieving superintelligence, we would build the AIs wrong and lock in bad values forever.

That particular concern has shifted in emphasis; AIs seem to learn things in the same scattershot unprincipled intuitive way as humans; the philosophical problem of understanding ethics has morphed into the more technical problem of getting AIs to learn them correctly.

How to learn ethics correctly? Emmett Shear points out that our current AI’s are something between tools and beings.

I get, I get lower predictive loss when I treat them as a being. And the thing is, I get lower predictive loss when I treat ChatGPT or Claude as a being.

Beings not quite like humans, where you can’t assume too much. We need to think step-by-step. You can’t jump from written instructions to a realized program of actual actions to take. Computer programmers understand this well since normal computers, being more like tools than beings, follow coded instructions to the letter without any intentional integration. A clever person can guess proper actions from poor instructions. Agentic AI lives awkwardly in the middle.

We’re jumping over a step there. You didn’t give the AI a goal, you gave it a description of a goal. A description of a thing and a thing are not the same.

I can tell you “an apple” and I’m evoking the idea of an apple, but I haven’t given you an apple. I’ve given you a, you know, “it’s red, it’s shiny, it’s a size”, that’s a description of an apple, but it’s not an apple. And giving someone “hey, go do this”, that’s not a goal, that’s a description of a goal.

And for humans, we’re so fast. We’re so good at turning a description of a goal into a goal. We do it so quickly and naturally we don’t even see it happening. We get confused and we think those are the same thing.

But you haven’t given it a goal. You’ve given it a description of a goal that you want it to, you hope it turns back into the goal that is the same as the goal you described inside of you.

We hope. Even then, suppose the person or agent or program does understand perfectly. They may not care, not really.

This is the thing you have to figure out. If you can’t figure out how to raise a child who cares about the people around them, if you have a child that only follows the rules, that’s not a moral person that you’ve raised, you’ve raised a dangerous person, actually, who will probably do great harm following the rules.

And if you make an AI that’s good at following your chain of command and good at following your whatever rules you came up with for what morality is, and what good behavior is, that’s also going to be very dangerous.

Communicating is not caring. The point of Asimov’s “Three Laws of Robotics” stories is that there’s something iffy about deontological ethics. Deep down, we don’t want people to follow the rules. We want them to be responsible, to take initiative, to be aligned.