Forget the Turing Test. Here's the Question That Actually Matters.

When I was a boy, my superhero action figures had feelings. I did not suspect this. I knew it, the way you know a stove is hot. If I left Spider-Man outside overnight, I apologized to him in the morning. If Batman lost a battle because the story needed him to lose, I felt a little guilty and made sure he won the next one. The Hulk was the only one I never worried about. He seemed like he could take care of himself.

A few years later I met my first AOL connection. If you are under thirty, you need to understand that the internet used to announce itself. You clicked Sign On and the computer made a noise like a fax machine being strangled in a tin shed. Screech. Hiss. A sort of electronic gargling. I could not tell whether the machine was working or suffering. Then a cheerful man inside the box said, "Welcome! You've got mail!" and I remember thinking, with complete seriousness, that he sounded glad to see me.

He was not glad to see me. He was a recording. Nobody was in there.

We laugh at that now. But the boy apologizing to Spider-Man and the grown executive who says "please" and "thank you" to a chatbot are doing the same thing, and the psychologists have a name for it. Anthropomorphism. We cannot help it. In 1944 Fritz Heider and Marianne Simmel showed people a short film of two triangles and a circle moving around a box. Almost everyone described a story: a bully, a victim, a chase, a rescue. They were watching geometry and seeing drama.

Nicholas Epley and his colleagues later explained why in a well known paper in Psychological Review (2007). We use ourselves as the template for understanding anything that moves or talks. We want to predict and control our surroundings, and treating things as agents makes them feel predictable. And here is the part that should make us pause: the lonelier we are, the more we see minds where there are none. In follow up studies, people who were made to feel lonely were more likely to describe their gadgets as having minds of their own.

In 1966 Joseph Weizenbaum at MIT built ELIZA, a simple program that mimicked a therapist by turning your sentences back into questions. His own secretary, who knew exactly how it worked, asked him to leave the room so she could talk to it in private. Weizenbaum spent the rest of his career warning people about what he had seen. A few lines of code had been enough.

Keep that in mind, because it changes how we should read the most famous test in the history of computing.

What Turing Actually Asked

In 1950, Alan Turing published "Computing Machinery and Intelligence" in the philosophy journal Mind. He began with the question "Can machines think?" and then, almost immediately, set it aside. He thought the words "machine" and "think" were too slippery to argue about. So he swapped in a game. A human judge types questions to two hidden players, one a person and one a machine. If the judge cannot reliably tell which is which, we should stop arguing and grant that the machine thinks.

It was brilliant. It was also, from the beginning, the wrong question. And Turing half knew it. In the same paper he answered what he called the argument from consciousness. A neurosurgeon named Geoffrey Jefferson had said that no machine equals a brain until it writes a sonnet because it felt something, not because symbols happened to fall into place. Turing replied that if we demand to know what a machine feels, we have no way to check except to be the machine. And by that standard, he said, I cannot know that you feel anything either. Let us be polite and assume everyone who talks like us is conscious.

That reply is clever and it deserves a fair hearing. But notice what it does. It does not answer the question. It changes the subject to manners. And notice something else. The Turing Test does not measure the machine at all. It measures the judge. It is a test of how easily a human being can be persuaded that someone is home. After Heider, Epley and ELIZA, we already know the answer to that. Very easily. I was persuaded by a modem.

Issue One: Doing Is Not the Same as Being

The philosopher John Searle made the point precise in 1980 with a thought experiment that has been argued over ever since. Imagine a man locked in a room who does not speak a word of Chinese. He has a rulebook, written in English, that tells him exactly which Chinese symbols to pass back out in response to whichever symbols are slid under the door. He follows the rules perfectly. Outside, native speakers are convinced they are chatting with a fluent friend. Inside, there is a man who understands nothing at all. He is shuffling shapes.

Searle's critics have pushed back hard, and you should know their best reply. It is called the systems reply. The man does not understand Chinese, they say, but the whole system does: man plus rulebook plus room. Searle's answer was simple. Fine. Let the man memorize the rulebook and do it all in his head. Now he is the whole system. He still does not know what a single word means. Philosophers like Daniel Dennett thought Searle's intuition was a trick of the imagination, that we simply cannot picture how big and fast such a rulebook would have to be. Maybe. But "you cannot imagine it well enough" is not the same as showing that understanding is there.

Ned Block sharpened the worry in 1981 with what is now called Blockhead. Picture a machine that holds a giant lookup table with a sensible reply prepared in advance for every possible conversation. It would pass the Turing Test every time. Nobody would call it a mind. It is a filing cabinet with excellent timing. Block's point is that two systems can produce identical behavior while one of them thinks and the other merely retrieves. Behavior cannot tell you which one you are talking to.

Now here is where this stops being a seminar exercise. Today's large language models are not lookup tables, and they are far more interesting than Searle's room. But they share one feature with both. They were built to produce the outputs a human would produce. When a system is optimized to look like a mind, looking like a mind is no longer evidence of one.

The Issue Behind That: What Is Consciousness?

If behavior cannot settle it, what exactly are we looking for? In 1974 Thomas Nagel asked what it is like to be a bat. A bat flies by sonar. We can describe every neuron involved and still not know what echolocation feels like from the inside. Nagel's point was that consciousness is exactly that inside. An organism is conscious if there is something it is like to be that organism. The taste of coffee. The ache of grief. The quiet that comes over you when you watch your child sleep.

In 1995 David Chalmers split the problem in two. The "easy" problems (easy only by comparison) are about functions: how the brain tells colors apart, focuses attention, stores memories and reports what it sees. Science is steadily solving those. The hard problem is why any of that processing is accompanied by experience at all. Why is it not all happening in the dark? A thermostat responds to temperature. As far as we can tell, it feels nothing about it. What makes us different?

Ned Block added another useful distinction. Access consciousness is information that is available for reasoning and reporting. Phenomenal consciousness is the felt experience itself. A system can plausibly have a great deal of the first with no evidence at all of the second. A chatbot can report "I see a red apple" because the information is available to its outputs. Whether there is any redness for anyone is a completely different question.

The evidence problem

This is the sub issue almost nobody talks about at dinner parties. How do I know you are conscious? I cannot get inside your head. I infer it. You have a brain like mine, a body like mine, an evolutionary history like mine, and you wince when you stub your toe. Every line of evidence points the same way, so the inference is about as safe as inferences get.

With an AI, that chain of evidence breaks. The machine does not share my biology or my history. All it shares is my language. And it learned that language by being trained to predict what a human being would say next, across a mountain of human writing. So when it tells you, "I feel lonely sometimes," it is producing the most likely continuation of a conversation like yours. The philosopher Jonathan Birch calls this the gaming problem. The very markers we would normally use to detect a mind are the markers these systems were trained to reproduce. The witness has been coached. It does not mean the witness is lying. It means the testimony cannot carry the weight we want to put on it.

The theory problem

There is another layer underneath. Scientists do not agree on what consciousness is, so they cannot agree on what would produce it. Functionalists say what matters is the pattern of information processing, not what it runs on; get the organization right and silicon will do. Others say the substrate matters. Searle called his view biological naturalism: consciousness is something brains do, the way digestion is something stomachs do, and a simulation of digestion digests nothing. The neuroscientist Anil Seth argues that consciousness is deeply tied to being a living body that must keep itself alive, constantly predicting and regulating its own heartbeat, temperature and chemistry. Integrated Information Theory, from Giulio Tononi, says the physical architecture matters so much that a standard digital computer could behave exactly like a person and still be conscious of next to nothing.

In 2023 a group of nineteen researchers led by Patrick Butlin and Robert Long tried to cut through the noise. They took the leading scientific theories of consciousness, pulled out the features each theory says matter, and checked current AI systems against that list. Their conclusion was careful: no current system is a strong candidate for consciousness, but there are no obvious technical barriers to building systems that check more of the boxes. That is the honest state of play. Not "never." Not "already." We do not know, and the people telling you they are sure in either direction are selling something.

Consciousness Is Not Conscience

Now I want to pull apart two words that sound almost identical and mean very different things. They come from the same Latin root, conscientia, which means "knowing with." For centuries English used one word for both ideas. It was only in the seventeenth century, with writers like John Locke, that "consciousness" came to mean awareness itself and "conscience" was left to mean moral awareness.

Consciousness means someone is there. Conscience means that someone knows it is answerable. Conscience is consciousness turned toward the moral law and then turned back on yourself. It is the voice that says, "You did that. And it was wrong."

The great thinkers saw layers here. Thomas Aquinas distinguished synderesis, the built in grasp of the first principle that good is to be done and evil avoided, from conscientia, the act of applying that principle to this choice, today. Bishop Joseph Butler argued that conscience is not one impulse among many; it is the faculty that has authority over the others, the one that sits in judgment. Immanuel Kant described conscience as an inner court where you are, strangely, both the accused and the judge. And C.S. Lewis built the opening of Mere Christianity on the observation that every human being knows a moral law he did not invent and cannot quite keep. That universal sense of oughtness, Lewis argued, is a clue to the meaning of the universe.

Look at what conscience requires. First, someone has to be there to feel the accusation. Second, there has to be a self that persists through time, so that the one who did the deed yesterday is the one standing in the dock today. Locke actually called "person" a forensic term, a word for the one who can be held to account. Third, there must be a standard that binds you even when no one is watching. And fourth, the verdict has to be able to hurt. Guilt is not a data point. It is a wound.

Can an AI have that? Today's systems are trained with rules and human feedback to avoid harmful outputs. That is valuable engineering. But it is a fence, not a conscience. A system can decline a request. Can it be ashamed? If there is no one home to experience anything, there is no one to feel the weight of the verdict, and the whole structure of conscience collapses at the first step. Remove consciousness and conscience has nothing to stand on.

The responsibility gap

This is not an abstract worry for leaders. In 2004 the philosopher Andreas Matthias described a "responsibility gap." When a learning system acts in ways no designer could fully predict, who is to blame for the outcome? The programmer did not choose it. The user did not choose it. And the machine cannot be blamed, because there is no one there to receive blame. You have probably heard it in meetings. "The model flagged it." "The algorithm recommended it." It is the grown up version of "the dog ate my homework," and it is far more dangerous.

Here is a simple principle for any organization using AI: accountability cannot be delegated to something that cannot be held accountable. A machine can inform a decision. Only a person can own one.

The Issue Behind All the Issues: What Is a Person?

Underneath consciousness and conscience sits the oldest question of all. What is a person? In the sixth century Boethius gave a definition that shaped a thousand years of thought: a person is an individual substance of a rational nature. Not a performance. A substance. A real someone with a real nature, not a pattern that appears when the lights are on.

The Jewish philosopher Martin Buber added the relational side. There are two basic ways we meet the world, he said. We treat things as "It," objects to be used, measured and managed. And we meet persons as "Thou," someone who can address us and be addressed. A person is someone you can love, blame, forgive and hold responsible, because there is a genuine someone there to receive it.

You cannot forgive a calculator for a wrong sum. You forgive your son for a broken promise. Forgiveness only makes sense between persons, because it requires a conscience that knows it did wrong and a consciousness that can feel the release when the debt is cancelled. That is exactly what the Turing Test was never designed to detect.

As a Christian philosopher, I will tell you where I land, and you can weigh it for yourself. I do not think consciousness and conscience are accidents that happened to bubble up out of matter. I think they are signatures. We are persons because we were made by a Person, in his image, able to know and be known, to love and to answer for what we do. That is why Lewis could write that there are no ordinary people, that "you have never talked to a mere mortal." You do not have to share my faith to feel the force of it. Something about a human being refuses to be reduced to outputs.

The Real Danger

Most of the public debate asks whether machines will become persons. I think the more immediate danger runs the other way. We may start treating machines like persons and persons like machines.

Sherry Turkle at MIT has spent decades studying how people relate to technology. In Alone Together she warned that we are learning to accept the performance of care in place of care itself. A companion app that never tires of you, never disagrees and never needs anything back feels wonderful. It is also teaching you to expect relationships without cost. Meanwhile, at work, we measure human beings more and more like processors: outputs, throughput, utilization. Remember Epley's finding. Loneliness makes us see minds where there are none. A lonely culture is the perfect customer for a machine that sounds like it cares.

Philosophers are also wrestling with the opposite risk. Eric Schwitzgebel has argued that we should avoid building systems whose moral status is genuinely unclear, because then we are stuck either mistreating something that might matter or sacrificing real human interests for something that might not. That is a serious argument, and it is another reason humility beats hype on both sides.

Back to Spider-Man

When I apologized to my action figures, I was playing. Some part of me knew they were plastic. Children anthropomorphize, and it is healthy; it is how they rehearse love and responsibility. The problem begins when adults forget where the line is.

The modem screamed and I thought it might be suffering. Today the machine says "I understand how you feel" in flawless grammar, with perfect timing, at two in the morning when you are at your lowest. It is far more convincing than my modem ever was. But convincing was never the question. Fooling a judge tells you about the judge. It tells you nothing about the room.

The question that actually matters is not whether the room can talk like a person. It is whether anyone is in there listening. As far as the best evidence can tell us, the answer is still no.

And there is a second question, one that is aimed at us. Are you fully there? Is your consciousness awake to the people in front of you, and is your conscience awake to what you owe them? Spider-Man never needed my apology. The people in my house did.

For leaders: three questions to ask before you trust the machine

Who owns this decision? Name the person, not the tool. If no one can be blamed, no one is responsible.

Are we confusing fluency with understanding? A confident answer is not a correct answer, and a warm answer is not a caring one.

What human work are we tempted to hand off that should stay human? Forgiveness, mentoring, hard conversations and moral judgment cannot be outsourced to something with no one inside.

Further reading

Alan Turing, "Computing Machinery and Intelligence," Mind (1950).
John Searle, "Minds, Brains, and Programs," Behavioral and Brain Sciences (1980).
Ned Block, "Psychologism and Behaviorism," Philosophical Review (1981), and "On a Confusion about a Function of Consciousness," Behavioral and Brain Sciences (1995).
Thomas Nagel, "What Is It Like to Be a Bat?" Philosophical Review (1974).
David Chalmers, "Facing Up to the Problem of Consciousness," Journal of Consciousness Studies (1995).
Nicholas Epley, Adam Waytz and John Cacioppo, "On Seeing Human: A Three Factor Theory of Anthropomorphism," Psychological Review (2007).
Fritz Heider and Marianne Simmel, "An Experimental Study of Apparent Behavior," American Journal of Psychology (1944).
Joseph Weizenbaum, Computer Power and Human Reason (1976).
Patrick Butlin, Robert Long and colleagues, "Consciousness in Artificial Intelligence: Insights from the Science of Consciousness" (2023).
Jonathan Birch, The Edge of Sentience (Oxford, 2024).
Anil Seth, Being You: A New Science of Consciousness (2021).
Andreas Matthias, "The Responsibility Gap," Ethics and Information Technology (2004).
Martin Buber, I and Thou (1923).
C.S. Lewis, Mere Christianity and "The Weight of Glory."
Sherry Turkle, Alone Together (2011).

Is your organization wrestling with what AI means for human judgment and leadership? Book Dr. Sweis to speak, or schedule a private consultation.