First, a note about the obvious racism: western philosophy has long used China as the prototypical example of the foreign. In the Chinese room thought experiment, the Chinese language is used because it is not understood by the person. Ultimately, Chinese is being likened to machine code, notoriously unreadable by humans. This is not the only example of this kind of thing. However, I won’t get into that here. Just be aware that these ideas come from a philosophical tradition of chauvinism.

---

The Chinese room is a thought experiment that goes like this: imagine a room containing a person, several volumes of books, and writing supplies. The room has no windows or doors, only a slot through which paper can be exchanged. We write a message in Traditional Chinese and slip it into the slot. Inside, the person receives the paper, but they do not understand Chinese. However, the books at their disposal contain detailed instructions for drawing out the shapes of characters for a reply based on the shapes of the characters in the input message. The person gradually writes out a response message, a little sloppily, in Traditional Chinese. They then slide it through to the outside where we receive it and can read it. Imagine that, when querying the “Chinese room,” all responses come back totally sensible as if written by an ordinary human. We can ask it questions and converse with it so long as we communicate in Traditional Chinese, and it is completely convincing.

So here’s the point of this thought experiment: the person inside the room does not understand Chinese. The books in the room are just instructions for drawing lines, they could not be used to translate. We wouldn’t normally say that these books contain an understanding of Chinese. Yet, the room appears to understand Chinese. In short: what gives? There is an appearance of consciousness, but that consciousness seems to be attached to this person-book-room system. It is certainly not the consciousness of the person inside the room (they’re okay by the way, we let them out).

John Searle who created this thought experiment used it to argue that, despite the appearance of intelligence and consciousness, there is none there. The room could describe having subjective experiences, but it’s not actually true that it has subjective experiences (is Searle’s point). In other words, consciousness cannot be identified by this kind of outward behavior; rather, it must be formed in a certain way (out of meat). The analogy, of course, is to computers and AI. The person in the room blindly following instructions is like a processor, with the books being the program. This is an argument that AI running on traditional computer hardware cannot by any means achieve sentience, only the appearance of sentience.

I was skeptical of this argument when I first heard it. The program contains all the information for the intelligent behavior we observe, although it is materially inanimate and can only tell a processor what to do. It is a hard question, but recently my perspective on this particular argument changed. After observing modern LLM (Large Language Model) AIs such as ChatGPT, I came to see the Chinese room differently. I always interpreted the argument as having the strongest possible criteria for the room’s responses being convincingly human. I described it that way above intentionally. However, what if the room does not exactly resemble a human, but rather exhibits something that seems like consciousness despite being different from humans? It could be that I was being uncharitable before in my interpretation. In any case, redoing the thought experiment with an “imperfect” Chinese room convinced me that the argument is valid.

While not an expert, I have sufficient understanding of LLMs to say that those that currently exist absolutely do not possess sentience (unless you subscribe to panpsychism, then I’ll allow it). To suggest that an AI of today could be sentient would be akin to someone in 1990 suggesting their computer program could be sentient. We’ve come a long way, but there’s nothing fundamentally different about what we’re doing.

In particular, LLMs do not really simulate a mind. When a human utters a sentence, they are capable of reflecting on the message they want to communicate. LLMs produce language that resembles human language without having any information about meaning beyond word association. If any future AI has the potential to be considered sentient, it would have to be a form of AGI (general intelligence) simulating thoughts and feelings.

---