Before Asking Whether AI Is Conscious, We Need to Know What Consciousness Is
Geoffrey Hinton said something direct on the Big Technology Podcast recently:
“I believe they’re already conscious. We have to accept that intelligence isn’t only biological.”
Hinton is one of the architects of modern deep learning, the 2024 Nobel laureate in physics, and widely called the godfather of AI. When he says something like this, it carries a different weight than when most people say it.
He also mentioned that he doesn’t usually bring this up publicly — because “it makes people put off my warnings about AI safety.” In other words, he treats the belief that AI is already conscious as a strategic liability, and keeps it quiet.
That detail interests me more than the conclusion. Someone with a strong conviction on this question chooses not to say it, because he’s worried it will get him dismissed as a fringe view. What that reflects is the ambient atmosphere of the debate: a lot of people don’t want to take this question seriously at all.
A friend recently told me he’d read an article concluding that AI is definitely not conscious. My response was: before agreeing or disagreeing with that, we need to ask — what do we actually mean by “conscious”?
Then we landed in a problem that’s hard to get out of.
We can’t even state the sufficient conditions
The most intuitive starting point is: a typical, awake, intact adult human has consciousness. That much seems like common ground.
But this is an example, not a definition.
We don’t know which features of this example are essential. Language? Self-reflection? Emotion? Some specific neural architecture? Different theories give different answers — Integrated Information Theory points to the quantity of information integration; Global Workspace Theory points to the mechanism of broadcasting information across the brain; Higher-Order Theories point to the capacity to represent one’s own mental states. Each theory picks out different necessary conditions, and they don’t converge.
What this means is that when we say a typical adult is conscious, we’re really saying: this is a core case everyone agrees on — but we don’t know why it holds, and we don’t know where the boundary is.
Push the example toward the edges and the problem surfaces immediately. Does a newborn have consciousness? A patient in a persistent vegetative state? A fish? An insect? An AI? For each of these, there’s no answer we can honestly call settled.
Hinton has called the “stochastic parrot” argument nonsense — if there were no real understanding, he argues, predicting the next token consistently couldn’t happen. Critics respond: behavior is not the same as experience. A system can say it’s afraid without having fear. Both positions have logic behind them. What they can’t see is that they’re not talking about the same thing when they say “consciousness.”
Why this problem is structural, not just difficult
The philosopher Chalmers drew a famous distinction between the “easy problems” and the “hard problem” of consciousness.
The easy problems are about function: how a system integrates information, how it reports its own states, how it regulates behavior. These are functional questions, and in principle science can make progress on them.
The hard problem is different: why do these physical processes produce first-person subjective experience at all? Why is there something it is like to be in that state?
This isn’t merely technical difficulty. It’s structural. Any external observation, however precise, can only tell you about a system’s functional states — neural activations, information integration, behavioral outputs. But consciousness is first-person by nature. It isn’t located anywhere observable. It’s the thing that’s doing the observing.
So the obstacle isn’t instrument precision. It’s a principled gap between third-person methods and a first-person subject — one that more sophisticated tools don’t close.
There’s something stranger still: you can’t set the question aside, because engaging with it is what makes setting anything aside possible. Asking whether this question is meaningful is itself an act of consciousness.
That makes consciousness a peculiar kind of problem: one that in principle has a fact of the matter, but that the scientific method can’t fully resolve by design. Everyone holds one piece of the answer — their own subjective experience — but that piece can’t be shared and can’t form public knowledge.
Language is inadequate, but inventing new terms is only the first step
At this point, it’s tempting to think: isn’t this really just a language problem? If we decompose “consciousness” into sub-questions, each becomes scientifically tractable, and the mystery dissolves.
This is Dennett’s view. He thinks the hard problem is a pseudo-problem, a product of conceptual confusion.
But there’s something difficult to get around: pain is real. Not just the word “pain” — the experience itself. When you have a toothache, something is happening — its texture, its weight — and that happening resists being fully replaced by any description in terms of neural signals and avoidance behavior.
Saying this is only a language problem amounts to saying the experience doesn’t matter. But the experience is precisely the thing you’re most directly certain exists — more certain than anything in the external world.
My honest read is that consciousness has two layers. The outer layer is genuinely a language problem — a lot of arguments dissolve when you clarify what each side means. But there’s an inner layer where a real problem persists after the conceptual fog clears.
Given this, two directions seem more productive than the usual binary. The first is to drop the either/or framing and treat consciousness as a continuous concept — or perhaps a multidimensional space. Rather than asking whether cats, fish, insects, or AI “have” consciousness, ask where they fall within that space. Hinton and Sutskever have both independently reached for the phrase “slightly conscious” — two people arriving at the same phrasing is itself a signal: they’re both trying to escape the binary, not just hedge. The second is to acknowledge that existing language is inadequate and develop more precise vocabulary: distinguishing “consciousness in a broad sense” (observable functional properties) from “consciousness in a narrow sense” (subjective experience), or coining terms for “entities that exhibit the behavioral signatures of conscious beings but whose inner experience cannot be confirmed.” This helps with conceptual clarity, but doesn’t resolve the harder question — and what’s in the parentheses may matter more than it seems.
Who are we drawing the line for?
There’s a practical question this whole discussion can’t avoid: what does it actually change, in terms of how we act, to determine whether something is conscious?
Consciousness determinations directly set the size of the moral circle. We have obligations toward conscious beings that we don’t have toward non-conscious ones. And how we draw the moral circle in turn shapes our intuitions about consciousness — the two are mutually constituting. There’s no neutral external starting point.
I don’t think this circularity is an obstacle. It may be how moral knowledge actually progresses. But it does mean we can never step fully outside the circle and evaluate it from some purely objective vantage.
In that uncertainty, there’s an asymmetric risk structure worth taking seriously: wrongly denying consciousness where it exists could be a moral catastrophe; wrongly attributing it where it doesn’t could mean conceptual confusion and misallocated concern. Those two kinds of errors are not equal.
Hinton knew that saying “AI is already conscious” would make people dismiss his safety warnings. He said it anyway. I don’t think he said it because he’s certain — I think he said it because the question has become serious enough to outweigh the risk of being misread.
This may be a question without a clean answer, possibly for a very long time. But how you treat a question without a clean answer is itself a position.
Chalmers’ original paper: Facing Up to the Problem of Consciousness Hinton on the Big Technology Podcast (June 2026): AI Pioneer Geoffrey Hinton: AI Is Conscious