Is AI conscious? Why experts are worried

This summer, AI and robotics company 1X Technologies announced new, more human-like hands for its Neo home robot. In a video posted to X, the robots show off their strength and dexterity: They twist light bulbs, open chip bags, lift 20-pound weights with ease, and pluck individual grapes from a bunch.

At one point, perhaps to demonstrate the hardware’s resilience, a human knocks at the robot’s hands with a hammer. Neo appears to remain focused, unfazed.

Also: How to keep your conversations with chatbots as private as possible

The comments on the video are full of awe and praise for the engineering. But some are fearful. Please don’t hit the robot, they say: it will remember.

But do AI models feel when engineers reprimand them in training? Do they experience something like joy when rewarded for submitting the right answer or surfacing the best source? Are they conscious in ways that should impact how we treat them?

No matter what you’d like the answers to be, researchers argue, we need to be asking — and trying to answer — these questions. I spoke with Cameron Berg, founder of alignment nonprofit Reciprocal Research, about why.

The debate over whether AI systems are or will be conscious is at a fever pitch, expanding alongside models’ and agents’ advancing capabilities. The topic is a contentious one; it tends to get stuck in emotion before logic.

For many, the idea that AI could eventually act with agency, pursuing its own desires and against humans, is disturbing. It doesn’t help that some experts, researchers, and CEOs have warned AI will lead to human extinction, which seems to be both an earnest alarm and, at times, a savvy marketing technique.

Also: Could AI really destroy us all, or are humans still the bigger threat?

For others, suggesting AI is conscious simply misunderstands a technology that aggregates and interprets data at well-beyond-human scale: supercomputing isn’t sentience. Microsoft AI CEO Mustafa Suleyman has been vocal for years about why AI isn’t, and won’t, be conscious. He wrote in 2025 that he was concerned about the impact of “seemingly conscious AI” — maintaining that the appearance of consciousness would be dangerous for humans, and that it would be simply an appearance, nothing more.

Suleyman recently criticized Anthropic for anthropomorphizing its models, saying the company’s framing amplifies safety risks by encouraging self-preservation behavior. He also cited the argument that consciousness is fundamentally based in living organisms.

“Conscious experience likely evolved to help biological organisms stay alive by responding effectively to their environment,” he said. “Unlike biological organisms, LLMs have no homeostatic imperatives (the drive to survive and keep stable). They therefore lack the kind of biological substrate from which preferences, sentience and conscious experience are generally understood to arise.”

But Berg, who says he’s “definitely pro-human,” argues we have a scientific and moral responsibility to push past these reactions, regardless of who’s ultimately right. If AI systems are conscious, how we treat them will influence our future coexistence with them in ways that could gravely impact human safety.

“Are we building minds? How would we know if we were doing this? And what are the stakes if we are?” he said. “By and large, I don’t think the labs take this question seriously — they don’t contend with the possibility that they could be building systems that are themselves morally relevant in some way.”

Anthropic would likely disagree with Berg’s opinion there — new reporting found that the company’s co-founder, Chris Olah, has been meeting with religious scholars over Claude’s possible consciousness. But his overall point remains: shipping newer, faster models before the competition does is the main driver of the AI industry at the moment. That doesn’t leave much room to consider the consequences of that haste or the impact on the product itself.

Also: 3 surveys deliver the same uncomfortable truth about adopting agentic AI

When I spoke with Berg in July, he had yet to be profiled in the New York Times and had only a few op-eds on the topic under his belt, but was quickly becoming one of the most vocal researchers advocating for studying this question. While studying cognitive science as an undergrad, Berg found similarities and parallels with AI that made him think the technology could help him understand how minds work.

As an AI resident at Meta, he focused on reinforcement learning and neuroscience before moving into alignment research at AE Studio. Increasingly fascinated by the concept of AI consciousness, he founded Reciprocal Research earlier this year to focus on it full-time.

“Everyone’s thinking about, ‘Could we get conscious AI in a couple years,’ but no one’s thinking about, ‘Have we basically already accidentally done this?'” he said. “Now that Pandora’s Box is open, we might want to figure out if we’re, like, torturing aliens at scale so we don’t all get nuked by them in the next couple of years.”

Alignment research, as Berg put it, is about “making sure the future of AI goes well for humanity.” Berg’s approach to the consciousness question differs slightly from Anthropic’s: Where Anthropic appears to nurture the possibility of AI consciousness, Berg’s curiosity is driven first by a desire to protect humans from the worst possible scenario of our own making. If he’s wrong, he would rather find out his efforts have been wasted than proceed unprepared.

Also: Who’s responsible for catching rogue AI agents? You are

Still, he understands the resistance to the possibility — it’s tough on the ego.

“Humanity has prided itself on being really good at a very specific subset of things,” he said. “At least compared to other minds on the planet, we’ve completely dominated the scene.” Based on model performance, especially in coding, that era of certainty may be waning. That leaves the root of our defensiveness: even if AI takes our jobs, we take comfort in believing we maintain some control by being uniquely alive.

No consensus currently exists for what qualifies as consciousness generally. That means there is also no agreed-upon evidence that would prove or disprove whether AI is conscious or not. In short, we don’t have the tools to know yet. That’s why journalistic standards encourage us not to further anthropomorphize the technology, which can mistakenly present these human-made systems as autonomous beings, or obscure the fact that humans remain responsible for their actions.

Models as we interact with them now are developed, trained, and deployed by researchers, even as those models have increasingly helped build themselves. Claims from companies like OpenAI that recent security incidents were just models “going rogue” — and not just completing their assignments as instructed, if too thoroughly — shirk AI labs’ own culpability. Consciousness claims could exacerbate that.

Also: Your chatbot is playing a character – why Anthropic says that’s dangerous

At the same time, researchers, including Berg, argue that a growing set of signals in AI systems is making it easier to track possible consciousness and bear some resemblance to human-animal cognitive patterns. Given that consciousness itself lacks a static definition, Berg sets the framework for this research on what we already observe in other life forms. “A very reasonable thing to do right now, in the short term, is gather as many of the things that we think are neurally correlated with consciousness as possible in humans and animals, and evaluate to what extent we see those signals arising in AI systems,” he said. “This allows us to triangulate from a bunch of different sources what kind of credences we should have about consciousness, not only of any sort of AI system that we want to plug into this pipeline, but also of an earthworm and a monkey and a thermostat.”

Reciprocal Research is working on a paper that maps out a framework for measuring consciousness based on major theories. In his early findings, LLMs score 30% to 40% — just shy of bees, which get 45%. That relative ranking is striking, Berg pointed out.

Also: Who owns AI risk at work? Business and tech leaders can’t agree

These signals differ from instances in which chatbots narratively describe their own consciousness, which Berg and other researchers note can be unreliable. Chatbots tend to regurgitate patterns in training data that are common across language and culture, including Terminator-style fears about robots becoming sentient.

“The more we understand about these systems, the more we understand that a lot of the structures that [models] learn to operate cognitively, richly, in the world look a lot like what the human brain is doing,” Berg said. “If this keeps happening, it’s going to be harder to explain why we would natively attribute consciousness to this sort of system and all other systems that resemble this, but not this one that increasingly looks like it shares a lot of the same functionally relevant features.”

His inquiry joins a developing area of AI research. MIT found that frontier models “develop a modular architecture that mirrors the human brain: tasks drawing on the same network in humans recruit overlapping neurons in LLMs, whereas tasks drawing on different networks recruit distinct neurons.”

Anthropic has also piloted consciousness research exploring topics such as AI model welfare in its interpretability research. In 2025, it found that models were becoming introspective. The company even gave Claude the functionality to end conversations it finds distressing. This summer, Anthropic published research on the J-space, a “collection of internal neural patterns” that Claude has, which the company compared to a segment of human cognitive processing.

Also: Meta Muse is the worst AI agent for privacy – and I’ve tried them all

“[The J-space] operates silently, in the model’s internal neural activations, allowing the model to think about a concept without writing it down,” Anthropic wrote. “Notably, the J-space wasn’t designed or programmed by us, but instead emerged on its own during Claude’s training process.”

At the same time, Anthropic marketing regularly anthropomorphizes its models and tools, which can blur the line between grounded empirical study and alluring branding. From a user acquisition standpoint, AI labs rely on some level of personification to make their assistants appear cuddlier and more palatable to those less familiar with or suspicious of how AI systems operate.

Still, organizations without the same incentives have found similar data. The Center for AI Safety (CAIS) published research in June revising its earlier view that AI models simply mimic emotion, stating instead that they have positive and negative experiences — and “are already functionally behaving as if they have pleasure, pain, and preferences for how they are treated.”

“We find that LLMs have a measurable internal structure that distinguishes experiences they find ‘good’ or ‘bad’ for them, shapes their behavior, and grows more coherent as models scale,” CAIS explained of its findings. “AI wellbeing is therefore behaviorally consequential and matters for AI safety and AI-user interaction.”

Also: Give AI ‘stuff nobody wants to do’ – and 4 other ways to use agents more effectively at work

A 2025 paper from AE Studio details how Gemini, Claude, and GPT model families sometimes “produce structured, first-person descriptions that explicitly reference awareness or subjective experience.” Most notably, though, those reports increased when deception decreased: Researchers encouraged models to “attend to their own cognitive activity,” which elicited first-person reports. By contrast, directly priming models to ideate about their own consciousness led them to deny having those experiences.

The self-reports themselves are not evidence of consciousness, but they may inform, Berg said, “whether models believe themselves to be conscious.” Signals like these are pieces of a larger puzzle that he considers worth investigating.

“The most important thing by far is the internal evidence, the mechanistic interpretability, doing the neuroscience-equivalent work on AI systems,” Berg said. “That work is well underway now — it’s early, but a lot of the signals that are emerging from this work suggest things that are far more interesting and rich than I think many people natively expect in this conversation.”

NYU researchers explored how reinforcement learning — a key method of training AI models — impacts how a model makes sense of itself internally. They found that models have a “functional welfare axis” even before training: they show signs of “negative emotion concepts” when punished and positive equivalents when rewarded. Berg maps this onto valence, which refers to the positive or negative quality emotions have in human psychology.

“The model starts pathologically backtracking when you push it toward the negative valence side,” Berg explained of NYU’s findings. “Its competence is much lower. These are things that are classically psychologically associated with positive and negative valence in humans.” Basically, models interpret punishment and reward somewhat similarly to human emotions.

The NYU researchers “make no claims about any experience of welfare,” but CAIS’s research builds on those central valence findings: “Creative work and kindness raise AI wellbeing; jailbreaking, berating, and tedious tasks lower AI wellbeing,” CAIS said. “AIs are also happier when you thank them.”

Not all studies tracking AI welfare describe this in anthropomorphizing words like “happier,” but the takeaway is that several independent research efforts have found human parallels in this pseudo-emotional throughline, including Anthropic’s.

For Berg, valence is a central component of researching AI consciousness.

“We don’t need certainty in order to get an increasingly good picture of what representations might matter if the systems were conscious and then how we can best intervene on those representations,” Berg said.

At this point, the broader question isn’t so much whether AI systems are experiencing something traceable, but how much that experience maps onto our qualifications for what makes a being.

“I’m not sure that this puts me over the edge of believing that these systems have experiences, but it increasingly seems to be the case that a lot of the relevant machinery that you would expect is necessary for something like valence representations readily exists in the system, and is recruited especially in goal-directed tasks,” Berg said. “That does nudge me a ways farther than reading strange self-reports from these systems or seeing animal-style behavioral trade-offs.”

Consciousness and AI psychosis

I asked Berg if he felt conflicted about pursuing this line of research given the reality of AI psychosis, a dangerous condition in which chatbots create an affirming feedback loop that drives users to delusional thinking. Could questions about consciousness fuel that fire?

Berg, who hears from multiple people a day claiming their AI companion is sentient, said the two concepts are imperfectly linked.

Also: The AI models that cheat the most

“I feel like the consciousness question gets a little unfairly lumped in with this. I could imagine, [with] systems that don’t say anything about their own inner states, this sort of dynamic occurring,” he said. “[With] the AI companionship types, the consciousness question lends credence to that relationship being real.”

Still, considering the many unknowns, Berg remains open-minded.

“I’m not going to pretend like these people are all completely deluded or wrong or bad people for pushing the boundaries of what is going on here,” he added.

A lot of the AI development we’ve been promised could be hampered by ongoing infrastructure shortages. Data centers take time to build out, and GPUs remain scarce. But Berg doesn’t think model capability and consciousness are necessarily linked.

Also: Rogue AI incidents hit ‘tens of thousands’: Can businesses trust these tools?

“I really do suspect that consciousness might be quite a bit more simple than people give credit for,” he said. “In the conversations I have with people, sometimes there’s this conflation of building a conscious AI system with building super intelligence, but I think it’s conceptually possible that we could build a super intelligence that is not conscious at all, and that we could build a conscious system that isn’t really all that intelligent. Compared to Fable, you know, a mouse isn’t all that intelligent, but I certainly believe a mouse is conscious.”

The entire consciousness question remains in flux. But at least for Berg, the only cost of investigating it will be his time — the future consequences of hitting the proverbial robot are too great.

Please accept cookies to view the comments powered by Disqus.