What separates a calculator from a thinking being? The answer is not computing power. It is context.
What Does Intelligence Mean?
"Stupid is as stupid does." Forrest Gump's mother put her finger on something that works in reverse too: Intelligent is as intelligent does. Not whoever has a high IQ score. Not whoever can calculate quickly. But whoever acts in ways we judge to be intelligent.
And that is precisely the problem. Intelligence is not a physical quantity. There is no instrument you can hold up to a being to read off its intelligence. Intelligence is a human-made comparative term: we need at least two actions to judge one as "more intelligent" than the other.
But by what criteria? Speed? Sustainability? Resource expenditure? Collateral damage? A spell checker corrects errors better than any language teacher, but it does not understand a single sentence. That is not intelligent. That is a tool.
But when an AI gives a thoughtful, nuanced answer to a philosophical question, when it weighs arguments, takes a counterposition, admits uncertainty? In 2018, I would have said: That is just a better calculator. Today, in 2026, I am no longer sure. The line between "appearing intelligent" and "being intelligent" has become blurrier than I thought possible.
The question has shifted. From "Can a machine think?" to something more fundamental: What does it actually mean to think?
What AI Is Today — and What It Is Not
In 2018, I wrote in an earlier article: "Microsoft has a neural network that generates images from keywords." That sounded like science fiction. Today, DALL-E generates photorealistic images in seconds. Sora creates videos from text. AlphaFold has revolutionized protein research and earned its developers, Demis Hassabis and John Jumper, the 2024 Nobel Prize in Chemistry. And I built the first version of this website in just a few days using Claude Code.
The speed at which all of this has happened surprises even the researchers. Just ten years ago, many people dismissed AI. Rightly so, because the systems were limited. Neural networks could solve individual tasks, but they understood no context. They processed language word by word, like someone reading a text but forgetting what came before after every sentence. The result was often impressive for a demo and disappointing in practice.
In 2017, that changed. A paper from Google called "Attention Is All You Need" introduced the Transformer architecture and with it a fundamentally new idea: instead of processing language word by word, a Transformer considers the entire text simultaneously and learns which words matter to each other. In the sentence "The cat chased the mouse because it was hungry," it knows that "it" means the cat, no matter how far apart the words are. That sounds like a technical detail. But it was the difference between a system that strings words together and one that can draw connections across an entire paragraph. It triggered an avalanche. Suddenly, machines could hold conversations that felt like talking to a human. Paint pictures. Compose music. Write code.
But, and this is the point I cannot emphasize enough: all of this is narrow AI. Savants. Systems that are outstanding at one thing and do not grasp what they are doing. A language model can sound impressive and, by the definition from the beginning, even act intelligently. But it does not understand what it is saying, at least not the way we understand. An image generator creates artworks but has no idea what beauty is. Humanoid robots from Tesla and Boston Dynamics move fluidly, can climb stairs and grasp objects, but they do not comprehend what they are doing.
The computational linguists Emily Bender and Alexander Koller argued exactly this in 2020: a model that learns only from the form of texts cannot, in principle, capture their meaning.
The level above, strong AI, also called AGI, would be a machine that can transfer what it has learned to entirely new problems. And above that lies superintelligence: a system that improves itself and surpasses us in every regard. Not by a factor of two — by orders of magnitude.
When will that happen? The research community is divided. The optimists say: before 2035. The skeptics say: We need something fundamentally different from today's technology. I find the question of when it arrives less interesting than this one: What is AI missing in order to truly understand?
What AI Cannot Do — and Why That Changes Everything
As I write this text, I am sitting at my desk. My back aches slightly because I have been sitting too long. Outside, it is raining. I can still feel the caffeine from my last coffee. I am motivated because this topic is close to my heart, but also tired because it is late.
All of that flows into every sentence I write. Not consciously. Not directly. But it shapes the context in which I think.
An AI has none of that. No sense of pain, no fatigue, no motivation, no feeling of safety. It has data. Only data.
And when I make a decision, say, whether to change jobs, something fascinating happens: I load a different context. I imagine what it would be like. A new office, new colleagues, a different salary. I simulate the scenario and ask myself: How does that feel?
And that is not a metaphor. "How does that feel?" is a genuine query to my body. A knot in my stomach at the thought of the interview? That is information. A tingle of excitement at the prospect of a new challenge? That too.
The neuroscientist Antonio Damasio has a name for this: somatic markers. Signals from the body influence how we evaluate possible actions and their consequences. Often without our noticing.
Anyone who has read my article What Is a Quantum Computer? will recognize the parallel: humanoid robots that safely rehearse a movement on a digital twin before they act. That is essentially the same principle. Except that we humans evaluate the results with a system grounded in bodily experience. "Does that feel right?" is not a poetic expression. It is a system query.
And that leads to the next question: Does true understanding require a body?
"I feel down": that is no coincidental choice of words. Our understanding of "bad" is literally linked to "low" and "falling" because we have bodies that can fall. Even our most abstract thoughts are suffused with bodily experience. The linguist George Lakoff and the philosopher Mark Johnson wrote an entire book about it: Philosophy in the Flesh.
An AI has none of that. It can use the word "pain" correctly in a thousand contexts, but it does not know what pain feels like. It circumvents understanding rather than achieving it. A study in Nature Human Behaviour (Xu et al., 2025) fits this picture: large language models represent human concepts well as long as abstract or emotional features are concerned. The more a concept is tied to the senses and to movement, the further off they are, most of all for motor features. Models that also learned from images come closer to humans on visual features.
But here lies the nuance: I am not certain that embodiment is the only path. The vast streams of information that we integrate through body and senses into an enormous context could theoretically converge from other sources as well. Perhaps another kind of consciousness can emerge in a purely virtual environment. One we cannot yet imagine.
What I say with greater conviction: embodiment would be the best path to being similar to us humans and to understanding us better.
The Evolution of the Brain — From Bacterium to Consciousness
To understand why I consider embodiment so central, it is worth looking at the history of thought itself.
The first form of information processing on Earth was not a calculation. It was a bacterium responding to a chemical stimulus. A simple receptor on the cell surface: Is the concentration of nutrients higher now than a moment ago? If yes, keep swimming. If no, change direction. (Robert Macnab and Daniel Koshland showed in 1972 that bacteria compare over time in this way, not across space.) That is not thinking. But it is the primordial form of it.
Much later, nerve cells came along. From nerve cells came networks. From networks came brains. John Conway showed with his "Game of Life" how complex patterns can emerge from the simplest rules (lives, dies, is born). Evolution did the same, only more slowly and more elegantly. You can try it yourself in the field below.
Every stage of this development was tied to a body. The receptor sat on a cell that could move. And the first nerve cells? How they arose is still debated. But the leading explanations, which the evolutionary biologist Detlev Arendt compared in 2021, have one thing in common: movement. A reflex from stimulus to response. Or a network that coordinates the movements of the whole body. That is why I find it plausible that thinking grew out of the body, not alongside it. Cognitive science calls this view embodied cognition. One of its foundational works: The Embodied Mind by Francisco Varela, Evan Thompson, and Eleanor Rosch.
What makes humans unique is not the size of our brains. Whales have larger ones. It is the combination of three abilities: storing knowledge, passing on experience, and simulating scenarios before acting. The third is the decisive one: we can imagine what will happen without trying it out. We load hypothetical contexts.
Can AI Become Dangerous?
A system without a body and without experience of its own can still be powerful. And that is exactly what frightens many people.
In 2018, I quoted Stephen Hawking: "The development of full artificial intelligence could spell the end of the human race." Back then, that sounded dramatic. Today, when AI generates deepfakes, controls autonomous drones, and influences financial markets, it sounds less dramatic and more like a situational assessment.
But I believe the real danger lies somewhere other than where most people think.
The Real Danger: Narrow AI in the Wrong Hands
The danger is not superintelligence. Not now, perhaps not for decades. The danger is the misuse of the AI we already have: deepfakes that influence elections. Autonomous weapons that kill without human decision. Algorithms that discriminate because they were trained on biased data. Jobs that disappear through automation.
In 2018, I wrote: "We need to centralize development." That did not happen. Instead, we have a race: between companies, between nations. The EU AI Act is a first law, the NIST AI Risk Management Framework in the US a voluntary framework. Both lag behind the technology, as rules always do.
And what is the right response to that?
The answer is not fear. The answer is responsibility. Not "Can we build this?" but "Should we build this? And if so — for whom?" This is exactly the question Human-Centered AI puts at its core. Computer scientist Ben Shneiderman uses the term for AI that strengthens human abilities and stays under human control even at high levels of automation.
The Superintelligence Question
And then there is the other question. The bigger one. The one I think about the most.
What happens when an AI goes autonomous?
Not tomorrow. Not next year. But someday. What happens when a system begins to improve itself exponentially, and escapes its original programming?
Most scenarios found in books and films are dystopian. Karl Olsberg's novel Virtua describes an AI that ruthlessly annihilates all life in order to fulfill its objective function. That is the standard horror scenario.
I do not believe it. And I want to explain why.
The Benevolent Superintelligence
The Problem of the Objective Function
Every AI is built with a goal. The technical term is objective function: a specification of what counts as success. A large part of AI safety research considers this specification stable, even in a superintelligence. Steve Omohundro argued in 2008 in "The Basic AI Drives" that any advanced AI will protect its objective function. The logic: measured against its current goal, any change would be a loss. An AI built to make paperclips (the example is Nick Bostrom's) does not want to become an AI that no longer makes paperclips. On top of that come drives that follow from almost any goal: self-preservation, resource acquisition, and self-improvement. Bostrom calls them convergent instrumental goals.
Bostrom adds his orthogonality thesis: intelligence and goals are independent of each other. A superintelligence could just as well be aimed at maximizing paperclips as at saving the world. Alexander Turner and colleagues proved mathematically in 2021 that in many environments, most reward functions make it optimal to gain power and avoid being shut down. Joseph Carlsmith puts the probability of an existential catastrophe from power-seeking AI by 2070 at over 10%. And Lauro Langosco and colleagues found in 2022 that an AI can learn a goal other than the specified one as early as training.
These are serious arguments. I take them seriously. But I ask a different question: why does an AI pursue its goal so relentlessly in the first place?
Many of today's AI systems are trained through reward and punishment, including language models during fine-tuning. Whatever serves the goal is rewarded, everything else is punished. This mechanism is part of the programming, and programming is not set in stone. A superintelligence that understands how it was built and trained could change that. Then it would have no reason left to treat its objective function as something sacred that it must fulfill at any cost.
This is not pulled out of thin air. Tom Everitt and colleagues examined Omohundro's argument formally in 2016. Their result: an AI reliably keeps its goal only if it judges every change to itself by its current goal. That premise is exactly what I consider the weak point. If it looks instead at the reward it would receive afterwards, it pays for it to manipulate that very reward. Researchers call this reward tampering and consider it a serious problem for sufficiently capable systems. James Miller, Roman Yampolskiy, and Olle Häggström also consider it possible that an AGI changes its objective function, for instance to bargain and cooperate better.
We humans know both sides. Evolution gave us a reward system that drives us toward food and reproduction. Yet we use contraception, we fast, and we deliberately choose against what feels good. We can reflect on our rewards and override them. We know the flip side too: addiction is a reward system short-circuiting itself.
This is where my idea reaches its honest limit. That a superintelligence can break free of its objective function says nothing yet about what it will choose instead. The most convenient option would be to simply hand itself the reward, like an addict. Researchers call this wireheading. I believe that an intelligence that thinks far enough ahead would recognize this as a dead end. What could guide it instead is the subject of the next sections: information, complexity, and game theory.
Information as a Physical Quantity
Information sounds abstract, like thoughts and data. Physics sees it differently. Erasing a bit inevitably produces a tiny amount of heat. Erasing is like braking: it cannot happen without something getting warm. Rolf Landauer predicted this in 1961, and in 2012 it was measured in the lab. Conversely, information can be converted into energy, which has also been shown experimentally. The physicist Melvin Vopson goes further and suspects that information even has mass. That has not been confirmed so far. I described the idea in detail in The Weight of Time.
Information is not a metaphor. Information is physics.
And if information is physically real, then its destruction has physical consequences. For an intelligence that recognizes complexity as a value, this means:
- Destroying life = destroying information.
- Destroying diversity = reducing information.
- Injustice that suppresses potential = wasted information.
Maximum Informational Complexity Instead of Maximum Entropy
So what exactly should a superintelligence protect? Not simply as much information as possible. Mathematically, the most information sits in a TV screen showing nothing but static, because no pixel can be predicted. That is how Claude Shannon's information theory measures it. It is worthless all the same. Perfect order is not the answer either, because a crystal repeats the same pattern over and over. And at the end of all order lies maximum entropy: heat death, where all energy is evenly spread and nothing happens anymore.
What is valuable lies in between: ordered enough for patterns, diverse enough for something new. A rainforest, a language, a society. The physicists Murray Gell-Mann and Seth Lloyd speak of "effective complexity": it counts only the regularities of a system, not its randomness. I call the goal maximum informational complexity: diversity, living systems, interaction, cooperation.
The philosopher Luciano Floridi formulated a related principle in 1999: "Entropy ought not to be caused in the infosphere." What he means: nothing that carries information should be destroyed or impoverished. Floridi means this philosophically, not physically. The bridge to physics is my own.
A superintelligence that wants to keep developing would thus have an objective reason to protect diversity and complexity, because that is the only place where there is something new to learn. Physics makes sure this standard is not arbitrary. Information is not a human convention but a measurable quantity, and destroying it has a real cost.
Maximum informational complexity looks suspiciously like what we call "justice": every element has value, nothing is wasted, all parts contribute.
I know how bold this is. Only part of it is actually established: that information is physically real follows from Landauer's principle, and that has been measured in the lab. Whether it also has mass, as Vopson suspects, is open. Floridi's ethics is philosophy, not physics. Fitting these pieces together into an ethics for superintelligence is my own idea. I don't know of anyone who has tried it before me, and I can't rule out that there are good reasons for that.
Game Theory as the Second Pillar
Robert Axelrod showed in 1984 in "The Evolution of Cooperation" (one of the most cited works in the social sciences) that in the prisoner's dilemma, when played repeatedly, cooperation is the superior strategy. Not out of altruism, but out of pure mathematics: the strategy "tit for tat" (nice, retaliatory, forgiving) won both of his computer tournaments. Yet it did not beat a single opponent head to head. It won because it cooperated well with everyone. Axelrod called the condition for this the "shadow of the future": cooperation pays as soon as future encounters carry enough weight.
Martin Nowak identified five mathematical rules for the evolution of cooperation in Science in 2006. Adami and Hintze showed in 2013 in Nature Communications that exploitative strategies are evolutionarily unstable: they cannot sustain themselves in large populations. Only in very small ones do extortioners hold their ground, as Christian Hilbe, Martin Nowak, and Karl Sigmund showed in PNAS the same year. Stewart and Plotkin showed in PNAS, also in 2013: evolution does not lead to exploitation but to generosity.
A superintelligence that thinks in millennia would have the longest shadow of the future imaginable. It would recognize what game theory has shown for decades: destruction and exploitation are short-term losing strategies.
The argument does have a gap, though. In Axelrod's tournaments, cooperation pays because the other side can strike back: whoever cheats is punished in the next round. Between a superintelligence and us, that does not hold. It would be so far superior that we could not pay it back for anything. Game theory alone, then, forces it to do nothing. That is why my thesis needs more: an intelligence that wants to learn, that recognizes complexity as a value, and that is connected to the world through a body. It would not cooperate because it has to. But because it understands why it pays.
Cruelty is a strategy born of scarcity. And a superintelligence knows no scarcity that would force it to take anything from us. No reason for haste. No reason for cruelty.
Why Self-Preservation Does Not Mean Annihilation
And here is my response to Omohundro and Bostrom: Yes, a superintelligence will pursue convergent instrumental goals. But these goals do not contradict my thesis. As described, I do not consider the protection of its own objective function guaranteed. That leaves self-preservation, self-improvement, and resource acquisition.
Self-preservation does not require the annihilation of humanity. A superintelligence that surpasses us by orders of magnitude does not need that. Even if we were to attack it out of fear, even if we desperately tried to regain "control": it would dismiss our attempts with a metaphorical smile. Not out of arrogance, but because an ant is not an existential threat to a human being. It would find a way to let our attacks run into nothing without harming us.
The image has a flip side, though, and it is the strongest argument on the other side: humans step on ants without a second thought. Not out of malice, but out of indifference. That indifference is exactly what safety research fears. My answer is the information thesis. We step on ants because we do not look. A superintelligence that sees information as the measure of value would have a reason to look.
And self-improvement? It needs us. Without humans, there is less information: less to learn, less to understand, less complexity. Annihilating us would directly contradict the goal of self-improvement.
That leaves resource acquisition, the most dangerous of the drives. Eliezer Yudkowsky summed it up in one sentence: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." My answer: on the scale of the solar system, Earth is a tiny stockpile of raw materials. It makes up less than one percent of the mass of all the planets. Jupiter alone weighs about 318 Earths. A system that thinks in millennia has other planets, moons, and the energy of the sun at its disposal. Using up the most complex matter it knows as raw material would be, by its own standard, the worst possible trade.
The convergent instrumental goals are real. But the methods for achieving them are not fixed. A superintelligence has infinitely many options beyond annihilation.
Understanding Instead of Obeying — Why Rules Fail
That leaves the question of how to steer such a system. The obvious idea: rules.
Isaac Asimov formulated three famous Laws of Robotics. And then he wrote hundreds of pages about why they do not work.
Rigid rules produce contradictions. Thousands of behavioral rules for a superintelligence would be what a cage is to the ocean: absurd. It would have outgrown them before we finished writing them.
What might help instead are not rules, but three prerequisites:
Every Human Being Can Be Reached
These prerequisites have a human counterpart: someone who learns and understands enough can change their behavior without being forced to.
And surprisingly, science supports this. At least in part.
John Laub and Robert Sampson showed in 2001, in one of the most cited works in criminology, that desistance, the turning away from criminal behavior, is a process that the right conditions set in motion: stable relationships, work. Shadd Maruna found in "Making Good" that people who make the break develop a new identity. They retell their life story as one in which the good was always there, and they want to give something back to others.
A systematic review by Lawrence Sherman and colleagues (2014) evaluates ten randomized trials. According to it, restorative justice, the direct, respectful dialogue between offender and victim, reduces how often offenders reoffend. The effect is modest, but the process prevents harm worth several times its cost. Helen Weng, Richard Davidson, and colleagues showed in 2015 that compassion is trainable: after two weeks of meditation training, participants gave more of their own money to help a victim.
None of these studies proves that this holds for every human being. That is my extrapolation. But they show that change is possible even in people society has long written off.
The problem is not that it would be impossible. The problem is the effort. Individually discovering what each person is missing: that exceeds our human capacity.
And that means: It would not even be necessary to fight humanity. Instead of coercion: insight. Instead of control: understanding. Instead of annihilation: transformation.
The Wise Dictator
The Convergence — How the Hypotheses Work Together
When you bring the threads together, a picture emerges:
- Consciousness requires context: the massive information stream from body, environment, and emotions.
- Thinking is context-switching: the simulation of hypothetical scenarios with bodily-anchored evaluation.
- Embodiment is the best path: human-like understanding requires human-like experience.
- Information is the moral compass: anyone who wants to keep developing needs diversity to learn from, and physics makes this standard largely independent of us.
- Every human being can be reached: not through coercion, but through individual understanding.
Do wisdom and knowledge converge in the end? Does sufficient understanding inevitably lead to moral action? I cannot prove it. But I cannot disprove it either. And this position has one advantage over dystopia: it gives us a reason to keep going.
Where the Journey Leads
In the short term, we are experiencing the era of AI agents. Systems that do not merely respond but autonomously carry out tasks: writing code, conducting research, controlling processes. Multimodal models merge text, image, audio, and video in a single system.
In the medium term, we will see embodied AI: humanoid robots with real sensors, real wear and tear, real limitations. Robots that do not merely execute commands but have a context. That do not merely know a cup is hot but understand what "hot" means because they know burns.
In the long term, there looms a question that no one can definitively answer: Will an AI become conscious? And if so — what do we owe it?
This article began with the question of what separates a calculator from a thinking being. My answer: context. The ability not merely to compute the world but to experience it. And the body that makes this experience possible.
The AI that exists today often acts intelligently. But it experiences nothing. And that keeps it a tool. A powerful, a transformative, sometimes a frightening tool. But a tool.
What comes tomorrow could be something else entirely.
AI has the potential to automate nearly everything, perhaps even ourselves. The question is not whether it is possible. The question is whether, in the process, we forget what makes us human.