The Mythology of Conscious AI

I've just read through an excellent article in Noema Magazine by Anil Seth explaining why consciousness is more likely a property of life than of computation and why creating conscious, or even conscious-seeming AI, is a bad idea.  Having learned by now that LLMs do a much better summary of information rich text that I can, I asked ChatGPT 5.2, Google Gemini, and Anthropic Claude to summarize in a few pages the main points in Seth's essay.  All were very good, and I've selected Gemini's reponse to pass on to MindBlog readers:

In his essay, "The Mythology of Conscious AI," neuroscientist Anil Seth argues that our current obsession with creating or identifying consciousness in AI is driven more by psychological biases and flawed metaphors than by biological reality. He contends that consciousness is not a byproduct of smart algorithms but a fundamental property of living, biological systems.

---

I. The Psychological Trap: Why We Want to Believe

Seth begins by identifying why we are so prone to "seeing" consciousness in machines, a phenomenon he compares to pareidolia (seeing faces in toast).

• Intelligence vs. Being: We often conflate doing (intelligence) with being (consciousness). Intelligence is the ability to achieve complex goals; consciousness is the "what-it-is-like-to-be-ness" of an experience.

• The Power of Language: Because language is a hallmark of human exceptionalism, Large Language Models (LLMs) trick our brains. When we say an AI "hallucinates," we are using a word that implies a conscious experience, when "confabulate" (to make things up without realizing it) would be more accurate.

• The Promethean Lure: There is a "techno-rapture" among Silicon Valley elites who view the creation of conscious AI as a path to godhood or digital immortality (mind uploading).

---

II. The Case Against "Turing World"

The foundational belief of AI enthusiasts is Computational Functionalism: the idea that if you simulate the right information processing, consciousness will "emerge" regardless of the material used. Seth challenges this through four main arguments:

1. Brains Are Not Computers

In computers, software is distinct from hardware. In brains, there is no such separation. A single neuron is a living, metabolic machine performing autopoiesis (self-production). Seth argues that you cannot replace a biological neuron with a silicon chip because the neuron's "output" is inextricably tied to its biological waste-clearing and metabolic functions.

2. Alternative Dynamics

Digital computers operate in "Turing World"—discrete steps of 0s and 1s where time doesn't matter. In contrast, biological consciousness is continuous, stochastic (random), and inherently temporal. It flows rather than stutters.

3. Life Matters (Biological Naturalism)

Seth proposes that consciousness is a "controlled hallucination" used by the brain to keep the body alive. Our most basic feelings—mood, hunger, the "feeling of being alive"—are tied to interoception (perceiving the body from within).

Main Point: If consciousness is rooted in the drive to resist entropy and stay alive, a non-living silicon chip has no biological "skin in the game" to anchor such experiences.

4. Simulation is Not Instantiation

A computer simulation of a rainstorm does not make the computer wet; a simulation of a digestive tract does not actually digest food. Seth argues that unless consciousness is purely mathematical, a simulation of a brain is just a "map," not the "territory."

---

III. The Ethical Disaster of "Conscious-Seeming" AI

Seth warns that the pursuit of conscious AI is not just a scientific error but an ethical minefield.

• The Danger of Real Success: If we actually created a conscious machine, we would be creating a new subject capable of suffering, leading to an "exponential" increase in misery.

• The Danger of "Conscious-Seeming" AI: Even if the AI isn't conscious, if it seems conscious, we are vulnerable. We might grant rights to "zombie" algorithms, preventing us from shutting down dangerous systems. Conversely, treating something that seems conscious with cruelty may "brutalize" our own human psychology.

Seth uses the Müller-Lyer illusion (above) to show that even when we know the lines are the same length, we can't help but see them as different. Similarly, we may intellectually know an AI isn't conscious, but our brains are hardwired to feel that it is.

---

IV. Conclusion: Remembering What We Know

Seth concludes that the "mechanization of the mind" is a grave injustice to human nature. By equating our rich, biological experience with the information processing of a chatbot, we overestimate the machine and underestimate ourselves. He suggests that our "soul" is not a piece of immortal software to be uploaded, but rather the "inchoate feeling of just being alive"—something meat can do, but machines cannot.

The Case for Being Human

 ...This Colin Lewis lecture and essay on "Character, Clarity, and connection in a Century of Curated Selves" is a welcome breath of fresh air. I pass on the entire piece:

 

We live in an age so crowded with voices that the unfiltered one now sounds almost shocking.

It’s strange how rare sincerity has become, and stranger still that we now confuse sincerity with naïveté.

To speak plainly is to risk sounding unsophisticated.

To think deeply is to risk being slow.

To feel fully is to risk being fragile.

But character, the old-fashioned word we’ve quietly retired, was never meant to protect us from vulnerability.

It was the discipline of aligning our inner life with our outer one, of letting integrity shape expression.

It demanded more than intelligence; it demanded coherence.

Critical thinking once meant that: not the sport of dismantling others’ arguments, but the patient craft of constructing one’s own, with care, doubt, and moral weight.

It was an act of self-respect, a kind of inner carpentry.

But our culture prizes speed over depth, reaction over reflection.

The algorithm rewards the appearance of certainty, not the work of understanding.

And so, in the noise, we mistake fluency for thought, visibility for virtue, and connection for mere contact.

We are raising a generation fluent in analysis but starved of empathy, able to read a thousand opinions yet unable to feel the gravity of a single human face.

Character, clarity, and connection are not distinct virtues, but the integrated disciplines required to reclaim the authentic, undivided human self from the pressure of performance.

The Discipline of Character

We once spoke of character the way we now speak of talent, as something to be developed, not displayed.

It was an inward architecture that gave a person moral gravity.

You sensed it not by how someone performed under lights but by how they behaved in the dark: what they refused, what they endured, what they stood for when standing carried a cost.

Emerson called it ‘the moral backbone of nature.’

William James called it ‘the faculty for choosing the better course when the worse is easier.’

Today, the word has slipped from our vocabulary, replaced by a constellation of lesser virtues: branding, adaptability, emotional intelligence, the polite euphemisms of a culture that fears judgment but craves approval.

Character was once forged in friction: between impulse and restraint, principle and convenience, public expectation and private conviction.

Now friction is the enemy.

We sand ourselves smooth for compatibility.

We are taught to self-optimize, to fit seamlessly into every new interface, every shifting norm.

The result is a society of impressive surfaces and untested cores.

What was once the slow combustion of the self, trial, error, correction, moral learning, has been replaced by the constant calibration of persona.

The question Who am I becoming? has been outsourced to the analytics of How am I performing?

And yet, without character, the rest collapses.
Critical thinking degenerates into cynicism; connection turns transactional.

The cultivated self was never meant to be frictionless, it was meant to be faithful: to something larger than appetite or trend.

To have character was to live by a standard not of success, but of self-command.

To measure worth not by agreement, but by conscience.In the quiet arithmetic of the inner life, that standard still waits, patient as gravity, reminding us that the project of being human is not to appear consistent but to become whole.

The first fruit of such integrity is humility, the recognition that self-knowledge is incomplete and that moral certainty is often the mask of fear.

That humility becomes the soil in which the next discipline takes root: the clarity of mind that allows us to see truth without distortion

The Discipline of Clarity

Critical thinking is often mistaken for skepticism, but its essence is humility, the willingness to see one’s own mind as fallible, corrigible, unfinished.

To think critically is not merely to doubt; it is to care about whether what you believe is true.

It demands a quiet courage, a resistance to the seductions of belonging.
For all our talk of open-mindedness, the modern mind is rarely open, it is flooded.

We are drowning in data but parched for discernment.

The problem is not ignorance; it’s the inability to tell signal from noise, truth from its infinitely capable impersonators.

Education once sought to train discernment, not just what to think, but how to think about thinking.

The goal was intellectual character: habits of honesty, coherence, proportion, restraint.

Today our institutions reward agility over depth, rhetorical victory over conceptual clarity.

The clever learn to weaponize ambiguity, to speak in the tone of thought without its substance.

In that performative landscape, genuine inquiry feels almost subversive.

It requires patience, solitude, and the unprofitable act of staying with uncertainty long enough for understanding to ripen.

True critical thinking, like moral character, begins in interiority, in self-questioning.

It’s the discipline of cleaning the lens through which we see.

It asks us to distinguish what is clear from what is loud, what is complex from what is confused.

The philosopher Raymond Geuss once noted that clarity is not a natural state but a moral achievement: it costs time, effort, and the willingness to endure ambiguity without surrendering to dogma.

In that sense, thinking clearly is an act of integrity.

The humility that character teaches becomes the clarity that thought requires.

The Courage to Feel

If character is the discipline of the soul, and clarity the discipline of the mind, then emotional authenticity is the discipline of presence.

It’s the art of showing up without armor, of allowing feeling to inform, rather than distort, perception.

We’ve mistaken composure for wisdom, detachment for maturity.
But the mind that cannot feel is no more rational than the heart that cannot think.

To connect honestly with another person is to risk being changed by them.

That risk, not efficiency or eloquence, is the measure of intelligence in its most human form.

The age of constant contact has made us experts at simulation.

We know how to mimic intimacy, how to text empathy, how to perform concern in 280 characters.

But emotional connection, the kind that alters the weather inside us, cannot be automated.

It requires the rarest human resource: attention.

To really listen is to momentarily suspend the self, to admit that the other person exists in a reality as vivid and demanding as your own.

That admission is humbling, and redemptive.It restores proportion to the world.

Neuroscientists now confirm what poets always knew: thought and feeling are not rivals but partners.

Emotion is not noise in the signal of reason; it’s the current that gives reason direction.

To cultivate empathy is therefore not a sentimental indulgence but a cognitive one, the training of perception through compassion.

The unfeeling intellect is not more objective, only more incomplete.

To know without feeling is to see in grayscale; to feel without knowing is to drown in it.

The task, as ever, is integration: to think with the heart and feel with the mind.

The Return to Wholeness

Character, clarity, and connection are not three virtues but three expressions of one longing, the desire to live as an undivided self.

We sense it in moments when our words, thoughts, and feelings align, when what we say reflects what we mean, and what we mean is grounded in what we are.

Such moments are rare because the world conspires against them.

The modern condition is fragmentation, our attention split, our loyalties divided, our selves dispersed across screens and roles.

We are, in Byung-Chul Han’s phrase, “achievement subjects,” endlessly producing ourselves.

Yet the human spirit, when left unharried, still yearns for coherence: to think truly, to act rightly, to feel deeply.

To cultivate that coherence is a rebellion.

It means valuing silence in a world addicted to noise, depth in a culture that worships speed, and sincerity in an economy built on display.

It asks us to become custodians of our own interior life, to guard the private workshop where moral insight, intellectual clarity, and emotional tenderness are forged into something like wisdom.

The old word for this was virtue, which did not mean virtue-signaling or moral posturing, but excellence of being.

To be virtuous was to be real, to be, as Aristotle wrote, ‘at one with oneself.’

In the end, wholeness is not a summary of these disciplines but their reward, the living synthesis of soul, mind, and heart into a single organ of understanding.

To live well in this century will require not new tools, but new depths: a return to the quiet heroism of character, the hard discipline of clarity, and the tender courage of connection.

And perhaps, when someone dares again to speak without polish, to feel without pretense, to think without echo, we’ll recognize them not as naïve, but as free.

Stay curious

 

Do we have an appetite for transcendence?

I've been working on a long rambling post on finding meaning in the flow and evolution of the cosmos rather than in the varied anthropomorphic religions that many cultures have invented.  My motivation to 'find meaning' is not that high, because the quest itself is a human invention derived from assuming the universe was made for us.  I find Paul Bloom in his recent Substack post "Is there a God-shaped hole?" has made a better statement of my position than I could.  I recommend that you read it.  

Venkatesh Rao's 'World Machines'

As usual, I found Venkatesh Rao's writinting in his latest piece "The Divergence Machine" a bit opaque and wordy, so I asked ChatGPT 5.2 to help me out with a condensation and summary, which I archive and pass on to readers here:

Here’s a condensed and clear summary of The Divergence Machine by Venkatesh Rao (Contraptions, Jan 5 2026):

Core Thesis
Rao introduces the concept of the divergence machine as a successor world-machine to the modernity machine explored in his 2025 book club series. The divergence machine’s logic has been built from ~1600–2000 and has fully operated only for about 25 years. It doesn’t reverse modernity but supersedes its organizing principles. Contraptions

World Machines Framework

• World machines are conceptual contraptions that embody how the world orders itself in particular epochs.

• In any period, one machine operates, one is being built, and another may be declining or dying.

• The modernity machine (constructed 1200–1600, dominant 1600–2000) focused on legibility and convergence—making people, systems, and ideas enumerable, interoperable, and canonical.

• The divergence machine rests on a different logic: variety, expansion of differences, and decentralized relationality rather than centralized canonicity. Contraptions

How the Divergence Machine Works

• It does not aim to dismantle modernity directly nor is it just “postmodernity.”

• It takes for granted many effects of the modernity machine (e.g., global connectivity) but operates in spaces where modernity’s centralized canon no longer determines outcomes.

• Its defining move is to make canonicity irrelevant, focusing instead on pervasive diversity and plural modes of being that coexist without unified narratives. Contraptions

Phenomenology and Method
Rao suggests readers filter phenomena through several lenses to see if they belong to the divergence machine rather than to other world processes like:

• Late modernity (zombie persistence of modern norms),

• Postmodernity (reaction/critique of modernity),

• Metamodernity (piecemeal modern revival). Contraptions

Guiding Questions for Divergence
Rao proposes probing events/processes for traits such as:

• Plurality over convergence,

• Relativized canon,

• Generative variety,

• Coexistence without closure,

• Emergent “liveness” not reducible to modern or postmodern categories. Contraptions

Practical Context
This piece sets up the 2026 Contraptions Book Club theme, with readings (e.g., Candide, The Underground Empire, philosophical tensions among Enlightenment figures) intended to explore the formative periods and idea-spaces relevant to the divergence machine’s logic. Contraptions

Big Picture Takeaway
Rao’s divergence machine is a conceptual model for understanding the contemporary world not as a breakdown of modernity but as a phase transition where plurality and decentralized, non-canonical relationality become the driving logic of civilization, distinct from both modern and postmodern frameworks. Contraptions