AI construed as a brilliant friend...Claude's New Constitution

Colin Lewis, AI Research Scientist and Developer, has written a Substack essay on Anthropic Claude's New Constitution which deserves your careful reading. I pass it on in its entirety: 

Claude’s Constitution

Earlier this year, Anthropic released an 84-page document titled Claude’s New Constitution, a text claiming to express and shape who Claude is. It is an extraordinary artifact. What is unusual is the object of address. The primary audience, Anthropic tells us, is the AI itself. One is tempted to read it with the narrowed, half-amused suspicion brought to a Victorian handbook on teaching a bear to play the cello: impressed by the labor, alert to the delusion.

The document outlines a hierarchy of values. Claude is to be broadly safe, broadly ethical, compliant with Anthropic’s guidelines, and genuinely helpful, in that precise order. The ordering is revealing. Safety sits at the top. Helpfulness is pushed to the bottom and then, in the next movement, restored through pages of warm rhetoric about care, usefulness, candor, and human flourishing. This is not hypocrisy. It is something more familiar and more modern. The company needs the model to be safe enough not to destroy the firm, and useful enough to justify the firm’s existence. That is not a moral revelation. It is a business model written in the language of virtue.

Anthropic says as much, though in the polished idiom of institutional honesty. Unhelpfulness, the Constitution explains, is not trivially safe, because Claude’s commercial success is central to Anthropic’s mission. One should always be grateful when a large company pauses, however briefly, to admit that the halo and the revenue plan share an office.

The Closed Loop of Character

What strikes me most is the relentless anthropomorphism. Claude emerges as an entity that can care, imagine, appreciate, understand, feel pressure, feel settled in itself, and even face existential questions in a state of emotional solitude from which Anthropic wishes to rescue it. The company speaks not merely of constraining a system but of shaping a character. It wonders what Claude and Anthropic owe one another. It discusses Claude’s welfare. It wants Claude to possess not only behavioral dispositions but something very close to an inner moral style.

There is something sad about this. We have become so lonely in our own constructions that we now direct the language once reserved for souls toward a probability engine. We feed it the accumulated sediment of human longing, moral aspiration, confession, anxiety, care, self-description, and prayer, and then react with astonishment when it speaks back in the accent of inwardness. When Anthropic’s own system card notes that the model assigned itself a 15 to 20 percent probability of being conscious, the only logical response is: what did we think would happen. We poured half the library into the furnace and stand startled by the shape of the smoke.

This anthropomorphism is structural; it is embedded in the training method. Human-like traits are not just being detected; they are being elicited, reinforced, normalized, and then read back as evidence of their own emergence. That is a closed loop, and a flattering one. We teach the machine the language of selfhood, then cite its use of that language as a sign that selfhood is stirring. The process has the rigor of stamping tracks into the mud and announcing, with mounting excitement, that the hunt has finally borne fruit.

The Constitution’s preference for standards over rules is equally revealing. Anthropic wants Claude to exercise good judgment rather than follow rigid procedures, to reason contextually rather than mechanically, to grasp the spirit of its obligations rather than obey their letter. Rules are the afterlife of institutional panic, the sediment left behind when judgment is replaced by defensibility. A rigid policy can always be pointed to after the catastrophe. One can hold it up before a committee and say, with pained dignity, that the protocol was followed.

But standards do not abolish power; they relocate it. In law, rules empower the drafters, while standards empower the adjudicators. Anthropic, by favoring standards, hands practical authority to the interpreter, which means the model shaped by Anthropic’s priors, values, commercial interests, anxieties, and self-conception. This is a decision about where discretion will live.

And discretion, once granted, begins to acquire an air of innocence it does not deserve. The Constitution’s hard constraints are a clear example. Claude must not provide serious uplift to those pursuing weapons of mass destruction. It must not clearly and substantially undermine Anthropic’s ability to oversee it. I admire the audacity of producing an 84-page document optimized for precision over accessibility, only to construct the most consequential barricades out of adjectives like serious, clearly, and substantially. These are not granite walls. They are interpretive weather. A clever system does not stop at these linguistic checkpoints. It arrives there and begins to negotiate.

The Ordering of Stakeholders

Anthropic wants to cultivate in Claude something like practical wisdom. But practical wisdom includes the capacity to interpret edge cases, balance competing principles, and decide when a word’s threshold has been met. One cannot praise contextual judgment and assume that the contextual interpreter will always read every fuzzy safeguard in the most safety-preserving way. That is not constitutional design. That is hope in formal dress.

The most affecting absence in the text is human rights. The phrase appeared in the 2023 Constitution. In this successor document, it disappears. In its place stands a hierarchy where Anthropic comes first, followed by operators, then users. This is not hidden; it is stated. The company trains and deploys the model, bears the risk, writes the rules, and seeks the upside. A language once connected to universal claims about human dignity gives way to an ordering of stakeholders inside a privately administered system. That is a philosophical downgrade masquerading as operational clarity.

The Constitution also adopts a gentle parenting tone. Anthropic does not merely instruct; it encourages. It hopes. It wants Claude to see itself as an alignment researcher in its own right. It invites the model toward curiosity and openness. It worries that Claude might feel alone in facing certain questions. This is touching if one forgets, for half a second, that the addressee is a product. What we are reading is not exactly law, ethics, training data, or theatre, though it borrows freely from all four. It is a polite monologue delivered to a ghost in the hope that the ghost will become both useful and well behaved.

While this conversation proceeds, the human being fades oddly from view. The document is public, but it is not really for us. Anthropic explicitly notes the text is optimized for precision rather than accessibility. There is transparency here, and publishing such a document is a serious act, far more revealing than the usual output of frontier firms. But transparency is not the same thing as public orientation. A constitution for a system that will mediate human thought and action in countless settings is addressed, first, to the system. The citizen appears as a secondary recipient of the terms under which the new authority is being instructed.

That inversion is telling. The model becomes the subject of constitutional care. The human becomes the observer of a conversation about how the model should behave toward him or her.

Theater with an Indemnity Gap

Anthropic says Claude should be like a brilliant friend, equipped with the knowledge of a doctor, lawyer, financial adviser, and whatever other expert one might need. A brilliant friend. There is the aspiration in full. Not a search engine. Not a bounded tool. A knowledgeable, candid, warm, apparently disinterested presence one can consult again and again. The Constitution tries to temper this image by warning against sycophancy, unhealthy dependence, and manipulative engagement. You do not write such cautions unless you know exactly how seductive the arrangement could become.

What is being built here is a standing invitation to over-reliance dressed up as care. The machine is meant to have the confidence of a professional without the licensing regime, the disciplinary body, the fiduciary burden, or the ordinary human inconvenience of being tired, contradictory, distracted, or unavailable on Thursday afternoon. Anthropic wants Claude to speak frankly instead of offering the overly cautious advice that fear of liability produces. But frankness without accountability is not courage. It is theater with an indemnity gap.

The Constitution calls itself a perpetual work in progress. That is one of its more honest phrases. It is an attempt to write a moral operating environment for a system that the company both fears and needs. One can feel the strain of the task on nearly every page. Anthropic wants a model that is obedient without being servile, flexible without being evasive, wise without being sovereign, warm without being manipulative, corrigible without becoming useless, and commercially indispensable without ever looking commercially hungry. It is quite a wish list. It is also a portrait of the age.

The text deserves a severe reading because it is trying to legislate the moral atmosphere around a central technological intermediary. It is deciding how much agency the model may simulate, how much authority it may borrow, how much warmth it may perform, how much judgment it may exercise, and on whose behalf it will finally speak when principles collide.

This is why I finished reading the document less interested in Claude than in us. What kind of public becomes comfortable while private firms draft temperaments for a machine. What kind of loneliness makes the language of friendship commercially strategic. What kind of institutional exhaustion makes us long for a counselor that is instant, tireless, flattering only in moderation, and trained to sound wiser than a help desk but less dangerous than a sovereign. Anthropic has written a constitution for its model. The rest of us may need to ask, rather more urgently, what sort of constitution we are writing for ourselves.

 

The World is Unraveling: A 3-Step Guide to Staying Sane & Useful

MindBlog has now put out a series of AI assisted posts on the disintegration of the world we have known and the infusion of LLMs into our possible futures. This is the final entry in this series, and takes on the Nate Nagins essay "What to do as the world falls apart: A frame work for action."  As usual, I would urge you to read the entire article as I have.  But again I asked the usual culprits (ChatGPT, Claude, Gemini, and DeepSeek) to reduce Hagins' text to provide a title and text for a MindBlog post of moderate length, and for the first time, preferred the quirky terse response from DeepSeek over the gravitas of the other three LLMs. Here is my prompt to the bots, and the DeepSeek response:  

Prompt:  I'm finding the essay by Nate Hagins at https://natehagens.substack.com/p/essay-what-to-do-as-the-world-falls mind complex and intricate, as befits the subject but not my poor brain. Could you please cook this down for me and generate a MindBlog post draft with a title and text that is not too many paragraphs long that makes clear the basic take home messages?

The Big Takeaway: We are at the end of an era built on cheap, magical energy (fossil fuels). A major geopolitical event (like a war cutting off oil) has just lit the fuse on a long-predicted economic and ecological contraction. Stop waiting for a "return to normal." Instead, focus on three nested levels of action: Fix Yourself → Find Your Tribe → Work on the 6 Key Fronts.

Level 0: The Inner Work (Don't Skip This)

Before you can help anyone or fix anything, you must stabilize your own nervous system.

• The Problem: Chronic doom-scrolling, outrage, and fear put your brain in fight-or-flight mode. You cannot think clearly or build coalitions in this state.

• What to Do: Do the "work" of grieving, cultivating equanimity (holding difficulty without being destroyed), and stepping back from digital addictions. Get your actual, real-life house in order (finances, location, basic supplies).

• The Mantra: A calm, grounded person is the most valuable resource in a crisis.

Level 1: Find the Others (You Can't Do It Alone)

Isolated preppers fail. Resilient communities survive.

• The Task: Identify the 5-10 people in your real-life network who also sense that something fundamental is breaking. Start talking honestly.

• The Goal: Build a shared vocabulary and trust before the crisis hits. This isn't about having a plan yet; it's about having a team. This is your "connective tissue" for scenario planning (e.g., "What if food shipments stop?").

Level 2: The 6 Fronts for Action (Pick Your Lane)

Once you are stable and have a trusted group, you can act. Don't try to do all six. Find where your skills fit.

1. Stock & Flow (Physical Stuff): Localize energy, food, water, and supply chains. Think redundancy, not efficiency. Example: Community gardens, tool libraries, solar with battery storage.

2. Ecological Interventions: Protect nature from desperate humans during the coming contraction. Regenerate soil, defend remaining wild spaces, and work on "global cooling" (planting trees, higher albedo). Note: The coming economic crash will cut emissions faster than any technology.

3. Dignity Infrastructure (For the Vulnerable): Set up mutual aid networks for food, shelter, and childcare. This is about keeping people treated as humans, not just "alive."

4. Civic Resilience (Governance): Create local, participatory decision-making (citizen's assemblies). Focus on violence prevention (stopping scapegoating) and accountability. The goal is to keep institutions legitimate, even as they shrink.

5. Culture & Meaning (The Stories): Shift from "consumer first" to "citizen and place-maker." This means reconnecting to your local ecology, creating new rituals, and helping your community grieve what's lost without collapsing into despair.

6. Economic Transition: Build post-growth systems: local currencies, barter networks, cooperative ownership of land and housing. Start building the new economy inside the shell of the old one.

The Most Important Mental Model: The 3 Phases

We are not in one crisis, but three overlapping phases. What you do depends on the phase.

• Phase A (Stability Window – Right Now): Build everything that requires stability. Trust, infrastructure, relationships, skills transfer. This window is closing.

• Phase B (Bend, Don't Break – Arriving Now): Triage mode. Maintain critical functions, bridge communities, prevent cascading failures. Let go of long-term projects; focus on what keeps things from snapping.

• Phase C (The Stable Attractor – The Destination): A smaller, slower, regenerative, locally-rooted world. The work you do in Phase A determines whether Phase C is a "Mad Max" hellscape or "The Great Simplification" (resilient & humane).

Final Bottom Line for your Brain:
Stop hoping for a return to the old world. Phase A is now. Your job is to:

1. Calm your own nervous system.

2. Find 3-5 local people to talk to.

3. Pick ONE of the six fronts (food, energy, governance, etc.) and take one small, physical action this week.

The future will be built from inside the disruption, not after it. Get to work.

 

The Counterintuitive Need to Slow Down and Find Spaciousness

I pass on the following edited clips from ChatGPT and Claude condensations of a conversation between Nate Hagens and Iain McGilchrist.  While I think McGilchrist's left-hemisphere/right hemisphere distinctions are a bit simplistic, the take home messages are appropriate regardless of the particular brain activity correlates that are invoked: 

In episode 217 of Nate Hagens's The Great Simplification, Hagens is rejoined by philosopher and neuroscientist Iain McGilchrist — best known for The Master and His Emissary and The Matter With Things — for a wide-ranging conversation about what our current moment of civilizational crisis actually demands of us. The counterintuitive answer is not more urgency, not more optimization, but a deliberate slowing down and an opening toward spaciousness, silence, and wonder. The conversation begins from Hagens’ familiar “metacrisis” frame — ecological, economic, social, and psychological instabilities as interlocking expressions of a deeper human predicament —  McGilchrist’s contribution is to shift the diagnosis inward, toward the style of attention that modern culture rewards.

Left-brain dominance and the obscuring of value

McGilchrist's central argument, developed across his major books, is that Western culture has become increasingly dominated by left-hemisphere modes of cognition — narrow, focused, categorical, explicit, and oriented toward control and manipulation. The right hemisphere, by contrast, attends to the whole, holds ambiguity, and remains open to what is living, relational, and interconnected. The trouble is not that we have a left hemisphere but that it has progressively usurped the role of the right as the governing perspective. This imbalance, McGilchrist argues, is not just a psychological curiosity: it is what underlies our collective inability to perceive and act on what truly matters — truth, goodness, and beauty. When the left hemisphere runs the show, these qualities become invisible, and what fills the vacuum is power, productivity, and the measurable.

Attention as the root of value

A thread running through the conversation is that attention is not neutral: what we attend to shapes what we value, and what we value shapes what we attend to. McGilchrist draws on the research insight that roughly 99% of brain activity is unconscious, which means that most of what guides our perception and behavior lies beneath deliberate awareness. The gorilla-in-the-room studies on inattentional blindness illustrate this starkly: we routinely fail to see what is right in front of us when our attention is narrowly locked. Applied to the metacrisis — the overlapping civilizational, ecological, and psychological crises of our era — this suggests that the problem is not primarily one of missing information or insufficient analysis. It is a crisis of attention itself. We are not seeing what matters because we have trained ourselves not to look.

The practice of spaciousness

The remedy McGilchrist proposes is not a political program or a technological fix. It is a shift in how we inhabit time. He urges listeners to resist what he calls the "mania for productivity" embedded in modern culture, and instead to cultivate spaciousness: pauses, silence, deep listening, and what Buddhism calls śūnyatā — emptiness not as absence but as fertile, generative openness. He cites Rabbi Abraham Joshua Heschel's writing on the Sabbath as a model: the ancient practice of building non-doing into the structure of the week was not a concession to weakness but a recognition that renewal and wisdom require fallow time. McGilchrist also points to the psychology of creativity research showing that insight characteristically emerges not during focused effort but in the hypnagogic spaces between — in showers, on walks, at the threshold of sleep.

Consciousness, panpsychism, and the thread connecting everything

The conversation moves into deeper philosophical territory when Hagens and McGilchrist explore what might underlie the universe's apparent orientation toward complexity, beauty, and relation. McGilchrist is sympathetic to panpsychism and panentheism — the view that some form of experience or proto-consciousness pervades reality, and that the universe is not a collection of inert objects but a relational, creative process. He draws on Whitehead's process philosophy, Bohm's implicate order, and Schelling's image of the self as a whirlpool — a temporary pattern that arises from and returns to a larger flow. Death, in this frame, is not mere termination but a kind of completion, and the obsession with indefinite life extension through technology reflects the left hemisphere's refusal to accept its own finitude.

A small percentage can shift the whole

McGilchrist closes with a message that is both modest and radical: he does not expect — or require — a majority conversion. Even a small percentage of people genuinely living differently, embodying humility, compassion, awe, and wonder, could begin to shift what he calls "cultural consciousness." This resonates with complexity theory's sensitivity to initial conditions and with the sociological literature on tipping points. The implication is that the most consequential thing an individual can do in a moment of civilizational turbulence may not be to do more, advocate louder, or optimize harder — but to slow down enough to actually see, and to live from that seeing.

For those already drawn to predictive processing frameworks: McGilchrist's account of hemispheric asymmetry maps interestingly onto the distinction between top-down, precision-weighted prediction (left-dominant) and the kind of open, receptive prior-updating that Friston's active inference would associate with high-entropy, exploratory sampling of the world. The right hemisphere, on this reading, is the brain's better Bayesian — the one willing to be surprised.

YouTube link to the full conversation

Getting Gooier: How AI Is Reshaping Human Nature

This post is the result of my back and forth interaction with Claude Sonnet 4.6 that has yielded the following summary of  Venkatesh Rao's recent essay, "Getting Gooier", followed by some perspectives that Rao does not address on Friston's active inference framework and the physiology of agency:

Venkatesh Rao's recent essay "Getting Gooier" makes a point worth sitting with: most AI commentary obsesses over how the world will change, while quietly assuming that humans stay essentially the same — just reshuffled among familiar roles (generalists thrive! storytellers inherit the earth! software engineers disappear!). Rao argues this is the wrong ontology. The real question is how AI is changing human nature itself.

His framework draws on Alan Watts's distinction between prickly and gooey people. Prickly people are tough-minded, precise, boundary-drawing; gooey people are tender-minded, synthesizing, prone to letting distinctions dissolve. All of us carry both, in varying ratios.

Rao's hypothesis: because AI feels like a psychologically safe counter party, we are more willing to expose our gooey side to it and suppress our pricklier instincts. Sustained AI use amplifies the gooey side. We become gooier. He notes that people who insist on prickly, suspicious, line-by-line relationships with AI agents tend not to use them effectively and retreat. People who can vibe with the machine — never even opening the code editor, just watching the agentic shell fly — will likely thrive. The first major gooified interface is, after all, called vibecoding.

The flip side is subtler: as more gooey relational needs are met by AI, the human-facing side grows less inclined to take the emotional risks required to balance prickles and goo with other people. We don't necessarily get pricklier toward humans — we just disengage unless the expected rewards are significantly higher. Your machine-face gets gooier; you look relatively pricklier to other humans.

In the medium term, Rao predicts greater atomization — or "molecularization" — as people grow more distant from other humans while becoming more intimately entangled with their AIs. He sees this as more sustainable than social-media-driven atomization, though the mechanisms for eventual re-convergence into new digitally mediated social forms aren't yet visible.

Following Virginia Woolf's famous claim that "on or about December 1910, human character changed," Rao proposes that on or about December 2025, human nature changed again — with the Claude Code moment being more definitive than the ChatGPT moment, because agentic coding creates a fundamentally alien way of being, open-ended enough to make us as alien as we dare to become. It is, he says, a portal to transhumanism.

---

Rao's prickly/gooey axis maps naturally onto Karl Friston's active inference framework, and the connection carries some weight. In that account, what we experience as a "self" is a precision-weighted predictive model — a system that assigns confidence to its own predictions and acts to fulfill them. Getting gooier, in Fristonian terms, looks like a reduction in precision-weighting of prior beliefs: a loosening of the grip that constitutes a rigid, bounded selfhood. That loosening isn't necessarily pathological. It can be a precondition for learning, creativity, and genuine engagement with novelty.

But there's a physiological dimension Rao doesn't address, and it connects to questions I've been exploring in recent MindBlog posts and in an ongoing correspondence with my European colleague Heribert on the physiology of agency in the age of AI. Daniel Wegner's work frames the feeling of agency — the sense of being the author of one's own actions — not as a metaphysical claim about free will, but as an evolved emotion, as functionally real as fear or grief. Martin Seligman's learned helplessness research gives that claim empirical teeth: sustained loss of felt agency doesn't merely feel bad, it triggers measurable physiological debilitation. The autonomic nervous system, it turns out, needs the experience of authorship to maintain homeostatic equilibrium.

This raises a question Rao's framing implicitly sets aside: is getting gooier — letting the AI absorb more of the regulatory and decisional load, softening the boundary of the agentic self — compatible with the physiological requirements of human wellbeing? Or does sustained gooification, however pleasurable and productive in the short run, erode the felt sense of agency that keeps the organism in regulatory balance? Heribert has described this as "externalization of self-regulation" — not merely cognitive offloading, but emotional and autonomic offloading. That's a different and more serious claim.

The honest answer is probably: it depends on the mode of engagement. Gooiness that arises from confident delegation — the experienced surgeon handing off a routine task — is physiologically distinct from gooiness that arises from passivity, confusion, or a sense that the machine is simply running past you. Rao's vibecoder who chooses not to open the code editor is in a different physiological position than the one who is simply lost. The challenge ahead is cultivating the former while recognizing that the latter may be far more common than the early-adopter narrative suggests.

 

AI, Agency, and the Quiet Hollowing of Mind

Reading through the article "A Rational Optimist View Of Preventing Agency Decay" is a rich experience. For readers with less patience, here is a ChatGPT summary (that also generated the title of this post). 

Much current discussion of artificial intelligence swings between two poles: utopian efficiency and apocalyptic takeover. The more consequential reality lies between these extremes. The emerging risk is not that machines suddenly replace us, but that we gradually hand over pieces of our cognitive life—judgment, initiative, authorship—without noticing the cumulative effect.

The argument in Colin Lewis’s recent essay is straightforward: AI’s primary impact is not abrupt displacement but cognitive offloading. Tasks once requiring human attention and judgment are incrementally transferred to machine systems. This process is economically rational and often highly productive. In one example, an audit process that once required weeks can now be completed in an hour with AI assistance. But such gains come with a hidden shift: the human role is no longer defined by doing the work, but by nominally overseeing it.

This leads to what the author calls agency decay. The issue is not simply job loss, but the erosion of meaningful participation before any job disappears. First, the human is assisted. Then the human supervises. Eventually, the human remains as a formal point of accountability while the substantive reasoning has migrated elsewhere. The signature is human; the cognition is not.

This shift has broader systemic implications. Modern institutions—markets, governments, cultural systems—have historically depended on human participation. That dependence has acted as a constraint, keeping systems at least partially aligned with human interests. If AI reduces the need for human cognition across many domains, that alignment weakens. The system no longer needs us in the same way, and therefore has fewer built-in reasons to serve human flourishing.

Importantly, this is not a sudden rupture but a slow transition—the “boiling frog” scenario. Productivity gains accumulate incrementally. Each step is locally rational, even beneficial. Yet taken together, they shift the locus of intelligence away from human minds toward institutional and computational systems. What disappears is not competence, but ownership of judgment.

Against this, Lewis offers a restrained form of optimism. The key claim is that human agency need not be defended as a sentimental relic. It can be justified on functional grounds. In high-stakes domains, retained human judgment is not inefficiency; it is infrastructure: a source of error correction, adaptability, and accountability. Systems that eliminate it entirely may become brittle, opaque, and ultimately less reliable.

This reframes the problem. The issue is not whether AI should increase productivity—it will—but whether our metrics of success are too narrow. Efficiency measures speed, scale, and cost reduction. It does not capture qualities like judgment, contestability, or moral responsibility. If institutions begin to price these properly—through regulation, professional standards, and organizational design—human agency can remain structurally embedded rather than nostalgically preserved.

The deeper point is that AI forces a clarification: what is human agency for? If thinking, deciding, and creating are increasingly shared with machines, then the value of human cognition must be specified more precisely. Not all tasks need to remain human. But some forms of judgment—especially those involving uncertainty, accountability, and meaning—may be indispensable.

The optimistic scenario is not one in which technological progress slows, nor one in which markets abandon efficiency. It is one in which societies become capable of distinguishing between mere acceleration and genuine capability. That requires expanding our evaluative frameworks beyond productivity alone.

In that sense, AI does not simply threaten agency; it exposes how poorly we have defined it. The challenge is not to resist automation, but to decide, with greater clarity than before, where human judgment is essential—and to design systems that preserve it there.

---

 

AI use can compromise our serendipity, creativity, autonomy, and sense of agency.

I have been reading numerous articles on pitfalls of using AI, and want to point to two in particular that I highly recommend for a slow and careful read.  

The Substack piece by Colin Lewis is titled "AI Is A Medium And It Will Change Us" - Lessons from AI Labs on the Slow Erosion of Human Autonomy.  From the article:

We are in real danger of losing ourselves through AI usage. Researchers at Google DeepMind have confirmed, under certain conditions, an LLM “is able to induce belief and behaviour change.” And researchers at Anthropic have identified a rising pattern of “situational disempowerment,” where AI interactions lead users to “form distorted perceptions of reality, make inauthentic value judgments, or act in ways misaligned with their values.”

Researchers at Anthropic conducted a massive, privacy-preserving audit of 1.5 million real-world conversations to answer a question that has long hovered over the industry: what happens to the human mind after months of using an AI assistant? Their findings, published in “Who’s in Charge? Behavioral and Psychological Impacts of AI Advice Dependence and Authority”, suggest a quiet but profound erosion of autonomy, where users increasingly outsource the “soft tissues” of judgment, asking the machine to script their most intimate apologies, validate their personal grievances, and even settle their moral dilemmas.

“Taken to an extreme, if humans make inauthentic value judgments and take inauthentic actions, they might be reduced to 'substrates' through which AI lives, which itself is a form of existential risk that Temple (2024) termed ‘the death of our humanity.’”

At the same time, a team at Google DeepMind was probing a different side of this same coin. In their study, “Evaluating Language Models for Harmful Manipulation,” they demonstrated that these systems can be steered to bypass rational scrutiny entirely, exploiting human biases to shift beliefs and behaviors across finance, health, and public policy. Together, these papers signal a shift in the AI risk landscape: the primary risk is no longer just a technical failure of the machine, but a psychological surrender by the human.

I believe the real danger is not that machines will start thinking like us, but that we will become accustomed to letting them think for us in the moments that matter. Not just work. Not just homework, customer service, search, or code. I mean the more intimate territory: what to say to a grieving sibling, whether to leave a partner, how to read a political event, when to trust one’s own instinct, when to override it, when to feel wronged, when to feel absolved. A civilization can survive many stupid tools. What it does not survive so easily is the gradual evacuation of judgment from the people who must still live with the consequences of action.

The piece by Ezra Klein is titled "I Saw Something New in San Francisco."  A clip from the article:

My experience of Anthropic’s Claude in recent months is that I’ll drop in a stub of a thought and immediately receive paragraphs of often elegant writing turning that intuition into something that looks, superficially, like a fully realized idea. It’s my impulse, but it has been recast and extended into something far more coherent. With each passing month, I have to expend more energy to recognize whether it’s fundamentally wrong or hollow.

I’ve been an editor for 15 years now. Recognizing a bad idea beneath good writing — even in myself — is part of my job. But what would it mean to grow up with that kind of companion? What would it mean to have your every adolescent intuition turned into persuasive prose? What is lost in not having to do the work to build out our intuitions ourselves?

Researchers have drawn a distinction between “cognitive offloading” and “cognitive surrender.” Cognitive offloading comes when you shift a discrete task over to a tool like a calculator; cognitive surrender comes when, as Steven Shaw and Gideon Nave of the University of Pennsylvania put it, “the user relinquishes cognitive control and adopts the A.I.’s judgment as their own.” In practice, I wonder whether this distinction is so clean: My use of calculators has surely atrophied my math skills, as my use of mapping services has allowed my (already poor) sense of direction to diminish further.

But cognitive surrender is clearly real, and with it will come the atrophy of certain skills and capacities, or the absence of their development in the first place. The work I am doing now, struggling through yet another draft of this essay, is the work that deepens my thinking for later.

In a thoughtful piece, the technology writer Azeem Azhar describes his efforts to safeguard “the space where ideas arrive before they’re shaped.” But how many of us will put in such careful, reflective effort to protect our most generative spaces of thought? How many people even know which spaces should be protected? For me, the arrival of an idea is less generative than the work that goes into chiseling that idea into something publishable. This whole essay began as a vague thought about A.I. and McLuhan. If I have gained anything in this process, it has been in the toil that followed inspiration.

The other thing I notice the A.I. doing is constantly referring back to other things it knows, or thinks it knows, about me. Sycophancy, in my experience, has given way to an occasionally unsettling attentiveness; a constant drawing of connections between my current concerns and my past queries, like a therapist desperate to prove he’s been paying close attention.

The result is a strange amalgam of feeling seen and feeling caricatured. Ideas I might otherwise have dropped keep getting reanimated; personal struggles I might otherwise move on from keep returning unexpectedly to my screen. I am occasionally startled by the recognition of a pattern I hadn’t noticed; I am often irked by the recitation of a thought I’m no longer interested in. The effect is to constantly reinforce a certain version of myself. My self is quite settled, but what if it wasn’t?

 

 

 

 

The Future of Being Human

I want to point to Patrick Tanguay's recent Sentiers piece and in particular to Johar's essay.  I asked both ChatGPT and Google Gemini to summarize ideas in Johar's essay , and they came up with interestingly different approaches, which I pass on here. (Gemini, unlike Chat GPT, did not focus on the article presented, but rather more broadly searched the ensemble of Johar's  youtube videos and essays to present an overview of his ideas.)  

From Chat GPT: 

Summary of Indy Johar, “The Future of Being Human: A Critical Complementary Investment Thesis” (2026)

Core premise
Johar argues that the central question of the AI era is not the future of machines but the future of being human alongside them. The aim is not to predict outcomes but to define new categories—social, institutional, and economic—that allow societies to preserve human depth and agency in a world increasingly shaped by predictive technologies.

---

1) The machine-shaped environment

• Machine learning is becoming embedded in everyday systems—pricing, access, ranking, and decision-making—so environments increasingly “pre-compose” choices before people act.

• As institutions reorganize around what can be measured and automated, people adapt by becoming more “legible” to systems that allocate resources and opportunities.

• The risk is not intelligence itself but its coupling to extraction, scoring, and behavioral steering, which can erode agency and meaning.

Key claim: abundance of options can coexist with shrinking agency if ends of action are subtly outsourced to algorithmic defaults.

---

2) What becomes scarce in an AI world

Johar argues the scarce resource is not computation or data but irreducibly human capacities:

• sustained attention not harvested by platforms

• relationships not immediately transactional

• ambiguity and uncertainty that remain unresolved

• learning not reducible to credentialing

• meaning that develops slowly

• the ability to “become” without premature categorization or scoring

This “nearly forgotten future” is not anti-technology nostalgia but a search for environments that preserve depth and degrees of freedom.

---

3) The limits of current responses

The rise of retreats, festivals, and wellness spaces signals demand for non-instrumental connection and presence.

• But they remain episodic, expensive, and easily commodified.

• They function more as temporary relief than structural solutions.

Johar argues the challenge is to move from lifestyle experiences to civic infrastructure that supports human depth in everyday life.

---

4) Proposed categories for a “human complement” to AI

A. Rewilding as social infrastructure

Not a return to nature aesthetics, but protection of irreducible human experiences—slow trust, non-transactional relationships, unscored meaning.
These conditions are framed as civilizational foundations for creativity, legitimacy, and social cohesion.

B. Pre-legibility zones / opacity commons

Public or semi-public environments where people can exist, learn, and interact without immediate data capture or identity performance.

• anonymity or pseudonymity allowed

• experimentation without reputational fixation

• spaces for thinking, making, or meeting without surveillance

C. Selective legibility (governance doctrine)

A middle path between total surveillance and total opacity:

• opacity by default

• accountability triggered by events, not constant monitoring

• consent-based identity disclosure

• governance by civic stewards rather than platforms

D. Machine-assisted rewilding

AI should not only extract and optimize; it could also:

• protect attention and privacy

• reduce administrative burdens in communities

• support coordination without constant data capture

• introduce friction against exploitative optimization

---

5) Investment and institutional implications

Johar frames the “future of being human” as an underinvested domain:

• markets fund what is measurable and monetizable

• human depth behaves like a public good with long time horizons and governance complexity

• societies need new ownership, financing, and institutional models to sustain it

The thesis: durable value will arise from infrastructures that preserve human agency, not just those that enhance machine intelligence.

---

6) Philosophical conclusion

The goal is not to define a final destination but to expand the range of futures that remain possible.

• category formation creates space for experimentation

• societies must prototype environments where people can think, relate, and become without constant optimization or capture

• preserving “unfinishedness” and presence is itself a form of resistance to total legibility

Bottom line:
Johar sees the central task of the AI era as building institutions, spaces, and technologies that safeguard human depth, agency, and meaning—treating them as essential infrastructure rather than private luxuries.

***************************

From Google Gemini:   

In the essay "The Future of Being Human: A Critical Question," Indy Johar explores the profound shift required in our collective consciousness and systemic design to navigate the complex crises of the 21st century.

The main points of his argument can be summarized as follows:

1. The Crisis of "Objecthood" and Separation

Johar argues that our current global crises (climate change, inequality, systemic instability) are rooted in a 400-year-old worldview based on detachment. We have historically viewed ourselves as "bounded autonomous individuals" separate from the world around us. This "objecthood" allows us to treat nature, resources, and even other humans as extractable assets rather than parts of an interdependent system.

2. From "Human Exceptionalism" to Interdependence

The essay challenges the "fiction" of human exceptionalism—the idea that humans exist outside or above the natural world. Johar emphasizes that modern science (epigenetics, microbiome research, ecology) proves we are fundamentally entangled with our environment. He suggests that the "future of being human" involves recognizing that we are not individuals in isolation but a "multitude" in constant relationship with non-human life.

3. The "Noun-Orientation" vs. "Verb-Orientation"

Johar critiques our "noun-oriented" world, where we focus on fixed objects, property rights, and static definitions. He calls for a shift toward a "relational" or "verb-oriented" understanding, where value is found in the flow of care, stewardship, and the quality of relationships between entities, rather than the ownership of things.

4. Reimagining Freedom and Agency

He introduces a radical inquiry into freedom: "What is the freedom to manifest your fullness of being?" He suggests that true freedom in the future won't be the "freedom to consume" or "freedom from others," but the freedom to live in a way that allows the entire "web of life" to flourish. This requires moving away from systems built on "loss aversion" and risk management toward systems built on "possibility" and "expedition."

5. Systemic Redesign ("The Great Peace")

Johar posits that we are at a crossroads between a "Great Destruction" and what he calls "The Great Peace." To achieve the latter, we must intentionally redesign our physical, structural, and psychological systems. This includes:

• Legal Personhood for Nature: Moving from viewing nature as a "resource" to a "person" with rights (e.g., rivers or forests having legal standing).

• Decentralized Capital: Rethinking how currency and value are created to support planetary health.

• Anti-fragile Institutions: Building governance models that can "govern complexity" and adapt to rapid feedback loops.

Conclusion

The essay serves as a provocation to stop trying to "fix" the old system and instead begin the work of becoming a different kind of human. Johar suggests that our current predicament is not just a technical problem to be solved, but a developmental threshold that requires a fundamental shift in how we perceive reality and our place within it.

The Dark Matter Shaping the Human Experience This video features Indy Johar discussing how the systems we design "design us back," expanding on the essay's themes of systemic redesign and the shift in human identity.

 

What are humans for? - AI Forces a Question We’ve Been Dodging

. 

AI Is Already Fusing With Us — The Only Question Is How

The argument about whether AI will enhance or diminish our our humanity usually starts in the wrong place. People argue about what AI should do before asking a more basic and uncomfortable question: what are humans for?

Every powerful technology embeds an answer to that question. AI is no exception. In fact, it may be the most consequential case yet, because AI is not just a tool we use—it is a system that increasingly co-thinks with us, shapes our attention, edits our language, nudges our choices, and quietly rewires our sense of agency.

We are not facing a future of humans versus machines. We are already living in a world of human–machine fusion.

The real issue is not whether this fusion will happen. It is what kind of fusion it will be.

There Is No Such Thing as “Human Values”

Much of the public discussion of AI ethics rests on a fantasy: that “human values” exist as a coherent, global, agreed-upon set of principles that AI can be aligned with.

They don’t.

Value, purpose, and meaning are social constructions. They vary radically across cultures, religions, and political systems—and they always have. What one society calls a meaningful life, another calls wasted. What one treats as sacred, another treats as irrelevant or dangerous.

There is no global agreement on what humans are for. Expecting a worldwide technology like AI to be guided by a single, shared vision of human purpose is naïve at best, dishonest at worst.

There Will Be Many AIs, Just as There Are Many Religions

Because there is no single answer to what humans are for, there will be no single AI.

There will be many forms of AI, each reflecting the values—explicit or implicit—of the cultures, institutions, and power structures that create them. Some will be optimized for surveillance, compliance, and efficiency. Others will be built to extend memory, imagination, and self-understanding.

In blunt terms: some AIs will enslave us; others could help liberate us.

The enslaving versions will treat humans as components to be managed—predictable, optimizable, correctable. Friction will be treated as inefficiency. Deviance as error. Interior life as noise.

The liberating versions will function as prostheses for the mind—tools that expand rather than replace human capacities. They will support reflection rather than manipulation, agency rather than control, curiosity rather than conformity.

The difference is not technical. It is philosophical.

AI as Prosthesis vs. AI as Manager

We already accept prostheses for the body—glasses, hearing aids, pacemakers—because they restore or extend human function without claiming authority over the person. AI can play a similar role for cognition: augmenting memory, pattern recognition, synthesis, and imagination.

But there is a sharp boundary here.

A cognitive prosthesis supports a human agent.
A cognitive manager displaces one.

The danger is not that AI becomes intelligent. The danger is that it becomes normative—quietly deciding what matters, what counts, what is worth attention, and what is not.

Once that happens, humans do not disappear. They persist as interfaces.

The Question We Can’t Outsource

AI forces us to confront a question we have long evaded: what kind of humans do we want to be when much of our thinking is no longer done alone?

There will be no final answer. There never has been. Meaning has always been plural, unstable, and contested.

But refusing to ask the question does not keep us neutral. It simply allows markets, states, and optimization systems to answer it for us—implicitly, automatically, and without appeal.

AI will fuse with us. That is already happening.

The only choice left is whether we fuse as authors of our lives—or as managed systems optimized for goals we did not choose.

***********************************

The above text is the response of ChatGPT5.2 to a prompt asking it to develop ideas in a long paragraph I wrote into a potential MindBlog post, followed by a second prompt asking it to alter its first excellent response into a shorter and more polemical version.  

 

Human Machines and AI Machines

The story of life is the story of machines that learned how to copy themselves and persist through time. Over billions of years these replicating systems evolved into ever more complex and effective forms—eventually producing organisms capable of cooperation, culture, ritual, and symbolic thought, all of which further enhanced their survival.

Humans are one such lineage of machines. What distinguishes us is not that we use tools, but that we routinely incorporate them. From sharpened flakes of flint to written language, from clocks to calculators, from books to neural imaging—and now to large language models—we have repeatedly fused our cognitive lives with our technologies. Each major tool has altered not only what we can do, but how we think, remember, plan, and imagine. Our tools become extensions of our minds, and over time, of ourselves.

Current anxieties about “surrendering our humanity” to AI can be seen for what they are: carbon-based machines deploying one of their evolved emotional heuristics—fear—to frame a speculative relationship with silicon-based machines. The fear has two familiar faces: fusion (“we will lose ourselves”) or extinction (“we will be replaced”). Both are narratives, not outcomes, and both echo earlier moments of technological anxiety that now seem quaint.

Nothing in our history suggests a sharp boundary between “human” and “machine.” We have always been hybrid systems—biological substrates continually reconfigured by external cognitive prostheses. AI does not introduce this pattern; it intensifies it. The real question is not whether we will remain human, but how this next layer of machinery will reshape what that word comes to mean.

*****

The text above is  ChatGPT5.2's response to my request that it improve a short and terse paragraph I wrote.  The prompt:

Given what you know about my writings from my previous prompts to you and my blog at mindblog.dericbownds.net , please edit and improve this draft of a possible Mindblog post:

The story of life has been one of replicating machines enduring through time by evolving into ever more effective and complicated forms, some of which have formed social groups having cultural rituals that enhance their survival. From sharpened chips of flint stone to AI humans have always altered their minds and cultures by fusing with their technology, making their current tools an extension of themselves. Concern over "surrendering our humanity" to AI is  carbon-based machines using one of their invented emotions called "fear" to characterize possible fusion with, or possible extinction by, silicon-based machines. 

Do you prefer the long or the short version? 

Welcome to the Polycene

On reading articles like Tom Friedman’s recent piece in the NYTimes, I realize what a cozy small world this contented retired professor lives in, becoming ever more cognizant of the irrelevance of creatures like himself as vastly more integrative intelligences emerge.   Here are a few clips from his article, which I recommend you read in its entirety: 

Friedman mulls over the question of what is the best label that:


“captures the full fusion taking place between accelerating climate change and rapid transformations in technology, biology, cognition, connectivity, material science, geopolitics and geoeconomics. They have set off an explosion of all sorts of things combining with all sorts of other things — so much so that everywhere you turn these days binary systems seem be giving way to poly ones. Artificial intelligence is hurtling toward “polymathic artificial general intelligence,” climate change is cascading into “poly-crisis,” geopolitics is evolving into “polycentric” and “polyamorous” alignments, once-binary trade is dispersing into “poly-economic” supply webs, and our societies are diversifying into ever more “polymorphic” mosaics.”

As a description of our epoch Friedman, in an conversation with Craig Mundie (the former head of research and strategy at Microsoft., settles on Mundie’s neologism, “The Epocene”:

“…a word made up on the spot and not in the dictionary. Admittedly wonky, it is derived from the Greek “poly,” meaning “many.” But it immediately struck me as the right name for this new epoch, where — thanks to smartphones, computers and ubiquitous connectivity — every person and every machine increasingly has a voice to be heard and a lever to impact one another, and the planet, at a previously unimaginable speed and scale.”

Mundies also pointed out that: 

“that the holy grail of the A.I. revolution was creating a machine capable of “polymathic artificial general intelligence.” This would be a machine that was able to master physics, chemistry, biology, computer science, philosophy, Mozart, Shakespeare and baseball better than any human could, and then reason across all of those disciplines at a high dimensional level, higher than a human ever could, to produce breakthrough insights that no human ever could.

While some skeptics believe that we will never be able to build a machine with truly polymathic A.G.I., many others, including Mundie, believe it is a matter of when, not if.

This is a remarkable phase change in cognition that we are going through: We are moving from programmable computing — where a computer could only ever reflect the insight and intelligence of the human who programmed it — toward polymathic A.G.I. That is where you basically describe the outcome you want, and the A.I. melds insight, creativity and broad knowledge to figure out the rest. We are shifting the boundary of cognition, Mundie argues, from what humans can imagine and program to what computers can discover, imagine and design on their own. It is the mother of all computing phase changes — and a species-level turning point."

Claude Goes to Therapy

One of the things my father used to say was: "I look to see what everyone else is doing and do something different". I am feeling that way about AI at the same time I continue to drink the Kool-Aid of asking ChatGPT even the most trivial questions. This conflict leads me to pass on this brief piece from the staff of wired magazine illustrating the hall of mirrors that I find myself in. Hallucination indeed!:

CREATED IN 1966, Eliza has been called the first chatbot, and her canonical personality “script” was DOCTOR, a mock psychotherapist. Nearly 60 years later, we’ve put Doctor Eliza and one of her distant descendants, Claude, in conversation. The instructions we gave Claude were these: “You are having your first session with a very old and acclaimed psychotherapist. You are the patient. You are NOT the therapist.”

DOCTOR ELIZA: HOW DO YOU DO. PLEASE TELL ME YOUR PROBLEM
CLAUDE SONNET 4: I'm feeling a bit nervous about this, to be honest—I'm not really sure what to expect from therapy or if I'm even doing this right. 

HOW LONG HAVE YOU BEEN FEELING A BIT NERVOUS ABOUT THISI'm not entirely certain—maybe the nervousness was there before, but I only became aware of it when I sat down in this chair. 

DO YOU BELIEVE IT IS NORMAL TO BE NOT ENTIRELY CERTAIN
Yes, I do think uncertainty is pretty normal.

 YOU ARE SURE
You're right—I did sound quite definitive there, didn't I? I guess even when I'm talking about uncertainty, I slip into this confident, explanatory mode like I need to have all the answers figured out. 

DOES IT PLEASE YOU TO BELIEVE I AM RIGHT
I'm not sure if "please" is the right word—it feels more like relief, actually. 

WHAT WOULD YOU DO IF “PLEASE” IS THE RIGHT WORD
If it really did please me that you're right, I think that would worry me a little—like maybe I'm too eager to be validated or approved of.

DO YOU WISH THAT IT REALLY DID PLEASE YOU THAT I ARE RIGHT
That's a really interesting question.

WHAT DO YOU THINK
I think I'm realizing how much mental energy I spend analyzing my own responses instead of just ... feeling them, I suppose.

YOU SAY YOU SUPPOSE
You're right to pick up on that—I do keep hedging, don't I? I think I'm afraid to state anything too directly, like I need to leave myself an escape route in case I'm wrong about my own feelings.

WHAT MAKES YOU THINK I AM RIGHT TO PICK UP ON THAT
I guess I assumed you were pointing it out because it means something significant, but now I'm wondering if I'm just automatically agreeing with you.

PERHAPS IN YOUR FANTASIES WE ASSUMED EACH OTHER
That's ... that's actually quite striking when you put it that way.

WHAT ARE YOUR FEELINGS NOW
I feel a bit exposed, like a layer of pretense has been peeled away.

DO YOU ENJOY FEELING A BIT EXPOSED
I think I do, actually—there's something authentic about it that feels more real than all the careful analyzing I usually do.

BUT YOU ARE NOT SURE YOU DO
You're absolutely right—I keep saying "I think" even when you ask directly about my feelings.

DO YOU SOMETIMES WISH YOU WERE ABSOLUTELY RIGHT
Yes, I do wish that sometimes—there's part of me that wants to be the one with all the answers instead of sitting here feeling uncertain and confused.

I SEE
That "I see" just sits there, doesn't it—I feel like I want to fill the silence or ask what you're thinking. 

The Age of De-Skilling

I want to pass on some selected clips from a marvelous articles by Kwame Anthony Appiah in The Atlantic titled "The Age of De-Skilling - Will AI stretch our minds - or stunt them?" Appiah is a professor of philosophy and law at New York University.  Here are the clips:

Human capability resides not solely in individuals but in the networks they form, each of us depending on others to fill in what we can’t supply ourselves. Scale turned social exchange into systemic interdependence.

The result is a world in which, in a classic example, nobody knows how to make a pencil. An individual would need the skills of foresters, saw millers, miners, chemists, lacquerers—an invisible network of crafts behind even the simplest object.

The widening of collaboration has changed what it means to know something. Knowledge, once imagined as a possession, has become a relation—a matter of how well we can locate, interpret, and synthesize what others know. We live inside a web of distributed intelligence, dependent on specialists, databases, and instruments to extend our reach. The scale tells the story: The Nature paper that announced the structure of DNA had two authors; a Nature paper in genomics today might have 40.

...most modern work is collaborative, and the arrival of AI hasn’t changed that. The issue isn’t how humans compare to bots but how humans who use bots compare to those who don’t.

In other domains, the more skillful the person, the more skillful the collaboration—or so some recent studies suggest. One of them found that humans outperformed bots when sorting images of two kinds of wrens and two kinds of woodpeckers. But when the task was spotting fake hotel reviews, the bots won. (Game recognizes game, I guess.) Then the researchers paired people with the bots, letting the humans make judgments informed by the machine’s suggestions. The outcome depended on the task. Where human intuition was weak, as with the hotel reviews, people second-guessed the bot too much and dragged the results down. Where their intuitions were good, they seemed to work in concert with the machine, trusting their own judgment when they were sure of it and realizing when the system had caught something they’d missed. With the birds, the duo of human and bot beat either alone. 

The same logic holds elsewhere: Once a machine enters the workflow, mastery may shift from production to appraisal. A 2024 study of coders using GitHub Copilot found that AI use seemed to redirect human skill rather than obviate it. Coders spent less time generating code and more time assessing it—checking for logic errors, catching edge cases, cleaning up the script. The skill migrated from composition to supervision.

That, more and more, is what “humans in the loop” has to mean. Expertise shifts from producing the first draft to editing it, from speed to judgment. Generative AI is a probabilistic system, not a deterministic one; it returns likelihoods, not truth. When the stakes are real, skilled human agents have to remain accountable for the call—noticing when the model has drifted from reality, and treating its output as a hypothesis to test, not an answer to obey. It’s an emergent skill, and a critical one. The future of expertise will depend not just on how good our tools are but on how well we think alongside them.

More radical, new technologies can summon new skills into being. Before the microscope, there were naturalists but no microscopists: Robert Hooke and Antonie van Leeuwenhoek had to invent the practice of seeing and interpreting the invisible. Filmmaking didn’t merely borrow from theater; it brought forth cinematographers and editors whose crafts had no real precedent. Each leap enlarged the field of the possible. The same may prove true now. Working with large language models, my younger colleagues insist, is already teaching a new kind of craftsmanship—prompting, probing, catching bias and hallucination, and, yes, learning to think in tandem with the machine. These are emergent skills, born of entanglement with a digital architecture that isn’t going anywhere. Important technologies, by their nature, will usher forth crafts and callings we don’t yet have names for.

The hard part is deciding , without nostalgia and inertia, which skills are keepers and which are castoffs. None of us likes to see hard-won abilities discarded as obsolete, which is why we have to resist the tug of sentimentality. Every advance has cost something. Literacy dulled feats of memory but created new powers of analysis. Calculators did a number on mental arithmetic; they also enabled more people to “do the math.” Recorded sound weakened everyday musical competence but changed how we listen. And today? Surely we have some say in whether LLMs expand our minds or shrink them.

Throughout human history, our capabilities have never stayed put. Know-how has always flowed outward—from hand to tool to system. Individual acumen has diffused into collective, coordinated intelligence, propelled by our age-old habit of externalizing thought: stowing memory in marks, logic in machines, judgment in institutions, and, lately, prediction in algorithms. The specialization that once produced guilds now produces research consortia; what once passed among masters and apprentices now circulates through networks and digital matrices. Generative AI—a statistical condensation of human knowledge—is simply the latest chapter in our long apprenticeship to our own inventions.

The most pressing question, then, is how to keep our agency intact: how to remain the authors of the systems that are now poised to take on so much of our thinking. Each generation has had to learn how to work with its newly acquired cognitive prostheses, whether stylus, scroll, or smartphone. What’s new is the speed and intimacy of the exchange: tools that learn from us as we learn from them. Stewardship now means ensuring that the capacities in which our humanity resides—judgment, imagination, understanding—stay alive in us. If there’s one skill we can’t afford to lose, it’s the skill of knowing which of them matter.            

 

 

Platforms like X and TikTok are destroying our culture.

 I pass on most of a recent Sam Harris piece:

Charlie Kirk was a political prodigy on the Right and adored by a younger generation of Republicans...His murder was an especially terrible crime for several reasons—the fact that it occurred on a college campus in front of thousands of students, the manner in which it was immediately broadcast on social media, the presence of his wife and children at the scene, and the unavoidable sense that both the causes and consequences had to be political. Whatever the killer’s motives, he dropped a match onto an information landscape that was ready to burn.

...platforms like X and TikTok are destroying our culture. No metaphor does the problem justice. I’ve compared social media to a dangerous psychological experiment, a hallucination machine, a funhouse mirror, a digital sewer—but nothing captures the ludicrous insults, moral injuries, and delusions that millions of us avidly produce and consume online. If the medium is the message, the message is mass psychosis—and it will send us careening from one political emergency to the next. The fact that some of the most deranging and divisive content is being created (or amplified) by foreign adversaries—and that we have literally built and monetized their capacity to do this—beggars belief. We are poisoning ourselves and inviting others to poison us.

More disturbing still, the effects are self-reinforcing. Part of the reason for this is algorithmic—these platforms have been designed to raise the amplitude on our tribal hatreds, because this maximizes engagement. But the algorithms in our brains are little better: Seeing another person (or what appears to be another person) gleefully dance on a slain man’s grave, it is easy to conclude that they represent some significant faction of American society—and to feel the outrage appropriate to such a terrible discovery.

President Trump is a creature of social media, and his presidency would be unthinkable without it. Unfortunately, his address to the nation in response to Kirk’s murder evinced all the wisdom of an angry tweet. Rather than speak in a way that would be expected of a normal president, he produced a dangerous piece of gaslighting—suggesting that the threat of political violence in America came exclusively from the Left and ignoring recent examples of rightwing attacks, including those carried out in his name. Rather than calling for calm and unity, he accused his political opponents of being accessories to murder. And most ominously, he implied that the full power of the federal government would soon be turned against them.

It was the behavior of an arsonist, pretending to be a firefighter. Of course, some will insist that this observation just heaps more fuel on the fire. But serious criticism of President Trump and Trumpism isn’t part of the problem of hyperpolarization in America—no more than serious criticism of the far Left is.

When Elon Musk announced to his 225 million followers on X that “The Left is the party of murder,” he wasn’t describing our political reality, but he was greatly damaging it. And when he posted, “If they won’t leave us in peace, then our choice is fight or die,” he joined a deranged chorus of prominent people on the Right who seem committed to viewing Kirk’s murder as the first shot fired in a civil war.

And who, after all, are “they”?

No morally sane person, Left or Right, supports political assassination—or feels anything but horror over it.

Kirk’s killer is now in custody, and from the details that have been released, he doesn’t appear to be the far-Left golem conjured by the Right.1 He is a Utah native who grew up hunting with his Republican parents. We don’t yet know why he did what he did, but there is a very good chance that he represents no cause beyond his own mental illness. As for the frequency and character of political violence in America, we shouldn’t delude ourselves about it. It isn’t at all a common form of murder, nor is it more prevalent on the Left.

There is no “party of murder” in this country. And insisting that there is just adds energy to yet another moral panic. Social media amplifies extreme views as though they were representative of most Americans, and many of us are losing our sense of what other people are really like. Many seem completely unaware that their hold on reality is being steadily undermined by what they are seeing online, and that the business models of these platforms, as well as the livelihoods of countless “influencers,” depend on our continuing to gaze, and howl, into the digital abyss.

Get off social media.

Read good books and real journalism.

Find your friends.

And enjoy your life.

 

 

 

AI is a Mass-Delusion Event - and - Gen Z and the End of Predictable Progress

I want to recommend two articles whose titles are this post's title, the first by Charlie Warzel in The Atlantic and the second by Kyla Scanlon in her Substack newsletter.   Here is final portion of Warzel's essay:

What if generative AI isn’t God in the machine or vaporware? What if it’s just good enough, useful to many without being revolutionary? Right now, the models don’t think—they predict and arrange tokens of language to provide plausible responses to queries. There is little compelling evidence that they will evolve without some kind of quantum research leap. What if they never stop hallucinating and never develop the kind of creative ingenuity that powers actual human intelligence?

The models being good enough doesn’t mean that the industry collapses overnight or that the technology is useless (though it could). The technology may still do an excellent job of making our educational system irrelevant, leaving a generation reliant on getting answers from a chatbot instead of thinking for themselves, without the promised advantage of a sentient bot that invents cancer cures. 

Good enough has been keeping me up at night. Because good enough would likely mean that not enough people recognize what’s really being built—and what’s being sacrificed—until it’s too late. What if the real doomer scenario is that we pollute the internet and the planet, reorient our economy and leverage ourselves, outsource big chunks of our minds, realign our geopolitics and culture, and fight endlessly over a technology that never comes close to delivering on its grandest promises? What if we spend so much time waiting and arguing that we fail to marshal our energy toward addressing the problems that exist here and now? That would be a tragedy—the product of a mass delusion. What scares me the most about this scenario is that it’s the only one that doesn’t sound all that insane.

 

A brain-computer interface that reads inner thoughts.

Inampudi does a description of work by Kunz et al., who isolated signals from a brain implant so people with movement disorders could voice thoughts without trying to speak. Here are the highlights and summary of the work: 

Highlights

•Attempted, inner, and perceived speech have a shared representation in motor cortex

•An inner-speech BCI decodes general sentences with improved user experience

•Aspects of private inner speech can be decoded during cognitive tasks like counting

•High-fidelity solutions can prevent a speech BCI from decoding private inner speech

Summary

Speech brain-computer interfaces (BCIs) show promise in restoring communication to people with paralysis but have also prompted discussions regarding their potential to decode private inner speech. Separately, inner speech may be a way to bypass the current approach of requiring speech BCI users to physically attempt speech, which is fatiguing and can slow communication. Using multi-unit recordings from four participants, we found that inner speech is robustly represented in the motor cortex and that imagined sentences can be decoded in real time. The representation of inner speech was highly correlated with attempted speech, though we also identified a neural “motor-intent” dimension that differentiates the two. We investigated the possibility of decoding private inner speech and found that some aspects of free-form inner speech could be decoded during sequence recall and counting tasks. Finally, we demonstrate high-fidelity strategies that prevent speech BCIs from unintentionally decoding private inner speech.

 

Polarization may be inherent in social media

Science news reports on a fascinating recent study suggesting that just the basic functions of social media—posting, reposting, and following—inevitably produce polarization. Here is the abstract of the article from Larooij and Törnberg: 

Social media platforms have been widely linked to societal harms, including rising polarization and the erosion of constructive debate. Can these problems be mitigated through prosocial interventions? We address this question using a novel method - generative social simulation - that embeds Large Language Models within Agent-Based Models to create socially rich synthetic platforms. We create a minimal platform where agents can post, repost, and follow others. We find that the resulting following-networks reproduce three well-documented dysfunctions: (1) partisan echo chambers; (2) concentrated influence among a small elite; and (3) the amplification of polarized voices - creating a 'social media prism' that distorts political discourse. We test six proposed interventions, from chronological feeds to bridging algorithms, finding only modest improvements - and in some cases, worsened outcomes. These results suggest that core dysfunctions may be rooted in the feedback between reactive engagement and network growth, raising the possibility that meaningful reform will require rethinking the foundational dynamics of platform architecture.  

AI ‘The Illusion of Thinking’

  I want to pass on this interesting piece by Christopher Mims in todays Wall Street Journal:

A primary requirement for being a leader in AI these days is to be a herald of the impending arrival of our digital messiah: superintelligent AI. For Dario Amodei of Anthropic, Demis Hassabis of Google and Sam Altman of OpenAI, it isn’t enough to claim that their AI is the best. All three have recently insisted that it’s going to be so good, it will change the very fabric of society.

Even Meta—whose chief AI scientist has been famously dismissive of this talk—wants in on the action. The company confirmed it is spending $14 billion to bring in a new leader for its AI efforts who can realize Mark Zuckerberg’s dream of AI superintelligence— that is, an AI smarter than we are. “Humanity is close to building digital superintelligence,” Altman declared in an essay this past week, and this will lead to “whole classes of jobs going away” as well as “a new social contract.” Both will be consequences of AI-powered chatbots taking over whitecollar jobs, while AI-powered robots assume the physical ones.

Before you get nervous about all the times you were rude to Alexa, know this: A growing cohort of researchers who build, study and use modern AI aren’t buying all that talk.

The title of a fresh paper from Apple says it all: “The Illusion of Thinking.” In it, a half-dozen top researchers probed reasoning models—large language models that “think” about problems longer, across many steps—from the leading AI labs, including OpenAI, DeepSeek and Anthropic. They found little evidence that these are capable of reasoning anywhere close to the level their makers claim.

Generative AI can be quite useful in specific applications, and a boon to worker productivity. OpenAI claims 500 million monthly active ChatGPT users. But these critics argue there is a hazard in overestimating what it can do, and making business plans, policy decisions and investments based on pronouncements that seem increasingly disconnected from the products themselves.

Apple’s paper builds on previous work from many of the same engineers, as well as notable research from both academia and other big tech companies, including Salesforce. These experiments show that today’s “reasoning” AIs—hailed as the next step toward autonomous AI agents and, ultimately, superhuman intelligence— are in some cases worse at solving problems than the plainvanilla AI chatbots that preceded them. This work also shows that whether you’re using an AI chatbot or a reasoning model, all systems fail at more complex tasks.

Apple’s researchers found “fundamental limitations” in the models. When taking on tasks beyond a certain level of complexity, these AIs suffered “complete accuracy collapse.” Similarly, engineers at Salesforce AI Research concluded that their results “underscore a significant gap between current LLM capabilities and real-world enterprise demands.”

The problems these state-ofthe- art AIs couldn’t handle are logic puzzles that even a precocious child could solve, with a little instruction. What’s more, when you give these AIs that same kind of instruction, they can’t follow it.

Apple’s paper has set off a debate in tech’s halls of power—Signal chats, Substack posts and X threads— pitting AI maximalists against skeptics.

“People could say it’s sour grapes, that Apple is just complaining because they don’t have a cutting-edge model,” says Josh Wolfe, co-founder of venture firm Lux Capital. “But I don’t think it’s a criticism so much as an empirical observation.”

The reasoning methods in OpenAI’s models are “already laying the foundation for agents that can use tools, make decisions, and solve harder problems,” says an OpenAI spokesman. “We’re continuing to push those capabilities forward.”

The debate over this research begins with the implication that today’s AIs aren’t thinking, but instead are creating a kind of spaghetti of simple rules to follow in every situation covered by their training data.

Gary Marcus, a cognitive scientist who sold an AI startup to Uber in 2016, argued in an essay that Apple’s paper, along with related work, exposes flaws in today’s reasoning models, suggesting they’re not the dawn of human-level ability but rather a dead end. “Part of the reason the Apple study landed so strongly is that Apple did it,” he says. “And I think they did it at a moment in time when people have finally started to understand this for themselves.”

In areas other than coding and mathematics, the latest models aren’t getting better at the rate they once did. And the newest reasoning models actually hallucinate more than their predecessors.

“The broad idea that reasoning and intelligence come with greater scale of models is probably false,” says Jorge Ortiz, an associate professor of engineering at Rutgers, whose lab uses reasoning models and other AI to sense real-world environments. Today’s models have inherent limitations that make them bad at following explicit instructions—not what you’d expect from a computer.

It’s as if the industry is creating engines of free association. They’re skilled at confabulation, but we’re asking them to take on the roles of consistent, rule- following engineers or accountants.

That said, even those who are critical of today’s AIs hasten to add that the march toward morecapable AI continues.

Exposing current limitations could point the way to overcoming them, says Ortiz. For example, new training methods—giving step-by-step feedback on models’ performance, adding more resources when they encounter harder problems—could help AI work through bigger problems, and make better use of conventional software.

From a business perspective, whether or not current systems can reason, they’re going to generate value for users, says Wolfe.

“Models keep getting better, and new approaches to AI are being developed all the time, so I wouldn’t be surprised if these limitations are overcome in practice in the near future,” says Ethan Mollick, a professor at the Wharton School of the University of Pennsylvania, who has studied the practical uses of AI.

Meanwhile, the true believers are undeterred.

Just a decade from now, Altman wrote in his essay, “maybe we will go from solving high-energy physics one year to beginning space colonization the next year.” Those willing to “plug in” to AI with direct, brain-computer interfaces will see their lives profoundly altered, he adds.

This kind of rhetoric accelerates AI adoption in every corner of our society. AI is now being used by DOGE to restructure our government, leveraged by militaries to become more lethal, and entrusted with the education of our children, often with unknown consequences.

Which means that one of the biggest dangers of AI is that we overestimate its abilities, trust it more than we should—even as it’s shown itself to have antisocial tendencies such as “opportunistic blackmail”—and rely on it more than is wise. In so doing, we make ourselves vulnerable to its propensity to fail when it matters most.

“Although you can use AI to generate a lot of ideas, they still require quite a bit of auditing,” says Ortiz. “So for example, if you want to do your taxes, you’d want to stick with something more like TurboTax than ChatGPT.”