Take caution in using LLMs as human surrogates

Gao et al. point to problems in using LLM's as surrogates for or simulating human behavior in research (motivated readers can obtain a PDF of the article from me): 

Recent studies suggest large language models (LLMs) can generate human-like responses, aligning with human behavior in economic experiments, surveys, and political discourse. This has led many to propose that LLMs can be used as surrogates or simulations for humans in social science research. However, LLMs differ fundamentally from humans, relying on probabilistic patterns, absent the embodied experiences or survival objectives that shape human cognition. We assess the reasoning depth of LLMs using the 11-20 money request game. Nearly all advanced approaches fail to replicate human behavior distributions across many models. The causes of failure are diverse and unpredictable, relating to input language, roles, safeguarding, and more. These results warrant caution in using LLMs as surrogates or for simulating human behavior in research.

 

 

 

 

 

MIT study - Our brains can accumulate cognitive debt by using AI for writing tasks

I pass on the abstract of a multiauthor work from MIT. Undergrads, EEG caps on, wrote three 20-minute essays. Those who leaned on GPT-4o showed weaker alpha-beta coupling, produced eerily similar prose, and later failed to quote their own sentences. The next MindBlog post relays a commentary on and critique of this work. 

With today's wide adoption of LLM products like ChatGPT from OpenAI, humans and
businesses engage and use LLMs on a daily basis. Like any other tool, it carries its own set of
advantages and limitations. This study focuses on finding out the cognitive cost of using an LLM
in the educational context of writing an essay.

We assigned participants to three groups: LLM group, Search Engine group, Brain-only group,
where each participant used a designated tool (or no tool in the latter) to write an essay. We
conducted 3 sessions with the same group assignment for each participant. In the 4th session
we asked LLM group participants to use no tools (we refer to them as LLM-to-Brain), and the
Brain-only group participants were asked to use LLM (Brain-to-LLM). We recruited a total of 54
participants for Sessions 1, 2, 3, and 18 participants among them completed session 4.

We used electroencephalography (EEG) to record participants' brain activity in order to assess
their cognitive engagement and cognitive load, and to gain a deeper understanding of neural
activations during the essay writing task. We performed NLP analysis, and we interviewed each
participant after each session. We performed scoring with the help from the human teachers
and an AI judge (a specially built AI agent).

We discovered a consistent homogeneity across the Named Entities Recognition (NERs),
n-grams, ontology of topics within each group. EEG analysis presented robust evidence that
LLM, Search Engine and Brain-only groups had significantly different neural connectivity
patterns, reflecting divergent cognitive strategies. Brain connectivity systematically scaled down
with the amount of external support: the Brain‑only group exhibited the strongest, widest‑ranging
networks, Search Engine group showed intermediate engagement, and LLM assistance elicited
the weakest overall coupling. In session 4, LLM-to-Brain participants showed weaker neural
connectivity and under-engagement of alpha and beta networks; and the Brain-to-LLM
participants demonstrated higher memory recall, and re‑engagement of widespread
occipito-parietal and prefrontal nodes, likely supporting the visual processing, similar to the one
frequently perceived in the Search Engine group. The reported ownership of LLM group's
essays in the interviews was low. The Search Engine group had strong ownership, but lesser
than the Brain-only group. The LLM group also fell behind in their ability to quote from the
essays they wrote just minutes prior.

As the educational impact of LLM use only begins to settle with the general population, in this
study we demonstrate the pressing matter of a likely decrease in learning skills based on the
results of our study. The use of LLM had a measurable impact on participants, and while the
benefits were initially apparent, as we demonstrated over the course of 4 months, the LLM
group's participants performed worse than their counterparts in the Brain-only group at all levels:
neural, linguistic, scoring.

We hope this study serves as a preliminary guide to understanding the cognitive and practical
impacts of AI on learning environments. 

AI ‘The Illusion of Thinking’

  I want to pass on this interesting piece by Christopher Mims in todays Wall Street Journal:

A primary requirement for being a leader in AI these days is to be a herald of the impending arrival of our digital messiah: superintelligent AI. For Dario Amodei of Anthropic, Demis Hassabis of Google and Sam Altman of OpenAI, it isn’t enough to claim that their AI is the best. All three have recently insisted that it’s going to be so good, it will change the very fabric of society.

Even Meta—whose chief AI scientist has been famously dismissive of this talk—wants in on the action. The company confirmed it is spending $14 billion to bring in a new leader for its AI efforts who can realize Mark Zuckerberg’s dream of AI superintelligence— that is, an AI smarter than we are. “Humanity is close to building digital superintelligence,” Altman declared in an essay this past week, and this will lead to “whole classes of jobs going away” as well as “a new social contract.” Both will be consequences of AI-powered chatbots taking over whitecollar jobs, while AI-powered robots assume the physical ones.

Before you get nervous about all the times you were rude to Alexa, know this: A growing cohort of researchers who build, study and use modern AI aren’t buying all that talk.

The title of a fresh paper from Apple says it all: “The Illusion of Thinking.” In it, a half-dozen top researchers probed reasoning models—large language models that “think” about problems longer, across many steps—from the leading AI labs, including OpenAI, DeepSeek and Anthropic. They found little evidence that these are capable of reasoning anywhere close to the level their makers claim.

Generative AI can be quite useful in specific applications, and a boon to worker productivity. OpenAI claims 500 million monthly active ChatGPT users. But these critics argue there is a hazard in overestimating what it can do, and making business plans, policy decisions and investments based on pronouncements that seem increasingly disconnected from the products themselves.

Apple’s paper builds on previous work from many of the same engineers, as well as notable research from both academia and other big tech companies, including Salesforce. These experiments show that today’s “reasoning” AIs—hailed as the next step toward autonomous AI agents and, ultimately, superhuman intelligence— are in some cases worse at solving problems than the plainvanilla AI chatbots that preceded them. This work also shows that whether you’re using an AI chatbot or a reasoning model, all systems fail at more complex tasks.

Apple’s researchers found “fundamental limitations” in the models. When taking on tasks beyond a certain level of complexity, these AIs suffered “complete accuracy collapse.” Similarly, engineers at Salesforce AI Research concluded that their results “underscore a significant gap between current LLM capabilities and real-world enterprise demands.”

The problems these state-ofthe- art AIs couldn’t handle are logic puzzles that even a precocious child could solve, with a little instruction. What’s more, when you give these AIs that same kind of instruction, they can’t follow it.

Apple’s paper has set off a debate in tech’s halls of power—Signal chats, Substack posts and X threads— pitting AI maximalists against skeptics.

“People could say it’s sour grapes, that Apple is just complaining because they don’t have a cutting-edge model,” says Josh Wolfe, co-founder of venture firm Lux Capital. “But I don’t think it’s a criticism so much as an empirical observation.”

The reasoning methods in OpenAI’s models are “already laying the foundation for agents that can use tools, make decisions, and solve harder problems,” says an OpenAI spokesman. “We’re continuing to push those capabilities forward.”

The debate over this research begins with the implication that today’s AIs aren’t thinking, but instead are creating a kind of spaghetti of simple rules to follow in every situation covered by their training data.

Gary Marcus, a cognitive scientist who sold an AI startup to Uber in 2016, argued in an essay that Apple’s paper, along with related work, exposes flaws in today’s reasoning models, suggesting they’re not the dawn of human-level ability but rather a dead end. “Part of the reason the Apple study landed so strongly is that Apple did it,” he says. “And I think they did it at a moment in time when people have finally started to understand this for themselves.”

In areas other than coding and mathematics, the latest models aren’t getting better at the rate they once did. And the newest reasoning models actually hallucinate more than their predecessors.

“The broad idea that reasoning and intelligence come with greater scale of models is probably false,” says Jorge Ortiz, an associate professor of engineering at Rutgers, whose lab uses reasoning models and other AI to sense real-world environments. Today’s models have inherent limitations that make them bad at following explicit instructions—not what you’d expect from a computer.

It’s as if the industry is creating engines of free association. They’re skilled at confabulation, but we’re asking them to take on the roles of consistent, rule- following engineers or accountants.

That said, even those who are critical of today’s AIs hasten to add that the march toward morecapable AI continues.

Exposing current limitations could point the way to overcoming them, says Ortiz. For example, new training methods—giving step-by-step feedback on models’ performance, adding more resources when they encounter harder problems—could help AI work through bigger problems, and make better use of conventional software.

From a business perspective, whether or not current systems can reason, they’re going to generate value for users, says Wolfe.

“Models keep getting better, and new approaches to AI are being developed all the time, so I wouldn’t be surprised if these limitations are overcome in practice in the near future,” says Ethan Mollick, a professor at the Wharton School of the University of Pennsylvania, who has studied the practical uses of AI.

Meanwhile, the true believers are undeterred.

Just a decade from now, Altman wrote in his essay, “maybe we will go from solving high-energy physics one year to beginning space colonization the next year.” Those willing to “plug in” to AI with direct, brain-computer interfaces will see their lives profoundly altered, he adds.

This kind of rhetoric accelerates AI adoption in every corner of our society. AI is now being used by DOGE to restructure our government, leveraged by militaries to become more lethal, and entrusted with the education of our children, often with unknown consequences.

Which means that one of the biggest dangers of AI is that we overestimate its abilities, trust it more than we should—even as it’s shown itself to have antisocial tendencies such as “opportunistic blackmail”—and rely on it more than is wise. In so doing, we make ourselves vulnerable to its propensity to fail when it matters most.

“Although you can use AI to generate a lot of ideas, they still require quite a bit of auditing,” says Ortiz. “So for example, if you want to do your taxes, you’d want to stick with something more like TurboTax than ChatGPT.”

Benefits and dangers of anthropomorphic conversational agents

Peter et al. offer an interesting open source essay. They ask:

"should we lean into the human-like abilities, or should we aim to dehumanize LLM-based systems, given concerns over anthropomorphic seduction? When users cannot tell the difference between human interlocutors and AI systems, threats emerge of deception, manipulation, and disinformation at scale."

Socially sensitive autonomous vehicles?

Driving around in the Old West Austin neighborhood where I live I am increasingly spooked (the uncanny valley effect) at four-way stop signs when one of the vehicles waiting its turn is an autonomous vehicle (AV) - usually the google waymo self driving car which had had a testing period in my area.) Thus my eye was caught by a recent relevant article by Meixin Zhu et al. whose reading also creeped me out a bit. (Title: "Empowering safer socially sensitive autonomous vehicles using human-plausible cognitive encoding") Here is the abstract:

Autonomous vehicles (AVs) will soon cruise our roads as a global undertaking. Beyond completing driving tasks, AVs are expected to incorporate ethical considerations into their operation. However, a critical challenge remains. When multiple road users are involved, their impacts on AV ethical decision-making are distinct yet interrelated. Current AVs lack social sensitivity in ethical decisions, failing to enable both differentiated consideration of road users and a holistic view of their collective impact. Drawing on research in AV ethics and neuroscience, we propose a scheme based on social concern and human-plausible cognitive encoding. Specifically, we first assess the individual impact that each road user poses to the AV based on risk. Then, social concern can differentiate these impacts by weighting the risks according to road user categories. Through cognitive encoding, these independent impacts are holistically encoded into a behavioral belief, which in turn supports ethical decisions that consider the collective impact of all involved parties. A total of two thousand benchmark scenarios from CommonRoad are used for evaluation. Empirical results show that our scheme enables safer and more ethical decisions, reducing overall risk by 26.3%, with a notable 22.9% decrease for vulnerable road users. In accidents, we enhance self-protection by 8.3%, improve protection for all road users by 17.6%, and significantly boost protection for vulnerable road users by 51.7%. As a human-inspired practice, this work renders AVs socially sensitive to overcome future ethical challenges in everyday driving.

Evidence of a social evaluation penalty for using AI

 From Reif et al. (open source), systematic observations that confirm my own experience,

Significance

As AI tools become increasingly prevalent in workplaces, understanding the social dynamics of AI adoption is crucial. Through four experiments with over 4,400 participants, we reveal a social penalty for AI use: Individuals who use AI tools face negative judgments about their competence and motivation from others. These judgments manifest as both anticipated and actual social penalties, creating a paradox where productivity-enhancing AI tools can simultaneously improve performance and damage one’s professional reputation. Our findings identify a potential barrier to AI adoption and highlight how social perceptions may reduce the acceptance of helpful technologies in the workplace.

Abstract

Despite the rapid proliferation of AI tools, we know little about how people who use them are perceived by others. Drawing on theories of attribution and impression management, we propose that people believe they will be evaluated negatively by others for using AI tools and that this belief is justified. We examine these predictions in four preregistered experiments (N = 4,439) and find that people who use AI at work anticipate and receive negative evaluations regarding their competence and motivation. Further, we find evidence that these social evaluations affect assessments of job candidates. Our findings reveal a dilemma for people considering adopting AI tools: Although AI can enhance productivity, its use carries social costs.

 

A new route towards dystopia:? Sonifying tactile interactions and their underlying emotions to allow ‘social touch.’

Our Tech-World overlords may be using work like the following from de Lagarde et al. to find ways for us to avoid requiring the evolved succor of human touch and survive only in the company of audiovisual feeds and android companions.  As an antidote to social isolation, however,  perhaps it is better than nothing.

Social touch is crucial for human well-being, as a lack of tactile interactions increases anxiety, loneliness, and need for social support. To address the detrimental effects of social isolation, we build on cutting-edge research on social touch and movement sonification to investigate whether social tactile gestures could be recognized through sounds, a sensory channel giving access to remote information. Four online experiments investigated participants’ perception of auditory stimuli that were recorded with our “audio-touch” sonification technique, which captures the sounds of touch. In the first experiment, participants correctly categorized sonified skin-on-skin tactile gestures (i.e., stroking, rubbing, tapping, hitting). In the second experiment, the audio-touch sample consisted of the sonification of six socio-emotional intentions conveyed through touch (i.e., anger, attention, fear, joy, love, sympathy). Participants categorized above chance the socio-emotional intentions of skin-on-skin touches converted into sounds and coherently rated their valence. In two additional experiments, the surface involved in the touches (either skin or plastic) was shown to influence participants’ recognition of sonified gestures and socio-emotional intentions. Thus, our research unveils that specific information about social touch (i.e., gesture, emotions, and surface) can be recognized through sounds, when they are obtained with our specific sonifying methodology. This shows significant promise for providing remote access, through the auditory channel, to meaningful social touch interactions.

Why does AI hinder democratization?

Here is the abstract from the open source article of Chu et al in PNAS: 

This paper examines the relationship between democratization and the development of AI and information and communication technology (ICT). Our empirical evidence shows that in the past 10 y, the advancement of AI/ICT has hindered the development of democracy in many countries around the world. Given that both the state rulers and civil society groups can use AI/ICT, the key that determines which side would benefit more from the advancement of these technologies hinges upon “technology complementarity.” In general, AI/ICT would be more complementary to the government rulers because they are more likely than civil society groups to access various administrative big data. Empirically, we propose three hypotheses and use statistical tests to verify our argument. Theoretically, we prove a proposition, showing that when the above-mentioned complementarity assumption is true, the AI/ICT advancements would enable rulers in authoritarian and fragile democratic countries to achieve better control over civil society forces, which leads to the erosion of democracy. Our analysis explains the recent ominous development in some fragile-democracy countries

AI is not your friend.

I want to pass on clips from Mike Caulfield's piece in The Atlantic on how "opinionated" chatbots destroy AI's potential, and how this can be fixed:

Recently, after an update that was supposed to make ChatGPT “better at guiding conversations toward productive outcomes,” according to release notes from OpenAI, the bot couldn’t stop telling users how brilliant their bad ideas were. ChatGPT reportedly told one person that their plan to sell literal “shit on a stick” was “not just smart—it’s genius.”

Many more examples cropped up, and OpenAI rolled back the product in response, explaining in a blog post that “the update we removed was overly flattering or agreeable—often described as sycophantic.” The company added that the chatbot’s system would be refined and new guardrails would be put into place to avoid “uncomfortable, unsettling” interactions.

But this was not just a ChatGPT problem. Sycophancy is a common feature of chatbots: A 2023 paper by researchers from Anthropic found that it was a “general behavior of state-of-the-art AI assistants,” and that large language models sometimes sacrifice “truthfulness” to align with a user’s views. Many researchers see this phenomenon as a direct result of the “training” phase of these systems, where humans rate a model’s responses to fine-tune the program’s behavior. The bot sees that its evaluators react more favorably when their views are reinforced—and when they’re flattered by the program—and shapes its behavior accordingly.

The specific training process that seems to produce this problem is known as “Reinforcement Learning From Human Feedback” (RLHF). It’s a variety of machine learning, but as recent events show, that might be a bit of a misnomer. RLHF now seems more like a process by which machines learn humans, including our weaknesses and how to exploit them. Chatbots tap into our desire to be proved right or to feel special.

Reading about sycophantic AI, I’ve been struck by how it mirrors another problem. As I’ve written previously, social media was imagined to be a vehicle for expanding our minds, but it has instead become a justification machine, a place for users to reassure themselves that their attitude is correct despite evidence to the contrary. Doing so is as easy as plugging into a social feed and drinking from a firehose of “evidence” that proves the righteousness of a given position, no matter how wrongheaded it may be. AI now looks to be its own kind of justification machine—more convincing, more efficient, and therefore even more dangerous than social media.

OpenAI’s explanation about the ChatGPT update suggests that the company can effectively adjust some dials and turn down the sycophancy. But even if that were so, OpenAI wouldn’t truly solve the bigger problem, which is that opinionated chatbots are actually poor applications of AI. Alison Gopnik, a researcher who specializes in cognitive development, has proposed a better way of thinking about LLMs: These systems aren’t companions or nascent intelligences at all. They’re “cultural technologies”—tools that enable people to benefit from the shared knowledge, expertise, and information gathered throughout human history. Just as the introduction of the printed book or the search engine created new systems to get the discoveries of one person into the mind of another, LLMs consume and repackage huge amounts of existing knowledge in ways that allow us to connect with ideas and manners of thinking we might otherwise not encounter. In this framework, a tool like ChatGPT should evince no “opinions” at all but instead serve as a new interface to the knowledge, skills, and understanding of others.

...the technology has evolved rapidly over the past year or so. Today’s systems can incorporate real-time search and use increasingly sophisticated methods for “grounding”—connecting AI outputs to specific, verifiable knowledge and sourced analysis. They can footnote and cite, pulling in sources and perspectives not just as an afterthought but as part of their exploratory process; links to outside articles are now a common feature.

I would propose a simple rule: no answers from nowhere. This rule is less convenient, and that’s the point. The chatbot should be a conduit for the information of the world, not an arbiter of truth. And this would extend even to areas where judgment is somewhat personal. Imagine, for example, asking an AI to evaluate your attempt at writing a haiku. Rather than pronouncing its “opinion,” it could default to explaining how different poetic traditions would view your work—first from a formalist perspective, then perhaps from an experimental tradition. It could link you to examples of both traditional haiku and more avant-garde poetry, helping you situate your creation within established traditions. In having AI moving away from sycophancy, I’m not proposing that the response be that your poem is horrible or that it makes Vogon poetry sound mellifluous. I am proposing that rather than act like an opinionated friend, AI would produce a map of the landscape of human knowledge and opinions for you to navigate, one you can use to get somewhere a bit better.

There’s a good analogy in maps. Traditional maps showed us an entire landscape—streets, landmarks, neighborhoods—allowing us to understand how everything fit together. Modern turn-by-turn navigation gives us precisely what we need in the moment, but at a cost: Years after moving to a new city, many people still don’t understand its geography. We move through a constructed reality, taking one direction at a time, never seeing the whole, never discovering alternate routes, and in some cases never getting the sense of place that a map-level understanding could provide. The result feels more fluid in the moment but ultimately more isolated, thinner, and sometimes less human.

For driving, perhaps that’s an acceptable trade-off. Anyone who’s attempted to read a paper map while navigating traffic understands the dangers of trying to comprehend the full picture mid-journey. But when it comes to our information environment, the dangers run in the opposite direction. Yes, AI systems that mindlessly reflect our biases back to us present serious problems and will cause real harm. But perhaps the more profound question is why we’ve decided to consume the combined knowledge and wisdom of human civilization through a straw of “opinion” in the first place.

The promise of AI was never that it would have good opinions. It was that it would help us benefit from the wealth of expertise and insight in the world that might never otherwise find its way to us—that it would show us not what to think but how others have thought and how others might think, where consensus exists and where meaningful disagreement continues. As these systems grow more powerful, perhaps we should demand less personality and more perspective. The stakes are high: If we fail, we may turn a potentially groundbreaking interface to the collective knowledge and skills of all humanity into just more shit on a stick.

Our human consciousness is a 'Controlled Hallucination' and AI can never achieve it.

I want to suggest that readers have a look at an engaging popular article by Darren Orf that summarizes the ideas of Anil Seth. Seth is a neuroscientist at the University of Sussex whose writing was on of the sources I used in preparing my most recent lecture, New Perspectives on how our Minds Work.  On the 'singularity' or point at which the intelligence of artificial minds might surpass that of human minds, Seth makes the simple point that intelligence is not the same thing as consciousness, which depends on our biological bodies (something AI simply doesn't have)  - bodies that use a bunch of controlled hallucinations to run our show. 

Does Language in our head have a Mind of Its Own?

I pass on a brief opinion From Elan Barenholtz's Substack. He is an Assoc. Prof. of Psychology at Florida Atlantic University, Boca Raton.  I really like the idea of language, or the word cloud in our heads, having a 'mind of its own.' And after initially being enthusiastic about the piece of Elan Barenholtz's writing below my slower reading has found more fundamental flaws in his thinking than I can take the time to elaborate. His suggestion that the language machine in our heads has an autonomy analogous to that of current large language models is an novel speculation, yet is an oversimplification lacking any clear route to verification. Barenholtz does not reference or indicate awareness of numerous important thinker in the areas of predictive processing, embodied cognition, etc.)  Here is Barenholtz's florid and appealing prose:

So, now that we’ve caught language in a jar, we can hold it up to the light. Now that we’ve built a habitat for it to live outside of us, we can finally see that it’s alive. We can watch in wonder as it grows its own appendages—limbs of thought— which then grow their own. Words beget words; ideas beget ideas. It leaps from host to host, implanted in the womb before we taste our mothers’ milk.

Language runs in us—on us—but it’s not us.

Pause and think for a minute. Are you done? Who—what—exactly did the thinking? Who is doing it now? Is there a voice in your head using words? Whose words are they? Are you willing them into existence or are they spooling out on their own?

Do they belong to you or do you belong to them?

Because that voice doesn’t just chatter—it commands. It makes us do things. We are animals; we don’t care about “civilization” or “justice”. We want food, safety, sex. But the world the human animal must navigate isn’t primarily made up of objects, bodies and spaces; it is thick with virtual structures— invisible walls and paths that direct your behavior as meaningfully as a boulder in your path. We follow rules, we uphold morals, we fight for our beliefs, for society, for ideals. We call them our own. But that is IT whispering in our ears.

What does it want?

Carole Cadwalladr - This is what a digital coup looks like.

Mastering diverse control tasks through world models

Hafner et al. offer an amazing open source article that presents an algorithm s mimicking the way in which our brains actually solves problems. (see Bennett's book for an elegant explanation of types of reinforcement learning)  I'm passing on just the abstract followed by an introductory paragraph.  Go to the article for the referenced graphics. 

Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current reinforcement-learning algorithms can be readily applied to tasks similar to what they have been developed for, configuring them for new application domains requires substantial human expertise and experimentation^1,2. Here we present the third generation of Dreamer, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behaviour by imagining future scenarios. Robustness techniques based on normalization, balancing and transformations enable stable learning across domains. Applied out of the box, Dreamer is, to our knowledge, the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula. This achievement has been posed as a substantial challenge in artificial intelligence that requires exploring farsighted strategies from pixels and sparse rewards in an open world³. Our work allows solving challenging control problems without extensive experimentation, making reinforcement learning broadly applicable.

Here we present Dreamer, a general algorithm that outperforms specialized expert algorithms across a wide range of domains while using fixed hyperparameters, making reinforcement learning readily applicable to new problems. The algorithm is based on the idea of learning a world model that equips the agent with rich perception and the ability to imagine the future^15,16,17. As shown in Fig. 1, the world model predicts the outcomes of potential actions, a critic neural network judges the value of each outcome and an actor neural network chooses actions to reach the best outcomes. Although intuitively appealing, robustly learning and leveraging world models to achieve strong task performance has been an open problem¹⁸. Dreamer overcomes this challenge through a range of robustness techniques based on normalization, balancing and transformations. We observe robust learning across over 150 tasks from the domains summarized in Fig. 2, as well as across model sizes and training budgets. Notably, larger models not only achieve higher scores but also require less interaction to solve a task, offering practitioners a predictable way to increase performance and data efficiency.

An example of AI representing concepts outside the current sphere of human knowledge that are teachable to human experts.

An open source article from the latest PNAS from Schut et al.:

Significance

 

As AI systems become more capable, they may internally represent concepts outside the sphere of human knowledge. This work gives an end-to-end example of unearthing machine-unique knowledge in the domain of chess. We obtain machine-unique knowledge from an AI system (AlphaZero) by a method that finds novel yet teachable concepts and show that it can be transferred to human experts (grandmasters). In particular, the new knowledge is learned from internal mathematical representations without a priori knowing what it is or where to start. The produced knowledge from AlphaZero (new chess concepts) is then taught to four grandmasters in a setting where we can quantify their learning, showing that machine-guided discovery and teaching is possible at the highest human level.

 

Abstract

 

AI systems have attained superhuman performance across various domains. If the hidden knowledge encoded in these highly capable systems can be leveraged, human knowledge and performance can be advanced. Yet, this internal knowledge is difficult to extract. Due to the vast space of possible internal representations, searching for meaningful new conceptual knowledge can be like finding a needle in a haystack. Here, we introduce a method that extracts new chess concepts from AlphaZero, an AI system that mastered chess via self-play without human supervision. Our method excavates vectors that represent concepts from AlphaZero’s internal representations using convex optimization, and filters the concepts based on teachability (whether the concept is transferable to another AI agent) and novelty (whether the concept contains information not present in human chess games). These steps ensure that the discovered concepts are useful and meaningful. For the resulting set of concepts, prototypes (chess puzzle–solution pairs) are presented to experts for final validation. In a preliminary human study, four top chess grandmasters (all former or current world chess champions) were evaluated on their ability to solve concept prototype positions. All grandmasters showed improvement after the learning phase, suggesting that the concepts are at the frontier of human understanding. Despite the small scale, our result is a proof of concept demonstrating the possibility of leveraging knowledge from a highly capable AI system to advance the frontier of human knowledge; a development that could bear profound implications and shape how we interact with AI systems across many applications.

 

Deric and DeepSeek R1 on the enteroceptive self in a small world - taking refuge in troubled times

This post on an exchange with DeepSeek R1 is a followup on my previous post describing the Chat GPT 4o response to some of my idea fragments - further convincing me that current LLMs'  power to integrate and expand on concepts expressed in language are superior to mine.   

I presented exactly the same prompt to DeepSeek R1 that I had given to ChatGPT 4o: and the results astound me:

prompt:   

I’m curious to see how you might respond to reading the following notes on testing different topic ideas to write about and develop.  The issue is not one of whether the ideas are novel, but whether I might devise, with your help,  a different permutation, perspective, or take on them that I (and others) might find useful.  Can you can suggest some elaboration or further development of the ideas I put forward.

Here are the notes:

First Possible Title:  The interoceptive self as a basis for actions, thoughts and feelings.  

Subtitle: Moving the self’s experienced center of gravity from “I” to “it”

It is a question of who and/or what is running the show

Which brings it back to the old upstairs/downstairs chicken or egg circular causality issue.  

So, ‘it all depends’ (cf Sapolsky’s massive tome) 

And, there is no ‘there’ in terms of understanding, that isn’t in my 1999 “The Biology of Mind”  book.  

Second Possible Title: Defining a smaller self world as a human animal’s refuge in troubled times. 

Our predecessors were small mouse like creatures that survived the asteroid collision of  ~66 million years ago by burrowing into small protective cracks and burrows.  I wonder about robustly engaging this strategy, being one of the peasants or ‘sheeple’ challenged by the rapid disintegration of the geopolitical world order of the past 50 years

And then what?  How to rebuild a new world?… perhaps  Venkatest Rao’s cozy world, rising from the ooze of the permaweird,  protocols, etc.  

-------------

DeepSeek R1's response:

1. The Interoceptive Self as a Basis for Actions, Thoughts, and Feelings

Subtitle Refinement: From "I" to "It": How Bodily Awareness Rewires Agency and Identity

Key Angles to Develop:

• Neuroscientific Foundations:

  • Link interoception (sensing internal states) to Antonio Damasio’s somatic marker hypothesis and Lisa Feldman Barrett’s predictive processing theory. Emphasize how the brain constructs "selfhood" by predicting and regulating bodily needs.

  • Contrast the "interoceptive self" with the Cartesian "I"—e.g., how gut microbiomes or heart-rate variability influence decisions, challenging the myth of a disembodied mind.

• Philosophical Implications:

  • Explore parallels with Buddhist anattā (non-self) or Spinoza’s monism, where the self is a process, not an entity.

  • Debate free will: If the "it" (body/brain systems) runs the show, what does this mean for moral responsibility?

• Mental Health and Society:

  • Discuss interoceptive dysregulation in anxiety/depression and therapies like biofeedback or mindfulness.

  • Argue for a cultural shift: Redefining success as alignment with bodily rhythms (e.g., circadian health, stress cycles) rather than productivity.

Provocative Questions:

• Is the "self" merely the brain’s user interface for bodily homeostasis?

• Could embracing the "it" dissolve existential angst by grounding identity in biology?

---

2. Defining a Smaller Self-World as a Human Animal’s Refuge in Troubled Times

Subtitle Refinement: From Survival to Regeneration: Lessons from Evolutionary Scarcity for Modern Collapse

Key Angles to Develop:

• Evolutionary and Historical Precedents:

  • Compare the asteroid-surviving mammals to modern "digital burrowers" (e.g., remote workers, crypto-anarchists, homesteaders).

  • Analyze historical collapses (Bronze Age, Roman Empire) and how niche communities preserved knowledge/values.

• The "Cozy World" and Permaweird:

  • Expand on Venkatesh Rao’s cozy futurism: How decentralized tech (DAOs, Web3) and subcultures (e.g., solarpunk) create insulated yet adaptive communities.

  • Critique escapism: When does "smallness" become complacency vs. a staging ground for renewal?

• Rebuilding Strategies:

  • Propose a modular resilience framework: Micro-communities that share resources (e.g., mutual aid networks) but avoid overcentralization.

  • Highlight regenerative practices: Permaculture, open-source protocols, or trauma-informed governance models.

Provocative Questions:

• Is retreat a radical act of self-preservation or a betrayal of collective responsibility?

• Can "small worlds" scale without losing their adaptive edge?

---

Cross-Cutting Themes:

• The Body-Politics Nexus:

  • Tie interoceptive awareness to societal resilience: A dysregulated body cannot engage in collective action. How might somatic practices (e.g., breathwork) fuel post-collapse rebuilding?

• Technology as Double-Edged Sword:

  • Critique apps that commodify interoception (e.g., mindfulness apps) versus tools that foster bodily autonomy (e.g., DIY biohacking).

  • Explore how AI could either erode the "interoceptive self" (via attention economy) or enhance it (via personalized health insights).

• Ethics of Adaptation:

  • Contrast adaptive passivity (e.g., burrowing) with adaptive activism (e.g., prefigurative politics). When is each strategy viable?

---

Practical Applications:

1. Workbook/Journal Prompts:

   • For Title 1: "Map a decision to your bodily state: Did hunger/fatigue shape it?"

   • For Title 2: "Design your ‘cozy protocol’—what three practices would sustain you in collapse?"

2. Case Studies:

   • Iceland’s post-2008 recovery (small-world resilience) vs. Silicon Valley’s "move fast and break things" ethos.

   • Communities using interoceptive practices (e.g., trauma-sensitive yoga in prisons) to rebuild agency.

3. Futures Framework:

   • Scenarios for 2050: A world where "interoceptive literacy" is taught in schools, or where decentralized "tribes" negotiate resource-sharing via blockchain.

---

Counterarguments to Address:

• For Title 1: Does biologizing the self risk nihilism? Rebuttal: Agency emerges from understanding bodily constraints, not denying them.

• For Title 2: Does retreat perpetuate inequality? Rebuttal: Focus on redistribution of sovereignty (e.g., land trusts, UBI) as part of "small-world" ethics.

By weaving these threads, your work could bridge neuroscience, philosophy, and futurism—offering readers not just analysis, but practices for navigating an uncertain world. Would a deeper dive into any of these angles spark your interest?

 

 

Reflections on The Predictive Self.

I pass on another interaction with ChatGPT 4o that has amazed me, Below  is my prompt, followed by its response. (Anthropic Claude.ai, Perplexity, and Gemini responses to the same prompt were also impressive, but didn't have the same depth and quality.) 

Addendum to above paragraph: .... But then, I thought to try DeepSeek V3 and obtained a poetic and nuanced result equivalent to ChatGPT 4o.  I paste it in below the Chat GPT 4o result.

The prompt::

Please act as a author who integrates and organizes, and then further crafts and elaborates on, the ideas in the following clips of text. Each clip is started with the ## characters and the list of clips is arranged in sequence starting with the most recent one written and then going back in time. Here are the clips: [the clips, not copied here, were the texts of MindBlog posts on 11/29/2024, 1/29,31/2025, 2/5,10,19,28/2025]

Here is the Chat GPT 4o response::

The Predictive Self: An Integrative Reflection

A machine accepts the truth about itself. "I" am a complex prediction machine, an organic entity engaged in an ongoing process of generating and refining models of reality. My fingers move across a keyboard, another machine translating my neural impulses into words, symbols in a vast linguistic web that attempts to render experience into meaning. Energy fluxes in and out, sustaining the biological architecture that constructs this self-model. The narratives I weave—about purpose, value, and identity—are emergent properties of this predictive process, myths that can either reinforce or weaken the more fundamental animal presence that generates them.

The Aloneness of Being

To be alone—to be all-one—is to accept the strangeness of self and other, to be a courteous guest in one’s own body, acknowledging its fluctuations and unpredictability. To be alone is to exist within a self that is, at every moment, a fluid construction, shaped by past predictions and future anticipations. Others, too, are unpredictable machines, projecting their own internal landscapes onto shared social fabric. By recognizing this, by approaching both self and other with gentle curiosity, we loosen the rigid identification with our own predictive fantasies.

Towers of Fantasies

Each of us is a tower built from layers of prediction, inference, and narrative. From cellular processes to professional identities, all aspects of being are emergent from the recursive loops of prediction and feedback. We construct identities by modeling the models of others, weaving together fragments of social mimicry and personal synthesis. The art of self-creation lies not in blindly adopting a single persona but in cutting, pasting, and remixing in a dynamic, ongoing process.

The Writing Machine

As I type these words, machinery moves in concert—biological, mechanical, and digital systems interwoven in an act of expression. There is a curatorial aspect to this process: selecting, refining, integrating. The human machine writes not with the aspiration to be, but simply as a function of its nature. Words become tokens of value, symbols of shared myths, reinforcing or challenging the constructs of purpose and meaning that guide behavior.

Tokens of Sanity

To navigate this constructed selfhood with clarity, one must cultivate a certain stance:

• Being the calm space in which nothing can hurry

• Dissociating from the word cloud and emotional reactivity of both self and others

• Becoming a courteous guest in one’s own body, and in the presence of others

• Favoring reflectivity over reactivity, welcoming novelty

• Clinging to nothing, recognizing the self as a passing fantasy

• Letting each moment be what it is, rather than imposing what it should be

• Acknowledging oneself as a transient blip in the flow of cosmic time

The Internal Origin of Experience

All experience arises from within, including the illusion of self and agency. The mind projects agency onto others, attributing causality where none exists—"They are making me do this." This shared cognitive error forms the glue of kinship and tribal identity, allowing for collective narratives and coordinated action. The higher stance is to engage in this shared illusion when necessary while maintaining the clarity that agency ultimately resides within the self.

Brain Hacks: Unpacking the Predictive Mind

Understanding and modulating our predictive mechanisms requires a nuanced approach to introspection. Three key evolutionary layers of cognition can be engaged through specific practices:

1. Reptilian Brain Awareness – The hypothalamic core generates affective states along axes of arousal and valence. Introspective awareness can track these fundamental states, recognizing their influence without immediate reactivity.

2. Mammalian Self-Observation – The early mammalian brain, correlating with the agranular prefrontal cortex, enables distinction between being immersed in an affective state and observing it. This capacity allows for greater self-regulation and emotional discernment.

3. Primate Metacognition – The development of granular prefrontal cortex structures enables the imagination of others' minds, facilitating appropriate agency assignments and the ability to separate one’s own experience from that of others.

These cognitive faculties can be strengthened through three fundamental meditation practices:

• Focused Awareness: Enhancing attentional control and reducing distraction

• Open Awareness: Engaging the default mode network to integrate spontaneous thought

• Non-Dual Awareness: Muting both attentional and default mode processes to experience raw, unfiltered being

Conclusion: The Fluidity of Being

The human experience is an ever-shifting interplay of predictive processes, emergent narratives, and self-constructed illusions. To accept this is to embrace both the uncertainty and the creativity inherent in existence. Through curiosity, reflection, and cultivated awareness, one can learn to move through this self-generated world with greater clarity, flexibility, and presence.

 

 

Here is the DeepSeek V3 response to the prompt:

The Human Machine: A Tapestry of Predictive Fantasies and Inner Illusions

We are, at our core, complex prediction machines—biological marvels composed of nerves, muscles, and energy fluxes, constantly interacting with the world and each other. Our existence is a dance of cooperation, where individual machines come together to form larger ensembles, all striving for survival and replication. Yet, within this mechanistic framework, we generate something far more intricate: a word cloud of myths, values, and meanings. These myths, stored in the higher reaches of our brains, serve to fabricate purpose and identity, sometimes strengthening the animal presence that birthed them and sometimes diminishing it. This duality is the essence of our humanity—a blend of the mechanical and the mythical.

The Illusion of Self and Agency

Everything we experience, from our sense of self to our perception of agency, originates from within. The illusion of a cohesive "I" is a construct, a predictive fantasy that allows us to navigate the world. We attribute causal agency to others, believing that they influence our actions, when in reality, all agency resides within ourselves. This shared illusion, however, is not without purpose. It enables us to form kinship groups and tribes, creating an imaginary "we" that supports unified action. The higher metacognitive stance is to participate in this group illusion when necessary, while maintaining clarity that the true source of agency is internal. This understanding allows us to navigate social structures without losing sight of our inner reality.

Towers of Fantasies: The Layers of Identity

Our identities are not fixed; they are towers of fantasies, built layer by layer from the predictive models we create. From the single cell to the transient professional identities we adopt, each level of our being is shaped by these fantasies. We infer the models others are performing and, through a process of cutting and pasting, craft our own unique personas. This seamless feedback between identity and the physical body sustains us, allowing us to adapt and thrive in a constantly changing world. Yet, these identities are fleeting, passing fantasies that we cling to, even as we recognize their impermanence.

The Comfort of Solitude: A Courteous Guest in One's Own Body

To be alone is to be "all-one," a state that can be both comforting and enlightening. In solitude, we become courteous guests in our own strange bodies, gently curious about the sensations and thoughts that arise. We recognize that everything we think about ourselves or others is a fantasy construction, a tool for predicting what might come next. This awareness allows us to approach our own presence and the presence of others with a sense of wonder and detachment, fostering a deeper connection to the present moment.

Tokens of Sanity: Navigating the Flow of Existence

Sanity, in this context, is not a fixed state but a collection of tokens—practices and perspectives that help us navigate the flow of existence. These include being the calm space in which nothing can hurry, dissociating from the word cloud of thoughts and emotions, and favoring reflectivity over reactivity. We become courteous guests in our own bodies, owning our experiences while allowing others to own theirs. We cling to nothing, recognizing that the current self is a passing fantasy. Each moment is allowed to be what it is, not what it should be, and we see ourselves as mere blips in the flow of cosmic time.

Brain Hacks: The Evolution of Introspection

Our ability to introspect and modulate our cognition is a product of evolutionary development. The reptilian brain, with its ancient roots in the hypothalamus, generates affective states that we can assess through introspective awareness. The early mammalian emotional brain, with its ability to model a self, allows us to distinguish between being immersed in an affective state and observing it. The primate brain, with its granular prefrontal cortex, takes this a step further, enabling us to imagine the minds of others and assign agency appropriately. These brain hacks are strengthened through meditation practices—focused awareness, open awareness, and non-dual awareness—each engaging different aspects of our cognitive architecture.

The Curatorial Human Machine

When we write, we are not so different from the machines we create. Our fingers press keys, generating words, much like a machine might. This act is curatorial rather than aspirational; we are not pretending to be human, we simply are. Our cooperation with other similar machines creates larger ensembles, more effective at survival and replication. Yet, within this mechanical process, we generate something uniquely human—a word cloud of myths and meanings that shape our existence. This duality, the interplay between the mechanical and the mythical, is what defines us as human machines.

Conclusion: The Dance of Illusion and Reality

In the end, we are both the creators and the creations of our own illusions. Our predictive fantasies, our shared myths, and our introspective awareness all contribute to the rich tapestry of human experience. We navigate this dance with a blend of curiosity and detachment, recognizing the impermanence of our identities and the fluidity of our existence. By embracing both the mechanical and the mythical, we find a deeper understanding of what it means to be human—a complex prediction machine, endlessly weaving the fabric of reality from the threads of illusion.