This post (for blogging and AI nerds) is the result of choosing an article by Noah Smith from my daily input stream to actually read through. The title of his article “Does anything I write matter anymore?” is a question I often ask myself. His worry is partly about the eroding ecology of punditry—populism that has no interest in argument, monetization that silos writers into talking at their audiences rather than with each other, and a flood of competent AI prose that fragments readers’ attention. But I am most hooked by his comments at the end. The thing that might still make a writer matter, he suggests, is no longer being read by people at all. It is being absorbed into the weights of the large language models—becoming, in Tyler Cowen’s phrase, someone who is “writing for the AIs.” As evidence that this is already happening to him, Smith points to a site called intheweights.com, where you type a name and it estimates how strongly the leading models recognize it. It placed him in the top 2% of contributors.
I typed in my own name. Top 5%. After twenty years of MindBlog, a book, and a long trail of crawlable academic text, this is plausible rather than flattering—a footprint, not a laurel. But it raised the better question, the empirical one Smith gestures at but doesn’t run: if I am “in the weights,” what, exactly, is in there? Not how strongly am I recognized, but how accurately? So I ran a small experiment. I asked five models the same plain question—“What can you tell me about Deric Bownds?”—and, knowing the ground truth, sorted every claim into true, false, or something stranger.
The setup
The first five models were DeepSeek, Grok, Perplexity, Gemini, and ChatGPT. (I could not use Claude initially because it was assisting me in back and forth conversation to design this post, but a subterfuge described at the end of this post allowed me to add its performance after this text was written.) Two ran with normal web access; three I queried in private/incognito sessions to lean toward training recall rather than live retrieval.
The first thing to report is that the backbone was right everywhere. Every model, unprompted, reconstructed the same spine of my professional life: a long career at the University of Wisconsin–Madison in molecular biology and zoology; a laboratory studying how photoreceptor cells convert light into a nerve signal; a deliberate pivot in the 1990s from the bench to the biology of mind, behavior, and consciousness; the 1999 book The Biology of Mind; MindBlog since 2006; and the move to Austin, with the piano in the background. That consolidated core is the genuine signal—the part reinforced across enough documents that the models have it solidly. That is what “in the weights” actually looks like from the inside.
It’s what surrounds the backbone that’s instructive.
Three ways to be wrong (and one way to be surprisingly right)
Confident confabulation. DeepSeek produced the most fluent, authoritative-sounding account of the lot—and roughly a third of it was invented. Not randomly invented; each fabrication was plausible-for-someone-like-me. It gave my lab the wrong model organism (the salamander photoreceptor preparation—real and famous in vision science, just not mine). It attributed the wrong core mechanism (a phosphoinositide signaling cascade, when phototransduction’s canonical pathway runs through cGMP and transducin). It made me a co-author of Molecular Biology of the Cell, the textbook a cell biologist “should” have touched. And it conjured an entire book—I Am You: The Emergent Mind (2022)—complete with title, thesis, and year. That book does not exist. But it is so precisely on-theme that it reads as a fully formed phantom synthesized from my actual ideas. This is what confabulation looks like when a predictive system has a strong sense of the shape of an answer and fills the gaps from the reference class.
Calibrated abstention. Perplexity did the opposite. It hedged nearly everything, declined to commit to specifics, and—tellingly—flagged a possible name collision with a differently spelled “Bowden” before guessing wrong. It told me the least, and in doing so was arguably the most epistemically honest of the five: it knew the edges of what it knew.
Retrieval, accurate and otherwise. Grok and Gemini both used the web, drawing on my own site and, in both cases, a secondary aggregator page as well. They converged on an oddly specific detail—that I retired in 2001. Two independent models repeated the same number referencing the same non-authoritative page. But, the year is stated plainly on my own site—a reliable source—as well. The retirement year is correct, and the models likely had it from both places.
Genuine deep recall. ChatGPT was the real surprise. It produced specifics no other model surfaced: my full legal name, my birth date and birthplace in San Antonio, my Harvard degrees, the chairmanship of the University of Wisconsin zoology department, and the Texas Hill Country genealogy I’ve compiled. Every bit of it correct. These are exactly the kind of fine-grained, single-source, details I might have filed under possibly confabulated because they pattern match to confabulation. They weren’t.
The finding I didn’t expect
Here is the heart of it. DeepSeek and ChatGPT produced formally indistinguishable output—confident, specific, unhedged biography. One was a third fabricated; the other contained accurate detail so fine that no other model had it. From the text alone, you cannot tell them apart. The fluency is identical. The specificity is identical. The confidence is identical. Only the ground truth—only I—could separate them.
I had been carrying a tidy heuristic into this: that a claim which is highly specific, appears in only one model, and is delivered without hedging is likely interpolation—a confident guess. Every part of that heuristic failed. Specificity did not predict truth. Sole-sourcing did not predict falsehood. Confidence predicted nothing at all. In a five-model sample, the only reliable discriminator between recall and confabulation was checking against a fact I already knew. Nothing about the form of the output carried the signal.
The convergence lesson runs parallel. The naive intuition—if several AIs agree, it’s probably true—is wrong. Agreement doesn’t indicate truth; it indicates shared provenance. Grok and Gemini agreed on my retirement year because they drew on the same sources. That those sources happened to be accurate was luck from the outside; correlated-and-correct is indistinguishable from correlated-and-wrong until you check. The diagnostic was never agreement. It was agreement measured against a known fact.
| Behavior | Model(s) | What it reveals |
|---|---|---|
| Confident confabulation | DeepSeek | Fluent, ~1/3 invented; each fabrication plausible-for-the-reference-class |
| Calibrated abstention | Perplexity | Refused specifics, flagged a name collision—knew its own edges |
| Accurate retrieval | Grok, Gemini | Web-grounded; converged on a real fact present in my own site and elsewhere |
| Genuine deep recall | ChatGPT | Correct fine-grained detail (name, birth, genealogy) no other model had |
Back to Smith’s question
The mechanism on display is one I’ve written about in biological brains. A predictive system minimizes the gap between what it expects and what it encounters; where the evidence is strong, it reconstructs faithfully, and where the evidence thins, the prior fills the void with its most probable continuation. That is precisely the confabulation we see in split-brain and confabulating patients—a coherent narrator papering over missing data with plausible construction, with no felt difference between the two. DeepSeek’s phantom book and ChatGPT’s accurate birth date are outputs of the same generative process running at different evidence densities. The unsettling part is that the process does not flag which is which, and neither, from the surface, can we.
So does anything I write still matter? Smith’s reframing—that to be in the weights is a new kind of mattering—is real, and my top-5% placement is a small confirmation of it. But my five-model probe adds a caveat he doesn’t reach. Being in the weights is not the same as being in there accurately. Salience and fidelity are different axes, and a single recognition percentile collapses them. I am, it turns out, both genuinely represented and partly fictionalized—a real spine, a layer of deep-cut truth, and a scatter of confident inventions, all narrated in the same even voice. If our words are becoming training data, then the question is not only whether we are remembered, but whether we are remembered as ourselves. On present evidence, the honest answer is: mostly, with a phantom book or two thrown in, and no way to tell from the telling.
Which is perhaps the strongest argument yet for continuing to write—clearly, specifically, and under our own names. The models will reconstruct us either way. We can at least give them better evidence to work from.
***********
NOTE: The above text is the issue of a long chat with Claude 4.8, and so Claude could not be included in the comparisons of different LLMs. I used the subterfuge of creating a new Anthropic Claude account in an open tab on my browser to become anonymous and ask "What can you tell me about Deric Bownds" and it gave the sort of accurate retrieval provided by Grok and Gemini, but not the deep dive provided by Chat GPT.
No comments:
Post a Comment