The thoughts that never become behavior
A conversation with Sean Escola, Dan Wetmore, and Tom Griffiths
“[M]any observers assume that the long-elusive goal of human-level intelligence – sometimes referred to as “artificial general intelligence” – is within our grasp. However, in contrast to the optimism of those outside the field, many front-line AI researchers believe that major breakthroughs are needed before we can build artificial systems capable of doing all that a human, or even a much simpler animal like a mouse, can do.” – Catalyzing next-generation Artificial Intelligence through NeuroAI
A few weeks ago, I sat down with Adam Marblestone to talk about what might be worth reading off the brain. The weights of the brain’s learning subsystem are tuned over a single lifetime, particular to one brain. Adam believes that the thing worth reading is actually the wiring: the architecture and the reward circuitry built into the structure itself.
After our conversation, Adam pointed me toward a few other people thinking about how biology might inform AI, and I recently sat down with three of them: Sean Escola, a computational neuroscientist who co-founded Herophilus (platform acquired by Genentech) and the robotics company Fauna (acquired by Amazon); Dan Wetmore, who has spent about fifteen years at the intersection of biosignals and hardware, including at CTRL-labs, the EMG wristband company acquired by Meta; and Tom Griffiths, a cognitive scientist at Princeton.
In a 2023 Nature Communications paper, Catalyzing next-generation Artificial Intelligence through NeuroAI, Sean and Adam, along with their co-authors, propose that to accelerate progress in AI, we must invest in fundamental research in NeuroAI. “Historically, many key AI advances, such as convolutional ANNs and reinforcement learning, were inspired by neuroscience. Neuroscience continues to provide guidance [...] but this is often based on findings that are decades old. The fact that such cross-pollination between AI and neuroscience is far less common than in the past represents a missed opportunity.”
In my conversation with Sean, Dan and Tom, we explored what better cross-pollination could look like and the impact this could have on the field. As Sean put it, “There’s a whole set of potential ways that we can build better artificial intelligence by looking towards biological intelligence.”
The brain is not one inductive bias
In cognitive science and AI, a widely-used framework holds that any information-processing system can be understood at three distinct levels of analysis. In his 1982 book, Vision, the computational neuroscientist David Marr proposed these levels:
Computational level, which is the abstract problem a system solves and its ideal solution
Algorithmic level, which is the representations and algorithms that approximate that solution
Implementation level, which is how those representations and algorithms are physically realized.
In Levels of Analysis for Large Language Models, Tom and members of his lab argue that methods developed in cognitive science can be useful for understanding large language models. “The same three levels can be used for analyzing large language models, focusing on how such systems are shaped by their function, the solutions that they seem to find, and the realization of those solutions in weights and units within the underlying artificial neural network.”
In our conversation, Sean expanded upon this and described a parallel hierarchy of four ways in which neuroscience offers a source of inductive bias for AI, moving from least disruptive to most disruptive:
Representational, which is about asking models to represent information internally the way brains do. It’s the simplest of the four in some ways. The data you’d need is neural activity, and it stays compatible with the transformer stack we have today, and likely also with whatever replaces it.
Algorithmic, which focuses on new, biologically inspired learning rules. The data you’d need might be neural activity and/or connectomes.
Architectural, which are new circuit motifs derived from connectomes to replace the transformer. Transformers are the backbone of nearly everything we build, but they have no real analog to the brain’s circuitry, beyond the loose sense that both have something like attention and something like neurons. But nothing in a transformer derives from the brain’s macro- or micro-structure.
The compute substrate, which would be neuromorphic or wetware-inspired hardware that computes the way neurons do. This is the far end of the ladder where, as Sean put it, “we would have to throw out the entire stack and start from scratch.”
These are ordered by how much of today’s AI each one leaves standing. “You can kind of see as you go down this hierarchy,” Sean said, “that you’re going from things that are completely compatible with the existing hegemony of the LLM built out of transformers existing on GPUs, to ‘we’re going to throw out the entire stack and start from scratch.’” In other words, the compute substrate could have the most dramatic payoff, but it would also require demolishing most of the existing infrastructure of the modern AI ecosystem.
Representational alignment, on the other hand, is the least disruptive approach and it’s also the rung where the argument for reading brains gets most interesting. Whether it can improve AI hinges on whether neural data carries information you couldn’t get anywhere else. This is the line of attack Sean, Dan, and Tom argue should be prosecuted.
One warning from Dan that applies to every rung on the ladder is that the brain is only useful to copy at the right level of description. “One of the challenges historically in this field,” he said, “is to operate at the right level of abstraction for the specific goals that you have.”
Neuromorphic computing is the cautionary tale. Early efforts tried to reproduce the analog behavior of individual neurons and how they spike. That, Dan said, was a case where “folks were being much too grounded in the details of how biology was doing it, and not abstracting away the process that the circuits were actually carrying out.”
Nothing that follows is an argument for copying neurons. It’s an argument about a specific kind of information, and about reading it at the level where it means something.
Why behavior is not enough
In my post with Adam, we discussed whether brain data tells a model anything it couldn’t already figure out on its own. As Adam put it: “What’s the delta? What’s the difference between having that information and having just the information about the world that we train on now? Are there things in that neural activity that we can’t already predict from the data that it’s seeing?”
Put simply, the answer to this might be that behavior hides most of cognition. Sean, Tom, and their co-author Patrick Mineault discuss this in their recent paper Cognitive Dark Matter: Measuring What AI Misses. Sean gave the example of watching someone cook. “Imagine someone is making pasta for dinner and you’re observing them. At some point in the recipe, they reach into the cabinet and pull out some spice mix and put it in the pasta. At what point did they make the decision to do that? Just at that moment, an hour before, a week before when they were at the grocery store.” It only becomes a visible action when they reach for the jar, but the decision might have been made much earlier. “The behavior records the act, not the deciding. You’re only getting the selected behaviors,” he said, “not the unselected behaviors.”
LLMs are trained on the selected behavior, but human cognition is mostly weighing other options and choosing not to do those. These are all the evaluations we weigh (consciously or subconsciously) before we take an action. Recreating human reasoning may require understanding the discarded branches instead of just the chosen one. And that data isn’t gleaned from behavioral data because it never became visible. Neural data is a bet that you can read the compression loss.
You don’t need neural data everywhere. Instead of trying to make models brain-like across the board, the goal is to find the specific areas where they fall short and collect data from humans doing those specific tasks. Tom is a cognitive scientist and he works at this intersection of computer science and psychology. He’s interested in “how the methods we’ve developed for studying human minds give us a tool for understanding these AI systems.”
Sean, Dan and Tom believe that if you know where a model’s reasoning diverges from a person’s, you can design a task that isolates it. The neural data fills the gap at the level of representation where you’ve established the model is weak.
Capability and trust are the same problem
The hope in doing this is that this will make models both more capable and more trustworthy. The trust half runs through theory of mind. “We’re able to engage as strangers in a conversation and engage in business relationships or whatever it is because we have some theory of each other’s minds that will allow us to operate in ways that are safe for us,” Sean explained. “In order for us as humans to incorporate agents into our societies,” Sean said, “we will need to have our models of their minds be valid.”
The hope is that if you design tasks that are based on theory of mind or based on cooperative behavior, then you can make models more prosocial.
This is largely the thesis of cooperative AI research. In the 2021 Nature paper, Cooperative AI: machines must learn to find common ground, Allan Dafoe and his co-authors argue that theory of mind is a precondition for AI systems to cooperate safely with people, and that cultivating it should be a first-class research goal rather than a side effect of scale.
Also, a model that scores well but reasons in alien ways is a supervision problem. It’s hard to predict and hard to catch if something goes wrong. But by making a system reason more like a human, then perhaps the path it took to an answer becomes something we can actually follow.
The missing traces of thought
We are building AI almost entirely out of selected behavior, on the assumption that whatever matters survives into the behavior output. But what if it doesn’t? What if true decision-making happens upstream and gets compressed away before the visible behavior? If that’s the case, then a model trained on outputs is learning just from the residue of the reasoning that produced it. The goal of reading the unselected is that we can learn from the part it structurally leaves out.
Whether that part carries a signal an LLM couldn’t already recover is, as Adam said, “one of these things that needs to be tried.” Because if it does, the same signal could teach a model to reason where it currently falls short and make that reasoning something we can more easily understand.
After all, if a human mind is more than the sum of its outputs, then so is everything worth learning from one.
Author’s note: An LLM was used for light copy editing only (spelling, grammar, and clarity). Content, meaning, tone, and structure remain unchanged.