What Capable Agents Must Know: Why AI Consciousness May Be an Inevitable Byproduct of Capability

Summary

[No LLMs were used (or harmed!) in the writing of this blogpost!] Technical results can all be found in my UAI 2026 paper: https://arxiv.org/abs/2603.02491 This work, and this post about this work, was borne out of a frustration. The frustration first emerged over a year ago when I was at a Dave & Buster’s for the first time in years (for an annual ML department event, no less!), surrounded by flashing lights and NPC agents, not being able to objectively rule out that they were conscious or not, even though I strongly felt these particular programmed agents weren’t. I wasn’t particularly interested in playing the games there, as I mainly sat in confusion the whole time watching the various games running about around me… So, given my background in NeuroAI (though I prefer the term "natural science of intelligence", but "NeuroAI" is apparently catchier!) and wanting to make claims about the mind and brain quantitative, I set about trying to design empirical tests for leading theories of consciousness (e.g. global workspaces) that one could falsify in human brains (which we agree are consciousness!), as well as potentially corroborate general signatures of in animal brains, and possibly LLMs, where we have self-report and direct access but no consensus on their sentience—all to try to converge on substrate-independent architectures that give rise to this subjective experience. After a few months and about 129 (!) pages of failed attempts (briefly summarized in my BAMΞ talk here), I felt deeply dissatisfied with this as any sort of long-term research program, in part because it seemed somewhat ad-hoc and non-normative, tailored to each theory without really knowing why I should a priori accept a particular theory's property for deeming something conscious or not. One notable exception here is Integrated Information Theory (IIT), which my friend & longtime quantum complexity collaborator @ScottAaronson showed over a decade ago leads to absurdities, like labelling simple expander graphs "conscious". In fact, even current definitions of IIT's Phi (IIT 4.0) still fall to Scott's loophole! Not to mention recent negative results in Nature currently finding support for neither leading theory of IIT or global workspaces. Although absence of evidence is not always evidence of absence, I think more work should continue to be done in this vein looking at deeper brain structures like thalamus, which connect to many other brain regions, rather than examining the cortical surface alone. In fact, I used to think AI consciousness—the first-person subjective experience of what it’s like to be oneself—would not be necessarily guaranteed to “pop out” via task competence, possibly requiring other specific architectural conditions. In other words, I treated intelligence and consciousness as potentially orthogonal axes (sharing a view previously espoused by Anil Seth), and that intelligence did not necessarily imply consciousness. In this post, I will mention recent results from my UAI 2026 paper “What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty” (technical results summarized here) which have updated my views and take on a more normative approach to this question. Specifically, I prove selection theorems  about what capable agents must necessarily have, constituting the first arrow in this diagram: Capability (UAI 2026 paper) ↓ {world models, belief-like memory, emotion primitives, representational convergence} ? ↓ first-person subjective experience I emphasize that the second arrow has not been proven, and is a major open question for future work. I will suggest some preliminary empirical evidence towards it though, and why I think it’s eminently reasonable given what we know both in neuroscience and with modern LLMs. In other words, I think consciousness is likely far more common and less mysterious than initially meets the eye, especially when we take a control-theoretic perspective. Arrow 1: What Capable Agents Must Know Let’s start with the first arrow, which is what my UAI paper is about. There were three principal inspirations for this line of work: The first inspiration was the point of view established in NeuroAI over the past decade now, that task-optimized neural networks are the best predictors of large-scale neural population responses, across brain areas and species (check out my PhD thesis & recording to learn more at a high level – though I recommend my more recent CMU Robotics Institute talk for the latest in this domain, especially as it connects to agency). In other words, doing well on the AI goals of building more intelligent systems in the world, gives rise to better brain models too. Thus, NeuroAI has empirically shown that how competent you are on tasks is a strong forcing function of the internals. In fact, it was my former PhD advisor Dan Yamins who first suggested to me 6 months ago that capable robots may inevitably become sentient, encouraging me to watch S