Motivated reasoning, confirmation bias, and AI risk theory

Summary

Of the fifty-odd biases discovered by Kahneman, Tversky, and their successors, forty-nine are cute quirks, and one is destroying civilization. This last one is confirmation bias. - From Scott Alexander's review of Julia Galef's The Scout Mindset. Alexander goes on to argue that this bias is the source of polarization in society, which is distorting our beliefs and setting us at each other's throats. How could someone believe such different things unless they're either really stupid or lying to conceal their selfishness? I think smart people who care about the truth go on believing conflicting things largely because of confirmation bias and motivated reasoning. The corner of civilization I'm most worried about is the one figuring out how to handle the advent of strong AI. I think confirmation bias makes us each a little to a lot overconfident in our beliefs about alignment and AI impacts, and that's pretty bad for collectively finding the truth. I think the effects of biases are still strong and still overlooked in this field, despite its strong values of truth-seeking and relative awareness of biases. Bias has more influence where there's less direct evidence, and that's the case in alignment theory and predicting AI impacts. I think the effects are underappreciated in part because empirically measured effect sizes tend to understate the problem. Confirmation bias happens at multiple stages of cognition, so it compounds during complex thinking. In this article, I'll talk about the relevant empirical research, challenges to reasoning about complex topics with a human brain, and some implications for AI risk and alignment thinking. I studied the brain basis of cognitive biases on an IARPA program on understanding biases in intelligence analysis, from 2011-2014. I became fascinated by motivated reasoning, and kept it as a research interest until switching to alignment in 2022. Confirmation bias is well known, and careful thinkers already try to avoid its effects. But the mechanistic explanations of confirmation bias are rarely discussed. Confirmation bias seems to be caused by several locally or partly rational effects.[1] The primary sources seem to be motivated reasoning; differing prior beliefs; discounting evidence; and coherence bias (§2.4). I focus on motivated reasoning and the cognitive limitations or problem complexities that create fertile ground for confirmation bias. Understanding our biases and limitations does not cure them, but it's a start at correcting for and working around them. Confirmation bias may play a large role in group and personal epistemics. Measured effects on specific tasks are modest, but they can compound in complex problems. Studies have demonstrated confirmation bias in selecting evidence or arguments, in evaluating them, and in remembering them. There are also biases resulting from choice of framings and hypotheses for evaluation, and social effects of weighting evidence and opinions of some experts over others. That's five layers across which biases can cascade or compound, and biases are usually pushing in the same direction at each layer. Section 4.3 contains some rough estimates of total effect sizes; they go from large on up, depending on assumptions about how carefully you're debiasing your thinking. I recently realized that motivated reasoning was stopping me from writing this article. I was afraid of writing it badly and motivating readers against the topic itself. This fear was giving me a negative reward signal, because motivated reasoning could be a major factor in alignment thinking, and I care a lot that we collectively get this right. To allay my remaining fears: I'm not telling anyone they're wrong about AI impacts or alignment. Despite thinking about and researching these questions a lot, I'm not confident where the truth lies. Motivated reasoning could easily go in multiple directions. It could be simple motivation to look forward to a bright future. Or it could spring from attachments to theories or group membership, or identities as farsighted or willing to look doom in the face. These sources of confirmation bias are pernicious and difficult to correct. I don't think I'm close to correcting all of my own. But I think the effort is worthwhile. I don't know how much effort you've put into correcting for motivated reasoning and other sources of confirmation bias, but I suspect for most there's still low-hanging fruit and benefits to claim. I discuss some and speculate on more in the last section. 1.1 Motivated reasoning[2] Confirmation bias is an effect in which we are irrational in favor of beliefs we already hold. Motivated reasoning is one cause of that effect. Loosely defined, it is our tendency to believe what's comfortable or useful. Motivated reasoning in this sense is largely non-conscious. The term is sometimes used for deliberately selectively presenting evidence and arguments. But here and in the academic literature, motivated reasoning refer