Announcing the Corrigibility Research Fund

Summary

TLDR: I'm managing a new fund, housed at Lightcone Infrastructure, that will award at least $200,000 in grants and prizes for corrigibility research in 2026. Roughly half will go to traditional grants (first application deadline August 23rd) and half for prizes recognizing excellent work done this year. If you have interest in working on corrigibility, now is a good time to start! Apply via email: [email protected] Why this fund exists When I first dived into AI safety and alignment in 2009, the field was basically nonexistent. I've been relieved and gratified to see attention and funding grow, especially in the past few years. But even now, nearly all AI safety funding goes to evals, control, or interpretability. Work on alignment itself still remains deeply neglected, and it's only through alignment research that the core problems get solved. At this year's LessOnline I was talking about this dynamic with Peter McCluskey, particularly around our shared interest in corrigibility. In the wake of that conversation, Peter, being a long-time patron of alignment work, directed a portion of his philanthropy towards launching this fund, with the goal of increasing the amount of corrigibility research happening around the world. Lightcone Infrastructure agreed to house the fund and appointed me as its manager, due to my expertise on the subject. Why corrigibility Corrigibility is one of (if not the most) promising angle on creating superhuman artificial intelligences that reliably act in alignment with human values. Training for ethical behavior and direct alignment with humanity runs headlong into known challenges: prosaic methods can't reliably distinguish reward proxies from true goals, instrumental convergence means that even partly-aligned agents will become self-preserving and subversive,[1] and the philosophy of ethics remains woefully unsolved, such that we wouldn't even know what values to instill, even if we could reliably write the AI's values by hand. Corrigibility, by contrast, offers a solution: build an AI that aims to keep the human principal in the driver's seat, empowering them to make wise choices (perhaps aided by the AI's counsel), rather than relying on the AI's direct judgment. This runs the risk of concentrating power in the hands of humans who might misuse it, but human alignment is a less-fraught problem, and is amenable to known strategies, such as democratic oversight. Thanks to the nature of corrigibility, a purely-corrigible agent can be expected to avoid scheming and other instrumentally-convergent strategies. And while corrigibility itself does not solve the limitations of machine learning, it is a simpler target than all of morality, and there are reasons to hope that in practice, imperfectly-corrigible agents still cooperate with their principals to surface their flaws and assist in pushing towards even more corrigible assistants. This robustness gives hope in something closer to an iterative approach, where control and interpretability techniques come together to produce a realistic plan for scaling up to the level of human-intelligence and beyond. (For more of my thoughts, see CAST: Corrigibility As Singular Target) I am not alone in placing a high level of emphasis on the need for corrigibility. Eliezer Yudkowsky and Paul Christiano have both written at length about how it is central to their best hopes for alignment, as well as many other brilliant alignment researchers.[2] The latest constitution from Anthropic mentions the term sixteen times, and has a dedicated section for it. OpenAI is similarly bullish on creating AIs that are tool-like and deferent. (Arguably more so than Anthropic!) Despite this, the number of people working directly on corrigibility, such as on clarifying the concept, formalizing it, testing whether and how it can be trained into current systems, and mapping where it breaks, is vanishingly tiny. My hope is that this fund shifts that, both by directly paying for work and by broadly signaling that the work is valuable. What counts as corrigibility research For the purposes of this fund, anything that predictably advances humanity's understanding of the subject is fair game. This spans the full range from pure theory (e.g. formal models, impossibility results, decision-theoretic analysis) to pure empirical work (e.g. training experiments, evaluations of corrigible behavior in frontier models, surveys of how laypeople think about the topic). Distillation of existing work is also welcome. The fund will be prioritizing efforts that cut to the heart of the subject, but feel free to apply for funds even if your research is only tangentially related. The goal is to impact the AIs that actually get built. We're looking for work that is legible and relevant to the people making decisions about real systems. Theoretical work that's judged as too esoteric to be of interest to someone like Joe Carlsmith is unlikely to get funding. Work that's incompat