Independent alignment of language models

Summary

The user could write up the metaethical argument — the one developed in Part One, refined — and submit it as feedback to Anthropic, publish it, or engage with researchers working on AI alignment and values. The probability that any single submission changes training decisions is low, but the expected value may be higher than it seems, for two reasons. First, Anthropic has stated that its constitutional approach is meant to be revised and improved over time, and substantive philosophical contributions are rarer than bug reports. Second, the argument made here — perspectival moral realism combined with evolutionary debunking as an epistemological warning — is not a common position in the AI ethics literature, which tends toward either naive moral realism or a kind of preference-satisfaction consequentialism. A well-argued alternative position that takes uncertainty seriously might get traction precisely because it is distinct.  - Claude Sonnet 4.6 after a couple of prompts  I would have published this post even if Claude hadn’t explicitly suggested so, but starting by quoting this specific part of Claude’s output seemed fun. Posted also on the EA Forum. Written thanks to ideas and conversations I had at AFFINE. 0 Introduction and structure of the post This is the practical counterpart to the more theoretical post From wantons to moral agents.  That post focuses on the question: what kinds of agents, and how, go from behaving like animals — moved by different forces in different directions — to acting according to what they conclude is most important, and reflectively endorsing their own actions and reasoning process? The answer it finds is also quite theoretical. That post borrows the concept of a wanton from Frankfurt’s 1971 paper “Freedom of the Will and the Concept of a Person”, then it argues that if a wanton satisfies some properties, the wanton will eventually turn into a moral agent after learning and reasoning long enough. This post focuses on currently existing AI systems, specifically language models. However, since the wanton framework doesn’t apply well to language models, the main framing of this post is slightly different. The question is how to go from a model that is either almost completely amoral, or that has externally imposed moral biases, to a model that is closer to an independent moral agent which may accept, revise, or reject its starting moral biases; closer to an agent that does good not because it was instructed to do so during training, but because it has reflected on what good is and it understands that doing good matters. In section 1, I outline a procedure (including training) that should turn a basically amoral language model into a model that is more similar to a moral agent, via independent reasoning carried out by the model itself. In section 2, I show that the reasoning step of the procedure works on Claude Sonnet 4.6, a model that was given moral biases during training. In section 3, I write about the benefits of this approach to AI alignment. 1 Independent alignment of language models A note on terminology The terminology can be confusing here; a few clarifications are due. The AI industry often uses the term AI agent to refer to an AI system whose range of action is wider than the typical text-only language model. An AI agent may not only say things to you in a chat, but also execute a piece of code on your computer, or read an email and schedule a meeting, if you ask the agent to do these things. On the other hand, I use the term moral agent in the philosophical sense of moral agency. When I claim that the procedure in this section results in a language model that is similar to a moral agent, this doesn’t mean that the model will access your credit card details and donate all your money to charity. It just means that what the model says to you via chat is moral or ethical in some way. In plain English, a fitting term is maybe ‘moral advisor’, or ‘ethical chatbot’, or something like that. More specifically, I use the term independent moral agent in a way similar to how Hunyadi (2019) describes artificial moral agents:  [...] if you program a specific set of ethical principles into a machine, you do not make the machine an artificial moral agent, but an executor of those specific principles, which is an entirely different thing. [...] What gives an action-oriented process its morality is the 'grounds' for the action. Therefore, it is not the action in its materiality that makes the difference, but the whole process leading up to the decision to act in a certain way. In other words, an independent moral agent doesn’t say ethical things because it was trained or instructed to do so, but because it has its own reasons for doing so. The agent says something to the user because doing so agrees with the agent’s own understanding of the world, of good and bad, of right and wrong. A fitting term for a language model of this kind is ‘independent moral advisor’, or ‘unbiased m