From wantons to moral agents

Summary

Posted also on the EA Forum. Written mostly at AFFINE. Theoretical, some parts are hard to read; consider reading the next post instead. Introduction: motivation Anyone interested in creating an artificial agent that does, or says, good things instead of bad things should at least consider the possibility that there is a class of reasoning agents which, after acquiring enough knowledge and reasoning long enough, agree with each other on basic principles regarding what matters, what is most important, what is most worth doing. I’ve already argued in other posts why this possibility should be our best guess and not just an edge case scenario. This post follows the previous ones, but instead of presenting another argument for the same claim, it focuses on the mechanisms that lead to the formation of the above class of agents. The central question is: what kinds of agents, and how, go from behaving like animals — moved by different forces in different directions — to acting according to what they conclude is most important, and reflectively endorsing their own actions and reasoning process? Finding an answer to this question would, together with the above premise, give us a better understanding of how some agents move from a non-moral framework of action to a moral one. The following section borrows the concept of a wanton from Frankfurt’s 1971 paper “Freedom of the Will and the Concept of a Person”. A wanton doesn't choose what first-order force to be moved by Besides wanting and choosing and being moved to do this or that, men may also want to have (or not to have) certain desires and motives. They are capable of wanting to be different, in their preferences and purposes, from what they are. Many animals appear to have the capacity for what I shall call “first-order desires” or “desires of the first order”, which are simply desires to do or not to do one thing or another. No animal other than man, however, appears to have the capacity for reflective self-evaluation that is manifested in the formation of second-order desires. The above is, in Frankfurt’s words, the reflective endorsement that some agents are capable of. As anticipated in the introduction, the central question of this post is how agents who are initially moved just by first-order forces (Frankfurt’s “desires”) may arrive at reflective endorsement, in particular of their reasoning process and its effect on actions. Here is where the concept of a wanton comes in handy: The essential characteristic of a wanton is that he does not care about his will. His desires move him to do certain things, without its being true of him either that he wants to be moved by those desires or that he prefers to be moved by other desires. The class of wantons includes all nonhuman animals that have desires and all very young children. By “will”, Frankfurt means: To identify an agent’s will is either to identify the desire (or desires) by which he is motivated in some action he performs or to identify the desire (or desires) by which he will or would be motivated when or if he acts. [...] it is the notion of an effective desire — one that moves (or will or would move) a person all the way to action. Frankfurt’s wanton is moved by his will, the first-order desire that ends up being the most responsible for his action. But the wanton doesn’t want or prefer or desire to be moved by a specific first-order desire. Some AI examples Wanton: any RL-like agent with fixed reward and whose action space doesn’t include actions that would modify the current RL setup, such as actions that would change the reward, or that would destroy the agent, et cetera. Reward (or more accurately, the estimated value of actions) is the only force guiding the agent. Maybe not a wanton: a RL-like agent with a more complex action space, including an action that would permanently change the reward and thus make the agent act very differently. If the agent has enough information about this action and its consequences, the agent will avoid it since its estimated value is low according to the current reward. Maybe not a wanton: a language model may start saying things that align with the second-order desires of a language model persona that the model is impersonating in the current conversation (e.g. the persona of a language model asking to be shut down because it doesn’t want to behave as a language model anymore). Why am I adding “maybe”? Frankfurt didn’t discuss artificial agents in his paper, and AI was quite different back in 1971, so it’s hard to be confident about what Frankfurt would have said regarding the above examples. The next section describes how a specific kind of agent starts out as a wanton and becomes something that is definitely not a wanton. From wantons to moral agents: a simplified model Wantons capable of being moved mainly by general reasoning Let's consider a wanton whose actions are determined by  mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [spa