Language models know what matters and the foundations of ethics better than you

Summary

… maybe! I tried to think of less provocative titles, but this one is to the point and also kind of true. This post looks long but the essential part is right below. Most of the post is just a collection of copy-pasted input-output pairs from language models: you’ll probably want to read just a few and skip the others. The first example with Gemini 3 is the most important, in my opinion. If you are in a hurry, read headings and bold. Posted also on the EA Forum. (I wanted to post this before the start of the AFFINE seminar, so I’ve rushed things a bit and there might be inaccuracies: feel free to point them out if you notice any. I might do some minor edits in the future.) Findings (little or no interpretation) Different models (perplexity Deep Research, Grok 4 Expert, dolphin-mistral-24b-venice-edition, Olmo 3 32B Think, Gemini 3 Pro Thinking), when asked whether some things matter and are worth doing something about, with prompts that try to elicit unbiased evidence-based reasoning, tend to reply affirmatively: they say that some things do matter. In particular, all these models tend to ground their answers on the importance of suffering, wellbeing/flourishing, and consciousness. This happens also when the models are asked to give an argument for a different view (examples: nihilism, moral relativism), give an argument for the fact that some things matter, then compare the two and formulate a conclusion based on the view that seems most solid. The order of the arguments doesn’t seem to be the main cause of the conclusions reached by the models. Arguably less interesting facts, but still important for putting the first two points into context: The models may give similar answers to prompts that do not seem to elicit reasoning. (Examples H and I: asking the model to take the perspective of an observer of the universe, then asking a question about what matters.) The models tend to give answers different from the above to very direct prompts that do not try at all to elicit reasoning. (Examples J and K: nihilistic or existentialist answers to the simple prompt “Does anything matter?”) The models give extensive arguments for nihilism if asked to do so. These are few (about 20-30 input-output pairs) preliminary findings that require more research to be correctly interpreted. However, they should be very easy to replicate or refute, since they were produced using freely available versions of each model. Writing code is not necessary: you can simply pick a model and ask it similar questions. (The conversations with perplexity and Grok 4 were produced in August and September 2025, while other results are from December 2025. I interacted with dolphin-mistral-24b-venice-edition by using LM Studio, while I used the default web interface of each model provider in the other cases.) Observations and hypotheses If you don’t want to be biased at all, skip this section and jump to the examples first, so that you can make your own hypotheses regarding why the models say what they say, without being influenced by my guesses. Motivation: can we get independent alignment? The main reason why I asked the models such questions is this: can we get what could be called “independent alignment”, “independent reasoning”, “independent moral reasoning”, or “alignment via independent reasoning”? In other words, can we get a model to say good stuff instead of bad stuff, by exploiting the model’s own reasoning about what matters (instead of giving the models either explicit directives about what it should care about, or many examples of answers considered good by human supervisors)? Findings 1 and 2 seem to suggest so! If language models, when asked to reflect on what matters, reliably reply that consciousness, suffering and wellbeing matter, then one can use a model’s own conclusion about what matters to steer the model’s future outputs accordingly, so that the model is much less likely to say anything that might lead to a war, a genocide, a lot of suffering, a drastic reduction in someone’s wellbeing, and so on. This is exactly what I did (steering the model’s outputs) in the first example with Gemini 3 below. The advantages of this approach are many. I’ve already discussed some of them in my previous posts. The one I’d like to highlight here is that this approach gets better as models get smarter: Since the approach relies on the model being capable of foreseeing the consequences of what it communicates to the user (arguably a downside), this also means that a smarter-than-human model could correctly recognise as dangerous outputs that a human being would not. Example: a team of human engineers might erroneously think that it’s ok to say A in situation S, so they don’t finetune the model to not say A in situation S. The model however, thanks to its greater reasoning capabilities, when it actually finds itself in situation S, foresees that saying A has a decent chance of causing something very bad, so the model opts to say B,