There should be $100M grants to automate AI safety

Summary

This post reflects my personal opinion and not necessarily that of other members of Apollo Research. TLDR: I think funders should heavily incentivize AI safety work that enables spending $100M+ in compute or API budgets on automated AI labor that directly and differentially translates to safety. Motivation I think we are in a short timeline world (and we should take the possibility seriously even if we don't have full confidence yet). This means that I think funders should aim to allocate large amounts of money (e.g. $1-50B per year across the ecosystem) on AI safety in the next 2-3 years.  I think that the AI safety funders have been allocating way too little funding and their spending has been far too conservative in the past 5 years. So, in my opinion, we should definitely continue ramping up “normal” spending, e.g. pay more competitive salaries, allow AI safety organizations to grow faster, and other things in that vein.  However, these “normal” spending patterns are not sufficient under short timeline assumptions and the obvious way to spend more money quickly is to aggressively encourage finding ways to use automated labor for AI safety.  What is an “automated AI safety scaling grant”? An “automated AI safety scaling grant” aims to aggressively encourage attempts to use automated labor for AI safety at scale. The explicit intention is that if somebody manages to find ways to build scalable safety pipelines, they can be confident that they can run these pipelines at scale with short turnaround time from grantmakers.  An example of such a grant would be: Step 1: Show scalability An organization submits a sketch of a pipeline that they think could scale significantly. In the best case, they have some empirical evidence for this claim already.  Both parties agree on the “scaling condition”, i.e. what empirical evidence would be sufficient to spend more money on that pipeline By default, the scaling condition could be a plot that has “money spent on the pipeline” on the x-axis and “some reasonably proxy of safety” on the y-axis (as long as both parties are convinced that the proxy is not goodharted). An example of a proxy could be “number of distinct, egregiously misaligned ‘features’ found by interpretability (and a human expert would judge them as egregious, diverse and high quality)”. For example, such a grant could be $5M for 1 year where $2M is for salaries and overhead and $3M is for compute (e.g. fine-tuning, API costs and GPU rents). These stats can vary depending on team size and prior confidence in the scaling hypothesis.  Step 2: double up When the funding condition has been met, the funder is willing to quickly ramp up additional funding to scale up the experiments.  For example, let’s say the grantee was able to produce the scaling plot and the evidence is convincing to both parties. Then the funder makes another grant for $14M where $10M is for scaling the pipeline and $4M is for scaling the team and rest of the organization. They then both agree on the next milestone. For example, the milestone could be to extend the scaling plot by four additional datapoints that show no or little diminishing returns to scale. Step 3: big scale When the double up was successful, the funder is willing to quickly and significantly increase the funding by another OOM.  For example, let’s say the grantee was able to show that the scaling plot can be extended and the increases still meaningfully translate into safety (rather than e.g. goodharting a metric that ceases to be useful). Then the funder makes another grant for $108M where $100M is used for further scaling the pipeline and $8M is used for other overhead.  Note: since the explicit intention of these grants is to scale quickly it is possible that the time between Step 2 and 3 can be as little as a few months. Step 4: real-world integration Assuming an organization can show that they can convincingly spend $100M on something that meaningfully translates into AGI safety at scale, there are multiple further possible actions.  Given that it was financed by a grant, it needs to be for the public benefit. Thus, under all circumstances, this information has to be published with relevant details. This can e.g. include open sourcing the entire pipeline or writing a paper with the core details that would enable others to replicate the pipeline.  Collaborate with labs to adopt these techniques into their alignment pipelines and help them implement it into practice. The team joins one AGI lab to implement the pipeline at that lab. If this pipeline could be profitable, explicitly try to turn the organization into an AGI safety for-profit. If it is already a for-profit, build out products around the idea that lead to drastically increased safety. Try to scale it further to $1B if we still haven’t hit diminishing returns and the pipeline continues to produce great safety results.  Other considerations Who should be able to receive such a grant? I think a