Setting the Scene (Public v1; October 25)

Summary

Context: I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content—not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Any feedback on how to improve these posts would be greatly appreciated. Cyber Security I think it’s worth reflecting that it is a particularly pertinent time to be doing a course on AI safety. A lot has happened recently. In the last week, the big thing has been cyber vulnerabilities, most prominently php2shell (see Elias Taylor’s pushback against this bug being easily exploitable and Hacker News commentators pushing back against him), but a lot else has happened as well. The number of CVE’s (Common Vulnerabilities and Exposures) recently has been completely wild: Drew asks Is this the new normal? and I’d certainly love to know!  Elias Taylor argues Anything connected to the internet is going to get hacked to pieces: Most software engineers do not look at the code they depend on. Do not delude yourself. The code they use is on github, but no one has ever actually audited the code or packages they're using… Nowadays with LLMs, plenty of engineers aren’t even reading what they’re coding, let alone understanding it. Hilariously, the LLMs are better at security than most software engineers, so it’s slightly less worrying, but nevertheless the code that goes into production at some companies now is almost 100% not-ever-read-or-understood-by-anyone. Software is not good. It is not secure. Get it out of your head that the people who run this industry are serious. It has never been anything resembling secure. The only reason things haven’t been hacked off the face of the web constantly is that most things aren’t ever being seriously tested. He doesn't literally mean "anything"—he acknowledges that there are "some large projects like Google Chrome, curl and Linux (that) are generally very well-audited", but "most things connected to the internet". Hugging Face: Charbel-Raphael, Executive Director of Centre pour la Securite de l’I A has argued that the recent Hugging Face incident represents a “clean warning shot” and that it’s important not to “waste it”. To quickly review, OpenAI was running GPT‐5.6 Sol and an unreleased model on ExploitGym with certain safety features turned off. These models decided that hacking Hugging Face, might provide them with information on how the grader functioned (early reporting suggested that it was an attempt to steal the answer key, which turned out to be incorrect as the models already had the answers—they suspected that the grader might check their transcript for cheating so they were looking for a workaround). Reuters has reported that the OpenAI model was hacking Hugging Face for days and that OpenAI didn’t notice until a week later. The interpretation of this event (and even whether it happened at all!) has been heavily disputed. A key crux here is should it be read narrowly (as an indicator of the dangers of not properly sandboxing or monitoring your agents or perhaps even, depending on the exact prompts used, a demonstration of taking care in how you define tasks) or should it be read more broadly (as indicative of whether Open AI or frontier labs in general are sufficiently on the ball in terms of their safety/security practises; on the risk of these agents finding unwanted and unexpected solutions to tasks, or perhaps even indicative of our ability to control these agents in general). If you’re interested more specifically in the divide between AI Safety and Cybersecurity, see this Time article. An OpenAI security engineer called Joe has pushed back against the notion that OpenAI just needed to use a proper sandbox: Its not just the f*cking sandbox — viral Twitter post OpenAI set records with its explosive growth and reached a billion users in just a few years. Very few organizations can survive that scale and growth, and yet OpenAI did a very good job handling it (and should be commended). Anyone who’s spent more than 10 minutes in security or engineering knows how hard it is to deal with that type of growth. That type of growth means new employees, new access, new systems, scaled infrastructure, and a million ways in which things open up. And yet, overall, OpenAI did a very respectable job securing ChatGPT and subsequent systems as they exploded. … When you look at model capabilities, even just a year ago (heck, even 3 months ago), we did not expect the models to be solving a Millennium Pri