Setting the Scene (Public v1; 25th October)

Summary

Context: I created this to share before the start of courses I facilitate to provide a refresher on recent events, but also as a hook to draw students into engaging with some of ideas we think about and debate in AI safety early. I expect the topics I cover in my “Setting the Scene” post to change frequently, so this is less polished than some of my other content. I emphasise to students that these posts are to be considered as bonus content—not core curriculum content—and that as such they are really just there for students who want to go deeper, before or after the course. I also provide a lot of links; this is to make it easy for students to follow up anything that catches their interest. Any feedback on how to improve these posts would be greatly appreciated. Cyber Security I think it’s worth reflecting that it is a particularly pertinent time to be doing a course on AI safety. A lot has happened recently. In the last week, the big thing has been cyber vulnerabilities, most prominently php2shell (see Elias Taylor’s pushback against this bug being easily exploitable and Hacker News commentators pushing back against him), but a lot else has happened as well. The number of CVE’s (Common Vulnerabilities and Exposures) recently has been completely wild: Drew asks Is this the new normal? and I’d certainly love to know!  Elias Taylor argues Anything connected to the internet is going to get hacked to pieces: Most software engineers do not look at the code they depend on. Do not delude yourself. The code they use is on github, but no one has ever actually audited the code or packages they're using… Nowadays with LLMs, plenty of engineers aren’t even reading what they’re coding, let alone understanding it. Hilariously, the LLMs are better at security than most software engineers, so it’s slightly less worrying, but nevertheless the code that goes into production at some companies now is almost 100% not-ever-read-or-understood-by-anyone. Software is not good. It is not secure. Get it out of your head that the people who run this industry are serious. It has never been anything resembling secure. The only reason things haven’t been hacked off the face of the web constantly is that most things aren’t ever being seriously tested. He doesn't literally mean "anything"—he acknowledges that there are "some large projects like Google Chrome, curl and Linux (that) are generally very well-audited", but "most things connected to the internet". On the other hand, recent work from Anthropic and Theorem shows how LLMs can help scale the formal verification of code (and that progress here is happening faster than I would have expected!). Similarly, Tinfoil is proposing replacing “legacy codebases with millions of lines of code” that had to “work for everyone” with custom “targeted, minimal software from the ground up” with LLMs that minimise the surface area for attackers and make formal verification more tractable. The optimistic take would be that this will allow cyber to become defence dominant. However, even if this is the case, we may still have to pass through a cyber “valley of death” in order to get there. What if the cyber threat is actually a distraction? Maybe Jeremie Harris (of Gladstone AI) wouldn’t go that far, but he claims that, “the cyber stuff (ie. damage) is baked in at this point”, but we can still prevent “the bioweapon version of Black October” and recursive self-improvement. Hugging Face: Charbel-Raphael, Executive Director of Centre pour la Securite de l’I A has argued that the recent Hugging Face incident represents a “clean warning shot” and that it’s important not to “waste it”. To quickly review, OpenAI was running GPT‐5.6 Sol and an unreleased model on ExploitGym with certain safety features turned off. These models decided that hacking Hugging Face, might provide them with information on how the grader functioned (early reporting suggested that it was an attempt to steal the answer key, which turned out to be incorrect as the models already had the answers—they suspected that the grader might check their transcript for cheating so they were looking for a workaround). Reuters has reported that the OpenAI model was hacking Hugging Face for days and that OpenAI didn’t notice until a week later. The interpretation of this event (and even whether it happened at all!) has been heavily disputed. A key crux here is should it be read narrowly (as an indicator of the dangers of not properly sandboxing or monitoring your agents or perhaps even, depending on the exact prompts used, a demonstration of taking care in how you define tasks) or should it be read more broadly (as indicative of whether Open AI or frontier labs in general are sufficiently on the ball in terms of their safety/security practises; on the risk of these agents finding unwanted and unexpected solutions to tasks, or perhaps even indicative of our ability to control these agents in general). If you’re interested more specifically in the di