The summer and autumn of this year has felt like the time period in which questions of AI governance entered clearly and forcefully into the public imagination. From the announcement of the attack by OpenAI agents on HuggingFace to the resignation of Jacob Coxon from Anthropic and his subsequent push in the media regarding how AI developers see risks of extinction, it feels like important topics have finally become salient. 

I’m not here to argue, somehow, that these disclosures shouldn’t have occurred, or that the publicity associated with them is deleterious to the wider cause of AI safety – they are not. I believe that the risks from frontier AI are real, as is the need to talk about them openly. This brief blog post, however, will instead focus on a basic challenge related to AI governance: which question is the important one to ask?

While the immediate answer for someone concerned about the existential risk of frontier AI might be: ‘How do we avoid human extinction?’, and while the question from someone primarily concerned with cybersecurity could instead read: ‘How do we protect our critical cyber systems from AI incursions, autonomous or otherwise?’, both of these questions depend on prior assumptions which must be properly enumerated before action can be taken.

A major challenge which we face is the fact that when we deal with a problem which involves exponential curves, as it seems the development of AI capabilities does, the temporal gap between being ridiculously early and slightly too late to stop catastrophic outcomes can be extremely narrow. People are not built to understand exponential development, and this hinders us in our work on AI safety. 

So, which questions are important to consider in our work on governance? From the perspective of absolute risk mitigation, the important question concerns when human beings might go extinct under the current regime of development This is, of course, complicated by the current regime not moving in a linear fashion: ‘we’re completely safe’ and ‘we’re about to be made into paper clips’ might only be a few days apart.

This question, though, is not the same as asking when it is too late for governments and other regulatory bodies to put a stop to the sort of AI development which poses catastrophic risks. It is, indeed, on almost all reasonable scenarios a lower threshold. Loss of control of AI systems does not mean existential risk, even if it should also be considered an unacceptable outcome. 

Similarly, the question of when governments might be too late to stop AI development is not the same as asking when AI technologies will be sufficiently capable that the regulatory options required to contain them would be more disruptive to civil society than simply placing restrictions on supply chains and infrastructure at this present point in time. Pushing these technologies as far as we possibly can, without violating our existential safety, before reigning them in is not the sensible way to approach this topic. 

Each of these questions carries implications for how we govern, based upon what our publics are willing to accept, the pressure that our governments feel and the capabilities that frontier AI developers are willing to discuss openly. Existential risk is clearly the most pressing question, but it is not the only one that we have to consider. If we are to remain human-focused, concerned with our broader wellbeing, then we have to move adroitly between different layers of analysis as different situations require it.