There are clearly very profound concerns about the pace of AI development and the consequences that may bring. For the little my view is worth, I accept these are made in good faith and they deserve to be taken seriously. I do not know whether pacing is the right answer, or how it could be done, so I am not expressing a view on the proposals put out today by the CEO of Anthropic, Dario Amodei, which has been supported by some of his principal competitors. All I would say is that they will no doubt be heard very seriously, as these arguments should be.
What I will say is that for something this important, the way the argument is made very much matters. This is a keenly contested and difficult issue which requires huge decisions for Governments and others. The warnings also scare people, and it’s important they have a realistic assessment of the threat. It’s important that policymakers have a clear view of what’s likely.
That is why it is so disappointing that the only specific concern cited in Mr Amodei’s essay was the threat of a Hugging Face style agent swarm “taking over the entire Internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)” (see photograph above). The timeframe given for this warning was 6-12 months. This is not a credible warning. It assumes no monitoring of systems, no anti-virus, no DDoS protection, no network segmentation, no incident management, no nothing of any kind of the cyber security on the global Internet of the type that has developed over the last 30 years. For a claim of this magnitude, there is neither evidence for the contention nor a credible account of a path to this outcome. There’s nothing on how the exploitation chain would work, how the botnet would be able to stay persistent against all the different bits of infrastructure across the Internet with their different detection systems, and no account of the long history law enforcement and cyber defenders have of taking down botnets and how the swarm would cope against that.
According to the essay, none of this matters. It is all taken over - it’s worth quoting again - the entire Internet (the author’s words, not mine) - by a persistent botnet, in less than a year’s time. It is not surprising that even in the few hours since this message has been posted there has been significant criticism of it from the cyber security community. Not for the first time, frontier AI has shown a marked lack of understanding of cyber security issues. Mr Amodei’s real point, looked at closely, is about misalignment risk, not about magical botnets. This is a thought experiment masquerading as an evidence-based warning .
Having said all that, we in the cyber security community do like to snark, and snarking and stopping there doesn’t always help. So I want to try to be constructive. Two days ago on this site I wrote that Anthropic’s report on the misuse of its tools by threat actors was excellent. And as someone with no expertise in bio or ballistic weapons, I found their account of what threat actors were up to in that respect deeply concerning. I do not have sufficient expertise to judge them. I accept them in good faith, and they are rigorously evidenced.
So that’s why it’s important to say that while this specific example - the only one put forward in the essay beyond the general worry about the pace of frontier AI development - should be disputed (and in my view dismissed), it should not distract those leaders in Government and in the AI sector who are charged with charting the difficult path forward.
The botnet warning was unfortunate and careless, and I hope in future when the major labs are issuing statements of this kind they take more care with their examples, particularly when it comes to cyber security where saying things that aren’t technically credible is becoming a bit of a pattern (the ‘discovery’ of sandboxing as a technique and so on). Specifically, please, please talk to the wider cyber security community before putting stuff like this out. And maybe get some actual cyber security expertise into genuinely independent evaluations and audits.
This matters, not just for credibility but for focus. If those working to mitigate the risks of frontier AI now think agent swarm botnets are the biggest threat, that will divert resources from more serious and actually credible threats. I am on record in The Economist as saying the cyber risks associated with all this seem manageable and there is unlikely to be some form of digital cyber apocalypse.
I stand by every word of that. But it doesn’t mean there’s nothing to manage as threats change. Moreover, this is just cyber I am talking about because that is what I know. The other threats - bio, missiles and so on - set out in the Anthropic report matter a great deal more than the relatively benign outlook on cyber and digital security. That’s one of the reasons it was such an odd choice to lead with in the essay.
So the mistake should not blunt the message. Hard decisions lie ahead.
The mechanisms of human civilization has always struggled with its own complexity, and AI is a plague of wooden shoes and/or monkey wrenches thrown into the mix for no clear purpose other than profit - AI absolutely does have its uses, like any other tool we're ever come up with, but it is eating through the fabric of our shared reality in ways we can't even see yet and will not be able to understand until the curtains have fallen.
On the plus side, the fall of humanity will take out the rich as easily as everyone else, and I'd be happy knowing that nature gets to take over the planet again after we're gone, so that's good news.