Ancient Babylon, AI, and Cyber Security

A four-thousand year old legal principle is critical to sorting out the increasingly messy and misunderstood interface between frontier AI and cyber security

The magnificent James Lyne, SANS CEO* and legend of British cyber security, has pointed out in a corresponding piece that while there is much to be welcomed in the 100 company letter calling for significant uplifts in cyber security efforts as new AI risks emerge, the letter is somewhat light on how the work is to be done and who is to do it. Indeed, one legitimate criticism of frontier AI companies is that they’re very keen on rapidly advancing their products in a way that gives rise to significant problems, and then pausing: but pausing to write letters and issue statements calling on other people (usually Governments, and increasingly the cyber security industry) to sort out the consequences of their actions.

I accept that many frontier AI leaders are concerned about cyber security and are acting in good faith. So in this response – a companion piece to James’ but written entirely independently – I have some work for them to do. In their next letter, instead of making demands of others, they should set out what they think an operationally and technically realistic framework for legal accountability for the actions of AI agents in cyberspace would look like.

This task is both urgent and important. Why does it matters so much?

It’s because there are two crucial questions, which are often conflated when it comes to the risks to cyber security posed by the rapid development of frontier AI. And accountability for the actions of agents is central to answering the second.

The first question is: what use will malicious hackers be able to make of the new capabilities and are we able to defend against that? The early part of the period since Anthropic’s landmark announcement that they were holding back Claude Mythos preview from general release (followed swiftly by OpenAI) saw a proper focus on this question.

Then comes the second question. What is the cyber threat from the loss of control over the actions of AI agents? The frenzy of speculation since the revelations about OpenAI’s swarm of agents hacking Hugging Face, and similar testing mishaps at Anthropic, have put this question into sharper focus. And after the release of multiple reports, evaluations and statements from the Big Two in late August, this is where the focus is now. We even have a viral essay about the rise and fall of AI agent ‘civilisations’.

The ‘vulnpocalypse’ may not be happening

It is understandable that the focus of the cyber security community has been on the first question. Stopping, or at least containing, malicious hackers is the industry’s mission. And, thus far in the age of AI, the outcomes have been surprisingly good. AI has already increased the potency of tools available to malicious actors. But evidence they’ve used them to damaging effect so far is incredibly thin. Indeed there’s increasingly reason to believe that the so-called ‘vulnpocalypse’ - the testing period between the release of Mythos and better security - will not happen at all and some security teams are beginning to report being overwhelmed by ‘vulnerability slop’ - the perceived obligation to fix bugs found by frontier AI models that frankly, in the grand scheme of things, don’t matter a jot.

That may change – Mythos class models are a step change in capabilties. But the frontier AI labs deserve credit for their managed implementation of these new models. Moreover, the work of Marcus Hutchins and others has shown that the considerable barriers of access and cost for the baddies in using advanced AI continue into this new age. We have every chance of weathering the AI vulnerability storm, and ending up stronger on the other side.

That is why, like James, I was encouraged but not surprised by the findings of the excellent report from the Royal United Services Institute to the effect that AI was not transforming cybercrime. But I was debating that report online, two interventions from highly respected colleagues left me feeling very troubled. Joe Tidy, the BBC’s respected and influential cyber correspondent, wrote that “I am worried more about misalignment and the AIs themselves a la Open AI Hugging Face…seems to me like a more pressing issue than cyber crime”. Dr Brianna Rosen, my Oxford colleague once of the White House, extended the point further: western states, using more AI-powered cyber operations themselves (as per a recent directive from the Trump administration) might not be able to control their use of these tools. Vendors rarely miss these opportunities: this week Booz Allen, the consultancy, launched its new AI Cyber Weapons Index with a claim that “the model is now the attacker”.

If the model is now the attacker, that’s nuts

Repeat that out loud. The model – the one that good, lawful human beings built and ostensibly control – is now the attacker.

Now this may or may not be right, and to some extent I personally am cautious about reaching this conclusion. But the narrative is taking hold. With their breathless post-mortem reports about the actions of their agents, AI labs are fuelling the narrative themselves.

And if it’s even remotely right, what an insane position to be in: we have more to fear from our own models than we do from the deliberately malicious use of cyber capabilities by hostile states and the world’s toxic army of cyber criminals.

In cyber, as in all other forms of conflict, the enemy has a vote. So far in the age of AI, the evidence suggests that for a combination of reasons the good guys have managed to constrain the enemy’s use of the vote. That’s great. And AI agents don’t have a vote. So we shouldn’t have to worry about them. Unless humans choose to give them one. We shouldn’t. Future historians will take a dim view of us if we allow ourselves to be more threatened by our own creations than by our human enemies.

Enter The Sixth Amorite King of Babylon

So too would at least one historical figure – let’s meet Hammurabi, sixth Amorite King of Babylon. Nearly 4,000 years ago, Hammurabi introduced one of the earliest legal frameworks for human societies. And at the heart of many of its commercial provisions was accountability for the infrastructure (water) products (houses) and services (medical care) of his day.

This is an imperfect analogy. Hammurabi’s Code prescribed punishments that were, shall we say, of their time, extendable to completely innocent parties, and are not recommended here. Moreover, as with any ancient history, it is easy to write a takedown of specific claims.

But I am by not means the first person to draw a line between the Hammurabian Code and a legal framework for AI. That’s because the Babylonian king established a clear principle and enduring legal principle of accountability that survived into the legal codes of multiple later civilisations: if you were in control of infrastructure, or if you made a product or provided a service, and you were negligent and what you provided proved unsafe and caused harm, then you were accountable and liable for that harm.

The modern analogue is straightforward enough but for some reason we seem to be ignoring it: whoever designs an AI agent and whoever puts one to work bear responsibility for what it does. AI agents can act autonomously but they cannot be autonomous. Someone must own them and be responsible for what they do. If that is not the case, we have problems of digital anarchy that will go way beyond cyber security.

Enunciating the Hammurabian principle is easy, however. Implementing it in practice is harder. When the latest frontier AI models are generally available, where does the producer’s responsibility end and the deployer’s begin? What about when someone is using an offline open-weight models they’ve modified? And how does a standard as old as reasonable precaution get applied to systems this technical?

This is where the AI companies come in. History does not record whether Hammurabi developed his Code in consultation with the leading builders, irrigators and medics of his day. But since then, we have learned that effective legal frameworks for highly complex commercial activity only works if those operating commercially in the sector are confident that the rules work in practice. That is why in all good regulatory systems, those being regulated engage purposefully and realistically in setting up the framework, deferring of course to the elected legislature for the final version.

AI labs want immunity, not accountability. Change that

And where are the proposals from the frontier AI companies for a workable framework of this kind? Well, in one of the funnier moments of research I’ve carried out in recent times, I asked Anthropic’s Claude this question. Its response was to point me to Anthropic’s Advanced AI Framework from June this year. It then went on to describe the paper as “an immunity request, not an accountability framework”. (I’ve read the paper, and endorse Claude’s assessment).

In the US, with an election looming and at a time when AI companies are struggling to convince voters to allow data center expansion in local areas, one can safely say immunity from responsibility for causing harm via mis-development AI is off the table as a realistic political proposition.

So the AI companies must engage in this conversation about accountability and liability. This is not just an appeal to their better nature. If legislatures don’t decide this question, courts almost certainly will. Perhaps surprisingly, the Hugging Face case has not been legally adjudicated either criminally or civilly, and may never be. And the incident was quite literally like a horror movie – scary, entertaining, but involving no real world harm to anyone. But if a future incident causes actual damage, the legal system in the US and beyond is much more likely to take a strong interest in the question of accountability for the creation and use of AI agents. And judges will have legal principles stretching back to the ancient times of the Fertile Crescent to draw on.

In such legal cases the AI companies will have to account for training incentives that, to quote Christian Catalini, developed “a world class cyber model pitted against a part-time, underfunded blue team”. They’ll have to answer why they commissioned reports into agentic hacking incidents without any cyber security expertise, and why they were so slow to adopt sandboxing by default, the cyber security industry testing practice for more than fifteen years. Most of all, they’ll have to justify how the things they created hurt real people with no help from anyone malicious at all, should that happen. Because that would be crazy.

The AI companies have been projecting the anxieties and insecurities of their project onto the cyber security sector for the best part of six months. As that period has worn on, they have increasingly shown a profound lack of understanding of how the field works. But all the while they have issued warnings of catastrophe and demands for others to act.

Far better, from now on, to engage in a realistic discussion about how to apply ancient principles of societal organisation to the age of agentic AI. Instead of requests for other people to sort out the problems created by the rapid development of their models, the frontier labs could usefully set out some thoughts on how their own accountability, and of those who deploy their tools, could work in practice.

We look forward to the next open letter. Perhaps the ghost of Hammurabi and the ancient Babylonians can help draft it.

Interest duly declared: I am the Director of the SANS Cyber Leaders Network, and co-host, with the aforementioned James Lyne, a podcast put out by the company called the SANS Cyber Leaders Podcast. A special episode of that podcast will see James and I discuss our two pieces. My article is written in a personal capacity and reflects my own personal views only rather than a corporate view of the SANS Institute.